Decision models on robotics boards, measured
Seven decision models, three Jetson boards: how model choice and execution path change latency, memory, energy and answer quality.
A robot asks itself small questions all the time. Is the gripper holding the cup? Is someone in the way? Did that grasp work, or should it try again? A decision model answers questions like these. It takes the robot's state (text, and for some models an image or audio) and a typed question, and returns a probability for each answer option.
These questions sit inside the robot's loop, so answers have to come back fast, on the computer inside the robot. That raises the question robotics teams already ask about VLAs: how does it run on my board?
The answer depends on three things together: the model, the hardware and the execution path, which is how the model actually runs on that hardware (the runtime or compiler, the precision and the settings). By the time a team asks, the model is usually chosen and the board is often bought. The execution path is what they can still change.
ixian-bench now has results for seven decision models, from d1-omni-600M up to JEV-9B, on AGX Thor, AGX Orin and Orin Nano. Each result puts answer accuracy next to latency, memory and energy per request, with the setup that produced it. Every speedup in this post is measured against PyTorch eager, the reference path on ixian-bench for decision models.
One model, four setups
Take d1-2B, one of the models measured on both AGX Thor and AGX Orin. We ran it on each board in two ways: PyTorch eager, the default way to run a PyTorch model, and a ZETIC build, the same model optimized by ZETIC for that board and run in mixed precision. ZETIC builds are listed separately on the site, under Optimized builds.
Median latency per request (p50).
Energy per request.
Peak board memory above idle, including model loading.
From the slowest setup to the fastest, the same model is about 9× apart: 78.6 ms against 8.6 ms per request. Moving from Orin to Thor accounts for 3.3×, both in eager. Changing only the execution path gives 4.9× on Orin and 2.8× on Thor.
Now compare across the boards. On AGX Orin with a ZETIC build, d1-2B answers in 15.9 ms on 0.73 J per request. On AGX Thor in eager, it takes 23.9 ms and 1.74 J. The previous-generation board with a better execution path answers faster than the newer one in eager, on less than half the energy.
The execution path can even reorder models. On AGX Orin, d1-3B is faster than d1-2B in eager (68.6 against 78.6 ms) and slower with ZETIC builds (22.1 against 15.9 ms). A ranking measured on one execution path doesn't carry over to another.
When the board is too small
Orin Nano has 8 GB of memory, and there the board makes part of the decision for you. The original d1-3B and d1-2B run out of memory on it in the two configurations we tested, PyTorch eager and torch.compile. That says nothing about other runtimes or quantized configurations, which we haven't measured.
| Model | PyTorch eager | ZETIC build | Build accuracy |
|---|---|---|---|
| d1-3B | Out of memory | Runs; latency not reported | 67.0% |
| d1-2B | Out of memory | Runs; latency not reported | 67.2% |
| d1-omni-600M | 50.6 ms | 9.2 ms 5.5× faster | 60.5% |
Out of memory applies to the PyTorch eager and torch.compile configurations tested, not to every runtime or quantized configuration. Original FP32 accuracy was measured on AGX boards: 67.2% for d1-3B and d1-2B; 60.7% for d1-omni-600M. No original accuracy is reported for Orin Nano. Results as of Nov 3, 2026.
Their ZETIC builds run, and score 67.0% and 67.2% on DecisionBench 1,000, against 67.2% for the originals on the larger boards. In the Nov 3 comparison snapshot, latency, energy and memory aren't reported for these builds on Orin Nano, so we claim no speedup there.
d1-omni-600M fits either way, and its ZETIC build is 5.5× faster (50.6 to 9.2 ms) with 73% less energy per request and 53% less peak memory. But it scores lower: the original gets 60.7% on DecisionBench and the ZETIC build 60.5%, against 67.2% for the larger two. On the smallest board, the choice is a smaller model, or a larger one on an execution path that fits in 8 GB.
Picking the model
The model is a trade-off of its own. The FP32 originals of Perplexity Flash-4B and d3-mini-4B answer about 78% of DecisionBench questions correctly, against 67.2% for d1-3B. Running in PyTorch eager on AGX Orin, they take about twice as long per request (about 144 ms against 68.6 ms) and need more memory (11.5 and 11.8 GB against 7.9 GB).
| Model | Answer accuracy (%) | Latency p50 (ms) | Peak memory (GB) |
|---|---|---|---|
| d3-mini-4B | 78.2 | 144.1 | 11.8 |
| Perplexity Flash-4B | 78.1 | 143.8 | 11.5 |
| JEV-9B | 75.3 | 144.9 | 20.8 |
| d1-3B | 67.2 | 68.6 | 7.9 |
| d1-2B | 67.2 | 78.6 | 7.4 |
| d1-omni-600M | 60.7 | 40.6 | 4.3 |
Higher accuracy is better; lower latency and memory are better. Peak memory is board memory above idle. Kev-4B has no published AGX Orin result in this snapshot.
The execution path changes that trade. Perplexity Flash-4B's ZETIC build answers in 56.8 ms at 78.3% accuracy: faster than d1-3B in eager, and 11 points more accurate. Say the robot needs an answer within 100 ms on AGX Orin. In eager, only the three d1 models fit, at 67.2% accuracy or less. With ZETIC builds, Perplexity Flash-4B (56.8 ms), d3-mini-4B (59.4 ms) and JEV-9B (70.0 ms) fit too, at 75% to 78%. One caveat: this uses median single-question latency, one question per request on an otherwise idle board. Latency inside a real robot loop, with perception and control running alongside, will differ and has to be measured in that loop.
The headline score isn't the whole story either. Robot frames, the question set drawn from robot datasets (still frames only), splits results into three roles that are never blended: scene understanding (RoboBench, ERQA), success detection (RoboReward, UR5-Fail) and safety judgment (ASIMOV, HomeSafe). d3-mini-4B scores 78.2% on DecisionBench, but 61.2% on scene understanding, 81.9% on success detection and 65.7% on safety judgment. Check the role your robot depends on, not only DecisionBench.
Do the answers hold?
A faster decision model only helps if its answers hold up. ixian-bench checks two things: answer accuracy against the gold answers, and agreement, the share of questions where the optimized build picks the same top answer as the FP32 original model. Both are measured on DecisionBench 1,000, and a similar aggregate score doesn't mean identical outputs on every question.
| Model | Speedup | Same top answer (%) | Accuracy: original → build (%) |
|---|---|---|---|
| d1-omni-600M | 7.5× | 94.8 | 60.7 → 61.2 |
| d1-2B | 4.9× | 97.8 | 67.2 → 67.1 |
| d1-3B | 3.1× | 98.2 | 67.2 → 66.5 |
| Perplexity Flash-4B | 2.5× | 97.4 | 78.1 → 78.3 |
| d3-mini-4B | 2.4× | 98.3 | 78.2 → 78.1 |
| JEV-9B | 2.1× | 97.1 | 75.3 → 75.4 |
Same top answer is agreement with the FP32 original, not answer accuracy or a guarantee of identical outputs on other tasks.
On AGX Orin, the ZETIC builds of six models, from d1-omni-600M to JEV-9B, run 2.1× to 7.5× faster than eager. They pick the same top answer as the original on 94.8% to 98.3% of DecisionBench questions, and DecisionBench accuracy moves by 0.7 points or less. That is a result on this question set, not a guarantee for every task: an optimized build still has to be checked on the questions your robot actually asks. On Robot frames the gains are smaller, 1.5× to 2.1× on AGX Orin; d1-2B goes from 454.6 ms to 215.1 ms per request.
Three moves
Put together, a team usually has three moves:
- A bigger board, when no execution path on the current one meets the loop's latency or memory needs.
- A different model, when another one scores better on the role your robot depends on and still fits the budget.
- A better execution path on the board and model you already have. It's the only move that keeps both, and on Orin Nano it's the only path we measured that runs the larger d1 models.
Accuracy, memory and energy decide which move is acceptable. ixian-bench publishes the measurements and the conditions, and the call stays with you.
How we measure decision models
The full method is on the Methodology page. These are the parts that matter when you compare results:
- Latency runs from the raw input to a probability for each answer option, at batch 1, in milliseconds per request. The headline number, p50, is the median of four round medians.
- Single question, one question per request, is the headline scenario. Robot frames uses still frames from the role datasets and skips requests over 3,072 tokens.
- The timing baseline is the original model in PyTorch eager (BF16, or FP16 where noted). Answer accuracy and top-1 agreement use the FP32 original as the quality reference; this is distinct from the timing baseline.
- DecisionBench 1,000 is the one question set that compares directly across models. Robot-frames scene-understanding question sets differ between d1-2B and d1-3B, so those scores should not be compared directly.
- Anything not measured is marked as not reported. Failure diagnosis is held while its sources and labels are reviewed. These are benchmark results, not tests on a physical robot.
When your current setup falls short
Sometimes the baseline doesn't meet the robot's needs. The answer is too slow for the loop, or the model doesn't fit next to everything else on the board. That's the work ZETIC does: we improve the execution path on your target hardware, then validate the result against your quality targets, so a speedup only counts if quality holds on your task.
Optimized builds on ixian-bench puts our runs next to the reference results on the same board, labeled separately, with the precision, quality result and conditions for each. Check each build's reported quality result against the requirements of your task.
JEV-9B, the largest decision model in this comparison, shows one such result. On AGX Orin our build is 2.1× faster, uses 56% less peak memory (20.8 GB to 9.1 GB) and 48% less energy per request, and picks the same top answer as the FP32 original on 97.1% of DecisionBench questions.
If your model or board isn't on the site yet, including a fine-tuned version, start with the Benchmark your model form on ixian-bench. Add your target board and your goals for latency, energy and memory.
Compare decision models on ixian-bench, with the conditions behind each result.
Measurements and dashboard screenshots as of Nov 3, 2026. Reported results describe the tested workloads and configurations, not performance inside a complete robot loop.
Exception to the 95% agreement target: d1-omni-600M on AGX Orin recorded 94.8% top-1 agreement with the FP32 original on DecisionBench 1,000.