How To Evaluate A VLA Policy: Benchmarks, Rollouts, And Reproducibility

Evaluation Is The Bottleneck, Not Training

Training a vision-language-action policy has gotten easier. Open checkpoints, open datasets, and a handful of solid recipes mean a lab can get a reasonable policy running in a simulator within a week. Knowing whether that policy is actually good is the hard part.

Most teams still evaluate the way robotics has always evaluated: a hand-built harness for one benchmark, a couple hundred rollouts run once, and a single aggregate success-rate number reported in a paper or a Slack message. That number is usually noisier than anyone treats it as, it rarely says which tasks are actually failing, and it is almost never reproducible by anyone outside the lab that ran it.

This guide is a working checklist for evaluating a VLA policy properly: which benchmarks to run and what they actually measure, how many rollouts you need before the number means anything, what to publish so someone else can reproduce your run, and how to read failures instead of just the top-line score.

Pick Benchmarks That Test Different Things

"Run it on LIBERO" is not a specification. LIBERO and RoboCasa test different capabilities, and a policy that is strong on one can be mediocre on the other. Treat benchmark choice as part of the experiment design, not an afterthought.

LIBERO: Knowledge Transfer, Not Just Manipulation

LIBERO was built to study lifelong robot learning: whether a policy can transfer knowledge across tasks rather than just perform one task well (Liu, Zhu, Gao, Feng, Liu, Zhu, and Stone, NeurIPS 2023). That framing matters for how it's structured. The benchmark separates two kinds of knowledge a manipulation policy needs: declarative knowledge (facts about objects and layouts) and procedural knowledge (the actual motor behaviors).

LIBERO ships as four task suites, each isolating a different axis of variation:

• LIBERO-Spatial holds the objects and goal fixed and varies where they're placed, testing whether a policy generalizes across spatial layout rather than memorizing one arrangement.
• LIBERO-Object holds the layout fixed and swaps in different objects, testing generalization across object identity.
• LIBERO-Goal holds the objects and layout fixed and varies the task goal, testing whether a policy can follow different instructions on an otherwise identical scene.
• LIBERO-100 is a pool of 100 tasks (split into a 90-task training set, LIBERO-90, and a 10-task held-out set, LIBERO-10) with entangled objects, layouts, and goals, closer to how a generalist policy actually gets stress-tested.

Each of LIBERO-Spatial, LIBERO-Object, and LIBERO-Goal has 10 tasks, and every task ships with 50 human-teleoperated demonstrations for training. The standard evaluation protocol runs roughly 50 rollouts per task, so a full suite is on the order of 500 rollouts, not the 20 or 30 spot-check runs a lot of papers quietly settle for.

The practical takeaway: if you only report one LIBERO number, say which suite it's from. A strong LIBERO-Object score and a weak LIBERO-Spatial score describe two different failure modes (object generalization is fine, spatial generalization is not) and averaging them into one figure erases that signal.

RoboCasa: Long-Horizon Tasks In A Real Kitchen Layout

RoboCasa is a different kind of test. It's a large-scale simulation of everyday household environments, built on an extended RoboSuite running on the MuJoCo physics engine (Nasiriany, Maddukuri, Zhang, Parikh, Lo, Joshi, Mandlekar, and Zhu, RSS 2024). Where LIBERO isolates specific generalization axes on a tabletop, RoboCasa asks whether a policy can operate in a full kitchen: 120 distinct scenes generated from combinations of floor plans and kitchen styles, populated with thousands of interactable 3D objects across more than 150 categories.

RoboCasa's benchmark suite has 100 tasks, split into 25 atomic tasks (pick-place, opening and closing, turning knobs, pressing buttons) and 75 composite tasks that chain atomic skills into the kind of multi-step activity a household robot would actually need to do, like clearing a counter into a sink or putting away groceries. The composite tasks were generated with LLM assistance, which is part of what lets the benchmark scale past what a team could hand-author.

For a VLA policy, RoboCasa is the closer proxy for "can this thing do a chore," and LIBERO is the closer proxy for "does this policy generalize instead of memorize." Neither answers the other's question. If your research claim is about generalization, don't lean on RoboCasa's composite tasks to make the case, and vice versa.

Isaac Lab, MuJoCo playgrounds, Genesis scenes, and your own custom tasks each add a third axis: physics fidelity and rendering realism, which matters most when you're trying to say something about sim-to-real transfer rather than sim-only performance. A recent benchmarking paper out of NVIDIA lays out the case directly: simulation benchmarks have driven real progress in manipulation research, but evaluating whether that progress holds up in the real world has lagged behind, and the paper argues for higher visual fidelity, systematically increasing task difficulty and perturbation, and explicitly measuring how well sim performance predicts real performance rather than assuming it does (Yang, Eppner, Tremblay, Fox, Birchfield, and Ramos, 2025).

How Many Rollouts Do You Actually Need?

This is the part most eval writeups skip, and it's the part that decides whether your number means anything. Success rate is a binomial proportion. A binomial proportion measured on a small sample has a wide confidence interval, and that interval doesn't visibly shrink just because the point estimate looks clean.

Recent work auditing the field's own practices found that most published evaluations of VLA policies report a single binary success rate at a fixed timeout, with 25 rollouts or fewer per condition, no confidence interval, and no paired statistical test between policies (see the evaluation-methodology discussion in recent robot-policy-comparison work, e.g. the PhAIL benchmarking paper, arXiv:2605.29710). That's an order of magnitude below what's needed to say two policies are actually different rather than different by chance.

The math is unforgiving. NVIDIA's own writeup on evaluating general-purpose policies works the numbers directly: an observed 90% success rate measured over 70 rollouts carries a 95% (Clopper-Pearson) confidence interval spanning roughly 80.5% to 95.9%, a 15-point-wide band around a number that looks precise on the page. Tightening that to roughly plus or minus 2 points takes about 15x more rollouts, on the order of 1,000 instead of 70; halving it again to a true 2-point-wide band pushes you past 3,000 (NVIDIA Technical Blog). Toyota Research Institute has made a similar case for treating policy comparisons like proper A/B tests rather than reading two point estimates off a table (Toyota Research Institute, Medium).

What this means in practice:

• Report a rollout count, every time. A success rate with no denominator is not a result.
• A few hundred rollouts is a floor, not a target, if you want to distinguish two policies that are within a few points of each other, which describes most real comparisons.
• A single seed is not a run. Simulator randomness, initial object poses, and policy sampling temperature all move the number. Vary seeds and report the spread, not just the mean.
• Don't rank policies on a difference smaller than the confidence interval. If policy A beats policy B by 2 points and your interval is 10 points wide, you don't have a result yet, you have noise.

None of this is exotic statistics. It's the same reasoning any A/B test needs, applied to a domain that has mostly skipped it because rollouts used to be too slow and too expensive to run at the volume the reasoning requires.

A Reproducibility Checklist

A policy evaluation that can't be rerun by someone else isn't a benchmark result, it's an anecdote. Robot sim benchmarks have a specific set of things that quietly drift between runs and between labs:

1. Pin the simulator version. LIBERO, RoboCasa, Isaac Lab, and MuJoCo all ship updates that change physics, rendering, or task logic in ways that move success rates. Record the exact commit or release tag you evaluated against, not just "LIBERO."
2. Fix and report your seeds. Publish the seed list, not just "we averaged over multiple seeds."
3. Report per-suite and per-task numbers, not just an aggregate. A LIBERO-100 average of 70% could be a policy that's uniformly mediocre or a policy that's excellent on 80 tasks and fails completely on 20. Those are different bugs with different fixes, and an aggregate hides the difference.
4. Publish your success criteria and episode length. Two papers can both report "success rate on LIBERO-Object" and mean different things, because they used different max-episode-step budgets or different definitions of task success (final object pose vs. gripper state vs. a scripted checker). State the exact criterion.
5. Separate training checkpoint selection from evaluation. Picking the checkpoint that scores highest on your test rollouts and then reporting that same rollout's score is circular. Select on a held-out validation split, or at minimum say plainly that you didn't.
6. Make the run itself citable. A results table in a paper appendix is not something another lab can point at, rerun, or diff against their own policy. A run with a permanent identifier, method, seed list, and simulator version attached is.

None of these are individually hard. Most eval writeups skip several of them anyway, because doing this by hand across five benchmarks and a few thousand rollouts is genuinely tedious.

Look Past The Aggregate Score

A single success-rate number tells you that something is wrong, not what. Two policies that both score 62% on RoboCasa can be broken in completely different ways: one might fail every task that involves opening an articulated door, and the other might fail every task that needs a two-step sequence longer than four actions. Averaging those into the same number makes them look identical, and the fix for each is unrelated.

The more useful unit of analysis is the failed episode, clustered by what actually went wrong: grasp failure, wrong-object selection, premature task termination, getting stuck against furniture, a plan that never gets past step one. Grouping failures this way turns "62%" into "opening articulated doors is the problem," which is something you can actually go fix in the next training run. It also tells you where a benchmark's difficulty is concentrated, which suites are actually discriminating between good and bad policies, and which composite tasks are secretly gated on one atomic skill.

This kind of analysis is normal practice for anyone who has sat through the tedium of watching a few hundred rollout videos by hand. It's rarely done at the scale a real benchmark run demands, because doing it by hand across a few thousand episodes just doesn't scale.

How Manifold Automates This

Bifrost built Manifold because we kept re-deriving this checklist by hand, per policy, per benchmark, per lab. Manifold is one harness across LIBERO, RoboCasa, Isaac Lab, MuJoCo, Genesis, and custom scenarios, so switching benchmarks doesn't mean rebuilding the evaluation pipeline from scratch.

It addresses the rollout-count problem directly by sharding rollouts across GPUs, so runs on LIBERO come back roughly 8x faster than a single-GPU baseline. That makes a statistically defensible rollout count (hundreds to thousands, not 25) something you can run before lunch instead of something you skip because it's inconvenient.

Every run is checked against verified baselines on the same benchmark, and every run gets a citable manifold:// URI, so a result isn't just a number in a spreadsheet, it's something another lab can point at and reproduce. And instead of stopping at an aggregate score, Manifold clusters failed episodes by failure mode automatically, so you see the specific task families breaking your policy rather than a single percentage.

Manifold is free for research partners, and an open-source release of the runner, harness, and leaderboard schema is coming. If you're evaluating a VLA policy and want the rollout count, reproducibility, and failure analysis above without building it yourself, get in touch through the robotics page or get early access at manifold.bifrost.ai.

Also Read

• Top Datasets For Maritime AI, USV Perception & Obstacle Detection
• Building The Data Engine To Power Physical AI