Is LIBERO Saturated? What Top Scores Hide And What To Report

Is LIBERO Saturated?

For the standard protocol, largely yes. As of October 2026, the strongest open VLA models report LIBERO averages between 96.85% and 98.1% across the four standard suites, which leaves less than 4 points of headroom and puts the gaps between top models inside the confidence interval of a 500-episode suite. The perturbation benchmarks show what that ceiling hides. On LIBERO-PRO, OpenVLA, pi0 and pi0.5 score 0% to 1% when the task is modified, and OpenVLA and pi0 score 0% when object positions move, and on LIBERO-Plus OpenVLA-OFT drops from 97.1% to 37.2% when the robot's initial state changes. LIBERO is still a useful regression test, but it no longer separates the best policies, and it says little about generalization on its own.

The rest of this post shows the ceiling with sourced numbers, summarizes what the two perturbation benchmarks found, lists what still discriminates between policies, and ends with a short recommendation for what to report next to a LIBERO score.

The Ceiling On Standard LIBERO

These are the four-suite averages reported by each source, with LIBERO-10 shown separately because it is where most of the remaining spread sits.

OpenVLA's 76.5% average was added to its paper in September 2024, and by early 2025 OpenVLA-OFT reported 97.1%. Two facts about the protocol explain why the top rows bunch up.

The first is sample size. A standard suite is 10 tasks with 50 episodes each, so 500 episodes. At a true success rate of 97%, the 95% confidence interval on a 500-episode estimate is about plus or minus 1.5 points (normal approximation), and at LeRobot's recommended 10 episodes per task it is wider still. The difference between 96.85% and 98.1% is smaller than that margin, which means the ordering among the top five rows above is not established by the published numbers.

The second is what the test measures. The LIBERO-PRO authors point out that LIBERO's evaluation tasks are the same tasks used for training, and that the only difference at test time is a small change in the initial placement of objects. Each task ships a fixed set of initial states, and the standard scripts evaluate the same 50 of them every time. A policy can score in the high 90s by learning a reliable mapping from these scenes to these trajectories. LIBERO was designed to study knowledge transfer in lifelong learning (Liu et al., 2023), and the high-90s scores come from fine-tuning a large pretrained model on each suite and testing on the same suite, which is a different use from the one the benchmark was built for.

What LIBERO-PRO Does To Top Models

LIBERO-PRO (Zhou et al., 2025) keeps the LIBERO scenes and perturbs five things: the manipulated objects (appearance, size, color), their initial positions, the instruction wording, the task itself (new goals built from the same scene), and the environment. It evaluates 50 episodes per task, matching the original protocol. The table below gives the range across the four standard suites, from the original-task results and the leaderboard in the project README.

The authors read the stable rows with caution. They report that models keep grasping when the target object is replaced with an irrelevant item, and that outputs stay unchanged when the instruction is corrupted or replaced with nonsense tokens, so the high scores under paraphrased instructions may mean the policy ignores the instruction rather than understands it. The one position result above zero, pi0.5 at 0.38 on LIBERO-Goal, is the kind of difference between models that the standard benchmark hides, since all three score above 0.9 on the original tasks.

LIBERO-PRO's task and position perturbations are large by design, and a 0% score under a modified task partly reflects that the policy was never trained on that task. The finding that holds up is that the same models cannot recover when objects start somewhere new in a scene they have seen thousands of times.

What LIBERO-Plus Does To Top Models

LIBERO-Plus (Fei et al., 2025) takes a finer-grained approach. It perturbs seven factors (objects layout, camera viewpoint, robot initial state, language instruction, lighting, background texture, and sensor noise) across 21 sub-dimensions, and grades each variant into five difficulty levels. The released benchmark has 10,030 tasks. Table 1 of the paper reports success rates (%) for each perturbation type:

Camera viewpoint and robot initial state are the hardest factors, and the authors attribute that to both requiring an understanding of spatial geometry and proprioception, while lighting and background are easier. Rankings also reverse under perturbation, as pi0-FAST sits 8.7 points below pi0 on original LIBERO but scores 66.4% to pi0's 15.8% under camera changes. In a blank-instruction experiment, OpenVLA-OFT's performance on LIBERO-Object stayed largely unchanged with no language input at all, and dropped substantially only on LIBERO-10. The authors conclude that the model behaves more like a vision-action policy than a vision-language-action one.

LIBERO-Plus also has headroom that standard LIBERO lacks. Across its full benchmark (Table 2), totals range from 15.6% for OpenVLA to 69.6% for OpenVLA-OFT, and the authors' own post-training on more than 20,000 perturbed trajectories reaches 79.6%.

What Still Discriminates Between Policies

LIBERO-10. The long-horizon suite is the one standard suite with a wide spread left. Reported LIBERO-10 scores run from 53.7% (fine-tuned OpenVLA) and 60.2% (pi0-FAST, as compiled by OpenVLA-OFT) through 71% (SmolVLA) to 94.5% (OpenVLA-OFT) and 97.6% (X-VLA). LIBERO-Plus also found it was the only suite where removing the instruction hurt OpenVLA-OFT substantially. If you report one standard suite to show a difference between two policies, this is the one.

Perturbation suites. LIBERO-PRO and LIBERO-Plus reuse LIBERO's scenes and assets, so they cost little extra setup if you already run LIBERO, and both separate models that tie on the original benchmark. LeRobot already supports LIBERO-plus as an environment (--env.type=libero_plus), with the caveat in its docs that installing it replaces vanilla LIBERO in the same Python environment.

Benchmarks with harder task structure. RoboCasa is far from saturated. In the GR00T N1 paper, GR00T-N1-2B averages 32.1% across 24 RoboCasa kitchen tasks with 100 demonstrations per task and 49.6% with 300, and GR00T N1 stays below 15% on individual tasks such as opening a double door. The newer RoboCasa365 release has 365 tasks across 2,500 kitchen environments and is also available through LeRobot. When a policy scores 97% on LIBERO and struggles on RoboCasa, the gap shows limits that LIBERO alone would not have revealed.

Real robots. Simulation scores, perturbed or not, are a proxy. The case for measuring how well sim performance predicts real performance, rather than assuming it, is made well in recent NVIDIA work on sim benchmarking (Yang et al., 2025), which we discuss in How To Evaluate A VLA Policy.

What To Report Alongside LIBERO

If you publish or circulate a LIBERO result, the following makes it informative even when the headline number is near the ceiling.

1. Per-suite scores, with LIBERO-10 shown separately, plus the number of episodes per task and the seeds. A four-suite average of 97% says less than a LIBERO-10 score with its confidence interval.
2. The evaluation pipeline. Name the script and commit, the max steps per suite, the action chunk length executed per query, and the image preprocessing, including whether frames were rotated 180 degrees. These details move standard LIBERO scores by several points, as covered in How To Run LIBERO Evaluation and Your Robot May Be Learning The World Flipped.
3. At least one perturbation result. LIBERO-Plus or LIBERO-PRO, broken out by factor. Camera viewpoint, robot initial state and object position are the factors that have separated models most in published results.
4. A blank or corrupted instruction check. If the score does not move when the instruction is removed, the policy is not using language on that suite, and any claim about instruction following needs other evidence.
5. One benchmark outside the LIBERO family, such as RoboCasa, for any claim about general manipulation ability.

Where Manifold Fits

Manifold is Bifrost's platform for evaluating robot policies in simulation. Its benchmark list includes LIBERO and LIBERO-Plus alongside RoboCasa, RoboMimic, CALVIN, SIMPLER, RoboTwin 2.0 and RLBench, run through one harness, so adding a perturbation suite or a second benchmark next to LIBERO does not mean building a second evaluation pipeline. Rollouts are sharded across GPUs, which brings a LIBERO sweep that takes 8 hours on a single GPU down to about 30 minutes (roughly 16x faster) and makes larger episode counts practical. Manifold also clusters failed episodes by failure mode, which is the view you want when a policy that scores 97% on LIBERO drops under camera or initial-state perturbations. If you only need LIBERO-Plus for one checkpoint, the LeRobot integration above works well. Manifold is in early access through the waitlist on the Manifold page, and Introducing Manifold explains why we built it.

Sources

• Liu et al., LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning
• Zhou et al., LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization, and the LIBERO-PRO repository and leaderboard
• Fei et al., LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models
• Kim, Finn and Liang, Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success (OpenVLA-OFT)
• Zheng et al., X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Model
• Physical Intelligence, openpi LIBERO example
• Shukor et al., SmolVLA
• NVIDIA, GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
• Hugging Face, LeRobot LIBERO and LIBERO-plus documentation