LiLo-VLA: Compositional Long-Horizon Manipulation via Linked Object-Centric Policies

85% vs 0%

success on reordered skill sequences

LiLo-VLA vs Pi0.5 · same Suite-1 tasks, only the order changes

69% / 86%

overall success rate / average progress

Pi0.5 28% / 31% · OpenVLA-OFT 2% / 4%

16 skills

longest task in the benchmark

21 configurations over 9 scenarios · prior LIBERO-Long suites run 3–4

85%

on a real Franka Panda

8 configurations, up to 8 skills, trained on atomic skills only

Overview — sound on

Reordering should be free

A long-horizon task is a sequence of atomic skills, and the same skills usually admit many valid orderings. Put the mug away before the bowl or after it; the skills are identical either way. A policy that has learned the skills should not care which order it is asked for.

Monolithic VLA policies care a great deal. Fine-tuned on the original long-horizon demonstrations of six LIBERO-Long++ tasks, Pi0.5 solves 83% of them. Ask for the same tasks with the skills in a different order and it solves none of them.

The failure has a specific cause rather than a mysterious one: Pi0.5 frequently ignores the re-ordered language instruction and replays the order it was trained on. It has memorised a trajectory, not grounded a command.

LiLo-VLA is never trained on a task-level ordering at all — only on individual atomic skills, each starting from a canonical approach pose that a motion planner is responsible for reaching. No ordering is more in-distribution than another, so the reordered sequences of Suite 1 score no worse than the original ones. Everything below follows from that one decision.

Linking object-centric policies

Execution is split in two. A Reaching Module moves the end-effector from wherever the last skill ended to a per-object approach pose, using a collision-aware motion planner over the scene point cloud (MPLib). An Interaction Module then performs the contact-rich part with an object-centric VLA policy. A geometric verifier checks each skill’s effect, and a closed-loop wrapper decides what to do when it fails.

Architecture. The Reaching Module plans a collision-free path to an approach pose defined in the reference object’s frame; the pose is perturbed during demonstration generation so the downstream policy tolerates planner and perception error. The Interaction Module then executes the atomic skill from a wrist view with non-target objects masked out. A failed skill falls back to the Reaching Module for a reset rather than a blind retry.

Recovery, not retry

Each skill ends with a geometric verifier: LIBERO’s own contact, position and height conditions for a place, displacement or rotation for an articulation, and an added predicate for a pick, which succeeds when the target rises by at least 3 cm.

When a skill fails, what happens next depends on whether the gripper is holding something. A failed pick or articulation is retried locally — the object pose is re-estimated and the Reaching Module returns the end-effector to the approach pose first, so the retry starts from a state the policy trained on. A failed place means the held object’s location is uncertain, so execution backtracks to the most recent pick and re-acquires it. Every skill position keeps its own retry counter that no other skill’s success resets, with a budget of K = 10, which bounds the whole episode at O(NK) failures.

A benchmark of 21 configurations

The LIBERO-Long tasks this builds on stop at three or four skills. The benchmark here keeps that regime and adds one an order of magnitude longer, and every scenario ships with permuted skill orderings — those permutations are what no policy is trained on.

Results in simulation

Two things in that table are worth reading slowly.

Suite 2 is not solved. Both baselines score 0% on every Ultra-Long configuration, and LiLo-VLA reaches 44%. The gap is attributable to coupling: as layouts evolve across stages, a policy that owns transport as well as interaction needs intractable amounts of transition data, whereas delegating transport to a planner leaves the interaction policy invariant to those shifts. But 44% means most 9-to-16-skill episodes still end short of the goal.

Read AP next to SR. Suite 2 averages 44% SR at 79% AP — a typical trial completes about four fifths of its sequence. At these horizons SR collapses each trial to a single bit and discards most of the signal; AP is the more informative, and the more stable, number.

What each module buys

Each ablation removes one component and leaves the rest of the system intact.

Masking matters on top of viewpoint. Removing random erasing costs 21 points overall, 69% to 48%, and takes Suite 2 to 0%. Even a wrist camera admits enough of the surrounding scene that object-centricity has to be enforced explicitly.

Recovery is not just more tries. A naive-retry control keeps the same budget of K = 10 but re-runs a failed skill from wherever the arm ended up, without re-estimating the object pose, resetting to the approach pose, or backtracking to the last pick:

Across the three Ultra-Long scenarios recovery triggers on 26.8% of executed skills and resolves 79.2% of them, and the yield is bounded by the atomic policy underneath it: Cooking Preparation triggers most often, on 37.5% of skills, and resolves only 46.7%, because some picks on geometrically complex objects never succeed.

Why wrist-only. On BOSS-C1, a benchmark that stress-tests policies against observation-space shift:

Real-robot deployment

A Franka Emika Panda with a Robotiq 2F-85 gripper and the dual-camera setup of DROID: a static ZED 2 gives the Reaching Module global context, and a wrist-mounted ZED Mini feeds the Interaction Module. Two things differ from simulation, and both are the point of this section — the backbone is Pi0.5 rather than OpenVLA-OFT, so the framework does not bind to one architecture, and the training data is teleoperated atomic skills only, with no long-horizon sequence data at all.

Every rollout below is real hardware. Clips play as they scroll into view; each carries its task number and protocol in the frame, with the measured success rate overlaid.

Run your own policy on it

The benchmark ships separately from the method. Evaluating your own policy on the 21 configurations needs neither the LiLo-VLA framework nor a GPU-sized dependency tree.

git clone https://github.com/YY-GX/LiLo-VLA.git && cd LiLo-VLA
pip install -e ".[libero]"          # 9 packages, not 39

import lilo_vla.benchmark                      # registers the suites AND patches LIBERO
from libero.libero.benchmark import get_benchmark

bench = get_benchmark("ultra_long")()          # or "libero_long_plus_plus",
                                               # "ultra_long_all", "libero_long_plus_plus_all"
for i in range(bench.get_num_tasks()):
    task = bench.get_task(i)
    env, obs = bench.make_env(i)               # the published evaluation protocol
    # ... roll out YOUR policy ...
    success = env.env._check_success()
    env.close()

make_env() builds the environment exactly as every number on this page was measured, so results are comparable without matching a protocol by hand. Importing lilo_vla also applies the LIBERO compatibility patches the benchmark depends on — three of them change success criteria, so an unpatched LIBERO does not crash, it silently scores the same rollouts differently. The checkpoint and the atomic-skill dataset are released alongside it.

Limitations

• Perception is external. On hardware the Reaching Module plans from detections and 6D poses supplied by YOLOE and FoundationPose, which struggle with transparent or occluded objects.
• Recovery cannot exceed the atomic policy. The backbone’s proficiency at a single skill caps what closed-loop recovery can achieve: with per-skill success $p_i$ and budget $K$, the recovered rate is $1-(1-p_i)^K$, which stays 0 for any $K$ when $p_i$ is 0.
• Verification is geometric, and human on hardware. Each skill is checked against BDDL predicates in simulation; on the real robot an operator judges success, because occlusion prevents reliable automatic verification.

A natural extension integrates an LLM/VLM planner that produces the skill sequence, and evaluates planning and execution end to end.

BibTeX

@article{yang2026lilo,
  title={LiLo-VLA: Compositional Long-Horizon Manipulation via Linked Object-Centric Policies},
  author={Yang, Yue and Cheng, Shuo and Fang, Yu and Bharadhwaj, Homanga and Ding, Mingyu and Bertasius, Gedas and Szafir, Daniel},
  journal={arXiv preprint arXiv:2602.21531},
  year={2026}
}