Sensing Which Modality Matters: Evidence-Gated Regularization for Robust VLA Policies

30 → 85%

under unseen physical distractors

bi-manual real robot, +183%

2.8 → 6.1%

when only one useful sensor survives

simulated manipulation, +120%

zero

added inference cost

a training objective only

47

benchmark skills for modality entanglement

BEHAVIOR-1K · 11 NAV + 36 MAN

Three-minute overview — sound on

Modality entanglement

VLA policies build on large-model backbones that fuse vision, language, and proprioception early. Early fusion is known to invite shortcut learning, and robot demonstrations — small, and narrow in operator, environment, and object diversity — make it worse. The result is a policy whose behavioral dependence on a sensor is misaligned with how much that sensor actually matters at the current state.

I. Nuisance sensitivity

II. Single-modality insufficiency

Evidence-Gated Regularization

EGR separates how task-relevant each sensor is at each frame from how that relevance shapes the policy. The first part is modality-specific; the second is shared across every sensor type. Nothing changes at inference — EGR is a training objective only.

Invariance loss — low-evidence sensor corrupted

Sufficiency loss — only the high-evidence sensor preserved

The evidence score

For each frame $t$ and sensor $m \in \mathcal{M}$ we assign an evidence score $E_{t,m} \in [0,1]$, normalized across modalities so it captures relative importance. It is computed from task structure rather than learned end-to-end from the very demonstrations that caused the problem. It is a heuristic measure of task relevance, not ground truth — its job is to indicate which failure mode a frame is susceptible to, so the right regularizer fires.

• Vision. Evidence is the amount of task-relevant geometry in view. With per-frame instance segmentation and task annotations of focal objects (what is manipulated) and interaction objects (the target or receptacle), the per-camera score is $s_{t,m} = a^{\mathrm{focal}}_{t,m} + \alpha \cdot g_{t,m} \cdot a^{\mathrm{inter}}_{t,m}$, where the gate $g_{t,m}$ admits interaction objects only when the focal object is visible in the same camera — otherwise a large table would inflate cameras that never see the focal object. Cross-camera normalization gives $E_{t,m}$, and a global gate $G_t$ disables both losses on search frames where no camera carries meaningful evidence.
• Tactile. In the sparse-contact regime (grasping, pressing, insertion) contact is the task-relevant event, so $E_{t,\mathrm{tactile}}$ is a binary contact indicator. In the rich-contact regime, where contact persists throughout, a task-feature operator $\phi_{\mathrm{task}}$ scores the current tactile reading against reference signatures from demonstrations.

Two gated objectives

The score selects which failure mode each frame is susceptible to, and therefore which consistency to enforce:

$\mathcal{L}_{\mathrm{inv}}$ fires on low-evidence sensors: corrupting what does not matter must not change the action. $\mathcal{L}_{\mathrm{suff}}$ fires on high-evidence sensors: that sensor alone must be enough to act. Both ride on top of the base flow-matching imitation loss,

Because the weights are hinges rather than hard constraints, evidence acts as a soft inductive bias — the policy stays free to find extra structure in any sensor.

Two forward passes, not 2|M|

Written literally, both losses need a corrupted forward pass per sensor, so cost would grow with the number of sensors. An importance-sampled estimator collapses this to two extra passes per step regardless of $|\mathcal{M}|$: instead of summing over every modality, sample one according to the evidence-derived weights and reweight. The appendix proves both estimators unbiased.

Benchmark

We build a benchmark on BEHAVIOR-1K targeting modality entanglement directly: 47 curated skill segments — 11 navigation and 36 manipulation — drawn from 11 usable tasks and filtered from 118 candidates by three vision-evidence criteria, with per-segment initial simulator states recovered by demonstration replay.

Results in simulation

Both suites agree. Vanilla π₀.₅ shows the entanglement, and EGR reduces it without giving up clean-input performance — manipulation success under full modalities actually goes up, suggesting evidence gating also helps the policy learn more task-relevant features in the first place.

Suite 2 — task success

The headline row: manipulation SR rises 12.5 → 16.4 under full modalities (+31%), 9.4 → 16.5 under NoUseless (+75%), and 2.8 → 6.1 under UsefulOnly (+120%). On navigation, clean performance is essentially preserved (42.3 → 40.9) while SingleUseful more than doubles, 15.5 → 37.3 (+141%).

These absolute rates are low for every method, and deliberately so: these are long-horizon BEHAVIOR-1K skills that no current VLA solves reliably. The comparison to read here is between methods on identical segments, not a claim about deployment reliability.

Suite 1 — action deviation

The vanilla rows are the diagnosis: perturbing sensors the filter labeled useless still moves the action distribution by +31.4% (NAV) and +12.7% (MAN), and dropping to the single useful sensor moves it by +158.5%. ModDrop shows the same pattern — random modality dropout does not remove the entanglement. EGR lowers every ratio.

Read the Δ columns, not the absolute ones. Each method’s deviation is measured against its own Full baseline, so absolute magnitudes are not comparable across rows — EGR’s absolute deviation is in fact higher than vanilla’s, which says nothing about how much a perturbation moves it.

Real-robot validation

We test two platforms with deliberately different embodiments and sensor sets, plus a RealDistractor condition in which a physical out-of-distribution object enters the camera’s view — the only non-synthetic corruption in the evaluation.

Vision-tactile platform

Bi-manual platform

Bi-manual vision platform

Two Kinova arms and three RGB cameras, on four tasks: stack bowls, put drumstick on plate, put sushi in pot, and close microwave door. Four conditions, 10 trials each.

Vision-tactile platform

One MELFA ASSISTA arm, one RGB camera, and two GelSight sensors, on two tasks: reading a pencil’s serial code and localizing a board crack. Three conditions, 10 trials each.

The largest relative gain on both platforms comes on RealDistractor, where the distractor is a physical object the policy has never seen — the strongest out-of-distribution signal available to us. The tactile platform confirms that the gains carry to a sensor combination the vision instantiation never touches.

Limitations

• Evidence must be specifiable in advance. EGR needs a task-grounded relevance signal that can be written down from task structure. It does not cover settings where relevance must itself be learned — in-hand object reorientation, for instance.
• “Modality-agnostic” is demonstrated across image-like sensors. The regularizer is sensor-agnostic once evidence exists, but both instantiations here consume images: an RGB camera and a GelSight, whose output is also RGB. Depth, force, and audio — where the erasing corruption operator is not even well defined — remain untested.
• The benchmark covers a slice of BEHAVIOR-1K. 47 segments drawn from 11 usable tasks, filtered by vision-evidence criteria that share a visible-geometry root with EGR’s own vision evidence.
• The evidence score is a hand-designed proxy. Visible pixel area is not the same as information value, and the score is treated as a heuristic throughout.

BibTeX

@article{yang2026sensing,
  title={Sensing Which Modality Matters: Evidence-Gated Regularization for Robust VLA Policies},
  author={Yang, Yue and Romeres, Diego and Hori, Chiori and Bertasius, Gedas and Szafir, Daniel and Jain, Siddarth},
  journal={arXiv preprint arXiv:2609.03142},
  year={2026}
}