Zero-Shot Generalization Limits in Manipulation Robots

Zero-shot generalization in manipulation robots means a policy trained on certain objects, scenes, and tasks should handle new ones without anyone retraining it. That's the pitch behind every generalist robot policy paper released in the last two years, and it holds up in one narrow sense while getting badly oversold in three others. Vision-language-action models, the architecture running the show right now, genuinely do generalize across language instructions and object categories. But a model scoring above 95% on LIBERO can still fail on a table it's never seen, in ways that have nothing to do with that benchmark number.

The confusion starts with a definitional problem nobody's settled: what counts as "in-distribution" for a VLA model isn't fixed the way it is for an image classifier. A benchmark score can look bulletproof and still hide brittleness, because the test set and the deployment world don't share the same idea of what counts as new. Once you dig into where these models actually break, the failures aren't random noise scattered across every input. They cluster along three distinct planes: visual and perceptual, skill and motion, and compositional. Lump them together and the diagnosis stays stuck at "the robot needs more data." Split them apart, and something closer to an actual fix shows up. That split is the whole point of this piece, and it's worth saying upfront which of the three is the one most people get wrong: compositional failure, the one many assume scale will eventually fix, is the one published evidence suggests scale does not touch.

How the VLA architecture creates a built-in ceiling on physical generalization

VLAs work by taking a pretrained vision-language model, the kind that already knows what a "shoelace" or a "bowl" looks like from scraping the internet, and bolting on an action head that predicts motor commands. The language side is doing real work here. Ask a VLA to find "the red mug" in a pile of objects it's never trained on, and there's a decent chance it succeeds, because that semantic association came pre-baked from the vision-language pretraining.

Here's the catch: that pretraining teaches what to do, not how to do it. No representation of contact dynamics sits inside a captioned image dataset, and no spatial geometry gets learned from a caption like "a man tying his shoe." So a VLA can correctly parse "untie the shoelace" as the task, look at the right shoelace, and still have no idea how to move a gripper to actually untie it, because that exact motion was never in the robot's own training data.

This matters a lot for a certain class of planning system. Frameworks like SayCan and Code-as-Policies chain high-level reasoning together with low-level skill execution, assuming the execution layer just works once the plan is correct. One misfired motion primitive and the whole chain snaps, the way a car runs fine until the timing belt slips and suddenly nothing downstream matters.

Recent work makes the sharpest claim here, and it's worth building an argument around: this isn't a data-coverage problem fixable by throwing more examples at it. A model can see substantial additional in-distribution demonstrations and still lack the representational structure needed for inductive generalization, meaning the ability to extrapolate a rule rather than memorize a pattern. That's an architectural ceiling, not a data shortage. Anyone still pitching "just scale the dataset" as the fix for compositional failure is arguing against evidence that already exists, and that's exactly why the three planes below need three separate diagnoses instead of one scaling prescription.

The visual and perceptual plane: how appearance shifts degrade policy execution

Start with the floor, because it's steep. THE COLOSSEUM, a 2024 evaluation, tested five state-of-the-art manipulation models and found success rates drop 30 to 50% under individual perturbation factors, things like changed lighting or a shifted camera angle. Stack multiple perturbations at once (background plus lighting plus object layout) and degradation climbs past 75%. That's most of a model's competence gone because someone moved a lamp.

What counts as a "perturbation" here is broader than most people assume: background textures, lighting, camera viewpoints, sensor noise, and object arrangement all get tested systematically in benchmarks like LIBERO-Plus and RoboTwin-Clean2Rand. RoboTwin 2.0 pushes the coverage question further by throwing in distractor clutter drawn from a pool of 731 objects across 147 categories, which gives some sense of how large the space of "things that could visually confuse a robot" actually is.

One named failure mode is worth sitting with: the perceptual shortcut, described in AC-VLA research. Action tokens end up over-anchored to wrist-view camera textures that were tightly tied to specific objects during training. Change the task configuration even slightly, and the model keeps staring at the same close-up visual cues instead of reading the broader spatial layout it needs for accurate placement. Picture a delivery driver who memorized a route by counting stop signs instead of reading street names. Change one intersection and the whole route falls apart.

LIBERO-PRO, from 2025, gives the sharpest concrete example here. Swap a salad dressing bottle for a can of alphabet soup, and a model told to "pick up the salad dressing" will still reach for and grab the wrong object entirely, confidently executing the wrong grasp on the wrong item. That's not pixel noise confusing a sensor. That's a model that learned a spurious co-occurrence, salad-dressing-shaped-object-in-that-spot, instead of the actual concept "salad dressing."

Sim-to-real transfer is really just this plane wearing a different hat. Simulated training data is cheap to generate at scale, but the gap between simulated and real appearance, geometry, and material behavior blocks a lot of that value from transferring. CMU's SplatSim project, presented at a CoRL workshop, tested something clever: keep the MuJoCo physics engine untouched, but swap the mesh-based rendering for Gaussian Splats, a rendering method that produces far more photorealistic scenes. That single change pushed zero-shot sim-to-real success to 86.25%, against 97.5% for models trained on real data. The rendering layer, not the physics, accounted for most of the remaining gap.

The skill and motion plane: why semantic competence does not transfer to novel physical execution

Here the failure looks different. The policy correctly names the task. It looks at the right object. It still can't do the thing, because the motor primitive required simply doesn't exist anywhere in its training data.

NVIDIA's DREAMZERO research, released in February 2026 as part of a broader push into what it calls World Action Models, frames this precisely: a VLA fails at "untie the shoelace" not because it misunderstands the instruction but because no physical representation of that motion was ever learned. The model's action space just doesn't contain it, the same way a phone number never memorized can't be recalled no matter how hard someone tries.

The proposed fix is worth unpacking, because it changes what the model actually learns. Instead of training a policy to copy state-action pairs directly, DREAMZERO jointly predicts future video frames and the actions that produce them, treating video as a dense stand-in for world physics. That turns the learning problem into inverse dynamics rather than pure imitation, and it gives the model spatiotemporal priors, basically intuitions about how objects move and deform, that a standard vision-language model never picked up from captioned internet images. DREAMZERO itself is a 14-billion-parameter model running closed-loop control at 7Hz, and in real-robot testing it more than doubles generalization performance to new tasks and environments compared to prior state-of-the-art VLAs.

Cross-embodiment transfer is the more surprising part. Feed the model video-only demonstrations from a different robot, or even from a human, and with just 10 to 20 minutes of that footage, it picks up a relative improvement of over 42% on unseen tasks. That's a small amount of video buying a large amount of new competence, which points to the bottleneck being the physical representation, not raw data volume.

Deformable objects remain the plane's hardest test case. TC-IDM, from 2025, hits a 38.46% zero-shot success rate on deformable-object manipulation tasks, things like folding cloth or handling cables. Read that number two ways at once: it's genuine progress on a genuinely hard problem, and it's also a plain signal that well over half the attempts still fail. Contact-rich tasks and tool use compound the issue further, because a verbal instruction sits at a high level of abstraction ("screw in the bolt") while the actual execution demands geometric precision measured in millimeters. That gap between the register of language and the register of contact physics is where a lot of industrial-style manipulation tasks quietly fall apart.

Scaling real-robot datasets helps here, but unevenly. Rigid-object performance improves fairly reliably with more data. Deformable manipulation resists that fix, because the object's geometry keeps changing shape as it's handled, so the model keeps running into states it's never technically seen before, no matter how much data it started with.

The compositional plane: when individually learned skills fail to recombine

This is the plane that makes the previous two look almost tidy by comparison. Compositional generalization asks whether a policy can take sub-skills it has separately mastered and combine them into a new sequence it has never performed as a whole.

The canonical example, from a 2025 paper titled "VLAs are Confined," is almost comic in its simplicity. Train a model to succeed at "put the cream cheese in the bowl." Train it separately to succeed at "put the bowl on top of the cabinet." Now ask it to do "put the cream cheese on top of the cabinet," a task built entirely from motions the model already knows, and it fails. Every required motion primitive sits right there in the model's repertoire. The model just can't figure out how to recombine them into a route it hasn't already memorized.

AC-VLA's July 2026 research names the underlying mechanism as trajectory overfitting: models memorize whole motion sequences tied to specific task combinations, instead of learning reusable, separable sub-skills like "grasp object A" and "place it at location B." The numbers tell a clean story about which fix targets which problem. Baseline out-of-distribution success rates sit at 35.5 and 46.6 across two test suites. Suppress the wrist-view inputs during the placement phase, addressing the perceptual shortcut from the visual plane, and OOD scores climb to 47.3 and 67.0. Add compositional learning on top of that, and scores rise again to 64.2 and 73.3, all while in-distribution performance holds steady at 96.7. The fixes stack. Nobody has to trade compositional skill for baseline competence to get there.

LIBERO-PRO documents the same phenomenon from the benchmark side: models clear 95% in-distribution while showing, on qualitative inspection, an inability to string individually learned subtasks into coherent multi-step sequences. The instruction's actual meaning gets bypassed in favor of recalling a memorized trajectory that merely resembles the right answer.

The Macaluso finding from June 2026 lands hardest here, and it's the one worth taking a hard position on: more in-distribution data, even synthetically generated data through a tool like MimicGen, produces no improvement on the structured compositional test axes. Inductive generalization is a separate capability, full stop. It doesn't show up as a free side effect of scale or broader data coverage, and testing against ordinary domain shifts, different lighting, more clutter, doesn't reveal this failure mode at all. Only deliberately structured, progressively harder out-of-distribution task combinations expose it. Anyone still betting the roadmap on "the next dataset fixes composition" is betting against a result that's already published, and that bet is going to lose.

Why benchmark design has been obscuring all three planes simultaneously

Put the three planes side by side and the benchmark problem becomes obvious: most standard evaluations test interpolation within a known distribution using unstructured shifts. That measures how well a model resists noise. It does not measure whether a model can extrapolate a rule it's never been shown directly, and those are not the same skill no matter how often a leaderboard treats them as one.

LIBERO-PRO makes this concrete rather than abstract. The same models scoring above 95% on the standard LIBERO suite show object hallucination, insensitivity to instruction wording, and outright compositional failure once someone runs qualitative probes on them. The headline number and the failure modes exist in the same model at the same time, which is exactly the gap this piece opened with.

Each plane needs its own kind of test to even become visible. The visual plane shows up under systematic, orthogonal perturbation testing, varying background, lighting, viewpoint, and clutter independently, the approach taken by LIBERO-Plus, RoboTwin-Clean2Rand, and THE COLOSSEUM. The skill and motion plane needs entire motion primitives held out, novel embodiments tested, or deformable and contact-rich tasks probed that were absent from training. The compositional plane only shows up under structured extrapolation tests, ones deliberately built from known sub-skills recombined in an unfamiliar order.

Big datasets don't solve this on their own, though they help with coverage. AgiBot World contributes over a million trajectories across 217 tasks. RoboMIND adds tens of thousands of episodes spanning 479 tasks across four robot platforms. Open X-Embodiment spans 22 different robot embodiments. All genuinely useful, and none of it redefines what the evaluation is actually measuring. A published success rate tells a reader which distribution the model interpolates well. It says almost nothing about which of the three planes will be the one that breaks first once the robot leaves the lab.

Approaches that target specific planes rather than hoping scale alone closes the gap

If the three planes need three diagnoses, they also need three separate treatments, and some genuinely promising work is starting to isolate them. Scale is not one of those treatments, and any pitch that treats it as one deserves the same skepticism a used-car salesman gets for pointing at the odometer instead of popping the hood.

On the visual plane, the general strategy is separation: stop asking one model to handle perception and motor execution as a single tangled task. PEEK, released in September 2025, fine-tunes vision-language models to predict keypoint paths and task-relevant masks, trained across more than 20 robot datasets spanning 9 embodiments, and overlays those predictions directly onto the robot's observations. The low-level policy then only has to learn how to move, not also what to look at. Across 535 real-world evaluations spanning 17 task variations, this produced a 41.4x improvement for a 3D policy trained purely in simulation, and gains of 2 to 3.5x for both large VLAs and smaller manipulation-specific policies. SplatSim's rendering fix belongs in this same category: close the sim-to-real gap at the rendering layer instead of paying to collect more real-world data.

On the skill and motion plane, the fix looks less like better perception and more like a better foundation for physics itself. World Action Models like DREAMZERO replace direct state-action imitation with joint video-and-action prediction, borrowing spatiotemporal priors from video diffusion pretraining that standard vision-language pretraining never provided. Few-shot embodiment adaptation backs this up quantitatively: roughly 30 minutes of play data is enough to move a policy to a new robot embodiment.

On the compositional plane, the fix is architectural discipline: training that actively forces sub-skill separation rather than letting the model memorize whole trajectories. AC-VLA's combined approach, compositional learning plus wrist-view suppression during placement, took OOD scores from a 35.5/46.6 baseline up to 64.2/73.3 without giving up any in-distribution accuracy. If a team has budget for exactly one intervention, this is the one with the receipts to back it. It is, so far, the only fix in this piece that directly targets the plane Macaluso says scale cannot touch.

Retrieval-based methods try to hit the visual and skill planes at once, though they carry their own ceiling: they're only as good as the demonstration database behind them. RobMRAG, a Gaussian-splat-enhanced retrieval approach, improved zero-shot success by 7.76 percentage points over the best zero-shot baseline and 6.54 points over the strongest supervised baseline, tested on a 30-category household object set. Solid numbers, but coverage limits still apply the moment a genuinely new task or scene shows up.

Worth a mention, too: a runtime safety net running alongside the policy itself, out-of-distribution detection. Autoencoder-based anomaly detection reaches an AUROC above 0.96 on failures like cabling and screwing errors, catching the robot before it makes things worse. Subtler failure modes, like a polishing task that's just slightly off, still slip through undetected, and closing that gap is active research territory (AHA, I-FailSense, and PATCH among the approaches working on it).

What the three-plane framework means for teams deploying manipulation robots today

Not all three planes are equally stubborn, and pretending otherwise is how deployment timelines blow up. The visual plane responds best to current fixes: rendering-layer corrections and VLM-guided perception both make measurable dents. The compositional plane is the hardest of the three, precisely because That finding rules out the easy answer. More data alone doesn't touch it. Any vendor claiming their next dataset release solves compositional generalization is selling something the evidence, so far, doesn't support. Say that plainly instead of hedging around it, because the roadmap decisions riding on it are real, and the ones betting on scale alone are the ones that should worry a buyer most.

A practical checklist falls out of this fairly naturally. On the visual plane, test against the specific appearance conditions of the actual deployment environment, not whatever the benchmark happened to vary. A benchmark's robustness numbers don't automatically carry over to a factory floor's lighting or a warehouse's particular clutter. On the skill and motion plane, audit the training data directly and ask whether every motion the deployment requires is actually represented in it, with deformable objects, contact-rich tasks, and tool use flagged as the highest-risk categories. On the compositional plane, run structured extrapolation tests, sub-skills recombined in ways the training data never showed, before assuming a policy will handle a new multi-step sequence just because it nailed the individual steps separately.

No generalist policy currently clears all three planes at once. The most capable systems out there, DREAMZERO and PEEK-augmented policies among them, show real, measurable gains in specific generalization dimensions while leaving others only partly addressed. That's not a failure of the field so much as an honest description of where the field actually sits right now.

The real value of splitting the problem into three planes isn't theoretical tidiness. It's diagnostic. "The robot failed" is a dead end, with nowhere to go from there. "The robot failed because of a perceptual shortcut tied to wrist-view textures" is an actual bug report, one that points to a specific fix, a specific dataset gap, or a specific architecture change. Teams that build their evaluation pipelines around these three axes, instead of one undifferentiated generalization score, get to separate what's genuinely solved from what still needs targeted data collection, new architecture, or compositional training built in from the start.