Foundation Models for Robot Manipulation in Production

Foundation models for robot manipulation now come with names, price tags, and vendor roadmaps, not just conference papers. This piece is about the gap between a demo where a robot arm stacks blocks under studio lighting and a robot arm that does the same job for eight hours on a warehouse floor with dust on the lens. Robot foundation models (RFMs), usually built as Vision-Language-Action (VLA) systems, take in camera images and a plain-language instruction and output motor commands directly. That sounds simple, but it is not, and the reasons why sit at the center of every decision covered below.

Manipulation is hard in a way that text generation and image classification are not, because contact-rich tasks demand precise geometry, force feedback, and the ability to replan in real time when a gripper slips or a box shifts mid-grasp. None of that falls out of language model pretraining for free. Two design philosophies keep competing for dominance here: monolithic end-to-end VLAs like RT-2, Gato, and Physical Intelligence's π0, which run pixels and language through one pipeline straight to action, versus modular stacks in the tradition of SayCan, which chain a language model planner to a vision-language perception layer to a library of discrete, swappable skills. The promise on both sides is the same: better transfer across tasks and robot bodies, understanding of open-world scenes, and long-horizon planning that doesn't require hand-coding every step. Worth sitting with from the start: the generality that makes these models exciting in a lab is the same generality that makes them unpredictable on a factory floor. This piece is about navigating that gap, not pretending it isn't there.

The landscape of models teams are actually choosing from in 2025–2026

Physical Intelligence's π0 pairs a PaLI-Gemma vision-language backbone with a flow matching action head, a continuous normalizing flow that avoids the discretization artifacts that come from chopping actions into tokens. It's built for dexterous, two-handed manipulation and is among the more capable reported models for contact-rich work. The fine-tuning barrier is notably low: company experiments reportedly needed only a small number of hours of demonstration data to adapt the model to a new task, and checkpoints ship for platforms including ALOHA and DROID with open inference and fine-tuning code. Physical Intelligence has since pushed two follow-ups: π0.5, aimed at open-world generalization, in 2025, and π0.7, pitched as a steerable generalist with emergent capabilities, expected in 2026.

NVIDIA entered this race in March 2025 with Isaac GR00T N1, announced at GTC. Its architecture splits cognition into two systems: a slow, deliberate vision-language planner (System 2) that hands instructions to a fast, reflexive motor controller (System 1), a split that mirrors how a warehouse supervisor thinks through a task before a forklift driver actually does it. Training data mixes real robot trajectories, human video, and synthetic data generated through NVIDIA's Isaac GR00T Blueprint pipeline, which can produce hundreds of thousands of synthetic trajectories in hours — roughly the throughput of many months of continuous human demonstration collection, by rough estimate. In NVIDIA's own reported benchmarks, the model beat imitation-learning baselines across multiple robot bodies in simulation; material handling, packaging, and inspection are the target uses named.

Google DeepMind's Gemini Robotics family takes a different angle on trust: Gemini Robotics 1.5, released in September 2025, shows its reasoning before it acts, treating transparency as a design goal rather than a nice-to-have. Gemini Robotics On-Device runs without a network connection at all, aimed squarely at places where latency matters or Wi-Fi doesn't reach, like the back corner of a distribution center, and it's built to fine-tune on a small number of demonstrations. Gemini Robotics 2 controls three different robot bodies from a single checkpoint, though dexterous multi-finger manipulation still lands at medium-to-high success rates while gripper and whole-body tasks do noticeably better. Right now the reasoning model sits on Google AI Studio; the VLA and On-Device variants are in early-access partner programs.

Covariant's RFM-1 and its 2025 logistics-focused sibling PRIME-1 are worth flagging for a narrower reason than most coverage gives them credit for: they have documented deployments in warehouse logistics specifically, which is rarer than the number of papers in this space would suggest. RFM-1's visuo-motor backbone trains on large volumes of real pick actions and is built for zero-shot generalization to objects it hasn't seen. PRIME-1 narrows that further to parcel workflows and fast adaptation at a new site. Covariant's distribution deal with ABB is the detail that separates this from a research paper looking for citations — a distributor doesn't sign on for a demo.

For teams that want to kick the tires before committing budget, OpenVLA remains the accessible open-weight baseline: a 7-billion parameter LLaMA-2 backbone paired with DINOv2 and SigLIP vision encoders, fine-tuned on the Open X-Embodiment dataset. Inference takes a few seconds per action, which is fine for slow control loops but not fine for anything that needs to react fast. It's the default starting point for academic labs and smaller industrial teams doing feasibility work before they write a check for something commercial.

None of these five are interchangeable, and picking one is already an architectural decision whether a team treats it that way or not. The position to take after sitting with the tradeoffs above is this: teams that pick a monolithic end-to-end model for anything safety-critical are buying a debugging headache they haven't priced in yet, because when a single-pipeline VLA fails, there's no clean way to tell whether the camera misread the scene, the model picked the wrong sub-goal, or the motor output just missed. That cost shows up months later during integration, not on day one, which is exactly when it's most expensive to fix.

Why the training data situation looks different from every other ML domain

Text and images are everywhere online, but well-labeled robot trajectories are not, and there is no web crawl for robot arms. Collecting that data means buying or renting physical hardware, running human teleoperation or hands-on demonstration sessions, and burning real hours per episode, a cost structure that looks nothing like scraping Common Crawl. Then there's the embodiment problem: a Franka arm, a Fetch mobile base, and a Unitree humanoid have different degrees of freedom, different sensors, different kinematics, and no single way of representing "action" transfers cleanly between them.

The field's collective answer to this is Open X-Embodiment (OXE), a dataset pooled from 21 institutions and 22 robot platforms, covering 527 distinct skills across 160,266 task episodes. Training RT-1 on this combined dataset, producing RT-1-X, delivered roughly 50% higher mean success than single-robot baselines, and RT-2-X showed a threefold jump in success on entirely new, emergent skills. The scale keeps climbing: by 2024 to 2025, cross-embodiment datasets passed a million trajectories, with newer efforts like AgiBot World and EO-1.5M pushing multimodal data fusion 10 to 100 times beyond what earlier work managed.

Synthetic data closes part of the gap, not all of it, and that distinction is worth sitting with rather than skating past. NVIDIA's Blueprint pipeline shows the speed advantage plainly, generating in hours what would take months to collect by hand. Approaches like Real-Is-Sim build dynamic digital twins using Embodied Gaussians, running the same twin through data collection, training, and deployment to keep the sim-to-real gap from widening. But contact-rich tasks and anything involving deformable objects (fabric, cardboard that's already a little crushed, a bag of loose parts) remain the cases where simulated physics still doesn't quite match the real thing. A block is a block in simulation, but a crumpled bag of chips is a physics problem nobody has fully solved.

Can robots eventually gather their own training data, closing the loop without a human in it? Not reliably yet, and filtering noisy, mixed-quality trajectories at scale, keeping the useful signal and tossing the garbage, remains unsolved. In practice, this means a team deploying an RFM in a new facility should not assume the pretraining data covers their specific bins, lighting, and SKU mix. The gap between a benchmark task and a production task is usually wider than it looks from the outside.

What lab benchmarks measure and what they leave out

Benchmark design has genuinely gotten better. VLABench, presented at ICCV 2025, covers 100 task categories and over 2,000 objects with heavy randomization, built specifically to stress long-horizon reasoning rather than single-step pick-and-place. ManipBench and EmbodiedBench, both from 2025, extend that coverage into low-level manipulation precision and multi-modal embodied agents respectively, and that's real progress, not window dressing.

But sit with what a high benchmark score doesn't tell anyone, because this is where the field's own reporting undersells the risk: even good benchmarks miss things that matter on a real floor. Environmental brittleness is the big one. A VLA trained in environment X can perform poorly in environment Y, and something as mundane as a shift in warehouse lighting or a rearranged shelf can break a policy that scored well in the lab. Benchmarks report mean success rates; a plant manager cares about the tail, the rare runs where the robot damages product or, worse, moves unpredictably near a person. Autoregressive VLAs also have limited ability to replan mid-task, so if a human reaches in to grab a part or the environment shifts, recovery isn't guaranteed. And there's a specific failure mode worth naming directly: models can produce action sequences that sound plausible inside the model's own representation but are physically impossible, off by a few centimeters in a task where a few centimeters is the whole ballgame.

That last point traces back to weak geometric grounding. Open-set 3D pose estimation in cluttered, moving scenes is still unreliable, and precise manipulation tasks expose that weakness far more than generic pick-and-place benchmarks do. ResponsibleRobotBench is an early signal that the field has noticed; safety and responsible behavior are starting to get their own formal evaluation, though the tooling here is young. So the practical read on any benchmark table, after working through what these numbers actually cover: a high score is necessary, not sufficient. The better question to ask isn't what the score was, it's what conditions the benchmark controlled for that a factory floor never will.

The architectural choices that actually determine production viability

Monolithic versus modular isn't a style preference; it changes what happens when something breaks, and this is where the position from the first section pays off. A single end-to-end VLA is simpler to bolt into a pipeline, but debugging it after a failure means guessing between three possible culprits with no clean way to isolate which one. A modular stack, LLM planner plus VLM perception plus an explicit skill library, costs more in integration complexity and adds latency at every handoff between components, but each piece can be debugged and swapped on its own. Most production teams coming out of industrial automation lean modular, largely because it lets them keep existing, safety-certified motion controllers in place and use the RFM purely as a high-level planner sitting on top of them.

Action representation is a detail that matters more than most people expect walking in. Tokenized action outputs, common in earlier VLAs, introduce discretization artifacts: fine for coarse pick-and-place, a real problem for precision assembly where a rounding error in the token space becomes a millimeter of drift at the gripper. Flow matching heads, π0's approach, produce smoother continuous control and dodge this specific failure mode, worth knowing before evaluating a model for anything contact-rich.

GR00T N1's System 1 / System 2 split formalizes something modular teams have often built by hand anyway: slow deliberate planning separated from fast reflexive execution. Making that split explicit tends to help both latency and reliability, since the fast loop doesn't wait on the slow loop to finish thinking. Separately, models that reason purely over 2D images carry a geometric grounding deficit for anything needing millimeter precision; the emerging 3D-VLA line of research argues this is a limitation of the architecture, not something more data alone fixes.

Then there's the on-device versus cloud question, which is really a question about what happens when the network hiccups. Cloud dependency means latency and availability risk, and a factory floor or a distribution center does not always have a clean, dedicated Wi-Fi channel. Both NVIDIA (with GR00T N1 post-training paths) and Google (with Gemini Robotics On-Device) have built explicit on-device inference options, which says something about where the field expects production constraints to push everyone eventually. Digital twins built along Real-Is-Sim lines double as evaluation infrastructure too, letting a team watch how a policy behaves offline before it's anywhere near a live production line.

Safety constraints that engineering must own regardless of model capability

Start from this and don't relax it: a VLA's output is not safe just because it came from a capable model, and safety has to be built into the system around the model, not assumed to live inside it. That means speed and force limits enforced at the controller level rather than trusted to the model's judgment, collision avoidance backed by real workspace monitoring, and an emergency stop circuit that's independent of the software stack entirely, so a model hallucination can't take the kill switch down with it.

SafeVLA, from 2025, is one research response worth watching: it treats safety alignment as a constraint-learning problem, training the model to respect task rules and safety measures directly rather than relying on filtering bad outputs after the fact. Runtime monitoring is a separate, complementary layer. Guardian, also 2025, is a VLM-based system that watches execution in real time for specific failure signatures, an object slipping out of a grip, steps happening out of order, and flags them as they occur. Related recovery frameworks have a VLM check the pre- and post-conditions of each skill, figure out what condition got missed, and suggest a recovery action, treating failure recovery as something designed into the system rather than bolted on after something goes wrong.

Worth pausing on why hallucination means something different here than it does for a chatbot — this is the distinction worth returning to when people wave away model errors as a solved UX problem. A language model that invents a fact is embarrassing, but a manipulation model that outputs a physically impossible action sequence can crush a part, snap a gripper, or put a person at risk; the stakes shift from wrong to dangerous the moment actuators get involved. Human-robot collaboration tightens every constraint further: when a person is standing in the workspace, speed limits and collision zones need to flex dynamically, not sit at some fixed value calibrated for an empty room. None of this happens in a regulatory vacuum, either. ISO/TS 15066 governs collaborative robot safety specifically for human-robot interaction, and ISO 13849 covers functional safety more broadly; the model is one part inside a system that gets certified as a whole, not piece by piece.

Adapting a pretrained model to a specific production task without starting from scratch

The pattern that's emerged looks consistent across vendors: pretrain broad on something like OXE or a large proprietary dataset, then fine-tune narrow on demonstrations collected in the actual target environment. How much fine-tuning data that takes varies a lot by model and by task. π0's reported adaptation, a small number of hours of demonstration data in company experiments, sets a fairly low bar of entry for a task that's well scoped and close to what the base model already knows. Gemini Robotics On-Device is built around the same idea, fine-tuning on a small number of demonstrations as a design goal rather than a happy accident. Push further from the pretraining distribution, a genuinely new robot body, a task far outside what the base model saw, and the data requirement climbs fast.

Fine-tuning is not a universal fix, and after tracing through where it does and doesn't help, it's worth being blunt about the limits. It sharpens task-specific performance and helps the model adjust to a particular environment's quirks, but it does not repair fundamental limits like weak geometric grounding or discretization artifacts in the action space; those need a different model or a redesigned action head, not more hours of demonstration data. And it doesn't transfer on its own: a model fine-tuned to a flawless degree for one warehouse layout can degrade fast the moment it's dropped into a different warehouse with different shelving, different lighting, different clutter. That's the whole story of this piece in miniature: the general capability is real, but every step from lab to production adds a constraint the benchmark never had to satisfy.