Simulation Fidelity Requirements for Manipulation Policy Transfer

Manipulation policies trained in simulation still break the moment they meet a real gripper and a real object. Simulators struggle with this particular aspect of physics. It's that the gap between simulated and real environments, what Tobin and colleagues named the "reality gap" back at IROS 2017, hits different physical properties with wildly different severity. Some mismatches a policy shrugs off. Others turn a working grasp into a dropped part. The field's real work right now is figuring out which is which.

Training in simulation costs almost nothing and breaks nothing. A policy can fail ten thousand times overnight without damaging a single part or requiring a human to reset a workspace. Deployment is where the bill comes due. Unmodeled effects like gear backlash, joint wear, and basic nonrigidity never show up in the physics engine's equations. Camera renderers, no matter how advanced, still don't reproduce the noise and clutter of a real sensor pointed at a real bin. And contact, the actual moment metal touches metal or a fingertip touches cloth, tends to punish even small parameter errors with dynamics that look nothing like the real thing. Zero-shot transfer, where a policy trained purely in sim goes straight to a real robot with no adjustment, fails often in contact-heavy tasks. That doesn't make simulation useless. Even a policy that won't transfer directly can still hand a real-world learning loop a strong starting point, cutting down what would otherwise be a slow, expensive exploration process on hardware. The shift underway favors higher fidelity in select areas. It's toward figuring out exactly where fidelity has to be right and where it can afford to be wrong.

The distinct fidelity dimensions that shape whether a policy transfers

Fidelity isn't one setting a researcher turns up or down. It splits into separate axes, and each one affects transfer on its own terms.

Visual fidelity is often the first wall a policy hits. Any policy conditioned on RGB images is sensitive to texture, lighting, and the appearance of the object itself, and small mismatches here are one of the main reasons sim-to-real learning struggles to scale. Physics and dynamics fidelity is a second, separate axis. Get the mass, the friction coefficient, or the contact parameters wrong, and the object moves in ways that don't resemble reality at all. What's easy to miss is that kinematic parameters, the link lengths and joint offsets researchers usually just assume are known exactly, matter for transfer just as much as the dynamics parameters everyone worries about.

Then there's the hardware layer. Binding actuator torque and velocity limits to the real robot's actual specs, rather than leaving them at whatever the simulator defaults to, is a recurring recommendation in physics-aware sim-to-real work. Sensor noise and observation delay sit next to this problem and mostly go unaddressed, since most physics engines simply don't model them. Soft-body and deformable objects form their own, harder category: existing simulators struggle to represent the coupled visual and physical behavior of something like cloth or a stuffed toy, and that struggle is qualitatively different from anything rigid-body simulation deals with.

Put together, this means a policy can be strong in one dimension and weak in another, and the failure will show up exactly where the weak dimension lives. A policy trained with cheap, low-fidelity graphics but tight contact modeling might handle a force-sensitive insertion task fine and fail the moment it needs to recognize a real object by sight. Flip the fidelity profile, and the failure flips with it.

Domain randomization as the first answer to fidelity uncertainty, and where it falls short

Domain randomization, DR for short, was the field's first serious answer to not knowing exactly how the real world would differ from the simulator. Instead of guessing one true value for friction or lighting, DR trains the policy across a spread of values, textures, and lighting conditions, on the theory that a policy robust to variation in simulation will be robust to the specific, unknown values of the real thing.

What gets randomized has expanded over time: textures and lighting first, then masses and friction, and eventually kinematic parameters too. That last one turns out to matter more than people expected. Randomizing kinematic parameters, the values usually treated as fixed and known, outperforms randomizing dynamics parameters alone for manipulation transfer. That's a counterintuitive result if the intuition is "physics is where the uncertainty lives."

But DR has real limits, and they're not minor. Setting good randomization ranges takes real tuning, and getting it wrong in either direction costs the policy something. Physics-aware recommendations in the literature suggest constraining randomization to a physically grounded range for parameters that matter most, rather than throwing a wide, unconstrained range at the problem, because wide ranges start training the policy on scenarios that could never happen on real hardware. And DR only ever covers the parameters someone thought to randomize. Sensor noise, observation delay, and unmodeled environmental effects sit entirely outside its reach, because DR is a hedge against uncertainty in values the model already represents, not a fix for physical phenomena the model never included in the first place.

System identification and digital twins: raising fidelity rather than tolerating the gap

Where DR spreads uncertainty across a range, system identification tries to close it. The idea is to calibrate the simulator's parameters against real measured behavior, iteratively, until the simulated system moves the way the real one does. Methods range from optimization approaches that match simulated and real system behavior to classifiers working directly off sensor data.

Digital twins push this further. Instead of tuning toward a generic physics model, the simulation gets aligned to the specific real system it's meant to represent, the actual robot on the actual bench, not an average version of that robot. This pays off most clearly in contact-rich, precision-dependent tasks, precision-dependent insertion tasks being a representative example, where a tiny parameter error is the entire difference between success and failure.

Even incremental, low-level calibration steps count as system identification in practice, and they are a useful reminder that some of this work is mechanical rather than exotic. But there's a ceiling. No amount of calibration recovers gear backlash, accumulated wear, or genuine nonrigidity, because those effects are not reliably captured in current simulators. And both system identification and digital twin construction take real manual effort per system, which limits how far either approach scales across a fleet of different robots handling a fleet of different objects.

Where visual fidelity actually matters, and the Gaussian Splatting shift

For any policy conditioned on camera images, the appearance gap is often the first and most direct cause of failure: the policy has never seen the real scene's textures, shadows, or clutter, and no amount of contact-parameter tuning fixes that. Earlier fixes leaned on style transfer, retraining the visual encoder to make simulated images look more like real ones, or adversarial domain adaptation, aligning feature distributions between the two domains statistically rather than pixel by pixel.

3D Gaussian Splatting has changed the shape of this problem. Instead of trying to make a rendered scene look real, 3DGS reconstructs an actual photorealistic 3D scene directly from real-world images and feeds that reconstruction back into the simulator as the environment itself. RoboSimGS builds on this with a hybrid setup: 3DGS handles the photorealistic appearance, while separate mesh primitives handle the objects that actually need to be interacted with, and a multimodal LLM infers physical properties like density and stiffness, along with kinematic structure like hinges or sliding rails, straight from the visual reconstruction. Policies trained on RoboSimGS-generated data have been evaluated across a range of manipulation tasks. SplatSim, presented at ICRA 2025, demonstrates the same underlying idea applied specifically to RGB-conditioned manipulation policies.

A separate real-to-sim framework, tested on deformable tasks including plush toy packing, rope routing, and pushing an irregularly shaped block, found that simulated rollouts correlated strongly with what actually happened on the real robot (Zhang et al., arXiv:2511.04665). That correlation matters because for these particular tasks, visual fidelity and soft-body physical fidelity aren't separable problems, getting the look right and getting the deformation right have to happen together. Genie Sim 3.0 takes the reconstruction idea to a larger scale, combining 3DGS-reconstructed assets with LLM-generated scene variety, and open-sourcing 5,140 object assets alongside more than 10,000 hours of synthetic data and 100,000 evaluation scenarios.

The pattern across all of this: photorealism is no longer the hard part. Reconstructing a real scene's appearance from photographs is now a solved-enough problem that the bottleneck has moved back to physics and contact.

Contact and deformable object fidelity: the hardest dimension to approximate

Contact is where fidelity errors stop being cosmetic. A slightly wrong friction coefficient or contact stiffness doesn't just make the simulation a little noisier, it produces dynamics that are qualitatively different from what actually happens when two surfaces touch.

MuJoCo, introduced by Todorov and colleagues at IROS 2012, remains the standard engine for rigid-body contact, and most published evaluations still stick to rigid bodies or the simplest soft-body approximations available. Deformable objects, cloth, rope, foam, anything that bends or compresses in complicated ways, are handled far less well, precisely because the visual complexity and the physical complexity of deformation are coupled and neither simulator community has fully cracked that combination. FLASH addresses this directly with a GPU-optimized architecture built around a custom non-smooth Newton solver, hitting real-time simulation at 30 frames per second for scenes with more than 3 million degrees of freedom, and its fully automated pipeline learns vision-based manipulation policies at that scale using simulated interaction and systematic domain randomization, with no real-world demonstrations required.

Tactile and force sensing close a different part of the gap. For contact-rich tasks, high-resolution tactile data fused with joint-level torque control turns out to matter more than any amount of visual fidelity. Zero-Shot Dexterous Force Grasping reportedly demonstrated the first controllable grasping on a multi-finger dexterous hand trained entirely in simulation and transferred with zero real-world fine-tuning, using an asymmetric actor-critic PPO training pipeline to achieve controllable grasp force tracking. Contact-Aware Neural Dynamics trains a base model on a large batch of simulated trajectories, 8,000 for single-object tasks and 15,000 for multi-object tasks, then fine-tunes that model on a small amount of real-world interaction data that includes actual tactile sensor readings.

The takeaway across these projects is consistent: for deformable or contact-heavy manipulation, neither visual realism nor physics randomization alone gets the job done. Tactile sensing and force feedback have to be built into the simulation architecture itself, not bolted on afterward.

How the field uses real-world data to patch what simulation cannot model

If perfect simulation fidelity isn't achievable, and increasingly it looks like it isn't, the real question becomes how to spend real-world data as efficiently as possible on exactly the gap simulation leaves behind.

Simulation-Guided Fine-Tuning, presented at ICLR 2025, pulls a structural prior out of the simulator, specifically a value function learned there, and uses it to steer exploration once the policy moves to real hardware. Tested across five dexterous manipulation tasks where zero-shot transfer fails outright, it needed up to an order of magnitude fewer real-world samples than standard fine-tuning to reach comparable performance. Contact-Aware Neural Dynamics follows a similar logic in its own fine-tuning step: the base model already knows the general shape of the task from simulated trajectories, and the real tactile data only has to correct the contact-fidelity gap specifically, not relearn the whole task from scratch.

A surrogate reinforcement learning approach trains a high-fidelity model on historical real-world data and uses it for safe pretraining before fine-tuning under real operational constraints, with reported success rates as high as 97.8 percent on reaching tasks while cutting labeled-data requirements roughly in half. Contact-Anchored Policies, or CAP, take a different angle on the same problem: instead of conditioning on language instructions, CAP conditions on physical contact coordinates, trains on just 23 hours of demonstration data, and beats state-of-the-art vision-language-action models in zero-shot evaluation by 56 percent. CAP also uses EgoGym, a deliberately lightweight simulation benchmark, to iterate quickly on failure modes before anything touches real hardware, trading photorealism for speed and scene variety. X-Sim tackles a related but distinct problem, cross-embodiment learning through a real-to-sim-to-real pipeline.

The logic threading through all of these is the same: simulation supplies scale, real-world data patches the specific residual gap simulation can't close, and the engineering problem is drawing that boundary correctly rather than pretending one side can do the other's job.

The evaluation bottleneck: when simulation fidelity fails as a proxy for real-world performance

Real-robot evaluation is expensive in a way that's easy to underestimate. Getting a statistically meaningful comparison between two policies on real hardware requires an evaluation schedule demanding enough that it's become one of the field's biggest practical bottlenecks, full stop.

Simulation looks like the obvious fix: it's reproducible, it's cheap, and it scales. Even small sim-real differences make a simulation metric a poor guide to real-world improvement. Push a policy's training loss or test loss down in simulation, and there's no guarantee that number says anything about how the policy performs on an actual robot.

Some recent work tries to close this specific gap rather than the training-time gap. The Interactive World Simulator is an action-conditioned video prediction model trained on real robot interaction data, capable of stable long-horizon prediction beyond 10 minutes at 15 frames per second on a single RTX 4090, and it shows strong correlation between simulated and real policy performance across rigid, deformable, and pile-interaction tasks. Policies trained on data generated by this world model perform comparably to policies trained on an equivalent amount of real-world data, which is a meaningfully different claim than anything domain randomization or system identification has offered. PolaRiS focuses on scalable real-to-sim evaluation for generalist robot policies, and RoboEval offers a structured, scalable evaluation framework for manipulation specifically. EgoGym, mentioned earlier as part of CAP, takes yet another position: it prioritizes object and scene diversity over photorealism, on the reasoning that generalization is what evaluation should be measuring, not visual polish.

This tension drives all of it and deserves direct statement: the fidelity that makes a simulator good for fast policy training is not necessarily the fidelity that makes it trustworthy for evaluation. A simulator optimized to let a policy iterate quickly through thousands of episodes may be exactly the wrong environment to trust when deciding whether that policy is actually ready for a real robot.

What current benchmarks and platforms reveal about which fidelity gaps remain open

Every benchmark platform encodes a specific set of fidelity trade-offs, whether its authors say so explicitly or not, and reading the landscape of platforms together says more about the field's real priorities than any single paper does.

RLBench, introduced by James and colleagues in IEEE Robotics and Automation Letters in 2020, established much of the early structure for robot learning benchmarks, built on a physics engine with a deliberately diverse task set. MuJoCo remains the dominant engine underneath model-based control research generally, and its near-universal use is a big part of why soft-body fidelity has lagged. It's genuinely good at rigid-body contact and comparatively weak at deformables, and given how much of the field builds on it, that weakness propagates. RoboVerse arrived more recently as an attempt to unify platform, dataset, and benchmark into one system, directly addressing the fragmentation that resulted from every lab building its own separate benchmark. Genie Sim 3.0, again, brings scale: 5,140 object assets, over 10,000 hours of synthetic data, 100,000 evaluation scenarios, and scene generation driven by a language model aimed at generalization along multiple axes, along with the first benchmark to run automated evaluation through a language model using a vision-language-model assessment pipeline.

Read together, the platforms tell a consistent story. Visual fidelity is increasingly handled well, largely thanks to 3DGS-based reconstruction pipelines that turn real photographs directly into simulation assets. Rigid-body contact is reasonably well served by existing engines, provided someone puts in the calibration work system identification demands. Deformable and soft-body simulation remains the field's most open problem. The physical modeling is harder, the tools are younger, and the coupling between visual and physical complexity in soft bodies still resists the kind of clean separation that made progress possible on the rigid-body side. That's where the next real gains sit, and everyone building these systems knows it.