Reward Function Design for Industrial Manipulation Tasks

Reward function design for industrial manipulation, meaning factory assembly arms, warehouse pick-and-place stations, and insertion tasks, comes down to a chain of decisions that each shape what a robot actually learns to do. Get any one of those decisions wrong and the robot doesn't fail loudly. It succeeds by the metric while doing something you never wanted.

Compression is the real problem: a reward function has to take a task with many moving parts and squeeze all of that into one number per timestep. A reward function has to take a task with many moving parts, precision requirements, timing, safety margins, contact forces, and squeeze all of that into one number per timestep. Every gap left by that compression is a place the robot can exploit, and hand-crafting a reward by trial and error only makes the gap harder to see: it's slow, it has to be redone for every new task, and it tends to produce policies that are subtly wrong rather than obviously broken. A 2026 paper on the Reward Design Agent (RDA), from researchers at Meta, Holiday Robotics, and Boston Dynamics, gives a clean example. A humanoid tasked with package delivery, rewarded mainly for getting the object to a goal location, learned to reach the goal by a means other than the intended carry-and-place behavior. The metric said success. The behavior said otherwise. Nearly everything that follows here, from the sparse-versus-dense choice down to how a vision-language model scores a trajectory, is an attempt to close that gap between what gets measured and what you actually wanted.

Sparse versus dense reward: the first and most consequential choice

Sparse reward gives the agent nothing until the task is done, then a single non-zero signal at the end. It's binary, cheap to write, and light on sensors. In principle, an insertion task could draw its sparse reward from a single end-state check rather than tracking the peg's position the whole way through, no need for precise pose sensing along the way, just a pass or fail check at the finish.

Dense reward gives the agent something to react to at every step, usually built from measurable quantities: how far the gripper sits from the object, whether contact has happened, how high the object has lifted, how close it sits to the goal once placed. That continuous signal speeds up learning by a wide margin, especially in high-dimensional continuous control, but more sensors mean more noise leaking into the training signal. Dense shaping means more sensors, and more sensors mean more noise leaking into the training signal.

The algorithm choice isn't separate from any of this, either. Soft Actor-Critic (SAC), with automatic entropy tuning, tends to converge faster on continuous, high-dimensional manipulation tasks in simulation. PPO tends to fit better when the action space is discrete or low-dimensional. Reward formulation and algorithm choice get decided together, not one after the other.

For industrial arms, dense reward should be the default, not a fallback. Most deep RL work on these tasks already leans that way, because the learning-speed gain from step-by-step feedback is worth the added sensor cost, and a purely sparse signal on a six-stage insertion task can leave an agent exploring blind for far too long to be practical on a factory floor. This default only pays off if the shaping terms are built carefully. Sloppy shaping is worse than no shaping, because it doesn't just slow the agent down, it teaches the wrong lesson convincingly.

Structuring a dense reward function for pick-and-place and insertion tasks

Before writing a single reward term, nail down what success actually means as a boolean check on observable state. The Claru (2026) framework recommends framing success as a boolean check on observable state conditions such as object position relative to the goal, gripper contact status, and object velocity. Test that boolean against hand-built state configurations before it ever touches a training run. A false positive in the success check poisons everything downstream. What counts as observable also depends on where the robot runs: in simulation you typically get full state for free, but on a real arm the reward has to come from cameras, joint encoders, and force-torque sensors, which decides up front whether a state-based reward is even possible or whether you're building something vision-based instead.

The Claru framework splits the reward into a small number of named pieces. Gripper-to-object distance is penalized by r_reach. r_grasp gives a bonus on contact detection. r_lift rewards the object rising above its starting height. r_place penalizes object-to-goal distance, but only counts once the object is actually grasped. an action-smoothness penalty term discourages large or jerky motions, and a final sparse bonus term rewards task completion.

Getting the order these get weighted in backwards makes shaping terms stop guiding progress and start substituting for it, more than any single coefficient does. Success has to dominate everything else, placement has to outweigh grasping, and grasping has to outweigh reaching. Get that hierarchy backwards, and shaping terms stop guiding progress and start substituting for it.

Gating is what actually makes this work. r_place should switch on only after the object is grasped, and r_lift only after first contact. If the gating is skipped, an agent finds the shortcut fast: it pushes the object toward the goal along the table surface, collects the placement reward, and never bothers picking it up.

Check the reward against two baselines before any real training run: a scripted oracle policy and a random one, each run for many episodes. The gap between their returns should be large relative to the noise in the random policy's own returns, the oracle's episode reward should climb roughly monotonically over time, and every shaping component, checked on its own, should score higher under the oracle than under random behavior. If this step is skipped, the reward turns out to be broken only after burning compute on a policy that learned nothing, or worse, learned the wrong thing convincingly. Logging each component separately during training, not just the summed return, is what lets you catch a dominant or exploited term while there's still time to fix it.

Reward hacking: the exploitation patterns that appear most in manipulation training

Reward hacking in manipulation is structural. It's structural. The same handful of exploits keep appearing because they follow directly from how the reward components interact with each other.

The most common one: an agent hovers near the object, approaching and retreating over and over, harvesting distance-based shaping reward without ever committing to a grasp. Call it the approach-oscillation exploit, the manipulation equivalent of a student padding a word count without saying anything new.

Without proper gating, there's also the pushing exploit already mentioned: sliding the object across the surface toward the goal instead of lifting and placing it. Gating handles this in principle, but it comes back the moment the gate condition is implemented slightly wrong, say if contact detection fires on a near-miss instead of an actual grasp.

Then there's outright mis-specification, which produces behavior that isn't exploitative so much as misaligned from the start. The RDA paper's package-delivery case again: Eureka's generated reward, focused mainly on getting the object to the destination, led the humanoid to throw the package. Numerically, task complete. Behaviorally, wrong.

Catching any of this takes more than watching the return curve climb. The Claru monitoring protocol from mid-2026 calls for logging total return per episode, logging the per-component breakdown, tracking task success with a boolean check that's entirely independent of the reward function, and recording video of evaluation rollouts every 10,000 steps. A numerical log can look perfectly healthy while the video shows a robot throwing objects across the workspace. The independent success check is the real diagnostic here: high reward paired with low success means an exploit has been found, while low reward and low success just means the signal isn't informative yet.

Reward shaping methods that go beyond hand-tuned distance functions

Hand-crafted shaping hits a scaling wall fast. It needs rich state access, it's specific to one task, and every new manipulation variant means writing new terms from scratch, which is a real cost on a line where product variants change every quarter.

Semi-supervised reward shaping, described in a 2025 paper (arXiv:2501.19128), tackles this directly. The core insight: zero-reward transitions aren't dead weight, they're informative, and standard supervised methods throw them out. The semi-supervised approach uses both zero and non-zero transitions when inferring reward, and combined with a data augmentation technique the paper calls the double entropy method, it produced a 15.8% jump in best score over competing augmentation methods. In sparse-reward settings it hit roughly double the peak scores of supervised baselines, tested across Atari and robotic manipulation benchmarks. There's more usable signal sitting in existing training data than most pipelines pull out of it, once zero-reward transitions get treated as labeled negatives instead of noise to throw away.

Potential-based shaping is the cleaner option on paper. It guarantees, by construction, that adding the shaping term won't change what the optimal policy actually is, only how fast the agent finds it. That guarantee sounds great until you try to write the potential function for contact-rich manipulation with changing geometries, where defining it over the state space is genuinely hard.

Preference-based learning, the RLHF approach applied to robotics, skips explicit reward components. Instead of hand-specifying r_reach and r_grasp, it infers reward from comparisons: this trajectory looks better than that one. It cuts down on hand-tuning, but it drags in its own problems. States seen during RL training drift away from the states the preference data was collected on. Human raters are slow and expensive to use at scale. And the learned reward model, once trained, gets exploited by the RL agent just like a hand-crafted one would.

None of these four is a universal answer, and picking one isn't a matter of taste. Sparse binary reward makes sense when the success signal is clean and exploration stays tractable. Dense hand-crafted shaping fits when state access is solid and the task has only a few sub-stages. Semi-supervised shaping earns its complexity when zero-reward data is already sitting around in bulk. Preference-based methods suit tasks where success is easy to judge by eye but hard to write down as math. All four still take real human effort to set up, which is exactly the opening the next generation of tools is trying to close.

LLM-generated reward functions: what EUREKA, Text2Reward, and their successors do

EUREKA (Ma et al., ICLR 2024) uses GPT-4 to write reward code straight from environment source and a task description, then runs an evolutionary loop: propose several reward candidates, train an agent on each, keep the best, refine using per-component reward statistics, repeat. No task-specific prompting, no reward templates to fill in. Across 29 open-source RL environments spanning 10 robot morphologies, EUREKA's rewards beat human expert-designed ones on 83% of tasks, with an average normalized improvement of 52%. It even supports a gradient-free version of RLHF, folding human feedback into the reward search without touching model weights. But the reflection loop only sees numerical statistics per component, mean, min, max, and two very different failure modes, say throwing an object versus pushing it, can produce nearly identical numbers. The system has no way to tell them apart, let alone fix the specific behavior causing the failure.

Text2Reward (Xie et al., 2024) takes a natural-language goal and a simplified sketch of the environment written in a general-purpose programming language, then has GPT-4 write dense reward code directly and refine it through self-execution and optional human feedback. Policies trained on this generated code matched or beat expert-written rewards on 13 of 17 simulated manipulation tasks. The generated code tends to lock in fixed numeric weights on each component, so if the first draft grabs onto a fragile or poorly chosen feature, the downstream RL agent struggles to learn anything useful.

Reward-Self-Align (Zeng et al., 2024) and R* (Li et al., 2025) push past code generation into weight tuning. Reward-Self-Align has the LLM write feature templates first, then iteratively adjusts their weights using pairwise ranking of actual rollouts. R splits the problem in two: reward structure evolution, where an LLM iterates over modular reward functions, and parameter alignment, where trajectories are compared to tune the weights. Both go directly after the static-weight problem that Text2Reward left unresolved.

RDA, the Reward Design Agent from Lee et al. (2026), changes the reflection step itself. Instead of numerical statistics, it runs visual analysis on the resulting trajectories: break the instruction into subtasks, propose reward candidates conditioned on those subtasks, train policies, watch the rollouts to score how well each subtask actually got completed, diagnose the failure, keep the best candidates, revise. Tested across 12 tabletop tasks in ManiSkill (pick-and-place, non-prehensile manipulation like rolling and dragging, insertion tasks including peg and charger insertion) plus 4 whole-body tasks in HumanoidBench, RDA produced policies noticeably better aligned with the actual instruction than either EUREKA or human-designed rewards, while matching them on raw success rate. In the package-delivery case, both EUREKA and the human baseline ended in kicking or throwing the package. RDA's reward produced a more behaviorally aligned policy.

The pattern across this whole line of work is incremental: each paper fixes exactly one gap and leaves the next one exposed. Text2Reward killed the need for hand-written reward code. Reward-Self-Align and R* killed static weights. RDA killed coarse numerical feedback. None of them has closed the loop entirely, and the next gap is already visible in how these systems fall apart on vision-heavy, contact-rich tasks. That is where the field is headed next.

Where VLM-based reward models struggle on contact-rich tasks

A lot of real industrial manipulation doesn't reduce cleanly to state variables. Deformable objects, changing geometry, occlusion, multi-step contact sequences: none of that reads off a position sensor. That's the gap reward models built straight from vision are trying to close. LIV, VLC, GVL, VICtoR, REDS, ReWiND, and SARM all score manipulation directly from camera input paired with task text, with no explicit object-pose tracking required.

The catch, confirmed in the SARM paper at ICLR 2026 (arXiv:2509.25358), is that most of these models process the entire trajectory starting from the very first frame just to figure out what's happening over time. That's expensive in both data and compute, and it caps how well the approach scales to long, contact-heavy tasks.

SARM, short for Stage-Aware Reward Modeling, goes after exactly that class of task, with T-shirt folding as the running example: changing geometry, occlusion, fabric that behaves differently every time, and a sequence of steps that has to land in order. Reward models built on goal-distance alone miss the intermediate progress in a task like this. Reward models built on frame-index progress hit a different wall: frame count doesn't line up with the actual semantic stages of the task. SARM's fix swaps frame-index labels for progress labels tied to natural-language sub-task annotations, paired with a dual-head architecture that predicts the high-level stage and the fine-grained progress within that stage at the same time.

Even with that fix in place, the underlying difficulty doesn't go away. Long-horizon, contact-rich, deformable-object manipulation stays the hardest category for vision-language reward models to score reliably. SARM's stage-aware labels mark a real improvement on the problem. They don't close it.