Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning

July 13, 2026

on-policy RL - updates are calculated using trajectories/rewards from the currently policy

off-policy RL - update signals are calculated by a separate policy, human feedback, etc.

online RL - policy interacts with the environment, the agent collects new experiences while it's learning

-could be hard to explore reward-bearing states if the rewards are sparse and the state space is very big

-supplement with behavior cloning

offline RL - trajectories are already collected, sample transitions from the dataset and update the policy/value function

-very hard to learn accurate out-of-distribution actions

-during training, the system queries a human operator for potential corrections

-used for off-policy updates

-better than IL on the same amount of human data

-101% average improvement in terms of success rate and 1.8x faster cycle time

Training Process:

1.Data collection to train reward classifier - binary sparse rewards

2.Collect 20-30 human demonstrations of the task, used to initialize demo replay buffer + initial behavior cloning

3.Start policy training using SAC, intervene as minimally as possible

1.RL, rewards, and human intervention to improve policy

2.Restart rollouts from a (somewhat) random position