TMRL Diffusion: Timestep-Modulated Pretraining Enables Exploration for Efficient Policy Finetuning
July 13, 2026
-current paradigm: large scale pre-training on demonstration data via imitation learning, fine-tuning with RL
-behavior cloning is very good at in-distribution tasks, not good with slight variations (especially taking a robot's own errors into account)
context-smoothed policies - pre-training framework that expands the support of BC policies by injecting noise into context inputs via a forward diffusion process
-used to help RL policies explore more actions during post-training by smoothing action distributions
-requires retraining or fine-tuning current policies (i.e. )
timestep modulated RL - used to dynamically adjust the diffusion timestep during RL
-based on the timestep of a rollout, the RL policy changes the diffusion timestep to pass into the CSP policy, to change how strongly the policy conditions on the current observation
Summary:
-current BC stays very close to - action based on context
-CSP allows the policy to smoothly move towards - overall marginal action distribution
-the broader policy gives RL more opportunities to explore and discover behaviors