Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
August 4, 2026
-imitation learning is hard for tasks that require precision and visual feedback
-small errors in predicted actions incur large differences in state
-traditional behavior cloning minimizes MSE between predicted actions and demonstrated actions
-this collapses the multiple valid trajectories (modes) from demonstration data into an average, which is not always admissible
-standard regression learns the conditional mean
Variational autoencoder (VAE)
-an autoencoder whose encoder provides outputs in a continuous latent space, where similar inputs occupy nearby regions
-decoder outputs aim to reconstruct the original input
-reconstruction loss preserves information in the data
-preserves modes and forces the encoder to learn distinguishing characteristics
-KL loss organizes that information into a smooth, continuous latent space
-prevents the encoder from assigning an arbitrary code or latent representation to each different observation
-rewards smoothness
-learns distribution without input conditioning
-a conditional VAE (CVAE) provides an input observation to condition the output
-“Generate a sentence” VAE; “What’s 2+2?” CVAE
-during inference, sample latent , then run the decoder to sample from and produce the output
Action chunking
-sequences of actions are grouped together and executed as one unit
-policy predicts target joint positions for the next k timesteps, rather than just one timestep
-mitigates compounding errors by reducing the task horizon by a factor of
-temporal ensembling: query the policy every steps and average across overlapping chunk predictions
-use transformers for sequence prediction, trained as a CVAE
-joint positions from ALOHA are used as actions; observations come from four cameras