Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations

August 16, 2026

-pre-training a policy with BC and subsequent fine-tuning with policy gradient + augmented loss to stay close to demonstration

- purely trained in MuJoCo, demonstrations collected using VR, model-free RL

- sparse rewards

policy gradient: $\nabla_\theta \ln \pi(a \mid s) A(s, a)$

\nabla_\theta \ln \pi(a \mid s)

— how to move the weights to increase the likelihood of taking

a

from

s

A(s, a)

— which direction to move the weights in

- $A > 0$ increase likelihood of $a$

- $A < 0$ decrease likelihood of $a$

- advantage calculate using methods like GAE or $A(s, a) = Q(s, a) - V(s)$

Demo Augmented Policy Gradient (DAPG)

1.Pre-training with BC using a maximum-likelihood objective:

- $\rho_D$ — demonstration data

- learns $(\mu_\theta(s), \sigma_\theta(s))$

\text{maximize} \sum_{(s, a) \in \rho_D} \ln \pi_\theta(a \mid s)

2.RL fine-tuning with augmented loss

- BC doesn’t optimally use the information present in demonstration data

- augmented policy gradient:

g_\text{aug} = \sum_{(s, a) \in \rho_{\pi}} \nabla_\theta \ln \pi_\theta(a \mid s) A^{\pi}(s, a) + \sum_{(s, a) \in \rho_D} \nabla_\theta \ln \pi_\theta(a \mid s) w(s, a)

- use $w(s, a) = \lambda_0 \lambda_1 \max_{(s', a') \in \rho_{\pi}} A^{\pi}(s', a')$ for all $(s, a) \in \rho_D$ , where $\lambda_0 = 0.1$ and $\lambda_1 = 0.95$