Asymmetric Actor Critic for Image Based Robot Learning

August 2, 2026

-training on full states end-to-end is hard because it depends on good state predictors; training on just RGBD is hard because of partial observability and dimensionality
-full-state prediction from partial observations is hard
-accurate inference from partial data is also hard

Asymmetric actor/critic

-actor takes visual partial observations as inputs
-critic takes full state as input
-learns Q(s,a)Q(s, a) much faster, leading to better actor updates

Sim-to-real transfer methods

-very realistic simulators
-domain adaptation
-real-world fine-tuning
-domain randomization in simulation

Bellman error loss

Lc=(Q(st,at)−yt)2\mathcal{L}_c = (Q(s_t, a_t) - y_t)^2, where yt=rt+γQ(st+1,π(st+1))y_t = r_t + \gamma Q(s_{t+1}, \pi(s_{t+1}))

-same loss used in SAC

Deep Deterministic Policy Gradient (DDPG)

-online, off-policy
-rollouts generated with πb(s)=π(s)+N(0,1)\pi_b(s) = \pi(s) + \mathcal{N}(0, 1), where N\mathcal{N} is Normal noise
-transitions (st,at,rt,st+1)(s_t, a_t, r_t, s_{t+1}) are stored in a replay buffer and used to train the critic
-actor minimized with La=−Es[Q(s,π(s))]\mathcal{L}_a = -E_s[Q(s, \pi(s))]
-∇La\nabla\mathcal{L}_a is the deterministic policy gradient

Universal value function approximators (UVFA)

-make an RL agent learn many value functions at once by conditioning the value function on state and goal: Vπ(s)→Vπ(s,g)V^{\pi}(s) \rightarrow V^{\pi}(s, g)

Hindsight experience replay (HER)

-improves goal-conditioned RL when rewards are sparse
-relabels failed trajectories with goals actually achieved during the trajectory
-provides a denser learning signal despite a sparse reward function

Bottleneck layer

-constrain an intermediary actor-network layer to predict the full state
-forces policy-network latents to align with the underlying physical state