Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (Haarnoja et al.)

July 28, 2026

-maximize expected reward while maximizing entropy = acting as randomly as possible

- assigns meaningful probabilities to a wider range of actions, prevents collapsing to local maxima or near-optimal behaviors early in training

-on-policy RL must be online, with new trajectories collected for each gradient step / policy iteration

- off-policy RL reuses past experiences

Critic ($`Q_{\theta}(s, a)`$)

-soft Q-value: expected discounted reward plus an entropy bonus

-off-policy: train from transitions

(s, a, r, s')

sampled from a replay buffer

-use two Q-networks (clipped double-Q) and take their minimum to reduce overestimation bias

- online critic is updated directly with Adam + LR from current weights

- target critic is updated using Polyak averaging from its corresponding online critic after each step

\bar{\theta}_i \leftarrow \rho\bar{\theta}_i + (1-\rho)\theta_i, \qquad \rho \approx 1

Soft Bellman target

For a sampled next action $a' \sim \pi_{\phi}(\cdot \mid s')$ :

$y = r + \gamma \left[\min_{i=1,2} Q_{\bar{\theta}_i}(s', a') - \alpha \log \pi_{\phi}(a' \mid s')\right]$

Q_{\bar{\theta}_i}

are target critics

-\alpha \log \pi_{\phi}(a' \mid s')

is the entropy bonus: unlikely actions contribute more to the target

- $\alpha$ = temperature

Critic loss and gradient

Fit each critic with squared Bellman error:

$J_Q(\theta_i) = \mathbb{E}_{(s,a,r,s') \sim \mathcal{D}}\left[\frac{1}{2}\left(Q_{\theta_i}(s,a) - y\right)^2\right]$

Taking the gradient gives:

$\nabla_{\theta_i} J_Q = \mathbb{E}_{\mathcal{D}}\left[\left(Q_{\theta_i}(s,a) - y\right)\nabla_{\theta_i}Q_{\theta_i}(s,a)\right]$

-treat

y

as a constant when differentiating; gradients do not flow through the target critics or the next action

-minimize this loss for both critics

Actor (policy) objective

The policy should choose actions that have high value while remaining stochastic. Equivalently, minimize:

$J_{\pi}(\phi) = \mathbb{E}_{s \sim \mathcal{D},\, a \sim \pi_{\phi}}\left[\alpha \log \pi_{\phi}(a \mid s) - \min_{i=1,2}Q_{\theta_i}(s,a)\right]$

-this is the negative of maximizing

Q(s,a) - \alpha\log\pi(a\mid s)

-the first term rewards entropy; the second term favors actions with high critic value

-use the online critics

Q_{\theta_i}

for this actor update. The Polyak-averaged target critics

Q_{\bar{\theta}_i}

are used only in the next-state Bellman target above, which stabilizes critic learning

Actor gradient

Use the reparameterization $a = f_{\phi}(\epsilon; s)$ , with $\epsilon$ sampled from fixed noise. The gradient is:

$\nabla_{\phi}J_{\pi} = \mathbb{E}_{s,\epsilon}\left[\nabla_{\phi}\left(\alpha\log\pi_{\phi}(f_{\phi}(\epsilon;s)\mid s) - \min_i Q_{\theta_i}(s,f_{\phi}(\epsilon;s))\right)\right]$

-gradients pass through the sampled action into the actor, but the critic parameters remain fixed during the actor update

-this directly moves the actor toward actions that are both high-value and sufficiently diverse

Temperature α

-controls the exploration/exploitation tradeoff (entropy weight)