Policy Gradients and Actor-Critic Methods

Policy-gradient methods optimize the policy directly. Instead of learning only which action has the highest value, they adjust the parameters of ${\pi }_{\theta }(a\mid s)$ so actions that led to high return become more likely.

Policy Gradient

The objective is expected return:

$$J(\theta )={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}[G_0].$$

Here $\theta$ parameterizes the policy, $\tau$ is a sampled trajectory generated by that policy, and $G_0$ is the discounted return from the start of the trajectory. Maximizing $J$ means changing the policy so future sampled trajectories have higher return on average.

A common gradient estimator is

$${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{{\pi }_{\theta }}\left[{\nabla }_{\theta }\log {\pi }_{\theta }(A_t\mid S_t)A^{\pi }(S_t,A_t)\right],$$

where $A^{\pi }(s,a)=Q^{\pi }(s,a)-V^{\pi }(s)$ is the advantage. Positive advantage means the action was better than the baseline state value, so the update increases its probability. Negative advantage lowers it.

Actor-Critic Mechanism

Actor-critic methods split the work:

Part	Learns	Role
Actor	${\pi }_{\theta }(a\mid s)$	chooses actions
Critic	$V_{\psi }(s)$ or $Q_{\psi }(s,a)$	estimates future return and advantage

The critic reduces variance because the actor no longer treats every sampled return as equally informative. The actor still learns from sampled experience, but the critic supplies a shaped learning signal.

flowchart TD
  State[State] --> Actor[Actor: policy network]
  State --> Critic[Critic: value network]
  Actor --> Action[Action]
  Action --> Environment[Environment]
  Environment --> Feedback[Reward and next state]
  Feedback --> Critic
  Critic --> Advantage[Advantage estimate]
  Advantage --> Actor

PPO

Proximal Policy Optimization (PPO) limits how far the new policy moves from the old policy during one update. In LLM training, a “PPO loop” usually means repeatedly sampling model responses, scoring them with a reward model, applying a KL penalty against a reference policy, and updating the policy with PPO. With probability ratio

$${\rho }_t(\theta )=\frac{{\pi }_{\theta }(A_t\mid S_t)}{{\pi }_{{\theta }_{\text{old}}}(A_t\mid S_t)},$$

The ratio ${\rho }_t$ compares the new policy’s probability for the sampled action with the old policy’s probability. Values above 1 mean the update is making that action more likely; values below 1 mean it is becoming less likely.

PPO uses a clipped surrogate objective:

$$L^{\text{CLIP}}(\theta )=\mathbb{E}\left[\min \left({\rho }_t(\theta ){\hat{A}}_t,\mathrm{clip}({\rho }_t(\theta ),1-ϵ,1+ϵ){\hat{A}}_t\right)\right].$$

The clipping term discourages a large policy update from making a once-good action suddenly much more likely. That makes PPO practical for many deep RL and RLHF systems. See Proximal Policy Optimization for generalized advantage estimation, the full training loop, and why PPO dominates RLHF.

Soft Actor-Critic

Soft Actor-Critic adds entropy to the objective:

$$J(\pi )=\mathbb{E}\left[\sum _t{\gamma }^t\left(R(S_t,A_t)+\alpha \mathcal{H}(\pi (\cdot \mid S_t))\right)\right].$$

The entropy term rewards stochasticity. This helps exploration and often improves robustness in continuous-control tasks because the policy is not pushed too early into a brittle deterministic action.

Caveats

Policy-gradient methods can be sample-inefficient because they learn from trajectories generated by the current or recent policy. Advantage normalization, trust-region or clipping constraints, entropy bonuses, and careful evaluation are not minor details; they often determine whether training works.

Connections

• Value Functions and Bellman Equations explain the critic’s target.
• Reinforcement Learning from Human Feedback often uses PPO or direct preference optimization after reward modeling.
• Q-Learning and DQN contrasts direct policy optimization with value-based action selection.

References

• Williams, 1992, Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
• Mnih et al., 2016, Asynchronous Methods for Deep Reinforcement Learning
• Schulman et al., 2017, Proximal Policy Optimization Algorithms
• Haarnoja et al., 2018, Soft Actor-Critic Algorithms and Applications

Nav

Section — Reinforcement Learning

← Q-Learning and DQN Proximal Policy Optimization →

Learning path — Reinforcement learning

← Q-Learning and DQN Reinforcement Learning from Human Feedback →