Proximal Policy Optimization

Why PPO Was Created: Trust Region Motivation

In supervised learning, if a gradient step is too large, the next mini-batch corrects it.

In Reinforcement Learning, taking an overly aggressive policy step $\Delta \theta$ ruins data collection: the degraded policy generates bad trajectory data, causing irrecoverable policy collapse.

• TRPO (Trust Region Policy Optimization): Enforces strict KL constraint $D_{KL}(\pi_{\text{old}} \parallel \pi_\theta) \le \delta$ using complex Second-Order Conjugate Gradient optimization.
• PPO (Proximal Policy Optimization): Achieves the stability of TRPO using First-Order SGD with a Clipped Loss Objective!

The Clipped Surrogate Objective $L^{\text{CLIP}}(\theta)$

Let the Probability Ratio $r_t(\theta)$ be:

$$r_t(\theta) = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\text{old}}(a_t \mid s_t)}, \quad \text{where } r_t(\theta_{\text{old}}) = 1.0$$ $$L^{\text{CLIP}}(\theta) = \hat{\mathbb{E}}_t \left[ \min\left( r_t(\theta) \hat{A}_t, \text{clip}\left( r_t(\theta), 1-\epsilon, 1+\epsilon \right) \hat{A}_t \right) \right]$$

Standard hyperparameter setting: $\epsilon = 0.2$ (Clips ratio $r_t \in [0.8, 1.2]$).

   Positive Advantage (A_t > 0)                       Negative Advantage (A_t < 0)
   L_CLIP                                              L_CLIP
     │                                                   │
  1+ε├─────────────── Upper Bound                        │
     │              /                                    │               /
     │             /                                  1-ε├──────────────/─ Lower Bound
     │            /                                      │             /
  1.0├───────────/                                    1.0├────────────/
     └───────────┴─────────────► r_t                     └────────────┴──────────────► r_t
     1.0      1+ε                                         1-ε        1.0

• Positive Advantage ($\hat{A}_t > 0$): Wants to increase $r_t$. Clipped at $1+\epsilon$ so policy cannot gain extra reward by pushing $r_t > 1.2$.
• Negative Advantage ($\hat{A}_t < 0$): Wants to decrease $r_t$. Clipped at $1-\epsilon$ so policy cannot gain extra reward by pushing $r_t < 0.8$.

Complete PPO Combined Loss

$$L^{\text{PPO}}(\theta) = \hat{\mathbb{E}}_t \left[ L^{\text{CLIP}}(\theta) - c_1 L^{\text{VF}}(\theta) + c_2 S[\pi_\theta](s_t) \right]$$

• $L^{\text{CLIP}}(\theta)$: Clipped policy surrogate loss.
• $L^{\text{VF}}(\theta) = (V_\theta(s_t) - V_t^{\text{target}})^2$: Critic Squared Error Value Loss.
• $S[\pi_\theta](s_t)$: Entropy Bonus encouraging exploration.

Say this out loud

"PPO stabilizes policy gradient updates using a Clipped Surrogate Objective L_CLIP(θ) = E [ min( r_t A_t, clip(r_t, 1-ε, 1+ε) A_t ) ]. By capping probability ratio r_t = π_θ / π_old within [1-ε, 1+ε] (typically [0.8, 1.2]), PPO prevents destructive large policy steps, providing smooth, monotonic policy improvements in RL and LLM RLHF."

Follow-ups to expect

• How is PPO used in LLM RLHF alignment? The Policy LLM generates responses $y$ for prompts $x$. PPO updates LLM parameters $\theta$ using Advantage $A(x,y) = R_{\psi}(x,y) - \beta D_{KL}(\pi_\theta \parallel \pi_{\text{ref}}) - V_\phi(x)$.
• What is PPO-Penalty vs PPO-Clip? PPO-Penalty enforces KL divergence as an adaptive loss penalty term $\beta D_{KL}$. PPO-Clip uses hard ratio clipping. PPO-Clip is almost universally preferred due to superior empirical stability.