GRPO & Group-Relative Methods

The Paradigm Shift: PPO vs GRPO

  TRADITIONAL PPO RLHF:
  Prompt x ──► Policy LLM ──► Response y ──► Reward Model r(x,y)
                    │
                    ▼
     Requires CRITIC MODEL V_ϕ(x) to compute Advantage: A(x,y) = r(x,y) - V_ϕ(x)
     (Massive VRAM Overhead: Actor + Critic + Ref + Reward = 4 Models!)

  GROUP RELATIVE POLICY OPTIMIZATION (GRPO):
  Prompt x ──► Policy LLM ──► Samples Group of G = 4 Responses: {y_1, y_2, y_3, y_4}
                    │
                    ▼
     Scalar Rewards: {r_1, r_2, r_3, r_4}
     Group Mean:     μ_r = mean(r)
     Group Std:      σ_r = std(r)
     Group Advantage: A_i = (r_i - μ_r) / (σ_r + ε)  ──► NO CRITIC MODEL NEEDED!
     (50% Lower VRAM: Actor + Ref + Reward = 3 Models!)

The GRPO Mathematical Objective

For prompt $x$ and group of $G$ sampled outputs $\{y_1, y_2, \dots, y_G\}$:

$$\mathcal{L}_{\text{GRPO}}(\theta) = \hat{\mathbb{E}} \left[ \frac{1}{G} \sum_{i=1}^G \min\left( \frac{\pi_\theta(y_i \mid x)}{\pi_{\text{old}}(y_i \mid x)} A_i, \text{clip}\left( \frac{\pi_\theta(y_i \mid x)}{\pi_{\text{old}}(y_i \mid x)}, 1-\epsilon, 1+\epsilon \right) A_i \right) - \beta D_{KL}(\pi_\theta \parallel \pi_{\text{ref}}) \right]$$

Group Advantage $A_i$:

$$A_i = \frac{r_i - \text{mean}(\{r_1, \dots, r_G\})}{\text{std}(\{r_1, \dots, r_G\}) + \epsilon}$$

Memory & VRAM Breakdown

Why GRPO Excels in Reasoning Models (DeepSeek-R1)

For math and coding tasks, binary rule-based verifiers provide exact execution rewards ($r_i = 1.0$ if code passes unit tests; $r_i = 0.0$ if it fails).

Sampling $G = 8$ candidate reasoning chains per prompt creates a natural intra-prompt tournament:

• Responses that solve the math problem receive positive normalized Advantage $A_i > 0$.
• Responses that fail receive negative normalized Advantage $A_i < 0$.

Grades update the Policy LLM without needing any neural value estimator!

Say this out loud

"GRPO eliminates the memory-heavy Critic Value Model in RLHF. For each prompt, GRPO samples a group of G responses, computing baseline value dynamically as group average reward. Advantage for response i is normalized against group mean and std: A_i = (r_i - μ_r) / σ_r. DeepSeek-R1 used GRPO to cut VRAM memory by 50% while scaling reasoning performance."

Follow-ups to expect

• What is the optimal Group Size G in GRPO? DeepSeek-Math empirically evaluated group sizes $G \in [4, 16]$, finding $G = 4\text{--}8$ provides optimal variance reduction without inflating generation rollout time.
• How does GRPO handle KL Divergence? Instead of adding KL penalty directly into reward $r_i$, GRPO adds an explicit KL divergence loss term $D_{KL}(\pi_\theta \parallel \pi_{\text{ref}})$ directly to the main loss function.