Value-Based vs Policy-Based Methods

Comparing the two core families of Reinforcement Learning algorithms: learning action values vs directly optimizing policy parameters.

🟡 intermediate5 min readrlmust-know

Reinforcement Learning algorithms divide into Value-Based methods (Q-Learning, DQN) and Policy-Based methods (REINFORCE, PPO). Value-based methods learn action-value function Q(s, a), deriving implicit greedy policies a = argmax_a Q(s, a). Policy-based methods parameterize policy π_θ(a|s) directly, optimizing parameters θ via gradient ascent on expected return. Actor-Critic methods combine both: an Actor policy π_θ(a|s) and a Critic value function V_ϕ(s).

Comparison Matrix

       VALUE-BASED (DQN)                        POLICY-BASED (REINFORCE)                     ACTOR-CRITIC (PPO)
  Learns Q(s, a)                         Learns π_θ(a|s)                              Learns BOTH!
  Policy: a = argmax_a Q(s, a)           Optimizes θ via ∇_θ J(θ)                     Actor π_θ(a|s) + Critic V_ϕ(s)

Property	Value-Based (DQN, SARSA)	Policy-Based (REINFORCE)	Actor-Critic (PPO, SAC, A2C)
Core Target Learned	Action-Value $Q(s, a)$	Policy Distribution $\pi_\theta(a \mid s)$	Both Policy $\pi_\theta(a \mid s)$ & Value $V_\phi(s)$
Action Space	Discrete (Buttons, Moves)	Continuous & Discrete	Continuous & Discrete
Policy Type	Deterministic (Greedy $\arg\max_a$ )	Stochastic / Continuous Gaussian	Stochastic / Continuous
Sample Efficiency	High (Off-policy Replay Buffer)	Low (On-policy Monte Carlo)	Moderate to High
Variance	Low (Bootstrapping)	High (Monte Carlo Return $G_t$ )	Low (Advantage $A = Q - V$ )

Mathematical Objectives

1. Value-Based Loss (DQN)

Minimizes Bellman Mean Squared Error over replay buffer $D$ :

\mathcal{L}(\theta) = \mathbb{E}_{(s,a,r,s') \sim D} \left[ \left( r + \gamma \max_{a'} Q(s', a'; \theta^-) - Q(s, a; \theta) \right)^2 \right]

2. Policy-Based Objective (Policy Gradient Theorem)

Maximizes expected trajectory return $J(\theta) = \mathbb{E}_{\pi_\theta}[G_0]$ via gradient ascent:

\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta} \left[ \nabla_\theta \ln \pi_\theta(a \mid s) \cdot G_t \right]

3. Actor-Critic Objective

Replaces Monte Carlo return $G_t$ with Advantage Function $A^\pi(s, a) = Q^\pi(s, a) - V^\pi(s)$ :

\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta} \left[ \nabla_\theta \ln \pi_\theta(a \mid s) \cdot A^\phi(s, a) \right]

Say this out loud

"Value-based methods (DQN) learn action values Q(s,a) for discrete actions, deriving greedy policies via argmax_a Q(s,a). Policy-based methods (REINFORCE) parameterize policy π_θ(a|s) directly using gradient ascent, handling continuous action spaces naturally. Actor-Critic methods (PPO) combine both: the Actor learns policy π_θ and the Critic learns value baseline V_ϕ(s) to reduce gradient variance."

Follow-ups to expect

What is On-Policy vs Off-Policy learning? On-policy methods (PPO, REINFORCE) evaluate and update the exact policy currently generating environment samples. Off-policy methods (DQN, SAC) evaluate one policy while using samples generated by an older/different exploration policy stored in a Replay Buffer.
What is Soft Actor-Critic (SAC)? Off-policy maximum-entropy actor-critic algorithm that adds entropy bonus $\mathcal{H}(\pi(\cdot|s))$ to the reward objective, encouraging exploration in continuous control tasks.

Check yourself

Question 1 of 3

Why do Value-Based methods (Q-Learning / DQN) struggle with continuous action spaces (e.g. steering wheel angle from -180° to +180°)?

More in Reinforcement Learning

See all →

Multi-Armed Bandits4 min Proximal Policy Optimization5 min Markov Decision Processes4 min