Deep Q-Networks & Replay Buffers

The DQN Architecture

  Raw Input Screen (84x84x4 pixels) ──► [ CNN Layers + FFN ] ──► Vector of Q-Values [ Q(s, a1), Q(s, a2), ..., Q(s, aK) ]

A single forward pass computes $Q$-values for all discrete actions simultaneously!

The Two Core Stabilizing Innovations

When combining Q-Learning with Deep Neural Networks, 3 sources of instability arise:

1. Correlations between sequential observations ($s_t, s_{t+1}$).
2. Non-stationary data distribution (Policy changes as $Q$ updates).
3. Chasing a Moving Target: Updating $\theta$ changes both current $Q(s, a; \theta)$ AND target $r + \gamma \max_{a'} Q(s', a'; \theta)$.

                            DQN STABILIZATION TRIAD
┌───────────────────────────────────────┬───────────────────────────────────────┐
│ 1. EXPERIENCE REPLAY BUFFER (D)       │ 2. SEPARATE TARGET NETWORK (θ^-)      │
├───────────────────────────────────────┼───────────────────────────────────────┤
│ Store 1M transitions (s, a, r, s').   │ Maintain periodic copy of weights θ^-.│
│ Sample random mini-batches for SGD.   │ Freeze θ^- for C steps (e.g. C=10k).  │
│ Breaks temporal correlation -> i.i.d.! │ Eliminates moving target feedback!    │
└───────────────────────────────────────┴───────────────────────────────────────┘

Loss Function & Target Update

DQN minimizes Loss $\mathcal{L}_i(\theta_i)$ over random mini-batch $U(D)$:

$$\mathcal{L}_i(\theta_i) = \mathbb{E}_{(s, a, r, s') \sim U(D)} \left[ \left( \underbrace{r + \gamma \max_{a'} Q(s', a'; \theta_i^-)}_{\text{Target (Frozen Weights } \theta^-)} - \underbrace{Q(s, a; \theta_i)}_{\text{Prediction (Active Weights } \theta)} \right)^2 \right]$$

Every $C$ steps, copy active weights to target network: $\theta^- \leftarrow \theta$.

DQN Extension Family

• Double DQN: Decouples action selection from evaluation to fix overestimation bias.
• Dueling DQN: Splits network architecture into two streams: State-Value $V(s)$ and Advantage $A(s, a)$, recombining $Q(s,a) = V(s) + (A(s,a) - \frac{1}{|A|}\sum A)$.
• Prioritized Experience Replay (PER): Samples transitions proportional to TD error magnitude $|\delta_i|$, re-playing high-learning-value mistakes more frequently.

Say this out loud

"DQN scales Q-learning to high-dimensional pixel inputs using deep CNNs. Training is stabilized using Experience Replay to break sample autocorrelation and a frozen Target Network Q(s,a; θ^-) to eliminate the moving target problem. Double DQN further decouples action selection from evaluation to eliminate overestimation bias."

Follow-ups to expect

• What is Dueling DQN architecture? Separates network branches into $V(s)$ and $A(s, a)$ before combining: $Q(s, a) = V(s) + \left( A(s, a) - \frac{1}{|A|} \sum_{a'} A(s, a') \right)$. This allows the model to learn state values $V(s)$ without needing to visit every individual action.
• Can DQN handle continuous action spaces? No. Calculating $\max_a Q(s,a)$ over continuous actions requires solving an optimization problem at every step. Use Actor-Critic algorithms (DDPG, SAC) for continuous control.