Policy Gradients & REINFORCE

The Policy Objective Function $J(\theta)$

We parameterize a policy $\pi_\theta(a \mid s)$ using neural network weights $\theta$.

The goal is to maximize total expected discounted return over trajectories $\tau = (s_0, a_0, r_0, s_1, a_1, r_1, \dots)$:

$$J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[R(\tau)] = \int P(\tau; \theta) R(\tau) d\tau$$

Where trajectory probability $P(\tau; \theta) = P(s_0) \prod_{t=0}^T P(s_{t+1} \mid s_t, a_t) \pi_\theta(a_t \mid s_t)$.

The Log-Derivative Trick & Policy Gradient Theorem

Taking the gradient with respect to $\theta$:

$$\nabla_\theta J(\theta) = \nabla_\theta \int P(\tau; \theta) R(\tau) d\tau = \int \nabla_\theta P(\tau; \theta) R(\tau) d\tau$$

Applying Log-Derivative Trick ($\nabla_\theta P(\tau; \theta) = P(\tau; \theta) \nabla_\theta \ln P(\tau; \theta)$):

$$\nabla_\theta J(\theta) = \int P(\tau; \theta) \nabla_\theta \ln P(\tau; \theta) R(\tau) d\tau = \mathbb{E}_{\tau \sim \pi_\theta} \left[ \nabla_\theta \ln P(\tau; \theta) R(\tau) \right]$$

Notice that $\ln P(\tau; \theta) = \ln P(s_0) + \sum_{t=0}^T \ln P(s_{t+1} \mid s_t, a_t) + \sum_{t=0}^T \ln \pi_\theta(a_t \mid s_t)$.

Taking derivatives eliminates unknown environment terms $P(s_{t+1} \mid s_t, a_t)$ completely!

$$\mathbf{\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta} \left[ \sum_{t=0}^T \nabla_\theta \ln \pi_\theta(a_t \mid s_t) \cdot G_t \right]}$$

                           Intuition of Policy Gradient Step
                             θ ← θ + α · ∇_θ ln π_θ(a_t | s_t) · G_t
  If Trajectory Return G_t > 0 ──► INCREASE probability of taking action a_t in state s_t!
  If Trajectory Return G_t < 0 ──► DECREASE probability of taking action a_t in state s_t!

The REINFORCE Algorithm (Monte Carlo Policy Gradient)

# REINFORCE Update Loop
for episode in range(num_episodes):
    trajectory = generate_trajectory(policy_net)  # Collect full episode
    for t in range(len(trajectory)):
        G_t = sum(gamma**k * r for k, r in enumerate(rewards[t:]))  # Monte Carlo Return
        loss = -log_prob(actions[t]) * G_t                          # Policy Gradient Loss
        loss.backward()
    optimizer.step()

Say this out loud

"Policy Gradient methods optimize policy parameters θ directly via gradient ascent on expected return J(θ). The Policy Gradient Theorem uses the log-derivative trick to eliminate unknown environment transition dynamics, deriving ∇_θ J(θ) = E [ ∇_θ ln π_θ(a|s) · G_t ]. REINFORCE uses Monte Carlo episode returns G_t as unbiased estimators, but suffers from high variance."

Follow-ups to expect

• How do you reduce variance in REINFORCE? Subtract a state-dependent Baseline $b(s)$ from return $G_t$: $\nabla_\theta J = \mathbb{E}[ \nabla_\theta \ln \pi_\theta(a|s) (G_t - b(s)) ]$. Setting $b(s) = V^\phi(s)$ yields the Advantage Function $A(s,a) = G_t - V(s)$, significantly lowering variance without introducing bias.
• What is Continuous Action Policy Parametrization? For continuous actions $a \in \mathbb{R}^d$, the policy network outputs Mean $\mu_\theta(s)$ and Standard Deviation $\sigma_\theta(s)$ of a Gaussian distribution $\mathcal{N}(\mu, \sigma^2)$, sampling actions via reparameterization.