Precision@k, Recall@k & NDCG

Position Matters in Ranking Systems

In classification, getting a positive prediction correct scores the same regardless of order.

In search and recommendation systems, position order is everything:

• User sees item at Rank 1 $\implies$ High utility and satisfaction!
• User sees item buried at Rank 20 $\implies$ User leaves the platform before scrolling down!

Ranking evaluation metrics measure the quality of Top-K recommended items.

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. PRECISION@K           │ 2. RECALL@K              │ 3. NDCG@K                │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Proportion of top-K      │ Fraction of ALL true     │ Graded relevance metric  │
│ slots containing binary  │ positive items retrieved │ with logarithmic position│
│ relevant items.          │ inside top-K slots.      │ discounting. Gold Standard!│
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

1. Precision@K and Recall@K

Suppose a user liked 5 movies globally ($5$ Total Relevant Items). We recommend a top-5 list ($K = 5$) containing 3 liked movies:

$$\text{Precision@5} = \frac{\text{Relevant Items in Top 5}}{5} = \frac{3}{5} = 0.60$$ $$\text{Recall@5} = \frac{\text{Relevant Items in Top 5}}{\text{Total Relevant Items Globally}} = \frac{3}{5} = 0.60$$

• Limitation: Neither Precision@K nor Recall@K cares about order inside the top-K slots! A list with relevant items at ranks [1, 2, 3] gets the exact same score as [3, 4, 5].

2. Normalized Discounted Cumulative Gain (NDCG@K)

NDCG solves the position problem and supports Multi-Level Graded Relevance (for example $0 = \text{Irrelevant}, 1 = \text{Fair}, 2 = \text{Good}, 3 = \text{Perfect}$).

Step 1: Discounted Cumulative Gain (DCG@K)

$$\text{DCG}@K = \sum_{i=1}^K \frac{2^{r_i} - 1}{\log_2(i + 1)}$$

Logarithmic discounting $\frac{1}{\log_2(i+1)}$ heavily penalizes relevant items placed lower down the list.

Step 2: Ideal DCG@K (IDCG@K)

Re-sort the top-K items in perfect descending relevance order and compute DCG@K.

Step 3: Normalization (NDCG@K)

$$\text{NDCG}@K = \frac{\text{DCG}@K}{\text{IDCG}@K}$$

Normalizing against ideal sorting outputs a standardized score between $0.0$ and $1.0$.

Metric Comparison Table

Say this out loud

Ranking metrics evaluate ordered recommendation lists. Precision at K measures the proportion of relevant items inside top K slots. Recall at K measures the fraction of total relevant items captured. NDCG at K is the industry gold standard metric for multi level graded relevance, discounting item relevance logarithmically based on rank position and normalizing against ideal sorting.

Followups to expect

1. What is Mean Reciprocal Rank (MRR)? Evaluates reciprocal rank $1 / \text{rank}$ of the first relevant item, suitable for navigational search where users seek a single target result.
2. How do you choose K in offline evaluation? Match K to actual user interface screen capacity: $K = 5$ or $K = 10$ for mobile feed cards, $K = 20$ for web search engine pages.