1. The Problem with RNNs and the Birth of Attention

1.1 The Sequential Bottleneck

Before Transformers, the industry standard for processing text was the Recurrent Neural Network (RNN). RNNs read a sentence word-by-word from left to right. When processing the 10th word, the network relied on a "hidden state" that summarized the previous 9 words. This sequential design suffered from two fatal flaws.

First, it suffered from catastrophic forgetting. By the time an RNN reached the end of a long paragraph, the hidden state had often "forgotten" the critical context from the first sentence. Second, because it processed words sequentially, it could not leverage modern GPU parallelization. You cannot process word 10 until you finish word 9.

1.2 The Transformer Solution

Transformers discard sequence entirely. They ingest the entire sentence at once. To figure out how words relate to each other, the Transformer uses Self-Attention. Every single word in the sentence looks at every other word in the sentence simultaneously, calculates a "relevance score," and blends their meanings together. Because there is no sequential dependence, we can calculate these scores for all words at the exact same time using massive matrix multiplications on a GPU.

2. The Cocktail Party: Queries, Keys, and Values

The genius of Self-Attention is its use of a database retrieval analogy. Imagine you are at a crowded cocktail party (the sentence). You are a specific word, and you want to find other words to talk to in order to understand your own context.

• Query (Q): What you are looking for. (e.g., "I am an adjective looking for the noun I modify.")
• Key (K): What other words advertise about themselves. (e.g., "I am a singular, animate noun.")
• Value (V): The actual substantive meaning or context the word provides once a match is found.

To create these Q, K, and V vectors, the model maintains three learnable weight matrices ($W_Q$, $W_K$, $W_V$). We take our initial word embedding vector and multiply it by these three matrices to generate the word's Query, Key, and Value.

import numpy as np

# Let's say we have 3 words, each represented by an embedding of size 4
X = np.array([
    [1.0, 0.0, 1.0, 0.0],  # "The"
    [0.0, 2.0, 0.0, 2.0],  # "cat"
    [1.0, 1.0, 1.0, 1.0]   # "sat"
])

# Initialize random weight matrices (size 4x3) to project into dimension d_k = 3
W_Q = np.random.randn(4, 3)
W_K = np.random.randn(4, 3)
W_V = np.random.randn(4, 3)

# Project the input to get Q, K, V
Q = np.dot(X, W_Q)  # Shape: (3, 3)
K = np.dot(X, W_K)  # Shape: (3, 3)
V = np.dot(X, W_V)  # Shape: (3, 3)

3. The Attention Formula and The Worked Trace

3.1 Calculating Relevance Scores

To figure out how much the word "cat" should pay attention to "sat", we take the Query vector of "cat" and calculate the dot product with the Key vector of "sat". The dot product is a mathematical measure of similarity. A high dot product means the Query and Key align perfectly; the words are highly relevant to each other.

Because we want to do this for all words simultaneously, we multiply the entire Query matrix by the transpose of the Key matrix: $Q \times K^T$. This results in a square "Attention Score" matrix where row $i$ and column $j$ represent how much word $i$ attends to word $j$.

3.2 The Worked Trace

$$Attention(Q, K, V) = ext{softmax}\left(rac{Q K^T}{\sqrt{d_k}} ight) V$$

import math

# Step 1 & 2: Dot product and Scale
d_k = Q.shape[1]
scores = np.dot(Q, K.T) / math.sqrt(d_k)

# Step 3: Softmax (row-wise)
exp_scores = np.exp(scores - np.max(scores, axis=1, keepdims=True)) # stability trick
attention_weights = exp_scores / np.sum(exp_scores, axis=1, keepdims=True)

# Step 4: Multiply by Values
output = np.dot(attention_weights, V)

4. Why Divide by the Square Root? (The Scaling Factor)

One of the most common questions is why the original Google researchers included the $rac{1}{\sqrt{d_k}}$ scaling factor. Why not just take the raw dot product?

Assume the elements in the $Q$ and $K$ vectors are independent random variables with a mean of 0 and a variance of 1. When you take the dot product of two vectors of dimension $d_k$, you are summing $d_k$ products. The mean of this sum is still 0, but the variance grows to $d_k$. A large variance means the dot products will produce extremely large positive and negative numbers.

The Softmax function uses exponentials ($e^x$). If you feed Softmax extremely large numbers (like 112), $e^{112}$ becomes astronomically larger than $e^{12}$. The Softmax output will slam to 1.0 for the highest value and 0.0 for everything else. This is a disaster for neural networks because the gradient (the derivative used for backpropagation) in these "flat" regions of the Softmax curve approaches exactly zero. The network completely stops learning—a phenomenon known as the vanishing gradient problem.

By dividing by $\sqrt{d_k}$, we reduce the variance back to 1. The numbers stay small, the Softmax curve remains gentle, and gradients flow smoothly during training.

5. Multi-Head Attention: Multiple Perspectives

If a single Attention block is good, multiple blocks are better. When reading the sentence "The bank of the river," the word "bank" needs to understand its syntactic role (it is a noun), its semantic meaning (related to water), and its grammatical relations. A single set of $Q, K, V$ matrices might struggle to capture all these distinct concepts simultaneously.

Instead of one massive attention operation, we split our embeddings into multiple "Heads." We project the input into $h$ different, smaller sets of $Q, K, V$ matrices. Head 1 might learn to attend strictly to adjectives. Head 2 might learn to look backwards at previous verbs. Each head performs the scaled dot-product attention completely independently in parallel.

Finally, we concatenate the outputs of all $h$ heads together, multiply them by a final output weight matrix ($W_O$), and pass them forward. This allows the model to jointly attend to information from different representation subspaces at different positions.

6. Frequently Asked Questions