1. The Sequence Problem: Beyond Recurrent Neural Networks (RNNs)

1.1 The Sequential Bottleneck

To model text, a neural network must track connections between words across long sentences. Recurrent Neural Networks (RNNs) process text word-by-word, maintaining a hidden state vector $h_t$ that encapsulates the history of all previous tokens. To calculate state $h_t$, the network must first compute $h_{t-1}$. This sequential dependency makes it impossible to parallelize training over time: you cannot process the 100th word in a sentence until the previous 99 words have finished computing.

Furthermore, RNNs suffer from vanishing gradients. As the hidden state is multiplied repeatedly across long sequences, information about early tokens fades away, making it difficult for the network to link pronouns to distant subject nouns.

1.1 RNN Sequential Updates vs Transformer Parallel Attention

RNNs update states recursively step-by-step. Transformers compute attention scores between all tokens in a single parallel matrix multiplication operation, bypassing time dependencies.

Common Misconception — Transformers contain recurrent loops: A common developer misconception is that Transformers process text sequentially during training. In reality, Transformers have zero recurrence. The entire input sequence is fed into the network simultaneously, and self-attention computes relationships between all tokens in parallel, which is why they train so efficiently on modern GPUs.

2. The Self-Attention Mechanism: Queries, Keys, and Values

2.1 The QKV Projection

The core innovation of the Transformer is the Query-Key-Value (QKV) model, derived from database retrieval systems. When looking up a video, you submit a *Query* (e.g. "cute cats"), the database matches it against *Keys* (video tags/titles), and retrieves the best-matching *Values* (the actual videos). In self-attention, we project each input token embedding into three separate vector spaces:

• **Query ($Q$)**: What the current token is looking for in other words.
• **Key ($K$)**: What information the current token offers to other words.
• **Value ($V$)**: The actual semantic content of the token to be weighted and aggregated.

Mermaid Diagram: The projection of raw token embeddings into Query, Key, and Value vectors for self-attention calculations.

3. Scaled Dot-Product Attention Math

3.1 The Attention Formula

To calculate the relationship between all words, we take the dot product of the Query matrix $Q$ and Key matrix $K$. A higher dot product indicates a stronger semantic connection between two tokens. The resulting similarity scores are scaled down and normalized using Softmax to yield attention weights, which are then used to compute a weighted sum of the Value matrix $V$:

Where $d_k$ is the dimensionality of the Key vectors. The scaling factor $1 / \sqrt{d_k}$ is critical: for large key dimensions, the dot products grow large, pushing the Softmax function into regions with extremely small gradients. Dividing by $\sqrt{d_k}$ keeps the values in active, gradient-friendly regions during backpropagation.

4. Multi-Head Attention: Looking in Parallel

4.1 Subspace Projections

A single attention head calculates one set of relationships. However, a word can relate to other words in multiple ways simultaneously (e.g. a noun relates to its qualifying adjective, its verb action, and referring pronouns). To capture this, **Multi-Head Attention** projects the Q, K, and V matrices into multiple lower-dimensional subspaces in parallel.

Each head calculates attention independently in its subspace. The outputs of all heads are then concatenated and projected back to the original dimensionality, allowing the model to capture diverse contextual relationships across different token distances.

5. Positional Encoding: Restoring Word Order

5.1 Sinusoidal Signals

Because self-attention processes all tokens simultaneously, it is permutation-invariant: scrambling the order of words in a sentence yields the exact same attention outputs. In language, word order is critical. To restore sequence coordinates, the Transformer adds a **Positional Encoding** vector to each input embedding before feeding it into the model.

The positional encoding uses sine and cosine functions of different frequencies, mapping each sequence position to a unique coordinate space. This allows the model to learn relative distances between words easily.

5.2 Positional Encoding Formulations and Linear Shift Projections

To construct unique positional coordinates, the Transformer uses sine and cosine functions of different wavelengths across the hidden dimensions. Let $pos$ represent the position of the token in the sequence (from $0$ to $N-1$), and let $i$ represent the index of the hidden feature dimension (from $0$ to $d_{\text{model}}/2 - 1$). The positional encoding values $PE$ are calculated as:

This specific formulation is chosen because it allows the model to learn to attend by relative positions. For any fixed offset $k$, the positional encoding at position $pos+k$ can be represented as a linear function of the encoding at position $pos$. That is, there exists a linear transformation matrix $M_k \in \mathbb{R}^{d_{\text{model}} \times d_{\text{model}}}$ such that:

This enables the self-attention layer to compute relative distances between words (e.g. knowing that a word is 3 positions to the left of another word) regardless of their absolute positions in the sentence. These encoding values are added directly to the token embeddings:

This ensures that the combined representation contains both semantic content and spatial sequence coordinates. Below is a comparison table of positional encoding strategies:

Pitfall — Scaling context length without RoPE adjustment: If you train a model with sinusoidal or learned embeddings on a context window of 2,048 tokens and attempt to run inference at 4,096 tokens, the attention scores will degrade because the model has never encountered coordinates in that range. RoPE solves this by decaying attention values dynamically as distance increases.

6. The Transformer Encoder-Decoder Pipeline

6.1 Feed-Forward and Normalization Blocks

The classic Transformer consists of an Encoder (which reads the input sequence) and a Decoder (which generates the output sequence token-by-token). Each encoder block contains a multi-head self-attention layer followed by a position-wise fully connected feed-forward network. Residual skip connections and Layer Normalization wrap each sub-layer to stabilize gradient flow.

The decoder block includes an additional **Encoder-Decoder Attention** layer, allowing the decoder to query the representations generated by the encoder, translating input features into target sequences.

6.2 Position-Wise Feed-Forward Networks and GELU Activations

While self-attention allows tokens to exchange context information with each other, it does not apply non-linear transformations to individual token representations. This is performed by the **Position-Wise Feed-Forward Network (FFN)** block, which is applied to each token position independently and identically. The FFN consists of two linear transformations with a non-linear activation function in between:

Where $W_1 \in \mathbb{R}^{d_{\text{model}} \times d_{ff}}$ projects the token embedding to a higher-dimensional space (typically $d_{ff} = 4 \cdot d_{\text{model}}$), and $W_2 \in \mathbb{R}^{d_{ff} \times d_{\text{model}}}$ projects it back. In the original paper, the activation was a Rectified Linear Unit (ReLU). Modern models use the **Gaussian Error Linear Unit (GELU)**, which scales input values by the cumulative distribution function of the standard normal distribution $\Phi(x)$:

This smooths the activation boundary, avoiding dead neurons that block gradient updates during training. In the decoder block, **Encoder-Decoder Cross-Attention** aligns representations. The query matrix $Q$ is projected from the decoder's previous layer, while the key $K$ and value $V$ matrices are projected from the final outputs of the encoder. This allows the decoder to align each generated target word with relevant source context. Below is a comparison table of Transformer sub-blocks:

Pitfall — Decoder exposure bias during inference: During training, the decoder uses teacher forcing: it receives the ground-truth target tokens as input. During inference, however, it must use its own predicted tokens from the previous steps. This mismatch (exposure bias) can cause errors to accumulate, leading to repetitive loops or gibberish output. Use scheduled sampling to mitigate this.

7. Advanced: The Vanishing Gradient Problem in Softmax and Layer Normalization

7.1 Normalization Layers

As neural networks grow deeper, activations can drift, leading to vanishing or exploding gradients. The Transformer resolves this by implementing **Layer Normalization (LayerNorm)** instead of Batch Normalization, calculating mean and variance statistics across features rather than batches.

This makes normalization independent of batch size, which is critical during autoregressive generation where batch sizes are small.

7.2 Layer Normalization and Residual Path Mathematics

Unlike Batch Normalization, which computes statistics across the batch dimension, **Layer Normalization** computes statistics across the hidden feature dimension of a single token. Let $\vec{x}_i \in \mathbb{R}^d$ represent the embedding vector of token $i$ with dimensionality $d$. The mean $\mu_i$ and variance $\sigma_i^2$ for the token's features are computed as:

Each feature element $x_{ij}$ is normalized and then scaled and shifted using learnable parameter vectors $\vec{\gamma}$ and $\vec{\beta}$ (initialized to 1 and 0, respectively):

Where $\epsilon$ is a small constant (e.g. $10^{-5}$) to prevent division by zero. By normalizing across features, LayerNorm ensures that the activation scale remains stable, preserving gradients during training.

The placement of LayerNorm relative to residual connections is a key design decision. Early Transformers used **Post-LN**, placing LayerNorm on the residual path after the sub-layer addition:

While Post-LN yields high performance, it makes training deep models difficult because gradients at output layers are much larger than at input layers, requiring careful learning rate warmup. Modern architectures prefer **Pre-LN**, applying normalization to inputs *before* the sub-layer, passing the raw residual signal directly through skip connections:

This direct highway enables smooth gradient flow, allowing models to scale to hundreds of layers. Below is a comparison table of normalization strategies:

Pitfall — Incorrect Pre-LN vs Post-LN learning rates: If you switch a model config from Pre-LN to Post-LN without introducing a learning rate warmup phase, gradients will explode on step 1, causing loss values to read NaN. Post-LN requires slow learning rate warmups to stabilize early layers.

8. Advanced: Comparison of Sequence Modeling Architectures

8.1 Architecture Strategy Matrix

The table below compares the training efficiency, maximum path length, and sequential complexity of different sequence architectures:

Pitfall — The Quadratic Context Bottleneck: Because self-attention calculates similarity between all pairs of tokens, the computational complexity scales quadratically ($O(N^2)$) with the sequence length $N$. Storing this attention matrix consumes massive VRAM, clamping maximum context window boundaries on standard GPUs.

9. Complete Scaled Dot-Product Attention Implementation in JavaScript

9.1 The JS Matrix Code

The following JavaScript code implements scaled dot-product attention, executing matrix transposition, multiplication, and Softmax scaling:

10. Interactive: Self-Attention Matrix Visualizer

Click "Calculate Attention" to evaluate dot product similarities. Watch how the Softmax normalization highlights matching contextual associations:

Training Latency vs Sequence Length

The chart below compares the training time (in seconds) of Recurrent architectures (RNN/LSTM) vs self-attention (Transformer) models as sequence context window lengths increase:

12. Frequently Asked Questions