Beginner Basics

Article 2 explains how attention, position, and stable training operate inside a Generative Pre-Trained Transformer block.

Decoder-only Transformer blocks use projections, attention, causal masking, and feed-forward layers to build stable, context-aware language models.

Article 2 explains how attention, position, and stable training operate inside a Generative Pre-Trained Transformer block.

Key Takeaways

  • Decoder-only Transformer blocks replace the original encoder-decoder design: stacked blocks use self-attention and feed-forward layers to predict next tokens from contextualized embeddings.
  • Token embeddings are projected into Query, Key, and Value matrices via learned weights; queries encode what a position seeks, keys describe what it offers, and values carry the semantic information used to update representations.
  • Attention computes similarity scores with QKᵀ, scales them by √d_head to stabilize gradients, and applies a causal mask that blocks future-token attention, keeping autoregressive generation valid.
  • The article connects this architecture to practical PyTorch-style implementation, emphasizing stable training, multi-head parallelization, and feed-forward expansion/contraction of embeddings.

Meta description: Decoder-only Transformer blocks use 128-dimensional embeddings, four heads, matrix projections, scaled dot-product attention, causal masks, and PyTorch code.

Article 2 explains how attention, position, and stable training operate inside a Generative Pre-Trained Transformer bloc

Tags: Decoder-only Transformers, MultiHeadAttention, Scaled Dot-Product Attention, PyTorch, Causal Masking

Featured image: Abstract friendly illustration of token vectors becoming attention scores, no text, no logos, no people.

When the seminal 2017 paper “Attention Is All You Need” introduced the original Transformer, it paired an Encoder with a Decoder. Modern Large Language Models (LLMs) such as GPT, Llama, Mistral, and DeepSeek removed the encoder and the cross-attention path, leaving a streamlined decoder-only Transformer architecture built to predict the next token in a sequence. For readers who build or evaluate language models, the practical question is how the remaining blocks turn token vectors into context-aware outputs, and how PyTorch can implement that process.

From Encoder-Decoder Design to Decoder-Only Blocks

The original encoder processed input text in parallel, while the decoder generated output text token-by-token and relied on Cross-Attention to look back at the encoder’s data. Today, when engineers talk about a “Transformer block” or a “GPT block,” they usually mean the modern decoder-only variant. A decoder-only language model stacks dozens or hundreds of these blocks. Each block takes a sequence of vectors, lets tokens exchange information through self-attention, and then processes each token’s updated representation independently through a feed-forward network.

In the teaching model, the embedding width is d_model = 128. That means each token has a vector representation with 128 dimensions, a way of trying to comprehend 128 different features of a token, which is not always a word or a character. The model uses four attention heads (heads = 4), parallelizing learning across four threads so each head can learn its own weights independently. After those features are learned, they are combined and passed through a feed-forward network that first expands, then contracts, the representation to an output of the exact same size. The point is to pass an embedding through learning blocks to interpret its context relative to all other tokens in the sequence. In production models, these architectural choices vary significantly; the numbers here are concrete examples, not a universal recipe.

Matrix Projections Create Queries, Keys, and Values

A matrix is data arranged in rows and columns. In an LLM, the input data matrix X has dimensions of (Context Length × d_model). For a production workload, this could be 32,000 × 512, where 32,000 is the context length in number of tokens and 512 is the input embedding size. Matrix multiplication requires that the columns of the first matrix match the rows of the second: multiplying (A × B) by (B × C) yields a new matrix of size (A × C).

For each embedding representation X, the layer applies learned linear projections:

Article 2 explains how attention, position, and stable training operate inside a Generative Pre-Trained Transformer bloc

Q = X · Wq
K = X · Wk
V = X · Wv

X is the input embedding matrix, multiplied by learned weight matrices Wq, Wk, and Wv. These weights are initialized randomly, using frameworks like PyTorch via torch.tensor(requires_grad=True), and are optimized during training via backpropagation.

A concrete example uses sequence length 4 and d_model = 4. Let X = [[1,0,1,2],[0,2,1,0],[1,1,0,1],[2,0,2,1]] and Wq = [[1,0,1,0],[0,1,0,1],[1,-1,0,0],[0,1,1,-1]]. Multiplying row by row gives [2,1,3,-2], [1,1,0,2], [1,2,2,0], and [4,-1,3,-1]. The source then displays a final Q whose third row is [1,2,2,1], so the example contains a mismatch between the row calculation and the final matrix.

Initially, all weight matrices hold small random values near zero. Over the course of training, they learn to capture distinct features. The transformed vectors serve different functional roles: Query (Q) asks what a token position is looking for, Key (K) describes what the token position offers to the context, and Value (V) carries the actual semantic information to be combined. The projection shape can be visualized as [Context × d_model] × [d_model × d_out] = [Context × d_out]. For example, [32,000 × 512] × [512 × 4,096] = [32,000 × 4,096].

Scaled Dot-Product Attention Adds Causal Masks

The attention mechanism allows token embeddings to calculate their mutual similarities across the provided context window. For a single head, the equation is:

Attention(Q, K, V) = Softmax((Q · Kᵀ / √d_head) + Causal Mask) · V

First, Q and Kᵀ are multiplied. If Q is of size (32,000 × 4,096), it is multiplied by the transpose of the Key matrix Kᵀ, which has dimensions of (4,096 × 32,000). This matrix multiplication results in an attention score matrix of size (32,000 × 32,000), mapping the raw relationship score between every pair of positions.

Article 2 explains how attention, position, and stable training operate inside a Generative Pre-Trained Transformer bloc

Second, the scores are divided by √d_head. As the head dimension grows larger, the dot products grow larger in magnitude. This pushes the softmax function into regions with extremely small gradients. Dividing by √d_head scales these values down, stabilizing the gradient flow during backpropagation.

Third, an autoregressive decoder must not look into the future. When predicting the next word, it should only have access to current and past tokens. To enforce this condition, a causal mask is added to the scaled scores before applying the softmax function. This mask sets all upper-triangular positions, representing future tokens, to negative infinity (-∞). Since e^(-∞) = 0, the softmax operation assigns a probability of exactly 0 to those future tokens.

A small example shows the sequence. Raw QKᵀ scores are [12,4,8], [3,9,2], and [7,6,11]. After causal masking, they become [12,-∞,-∞], [3,9,-∞], and [7,6,11]. After softmax, the rows become [1.0,0.0,0.0], [0.002,0.998,0.0], and [0.02,0.01,0.97]. Applying softmax guarantees that the weights across each row sum exactly to 1.0, turning them into a valid probability distribution.

For instance, at token position 4, the model might assign the weights [0.05, 0.15, 0.70, 0.10] to positions 1 through 4. The final output vector for position 4 is computed as a weighted mixture of those historical value vectors:

Output_4 = 0.05·V₁ + 0.15·V₂ + 0.70·V₃ + 0.10·V₄

Implementing MultiHeadAttention and GroupedQueryAttention in PyTorch

The source presents a PyTorch path for four major attention variants used across the industry, then defines MultiHeadAttention and GroupedQueryAttention. The implementation can be followed in numbered steps.

  1. Create a torch.nn.Module for MultiHeadAttention with d_model and num_heads. Assert that d_model % num_heads == 0, with the message “d_model must be divisible by num_heads,” because each head receives d_head = d_model // num_heads dimensions.
  2. Build four nn.Linear projections without bias: W_q, W_k, W_v, and out_proj. In standard MultiHeadAttention, the module allocates distinct W_q, W_k, and W_v paths for every single head.
  3. In forward, unpack batch size B, sequence length T, and embedding dimension C from x.shape.
  4. Project x through W_q, W_k, and W_v; reshape the flat channel dimension into num_heads and d_head; then transpose so the heads dimension comes before T. This allows fast, parallel matrix multiplication across all batches and heads at once.
  5. Compute scores = (q @ k.transpose(-2, -1)) / (self.d_head ** 0.5). Add a causal mask created with torch.triu(torch.full((T, T), float('-inf'), device=x.device), diagonal=1). The zeros on the lower half allow past tokens to talk, while -inf blocks future tokens.
  6. Apply F.softmax to the masked scores, multiply the result by v to get context vectors for each head, transpose heads and sequence length back to the original order, call contiguous, view the tensor back to B, T, C, and pass the context through out_proj.
  7. For GroupedQueryAttention, accept d_model, num_query_heads, and num_kv_groups. Set d_head = d_model // num_query_heads and queries_per_group = num_query_heads // num_kv_groups.
  8. Make W_q project to num_query_heads * d_head, while W_k and W_v project to num_kv_groups * d_head. Instead of matching the number of Query heads, this restricts the Key and Value heads to a smaller number to compress the memory footprint.
  9. In forward, project q, k, and v, then use repeat_interleave(queries_per_group, dim=1) on k and v so the shared Key and Value heads are duplicated across the head dimension and match the exact shape of the Query matrix for fast, parallel batch matrix multiplication.

Common mistakes

The most common implementation errors trace directly to the rules above. First, setting num_heads so that d_model is not divisible by num_heads breaks the head split. Second, omitting the causal mask lets a decoder train on future tokens, which ruins next-token prediction. Third, forgetting the √d_head scaling can make softmax scores too large, weakening gradient flow. Fourth, treating the teaching numbers d_model = 128, heads = 4, or the production example 32,000 × 512 as universal choices can mislead readers; production architecture varies. Fifth, in GroupedQueryAttention, failing to repeat_interleave shared Key and Value heads prevents the tensor shapes from matching the Query heads.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.