Attention Mechanisms and the Transformer Architecture, Explained

Every LLM you’ve used runs on this idea. Attention is the mechanism that lets a model figure out which parts of the input actually matter for the prediction it’s making right now, instead of treating every word as equally important. This post walks through how that works, from the basic idea up to a full Transformer encoder layer.

Why Attention Exists

Older neural network architectures compressed an entire input sequence into one fixed-size representation. Fine for short sequences. It falls apart as sequences get longer, since you’re trying to cram more and more information into the same fixed space.

There’s a second problem, separate from length: not every part of an input matters equally for every prediction. Take this sentence:

“The animal didn’t cross the street because it was too tired.”

To figure out what “it” refers to, a model needs to weight “animal” much more heavily than the other words. Attention is the mechanism that makes that possible: instead of treating every input equally, it assigns different importance scores to different parts of the input. Higher scores mean the model pays more attention there. This dynamic focusing is the core idea that made modern Transformers work.

How Attention Works, at a High Level

For each prediction, attention scores every input element based on relevance:

1. Calculate a relevance score for each input.
2. Convert those scores into attention weights.
3. Combine information from across the inputs, weighted by those scores.

Back to “it was too tired”: the model can assign high attention to “animal” and low attention to everything else, which lets it correctly resolve the reference even though “animal” and “it” are several words apart.

Query, Key, and Value

Modern attention runs on three vectors, generated for every input token:

• Query (Q): what this token is looking for.
• Key (K): what information this token contains.
• Value (V): the actual information this token would contribute if selected.

Here’s how they interact. A token’s query gets compared against every other token’s key. Tokens whose keys match well get higher attention scores. Those scores become weights, and the final output is a weighted combination of every token’s value vector.

Concretely: when processing “it,” its query might strongly match the key belonging to “animal.” That match means “animal“‘s value vector gets weighted heavily in the output for “it.”

Short version: queries ask, keys answer, values deliver.

Scaled Dot-Product Attention

This is the actual math behind turning Q, K, and V into an output.

Step 1, score: compare a token’s query against every key using a dot product. Higher dot product, stronger relevance.

Step 2, scale: divide those scores by the square root of the key dimension. Skip this step and scores can grow large enough to make training unstable.

Step 3, softmax: run the scaled scores through softmax, converting them into probabilities that sum to 1. These are your final attention weights.

Step 4, combine: multiply each value vector by its attention weight, then sum them all up. That sum is the attention output.

The full pipeline:

Query + Keys
     ↓
Relevance Scores
     ↓
Scaling
     ↓
Softmax
     ↓
Attention Weights
     ↓
Weighted Values
     ↓
Attention Output

Self-Attention

Self-attention is the specific case where a sequence attends to itself. Every query, key, and value comes from the same input sequence, which means every token can pull in information from every other token in that same sequence.

The steps:

1. Every token generates its own query, key, and value.
2. Each token’s query gets compared against every key in the sequence.
3. Those comparisons produce attention weights.
4. The weights combine the value vectors into the final output.

Back to the animal sentence one more time: when self-attention processes “it,” it can directly attend to “animal” and pull in that context, which is how the model resolves what “it” means.

What this buys you over older architectures:

• Captures relationships between distant words directly, no matter how far apart they are.
• Lets information flow straight between any two tokens, not just neighboring ones.
• Processes every token in parallel, unlike RNNs, which have to go step by step.
• Lets each token decide for itself which other tokens actually matter to its meaning.

Multi-Head Attention

One attention mechanism catches one kind of relationship. Transformers run several in parallel, called heads, each learning something different from the same input.

The input gets projected into multiple separate sets of queries, keys, and values, one set per head. Each head does its own self-attention calculation independently, and the outputs get combined at the end.

Why bother with multiple heads: different heads specialize. In the animal sentence, one head might pick up on grammatical structure, another on semantic meaning, another on positional relationships between words. Combined, that’s a lot richer than any single head could capture alone. It’s the model looking at the same sequence from several angles at once, in parallel, in a single pass.

Positional Encoding

Self-attention has a blind spot: it processes every token at once, which means it has no built-in sense of word order. That’s a real problem, since word order changes meaning:

“The cat chased the dog”

vs.

“The dog chased the cat”

Same words, completely different meaning. Without some way to encode position, a Transformer literally can’t tell these two sentences apart.

Positional encoding fixes this by adding position information directly into each token’s embedding. Two signals now travel together:

• Token embeddings: what the word means.
• Positional encodings: where the word sits in the sequence.

Transformers rely on both together, not either one alone.

The Transformer Architecture

A Transformer is built entirely around attention, structured into two main pieces.

Encoder

Processes the input sequence using self-attention and feed-forward networks, and produces contextual representations of every input token, essentially a rich understanding of the input.

Decoder

Generates the output sequence one token at a time, using three components: masked self-attention, encoder-decoder attention, and feed-forward networks. It attends both to tokens it’s already generated and to the encoder’s output.

The short version: encoder understands the input, decoder generates the output.

Why this architecture scales so well:

• Every token processes in parallel, not sequentially.
• Long-range dependencies get captured directly, without degrading over distance.
• Scales well to very large datasets and very large models, which is a big part of why it underlies modern LLMs.

Common applications: machine translation, text generation, question answering, summarization, and large language models generally.

Masked Self-Attention

During generation, a model absolutely cannot be allowed to see future tokens. If it could, it wouldn’t be learning to predict anything, it would just be copying the answer.

Take:

“The cat sat on the ___”

Predicting “mat,” the model should only ever see “The cat sat on the.” Not “mat” itself.

How masking works: future positions get assigned very large negative values in the attention scores, before softmax runs. After softmax, those positions end up with attention weights close to zero, effectively invisible. Each token ends up attending only to itself and whatever came before it.

Why this matters: it keeps training consistent with how generation actually works at inference time, one token at a time, using only past context. This is what makes autoregressive generation possible in the first place. Information simply can’t flow backward in time through the model.

Encoder-Decoder Attention

Beyond self-attention, the decoder also runs encoder-decoder attention, which is how it stays connected to the original input while generating output.

The roles here split cleanly:

• Queries come from the decoder.
• Keys come from the encoder.
• Values come from the encoder.

The decoder compares its queries against the encoder’s keys to figure out which parts of the input are most relevant right now.

Example, translation:

Input: “Bonjour” Output: “Hello”

While generating “Hello,” the decoder attends back to the encoder’s representation of “Bonjour.” That’s the mechanism making an accurate translation possible.

Put simply: self-attention handles the decoder’s own previously generated tokens, encoder-decoder attention handles the original input. The encoder builds representations, the decoder queries them.

Feed-Forward Networks in Transformers

Attention isn’t the whole story. After the attention layer, every token individually passes through a feed-forward network, the same network applied independently to each token.

Structurally, it’s simple: a linear layer, an activation function, another linear layer.

The division of labor is clean:

Attention decides which information matters. The feed-forward network decides how that information gets transformed.

Without feed-forward layers, the model could combine information from other tokens but couldn’t do much to actually transform it. Feed-forward networks are where a lot of the model’s real learning capacity lives.

Residual Connections

Deeper networks are harder to train. Information and gradients can degrade badly as they pass through many stacked layers.

Residual connections are the fix: instead of only passing the transformed output forward, you add the original input back in too.

Output = Layer(Input) + Input

What this buys you:

• Easier information flow through the network.
• Easier gradient propagation during backpropagation.
• Less vanishing gradient trouble.
• The ability to actually train Transformers with many stacked layers.
• More stable training overall.
• Better preservation of information from earlier layers.

The framing that makes this click: instead of forcing each layer to learn a totally new representation from scratch, residual connections let each layer just learn a modification to what it already received. That’s a much easier optimization problem.

Layer Normalization

As activations pass through a deep network, their distribution can shift layer to layer, making training unstable. Layer normalization keeps this in check by normalizing activations within each individual sample.

For a given token representation:

1. Calculate the mean.
2. Calculate the standard deviation.
3. Normalize using those values.
4. Scale and shift the result using learnable parameters.

What it gets you: more stable training, better gradient flow, less sensitivity to how you initialized your parameters, better convergence, and it holds up well even in very deep models.

In practice, Transformers apply layer normalization after residual connections, which keeps activations in a consistent range as they move through the network.

Putting a Full Encoder Layer Together

A Transformer encoder layer combines four components: multi-head attention, residual connections, layer normalization, and a feed-forward network.

The flow through one layer:

1. Multi-head attention: every token gathers information from other relevant tokens.
2. Residual connection: the original input gets added back to the attention output.
3. Layer normalization: the result gets normalized for stable training.
4. Feed-forward network: each token gets independently transformed.
5. Residual connection and layer normalization, again: the feed-forward output combines with its input and gets normalized once more.

Each component has a distinct job:

• Attention: lets tokens communicate.
• Feed-forward networks: transform each token’s representation.
• Residual connections: keep information and gradients flowing through the network.
• Layer normalization: keeps training stable.

One encoder layer, in short, gathers information via attention and refines it via feed-forward transformation. Stack a lot of these layers, and the model builds increasingly rich representations of the input sequence, layer by layer.

FAQ

Why is attention important in deep learning?

It lets a model focus computational weight on the most relevant parts of an input, instead of treating every part equally regardless of how relevant it actually is.

What are Query, Key, and Value in attention?

Query represents what a token is looking for, key represents what information a token holds, and value is the actual information passed forward once a token is selected as relevant.

What is self-attention?

A mechanism where every token attends to every other token in the same sequence, with queries, keys, and values all generated from that sequence.

Why does a Transformer need positional encoding?

Self-attention processes all tokens simultaneously and has no inherent sense of order. Positional encoding adds explicit information about each token’s position in the sequence.

What is multi-head attention?

Running multiple attention mechanisms in parallel, each learning to focus on different relationships or patterns within the same input.

What is masked self-attention?

A technique that blocks a token from attending to future tokens during generation, so the model learns to predict the next token using only past context.

What is encoder-decoder attention?

An attention mechanism where the decoder’s queries are compared against the encoder’s keys and values, letting the decoder pull in relevant information from the original input while generating output.

What are residual connections?

Connections that add a layer’s original input back to its transformed output, which helps information and gradients flow more easily through deep networks.

What is layer normalization?

A technique that normalizes activations within each individual sample, helping stabilize and speed up training in deep networks.

What are the main components of a Transformer encoder layer?

Multi-head attention, residual connections, layer normalization, and a feed-forward network, applied in sequence.