Multi-Head Attention

Concepts Covered

• Multi-head attention
• Attention heads
• Learned projections
• Parallel context mixing
• Representation subspaces
• Combining head outputs
• Capacity and compute tradeoffs

Definition

Multi-head attention runs several attention computations in parallel over learned projections of the current representations, then combines their outputs.

The beginner version is:

one attention operation -> one learned way to mix context
multiple heads -> several learned ways to mix context in the same layer

The model does not ask a human to assign a grammar head, a reference head, and a code head. Training learns how the heads become useful.

Why This Concept Exists

One token position may need more than one kind of context signal at once.

In a sentence, a token might need:

• nearby phrase structure
• a far-away subject
• the object being referenced
• punctuation or formatting boundaries

Multi-head attention gives a transformer layer more room to form different attention patterns and value mixtures in parallel instead of squeezing every relationship through one attention view.

The Mechanical Shape

Each head gets learned projections for its attention computation.

A simplified layer flow is:

input representations
  -> head 1 attention output
  -> head 2 attention output
  -> head N attention output
  -> combine head outputs
  -> project back into the model representation

The heads are parallel parts of one layer. Their outputs are combined before later transformer computation continues.

Why Learned Projections Matter

Heads do not all look at an identical representation through identical parameters.

Learned projections create different query, key, and value views for the heads. That is why "multi-head" means more than repeating the exact same comparison several times.

The architecture gives capacity for different relationships. Training decides what becomes useful for the objective and data.

A Careful Mental Model

It is tempting to say:

head 1 handles grammar
head 2 handles facts
head 3 handles code

That can make the first picture intuitive, but it is too rigid.

A better mental model is:

different heads can learn different context-mixing patterns

Some patterns may become interpretable. Others are distributed across heads, layers, and feed-forward computation.

Capacity, Compute, And Design

Multi-head attention increases the structure available inside an attention layer.

Engineers care because architecture choices affect:

• representation capacity
• memory movement
• attention-kernel efficiency
• KV cache shape during inference
• how model width is divided across heads

Those details become sharper when we reach KV cache and attention optimizations. For now, remember that more architectural structure is not free. It must fit the model's quality, training, and serving budget.

Common Confusions

A head is not a separate model.

It is a component inside an attention layer.

More heads does not automatically mean better product behavior.

Model quality depends on data, objective, scale, training, architecture balance, evaluation, and product system design.

Multi-head attention is not a committee of human-readable specialists.

It is parallel learned computation over projected representations.

Related Topics

• Attention
• Masked Attention
• Transformer Architecture
• Parameters And Weights
• Loss, Optimization, And Gradient Descent