Transformer Architecture

Concepts Covered

• Transformer architecture
• Token representations
• Attention layers
• Feed-forward layers
• Repeated blocks
• Residual connections
• Encoder-decoder and decoder-only shapes
• Parallel training over tokens
• Autoregressive generation

Definition

A transformer is a neural-network architecture that processes token representations through repeated blocks built around attention and learned transformations.

For language models, keep this first mental picture:

tokens
  -> token representations
  -> repeated transformer blocks
  -> scores for possible next tokens

The block is the important unit. It lets the model mix information across tokens, transform that information, and repeat the process many times.

Why This Concept Exists

Language is not only a bag of words.

The meaning of a token can depend on tokens before it, after it, far away from it, or serving a different role in the sentence.

Older sequence models often carried context forward step by step. Transformers made attention the central way tokens can read from other token positions, which made large-scale training and modern language-model architectures practical.

You do not need every equation first. You need to know what kind of machine people mean when they say:

LLMs are built from transformers

The Beginner Block Diagram

A simplified transformer block has two jobs:

1. let token positions exchange relevant information
2. transform each position with learned neural-network computation

That becomes:

token representations
  -> attention
  -> feed-forward transformation
  -> updated token representations

Real blocks add details that matter for stable training and scale, including normalization, residual connections, and multiple attention heads.

The architecture repeats these blocks. Early blocks may build local or surface-level signals. Later blocks can build richer task-relevant representations from the context available to them.

What Enters The Transformer

The model does not receive raw words directly.

Text is tokenized. Tokens become learned vector representations. The model also needs position information so it can distinguish:

dog bites person
person bites dog

Those representations are what transformer blocks update.

This is why tokens, embeddings, positional information, attention, and next-token prediction keep appearing together in LLM discussions.

Encoder, Decoder, And Decoder-Only

The original transformer architecture described an encoder and a decoder.

At a high level:

• an encoder builds representations from an input sequence
• a decoder produces an output sequence while using allowed context

Modern generative LLM conversations often focus on decoder-only transformers.

A decoder-only language model reads the current context and predicts the next token repeatedly:

context -> next token
context + next token -> next token

That generation loop is sequential at inference time even though transformer training can process many token positions in parallel under the training setup.

Where Attention Fits

Attention is the information-routing part of the block.

For a given token position, attention helps answer:

Which other token positions matter for updating this representation right now?

Without that idea, "transformer architecture" sounds like a brand name. With it, the model shape becomes easier to inspect:

• attention mixes context
• feed-forward layers transform representations
• repeated blocks refine them

Where The Training Objective Fits

Architecture and objective are different layers.

The architecture defines the computation the model can perform.

The training objective supplies the signal that pushes parameters toward useful behavior. For a next-token language model, training rewards assigning better scores to the target continuation under the context.

That is why the training mechanics page comes before this page. Loss and optimization explain how a transformer becomes a trained model instead of a stack of untrained operations.

Common Confusions

Transformer does not mean chatbot.

Transformers are an architecture family. A product can add instruction tuning, retrieval, tools, safety controls, memory-like product features, and user experience around a model.

Attention is not the whole transformer.

Attention is central, but feed-forward layers, parameters, normalization, residual paths, token representations, and the training setup also matter.

Generation is not one giant answer lookup.

For autoregressive language models, output is produced token by token from the current context and the model state created by the forward computation.

Related Topics

• What Is A Large Language Model?
• Tokens And Tokenization
• Loss, Optimization, And Gradient Descent
• Attention