How AI works: A light introduction to Transformers

A visual introduction to Transformers, the design behind many modern language models. Follow how text becomes small pieces and numbers, how the model combines information from them, and how an answer grows one piece at a time. The guide also explains training, memory, speed, and why fluent answers can still be wrong. No AI background required.

You paste an incident report into an AI assistant. Seconds later, it gives you a tidy summary. What happened between the paste and the prose? Let’s follow the data—from text to small pieces called tokens, through attention—a calculation that mixes information from those pieces—and back to an answer.

This guide is for people who build with AI, evaluate it, or have to explain its behavior to someone else. No machine-learning degree required. We will use one small example, four diagrams, and a little optional math. The math can wait its turn.

A few names first. An AI assistant is the application you use. A language model is the mathematical system inside it that processes and generates language using learned numerical settings. Training adjusts those settings from examples. A Transformer is one design for arranging the model’s calculations; that arrangement is its architecture.

Our scope: a typical decoder-only Transformer. To unpack that name, start with the original 2017 translation design in Attention Is All You Need. Its encoder processes an input sentence into numerical representations that capture information from across it. Its decoder uses those representations and the translation generated so far to predict the next piece of the translation. For French-to-English translation, the encoder processes the French and the decoder builds the English.

Decoder-only means there is no separate encoder component. The model processes the supplied text and generates its continuation through the same stack of processing stages, called layers. It still processes the input; “only” does not mean it skips that work. AI is a much broader field, and this guide does not claim to describe the undisclosed internals of any particular assistant.

Reading the map — represent the text. A vector is a list of numbers; an embedding is a learned vector representing a token. Token IDs identify which vectors to look up. A position is a token’s place in the text.

Combine and transform. Causal self-attention combines information from the current and earlier positions in the same text. A feed-forward network transforms the numerical information at each position. Normalization rescales values; residual connections add new results to existing representations.

Score the next piece. Repeating these calculations produces scores for the vocabulary, the set of possible tokens. A forward pass is one trip through the calculations; parameters are the learned numerical settings it uses.

Read your way: The example · Tokens · Attention · Layers · Generation · Training · Speed & memory · Practical takeaways. The attention code is optional.

One incident. Two very different summaries.

Imagine giving the assistant this note. An API (application programming interface) lets software systems send requests and responses. Here, 503 means the service was unavailable; a deployment puts a new software version into service, and a rollback restores the previous version.

The checkout API returned 503 after the deployment.
Rolling back restored service.

Summarize the incident in one sentence.

A grounded summary might be: “The checkout API returned 503 after a deployment, and service recovered following a rollback.”

A less careful one might announce: “A broken database migration caused the outage.” A database migration changes a database’s structure or data. That is quite a promotion for a migration nobody mentioned.

Keep that difference in mind. We are going to explain how a model can produce a useful continuation. That mechanism does not, by itself, establish that every statement in the continuation is true. Both summaries here are invented examples, not results from a model test.

1. Text becomes tokens, then vectors

A tokenizer splits text into tokens and assigns their IDs. Its vocabulary is the set of pieces it can represent; a sequence is those pieces in order. A token can cover a word, part of a word, punctuation, or bytes—the small units used to encode text digitally. The exact split depends on the tokenizer. checkout, 503, and a space before a word do not come with universal boundaries.

For our diagrams, we shorten the note to The / API / failed / . and treat those four pieces as tokens. That is a teaching convention—not output captured from a real tokenizer. The full summarization request still contains the report and its instruction; this miniature sequence just makes the mechanics easier to see. Hugging Face’s tokenization walkthrough shows how one common family of tokenizers builds and applies its vocabulary.

Each ID selects a row from a learned embedding table: a collection of vectors, with one numerical list for each vocabulary entry. The ID acts like an index; its numeric value does not measure meaning. Token 900 is not nine times more meaningful than token 100.

The vector gives the network numerical features—values it can combine and transform. A vector’s width, or dimensionality, is its number of entries. Each entry holds one numerical component. In the embedding table, these values are learned rather than hand-labelled “database,” “outage,” or “Friday afternoon.” Later layers build contextual representations: vectors shaped by the tokens available at or before that position. Context is the information available for the current prediction. Google’s embedding guide.

Order matters too. “Deployment before failure” and “failure before deployment” contain the same words and tell different stories. The network therefore needs position information.

One approach, rotary position embeddings (RoPE), changes pairs of numbers in query and key vectors according to their positions. A query is a vector used to look for matches; a key is a vector it is compared with. Rotating their coordinates changes those comparisons in a way that reflects how far apart the positions are. The attention section explains those roles in detail. You do not need to implement the rotations to understand their job. RoFormer paper.

Why you care: token counts affect context budgets—the allowance for the text supplied and generated in a request—and often charges for calling a model through an API. Measure your actual text with the relevant tokenizer; “one token equals one word” is a surprisingly expensive approximation.

2. Attention lets token positions exchange information

A token vector starts as a local representation. Self-attention lets a position mix information from other permitted positions in the same sequence. In our incident note, the representation used to generate the summary can draw on the service name, the failure, and the rollback rather than handling each piece in isolation.

A matrix is a rectangular grid of numbers. A projection here multiplies a vector by a learned matrix: each output entry is a weighted combination of input entries, producing a new vector. Attention uses three such projections:

For one query, the model compares it with each allowed key using a dot product: multiply matching entries, then add them. It divides these scores by the square root of the number of entries in each key vector, keeping their magnitude more manageable. Softmax then converts them into nonnegative weights that sum to one. A weighted sum multiplies each value by its assigned share, then adds the results. That is scaled dot-product attention. Attention Is All You Need.

These attention weights are mixing amounts calculated for the current text. They differ from the model’s learned parameters, also commonly called weights, which training adjusts. The diagram’s causal mask hides later positions so each position can use only itself and earlier pieces.

Read the diagram row by row. At failed, our invented weights are 0.10 for The, 0.60 for API, and 0.30 for failed. The later period is unavailable. The weighted result is 0.10 × V(The) + 0.60 × V(API) + 0.30 × V(failed).

To make that tangible, suppose those value vectors were [1, 0], [0, 2], and [3, 1]. Take 10% of the first list, 60% of the second, and 30% of the third, then add matching entries. The mixture is [1.0, 1.5]. We made up the vectors too. Their purpose is to show the arithmetic, not to reveal what a real model thinks about APIs.

At failed, the period is the target—the next token to predict. Its information must remain hidden. That is the practical consequence of the causal mask, even though the whole sentence is available during training.

A head is one set of learned query, key, and value projections. Multi-head attention performs several such mixtures and combines their outputs. Different heads can capture different relationships. They are not guaranteed to divide into a neat grammar department, facts department, and emotional-support department. Also, an attention weight is not a truth score or a complete explanation of a decision; research on attention and explanation cautions against that leap.

# Teaching sketch: one head, one sequence.
import math
import torch

# Assume X and learned W_q, W_k, W_v are initialized.
# Q, K: [tokens, key_width]; V: [tokens, value_width]
Q, K, V = X @ W_q, X @ W_k, X @ W_v
scores = (Q @ K.T) / math.sqrt(Q.shape[-1])
future = torch.triu(
    torch.ones_like(scores, dtype=torch.bool), diagonal=1
)
scores = scores.masked_fill(future, float("-inf"))
weights = torch.softmax(scores, dim=-1)
mixed = weights @ V

3. Attention is one part of a repeated block

If attention were the entire model, the rest of the architecture diagram could go home early. It cannot.

A decoder block is a repeated group of processing steps. Alongside attention, it contains a feed-forward network, often called an MLP (multilayer perceptron): a series of learned calculations at each position. Its nonlinear functions can respond differently depending on the input values, rather than only applying a fixed weighted sum. This lets successive layers represent relationships that one linear transformation—a fixed multiplication and addition of inputs—cannot. Attention mixes information across positions; the MLP transforms the features at each position using that context. The model repeats these operations across many layers.

Residual connections add a processing step’s result to the representations already present instead of replacing them wholesale. A sublayer is one such component, such as attention or the feed-forward network. Normalization rescales groups of values to help keep their numerical magnitudes controlled. These are part of making deep networks train and operate effectively, not decorative plumbing.

Implementations differ. The LLaMA paper describes pre-normalization (normalizing before each sublayer), RMSNorm (root mean square normalization, which rescales a vector using the square root of the average of its squared entries), a gated feed-forward function (one learned set of values controls how much another contributes), and RoPE. The pipeline diagram here omits those ordering details; the original Transformer used a different normalization arrangement. The layers share an architectural pattern, but normally have their own learned parameters.

At the end, a projection maps the final representation to one score per vocabulary token. These scores are called logits. The representation is no longer just the embedding we looked up at the entrance: it has passed through repeated context mixing and feature transformations.

4. Scores become a token. Then we do it again.

These vocabulary scores are different from the attention weights inside the network: attention mixes context; the output scores rank possible next tokens.

The prompt is the supplied input and instructions. In our full summarization request, its final token position supplies the logits for the first token of the summary. A decoding rule, the selection procedure for output tokens, chooses a token, appends it to the sequence, and repeats until a stop condition is reached. That step-by-step dependence is what autoregressive generation means.

Softmax converts output logits into probabilities over vocabulary tokens. With greedy decoding, choose the highest-scoring token. With sampling, draw from a probability distribution: the list of possible tokens and their assigned probabilities. Temperature changes how concentrated those probabilities are. Top-p sampling keeps the smallest set of highest-probability tokens whose probabilities add up to at least a chosen threshold; with p = 0.9, that set covers at least 90% of the probability before selection is made within it. The retained probabilities are then rescaled to sum to one. generation guide.

Lower temperature is not a fact-checking setting. It changes the choice among candidate continuations; it does not add missing incident logs. A model can be consistently wrong in a very calm voice.

The incident summary emerges through successive token choices. A token may finish a word rather than a sentence. The model is not necessarily drafting the whole paragraph somewhere and revealing it letter by letter.

Why you care: the next-token objective tells you the form of prediction, not a complete account of the capabilities learned during training. It is reasonable to study the model’s reasoning behavior. It is a mistake to treat fluent wording as proof that the reasoning or its premises were correct.

5. Training changes the model. A prompt changes its context.

During pretraining, the initial broad learning stage, a causal language model—one that predicts later tokens from earlier ones—is shown token sequences and trained to predict their next tokens. The targets come from the sequences themselves. A loss measures how poorly the predicted distributions assign probability to those targets.

Backpropagation computes gradients: numbers describing how small changes to each parameter would affect the loss. An optimizer, the update algorithm, uses them to adjust the parameters. Repeat across many examples, and the model can learn patterns that improve its predictions.

The figure’s targets are the actual next tokens being predicted. Inference means running a trained model to produce an answer; its forward pass calculates predictions from the input using the current parameters.

Many positions in a training sequence can be evaluated in parallel because the target sequence is already available. The causal mask still blocks future tokens at each position. Training gets a worksheet with an answer key; generation has to produce the next answer before it can use it as context. next-token training walkthrough.

Predicting text well can benefit from learning regularities in grammar, code, facts, and relationships described in that text. The training signal is the next token; the learned representations can support richer behavior. The GPT-3 research documents one example of varied task performance from an autoregressive model. The objective itself does not guarantee any particular skill.

That is how the projection matrices, embeddings, and other parameters acquire useful patterns. Ordinary answering uses those parameters without running an optimizer update for each message. The conversation changes the temporary inputs and computed states, not the deployed weights.

Post-training then shapes behavior using methods such as supervised learning (learning from examples of desired responses), preference-based learning (using comparisons of better and worse responses), or reinforcement learning (adjusting behavior using reward feedback). The InstructGPT paper is one documented example. Following instructions, using tools, and producing a helpful chat response involve more than selecting a base architecture.

A service may log conversations or use eligible data in a later training process. That is a separate product-policy question. “The weights do not update during this response” does not mean “this service stores nothing.”

6. Why the first token and the next token cost different work

Prefill processes the prompt and produces the state needed to begin generation. Decode advances the continuation. In a standard causal attention model, earlier positions do not gain access to future tokens, so their already-computed keys and values can be reused.

A KV (key–value) cache saves calculations for reuse. It keeps those key and value activations—intermediate numerical results computed for this input—at each layer. As a new token is processed, it contributes new K/V entries and attends using the available cached entries. The cache saves recomputation; it is not a store of verified facts and it does not retrain the model. Hugging Face’s cache guide.

The figure calls this runtime state: temporary information kept while a request is being processed. The cache stores calculated values, not changes to the learned parameters.

With full attention, each position can compare against every position the causal rule allows. The number of position pairs grows quadratically: doubling a long prompt gives roughly four times as many pairs. That does not mean total response time must increase exactly fourfold.

FlashAttention is an implementation that makes exact attention—the same attention calculation rather than an approximation—more efficient by reducing expensive data movement. It runs on a GPU (graphics processing unit), a processor that performs many numerical operations in parallel; it does not make the conceptual all-pairs interaction vanish.

Sparse attention uses a selected subset of position pairs; sliding-window attention limits comparisons to nearby tokens; hybrid designs combine different kinds of processing layers. These designs can change the cost.

Practical consequence: long prompts, long answers, and concurrent requests (requests running at once) stress different parts of serving, the system that runs models for users. Time to first token measures the wait until output starts; output tokens per second measures the generation rate. They answer different questions. The KV cache also consumes memory, so “the model weights fit on the GPU” does not mean every workload will fit.

For a concrete serving experiment, our single-GPU Qwen performance article separates the workload and measurements. Its results describe that setup; they are not universal Transformer speeds.

7. What this changes in your work

Return to the incident note. The useful task is to preserve what the report supports. A claim about a database migration needs evidence beyond the words “after the deployment.” No amount of softmax can supply a log entry that was never provided.

Here, an agent means an application that uses a model in a loop to select actions, receive results, and continue a task. A tool is an operation that the surrounding software can execute, such as searching documents or running code.

Retrieval-augmented generation (RAG) looks up relevant material, such as documents, and supplies it to the model as evidence for its response; the original RAG paper studies combining retrieval with a generator. Retrieval can provide missing evidence, but selecting the right evidence and using it correctly remain jobs to evaluate.

Likewise, a large context window—the maximum number of tokens available together to the model—is a capacity limit, not a promise of perfect recall. Lost in the Middle found position-sensitive performance in the models and tasks it tested. That is a reason to test your own workload, not to assume every current model has the same curve.

A chat application may add instructions, retrieved documents, tools, persistent storage, and several model calls. Those surrounding systems explain much of the difference between a model demonstration and a useful product. Keep their responsibilities visible.

The version worth keeping in your head

• Tokens are the input units. Embeddings turn their IDs into vectors, and position handling supplies order information.
• Attention mixes information across allowed positions. Feed-forward networks transform it; repeated blocks build contextual representations.
• The model produces next-token scores. A decoding rule turns them into a continuation.
• Training updates learned parameters. Ordinary inference uses them; context and caches are different kinds of state.
• A useful answer still needs evidence and evaluation. Architecture explains the machinery, not the truth of every sentence it produces.

You now have enough of the mechanism to ask better questions. What text actually reached the model? Which state persists? What work is being cached? What did we test? And did the database migration really exist, or did the summary hire one for the occasion?

Sources and a route deeper

The sources are linked beside the relevant explanations. Start with Attention Is All You Need for the architecture, Hugging Face’s tokenization walkthrough for text processing, and the next-token training walkthrough for the learning objective. The LLaMA paper shows how a later decoder model differs from the original design. PyTorch’s attention reference and Hugging Face’s cache guide connect the concepts to implementation.

About this guide: Co-written and co-edited by Joseph Gabito and Astra (AI). The diagrams and incident examples were created for this article. Token boundaries, attention weights, and tiny vectors used for teaching are explicitly illustrative. This is a conceptual guide, not a benchmark or a disclosure of proprietary model internals.

For a contrasting approach, see Jev by TypeSafe AI, a model designed for structured decisions rather than text generation.