How Large Language Models Actually Work: From Tokenization to Autoregressive Generation

A large language model is, at the level of its interface, a function that takes a sequence of integers and returns a probability distribution over the next integer. Its apparent reasoning, stylistic range and factual recall all come from what that function learned during training, and from the loop that calls it repeatedly at inference time.

Keeping that interface in mind explains several common behaviours. Why models struggle to count letters. Why they cost more to run per token as the conversation grows. Why the same prompt gives different answers. Why training is parallel but generation is sequential. This article follows a single token through a modern decoder-only transformer (the architecture family behind GPT, Llama, Mistral, Qwen and most open models) and then looks at how the model is trained and how text is produced from it.

1. Tokenization: from text to integers

Neural networks operate on vectors, so text must first become a sequence of discrete symbols from a fixed vocabulary. Characters make sequences too long, and words make the vocabulary unbounded and fail on unseen words. Modern LLMs use subword tokenization, almost always some form of byte-pair encoding (BPE).

BPE training is a greedy compression algorithm:

1. Start with a base vocabulary of all 256 byte values. (Byte-level BPE, used since GPT-2, can therefore encode any string, so there is no “unknown token”.)
2. Count every adjacent pair of symbols in the training corpus.
3. Merge the most frequent pair into a new symbol, and record the merge.
4. Repeat until the vocabulary reaches its target size.

Encoding new text replays the learned merges in order. Frequent words become single tokens ( the, model), rarer words split into pieces ( token, ization), and rare strings fall back toward bytes.

Vocabulary size is a real design trade-off. GPT-2 used about 50K tokens. Llama 2 used 32K. Llama 3 moved to about 128K. A larger vocabulary means fewer tokens per document, so shorter sequences, cheaper attention and more text per context window, especially for code and non-English languages. The cost is a bigger embedding table and output layer, and rarer tokens that each get fewer training updates.

Tokenization explains several well-known quirks:

• Character-level tasks are hard. The model never sees the letters in strawberry, only one or a few token IDs. Counting the r’s requires the model to have memorized the spelling of those tokens.
• Arithmetic depends on how numbers split. If 12345 becomes 123 + 45, digit alignment for addition is not visible in the input. Some tokenizers split numbers into single digits or fixed groups of three for this reason.
• Whitespace and casing matter. Hello, Hello and hello are usually different tokens with different learned embeddings.
• Cost and context are measured in tokens, and the same text can cost very different numbers of tokens in different languages.

2. Token embeddings

Each token ID $t$ indexes a row of a learned embedding matrix $E \in \mathbb{R}^{|V| \times d}$, where $d$ is the model width (4,096 in Llama 2 7B). The row $E_t$ is the token’s initial vector.

These vectors are not meaningful in isolation at first. They are trained jointly with everything else, and they end up encoding whatever helps predict the next token: syntactic role, semantic neighbourhood, common continuations. At this point though, the vector for bank is the same whether the sentence is about rivers or money. Turning that static vector into a contextual one is the job of the transformer layers.

Many models tie the input embedding and the output projection (using $E^\top$ to produce logits at the end), which saves $|V| \cdot d$ parameters. That is significant: with a 128K vocabulary and $d = 4096$, one matrix is about 0.5B parameters.

3. Position: why attention needs it, and how RoPE provides it

Self-attention, described below, is permutation-equivariant: it treats its input as a set. Without extra information, “dog bites man” and “man bites dog” would produce the same set of representations. Position must be injected.

The original transformer added fixed sinusoidal vectors to the embeddings. GPT-2 added learned position vectors. Most modern LLMs instead use rotary position embeddings (RoPE) (Su et al., 2021), which encode position by rotating the query and key vectors inside attention.

Split a $d_h$-dimensional query or key into $d_h/2$ pairs of coordinates. For position $m$, rotate pair $i$ by the angle $m\theta_i$, where

$$\theta_i = b^{-2i/d_h}, \qquad b = 10{,}000 \text{ (typically)}.$$

For one pair, the rotation is

$$R_{m,i} = \begin{pmatrix} \cos m\theta_i & -\sin m\theta_i \\ \sin m\theta_i & \cos m\theta_i \end{pmatrix}.$$

The key property comes from the fact that rotations compose: $R_m^\top R_n = R_{n-m}$. So the attention score between a query at position $m$ and a key at position $n$ is

$$(R_m \mathbf{q})^\top (R_n \mathbf{k}) = \mathbf{q}^\top R_m^\top R_n \mathbf{k} = \mathbf{q}^\top R_{n-m}\, \mathbf{k}.$$

The score depends on the content of $\mathbf{q}$ and $\mathbf{k}$ and on their relative offset $n - m$, never on absolute position. Each pair rotates at a different frequency: early pairs rotate quickly and resolve fine local order, and late pairs rotate slowly and carry long-range position. Because position lives inside the dot product rather than being added to the content vector, it does not consume part of the representation, and it extends more gracefully to longer sequences.

That extension is not free, though. A model trained on 4K-token sequences has never seen rotation angles beyond that range. Context-extension methods such as position interpolation, NTK-aware scaling and YaRN rescale the frequencies so that longer sequences map into angles the model knows, usually followed by a short fine-tuning run on long documents.

4. The transformer block

A decoder-only LLM is a stack of $L$ identical blocks (32 in Llama 2 7B, 80 in Llama 2 70B). Each block reads and writes a shared vector per position, the residual stream. In the pre-normalization layout that modern models use, a block is

$$\begin{aligned} \mathbf{x}' &= \mathbf{x} + \text{Attention}(\text{Norm}(\mathbf{x})) \\ \mathbf{x}'' &= \mathbf{x}' + \text{FFN}(\text{Norm}(\mathbf{x}')) \end{aligned}$$

A useful mental model is that attention moves information between positions, and the feed-forward network transforms information within a position. The residual stream is the shared workspace that both read from and add to.

Figure 1. A pre-norm decoder-only transformer. Each sublayer reads a normalized copy of the residual stream and adds its output back to it.

4.1 Self-attention, from the equation up

For each position, the normalized input $\mathbf{x}_i \in \mathbb{R}^d$ is projected into three vectors:

$$\mathbf{q}_i = W_Q \mathbf{x}_i, \qquad \mathbf{k}_i = W_K \mathbf{x}_i, \qquad \mathbf{v}_i = W_V \mathbf{x}_i.$$

The intuition is a soft dictionary lookup. The query says what this position is looking for. The key says what each position offers to be found by. The value is what each position hands over if it is selected. Stacking all positions into matrices $Q, K, V \in \mathbb{R}^{n \times d_h}$:

$$\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_h}} + M\right) V.$$

Reading the equation piece by piece:

• $QK^\top$ is an $n \times n$ matrix of scores. Entry $(i, j)$ is how well position $i$‘s query matches position $j$‘s key.
• Dividing by $\sqrt{d_h}$ keeps the variance of the scores near 1. Without it, for random vectors the dot product’s variance grows with $d_h$, the softmax saturates toward one-hot, and gradients vanish.
• $M$ is the causal mask: $M_{ij} = 0$ if $j \le i$ and $-\infty$ otherwise. A position can attend only to itself and earlier positions, which is what lets the model be trained to predict the future without seeing it.
• The softmax turns each row into weights that sum to 1.
• Multiplying by $V$ makes each output a weighted average of the value vectors of the positions it attends to.

import torch, math

def causal_self_attention(x, W_q, W_k, W_v, W_o, n_heads):
    B, n, d = x.shape
    d_h = d // n_heads
    # project and split into heads: (B, heads, n, d_h)
    q = (x @ W_q).view(B, n, n_heads, d_h).transpose(1, 2)
    k = (x @ W_k).view(B, n, n_heads, d_h).transpose(1, 2)
    v = (x @ W_v).view(B, n, n_heads, d_h).transpose(1, 2)
    # (RoPE would rotate q and k here)
    scores = q @ k.transpose(-2, -1) / math.sqrt(d_h)          # (B, heads, n, n)
    mask = torch.triu(torch.ones(n, n, dtype=torch.bool), diagonal=1)
    scores = scores.masked_fill(mask, float("-inf"))
    out = torch.softmax(scores, dim=-1) @ v                      # (B, heads, n, d_h)
    out = out.transpose(1, 2).reshape(B, n, d)                   # concatenate heads
    return out @ W_o

4.2 Multi-head attention

One attention operation produces one weighting per position, so it can only “look” in one way at a time. Multi-head attention runs $h$ attention operations in parallel on lower-dimensional projections ($d_h = d/h$, for example 32 heads of 128 dimensions) and concatenates the results before a final output projection $W_O$:

$$\text{MHA}(\mathbf{x}) = W_O\,[\text{head}_1; \dots; \text{head}_h].$$

The cost is about the same as a single full-width head, but the model can attend to different things at once: one head might track the previous token, another the subject of the sentence, another a matching bracket. Interpretability research has identified such specialized heads, including induction heads, which find an earlier occurrence of the current token and copy what followed it. They are thought to be an important mechanism behind in-context learning.

Many recent models share keys and values across groups of query heads (grouped-query attention). That change matters mostly for inference memory, and is covered in my article on the KV cache.

4.3 The feed-forward network

After attention mixes information across positions, each position passes independently through a two-layer MLP. The original form was $\text{FFN}(\mathbf{x}) = W_2\,\sigma(W_1 \mathbf{x})$ with a hidden width of $4d$. Most current LLMs use a gated variant, SwiGLU:

$$\text{FFN}(\mathbf{x}) = W_2\big(\text{SiLU}(W_1 \mathbf{x}) \odot W_3 \mathbf{x}\big),$$

where one projection produces a gate that multiplies another elementwise. With three matrices instead of two, the hidden width is usually reduced to about $\tfrac{8}{3}d$ to keep the parameter count similar (11,008 for $d = 4096$ in Llama 2 7B).

The FFN holds most of the parameters, about two-thirds of each block, and is where much of the model’s factual knowledge appears to be stored. One influential view treats each FFN as a key-value memory: rows of $W_1$ detect patterns in the residual stream, and the matching columns of $W_2$ write associated information back into it.

4.4 Residual connections and normalization

Each sub-layer adds its output to the residual stream instead of replacing it. This gives gradients a direct path from the loss to every layer, which is what makes stacks of 80+ layers trainable, and it lets each layer make small, incremental edits to a shared representation.

Normalization keeps the scale of activations stable. Modern LLMs normalize the input of each sub-layer (pre-norm) rather than the output (post-norm, as in the original transformer). Pre-norm trains much more stably at depth, because the residual path itself is never normalized. Most also replace LayerNorm with RMSNorm, which drops the mean-centering:

$$\text{RMSNorm}(\mathbf{x}) = \frac{\mathbf{x}}{\sqrt{\frac{1}{d}\sum_{j} x_j^2 + \epsilon}} \odot \boldsymbol{\gamma}.$$

It is slightly cheaper and works as well in practice.

4.5 From the last layer to a distribution

After the final block and a final normalization, the vector at each position is projected to the vocabulary to give logits $\mathbf{z} \in \mathbb{R}^{|V|}$, and a softmax turns them into probabilities:

$$p(x_{t+1} = v \mid x_{\le t}) = \frac{\exp(z_v)}{\sum_{u \in V} \exp(z_u)}.$$

4.6 Where the parameters are

For a block with width $d$, attention has four $d \times d$ matrices ($4d^2$ parameters, with full multi-head attention), and a SwiGLU FFN with hidden width $d_{ff}$ has $3\,d\,d_{ff}$. For Llama 2 7B ($d = 4096$, $d_{ff} = 11008$, $L = 32$, $|V| = 32000$, untied embeddings):

$$\underbrace{32 \times (4 \cdot 4096^2 + 3 \cdot 4096 \cdot 11008)}_{\approx\, 6.48\text{B}} + \underbrace{2 \cdot 32000 \cdot 4096}_{\approx\, 0.26\text{B}} \approx 6.74\text{B},$$

which matches the published size. Normalization weights are a rounding error.

5. Pretraining: next-token prediction

The model is trained on trillions of tokens of text with one objective: predict each token from the ones before it. For a sequence $x_1, \dots, x_T$, the loss is the average negative log-likelihood (cross-entropy):

$$\mathcal{L}(\theta) = -\frac{1}{T}\sum_{t=1}^{T} \log p_\theta(x_t \mid x_{<t}).$$

Its exponential, $\exp(\mathcal{L})$, is the perplexity: roughly, the number of equally likely tokens the model is choosing between on average.

The objective is simple, but minimizing it on diverse text is extremely demanding. Predicting the next token of a proof requires following the proof. Predicting the next line of code requires tracking variable types. Predicting dialogue requires modelling what the characters know. Grammar, facts, style, and some forms of reasoning are all useful for compression, so they are learned as a side effect.

Training is parallel. Thanks to the causal mask, a single forward pass over a sequence of $T$ tokens produces predictions for all $T$ positions at once, each conditioned only on its own past. The model is fed the true previous tokens at every position (this is called teacher forcing), so there is no sequential dependency during training. That is why transformers train so efficiently on GPUs, and also why there is a mismatch at inference time, discussed below.

6. Autoregressive generation

At inference time there is no ground truth to feed in. Generation is a loop:

def generate(model, prompt_ids, max_new_tokens, sample_fn):
    ids = list(prompt_ids)
    for _ in range(max_new_tokens):
        logits = model(ids)[-1]          # distribution for the next position only
        next_id = sample_fn(logits)
        ids.append(next_id)
        if next_id == EOS_ID:
            break
    return ids

Each new token depends on all previous tokens, including the ones the model just generated, so generation is inherently sequential: producing 500 tokens needs 500 forward passes. Written naively like this, each pass also recomputes the keys and values of every earlier token. In practice those are cached (the KV cache), so each step only processes the newest token, but the step still has to read all of the model’s weights and the whole cache from memory.

Because the model’s own outputs become its inputs, errors compound. A model trained only on correct prefixes (teacher forcing) is then run on prefixes containing its own mistakes. This exposure bias is one reason long generations can drift, and one motivation for post-training methods that train on the model’s own samples.

7. Sampling: turning a distribution into text

The model produces a distribution, and a decoding strategy chooses from it.

Greedy decoding always picks the most likely token. It is deterministic, and good for short, factual or structured outputs. For open-ended text it tends to loop and repeat, because a repeated phrase often becomes the most likely continuation of itself.

Temperature rescales the logits before the softmax:

$$p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}.$$

$T < 1$ sharpens the distribution toward the top tokens, $T > 1$ flattens it, $T \to 0$ approaches greedy, and $T = 1$ is the model’s own distribution. Temperature changes how often low-probability tokens are picked, not which tokens are possible.

Top-k sampling keeps only the $k$ most likely tokens, renormalizes, and samples. Its weakness is that a fixed $k$ suits neither case well. After “The capital of France is”, almost all probability is on one token, and $k = 50$ admits 49 bad options. In an open creative context, $k = 50$ may cut off many reasonable ones.

Top-p (nucleus) sampling (Holtzman et al., 2019) adapts instead. It keeps the smallest set of tokens whose cumulative probability reaches $p$:

$$V_p = \arg\min_{V' \subseteq V} |V'| \quad \text{subject to} \quad \sum_{v \in V'} p(v) \ge p.$$

When the model is confident the set is tiny, and when it is uncertain the set grows.

Figure 2. Temperature reshapes the distribution and top-p adapts to it. At low temperature one token holds 90% of the mass; at high temperature the nucleus grows to several tokens.

def sample(logits, temperature=0.8, top_k=None, top_p=0.95):
    if temperature == 0:
        return int(torch.argmax(logits))
    logits = logits / temperature
    if top_k is not None:
        kth = torch.topk(logits, top_k).values[-1]
        logits = logits.masked_fill(logits < kth, float("-inf"))
    probs = torch.softmax(logits, dim=-1)
    if top_p is not None:
        sorted_p, idx = torch.sort(probs, descending=True)
        cumulative = torch.cumsum(sorted_p, dim=-1)
        drop = cumulative - sorted_p > top_p        # keep tokens until mass reaches p
        sorted_p[drop] = 0.0
        probs = torch.zeros_like(probs).scatter(0, idx, sorted_p)
        probs = probs / probs.sum()
    return int(torch.multinomial(probs, 1))

Other strategies are common too. Min-p keeps tokens whose probability is at least a fraction of the top token’s probability. Repetition and frequency penalties down-weight tokens that already appeared. Beam search keeps several candidate sequences, which helps translation but tends to produce bland text in open-ended generation. Constrained decoding masks out tokens that would violate a grammar or JSON schema, which guarantees valid structured output.

A common misconception is that temperature 0 makes a model “accurate”. It makes it deterministic (up to floating-point and batching effects). The most likely token at each step is not the same as the most likely, or most correct, whole answer.

8. Training versus inference: two different workloads

The same network behaves very differently as a computational workload in training and in inference.

	Training	Inference: prefill	Inference: decode
Tokens processed per pass	Whole sequences, whole batch	Entire prompt at once	One new token per sequence
Parallel over positions	Yes	Yes	No
Bottleneck	Compute (matrix multiplies)	Compute	Memory bandwidth
Extra state	Activations for backprop, optimizer state	Builds KV cache	Reads full weights and KV cache every step

A useful rule from Kaplan et al. (2020) is that a forward pass costs about $2N$ floating-point operations per token for a model with $N$ parameters (one multiply and one add per weight), and training costs about $6N$ per token, because the backward pass is roughly twice the forward pass. Total training compute is then approximately

$$C_{\text{train}} \approx 6\,N\,D,$$

where $D$ is the number of training tokens. For a 7B model trained on 2 trillion tokens, that is about $8.4 \times 10^{22}$ FLOPs.

Training memory is dominated not by weights but by optimizer state and activations. With mixed precision and Adam, each parameter typically needs about 16 bytes: 2 for BF16 weights, 2 for gradients, and 12 for FP32 master weights plus two Adam moments. A 7B model therefore needs on the order of 112 GB just for that state, before activations, which is why training is sharded across many GPUs (ZeRO, FSDP, tensor and pipeline parallelism).

Inference decode is limited by memory bandwidth. To produce one token, every weight must be read from GPU memory. A 7B model in 16-bit precision is about 14 GB. On a GPU with about 2 TB/s of bandwidth, that alone takes about 7 ms, setting a floor of roughly 140 tokens per second for a single sequence, however fast the arithmetic units are. Batching many requests together reuses each weight read for many tokens, which is the core of efficient LLM serving. Quantizing weights to 8 or 4 bits speeds up decoding mainly because it reduces the bytes that must be read.

9. Complexity and scaling

For a sequence of length $n$, one layer costs roughly

$$\underbrace{O(n\,d^2)}_{\text{projections and FFN}} + \underbrace{O(n^2 d)}_{\text{attention scores and weighted sum}}.$$

For short sequences the $n\,d^2$ term dominates. For long sequences the quadratic attention term takes over, both in compute and, in a naive implementation, in memory, since the $n \times n$ score matrix is materialized per head. FlashAttention (Dao et al., 2022) computes exact attention in tiles that fit in on-chip SRAM, using an online softmax, and never writes the full matrix to GPU memory. It does not reduce FLOPs, but it cuts memory traffic so much that it is substantially faster, and it reduces attention’s memory use from quadratic to linear in $n$.

On the scaling side, Kaplan et al. found that loss falls as a smooth power law in parameters, data and compute. Hoffmann et al. (2022, “Chinchilla”) revisited the compute-optimal balance and concluded that parameters and tokens should grow together, at roughly 20 training tokens per parameter. Current practice deliberately trains past that point. Llama 3 8B was trained on more than 15 trillion tokens, nearly 100 tokens per parameter, because a smaller model trained longer is more expensive to train but cheaper to serve, and serving cost dominates over a model’s lifetime.

10. What this picture explains, and what it does not

This mechanical view accounts for a lot of observed behaviour:

• Hallucination follows naturally from the objective. The model is trained to produce a plausible continuation, and a fluent, wrong answer can be highly plausible. Nothing in next-token prediction alone separates “true” from “typical”.
• Prompt sensitivity follows from conditioning. Every token in the context shifts the distribution. Phrasing, examples and order change what is probable.
• Long conversations get slower and more expensive, because each generated token attends to, and reads the cached keys and values of, every earlier token.
• Models can be steered without retraining, because in-context examples change the conditional distribution. In-context learning is real, but it happens within a forward pass, and nothing is stored after the conversation ends.

Some things it leaves open. The base model produced by pretraining is a document completer, not an assistant. It will continue a question with more questions as readily as with an answer. The behaviour people associate with chat models comes from post-training (supervised fine-tuning and preference optimization), which is the subject of my article on alignment. And knowing the architecture does not mean we understand the algorithms the trained weights implement. Mechanistic interpretability has found circuits for specific behaviours, such as induction heads, but most of what a large model computes is not yet understood at that level.