Skip to content
CLAY CLOUD DOCSMachine learningConceptsStatusDashboard

Machine learning in Clay Cloud / chapter 2 of 8

2. How a language model writes text

It predicts one next piece of text, appends it, and goes round again.

Tokens

A model does not read letters or words. A tokenizer first cuts the text into tokens: common pieces such as Hello, ,, my (note the leading space), or 'm. Each token is a whole number, its id, in a fixed vocabulary; GPT-2's has 50,257 entries. "The dog" is the two ids 464, 3290. Going back from ids to text is decoding.

The tokenizer is ordinary software, not part of the model's arithmetic, but it must match the model exactly: a different cutting gives different ids and nonsense. In our implementation it is a separate Tokenizer ability with two operations, encode: string => list<int> and decode: list<int> => string, answered by the host (docs/ml-m2-gpt2.md).

Embeddings: ids become vectors

The first step turns each token id into a vector of 768 numbers by looking up one row of a big learned table (the embedding table, 50,257 rows). A second table, indexed by position (0, 1, 2, ...), supplies a "where am I in the sentence" vector, and the two are added. In the Clay code this is two Tensor.gather calls and an add. From here on, text is only vectors.

Attention: each token looks back

The model then runs the vectors through 12 identical-shaped blocks. The interesting part of a block is attention. In plain words: for every token, the block asks "which of the tokens before me matter for what I should become?", gives each earlier token a score, turns the scores into percentages that add to 100 (that is what softmax does), and mixes the earlier tokens' information in those proportions. A token can only look backwards, never at the future; that is the causal mask.

The score comes from two derived vectors per token, a query ("what am I looking for?") and a key ("what do I offer?"), compared by multiplication. What gets mixed is a third, the value. All three are produced by matrix multiplies against learned weights. GPT-2 does this in 12 independent heads of 64 numbers each, so that different heads can look for different things. After attention, a small two-layer network (the MLP) processes each token separately, and the result is added back to the token's vector. A block is: attention, then MLP, each with a normalisation step (layer norm) to keep numbers in a sane range.

Figure 2.1. A toy attention table (illustrative; the percentages are invented, GPT-2's differ for every head and every sentence). Row = the token reading; columns = the earlier tokens it can look at. The empty upper right is the causal mask. Press Step to move to the next reading token.

Logits and choosing the next token

After the 12 blocks, the last token's vector is multiplied once more by the embedding table (turned on its side), producing 50,257 numbers, one score for every possible next token. These raw scores are logits. The simplest rule, greedy decoding, picks the largest (Tensor.argmax). Other rules sample randomly in proportion to the scores; we use greedy because the answer is then exactly checkable (chapter 6).

The loop

The model produces one token. To write more, append it to the text and run again. A reply of 100 tokens is 100 trips through all 12 blocks. In Clay this loop is plain code: decode in gpt2.clay calls step, takes the chosen token, and calls itself with that token as the new input.

Why a KV cache exists

Step through the figure. If each trip re-read the whole text so far, the work would grow with every token. But look at what attention needs from the earlier tokens: only their keys and values, and those never change once computed (earlier tokens cannot see later ones). So we keep them: the KV cache. Each new step computes keys and values only for the one new token, appends them, and reads the rest from the cache. The first step, the prompt pass, processes all the prompt tokens at once and fills the cache.

Figure 2.2. The generation loop with a growing KV cache. Text: the published GPT-2 greedy output for the prompt "Hello, my dog is a little" (docs/ml-m0-scope.md section 4; the prompt is 7 tokens, docs/ml-m2-gpt2.md p02). Where the token boundaries fall is not stated in our docs; this split is consistent with the stated total of 20 tokens. Cache size per position, 12 layers x 2 (keys and values) x 768 x 4 bytes = 73,728 bytes, is inferred from the shapes (docs/ml-m0-scope.md).

The cache is the price of that saving: memory that grows with the text. In our code the cache is a list of 12 records, each holding a keys tensor and a values tensor shaped [12, positions, 64] (gKv in gpt2.clay). The honest cost today: appending copies the whole cache each step (Tensor.concat copies), because Clay values are immutable and no last-use analysis exists yet to do it in place (docs/ml-workloads.md section 5, wishlist W17).

To remember. Text is ids; ids become vectors; each block lets tokens look back and then processes each one; the last vector becomes 50,257 scores; pick one; repeat. The KV cache turns "redo everything" into "do the one new token".