Skip to content
CLAY CLOUD DOCSMachine learningConceptsStatusDashboard

Machine learning in Clay Cloud / chapter 1 of 8

1. What is a model, really?

A pile of numbers, and a short recipe for combining them with your input. Nothing else.

Evidence labels used in this book: measured we ran it and wrote the number down; computed arithmetic done here on numbers the repository states; inferred reasoned, not measured; illustrative made up to draw a picture. Every figure names the repository document it came from.

Two things: weights and a recipe

A trained model, such as GPT-2, is two things you can keep in separate files.

Running a trained model on an input (inference) means following the recipe with the weights. That is the whole workload this book is about. In Clay terms: the recipe is an ordinary Clay program (it has a hash), and the weights are a file addressed by its own hash (chapter 5).

Weights124,439,808 numbers Recipeabout 200 lines of Clay runinference your input outputscores
Figure 1.1. A model is data plus a recipe. Sizes: 124,439,808 weights; recipe length about 200 lines of Clay (packages/tensor-host/clay/gpt2.clay).

Vectors, matrices, tensors

These three words only describe the shape of a block of numbers.

Inference is multiply-adds

Almost all of the arithmetic is one operation: the matrix multiply ("matmul"). To make one output number, take a row of the first matrix and a column of the second, multiply the pairs, and add the products up. A layer is "take my 768 numbers, produce 2,304 new numbers, each a weighted sum of all 768". That is 2,304 such row-meets-column sums. Step through a tiny one below.

Figure 1.2. Matrix multiply with toy numbers (illustrative). Each green cell is built from four multiply-adds. GPT-2's real multiplies are the same thing with rows of 768 numbers.

Nothing about this is clever, which is the point: it is a very large number of simple multiply-adds, and the speed of a model is how fast a machine can do them and, as chapter 3 shows, how fast it can fetch the weights to do them with.

What "GPT-2 small, 124M parameters, 548 MB" means

GPT-2 small is a language model published by OpenAI (Hub repository openai-community/gpt2, MIT licence, revision 607a30d7... pinned). We use it as the first real workload because it is small and simple, and because transformers' own test suite publishes exact expected outputs for it (chapter 6).

where the 124,439,808 numbers live (bar length proportional) word-embedding table: 38,597,376 (31%) 12 blocks: 84,934,656 (68%) position table: 786,432 (0.6%) norms and biases: 121,344 (0.1%)
Figure 1.3. Parameter counts computed from the checkpoint shapes in docs/ml-m2-gpt2.md: embeddings 50,257 x 768; positions 1,024 x 768; per block 768 x 2,304 + 768 x 768 + 768 x 3,072 + 3,072 x 768 = 7,077,888, times 12; the rest is norms and biases (121,344, the remainder). Percentages rounded.

What one token costs

To generate one word-piece, the recipe pushes one vector of 768 numbers through every weight matrix once. So the arithmetic per token is about one multiply-add per weight: 124 million multiply-adds, 0.249 GFLOP (a FLOP is one add or one multiply, so a multiply-add is two) measured by counting. Of that, about 85 million are in the 12 blocks and 38.6 million in the final step that turns the vector into a score for each of the 50,257 possible tokens computed (the same split as Figure 1.3's matrices, docs/ml-m0-scope.md).

To remember. The weights are fixed data; the recipe is a program; running is multiply-adds; and for every token generated, every weight is used once. That last fact decides almost everything in chapter 3.