Machine learning in Clay Cloud / chapter 1 of 8
1. What is a model, really?
A pile of numbers, and a short recipe for combining them with your input. Nothing else.
Evidence labels used in this book: measured we ran it and wrote the number down; computed arithmetic done here on numbers the repository states; inferred reasoned, not measured; illustrative made up to draw a picture. Every figure names the repository document it came from.
Two things: weights and a recipe
A trained model, such as GPT-2, is two things you can keep in separate files.
- The weights (also called parameters): a very large list of ordinary decimal numbers. Training, which this book is not about, is the process that found good values for them. After training they never change; they are data, like a lookup table that took months of computing to produce.
- The recipe: the architecture. A short program that says "take the input, multiply it by this group of weights, add that group, squash it with this function, repeat twelve times". It is a few hundred lines. It contains no learned knowledge at all.
Running a trained model on an input (inference) means following the recipe with the weights. That is the whole workload this book is about. In Clay terms: the recipe is an ordinary Clay program (it has a hash), and the weights are a file addressed by its own hash (chapter 5).
packages/tensor-host/clay/gpt2.clay).Vectors, matrices, tensors
These three words only describe the shape of a block of numbers.
- A vector is a list:
[0.2, -1.5, 0.7]. GPT-2 represents each piece of text as a vector of 768 numbers. - A matrix is a table with rows and columns. A layer's weights are a matrix, for example 768 rows by 2,304 columns.
- A tensor is the general word: a block with any number of dimensions (a list, a table, a stack of tables). Libraries say "tensor" for all of them. Its shape is just the list of sizes, written
[768, 2304]. The ClayTensorability has the same notion; shapes arelist<int>there.
Inference is multiply-adds
Almost all of the arithmetic is one operation: the matrix multiply ("matmul"). To make one output number, take a row of the first matrix and a column of the second, multiply the pairs, and add the products up. A layer is "take my 768 numbers, produce 2,304 new numbers, each a weighted sum of all 768". That is 2,304 such row-meets-column sums. Step through a tiny one below.
Nothing about this is clever, which is the point: it is a very large number of simple multiply-adds, and the speed of a model is how fast a machine can do them and, as chapter 3 shows, how fast it can fetch the weights to do them with.
What "GPT-2 small, 124M parameters, 548 MB" means
GPT-2 small is a language model published by OpenAI (Hub repository openai-community/gpt2, MIT licence, revision 607a30d7... pinned). We use it as the first real workload because it is small and simple, and because transformers' own test suite publishes exact expected outputs for it (chapter 6).
- 124M parameters: 124,439,808 weights, i.e. numbers.
- Each is a 32-bit float, 4 bytes: 124,439,808 x 4 = 497,759,232 bytes, about 500 MB computed.
- 548 MB is the size of the file, 548,105,171 bytes. The extra ~50 MB is 12 unused "mask" tables (12 x 1024 x 1024 numbers) stored in the file that the recipe never reads, plus a 14,283-byte header that names each table and its shape computed, per
docs/ml-m0-scope.md.
What one token costs
To generate one word-piece, the recipe pushes one vector of 768 numbers through every weight matrix once. So the arithmetic per token is about one multiply-add per weight: 124 million multiply-adds, 0.249 GFLOP (a FLOP is one add or one multiply, so a multiply-add is two) measured by counting. Of that, about 85 million are in the 12 blocks and 38.6 million in the final step that turns the vector into a score for each of the 50,257 possible tokens computed (the same split as Figure 1.3's matrices, docs/ml-m0-scope.md).
To remember. The weights are fixed data; the recipe is a program; running is multiply-adds; and for every token generated, every weight is used once. That last fact decides almost everything in chapter 3.