Machine learning in Clay Cloud / chapter 8 of 8
8. Status and glossary
What exists today, what was measured, and every term used in this book.
What exists today
As of 2026-11-11 (docs/ml-workloads.md section 10, docs/roadmap.md). Rungs are the project's plan; "done" means the acceptance check in the document passed on the developer's machine.
| Rung | What | State |
|---|---|---|
| M0 | Scope: operation inventory, pinned models, harness design | done 2026-11-10 (docs/ml-m0-scope.md) |
| M1 | Tensor ability, eager CPU host in Rust, counting and simulated handlers, a tiny network | done 2026-11-10: 22 operations plus 3 arena operations; about 100 GFLOP/s matmul; 7 microseconds per call end to end |
| M2 | GPT-2 small in Clay with KV cache and greedy decode; native tokenizer; oracle and harness | done 2026-11-11 (docs/ml-m2-gpt2.md): 20 of 20 prompts pass; 54 to 61 tokens/s |
| M3 | Runner integration: the host as a service behind the job contract, guards that enforce pins, usage reports | not started |
| M4 | A second and third architecture (Llama-style, BERT), possibly via tinygrad | not started (needs download approval) |
| M5 | GPU backend | not started |
| M6 | Graph-recording handler and fusion | not started; waits on the compiled tier |
What is measured, and what is not
- Measured (CPU, one machine, native, one thread): tokens per second, the time split, 433 calls and 0.249 GFLOP per token, 7.3 microseconds per call, about 40 GB/s, peak memory 1.02 GB, 20 prompts against the oracle, mutation checks of the harness. All in
docs/ml-m2-gpt2.md. - Not verified: any GPU; a browser or phone; other models (SmolLM2, MiniLM, Llama-style); bit-level determinism across machines or thread counts; the host checking pinned digests at run time; per-position logits for the 1,000-token prompt; exotic whitespace in the tokenizer; running on the 4 GB server; multi-tenant behaviour. The list is the "What was NOT verified" section of
docs/ml-m2-gpt2.mdplus those chapter 7 marks open. - Illustrative, not data: the matrix numbers in Figure 1.2 and the attention percentages in Figure 2.1. Inferred: the batch curve in Figure 3.1, the cache size per token, GPU generalities.
Glossary
- Activation function
- A fixed non-linear function applied to numbers between matrix multiplies (GPT-2 uses a "gelu" variant). Without it, stacked matrix multiplies collapse into one.
- Arithmetic intensity
- Multiply-adds done per byte fetched from memory. Low means memory-bound, high means compute-bound (chapter 3).
- Attention
- The step where each token mixes in information from earlier tokens in proportions it computes (chapter 2).
- Batching
- Processing several sequences or tokens together so one read of the weights serves many.
- Block (transformer block)
- One repeating unit: attention then a small network, each with normalisation and a residual add. GPT-2 small has 12.
- Compute-bound / memory-bound
- Limited by arithmetic speed / by how fast data can be read from memory.
- CUDA
- NVIDIA's platform for running programs on its GPUs. Background knowledge; nothing here uses it yet.
- Decode / decoding
- Turning token ids back into text. (Also used for the generation loop; the Clay function
decodeingpt2.clayis the loop, theTokenizer.decodeoperation is the text conversion.) - Embedding
- A learned vector for each token (and position); the model's first step is a table lookup of these.
- f32, f16, bf16
- Number formats of 32, 16 and 16 bits (the last with a wider range, fewer digits). GPT-2's file is all f32; the host widens other formats to f32 on load.
- FLOP, GFLOP, GFLOP/s
- One floating-point add or multiply; a billion of them; a billion per second. A multiply-add is two FLOPs.
- Golden
- A stored expected output from the oracle, which a port is compared against.
- GPU
- A processor with thousands of simple arithmetic units and fast memory; suits large batched multiplies.
- Greedy decoding
- Always pick the highest-scoring next token. Deterministic, so checkable.
- Handle
- A small value in Clay (an id) standing for a tensor that lives in the host's memory.
- Head
- One of several parallel attention computations in a block (12 heads of 64 numbers in GPT-2).
- Hugging Face hub
- A public site hosting model repositories; each is a git repository, so a revision is a hash.
- Inference
- Running a trained model on inputs. Contrast training, finding the weights; out of scope here.
- Kernel
- A native routine that does one heavy operation (such as matmul). Not the operating system kind.
- KV cache
- The stored keys and values of earlier tokens, so each new token only computes its own (chapter 2).
- Layer norm
- Rescales a vector's numbers to a standard mean and spread. Keeps values in range between steps.
- Logits
- The raw scores for every possible next token (50,257 for GPT-2).
- Matmul (matrix multiply)
- Rows of one matrix against columns of another, products summed (chapter 1). The one operation that is almost all of the work.
- Causal mask
- The rule that a token may only attend to itself and earlier tokens.
- MIG, MPS
- NVIDIA mechanisms for sharing one GPU: MIG splits it into isolated slices, MPS lets several processes share it. Background knowledge, mentioned only as a hosting question (chapter 7).
- MLP
- The small two-layer network applied to each token separately inside a block.
- Oracle
- A trusted independent implementation used to check another (here, the pure-Python one for the harness; UCM is the project's other oracle, for hashing).
- Parameter / weight
- One learned number. GPT-2 small has 124,439,808.
- Quantisation
- Storing weights with fewer bits (8 or 4 instead of 32) to shrink memory and the bytes read per token, at some cost in accuracy. Not built here: we start with f32 and plan f16/bf16 and one 8- or 4-bit format (
docs/ml-workloads.mdsection 9). - Query, key, value
- Three vectors derived from each token: what it looks for, what it offers, and what it contributes when attended to.
- Roofline
- A chart of speed against arithmetic intensity, with a memory slope and a compute ceiling (Figure 3.1).
- safetensors
- The weights file format: a JSON header naming each tensor and shape, then raw numbers.
- Softmax
- Turns a list of scores into percentages that add to 100%, keeping the order.
- Tensor
- A block of numbers with any number of dimensions; its shape is the list of sizes.
- Token, tokenizer, vocabulary
- A piece of text with a whole-number id; the program that cuts text into tokens; the fixed list of all of them.
- Tolerance
- How much two numeric results may differ and still count as equal, because float arithmetic is order-dependent.
- Ability, handler, perform
- Clay terms: an ability is a set of operations a function may request; performing one suspends the function; the nearest handler answers and resumes it.
- Arena (push, pop, keep)
- Host memory scoping: tensors made between
pushandpopare freed atpopunlesskeepmoves them out. Keeps a token's temporaries from piling up.