Skip to content
CLAY CLOUD DOCSMachine learningConceptsStatusDashboard

Machine learning in Clay Cloud / chapter 8 of 8

8. Status and glossary

What exists today, what was measured, and every term used in this book.

What exists today

As of 2026-11-11 (docs/ml-workloads.md section 10, docs/roadmap.md). Rungs are the project's plan; "done" means the acceptance check in the document passed on the developer's machine.

RungWhatState
M0Scope: operation inventory, pinned models, harness designdone 2026-11-10 (docs/ml-m0-scope.md)
M1Tensor ability, eager CPU host in Rust, counting and simulated handlers, a tiny networkdone 2026-11-10: 22 operations plus 3 arena operations; about 100 GFLOP/s matmul; 7 microseconds per call end to end
M2GPT-2 small in Clay with KV cache and greedy decode; native tokenizer; oracle and harnessdone 2026-11-11 (docs/ml-m2-gpt2.md): 20 of 20 prompts pass; 54 to 61 tokens/s
M3Runner integration: the host as a service behind the job contract, guards that enforce pins, usage reportsnot started
M4A second and third architecture (Llama-style, BERT), possibly via tinygradnot started (needs download approval)
M5GPU backendnot started
M6Graph-recording handler and fusionnot started; waits on the compiled tier

What is measured, and what is not

Glossary

Activation function
A fixed non-linear function applied to numbers between matrix multiplies (GPT-2 uses a "gelu" variant). Without it, stacked matrix multiplies collapse into one.
Arithmetic intensity
Multiply-adds done per byte fetched from memory. Low means memory-bound, high means compute-bound (chapter 3).
Attention
The step where each token mixes in information from earlier tokens in proportions it computes (chapter 2).
Batching
Processing several sequences or tokens together so one read of the weights serves many.
Block (transformer block)
One repeating unit: attention then a small network, each with normalisation and a residual add. GPT-2 small has 12.
Compute-bound / memory-bound
Limited by arithmetic speed / by how fast data can be read from memory.
CUDA
NVIDIA's platform for running programs on its GPUs. Background knowledge; nothing here uses it yet.
Decode / decoding
Turning token ids back into text. (Also used for the generation loop; the Clay function decode in gpt2.clay is the loop, the Tokenizer.decode operation is the text conversion.)
Embedding
A learned vector for each token (and position); the model's first step is a table lookup of these.
f32, f16, bf16
Number formats of 32, 16 and 16 bits (the last with a wider range, fewer digits). GPT-2's file is all f32; the host widens other formats to f32 on load.
FLOP, GFLOP, GFLOP/s
One floating-point add or multiply; a billion of them; a billion per second. A multiply-add is two FLOPs.
Golden
A stored expected output from the oracle, which a port is compared against.
GPU
A processor with thousands of simple arithmetic units and fast memory; suits large batched multiplies.
Greedy decoding
Always pick the highest-scoring next token. Deterministic, so checkable.
Handle
A small value in Clay (an id) standing for a tensor that lives in the host's memory.
Head
One of several parallel attention computations in a block (12 heads of 64 numbers in GPT-2).
Hugging Face hub
A public site hosting model repositories; each is a git repository, so a revision is a hash.
Inference
Running a trained model on inputs. Contrast training, finding the weights; out of scope here.
Kernel
A native routine that does one heavy operation (such as matmul). Not the operating system kind.
KV cache
The stored keys and values of earlier tokens, so each new token only computes its own (chapter 2).
Layer norm
Rescales a vector's numbers to a standard mean and spread. Keeps values in range between steps.
Logits
The raw scores for every possible next token (50,257 for GPT-2).
Matmul (matrix multiply)
Rows of one matrix against columns of another, products summed (chapter 1). The one operation that is almost all of the work.
Causal mask
The rule that a token may only attend to itself and earlier tokens.
MIG, MPS
NVIDIA mechanisms for sharing one GPU: MIG splits it into isolated slices, MPS lets several processes share it. Background knowledge, mentioned only as a hosting question (chapter 7).
MLP
The small two-layer network applied to each token separately inside a block.
Oracle
A trusted independent implementation used to check another (here, the pure-Python one for the harness; UCM is the project's other oracle, for hashing).
Parameter / weight
One learned number. GPT-2 small has 124,439,808.
Quantisation
Storing weights with fewer bits (8 or 4 instead of 32) to shrink memory and the bytes read per token, at some cost in accuracy. Not built here: we start with f32 and plan f16/bf16 and one 8- or 4-bit format (docs/ml-workloads.md section 9).
Query, key, value
Three vectors derived from each token: what it looks for, what it offers, and what it contributes when attended to.
Roofline
A chart of speed against arithmetic intensity, with a memory slope and a compute ceiling (Figure 3.1).
safetensors
The weights file format: a JSON header naming each tensor and shape, then raw numbers.
Softmax
Turns a list of scores into percentages that add to 100%, keeping the order.
Tensor
A block of numbers with any number of dimensions; its shape is the list of sizes.
Token, tokenizer, vocabulary
A piece of text with a whole-number id; the program that cuts text into tokens; the fixed list of all of them.
Tolerance
How much two numeric results may differ and still count as equal, because float arithmetic is order-dependent.
Ability, handler, perform
Clay terms: an ability is a set of operations a function may request; performing one suspends the function; the nearest handler answers and resumes it.
Arena (push, pop, keep)
Host memory scoping: tensors made between push and pop are freed at pop unless keep moves them out. Keeps a token's temporaries from piling up.