Machine learning in Clay Cloud / chapter 4 of 8
4. How Python does it, and how Clay does it
Both are glue over native kernels. The difference is where the seam between them is drawn.
Python's arrangement
Almost no machine-learning arithmetic runs in Python. A PyTorch program is Python code that calls a library (PyTorch, NumPy); each call, such as "matrix multiply these two tensors", jumps into compiled C++ or CUDA code that does the arithmetic and returns a tensor handle. Python owns the control flow: the loop over tokens, the cache, batching, which model to load. The kernels, the part that needs to be fast, are native and heavily tuned over years. That is what we mean by "Python is glue".
The weight of the ecosystem is the real asset: thousands of operators, GPU kernels tuned for every card, hundreds of architectures on the Hugging Face hub. docs/ml-workloads.md section 9 says so plainly: "The ecosystem is the moat".
Clay's arrangement
Clay plays Python's role with one difference: the seam is an ability. Recall from the language docs that an ability is a set of operations a function may perform, which some enclosing handler answers. We declare one ability, Tensor, with 22 operations (matmul, add, softmax, layerNorm, gather, reshape, ...), plus 3 for arena memory (push, pop, keep) (docs/ml-m0-scope.md). A tensor in Clay code is a tiny opaque handle {id: nat}; the actual numbers stay in the host's memory. The model is then an ordinary Clay program that performs Tensor.* and has no idea who answers.
packages/tensor-host/models/gpt2.json, layer 5 has CPU, counting and simulated handlers). Highlighted: the ability and the handlers, which Figure 4.2 animates.Handlers are the backends
This is the new thing relative to Python. In PyTorch, "run on GPU instead" means moving tensors to a device and relying on the library to dispatch. In Clay it means wrapping the same program in a different handler. Try it: pick a handler and step through one Tensor.matmul call.
packages/tensor-host/clay/tensor.clay (the CPU host is the Rust process packages/tensor-host; countTensor and simulateTensor are Clay). The FLOP figure is computed: 2 x 1 x 768 x 2,304 for GPT-2's first attention projection. The two greyed buttons are designed (docs/ml-workloads.md section 4) and not built.Here are the two real clauses that differ, copied from tensor.clay. The counting handler forwards the call to whatever answers Tensor outside it and bumps a counter; the simulated one answers without any host.
// countTensor (line 104)
Tensor.matmul(a, b, resume) => {
let r = Tensor.matmul(a, b);
resume(r, n + 1)
};
// simulateTensor (line 188)
Tensor.matmul(a, b, resume) => resume(mkTensor(n), n + 1n);
What the model code looks like
This is the real code of one GPT-2 block, from packages/tensor-host/clay/gpt2.clay. It reads like NumPy: calls on tensors, ordinary records, ordinary functions. The <Tensor> after the name is the declared ability requirement.
let affine<Tensor> = (l: gLinear, x: tensor): tensor => Tensor.add(Tensor.matmul(x, l.w), l.b)
let layerNormed<Tensor> = (n: gNorm, x: tensor): tensor => Tensor.layerNorm(x, n.g, n.b, 0.00001)
let block<Tensor> = (b: gBlock, x: tensor, past: option<gKv>, t: int, offset: int): (tensor, gKv) => {
let (h, kv) = attend(b, layerNormed(b.ln1, x), past, t, offset);
let x1 = Tensor.add(x, h);
let m = affine(b.out, Tensor.act(affine(b.fc, layerNormed(b.ln2, x1)), "gelu_new"));
(Tensor.add(x1, m), kv)
}
Because the program is an ordinary Clay definition it has a content hash like any other, so "the GPT-2 forward pass" can be named, reviewed and pinned exactly. The same file runs unchanged under the counting and simulated handlers (docs/ml-m2-gpt2.md, "The model in Clay").
An honest comparison of overheads
| Python + PyTorch | Clay + tensor host | |
|---|---|---|
| Who does the arithmetic | native kernels (CPU libraries, CUDA) | a native Rust host (matrixmultiply crate and a custom matrix-vector kernel); CPU only today |
| Cost of one tensor operation call | not measured: no PyTorch exists in this repository (docs/ml-m0-scope.md chose a dependency-free oracle instead) | 7.3 microseconds end to end measured (suspend, JSON, host, resume) |
| Share of a GPT-2 token spent outside the kernels | not measured | about 27% (engine 3.9 ms + glue 1.0 ms of 16 to 18.5) measured |
| Interpreter speed | CPython is the baseline | about 1.4 to 4 times slower than CPython 3.9 on the twelve benchmarks of docs/engine-performance.md (docs/ml-workloads.md section 8 rounds this to "1.4 to 5x"); Clay interpretation proper is well under 1 ms per token |
| Switching backend | move tensors to a device | wrap in another handler; the model is unchanged |
| Identity of the program | a file in a repository | a content hash; the effects it may perform are in its type |
| Ecosystem, GPU, training | mature | none of GPU or training exist; operators we have: 22 |
One measured comparison we do have, with a caveat. In M0 a pure-Python dot product took 22.7 ns per multiply-add measured; the host's matrix-vector kernel does a token's 124 M multiply-adds in about 13 ms, roughly 0.1 ns each computed, over 200 times faster. That compares native code with pure Python, which nobody uses for this; it explains why the independent oracle needs 18 minutes for a 313-token prompt that the host does in 1.7 s, and it says nothing about PyTorch.
To remember. The seam in Clay is a language feature, not a library convention. The price today is about a quarter of a token's time at batch one, on small kernels (engine boundary, chapter 3); the gains are interchangeable backends, a hashed, effect-typed model, and a single description that runs in a test, a counter or the real thing.