Skip to content
CLAY CLOUD DOCSMachine learningConceptsStatusDashboard

Machine learning in Clay Cloud / chapter 3 of 8

3. Where the time goes

Doing the arithmetic is cheap. Bringing the weights to the arithmetic is not.

All numbers in this chapter are from one machine: an 8-core Apple Silicon laptop, native release build, one host thread, CPU only (docs/ml-m2-gpt2.md, "Speed and where the time goes"). They are measured unless labelled otherwise. Nothing here was measured on a GPU, in a browser, or on the 4 GB server.

Two ways to be slow

A processor can be limited by two different things.

Which one applies depends on arithmetic intensity: how many multiply-adds you do for each byte you fetch. If you fetch a weight and use it once, intensity is low and memory sets the speed. If you fetch it once and use it many times, the arithmetic becomes the limit.

Why one token is memory-bound

Generating one token multiplies a single row (the one new token's vector) by each weight matrix. Each weight is fetched, used for exactly one multiply-add, and not needed again until the next token. A token therefore reads all of the weights, about 500 MB of them, to do 0.249 GFLOP of work. In numbers:

QuantityValueLabelSource
Work per token0.249 GFLOP (124 M multiply-adds)measuredml-m2-gpt2.md
Weights read per tokenabout 500 MB (f32)measuredml-m2-gpt2.md
Intensity at one token0.249 GFLOP / 0.5 GB = 0.5 FLOP per bytecomputedfrom the two rows above
Host kernel time per token12 to 14 ms (the 73 matmul calls: 11.6 to 13.6 ms)measuredml-m2-gpt2.md
Memory speed achieved500 MB in about 12 ms is about 40 GB/s on one coremeasuredml-m2-gpt2.md
Best matrix-multiply speed seenabout 100 GFLOP/s (M1, large multiplies)measuredml-workloads.md rung M1
Speed we got at one token0.249 GFLOP / 12 to 14 ms = 18 to 21 GFLOP/scomputedfrom the rows above

So at one token the machine runs at about a fifth of its demonstrated arithmetic speed, and exactly as fast as memory lets it. That is memory-bound.

Figure 3.1. A "roofline" picture. The sloped line is what memory allows (40 GB/s times the intensity); the flat line is what the arithmetic units allow (~100 GFLOP/s). They meet at 2.5 FLOP per byte computed. The green dot is our measured single-token run; it lies on the slope. Step through batch sizes to see the INFERRED effect of sharing one read of the weights between several tokens (simple model: intensity rises in proportion to the batch; real batching adds costs this ignores, and we have not measured it).

Why GPUs help, and what batching is

inferred / background knowledge, not measured by us. A graphics card is a processor with thousands of simple arithmetic units and memory several times (often an order of magnitude) faster than a laptop's. Both help a memory-bound job: more bytes per second per token, and more arithmetic when the job turns compute-bound. A GPU is much less helpful if you feed it one token at a time and waste its arithmetic.

Batching is the standard remedy: process several sequences (or several prompt tokens) in one go, so that one fetch of a weight serves many multiply-adds. That moves a job to the right on Figure 3.1. The prompt pass already does this for free: all the prompt tokens go through together. A measured hint: the 313-token prompt plus 16 generated tokens took 1.7 s in total; the 1,000-token prompt plus 8 tokens, 4.4 s. Serving many users in one batch is a scheduler question for Clay Cloud (docs/cloud-workloads.md) and has not been built or measured for models.

The time split of one token

If the arithmetic were all, a token would cost only the host-kernel slice. The rest is the cost of driving the host from Clay: each tensor operation is an ability call that the engine suspends at, packages as JSON, hands to the host, and resumes from.

Figure 3.2. One generated token, from docs/ml-m2-gpt2.md (greedy decode, the 5-token prompt "The meaning of life is", per-token cost from runs differing only in the number of new tokens, median of several). Segments are drawn at the midpoint of each measured range. 433 calls and 0.249 GFLOP per token; a bare Tensor.shape call costs 7.3 microseconds end to end, so 433 calls is about 3.2 ms computed.

What was tried, and what it taught

To remember. At one token per step the job is limited by how fast 500 MB can be read, not by arithmetic. Faster memory, sharing a read between tokens, or smaller weights (quantisation, chapter 8) help; a faster multiplier alone would not.