How we measure AI speed

Engineers measure generation speed in tokens per second. That unit is almost useless for comparing models, because a token is not a fixed amount of text — models with larger vocabularies pack more meaning into each token, so an identical sentence costs a different number of tokens depending on who generated it.

We count actual words instead, and compare them against the speed a person reads. Adults read roughly 200–300 words per minute, so we use 250 wpm ≈ 4.2 words per second as the reference point. A model running at 12 words per second is about three times reading speed; one at 40 is roughly ten times.

The tokens-per-second toggle is still there for anyone who wants it. When a provider reports its own token counts we use those; otherwise we estimate at about 0.75 words per token and label the figure as an estimate.

Thinking delay is the pause between asking and the first word appearing. It is dominated by queueing, prompt processing and network latency rather than generation speed, which is why we report it separately instead of folding it into the headline rate.

A reasoning model starts emitting tokens quickly, but they are hidden thinking that you never see. Judged on time to first token it looks fast. Judged on when readable text appears, it can be several seconds slower.

So we report both: time to first token for engine responsiveness, and time to first answer token for the wait a person actually experiences. In one measurement, Claude with extended thinking produced its first token in about 1.5 seconds and its first readable word at 4.5 seconds, after 65 words of hidden reasoning.

Output speed for these models is measured over the answer only. A long reasoning chain should not be counted as slow typing.

Benchmarks usually start the clock on the first chunk that arrives over the network. That is trivially gameable: a server can flush an empty chunk the instant the connection opens, long before it has processed anything, and report a latency that is really just network round-trip time.

We start the clock only on the first chunk containing non-whitespace content. Our own proxy applies the same rule in reverse — it forwards nothing until the provider sends real text — so we cannot flatter ourselves by accident.

When a provider finishes generating before your browser reads the socket, the entire response arrives in one burst. Divide the words by the time between first and last chunk and you get a number that describes the network, not the model.

Where a provider reports its own generation time we use that figure instead of our observation. Where it does not, any observed rate above 500 tokens per second is flagged as buffered delivery rather than published as a clean measurement.

On-device speed varies with GPU clock ramping, thermal state and whatever else your machine is doing. Consecutive single runs on one MacBook Air produced 15.8, 24.4, 25.5 and 31.3 words per second — a spread of about ±25% on identical hardware and an identical prompt.

So a local test runs a discarded warm-up pass, then 3 measured passes, and reports the median. The same machine then reads within a few percent run to run. The median is used rather than the mean because one stalled pass should not drag the headline number with it, and because with an odd number of passes the median is a real run — so the transcript and timings you see belong to the same pass as the number.

Cloud runs are a single pass. Their variance is dominated by server queueing rather than your device, and repeating them would burn through the daily free allowance for no real gain.

Text arriving in even intervals reads comfortably; the same average speed delivered in bursts with pauses does not. We record the gap between consecutive chunks and judge the spread relative to the average gap, not in absolute milliseconds — otherwise a slow but perfectly steady model would be marked as stuttering merely for being slow.

When delivery is batched, the gaps collapse toward zero and would score as flawlessly smooth while measuring nothing. Those runs are reported as not measurable instead.

Every model receives the identical prompt with the same output cap — 150 tokens for the quick test — and temperature pinned to zero where the provider allows it. Output length is capped by the request, never by asking the model for a word count, because models do not reliably comply and a variable length would change the measurement window between providers.

On-device model download time is excluded. That is setup, not inference.

The on-device test runs Llama 3.2 1B in your browser through WebGPU, in a background worker so the page stays responsive. Weights are downloaded once and cached. Nothing you type or generate leaves your machine, and it costs nothing to run because it uses your hardware.

We use the q4f32 build rather than q4f16 because f16 shaders are unavailable on a meaningful share of consumer GPUs, and a benchmark that fails to start is worse than one that downloads a little more.

Provider speeds change, model catalogues rotate, and device results depend on thermal conditions. Any figure quoted here is a measurement from a particular moment, not a permanent property of the model. The way to get a current number is to run the test yourself.