An int8 model was 40% faster on our research box and 2.6× slower on our production CPUs. Why speed results flip between machines, and how to test.
An int8 model ran 40% faster on our research box. On our production CPUs it ran 2.6× slower. It never shipped.
We caught it because the extraction benchmarks ran on our own cluster, not only on a research machine. Without that, the research number would have looked like a free 40%, and we’d have shipped a slowdown.
This is why a speed result can flip between machines, and how to test so it doesn’t catch you.
What int8 is supposed to buy you
A machine-learning model is mostly large tables of numbers called weights. Normally each weight is a 32-bit floating-point number, called fp32.
Quantisation stores those numbers in fewer bits. With int8, each one becomes an 8-bit whole number, one of 256 possible values, plus a scale that maps it back to roughly its real value.
Two things change. The model gets four times smaller, because 8 bits is a quarter of 32. And the maths can get faster, because a processor can work on far more 8-bit numbers at once than 32-bit ones. The price is some accuracy, so you check that too.
“Can get faster” is doing a lot of work in that sentence.
Why the same model can win on one machine and lose on another
I’m not going to claim which of these applied to us. These are the usual reasons a result flips, and any one of them is enough.
The speed-up needs the right instructions. An instruction set is the list of operations a processor can run directly. Int8 is only fast on chips with instructions built for it. On x86 the main one is VNNI, short for Vector Neural Network Instructions, which does an 8-bit multiply and add in one step where older chips need three. Intel brought it to Xeon servers with Cascade Lake, and AMD added it with Zen 4. Newer Intel Xeons add AMX, built for matrix maths. Arm chips have their own int8 instructions.
On a chip without them, the runtime falls back to slower code. It still has to convert numbers between float and int8 around the quantised parts of the model, and that overhead can eat the whole saving and more. ONNX Runtime’s own documentation warns that quantised models can end up slower than the original on CPUs without VNNI.
The machine around the chip matters too.
- Cores and threads. Runtimes split work across threads. A research box with many free cores can make a model look faster than it will be in a container that gets two.
- CPU limits. In Kubernetes, a container with a CPU limit is paused once it uses its share of each short scheduling period. That can wreck timings. More in CPU throttling in Kubernetes.
- Cache and memory speed. Smaller weights fit in cache more easily, and how much that helps depends on the chip.
Two machines with the same number of cores can sit on opposite sides of any of these lines. A spec sheet won’t tell you which side you’re on.
The principle: a benchmark measures a model on a machine
A speed result isn’t a property of the model. It belongs to the model, the runtime and the machine together. Change any of them and you have a different result.
So a research box is a good place to find candidates and a bad place to decide. The decision belongs on the hardware that will run the model and send you the bill.
How to test so it doesn’t catch you
- Benchmark on the same CPU type and instance size you’ll pay for, inside the same container limits.
- Compare CPU flags before you trust a speed-up. On Linux, look for avx512_vnni, avx_vnni or amx_int8 in /proc/cpuinfo, on the research box and inside a container on a production node. If one has a flag the other doesn’t, treat the research box’s int8 numbers as a guess.
- Run the quantised and full-precision models side by side, on the same machine, with the same thread count.
- Throw away the warm-up runs, then time enough pages to see the slow ones, not only the average.
- Check accuracy as well as speed. A faster model that reads tables worse isn’t a win.
- Test again when you change instance type or cloud provider.
What it changed
The model that was 40% faster on the research box would have run 2.6× slower in production. Because the measurement ran where the model would live, it never shipped, and the slowdown never reached a customer.
The lesson
Benchmark on the hardware you’ll actually pay for. The research box made the wrong option look like a free 40%.
The rest of the extraction work is in the extraction case study.
Quick answers
- What is int8 quantisation?
- It stores a model's numbers as 8-bit integers instead of 32-bit floating-point numbers, with a scale that maps them back to roughly their real values. The model becomes four times smaller and can run faster on CPUs with fast int8 instructions.
- Can int8 quantisation make a model slower?
- Yes. On CPUs without fast int8 instructions such as VNNI, the runtime falls back to slower code paths, and converting numbers back and forth can cost more than the smaller maths saves.
- What is VNNI?
- VNNI (Vector Neural Network Instructions) is an x86 extension that does an 8-bit multiply and add in one instruction instead of three. Intel brought it to Xeon servers with Cascade Lake, and AMD added it with Zen 4.
- How do I check if my CPU supports VNNI?
- On Linux, look for avx512_vnni or avx_vnni in the CPU flags in /proc/cpuinfo. Check inside the container, on the same kind of machine you run in production.