Inference performance is shaped by more than GPU specifications. Quantisation, batching, caching and speculative decoding can change memory use and service behaviour, but each introduces conditions that need testing.
Quantisation
Quantisation represents weights or activations with lower precision. It can reduce memory footprint and sometimes improve throughput. Results depend on the checkpoint, method, kernel and hardware.
Record the exact converted artefact, quantisation method and calibration data where applicable. “Four bit” is not a complete identity.
Quality must be checked against the accepted task. A model that loads is not necessarily a model that preserves the required answer quality.
Continuous batching
A serving engine can combine work from several requests as sequences enter and leave the active batch. This can raise aggregate utilisation and throughput.
The trade-off may appear in queueing or individual request latency. Benchmark the intended arrival pattern and concurrency rather than an unlimited synthetic queue alone.
KV-cache management
The key-value cache holds attention state for active sequences. Its demand grows with model architecture, context and active tokens. Paged allocation and related techniques can reduce fragmentation, but they do not remove the physical capacity limit.
Disclose cache precision and allocation settings when they materially affect memory or quality.
Prefix caching
When requests share a long prefix, a runtime may reuse previously computed state. This can reduce repeated prompt work for suitable traffic.
The benefit depends on actual shared content and cache hits. A benchmark with a common system prompt should not be presented as representative of unrelated prompts without saying so.
Caching also creates a data-boundary question. Review tenancy, eviction and whether sensitive prefixes can influence cache state across users.
Speculative decoding
A smaller draft method can propose tokens that the target model verifies. Accepted proposals may reduce generation latency in suitable conditions.
The result depends on draft quality, target model, workload and implementation. Report the draft configuration and acceptance behaviour alongside the headline speed.
Optimisations can interact
Lower-precision weights may free capacity for a larger batch or longer cache. A larger batch can raise throughput but worsen tail latency. Prefix caching may reduce prompt work without changing decode performance.
Test the complete combination. Do not add isolated best-case improvements together.
Controlled evaluation
Keep a baseline and change one group of settings at a time. Record:
- immutable checkpoint and converted artefact;
- runtime and kernel versions;
- quantisation and cache precision;
- scheduler, batch and concurrency limits;
- prefix reuse and cache state;
- speculative decoding configuration;
- quality, latency percentiles, throughput, errors and memory.
Select the configuration that passes the business acceptance set with headroom, not the one that wins one synthetic metric.
Technical context
See the physical and operating boundary
Use these views to connect the guide to the machine, its airflow and its operating environment. Captions state the limits of what each image shows.
Primary sources
Sources are checked at the review date. Platform terms, prices and public guidance can change; verify them at the point of decision.