Quick estimate for single-stream LLM inference speed

2026-07-26 · 2 min read

When generating one token at a time with a single request, inference is usually memory-bandwidth bound. A simple rule of thumb is:

Both values should use the same units (e.g., GB/s and GB).

Why this works

Each new token requires the full model weights to be read from memory. The arithmetic is fast enough that the GPU/ASIC is mostly waiting on memory, so throughput is capped by how fast weights can stream through.

This estimate ignores context length (KV-cache bandwidth), quantization, batching, and per-token overhead. It works best for dense transformer models running one token per step.

Examples

Example 1: Mid-range GPU, 9B model

• GPU memory bandwidth: 608 GB/s
• Model size: ~18 GB (Qwen3.5-9B at BF16)

Example 2: High-end GPU, 70B model

• GPU memory bandwidth: ~2,000 GB/s
• Model size: ~140 GB (Llama 3.3-70B at FP16)

Example 3: Quantized model on consumer GPU

• GPU memory bandwidth: 600 GB/s
• Model size: ~35 GB (Qwen3.5-32B at INT4/4-bit)

Common bandwidth references