Intel ARC B70 Pro --max-num-batched-tokens sweep

2026-07-27 · 3 min read

Seems like best options are between 512 and 1024; 640 having the lowest time to first token.

These results are collected from following benchmark template:

vllm serve Qwen/Qwen3.5-9B --max-model-len 8192 --max-num-batched-tokens $N

vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model Qwen/Qwen3.5-9B \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 128 \
  --num-prompts $NPROMPTS \
  --max-concurrency $C \
  --request-rate inf \
  --ignore-eos \
  --save-result \
  --result-dir /tmp \
  --result-filename bench_result_c${C}.json

Output throughput (tok/s)

Mean TTFT (ms)