Seems like best options are between 512 and 1024; 640 having the lowest time to first token. These results are collected from following benchmark template:

When generating one token at a time with a single request, inference is usually memory-bandwidth bound. A simple rule of thumb is: Both values should use the same units (e.g., GB/s and GB). Each new…

This guide uses Terraform with Helm, since that's how I manage services and configuration.

• Part 1 of 5: Kernel
• Part 2 of 5: Kubernetes node-feature-discovery (NFD)
• Part 3 of 5: Kubernetes intel-gpu-plugin
• Part 4 of 5: Build vLLM
• Part 5 of 5: Deploy vLLM

Restart Poweroff

This script deletes all container registry repositories for a given GitLab project via the API. It fetches repository IDs, then issues a DELETE request for each one.

New blog in place. It's running this: https://gitlab.com/hexide/note