MLOps & Infrastructure · advanced · concept 146 of 176

🔧Inference Optimization

The engineering that makes serving a model affordable. Generation splits into a compute-bound prefill over the prompt and a memory-bound decode of one token at a time, so throughput and cost are set by the KV cache, continuous batching, attention memory layout, and speculative decoding, where a small draft model proposes tokens a larger model verifies in one pass. Two deployments of the same weights can differ by an order of magnitude in cost per token. Attention itself gets slimmed too: multi-query and grouped-query attention shrink the KV cache that dominates serving memory. Routing sends easy queries to cheap models and hard ones to frontier models; cascades escalate only on failure.