MLOps & Infrastructure · intermediate · concept 147 of 187

💸Inference Cost & Token Economics

Inference cost is the recurring price of serving a model, billed per token and split between input and output. The two are not symmetric: input arrives in one parallel prefill pass, while output is generated one token at a time, each step re-reading model weights from memory. Practitioners tune prompt length and ignore output length, which is priced higher and often dominates the bill. Context length also charges twice, through prefill work and through KV cache memory that caps batch size and throughput. Caching, smaller models, and routing are the main levers.