MLOps & Infrastructure · beginner · concept 199 of 311 · Updated Nov 4, 2026
Batch vs Online Inference
Batch inference computes predictions ahead of time in bulk; online inference computes each one when a request arrives. Batch is cheap and simple, but it can go stale and cannot handle inputs it has never seen. Online handles anything but must be fast and always available. Many systems combine them, and LLM batch APIs often cost less than live calls.
L2 · Confident professionalDevelopersTech leadersLeadershipProduct leadersQA and testingSEO and content
Key terms
Static inferenceDynamic inferenceLatencyStalenessBatch API
Learn these first
Where you meet it in the real world
Nightly recommendations, real-time fraud checks, bulk document processing
Guides and articles