Batched in Back: Characterizing and Optimizing Offline LLM Inference in Production with ACDC
Paper Reading · Part 13 of 13Reading notes on what Batch APIs offer users and model providers, how batch requests behave, and how ACDC uses those characteristics.