Batched in Back: Characterizing and Optimizing Offline LLM Inference in Production with ACDC
What Batch APIs offer users and providers
Product tagging, bulk translation, and document summarization often involve large volumes of data and can tolerate delayed responses. A Batch API lets users submit those requests as one job and retrieve the results after the platform processes them asynchronously. OpenAI Batch API
Users can trade a more flexible completion deadline for a lower price. If I need to tag 100,000 product descriptions, what matters is getting the full result by tomorrow. The time it takes for any one record to start generating is less important.
Model providers can run batch jobs on capacity reserved for online traffic peaks when that capacity sits idle. Relaxing per-request latency requirements also gives them room to reorder requests and choose configurations that get more throughput from the same GPUs.
Batch request characteristics
- Throughput matters at the job level. Completion time and cost matter more than the latency of an individual request.
- Inputs are available up front. The scheduler can sample requests, reorder them, and plan cache reuse before dispatch.
- Lengths often cluster within a job. Different jobs can have different input and output length distributions, even on the same model.
- Requests in a job can share long prefixes. A common prompt template creates reuse opportunities. The paper reports a median prefix-sharing ratio above 80% for Model-B.
- Reuse varies by modality. The rate of repeated images depends on the workload, so the case for caching image encodings needs a separate look.
- Jobs vary widely in size and duration. Job count is a poor proxy for work. The cost of preprocessing and changing configurations also has to be amortized over the job.
ACDC’s architecture and optimizations
Reusing online serving and isolating resources
I see two options here. A provider can reuse the online serving stack or allocate separate resources for offline inference. The trade-off is in maintenance, interference, and how much control the offline scheduler needs.
Reuse online serving. One serving stack is easier to operate. Batch jobs can fill spare capacity that online serving reserves for traffic bursts.
With shared resources, batch capacity depends on how accurately the platform estimates online demand. Underestimating demand or admitting too many batch requests can hurt online latency. Raising the priority of online traffic helps only so much once batch requests are already queued or their KV caches are resident on the GPU.
Allocate separate offline resources. Isolation can happen at the resource pool, instance, or execution stage. It reduces interference and lets offline instances use throughput-oriented settings without being bound by online time-per-output-token targets.
The cost is a second set of serving configurations to maintain. If offline jobs still depend on capacity released by online serving, their GPU supply fluctuates with online traffic. With disaggregated prefill and decode, that fluctuation can change the balance between the two stages. Extra GPUs only help if they add capacity where the job is bottlenecked.
What ACDC does
ACDC uses dedicated offline instances with their own execution configurations and scheduling policies. It profiles each job and uses request lengths, shared prefixes, and historical measurements to guide three optimizations (§4).
| Design | Approach | Implementation requirements |
|---|---|---|
| Parallelism | Run a uniform 5% sample, profile input and output lengths, and match the job to a measured tensor, data, and expert parallelism configuration. Reassess when the length distribution changes. | Benchmark candidate configurations, maintain the results, and hand off requests between instances when switching. |
| Cache reuse | Build a prefix tree over the full input set and dispatch requests in depth-first order. Prioritize KV blocks from completed subtrees for eviction. | Preserve prefix relationships during routing and coordinate with the inference engine’s cache manager. |
| Speculative decoding | Choose an initial speculation length γ using warmup batch sizes, acceptance rates, and historical measurements. Reassess it periodically during execution. | Measure candidate settings and define when and by how much to adjust γ. |
ACDC system architecture, Figure 10 from the paper by Leping Yang et al. Paper, §4 · CC BY 4.0
The workload observations explain why these optimizations could help. Carrying the gains into production depends on the surrounding systems. Per-job parallelism needs a maintained performance database, while changing a running configuration also takes instance scheduling and cache handoff. For dynamic speculation length, I would measure how much it adds over a fixed setting under realistic batch sizes and acceptance rates. Request reordering is easier to try with an existing prefix cache.
Prestarted instances with Sleep Mode could reduce repeated initialization. Each parallelism configuration still needs its own execution instance, and standard level 1 sleep discards the KV cache. Cache reconstruction belongs in the handoff cost. vLLM Sleep Mode
Request reordering and cache lifetime
If requests with a shared prefix are spread far apart, the engine can evict the corresponding KV blocks before the next request uses them. ACDC uses a prefix tree to run those requests close together. It also tracks pending requests to determine which cached prefixes are no longer needed (§4.2).
- Build the index. Keep request IDs and input token sequences, build the prefix tree, and count pending requests in each subtree.
- Reorder and dispatch. Traverse the tree depth first. Route requests with shared prefixes to the same instance where possible and run them close together. Use request IDs to associate results with the original inputs.
- Reclaim cache. Update the remaining reference counts when a request finishes. If a subtree has no pending requests and no running request references the corresponding KV blocks, prioritize those blocks for eviction.
I would start by reordering at the dispatch layer and using the engine’s existing prefix cache. Tracking remaining references can come later if eviction order is still a bottleneck. Reordering cuts repeated prefill, and reference tracking lets the engine keep blocks that pending requests will need.
References
- Batched in Back: Characterizing and Optimizing Offline LLM Inference in Production with ACDC
- OpenAI Batch API
This post was written with assistance from Codex.