Prefill-Decode Model Serving Report

Task: As of 2026-07-01, determine when separating LLM prefill and decode improves serving over a monolithic design. Compare model sizes, prompt/output mixes, concurrency and KV-cache paths; assess TTFT, inter-token and tail latency, throughput, utilization, network overhead, reliability, complexity and total cost.

Research protocol: Define prefill, decode, disaggregation boundaries and every latency and throughput metric. Freeze model and quantization, framework release or commit, accelerator, interconnect, scheduler, batching, cache format, request traces and measurement window. Use current papers, framework docs and source, hardware guidance and reproducible tests; treat cloud or vendor benchmarks as claims unless their workload and environment can be matched. Run monolithic and split deployments on identical steady, mixed-length and burst traces, showing distributions rather than headline averages. Include KV-cache serialization and transfer, queueing, failure domains, autoscaling, observability and operator burden alongside accelerator cost. Explain version drift, incompatible metrics and missing evidence. Deliver an architecture map, experiment matrix, raw assumptions, comparable results, break-even conditions, operational risks and a workload-bound recommendation.

Task: As of 2026-07-01, determine when separating LLM prefill and decode improves serving over a monolithic design. Compare model sizes, prompt/output mixes, concurrency and KV-cache paths; assess TTFT, inter-token and tail latency, throughput, utilization, network overhead, reliability, complexity and total cost.

Research protocol: Define prefill, decode, disaggregation boundaries and every latency and throughput metric. Freeze model and quantization, framework release or commit, accelerator, interconnect, scheduler, batching, cache format, request traces and measurement window. Use current papers, framework docs and source, hardware guidance and reproducible tests; treat cloud or vendor benchmarks as claims unless their workload and environment can be matched. Run monolithic and split deployments on identical steady, mixed-length and burst traces, showing distributions rather than headline averages. Include KV-cache serialization and transfer, queueing, failure domains, autoscaling, observability and operator burden alongside accelerator cost. Explain version drift, incompatible metrics and missing evidence. Deliver an architecture map, experiment matrix, raw assumptions, comparable results, break-even conditions, operational risks and a workload-bound recommendation.