Weekly Research Intelligence — OR for Generative AI

Key contributions demonstrate significant quantitative improvements across various facets of LLM serving. Dynamo achieved 67-340% average SLO attainment improvement through ILP-based offline deployment and adaptive routing. Lin et al. developed a multiclass many-server queueing network model with a fluid approximation and LP-based 'Gate-and-Route' policy, proving effective for prefill-decode contention. The Petals framework saw 60-80% latency reduction using a decomposed MILP heuristic for joint block placement and request routing. For MoE inference, ILP-based expert placement reduced network hops by up to 39.1% with a load-aware objective. SageServe, combining ILP with ARIMA forecasting, delivered 25% GPU-hours savings and $2.5M/month savings on real Microsoft O365 traces. Finally, Cascadia's bi-level optimization (MILP + Chebyshev) yielded 2.3x average throughput gains for cascade serving. These results underscore the efficacy of OR in enhancing LLM inference performance and cost-efficiency.

This front is rapidly emerging, driven by the escalating scale and complexity of LLM serving infrastructure. The consistent and successful application of ILP/MILP and queueing theory across diverse problem settings indicates a maturing methodology for applying OR to this domain. The trajectory suggests that future research will focus on integrating these OR solutions into more comprehensive, dynamic, and adaptive systems, addressing the limitations of current static or heuristic approaches. Expect to see more sophisticated modeling of real-world constraints, such as heterogeneous hardware, dynamic and multi-tenant workloads, and multi-objective optimization, as the field progresses towards fully autonomous and optimized LLM serving platforms.

Key contributions include Staggered Batch Scheduling (SBS) which reduces TTFT by 30-40% and improves overall throughput by 15-20% on DeepSeek-V3 workloads by enforcing buffering windows and using IQR-aware lexicographical decode scheduling. Helix formulates distributed LLM serving on heterogeneous clusters as a max-flow MILP, achieving up to 3.3x decode throughput over Swarm on LLaMA 70B by dynamically routing requests based on edge capacities. Furthermore, the 'Nested WAIT' algorithm, based on fluid dynamics approximation, provides asymptotic optimality proofs and high-probability bounds against memory overflow for online LLM scheduling, outperforming heuristic baselines like vLLM and Sarathi.

This front is emerging, driven by the increasing scale and complexity of LLM serving infrastructure. The papers demonstrate a clear shift towards more rigorous, theoretically-backed OR formulations over purely heuristic approaches. The trajectory suggests that future work will focus on integrating these diverse advanced scheduling techniques, scaling MILP solutions for larger clusters, and adapting fluid-guided methods to multi-GPU and highly dynamic environments, while also developing analytical frameworks for up-to-date LLM inference models.

Key contributions include the Robust Multi-Objective Decoding (RMOD) framework, which formulates multi-objective decoding as a zero-sum game solvable via convex optimization, demonstrating gains of +1.2% WCWR on LLM-as-Judge and +0.17 worst-case reward on HH benchmarks. Concurrently, the Safety Game introduces a novel game-theoretic formulation for black-box LLM agent alignment, employing an LP solver at inference time to balance safety and helpfulness. This approach achieved up to a two-fold improvement in accuracy on SafetyBench, outperforming baselines in 11 of 15 test cases by guaranteeing that stochastic policies are no worse than a safe fallback.

This front is emerging, showcasing novel applications of operations research techniques like game theory and robust optimization to critical LLM challenges at inference time. The trajectory suggests a move towards more sophisticated, dynamic control mechanisms for LLM behavior. The next likely paper will focus on extending these robust game-theoretic frameworks to handle sequential dialogues, a larger number of objectives, or dynamically tune trade-off parameters in real-time.

Paper [1] introduces a Bayesian adaptive stopping criterion for majority voting, leveraging Dirichlet process priors and Bayes factors to reduce test-time compute by 2-5x for LLM reasoning tasks on benchmarks like AIME2025. It also formulates LLM ensemble optimization as a Mixed-Integer Linear Program (MILP), showing superior performance (e.g., 93.3% accuracy vs 90.0% for single LLMs). Paper [2] presents PROBE, a framework for MoE inference optimization that employs Continuous Lookahead Pipelining and predictive prefetching. PROBE demonstrates significant performance gains, achieving a 1.32x speedup in prefill latency with SGLang and 1.26x higher decoding throughput compared to DeepSeek-EPLB on large MoE models.

This front is clearly emerging, showcasing novel applications of Operations Research and AI techniques to critical LLM inference challenges. The trajectory suggests a focus on integrating sophisticated decision-making into LLM serving, moving beyond static configurations. Future work will likely involve refining adaptive sampling mechanisms, extending MILP formulations for dynamic ensemble management, and developing more advanced, specialized solvers within predictive pipelining architectures to further optimize resource utilization and throughput.