Weekly Research Intelligence — OR for Generative AI

Operations Research for Heterogeneous LLM Infrastructure and Agentic Workflows

This research front focuses on applying advanced Operations Research (OR) techniques, including Mixed-Integer Linear Programming (MILP), Dynamic Programming (DP), Convex Optimization, and Stochastic Control, to address critical resource allocation, scheduling, and efficiency challenges in large-scale AI systems. The core theme revolves around optimizing LLM serving, training, and agentic workflows, often in heterogeneous or distributed environments. Specific applications span fine-grained tensor management for ultra-long context LLM training, carbon-aware caching, fair scheduling for LLM applications, multimodal inference optimization, and efficient RLHF scheduling on diverse GPU clusters.

Key contributions include `MEMO` (2025-01) achieving 1.97x MFU improvement for long-context LLM training using bi-level MIP, and `Justitia` (2025-10) reducing average job completion time by ~60% on vLLM through virtual-time fair queuing. `HetRL` (2025-12) demonstrated 3-9x throughput gains for RLHF scheduling on heterogeneous GPUs using a multi-level search framework, while `FlexSP` (2025-02) accelerated LLM training by up to 1.98x with MILP-driven adaptive sequence parallelism. `HeteroServe` (2026-03) improved multimodal LLM inference cost-efficiency by 37% via modality-level partitioning. Furthermore, `RLTune` (2025-12) combined PPO with MILP for DL workload scheduling, reducing makespan by ~25%, and `Global Resolution` (2025-11) achieved 10,000x faster optimal multi-draft speculative sampling using convex minimization. The `CPU-Centric Perspective` (2025-11) revealed CPU bottlenecks in agentic AI, proposing micro-batching for 2.1x P50 latency speedup.

This front is rapidly emerging, with papers from 2025-2026 indicating active and foundational research. The trajectory suggests a strong focus on integrating these OR techniques into more complex, dynamic, and heterogeneous AI architectures, particularly for multimodal and agentic LLMs. Future work will likely address real-world deployment challenges such as dynamic load, fault tolerance, and energy efficiency, with a clear trend towards hybrid approaches that combine learning-based methods (e.g., RL, Bayesian Optimization) with symbolic solvers (MILP, DP) for robust and adaptive system optimization.

OR and RL for Heterogeneous LLM Serving and Agentic System Design

This research front is defined by the rigorous application of Operations Research (OR) techniques, primarily Mixed-Integer Linear Programming (MILP) and Integer Linear Programming (ILP), alongside Reinforcement Learning (RL) and multi-agent frameworks, to optimize complex Large Language Model (LLM) systems. A central theme is the efficient serving and scheduling of LLMs on heterogeneous GPU clusters, as seen in frameworks like MuxTune, JIGSAWSERVE, Cascadia, SageServe, and Helix. Concurrently, significant effort is directed towards designing, orchestrating, and ensuring the robustness of multi-agent LLM systems, with notable contributions from MaMa, SAMPO (ARLArena), AdaptOrch, MAESTRO, and Glia.

Key contributions include substantial improvements in LLM serving efficiency and cost reduction. Papers like MuxTune, JIGSAWSERVE, Cascadia, SageServe, and Helix demonstrate throughput gains of 2-11x and cost reductions of 25-50% by leveraging MILP/ILP for joint optimization of model placement, parallelism strategies, and GPU spatial partitioning. Inference-fleet-sim uses queueing theory to achieve 43% cost savings in fleet sizing, while Puzzle employs MIP for Neural Architecture Search, compressing Llama-70B to 51B with a 2.17x throughput increase. In multi-agent systems, MaMa's adversarial co-evolution reduces attack success rates from ~50% to 15-25%, and SAMPO stabilizes agentic RL, boosting ALFWorld success to 92%. Glia introduces an agentic workflow for system design, outperforming AlphaEvolve by 1.3-1.7x in response time with superior sample efficiency, and ETS uses ILP to reduce KV cache size by 1.8x and increase throughput by 1.4x.

This front is rapidly emerging, characterized by a concerted effort to formalize and solve previously ad-hoc problems in LLM deployment and multi-agent coordination using robust OR and AI methodologies. The trajectory indicates a strong push towards integrating these solutions into real-time, dynamic systems, enhancing their scalability to larger, more complex heterogeneous clusters and agentic tasks. Future work will likely focus on developing adaptive control loops that can handle uncertainty and changing workloads, while also improving the interpretability and auditability of these sophisticated AI-driven systems, potentially through advanced LLM-driven reasoning and distillation techniques.

OR-Driven Optimization for LLM Serving and Multi-Agent Systems

This research front is characterized by the application of rigorous Operations Research (OR) methodologies, including Integer Linear Programming (ILP), Mixed-Integer Programming (MIP), and Queueing Theory, to optimize various facets of Large Language Model (LLM) serving infrastructure and multi-agent systems. The unifying theme is a shift from heuristic-based approaches to mathematically grounded optimization for problems such as dynamic parallelism planning, GPU resource allocation, KV cache management, and the orchestration of complex agentic workflows. Key frameworks like HAP, FineEP, and Staggered Batch Scheduling (SBS) exemplify this trend in LLM serving, while AgentConductor and BAMAS apply similar principles to multi-agent coordination.

Significant contributions include the use of ILP/MIP for dynamic selection of hybrid parallel strategies in MoE inference (HAP, achieving up to 1.77x speedup), fine-grained load balancing (FineEP, 37-47% throughput gains), and optimal resource allocation for distributed LLM inference (AMPD, BAMAS, MoETuner, showing 60-80% latency reduction or ~80% cost reduction). Queueing-theoretic frameworks (Li et al., Bari et al.) establish throughput optimality for LLM inference scheduling, while systems like SBS reduce Time-to-First-Token by 30-40%. For multi-agent systems, RL-based policy optimization with OR-inspired reward signals (AgentConductor, UMAD) and dynamic rollout allocation (DynaMO, +11.8% Pass@1) demonstrate improved performance and sample efficiency. Novel techniques like embedding-based output length prediction (TRAIL, 2x lower latency) and convex-hull relaxation for KV cache eviction (Tang et al., 80% cache reduction) further highlight the impact of OR.

This front is rapidly maturing, demonstrating a clear trajectory towards integrating formal OR methods into the core design and runtime of LLM-powered systems. The next wave of research will likely focus on developing more adaptive and robust OR-driven solutions that can handle the inherent uncertainties of real-world LLM workloads, such as dynamic request patterns, heterogeneous hardware, and multi-modal inputs. Expect to see deeper integration of hybrid OR-RL architectures for real-time decision-making, advanced techniques for managing stochasticity (e.g., chance-constrained programming, generative ambiguity sets), and the extension of these optimization paradigms to more complex, long-horizon multi-agent and multi-modal tasks.