68 recent papers · updated 2026-09-03 · sources deduplicated from arXiv, OpenAlex and Semantic Scholar
2026-09-01 | Meta Platforms, Inc. Method: Optimal bipartite assignment using Hungarian algorithm on LLM hidden states for O(1) pass decoding | LLM role: score_matrix_generator_from_hidden_states
HLLM replaces the sequential autoregressive decoding of generative rankers with a single forward pass, using a lightweight self-attention head and the Hungarian algorithm to decode the optimal permutation directly from the LLM's prefill hidden states. The results are strongly backed by empirical numbers, demonstrating a 64x speedup (down to 28ms latency) on a proprietary dataset and a 45x speedup on Amazon Beauty, while maintaining lossless ranking quality compared to the autoregressive teacher. The key insight is the architectural pattern of bypassing token-by-token generation for constrained outputs by training with a differentiable relaxation (Sinkhorn) and decoding with an exact combinatorial solver on the hidden states.
2026-08-26 | University of Science and Technology of China, Hefei University of Technology Method: Task-Oriented Prefix-Aware Scheduler (TOPAS) that jointly determines prefix residency and request admission via a JCT-oriented utility function and hierarchical state search | LLM role: none
TOPAS is an online scheduler for multi-agent LLM serving that jointly optimizes KV cache prefix residency and request admission to minimize task-level job completion time. Backed by empirical numbers, it reduces mean and p99 JCT by up to 39.8% and 49.4% on synthetic DAGs and significantly improves performance on real MetaGPT workflows compared to baselines like Shortest-Path-First. The critical insight is treating prefix residency as an explicit, workflow-aware scheduling decision rather than a reactive byproduct of request ordering, using a utility function that balances the reduction of a task's longest remaining service path against near-term prefix reuse. This is highly relevant for research in OR formulations for LLM serving scheduling, offering a concrete heuristic search approach to manage the memory-compute tradeoff in complex multi-agent deployments.
2026-08-22 | Microsoft Azure Research, Cornell University, Cornell Tech Method: Karush–Kuhn–Tucker (KKT) online solver for convex-relaxed joint allocation | LLM role: none
POWERSLIDER optimizes LLM serving under dynamic power caps by disaggregating the pipeline into Prefill, Think and Answer stages and using a KKT online solver to dynamically adjust GPU allocation and frequencies based on phase asymmetry. The results are rigorously backed by empirical data, demonstrating 1.64x higher goodput than the best baseline at a 30% power cap reduction while solving the allocation problem in just 7.7ms. The key insight is that reasoning workloads break standard prefill-decode disaggregation due to massive KV-cache accumulation during the 'thinking' phase; isolating this into a three-stage pipeline allows an online convex solver to aggressively scale down frequencies for memory-bound decode stages without starving compute-bound prefill.
2026-08-12 | The Hong Kong University of Science and Technology (Guangzhou), IDEA Research, DataArcTech Ltd. Method: Mixed-integer linear programming (MILP) for joint optimization of checkpoint selection, activation placement, recomputation and CPU-GPU-NVMe communication overlap | LLM role: none
LazyTrain formulates limited-resource LLM training, specifically activation offloading, recomputation and communication overlap, as a mixed-integer linear programming (MILP) scheduling problem. The results are backed by concrete hardware measurements, demonstrating a 1.24x TFLOPS improvement over heuristic baselines on a single H800 for a 27B model. The key insight is that treating heterogeneous memory offloading as a joint MILP path-selection problem, rather than a greedy heuristic, enables the solver to perfectly hide PCIe and NVMe transfer costs within compute windows.
2026-08-08 | Beihang University, University of Leeds, University of Sydney Method: Lagrangian decomposition with projected subgradient ascent for resource allocation, DNN for interference prediction and Kubernetes-native middleware for orchestration | LLM role: none
ElastiCo is a GPU cluster scheduler that co-locates deep learning training and offline LLM inference by treating job configurations as elastic variables and using shadow pricing for multi-resource allocation. The results are backed by strong empirical evidence on a 64-GPU testbed and 512-GPU simulation, demonstrating a 2.94x reduction in average job completion time and an increase in GPU utilization from 25% to 46% compared to static partitioning baselines. The key insight is the combination of 'Resource Shape Transformation'—exposing a family of valid resource-performance profiles for each job—with a Lagrangian decomposition that uses dynamic shadow prices to resolve cluster-wide multi-resource contention.
2026-08-06 | National University of Defense Technology Method: Hybrid Adaptive Thread-Tuning using Physics-Informed Neural Operator (PINO) guided by a finite-source M/M/1 queueing model, combined with a three-tier dynamic adjustment strategy | LLM role: none
Su et al. propose AutoThread, a hybrid adaptive thread-tuning method that dynamically optimizes worker thread counts for reinforcement learning simulation environments. The authors demonstrate real empirical gains, achieving up to an 83.8% runtime reduction and 1.8x throughput improvement over baseline tuning methods on AMD and Intel platforms. The key insight is the use of a Physics-Informed Neural Operator (PINO) where a finite-source M/M/1 queueing model acts as a structural constraint on the neural network's loss function, ensuring predictions remain physically plausible under dynamic workloads.
2026-08-07 | Tsinghua University Method: Global resource allocation using progressively refinable prototype trees and a utility-guided greedy budget flow with a budget-aware singleton floor. | LLM role: target_of_optimization
GraceKV formulates KV cache compression for long-context LLMs as a global resource allocation problem, using prototype trees and greedy marginal-utility allocation to balance local resolution and broad coverage. The method is backed by strong empirical results, achieving state-of-the-art performance across 24 of 32 settings on LongBench and RULER while reducing KV cache memory by up to 92% at 30K context lengths. The key insight is treating KV cache compression not as a fixed eviction or merging rule, but as a unified knapsack-like allocation problem across layers, heads and context slots, enabling dynamic, input-conditioned memory optimization.
2026-08-06 | The University of Hong Kong, Imperial College London Method: Polynomial-time algorithm with constant approximation guarantee based on problem decomposition, enumeration of feasible parameters and sorting for user scheduling | LLM role: none
Qu et al. propose an edge LLM inference scheduling framework that maximizes task throughput by jointly assigning users to either autoregressive or speculative decoding while partitioning GPU memory and compute. The results are backed by numerical simulations and device measurements, demonstrating ~30-38% throughput improvements over single-mode baselines. The key insight is that the latency-memory trade-off between AD (low memory, high latency) and SD (high memory, low latency) can be effectively optimized by decoupling the modes and applying greedy knapsack heuristics based on user-specific latency contributions.
2026-08-04 | Imperial College London, University of Cambridge Method: Event-driven simulation framework (HeteroPanacea) with roofline execution model for NPU architectures, supporting disaggregated quantization and parallelization scheduling | LLM role: none
Forys et al. introduce HeteroPanacea, a simulation framework to evaluate 4-way (Prefill-Decode-Attention-FFN) disaggregated LLM serving across heterogeneous hardware and parallelization strategies. Backed by extensive simulation data, they demonstrate that this PDAF disaggregation yields up to a 2.06x throughput gain over non-disaggregated serving for prefill-heavy agentic workloads on custom NPUs. The key insight is that the hardware demands of attention (KV-bandwidth bound) and FFN (compute/weight bound) diverge so sharply in agentic workloads that they require entirely different hardware profiles to avoid resource stranding.
2026-08-03 | University of Science and Technology of China, China Telecom Cloud Computing Research Institute, Xidian University, SKLP, ICT, CAS Method: Two-level control plane (Global Scheduler using Integer Linear Program, Local DVFS Controller) combined with an interleaved A/F pipeline with dynamic microbatch depth and adaptive request batching | LLM role: none
AFlex optimizes energy consumption in LLM serving by disaggregating Attention and FFN operators and applying an Integer Linear Program (ILP) alongside a local controller to dynamically scale GPU frequencies. The results are empirically validated on NVIDIA A800 GPUs, demonstrating up to a 49% reduction in energy per token compared to state-of-the-art disaggregated serving systems while maintaining strict latency SLOs. The key insight is that Attention and FFN operators exhibit distinct frequency sensitivities; formulating an ILP to independently provision and scale frequencies for these operators yields massive energy savings over coarse-grained phase-level controls.
2026-08-03 | The University of Sydney Method: Joint temporal-spatial scheduling with slack-based deadline protection, shared-stage path rotation and bandwidth-guided SM throttling | LLM role: none
HorizonServe introduces a joint temporal-spatial scheduler for single-GPU omni-model serving that coordinates request admission and streaming multiprocessor (SM) allocation to meet heterogeneous service-level objectives (SLOs). Backed by strong empirical results, it improves SLO attainment by up to 7.0x and reduces p95 first-response latency by up to 63.7% compared to vLLM-Omni and EDF baselines on RTX 6000 GPUs. The key insight is that the critical bottleneck in omni-model serving is cross-stage memory bandwidth contention between the shared multimodal backbone and downstream generators; bounding the shared-stage SM allocation during co-running prevents goodput collapse for tight-SLO text requests.
2026-08-01 | The Hong Kong Polytechnic University Method: Causality-inspired replay valuation and shrinkage-based packet-value prediction for resource-constrained semantic packet admission | LLM role: none
BANDMAS optimizes multi-agent LLM communication by breaking messages into semantic packets and scheduling their transmission based on a causality-inspired value predictor and resource constraints. Backed by strong empirical results, it reduces transmitted bytes by 53-77% on QA benchmarks while maintaining or improving task accuracy compared to pruning baselines. The key insight is the 'causality-inspired replay valuation'—using offline counterfactual removal (testing sufficiency and necessity of individual message chunks) to train a lightweight predictor of a message's contribution to the final outcome.
2026-07-24 | Imperial College London, Peking University, Microsoft Research, University of Edinburgh, Shanghai Jiao Tong University, Tile-AI Method: Unified tile-centric analytical execution engine with hierarchical pipeline envelope modeling, tile reuse-distance cache modeling and placement-based cross-device tile access | LLM role: none
TileSight is a first-principles, tile-centric analytical performance model that predicts GPU kernel and end-to-end LLM serving latency by simulating intra-tile resource usage, inter-tile cache reuse and cross-device communication. The results are rigorously backed by hardware measurements, achieving 12.35% MAPE on single-GPU kernels and 13.52% wMAPE on end-to-end vLLM serving across diverse architectures (A100 to B6000), outperforming learned predictors. The key insight is that lifting performance modeling to the tile abstraction (rather than thread or cache-line level) enables fast, schedule-sensitive and deterministic latency predictions without requiring per-architecture ML training. This is highly relevant for research in LLM serving scheduling and GPU resource allocation, as TileSight can serve as a highly accurate, white-box cost estimator for optimization formulations.
2026-07-18 | Arizona State University Method: Wasserstein distributionally robust optimization (DRO) with block coordinate descent (BCD) algorithm | LLM role: none
Cheng et al. formulate KV cache reservation in LLM serving as a Wasserstein distributionally robust optimization (DRO) problem to jointly optimize GPU parallelism, routing and prefix caching under output length uncertainty. The results are rigorously backed by trace-driven simulations on production workloads (BurstGPT, Azure), demonstrating up to 56% lower costs compared to fixed-quantile baselines while maintaining competitive P99 latency. The key insight is the derivation of a critical fractile structure that automatically adapts the reservation quantile based on the ratio of preemption to memory-waste costs, eliminating the need for manual tuning.
2026-07-20 | Peking University, Independent Researcher Method: Hybrid disaggregation-colocation architecture with adaptive persistent kernels (APKs) and attention-initiated one-sided MoE communication, optimized by a cross-stack placement optimizer | LLM role: none
ExpertPlex introduces a hybrid LLM serving architecture for Mixture-of-Experts (MoE) models that shares massive MoE weights across prefill and decode phases while disaggregating attention, supported by tile-level GPU scheduling and a cross-stack placement optimizer. The system demonstrates strong empirical results, achieving up to 2.01x higher goodput over standard prefill-decode disaggregation on real-world MoE models like MiniMax-M2.7 and GLM-5.1-FP8. The key insight is the tile-aware latency model for MoE computation: because sparse routing changes latency based on per-expert packing rather than just token volume, resource allocation models must account for active expert counts rather than raw token counts.
2026-07-19 Method: Workload-aware system combining model-free SuffixDecoding and cache-aware scheduling | LLM role: target_model_inference_acceleration
WAR optimizes synchronous agentic RL rollouts by dynamically applying model-free speculative decoding (SuffixDecoding) under low load and cache-aware request scheduling under high load. The results are backed by strong empirical evidence, demonstrating 1.4x to 1.6x throughput improvements on 16 H100 GPUs using Qwen3-32B for software engineering agent tasks. The key insight is that the effectiveness of rollout optimizations is highly workload-dependent; speculative decoding wastes compute under high load where standard batched decoding already saturates GPUs, making cache-aware scheduling (prioritizing KV-cache locality and shorter trajectories) the better lever.
2026-07-19 | Vienna University of Technology, University of Klagenfurt, University of Messina Method: Lightweight heuristic approximating Binary Integer Linear Programming (BILP) optimization, leveraging five XGBoost/Random Forest ML models for query-specific performance prediction | LLM role: none
LMEdge orchestrates LLM inference across heterogeneous edge devices by using ML predictors and a lightweight heuristic to jointly optimize model selection, quantization and placement under QoS constraints. The results are backed by empirical measurements from a 57-device Kubernetes testbed, demonstrating improved latency and resource utilization over standard load-aware baselines. The key insight is the practical integration of query-specific XGBoost predictors for accuracy and latency directly into the scheduling constraints, alongside a released dataset of 59,000 inference measurements that could be highly valuable for offline scheduling simulations. This work is directly relevant to operations research formulations for LLM serving scheduling, though its methods are relatively standard and primarily target edge environments rather than large-scale cloud GPU clusters.
2026-07-17 | Tsinghua University, Peking University, Beihang University, Beijing Institute of Technology, JDT AI Infra Method: Service-oriented architecture with decoupled Training Model Service, Inference Model Service and Environment Service, employing multi-tenant group batching for shared VLM backbone forward passes. | LLM role: none
JoyNexus proposes a multi-tenant service architecture for Vision-Language-Action model post-training that decouples training, inference and environment interaction into separate, independently scheduled services. The results are backed by empirical simulations, demonstrating a 28.3% reduction in aggregate GPU time and up to 3.24x shared-forward throughput speedup. The key insight is the use of group batching for heterogeneous requests, which canonicalizes different tenant data schemas to share a single forward pass through a frozen base model before splitting features for tenant-specific updates. This systems engineering pattern is highly relevant for scaling LLM evolutionary search infrastructure, as decoupling generation, evaluation and update phases allows for asynchronous scaling and better GPU utilization during bursty search workloads.
2026-07-10 | Hong Kong University of Science and Technology, Shanghai Jiao Tong University Method: Regime decomposition with budget-doubling greedy for rectangle strips and geometric slicing SPS for response-heavy jobs, combined by a routing meta-scheduler | LLM role: none
Feng et al. introduce a non-clairvoyant scheduling algorithm for batched LLM inference that minimizes total completion time under hard KV-cache constraints. The authors mathematically prove the first O(1)-competitive theoretical guarantee against an optimal clairvoyant scheduler for arbitrary prompt and response lengths. The key insight is that a single global priority rule cannot simultaneously maintain high memory utilization and area-consistent completion order; instead, requests must be dynamically routed into three geometric regimes (large, prompt-heavy and response-heavy).
2026-05-21 | MIT, Kensho Technologies, Ben-Gurion University Method: Integer Programming (IP) for vocabulary selection, solved via Linear Programming (LP) relaxation and rounding heuristic, combined with greedy split tree construction and recursive inference | LLM role: code_writer, wording_improvement
Schmidt et al. introduce ToaST, a tokenization method that replaces greedy heuristics like BPE by formulating vocabulary selection as an Integer Program to minimize total token count. The results are rigorously backed by numbers, demonstrating an 11% compression improvement over BPE and significantly better downstream language model performance. The key insight is that formulating the selection problem over a root-to-leaf tree structure yields an exceptionally tight Linear Programming relaxation, allowing exact OR solvers to scale to massive AI infrastructure problems.
2026-05-20 | Harvard University, Boston University Method: Closed-loop control system with offline power-performance random forest regression models and online PID feedback controller for joint hardware (power caps) and software (batch size) optimization | LLM role: none
PALS introduces a power-aware runtime for LLM serving that jointly optimizes GPU power caps and batch sizes using a closed-loop controller. Backed by hardware measurements on multi-GPU setups, it achieves up to 26.3% energy efficiency improvements and a 4x-7x reduction in QoS violations. The key insight is that for communication-bound MoE models, increasing power beyond a specific threshold degrades efficiency by accelerating communication overheads rather than useful computation.
2026-05-20 | The Chinese University of Hong Kong, Anuttacon, StepFun Method: Discrete-event simulation with disaggregated abstraction and hardware-aware predictors | LLM role: none
Frontier is a discrete-event simulator for modern LLM inference serving that accurately models disaggregated architectures, complex parallelism and stateful reasoning workloads. Backed by extensive physical H800 GPU profiling, it reduces end-to-end latency prediction error from over 45% in existing simulators to under 7% by replacing average-case analytical proxies with hardware-aware predictors. The key insight is that coarse analytical models for KV-cache and operator runtimes distort SLA predictions and can reverse optimization conclusions; accurate evaluation requires modeling the closed-loop dynamics of memory state and batch composition.
2026-05-20 | National University of Singapore, Nanyang Technological University, Beihang University, Shanghai Qiji Zhifeng Co., Ltd., Infrawaves, Shanghai Innovation Institute Method: Cluster-level multiplexing of unified LLM services using spatio-temporal scheduling, affinity-aware placement and centralized state management | LLM role: workload_target
PlexRL is a cluster-level runtime that multiplexes LLM execution across multiple Reinforcement Learning with Verifiable Rewards (RLVR) jobs to reclaim idle GPU capacity caused by long-tailed rollouts and phase alternation. The results are backed by strong empirical numbers on a 2048-GPU cluster, demonstrating up to a 37.58% reduction in GPU-hour costs for 7B to 235B models compared to asynchronous split deployments. The key insight is that decoupling algorithm control from model execution allows the system to treat rollout and training as shared cluster services, enabling spatio-temporal packing that interleaves jobs to hide the massive latency of long-tailed generation and tool-use stalls.
2026-05-19 | Rice University, University of Central Florida, Mobi.AI Method: Interval-based expert refresh strategy with I/O-aware expert offload, optimized by mathematical programming and greedy search for optimal interval | LLM role: none
TIDE optimizes inference for Mixture-of-Experts diffusion LLMs by formulating the GPU-CPU expert offloading schedule as a mathematical programming problem based on the temporal stability of expert activations. The results are backed by empirical hardware profiling, achieving up to 1.5x throughput improvements on LLaDA2.0 models using a single GPU-CPU setup. The key insight is that the temporal locality of expert routing in diffusion models allows for interval-based expert refreshing, where the optimal interval balancing I/O overhead and CPU compute can be solved analytically via mathematical programming.
2026-05-13 | LayerScale, Inc. Method: Data-driven stateful transformer inference with decoupled data and query planes | LLM role: none
Norgren introduces a stateful transformer inference architecture that decouples data ingestion from query processing, maintaining a persistent KV cache to achieve constant-time query latency for streaming workloads. The approach is backed by strong empirical results, demonstrating a 2.4x to 5.9x speedup over state-of-the-art engines like vLLM and SGLang on streaming benchmarks while maintaining approximately 43ms latency regardless of context size. The key insight is the use of Flash Queries, which utilize idle GPU cycles between data arrivals to pre-compute answers to registered queries against the evolving context, effectively pushing user-visible latency toward zero.
2026-05-13 | Zhejiang University, Northeastern University, University of Surrey, Zhongguancun Institute of Artificial Intelligence Method: Token-batch pipeline scheduling via dynamic programming and dual-threshold NAV triggering with Bayesian optimization autotuner | LLM role: inference_engine
PipeSD accelerates cloud-edge collaborative LLM inference by using dynamic programming to optimally batch and pipeline draft tokens, alongside a Bayesian optimization-tuned dual-threshold mechanism for triggering speculative verification. The results are backed by empirical hardware measurements, demonstrating 1.16x–2.16x speedups and up to 25% energy reduction over baselines like EdgeLLM on standard benchmarks. The key insight is formulating the token-batching decision—balancing communication startup overhead against immediate transmission—as a dynamic programming problem to perfectly overlap edge-side autoregressive generation with network transmission.
2026-05-10 | ByteDance Seed, University of Washington, Cornell University Method: Disaggregated component placement with multi-stage AF-Pipe and roofline-model guided adaptive worker allocation | LLM role: none
DisagMoE optimizes large-scale Mixture-of-Experts (MoE) model training by disaggregating attention and feed-forward network (FFN) layers onto separate GPU groups and using a multi-stage pipeline to overlap communication and computation. The results are backed by strong empirical evidence, demonstrating up to 1.81x throughput speedups over Megatron-LM and 1.34x over state-of-the-art overlap methods on a 128-GPU H800 cluster. The key insight is the use of a Compute-Communication roofline model, solved via MILP, to asymmetrically allocate GPU and NIC resources based on the distinct arithmetic intensities of different model components (compute-bound attention vs. communication-bound FFN).
2026-05-08 | University of Science and Technology of China Method: CP-SAT-backed frontier planner with horizon-aware candidate scoring and state-conditional cost estimation | LLM role: none
FATE introduces a future-state-aware scheduler for heterogeneous LLM workflows (e.g., multi-agent DAGs) that uses a CP-SAT solver to optimize both immediate execution costs and downstream state preservation. The results are backed by solid empirical evidence, showing an 8.9% reduction in normalized makespan and an 8.8% reduction in P95 latency over the strongest baseline on a WfCommons-derived benchmark. The key insight is that LLM workflow scheduling cannot be myopic; it must explicitly model how current placement decisions alter future execution states, specifically regarding model residency, cross-device transfer and KV-cache/prefix reuse. This is highly relevant to the team's work on OR formulations for LLM serving scheduling, as the rolling-horizon constraint programming formulation and state-conditional cost estimators offer a directly implementable approach for optimizing multi-agent execution on GPU clusters.
2026-05-07 | Indian Institute of Technology Bombay Method: Reinforcement Learning-based prefix-aware scheduler with Chunked Hash Tree for fast prefix detection | LLM role: none
Feather introduces a prefix-aware LLM inference scheduler that uses a novel Chunked Hash Tree for fast prefix detection and a reinforcement learning policy to dynamically balance batch size against prefix homogeneity. The results are strongly backed by empirical evidence, demonstrating 2-10x higher end-to-end throughput compared to vLLM and SGLang baselines while reducing CPU scheduling overhead by up to 1000x. The key insight is a paradigm shift for inference batching: maximizing batch size is sub-optimal for prefix-shared workloads, as moderately small, prefix-homogeneous batches achieve higher throughput by maximizing spatial and temporal locality in KV cache accesses.
2026-05-06 | University of California, San Diego, GenseeAI Inc. Method: Adaptive Tensor Parallelism (TP) with TP-aware weight reuse and pipelined KV migration for dynamic GPU allocation and SLO-aware request scheduling | LLM role: none
Nitsum dynamically adjusts Tensor Parallelism (TP) levels and prefill/decode GPU allocations at runtime to maximize SLO-compliant goodput for multi-tenant LLM serving. The system achieves up to 5.3x higher goodput than state-of-the-art baselines like Llumnix, backed by rigorous experiments on real-world Azure and Alibaba traces. The key insight is that TP can be treated as a dynamic runtime control surface rather than a static deployment choice, enabled by keeping full weight copies on each GPU and using pipelined KV migration to reduce switching overhead to milliseconds.
2026-05-01 | The University of Hong Kong, Stellaris AI Limited, Brain Investing Limited Method: Distributed workflow-atomic scheduling for AI agent inference on GPU clusters, utilizing Agent Execution Graphs for predictive KV cache management (WA-LRU, Tool-Call-Aware TTL, speculative prefetching), session-affinity batching with work stealing and Agent Fair Share for task-level fairness. | LLM role: none
SAGA introduces a workflow-atomic scheduler for AI agent inference on GPU clusters that uses Agent Execution Graphs to proactively manage KV cache retention across multi-step reasoning and tool-call boundaries. Backed by strong empirical results on a 64-GPU cluster, it achieves a 1.64x geometric mean speedup in task completion time and 1.22x better memory utilization over state-of-the-art vLLM with automatic prefix caching. The key insight is the Workflow-Aware LRU (WA-LRU) eviction policy, which uses the agent's execution graph to predict future KV cache reuse probabilities, effectively bridging the gap between online cache management and the offline-optimal Bélády policy.
2026-05-02 | HKUST Method: Mathematical optimization and principled algorithmic design for LLM serving systems | LLM role: none
This position paper argues that LLM inference serving must transition from generic heuristics to rigorous mathematical optimization, synthesizing recent advances in applying operations research to AI infrastructure. Rather than presenting new experiments, it aggregates empirical evidence from recent literature—such as LP-based load balancers for Mixture-of-Experts and online integer programming for data parallelism—to demonstrate that OR methods consistently outperform heuristics. The key insight is the formalization of LLM serving bottlenecks, like dynamically growing KV caches and prefill-decode asymmetry, into specific OR problem classes, alongside a roadmap of open problems including scheduling for agentic workloads.
2026-04-28 | University of Illinois Urbana-Champaign, National University of Singapore Method: 3D-parallel KV cache restoration framework with batch-aware two-pointer scheduler | LLM role: none
CacheFlow optimizes KV cache restoration in long-context LLM serving by formulating it as a 3D-parallel scheduling problem (token, layer, GPU) that overlaps recomputation and I/O transfer. The results are backed by strong empirical numbers, showing a 10%-62% reduction in Time-To-First-Token (TTFT) compared to state-of-the-art frameworks like vLLM and SGLang across diverse hardware and bandwidth conditions. The key insight is that the optimal balance between recomputation (which scales quadratically) and I/O loading (which scales linearly but is bandwidth-bound) can be achieved using a batch-aware two-pointer scheduler that prioritizes I/O for requests with the longest remaining lengths to restore.
2026-04-19 | Peking University Method: Hive multi-agent inference infrastructure with Logits Cache and Agent-Aware Scheduling | LLM role: inference_engine_optimization
Hive is an LLM inference infrastructure that optimizes multi-agent and test-time scaling workloads by introducing Logits Cache for redundant sampling paths and Agent-Aware Scheduling for KV cache eviction. The results are backed by solid empirical evidence, demonstrating a 1.11x-1.76x speedup for re-sampling and a 33%-51% reduction in KV cache miss rates on Qwen3-8B. The key insight is that caching intermediate logits (not just KV states) allows the engine to skip expensive forward passes during stochastic resampling of shared prefixes, while evicting KV cache based on an agent's structural contribution outperforms standard LRU. This is highly relevant for scaling LLM evolutionary search and multi-agent optimization, as it provides concrete systems-level techniques to drastically reduce the inference costs associated with branching generation and complex agent coordination.
2026-04-22 | Tsinghua University Method: Joint optimization of batching, user-to-batch assignment and integer draft lengths formulated as a fractional mixed-integer program, solved by scanning batch counts and iteratively alternating between MILP-based subproblems using Dinkelbach's method. | LLM role: none
This paper formulates the scheduling of distributed speculative decoding (local drafting, centralized verification) as a fractional mixed-integer program to maximize multi-user token throughput. The authors demonstrate up to 1.93x throughput improvements over greedy batching in simulated edge deployments using Qwen3 models. The key insight is that the complex fractional objective of throughput (expected accepted tokens per unit time) can be efficiently decoupled and solved using the Dinkelbach method combined with alternating optimization for batch assignment and draft lengths.
2026-04-16 | Imperial College London, Independent Researcher Method: Aggregate LLM Pipeline for performance prediction combined with a hierarchical heuristic search for joint throughput/latency optimization of GPU allocations and topology-aware fractional placement | LLM role: none
Scepsy is a serving system that schedules multi-LLM agentic workflows onto GPU clusters by profiling relative LLM execution times to create an Aggregate LLM Pipeline and using a hierarchical heuristic for fractional GPU allocation. The results are strongly backed by empirical numbers on a 16-GPU cluster, showing up to 2.4x higher throughput and 27x lower latency compared to baselines like Kubernetes HPA and Ayo. The key insight is that instead of modeling the highly variable end-to-end latency of dynamic agentic workflows, systems can achieve stable steady-state performance predictions by modeling the aggregate fractional demand each LLM places on the system.
2026-04-06 | University of Washington, NVIDIA, Rice University, University of Waterloo, Cisco Research, Independent Researcher Method: SLO-aware dynamic programming scheduler with intelligent video preemption, elastic sequence parallelism and dynamic batching | LLM role: none
GENSERVE optimizes the co-serving of text-to-image and text-to-video diffusion models on shared GPUs using a dynamic programming scheduler that jointly manages step-level preemption, dynamic batching and elastic sequence parallelism. The results are empirically backed, demonstrating up to a 44% improvement in SLO attainment over baselines like SRTF on an 8-GPU cluster. The most useful takeaway is their two-stage DP formulation: they first generate a small set of anchored candidate actions (hold, resume, scale SP) per request, then run a knapsack DP to maximize global SLOs in under 2ms.
2026-03-27 | IMDEA Software Institute, Universidad Politécnica de Madrid Method: Modality-aware dynamic priority scheduling with aging mechanism | LLM role: target_of_optimization
RPS-Serve introduces a modality-aware scheduler for multimodal LLMs that classifies requests into 'rocks' (video), 'pebbles' (image) and 'sand' (text) based on predicted prefill latency and memory, using dynamic priorities and aging to prevent head-of-line blocking. The results are real and backed by solid systems experiments on vLLM, showing a 78.5% reduction in time-to-first-token for latency-critical text requests compared to FCFS and EDF baselines. The core takeaway is that multimodal workloads completely break standard text-only LLM serving assumptions because video/image prefill times and KV-cache footprints are orders of magnitude larger.
2026-03-26 | Microsoft Gray Systems Lab, SK Telecom, Yonsei University Method: Data-driven co-optimization of 3D parallelism configuration and runtime microbatch scheduling using empirical profiling, an expected makespan minimization algorithm and a hybrid ILP/LPT scheduler. | LLM role: none
DFLOP optimizes distributed 3D parallelism for multimodal LLM training by combining offline profiling with an online ILP-based microbatch scheduler to minimize pipeline bubbles caused by heterogeneous data inputs. The results are real and backed by extensive hardware experiments, showing up to 3.6x throughput improvements over Megatron-LM.
2026-03-19 | Independent Researcher, Shenzhen, China Method: GPU-accelerated metaheuristic framework with 'one block evolves one solution' CUDA architecture, two-level adaptive operator selection (AOS) and hardware-aware resource management. | LLM role: modeling_assistant
cuGenOpt is a GPU-accelerated metaheuristic framework that uses a 'one block evolves one solution' CUDA architecture and JIT compilation to solve combinatorial optimization problems. The results are rigorously backed by hardware benchmarks, showing it matches OR-Tools on small instances and vastly outperforms MIP solvers, though it struggles with large-scale VRP (>200 nodes) due to memory limits.
2026-03-17 | Carnegie Mellon University, UC Berkeley, UNC-Chapel Hill, UC Santa Cruz Method: Continual meta-learning with LLM-based gradient-free skill evolution and RL-based LoRA fine-tuning using a process reward model | LLM role: policy_executor, skill_generator, reward_model
MetaClaw is a continual learning framework for LLM agents that combines gradient-free skill evolution (distilling failures into reusable prompt instructions) with asynchronous RL fine-tuning guided by a process reward model. The results are backed by solid empirical gains, showing an 8.25x improvement in end-to-end task completion on a 30-day simulated CLI benchmark.
2026-03-17 | National University of Singapore Method: Workflow-aware LLM serving framework integrating proactive KV cache management, global prompt caching and cost-based cache-aware scheduling based on a templated radix tree | LLM role: none
Helium optimizes LLM serving for batch agentic workflows by modeling them as query plans and using a Templated Radix Tree (TRT) to enable proactive KV caching and cache-aware scheduling. The results are rigorously backed by numbers, demonstrating up to 1.56x speedups over state-of-the-art systems (vLLM, Parrot, KVFlow) on complex multi-agent workflows and the authors even validate their greedy scheduler's optimality gap against an MILP solver. The most valuable takeaway is the TRT abstraction, which captures global prefix hierarchies across a DAG of LLM calls to maximize KV cache reuse, rather than relying on reactive, per-call caching.
2026-03-18 | Shanghai Jiao Tong University Method: VCG-based Min-Cost Max-Flow (MCMF) for bipartite matching, guided by Hoeffding Tree predictive QoS models | LLM role: none
IEMAS routes client requests to distributed LLM agents by formulating the assignment as a Min-Cost Max-Flow bipartite matching problem, using VCG auctions to align economic incentives with KV-cache reuse. The results are backed by solid vLLM simulations, demonstrating an 80.2% KV-cache hit rate and a 35% cost reduction over baselines like GraphRouter.
2026-03-17 | MBZUAI, McGill University, University of Chicago, Tensormesh Inc Method: Two-phase optimization combining M/G/c Kimura approximation for analytical sweep and discrete-event simulation (DES) for verification | LLM role: none
This paper introduces a two-phase capacity planner (M/G/c analytical sweep followed by discrete-event simulation) to find minimum-cost GPU fleet configurations for LLM inference under strict latency SLOs. The results are rigorously backed by simulation across multiple GPU profiles and workload traces, demonstrating that intuitive sizing rules often fail (e.g., slower GPUs can be cheaper due to KV-slot multipliers and analytical models approve broken fleets for high-variance agent traffic).
2026-03-17 | Shanghai Jiao Tong University, Chinese Academy of Sciences, University of Chinese Academy of Sciences Method: GPU coroutines abstraction decoupling logical execution contexts (vCtx) from physical GPU resources (pCtx) via dynamic context binding and cooperative preemption | LLM role: none
DETSHARE introduces 'GPU coroutines' to decouple logical execution contexts from physical GPU resources, enabling fine-grained spatial sharing without modifying kernels to preserve semantic determinism. The results are highly credible and backed by strong empirical numbers on A800/Hopper GPUs, demonstrating up to 79% higher training throughput and 69% lower inference latency compared to temporal sharing baselines.
2026-03-13 | University of Illinois Urbana-Champaign Method: Phase-aware runtime (HeteroServe) with modality-level partitioning, embedding-only transfer and cross-type work stealing | LLM role: none
Yu et al. demonstrate that partitioning multimodal LLM inference at the modality boundary (vision encoder vs. language decoder) reduces cross-device transfer costs by O(L), dropping requirements from GB-scale NVLink to MB-scale PCIe. This enables heterogeneous serving architectures where cheap compute-dense GPUs (RTX 4090) handle vision and expensive bandwidth-dense GPUs (A100) handle language. Results are strongly backed by real hardware deployments, showing a 37% improvement in cost-efficiency over homogeneous vLLM baselines.
2026-03-09 | University of Illinois Urbana-Champaign Method: Mixed Integer Linear Programming (MILP) for joint optimization of model variants, GPU spatial partitions and task-graph-informed budgeting | LLM role: none
JIGSAWSERVE uses a Mixed Integer Linear Programming (MILP) formulation to jointly optimize model variant selection (accuracy scaling) and fine-grained GPU spatial partitioning (MIG/MPS) for serving compound inference DAGs. The results are strongly backed by empirical numbers on real hardware (H100s), demonstrating an 11.3x capacity improvement over the closest prior work (Loki) while maintaining under 0.6% SLO violations.
2026-03-05 | Kingsoft Cloud Method: Hybrid theoretical modeling (M/M/1 queuing theory) and empirical benchmarking for P/D resource calculation | LLM role: none
This paper proposes a hybrid resource allocation method for disaggregated Prefill/Decode (P/D) inference, using M/M/1 queuing theory to model prefill throughput under TTFT constraints and empirical profiling for decode. The results are real and validated on NVIDIA H200 clusters running DeepSeek-V3.1.
2026-03-05 | Nanyang Technological University, Unaffiliated Method: SLO-aware elastic system combining a two-layer Prompt Bank for initial prompt selection and a Workload Scheduler for dynamic multi-GPU allocation | LLM role: feature_extractor
PromptTuner is a cluster management system for LLM prompt tuning that combines a 'Prompt Bank' (retrieving similar past prompts to speed up convergence) with a hierarchical scheduler (warm/cold GPU pools) to meet latency SLOs. The authors demonstrate real-world efficacy on 32-96 GPU clusters, showing 4-8x reductions in SLO violations compared to INFless and ElasticFlow.
2026-03-05 | Fudan University, Shanghai Innovation Institute Method: Band-constrained Policy Optimization (BandPO) using a Band operator to project f-divergence trust regions into dynamic, probability-aware clipping intervals | LLM role: policy_agent
BandPO replaces the standard static clipping in PPO/GRPO with dynamic bounds derived from projecting f-divergence trust regions, specifically addressing a bottleneck where allowable updates vanish for low-probability tokens. Empirical results are rigorous, showing consistent gains (2-10%) on math benchmarks and crucially, maintaining policy entropy where baselines collapse. The key takeaway is that standard clipping scales update margins linearly with probability, effectively freezing rare tokens; BandPO decouples this, allowing the model to actually reinforce novel, high-advantage tail strategies.
2026-03-03 | Fudan University, Shanghai Innovation Institute, Shanghai Academy of AI for Science Method: Bayesian Adversarial Multi-agent Framework for AI4S (BAMF-AI4S) with recursive co-optimization of generated code, test cases and prompts, guided by a non-LLM-based Bayesian updating rule and Bayesian Optimization for code performance estimation. | LLM role: code_writer, decomposition_guide, prompt_optimizer, test_case_generator, solution_generator
The authors propose a multi-agent framework for scientific code generation that couples an adversarial 'Challenger' (generating difficult test cases) with a 'Solver', governed by a Bayesian update rule. Crucially, they employ Bayesian Optimization with a kernel based on code embeddings (AST + text) to estimate solution quality before running expensive tests, effectively acting as a learned surrogate model. Results on SciCode and ScienceAgentBench are strong, showing small models (Qwen-32B) outperforming GPT-4o when using this loop.
2026-03-03 | Tencent Hunyuan, Hong Kong University of Science and Technology Method: Native visual-generation agentic model (VisionCreator) unifying Understanding, Thinking, Planning and Creation (UTPC) capabilities, optimized via Progressive Specialization Training (PST) and Virtual Reinforcement Learning (VRL) with LtrReward in VisGenEnv. | LLM role: agentic_model
This paper introduces VisionCreator, an agent trained via 'Virtual Reinforcement Learning' (VRL) where tool outputs and logic are simulated to train long-horizon planning policies without incurring expensive real-world execution costs. They employ a 'Plan-Driven Reward' model (combining LLM-based plan verification with rule-based execution checks) and prove theoretical bounds for the sim-to-real transfer, achieving performance superior to GPT-5 on visual tasks.
2026-03-03 | University of Minnesota-Twin Cities Method: Multi-agent framework with rubric-based agentic reinforcement learning (GRPO) | LLM role: decomposition_guide, code_writer, evaluator
StitchCUDA automates end-to-end GPU program generation using a multi-agent framework, but its core contribution is a training recipe that solves reward hacking in code optimization. They decompose expensive multi-turn agentic RL into single-turn 'atomic skills' (generation vs. refinement) and use GRPO with an LLM-evaluated 'Rubric Reward' (e.g., 'Did you use tiling?') rather than just sparse outcome metrics. This prevents the model from gaming the system (e.g., wrapping PyTorch code) and forces actual optimization behavior.
2026-03-04 | Nanjing University, Suzhou Laboratory, Shanghai Artificial Intelligence Laboratory Method: Neuro-symbolic framework integrating Sparse Monte Carlo Tree Search (MCTS) with Sibling-Aware Expansion, Memory-Driven Global Planning and a Differentiable Physics Engine for continuous ratio optimization. | LLM role: semantic_generator
Chen et al. introduce a neuro-symbolic MCTS framework for mixed discrete-continuous optimization, applying it to solvent design. They solve the LLM context bottleneck via 'Sparse State Storage' (storing only state abstractions and reconstructing paths on-demand) and fix mode collapse using 'Sibling-Aware Expansion' (conditioning the generator on sibling nodes to force orthogonality).
2026-03-03 | DoorDash, WithMetis.ai Method: Prompt-level optimization using GEPA and MAMUT GEPA | LLM role: evaluator, evolutionary_search, decomposition_guide, user_simulator
This paper presents a production-grade framework for optimizing multi-agent systems by jointly evolving prompt bundles (MAMUT) rather than optimizing agents in isolation. They validate this on a grocery assistant, showing that system-level optimization outperforms local sub-agent optimization by ~7% because it captures coordination dynamics (e.g., context passing) that local metrics miss. The most stealable insight is their 'Judge Calibration' loop: they use evolutionary search (GEPA) to optimize the evaluator's prompt to match human labels (91.4% agreement) before using that judge to optimize the agents.
2026-03-03 | Shanghai Jiao Tong University, National University of Singapore Method: Hierarchical spatial-temporal backbone multiplexing with unified PEFT representations, dynamic programming for task fusion, priority-based subgraph scheduling and chunk-based data alignment | LLM role: subject_of_optimization
MuxTune introduces a hierarchical scheduler for multi-tenant PEFT that uses Dynamic Programming to optimally fuse tasks (spatial batching) or interleave them (temporal multiplexing) based on a pipeline cost model. Empirical results on H100s show up to 5x throughput gains over NeMo and S-LoRA, validated by ablation studies.
2026-02-26 | City University of Hong Kong, Southern University of Science and Technology Method: Adaptive Iterated Local Search (AILS) with LLM-driven Evolutionary Computation for Automatic Heuristic Design (AHD) of ruin heuristics | LLM role: heuristic_generator
This paper integrates LLM-driven evolutionary search into the AILS framework to evolve 'ruin' heuristics for CVRP, employing a Chain-of-Thought 'voting' mechanism to filter out poor heuristics before expensive evaluation. The results are empirically strong: they claim 8 new Best-Known Solutions on the CVRPLib large-scale benchmark, outperforming HGS and AILS-II. This is a direct proof-of-concept that LLM-evolved components can beat hand-crafted SOTA on hard OR instances.
2026-02-26 | Alibaba Group, Harbin Institute of Technology, Shenzhen Method: Test-time rectify-or-reject pruning framework with retrieval-augmented rectifier, failure-driven indicator pool and dual-stage deduplication | LLM role: rectifier, teacher, deduplicator, reasoning_engine
Wang et al. propose a test-time 'firewall' for multi-agent systems that intercepts messages and validates them against a retrieved set of error patterns (mined from offline failure trajectories). They achieve ~6% accuracy gains on math benchmarks by iteratively rectifying or pruning erroneous outputs before they propagate. This effectively turns the 'graveyard' of failed runs into a persistent memory that improves sample efficiency.
2026-02-25 | DeepSeek-AI, Peking University, Tsinghua University Method: Dual-path KV-Cache loading architecture with CNIC-centric traffic management and adaptive request scheduling | LLM role: none
Wu et al. introduce DualPath, a system that breaks the storage I/O bottleneck in agentic inference by utilizing idle decode-node bandwidth to load KV-cache and transferring it to prefill nodes via RDMA.
2026-02-25 | Caltech Method: Strategically Risk-Averse Policy Optimization (SRPO) based on Risk-Averse Quantal Response Equilibria (RQE) | LLM role: collaborative_agent
Qu et al. introduce Strategically Risk-Averse Policy Optimization (SRPO), which trains agents against a 'constrained adversary' that minimizes their reward within a KL-divergence bound of the partner's current policy. Theoretical results prove this objective eliminates free-riding equilibria and experiments on GSM8K multi-agent debate show it prevents 'lazy' agreement, improving joint accuracy by up to 19% when pairing heterogeneous LLMs (e.g., 0.6B with 4B). The key takeaway is that robustness to partner deviation—enforced via this specific adversarial objective—is a more principled way to fix lazy agent behavior than prompt engineering or simple dropout.
2026-02-22 | Meituan, Tsinghua University, Fudan University, Peking University Method: Dual-pronged optimization framework (DynaMO) combining variance-minimizing dynamic rollout allocation and gradient-aware advantage modulation for GRPO-based policy optimization | LLM role: policy_model
DynaMO introduces a dual-pronged optimization for RLVR: a dynamic rollout allocation strategy that prioritizes problems with high gradient variance (proxied by Bernoulli variance of success/failure) and a gradient modulation technique to stabilize updates. The results are strong (+11.8% Pass@1 over GRPO on Qwen-7B) and backed by clear ablations.
2026-02-25 | University of California, Los Angeles, University of Wisconsin–Madison Method: Stable Agentic Multi-turn Policy Optimization (SAMPO) integrating sequence-level clipping, fine-grained advantage estimation and dynamic filtering | LLM role: policy
The authors dissect why standard RL (GRPO/PPO) fails in multi-turn agentic tasks, identifying that token-level importance sampling (IS) clipping allows negative-advantage outliers to destabilize training. They propose SAMPO, which enforces sequence-level clipping and integrates fine-grained step-level advantages (similar to process rewards) to stabilize learning. The results are rigorous, showing a jump from ~50% to 92% success on ALFWorld by fixing the gradient update mechanics rather than just prompt engineering.
2026-02-25 | Georgia Institute of Technology, Korea University, Sogang University, Ewha Womans University Method: Multi-Agent-based Electrocatalyst Search Through Reasoning and Optimization (MAESTRO) framework using LLM agents and a Machine Learning Force Field (MLFF) surrogate model | LLM role: evolutionary_search
Mok et al. propose MAESTRO, a multi-agent LLM framework for optimizing single-atom catalysts that explicitly separates search into exploration and exploitation phases, bridged by a textual 'Exploration Report.' Results are validated against high-fidelity DFT calculations, showing the system learns to break theoretical scaling relations via in-context learning, outperforming memory-less baselines.
2026-02-23 | Cornell University, University of Maryl and College Park Method: Differentiable Scheduling Optimization via Gaussian Reparameterization with Augmented Lagrangian Method | LLM role: none
GauS replaces the standard categorical (Gumbel-Softmax) relaxation in differentiable scheduling with Gaussian variables defined by mean and variance, reducing parameter space from O(N*D) to O(N). Results are strong: it scales to 57k nodes where previous differentiable methods OOM and exact solvers timeout, while maintaining near-100% GPU utilization. The key takeaway is a specific modeling technique: using Gaussian distributions to represent discrete ordinal values (like time steps) naturally captures temporal proximity and provides smoother gradients than categorical buckets.
2026-02-19 | Shanghai Jiao Tong University, Meituan Method: Reinforcement learning (GRPO) optimized multi-agent system with LLM-based orchestrator agent for dynamic layered DAG topology generation | LLM role: orchestrator
AgentConductor trains an LLM orchestrator via GRPO to dynamically generate and refine layered DAG interaction topologies (output as YAML) for code generation, optimizing for both correctness and token efficiency. The key innovation is a multi-objective reward that combines execution correctness with a 'difficulty-aware density penalty,' forcing the model to learn a policy that scales graph complexity with task hardness. Results are strong, showing ~14% gains on APPS while reducing token costs.
2026-02-20 | Lehigh University Method: APEMO (Affect-aware Peak-End Modulation for Orchestration), a runtime scheduling layer that reallocates reasoning effort and repair across a trajectory under fixed computational budgets by operationalizing temporal-affective signals. | LLM role: agents_being_orchestrated
Shi et al. introduce APEMO, a runtime orchestration layer that monitors agent trajectories for behavioral instability (e.g., repetition, drift) and dynamically reallocates a fixed compute budget to 'repair' these segments rather than spreading compute uniformly. The results are statistically rigorous, using bootstrap CIs to demonstrate significant improvements in trajectory robustness and completion rates without model retraining.
2026-02-18 | Korea National Open University Method: Task-adaptive topology routing algorithm based on DAG structural properties (parallelism width, critical path depth, coupling density) combined with an adaptive synthesis protocol | LLM role: decomposition_guide, executor, arbiter, synthesizer
AdaptOrch introduces a control layer that dynamically routes tasks to one of four agent topologies (Parallel, Sequential, Hierarchical, Hybrid) by analyzing the task's dependency graph properties (parallelism width, coupling density). The results are strong and credible, showing a 9.8% improvement on SWE-bench over single-model baselines and significantly outperforming static multi-agent architectures like standard MoA. The most valuable takeaway is the Topology Routing Algorithm: a linear-time heuristic that maps DAG structure to optimal agent coordination patterns.
2026-02-16 | University of Cambridge, Peking University, Shanghai Jiao Tong University, Ant Group, Southeast University Method: Adaptive routing and prefill reordering for online scheduling, combined with an Integer Linear Programming (ILP) based offline deployment planner | LLM role: none
AMPD introduces a disaggregated serving framework tailored for multi-round LLM agents, utilizing an offline ILP solver to optimize resource allocation (TP/DP configurations) and an online adaptive routing mechanism to handle incremental prefill tasks. The results are strong, showing 67-340% improvements in SLO attainment over vLLM and NVIDIA Dynamo by dynamically routing incremental prefill to decode workers when slack exists. 5) for partitioning prefill/decode resources under global GPU constraints and the insight that multi-agent workflows create a unique 'incremental prefill' bottleneck that standard disaggregation handles poorly.
Same pipeline, pointed at whatever you're working on — with relevance scored to how you read.
Request access →