|
2026-04-14 | Brown University, Fidelity Investments | 2604.12955
This paper introduces Text2Zinc, a solver-agnostic benchmark of 1,775 natural language combinatorial problems and evaluates various LLM copilot strategies (CoT, knowledge graphs, grammar validation) for generating formal MiniZinc models. The results are backed by extensive empirical evaluation, demonstrating that even advanced models like GPT-4o struggle, achieving only ~40-50% solution accuracy despite much higher execution (compilation) accuracy. The key insight is that decoupling syntax enforcement from generation via post-hoc grammar validation significantly improves execution accuracy without requiring constrained decoding, though capturing the underlying optimization logic remains a major bottleneck.
|