Tool Reviews

A Terminal-Style Mathematical Modeling Benchmark Is the First Agentic and Verifiable Benchmark.

TB-MathModel tests AI agents on real mathematical modeling via terminal tasks, formal checks, and contest data, revealing capability and cost gaps.

A Terminal-Style Mathematical Modeling Benchmark Is the First Agentic and Verifiable Benchmark.

Key Points

  • TB-MathModel is a terminal-style benchmark that tests whether LLM agents can handle messy, real-world mathematical modeling tasks rather than neatly posed exercises.
  • It uses CUMCM/MCM contest problems inside containerized terminals, requiring agents to read data, analyze figures, write code, run solvers, and output structured JSON results.
  • The benchmark scores numeric answers with direction-aware evaluation, multiple human expert references, and formal feasibility checks using Lean 4 to enforce logical and physical constraints.
  • Early results show GPT-6 Astra at 64.6%, Claude Fable 5.1 at 63.1%, and a low-cost open model at 59.8% for 1/100 the price, suggesting reliable modeling is still unsolved.
  • TB-MathModel points to a future direction for agent evaluation: uniform environments, verifiable rewards, and growing trajectory corpora that force models to make assumptions and produce testable outputs.

A new benchmark for mathematical modeling, TB-MathModel, is giving developers and AI researchers a clearer way to judge whether large language model agents can handle messy real-world tasks rather than neatly written exercises. The first published scores are not flattering to the leading models: GPT-6 Astra reaches 64.6 percent, Claude Fable 5.1 reaches 63.1 percent, and an open model costing $2.54 reaches 59.8 percent at 1/100 the cost. The project releases code and data at https://github.com/wxj630/terminal-bench-math-modeling.

A Terminal-Style Mathematical Modeling Benchmark Is the First Agentic and Verifiable Benchmark.

Why mathematical modeling is a hard test for agents

Today's LLM agents are strong when the task is already well posed. If the user asks them to write code, solve a math problem, or call an API, they can usually find a path. Real work rarely arrives that way. A business question, an engineering issue, or a policy analysis often lacks defined variables, a chosen model form, clean data, and a clear way to check the answer.

Mathematical modeling is the layer between a messy situation and a computable question. It asks an agent to decide what matters, make assumptions, build a model, test it, and produce a result that can survive review. That is why the benchmark matters: it measures the step that separates automated coding help from practical problem solving.

How TB-MathModel turns contest problems into real work

TB-MathModel uses the China Undergraduate Mathematical Contest in Modeling and the Mathematical Contest in Modeling, known as CUMCM/MCM, as its entry point. The reason is not contest promotion. These problems are public, complete, unstructured real-world tasks with genuine attachments, official figures, and human expert solutions. That makes them useful as a test corpus.

Eighteen problems are placed inside containerized terminals. The model receives a task in a miniature environment: read the statement, open spreadsheets, look at figures, write code, run the solver, and emit one structured JSON file, a common machine-readable format. That workflow is close to a real modeling job, compressed into a reproducible computer task.

Why the scoring checks numbers, not writing quality

The benchmark does not grade the report. It checks the numeric core of the final question. The scoring is direction-aware and one-sided: if a model beats the human paper on that numeric result, it never loses points for doing better. That avoids punishing a model for finding a stronger answer than the human reference.

The reference layer is also wider than a single answer. Because modeling has no unique solution, TB-MathModel uses 94 reference answers. Every Outstanding-award paper gets a vote, so a model must land close to many human experts at once rather than memorize one official answer.

Feasibility is checked separately. A programmatic gate examines six decidable invariant forms, and 624 kernel-checked Lean 4 obligations are used to verify constraints. Lean 4 is a formal proof assistant, so these checks are machine-readable. The benchmark avoids Mathlib and avoids sorry, a placeholder that would weaken a formal proof. In plain terms, a number is not enough. The result must pass formal checks that keep the model honest about physical and logical constraints.

What the scores say about capability and cost

The result shows a gap between fluent AI output and reliable real-world modeling. GPT-6 Astra, the best entry, reaches 64.6 percent. Claude Fable 5.1 reaches 63.1 percent. The open model priced at $2.54 reaches 59.8 percent at 1/100 the cost.

For readers who care about AI deployment, the cost point is important. The leading entries still score only modestly above the cheap open model. The benchmark does not say which model is better for every task, but it suggests that mathematical modeling is not yet a solved problem, even when a model can write code and explain an answer well.

What comes next for agent evaluation

TB-MathModel is aimed at a missing layer in agent evaluation. The project says the road to useful real-world agents needs a uniform environment, a verifiable reward, and an agent-trajectory corpus that compounds over time. In other words, better agents may not come only from better prompts or larger reports. They may come from evaluation settings that force models to make assumptions, produce testable outputs, and learn from repeated, machine-checkable work.

The code and data are available at https://github.com/wxj630/terminal-bench-math-modeling. For now, the message is straightforward: GPT-6 Astra and Claude Fable 5.1 are capable, but they still need help crossing the line from solving clean exercises to doing messy real work.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.