Beginner Basics

Massive multitask decision making is measured.

MMDM evaluates decision models on 5,448 questions, 16 tasks, long contexts, and 1,024 options, revealing weak logic, constraints, and multi-select.

Massive multitask decision making is measured.

Key Takeaways

  • MMDM is a new 5,448-question benchmark across 16 task categories designed to evaluate System-1 Decision Models in long-context, large-option, and multi-select decision settings.
  • The benchmark tests direct structured probabilistic decision making, where models choose among defined options in a single forward pass without generating intermediate reasoning or textual responses.
  • It includes fresh, source-grounded tasks such as agent state tracking, tool and skill selection, preference judgments, and rubric scoring, with option spaces up to 1,024 choices and inputs up to 64,817 tokens.
  • Evaluation of 28 decision models shows strong performance on common knowledge but major drops on option logic, constraint planning, and complete recovery of multiple correct answers; the best Hard Multi-select Accuracy was 36.81%.
  • The results highlight three development priorities: joint reasoning over interdependent options, constraint-aware selection, and full multi-select answer recovery, suggesting that world knowledge alone is insufficient for complex decision making.

Meta description: MMDM is a 5,448-question benchmark across 16 tasks that tests System-1 Decision Models on long contexts, large option spaces, and multi-select choices.

Massive multitask decision making is measured.

Tags: MMDM benchmark, decision models, multi-select tasks, model evaluation

![Abstract decision nodes and large option lists, no people, logos, or text](abstract-decision-benchmark.png)

This article is for researchers and builders working with System-1 Decision Models. The MMDM benchmark gives them a clearer way to test whether these models can make structured probabilistic decisions when inputs are long, options are numerous, and decisions depend on logic or constraints.

Massive multitask decision making is measured.

System-1 Decision Models directly choose among query-specified options given a state in a single forward pass. In practice, that means the model produces its choice from the current state without writing out intermediate reasoning steps or textual responses. This approach is relevant for agent state tracking, tool and skill selection, preference judgments, and rubric scoring, where the task is to pick among defined choices.

Why the benchmark was created

Recent evaluations covered diverse tasks, but many relied on existing benchmarks. When test material has been seen before, it can obscure how well a model generalizes to fresh contexts. Real-world decision making also combines three hard features: long contexts, large option spaces, and complex logical dependencies. Existing benchmarks did not adequately capture those features together.

What MMDM tests

MMDM, short for Massive Multitask Decision Making, contains 5,448 high-quality questions across 16 task categories. It combines newly constructed tasks with source-grounded recompositions and rewrites. The tasks include single-select, multi-select, and soft-label decision making in fresh contexts.

Scenarios include agent state tracking, tool and skill selection, preference judgments, and rubric scoring. The option spaces range from 2 to 1,024 choices, and inputs extend to 64,817 tokens. That makes the benchmark useful for systems that must compare many possibilities while holding detailed context in mind.

Massive multitask decision making is measured.

How models performed

The study conducted a comprehensive quantitative evaluation of 28 decision models. Many models performed well on common knowledge, but their accuracy fell when options depended on each other, when decisions had constraints, or when multiple correct choices had to be recovered completely.

For example, Perplexity Decider-V1.2 reached 78.75% Hard Single-select Accuracy on world knowledge. It scored only 53.95% on option logic and 47.95% on constraint planning. Across all evaluated models, the best Hard Multi-select Accuracy was 36.81%.

What the results mean

The findings point to three priorities for decision model development: joint reasoning over interdependent options, constraint-aware selection, and complete recovery of multi-select answers. MMDM provides a testbed for measuring progress on those issues. For readers building agents or evaluation pipelines, the benchmark suggests that strong world knowledge alone is not enough; the harder test is choosing correctly inside large, structured, context-heavy decision spaces.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.