Tool Reviews

A human-grounded granular evaluation for proactive agents is proposed.

HALO adds human annotations to evaluate proactive agents, finding over-proactivity in timing and intensity is the main failure mode.

A human-grounded granular evaluation for proactive agents is proposed.

Key Takeaways

  • HALO introduces a human-grounded, granular evaluation framework for proactive agents that must decide whether, when, and how much to help without an explicit request.
  • HALO-Arena tests agents across 4 proactive need types, 10 everyday domains, and 50 personal environments, requiring evidence retrieval from structured or unstructured sources before intervention.
  • HALO-8K contains 8,000 expert-annotated responses from 10 models across 400 scenarios, recording acceptability, action level, timing, intensity, reasoning tokens, and retrieval rounds.
  • The main failure mode is over-proactivity: premature timing or excessive intensity account for 61.20% of rejected responses, showing that acting too early or too much is a bigger problem than simply failing to act.
  • Top models still leave room for improvement, with the best proprietary model reaching 72.88% acceptability and the best open-source model reaching 56.63%, highlighting the need for more calibrated, context-aware proactive assistance.

Meta description: HALO-Arena and HALO-8K add human annotations to evaluate proactive agents, revealing over-proactivity as the main failure mode in 8,000 responses.

A human-grounded granular evaluation for proactive agents is proposed.

Tags: proactive agents, AI evaluation, HALO, benchmarks

Featured image: abstract friendly illustration of nodes representing agents reviewing evidence from structured databases and unstructured documents, with timing and intensity gauges; no people, logos, or text.

A new framework called HALO, Human-Grounded GrAnular EvaLuation of PrOactive Agents, is designed to test proactive agents in a more human way. It focuses on a problem that is easy to describe and hard to score: an assistant may decide whether, when, and how far to help without an explicit request. Existing evaluations often rely on automated scores or binary verdicts, which can say whether a response was accepted but not why. HALO adds human diagnosis to evidence use and intervention quality.

What HALO adds

The framework has two main parts. HALO-Arena is the test environment. It covers 4 proactive need types: Dependency, Blockage, Risk, and Opportunity. These scenarios are spread across 10 everyday domains and 50 personal environments. The tasks are built so agents cannot simply guess whether to help. They must retrieve evidence from structured databases or unstructured documents before judging whether assistance is warranted and choosing its scope.

A human-grounded granular evaluation for proactive agents is proposed.

The second part is HALO-8K, a dataset built from the arena. It contains 8,000 responses from 10 models across 400 scenarios. The responses are annotated by 20 experts. The annotations record action level, acceptability verdicts, and human diagnosis. For rejected responses, the diagnosis covers timing, intensity, and other issues. The framework also records reasoning tokens and retrieval rounds, which measure the effort behind each response.

Why the scoring gap matters

The headline numbers are straightforward. The best proprietary model reaches 72.88% acceptability. The best open-source model reaches 56.63%. Those scores show that even top systems leave room for improvement when they are asked to act before being asked. More important is the reason for the failures. HALO’s central finding is that over-proactivity is the dominant observed failure mode. Premature timing or Excessive intensity account for 61.20% of rejected responses.

This matters for readers because proactive agents are meant to feel helpful, not intrusive. If an agent acts too early, it may jump in before the user actually needs help. If it acts too intensely, it may do too much or go beyond what the situation calls for. A single acceptability score can hide those errors. HALO’s human annotations make them visible.

Evidence first, action second

Another part of the design is the evidence step. Many assistant tasks are judged only by the final answer. HALO-Arena asks agents to retrieve evidence first, from structured databases or unstructured documents, and then decide whether assistance is warranted. This tests a different skill: not only whether the model can produce a plausible response, but whether it can use available information to judge the situation.

A human-grounded granular evaluation for proactive agents is proposed.

That is useful for everyday environments, where the right level of help depends on context. A response that seems helpful in one personal environment may be too active in another. By covering 10 everyday domains and 50 personal environments, the framework tests whether agents can adapt rather than repeat one default behavior.

What the further analyses show

The dataset and arena also support complementary analyses. They look at the balance between intervention and restraint, evidence acquisition, and the calibration of timing and intensity. Together, these findings suggest that the challenge is not simply whether a model should act. The challenge is whether it can match action to need, evidence to context, and intensity to the moment.

For developers, the value is diagnostic. Instead of only knowing that a model failed, a team can see whether the failure was early, too strong, poorly evidenced, or misjudged. For users, the value is practical: a better evaluation could lead to assistants that are more careful about when they step in.

What comes next

HALO does not remove the need for broader evaluation. It addresses a specific gap: human-grounded, granular testing of proactive agents. The recorded reasoning tokens and retrieval rounds, plus expert annotations, turn rejection into something that can be studied. The numbers show progress, but also a clear problem. As proactive agents move closer to everyday tasks, the next step will be to reduce over-proactivity without making assistants too passive to be useful.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.