Tutorials

Apply decision models in Amazon Bedrock AgentCore Evaluations

AWS sample uses Jev and Strands Decider 2B in AgentCore Evaluations for calibrated AI agent scoring.

Key Takeaways

  • Amazon Bedrock AgentCore Evaluations supports decision model backends, such as Jev and Strands Decider 2B, through code-based evaluators for calibrated agent scoring.
  • Decision models are best for narrow, repeatable evaluation checks because they return probability-based labels instead of free-form text with variable confidence.
  • The AWS sample shows how to use these models during development, batch testing, and online evaluation, including groundedness, helpfulness, and goal completion.
  • The sample uses an HR Assistant agent and deterministic Lambda evaluators to demonstrate how decision models can work alongside built-in evaluators and LLM-as-a-judge checks.

Meta description: AWS sample shows how Amazon Bedrock AgentCore Evaluations can use Jev and Strands Decider 2B for calibrated agent scoring in development and production.

Apply decision models in Amazon Bedrock AgentCore Evaluations

Tags: Amazon Bedrock AgentCore, AI agent evaluation, decision models, AWS Lambda, LLM-as-a-judge

Featured image: abstract friendly illustration of calibrated decision points feeding an agent evaluation pipeline, no real people, no logos, no text.

For teams that evaluate AI agents, Amazon Bedrock AgentCore Evaluations can use decision model backends to return calibrated labels for narrow checks. A published AWS sample shows how to connect Jev, a hosted decision model from TypeSafe, and Strands Decider 2B, an open-source 2-billion-parameter model from the Strands Agents team, through code-based evaluators. The sample gives a way to score groundedness, helpfulness, and goal completion during development, batch testing, and online evaluation.

Why decision models matter

Some agent evaluation questions are narrow and repeatable: did a response stay on policy, is a tool call safe to approve, or does an answer match retrieved context? Decision models take a structured state and a fixed set of allowed answers, then return a calibrated probability instead of free-form text whose confidence can drift from run to run. That produces a label and a number that teams can threshold and trust.

A general-purpose large language model (LLM) evaluator usually returns a score with a rationale. A decision model returns a probability distribution over predefined answers. That direct confidence measure can be aggregated and acted on programmatically. Jev and Strands Decider 2B support three question types. A noul question asks yes or no and returns a probability between 0 (no) and 1 (yes). A choice question asks which of up to 255 options applies and returns the most probable option, the full distribution, and a confidence. A score question asks which of 2 to 10 ordered levels applies and returns a probability-weighted score, the distribution over levels, and a confidence.

These models fit questions an informed person could decide at a glance, such as whether the agent refused a request or whether a response stayed on topic. They are most useful when teams want a well-calibrated confidence score alongside the label, and when they want to enforce a specific rubric consistently without asking a general-purpose LLM to interpret it. They are not a replacement for LLM evaluators. They are an addition enabled by the code-based evaluator framework, best used for narrow, well-defined checks alongside broader built-in assessments.

How AgentCore Evaluations connects them

Amazon Bedrock AgentCore Evaluations scores an agent through its lifecycle: on demand during development, in batch jobs against stored sessions, and online against production traffic. Alongside 15 built-in evaluators, it supports custom evaluators defined as either an LLM-as-a-judge or a code-based evaluator. A custom LLM-as-a-judge evaluator applies instructions and a rating scale with a model in Amazon Bedrock. A code-based evaluator runs custom logic in an AWS Lambda function, and that logic can call an external system, model, or API, including a decision model.

The sample uses both backends as a screening layer. Because each backend can answer several questions about the same state in a single request, teams can run pass or fail signals across metrics and select the turns that need deeper assessment by built-in evaluators, whose explanations help investigate what went wrong. After a question is validated on real traffic, a decision model can be sufficient for that decision or score, and the same approach can extend to specific harmfulness or stereotyping categories in a policy.

What the sample builds and requires

The end-to-end example uses an HR Assistant agent built with Strands Agents and deployed to AgentCore runtime, a capability of Amazon Bedrock AgentCore. The agent answers employee questions using five tools that return fixed mock data: paid time off balances, paid time off requests, HR policies, benefits, and pay stubs. Fixed data makes it straightforward to verify evaluator behavior from LLM-as-a-judge and decision models. When teams also have ground truth, they can confirm whether each evaluator catches the right failures.

To deploy the sample, an AWS account must be in a Region where AgentCore Evaluations is available; the sample defaults to US West (Oregon). It requires Amazon CloudWatch Transaction Search enabled, access to a foundation model in Amazon Bedrock, Python 3.12 or later, the AWS Command Line Interface (AWS CLI), and Docker. Jev requires a TypeSafe API key, with access details at typesafe.ai. Strands Decider requires a host running the Strands Decider server, either locally for testing or on Amazon Elastic Compute Cloud (Amazon EC2) inside a virtual private cloud so the Lambda function can reach it.

How to run and validate the evaluators

The full code sample is in the AgentCore samples GitHub repository. Two Python scripts handle the workflow: ../utils/deploy.py deploys the HR Assistant to AgentCore runtime, and evaluate.py deploys evaluators, invokes the agent, and runs evaluation modes.

Running evaluate.py without flags executes five steps:

  1. Creates the Lambda execution role AgentCoreLambdaEvaluatorRole.
  2. Packages and deploys two deterministic Lambda evaluators: hr-response-length, a trace-level evaluator that checks whether the agent's response stays within length bounds, and hr-fact-checker, a session-level evaluator that regex-validates HR facts against the known mock data.
  3. Registers both as AgentCore code-based evaluators through the control plane.
  4. Runs on-demand evaluation with EvaluationClient and OnDemandEvaluationDatasetRunner across five labeled scenarios, combining the code-based evaluators with Builtin.Correctness, Builtin.Helpfulness, and Builtin.ResponseRelevance.
  5. Creates an online evaluation configuration that continuously scores live sessions at 100 percent sampling.

Decision-model evaluators are opt-in and run alongside the base evaluators with no changes to the existing flow. Using the flags --with-jev and --jev-secret-arn with the secret's Amazon Resource Name deploys jev-evaluator and registers JevGroundedness, JevHelpfulness, and JevGoalCompletion. Using --with-decider deploys strands-decider-evaluator and registers the same three evaluators backed by Strands Decider 2B. Adding --decider-ec2 auto-provisions the Amazon EC2 instance, security groups, Identity and Access Management role, and Lambda virtual private cloud configuration without manual console steps.

Each backend contributes three evaluators:

EvaluatorLevelQuestion typeLabelValue
JevGroundedness or DeciderGroundednessTRACEnoulGrounded or UngroundedP(grounded)
JevHelpfulness or DeciderHelpfulnessTRACEscore (4-level)Most probable levelExpected level, scaled 0 to 1
JevGoalCompletion or DeciderGoalCompletionSESSIONchoice (3-way)Most probable optionProbability-weighted value

Inside the evaluator Lambda, the function receives OpenTelemetry spans of the agent. The spans.py module normalizes the shapes OpenTelemetry content can arrive in, then formats them for the decision model call. For trace-level evaluation, only the targeted turn and brief prior history are sent; for session-level evaluation, the full conversation is included. The adapter converts the model's probability distribution into an AgentCore evaluation result. For a noul question, the probability of true becomes the value and the most probable answer becomes the label. For a score question, the value is the probability-weighted expected level scaled to [0, 1]. For a choice question, it is the weighted average of predefined option values.

Validation is the critical step. When a rubric is rewritten as a decision model question, the result is a new measurement rather than a copy of the old one. A reframed rubric often lands stricter or looser than intended, and that drift may not be visible until it is checked against real sessions. The sample describes a five-step validation procedure, beginning with building a labeled sample from stored sessions that covers failures and edge cases, not just the happy path. It then recommends getting human reference labels from three independent labelers, taking majority vote, and tracking unanimous versus split votes, because split-vote turns reveal ambiguity in the rubric itself before the model sees it. The next step is to run the decision-model evaluator and the matching built-in evaluator over the same inputs.

Common mistakes

The main mistake is treating a decision model as a drop-in replacement for a built-in LLM evaluator. Decision models return labels and probabilities, not explanations, so they work best as a screening layer for narrow, repeatable checks. Another mistake is skipping validation on a labeled sample of real traffic; a question that sounds equivalent to an old rubric can become stricter or looser in practice. Teams should also avoid sending full conversations to trace-level checks when only the targeted turn and brief prior history are needed, and they should not gate a release on a new decision-model evaluator until they have compared it against human labels and the matching built-in evaluator. For teams that already use AgentCore Evaluations, the sample provides a practical path to add calibrated decision-model scores while keeping broader LLM-as-a-judge assessment for the cases that need explanation.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.