Construct a Jev-style decision model using an LLM

Meta description: Use logit bias and logprobs in TensorRT Edge-LLM to turn LLMs into single-pass decision models, with rules, limits, and checks for failure cases.

Tags: LLM routing, TensorRT Edge-LLM, logit bias, logprobs, agentic systems

Featured image: Abstract illustration of labels, score bars, and one arrow; no people, logos, or text.

For teams building agentic systems on TensorRT Edge-LLM, this method creates a single-pass decision model: one prompt read, one score set, no generated text. It uses logit bias and logprobs to make an ordinary LLM score fixed labels such as tool names, categories, or routing targets. It needs no training, and for builders it can reduce the repeated compute cost of text generation.

Why token-by-token choices are slow

Agentic systems choose from lists known ahead of time. They decide which tool to call, which category a request falls into, or where a message should go. An LLM is still a text generator. It emits one token at a time, and each token needs another model pass. Even a structured answer such as {"tool": "search_docs"} takes around ten tokens, so a simple choice pays for ten sequential passes.

A system-one decision model avoids that cost. The name borrows from Kahneman's two systems: system one answers at a glance, while system two reasons step by step. A Jev-style decision model works the first way. It receives the prompt and allowed answers, reads the prompt once, and returns a score for each answer. That reading pass is the prefill, and it is the only computation.

From logits to a label-only decision

An LLM assigns a raw score, or logit, to every vocabulary token. Larger logits make selection more likely. It then samples from a probability distribution made by the softmax: exponentiate each logit, then divide by the sum of all exponentiated logits.

Real vocabularies are around 150,000 tokens, so rare tokens can receive probabilities near 10^-44. Those values are too small for reliable computer arithmetic, so the model uses logprobs, or log probabilities. Taking the logarithm turns 10^-44 into about -101. In log space, a product of several token probabilities becomes a sum, which is easier and more stable.

The log of the softmax turns each logprob into the original score minus one shared number. In a toy vocabulary of four tokens, "The", "A", "B", and "C", with scores 3, 2, 1, and 0, the exponentiated scores sum to 31.19, and its log is 3.440. Each logprob is score minus 3.440, so gaps between scores and gaps between logprobs are the same.

A decision model only cares about allowed labels. If a ticket must go to A, B, or C, those are labels, while "The" stands for everything else. The desired value is a softmax over the labels only: q_i = exp(z_i) divided by the sum over labels exp(z_j). With toy scores, A, B, and C are e^2, e^1, and e^0, or 7.39, 2.72, and 1.00. They sum to 11.11, so q(A) = 0.665, q(B) = 0.245, and q(C) = 0.090. The same number appears from ordinary probabilities: 0.237 / (0.237 + 0.087 + 0.032) = 0.665. q is a conditional probability, not a correctness probability. It can hide weak label mass: in the toy, the labels together received only 0.356, while "The" received 0.644.

Bring labels into the window with one shared head start

Computing q needs the labels' scores or logprobs. Servers return logprobs only for the top N likely tokens, not all 150,000. In the toy, N = 3 returns "The", A, and B, so C is invisible. In a real model, labels might rank 21st, 11th, and 23rd, with words like "The", "I", or "Sure" above them. A larger window helps only if the caller knows how deep the labels sit, and the runtime caps it at 50.

The fix is to add the same large number, c, to every label score and leave other tokens alone. Most servers expose this as logit bias. With c = 100, labels jump above every other token, so a window equal to the label count returns only the labels. The decision is undistorted because softmax cares about gaps. A's lead over B is 2 - 1 = 1 before bias and 102 - 101 = 1 after bias. Adding c multiplies every label term by exp(c), and that common factor cancels.

The head start must lift the weakest label above the strongest non-label. In the toy, any c above 3 works because the best non-label is 3 and the weakest label is 0. In real models, labels can sit 20 or more points below the top token. The runtime allows a bias of at most 100, so the trick works only while that gap stays under 100.

With c = 10, the exponentiated scores become e^12 + e^11 + e^10 + e^3 = 244,675, and its log is 12.408. The top three tokens are A, B, and C, and the returned logprobs read 0.665, 0.245, and 0.090, matching the label-only softmax. The server returns biased scores minus 12.408; exponentiating them and dividing by their sum cancels the shared bias.

The returned values may not sum to 1 because a small probability, epsilon, can remain on non-labels. With c = 10, epsilon is 20.09 / 244,675 = 0.00008, so labels sum to 0.99992. With c = 100, epsilon is about 7 x 10^-44, which a 32-bit float cannot register, so the label sum is exactly 1.0 and the returned logprobs already equal q. Renormalization still matters because servers may print limited digits and show 0.999998, because a clipped bias still gives a usable sum, and because subtracting the largest logprob before exponentiating protects against underflow. Overflow is not a risk here because logprobs are never above 0.

Run it on TensorRT Edge-LLM

TensorRT Edge-LLM has no built-in scoring endpoint, but it provides per-token logit bias and top-K logprobs. Label token ids must come from the Hugging Face tokenizer of the exported model. Runtime behavior changed between releases, so versions matter.

The parameters are separate: top_k decides which token is sampled and emitted, while top_logprobs decides how many candidates are reported. The decision scores come from top_logprobs. The trick works because the runtime adds the bias after the prompt read, picks the next token, and then computes logprobs from the biased scores.

Before exponentiating, the log-softmax subtracts the largest score, so a biased score near 100 never becomes e^100, which would exceed a 32-bit float. Logprobs are computed at temperature 1 regardless of sampling temperature. Scores are kept in 32-bit floats. The limits are firm: bias must stay between -100 and 100, and at most 50 logprobs come back per step.

For a local check, compare the renormalized output with a softmax computed directly from the Hugging Face model's scores on the same prompts.

Use these rules:

1. Use single-token labels such as A, B, C or 1, 2, 3, and explain them in the prompt. Multi-token labels need a different method.
2. End the prompt where the answer goes, such as Label:. With a chat template, tokenize labels without a leading space because the assistant turn starts after the prefix.
3. For Qwen3, keep enable_thinking = False, otherwise the model may start by thinking instead of answering.
4. Match tokenizer and vocabulary. Label ids must exist in the exported engine, including reduced-vocabulary engines.
5. Use at most 50 labels because of the logprob cap. On v0.10.0, disable speculative decoding because it rejects logit_bias. From v0.10.1, the two work together.
6. Add an escape label such as OTHER or ESCALATE so the model has a valid place for probability when no real choice fits.

Common mistakes

The biggest mistake is reading q as proof that a label is correct. The method gives a conditional preference. If the model never wanted any label, the bias can still lift labels into the window. The toy example shows the risk: q(A) = 0.665 can come from total label mass 0.356.

Other common mistakes are:

• Choosing a bias too small to lift the weakest label above the strongest non-label.
• Forgetting the top-N window. If any label is missing from the returned logprobs, q cannot be reconstructed.
• Exceeding 50 labels or exceeding the -100 to 100 bias range.
• Using ids from the wrong tokenizer or a vocabulary that lacks the label.
• Using multi-token labels as if they were one-token answers.
• Skipping renormalization when server output rounds, clips, or leaves residual probability on non-labels.
• Ignoring release rules, especially speculative decoding on v0.10.0 and enable_thinking = False for Qwen3.