Prompts

An Ensemble Routing Method Makes LLM Inference Prompt-Adaptive and Cost-Aware.

PISA and PISA-E route LLM prompts and ensembles by cost and quality, lowering inference costs by 36% in-distribution and 25% out-of-distribution.

An Ensemble Routing Method Makes LLM Inference Prompt-Adaptive and Cost-Aware.

Key Takeaways

  • PISA is a cost-aware LLM router that selects a single model per prompt, while PISA-E extends routing to ensembles with prompt-adaptive model selection.
  • PISA-E decides which and how many models to call by estimating ensemble utility from individual model capability predictions, without ensemble-specific training.
  • On four routing benchmarks, PISA achieved the best cost-performance hypervolume among single-model routers, and PISA-E achieved the best quality-focused ensemble performance.
  • PISA-E can deliver higher performance at about 36% lower average inference cost in-distribution and 25% lower out-of-distribution compared with the strongest competing ensembles.
  • The practical takeaway is that LLM routing can be adaptive: a router can choose one model or a smaller ensemble based on the prompt and the user’s cost-quality preference.

PISA-E Offers LLM Routing With 36 and 25 Percent Lower Ensemble Costs

An Ensemble Routing Method Makes LLM Inference Prompt-Adaptive and Cost-Aware.

Meta description: PISA and PISA-E route LLM prompts and ensembles by cost, cutting average inference costs 36 percent in-distribution and 25 percent out-of-distribution.

Tags: LLM routing, PISA, PISA-E, ensembles, inference cost

Featured image: Abstract illustration of prompts moving to different LLM models and an ensemble panel with cost and quality dials, no text or logos.

An Ensemble Routing Method Makes LLM Inference Prompt-Adaptive and Cost-Aware.

This paper is for readers who use LLM routing to decide which large language model answers each prompt. It introduces PISA, a cost-aware single-model router, and PISA-E, which extends LLM routing to ensembles with prompt-adaptive composition and size.

Why routing and ensembles differ
Most LLM routing approaches select one model per prompt. Ensembling asks another question: which models should be combined, and how many are worth calling. More models can improve quality but raise inference cost, so the router must balance cost and quality for each prompt.

How PISA-E chooses models
PISA is a cost-aware single-model router. PISA-E estimates ensemble utility from individual model capability predictions, without requiring ensemble-specific training. It selects the models to invoke based on the prompt and a configurable cost-quality preference.

An Ensemble Routing Method Makes LLM Inference Prompt-Adaptive and Cost-Aware.

Benchmark results
The paper evaluates both methods across four routing benchmarks under in-distribution and out-of-distribution settings. PISA achieved the highest macro-average cost-performance hypervolume among the evaluated single-model routers. PISA-E achieved the highest macro-average ensemble task performance when prioritizing quality. It also offered operating points with higher performance at approximately 36 percent lower average inference cost in-distribution and 25 percent lower average inference cost out-of-distribution than the strongest competing ensembles.

What readers can take away
For teams running LLM applications, the practical idea is that routing does not have to be fixed. A router can choose one model or a smaller ensemble based on the prompt and the user's cost-quality setting. The paper invites feedback, especially on the ensemble-selection formulation and adaptive ensemble cardinality.

Who should use it / who should skip it
Readers evaluating LLM routers, ensemble methods, or cost-quality tradeoffs should use it. Readers looking only for model training recipes or final product benchmarks may skip it.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.