- Comparing routers requires a dataset of prompts with the ground-truth performance of every candidate model on each prompt
- Early routing papers borrowed general LLM benchmarks — MT-Bench, MMLU, GSM8K — as a stand-in for routing quality
- Purpose-built router benchmarks followed, scoring many models against many prompts so routers can be compared apples-to-apples (e.g. RouterBench, LLMRouterBench)
- Still unresolved: benchmarks are a fixed snapshot, but the model lineup keeps changing and a benchmark’s prompt mix rarely matches a given production workload
- This gap between benchmark results and real deployments is a big reason the “gap to oracle” problem (next section) persists