Large Language Model Routing with Benchmark Datasets (Shnitzer et al., IBM, NeurIPS 2023 workshop / COLM 2024) — reframes routing as reusing existing benchmark datasets to train a router via binary classification, without needing labeled data for the new task.
Taxonomy: decision = difficulty (classifier trained on benchmark data) · workflow = single choice · offline learning · also a benchmarks storyLLM-Blender (Jiang, Ren & Lin, 2023) — an ensembling/routing hybrid: pairwise ranking across candidate LLM outputs plus generative fusion of the top ones.
Taxonomy: workflow = run several models in parallel and aggregate · decision = user preference (pairwise ranking of outputs) · offlineAutoMix (Madaan et al., 2023) — few-shot self-verification: a small model answers, verifies its own confidence, and a POMDP-style policy decides whether to escalate.
Taxonomy: workflow = cascade with a verifier and escalation policy · decision = uncertainty (self-verification) · statelessHybridLLM (Ding et al., Microsoft, ICLR 2024) — trains a DeBERTa-based router to predict query difficulty and route to the right-sized model to meet a target quality bar. Easy to test because it's a clean binary/threshold decision.
Taxonomy: decision = difficulty · workflow = single choice · metrics = minimum cost at a target quality bar · offlineRouterBench: A Benchmark for Multi-LLM Routing System — addresses an important problem: how do you actually compare routers? RouterBench provides more than 405K inference outcomes from representative LLMs, a routing evaluation framework, and a theoretical formulation of the routing problem.
Taxonomy: benchmarks · supplies the per-model ground truth the gap-to-oracle analysis needs