- LLM routing has moved from a simple “pick the cheapest model that works” idea to a full discipline: goals, metrics, workflows, statefulness, learning, and evaluation are all now active design choices
- Production systems have followed: routing now ships as infrastructure (LLMRouter) and as part of the inference stack itself (vLLM Semantic Router), not just a research idea
- But the field has not solved the problem: diverse router designs converge to a similar accuracy ceiling well below the oracle, and that gap has not meaningfully closed as more approaches are tried
- What’s next: better evaluators may matter more than better routers, benchmarks need to track real production traffic and agentic workflows, and with the rapid pace of new model releases, routers that can handle new models without full retraining will be preferred
“The more I look at modern routing systems, the more I am convinced that the evaluator, rather than the router, is where the real competitive advantage is accumulating” — Pratik Bhavsar, Pratik’s Pakodas