Setting the stage

Introduction

Problem statement: given a choice of multiple LLMs with different capabilities and costs, route my prompt to the best LLM
As people started using LLMs more widely and paying for inference, smarter choices of models became attractive
LLM routing seems to be of wide interest
There are numerous tools available
However, this is not close to a solved problem. The ability of LLM routers today leaves a large gap to what’s theoretically possible.