- As the goal of LLM routing has become more complex, more factors can be included in the metric we’re trying to optimize
- For simple one-shot routing, basic metric to minimize is inference cost
- Can get a little more sophisticated and add penalty for inference time required
- Evolve to modeling the human/business cost to getting a task done, including verification, retries, etc.
- Incremental Value = (Increased probability of success * Value of success) - Incremental Task Cost
- The net change in cost is increased inference fees less reduced human effort (such as verification)