- Simplest workflow is to make a prior decision, selecting which LLM
- Expensive workflow step is to run multiple LLMs in parallel, when the value of a good answer is very high
- Consensus voting by the same LLM (e.g. best of 5) is a form of this
- Can integrate routing decisions while generating — e.g., use low model confidence to escalate to a costlier LLM
- Can pipeline a verifier/evaluator after generation to decide whether to accept response or escalate
- Escalation policies can use a wide variety of techniques, but simplest is redo prompt with a costlier LLM