Case study

FrugalGPT

“FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance” — Chen et al. (2023), arXiv:2305.05176

Three strategies:

Prompt adaptation (reduce few-shot examples or combine queries)
LLM approximation (cache or fine-tuned smaller model)
LLM cascade

Diagram of an LLM cascade: a query is sent to GPT-J, then GPT-3, then GPT-4 in sequence, accepting the answer as soon as a scoring function is confident enough. — (e) LLM cascade — escalate through GPT-J → GPT-3 → GPT-4 only as needed

FrugalGPT’s cost-performance profile not only allowed less expensive inference, but even achieved better performance than the largest model. Results for CoQA on reading comprehension:

Scatter plot of accuracy versus cost in dollars for individual LLMs (CHATGPT, GPT-J, J1, GPT-Neo, FQ, CoHere, GPT-3, GPT-4) compared against the FrugalGPT cost-accuracy curve, which reaches higher accuracy than GPT-4 at a fraction of the cost. — FrugalGPT’s curve dominates every individual model on the cost/accuracy trade-off