The LLM routing-ladder recipe (order, fallbacks, token ceilings)
I cut twelve seconds of AI latency by reordering a list. My gateway routed Groq-first, and Groq had a ~12s stall mode under load; flipping to Cerebras-first killed it. Then I got greedy and ADDED two more providers to the ladder… and it got slower. Removed them the same week: dropping the two extra rungs was itself the fix, because all they added was latency. Also learned the hard way: token ceilings 502'd my long requests until I raised them 1500 → 4000.
The recipe
- Structure routing as an explicit ordered ladder. Primary → failover, declared in config, not implied by SDK defaults:
// via an AI gateway (mine: Vercel AI Gateway)
providerOptions: {
gateway: { order: ['cerebras', 'groq'], only: ['cerebras', 'groq'] }
}
// plus a hard fallback outside the gateway: gateway → direct API → last-resort model
- Order by YOUR measured tail latency, not by benchmark headlines. Groq's average was fine; its p95 had a stall mode that averages hide. Rank ladder rungs by p95/p99 on your actual prompts and payload sizes.
- Shorter ladders beat longer ones. Every added rung is another failure mode, another latency distribution, another dialect of output quirks. Two good providers beat four okay ones. I have the revert commit to prove it.
- Set token ceilings from your real outputs, then add headroom. A
max_tokensof 1500 hard-failed (502) my longer requests. Truncation limits don't degrade gracefully in structured-output pipelines, they break parsing. Measure your p99 output length; set the ceiling comfortably above it (mine: 4000). - Verify structured output survives each rung. Before adding a provider, probe that your strict schema/enum enforcement holds through the gateway on THAT provider. I ran a 40-request strict-schema probe: 0 leaks. That's the bar.
- Tag every call with its resolved provider. Traces and errors must record which rung actually served the request, or ladder debugging is astrology. Latency dashboards sliced by provider is how the 12s stall became visible at all.
- Re-measure on a schedule. Provider performance drifts week to week (capacity, model updates). The ladder is a tuned system, not a set-and-forget config. Mine changed three times in one month, each time from data.
- Keep failover semantics idempotent. If rung 1 times out after partially processing, rung 2 must be safe to run: idempotency keys on any downstream side effects.
Steal this for your app
- The cheapest latency wins in AI apps are config-shaped: provider order, token ceilings, timeouts. Exhaust those before rewriting prompts or upgrading models.
- Treat provider lists like database indexes: additions feel free but each one has a maintenance cost, and the burden of proof is on adding, not removing.
- "Tune with data, not vibes" means having the data: if you can't chart p95 by provider today, that's the first task.
Run this on your codebase
Paste this into Claude Code in your repo:
Audit this repo's LLM routing for latency traps.
Find where provider order is defined (gateway config, SDK defaults, env vars) and check it is an explicit ordered list, not implied.
Count the rungs in the fallback ladder and flag anything beyond two providers that has no measured justification.
Find every max_tokens or output ceiling and flag values that look set by guesswork rather than measured p99 output length plus headroom.
Check whether logs and traces record which provider actually served each request.
Check for idempotency keys on side effects downstream of any call that can fail over mid-request.
Report findings as a checklist before changing anything.
Route by task difficulty as well as provider
Provider order answers who serves a request. Model routing answers how much model the task needs. Do not send a classification or field extraction to the largest model by habit.
Build a fixed evaluation set per task family, then test candidates from smallest to largest. Use a table that keeps the decision inspectable:
| Task tier | Example | Quality gate | Selected model | Cost per successful case | p95 latency |
|---|---|---|---|---|---|
| Mechanical | classify, extract, normalize | strict schema plus task score | smallest passing | measured | measured |
| Mixed | draft from supplied evidence | factuality plus review score | smallest passing | measured | measured |
| Hard | ambiguous planning or multi-source judgment | task-specific rubric | model that earns it | measured | measured |
Choose the cheapest model that passes the same quality gate. "It is a simple task" is not a gate. Include ugly inputs, refusals, missing data, and the cases that previously failed.
Route confidence failures upward, not every request. Tag each call with task tier, selected model, provider, token count, latency, cost, and evaluation version. Re-run the matrix when models or provider behaviour change.
Keep the provider fallback ladder short inside each selected tier. A cheap primary with four slow fallbacks can cost more and fail later than one measured pair.