The provider-swap compatibility suite (null tests per model)

Every open-source model breaks your API in the same boring place. When I routed traffic to an open-weight model (Cerebras gpt-oss), it emitted explicit nulls where closed models simply omit optional fields. My schema's .default() values don't fire on null, so every voice expense 400'd and silently vanished from users' screens. The schema, not the prompt, is the real model-compatibility layer.

The one-line class of fix

// omission-tolerant only:
note: z.string().default("")
// null-AND-omission tolerant (what multi-provider actually requires):
note: z.string().nullish().transform(v => v ?? "")

The compatibility suite

Enumerate provider "dialects." Same JSON-mode request, different habits: explicit nulls vs omitted fields, numbers as strings ("80"), unrequested extra keys, enum case drift ("Food" vs "food"), varying refusal formats. Each provider you add is a dialect to cover.
Golden-output fixtures per provider. Capture REAL raw outputs from each model for your core tasks and check them into the repo. Your parse + schema layer must pass all of them:

fixtures/llm/
  gemini/expense-simple.json      cerebras-gpt-oss/expense-simple.json
  gemini/expense-no-note.json     cerebras-gpt-oss/expense-no-note.json  ← nulls live here

A null suite per endpoint. For every AI-writable schema, generate cases with each optional field: omitted, null, and empty-string/zero. Three states, three tests. An undefined passing tells you nothing about null.
Run the suite BEFORE flipping routing weights. Provider swaps ship via config in seconds; make the fixture suite the gate. New provider = capture fixtures → suite green → then ramp traffic.
Normalize at one boundary. All null-coercion and case-fixing lives in the schema/transform layer, never scattered "if null" patches in business logic. One place to audit when the next dialect arrives.
Alert on validation-failure rate by provider. A 400-rate spike scoped to one provider tag is your early warning that a dialect shifted (models get silently updated behind APIs).
Keep the failure loud for users. The compound bug was schema strictness + optimistic UI silently dropping the rejected write. Even with a perfect suite, a validation reject must surface to the user, not vanish.

Steal this for your app

If you support (or might ever support) more than one model, build the suite at provider #1. It's 10× cheaper than diagnosing vanishing data at provider #2.
Treat model swaps like DB migrations: staged, gated by tests, reversible by config.
The boring bugs are the portable ones: nulls, casing, stringly numbers. Nail those and "provider-agnostic" stops being a slide-deck word.

Run this on your codebase

Paste this into Claude Code in your repo:

Audit this repo's LLM output schemas for provider dialect breakage.
Find every schema that validates model output and list optional fields using .default() without .nullish(); those break on explicit nulls.
Check handling of the boring dialects: numbers as strings, unrequested extra keys, enum case drift, refusal formats.
Look for golden-output fixtures per provider checked into the repo; flag any provider with none.
For each AI-writable schema, check tests cover each optional field in three states: omitted, null, and empty or zero.
Verify null-coercion lives at one schema/transform boundary, not scattered "if null" patches in business logic.
Check validation failures are alerted per provider tag and surfaced to the user instead of silently dropped.
Report findings as a checklist before changing anything.

Constrain generation before validation

"Return JSON" is a writing instruction. The model can still add a sentence, miss a key, change a type, or stop halfway through. JSON mode usually guarantees valid JSON syntax, not your business schema.

Choose the strongest mode the provider supports:

Strict structured output: provide the JSON Schema and require a matching object.
Tool call: define typed arguments and let the model request that tool.
JSON mode: useful syntax protection, followed by full application validation.
Prompt-only JSON: last resort, with parsing failures treated as expected failures.

Every mode still needs the application boundary from this guide. Validate required keys, types, enums, additional keys, nulls, and domain rules before business code sees the value.

Keep refusal separate from malformed output. A provider may return a structured refusal outside the requested object. Do not coerce it into empty values and continue. Record the refusal, show the user an honest state, and skip downstream actions.

Capture raw fixtures for: valid output, omitted optional value, explicit null, wrong type, extra key, refusal, truncation, and non-JSON text. One green happy path proves almost nothing.