Arrange Prompts So Prefix Caching Can Work

Prompt caching matches from the start. The reusable part is a stable prefix the provider has already processed. Matching stops at the first change, so a timestamp, request ID, or user message near the top can make everything after it miss the cache.

This is not ordinary response caching. The model still produces a new answer. The provider reuses work for the matching prompt prefix, subject to that provider's token minimums, lifetime, and usage rules.

Put stable material first

A cache-friendly prompt normally follows this order:

1. System instructions
2. Tool and output schemas
3. Stable reference documents
4. Stable few-shot examples
5. Conversation or request data that changes
6. The current user message

Do not put request_id, current time, random delimiters, or per-user metadata before the long stable block. Even harmless whitespace changes can alter the prefix sent to the provider.

Draw the first-difference line

Serialize two real requests and compare them byte for byte from the beginning. Mark the first difference. Everything after that point is outside the shared prefix.

This catches subtle misses:

object keys emitted in random order;
tools listed in a different order;
documents sorted by retrieval score;
a generated nonce in the system message;
one environment adding a trailing newline;
examples changing with every deploy.

Stability should not make facts stale. If a reference document changes, change the prefix and accept the miss. Correctness is more important than keeping a cache entry warm.

Read provider telemetry

Measure explicit cache creation and cache read fields from the provider response. Track tokens and cost for the same prompt shape before and after the reorder. Lower latency alone does not prove a cache hit because model load varies.

Also record the model, region, cache policy, prefix token count, and time between requests. Cache behaviour and minimum sizes differ by provider and can change.

Test one controlled pair

Send request A, then request B with the exact same stable prefix and a different final user input. Confirm the second response reports cache reads. Then move one variable field above the stable documents and prove the read disappears. That negative control tells you the instrumentation is actually observing the mechanism.

Claude can help reorganize prompt assembly and find unstable fields. Code should perform canonical serialization and capture the usage metadata.

Run the starter locally

npm test
npm run validate
npm run sample

The starter validates fictional prompt-layout evidence. Live mode can ask Claude for a draft, but it does not alter a production prompt or promise a provider-specific saving.