New: outcomized services. Pay per result, with a monthly cap you set. See pricing →
Engineering

What is Prompt Caching?

Prompt Caching: A model API feature that stores the processed start of a prompt so repeat requests sharing it cost less and respond faster.

Prompt caching lets a model provider reuse work it already did on the beginning of a prompt. When many requests share the same prefix (a long system prompt, tool definitions, a reference document, or earlier turns of a conversation), the provider caches the processed prefix. Later requests that start with the exact same content read from the cache, which is billed at a fraction of the normal input price and reduces time to first token.

Providers implement it differently: some cache automatically, others require you to mark cache breakpoints, and caches expire after a few minutes of inactivity unless refreshed or extended. To benefit, structure prompts with the stable content first and the changing content (the user's question, the specific document) last, because any change early in the prompt invalidates the cache for everything after it.

Why it matters when you build or buy AI

For agents and assistants with long instructions or many tools, caching is one of the simplest ways to cut cost per task and latency without touching quality. It takes prompt structure discipline: a timestamp or user name near the top of the system prompt can silently break caching on every request. Check cache hit rates in your usage data.

Example

A support agent has a 12,000-token system prompt with policies and tool definitions. With caching, every conversation after the first reuses it at a discounted rate, and only the customer's messages are billed at full price.

Did the model bill surprise you?A $6,500 week to cap cost per task without killing the product.

Related terms

More in Engineering

Next step

Book 20 minutes, or send a brief

A former CTO replies. You leave with a written plan and a fixed estimate.

No spam. One reply within one business day. Or skip the form and book a call.

Thank you. Your brief is on its way. Grab a time now so we can talk it through.

Prefer LinkedIn? Connect with Harish.

Book a callFree lead test