Skip to main content

Prompt Caching

By reusing common prefixes across requests (system / tool definitions / conversation history), caching can significantly reduce latency and lower costs. The matrix below shows which protocols (v1/chat/completions / v1/messages / v1/responses) support caching for each model.

There are two mechanisms, distinguished by how they are triggered:

  • Implicit caching — The platform automatically identifies common prefixes across requests; the client requires no additional parameters. Cache hits are billed at a discounted rate with no write overhead. The vast majority of models — OpenAI, DeepSeek, Gemini, Doubao, Qwen, etc. — use this approach.
  • Explicit caching — The client actively marks prefix checkpoints in the request via the cache_control field, with an optional TTL of 5 minutes or 1 hour. Writes and reads are billed separately. All Anthropic Claude models use this approach.

The protocols, caching capabilities, and pricing actually supported by each model are authoritative in the Model Catalog — the API row on each model card lists the supported protocols, and the "Cache Read / Cache Write" columns in the pricing table show per-token rates.

Protocol × Provider Caching Matrix​

The table below summarizes the caching behavior of major providers across the three protocols.

Provider / Model RangeCaching Mechanismv1/chat/completionsv1/messagesv1/responsesClient Configuration Required
Anthropic Claude (4.x / 5.x)Explicit cache_control✅ tools field is most reliable; system / messages depend on whether the SDK wraps content in OpenAI-standard content block format — may not propagate correctly✅ Full native support (system / messages / tools all fields)❌ Not supported upstreamRequired
Azure OpenAI (GPT-4o / GPT-4.1 / GPT-5.x / o1·o3·o4)Implicit automatic✅ Automatic✅ Automatic✅ Automatic (GPT-5.x / o3 / o4 / o1 and other supported models)Not required
DeepSeek V4 (deepseek-v4-pro / flash)Implicit automatic✅ Automatic✅ Automatic❌ Not supported upstreamNot required
Google Gemini (2.5 / 3.x Pro)Implicit automatic✅ Automatic❌ Not supported❌ Not supported upstreamNot required
Doubao 1.6+Implicit automatic✅ Automatic❌ Not supported❌ Not supported upstreamNot required
Alibaba Qwen (qwen3.5-plus / qwen3.6-plus·flash·max)Implicit automatic (upstream DashScope)⚠ Upstream response contains cached_tokens; Turing currently does not apply differential billing (billed at standard input rate)⚠ Same as left❌ Not supported upstreamNot required

The complete list of model IDs and current protocol support is authoritative in the API field of each model card in the Model Catalog. This table lists representative ranges only. v1/responses is a new protocol introduced by OpenAI (for stateful reasoning / MCP connectors in the GPT-5 series) and is currently supported only by Azure OpenAI models.

Upstream Caching Documentation by Provider​

Implicit Caching​

How It Works​

  • The upstream provider (Azure OpenAI / DashScope / Vertex AI / Volcengine) automatically reuses KV caches for common prefixes of recent requests within its own inference cluster.
  • A hit requires a strict prefix match: from the start of the request up to a given token, the content must be exactly identical to a previously cached request (including system, conversation history messages, and tools).
  • TTL is managed dynamically by the upstream platform (typically a few minutes to one hour; entries are evicted after prolonged inactivity). Azure OpenAI supports prompt_cache_retention with in_memory or 24h retention policies (availability depends on the model).
  • The cached portion is billed at the cache-read rate, with no write overhead.

Protocol Support​

  • v1/chat/completions — The default entry point for all models that support implicit caching.
  • v1/responses — Available only for Azure OpenAI models (GPT-5.x / o1·o3·o4, etc.); implicit caching works as normal, and the cached_tokens field appears in the response.
  • v1/messages — Azure OpenAI and DeepSeek V4 can be called via the messages protocol and benefit from implicit caching. (Qwen 3.5/3.6 plus series can also use the messages protocol, but Turing does not yet apply differential billing — see the matrix above.) Response field format follows the messages protocol (see "Response Field Reference" below).

Triggering and Hit Verification​

The minimum prefix length required for a hit varies significantly by provider:

ProviderMinimum Match LengthUpstream Documentation
Azure OpenAI (GPT / o series)~1,024 tokensOpenAI Prompt Caching
DeepSeek V4256 tokensDeepSeek KV Cache
Gemini 2.5 / 3.x Pro~1,024 tokensVertex AI Pricing
Doubao 1.6+See Volcengine documentationVolcengine Context Cache
Qwen 3.5+ / 3.6 seriesSee DashScope documentationBailian Context Cache

For specific model IDs, implicit caching support, and cache-read rates, see the Model Catalog — a non-empty value in the "Cache Read" column indicates that the model has implicit caching enabled.

Azure OpenAI Cache Control Fields​

Azure OpenAI does not provide a separate "create cache" API, nor does it require marking cache_control on content blocks the way Claude does. The first qualifying request automatically warms the cache; for subsequent requests with the same long prefix, usage.prompt_tokens_details.cached_tokens in the response will be greater than 0.

Azure supports two optional fields to improve hit stability or adjust retention duration:

FieldPurposeRequired
prompt_cache_keyParticipates in routing alongside the prefix hash, making requests that share a long prefix more likely to land on the same cache slotNo
prompt_cache_retentionRequests a cache retention policy; common values are in_memory / 24h; available values and defaults vary by modelNo

When these two fields are omitted, Azure still enables prompt caching by default; hit rate and retention duration are entirely governed by the upstream default policy. Azure requires at least approximately 1,024 tokens per request, with the first 1,024 tokens being completely identical; every additional 128 identical tokens continue to count toward cache hits.

Chat Completions example:

{
  "model": "turing/gpt-5.5",
  "messages": [
    { "role": "system", "content": "Stable system prompt — do not mix in timestamps or random IDs." },
    { "role": "user", "content": "Place a long prefix of more than 1,024 tokens here that remains completely unchanged in subsequent requests..." }
  ],
  "prompt_cache_key": "my-stable-prefix-v1",
  "prompt_cache_retention": "24h",
  "max_completion_tokens": 256
}

Responses example:

{
  "model": "turing/gpt-5.5",
  "input": [
    {
      "type": "message",
      "role": "user",
      "content": [
        { "type": "input_text", "text": "Place a long prefix of more than 1,024 tokens here that remains completely unchanged in subsequent requests..." }
      ]
    }
  ],
  "prompt_cache_key": "my-stable-prefix-v1",
  "prompt_cache_retention": "24h",
  "max_output_tokens": 256
}

Best practices:

  • Place large, stable content at the very beginning of messages / input, and put dynamic questions at the end.
  • system, tools schema, and assistant history can all participate in caching, but any change to a character, ordering, image detail, or tool definition anywhere in the prefix will cause a miss.
  • Use a stable prompt_cache_key for the same class of long prefix — do not generate a new one randomly each turn.
  • The first request is typically a warm-up; cached_tokens may be 0. Hits should only be observed from the second request onward.

Minimal example (no additional fields required):

curl $TURING_BASE_URL/chat/completions \
  -H "Authorization: Bearer $TURING_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-v4-pro",
    "messages": [
      { "role": "system", "content": "[Shared long system prompt, >= 256 tokens]" },
      { "role": "user", "content": "Today'\''s question" }
    ]
  }'

As long as system and the conversation history messages are exactly the same in subsequent requests, the platform will hit the cache automatically. Check the cache hit count in the response:

{
  "usage": {
    "prompt_tokens": 1280,
    "completion_tokens": 80,
    "total_tokens": 1360,
    "prompt_tokens_details": {
      "cached_tokens": 1024
    }
  }
}

Explicit Caching (Anthropic Claude)​

How It Works​

  • The client adds cache_control: { "type": "ephemeral" } to the content object to be cached, marking a prefix checkpoint — all content from the start of the request up to that checkpoint is cached.
  • TTL is either 5 minutes (default) or 1 hour (select models; see table below).
  • A maximum of 4 checkpoints per request.
  • Strict prefix match: if any token before the checkpoint differs, the cache is not hit.
  • Cache writes (creation) and reads (hits) are billed separately: the write rate is higher than the standard input rate; the read rate is far lower.

Cache Configuration (All Models)​

ConfigurationValue
Minimum token count1,024 tokens (claude-opus-4.5 / claude-haiku-4.5 require 4,096)
Maximum checkpoints per request4
TTL options5 minutes (default) / 1 hour (supported from Sonnet 4.5 onward; earlier 4.x models support 5 minutes only)

For specific model IDs and cache-read / cache-write rates, see Model Catalog → Claude — the "Cache Read", "Cache Write 5m", and "Cache Write 1h" columns show the corresponding per-token rates.

Protocol Differences (Anthropic-Specific)​

Claude models are exposed via both v1/messages and v1/chat/completions. Both support cache_control, but the scope of effect differs:

warning
  • v1/messages (Anthropic SDK): The native upstream protocol. cache_control takes full effect on all three fields: system, messages, and tools.
  • v1/chat/completions (OpenAI SDK): tools[*].cache_control is the most reliable. Whether cache_control on system / messages takes effect depends on whether the SDK wraps content in OpenAI-standard content block format — it may not propagate correctly. You can verify by checking whether usage.cache_creation_input_tokens is non-zero in the response.

Using Explicit Caching with v1/messages​

The example below demonstrates marking a 1-hour TTL checkpoint on system and a 5-minute TTL checkpoint at the end of tools. To cache conversation history in messages, add cache_control to the corresponding content block using the same syntax.

curl $TURING_BASE_URL/messages \
  -H "Authorization: Bearer $TURING_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "turing/claude-sonnet-5",
    "max_tokens": 1024,
    "system": [
      {
        "type": "text",
        "text": "You are a senior legal advisor...[thousands of tokens of fixed background knowledge]",
        "cache_control": { "type": "ephemeral", "ttl": "1h" }
      }
    ],
    "tools": [
      { "name": "search_orders", "description": "...", "input_schema": { "type": "object", "properties": {} } },
      { "name": "get_customer_info", "description": "...", "input_schema": { "type": "object", "properties": {} },
        "cache_control": { "type": "ephemeral" }
      }
    ],
    "messages": [
      { "role": "user", "content": "Please check the status of order ORD-2024-001" }
    ]
  }'

Using Explicit Caching with v1/chat/completions​

Under the OpenAI-compatible protocol, placing cache checkpoints on tools[*] is the reliably effective approach. For cache_control behavior on system / messages, see the protocol differences section above.

curl $TURING_BASE_URL/chat/completions \
  -H "Authorization: Bearer $TURING_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "turing/claude-sonnet-5",
    "max_tokens": 1024,
    "messages": [
      { "role": "user", "content": "Please check the status of order ORD-2024-001" }
    ],
    "tools": [
      {
        "type": "function",
        "function": { "name": "search_orders", "description": "...", "parameters": {} }
      },
      {
        "type": "function",
        "function": { "name": "update_order_status", "description": "...", "parameters": {} },
        "cache_control": { "type": "ephemeral" }
      }
    ]
  }'

For the cache_control syntax that takes full effect on system / messages, use the v1/messages protocol. See Choosing the Right Endpoint for details.

Response Field Reference​

You can add turing_options.include_service_usages: true to the request to obtain a detailed breakdown (including 5m / 1h TTL split, billed amount, and latency).

v1/chat/completions Response​

{
  "usage": {
    "prompt_tokens": 5139,
    "completion_tokens": 377,
    "total_tokens": 5516,
    "prompt_tokens_details": {
      "cached_tokens": 0,
      "cache_creation_tokens": 5136
    },
    "cache_creation_input_tokens": 5136,
    "cache_read_input_tokens": 0
  }
}

v1/messages Response​

{
  "usage": {
    "input_tokens": 5139,
    "output_tokens": 377,
    "cache_creation_input_tokens": 5136,
    "cache_read_input_tokens": 0
  }
}

On a subsequent cache hit, cache_creation_input_tokens = 0 and cache_read_input_tokens = 5136.

Field Descriptions​

FieldPresent InDescription
usage.prompt_tokens_details.cached_tokensv1/chat/completions onlyNumber of tokens served from cache in this request (includes both implicit and explicit)
usage.cache_creation_input_tokensBothNumber of tokens written to cache in this request (non-zero only for explicit caching)
usage.cache_read_input_tokensBothNumber of tokens read from cache in this request (explicit caching)
service_usages[].usage_detail.prompt_tokens_details.cached_tokensRequires include_service_usagesSame as the row above; persisted for billing reconciliation
service_usages[].usage_detail.prompt_tokens_details.cache_creation_token_details.ephemeral_5m_input_tokensSame as aboveNumber of tokens created with a 5-minute TTL
service_usages[].usage_detail.prompt_tokens_details.cache_creation_token_details.ephemeral_1h_input_tokensSame as aboveNumber of tokens created with a 1-hour TTL
service_usages[].bill_amount / bill_currencySame as aboveActual billed amount and currency for this service call

Important Notes​

  1. Implicit caching is sensitive to prefix stability: Do not mix timestamps, UUIDs, current usernames, or other dynamic content into long system prompts — this will invalidate the entire cache. Azure OpenAI's prompt_cache_key can improve hit stability for the same long prefix, but it does not relax the strict prefix-match requirement.
  2. Caches are not shared across models: Different model IDs (including pro / flash / mini variants within the same family) each maintain their own cache. Switching models invalidates the cache.
  3. Strict prefix matching (applies to both modes): Every token before the cached content must be exactly identical to the prior request; otherwise there is no hit.
  4. Minimum token threshold for explicit caching: Claude Opus 4.5 / Haiku 4.5 require ≥ 4,096 tokens; all other Claude models require 1,024 tokens. If the threshold is not met, the cache entry will not be created.
  5. Checkpoint limit for explicit caching: A maximum of 4 cache_control checkpoints per request, distributed across system / tools / messages.
  6. Break-even threshold for explicit caching: The initial write costs more than standard input. The cache must be hit ≥ 2 times by subsequent requests to achieve an overall cost reduction. Enabling caching may not be cost-effective when request repetition is low. See the Model Catalog for specific rate comparisons.

To view actual billing details, see Usage and Billing.