Skip to main content

Custom Timeout, Retry, and Fallback

All configuration options are passed via turing_options.

Timeout​

ParameterDefaultRange
timeout300 seconds (same for streaming and non-streaming)0–600 seconds

For streaming requests, this timeout governs the maximum silence period both before the first byte and between consecutive chunks. Time to first token grows with input size (the model produces no output during the prefill phase), so increase this value as needed for long-context workloads.

from openai import OpenAI

client = OpenAI()

completion = client.chat.completions.create(
    model="turing/gpt-4.1",
    messages=[
        {"role": "user", "content": "Hello!"}
    ],
    turing_options={
        "timeout": 30  # seconds
    }
)
curl $TURING_BASE_URL/chat/completions \
  -H "Authorization: Bearer $TURING_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "turing/gpt-4.1",
    "messages": [
      {"role": "user", "content": "Hello!"}
    ],
    "turing_options": {
      "timeout": 30
    }
  }'

Retry​

ParameterDefaultRange
max_retries01–3
completion = client.chat.completions.create(
    model="turing/gpt-4.1",
    messages=[{"role": "user", "content": "Hello!"}],
    turing_options={
        "max_retries": 2
    }
)

Fallback​

ParameterType
fallbacksstring | object | array

Parameter Inheritance Rules​

ParameterBehavior
Request parameters such as messages, tools, temperature, streamInherited automatically
turing_options.timeoutInherited automatically
turing_options.max_retries, turing_options.fallbacksNot inherited

String Format​

turing_options={
    "fallbacks": "doubao-seed-2.1-turbo"
}

Object Format (with parameter overrides)​

turing_options={
    "fallbacks": {
        "model": "doubao-seed-2.1-turbo",
        "thinking": {"type": "enabled"},
        "temperature": 0.7
    }
}

Array Format (multiple fallbacks)​

turing_options={
    "fallbacks": [
        "doubao-seed-2.1-turbo",
        "turing/gpt-5.4-mini"
    ]
}

Auto Mode (automatic fallback)​

When fallbacks: "auto" is set, the platform automatically generates a cross-provider fallback chain based on the current model.

turing_options={
    "fallbacks": "auto"
}

Auto fallback strategy:

  1. First tries other generation models in the same family (e.g., the previous generation from the same provider).
  2. Then tries the latest models from other families in order (cross-provider safety net).
Risk Notice

Auto fallback switches requests to models from different providers. Parameter support and capabilities vary across models. Please evaluate whether this is appropriate for your use case:

  • Multimodal capabilities are not interchangeable: if the primary model supports video/image understanding, the fallback model may not have that capability.
  • Specialized parameters may be incompatible: reasoning/thinking mode, tool calling, and similar behaviors differ across providers.
  • Not suitable for scenarios that depend on a specific model's capabilities: if an Agent relies on features unique to a particular model, specify the fallback manually.

Auto mode is best suited for general text generation scenarios where there is no strong dependency on a specific model's capabilities.

Combined Usage​

from openai import OpenAI

client = OpenAI()

completion = client.chat.completions.create(
    model="deepseek-v4-flash",
    messages=[{"role": "user", "content": "Hello!"}],
    turing_options={
        "timeout": 10,
        "max_retries": 1,
        "fallbacks": [
            "doubao-seed-2.1-turbo",
            "turing/gpt-5.4-mini"
        ]
    }
)

Observing Actual Results (Response Headers)​

After configuring max_retries / fallbacks, you can inspect the non-streaming response headers to confirm what actually happened during the call:

HeaderDescription
x-turing-model-idThe model ID that actually served this request (use this to confirm which model was selected when fallbacks: "auto" is set)
x-turing-retriesThe actual number of retries that occurred
x-turing-fallbacksThe actual number of fallback switches that occurred
response = client.chat.completions.with_raw_response.create(
    model="deepseek-v4-flash",
    messages=[{"role": "user", "content": "Hello!"}],
    turing_options={
        "max_retries": 2,
        "fallbacks": "auto",
    },
)

print(response.headers.get("x-turing-model-id"))   # model that ultimately served the request
print(response.headers.get("x-turing-retries"))    # retry count
print(response.headers.get("x-turing-fallbacks"))  # fallback count

completion = response.parse()

If all fallbacks are exhausted and an error is returned, these three headers are still attached to the error response, making it easy to determine how far along the chain the request progressed. Streaming responses do not carry these three headers. For the complete response header specification, see Headers Protocol.

FAQ​

Why am I seeing timeout errors?​

Timeout errors may be caused by:

  1. Large file transfers: uploading a large number of images or files takes more time.
  2. Long reasoning problems: models that use reasoning mode (e.g., the o-series, DeepSeek R1) may require extended inference time on complex problems.
  3. Model silence periods: the model may go silent for a period while processing a request.

Recommended troubleshooting steps:

  1. Verify using streaming mode: switch to stream=True and check whether the first chunk is returned.

    completion = client.chat.completions.create(
        model="turing/deepseek-r1",
        messages=[{"role": "user", "content": "A complex math problem..."}],
        stream=True  # enable streaming mode
    )

    for chunk in completion:
        print(chunk)  # observe when the first chunk arrives

  2. If the first chunk is returned but a timeout occurs later: the model is working but simply taking longer to process; increase the timeout value.

    completion = client.chat.completions.create(
        model="turing/deepseek-r1",
        messages=[{"role": "user", "content": "A complex math problem..."}],
        turing_options={
            "timeout": 180  # increase to 3 minutes
        }
    )

  3. If nothing is returned at all: this may indicate a network issue or an invalid request parameter. Check your network connection and request format.

Client-Side Timeout​

The OpenAI SDK timeout is a client-side timeout, while turing_options.timeout is a server-side timeout. It is recommended to set the client timeout greater than the server timeout:

client = OpenAI(
    timeout=60.0  # client-side timeout
)

completion = client.chat.completions.create(
    model="turing/gpt-4.1",
    messages=[{"role": "user", "content": "Hello!"}],
    turing_options={
        "timeout": 30  # server-side timeout
    }
)