Timeouts & Model Instability

Background

Due to network fluctuations or provider instability, model calls may occasionally time out or fail. The Turing Platform provides three configuration options to address this:

Option	Purpose
timeout	Controls the maximum wait time for a single request
max_retries	Automatically retries on failure
fallbacks	Switches to a backup model when the primary model is unavailable

---

Why Do Timeouts Occur?

If you encounter timeout errors, the likely causes are:

1. Large file transfers: Uploading large images or files takes more time
2. Long reasoning problems: Models with reasoning mode (e.g., the o-series, DeepSeek R1) may require extended inference time on complex problems
3. Model silent periods: The model may not return data for a period of time while processing a request

Troubleshooting: Use stream=True mode to verify whether the first chunk is returned

completion = client.chat.completions.create(
    model="turing/deepseek-r1",
    messages=[{"role": "user", "content": "A complex math problem..."}],
    stream=True
)

for chunk in completion:
    print(chunk)  # Observe the time to first chunk

• First chunk returned but subsequent timeout: The model is working — simply increase the timeout value
• No response at all: Likely a network issue or invalid request parameters

---

Practical Example

Suppose your application has a response-time requirement: complete the LLM call within 10 seconds.

Under normal conditions, deepseek-v4-flash responds in an average of 3–4 seconds. However, timeouts or failures occasionally occur.

Solution: Set timeout: 5 seconds and use fallbacks to fall back to doubao-seed-2.1-turbo.

from openai import OpenAI

client = OpenAI()

completion = client.chat.completions.create(
    model="deepseek-v4-flash",
    messages=[{"role": "user", "content": "Hello"}],
    turing_options={
        "timeout": 5,
        "fallbacks": "doubao-seed-2.1-turbo"
    }
)

Result:

• Normal case: deepseek-v4-flash responds in 3–4 seconds
• Primary model times out (>5 s) or fails: automatically switches to doubao-seed-2.1-turbo
• Total latency kept within 10 seconds

If you want the primary model to have one retry before falling back:

turing_options={
    "timeout": 5,
    "max_retries": 1,
    "fallbacks": "doubao-seed-2.1-turbo"
}

---

Configuring Multiple Fallback Models

turing_options={
    "timeout": 10,
    "max_retries": 1,
    "fallbacks": [
        "doubao-seed-2.1-turbo",     # First fallback
        "turing/gpt-5.4-mini"        # Second fallback
    ]
}

---

Overriding Parameters for Fallback

If the fallback model requires different parameter settings:

turing_options={
    "fallbacks": {
        "model": "doubao-seed-2.1-turbo",
        "thinking": {"type": "enabled"},  # Enable extended reasoning
        "temperature": 0.7
    }
}

---

Client-Side Timeout Explained

The OpenAI SDK timeout is a client-side timeout, while turing_options.timeout is a server-side timeout — both take effect independently. It is recommended to set the SDK client timeout longer than the server-side timeout:

client = OpenAI(
    timeout=60.0  # Client timeout: 60 seconds
)

completion = client.chat.completions.create(
    model="turing/gpt-4.1",
    messages=[{"role": "user", "content": "Hello!"}],
    turing_options={
        "timeout": 30  # Server-side timeout: 30 seconds
    }
)

---

Parameter Reference

Parameter	Type	Default	Range	Description
timeout	float	stream: 90s, non-stream: 120s	0–300	Request timeout in seconds
max_retries	int	0	1–3	Maximum number of retries
fallbacks	string | object | array	null	-	Fallback model configuration

---

Important Notes

1. Streaming requests that have already started returning chunks will not retry or trigger a fallback if an error occurs mid-stream
2. Ensure the fallback model supports the features used in the original request (e.g., tools, vision)
3. Fallback requests are billed independently, based on the model actually used