Skip to main content

Streaming

Receive model output chunk by chunk via SSE (Server-Sent Events), reducing time to first token (TTFT) from seconds to milliseconds. The standard choice for chat UIs, text-to-speech, and real-time code completion.

When to Use​

  • Interactive chat UIs (typewriter effect)
  • Long responses (> 500 tokens) where perceived latency matters
  • Scenarios requiring early cancellation (user interruption, cancel on condition)
  • Downstream streaming consumers (TTS, video captions)

Avoid for: backend batch processing or cases where only the final result is needed — non-streaming is simpler and more reliable.

Basic Usage​

Chat Completions (OpenAI Protocol)​

from openai import OpenAI

client = OpenAI(
    api_key="your-api-key",
    base_url="https://live-turing.cn.llm.tcljd.com/api/v1",
)

stream = client.chat.completions.create(
    model="turing/gpt-5.4-mini",
    messages=[{"role": "user", "content": "Write a short poem about spring"}],
    stream=True,
    stream_options={"include_usage": True},  # send one extra usage chunk at the end of the stream
)

for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
    if chunk.usage:
        print(f"\n[total={chunk.usage.total_tokens}]")
curl -N $TURING_BASE_URL/chat/completions \
  -H "Authorization: Bearer $TURING_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "turing/gpt-5.4-mini",
    "messages": [{"role": "user", "content": "Write a short poem about spring"}],
    "stream": true,
    "stream_options": {"include_usage": true}
  }'

The response is an SSE stream:

data: {"id":"...","choices":[{"delta":{"content":"Spring"},"index":0}]}

data: {"id":"...","choices":[{"delta":{"content":" day"},"index":0}]}

data: {"id":"...","choices":[{"delta":{},"index":0,"finish_reason":"stop"}]}

data: {"choices":[],"usage":{"prompt_tokens":12,"completion_tokens":45,"total_tokens":57}}

data: [DONE]
SSE Event Delimiter

Starting 2025-04-27, the Turing Platform uses single \n delimiters for Portal clients and remains compatible with double \n\n for older clients. Standard SSE parsers (eventsource-parser, httpx, openai SDK) handle both automatically.

Messages (Anthropic Protocol)​

The Anthropic stream has more granular event types: message_start / content_block_start / content_block_delta / content_block_stop / message_delta / message_stop / ping / error.

from anthropic import Anthropic

client = Anthropic(
    base_url="https://live-turing.cn.llm.tcljd.com/api/v1",
    auth_token="your-api-key",
)

with client.messages.stream(
    model="turing/claude-sonnet-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Write a short poem about spring"}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

Raw event stream structure:

event: message_start
data: {"type":"message_start","message":{"id":"msg_…","role":"assistant",…}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Spring"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" day"}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"output_tokens":45}}

event: message_stop
data: {"type":"message_stop"}

Responses (OpenAI Responses Protocol)​

Event names are longer and more granular: response.created / response.output_item.added / response.output_text.delta / response.output_text.done / response.completed, etc. The SDK abstracts this layer — you can consume text directly.

response = client.responses.create(
    model="turing/gpt-5.4-mini",
    input="Write a short poem about spring",
    stream=True,
)

for event in response:
    if event.type == "response.output_text.delta":
        print(event.delta, end="", flush=True)

Usage in Streams​

By default, streams do not include usage data. To obtain token statistics:

ProtocolHow to Enable
Chat Completionsstream_options: {"include_usage": true} → an extra chunk with choices=[] and usage={...} is appended at the end
MessagesThe final message_delta event naturally includes usage.output_tokens
ResponsesThe response.completed event carries the full usage

Tool Use in Streams​

Tool calls are also returned incrementally — function.arguments arrives in fragments that the client must concatenate.

stream = client.chat.completions.create(
    model="turing/gpt-5.4-mini",
    messages=[{"role": "user", "content": "Check the weather in Shanghai"}],
    tools=[...],
    stream=True,
)

tool_calls = {}  # tool_call_id -> accumulated args
for chunk in stream:
    delta = chunk.choices[0].delta
    for tc in (delta.tool_calls or []):
        if tc.id:  # new call
            tool_calls[tc.id] = {"name": tc.function.name, "args": ""}
        # use index to locate the in-progress call (id only appears in the start event)
        call = tool_calls[list(tool_calls)[tc.index]]
        if tc.function.arguments:
            call["args"] += tc.function.arguments

For full documentation, see Function Calling / Streaming.

Streaming Error Handling​

Scenario: the server errors out or the network drops after the first chunk has been received.

  • OpenAI SDK: the for chunk in stream: loop raises an exception — wrap it in try/except
  • Anthropic SDK: an error event appears in the stream; no exception is raised directly
  • Raw SSE: parse each data: line with JSON error tolerance

Best practice:

import logging

logger = logging.getLogger(__name__)

try:
    for chunk in stream:
        handle(chunk)
except Exception as e:
    # log content received so far + trace_id (see Request Tracing)
    logger.error("stream interrupted after N chars, trace_id=...", exc_info=e)
    raise

To validate capacity before initiating a stream (avoiding a 429 midway through), use count_tokens to estimate input size first.

Parameter Reference​

Billing Impact​

  • Streaming and non-streaming are billed identically (per token)
  • Streaming does not affect Prompt Caching pricing
  • When a stream times out or is interrupted, tokens already generated are still billed (the backend cannot "roll back")

FAQ​

  • First chunk is very slow (> 10s) → The model is most likely reasoning. See Thinking & Reasoning or switch to a non-reasoning model.
  • Stream stalls with no chunks mid-way → The backend may be executing a tool call or reasoning; some providers do not push chunks during this phase. Set a reasonable timeout.
  • Cannot get usage → You forgot stream_options: {"include_usage": true}.
  • SDK reports "Unexpected end of JSON" → Usually a gateway middleware (nginx, CDN) cut the stream. Add Cache-Control: no-cache or inspect your proxy.
  • Truncated multibyte characters → Multi-byte characters may span chunk boundaries; always concatenate complete UTF-8 before rendering.

See Also​