Timeouts for Long Streaming Requests: A Walkthrough

Formatos de comunicaçãoEN2026-10-25

A long streaming response needs a timeout policy that distinguishes waiting for a connection, waiting for more data, and waiting for the entire task to finish. These are different limits: HTTPX, for example, defines its read timeout around receiving the next chunk, not completing the whole response. ([python-httpx.org](https://www.python-httpx.org/advanced/timeouts/?utm_source=openai)) This walkthrough shows how to configure those boundaries and investigate interrupted requests.

§ Step 1: Record where the request stops

Before changing settings, instrument one representative request. Record the request start, response headers, first stream event, first visible text, last received event, and connection close. Keep the exception type, HTTP status when available, and any request identifier returned.

Compare several failures. Treat a repeated cutoff near the same elapsed time as a clue to investigate a deadline. If the cutoff instead follows a quiet interval, investigate an idle timeout. Test the same request directly from your backend and through your normal application path to narrow the search.

§ Step 2: Separate the timeout budgets

Define your application policy explicitly rather than relying on a single setting called timeout. Transport libraries distinguish connection, read, write, and connection-pool waits; an overall application deadline is a separate concern. ([python-httpx.org](https://www.python-httpx.org/advanced/timeouts/?utm_source=openai))

Connection budget: How long you allow connection establishment.
First-event budget: How long your application waits for the first stream event.
Read-idle budget: How long you allow between received chunks.
Overall deadline: How long the complete operation may run, including any retries.

For an initial test, you might choose a 10-second connection budget, a 120-second first-event budget, a 120-second read-idle budget, and a 15-minute overall deadline. These are illustrative application settings, not published Vana API limits. Adjust them against measured workloads and your acceptable waiting time.

§ Step 3: Configure the client deliberately

Vana API provides one endpoint for your model requests. Keep that endpoint configuration separate from timeout configuration so you can diagnose each independently. ([vanaapi.com](https://vanaapi.com/)) In an OpenAI SDK integration, inspect the timeout and retry options supported by your installed version. The official Python library exposes configurable timeouts and retries; do not assume every SDK uses identical settings or semantics. ([github.com](https://github.com/openai/openai-python/blob/main/README.md?plain=1&utm_source=openai))

During diagnosis, disable automatic retries where supported. Run a single attempt and capture its actual duration. Implement the overall deadline at the application layer if your transport timeout does not provide one. Keep a user cancellation path rather than making every wait unlimited.

§ Step 4: Check the proxy path

If your backend uses NGINX, inspect both proxy_read_timeout and proxy_buffering. The read timeout applies between successive upstream reads, not to the total response duration. Buffering changes how upstream data is passed downstream. For a streaming route, consider proxy_buffering off and choose a read timeout compatible with your intended idle budget. ([nginx.org](https://nginx.org/en/docs/http/ngx_http_proxy_module.html?utm_source=openai))

Inventory every other hop you control: application server, ingress, load balancer, CDN, and hosting runtime. Check each product's documented idle and execution limits. Do not assume increasing the SDK timeout also changes infrastructure limits.

§ Step 5: Distinguish activity from completion

Use the event parser for the protocol you selected. Anthropic Messages streams can include ping events and in-stream errors, and normal completion includes message_stop. A ping indicates connection activity, not new answer text or successful completion. ([platform.claude.com](https://platform.claude.com/docs/en/build-with-claude/streaming?trk=article-ssr-frontend-pulse_little-text-block&utm_source=openai))

Track transport activity separately from visible progress. Preserve partial text, but mark the response incomplete unless your protocol's completion condition is satisfied. In your own downstream SSE layer, you may send valid heartbeat comments; do not assume those heartbeats extend an upstream deadline.

§ Step 6: Test recovery before deployment

Test a slow start, an extended pause, a midstream disconnect, and user cancellation. For each, verify that the application closes resources, preserves useful partial output, and displays an accurate completion state.

Review retry settings explicitly because the OpenAI Python SDK retries certain failures automatically. ([github.com](https://github.com/openai/openai-python/blob/main/README.md?plain=1&utm_source=openai)) As an application policy, avoid silently appending a fresh attempt to partial output. Label a restart, limit attempts, and keep them inside the overall deadline.

Create a Vana API account to test this policy with your own workload. Keep the measurements, then tune the smallest timeout responsible for the interruption rather than increasing every limit.