Long-Running Responses

A non-streaming completion that takes a long time is answered differently from a fast one. The response is still valid OpenAI-shaped JSON, but three things change, and one of them will mislead a client that only checks the HTTP status code.

This page is the whole contract for that case. It applies to non-streaming requests only — streaming is unaffected.

What changes

A non-streaming request that is still running after a threshold commits its response early: the status line and headers are sent before the completion exists, so the connection survives intermediaries that would otherwise time it out. Everything after that point travels in the body.

Three observable consequences:

The leading whitespace is insignificant JSON — JSON.parse and every standard SDK handle it without changes. You do not need to trim it.

Reading an error: the status code is no longer enough

On a long request you will receive 200 even when the request failed. The status was already committed before the outcome was known, so it cannot be changed. The real outcome is in the body.

Detect failure by inspecting the body, not the status. The reliable test on both surfaces:

const res  = await fetch(url, { method: 'POST', headers, body });
const data = await res.json();

// Works for fast AND long requests, on every surface.
if (data.error || !data.choices) {
  // failed — read data.error.code / data.error.type / data.error.message
}

Checking res.ok alone, or res.status === 200, will treat a failed generation as a success.

OpenAI-compatible surfaces

/v1/chat/completions, /v1/completions, /v1/responses:

{
  "error": {
    "type": "service_unavailable",
    "code": "no_healthy_sellers",
    "message": "..."
  }
}

There is no choices array on a failed response, which is the simplest thing to branch on. error.code carries the real cause — the same code a fast request would have returned in its status envelope, so existing error-handling code keyed on code keeps working. error.type reads service_unavailable for a failure discovered after the commit; do not key on it to distinguish causes.

Anthropic-compatible surface

/anthropic/v1/messages keeps its own error shape, which carries no code:

{
  "type": "error",
  "error": { "type": "api_error", "message": "..." }
}

A note on upstream errors

A 200 carrying an error object can also come straight from the upstream provider, independently of anything on this page. The detection recipe above covers both cases, which is why it is the one to implement.

Where the headers went

On a long request the routing disclosures cannot be sent as headers, so they are published additively under a single surplus key:

{
  "id": "chatcmpl-…",
  "choices": [ … ],
  "usage": { "prompt_tokens": 596, "completion_tokens": 2266, "buyer_cost_micro": 2132 },
  "surplus": {
    "provider": "venice",
    "served_by": "marketplace",
    "marketplace_status": "served",
    "attempts": 1,
    "routing_decision_ms": 19,
    "adapted_params": ["max_tokens"],
    "truncated": false
  }
}

surplus is additive — nothing else in the response is rewritten — and it appears only on a committed response. Fast requests are byte-identical to what they have always returned, with every header intact and no surplus key. Fields with nothing to report are omitted rather than sent as null.

What you get, and the header it replaces

Numbers arrive as numbers, booleans as booleans, and adapted_params as an array — not the CSV string the header used.

Where to find what you were billed

usage.buyer_cost_micro, always. It is in the body of every non-streaming response, fast or long, and is not affected by any of the above. Integer microdollars, where 1_000_000 = 1 USD.

The x-si-buyer-cost-micro header is absent on a committed response — use the usage field, which is the canonical place on both wires.

Checklist

• Branch on data.error / data.choices, never on HTTP status alone.
• Read the charge from usage.buyer_cost_micro.
• Read routing detail from surplus when present, falling back to the x-si-* headers.
• Do not trim the leading whitespace; hand the body to your JSON parser as-is.
• Streaming is unaffected — SSE requests behave exactly as documented elsewhere.