Media Inputs

Sending an image, video, document, or audio file to a model, and what the marketplace does with it.

This page covers media understanding. Media generation is a separate surface: see Image Generations, Video Generations, and Music Generations.

Generation vs Understanding

The two are separate surfaces with separate models. Nothing rejects a cross-surface call up front, so sending a generation model to /v1/chat/completions, or a vision model to /v1/images/generations, surfaces as 404 no_sellers_for_model or an opaque provider error rather than a clear message.

Generation endpoints take a source image in dedicated top-level fields (image, image_url, input_images). Understanding endpoints take it inside messages[].content.

Capability Discovery

GET /v1/models is the capability surface, public and unauthenticated. Two fields matter:

• architecture.input_modalities, with values from text, image, video, file, audio.
• supported_features, where vision and file_input are relevant.

Check both. The two disagree on a meaningful number of rows, and the router treats a model as image-capable when either says so. claude-opus-5 declares vision in supported_features while input_modalities reads ["text"], and it does accept images.

# Models accepting image input, by either signal
curl -s https://api.surplusintelligence.ai/v1/models \
  | jq -r '.data[]
      | select(((.architecture.input_modalities // []) | index("image"))
               or ((.supported_features // []) | index("vision")))
      | .id'

# Everything one model declares
curl -s https://api.surplusintelligence.ai/v1/models \
  | jq '.data[] | select(.id=="gemini-3.8-flash")
      | {id, input: .architecture.input_modalities, features: .supported_features}'

# Models accepting image input, by either signal
curl -s $SURPLUS_BASE_URL/models \
  | jq -r '.data[]
      | select(((.architecture.input_modalities // []) | index("image"))
               or ((.supported_features // []) | index("vision")))
      | .id'

# Everything one model declares
curl -s $SURPLUS_BASE_URL/models \
  | jq '.data[] | select(.id=="gemini-3.8-flash")
      | {id, input: .architecture.input_modalities, features: .supported_features}'

A 2026-09-07 snapshot returned 403 catalog rows: 99 accepting image input, 18 declaring file, 15 declaring audio, 14 declaring video. Counts change; read the endpoint. A declared modality means the upstream model accepts it, not that a given wire spelling reaches that model through the marketplace.

Image Input

Send an OpenAI-format image content part. A remote URL and an inline data URL both work. There is no upload step and no file id to create first.

curl https://api.surplusintelligence.ai/v1/chat/completions \
  -H "Authorization: Bearer inf_xxx" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-3.8-flash",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "text", "text": "What is in this image?"},
        {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}}
      ]
    }]
  }'

curl $SURPLUS_BASE_URL/chat/completions \
  -H "Authorization: Bearer inf_xxx" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-3.8-flash",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "text", "text": "What is in this image?"},
        {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}}
      ]
    }]
  }'

{
  "type": "image_url",
  "image_url": { "url": "data:image/png;base64,iVBORw0KGgo..." }
}

Marketplace Handling

messages is forwarded to the chosen seller whole. The marketplace does not fetch buyer URLs, re-encode bytes, or validate the MIME type declared in a data URL. Three behaviors apply:

Presence detection. A part typed image_url, input_image, or image marks the request as carrying an image. A capability gate can reject an image sent to a text-only model with 400 model_does_not_support_images. When that gate is inactive the request is forwarded, the provider rejects it, and the router may try a bounded number of other offers (a capability fault is capped at three upstream attempts in total) before surfacing the provider's error. That failover is gated by ADAPTER_FAILOVER, which ships off, in which case the first provider's error surfaces directly. The remedy is the same in every case: use a vision-capable model.

Refusal for images referenced by id. An image part whose only content is a file_id returns 400 unresolvable_file_reference. A file id names a file in the buyer's own provider account; the marketplace has no files API, another seller's key cannot resolve the id, and the chat wire has no field for one. Inline the bytes as a data URL instead.

Anthropic wire translation. On a seller served over the Anthropic wire, image_url is rewritten into an Anthropic image block. A data: URL becomes {"source": {"type": "base64", "media_type": "<as declared>", "data": "..."}}; an https URL becomes {"source": {"type": "url", "url": "..."}}. The media_type is copied verbatim from the data URL, not inferred from the bytes.

Video Input

The marketplace defines no contract for sending video to a chat model. No request path recognizes, validates, fetches, transcodes, or samples video, and no wire spelling for video input exists.

The data:video/mp4 Payload

A payload of this shape has been reported in the wild:

{"type": "image_url", "image_url": {"url": "data:video/mp4;base64,<BASE64>"}}

It is not universally supported and is not a marketplace feature. Actual behavior:

• The MIME type is never parsed, so the part is forwarded unchanged to whichever seller wins the route. It succeeds only if that provider accepts a video payload in the image_url field.
• Provider selection is a per-request routing decision, so an identical body can succeed on one request and fail on the next as liquidity and health change.
• It cannot work on an Anthropic-format seller. The bridge maps it to an Anthropic image block with media_type: "video/mp4", which that block type does not accept.
• The 10 MB JSON body limit applies, which is small for video.

Clients depending on this behavior should pin the route with a provider hint naming the verified provider, or use a BYOK priority provider. Confirm against that provider's own documentation and treat a change of provider as a change of contract.

YouTube URLs

Not supported. No marketplace code path recognizes a YouTube URL, and the OpenAI-compatible wire has no field that carries one.

• A YouTube link in a text part reaches the model as a literal string.
• A YouTube link in image_url.url is forwarded as an image URL and rejected by the provider.
• Providers with native YouTube support expose it through their own API shape rather than the OpenAI-compatible wire, so it is unreachable through the marketplace even for models whose upstream supports it.

Extract frames or a transcript locally, or call that provider directly with an own-account key.

File Input

A file content part is forwarded unchanged, so a provider accepting inline document bytes works:

{
  "type": "file",
  "file": { "filename": "contract.pdf", "file_data": "data:application/pdf;base64,JVBERi0x..." }
}

• No capability gate applies to file parts. Few catalog rows declare file_input, and some models accept PDFs without declaring it. Check supported_features first and expect a provider-side error otherwise.
• A file_id on a file part is forwarded as sent. It resolves only when the credential serving the request belongs to the account owning the id, which in practice means routing to a BYOK provider.
• A file_id on an image part is refused up front, as described above.

Audio Input

POST /v1/audio/transcriptions is the speech-to-text upload endpoint, and one of only two paths that take a multipart/form-data file body (the other is POST /v1/images/edits). It takes a file part, capped at 25 MB, against a speech-to-text model such as venice-whisper-large-v3. Chat models declaring audio in input_modalities accept provider-native audio parts on the chat wire under the same passthrough rules as everything else on this page.

Limits

Providers apply their own limits, usually smaller. A payload within the marketplace limit can still be rejected upstream.

Errors

See Also

• Chat Completions for the full request contract.
• Provider & Model Verification for what a provider hint guarantees.
• Reasoning Controls for the other per-model capability requiring lookup.
• Models & Providers for name resolution and the provider allowlist.