Runs & streaming

A run is one turn: input messages in, events while it works, an output record at the end. Tools pause runs, approvals gate them, and usage and traces are recorded against them.

Anatomy of a run

POST /v1/smiths/{sid}/runs
  → queued → running → completed
                     ↘ paused_for_tool      (an external-execution tool needs you)
                     ↘ paused_for_approval  (a human must approve)
                     ↘ failed / cancelled

A paused run resumes when you submit the missing piece via /submit, the same endpoint for tool results, approval decisions, and cancellation.

A run that a schedule, a channel message or a background response started resumes through the queue: submitting an approval answers 202 with the run queued, and it is running again once a worker takes it. A queued run can therefore have history.

Every run carries an actor — who started it, resolved from the token at the point of action: { "kind": "smith" | "tenant" | "operator", "id": …, "token_id": …, "email": … }. A run a tenant-admin token drove names the tenant and that token; one a schedule or an inbound channel message drove names the smith with an empty token_id (no token acted). email names the human when there is one — a signed-in console user or an Ingram operator — and is empty for a machine caller. Every event the run produces carries the same actor.

Threads carry conversation history: pass a stable thread_id per conversation (any string of yours, or omit it and one is minted with a thr_ prefix and returned on the run) and the smith sees the recent turns of that thread.

metadata.deployment_id marks a run a deployment started, so a run you create may not set it: 422 invalid_request.

input is a list of messages: { "role": "user" | "assistant", "content": "<text>" }. content is a plain string for a text turn, or an array of typed parts ({ "type": "text" | "image" | "file", … }) for a multimodal turn — when you read a run back, a turn that carried an image or file comes back in the parts form. Multiple messages are allowed (e.g. to replay context); system behaviour comes from the smith's resolved instructions, not a system message. For long turns prefer "stream": true. A synchronous call holds the connection open for the whole turn.

Synchronous runs

# Authorization: tenant-admin token (server-side only), or a smith token
curl https://api.cloud.ingram.tech/v1/smiths/smt_…/runs \
  -H "Authorization: Bearer $IC_TOKEN" \
  -H "IC-Api-Version: 2026-05-01" \
  -H "Content-Type: application/json" \
  -d '{ "input": [{ "role": "user", "content": "Summarize what we discussed yesterday." }],
        "thread_id": "chat_42" }'

The response is the run record:

{ "id": "run_…", "smith_id": "smt_…", "thread_id": "chat_42",
  "status": "queued | running | completed | paused_for_approval | failed | cancelled",
  "output": { "content": "…", "tool_calls": [ { "id": "call_1", "type": "function",
               "function": { "name": "get_weather", "arguments": "{…}" } } ] },
  "stop_reason": "end_turn",
  "warnings": [],
  "usage": { "input_tokens": 0, "output_tokens": 0, "total_tokens": 0,
             "cost": 0.0241 } }

When a prompt-cache slice was served or written, usage also carries cache_read_tokens / cache_write_tokens — see Models → Prompt caching.

Why a run ended

stop_reason is why the turn stopped. It is the same value on the run record, on the event feed, and on every stream frame.

A cancelled run carries your own /submit kind=cancel reason instead (or run_timeout when the platform cancelled it at its deadline, or client_disconnected when you dropped the connection of a one-shot structured or client-tool turn).

Branch on max_tokens to retry or continue a cut-off answer: status is completed and output.content holds the partial text.

output.tool_calls follows the OpenAI tool_calls shape ({ id, type, function }) for a model-driven call; an approval pause instead surfaces a pending-call object naming the tool and its arguments. A structured run (one that passed response_format) sets output.content_type (e.g. application/json) so a reader knows how to parse output.content. When the agent offered quick-reply chips on a chat channel, output.suggested_replies lists the labels it presented. usage reports this run's own token counts and, when the model has a price-book entry, its priced cost in your account currency — the run's own line-item, no separate query. (A turn on an unpriced model carries tokens but omits cost.) The usage API is where those are aggregated across runs into your billing summary.

Reads: GET /v1/smiths/{sid}/runs/{rid} (one), GET /v1/smiths/{sid}/runs (per smith), GET /v1/runs?status=&smith_id=&agent_id= (project-wide feed; agent_id pulls every run across one agent's smiths), GET /v1/smiths/{sid}/runs/{rid}/events (the run's events as SSE: those recorded for its current attempt, then each as it is recorded, until the event that finishes or pauses the run; each frame's id: is its position in the run's log, counting from 0, so a reconnect with the Last-Event-ID header continues past the last frame you saw, through a pause and a resume, instead of starting over; ?follow=false answers the whole recorded log and closes). The frames are the stored events, so a terminal one carries usage, which the stream you start with stream: true leaves out.

The log is the record of a run's output, in the order it happened. Text is stored as message.delta rows, one per stretch of text between two other events, so a turn that talks, calls a tool, then talks again replays as message.delta, tool.executing, tool.completed, message.delta. A stored row holds about 300 ms of generation, so a replay has fewer, larger message.delta frames than the live stream. Runs from before the log kept that order hold their text as one message.delta ahead of the run's end, and a run that resumed from an approval before then has lost the text it wrote before the pause.

What the run ran without

warnings is what went wrong during the run without stopping it. It is always present — [] on a run that got everything it asked for — so absence never has to be interpreted:

"warnings": [{ "code": "tool_source_unavailable",
  "message": "MCP server \"librarian\" was skipped: runtime load failed. Its tools did not reach the model." }]

A run whose tools were unreachable still completes. The model answers from what it had left, and that answer is shaped exactly like a real one — so a consumer that branches only on status cannot tell the two apart. Branch on warnings being non-empty when a degraded answer is not acceptable: retry the run, or fail it on your side. The same list rides the run.completed event and stream frame, so a caller watching only the stream sees it too. metadata.tools has the full detail behind each one.

The trace for a run

GET /v1/runs/{run_id}/trace returns the run's execution trace — the timed span waterfall (model calls, tool calls, memory ops) with token counts and cost — so "why was this run slow / what did it cost" is a lookup keyed by the run. Every trace also carries its run_id, so GET /v1/traces?run_id={rid} filters the trace list the same way.

Every run gets one, whichever surface started it — native, Chat Completions, or Responses. A 404 here means the run predates tracing, not that its surface skips it.

Which tools the run resolved

Every run records the tool set it resolved under metadata.tools, so "did my MCP servers actually reach the model?" is a single read:

"metadata": { "tools": {
  "total": 4,
  "mcp": [{ "server": "librarian", "tools": 3 }],
  "hosted": ["web_search"], "deployment": 0, "memory": 0,
  "errors": [{ "server": "acme", "error": "runtime load failed: stored secret could not be decoded" }],
  "skills": [{ "name": "vat-filing", "version": 5 }],
  "skill_errors": [{ "skill": "skl_123", "error": "version 5 does not exist" }]
} }

mcp lists each registered MCP server with the number of its tools that passed the allow-list and reached the model — an empty list or a 0 count is the signal that a server contributed nothing to this run. errors names any registered server the run skipped and why (e.g. a stored secret that can no longer be decoded), so a missing tool reads as a recorded reason rather than a guess — the same failure also flips that server to degraded on GET /v1/tenant/mcp, and surfaces on the run's own warnings.

skills lists each skill that reached the model, with the version it resolved to; skill_errors names any the run could not serve and why — a reference to a deleted skill, a version that no longer exists, or a skill dropped because the run's skills exceeded the search-index ceiling. Both also surface on the run's own warnings.

Every run also records the config it resolved under metadata.config — the agent version and the effective model/instructions/tools it ran with — so a run is self-describing after the fact, and replay can pin it.

Streaming with SSE

A refusal always arrives before the stream opens — as an HTTP status, never as a frame part-way through an answer. When retrying could help, the response carries Retry-After.

"stream": true returns text/event-stream. Every event is one envelope: event: carries the type, data: a JSON object that always includes { "v": 1, "run_id": "…" }. For an in-app chat tab, prefer off-the-shelf SDKs: the OpenAI-compatible API projects this same loop onto the OpenAI Chat Completions wire format, and the Vercel AI SDK adapter wraps that in a streamText/useChat integration. Reach for this native envelope when you want live tool.executing frames, structured output, or server-to-server calls.

The turn runs on the server and the stream reads the run's log, so a client that closes the connection stops the stream and nothing else. The run ends when it finishes, when you cancel it (/submit kind=cancel) or after 15 minutes. Read the rest with GET …/runs/{rid}/events and Last-Event-ID.

Both hosted tools (e.g. web_search) and your MCP tools execute server-side. Ingram Cloud calls them and the stream just keeps going, surfacing the informational tool.executing / tool.completed frames as they happen. Those frames are also mirrored to the run timeline and the event feed, so a run's tool activity stays auditable after the fact. You only act on an approval.required pause.

When a run pauses you first get an approval.required event per pending call and then a single run.paused as the terminal marker. Act on the per-call events; treat run.paused as the state change. The streamed run.completed carries stop_reason, never usage — read the run record for token counts and cost.

One run at a time per smith

A smith runs one turn at a time. Its memory is a single doc shared by every one of its conversations, so two turns in flight would be two writers of one document — the second would silently overwrite what the first learned. Start a run while that smith already has one going and you get 400 with code conversation_locked:

{
  "error": {
    "message": "Another process is currently operating on this conversation. Please retry in a few seconds.",
    "type": "invalid_request_error",
    "code": "conversation_locked"
  }
}

The response carries a Retry-After header saying how long to wait — the running turn ends within its deadline. A different thread_id does not avoid this: the lock is the smith, not the conversation. A run paused awaiting an approval does not hold it, so an unanswered approval never blocks the next message.

Two things are never locked, because neither touches memory: a smith with auto_memory off, and the stateless structured one-shot (response_format with no thread_id). Both stay fully parallel — use them when you want a smith to serve many requests at once.

Autonomous turns — schedules, inbound channel messages — queue instead of failing, and run when the smith is free.

Idempotent run creation

Creating a run costs money. A retry after a network timeout would otherwise start (and bill) a second run. Send an Idempotency-Key header on POST /v1/smiths/{sid}/runs and a replay with the same key returns the original run instead of starting a new one:

# Authorization: tenant-admin token (server-side only), or a smith token
curl https://api.cloud.ingram.tech/v1/smiths/smt_…/runs \
  -H "Authorization: Bearer $IC_TOKEN" \
  -H "IC-Api-Version: 2026-05-01" \
  -H "Idempotency-Key: 8f3c…" \
  -H "Content-Type: application/json" \
  -d '{ "input": [{ "role": "user", "content": "…" }], "stream": true }'

This covers streaming runs too, which are the most likely to time out mid-turn. A retried key does not start a second stream: it returns a single run.duplicate event carrying the existing run's id and current state; reconnect to that run's recorded event log (GET /v1/smiths/{sid}/runs/{rid}/events) or poll GET …/runs/{rid}. A synchronous retry simply returns the original run record. Keys are scoped to your tenant and honoured for 24 hours; use a fresh key per distinct run.

Replay a run

Re-run a recorded run's input through the smith as it stands now:

# Authorization: tenant-admin token (server-side only), or a smith token
curl https://api.cloud.ingram.tech/v1/smiths/smt_…/runs/run_…/replay \
  -H "Authorization: Bearer $IC_TOKEN" \
  -H "IC-Api-Version: 2026-05-01" \
  -H "Content-Type: application/json" \
  -d '{ "stream": false }'

The reply is a fresh run record, carrying the same input, on its own new thread (the original conversation is untouched), with metadata.replay_of set to the source run id. "stream": true streams the replay like any create.

Replay pins the config the original run recorded (metadata.config — the resolved model, instructions, tools, and the agent version in effect then), so a config change since — an agent version rolled forward, the smith's overrides edited — doesn't make the replay diverge. The response says which config it used: metadata.replay.deterministic (true when pinned) and metadata.replay.agent_version. Runs created before config snapshotting shipped carry no snapshot and fall back to the smith's current config (deterministic: false). Memory and model-provider state are always live, not pinned — replay reproduces the config, not the whole world, so output can still differ. A file attachment on the original input is replayed by its file_id and read from storage again; if the file has since been deleted, replay returns 400 file_not_found.

Pause and resume: the universal /submit

One endpoint resumes everything, discriminated by kind:

# Authorization: tenant-admin token (server-side only), or a smith token
curl https://api.cloud.ingram.tech/v1/smiths/smt_…/runs/run_…/submit \
  -H "Authorization: Bearer $IC_TOKEN" \
  -H "IC-Api-Version: 2026-05-01" \
  -H "Content-Type: application/json" \
  -d '{ "kind": "tool_result", "tool_call_id": "tc_…",
        "result": { "events": ["standup at 10:00"] }, "stream": true }'

• kind: "approval_decision": { approval_id, decision: "approve" | "reject", actor }. The common resume: on approve, Ingram Cloud calls your MCP tool and continues; a rejection completes the run with stop_reason: "approval_rejected". Pass "stream": true to pump the continuation back in the same envelope. For a run a schedule, channel message or background response started, /submit returns 202 with the run queued instead of waiting; follow it with GET …/runs/{rid} or the event log. A rejection still completes the run in the response. A queued approval waits for the smith's other turns to finish. For any other run, approving takes the smith's turn slot back: if the smith is running another turn, /submit returns 400 conversation_locked and the run stays paused. Retry. A reject never waits on the slot.

  A rejection is recorded in the conversation, so the smith's next turn on the thread knows the call never ran. On a smith with auto_memory on, a new message on that thread while the run is paused ends it with stop_reason: "approval_superseded", recorded the same way; a later decision on it returns 409 run_not_paused.

  When the approval carries an elicitation (an MCP tool's server asked the end-user a question mid-call), add content: a JSON object matching its requested_schema. approve answers and the tool call retries with the answer; reject sends the server a decline, and the run carries on with whatever the tool then returns rather than ending approval_rejected.

• kind: "tool_result": { tool_call_id, result }, only for an external-execution tool that paused the run (MCP tools run in-process and never need this).

• kind: "cancel": { reason }; the run ends as cancelled and its in-flight model stream is aborted, so token generation — and the billing for it — stops immediately. Allowed from any state, and final: a turn still finishing elsewhere cannot reopen the run as completed, and is not billed. A client that drops the connection mid-stream does not cancel the run.

Generation is bounded either way: a model stream still running after 15 minutes is cut off, ending the run cancelled with stop_reason: "run_timeout".

A run left paused for approval expires after 24 hours: it completes with stop_reason: "approval_expired" and its approval can no longer be submitted. Decide within the window, or start a fresh run.

Structured output

For server-side calls that must return schema-valid JSON (classify, extract, route), pass response_format. On its own — no thread_id, against an agent with no tools of its own — the run is a one-shot model call (no tools, no memory, no streaming) using the smith's configured model:

# Authorization: tenant-admin token (server-side only)
curl https://api.cloud.ingram.tech/v1/smiths/smt_…/runs \
  -H "Authorization: Bearer $IC_TOKEN" \
  -H "IC-Api-Version: 2026-05-01" \
  -H "Content-Type: application/json" \
  -d '{ "input": [{ "role": "user", "content": "<source text>" }],
        "response_format": { "type": "json_schema", "name": "Ticket",
          "strict": true, "schema": { "type": "object", "additionalProperties": false,
            "properties": { "priority": { "enum": ["low", "high"] } },
            "required": ["priority"] } } }'
# → output.content is a JSON *string* that parses and validates

Schema rules. Every object node must set "additionalProperties": false and list all of its keys in "required" — the strict json_schema contract the model provider enforces (strict: true or false makes no difference here). A schema that breaks this is rejected with 422 schema_error, the error message carrying the provider's reason; the run record is marked failed with stop_reason: schema_error. (Generating the schema with z.toJSONSchema satisfies this automatically; the trap is hand-written schemas.) If the model can't produce valid output against a valid schema, the API retries, then returns 500 structured_output_failed.

Use a dedicated smith with auto_memory: false for these utility calls.

With tools and memory. response_format never removes a capability. If the agent has tools — MCP servers, hosted tools, an attached vector store — or you send a thread_id, the turn stays a full agent run (its tools, its memory, as many steps as it needs) and only the final message is typed. The intermediate tool-calling steps are unconstrained, so an unattended run can do real work and still hand back JSON your code can parse. output.content is the JSON string either way, with output.content_type: "application/json"; a turn that can't produce valid output against a valid schema fails with 500 structured_output_failed exactly as the one-shot does. Streaming a typed turn delivers the answer whole — the model's intermediate prose is not streamed, since it isn't the answer.

The same two shapes run on the OpenAI-compatible surfaces: response_format (nested json_schema) on Chat Completions, text.format on Responses.

Inspecting runs in the console

Observe → Runs lists every turn with status, tokens, and cost. A run's detail page opens with the run's identity (status, smith, thread) and its vitals (duration, tokens, cost, spans, model turns), then the timed span waterfall (model calls, tool calls, retrievals, per-span cost) when the run has a trace. Below it a detail band switches between Transcript (the conversation with tool calls), Events (the lifecycle log), and Attributes (run metadata and trace attributes). When a run is paused on an approval.required, the page surfaces the pending tool call and its arguments inline with Approve / Reject — resolving submits the decision on the run's standard tool-call channel and resumes it. Replay (top right) re-runs the same input as a fresh run and opens it. Runs fired from the Playground land here too.

Observe → Threads folds the recent run feed into threads (turns sharing one thread_id) — smith, channel, run count, and last activity — so a thread is browsable on its own; a row opens its latest run.