Parallel & Sequential Tool Calls

Parallel tool use is on by default on current models. When the requests are independent — three lookups, four file reads, a fan-out across candidates — the model emits them together, and the latency win is real: three sequential 400ms calls plus two extra model round trips becomes one 400ms wait.

The rule that matters is the message shape. All results go back in one user message, one tool_result block per tool_use_id. Splitting them across several user messages is accepted by the API but changes behaviour — it trains the model, within that conversation, to stop making parallel calls, and the effect is invisible unless you are watching traces. Partial returns are worse: every requested call needs a result, and a failed one returns an error result rather than being omitted.

Sequential calls are the other half, and they are sequential for a reason — the second call's arguments depend on the first call's output. Look up the customer, then fetch their orders. That dependency costs a full model round trip per step, which is where agent latency actually goes: not in your tools, but in the model turns between them.

Two ways to compress that. Give the model tools at the right altitude — a single get_customer_orders_by_email beats forcing two round trips through find_customer then get_orders. Coarse-grained tools that match real tasks are faster and more reliable than a fine-grained API surface faithfully mirrored into tool definitions.

Programmatic tool calling goes further: the model writes a script that invokes tools in a sandbox, with loops and branching, and only the final output returns to the context. Intermediate results never enter the window at all. For "check these forty items and tell me which three failed", that is the difference between forty round trips filling the context and one.

Concurrency is your responsibility, and so are its consequences. Read-only tools are safe to run in parallel; writes may not be. If two calls in one turn touch the same resource, you need ordering or locking — the model has no idea your tools contend.

Latency is the main complaint about agentic features, and most of it is model round trips rather than tool execution — so knowing that the levers are parallelism, tool granularity, and programmatic calling is a concrete performance answer rather than a vague one. The single-message rule is also the kind of subtle protocol detail that quietly degrades a system: performance regresses over a conversation and nobody can explain why.

• Parallel tool use is on by default; independent calls arrive together in one assistant turn.
• Return all results in a single user message with matching tool_use_ids — splitting them across messages suppresses future parallel calls.
• Every requested call needs a result, including failures, which return an error result rather than being dropped.
• Sequential chains cost a full model round trip per step; that, not tool execution, is where agent latency accumulates.
• Design tools at task altitude rather than mirroring your REST endpoints — one coarse tool often replaces a three-step chain.
• Programmatic tool calling lets the model script tool invocations in a sandbox so intermediate results never enter the context.
• You own concurrency: parallelise reads freely, and add ordering or locking when two calls in a turn touch the same resource.
• Disable parallel calls deliberately when correctness requires strict ordering — it is a supported option, not something to fake in the prompt.