Realtime Tools cause agent response to cut off

[aligokalppeker]

Describe the bug

In Agent SDK, when a tool is registered to a RealtimeAgent, the tool's execution triggers a "create.response" event. That trigger mostly works correctly. But when there is a response in progress, the tool originated "create.response" event causes a cut-off with a RealtimeError.

RealtimeError(message='Conversation already has an active response in progress: resp_DUvxzRrEe3YZSdwIqBdVn. Wait until the response is finished before creating a new one.', type='invalid_request_error', code='conversation_already_has_active_response', event_id=None, param=None)

This causes the agent to get stuck and wait for the customer request to respond.

When a tool is called in the agent, the following is the path that results in the agent response for v0.13.0:

agents.realtime.session.RealtimeSession._enqueue_tool_call_task
agents.realtime.session.RealtimeSession._handle_tool_call
agents.realtime.openai_realtime.OpenAIRealtimeWebSocketModel.send_event
agents.realtime.openai_realtime.OpenAIRealtimeWebSocketModel._send_tool_output

As there are limitless scenarios that can cause this, it primarily happens within the following case;

• Async tools calls, where outputs are produced simultaneously.
• When "create.response" is triggered outside the agent, and then the agent runs the tool

This process is improved with v0.13.1, but edge cases seem to exist.

Debug information

• Agents SDK version in production: v0.14.2
• Python version: Python 3.11

Repro steps

Trigger multiple tools in an async fashion where each one generates an output.

Expected behavior

Tool-triggered "create.response" messages should be scheduled to occur after the response is completed if a response is in progress.

[nileshpatil6]

Hi! I've traced through the code and would like to work on this.

After reading through the sequencer in openai_realtime.py, the _send_response_create_when_idle / wait_for_response_create_slot machinery handles the serialization of concurrent response.create sends correctly in the common case. However I can see two edge cases where it may still fire prematurely:

1. Race between response.done and the next reserve_response_create_request: If response.done arrives and clears _ongoing_response before a second async tool output has had a chance to call reserve_response_create_request, a background task from a concurrent source (e.g. an external response.create sent via send_event) can slip through the now-free slot and conflict with the tool's pending create.

2. async_tool_calls=True + function_call events arriving mid-response: Tool tasks are spawned immediately on each function_call event (before response.done), but they may complete and attempt response.create before the model response is fully finished, relying entirely on _ongoing_response being accurate — which may lag if response.created and response.done arrive out of order under heavy load.

Would a PR be welcome? My proposed fix is to track the number of outstanding tool-call tasks that have reserved a response.create slot and only allow the sequencer to dispatch after all active-response tool tasks have registered their versions — preventing any external or concurrent response.create from slipping in between them.

[seratch]

Thanks for the report. I took a first look against v0.14.2.

One important detail: the code snippet in the issue does not appear to match the v0.14.2 implementation. In v0.14.2, _send_tool_output() does not send response.create directly after the tool output. It reserves a response-create request and starts the deferred response-create path:

• _send_tool_output() -> _reserve_response_create_request() -> _start_response_create()
• _send_response_create_when_idle() then waits on _ResponseCreateSequencer.wait_for_response_create_slot()
• that wait condition requires no ongoing response before sending the actual response.create

This behavior is covered by existing tests, including the cases where tool outputs complete while a response is active and where multiple tool outputs are queued for the same turn. Those tests assert that response.create is not sent until after response.done, and that stacked tool outputs coalesce to a single follow-up response.create.

So I do not think the issue, as currently described, is enough to identify a v0.14.2 SDK bug. The reported server error is valid, but it may be coming from a different path, for example:

• a raw response.create being sent outside the SDK's send_event() / sequencer path
• another client or external layer creating a response on the same conversation
• an older SDK version or locally modified code path
• an event ordering case not covered by the current reproduction description

Could you share a minimal runnable repro, plus the ordered client/server event log around the failure? The most useful log would include outgoing conversation.item.create, outgoing response.create, incoming response.created, incoming response.done, and the incoming error, including event_id values where present.

For the proposed fix in the follow-up comment: I would hold off on adding another outstanding-tool-call counter until we have a concrete repro. The current sequencer already tracks pending response-create requests and coalesces automatic follow-up creates, so an additional tool-task gate may solve a different race than the one actually happening here.

[aligokalppeker]

Thx for the details @seratch. I updated the trace with v0.13.0; it seems to be improved with the recent Agent SDK updates.

But the solution in the agent's response to events can have edge cases. The "deferred" solution might be inspected in detail.

As I give the traces of the version v0.13.0, the Agent SDK library version used in production is the latest version. We will investigate the issue with the latest version deployed in the system.

The code to reproduce this issue is straightforward, but it may not be easy to create the edge cases. I will inspect #2763 with respect to our final results.

[aligokalppeker]

@seratch In here, another additional concern is the tool calls to create a response when they are executed. Response trigger at the end of the tool should be provided as an option to the SDK user; not every tool result is required to create a response, so when creating a tool, there should be an option to determine whether or not to trigger a response.