Skip to content

feat(soniox): add TTS speed support#2005

Open
rosetta-livekit-bot[bot] wants to merge 8 commits into
mainfrom
meters-crane-brazen
Open

feat(soniox): add TTS speed support#2005
rosetta-livekit-bot[bot] wants to merge 8 commits into
mainfrom
meters-crane-brazen

Conversation

@rosetta-livekit-bot

@rosetta-livekit-bot rosetta-livekit-bot Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add Soniox TTS support to the JS plugin, including the Python source PR's speed option, [0.7, 1.3] validation range, runtime updates, and WebSocket request field.
  • Export the TTS API and update the API report.
  • Make pooled request cleanup deterministic across completion, provider errors, cancellation, abort, and retry.
  • Add an exactly-once base SynthesizeStream completion hook so provider ownership spans the entire logical retry lifecycle.
  • Add focused request-shaping, retry-output, cancellation, and lifecycle regression coverage with a patch changeset.

Source diff coverage

The source change from livekit/agents#6339 adds bounded Soniox TTS speed configuration. The JS port preserves its default, validation, update behavior, and initial WebSocket configuration field. JS-specific pooled WebSocket ownership and base-class retry behavior are covered by focused tests.

The Python implementation can pass through encoded formats. The JS TTS interface emits decoded AudioFrames, so this port explicitly accepts pcm_s16le and rejects unsupported formats instead of silently decoding them incorrectly.

Review fixes

  • Reject non-finite speed values.
  • Unregister chunked and streaming requests on normal and error paths.
  • Settle cancellation locally when a provider never sends a terminal response.
  • Remove settled requests from the owning TTS state exactly once after the full logical stream lifecycle.
  • Replay cached streaming input on retry without conflating attempt cleanup with user cancellation.
  • Preserve consumer-visible retry audio by leaving queue/output termination to the base lifecycle.
  • Keep active retries owned so TTS.close() cancels and settles them before pooled socket shutdown.

Verification

  • Soniox TTS/STT tests: 42 passed.
  • Focused base and Soniox lifecycle tests: 32 passed.
  • Repository build: 37/37 successful.
  • Repository lint: 37/37 successful; unrelated existing warnings only.
  • Repository format check: passed.
  • Soniox API/type check: passed.
  • Independent full-range correctness and architecture reviews: approved with no Critical or Important findings.

Cue voice E2E

Verified at exact PR head 0d19988a469404d6ed2327bcecd70f0164871ab8 in voice mode with a disclosed local protocol-compatible Soniox WebSocket boundary because SONIOX_API_KEY was unavailable.

  • Session: sid_ec11dfb4a2b4.
  • Input: Please verify the Soniox speaking speed.
  • Completion predicate waited for final user transcription, AS_SPEAKING, the assistant conversation item, and return to AS_LISTENING/AS_IDLE; worker output independently reported playout completed without interruption.
  • Result: resolved in 5,556 ms, exit code 0.
  • Boundary captured speed: 1.3, 24 kHz pcm_s16le, 48,000 PCM bytes, terminal response, and socket cleanup; SHA-256 8b3e603753d90668c5890686a4fc550367eecfe243d4d709ae87d3a7afcb087f.
  • Framework events included final STT, thinking, Soniox TTS usage with one second of audio, speaking, assistant output, and return to listening.
  • Session WAV: stereo PCM16, 48 kHz, 16.114917 s, both channels nonzero, SHA-256 04a15610aa43d6ee8ee7537c254e1d6ff60add2452b62bb464022dba2fe2b398.
  • Command WAV: stereo PCM16, 48 kHz, 5.599938 s, both channels nonzero, SHA-256 a477293786c1eba1ca0a90d8fb4740665f5faceb93971f71192e3882cbf100e0.
  • events.jsonl SHA-256: aaaaf5aa43021a91888b04bdc0b36e3a7775c7abcf8223b8d09f813df091b871.
  • result.json SHA-256: 12ef54ef40dee1946473821ee5eecc4481916bc42ac101b23afa7f2d62997ad1.

Ported from livekit/agents#6339

@rosetta-livekit-bot rosetta-livekit-bot Bot requested a review from a team as a code owner July 9, 2026 05:37
@changeset-bot

changeset-bot Bot commented Jul 9, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 0d19988

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 36 packages
Name Type
@livekit/agents-plugin-soniox Patch
@livekit/agents Patch
@livekit/agents-plugin-anam Patch
@livekit/agents-plugin-anthropic Patch
@livekit/agents-plugin-assemblyai Patch
@livekit/agents-plugin-baseten Patch
@livekit/agents-plugin-bey Patch
@livekit/agents-plugin-cartesia Patch
@livekit/agents-plugin-cerebras Patch
@livekit/agents-plugin-deepgram Patch
@livekit/agents-plugin-did Patch
@livekit/agents-plugin-elevenlabs Patch
@livekit/agents-plugin-fishaudio Patch
@livekit/agents-plugin-google Patch
@livekit/agents-plugin-hedra Patch
@livekit/agents-plugin-hume Patch
@livekit/agents-plugin-inworld Patch
@livekit/agents-plugin-lemonslice Patch
@livekit/agents-plugin-liveavatar Patch
@livekit/agents-plugin-livekit Patch
@livekit/agents-plugin-minimax Patch
@livekit/agents-plugin-mistral Patch
@livekit/agents-plugin-mistralai Patch
@livekit/agents-plugin-neuphonic Patch
@livekit/agents-plugin-openai Patch
@livekit/agents-plugin-perplexity Patch
@livekit/agents-plugin-phonic Patch
@livekit/agents-plugin-resemble Patch
@livekit/agents-plugin-rime Patch
@livekit/agents-plugin-runway Patch
@livekit/agents-plugin-sarvam Patch
@livekit/agents-plugin-silero Patch
@livekit/agents-plugin-tavus Patch
@livekit/agents-plugin-trugen Patch
@livekit/agents-plugin-xai Patch
@livekit/agents-plugins-test Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

devin-ai-integration[bot]

This comment was marked as resolved.

toubatbrian and others added 3 commits July 15, 2026 09:06
Reject non-finite speeds and release completed or failed streams so pooled connections and streaming inputs cannot leak.

Co-authored-by: Cursor <cursoragent@cursor.com>
Release local waiters when callers cancel so nonresponsive servers cannot retain pooled stream state.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
devin-ai-integration[bot]

This comment was marked as resolved.

toubatbrian and others added 4 commits July 15, 2026 10:21
Keep provider-attempt cleanup from terminating consumer output before the base retry lifecycle completes.

Co-authored-by: Cursor <cursoragent@cursor.com>
Keep failed attempts owned until retry completion and drain cancellation before closing the pooled socket.

Co-authored-by: Cursor <cursoragent@cursor.com>
Distinguish recoverable attempts from terminal failures so retries remain cancellable without retaining completed streams.

Co-authored-by: Cursor <cursoragent@cursor.com>
Centralize provider cleanup after the base retry lifecycle so plugins do not duplicate private attempt policy.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant