RealtimeAudioClient
Interface for realtime voice AI model connections.
Lifecycle:
- connect(config) — establish WebSocket, send initial config
- sendAudio(frame) — stream user audio to model
- on(‘tool-call’, …) — handle tool calls from model
- sendToolResponse(…) — return tool results
- on(‘audio’, …) — receive model audio output
- updateConfig(…) — change prompt/tools mid-session (e.g., on flow transition)
- disconnect() — close cleanly
NOTE: For SDK-based implementations (e.g. @google/genai), session.receive()
is an async iterator that completes after each turn and must be re-entered
inside a while(true) loop. Raw-wire implementations (fetch()+Upgrade on
Cloudflare Workers, direct ws WebSocket usage) do not have this quirk; the
underlying WebSocket stays open for the session lifetime. Future authors
porting this interface to a new transport should be aware that the SDK
wrapper and the raw-wire path diverge at this seam.
Properties
Section titled “Properties”capabilities
Section titled “capabilities”readonly capabilities: RealtimeCapabilities;Capability flags. Static per implementation; declared at construction. Implementations MUST NOT throw from this accessor.
connected
Section titled “connected”readonly connected: boolean;Whether currently connected to the model.
readonly model: string;Model identifier configured for this client, e.g. "gpt-realtime".
provider
Section titled “provider”readonly provider: string;Short stable identifier for the provider, e.g. "gemini", "openai".
Methods
Section titled “Methods”connect()
Section titled “connect()”connect(config): Promise<void>;Connect to the AI service with initial configuration.
Parameters
Section titled “Parameters”| Parameter | Type |
|---|---|
|
|
Returns
Section titled “Returns”Promise<void>
disconnect()
Section titled “disconnect()”disconnect(): Promise<void>;Disconnect gracefully.
Returns
Section titled “Returns”Promise<void>
off<K>(event, handler): void;Unsubscribe from events.
Type Parameters
Section titled “Type Parameters”| Type Parameter |
|---|
|
|
Parameters
Section titled “Parameters”| Parameter | Type |
|---|---|
|
|
|
|
|
Returns
Section titled “Returns”void
on<K>(event, handler): void;Subscribe to events from the model.
Type Parameters
Section titled “Type Parameters”| Type Parameter |
|---|
|
|
Parameters
Section titled “Parameters”| Parameter | Type |
|---|---|
|
|
|
|
|
Returns
Section titled “Returns”void
ping()
Section titled “ping()”ping(): Promise<boolean>;Check connection health (WebSocket ping).
Returns
Section titled “Returns”Promise<boolean>
requestResponse()?
Section titled “requestResponse()?”optional requestResponse(instruction?): void;Trigger a model response after config update. Optional — not all providers support it.
Parameters
Section titled “Parameters”| Parameter | Type |
|---|---|
|
|
|
Returns
Section titled “Returns”void
sendAudio()
Section titled “sendAudio()”sendAudio(frame): void;Send a PCM audio frame to the model.
Parameters
Section titled “Parameters”| Parameter | Type |
|---|---|
|
|
|
Returns
Section titled “Returns”void
sendToolResponse()
Section titled “sendToolResponse()”sendToolResponse(responses): void;Send tool call results back to the model.
Parameters
Section titled “Parameters”| Parameter | Type |
|---|---|
|
|
Returns
Section titled “Returns”void
updateConfig()
Section titled “updateConfig()”updateConfig(config): Promise<void>;Update session configuration mid-call (system prompt, tools). Used when CapabilityHost signals ‘reconfigure’ after a flow transition.
Implementation may:
- Send in-place update (if model supports it)
- Disconnect and reconnect with new config (fallback)
- Use session resumption handle for continuity
Parameters
Section titled “Parameters”| Parameter | Type |
|---|---|
|
|
|
Returns
Section titled “Returns”Promise<void>