Build a conversation.
Make it yours.

Connect your app, robot or device to a voice agent. Stream speech, bring conversation history and integrate the tools your application needs.

Voice API · Developer guideYOUR API ORIGIN
S Sophea Developers
01

Get a key

Your workspace owner provides a key and API URL. Use both from the same service.

02

Choose your agent

Sophea assistant keys use a published agent. Keys for your own agent let your app supply instructions and functions.

03

Start streaming

Use the Realtime WebSocket interface for SDK integration, or the direct device protocol below.

JAVASCRIPT QUICKSTART

Connect with JavaScript

Run on Node.js with npm install ws. Save the example as realtime-node.mjs, set SOPHEA_URL to the matching HTTPS API origin and SOPHEA_DEVICE_KEY to your key, then run node realtime-node.mjs. Both connection types work with the matching URL shown beside your key.

This example keeps the service or published assistant’s voice and language settings, sends text and saves a playable reply.wav. For voice input, set SOPHEA_INPUT_PCM to a raw mono PCM16, 24 kHz file. It sends audio at real-time speed and reports first received audio plus server timing metrics. For a private CA, set NODE_EXTRA_CA_CERTS to the trusted CA file.

JavaScript · Node.js · Realtime
// npm install ws
// Set SOPHEA_URL (HTTPS origin) and SOPHEA_DEVICE_KEY, then:
// node examples/realtime-node.mjs
// Optional: SOPHEA_INPUT_PCM=/path/to/mono-24000hz-s16le.pcm
import WebSocket from 'ws';
import { readFileSync, writeFileSync } from 'node:fs';
import { setTimeout as sleep } from 'node:timers/promises';

if (!process.env.SOPHEA_URL || !process.env.SOPHEA_DEVICE_KEY) {
  throw new Error('Set SOPHEA_URL and SOPHEA_DEVICE_KEY.');
}
const url = new URL(process.env.SOPHEA_URL.replace(/\/$/, '') + '/v1/realtime');
if (url.protocol !== 'https:') throw new Error('Use a trusted HTTPS origin.');
url.protocol = 'wss:';
url.searchParams.set('model', 'sophea-realtime');
const input = process.env.SOPHEA_INPUT_PCM
  ? readFileSync(process.env.SOPHEA_INPUT_PCM) : null;
if (input && (input.length < 4800 || input.length > 30 * 48000 || input.length % 2)) {
  throw new Error('Input must be 0.1–30 seconds of raw mono PCM16 at 24 kHz, without a WAV header.');
}
const ws = new WebSocket(url, {
  headers: { Authorization: `Bearer ${process.env.SOPHEA_DEVICE_KEY}` },
  handshakeTimeout: 15000,
});
const send = event => ws.send(JSON.stringify(event));
const chunks = [];
let started = 0, firstAudioMs = null, finished = false, submitted = false;
function fail(message) {
  if (finished) return;
  finished = true;
  console.error(message);
  process.exitCode = 1;
  ws.terminate();
}
const deadline = setTimeout(() => fail('Conversation timed out.'), 120000);
ws.on('error', () => fail('Connection failed. Check the URL, key and trusted CA.'));
ws.on('close', () => {
  clearTimeout(deadline);
  if (!finished) fail('Connection closed before the reply completed.');
});
ws.on('message', async raw => {
  try {
    const event = JSON.parse(raw.toString());
    if (event.type === 'session.created') {
      send({ type: 'session.update', session: {
        audio: { input: { turn_detection: null } },
        // Keys for your own agent: add instructions and tools here if needed.
        // Omitted instructions, voice and language retain the service/assistant defaults.
      } });
    } else if (event.type === 'session.updated' && !submitted) {
      submitted = true;
      if (input) {
        for (let offset = 0; offset < input.length; offset += 960) {
          if (finished) return;
          send({ type: 'input_audio_buffer.append',
            audio: input.subarray(offset, offset + 960).toString('base64') });
          await sleep(20); // 20 ms of audio, paced in real time.
        }
        send({ type: 'input_audio_buffer.commit' });
      } else {
        send({ type: 'conversation.item.create', item: {
          type: 'message', role: 'user',
          content: [{ type: 'input_text', text: 'Hello!' }],
        } });
      }
      started = Date.now();
      send({ type: 'response.create' });
    } else if (event.type === 'response.output_audio.delta') {
      firstAudioMs ??= Date.now() - started;
      chunks.push(Buffer.from(event.delta, 'base64'));
    } else if (event.type === 'response.output_audio_transcript.delta') {
      process.stdout.write(event.delta);
    } else if (event.type === 'error') {
      fail(event.error?.message || 'The API rejected the request.');
    } else if (event.type === 'response.done') {
      if (event.response.status !== 'completed' || !chunks.length) {
        fail('Reply did not complete with audio.');
        return;
      }
      const pcm = Buffer.concat(chunks);
      const header = Buffer.alloc(44);
      header.write('RIFF', 0); header.writeUInt32LE(36 + pcm.length, 4);
      header.write('WAVEfmt ', 8); header.writeUInt32LE(16, 16);
      header.writeUInt16LE(1, 20); header.writeUInt16LE(1, 22);
      header.writeUInt32LE(24000, 24); header.writeUInt32LE(48000, 28);
      header.writeUInt16LE(2, 32); header.writeUInt16LE(16, 34);
      header.write('data', 36); header.writeUInt32LE(pcm.length, 40);
      writeFileSync('reply.wav', Buffer.concat([header, pcm]));
      console.log('\nSaved reply.wav', {
        firstReceivedAudioMs: firstAudioMs,
        serverMetrics: event.response.metadata,
      }); // Receive timing does not include speaker playback.
      finished = true;
      clearTimeout(deadline);
      ws.close();
    }
  } catch {
    fail('Could not process this response or write reply.wav.');
  }
});

This is server/device JavaScript. Keep the key in your backend or trusted device. Browser apps should connect through your authenticated backend; do not ship a permanent key in browser code. Sophea does not provide the OpenAI ephemeral-key or WebRTC endpoints.

QUICKSTART

Connect with Python

Install openai[realtime], for example with python -m pip install "openai[realtime]". Set SOPHEA_URL to your API origin and SOPHEA_DEVICE_KEY to your key. Use trusted HTTPS. This example sends text and saves the agent’s reply as playable audio.

Python · Realtime
import asyncio, base64, os, wave
from openai import AsyncOpenAI

async def main():
    async with AsyncOpenAI(
        api_key=os.environ["SOPHEA_DEVICE_KEY"],
        base_url=os.environ["SOPHEA_URL"].rstrip("/") + "/v1",
    ) as client:
        async with client.realtime.connect(model="sophea-realtime") as connection:
            await connection.session.update(session={
                "audio": {"input": {"turn_detection": None}}
            })
            await connection.conversation.item.create(item={
                "type": "message", "role": "user",
                "content": [{"type": "input_text", "text": "Hello!"}]
            })
            await connection.response.create()
            with wave.open("reply.wav", "wb") as output:
                output.setnchannels(1)
                output.setsampwidth(2)
                output.setframerate(24000)
                async for event in connection:
                    if event.type == "response.output_audio.delta":
                        output.writeframes(base64.b64decode(event.delta))
                    elif event.type == "error":
                        raise RuntimeError(event.error.message)
                    elif event.type == "response.done":
                        if event.response.status != "completed":
                            raise RuntimeError("Response did not complete")
                        break

asyncio.run(main())

This API implements an OpenAI Realtime GA WebSocket subset. Use model sophea-realtime and Sophea voice names. WebRTC, SIP, semantic VAD, images are not supported.

Voice, language & instructions

Choose how your assistant sounds, which language it replies in, and how it behaves. Configure these settings with session.update.

1. Choose a voice

Request GET /api/platform/voice-api/voices with Authorization: Bearer YOUR_KEY. Use the returned catalogue to choose a voice and check its supported languages before the first audio response.

  • Multilingual voices: choose a voice and language from the catalogue.
  • For Greek, select Maria and language el.
  • Use voices_by_language to find available voices for each language.

Choose the voice before the first audio response. It stays fixed during the session.

Use a saved designed voice
  1. Create a preview in the platform’s Voice Designer, name it and choose Save voice.
  2. Find saved voice names in voice_labels.
  3. Set session.audio.output.voice to the saved custom_… ID.

Use only the languages listed for your saved voice.

Removing a saved voice makes it unavailable for future sessions and Realtime turns. Update any assistants using it before removal.

2. Configure a session

These examples select Nikos with automatic reply-language selection. Replace the voice with an ID from your catalogue.

Python · session settings
await connection.send({
    "type": "session.update",
    "session": {"audio": {"output": {"voice": "Nikos"}}},
    "extra_body": {"language": "auto", "speech_effects": "expressive"}
})
JavaScript · session settings
send({
  type: "session.update",
  session: { audio: { output: { voice: "Nikos" } } },
  extra_body: { language: "auto", speech_effects: "expressive" }
});

extra_body is a Sophea event extension. Send it with Python connection.send or the JavaScript send helper from the JavaScript quickstart.

Update settings between responses, after clearing or committing pending audio. Language state belongs to the connection; reapply any required language setting after reconnecting.

Emotions and vocal effects

Enabled by default. New sessions without an explicit mode use expressive: the assistant chooses emotional tone and may add appropriate vocal effects. You do not need to enable it or put emotion tags in your instructions. Effects are used selectively, not in every reply.

  • expressive (default): automatic emotional tone and context-appropriate effects.
  • requested: automatic emotional tone; effects only when explicitly requested.
  • off: no vocal effects; emotional tone remains automatic.

Set extra_body.speech_effects using either session-settings example above. Supported effects include laughter, sighs, pauses, whispering and crying, depending on the voice. Requests to stop an effect take priority. Your selected voice stays the same, and private directions never appear in response.output_audio_transcript.delta.

The Python Realtime SDK's connection.session.update() helper has no extra_body parameter. Use connection.send(...) as shown above. The resolved mode is returned as session.speech_effects in session.created and session.updated.

For HTTP session creation, send {"extra_body":{"speech_effects":"expressive"}} in the JSON body. Explicit settings are preserved; omitting the field in an update keeps the current mode. Conflicting values are rejected. To use the new default after reconnecting, omit any older explicit requested setting in your client or assistant configuration.

3. Set the reply language

Your goalSettingBehavior
Follow the speakerlanguage: "auto"Auto follows supported language changes without changing the voice.
Keep one languageA specific code, such as "en" or "el"Fixes the reply language. Greek uses Maria.
Reset a spoken preferenceSet language: "auto" again between turnsClears a request such as “reply in Chinese”. The speaker can also say “follow my language again”.

Set extra_body.language. Select Greek explicitly.

How automatic language switching behaves
  • Explicit requests persist: “reply in Chinese” keeps that reply language even if the user continues speaking Greek, until the preference is cleared.
  • Clear greetings and corrections can switch language: for example, “hi” and “I said HI”.
  • Ambiguous speech keeps the current language: acknowledgements such as “OK” and unclear mixed speech do not force a switch.
  • When a reply cannot be completed: the assistant may ask you to try again. Test language changes with representative speech.

Accuracy depends on the recognizer’s transcript. Automatic switching does not guarantee word-by-word mixed-language transcription.

4. Add instructions & conversation history

Instructions define behavior. Set session.instructions through session.update to customize the assistant. Omit it to keep the published assistant’s instructions. Voice pacing and tool permissions still apply.

History provides context. Add previous messages with conversation.item.create before response.create:

  • User messages use text content with input_text.
  • Assistant messages use text content with output_text.

Use session.update for session-wide instructions, rather than adding them as conversation history.

Diagnostics & timing reference

Check the status in response.done before treating a response as complete. Measure the time to received audio and audible playback in your client.

Server timing metadata, when present, does not include your device’s playback buffering. Timing fields can overlap and should not be added together. Keep diagnostics free of API keys, session tickets and conversation content when sharing a support report.

Stream audio in both directions

Realtime eventClient behavior
input_audio_buffer.appendSend base64 mono PCM16 at 24,000 Hz, paced in real time.
input_audio_buffer.commitIn manual mode, commit at least 100 ms of audio, then send response.create.
response.output_audio.deltaDecode base64 and play mono PCM16 at 24,000 Hz.
response.output_audio_transcript.deltaAppend spoken text to your transcript.
response.cancelCancel generation and discard queued playback. Truncate the assistant audio item to the milliseconds actually played.
response.doneCheck status. Timing metadata reports server-send latency; measure playback latency in your client.

For automatic turn detection, use audio.input.turn_detection: {"type":"server_vad"}. On input_audio_buffer.speech_started, stop queued playback and truncate the previous assistant audio to what was heard. Do not also create manual responses for VAD-triggered turns.

Let the agent call your functions

Keys for your own agent support client-side functions supplied in session.tools as flat objects with type: "function", name, description and a JSON parameters schema. Sophea assistant keys use the published agent’s server tools.

When a completed response contains a function_call, validate its name and JSON arguments against your own allowlist, run the function, and return its result:

Return a function result
await connection.conversation.item.create(item={
    "type": "function_call_output",
    "call_id": call.call_id,
    "output": json.dumps(result)
})
await connection.response.create()

Import json in your client. Handle each call ID once. Results belong to the connection that requested them. Return results within 30 seconds; a timeout does not prove an external action failed, so do not blindly retry side effects. Up to 16 functions and 16,000 characters per result are supported.

Direct device protocol

Use this alternative if your device sends binary audio without the Realtime SDK. Create a session through the reference below, then connect to the returned ws_path on this origin with ?ticket= plus the URL-encoded ws_ticket. Connect before expires_in; tickets are single-use secrets.

This protocol uses 16,000 Hz mono PCM16 binary frames, not the Realtime interface’s 24 kHz base64 audio. Prefer 20 ms / 640-byte frames at real-time speed. Parse text frames as JSON events and play binary frames as audio.

Instructions, history and device function results

Send instructions or leading system/developer messages, not both. History uses chronological role/content messages. Set respond_on_connect: true to answer a final user message immediately. Otherwise the agent waits for input. History suppresses the opening greeting.

Keys for your own agent accept tools in OpenAI-style {"type":"function","function":{...}} wrappers. Execute only function_call events, returning {"type":"function_result","call_id":"...","result":{...}} within 30 seconds. Observational tool_call events are not execution requests. Sophea assistant keys run published server tools.

session_metrics and turn_metrics measure server timing, not device playback. Close the WebSocket to end the session.

Handle errors and limits

Keys are scoped to their service and can expire or be revoked. Keep them out of URLs, logs and browser bundles. Reconnect with a fresh session after a connection closes.

StatusWhat to do
401 / 403Check the key, expiry and API origin. Both key types use their matching service.
409The selected assistant must be active and published.
422 / error eventCorrect the request using its validation message. Do not retry unchanged input.
429Honor Retry-After when present, use backoff, and end unused sessions. Limits depend on your account.
503Retry connection setup with backoff. Never replay external actions automatically.

Sessions last at most 30 minutes and close after 60 seconds without input. Keep audio paced in real time. Direct frames are limited to 32,000 bytes and 100 frames/second. Keep Realtime JSON messages below 256 KiB and send short audio chunks. Close codes: 4401 invalid access, 4408 timeout, 4409 audio limit, 4429 throttled, 1009 message too large.

EXPLORE

API reference

Authorize with your key, without the Bearer prefix. Credentials stay in memory and clear on reload. Creating a session uses account quota. Try it out does not connect a microphone. Use the SDK for the WebSocket endpoint.