Modes

Set mode once in your config:

const config = {
  mode: "call", // "call" | "tts" | "audio"
  // ...
};

call — Live voice conversation

The default. Bidirectional voice over WebSocket. Your user speaks, the avatar listens, thinks, and responds — all in real time.

<AvatarProvider
  config={{
    getSessionToken: async () => fetchSessionToken(),
    mode: "call",
    avatar: {
      avatarId: "default",
      systemPrompt: "Returning customer, has an open ticket about billing.",
    },
  }}
>
  <div style={{ width: 400, height: 400 }}>
    <Avatar />
  </div>
</AvatarProvider>

How it works: the SDK captures the user's microphone and streams it to the server. The server understands what was said, generates a reply, and sends back the spoken audio along with the timing data that drives lip-sync.

Best for: Support agents, AI companions, interactive tutors.

Turn taking

Inside call mode, a second choice: who decides a turn is over. This is session wiring, not part of the avatar — the same avatar can run hands-free in one app and push-to-talk in another.

turnTaking: { mode: "auto" }          // the server's endpointer decides, from the audio
turnTaking: { mode: "push-to-talk" }  // the user decides, by holding a button

Push-to-talk is deliberately forgiving at both ends: 200 ms of already-captured audio is released on press, because people start talking a hair before their thumb lands, and 250 ms of real audio is still sent after release, so a syllable the user let go on is not cut off. A hold longer than two minutes is treated as a stuck key rather than a person, and the turn is closed.

The built-in controls render the hold-to-talk button for you. To build your own, see useAvatar → Push to talk.

Camera

call mode can also let the avatar see, if you opt in with perception: { camera: true }. The camera stays closed until the avatar actually needs to look at something. See Camera perception.

tts — Text-to-speech

Send text programmatically — the avatar speaks it. No microphone, no voice input.

function TTSControls() {
  const { speakText, start, isReady } = useAvatar();

  return (
    <div>
      <button onClick={start}>Connect</button>
      <button
        onClick={() => speakText?.("Hello! Welcome to our platform.")}
        disabled={!isReady}
      >
        Speak
      </button>
    </div>
  );
}

The server processes the text, generates audio, and returns it with alignment data for lip-sync.

Best for: Announcements, notifications, narration, scripted onboarding flows.

audio — Bring your own audio

You already have audio (from another TTS provider, a pre-recorded file, or a custom pipeline). Send it to the avatar to lip-sync and play.

function AudioControls() {
  const { speakAudio, start, isReady } = useAvatar();

  const handleFile = async (file: File) => {
    const buffer = await file.arrayBuffer();
    speakAudio?.(buffer);
  };

  return (
    <div>
      <button onClick={start}>Connect</button>
      <input
        type="file"
        accept="audio/*"
        onChange={(e) => e.target.files?.[0] && handleFile(e.target.files[0])}
        disabled={!isReady}
      />
    </div>
  );
}

Accepted input types: ArrayBuffer, Float32Array, Blob.

Best for: Custom TTS pipelines (ElevenLabs, Play.ai, etc.), pre-recorded content with dynamic delivery.

Mode comparison