View as Markdown Open in ChatGPT Open in Claude

useAvatar

Control the avatar and read its live status from your own components.

useAvatar gives you everything you need to build custom controls: methods to start, stop, mute, and send speech, plus live status flags that update as the session changes.

Call it inside any component nested under <AvatarProvider>.

tsx

"use client";
import { useAvatar } from "@avatarfactory/react";

function Controls() {
  const { start, stop, isIdle, isConnecting } = useAvatar();

  if (isConnecting) return <button disabled>Connecting…</button>;

  return (
    <button onClick={isIdle ? start : stop}>
      {isIdle ? "Start conversation" : "End session"}
    </button>
  );
}

That button is correct from the first render. It says "Start conversation" while the avatar is still loading — and it works, because start() waits the load out for you.

State

Two independent axes. Never infer one from the other: isIdle does not mean there is no avatar, and isReady does not mean you are in a call. See State & Lifecycle.

The avatar (paint axis)

Prop	Type	Default	Description
`isReady`Optional	`boolean`	No default	The avatar is on screen and animating. Unaffected by a session starting or ending.
`isLoading`Optional	`boolean`	No default	The avatar is still coming up — resolving config, or resolved with the canvas unfinished.
`isFailed`Optional	`boolean`	No default	The avatar could not be loaded; nothing is on screen. Survives a session ending, so label the action "Retry" rather than "Connect".

The session

Prop	Type	Default	Description
`isIdle`Optional	`boolean`	No default	No session; start() is available. True while the avatar is still loading, since start() waits the load out.
`isConnecting`Optional	`boolean`	No default	start() is in flight — token, config, socket, and init ack, end to end.
`isConnected`Optional	`boolean`	No default	The session is live. The turn states below only apply from here.

The turn

Prop	Type	Default	Description
`isListening`Optional	`boolean`	No default	Waiting for the user's voice. Call mode.
`isThinking`Optional	`boolean`	No default	The server is working out the reply, between listening and speaking.
`isSpeaking`Optional	`boolean`	No default	The avatar is talking, with lip-sync running.
`isResponseSlow`Optional	`boolean`	No default	The current turn is taking unusually long. Advisory — the turn is fine, just slow. Auto-clears when speech starts or the turn ends. The built-in thinking indicator renders off this.

Mute and push-to-talk

Prop	Type	Default	Description
`isMicMuted`Optional	`boolean`	No default	The user's mic is muted — outbound audio is dropped, so the server hears silence. Call mode.
`isSpeakerMuted`Optional	`boolean`	No default	The avatar's audio is muted. Playback is silent but lip-sync still animates.
`isPushToTalk`Optional	`boolean`	No default	turnTaking.mode is "push-to-talk". Static for the session.
`isTalking`Optional	`boolean`	No default	The user is holding the talk button, so their audio is reaching the server.

Live data

Prop	Type	Default	Description
`error`Optional	`AvatarError \| null`	No default	The most recent error, or null. Sticky — it stays until the next start(), and never clears on a timer.
`notice`Optional	`SessionNotice \| null`	No default	The last server notice (limit, plan, test key). Not an error — the session is ending gracefully. Drive your upgrade CTA off this.
`health`Optional	`ConnectionHealth`	No default	Live connection health. health.concern is the blame-resolved banner state (null when all is well); health.services is the raw per-service map.
`transcript`Optional	`TranscriptEntry[]`	No default	Conversation lines, oldest first. Empty unless transcript.enabled.
`availableActions`Optional	`string[]`	No default	Gesture names the loaded rig advertises.
`avatarId`Optional	`string \| undefined`	No default	The currently loaded avatar's ID.
`controls`Optional	`{ enabled: boolean; stopSpeaking?: boolean }`	No default	Whether the built-in control bar is shown, and whether it includes the stop-speaking button.

Session control

start()

Open the session. In call mode this requests microphone permission. If the avatar is still loading, start() waits for it rather than failing.

tsx

const { start } = useAvatar();
<button onClick={start}>Begin</button>

Call start() from a real click or tap. A session started outside a user gesture connects but stays mute — browsers block audio that no one asked for.

stop()

End the session. The avatar stays painted, so restarting is cheap and isIdle flips back to true.

interrupt()

Cut the avatar off mid-sentence. It returns to listening.

stopSpeaking()

A manual interrupt, aimed at a "stop talking" button in your own UI. Cuts the avatar off and goes back to listening.

tsx

const { stopSpeaking, isSpeaking } = useAvatar();
{isSpeaking && <button onClick={stopSpeaking}>Stop talking</button>}

Prefer controls.stopSpeaking: true if you are using the built-in bar — it renders the same button and shows it only while the avatar speaks.

Mute

tsx

function MuteButtons() {
  const { isMicMuted, setMicMuted, isSpeakerMuted, setSpeakerMuted } = useAvatar();

  return (
    <>
      <button
        onClick={() => setMicMuted(!isMicMuted)}
        aria-pressed={isMicMuted}
      >
        {isMicMuted ? "Unmute mic" : "Mute mic"}
      </button>

      <button
        onClick={() => setSpeakerMuted(!isSpeakerMuted)}
        aria-pressed={isSpeakerMuted}
      >
        {isSpeakerMuted ? "Unmute avatar" : "Mute avatar"}
      </button>
    </>
  );
}

Muting the mic drops outbound audio, so the server genuinely hears silence — it is not a UI-only flag. Muting the speaker silences playback while lip-sync keeps animating, so the avatar does not appear frozen.

Push to talk

Set turnTaking: { mode: "push-to-talk" } on the config, then wire the press and release. Pressing cuts the avatar off if it was speaking, which is what makes barge-in work.

tsx

function TalkButton() {
  const { isPushToTalk, isTalking, startTalking, stopTalking } = useAvatar();

  if (!isPushToTalk) return null;

  return (
    <button
      onPointerDown={startTalking}
      onPointerUp={stopTalking}
      onPointerCancel={stopTalking}
      onPointerLeave={stopTalking}
      aria-pressed={isTalking}
    >
      {isTalking ? "Listening…" : "Hold to talk"}
    </button>
  );
}

Handle onPointerCancel and onPointerLeave, not just onPointerUp. A pointer that leaves the button or gets cancelled by the OS otherwise leaves the turn open forever.

By default the SDK also accepts the spacebar. Set turnTaking.spacebar: false if your page binds it too.

Mode-specific methods

These are undefined outside their mode, so always use optional chaining.

speakText(sentence)

tts mode. The avatar speaks the string with full lip-sync.

tsx

const { speakText } = useAvatar();
speakText?.("Welcome! How can I help you today?");

Text over 5000 characters is rejected with a non-fatal INVALID_INPUT.

speakAudio(audio)

audio mode. Feed audio from any source and the avatar lip-syncs to it.

tsx

const { speakAudio } = useAvatar();

const buffer = await file.arrayBuffer();
speakAudio?.(buffer);

// Accepts: ArrayBuffer | Float32Array | Blob

See Limits for the required PCM format.

play(job)

player mode. Play a pre-rendered AvatarSpeech job locally.

tsx

const { play } = useAvatar();

play?.({
  audioChunks: ["base64-audio..."],
  alignments: [{
    characters: ["H", "e", "l", "l", "o"],
    character_start_times_seconds: [0, 0.08, 0.16, 0.24, 0.32],
    character_end_times_seconds:   [0.08, 0.16, 0.24, 0.32, 0.4],
  }],
});

addLiveContext(args)

call mode, requires a live session. Tell the avatar what the user is doing right now.

tsx

const { addLiveContext } = useAvatar();

// Append to the existing context
addLiveContext?.({ context: "User is viewing the Premium Plan pricing page." });

// Replace it entirely
addLiveContext?.({ context: "User just navigated to checkout.", update: true });

tsx

type LiveContextArgs = {
  context: string;   // non-empty, max 2000 characters
  update?: boolean;  // true replaces the previous context
};

With no live session the context is dropped and a non-fatal NOT_CONNECTED error is emitted. Empty or over-length context emits a non-fatal INVALID_INPUT. Neither ends the session.

Gestures

Rigs can advertise named gestures. Read what is available, then fire one.

tsx

function GestureButtons() {
  const { availableActions, triggerAction } = useAvatar();

  return availableActions?.map((name) => (
    <button key={name} onClick={() => triggerAction?.(name)}>
      {name}
    </button>
  ));
}

availableActions comes from the loaded rig, so it is empty until the avatar is ready and differs between characters. Never hard-code a gesture name — render from the list.

Server notices

A notice is not an error. It is the server ending the session deliberately — a plan limit, a demo cap, a test key expiring — and the avatar speaks the message before the session closes.

tsx

function NoticeBanner() {
  const { notice } = useAvatar();
  if (!notice) return null;

  return (
    <div role="status">
      <p>{notice.message}</p>
      {notice.action === "upgrade" && <a href="/pricing">See plans</a>}
      {notice.action === "add_key" && <a href="/profile?tab=developer">Add a key</a>}
    </div>
  );
}

Unlike error.message, notice.message is written for your users — it is the same text the avatar just said out loud.

Connection health

health.concern is the single thing to render from; the per-service map is for a debug panel.

tsx

function HealthHint() {
  const { health } = useAvatar();
  const concern = health?.concern;
  if (!concern) return null;

  const COPY = {
    network: "Your connection looks unstable.",
    device:  "Your device is struggling to keep up.",
    service: "Reconnecting…",
  };

  return <p role="status">{COPY[concern.scope]}</p>;
}

You get this for free — the built-in status banner renders exactly this and is on by default. Write your own only if you need it somewhere else on the page, and turn the built-in one off with statusBanner: { enabled: false }.

Event subscription with on/off

For subscribing outside the component lifecycle, or grouping events in one useEffect. For everything else prefer useAvatarEvent, which cleans up for you.

tsx

const { on, off } = useAvatar();

useEffect(() => {
  const onStart = () => console.log("Session started");
  const onError = (err) => console.error(err.code, err.message);

  on("start", onStart);
  on("error", onError);

  return () => {
    off("start", onStart);
    off("error", onError);
  };
}, [on, off]);

AvatarWidget

useAvatarEvent