Camera perception

Perception lets the avatar answer questions about what the camera can see ("what am I holding?", "does this look right?"). It is off unless you turn it on, and turning it on does not open the camera.

To let the avatar see the user's screen rather than their camera, see Screen share.

const config = {
  getSessionToken: async () => fetchSessionToken(),
  mode: "call",
  perception: { camera: true }, // permission to ask — not an instruction to open
  avatar: { avatarId: "default" },
};

Just-in-time by design

Even with perception.camera enabled, the SDK opens the camera only when the server asks for a look, mid-conversation. It is never opened on connect.

That ordering is the point. A camera that switches on at the start of every call and stays on is a camera the user stops thinking about. One that opens when the avatar needs to see something is one the user can connect to a reason.

Three independent states

Do not collapse these — "permission granted" does not mean "a picture is arriving", and neither means "the session is over".

If the user switches video off, the SDK will not silently re-open it — only an explicit toggle back on will. A camera the user turned off staying off is not negotiable.

What is captured

A freshly-opened camera has an empty buffer — getUserMedia resolves before the hardware yields a decodable frame, typically by more than a second. The SDK waits for the first real frame rather than sending nothing, so the first look after a cold grant is slower than later ones.

The self-view

While a camera session is open, the SDK renders a small live preview of the user's own camera. It is the mechanism by which the user knows the camera is on.

<Avatar
  usePictureInPicture
  pictureInPictureCorner="top-right"
/>

When the camera goes away

The SDK distinguishes the user's intent from everything else, and reports the difference to the server so the avatar does not ask to see something it cannot:

Mood reading

Mood reading comes with the camera: there is nothing to configure. Whenever the camera is on, the self-view shows a Detect mood switch. It is off until the user turns it on. While it is on, the SDK reads their expression on their own device and the avatar adjusts its tone: warmer when they look happy, calmer and more patient when they look frustrated. It never opens the camera itself, and it never names the user's mood unless they ask.

A read happens only when all of these are true:

A custom toggle

The built-in switch lives in the self-view. To put one in your own UI, read and set the same state from useAvatar — the two stay in sync.

function MoodToggle() {
  const { isMoodDetectionAvailable, isMoodDetectionOn, setMoodDetection } =
    useAvatar();
  if (!isMoodDetectionAvailable) return null;
  return (
    <label>
      <input
        type="checkbox"
        checked={isMoodDetectionOn}
        onChange={(event) => setMoodDetection(event.target.checked)}
      />
      Detect mood
    </label>
  );
}

isMoodDetectionAvailable is true once a call is connected with camera perception on. Reads still only happen while the camera is on and the switch is on.

What it needs from the page

The face model is downloaded once per device (about 16 MB, cached afterwards) and runs in a Web Worker, off your page's main thread. A strict Content Security Policy must allow WebAssembly and blob: workers:

script-src 'self' 'wasm-unsafe-eval' blob:;
worker-src blob:;
connect-src https://api.avatarfactory.in wss://api.avatarfactory.in blob:;

If the model cannot load, mood reading switches itself off for that session; the call and the camera are unaffected. Mood reading is web-only — on React Native, isMoodDetectionAvailable is always false.

Front and back cameras

On devices with more than one camera, the avatar can switch between them. The SDK detects whether a flip is actually possible before advertising it, and keeps the current camera if the target one fails to open — a failed flip never leaves the user with no picture.

React Native

Camera perception is supported on React Native, with the same opt-in config and the same just-in-time opening. The self-view corner and hideCameraPreview work the same way; picture-in-picture is web-only.

Camera permission on mobile is a native permission — see Permissions for the Expo config-plugin setup.

Privacy checklist

Before shipping perception, make sure your own product does these — the SDK cannot do them for you:

• Say so before the call. The browser prompt asks for the camera; it does not explain why your avatar wants it.
• Leave the self-view on. It is the user's only continuous signal.
• Give the user a video toggle in your own UI if you are not using the built-in controls.
• Cover it in your privacy policy — frames are transient, but they are still images of a person, sent to a model vendor.
• Tell users about mood reading if you enable the camera. Reading emotions is regulated in some places (in the EU it is restricted at work and in education), so check it fits where and how you deploy.