<!-- AvatarFactory Docs · https://avatarfactory.in/docs/sdk/react/perception · Full map: https://avatarfactory.in/llms.txt -->

# Camera perception

Let the avatar see. Opt-in, opened just-in-time, held in memory only — and what the user sees while it is on.

Perception lets the avatar answer questions about what the camera can see
("what am I holding?", "does this look right?"). It is off unless you turn it
on, and turning it on does **not** open the camera.

```tsx
const config = {
  getSessionToken: async () => fetchSessionToken(),
  mode: "call",
  perception: { camera: true }, // permission to ask — not an instruction to open
  avatar: { avatarId: "default" },
};
```

## Just-in-time by design

Even with `perception.camera` enabled, the SDK opens the camera only when the
server asks for a look, mid-conversation. It is never opened on connect.

That ordering is the point. A camera that switches on at the start of every call
and stays on is a camera the user stops thinking about. One that opens when the
avatar needs to see something is one the user can connect to a reason.

> `perception: { camera: true }` is a **grant**, not a switch. The browser still
>   shows its own permission prompt the first time, and the user can refuse. Plan
>   for a session where the avatar never gets to see anything.

## Three independent states

Do not collapse these — "permission granted" does not mean "a picture is
arriving", and neither means "the session is over".

| State | What it tracks |
| --- | --- |
| Session open | The camera session exists and the self-view is mounted. Ends only when the call ends. |
| Video on / off | The user's own toggle. Off releases the camera hardware and drops the frame buffer, but keeps the session and the self-view up. |
| Permission | Granted once and remembered. It survives the video toggle and the end of the call, so re-opening never re-prompts. |

If the user switches video off, the SDK will **not** silently re-open it — only
an explicit toggle back on will. A camera the user turned off staying off is not
negotiable.

## What is captured

| Property | Value | Why |
| --- | --- | --- |
| Capture rate | 3 fps | Enough to catch a gesture; low enough not to compete with the call for CPU. |
| Rolling buffer | 12 s | So a retroactive question (\"what did I just do?\") is answerable. |
| Frame size | 512 px longest edge | Downscaled before encoding. The model does not need more, and the upload does not want it. |
| Encoding | JPEG, quality 0.7 | Small enough to send inside a live call's latency budget. |
| Frames sent per look | 3, newest last | At 3 fps that is the last second. Older frames are near-duplicates and only add model latency. |

> **Frames live in memory and are never persisted.** The buffer is a fixed
>   12-second window that discards anything older on every capture, and it is
>   dropped entirely when video is switched off or the session ends. Nothing is
>   written to disk, and nothing survives the call.

A freshly-opened camera has an **empty** buffer — `getUserMedia` resolves before
the hardware yields a decodable frame, typically by more than a second. The SDK
waits for the first real frame rather than sending nothing, so the first look
after a cold grant is slower than later ones.

## The self-view

While a camera session is open, the SDK renders a small live preview of the
user's own camera. It is the mechanism by which the user knows the camera is on.

| Prop | Type | Default | Description |
| --- | --- | --- | --- |
| `cameraPreviewCorner` | `CameraPreviewCorner` | `"bottom-right"` | Which corner the self-view sits in. Ignored in picture-in-picture. |
| `hideCameraPreview` | `boolean` | `false` | Hide the self-view. Discouraged — capture continues either way, so all this removes is the user's ability to see that it is happening. Ignored in picture-in-picture. |
| `usePictureInPicture` | `boolean` | `false` | Swap the layout: camera on the stage, avatar in the corner, once a camera session opens. Takes precedence over the two props above. |
| `pictureInPictureCorner` | `CameraPreviewCorner` | `"bottom-right"` | The avatar's corner in picture-in-picture. Independent of cameraPreviewCorner, since the two modes place different things. |

```tsx
<Avatar
  usePictureInPicture
  pictureInPictureCorner="top-right"
/>
```

> Think hard before setting `hideCameraPreview`. The self-view is a privacy
>   signal, not decoration — hiding it means the camera is running with nothing on
>   screen to say so.

## When the camera goes away

The SDK distinguishes the user's intent from everything else, and reports the
difference to the server so the avatar does not ask to see something it cannot:

| Event | What happens |
| --- | --- |
| The user toggles video off | Camera released, buffer dropped, session and self-view stay. A later look can legitimately re-open it only after the user toggles back on. |
| The camera is unplugged, or another app takes it | Treated as an involuntary loss, not a user decision. The feed is reported as off; a later request may re-open it. |
| Permission is revoked mid-call | Both axes collapse — access denied, feed off. The next open would prompt again. |
| No camera hardware, or an insecure context | Reported as unavailable rather than denied. Retrying will not help. |

## Front and back cameras

On devices with more than one camera, the avatar can switch between them. The
SDK detects whether a flip is actually possible before advertising it, and keeps
the current camera if the target one fails to open — a failed flip never leaves
the user with no picture.

## React Native

Camera perception is supported on React Native, with the same opt-in config and
the same just-in-time opening. The self-view corner and `hideCameraPreview` work
the same way; picture-in-picture is web-only.

Camera permission on mobile is a native permission — see
[Permissions](/docs/sdk/react-native/permissions) for the Expo config-plugin
setup.

## Privacy checklist

Before shipping perception, make sure your own product does these — the SDK
cannot do them for you:

- **Say so before the call.** The browser prompt asks for the camera; it does
  not explain why your avatar wants it.
- **Leave the self-view on.** It is the user's only continuous signal.
- **Give the user a video toggle** in your own UI if you are not using the
  built-in controls.
- **Cover it in your privacy policy** — frames are transient, but they are still
  images of a person, sent to a model vendor.

> If you only need the avatar to know *about* something rather than *see* it,
>   [`addLiveContext`](/docs/sdk/react/use-avatar#addlivecontextargs) is cheaper,
>   faster, and asks nothing of the user.