@snap/camera-kit 0.11.0 → 0.12.0

package/README.md

@@ -2,31 +2,60 @@
 
 The Camera Kit Web SDK allows web developers to build Snap's core AR Lens technology into their applications.
 
-## Getting started
+## Minimum browser requirements
 
-First, familiarize yourself with the [Camera Kit docs](https://docs.snap.com/camera-kit/home). This covers access to the Camera Kit Portal, where to find your API Token, how to manage lenses and lens groups, etc.
+- **Chrome 73+**
+- **Safari 15+**
+  - MacOS 12+, iOS 15+, iPadOS 15+
+  - SIMD: Unsupported
+  - Web Worker Mode\*: Unsupported
+- **Edge 79+**
+  - Edge is still currently under evaluation. However, since New Edge is Chromium based, the expectations are similar to that of Chrome.
+- **Firefox** - Under Evaluation
+  - Web Worker Mode\*: Firefox 105+
 
-Most of the content found here can also be found on the [Camera Kit docs](https://docs.snap.com/camera-kit/quick-start/integrate-sdk/integrate-sdk-web/web-configuration) site.
+\*Web Worker Mode requires `OffscreenCanvas` support.
 
-### Installing the SDK
+## Prerequisites
+
+### Snap Developer set up
+You'll need a Snap Developer account, and you'll need to apply for access to Camera Kit Web SDK. You can find more info on that [here](camera-kit/quick-start/setting-up-accounts).
+
+You may also want to familiarize yourself with how to access and manage AR content (i.e. Lenses). You read about that [here](/camera-kit/quick-start/build-manage-ar-content/camera-kit-portal).
+
+### Development environment
+This guide assumes you've already set up an [NPM](https://www.npmjs.com/) package, you're using [TypeScript](https://www.typescriptlang.org/), and have some way to build and host your project during development (e.g. using [Webpack](https://webpack.js.org/)).
+
+#### Using the SDK in a JavaScript project
+
+The SDK is authored in TypeScript, and distributes type definitions. All the examples here are presented in TypeScript. We encourage the use of TypeScript in projects that consume the SDK – but it's also fully compatible with JavaScript projects as well.
+
+### Content Security Policy
+
+If your project already has a [Content Security Policy](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy) in place, you'll likely need to make some changes in order for Camera Kit Web SDK to work.
+
+When it bootstraps, Camera Kit Web SDK downloads an executable WebAssembly file containing the Lens rendering engine. This file is served from an optimized CDN managed by Snap. You'll need to make sure your Content Security Policy allows this file to be executed.
+
+- `connect-src` must include `https://*.snapar.com`, otherwise Camera Kit Web will fail to initialize.
+- `script-src` must include `https://cf-st.sc-cdn.net/ blob: 'wasm-unsafe-eval'`.
+
+_Note: Some older browser versions may not support the `'wasm-unsafe-eval'` source value, and it may be necessary to use `'unsafe-eval'` to allow Camera Kit's downloaded WebAssembly to run._
+
+## Installing the SDK
 
 ```
 npm install @snap/camera-kit
 ```
 
-### Using the SDK in a Javascript project
-
-The SDK is authored in TypeScript, and distributes type definitions. All the examples here are presented in TypeScript. We encourage the use of TypeScript in projects that consume the SDK – but it's also fully compatible with Javascript projects as well.
+You can find the Camera Kit Web SDK package on npmjs.com [here](https://www.npmjs.com/package/@snap/camera-kit).
 
-### Importing the SDK
+### Importing Camera Kit
 
-Currently, the SDK distributes [Javascript modules](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules). We may support other module formats in the future (e.g. CommonJS), but for now you'll need to use `import` syntax to use the Camera Kit Web SDK.
+Currently, the SDK distributes [JavaScript modules](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules). We may support other module formats in the future (e.g. CommonJS), but for now you'll need to use `import` syntax to use the Camera Kit Web SDK.
 
-If you're working in a project that uses a bundler (e.g. Webpack or Rollup), or builds using the TypeScript compiler, this shouldn't be an issue. If you're not using a bundler, many browsers now support `import` syntax natively – if you don't need support for older browsers (or IE), no bundler is necessary.
+## Bootstrapping the SDK
 
-### Bootstrapping the SDK
-
-With the SDK installed and capable of being imported, the first thing to do is bootstrap the SDK. This tells the SDK to download the LensCore WebAssembly runtime, which will be used to render Lenses. It also allows you to configure the SDK according to your needs.
+With the SDK installed and imported, the first thing to do is bootstrap the SDK. When bootstrapping, the SDK will download the WebAssembly runtime which is used to render Lenses. This is also where you'll configure the SDK according to your needs.
 
 To call {@link bootstrapCameraKit}, you'll need to provide a `apiToken`. Once you've completed the [Getting set up in our portals](https://docs.snap.com/camera-kit/guides/quick-start/integrate-sdk/setting-up-accounts) section of the Getting Started guide, you'll be able to find this in the [Snap Kit Portal](https://devportal.snap.com/manage).
 
@@ -34,14 +63,14 @@ To call {@link bootstrapCameraKit}, you'll need to provide a `apiToken`. Once yo
 import { boostrapCameraKit } from "@snap/camera-kit";
 
 (async function main() {
-    const apiToken = "API Token value copied from the Camera Kit Portal";
+    const apiToken = "Your API Token value copied from the Camera Kit Portal";
     const cameraKit = await bootstrapCameraKit({ apiToken });
 })();
 ```
 
-### Creating a CameraKitSession
+## Creating a `CameraKitSession`
 
-In order to render Lenses, you must first create a {@link CameraKitSession}. Each session represents a rendering pipeline - it connects an input media source (e.g. a webcam) to the LensCore AR engine, applies a Lens, and renders the output to a `<canvas>` element.
+In order to render Lenses, you must first create a {@link CameraKitSession}. Each session represents a rendering pipeline - it connects an input media source (e.g. a webcam) to Camera Kit's AR engine, applies a Lens, and renders the output to a `<canvas>` element.
 
 There are two ways to create a session. If you already have a `<canvas>` element on your page that you'd like for the {@link CameraKitSession} to render into, you can pass it to Camera Kit when creating the session. Otherwise, the session will create a new `<canvas>` element which you can add to the DOM.
 
@@ -56,63 +85,71 @@ const session = await cameraKit.createSession();
 canvasContainer.appendChild(session.output.live);
 ```
 
-_There are actually two different output canvases: `live` and `capture`. These correspond to the two different RenderTargets to which a Lens may render. The `live` output renders a view suitable for presentation to the current user. Some Lenses may also render to the `capture` output, which renders content suitable for viewing by other users._
+There are actually two different output canvases:
+
+- `live`: This canvas renders content intended for the Lens user. Depending on the Lens being used, this canvas may include UI elements, prompts, or other content that is only meant to be seen by the user of the Lens.
+- `capture`: This canvas renders content intended for presenting to other users.
+
+These two output canvases correspond to the two different [RenderTargets](https://docs.snap.com/lens-studio/references/guides/lens-features/scene-set-up/camera#render-target) a Lens may use to render its content. Not all Lenses will render different content to `live` vs. `capture`, so it's important to understand how the Lenses you'll be using use these two different outputs.
+
+> No rendering will happen yet, and the output canvas associated with this new `CameraKitSession` will be blank. Frames are not processed until you start [playback](#playback) by calling the `CameraKitSession`'s `play()` method. This will be discussed below.
+
+## Creating a `CameraKitSource`
+
+In order for Camera Kit Web SDK to render anything, it must have a source of imagery to be processed by the AR engine. The Lens content will be rendered on top of this source media.
+
+The most common source of input media is the user's webcam. Camera Kit Web SDK provides a helper method to create a `CameraKitSource` object from a [`MediaStream`](https://developer.mozilla.org/en-US/docs/Web/API/MediaStream). You can use [`getUserMedia`](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia) to obtain a `MediaStream` with video from the user's webcam.
 
-### Obtaining a media source
+> Note that calling `getUserMedia` will prompt the user to grant the webpage access to their camera.
 
-The most common source will be the user's webcam. Camera Kit Web SDK provides helper methods to create a {@link CameraKitSource} object from
-a [`MediaStream`](https://developer.mozilla.org/en-US/docs/Web/API/MediaStream/MediaStream), or directly from a user's webcam via [`getUserMedia`](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia).
+Once we have a `CameraKitSource`, we can tell the `CameraKitSession` to use this source for rendering.
 
 ```ts
-import { createMediaStreamSource } from "@snap/camera-kit";
+import { createMediaStreamSource, Transform2D } from "@snap/camera-kit"
 
-// ...
 const stream = await navigator.mediaDevices.getUserMedia({ video: true });
-const source = createMediaStreamSource(stream, {
-    transform: Transform2D.MirrorX,
-    cameraType: "front",
-});
-session.setSource(source);
+const source = createMediaStreamSource(stream, { transform: Transform2D.MirrorX, cameraType: 'front' });
+await session.setSource(source);
 ```
 
-_Note that calling `getUserMedia` will prompt the user to grant the webpage access to the webcam._
+In this example, we also mirror the source stream (which feels more natural in most cases), and we indicate this source comes from a front-facing camera. To read more about these options, see [below](#customizing-camerakitsource).
+
+Camera Kit Web SDK has helper methods to create a `CameraKitSource` from:
+
+- A `MediaStream` object, which could come from the user's camera, a WebRTC connection, a `<canvas>` via the [`captureStream()` method](https://developer.mozilla.org/en-US/docs/Web/API/HTMLCanvasElement/captureStream), or elsewhere.
+- A `<video>` element (i.e. `HTMLVideoElement`)
+- An `<img>`element (i.e. `HTMLImageElement`)
 
-### Applying a Lens
+## Loading, applying, and removing Lenses
 
-In order to apply a lens to our session, we first have to fetch metadata for the lens we wish to apply. We can configure a group of lenses and get all the lenses in that group – this could be useful if we want to allow the user to select a lens from the group. Or if we know the specific lens we want to apply, we can provide its unique ID (along with the ID of a group that contains that lens) and get back just the single lens.
+In Camera Kit Web SDK -- just like in the Snapchat app itself -- each individual AR experience is called a Lens. Lenses are created using [Lens Studio](https://ar.snap.com/en-US/lens-studio). You can find more information about acquiring Lenses [here](https://docs.snap.com/camera-kit/quick-start/build-manage-ar-content/ar-overview).
 
-In the [Camera Kit Portal](https://camera-kit.snapchat.com/), you can find available lenses and add them to lens groups. Once you've chosen a lens or lens group to load, you can use the Lens repository to fetch the lens metadata. Then you can apply the lens to the session.
+You can find Lens and Lens Groups in the [Camera Kit Portal](https://camera-kit.snapchat.com/). This is where you can manage your Lenses and Lens Groups and where you can find Lens IDs and Lens Group IDs. You can read more about managing Lenses in the Camera Kit Portal [here](https://docs.snap.com/camera-kit/quick-start/build-manage-ar-content/camera-kit-portal).
 
-_The [Camera Kit docs](https://docs.snap.com/camera-kit/guides/quick-start/build-manage-ar-content/camera-kit-portal) include info on how to set up Lenses and Lens Groups._
+Before applying a Lens to our session, we have to load metadata about that Lens. Lenses can be loaded either one at a time, or entire Lens Groups can be loaded at once. You'll need a Lens ID and its Lens Group ID to complete this step.
 
 ```ts
-// A single lens
-const lens = await cameraKit.lensRepository.loadLens("A lens ID found in the Camera Kit Portal", "A lens group ID");
-session.applyLens(lens);
+// Loading a single lens
+const lens = await cameraKit.lensRepository.loadLens("<Lens ID>", "<Lens Group ID>");
+await session.applyLens(lens);
 
-// One or more lens groups – lenses from all groups are returned as a single array of lenses.
+// Loading one or more Lens Groups – Lenses from all groups are returned as a single array of lenses.
 const { lenses } = await cameraKit.lensRepository.loadLensGroups([
-    "A lens group ID",
-    "These can be found in the Camera Kit Portal",
+    "<Lens Group ID 1>",
+    "<Lens Group ID 2>",
 ]);
-session.applyLens(lenses[0]);
+await session.applyLens(lenses[0]);
 ```
 
-### Setting the render size
-
-By default, Camera Kit will render its output at the same resolution as the video input you provided. But you can also tell Camera Kit to render at a different resolution.
-
-Keep in mind that this controls LensCore's render resolution, and not (necessarily) the size at which the output canvas is displayed to the user. The output canvas may be sized using HTML and CSS, and may apply its own scaling to the rendered output.
-
-Most of the time you'll not need to set the render size – but it could be useful if your video source is, say, very high resolution. In that case, you may observe better performance by telling Camera Kit to render at a lower resolution.
+### Removing the Lens
 
+You can also remove the currently-applied Lens:
 ```ts
-session.setRenderSize(width, height);
+await session.removeLens();
 ```
+After removing the Lens, when the session renders it will simply render the input media directly to the output canvas.
 
-_Note that when calling [`getUserMedia`](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia), best performance can be achieved by requesting the resolution you want to display. This can be done using [constraints](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia#parameters)_
-
-### Session playback
+## Playback
 
 The {@link CameraKitSession} will only process input video frames and render them to the output when you tell it to do so – this way, you can control when Camera Kit is using a client's resources (e.g. CPU and GPU compute cycles). You can tell the session to `play` or `pause`.
 
@@ -123,9 +160,45 @@ session.play();
 session.pause();
 ```
 
-_You can also tell the session which output – `live` or `capture` - to play / pause. By default, `play` will start rendering the `live` output and `pause` will pause rendering all outputs._
+### Playing and pausing the different RenderTargets
+
+By default, `play()` will only begin rendering the `live` output canvas. You can specify which canvas to render by passing an argument:
+
+```ts
+session.play('live');
+session.play('capture');
+```
+
+Calling `pause()` with no arguments will pause both outputs. But just like with `play()`, you can pass an argument to select which canvas to pause.
+
+```ts
+session.pause('live');
+session.pause('capture');
+```
+
+> - _`live`: This canvas renders content intended for the Lens user. Depending on the Lens being used, this canvas may include UI elements, prompts, or other content that is only meant to be seen by the user of the Lens._
+> - _`capture`: This canvas renders content intended for presenting to other users._
+
+## Error Handling
 
-### Putting it all together
+Camera Kit Web SDK methods may throw errors, or return rejected Promises -- these are documented in the [API docs](https://kit.snapchat.com/reference/CameraKit/web/0.12.0/index.html). It is good practice to handle such cases, to provide a good experience to your users.
+
+Errors may also occur during Lens rendering. For example, Lenses contain their own scripting, which could throw an error. A rendering error could also occur if a Lens attempts to use a feature that is not supported by Camera Kit Web SDK.
+
+When a `LensExecutionError` such as these occurs, the Lens is **automatically removed** from the `CameraKitSession`. An error event is emitted so that your application can respond appropriately. You can listen to these error events like so:
+
+```ts
+session.events.addEventListener('error', (event) => {
+  console.error(event.detail.error);
+
+  if (event.detail.error.name === 'LensExecutionError') {
+    // The currently-applied Lens encountered a problem that is most likely unrecoverable and the Lens has been removed.
+    // Your application may want to prevent this Lens from being applied again.
+  }
+});
+```
+
+## Putting it all together
 
 Using the examples above, here's a complete example of the minimal Camera Kit Web SDK integration:
 
@@ -133,37 +206,139 @@ Using the examples above, here's a complete example of the minimal Camera Kit We
 import { boostrapCameraKit, createMediaStreamSource } from "@snap/camera-kit";
 
 (async function main() {
-    const apiToken = "API Token value copied from the SnapKit developer portal";
+    const apiToken = "Your API Token value copied from the SnapKit developer portal";
     const cameraKit = await bootstrapCameraKit({ apiToken });
 
     const canvas = document.getElementById("my-canvas");
-    const session = await cameraKit.createSession(canvas);
+    const session = await cameraKit.createSession({ liveRenderTarget: canvas });
+    session.events.addEventListener('error', (event) => {
+      if (event.detail.error.name === 'LensExecutionError') {
+        console.log('The current Lens encountered an error and was removed.', event.detail.error);
+      }
+    });
 
     const stream = await navigator.mediaDevices.getUserMedia({ video: true });
-    const source = createMediaStreamSource(stream, {
-        transform: Transform2D.MirrorX,
-        cameraType: "front",
-    });
-    session.setSource(source);
+    const source = createMediaStreamSource(stream, { transform: Transform2D.MirrorX, cameraType: 'front' });
+    await session.setSource(source);
 
-    const lens = await cameraKit.lensRepository.loadLens("A lens ID found in the Camera Kit Portal", "A lens group ID");
+    const lens = await cameraKit.lensRepository.loadLens("<Lens ID>", "<Lens Group ID>");
     await session.applyLens(lens);
 
     await session.play();
     console.log("Lens rendering has started!");
 })();
+
+main();
 ```
 
-### Content Security Policy
+## Advanced use cases
+
+### Logging
 
-Camera Kit Web SDK downloads an executable WebAssembly file containing the Lens rendering engine. This file is served from an optimized CDN. If you have a [Content Security Policy](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy) in place on your page, you'll likely need to make some changes in order for Camera Kit to work.
+By default, Camera Kit Web SDK does very minimal logging. Specifying `'console'` as logger will cause more log statements to be printed to the browser's console, which may be useful during development.
 
-#### `connect-src`
+```ts
+const cameraKit = await bootstrapCameraKit({
+  apiToken: '<apiToken>',
+  logger: 'console',
+});
+```
 
-Must include `https://*.snapchat.com`, otherwise Camera Kit Web will fail to initialize.
+### Keyboard support for Lenses
 
-#### `script-src`
+Some Lenses allow for keyboard input. The SDK provides a `<textarea>` (`HTMLTextAreaElement`) element that can be added to your page, and will send its text to the active Lens whenever the user presses the `Enter` key.
 
-Must include `https://cf-st.sc-cdn.net/ blob: 'wasm-unsafe-eval'`.
+```ts
+const textAreaContainer = document.getElementById('text-area-container');
+const textArea = session.keyboard.getElement();
+textAreaContainer.appendChild(textArea);
+```
 
-_Note: Some older browser versions may not support the `'wasm-unsafe-eval'` source value, and it may be necessary to use `'unsafe-eval'` to allow Camera Kit's downloaded WebAssembly to run._
+Alternatively, an event is emitted when a Lens is expecting text input. You can then implement your own UI and logic for obtaining text input from your users, and then send it back to the Lens. For example, something like:
+
+```ts
+const input = document.getElementById('my-text-area');
+session.keyboard.addEventListener('active', () => {
+  input.classList.remove('hidden');
+});
+
+input.addEventListener('keyup', () => {
+  session.keyboard.sendInputToLens(input.value);
+})
+```
+
+See an example of this [here](https://camera-kit.snapchat.com/websdk/sample/keyboard).
+
+### Customizing `CameraKitSource`
+
+#### Setting the render size
+
+By default, Camera Kit will render its output at the same resolution as the video input you provided. But you can also tell Camera Kit to render at a different resolution.
+
+Keep in mind that this controls Camera Kit's render resolution, and not (necessarily) the size at which the output canvas is displayed to the user. The output canvas may be sized using HTML and CSS, and may apply its own scaling to the rendered output.
+
+Most of the time you'll not need to set the render size – but it could be useful if your video source is, say, very high resolution. In that case, you may observe better performance by telling Camera Kit to render at a lower resolution.
+
+```ts
+await session.setSource(source)
+// This must be done *after* calling `setSource()`
+await source.setRenderSize(width, height);
+```
+
+> When calling [`getUserMedia`](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia), best performance can be achieved by requesting the resolution at which you want to render. This can be done using [constraints](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia#parameters). Then you won't have to use `setRenderSize` at all.
+
+#### Camera type
+
+When setting up a `CameraKitSource`, you can specify whether or not it is a front facing camera or a back facing camera. By default, the media source will be treated as a front facing camera. If it is a back facing camera be sure to specify it as such when you create your source; this ensures that Lenses created for back facing camera's (e.g. World AR) render properly.
+
+```ts
+const stream = await navigator.mediaDevices.getUserMedia(constraints);
+const source = createMediaStreamSource(stream, { cameraType: 'back' });
+await session.setSource(source);
+```
+
+#### FPS limit
+
+In some cases, it may be desirable to set a limit on the FPS at which Camera Kit renders. By default, Camera Kit will attempt to render frames at the same rate as the source media. But, for example, if your input media source has a very high framerate you may want to limit Camera Kit's rendering framerate to achieve better performance.
+
+This can be done at the `CameraKitSession` level:
+
+```ts
+await session.setFPSLimit(60);
+```
+
+Or at the `CameraKitSource` level:
+
+```ts
+const stream = await navigator.mediaDevices.getUserMedia(constraints);
+const source = createMediaStreamSource(stream, { fpsLimit: 60 });
+```
+
+These options can also be used with any of the [create source helpers](#creating-a-camerakitsource).
+
+#### 2D transforms
+
+Any `CameraKitSource` can be transformed using a matrix, to rotate, scale, or mirror the source media. This is most often used to mirror a front-facing camera stream so that it appears more natural to the user.
+
+```ts
+import { Transform2D } from "@snap/camera-kit";
+
+await session.setSource(source)
+// This must be done *after* calling `setSource()`
+source.setTransform(Transform2D.MirrorX);
+```
+
+### Metrics reporting
+
+Camera Kit Web SDK reports certain important events which may be of interest to the application.
+
+> This is an unstable portion of the Camera Kit Web SDK's public API -- the specific events and their properties may change between versions of the SDK, without warning. We currently offer these events as a convenient way to gather more information from the SDK, but if you need to report accurate metrics (e.g. number of lens views), that is currently something you will have to implement in your application.
+
+Currently, the only event that may be of interest is the `lensView` event. It is emitted whenever a lens is _removed_ from the `CameraKitSession`, indicating a complete lens view. It contains information about the lens' performance (e.g. fps, frame processing times, etc.) and how long the lens was applied.
+
+You may listen to these events like so:
+```ts
+cameraKit.metrics.addEventListener('lensView', (event) => {
+  console.debug(event.detail);
+});
+```