@reactor-team/js-sdk 3.3.0 → 3.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +104 -9
- package/dist/index.cjs +1013 -45
- package/dist/index.cjs.map +1 -1
- package/dist/index.d.cts +257 -47
- package/dist/index.d.ts +257 -47
- package/dist/index.js +1013 -45
- package/dist/index.js.map +1 -1
- package/dist/wasm/README.md +36 -9
- package/dist/wasm/reactor_wasm.d.ts +143 -10
- package/dist/wasm/reactor_wasm.js +216 -14
- package/dist/wasm/reactor_wasm_bg.wasm.d.ts +11 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -167,7 +167,7 @@ const reactor = new Reactor({ modelName: "my-stepped-model", generation: "steppe
|
|
|
167
167
|
await reactor.connect();
|
|
168
168
|
|
|
169
169
|
// Input frames never travel over RTP in a stepped session: claim the input
|
|
170
|
-
// track
|
|
170
|
+
// track and push frames through it; they reach the runtime ahead of the next step.
|
|
171
171
|
const camera = await reactor.publishTrack("camera");
|
|
172
172
|
await camera.pushFrame(canvasCtx.getImageData(0, 0, 640, 360));
|
|
173
173
|
|
|
@@ -188,8 +188,8 @@ try {
|
|
|
188
188
|
await reactor.step();
|
|
189
189
|
```
|
|
190
190
|
|
|
191
|
-
- `step({ state, commands })` sends `state`, `commands` and
|
|
192
|
-
|
|
191
|
+
- `step({ state, commands })` sends `state`, `commands` and any frames still buffered in one
|
|
192
|
+
request. The runtime applies them in order — each `state` key as the
|
|
193
193
|
model's `set_<key>` command, then `commands`, then the frames — and then generates. The
|
|
194
194
|
runtime checks the whole request before it applies any of it, so a refused step changes
|
|
195
195
|
nothing; past that check they are separate calls, and a handler that fails does not undo
|
|
@@ -197,7 +197,7 @@ await reactor.step();
|
|
|
197
197
|
`sendCommand()` outside a step still works and changes the state the next step uses. A file
|
|
198
198
|
input goes in the way it does for a command: `uploadFile()` it, then put the `FileRef` in a
|
|
199
199
|
command's `data` or a state value.
|
|
200
|
-
- It resolves with `{ step, tracks: [{ track, kind, rate, frames }], elapsed, files, error? }`,
|
|
200
|
+
- It resolves with `{ step, tracks: [{ track, kind, rate, frames }], elapsed, files, error?, data? }`,
|
|
201
201
|
`step` being the step's number in the session, counted from 1.
|
|
202
202
|
`rate` is the rate a track's frames play at as the runtime reported it — frames per second
|
|
203
203
|
for video, samples per second for audio, `0` when the runtime did not say. `elapsed` is the seconds the step's `generate()` took (or the time the model reported for
|
|
@@ -207,16 +207,24 @@ await reactor.step();
|
|
|
207
207
|
file the model kept with the step; the bytes are served over HTTP from the session's step
|
|
208
208
|
results (`GET /sessions/{id}/steps/{step}`), not carried in the result. `error` is set when
|
|
209
209
|
the step ran and the model reported a failure — such a step resolves, it does not reject.
|
|
210
|
+
`data` is a `Uint8Array` of what the model returned for the step besides its media, opaque
|
|
211
|
+
to the SDK: the bytes it returned as they are, or a dict it returned encoded as JSON
|
|
212
|
+
(`JSON.parse(new TextDecoder().decode(result.data))`). It arrives with the step's frames, so a
|
|
213
|
+
model whose output is data alone needs no output track; it is absent when the model returned none.
|
|
210
214
|
- `publishTrack(name)` resolves with a handle whose `pushFrame(frame, { metadata?,
|
|
211
|
-
captureTimeUs? })`
|
|
215
|
+
captureTimeUs? })` hands a frame to the runtime (`reactor.pushFrame(name, frame)` does
|
|
212
216
|
the same, for a track published on this connection). The handle stops working once the track
|
|
213
217
|
is unpublished or its connection ends; publish again for a new one. It takes a `VideoFrame`, an `ImageData`, `{ data, width, height, format? }` raw
|
|
214
218
|
`"rgba"`/`"rgb"`/`"bgra"` pixels, or `{ samples, sampleRate, channels? }` interleaved
|
|
215
219
|
`Int16Array` PCM for an audio track. Pushes keep their order, and `step()` waits for every
|
|
216
220
|
push called before it. A refused push rejects, so await it, or `.catch()` it in a frame loop.
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
221
|
+
Once the session is ready, against a runtime that takes pushed frames, each push is sent at
|
|
222
|
+
once over the data channel, ahead of any step after it, and one the channel cannot take is
|
|
223
|
+
dropped and raised as an error event. A frame pushed before the session is ready, or to an
|
|
224
|
+
older runtime, is buffered instead and sent with the next push or step. At most one
|
|
225
|
+
message's worth of frames waits in that buffer: pushing faster than steps run drops the
|
|
226
|
+
oldest frames, across tracks. `clearStepInputs()` drops whatever is waiting; frames already
|
|
227
|
+
sent to the runtime are not recalled.
|
|
220
228
|
- The `MediaStreamTrack` argument of `publishTrack()` is optional in a stepped session and
|
|
221
229
|
ignored there, with a console warning: no RTP is sent, frames go through `pushFrame()`. In
|
|
222
230
|
a streamed session it is still required, and `publishTrack(name)` without one rejects with
|
|
@@ -229,9 +237,96 @@ await reactor.step();
|
|
|
229
237
|
`command_code` the code its handler raised), `StepSessionEndedError`
|
|
230
238
|
(`step_session_ended`: the session ended before the step's result came back),
|
|
231
239
|
`StepResultTooLargeError` (`step_result_too_large`), `UnsupportedFrameEncodingError`, and
|
|
232
|
-
`MessageTooLargeError` when the request is too large to send. The frames
|
|
240
|
+
`MessageTooLargeError` when the request is too large to send. The frames buffered for a step
|
|
233
241
|
belong to it: when its request cannot be sent, the step rejects and its frames are dropped
|
|
234
242
|
with it, not kept for the next step.
|
|
243
|
+
- A step's result travels in one data-channel message, whose limit is the smaller of the
|
|
244
|
+
client's `maxMessageSize` option (default 64 MiB) and the runtime's, which is 64 MiB unless
|
|
245
|
+
the model raises it in its `reactor.yaml` (`runtime.webrtc.max_message_size`). Raise
|
|
246
|
+
`maxMessageSize` only for a model whose steps need it: a message is held whole in the tab's
|
|
247
|
+
memory while it arrives.
|
|
248
|
+
|
|
249
|
+
### Encoded step frames
|
|
250
|
+
|
|
251
|
+
Video step frames travel encoded both ways when the browser and the runtime share a codec, so a
|
|
252
|
+
720p frame costs tens of KB rather than 2.7 MB. The SDK does it with WebCodecs: the frames you
|
|
253
|
+
push on an input track are encoded before they are sent, and the frames of an output track are
|
|
254
|
+
decoded before `step()` resolves. Your code still pushes and receives RGB; a result frame's
|
|
255
|
+
`encoding` names the codec it travelled in.
|
|
256
|
+
|
|
257
|
+
```ts
|
|
258
|
+
reactor.setCodecPreference(["vp9", "h264"]); // VP9 when both ends have it
|
|
259
|
+
reactor.setCodecPreference([]); // raw frames
|
|
260
|
+
reactor.setCodecPreference(["av1"], { track: "main_video" });
|
|
261
|
+
await reactor.connect();
|
|
262
|
+
reactor.getStepFrameEncodings(); // { main_video: "av1", camera: "vp9" }
|
|
263
|
+
```
|
|
264
|
+
|
|
265
|
+
- The codecs are `"h264"`, `"vp9"`, `"av1"` and `"vp8"`; H.264 comes first by default. A list
|
|
266
|
+
is an order, not a limit: the codecs it leaves out follow it, so a runtime without the
|
|
267
|
+
preferred ones takes another. Of those, the ones this browser can encode (input tracks) or
|
|
268
|
+
decode (output tracks), as `VideoEncoder.isConfigSupported` and
|
|
269
|
+
`VideoDecoder.isConfigSupported` report, are offered, and the runtime takes per track the
|
|
270
|
+
first it shares. A track they share no codec for, an audio track, and every track of a
|
|
271
|
+
runtime without encoded step frames stay raw.
|
|
272
|
+
- `setCodecPreference()` can be called before `connect()` or during a session. It takes effect
|
|
273
|
+
on the next connection or reconnect; the live connection keeps what it negotiated.
|
|
274
|
+
- Each track is one stream for the whole connection. A track's first frame on a connection is
|
|
275
|
+
a keyframe, and so is the next one after the runtime could not decode the track's pushed
|
|
276
|
+
frames. When an output track's decoder loses its stream, that step rejects with a
|
|
277
|
+
`DecodeError`, and the SDK pauses and resumes that track so the runtime starts it again at a
|
|
278
|
+
keyframe; its frames fail until that keyframe arrives.
|
|
279
|
+
|
|
280
|
+
### Encoded video you encode yourself
|
|
281
|
+
|
|
282
|
+
A step can also carry video you encoded yourself, its bytes as they are: one H.264 access unit
|
|
283
|
+
(Annex B), one VP8 or VP9 frame, or one AV1 temporal unit. The runtime decodes a track's
|
|
284
|
+
encoded frames as one stream, so the model reads RGB frames exactly as if they had been pushed
|
|
285
|
+
raw. Give the frames in decode order, starting at a keyframe. A WebCodecs `VideoEncoder`'s
|
|
286
|
+
chunks go in as they are; configure it for Annex B, as its default `avc` format is
|
|
287
|
+
length-prefixed:
|
|
288
|
+
|
|
289
|
+
```ts
|
|
290
|
+
import type { EncodedFrame } from "@reactor-team/js-sdk";
|
|
291
|
+
|
|
292
|
+
let gop: EncodedFrame[] = [];
|
|
293
|
+
const encoder = new VideoEncoder({
|
|
294
|
+
output: (chunk) => {
|
|
295
|
+
if (chunk.type === "key") gop = [];
|
|
296
|
+
gop.push({ codec: "h264", data: chunk, width: 1280, height: 720 });
|
|
297
|
+
},
|
|
298
|
+
error: console.error,
|
|
299
|
+
});
|
|
300
|
+
encoder.configure({ codec: "avc1.42001f", width: 1280, height: 720, avc: { format: "annexb" } });
|
|
301
|
+
|
|
302
|
+
// ... encode frames, then send the ones from the last keyframe with the step:
|
|
303
|
+
const result = await reactor.step({ media: { camera: gop } });
|
|
304
|
+
|
|
305
|
+
// The same, pushed ahead of the step:
|
|
306
|
+
for (const frame of gop) await camera.pushEncodedFrame(frame);
|
|
307
|
+
await reactor.step();
|
|
308
|
+
```
|
|
309
|
+
|
|
310
|
+
- An `EncodedFrame` is `{ codec, data, width, height, metadata?, captureTimeUs?, keyframe? }`:
|
|
311
|
+
`codec` one of `"h264"`, `"vp8"`, `"vp9"` or `"av1"`, `data` one frame as a `Uint8Array`, an
|
|
312
|
+
`ArrayBuffer` or an `EncodedVideoChunk` (its bytes are copied out when pushed), `width` and
|
|
313
|
+
`height` the frame it decodes to, and `keyframe` whether a decoder can start from it (read
|
|
314
|
+
from a chunk's `type` when not given). `codec` is what tells it from a raw frame in `media`.
|
|
315
|
+
- `media` maps input track names to frames: `EncodedFrame`s, or anything `pushFrame()` takes.
|
|
316
|
+
They go after the frames pushed to the same track since the previous step, and only once
|
|
317
|
+
`state` and `commands` are known to be well formed and the session is ready, so a step
|
|
318
|
+
refused for those leaves the buffer as it was. Each track must be published first. A bad
|
|
319
|
+
track or frame anywhere in `media` fails the step before any of it is pushed, leaving the
|
|
320
|
+
buffer as it was too. A session that leaves ready once the media is in has the step refused
|
|
321
|
+
and keeps the frames for the next one, as pushed frames are. A push or step called after
|
|
322
|
+
`step()` waits for it to take the buffer, so its frames go to the next step, not this one.
|
|
323
|
+
- `pushEncodedFrame()` rejects with an `InvalidStateError` outside a stepped session, on an
|
|
324
|
+
audio or recvonly track, a track not published on this connection, empty data or a zero
|
|
325
|
+
dimension. A frame already sent to the runtime that does not decode raises an
|
|
326
|
+
`input_decode_failed` error event naming the track, whose next frame must then be a
|
|
327
|
+
keyframe. A step whose buffered frames do not decode, or do not start at a keyframe, rejects
|
|
328
|
+
with a `StepRefusedError` naming the frame in `entry`; a runtime that predates encoded frames
|
|
329
|
+
refuses them the same way.
|
|
235
330
|
|
|
236
331
|
## Recording and clips
|
|
237
332
|
|