@reactor-team/js-sdk 3.2.0 → 3.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +131 -0
- package/dist/index.cjs +752 -9
- package/dist/index.cjs.map +1 -1
- package/dist/index.d.cts +399 -5
- package/dist/index.d.ts +399 -5
- package/dist/index.js +743 -9
- package/dist/index.js.map +1 -1
- package/dist/wasm/README.md +38 -2
- package/dist/wasm/reactor_wasm.d.ts +229 -2
- package/dist/wasm/reactor_wasm.js +348 -8
- package/dist/wasm/reactor_wasm_bg.wasm.d.ts +10 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -154,6 +154,137 @@ matching `schemaReceived`/`capabilitiesReceived` events. See
|
|
|
154
154
|
[`getSchema()`](https://docs.reactor.inc/sdk-reference/reactor-class#getschema) and
|
|
155
155
|
[`Capabilities`](https://docs.reactor.inc/sdk-reference/types#capabilities) for the full shape.
|
|
156
156
|
|
|
157
|
+
## Stepped generation
|
|
158
|
+
|
|
159
|
+
A model that generates one step at a time — rather than streaming continuously — is driven
|
|
160
|
+
with `step()`. Create the client with `generation: "stepped"`; `connect()` fails with a
|
|
161
|
+
`GenerationModeMismatchError` when the runtime does not start the session that way.
|
|
162
|
+
|
|
163
|
+
```ts
|
|
164
|
+
import { Reactor, StepRefusedError, stepFrameToImageData } from "@reactor-team/js-sdk";
|
|
165
|
+
|
|
166
|
+
const reactor = new Reactor({ modelName: "my-stepped-model", generation: "stepped", jwt });
|
|
167
|
+
await reactor.connect();
|
|
168
|
+
|
|
169
|
+
// Input frames never travel over RTP in a stepped session: claim the input
|
|
170
|
+
// track, push frames through it, and the next step carries them.
|
|
171
|
+
const camera = await reactor.publishTrack("camera");
|
|
172
|
+
await camera.pushFrame(canvasCtx.getImageData(0, 0, 640, 360));
|
|
173
|
+
|
|
174
|
+
try {
|
|
175
|
+
const result = await reactor.step({
|
|
176
|
+
state: { prompt: "a cat" },
|
|
177
|
+
commands: [{ command: "jump", data: { height: 2 } }],
|
|
178
|
+
});
|
|
179
|
+
if (result.error) console.warn(`step ${result.step} failed:`, result.error);
|
|
180
|
+
const frame = result.tracks.find((t) => t.track === "main_video")?.frames[0];
|
|
181
|
+
if (frame) outputCtx.putImageData(stepFrameToImageData(frame), 0, 0);
|
|
182
|
+
} catch (error) {
|
|
183
|
+
if (error instanceof StepRefusedError) console.warn("refused:", error.entry, error.message);
|
|
184
|
+
else throw error;
|
|
185
|
+
}
|
|
186
|
+
|
|
187
|
+
// Every part of a step is optional: this one runs on the model's current state.
|
|
188
|
+
await reactor.step();
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
- `step({ state, commands })` sends `state`, `commands` and every frame pushed since the
|
|
192
|
+
previous step in one request. The runtime applies them in order — each `state` key as the
|
|
193
|
+
model's `set_<key>` command, then `commands`, then the frames — and then generates. The
|
|
194
|
+
runtime checks the whole request before it applies any of it, so a refused step changes
|
|
195
|
+
nothing; past that check they are separate calls, and a handler that fails does not undo
|
|
196
|
+
the ones before it.
|
|
197
|
+
`sendCommand()` outside a step still works and changes the state the next step uses. A file
|
|
198
|
+
input goes in the way it does for a command: `uploadFile()` it, then put the `FileRef` in a
|
|
199
|
+
command's `data` or a state value.
|
|
200
|
+
- It resolves with `{ step, tracks: [{ track, kind, rate, frames }], elapsed, files, error? }`,
|
|
201
|
+
`step` being the step's number in the session, counted from 1.
|
|
202
|
+
`rate` is the rate a track's frames play at as the runtime reported it — frames per second
|
|
203
|
+
for video, samples per second for audio, `0` when the runtime did not say. `elapsed` is the seconds the step's `generate()` took (or the time the model reported for
|
|
204
|
+
the step); each frame has `dtype`, `shape`, typed-array `data` (`Uint8Array` RGB
|
|
205
|
+
`(height, width, 3)` for video, `Int16Array` planar `(channels, samples)` for audio), and
|
|
206
|
+
optional `metadata` / `captureTimeUs`. `files` lists `{ name, contentType, size }` for each
|
|
207
|
+
file the model kept with the step; the bytes are served over HTTP from the session's step
|
|
208
|
+
results (`GET /sessions/{id}/steps/{step}`), not carried in the result. `error` is set when
|
|
209
|
+
the step ran and the model reported a failure — such a step resolves, it does not reject.
|
|
210
|
+
- `publishTrack(name)` resolves with a handle whose `pushFrame(frame, { metadata?,
|
|
211
|
+
captureTimeUs? })` buffers a frame for the next step (`reactor.pushFrame(name, frame)` does
|
|
212
|
+
the same, for a track published on this connection). The handle stops working once the track
|
|
213
|
+
is unpublished or its connection ends; publish again for a new one. It takes a `VideoFrame`, an `ImageData`, `{ data, width, height, format? }` raw
|
|
214
|
+
`"rgba"`/`"rgb"`/`"bgra"` pixels, or `{ samples, sampleRate, channels? }` interleaved
|
|
215
|
+
`Int16Array` PCM for an audio track. Pushes keep their order, and `step()` waits for every
|
|
216
|
+
push called before it. A refused push rejects, so await it, or `.catch()` it in a frame loop.
|
|
217
|
+
At most one message's worth of frames waits for the next step:
|
|
218
|
+
pushing faster than steps run drops the oldest frames, across tracks. `clearStepInputs()`
|
|
219
|
+
drops whatever is waiting.
|
|
220
|
+
- The `MediaStreamTrack` argument of `publishTrack()` is optional in a stepped session and
|
|
221
|
+
ignored there, with a console warning: no RTP is sent, frames go through `pushFrame()`. In
|
|
222
|
+
a streamed session it is still required, and `publishTrack(name)` without one rejects with
|
|
223
|
+
an `InvalidStateError`.
|
|
224
|
+
- Refusals are typed: `StepRefusedError` (`step_refused`: nothing the step carried was
|
|
225
|
+
applied, and `entry` names the part of the request at fault), `StepNotReadyError`
|
|
226
|
+
(`step_not_ready`: the session cannot take a step yet, and the step did not run),
|
|
227
|
+
`StepCommandFailedError` (`step_command_failed`: a state key or command failed, the calls
|
|
228
|
+
before it stay applied, and the step did not run; `entry` names the call and
|
|
229
|
+
`command_code` the code its handler raised), `StepSessionEndedError`
|
|
230
|
+
(`step_session_ended`: the session ended before the step's result came back),
|
|
231
|
+
`StepResultTooLargeError` (`step_result_too_large`), `UnsupportedFrameEncodingError`, and
|
|
232
|
+
`MessageTooLargeError` when the request is too large to send. The frames pushed for a step
|
|
233
|
+
belong to it: when its request cannot be sent, the step rejects and its frames are dropped
|
|
234
|
+
with it, not kept for the next step.
|
|
235
|
+
- A step's result travels in one data-channel message, whose limit is the smaller of the
|
|
236
|
+
client's `maxMessageSize` option (default 64 MiB) and the runtime's, which is 64 MiB unless
|
|
237
|
+
the model raises it in its `reactor.yaml` (`runtime.webrtc.max_message_size`). Raise
|
|
238
|
+
`maxMessageSize` only for a model whose steps need it: a message is held whole in the tab's
|
|
239
|
+
memory while it arrives.
|
|
240
|
+
|
|
241
|
+
### Encoded video in a step
|
|
242
|
+
|
|
243
|
+
A step can carry video already encoded as H.264, its bytes as they are: much smaller than raw
|
|
244
|
+
RGB. The runtime decodes a track's encoded frames before the step runs, so the model reads RGB
|
|
245
|
+
frames exactly as if they had been pushed raw. Give the frames of one step in decode order,
|
|
246
|
+
starting at a keyframe. A WebCodecs `VideoEncoder`'s chunks go in as they are; configure it for
|
|
247
|
+
Annex B, as its default `avc` format is length-prefixed:
|
|
248
|
+
|
|
249
|
+
```ts
|
|
250
|
+
import type { EncodedFrame } from "@reactor-team/js-sdk";
|
|
251
|
+
|
|
252
|
+
let gop: EncodedFrame[] = [];
|
|
253
|
+
const encoder = new VideoEncoder({
|
|
254
|
+
output: (chunk) => {
|
|
255
|
+
if (chunk.type === "key") gop = [];
|
|
256
|
+
gop.push({ codec: "h264", data: chunk, width: 1280, height: 720 });
|
|
257
|
+
},
|
|
258
|
+
error: console.error,
|
|
259
|
+
});
|
|
260
|
+
encoder.configure({ codec: "avc1.42001f", width: 1280, height: 720, avc: { format: "annexb" } });
|
|
261
|
+
|
|
262
|
+
// ... encode frames, then send the ones from the last keyframe with the step:
|
|
263
|
+
const result = await reactor.step({ media: { camera: gop } });
|
|
264
|
+
|
|
265
|
+
// The same, pushed ahead of the step:
|
|
266
|
+
for (const frame of gop) await camera.pushEncodedFrame(frame);
|
|
267
|
+
await reactor.step();
|
|
268
|
+
```
|
|
269
|
+
|
|
270
|
+
- An `EncodedFrame` is `{ codec: "h264", data, width, height, metadata?, captureTimeUs? }`:
|
|
271
|
+
`data` one access unit as a `Uint8Array`, an `ArrayBuffer` or an `EncodedVideoChunk` (its
|
|
272
|
+
bytes are copied out when pushed), `width` and `height` the frame it decodes to. `codec` is
|
|
273
|
+
what tells it from a raw frame in `media`.
|
|
274
|
+
- `media` maps input track names to frames: `EncodedFrame`s, or anything `pushFrame()` takes.
|
|
275
|
+
They go after the frames pushed to the same track since the previous step, and only once
|
|
276
|
+
`state` and `commands` are known to be well formed and the session is ready, so a step
|
|
277
|
+
refused for those leaves the buffer as it was. Each track must be published first. A bad
|
|
278
|
+
track or frame anywhere in `media` fails the step before any of it is pushed, leaving the
|
|
279
|
+
buffer as it was too. A session that leaves ready once the media is in has the step refused
|
|
280
|
+
and keeps the frames for the next one, as pushed frames are. A push or step called after
|
|
281
|
+
`step()` waits for it to take the buffer, so its frames go to the next step, not this one.
|
|
282
|
+
- `pushEncodedFrame()` rejects with an `InvalidStateError` outside a stepped session, on an
|
|
283
|
+
audio or recvonly track, a track not published on this connection, empty data or a zero
|
|
284
|
+
dimension. A step whose frames do not decode, or do not start at a keyframe, rejects with a
|
|
285
|
+
`StepRefusedError` naming the frame in `entry`; a runtime that predates encoded frames
|
|
286
|
+
refuses them the same way.
|
|
287
|
+
|
|
157
288
|
## Recording and clips
|
|
158
289
|
|
|
159
290
|
```ts
|