@reactor-team/js-sdk 3.2.0 → 3.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -154,6 +154,85 @@ matching `schemaReceived`/`capabilitiesReceived` events. See
154
154
  [`getSchema()`](https://docs.reactor.inc/sdk-reference/reactor-class#getschema) and
155
155
  [`Capabilities`](https://docs.reactor.inc/sdk-reference/types#capabilities) for the full shape.
156
156
 
157
+ ## Stepped generation
158
+
159
+ A model that generates one step at a time — rather than streaming continuously — is driven
160
+ with `step()`. Create the client with `generation: "stepped"`; `connect()` fails with a
161
+ `GenerationModeMismatchError` when the runtime does not start the session that way.
162
+
163
+ ```ts
164
+ import { Reactor, StepRefusedError, stepFrameToImageData } from "@reactor-team/js-sdk";
165
+
166
+ const reactor = new Reactor({ modelName: "my-stepped-model", generation: "stepped", jwt });
167
+ await reactor.connect();
168
+
169
+ // Input frames never travel over RTP in a stepped session: claim the input
170
+ // track, push frames through it, and the next step carries them.
171
+ const camera = await reactor.publishTrack("camera");
172
+ await camera.pushFrame(canvasCtx.getImageData(0, 0, 640, 360));
173
+
174
+ try {
175
+ const result = await reactor.step({
176
+ state: { prompt: "a cat" },
177
+ commands: [{ command: "jump", data: { height: 2 } }],
178
+ });
179
+ if (result.error) console.warn(`step ${result.step} failed:`, result.error);
180
+ const frame = result.tracks.find((t) => t.track === "main_video")?.frames[0];
181
+ if (frame) outputCtx.putImageData(stepFrameToImageData(frame), 0, 0);
182
+ } catch (error) {
183
+ if (error instanceof StepRefusedError) console.warn("refused:", error.entry, error.message);
184
+ else throw error;
185
+ }
186
+
187
+ // Every part of a step is optional: this one runs on the model's current state.
188
+ await reactor.step();
189
+ ```
190
+
191
+ - `step({ state, commands })` sends `state`, `commands` and every frame pushed since the
192
+ previous step in one request. The runtime applies them in order — each `state` key as the
193
+ model's `set_<key>` command, then `commands`, then the frames — and then generates. The
194
+ runtime checks the whole request before it applies any of it, so a refused step changes
195
+ nothing; past that check they are separate calls, and a handler that fails does not undo
196
+ the ones before it.
197
+ `sendCommand()` outside a step still works and changes the state the next step uses. A file
198
+ input goes in the way it does for a command: `uploadFile()` it, then put the `FileRef` in a
199
+ command's `data` or a state value.
200
+ - It resolves with `{ step, tracks: [{ track, kind, rate, frames }], elapsed, files, error? }`,
201
+ `step` being the step's number in the session, counted from 1.
202
+ `rate` is the rate a track's frames play at as the runtime reported it — frames per second
203
+ for video, samples per second for audio, `0` when the runtime did not say. `elapsed` is the seconds the step's `generate()` took (or the time the model reported for
204
+ the step); each frame has `dtype`, `shape`, typed-array `data` (`Uint8Array` RGB
205
+ `(height, width, 3)` for video, `Int16Array` planar `(channels, samples)` for audio), and
206
+ optional `metadata` / `captureTimeUs`. `files` lists `{ name, contentType, size }` for each
207
+ file the model kept with the step; the bytes are served over HTTP from the session's step
208
+ results (`GET /sessions/{id}/steps/{step}`), not carried in the result. `error` is set when
209
+ the step ran and the model reported a failure — such a step resolves, it does not reject.
210
+ - `publishTrack(name)` resolves with a handle whose `pushFrame(frame, { metadata?,
211
+ captureTimeUs? })` buffers a frame for the next step (`reactor.pushFrame(name, frame)` does
212
+ the same, for a track published on this connection). The handle stops working once the track
213
+ is unpublished or its connection ends; publish again for a new one. It takes a `VideoFrame`, an `ImageData`, `{ data, width, height, format? }` raw
214
+ `"rgba"`/`"rgb"`/`"bgra"` pixels, or `{ samples, sampleRate, channels? }` interleaved
215
+ `Int16Array` PCM for an audio track. Pushes keep their order, and `step()` waits for every
216
+ push called before it. A refused push rejects, so await it, or `.catch()` it in a frame loop.
217
+ At most one message's worth of frames waits for the next step:
218
+ pushing faster than steps run drops the oldest frames, across tracks. `clearStepInputs()`
219
+ drops whatever is waiting.
220
+ - The `MediaStreamTrack` argument of `publishTrack()` is optional in a stepped session and
221
+ ignored there, with a console warning: no RTP is sent, frames go through `pushFrame()`. In
222
+ a streamed session it is still required, and `publishTrack(name)` without one rejects with
223
+ an `InvalidStateError`.
224
+ - Refusals are typed: `StepRefusedError` (`step_refused`: nothing the step carried was
225
+ applied, and `entry` names the part of the request at fault), `StepNotReadyError`
226
+ (`step_not_ready`: the session cannot take a step yet, and the step did not run),
227
+ `StepCommandFailedError` (`step_command_failed`: a state key or command failed, the calls
228
+ before it stay applied, and the step did not run; `entry` names the call and
229
+ `command_code` the code its handler raised), `StepSessionEndedError`
230
+ (`step_session_ended`: the session ended before the step's result came back),
231
+ `StepResultTooLargeError` (`step_result_too_large`), `UnsupportedFrameEncodingError`, and
232
+ `MessageTooLargeError` when the request is too large to send. The frames pushed for a step
233
+ belong to it: when its request cannot be sent, the step rejects and its frames are dropped
234
+ with it, not kept for the next step.
235
+
157
236
  ## Recording and clips
158
237
 
159
238
  ```ts