@kubohiroya/turbowarp-teachable-machine 3.5.6 → 3.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,22 +1,16 @@
1
1
  # TurboWarp Teachable Machine
2
2
 
3
- Use a [Teachable Machine](https://teachablemachine.withgoogle.com/) pose, image, or audio model as
4
- an input for TurboWarp projects. The extension turns each camera frame or microphone window into labels,
5
- confidence scores, and Boolean conditions that Scratch-style scripts can use.
3
+ <img src="assets/block-icon.svg" alt="" width="48" height="48">
6
4
 
7
- TurboWarp Teachable Machine uses the Scratch extension ID `kubohiroyatm` from the 2.x release line onward to
8
- avoid short-ID collisions while the package, repository, Pages URL, and browser build use
9
- `turbowarp-teachable-machine`.
5
+ English | [日本語](README.ja.md)
10
6
 
11
- > The package, repository, and Pages URL were previously named `turbowarp-tm`
12
- > (`@kubohiroya/turbowarp-tm`, last published as 3.2.0). Replace that name with
13
- > `turbowarp-teachable-machine` in CDN URLs, `pnpm add`, and `import` specifiers.
7
+ Use a published [Teachable Machine](https://teachablemachine.withgoogle.com/) pose, image, or audio
8
+ model as an input for TurboWarp projects. Every camera frame or microphone window becomes a label, a
9
+ confidence score, and Boolean conditions that Scratch-style scripts can act on.
14
10
 
15
- **[Open the illustrated user guide (English)](https://kubohiroya.github.io/turbowarp-teachable-machine/)** ·
16
- **[日本語ガイド](https://kubohiroya.github.io/turbowarp-teachable-machine/ja/)** ·
17
- [Block reference](#blocks)
11
+ **User guide:** [English](https://kubohiroya.github.io/turbowarp-teachable-machine/) | [日本語](https://kubohiroya.github.io/turbowarp-teachable-machine/ja/) · **[Block reference](#blocks)**
18
12
 
19
- ## What The Extension Does
13
+ ## What it does
20
14
 
21
15
  ```mermaid
22
16
  flowchart LR
@@ -26,318 +20,80 @@ flowchart LR
26
20
  Scores --> Blocks["TurboWarp blocks"]
27
21
  ```
28
22
 
29
- - loads a published Teachable Machine Pose, Image, or Audio model;
30
- - switches recognition mode between `pose`, `image`, and `audio` before input startup;
31
- - starts and stops the camera independently from recognition;
32
- - places a configurable camera preview over the TurboWarp stage;
33
- - optionally overlays configurable SVG pose joints and bones on that preview in pose mode;
34
- - reports the best label, its confidence, and the confidence of any named label;
35
- - tests labels with a fixed or custom confidence threshold;
36
- - optionally smooths decisions with time-decayed accumulated scores;
37
- - reports startup timings and explicit runtime errors.
23
+ - Loads a published Teachable Machine pose, image, or audio model from its model folder URL.
24
+ - Switches recognition between `pose`, `image`, and `audio` before the input starts.
25
+ - Starts and stops the camera independently from recognition.
26
+ - Places a camera preview over the stage, at six positions, with adjustable opacity and mirroring.
27
+ - Draws an optional SVG overlay of pose joints and bones on that preview.
28
+ - Reports the best label, its confidence, and the confidence of any named class.
29
+ - Tests a label against a fixed or custom confidence threshold.
30
+ - Smooths flickering decisions with time-decayed accumulated scores.
31
+ - Reports startup timings and the last runtime error.
38
32
 
39
- The [illustrated guide](https://kubohiroya.github.io/turbowarp-teachable-machine/) explains the complete flow,
40
- preview layout, score behavior, privacy, and troubleshooting. English is the default; the
41
- [Japanese version](https://kubohiroya.github.io/turbowarp-teachable-machine/ja/) has the same content.
33
+ Recognition runs in the browser. Camera frames and microphone audio are never uploaded; only the
34
+ runtime libraries and the published model are fetched.
42
35
 
43
36
  ## Requirements
44
37
 
45
- - a published Teachable Machine Pose, Image, or Audio model URL;
46
- - a camera or microphone and permission to use it in the browser;
47
- - network access for the reviewed Teachable Machine browser runtime and the model files;
48
- - TurboWarp's **Run without sandbox** option, which is only available when the extension is
49
- loaded from a file (see [Installation](#installation)).
38
+ - A published Teachable Machine pose, image, or audio model URL — the model folder URL, not the
39
+ editor URL.
40
+ - A camera or microphone, and browser permission to use it.
41
+ - Network access for the reviewed Teachable Machine browser runtime and the model files.
42
+ - TurboWarp's **Run without sandbox** option, which only exists when the extension is loaded from a
43
+ file. See [Installation](#installation).
50
44
 
51
45
  > [!IMPORTANT]
52
- > This is an unsandboxed extension because it needs camera, microphone, and stage access. Only load extension
53
- > code you trust. Camera APIs also require a secure browser context such as HTTPS or localhost.
46
+ > This extension is unsandboxed because it needs camera, microphone, and stage access. Load
47
+ > extension code only from sources you trust. Camera APIs also require a secure browser context
48
+ > such as HTTPS or `localhost`.
54
49
 
55
50
  ## Installation
56
51
 
57
- Download [`dist/tm.js`](dist/tm.js), then in TurboWarp's custom extension dialog open the
58
- **Files** tab, select that file, and keep **Run without sandbox** enabled.
52
+ Download [`dist/tm.js`](dist/tm.js). In TurboWarp's custom extension dialog, open the **Files** tab,
53
+ select that file, and keep **Run without sandbox** enabled.
59
54
 
60
55
  > [!IMPORTANT]
61
56
  > The dialog's **URL** tab cannot load this extension. TurboWarp always sandboxes an extension
62
- > loaded from an untrusted URL — the URL tab has no **Run without sandbox** option at all — and TM
63
- > refuses to start sandboxed because it needs camera, microphone, and stage access. Pasting the
64
- > jsDelivr link below into the URL tab leaves the dialog waiting with no message on screen and only
65
- > `TM must run without the extension sandbox.` in the browser console. Download the file and use
66
- > the **Files** tab instead.
57
+ > loaded from an untrusted URL — the URL tab has no **Run without sandbox** option at all — and this
58
+ > extension refuses to start sandboxed. Pasting the jsDelivr link below into the URL tab leaves the
59
+ > dialog waiting with nothing on screen and only `TM must run without the extension sandbox.` in the
60
+ > browser console. Download the file and use the **Files** tab.
67
61
 
68
- That download, and the artifact a composite runtime fetches, is the version-pinned build on
69
- jsDelivr:
62
+ The download, and the artifact a composite runtime fetches, is the version-pinned build on jsDelivr:
70
63
 
71
64
  ```text
72
- https://cdn.jsdelivr.net/npm/@kubohiroya/turbowarp-teachable-machine@3.5.6/dist/tm.js
65
+ https://cdn.jsdelivr.net/npm/@kubohiroya/turbowarp-teachable-machine@3.6.0/dist/tm.js
73
66
  ```
74
67
 
75
- The standalone extension loads one reviewed browser runtime that contains one TensorFlow.js 4.22.0
76
- module graph together with Teachable Machine Pose 0.8.6, Teachable Machine Image 0.8.5, and
77
- TensorFlow.js Speech Commands 0.5.4. That runtime ships the WebGL and CPU kernels; the WebGPU and
78
- WebAssembly backends live in separate bundles next to it and are fetched only when a project uses
79
- them. See [Compute backends](#compute-backends).
80
- Composite runtimes can load or embed the same artifact without rewriting a minified third-party
81
- bundle:
68
+ A composite runtime can load or embed the same reviewed artifact without rewriting a minified
69
+ third-party bundle:
82
70
 
83
71
  ```text
84
- https://cdn.jsdelivr.net/npm/@kubohiroya/turbowarp-teachable-machine@3.5.6/dist/runtime.js
72
+ https://cdn.jsdelivr.net/npm/@kubohiroya/turbowarp-teachable-machine@3.6.0/dist/runtime.js
85
73
  ```
86
74
 
87
75
  To add the published package to another project:
88
76
 
89
77
  ```sh
90
- pnpm add --save-exact @kubohiroya/turbowarp-teachable-machine@3.5.6
91
- ```
92
-
93
- ### Offline PoseNet bundle API
94
-
95
- `@kubohiroya/turbowarp-teachable-machine/posenet` owns the fixed PoseNet MobileNetV1 0.75 / stride 16
96
- manifest, package asset specifiers, SHA-256 verification, bounded Base64 project descriptor, and
97
- the Teachable Machine Pose fetch adapter. Host applications only decide where that explicit model
98
- descriptor is stored. The model stays as one JSON file and two binary shards; it is never disguised
99
- as an image, sound, or costume.
100
-
101
- ```js
102
- import {readFile} from 'node:fs/promises';
103
- import {
104
- createBundledTMRuntime,
105
- createPoseNetProjectBundleFromLoader,
106
- } from '@kubohiroya/turbowarp-teachable-machine/posenet';
107
-
108
- const projectBundle = await createPoseNetProjectBundleFromLoader((file) =>
109
- readFile(new URL(import.meta.resolve(file.packageSpecifier))),
110
- );
111
-
112
- // A browser host can store projectBundle in its own project-data channel.
113
- // Decode and hash verification start only on the first loadFromFiles() call.
114
- const offlineRuntime = createBundledTMRuntime({
115
- runtime: tmPose,
116
- projectBundle,
117
- });
118
- ```
119
-
120
- The model supply contains exactly 5,082,500 bytes. Its three SHA-256 operations run independently,
121
- and the first shared verification overlaps the Teachable Machine classifier load. The intercepted
122
- PoseNet fetch never exposes bytes until that verification succeeds. Later model requests reuse the
123
- same verified supply. Tampering, missing or incorrectly sized files, and unexpected PoseNet network
124
- requests fail closed with `TM-POSENET-*` error codes.
125
-
126
- ### Composition API
127
-
128
- Composite runtimes can import `@kubohiroya/turbowarp-teachable-machine/composition` without registering the
129
- Standalone extension or adding blocks. The caller supplies an already-bundled Teachable Machine
130
- Pose runtime and validated model bytes, so this path does not download runtime scripts or model
131
- files.
132
-
133
- ```js
134
- import {createTMComposition} from '@kubohiroya/turbowarp-teachable-machine/composition';
135
-
136
- const pose = createTMComposition({
137
- runtime: bundledTMRuntime,
138
- modelInitializationPolicy: 'latest-needed',
139
- // Optional latency-first mode; it remains off by default.
140
- // parallelModelInitialization: true,
141
- });
142
- const cameras = await pose.listCameraDevices();
143
- if (cameras[0]) await pose.selectCamera({deviceId: cameras[0].deviceId});
144
- const modelDemand = new AbortController();
145
- await pose.registerPoseModel(
146
- {
147
- name: 'RescuePose',
148
- files: [
149
- {path: 'model.json', bytes: modelBytes},
150
- {path: 'weights.bin', bytes: weightsBytes},
151
- {path: 'metadata.json', bytes: metadataBytes},
152
- ],
153
- },
154
- {signal: modelDemand.signal},
155
- );
156
- pose.activatePoseModel('RescuePose');
157
- pose.setPreviewOpacity(0.2);
158
- pose.setPreviewPosition('full-stage');
159
- pose.setPreviewMirroring('unmirrored');
160
- pose.setPoseJointStyle('leftWrist', {color: '#ff00aa', opacity: 0.8, radius: 6});
161
- pose.setPoseBoneStyle({color: '#00e5ff', opacity: 0.9, width: 3});
162
- pose.setPoseOverlayMinimumConfidence(0.5);
163
- pose.setPoseOverlayConfidenceScaling({
164
- jointOpacity: true,
165
- jointRadius: true,
166
- boneOpacity: true,
167
- boneWidth: true,
168
- });
169
- pose.configureAccumulatedPose({
170
- accumulationPerSecond: 1,
171
- decayPerSecond: 0.9,
172
- scoreThreshold: 0,
173
- });
174
- const unsubscribe = pose.subscribeAccumulatedPose((event) => {
175
- // event.poseName is the one currently selected candidate, or an empty string.
176
- });
177
- await pose.startRecognition();
78
+ pnpm add --save-exact @kubohiroya/turbowarp-teachable-machine@3.6.0
178
79
  ```
179
80
 
180
- Each composition owns its model registry, active model, camera, and recognition state. Release one
181
- model with `releasePoseModel(name)` or release the complete instance with `releaseAll()`. Calling
182
- `releaseAll()` also removes accumulated-pose listeners. `subscribeAccumulatedPose()` returns an
183
- idempotent unsubscribe function.
184
-
185
- `releasePoseModel(name)` stops recognition when the selected model is active and disposes only that
186
- model. It does not stop the camera stream. Camera capture is an independent lifecycle: stop it with
187
- `stopCamera()`, or release the complete composition with `releaseAll()`.
188
-
189
- The default `modelInitializationPolicy` is `legacy`, preserving independent registration calls.
190
- Opt in to `latest-needed` when a host has one current label-model demand, such as a story that can
191
- skip scenes. At most one heavy runtime load is active and one latest request is pending. A newer
192
- request cancels the active request cooperatively and replaces an older request that has not started.
193
- For A loading, then B pending, then C requested, B never reaches the TensorFlow loader; A is cleaned
194
- up at its next safe boundary and only C starts afterward.
195
- Repeated `latest-needed` registrations with the same name, byte-identical files, and the same
196
- `AbortSignal` share one pending operation instead of invoking the runtime twice. Different signals
197
- remain separate demands so cancelling one caller cannot silently cancel another caller's work.
198
-
199
- Pass an optional `AbortSignal` to `registerPoseModel()` when the host no longer needs that demand. A
200
- queued request rejects with `AbortError` without invoking the runtime. TensorFlow.js 4.22.0 and Web
201
- Crypto do not expose physical cancellation for every operation already in progress, so a running
202
- request becomes stale immediately, starts no later phase where the runtime provides a boundary, and
203
- disposes every late classifier/PoseNet result exactly once before its promise settles. A cancelled
204
- model is never published in the registry. Shared fixed-PoseNet verification may finish and remain
205
- cached when the next latest request also needs it. Model cancellation does not stop the camera.
206
-
207
- The composition API does not infer scene or action reachability. A host should retain the signal when the same
208
- model is needed by an imminent action, abort it when a skipped path has no nearby pose demand, or
209
- submit the replacement model when a skip changes that demand. `releaseAll()` cancels both active and
210
- pending registration work and waits for safe cleanup.
211
-
212
- Runtime model phases remain sequential by default. Cancellation is checked after TensorFlow
213
- readiness, classifier loading, metadata parsing, and PoseNet loading, so a skipped demand does not
214
- start the next expensive phase. Set `parallelModelInitialization: true` on the composition or
215
- `createBundledTMRuntime()` only when lower startup latency is more important than those phase
216
- boundaries. That opt-in starts classifier, metadata, and PoseNet initialization together; work
217
- already accepted by TensorFlow cannot be physically interrupted, but every completed stale resource
218
- is still disposed before cancellation settles. The `latest-needed` queue continues to permit only
219
- one active demand and one newest pending demand in either mode.
220
-
221
- `setPreviewMirroring('mirrored' | 'unmirrored')` changes only the camera preview. It can be called
222
- before camera startup, while the camera is running, or after it has stopped. The default remains
223
- `mirrored`, and the recognition input keeps the Teachable Machine runtime's existing horizontal
224
- flip regardless of the preview setting.
225
-
226
- `setPreviewOpacity()` accepts a finite number from `0` to `1` and applies before or after camera
227
- startup. `setPreviewPosition()` accepts `top-left`, `top-right`, `bottom-left`, `bottom-right`,
228
- `center`, or `full-stage`. `hidePreview()` and `showPreview()` change only preview visibility; they
229
- do not stop or restart the camera stream. Use `stopCamera()` when the MediaStream tracks must be
230
- released.
231
-
232
- The composition API enables its SVG pose overlay and exposes `showPoseOverlay()`,
233
- `hidePoseOverlay()`, and `isPoseOverlayVisible()`. `setPoseJointStyle()` configures the CSS color,
234
- opacity, and radius for any of the 17 PoseNet joint names. `setPoseBoneStyle()` configures the color,
235
- opacity, and width shared by the 12 standard PoseNet bone connections. Opacity must be from `0` to
236
- `1`; radii and widths must be finite and non-negative. `setPoseOverlayMinimumConfidence()` accepts
237
- `0` through `1` and hides a bone when either endpoint is below the threshold.
238
-
239
- `setPoseOverlayConfidenceScaling()` independently controls `jointOpacity`, `jointRadius`,
240
- `boneOpacity`, and `boneWidth`. An enabled joint property is multiplied by that joint's confidence,
241
- clamped to `0`–`1`. An enabled bone property is multiplied by the lower confidence of its two
242
- endpoints. Hiding the overlay does not stop recognition. Hiding the camera preview also hides the
243
- overlay, while stopping recognition clears every SVG mark and stopping the camera removes the SVG
244
- element.
245
-
246
- `listCameraDevices()` performs a fresh browser enumeration on every call. It returns a deeply
247
- frozen canonical copy containing only unique, non-empty video device IDs in enumeration order;
248
- labels remain the empty string when permission has not made them available. Empty IDs are omitted,
249
- and the first occurrence wins when an ID is duplicated. Enumeration failures use
250
- `TM-COMPOSITION-012` and never expose `MediaDeviceInfo`, `MediaStreamTrack`, or a stream.
251
-
252
- Select `default`, `front`, or `back` as a camera preference, or wrap a session-only browser ID as
253
- `{deviceId}`. The object form is intentionally distinct, so a physical device whose ID is literally
254
- `default`, `front`, or `back` is not mistaken for a preference. A selection can be made before
255
- camera startup or while recognition is running. Running switches preserve the prepared model,
256
- recognition state, and all preview settings. Concurrent selections run in call order; a failed
257
- switch restores the preceding successful camera and selection. Invalid selections use
258
- `TM-COMPOSITION-011`. `getCameraSelection()` returns an isolated immutable copy, and
259
- `getActiveCamera()` returns an immutable `{deviceId, label}` only while an identifiable camera is
260
- running. The extension never persists device IDs. `releaseAll()` rejects queued selection work, waits for
261
- an in-progress switch to become quiescent, and then stops the final stream; all four camera-device
262
- methods fail closed after release.
263
-
264
- The extension owns the camera canvas used by Teachable Machine and TensorFlow.js and requests a normal
265
- Canvas2D context for it. Physical-camera measurements of its 320x240, one-draw/one-read path
266
- did not show a repeatable end-to-end benefit from `willReadFrequently`; that hint improved a
267
- different read-heavy condition with four or more reads per draw, but could make video drawing more
268
- expensive. Chrome may therefore emit its Canvas2D readback warning during CPU inference. Preview
269
- rendering and model input continue to use the same canvas, and no downstream Canvas prototype or
270
- Console patch is required. Reconsider the hint only with a reproducible benchmark of this exact
271
- camera path.
272
-
273
- Teachable Machine Pose 0.8.6 exposes the classifier as `CustomPoseNet.model` and PoseNet as
274
- `CustomPoseNet.posenetModel`, while its top-level `dispose()` releases PoseNet only. The composition
275
- therefore disconnects an active model from recognition first and disposes those two public
276
- resources separately and exactly once; it does not call the incomplete top-level disposer for this
277
- official shape. A custom injected runtime without those fields can retain the legacy single
278
- top-level `dispose()` contract. A runtime that exposes only part of the official shape is rejected
279
- with `TM-COMPOSITION-009`, after every safely identifiable resource has been attempted.
280
-
281
- Model, weights, and metadata `File` objects exist only for the pending `loadFromFiles()` call and
282
- are not stored in the named registry. `releasePoseModel()` and `releaseAll()` invalidate and wait for
283
- matching pending registrations, so their promises do not resolve before a late loaded model has
284
- been disposed. Switching to an already prepared model after stopping recognition keeps the camera
285
- stream alive; releasing the old, no-longer-active model does not request camera permission again.
286
- Releasing the current model with no prepared successor stops recognition but keeps the camera
287
- stream alive. `stopCamera()` and `releaseAll()` stop the stream explicitly.
288
-
289
- The accumulated-pose API chooses one candidate for an async-input consumer. It is deliberately
290
- separate from an Actor action that waits for multiple pose steps in sequence: a sequence consumer
291
- should read `confidenceOf(name)` and own its per-step progress instead of using or resetting this
292
- candidate-selection state.
293
-
294
- Accumulated scores use elapsed wall-clock seconds, not recognition counts:
295
-
296
- ```text
297
- nextScore = previousScore * decayPerSecond^elapsedSeconds
298
- + confidence * accumulationPerSecond * elapsedSeconds
299
- ```
300
-
301
- `accumulationPerSecond` is a finite number greater than or equal to zero. `decayPerSecond` is the
302
- finite fraction retained per second from zero through one; a changed decay takes effect in the next
303
- recognition session. `scoreThreshold` is a finite number greater than or equal to zero. An event is
304
- published only when the selected pose name changes, including one transition to an empty name on
305
- reset or stop; score-only changes do not publish another event. The standalone extension enables
306
- temporal scoring and keeps only the event feature flag off by default.
307
-
308
- ## Upgrading from 2.x
309
-
310
- The block surface, the composition API, and the offline PoseNet bundle API are unchanged. A project
311
- built on 2.x keeps working after swapping the extension file. New blocks appear that 2.x did not
312
- have: three for compute backends, and six for accumulated pose scoring.
313
-
314
- What changed is the runtime this extension bundles.
315
-
316
- | | 2.x | 3.0.0 |
317
- |---|---|---|
318
- | TensorFlow.js | 1.3.1 | 4.22.0 |
319
- | Teachable Machine Pose | 0.8.3 | 0.8.6 |
320
- | TensorFlow.js Speech Commands | 0.4.0 | 0.5.4 |
321
- | Compute backends | WebGL only | WebGPU, WebGL, WASM, CPU, negotiated at load |
322
-
323
- That runtime is published as `globalThis.tf`, so a host application reading it directly moves from
324
- the TensorFlow.js 1.x API to 4.x — the reason this is a major release. Models exported by Teachable
325
- Machine are not affected: pose, image, and audio models exported for the 1.x-era libraries load and
326
- run unchanged on 4.22.0.
81
+ ### Names
327
82
 
328
- Two behavioral notes for existing projects:
83
+ The Scratch extension ID is `kubohiroyatm`, from the 2.x release line onward, so that it does not
84
+ collide with other short IDs. The package, repository, Pages URL, and browser build all use
85
+ `turbowarp-teachable-machine`.
329
86
 
330
- - With no `set compute mode to` block, recognition now negotiates a backend instead of always using
331
- WebGL. `active compute backend` reports which one won, and `last error` explains any substitution.
332
- - The first `start camera` or `load model` downloads a larger runtime than 2.x did (1.24 MB versus
333
- 1.04 MB), so plan the startup sequence accordingly. See [Compute backends](#compute-backends).
87
+ > The package, repository, and Pages URL were previously named `turbowarp-tm`
88
+ > (`@kubohiroya/turbowarp-tm`, last published as 3.2.0). Replace that name with
89
+ > `turbowarp-teachable-machine` in CDN URLs, `pnpm add`, and `import` specifiers.
334
90
 
335
91
  ## Quick start
336
92
 
337
93
  1. Train classes such as `jump`, `card`, or `clap` in Teachable Machine.
338
94
  2. Export the model, upload it, and copy the model folder URL.
339
95
  3. Set that URL with `set model URL to [URL]`.
340
- 4. Run `start recognition`, allow camera or microphone access, and use a result block in your script.
96
+ 4. Run `start recognition`, allow camera or microphone access, and read a result block.
341
97
 
342
98
  ```text
343
99
  when green flag clicked
@@ -351,208 +107,361 @@ forever
351
107
  end
352
108
  ```
353
109
 
354
- `start recognition` starts the required input and loads the configured model when necessary. Pose and
355
- image modes use the camera; audio mode uses the microphone. A separate `start camera` or `load model`
356
- step is only needed when a camera project wants to control startup phases individually.
110
+ `start recognition` starts the input it needs and loads the configured model when necessary. Pose
111
+ and image modes use the camera; audio mode uses the microphone. A separate `start camera` or
112
+ `load model` step is only needed when a project wants to control the startup phases itself — for
113
+ example to show the preview before the model is ready, or to read `model load time (ms)`.
357
114
 
358
115
  ## Compute backends
359
116
 
360
- Recognition runs on a TensorFlow.js compute backend. `set compute mode to [MODE]` chooses it and
117
+ Recognition runs on a TensorFlow.js compute backend. `set compute mode to [MODE]` chooses one and
361
118
  `active compute backend` reports the one actually in use.
362
119
 
363
120
  | Mode | What it does |
364
121
  |---|---|
365
- | `auto` (default) | Tries WebGPU, WebGL, WASM, and CPU in that order and keeps the first one that activates and computes a verification convolution correctly |
366
- | `webgpu` | Modern GPU path with the lowest per-operation overhead |
122
+ | `auto` (default) | Tries WebGPU, WebGL, WASM, and CPU in that order, keeping the first that activates and computes a verification convolution correctly |
123
+ | `webgpu` | Modern GPU path, lowest per-operation overhead |
367
124
  | `webgl` | The long-standing GPU path, available in every browser this extension supports |
368
125
  | `wasm` | SIMD WebAssembly kernels; steady on machines with weak or unreliable GPU drivers |
369
- | `cpu` | Plain JavaScript kernels; correct everywhere and slowest |
126
+ | `cpu` | Plain JavaScript kernels; correct everywhere, and slowest |
370
127
 
371
- A named mode is still only a preference. If the browser cannot provide it, the same order takes
372
- over, `active compute backend` shows what won, and `last error` explains the substitution. That is
373
- why a project should read `active compute backend` rather than assume the requested mode.
374
-
375
- ```text
376
- when green flag clicked
377
- set compute mode to [webgpu]
378
- set model URL to [https://teachablemachine.withgoogle.com/models/.../]
379
- start recognition
380
- ```
381
-
382
- The reviewed browser runtime carries the WebGL and CPU kernels. The WebGPU and WebAssembly backends
383
- are separate bundles next to `dist/runtime.js`, fetched only when a project reaches for them, and
384
- the WebAssembly binaries come from `dist/wasm/` in this same package. Everything resolves against
385
- the one TensorFlow.js instance the runtime published, so no page ever runs two TensorFlow.js cores.
128
+ A named mode is a preference, not a guarantee. When the browser cannot provide it the same order
129
+ takes over, `active compute backend` shows what won, and `last error` explains the substitution.
130
+ Read `active compute backend` rather than assuming the requested mode.
386
131
 
387
132
  TensorFlow.js binds every tensor to the backend that was active when the tensor was created, so the
388
133
  mode has to be settled before a model loads. Changing the mode therefore releases a model that was
389
134
  loaded from a URL, and the next `load model` or `start recognition` rebuilds it on the new backend.
390
135
  Changing the mode during recognition, or while a host-prepared model is active, is refused instead.
391
136
 
392
- Host applications reach the same negotiation through the composition API:
393
-
394
- ```js
395
- const composition = createTMComposition({
396
- runtime: tmPose,
397
- compute: tmCompute, // published by dist/runtime.js
398
- computeMode: 'auto',
399
- });
400
-
401
- const selection = await composition.selectComputeBackend('webgpu');
402
- // selection.backend === 'webgpu' | 'webgl' | 'wasm' | 'cpu'
403
- // selection.fallback === true when the requested backend was unusable
404
- ```
137
+ The reviewed browser runtime carries the WebGL and CPU kernels. WebGPU and WebAssembly live in
138
+ separate bundles next to `dist/runtime.js`, fetched only when a project reaches for them, with the
139
+ WebAssembly binaries coming from `dist/wasm/` in this same package. Everything resolves against the
140
+ one TensorFlow.js instance the runtime published, so no page ever runs two TensorFlow.js cores.
405
141
 
406
- `selectComputeBackend()` is rejected with `TM-COMPOSITION-018` while pose models are registered,
407
- because their tensors live in the memory of the backend that loaded them. Release them first.
408
-
409
- ## Reading recognition results
142
+ ## Reading results
410
143
 
411
144
  | Block | Result |
412
145
  |---|---|
413
- | `current label` | Class with the highest probability in the latest frame |
414
- | `confidence` | Current label probability, rounded to two decimal places |
415
- | `confidence of [NAME]` | Probability of one named class |
146
+ | `current label` | The class with the highest probability in the latest frame |
147
+ | `confidence` | That class's probability, rounded to two decimal places |
148
+ | `confidence of [NAME]` | The probability of one named class |
416
149
  | `label is [NAME]?` | Whether the named class has at least `0.75` confidence |
417
- | `label is [NAME] with confidence at least [THRESHOLD]?` | Same test with a custom `0`–`1` threshold |
150
+ | `label is [NAME] with confidence at least [THRESHOLD]?` | The same test with a custom `0`–`1` threshold |
151
+
152
+ Live confidence reacts quickly and fluctuates near a decision boundary. Better training data,
153
+ lighting or audio quality, framing, and a threshold chosen for the project usually matter more than
154
+ anything this extension can do for you. When a decision should survive a single bad frame, use
155
+ [accumulated pose scoring](#accumulated-pose-scoring) instead of raising the threshold.
418
156
 
419
- Live confidence reacts quickly and can fluctuate near a decision boundary. Better training data,
420
- lighting or audio quality, camera or microphone framing, and a suitable threshold usually improve
421
- the result.
157
+ ## Camera, preview, and what the model sees
422
158
 
423
- ## Camera selection, preview, and stopping
159
+ ### Choosing a camera
424
160
 
425
- Use `refresh camera list` to detect video inputs, then choose `default camera`, `front camera`,
426
- `back camera`, or a detected device with `set camera to [CAMERA]`. Camera labels may be unavailable
427
- until the browser grants camera permission. Device IDs are browser- and machine-specific, so use
428
- the portable front/back choices when a project moves between devices. Changing the selection while
429
- the camera is running restarts only the camera stream, preserving the loaded model, recognition
430
- state, and preview settings. If switching fails, the extension attempts to restore the previous camera and
431
- records the error in `last error`.
161
+ `refresh camera list` detects video inputs. `set camera to [CAMERA]` then selects `default camera`,
162
+ `front camera`, `back camera`, or one detected device. Labels may stay empty until the browser
163
+ grants camera permission, and device IDs are browser- and machine-specific, so prefer the portable
164
+ front/back choices when a project moves between devices.
432
165
 
433
- `camera count` reports the latest refreshed count. `camera device ID` and `camera device name`
434
- report the active input when the browser provides those values.
166
+ Changing the selection while the camera runs restarts only the camera stream and preserves the
167
+ loaded model, the recognition state, and every preview setting. A failed switch restores the
168
+ previous camera and records the reason in `last error`. `camera count`, `camera device ID`, and
169
+ `camera device name` report the latest refreshed count and the active input.
435
170
 
436
- The preview is a camera canvas placed over the TurboWarp stage. It can be shown, hidden, moved to
437
- six stage positions, made transparent, or expanded to fill the stage. Hiding the preview does not
438
- stop recognition. The preview is mirrored by default. Use `set camera preview to [MIRRORING]` to
439
- switch between `mirrored` and `unmirrored`, including while the camera is running. This display-only
440
- setting does not change the frames used for recognition. The `camera preview mirroring` reporter
441
- returns the current setting.
171
+ ### The preview and the model input
442
172
 
443
- - `stop recognition` clears current results, stops audio listening, but leaves the camera available;
444
- - `stop camera` also stops recognition, releases the camera, and removes the preview and SVG.
173
+ The preview is the camera's own video element placed over the stage, at the camera's resolution and
174
+ aspect ratio. It can be shown, hidden, moved to six positions, made transparent, or expanded to
175
+ fill the stage, and it is mirrored by default — `set camera preview to [MIRRORING]` switches that,
176
+ including while the camera runs. Mirroring is display-only.
177
+
178
+ The model is fed a square centre crop of the frame, mirrored, as a tensor. `fromPixels` uploads the
179
+ video to a texture on the WebGL and WebGPU backends and the crop and mirroring are tensor ops, so
180
+ no Canvas2D stage sits between the camera and the model. The square also makes PoseNet's padding
181
+ zero.
182
+
183
+ - `stop recognition` clears current results and stops audio listening, but leaves the camera
184
+ available.
185
+ - `stop camera` also stops recognition, releases the camera, and removes the preview and the SVG
186
+ overlay.
187
+
188
+ Hiding the preview does not stop recognition.
445
189
 
446
190
  ### Sharing one camera with other extensions
447
191
 
448
- When [TurboWarp-Camera-Source](https://github.com/kubohiroya/turbowarp-camera-source) is loaded, this
449
- extension takes a lease on the camera it shares under the `pose` role instead of opening a device of
450
- its own. One physical camera then serves recognition and anything else reading frames — a QR reader,
451
- a preview, another classifier — rather than each extension competing for the device.
192
+ When [TurboWarp-Camera-Source](https://github.com/kubohiroya/turbowarp-camera-source) is loaded,
193
+ this extension takes a lease on the camera it shares under the `pose` role instead of opening a
194
+ device of its own. One physical camera then serves recognition and anything else reading frames — a
195
+ QR reader, a preview, another classifier — rather than each extension competing for the device.
452
196
 
453
197
  Camera Source owns the stream: it stops the camera when the last lease is released, and `stop
454
- camera` here releases this extension's lease without stopping tracks that other consumers may still
455
- be reading. The frames recognition sees are unchanged either way.
456
-
457
- Camera Source is not required. It is a separate extension, and this one is also loaded directly by
458
- URL, so when it is absent the extension opens and stops its own camera exactly as before. Nothing
459
- in a project needs to change for either configuration.
460
-
461
- 3.5.0 changes what the stage shows and what the model is fed. The preview is the camera's own
462
- video rather than a 320×240 readback canvas scaled to fit, so it carries the camera's resolution
463
- and aspect ratio. The model is fed the whole square centre crop as a tensor; the canvas used to
464
- hold only its top 240 rows, so a quarter of the subject's height never reached it. Blocks,
465
- opcodes and the composition API are unchanged, and a project needs no edit — but a pose model
466
- will see a subject it was not seeing before, which is the point.
467
-
468
- 3.5.1 fixes what image models are fed. Image mode handed `predict` the camera's video element
469
- without asking for mirroring, so the model saw unmirrored pixels while Teachable Machine had
470
- trained it on a preview mirrored in CSS. The square it sees was already the frame's centre crop;
471
- only the mirroring was missing, and pose and image mode now feed the same square the same way
472
- round. Pose recognition is unchanged.
473
-
474
- Leasing is fixed in 3.4.1. In 3.4.0 a leased start threw `Webcam canvas does not provide a 2D
475
- context` and recognition never began, because the upstream camera helper builds its readback canvas
476
- in the same branch that opens the device — the branch a supplied element skips. Use 3.4.1 wherever
477
- Camera Source is loaded; 3.4.0 is unaffected only when it is not.
478
-
479
- ## Camera preview and what the model sees
480
-
481
- The preview on the stage is the camera's own video element, at the camera's resolution and aspect
482
- ratio. It used to be a 320×240 readback canvas scaled to fit, so a 1080p camera looked like a
483
- 320×240 one and a 16:9 camera was shown in a 4:3 window.
484
-
485
- The model is fed a square centre crop of the frame, mirrored, as a tensor. `fromPixels` uploads
486
- the video to a texture on the WebGL and WebGPU backends and the crop and mirroring are tensor ops,
487
- so no Canvas2D stage sits between the camera and the model. The square also makes PoseNet's
488
- padding zero, where the old 4:3 canvas was padded back to square and left the subject about 25%
489
- smaller than the frames the model was trained on. The crop is the whole square now: the readback
490
- canvas held only its top 240 rows and discarded the rest.
491
-
492
- The pose overlay spans the preview and its keypoints are in the camera frame's own coordinates.
493
-
494
- The extension performs pose, image, or audio classification in the browser and does not upload
495
- camera frames or microphone audio. It does fetch its runtime libraries and the published model. Stop
496
- recognition and the camera when the project no longer needs them.
497
-
498
- ## Optional SVG pose overlay
499
-
500
- The `poseOverlay` feature flag is **on by default** for the standalone TurboWarp extension, adding
501
- blocks for overlay visibility, per-joint circle color/opacity/radius, shared bone
502
- color/opacity/width, minimum keypoint confidence, and confidence scaling. The overlay's SVG coordinate system is the
503
- camera frame's own, so it follows a camera of any resolution or aspect ratio, and it honours all
504
- six preview positions and preview mirroring. The Composition API enables this isolated overlay layer directly.
505
-
506
- Confidence scaling can be switched independently for joint opacity, joint radius, bone opacity,
507
- and bone width. Joint properties use the corresponding keypoint confidence; bone properties use
508
- the smaller confidence of the connected endpoints. This makes every enabled property vary from
509
- zero through its configured value without changing the recognition classifier input.
198
+ camera` here releases only this extension's lease. Camera Source is not required; without it the
199
+ extension opens and stops its own camera exactly as before, and a project needs no edit either way.
200
+
201
+ ## Pose overlay
202
+
203
+ The `poseOverlay` feature flag is **on by default**, so the overlay blocks are in the palette:
204
+ visibility, per-joint circle colour/opacity/radius for the 17 PoseNet joints, shared bone
205
+ colour/opacity/width for the 12 standard connections, a minimum keypoint confidence, and confidence
206
+ scaling.
207
+
208
+ The overlay's SVG coordinate system is the camera frame's own, so it follows a camera of any
209
+ resolution or aspect ratio and honours all six preview positions and preview mirroring.
210
+
211
+ Confidence scaling switches independently for joint opacity, joint radius, bone opacity, and bone
212
+ width. A joint property is multiplied by that keypoint's confidence; a bone property by the lower
213
+ confidence of its two endpoints. A bone is hidden when either endpoint falls below the minimum
214
+ confidence. None of this changes the classifier input.
215
+
216
+ Hiding the overlay does not stop recognition. Hiding the preview also hides the overlay; stopping
217
+ recognition clears every mark; stopping the camera removes the SVG element.
510
218
 
511
219
  ## Accumulated pose scoring
512
220
 
513
- Live confidence flickers frame to frame near a decision boundary. The accumulated score blocks are
514
- the built-in answer: they combine evidence over time for poses that should be held steadily, so a
515
- brief misrecognition cannot flip a decision on its own. The `temporalPoseScoring` feature flag is
516
- **on by default**, so these blocks are in the palette.
221
+ Live confidence flickers frame to frame near a decision boundary. Accumulated scores combine
222
+ evidence over time, so a brief misrecognition cannot flip a decision on its own. The
223
+ `temporalPoseScoring` feature flag is **on by default**.
517
224
 
518
225
  ```text
519
- previous × decay^elapsedSeconds + probability × accumulation × elapsedSeconds
226
+ nextScore = previousScore × decayPerSecond^elapsedSeconds
227
+ + confidence × accumulationPerSecond × elapsedSeconds
520
228
  ```
521
229
 
522
- The accumulation coefficient is a per-second rate. The decay coefficient is the fraction retained
523
- after one second and is clamped to `0`–`1`; changes to decay take effect the next time recognition
524
- starts. Accumulation and decay both pause while the document is hidden.
230
+ Scores use elapsed wall-clock seconds, not recognition counts. The accumulation coefficient is a
231
+ per-second rate of zero or more. The decay coefficient is the fraction retained after one second,
232
+ clamped to `0`–`1`; a changed decay takes effect the next time recognition starts. Both pause while
233
+ the document is hidden.
525
234
 
526
235
  `accumulated pose` returns the highest positive accumulated pose only when it meets the configured
527
- threshold, or an empty string otherwise. The accumulated score reporters continue to return their
528
- unrounded values below that threshold. Resetting or stopping recognition clears all accumulated
529
- scores.
236
+ threshold, and an empty string otherwise; the score reporters keep returning their unrounded values
237
+ below it. Resetting or stopping recognition clears every accumulated score.
530
238
 
531
- ### Accumulated pose change events
239
+ This chooses one candidate. A consumer that waits for several pose steps in sequence should read
240
+ `confidence of [NAME]` and own its own per-step progress rather than resetting this shared
241
+ candidate-selection state.
242
+
243
+ ### Change events
532
244
 
533
245
  The `accumulatedPoseEvents` feature flag is **off by default** and requires `temporalPoseScoring`.
534
- When both are enabled, other unsandboxed extensions can check
246
+ With both enabled, other unsandboxed extensions can check
535
247
  `runtime.ext_kubohiroyatm.supportsAccumulatedPoseEvents()` and subscribe to
536
248
  `TM_ACCUMULATED_POSE_CHANGED` on the TurboWarp runtime.
537
249
 
538
- Each version 2 event includes `poseName`, `previousPoseName`, `score`, `reason` (`recognition`,
539
- `reset`, or `stop`), and a monotonic `timestamp`. Score-only updates do not emit an event.
250
+ A version 2 event carries `poseName`, `previousPoseName`, `score`, `reason` (`recognition`, `reset`,
251
+ or `stop`), and a monotonic `timestamp`. An event is published only when the selected name changes,
252
+ including the transition to an empty name on reset or stop; score-only updates publish nothing.
253
+
254
+ ## Host application APIs
255
+
256
+ These two entry points are for applications that embed the runtime. A project built in the TurboWarp
257
+ editor needs neither.
258
+
259
+ ### Composition API
260
+
261
+ `@kubohiroya/turbowarp-teachable-machine/composition` gives a composite runtime the recognition
262
+ engine without registering the standalone extension or adding blocks. The caller supplies an
263
+ already-bundled Teachable Machine Pose runtime and validated model bytes, so this path downloads no
264
+ runtime script and no model file.
265
+
266
+ ```js
267
+ import {createTMComposition} from '@kubohiroya/turbowarp-teachable-machine/composition';
268
+
269
+ const pose = createTMComposition({
270
+ runtime: bundledTMRuntime,
271
+ compute: tmCompute, // published by dist/runtime.js
272
+ modelInitializationPolicy: 'latest-needed',
273
+ });
274
+
275
+ const cameras = await pose.listCameraDevices();
276
+ if (cameras[0]) await pose.selectCamera({deviceId: cameras[0].deviceId});
277
+
278
+ await pose.registerPoseModel({
279
+ name: 'RescuePose',
280
+ files: [
281
+ {path: 'model.json', bytes: modelBytes},
282
+ {path: 'weights.bin', bytes: weightsBytes},
283
+ {path: 'metadata.json', bytes: metadataBytes},
284
+ ],
285
+ });
286
+ pose.activatePoseModel('RescuePose');
287
+
288
+ pose.setPreviewPosition('full-stage');
289
+ pose.setPoseBoneStyle({color: '#00e5ff', opacity: 0.9, width: 3});
290
+ pose.configureAccumulatedPose({accumulationPerSecond: 1, decayPerSecond: 0.9, scoreThreshold: 0});
291
+ const unsubscribe = pose.subscribeAccumulatedPose((event) => {
292
+ // event.poseName is the currently selected candidate, or an empty string.
293
+ });
294
+
295
+ await pose.startRecognition();
296
+ ```
297
+
298
+ **Lifecycle.** Each composition owns its model registry, active model, camera, and recognition
299
+ state. `releasePoseModel(name)` stops recognition when that model is active and disposes only it;
300
+ `releaseAll()` releases the whole instance, cancels queued and active registration work, removes
301
+ accumulated-pose listeners, and waits for safe cleanup. `subscribeAccumulatedPose()` returns an
302
+ idempotent unsubscribe. Camera capture is an independent lifecycle: only `stopCamera()` and
303
+ `releaseAll()` stop the stream. `File` objects live only for the pending `loadFromFiles()` call and
304
+ are never stored in the registry.
305
+
306
+ **Model initialization policy.** The default `legacy` policy keeps every registration independent.
307
+ `latest-needed` suits a host with one current label-model demand, such as a story that can skip
308
+ scenes: at most one heavy load is active and one newest request pending, and a newer request
309
+ cancels the active one cooperatively and replaces an older request that has not started. For A
310
+ loading, B pending, then C requested, B never reaches the TensorFlow loader. Repeated registrations
311
+ with the same name, byte-identical files, and the same `AbortSignal` share one pending operation;
312
+ different signals stay separate, so cancelling one caller cannot cancel another's work.
313
+
314
+ Pass an `AbortSignal` to `registerPoseModel()` when the host no longer needs that demand. A queued
315
+ request rejects with `AbortError` without invoking the runtime. TensorFlow.js 4.22.0 and Web Crypto
316
+ cannot physically cancel every operation already running, so a running request becomes stale at
317
+ once, starts no later phase, and disposes every late classifier and PoseNet result exactly once
318
+ before its promise settles. A cancelled model is never published in the registry; shared PoseNet
319
+ verification may finish and stay cached. Cancellation never stops the camera.
320
+
321
+ Phases run sequentially by default, with cancellation checked after TensorFlow readiness,
322
+ classifier loading, metadata parsing, and PoseNet loading. Set `parallelModelInitialization: true`
323
+ only when startup latency matters more than those boundaries; it starts the three in parallel, and
324
+ completed stale resources are still disposed before cancellation settles.
325
+
326
+ The composition does not infer scene reachability. A host should retain the signal when the model is
327
+ needed by an imminent action, abort it when a skipped path has no nearby demand, and submit the
328
+ replacement model when a skip changes that demand.
329
+
330
+ **Cameras.** `listCameraDevices()` enumerates afresh on every call and returns a deeply frozen copy
331
+ holding only unique, non-empty video device IDs in enumeration order; labels stay empty until
332
+ permission makes them available. Enumeration failures use `TM-COMPOSITION-012` and never expose
333
+ `MediaDeviceInfo`, `MediaStreamTrack`, or a stream. Select `default`, `front`, or `back` as a
334
+ preference, or wrap a session-only browser ID as `{deviceId}` — the object form is deliberately
335
+ distinct, so a device whose ID is literally `default` is not mistaken for the preference. Selections
336
+ run in call order; a failed switch restores the preceding camera; invalid selections use
337
+ `TM-COMPOSITION-011`. Device IDs are never persisted, and all four camera methods fail closed after
338
+ release.
339
+
340
+ **Preview and overlay.** `setPreviewOpacity()` takes `0`–`1`; `setPreviewPosition()` takes
341
+ `top-left`, `top-right`, `bottom-left`, `bottom-right`, `center`, or `full-stage`;
342
+ `setPreviewMirroring()` takes `mirrored` (the default) or `unmirrored` and may be called before,
343
+ during, or after camera startup. `showPreview()` and `hidePreview()` change visibility only. The
344
+ overlay is enabled directly, with `showPoseOverlay()`, `hidePoseOverlay()`,
345
+ `isPoseOverlayVisible()`, `setPoseJointStyle()`, `setPoseBoneStyle()`,
346
+ `setPoseOverlayMinimumConfidence()`, and `setPoseOverlayConfidenceScaling()`.
347
+
348
+ **Compute.** `selectComputeBackend('webgpu')` resolves to `{backend, fallback}`, where `fallback`
349
+ is `true` when the requested backend was unusable. It is rejected with `TM-COMPOSITION-018` while
350
+ pose models are registered, because their tensors live in the memory of the backend that loaded
351
+ them — release them first.
352
+
353
+ **Disposal shape.** Teachable Machine Pose 0.8.6 exposes the classifier as `CustomPoseNet.model`
354
+ and PoseNet as `CustomPoseNet.posenetModel`, while its top-level `dispose()` releases PoseNet only.
355
+ The composition therefore disconnects an active model from recognition first and disposes those two
356
+ public resources separately and exactly once, rather than calling the incomplete top-level
357
+ disposer. A custom injected runtime without those fields keeps the legacy single-`dispose()`
358
+ contract; a runtime exposing only part of the official shape is rejected with `TM-COMPOSITION-009`
359
+ after every safely identifiable resource has been attempted.
360
+
361
+ **Canvas note.** The extension owns the camera canvas used by Teachable Machine and TensorFlow.js
362
+ and requests a normal Canvas2D context for it. Physical-camera measurements of its 320×240,
363
+ one-draw/one-read path showed no repeatable benefit from `willReadFrequently`, and that hint can
364
+ make video drawing more expensive, so Chrome may emit its Canvas2D readback warning during CPU
365
+ inference. Reconsider the hint only with a reproducible benchmark of this exact path.
366
+
367
+ ### Offline PoseNet bundle API
368
+
369
+ `@kubohiroya/turbowarp-teachable-machine/posenet` owns the fixed PoseNet MobileNetV1 0.75 /
370
+ stride 16 manifest, the package asset specifiers, SHA-256 verification, a bounded Base64 project
371
+ descriptor, and the Teachable Machine Pose fetch adapter. A host application only decides where
372
+ that explicit descriptor is stored. The model stays one JSON file and two binary shards; it is
373
+ never disguised as an image, sound, or costume.
374
+
375
+ ```js
376
+ import {readFile} from 'node:fs/promises';
377
+ import {
378
+ createBundledTMRuntime,
379
+ createPoseNetProjectBundleFromLoader,
380
+ } from '@kubohiroya/turbowarp-teachable-machine/posenet';
381
+
382
+ const projectBundle = await createPoseNetProjectBundleFromLoader((file) =>
383
+ readFile(new URL(import.meta.resolve(file.packageSpecifier))),
384
+ );
385
+
386
+ // A browser host can store projectBundle in its own project-data channel.
387
+ // Decoding and hash verification start on the first loadFromFiles() call.
388
+ const offlineRuntime = createBundledTMRuntime({runtime: tmPose, projectBundle});
389
+ ```
390
+
391
+ The model supply is exactly 5,082,500 bytes. Its three SHA-256 operations run independently, and the
392
+ first shared verification overlaps the classifier load. The intercepted PoseNet fetch exposes no
393
+ bytes until verification succeeds, and later requests reuse the same verified supply. Tampering,
394
+ missing or wrongly sized files, and unexpected PoseNet network requests all fail closed with
395
+ `TM-POSENET-*` error codes.
540
396
 
541
397
  ## Troubleshooting
542
398
 
543
- Read `last error` first when setup fails. Common causes are denied camera permission, a model editor
544
- URL instead of the published model folder URL, blocked network requests, or loading the extension in
545
- the sandbox. A sandboxed load shows nothing in the editor: the custom extension dialog simply keeps
546
- waiting, and only the browser console carries `TM must run without the extension sandbox.` Load the
547
- downloaded file from the dialog's **Files** tab, never from its **URL** tab.
399
+ Read `last error` first. The usual causes are denied camera permission, a model *editor* URL instead
400
+ of the published model folder URL, blocked network requests, and a sandboxed load.
401
+
402
+ A sandboxed load shows nothing in the editor: the custom extension dialog simply keeps waiting, and
403
+ only the browser console carries `TM must run without the extension sandbox.` Load the downloaded
404
+ file from the dialog's **Files** tab, never from its **URL** tab.
548
405
 
549
406
  `start camera` also needs the page to have been interacted with at least once. Browsers refuse
550
- `video.play()` on a page with no user activation, which surfaces as `NotAllowedError: play() failed
551
- because the user didn't interact with the document first.` Clicking the green flag counts, so an
552
- ordinary project is fine; a packaged project that starts recognition on its own has to wait for a
553
- button press before `start camera`. See the illustrated guide's [troubleshooting section](https://kubohiroya.github.io/turbowarp-teachable-machine/#troubleshooting)
554
- or [Japanese troubleshooting section](https://kubohiroya.github.io/turbowarp-teachable-machine/ja/#troubleshooting)
555
- for step-by-step checks.
407
+ `video.play()` without user activation, which surfaces as `NotAllowedError: play() failed because
408
+ the user didn't interact with the document first.` Clicking the green flag counts, so an ordinary
409
+ project is fine; a packaged project that starts recognition on its own has to wait for a button
410
+ press first.
411
+
412
+ The guide has step-by-step checks:
413
+ [English](https://kubohiroya.github.io/turbowarp-teachable-machine/#troubleshooting) ·
414
+ [日本語](https://kubohiroya.github.io/turbowarp-teachable-machine/ja/#troubleshooting).
415
+
416
+ ## Version notes
417
+
418
+ ### Upgrading from 2.x
419
+
420
+ The block surface, the composition API, and the offline PoseNet bundle API are unchanged; a project
421
+ built on 2.x keeps working after swapping the extension file. Nine blocks are new — three for
422
+ compute backends, six for accumulated pose scoring. What changed is the bundled runtime:
423
+
424
+ | | 2.x | 3.0.0 |
425
+ |---|---|---|
426
+ | TensorFlow.js | 1.3.1 | 4.22.0 |
427
+ | Teachable Machine Pose | 0.8.3 | 0.8.6 |
428
+ | TensorFlow.js Speech Commands | 0.4.0 | 0.5.4 |
429
+ | Compute backends | WebGL only | WebGPU, WebGL, WASM, CPU, negotiated at load |
430
+
431
+ That runtime is published as `globalThis.tf`, so a host application reading it directly moves from
432
+ the TensorFlow.js 1.x API to 4.x — the reason this is a major release. Models exported by Teachable
433
+ Machine are unaffected: pose, image, and audio models exported for the 1.x-era libraries load and
434
+ run unchanged on 4.22.0.
435
+
436
+ Two behavioural notes for existing projects:
437
+
438
+ - Without a `set compute mode to` block, recognition now negotiates a backend instead of always
439
+ using WebGL. `active compute backend` reports which one won.
440
+ - The first `start camera` or `load model` downloads a larger runtime than 2.x did (1.24 MB against
441
+ 1.04 MB), so plan the startup sequence accordingly.
442
+
443
+ ### 3.4.1 — camera leasing
444
+
445
+ In 3.4.0 a leased start threw `Webcam canvas does not provide a 2D context` and recognition never
446
+ began, because the upstream camera helper builds its readback canvas in the same branch that opens
447
+ the device — the branch a supplied element skips. Use 3.4.1 or later wherever Camera Source is
448
+ loaded; 3.4.0 is unaffected only when it is not.
449
+
450
+ ### 3.5.0 — preview and crop
451
+
452
+ The preview became the camera's own video rather than a 320×240 readback canvas scaled to fit, so it
453
+ carries the camera's resolution and aspect ratio. The model is fed the whole square centre crop as a
454
+ tensor; the canvas used to hold only its top 240 rows, so a quarter of the subject's height never
455
+ reached the model, and the 4:3 canvas was padded back to square, leaving the subject about 25%
456
+ smaller than the frames the model was trained on. Blocks, opcodes, and the composition API are
457
+ unchanged — but a pose model now sees a subject it was not seeing before, which is the point.
458
+
459
+ ### 3.5.1 — image-mode mirroring
460
+
461
+ Image mode handed `predict` the camera's video element without asking for mirroring, so the model
462
+ saw unmirrored pixels while Teachable Machine had trained it on a CSS-mirrored preview. The square
463
+ it sees was already the frame's centre crop; only the mirroring was missing. Pose and image mode now
464
+ feed the same square the same way round. Pose recognition is unchanged.
556
465
 
557
466
  ## Blocks
558
467
 
@@ -1038,41 +947,44 @@ pnpm install --frozen-lockfile
1038
947
  pnpm check
1039
948
  ```
1040
949
 
1041
- The check runs type checking, tests, the production build, generated-documentation validation,
950
+ `pnpm check` runs type checking, tests, the production build, generated-documentation validation,
1042
951
  Pages link validation, distribution reproducibility, and an npm package dry run. The build produces
1043
- `dist/tm.js`, `dist/composition.js`, `dist/runtime.js`, `dist/posenet.js`, and the three raw
1044
- PoseNet model assets under `dist/posenet/`.
952
+ `dist/tm.js`, `dist/composition.js`, `dist/runtime.js`, `dist/posenet.js`, and the three raw PoseNet
953
+ model assets under `dist/posenet/`.
1045
954
 
1046
- The version in `package.json` is the release source of truth. The runtime reporter appends
1047
- `-typescript` to that version, and the release consistency check keeps the generated bundles,
1048
- exact-version README examples, Pages badges, and release tag aligned with it.
955
+ The block reference above is generated from `src/block-definitions.json` by `pnpm docs`; edit the
956
+ definitions, not the generated section. The version in `package.json` is the release source of
957
+ truth: the runtime reporter appends `-typescript` to it, and the release consistency check keeps the
958
+ generated bundles, the exact-version README examples, the Pages badges, and the release tag aligned
959
+ with it.
1049
960
 
1050
961
  ### npm publishing
1051
962
 
1052
- npm publication uses GitHub Actions trusted publishing rather than a long-lived write token. The
1053
- npm package settings must trust `kubohiroya/turbowarp-teachable-machine`, workflow file
1054
- `publish-npm.yml`, for `npm publish`. After the annotated release tag and GitHub Release exist,
1055
- dispatch **Publish npm package** with that tag. The workflow checks out the tag, verifies that it
1056
- exactly matches `package.json`, runs the complete check, and publishes through a short-lived OIDC
1057
- credential. Roll back the automation by removing the trusted publisher in npm package settings;
1058
- do not add a write token to the repository.
963
+ npm publication uses GitHub Actions trusted publishing rather than a long-lived write token. The npm
964
+ package settings must trust `kubohiroya/turbowarp-teachable-machine`, workflow file
965
+ `publish-npm.yml`, for `npm publish`. Once the annotated release tag and the GitHub Release exist,
966
+ dispatch **Publish npm package** with that tag: the workflow checks out the tag, verifies that it
967
+ matches `package.json` exactly, runs the complete check, and publishes through a short-lived OIDC
968
+ credential. Roll the automation back by removing the trusted publisher in npm package settings; do
969
+ not add a write token to the repository.
1059
970
 
1060
971
  ## External libraries
1061
972
 
1062
973
  The version-pinned `dist/runtime.js` artifact bundles TensorFlow.js 4.22.0, Teachable Machine Pose
1063
- 0.8.6, Teachable Machine Image 0.8.5, TensorFlow.js Speech Commands 0.5.4, and its PoseNet 2.2.2
1064
- runtime in one reviewed module graph. The standalone extension loads that single artifact from
1065
- jsDelivr when no runtime is injected. The optional `dist/backend-wasm.js` and
1066
- `dist/backend-webgpu.js` bundles resolve TensorFlow.js against the instance that runtime already
1067
- published, so a page never ends up with two TensorFlow.js cores; the WebAssembly binaries under
1068
- `dist/wasm/` and the fixed offline PoseNet model data under `dist/posenet/` are distributed as their
1069
- original binary files.
974
+ 0.8.6, Teachable Machine Image 0.8.5, TensorFlow.js Speech Commands 0.5.4, and PoseNet 2.2.2 in one
975
+ reviewed module graph. The standalone extension loads that single artifact from jsDelivr when no
976
+ runtime is injected. The optional `dist/backend-wasm.js` and `dist/backend-webgpu.js` bundles
977
+ resolve TensorFlow.js against the instance that runtime already published, so a page never ends up
978
+ with two TensorFlow.js cores. The WebAssembly binaries under `dist/wasm/` and the fixed offline
979
+ PoseNet model data under `dist/posenet/` are distributed as their original binary files.
1070
980
 
1071
- ## License
981
+ Third-party licences are listed in [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).
1072
982
 
1073
- MPL-2.0
983
+ ## Fixed input editors
1074
984
 
985
+ The opt-in fixed input forms, and the rules that keep existing data intact, are described in
986
+ [the fixed input editor contract](docs/fixed-input-editors.ja.md) (Japanese).
1075
987
 
1076
- ## 固定入力の編集
988
+ ## License
1077
989
 
1078
- 既定OFFの固定入力フォームと既存データの保持規則は[固定入力エディターの契約](docs/fixed-input-editors.ja.md)を参照してください。
990
+ MPL-2.0