@kubohiroya/turbowarp-teachable-machine 3.5.6 → 3.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.ja.md +427 -0
- package/README.md +377 -465
- package/assets/block-icon.svg +1 -0
- package/dist/backend-wasm.js +1 -1
- package/dist/backend-webgpu.js +1 -1
- package/dist/composition.js +65 -10
- package/dist/posenet.js +6 -4
- package/dist/runtime.js +2 -2
- package/dist/tm.js +2916 -2861
- package/dist/types/extension.d.ts +28 -0
- package/docs/fixed-input-editors.ja.md +1 -1
- package/package.json +6 -4
package/README.md
CHANGED
|
@@ -1,22 +1,16 @@
|
|
|
1
1
|
# TurboWarp Teachable Machine
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
an input for TurboWarp projects. The extension turns each camera frame or microphone window into labels,
|
|
5
|
-
confidence scores, and Boolean conditions that Scratch-style scripts can use.
|
|
3
|
+
<img src="assets/block-icon.svg" alt="" width="48" height="48">
|
|
6
4
|
|
|
7
|
-
|
|
8
|
-
avoid short-ID collisions while the package, repository, Pages URL, and browser build use
|
|
9
|
-
`turbowarp-teachable-machine`.
|
|
5
|
+
English | [日本語](README.ja.md)
|
|
10
6
|
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
7
|
+
Use a published [Teachable Machine](https://teachablemachine.withgoogle.com/) pose, image, or audio
|
|
8
|
+
model as an input for TurboWarp projects. Every camera frame or microphone window becomes a label, a
|
|
9
|
+
confidence score, and Boolean conditions that Scratch-style scripts can act on.
|
|
14
10
|
|
|
15
|
-
**
|
|
16
|
-
**[日本語ガイド](https://kubohiroya.github.io/turbowarp-teachable-machine/ja/)** ·
|
|
17
|
-
[Block reference](#blocks)
|
|
11
|
+
**User guide:** [English](https://kubohiroya.github.io/turbowarp-teachable-machine/) | [日本語](https://kubohiroya.github.io/turbowarp-teachable-machine/ja/) · **[Block reference](#blocks)**
|
|
18
12
|
|
|
19
|
-
## What
|
|
13
|
+
## What it does
|
|
20
14
|
|
|
21
15
|
```mermaid
|
|
22
16
|
flowchart LR
|
|
@@ -26,318 +20,80 @@ flowchart LR
|
|
|
26
20
|
Scores --> Blocks["TurboWarp blocks"]
|
|
27
21
|
```
|
|
28
22
|
|
|
29
|
-
-
|
|
30
|
-
-
|
|
31
|
-
-
|
|
32
|
-
-
|
|
33
|
-
-
|
|
34
|
-
-
|
|
35
|
-
-
|
|
36
|
-
-
|
|
37
|
-
-
|
|
23
|
+
- Loads a published Teachable Machine pose, image, or audio model from its model folder URL.
|
|
24
|
+
- Switches recognition between `pose`, `image`, and `audio` before the input starts.
|
|
25
|
+
- Starts and stops the camera independently from recognition.
|
|
26
|
+
- Places a camera preview over the stage, at six positions, with adjustable opacity and mirroring.
|
|
27
|
+
- Draws an optional SVG overlay of pose joints and bones on that preview.
|
|
28
|
+
- Reports the best label, its confidence, and the confidence of any named class.
|
|
29
|
+
- Tests a label against a fixed or custom confidence threshold.
|
|
30
|
+
- Smooths flickering decisions with time-decayed accumulated scores.
|
|
31
|
+
- Reports startup timings and the last runtime error.
|
|
38
32
|
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
[Japanese version](https://kubohiroya.github.io/turbowarp-teachable-machine/ja/) has the same content.
|
|
33
|
+
Recognition runs in the browser. Camera frames and microphone audio are never uploaded; only the
|
|
34
|
+
runtime libraries and the published model are fetched.
|
|
42
35
|
|
|
43
36
|
## Requirements
|
|
44
37
|
|
|
45
|
-
-
|
|
46
|
-
|
|
47
|
-
-
|
|
48
|
-
-
|
|
49
|
-
|
|
38
|
+
- A published Teachable Machine pose, image, or audio model URL — the model folder URL, not the
|
|
39
|
+
editor URL.
|
|
40
|
+
- A camera or microphone, and browser permission to use it.
|
|
41
|
+
- Network access for the reviewed Teachable Machine browser runtime and the model files.
|
|
42
|
+
- TurboWarp's **Run without sandbox** option, which only exists when the extension is loaded from a
|
|
43
|
+
file. See [Installation](#installation).
|
|
50
44
|
|
|
51
45
|
> [!IMPORTANT]
|
|
52
|
-
> This is
|
|
53
|
-
> code you trust. Camera APIs also require a secure browser context
|
|
46
|
+
> This extension is unsandboxed because it needs camera, microphone, and stage access. Load
|
|
47
|
+
> extension code only from sources you trust. Camera APIs also require a secure browser context
|
|
48
|
+
> such as HTTPS or `localhost`.
|
|
54
49
|
|
|
55
50
|
## Installation
|
|
56
51
|
|
|
57
|
-
Download [`dist/tm.js`](dist/tm.js)
|
|
58
|
-
|
|
52
|
+
Download [`dist/tm.js`](dist/tm.js). In TurboWarp's custom extension dialog, open the **Files** tab,
|
|
53
|
+
select that file, and keep **Run without sandbox** enabled.
|
|
59
54
|
|
|
60
55
|
> [!IMPORTANT]
|
|
61
56
|
> The dialog's **URL** tab cannot load this extension. TurboWarp always sandboxes an extension
|
|
62
|
-
> loaded from an untrusted URL — the URL tab has no **Run without sandbox** option at all — and
|
|
63
|
-
> refuses to start sandboxed
|
|
64
|
-
>
|
|
65
|
-
>
|
|
66
|
-
> the **Files** tab instead.
|
|
57
|
+
> loaded from an untrusted URL — the URL tab has no **Run without sandbox** option at all — and this
|
|
58
|
+
> extension refuses to start sandboxed. Pasting the jsDelivr link below into the URL tab leaves the
|
|
59
|
+
> dialog waiting with nothing on screen and only `TM must run without the extension sandbox.` in the
|
|
60
|
+
> browser console. Download the file and use the **Files** tab.
|
|
67
61
|
|
|
68
|
-
|
|
69
|
-
jsDelivr:
|
|
62
|
+
The download, and the artifact a composite runtime fetches, is the version-pinned build on jsDelivr:
|
|
70
63
|
|
|
71
64
|
```text
|
|
72
|
-
https://cdn.jsdelivr.net/npm/@kubohiroya/turbowarp-teachable-machine@3.
|
|
65
|
+
https://cdn.jsdelivr.net/npm/@kubohiroya/turbowarp-teachable-machine@3.6.0/dist/tm.js
|
|
73
66
|
```
|
|
74
67
|
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
TensorFlow.js Speech Commands 0.5.4. That runtime ships the WebGL and CPU kernels; the WebGPU and
|
|
78
|
-
WebAssembly backends live in separate bundles next to it and are fetched only when a project uses
|
|
79
|
-
them. See [Compute backends](#compute-backends).
|
|
80
|
-
Composite runtimes can load or embed the same artifact without rewriting a minified third-party
|
|
81
|
-
bundle:
|
|
68
|
+
A composite runtime can load or embed the same reviewed artifact without rewriting a minified
|
|
69
|
+
third-party bundle:
|
|
82
70
|
|
|
83
71
|
```text
|
|
84
|
-
https://cdn.jsdelivr.net/npm/@kubohiroya/turbowarp-teachable-machine@3.
|
|
72
|
+
https://cdn.jsdelivr.net/npm/@kubohiroya/turbowarp-teachable-machine@3.6.0/dist/runtime.js
|
|
85
73
|
```
|
|
86
74
|
|
|
87
75
|
To add the published package to another project:
|
|
88
76
|
|
|
89
77
|
```sh
|
|
90
|
-
pnpm add --save-exact @kubohiroya/turbowarp-teachable-machine@3.
|
|
91
|
-
```
|
|
92
|
-
|
|
93
|
-
### Offline PoseNet bundle API
|
|
94
|
-
|
|
95
|
-
`@kubohiroya/turbowarp-teachable-machine/posenet` owns the fixed PoseNet MobileNetV1 0.75 / stride 16
|
|
96
|
-
manifest, package asset specifiers, SHA-256 verification, bounded Base64 project descriptor, and
|
|
97
|
-
the Teachable Machine Pose fetch adapter. Host applications only decide where that explicit model
|
|
98
|
-
descriptor is stored. The model stays as one JSON file and two binary shards; it is never disguised
|
|
99
|
-
as an image, sound, or costume.
|
|
100
|
-
|
|
101
|
-
```js
|
|
102
|
-
import {readFile} from 'node:fs/promises';
|
|
103
|
-
import {
|
|
104
|
-
createBundledTMRuntime,
|
|
105
|
-
createPoseNetProjectBundleFromLoader,
|
|
106
|
-
} from '@kubohiroya/turbowarp-teachable-machine/posenet';
|
|
107
|
-
|
|
108
|
-
const projectBundle = await createPoseNetProjectBundleFromLoader((file) =>
|
|
109
|
-
readFile(new URL(import.meta.resolve(file.packageSpecifier))),
|
|
110
|
-
);
|
|
111
|
-
|
|
112
|
-
// A browser host can store projectBundle in its own project-data channel.
|
|
113
|
-
// Decode and hash verification start only on the first loadFromFiles() call.
|
|
114
|
-
const offlineRuntime = createBundledTMRuntime({
|
|
115
|
-
runtime: tmPose,
|
|
116
|
-
projectBundle,
|
|
117
|
-
});
|
|
118
|
-
```
|
|
119
|
-
|
|
120
|
-
The model supply contains exactly 5,082,500 bytes. Its three SHA-256 operations run independently,
|
|
121
|
-
and the first shared verification overlaps the Teachable Machine classifier load. The intercepted
|
|
122
|
-
PoseNet fetch never exposes bytes until that verification succeeds. Later model requests reuse the
|
|
123
|
-
same verified supply. Tampering, missing or incorrectly sized files, and unexpected PoseNet network
|
|
124
|
-
requests fail closed with `TM-POSENET-*` error codes.
|
|
125
|
-
|
|
126
|
-
### Composition API
|
|
127
|
-
|
|
128
|
-
Composite runtimes can import `@kubohiroya/turbowarp-teachable-machine/composition` without registering the
|
|
129
|
-
Standalone extension or adding blocks. The caller supplies an already-bundled Teachable Machine
|
|
130
|
-
Pose runtime and validated model bytes, so this path does not download runtime scripts or model
|
|
131
|
-
files.
|
|
132
|
-
|
|
133
|
-
```js
|
|
134
|
-
import {createTMComposition} from '@kubohiroya/turbowarp-teachable-machine/composition';
|
|
135
|
-
|
|
136
|
-
const pose = createTMComposition({
|
|
137
|
-
runtime: bundledTMRuntime,
|
|
138
|
-
modelInitializationPolicy: 'latest-needed',
|
|
139
|
-
// Optional latency-first mode; it remains off by default.
|
|
140
|
-
// parallelModelInitialization: true,
|
|
141
|
-
});
|
|
142
|
-
const cameras = await pose.listCameraDevices();
|
|
143
|
-
if (cameras[0]) await pose.selectCamera({deviceId: cameras[0].deviceId});
|
|
144
|
-
const modelDemand = new AbortController();
|
|
145
|
-
await pose.registerPoseModel(
|
|
146
|
-
{
|
|
147
|
-
name: 'RescuePose',
|
|
148
|
-
files: [
|
|
149
|
-
{path: 'model.json', bytes: modelBytes},
|
|
150
|
-
{path: 'weights.bin', bytes: weightsBytes},
|
|
151
|
-
{path: 'metadata.json', bytes: metadataBytes},
|
|
152
|
-
],
|
|
153
|
-
},
|
|
154
|
-
{signal: modelDemand.signal},
|
|
155
|
-
);
|
|
156
|
-
pose.activatePoseModel('RescuePose');
|
|
157
|
-
pose.setPreviewOpacity(0.2);
|
|
158
|
-
pose.setPreviewPosition('full-stage');
|
|
159
|
-
pose.setPreviewMirroring('unmirrored');
|
|
160
|
-
pose.setPoseJointStyle('leftWrist', {color: '#ff00aa', opacity: 0.8, radius: 6});
|
|
161
|
-
pose.setPoseBoneStyle({color: '#00e5ff', opacity: 0.9, width: 3});
|
|
162
|
-
pose.setPoseOverlayMinimumConfidence(0.5);
|
|
163
|
-
pose.setPoseOverlayConfidenceScaling({
|
|
164
|
-
jointOpacity: true,
|
|
165
|
-
jointRadius: true,
|
|
166
|
-
boneOpacity: true,
|
|
167
|
-
boneWidth: true,
|
|
168
|
-
});
|
|
169
|
-
pose.configureAccumulatedPose({
|
|
170
|
-
accumulationPerSecond: 1,
|
|
171
|
-
decayPerSecond: 0.9,
|
|
172
|
-
scoreThreshold: 0,
|
|
173
|
-
});
|
|
174
|
-
const unsubscribe = pose.subscribeAccumulatedPose((event) => {
|
|
175
|
-
// event.poseName is the one currently selected candidate, or an empty string.
|
|
176
|
-
});
|
|
177
|
-
await pose.startRecognition();
|
|
78
|
+
pnpm add --save-exact @kubohiroya/turbowarp-teachable-machine@3.6.0
|
|
178
79
|
```
|
|
179
80
|
|
|
180
|
-
|
|
181
|
-
model with `releasePoseModel(name)` or release the complete instance with `releaseAll()`. Calling
|
|
182
|
-
`releaseAll()` also removes accumulated-pose listeners. `subscribeAccumulatedPose()` returns an
|
|
183
|
-
idempotent unsubscribe function.
|
|
184
|
-
|
|
185
|
-
`releasePoseModel(name)` stops recognition when the selected model is active and disposes only that
|
|
186
|
-
model. It does not stop the camera stream. Camera capture is an independent lifecycle: stop it with
|
|
187
|
-
`stopCamera()`, or release the complete composition with `releaseAll()`.
|
|
188
|
-
|
|
189
|
-
The default `modelInitializationPolicy` is `legacy`, preserving independent registration calls.
|
|
190
|
-
Opt in to `latest-needed` when a host has one current label-model demand, such as a story that can
|
|
191
|
-
skip scenes. At most one heavy runtime load is active and one latest request is pending. A newer
|
|
192
|
-
request cancels the active request cooperatively and replaces an older request that has not started.
|
|
193
|
-
For A loading, then B pending, then C requested, B never reaches the TensorFlow loader; A is cleaned
|
|
194
|
-
up at its next safe boundary and only C starts afterward.
|
|
195
|
-
Repeated `latest-needed` registrations with the same name, byte-identical files, and the same
|
|
196
|
-
`AbortSignal` share one pending operation instead of invoking the runtime twice. Different signals
|
|
197
|
-
remain separate demands so cancelling one caller cannot silently cancel another caller's work.
|
|
198
|
-
|
|
199
|
-
Pass an optional `AbortSignal` to `registerPoseModel()` when the host no longer needs that demand. A
|
|
200
|
-
queued request rejects with `AbortError` without invoking the runtime. TensorFlow.js 4.22.0 and Web
|
|
201
|
-
Crypto do not expose physical cancellation for every operation already in progress, so a running
|
|
202
|
-
request becomes stale immediately, starts no later phase where the runtime provides a boundary, and
|
|
203
|
-
disposes every late classifier/PoseNet result exactly once before its promise settles. A cancelled
|
|
204
|
-
model is never published in the registry. Shared fixed-PoseNet verification may finish and remain
|
|
205
|
-
cached when the next latest request also needs it. Model cancellation does not stop the camera.
|
|
206
|
-
|
|
207
|
-
The composition API does not infer scene or action reachability. A host should retain the signal when the same
|
|
208
|
-
model is needed by an imminent action, abort it when a skipped path has no nearby pose demand, or
|
|
209
|
-
submit the replacement model when a skip changes that demand. `releaseAll()` cancels both active and
|
|
210
|
-
pending registration work and waits for safe cleanup.
|
|
211
|
-
|
|
212
|
-
Runtime model phases remain sequential by default. Cancellation is checked after TensorFlow
|
|
213
|
-
readiness, classifier loading, metadata parsing, and PoseNet loading, so a skipped demand does not
|
|
214
|
-
start the next expensive phase. Set `parallelModelInitialization: true` on the composition or
|
|
215
|
-
`createBundledTMRuntime()` only when lower startup latency is more important than those phase
|
|
216
|
-
boundaries. That opt-in starts classifier, metadata, and PoseNet initialization together; work
|
|
217
|
-
already accepted by TensorFlow cannot be physically interrupted, but every completed stale resource
|
|
218
|
-
is still disposed before cancellation settles. The `latest-needed` queue continues to permit only
|
|
219
|
-
one active demand and one newest pending demand in either mode.
|
|
220
|
-
|
|
221
|
-
`setPreviewMirroring('mirrored' | 'unmirrored')` changes only the camera preview. It can be called
|
|
222
|
-
before camera startup, while the camera is running, or after it has stopped. The default remains
|
|
223
|
-
`mirrored`, and the recognition input keeps the Teachable Machine runtime's existing horizontal
|
|
224
|
-
flip regardless of the preview setting.
|
|
225
|
-
|
|
226
|
-
`setPreviewOpacity()` accepts a finite number from `0` to `1` and applies before or after camera
|
|
227
|
-
startup. `setPreviewPosition()` accepts `top-left`, `top-right`, `bottom-left`, `bottom-right`,
|
|
228
|
-
`center`, or `full-stage`. `hidePreview()` and `showPreview()` change only preview visibility; they
|
|
229
|
-
do not stop or restart the camera stream. Use `stopCamera()` when the MediaStream tracks must be
|
|
230
|
-
released.
|
|
231
|
-
|
|
232
|
-
The composition API enables its SVG pose overlay and exposes `showPoseOverlay()`,
|
|
233
|
-
`hidePoseOverlay()`, and `isPoseOverlayVisible()`. `setPoseJointStyle()` configures the CSS color,
|
|
234
|
-
opacity, and radius for any of the 17 PoseNet joint names. `setPoseBoneStyle()` configures the color,
|
|
235
|
-
opacity, and width shared by the 12 standard PoseNet bone connections. Opacity must be from `0` to
|
|
236
|
-
`1`; radii and widths must be finite and non-negative. `setPoseOverlayMinimumConfidence()` accepts
|
|
237
|
-
`0` through `1` and hides a bone when either endpoint is below the threshold.
|
|
238
|
-
|
|
239
|
-
`setPoseOverlayConfidenceScaling()` independently controls `jointOpacity`, `jointRadius`,
|
|
240
|
-
`boneOpacity`, and `boneWidth`. An enabled joint property is multiplied by that joint's confidence,
|
|
241
|
-
clamped to `0`–`1`. An enabled bone property is multiplied by the lower confidence of its two
|
|
242
|
-
endpoints. Hiding the overlay does not stop recognition. Hiding the camera preview also hides the
|
|
243
|
-
overlay, while stopping recognition clears every SVG mark and stopping the camera removes the SVG
|
|
244
|
-
element.
|
|
245
|
-
|
|
246
|
-
`listCameraDevices()` performs a fresh browser enumeration on every call. It returns a deeply
|
|
247
|
-
frozen canonical copy containing only unique, non-empty video device IDs in enumeration order;
|
|
248
|
-
labels remain the empty string when permission has not made them available. Empty IDs are omitted,
|
|
249
|
-
and the first occurrence wins when an ID is duplicated. Enumeration failures use
|
|
250
|
-
`TM-COMPOSITION-012` and never expose `MediaDeviceInfo`, `MediaStreamTrack`, or a stream.
|
|
251
|
-
|
|
252
|
-
Select `default`, `front`, or `back` as a camera preference, or wrap a session-only browser ID as
|
|
253
|
-
`{deviceId}`. The object form is intentionally distinct, so a physical device whose ID is literally
|
|
254
|
-
`default`, `front`, or `back` is not mistaken for a preference. A selection can be made before
|
|
255
|
-
camera startup or while recognition is running. Running switches preserve the prepared model,
|
|
256
|
-
recognition state, and all preview settings. Concurrent selections run in call order; a failed
|
|
257
|
-
switch restores the preceding successful camera and selection. Invalid selections use
|
|
258
|
-
`TM-COMPOSITION-011`. `getCameraSelection()` returns an isolated immutable copy, and
|
|
259
|
-
`getActiveCamera()` returns an immutable `{deviceId, label}` only while an identifiable camera is
|
|
260
|
-
running. The extension never persists device IDs. `releaseAll()` rejects queued selection work, waits for
|
|
261
|
-
an in-progress switch to become quiescent, and then stops the final stream; all four camera-device
|
|
262
|
-
methods fail closed after release.
|
|
263
|
-
|
|
264
|
-
The extension owns the camera canvas used by Teachable Machine and TensorFlow.js and requests a normal
|
|
265
|
-
Canvas2D context for it. Physical-camera measurements of its 320x240, one-draw/one-read path
|
|
266
|
-
did not show a repeatable end-to-end benefit from `willReadFrequently`; that hint improved a
|
|
267
|
-
different read-heavy condition with four or more reads per draw, but could make video drawing more
|
|
268
|
-
expensive. Chrome may therefore emit its Canvas2D readback warning during CPU inference. Preview
|
|
269
|
-
rendering and model input continue to use the same canvas, and no downstream Canvas prototype or
|
|
270
|
-
Console patch is required. Reconsider the hint only with a reproducible benchmark of this exact
|
|
271
|
-
camera path.
|
|
272
|
-
|
|
273
|
-
Teachable Machine Pose 0.8.6 exposes the classifier as `CustomPoseNet.model` and PoseNet as
|
|
274
|
-
`CustomPoseNet.posenetModel`, while its top-level `dispose()` releases PoseNet only. The composition
|
|
275
|
-
therefore disconnects an active model from recognition first and disposes those two public
|
|
276
|
-
resources separately and exactly once; it does not call the incomplete top-level disposer for this
|
|
277
|
-
official shape. A custom injected runtime without those fields can retain the legacy single
|
|
278
|
-
top-level `dispose()` contract. A runtime that exposes only part of the official shape is rejected
|
|
279
|
-
with `TM-COMPOSITION-009`, after every safely identifiable resource has been attempted.
|
|
280
|
-
|
|
281
|
-
Model, weights, and metadata `File` objects exist only for the pending `loadFromFiles()` call and
|
|
282
|
-
are not stored in the named registry. `releasePoseModel()` and `releaseAll()` invalidate and wait for
|
|
283
|
-
matching pending registrations, so their promises do not resolve before a late loaded model has
|
|
284
|
-
been disposed. Switching to an already prepared model after stopping recognition keeps the camera
|
|
285
|
-
stream alive; releasing the old, no-longer-active model does not request camera permission again.
|
|
286
|
-
Releasing the current model with no prepared successor stops recognition but keeps the camera
|
|
287
|
-
stream alive. `stopCamera()` and `releaseAll()` stop the stream explicitly.
|
|
288
|
-
|
|
289
|
-
The accumulated-pose API chooses one candidate for an async-input consumer. It is deliberately
|
|
290
|
-
separate from an Actor action that waits for multiple pose steps in sequence: a sequence consumer
|
|
291
|
-
should read `confidenceOf(name)` and own its per-step progress instead of using or resetting this
|
|
292
|
-
candidate-selection state.
|
|
293
|
-
|
|
294
|
-
Accumulated scores use elapsed wall-clock seconds, not recognition counts:
|
|
295
|
-
|
|
296
|
-
```text
|
|
297
|
-
nextScore = previousScore * decayPerSecond^elapsedSeconds
|
|
298
|
-
+ confidence * accumulationPerSecond * elapsedSeconds
|
|
299
|
-
```
|
|
300
|
-
|
|
301
|
-
`accumulationPerSecond` is a finite number greater than or equal to zero. `decayPerSecond` is the
|
|
302
|
-
finite fraction retained per second from zero through one; a changed decay takes effect in the next
|
|
303
|
-
recognition session. `scoreThreshold` is a finite number greater than or equal to zero. An event is
|
|
304
|
-
published only when the selected pose name changes, including one transition to an empty name on
|
|
305
|
-
reset or stop; score-only changes do not publish another event. The standalone extension enables
|
|
306
|
-
temporal scoring and keeps only the event feature flag off by default.
|
|
307
|
-
|
|
308
|
-
## Upgrading from 2.x
|
|
309
|
-
|
|
310
|
-
The block surface, the composition API, and the offline PoseNet bundle API are unchanged. A project
|
|
311
|
-
built on 2.x keeps working after swapping the extension file. New blocks appear that 2.x did not
|
|
312
|
-
have: three for compute backends, and six for accumulated pose scoring.
|
|
313
|
-
|
|
314
|
-
What changed is the runtime this extension bundles.
|
|
315
|
-
|
|
316
|
-
| | 2.x | 3.0.0 |
|
|
317
|
-
|---|---|---|
|
|
318
|
-
| TensorFlow.js | 1.3.1 | 4.22.0 |
|
|
319
|
-
| Teachable Machine Pose | 0.8.3 | 0.8.6 |
|
|
320
|
-
| TensorFlow.js Speech Commands | 0.4.0 | 0.5.4 |
|
|
321
|
-
| Compute backends | WebGL only | WebGPU, WebGL, WASM, CPU, negotiated at load |
|
|
322
|
-
|
|
323
|
-
That runtime is published as `globalThis.tf`, so a host application reading it directly moves from
|
|
324
|
-
the TensorFlow.js 1.x API to 4.x — the reason this is a major release. Models exported by Teachable
|
|
325
|
-
Machine are not affected: pose, image, and audio models exported for the 1.x-era libraries load and
|
|
326
|
-
run unchanged on 4.22.0.
|
|
81
|
+
### Names
|
|
327
82
|
|
|
328
|
-
|
|
83
|
+
The Scratch extension ID is `kubohiroyatm`, from the 2.x release line onward, so that it does not
|
|
84
|
+
collide with other short IDs. The package, repository, Pages URL, and browser build all use
|
|
85
|
+
`turbowarp-teachable-machine`.
|
|
329
86
|
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
-
|
|
333
|
-
1.04 MB), so plan the startup sequence accordingly. See [Compute backends](#compute-backends).
|
|
87
|
+
> The package, repository, and Pages URL were previously named `turbowarp-tm`
|
|
88
|
+
> (`@kubohiroya/turbowarp-tm`, last published as 3.2.0). Replace that name with
|
|
89
|
+
> `turbowarp-teachable-machine` in CDN URLs, `pnpm add`, and `import` specifiers.
|
|
334
90
|
|
|
335
91
|
## Quick start
|
|
336
92
|
|
|
337
93
|
1. Train classes such as `jump`, `card`, or `clap` in Teachable Machine.
|
|
338
94
|
2. Export the model, upload it, and copy the model folder URL.
|
|
339
95
|
3. Set that URL with `set model URL to [URL]`.
|
|
340
|
-
4. Run `start recognition`, allow camera or microphone access, and
|
|
96
|
+
4. Run `start recognition`, allow camera or microphone access, and read a result block.
|
|
341
97
|
|
|
342
98
|
```text
|
|
343
99
|
when green flag clicked
|
|
@@ -351,208 +107,361 @@ forever
|
|
|
351
107
|
end
|
|
352
108
|
```
|
|
353
109
|
|
|
354
|
-
`start recognition` starts the
|
|
355
|
-
image modes use the camera; audio mode uses the microphone. A separate `start camera` or
|
|
356
|
-
step is only needed when a
|
|
110
|
+
`start recognition` starts the input it needs and loads the configured model when necessary. Pose
|
|
111
|
+
and image modes use the camera; audio mode uses the microphone. A separate `start camera` or
|
|
112
|
+
`load model` step is only needed when a project wants to control the startup phases itself — for
|
|
113
|
+
example to show the preview before the model is ready, or to read `model load time (ms)`.
|
|
357
114
|
|
|
358
115
|
## Compute backends
|
|
359
116
|
|
|
360
|
-
Recognition runs on a TensorFlow.js compute backend. `set compute mode to [MODE]` chooses
|
|
117
|
+
Recognition runs on a TensorFlow.js compute backend. `set compute mode to [MODE]` chooses one and
|
|
361
118
|
`active compute backend` reports the one actually in use.
|
|
362
119
|
|
|
363
120
|
| Mode | What it does |
|
|
364
121
|
|---|---|
|
|
365
|
-
| `auto` (default) | Tries WebGPU, WebGL, WASM, and CPU in that order
|
|
366
|
-
| `webgpu` | Modern GPU path
|
|
122
|
+
| `auto` (default) | Tries WebGPU, WebGL, WASM, and CPU in that order, keeping the first that activates and computes a verification convolution correctly |
|
|
123
|
+
| `webgpu` | Modern GPU path, lowest per-operation overhead |
|
|
367
124
|
| `webgl` | The long-standing GPU path, available in every browser this extension supports |
|
|
368
125
|
| `wasm` | SIMD WebAssembly kernels; steady on machines with weak or unreliable GPU drivers |
|
|
369
|
-
| `cpu` | Plain JavaScript kernels; correct everywhere and slowest |
|
|
126
|
+
| `cpu` | Plain JavaScript kernels; correct everywhere, and slowest |
|
|
370
127
|
|
|
371
|
-
A named mode is
|
|
372
|
-
over, `active compute backend` shows what won, and `last error` explains the substitution.
|
|
373
|
-
|
|
374
|
-
|
|
375
|
-
```text
|
|
376
|
-
when green flag clicked
|
|
377
|
-
set compute mode to [webgpu]
|
|
378
|
-
set model URL to [https://teachablemachine.withgoogle.com/models/.../]
|
|
379
|
-
start recognition
|
|
380
|
-
```
|
|
381
|
-
|
|
382
|
-
The reviewed browser runtime carries the WebGL and CPU kernels. The WebGPU and WebAssembly backends
|
|
383
|
-
are separate bundles next to `dist/runtime.js`, fetched only when a project reaches for them, and
|
|
384
|
-
the WebAssembly binaries come from `dist/wasm/` in this same package. Everything resolves against
|
|
385
|
-
the one TensorFlow.js instance the runtime published, so no page ever runs two TensorFlow.js cores.
|
|
128
|
+
A named mode is a preference, not a guarantee. When the browser cannot provide it the same order
|
|
129
|
+
takes over, `active compute backend` shows what won, and `last error` explains the substitution.
|
|
130
|
+
Read `active compute backend` rather than assuming the requested mode.
|
|
386
131
|
|
|
387
132
|
TensorFlow.js binds every tensor to the backend that was active when the tensor was created, so the
|
|
388
133
|
mode has to be settled before a model loads. Changing the mode therefore releases a model that was
|
|
389
134
|
loaded from a URL, and the next `load model` or `start recognition` rebuilds it on the new backend.
|
|
390
135
|
Changing the mode during recognition, or while a host-prepared model is active, is refused instead.
|
|
391
136
|
|
|
392
|
-
|
|
393
|
-
|
|
394
|
-
|
|
395
|
-
|
|
396
|
-
runtime: tmPose,
|
|
397
|
-
compute: tmCompute, // published by dist/runtime.js
|
|
398
|
-
computeMode: 'auto',
|
|
399
|
-
});
|
|
400
|
-
|
|
401
|
-
const selection = await composition.selectComputeBackend('webgpu');
|
|
402
|
-
// selection.backend === 'webgpu' | 'webgl' | 'wasm' | 'cpu'
|
|
403
|
-
// selection.fallback === true when the requested backend was unusable
|
|
404
|
-
```
|
|
137
|
+
The reviewed browser runtime carries the WebGL and CPU kernels. WebGPU and WebAssembly live in
|
|
138
|
+
separate bundles next to `dist/runtime.js`, fetched only when a project reaches for them, with the
|
|
139
|
+
WebAssembly binaries coming from `dist/wasm/` in this same package. Everything resolves against the
|
|
140
|
+
one TensorFlow.js instance the runtime published, so no page ever runs two TensorFlow.js cores.
|
|
405
141
|
|
|
406
|
-
|
|
407
|
-
because their tensors live in the memory of the backend that loaded them. Release them first.
|
|
408
|
-
|
|
409
|
-
## Reading recognition results
|
|
142
|
+
## Reading results
|
|
410
143
|
|
|
411
144
|
| Block | Result |
|
|
412
145
|
|---|---|
|
|
413
|
-
| `current label` |
|
|
414
|
-
| `confidence` |
|
|
415
|
-
| `confidence of [NAME]` |
|
|
146
|
+
| `current label` | The class with the highest probability in the latest frame |
|
|
147
|
+
| `confidence` | That class's probability, rounded to two decimal places |
|
|
148
|
+
| `confidence of [NAME]` | The probability of one named class |
|
|
416
149
|
| `label is [NAME]?` | Whether the named class has at least `0.75` confidence |
|
|
417
|
-
| `label is [NAME] with confidence at least [THRESHOLD]?` |
|
|
150
|
+
| `label is [NAME] with confidence at least [THRESHOLD]?` | The same test with a custom `0`–`1` threshold |
|
|
151
|
+
|
|
152
|
+
Live confidence reacts quickly and fluctuates near a decision boundary. Better training data,
|
|
153
|
+
lighting or audio quality, framing, and a threshold chosen for the project usually matter more than
|
|
154
|
+
anything this extension can do for you. When a decision should survive a single bad frame, use
|
|
155
|
+
[accumulated pose scoring](#accumulated-pose-scoring) instead of raising the threshold.
|
|
418
156
|
|
|
419
|
-
|
|
420
|
-
lighting or audio quality, camera or microphone framing, and a suitable threshold usually improve
|
|
421
|
-
the result.
|
|
157
|
+
## Camera, preview, and what the model sees
|
|
422
158
|
|
|
423
|
-
|
|
159
|
+
### Choosing a camera
|
|
424
160
|
|
|
425
|
-
|
|
426
|
-
`back camera`, or
|
|
427
|
-
|
|
428
|
-
|
|
429
|
-
the camera is running restarts only the camera stream, preserving the loaded model, recognition
|
|
430
|
-
state, and preview settings. If switching fails, the extension attempts to restore the previous camera and
|
|
431
|
-
records the error in `last error`.
|
|
161
|
+
`refresh camera list` detects video inputs. `set camera to [CAMERA]` then selects `default camera`,
|
|
162
|
+
`front camera`, `back camera`, or one detected device. Labels may stay empty until the browser
|
|
163
|
+
grants camera permission, and device IDs are browser- and machine-specific, so prefer the portable
|
|
164
|
+
front/back choices when a project moves between devices.
|
|
432
165
|
|
|
433
|
-
|
|
434
|
-
|
|
166
|
+
Changing the selection while the camera runs restarts only the camera stream and preserves the
|
|
167
|
+
loaded model, the recognition state, and every preview setting. A failed switch restores the
|
|
168
|
+
previous camera and records the reason in `last error`. `camera count`, `camera device ID`, and
|
|
169
|
+
`camera device name` report the latest refreshed count and the active input.
|
|
435
170
|
|
|
436
|
-
The preview
|
|
437
|
-
six stage positions, made transparent, or expanded to fill the stage. Hiding the preview does not
|
|
438
|
-
stop recognition. The preview is mirrored by default. Use `set camera preview to [MIRRORING]` to
|
|
439
|
-
switch between `mirrored` and `unmirrored`, including while the camera is running. This display-only
|
|
440
|
-
setting does not change the frames used for recognition. The `camera preview mirroring` reporter
|
|
441
|
-
returns the current setting.
|
|
171
|
+
### The preview and the model input
|
|
442
172
|
|
|
443
|
-
|
|
444
|
-
|
|
173
|
+
The preview is the camera's own video element placed over the stage, at the camera's resolution and
|
|
174
|
+
aspect ratio. It can be shown, hidden, moved to six positions, made transparent, or expanded to
|
|
175
|
+
fill the stage, and it is mirrored by default — `set camera preview to [MIRRORING]` switches that,
|
|
176
|
+
including while the camera runs. Mirroring is display-only.
|
|
177
|
+
|
|
178
|
+
The model is fed a square centre crop of the frame, mirrored, as a tensor. `fromPixels` uploads the
|
|
179
|
+
video to a texture on the WebGL and WebGPU backends and the crop and mirroring are tensor ops, so
|
|
180
|
+
no Canvas2D stage sits between the camera and the model. The square also makes PoseNet's padding
|
|
181
|
+
zero.
|
|
182
|
+
|
|
183
|
+
- `stop recognition` clears current results and stops audio listening, but leaves the camera
|
|
184
|
+
available.
|
|
185
|
+
- `stop camera` also stops recognition, releases the camera, and removes the preview and the SVG
|
|
186
|
+
overlay.
|
|
187
|
+
|
|
188
|
+
Hiding the preview does not stop recognition.
|
|
445
189
|
|
|
446
190
|
### Sharing one camera with other extensions
|
|
447
191
|
|
|
448
|
-
When [TurboWarp-Camera-Source](https://github.com/kubohiroya/turbowarp-camera-source) is loaded,
|
|
449
|
-
extension takes a lease on the camera it shares under the `pose` role instead of opening a
|
|
450
|
-
its own. One physical camera then serves recognition and anything else reading frames — a
|
|
451
|
-
a preview, another classifier — rather than each extension competing for the device.
|
|
192
|
+
When [TurboWarp-Camera-Source](https://github.com/kubohiroya/turbowarp-camera-source) is loaded,
|
|
193
|
+
this extension takes a lease on the camera it shares under the `pose` role instead of opening a
|
|
194
|
+
device of its own. One physical camera then serves recognition and anything else reading frames — a
|
|
195
|
+
QR reader, a preview, another classifier — rather than each extension competing for the device.
|
|
452
196
|
|
|
453
197
|
Camera Source owns the stream: it stops the camera when the last lease is released, and `stop
|
|
454
|
-
camera` here releases this extension's lease
|
|
455
|
-
|
|
456
|
-
|
|
457
|
-
|
|
458
|
-
|
|
459
|
-
|
|
460
|
-
|
|
461
|
-
|
|
462
|
-
|
|
463
|
-
|
|
464
|
-
|
|
465
|
-
|
|
466
|
-
|
|
467
|
-
|
|
468
|
-
|
|
469
|
-
|
|
470
|
-
|
|
471
|
-
|
|
472
|
-
|
|
473
|
-
|
|
474
|
-
Leasing is fixed in 3.4.1. In 3.4.0 a leased start threw `Webcam canvas does not provide a 2D
|
|
475
|
-
context` and recognition never began, because the upstream camera helper builds its readback canvas
|
|
476
|
-
in the same branch that opens the device — the branch a supplied element skips. Use 3.4.1 wherever
|
|
477
|
-
Camera Source is loaded; 3.4.0 is unaffected only when it is not.
|
|
478
|
-
|
|
479
|
-
## Camera preview and what the model sees
|
|
480
|
-
|
|
481
|
-
The preview on the stage is the camera's own video element, at the camera's resolution and aspect
|
|
482
|
-
ratio. It used to be a 320×240 readback canvas scaled to fit, so a 1080p camera looked like a
|
|
483
|
-
320×240 one and a 16:9 camera was shown in a 4:3 window.
|
|
484
|
-
|
|
485
|
-
The model is fed a square centre crop of the frame, mirrored, as a tensor. `fromPixels` uploads
|
|
486
|
-
the video to a texture on the WebGL and WebGPU backends and the crop and mirroring are tensor ops,
|
|
487
|
-
so no Canvas2D stage sits between the camera and the model. The square also makes PoseNet's
|
|
488
|
-
padding zero, where the old 4:3 canvas was padded back to square and left the subject about 25%
|
|
489
|
-
smaller than the frames the model was trained on. The crop is the whole square now: the readback
|
|
490
|
-
canvas held only its top 240 rows and discarded the rest.
|
|
491
|
-
|
|
492
|
-
The pose overlay spans the preview and its keypoints are in the camera frame's own coordinates.
|
|
493
|
-
|
|
494
|
-
The extension performs pose, image, or audio classification in the browser and does not upload
|
|
495
|
-
camera frames or microphone audio. It does fetch its runtime libraries and the published model. Stop
|
|
496
|
-
recognition and the camera when the project no longer needs them.
|
|
497
|
-
|
|
498
|
-
## Optional SVG pose overlay
|
|
499
|
-
|
|
500
|
-
The `poseOverlay` feature flag is **on by default** for the standalone TurboWarp extension, adding
|
|
501
|
-
blocks for overlay visibility, per-joint circle color/opacity/radius, shared bone
|
|
502
|
-
color/opacity/width, minimum keypoint confidence, and confidence scaling. The overlay's SVG coordinate system is the
|
|
503
|
-
camera frame's own, so it follows a camera of any resolution or aspect ratio, and it honours all
|
|
504
|
-
six preview positions and preview mirroring. The Composition API enables this isolated overlay layer directly.
|
|
505
|
-
|
|
506
|
-
Confidence scaling can be switched independently for joint opacity, joint radius, bone opacity,
|
|
507
|
-
and bone width. Joint properties use the corresponding keypoint confidence; bone properties use
|
|
508
|
-
the smaller confidence of the connected endpoints. This makes every enabled property vary from
|
|
509
|
-
zero through its configured value without changing the recognition classifier input.
|
|
198
|
+
camera` here releases only this extension's lease. Camera Source is not required; without it the
|
|
199
|
+
extension opens and stops its own camera exactly as before, and a project needs no edit either way.
|
|
200
|
+
|
|
201
|
+
## Pose overlay
|
|
202
|
+
|
|
203
|
+
The `poseOverlay` feature flag is **on by default**, so the overlay blocks are in the palette:
|
|
204
|
+
visibility, per-joint circle colour/opacity/radius for the 17 PoseNet joints, shared bone
|
|
205
|
+
colour/opacity/width for the 12 standard connections, a minimum keypoint confidence, and confidence
|
|
206
|
+
scaling.
|
|
207
|
+
|
|
208
|
+
The overlay's SVG coordinate system is the camera frame's own, so it follows a camera of any
|
|
209
|
+
resolution or aspect ratio and honours all six preview positions and preview mirroring.
|
|
210
|
+
|
|
211
|
+
Confidence scaling switches independently for joint opacity, joint radius, bone opacity, and bone
|
|
212
|
+
width. A joint property is multiplied by that keypoint's confidence; a bone property by the lower
|
|
213
|
+
confidence of its two endpoints. A bone is hidden when either endpoint falls below the minimum
|
|
214
|
+
confidence. None of this changes the classifier input.
|
|
215
|
+
|
|
216
|
+
Hiding the overlay does not stop recognition. Hiding the preview also hides the overlay; stopping
|
|
217
|
+
recognition clears every mark; stopping the camera removes the SVG element.
|
|
510
218
|
|
|
511
219
|
## Accumulated pose scoring
|
|
512
220
|
|
|
513
|
-
Live confidence flickers frame to frame near a decision boundary.
|
|
514
|
-
|
|
515
|
-
|
|
516
|
-
**on by default**, so these blocks are in the palette.
|
|
221
|
+
Live confidence flickers frame to frame near a decision boundary. Accumulated scores combine
|
|
222
|
+
evidence over time, so a brief misrecognition cannot flip a decision on its own. The
|
|
223
|
+
`temporalPoseScoring` feature flag is **on by default**.
|
|
517
224
|
|
|
518
225
|
```text
|
|
519
|
-
|
|
226
|
+
nextScore = previousScore × decayPerSecond^elapsedSeconds
|
|
227
|
+
+ confidence × accumulationPerSecond × elapsedSeconds
|
|
520
228
|
```
|
|
521
229
|
|
|
522
|
-
|
|
523
|
-
|
|
524
|
-
|
|
230
|
+
Scores use elapsed wall-clock seconds, not recognition counts. The accumulation coefficient is a
|
|
231
|
+
per-second rate of zero or more. The decay coefficient is the fraction retained after one second,
|
|
232
|
+
clamped to `0`–`1`; a changed decay takes effect the next time recognition starts. Both pause while
|
|
233
|
+
the document is hidden.
|
|
525
234
|
|
|
526
235
|
`accumulated pose` returns the highest positive accumulated pose only when it meets the configured
|
|
527
|
-
threshold,
|
|
528
|
-
|
|
529
|
-
scores.
|
|
236
|
+
threshold, and an empty string otherwise; the score reporters keep returning their unrounded values
|
|
237
|
+
below it. Resetting or stopping recognition clears every accumulated score.
|
|
530
238
|
|
|
531
|
-
|
|
239
|
+
This chooses one candidate. A consumer that waits for several pose steps in sequence should read
|
|
240
|
+
`confidence of [NAME]` and own its own per-step progress rather than resetting this shared
|
|
241
|
+
candidate-selection state.
|
|
242
|
+
|
|
243
|
+
### Change events
|
|
532
244
|
|
|
533
245
|
The `accumulatedPoseEvents` feature flag is **off by default** and requires `temporalPoseScoring`.
|
|
534
|
-
|
|
246
|
+
With both enabled, other unsandboxed extensions can check
|
|
535
247
|
`runtime.ext_kubohiroyatm.supportsAccumulatedPoseEvents()` and subscribe to
|
|
536
248
|
`TM_ACCUMULATED_POSE_CHANGED` on the TurboWarp runtime.
|
|
537
249
|
|
|
538
|
-
|
|
539
|
-
|
|
250
|
+
A version 2 event carries `poseName`, `previousPoseName`, `score`, `reason` (`recognition`, `reset`,
|
|
251
|
+
or `stop`), and a monotonic `timestamp`. An event is published only when the selected name changes,
|
|
252
|
+
including the transition to an empty name on reset or stop; score-only updates publish nothing.
|
|
253
|
+
|
|
254
|
+
## Host application APIs
|
|
255
|
+
|
|
256
|
+
These two entry points are for applications that embed the runtime. A project built in the TurboWarp
|
|
257
|
+
editor needs neither.
|
|
258
|
+
|
|
259
|
+
### Composition API
|
|
260
|
+
|
|
261
|
+
`@kubohiroya/turbowarp-teachable-machine/composition` gives a composite runtime the recognition
|
|
262
|
+
engine without registering the standalone extension or adding blocks. The caller supplies an
|
|
263
|
+
already-bundled Teachable Machine Pose runtime and validated model bytes, so this path downloads no
|
|
264
|
+
runtime script and no model file.
|
|
265
|
+
|
|
266
|
+
```js
|
|
267
|
+
import {createTMComposition} from '@kubohiroya/turbowarp-teachable-machine/composition';
|
|
268
|
+
|
|
269
|
+
const pose = createTMComposition({
|
|
270
|
+
runtime: bundledTMRuntime,
|
|
271
|
+
compute: tmCompute, // published by dist/runtime.js
|
|
272
|
+
modelInitializationPolicy: 'latest-needed',
|
|
273
|
+
});
|
|
274
|
+
|
|
275
|
+
const cameras = await pose.listCameraDevices();
|
|
276
|
+
if (cameras[0]) await pose.selectCamera({deviceId: cameras[0].deviceId});
|
|
277
|
+
|
|
278
|
+
await pose.registerPoseModel({
|
|
279
|
+
name: 'RescuePose',
|
|
280
|
+
files: [
|
|
281
|
+
{path: 'model.json', bytes: modelBytes},
|
|
282
|
+
{path: 'weights.bin', bytes: weightsBytes},
|
|
283
|
+
{path: 'metadata.json', bytes: metadataBytes},
|
|
284
|
+
],
|
|
285
|
+
});
|
|
286
|
+
pose.activatePoseModel('RescuePose');
|
|
287
|
+
|
|
288
|
+
pose.setPreviewPosition('full-stage');
|
|
289
|
+
pose.setPoseBoneStyle({color: '#00e5ff', opacity: 0.9, width: 3});
|
|
290
|
+
pose.configureAccumulatedPose({accumulationPerSecond: 1, decayPerSecond: 0.9, scoreThreshold: 0});
|
|
291
|
+
const unsubscribe = pose.subscribeAccumulatedPose((event) => {
|
|
292
|
+
// event.poseName is the currently selected candidate, or an empty string.
|
|
293
|
+
});
|
|
294
|
+
|
|
295
|
+
await pose.startRecognition();
|
|
296
|
+
```
|
|
297
|
+
|
|
298
|
+
**Lifecycle.** Each composition owns its model registry, active model, camera, and recognition
|
|
299
|
+
state. `releasePoseModel(name)` stops recognition when that model is active and disposes only it;
|
|
300
|
+
`releaseAll()` releases the whole instance, cancels queued and active registration work, removes
|
|
301
|
+
accumulated-pose listeners, and waits for safe cleanup. `subscribeAccumulatedPose()` returns an
|
|
302
|
+
idempotent unsubscribe. Camera capture is an independent lifecycle: only `stopCamera()` and
|
|
303
|
+
`releaseAll()` stop the stream. `File` objects live only for the pending `loadFromFiles()` call and
|
|
304
|
+
are never stored in the registry.
|
|
305
|
+
|
|
306
|
+
**Model initialization policy.** The default `legacy` policy keeps every registration independent.
|
|
307
|
+
`latest-needed` suits a host with one current label-model demand, such as a story that can skip
|
|
308
|
+
scenes: at most one heavy load is active and one newest request pending, and a newer request
|
|
309
|
+
cancels the active one cooperatively and replaces an older request that has not started. For A
|
|
310
|
+
loading, B pending, then C requested, B never reaches the TensorFlow loader. Repeated registrations
|
|
311
|
+
with the same name, byte-identical files, and the same `AbortSignal` share one pending operation;
|
|
312
|
+
different signals stay separate, so cancelling one caller cannot cancel another's work.
|
|
313
|
+
|
|
314
|
+
Pass an `AbortSignal` to `registerPoseModel()` when the host no longer needs that demand. A queued
|
|
315
|
+
request rejects with `AbortError` without invoking the runtime. TensorFlow.js 4.22.0 and Web Crypto
|
|
316
|
+
cannot physically cancel every operation already running, so a running request becomes stale at
|
|
317
|
+
once, starts no later phase, and disposes every late classifier and PoseNet result exactly once
|
|
318
|
+
before its promise settles. A cancelled model is never published in the registry; shared PoseNet
|
|
319
|
+
verification may finish and stay cached. Cancellation never stops the camera.
|
|
320
|
+
|
|
321
|
+
Phases run sequentially by default, with cancellation checked after TensorFlow readiness,
|
|
322
|
+
classifier loading, metadata parsing, and PoseNet loading. Set `parallelModelInitialization: true`
|
|
323
|
+
only when startup latency matters more than those boundaries; it starts the three in parallel, and
|
|
324
|
+
completed stale resources are still disposed before cancellation settles.
|
|
325
|
+
|
|
326
|
+
The composition does not infer scene reachability. A host should retain the signal when the model is
|
|
327
|
+
needed by an imminent action, abort it when a skipped path has no nearby demand, and submit the
|
|
328
|
+
replacement model when a skip changes that demand.
|
|
329
|
+
|
|
330
|
+
**Cameras.** `listCameraDevices()` enumerates afresh on every call and returns a deeply frozen copy
|
|
331
|
+
holding only unique, non-empty video device IDs in enumeration order; labels stay empty until
|
|
332
|
+
permission makes them available. Enumeration failures use `TM-COMPOSITION-012` and never expose
|
|
333
|
+
`MediaDeviceInfo`, `MediaStreamTrack`, or a stream. Select `default`, `front`, or `back` as a
|
|
334
|
+
preference, or wrap a session-only browser ID as `{deviceId}` — the object form is deliberately
|
|
335
|
+
distinct, so a device whose ID is literally `default` is not mistaken for the preference. Selections
|
|
336
|
+
run in call order; a failed switch restores the preceding camera; invalid selections use
|
|
337
|
+
`TM-COMPOSITION-011`. Device IDs are never persisted, and all four camera methods fail closed after
|
|
338
|
+
release.
|
|
339
|
+
|
|
340
|
+
**Preview and overlay.** `setPreviewOpacity()` takes `0`–`1`; `setPreviewPosition()` takes
|
|
341
|
+
`top-left`, `top-right`, `bottom-left`, `bottom-right`, `center`, or `full-stage`;
|
|
342
|
+
`setPreviewMirroring()` takes `mirrored` (the default) or `unmirrored` and may be called before,
|
|
343
|
+
during, or after camera startup. `showPreview()` and `hidePreview()` change visibility only. The
|
|
344
|
+
overlay is enabled directly, with `showPoseOverlay()`, `hidePoseOverlay()`,
|
|
345
|
+
`isPoseOverlayVisible()`, `setPoseJointStyle()`, `setPoseBoneStyle()`,
|
|
346
|
+
`setPoseOverlayMinimumConfidence()`, and `setPoseOverlayConfidenceScaling()`.
|
|
347
|
+
|
|
348
|
+
**Compute.** `selectComputeBackend('webgpu')` resolves to `{backend, fallback}`, where `fallback`
|
|
349
|
+
is `true` when the requested backend was unusable. It is rejected with `TM-COMPOSITION-018` while
|
|
350
|
+
pose models are registered, because their tensors live in the memory of the backend that loaded
|
|
351
|
+
them — release them first.
|
|
352
|
+
|
|
353
|
+
**Disposal shape.** Teachable Machine Pose 0.8.6 exposes the classifier as `CustomPoseNet.model`
|
|
354
|
+
and PoseNet as `CustomPoseNet.posenetModel`, while its top-level `dispose()` releases PoseNet only.
|
|
355
|
+
The composition therefore disconnects an active model from recognition first and disposes those two
|
|
356
|
+
public resources separately and exactly once, rather than calling the incomplete top-level
|
|
357
|
+
disposer. A custom injected runtime without those fields keeps the legacy single-`dispose()`
|
|
358
|
+
contract; a runtime exposing only part of the official shape is rejected with `TM-COMPOSITION-009`
|
|
359
|
+
after every safely identifiable resource has been attempted.
|
|
360
|
+
|
|
361
|
+
**Canvas note.** The extension owns the camera canvas used by Teachable Machine and TensorFlow.js
|
|
362
|
+
and requests a normal Canvas2D context for it. Physical-camera measurements of its 320×240,
|
|
363
|
+
one-draw/one-read path showed no repeatable benefit from `willReadFrequently`, and that hint can
|
|
364
|
+
make video drawing more expensive, so Chrome may emit its Canvas2D readback warning during CPU
|
|
365
|
+
inference. Reconsider the hint only with a reproducible benchmark of this exact path.
|
|
366
|
+
|
|
367
|
+
### Offline PoseNet bundle API
|
|
368
|
+
|
|
369
|
+
`@kubohiroya/turbowarp-teachable-machine/posenet` owns the fixed PoseNet MobileNetV1 0.75 /
|
|
370
|
+
stride 16 manifest, the package asset specifiers, SHA-256 verification, a bounded Base64 project
|
|
371
|
+
descriptor, and the Teachable Machine Pose fetch adapter. A host application only decides where
|
|
372
|
+
that explicit descriptor is stored. The model stays one JSON file and two binary shards; it is
|
|
373
|
+
never disguised as an image, sound, or costume.
|
|
374
|
+
|
|
375
|
+
```js
|
|
376
|
+
import {readFile} from 'node:fs/promises';
|
|
377
|
+
import {
|
|
378
|
+
createBundledTMRuntime,
|
|
379
|
+
createPoseNetProjectBundleFromLoader,
|
|
380
|
+
} from '@kubohiroya/turbowarp-teachable-machine/posenet';
|
|
381
|
+
|
|
382
|
+
const projectBundle = await createPoseNetProjectBundleFromLoader((file) =>
|
|
383
|
+
readFile(new URL(import.meta.resolve(file.packageSpecifier))),
|
|
384
|
+
);
|
|
385
|
+
|
|
386
|
+
// A browser host can store projectBundle in its own project-data channel.
|
|
387
|
+
// Decoding and hash verification start on the first loadFromFiles() call.
|
|
388
|
+
const offlineRuntime = createBundledTMRuntime({runtime: tmPose, projectBundle});
|
|
389
|
+
```
|
|
390
|
+
|
|
391
|
+
The model supply is exactly 5,082,500 bytes. Its three SHA-256 operations run independently, and the
|
|
392
|
+
first shared verification overlaps the classifier load. The intercepted PoseNet fetch exposes no
|
|
393
|
+
bytes until verification succeeds, and later requests reuse the same verified supply. Tampering,
|
|
394
|
+
missing or wrongly sized files, and unexpected PoseNet network requests all fail closed with
|
|
395
|
+
`TM-POSENET-*` error codes.
|
|
540
396
|
|
|
541
397
|
## Troubleshooting
|
|
542
398
|
|
|
543
|
-
Read `last error` first
|
|
544
|
-
|
|
545
|
-
|
|
546
|
-
|
|
547
|
-
|
|
399
|
+
Read `last error` first. The usual causes are denied camera permission, a model *editor* URL instead
|
|
400
|
+
of the published model folder URL, blocked network requests, and a sandboxed load.
|
|
401
|
+
|
|
402
|
+
A sandboxed load shows nothing in the editor: the custom extension dialog simply keeps waiting, and
|
|
403
|
+
only the browser console carries `TM must run without the extension sandbox.` Load the downloaded
|
|
404
|
+
file from the dialog's **Files** tab, never from its **URL** tab.
|
|
548
405
|
|
|
549
406
|
`start camera` also needs the page to have been interacted with at least once. Browsers refuse
|
|
550
|
-
`video.play()`
|
|
551
|
-
|
|
552
|
-
|
|
553
|
-
|
|
554
|
-
|
|
555
|
-
|
|
407
|
+
`video.play()` without user activation, which surfaces as `NotAllowedError: play() failed because
|
|
408
|
+
the user didn't interact with the document first.` Clicking the green flag counts, so an ordinary
|
|
409
|
+
project is fine; a packaged project that starts recognition on its own has to wait for a button
|
|
410
|
+
press first.
|
|
411
|
+
|
|
412
|
+
The guide has step-by-step checks:
|
|
413
|
+
[English](https://kubohiroya.github.io/turbowarp-teachable-machine/#troubleshooting) ·
|
|
414
|
+
[日本語](https://kubohiroya.github.io/turbowarp-teachable-machine/ja/#troubleshooting).
|
|
415
|
+
|
|
416
|
+
## Version notes
|
|
417
|
+
|
|
418
|
+
### Upgrading from 2.x
|
|
419
|
+
|
|
420
|
+
The block surface, the composition API, and the offline PoseNet bundle API are unchanged; a project
|
|
421
|
+
built on 2.x keeps working after swapping the extension file. Nine blocks are new — three for
|
|
422
|
+
compute backends, six for accumulated pose scoring. What changed is the bundled runtime:
|
|
423
|
+
|
|
424
|
+
| | 2.x | 3.0.0 |
|
|
425
|
+
|---|---|---|
|
|
426
|
+
| TensorFlow.js | 1.3.1 | 4.22.0 |
|
|
427
|
+
| Teachable Machine Pose | 0.8.3 | 0.8.6 |
|
|
428
|
+
| TensorFlow.js Speech Commands | 0.4.0 | 0.5.4 |
|
|
429
|
+
| Compute backends | WebGL only | WebGPU, WebGL, WASM, CPU, negotiated at load |
|
|
430
|
+
|
|
431
|
+
That runtime is published as `globalThis.tf`, so a host application reading it directly moves from
|
|
432
|
+
the TensorFlow.js 1.x API to 4.x — the reason this is a major release. Models exported by Teachable
|
|
433
|
+
Machine are unaffected: pose, image, and audio models exported for the 1.x-era libraries load and
|
|
434
|
+
run unchanged on 4.22.0.
|
|
435
|
+
|
|
436
|
+
Two behavioural notes for existing projects:
|
|
437
|
+
|
|
438
|
+
- Without a `set compute mode to` block, recognition now negotiates a backend instead of always
|
|
439
|
+
using WebGL. `active compute backend` reports which one won.
|
|
440
|
+
- The first `start camera` or `load model` downloads a larger runtime than 2.x did (1.24 MB against
|
|
441
|
+
1.04 MB), so plan the startup sequence accordingly.
|
|
442
|
+
|
|
443
|
+
### 3.4.1 — camera leasing
|
|
444
|
+
|
|
445
|
+
In 3.4.0 a leased start threw `Webcam canvas does not provide a 2D context` and recognition never
|
|
446
|
+
began, because the upstream camera helper builds its readback canvas in the same branch that opens
|
|
447
|
+
the device — the branch a supplied element skips. Use 3.4.1 or later wherever Camera Source is
|
|
448
|
+
loaded; 3.4.0 is unaffected only when it is not.
|
|
449
|
+
|
|
450
|
+
### 3.5.0 — preview and crop
|
|
451
|
+
|
|
452
|
+
The preview became the camera's own video rather than a 320×240 readback canvas scaled to fit, so it
|
|
453
|
+
carries the camera's resolution and aspect ratio. The model is fed the whole square centre crop as a
|
|
454
|
+
tensor; the canvas used to hold only its top 240 rows, so a quarter of the subject's height never
|
|
455
|
+
reached the model, and the 4:3 canvas was padded back to square, leaving the subject about 25%
|
|
456
|
+
smaller than the frames the model was trained on. Blocks, opcodes, and the composition API are
|
|
457
|
+
unchanged — but a pose model now sees a subject it was not seeing before, which is the point.
|
|
458
|
+
|
|
459
|
+
### 3.5.1 — image-mode mirroring
|
|
460
|
+
|
|
461
|
+
Image mode handed `predict` the camera's video element without asking for mirroring, so the model
|
|
462
|
+
saw unmirrored pixels while Teachable Machine had trained it on a CSS-mirrored preview. The square
|
|
463
|
+
it sees was already the frame's centre crop; only the mirroring was missing. Pose and image mode now
|
|
464
|
+
feed the same square the same way round. Pose recognition is unchanged.
|
|
556
465
|
|
|
557
466
|
## Blocks
|
|
558
467
|
|
|
@@ -1038,41 +947,44 @@ pnpm install --frozen-lockfile
|
|
|
1038
947
|
pnpm check
|
|
1039
948
|
```
|
|
1040
949
|
|
|
1041
|
-
|
|
950
|
+
`pnpm check` runs type checking, tests, the production build, generated-documentation validation,
|
|
1042
951
|
Pages link validation, distribution reproducibility, and an npm package dry run. The build produces
|
|
1043
|
-
`dist/tm.js`, `dist/composition.js`, `dist/runtime.js`, `dist/posenet.js`, and the three raw
|
|
1044
|
-
|
|
952
|
+
`dist/tm.js`, `dist/composition.js`, `dist/runtime.js`, `dist/posenet.js`, and the three raw PoseNet
|
|
953
|
+
model assets under `dist/posenet/`.
|
|
1045
954
|
|
|
1046
|
-
The
|
|
1047
|
-
|
|
1048
|
-
|
|
955
|
+
The block reference above is generated from `src/block-definitions.json` by `pnpm docs`; edit the
|
|
956
|
+
definitions, not the generated section. The version in `package.json` is the release source of
|
|
957
|
+
truth: the runtime reporter appends `-typescript` to it, and the release consistency check keeps the
|
|
958
|
+
generated bundles, the exact-version README examples, the Pages badges, and the release tag aligned
|
|
959
|
+
with it.
|
|
1049
960
|
|
|
1050
961
|
### npm publishing
|
|
1051
962
|
|
|
1052
|
-
npm publication uses GitHub Actions trusted publishing rather than a long-lived write token. The
|
|
1053
|
-
|
|
1054
|
-
`publish-npm.yml`, for `npm publish`.
|
|
1055
|
-
dispatch **Publish npm package** with that tag
|
|
1056
|
-
|
|
1057
|
-
credential. Roll
|
|
1058
|
-
|
|
963
|
+
npm publication uses GitHub Actions trusted publishing rather than a long-lived write token. The npm
|
|
964
|
+
package settings must trust `kubohiroya/turbowarp-teachable-machine`, workflow file
|
|
965
|
+
`publish-npm.yml`, for `npm publish`. Once the annotated release tag and the GitHub Release exist,
|
|
966
|
+
dispatch **Publish npm package** with that tag: the workflow checks out the tag, verifies that it
|
|
967
|
+
matches `package.json` exactly, runs the complete check, and publishes through a short-lived OIDC
|
|
968
|
+
credential. Roll the automation back by removing the trusted publisher in npm package settings; do
|
|
969
|
+
not add a write token to the repository.
|
|
1059
970
|
|
|
1060
971
|
## External libraries
|
|
1061
972
|
|
|
1062
973
|
The version-pinned `dist/runtime.js` artifact bundles TensorFlow.js 4.22.0, Teachable Machine Pose
|
|
1063
|
-
0.8.6, Teachable Machine Image 0.8.5, TensorFlow.js Speech Commands 0.5.4, and
|
|
1064
|
-
|
|
1065
|
-
|
|
1066
|
-
|
|
1067
|
-
|
|
1068
|
-
|
|
1069
|
-
original binary files.
|
|
974
|
+
0.8.6, Teachable Machine Image 0.8.5, TensorFlow.js Speech Commands 0.5.4, and PoseNet 2.2.2 in one
|
|
975
|
+
reviewed module graph. The standalone extension loads that single artifact from jsDelivr when no
|
|
976
|
+
runtime is injected. The optional `dist/backend-wasm.js` and `dist/backend-webgpu.js` bundles
|
|
977
|
+
resolve TensorFlow.js against the instance that runtime already published, so a page never ends up
|
|
978
|
+
with two TensorFlow.js cores. The WebAssembly binaries under `dist/wasm/` and the fixed offline
|
|
979
|
+
PoseNet model data under `dist/posenet/` are distributed as their original binary files.
|
|
1070
980
|
|
|
1071
|
-
|
|
981
|
+
Third-party licences are listed in [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).
|
|
1072
982
|
|
|
1073
|
-
|
|
983
|
+
## Fixed input editors
|
|
1074
984
|
|
|
985
|
+
The opt-in fixed input forms, and the rules that keep existing data intact, are described in
|
|
986
|
+
[the fixed input editor contract](docs/fixed-input-editors.ja.md) (Japanese).
|
|
1075
987
|
|
|
1076
|
-
##
|
|
988
|
+
## License
|
|
1077
989
|
|
|
1078
|
-
|
|
990
|
+
MPL-2.0
|