@kubohiroya/turbowarp-camera-source 0.12.0 → 0.14.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.ja.md +36 -2
- package/README.md +41 -5
- package/dist/camera-source.js +118 -4
- package/dist/runtime.d.ts +59 -3
- package/dist/runtime.d.ts.map +1 -1
- package/dist/runtime.js.map +1 -1
- package/package.json +2 -1
- package/schemas/extension-manifest.schema.json +0 -49
package/README.ja.md
CHANGED
|
@@ -26,13 +26,13 @@ TurboWarp-Camera-Sourceは、MediaDevicesのカメラストリームを名前付
|
|
|
26
26
|
次のURLをunsandboxed custom extensionとして読み込みます。
|
|
27
27
|
|
|
28
28
|
```text
|
|
29
|
-
https://cdn.jsdelivr.net/npm/@kubohiroya/turbowarp-camera-source@0.
|
|
29
|
+
https://cdn.jsdelivr.net/npm/@kubohiroya/turbowarp-camera-source@0.14.0/dist/camera-source.js
|
|
30
30
|
```
|
|
31
31
|
|
|
32
32
|
npm hostでは次を使います。
|
|
33
33
|
|
|
34
34
|
```bash
|
|
35
|
-
pnpm add @kubohiroya/turbowarp-camera-source@0.
|
|
35
|
+
pnpm add @kubohiroya/turbowarp-camera-source@0.14.0
|
|
36
36
|
```
|
|
37
37
|
|
|
38
38
|
## Quick start
|
|
@@ -122,6 +122,32 @@ await qrLease.release();
|
|
|
122
122
|
await poseLease.release();
|
|
123
123
|
```
|
|
124
124
|
|
|
125
|
+
consumerが`MediaStream`を既に持っている場合は`adoptCamera()`を使います。`<usermedia>` capability
|
|
126
|
+
elementで取得したもの、別フレームから渡されたもの、テスト用の作りものでも同じです。Camera Sourceは
|
|
127
|
+
それを`cameraId`のもとで共有し、どのconsumerも同じ`getFrameSource()`から読みます。
|
|
128
|
+
|
|
129
|
+
**採用したstreamのtrackをCamera Sourceは止めません。** 最後のleaseを解放するとセッションは終わり、
|
|
130
|
+
streamはdetachされます。カメラを使い終えたときに止めるのは呼び出し側です。trackの停止は取り消せず、
|
|
131
|
+
同じstreamを見ているほかのconsumer全員からフレームを奪うので、それが正しいときを知りうるのは
|
|
132
|
+
streamを開いた側だけです。自分で取得したstreamは従来どおりCamera Sourceが止めます。
|
|
133
|
+
|
|
134
|
+
```js
|
|
135
|
+
if (typeof cameraSource.adoptCamera === "function") {
|
|
136
|
+
const hostLease = await cameraSource.adoptCamera({
|
|
137
|
+
owner: "app-shell",
|
|
138
|
+
cameraId: "pose",
|
|
139
|
+
stream: streamTheHostAlreadyHolds,
|
|
140
|
+
});
|
|
141
|
+
const frame = hostLease.getFrameSource();
|
|
142
|
+
await hostLease.release();
|
|
143
|
+
}
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
このメソッドはfeature detectしてください。このsurfaceはversionではなく形で絞られるため、現行の宣言に
|
|
147
|
+
対してコンパイルしたconsumerが、それより前のビルドに出会うことがあります。既に別のstreamが動いている
|
|
148
|
+
`cameraId`へ2つめを採用しようとすると拒否します。呼び出し側に他人のフレームを返すと、自分のstreamが
|
|
149
|
+
使われていると誤解させるためです。同じstreamをもう一度採用した場合はそのセッションに参加します。
|
|
150
|
+
|
|
125
151
|
`preview: true`を指定したleaseは、GPU-backed stage previewを利用します。専用video skinは共有
|
|
126
152
|
`HTMLVideoElement`を`texImage2D(video)`でアップロードし、preview drawableのX scaleで左右反転します。
|
|
127
153
|
`drawImage()`、`getImageData()`、CPUでのフレームピクセル走査は行いません。既定値は`false`で、同じ
|
|
@@ -159,6 +185,8 @@ Scratchのtouchingまたはcolor sensing用の画像sourceとしては扱いま
|
|
|
159
185
|
|
|
160
186
|
**プロファイルを自分でスケールせず、内部行列を要求すること。** 校正時と違う解像度で届く場合、必要な計算は何が起きたかで変わる。単純な縮小なら`fx`・`fy`・`cx`・`cy`を比率倍するが、cropなら焦点距離はそのままで主点が平行移動する。trackの`resizeMode`を見られるのは本拡張だけなので、両者を区別できるのも本拡張だけ。**利用側がそれぞれ推測すれば、それぞれ違う推測をし、同じカメラがどの拡張から尋ねたかで違う幾何を返すことになる。** `camera intrinsics JSON`は現在のフレームに適合済みの数値を返し、cropやアスペクト変更で主点を置けない場合は空文字を返す。
|
|
161
187
|
|
|
188
|
+
**フレームの撮影時刻。** カメラがフレームを1枚表示すると、`getFrameSource()`は`frameTime`も返す。`timestampUs`はページの単調時計でのUnix epochからのマイクロ秒、`source`は`requestVideoFrameCallback`の`captureTime`なら`capture`、ブラウザが`presentationTime`しか報告しなければ`presentation`、`presentedFrames`は使用済みのフレームと新しいフレームを見分けるための番号である。同じページのカメラは同じ時計を共有するので、フレームをそのまま並べて比べられる。ほかのコンピュータとは同期していない。時刻はframe sourceを取得した時点で表示されていたフレームのものなので、画素を読む直前に取得すること。無いときは不明であって、現在時刻ではない。
|
|
189
|
+
|
|
162
190
|
**previewを反転してもフレームは反転しない。** `getFrameSource()`が返す`previewFlip`は、stageがどう描いているかだけを述べる。その裏にあるフレームは常にカメラが撮ったままである。**反転表示したpreview上で拾った座標は、これらのフレームと使う前に戻さなければならない** — previewの座標をそのままsolveへ渡すと、左右反転した姿勢に収束し、しかも再投影誤差は小さいまま出る。`horizontal`は左右反転で、`cv::flip(…, 1)`・ffmpegの`hflip`・CSSの`scaleX(-1)`と同じ。回転は別の関心事なので、同じenumには混ぜていない。
|
|
163
191
|
|
|
164
192
|
**契約は書き写さずimportすること。** 型は専用のentry pointとして公開してある。
|
|
@@ -251,6 +279,12 @@ if <(stored camera profile result for [pose]) = [restored]> then
|
|
|
251
279
|
|
|
252
280
|
## 互換性
|
|
253
281
|
|
|
282
|
+
0.14.0では呼び出し側が持っているstreamを共有する`adoptCamera`と、`./runtime`に`CameraAdoptOptions`・
|
|
283
|
+
`CameraLeaseOptions`の宣言を追加した。`CameraAcquireOptions`のメンバーは変えていない。既存のブロックと
|
|
284
|
+
capabilityのメンバーは変えていない。
|
|
285
|
+
|
|
286
|
+
0.13.0ではframe sourceに`frameTime`を追加した。既存のブロックとcapabilityのメンバーは変えていない。
|
|
287
|
+
|
|
254
288
|
0.12.0ではデバイスで絞る復元、`bind camera profile ... to its current device`、`camera profile ... belongs to its current device?`、`start shared camera [CAMERA_ID] requested by this page`を追加した。既存のブロックとcapabilityのメンバーは変えていない。
|
|
255
289
|
|
|
256
290
|
0.11.0では校正ファイルをROSの`camera_info` YAMLで読み書きするようにした。`camera profile YAML`を追加し、`register camera profile`はYAMLも受け取り、処理だけを持つ関数を`./profile`で公開した。既存のブロックとcapabilityのメンバーは変えていない。
|
package/README.md
CHANGED
|
@@ -30,13 +30,13 @@ or select separate cameras for separate roles.
|
|
|
30
30
|
Load this URL as an unsandboxed custom extension:
|
|
31
31
|
|
|
32
32
|
```text
|
|
33
|
-
https://cdn.jsdelivr.net/npm/@kubohiroya/turbowarp-camera-source@0.
|
|
33
|
+
https://cdn.jsdelivr.net/npm/@kubohiroya/turbowarp-camera-source@0.14.0/dist/camera-source.js
|
|
34
34
|
```
|
|
35
35
|
|
|
36
36
|
For npm hosts:
|
|
37
37
|
|
|
38
38
|
```bash
|
|
39
|
-
pnpm add @kubohiroya/turbowarp-camera-source@0.
|
|
39
|
+
pnpm add @kubohiroya/turbowarp-camera-source@0.14.0
|
|
40
40
|
```
|
|
41
41
|
|
|
42
42
|
## Quick Start
|
|
@@ -469,6 +469,35 @@ await qrLease.release();
|
|
|
469
469
|
await poseLease.release();
|
|
470
470
|
```
|
|
471
471
|
|
|
472
|
+
Use `adoptCamera()` instead when the consumer already holds a `MediaStream` — one obtained
|
|
473
|
+
through the `<usermedia>` capability element, from another frame, or from a test fixture.
|
|
474
|
+
Camera Source shares it under a `cameraId` exactly as an acquired stream, and every consumer
|
|
475
|
+
reads it through the same `getFrameSource()`.
|
|
476
|
+
|
|
477
|
+
**Camera Source never stops the tracks of an adopted stream.** Releasing the last lease ends
|
|
478
|
+
the session and detaches the stream, leaving the caller to stop it when it is finished with the
|
|
479
|
+
camera. Stopping a track cannot be undone and takes the frames away from every other consumer
|
|
480
|
+
of the same stream, so the side that opened it is the only side that can know when that is
|
|
481
|
+
right. An acquired stream is still stopped by Camera Source as before.
|
|
482
|
+
|
|
483
|
+
```js
|
|
484
|
+
if (typeof cameraSource.adoptCamera === "function") {
|
|
485
|
+
const hostLease = await cameraSource.adoptCamera({
|
|
486
|
+
owner: "app-shell",
|
|
487
|
+
cameraId: "pose",
|
|
488
|
+
stream: streamTheHostAlreadyHolds,
|
|
489
|
+
});
|
|
490
|
+
const frame = hostLease.getFrameSource();
|
|
491
|
+
await hostLease.release();
|
|
492
|
+
}
|
|
493
|
+
```
|
|
494
|
+
|
|
495
|
+
Feature detect the method. This surface is narrowed by shape rather than by a version, so a
|
|
496
|
+
consumer compiled against the current declarations can still meet a build that predates it.
|
|
497
|
+
Adopting a second, different stream under a `cameraId` that already runs one is refused:
|
|
498
|
+
answering the caller with somebody else's frames would leave it believing its own stream is in
|
|
499
|
+
use. Adopting the same stream again joins the session.
|
|
500
|
+
|
|
472
501
|
`preview: true` opts that lease into the GPU-backed stage preview. The dedicated video skin uploads
|
|
473
502
|
the shared `HTMLVideoElement` with `texImage2D(video)` and mirrors by changing the preview drawable's
|
|
474
503
|
X scale. It does not call `drawImage()`, `getImageData()`, or scan frame pixels on the CPU. The option
|
|
@@ -515,6 +544,8 @@ const assessment = calibration.assessProfile("pose");
|
|
|
515
544
|
|
|
516
545
|
**Ask for intrinsics rather than scaling a profile yourself.** When the camera delivers a size the calibration was not solved at, the arithmetic depends on what happened: a pure downscale multiplies `fx`, `fy`, `cx` and `cy`, while a crop leaves the focal lengths alone and shifts the principal point instead. Only this extension sees the track's `resizeMode`, so only it can tell the two apart — and if every consumer guesses, they guess differently and the same camera yields different geometry depending on which extension asked. `camera intrinsics JSON` returns numbers already adapted to the current frame, or an empty string when the difference is a crop or an aspect change and the principal point cannot be placed.
|
|
517
546
|
|
|
547
|
+
**Frame capture time.** `getFrameSource()` also carries `frameTime` once the camera has presented a frame: `timestampUs` in microseconds since the Unix epoch on the page's monotonic clock, `source` (`capture` for `captureTime` from `requestVideoFrameCallback`, `presentation` where the browser reports only `presentationTime`), and `presentedFrames`, which tells a frame already used from a new one. Cameras on one page share that clock, so their frames can be lined up directly; it is not synchronized with any other computer. Take the frame source immediately before reading pixels: the time belongs to the frame presented when the source was taken. Absent means unknown, never now.
|
|
548
|
+
|
|
518
549
|
**Turning the preview over does not turn the frames over.** `getFrameSource()` reports `previewFlip`, which describes how the stage is drawing the image and nothing else; the frames behind it are always the ones the camera captured. Coordinates picked off a flipped preview must be turned back before they are used with these frames — a solve fed the preview's coordinates converges on a left-right reflected pose and reports a small reprojection error while doing it. `horizontal` is the left-right mirror, matching `cv::flip(…, 1)`, ffmpeg's `hflip` and CSS `scaleX(-1)`; rotation is a separate concern and is deliberately not folded into the same enum.
|
|
519
550
|
|
|
520
551
|
**Import these declarations rather than re-writing them.** The contract is published as its own
|
|
@@ -649,6 +680,12 @@ profile), and save it. An app that opens the calibration app for one camera name
|
|
|
649
680
|
|
|
650
681
|
## Compatibility
|
|
651
682
|
|
|
683
|
+
Version 0.14.0 adds `adoptCamera` for sharing a stream the caller already holds, and the
|
|
684
|
+
`CameraAdoptOptions` and `CameraLeaseOptions` declarations to `./runtime`. `CameraAcquireOptions`
|
|
685
|
+
keeps every member it had. No existing block or capability member changes.
|
|
686
|
+
|
|
687
|
+
Version 0.13.0 adds `frameTime` to the frame source. No existing block or capability member changes.
|
|
688
|
+
|
|
652
689
|
Version 0.12.0 adds device-scoped restore, `bind camera profile ... to its current device`,
|
|
653
690
|
`camera profile ... belongs to its current device?` and `start shared camera [CAMERA_ID] requested by this page`.
|
|
654
691
|
No existing block or capability member changes.
|
|
@@ -712,10 +749,9 @@ The generated JavaScript is a single, non-minified TurboWarp extension file with
|
|
|
712
749
|
- `src/block-definitions.json`: canonical block metadata used by both the extension and README generator
|
|
713
750
|
- `src/extension.ts`: extension implementation
|
|
714
751
|
- `src/video-preview.ts`: GPU-backed `HTMLVideoElement` preview skin and drawable lifecycle
|
|
715
|
-
- `src/extension-manifest.ts`: canonical manifest generator and Vite output plugin
|
|
716
752
|
- `src/index.ts`: extension registration entry point
|
|
717
753
|
- `src/globals.d.ts`: Scratch API declarations used by the project
|
|
718
|
-
-
|
|
754
|
+
- the manifest generator and its JSON Schema come from [`@kubohiroya/turbowarp-extension-manifest`](https://github.com/kubohiroya/turbowarp-extension-manifest)
|
|
719
755
|
- `scripts/generate-readme.ts`: updates the generated README block section
|
|
720
756
|
- `tests/`: unit tests
|
|
721
757
|
- `vite.config.ts`: TurboWarp-compatible Vite build configuration
|
|
@@ -727,7 +763,7 @@ Each build emits `dist/extension-manifest.json` with `formatVersion: 1`. It reco
|
|
|
727
763
|
block opcodes and types, argument IDs and types, and menu references in a deterministic order. Tools
|
|
728
764
|
such as `sb3-toolchain` can compare this contract before updating an embedded extension or migrating
|
|
729
765
|
its ID. See [the architecture document](docs/architecture.md) and the
|
|
730
|
-
[JSON Schema](schemas/extension-manifest.schema.json) for the v1 contract.
|
|
766
|
+
[JSON Schema](https://github.com/kubohiroya/turbowarp-extension-manifest/blob/main/schemas/extension-manifest.schema.json) for the v1 contract.
|
|
731
767
|
|
|
732
768
|
After changing runtime or block metadata, regenerate and verify the tracked release artifacts:
|
|
733
769
|
|
package/dist/camera-source.js
CHANGED
|
@@ -2236,6 +2236,55 @@
|
|
|
2236
2236
|
});
|
|
2237
2237
|
}
|
|
2238
2238
|
//#endregion
|
|
2239
|
+
//#region src/frame-time.ts
|
|
2240
|
+
function finite(value) {
|
|
2241
|
+
return typeof value === "number" && Number.isFinite(value) ? value : void 0;
|
|
2242
|
+
}
|
|
2243
|
+
/** The time of one presented frame, or the previous one when the metadata carries no time. */
|
|
2244
|
+
function frameTimeFrom(metadata, timeOriginMs, previous) {
|
|
2245
|
+
const capture = finite(metadata.captureTime);
|
|
2246
|
+
const presentation = finite(metadata.presentationTime);
|
|
2247
|
+
const milliseconds = capture ?? presentation;
|
|
2248
|
+
if (milliseconds === void 0) return previous;
|
|
2249
|
+
return Object.freeze({
|
|
2250
|
+
timestampUs: Math.round((timeOriginMs + milliseconds) * 1e3),
|
|
2251
|
+
source: capture === void 0 ? "presentation" : "capture",
|
|
2252
|
+
presentedFrames: finite(metadata.presentedFrames) ?? (previous?.presentedFrames ?? 0) + 1
|
|
2253
|
+
});
|
|
2254
|
+
}
|
|
2255
|
+
/**
|
|
2256
|
+
* Follows one video element for as long as its camera runs.
|
|
2257
|
+
*
|
|
2258
|
+
* Nothing is recorded where the browser has no `requestVideoFrameCallback`; frame sources then carry
|
|
2259
|
+
* no time, which a consumer has to treat as unknown rather than as now.
|
|
2260
|
+
*/
|
|
2261
|
+
var FrameTimeWatcher = class {
|
|
2262
|
+
constructor(element, timeOriginMs) {
|
|
2263
|
+
this.disposed = false;
|
|
2264
|
+
this.element = element;
|
|
2265
|
+
this.timeOriginMs = timeOriginMs;
|
|
2266
|
+
this.schedule();
|
|
2267
|
+
}
|
|
2268
|
+
latest() {
|
|
2269
|
+
return this.frame;
|
|
2270
|
+
}
|
|
2271
|
+
dispose() {
|
|
2272
|
+
this.disposed = true;
|
|
2273
|
+
if (this.handle !== void 0) {
|
|
2274
|
+
this.element.cancelVideoFrameCallback(this.handle);
|
|
2275
|
+
this.handle = void 0;
|
|
2276
|
+
}
|
|
2277
|
+
}
|
|
2278
|
+
schedule() {
|
|
2279
|
+
if (this.disposed || typeof this.element.requestVideoFrameCallback !== "function") return;
|
|
2280
|
+
this.handle = this.element.requestVideoFrameCallback((_now, metadata) => {
|
|
2281
|
+
this.handle = void 0;
|
|
2282
|
+
this.frame = frameTimeFrom(metadata, this.timeOriginMs, this.frame);
|
|
2283
|
+
this.schedule();
|
|
2284
|
+
});
|
|
2285
|
+
}
|
|
2286
|
+
};
|
|
2287
|
+
//#endregion
|
|
2239
2288
|
//#region src/extension.ts
|
|
2240
2289
|
var blockDefinitions = block_definitions_default.blocks;
|
|
2241
2290
|
var menuDefinitions = block_definitions_default.menus;
|
|
@@ -2469,9 +2518,31 @@
|
|
|
2469
2518
|
async acquireCamera(options = {}) {
|
|
2470
2519
|
const cameraId = normalizeId(options.cameraId);
|
|
2471
2520
|
const session = this.session(cameraId);
|
|
2472
|
-
const token = Symbol(String(options.owner ?? "camera-lease"));
|
|
2473
2521
|
if (session.startPromise) await session.startPromise;
|
|
2474
2522
|
else if (!session.stream) await this.start(session, options);
|
|
2523
|
+
return this.lease(session, options);
|
|
2524
|
+
}
|
|
2525
|
+
/**
|
|
2526
|
+
* Shares a stream the caller already holds, under a camera id, without opening a camera.
|
|
2527
|
+
*
|
|
2528
|
+
* Refuses when the id already runs a different stream rather than quietly joining it: the caller
|
|
2529
|
+
* asked for its own stream to be shared, and answering with somebody else's would leave it
|
|
2530
|
+
* believing a stream is in use that nobody reads. Adopting the same stream twice is the one case
|
|
2531
|
+
* that joins, because that is what the caller asked for and already got.
|
|
2532
|
+
*/
|
|
2533
|
+
async adoptCamera(options) {
|
|
2534
|
+
const stream = options?.stream;
|
|
2535
|
+
if (typeof stream !== "object" || stream === null || typeof stream.getTracks !== "function") throw new TypeError("Camera Source adoptCamera requires a MediaStream.");
|
|
2536
|
+
const cameraId = normalizeId(options.cameraId);
|
|
2537
|
+
const session = this.session(cameraId);
|
|
2538
|
+
if (session.startPromise) await session.startPromise;
|
|
2539
|
+
if (session.stream && session.stream !== stream) throw new Error(`Camera id ${cameraId} already runs another stream. Release its leases before adopting.`);
|
|
2540
|
+
if (!session.stream) await this.adopt(session, stream);
|
|
2541
|
+
return this.lease(session, options);
|
|
2542
|
+
}
|
|
2543
|
+
lease(session, options) {
|
|
2544
|
+
const cameraId = session.cameraId;
|
|
2545
|
+
const token = Symbol(String(options.owner ?? "camera-lease"));
|
|
2475
2546
|
session.leases.add(token);
|
|
2476
2547
|
try {
|
|
2477
2548
|
if (options.preview === true) {
|
|
@@ -2959,11 +3030,49 @@
|
|
|
2959
3030
|
video: null,
|
|
2960
3031
|
preview: null,
|
|
2961
3032
|
startPromise: null,
|
|
2962
|
-
activeDeviceId: ""
|
|
3033
|
+
activeDeviceId: "",
|
|
3034
|
+
frameTimes: null,
|
|
3035
|
+
adopted: false
|
|
2963
3036
|
};
|
|
2964
3037
|
this.sessions.set(cameraId, session);
|
|
2965
3038
|
return session;
|
|
2966
3039
|
}
|
|
3040
|
+
/**
|
|
3041
|
+
* Brings a caller's stream into a session, taking the same path `start` takes after acquisition.
|
|
3042
|
+
*
|
|
3043
|
+
* `adopted` is set before anything can fail, so no failure path can reach the tracks.
|
|
3044
|
+
*/
|
|
3045
|
+
async adopt(session, stream) {
|
|
3046
|
+
session.adopted = true;
|
|
3047
|
+
session.startPromise = (async () => {
|
|
3048
|
+
let video = null;
|
|
3049
|
+
try {
|
|
3050
|
+
if (!session.active) throw new Error("Camera adoption was cancelled.");
|
|
3051
|
+
video = document.createElement("video");
|
|
3052
|
+
video.muted = true;
|
|
3053
|
+
video.playsInline = true;
|
|
3054
|
+
video.srcObject = stream;
|
|
3055
|
+
await video.play();
|
|
3056
|
+
if (!session.active) throw new Error("Camera adoption was cancelled.");
|
|
3057
|
+
session.stream = stream;
|
|
3058
|
+
session.video = video;
|
|
3059
|
+
session.frameTimes = new FrameTimeWatcher(video, globalThis.performance?.timeOrigin ?? 0);
|
|
3060
|
+
this.watchStreamEnd(session, stream);
|
|
3061
|
+
this.updateActiveDevice(session);
|
|
3062
|
+
this.cameraFailures.delete(session.cameraId);
|
|
3063
|
+
} catch (error) {
|
|
3064
|
+
if (video) video.srcObject = null;
|
|
3065
|
+
throw error;
|
|
3066
|
+
}
|
|
3067
|
+
})();
|
|
3068
|
+
try {
|
|
3069
|
+
await session.startPromise;
|
|
3070
|
+
} catch (error) {
|
|
3071
|
+
if (this.sessions.get(session.cameraId) === session) this.stopCameraSession(session.cameraId);
|
|
3072
|
+
this.cameraFailures.set(session.cameraId, cameraFailure(error));
|
|
3073
|
+
throw error;
|
|
3074
|
+
}
|
|
3075
|
+
}
|
|
2967
3076
|
async start(session, options) {
|
|
2968
3077
|
session.startPromise = (async () => {
|
|
2969
3078
|
const stream = await mediaDevices().getUserMedia(videoConstraints(options));
|
|
@@ -2978,6 +3087,7 @@
|
|
|
2978
3087
|
if (!session.active) throw new Error("Camera acquisition was cancelled.");
|
|
2979
3088
|
session.stream = stream;
|
|
2980
3089
|
session.video = video;
|
|
3090
|
+
session.frameTimes = new FrameTimeWatcher(video, globalThis.performance?.timeOrigin ?? 0);
|
|
2981
3091
|
this.watchStreamEnd(session, stream);
|
|
2982
3092
|
this.updateActiveDevice(session);
|
|
2983
3093
|
this.cameraFailures.delete(session.cameraId);
|
|
@@ -3035,13 +3145,15 @@
|
|
|
3035
3145
|
}
|
|
3036
3146
|
getFrameSource(session) {
|
|
3037
3147
|
if (!session.video || !session.stream) throw new Error("Shared camera is not running.");
|
|
3148
|
+
const frameTime = session.frameTimes?.latest();
|
|
3038
3149
|
return Object.freeze({
|
|
3039
3150
|
kind: "video",
|
|
3040
3151
|
element: session.video,
|
|
3041
3152
|
width: session.video.videoWidth,
|
|
3042
3153
|
height: session.video.videoHeight,
|
|
3043
3154
|
previewFlip: this.previewFlip(session),
|
|
3044
|
-
deviceId: session.activeDeviceId
|
|
3155
|
+
deviceId: session.activeDeviceId,
|
|
3156
|
+
...frameTime === void 0 ? {} : { frameTime }
|
|
3045
3157
|
});
|
|
3046
3158
|
}
|
|
3047
3159
|
stopWhenUnused(session) {
|
|
@@ -3054,7 +3166,9 @@
|
|
|
3054
3166
|
session.active = false;
|
|
3055
3167
|
this.nextPreviewBlockRevision(cameraId);
|
|
3056
3168
|
session.preview?.dispose();
|
|
3057
|
-
session.
|
|
3169
|
+
session.frameTimes?.dispose();
|
|
3170
|
+
session.frameTimes = null;
|
|
3171
|
+
if (!session.adopted) session.stream?.getTracks().forEach((track) => track.stop());
|
|
3058
3172
|
if (session.video) session.video.srcObject = null;
|
|
3059
3173
|
session.stream = null;
|
|
3060
3174
|
session.video = null;
|
package/dist/runtime.d.ts
CHANGED
|
@@ -38,17 +38,37 @@ export declare const cameraSourceCapabilityVersion: 1;
|
|
|
38
38
|
* rotation and not a flip, and once both exist their order matters.
|
|
39
39
|
*/
|
|
40
40
|
export type Flip = 'none' | 'horizontal' | 'vertical' | 'both';
|
|
41
|
-
|
|
41
|
+
/** What taking a lease needs, however the stream behind it was obtained. */
|
|
42
|
+
export interface CameraLeaseOptions {
|
|
42
43
|
/** Free text naming the caller. Appears in diagnostics; never used to decide anything. */
|
|
43
44
|
owner?: string;
|
|
44
45
|
/** The role name of the stream, such as `pose` or `qr`. Callers naming the same one share it. */
|
|
45
46
|
cameraId?: string;
|
|
46
|
-
deviceId?: string;
|
|
47
|
-
video?: MediaTrackConstraints | boolean;
|
|
48
47
|
/** How this consumer wants the stage preview drawn. Only meaningful together with `preview`. */
|
|
49
48
|
previewFlip?: Flip;
|
|
50
49
|
preview?: boolean;
|
|
51
50
|
}
|
|
51
|
+
export interface CameraAcquireOptions extends CameraLeaseOptions {
|
|
52
|
+
deviceId?: string;
|
|
53
|
+
video?: MediaTrackConstraints | boolean;
|
|
54
|
+
}
|
|
55
|
+
/**
|
|
56
|
+
* What adopting a stream the caller already holds needs.
|
|
57
|
+
*
|
|
58
|
+
* No `deviceId` or `video`: the stream exists, so there is nothing left to constrain. Which device
|
|
59
|
+
* it came from is readable from the track, and `CameraFrameSource.deviceId` reports it.
|
|
60
|
+
*/
|
|
61
|
+
export interface CameraAdoptOptions extends CameraLeaseOptions {
|
|
62
|
+
/**
|
|
63
|
+
* The stream to share. Camera Source never stops its tracks.
|
|
64
|
+
*
|
|
65
|
+
* Releasing the last lease ends the session and detaches the stream, leaving the caller to stop
|
|
66
|
+
* it when it is finished with the camera. Stopping a track cannot be undone and takes the frames
|
|
67
|
+
* away from every other consumer of the same stream, so the side that opened it is the only side
|
|
68
|
+
* that can know when that is right.
|
|
69
|
+
*/
|
|
70
|
+
stream: MediaStream;
|
|
71
|
+
}
|
|
52
72
|
export interface CameraFrameSource {
|
|
53
73
|
readonly kind: 'video';
|
|
54
74
|
readonly element: HTMLVideoElement;
|
|
@@ -66,6 +86,33 @@ export interface CameraFrameSource {
|
|
|
66
86
|
*/
|
|
67
87
|
readonly previewFlip: Flip;
|
|
68
88
|
readonly deviceId: string;
|
|
89
|
+
/**
|
|
90
|
+
* When the frame the element is showing was captured, as of the moment this source was taken.
|
|
91
|
+
*
|
|
92
|
+
* Absent until the camera has presented a frame, and where the browser has no
|
|
93
|
+
* `requestVideoFrameCallback`; absent means unknown, never now. The element can present a newer
|
|
94
|
+
* frame between taking the source and reading pixels from it, so the time belongs to the frame
|
|
95
|
+
* presented when the source was taken — take the source immediately before reading pixels.
|
|
96
|
+
*/
|
|
97
|
+
readonly frameTime?: CameraFrameTime;
|
|
98
|
+
}
|
|
99
|
+
/**
|
|
100
|
+
* The capture time of a presented frame.
|
|
101
|
+
*
|
|
102
|
+
* `timestampUs` is microseconds since the Unix epoch on the page's monotonic clock
|
|
103
|
+
* (`performance.timeOrigin + time`). Cameras on one page share that clock, so their frames can be
|
|
104
|
+
* compared directly; it is not synchronized with any other computer.
|
|
105
|
+
*/
|
|
106
|
+
export interface CameraFrameTime {
|
|
107
|
+
readonly timestampUs: number;
|
|
108
|
+
/**
|
|
109
|
+
* `capture` is when the frame left the device (`captureTime`). `presentation` is when the browser
|
|
110
|
+
* presented it (`presentationTime`), later by the capture and decode pipeline, and used only where
|
|
111
|
+
* the browser reports no capture time.
|
|
112
|
+
*/
|
|
113
|
+
readonly source: 'capture' | 'presentation';
|
|
114
|
+
/** Increases with every presented frame, so a frame already used can be recognized. */
|
|
115
|
+
readonly presentedFrames: number;
|
|
69
116
|
}
|
|
70
117
|
export interface CameraLease {
|
|
71
118
|
getFrameSource(): CameraFrameSource;
|
|
@@ -79,6 +126,15 @@ export interface CameraLease {
|
|
|
79
126
|
*/
|
|
80
127
|
export interface CameraSourceRuntime {
|
|
81
128
|
acquireCamera(options?: CameraAcquireOptions): Promise<CameraLease>;
|
|
129
|
+
/**
|
|
130
|
+
* Shares a stream the caller already obtained, instead of opening a camera.
|
|
131
|
+
*
|
|
132
|
+
* Optional because this surface is narrowed by shape rather than by a version, so a consumer
|
|
133
|
+
* compiled against these declarations can still meet a build that predates the method. Feature
|
|
134
|
+
* detect it: `typeof runtime.adoptCamera === 'function'`. A caller reaching for it is already
|
|
135
|
+
* feature detecting whatever produced the stream.
|
|
136
|
+
*/
|
|
137
|
+
adoptCamera?(options: CameraAdoptOptions): Promise<CameraLease>;
|
|
82
138
|
}
|
|
83
139
|
/**
|
|
84
140
|
* The profile contract, re-exported so a consumer states it rather than restating it.
|
package/dist/runtime.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"runtime.d.ts","sourceRoot":"","sources":["../src/runtime.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;GAWG;AAEH,0FAA0F;AAC1F,eAAO,MAAM,uBAAuB,2BAA2B,CAAC;AAEhE;;;;;GAKG;AACH,eAAO,MAAM,sBAAsB,+BAA+B,CAAC;AAEnE;;;;;;;GAOG;AACH,eAAO,MAAM,yBAAyB,qCAAqC,CAAC;AAE5E,wFAAwF;AACxF,eAAO,MAAM,6BAA6B,EAAG,CAAU,CAAC;AAExD;;;;;;GAMG;AACH,MAAM,MAAM,IAAI,GAAG,MAAM,GAAG,YAAY,GAAG,UAAU,GAAG,MAAM,CAAC;AAE/D,MAAM,WAAW,
|
|
1
|
+
{"version":3,"file":"runtime.d.ts","sourceRoot":"","sources":["../src/runtime.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;GAWG;AAEH,0FAA0F;AAC1F,eAAO,MAAM,uBAAuB,2BAA2B,CAAC;AAEhE;;;;;GAKG;AACH,eAAO,MAAM,sBAAsB,+BAA+B,CAAC;AAEnE;;;;;;;GAOG;AACH,eAAO,MAAM,yBAAyB,qCAAqC,CAAC;AAE5E,wFAAwF;AACxF,eAAO,MAAM,6BAA6B,EAAG,CAAU,CAAC;AAExD;;;;;;GAMG;AACH,MAAM,MAAM,IAAI,GAAG,MAAM,GAAG,YAAY,GAAG,UAAU,GAAG,MAAM,CAAC;AAE/D,4EAA4E;AAC5E,MAAM,WAAW,kBAAkB;IACjC,0FAA0F;IAC1F,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,iGAAiG;IACjG,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,gGAAgG;IAChG,WAAW,CAAC,EAAE,IAAI,CAAC;IACnB,OAAO,CAAC,EAAE,OAAO,CAAC;CACnB;AAED,MAAM,WAAW,oBAAqB,SAAQ,kBAAkB;IAC9D,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,KAAK,CAAC,EAAE,qBAAqB,GAAG,OAAO,CAAC;CACzC;AAED;;;;;GAKG;AACH,MAAM,WAAW,kBAAmB,SAAQ,kBAAkB;IAC5D;;;;;;;OAOG;IACH,MAAM,EAAE,WAAW,CAAC;CACrB;AAED,MAAM,WAAW,iBAAiB;IAChC,QAAQ,CAAC,IAAI,EAAE,OAAO,CAAC;IACvB,QAAQ,CAAC,OAAO,EAAE,gBAAgB,CAAC;IACnC,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;IACvB,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC;IACxB;;;;;;;;;OASG;IACH,QAAQ,CAAC,WAAW,EAAE,IAAI,CAAC;IAC3B,QAAQ,CAAC,QAAQ,EAAE,MAAM,CAAC;IAC1B;;;;;;;OAOG;IACH,QAAQ,CAAC,SAAS,CAAC,EAAE,eAAe,CAAC;CACtC;AAED;;;;;;GAMG;AACH,MAAM,WAAW,eAAe;IAC9B,QAAQ,CAAC,WAAW,EAAE,MAAM,CAAC;IAC7B;;;;OAIG;IACH,QAAQ,CAAC,MAAM,EAAE,SAAS,GAAG,cAAc,CAAC;IAC5C,uFAAuF;IACvF,QAAQ,CAAC,eAAe,EAAE,MAAM,CAAC;CAClC;AAED,MAAM,WAAW,WAAW;IAC1B,cAAc,IAAI,iBAAiB,CAAC;IACpC,OAAO,IAAI,OAAO,CAAC,IAAI,CAAC,CAAC;CAC1B;AAED;;;;;GAKG;AACH,MAAM,WAAW,mBAAmB;IAClC,aAAa,CAAC,OAAO,CAAC,EAAE,oBAAoB,GAAG,OAAO,CAAC,WAAW,CAAC,CAAC;IACpE;;;;;;;OAOG;IACH,WAAW,CAAC,CAAC,OAAO,EAAE,kBAAkB,GAAG,OAAO,CAAC,WAAW,CAAC,CAAC;CACjE;AAED;;;;;;;GAOG;AACH,YAAY,EACV,iBAAiB,EACjB,wBAAwB,EACzB,MAAM,yBAAyB,CAAC;AAEjC,YAAY,EACV,kBAAkB,EAClB,qBAAqB,EACrB,gBAAgB,EAChB,kBAAkB,EAClB,gBAAgB,EAChB,wBAAwB,EACxB,gBAAgB,EAChB,WAAW,EACX,eAAe,EACf,YAAY,EACZ,gBAAgB,EAChB,aAAa,EACd,MAAM,wBAAwB,CAAC;AAEhC,YAAY,EACV,cAAc,EACd,eAAe,EACf,iBAAiB,EACjB,gBAAgB,EACjB,MAAM,6BAA6B,CAAC;AAErC,YAAY,EACV,iBAAiB,EACjB,oBAAoB,EACpB,mBAAmB,EACnB,kBAAkB,EAClB,YAAY,EACb,MAAM,gCAAgC,CAAC;AAExC,YAAY,EAAC,gBAAgB,EAAC,MAAM,6BAA6B,CAAC;AAElE,OAAO,KAAK,EAAC,wBAAwB,EAAC,MAAM,yBAAyB,CAAC;AAEtE,oGAAoG;AACpG,wBAAgB,uBAAuB,CAAC,OAAO,EAAE,OAAO,GAAG,mBAAmB,GAAG,SAAS,CAMzF;AAED;;;;;;;;;;GAUG;AACH,wBAAgB,0BAA0B,CACxC,OAAO,EAAE,OAAO,GACf,wBAAwB,GAAG,SAAS,CAMtC"}
|
package/dist/runtime.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"runtime.js","sourceRoot":"","sources":["../src/runtime.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;GAWG;AAEH,0FAA0F;AAC1F,MAAM,CAAC,MAAM,uBAAuB,GAAG,wBAAwB,CAAC;AAEhE;;;;;GAKG;AACH,MAAM,CAAC,MAAM,sBAAsB,GAAG,4BAA4B,CAAC;AAEnE;;;;;;;GAOG;AACH,MAAM,CAAC,MAAM,yBAAyB,GAAG,kCAAkC,CAAC;AAE5E,wFAAwF;AACxF,MAAM,CAAC,MAAM,6BAA6B,GAAG,CAAU,CAAC;
|
|
1
|
+
{"version":3,"file":"runtime.js","sourceRoot":"","sources":["../src/runtime.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;GAWG;AAEH,0FAA0F;AAC1F,MAAM,CAAC,MAAM,uBAAuB,GAAG,wBAAwB,CAAC;AAEhE;;;;;GAKG;AACH,MAAM,CAAC,MAAM,sBAAsB,GAAG,4BAA4B,CAAC;AAEnE;;;;;;;GAOG;AACH,MAAM,CAAC,MAAM,yBAAyB,GAAG,kCAAkC,CAAC;AAE5E,wFAAwF;AACxF,MAAM,CAAC,MAAM,6BAA6B,GAAG,CAAU,CAAC;AAmKxD,oGAAoG;AACpG,MAAM,UAAU,uBAAuB,CAAC,OAAgB;IACtD,IAAI,OAAO,OAAO,KAAK,QAAQ,IAAI,OAAO,KAAK,IAAI;QAAE,OAAO,SAAS,CAAC;IACtE,MAAM,SAAS,GAAI,OAAmC,CAAC,sBAAsB,CAAC,CAAC;IAC/E,IAAI,OAAO,SAAS,KAAK,QAAQ,IAAI,SAAS,KAAK,IAAI;QAAE,OAAO,SAAS,CAAC;IAC1E,MAAM,EAAC,aAAa,EAAC,GAAG,SAAsC,CAAC;IAC/D,OAAO,OAAO,aAAa,KAAK,UAAU,CAAC,CAAC,CAAE,SAAiC,CAAC,CAAC,CAAC,SAAS,CAAC;AAC9F,CAAC;AAED;;;;;;;;;;GAUG;AACH,MAAM,UAAU,0BAA0B,CACxC,OAAgB;IAEhB,IAAI,OAAO,OAAO,KAAK,QAAQ,IAAI,OAAO,KAAK,IAAI;QAAE,OAAO,SAAS,CAAC;IACtE,MAAM,SAAS,GAAI,OAAmC,CAAC,yBAAyB,CAAC,CAAC;IAClF,IAAI,OAAO,SAAS,KAAK,QAAQ,IAAI,SAAS,KAAK,IAAI;QAAE,OAAO,SAAS,CAAC;IAC1E,MAAM,EAAC,cAAc,EAAC,GAAG,SAAuC,CAAC;IACjE,OAAO,OAAO,cAAc,KAAK,UAAU,CAAC,CAAC,CAAE,SAAsC,CAAC,CAAC,CAAC,SAAS,CAAC;AACpG,CAAC"}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@kubohiroya/turbowarp-camera-source",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.14.0",
|
|
4
4
|
"author": "Hiroya Kubo <hiroya@cuc.ac.jp>",
|
|
5
5
|
"description": "A TurboWarp capability extension for sharing one MediaDevices camera stream across extensions.",
|
|
6
6
|
"type": "module",
|
|
@@ -61,6 +61,7 @@
|
|
|
61
61
|
},
|
|
62
62
|
"devDependencies": {
|
|
63
63
|
"@eslint/js": "^9.39.5",
|
|
64
|
+
"@kubohiroya/turbowarp-extension-manifest": "0.2.0",
|
|
64
65
|
"@kubohiroya/vite-plugin-turbowarp-extension": "0.4.0",
|
|
65
66
|
"@types/node": "^24.13.3",
|
|
66
67
|
"ajv": "^8.20.0",
|
|
@@ -1,49 +0,0 @@
|
|
|
1
|
-
{
|
|
2
|
-
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
3
|
-
"$id": "https://raw.githubusercontent.com/kubohiroya/turbowarp-extension-template/main/schemas/extension-manifest.schema.json",
|
|
4
|
-
"title": "TurboWarp Extension API Manifest",
|
|
5
|
-
"type": "object",
|
|
6
|
-
"additionalProperties": false,
|
|
7
|
-
"required": ["formatVersion", "id", "blocks", "menus"],
|
|
8
|
-
"properties": {
|
|
9
|
-
"formatVersion": {"const": 1},
|
|
10
|
-
"id": {"type": "string", "pattern": "^[a-z0-9]+$"},
|
|
11
|
-
"blocks": {
|
|
12
|
-
"type": "array",
|
|
13
|
-
"items": {
|
|
14
|
-
"type": "object",
|
|
15
|
-
"additionalProperties": false,
|
|
16
|
-
"required": ["opcode", "blockType", "arguments"],
|
|
17
|
-
"properties": {
|
|
18
|
-
"opcode": {"type": "string", "minLength": 1},
|
|
19
|
-
"blockType": {"type": "string", "minLength": 1},
|
|
20
|
-
"arguments": {
|
|
21
|
-
"type": "array",
|
|
22
|
-
"items": {
|
|
23
|
-
"type": "object",
|
|
24
|
-
"additionalProperties": false,
|
|
25
|
-
"required": ["id", "type"],
|
|
26
|
-
"properties": {
|
|
27
|
-
"id": {"type": "string", "minLength": 1},
|
|
28
|
-
"type": {"type": "string", "minLength": 1},
|
|
29
|
-
"menu": {"type": "string", "minLength": 1}
|
|
30
|
-
}
|
|
31
|
-
}
|
|
32
|
-
}
|
|
33
|
-
}
|
|
34
|
-
}
|
|
35
|
-
},
|
|
36
|
-
"menus": {
|
|
37
|
-
"type": "array",
|
|
38
|
-
"items": {
|
|
39
|
-
"type": "object",
|
|
40
|
-
"additionalProperties": false,
|
|
41
|
-
"required": ["id", "acceptReporters"],
|
|
42
|
-
"properties": {
|
|
43
|
-
"id": {"type": "string", "minLength": 1},
|
|
44
|
-
"acceptReporters": {"type": "boolean"}
|
|
45
|
-
}
|
|
46
|
-
}
|
|
47
|
-
}
|
|
48
|
-
}
|
|
49
|
-
}
|