@kubohiroya/turbowarp-realtime-motion-capture 0.2.0

package/docs/architecture.ja.md

@@ -0,0 +1,256 @@
+# アーキテクチャ
+
+[利用ガイド](../README.ja.md) | [TurboWarp機能拡張API](turbowarp-extension-api.ja.md) |
+[English](architecture.md)
+
+## ビルド出力
+
+このプロジェクトは実行時の動作と互換性メタデータを分離し、リポジトリに保存された同じソース定義から両方を生成します。
+
+```text
+src/index.ts + src/extension.ts
+  -> vite-plugin-turbowarp-extension
+  -> dist/<extension>.js
+
+src/config.ts + src/block-definitions.json
+  -> extension-api-manifest Viteプラグイン
+  -> dist/extension-manifest.json
+```
+
+manifestプラグインはViteのビルド後フェーズで実行されます。これにより、JavaScriptプラグインの単一出力検証を維持しながら、TurboWarpバンドルの完成後にだけmanifestを追加します。
+
+## 拡張機能API manifest v1
+
+`schemas/extension-manifest.schema.json`が規範となるJSON Schemaです。`formatVersion`は`1`で、互換性のないmanifest形式を導入するときに変更する必要があります。
+
+v1契約は次の情報を含みます。
+
+- TurboWarp拡張機能のID
+- 各ブロックのopcodeとブロック種類
+- 各引数のID、引数種類、任意のメニュー参照
+- 各メニューのIDとReporterブロックを受け付けるかどうか
+
+ブロック、引数、メニューは、シリアライズ前に識別子で並べ替えられます。テキスト、説明、既定値、静的メニュー項目は、保存済みプロジェクトのAPI参照を識別しないため、意図的に除外しています。そのため互換性チェッカーは、API変更とドキュメントまたはローカライズの変更を区別できます。
+
+manifestとTurboWarp blockがpackageの公開APIです。source levelのcontroller／portは内部の実装境界で、
+npm exportではありません。読み込み順、lifecycle値、error、data contractは
+[機能拡張APIリファレンス](turbowarp-extension-api.ja.md)を参照してください。
+
+## 差分の検出
+
+`dist/`はリリース成果物としてコミットされます。`npm run check:dist`は両方のファイルを再ビルドし、`dist/`配下に変更、削除、未追跡ファイルがある場合に失敗します。これにより、ローカル検証とCIの両方でmanifestとバンドルの差分を検出できます。
+
+## offer QRの縦切り
+
+`config/feature-flags.ts`の`qrCourierPairing`は起動時固定・既定OFFです。有効時は
+`kubohiroyaWebRtcCapability` version 2を要求し、`createOffer(peer)`の完了後に
+`getOffer(peer)`でpairing codeを取得します。別機能拡張のprivate instanceには依存しません。
+
+printable ASCIIのpairing codeを変更せず、version付き`twmp-qr/1` partへ格納します。
+session ID、peer／message識別子、0始まりの順序、原文長、SHA-256 digestを全partに含めます。
+byte modeで保守的に容量計算し、QR Version 40を上限とします。入力は128 KiB、64 part、
+chunkごとに4096文字までです。
+
+表示はrenderer上だけの一時SVG skinです。現在のdrawable skinを保持し、VM costumeを追加せず
+QR skinへ切り替えます。明示終了、project停止、extension dispose、表示target削除で元skinを
+復元し、一時resourceとQR dataを破棄します。
+
+## 固定version application contract codec
+
+`protocolV1Codec`は独立した起動時固定・既定OFF flagです。最大1 MiBのJSONをparseし、rootの
+`schema`と`version`から明示対応した5種類のv1 TypeBox schemaだけへdispatchします。
+version推測やfallbackは行いません。decode成功時はcompact JSONを1件保持し、失敗時は保持値を
+消去して、最初の診断をJSON Pointer pathとmessageとして公開します。
+
+契約は本packageが所有します。正本は`src/protocol/schemas.ts`、`pnpm run schemas`が`schemas/`配下の
+配布用JSON Schemaを生成し、repository checkは、生成物が定義からdriftした場合、file名が宣言した
+`version` literalや`$id`と食い違う場合、dispatch対象のversionに対応するfileが無い場合に失敗します。
+applicationは本packageと配布された`schemas/`を通して契約を利用します。本packageがapplication
+repositoryから契約定義を読むことはなく、依存方向はapplication → extensionの一方向に保たれます。
+
+契約はversionを切って追加し、公開済みversionを書き換えません。`protocolSchemas`はschema識別子と
+versionの2段でdispatchするため、`twrmc/pose-frame-2d`はv1とv2を受理し、v1利用者はv2 payloadを
+拒否し続けます。PoseFrame2D v2は人物ごとに最大4件のサイリウムmarkerを追加します。各markerは、
+一意な色の発光体を観測したCOCO-17 keypoint、`#RRGGBB`の色、patch内で色が占めた割合を持ちます。
+色はkeypointと同一の映像frame・同一のcapture timestampの観測なので、別messageではなくpose frame
+内で運び、受信側での時刻対応付けを不要にします。
+
+TypeBoxのtuple／array制約でCOCO-17順序、6人上限、matrix size、各数値境界を保証します。
+別の再帰key guardにより、WebRTC offer／answer、SDP、ICE／DTLS material、credential fieldを
+永続化前に拒否します。SessionPolicyはapplication validationとして`expiresAt > issuedAt`と
+未期限切れも検証します。
+
+PoseFrame2Dの`captureTimestampUs`とPoseFrame3Dの`timestampUs`は、別実装の同期済みlocal
+time serviceから受け取る不透明値です。この機能拡張はclock同期、offset推定、probe、ping、
+pongを実装せず、受け取ったtimestampを変更せずprotocolへ格納します。
+
+## camera calibration workflow
+
+`cameraCalibrationV1`は起動時固定・既定OFFです。camera／calibration ID、inner cornerが縦横
+3〜20のchessboard、meter単位のsquare sizeを検証してから、named Camera Source leaseを取得
+します。最初の実frame解像度をsession中は固定し、途中変更は検出前に拒否します。
+
+明示的なsample要求ごとに一度だけvideoから一時canvasへcopyします。exact pinしたOpenCV.js
+4.12 WebAssembly backendはbundle内に含め、最初のsampleまたはsolveで遅延初期化します。
+完全なchessboardを検出してsubpixel精度へ補正し、board coverageと
+Laplacian sharpnessから品質を評価します。quality 0.2未満、および保持viewのいずれかと正規化
+RMS corner変位0.015未満のviewは拒否します。8〜40の多様なsampleを保持し、継続的なCPU
+sampling loopや別の参照solverは実装しません。
+
+`calibrateCamera`がintrinsic matrix、distortion vector、RMS reprojection errorを求めます。
+最後のsampleを舞台world board配置として予約し、そのrotation／translationを反転してrow-major
+4×4 `worldFromCameraMatrix`を作ります。設定RMS上限またはCameraCalibration v1境界を超える結果は
+最後の有効profileを置き換えません。
+
+cancel、project reload、extension disposeではleaseと一時sampleを解放し、最後の検証済みprofile
+はmemoryに維持します。明示cleanupだけがprofileも消去します。importもexact v1 schemaで検証し、
+pairing-secret keyを再帰的に拒否します。offline会場LANで唯一のproduction backendを使えるよう
+非圧縮bundle約11 MB増を受け入れ、実camera／board幾何精度とWebAssembly起動はbrowser E2Eで
+検証します。
+
+## WebGPU MoveNet MultiPoseの縦切り
+
+`webgpuMoveNetMultiPose`は独立した起動時固定・既定OFF flagです。TensorFlow.jsではWebGPU
+backendだけをimportし、`setBackend("webgpu")`の結果を検証してから、trackingとbounding-box
+trackerを有効にした`MULTIPOSE_LIGHTNING`を生成します。backendがWebGPU以外ならfail closedし、
+CPU／WASM／WebGL推論fallbackは行いません。
+
+controllerはCamera Sourceから`{cameraId: "pose"}`のleaseを取得し、media captureを所有しません。
+同時に呼ばれた推論blockは1つのPromiseを共有するため、detector実行は重ならず、古いframe要求を
+蓄積しません。成功時は最新video frameから最大6人を推定し、model tracking IDと17個すべての
+名前付きCOCO keypointを必須として、`twrmc/pose-frame-2d` version 1へserializeします。
+
+停止時は実行中の初期化／推論を待ち、detectorをdisposeし、camera leaseと最新frameを解放します。
+TensorFlow.js backendはprocess全体で共有されるためresetせず、本機能が所有するmodel resourceは
+detectorのdisposeで解放します。
+
+## PoseFrame3D avatar retarget
+
+`avatarRetargetV1`は独立した起動時固定・既定OFF flagです。runtime key
+`turbowarpAFrameCapability`へ`requireVersion(1)`を呼び、TurboWarp-A-Frame capabilityの公開同期
+scene操作7種だけを利用します。A-Frame DOM、Three.js `object3D`、GLTF内部boneへはアクセス
+しません。capability v1は`@kubohiroya/turbowarp-aframe@0.3.0`で公開済みです。
+
+asset登録では宣言的template JSONをA-Frameへ送り、検証済みrig mappingを保持します。各boneは
+対応するKalidokit pose rig出力、`{avatar}`を含むselector、任意Euler offset degreeで定義します。
+適用時は対応するexact-v1 PoseFrame3DとPoseFrame2Dの両方を要求します。PoseFrame3Dの`personId`と
+PoseFrame2Dの`trackingId`が一致するpersonだけを結合しますが、これは時刻alignmentではありません。
+
+adapterは両方のCOCO-17 recordを、exact pinした`kalidokit@1.1.5`が要求する33 positionへ
+決定論的に変換します。screen座標にはPoseFrame2Dの`frameWidth`／`frameHeight`を使い、world座標は
+外部serviceの値を維持します。不足するBlazePose face／hand／foot pointは低visibilityで中点補間
+または複製します。`runtime: "tfjs"`、`enableLegs: true`のKalidokit `Pose.solve`だけをrotation
+solverとし、radian出力をA-Frame degreeへ変換します。hips結果にroot scale／offsetを適用し、
+自前rotation fallbackは持ちません。joint／personがbinding threshold未満なら該当transformだけを
+skipし、直前値を維持します。
+
+最大6 person IDを一意なtemplate instanceへbindします。recognition遷移は設定可能なA-Frame
+eventで通知し、application側がPerformance DSLのstart／end effectへ接続できます。1人の
+capability失敗は`partial`診断へ集約し、他avatarを継続します。rebind、明示reset、project
+lifecycle reset、disposeでは可能ならend eventを送り、生成instanceと一時状態をcleanupします。
+
+PoseFrame3Dは別実装の3D serviceから届くexact v1境界dataです。`timestampUs`は不透明値として
+recognition event dataへcopyするだけです。frame alignment、履歴保持／query、triangulation、
+3D solveは行いません。Kalidokitは上流でdeprecatedでありnative BlazePose landmarkを想定するため、
+このCOCO-17拡張は明示的な精度制約です。release前に対象GLTF rigを実browserで検証します。
+
+## フレーム同期パターンの縦切り
+
+`frameSyncPatternV1`は独立した起動時固定・既定OFF flagです。fusion application向けに、
+「ある出来事の後、各カメラPCがそれを写したフレームを記録し終えるまで何ms遅れるか」だけを答えます。
+
+表示側は画面全体のoverlayに、黒地の4×4パネルを描きます。12セルが4096msで一周するミリ秒
+カウンタ、4セルがそのカウンタから導くcheck bitです。露光が画面のリフレッシュをまたぐと2つの
+codeが混ざりますが、check bitがその読み取りを拒否するので、誤った時刻は通りません。1つの
+animation frameで描いた内容は次のリフレッシュで画面に出るため、符号化する時刻は現在の時計に
+実測したリフレッシュ間隔を1つ足した値です。残るプロジェクタ遅延は全カメラ共通なので、
+カメラ間のoffsetでは相殺されます。
+
+カメラ側は名前付きのCamera Source leaseを取得し、`getUserMedia`は呼びません。届いたframeは
+240×180の輝度bufferへ縮小します。bufferはdecoderが同期的に読み終えるため再利用します。
+キャリブレーションは実際のパターンに対して2段階で走ります。前半60%で画素ごとの輝度min/maxを
+記録し、高レンジ画素の最大連結領域のうちパネル形状のものをbounding boxとして採用します。
+後半でセルごとの明暗レベルを学習し、復号成功率を測ります。投影は不均一なのでレベルはセル単位で
+持ち、学習済みレベルの中間に落ちた読み取りは捨てます。信用できないlatencyを返す代わりに、
+`panel-not-found`、`low-contrast`、`decode-unstable`で失敗します。受け付ける窓は6.2秒からです。
+最も遅いセルの変化周期2048msと、レベル学習に割り当てる窓の割合から導いた下限で、全セルを
+両方のレベルで観測できない窓は、後からlow contrastとして失敗する代わりに最初に拒否します。
+どちらの段階も共有時計で終了するため、カメラが止まってもcontrollerが待ち続けることはありません。
+また、キャリブレーション中に停止した場合は、camera faultとして記録せずに終了します。
+
+復号できたframeはobservationとしてqueueに入ります。timestampは外部の同期時刻サービスから
+読み取った不透明な値で、frameがアプリケーションへ届いた時点で取得します。センサの露光時刻が
+必要な呼び出し側のために、browserが報告するframe ageは別に公開します。clock probe、latency
+サンプル、カメラ別の集計レポートはWebRTC機能拡張側の責務なので、clock・offset・ping・pongの
+ロジックはここには置きません。
+## 多視点3D pose fusion
+
+`poseFusion3D`は独立した起動時固定・既定OFF flagです。bufferへ入力するPoseFrame2D JSONは
+fusion appがWebRTC data channelで受信したものであり、本機能拡張はtransportもclockも所有しません。
+
+cameraごとにtimestamp順のring bufferを持ちます。slot数は設定delayとjitter windowから算出し、
+16〜600 frameに制限します。buffer対象cameraは最大16台です。jitter window内で順序が入れ替わった
+frameは、ringの短い側をずらしてtimestamp位置へ挿入するため、通常の順序どおりの追加はO(1)の
+ままです。timestampの重複、最新frameからjitter windowより古い到着、満杯ringの最古frameより
+古い到着はdropとして計上し、bufferしません。ここではcameraごとのclock offsetを推定しません。
+capture timestampは別実装の同期済みlocal time serviceが与える不透明値のまま扱います。
+
+frameをbufferするのは、その`cameraId`のcalibration profileが読み込み済みで、かつそのprofileが
+frameを説明できる場合だけです。`calibrationId`や解像度が一致しないframeは、誤ったintrinsicで
+そのまま三角測量されてしまうため拒否します。これらは重複や遅延到着と同様にdropとして計上し、
+throwしません。ingestはdata channelのhot pathであり、設定を誤ったpeer 1台で実行中のscriptを
+止めるべきではないからです。calibration済みcameraしかbufferしないので、未知のcamera IDが
+ring bufferを占有することもありません。
+
+`fuse PoseFrame3D at buffered delay`は、最新のbuffered timestampから設定delayを引いた1つの過去の
+瞬間を解決し、全cameraをその瞬間で再sampleします。前後のframeで挟めたkeypointは線形補間し、
+片側がocclusionのkeypointは低信頼値を混ぜず見えている側の観測を採用します。挟めないcamera、
+またはjitter windowの2倍より広い間隔しかないcameraは、最大1 jitter windowだけ直近frameを保持し、
+それを超える場合は寄与しません。
+
+camera間の対応付けは、異なるcameraの追跡人物のすべての組について、共有する確信のあるkeypoint
+での2視点reprojection誤差の平均をcostとし、共有keypointは4点以上・最大12点で打ち切ります。costの
+小さい組から貪欲にmergeし、同一cameraの2視点が1人になるmergeは拒否します。2視点の三角測量は
+2本の視線の最短距離の中点を閉形式で求め、この二乗オーダーの段を反復解法から外します。3視点
+以上はscore重み付き線形解法（Jacobiは相対収束判定）を使います。16 camera×6人の上限で1回の統合
+は約80 ms、4 camera×2人では約1 msです。
+
+2台以上のcameraが覆うclusterは、keypointごとにcheirality判定とreprojection判定付きで三角測量
+します。全視点が一致しない場合は、2視点ごとの仮解に対してreprojection閾値内に収まる視点数を
+数え、最大の一致集合で三角測量し直します。これにより少数の誤検出はkeypointを引きずらずに
+捨てられます（視点が2つの解に均等に割れる場合は原理的に区別できません）。pixel観測はprofileの
+OpenCV rational modelで歪み補正するため、係数0／4／5／8個に対応し、それ以外はprofile読み込み時
+に拒否します。
+
+registryはcameraとtracking IDの重なりから`person-N`のidentityを維持し、keypointごとに最後に
+三角測量できた位置を保持します。確信のある視点が2つ未満のkeypointはその位置を保持してscore `0`
+を返し、実測値と保持値を利用側が区別できるようにします。組み立てたframeは保持する前に、
+pinnedのPoseFrame3D v1 schemaで検証します。
+
+bufferが空、覆うcameraが2台未満、多視点で見えた人物がいない場合は想定内の一時状態として
+`false`を返し、直前の統合結果を保持したままerror codeを公開します。不正なJSON、他contractの
+schema、不正なcalibration profileはerrorになります。停止ボタン、project reload、extension dispose
+ではbufferと統合結果を解放し、明示cleanupでは読み込み済みcalibration profileも解放します。
+`PROJECT_RUN_STOP`では解放しません。runtimeはthread queueが空になるたびにこのeventを出すため、
+hat scriptでframeをbufferするevent駆動のprojectがmessageの合間にjitter bufferを失ってしまいます。
+camera leaseと一時skinはこのeventでも解放します。
+
+## サイリウムによる識別と向きの補正
+
+`glowStickMarkers`は独立した起動時固定・既定OFF flagです。camera側では推論のたびに、追跡中の各
+人物について設定したkeypointごとに1つのpatchを、keypointと同一の映像frameから、直後に破棄する
+一時canvas経由でsampleします。patchはhueで評価します。彩度または明度が閾値未満のpixelは無視し、
+残りのhueを円環平均するため、0度をまたぐ赤は赤のまま扱えます。色が占める割合が閾値を超えた場合
+だけmarkerとして採用し、keypointごとに最も強い観測をPoseFrame2D v2のmarkerにします。sampleを
+止めている間、frameはv1のままです。
+
+fusion側では、performance DSLが演者の色を与え、演者がサイリウムを持つkeypointはfusion側の設定と
+します（その契約は持ち手を記述しないため）。cameraのsampleごとに、観測の強い順で貪欲に割り当て、
+1人の演者が同じcameraで2人を占めることはありません。
+
+割り当ては2つの効果を持ちます。1つは識別です。統合された人物は演者IDを`personId`とするため、
+occlusion、再入場、tracking ID変化をまたいで同一性が保たれ、対応付けでは別演者の視点の統合を拒否し、
+同一演者の視点はreprojection costより先に統合します。もう1つは向きの補正です。指定keypointの
+左右反転側で色が見つかった場合、そのcameraは背面を腹面として読んでいるため、三角測量の前に
+その視点の左右keypointを入れ替えます。補正しなければ、その視点は手足を体の反対側へ引きずるか、
+reprojection判定で落ちてしまいます。

package/docs/architecture.md

@@ -0,0 +1,294 @@
+# Architecture
+
+[User guide](../README.md) | [TurboWarp extension API](turbowarp-extension-api.md) |
+[日本語](architecture.ja.md)
+
+## Build outputs
+
+The project keeps runtime behavior and compatibility metadata separate while generating both from
+the same checked-in source definitions.
+
+```text
+src/index.ts + src/extension.ts
+  -> vite-plugin-turbowarp-extension
+  -> dist/<extension>.js
+
+src/config.ts + src/block-definitions.json
+  -> extension-api-manifest Vite plugin
+  -> dist/extension-manifest.json
+```
+
+The manifest plugin runs in Vite's post-build phase. This preserves the JavaScript plugin's
+single-output validation and adds the manifest only after the TurboWarp bundle is complete.
+
+## Extension API manifest v1
+
+`schemas/extension-manifest.schema.json` is the normative JSON Schema. `formatVersion` is `1` and
+must change when an incompatible manifest shape is introduced.
+
+The v1 contract contains:
+
+- the TurboWarp extension ID;
+- each block opcode and block type;
+- each argument ID, argument type, and optional menu reference;
+- each menu ID and whether it accepts reporter blocks.
+
+Blocks, arguments, and menus are sorted by their identifiers before serialization. Text,
+descriptions, default values, and static menu items are intentionally excluded because they do not
+identify saved-project API references. A compatibility checker can therefore distinguish API
+changes from documentation or localization changes.
+
+The manifest and TurboWarp blocks are the package's public API. Source-level controllers and ports
+are internal implementation seams and are not npm exports. See the
+[extension API reference](turbowarp-extension-api.md) for loading order, lifecycle values, errors,
+and data contracts.
+
+## Drift detection
+
+`dist/` is committed as a release artifact. `npm run check:dist` rebuilds both files and fails when
+Git reports any modified, deleted, or untracked file below `dist/`. This catches manifest and bundle
+drift in local checks and CI.
+
+## Offer QR vertical slice
+
+`config/feature-flags.ts` keeps `qrCourierPairing` startup-fixed and default OFF. When enabled, the
+extension requires `kubohiroyaWebRtcCapability` version 2, calls `createOffer(peer)`, and retrieves
+the completed pairing code with `getOffer(peer)`. It never reaches into another extension's private
+instance.
+
+The unchanged printable-ASCII pairing code is wrapped in versioned `twmp-qr/1` parts. Every part
+contains a random session ID, peer and message identity, zero-based ordering, total source length,
+and a SHA-256 digest. A conservative byte-mode capacity check bounds each QR to Version 40. The
+wire input is limited to 128 KiB, 64 parts, and 4096 characters per chunk.
+
+Display uses renderer-only SVG skins. The current drawable skin is retained, a generated skin is
+attached without creating a VM costume, and cleanup restores the retained skin before destroying
+the temporary resource. Project stop, extension disposal, explicit cleanup, and displayed-target
+removal all release sensitive QR data.
+
+## WebGPU MoveNet MultiPose vertical slice
+
+`webgpuMoveNetMultiPose` is an independent startup-fixed, default-OFF flag. Startup imports only the
+TensorFlow.js WebGPU backend, explicitly calls `setBackend("webgpu")`, verifies the selected backend,
+and then creates `MoveNet` with `MULTIPOSE_LIGHTNING`, tracking enabled, and bounding-box tracking.
+Any non-WebGPU result fails closed; there is no CPU, WASM, or WebGL inference fallback.
+
+The controller acquires `{cameraId: "pose"}` through Camera Source and never owns media capture.
+Concurrent inference block calls share one promise, so detector invocations do not overlap and old
+frame requests do not accumulate. Each successful call reads the latest video frame, requests no
+more than six poses, requires model tracking IDs and all 17 named COCO keypoints, and serializes the
+result to the `twrmc/pose-frame-2d` version 1 contract.
+
+Stopping waits for in-flight initialization/inference, disposes the detector, releases the camera
+lease, and clears the last frame. The TensorFlow.js backend is process-global and is not reset,
+because doing so would invalidate resources owned by other extensions; disposing the detector
+releases this feature's model resources.
+
+## Owned application contracts and their codec
+
+`protocolV1Codec` is an independent startup-fixed, default-OFF flag. The codec parses at most 1 MiB
+of JSON, reads the root `schema` and `version`, and dispatches only explicitly supported TypeBox
+schemas. It performs no version inference or fallback. A successful decode retains one compact JSON
+value; any failed decode clears it and exposes the first diagnostic as a JSON Pointer path and
+message.
+
+This package owns the contracts. `src/protocol/schemas.ts` is the source of truth, `pnpm run schemas`
+generates the published JSON Schemas under `schemas/`, and the repository check fails when a
+generated file drifts from its definition, when a file name disagrees with the `version` literal or
+`$id` it declares, or when a dispatched version has no published file. Applications consume these
+definitions through this package and through the published `schemas/` directory; nothing here reads
+contract definitions from an application repository, which keeps the dependency pointing from the
+application to the extension.
+
+Contracts are versioned, never edited in place. `protocolSchemas` dispatches by schema identifier and
+then by version, so `twrmc/pose-frame-2d` accepts v1 and v2 while a v1 consumer still rejects a v2
+payload. PoseFrame2D v2 adds up to four glow stick markers per person, each naming the COCO-17
+keypoint where a uniquely colored light was observed, its `#RRGGBB` color, and the patch coverage
+that produced it. The color is observed on the same video frame and at the same capture timestamp as
+the keypoints, so it travels inside the pose frame instead of a second message that a receiver would
+have to time-align.
+
+TypeBox tuple and array constraints enforce COCO-17 ordering, six-person limits, matrix sizes, and
+all bounded values. A separate recursive key guard rejects WebRTC offers, answers, SDP, ICE/DTLS
+material, and credential fields before persistence, even if a later schema accidentally permits a
+nested extension point. SessionPolicy also enforces `expiresAt > issuedAt` and rejects expired
+policies at application-validation time.
+
+PoseFrame2D `captureTimestampUs` and PoseFrame3D `timestampUs` are opaque values from a separate
+synchronized local time service. This extension carries the supplied timestamp unchanged and does
+not implement clock initialization, offset estimation, probes, ping, or pong logic.
+
+## Camera calibration workflow
+
+`cameraCalibrationV1` is startup-fixed and default OFF. A session validates the camera and
+calibration identifiers plus a 3–20 by 3–20 inner-corner chessboard and a square size in meters,
+then obtains a named Camera Source lease. The actual first frame dimensions become immutable for
+the session; later resolution changes fail before detection.
+
+Each explicitly requested sample performs one temporary video-to-canvas copy. The exact-pinned
+OpenCV.js 4.12 WebAssembly backend is bundled but lazily initialized by the first sample or solve.
+It detects the complete chessboard, refines corners to subpixel
+precision, and scores board coverage plus Laplacian sharpness. Quality below 0.2 and views within
+0.015 normalized RMS corner displacement of any retained view are rejected. The controller retains
+8–40 diverse samples and does not run a continuous CPU sampling loop or a second reference solver.
+
+`calibrateCamera` produces the intrinsic matrix, distortion vector, and RMS reprojection error. The
+last sample is deliberately reserved as the stage-world board placement. Its returned rotation and
+translation are inverted into the row-major 4 by 4 `worldFromCameraMatrix`. Results above the
+configured RMS threshold or outside CameraCalibration v1 bounds do not replace the last valid
+profile.
+
+Cancel, project reload, and extension disposal release the lease and temporary samples while
+keeping the last validated profile in memory. Explicit cleanup clears that profile too. Import uses
+the same exact v1 schema and recursively rejects pairing-secret keys. The 11 MB uncompressed bundle
+increase is accepted to keep the sole production backend available on an offline venue LAN; real
+camera/board geometry and WebAssembly startup remain browser E2E responsibilities.
+
+## PoseFrame3D avatar retargeting
+
+`avatarRetargetV1` is an independent startup-fixed, default-OFF flag. It requires runtime key
+`turbowarpAFrameCapability`, calls `requireVersion(1)`, and uses only the seven public synchronous
+scene operations from the TurboWarp-A-Frame capability. The consumer never accesses A-Frame DOM,
+Three.js `object3D`, or GLTF bone internals. Capability v1 is published in
+`@kubohiroya/turbowarp-aframe@0.3.0`.
+
+An asset registration sends declarative template JSON to A-Frame and retains a validated rig map.
+Each bone maps one supported Kalidokit pose rig output to a selector containing `{avatar}`, plus
+optional Euler offset degrees. Applying a frame requires corresponding exact-v1 PoseFrame3D and
+PoseFrame2D values. A person is joined only by PoseFrame3D `personId` equal to PoseFrame2D
+`trackingId`; this is not temporal alignment.
+
+The adapter maps both COCO-17 records deterministically to the 33 positions required by
+exact-pinned `kalidokit@1.1.5`. Screen coordinates use PoseFrame2D `frameWidth` and `frameHeight`;
+world coordinates retain the external service coordinate values. Missing BlazePose face, hand, and
+foot points are midpoint-interpolated or duplicated with reduced visibility. Kalidokit `Pose.solve`
+with `runtime: "tfjs"` and `enableLegs: true` is the sole rotation solver. Its radians are converted
+to A-Frame degrees, and its hips result drives the configured root scale and offset. There is no
+custom rotation fallback. Joint or person confidence below the binding threshold skips only that
+transform and preserves its prior value.
+
+At most six person IDs bind to unique template instances. Recognition transitions use configurable
+A-Frame events, allowing the application to connect Performance DSL start/end effects without this
+adapter owning effect execution. Per-person capability failures are collected as `partial` state so
+other avatars continue. Rebind, explicit reset, project lifecycle reset, and disposal emit end when
+possible, delete created instances, and clear temporary state.
+
+PoseFrame3D is exact v1 boundary data from a separate 3D service. Its `timestampUs` is opaque and is
+only copied into recognition event data. This extension performs no frame alignment, history
+retention/query, triangulation, or 3D solve. Kalidokit is deprecated upstream and expects native
+BlazePose landmarks; the deterministic COCO-17 expansion is therefore an explicit accuracy
+constraint, and intended GLTF rigs require real-browser validation before release.
+
+## Frame sync pattern vertical slice
+
+`frameSyncPatternV1` is startup-fixed and default OFF. It answers one question for the fusion
+application: how long after an event each camera computer finishes recording the frame that shows
+it.
+
+The display side paints a full-screen overlay with a 4 by 4 panel on black. Twelve cells hold a
+millisecond counter that wraps every 4096 ms; four hold check bits derived from the counter. A
+camera exposure that straddles a display refresh mixes two codes, and the check bits reject that
+reading instead of letting a wrong time through. What is drawn during one animation frame reaches
+the screen at the next refresh, so the encoded time is the current clock reading plus one measured
+refresh interval; the remaining projector delay is common to every camera and cancels out of the
+per-camera offsets.
+
+The camera side takes a named Camera Source lease and never calls `getUserMedia`. Each delivered
+frame is downscaled to a 240 by 180 luminance buffer that is reused between callbacks because the
+decoder reads it synchronously. Calibration runs in two phases on the live pattern: the first 60%
+of the window records per-pixel minimum and maximum luminance and takes the largest connected
+high-range region whose bounding box is panel shaped, and the rest learns each cell's own light and
+dark level and measures how often readings decode. Uneven projection is why levels are per cell, and
+a reading that lands between a cell's learned levels is discarded. Calibration fails with
+`panel-not-found`, `low-contrast`, or `decode-unstable` rather than returning untrusted latencies.
+The accepted window starts at 6.2 seconds, derived from the slowest cell's 2048 ms change period and
+the share of the window the levels phase gets, so a window that cannot see every cell at both levels
+is refused up front instead of failing later as low contrast. Both phases end on the shared clock, so
+a stalled camera never leaves the controller waiting, and stopping mid-calibration settles the
+pending run without recording a camera fault.
+
+Decoded frames are queued as observations. Timestamps are opaque readings from the external
+synchronized time service, taken when the frame reached the application; the browser-reported frame
+age is exposed separately for callers that want the sensor exposure moment instead. Clock probing,
+latency samples, and the aggregated per-camera report belong to the WebRTC extension, so no clock,
+offset, ping, or pong logic lives here.
+## Multi-camera 3D pose fusion
+
+`poseFusion3D` is an independent startup-fixed, default-OFF flag. The fusion app feeds the buffer
+with PoseFrame2D JSON received over WebRTC data channels; this extension owns neither the transport
+nor the clock.
+
+Every camera gets its own timestamp-ordered ring buffer. Slot count is derived from the configured
+delay and jitter window, bounded to 16 to 600 frames, and at most 16 cameras are buffered. An
+in-window reorder is inserted at its timestamp position by shifting the shorter side of the ring, so
+the common in-order append stays O(1). A duplicate timestamp, an arrival older than the newest frame
+minus the jitter window, and an arrival older than the oldest retained frame of a full ring are
+counted as dropped instead of buffered. None of this estimates a per-camera clock offset: capture
+timestamps stay opaque values from the separate synchronized local time service.
+
+A frame is only buffered when a calibration profile for its `cameraId` is loaded and that profile
+describes it: a mismatched `calibrationId` or frame size would otherwise be triangulated silently
+with the wrong intrinsics. Such frames, like duplicates and late arrivals, are counted as dropped
+rather than thrown, because ingest runs on the data-channel hot path and one misconfigured peer must
+not break a running project script. Because only calibrated cameras are buffered, stray camera IDs
+cannot occupy the ring buffers either.
+
+`fuse PoseFrame3D at buffered delay` resolves the newest buffered timestamp minus the configured
+delay, and every camera is resampled at that single past instant. A keypoint bracketed by two frames
+is interpolated linearly; a keypoint that is occluded on one side of the bracket keeps the visible
+observation instead of blending a low-confidence estimate into it; a camera without a bracket, or
+with a bracket wider than twice the jitter window, holds its nearest frame for at most one jitter
+window and otherwise contributes nothing.
+
+Cross-camera association scores every pair of tracked persons from different cameras by the mean
+two-view reprojection error over their shared confident keypoints, requiring at least four shared
+keypoints and stopping at twelve. Pairs are merged greedily from the lowest cost, and a merge that
+would place two views of the same camera in one person is rejected. Two-view triangulation uses the
+closed-form midpoint of both viewing rays, which keeps this quadratic stage off the iterative
+solver; the general case still uses the score-weighted linear solver with a relative Jacobi
+convergence threshold. One fusion at the 16 camera by 6 person limit measures about 80 ms, against
+about 1 ms for four cameras and two performers.
+
+Each cluster covered by at least two cameras is triangulated per keypoint with a cheirality check
+and a reprojection check. When the full view set does not agree, every two-view seed is scored by
+how many views fall inside the reprojection threshold, and the largest consensus set is
+re-triangulated; a minority of wrong detections is therefore discarded instead of dragging the
+keypoint away from the truth, while views split evenly between two consistent answers stay
+ambiguous. Pixel observations are undistorted with the OpenCV rational model of the profile, so 0,
+4, 5, or 8 coefficients are supported and anything else is rejected when the profile is loaded.
+
+A registry keeps stable `person-N` identifiers by camera and tracking-ID overlap, and keeps the last
+triangulated position of every keypoint. A keypoint left with fewer than two confident views holds
+that last position and reports score `0`, so consumers can distinguish measured from held values.
+The assembled frame is checked against the pinned PoseFrame3D v1 schema before it is retained.
+
+Empty buffers, fewer than two covering cameras, and an instant with no multi-camera person are
+expected transient states: they report `false`, keep the last fused frame, and expose an error code.
+Invalid JSON, a foreign schema, and an invalid calibration profile throw. The stop button, project
+reload, and extension disposal clear buffers and fused results; explicit cleanup also clears the
+loaded calibration profiles. `PROJECT_RUN_STOP` deliberately does not: the runtime emits it whenever
+the thread queue empties, and an event-driven fusion project that buffers frames from hat scripts
+would otherwise lose its jitter buffer between messages. Camera leases and temporary skins are still
+released there.
+
+## Glow stick assisted identity and orientation
+
+`glowStickMarkers` is an independent startup-fixed, default-OFF flag. On the camera side, each
+inference samples one patch per configured keypoint of every tracked person from the same video
+frame, in one temporary canvas that is released immediately. A patch is scored by hue: pixels below
+the saturation or value threshold are ignored, the remaining hues are averaged circularly so a
+wrapping red stays red, and a patch qualifies only when enough of it carried the color. The strongest
+observation per keypoint becomes a PoseFrame2D v2 marker; the frame stays v1 while sampling is off.
+
+On the fusion side the Performance DSL supplies the performer colors, and a fusion-side setting says
+which keypoint each performer carries the light at, because that contract does not describe the
+carrying hand. Each camera sample is assigned greedily from the strongest observation, and one
+performer never claims two people in the same camera.
+
+An assignment does two things. It fixes identity: the fused person takes the performer's identifier,
+so `personId` survives occlusion, re-entry, and tracking-ID churn, and the association step refuses
+to merge views of different performers while merging views of the same performer before any
+reprojection cost is considered. It also fixes orientation: a color found on the mirror of the
+performer's keypoint means that camera labelled a back view as a front view, so the view's left and
+right keypoints are swapped before triangulation. Without that correction the swapped view pulls
+every limb across the body or fails the reprojection gate entirely.

package/docs/index.html

@@ -0,0 +1,83 @@
+<!doctype html>
+<html lang="en">
+<head>
+  <meta charset="utf-8">
+  <meta name="viewport" content="width=device-width,initial-scale=1">
+  <title>TurboWarp Realtime Motion Capture documentation</title>
+  <style>
+    body{font-family:system-ui,-apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif;max-width:900px;margin:0 auto;padding:2rem;line-height:1.6;color:#222}
+    code{background:#f3f3f3;padding:.1em .3em;border-radius:.25em}
+    pre{background:#f6f8fa;padding:1rem;overflow:auto;border-radius:.5rem}
+    table{border-collapse:collapse;width:100%}th,td{border:1px solid #ddd;padding:.5rem;text-align:left}
+    nav a{margin-right:1rem}
+  </style>
+</head>
+<body>
+  <h1>TurboWarp Realtime Motion Capture</h1>
+  <p>Composite blocks for realtime motion capture camera and fusion applications.</p>
+
+  <nav>
+    <a href="https://github.com/kubohiroya/turbowarp-realtime-motion-capture#readme">User guide</a>
+    <a href="https://github.com/kubohiroya/turbowarp-realtime-motion-capture/blob/main/docs/turbowarp-extension-api.md">Extension API</a>
+    <a href="https://github.com/kubohiroya/turbowarp-realtime-motion-capture/blob/main/docs/turbowarp-extension-api.ja.md">API（日本語）</a>
+    <a href="https://github.com/kubohiroya/turbowarp-realtime-motion-capture/blob/main/docs/architecture.md">Architecture</a>
+  </nav>
+
+  <h2>Install and configure</h2>
+  <p>Load the provider extensions you use first, then load the version-pinned bundle as an unsandboxed custom extension:</p>
+  <pre>https://cdn.jsdelivr.net/npm/@kubohiroya/turbowarp-realtime-motion-capture@0.2.0/dist/turbowarp-realtime-motion-capture.js</pre>
+  <p>All feature groups are OFF by default. Set the complete startup configuration before loading the bundle:</p>
+  <pre><code>globalThis.__TWMP_FEATURE_FLAGS__ = {
+  qrCourierPairing: true,
+  webgpuMoveNetMultiPose: true,
+  protocolV1Codec: true,
+  cameraCalibrationV1: true,
+  avatarRetargetV1: true,
+  frameSyncPatternV1: true,
+  poseFusion3D: true,
+  glowStickMarkers: true,
+};
+globalThis.__TWMP_QR_CONFIG__ = {errorCorrectionLevel: "M"};</code></pre>
+  <p>Omit features you do not use. Reload the project and extension after changing this startup-fixed object.</p>
+
+  <h2>Offer QR blocks</h2>
+  <h3><code>prepare offer QR for peer [PEER]</code></h3>
+  <p>Creates a named WebRTC offer and prepares one or more QR courier parts.</p>
+  <h3><code>show offer QR part [INDEX] on this sprite</code></h3>
+  <p>Displays a one-based part as a temporary, lossless SVG sprite skin.</p>
+  <h3><code>end offer QR display</code></h3>
+  <p>Restores original skins and discards sensitive temporary session data.</p>
+  <p>The extension exposes blocks across eight independently enabled feature groups. See the <a href="https://github.com/kubohiroya/turbowarp-realtime-motion-capture/blob/main/docs/turbowarp-extension-api.md#block-reference">generated block reference</a> for every opcode, argument, type, and default.</p>
+
+  <h2>Enablement and safety</h2>
+  <p>The startup-fixed <code>qrCourierPairing</code> flag is OFF by default. The extension requires the version 2 TurboWarp WebRTC runtime capability and must run unsandboxed.</p>
+  <p>Pairing QR codes can expose local addressing and ICE credentials. Display them only inside a trusted venue and remove courier photographs after pairing.</p>
+  <h2>WebGPU MoveNet MultiPose</h2>
+  <p>The independent <code>webgpuMoveNetMultiPose</code> flag is OFF by default. When enabled, the extension leases frames from TurboWarp Camera Source, explicitly requires the TensorFlow.js <code>webgpu</code> backend, and returns up to six tracked COCO-17 poses as PoseFrame2D version 1 JSON.</p>
+  <p>No CPU, WASM, or WebGL inference fallback is provided. The first model load generally requires network access; prepare the browser cache before an offline venue run.</p>
+
+  <h2>Application contract codec</h2>
+  <p>The independent <code>protocolV1Codec</code> flag is OFF by default. When enabled, blocks validate and round-trip SessionPolicy, CameraCalibration, PoseFrame2D (v1 and v2), PoseFrame3D, and Performance DSL JSON. This extension owns those contracts and publishes their generated JSON Schemas; applications depend on the extension, never the reverse.</p>
+  <p>Unknown schemas, versions, fields, malformed COCO-17 tuples, and pairing credential keys fail closed. Diagnostic reporters expose the JSON Pointer path and message.</p>
+  <p>Pose timestamps are opaque values supplied by a separate synchronized local time service. This extension carries them unchanged and implements no clock probes, offset estimation, ping, or pong logic.</p>
+
+  <h2>Camera calibration</h2>
+  <p>The independent <code>cameraCalibrationV1</code> flag is OFF by default. A high-level state machine leases the named Camera Source stream, fixes its real resolution, accepts diverse chessboard samples, and exports an exact CameraCalibration v1 profile.</p>
+  <p>The bundled, exact-pinned OpenCV.js 4.12 WebAssembly backend is the sole solver. It estimates intrinsic, distortion, and world-from-camera values and rejects insufficient samples, resolution changes, weak views, excessive reprojection RMS, unknown profile versions, and pairing credentials.</p>
+
+  <h2>PoseFrame3D avatar retargeting</h2>
+  <p>The independent <code>avatarRetargetV1</code> flag is OFF by default. The adapter requires TurboWarp-A-Frame 0.3.0 scene capability v1 and uses only its public template, selector transform, event, count, and delete operations.</p>
+  <p>Corresponding external PoseFrame3D and PoseFrame2D v1 values can drive up to six declarative rigs. The adapter matches person IDs, expands COCO-17 to BlazePose-33, and uses exact-pinned Kalidokit 1.1.5 as its only rotation solver before sending public selector transforms to A-Frame.</p>
+  <p>The deterministic expansion uses reduced-visibility duplicates for unavailable hand, foot, and face landmarks and is less precise than native BlazePose input. Kalidokit is deprecated upstream, so intended GLTF rigs require real-browser validation. This extension provides no custom solver fallback and does not align frames, query history, triangulate, or solve 3D data.</p>
+  <h2>Multi-camera 3D pose fusion</h2>
+  <p>The independent <code>poseFusion3D</code> flag is OFF by default. When enabled, blocks buffer PoseFrame2D JSON per camera in timestamp-ordered ring buffers, resample every camera at one past instant behind a configured delay, and triangulate the synchronized 2D sets into PoseFrame3D version 1 JSON.</p>
+  <p>Out-of-order frames inside the jitter window are reordered, while duplicates, late arrivals, and frames that no loaded calibration profile describes are counted as dropped. Occluded keypoints are filled from the visible side of their bracket, disagreeing views are resolved by the largest consensus set, and a keypoint without two confident views holds its last triangulated position with score zero.</p>
+
+  <h2>Glow stick markers</h2>
+  <p>The independent <code>glowStickMarkers</code> flag is OFF by default. When enabled, the camera app samples the keypoints you name for a uniquely colored glow stick on the same video frame as the pose and reports PoseFrame2D version 2.</p>
+  <p>The fusion app maps those colors to performers with the Performance DSL palette. An identified person keeps the performer's identifier across occlusion and re-entry, views of different performers never merge, and a color seen on the mirrored keypoint tells the fusion that the camera read a back view as a front view, so that view's left and right labels are swapped before triangulation.</p>
+
+  <h2>Development</h2>
+  <p>The public API is the TurboWarp extension and its generated manifest. Source-level controllers are internal and are not npm exports. See the <a href="https://github.com/kubohiroya/turbowarp-realtime-motion-capture#readme">repository README</a> and <a href="https://github.com/kubohiroya/turbowarp-realtime-motion-capture/blob/main/docs/architecture.md">architecture documentation</a> for build, test, release, and compatibility details.</p>
+</body>
+</html>