wenay-common2 1.0.65 → 1.0.66

package/CLAUDE.md

@@ -51,6 +51,26 @@ Write new code in this style by default — no need to ask.
 ## Comments
 - Section dividers like `// ===...===` with a block heading.
 - Explain "why", not "what". Keep them short.
+
+## Work progress files
+
+- For any task that is more than a tiny/local edit, create a temporary progress file before
+  starting broad changes.
+- Put progress files under `doc/progress/`. This is the working-doc area, separate from public
+  API docs, roadmap docs, and release notes.
+- Name them by task, for example `doc/progress/replay-route-handoff.md`.
+- Keep the file short: goal, current checkpoint list, notable decisions, blockers, and verification
+  already run or still needed.
+- Update the progress file as checkpoints are completed, especially before switching context or
+  making broad edits.
+- When the task is finished, delete the progress file. Preserve only the durable outcome:
+  final response, commit message, and, for publishable changes, the matching `doc/changes/<version>.md`
+  entry.
+- If work is paused or blocked locally, leave the progress file in place and make the next required
+  action explicit. If the paused state must be committed or handed off, promote the useful part into
+  a durable doc (`ROADMAP`, `RECOMMENDATIONS`, `doc/target`, or `doc/changes`) instead of relying on
+  an ignored progress file.
+
 ## Documentation and release notes
 
 - `README.md` is navigation only. Do not put API guides, examples, or long explanations there.

package/doc/ROADMAP.md

@@ -35,39 +35,98 @@ This implies a useful split:
 - **API surface** remains facade-shaped and typed.
 - **Transport/routing layer** may substitute sockets and promote paths.
 - **Replay layer** resumes streams and mirrors from the last known `seq`.
-- **Authority layer** decides whether an incoming write is accepted, predicted, reconciled, or merged.
-
-### 0.1 Optional account-to-account route plane ("wrapper over wrappers") 🔴
-
-Some deployments need separate client/account identities to communicate directly when policy and
-network conditions allow it, while still allowing the server/relay to step back into the path later.
-This is not just a socket trick; it is an account-aware routing shell above the existing facade,
-mirror, and replay primitives.
-
-- **Account model:** every participant has its own account/session identity and a scoped facade/store
-  set. Runtime-account maps are a dynamic keyspace, so they should look like `noStrict(accountMap)`
-  rather than a fixed schema. Access checks stay in the facade/policy layer; `noStrict` is not an ACL.
-- **Wrapper over wrappers:** an app-level coordinator owns the set of per-account clients, exposed
-  facades, mirrors, replay subscriptions, and route state. "State of other accounts" is represented as
-  selected `Observe` mirrors/replay/offline resources, ideally started and stopped through
-  `createStoreManager`, not as a new global store core.
-- **Route states:** a pair of accounts may be `relay`, `direct`, `direct+shadowRelay` (audit/observe
-  copy), `blocked`, or `fallback`. Direct links are optional optimizations, useful for latency or
-  traffic cost, not a semantic change.
-- **Re-interposition:** if NDA/privacy policy, audit, moderation, throttling, reauth, direct-link
-  failure, or group topology changes require it, the relay must be able to re-enter the data path.
-  This is the reverse of direct promotion: open the replacement route, resume from the last `seq`,
-  switch consumers after catch-up, then close or demote the old route.
-- **Privacy rule:** direct account links are opt-in and policy-gated. Peers only receive the endpoint
-  and session material needed for that specific relationship; no implicit broad account discovery.
-
-Open questions: account ACL/policy shape; signaling format; whether the relay sees payloads or only
-coordinates encrypted direct streams; how route state is exposed to the app; backpressure across
-multi-hop and direct paths; revocation in the middle of a live stream.
-
-Status: 🔴 not started. This belongs to project-0 architecture: no store-core change, but it defines
-the account/routing shell that later direct-connection work plugs into.
-
+- **Authority layer** decides whether an incoming write is accepted, predicted, reconciled, or merged.
+
+Critical ordering rule: do **not** start with WebRTC/NAT plumbing or CRDTs. The next useful layer is a
+small route/account/policy coordinator with a fake/in-process transport adapter. Direct transport and
+multi-writer merge are expensive adapters; they only become safe after the coordinator state machine is
+boring and well-tested.
+
+### 0.1 Account route coordinator ("wrapper over wrappers") 🔴
+
+Some deployments need separate client/account identities to communicate directly when policy and
+network conditions allow it, while still allowing the server/relay to step back into the path later.
+This is not just a socket trick; it is an account-aware routing shell above the existing facade,
+mirror, and replay primitives.
+
+- **Account model:** every participant has its own account/session identity and a scoped facade/store
+  set. Runtime-account maps are a dynamic keyspace, so they should look like `noStrict(accountMap)`
+  rather than a fixed schema. Access checks stay in the facade/policy layer; `noStrict` is not an ACL.
+- **Wrapper over wrappers:** an app-level coordinator owns the set of per-account clients, exposed
+  facades, mirrors, replay subscriptions, and route state. "State of other accounts" is represented as
+  selected `Observe` mirrors/replay/offline resources, ideally started and stopped through
+  `createStoreManager`, not as a new global store core.
+- **Route states:** a pair of accounts may be `relay`, `direct`, `direct+shadowRelay` (audit/observe
+  copy), `blocked`, or `fallback`. Direct links are optional optimizations, useful for latency or
+  traffic cost, not a semantic change.
+- **Re-interposition:** if NDA/privacy policy, audit, moderation, throttling, reauth, direct-link
+  failure, or group topology changes require it, the relay must be able to re-enter the data path.
+  This is the reverse of direct promotion: open the replacement route, resume from the last `seq`,
+  switch consumers after catch-up, then close or demote the old route.
+- **Privacy rule:** direct account links are opt-in and policy-gated. Peers only receive the endpoint
+  and session material needed for that specific relationship; no implicit broad account discovery.
+
+Concrete next API shape, not final names:
+
+```ts
+createRouteCoordinator({
+  policy,
+  routes,
+  resources,
+}) -> {
+  pair(a, b),
+  state(pair),
+  promoteDirect(pair, opts?),
+  reinterposeRelay(pair, reason?),
+  block(pair, reason?),
+  fallback(pair, reason?),
+  onRoute(cb),
+}
+```
+
+Policy must run **before** transport promotion:
+
+- `canDirect(pair, ctx)` - may these accounts attempt direct at all?
+- `mustRelay(pair, ctx)` - force relay path because of NDA/audit/moderation/reauth.
+- `mustShadowRelay(pair, ctx)` - allow direct payload path, but keep audit/observe copy.
+- `canExposeEndpoint(pair, ctx)` - whether signaling may reveal endpoint/session material.
+- `canReinterpose(pair, ctx)` - whether/when relay is allowed or required to step back in.
+
+Minimum state machine:
+
+- `relay` -> `direct:connecting` -> `direct` -> `relay:reinterposing` -> `relay`
+- `relay` -> `direct:connecting` -> `fallback` -> `relay`
+- `direct` -> `direct+shadowRelay`
+- any state -> `blocked`
+
+Required failure modes:
+
+- direct setup never completes: keep relay active and mark fallback;
+- replacement route catches up too slowly: keep old route active and fail the switch;
+- policy changes mid-stream: re-interpose relay through replay hand-off;
+- account reauth changes facade/ACL: rebuild route policy before accepting new writes;
+- endpoint/session material is revoked: close direct and resume relay from `seq`;
+- audit/shadow route lags: decide whether to throttle, fallback, or block by policy.
+
+Acceptance tests for 0.1:
+
+- policy denial: direct is never attempted;
+- direct promotion: old relay stays live until replacement catches up;
+- failed direct: old relay continues with no data gap;
+- re-interposition: direct -> relay resumes from `seq`;
+- `direct+shadowRelay`: direct path is active while relay/audit mirror observes;
+- revocation: direct closes and relay resumes without changing facade API;
+- account map uses `noStrict`, but all access checks live in policy/facade code;
+- `createStoreManager` starts/stops selected per-account mirrors without store-core changes.
+
+Open questions: exact policy object shape; signaling envelope; whether the relay sees payloads or only
+coordinates encrypted direct streams; app-facing route-state events; backpressure across multi-hop and
+direct paths; group topology beyond a pair of accounts.
+
+Status: 🔴 not started. This belongs to project-0 architecture: no store-core change, but it defines
+the account/routing shell that later direct-connection work plugs into. This is the next important
+roadmap item; WebRTC and CRDT work should wait until this state machine is proven with fake transports.
+
 ## 1. Connection hand-off — relay ↔ direct promotion ("port forwarding") 🟡
 
 A relay/intermediary bootstraps a connection between two parties, then — on a signal — **steps out

package/doc/changes/1.0.65.md

@@ -4,4 +4,5 @@
 - Add `Observe.syncStoreReplayRoute` for store mirrors that can switch relay/direct routes without gaps or duplicate patch delivery.
 - Add `replay/route-handoff.test.ts` covering relay -> direct promotion, direct -> relay re-interposition, failed replacement fallback, and store mirror convergence.
 - Update roadmap/replay docs to mark connection hand-off as partially implemented and keep signaling/WebRTC/policy triggers explicitly open.
-- Normalize package metadata for `wenay-common2@1.0.65`.
+- Normalize package metadata for `wenay-common2@1.0.65`.
+- Add `doc/progress/` working-checkpoint rules for non-trivial tasks and keep task progress files out of package output.

package/doc/changes/1.0.66.md

@@ -0,0 +1,8 @@
+# 1.0.66
+
+- Add `Media.createAudioSource` and `Media.createVideoSource` browser capture helpers that expose media as ordinary binary `Listen` sources.
+- Add fixed-header media frame helpers: `Media.encodeMediaFrame` and `Media.decodeMediaFrame` (`Uint8Array` header + raw payload, no JSON envelope).
+- Add optional `replay:true` media source mode so `createRpcServerAuto` can expose media listeners with legacy + replay surfaces under the same key.
+- Add `wenay-common2/media` package export and root `Media` namespace export.
+- Add `replay/media-socket.test.ts` covering media binary frame headers, no-device typed state, replay media source shape, and real Socket.IO/RPC binary delivery.
+- Document media-over-socket backpressure defaults and reserve WebRTC as a future explicit SFU/signaling opt-in, not the default transport.

package/doc/wenay-common2-rare.md

@@ -205,6 +205,50 @@ Contract:
 - Static dotted keys are also supported: `api["a.b"].c` is distinct from `api.a.b.c`. Internal route/listen/cache identity must stay lossless; dotted strings are only a debug display form.
 - If a branch is a fixed public API, keep it strict. If a branch is a personal/dynamic keyspace, wrap that branch in `noStrict` instead of trying to publish all current keys as schema.
 
+## 🎙️ Media over socket — browser capture as binary Listen
+> `import { Media } from "wenay-common2"` or `import * as Media from "wenay-common2/media"`.
+> The hot path event is ONE `Uint8Array`: fixed 40-byte common2 media header + raw payload. No JSON envelope.
+```
+Media.createAudioSource(opts?) -> [emit, listen] & control
+Media.createVideoSource(opts?) -> [emit, listen] & control
+Media.encodeMediaFrame(meta, payload) -> Uint8Array
+Media.decodeMediaFrame(frame) -> {kind, codec, seq, tMono, payload, sampleRate?, channels?, nSamples?, width?, height?}
+
+control: start() -> Promise<MediaSourceState> · stop() · getStats() · setDevice(id) · listDevices()
+state: 'idle'|'requesting'|'live'|'denied'|'no-device'|'error'
+```
+Audio source:
+- default `mode:'pcm'`, `format:'int16'`, raw PCM payload; uses `AudioWorklet` when available and falls back to `ScriptProcessor` only when the browser cannot run a worklet.
+- `mode:'record'` uses `MediaRecorder` chunks (`webm-opus`) for record/upload flows, not live STT.
+- `getStats().rms` gives a VU-meter signal; permission denied/no device returns typed state, not a thrown public failure.
+
+Video source:
+- default snapshots, not a 30fps video stream: `video->canvas`, JPEG, `fps` default 3, `quality` default 0.82.
+- each frame carries absolute image bytes, so `replay:true` can safely keep the latest frame for lag recovery.
+- `worker` is API-reserved; current implementation stays main-thread. Any future worker path must transfer `ArrayBuffer` payloads, never structured-clone frame objects.
+
+Replay/RPC wiring:
+```ts
+const audio = Media.createAudioSource({sourceId: 'mic'})                 // plain lossless queue Listen
+const video = Media.createVideoSource({sourceId: 'cam', fps: 2, replay: true})
+
+createRpcServerAuto({
+    socket, socketKey: 'media',
+    object: {audio: audio[1], video: video[1]},
+    replayOpts: {highWater: 64, lowWater: 8},
+})
+```
+`replay:true` makes the returned listen a `Replay.replayListen` surface before capture emits into it, so `createRpcServerAuto` brand-detects it and exposes legacy + replay under the same key. Defaults differ by media kind: audio replay is a sacred queue (`history:1024`, no keyframe/frame, do not drop samples); video replay is keep-latest (`history:256`, `current:'last'`, `frame` returns the newest covered frame). Pass `replay:{...}` for custom history/current/frame.
+
+Backpressure rule: audio consumers should use the default queue policy unless the app explicitly accepts loss. Video consumers can use `Replay.replaySubscribe(remote.video, cb, {policy:'frame'})`; a slow socket drains to the latest frame instead of accumulating stale images. The binary frame itself is RPC-safe because `rpc-walk` passes `TypedArray`/`ArrayBuffer` leaves through natively and applies `maxBinaryLen`.
+
+WebRTC future contract:
+- `transport:'socket'` is the implemented default today.
+- `transport:'webrtc'` is reserved and currently reports `state:'error'` on `start()`; it is not a hidden second transport.
+- Future WebRTC support must be explicit opt-in for sub-200ms human duplex. Signaling belongs on the existing socket/RPC control channel (offer/answer/ICE), and backend/AI access requires an SFU that re-emits media bytes into the same `Media` Listen/replay surface. Downstream RPC/replay/store consumers must not change.
+
+Oracle: `npx ts-node replay/media-socket.test.ts` checks header decode, plain Listen shape, `replay:true`, typed no-device state in Node, and real Socket.IO binary delivery.
+
 ## 📈 exchange — params (`CParams`)
 ```
 class CParams / CParamsReadonly implements IParams

package/doc/wenay-common2.md

@@ -143,6 +143,17 @@ await l.ticks.frame(mySeq)                                // pull at YOUR pace (
 // full guide + examples → rpc.md; frame model / lag policies → 🎞️ recipe below and rare docs
 ```
 
+## 🎙️ Media over socket — binary Listen frames
+```
+import { Media } from "wenay-common2"        // or: import * as Media from "wenay-common2/media"
+
+Media.createAudioSource({format?: 'int16'|'float32', mode?: 'pcm'|'record', replay?}) -> [emit, listen] & control
+Media.createVideoSource({fps? = 3, codec? = 'jpeg', quality?, replay?}) -> [emit, listen] & control
+control: start() -> Promise<'idle'|'requesting'|'live'|'denied'|'no-device'|'error'> · stop() · getStats() · setDevice(id) · listDevices() · state
+Media.encodeMediaFrame(meta, payload) / Media.decodeMediaFrame(frame)     // one Uint8Array = 40-byte fixed header + raw payload
+```
+Audio default is PCM frames from `AudioWorklet` where available (`mode:'record'` uses MediaRecorder chunks). Video default is camera snapshots (`video->canvas`, JPEG, low fps for vision). Put `listen` into `createRpcServerAuto` like any other Listen; with `replay:true`, the returned listen is a replay line, so RPC auto exposes legacy + replay surfaces under the same key. Backpressure policy: audio is lossless queue; video `replay:true` defaults to keep-latest frame recovery. `transport:'webrtc'` is reserved for a future SFU/signaling adapter; socket binary is the default today.
+
 ## 🔁 Observe — reactive state + store/mirror API
 > `import { Observe } from "wenay-common2"` or `import * as Observe from "wenay-common2/observe"`.
 > This is the documented v2 reactive/store surface.

package/lib/Common/media/media-index.d.ts

@@ -0,0 +1 @@
+export * from './media-source';

package/lib/Common/media/media-index.js

@@ -0,0 +1,17 @@
+"use strict";
+var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    var desc = Object.getOwnPropertyDescriptor(m, k);
+    if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) {
+      desc = { enumerable: true, get: function() { return m[k]; } };
+    }
+    Object.defineProperty(o, k2, desc);
+}) : (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    o[k2] = m[k];
+}));
+var __exportStar = (this && this.__exportStar) || function(m, exports) {
+    for (var p in m) if (p !== "default" && !Object.prototype.hasOwnProperty.call(exports, p)) __createBinding(exports, m, p);
+};
+Object.defineProperty(exports, "__esModule", { value: true });
+__exportStar(require("./media-source"), exports);

package/lib/Common/media/media-source.d.ts

@@ -0,0 +1,94 @@
+import { Listener, ListenApi } from '../events/Listen';
+import { ListenReplayApi, ReplayListenUseOptions } from '../events/replay-listen';
+export type MediaSourceKind = 'audio' | 'video';
+export type MediaSourceState = 'idle' | 'requesting' | 'live' | 'denied' | 'no-device' | 'error';
+export type MediaFrameKind = 'audio-pcm' | 'audio-record' | 'video-frame';
+export type MediaFrameCodec = 'pcm16' | 'float32' | 'jpeg' | 'png' | 'webp' | 'webm-opus';
+export type MediaTransportMode = 'socket' | 'webrtc';
+export type MediaSourceDevice = {
+    deviceId: string;
+    label: string;
+    kind: string;
+};
+export type MediaReplayOpts = true | ReplayListenUseOptions<[Uint8Array]>;
+export type AudioSourceOpts = {
+    sourceId?: string;
+    deviceId?: string;
+    mode?: 'pcm' | 'record';
+    format?: 'int16' | 'float32';
+    channels?: number;
+    sampleRate?: number;
+    worklet?: boolean;
+    bufferSize?: number;
+    recordMimeType?: string;
+    recordTimesliceMs?: number;
+    transport?: MediaTransportMode;
+    replay?: MediaReplayOpts;
+};
+export type VideoSourceOpts = {
+    sourceId?: string;
+    deviceId?: string;
+    fps?: number;
+    width?: number;
+    height?: number;
+    codec?: 'jpeg' | 'png' | 'webp';
+    quality?: number;
+    video?: any;
+    canvas?: any;
+    worker?: false | {
+        transferable?: true;
+    };
+    transport?: MediaTransportMode;
+    replay?: MediaReplayOpts;
+};
+export type MediaStats = {
+    sourceId: string;
+    kind: MediaSourceKind;
+    state: MediaSourceState;
+    seq: number;
+    frames: number;
+    bytes: number;
+    dropped: number;
+    startedAt: number;
+    lastAt: number;
+    fps: number;
+    rms?: number;
+    error?: string;
+};
+export type MediaSourceControl = {
+    start(): Promise<MediaSourceState>;
+    stop(): void;
+    getStats(): MediaStats;
+    setDevice(id: string): Promise<MediaSourceState>;
+    listDevices(): Promise<MediaSourceDevice[]>;
+    readonly state: MediaSourceState;
+    readonly sourceId: string;
+    readonly kind: MediaSourceKind;
+    readonly transport: MediaTransportMode;
+};
+export type MediaListen = ListenApi<[Uint8Array]>;
+export type MediaReplayListen = ListenReplayApi<[Uint8Array]>;
+export type MediaAnyListen = MediaListen | MediaReplayListen;
+export type MediaEmit = Listener<[Uint8Array]>;
+export type MediaSource = readonly [MediaEmit, MediaAnyListen] & MediaSourceControl;
+export type MediaFrameMeta = {
+    kind: MediaFrameKind;
+    codec: MediaFrameCodec;
+    seq: number;
+    tMono: number;
+    sampleRate?: number;
+    channels?: number;
+    nSamples?: number;
+    width?: number;
+    height?: number;
+};
+export type DecodedMediaFrame = MediaFrameMeta & {
+    payload: Uint8Array;
+};
+export declare const MEDIA_FRAME_MAGIC = 1464028466;
+export declare const MEDIA_FRAME_VERSION = 1;
+export declare const MEDIA_FRAME_HEADER_BYTES = 40;
+export declare function encodeMediaFrame(meta: MediaFrameMeta, payload: ArrayBuffer | ArrayBufferView): Uint8Array<ArrayBuffer>;
+export declare function decodeMediaFrame(frameLike: ArrayBuffer | ArrayBufferView): DecodedMediaFrame;
+export declare function createAudioSource(opts?: AudioSourceOpts): MediaSource;
+export declare function createVideoSource(opts?: VideoSourceOpts): MediaSource;

package/lib/Common/media/media-source.js

@@ -0,0 +1,537 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.MEDIA_FRAME_HEADER_BYTES = exports.MEDIA_FRAME_VERSION = exports.MEDIA_FRAME_MAGIC = void 0;
+exports.encodeMediaFrame = encodeMediaFrame;
+exports.decodeMediaFrame = decodeMediaFrame;
+exports.createAudioSource = createAudioSource;
+exports.createVideoSource = createVideoSource;
+const Listen_1 = require("../events/Listen");
+const replay_listen_1 = require("../events/replay-listen");
+exports.MEDIA_FRAME_MAGIC = 0x57434d32;
+exports.MEDIA_FRAME_VERSION = 1;
+exports.MEDIA_FRAME_HEADER_BYTES = 40;
+const KIND_TO_CODE = {
+    'audio-pcm': 1,
+    'audio-record': 2,
+    'video-frame': 3,
+};
+const CODE_TO_KIND = {
+    1: 'audio-pcm',
+    2: 'audio-record',
+    3: 'video-frame',
+};
+const CODEC_TO_CODE = {
+    pcm16: 1,
+    float32: 2,
+    jpeg: 10,
+    png: 11,
+    webp: 12,
+    'webm-opus': 20,
+};
+const CODE_TO_CODEC = {
+    1: 'pcm16',
+    2: 'float32',
+    10: 'jpeg',
+    11: 'png',
+    12: 'webp',
+    20: 'webm-opus',
+};
+function nowMono() {
+    const perf = globalThis.performance;
+    return typeof perf?.now == 'function' ? perf.now() : Date.now();
+}
+function toBytes(data) {
+    if (ArrayBuffer.isView(data))
+        return new Uint8Array(data.buffer, data.byteOffset, data.byteLength);
+    return new Uint8Array(data);
+}
+function clonePayload(data) {
+    const src = toBytes(data);
+    const out = new Uint8Array(src.byteLength);
+    out.set(src);
+    return out;
+}
+function writeU32(view, offset, value) {
+    view.setUint32(offset, Math.max(0, Math.min(0xffffffff, Math.floor(value ?? 0))), true);
+}
+function readPayload(frame, headerBytes) {
+    return frame.subarray(headerBytes);
+}
+function encodeMediaFrame(meta, payload) {
+    const body = toBytes(payload);
+    const out = new Uint8Array(exports.MEDIA_FRAME_HEADER_BYTES + body.byteLength);
+    const view = new DataView(out.buffer, out.byteOffset, exports.MEDIA_FRAME_HEADER_BYTES);
+    view.setUint32(0, exports.MEDIA_FRAME_MAGIC, true);
+    view.setUint8(4, exports.MEDIA_FRAME_VERSION);
+    view.setUint8(5, KIND_TO_CODE[meta.kind]);
+    view.setUint8(6, CODEC_TO_CODE[meta.codec]);
+    view.setUint8(7, 0);
+    view.setUint16(8, exports.MEDIA_FRAME_HEADER_BYTES, true);
+    view.setUint16(10, 0, true);
+    writeU32(view, 12, meta.seq);
+    view.setFloat64(16, meta.tMono, true);
+    if (meta.kind == 'video-frame') {
+        writeU32(view, 24, meta.width);
+        writeU32(view, 28, meta.height);
+    }
+    else {
+        writeU32(view, 24, meta.sampleRate);
+        writeU32(view, 28, meta.channels);
+        writeU32(view, 32, meta.nSamples);
+    }
+    writeU32(view, 36, body.byteLength);
+    out.set(body, exports.MEDIA_FRAME_HEADER_BYTES);
+    return out;
+}
+function decodeMediaFrame(frameLike) {
+    const frame = toBytes(frameLike);
+    if (frame.byteLength < exports.MEDIA_FRAME_HEADER_BYTES)
+        throw new Error('media frame too short');
+    const view = new DataView(frame.buffer, frame.byteOffset, exports.MEDIA_FRAME_HEADER_BYTES);
+    if (view.getUint32(0, true) != exports.MEDIA_FRAME_MAGIC)
+        throw new Error('media frame magic mismatch');
+    const version = view.getUint8(4);
+    if (version != exports.MEDIA_FRAME_VERSION)
+        throw new Error(`media frame version ${version} is not supported`);
+    const kind = CODE_TO_KIND[view.getUint8(5)];
+    const codec = CODE_TO_CODEC[view.getUint8(6)];
+    if (!kind || !codec)
+        throw new Error('media frame kind/codec is not supported');
+    const headerBytes = view.getUint16(8, true);
+    if (headerBytes < exports.MEDIA_FRAME_HEADER_BYTES || headerBytes > frame.byteLength)
+        throw new Error('media frame header size is invalid');
+    const payloadBytes = view.getUint32(36, true);
+    const payload = readPayload(frame, headerBytes);
+    if (payload.byteLength != payloadBytes)
+        throw new Error('media frame payload size mismatch');
+    const base = {
+        kind,
+        codec,
+        seq: view.getUint32(12, true),
+        tMono: view.getFloat64(16, true),
+        payload,
+    };
+    if (kind == 'video-frame') {
+        return {
+            ...base,
+            width: view.getUint32(24, true),
+            height: view.getUint32(28, true),
+        };
+    }
+    return {
+        ...base,
+        sampleRate: view.getUint32(24, true),
+        channels: view.getUint32(28, true),
+        nSamples: view.getUint32(32, true),
+    };
+}
+function createStats(sourceId, kind) {
+    return {
+        sourceId,
+        kind,
+        state: 'idle',
+        seq: 0,
+        frames: 0,
+        bytes: 0,
+        dropped: 0,
+        startedAt: 0,
+        lastAt: 0,
+        fps: 0,
+    };
+}
+function hasGetUserMedia() {
+    const nav = globalThis.navigator;
+    return typeof nav?.mediaDevices?.getUserMedia == 'function';
+}
+async function listDevices(kind) {
+    const nav = globalThis.navigator;
+    if (typeof nav?.mediaDevices?.enumerateDevices != 'function')
+        return [];
+    const devices = await nav.mediaDevices.enumerateDevices();
+    const prefix = kind == 'audio' ? 'audioinput' : 'videoinput';
+    return devices
+        .filter(d => d?.kind == prefix)
+        .map(d => ({ deviceId: String(d.deviceId ?? ''), label: String(d.label ?? ''), kind: String(d.kind ?? '') }));
+}
+function stopTracks(stream) {
+    const tracks = typeof stream?.getTracks == 'function' ? stream.getTracks() : [];
+    for (const track of tracks)
+        track.stop?.();
+}
+function stateFromMediaError(e) {
+    const name = String(e?.name ?? '');
+    if (name == 'NotAllowedError' || name == 'SecurityError' || name == 'PermissionDeniedError')
+        return 'denied';
+    if (name == 'NotFoundError' || name == 'DevicesNotFoundError')
+        return 'no-device';
+    return 'error';
+}
+function defaultReplayOptions(kind) {
+    if (kind == 'video') {
+        return {
+            history: 256,
+            current: 'last',
+            frame: tail => tail.length ? [tail[tail.length - 1]] : [],
+        };
+    }
+    return { history: 1024 };
+}
+function createSourceShell(kind, sourceId, transport, replay) {
+    const replayOpts = replay == true ? defaultReplayOptions(kind) : replay;
+    const [emit, listenApi] = replayOpts ? (0, replay_listen_1.replayListen)(replayOpts) : (0, Listen_1.listen)();
+    const stats = createStats(sourceId, kind);
+    let state = 'idle';
+    let error;
+    function setState(next, err) {
+        state = next;
+        stats.state = next;
+        error = err ? String(err?.message ?? err) : undefined;
+        stats.error = error;
+    }
+    function emitFrame(frame, seq) {
+        stats.seq = seq;
+        stats.frames++;
+        stats.bytes += frame.byteLength;
+        stats.lastAt = nowMono();
+        const age = Math.max(1, stats.lastAt - (stats.startedAt || stats.lastAt));
+        stats.fps = Math.round((stats.frames * 10000) / age) / 10;
+        emit(frame);
+    }
+    return { emit, listenApi, stats, get state() { return state; }, get error() { return error; }, setState, emitFrame };
+}
+function attachControl(pair, control) {
+    return Object.assign(pair, control);
+}
+function ensureSocketTransport(mode) {
+    if (mode == 'webrtc') {
+        throw new Error('WebRTC media transport is reserved for a future opt-in SFU/signaling adapter; use transport:"socket" today');
+    }
+}
+function int16PcmFromFloat(input) {
+    const out = new Int16Array(input.length);
+    for (let i = 0; i < input.length; i++) {
+        const x = Math.max(-1, Math.min(1, input[i]));
+        out[i] = x < 0 ? x * 0x8000 : x * 0x7fff;
+    }
+    return new Uint8Array(out.buffer);
+}
+function interleaveChannels(buffers, frames, channels) {
+    const out = new Float32Array(frames * channels);
+    for (let i = 0; i < frames; i++) {
+        for (let ch = 0; ch < channels; ch++)
+            out[i * channels + ch] = buffers[ch]?.[i] ?? 0;
+    }
+    return out;
+}
+function rmsOf(input) {
+    if (input.length == 0)
+        return 0;
+    let sum = 0;
+    for (let i = 0; i < input.length; i++)
+        sum += input[i] * input[i];
+    return Math.sqrt(sum / input.length);
+}
+function audioWorkletCode() {
+    return `
+class WenayCommon2PcmProcessor extends AudioWorkletProcessor {
+    process(inputs) {
+        const input = inputs[0]
+        if (!input || !input.length || !input[0]) return true
+        const channels = input.length
+        const frames = input[0].length
+        const out = new Float32Array(frames * channels)
+        for (let i = 0; i < frames; i++) {
+            for (let ch = 0; ch < channels; ch++) out[i * channels + ch] = input[ch] ? input[ch][i] : 0
+        }
+        this.port.postMessage({sampleRate, channels, frames, samples: out}, [out.buffer])
+        return true
+    }
+}
+registerProcessor('wenay-common2-pcm', WenayCommon2PcmProcessor)
+`;
+}
+function createAudioSource(opts = {}) {
+    const sourceId = opts.sourceId ?? 'audio';
+    const transport = opts.transport ?? 'socket';
+    const shell = createSourceShell('audio', sourceId, transport, opts.replay);
+    let deviceId = opts.deviceId;
+    let stream = null;
+    let audioCtx = null;
+    let mediaNode = null;
+    let workletNode = null;
+    let recorder = null;
+    let seq = 0;
+    function emitPcm(samples, sampleRate, channels, frames) {
+        const codec = opts.format == 'float32' ? 'float32' : 'pcm16';
+        const payload = codec == 'float32' ? new Uint8Array(samples.buffer, samples.byteOffset, samples.byteLength) : int16PcmFromFloat(samples);
+        shell.stats.rms = rmsOf(samples);
+        shell.emitFrame(encodeMediaFrame({
+            kind: 'audio-pcm',
+            codec,
+            seq: ++seq,
+            tMono: nowMono(),
+            sampleRate,
+            channels,
+            nSamples: frames,
+        }, payload), seq);
+    }
+    async function startPcm(nextStream) {
+        const AudioContextCtor = globalThis.AudioContext ?? globalThis.webkitAudioContext;
+        if (!AudioContextCtor)
+            throw new Error('AudioContext is not available');
+        audioCtx = new AudioContextCtor(opts.sampleRate ? { sampleRate: opts.sampleRate } : undefined);
+        mediaNode = audioCtx.createMediaStreamSource(nextStream);
+        if (opts.worklet != false && audioCtx.audioWorklet && globalThis.Blob && globalThis.URL) {
+            const blob = new globalThis.Blob([audioWorkletCode()], { type: 'text/javascript' });
+            const url = globalThis.URL.createObjectURL(blob);
+            try {
+                await audioCtx.audioWorklet.addModule(url);
+            }
+            finally {
+                globalThis.URL.revokeObjectURL(url);
+            }
+            const AudioWorkletNodeCtor = globalThis.AudioWorkletNode;
+            workletNode = new AudioWorkletNodeCtor(audioCtx, 'wenay-common2-pcm', {
+                numberOfInputs: 1,
+                numberOfOutputs: 0,
+                channelCount: opts.channels ?? 1,
+            });
+            workletNode.port.onmessage = function onWorkletSamples(ev) {
+                const data = ev.data ?? {};
+                emitPcm(data.samples, data.sampleRate ?? audioCtx.sampleRate, data.channels ?? 1, data.frames ?? 0);
+            };
+            mediaNode.connect(workletNode);
+            return;
+        }
+        const channels = opts.channels ?? 1;
+        const processor = audioCtx.createScriptProcessor(opts.bufferSize ?? 2048, channels, channels);
+        processor.onaudioprocess = function onAudioProcess(ev) {
+            const input = ev.inputBuffer;
+            const frames = input.length;
+            const buffers = [];
+            for (let ch = 0; ch < channels; ch++)
+                buffers.push(input.getChannelData(Math.min(ch, input.numberOfChannels - 1)));
+            emitPcm(interleaveChannels(buffers, frames, channels), input.sampleRate ?? audioCtx.sampleRate, channels, frames);
+            const out = ev.outputBuffer;
+            if (out)
+                for (let ch = 0; ch < out.numberOfChannels; ch++)
+                    out.getChannelData(ch).fill(0);
+        };
+        mediaNode.connect(processor);
+        processor.connect(audioCtx.destination);
+        workletNode = processor;
+    }
+    function startRecord(nextStream) {
+        const Recorder = globalThis.MediaRecorder;
+        if (!Recorder)
+            throw new Error('MediaRecorder is not available');
+        const mimeType = opts.recordMimeType ?? 'audio/webm;codecs=opus';
+        recorder = new Recorder(nextStream, Recorder.isTypeSupported?.(mimeType) ? { mimeType } : undefined);
+        recorder.ondataavailable = async function onRecordChunk(ev) {
+            if (!ev.data || ev.data.size == 0)
+                return;
+            const payload = new Uint8Array(await ev.data.arrayBuffer());
+            shell.emitFrame(encodeMediaFrame({
+                kind: 'audio-record',
+                codec: 'webm-opus',
+                seq: ++seq,
+                tMono: nowMono(),
+            }, payload), seq);
+        };
+        recorder.start(opts.recordTimesliceMs ?? 1000);
+    }
+    function stop() {
+        recorder?.stop?.();
+        recorder = null;
+        workletNode?.disconnect?.();
+        mediaNode?.disconnect?.();
+        audioCtx?.close?.();
+        stopTracks(stream);
+        stream = null;
+        audioCtx = null;
+        mediaNode = null;
+        workletNode = null;
+        shell.setState('idle');
+    }
+    async function start() {
+        try {
+            ensureSocketTransport(transport);
+            if (!hasGetUserMedia()) {
+                shell.setState('no-device');
+                return shell.state;
+            }
+            stop();
+            shell.setState('requesting');
+            shell.stats.startedAt = nowMono();
+            const constraints = { audio: { deviceId: deviceId ? { exact: deviceId } : undefined, channelCount: opts.channels, sampleRate: opts.sampleRate } };
+            stream = await globalThis.navigator.mediaDevices.getUserMedia(constraints);
+            if (opts.mode == 'record')
+                startRecord(stream);
+            else
+                await startPcm(stream);
+            shell.setState('live');
+            return shell.state;
+        }
+        catch (e) {
+            stopTracks(stream);
+            stream = null;
+            shell.setState(stateFromMediaError(e), e);
+            return shell.state;
+        }
+    }
+    const control = {
+        start,
+        stop,
+        getStats: () => ({ ...shell.stats }),
+        setDevice: async (id) => {
+            deviceId = id;
+            return shell.state == 'live' ? start() : shell.state;
+        },
+        listDevices: () => listDevices('audio'),
+        get state() { return shell.state; },
+        sourceId,
+        kind: 'audio',
+        transport,
+    };
+    return attachControl([shell.emit, shell.listenApi], control);
+}
+function mimeForVideo(codec) {
+    if (codec == 'png')
+        return 'image/png';
+    if (codec == 'webp')
+        return 'image/webp';
+    return 'image/jpeg';
+}
+function canvasToBlob(canvas, mime, quality) {
+    if (typeof canvas.convertToBlob == 'function')
+        return canvas.convertToBlob({ type: mime, quality });
+    return new Promise((resolve, reject) => {
+        if (typeof canvas.toBlob != 'function') {
+            reject(new Error('canvas.toBlob is not available'));
+            return;
+        }
+        canvas.toBlob((blob) => blob ? resolve(blob) : reject(new Error('canvas.toBlob returned null')), mime, quality);
+    });
+}
+function createVideoSource(opts = {}) {
+    const sourceId = opts.sourceId ?? 'video';
+    const transport = opts.transport ?? 'socket';
+    const shell = createSourceShell('video', sourceId, transport, opts.replay);
+    let deviceId = opts.deviceId;
+    let stream = null;
+    let video = opts.video;
+    let canvas = opts.canvas;
+    let ctx = null;
+    let timer = null;
+    let busy = false;
+    let seq = 0;
+    async function captureOne() {
+        if (busy || shell.state != 'live') {
+            shell.stats.dropped++;
+            return;
+        }
+        busy = true;
+        try {
+            const w = opts.width ?? video.videoWidth ?? video.width ?? 0;
+            const h = opts.height ?? video.videoHeight ?? video.height ?? 0;
+            if (!w || !h)
+                return;
+            if (canvas.width != w)
+                canvas.width = w;
+            if (canvas.height != h)
+                canvas.height = h;
+            ctx.drawImage(video, 0, 0, w, h);
+            const codec = opts.codec ?? 'jpeg';
+            const blob = await canvasToBlob(canvas, mimeForVideo(codec), opts.quality ?? 0.82);
+            const payload = new Uint8Array(await blob.arrayBuffer());
+            shell.emitFrame(encodeMediaFrame({
+                kind: 'video-frame',
+                codec,
+                seq: ++seq,
+                tMono: nowMono(),
+                width: w,
+                height: h,
+            }, payload), seq);
+        }
+        catch (e) {
+            shell.setState('error', e);
+        }
+        finally {
+            busy = false;
+        }
+    }
+    function stop() {
+        if (timer) {
+            clearInterval(timer);
+            timer = null;
+        }
+        stopTracks(stream);
+        stream = null;
+        if (!opts.video && video) {
+            try {
+                video.srcObject = null;
+            }
+            catch { }
+        }
+        shell.setState('idle');
+    }
+    async function start() {
+        try {
+            ensureSocketTransport(transport);
+            if (!hasGetUserMedia()) {
+                shell.setState('no-device');
+                return shell.state;
+            }
+            const doc = globalThis.document;
+            if (!video && !doc?.createElement) {
+                shell.setState('error', 'document.createElement is not available');
+                return shell.state;
+            }
+            stop();
+            shell.setState('requesting');
+            shell.stats.startedAt = nowMono();
+            stream = await globalThis.navigator.mediaDevices.getUserMedia({
+                video: {
+                    deviceId: deviceId ? { exact: deviceId } : undefined,
+                    width: opts.width,
+                    height: opts.height,
+                },
+            });
+            video = video ?? doc.createElement('video');
+            video.muted = true;
+            video.playsInline = true;
+            video.srcObject = stream;
+            await video.play?.();
+            canvas = canvas ?? doc.createElement('canvas');
+            ctx = canvas.getContext('2d');
+            if (!ctx)
+                throw new Error('2d canvas context is not available');
+            shell.setState('live');
+            const ms = Math.max(1, Math.round(1000 / (opts.fps ?? 3)));
+            await captureOne();
+            timer = setInterval(captureOne, ms);
+            return shell.state;
+        }
+        catch (e) {
+            stopTracks(stream);
+            stream = null;
+            shell.setState(stateFromMediaError(e), e);
+            return shell.state;
+        }
+    }
+    const control = {
+        start,
+        stop,
+        getStats: () => ({ ...shell.stats }),
+        setDevice: async (id) => {
+            deviceId = id;
+            return shell.state == 'live' ? start() : shell.state;
+        },
+        listDevices: () => listDevices('video'),
+        get state() { return shell.state; },
+        sourceId,
+        kind: 'video',
+        transport,
+    };
+    return attachControl([shell.emit, shell.listenApi], control);
+}

package/lib/client.d.ts

@@ -13,6 +13,7 @@ export * from "./Common/events/event";
 export * from "./Common/events/SocketBuffer";
 export * from "./Common/events/SocketServerHook";
 export * from "./Common/events/joinListens";
+export * from "./Common/media/media-index";
 export * from "./Common/rcp/rpc-index";
 export * from "./Common/Color";
 export * from "./Common/funcTimeWait";
@@ -27,3 +28,4 @@ export * as BaseTypes from "./Common/core/BaseTypes";
 export * as Params from "./Exchange/CParams";
 export * as Time from "./Common/Time";
 export * as Color from "./Common/Color";
+export * as Media from "./Common/media/media-index";

package/lib/client.js

@@ -36,7 +36,7 @@ var __importStar = (this && this.__importStar) || (function () {
     };
 })();
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.Color = exports.Time = exports.Params = exports.BaseTypes = exports.Bars = exports.Math = void 0;
+exports.Media = exports.Color = exports.Time = exports.Params = exports.BaseTypes = exports.Bars = exports.Math = void 0;
 __exportStar(require("./Common/core/Decorator"), exports);
 __exportStar(require("./Common/core/BaseTypes"), exports);
 __exportStar(require("./Common/core/common"), exports);
@@ -52,6 +52,7 @@ __exportStar(require("./Common/events/event"), exports);
 __exportStar(require("./Common/events/SocketBuffer"), exports);
 __exportStar(require("./Common/events/SocketServerHook"), exports);
 __exportStar(require("./Common/events/joinListens"), exports);
+__exportStar(require("./Common/media/media-index"), exports);
 __exportStar(require("./Common/rcp/rpc-index"), exports);
 __exportStar(require("./Common/Color"), exports);
 __exportStar(require("./Common/funcTimeWait"), exports);
@@ -66,3 +67,4 @@ exports.BaseTypes = __importStar(require("./Common/core/BaseTypes"));
 exports.Params = __importStar(require("./Exchange/CParams"));
 exports.Time = __importStar(require("./Common/Time"));
 exports.Color = __importStar(require("./Common/Color"));
+exports.Media = __importStar(require("./Common/media/media-index"));

package/lib/index.d.ts

@@ -17,6 +17,7 @@ export * from "./Common/events/SocketBuffer";
 export * from "./Common/events/SocketServerHook";
 export * from "./Common/events/joinListens";
 export * from "./Common/events/mapListen";
+export * from "./Common/media/media-index";
 export * from "./Common/rcp/rpc-index";
 export * from "./Common/Color";
 export * from "./Common/funcTimeWait";
@@ -33,3 +34,4 @@ export * as Time from "./Common/Time";
 export * as Color from "./Common/Color";
 export * as Observe from "./Common/Observe";
 export * as Replay from "./Common/events/replay-index";
+export * as Media from "./Common/media/media-index";

package/lib/index.js

@@ -36,7 +36,7 @@ var __importStar = (this && this.__importStar) || (function () {
     };
 })();
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.Replay = exports.Observe = exports.Color = exports.Time = exports.Params = exports.Bars = exports.Math = void 0;
+exports.Media = exports.Replay = exports.Observe = exports.Color = exports.Time = exports.Params = exports.Bars = exports.Math = void 0;
 __exportStar(require("./Common/node_console"), exports);
 __exportStar(require("./Common/core/Decorator"), exports);
 __exportStar(require("./Common/core/BaseTypes"), exports);
@@ -56,6 +56,7 @@ __exportStar(require("./Common/events/SocketBuffer"), exports);
 __exportStar(require("./Common/events/SocketServerHook"), exports);
 __exportStar(require("./Common/events/joinListens"), exports);
 __exportStar(require("./Common/events/mapListen"), exports);
+__exportStar(require("./Common/media/media-index"), exports);
 __exportStar(require("./Common/rcp/rpc-index"), exports);
 __exportStar(require("./Common/Color"), exports);
 __exportStar(require("./Common/funcTimeWait"), exports);
@@ -72,3 +73,4 @@ exports.Time = __importStar(require("./Common/Time"));
 exports.Color = __importStar(require("./Common/Color"));
 exports.Observe = __importStar(require("./Common/Observe"));
 exports.Replay = __importStar(require("./Common/events/replay-index"));
+exports.Media = __importStar(require("./Common/media/media-index"));

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "wenay-common2",
-  "version": "1.0.65",
+  "version": "1.0.66",
   "description": "Common library",
   "strict": true,
   "main": "lib/index.js",
@@ -11,6 +11,7 @@
     "CLAUDE.md",
     "rpc.md",
     "!doc/target/**/*",
+    "!doc/progress/**/*",
     "!**/*.tsbuildinfo"
   ],
   "author": "wenay",
@@ -33,7 +34,8 @@
     "./lib/server": "./lib/server.js",
     "./client": "./lib/client.js",
     "./server": "./lib/server.js",
-    "./package.json": "./package.json"
+    "./package.json": "./package.json",
+    "./media": "./lib/Common/media/media-index.js"
   },
   "dependencies": {
     "express": "^5.2.1"