@cyberart-io/engine 0.0.5 → 0.0.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +117 -14
- package/dist/headless.d.ts +483 -6
- package/dist/headless.js +1 -1
- package/dist/index.d.ts +883 -6
- package/dist/index.js +1 -1
- package/docs/audio.md +152 -0
- package/docs/calculation-carts.md +134 -0
- package/docs/capability-manifest.md +5 -2
- package/docs/compositor.md +8 -3
- package/docs/executable-modules.md +112 -0
- package/docs/midi.md +128 -0
- package/docs/runtime-group.md +3 -3
- package/docs/snapshots.md +102 -0
- package/docs/visual-layers.md +151 -0
- package/package.json +1 -1
package/docs/audio.md
ADDED
|
@@ -0,0 +1,152 @@
|
|
|
1
|
+
# Audio unlock broker and cues
|
|
2
|
+
|
|
3
|
+
Page-level unlock, channel mix, and frame-stepped cue scheduling. One user gesture unlocks authorized carts onto a single audio graph. Cue traces are deterministic; **PCM output is not**.
|
|
4
|
+
|
|
5
|
+
Back to the [package README](../README.md). Related: [presentation cue](presentation-cue.md) (scheduling), [asset resolver](asset-resolver.md) (`ASSET_READY_EVENT` / `ASSET_FAILED_EVENT`), [runtime group](runtime-group.md), [deterministic mode](deterministic-mode.md).
|
|
6
|
+
|
|
7
|
+
## One-command reproduce (this repo)
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
pnpm exec vitest run packages/engine/src/canvas/cyb-67-audio.repro.spec.ts
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
## Why this exists
|
|
14
|
+
|
|
15
|
+
Browser autoplay policy, load timing, and several carts on one page make it easy to start extra `AudioContext`s or invent per-cart mute/timing rules. Hosts create **one** `createAudioBroker` and pass it into `createRuntime` / `createRuntimeGroup`. Carts that only set `metadata.audio: 'tone'` keep the existing `start()` / `unlockAudio()` path when the broker is omitted.
|
|
16
|
+
|
|
17
|
+
Silent carts never import Tone. Destroying a cart tears down that participant’s channels; it does not close Tone for remaining carts. There is still **one audio graph per page**.
|
|
18
|
+
|
|
19
|
+
## `createAudioBroker(options?)`
|
|
20
|
+
|
|
21
|
+
```ts
|
|
22
|
+
import { createAudioBroker, createRuntime, createRuntimeGroup } from '@cyberart-io/engine';
|
|
23
|
+
|
|
24
|
+
const broker = createAudioBroker({
|
|
25
|
+
// tests inject a no-op; default dynamically loads the Tone adapter
|
|
26
|
+
toneStart: async () => undefined,
|
|
27
|
+
reducedSensory: false,
|
|
28
|
+
});
|
|
29
|
+
|
|
30
|
+
await broker.unlock(); // first call runs toneStart; later calls share status
|
|
31
|
+
broker.authorize('effects');
|
|
32
|
+
broker.authorize('ambience');
|
|
33
|
+
|
|
34
|
+
const group = createRuntimeGroup({
|
|
35
|
+
audioBroker: broker,
|
|
36
|
+
participants: [/* … */],
|
|
37
|
+
});
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
| Member | Meaning |
|
|
41
|
+
|---|---|
|
|
42
|
+
| `unlock()` | Idempotent. First call runs `toneStart`. Default dynamically loads the Tone adapter: `load()` when `navigator.userAgent` matches `/Headless/i` (same as `AnimationManager.unlockAudio`), otherwise `unlock()`. Returns `{ state, error? }`. |
|
|
43
|
+
| `status()` | `locked` / `unlocking` / `unlocked` / `failed`. |
|
|
44
|
+
| `authorize(id)` / `revoke(id)` / `isAuthorized(id)` | Runtime-group participant ids. Empty set = all cues allowed (single-cart host). |
|
|
45
|
+
| `setChannelGain(id, gain)` | Channel gain in `0…1`. |
|
|
46
|
+
| `setPriority(id, n)` | Higher-priority active cues duck lower-priority channels (`DEFAULT_DUCK_GAIN` = `0.25`). |
|
|
47
|
+
| `mute()` / `unmute()` | Page mute. Cues still record; playback is skipped (`audio.cue.skipped`, reason `muted`). |
|
|
48
|
+
| `muteChannel(id)` / `unmuteChannel(id)` | Per-channel mute. |
|
|
49
|
+
| `duck(id, gain?)` / `unduck(id)` | Manual duck; `unduck` restores auto priority ducking. |
|
|
50
|
+
| `effectiveGain(id)` | `0` when page/channel muted, else `gain * duckGain`. |
|
|
51
|
+
| `handleHostEvent(event)` | Records `ASSET_READY_EVENT` / `ASSET_FAILED_EVENT` by payload `id`. |
|
|
52
|
+
| `teardown(participantId)` | Drop that cart’s authorization, channels, and active cues. Does **not** close Tone. |
|
|
53
|
+
| `inspect()` | JSON-serializable `{ status, reducedSensory, muted, authorized, channels, assets }`. |
|
|
54
|
+
| `destroy()` | Tear down all participants and listeners. Unlock status stays `unlocked` if it was; the shared context is left running. |
|
|
55
|
+
|
|
56
|
+
`reducedSensory: true` skips playback and still records cue events (`audio.cue.skipped`, reason `reduced-sensory`). It does **not** pass presentation `reducedMotion`, so authored `delayFrames` still appear on `atFrame`.
|
|
57
|
+
|
|
58
|
+
`createAudioCueTimeline()` with **no** broker starts cues (no unauthorized skip). Pass a broker when you need authorization, mute, or asset readiness.
|
|
59
|
+
|
|
60
|
+
Repeat (`CueSpec.repeat`) keeps the audio record across cycles: each presentation `cue.started` can emit another `audio.cue.started`, and `noteCueEnded` runs at the end of a cycle so ducking does not stick after the first loop.
|
|
61
|
+
|
|
62
|
+
`createRuntime({ audioBroker, audioParticipantId })` authorizes that id, routes mailbox `ASSET_*` events into the broker, and `teardown`s the id on `runtime.destroy()`. `createRuntimeGroup({ audioBroker })` passes each participant id. Carts that only declare `metadata.audio: 'tone'` still unlock on `start()` / `unlockAudio()` when **no** broker is passed.
|
|
63
|
+
|
|
64
|
+
## `createAudioCueTimeline(options?)` / `scheduleAudioCue`
|
|
65
|
+
|
|
66
|
+
Composes [`createPresentationTimeline`](presentation-cue.md). Drive `step` from the same clock as deterministic `cart.step`. Duplicate policy, delay, and easing are the presentation cue’s.
|
|
67
|
+
|
|
68
|
+
```ts
|
|
69
|
+
import {
|
|
70
|
+
createAudioCueTimeline,
|
|
71
|
+
scheduleAudioCue,
|
|
72
|
+
AUDIO_CUE_SCHEDULED_EVENT,
|
|
73
|
+
AUDIO_CUE_STARTED_EVENT,
|
|
74
|
+
} from '@cyberart-io/engine';
|
|
75
|
+
|
|
76
|
+
const timeline = createAudioCueTimeline({
|
|
77
|
+
broker,
|
|
78
|
+
originFrame: 0,
|
|
79
|
+
reducedSensory: false,
|
|
80
|
+
});
|
|
81
|
+
|
|
82
|
+
scheduleAudioCue(timeline, {
|
|
83
|
+
name: 'overlay',
|
|
84
|
+
idempotencyKey: 'ripple-overlay',
|
|
85
|
+
assetId: 'overlay',
|
|
86
|
+
participantId: 'ambience',
|
|
87
|
+
channelId: 'ambience',
|
|
88
|
+
durationFrames: 4,
|
|
89
|
+
delayFrames: 1,
|
|
90
|
+
onDuplicate: 'ignore',
|
|
91
|
+
});
|
|
92
|
+
|
|
93
|
+
timeline.step(6);
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
`AudioCueSpec` is `CueSpec` plus `assetId` (required), `channelId?`, `participantId?`, `priority?`, `gain?`. `channelId` defaults to `participantId` or `'master'`.
|
|
97
|
+
|
|
98
|
+
| `onDuplicate` | Effect |
|
|
99
|
+
|---|---|
|
|
100
|
+
| `replace` (presentation default) | Replace the live cue; emit a new `audio.cue.scheduled`. |
|
|
101
|
+
| `ignore` | Keep the existing cue; no second `scheduled`. |
|
|
102
|
+
| `reject` | `{ ok: false, reason: 'duplicate' }`. |
|
|
103
|
+
|
|
104
|
+
## Cue events (stable names)
|
|
105
|
+
|
|
106
|
+
| `type` | When |
|
|
107
|
+
|---|---|
|
|
108
|
+
| `audio.cue.scheduled` | `play` / `scheduleAudioCue` accepted |
|
|
109
|
+
| `audio.cue.started` | Presentation start, authorized, not muted, asset ready |
|
|
110
|
+
| `audio.cue.skipped` | Reduced-sensory, muted, or unauthorized (reasons above) |
|
|
111
|
+
| `audio.cue.failed` | Failed audio asset (`asset-failed`) or cart teardown (`torn-down`) |
|
|
112
|
+
|
|
113
|
+
Each event: `{ type, atFrame, name, idempotencyKey, assetId, channelId, participantId?, reason?, progress }`. Two identical `play` / `step` / asset-event tapes produce identical lists.
|
|
114
|
+
|
|
115
|
+
A failed `ASSET_FAILED_EVENT` for the cue’s `assetId` emits `audio.cue.failed` and does not wait on I/O. Unknown assets stay scheduled until ready, failed, or the presentation cue completes (then `asset-failed` so `step` cannot hang).
|
|
116
|
+
|
|
117
|
+
## Headless adapter
|
|
118
|
+
|
|
119
|
+
Event-only. No Web Audio. Import from **`@cyberart-io/engine/headless`** (also re-exported from `@cyberart-io/engine`).
|
|
120
|
+
|
|
121
|
+
```ts
|
|
122
|
+
import { createHeadlessAudioAdapter } from '@cyberart-io/engine/headless';
|
|
123
|
+
|
|
124
|
+
const adapter = createHeadlessAudioAdapter({ originFrame: 0 });
|
|
125
|
+
await adapter.unlock();
|
|
126
|
+
adapter.handleHostEvent({
|
|
127
|
+
type: 'cyberart.asset.ready',
|
|
128
|
+
kind: 'state',
|
|
129
|
+
payload: { id: 'overlay' },
|
|
130
|
+
});
|
|
131
|
+
adapter.play({ /* AudioCueSpec */ });
|
|
132
|
+
adapter.step(4);
|
|
133
|
+
const { events } = adapter.snapshot();
|
|
134
|
+
adapter.destroy();
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
Replay two runs with the same seed/tape and compare `events`. Do not compare speakers, meters, or decoded PCM — those are not deterministic across machines or browsers.
|
|
138
|
+
|
|
139
|
+
## Host wiring
|
|
140
|
+
|
|
141
|
+
```ts
|
|
142
|
+
const broker = createAudioBroker();
|
|
143
|
+
button.addEventListener('click', () => broker.unlock());
|
|
144
|
+
|
|
145
|
+
const runtime = createRuntime({
|
|
146
|
+
container,
|
|
147
|
+
audioBroker: broker,
|
|
148
|
+
audio: 'tone', // still required for start() to unlock when the cart declares it
|
|
149
|
+
});
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
`runtime.unlockAudio()` / `cart.start()` call `broker.unlock()` when the cart (or `createRuntime({ audio })` hint) needs audio. A silent cart still never imports Tone.
|
|
@@ -0,0 +1,134 @@
|
|
|
1
|
+
# Calculation carts
|
|
2
|
+
|
|
3
|
+
Carts that run `getDefaultState`, `update`, and host-channel events without a paint surface. They share a lockstep clock and router with sibling render carts. They do not create a canvas and the runtime does not call `render`, even if a no-op is supplied.
|
|
4
|
+
|
|
5
|
+
Generic names only: telemetry, world, overlay. Event types stay on `host.intent.*` / `host.state.*`. Scene ids such as `alpha` and overlay ids such as `banner` are payload fields, not product names.
|
|
6
|
+
|
|
7
|
+
Related: [runtime group](runtime-group.md), [events](events.md), [capability manifest](capability-manifest.md), [deterministic mode](deterministic-mode.md).
|
|
8
|
+
|
|
9
|
+
Back to the [package README](../README.md).
|
|
10
|
+
|
|
11
|
+
## One-command reproduce (this repo)
|
|
12
|
+
|
|
13
|
+
```bash
|
|
14
|
+
pnpm exec vitest run packages/engine/src/canvas/cyb-20-calculation-cart.repro.spec.ts
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
## When to use
|
|
18
|
+
|
|
19
|
+
| Surface | Import | Use |
|
|
20
|
+
|---|---|---|
|
|
21
|
+
| Production host | `createRuntime({ kind: 'calculation' })` or `createRuntimeGroup` participant `kind: 'calculation'` from `@cyberart-io/engine` | World / telemetry carts that emit and consume events for a sibling overlay (or other render) cart. |
|
|
22
|
+
| Vitest / jsdom | `createHeadlessMultiCartHarness` from `@cyberart-io/engine/headless` | Same group API after `installHeadlessCanvas()`. |
|
|
23
|
+
|
|
24
|
+
`kind` defaults to `'render'`. Existing carts, `/art`, and `metadata.audio: 'tone'` pieces are unchanged: they still get a canvas and a paint phase.
|
|
25
|
+
|
|
26
|
+
## `createRuntime({ kind: 'calculation' })`
|
|
27
|
+
|
|
28
|
+
```ts
|
|
29
|
+
import { createRuntime } from '@cyberart-io/engine';
|
|
30
|
+
import type { AnimationCart } from '@cyberart-io/engine';
|
|
31
|
+
|
|
32
|
+
type WorldState = { sceneId: string | null };
|
|
33
|
+
|
|
34
|
+
const worldCart: AnimationCart<WorldState> = {
|
|
35
|
+
metadata: { id: 'world', name: 'World', frameRate: 30 },
|
|
36
|
+
getDefaultState: () => ({ sceneId: null }),
|
|
37
|
+
update: (_R, _f, _p, _d, state, _k, _ptr, _gm, _t, _fs, hostChannel) => {
|
|
38
|
+
const events = hostChannel?.consume() ?? [];
|
|
39
|
+
let sceneId = state.sceneId;
|
|
40
|
+
for (const event of events) {
|
|
41
|
+
if (event.type === 'host.state.scene-changed') {
|
|
42
|
+
sceneId = (event.payload as { sceneId?: string } | undefined)?.sceneId ?? null;
|
|
43
|
+
hostChannel?.emit({
|
|
44
|
+
type: 'host.intent.overlay',
|
|
45
|
+
kind: 'intent',
|
|
46
|
+
payload: { overlayId: 'banner' },
|
|
47
|
+
});
|
|
48
|
+
}
|
|
49
|
+
}
|
|
50
|
+
return { sceneId };
|
|
51
|
+
},
|
|
52
|
+
// render omitted
|
|
53
|
+
};
|
|
54
|
+
|
|
55
|
+
const runtime = createRuntime({
|
|
56
|
+
container,
|
|
57
|
+
kind: 'calculation',
|
|
58
|
+
deterministic: { origin: 0 },
|
|
59
|
+
seed: `0x${'20'.repeat(32)}`,
|
|
60
|
+
});
|
|
61
|
+
const cart = runtime.mount(worldCart);
|
|
62
|
+
// cart.canvas === undefined; container has no <canvas>
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
`runtime.kind` is `'calculation'`. `CartHandle.canvas` is `undefined`. `step` / `advance` still run `update`. A no-op `render` is not invoked.
|
|
66
|
+
|
|
67
|
+
## Runtime group
|
|
68
|
+
|
|
69
|
+
```ts
|
|
70
|
+
import { createRuntimeGroup } from '@cyberart-io/engine';
|
|
71
|
+
|
|
72
|
+
const group = createRuntimeGroup({
|
|
73
|
+
origin: 0,
|
|
74
|
+
participants: [
|
|
75
|
+
{
|
|
76
|
+
id: 'world',
|
|
77
|
+
cart: worldCart,
|
|
78
|
+
kind: 'calculation',
|
|
79
|
+
seed: `0x${'20'.repeat(32)}`,
|
|
80
|
+
emit: ['host.intent.*'],
|
|
81
|
+
subscribe: ['host.state.*'],
|
|
82
|
+
},
|
|
83
|
+
{
|
|
84
|
+
id: 'overlay',
|
|
85
|
+
cart: overlayCart,
|
|
86
|
+
kind: 'render',
|
|
87
|
+
subscribe: ['host.intent.*'],
|
|
88
|
+
},
|
|
89
|
+
{
|
|
90
|
+
id: 'telemetry',
|
|
91
|
+
cart: telemetryCart,
|
|
92
|
+
kind: 'calculation',
|
|
93
|
+
subscribe: ['host.state.*', 'host.intent.*'],
|
|
94
|
+
},
|
|
95
|
+
],
|
|
96
|
+
});
|
|
97
|
+
|
|
98
|
+
group.publish({
|
|
99
|
+
type: 'host.state.scene-changed',
|
|
100
|
+
kind: 'state',
|
|
101
|
+
payload: { sceneId: 'alpha' },
|
|
102
|
+
});
|
|
103
|
+
await group.step(2);
|
|
104
|
+
const { participants } = await group.inspect();
|
|
105
|
+
// participants.world.kind === 'calculation'
|
|
106
|
+
// group.participant('world').cart.canvas === undefined
|
|
107
|
+
// overlay still paints
|
|
108
|
+
group.destroy();
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
`createRuntimeGroup` passes each participant `kind` into `createRuntime`. Calculation participants keep a mount container (for the runtime) but never attach a canvas. `resize` skips missing canvases. Compositor layers that point at a calculation id are skipped (`cart.canvas` is absent).
|
|
112
|
+
|
|
113
|
+
## Capability manifest
|
|
114
|
+
|
|
115
|
+
Optional `kind: 'calculation'` on the manifest (omitted means render). Hosts that want to enforce it pass `kinds: ['render', 'calculation']`. Existing manifests and hosts that omit `kinds` are unchanged.
|
|
116
|
+
|
|
117
|
+
```ts
|
|
118
|
+
import { defineCapabilityManifest, validateCapabilityManifest } from '@cyberart-io/engine';
|
|
119
|
+
|
|
120
|
+
const defined = defineCapabilityManifest({
|
|
121
|
+
id: 'world.telemetry',
|
|
122
|
+
kind: 'calculation',
|
|
123
|
+
runtime: { minContractVersion: 1, features: ['router'] },
|
|
124
|
+
phases: ['ready'],
|
|
125
|
+
managers: ['hostChannel'],
|
|
126
|
+
assets: { kinds: [], declarations: [] },
|
|
127
|
+
acceptedEvents: ['host.state.*'],
|
|
128
|
+
emittedEvents: ['host.intent.overlay'],
|
|
129
|
+
permissions: { emit: ['host.intent.*'], subscribe: ['host.state.*'] },
|
|
130
|
+
integrations: [],
|
|
131
|
+
});
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
`CAPABILITY_CART_KINDS` is `['render', 'calculation']`. Diagnostics: `invalid-kind`, `unsupported-kind`.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Capability manifest
|
|
2
2
|
|
|
3
|
-
Versioned JSON for what a cart needs from the host: runtime contract, lifecycle phases, managers, asset kinds, event patterns, permissions, and integrations. Related: [events](events.md) (emit / subscribe patterns), [asset resolver](asset-resolver.md) (kinds only — this file does not resolve URLs).
|
|
3
|
+
Versioned JSON for what a cart needs from the host: runtime contract, lifecycle phases, managers, asset kinds, event patterns, permissions, and integrations. Related: [events](events.md) (emit / subscribe patterns), [asset resolver](asset-resolver.md) (kinds only — this file does not resolve URLs), [calculation carts](calculation-carts.md) (`kind`).
|
|
4
4
|
|
|
5
5
|
Back to the [package README](../README.md).
|
|
6
6
|
|
|
@@ -26,6 +26,7 @@ pnpm exec vitest run packages/engine/src/canvas/cyb-25-capability-manifest.repro
|
|
|
26
26
|
| `acceptedEvents` / `emittedEvents` | Dotted type patterns (`*` = one segment). |
|
|
27
27
|
| `permissions` | `emit` / `subscribe` patterns; optional `authoritative`. |
|
|
28
28
|
| `integrations` | Required host libraries: `tone` \| `midi`. |
|
|
29
|
+
| `kind` | Optional. `'render'` (omitted) or `'calculation'`. |
|
|
29
30
|
|
|
30
31
|
Emitted types must be covered by `permissions.emit`; accepted types by `permissions.subscribe`.
|
|
31
32
|
|
|
@@ -84,12 +85,14 @@ const parsed = parseCapabilityManifest(JSON.stringify(defined.manifest));
|
|
|
84
85
|
| `unknown-permission` | Cart emit/subscribe permission is not on the host allowlist (only if host listed `emit` / `subscribe`). |
|
|
85
86
|
| `invalid-surface` / `invalid-clear-policy` / `invalid-blend` | Cart or host `surface` / `layers` values are the wrong shape or not in the allowlist. |
|
|
86
87
|
| `unsupported-surface` / `unsupported-layer` | Host omitted `surface.alpha` / `clearPolicy` or compositor blend modes the cart requires. |
|
|
88
|
+
| `invalid-kind` / `unsupported-kind` | Manifest `kind` is not `render` / `calculation`, or the host `kinds` list does not include it. |
|
|
87
89
|
|
|
88
90
|
Optional additive keys (omitted on existing carts):
|
|
89
91
|
|
|
90
92
|
```ts
|
|
91
93
|
surface: { alpha: true, clearPolicy: 'transparent' }
|
|
92
94
|
layers: { compositor: true, blend: ['source-over', 'screen'] }
|
|
95
|
+
kind: 'calculation'
|
|
93
96
|
```
|
|
94
97
|
|
|
95
|
-
See [
|
|
98
|
+
`host.kinds` is optional. When omitted, `unsupported-kind` is not checked. See [calculation carts](calculation-carts.md).
|
package/docs/compositor.md
CHANGED
|
@@ -48,7 +48,7 @@ const frame = compositor.captureComposedFrame();
|
|
|
48
48
|
compositor.destroy();
|
|
49
49
|
```
|
|
50
50
|
|
|
51
|
-
Typical stack: host image, then opaque overlay carts (`source-over`), then additive layers (`screen` so RGB `0,0,0` backing is identity unless `clearPolicy: 'transparent'` knocks it out). Later `order` values draw on top.
|
|
51
|
+
Typical stack: host image, then opaque overlay carts (`source-over`), then additive layers (`screen` so RGB `0,0,0` backing is identity unless `clearPolicy: 'transparent'` knocks it out). Later `order` values draw on top. Versioned overlay swaps (`v1` → `v2`) belong on [`createVisualLayerController`](visual-layers.md), which calls `setLayer` / `addLayer` with `source: { kind: 'image', image }` so a failed asset never becomes the live frame.
|
|
52
52
|
|
|
53
53
|
### Options
|
|
54
54
|
|
|
@@ -65,7 +65,7 @@ Typical stack: host image, then opaque overlay carts (`source-over`), then addit
|
|
|
65
65
|
|
|
66
66
|
| Field | Default | Meaning |
|
|
67
67
|
|---|---|---|
|
|
68
|
-
| `id` | required | Runtime-group participant id. |
|
|
68
|
+
| `id` | required | Runtime-group participant id, or a visual-layer slot with an `image` / `canvas` source. |
|
|
69
69
|
| `order` | required | Lower draws first. |
|
|
70
70
|
| `visible` | `true` | Skip compose and pointer hits when false. |
|
|
71
71
|
| `opacity` | `1` | 0..1. |
|
|
@@ -73,6 +73,11 @@ Typical stack: host image, then opaque overlay carts (`source-over`), then addit
|
|
|
73
73
|
| `clip` | none | Pixel rect; compose and pointer hits are clipped. |
|
|
74
74
|
| `pointerEvents` | `auto` | `none` skips hit testing and sets `canvas.style.pointerEvents`. |
|
|
75
75
|
| `clearPolicy` | compositor default | `transparent` knocks out black backing. |
|
|
76
|
+
| `source` | `{ kind: 'participant' }` | `participant` reads the group canvas. `image` clones `ImageData`. `canvas` rereads that canvas each compose. |
|
|
77
|
+
|
|
78
|
+
`setLayer` applies a patch, including an atomic `source` swap. `addLayer` appends a declaration (ImageData slots do not need a group participant). `unmountLayer` detaches a participant when one exists; ImageData-only layers are removed from the compose list.
|
|
79
|
+
|
|
80
|
+
`inspect().layers[].sourceKind` is `participant` | `image` | `canvas`.
|
|
76
81
|
|
|
77
82
|
### Handle
|
|
78
83
|
|
|
@@ -81,7 +86,7 @@ Typical stack: host image, then opaque overlay carts (`source-over`), then addit
|
|
|
81
86
|
| `compose()` | Blend host + visible layers; returns `ImageData`. |
|
|
82
87
|
| `captureComposedFrame()` | `{ imageData, pngDataUrl, declaredOrder, width, height }`. |
|
|
83
88
|
| `resize(w, h, dpr?)` | Shared viewport; calls `group.resize` on the buffer size. Order/blend survive `group.step`. Carts that cache `DimensionContext` should `group.reset()` after a size change. |
|
|
84
|
-
| `setLayer` / `layer` / `layers` | Mutate / inspect declarative settings. |
|
|
89
|
+
| `setLayer` / `addLayer` / `layer` / `layers` | Mutate / inspect declarative settings. `setLayer` can swap `source`. |
|
|
85
90
|
| `pointerTarget(x, y)` | Top-most visible layer with `pointerEvents: 'auto'` under the pixel. |
|
|
86
91
|
| `unmountLayer(id)` | `group.detach(id)` without clearing sibling canvases. |
|
|
87
92
|
| `inspect()` | Viewport, clear policy, layers, `declaredOrder` ids. |
|
|
@@ -0,0 +1,112 @@
|
|
|
1
|
+
# Executable modules
|
|
2
|
+
|
|
3
|
+
Trusted, versioned factories behind a host allowlist. The engine never `eval`s or `new Function`s untrusted strings. Unknown ids, wrong versions, and unauthorized refs fail closed without invoking. A throwing module does not stop the next allowlisted call.
|
|
4
|
+
|
|
5
|
+
Related: [capability manifest](capability-manifest.md) (`modules.refs`).
|
|
6
|
+
|
|
7
|
+
Back to the [package README](../README.md).
|
|
8
|
+
|
|
9
|
+
## One-command reproduce (this repo)
|
|
10
|
+
|
|
11
|
+
```bash
|
|
12
|
+
pnpm exec vitest run packages/engine/src/canvas/cyb-27-executable-module.repro.spec.ts
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
## `createExecutableModuleHost(options)`
|
|
16
|
+
|
|
17
|
+
```ts
|
|
18
|
+
import { createExecutableModuleHost } from '@cyberart-io/engine';
|
|
19
|
+
|
|
20
|
+
const overlayCalls = { n: 0 };
|
|
21
|
+
const host = createExecutableModuleHost({
|
|
22
|
+
allowlist: [
|
|
23
|
+
{ id: 'overlay-fx', version: '1.0.0' },
|
|
24
|
+
{ id: 'host.module', version: '1.0.0' },
|
|
25
|
+
],
|
|
26
|
+
limits: { maxInvokeMs: 16, maxInvokesPerTurn: 4 },
|
|
27
|
+
capabilities: { seed: '0x27' },
|
|
28
|
+
modules: [
|
|
29
|
+
{
|
|
30
|
+
id: 'overlay-fx',
|
|
31
|
+
version: '1.0.0',
|
|
32
|
+
create: (capabilities) => ({
|
|
33
|
+
invoke: (input, { signal }) => {
|
|
34
|
+
if (signal.aborted) return;
|
|
35
|
+
overlayCalls.n += 1;
|
|
36
|
+
return { seed: capabilities.seed, input };
|
|
37
|
+
},
|
|
38
|
+
}),
|
|
39
|
+
},
|
|
40
|
+
{
|
|
41
|
+
id: 'host.module',
|
|
42
|
+
version: '1.0.0',
|
|
43
|
+
create: () => ({
|
|
44
|
+
invoke: () => {
|
|
45
|
+
throw new Error('host.module boom');
|
|
46
|
+
},
|
|
47
|
+
}),
|
|
48
|
+
},
|
|
49
|
+
],
|
|
50
|
+
});
|
|
51
|
+
|
|
52
|
+
const allowed = await host.invoke({ id: 'overlay-fx', version: '1.0.0' }, { tape: 'cyb-27' });
|
|
53
|
+
// allowed.ok === true — overlay-fx ran
|
|
54
|
+
|
|
55
|
+
await host.invoke({ id: 'leak.module', version: '1.0.0' });
|
|
56
|
+
// { ok: false, error: { code: 'unknown-module' | 'not-allowlisted' } }
|
|
57
|
+
|
|
58
|
+
await host.invoke({ id: 'overlay-fx', version: '2.0.0' });
|
|
59
|
+
// { ok: false, error: { code: 'version-mismatch' } } — factory is not called
|
|
60
|
+
|
|
61
|
+
const boom = await host.invoke({ id: 'host.module', version: '1.0.0' });
|
|
62
|
+
// boom.error.code === 'invoke-failed'; overlay-fx still runs next:
|
|
63
|
+
await host.invoke({ id: 'overlay-fx', version: '1.0.0' });
|
|
64
|
+
|
|
65
|
+
host.beginTurn();
|
|
66
|
+
host.destroy();
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
Register only trusted `create` functions you compiled with the host. Carts declare required refs on the capability manifest; they do not ship source strings into this API.
|
|
70
|
+
|
|
71
|
+
### Options
|
|
72
|
+
|
|
73
|
+
| Option | Default | Meaning |
|
|
74
|
+
|---|---|---|
|
|
75
|
+
| `allowlist` | required | `{ id, version }[]`. `id` may be a dotted segment pattern (`host.*`). **Version is always exact** — `*` never loads. |
|
|
76
|
+
| `modules` | required | Trusted factories keyed by exact `id` + `version`. Duplicate refs throw. |
|
|
77
|
+
| `limits.maxInvokeMs` | none | Cooperative timeout. `invoke` receives `AbortSignal`; hanging thenables fail with `timeout`. |
|
|
78
|
+
| `limits.maxInvokesPerTurn` | none | Allowlisted invoke budget. Reset with `beginTurn()`. |
|
|
79
|
+
| `capabilities` | `{}` | Frozen bag passed into each factory. Nested plain objects are frozen; class instances stay shared handles. |
|
|
80
|
+
|
|
81
|
+
Per-registration `capabilities` overlay the host bag, then the result is frozen.
|
|
82
|
+
|
|
83
|
+
### Handle
|
|
84
|
+
|
|
85
|
+
| Member | Meaning |
|
|
86
|
+
|---|---|
|
|
87
|
+
| `load(ref)` | Instantiate the factory if the exact ref is registered and allowlisted. Idempotent. |
|
|
88
|
+
| `invoke(ref, input?)` | `load` if needed, then call `invoke`. Returns `{ ok: true, value }` or `{ ok: false, error }`. |
|
|
89
|
+
| `beginTurn()` | Increment the turn counter and reset `maxInvokesPerTurn`. |
|
|
90
|
+
| `inspect()` | Allowlist, registered/loaded refs, turn, diagnostics. |
|
|
91
|
+
| `destroy()` | Drop the registry, abort in-flight signals, call optional instance `destroy()`. Idempotent. |
|
|
92
|
+
|
|
93
|
+
`error.code` is one of: `invalid-ref`, `unknown-module`, `version-mismatch`, `not-allowlisted`, `load-failed`, `invoke-failed`, `timeout`, `rate-limited`, `destroyed`. Failures are recorded on `inspect().diagnostics`. Sibling modules and the cart continue.
|
|
94
|
+
|
|
95
|
+
## Isolation
|
|
96
|
+
|
|
97
|
+
This boundary is **in-process**. Factories run on the cart’s thread; there is no worker, iframe, or separate origin. Isolation is the allowlist, exact id+version matching, a frozen capability bag, invoke budgets, and `{ ok: false }` on throw so a sibling can still run. Do not pass untrusted source strings. Untrusted code needs a host-owned worker/iframe **outside** this API.
|
|
98
|
+
|
|
99
|
+
## Capability manifest
|
|
100
|
+
|
|
101
|
+
Optional additive key (omitted on existing carts):
|
|
102
|
+
|
|
103
|
+
```ts
|
|
104
|
+
modules: {
|
|
105
|
+
refs: [
|
|
106
|
+
{ id: 'overlay-fx', version: '1.0.0' },
|
|
107
|
+
{ id: 'host.module', version: '1.0.0' },
|
|
108
|
+
],
|
|
109
|
+
}
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
`validateCapabilityManifest` returns `unsupported-module` when the host does not list a required exact ref on `host.modules.refs`.
|
package/docs/midi.md
ADDED
|
@@ -0,0 +1,128 @@
|
|
|
1
|
+
# MIDI controller
|
|
2
|
+
|
|
3
|
+
Host-owned live MIDI in and out. Carts and hosts construct `MidiManager`. The runtime does not pass it into `getDefaultState`. This is controller I/O (Web MIDI), not MIDI file playback.
|
|
4
|
+
|
|
5
|
+
Capability manifests already list `midi` under `integrations`. Related: [capability manifest](capability-manifest.md).
|
|
6
|
+
|
|
7
|
+
Back to the [package README](../README.md).
|
|
8
|
+
|
|
9
|
+
## One-command reproduce (this repo)
|
|
10
|
+
|
|
11
|
+
```bash
|
|
12
|
+
pnpm exec vitest run packages/engine/src/canvas/cyb-19-midi.repro.spec.ts
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
## Why this exists
|
|
16
|
+
|
|
17
|
+
A piece that maps a keyboard or fader to parameters needs the same API in a browser (real ports) and in CI (no hardware). `inject` delivers inbound note / CC / pitch without `navigator.requestMIDIAccess`. `send` writes note-on/off, CC, pitch, or raw bytes through an injectable port. Missing Web MIDI or a denied permission is a structured result, so silent carts keep running.
|
|
18
|
+
|
|
19
|
+
## `new MidiManager(options?)`
|
|
20
|
+
|
|
21
|
+
```ts
|
|
22
|
+
import { MidiManager } from '@cyberart-io/engine';
|
|
23
|
+
|
|
24
|
+
const sent: number[][] = [];
|
|
25
|
+
const midi = new MidiManager({
|
|
26
|
+
output: {
|
|
27
|
+
send(data) {
|
|
28
|
+
sent.push([...data]);
|
|
29
|
+
},
|
|
30
|
+
},
|
|
31
|
+
});
|
|
32
|
+
|
|
33
|
+
const stop = midi.subscribe('note', (message) => {
|
|
34
|
+
if (message.kind === 'noteon') {
|
|
35
|
+
// message.channel 0–15, message.note, message.velocity, message.status
|
|
36
|
+
}
|
|
37
|
+
});
|
|
38
|
+
|
|
39
|
+
midi.inject({ kind: 'noteon', channel: 0, note: 60, velocity: 100 });
|
|
40
|
+
midi.sendNoteOn(0, 64, 90);
|
|
41
|
+
midi.sendCc(3, 1, 127);
|
|
42
|
+
midi.send([0xf8]);
|
|
43
|
+
|
|
44
|
+
const access = await midi.requestAccess();
|
|
45
|
+
// jsdom / Node: { ok: false, reason: 'unavailable' }
|
|
46
|
+
// user gesture denied: { ok: false, reason: 'denied' }
|
|
47
|
+
// browser with ports: { ok: true, inputs, outputs, sysexEnabled }
|
|
48
|
+
|
|
49
|
+
stop();
|
|
50
|
+
midi.destroy();
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
Headless and jsdom have no `navigator.requestMIDIAccess`. Inject and send still work when you pass a test `output`.
|
|
54
|
+
|
|
55
|
+
### Options
|
|
56
|
+
|
|
57
|
+
| Option | Default | Meaning |
|
|
58
|
+
|---|---|---|
|
|
59
|
+
| `output` | none | Port used by `send`. Tests pass a recording fake. When omitted, `requestAccess` adopts the first hardware output and re-adopts after hotplug if that port disconnects. A constructor `output` is never replaced. |
|
|
60
|
+
| `requestMIDIAccess` | `navigator.requestMIDIAccess` | Override for tests (fake access or a rejecting function). |
|
|
61
|
+
|
|
62
|
+
`statechange` on the access object attaches new inputs and **detaches disconnected ports immediately** (not only on `destroy()`). Stale `midimessage` listeners are removed when the input leaves `access.inputs`.
|
|
63
|
+
|
|
64
|
+
### Inbound
|
|
65
|
+
|
|
66
|
+
| Member | Meaning |
|
|
67
|
+
|---|---|
|
|
68
|
+
| `subscribe(kind, listener)` | `kind`: `note` (on and off) \| `cc` \| `pitch` \| `raw` \| `*`. Returns unsubscribe. |
|
|
69
|
+
| `inject(input)` | Deliver without hardware. Voice fields, `Uint8Array`, or raw `number[]`. No-op after `destroy`. |
|
|
70
|
+
|
|
71
|
+
### Outbound
|
|
72
|
+
|
|
73
|
+
| Member | Meaning |
|
|
74
|
+
|---|---|
|
|
75
|
+
| `sendNoteOn(channel, note, velocity?)` | Status `0x90 \| channel`. Velocity default 100. |
|
|
76
|
+
| `sendNoteOff(channel, note, velocity?)` | Status `0x80 \| channel`. Velocity default 0. |
|
|
77
|
+
| `sendCc(channel, controller, value)` | Status `0xB0 \| channel`. |
|
|
78
|
+
| `sendPitch(channel, value)` | 14-bit pitch bend `0–16383` (center `8192`). Status `0xE0 \| channel`. |
|
|
79
|
+
| `send(bytes)` | Raw bytes through the port. |
|
|
80
|
+
|
|
81
|
+
Each send returns `{ ok: true, data }` or `{ ok: false, reason: 'no-port' \| 'invalid' \| 'destroyed' }`. Out-of-range channel (not 0–15), 7-bit data, or pitch is `invalid`, not a throw.
|
|
82
|
+
|
|
83
|
+
### Access and teardown
|
|
84
|
+
|
|
85
|
+
| Member | Meaning |
|
|
86
|
+
|---|---|
|
|
87
|
+
| `requestAccess({ sysex? })` | Wraps Web MIDI when present. Never throws. |
|
|
88
|
+
| `destroy()` | Removes hardware listeners and subscribers. Idempotent. |
|
|
89
|
+
|
|
90
|
+
`requestAccess` results:
|
|
91
|
+
|
|
92
|
+
| Result | When |
|
|
93
|
+
|---|---|
|
|
94
|
+
| `{ ok: true, inputs, outputs, sysexEnabled }` | Access granted. Inputs are attached; a constructor `output` is kept. |
|
|
95
|
+
| `{ ok: false, reason: 'unavailable' }` | No `navigator.requestMIDIAccess` (headless / jsdom / insecure context). |
|
|
96
|
+
| `{ ok: false, reason: 'denied' }` | The request rejected (permission or security). |
|
|
97
|
+
| `{ ok: false, reason: 'destroyed' }` | `destroy()` ran before the promise settled, or after teardown. |
|
|
98
|
+
|
|
99
|
+
## Types, channels, status bytes
|
|
100
|
+
|
|
101
|
+
Channel is **0–15** (MIDI channels 1–16) in the status low nibble. Command lives in the high nibble.
|
|
102
|
+
|
|
103
|
+
| Export | Value | Role |
|
|
104
|
+
|---|---|---|
|
|
105
|
+
| `MIDI_NOTE_OFF` | `0x80` | Note Off command |
|
|
106
|
+
| `MIDI_NOTE_ON` | `0x90` | Note On command |
|
|
107
|
+
| `MIDI_CONTROL_CHANGE` | `0xB0` | Control Change command |
|
|
108
|
+
| `MIDI_PITCH_BEND` | `0xE0` | Pitch Bend command |
|
|
109
|
+
| `MIDI_CHANNEL_MIN` / `MIDI_CHANNEL_MAX` | `0` / `15` | Valid channel range |
|
|
110
|
+
| `MIDI_DATA_MAX` | `127` | Max 7-bit data byte (note, velocity, CC) |
|
|
111
|
+
| `MIDI_PITCH_CENTER` / `MIDI_PITCH_MAX` | `8192` / `16383` | 14-bit pitch bend |
|
|
112
|
+
| `midiStatus(command, channel)` | `(command & 0xF0) \| (channel & 0x0F)` | Build a status byte |
|
|
113
|
+
| `midiChannelFromStatus(status)` | `status & 0x0F` | Channel 0–15 from a status byte |
|
|
114
|
+
| `isMidiChannel(value)` | boolean | Integer 0–15 |
|
|
115
|
+
| `isMidiData(value)` | boolean | Integer 0–127 (`MIDI_DATA_MAX`) |
|
|
116
|
+
| `encodeMidiMessage` / `parseMidiBytes` | bytes ↔ `MidiMessage` | Shared by inject and hardware |
|
|
117
|
+
|
|
118
|
+
`MidiMessage` kinds: `noteon` / `noteoff` / `cc` / `pitch` / `raw`. Each carries `status` and a copied `data` `Uint8Array`. Note-on with velocity `0` parses as `noteoff` (MIDI convention); the status byte stays `0x9n`.
|
|
119
|
+
|
|
120
|
+
Public types for the `requestMIDIAccess` option (tests inject a fake; browsers pass the real Web MIDI objects):
|
|
121
|
+
|
|
122
|
+
| Type | Role |
|
|
123
|
+
|---|---|
|
|
124
|
+
| `MidiRequestAccess` | `(options?: { sysex?: boolean }) => Promise<MidiAccessLike>` |
|
|
125
|
+
| `MidiAccessLike` | `inputs` / `outputs` (`forEach`), `sysexEnabled`, optional `statechange` |
|
|
126
|
+
| `MidiInputLike` | `addEventListener` / `removeEventListener` for `midimessage` (`event.data`) |
|
|
127
|
+
|
|
128
|
+
Hosts that need a controller construct `MidiManager` themselves. Do not add it to `getDefaultState`.
|
package/docs/runtime-group.md
CHANGED
|
@@ -4,7 +4,7 @@ Host helper that mounts several production `createRuntime` carts, attaches each
|
|
|
4
4
|
|
|
5
5
|
Capability-manifest integration is optional and structural (`capability?: { emit, subscribe, authoritative }`). This module does not import the manifest.
|
|
6
6
|
|
|
7
|
-
Related: [events](events.md), [headless harness](headless-harness.md), [deterministic mode](deterministic-mode.md), [compositor](compositor.md), [replay inspector](replay-inspector.md).
|
|
7
|
+
Related: [events](events.md), [headless harness](headless-harness.md), [deterministic mode](deterministic-mode.md), [compositor](compositor.md), [replay inspector](replay-inspector.md), [calculation carts](calculation-carts.md).
|
|
8
8
|
|
|
9
9
|
Back to the [package README](../README.md).
|
|
10
10
|
|
|
@@ -82,7 +82,7 @@ Example mapping: an accepted `host.state.accepted` is consumed by the effects ca
|
|
|
82
82
|
|---|---|
|
|
83
83
|
| `id` | Router participant id. Reserved: `host`, `router`. |
|
|
84
84
|
| `cart` | `AnimationCart` mounted through `createRuntime`. |
|
|
85
|
-
| `kind` | `'render'` (default) or `'calculation'` (no
|
|
85
|
+
| `kind` | `'render'` (default) or `'calculation'` (no canvas / paint; `update` and host-channel events still run). See [calculation carts](calculation-carts.md). |
|
|
86
86
|
| `seed` | `0x` + 64 hex, or any seed `createRuntime` accepts. Default: derived from `id`. |
|
|
87
87
|
| `container` | Injected mount node. If omitted, the group creates and owns a sized `div`. |
|
|
88
88
|
| `emit` / `subscribe` / `authoritative` | Passed to `router.attach`. |
|
|
@@ -117,6 +117,6 @@ await harness.step(2);
|
|
|
117
117
|
harness.destroy();
|
|
118
118
|
```
|
|
119
119
|
|
|
120
|
-
Calls `installHeadlessCanvas()` then `createRuntimeGroup`. Same handle. Three-cart CI fixture: effects + ambience + a calculation telemetry cart.
|
|
120
|
+
Calls `installHeadlessCanvas()` then `createRuntimeGroup`. Same handle. Three-cart CI fixture: effects + ambience + a calculation telemetry cart (no canvas).
|
|
121
121
|
|
|
122
122
|
Cleanup: `afterEach(() => group.destroy())` so a failed assertion does not leak nodes. After `destroy`, `step` throws and owned containers are gone from `document.body`.
|