dsh-realtime-audio-ws 0.2.2 → 0.2.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -43,8 +43,27 @@ first — which is the only reason to trust it.
43
43
 
44
44
  ## The wire
45
45
 
46
- Binary frames only, both directions, in the session's declared input/output format (PCM16, 24 kHz, mono, for
47
- the shipped adapter). A text frame is ignored rather than fatal: nothing on this wire is JSON.
46
+ Binary frames carry audio, both directions, in the session's declared input/output format (PCM16, 24 kHz,
47
+ mono, for the shipped adapter). **Text frames carry control:**
48
+
49
+ | Frame | Answer |
50
+ |---|---|
51
+ | `status` | this route, the live voice, every setting, the journal's last entry |
52
+ | `start` | the session that opened — an outcome, not an acknowledgement |
53
+ | `stop` | the state afterwards |
54
+ | `steer <sessionId>` | the setting that declares a live `sessionId`, changed |
55
+ | `set <key>=<value>` | the setting named by its `<owner>.<field>` key, changed |
56
+
57
+ One frame in, exactly one frame out, and **every** frame is answered — a verb it does not have, an argument
58
+ where none belongs, a field frozen by its class, a value the setting's own rules reject. The reply is JSON:
59
+ `{"ok":true,…}` with the fields for that verb, or `{"ok":false,"verb":…,"code":…,"reason":…}` with a reason
60
+ written to be relayed verbatim. `set realtime-agent.autoStart=true` answers with the *restart* it needs
61
+ rather than a silence that reads as a broken control.
62
+
63
+ This was a deliberate widening of a contract that used to say the opposite — a text frame was ignored,
64
+ because nothing on this wire was JSON. The socket is the only duplex connection the client already holds,
65
+ one text frame in / one text frame out is smaller than a second route with its own authentication, and the
66
+ audio path is untouched: the two are told apart by the frame's own type.
48
67
 
49
68
  Frames are forwarded, never queued. A client that cannot keep up drops audio instead of accumulating it,
50
69
  which matches the `realtime-agent/audio` contract — a queue that grows while nothing drains it presents
@@ -60,6 +79,14 @@ A frame larger than `maxFrameBytes` closes the connection with 1009 rather than
60
79
  | `maxFrameBytes` | `480000` | Ten seconds of 24 kHz mono PCM16. Above this the connection is closed with 1009. |
61
80
  | `maxConnections` | `1` | A second connection is closed with 1013 rather than queued. One microphone is the honest default. |
62
81
 
82
+ `openSessionOnConnect` (default `true`) is **live**: read at the moment an authenticated client connects
83
+ and at the moment the last one leaves, so the seam's settings surface turns it off and on without a
84
+ restart — `set realtime-audio-ws.openSessionOnConnect=false`. Freezing it was the failure this plugin's
85
+ own docs describe: with it off and no way to change it, a working microphone produces silence that looks
86
+ like a fault anywhere but in the config. The rest of this row's fields (`path`, `maxFrameBytes`,
87
+ `maxConnections`, `diagnosticsPath`) are claimed against the web server's registry or enforced by the
88
+ socket at load, so they are restart-bound and get no control.
89
+
63
90
  ## Where it waits
64
91
 
65
92
  This row appears **unloaded** in a composition without the web server and connection services, and that is
@@ -70,20 +97,41 @@ visibly waits rather than one that loads and silently claims nothing. The bundle
70
97
  ## The client half
71
98
 
72
99
  `dsh-realtime-audio-ws/client` is the browser face: it opens this socket, captures the microphone into it at
73
- 24 kHz PCM16, and plays what comes back. No client services, one file, and it injects nothing — a client
74
- face that needs nothing cannot be broken by another plugin's absence.
100
+ 24 kHz PCM16, plays what comes back, and drives the strip. No client services, one file.
101
+
102
+ **How it runs.** The strip in the app's own page is the way in — it is mounted by the injected row and by
103
+ the bundle itself, whichever lands second finding the panel already there:
104
+
105
+ ```
106
+ [ microphone: idle ] [ Connect microphone ] [ Refresh ]
107
+ voice: open · fake/gpt-live-1
108
+ realtime-responder.sessionId [ sess-1 ▾ ] [ Steer ] applied
109
+ realtime-responder.answerTimeoutMs [ 45000 ] [ Set ]
110
+ realtime-agent.model fixed when the session opens — reconnect to apply
111
+ ```
112
+
113
+ A live field gets a control, a session-bound field gets the value it will take plus the words "reconnect to
114
+ apply", and a restart-bound field gets no row at all — a control the protocol cannot honour is worse than no
115
+ control. Every `set` reports its outcome beside the field it belongs to, and a refusal is relayed verbatim
116
+ with the host's own code: `FROZEN_SETTING: "…" is claimed when the plugin loads — restart to change it`.
75
117
 
76
- **How it runs.** There is no UI surface yet, so it publishes itself on a global:
118
+ The same handle stays published on `globalThis[GLOBAL_KEY]` for the injected bootstrap and for anything else
119
+ that holds it — `start`, `stop`, `state`, `request`, `mount` — but it is no longer the on-switch:
77
120
 
78
121
  ```js
79
- await __dshRealtimeAudio.start() // asks for the microphone, then connects
80
- __dshRealtimeAudio.state() // { kind: 'idle' | 'live' | 'failed' }
81
- __dshRealtimeAudio.stop()
122
+ __dshRealtimeAudio.start() // asks for the microphone, then connects
123
+ __dshRealtimeAudio.state() // { kind: 'idle' | 'live' | 'failed' }
124
+ __dshRealtimeAudio.request('status') // one text frame in, exactly one reply out
82
125
  ```
83
126
 
84
127
  `start()` **reports rather than throws**, so a refused permission or a missing API arrives as
85
128
  `{ kind: 'failed', reason }` — the reason is what tells someone whether to grant something or to look
86
- somewhere else. A settings card is its own increment.
129
+ somewhere else.
130
+
131
+ **Requests are serialised.** One frame is in flight at a time: the host answers in arrival order, and two
132
+ frames out at once would make pairing a reply with its request a guess the moment anything on the host
133
+ slowed down. A request with no socket is a rejection — reporting `{ok: false}` there would be
134
+ indistinguishable from the host refusing.
87
135
 
88
136
  **Two decisions inside it worth knowing.**
89
137
 
@@ -99,6 +147,10 @@ will not open at 24 kHz the client fails and says which rate it got.
99
147
 
100
148
  ## What it does not do yet
101
149
 
102
- **No format negotiation, and no reconnect.** A `ready` frame carrying sample rate and channel count would be
103
- the natural first control message; a dropped socket currently ends the conversation rather than being
104
- rejoined. Both are deliberately absent rather than half-built.
150
+ **No format negotiation, and no reconnect.** A `ready` frame carrying sample rate and channel count is the
151
+ obvious next control *frame* — the control channel is now there to carry one — and a dropped socket still
152
+ ends the conversation rather than being rejoined. Both are deliberately absent rather than half-built.
153
+
154
+ **The panel has no keyboard path and no permissions row.** The strip is clickable controls and text, so a
155
+ screen reader gets it, but nothing focuses it and the microphone's permission state is reported rather than
156
+ requested ahead of time. Both are increments, not gaps in the channel.
package/lib/bridge.d.ts CHANGED
@@ -12,6 +12,20 @@ export interface AudioSocketBridgeDeps {
12
12
  readonly subscribeAudio: (listener: (pcm16: Uint8Array) => void) => () => void;
13
13
  /** Longest inbound frame accepted, in bytes. */
14
14
  readonly maxFrameBytes: number;
15
+ /**
16
+ * Handle one **control frame** — a text frame on this socket.
17
+ *
18
+ * Until S2 story 2 a text frame was not part of this contract, and the comment that said so was right
19
+ * about the wire as it was: the socket carried PCM16 in both directions and nothing else. It now
20
+ * carries a channel too, because it is the only duplex connection the client already holds, and one
21
+ * text frame in / one text frame out is a smaller contract than a second route with its own
22
+ * authentication would have been.
23
+ *
24
+ * `reply` writes exactly one text frame back, and silently drops it once the socket is gone: a control
25
+ * handler is asynchronous, and a reply to a closed socket would throw from inside a promise nobody
26
+ * awaits.
27
+ */
28
+ readonly onControl: (frame: string, reply: (text: string) => void) => void;
15
29
  /** Called exactly once when the bridge stops, however it stops. */
16
30
  readonly onDetach: () => void;
17
31
  }
@@ -26,6 +40,16 @@ export interface AudioSocketBridgeDeps {
26
40
  * @returns the payload as bytes.
27
41
  */
28
42
  export declare function toBytes(data: unknown): Uint8Array;
43
+ /**
44
+ * Normalise a `ws` message payload to text.
45
+ *
46
+ * The same three shapes {@link toBytes} handles, decoded as UTF-8 — a control frame arrives as a Buffer
47
+ * whichever way it was framed, and a fragmented one as an array of them.
48
+ *
49
+ * @param data - the payload as `ws` delivered it.
50
+ * @returns the payload as text.
51
+ */
52
+ export declare function toText(data: unknown): string;
29
53
  /**
30
54
  * Wire one accepted socket to the agent's audio events.
31
55
  *
@@ -1 +1 @@
1
- {"version":3,"file":"bridge.d.ts","sourceRoot":"","sources":["../src/bridge.ts"],"names":[],"mappings":"AAAA;;;;;GAKG;AAEH,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,YAAY,CAAA;AAE7C,MAAM,WAAW,qBAAqB;IACpC,6DAA6D;IAC7D,QAAQ,CAAC,OAAO,EAAE,CAAC,KAAK,EAAE,UAAU,KAAK,IAAI,CAAA;IAC7C,qEAAqE;IACrE,QAAQ,CAAC,cAAc,EAAE,CAAC,QAAQ,EAAE,CAAC,KAAK,EAAE,UAAU,KAAK,IAAI,KAAK,MAAM,IAAI,CAAA;IAC9E,gDAAgD;IAChD,QAAQ,CAAC,aAAa,EAAE,MAAM,CAAA;IAC9B,mEAAmE;IACnE,QAAQ,CAAC,QAAQ,EAAE,MAAM,IAAI,CAAA;CAC9B;AAED;;;;;;;;;GASG;AACH,wBAAgB,OAAO,CAAC,IAAI,EAAE,OAAO,GAAG,UAAU,CAKjD;AAED;;;;;;GAMG;AACH,wBAAgB,iBAAiB,CAAC,MAAM,EAAE,WAAW,EAAE,IAAI,EAAE,qBAAqB,GAAG,MAAM,IAAI,CAwC9F"}
1
+ {"version":3,"file":"bridge.d.ts","sourceRoot":"","sources":["../src/bridge.ts"],"names":[],"mappings":"AAAA;;;;;GAKG;AAEH,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,YAAY,CAAA;AAE7C,MAAM,WAAW,qBAAqB;IACpC,6DAA6D;IAC7D,QAAQ,CAAC,OAAO,EAAE,CAAC,KAAK,EAAE,UAAU,KAAK,IAAI,CAAA;IAC7C,qEAAqE;IACrE,QAAQ,CAAC,cAAc,EAAE,CAAC,QAAQ,EAAE,CAAC,KAAK,EAAE,UAAU,KAAK,IAAI,KAAK,MAAM,IAAI,CAAA;IAC9E,gDAAgD;IAChD,QAAQ,CAAC,aAAa,EAAE,MAAM,CAAA;IAC9B;;;;;;;;;;;;OAYG;IACH,QAAQ,CAAC,SAAS,EAAE,CAAC,KAAK,EAAE,MAAM,EAAE,KAAK,EAAE,CAAC,IAAI,EAAE,MAAM,KAAK,IAAI,KAAK,IAAI,CAAA;IAC1E,mEAAmE;IACnE,QAAQ,CAAC,QAAQ,EAAE,MAAM,IAAI,CAAA;CAC9B;AAED;;;;;;;;;GASG;AACH,wBAAgB,OAAO,CAAC,IAAI,EAAE,OAAO,GAAG,UAAU,CAKjD;AAED;;;;;;;;GAQG;AACH,wBAAgB,MAAM,CAAC,IAAI,EAAE,OAAO,GAAG,MAAM,CAK5C;AAED;;;;;;GAMG;AACH,wBAAgB,iBAAiB,CAAC,MAAM,EAAE,WAAW,EAAE,IAAI,EAAE,qBAAqB,GAAG,MAAM,IAAI,CAkE9F"}
package/lib/bridge.js CHANGED
@@ -23,6 +23,24 @@ export function toBytes(data) {
23
23
  return new Uint8Array(data);
24
24
  return new Uint8Array(Buffer.from(data));
25
25
  }
26
+ /**
27
+ * Normalise a `ws` message payload to text.
28
+ *
29
+ * The same three shapes {@link toBytes} handles, decoded as UTF-8 — a control frame arrives as a Buffer
30
+ * whichever way it was framed, and a fragmented one as an array of them.
31
+ *
32
+ * @param data - the payload as `ws` delivered it.
33
+ * @returns the payload as text.
34
+ */
35
+ export function toText(data) {
36
+ if (typeof data === 'string')
37
+ return data;
38
+ if (Array.isArray(data))
39
+ return Buffer.concat(data).toString('utf8');
40
+ if (data instanceof Uint8Array)
41
+ return Buffer.from(data.buffer, data.byteOffset, data.byteLength).toString('utf8');
42
+ return Buffer.from(data).toString('utf8');
43
+ }
26
44
  /**
27
45
  * Wire one accepted socket to the agent's audio events.
28
46
  *
@@ -48,11 +66,37 @@ export function attachAudioSocket(client, deps) {
48
66
  unsubscribe();
49
67
  deps.onDetach();
50
68
  };
69
+ /**
70
+ * Answer one control frame, if the socket is still there.
71
+ *
72
+ * The guard is here rather than at the call site because this is the only place that knows whether the
73
+ * bridge is still live, and because a control handler settles asynchronously: a reply that arrives after
74
+ * the socket went away must be dropped, not thrown.
75
+ *
76
+ * **Total**, including the write: a socket can die between the check above and the send, and a throw here
77
+ * would escape into a promise nobody awaits — and, in the plugin, would break the reply queue for every
78
+ * frame behind it.
79
+ * @param text - the reply frame.
80
+ */
81
+ const reply = (text) => {
82
+ if (stopped)
83
+ return;
84
+ try {
85
+ client.send(text);
86
+ }
87
+ catch {
88
+ stop();
89
+ }
90
+ };
51
91
  client.on('message', (data, isBinary) => {
52
- // A text frame is not part of this contract. Ignoring one is cheaper than closing the connection and
53
- // no less correct: nothing on this wire is JSON.
54
- if (!isBinary)
92
+ // A **text** frame is a control frame. This is the deliberate widening of the contract that used to
93
+ // say the opposite: the socket is the one duplex connection the client already holds, and a control
94
+ // plane that needs a second route with a second authentication is a control plane nobody builds.
95
+ // Checked before the audio path, because a text frame is never audio whatever it contains.
96
+ if (!isBinary) {
97
+ deps.onControl(toText(data), reply);
55
98
  return;
99
+ }
56
100
  const pcm16 = toBytes(data);
57
101
  if (pcm16.byteLength > deps.maxFrameBytes) {
58
102
  // 1009 = message too big. `ws` enforces its own `maxPayload` first, so reaching this means the bound
package/lib/bridge.js.map CHANGED
@@ -1 +1 @@
1
- {"version":3,"file":"bridge.js","sourceRoot":"","sources":["../src/bridge.ts"],"names":[],"mappings":"AAAA;;;;;GAKG;AAeH;;;;;;;;;GASG;AACH,MAAM,UAAU,OAAO,CAAC,IAAa;IACnC,IAAI,IAAI,YAAY,UAAU;QAAE,OAAO,IAAI,CAAA;IAC3C,IAAI,KAAK,CAAC,OAAO,CAAC,IAAI,CAAC;QAAE,OAAO,IAAI,UAAU,CAAC,MAAM,CAAC,MAAM,CAAC,IAA6B,CAAC,CAAC,CAAA;IAC5F,IAAI,IAAI,YAAY,WAAW;QAAE,OAAO,IAAI,UAAU,CAAC,IAAI,CAAC,CAAA;IAC5D,OAAO,IAAI,UAAU,CAAC,MAAM,CAAC,IAAI,CAAC,IAAuB,CAAC,CAAC,CAAA;AAC7D,CAAC;AAED;;;;;;GAMG;AACH,MAAM,UAAU,iBAAiB,CAAC,MAAmB,EAAE,IAA2B;IAChF,IAAI,OAAO,GAAG,KAAK,CAAA;IAEnB,MAAM,WAAW,GAAG,IAAI,CAAC,cAAc,CAAC,CAAC,KAAK,EAAE,EAAE;QAChD,qGAAqG;QACrG,gGAAgG;QAChG,IAAI,OAAO;YAAE,OAAM;QACnB,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,CAAA;IACpB,CAAC,CAAC,CAAA;IAEF,MAAM,IAAI,GAAG,GAAS,EAAE;QACtB,sGAAsG;QACtG,qDAAqD;QACrD,IAAI,OAAO;YAAE,OAAM;QACnB,OAAO,GAAG,IAAI,CAAA;QACd,WAAW,EAAE,CAAA;QACb,IAAI,CAAC,QAAQ,EAAE,CAAA;IACjB,CAAC,CAAA;IAED,MAAM,CAAC,EAAE,CAAC,SAAS,EAAE,CAAC,IAAI,EAAE,QAAQ,EAAE,EAAE;QACtC,qGAAqG;QACrG,iDAAiD;QACjD,IAAI,CAAC,QAAQ;YAAE,OAAM;QACrB,MAAM,KAAK,GAAG,OAAO,CAAC,IAAI,CAAC,CAAA;QAC3B,IAAI,KAAK,CAAC,UAAU,GAAG,IAAI,CAAC,aAAa,EAAE,CAAC;YAC1C,qGAAqG;YACrG,mGAAmG;YACnG,IAAI,EAAE,CAAA;YACN,MAAM,CAAC,KAAK,CAAC,IAAI,EAAE,iBAAiB,CAAC,CAAA;YACrC,OAAM;QACR,CAAC;QACD,IAAI,CAAC,OAAO,CAAC,KAAK,CAAC,CAAA;IACrB,CAAC,CAAC,CAAA;IACF,MAAM,CAAC,EAAE,CAAC,OAAO,EAAE,IAAI,CAAC,CAAA;IACxB,MAAM,CAAC,EAAE,CAAC,OAAO,EAAE,IAAI,CAAC,CAAA;IAExB,OAAO,GAAG,EAAE;QACV,IAAI,EAAE,CAAA;QACN,MAAM,CAAC,SAAS,EAAE,CAAA;IACpB,CAAC,CAAA;AACH,CAAC"}
1
+ {"version":3,"file":"bridge.js","sourceRoot":"","sources":["../src/bridge.ts"],"names":[],"mappings":"AAAA;;;;;GAKG;AA6BH;;;;;;;;;GASG;AACH,MAAM,UAAU,OAAO,CAAC,IAAa;IACnC,IAAI,IAAI,YAAY,UAAU;QAAE,OAAO,IAAI,CAAA;IAC3C,IAAI,KAAK,CAAC,OAAO,CAAC,IAAI,CAAC;QAAE,OAAO,IAAI,UAAU,CAAC,MAAM,CAAC,MAAM,CAAC,IAA6B,CAAC,CAAC,CAAA;IAC5F,IAAI,IAAI,YAAY,WAAW;QAAE,OAAO,IAAI,UAAU,CAAC,IAAI,CAAC,CAAA;IAC5D,OAAO,IAAI,UAAU,CAAC,MAAM,CAAC,IAAI,CAAC,IAAuB,CAAC,CAAC,CAAA;AAC7D,CAAC;AAED;;;;;;;;GAQG;AACH,MAAM,UAAU,MAAM,CAAC,IAAa;IAClC,IAAI,OAAO,IAAI,KAAK,QAAQ;QAAE,OAAO,IAAI,CAAA;IACzC,IAAI,KAAK,CAAC,OAAO,CAAC,IAAI,CAAC;QAAE,OAAO,MAAM,CAAC,MAAM,CAAC,IAA6B,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAA;IAC7F,IAAI,IAAI,YAAY,UAAU;QAAE,OAAO,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,MAAM,EAAE,IAAI,CAAC,UAAU,EAAE,IAAI,CAAC,UAAU,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAA;IAClH,OAAO,MAAM,CAAC,IAAI,CAAC,IAAuB,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAA;AAC9D,CAAC;AAED;;;;;;GAMG;AACH,MAAM,UAAU,iBAAiB,CAAC,MAAmB,EAAE,IAA2B;IAChF,IAAI,OAAO,GAAG,KAAK,CAAA;IAEnB,MAAM,WAAW,GAAG,IAAI,CAAC,cAAc,CAAC,CAAC,KAAK,EAAE,EAAE;QAChD,qGAAqG;QACrG,gGAAgG;QAChG,IAAI,OAAO;YAAE,OAAM;QACnB,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,CAAA;IACpB,CAAC,CAAC,CAAA;IAEF,MAAM,IAAI,GAAG,GAAS,EAAE;QACtB,sGAAsG;QACtG,qDAAqD;QACrD,IAAI,OAAO;YAAE,OAAM;QACnB,OAAO,GAAG,IAAI,CAAA;QACd,WAAW,EAAE,CAAA;QACb,IAAI,CAAC,QAAQ,EAAE,CAAA;IACjB,CAAC,CAAA;IAED;;;;;;;;;;;OAWG;IACH,MAAM,KAAK,GAAG,CAAC,IAAY,EAAQ,EAAE;QACnC,IAAI,OAAO;YAAE,OAAM;QACnB,IAAI,CAAC;YACH,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,CAAA;QACnB,CAAC;QAAC,MAAM,CAAC;YACP,IAAI,EAAE,CAAA;QACR,CAAC;IACH,CAAC,CAAA;IAED,MAAM,CAAC,EAAE,CAAC,SAAS,EAAE,CAAC,IAAI,EAAE,QAAQ,EAAE,EAAE;QACtC,oGAAoG;QACpG,oGAAoG;QACpG,iGAAiG;QACjG,2FAA2F;QAC3F,IAAI,CAAC,QAAQ,EAAE,CAAC;YACd,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC,IAAI,CAAC,EAAE,KAAK,CAAC,CAAA;YACnC,OAAM;QACR,CAAC;QACD,MAAM,KAAK,GAAG,OAAO,CAAC,IAAI,CAAC,CAAA;QAC3B,IAAI,KAAK,CAAC,UAAU,GAAG,IAAI,CAAC,aAAa,EAAE,CAAC;YAC1C,qGAAqG;YACrG,mGAAmG;YACnG,IAAI,EAAE,CAAA;YACN,MAAM,CAAC,KAAK,CAAC,IAAI,EAAE,iBAAiB,CAAC,CAAA;YACrC,OAAM;QACR,CAAC;QACD,IAAI,CAAC,OAAO,CAAC,KAAK,CAAC,CAAA;IACrB,CAAC,CAAC,CAAA;IACF,MAAM,CAAC,EAAE,CAAC,OAAO,EAAE,IAAI,CAAC,CAAA;IACxB,MAAM,CAAC,EAAE,CAAC,OAAO,EAAE,IAAI,CAAC,CAAA;IAExB,OAAO,GAAG,EAAE;QACV,IAAI,EAAE,CAAA;QACN,MAAM,CAAC,SAAS,EAAE,CAAA;IACpB,CAAC,CAAA;AACH,CAAC"}
@@ -28,7 +28,7 @@ window.__ModuleLoader__.load({ id: "dsh-realtime-audio-ws",
28
28
  * actually been read, rather than shipping a capture path that a policy might silently kill.
29
29
  */
30
30
  Object.defineProperty(exports, "__esModule", { value: true });
31
- exports.INJECTED_KEY = exports.GLOBAL_KEY = exports.MAX_BACKLOG_SECONDS = exports.CAPTURE_BUFFER = exports.SAMPLE_RATE = exports.DEFAULT_PATH = exports.inject = exports.name = void 0;
31
+ exports.STRIP_ELEMENT_ID = exports.INJECTED_KEY = exports.GLOBAL_KEY = exports.MAX_BACKLOG_SECONDS = exports.CAPTURE_BUFFER = exports.SAMPLE_RATE = exports.DEFAULT_PATH = exports.inject = exports.name = void 0;
32
32
  exports.socketUrl = socketUrl;
33
33
  exports.pageAuthority = pageAuthority;
34
34
  exports.withToken = withToken;
@@ -36,6 +36,9 @@ window.__ModuleLoader__.load({ id: "dsh-realtime-audio-ws",
36
36
  exports.float32FromPcm16 = float32FromPcm16;
37
37
  exports.defaultDeps = defaultDeps;
38
38
  exports.createAudioClient = createAudioClient;
39
+ exports.stripRoot = stripRoot;
40
+ exports.renderStrip = renderStrip;
41
+ exports.mountStrip = mountStrip;
39
42
  exports.apply = apply;
40
43
  /** Browser-side plugin name. */
41
44
  exports.name = 'realtime-audio-client';
@@ -170,12 +173,20 @@ window.__ModuleLoader__.load({ id: "dsh-realtime-audio-ws",
170
173
  let processor;
171
174
  let silence;
172
175
  let playsAt = 0;
176
+ /** The control request waiting for its reply, if any. The host answers one frame with one frame. */
177
+ let pending;
178
+ /** The tail of the request chain, so two frames are never in flight at once. */
179
+ let chain = Promise.resolve();
173
180
  const fail = (reason) => {
174
181
  current = { kind: 'failed', reason };
175
182
  return current;
176
183
  };
177
184
  /** Release everything held. Idempotent, because stop, an error and a close all reach it. */
178
185
  const release = () => {
186
+ // A request whose socket is going away must fail rather than hang: the panel disables half of itself
187
+ // while one is in flight, and a promise that never settles would leave it disabled for ever.
188
+ pending?.reject(new Error('the audio socket closed before the host answered'));
189
+ pending = undefined;
179
190
  socket?.close();
180
191
  socket = undefined;
181
192
  processor?.disconnect();
@@ -220,6 +231,12 @@ window.__ModuleLoader__.load({ id: "dsh-realtime-audio-ws",
220
231
  const source = context.createMediaStreamSource(stream);
221
232
  const node = context.createScriptProcessor(exports.CAPTURE_BUFFER, 1, 1);
222
233
  node.onaudioprocess = (event) => {
234
+ // Every quantum ships, **silent ones included, and that is load-bearing rather than incidental**: the
235
+ // provider's session timeline advances with the audio this client sends, and a context append is
236
+ // placed only while that timeline is advancing — so a client that skipped silent blocks (an
237
+ // obvious-looking saving) would silently freeze the session and lose every append after it. Measured,
238
+ // with the mechanism, in `docs/usable-window.md`; guarded by the capture-continuity case in
239
+ // `tests/client.spec.ts`.
223
240
  const pcm16 = pcm16FromFloat32(event.inputBuffer.getChannelData(0));
224
241
  socket?.send(new Uint8Array(pcm16.buffer));
225
242
  };
@@ -276,8 +293,26 @@ window.__ModuleLoader__.load({ id: "dsh-realtime-audio-ws",
276
293
  socket = opened;
277
294
  opened.binaryType = 'arraybuffer';
278
295
  opened.onmessage = (event) => {
279
- if (event.data instanceof ArrayBuffer)
296
+ // Binary frames are audio; a string is the answer to a control frame. The frame's own type is what
297
+ // tells them apart, which is why the control channel cost this path nothing.
298
+ if (event.data instanceof ArrayBuffer) {
280
299
  play(new Uint8Array(event.data));
300
+ return;
301
+ }
302
+ if (typeof event.data !== 'string')
303
+ return;
304
+ const waiting = pending;
305
+ pending = undefined;
306
+ if (waiting === undefined)
307
+ return;
308
+ try {
309
+ waiting.resolve(JSON.parse(event.data));
310
+ }
311
+ catch {
312
+ // A reply that is not JSON is a host fault, and the caller has to hear about it rather than be left
313
+ // waiting: the same rule the channel itself follows, one side out.
314
+ waiting.reject(new Error('the host answered a control frame with something that is not a reply'));
315
+ }
281
316
  };
282
317
  opened.onclose = () => {
283
318
  release();
@@ -288,23 +323,329 @@ window.__ModuleLoader__.load({ id: "dsh-realtime-audio-ws",
288
323
  opened.onerror = () => { resolve(fail('the host refused the audio socket')); };
289
324
  });
290
325
  };
291
- return { start, stop, state: () => current };
326
+ /**
327
+ * Send one frame and wait for its reply.
328
+ *
329
+ * The socket is looked up at the moment of use rather than closed over, so a request made after a
330
+ * reconnect goes out on the socket that exists now — the same rule every other live value in this
331
+ * project follows.
332
+ * @param frame - the control frame.
333
+ * @returns the parsed reply.
334
+ */
335
+ const sendFrame = (frame) => {
336
+ const current = socket;
337
+ if (current === undefined) {
338
+ return Promise.reject(new Error('not connected: connect the microphone first'));
339
+ }
340
+ return new Promise((resolve, reject) => {
341
+ pending = { resolve, reject };
342
+ current.send(frame);
343
+ });
344
+ };
345
+ const request = (frame) => {
346
+ // Chained, not concurrent: the host answers in arrival order, and a client that pipelined would be
347
+ // pairing replies with frames by guesswork. The chain is kept alive across a failure, because one
348
+ // refused frame must not silently stop every frame behind it.
349
+ const next = chain.then(() => sendFrame(frame));
350
+ chain = next.catch(() => undefined);
351
+ return next;
352
+ };
353
+ return { start, stop, state: () => current, request };
354
+ }
355
+ // ---- the strip: the panel this plugin puts in the app ---------------------------------------------------
356
+ /** The element the injected markup provides. Duplicated in the host half, which mints the row. */
357
+ exports.STRIP_ELEMENT_ID = 'dsh-realtime-strip';
358
+ /**
359
+ * Where the injected markup puts the panel, or `null` when this page does not carry it.
360
+ * @param scope - the page scope, injectable so both cases are testable without a DOM.
361
+ * @returns the panel's element, or `null`.
362
+ */
363
+ function stripRoot(scope) {
364
+ return scope.document?.getElementById(exports.STRIP_ELEMENT_ID) ?? null;
365
+ }
366
+ /**
367
+ * Escape text for an element's body and for a quoted attribute at once.
368
+ *
369
+ * One function rather than two because the two contexts differ only in which characters break out, and
370
+ * every value a panel renders came from a plugin's own configuration — a session id, a model name, or
371
+ * `instructions` somebody typed.
372
+ * @param text - the text to escape.
373
+ * @returns the text, safe in either position.
374
+ */
375
+ function escapeHtml(text) {
376
+ return text
377
+ .replaceAll('&', '&amp;')
378
+ .replaceAll('<', '&lt;')
379
+ .replaceAll('>', '&gt;')
380
+ .replaceAll('"', '&quot;');
381
+ }
382
+ /**
383
+ * Show one value as text.
384
+ * @param value - a setting's value as `status` reported it.
385
+ * @returns the text to render.
386
+ */
387
+ function show(value) {
388
+ if (value === undefined || value === null)
389
+ return '—';
390
+ return typeof value === 'string' ? value : JSON.stringify(value);
391
+ }
392
+ /**
393
+ * Render the whole panel.
394
+ *
395
+ * A pure function of the state, because that is what makes the panel testable at all: the injected script
396
+ * is three lines and the DOM is one assignment, so everything worth checking lives here.
397
+ * @param state - what the panel is showing.
398
+ * @returns the markup for the panel's element.
399
+ */
400
+ function renderStrip(state) {
401
+ const reply = state.reply;
402
+ const lines = [`<p data-dsh-strip-voice>${escapeHtml(summariseVoice(reply, state.audio))}</p>`];
403
+ if (reply !== null)
404
+ lines.push(`<p data-dsh-strip-route>${escapeHtml(summariseRoute(reply))}</p>`);
405
+ if (state.notice !== '')
406
+ lines.push(`<p data-dsh-strip-notice>${escapeHtml(state.notice)}</p>`);
407
+ lines.push(`<p data-dsh-strip-buttons>${buttons(state.audio)}</p>`);
408
+ // Restart-bound settings are omitted rather than shown read-only: the gate says no control at all, and a
409
+ // row that cannot be changed is a row a reader will try to change.
410
+ const settings = (reply?.settings ?? []).filter(entry => entry.scope !== 'restart');
411
+ lines.push(settings.length === 0
412
+ ? '<p data-dsh-strip-empty>no settings to show</p>'
413
+ : `<ul>${settings.map(entry => settingRow(entry, state.notes)).join('')}</ul>`);
414
+ return lines.join('');
415
+ }
416
+ /**
417
+ * The voice line: whether a session is live, and what it is.
418
+ * @param reply - the last status, if any.
419
+ * @param audio - what the microphone is doing.
420
+ * @returns one line of plain text.
421
+ */
422
+ function summariseVoice(reply, audio) {
423
+ const microphone = audio.kind === 'failed' ? `microphone: failed — ${audio.reason}` : `microphone: ${audio.kind}`;
424
+ if (reply === null)
425
+ return `no answer yet · ${microphone}`;
426
+ const voice = reply.voice;
427
+ // `null` is "nobody answered the question", which is a composition without an agent row — deliberately
428
+ // not the same statement as a session that is merely closed.
429
+ if (voice === null || voice === undefined)
430
+ return `voice: no agent is mounted in this profile · ${microphone}`;
431
+ const where = voice.provider === undefined ? '' : ` · ${voice.provider}/${voice.model ?? '?'}`;
432
+ const which = voice.sessionId === undefined ? '' : ` · ${voice.sessionId}`;
433
+ return `voice: ${voice.open ? 'open' : 'closed'}${where}${which} · ${microphone}`;
292
434
  }
293
435
  /**
294
- * The client plugin. Publishes the client on a global and releases it with the plugin.
436
+ * The route line: what this socket is, and the last thing the journal recorded.
437
+ * @param reply - the last status.
438
+ * @returns one line of plain text.
439
+ */
440
+ function summariseRoute(reply) {
441
+ const last = reply.journal?.last?.kind;
442
+ return [
443
+ reply.audio?.path ?? 'the audio route',
444
+ `${String(reply.audio?.clients ?? 0)} socket(s)`,
445
+ `journal ${String(reply.journal?.size ?? 0)}`,
446
+ ...last === undefined ? [] : [`last ${last}`],
447
+ ].join(' · ');
448
+ }
449
+ /**
450
+ * The buttons above the settings.
295
451
  *
296
- * There is no UI surface yet, so a global is how it is driven — a settings card is its own increment and a
297
- * bigger one than this. `inject` is empty for the same reason: nothing here needs another plugin.
452
+ * The microphone's own button is the on-switch this sprint exists to provide: what used to be a
453
+ * `globalThis` call in a devtools console is now the first control in the panel.
454
+ * @param audio - what the microphone is doing.
455
+ * @returns the markup.
456
+ */
457
+ function buttons(audio) {
458
+ const live = audio.kind === 'live';
459
+ return [
460
+ `<button data-action="${live ? 'disconnect' : 'connect'}">${live ? 'Disconnect microphone' : 'Connect microphone'}</button>`,
461
+ '<button data-action="voice-start">Start voice</button>',
462
+ '<button data-action="voice-stop">Stop voice</button>',
463
+ '<button data-action="refresh">Refresh</button>',
464
+ ].join(' ');
465
+ }
466
+ /**
467
+ * One setting's row.
468
+ *
469
+ * Three classes, three treatments, and the gate's rule is the reason: a live field gets a control, a
470
+ * session-bound field says what it would take to change it, and a restart-bound field is not rendered.
471
+ * @param entry - the setting as `status` reported it.
472
+ * @param notes - what the last change to each setting produced.
473
+ * @returns the markup for one row.
474
+ */
475
+ function settingRow(entry, notes) {
476
+ const key = escapeHtml(entry.key);
477
+ const label = `<span data-dsh-strip-label>${escapeHtml(entry.describe ?? entry.field)}</span>`;
478
+ const note = notes[entry.key] === undefined
479
+ ? ''
480
+ : `<em data-note="${key}">${escapeHtml(notes[entry.key])}</em>`;
481
+ if (entry.scope !== 'live') {
482
+ return `<li data-key="${key}" data-scope="session">${label}<b>${escapeHtml(show(entry.value))}</b>`
483
+ + `<em>fixed when the session opens — reconnect to apply</em>${note}</li>`;
484
+ }
485
+ // The channel has a verb for exactly one purpose — steering at a session — and this is the field it
486
+ // means. Everything else goes through `set <key>=<value>`.
487
+ const action = entry.field === 'sessionId' && entry.kind === 'string' ? 'steer' : 'set';
488
+ const control = entry.choices !== undefined && entry.choices.length > 0
489
+ ? `<select data-input="${key}">${choiceOptions(entry, entry.choices)}</select>`
490
+ : `<input data-input="${key}" value="${escapeHtml(show(entry.value))}">`;
491
+ const name = action === 'steer' ? 'Steer' : 'Set';
492
+ return `<li data-key="${key}" data-scope="live">${label}${control}`
493
+ + `<button data-action="${action}" data-key="${key}">${name}</button>${note}</li>`;
494
+ }
495
+ /**
496
+ * A picker's options, with the value in force always among them.
497
+ * @param entry - the setting, for its value.
498
+ * @param candidates - the values it declared. Passed in rather than read here: this is only ever called
499
+ * for a setting that has a non-empty list, and a fallback for the case that cannot happen is a branch
500
+ * the coverage gate rightly flags as dead.
501
+ * @returns the markup for each option.
502
+ */
503
+ function choiceOptions(entry, candidates) {
504
+ const current = show(entry.value);
505
+ // A value that is not among the candidates is offered anyway. A select that silently showed the first
506
+ // candidate instead of the value actually in force would be lying about the state, which is the one
507
+ // thing a control plane must not do.
508
+ const all = candidates.includes(current) ? candidates : [current, ...candidates];
509
+ return all.map(choice => `<option value="${escapeHtml(choice)}"${choice === current ? ' selected' : ''}>${escapeHtml(choice)}</option>`).join('');
510
+ }
511
+ /**
512
+ * Mount the panel on its element.
513
+ *
514
+ * Reads `status` once, then renders — and renders again after every action, because a control that
515
+ * reported an outcome without showing the state it produced would leave the reader unsure which of the two
516
+ * they are looking at.
517
+ * @param deps - the element, the transport and the microphone's own controls.
518
+ * @returns the panel's handle.
519
+ */
520
+ function mountStrip(deps) {
521
+ let reply = null;
522
+ const notes = {};
523
+ let notice = '';
524
+ const paint = () => {
525
+ deps.root.innerHTML = renderStrip({ reply, notes, notice, audio: deps.audio() });
526
+ };
527
+ const refresh = async () => {
528
+ try {
529
+ reply = await deps.request('status');
530
+ }
531
+ catch (error) {
532
+ // Nothing was sent, or nothing came back: the panel says which, rather than showing a stale state as
533
+ // if it were current.
534
+ notice = error instanceof Error ? error.message : String(error);
535
+ }
536
+ paint();
537
+ };
538
+ /**
539
+ * Send one frame and report what came back, then re-render.
540
+ * @param frame - the control frame.
541
+ * @param fieldKey - the setting the change was about, when it was about one.
542
+ * @param done - what to say when it worked.
543
+ */
544
+ const send = async (frame, fieldKey, done) => {
545
+ let line;
546
+ try {
547
+ const answer = await deps.request(frame);
548
+ // A refusal carries the reason written to be relayed verbatim, so it is relayed rather than
549
+ // translated — including for a field frozen by its class.
550
+ line = answer.ok ? done : `${answer.code ?? 'refused'}: ${answer.reason ?? 'no reason given'}`;
551
+ }
552
+ catch (error) {
553
+ line = error instanceof Error ? error.message : String(error);
554
+ }
555
+ if (fieldKey === undefined)
556
+ notice = line;
557
+ else
558
+ notes[fieldKey] = line;
559
+ await refresh();
560
+ };
561
+ const onClick = (event) => {
562
+ const action = event.target?.dataset?.action;
563
+ if (action === undefined)
564
+ return;
565
+ const key = event.target?.dataset?.key;
566
+ const value = key === undefined ? undefined : deps.root.querySelector(`[data-input="${key}"]`)?.value;
567
+ const act = async () => {
568
+ // Cleared per action: a notice from the last one is about the last one, and leaving it up would make
569
+ // a successful reconnect read as a failed one.
570
+ notice = '';
571
+ switch (action) {
572
+ case 'refresh':
573
+ await refresh();
574
+ return;
575
+ case 'disconnect':
576
+ deps.disconnect();
577
+ await refresh();
578
+ return;
579
+ case 'connect': {
580
+ const reached = await deps.connect();
581
+ if (reached.kind === 'failed')
582
+ notice = reached.reason;
583
+ await refresh();
584
+ return;
585
+ }
586
+ case 'voice-start':
587
+ await send('start', undefined, 'voice started');
588
+ return;
589
+ case 'voice-stop':
590
+ await send('stop', undefined, 'voice stopped');
591
+ return;
592
+ case 'steer':
593
+ await send(`steer ${value ?? ''}`, key, 'steered');
594
+ return;
595
+ case 'set': await send(`set ${key ?? ''}=${value ?? ''}`, key, 'applied');
596
+ }
597
+ };
598
+ void act();
599
+ };
600
+ deps.root.addEventListener('click', onClick);
601
+ return { refresh, state: () => ({ reply, notes, notice, audio: deps.audio() }) };
602
+ }
603
+ /**
604
+ * The client plugin. Publishes the client and the strip on a global, and releases them with the plugin.
605
+ *
606
+ * `inject` is empty on purpose: nothing here needs another plugin, and a client face that needs nothing
607
+ * cannot be broken by another plugin's absence.
608
+ *
609
+ * The global is **no longer the on-switch**. S2 story 3 put a panel in the app, and the panel's buttons
610
+ * are how anyone starts a microphone; the global is now the seam between the injected markup and this
611
+ * bundle, which is a different thing from a user interface that only exists in a devtools console.
298
612
  *
299
613
  * @param ctx - the client context, used only to own the lifetime.
300
614
  */
301
615
  function apply(ctx) {
302
616
  const client = createAudioClient(defaultDeps());
617
+ let mounted = false;
618
+ /**
619
+ * Render the panel into the element the injected markup provides.
620
+ *
621
+ * Called twice on purpose — once here, and once by the injected bootstrap script — because the two
622
+ * orders are both possible: this bundle is materialised by the module table, and the markup arrives with
623
+ * the body rows. Whichever runs second finds the panel already there and does nothing.
624
+ */
625
+ const mount = () => {
626
+ if (mounted)
627
+ return;
628
+ const root = stripRoot(globalThis);
629
+ if (root === null)
630
+ return;
631
+ mounted = true;
632
+ const strip = mountStrip({
633
+ root,
634
+ request: (frame) => client.request(frame),
635
+ connect: () => client.start(),
636
+ disconnect: () => { client.stop(); },
637
+ audio: () => client.state(),
638
+ });
639
+ void strip.refresh();
640
+ };
303
641
  globalThis[exports.GLOBAL_KEY] = {
304
642
  start: () => client.start(),
305
643
  stop: () => { client.stop(); },
306
644
  state: () => client.state(),
645
+ request: (frame) => client.request(frame),
646
+ mount,
307
647
  };
648
+ mount();
308
649
  // A held microphone and a live audio graph must not outlive the plugin that opened them.
309
650
  ctx.effect(() => () => { client.stop(); }, 'realtime-audio-client');
310
651
  }