@kubohiroya/turbowarp-openai-realtime-api 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md ADDED
@@ -0,0 +1,415 @@
1
+ # TurboWarp-OpenAI-Realtime-API
2
+
3
+ [English](README.md) | [日本語](README.ja.md)
4
+
5
+ A TurboWarp extension for talking with the OpenAI Realtime API by voice or text, and for letting the model call functions written in blocks. The OpenAI API key stays in a localhost relay; the browser only receives short-lived ephemeral keys.
6
+
7
+ **User guide:** [English](https://kubohiroya.github.io/turbowarp-openai-realtime-api/)
8
+
9
+ ## What it does
10
+
11
+ - Connects a TurboWarp project to the OpenAI Realtime API over WebRTC, with or without the microphone.
12
+ - Sends text messages and reports the assistant's text or spoken transcript.
13
+ - Turns `define function ... export as tool` scripts into tools the model can call, and returns each script's `return` value to the model.
14
+ - Keeps the OpenAI API key in [`@kubohiroya/capability-proxy`](https://github.com/kubohiroya/capability-proxy) on localhost. The extension pairs with the relay using a one-time code and asks it for an ephemeral key per connection.
15
+
16
+ ## Requirements and safety
17
+
18
+ - TurboWarp Desktop or TurboWarp Web with custom extensions enabled.
19
+ - [`@kubohiroya/capability-proxy`](https://github.com/kubohiroya/capability-proxy) running on `127.0.0.1` with the `openai` provider configured.
20
+ - A browser with WebRTC and microphone access. Output audio plays through the page.
21
+
22
+ > [!IMPORTANT]
23
+ > This extension must run unsandboxed. It needs microphone access, WebRTC, access to the localhost relay, and the VM runtime to start function scripts.
24
+ > Load extensions only from sources you trust.
25
+
26
+ - The relay endpoint must be a loopback origin such as `http://127.0.0.1:8787`. The extension refuses other hosts so the relay token never leaves the machine.
27
+ - The relay token and ephemeral keys are kept only in memory. They are never written into blocks or the `.sb3` project.
28
+ - Functions exported as tools can be called by the model with arguments it chooses. Treat every argument as untrusted input.
29
+
30
+ ## Installation
31
+
32
+ ### Built JavaScript
33
+
34
+ 1. Download [`dist/turbowarp-openai-realtime-api.js`](dist/turbowarp-openai-realtime-api.js?raw=1).
35
+ 2. Open **Extensions** in TurboWarp.
36
+ 3. Choose **Custom Extension** and load the file.
37
+ 4. Enable **Run without sandbox**.
38
+
39
+ The reviewed JavaScript build is committed to this repository, so users do not need Node.js or a build environment.
40
+
41
+ ### npm package
42
+
43
+ Install an exact version that you have reviewed:
44
+
45
+ ```bash
46
+ pnpm add --save-exact @kubohiroya/turbowarp-openai-realtime-api@0.1.0
47
+ ```
48
+
49
+ Standalone bundle:
50
+
51
+ ```text
52
+ node_modules/@kubohiroya/turbowarp-openai-realtime-api/dist/turbowarp-openai-realtime-api.js
53
+ ```
54
+
55
+ ## Quick start
56
+
57
+ 1. Start `capability-proxy` with the `openai` provider (see its README) and note the eight-digit pairing code it prints.
58
+ 2. Load this extension unsandboxed.
59
+ 3. Run the blocks below. Say something to the microphone, or use `send text`.
60
+
61
+ ```text
62
+ configure local relay [http://127.0.0.1:8787]
63
+ pair local relay with one-time code [12345678]
64
+ set instructions to [You are a friendly assistant. Answer briefly.]
65
+ connect to Realtime with microphone [on]
66
+ send text [Hello!]
67
+
68
+ when assistant finishes responding
69
+ say (last assistant response)
70
+
71
+ define function [get_score] description [Returns the player's score.] args schema [{"type":"object","properties":{}}] export as [tool]
72
+ return (join [{"score":] (join (score) [}]))
73
+ ```
74
+
75
+ ## Block reference
76
+
77
+ The block reference is generated from
78
+ [`src/block-definitions.json`](src/block-definitions.json). Do not edit the
79
+ generated section manually.
80
+
81
+ <!-- BEGIN GENERATED BLOCKS -->
82
+
83
+ ### `configure local relay [ENDPOINT]`
84
+
85
+ Sets the loopback origin of the local capability-proxy relay. Clears any previous pairing.
86
+
87
+ | Property | Value |
88
+ |---|---|
89
+ | Type | Command |
90
+ | Opcode | `configureRelay` |
91
+ | `ENDPOINT` | String, default: `http://127.0.0.1:8787` |
92
+
93
+ ### `pair local relay with one-time code [CODE]`
94
+
95
+ Exchanges the eight-digit code printed by the relay for a session token kept only in memory.
96
+
97
+ | Property | Value |
98
+ |---|---|
99
+ | Type | Command |
100
+ | Opcode | `pairRelay` |
101
+ | `CODE` | String, default: `00000000` |
102
+
103
+ ### `local relay paired?`
104
+
105
+ Reports whether an unexpired relay session token is held in memory.
106
+
107
+ | Property | Value |
108
+ |---|---|
109
+ | Type | Boolean |
110
+ | Opcode | `isRelayPaired` |
111
+
112
+ ### `set instructions to [TEXT]`
113
+
114
+ Sets the system instructions used by the next connection.
115
+
116
+ | Property | Value |
117
+ |---|---|
118
+ | Type | Command |
119
+ | Opcode | `setInstructions` |
120
+ | `TEXT` | String, default: `You are a friendly assistant. Answer briefly.` |
121
+
122
+ ### `set voice to [VOICE]`
123
+
124
+ Sets the output voice used by the next connection.
125
+
126
+ | Property | Value |
127
+ |---|---|
128
+ | Type | Command |
129
+ | Opcode | `setVoice` |
130
+ | `VOICE` | String, default: `marin`, choices: `alloy`, `ash`, `ballad`, `cedar`, `coral`, `echo`, `marin`, `sage`, `shimmer`, `verse` |
131
+
132
+ ### `set output to [MODE]`
133
+
134
+ Chooses spoken audio or text-only responses for the next connection.
135
+
136
+ | Property | Value |
137
+ |---|---|
138
+ | Type | Command |
139
+ | Opcode | `setOutputMode` |
140
+ | `MODE` | String, default: `audio`, choices: `audio`, `text` |
141
+
142
+ ### `set model to [MODEL]`
143
+
144
+ Chooses the Realtime model for the next connection. The relay only accepts models its operator allows; empty uses the relay default.
145
+
146
+ | Property | Value |
147
+ |---|---|
148
+ | Type | Command |
149
+ | Opcode | `setModel` |
150
+ | `MODEL` | String, default: `gpt-realtime-2.1-mini`, choices: `gpt-realtime-2.1-mini`, `gpt-realtime-2.1` |
151
+
152
+ ### `set session time limit to [SECONDS] seconds`
153
+
154
+ Disconnects automatically after this many seconds from the next connection on. 0 disables the limit.
155
+
156
+ | Property | Value |
157
+ |---|---|
158
+ | Type | Command |
159
+ | Opcode | `setSessionTimeLimit` |
160
+ | `SECONDS` | Number, default: `600` |
161
+
162
+ ### `connect to Realtime with microphone [MICROPHONE]`
163
+
164
+ Mints an ephemeral key through the relay and opens a WebRTC session. Exported functions become tools.
165
+
166
+ | Property | Value |
167
+ |---|---|
168
+ | Type | Command |
169
+ | Opcode | `connect` |
170
+ | `MICROPHONE` | String, default: `on`, choices: `on`, `off` |
171
+
172
+ ### `disconnect from Realtime`
173
+
174
+ Closes the WebRTC session, stops the microphone, and fails pending function calls.
175
+
176
+ | Property | Value |
177
+ |---|---|
178
+ | Type | Command |
179
+ | Opcode | `disconnect` |
180
+
181
+ ### `connected to Realtime?`
182
+
183
+ Reports whether the Realtime session is open.
184
+
185
+ | Property | Value |
186
+ |---|---|
187
+ | Type | Boolean |
188
+ | Opcode | `isConnected` |
189
+
190
+ ### `Realtime connection state`
191
+
192
+ Reports disconnected, connecting, connected, or failed.
193
+
194
+ | Property | Value |
195
+ |---|---|
196
+ | Type | Reporter |
197
+ | Opcode | `connectionState` |
198
+
199
+ ### `Realtime model`
200
+
201
+ Reports the model the relay used for the current or last connection.
202
+
203
+ | Property | Value |
204
+ |---|---|
205
+ | Type | Reporter |
206
+ | Opcode | `currentModel` |
207
+
208
+ ### `session elapsed seconds`
209
+
210
+ Reports seconds since the current session connected, or 0 when disconnected.
211
+
212
+ | Property | Value |
213
+ |---|---|
214
+ | Type | Reporter |
215
+ | Opcode | `sessionElapsed` |
216
+
217
+ ### `when session time limit is reached`
218
+
219
+ Starts after the session time limit disconnects the session.
220
+
221
+ | Property | Value |
222
+ |---|---|
223
+ | Type | Hat |
224
+ | Opcode | `whenSessionTimeLimitReached` |
225
+
226
+ ### `send text [TEXT]`
227
+
228
+ Adds a user text message to the conversation and asks for a response.
229
+
230
+ | Property | Value |
231
+ |---|---|
232
+ | Type | Command |
233
+ | Opcode | `sendText` |
234
+ | `TEXT` | String, default: `Hello!` |
235
+
236
+ ### `when assistant finishes responding`
237
+
238
+ Starts when a response that contains assistant text or transcript completes.
239
+
240
+ | Property | Value |
241
+ |---|---|
242
+ | Type | Hat |
243
+ | Opcode | `whenResponseDone` |
244
+
245
+ ### `last assistant response`
246
+
247
+ Reports the text or audio transcript of the most recent completed assistant response.
248
+
249
+ | Property | Value |
250
+ |---|---|
251
+ | Type | Reporter |
252
+ | Opcode | `lastResponseText` |
253
+
254
+ ### `define function [NAME] description [DESCRIPTION] args schema [SCHEMA] export as [EXPORT]`
255
+
256
+ Defines a function value. With export as tool, the model can call it; NAME, DESCRIPTION, and SCHEMA must be literal text.
257
+
258
+ | Property | Value |
259
+ |---|---|
260
+ | Type | Hat |
261
+ | Opcode | `defineFunction` |
262
+ | `NAME` | String, default: `get_score` |
263
+ | `DESCRIPTION` | String, default: `Returns the player's current score.` |
264
+ | `SCHEMA` | String, default: `{"type":"object","properties":{}}` |
265
+ | `EXPORT` | String, default: `tool`, choices: `tool`, `none` |
266
+
267
+ ### `function argument [PATH]`
268
+
269
+ Inside a function, reports the argument at a dotted path such as city or items.0.name.
270
+
271
+ | Property | Value |
272
+ |---|---|
273
+ | Type | Reporter |
274
+ | Opcode | `functionArgument` |
275
+ | `PATH` | String, default: `city` |
276
+
277
+ ### `function arguments JSON`
278
+
279
+ Inside a function, reports all arguments as JSON text.
280
+
281
+ | Property | Value |
282
+ |---|---|
283
+ | Type | Reporter |
284
+ | Opcode | `functionArgumentsJson` |
285
+
286
+ ### `return [VALUE]`
287
+
288
+ Inside a function, returns a value and ends the script. JSON text is returned as JSON; other text is returned as a string.
289
+
290
+ | Property | Value |
291
+ |---|---|
292
+ | Type | Command |
293
+ | Opcode | `returnValue` |
294
+ | `VALUE` | String, default: `{"score":10}` |
295
+
296
+ ### `Realtime usage [FIELD]`
297
+
298
+ Reports token usage totals since the last reset. costUSD is an estimate from published prices; the OpenAI dashboard is authoritative.
299
+
300
+ | Property | Value |
301
+ |---|---|
302
+ | Type | Reporter |
303
+ | Opcode | `usageValue` |
304
+ | `FIELD` | String, default: `costUSD`, choices: `costUSD`, `responses`, `inputTokens`, `outputTokens`, `cachedInputTokens`, `textInputTokens`, `audioInputTokens`, `textOutputTokens`, `audioOutputTokens` |
305
+
306
+ ### `reset Realtime usage`
307
+
308
+ Clears the usage totals.
309
+
310
+ | Property | Value |
311
+ |---|---|
312
+ | Type | Command |
313
+ | Opcode | `resetUsage` |
314
+
315
+ ### `last Realtime error`
316
+
317
+ Reports the most recent relay, connection, or API error, or an empty string.
318
+
319
+ | Property | Value |
320
+ |---|---|
321
+ | Type | Reporter |
322
+ | Opcode | `lastError` |
323
+
324
+ <!-- END GENERATED BLOCKS -->
325
+
326
+ ## Important behavior
327
+
328
+ | Situation | Behavior |
329
+ |---|---|
330
+ | Relay endpoint is not a loopback origin | The block records an error in `last Realtime error`; the previous endpoint is kept. |
331
+ | Connecting before pairing, or after the relay session expired | `connect` records an error and does not contact OpenAI. |
332
+ | Invalid `define function` hats (non-literal NAME/DESCRIPTION/SCHEMA, invalid JSON schema, duplicate names, a tool without description) | `connect` records all problems and does not connect. |
333
+ | Settings changed while connected | Instructions, voice, output, and tools apply to the next connection. |
334
+ | The model calls a function | Every `define function` hat is started; only the one with the matching NAME runs. Its `return` value is sent back as JSON, then a new response is requested. |
335
+ | A function script ends without `return` | The result is `null`. |
336
+ | A function takes longer than 30 seconds | The model receives `{"error": "Function ... timed out."}`. |
337
+ | Several calls to the same function | They run one at a time in order. Different functions run concurrently. |
338
+ | The model calls a function that is not exported as a tool | The call is refused and the model receives an error. |
339
+ | Project stop | Pending function calls fail; the Realtime session stays open until `disconnect`. |
340
+ | Disconnect or connection loss | The microphone is released, audio stops, and pending function calls fail. |
341
+ | `set model to` | The model is sent with the next connection. The relay accepts only models listed in its `allowedModels`; empty uses the relay default. `Realtime model` reports the model actually used. |
342
+ | `set session time limit to` | From the next connection, the session disconnects after that many seconds and `when session time limit is reached` starts. 0 disables the limit. |
343
+ | `Realtime usage [FIELD]` | Totals from every `response.done`, including function-call rounds, since the last reset. `costUSD` is an estimate from published prices (see `src/usage.ts`); the OpenAI dashboard is authoritative. |
344
+
345
+ ## Composition API
346
+
347
+ Importing the Composition API does not register the standalone TurboWarp extension. A downstream
348
+ extension such as a voice-chat extension owns its blocks and hats and forwards them here. Named
349
+ functions come from [`@kubohiroya/turbowarp-named-functions`](https://github.com/kubohiroya/turbowarp-named-functions);
350
+ pass a shared instance with `functions` when the downstream extension also uses it.
351
+
352
+ ```ts
353
+ import {createRealtimeComposition} from '@kubohiroya/turbowarp-openai-realtime-api/composition';
354
+
355
+ const realtime = createRealtimeComposition({
356
+ runtime: Scratch.vm.runtime,
357
+ functionHatOpcode: 'myextension_defineFunction'
358
+ });
359
+
360
+ realtime.subscribe((event) => {
361
+ if (event.type === 'response') console.log(event.text);
362
+ if (event.type === 'sessionTimeLimitReached') console.log('time for a break');
363
+ });
364
+
365
+ realtime.configureRelay('http://127.0.0.1:8787');
366
+ await realtime.pairRelay('12345678');
367
+ realtime.setModel('gpt-realtime-2.1-mini');
368
+ realtime.setOutputMode('text');
369
+ realtime.setSessionTimeLimit(600);
370
+ await realtime.connect({microphone: false});
371
+ realtime.sendText('Hello!');
372
+
373
+ realtime.release();
374
+ ```
375
+
376
+ | Member | Purpose |
377
+ |---|---|
378
+ | `configureRelay` / `pairRelay` / `isRelayPaired` | Pair with the localhost relay |
379
+ | `setModel` / `setInstructions` / `setVoice` / `setOutputMode` / `setSessionTimeLimit` | Settings for the next connection |
380
+ | `connect` / `disconnect` / `state` / `activeModel` / `sessionElapsedSeconds` | Session lifecycle |
381
+ | `sendText` / `lastResponseText` | Text conversation |
382
+ | `matchFunctionHat` / `functionArguments` / `returnFromFunction` / `functions` | Bridge to named functions exported as tools |
383
+ | `usage` / `resetUsage` | Token usage and estimated cost |
384
+ | `subscribe` | Events: `state`, `response`, `usage`, `error`, `sessionTimeLimitReached` |
385
+ | `release` | Disconnect and detach listeners |
386
+
387
+ ## Compatibility
388
+
389
+ | Identifier | Value | Stability |
390
+ |---|---|---|
391
+ | Product name | `TurboWarp-OpenAI-Realtime-API` | Human-facing |
392
+ | Repository | `kubohiroya/turbowarp-openai-realtime-api` | Current source location |
393
+ | npm package | `@kubohiroya/turbowarp-openai-realtime-api` | Public package contract |
394
+ | Extension ID | `kubohiroyaopenairealtime` | Stored in SB3; migration required to change |
395
+ | Relay route | `POST /v1/openai/realtime/client-secrets` | Contract with `capability-proxy` |
396
+
397
+ ## Development
398
+
399
+ Use Node.js 22.18.0 or newer and the pnpm version declared by `packageManager`.
400
+
401
+ ```bash
402
+ corepack enable
403
+ pnpm install --frozen-lockfile
404
+ pnpm check
405
+ ```
406
+
407
+ To also run the integration test on a real headless TurboWarp VM, point `SCRATCH_VM_PATH` at a TurboWarp `scratch-vm` checkout:
408
+
409
+ ```bash
410
+ SCRATCH_VM_PATH=/path/to/TurboWarp/scratch-vm pnpm test
411
+ ```
412
+
413
+ ## License
414
+
415
+ [Mozilla Public License 2.0](LICENSE) (SPDX: `MPL-2.0`).