@fatihdogmus/opencode-voice-input 0.0.0-stage → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +152 -2
- package/dist/gui-experimental.cjs +13 -0
- package/package.json +59 -3
- package/src/contract.ts +9 -0
- package/src/gui.tsx +274 -0
- package/src/index.ts +115 -0
- package/src/inject.tsx +167 -0
- package/src/main.ts +13 -0
- package/src/rpc.ts +52 -0
- package/src/tui-audio.ts +122 -0
- package/src/tui.tsx +190 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Muhammed Fatih Doğmuş
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -1,3 +1,153 @@
|
|
|
1
|
-
#
|
|
1
|
+
# @fatihdogmus/opencode-voice-input
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
OpenCode plugin that adds voice prompts to OpenCode Desktop and the TUI, transcribed with the OpenAI account selected in
|
|
4
|
+
OpenCode. The desktop app:
|
|
5
|
+
|
|
6
|
+
- **Mic button** next to the send button. Click to record (it turns red with a timer), click again to transcribe. The
|
|
7
|
+
text is typed into the prompt at the cursor for you to review and send (nothing is sent automatically). Works in
|
|
8
|
+
session composers and the new-session composer.
|
|
9
|
+
- This placement uses the composer's internal DOM, not the plugin API: the button is inserted into the static toolbar
|
|
10
|
+
row before `[data-slot="composer-actions"]` and styled with the app's `icon-button-v2` styles. If an app update
|
|
11
|
+
moves that element, the plugin falls back to a **🎙 Dictate** pill above the composer (plugin API), so it keeps
|
|
12
|
+
working.
|
|
13
|
+
- Set `"placement": "pill"` in the options to use only the pill.
|
|
14
|
+
- **Shortcut** `⌘⌥M` (changeable in **Settings > Shortcuts**), also `/dictate` and the command palette.
|
|
15
|
+
- **Cancel voice dictation** discards a recording in progress.
|
|
16
|
+
|
|
17
|
+
## Install
|
|
18
|
+
|
|
19
|
+
Add the package to your global `~/.config/opencode/opencode.json(c)`. OpenCode Desktop loads GUI parts only from the
|
|
20
|
+
global config, and the TUI loads the TUI part of every plugin its server runs:
|
|
21
|
+
|
|
22
|
+
```jsonc
|
|
23
|
+
{
|
|
24
|
+
"$schema": "https://opencode.ai/config.json",
|
|
25
|
+
"plugins": ["@fatihdogmus/opencode-voice-input"],
|
|
26
|
+
}
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
Connect an OpenAI account with `/connect` if you haven't. The TUI also needs `ffmpeg` (`brew install ffmpeg`).
|
|
30
|
+
|
|
31
|
+
## For agents
|
|
32
|
+
|
|
33
|
+
Follow these steps when a user asks you to install this plugin.
|
|
34
|
+
|
|
35
|
+
1. **Check the OpenAI account.** `opencode auth list` must show an `OpenAI` row. If it doesn't, ask the user to run
|
|
36
|
+
`/connect` in OpenCode and sign in; don't start an OAuth flow yourself.
|
|
37
|
+
2. **Check ffmpeg** (only needed for the TUI) with `ffmpeg -version`. If it's missing, ask before installing it
|
|
38
|
+
(`brew install ffmpeg` on macOS, the distribution's `ffmpeg` package on Linux, `winget install Gyan.FFmpeg` on
|
|
39
|
+
Windows).
|
|
40
|
+
3. **Install it globally** with OpenCode's plugin command. It installs the package and adds it to the user's global
|
|
41
|
+
config. Never add this plugin to a project config: OpenCode Desktop loads GUI parts only from the global config.
|
|
42
|
+
Don't edit OpenCode's database or run `npm install` yourself.
|
|
43
|
+
|
|
44
|
+
```sh
|
|
45
|
+
opencode plugin add @fatihdogmus/opencode-voice-input
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
Set [options](#options) such as `language` only if the user asks; that means replacing the entry in
|
|
49
|
+
`~/.config/opencode/opencode.json(c)` with the object form and keeping every other entry.
|
|
50
|
+
4. **Verify** that `opencode plugin list` has a row with the ID `voice-input`. Config changes reload on their own;
|
|
51
|
+
don't restart OpenCode or its service. Optionally, check transcription end to end. This sends a short clip through
|
|
52
|
+
the user's OpenAI account; on macOS:
|
|
53
|
+
|
|
54
|
+
```sh
|
|
55
|
+
say -o /tmp/voice-check.aiff "hello from voice input"
|
|
56
|
+
ffmpeg -loglevel error -y -i /tmp/voice-check.aiff -c:a libopus /tmp/voice-check.webm
|
|
57
|
+
opencode api post /api/rpc/voice-input/transcribe \
|
|
58
|
+
--data "{\"input\":{\"audio\":\"$(base64 < /tmp/voice-check.webm | tr -d '\n')\",\"mimeType\":\"audio/webm\"}}"
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
The reply's `output.text` should contain the sentence, and `output.account` names the account used.
|
|
62
|
+
5. **Tell the user** how to use it: the mic button next to send (or `⌘⌥M`) in OpenCode Desktop, and `ctrl+x v` (or
|
|
63
|
+
`/dictate`) in the TUI. macOS asks for microphone access the first time they record. A TUI that was already open may
|
|
64
|
+
need to be reopened before the shortcut works.
|
|
65
|
+
|
|
66
|
+
To remove it: `opencode plugin remove @fatihdogmus/opencode-voice-input`.
|
|
67
|
+
|
|
68
|
+
## TUI
|
|
69
|
+
|
|
70
|
+
The same package has a `./tui` part (`src/tui.tsx`), which the TUI loads automatically because the plugin is in the
|
|
71
|
+
server's config.
|
|
72
|
+
|
|
73
|
+
- **`<leader>v`** (`ctrl+x v` by default) or `/dictate` starts recording and stops it. The prompt footer shows
|
|
74
|
+
nothing while idle; it shows `● 0:07 stop` while recording and `◌ transcribing…` after. **Cancel voice dictation** and
|
|
75
|
+
**Copy last dictation** are in the command palette. Rebind the key in `cli.json` under `keybinds` as
|
|
76
|
+
`"voice-input.dictate"`.
|
|
77
|
+
- Recording uses `ffmpeg` (`brew install ffmpeg`) with the system default microphone (macOS AVFoundation, Linux
|
|
78
|
+
PulseAudio). macOS asks for microphone access for your terminal app the first time.
|
|
79
|
+
- The TUI plugin API cannot write to the prompt, so the transcript is put on the clipboard and the TUI's own
|
|
80
|
+
`prompt.paste` command inserts it at the cursor. Your previous clipboard text is restored 1.5 s later (set
|
|
81
|
+
`"restoreClipboard": false` to keep the transcript on the clipboard). If the paste can't happen, **Copy last
|
|
82
|
+
dictation** copies it again.
|
|
83
|
+
- After adding the `./tui` export to an already running shared service, run `opencode service restart` once: Bun
|
|
84
|
+
caches `package.json` exports, so the running service doesn't see the new part until it restarts.
|
|
85
|
+
- For testing without a microphone, `OPENCODE_VOICE_INPUT_TEST_FILE=/path/to/audio opencode` records that file instead.
|
|
86
|
+
|
|
87
|
+
## How transcription works
|
|
88
|
+
|
|
89
|
+
The window records WebM/Opus with `MediaRecorder` and sends it to the plugin's server part (`src/index.ts`) through
|
|
90
|
+
its RPC, `voice-input.transcribe`:
|
|
91
|
+
|
|
92
|
+
1. `ctx.integration.connection.active("openai")` returns the OpenAI account selected in OpenCode (`/connect`,
|
|
93
|
+
`opencode auth switch openai <label>`).
|
|
94
|
+
2. `ctx.integration.connection.resolve(connection)` returns its credential. OpenCode refreshes an OAuth token that
|
|
95
|
+
expires within 5 minutes and stores the rotated refresh token, so the plugin never refreshes tokens itself.
|
|
96
|
+
3. The audio is uploaded depending on the account's sign-in method:
|
|
97
|
+
- **Codex browser / device code** (`chatgpt-browser`, `chatgpt-headless`): `POST
|
|
98
|
+
https://chatgpt.com/backend-api/transcribe` with the ChatGPT access token, `ChatGPT-Account-Id`,
|
|
99
|
+
`originator: Codex Desktop` and a Codex Desktop user agent, as [codex-asr](https://github.com/Wangnov/codex-asr)
|
|
100
|
+
does. This is Codex Desktop's dictation endpoint; it is not a public API and can change.
|
|
101
|
+
- **API key**, `OPENAI_API_KEY`, or **Sign in with ChatGPT** (token sharing): `POST
|
|
102
|
+
https://api.openai.com/v1/audio/transcriptions` with model `gpt-4o-mini-transcribe` (option `model`).
|
|
103
|
+
|
|
104
|
+
The token stays in the OpenCode server; the window only sends audio and receives text.
|
|
105
|
+
|
|
106
|
+
Test it without the GUI:
|
|
107
|
+
|
|
108
|
+
```sh
|
|
109
|
+
opencode api post /api/rpc/voice-input/transcribe --data '{"input":{"audio":"<base64>","mimeType":"audio/webm"}}'
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
## Options
|
|
113
|
+
|
|
114
|
+
```jsonc
|
|
115
|
+
{
|
|
116
|
+
"plugins": [
|
|
117
|
+
{
|
|
118
|
+
"package": "@fatihdogmus/opencode-voice-input",
|
|
119
|
+
"options": {
|
|
120
|
+
"language": "tr", // optional language hint
|
|
121
|
+
"model": "gpt-4o-mini-transcribe", // public-API model for API-key / token-sharing accounts
|
|
122
|
+
"placement": "composer", // desktop: "composer" (mic next to send, default) | "pill"
|
|
123
|
+
"microphone": "default", // TUI: ffmpeg input device (avfoundation index/name, pulse source, or dshow name)
|
|
124
|
+
"ffmpegPath": "/opt/homebrew/bin/ffmpeg", // TUI: optional
|
|
125
|
+
"restoreClipboard": true, // TUI: put the previous clipboard text back after pasting
|
|
126
|
+
},
|
|
127
|
+
},
|
|
128
|
+
],
|
|
129
|
+
}
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
## Files
|
|
133
|
+
|
|
134
|
+
- `src/index.ts`: server part, the `voice-input.transcribe` RPC.
|
|
135
|
+
- `src/rpc.ts`: the RPC contract.
|
|
136
|
+
- `src/gui.tsx`: window part: recorder, commands, pill, inserting the text.
|
|
137
|
+
- `src/inject.tsx`: the mic button next to the send button.
|
|
138
|
+
- `src/tui.tsx`, `src/tui-audio.ts`: TUI part: shortcut and commands, recording status, ffmpeg recorder, clipboard
|
|
139
|
+
paste.
|
|
140
|
+
- `src/main.ts`, `src/contract.ts`: asks macOS for microphone access from the app's main process.
|
|
141
|
+
|
|
142
|
+
## Develop
|
|
143
|
+
|
|
144
|
+
This package lives in the [opencode-plugins](https://github.com/fatihdogmus/opencode-plugins) monorepo. From the
|
|
145
|
+
repository root:
|
|
146
|
+
|
|
147
|
+
```sh
|
|
148
|
+
pnpm install
|
|
149
|
+
pnpm --filter @fatihdogmus/opencode-voice-input check # typecheck + build the desktop bundle
|
|
150
|
+
pnpm --filter @fatihdogmus/opencode-voice-input dev # vite build --watch; each rebuild reloads the GUI
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
To try a local checkout, point the plugin entry at `packages/voice-input` instead of the package name.
|
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
Object.defineProperties(exports,{__esModule:{value:!0},[Symbol.toStringTag]:{value:`Module`}});let e=require("solid-js"),t=require("@opencode/plugin/gui"),n=require("effect"),r=require("solid-js/web"),i=require("@opencode/plugin/rpc"),a=require("electron");var o=t.Ipc.define({id:`voice-input.voice`,methods:{requestMicrophone:{output:n.Schema.Boolean}}}),s=(0,r.template)(`<svg viewBox="0 0 16 16"fill=none aria-hidden=true><rect x=5.5 y=1.5 width=5 height=8.5 rx=2.5 stroke=currentColor></rect><path d="M3 7.75a5 5 0 0 0 10 0M8 12.75v1.75M5.75 14.5h4.5"stroke=currentColor stroke-linecap=round>`),c=(0,r.template)(`<svg viewBox="0 0 16 16"aria-hidden=true><rect x=4 y=4 width=8 height=8 rx=1.5 fill=currentColor>`),l=(0,r.template)(`<svg viewBox="0 0 16 16"fill=none aria-hidden=true><circle cx=8 cy=8 r=5.5 stroke=currentColor stroke-opacity=.25 stroke-width=1.5></circle><path d="M13.5 8A5.5 5.5 0 0 0 8 2.5"stroke=currentColor stroke-width=1.5 stroke-linecap=round>`),u=(0,r.template)(`<span data-slot=voice-input-dot>`),d=(0,r.template)(`<span data-slot=voice-input-time>`),f=(0,r.template)(`<button type=button data-component=icon-button-v2 data-size=normal data-variant=ghost-muted>`),p=`[data-component="composer"] [data-slot="composer-actions"]`,m=`data-voice-input`,h=`
|
|
2
|
+
[${m}] { display: flex; align-items: center; gap: 4px; flex-shrink: 0; margin-inline-end: 6px; }
|
|
3
|
+
[${m}] > button[data-component="icon-button-v2"] { width: 28px; height: 28px; border-radius: 6px; }
|
|
4
|
+
[${m}] > button svg { width: 16px; height: 16px; display: block; }
|
|
5
|
+
[${m}][data-phase="recording"] > button { color: var(--v2-state-fg-danger); background-color: color-mix(in srgb, var(--v2-state-fg-danger) 14%, transparent); }
|
|
6
|
+
[${m}][data-phase="recording"] > button:hover { background-color: color-mix(in srgb, var(--v2-state-fg-danger) 22%, transparent); }
|
|
7
|
+
[${m}][data-phase="transcribing"] > button svg, [${m}][data-phase="starting"] > button svg { animation: voice-input-spin .9s linear infinite; }
|
|
8
|
+
[${m}] [data-slot="voice-input-time"] { font-size: 11px; font-weight: 530; font-variant-numeric: tabular-nums; color: var(--v2-state-fg-danger); }
|
|
9
|
+
[${m}][data-phase="recording"] [data-slot="voice-input-dot"] { width: 6px; height: 6px; border-radius: 9999px; background: var(--v2-state-fg-danger); animation: voice-input-pulse 1.2s ease-in-out infinite; }
|
|
10
|
+
@keyframes voice-input-spin { to { transform: rotate(360deg); } }
|
|
11
|
+
@keyframes voice-input-pulse { 50% { opacity: .25; } }
|
|
12
|
+
:root[data-motion="reduce"] [${m}] * { animation: none !important; }
|
|
13
|
+
`,g=()=>s(),_=()=>c(),v=()=>l();function y(t){let n=()=>{let e=t.model.phase(),n=t.model.owner();return e===`idle`||!n||n===t.host?e:`busy`};return(0,e.createEffect)(()=>{t.host.dataset.phase=n()===`busy`?`idle`:n()}),[(0,r.createComponent)(e.Switch,{get children(){return(0,r.createComponent)(e.Match,{get when(){return n()===`recording`},get children(){return[u(),(()=>{var e=d();return(0,r.insert)(e,()=>t.model.elapsed()),e})()]}})}}),(()=>{var i=f();return i.$$click=e=>{e.preventDefault(),e.stopPropagation(),t.model.onClick(t.host)},i.$$mousedown=e=>e.preventDefault(),(0,r.insert)(i,(0,r.createComponent)(e.Switch,{get fallback(){return(0,r.createComponent)(g,{})},get children(){return[(0,r.createComponent)(e.Match,{get when(){return n()===`recording`},get children(){return(0,r.createComponent)(_,{})}}),(0,r.createComponent)(e.Match,{get when(){return n()===`transcribing`||n()===`starting`},get children(){return(0,r.createComponent)(v,{})}})]}})),(0,r.effect)(e=>{var a=t.model.title(),o=t.model.title(),s=n()===`recording`,c=n()===`busy`||n()===`transcribing`||n()===`starting`;return a!==e.e&&(0,r.setAttribute)(i,`title`,e.e=a),o!==e.t&&(0,r.setAttribute)(i,`aria-label`,e.t=o),s!==e.a&&(0,r.setAttribute)(i,`aria-pressed`,e.a=s),c!==e.o&&(i.disabled=e.o=c),e},{e:void 0,t:void 0,a:void 0,o:void 0}),i})()]}function b(t){let n=new Map,[i,a]=(0,e.createSignal)(0),o=0,s=()=>{o=0;for(let e of document.querySelectorAll(p)){let i=e.parentElement;if(!i||i.querySelector(`:scope > [${m}]`))continue;let a=document.createElement(`div`);a.setAttribute(m,``),i.insertBefore(a,e),n.set(a,(0,r.render)(()=>(0,r.createComponent)(y,{host:a,model:t}),a))}for(let[e,t]of n)e.isConnected||(t(),n.delete(e));a(n.size)},c=new MutationObserver(()=>{o||=requestAnimationFrame(s)});return c.observe(document.body,{childList:!0,subtree:!0}),s(),{count:i,dispose(){c.disconnect(),o&&cancelAnimationFrame(o);for(let[e,t]of n)t(),e.remove();n.clear()}}}function x(e){let t=e?.closest(`[data-component="composer"]`);return t?.isConnected?t:document.activeElement?.closest(`[data-component="composer"]`)||[...document.querySelectorAll(`[data-component="composer"]`)].find(e=>e.offsetParent)}function S(e,t){let n=e.querySelector(`[data-component="composer-editor"]`);return n?(n.focus(),document.execCommand(`insertText`,!1,t)):!1}(0,r.delegateEvents)([`mousedown`,`click`]);var C={type:`object`,properties:{message:{type:`string`}},required:[`message`],additionalProperties:!1},w=i.Rpc.define({id:`voice-input`,methods:{transcribe:{input:{type:`object`,properties:{audio:{type:`string`,description:`Base64-encoded recording`},mimeType:{type:`string`},language:{type:`string`}},required:[`audio`,`mimeType`],additionalProperties:!1},output:{type:`object`,properties:{text:{type:`string`},account:{type:`string`}},required:[`text`,`account`],additionalProperties:!1},errors:{no_account:C,unsupported_account:C,upstream:C}}},events:{}}),T=e=>{e.provide(o,{requestMicrophone:async()=>process.platform!==`darwin`||a.systemPreferences.getMediaAccessStatus(`microphone`)===`granted`||a.systemPreferences.askForMediaAccess(`microphone`)})},E=t.Plugin.define({i18n:{en:{group:`Voice input`,"pill.idle":`🎙 Dictate`,"pill.idle.title":`Record a voice prompt and transcribe it with your OpenAI account`,"pill.starting":`🎙 Starting…`,"pill.recording":`Stop {{time}}`,"pill.recording.title":`Stop recording and transcribe`,"pill.transcribing":`Transcribing…`,"button.idle":`Dictate (⌘⌥M)`,"button.recording":`Stop and transcribe (⌘⌥M)`,"button.busy":`Transcribing…`,"command.toggle":`Start or stop voice dictation`,"command.cancel":`Cancel voice dictation`,"error.title":`Voice input failed`,"error.microphone":`Microphone access was denied. Allow OpenCode in System Settings › Privacy & Security › Microphone.`,nospeech:`No speech was heard`,copied:`No prompt is open, so the transcript was copied to the clipboard`}}}),D=6e5,O=[`audio/webm;codecs=opus`,`audio/webm`,`audio/ogg;codecs=opus`,`audio/mp4`],k=e=>new Promise((t,n)=>{let r=new FileReader;r.onload=()=>t(String(r.result).replace(/^data:[^,]*,/,``)),r.onerror=()=>n(r.error??Error(`Could not read the recording`)),r.readAsDataURL(e)}),A=e=>{let t=e?.data;return typeof t?.message==`string`?t.message:e instanceof Error?e.message:String(e)},j=class{stream;recorder;chunks=[];async start(){this.stream=await navigator.mediaDevices.getUserMedia({audio:{echoCancellation:!0,noiseSuppression:!0,autoGainControl:!0}});let e=O.find(e=>MediaRecorder.isTypeSupported(e));this.chunks=[],this.recorder=new MediaRecorder(this.stream,e?{mimeType:e}:void 0),this.recorder.ondataavailable=e=>{e.data.size>0&&this.chunks.push(e.data)},this.recorder.start(250)}stop(){let e=this.recorder;return e?new Promise(t=>{e.onstop=()=>{let n=new Blob(this.chunks,{type:e.mimeType||`audio/webm`});this.release(),t(n)},e.stop()}):Promise.resolve(void 0)}release(){this.recorder?.state===`recording`&&this.recorder.stop(),this.stream?.getTracks().forEach(e=>e.stop()),this.stream=void 0,this.recorder=void 0,this.chunks=[]}},M=n=>{let r=n.main(o),i=new j,[a,s]=(0,e.createSignal)({kind:`idle`}),[c,l]=(0,e.createSignal)(Date.now()),u,[d,f]=(0,e.createSignal)(),p=n.options.placement===`pill`?`pill`:`composer`,m=()=>typeof n.options.language==`string`?n.options.language:``,g=()=>{u&&clearInterval(u),u=void 0},_=e=>{g(),i.release(),s({kind:`idle`}),f(void 0),n.toast.show({variant:`error`,title:n.t(`error.title`),description:A(e)})},v=e=>{if(!e){n.toast.show({title:n.t(`nospeech`)});return}let t=n.screen.current()?.composer,r=x(d());if(!t&&r&&S(r,e))return;if(!t){n.system.copy(e),n.toast.show({title:n.t(`copied`)});return}let i=t.text();t.insert(i&&!/\s$/.test(i)?` ${e}`:e),t.focus()},y=async e=>{if(a().kind===`idle`){f(e),s({kind:`starting`});try{if(!await r.requestMicrophone())throw Error(n.t(`error.microphone`));await i.start();let e=Date.now();l(e),s({kind:`recording`,since:e}),u=setInterval(()=>{l(Date.now()),Date.now()-e>D&&C()},500)}catch(e){_(e)}}},C=async()=>{if(a().kind===`recording`){g(),s({kind:`transcribing`});try{let e=await i.stop();if(!e||e.size===0)throw Error(n.t(`nospeech`));let t=await k(e),r=(await n.client.rpc(w).transcribe({audio:t,mimeType:e.type,language:m()})).text;s({kind:`idle`}),v(r),f(void 0)}catch(e){_(e)}}},T=()=>{if(a().kind!==`recording`&&a().kind!==`starting`)return!1;g(),i.release(),s({kind:`idle`}),f(void 0)},E=e=>{let t=a().kind;t===`idle`?y(e??N()):t===`recording`&&C()},O=e=>{let t=Math.max(0,Math.floor((c()-e)/1e3));return`${Math.floor(t/60)}:${String(t%60).padStart(2,`0`)}`},M=p===`composer`?b({phase:()=>a().kind,owner:d,elapsed:()=>{let e=a();return e.kind===`recording`?O(e.since):``},title:()=>{let e=a().kind;return e===`recording`?n.t(`button.recording`):e===`idle`?n.t(`button.idle`):n.t(`button.busy`)},onClick:e=>E(e)}):void 0;M&&n.add(t.Style,h);function N(){return x(void 0)?.querySelector(`[data-voice-input]`)??void 0}n.add(t.ComposerPill,{id:`mic`,show:()=>{if(M&&M.count()>0)return;let e=a();switch(e.kind){case`idle`:return{label:n.t(`pill.idle`),title:n.t(`pill.idle.title`),run:()=>E()};case`starting`:return{label:n.t(`pill.starting`)};case`recording`:return{label:n.t(`pill.recording`,{time:O(e.since)}),title:n.t(`pill.recording.title`),icon:`stop`,tone:`danger`,run:()=>E()};case`transcribing`:return{label:n.t(`pill.transcribing`),icon:`outline-wave-audio`,tone:`warning`}}}}),n.add(t.Command,()=>({id:`toggle`,title:n.t(`command.toggle`),group:n.t(`group`),keywords:`voice mic microphone dictate speech record transcribe`,section:`prompt`,bind:`mod+alt+m`,editable:!0,slash:{name:`dictate`},run:()=>E()})),n.add(t.Command,()=>({id:`cancel`,title:n.t(`command.cancel`),group:n.t(`group`),section:`prompt`,enabled:a().kind===`recording`||a().kind===`starting`,run:()=>T()})),(0,e.onCleanup)(()=>{M?.dispose(),g(),i.release()})},N=t.Plugin.define({...E,setup:M,main:T});exports.default=N;
|
package/package.json
CHANGED
|
@@ -1,6 +1,62 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@fatihdogmus/opencode-voice-input",
|
|
3
|
-
"version": "
|
|
4
|
-
"
|
|
5
|
-
"
|
|
3
|
+
"version": "1.0.0",
|
|
4
|
+
"description": "Voice prompts for OpenCode Desktop and the TUI, transcribed with the OpenAI account selected in OpenCode",
|
|
5
|
+
"license": "MIT",
|
|
6
|
+
"type": "module",
|
|
7
|
+
"engines": {
|
|
8
|
+
"node": ">=26"
|
|
9
|
+
},
|
|
10
|
+
"repository": {
|
|
11
|
+
"type": "git",
|
|
12
|
+
"url": "git+https://github.com/fatihdogmus/opencode-plugins.git",
|
|
13
|
+
"directory": "packages/voice-input"
|
|
14
|
+
},
|
|
15
|
+
"homepage": "https://github.com/fatihdogmus/opencode-plugins/tree/main/packages/voice-input#readme",
|
|
16
|
+
"bugs": {
|
|
17
|
+
"url": "https://github.com/fatihdogmus/opencode-plugins/issues"
|
|
18
|
+
},
|
|
19
|
+
"keywords": [
|
|
20
|
+
"opencode",
|
|
21
|
+
"voice",
|
|
22
|
+
"dictation",
|
|
23
|
+
"speech-to-text",
|
|
24
|
+
"transcription",
|
|
25
|
+
"plugin"
|
|
26
|
+
],
|
|
27
|
+
"publishConfig": {
|
|
28
|
+
"access": "public",
|
|
29
|
+
"registry": "https://registry.npmjs.org/"
|
|
30
|
+
},
|
|
31
|
+
"exports": {
|
|
32
|
+
".": "./src/index.ts",
|
|
33
|
+
"./rpc": "./src/rpc.ts",
|
|
34
|
+
"./tui": "./src/tui.tsx",
|
|
35
|
+
"./gui-experimental": "./dist/gui-experimental.cjs"
|
|
36
|
+
},
|
|
37
|
+
"files": [
|
|
38
|
+
"src",
|
|
39
|
+
"dist"
|
|
40
|
+
],
|
|
41
|
+
"dependencies": {
|
|
42
|
+
"@opencode/plugin": "2.2.0",
|
|
43
|
+
"effect": "4.0.1"
|
|
44
|
+
},
|
|
45
|
+
"devDependencies": {
|
|
46
|
+
"@opencode/theme": "2.2.0",
|
|
47
|
+
"@opentui/core": "0.7.0",
|
|
48
|
+
"@opentui/solid": "0.7.0",
|
|
49
|
+
"@types/node": "^22.0.0",
|
|
50
|
+
"electron": "^39.0.0",
|
|
51
|
+
"solid-js": "^1.9.15",
|
|
52
|
+
"typescript": "^7.0.2",
|
|
53
|
+
"vite": "^8.3.5",
|
|
54
|
+
"vite-plugin-solid": "^2.11.0"
|
|
55
|
+
},
|
|
56
|
+
"scripts": {
|
|
57
|
+
"build": "vite build",
|
|
58
|
+
"dev": "vite build --watch",
|
|
59
|
+
"typecheck": "tsc --noEmit",
|
|
60
|
+
"check": "pnpm typecheck && pnpm build"
|
|
61
|
+
}
|
|
6
62
|
}
|
package/src/contract.ts
ADDED
package/src/gui.tsx
ADDED
|
@@ -0,0 +1,274 @@
|
|
|
1
|
+
import { createSignal, onCleanup } from "solid-js"
|
|
2
|
+
import { Command, ComposerPill, Plugin, Style, type ComposerPillView, type SetupContext } from "@opencode/plugin/gui"
|
|
3
|
+
import { Voice } from "./contract"
|
|
4
|
+
import { composerOf, CSS, injectComposerButtons, typeInto } from "./inject"
|
|
5
|
+
import { VoiceRpc } from "./rpc"
|
|
6
|
+
import { main } from "./main"
|
|
7
|
+
|
|
8
|
+
const definition = Plugin.define({
|
|
9
|
+
i18n: {
|
|
10
|
+
en: {
|
|
11
|
+
group: "Voice input",
|
|
12
|
+
"pill.idle": "🎙 Dictate",
|
|
13
|
+
"pill.idle.title": "Record a voice prompt and transcribe it with your OpenAI account",
|
|
14
|
+
"pill.starting": "🎙 Starting…",
|
|
15
|
+
"pill.recording": "Stop {{time}}",
|
|
16
|
+
"pill.recording.title": "Stop recording and transcribe",
|
|
17
|
+
"pill.transcribing": "Transcribing…",
|
|
18
|
+
"button.idle": "Dictate (⌘⌥M)",
|
|
19
|
+
"button.recording": "Stop and transcribe (⌘⌥M)",
|
|
20
|
+
"button.busy": "Transcribing…",
|
|
21
|
+
"command.toggle": "Start or stop voice dictation",
|
|
22
|
+
"command.cancel": "Cancel voice dictation",
|
|
23
|
+
"error.title": "Voice input failed",
|
|
24
|
+
"error.microphone":
|
|
25
|
+
"Microphone access was denied. Allow OpenCode in System Settings › Privacy & Security › Microphone.",
|
|
26
|
+
nospeech: "No speech was heard",
|
|
27
|
+
copied: "No prompt is open, so the transcript was copied to the clipboard",
|
|
28
|
+
},
|
|
29
|
+
},
|
|
30
|
+
})
|
|
31
|
+
|
|
32
|
+
type Ctx = SetupContext<typeof definition>
|
|
33
|
+
|
|
34
|
+
type Phase = { kind: "idle" } | { kind: "starting" } | { kind: "recording"; since: number } | { kind: "transcribing" }
|
|
35
|
+
|
|
36
|
+
const MAX_RECORDING_MS = 10 * 60_000
|
|
37
|
+
const MIME_TYPES = ["audio/webm;codecs=opus", "audio/webm", "audio/ogg;codecs=opus", "audio/mp4"]
|
|
38
|
+
|
|
39
|
+
const toBase64 = (blob: Blob) =>
|
|
40
|
+
new Promise<string>((resolve, reject) => {
|
|
41
|
+
const reader = new FileReader()
|
|
42
|
+
reader.onload = () => resolve(String(reader.result).replace(/^data:[^,]*,/, ""))
|
|
43
|
+
reader.onerror = () => reject(reader.error ?? new Error("Could not read the recording"))
|
|
44
|
+
reader.readAsDataURL(blob)
|
|
45
|
+
})
|
|
46
|
+
|
|
47
|
+
const message = (error: unknown) => {
|
|
48
|
+
// Plugin RPC errors carry the user-facing reason in data.message.
|
|
49
|
+
const data = (error as { data?: { message?: unknown } } | undefined)?.data
|
|
50
|
+
if (typeof data?.message === "string") return data.message
|
|
51
|
+
return error instanceof Error ? error.message : String(error)
|
|
52
|
+
}
|
|
53
|
+
|
|
54
|
+
class Recorder {
|
|
55
|
+
private stream?: MediaStream
|
|
56
|
+
private recorder?: MediaRecorder
|
|
57
|
+
private chunks: Blob[] = []
|
|
58
|
+
|
|
59
|
+
async start() {
|
|
60
|
+
this.stream = await navigator.mediaDevices.getUserMedia({
|
|
61
|
+
audio: { echoCancellation: true, noiseSuppression: true, autoGainControl: true },
|
|
62
|
+
})
|
|
63
|
+
const mimeType = MIME_TYPES.find((type) => MediaRecorder.isTypeSupported(type))
|
|
64
|
+
this.chunks = []
|
|
65
|
+
this.recorder = new MediaRecorder(this.stream, mimeType ? { mimeType } : undefined)
|
|
66
|
+
this.recorder.ondataavailable = (event) => {
|
|
67
|
+
if (event.data.size > 0) this.chunks.push(event.data)
|
|
68
|
+
}
|
|
69
|
+
this.recorder.start(250)
|
|
70
|
+
}
|
|
71
|
+
|
|
72
|
+
stop() {
|
|
73
|
+
const recorder = this.recorder
|
|
74
|
+
if (!recorder) return Promise.resolve(undefined)
|
|
75
|
+
return new Promise<Blob>((resolve) => {
|
|
76
|
+
recorder.onstop = () => {
|
|
77
|
+
const blob = new Blob(this.chunks, { type: recorder.mimeType || "audio/webm" })
|
|
78
|
+
this.release()
|
|
79
|
+
resolve(blob)
|
|
80
|
+
}
|
|
81
|
+
recorder.stop()
|
|
82
|
+
})
|
|
83
|
+
}
|
|
84
|
+
|
|
85
|
+
release() {
|
|
86
|
+
if (this.recorder?.state === "recording") this.recorder.stop()
|
|
87
|
+
this.stream?.getTracks().forEach((track) => track.stop())
|
|
88
|
+
this.stream = undefined
|
|
89
|
+
this.recorder = undefined
|
|
90
|
+
this.chunks = []
|
|
91
|
+
}
|
|
92
|
+
}
|
|
93
|
+
|
|
94
|
+
const setup = (ctx: Ctx) => {
|
|
95
|
+
const voice = ctx.main(Voice)
|
|
96
|
+
const recorder = new Recorder()
|
|
97
|
+
const [phase, setPhase] = createSignal<Phase>({ kind: "idle" })
|
|
98
|
+
const [now, setNow] = createSignal(Date.now())
|
|
99
|
+
let ticker: ReturnType<typeof setInterval> | undefined
|
|
100
|
+
const [owner, setOwner] = createSignal<HTMLElement>()
|
|
101
|
+
const placement = ctx.options.placement === "pill" ? "pill" : "composer"
|
|
102
|
+
|
|
103
|
+
const language = () => (typeof ctx.options.language === "string" ? ctx.options.language : "")
|
|
104
|
+
|
|
105
|
+
const stopTicker = () => {
|
|
106
|
+
if (ticker) clearInterval(ticker)
|
|
107
|
+
ticker = undefined
|
|
108
|
+
}
|
|
109
|
+
|
|
110
|
+
const fail = (error: unknown) => {
|
|
111
|
+
stopTicker()
|
|
112
|
+
recorder.release()
|
|
113
|
+
setPhase({ kind: "idle" })
|
|
114
|
+
setOwner(undefined)
|
|
115
|
+
ctx.toast.show({ variant: "error", title: ctx.t("error.title"), description: message(error) })
|
|
116
|
+
}
|
|
117
|
+
|
|
118
|
+
const insert = (text: string) => {
|
|
119
|
+
if (!text) {
|
|
120
|
+
ctx.toast.show({ title: ctx.t("nospeech") })
|
|
121
|
+
return
|
|
122
|
+
}
|
|
123
|
+
const composer = ctx.screen.current()?.composer
|
|
124
|
+
const target = composerOf(owner())
|
|
125
|
+
// The plugin API reaches the routed session's prompt; a new-session composer is only reachable through the DOM.
|
|
126
|
+
if (!composer && target && typeInto(target, text)) return
|
|
127
|
+
if (!composer) {
|
|
128
|
+
void ctx.system.copy(text)
|
|
129
|
+
ctx.toast.show({ title: ctx.t("copied") })
|
|
130
|
+
return
|
|
131
|
+
}
|
|
132
|
+
const current = composer.text()
|
|
133
|
+
composer.insert(current && !/\s$/.test(current) ? ` ${text}` : text)
|
|
134
|
+
composer.focus()
|
|
135
|
+
}
|
|
136
|
+
|
|
137
|
+
const start = async (host?: HTMLElement) => {
|
|
138
|
+
if (phase().kind !== "idle") return
|
|
139
|
+
setOwner(host)
|
|
140
|
+
setPhase({ kind: "starting" })
|
|
141
|
+
try {
|
|
142
|
+
if (!(await voice.requestMicrophone())) throw new Error(ctx.t("error.microphone"))
|
|
143
|
+
await recorder.start()
|
|
144
|
+
const since = Date.now()
|
|
145
|
+
setNow(since)
|
|
146
|
+
setPhase({ kind: "recording", since })
|
|
147
|
+
ticker = setInterval(() => {
|
|
148
|
+
setNow(Date.now())
|
|
149
|
+
if (Date.now() - since > MAX_RECORDING_MS) void stop()
|
|
150
|
+
}, 500)
|
|
151
|
+
} catch (error) {
|
|
152
|
+
fail(error)
|
|
153
|
+
}
|
|
154
|
+
}
|
|
155
|
+
|
|
156
|
+
const stop = async () => {
|
|
157
|
+
if (phase().kind !== "recording") return
|
|
158
|
+
stopTicker()
|
|
159
|
+
setPhase({ kind: "transcribing" })
|
|
160
|
+
try {
|
|
161
|
+
const blob = await recorder.stop()
|
|
162
|
+
if (!blob || blob.size === 0) throw new Error(ctx.t("nospeech"))
|
|
163
|
+
const audio = await toBase64(blob)
|
|
164
|
+
const result = await ctx.client.rpc(VoiceRpc).transcribe({ audio, mimeType: blob.type, language: language() })
|
|
165
|
+
const text = (result as { text: string }).text
|
|
166
|
+
setPhase({ kind: "idle" })
|
|
167
|
+
insert(text)
|
|
168
|
+
setOwner(undefined)
|
|
169
|
+
} catch (error) {
|
|
170
|
+
fail(error)
|
|
171
|
+
}
|
|
172
|
+
}
|
|
173
|
+
|
|
174
|
+
const cancel = () => {
|
|
175
|
+
if (phase().kind !== "recording" && phase().kind !== "starting") return false
|
|
176
|
+
stopTicker()
|
|
177
|
+
recorder.release()
|
|
178
|
+
setPhase({ kind: "idle" })
|
|
179
|
+
setOwner(undefined)
|
|
180
|
+
}
|
|
181
|
+
|
|
182
|
+
const toggle = (host?: HTMLElement) => {
|
|
183
|
+
const kind = phase().kind
|
|
184
|
+
if (kind === "idle") void start(host ?? hostOfFocusedComposer())
|
|
185
|
+
else if (kind === "recording") void stop()
|
|
186
|
+
}
|
|
187
|
+
|
|
188
|
+
const elapsed = (since: number) => {
|
|
189
|
+
const seconds = Math.max(0, Math.floor((now() - since) / 1000))
|
|
190
|
+
return `${Math.floor(seconds / 60)}:${String(seconds % 60).padStart(2, "0")}`
|
|
191
|
+
}
|
|
192
|
+
|
|
193
|
+
const buttons =
|
|
194
|
+
placement === "composer"
|
|
195
|
+
? injectComposerButtons({
|
|
196
|
+
phase: () => phase().kind,
|
|
197
|
+
owner,
|
|
198
|
+
elapsed: () => {
|
|
199
|
+
const current = phase()
|
|
200
|
+
return current.kind === "recording" ? elapsed(current.since) : ""
|
|
201
|
+
},
|
|
202
|
+
title: () => {
|
|
203
|
+
const kind = phase().kind
|
|
204
|
+
if (kind === "recording") return ctx.t("button.recording")
|
|
205
|
+
if (kind === "idle") return ctx.t("button.idle")
|
|
206
|
+
return ctx.t("button.busy")
|
|
207
|
+
},
|
|
208
|
+
onClick: (host) => toggle(host),
|
|
209
|
+
})
|
|
210
|
+
: undefined
|
|
211
|
+
if (buttons) ctx.add(Style, CSS)
|
|
212
|
+
|
|
213
|
+
function hostOfFocusedComposer() {
|
|
214
|
+
return composerOf(undefined)?.querySelector<HTMLElement>("[data-voice-input]") ?? undefined
|
|
215
|
+
}
|
|
216
|
+
|
|
217
|
+
ctx.add(ComposerPill, {
|
|
218
|
+
id: "mic",
|
|
219
|
+
// The pill stands in when the mic can't be placed next to the send button, e.g. after an app update moves it.
|
|
220
|
+
show: (): ComposerPillView | undefined => {
|
|
221
|
+
if (buttons && buttons.count() > 0) return undefined
|
|
222
|
+
const current = phase()
|
|
223
|
+
switch (current.kind) {
|
|
224
|
+
case "idle":
|
|
225
|
+
return {
|
|
226
|
+
label: ctx.t("pill.idle"),
|
|
227
|
+
title: ctx.t("pill.idle.title"),
|
|
228
|
+
run: () => toggle(),
|
|
229
|
+
}
|
|
230
|
+
case "starting":
|
|
231
|
+
return { label: ctx.t("pill.starting") }
|
|
232
|
+
case "recording":
|
|
233
|
+
return {
|
|
234
|
+
label: ctx.t("pill.recording", { time: elapsed(current.since) }),
|
|
235
|
+
title: ctx.t("pill.recording.title"),
|
|
236
|
+
icon: "stop",
|
|
237
|
+
tone: "danger",
|
|
238
|
+
run: () => toggle(),
|
|
239
|
+
}
|
|
240
|
+
case "transcribing":
|
|
241
|
+
return { label: ctx.t("pill.transcribing"), icon: "outline-wave-audio", tone: "warning" }
|
|
242
|
+
}
|
|
243
|
+
},
|
|
244
|
+
})
|
|
245
|
+
|
|
246
|
+
ctx.add(Command, (): Command => ({
|
|
247
|
+
id: "toggle",
|
|
248
|
+
title: ctx.t("command.toggle"),
|
|
249
|
+
group: ctx.t("group"),
|
|
250
|
+
keywords: "voice mic microphone dictate speech record transcribe",
|
|
251
|
+
section: "prompt",
|
|
252
|
+
bind: "mod+alt+m",
|
|
253
|
+
editable: true,
|
|
254
|
+
slash: { name: "dictate" },
|
|
255
|
+
run: () => toggle(),
|
|
256
|
+
}))
|
|
257
|
+
|
|
258
|
+
ctx.add(Command, (): Command => ({
|
|
259
|
+
id: "cancel",
|
|
260
|
+
title: ctx.t("command.cancel"),
|
|
261
|
+
group: ctx.t("group"),
|
|
262
|
+
section: "prompt",
|
|
263
|
+
enabled: phase().kind === "recording" || phase().kind === "starting",
|
|
264
|
+
run: () => cancel(),
|
|
265
|
+
}))
|
|
266
|
+
|
|
267
|
+
onCleanup(() => {
|
|
268
|
+
buttons?.dispose()
|
|
269
|
+
stopTicker()
|
|
270
|
+
recorder.release()
|
|
271
|
+
})
|
|
272
|
+
}
|
|
273
|
+
|
|
274
|
+
export default Plugin.define({ ...definition, setup, main })
|
package/src/index.ts
ADDED
|
@@ -0,0 +1,115 @@
|
|
|
1
|
+
import { readFileSync } from "node:fs"
|
|
2
|
+
import { Plugin } from "@opencode/plugin"
|
|
3
|
+
import { VoiceRpc, type TranscribeInput, type TranscribeOutput } from "./rpc"
|
|
4
|
+
|
|
5
|
+
// The ChatGPT backend's one-shot dictation endpoint that Codex Desktop uses (as codex-asr does). It takes the same
|
|
6
|
+
// ChatGPT access token as Codex logins: OpenCode's "Codex browser/device code" OpenAI sign-ins.
|
|
7
|
+
const CHATGPT_TRANSCRIBE = "https://chatgpt.com/backend-api/transcribe"
|
|
8
|
+
const CHATGPT_METHODS = new Set(["chatgpt-browser", "chatgpt-headless"])
|
|
9
|
+
// API keys and "Sign in with ChatGPT" (token sharing) tokens are used against the public API.
|
|
10
|
+
const OPENAI_TRANSCRIBE = "https://api.openai.com/v1/audio/transcriptions"
|
|
11
|
+
const DEFAULT_API_MODEL = "gpt-4o-mini-transcribe"
|
|
12
|
+
const CODEX_DESKTOP_VERSION = "26.429.30905"
|
|
13
|
+
|
|
14
|
+
const codexUserAgent = () => {
|
|
15
|
+
let version = CODEX_DESKTOP_VERSION
|
|
16
|
+
try {
|
|
17
|
+
const plist = readFileSync("/Applications/Codex.app/Contents/Info.plist", "utf8")
|
|
18
|
+
version = plist.match(/<key>CFBundleShortVersionString<\/key>\s*<string>([^<]+)<\/string>/)?.[1] ?? version
|
|
19
|
+
} catch {}
|
|
20
|
+
const os = process.platform === "darwin" ? "macos" : process.platform === "win32" ? "windows" : process.platform
|
|
21
|
+
const arch = process.arch === "arm64" ? "aarch64" : process.arch === "x64" ? "x86_64" : process.arch
|
|
22
|
+
return `Codex Desktop/${version} (${os}; ${arch})`
|
|
23
|
+
}
|
|
24
|
+
|
|
25
|
+
const accountFromToken = (token: string) => {
|
|
26
|
+
try {
|
|
27
|
+
const payload = JSON.parse(Buffer.from(token.split(".")[1] ?? "", "base64url").toString())
|
|
28
|
+
const id = payload?.["https://api.openai.com/auth"]?.chatgpt_account_id ?? payload?.chatgpt_account_id
|
|
29
|
+
return typeof id === "string" && id ? id : undefined
|
|
30
|
+
} catch {
|
|
31
|
+
return undefined
|
|
32
|
+
}
|
|
33
|
+
}
|
|
34
|
+
|
|
35
|
+
const extension = (mimeType: string) => {
|
|
36
|
+
if (mimeType.includes("ogg")) return "ogg"
|
|
37
|
+
if (mimeType.includes("mp4") || mimeType.includes("m4a")) return "m4a"
|
|
38
|
+
if (mimeType.includes("wav")) return "wav"
|
|
39
|
+
if (mimeType.includes("mpeg")) return "mp3"
|
|
40
|
+
return "webm"
|
|
41
|
+
}
|
|
42
|
+
|
|
43
|
+
const clip = (body: string) => body.split(/\s+/).join(" ").slice(0, 300)
|
|
44
|
+
|
|
45
|
+
class UpstreamError extends Error {}
|
|
46
|
+
|
|
47
|
+
const upload = async (url: string, headers: Record<string, string>, form: FormData, signal: AbortSignal) => {
|
|
48
|
+
const response = await fetch(url, { method: "POST", headers, body: form, signal })
|
|
49
|
+
const body = await response.text()
|
|
50
|
+
if (!response.ok) throw new UpstreamError(`Transcription failed with HTTP ${response.status}: ${clip(body)}`)
|
|
51
|
+
const parsed = JSON.parse(body) as { text?: unknown }
|
|
52
|
+
if (typeof parsed.text !== "string") throw new UpstreamError("Transcription response did not contain text")
|
|
53
|
+
return parsed.text.trim()
|
|
54
|
+
}
|
|
55
|
+
|
|
56
|
+
const audioForm = (input: TranscribeInput) => {
|
|
57
|
+
const type = input.mimeType.split(";")[0] || "audio/webm"
|
|
58
|
+
const form = new FormData()
|
|
59
|
+
form.append("file", new Blob([Buffer.from(input.audio, "base64")], { type }), `voice.${extension(type)}`)
|
|
60
|
+
if (input.language) form.append("language", input.language)
|
|
61
|
+
return form
|
|
62
|
+
}
|
|
63
|
+
|
|
64
|
+
export default Plugin.define({
|
|
65
|
+
id: "voice-input",
|
|
66
|
+
async setup(ctx) {
|
|
67
|
+
const apiModel = typeof ctx.options.model === "string" && ctx.options.model ? ctx.options.model : DEFAULT_API_MODEL
|
|
68
|
+
|
|
69
|
+
const registration = await ctx.rpc.register(VoiceRpc, {
|
|
70
|
+
transcribe: async (raw, call) => {
|
|
71
|
+
const input = raw as TranscribeInput
|
|
72
|
+
const connection = await ctx.integration.connection.active("openai")
|
|
73
|
+
if (!connection) {
|
|
74
|
+
return call.error("no_account", "No OpenAI account is connected", {
|
|
75
|
+
message: "Connect OpenAI with /connect (or `opencode auth login openai`), then try again.",
|
|
76
|
+
})
|
|
77
|
+
}
|
|
78
|
+
const account = connection.type === "credential" ? connection.label : connection.name
|
|
79
|
+
// resolve() refreshes an OAuth token that expires within 5 minutes and saves the rotated refresh token.
|
|
80
|
+
const credential = await ctx.integration.connection.resolve(connection)
|
|
81
|
+
|
|
82
|
+
try {
|
|
83
|
+
let text: string
|
|
84
|
+
if (credential?.type === "oauth" && CHATGPT_METHODS.has(credential.methodID)) {
|
|
85
|
+
const accountID = (credential.metadata?.accountID as string | undefined) ?? accountFromToken(credential.access)
|
|
86
|
+
const headers: Record<string, string> = {
|
|
87
|
+
Authorization: `Bearer ${credential.access}`,
|
|
88
|
+
originator: "Codex Desktop",
|
|
89
|
+
"User-Agent": codexUserAgent(),
|
|
90
|
+
}
|
|
91
|
+
if (accountID) headers["ChatGPT-Account-Id"] = accountID
|
|
92
|
+
text = await upload(CHATGPT_TRANSCRIBE, headers, audioForm(input), call.signal)
|
|
93
|
+
} else if (credential?.type === "key" || credential?.type === "oauth") {
|
|
94
|
+
const token = credential.type === "key" ? credential.key : credential.access
|
|
95
|
+
const form = audioForm(input)
|
|
96
|
+
form.append("model", apiModel)
|
|
97
|
+
form.append("response_format", "json")
|
|
98
|
+
text = await upload(OPENAI_TRANSCRIBE, { Authorization: `Bearer ${token}` }, form, call.signal)
|
|
99
|
+
} else {
|
|
100
|
+
return call.error("unsupported_account", "This OpenAI account can't transcribe", {
|
|
101
|
+
message: `The active OpenAI account "${account}" has no usable token. Switch accounts with /connect.`,
|
|
102
|
+
})
|
|
103
|
+
}
|
|
104
|
+
return { text, account } satisfies TranscribeOutput
|
|
105
|
+
} catch (error) {
|
|
106
|
+
if (call.signal.aborted) throw error
|
|
107
|
+
const message = error instanceof Error ? error.message : String(error)
|
|
108
|
+
return call.error("upstream", message, { message: `${message} (account "${account}")` })
|
|
109
|
+
}
|
|
110
|
+
},
|
|
111
|
+
})
|
|
112
|
+
|
|
113
|
+
return () => registration.dispose()
|
|
114
|
+
},
|
|
115
|
+
})
|
package/src/inject.tsx
ADDED
|
@@ -0,0 +1,167 @@
|
|
|
1
|
+
import { createEffect, createSignal, Match, Switch, type Accessor } from "solid-js"
|
|
2
|
+
import { render } from "solid-js/web"
|
|
3
|
+
|
|
4
|
+
// Internal DOM of OpenCode Desktop's composer (app/src/composer/editor/editor.tsx), not plugin API:
|
|
5
|
+
// <form data-component="composer"> … <div class="flex h-11 …"> ← static row
|
|
6
|
+
// <div>+ menu</div> <div data-slot="composer-controls"/> <div data-slot="composer-actions"/> ← send button
|
|
7
|
+
// The mic goes into the static row right before composer-actions, so Solid never reconciles over it.
|
|
8
|
+
const ACTIONS = '[data-component="composer"] [data-slot="composer-actions"]'
|
|
9
|
+
const HOST = "data-voice-input"
|
|
10
|
+
|
|
11
|
+
export type ButtonPhase = "idle" | "starting" | "recording" | "transcribing"
|
|
12
|
+
|
|
13
|
+
export interface ButtonModel {
|
|
14
|
+
phase: Accessor<ButtonPhase>
|
|
15
|
+
owner: Accessor<HTMLElement | undefined>
|
|
16
|
+
elapsed: Accessor<string>
|
|
17
|
+
title: Accessor<string>
|
|
18
|
+
onClick(host: HTMLElement): void
|
|
19
|
+
}
|
|
20
|
+
|
|
21
|
+
export const CSS = `
|
|
22
|
+
[${HOST}] { display: flex; align-items: center; gap: 4px; flex-shrink: 0; margin-inline-end: 6px; }
|
|
23
|
+
[${HOST}] > button[data-component="icon-button-v2"] { width: 28px; height: 28px; border-radius: 6px; }
|
|
24
|
+
[${HOST}] > button svg { width: 16px; height: 16px; display: block; }
|
|
25
|
+
[${HOST}][data-phase="recording"] > button { color: var(--v2-state-fg-danger); background-color: color-mix(in srgb, var(--v2-state-fg-danger) 14%, transparent); }
|
|
26
|
+
[${HOST}][data-phase="recording"] > button:hover { background-color: color-mix(in srgb, var(--v2-state-fg-danger) 22%, transparent); }
|
|
27
|
+
[${HOST}][data-phase="transcribing"] > button svg, [${HOST}][data-phase="starting"] > button svg { animation: voice-input-spin .9s linear infinite; }
|
|
28
|
+
[${HOST}] [data-slot="voice-input-time"] { font-size: 11px; font-weight: 530; font-variant-numeric: tabular-nums; color: var(--v2-state-fg-danger); }
|
|
29
|
+
[${HOST}][data-phase="recording"] [data-slot="voice-input-dot"] { width: 6px; height: 6px; border-radius: 9999px; background: var(--v2-state-fg-danger); animation: voice-input-pulse 1.2s ease-in-out infinite; }
|
|
30
|
+
@keyframes voice-input-spin { to { transform: rotate(360deg); } }
|
|
31
|
+
@keyframes voice-input-pulse { 50% { opacity: .25; } }
|
|
32
|
+
:root[data-motion="reduce"] [${HOST}] * { animation: none !important; }
|
|
33
|
+
`
|
|
34
|
+
|
|
35
|
+
const MicIcon = () => (
|
|
36
|
+
<svg viewBox="0 0 16 16" fill="none" aria-hidden="true">
|
|
37
|
+
<rect x="5.5" y="1.5" width="5" height="8.5" rx="2.5" stroke="currentColor" />
|
|
38
|
+
<path d="M3 7.75a5 5 0 0 0 10 0M8 12.75v1.75M5.75 14.5h4.5" stroke="currentColor" stroke-linecap="round" />
|
|
39
|
+
</svg>
|
|
40
|
+
)
|
|
41
|
+
|
|
42
|
+
const StopIcon = () => (
|
|
43
|
+
<svg viewBox="0 0 16 16" aria-hidden="true">
|
|
44
|
+
<rect x="4" y="4" width="8" height="8" rx="1.5" fill="currentColor" />
|
|
45
|
+
</svg>
|
|
46
|
+
)
|
|
47
|
+
|
|
48
|
+
const SpinnerIcon = () => (
|
|
49
|
+
<svg viewBox="0 0 16 16" fill="none" aria-hidden="true">
|
|
50
|
+
<circle cx="8" cy="8" r="5.5" stroke="currentColor" stroke-opacity=".25" stroke-width="1.5" />
|
|
51
|
+
<path d="M13.5 8A5.5 5.5 0 0 0 8 2.5" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" />
|
|
52
|
+
</svg>
|
|
53
|
+
)
|
|
54
|
+
|
|
55
|
+
function MicButton(props: { host: HTMLElement; model: ButtonModel }) {
|
|
56
|
+
// Only the composer that started a recording shows its progress; the others stay a plain, disabled mic.
|
|
57
|
+
const phase = () => {
|
|
58
|
+
const current = props.model.phase()
|
|
59
|
+
const owner = props.model.owner()
|
|
60
|
+
return current === "idle" || !owner || owner === props.host ? current : "busy"
|
|
61
|
+
}
|
|
62
|
+
|
|
63
|
+
createEffect(() => {
|
|
64
|
+
props.host.dataset.phase = phase() === "busy" ? "idle" : phase()
|
|
65
|
+
})
|
|
66
|
+
|
|
67
|
+
return (
|
|
68
|
+
<>
|
|
69
|
+
<Switch>
|
|
70
|
+
<Match when={phase() === "recording"}>
|
|
71
|
+
<span data-slot="voice-input-dot" />
|
|
72
|
+
<span data-slot="voice-input-time">{props.model.elapsed()}</span>
|
|
73
|
+
</Match>
|
|
74
|
+
</Switch>
|
|
75
|
+
<button
|
|
76
|
+
type="button"
|
|
77
|
+
data-component="icon-button-v2"
|
|
78
|
+
data-size="normal"
|
|
79
|
+
data-variant="ghost-muted"
|
|
80
|
+
title={props.model.title()}
|
|
81
|
+
aria-label={props.model.title()}
|
|
82
|
+
aria-pressed={phase() === "recording"}
|
|
83
|
+
disabled={phase() === "busy" || phase() === "transcribing" || phase() === "starting"}
|
|
84
|
+
// Keep focus and the caret in the prompt, as the composer's own controls do.
|
|
85
|
+
onMouseDown={(event) => event.preventDefault()}
|
|
86
|
+
onClick={(event) => {
|
|
87
|
+
event.preventDefault()
|
|
88
|
+
event.stopPropagation()
|
|
89
|
+
props.model.onClick(props.host)
|
|
90
|
+
}}
|
|
91
|
+
>
|
|
92
|
+
<Switch fallback={<MicIcon />}>
|
|
93
|
+
<Match when={phase() === "recording"}>
|
|
94
|
+
<StopIcon />
|
|
95
|
+
</Match>
|
|
96
|
+
<Match when={phase() === "transcribing" || phase() === "starting"}>
|
|
97
|
+
<SpinnerIcon />
|
|
98
|
+
</Match>
|
|
99
|
+
</Switch>
|
|
100
|
+
</button>
|
|
101
|
+
</>
|
|
102
|
+
)
|
|
103
|
+
}
|
|
104
|
+
|
|
105
|
+
/** Keeps a mic button before the send button of every composer in the window. Returns how many are mounted. */
|
|
106
|
+
export function injectComposerButtons(model: ButtonModel) {
|
|
107
|
+
const mounted = new Map<HTMLElement, () => void>()
|
|
108
|
+
const [count, setCount] = createSignal(0)
|
|
109
|
+
let frame = 0
|
|
110
|
+
|
|
111
|
+
const scan = () => {
|
|
112
|
+
frame = 0
|
|
113
|
+
for (const actions of document.querySelectorAll<HTMLElement>(ACTIONS)) {
|
|
114
|
+
const row = actions.parentElement
|
|
115
|
+
if (!row || row.querySelector(`:scope > [${HOST}]`)) continue
|
|
116
|
+
const host = document.createElement("div")
|
|
117
|
+
host.setAttribute(HOST, "")
|
|
118
|
+
row.insertBefore(host, actions)
|
|
119
|
+
mounted.set(
|
|
120
|
+
host,
|
|
121
|
+
render(() => <MicButton host={host} model={model} />, host),
|
|
122
|
+
)
|
|
123
|
+
}
|
|
124
|
+
for (const [host, dispose] of mounted) {
|
|
125
|
+
if (host.isConnected) continue
|
|
126
|
+
dispose()
|
|
127
|
+
mounted.delete(host)
|
|
128
|
+
}
|
|
129
|
+
setCount(mounted.size)
|
|
130
|
+
}
|
|
131
|
+
|
|
132
|
+
const observer = new MutationObserver(() => {
|
|
133
|
+
if (!frame) frame = requestAnimationFrame(scan)
|
|
134
|
+
})
|
|
135
|
+
observer.observe(document.body, { childList: true, subtree: true })
|
|
136
|
+
scan()
|
|
137
|
+
|
|
138
|
+
return {
|
|
139
|
+
count,
|
|
140
|
+
dispose() {
|
|
141
|
+
observer.disconnect()
|
|
142
|
+
if (frame) cancelAnimationFrame(frame)
|
|
143
|
+
for (const [host, dispose] of mounted) {
|
|
144
|
+
dispose()
|
|
145
|
+
host.remove()
|
|
146
|
+
}
|
|
147
|
+
mounted.clear()
|
|
148
|
+
},
|
|
149
|
+
}
|
|
150
|
+
}
|
|
151
|
+
|
|
152
|
+
/** The composer element a mic host, or failing that the focused or first visible composer, belongs to. */
|
|
153
|
+
export function composerOf(host: HTMLElement | undefined) {
|
|
154
|
+
const from = host?.closest<HTMLElement>('[data-component="composer"]')
|
|
155
|
+
if (from?.isConnected) return from
|
|
156
|
+
const focused = document.activeElement?.closest<HTMLElement>('[data-component="composer"]')
|
|
157
|
+
if (focused) return focused
|
|
158
|
+
return [...document.querySelectorAll<HTMLElement>('[data-component="composer"]')].find((el) => el.offsetParent)
|
|
159
|
+
}
|
|
160
|
+
|
|
161
|
+
/** Types text into a composer's editor through the browser's own input path, for composers the plugin API can't reach. */
|
|
162
|
+
export function typeInto(composer: HTMLElement, text: string) {
|
|
163
|
+
const editor = composer.querySelector<HTMLElement>('[data-component="composer-editor"]')
|
|
164
|
+
if (!editor) return false
|
|
165
|
+
editor.focus()
|
|
166
|
+
return document.execCommand("insertText", false, text)
|
|
167
|
+
}
|
package/src/main.ts
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
import { systemPreferences } from "electron"
|
|
2
|
+
import type { Plugin } from "@opencode/plugin/gui"
|
|
3
|
+
import { Voice } from "./contract"
|
|
4
|
+
|
|
5
|
+
export const main: Plugin.MainSetup = (ctx) => {
|
|
6
|
+
ctx.provide(Voice, {
|
|
7
|
+
requestMicrophone: async () => {
|
|
8
|
+
if (process.platform !== "darwin") return true
|
|
9
|
+
if (systemPreferences.getMediaAccessStatus("microphone") === "granted") return true
|
|
10
|
+
return systemPreferences.askForMediaAccess("microphone")
|
|
11
|
+
},
|
|
12
|
+
})
|
|
13
|
+
}
|
package/src/rpc.ts
ADDED
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
import { Rpc } from "@opencode/plugin/rpc"
|
|
2
|
+
|
|
3
|
+
const reason = {
|
|
4
|
+
type: "object",
|
|
5
|
+
properties: { message: { type: "string" } },
|
|
6
|
+
required: ["message"],
|
|
7
|
+
additionalProperties: false,
|
|
8
|
+
} as const
|
|
9
|
+
|
|
10
|
+
export const VoiceRpc = Rpc.define({
|
|
11
|
+
id: "voice-input",
|
|
12
|
+
methods: {
|
|
13
|
+
transcribe: {
|
|
14
|
+
input: {
|
|
15
|
+
type: "object",
|
|
16
|
+
properties: {
|
|
17
|
+
audio: { type: "string", description: "Base64-encoded recording" },
|
|
18
|
+
mimeType: { type: "string" },
|
|
19
|
+
language: { type: "string" },
|
|
20
|
+
},
|
|
21
|
+
required: ["audio", "mimeType"],
|
|
22
|
+
additionalProperties: false,
|
|
23
|
+
},
|
|
24
|
+
output: {
|
|
25
|
+
type: "object",
|
|
26
|
+
properties: {
|
|
27
|
+
text: { type: "string" },
|
|
28
|
+
account: { type: "string" },
|
|
29
|
+
},
|
|
30
|
+
required: ["text", "account"],
|
|
31
|
+
additionalProperties: false,
|
|
32
|
+
},
|
|
33
|
+
errors: {
|
|
34
|
+
no_account: reason,
|
|
35
|
+
unsupported_account: reason,
|
|
36
|
+
upstream: reason,
|
|
37
|
+
},
|
|
38
|
+
},
|
|
39
|
+
},
|
|
40
|
+
events: {},
|
|
41
|
+
})
|
|
42
|
+
|
|
43
|
+
export interface TranscribeInput {
|
|
44
|
+
audio: string
|
|
45
|
+
mimeType: string
|
|
46
|
+
language?: string
|
|
47
|
+
}
|
|
48
|
+
|
|
49
|
+
export interface TranscribeOutput {
|
|
50
|
+
text: string
|
|
51
|
+
account: string
|
|
52
|
+
}
|
package/src/tui-audio.ts
ADDED
|
@@ -0,0 +1,122 @@
|
|
|
1
|
+
import { execFile, spawn, type ChildProcess } from "node:child_process"
|
|
2
|
+
import { randomUUID } from "node:crypto"
|
|
3
|
+
import { existsSync } from "node:fs"
|
|
4
|
+
import { readFile, rm } from "node:fs/promises"
|
|
5
|
+
import { homedir, tmpdir } from "node:os"
|
|
6
|
+
import { delimiter, join } from "node:path"
|
|
7
|
+
|
|
8
|
+
const findBinary = (name: string, override?: unknown) => {
|
|
9
|
+
if (typeof override === "string" && override.trim()) return override.trim().replace(/^~(?=\/)/, homedir())
|
|
10
|
+
const dirs = [...(process.env.PATH ?? "").split(delimiter), "/opt/homebrew/bin", "/usr/local/bin", "/usr/bin"]
|
|
11
|
+
return dirs.filter(Boolean).map((dir) => join(dir, name)).find((path) => existsSync(path))
|
|
12
|
+
}
|
|
13
|
+
|
|
14
|
+
const run = (command: string, args: string[], input?: string) =>
|
|
15
|
+
new Promise<string>((resolve, reject) => {
|
|
16
|
+
const child = execFile(command, args, { maxBuffer: 16 * 1024 * 1024 }, (error, stdout) =>
|
|
17
|
+
error ? reject(error) : resolve(String(stdout)),
|
|
18
|
+
)
|
|
19
|
+
if (input !== undefined) child.stdin?.end(input)
|
|
20
|
+
})
|
|
21
|
+
|
|
22
|
+
// ffmpeg's microphone input for this OS: the system default device unless `device` names another.
|
|
23
|
+
const inputArgs = (device: string | undefined) => {
|
|
24
|
+
// For testing without a microphone: plays a file in real time as if it were spoken.
|
|
25
|
+
const testFile = process.env.OPENCODE_VOICE_INPUT_TEST_FILE
|
|
26
|
+
if (testFile) return ["-re", "-i", testFile]
|
|
27
|
+
if (process.platform === "darwin") return ["-f", "avfoundation", "-i", `:${device || "default"}`]
|
|
28
|
+
if (process.platform === "linux") return ["-f", "pulse", "-i", device || "default"]
|
|
29
|
+
if (process.platform === "win32" && device) return ["-f", "dshow", "-i", `audio=${device}`]
|
|
30
|
+
throw new Error("Set the plugin's `microphone` option to your DirectShow microphone name to record on Windows.")
|
|
31
|
+
}
|
|
32
|
+
|
|
33
|
+
export class FfmpegRecorder {
|
|
34
|
+
private child?: ChildProcess
|
|
35
|
+
private file?: string
|
|
36
|
+
private stderr = ""
|
|
37
|
+
private exited?: Promise<number | null>
|
|
38
|
+
|
|
39
|
+
constructor(private readonly options: { ffmpegPath?: unknown; microphone?: unknown }) {}
|
|
40
|
+
|
|
41
|
+
async start() {
|
|
42
|
+
const ffmpeg = findBinary("ffmpeg", this.options.ffmpegPath)
|
|
43
|
+
if (!ffmpeg) throw new Error("ffmpeg was not found. Install it (`brew install ffmpeg`) or set the `ffmpegPath` option.")
|
|
44
|
+
const device = typeof this.options.microphone === "string" ? this.options.microphone : undefined
|
|
45
|
+
this.file = join(tmpdir(), `opencode-voice-${randomUUID()}.webm`)
|
|
46
|
+
this.stderr = ""
|
|
47
|
+
const child = spawn(
|
|
48
|
+
ffmpeg,
|
|
49
|
+
["-hide_banner", "-loglevel", "error", ...inputArgs(device), "-ac", "1", "-ar", "16000", "-c:a", "libopus",
|
|
50
|
+
"-b:a", "32k", "-y", this.file],
|
|
51
|
+
{ stdio: ["pipe", "ignore", "pipe"] },
|
|
52
|
+
)
|
|
53
|
+
this.child = child
|
|
54
|
+
child.stdin?.on("error", () => undefined)
|
|
55
|
+
child.stderr?.on("data", (chunk) => (this.stderr = (this.stderr + chunk).slice(-2000)))
|
|
56
|
+
this.exited = new Promise((resolve) => {
|
|
57
|
+
child.once("exit", (code) => resolve(code))
|
|
58
|
+
child.once("error", () => resolve(-1))
|
|
59
|
+
})
|
|
60
|
+
// ffmpeg fails within a moment when the device or the microphone permission is missing.
|
|
61
|
+
const early = await Promise.race([this.exited, new Promise((resolve) => setTimeout(() => resolve("running"), 400))])
|
|
62
|
+
if (early !== "running") {
|
|
63
|
+
await this.cleanup()
|
|
64
|
+
throw new Error(`Could not record: ${this.stderr.trim().split("\n").pop() || `ffmpeg exited with ${early}`}`)
|
|
65
|
+
}
|
|
66
|
+
}
|
|
67
|
+
|
|
68
|
+
/** Stops recording and returns the WebM/Opus file's contents. */
|
|
69
|
+
async stop() {
|
|
70
|
+
const child = this.child
|
|
71
|
+
const file = this.file
|
|
72
|
+
if (!child || !file) throw new Error("Not recording")
|
|
73
|
+
// "q" makes ffmpeg finish the file properly; SIGINT does the same if it doesn't answer.
|
|
74
|
+
if (child.exitCode === null) child.stdin?.end("q")
|
|
75
|
+
const timer = setTimeout(() => child.kill("SIGINT"), 3000)
|
|
76
|
+
await this.exited
|
|
77
|
+
clearTimeout(timer)
|
|
78
|
+
try {
|
|
79
|
+
return await readFile(file)
|
|
80
|
+
} finally {
|
|
81
|
+
await this.cleanup()
|
|
82
|
+
}
|
|
83
|
+
}
|
|
84
|
+
|
|
85
|
+
async cancel() {
|
|
86
|
+
this.child?.kill("SIGKILL")
|
|
87
|
+
await this.cleanup()
|
|
88
|
+
}
|
|
89
|
+
|
|
90
|
+
private async cleanup() {
|
|
91
|
+
if (this.file) await rm(this.file, { force: true })
|
|
92
|
+
this.child = undefined
|
|
93
|
+
this.file = undefined
|
|
94
|
+
}
|
|
95
|
+
}
|
|
96
|
+
|
|
97
|
+
const clipboardCommands = () => {
|
|
98
|
+
if (process.platform === "darwin") return { read: ["pbpaste"], write: ["pbcopy"] }
|
|
99
|
+
if (process.platform === "win32")
|
|
100
|
+
return { read: ["powershell.exe", "-NoProfile", "-Command", "Get-Clipboard -Raw"], write: ["clip.exe"] }
|
|
101
|
+
if (process.env.WAYLAND_DISPLAY && findBinary("wl-copy"))
|
|
102
|
+
return { read: ["wl-paste", "--no-newline"], write: ["wl-copy"] }
|
|
103
|
+
if (findBinary("xclip"))
|
|
104
|
+
return { read: ["xclip", "-selection", "clipboard", "-o"], write: ["xclip", "-selection", "clipboard"] }
|
|
105
|
+
return undefined
|
|
106
|
+
}
|
|
107
|
+
|
|
108
|
+
export const clipboard = {
|
|
109
|
+
available: () => clipboardCommands() !== undefined,
|
|
110
|
+
async read() {
|
|
111
|
+
const commands = clipboardCommands()
|
|
112
|
+
if (!commands) return undefined
|
|
113
|
+
const [command, ...args] = commands.read
|
|
114
|
+
return run(command!, args).catch(() => undefined)
|
|
115
|
+
},
|
|
116
|
+
async write(text: string) {
|
|
117
|
+
const commands = clipboardCommands()
|
|
118
|
+
if (!commands) throw new Error("No clipboard tool found (pbcopy, wl-copy or xclip).")
|
|
119
|
+
const [command, ...args] = commands.write
|
|
120
|
+
await run(command!, args, text)
|
|
121
|
+
},
|
|
122
|
+
}
|
package/src/tui.tsx
ADDED
|
@@ -0,0 +1,190 @@
|
|
|
1
|
+
/** @jsxImportSource @opentui/solid */
|
|
2
|
+
import { createSignal, Show } from "solid-js"
|
|
3
|
+
import { Plugin } from "@opencode/plugin/tui"
|
|
4
|
+
import { VoiceRpc } from "./rpc"
|
|
5
|
+
import { clipboard, FfmpegRecorder } from "./tui-audio"
|
|
6
|
+
|
|
7
|
+
type Phase = "idle" | "starting" | "recording" | "transcribing"
|
|
8
|
+
|
|
9
|
+
const COMMAND = "voice-input.dictate"
|
|
10
|
+
const MAX_RECORDING_MS = 10 * 60_000
|
|
11
|
+
// prompt.paste reads the clipboard asynchronously; the previous clipboard text is put back after it has.
|
|
12
|
+
const RESTORE_DELAY_MS = 1500
|
|
13
|
+
|
|
14
|
+
const message = (error: unknown) => {
|
|
15
|
+
const data = (error as { data?: { message?: unknown } } | undefined)?.data
|
|
16
|
+
if (typeof data?.message === "string") return data.message
|
|
17
|
+
return error instanceof Error ? error.message : String(error)
|
|
18
|
+
}
|
|
19
|
+
|
|
20
|
+
export default Plugin.define({
|
|
21
|
+
id: "voice-input.tui",
|
|
22
|
+
setup(context) {
|
|
23
|
+
const options = context.options
|
|
24
|
+
const language = typeof options.language === "string" ? options.language : ""
|
|
25
|
+
const restoreClipboard = options.restoreClipboard !== false
|
|
26
|
+
const recorder = new FfmpegRecorder({ ffmpegPath: options.ffmpegPath, microphone: options.microphone })
|
|
27
|
+
const [phase, setPhase] = createSignal<Phase>("idle")
|
|
28
|
+
const [since, setSince] = createSignal(0)
|
|
29
|
+
const [now, setNow] = createSignal(Date.now())
|
|
30
|
+
let ticker: ReturnType<typeof setInterval> | undefined
|
|
31
|
+
let restoreTimer: ReturnType<typeof setTimeout> | undefined
|
|
32
|
+
let last = ""
|
|
33
|
+
|
|
34
|
+
const stopTicker = () => {
|
|
35
|
+
if (ticker) clearInterval(ticker)
|
|
36
|
+
ticker = undefined
|
|
37
|
+
}
|
|
38
|
+
|
|
39
|
+
const fail = (error: unknown) => {
|
|
40
|
+
stopTicker()
|
|
41
|
+
setPhase("idle")
|
|
42
|
+
context.ui.toast.show({ variant: "error", title: "Voice input failed", message: message(error) })
|
|
43
|
+
}
|
|
44
|
+
|
|
45
|
+
// The TUI plugin API cannot write to the prompt, so the transcript goes through the clipboard and the prompt's own
|
|
46
|
+
// paste command, which inserts it at the cursor. The previous clipboard text is restored afterwards.
|
|
47
|
+
const paste = async (text: string) => {
|
|
48
|
+
const pasteReachable = context.keymap.commands().some((command) => command.id === "prompt.paste")
|
|
49
|
+
const previous = restoreClipboard && pasteReachable ? await clipboard.read() : undefined
|
|
50
|
+
await clipboard.write(text)
|
|
51
|
+
if (!pasteReachable) {
|
|
52
|
+
context.ui.toast.show({ variant: "info", message: "Transcript copied. Paste it into the prompt." })
|
|
53
|
+
return
|
|
54
|
+
}
|
|
55
|
+
context.keymap.dispatch("prompt.paste")
|
|
56
|
+
if (previous === undefined || previous === "") return
|
|
57
|
+
if (restoreTimer) clearTimeout(restoreTimer)
|
|
58
|
+
restoreTimer = setTimeout(() => void clipboard.write(previous).catch(() => undefined), RESTORE_DELAY_MS)
|
|
59
|
+
}
|
|
60
|
+
|
|
61
|
+
const start = async () => {
|
|
62
|
+
if (phase() !== "idle") return
|
|
63
|
+
setPhase("starting")
|
|
64
|
+
try {
|
|
65
|
+
await recorder.start()
|
|
66
|
+
const started = Date.now()
|
|
67
|
+
setSince(started)
|
|
68
|
+
setNow(started)
|
|
69
|
+
setPhase("recording")
|
|
70
|
+
ticker = setInterval(() => {
|
|
71
|
+
setNow(Date.now())
|
|
72
|
+
if (Date.now() - started > MAX_RECORDING_MS) void stop()
|
|
73
|
+
}, 500)
|
|
74
|
+
} catch (error) {
|
|
75
|
+
fail(error)
|
|
76
|
+
}
|
|
77
|
+
}
|
|
78
|
+
|
|
79
|
+
const stop = async () => {
|
|
80
|
+
if (phase() !== "recording") return
|
|
81
|
+
stopTicker()
|
|
82
|
+
setPhase("transcribing")
|
|
83
|
+
try {
|
|
84
|
+
const audio = await recorder.stop()
|
|
85
|
+
if (audio.byteLength === 0) throw new Error("Nothing was recorded")
|
|
86
|
+
const result = await context.client.rpc(VoiceRpc).transcribe({
|
|
87
|
+
audio: audio.toString("base64"),
|
|
88
|
+
mimeType: "audio/webm",
|
|
89
|
+
language,
|
|
90
|
+
})
|
|
91
|
+
const text = (result as { text: string }).text.trim()
|
|
92
|
+
setPhase("idle")
|
|
93
|
+
if (!text) {
|
|
94
|
+
context.ui.toast.show({ variant: "info", message: "No speech was heard" })
|
|
95
|
+
return
|
|
96
|
+
}
|
|
97
|
+
last = text
|
|
98
|
+
await paste(`${text} `)
|
|
99
|
+
} catch (error) {
|
|
100
|
+
fail(error)
|
|
101
|
+
}
|
|
102
|
+
}
|
|
103
|
+
|
|
104
|
+
const cancel = async () => {
|
|
105
|
+
if (phase() !== "recording") return false
|
|
106
|
+
stopTicker()
|
|
107
|
+
await recorder.cancel()
|
|
108
|
+
setPhase("idle")
|
|
109
|
+
}
|
|
110
|
+
|
|
111
|
+
const toggle = () => {
|
|
112
|
+
if (phase() === "idle") void start()
|
|
113
|
+
else if (phase() === "recording") void stop()
|
|
114
|
+
}
|
|
115
|
+
|
|
116
|
+
const elapsed = () => {
|
|
117
|
+
const seconds = Math.max(0, Math.floor((now() - since()) / 1000))
|
|
118
|
+
return `${Math.floor(seconds / 60)}:${String(seconds % 60).padStart(2, "0")}`
|
|
119
|
+
}
|
|
120
|
+
|
|
121
|
+
context.keymap.layer(() => ({
|
|
122
|
+
mode: "global",
|
|
123
|
+
commands: [
|
|
124
|
+
{
|
|
125
|
+
id: COMMAND,
|
|
126
|
+
title: "Start or stop voice dictation",
|
|
127
|
+
group: "Voice input",
|
|
128
|
+
bind: "<leader>v",
|
|
129
|
+
palette: true,
|
|
130
|
+
slash: { name: "dictate" },
|
|
131
|
+
run: () => toggle(),
|
|
132
|
+
},
|
|
133
|
+
{
|
|
134
|
+
id: "voice-input.copy-last",
|
|
135
|
+
title: "Copy last dictation",
|
|
136
|
+
group: "Voice input",
|
|
137
|
+
palette: true,
|
|
138
|
+
enabled: () => last !== "",
|
|
139
|
+
run: () =>
|
|
140
|
+
void clipboard
|
|
141
|
+
.write(last)
|
|
142
|
+
.then(() => context.ui.toast.show({ variant: "success", message: "Last dictation copied" })),
|
|
143
|
+
},
|
|
144
|
+
{
|
|
145
|
+
id: "voice-input.cancel",
|
|
146
|
+
title: "Cancel voice dictation",
|
|
147
|
+
group: "Voice input",
|
|
148
|
+
palette: true,
|
|
149
|
+
enabled: () => phase() === "recording",
|
|
150
|
+
run: () => void cancel(),
|
|
151
|
+
},
|
|
152
|
+
],
|
|
153
|
+
}))
|
|
154
|
+
|
|
155
|
+
// The footer shows nothing while idle; it only shows that the mic is recording or transcribing.
|
|
156
|
+
const removeSlot = context.ui.slot({
|
|
157
|
+
append: "prompt.footer.status",
|
|
158
|
+
render: () => {
|
|
159
|
+
const theme = context.theme
|
|
160
|
+
const label = () => {
|
|
161
|
+
switch (phase()) {
|
|
162
|
+
case "recording":
|
|
163
|
+
return `● ${elapsed()} stop`
|
|
164
|
+
case "transcribing":
|
|
165
|
+
return "◌ transcribing…"
|
|
166
|
+
case "starting":
|
|
167
|
+
return "◌ mic…"
|
|
168
|
+
default:
|
|
169
|
+
return ""
|
|
170
|
+
}
|
|
171
|
+
}
|
|
172
|
+
const color = () => (phase() === "recording" ? theme.text.feedback.error.base : theme.text.feedback.warning.base)
|
|
173
|
+
return (
|
|
174
|
+
<Show when={phase() !== "idle"}>
|
|
175
|
+
<text fg={color()} onMouseDown={() => toggle()}>
|
|
176
|
+
{label()}
|
|
177
|
+
</text>
|
|
178
|
+
</Show>
|
|
179
|
+
)
|
|
180
|
+
},
|
|
181
|
+
})
|
|
182
|
+
|
|
183
|
+
return () => {
|
|
184
|
+
removeSlot()
|
|
185
|
+
stopTicker()
|
|
186
|
+
if (restoreTimer) clearTimeout(restoreTimer)
|
|
187
|
+
void recorder.cancel()
|
|
188
|
+
}
|
|
189
|
+
},
|
|
190
|
+
})
|