@tychenjiajun/pi-voice 0.3.0 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +40 -9
- package/package.json +1 -1
- package/src/chinese.ts +15 -0
- package/src/dictation-controller.ts +6 -2
- package/src/file-transcription.ts +9 -8
- package/src/onboarding.ts +76 -7
- package/src/openai-asr.ts +13 -47
- package/src/remote-asr.ts +212 -0
- package/src/runtime.ts +13 -11
- package/src/settings-menu.ts +66 -206
- package/src/settings.ts +79 -20
- package/src/transcription-service.ts +15 -13
- package/src/try-it.ts +15 -6
- package/src/ui-components.ts +8 -2
package/README.md
CHANGED
|
@@ -4,7 +4,7 @@ Speech-to-text for Pi — on-device model or OpenAI-compatible remote ASR, with
|
|
|
4
4
|
|
|
5
5
|
Press a keyboard shortcut, talk, and the resulting transcript will be put directly into your chat.
|
|
6
6
|
|
|
7
|
-
> **Fork notice.** `@tychenjiajun/pi-voice` is a fork of [`@earendil-works/pi-voice`](https://github.com/earendil-works/pi-voice) that adds a remote ASR backend (any OpenAI-compatible `input_audio` chat-completions endpoint)
|
|
7
|
+
> **Fork notice.** `@tychenjiajun/pi-voice` is a fork of [`@earendil-works/pi-voice`](https://github.com/earendil-works/pi-voice) that adds a remote ASR backend (any OpenAI-compatible `input_audio` chat-completions endpoint) in an exclusive local-only or remote-only setup, plus compaction-style post-processing that refines a transcript with Pi's own model. It is published independently under a personal npm scope. See the [changelog](#changelog) for what changed.
|
|
8
8
|
|
|
9
9
|
## Install
|
|
10
10
|
|
|
@@ -27,7 +27,7 @@ The extension registers:
|
|
|
27
27
|
- a configurable terminal shortcut (`Ctrl+Alt+Z` by default) to start and stop recording;
|
|
28
28
|
- `/voice`, which does the same as the shortcut. While recording, `Space` also stops and transcribes, and `Escape` discards;
|
|
29
29
|
- a `transcribe_file` tool that the agent can use to transcribe local audio or video files;
|
|
30
|
-
- `/voice-settings` for preferred languages, model, transcription language, post-processing, microphone, and shortcut settings.
|
|
30
|
+
- `/voice-settings` for preferred languages, model, transcription backend, transcription language, post-processing, microphone, and shortcut settings.
|
|
31
31
|
|
|
32
32
|
`/transcribe` remains available as a compatibility alias for `/voice-settings`.
|
|
33
33
|
|
|
@@ -35,7 +35,7 @@ The extension registers:
|
|
|
35
35
|
|
|
36
36
|
Pi Voice can send audio to a remote ASR service instead of transcribing with the local model. It speaks the OpenAI chat-completions `input_audio` dialect: the recording is uploaded as a base64 WAV data URL and the transcript is read from `choices[0].message.content`. Any endpoint that accepts that shape works.
|
|
37
37
|
|
|
38
|
-
Set it up in `/voice-settings` → **Transcription backend** → **Remote ASR (OpenAI-compatible)**. You are asked for:
|
|
38
|
+
Set it up either during first-run setup — `/voice-settings` starts by asking **Local model** or **Remote ASR (OpenAI-compatible)** — or later in `/voice-settings` → **Transcription backend** → **Remote ASR (OpenAI-compatible)**. You are asked for:
|
|
39
39
|
|
|
40
40
|
1. **Base URL** — endpoint root, without `/chat/completions`;
|
|
41
41
|
2. **Model** — the model name your endpoint expects;
|
|
@@ -49,12 +49,13 @@ export OPENAI_API_KEY=sk-...
|
|
|
49
49
|
|
|
50
50
|
Behavior:
|
|
51
51
|
|
|
52
|
-
- **
|
|
52
|
+
- **Remote-only mode.** Remote settings store no local model: setup skips the model download step, and a failed remote request (network, quota, HTTP error, or an unset API key) surfaces as an error instead of being retried offline. Failures are logged to `~/.pi/agent/pi-voice.log`.
|
|
53
53
|
- **Batch-only.** Remote transcription happens when recording stops (the ASR request carries the whole clip), so there is no streaming preview.
|
|
54
|
-
- **16 kHz mono WAV** is uploaded as a base64 data URL.
|
|
55
|
-
- Switching
|
|
54
|
+
- **16 kHz mono WAV** is uploaded as a base64 WAV data URL.
|
|
55
|
+
- **Exclusive modes.** Switching **Transcription backend** to **Local model** asks you to pick a model first; switching to **Remote ASR** drops the local model from settings. The **Model** and **Transcription language** settings rows appear only in local mode.
|
|
56
|
+
- `transcribe_file` uses whichever mode is configured — in remote mode the file audio is sent to the endpoint instead of a local model.
|
|
56
57
|
|
|
57
|
-
Transcripts still go through the **Chinese output** script conversion preference.
|
|
58
|
+
Transcripts still go through the **Chinese output** script conversion preference; because a remote response carries no detected language, Han characters in the transcript are what trigger the conversion when the language is set to automatic. The **Chinese output** row stays reachable in remote mode, since the endpoint can always produce Chinese.
|
|
58
59
|
|
|
59
60
|
## Post-processing (compaction-style refinement)
|
|
60
61
|
|
|
@@ -111,10 +112,40 @@ export PI_VOICE_FFMPEG_PATH=/path/to/ffmpeg
|
|
|
111
112
|
|
|
112
113
|
The legacy `PI_TRANSCRIBE_FFMPEG_PATH` variable remains supported when `PI_VOICE_FFMPEG_PATH` is not set.
|
|
113
114
|
|
|
114
|
-
When FFmpeg is unavailable, `transcribe_file` reports platform-specific guidance to the agent. The agent should ask before running a package-manager command. Model setup is still explicit: run `/voice-settings` once in the interactive TUI to choose and,
|
|
115
|
+
When FFmpeg is unavailable, `transcribe_file` reports platform-specific guidance to the agent. The agent should ask before running a package-manager command. Model setup is still explicit: run `/voice-settings` once in the interactive TUI to choose a mode and, in local mode, confirm the download of a local model.
|
|
115
116
|
|
|
116
117
|
## Changelog
|
|
117
118
|
|
|
119
|
+
### 0.5.0
|
|
120
|
+
|
|
121
|
+
**Changed — Exclusive local / remote modes (no fallback)**
|
|
122
|
+
|
|
123
|
+
- Local and remote transcription are now mutually exclusive: remote settings store no local model, and selecting a local model drops any remote configuration. Settings files with both are normalized to remote-only on read.
|
|
124
|
+
- First-run setup in remote mode skips the model step entirely; no local model is downloaded (onboarding jumps from step 2 to step 4, the Try it pane).
|
|
125
|
+
- Removed the local fallback for remote ASR. Network, quota, HTTP, and unset-API-key failures now surface directly and are logged to `~/.pi/agent/pi-voice.log`.
|
|
126
|
+
- The settings menu hides the **Model** and **Transcription language** rows in remote mode and always shows **Chinese output** there; switching the backend to local asks you to pick a model first, and switching to remote clears the local model.
|
|
127
|
+
- Try it shows a **Local model** row in remote mode; choosing a model from it switches setup to local mode.
|
|
128
|
+
- `transcribe_file` works in either mode and reports the remote model name in its details when remote.
|
|
129
|
+
|
|
130
|
+
### 0.4.0
|
|
131
|
+
|
|
132
|
+
**Added — Backend choice in first-run setup**
|
|
133
|
+
|
|
134
|
+
- `/voice-settings` (and the first shortcut press) now starts by asking **Local model** or **Remote ASR (OpenAI-compatible)**, so the remote backend no longer requires finishing local setup first. A local model is still downloaded as the fallback.
|
|
135
|
+
- The remote endpoint prompts (base URL, model, API-key env var name) are shared by first-run setup and the settings menu.
|
|
136
|
+
|
|
137
|
+
**Fixed — Settings reachability and remote consistency**
|
|
138
|
+
|
|
139
|
+
- `/voice-settings` opens the settings menu after recovering a missing model file, instead of returning and making you run the command a second time.
|
|
140
|
+
- **Chinese output** is shown for detecting models that support Chinese, so it is reachable when the transcription language is automatic.
|
|
141
|
+
- Remote transcripts apply the Chinese output preference when the transcript contains Han characters, not only when the configured language is Chinese.
|
|
142
|
+
- Settings and errors no longer assume local-only transcription: the **Transcription language** row explains that a remote endpoint detects the language itself, and the transcription error, `transcribe_file` setup guidance, and Try-it model row describe the remote backend.
|
|
143
|
+
|
|
144
|
+
**Changed**
|
|
145
|
+
|
|
146
|
+
- Onboarding progress counts the backend step (1 backend → 2 languages → 3 model → 4 Try it).
|
|
147
|
+
- Try it's **change model** keeps the configured transcription language for the new model.
|
|
148
|
+
|
|
118
149
|
### 0.3.0
|
|
119
150
|
|
|
120
151
|
**Added — Post-processing (compaction-style refinement)**
|
|
@@ -133,7 +164,7 @@ First release of `@tychenjiajun/pi-voice` — a fork of [`@earendil-works/pi-voi
|
|
|
133
164
|
**Added — Remote ASR backend**
|
|
134
165
|
|
|
135
166
|
- Recordings can be transcribed by any OpenAI-compatible chat-completions ASR endpoint via the `input_audio` content block.
|
|
136
|
-
- Configure in `/voice-settings` → **Transcription backend** → **Remote ASR (OpenAI-compatible)**: base URL, model, and the *name* of an env var holding the API key (the key itself is never stored in settings).
|
|
167
|
+
- Configure in `/voice-settings` → **Transcription backend** → **Remote ASR (OpenAI-compatible)**: base URL, model, and the *name* of an env var holding the API key (the key itself is never stored in settings). First-run setup also offers the remote backend directly.
|
|
137
168
|
- Audio is uploaded as a 16 kHz mono WAV base64 data URL. Remote transcription is batch-only (no streaming preview).
|
|
138
169
|
- **Local model fallback:** if a remote request fails (network, quota, HTTP error, or the API key is unset), the full recording is transcribed with the configured local model; the failure is logged to `~/.pi/agent/pi-voice.log`. Aborted/cancelled transcriptions are never retried locally.
|
|
139
170
|
|
package/package.json
CHANGED
package/src/chinese.ts
CHANGED
|
@@ -33,6 +33,21 @@ export function isChineseLanguage(language: string): boolean {
|
|
|
33
33
|
return base === "zh" || base === "yue";
|
|
34
34
|
}
|
|
35
35
|
|
|
36
|
+
/**
|
|
37
|
+
* CJK ranges worth converting: Ext A, the main block, compatibility ideographs,
|
|
38
|
+
* and Ext B+ (surrogate pairs are matched by their lead code unit).
|
|
39
|
+
*/
|
|
40
|
+
const HAN_CHARACTERS = /[\u3400-\u4dbf\u4e00-\u9fff\uf900-\ufaff\ud840-\ud87f]/;
|
|
41
|
+
|
|
42
|
+
/**
|
|
43
|
+
* Remote ASR reports no detected language, so Chinese output is decided by
|
|
44
|
+
* looking at the transcript: OpenCC is only worth loading when there is Han
|
|
45
|
+
* text (or the configured language is Chinese) to convert.
|
|
46
|
+
*/
|
|
47
|
+
export function containsChineseCharacters(text: string): boolean {
|
|
48
|
+
return HAN_CHARACTERS.test(text);
|
|
49
|
+
}
|
|
50
|
+
|
|
36
51
|
export async function convertChineseOutput(
|
|
37
52
|
text: string,
|
|
38
53
|
output: ChineseOutput,
|
|
@@ -1,7 +1,11 @@
|
|
|
1
1
|
import { CAPTURE_SAMPLE_RATE } from "./audio-constants.js";
|
|
2
2
|
import { describeError, log, logStep, sinceKeyPress } from "./log.js";
|
|
3
3
|
import { PcmChunker } from "./pcm-chunker.js";
|
|
4
|
-
import
|
|
4
|
+
import {
|
|
5
|
+
transcriptionModelLabel,
|
|
6
|
+
type MicrophoneSetting,
|
|
7
|
+
type TranscribeSettings,
|
|
8
|
+
} from "./settings.js";
|
|
5
9
|
import type { DictationReservation, TranscriptionService } from "./transcription-service.js";
|
|
6
10
|
|
|
7
11
|
/** What a recording captured. `interruption` is why it ended early, if it did. */
|
|
@@ -90,7 +94,7 @@ export class DictationController {
|
|
|
90
94
|
/** Reserves the model for a take; on failure the state says why. */
|
|
91
95
|
private reserve(settings: TranscribeSettings): Take | undefined {
|
|
92
96
|
this.readiness = "loading";
|
|
93
|
-
logStep("reserving model", settings
|
|
97
|
+
logStep("reserving model", transcriptionModelLabel(settings));
|
|
94
98
|
const reservedAt = this.now();
|
|
95
99
|
let reservation: DictationReservation;
|
|
96
100
|
try {
|
|
@@ -12,7 +12,7 @@ import { basename, join, resolve } from "node:path";
|
|
|
12
12
|
import { Type } from "typebox";
|
|
13
13
|
import { AsyncLimiter } from "./async-limiter.js";
|
|
14
14
|
import { polishTranscript } from "./polish.js";
|
|
15
|
-
import type
|
|
15
|
+
import { transcriptionModelLabel, type TranscribeSettings } from "./settings.js";
|
|
16
16
|
import type { TranscriptionService } from "./transcription-service.js";
|
|
17
17
|
|
|
18
18
|
const CONTEXT_LINE_LENGTH = 1_000;
|
|
@@ -84,7 +84,7 @@ export function registerFileTranscriptionTool(
|
|
|
84
84
|
pi.registerTool({
|
|
85
85
|
name: "transcribe_file",
|
|
86
86
|
label: "Transcribe File",
|
|
87
|
-
description: `Transcribe speech from a local audio or video file using Pi Voice's configured transcription backend (local model
|
|
87
|
+
description: `Transcribe speech from a local audio or video file using Pi Voice's configured transcription backend (local model or remote ASR endpoint), then applies Pi Voice's compaction-style post-processing when it is enabled. Requires the ffmpeg executable on PATH (or PI_VOICE_FFMPEG_PATH). Output is truncated to ${DEFAULT_MAX_LINES} lines or ${formatSize(DEFAULT_MAX_BYTES)}; a complete transcript is saved to a temporary file when needed.`,
|
|
88
88
|
promptSnippet: "Transcribe speech from local audio or video files",
|
|
89
89
|
promptGuidelines: [
|
|
90
90
|
"Use transcribe_file when speech in a local audio or video file needs to be read, analyzed, or transcribed.",
|
|
@@ -118,10 +118,11 @@ export function registerFileTranscriptionTool(
|
|
|
118
118
|
|
|
119
119
|
const configured = await options.getSettings();
|
|
120
120
|
const service = await options.getService();
|
|
121
|
+
const modelLabel = transcriptionModelLabel(configured);
|
|
121
122
|
if (fileOperations.saturated) {
|
|
122
123
|
onUpdate?.({
|
|
123
124
|
content: [{ type: "text", text: "Waiting for file transcription capacity…" }],
|
|
124
|
-
details: { inputPath, modelId:
|
|
125
|
+
details: { inputPath, modelId: modelLabel, seconds: 0 },
|
|
125
126
|
});
|
|
126
127
|
}
|
|
127
128
|
|
|
@@ -129,13 +130,13 @@ export function registerFileTranscriptionTool(
|
|
|
129
130
|
if (fileDecoders.saturated) {
|
|
130
131
|
onUpdate?.({
|
|
131
132
|
content: [{ type: "text", text: `Waiting to decode ${basename(inputPath)}…` }],
|
|
132
|
-
details: { inputPath, modelId:
|
|
133
|
+
details: { inputPath, modelId: modelLabel, seconds: 0 },
|
|
133
134
|
});
|
|
134
135
|
}
|
|
135
136
|
const audio = await fileDecoders.run(async () => {
|
|
136
137
|
onUpdate?.({
|
|
137
138
|
content: [{ type: "text", text: `Decoding ${basename(inputPath)} with FFmpeg…` }],
|
|
138
|
-
details: { inputPath, modelId:
|
|
139
|
+
details: { inputPath, modelId: modelLabel, seconds: 0 },
|
|
139
140
|
});
|
|
140
141
|
const { decodeFileAudio } = await import("./file-audio.js");
|
|
141
142
|
return decodeFileAudio(inputPath, operationSignal);
|
|
@@ -145,12 +146,12 @@ export function registerFileTranscriptionTool(
|
|
|
145
146
|
content: [
|
|
146
147
|
{
|
|
147
148
|
type: "text",
|
|
148
|
-
text: `Queued ${audio.seconds.toFixed(1)}s from ${basename(inputPath)} for
|
|
149
|
+
text: `Queued ${audio.seconds.toFixed(1)}s from ${basename(inputPath)} for transcription…`,
|
|
149
150
|
},
|
|
150
151
|
],
|
|
151
152
|
details: {
|
|
152
153
|
inputPath,
|
|
153
|
-
modelId:
|
|
154
|
+
modelId: modelLabel,
|
|
154
155
|
seconds: audio.seconds,
|
|
155
156
|
},
|
|
156
157
|
});
|
|
@@ -170,7 +171,7 @@ export function registerFileTranscriptionTool(
|
|
|
170
171
|
: rawTranscript;
|
|
171
172
|
const details: FileTranscriptionDetails = {
|
|
172
173
|
inputPath,
|
|
173
|
-
modelId:
|
|
174
|
+
modelId: modelLabel,
|
|
174
175
|
seconds: audio.seconds,
|
|
175
176
|
};
|
|
176
177
|
|
package/src/onboarding.ts
CHANGED
|
@@ -14,11 +14,13 @@ import {
|
|
|
14
14
|
} from "./recommendation-picker.js";
|
|
15
15
|
import { CATALOG_MODELS } from "./catalog.js";
|
|
16
16
|
import { recommendModels } from "./recommendations.js";
|
|
17
|
+
import { promptRemoteBackend, warnMissingApiKey } from "./remote-asr.js";
|
|
17
18
|
import { createShortcutPicker } from "./shortcuts.js";
|
|
18
19
|
import { tryVoice } from "./try-it.js";
|
|
19
20
|
import {
|
|
20
21
|
DEFAULT_MICROPHONE,
|
|
21
22
|
settingsForModel,
|
|
23
|
+
settingsRemoteOnly,
|
|
22
24
|
writeSettings,
|
|
23
25
|
type BackendSettings,
|
|
24
26
|
type ChineseOutput,
|
|
@@ -28,6 +30,10 @@ import {
|
|
|
28
30
|
type TranscriptionLanguage,
|
|
29
31
|
} from "./settings.js";
|
|
30
32
|
import { DEFAULT_SHORTCUT } from "./shortcut-core.js";
|
|
33
|
+
import {
|
|
34
|
+
SingleSelectPicker,
|
|
35
|
+
type SingleSelectChoice,
|
|
36
|
+
} from "./ui-components.js";
|
|
31
37
|
|
|
32
38
|
function requireTui(ctx: ExtensionContext): boolean {
|
|
33
39
|
if (ctx.mode === "tui") return true;
|
|
@@ -91,7 +97,7 @@ export async function runModelSelection(
|
|
|
91
97
|
}),
|
|
92
98
|
(settings) => {
|
|
93
99
|
configured = settings;
|
|
94
|
-
currentModelId = settings.model
|
|
100
|
+
currentModelId = settings.model?.id;
|
|
95
101
|
},
|
|
96
102
|
);
|
|
97
103
|
|
|
@@ -213,6 +219,7 @@ export async function changeOnboardingModel(
|
|
|
213
219
|
() => ({
|
|
214
220
|
shortcut: current.shortcut,
|
|
215
221
|
preferredLanguages: languages,
|
|
222
|
+
transcriptionLanguage: current.transcriptionLanguage,
|
|
216
223
|
microphone: current.microphone,
|
|
217
224
|
chineseOutput: current.chineseOutput,
|
|
218
225
|
backend: current.backend,
|
|
@@ -231,7 +238,7 @@ export async function changeOnboardingModel(
|
|
|
231
238
|
expanded: true,
|
|
232
239
|
onboardingStep: 3,
|
|
233
240
|
})
|
|
234
|
-
: await chooseCatalogModel(ctx, languages, chosen?.model
|
|
241
|
+
: await chooseCatalogModel(ctx, languages, chosen?.model?.id ?? current.model?.id, {
|
|
235
242
|
postActivation: "advance",
|
|
236
243
|
onActivate: activation.activate,
|
|
237
244
|
cancelLabel: "back",
|
|
@@ -264,12 +271,60 @@ export async function changeOnboardingModel(
|
|
|
264
271
|
}
|
|
265
272
|
}
|
|
266
273
|
|
|
274
|
+
export type SetupMode = "local" | "remote";
|
|
275
|
+
|
|
276
|
+
/**
|
|
277
|
+
* First screen of first-run setup: on-device model, or a remote ASR endpoint.
|
|
278
|
+
* The modes are exclusive: remote setup downloads no local model.
|
|
279
|
+
*/
|
|
280
|
+
async function chooseSetupMode(ctx: ExtensionContext): Promise<SetupMode | undefined> {
|
|
281
|
+
const choices: SingleSelectChoice<SetupMode>[] = [
|
|
282
|
+
{
|
|
283
|
+
value: "local",
|
|
284
|
+
label: "Local model",
|
|
285
|
+
description: "Transcribe fully on-device; no audio leaves this machine",
|
|
286
|
+
},
|
|
287
|
+
{
|
|
288
|
+
value: "remote",
|
|
289
|
+
label: "Remote ASR (OpenAI-compatible)",
|
|
290
|
+
description:
|
|
291
|
+
"Send audio to an OpenAI-compatible ASR endpoint; no local model is downloaded",
|
|
292
|
+
},
|
|
293
|
+
];
|
|
294
|
+
return ctx.ui.custom<SetupMode | undefined>((tui, theme, keybindings, done) =>
|
|
295
|
+
new SingleSelectPicker(
|
|
296
|
+
tui,
|
|
297
|
+
theme,
|
|
298
|
+
keybindings,
|
|
299
|
+
choices,
|
|
300
|
+
undefined,
|
|
301
|
+
{
|
|
302
|
+
title: "Pi Voice setup",
|
|
303
|
+
subtitle: "How should Pi Voice transcribe your speech?",
|
|
304
|
+
cancelLabel: "cancel",
|
|
305
|
+
onboardingStep: 1,
|
|
306
|
+
},
|
|
307
|
+
done,
|
|
308
|
+
),
|
|
309
|
+
);
|
|
310
|
+
}
|
|
311
|
+
|
|
267
312
|
export async function runOnboarding(
|
|
268
313
|
ctx: ExtensionContext,
|
|
269
314
|
shortcut = DEFAULT_SHORTCUT,
|
|
270
315
|
): Promise<TranscribeSettings | undefined> {
|
|
271
316
|
if (!requireTui(ctx)) return undefined;
|
|
272
317
|
|
|
318
|
+
const mode = await chooseSetupMode(ctx);
|
|
319
|
+
if (!mode) return undefined;
|
|
320
|
+
|
|
321
|
+
let backend: BackendSettings | undefined;
|
|
322
|
+
if (mode === "remote") {
|
|
323
|
+
backend = await promptRemoteBackend(ctx);
|
|
324
|
+
if (!backend) return undefined;
|
|
325
|
+
warnMissingApiKey(ctx, backend.apiKeyEnv);
|
|
326
|
+
}
|
|
327
|
+
|
|
273
328
|
let languages = defaultSpokenLanguages();
|
|
274
329
|
// Navigation cannot undo a completed settings commit. Keep it across a
|
|
275
330
|
// return to languages, even if the user then exits without another choice.
|
|
@@ -277,15 +332,27 @@ export async function runOnboarding(
|
|
|
277
332
|
while (true) {
|
|
278
333
|
const chosen = await chooseLanguages(ctx, languages, {
|
|
279
334
|
cancelLabel: "exit",
|
|
280
|
-
onboardingStep:
|
|
335
|
+
onboardingStep: 2,
|
|
281
336
|
});
|
|
282
337
|
if (!chosen?.confirmed) return configured;
|
|
283
338
|
languages = chosen.languages;
|
|
339
|
+
|
|
340
|
+
// Remote mode is remote-only: no model step, settings are complete once
|
|
341
|
+
// the endpoint and languages are chosen.
|
|
342
|
+
if (backend?.type === "openai-asr") {
|
|
343
|
+
const settings = settingsRemoteOnly({ backend, shortcut, preferredLanguages: languages });
|
|
344
|
+
if (!(await saveOnboardingSettings(ctx, settings))) return configured;
|
|
345
|
+
configured = settings;
|
|
346
|
+
return finishOnboarding(ctx, configured, (current) =>
|
|
347
|
+
changeOnboardingModel(ctx, current),
|
|
348
|
+
);
|
|
349
|
+
}
|
|
350
|
+
|
|
284
351
|
// The picks are judged on the benchmark rig; the Try it step measures
|
|
285
352
|
// the real wait on this machine.
|
|
286
353
|
const picks = recommendModels(CATALOG_MODELS, languages);
|
|
287
354
|
const { activate, waitForCommits } = createSettingsActivation(
|
|
288
|
-
() => ({ shortcut, preferredLanguages: languages }),
|
|
355
|
+
() => ({ shortcut, preferredLanguages: languages, backend }),
|
|
289
356
|
(settings) => {
|
|
290
357
|
configured = settings;
|
|
291
358
|
},
|
|
@@ -298,7 +365,9 @@ export async function runOnboarding(
|
|
|
298
365
|
languages,
|
|
299
366
|
picks,
|
|
300
367
|
activate,
|
|
301
|
-
{
|
|
368
|
+
{
|
|
369
|
+
onboardingStep: 3,
|
|
370
|
+
},
|
|
302
371
|
);
|
|
303
372
|
await waitForCommits();
|
|
304
373
|
if (!recommendation) return configured;
|
|
@@ -310,11 +379,11 @@ export async function runOnboarding(
|
|
|
310
379
|
continue;
|
|
311
380
|
}
|
|
312
381
|
|
|
313
|
-
const selection = await chooseCatalogModel(ctx, languages, configured?.model
|
|
382
|
+
const selection = await chooseCatalogModel(ctx, languages, configured?.model?.id, {
|
|
314
383
|
postActivation: "advance",
|
|
315
384
|
onActivate: activate,
|
|
316
385
|
cancelLabel: "back",
|
|
317
|
-
onboardingStep:
|
|
386
|
+
onboardingStep: 3,
|
|
318
387
|
title: "Browse all models",
|
|
319
388
|
});
|
|
320
389
|
await waitForCommits();
|
package/src/openai-asr.ts
CHANGED
|
@@ -1,22 +1,15 @@
|
|
|
1
1
|
import OpenAI from "openai";
|
|
2
2
|
import { CAPTURE_SAMPLE_RATE } from "./audio-constants.js";
|
|
3
|
-
import { convertChineseOutput, isChineseLanguage } from "./chinese.js";
|
|
3
|
+
import { containsChineseCharacters, convertChineseOutput, isChineseLanguage } from "./chinese.js";
|
|
4
4
|
import { describeError, writeLog } from "./log.js";
|
|
5
5
|
import type { BackendSettings, ChineseOutput } from "./settings.js";
|
|
6
6
|
import type {
|
|
7
7
|
DictationStream,
|
|
8
|
-
TranscribeCppBackend,
|
|
9
8
|
TranscriptionOptions,
|
|
10
9
|
} from "./transcription.js";
|
|
11
10
|
|
|
12
11
|
type OpenAIAsrSettings = Extract<BackendSettings, { type: "openai-asr" }>;
|
|
13
12
|
|
|
14
|
-
/** The slice of the local backend this module needs for offline fallback. */
|
|
15
|
-
export type LocalFallback = Pick<
|
|
16
|
-
TranscribeCppBackend,
|
|
17
|
-
"prepare" | "transcribe" | "dispose"
|
|
18
|
-
>;
|
|
19
|
-
|
|
20
13
|
export type AsrRequest = {
|
|
21
14
|
/** `data:audio/wav;base64,…` of the whole recording. */
|
|
22
15
|
dataUrl: string;
|
|
@@ -122,22 +115,20 @@ function createOpenAITransport(config: OpenAIAsrSettings): {
|
|
|
122
115
|
/**
|
|
123
116
|
* Trims, then applies the Chinese script preference exactly like the local
|
|
124
117
|
* backend. The chat-completions response carries no detected language, so
|
|
125
|
-
*
|
|
126
|
-
*
|
|
118
|
+
* Chinese is inferred from the text itself: Han characters mean the transcript
|
|
119
|
+
* is Chinese and the preference applies, matching local models that detect it.
|
|
127
120
|
*/
|
|
128
121
|
export async function finishRemoteTranscript(
|
|
129
122
|
text: string,
|
|
130
123
|
options: TranscriptionOptions,
|
|
131
124
|
): Promise<string> {
|
|
132
125
|
const trimmed = text.trim();
|
|
133
|
-
return isChineseLanguage(options.language ?? "")
|
|
126
|
+
return isChineseLanguage(options.language ?? "") || containsChineseCharacters(trimmed)
|
|
134
127
|
? convertChineseOutput(trimmed, options.chineseOutput ?? "simplified")
|
|
135
128
|
: trimmed;
|
|
136
129
|
}
|
|
137
130
|
|
|
138
131
|
export type OpenAIAsrBackendDeps = {
|
|
139
|
-
/** Creates the local model used when a remote call fails. */
|
|
140
|
-
createFallback?: () => Promise<LocalFallback>;
|
|
141
132
|
/** Overrides the network transport (tests). */
|
|
142
133
|
transport?: AsrTransport;
|
|
143
134
|
};
|
|
@@ -145,12 +136,11 @@ export type OpenAIAsrBackendDeps = {
|
|
|
145
136
|
/**
|
|
146
137
|
* Batch-only remote backend: `startStream` resolves to undefined so the
|
|
147
138
|
* service records the whole clip and transcribes it after stop, exactly like
|
|
148
|
-
* a non-streaming local model. Remote failures
|
|
149
|
-
*
|
|
139
|
+
* a non-streaming local model. Remote failures surface to the caller; there
|
|
140
|
+
* is no local fallback.
|
|
150
141
|
*/
|
|
151
142
|
export class OpenAIAsrBackend {
|
|
152
143
|
private client: { transport: AsrTransport; dispose: () => void } | undefined;
|
|
153
|
-
private fallback: Promise<LocalFallback> | undefined;
|
|
154
144
|
private disposed = false;
|
|
155
145
|
private readonly injected: AsrTransport | undefined;
|
|
156
146
|
|
|
@@ -164,8 +154,8 @@ export class OpenAIAsrBackend {
|
|
|
164
154
|
async prepare(): Promise<void> {
|
|
165
155
|
if (this.disposed) throw new Error("Remote ASR backend has been disposed");
|
|
166
156
|
// The transport is created lazily in transcribe() so that a missing API
|
|
167
|
-
// key
|
|
168
|
-
//
|
|
157
|
+
// key surfaces per recording instead of failing prepare() and blocking
|
|
158
|
+
// every subsequent transcription.
|
|
169
159
|
}
|
|
170
160
|
|
|
171
161
|
/** Batch-only: the dictation service drops live chunks and transcribes on stop. */
|
|
@@ -184,9 +174,9 @@ export class OpenAIAsrBackend {
|
|
|
184
174
|
if (!transport) {
|
|
185
175
|
if (this.disposed) throw new Error("Remote ASR backend has been disposed");
|
|
186
176
|
// Lazily build the transport inside the try so a missing API key or
|
|
187
|
-
// any construction failure
|
|
188
|
-
// cached; a failure is not, so a later call retries once
|
|
189
|
-
// fixed.
|
|
177
|
+
// any construction failure is reported like a request failure.
|
|
178
|
+
// Success is cached; a failure is not, so a later call retries once
|
|
179
|
+
// the env is fixed.
|
|
190
180
|
const client = this.client ?? createOpenAITransport(this.config);
|
|
191
181
|
this.client = client;
|
|
192
182
|
transport = client.transport;
|
|
@@ -205,20 +195,12 @@ export class OpenAIAsrBackend {
|
|
|
205
195
|
return await finishRemoteTranscript(text, options);
|
|
206
196
|
} catch (error) {
|
|
207
197
|
if (signal?.aborted || this.disposed) throw error;
|
|
208
|
-
if (!this.deps.createFallback) throw error;
|
|
209
198
|
writeLog(
|
|
210
199
|
"warn",
|
|
211
200
|
"voice",
|
|
212
|
-
`remote ASR failed (${describeError(error)})
|
|
201
|
+
`remote ASR failed (${describeError(error)})`,
|
|
213
202
|
);
|
|
214
|
-
|
|
215
|
-
const local = await this.fallbackBackend();
|
|
216
|
-
return await local.transcribe(pcm, options);
|
|
217
|
-
} catch (fallbackError) {
|
|
218
|
-
throw new Error(
|
|
219
|
-
`Remote ASR failed (${describeError(error)}) and the local fallback failed (${describeError(fallbackError)})`,
|
|
220
|
-
);
|
|
221
|
-
}
|
|
203
|
+
throw error;
|
|
222
204
|
}
|
|
223
205
|
}
|
|
224
206
|
|
|
@@ -227,21 +209,5 @@ export class OpenAIAsrBackend {
|
|
|
227
209
|
this.disposed = true;
|
|
228
210
|
this.client?.dispose();
|
|
229
211
|
this.client = undefined;
|
|
230
|
-
if (this.fallback) {
|
|
231
|
-
const local = await this.fallback.catch(() => undefined);
|
|
232
|
-
await local?.dispose().catch(() => undefined);
|
|
233
|
-
}
|
|
234
|
-
}
|
|
235
|
-
|
|
236
|
-
private fallbackBackend(): Promise<LocalFallback> {
|
|
237
|
-
if (!this.fallback) {
|
|
238
|
-
// A rejected fallback is not cached: a later attempt can re-create the
|
|
239
|
-
// local model instead of re-throwing a stale failure.
|
|
240
|
-
this.fallback = this.deps.createFallback!().catch((error) => {
|
|
241
|
-
this.fallback = undefined;
|
|
242
|
-
throw error;
|
|
243
|
-
});
|
|
244
|
-
}
|
|
245
|
-
return this.fallback;
|
|
246
212
|
}
|
|
247
213
|
}
|