@gotcos/glasses-server 6.72.0 → 6.73.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +22 -0
- package/README.md +50 -12
- package/package.json +2 -1
- package/server/lib/codex-model-catalog.ts +15 -4
- package/server/lib/dictation-stt.ts +117 -0
- package/server/lib/parakeet.ts +14 -4
- package/server/lib/prompt-draft-store.ts +5 -0
- package/server/lib/stt-runtime.ts +10 -2
- package/server/lib/stt-settings.ts +91 -2
- package/server/lib/transcribe-audio.ts +53 -4
- package/server/routes/prompt-drafts.ts +56 -17
- package/server/routes/transcribe.ts +6 -2
package/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,25 @@
|
|
|
1
|
+
## 6.73.1 — GPT tier selection correction
|
|
2
|
+
|
|
3
|
+
- Resolve GPT Max/Frontier to the newest provider-described frontier model and GPT Balanced to the newest workhorse/balanced model. A newer workhorse version no longer displaces a frontier model: GPT-6-Astra serves Max, GPT-6.1-Sol serves Balanced in the current catalog.
|
|
4
|
+
- Keep version-based selection when the provider does not describe both tiers; preserve saved tier preferences, explicit model overrides, discovery refresh, and last-known-good fallback.
|
|
5
|
+
- Server catalog and execution share the corrected selection. Existing companion builds can receive the correction without an EHPK update. No transcription settings or experimental flags change.
|
|
6
|
+
|
|
7
|
+
## [6.73.0] - 2026-10-13 - Prompts and dictation follow the speech-to-text settings
|
|
8
|
+
|
|
9
|
+
Paired clients: none required. Every phone, including the public 618, already accepts the new dictation text (it never checks a model on dictation). COS Control and the phone can read the new `dictation` block in `GET /api/stt-settings` to replace "Dictation is unchanged" in their copy.
|
|
10
|
+
|
|
11
|
+
- Dictation follows the settings (`docs/dictation-stt-contract.md`). The live provisional words (`/api/prompt-drafts/:id/peek`), the closed-chunk HUD text and the Fast mode final follow the Preview model. The final text with Fast mode off, and the speculative HQ warm, follow the HQ model. `/api/transcribe` (Quick VOICE, browser recordings) follows the same split. With the defaults (preview `turbo`, HQ `large-v3`) every dictation decode is the 6.72.0 Whisper decode, unchanged.
|
|
12
|
+
- Fast mode now means "the fast engine you picked", the Preview model. On a standard Mac the Whisper choice is the Turbo live worker; on a Max-tier Mac it is the live Large-v3 worker, as before. A `large-v3` or `nemotron-560` Preview choice keeps dictation on Whisper and says why.
|
|
13
|
+
- Parakeet Ultra runs for a prompt only when it is installed, warm, and the breaker is closed. When it is cold, failed, busy, or the audio is not 16 kHz WAV (the browser recorder posts WebM), the 6.72.0 Whisper engine decodes the same audio in the same request, so the user never sees a blank or an error because of Parakeet. The engine is chosen when each decode starts and never changes during it. A finalize takes one settings snapshot for all its chunks, and a chunk decoded under a different selection is decoded again instead of reused (cached records now carry `sttKey`; records from before 6.73.0 count as the 6.72.0 defaults).
|
|
14
|
+
- Parakeet dictation runs on a new interactive lane: normal priority, never queued behind a meeting batch, at most two at once. The live preview uses the existing one-at-a-time preview lane. Dictation Parakeet failures count on the same breaker as the meeting preview; a busy lane does not. Choosing a Parakeet HQ model now also warms Parakeet (never while a recording is live).
|
|
15
|
+
- Second opinion and resolver never run on dictation. The quiet gate never drops dictation audio: on clean spoken one-to-three-word commands the Silero rule (under 0.5 s of speech) would have dropped 18 of 30. The front end applies to Parakeet dictation decodes; Whisper dictation keeps raw audio as before (highpass changed the first word of Large-v3 HQ on 24 of 149 real prompt chunks).
|
|
16
|
+
- Additive response fields: `engine` (`whisper`, `parakeet-ultra`, or `mixed` on a finalize) and `engineReason` on `/api/transcribe`, finalize and retry; `engine` on `prompt_transcript` events. `backend` reads `parakeet-ultra` for a Parakeet decode. `actualQuality` keeps its three values, so a Parakeet HQ prompt is HQ, not degraded.
|
|
17
|
+
- `GET /api/stt-settings` adds `dictation` (`preview`, `fast`, `final`, each with `engine`, `follows`, `reason`, `reasonCode`; `frontEnd`, `quietGate: "none"`, `secondOpinion: "off"`, `note`) and `appliesTo.dictation`. Every 6.72.0 key keeps its value. The Parakeet preview option note now says dictation uses it on every phone.
|
|
18
|
+
- `COS_DICTATION_FOLLOWS_STT=0` holds every dictation decode on its 6.72.0 Whisper engine.
|
|
19
|
+
- Measured on this Mac (M3 Ultra, private engines on spare ports; full tables in the contract): on 152 real prompt chunks of 2 to 15 s, Parakeet Ultra decodes at p50 233 ms, p95 419 ms (plus the highpass front end, p50 58 ms), the live Large-v3 worker at 316 / 467 ms, the Turbo live worker at 195 / 241 ms, and whisper-cli Large-v3 HQ at 1,544 / 1,844 ms. On a synthetic set with known text, Parakeet Ultra matched 69% of name targets clean and 63% in G2 babble, against 98% and 86% for Large-v3 HQ; numbers with units and negations were level (negations 100% for every engine). Parakeet is clearly weaker on names for prompts.
|
|
20
|
+
- No language model is called by any transcription path. No new downloads.
|
|
21
|
+
- Tests: new suites for the dictation mapping, the Parakeet dictation runner, the plan in `transcribeAudioBuffer`, prompt-draft reuse across selections and the Parakeet live preview, the interactive lane and the HQ warm-up. Mutation gate: `node server/scripts/dictation-stt-mutation-gate.mjs` (repository only), green baseline first (100 tests in 8 files), each target matched exactly once: 22 of 22 killed, each by a named test.
|
|
22
|
+
|
|
1
23
|
## [6.72.0] - 2026-10-13 - Speech-to-text switcher, front-end fixes, Parakeet Ultra as an option
|
|
2
24
|
|
|
3
25
|
Paired clients: COS Control 0.5.289 (Settings, Transcription) and COS Glasses 6.10.639 (Settings, Voice). Older clients show no switcher, but the new defaults below apply to their meetings too; change them with the routes or `<data>/stt-settings.json`.
|
package/README.md
CHANGED
|
@@ -148,7 +148,7 @@ range is the exact Tailscale/CGNAT allocation (`100.64.0.0/10`), not all of
|
|
|
148
148
|
`COS_OPENAI_WHISPER_FALLBACK=1` opt-in and a configured key; a key alone never
|
|
149
149
|
uploads audio.
|
|
150
150
|
- HQ prompt dictation is requested by default; the phone's **Fast mode** switch
|
|
151
|
-
opts into
|
|
151
|
+
opts into the fast engine (the STT Preview model since 6.73.0). Server 6.16.0 reports whether full local large-v3 actually
|
|
152
152
|
ran, and compatible companions alert once if an HQ request used Fast or
|
|
153
153
|
Cloud instead of silently claiming HQ.
|
|
154
154
|
- Local-first spoken reply playback through Kokoro on Apple silicon. The first
|
|
@@ -679,7 +679,11 @@ GET returns (abridged):
|
|
|
679
679
|
"progress": {"state": "idle", "bytesDone": 0, "bytesTotal": 654356591, "error": null},
|
|
680
680
|
"warm": {"preview": "cold"}, "preview": {"breaker": {"open": false, "retryAt": null, "misses": 0}, "counters": {}}, "note": "..."},
|
|
681
681
|
"lastJob": {"at": "...", "meeting": "...", "outcome": "batch", "summary": {"windows": 12, "primaryFallbacks": 0, "timestampFallbacks": 0, "secondOpinionFailures": 0, "windowsLost": 0}},
|
|
682
|
-
"
|
|
682
|
+
"dictation": {"preview": {"engine": "whisper", "follows": "preview", "reason": null, "reasonCode": null},
|
|
683
|
+
"fast": {"engine": "whisper", "follows": "preview", "reason": null, "reasonCode": null},
|
|
684
|
+
"final": {"engine": "whisper", "follows": "hq", "reason": null, "reasonCode": null},
|
|
685
|
+
"frontEnd": "raw", "quietGate": "none", "secondOpinion": "off", "note": "..."},
|
|
686
|
+
"appliesTo": {"preview": "next live chunk", "hq": "next saved meeting", "dictation": "next dictation chunk"}
|
|
683
687
|
}
|
|
684
688
|
```
|
|
685
689
|
|
|
@@ -711,19 +715,52 @@ GET returns (abridged):
|
|
|
711
715
|
never gets gap fill). See the evaluation for what each measured.
|
|
712
716
|
- `COS_BATCH_HQ_TIMESTAMPS` still overrides HQ timestamps: `1` forces them on,
|
|
713
717
|
`0` off. `COS_WHISPER_MEETING_PREVIEW=0` still turns the meeting preview off.
|
|
714
|
-
-
|
|
715
|
-
|
|
716
|
-
|
|
717
|
-
|
|
718
|
-
|
|
719
|
-
|
|
720
|
-
|
|
721
|
-
preview hears the selected front end.
|
|
718
|
+
- On the meeting preview lane (`POST /api/transcribe-stream/preview`) a client
|
|
719
|
+
gets a non-Turbo preview only if it lists that model in the `previewModels`
|
|
720
|
+
query parameter (comma-separated; COS Glasses 6.10.639 sends it); every other
|
|
721
|
+
client gets Whisper Turbo, so a phone that only accepts Turbo never sees
|
|
722
|
+
another model. Playback audio filters are unchanged. The Whisper previews keep
|
|
723
|
+
raw audio; the Parakeet preview hears the selected front end.
|
|
724
|
+
- Since 6.73.0 the settings drive dictation too; see "Dictation" below.
|
|
722
725
|
- The quiet gate decides the prompt and the front end of a live chunk. The
|
|
723
726
|
silence-hallucination text drops (URL-only, repeated thank-you, a quiet
|
|
724
727
|
vocabulary echo) stay on the pre-6.72 RMS test, because a chunk holding one
|
|
725
728
|
short real name can be Silero-quiet.
|
|
726
729
|
|
|
730
|
+
### Dictation (6.73.0)
|
|
731
|
+
|
|
732
|
+
Prompts and dictation (Record Message, hold-to-talk, edit instructions, Quick
|
|
733
|
+
VOICE) follow the same settings. Contract and measurements:
|
|
734
|
+
`docs/dictation-stt-contract.md`.
|
|
735
|
+
|
|
736
|
+
| Dictation decode | Follows | Whisper choice (default) | `parakeet` / `parakeet-ultra` |
|
|
737
|
+
| --- | --- | --- | --- |
|
|
738
|
+
| Live provisional words (`/api/prompt-drafts/:id/peek`) | Preview model | the 6.72.0 preview sidecar | Parakeet Ultra |
|
|
739
|
+
| Closed-chunk HUD text and the Fast mode final | Preview model | the live worker (Turbo, or Large-v3 on a Max-tier Mac) | Parakeet Ultra |
|
|
740
|
+
| Final text with Fast mode off, and the HQ warm | HQ model | whisper-cli Large-v3 | Parakeet Ultra |
|
|
741
|
+
|
|
742
|
+
- Fast mode means "the fast engine you picked": the Preview model. With Fast
|
|
743
|
+
mode off the final text comes from the HQ model.
|
|
744
|
+
- Parakeet runs for a prompt only when installed, warm, and with the breaker
|
|
745
|
+
closed. Otherwise, and when a Parakeet decode fails, is busy, or the audio is
|
|
746
|
+
not 16 kHz WAV (the browser recorder posts WebM), the 6.72.0 Whisper engine
|
|
747
|
+
decodes that audio and the response says why (`engineReason`). Selecting a
|
|
748
|
+
Parakeet HQ model also warms Parakeet. Dictation Parakeet failures count on
|
|
749
|
+
the same breaker as the meeting preview.
|
|
750
|
+
- The engine is chosen when each decode starts. A finalize takes one snapshot
|
|
751
|
+
for all its chunks, and a chunk decoded under another selection is decoded
|
|
752
|
+
again rather than reused.
|
|
753
|
+
- Second opinion and resolver never run on dictation. The quiet gate never
|
|
754
|
+
drops dictation audio. The front end applies to Parakeet dictation decodes;
|
|
755
|
+
Whisper dictation decodes keep raw audio, as before.
|
|
756
|
+
- Responses gain `engine` (`whisper`, `parakeet-ultra`, or `mixed` on a
|
|
757
|
+
finalize) and `engineReason`; `prompt_transcript` events gain `engine`.
|
|
758
|
+
`actualQuality` keeps its values, so a Parakeet HQ prompt is not "degraded".
|
|
759
|
+
No phone needs a new request field: the phone never checks a model on
|
|
760
|
+
dictation text.
|
|
761
|
+
- `COS_DICTATION_FOLLOWS_STT=0` holds every dictation decode on its 6.72.0
|
|
762
|
+
Whisper engine.
|
|
763
|
+
|
|
727
764
|
### Parakeet (FluidAudio CoreML, offline)
|
|
728
765
|
|
|
729
766
|
`POST /api/stt-settings/parakeet/install` installs FluidAudio's `fluidaudiocli`
|
|
@@ -814,8 +851,9 @@ and empty workspace. Live preview and canonical meeting transcription are unchan
|
|
|
814
851
|
## HQ dictation
|
|
815
852
|
|
|
816
853
|
Prompt dictation defaults to HQ. The phone owns the preference: **Fast mode
|
|
817
|
-
OFF** requests HQ, and **Fast mode ON** requests
|
|
818
|
-
|
|
854
|
+
OFF** requests HQ, and **Fast mode ON** requests the fast engine (the live
|
|
855
|
+
worker, or Parakeet Ultra when it is the Preview model; see "Dictation (6.73.0)").
|
|
856
|
+
The Mac performs all decoding; the phone does not run Whisper.
|
|
819
857
|
|
|
820
858
|
For the recommended **Balanced** setup, run:
|
|
821
859
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@gotcos/glasses-server",
|
|
3
|
-
"version": "6.
|
|
3
|
+
"version": "6.73.1",
|
|
4
4
|
"description": "COS Glasses: self-hosted AI heads-up-display server for Even G2 smart glasses, powered by Claude Code, Codex, Cursor Agent CLI, or local Ollama",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
@@ -53,6 +53,7 @@
|
|
|
53
53
|
"!server/scripts/fact-check-mutation-gate.mjs",
|
|
54
54
|
"!server/scripts/stale-session-mutation-gate.mjs",
|
|
55
55
|
"!server/scripts/stt-mutation-gate.mjs",
|
|
56
|
+
"!server/scripts/dictation-stt-mutation-gate.mjs",
|
|
56
57
|
"!server/scripts/pairing-mutation-gate.mjs",
|
|
57
58
|
"!server/scripts/away-hold-mutation-gate.mjs",
|
|
58
59
|
"!server/scripts/nemotron-mutation-gate.mjs",
|
|
@@ -50,7 +50,7 @@ export interface CodexModelCatalog {
|
|
|
50
50
|
source: CodexCatalogSource
|
|
51
51
|
refreshedAt: string
|
|
52
52
|
autoUpdates: true
|
|
53
|
-
selectionPolicy: 'newest-generation
|
|
53
|
+
selectionPolicy: 'capability-tiers-then-newest-generation'
|
|
54
54
|
options: CodexModelOption[]
|
|
55
55
|
refreshError?: string
|
|
56
56
|
}
|
|
@@ -93,13 +93,24 @@ function isCapabilityCandidate(model: CodexCatalogModel): boolean {
|
|
|
93
93
|
return !/(?:fast and affordable|cost-efficient|small,? fast|ultra-fast)/i.test(model.description)
|
|
94
94
|
}
|
|
95
95
|
|
|
96
|
-
/**
|
|
96
|
+
/** Prefer the provider's frontier/workhorse tiers; use generation order when
|
|
97
|
+
* the catalog does not describe both tiers. A newer workhorse (6.1 Sol) must
|
|
98
|
+
* not replace the frontier (6 Astra) merely because its version is higher.
|
|
99
|
+
*/
|
|
97
100
|
export function selectTopCodexModels(models: CodexCatalogModel[]): CodexCatalogModel[] {
|
|
98
101
|
const visible = models.filter(model => !model.hidden && versionParts(model.id))
|
|
99
102
|
if (visible.length === 0) return []
|
|
100
103
|
|
|
101
104
|
const capable = visible.filter(isCapabilityCandidate)
|
|
102
105
|
const pool = capable.length > 0 ? capable : visible
|
|
106
|
+
const byNewest = (candidates: CodexCatalogModel[]): CodexCatalogModel | undefined =>
|
|
107
|
+
candidates.slice().sort((a, b) => compareVersionsDesc(versionParts(a.id) ?? [], versionParts(b.id) ?? []))[0]
|
|
108
|
+
const frontier = byNewest(pool.filter(model => /\bfrontier\b/i.test(model.description)))
|
|
109
|
+
const balanced = byNewest(pool.filter(model =>
|
|
110
|
+
!/\bfrontier\b/i.test(model.description) && /\b(workhorse|balanced)\b/i.test(model.description),
|
|
111
|
+
))
|
|
112
|
+
if (frontier && balanced) return [frontier, balanced]
|
|
113
|
+
|
|
103
114
|
const versions = pool
|
|
104
115
|
.map(model => versionParts(model.id))
|
|
105
116
|
.filter((parts): parts is number[] => !!parts)
|
|
@@ -144,7 +155,7 @@ export function buildCodexModelCatalog(
|
|
|
144
155
|
source,
|
|
145
156
|
refreshedAt,
|
|
146
157
|
autoUpdates: true,
|
|
147
|
-
selectionPolicy: 'newest-generation
|
|
158
|
+
selectionPolicy: 'capability-tiers-then-newest-generation',
|
|
148
159
|
options,
|
|
149
160
|
}
|
|
150
161
|
}
|
|
@@ -163,7 +174,7 @@ function cliDefaultCatalog(refreshError?: string): CodexModelCatalog {
|
|
|
163
174
|
source: 'cli-default',
|
|
164
175
|
refreshedAt: new Date().toISOString(),
|
|
165
176
|
autoUpdates: true,
|
|
166
|
-
selectionPolicy: 'newest-generation
|
|
177
|
+
selectionPolicy: 'capability-tiers-then-newest-generation',
|
|
167
178
|
options: [
|
|
168
179
|
{ ...common, preference: CODEX_FRONTIER_MODEL, displayName: 'GPT Frontier (auto)' },
|
|
169
180
|
{ ...common, preference: CODEX_BALANCED_MODEL, displayName: 'GPT Balanced (auto)', isDefault: false },
|
|
@@ -0,0 +1,117 @@
|
|
|
1
|
+
// Dictation speech-to-text (6.73.0): prompts and dictation follow the STT
|
|
2
|
+
// settings. The contract and the evidence behind every rule here:
|
|
3
|
+
// docs/dictation-stt-contract.md.
|
|
4
|
+
//
|
|
5
|
+
// live preview (peek) and Fast mode -> the Preview model
|
|
6
|
+
// final text (Fast mode off) -> the HQ model
|
|
7
|
+
//
|
|
8
|
+
// Only the Parakeet branch is new. Every Whisper path is the 6.72.0 code,
|
|
9
|
+
// unchanged, and it is also the fallback: a Parakeet decode that is cold,
|
|
10
|
+
// failed, busy or behind an open breaker leaves the request on Whisper with a
|
|
11
|
+
// reason code, so the user never sees a blank or an error because of it.
|
|
12
|
+
// The engine is chosen once when a decode starts and never changes during it.
|
|
13
|
+
// Second opinion and resolver never run here. No LLM is ever called.
|
|
14
|
+
|
|
15
|
+
import { transcribeParakeet, ParakeetError } from './parakeet.js'
|
|
16
|
+
import { asrFrontEnd } from './stt-live.js'
|
|
17
|
+
import {
|
|
18
|
+
currentSttCapabilities, ensurePreviewWarm, noteParakeetDictationHit, noteParakeetDictationMiss,
|
|
19
|
+
PARAKEET_PREVIEW_TIMEOUT_MS, PREVIEW_PARAKEET_MODEL,
|
|
20
|
+
} from './stt-runtime.js'
|
|
21
|
+
import {
|
|
22
|
+
dictationPlan, readSttSettings, type DictationPlan, type DictationSlot, type FrontEndId,
|
|
23
|
+
} from './stt-settings.js'
|
|
24
|
+
|
|
25
|
+
/** The plan for a dictation decode starting now. A Parakeet selection that is
|
|
26
|
+
* not warm yet starts a warm-up (never awaited) and decodes this one on Whisper. */
|
|
27
|
+
export function currentDictationPlan(slot: DictationSlot): DictationPlan {
|
|
28
|
+
let plan: DictationPlan
|
|
29
|
+
try {
|
|
30
|
+
const settings = readSttSettings()
|
|
31
|
+
plan = dictationPlan(slot, settings, currentSttCapabilities())
|
|
32
|
+
if (plan.reason === 'parakeet_warming' || plan.reason === 'parakeet_warm_failed') ensurePreviewWarm()
|
|
33
|
+
} catch (error) {
|
|
34
|
+
console.warn(`[stt] dictation settings unreadable, using Whisper: ${error instanceof Error ? error.message : error}`)
|
|
35
|
+
plan = { slot, engine: 'whisper', reason: null, frontEnd: 'highpass', key: slot === 'final' ? 'final:large-v3' : `${slot}:turbo` }
|
|
36
|
+
}
|
|
37
|
+
return plan
|
|
38
|
+
}
|
|
39
|
+
|
|
40
|
+
/** Parakeet reads 16 kHz mono PCM16 WAV. The browser recorder posts WebM, which stays on Whisper. */
|
|
41
|
+
export function isPcm16kMonoWav(audio: Buffer): boolean {
|
|
42
|
+
if (audio.length < 44 || audio.toString('ascii', 0, 4) !== 'RIFF' || audio.toString('ascii', 8, 12) !== 'WAVE') return false
|
|
43
|
+
let offset = 12
|
|
44
|
+
while (offset + 8 <= audio.length) {
|
|
45
|
+
const id = audio.toString('ascii', offset, offset + 4)
|
|
46
|
+
const size = audio.readUInt32LE(offset + 4)
|
|
47
|
+
const body = offset + 8
|
|
48
|
+
if (id === 'fmt ') {
|
|
49
|
+
return body + 16 <= audio.length
|
|
50
|
+
&& audio.readUInt16LE(body) === 1 && audio.readUInt16LE(body + 2) === 1
|
|
51
|
+
&& audio.readUInt32LE(body + 4) === 16_000 && audio.readUInt16LE(body + 14) === 16
|
|
52
|
+
}
|
|
53
|
+
offset = body + size + (size % 2)
|
|
54
|
+
}
|
|
55
|
+
return false
|
|
56
|
+
}
|
|
57
|
+
|
|
58
|
+
/** A warm Parakeet decodes 6 s in about 0.3 s. The budget is generous so a busy
|
|
59
|
+
* Neural Engine is not a miss, and bounded so a hang falls to Whisper quickly. */
|
|
60
|
+
export function parakeetDictationTimeoutMs(audio: Buffer): number {
|
|
61
|
+
const seconds = Math.max(0, (audio.length - 44) / 32_000)
|
|
62
|
+
return Math.min(15_000, Math.round(2_000 + 150 * seconds))
|
|
63
|
+
}
|
|
64
|
+
|
|
65
|
+
export type DictationLane = 'preview' | 'interactive'
|
|
66
|
+
|
|
67
|
+
export interface DictationParakeetDeps {
|
|
68
|
+
parakeet: (audio: Buffer, lane: DictationLane, timeoutMs: number) => Promise<{ text: string }>
|
|
69
|
+
frontEnd: (audio: Buffer, frontEnd: FrontEndId) => Promise<Buffer>
|
|
70
|
+
hit: () => void
|
|
71
|
+
miss: () => void
|
|
72
|
+
}
|
|
73
|
+
|
|
74
|
+
export const defaultDictationParakeetDeps: DictationParakeetDeps = {
|
|
75
|
+
parakeet: (audio, lane, timeoutMs) => transcribeParakeet(audio, { model: PREVIEW_PARAKEET_MODEL, lane, timeoutMs }),
|
|
76
|
+
frontEnd: (audio, fe) => asrFrontEnd(audio, fe),
|
|
77
|
+
hit: () => noteParakeetDictationHit(),
|
|
78
|
+
miss: () => noteParakeetDictationMiss(),
|
|
79
|
+
}
|
|
80
|
+
|
|
81
|
+
export type DictationParakeetOutcome =
|
|
82
|
+
| { ok: true; text: string }
|
|
83
|
+
| { ok: false; reason: 'parakeet_busy' | 'parakeet_failed' | 'audio_not_pcm_wav' }
|
|
84
|
+
|
|
85
|
+
/** Applies the front end only on the Parakeet branch (see the contract). */
|
|
86
|
+
export const DICTATION_PARAKEET_FRONT_END = true
|
|
87
|
+
|
|
88
|
+
/**
|
|
89
|
+
* One Parakeet decode for a dictation request. Never throws: any failure is an
|
|
90
|
+
* outcome the caller turns into its 6.72.0 Whisper decode of the same audio.
|
|
91
|
+
* Raw text only; the caller runs the same corrections and filters as Whisper text.
|
|
92
|
+
*/
|
|
93
|
+
export async function runParakeetDictation(
|
|
94
|
+
audio: Buffer,
|
|
95
|
+
plan: DictationPlan,
|
|
96
|
+
lane: DictationLane,
|
|
97
|
+
deps: DictationParakeetDeps = defaultDictationParakeetDeps,
|
|
98
|
+
): Promise<DictationParakeetOutcome> {
|
|
99
|
+
if (plan.engine !== 'parakeet-ultra') return { ok: false, reason: 'parakeet_failed' }
|
|
100
|
+
if (!isPcm16kMonoWav(audio)) return { ok: false, reason: 'audio_not_pcm_wav' }
|
|
101
|
+
const timeoutMs = lane === 'preview' ? PARAKEET_PREVIEW_TIMEOUT_MS : parakeetDictationTimeoutMs(audio)
|
|
102
|
+
let input = audio
|
|
103
|
+
if (DICTATION_PARAKEET_FRONT_END) {
|
|
104
|
+
// A front-end failure is not a Parakeet miss: decode the raw audio instead.
|
|
105
|
+
try { input = await deps.frontEnd(audio, plan.frontEnd) } catch { input = audio }
|
|
106
|
+
}
|
|
107
|
+
try {
|
|
108
|
+
const result = await deps.parakeet(input, lane, timeoutMs)
|
|
109
|
+
deps.hit()
|
|
110
|
+
return { ok: true, text: result.text }
|
|
111
|
+
} catch (error) {
|
|
112
|
+
const busy = error instanceof ParakeetError ? error.reason === 'busy' : (error as { reason?: string })?.reason === 'busy'
|
|
113
|
+
if (!busy) deps.miss()
|
|
114
|
+
console.warn(`[stt] Parakeet dictation ${plan.slot} ${busy ? 'busy' : 'missed'}; Whisper decodes it: ${error instanceof Error ? error.message : error}`)
|
|
115
|
+
return { ok: false, reason: busy ? 'parakeet_busy' : 'parakeet_failed' }
|
|
116
|
+
}
|
|
117
|
+
}
|
package/server/lib/parakeet.ts
CHANGED
|
@@ -383,8 +383,10 @@ type SpawnLike = (command: string, args: string[], options: SpawnOptions) => Chi
|
|
|
383
383
|
|
|
384
384
|
export interface ParakeetRunOptions {
|
|
385
385
|
model: ParakeetModelId
|
|
386
|
-
/** preview: one at a time, a busy lane is a miss. batch: queued, lower CPU priority.
|
|
387
|
-
|
|
386
|
+
/** preview: one at a time, a busy lane is a miss. batch: queued, lower CPU priority.
|
|
387
|
+
* interactive (6.73.0, dictation final and Fast mode): normal priority, never queued
|
|
388
|
+
* behind a meeting batch, at most INTERACTIVE_LANE_MAX at once; beyond that it is busy. */
|
|
389
|
+
lane: 'preview' | 'batch' | 'warmup' | 'interactive'
|
|
388
390
|
timeoutMs: number
|
|
389
391
|
signal?: AbortSignal
|
|
390
392
|
spawnImpl?: SpawnLike
|
|
@@ -510,6 +512,9 @@ export function padShortWav(wav: Buffer, minSec = MIN_PARAKEET_AUDIO_SEC): Buffe
|
|
|
510
512
|
|
|
511
513
|
let previewBusy = false
|
|
512
514
|
let batchTail: Promise<unknown> = Promise.resolve()
|
|
515
|
+
/** A prompt's Fast warm and HQ warm can overlap for one chunk; a third waits for Whisper. */
|
|
516
|
+
export const INTERACTIVE_LANE_MAX = 2
|
|
517
|
+
let interactiveRunning = 0
|
|
513
518
|
|
|
514
519
|
export function transcribeParakeet(wav: Buffer, options: ParakeetRunOptions): Promise<ParakeetResult> {
|
|
515
520
|
if (options.lane === 'preview') {
|
|
@@ -517,6 +522,11 @@ export function transcribeParakeet(wav: Buffer, options: ParakeetRunOptions): Pr
|
|
|
517
522
|
previewBusy = true
|
|
518
523
|
return runParakeet(wav, options).finally(() => { previewBusy = false })
|
|
519
524
|
}
|
|
525
|
+
if (options.lane === 'interactive') {
|
|
526
|
+
if (interactiveRunning >= INTERACTIVE_LANE_MAX) return Promise.reject(new ParakeetError('busy', 'the interactive lane is full'))
|
|
527
|
+
interactiveRunning++
|
|
528
|
+
return runParakeet(wav, options).finally(() => { interactiveRunning-- })
|
|
529
|
+
}
|
|
520
530
|
if (options.lane === 'batch') {
|
|
521
531
|
const job = batchTail.then(() => runParakeet(wav, options))
|
|
522
532
|
batchTail = job.catch(() => undefined)
|
|
@@ -581,7 +591,7 @@ function runParakeetChecked(wav: Buffer, options: ParakeetRunOptions): Promise<P
|
|
|
581
591
|
options.signal?.addEventListener('abort', onAbort, { once: true })
|
|
582
592
|
liveChildren.add(child)
|
|
583
593
|
child.once('close', () => liveChildren.delete(child))
|
|
584
|
-
if (child.pid && options.lane !== 'preview') {
|
|
594
|
+
if (child.pid && options.lane !== 'preview' && options.lane !== 'interactive') {
|
|
585
595
|
try { setPriority(child.pid, 10) } catch { /* runs at normal priority */ }
|
|
586
596
|
}
|
|
587
597
|
let stderr = ''
|
|
@@ -659,7 +669,7 @@ export function warmParakeet(model: ParakeetModelId, spawnImpl?: SpawnLike): Pro
|
|
|
659
669
|
}
|
|
660
670
|
|
|
661
671
|
export function __resetParakeetForTests(): void {
|
|
662
|
-
warm.clear(); warming.clear(); previewBusy = false; batchTail = Promise.resolve(); installing = null
|
|
672
|
+
warm.clear(); warming.clear(); previewBusy = false; interactiveRunning = 0; batchTail = Promise.resolve(); installing = null
|
|
663
673
|
progress = { state: 'idle', bytesDone: 0, bytesTotal: TOTAL_BYTES, error: null }
|
|
664
674
|
installedCache = null
|
|
665
675
|
}
|
|
@@ -22,6 +22,11 @@ export interface PromptDraftTranscriptRecord {
|
|
|
22
22
|
degraded: boolean
|
|
23
23
|
degradationReason?: string
|
|
24
24
|
acceptedDegraded?: boolean
|
|
25
|
+
/** 6.73.0: the setting this decode followed ("fast:parakeet", "final:large-v3").
|
|
26
|
+
* Absent on older records, which were all made under the 6.72.0 defaults. */
|
|
27
|
+
sttKey?: string
|
|
28
|
+
engine?: 'whisper' | 'parakeet-ultra'
|
|
29
|
+
engineReason?: string
|
|
25
30
|
}
|
|
26
31
|
|
|
27
32
|
/** Higher wins. Fast warm must never replace an HQ/cloud decode for the same audio. */
|
|
@@ -59,6 +59,11 @@ function notePreviewMiss(now = Date.now()): void {
|
|
|
59
59
|
}
|
|
60
60
|
function notePreviewHit(): void { breaker = { misses: 0, openUntil: 0, backoffMs: PREVIEW_BREAKER_BASE_MS } }
|
|
61
61
|
|
|
62
|
+
/** 6.73.0: dictation Parakeet decodes share the preview breaker, so a Parakeet that keeps
|
|
63
|
+
* failing stops being tried for prompts and meetings alike. A busy lane is not a miss. */
|
|
64
|
+
export function noteParakeetDictationMiss(now = Date.now()): void { notePreviewMiss(now) }
|
|
65
|
+
export function noteParakeetDictationHit(): void { notePreviewHit() }
|
|
66
|
+
|
|
62
67
|
export function __resetPreviewForTests(): void {
|
|
63
68
|
for (const k of Object.keys(previewStats) as Array<keyof typeof previewStats>) previewStats[k] = 0
|
|
64
69
|
breaker = { misses: 0, openUntil: 0, backoffMs: PREVIEW_BREAKER_BASE_MS }
|
|
@@ -80,6 +85,7 @@ export function currentSttCapabilities(): SttCapabilities {
|
|
|
80
85
|
parakeetPreviewWarmFailed: installed && warm === 'failed',
|
|
81
86
|
parakeetPreviewBreakerOpen: installed && previewBreakerState().open,
|
|
82
87
|
previewDisabledByEnv: process.env.COS_WHISPER_MEETING_PREVIEW === '0',
|
|
88
|
+
dictationDisabledByEnv: process.env.COS_DICTATION_FOLLOWS_STT === '0',
|
|
83
89
|
}
|
|
84
90
|
}
|
|
85
91
|
|
|
@@ -132,7 +138,9 @@ export const WARM_RETRY_MS = 120_000
|
|
|
132
138
|
*/
|
|
133
139
|
export function ensurePreviewWarm(now = Date.now()): 'started' | 'skipped' | 'deferred' {
|
|
134
140
|
const settings = readSttSettings()
|
|
135
|
-
|
|
141
|
+
// 6.73.0: a Parakeet HQ model also warms, because dictation final text uses it only warm.
|
|
142
|
+
const wanted = settings.preview === 'parakeet' || settings.hq === 'parakeet-ultra'
|
|
143
|
+
if (!wanted || !parakeetInstalled().installed) return 'skipped'
|
|
136
144
|
const w = parakeetWarmState(PREVIEW_PARAKEET_MODEL)
|
|
137
145
|
if (w.state === 'warm' || w.state === 'warming') return 'skipped'
|
|
138
146
|
if (w.state === 'failed' && now - w.at < WARM_RETRY_MS) return 'skipped'
|
|
@@ -144,7 +152,7 @@ export function ensurePreviewWarm(now = Date.now()): 'started' | 'skipped' | 'de
|
|
|
144
152
|
}
|
|
145
153
|
|
|
146
154
|
let warmTimer: ReturnType<typeof setInterval> | null = null
|
|
147
|
-
/** Server start: one try now, then every minute (a no-op unless Parakeet preview is selected and cold). */
|
|
155
|
+
/** Server start: one try now, then every minute (a no-op unless a Parakeet preview or HQ model is selected and cold). */
|
|
148
156
|
export function startPreviewWarmLoop(): void {
|
|
149
157
|
ensurePreviewWarm()
|
|
150
158
|
if (warmTimer) return
|
|
@@ -130,6 +130,8 @@ export interface SttCapabilities {
|
|
|
130
130
|
/** Repeated preview misses opened the preview breaker: Turbo until it closes. */
|
|
131
131
|
parakeetPreviewBreakerOpen?: boolean
|
|
132
132
|
previewDisabledByEnv: boolean
|
|
133
|
+
/** 6.73.0: COS_DICTATION_FOLLOWS_STT=0 keeps every dictation decode on its 6.72.0 Whisper engine. */
|
|
134
|
+
dictationDisabledByEnv?: boolean
|
|
133
135
|
}
|
|
134
136
|
|
|
135
137
|
export interface SttOption {
|
|
@@ -153,7 +155,7 @@ export function previewOptions(c: SttCapabilities): SttOption[] {
|
|
|
153
155
|
: c.parakeetPreviewBreakerOpen
|
|
154
156
|
? 'Parakeet missed several previews in a row; Whisper Turbo previews for a few minutes, then Parakeet is tried again.'
|
|
155
157
|
: c.parakeetPreviewWarm
|
|
156
|
-
? 'Parakeet Ultra on the Neural Engine, about 0.3 s per chunk. Experimental: compare it live yourself.
|
|
158
|
+
? 'Parakeet Ultra on the Neural Engine, about 0.3 s per chunk. Experimental: compare it live yourself. Meeting previews need COS Glasses 6.10.639 (older phones keep Whisper Turbo there). Dictation previews and Fast mode use it on every phone.'
|
|
157
159
|
: c.parakeetPreviewWarmFailed
|
|
158
160
|
? 'Parakeet could not start; Whisper Turbo previews. It is tried again every 2 minutes.'
|
|
159
161
|
: c.parakeetPreviewWarming
|
|
@@ -306,6 +308,92 @@ export function effectiveStt(settings: SttSettings, c: SttCapabilities): Effecti
|
|
|
306
308
|
}
|
|
307
309
|
}
|
|
308
310
|
|
|
311
|
+
// ── Dictation (6.73.0) ────────────────────────────────────────────────────────
|
|
312
|
+
//
|
|
313
|
+
// Prompting and dictation follow the same choices (docs/dictation-stt-contract.md):
|
|
314
|
+
// the live preview and Fast mode follow the Preview model, the final text follows
|
|
315
|
+
// the HQ model. Second opinion and resolver never run on dictation.
|
|
316
|
+
|
|
317
|
+
export type DictationSlot = 'preview' | 'fast' | 'final'
|
|
318
|
+
export type DictationEngine = 'whisper' | 'parakeet-ultra'
|
|
319
|
+
|
|
320
|
+
export interface DictationPlan {
|
|
321
|
+
slot: DictationSlot
|
|
322
|
+
engine: DictationEngine
|
|
323
|
+
/** Why the selection could not run, as a REASON_TEXT code; null when it runs as chosen. */
|
|
324
|
+
reason: string | null
|
|
325
|
+
/** What a Parakeet decode hears. Whisper dictation always decodes raw audio. */
|
|
326
|
+
frontEnd: FrontEndId
|
|
327
|
+
/** The decode path this follows: "preview:" or "fast:" plus turbo or parakeet, or
|
|
328
|
+
* "final:" plus large-v3 or parakeet-ultra. A cached decode is reused only under the same key. */
|
|
329
|
+
key: string
|
|
330
|
+
}
|
|
331
|
+
|
|
332
|
+
/** Which slot a decode is in: Fast mode follows the Preview model, otherwise the HQ model. */
|
|
333
|
+
export function dictationSlotForMode(mode: 'hq' | 'fast'): DictationSlot {
|
|
334
|
+
return mode === 'fast' ? 'fast' : 'final'
|
|
335
|
+
}
|
|
336
|
+
|
|
337
|
+
/** The 6.72.0 key every pre-6.73 cached decode was made under. */
|
|
338
|
+
export function legacyDictationKey(mode: 'hq' | 'fast'): string {
|
|
339
|
+
return mode === 'fast' ? `fast:${DEFAULT_STT_SETTINGS.preview}` : `final:${DEFAULT_STT_SETTINGS.hq}`
|
|
340
|
+
}
|
|
341
|
+
|
|
342
|
+
/** Pure: what a dictation decode in this slot will use, given the settings and this Mac. */
|
|
343
|
+
export function dictationPlan(slot: DictationSlot, settings: SttSettings, c: SttCapabilities): DictationPlan {
|
|
344
|
+
if (c.dictationDisabledByEnv) {
|
|
345
|
+
return { slot, engine: 'whisper', reason: 'dictation_disabled_by_env', frontEnd: settings.frontEnd, key: slot === 'final' ? 'final:large-v3' : `${slot}:turbo` }
|
|
346
|
+
}
|
|
347
|
+
const wantsParakeet = slot === 'final' ? settings.hq === 'parakeet-ultra' : settings.preview === 'parakeet'
|
|
348
|
+
// The key names the dictation decode path, not the raw id: every Whisper choice in a
|
|
349
|
+
// slot is the same 6.72.0 decode for a prompt, so switching among them re-decodes nothing.
|
|
350
|
+
const key = slot === 'final' ? (wantsParakeet ? 'final:parakeet-ultra' : 'final:large-v3') : `${slot}:${wantsParakeet ? 'parakeet' : 'turbo'}`
|
|
351
|
+
const base = { slot, frontEnd: settings.frontEnd, key }
|
|
352
|
+
if (!wantsParakeet) {
|
|
353
|
+
// large-v3 and nemotron-560 previews are not offered; dictation keeps its Whisper engine.
|
|
354
|
+
const reason = slot !== 'final' && settings.preview === 'large-v3' ? 'large_v3_preview_not_offered'
|
|
355
|
+
: slot !== 'final' && settings.preview === 'nemotron-560' ? 'nemotron_not_wired' : null
|
|
356
|
+
return { ...base, engine: 'whisper', reason }
|
|
357
|
+
}
|
|
358
|
+
const reason = !c.parakeetInstalled ? 'parakeet_not_installed'
|
|
359
|
+
: c.parakeetPreviewBreakerOpen ? 'parakeet_breaker_open'
|
|
360
|
+
: c.parakeetPreviewWarm ? null
|
|
361
|
+
: c.parakeetPreviewWarmFailed ? 'parakeet_warm_failed'
|
|
362
|
+
: 'parakeet_warming'
|
|
363
|
+
return { ...base, engine: reason ? 'whisper' : 'parakeet-ultra', reason }
|
|
364
|
+
}
|
|
365
|
+
|
|
366
|
+
export const DICTATION_REASON_TEXT: Record<string, string> = {
|
|
367
|
+
parakeet_busy: 'Parakeet was busy with another decode; Whisper decoded this one.',
|
|
368
|
+
parakeet_failed: 'Parakeet failed on this audio; Whisper decoded it instead.',
|
|
369
|
+
audio_not_pcm_wav: 'This audio is not 16 kHz WAV, which Parakeet needs; Whisper decoded it.',
|
|
370
|
+
dictation_disabled_by_env: 'Dictation is held on Whisper on this server (COS_DICTATION_FOLLOWS_STT=0).',
|
|
371
|
+
}
|
|
372
|
+
|
|
373
|
+
function dictationSlotView(plan: DictationPlan, follows: 'preview' | 'hq') {
|
|
374
|
+
const code = plan.reason
|
|
375
|
+
return { engine: plan.engine, follows, reason: code ? (REASON_TEXT[code] ?? DICTATION_REASON_TEXT[code] ?? code) : null, reasonCode: code }
|
|
376
|
+
}
|
|
377
|
+
|
|
378
|
+
export function dictationView(settings: SttSettings, c: SttCapabilities) {
|
|
379
|
+
const preview = dictationPlan('preview', settings, c)
|
|
380
|
+
const fast = dictationPlan('fast', settings, c)
|
|
381
|
+
const final = dictationPlan('final', settings, c)
|
|
382
|
+
const anyParakeet = settings.preview === 'parakeet' || settings.hq === 'parakeet-ultra'
|
|
383
|
+
const name = (p: DictationPlan) => p.engine === 'parakeet-ultra' ? 'Parakeet Ultra' : 'Whisper'
|
|
384
|
+
const note = `Dictation follows these settings: the live preview and Fast mode use the Preview model (now ${name(preview)}), the final text uses the HQ model (now ${name(final)}). `
|
|
385
|
+
+ 'Dictation never runs the second opinion or the resolver, and the quiet gate never drops a spoken prompt.'
|
|
386
|
+
return {
|
|
387
|
+
preview: dictationSlotView(preview, 'preview'),
|
|
388
|
+
fast: dictationSlotView(fast, 'preview'),
|
|
389
|
+
final: dictationSlotView(final, 'hq'),
|
|
390
|
+
frontEnd: anyParakeet ? settings.frontEnd : 'raw' as const,
|
|
391
|
+
quietGate: 'none' as const,
|
|
392
|
+
secondOpinion: 'off' as const,
|
|
393
|
+
note,
|
|
394
|
+
}
|
|
395
|
+
}
|
|
396
|
+
|
|
309
397
|
export interface ParakeetStatus {
|
|
310
398
|
installed: boolean
|
|
311
399
|
installable: boolean
|
|
@@ -337,6 +425,7 @@ export function settingsView(settings: SttSettings, c: SttCapabilities, parakeet
|
|
|
337
425
|
quietGate: { selected: settings.quietGate, options: QUIET_GATE_OPTIONS },
|
|
338
426
|
parakeet,
|
|
339
427
|
lastJob,
|
|
340
|
-
|
|
428
|
+
dictation: dictationView(settings, c),
|
|
429
|
+
appliesTo: { preview: 'next live chunk', hq: 'next saved meeting', dictation: 'next dictation chunk' },
|
|
341
430
|
}
|
|
342
431
|
}
|
|
@@ -22,6 +22,8 @@ import {
|
|
|
22
22
|
} from './hallucination-filter.js'
|
|
23
23
|
import { getOpenAIKey, tryGetOpenAIKey } from './openai-key.js'
|
|
24
24
|
import { getTranscriptionPolicySnapshot, isOpenAIWhisperFallbackReady } from './transcription-policy.js'
|
|
25
|
+
import { currentDictationPlan, runParakeetDictation, type DictationParakeetOutcome } from './dictation-stt.js'
|
|
26
|
+
import { dictationSlotForMode, type DictationEngine, type DictationPlan, type DictationSlot } from './stt-settings.js'
|
|
25
27
|
|
|
26
28
|
export { OpenAIWhisperBudgetExhaustedError, estimateAudioSeconds }
|
|
27
29
|
|
|
@@ -39,6 +41,22 @@ export interface TranscribeAudioResult {
|
|
|
39
41
|
degradationReason?: string
|
|
40
42
|
elapsedMs: number
|
|
41
43
|
audioBytes: number
|
|
44
|
+
/** 6.73.0: the engine that decoded this audio. */
|
|
45
|
+
engine: DictationEngine
|
|
46
|
+
/** Why the selected engine did not run (a reason code), when it did not. */
|
|
47
|
+
engineReason?: string
|
|
48
|
+
/** The setting this decode followed (see DictationPlan.key). */
|
|
49
|
+
dictationKey: string
|
|
50
|
+
}
|
|
51
|
+
|
|
52
|
+
/** Injected by tests; production reads the settings and runs Parakeet. */
|
|
53
|
+
export interface DictationSttDeps {
|
|
54
|
+
plan: (slot: DictationSlot) => DictationPlan
|
|
55
|
+
parakeet: (audio: Buffer, plan: DictationPlan) => Promise<DictationParakeetOutcome>
|
|
56
|
+
}
|
|
57
|
+
export const defaultDictationSttDeps: DictationSttDeps = {
|
|
58
|
+
plan: slot => currentDictationPlan(slot),
|
|
59
|
+
parakeet: (audio, plan) => runParakeetDictation(audio, plan, 'interactive'),
|
|
42
60
|
}
|
|
43
61
|
|
|
44
62
|
export type TranscriptionBackendPolicy = 'automatic' | 'local-only'
|
|
@@ -147,6 +165,9 @@ export async function transcribeAudioBuffer(
|
|
|
147
165
|
policy?: TranscriptionBackendPolicy
|
|
148
166
|
/** When false, turbo/server failures do not move the shared meeting circuit breaker. */
|
|
149
167
|
affectsCircuit?: boolean
|
|
168
|
+
/** A plan snapshot taken by the caller (prompt finalize takes one for every chunk). */
|
|
169
|
+
dictationPlan?: DictationPlan
|
|
170
|
+
sttDeps?: DictationSttDeps
|
|
150
171
|
} = {},
|
|
151
172
|
): Promise<TranscribeAudioResult> {
|
|
152
173
|
const requestedMode = opts.mode ?? 'hq'
|
|
@@ -160,14 +181,39 @@ export async function transcribeAudioBuffer(
|
|
|
160
181
|
console.log(`[transcribe] mode downgrade: hq → fast (audio ${audioSeconds.toFixed(1)}s > cap ${HQ_MAX_SECONDS}s)`)
|
|
161
182
|
}
|
|
162
183
|
|
|
163
|
-
let text
|
|
184
|
+
let text!: string
|
|
164
185
|
let words: WhisperWord[] | undefined
|
|
165
|
-
let backend
|
|
166
|
-
let actualQuality
|
|
186
|
+
let backend!: string
|
|
187
|
+
let actualQuality!: 'hq' | 'fast' | 'cloud'
|
|
167
188
|
let degradationReason: string | undefined = effectiveMode !== requestedMode ? 'audio_too_long' : undefined
|
|
168
189
|
const tStart = performance.now()
|
|
169
190
|
|
|
170
|
-
|
|
191
|
+
// 6.73.0: Fast mode follows the Preview model, otherwise the HQ model. One plan
|
|
192
|
+
// per decode; a long clip downgraded to Fast is planned for the Fast slot.
|
|
193
|
+
const sttDeps = opts.sttDeps ?? defaultDictationSttDeps
|
|
194
|
+
const slot = dictationSlotForMode(effectiveMode)
|
|
195
|
+
const plan = opts.dictationPlan && opts.dictationPlan.slot === slot ? opts.dictationPlan : sttDeps.plan(slot)
|
|
196
|
+
let engine: DictationEngine = 'whisper'
|
|
197
|
+
let engineReason: string | null = plan.reason
|
|
198
|
+
let decoded = false
|
|
199
|
+
if (plan.engine === 'parakeet-ultra') {
|
|
200
|
+
const outcome = await sttDeps.parakeet(audioBuffer, plan)
|
|
201
|
+
if (outcome.ok) {
|
|
202
|
+
text = outcome.text
|
|
203
|
+
backend = 'parakeet-ultra'
|
|
204
|
+
actualQuality = effectiveMode
|
|
205
|
+
engine = 'parakeet-ultra'
|
|
206
|
+
engineReason = null
|
|
207
|
+
decoded = true
|
|
208
|
+
} else {
|
|
209
|
+
engineReason = outcome.reason
|
|
210
|
+
}
|
|
211
|
+
}
|
|
212
|
+
|
|
213
|
+
if (decoded) {
|
|
214
|
+
// Parakeet ran: nothing below. Its word clocks are not passed to the prompt tail
|
|
215
|
+
// guard, whose confidence rule is calibrated on Whisper token probabilities.
|
|
216
|
+
} else if (effectiveMode === 'hq') {
|
|
171
217
|
try {
|
|
172
218
|
// A0 (2026-07-30): ffmpeg enhance light (highpass=f=80) was measured dropping
|
|
173
219
|
// leading speech on compose ("device just for your awareness"). Meeting batch
|
|
@@ -286,5 +332,8 @@ export async function transcribeAudioBuffer(
|
|
|
286
332
|
...(degradationReason ? { degradationReason } : {}),
|
|
287
333
|
elapsedMs,
|
|
288
334
|
audioBytes: audioBuffer.length,
|
|
335
|
+
engine,
|
|
336
|
+
...(engineReason ? { engineReason } : {}),
|
|
337
|
+
dictationKey: plan.key,
|
|
289
338
|
}
|
|
290
339
|
}
|
|
@@ -40,6 +40,8 @@ import { applyCorrections } from '../lib/whisper-local.js'
|
|
|
40
40
|
import { applyFuzzyCorrections } from '../lib/fuzzy-correct.js'
|
|
41
41
|
import { getAllSpeakerNames } from '../lib/speaker-embeddings.js'
|
|
42
42
|
import { transcribeWhisperPreview } from '../lib/whisper-preview.js'
|
|
43
|
+
import { currentDictationPlan, runParakeetDictation } from '../lib/dictation-stt.js'
|
|
44
|
+
import { dictationSlotForMode, legacyDictationKey, type DictationPlan } from '../lib/stt-settings.js'
|
|
43
45
|
import { autoCleanDictation, AUTOCLEAN_MAX_CHARS } from '../lib/dictation-clean.js'
|
|
44
46
|
import { getVocabulary } from '../lib/profile.js'
|
|
45
47
|
import { createBreaker } from '../lib/claude-circuit.js'
|
|
@@ -240,14 +242,25 @@ async function sendDraftError(res: Response, draftId: string, err: any): Promise
|
|
|
240
242
|
res.status(err.status ?? 500).json({ error: err.message })
|
|
241
243
|
}
|
|
242
244
|
|
|
243
|
-
|
|
244
|
-
|
|
245
|
+
// 6.73.0: the setting key is part of every decode key, so a decode made under
|
|
246
|
+
// one STT selection is never shared with or reused for another.
|
|
247
|
+
function modeQualityKey(draftId: string, chunkIndex: number, mode: 'hq' | 'fast', hash: string, sttKey: string): string {
|
|
248
|
+
return `${draftId}:${chunkIndex}:${mode}:${hash}:${sttKey}`
|
|
245
249
|
}
|
|
246
250
|
|
|
247
|
-
|
|
251
|
+
/** The setting key a stored record was made under (pre-6.73 records: the 6.72.0 defaults). */
|
|
252
|
+
export function recordSttKey(record: PromptDraftTranscriptRecord): string {
|
|
253
|
+
return record.sttKey ?? legacyDictationKey(record.requestedMode)
|
|
254
|
+
}
|
|
255
|
+
|
|
256
|
+
function planFor(mode: 'hq' | 'fast'): DictationPlan {
|
|
257
|
+
return currentDictationPlan(dictationSlotForMode(mode))
|
|
258
|
+
}
|
|
259
|
+
|
|
260
|
+
async function transcribeChunk(draftId: string, chunkIndex: number, audio: Buffer, mode: 'hq' | 'fast', purpose: 'warm' | 'final', plan: DictationPlan = planFor(mode)): Promise<string> {
|
|
248
261
|
const hash = audioHash(audio)
|
|
249
|
-
const purposeKey = `${draftId}:${chunkIndex}:${purpose}:${mode}:${hash}`
|
|
250
|
-
const sharedKey = modeQualityKey(draftId, chunkIndex, mode, hash)
|
|
262
|
+
const purposeKey = `${draftId}:${chunkIndex}:${purpose}:${mode}:${hash}:${plan.key}`
|
|
263
|
+
const sharedKey = modeQualityKey(draftId, chunkIndex, mode, hash, plan.key)
|
|
251
264
|
|
|
252
265
|
// HQ warm and HQ finalize must share one decode (plan step 8a).
|
|
253
266
|
const existingShared = modeQualityJobs.get(sharedKey)
|
|
@@ -276,6 +289,9 @@ async function transcribeChunk(draftId: string, chunkIndex: number, audio: Buffe
|
|
|
276
289
|
backend: warmMatches ? warm!.backend : 'shared-inflight',
|
|
277
290
|
degraded: warmMatches ? warm!.degraded : false,
|
|
278
291
|
...(warmMatches && warm!.degradationReason ? { degradationReason: warm!.degradationReason } : {}),
|
|
292
|
+
sttKey: plan.key,
|
|
293
|
+
...(warmMatches && warm!.engine ? { engine: warm!.engine } : {}),
|
|
294
|
+
...(warmMatches && warm!.engineReason ? { engineReason: warm!.engineReason } : {}),
|
|
279
295
|
}
|
|
280
296
|
if (
|
|
281
297
|
!cachedFinal
|
|
@@ -298,7 +314,7 @@ async function transcribeChunk(draftId: string, chunkIndex: number, audio: Buffe
|
|
|
298
314
|
try {
|
|
299
315
|
// Speculative warm is always local-only. Finalize may use automatic cloud fallback.
|
|
300
316
|
const policy = purpose === 'warm' ? 'local-only' as const : 'automatic' as const
|
|
301
|
-
const result = await transcribeAudioBuffer(audio, { mode, policy })
|
|
317
|
+
const result = await transcribeAudioBuffer(audio, { mode, policy, dictationPlan: plan })
|
|
302
318
|
if (!isCurrentChunk(draftId, chunkIndex, audio)) return ''
|
|
303
319
|
// 6.45.4 — the sentence Whisper invents after the speaker stops. Never trims
|
|
304
320
|
// audio; drops a trailing sentence only on the junk lexicon or on silence
|
|
@@ -313,9 +329,11 @@ async function transcribeChunk(draftId: string, chunkIndex: number, audio: Buffe
|
|
|
313
329
|
text, hash, requestedMode: result.requestedMode, actualQuality: result.actualQuality,
|
|
314
330
|
backend: result.backend, degraded: result.degraded,
|
|
315
331
|
...(result.degradationReason ? { degradationReason: result.degradationReason } : {}),
|
|
332
|
+
sttKey: result.dictationKey, engine: result.engine,
|
|
333
|
+
...(result.engineReason ? { engineReason: result.engineReason } : {}),
|
|
316
334
|
}
|
|
317
335
|
await markPromptDraftChunkTranscript(draftId, chunkIndex, record, purpose)
|
|
318
|
-
console.log(`[prompt-draft] chunk ${draftId}/${chunkIndex}: ${result.elapsedMs.toFixed(1)}ms | ${result.backend} | ${purpose}/${mode} | ${text.length} chars`)
|
|
336
|
+
console.log(`[prompt-draft] chunk ${draftId}/${chunkIndex}: ${result.elapsedMs.toFixed(1)}ms | ${result.backend}${result.engineReason ? ` (${result.engineReason})` : ''} | ${purpose}/${mode} | ${text.length} chars`)
|
|
319
337
|
return text
|
|
320
338
|
} catch (err) {
|
|
321
339
|
if (err instanceof NoSpeechDetectedError) {
|
|
@@ -324,7 +342,7 @@ async function transcribeChunk(draftId: string, chunkIndex: number, audio: Buffe
|
|
|
324
342
|
// recognized failure shape instead of persisting either the artifact
|
|
325
343
|
// or an empty gap. Other no-speech results retain existing behavior.
|
|
326
344
|
if (mode === 'hq' && isCaptionCreditHallucination(err.rawText)) {
|
|
327
|
-
const fastJob = modeQualityJobs.
|
|
345
|
+
const fastJob = [...modeQualityJobs.entries()].find(([key]) => key.startsWith(`${draftId}:${chunkIndex}:fast:${hash}:`))?.[1]
|
|
328
346
|
if (fastJob) {
|
|
329
347
|
try { await fastJob } catch { /* normal no-speech path below */ }
|
|
330
348
|
}
|
|
@@ -342,6 +360,7 @@ async function transcribeChunk(draftId: string, chunkIndex: number, audio: Buffe
|
|
|
342
360
|
requestedMode: 'hq',
|
|
343
361
|
degraded: true,
|
|
344
362
|
degradationReason: 'hq_caption_credit_hallucination',
|
|
363
|
+
sttKey: plan.key,
|
|
345
364
|
}, 'final')
|
|
346
365
|
}
|
|
347
366
|
console.warn(`[prompt-draft] dropped HQ caption-credit hallucination ${draftId}/${chunkIndex}; retained Fast text`)
|
|
@@ -349,7 +368,7 @@ async function transcribeChunk(draftId: string, chunkIndex: number, audio: Buffe
|
|
|
349
368
|
}
|
|
350
369
|
}
|
|
351
370
|
await markPromptDraftChunkTranscript(draftId, chunkIndex, {
|
|
352
|
-
text: '', hash, requestedMode: mode, actualQuality: mode, backend: 'no-speech', degraded: false,
|
|
371
|
+
text: '', hash, requestedMode: mode, actualQuality: mode, backend: 'no-speech', degraded: false, sttKey: plan.key,
|
|
353
372
|
}, purpose)
|
|
354
373
|
return ''
|
|
355
374
|
}
|
|
@@ -364,12 +383,15 @@ async function transcribeChunk(draftId: string, chunkIndex: number, audio: Buffe
|
|
|
364
383
|
return job
|
|
365
384
|
}
|
|
366
385
|
|
|
367
|
-
function isReusableTranscript(
|
|
386
|
+
export function isReusableTranscript(
|
|
368
387
|
cached: PromptDraftTranscriptRecord | undefined,
|
|
369
388
|
hash: string,
|
|
370
389
|
mode: 'hq' | 'fast',
|
|
390
|
+
sttKey: string,
|
|
371
391
|
): boolean {
|
|
372
392
|
if (!cached || cached.hash !== hash || cached.requestedMode !== mode) return false
|
|
393
|
+
// 6.73.0: a decode made under a different STT selection is decoded again.
|
|
394
|
+
if (recordSttKey(cached) !== sttKey) return false
|
|
373
395
|
if (cached.backend.startsWith('legacy')) return false
|
|
374
396
|
if (mode === 'fast') return cached.actualQuality === 'fast' || cached.actualQuality === 'cloud'
|
|
375
397
|
// mode === 'hq'
|
|
@@ -385,12 +407,14 @@ async function finalizeDraft(draftId: string, mode: 'hq' | 'fast', autoClean: Au
|
|
|
385
407
|
if (!meta) throw Object.assign(new Error('draft not found'), { status: 404 })
|
|
386
408
|
const texts: string[] = []
|
|
387
409
|
const transcriptRecords: PromptDraftTranscriptRecord[] = []
|
|
410
|
+
// One STT snapshot for the whole finalize: a settings change mid-finalize never mixes selections.
|
|
411
|
+
const plan = planFor(mode)
|
|
388
412
|
for (const chunk of readPromptDraftChunks(draftId)) {
|
|
389
413
|
try {
|
|
390
414
|
const hash = audioHash(chunk.audioBuffer)
|
|
391
415
|
// Await in-flight HQ warm before deciding cache miss (plan step 8b belt).
|
|
392
416
|
if (mode === 'hq') {
|
|
393
|
-
const inflight = modeQualityJobs.get(modeQualityKey(draftId, chunk.chunkIndex, 'hq', hash))
|
|
417
|
+
const inflight = modeQualityJobs.get(modeQualityKey(draftId, chunk.chunkIndex, 'hq', hash, plan.key))
|
|
394
418
|
if (inflight) {
|
|
395
419
|
try { await inflight } catch { /* finalize may retry with automatic below */ }
|
|
396
420
|
}
|
|
@@ -401,8 +425,8 @@ async function finalizeDraft(draftId: string, mode: 'hq' | 'fast', autoClean: Au
|
|
|
401
425
|
// When Settings HQ is available, never treat a degraded turbo warm as final
|
|
402
426
|
// — that left finished chats on Fast text while REVIEW looked "HQ ready".
|
|
403
427
|
// Legacy records stay excluded (reconstructed provenance).
|
|
404
|
-
const reusable = isReusableTranscript(cached, hash, mode)
|
|
405
|
-
const raw = reusable ? cached!.text : await transcribeChunk(draftId, chunk.chunkIndex, chunk.audioBuffer, mode, 'final')
|
|
428
|
+
const reusable = isReusableTranscript(cached, hash, mode, plan.key)
|
|
429
|
+
const raw = reusable ? cached!.text : await transcribeChunk(draftId, chunk.chunkIndex, chunk.audioBuffer, mode, 'final', plan)
|
|
406
430
|
const text = sanitizeTranscript(draftId, raw, !reusable)
|
|
407
431
|
if (text.trim()) {
|
|
408
432
|
texts.push(text.trim())
|
|
@@ -433,6 +457,8 @@ async function finalizeDraft(draftId: string, mode: 'hq' | 'fast', autoClean: Au
|
|
|
433
457
|
const degraded = mode === 'hq' && (actualQuality !== 'hq' || transcriptRecords.some(record => record.degraded))
|
|
434
458
|
const backends = [...new Set(transcriptRecords.map(record => record.backend))]
|
|
435
459
|
const degradationReason = transcriptRecords.find(record => record.degradationReason)?.degradationReason
|
|
460
|
+
const engines = [...new Set(transcriptRecords.map(record => record.engine ?? 'whisper'))]
|
|
461
|
+
const engineReason = transcriptRecords.find(record => record.engineReason)?.engineReason
|
|
436
462
|
return {
|
|
437
463
|
draftId,
|
|
438
464
|
text: finalText,
|
|
@@ -445,6 +471,9 @@ async function finalizeDraft(draftId: string, mode: 'hq' | 'fast', autoClean: Au
|
|
|
445
471
|
degraded,
|
|
446
472
|
backend: backends.length === 1 ? backends[0] : 'mixed',
|
|
447
473
|
...(degradationReason ? { degradationReason } : {}),
|
|
474
|
+
// 6.73.0, additive.
|
|
475
|
+
engine: engines.length === 1 ? engines[0] : engines.length === 0 ? plan.engine : 'mixed',
|
|
476
|
+
...(engineReason ? { engineReason } : {}),
|
|
448
477
|
}
|
|
449
478
|
}
|
|
450
479
|
|
|
@@ -513,15 +542,24 @@ promptDraftsRouter.post('/prompt-drafts/:draftId/peek', async (req, res) => {
|
|
|
513
542
|
peekTail = peekTail.then(async () => {
|
|
514
543
|
lease.setPhase('active')
|
|
515
544
|
try {
|
|
516
|
-
|
|
545
|
+
// 6.73.0: the live preview follows the Preview model. Parakeet runs on the
|
|
546
|
+
// shared one-at-a-time preview lane; any miss is the 6.72.0 Whisper peek.
|
|
547
|
+
const plan = currentDictationPlan('preview')
|
|
548
|
+
let raw: string | null = null
|
|
549
|
+
let engine: 'whisper' | 'parakeet-ultra' = 'whisper'
|
|
550
|
+
if (plan.engine === 'parakeet-ultra') {
|
|
551
|
+
const outcome = await runParakeetDictation(audio, plan, 'preview')
|
|
552
|
+
if (outcome.ok) { raw = applyCorrections(outcome.text); engine = 'parakeet-ultra' }
|
|
553
|
+
}
|
|
554
|
+
if (raw === null) raw = (await transcribeWhisperPreview(audio)).text
|
|
517
555
|
// 6.45.4: the same whole-sentence filler rule as the commit path, so an
|
|
518
556
|
// invented closing line never paints on the lens only to vanish later.
|
|
519
|
-
const text = sanitizeTranscript(draftId, guardPromptTail({ text:
|
|
557
|
+
const text = sanitizeTranscript(draftId, guardPromptTail({ text: raw, speech: SPEECH_UNKNOWN }).text, false)
|
|
520
558
|
if (!text) return
|
|
521
559
|
if (!loadPromptDraftMeta(draftId)) return
|
|
522
560
|
emitDisplay({
|
|
523
561
|
type: 'prompt_transcript',
|
|
524
|
-
data: { draftId, chunkIndex, text, provisional: true, peekGen },
|
|
562
|
+
data: { draftId, chunkIndex, text, provisional: true, peekGen, engine },
|
|
525
563
|
})
|
|
526
564
|
} catch (err: any) {
|
|
527
565
|
if (!(err instanceof NoSpeechDetectedError)) {
|
|
@@ -575,9 +613,10 @@ promptDraftsRouter.post('/prompt-drafts/:draftId/chunks', async (req, res) => {
|
|
|
575
613
|
// transcript only after rechecking that this exact audio still owns the
|
|
576
614
|
// chunk index; a retry/replacement must never paint stale words.
|
|
577
615
|
if (!isCurrentChunk(req.params.draftId, chunkIndex, audio)) return
|
|
616
|
+
const warm = loadPromptDraftMeta(req.params.draftId)?.warmTranscripts?.[String(chunkIndex)]
|
|
578
617
|
emitDisplay({
|
|
579
618
|
type: 'prompt_transcript',
|
|
580
|
-
data: { draftId: req.params.draftId, chunkIndex, text },
|
|
619
|
+
data: { draftId: req.params.draftId, chunkIndex, text, ...(warm?.requestedMode === 'fast' && warm.engine ? { engine: warm.engine } : {}) },
|
|
581
620
|
})
|
|
582
621
|
}).catch(err => {
|
|
583
622
|
console.warn(`[prompt-draft] warm transcription failed ${req.params.draftId}/${chunkIndex}: ${err.message}`)
|
|
@@ -1,4 +1,5 @@
|
|
|
1
|
-
// POST /api/transcribe
|
|
1
|
+
// POST /api/transcribe: one-shot transcription (message-query path). 6.73.0: the engine
|
|
2
|
+
// follows the STT settings (docs/dictation-stt-contract.md); Whisper is the fallback.
|
|
2
3
|
//
|
|
3
4
|
// The actual Whisper + cleanup pipeline lives in server/lib/transcribe-audio.ts so
|
|
4
5
|
// prompt-draft recovery and the legacy one-shot path share exactly the same behavior.
|
|
@@ -43,7 +44,7 @@ transcribeRouter.post('/transcribe', async (req, res) => {
|
|
|
43
44
|
}
|
|
44
45
|
|
|
45
46
|
const result = await transcribeAudioBuffer(audioBuffer, { mode: resolveMode(req) })
|
|
46
|
-
console.log(`[perf] /transcribe: ${result.elapsedMs.toFixed(1)}ms | mode=${result.mode} | ${result.backend} | ${result.audioBytes}b | ${result.text.length} chars`)
|
|
47
|
+
console.log(`[perf] /transcribe: ${result.elapsedMs.toFixed(1)}ms | mode=${result.mode} | ${result.backend}${result.engineReason ? ` (${result.engineReason})` : ''} | ${result.audioBytes}b | ${result.text.length} chars`)
|
|
47
48
|
res.json({
|
|
48
49
|
text: result.text,
|
|
49
50
|
backend: result.backend,
|
|
@@ -52,6 +53,9 @@ transcribeRouter.post('/transcribe', async (req, res) => {
|
|
|
52
53
|
actualQuality: result.actualQuality,
|
|
53
54
|
degraded: result.degraded,
|
|
54
55
|
...(result.degradationReason ? { degradationReason: result.degradationReason } : {}),
|
|
56
|
+
// 6.73.0, additive: which engine decoded it and, when the selection could not run, why.
|
|
57
|
+
engine: result.engine,
|
|
58
|
+
...(result.engineReason ? { engineReason: result.engineReason } : {}),
|
|
55
59
|
})
|
|
56
60
|
} catch (err: any) {
|
|
57
61
|
if (err instanceof MaintenanceLifecycleError) {
|