@gotcos/glasses-server 6.94.2 → 6.95.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,3 +1,17 @@
1
+ ## 6.95.0 - Speaker merge shadow mode (opt-in, log only)
2
+
3
+ 2026-10-28. Compatible with COS Control 0.5.352; no Control or glasses change required.
4
+
5
+ - **Why:** Mac recordings split one person into many anonymous `Voice N` labels (45 on one 60-minute call). The research canary (`community-1-fluidaudio-v1`, rounds 1 to 3) found a hybrid that fixes this safely on the reference call: production labels stay the base, and FluidAudio's offline Community-1 diarizer only proposes merges behind four gates. On the Oct 27 reference call (11 human-confirmed people) it cut fragmentation from 27 to 18 with no mixed people and the error proxy from 33% to 3%. Before any write path exists, the proposals have to be scored against Miles's own corrections, so this release only computes and logs them.
6
+ - **What:** with shadow mode on, every Mac recording whose finalization completes is queued. When no recording, save, finalization, one-shot transcription or dictation is live, a niced child process rebuilds the whole-call timeline from the retained chunk WAVs (each verified against the sidecar's `audioSha256`), runs the Parakeet install's `fluidaudiocli process --mode offline --overlapping-segments --export-embeddings` under `sandbox-exec` with network denied, applies the research gates (a port of `hybrid.decide()`), and appends one record per meeting to `<data>/speaker-shadow/proposals.jsonl`: accepted and rejected proposals with reasons, gate values, the post-hoc anchor ratio (reported, never decisive), co-occurring named labels as hints, and listen pointers by chunk and time (no transcript text). A readable report per finished meeting lands in `<data>/speaker-shadow/reports/<YYYY-MM-DD>/<title>__<session>.md`: "N voices would become M", each merge with its gate values and listen pointers (chunk file, time and words), rejections with reasons, and name hints ("this group also holds 65 s labelled ..."). `GET /api/speaker-shadow/proposals` and `/status` read them (token required); while it is on, `/api/health` reports `speaker_shadow` (absent when off, so the health shape is unchanged by default).
7
+ - **Log only:** nothing writes a meeting file, label, name, profile, embedding cache or correction ledger. Pinned by a data-home snapshot test (every file byte-identical after a run), an import allowlist and pinned write-call counts for every shadow module, and mutation-tested.
8
+ - **Exact port:** the TypeScript gates reproduce the Python tool's Driving decisions exactly (45 voices to 27, 29 proposed, 18 accepted, 11 rejected, every reason string and unrounded outlier distance bit for bit), and a real-CLI run of the TypeScript pipeline on the research copies reproduces the timeline PCM and WAV SHA-256 and the full round-3 report with zero differences. Python's half-to-even rounding and %-formatting, CPython 3.12+ compensated `sum()`, dict order and code-point string order are replicated and tested against CPython 3.15 vectors.
9
+ - **Switch, per Mac:** `POST /api/speaker-shadow/settings {"enabled":true}` (token required, no restart) saves it in `<data>/speaker-shadow/settings.json`; `COS_SPEAKER_SHADOW=1|0` pins it instead (409 `env_forced`); otherwise it is off. The package default is off for everyone, so installing this version changes nothing until a Mac turns it on.
10
+ - **How it stops:** the switch (default off; off creates nothing; off mid-run stops the run), a daily cap (12), a breaker (3 failures pause 1 hour), a 180 s wall clock and an RSS cap (3 GB) on the whole process group, and live work, which defers it or stops a run mid-way and requeues the meeting. No LLM; no network at run time.
11
+ - **Models:** never downloaded by the server. The pinned `FluidInference/speaker-diarization-coreml@df2625ac` files must be present (`models_missing` is logged otherwise); every file is hashed before every run and linked read-only into the CLI's private HOME.
12
+ - **Known limits:** a recording's first run with a cold CoreML cache leaves up to four empty `NSIRD_fluidaudiocli_*` folders in the per-user temp dir (counted in the record, not removed). Recovered (orphan) Mac captures are not queued. The gates were fixed on one truth-labeled call; scoring the logged proposals is the next step.
13
+ - **Switches:** the per-Mac setting and `COS_SPEAKER_SHADOW` (default off), `COS_SPEAKER_SHADOW_DAILY_RUNS`, `COS_SPEAKER_SHADOW_TIMEOUT_MS` (may only lower 180 s), `COS_SPEAKER_SHADOW_RSS_MB`, `COS_SPEAKER_SHADOW_MODELS`. No default-on change.
14
+
1
15
  ## 6.94.2 - Scoped profile matches for saved Mac recordings
2
16
 
3
17
  2026-10-28. Compatible with COS Control 0.5.352; no Control or glasses package change required.
package/README.md CHANGED
@@ -722,6 +722,58 @@ limits above: this release does not promise meeting-wide speaker deduplication
722
722
  or an attendee count. Older recordings require an explicit, validated repair;
723
723
  they are not silently rewritten on upgrade.
724
724
 
725
+ ### Speaker merge shadow mode (6.95.0, off by default, log only)
726
+
727
+ Mac recordings can split one person across many `Voice N` labels. Shadow mode asks
728
+ FluidAudio's offline Community-1 diarizer which of those labels belong together and
729
+ **logs** the answer for every finished Mac recording, with a readable report per meeting,
730
+ so the proposals can be checked by ear and against later human corrections. It changes
731
+ nothing: no meeting file, label, name, voice profile, embedding cache or correction ledger
732
+ is written.
733
+
734
+ - **On, for this Mac only** (no restart; the token is `COS_API_TOKEN` in `~/.cos-glasses/.env`):
735
+
736
+ ```sh
737
+ curl -s -X POST http://127.0.0.1:3141/api/speaker-shadow/settings \
738
+ -H "X-Cos-Token: $(sed -n 's/^COS_API_TOKEN=//p' ~/.cos-glasses/.env)" \
739
+ -H 'Content-Type: application/json' -d '{"enabled":true}'
740
+ ```
741
+
742
+ **Off:** the same with `{"enabled":false}`. The setting lives in
743
+ `~/.cos-glasses/data/speaker-shadow/settings.json`. `COS_SPEAKER_SHADOW=1` or `=0` in the
744
+ environment pins it instead (the POST then answers `409 env_forced`). With neither, it is
745
+ off, and off does nothing at all: no queue, no process, no file. Meetings already finished
746
+ when it is turned on are not queued; turning it off mid-run stops the run.
747
+ - **Needs:** the Parakeet install's verified `fluidaudiocli` 0.17.4, `sandbox-exec`, and the
748
+ pinned models from `FluidInference/speaker-diarization-coreml` at revision `df2625ac` in
749
+ `~/.cos-glasses/data/speaker-shadow/models/speaker-diarization/` (or
750
+ `COS_SPEAKER_SHADOW_MODELS`). The server never downloads them: missing or mismatched
751
+ models are logged as `models_missing` or `models_mismatch`. `GET /api/speaker-shadow/status`
752
+ shows `readiness` (models present, CLI installed and how it is trusted, sandbox). The models
753
+ are CC-BY-4.0 (scoped); see the repo's NOTICE.md.
754
+ - **When:** after a Mac recording's finalization is complete. A live recording, save,
755
+ batch finalization, one-shot transcription or dictation defers it, and one that starts
756
+ mid-run stops the run and puts the meeting back in the queue.
757
+ - **How it stops:** the switch, a daily cap (`COS_SPEAKER_SHADOW_DAILY_RUNS`, default 12), a
758
+ breaker (3 failures in a row pause it for 1 hour), a 180 s hard wall clock
759
+ (`COS_SPEAKER_SHADOW_TIMEOUT_MS` may only lower it) and an RSS cap on the whole run
760
+ (`COS_SPEAKER_SHADOW_RSS_MB`, default 3072). It runs in a niced child process, never on the
761
+ server's event loop. No LLM, and the CLI runs under `sandbox-exec` with network denied.
762
+ - **Gates** (the research tool's, unchanged): at least 0.8 of a voice's covered speech on one
763
+ cluster with at least half of it covered, at least 15 s of speech, and the voice's
764
+ embedding centroid inside the cluster's 95th-percentile spread (leave-fragment-out). Named
765
+ labels and `Ext` are never merged into or out of. The anchor ratio is reported for
766
+ scoring and decides nothing.
767
+ - **Read:** one report per finished Mac meeting in
768
+ `~/.cos-glasses/data/speaker-shadow/reports/<YYYY-MM-DD>/<title>__<session>.md`: "N voices
769
+ would become M", each accepted merge with its seconds, gate values and listen pointers
770
+ (chunk file, time, the chunk's words), rejections with reasons, and named labels sharing a
771
+ group as a hint only ("this group also holds 65 s labelled ..."). A meeting with no
772
+ proposals gets a one-line page saying why. The records are in `proposals.jsonl` beside it
773
+ (rotated past 8 MB) and `GET /api/speaker-shadow/proposals?limit=10&sessionId=mac_<uuid>`
774
+ (token required). While it is on, `/api/health` carries `speaker_shadow` (codes and counts
775
+ only); when off the key is absent.
776
+
725
777
  ### iPhone microphone meetings (6.93.0, off by default)
726
778
 
727
779
  With `COS_IPHONE_MICROPHONE_MEETINGS=1`, HQ set to Parakeet Ultra and Parakeet
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@gotcos/glasses-server",
3
- "version": "6.94.2",
3
+ "version": "6.95.0",
4
4
  "description": "COS Glasses: self-hosted AI heads-up-display server for Even G2 smart glasses, powered by Claude Code, Codex, Cursor Agent CLI, or local Ollama",
5
5
  "type": "module",
6
6
  "bin": {
@@ -58,6 +58,7 @@
58
58
  "!server/scripts/routines-mutation-gate.mjs",
59
59
  "!server/scripts/routines-canary.ts",
60
60
  "!server/scripts/stt-mutation-gate.mjs",
61
+ "!server/scripts/speaker-shadow-mutation-gate.mjs",
61
62
  "!server/scripts/dictation-stt-mutation-gate.mjs",
62
63
  "!server/scripts/pairing-mutation-gate.mjs",
63
64
  "!server/scripts/away-hold-mutation-gate.mjs",
package/server/index.ts CHANGED
@@ -74,7 +74,7 @@ import { ForkJobLedger, fileForkJobPersistence } from './lib/fork-job-ledger.js'
74
74
  import { nativeHead, realNativeHeadDeps, registerClaudeTranscriptHint } from './lib/native-head.js'
75
75
  import { threadOccupancy, holderActivity } from './lib/thread-occupancy.js'
76
76
  import { displayRouter } from './routes/display.js'
77
- import { transcribeStreamRouter, getActiveTranscriptionSessionCount } from './routes/transcribe-stream.js'
77
+ import { transcribeStreamRouter, getActiveTranscriptionSessionCount, getTranscriptionSessionLiveness } from './routes/transcribe-stream.js'
78
78
  import { meetingRouter, resumeAutonomousMeetingSaves, resumeMeetingFinalizationJobs, scheduleTranscriptCleanupSweep, transcriptCleanupRouteDeps } from './routes/meeting.js'
79
79
  import { createTranscriptCleanupRouter } from './routes/transcript-cleanup.js'
80
80
  import { meetingsRouter } from './routes/meetings.js'
@@ -110,6 +110,8 @@ import { voiceRouter } from './routes/voice.js'
110
110
  import { glossaryRouter } from './routes/glossary.js'
111
111
  import { homeAreaRouter } from './routes/home-area.js'
112
112
  import { createSttSettingsRouter } from './routes/stt-settings.js'
113
+ import { createSpeakerShadowRouter } from './routes/speaker-shadow.js'
114
+ import { configureSpeakerShadow, speakerShadowLeaseActive } from './lib/speaker-shadow.js'
113
115
  import { setRecordingLiveProbe, startPreviewWarmLoop, stopPreviewWarmLoop } from './lib/stt-runtime.js'
114
116
  import { findPinnedCli, killParakeetChildren, sweepParakeetStaging, sweepStaleParakeetTmp } from './lib/parakeet.js'
115
117
  import { handoffsRouter } from './routes/handoffs.js'
@@ -1229,6 +1231,8 @@ app.use('/api', voiceRouter)
1229
1231
  app.use('/api', glossaryRouter)
1230
1232
  app.use('/api', homeAreaRouter)
1231
1233
  app.use('/api', createSttSettingsRouter({ recordingLive: () => getActiveTranscriptionSessionCount() > 0 }))
1234
+ // 6.95.0: speaker merge shadow mode, read only (GET proposals and status).
1235
+ app.use('/api', createSpeakerShadowRouter())
1232
1236
  app.use('/api', handoffsRouter)
1233
1237
  app.use('/api', recoveryRouter)
1234
1238
  app.use('/api', promptEditRouter)
@@ -1541,6 +1545,14 @@ listenRequiredServers(listeners).then(() => {
1541
1545
  // defer while a capture is live or admissions are closed.
1542
1546
  startMeetingMergeScheduler()
1543
1547
 
1548
+ // 6.95.0: speaker merge shadow mode (off unless COS_SPEAKER_SHADOW=1; log only). Live
1549
+ // work is a recording that is still live, or any capture or dictation lease; a drain
1550
+ // or shutdown also holds it. Until this runs, the runner never starts a diarization.
1551
+ configureSpeakerShadow({
1552
+ captureActive: () => getTranscriptionSessionLiveness().live > 0 || speakerShadowLeaseActive(),
1553
+ admissionsOpen: () => !gracefulShutdownStarted && maintenanceAdmissionsOpen(),
1554
+ })
1555
+
1544
1556
  // AFTER the imports sweep, not before: the sweep removes a killed write's
1545
1557
  // debris, and re-driving a pending apply into a half-written decision is
1546
1558
  // exactly what the sweep exists to prevent. A pending action is one a
@@ -0,0 +1,466 @@
1
+ // Speaker merge shadow mode (6.95.0): the gates, ported EXACTLY from the research tool.
2
+ //
3
+ // SOURCE OF TRUTH. `community-1-fluidaudio-v1/scripts/hybrid.py` `decide()` (round 2) and
4
+ // the report assembly in `round3/propose.py`, on CPython 3.15. Production `Voice N` labels
5
+ // stay the base; FluidAudio's offline Community-1 clusters only PROPOSE that two of them
6
+ // are one person. A voice may join a merge only when all hold:
7
+ // (a) at least `majority` (0.8) of its FluidAudio-covered speech is on one cluster, and
8
+ // at least half of its speech is covered at all;
9
+ // (b) it has at least `floorSeconds` (15 s) of production speech;
10
+ // (c) its window-embedding centroid lies inside the target cluster's own spread,
11
+ // measured leave-fragment-out against the 95th percentile.
12
+ // Named labels, `Ext` and every non-Voice label are frozen: never merged into or out of.
13
+ // The post-hoc anchor ratio is computed and REPORTED ONLY. It decides nothing.
14
+ //
15
+ // LOG ONLY. Nothing here writes, and nothing calls a writer: the output is a proposal record.
16
+ //
17
+ // Order and rounding are part of the contract: Python dicts iterate in insertion order,
18
+ // max()/most_common(1) keep the FIRST maximum, sorted() is stable, gates compare ROUNDED
19
+ // values and reason strings use Python %-formatting. Every one of those is kept below,
20
+ // with the Python-compatible numerics in speaker-shadow-numeric.ts.
21
+
22
+ import { PY_VOICE_LABEL, codePointCompare, pyClock, pyFirstInt, pyFixed, pyRound, pyRoundInt, pySum } from './speaker-shadow-numeric.js'
23
+
24
+ export const SHADOW_SAMPLE_RATE = 16_000
25
+ /** The round-2 gates, fixed before scoring (RESULT-ROUND2.md). */
26
+ export const SHADOW_GATES = Object.freeze({ majority: 0.8, minCoveredFraction: 0.5, floorSeconds: 15, anchorRatioPostHoc: 1.5 })
27
+
28
+ /** One production segment on the reconstructed timeline: [lo, hi) samples and its label. */
29
+ export type ShadowSegment = readonly [lo: number, hi: number, label: string]
30
+ /** A FluidAudio turn in seconds (float32 values widened, end clipped to the audio). */
31
+ export interface ShadowTurn { start: number; end: number; speaker: string }
32
+ /** One `--export-embeddings` row (the fields decide() reads). */
33
+ export interface ShadowEmbeddingRow { startTime: number; endTime: number; cluster: number; embedding256: number[] }
34
+ /** Where a production segment came from, for listen pointers. Same order as the segments. */
35
+ export interface ShadowSegmentSource { chunkIndex: number; start: number; end: number; abs: number }
36
+
37
+ export function isShadowVoiceLabel(label: string): boolean {
38
+ return PY_VOICE_LABEL.test(label)
39
+ }
40
+
41
+ // ── Vector helpers (hybrid.py norm / mean_dir / cosdist / p95) ───────────────
42
+
43
+ export function norm(v: readonly number[]): number[] {
44
+ const squares = new Array<number>(v.length)
45
+ for (let i = 0; i < v.length; i++) squares[i] = v[i] * v[i]
46
+ const n = Math.sqrt(pySum(squares)) || 1.0
47
+ return v.map(x => x / n)
48
+ }
49
+
50
+ export function meanDir(vs: readonly number[][]): number[] {
51
+ const s = new Array<number>(vs[0].length).fill(0)
52
+ for (const v of vs) for (let i = 0; i < v.length; i++) s[i] += v[i]
53
+ return norm(s)
54
+ }
55
+
56
+ export function cosdist(a: readonly number[], b: readonly number[]): number {
57
+ const n = Math.min(a.length, b.length)
58
+ const products = new Array<number>(n)
59
+ for (let i = 0; i < n; i++) products[i] = a[i] * b[i]
60
+ return 1.0 - pySum(products)
61
+ }
62
+
63
+ export function p95(values: readonly number[]): number {
64
+ const xs = [...values].sort((x, y) => x - y)
65
+ const k = 0.95 * (xs.length - 1)
66
+ const f = Math.floor(k), c = Math.ceil(k)
67
+ return f === c ? xs[f] : xs[f] + (xs[c] - xs[f]) * (k - f)
68
+ }
69
+
70
+ // ── Interval helpers (common.py interval_samples / exact_overlap) ────────────
71
+
72
+ const KEY_SEP = '\u0000'
73
+ type Interval = readonly [lo: number, hi: number, label: string]
74
+
75
+ /** FluidAudio turns as sample intervals, validated like common.py. */
76
+ export function intervalSamples(turns: readonly ShadowTurn[], total: number): Interval[] {
77
+ return turns.map(t => {
78
+ const { start: a, end: b, speaker: label } = t
79
+ if (typeof label !== 'string' || !label || label.includes(KEY_SEP)) throw new Error('bad FluidAudio turn label')
80
+ if (!Number.isFinite(a) || !Number.isFinite(b) || !(a >= 0 && a < b && b <= total / SHADOW_SAMPLE_RATE + 1e-6)) throw new Error('bad FluidAudio turn time')
81
+ return [Math.max(0, pyRoundInt(a * SHADOW_SAMPLE_RATE)), Math.min(total, pyRoundInt(b * SHADOW_SAMPLE_RATE)), label] as const
82
+ })
83
+ }
84
+
85
+ /** Turns sorted by start with the longest span, so a query scans only what can overlap. */
86
+ export class IntervalIndex {
87
+ private readonly rows: Interval[]
88
+ private readonly maxSpan: number
89
+ constructor(intervals: readonly Interval[]) {
90
+ this.rows = [...intervals].sort((x, y) => x[0] - y[0])
91
+ this.maxSpan = this.rows.reduce((m, [a, b]) => Math.max(m, b - a), 0)
92
+ }
93
+ /** Every interval with a < hi and b > lo (the set common.py crops to). */
94
+ overlapping(lo: number, hi: number): Interval[] {
95
+ const rows = this.rows
96
+ let left = 0, right = rows.length
97
+ const from = lo - this.maxSpan
98
+ while (left < right) { const mid = (left + right) >> 1; if (rows[mid][0] < from) left = mid + 1; else right = mid }
99
+ const out: Interval[] = []
100
+ for (let i = left; i < rows.length && rows[i][0] < hi; i++) if (rows[i][1] > lo) out.push(rows[i])
101
+ return out
102
+ }
103
+ }
104
+
105
+ /**
106
+ * Samples of [lo, hi) by active-label set, in Python dict order (first appearance as the
107
+ * points ascend). Keys are the sorted labels joined by NUL; '' is "no cluster active".
108
+ */
109
+ export function exactOverlap(lo: number, hi: number, index: IntervalIndex): Map<string, number> {
110
+ if (!Number.isInteger(lo) || !Number.isInteger(hi) || !(lo >= 0 && lo < hi)) throw new Error('exact_overlap range')
111
+ const boundaries = new Set<number>([lo, hi])
112
+ const cropped: Array<[number, number, string]> = []
113
+ for (const [a0, b0, label] of index.overlapping(lo, hi)) {
114
+ const a = Math.max(lo, a0), b = Math.min(hi, b0)
115
+ if (a < b) { cropped.push([a, b, label]); boundaries.add(a); boundaries.add(b) }
116
+ }
117
+ const points = [...boundaries].sort((x, y) => x - y)
118
+ const counts = new Map<string, number>()
119
+ for (let i = 0; i + 1 < points.length; i++) {
120
+ const a = points[i], b = points[i + 1]
121
+ const active = [...new Set(cropped.filter(([x, y]) => x <= a && y >= b).map(([, , label]) => label))].sort(codePointCompare)
122
+ const key = active.join(KEY_SEP)
123
+ counts.set(key, (counts.get(key) ?? 0) + (b - a))
124
+ }
125
+ let sum = 0
126
+ for (const n of counts.values()) sum += n
127
+ if (sum !== hi - lo) throw new Error('exact_overlap accounting')
128
+ return counts
129
+ }
130
+
131
+ const isSingle = (key: string) => key !== '' && !key.includes(KEY_SEP)
132
+
133
+ function mergeIv(iv: Array<[number, number]>): Array<[number, number]> {
134
+ const out: Array<[number, number]> = []
135
+ for (const [a, b] of [...iv].sort((x, y) => (x[0] - y[0]) || (x[1] - y[1]))) {
136
+ const last = out[out.length - 1]
137
+ if (last && a <= last[1]) last[1] = Math.max(b, last[1])
138
+ else out.push([a, b])
139
+ }
140
+ return out
141
+ }
142
+
143
+ // ── decide() ─────────────────────────────────────────────────────────────────
144
+
145
+ export type ShadowOutlier =
146
+ | { pass: false; reason: string }
147
+ | { pass: boolean; fragmentCentroidDistance: number; clusterP95Distance: number; raw: { distance: number; p95: number } }
148
+
149
+ export interface ShadowVoiceRow {
150
+ seconds: number
151
+ coveredFraction: number
152
+ topCluster: string | null
153
+ majority: number
154
+ fragmentEmbeddings: number
155
+ otherClusterMembers: number
156
+ outlier: ShadowOutlier
157
+ eligible: boolean
158
+ reasons: string[]
159
+ }
160
+
161
+ export interface ShadowItem { vec: number[]; exportCluster: string; outputCluster: string | null; voice: string | null }
162
+ export interface ShadowGroup { cluster: string; voices: string[]; anchor: string | null; mergedIntoAnchor: string[] }
163
+
164
+ export interface ShadowDecision {
165
+ voices: Map<string, Array<[number, number]>>
166
+ rows: Map<string, ShadowVoiceRow>
167
+ groups: ShadowGroup[]
168
+ rejected: Array<{ voice: string; cluster: string; reasons: string[] }>
169
+ remap: Map<string, string>
170
+ proposed: number
171
+ accepted: number
172
+ agree: number
173
+ withOut: number
174
+ items: ShadowItem[]
175
+ voiceBefore: number
176
+ voiceAfter: number
177
+ index: IntervalIndex
178
+ }
179
+
180
+ export function decide(
181
+ segs: readonly ShadowSegment[],
182
+ total: number,
183
+ regularTurns: readonly ShadowTurn[],
184
+ emb: readonly ShadowEmbeddingRow[],
185
+ nSec: number = SHADOW_GATES.floorSeconds,
186
+ majority: number = SHADOW_GATES.majority,
187
+ ): ShadowDecision {
188
+ const SR = SHADOW_SAMPLE_RATE
189
+ const index = new IntervalIndex(intervalSamples(regularTurns, total))
190
+
191
+ const collected = new Map<string, Array<[number, number]>>()
192
+ for (const [lo, hi, label] of segs) {
193
+ if (!isShadowVoiceLabel(label)) continue
194
+ const list = collected.get(label)
195
+ if (list) list.push([lo, hi]); else collected.set(label, [[lo, hi]])
196
+ }
197
+ const voices = new Map<string, Array<[number, number]>>()
198
+ for (const [v, iv] of collected) voices.set(v, mergeIv(iv))
199
+
200
+ // Embedding attribution: a window belongs to the voice holding >= 50% of its span.
201
+ const items: ShadowItem[] = []
202
+ for (const e of emb) {
203
+ const lo = pyRoundInt(e.startTime * SR), hi = pyRoundInt(e.endTime * SR)
204
+ if (hi <= lo) continue
205
+ if (!Number.isInteger(e.cluster)) throw new Error('embedding cluster is not an integer')
206
+ let best: string | null = null, bestN = 0
207
+ for (const [v, iv] of voices) {
208
+ let n = 0
209
+ for (const [a, b] of iv) if (a < hi && b > lo) n += Math.max(0, Math.min(hi, b) - Math.max(lo, a))
210
+ if (n > bestN) { best = v; bestN = n }
211
+ }
212
+ const cover = lo < total ? exactOverlap(lo, Math.min(hi, total), index) : new Map<string, number>()
213
+ let outLabel: string | null = null, outN = -Infinity
214
+ for (const [key, n] of cover) if (isSingle(key) && n > outN) { outLabel = key; outN = n }
215
+ items.push({ vec: norm(e.embedding256), exportCluster: `S${e.cluster + 1}`, outputCluster: outLabel, voice: best !== null && bestN >= 0.5 * (hi - lo) ? best : null })
216
+ }
217
+ let agree = 0, withOut = 0
218
+ for (const x of items) if (x.outputCluster) { withOut++; if (x.exportCluster === x.outputCluster) agree++ }
219
+
220
+ const rows = new Map<string, ShadowVoiceRow>()
221
+ for (const [v, iv] of voices) {
222
+ let dur = 0
223
+ for (const [a, b] of iv) dur += b - a
224
+ const counts = new Map<string, number>()
225
+ for (const [a, b] of iv) for (const [key, n] of exactOverlap(a, b, index)) counts.set(key, (counts.get(key) ?? 0) + n)
226
+ let covered = 0
227
+ for (const [key, n] of counts) if (key !== '') covered += n
228
+ let top: string | null = null, topN = 0, topSeen = false
229
+ for (const [key, n] of counts) if (isSingle(key) && (!topSeen || n > topN)) { top = key; topN = n; topSeen = true }
230
+ rows.set(v, {
231
+ seconds: pyRound(dur / SR, 2),
232
+ coveredFraction: dur ? pyRound(covered / dur, 3) : 0,
233
+ topCluster: top,
234
+ majority: covered ? pyRound(topN / covered, 3) : 0,
235
+ fragmentEmbeddings: 0, otherClusterMembers: 0,
236
+ outlier: { pass: false, reason: '' }, eligible: false, reasons: [],
237
+ })
238
+ }
239
+
240
+ // (c) outlier check against the voice's top cluster; membership = export cluster label.
241
+ for (const [v, r] of rows) {
242
+ const k = r.topCluster
243
+ const frag = items.filter(x => x.voice === v).map(x => x.vec)
244
+ const members = items.filter(x => x.exportCluster === k && x.voice !== v).map(x => x.vec)
245
+ r.fragmentEmbeddings = frag.length
246
+ r.otherClusterMembers = members.length
247
+ if (k === null || frag.length < 3 || members.length < 5) {
248
+ r.outlier = { pass: false, reason: `insufficient embeddings (fragment ${frag.length}, others ${members.length})` }
249
+ continue
250
+ }
251
+ const c = meanDir(members)
252
+ const thr = p95(members.map(m => cosdist(m, c)))
253
+ const d = cosdist(meanDir(frag), c)
254
+ r.outlier = { pass: d <= thr, fragmentCentroidDistance: pyRound(d, 4), clusterP95Distance: pyRound(thr, 4), raw: { distance: d, p95: thr } }
255
+ }
256
+
257
+ for (const r of rows.values()) {
258
+ const reasons: string[] = []
259
+ if (!(r.coveredFraction >= 0.5)) reasons.push(`(a) covered ${pyFixed(r.coveredFraction, 2)} < 0.50`)
260
+ if (!(r.majority >= majority)) reasons.push(`(a) majority ${pyFixed(r.majority, 2)} < ${pyFixed(majority, 2)}`)
261
+ if (!(r.seconds >= nSec)) reasons.push(`(b) ${pyFixed(r.seconds, 1)} s < ${pyFixed(nSec, 0)} s`)
262
+ if (!r.outlier.pass) {
263
+ const o = r.outlier
264
+ reasons.push('(c) ' + ('reason' in o ? o.reason : `centroid ${pyFixed(o.fragmentCentroidDistance, 3)} > p95 ${pyFixed(o.clusterP95Distance, 3)}`))
265
+ }
266
+ r.eligible = reasons.length === 0
267
+ r.reasons = reasons
268
+ }
269
+
270
+ const byCluster = new Map<string, string[]>()
271
+ for (const [v, r] of rows) {
272
+ if (r.topCluster && r.majority > 0.5) {
273
+ const list = byCluster.get(r.topCluster)
274
+ if (list) list.push(v); else byCluster.set(r.topCluster, [v])
275
+ }
276
+ }
277
+ const longestFirst = (vs: string[]) => [...vs].sort((a, b) => rows.get(b)!.seconds - rows.get(a)!.seconds)
278
+ const remap = new Map<string, string>()
279
+ const rejected: ShadowDecision['rejected'] = []
280
+ const groups: ShadowGroup[] = []
281
+ let proposed = 0, accepted = 0
282
+ for (const [k, vs] of [...byCluster].sort((x, y) => codePointCompare(x[0], y[0]))) {
283
+ if (vs.length < 2) continue
284
+ proposed += vs.length - 1
285
+ const ok = longestFirst(vs.filter(v => rows.get(v)!.eligible))
286
+ const anchor = ok[0] ?? null
287
+ const merged = ok.slice(1)
288
+ accepted += merged.length
289
+ for (const v of merged) remap.set(v, anchor!)
290
+ for (const v of vs) {
291
+ if (!ok.includes(v)) rejected.push({ voice: v, cluster: k, reasons: rows.get(v)!.reasons })
292
+ else if (v === anchor && !merged.length) rejected.push({ voice: v, cluster: k, reasons: ['no other eligible voice in this cluster'] })
293
+ }
294
+ groups.push({ cluster: k, voices: longestFirst(vs), anchor, mergedIntoAnchor: merged })
295
+ }
296
+
297
+ return {
298
+ voices, rows, groups, rejected, remap, proposed, accepted, agree, withOut, items,
299
+ voiceBefore: voices.size,
300
+ voiceAfter: new Set([...voices.keys()].map(v => remap.get(v) ?? v)).size,
301
+ index,
302
+ }
303
+ }
304
+
305
+ // ── Report assembly (round3/propose.py), log only ────────────────────────────
306
+
307
+ export interface ShadowListenPointer { chunkIndex: number; inFileStart: number; inFileEnd: number; callTime: string }
308
+ export interface ShadowVoiceEntry {
309
+ voice: string
310
+ seconds: number
311
+ coveredFraction: number
312
+ topCluster: string | null
313
+ majority: number
314
+ outlierPass: boolean
315
+ outlierDistance: number | null
316
+ outlierP95: number | null
317
+ outlierNote: string | null
318
+ /** UNVALIDATED post-hoc check, reported only (fragment centroid vs the kept voice's spread). */
319
+ anchorRatioPostHoc: number | null
320
+ eligible: boolean
321
+ reasons: string[]
322
+ listen: ShadowListenPointer[]
323
+ }
324
+ /** A non-Voice label sharing the cluster, as a hint only ("cluster also holds 65 s of ..."). */
325
+ export interface ShadowOtherLabel { label: string; seconds: number }
326
+ export interface ShadowAccepted { cluster: string; keepLabel: string; merge: string[]; voices: ShadowVoiceEntry[]; otherLabelsInCluster: ShadowOtherLabel[] }
327
+ export type ShadowRejected = ShadowVoiceEntry & { cluster: string; wouldJoin: string | null; otherLabelsInCluster: ShadowOtherLabel[] }
328
+ export interface ShadowUntouched { voice: string; seconds: number; why: string }
329
+
330
+ export interface ShadowProposals {
331
+ productionVoices: number
332
+ voicesAfter: number
333
+ frozenOtherLabels: string[]
334
+ fluidAudioClusters: number
335
+ proposedMerges: number
336
+ acceptedMerges: number
337
+ rejectedMerges: number
338
+ embeddingClusterAgreement: [agree: number, withOutputCluster: number, total: number]
339
+ accepted: ShadowAccepted[]
340
+ rejected: ShadowRejected[]
341
+ untouched: ShadowUntouched[]
342
+ }
343
+
344
+ function listenPointers(label: string, byLabel: Map<string, ShadowSegmentSource[]>, k = 3): ShadowListenPointer[] {
345
+ const segs = byLabel.get(label) ?? []
346
+ if (!segs.length) return []
347
+ const pick = (score: (s: ShadowSegmentSource) => number, max: boolean) => segs.reduce((best, s) => (max ? score(s) > score(best) : score(s) < score(best)) ? s : best)
348
+ const picks = [pick(s => s.end - s.start, true), pick(s => s.abs, false), pick(s => s.abs, true)]
349
+ const seen = new Set<number>(), out: ShadowListenPointer[] = []
350
+ for (const s of picks) {
351
+ if (seen.has(s.chunkIndex)) continue
352
+ seen.add(s.chunkIndex)
353
+ out.push({ chunkIndex: s.chunkIndex, inFileStart: pyRound(s.start, 2), inFileEnd: pyRound(s.end, 2), callTime: pyClock(s.abs) })
354
+ }
355
+ return out.slice(0, k)
356
+ }
357
+
358
+ /** The proposal report for one meeting, exactly as round3/propose.py assembles it (minus transcript text and file paths). */
359
+ export function buildShadowProposals(
360
+ D: ShadowDecision,
361
+ segs: readonly ShadowSegment[],
362
+ sources: readonly ShadowSegmentSource[],
363
+ regularTurns: readonly ShadowTurn[],
364
+ ): ShadowProposals {
365
+ const SR = SHADOW_SAMPLE_RATE
366
+ const rows = D.rows
367
+ if (sources.length !== segs.length) throw new Error('segment sources out of step')
368
+ const byLabel = new Map<string, ShadowSegmentSource[]>()
369
+ segs.forEach(([, , label], i) => { const list = byLabel.get(label); if (list) list.push(sources[i]); else byLabel.set(label, [sources[i]]) })
370
+
371
+ const vecs = new Map<string, number[][]>()
372
+ for (const it of D.items) if (it.voice) { const list = vecs.get(it.voice); if (list) list.push(it.vec); else vecs.set(it.voice, [it.vec]) }
373
+ const ratio = (v: string, anchor: string): number | null => {
374
+ const A = vecs.get(anchor) ?? [], B = vecs.get(v) ?? []
375
+ if (v === anchor || A.length < 3 || B.length < 3) return null
376
+ const ca = meanDir(A)
377
+ const thr = p95(A.map(x => cosdist(x, ca)))
378
+ return thr > 0 ? pyRound(cosdist(meanDir(B), ca) / thr, 2) : null
379
+ }
380
+
381
+ // Named / non-Voice labels sharing each FluidAudio cluster (caution only; never merged).
382
+ const named = new Map<string, Map<string, number>>()
383
+ for (const [lo, hi, label] of segs) {
384
+ if (isShadowVoiceLabel(label)) continue
385
+ for (const [key, n] of exactOverlap(lo, hi, D.index)) {
386
+ if (!isSingle(key)) continue
387
+ let inner = named.get(key)
388
+ if (!inner) { inner = new Map(); named.set(key, inner) }
389
+ inner.set(label, (inner.get(label) ?? 0) + n)
390
+ }
391
+ }
392
+
393
+ const voiceEntry = (v: string, anchor: string): ShadowVoiceEntry => {
394
+ const r = rows.get(v)!
395
+ const o = r.outlier
396
+ return {
397
+ voice: v, seconds: r.seconds, coveredFraction: r.coveredFraction, topCluster: r.topCluster, majority: r.majority,
398
+ outlierPass: o.pass,
399
+ outlierDistance: 'reason' in o ? null : o.fragmentCentroidDistance,
400
+ outlierP95: 'reason' in o ? null : o.clusterP95Distance,
401
+ outlierNote: 'reason' in o ? o.reason : null,
402
+ anchorRatioPostHoc: ratio(v, anchor), eligible: r.eligible, reasons: r.reasons,
403
+ listen: listenPointers(v, byLabel),
404
+ }
405
+ }
406
+
407
+ const accepted: ShadowAccepted[] = [], rejected: ShadowRejected[] = []
408
+ const grouped = new Set<string>()
409
+ for (const g of D.groups) {
410
+ const anchor = g.anchor || g.voices[0]
411
+ for (const v of g.voices) grouped.add(v)
412
+ const members = g.voices.map(v => voiceEntry(v, anchor))
413
+ const caution: ShadowOtherLabel[] = [...(named.get(g.cluster) ?? new Map<string, number>())]
414
+ .sort((x, y) => y[1] - x[1])
415
+ .filter(([, n]) => n >= SR)
416
+ .map(([label, n]) => ({ label, seconds: pyRound(n / SR, 1) }))
417
+ if (g.mergedIntoAnchor.length) {
418
+ const kept = [g.anchor, ...g.mergedIntoAnchor]
419
+ accepted.push({ cluster: g.cluster, keepLabel: g.anchor!, merge: g.mergedIntoAnchor, voices: members.filter(m => kept.includes(m.voice)), otherLabelsInCluster: caution })
420
+ }
421
+ for (const m of members) {
422
+ const v = m.voice
423
+ if (g.mergedIntoAnchor.includes(v) || (v === g.anchor && g.mergedIntoAnchor.length)) continue
424
+ if (v === anchor) rejected.push({ cluster: g.cluster, wouldJoin: null, ...m, reasons: [...m.reasons, 'no other voice on this cluster passed every gate'], otherLabelsInCluster: caution })
425
+ else rejected.push({ cluster: g.cluster, wouldJoin: anchor, ...m, otherLabelsInCluster: caution })
426
+ }
427
+ }
428
+
429
+ const untouched: ShadowUntouched[] = []
430
+ const byNumber = [...rows].sort((x, y) => (pyFirstInt(x[0]) ?? 0) - (pyFirstInt(y[0]) ?? 0))
431
+ for (const [v, r] of byNumber) {
432
+ if (grouped.has(v)) continue
433
+ let why: string
434
+ if (!r.topCluster || r.coveredFraction === 0) why = 'FluidAudio did not cover this voice'
435
+ else if (r.majority <= 0.5) why = `no FluidAudio cluster holds a majority of it (top ${r.topCluster} ${pyFixed(r.majority, 2)})`
436
+ else why = `the only voice on FluidAudio cluster ${r.topCluster}`
437
+ untouched.push({ voice: v, seconds: r.seconds, why })
438
+ }
439
+
440
+ return {
441
+ productionVoices: D.voiceBefore,
442
+ voicesAfter: D.voiceAfter,
443
+ frozenOtherLabels: [...new Set(segs.map(([, , label]) => label).filter(label => !isShadowVoiceLabel(label)))].sort(codePointCompare),
444
+ fluidAudioClusters: new Set(regularTurns.map(t => t.speaker)).size,
445
+ proposedMerges: D.proposed,
446
+ acceptedMerges: D.accepted,
447
+ rejectedMerges: rejected.filter(x => x.wouldJoin).length,
448
+ embeddingClusterAgreement: [D.agree, D.withOut, D.items.length],
449
+ accepted, rejected, untouched,
450
+ }
451
+ }
452
+
453
+ /** adapt.py native_segments + to_turns: float32 times, end clipped to the audio, empty turns dropped. */
454
+ export function nativeTurns(segments: ReadonlyArray<{ speakerId: unknown; startTimeSeconds: unknown; endTimeSeconds: unknown }>, totalSeconds: number): { turns: ShadowTurn[]; clipped: number; dropped: number } {
455
+ const turns: ShadowTurn[] = []
456
+ let clipped = 0, dropped = 0
457
+ for (const s of segments) {
458
+ if (typeof s.speakerId !== 'string' || typeof s.startTimeSeconds !== 'number' || typeof s.endTimeSeconds !== 'number') throw new Error('bad FluidAudio segment')
459
+ const start = Math.fround(s.startTimeSeconds)
460
+ let end = Math.fround(s.endTimeSeconds)
461
+ if (end > totalSeconds) { end = totalSeconds; clipped++ }
462
+ if (!(start < end)) { dropped++; continue }
463
+ turns.push({ start, end, speaker: s.speakerId })
464
+ }
465
+ return { turns, clipped, dropped }
466
+ }