@fastagent-sh/voicenote 0.17.9 → 0.18.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +12 -4
- package/README.zh-CN.md +13 -5
- package/package.json +2 -1
- package/src/cli.ts +308 -175
- package/src/jobs.ts +437 -0
package/README.md
CHANGED
|
@@ -53,18 +53,20 @@ The first install creates `~/.config/voicenote/config.json`. A legacy `speakers.
|
|
|
53
53
|
Manual install:
|
|
54
54
|
|
|
55
55
|
```bash
|
|
56
|
+
bun remove -g @kid7st/voicenote 2>/dev/null || true # drop the pre-rebrand package if present (safe no-op otherwise)
|
|
56
57
|
bun add -g @fastagent-sh/voicenote
|
|
57
58
|
mkdir -p ~/.local/bin
|
|
58
59
|
ln -sf ~/.bun/bin/vn ~/.local/bin/vn
|
|
59
60
|
```
|
|
60
61
|
|
|
61
|
-
An older `git+…#main` install is replaced in place by `bun add -g @fastagent-sh/voicenote
|
|
62
|
+
An older `git+…#main` install is replaced in place by `bun add -g @fastagent-sh/voicenote`. The one exception is the **pre-rebrand `@kid7st/voicenote`** package: it ships the same `vn` bin, so the `bun remove -g` line above clears it first (the one-line `install.sh` does this automatically).
|
|
62
63
|
|
|
63
64
|
### Windows (CLI)
|
|
64
65
|
|
|
65
66
|
The CLI is cross-platform. Prerequisites: Bun, ffmpeg (provides `ffprobe.exe`), Node + pi.
|
|
66
67
|
|
|
67
68
|
```powershell
|
|
69
|
+
bun remove -g @kid7st/voicenote 2>$null # drop the pre-rebrand package if present (safe no-op otherwise)
|
|
68
70
|
bun add -g @fastagent-sh/voicenote
|
|
69
71
|
# Windows has no /Volumes mount points; set the recorder drive explicitly
|
|
70
72
|
setx VOICENOTE_RECORD_DIR "E:\RECORD"
|
|
@@ -177,6 +179,10 @@ Changes take effect on the next `vn run`. A legacy `~/.config/voicenote/speakers
|
|
|
177
179
|
6. The summary model (default: pi codex via ChatGPT Plus) reads the raw transcript directly, performing necessary cleanup, speaker restoration, and reconstruction of views/debates/consensus inside the notes-generation stage; if the summary fails, the next `vn run` / `vn run --latest` reuses the saved transcript and retries only the notes generation — no `vn forget` needed
|
|
178
180
|
7. Write notes / metadata; the system makes no archiving decisions — files stay in the configured workspace
|
|
179
181
|
|
|
182
|
+
A failing recording is retried on later runs, but at most **3 times** (whether it fails in transcription or in summarisation, and a run killed mid-job counts too). After that it is marked `Gave up` and left alone, so one broken file can't burn ASR/LLM budget on every scheduler tick — `vn forget <name>` drops the record and re-queues it. Re-queuing is not the same as re-transcribing: if the transcript is already on disk it is reused, so `vn forget` never re-pays for ASR. (`vn forget` takes the run lock, so it refuses while a run is in progress — wait for that run to finish and repeat.)
|
|
183
|
+
|
|
184
|
+
Records whose source file is no longer on the recorder are forgotten on the next scan (and the removal is logged), *unless* they already produced notes or a transcript — that history is kept. This is why swapping recorders, or deleting files from the device, no longer leaves permanent "failed" rows behind.
|
|
185
|
+
|
|
180
186
|
## Output locations
|
|
181
187
|
|
|
182
188
|
The installer defaults to `VOICENOTE_WORKSPACE=~/Documents/meetings`.
|
|
@@ -185,7 +191,7 @@ The installer defaults to `VOICENOTE_WORKSPACE=~/Documents/meetings`.
|
|
|
185
191
|
- Original audio: `${VOICENOTE_WORKSPACE}/_audio/YYYY-MM/`
|
|
186
192
|
- Full transcripts: `${VOICENOTE_WORKSPACE}/_transcripts/YYYY-MM/`
|
|
187
193
|
- Metadata: `${VOICENOTE_WORKSPACE}/_metadata/YYYY-MM/`
|
|
188
|
-
- State: `${VOICENOTE_WORKSPACE}/_state/processed.json`
|
|
194
|
+
- State: `${VOICENOTE_WORKSPACE}/_state/jobs.json` — one record per recording, holding its `state` — where it is in its lifecycle (`queued`, `running`, `done`, `filtered`, `error`, or `gave_up` once retries are spent) — plus a `code` saying why (`summary_failed`, `transcribe_failed`, `interrupted`, `too_small`, …), its attempt count and its output paths. `vn run` is the only writer; `vn jobs` and the GUI dashboard are pure reads of it, so what you see is what will run. A pre-0.18 `processed.json` is converted automatically on the first run and kept as `processed.json.v1.bak`.
|
|
189
195
|
- Index: `${VOICENOTE_WORKSPACE}/_index/notes.jsonl`
|
|
190
196
|
|
|
191
197
|
## Automation
|
|
@@ -223,15 +229,17 @@ bun run typecheck
|
|
|
223
229
|
bun src/cli.ts doctor
|
|
224
230
|
```
|
|
225
231
|
|
|
226
|
-
Distribution: vn ships as **source** with no build step — it only runs on bun (shebang + `bun:ffi` + `engines.bun`), and bun runs TypeScript natively, so `bin` points straight at `src/cli.ts` and the npm tarball only contains `src/{cli,envConfig,runLock}.ts`. The install script / `vn upgrade` install from the published npm package (`bun add -g @fastagent-sh/voicenote`); a `git+https` install also works directly (the git tree carries the source; no build or install script needed).
|
|
232
|
+
Distribution: vn ships as **source** with no build step — it only runs on bun (shebang + `bun:ffi` + `engines.bun`), and bun runs TypeScript natively, so `bin` points straight at `src/cli.ts` and the npm tarball only contains `src/{cli,envConfig,jobs,runLock}.ts`. The install script / `vn upgrade` install from the published npm package (`bun add -g @fastagent-sh/voicenote`); a `git+https` install also works directly (the git tree carries the source; no build or install script needed).
|
|
227
233
|
|
|
228
234
|
Routine release (tag triggers CI):
|
|
229
235
|
|
|
230
236
|
```bash
|
|
231
|
-
npm version patch
|
|
237
|
+
npm version patch # then sync `VERSION` in src/cli.ts to match
|
|
232
238
|
git push --follow-tags
|
|
233
239
|
```
|
|
234
240
|
|
|
241
|
+
`src/cli.ts` hardcodes `VERSION` for `vn --version`, and `npm version` does not touch it — update both in the same commit or the CLI will report a version it isn't.
|
|
242
|
+
|
|
235
243
|
The workflow lives at `.github/workflows/release.yml`: CI explicitly runs typecheck/test/build + an artifact smoke test, then `npm publish --ignore-scripts` (deterministic publishing, no lifecycle dependence). Publishing uses **npm trusted publishing (OIDC)**: no long-lived token (`id-token: write` + a Trusted Publisher configured on npmjs.com), with provenance attached automatically. A bare local `npm publish` is still guarded by `prepublishOnly` (typecheck+test+build).
|
|
236
244
|
|
|
237
245
|
> **First-release exception**: npm has no pending-publisher, so trusted publishing cannot publish a package's very first version. Publish once manually with `npm login` + `npm publish --ignore-scripts`, then add a Trusted Publisher on the package settings page at npmjs.com (repo `fastagent-sh/voicenote`, workflow `release.yml`); CI takes over afterwards (the npm account needs 2FA).
|
package/README.zh-CN.md
CHANGED
|
@@ -53,18 +53,20 @@ bash <(curl -fsSL https://raw.githubusercontent.com/fastagent-sh/voicenote/main/
|
|
|
53
53
|
手动安装:
|
|
54
54
|
|
|
55
55
|
```bash
|
|
56
|
+
bun remove -g @kid7st/voicenote 2>/dev/null || true # 若装过改名前的旧包则清掉(没装则安全跳过)
|
|
56
57
|
bun add -g @fastagent-sh/voicenote
|
|
57
58
|
mkdir -p ~/.local/bin
|
|
58
59
|
ln -sf ~/.bun/bin/vn ~/.local/bin/vn
|
|
59
60
|
```
|
|
60
61
|
|
|
61
|
-
旧版(`git+…#main`)安装会被 `bun add -g @fastagent-sh/voicenote`
|
|
62
|
+
旧版(`git+…#main`)安装会被 `bun add -g @fastagent-sh/voicenote` 直接替换,无需先卸载。唯一例外是**改名前的 `@kid7st/voicenote`** 包:它带同一个 `vn` 命令,所以上面的 `bun remove -g` 会先清掉它(一键 `install.sh` 会自动处理)。
|
|
62
63
|
|
|
63
64
|
### Windows(CLI)
|
|
64
65
|
|
|
65
66
|
CLI 已跨平台。前置:Bun、ffmpeg(提供 `ffprobe.exe`)、Node + pi。
|
|
66
67
|
|
|
67
68
|
```powershell
|
|
69
|
+
bun remove -g @kid7st/voicenote 2>$null # 若装过改名前的旧包则清掉(没装则安全跳过)
|
|
68
70
|
bun add -g @fastagent-sh/voicenote
|
|
69
71
|
# Windows 无 /Volumes 挂载点,录音盘按盘符设置
|
|
70
72
|
setx VOICENOTE_RECORD_DIR "E:\RECORD"
|
|
@@ -175,7 +177,11 @@ vn uninstall-launch-agent
|
|
|
175
177
|
4. 转写:火山豆包【大模型录音文件识别标准版 API】,本地音频先传到 TOS,提交任务后轮询结果,完成后默认删除 TOS 对象
|
|
176
178
|
5. 转写完成后立刻落盘原始 transcript(不做 lossy 清洗),避免后面步骤失败导致 ASR 费用白付
|
|
177
179
|
6. summary 模型(默认 pi codex 走 ChatGPT Plus)直接看原始 transcript,在纪要生成阶段内部完成必要清理、说话人还原、观点/争论/共识形成过程还原;如果 summary 失败,下一次 `vn run` / `vn run --latest` 会复用已保存 transcript,直接重试纪要生成,不需要 `vn forget`
|
|
178
|
-
7. 写出 notes / metadata
|
|
180
|
+
7. 写出 notes / metadata;系统不做任何归档决定,文件留在配置的 workspace 中
|
|
181
|
+
|
|
182
|
+
失败的录音会在后续运行中重试,但**最多 3 次**(转写失败、纪要失败、以及被中途 kill 的运行都算)。超过后标记为 `Gave up` 并不再自动重试,避免一个坏文件每个调度周期都烧一次 ASR/LLM 额度 —— `vn forget <name>` 会删掉该记录并重新入队。重新入队不等于重新转写:磁盘上已有 transcript 时会直接复用,所以 `vn forget` 不会让你再付一次 ASR。(`vn forget` 需要 run lock,因此在某次 run 进行中时会拒绝执行 —— 等该次 run 结束后重试即可。)
|
|
183
|
+
|
|
184
|
+
源文件已不在录音笔上的记录,会在下一次扫描时被遗忘(并记入日志),**已经产出纪要或 transcript 的除外** —— 那部分历史会保留。所以换录音笔、或从设备上删文件,不再会留下永久的 “失败” 条目。
|
|
179
185
|
|
|
180
186
|
## 输出位置
|
|
181
187
|
|
|
@@ -185,7 +191,7 @@ installer 默认设置:`VOICENOTE_WORKSPACE=~/Documents/meetings`。
|
|
|
185
191
|
- 原始音频:`${VOICENOTE_WORKSPACE}/_audio/YYYY-MM/`
|
|
186
192
|
- 完整转写:`${VOICENOTE_WORKSPACE}/_transcripts/YYYY-MM/`
|
|
187
193
|
- metadata:`${VOICENOTE_WORKSPACE}/_metadata/YYYY-MM/`
|
|
188
|
-
-
|
|
194
|
+
- 状态:`${VOICENOTE_WORKSPACE}/_state/jobs.json` —— 每条录音一条记录,包含 `state`(生命周期位置:`queued`、`running`、`done`、`filtered`、`error`,以及重试耗尽后的 `gave_up`)、`code`(原因:`summary_failed`、`transcribe_failed`、`interrupted`、`too_small` 等)、重试次数和产物路径。`vn run` 是唯一的写入方,`vn jobs` 和 GUI 面板都只是它的纯读取 —— 你看到的队列就是会跑的队列。0.18 之前的 `processed.json` 会在首次运行时自动转换,旧文件保留为 `processed.json.v1.bak`。
|
|
189
195
|
- 索引:`${VOICENOTE_WORKSPACE}/_index/notes.jsonl`
|
|
190
196
|
|
|
191
197
|
## 自动化
|
|
@@ -223,15 +229,17 @@ bun run typecheck
|
|
|
223
229
|
bun src/cli.ts doctor
|
|
224
230
|
```
|
|
225
231
|
|
|
226
|
-
分发:vn 以**源码**分发,没有构建步骤 —— 它只在 bun 上运行(shebang + `bun:ffi` + `engines.bun`),而 bun 原生跑 TypeScript,所以 `bin` 直接指向 `src/cli.ts`,npm tarball 只带 `src/{cli,envConfig,runLock}.ts`。安装脚本 / `vn upgrade` 从已发布的 npm 包安装(`bun add -g @fastagent-sh/voicenote`);`git+https` 安装也能直接用(git 树自带源码,无需 build 或安装脚本)。
|
|
232
|
+
分发:vn 以**源码**分发,没有构建步骤 —— 它只在 bun 上运行(shebang + `bun:ffi` + `engines.bun`),而 bun 原生跑 TypeScript,所以 `bin` 直接指向 `src/cli.ts`,npm tarball 只带 `src/{cli,envConfig,jobs,runLock}.ts`。安装脚本 / `vn upgrade` 从已发布的 npm 包安装(`bun add -g @fastagent-sh/voicenote`);`git+https` 安装也能直接用(git 树自带源码,无需 build 或安装脚本)。
|
|
227
233
|
|
|
228
234
|
日常发布(打 tag 触发 CI):
|
|
229
235
|
|
|
230
236
|
```bash
|
|
231
|
-
npm version patch
|
|
237
|
+
npm version patch # 然后把 src/cli.ts 里的 `VERSION` 同步成一样
|
|
232
238
|
git push --follow-tags
|
|
233
239
|
```
|
|
234
240
|
|
|
241
|
+
`src/cli.ts` 里硬编码了 `VERSION`(供 `vn --version` 用),而 `npm version` 不会改它 —— 请在同一个 commit 里一起更新,否则 CLI 会报一个它并不是的版本号。
|
|
242
|
+
|
|
235
243
|
workflow 位于 `.github/workflows/release.yml`:CI 显式跑 typecheck/test/build + 产物冒烟,再 `npm publish --ignore-scripts`(确定发布,不依赖 lifecycle)。发布走 **npm trusted publishing(OIDC)**:免长期 token(`id-token: write` + npmjs.com 上配好 Trusted Publisher),自动带 provenance。本地裸 `npm publish` 则由 `prepublishOnly`(typecheck+test+build)兼底。
|
|
236
244
|
|
|
237
245
|
> **首发例外**:npm 无 pending-publisher,trusted publishing 发不了包的第一个版本。先本机 `npm login` 后手动 `npm publish --ignore-scripts` 发一次,再到 npmjs.com 包设置页加 Trusted Publisher(repo `fastagent-sh/voicenote`、workflow `release.yml`),之后 CI 自动接管(需 npm 账号开 2FA)。
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@fastagent-sh/voicenote",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.18.0",
|
|
4
4
|
"description": "Voice recordings → diarized transcripts → integrated semantic Markdown notes. Currently optimized for the PHILIPS VTR6500 recorder, but the workflow is generic.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
|
@@ -29,6 +29,7 @@
|
|
|
29
29
|
"files": [
|
|
30
30
|
"src/cli.ts",
|
|
31
31
|
"src/envConfig.ts",
|
|
32
|
+
"src/jobs.ts",
|
|
32
33
|
"src/runLock.ts",
|
|
33
34
|
"README.md",
|
|
34
35
|
"LICENSE"
|
package/src/cli.ts
CHANGED
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
import { cac } from 'cac'
|
|
3
3
|
import { deriveNoProxy, envKeysToEmbed, hydrateFromFileEnv, parseFileEnv } from './envConfig'
|
|
4
4
|
import { parseLockOwner } from './runLock'
|
|
5
|
+
import { applyOutcome, buildJobsView, classify, emptyState, localIso, MAX_ATTEMPTS, migrateLegacyState, ownsOutput, parseJobsLimit, parseStateFile, parseStrictJson, patchJob, pruneUnseen, reconcileInterrupted, startAttempt, SUMMARY_FAILED_STATUS, type CurrentJob, type JobRecord, type StateFile } from './jobs'
|
|
5
6
|
import { createHash, createHmac, randomUUID } from 'node:crypto'
|
|
6
7
|
import { appendFile, chmod, mkdir, readFile, writeFile, copyFile, rename, unlink, stat, readdir, rm } from 'node:fs/promises'
|
|
7
8
|
import { existsSync, readFileSync, readdirSync, mkdirSync, writeFileSync, appendFileSync, openSync, closeSync, statSync, readSync, unlinkSync, renameSync } from 'node:fs'
|
|
@@ -11,7 +12,7 @@ import { fileURLToPath, pathToFileURL } from 'node:url'
|
|
|
11
12
|
import { spawn, spawnSync } from 'node:child_process'
|
|
12
13
|
import os from 'node:os'
|
|
13
14
|
|
|
14
|
-
const VERSION = '0.
|
|
15
|
+
const VERSION = '0.18.0'
|
|
15
16
|
const LAUNCH_AGENT_LABEL = 'sh.fastagent.voicenote'
|
|
16
17
|
const LAUNCH_AGENT_LABEL_LEGACY = 'com.kid7st.voicenote' // pre-fastagent installs; cleaned up on install
|
|
17
18
|
const TASK_NAME = 'VoiceNote' // Windows Task Scheduler name (mac uses LAUNCH_AGENT_LABEL)
|
|
@@ -402,25 +403,26 @@ async function readJson<T>(path: string, fallback: T): Promise<T> {
|
|
|
402
403
|
try { return JSON.parse(await readFile(path, 'utf8')) as T } catch (e) { warnSideEffect(`parse ${path}`, e); return fallback }
|
|
403
404
|
}
|
|
404
405
|
|
|
405
|
-
|
|
406
|
+
// Write via tmp+rename so readers only ever see a complete file. Anything whose
|
|
407
|
+
// mere existence is later treated as a signal MUST go through this: a half
|
|
408
|
+
// written file that still parses is worse than no file at all.
|
|
409
|
+
async function writeFileAtomic(path: string, body: string): Promise<void> {
|
|
406
410
|
await mkdir(dirname(path), { recursive: true })
|
|
407
411
|
const tmp = `${path}.tmp`
|
|
408
|
-
await writeFile(tmp,
|
|
412
|
+
await writeFile(tmp, body, 'utf8')
|
|
409
413
|
await rename(tmp, path)
|
|
410
414
|
}
|
|
411
415
|
|
|
416
|
+
const writeJson = (path: string, data: any) => writeFileAtomic(path, JSON.stringify(data, null, 2))
|
|
417
|
+
|
|
412
418
|
async function appendJsonl(path: string, data: any): Promise<void> {
|
|
413
419
|
await mkdir(dirname(path), { recursive: true })
|
|
414
420
|
await appendFile(path, JSON.stringify(data) + '\n', 'utf8')
|
|
415
421
|
}
|
|
416
422
|
|
|
417
|
-
const SUMMARY_FAILED_STATUS = 'summary_failed_transcript_saved'
|
|
418
423
|
const RAW_TRANSCRIPT_MARKER = '## Raw transcript (no lossy cleanup)\n\n'
|
|
419
424
|
const RAW_TRANSCRIPT_MARKER_LEGACY = '## 原始 transcript(不做 lossy 清洗)\n\n' // pre-0.18 files on disk
|
|
420
425
|
|
|
421
|
-
function isSummaryFailedEntry(entry: any): boolean {
|
|
422
|
-
return entry?.status === SUMMARY_FAILED_STATUS
|
|
423
|
-
}
|
|
424
426
|
|
|
425
427
|
|
|
426
428
|
// ────────────────────────────────────────────────────────────────────────────
|
|
@@ -480,6 +482,9 @@ function formatElapsed(ms: number): string {
|
|
|
480
482
|
|
|
481
483
|
function progressStep(step: number, total: number, title: string, detail?: string): void {
|
|
482
484
|
console.log(`▶ Step ${step}/${total}: ${title}${detail ? ` — ${detail}` : ''}`)
|
|
485
|
+
// Single hook for live progress: the dashboard shows the same string the log
|
|
486
|
+
// does, instead of regex-guessing the step from log text.
|
|
487
|
+
reportStep(title)
|
|
483
488
|
}
|
|
484
489
|
|
|
485
490
|
async function withHeartbeat<T>(label: string, work: () => Promise<T>, heartbeatSeconds = 60): Promise<T> {
|
|
@@ -785,42 +790,47 @@ function isCandidateFile(path: string): boolean {
|
|
|
785
790
|
return true
|
|
786
791
|
}
|
|
787
792
|
|
|
788
|
-
|
|
789
|
-
|
|
793
|
+
/**
|
|
794
|
+
* `complete` is false when any part of the listing was lost — the glob threw, or
|
|
795
|
+
* a file we had just seen could not be read. It gates pruning: "not in the scan"
|
|
796
|
+
* only means "gone from the recorder" if the scan actually saw everything, and
|
|
797
|
+
* treating a half-read device as authoritative would delete live queue entries
|
|
798
|
+
* along with their retry counters.
|
|
799
|
+
*/
|
|
800
|
+
async function scanRecordings(config: Config): Promise<{ recordings: Recording[]; complete: boolean }> {
|
|
801
|
+
if (!existsSync(config.recordDir)) return { recordings: [], complete: false }
|
|
790
802
|
const recordings: Recording[] = []
|
|
791
|
-
|
|
792
|
-
|
|
793
|
-
const
|
|
794
|
-
|
|
795
|
-
|
|
796
|
-
|
|
797
|
-
|
|
798
|
-
|
|
799
|
-
|
|
800
|
-
|
|
801
|
-
|
|
802
|
-
|
|
803
|
+
let complete = true
|
|
804
|
+
try {
|
|
805
|
+
for await (const file of new Bun.Glob('**/*').scan({ cwd: config.recordDir, absolute: true, dot: true })) {
|
|
806
|
+
if (!isCandidateFile(file)) continue
|
|
807
|
+
const st = await stat(file).catch(() => null)
|
|
808
|
+
// Listed a moment ago but unreadable now: the device is going away, or
|
|
809
|
+
// this file is. Either way the listing is no longer trustworthy.
|
|
810
|
+
if (!st) { complete = false; continue }
|
|
811
|
+
if (!st.isFile()) continue
|
|
812
|
+
try {
|
|
813
|
+
recordings.push({
|
|
814
|
+
sourcePath: file,
|
|
815
|
+
sizeBytes: st.size,
|
|
816
|
+
modifiedAt: st.mtime.toISOString(),
|
|
817
|
+
durationSeconds: await ffprobeDuration(file),
|
|
818
|
+
sourceId: await sourceIdFor(file),
|
|
819
|
+
recordedAt: parseRecordedAt(file),
|
|
820
|
+
})
|
|
821
|
+
} catch (e) { complete = false; warnSideEffect(`read ${basename(file)} during scan`, e) }
|
|
822
|
+
}
|
|
823
|
+
} catch (e) {
|
|
824
|
+
complete = false
|
|
825
|
+
warnSideEffect('scan recorder', e)
|
|
803
826
|
}
|
|
804
827
|
// Oldest first: backlog is drained in chronological order, so every file is
|
|
805
828
|
// guaranteed a turn before newer arrivals jump the queue.
|
|
806
|
-
|
|
829
|
+
recordings.sort((a, b) => a.recordedAt.getTime() - b.recordedAt.getTime())
|
|
830
|
+
return { recordings, complete }
|
|
807
831
|
}
|
|
808
832
|
|
|
809
|
-
|
|
810
|
-
const processed = state.processed_source_ids?.[rec.sourceId]
|
|
811
|
-
if (processed && !force) {
|
|
812
|
-
// A summary failure is not a completed job: the expensive transcript was
|
|
813
|
-
// saved, so the next normal notes run should resume at summary instead of
|
|
814
|
-
// requiring `vn forget` and paying ASR again.
|
|
815
|
-
if (mode === 'notes' && isSummaryFailedEntry(processed)) return [false, '']
|
|
816
|
-
return [true, 'already_processed']
|
|
817
|
-
}
|
|
818
|
-
const ageHours = (Date.now() - rec.recordedAt.getTime()) / 3600_000
|
|
819
|
-
if (config.maxAgeHours > 0 && ageHours > config.maxAgeHours) return [true, `too_old:${ageHours.toFixed(0)}h>${config.maxAgeHours}h`]
|
|
820
|
-
if (rec.sizeBytes < config.minBytes) return [true, `too_small:${rec.sizeBytes}<${config.minBytes}`]
|
|
821
|
-
if (rec.durationSeconds !== null && rec.durationSeconds < config.minDurationSeconds) return [true, `too_short:${rec.durationSeconds.toFixed(1)}<${config.minDurationSeconds}`]
|
|
822
|
-
return [false, '']
|
|
823
|
-
}
|
|
833
|
+
const limitsOf = (config: Config) => ({ maxAgeHours: config.maxAgeHours, minBytes: config.minBytes, minDurationSeconds: config.minDurationSeconds })
|
|
824
834
|
|
|
825
835
|
// ────────────────────────────────────────────────────────────────────────────
|
|
826
836
|
// File path planning
|
|
@@ -836,9 +846,9 @@ function initialLocalFiles(config: Config, rec: Recording): LocalFiles {
|
|
|
836
846
|
}
|
|
837
847
|
}
|
|
838
848
|
|
|
839
|
-
function localFilesFromState(config: Config, rec: Recording, entry:
|
|
849
|
+
function localFilesFromState(config: Config, rec: Recording, entry: JobRecord | undefined): LocalFiles {
|
|
840
850
|
const fallback = initialLocalFiles(config, rec)
|
|
841
|
-
const paths = entry?.
|
|
851
|
+
const paths = entry?.paths || {}
|
|
842
852
|
return {
|
|
843
853
|
audio: typeof paths.audio === 'string' ? paths.audio : fallback.audio,
|
|
844
854
|
transcript: typeof paths.transcript === 'string' ? paths.transcript : fallback.transcript,
|
|
@@ -847,11 +857,13 @@ function localFilesFromState(config: Config, rec: Recording, entry: any): LocalF
|
|
|
847
857
|
}
|
|
848
858
|
}
|
|
849
859
|
|
|
850
|
-
|
|
860
|
+
// Resume on the evidence, not on a state label: if the transcript is on disk,
|
|
861
|
+
// re-running ASR is money spent for nothing. Keying this off `notes_failed`
|
|
862
|
+
// instead meant `vn forget` (which drops the record) silently re-paid for ASR,
|
|
863
|
+
// even though the transcript was still sitting there.
|
|
864
|
+
function resumableTranscriptFiles(config: Config, rec: Recording, store: StateFile, mode: RunMode, force: boolean): LocalFiles | null {
|
|
851
865
|
if (force || mode !== 'notes') return null
|
|
852
|
-
const
|
|
853
|
-
if (!isSummaryFailedEntry(entry)) return null
|
|
854
|
-
const files = localFilesFromState(config, rec, entry)
|
|
866
|
+
const files = localFilesFromState(config, rec, store.jobs[rec.sourceId])
|
|
855
867
|
return existsSync(files.transcript) ? files : null
|
|
856
868
|
}
|
|
857
869
|
|
|
@@ -1762,7 +1774,11 @@ async function processRecording(config: Config, rec: Recording, opts: any): Prom
|
|
|
1762
1774
|
// if a later step (summary) blows up. We use the initial (untitled) path;
|
|
1763
1775
|
// if summary succeeds we'll move it to the titled path below.
|
|
1764
1776
|
await mkdir(dirname(files.transcript), { recursive: true })
|
|
1765
|
-
|
|
1777
|
+
// Atomic: "transcript exists on disk" is what makes a later run skip ASR, so
|
|
1778
|
+
// a run killed mid-write must not leave a truncated file behind. The raw
|
|
1779
|
+
// marker sits near the top, so a partial write would still pass
|
|
1780
|
+
// readSavedTranscript()'s checks and get summarised as if complete.
|
|
1781
|
+
await writeFileAtomic(files.transcript, transcriptMarkdown(config, rec, transcript, { mode }))
|
|
1766
1782
|
console.log(`✓ Transcript saved: ${files.transcript}`)
|
|
1767
1783
|
}
|
|
1768
1784
|
|
|
@@ -1825,7 +1841,9 @@ async function processRecording(config: Config, rec: Recording, opts: any): Prom
|
|
|
1825
1841
|
console.log(`✓ PDF: ${pdf}`)
|
|
1826
1842
|
}
|
|
1827
1843
|
} else if (needsNotes && summaryError) {
|
|
1828
|
-
|
|
1844
|
+
// No unconditional "just re-run" promise: after MAX_ATTEMPTS the job is
|
|
1845
|
+
// `gave_up` and further runs skip it, so the note has to name both ways out.
|
|
1846
|
+
const stubBody = `# Pending summary: ${basename(rec.sourcePath)}\n\n> ⚠ Transcription completed and saved, but the summary stage failed; retry needed.\n\n- Transcript file: \`${files.transcript}\`\n- Original audio: \`${rec.sourcePath}\`\n- Failure reason: ${meta.summary_error}\n- Retry: the next \`vn run\` reuses the saved transcript automatically (no new transcription cost). After ${MAX_ATTEMPTS} failed attempts it stops retrying — run \`vn forget ${basename(rec.sourcePath)}\` to queue it again.\n`
|
|
1829
1847
|
await writeFile(files.notes, stubBody, 'utf8')
|
|
1830
1848
|
console.log(`⚠ Stub notes (summary failed): ${files.notes}`)
|
|
1831
1849
|
} else if (opts.pdf) {
|
|
@@ -1857,6 +1875,132 @@ async function processRecording(config: Config, rec: Recording, opts: any): Prom
|
|
|
1857
1875
|
return meta
|
|
1858
1876
|
}
|
|
1859
1877
|
|
|
1878
|
+
// ────────────────────────────────────────────────────────────────────────────
|
|
1879
|
+
// Job state — `vn run` is the only writer; every view is a pure read of this.
|
|
1880
|
+
// ────────────────────────────────────────────────────────────────────────────
|
|
1881
|
+
|
|
1882
|
+
// Named for what it holds: every recording's job state, not just the processed
|
|
1883
|
+
// ones. (Pre-0.18 this was `processed.json` with two reason-keyed buckets.)
|
|
1884
|
+
const statePathFor = (config: Config) => join(config.workspace, '_state', 'jobs.json')
|
|
1885
|
+
|
|
1886
|
+
const legacyStatePathFor = (config: Config) => join(config.workspace, '_state', 'processed.json')
|
|
1887
|
+
|
|
1888
|
+
/**
|
|
1889
|
+
* Read-only load. On an un-migrated workspace this converts in memory and does
|
|
1890
|
+
* NOT write: `vn jobs` and the GUI's poll both come through here without the run
|
|
1891
|
+
* lock, and a write from a view could race a live `vn run`. Persisting the
|
|
1892
|
+
* conversion is migrateStateOnDisk()'s job, under the lock.
|
|
1893
|
+
*/
|
|
1894
|
+
// The legacy read is the one irreversible read in the codebase, so it gets the
|
|
1895
|
+
// same strictness as the new format — `readJson` swallows a parse failure and
|
|
1896
|
+
// returns `{}`, which here would mean "nothing was ever processed" and re-pay
|
|
1897
|
+
// for every recording's ASR.
|
|
1898
|
+
async function readLegacyState(config: Config): Promise<Json> {
|
|
1899
|
+
const path = legacyStatePathFor(config)
|
|
1900
|
+
return parseStrictJson(await readFile(path, 'utf8'), path) as Json
|
|
1901
|
+
}
|
|
1902
|
+
|
|
1903
|
+
async function loadState(config: Config): Promise<StateFile> {
|
|
1904
|
+
const path = statePathFor(config)
|
|
1905
|
+
if (!existsSync(path) && existsSync(legacyStatePathFor(config))) {
|
|
1906
|
+
return migrateLegacyState(await readLegacyState(config), nowIso())
|
|
1907
|
+
}
|
|
1908
|
+
const store = existsSync(path) ? parseStateFile(await readFile(path, 'utf8'), path) : emptyState()
|
|
1909
|
+
lastSavedState.set(path, JSON.stringify(store))
|
|
1910
|
+
return store
|
|
1911
|
+
}
|
|
1912
|
+
|
|
1913
|
+
// Inline rather than a repo script: most installs are the GUI's compiled
|
|
1914
|
+
// sidecar, which has no checkout to run a script from — and starting from empty
|
|
1915
|
+
// is not an option, it would re-transcribe everything and pay for ASR twice.
|
|
1916
|
+
// Call only with the run lock held.
|
|
1917
|
+
async function migrateStateOnDisk(config: Config): Promise<void> {
|
|
1918
|
+
const path = statePathFor(config)
|
|
1919
|
+
const legacy = legacyStatePathFor(config)
|
|
1920
|
+
if (existsSync(path) || !existsSync(legacy)) return
|
|
1921
|
+
const store = migrateLegacyState(await readLegacyState(config), nowIso())
|
|
1922
|
+
await writeJson(path, store)
|
|
1923
|
+
lastSavedState.set(path, JSON.stringify(store))
|
|
1924
|
+
await rename(legacy, `${legacy}.v1.bak`).catch(e => warnSideEffect('archive pre-0.18 state', e))
|
|
1925
|
+
console.log(`Converted ${basename(legacy)} → ${basename(path)} (${Object.keys(store.jobs).length} records; old file kept as .v1.bak)`)
|
|
1926
|
+
}
|
|
1927
|
+
|
|
1928
|
+
// The scheduler ticks every 60s and the workspace is often a synced folder
|
|
1929
|
+
// (iCloud/Dropbox). Rewriting an unchanged state file on every tick would be
|
|
1930
|
+
// pure sync noise, so writes are content-gated. Keyed by path, not a single
|
|
1931
|
+
// value: `vn serve` handles config.set (which can move the workspace) and runs
|
|
1932
|
+
// in one process, and a shared key could skip the first write to a new path.
|
|
1933
|
+
const lastSavedState = new Map<string, string>()
|
|
1934
|
+
async function saveState(config: Config, store: StateFile): Promise<void> {
|
|
1935
|
+
const path = statePathFor(config)
|
|
1936
|
+
const serialized = JSON.stringify(store)
|
|
1937
|
+
if (serialized === lastSavedState.get(path)) return
|
|
1938
|
+
await writeJson(path, store)
|
|
1939
|
+
lastSavedState.set(path, serialized)
|
|
1940
|
+
}
|
|
1941
|
+
|
|
1942
|
+
/** Upsert the scan-time facts; never touches lifecycle fields. */
|
|
1943
|
+
function recordFor(store: StateFile, rec: Recording): JobRecord {
|
|
1944
|
+
const existing = store.jobs[rec.sourceId]
|
|
1945
|
+
const next: JobRecord = existing ?? {
|
|
1946
|
+
name: basename(rec.sourcePath), source_path: rec.sourcePath, recorded_at: localIso(rec.recordedAt),
|
|
1947
|
+
size_bytes: rec.sizeBytes, duration_seconds: rec.durationSeconds,
|
|
1948
|
+
state: 'queued', code: null, detail: null, attempts: 0, updated_at: nowIso(), title: null, paths: null,
|
|
1949
|
+
}
|
|
1950
|
+
next.source_path = rec.sourcePath
|
|
1951
|
+
next.size_bytes = rec.sizeBytes
|
|
1952
|
+
next.duration_seconds = rec.durationSeconds
|
|
1953
|
+
store.jobs[rec.sourceId] = next
|
|
1954
|
+
return next
|
|
1955
|
+
}
|
|
1956
|
+
|
|
1957
|
+
function setJobState(store: StateFile, id: string, patch: Partial<JobRecord>): void {
|
|
1958
|
+
const entry = store.jobs[id]
|
|
1959
|
+
if (entry) patchJob(entry, patch, nowIso())
|
|
1960
|
+
}
|
|
1961
|
+
|
|
1962
|
+
// The live job, declared by the run itself. Lives next to run.lock (machine
|
|
1963
|
+
// state, not workspace data) and carries the pid so a reader can tell a live
|
|
1964
|
+
// job from one whose process was killed.
|
|
1965
|
+
const CURRENT_PATH = join(STATE_DIR, 'current.json')
|
|
1966
|
+
|
|
1967
|
+
function writeCurrent(sourceId: string, step: string, startedAt: string): void {
|
|
1968
|
+
try {
|
|
1969
|
+
mkdirSync(STATE_DIR, { recursive: true })
|
|
1970
|
+
// tmp+rename, same rule as writeFileAtomic: this file's existence and
|
|
1971
|
+
// contents are the live-job signal, and progressStep rewrites it at every
|
|
1972
|
+
// step. A truncated write would read back as null and show a running job
|
|
1973
|
+
// as queued.
|
|
1974
|
+
const tmp = `${CURRENT_PATH}.tmp`
|
|
1975
|
+
writeFileSync(tmp, JSON.stringify({ pid: process.pid, source_id: sourceId, step, started_at: startedAt } satisfies CurrentJob))
|
|
1976
|
+
renameSync(tmp, CURRENT_PATH)
|
|
1977
|
+
} catch (e) { warnSideEffect('write current job', e) }
|
|
1978
|
+
}
|
|
1979
|
+
|
|
1980
|
+
function clearCurrent(): void {
|
|
1981
|
+
try { unlinkSync(CURRENT_PATH) } catch (e: any) { if (e?.code !== 'ENOENT') warnSideEffect('clear current job', e) }
|
|
1982
|
+
}
|
|
1983
|
+
|
|
1984
|
+
// Step reporting from inside the pipeline: a job is only "the current job" for
|
|
1985
|
+
// as long as this run says so, so the step is written, never guessed from logs.
|
|
1986
|
+
let currentJobId: string | null = null
|
|
1987
|
+
let currentJobStartedAt = ''
|
|
1988
|
+
function reportStep(step: string): void {
|
|
1989
|
+
if (currentJobId) writeCurrent(currentJobId, step, currentJobStartedAt)
|
|
1990
|
+
}
|
|
1991
|
+
|
|
1992
|
+
function readCurrent(): CurrentJob | null {
|
|
1993
|
+
try {
|
|
1994
|
+
const c = JSON.parse(readFileSync(CURRENT_PATH, 'utf8'))
|
|
1995
|
+
return Number.isFinite(c?.pid) && typeof c?.source_id === 'string' ? c : null
|
|
1996
|
+
} catch { return null }
|
|
1997
|
+
}
|
|
1998
|
+
|
|
1999
|
+
function pidAlive(pid: number): boolean {
|
|
2000
|
+
if (!(pid > 0)) return false
|
|
2001
|
+
try { process.kill(pid, 0); return true } catch (e: any) { return e?.code === 'EPERM' }
|
|
2002
|
+
}
|
|
2003
|
+
|
|
1860
2004
|
async function runPipeline(opts: any): Promise<void> {
|
|
1861
2005
|
wireDailyLog()
|
|
1862
2006
|
const config = getConfig()
|
|
@@ -1874,10 +2018,16 @@ async function runPipeline(opts: any): Promise<void> {
|
|
|
1874
2018
|
|
|
1875
2019
|
async function runPipelineLocked(config: Config, opts: any): Promise<void> {
|
|
1876
2020
|
await ensureDirs(config)
|
|
1877
|
-
|
|
1878
|
-
|
|
1879
|
-
|
|
1880
|
-
|
|
2021
|
+
// --dry-run is a zero-side-effect diagnostic; the migration renames the legacy
|
|
2022
|
+
// file and permanently drops its `error:*` entries. loadState converts in
|
|
2023
|
+
// memory, so a dry run still sees the right picture.
|
|
2024
|
+
if (!opts.dryRun) await migrateStateOnDisk(config)
|
|
2025
|
+
const store = await loadState(config)
|
|
2026
|
+
// We hold the run lock, so nothing else can own a `running` record: any that
|
|
2027
|
+
// survive are debris from a killed run. Their attempt was already counted, so
|
|
2028
|
+
// this is what makes the retry cap cover crashes as well as thrown errors.
|
|
2029
|
+
const interrupted = reconcileInterrupted(store.jobs, nowIso())
|
|
2030
|
+
if (interrupted.length) console.log(`Reclaimed ${interrupted.length} job(s) left running by an interrupted run: ${interrupted.slice(0, 3).map(j => j.name).join(', ')}`)
|
|
1881
2031
|
|
|
1882
2032
|
if (!existsSync(config.recordDir)) {
|
|
1883
2033
|
if (shouldLogIdleStatus(`missing:${config.recordDir}`)) {
|
|
@@ -1885,25 +2035,31 @@ async function runPipelineLocked(config: Config, opts: any): Promise<void> {
|
|
|
1885
2035
|
}
|
|
1886
2036
|
return
|
|
1887
2037
|
}
|
|
1888
|
-
const recordings = await scanRecordings(config)
|
|
2038
|
+
const { recordings, complete: scanComplete } = await scanRecordings(config)
|
|
1889
2039
|
const mode = normalizeRunMode(opts)
|
|
1890
2040
|
const force = Boolean(opts.force)
|
|
1891
2041
|
const eligible: Recording[] = []
|
|
1892
2042
|
const skipCounts: Record<string, number> = {}
|
|
1893
2043
|
const skipSamples: Record<string, string[]> = {}
|
|
1894
2044
|
const verboseSkips = Boolean(opts.verbose || opts.dryRun)
|
|
2045
|
+
const seen = new Set<string>()
|
|
2046
|
+
const limits = limitsOf(config)
|
|
1895
2047
|
for (const rec of recordings) {
|
|
1896
|
-
|
|
1897
|
-
|
|
1898
|
-
|
|
1899
|
-
|
|
1900
|
-
|
|
1901
|
-
|
|
1902
|
-
|
|
1903
|
-
|
|
1904
|
-
|
|
1905
|
-
|
|
1906
|
-
|
|
2048
|
+
seen.add(rec.sourceId)
|
|
2049
|
+
const entry = recordFor(store, rec)
|
|
2050
|
+
const verdict = classify(rec, store.jobs[rec.sourceId], limits, { force, notesMode: mode === 'notes', now: Date.now() })
|
|
2051
|
+
if (verdict.run) { eligible.push(rec); continue }
|
|
2052
|
+
skipCounts[verdict.code] = (skipCounts[verdict.code] || 0) + 1
|
|
2053
|
+
;(skipSamples[verdict.code] ||= []).push(entry.name)
|
|
2054
|
+
if (verdict.persist) setJobState(store, rec.sourceId, { state: 'filtered', code: verdict.code, detail: verdict.detail })
|
|
2055
|
+
if (verboseSkips) console.log(` Skip: ${entry.name} (${verdict.code}${verdict.detail ? `: ${verdict.detail}` : ''})`)
|
|
2056
|
+
}
|
|
2057
|
+
// Only prune against a listing we believe to be complete: if the recorder went
|
|
2058
|
+
// away mid-glob the scan is partial, and pruning would wipe live queue entries
|
|
2059
|
+
// (they'd return on the next scan, but their retry counters would not).
|
|
2060
|
+
const dropped = pruneUnseen(store.jobs, seen, scanComplete && existsSync(config.recordDir))
|
|
2061
|
+
// The only routine path that deletes state — never do it silently.
|
|
2062
|
+
if (dropped.length) console.log(`Forgot ${dropped.length} record(s) whose source is no longer on the recorder: ${dropped.slice(0, 3).map(j => j.name).join(', ')}${dropped.length > 3 ? `…(+${dropped.length - 3})` : ''}`)
|
|
1907
2063
|
const skipSummary = Object.entries(skipCounts).map(([reason, count]) => `${reason}=${count}`).join(', ') || 'none'
|
|
1908
2064
|
const scanLine = `Scan summary: found=${recordings.length}; eligible=${eligible.length}; skipped=${recordings.length - eligible.length} (${skipSummary})`
|
|
1909
2065
|
const samplesLine = !verboseSkips && Object.keys(skipSamples).length
|
|
@@ -1917,7 +2073,7 @@ async function runPipelineLocked(config: Config, opts: any): Promise<void> {
|
|
|
1917
2073
|
// --dry-run, which is a zero-side-effect diagnostic and should still print the
|
|
1918
2074
|
// plan even on an unconfigured machine.
|
|
1919
2075
|
if (targets.length && !opts.dryRun) {
|
|
1920
|
-
const needsAsr = targets.some(rec => !resumableTranscriptFiles(config, rec,
|
|
2076
|
+
const needsAsr = targets.some(rec => !resumableTranscriptFiles(config, rec, store, mode, force))
|
|
1921
2077
|
if (needsAsr && !config.volcano) {
|
|
1922
2078
|
if (shouldLogIdleStatus(`asr-misconfig:${config.recordDir}`)) console.error('ASR not configured: Volcano needs VOLCANO_ASR_KEY / VOLCANO_TOS_*. Skipping; run `vn doctor`, fix config, then re-run.')
|
|
1923
2079
|
return
|
|
@@ -1938,20 +2094,59 @@ async function runPipelineLocked(config: Config, opts: any): Promise<void> {
|
|
|
1938
2094
|
if (samplesLine) console.log(samplesLine)
|
|
1939
2095
|
console.log(`Queue: processing ${targets.length} recording(s)${latestOnly ? ' (--latest)' : ''}. Remaining after this run: ${Math.max(0, eligible.length - targets.length)}`)
|
|
1940
2096
|
}
|
|
2097
|
+
if (opts.dryRun) {
|
|
2098
|
+
// Print the plan and touch nothing: no attempt counted, no state written.
|
|
2099
|
+
for (const rec of targets) {
|
|
2100
|
+
const plan = await processRecording(config, rec, { ...opts, resumeFromTranscriptFiles: resumableTranscriptFiles(config, rec, store, mode, force) })
|
|
2101
|
+
console.log(JSON.stringify(plan, null, 2))
|
|
2102
|
+
}
|
|
2103
|
+
return
|
|
2104
|
+
}
|
|
2105
|
+
await saveState(config, store)
|
|
2106
|
+
|
|
1941
2107
|
for (const rec of targets) {
|
|
2108
|
+
const entry = store.jobs[rec.sourceId]!
|
|
2109
|
+
// --force means "start over", so it refunds the retry budget too. Without
|
|
2110
|
+
// this it only skips one refusal: a spent record would be back at `gave_up`
|
|
2111
|
+
// the moment this attempt failed.
|
|
2112
|
+
if (force) patchJob(entry, { attempts: 0 }, nowIso())
|
|
2113
|
+
currentJobId = rec.sourceId
|
|
2114
|
+
currentJobStartedAt = nowIso()
|
|
2115
|
+
startAttempt(entry, nowIso())
|
|
2116
|
+
await saveState(config, store)
|
|
2117
|
+
writeCurrent(rec.sourceId, 'starting', currentJobStartedAt)
|
|
1942
2118
|
try {
|
|
1943
|
-
const resumeFromTranscriptFiles = resumableTranscriptFiles(config, rec,
|
|
2119
|
+
const resumeFromTranscriptFiles = resumableTranscriptFiles(config, rec, store, mode, force)
|
|
1944
2120
|
const result = await processRecording(config, rec, { ...opts, resumeFromTranscriptFiles })
|
|
1945
|
-
|
|
1946
|
-
|
|
1947
|
-
|
|
1948
|
-
}
|
|
2121
|
+
applyOutcome(entry, result.status === SUMMARY_FAILED_STATUS
|
|
2122
|
+
? { kind: 'summary_failed', title: result.title ?? null, paths: result.final_paths ?? null, message: String(result.summary_error ?? 'summary failed; transcript saved') }
|
|
2123
|
+
: { kind: 'done', title: result.title ?? null, paths: result.final_paths ?? null }, nowIso())
|
|
1949
2124
|
} catch (e: any) {
|
|
1950
|
-
|
|
1951
|
-
|
|
2125
|
+
const message = String(e?.message || e)
|
|
2126
|
+
console.error(`ERROR processing ${rec.sourcePath}: ${message}`)
|
|
2127
|
+
// Source vanished mid-run (recorder unplugged, file deleted) AND nothing
|
|
2128
|
+
// was produced: that's not a failed job, it's a job that no longer exists.
|
|
2129
|
+
// Drop it so it can't linger as a permanent "failed" row. A record that
|
|
2130
|
+
// already owns output is history — same rule pruneUnseen follows — and
|
|
2131
|
+
// deleting it would re-pay for ASR when the recorder comes back.
|
|
2132
|
+
if (!existsSync(rec.sourcePath) && !ownsOutput(entry)) {
|
|
2133
|
+
console.log(`Forgot ${entry.name}: source left the recorder before it produced anything`)
|
|
2134
|
+
delete store.jobs[rec.sourceId]
|
|
2135
|
+
} else {
|
|
2136
|
+
applyOutcome(entry, { kind: 'failed', message }, nowIso())
|
|
2137
|
+
}
|
|
2138
|
+
} finally {
|
|
2139
|
+
currentJobId = null
|
|
2140
|
+
clearCurrent()
|
|
2141
|
+
await saveState(config, store) // per job, not per batch: a kill -9 costs one job, not the batch
|
|
2142
|
+
}
|
|
2143
|
+
// Whole recorder went away — every remaining target would fail the same way
|
|
2144
|
+
// and churn ASR-free but noisy retries. Stop and let the next run rescan.
|
|
2145
|
+
if (!existsSync(config.recordDir)) {
|
|
2146
|
+
console.error(`Recorder disappeared mid-run (${config.recordDir}); stopping. Remaining recordings stay queued.`)
|
|
2147
|
+
break
|
|
1952
2148
|
}
|
|
1953
2149
|
}
|
|
1954
|
-
if (!opts.dryRun) await writeJson(statePath, state)
|
|
1955
2150
|
}
|
|
1956
2151
|
|
|
1957
2152
|
// ────────────────────────────────────────────────────────────────────────────
|
|
@@ -2316,20 +2511,24 @@ async function openTarget(arg?: string): Promise<void> {
|
|
|
2316
2511
|
|
|
2317
2512
|
async function forgetRecording(needle: string): Promise<void> {
|
|
2318
2513
|
const config = getConfig()
|
|
2319
|
-
|
|
2320
|
-
|
|
2321
|
-
|
|
2322
|
-
|
|
2323
|
-
|
|
2324
|
-
|
|
2325
|
-
|
|
2326
|
-
|
|
2514
|
+
// Under the run lock: `vn run` holds the state file in memory for the length
|
|
2515
|
+
// of a batch and re-saves after every job, so an unlocked delete here would be
|
|
2516
|
+
// silently resurrected by the next save.
|
|
2517
|
+
const lock = await acquireRunLock()
|
|
2518
|
+
if (!lock) { console.error('A voicenote run is in progress, so the state file is busy. Re-run this once it finishes (`vn jobs` shows what it is working on).'); process.exitCode = 1; return }
|
|
2519
|
+
try {
|
|
2520
|
+
await migrateStateOnDisk(config)
|
|
2521
|
+
const store = await loadState(config)
|
|
2522
|
+
let removed = 0
|
|
2523
|
+
for (const [id, entry] of Object.entries(store.jobs)) {
|
|
2524
|
+
if (id === needle || entry.source_path.includes(needle) || entry.name.includes(needle)) {
|
|
2525
|
+
delete store.jobs[id]
|
|
2327
2526
|
removed++
|
|
2328
2527
|
}
|
|
2329
2528
|
}
|
|
2330
|
-
|
|
2331
|
-
|
|
2332
|
-
|
|
2529
|
+
await saveState(config, store)
|
|
2530
|
+
console.log(`forgot ${removed} record(s)`)
|
|
2531
|
+
} finally { await lock.release() }
|
|
2333
2532
|
}
|
|
2334
2533
|
|
|
2335
2534
|
async function showLog(opts: { lines?: number; follow?: boolean; err?: boolean; date?: string }): Promise<void> {
|
|
@@ -2524,102 +2723,36 @@ async function collectDoctor() {
|
|
|
2524
2723
|
}
|
|
2525
2724
|
}
|
|
2526
2725
|
|
|
2527
|
-
//
|
|
2528
|
-
//
|
|
2529
|
-
//
|
|
2530
|
-
|
|
2531
|
-
|
|
2532
|
-
let name: string | null = null
|
|
2533
|
-
let processing = false
|
|
2534
|
-
let step = 'Preparing'
|
|
2535
|
-
for (const line of tail) {
|
|
2536
|
-
const m = line.match(/voicenote job:\s*(.+?)\s*===/)
|
|
2537
|
-
if (m) { name = m[1]!; processing = true; step = 'Preparing'; continue }
|
|
2538
|
-
if (/✓ Completed|Idle:|ERROR processing/i.test(line)) processing = false
|
|
2539
|
-
if (processing) {
|
|
2540
|
-
if (/Step 3|integrated semantic notes|generate/i.test(line)) step = 'Generating notes'
|
|
2541
|
-
else if (/Step 2|Transcribe|Volcano|transcrib/i.test(line)) step = 'Transcribing'
|
|
2542
|
-
else if (/Step 1|Copy audio/i.test(line)) step = 'Preparing'
|
|
2543
|
-
}
|
|
2544
|
-
}
|
|
2545
|
-
return processing && name ? { status: 'processing', name, step } : null
|
|
2546
|
-
}
|
|
2547
|
-
|
|
2548
|
-
// Unified processing status of recent recordings (for the GUI status board):
|
|
2549
|
-
// the live job (if any) + queued on the recorder (pending) + completed (done) +
|
|
2550
|
-
// transcript-saved-but-notes-failed (summary_failed) + errored (failed, will
|
|
2551
|
-
// auto-retry) + filtered (skipped: too_small/too_short).
|
|
2552
|
-
async function jobsListData(limit: number): Promise<{ items: Json[] }> {
|
|
2726
|
+
// The dashboard/CLI view of every recording's processing status. A pure read of
|
|
2727
|
+
// the state file `vn run` writes, grouped by jobs.ts. Nothing here rescans
|
|
2728
|
+
// the recorder or parses logs: the queue shown IS the queue that runs, and it
|
|
2729
|
+
// stays visible when the recorder is unplugged.
|
|
2730
|
+
async function jobsListData(limit: number): Promise<{ items: Json[]; total: number; queued_total: number; recorder_present: boolean }> {
|
|
2553
2731
|
const config = getConfig()
|
|
2554
|
-
const
|
|
2555
|
-
|
|
2556
|
-
//
|
|
2557
|
-
|
|
2558
|
-
const recTime = (name: string, at: string | null): string | null => {
|
|
2559
|
-
const m = name.match(/(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})/)
|
|
2560
|
-
if (m) return `${m[1]}-${m[2]}-${m[3]} ${m[4]}:${m[5]}`
|
|
2561
|
-
return at ? at.slice(0, 16).replace('T', ' ') : null
|
|
2562
|
-
}
|
|
2563
|
-
const items: Json[] = []
|
|
2564
|
-
const live = currentJobFromLog()
|
|
2565
|
-
if (live) items.push({ ...live, title: null, at: null, time: recTime(live.name, null), notes: null })
|
|
2566
|
-
const done: Json[] = []
|
|
2567
|
-
const knownPaths = new Set<string>()
|
|
2568
|
-
for (const [id, e] of Object.entries<any>(state.processed_source_ids || {})) {
|
|
2569
|
-
if (e.source_path) knownPaths.add(e.source_path)
|
|
2570
|
-
const name = basename(e.source_path || id)
|
|
2571
|
-
done.push({ status: isSummaryFailedEntry(e) ? 'summary_failed' : 'done', name, title: e.title ?? null, at: e.processed_at ?? null, time: recTime(name, e.processed_at ?? null), notes: e.final_paths?.notes ?? e.local_paths?.notes ?? null })
|
|
2572
|
-
}
|
|
2573
|
-
for (const [id, e] of Object.entries<any>(state.skipped_source_ids || {})) {
|
|
2574
|
-
if (e.source_path) knownPaths.add(e.source_path)
|
|
2575
|
-
const name = basename(e.source_path || id)
|
|
2576
|
-
const rawReason = String(e.reason || '')
|
|
2577
|
-
const isError = rawReason.startsWith('error')
|
|
2578
|
-
// Filter reasons are machine diagnostics (`too_small:1234<100000`); show a
|
|
2579
|
-
// human label instead. Error strings stay raw — that's the diagnostic.
|
|
2580
|
-
const reason = rawReason.startsWith('too_small') ? 'Recording too small, skipped' : rawReason.startsWith('too_short') ? 'Recording too short, skipped' : rawReason.startsWith('too_old') ? 'Recording too old, skipped' : (e.reason ?? null)
|
|
2581
|
-
done.push({ status: isError ? 'failed' : 'skipped', name, title: null, at: e.seen_at ?? null, time: recTime(name, e.seen_at ?? null), reason, notes: null })
|
|
2582
|
-
}
|
|
2583
|
-
done.sort((a, b) => String(b.at || '').localeCompare(String(a.at || '')))
|
|
2584
|
-
// Pending: candidate files on the recorder with no state entry yet. Matched by
|
|
2585
|
-
// source_path (not sourceId) so a status poll doesn't hash every file on the
|
|
2586
|
-
// recorder. ponytail: path match misses a re-recorded same-path file, and the
|
|
2587
|
-
// minDurationSeconds filter is absent (ffprobe per poll is too dear) — a
|
|
2588
|
-
// too-short file shows "pending" until a run flips it to "skipped". Fine for a
|
|
2589
|
-
// status view — the pipeline itself still dedupes by content hash and filters
|
|
2590
|
-
// by duration.
|
|
2591
|
-
const pending: Json[] = []
|
|
2592
|
-
if (existsSync(config.recordDir)) {
|
|
2593
|
-
for await (const file of new Bun.Glob('**/*').scan({ cwd: config.recordDir, absolute: true, dot: true })) {
|
|
2594
|
-
if (!isCandidateFile(file) || knownPaths.has(file)) continue
|
|
2595
|
-
if (config.maxAgeHours > 0 && Date.now() - parseRecordedAt(file).getTime() > config.maxAgeHours * 3600_000) continue
|
|
2596
|
-
const st = await stat(file).catch(() => null)
|
|
2597
|
-
if (!st?.isFile() || st.size < config.minBytes) continue
|
|
2598
|
-
const name = basename(file)
|
|
2599
|
-
pending.push({ status: 'pending', name, title: null, at: null, time: recTime(name, null), notes: null })
|
|
2600
|
-
}
|
|
2601
|
-
// Queue order: oldest first, same sort key as the pipeline (parseRecordedAt).
|
|
2602
|
-
pending.sort((a, b) => parseRecordedAt(String(a.name)).getTime() - parseRecordedAt(String(b.name)).getTime())
|
|
2603
|
-
// Cap the queue so a large backlog can't crowd done/failed out of the limit
|
|
2604
|
-
// window; the overflow collapses into one aggregate row.
|
|
2605
|
-
const PENDING_SHOWN = 10
|
|
2606
|
-
if (pending.length > PENDING_SHOWN) {
|
|
2607
|
-
const extra = pending.length - PENDING_SHOWN
|
|
2608
|
-
pending.length = PENDING_SHOWN
|
|
2609
|
-
pending.push({ status: 'pending', name: `…${extra} more queued`, title: null, at: null, time: null, notes: null })
|
|
2610
|
-
}
|
|
2611
|
-
}
|
|
2612
|
-
// Don't double-list the live job if it's also in pending/done.
|
|
2613
|
-
const liveName = live?.name
|
|
2614
|
-
for (const j of [...pending, ...done]) { if (liveName && j.name === liveName) continue; items.push(j) }
|
|
2615
|
-
return { items: items.slice(0, limit) }
|
|
2732
|
+
const store = await loadState(config)
|
|
2733
|
+
// One existsSync on the mount point — not the recursive glob the old pending
|
|
2734
|
+
// section ran on every poll, and always current.
|
|
2735
|
+
return buildJobsView(store, readCurrent(), { limit, alive: pidAlive, recorderPresent: existsSync(config.recordDir) })
|
|
2616
2736
|
}
|
|
2617
2737
|
|
|
2618
2738
|
async function jobsList(opts: { limit?: number; json?: boolean }): Promise<void> {
|
|
2619
|
-
|
|
2739
|
+
let limit: number
|
|
2740
|
+
try { limit = parseJobsLimit(opts.limit, 30) } catch (e: any) { console.error(e.message); process.exitCode = 1; return }
|
|
2741
|
+
const data = await jobsListData(limit)
|
|
2620
2742
|
if (opts.json) { console.log(JSON.stringify(data, null, 2)); return }
|
|
2621
|
-
if (!data.items.length) {
|
|
2622
|
-
|
|
2743
|
+
if (!data.items.length) {
|
|
2744
|
+
console.log(data.recorder_present ? 'No jobs yet.' : 'No jobs yet. (recorder not connected)')
|
|
2745
|
+
return
|
|
2746
|
+
}
|
|
2747
|
+
for (const j of data.items) {
|
|
2748
|
+
const suffix = [j.step, j.detail].filter(Boolean).join(' \u00b7 ')
|
|
2749
|
+
console.log(`[${j.status}] ${j.title || j.name}${suffix ? ' \u00b7 ' + suffix.slice(0, 140) : ''}`)
|
|
2750
|
+
}
|
|
2751
|
+
// Truncation used to be silent, which is how a 126-entry backlog read as 27.
|
|
2752
|
+
if (data.total > data.items.length) console.log(`\u2026 ${data.total - data.items.length} more (vn jobs --limit 0 to show all)`)
|
|
2753
|
+
if (!data.recorder_present) {
|
|
2754
|
+
console.log(data.queued_total ? `Recorder not connected \u2014 ${data.queued_total} recording(s) waiting for it.` : 'Recorder not connected.')
|
|
2755
|
+
}
|
|
2623
2756
|
}
|
|
2624
2757
|
|
|
2625
2758
|
async function doctor(opts: { json?: boolean } = {}): Promise<void> {
|
|
@@ -2691,7 +2824,7 @@ async function dispatchServe(req: any, send: (o: unknown) => void): Promise<void
|
|
|
2691
2824
|
case 'config.get': result = configGetData(); break
|
|
2692
2825
|
case 'config.set': result = await configSetData(params || {}); break
|
|
2693
2826
|
case 'doctor': result = await collectDoctor(); break
|
|
2694
|
-
case 'jobs': result = await jobsListData(
|
|
2827
|
+
case 'jobs': result = await jobsListData(parseJobsLimit(params?.limit, 40)); break
|
|
2695
2828
|
case 'ensure_agent': result = await ensureScheduler(!!params?.force); break
|
|
2696
2829
|
case 'run': {
|
|
2697
2830
|
// Long-running (minutes) like login: ack immediately so the GUI's 60s
|
|
@@ -2796,7 +2929,7 @@ cli.command('list', 'List notes in a month')
|
|
|
2796
2929
|
.action(listMeetings)
|
|
2797
2930
|
|
|
2798
2931
|
cli.command('last', 'Print summary of most recent processed recording').action(lastMeeting)
|
|
2799
|
-
cli.command('jobs', 'Show processing status
|
|
2932
|
+
cli.command('jobs', 'Show every recording\'s processing status (running, queued, done, failed, gave up, filtered)')
|
|
2800
2933
|
.option('--limit <n>', 'How many to list', { default: 30 })
|
|
2801
2934
|
.option('--json', 'Output as JSON (for the GUI)')
|
|
2802
2935
|
.action((opts: { limit?: number; json?: boolean }) => jobsList(opts))
|
|
@@ -2804,7 +2937,7 @@ cli.command('jobs', 'Show processing status of recordings (live + pending + done
|
|
|
2804
2937
|
|
|
2805
2938
|
cli.command('open [target]', 'Open notes dir, config dir (`config`), logs dir (`logs`), or a note matching the slug').action((target?: string) => openTarget(target))
|
|
2806
2939
|
|
|
2807
|
-
cli.command('forget <key>', '
|
|
2940
|
+
cli.command('forget <key>', 'Drop a recording\'s job record so it is queued again (a saved transcript on disk is still reused)').action((key: string) => forgetRecording(key))
|
|
2808
2941
|
|
|
2809
2942
|
cli.command('log', 'Print the daily log (today by default)')
|
|
2810
2943
|
.option('--lines <n>', 'How many trailing lines to print', { default: 30 })
|
package/src/jobs.ts
ADDED
|
@@ -0,0 +1,437 @@
|
|
|
1
|
+
// The job-state model: what should run (classify), what should be forgotten
|
|
2
|
+
// (pruneUnseen), and how it all reads back (buildJobsView).
|
|
3
|
+
//
|
|
4
|
+
// `vn jobs` used to assemble one view from three unrelated sources — a regex
|
|
5
|
+
// over the launchd log (live), a second glob of the recorder (pending), and the
|
|
6
|
+
// state file (everything else) — so the three could never agree. Now `vn run` is
|
|
7
|
+
// the only writer and every view is a pure read. Keeping the *decisions* here
|
|
8
|
+
// too (not just the grouping) is deliberate: the queue you see and the queue
|
|
9
|
+
// that runs must come from one function, or they drift apart again.
|
|
10
|
+
//
|
|
11
|
+
// No fs, no process, no clock — all injected — so every rule is testable
|
|
12
|
+
// (see jobs.test.ts).
|
|
13
|
+
//
|
|
14
|
+
// The invariant worth naming: a record left in `running` by a process that is
|
|
15
|
+
// no longer alive is NOT running. Liveness is a pid check injected by the
|
|
16
|
+
// caller, never inferred from log text — that inference is what used to wedge
|
|
17
|
+
// a failed job at "Processing" forever.
|
|
18
|
+
|
|
19
|
+
// Lifecycle position only. WHY a job is where it is lives in `code`, so no two
|
|
20
|
+
// fields have to agree about the same fact — an earlier cut expressed "gave up"
|
|
21
|
+
// as `code` while leaving `state` alone, and the view and the classifier
|
|
22
|
+
// promptly disagreed about what such a record was.
|
|
23
|
+
export type JobState = 'queued' | 'running' | 'done' | 'filtered' | 'error' | 'gave_up'
|
|
24
|
+
|
|
25
|
+
/** Which stage produced a failure; also carries the filter reason. */
|
|
26
|
+
export type JobCode = 'transcribe_failed' | 'summary_failed' | 'interrupted' | 'too_small' | 'too_short' | 'too_old' | null
|
|
27
|
+
|
|
28
|
+
export type JobRecord = {
|
|
29
|
+
name: string
|
|
30
|
+
source_path: string
|
|
31
|
+
/** Local wall-clock `YYYY-MM-DDTHH:mm:ss` — sorts lexicographically, no TZ drift. */
|
|
32
|
+
recorded_at: string
|
|
33
|
+
size_bytes: number
|
|
34
|
+
duration_seconds: number | null
|
|
35
|
+
state: JobState
|
|
36
|
+
code: JobCode
|
|
37
|
+
detail: string | null
|
|
38
|
+
attempts: number
|
|
39
|
+
updated_at: string
|
|
40
|
+
title: string | null
|
|
41
|
+
paths: Record<string, string | null> | null
|
|
42
|
+
}
|
|
43
|
+
|
|
44
|
+
export type StateFile = {
|
|
45
|
+
version: 2
|
|
46
|
+
jobs: Record<string, JobRecord>
|
|
47
|
+
}
|
|
48
|
+
|
|
49
|
+
/**
|
|
50
|
+
* Does this record own files on disk? Evidence, not a state enum: a
|
|
51
|
+
* `summary_failed` record has a transcript and a stub note, and enumerating
|
|
52
|
+
* states would have to remember that. Deleting such a record loses real work
|
|
53
|
+
* and re-pays for ASR when the recorder comes back.
|
|
54
|
+
*/
|
|
55
|
+
export const ownsOutput = (j: JobRecord): boolean => j.paths != null
|
|
56
|
+
|
|
57
|
+
// Give up auto-retrying a recording that keeps blowing up, so a permanently
|
|
58
|
+
// broken file can't burn an ASR (or LLM) call every scheduler tick. `vn forget`
|
|
59
|
+
// drops the record and puts it back in the queue.
|
|
60
|
+
export const MAX_ATTEMPTS = 3
|
|
61
|
+
|
|
62
|
+
/** The subset of JobCode a refusal may write to disk. */
|
|
63
|
+
export type FilterCode = Extract<JobCode, 'too_small' | 'too_short' | 'too_old'>
|
|
64
|
+
|
|
65
|
+
// Split by whether the refusal is persisted, so the code that reaches disk is
|
|
66
|
+
// typed as such. A single `code: string` needed a cast at the write site, and
|
|
67
|
+
// the cast was the only thing keeping a display-only reason out of the record.
|
|
68
|
+
export type Verdict =
|
|
69
|
+
| { run: true }
|
|
70
|
+
| { run: false; persist: true; code: FilterCode; detail: string | null }
|
|
71
|
+
| { run: false; persist: false; code: 'already_done' | 'gave_up'; detail: string | null }
|
|
72
|
+
|
|
73
|
+
export type ScanFacts = { recordedAt: Date; sizeBytes: number; durationSeconds: number | null }
|
|
74
|
+
export type Limits = { maxAgeHours: number; minBytes: number; minDurationSeconds: number }
|
|
75
|
+
|
|
76
|
+
/**
|
|
77
|
+
* The single answer to "should this recording run now, and in what form".
|
|
78
|
+
* `vn jobs` reads back what this decided instead of re-deriving it with looser
|
|
79
|
+
* rules, which is why the shown queue and the real queue can no longer disagree.
|
|
80
|
+
*/
|
|
81
|
+
export function classify(
|
|
82
|
+
rec: ScanFacts,
|
|
83
|
+
entry: JobRecord | undefined,
|
|
84
|
+
limits: Limits,
|
|
85
|
+
opts: { force: boolean; notesMode: boolean; now: number },
|
|
86
|
+
): Verdict {
|
|
87
|
+
if (opts.force) return { run: true }
|
|
88
|
+
if (entry?.state === 'done') return { run: false, persist: false, code: 'already_done', detail: null }
|
|
89
|
+
// Only the summary is outstanding, and this run doesn't make summaries. The
|
|
90
|
+
// transcription stage is genuinely finished — re-running it would pay for ASR
|
|
91
|
+
// again and then mark the job `done` with no notes, permanently.
|
|
92
|
+
if (entry?.code === 'summary_failed' && !opts.notesMode) return { run: false, persist: false, code: 'already_done', detail: null }
|
|
93
|
+
// Retries are spent. reconcileInterrupted() is what puts a record here, so
|
|
94
|
+
// this branch needs no knowledge of *how* the attempts were used up.
|
|
95
|
+
if (entry?.state === 'gave_up') return { run: false, persist: false, code: 'gave_up', detail: entry.detail }
|
|
96
|
+
|
|
97
|
+
// Filters are deterministic properties of the file, checked before anything
|
|
98
|
+
// stateful so a too-short file can't ping-pong between error and queued.
|
|
99
|
+
const ageHours = (opts.now - rec.recordedAt.getTime()) / 3600_000
|
|
100
|
+
if (limits.maxAgeHours > 0 && ageHours > limits.maxAgeHours) return { run: false, persist: true, code: 'too_old', detail: `${ageHours.toFixed(0)}h > ${limits.maxAgeHours}h` }
|
|
101
|
+
if (rec.sizeBytes < limits.minBytes) return { run: false, persist: true, code: 'too_small', detail: `${rec.sizeBytes} < ${limits.minBytes} bytes` }
|
|
102
|
+
if (rec.durationSeconds !== null && rec.durationSeconds < limits.minDurationSeconds) return { run: false, persist: true, code: 'too_short', detail: `${rec.durationSeconds.toFixed(0)}s < ${limits.minDurationSeconds}s` }
|
|
103
|
+
|
|
104
|
+
// `error`, `queued` and `running` are retryable — `error` used to be terminal,
|
|
105
|
+
// which is how 127 dead entries piled up without a single retry. Anything else
|
|
106
|
+
// came off disk hand-edited or from a newer build: refuse it rather than run
|
|
107
|
+
// it, so the scheduler and the view (which shows it as unrecognised) agree.
|
|
108
|
+
if (entry && !RUNNABLE_STATES.has(entry.state)) {
|
|
109
|
+
return { run: false, persist: false, code: 'gave_up', detail: `Unrecognised state '${entry.state}'; \`vn forget ${entry.name}\` to start over` }
|
|
110
|
+
}
|
|
111
|
+
return { run: true }
|
|
112
|
+
}
|
|
113
|
+
|
|
114
|
+
const RUNNABLE_STATES = new Set<JobState>(['queued', 'running', 'error'])
|
|
115
|
+
|
|
116
|
+
/** Every way a started attempt can end. */
|
|
117
|
+
export type Outcome =
|
|
118
|
+
| { kind: 'done'; title: string | null; paths: Record<string, string | null> | null }
|
|
119
|
+
| { kind: 'summary_failed'; title: string | null; paths: Record<string, string | null> | null; message: string }
|
|
120
|
+
| { kind: 'failed'; message: string }
|
|
121
|
+
| { kind: 'interrupted' }
|
|
122
|
+
|
|
123
|
+
/**
|
|
124
|
+
* The single place a started attempt is turned back into a record. It lives
|
|
125
|
+
* next to classify() and buildJobsView() on purpose: `running` used to be
|
|
126
|
+
* interpreted independently by all three, so they disagreed about what an
|
|
127
|
+
* interrupted-and-spent job was — the view called it "Queued" while the
|
|
128
|
+
* classifier refused to ever run it again.
|
|
129
|
+
*/
|
|
130
|
+
export function applyOutcome(entry: JobRecord, outcome: Outcome, now: string): void {
|
|
131
|
+
const spent = entry.attempts >= MAX_ATTEMPTS
|
|
132
|
+
const giveUp = (code: JobCode, why: string) => patchJob(entry, {
|
|
133
|
+
state: spent ? 'gave_up' : 'error',
|
|
134
|
+
code,
|
|
135
|
+
detail: spent
|
|
136
|
+
? `${why} — gave up after ${entry.attempts} attempts; \`vn forget ${entry.name}\` to retry`
|
|
137
|
+
: `${why} (attempt ${entry.attempts}/${MAX_ATTEMPTS})`,
|
|
138
|
+
}, now)
|
|
139
|
+
|
|
140
|
+
switch (outcome.kind) {
|
|
141
|
+
case 'done':
|
|
142
|
+
// Only a clean finish refunds the budget.
|
|
143
|
+
patchJob(entry, { state: 'done', code: null, detail: null, title: outcome.title, paths: outcome.paths, attempts: 0 }, now)
|
|
144
|
+
return
|
|
145
|
+
case 'summary_failed':
|
|
146
|
+
// The expensive transcript is on disk; keep its paths so a retry resumes
|
|
147
|
+
// there. Retrying notes re-runs the LLM, so it spends from the same budget.
|
|
148
|
+
patchJob(entry, { title: outcome.title, paths: outcome.paths }, now)
|
|
149
|
+
giveUp('summary_failed', outcome.message)
|
|
150
|
+
return
|
|
151
|
+
case 'failed':
|
|
152
|
+
giveUp('transcribe_failed', outcome.message)
|
|
153
|
+
return
|
|
154
|
+
case 'interrupted':
|
|
155
|
+
giveUp('interrupted', 'Run was interrupted before this job reported back')
|
|
156
|
+
return
|
|
157
|
+
}
|
|
158
|
+
}
|
|
159
|
+
|
|
160
|
+
/** Open an attempt. Counted here, not at the end: a run killed mid-job (kill -9,
|
|
161
|
+
* OOM, SIGTERM) never reaches an end, and an uncounted attempt retries forever. */
|
|
162
|
+
export function startAttempt(entry: JobRecord, now: string): void {
|
|
163
|
+
patchJob(entry, { state: 'running', code: null, detail: null, attempts: entry.attempts + 1 }, now)
|
|
164
|
+
}
|
|
165
|
+
|
|
166
|
+
/**
|
|
167
|
+
* Reclaim records left `running` by a dead run. Safe to do wholesale because the
|
|
168
|
+
* caller holds the run lock: no other run can own a `running` record right now.
|
|
169
|
+
*/
|
|
170
|
+
export function reconcileInterrupted(jobs: Record<string, JobRecord>, now: string): JobRecord[] {
|
|
171
|
+
const stale = Object.values(jobs).filter(j => j.state === 'running')
|
|
172
|
+
for (const j of stale) applyOutcome(j, { kind: 'interrupted' }, now)
|
|
173
|
+
return stale
|
|
174
|
+
}
|
|
175
|
+
|
|
176
|
+
/**
|
|
177
|
+
* Convert the pre-0.18 layout (two reason-keyed buckets) to the one-map form.
|
|
178
|
+
* Pure so the one irreversible step in this codebase is testable: `error:*`
|
|
179
|
+
* entries are dropped, and they never come back.
|
|
180
|
+
*/
|
|
181
|
+
export function migrateLegacyState(raw: Record<string, any>, now: string): StateFile {
|
|
182
|
+
const jobs: Record<string, JobRecord> = {}
|
|
183
|
+
const nameOf = (path: string, id: string) => String(path || id).split(/[/\\]/).pop()!
|
|
184
|
+
const recordedAt = (name: string, fallback: string | undefined): string => {
|
|
185
|
+
const m = name.match(/(20\d{2})(\d{2})(\d{2})(\d{2})(\d{2})(\d{2})/)
|
|
186
|
+
if (m) return `${m[1]}-${m[2]}-${m[3]}T${m[4]}:${m[5]}:${m[6]}`
|
|
187
|
+
const d = fallback ? new Date(fallback) : null
|
|
188
|
+
return d && !Number.isNaN(+d) ? localIso(d) : '1970-01-01T00:00:00'
|
|
189
|
+
}
|
|
190
|
+
for (const [id, e] of Object.entries<any>(raw.processed_source_ids ?? {})) {
|
|
191
|
+
const name = nameOf(e.source_path, id)
|
|
192
|
+
jobs[id] = {
|
|
193
|
+
name, source_path: e.source_path ?? '', recorded_at: recordedAt(name, e.processed_at),
|
|
194
|
+
size_bytes: e.size_bytes ?? 0, duration_seconds: e.duration_seconds ?? null,
|
|
195
|
+
state: e.status === SUMMARY_FAILED_STATUS ? 'error' : 'done',
|
|
196
|
+
code: e.status === SUMMARY_FAILED_STATUS ? 'summary_failed' : null,
|
|
197
|
+
detail: e.status === SUMMARY_FAILED_STATUS ? 'Summary failed before 0.18; the saved transcript will be reused' : null,
|
|
198
|
+
attempts: 0, updated_at: e.processed_at ?? now,
|
|
199
|
+
title: e.title ?? null, paths: e.final_paths ?? e.local_paths ?? null,
|
|
200
|
+
}
|
|
201
|
+
}
|
|
202
|
+
for (const [id, e] of Object.entries<any>(raw.skipped_source_ids ?? {})) {
|
|
203
|
+
const reason = String(e.reason ?? '')
|
|
204
|
+
// `error:*` entries were scan artifacts, not jobs — overwhelmingly ENOENT
|
|
205
|
+
// from a recorder unplugged mid-run. Dropping them re-queues whatever is
|
|
206
|
+
// still on the device and forgets the rest.
|
|
207
|
+
if (reason.startsWith('error')) continue
|
|
208
|
+
const code = reason.split(':')[0] as JobCode
|
|
209
|
+
const name = nameOf(e.source_path, id)
|
|
210
|
+
jobs[id] = {
|
|
211
|
+
name, source_path: e.source_path ?? '', recorded_at: recordedAt(name, e.seen_at),
|
|
212
|
+
size_bytes: e.size_bytes ?? 0, duration_seconds: e.duration_seconds ?? null,
|
|
213
|
+
state: 'filtered', code, detail: reason.slice((code ?? '').length + 1) || null,
|
|
214
|
+
attempts: 0, updated_at: e.seen_at ?? now, title: null, paths: null,
|
|
215
|
+
}
|
|
216
|
+
}
|
|
217
|
+
return { version: 2, jobs }
|
|
218
|
+
}
|
|
219
|
+
|
|
220
|
+
/**
|
|
221
|
+
* Drop records the scan no longer sees. A queued/filtered/error record whose
|
|
222
|
+
* file is gone was a scan artifact, not a job — keeping them is how 127 dead
|
|
223
|
+
* entries accumulated. Records that produced output are history and stay.
|
|
224
|
+
*
|
|
225
|
+
* `scanComplete` is the guard, not an optimisation: a partial listing (recorder
|
|
226
|
+
* yanked mid-glob) would otherwise wipe live queue entries. They'd come back on
|
|
227
|
+
* the next scan, but their retry counters wouldn't. It lives here rather than at
|
|
228
|
+
* the call site so the rule whose failure wipes a queue is covered by tests.
|
|
229
|
+
*/
|
|
230
|
+
export function pruneUnseen(jobs: Record<string, JobRecord>, seen: Set<string>, scanComplete: boolean): JobRecord[] {
|
|
231
|
+
if (!scanComplete) return []
|
|
232
|
+
const dropped: JobRecord[] = []
|
|
233
|
+
for (const [id, j] of Object.entries(jobs)) {
|
|
234
|
+
if (seen.has(id) || ownsOutput(j)) continue
|
|
235
|
+
delete jobs[id]
|
|
236
|
+
dropped.push(j) // the record, not the id: the caller must be able to name what it forgot
|
|
237
|
+
}
|
|
238
|
+
return dropped
|
|
239
|
+
}
|
|
240
|
+
|
|
241
|
+
export type CurrentJob = { pid: number; source_id: string; step: string; started_at: string }
|
|
242
|
+
|
|
243
|
+
export type JobView = {
|
|
244
|
+
status: 'running' | 'queued' | 'done' | 'notes_failed' | 'error' | 'gave_up' | 'filtered'
|
|
245
|
+
name: string
|
|
246
|
+
title: string | null
|
|
247
|
+
time: string | null
|
|
248
|
+
step: string | null
|
|
249
|
+
detail: string | null
|
|
250
|
+
notes: string | null
|
|
251
|
+
}
|
|
252
|
+
|
|
253
|
+
export const SUMMARY_FAILED_STATUS = 'summary_failed_transcript_saved'
|
|
254
|
+
|
|
255
|
+
/** Local wall clock, not UTC: recorder filenames are local time and the view sorts on this string. */
|
|
256
|
+
export function localIso(d: Date): string {
|
|
257
|
+
const p = (n: number) => String(n).padStart(2, '0')
|
|
258
|
+
return `${d.getFullYear()}-${p(d.getMonth() + 1)}-${p(d.getDate())}T${p(d.getHours())}:${p(d.getMinutes())}:${p(d.getSeconds())}`
|
|
259
|
+
}
|
|
260
|
+
|
|
261
|
+
/**
|
|
262
|
+
* Apply a patch, reporting whether anything actually changed.
|
|
263
|
+
*
|
|
264
|
+
* The no-op guard is load-bearing, not tidiness: `classify` re-derives the same
|
|
265
|
+
* verdict for every filtered recording on every 60s scan, so an unconditional
|
|
266
|
+
* `updated_at` bump would make the state file differ on each tick and defeat the
|
|
267
|
+
* content-gated write that keeps synced workspaces quiet.
|
|
268
|
+
*/
|
|
269
|
+
export function patchJob(entry: JobRecord, patch: Partial<JobRecord>, now: string): boolean {
|
|
270
|
+
if (Object.entries(patch).every(([k, v]) => (entry as any)[k] === v)) return false
|
|
271
|
+
Object.assign(entry, patch, { updated_at: now })
|
|
272
|
+
return true
|
|
273
|
+
}
|
|
274
|
+
|
|
275
|
+
export const emptyState = (): StateFile => ({ version: 2, jobs: {} })
|
|
276
|
+
|
|
277
|
+
/**
|
|
278
|
+
* One meaning of `limit` for both front doors (the CLI flag and the GUI's RPC):
|
|
279
|
+
* 0 = no limit, absent = `fallback`, anything else must be a non-negative
|
|
280
|
+
* integer. Coercing garbage to a default is how a truncated list gets mistaken
|
|
281
|
+
* for a complete one — the exact bug this module exists to remove.
|
|
282
|
+
*/
|
|
283
|
+
export function parseJobsLimit(raw: unknown, fallback: number): number {
|
|
284
|
+
if (raw === undefined || raw === null || raw === '') return fallback
|
|
285
|
+
const n = Number(raw)
|
|
286
|
+
if (!Number.isInteger(n) || n < 0) throw new Error(`Invalid limit '${raw}': expected a non-negative integer (0 = no limit).`)
|
|
287
|
+
return n === 0 ? Infinity : n
|
|
288
|
+
}
|
|
289
|
+
|
|
290
|
+
/**
|
|
291
|
+
* Parse a state file, throwing on anything that isn't one.
|
|
292
|
+
*
|
|
293
|
+
* Deliberately strict: a truncated or sync-mangled file that silently read as
|
|
294
|
+
* "nothing was ever processed" would re-transcribe the entire history, pay for
|
|
295
|
+
* ASR a second time, and then overwrite the evidence on the next save.
|
|
296
|
+
*/
|
|
297
|
+
export function parseStateFile(text: string, path: string): StateFile {
|
|
298
|
+
const parsed = parseStrictJson(text, path)
|
|
299
|
+
const version = (parsed as any)?.version
|
|
300
|
+
// A newer build's file must not be reinterpreted as v2: unknown states would
|
|
301
|
+
// be re-run by the classifier while the view calls them unrecognised.
|
|
302
|
+
if (Number.isFinite(version) && version > 2) {
|
|
303
|
+
throw new Error(`${path} was written by a newer voicenote (state version ${version}). Upgrade rather than risk re-processing everything.`)
|
|
304
|
+
}
|
|
305
|
+
const raw = (parsed as any)?.jobs
|
|
306
|
+
if (typeof raw !== 'object' || raw === null || Array.isArray(raw)) {
|
|
307
|
+
throw new Error(`${path} is not a job-state file (no \`jobs\` map). Refusing to continue rather than re-processing everything.`)
|
|
308
|
+
}
|
|
309
|
+
// Normalise at the boundary rather than trusting field by field downstream.
|
|
310
|
+
// `attempts` especially: a missing value makes `attempts >= MAX_ATTEMPTS`
|
|
311
|
+
// compare as NaN, which is false — the retry cap would silently never apply
|
|
312
|
+
// and a broken recording would burn ASR every scheduler tick.
|
|
313
|
+
const jobs: Record<string, JobRecord> = {}
|
|
314
|
+
for (const [id, j] of Object.entries<any>(raw)) {
|
|
315
|
+
if (!j || typeof j !== 'object') continue
|
|
316
|
+
jobs[id] = {
|
|
317
|
+
...j,
|
|
318
|
+
name: typeof j.name === 'string' ? j.name : id,
|
|
319
|
+
source_path: typeof j.source_path === 'string' ? j.source_path : '',
|
|
320
|
+
recorded_at: typeof j.recorded_at === 'string' ? j.recorded_at : '',
|
|
321
|
+
attempts: Number.isInteger(j.attempts) && j.attempts >= 0 ? j.attempts : 0,
|
|
322
|
+
paths: j.paths && typeof j.paths === 'object' ? j.paths : null,
|
|
323
|
+
}
|
|
324
|
+
}
|
|
325
|
+
return { version: 2, jobs }
|
|
326
|
+
}
|
|
327
|
+
|
|
328
|
+
/** JSON.parse with the message a user can act on. */
|
|
329
|
+
export function parseStrictJson(text: string, path: string): unknown {
|
|
330
|
+
let parsed: unknown
|
|
331
|
+
try { parsed = JSON.parse(text) } catch (e: any) {
|
|
332
|
+
throw new Error(`${path} is unreadable (${e?.message || e}). Move it aside to start over — but note that re-processing every recording costs ASR again.`)
|
|
333
|
+
}
|
|
334
|
+
if (!parsed || typeof parsed !== 'object') throw new Error(`${path} is not a JSON object. Refusing to continue rather than re-processing everything.`)
|
|
335
|
+
return parsed
|
|
336
|
+
}
|
|
337
|
+
|
|
338
|
+
const FILTER_LABELS: Record<string, string> = {
|
|
339
|
+
too_small: 'too small',
|
|
340
|
+
too_short: 'too short',
|
|
341
|
+
too_old: 'too old',
|
|
342
|
+
}
|
|
343
|
+
|
|
344
|
+
/** `2026-07-29T12:06:29` → `2026-07-29 12:06`. */
|
|
345
|
+
export function displayTime(recordedAt: string | null): string | null {
|
|
346
|
+
if (!recordedAt) return null
|
|
347
|
+
return recordedAt.slice(0, 16).replace('T', ' ')
|
|
348
|
+
}
|
|
349
|
+
|
|
350
|
+
function foldFiltered(records: JobRecord[]): JobView | null {
|
|
351
|
+
if (!records.length) return null
|
|
352
|
+
const counts = new Map<string, number>()
|
|
353
|
+
for (const r of records) {
|
|
354
|
+
const key = FILTER_LABELS[r.code ?? ''] ?? r.code ?? 'filtered'
|
|
355
|
+
counts.set(key, (counts.get(key) ?? 0) + 1)
|
|
356
|
+
}
|
|
357
|
+
const detail = [...counts].map(([label, n]) => `${label} ×${n}`).join(', ')
|
|
358
|
+
return {
|
|
359
|
+
status: 'filtered',
|
|
360
|
+
name: `${records.length} recording${records.length > 1 ? 's' : ''} filtered out`,
|
|
361
|
+
title: null, time: null, step: null, detail, notes: null,
|
|
362
|
+
}
|
|
363
|
+
}
|
|
364
|
+
|
|
365
|
+
export function buildJobsView(
|
|
366
|
+
state: StateFile,
|
|
367
|
+
current: CurrentJob | null,
|
|
368
|
+
opts: { limit: number; alive: (pid: number) => boolean; recorderPresent: boolean },
|
|
369
|
+
): { items: JobView[]; total: number; queued_total: number; recorder_present: boolean } {
|
|
370
|
+
// Two independent conditions must agree before a row is shown as running:
|
|
371
|
+
// the declaring process is alive, AND the record itself says `running`.
|
|
372
|
+
// current.json survives a kill -9, so the pid alone could be recycled by an
|
|
373
|
+
// unrelated long-lived process and wedge a finished job at "Processing" —
|
|
374
|
+
// the very failure this rewrite exists to remove.
|
|
375
|
+
const claimed = current && opts.alive(current.pid) ? current : null
|
|
376
|
+
const live = claimed && state.jobs?.[claimed.source_id]?.state === 'running' ? claimed : null
|
|
377
|
+
|
|
378
|
+
const running: JobView[] = []
|
|
379
|
+
const queued: JobView[] = []
|
|
380
|
+
const attention: JobView[] = []
|
|
381
|
+
const done: JobView[] = []
|
|
382
|
+
const filtered: JobRecord[] = []
|
|
383
|
+
|
|
384
|
+
for (const [id, j] of Object.entries(state.jobs ?? {})) {
|
|
385
|
+
const base = {
|
|
386
|
+
name: j.name,
|
|
387
|
+
title: j.title ?? null,
|
|
388
|
+
time: displayTime(j.recorded_at),
|
|
389
|
+
step: null,
|
|
390
|
+
detail: null as string | null,
|
|
391
|
+
notes: j.paths?.notes ?? null,
|
|
392
|
+
_t: j.recorded_at ?? '',
|
|
393
|
+
}
|
|
394
|
+
if (live && live.source_id === id) { running.push({ ...base, status: 'running', step: live.step }); continue }
|
|
395
|
+
switch (j.state) {
|
|
396
|
+
case 'done': done.push({ ...base, status: 'done' }); break
|
|
397
|
+
// A `running` record with no live process is a crashed run; the next run
|
|
398
|
+
// reconciles it. Until then it belongs with the work still to do.
|
|
399
|
+
case 'running':
|
|
400
|
+
case 'queued': queued.push({ ...base, status: 'queued' }); break
|
|
401
|
+
// A saved transcript with a failed summary reads better as its own row:
|
|
402
|
+
// the stub note is openable and the retry is cheap (no ASR).
|
|
403
|
+
case 'error': attention.push({ ...base, status: j.code === 'summary_failed' ? 'notes_failed' : 'error', detail: j.detail }); break
|
|
404
|
+
case 'gave_up': attention.push({ ...base, status: 'gave_up', detail: j.detail }); break
|
|
405
|
+
case 'filtered': filtered.push(j); break
|
|
406
|
+
// `state` comes off disk and could be hand-edited or written by a newer
|
|
407
|
+
// build. Showing an unknown value as "done" would hide unprocessed work,
|
|
408
|
+
// so surface it instead.
|
|
409
|
+
default: attention.push({ ...base, status: 'error', detail: `Unrecognised state '${j.state}'` })
|
|
410
|
+
}
|
|
411
|
+
}
|
|
412
|
+
|
|
413
|
+
// Sort and display share one key (recorded_at). They used to differ — list
|
|
414
|
+
// sorted by processing time, rows labelled with recording time — which is why
|
|
415
|
+
// the list looked shuffled.
|
|
416
|
+
const asc = (a: any, b: any) => String(a._t).localeCompare(String(b._t))
|
|
417
|
+
queued.sort(asc)
|
|
418
|
+
attention.sort((a, b) => -asc(a, b))
|
|
419
|
+
done.sort((a, b) => -asc(a, b))
|
|
420
|
+
|
|
421
|
+
const filteredRow = foldFiltered(filtered)
|
|
422
|
+
// Priority is the array order: live work, then the queue, then anything needing
|
|
423
|
+
// attention, then history. Everything is subject to `limit` — exempting the
|
|
424
|
+
// head would make one broken credential (every recording failing MAX_ATTEMPTS
|
|
425
|
+
// times into `attention`) an unbounded list, with `total` claiming it was whole.
|
|
426
|
+
const ordered = [...running, ...queued, ...attention, ...done]
|
|
427
|
+
const items: JobView[] = ordered.slice(0, Math.max(0, opts.limit - (filteredRow ? 1 : 0)))
|
|
428
|
+
if (filteredRow) items.push(filteredRow)
|
|
429
|
+
const total = ordered.length + (filteredRow ? 1 : 0)
|
|
430
|
+
|
|
431
|
+
for (const it of items) delete (it as any)._t
|
|
432
|
+
// `queued_total` is pre-truncation on purpose: "N recordings waiting" counted
|
|
433
|
+
// from the visible page would contradict the "… X more" line right above it.
|
|
434
|
+
// Read `recorder_present` live from the caller, never stored — a persisted
|
|
435
|
+
// flag would keep claiming the recorder is connected after the agent stops.
|
|
436
|
+
return { items, total, queued_total: running.length + queued.length, recorder_present: opts.recorderPresent }
|
|
437
|
+
}
|