dsh-pf-stream 0.2.4 → 0.3.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (80) hide show
  1. package/README.en.md +171 -34
  2. package/README.md +168 -31
  3. package/client.js +1193 -0
  4. package/icon.svg +18 -0
  5. package/lib/asr/catalog.d.ts +74 -0
  6. package/lib/asr/catalog.js +168 -0
  7. package/lib/asr/catalog.js.map +1 -0
  8. package/lib/asr/detect.d.ts +33 -0
  9. package/lib/asr/detect.js +117 -0
  10. package/lib/asr/detect.js.map +1 -0
  11. package/lib/asr/engine.d.ts +43 -0
  12. package/lib/asr/engine.js +143 -0
  13. package/lib/asr/engine.js.map +1 -0
  14. package/lib/asr/install.d.ts +56 -0
  15. package/lib/asr/install.js +317 -0
  16. package/lib/asr/install.js.map +1 -0
  17. package/lib/asr/verify.d.ts +22 -0
  18. package/lib/asr/verify.js +174 -0
  19. package/lib/asr/verify.js.map +1 -0
  20. package/lib/audio/acousticVad.d.ts +45 -0
  21. package/lib/audio/acousticVad.js +179 -0
  22. package/lib/audio/acousticVad.js.map +1 -0
  23. package/lib/audio/asrProviders.d.ts +51 -0
  24. package/lib/audio/asrProviders.js +251 -0
  25. package/lib/audio/asrProviders.js.map +1 -0
  26. package/lib/audio/asrRouter.d.ts +30 -0
  27. package/lib/audio/asrRouter.js +187 -0
  28. package/lib/audio/asrRouter.js.map +1 -0
  29. package/lib/audio/hallucinationGuard.d.ts +18 -0
  30. package/lib/audio/hallucinationGuard.js +97 -0
  31. package/lib/audio/hallucinationGuard.js.map +1 -0
  32. package/lib/audio/sidecar.d.ts +76 -0
  33. package/lib/audio/sidecar.js +362 -0
  34. package/lib/audio/sidecar.js.map +1 -0
  35. package/lib/audio/subtitleExtractor.d.ts +36 -0
  36. package/lib/audio/subtitleExtractor.js +218 -0
  37. package/lib/audio/subtitleExtractor.js.map +1 -0
  38. package/lib/audio/transcriptCache.d.ts +33 -0
  39. package/lib/audio/transcriptCache.js +126 -0
  40. package/lib/audio/transcriptCache.js.map +1 -0
  41. package/lib/audio/types.d.ts +136 -0
  42. package/lib/audio/types.js +10 -0
  43. package/lib/audio/types.js.map +1 -0
  44. package/lib/host/api.d.ts +23 -0
  45. package/lib/host/api.js +286 -0
  46. package/lib/host/api.js.map +1 -0
  47. package/lib/host/runtime.d.ts +167 -0
  48. package/lib/host/runtime.js +335 -0
  49. package/lib/host/runtime.js.map +1 -0
  50. package/lib/index.d.ts +229 -1
  51. package/lib/index.js +103 -2
  52. package/lib/index.js.map +1 -1
  53. package/lib/session/state.js +4 -0
  54. package/lib/session/state.js.map +1 -1
  55. package/lib/stage0/decoder.d.ts +1 -9
  56. package/lib/stage0/decoder.js +15 -45
  57. package/lib/stage0/decoder.js.map +1 -1
  58. package/lib/stage2/stateMachine.d.ts +4 -1
  59. package/lib/stage2/stateMachine.js +7 -1
  60. package/lib/stage2/stateMachine.js.map +1 -1
  61. package/lib/stage3/deterministicRouter.js +3 -3
  62. package/lib/stage3/deterministicRouter.js.map +1 -1
  63. package/lib/stage3/streamSampler.d.ts +9 -2
  64. package/lib/stage3/streamSampler.js +37 -16
  65. package/lib/stage3/streamSampler.js.map +1 -1
  66. package/lib/stage6/history.d.ts +2 -1
  67. package/lib/stage6/history.js +4 -1
  68. package/lib/stage6/history.js.map +1 -1
  69. package/lib/tool/handler.d.ts +9 -0
  70. package/lib/tool/handler.js +33 -3
  71. package/lib/tool/handler.js.map +1 -1
  72. package/lib/tool/schema.js +15 -2
  73. package/lib/tool/schema.js.map +1 -1
  74. package/lib/tool/streamCall.d.ts +11 -0
  75. package/lib/tool/streamCall.js +122 -9
  76. package/lib/tool/streamCall.js.map +1 -1
  77. package/lib/types.d.ts +9 -0
  78. package/locale/en.json +6 -0
  79. package/locale/zh.json +6 -0
  80. package/package.json +23 -2
package/README.en.md CHANGED
@@ -1,58 +1,195 @@
1
- # dsh-pf-stream
1
+ <p align="center">
2
+ <img src="https://raw.githubusercontent.com/Weilv-D/dsh-pf-stream/main/icon.svg" width="72" height="72" alt="Video understanding">
3
+ </p>
2
4
 
3
- [中文](README.md) | **English**
5
+ <h1 align="center">Video understanding</h1>
4
6
 
5
- A video plugin for DeepSeek Harness (DSH). The agent uses it to watch local videos. During motion, the plugin records the process. At settle, it captures detail frames. Exact timestamps support full-resolution inspection.
7
+ <p align="center">Frames, motion and speech from a video, aligned on a timeline for the agent.</p>
8
+
9
+ <p align="center">
10
+ <a href="https://www.npmjs.com/package/dsh-pf-stream"><img src="https://img.shields.io/npm/v/dsh-pf-stream?color=2f6feb" alt="npm version"></a>
11
+ <img src="https://img.shields.io/badge/license-Apache--2.0-2ea44f" alt="license">
12
+ <img src="https://img.shields.io/badge/DSH-plugin-6e56cf" alt="DSH plugin">
13
+ </p>
14
+
15
+ <p align="center"><a href="README.md">中文</a> · <b>English</b></p>
16
+
17
+ ---
18
+
19
+ ## Quick start
20
+
21
+ | Step | Do |
22
+ | :--- | :--- |
23
+ | 1 | Install ffmpeg ([commands below](#1-install-ffmpeg)); `ffmpeg -version` must print output |
24
+ | 2 | Left sidebar **Plugins** → **Video understanding** → switch it on |
25
+ | 3 | Click **Download local model** (~190 MB), then **Test transcription** |
26
+ | 4 | Give the agent a video path and ask |
27
+
28
+ Video works without transcription. Only videos that have no subtitles and need speech text require step 3.
29
+
30
+ ## What it does
31
+
32
+ - **See**: motion is recorded as it happens; the moment it settles, detail is captured. Any timestamp can be inspected at full resolution.
33
+ - **Hear**: subtitles are used when present; otherwise a transcription engine produces the speech and mounts it on the matching frames.
34
+ - **Optional setup**: video works without transcription; with it, videos that only have a voice track also work.
6
35
 
7
36
  ## Install
8
37
 
9
- Prerequisite: ffmpeg and ffprobe must be available.
38
+ ### 1. Install ffmpeg
39
+
40
+ The plugin decodes video with ffmpeg. Check that it runs:
41
+
42
+ ```bash
43
+ ffmpeg -version
44
+ ffprobe -version
45
+ ```
46
+
47
+ Both must print output. If not:
48
+
49
+ | OS | Command |
50
+ | :--- | :--- |
51
+ | Windows | `winget install Gyan.FFmpeg` |
52
+ | macOS | `brew install ffmpeg` |
53
+ | Ubuntu / Debian | `sudo apt install ffmpeg` |
10
54
 
11
- ### Option 1: delegate to the agent
55
+ > Open a new terminal and check again. To keep it out of PATH, set `ffmpegPath` and `ffprobePath` in the plugin config.
12
56
 
13
- Give this prompt to the agent:
57
+ ### 2. Install the plugin in DSH
58
+
59
+ 1. Open DSH and click **Plugins** in the left sidebar.
60
+ 2. Find **Video understanding** (package `dsh-pf-stream`).
61
+ 3. Open its detail page and switch it on (top right).
62
+ 4. On first enable, a setup card appears on the detail page.
63
+
64
+ If it is not listed, use the install entry on the Plugins page to install `dsh-pf-stream` from npm, then return here.
65
+
66
+ You can also hand the install to the agent:
14
67
 
15
68
  ```text
16
- Install the DSH plugin dsh-pf-stream. Follow these steps:
17
- 1. Install the latest dsh-pf-stream from npm with the plugin manager. The result must be "applied".
18
- 2. Check ffmpeg and ffprobe. If they are missing, install them:
19
- Windows: winget install Gyan.FFmpeg;
20
- macOS: brew install ffmpeg;
21
- Linux: sudo apt install ffmpeg.
22
- 3. Call probe on a local video. A returned duration and resolution means success.
23
- 4. Report the plugin version, the ffmpeg location, and the check result.
69
+ Install the DSH plugin dsh-pf-stream:
70
+ 1. Install the latest version from npm with the plugin manager; success means result applied;
71
+ 2. Check ffmpeg and ffprobe; install them if missing (Windows: winget install Gyan.FFmpeg / macOS: brew install ffmpeg / Linux: sudo apt install ffmpeg);
72
+ 3. Call probe on a local video; duration and resolution in the result means success;
73
+ 4. Report the plugin version, ffmpeg location, and check results.
24
74
  ```
25
75
 
26
- ### Option 2: manual install
76
+ ## First run: configure transcription
77
+
78
+ The transcription engine decides whether videos without subtitles get speech text. Without it, video still works; only the transcript is missing.
79
+
80
+ To open the settings: **Plugins in the left sidebar → Video understanding → the panel below the description**. Use the same path later to change settings.
81
+
82
+ <p><img src="https://raw.githubusercontent.com/Weilv-D/dsh-pf-stream/main/docs/assets/panel.png" width="720" alt="Transcription settings panel on the plugin detail page: decoder, local transcription and cloud transcription cards"></p>
27
83
 
28
- 1. Install ffmpeg and ffprobe:
84
+ Three cards at the top report video decode (ffmpeg), local transcription, and cloud transcription. Pick one path:
29
85
 
30
- ```bash
31
- winget install Gyan.FFmpeg # Windows
32
- brew install ffmpeg # macOS
33
- sudo apt install ffmpeg # Debian / Ubuntu
34
- ```
86
+ | Path | Best for | What to do |
87
+ | :--- | :--- | :--- |
88
+ | Local (recommended) | Free, offline, no API key | Click "Download local model" (~190 MB), then "Test transcription" |
89
+ | Cloud | You have a key, or the machine is slow | Pick OpenAI / Groq / a custom compatible endpoint, paste the key, save, then test |
90
+ | Skip for now | Videos have subtitles | Use it as is; speech-only videos get no transcript |
35
91
 
36
- 2. If the executables are outside PATH, set `ffmpegPath` and `ffprobePath` in the plugin config.
37
- 3. Add the package to the host: `pnpm add dsh-pf-stream`.
38
- 4. Mount the plugin as declared in `cordis.patch.yml` inside the package.
39
- 5. Verify: start a new DSH session. The tool `dsh-pf-stream` appears in the tool list.
92
+ ### Local model
40
93
 
41
- ## Usage
94
+ 1. Click "Download local model". The progress bar shows bytes received; cancel at any time.
95
+ 2. Clicking again after an interruption resumes the download.
96
+ 3. When done, the card reads "Ready · whisper.cpp".
97
+ 4. Click "Test transcription". "Local engine available" means it works.
42
98
 
43
- Ask the agent directly:
99
+ One-click install currently supports Windows x64. On other systems, install whisper.cpp and set `audio.asr.local.binPath` in the config file, or use cloud transcription.
100
+
101
+ ### Cloud key
102
+
103
+ 1. Choose a provider under "Cloud transcription key".
104
+ 2. Paste the API key and click "Save". After saving, only "Configured" is shown; the key is never echoed back.
105
+ 3. Click "Test transcription" to verify the key and network.
106
+
107
+ > **Auto policy**: cloud is used when key, endpoint and model are all set (faster, billed per use); otherwise the local engine. The badge next to the panel title shows the engine in use, such as "Transcription ready · Cloud". Under "Advanced" you can switch to local only, cloud only, or off.
108
+
109
+ ## Use it
110
+
111
+ Give the agent a video path and ask:
44
112
 
45
113
  - "What happens in this video?"
46
- - "What text is shown at 12.9 seconds?"
47
- - "Sample one frame every 5 seconds and describe the change."
114
+ - "What does the text at 12.9 seconds say?"
115
+ - "Take one frame every 5 seconds and describe how it changes."
116
+ - "Transcribe this interview with timestamps."
117
+
118
+ Example output (excerpt):
48
119
 
49
- For parameters and return content, see the [usage guide](docs/usage-guide.md).
120
+ ```text
121
+ [Packet #42 | Anchor: #105 | #101(P, 12.1s, Left-Top) → #105(F, 13.4s, Center)]
122
+ Audio @Packet #42: [12.1s-12.9s] "we prepared this for three years" | [13.0s-13.4s] "today we can finally say it"
123
+ ```
124
+
125
+ The text block is structured metadata: each canvas packet reports time, form, anchor, and unit positions; the audio part reports status, source, and per-line timings. The image block is one canvas per packet.
126
+
127
+ Four ways to look, by question:
128
+
129
+ | Goal | Ask |
130
+ | :--- | :--- |
131
+ | Overview first | "Look at this video" — streaming sampling; motion as process, settle as detail |
132
+ | One exact moment | "What is written at 12.9s" — deterministic high-resolution frame |
133
+ | Even coverage | "One frame every 5 seconds" — uniform sampling |
134
+ | A region | "Zoom into the bottom right" — crop by area, then look |
135
+
136
+ ## Design
137
+
138
+ ### Dual-scale streaming perception
139
+
140
+ Sampling is event-driven rather than fixed-rate. During motion, low-resolution process units (P) record state transitions; at settle, high-resolution freeze units (F) preserve detail. Both are delivered within a single canvas packet, so one observation carries the temporal process and the steady-state result together, and the model need not re-watch a clip to recover a detail.
141
+
142
+ Density adapts to picture activity: rapid change lowers the sampling rate to suppress redundant frames, while slow change extends the wait until a settle is detected; long static intervals maintain a temporal reference through heartbeats without retransmitting images.
143
+
144
+ ### Canvas packing and metering
145
+
146
+ Each frame is partitioned on a 42 px superpixel grid; P or F units are selected as needed and assembled into a canvas. A call may carry multiple packets, and each packet contains only the units new relative to the previous one; historical packets collapse into content-hash references, keeping context bounded in long sessions. Metering follows canvas grid cells (`r × (c + 1) + 2`).
147
+
148
+ ### Deterministic inspection
149
+
150
+ Beyond streaming, four deterministic paths are provided: `probe` parses container metadata only; `timestamps` extracts exact frames at millisecond timestamps; `uniform` samples at a fixed interval; `roi_hint` crops and magnifies a normalized region. None depends on motion detection; outputs are reproducible and suited to verifying text, numerals, and small objects.
151
+
152
+ ### Cascaded speech sources
153
+
154
+ Speech text is obtained through a three-stage cascade: a sidecar `.srt` / `.vtt` is read first, then embedded soft subtitles; without subtitles, silence detection and a speech-ratio estimate run, and an all-silent video skips transcription; finally ASR runs, either local whisper.cpp or OpenAI / Groq / a custom compatible endpoint. Segments are aligned by PTS and mounted on the corresponding packets, stitched across block boundaries with a ±300 ms tolerance.
155
+
156
+ The default policy is vision-first with background transcription: once a call exceeds the soft deadline (`graceMs`, 8 s by default), the visual result returns and transcription continues in the background; later calls on the same video hit the disk cache.
157
+
158
+ ### Cold regions in long videos
159
+
160
+ Historical packets outside the current window collapse into references (anchor hash, time, and transcript text) rather than retransmitting images; a timestamp call restores the detail when needed.
161
+
162
+ ## Troubleshooting
163
+
164
+ | Symptom | Read |
165
+ | :--- | :--- |
166
+ | ffmpeg not found | [ffmpeg](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#找不到-ffmpeg) |
167
+ | Speech but no transcript | [No transcript](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#有声音但没有台词) |
168
+ | Model download fails or is slow | [Download](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#下载失败) |
169
+ | Transcription is slow or times out | [Slow transcription](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#转写很慢或超时) |
170
+ | Settings page is read-only | [Read-only config](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#设置页只读) |
171
+
172
+ ## Docs
173
+
174
+ | Document | Contents |
175
+ | :--- | :--- |
176
+ | [Usage Guide](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/usage-guide.md) | Usage, parameters, output, metering, every config key |
177
+ | [使用指南](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/使用指南.md) | Chinese usage guide |
178
+ | [排障](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md) | Troubleshooting by symptom (Chinese) |
179
+ | [开发者指南](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/开发者指南.md) | Architecture, host contracts, tests, release (Chinese) |
180
+
181
+ ## Development
182
+
183
+ ```bash
184
+ pnpm build
185
+ pnpm test
186
+ pnpm verify:host
187
+ pnpm check:package
188
+ ```
50
189
 
51
- ## Documentation
190
+ The host half provides the `dsh-pf-stream` tool (`probe` / `pf_stream` / `timestamps` / `uniform`), the audio sidecar, the local engine installer, and the settings API. The client half is `client.js`, which registers the config panel and the setup card on the plugin detail page. The only runtime dependency is `pngjs`.
52
191
 
53
- - [使用指南](docs/使用指南.md) (Chinese)
54
- - [Usage Guide](docs/usage-guide.md) (English)
55
- - [发布规范](docs/发布规范.md) (release process)
192
+ Configuration is written through `ctx.configEditor.edit()`, which keeps fields the user did not touch; keys live in `ctx.credentials`, and the config file only holds a reference.
56
193
 
57
194
  ## License
58
195
 
package/README.md CHANGED
@@ -1,58 +1,195 @@
1
- # dsh-pf-stream
1
+ <p align="center">
2
+ <img src="https://raw.githubusercontent.com/Weilv-D/dsh-pf-stream/main/icon.svg" width="72" height="72" alt="视频理解">
3
+ </p>
2
4
 
3
- **中文** | [English](README.en.md)
5
+ <h1 align="center">视频理解</h1>
4
6
 
5
- DSH 视频感知插件。智能体用它观看本地视频。画面运动时,插件记录过程。画面停稳时,插件定格细节。指定时间点可以高清检视。
7
+ <p align="center">让智能体看懂视频:画面、动作与台词,按时间对齐。</p>
8
+
9
+ <p align="center">
10
+ <a href="https://www.npmjs.com/package/dsh-pf-stream"><img src="https://img.shields.io/npm/v/dsh-pf-stream?color=2f6feb" alt="npm 版本"></a>
11
+ <img src="https://img.shields.io/badge/license-Apache--2.0-2ea44f" alt="许可证">
12
+ <img src="https://img.shields.io/badge/DSH-plugin-6e56cf" alt="DSH 插件">
13
+ </p>
14
+
15
+ <p align="center"><b>中文</b> · <a href="README.en.md">English</a></p>
16
+
17
+ ---
18
+
19
+ ## 30 秒上手
20
+
21
+ | 步骤 | 做什么 |
22
+ | :--- | :--- |
23
+ | 1 | 安装 ffmpeg([命令见下](#1-安装-ffmpeg)),确认 `ffmpeg -version` 有输出 |
24
+ | 2 | 左侧栏「插件」→ 点「视频理解」→ 打开右上角开关 |
25
+ | 3 | 详情页点「下载本地模型」,约 190 MB;完成后点「测试转写」 |
26
+ | 4 | 在对话里给视频路径,直接提问 |
27
+
28
+ 不配置转写也能看画面。只有「没有字幕、又需要台词」的视频,才需要第 3 步。
29
+
30
+ ## 能做什么
31
+
32
+ - **看画面**:画面运动时记录过程,停稳时定格细节;指定时间点可以高清复核。
33
+ - **听声音**:有字幕就用字幕;没有字幕时,配置转写引擎后生成台词,并按时间挂到对应画面。
34
+ - **按需配置**:默认不转写也能处理画面;配置后可以处理带音轨的视频。
6
35
 
7
36
  ## 安装
8
37
 
9
- 前置条件:ffmpeg 与 ffprobe 必须可用。
38
+ ### 1. 安装 ffmpeg
39
+
40
+ 插件用 ffmpeg 解码视频。先确认它能运行:
41
+
42
+ ```bash
43
+ ffmpeg -version
44
+ ffprobe -version
45
+ ```
46
+
47
+ 两条都要有输出。没有的话:
48
+
49
+ | 系统 | 安装命令 |
50
+ | :--- | :--- |
51
+ | Windows | `winget install Gyan.FFmpeg` |
52
+ | macOS | `brew install ffmpeg` |
53
+ | Ubuntu / Debian | `sudo apt install ffmpeg` |
10
54
 
11
- ### 方式一:交给智能体
55
+ > 装完新开一个终端再验证。不想放进 PATH,可以在插件配置里写 `ffmpegPath` 与 `ffprobePath` 的绝对路径。
12
56
 
13
- 把下面的提示词交给智能体:
57
+ ### 2. 在 DSH 里安装插件
58
+
59
+ 1. 打开 DSH,点左侧栏的「插件」。
60
+ 2. 在列表里找「视频理解」(包名 `dsh-pf-stream`)。
61
+ 3. 点进详情页,把右上角的开关打开。
62
+ 4. 第一次开启后,详情页会出现引导卡片。
63
+
64
+ 列表里没有时,用插件页的安装入口从 npm 安装 `dsh-pf-stream`,装完再回到这一步。
65
+
66
+ 也可以把安装交给智能体:
14
67
 
15
68
  ```text
16
- 安装 DSH 插件 dsh-pf-stream,按以下步骤执行:
17
- 1. 用插件管理器从 npm 安装 dsh-pf-stream 最新版本。安装结果为 applied 才算成功。
18
- 2. 检查 ffmpeg 与 ffprobe 是否可用。缺失时安装:
19
- Windows 用 winget install Gyan.FFmpeg;
20
- macOS 用 brew install ffmpeg;
21
- Linux 用 sudo apt install ffmpeg。
22
- 3. 对一个本地视频调用 probe。返回时长与分辨率即成功。
23
- 4. 汇报插件版本、ffmpeg 位置、验证结果。
69
+ 安装 DSH 插件 dsh-pf-stream:
70
+ 1. 用插件管理器从 npm 安装最新版本,结果为 applied 才算成功;
71
+ 2. 检查 ffmpeg 与 ffprobe;缺失时安装(Windows: winget install Gyan.FFmpeg / macOS: brew install ffmpeg / Linux: sudo apt install ffmpeg);
72
+ 3. 对一个本地视频调用 probe;返回时长与分辨率即成功;
73
+ 4. 汇报插件版本、ffmpeg 位置与检查结果。
24
74
  ```
25
75
 
26
- ### 方式二:手动安装
76
+ ## 第一次配置转写
77
+
78
+ 转写引擎决定「没有字幕的视频能不能拿到台词」。不配置也能看画面,只是不会生成转写文本。
79
+
80
+ 进入配置页:**左侧栏「插件」→ 点「视频理解」→ 详情页描述下方的面板**。以后要改设置,走同一条路。
81
+
82
+ <p><img src="https://raw.githubusercontent.com/Weilv-D/dsh-pf-stream/main/docs/assets/panel.png" width="720" alt="插件详情页的转写设置面板:视频解码、本地转写与云端转写三张状态卡"></p>
27
83
 
28
- 1. 安装 ffmpeg 与 ffprobe:
84
+ 面板顶部有三张状态卡:视频解码(ffmpeg)、本地转写、云端转写。按需要选一条路线:
29
85
 
30
- ```bash
31
- winget install Gyan.FFmpeg # Windows
32
- brew install ffmpeg # macOS
33
- sudo apt install ffmpeg # Debian / Ubuntu
34
- ```
86
+ | 路线 | 适合 | 操作 |
87
+ | :--- | :--- | :--- |
88
+ | 本地转写(推荐) | 免费、离线、不填密钥 | 点「下载本地模型」,约 190 MB;下载完点「测试转写」 |
89
+ | 云端转写 | 已有 API Key,或机器性能较弱 | 选 OpenAI / Groq / 自定义兼容端点,填 Key,保存后点「测试转写」 |
90
+ | 暂不配置 | 视频都有字幕 | 直接使用;无字幕的语音视频不会生成转写文本 |
35
91
 
36
- 2. 可执行文件不在 PATH 时,在插件配置里写 `ffmpegPath` 与 `ffprobePath`。
37
- 3. 在宿主环境安装插件包:`pnpm add dsh-pf-stream`。
38
- 4. 按包内 `cordis.patch.yml` 写宿主装载配置。
39
- 5. 验证:新开 DSH 会话。工具列表出现 `dsh-pf-stream` 即成功。
92
+ ### 本地模型
40
93
 
41
- ## 使用
94
+ 1. 点「下载本地模型」。进度条显示已下载大小,可随时取消。
95
+ 2. 下载中断后再次点击会续传,不用从头开始。
96
+ 3. 完成后状态卡显示「已就绪 · whisper.cpp」。
97
+ 4. 点「测试转写」。返回「本地引擎可用」即配置成功。
42
98
 
43
- 对智能体直接提问:
99
+ 一键安装当前支持 Windows x64。其他系统请先安装 whisper.cpp,在配置文件 `audio.asr.local.binPath` 填写可执行文件路径,或改用云端转写。
100
+
101
+ ### 云端密钥
102
+
103
+ 1. 在「云端转写密钥」里选服务商。
104
+ 2. 粘贴 API Key,点「保存」。保存后只显示「已配置」,不再回显密钥。
105
+ 3. 点「测试转写」验证密钥与网络。
106
+
107
+ > **自动策略**:云端密钥、端点、模型都齐时用云端(更快,按量计费);否则用本地引擎。面板标题旁会显示当前实际使用的引擎,如「转写可用 · 云端」。可以在「高级」里改成仅本地、仅云端或不转写。
108
+
109
+ ## 开始使用
110
+
111
+ 在对话里直接给视频路径并提问:
44
112
 
45
113
  - 「这个视频里发生了什么?」
46
114
  - 「第 12.9 秒的文字写的是什么?」
47
115
  - 「每 5 秒抽一帧,看画面怎么变化。」
116
+ - 「把这段访谈里说的话按时间整理出来。」
117
+
118
+ 返回示例(节选):
48
119
 
49
- 参数与返回内容见[使用指南](docs/使用指南.md)。
120
+ ```text
121
+ [Packet #42 | Anchor: #105 | #101(P, 12.1s, Left-Top) → #105(F, 13.4s, Center)]
122
+ Audio @Packet #42: [12.1s-12.9s] "这件事我们准备了三年" | [13.0s-13.4s] "今天终于可以说了"
123
+ ```
124
+
125
+ 文本块是结构化元数据:每个画布包给出时间、形态、锚点与单元位置;音频部分给出状态、来源与逐句时间。图像块是每个画布包一张画布。
126
+
127
+ 常用问法对应四种取帧方式:
128
+
129
+ | 想要 | 说法 |
130
+ | :--- | :--- |
131
+ | 先看整体 | 「看看这个视频」——流式采样,运动记过程、停稳给细节 |
132
+ | 只看某一秒 | 「第 12.9 秒写了什么」——定点高清帧 |
133
+ | 均匀看一遍 | 「每 5 秒抽一帧」——等间隔抽帧 |
134
+ | 看局部 | 「放大右下角」——按区域裁剪后再看 |
135
+
136
+ ## 特点
137
+
138
+ ### 双尺度流式感知
139
+
140
+ 采样过程不再采用固定帧率,而由画面变化事件驱动:运动阶段以低分辨率过程单元(P)记录状态转移,停稳后以高分辨率定格单元(F)保留细节;两类单元在同一画布包内交付,使单次观测同时包含时序过程与稳态结果,模型无需为恢复某一细节而回看整段视频。
141
+
142
+ 采样密度随画面活动度自适应调整:快速变化时降低采样率以抑制冗余帧,缓慢变化时延长等待直至检测到停稳;长静止区间以心跳信号维持时间基准,不重复传输图像。
143
+
144
+ ### 画布打包与计费
145
+
146
+ 每帧按 42 px 超像素网格切分为单元,按需选取 P 或 F 单元并拼装为画布。单次调用可携带多个画布包;每个包仅包含相对上一包新增的单元,历史包以内容哈希引用折叠,使长会话中的上下文规模保持有界。计费按画布网格单元数计算(`r × (c + 1) + 2`)。
147
+
148
+ ### 确定性检视
149
+
150
+ 流式采样之外另设四条确定性路径:`probe` 仅解析容器元数据;`timestamps` 按毫秒级时间戳抽取精确帧;`uniform` 以固定间隔均匀采样;`roi_hint` 对指定归一化区域裁剪放大。上述路径不依赖运动检测,输出可复现,适用于文字、数字与小目标的核对。
151
+
152
+ ### 台词的级联来源
153
+
154
+ 转写文本按三级级联获取:优先读取同名外挂 `.srt` / `.vtt`,其次解析内嵌软字幕;无字幕时执行静音检测与语音占比估计,全静音视频直接跳过转写;最后调用语音识别,可选本地 whisper.cpp 或 OpenAI / Groq / 自定义兼容端点。识别结果按 PTS 对齐并挂载至相应画布包,跨块边界以 ±300 ms 容差缝合。
155
+
156
+ 默认策略为「视觉先行、转写后台补齐」:首次调用超过软截止(`graceMs`,默认 8 s)时先行返回视觉结果,转写任务在后台继续;同一视频的后续调用直接命中磁盘缓存。
157
+
158
+ ### 长视频的冷区
159
+
160
+ 超出当前窗口的历史画布包折叠为引用(锚点哈希、时间与台词文本),不再重复传输图像;需要复核时按时间戳重新取帧。
161
+
162
+ ## 遇到问题
163
+
164
+ | 现象 | 先看这里 |
165
+ | :--- | :--- |
166
+ | 提示找不到 ffmpeg | [排障:ffmpeg](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#找不到-ffmpeg) |
167
+ | 有声音,但没有台词 | [排障:没有台词](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#有声音但没有台词) |
168
+ | 下载模型失败或很慢 | [排障:下载失败](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#下载失败) |
169
+ | 转写很慢、超时 | [排障:转写慢](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#转写很慢或超时) |
170
+ | 设置页显示「只读」 | [排障:配置只读](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#设置页只读) |
50
171
 
51
172
  ## 文档
52
173
 
53
- - [使用指南](docs/使用指南.md)
54
- - [Usage Guide](docs/usage-guide.md)
55
- - [发布规范](docs/发布规范.md)
174
+ | 文档 | 内容 |
175
+ | :--- | :--- |
176
+ | [使用指南](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/使用指南.md) | 用法、参数、返回内容、计费、全部配置项 |
177
+ | [排障](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md) | 按现象查原因 |
178
+ | [开发者指南](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/开发者指南.md) | 架构、宿主契约、测试与发布 |
179
+ | [Usage Guide](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/usage-guide.md) | 英文使用指南 |
180
+
181
+ ## 本地开发
182
+
183
+ ```bash
184
+ pnpm build
185
+ pnpm test
186
+ pnpm verify:host
187
+ pnpm check:package
188
+ ```
189
+
190
+ Host 半提供工具 `dsh-pf-stream`(`probe` / `pf_stream` / `timestamps` / `uniform`)、音频边车、本地引擎安装器与设置接口;Client 半是 `client.js`,在插件详情页注册配置面板与引导卡片。运行时依赖只有 `pngjs`。
191
+
192
+ 配置通过 `ctx.configEditor.edit()` 写入,保留用户未触碰的字段;密钥通过 `ctx.credentials` 存取,配置文件只保留引用。
56
193
 
57
194
  ## 许可证
58
195