dsh-pf-stream 0.2.4 → 0.3.0

package/README.en.md

@@ -1,58 +1,170 @@
 # dsh-pf-stream
 
-[中文](README.md) | **English**
+[中文](https://github.com/Weilv-D/dsh-pf-stream/blob/main/README.md) | **English**
 
-A video plugin for DeepSeek Harness (DSH). The agent uses it to watch local videos. During motion, the plugin records the process. At settle, it captures detail frames. Exact timestamps support full-resolution inspection.
+Let the agent watch videos: adaptive sampling, plus the spoken lines.
+
+## What it does
+
+- **See**: motion is recorded as it happens; the moment it settles, detail is captured. Any timestamp can be inspected at full resolution.
+- **Hear**: subtitles are used when present; otherwise a transcription engine produces the speech and mounts it on the matching frames.
+- **Optional setup**: video works without transcription; with it, videos that only have a voice track also work.
 
 ## Install
 
-Prerequisite: ffmpeg and ffprobe must be available.
+### 1. Install ffmpeg
+
+The plugin decodes video with ffmpeg. Check that it runs:
+
+```bash
+ffmpeg -version
+ffprobe -version
+```
+
+Both must print output. If not:
 
-### Option 1: delegate to the agent
+| OS | Command |
+| --- | --- |
+| Windows | `winget install Gyan.FFmpeg` |
+| macOS | `brew install ffmpeg` |
+| Ubuntu / Debian | `sudo apt install ffmpeg` |
 
-Give this prompt to the agent:
+Open a new terminal and check again. To keep it out of PATH, set `ffmpegPath` and `ffprobePath` in the plugin config.
+
+### 2. Install the plugin in DSH
+
+1. Open DSH and click **Plugins** in the left sidebar.
+2. Find **Video understanding** (package `dsh-pf-stream`).
+3. Open its detail page and switch it on (top right).
+4. On first enable, a setup card appears on the detail page.
+
+If it is not listed, use the install entry on the Plugins page to install `dsh-pf-stream` from npm, then return here.
+
+You can also hand the install to the agent:
 
 ```text
-Install the DSH plugin dsh-pf-stream. Follow these steps:
-1. Install the latest dsh-pf-stream from npm with the plugin manager. The result must be "applied".
-2. Check ffmpeg and ffprobe. If they are missing, install them:
-   Windows: winget install Gyan.FFmpeg;
-   macOS: brew install ffmpeg;
-   Linux: sudo apt install ffmpeg.
-3. Call probe on a local video. A returned duration and resolution means success.
-4. Report the plugin version, the ffmpeg location, and the check result.
+Install the DSH plugin dsh-pf-stream:
+1. Install the latest version from npm with the plugin manager; success means result applied;
+2. Check ffmpeg and ffprobe; install them if missing (Windows: winget install Gyan.FFmpeg / macOS: brew install ffmpeg / Linux: sudo apt install ffmpeg);
+3. Call probe on a local video; duration and resolution in the result means success;
+4. Report the plugin version, ffmpeg location, and check results.
 ```
 
-### Option 2: manual install
+## First run: configure transcription
+
+The transcription engine decides whether videos without subtitles get speech text. Without it, video still works; only the transcript is missing.
 
-1. Install ffmpeg and ffprobe:
+To open the settings: **Plugins in the left sidebar → Video understanding → the panel below the description**. Use the same path later to change settings.
 
-   ```bash
-   winget install Gyan.FFmpeg   # Windows
-   brew install ffmpeg          # macOS
-   sudo apt install ffmpeg      # Debian / Ubuntu
-   ```
+Three cards at the top report video decode (ffmpeg), local transcription, and cloud transcription. Pick one path:
 
-2. If the executables are outside PATH, set `ffmpegPath` and `ffprobePath` in the plugin config.
-3. Add the package to the host: `pnpm add dsh-pf-stream`.
-4. Mount the plugin as declared in `cordis.patch.yml` inside the package.
-5. Verify: start a new DSH session. The tool `dsh-pf-stream` appears in the tool list.
+| Path | Best for | What to do |
+| --- | --- | --- |
+| Local (recommended) | Free, offline, no API key | Click "Download local model" (~190 MB), then "Test transcription" |
+| Cloud | You have a key, or the machine is slow | Pick OpenAI / Groq / a custom compatible endpoint, paste the key, save, then test |
+| Skip for now | Videos have subtitles | Use it as is; speech-only videos get no transcript |
 
-## Usage
+Local model steps:
 
-Ask the agent directly:
+1. Click "Download local model". The progress bar shows bytes received; cancel at any time.
+2. Clicking again after an interruption resumes the download.
+3. When done, the card reads "Ready · whisper.cpp".
+4. Click "Test transcription". "Local engine available" means it works.
+
+Cloud key steps:
+
+1. Choose a provider under "Cloud transcription key".
+2. Paste the API key and click "Save". After saving, only "Configured" is shown; the key is never echoed back.
+3. Click "Test transcription" to verify the key and network.
+
+The default policy is `auto`: use a cloud key when configured, otherwise the local engine. Under "Advanced" you can switch to local only, cloud only, or off.
+
+One-click install currently supports Windows x64. On other systems, install whisper.cpp and set `audio.asr.local.binPath` in the config file, or use cloud transcription.
+
+## Use it
+
+Give the agent a video path and ask:
 
 - "What happens in this video?"
-- "What text is shown at 12.9 seconds?"
-- "Sample one frame every 5 seconds and describe the change."
+- "What does the text at 12.9 seconds say?"
+- "Take one frame every 5 seconds and describe how it changes."
+- "Transcribe this interview with timestamps."
+
+Example output (excerpt):
+
+```text
+[Packet #42 | Anchor: #105 | #101(P, 12.1s, Left-Top) → #105(F, 13.4s, Center)]
+Audio @Packet #42: [12.1s-12.9s] "we prepared this for three years" | [13.0s-13.4s] "today we can finally say it"
+```
+
+The text block is structured metadata: each canvas packet reports time, form, anchor, and unit positions; the audio part reports status, source, and per-line timings. The image block is one canvas per packet.
 
-For parameters and return content, see the [usage guide](docs/usage-guide.md).
+Four ways to look, by question:
+
+| Goal | Ask |
+| --- | --- |
+| Overview first | "Look at this video" — streaming sampling; motion as process, settle as detail |
+| One exact moment | "What is written at 12.9s" — deterministic high-resolution frame |
+| Even coverage | "One frame every 5 seconds" — uniform sampling |
+| A region | "Zoom into the bottom right" — crop by area, then look |
+
+## Design
+
+### Dual-scale streaming perception
+
+Video is not sampled at a fixed rate. While the picture moves, low-resolution process units (P) record change; when it settles, high-resolution freeze units (F) keep detail. A single canvas packet carries both process and result, so the model does not need to re-watch a clip to recover one detail.
+
+The sampling cadence follows the picture: fast change lowers density to avoid duplicates; slow change waits for a settle before freezing. Long static stretches keep a heartbeat instead of repeating images.
+
+### Canvas packing and metering
+
+A frame is divided into 42 px superpixel-aligned units. P or F units are taken as needed and packed into one canvas. One call can carry several packets; each packet holds only the units that are new, and older packets collapse into reference hashes instead of resending images. Metering follows the canvas grid (`r × (c + 1) + 2`).
+
+### Deterministic inspection
+
+Besides streaming, three paths are deterministic: `probe` reads metadata only; `timestamps` returns exact frames in milliseconds; `uniform` samples at a fixed interval; `roi_hint` crops a region. These do not depend on motion detection, so results are reproducible and suited to reading text, numbers, and small objects.
+
+### Three sources for speech
+
+1. **Subtitles**: a sidecar `.srt` / `.vtt` with the same name first, then embedded soft subtitles.
+2. **Acoustic check**: without subtitles, silence detection and a speech-ratio estimate run first; an all-silent video skips transcription.
+3. **Transcription**: local whisper.cpp, or OpenAI / Groq / a custom compatible endpoint. Segments are mounted on packets by PTS and stitched across block boundaries with a ±300 ms tolerance.
+
+The default is "picture first, speech later": when the soft deadline (`graceMs`, 8 s by default) passes, the visual result returns and transcription continues in the background; the next call on the same video hits the disk cache.
+
+### Old regions
+
+Canvas packets outside the window collapse into references (anchor hash, time, and transcript text) instead of resending images. A timestamp call brings the detail back when needed.
+
+## Troubleshooting
+
+| Symptom | Read |
+| --- | --- |
+| ffmpeg not found | [ffmpeg](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#找不到-ffmpeg) |
+| Speech but no transcript | [No transcript](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#有声音但没有台词) |
+| Model download fails or is slow | [Download](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#下载失败) |
+| Transcription is slow or times out | [Slow transcription](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#转写很慢或超时) |
+| Settings page is read-only | [Read-only config](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#设置页只读) |
+
+## Docs
+
+- [Usage Guide](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/usage-guide.md) (English)
+- [使用指南](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/使用指南.md) (Chinese)
+- [排障](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md) (Chinese)
+- [开发者指南](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/开发者指南.md) (Chinese): architecture, host contracts, tests, release
+
+## Development
+
+```bash
+pnpm build
+pnpm test
+pnpm verify:host
+pnpm check:package
+```
 
-## Documentation
+The host half provides the `dsh-pf-stream` tool (`probe` / `pf_stream` / `timestamps` / `uniform`), the audio sidecar, the local engine installer, and the settings API. The client half is `client.js`, which registers the config panel and the setup card on the plugin detail page. The only runtime dependency is `pngjs`.
 
-- [使用指南](docs/使用指南.md) (Chinese)
-- [Usage Guide](docs/usage-guide.md) (English)
-- [发布规范](docs/发布规范.md) (release process)
+Configuration is written through `ctx.configEditor.edit()`, which keeps fields the user did not touch; keys live in `ctx.credentials`, and the config file only holds a reference.
 
 ## License
 

package/README.md

@@ -1,58 +1,170 @@
 # dsh-pf-stream
 
-**中文** | [English](README.en.md)
+**中文** | [English](https://github.com/Weilv-D/dsh-pf-stream/blob/main/README.en.md)
 
-DSH 视频感知插件。智能体用它观看本地视频。画面运动时，插件记录过程。画面停稳时，插件定格细节。指定时间点可以高清检视。
+让智能体看懂视频：动态抽帧，加上台词。
+
+## 能做什么
+
+- **看画面**：画面运动时记录过程，停稳时定格细节；指定时间点可以高清复核。
+- **听声音**：有字幕就用字幕；没有字幕时，配置转写引擎后生成台词，并按时间挂到对应画面。
+- **按需配置**：不配置转写也能处理画面；配置后可以处理带音轨的视频。
 
 ## 安装
 
-前置条件：ffmpeg 与 ffprobe 必须可用。
+### 第一步：安装 ffmpeg
+
+插件用 ffmpeg 解码视频。先确认它能运行：
+
+```bash
+ffmpeg -version
+ffprobe -version
+```
+
+两条都要有输出。没有的话：
 
-### 方式一：交给智能体
+| 系统 | 安装命令 |
+| --- | --- |
+| Windows | `winget install Gyan.FFmpeg` |
+| macOS | `brew install ffmpeg` |
+| Ubuntu / Debian | `sudo apt install ffmpeg` |
 
-把下面的提示词交给智能体：
+装完新开一个终端再验证。不想放进 PATH，可以在插件配置里写 `ffmpegPath` 与 `ffprobePath` 的绝对路径。
+
+### 第二步：在 DSH 里安装插件
+
+1. 打开 DSH，点左侧栏的「插件」。
+2. 在列表里找「视频理解」（包名 `dsh-pf-stream`）。
+3. 点进详情页，把右上角的开关打开。
+4. 第一次开启后，详情页会出现引导卡片。
+
+列表里没有时，用插件页的安装入口从 npm 安装 `dsh-pf-stream`，装完再回到这一步。
+
+也可以把安装交给智能体：
 
 ```text
-安装 DSH 插件 dsh-pf-stream，按以下步骤执行：
-1. 用插件管理器从 npm 安装 dsh-pf-stream 最新版本。安装结果为 applied 才算成功。
-2. 检查 ffmpeg 与 ffprobe 是否可用。缺失时安装：
-   Windows 用 winget install Gyan.FFmpeg；
-   macOS 用 brew install ffmpeg；
-   Linux 用 sudo apt install ffmpeg。
-3. 对一个本地视频调用 probe。返回时长与分辨率即成功。
-4. 汇报插件版本、ffmpeg 位置、验证结果。
+安装 DSH 插件 dsh-pf-stream：
+1. 用插件管理器从 npm 安装最新版本，结果为 applied 才算成功；
+2. 检查 ffmpeg 与 ffprobe；缺失时安装（Windows: winget install Gyan.FFmpeg / macOS: brew install ffmpeg / Linux: sudo apt install ffmpeg）；
+3. 对一个本地视频调用 probe；返回时长与分辨率即成功；
+4. 汇报插件版本、ffmpeg 位置与检查结果。
 ```
 
-### 方式二：手动安装
+## 第一次配置转写
+
+转写引擎决定「没有字幕的视频能不能拿到台词」。不配置也能看画面，只是不会生成转写文本。
 
-1. 安装 ffmpeg 与 ffprobe：
+进入配置页：**左侧栏「插件」→ 点「视频理解」→ 详情页描述下方的面板**。以后要改设置，走同一条路。
 
-   ```bash
-   winget install Gyan.FFmpeg   # Windows
-   brew install ffmpeg          # macOS
-   sudo apt install ffmpeg      # Debian / Ubuntu
-   ```
+面板顶部有三张状态卡：视频解码（ffmpeg）、本地转写、云端转写。按需要选一条路线：
 
-2. 可执行文件不在 PATH 时，在插件配置里写 `ffmpegPath` 与 `ffprobePath`。
-3. 在宿主环境安装插件包：`pnpm add dsh-pf-stream`。
-4. 按包内 `cordis.patch.yml` 写宿主装载配置。
-5. 验证：新开 DSH 会话。工具列表出现 `dsh-pf-stream` 即成功。
+| 路线 | 适合 | 操作 |
+| --- | --- | --- |
+| 本地转写（推荐） | 免费、离线、不填密钥 | 点「下载本地模型」，约 190 MB；下载完点「测试转写」 |
+| 云端转写 | 已有 API Key，或机器性能较弱 | 选 OpenAI / Groq / 自定义兼容端点，填 Key，保存后点「测试转写」 |
+| 暂不配置 | 视频都有字幕 | 直接使用；无字幕的语音视频不会生成转写文本 |
 
-## 使用
+本地模型的下载步骤：
 
-对智能体直接提问：
+1. 点「下载本地模型」。进度条显示已下载大小，可随时取消。
+2. 下载中断后再次点击会续传，不用从头开始。
+3. 完成后状态卡显示「已就绪 · whisper.cpp」。
+4. 点「测试转写」。返回「本地引擎可用」即配置成功。
+
+云端密钥的保存步骤：
+
+1. 在「云端转写密钥」里选服务商。
+2. 粘贴 API Key，点「保存」。保存后只显示「已配置」，不再回显密钥。
+3. 点「测试转写」验证密钥与网络。
+
+默认策略是 `auto`：云端密钥、端点、模型都齐时用云端（更快，按量计费）；否则用本地引擎。面板标题旁会显示当前实际使用的引擎，如「转写可用 · 云端」。可以在「高级」里改成仅本地、仅云端或不转写。
+
+一键安装当前支持 Windows x64。其他系统请先安装 whisper.cpp，在配置文件 `audio.asr.local.binPath` 填写可执行文件路径，或改用云端转写。
+
+## 开始使用
+
+在对话里直接给视频路径并提问：
 
 - 「这个视频里发生了什么？」
 - 「第 12.9 秒的文字写的是什么？」
 - 「每 5 秒抽一帧，看画面怎么变化。」
+- 「把这段访谈里说的话按时间整理出来。」
+
+返回示例（节选）：
+
+```text
+[Packet #42 | Anchor: #105 | #101(P, 12.1s, Left-Top) → #105(F, 13.4s, Center)]
+Audio @Packet #42: [12.1s-12.9s] "这件事我们准备了三年" | [13.0s-13.4s] "今天终于可以说了"
+```
+
+文本块是结构化元数据：每个画布包给出时间、形态、锚点与单元位置；音频部分给出状态、来源与逐句时间。图像块是每个画布包一张画布。
 
-参数与返回内容见[使用指南](docs/使用指南.md)。
+常用问法对应四种取帧方式：
+
+| 想要 | 说法 |
+| --- | --- |
+| 先看整体 | 「看看这个视频」——流式采样，运动记过程、停稳给细节 |
+| 只看某一秒 | 「第 12.9 秒写了什么」——定点高清帧 |
+| 均匀看一遍 | 「每 5 秒抽一帧」——等间隔抽帧 |
+| 看局部 | 「放大右下角」——按区域裁剪后再看 |
+
+## 特点
+
+### 双尺度流式感知
+
+视频不再平均抽帧。画面运动时，插件用低分辨率的过程单元（P）记录变化；画面停稳后，用高分辨率的定格单元（F）保留细节。同一个画布包里既有过程也有结果，模型不用为了一个细节反复回看整段视频。
+
+采样节奏由画面变化驱动：变化快时降低采样密度、避免重复帧；变化慢时等待停稳再定格。长静止画面用心跳保持时间感，不重复发图。
+
+### 画布打包与计费
+
+一帧画面被切成 42px 超像素对齐的单元，按需取 P 或 F，再拼成一张画布。一次调用可以带多个画布包；每个包只包含当前新增的单元，历史包以引用哈希折叠，不重复传图。计费按画布网格计算（`r × (c + 1) + 2`）。
+
+### 确定性检视
+
+流式采样之外，另外三条路径是确定性的：`probe` 只读元数据；`timestamps` 按毫秒取指定帧；`uniform` 按固定间隔取帧；`roi_hint` 裁剪指定区域。这几种不依赖运动检测，结果可复现，适合核对文字、数字与小目标。
+
+### 台词的三级来源
+
+1. **字幕**：优先用外挂同名 `.srt` / `.vtt`，其次内嵌软字幕。
+2. **声学判断**：没有字幕时先做静音检测与语音占比估计；全程静音的视频直接跳过转写。
+3. **语音转写**：本地 whisper.cpp，或 OpenAI / Groq / 自定义兼容端点。转写结果按 PTS 挂到对应画布包，跨块边界按 ±300ms 缝合。
+
+默认「先回画面、后台补台词」：首次调用超过软截止（`graceMs`，默认 8 秒）就先返回视觉结果，转写在后台继续；同一视频再调用时直接命中磁盘缓存。
+
+### 长视频的冷区
+
+超出窗口的历史画布包会折叠成引用（锚点哈希 + 时间 + 台词文本），不重复传图。需要回看时再按时间点取帧。
+
+## 遇到问题
+
+| 现象 | 先看这里 |
+| --- | --- |
+| 提示找不到 ffmpeg | [排障：ffmpeg](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#找不到-ffmpeg) |
+| 有声音，但没有台词 | [排障：没有台词](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#有声音但没有台词) |
+| 下载模型失败或很慢 | [排障：下载失败](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#下载失败) |
+| 转写很慢、超时 | [排障：转写慢](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#转写很慢或超时) |
+| 设置页显示「只读」 | [排障：配置只读](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#设置页只读) |
 
 ## 文档
 
-- [使用指南](docs/使用指南.md)
-- [Usage Guide](docs/usage-guide.md)
-- [发布规范](docs/发布规范.md)
+- [使用指南](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/使用指南.md)：用法、参数、返回内容、计费、全部配置项
+- [排障](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md)：按现象查原因
+- [开发者指南](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/开发者指南.md)：架构、宿主契约、测试与发布
+- [Usage Guide](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/usage-guide.md)（English）
+
+## 本地开发
+
+```bash
+pnpm build
+pnpm test
+pnpm verify:host
+pnpm check:package
+```
+
+Host 半提供工具 `dsh-pf-stream`（`probe` / `pf_stream` / `timestamps` / `uniform`）、音频边车、本地引擎安装器与设置接口；Client 半是 `client.js`，在插件详情页注册配置面板与引导卡片。运行时依赖只有 `pngjs`。
+
+配置通过 `ctx.configEditor.edit()` 写入，保留用户未触碰的字段；密钥通过 `ctx.credentials` 存取，配置文件只保留引用。
 
 ## 许可证