dsh-pf-stream 0.2.3 → 0.3.0

package/README.en.md

@@ -1,146 +1,170 @@
 # dsh-pf-stream
 
-[中文](README.md) | **English**
+[中文](https://github.com/Weilv-D/dsh-pf-stream/blob/main/README.md) | **English**
 
-Video perception plugin for DeepSeek Harness (DSH). One tool lets the model "see" local video files:
+Let the agent watch videos: adaptive sampling, plus the spoken lines.
 
-- **Continuous events** → adaptive streaming sampling. During motion, the sampler emits low-token process units by displacement integration. On settle or sharpness peaks, it freezes a high-resolution focus unit.
-- **Suspicious time points** → deterministic inspection: metadata probe, frame extraction at given timestamps, uniform slicing, and local RoI zoom-in.
+## What it does
 
-Every canvas aligns with the DeepSeek-V4.1-Flash vision pipeline: 42-px superpixel grid, billing `r×(c+1)+2`. All four forms bill ≤530 tokens, within the 1024-token per-image limit. The server encodes each canvas at its original resolution.
+- **See**: motion is recorded as it happens; the moment it settles, detail is captured. Any timestamp can be inspected at full resolution.
+- **Hear**: subtitles are used when present; otherwise a transcription engine produces the speech and mounts it on the matching frames.
+- **Optional setup**: video works without transcription; with it, videos that only have a voice track also work.
 
-## Capability routing
+## Install
 
-One tool, `dsh-pf-stream`, routes through `intent.action`:
+### 1. Install ffmpeg
 
-| action | engine | use |
-| --- | --- | --- |
-| `pf_stream` (default) | adaptive streaming | 336² process units (P) during motion; 672² focus units (F) on settle/peak. Schmitt four-state machine + two-layer adaptation (intent baseline + content feedback R1-R4) |
-| `timestamps` | deterministic inspection | high-resolution frames at given timestamps; `roi_hint` zoom-in |
-| `uniform` | deterministic inspection | uniform slicing by step or target frame count |
-| `probe` | deterministic inspection | metadata only (display size, rotation, last-frame PTS, stream summary); zero vision tokens |
+The plugin decodes video with ffmpeg. Check that it runs:
 
-## Install
+```bash
+ffmpeg -version
+ffprobe -version
+```
 
-### 1. Install the plugin
+Both must print output. If not:
 
-- Open the DSH plugin manager, search for `dsh-pf-stream`, and install the latest version.
-- Or tell the agent "install the plugin dsh-pf-stream"; the agent runs the same flow through plugin_manager.
+| OS | Command |
+| --- | --- |
+| Windows | `winget install Gyan.FFmpeg` |
+| macOS | `brew install ffmpeg` |
+| Ubuntu / Debian | `sudo apt install ffmpeg` |
 
-The host performs the install: it adds the package through pnpm and mounts the plugin from `cordis.patch.yml`. Peer dependencies `@deepseek-ai/cordis` and `@deepseek-ai/schemastery` resolve in the host runtime. The only runtime dependency is `pngjs`; canvas PNG encoding uses fixed parameters, so output bytes are deterministic.
+Open a new terminal and check again. To keep it out of PATH, set `ffmpegPath` and `ffprobePath` in the plugin config.
 
-### 2. Prepare ffmpeg and ffprobe
+### 2. Install the plugin in DSH
 
-Pick one install method:
+1. Open DSH and click **Plugins** in the left sidebar.
+2. Find **Video understanding** (package `dsh-pf-stream`).
+3. Open its detail page and switch it on (top right).
+4. On first enable, a setup card appears on the detail page.
 
-```bash
-winget install Gyan.FFmpeg   # Windows
-brew install ffmpeg          # macOS
-sudo apt install ffmpeg      # Debian / Ubuntu
+If it is not listed, use the install entry on the Plugins page to install `dsh-pf-stream` from npm, then return here.
+
+You can also hand the install to the agent:
+
+```text
+Install the DSH plugin dsh-pf-stream:
+1. Install the latest version from npm with the plugin manager; success means result applied;
+2. Check ffmpeg and ffprobe; install them if missing (Windows: winget install Gyan.FFmpeg / macOS: brew install ffmpeg / Linux: sudo apt install ffmpeg);
+3. Call probe on a local video; duration and resolution in the result means success;
+4. Report the plugin version, ffmpeg location, and check results.
 ```
 
-You can also download from [ffmpeg.org](https://ffmpeg.org/download.html), unpack, and point an environment variable at the `bin` directory. Executable search order: plugin config `ffmpegPath` / `ffprobePath` → `DSH_FFMPEG_PATH` / `FFMPEG_PATH` → `DSH_FFMPEG_DIR` → `~/.dsh/tools/ffmpeg/bin` → `~/.dsh/bin` → `PATH`.
+## First run: configure transcription
 
-### 3. Verify the installation
+The transcription engine decides whether videos without subtitles get speech text. Without it, video still works; only the transcript is missing.
 
-After installation the model-side tool `dsh-pf-stream` appears. Probe any local video:
+To open the settings: **Plugins in the left sidebar → Video understanding → the panel below the description**. Use the same path later to change settings.
 
-```json
-{ "video_path": "D:/videos/demo.mp4", "intent": { "action": "probe" } }
-```
+Three cards at the top report video decode (ffmpeg), local transcription, and cloud transcription. Pick one path:
 
-A response with display size, duration, and frame rate confirms a working installation.
+| Path | Best for | What to do |
+| --- | --- | --- |
+| Local (recommended) | Free, offline, no API key | Click "Download local model" (~190 MB), then "Test transcription" |
+| Cloud | You have a key, or the machine is slow | Pick OpenAI / Groq / a custom compatible endpoint, paste the key, save, then test |
+| Skip for now | Videos have subtitles | Use it as is; speech-only videos get no transcript |
 
-## Usage
+Local model steps:
 
-Streaming perception ("what text is on screen"):
+1. Click "Download local model". The progress bar shows bytes received; cancel at any time.
+2. Clicking again after an interruption resumes the download.
+3. When done, the card reads "Ready · whisper.cpp".
+4. Click "Test transcription". "Local engine available" means it works.
 
-```json
-{
-  "video_path": "D:/videos/demo.mp4",
-  "intent": { "action": "pf_stream", "focus": "detail", "mode": "dense" }
-}
-```
+Cloud key steps:
+
+1. Choose a provider under "Cloud transcription key".
+2. Paste the API key and click "Save". After saving, only "Configured" is shown; the key is never echoed back.
+3. Click "Test transcription" to verify the key and network.
 
-Point inspection (text found at 12.9 s in a streaming result):
-
-```json
-{
-  "video_path": "D:/videos/demo.mp4",
-  "intent": {
-    "action": "timestamps",
-    "timestamps_ms": [12900],
-    "roi_hint": { "x": 0.2, "y": 0.3, "w": 0.4, "h": 0.4 }
-  }
-}
+The default policy is `auto`: use a cloud key when configured, otherwise the local engine. Under "Advanced" you can switch to local only, cloud only, or off.
+
+One-click install currently supports Windows x64. On other systems, install whisper.cpp and set `audio.asr.local.binPath` in the config file, or use cloud transcription.
+
+## Use it
+
+Give the agent a video path and ask:
+
+- "What happens in this video?"
+- "What does the text at 12.9 seconds say?"
+- "Take one frame every 5 seconds and describe how it changes."
+- "Transcribe this interview with timestamps."
+
+Example output (excerpt):
+
+```text
+[Packet #42 | Anchor: #105 | #101(P, 12.1s, Left-Top) → #105(F, 13.4s, Center)]
+Audio @Packet #42: [12.1s-12.9s] "we prepared this for three years" | [13.0s-13.4s] "today we can finally say it"
 ```
 
-The top level also accepts common convenience fields from the video-inspection domain: `start_time`, `end_time`, `timestamps`, `fps`, `interval`, `num_frames`, `crop`, `probe_only`. The entry adapter normalizes them into the standard `intent` structure.
+The text block is structured metadata: each canvas packet reports time, form, anchor, and unit positions; the audio part reports status, source, and per-line timings. The image block is one canvas per packet.
 
-## Returned content
+Four ways to look, by question:
 
-Each call returns one or more canvas packets:
+| Goal | Ask |
+| --- | --- |
+| Overview first | "Look at this video" — streaming sampling; motion as process, settle as detail |
+| One exact moment | "What is written at 12.9s" — deterministic high-resolution frame |
+| Even coverage | "One frame every 5 seconds" — uniform sampling |
+| A region | "Zoom into the bottom right" — crop by area, then look |
 
-- **PNG canvas**: one of four forms (table below). Two-tone separator lines block edge-feature crosstalk between units; an optional superpixel badge renders unit IDs.
-- **Structured metadata**: `packet_id`, per-unit PTS, token-grid bbox, source-sampling bbox, dual-carrier SHA-256.
-- **Timeline hint line**: a model-readable P/F timeline, for example `[Packet #42 | Anchor: #105 | #101(P, 12.1s, Left-Top) → …]`.
+## Design
 
-Session capabilities:
+### Dual-scale streaming perception
 
-- Intent persists across calls.
-- Streaming cursor resume: the next call on the same video continues from the last position.
-- Two-tier history reference: the hot tier reuses the latest 2 full strips; the cold tier reuses up to 4 independent F-anchor carriers.
-- Epoch compaction: hot-to-cold conversion in batches of 6 packets, keeping the multi-turn Prefix KV-Cache prefix stable.
+Video is not sampled at a fixed rate. While the picture moves, low-resolution process units (P) record change; when it settles, high-resolution freeze units (F) keep detail. A single canvas packet carries both process and result, so the model does not need to re-watch a clip to recover one detail.
 
-## Canvas and billing
+The sampling cadence follows the picture: fast change lowers density to avoid duplicates; slow change waits for a settle before freezing. Long static stretches keep a heartbeat instead of repeating images.
 
-| form | layout | tokens |
-| --- | --- | --- |
-| `FORM_1_STANDARD_STRIP` | 4P + 1F, 1344×672 | 530 |
-| `FORM_2_COMPACT_4P` | 4P, 672×672 | 274 |
-| `FORM_3_COMPACT_1F` | 1F, 672×672 | 274 |
-| `FORM_4_PARTIAL_STRIP` | 1-3P + 1F, 1344×672 | 530 |
+### Canvas packing and metering
 
-## Canvas samples
+A frame is divided into 42 px superpixel-aligned units. P or F units are taken as needed and packed into one canvas. One call can carry several packets; each packet holds only the units that are new, and older packets collapse into reference hashes instead of resending images. Metering follows the canvas grid (`r × (c + 1) + 2`).
 
-`FORM_1` standard strip: four process units (P) on the left, one focus unit (F) on the right.
+### Deterministic inspection
 
-![FORM_1 standard strip](https://raw.githubusercontent.com/Weilv-D/dsh-pf-stream/main/assets/screenshot-1.png)
+Besides streaming, three paths are deterministic: `probe` reads metadata only; `timestamps` returns exact frames in milliseconds; `uniform` samples at a fixed interval; `roi_hint` crops a region. These do not depend on motion detection, so results are reproducible and suited to reading text, numbers, and small objects.
 
-`FORM_2` process-only: four process units.
+### Three sources for speech
 
-![FORM_2 process-only](https://raw.githubusercontent.com/Weilv-D/dsh-pf-stream/main/assets/screenshot-2.png)
+1. **Subtitles**: a sidecar `.srt` / `.vtt` with the same name first, then embedded soft subtitles.
+2. **Acoustic check**: without subtitles, silence detection and a speech-ratio estimate run first; an all-silent video skips transcription.
+3. **Transcription**: local whisper.cpp, or OpenAI / Groq / a custom compatible endpoint. Segments are mounted on packets by PTS and stitched across block boundaries with a ±300 ms tolerance.
 
-`FORM_3` single focus: RoI zoom review via `roi_hint`.
+The default is "picture first, speech later": when the soft deadline (`graceMs`, 8 s by default) passes, the visual result returns and transcription continues in the background; the next call on the same video hits the disk cache.
 
-![FORM_3 single focus RoI zoom](https://raw.githubusercontent.com/Weilv-D/dsh-pf-stream/main/assets/screenshot-3.png)
+### Old regions
 
-## Configuration
+Canvas packets outside the window collapse into references (anchor hash, time, and transcript text) instead of resending images. A timestamp call brings the detail back when needed.
 
-| option | default | description |
-| --- | --- | --- |
-| `renderVisualId` | `true` | superpixel inverse badge at each unit corner |
-| `maxPacketsPerCall` | `64` | packet count limit per call |
-| `maxFramesPerCall` | `128` | deterministic frame target limit per call (clamps `frame_count`) |
-| `maxTokensBudgetPerCall` | `65536` | vision-token budget per call |
-| `maxRequestBytes` | `56 MiB` | request byte watermark, Base64-inclusive |
-| `maxColdRefs` | `4` | cold-tier reference limit |
-| `defaultStreamWindowMs` | `10000` | window length in ms when `pts_window` is absent |
-| `streamWorkMaxEdge` | `2048` | working-resolution long edge for streaming decode; deterministic inspection always uses full resolution |
-| `epochCompactionWindow` | `6` | hot-to-cold batch window, in packets |
-| `meteringProfile.imageCountTiersEnabled` | `false` | gate for the ≥15-image 4096-px rule |
-
-## Testing
+## Troubleshooting
+
+| Symptom | Read |
+| --- | --- |
+| ffmpeg not found | [ffmpeg](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#找不到-ffmpeg) |
+| Speech but no transcript | [No transcript](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#有声音但没有台词) |
+| Model download fails or is slow | [Download](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#下载失败) |
+| Transcription is slow or times out | [Slow transcription](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#转写很慢或超时) |
+| Settings page is read-only | [Read-only config](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#设置页只读) |
+
+## Docs
+
+- [Usage Guide](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/usage-guide.md) (English)
+- [使用指南](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/使用指南.md) (Chinese)
+- [排障](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md) (Chinese)
+- [开发者指南](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/开发者指南.md) (Chinese): architecture, host contracts, tests, release
+
+## Development
 
 ```bash
-pnpm test        # boundary (pure logic) + e2e (fixture-driven), 20 boundary cases
-pnpm run build   # TypeScript compile
-pnpm verify:host # host-contract self-check (schema shape, keyword subset, dependency boundary)
+pnpm build
+pnpm test
+pnpm verify:host
+pnpm check:package
 ```
 
-## Release
+The host half provides the `dsh-pf-stream` tool (`probe` / `pf_stream` / `timestamps` / `uniform`), the audio sidecar, the local engine installer, and the settings API. The client half is `client.js`, which registers the config panel and the setup card on the plugin detail page. The only runtime dependency is `pngjs`.
 
-npm releases run automatically through GitHub Actions. Process and version policy: [docs/发布规范.md](docs/发布规范.md) (Chinese).
+Configuration is written through `ctx.configEditor.edit()`, which keeps fields the user did not touch; keys live in `ctx.credentials`, and the config file only holds a reference.
 
 ## License
 

package/README.md

@@ -1,147 +1,170 @@
 # dsh-pf-stream
 
-**中文** | [English](README.en.md)
+**中文** | [English](https://github.com/Weilv-D/dsh-pf-stream/blob/main/README.en.md)
 
-DeepSeek Harness（DSH）视频感知插件。模型用一个工具即可「看」本地视频：
+让智能体看懂视频：动态抽帧，加上台词。
 
-- **连续事件** → 自适应流式采样。运动时按位移积分抽低 token 过程单元；停稳与顿挫波峰时定格高清焦点单元。
-- **可疑时间点** → 确定性检视。能力：元数据探测、指定时间戳抽帧、均匀切片、局部 RoI 放大复核。
+## 能做什么
 
-全部画布对齐 DeepSeek-V4.1-Flash 视觉编码管线：42px 超像素网格，计费 `r×(c+1)+2`。四种形态计费均 ≤530 token，满足单图 1024 token 上限。服务端按画布原始分辨率计量。
+- **看画面**：画面运动时记录过程，停稳时定格细节；指定时间点可以高清复核。
+- **听声音**：有字幕就用字幕；没有字幕时，配置转写引擎后生成台词，并按时间挂到对应画面。
+- **按需配置**：不配置转写也能处理画面；配置后可以处理带音轨的视频。
 
-## 能力路由
+## 安装
 
-一个主工具 `dsh-pf-stream`，统一 `intent.action` 路由：
+### 第一步：安装 ffmpeg
 
-| action | 引擎 | 用途 |
-| --- | --- | --- |
-| `pf_stream`（默认） | 动态流式采样 | 运动中抽 336² 过程单元（P）；停稳/波峰定格 672² 焦点单元（F）。施密特四状态机 + 双层自适应调节（意图基准 + 内容自闭环 R1-R4） |
-| `timestamps` | 确定性检视 | 指定时间戳单帧/组合高清检视，支持 `roi_hint` 局部放大 |
-| `uniform` | 确定性检视 | 固定步长或目标帧数均匀切片 |
-| `probe` | 确定性检视 | 仅元数据（显示宽高/旋转/末帧 PTS/流摘要），零视觉 token |
+插件用 ffmpeg 解码视频。先确认它能运行：
 
-## 安装
+```bash
+ffmpeg -version
+ffprobe -version
+```
 
-### 1. 安装插件
+两条都要有输出。没有的话：
 
-- 在 DSH 插件管理器中搜索 `dsh-pf-stream`，安装最新版本。
-- 或直接对智能体说「安装插件 dsh-pf-stream」，智能体经 plugin_manager 完成同一流程。
+| 系统 | 安装命令 |
+| --- | --- |
+| Windows | `winget install Gyan.FFmpeg` |
+| macOS | `brew install ffmpeg` |
+| Ubuntu / Debian | `sudo apt install ffmpeg` |
 
-安装由宿主执行：宿主经 pnpm 安装包，读取 `cordis.patch.yml` 挂载插件。peer 依赖 `@deepseek-ai/cordis` 与 `@deepseek-ai/schemastery` 由宿主运行时解析；运行时直接依赖仅 `pngjs`，画布 PNG 编码参数固定，输出字节幂等。
+装完新开一个终端再验证。不想放进 PATH，可以在插件配置里写 `ffmpegPath` 与 `ffprobePath` 的绝对路径。
 
-### 2. 准备 ffmpeg 与 ffprobe
+### 第二步：在 DSH 里安装插件
 
-任选一种安装方式：
+1. 打开 DSH，点左侧栏的「插件」。
+2. 在列表里找「视频理解」（包名 `dsh-pf-stream`）。
+3. 点进详情页，把右上角的开关打开。
+4. 第一次开启后，详情页会出现引导卡片。
 
-```bash
-winget install Gyan.FFmpeg   # Windows
-brew install ffmpeg          # macOS
-sudo apt install ffmpeg      # Debian / Ubuntu
+列表里没有时，用插件页的安装入口从 npm 安装 `dsh-pf-stream`，装完再回到这一步。
+
+也可以把安装交给智能体：
+
+```text
+安装 DSH 插件 dsh-pf-stream：
+1. 用插件管理器从 npm 安装最新版本，结果为 applied 才算成功；
+2. 检查 ffmpeg 与 ffprobe；缺失时安装（Windows: winget install Gyan.FFmpeg / macOS: brew install ffmpeg / Linux: sudo apt install ffmpeg）；
+3. 对一个本地视频调用 probe；返回时长与分辨率即成功；
+4. 汇报插件版本、ffmpeg 位置与检查结果。
 ```
 
-也可从 [ffmpeg.org](https://ffmpeg.org/download.html) 下载解压，用环境变量指向 `bin` 目录。可执行文件定位顺序：插件配置 `ffmpegPath` / `ffprobePath` → `DSH_FFMPEG_PATH` / `FFMPEG_PATH` → `DSH_FFMPEG_DIR` → `~/.dsh/tools/ffmpeg/bin` → `~/.dsh/bin` → `PATH`。
+## 第一次配置转写
 
-### 3. 验证安装
+转写引擎决定「没有字幕的视频能不能拿到台词」。不配置也能看画面，只是不会生成转写文本。
 
-安装成功后，模型侧出现工具 `dsh-pf-stream`。用任一本地视频探测元数据：
+进入配置页：**左侧栏「插件」→ 点「视频理解」→ 详情页描述下方的面板**。以后要改设置，走同一条路。
 
-```json
-{ "video_path": "D:/videos/demo.mp4", "intent": { "action": "probe" } }
-```
+面板顶部有三张状态卡：视频解码（ffmpeg）、本地转写、云端转写。按需要选一条路线：
 
-返回显示宽高、时长、帧率，即安装成功。
+| 路线 | 适合 | 操作 |
+| --- | --- | --- |
+| 本地转写（推荐） | 免费、离线、不填密钥 | 点「下载本地模型」，约 190 MB；下载完点「测试转写」 |
+| 云端转写 | 已有 API Key，或机器性能较弱 | 选 OpenAI / Groq / 自定义兼容端点，填 Key，保存后点「测试转写」 |
+| 暂不配置 | 视频都有字幕 | 直接使用；无字幕的语音视频不会生成转写文本 |
 
-## 使用
+本地模型的下载步骤：
 
-流式感知（问「画面里写了什么」）：
+1. 点「下载本地模型」。进度条显示已下载大小，可随时取消。
+2. 下载中断后再次点击会续传，不用从头开始。
+3. 完成后状态卡显示「已就绪 · whisper.cpp」。
+4. 点「测试转写」。返回「本地引擎可用」即配置成功。
 
-```json
-{
-  "video_path": "D:/videos/demo.mp4",
-  "intent": { "action": "pf_stream", "focus": "detail", "mode": "dense" }
-}
-```
+云端密钥的保存步骤：
+
+1. 在「云端转写密钥」里选服务商。
+2. 粘贴 API Key，点「保存」。保存后只显示「已配置」，不再回显密钥。
+3. 点「测试转写」验证密钥与网络。
 
-定点复核（流式结果中 12.9s 处发现文字）：
-
-```json
-{
-  "video_path": "D:/videos/demo.mp4",
-  "intent": {
-    "action": "timestamps",
-    "timestamps_ms": [12900],
-    "roi_hint": { "x": 0.2, "y": 0.3, "w": 0.4, "h": 0.4 }
-  }
-}
+默认策略是 `auto`：云端密钥、端点、模型都齐时用云端（更快，按量计费）；否则用本地引擎。面板标题旁会显示当前实际使用的引擎，如「转写可用 · 云端」。可以在「高级」里改成仅本地、仅云端或不转写。
+
+一键安装当前支持 Windows x64。其他系统请先安装 whisper.cpp，在配置文件 `audio.asr.local.binPath` 填写可执行文件路径，或改用云端转写。
+
+## 开始使用
+
+在对话里直接给视频路径并提问：
+
+- 「这个视频里发生了什么？」
+- 「第 12.9 秒的文字写的是什么？」
+- 「每 5 秒抽一帧，看画面怎么变化。」
+- 「把这段访谈里说的话按时间整理出来。」
+
+返回示例（节选）：
+
+```text
+[Packet #42 | Anchor: #105 | #101(P, 12.1s, Left-Top) → #105(F, 13.4s, Center)]
+Audio @Packet #42: [12.1s-12.9s] "这件事我们准备了三年" | [13.0s-13.4s] "今天终于可以说了"
 ```
 
-顶层同时接受视频检视领域常见便捷字段：`start_time` / `end_time` / `timestamps` / `fps` / `interval` / `num_frames` / `crop` / `probe_only`。入口适配器将其归一化为标准 `intent` 结构。
+文本块是结构化元数据：每个画布包给出时间、形态、锚点与单元位置；音频部分给出状态、来源与逐句时间。图像块是每个画布包一张画布。
 
-## 返回内容
+常用问法对应四种取帧方式：
 
-每次调用返回一个或多个画布包：
+| 想要 | 说法 |
+| --- | --- |
+| 先看整体 | 「看看这个视频」——流式采样，运动记过程、停稳给细节 |
+| 只看某一秒 | 「第 12.9 秒写了什么」——定点高清帧 |
+| 均匀看一遍 | 「每 5 秒抽一帧」——等间隔抽帧 |
+| 看局部 | 「放大右下角」——按区域裁剪后再看 |
 
-- **PNG 画布**：四形态之一（见下表）。单元间双色隔离线阻断相邻边缘特征串扰；可选超像素角标渲染单元编号。画布 PNG 同时作为图像内容块进入模型上下文（单次调用至多 `maxImagesPerCall` 张）。
-- **结构化元数据**：`packet_id`、每单元 PTS、token 网格 bbox、源采样 bbox、双载体 SHA-256。
-- **时序提示行**：模型可读的 P/F 单元时间轴（如 `[Packet #42 | Anchor: #105 | #101(P, 12.1s, Left-Top) → …]`）。
+## 特点
 
-会话级能力：
+### 双尺度流式感知
 
-- 意图跨调用持久化。
-- 流式游标续窗：同视频下次调用从上次位置继续。
-- 两级历史引用：热区最近 2 包整条带复用；冷区至多 4 个 F 锚点独立载体复用。
-- Epoch Compaction：默认 6 包窗口批量热转冷，保持多轮对话 Prefix KV-Cache 前缀稳定。
+视频不再平均抽帧。画面运动时，插件用低分辨率的过程单元（P）记录变化；画面停稳后，用高分辨率的定格单元（F）保留细节。同一个画布包里既有过程也有结果，模型不用为了一个细节反复回看整段视频。
 
-## 画布与计费
+采样节奏由画面变化驱动：变化快时降低采样密度、避免重复帧；变化慢时等待停稳再定格。长静止画面用心跳保持时间感，不重复发图。
 
-| 形态 | 布局 | Token |
-| --- | --- | --- |
-| `FORM_1_STANDARD_STRIP` | 4P + 1F，1344×672 | 530 |
-| `FORM_2_COMPACT_4P` | 4P，672×672 | 274 |
-| `FORM_3_COMPACT_1F` | 1F，672×672 | 274 |
-| `FORM_4_PARTIAL_STRIP` | 1-3P + 1F，1344×672 | 530 |
+### 画布打包与计费
 
-## 画布样例
+一帧画面被切成 42px 超像素对齐的单元，按需取 P 或 F，再拼成一张画布。一次调用可以带多个画布包；每个包只包含当前新增的单元，历史包以引用哈希折叠，不重复传图。计费按画布网格计算（`r × (c + 1) + 2`）。
 
-`FORM_1` 标准条带：左半 4 个过程单元（P），右半 1 个焦点单元（F）。
+### 确定性检视
 
-![FORM_1 标准条带](https://raw.githubusercontent.com/Weilv-D/dsh-pf-stream/main/assets/screenshot-1.png)
+流式采样之外，另外三条路径是确定性的：`probe` 只读元数据；`timestamps` 按毫秒取指定帧；`uniform` 按固定间隔取帧；`roi_hint` 裁剪指定区域。这几种不依赖运动检测，结果可复现，适合核对文字、数字与小目标。
 
-`FORM_2` 纯过程：4 个过程单元。
+### 台词的三级来源
 
-![FORM_2 纯过程](https://raw.githubusercontent.com/Weilv-D/dsh-pf-stream/main/assets/screenshot-2.png)
+1. **字幕**：优先用外挂同名 `.srt` / `.vtt`，其次内嵌软字幕。
+2. **声学判断**：没有字幕时先做静音检测与语音占比估计；全程静音的视频直接跳过转写。
+3. **语音转写**：本地 whisper.cpp，或 OpenAI / Groq / 自定义兼容端点。转写结果按 PTS 挂到对应画布包，跨块边界按 ±300ms 缝合。
 
-`FORM_3` 单焦点：`roi_hint` 局部放大检视。
+默认「先回画面、后台补台词」：首次调用超过软截止（`graceMs`，默认 8 秒）就先返回视觉结果，转写在后台继续；同一视频再调用时直接命中磁盘缓存。
 
-![FORM_3 单焦点 RoI 放大](https://raw.githubusercontent.com/Weilv-D/dsh-pf-stream/main/assets/screenshot-3.png)
+### 长视频的冷区
 
-## 配置
+超出窗口的历史画布包会折叠成引用（锚点哈希 + 时间 + 台词文本），不重复传图。需要回看时再按时间点取帧。
 
-| 配置项 | 默认 | 说明 |
-| --- | --- | --- |
-| `renderVisualId` | `true` | 单元左上角超像素反色角标 |
-| `maxPacketsPerCall` | `64` | 单次调用返回包数上限 |
-| `maxFramesPerCall` | `128` | 确定性抽帧单次目标帧数上限（`frame_count` 受此钳制） |
-| `maxImagesPerCall` | `20` | 单次调用回传图像上限（宿主单条消息图像上限 20） |
-| `maxTokensBudgetPerCall` | `65536` | 单次调用视觉 token 预算 |
-| `maxRequestBytes` | `56 MiB` | 请求字节软水位（Base64 后口径） |
-| `maxColdRefs` | `4` | 冷区引用上限 |
-| `defaultStreamWindowMs` | `10000` | `pts_window` 缺省时窗口长度（毫秒） |
-| `streamWorkMaxEdge` | `2048` | 流式解码工作分辨率长边上限（定点检视恒全分辨率） |
-| `epochCompactionWindow` | `6` | 热转冷批量收敛窗口（包数） |
-| `meteringProfile.imageCountTiersEnabled` | `false` | ≥15 图单边 4096px 规则门控 |
-
-## 测试
+## 遇到问题
+
+| 现象 | 先看这里 |
+| --- | --- |
+| 提示找不到 ffmpeg | [排障：ffmpeg](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#找不到-ffmpeg) |
+| 有声音，但没有台词 | [排障：没有台词](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#有声音但没有台词) |
+| 下载模型失败或很慢 | [排障：下载失败](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#下载失败) |
+| 转写很慢、超时 | [排障：转写慢](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#转写很慢或超时) |
+| 设置页显示「只读」 | [排障：配置只读](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md#设置页只读) |
+
+## 文档
+
+- [使用指南](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/使用指南.md)：用法、参数、返回内容、计费、全部配置项
+- [排障](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/排障.md)：按现象查原因
+- [开发者指南](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/开发者指南.md)：架构、宿主契约、测试与发布
+- [Usage Guide](https://github.com/Weilv-D/dsh-pf-stream/blob/main/docs/usage-guide.md)（English）
+
+## 本地开发
 
 ```bash
-pnpm test        # boundary（纯逻辑）+ e2e（fixture 驱动），20 项边界用例
-pnpm run build   # TypeScript 编译
-pnpm verify:host # 宿主契约自检（Schema 形状/关键字子集/依赖边界）
+pnpm build
+pnpm test
+pnpm verify:host
+pnpm check:package
 ```
 
-## 发布
+Host 半提供工具 `dsh-pf-stream`（`probe` / `pf_stream` / `timestamps` / `uniform`）、音频边车、本地引擎安装器与设置接口；Client 半是 `client.js`，在插件详情页注册配置面板与引导卡片。运行时依赖只有 `pngjs`。
 
-npm 发布由 GitHub Actions 自动执行。流程与版本策略见 [docs/发布规范.md](docs/发布规范.md)。
+配置通过 `ctx.configEditor.edit()` 写入，保留用户未触碰的字段；密钥通过 `ctx.credentials` 存取，配置文件只保留引用。
 
 ## 许可证