dsh-video-insight 0.8.4 → 0.8.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +21 -7
- package/client/client.js +70 -44
- package/lib/asr.js +1 -1
- package/lib/assets.js +1 -1
- package/lib/budget.js +84 -0
- package/lib/frames.js +37 -22
- package/lib/index.js +4 -3
- package/lib/media.js +5 -2
- package/lib/ocr.js +19 -3
- package/lib/storyboard.js +136 -60
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
[](LICENSE)
|
|
4
4
|
[](https://github.com/deepseek-ai)
|
|
5
5
|
[](#已知限制)
|
|
6
|
-
[](package.json)
|
|
7
7
|
|
|
8
8
|

|
|
9
9
|
|
|
@@ -30,7 +30,7 @@
|
|
|
30
30
|
| 原文 | 字幕逐段列表(`[mm:ss] 文本`),可搜索过滤、点单条复制、一键复制全文;「双语」开关把每段翻译成中文/英文(`POST /translate`,按批送模型、结果按视频缓存),开启「跟随」后播放器播到哪、对应段落就高亮滚到视野里(时间轴同步由播放器外壳每秒回传) |
|
|
31
31
|
| 思维导图 | 模型把字幕整理成 3 层树状大纲(`GET /mindmap`,带缓存),面板渲染成 **SVG 图形化横向树**(节点卡片 + 贝塞尔连线、按分支配色),点节点折叠/展开,支持全部展开/收起、缩放、重新生成;节点上带可点的时间徽标(模型按字幕定位),点一下跳到视频对应位置 |
|
|
32
32
|
| 关键画面 | 关键画面总览(PPT 模式):与章节共用同一份大纲(`GET /outline`),一屏看到每章的代表画面;点「生成画面描述」让视觉模型一次性看完这套画面并给每章写一句描述(`POST /outline/visuals`,拼图后只发一张图),描述按视频缓存,章节页签也复用;点一张跳到那个画面并继续播放 |
|
|
33
|
-
| 分镜 |
|
|
33
|
+
| 分镜 | 按镜头切点抽帧、拼成**分镜拼图**(`POST /storyboard`,3 列;**帧数与张数按视频时长和剪辑密度自动定**:默认约每 20 秒一帧,一张最多 24 格,超过自动分成 2 张),每格左上角标时间;点任意一格跳到那个画面并继续播放。可选「附画面描述」让视觉模型逐张给每格写一句描述(悬停格子可见)。图片按视频缓存在 `~/.dsh/video-storyboards/`,切回页签秒开;低清视频和镜头切点也按视频缓存在 `~/.dsh/video-assets/`,转写、画面理解、分镜共用同一份,第二次打开页签不再重新下载 |
|
|
34
34
|
| 追问 | 左栏下方对话框:基于转写稿的 RAG 多轮问答(检索相关段落后再回答),回答带 `[mm:ss]`,下方「依据」chip 可点击跳转;对话按视频缓存。空态先给固定三句,再给 **AI 根据本视频总结生成的 3 条建议追问**(虚线 chip,点一下直接问) |
|
|
35
35
|
| 设置 | 填 B 站 Cookie(SESSDATA),保存到 `~/.dsh/video-credentials.json`(0600);可清除 |
|
|
36
36
|
| 历史 | 最近 100 条解析/分析记录(可搜索、可收藏星标),存在 `~/.dsh/video-history.json`(跨浏览器、跨重启都在),点击重新打开,可单条删除或清空 |
|
|
@@ -90,7 +90,7 @@ curl -L -o ggml-large-v3-turbo-q5_0.bin \
|
|
|
90
90
|
1. yt-dlp 下载 480p 以内的视频;
|
|
91
91
|
2. **按镜头切点抽帧**(`frameSampling: cuts`,默认):先扫一遍画面找剪辑点,一个镜头取一张,再按时间轴等分补齐(镜头太少时退化成均匀铺满);切分失败会自动退回「每 `ocrIntervalSeconds` 秒抽一帧」(`frameSampling: uniform`)。阈值 `cutThreshold` 默认 27,越高只抓越明显的转场;扫描只多一次 64×36 解码(实测 10 分钟片子约 4 秒),抽帧按时间点精确定位;
|
|
92
92
|
3. **本机 OCR**:用 macOS 自带的 Vision 框架离线识别中英文文字(首次使用会把一个小工具编译到 `~/.dsh/video-tools/ocr/`,约 10-20 秒,之后复用),完全不出本机;
|
|
93
|
-
4.
|
|
93
|
+
4. **画面理解(可选)**:把抽样的若干帧(帧数按时长自动定,默认约每 20 秒一帧、最多 96 帧;每 24 帧拼一张,可放多张)**拼成一张分镜图**(`visionPack: sheet`,默认 3 列、行数按帧数自动算)再交给支持图片输入的模型,让它按「第几格 = 第几秒」逐格描述场景、人物/物体、动作和画面文字;拼图失败会自动回落到逐帧发送(`visionPack: frames`)。默认 `frameMode: auto` 时,只有 OCR 平均每帧不足 4 字(比如纯动画、vlog)才追加这一轮;
|
|
94
94
|
5. 合并成带时间戳的文字稿:有画面描述时 `source: frame-vision`(视觉模型),只有本机文字时 `source: frame-ocr`;后续总结、章节、思维导图、追问都能用。
|
|
95
95
|
|
|
96
96
|
**为什么拼成一张图**:逐帧发时 12 帧就是 3 次请求、24 帧是 6 次,每张图都单独占一份图片 token;拼图后 24 帧只用 1 次请求、1 张图。实测同一段素材(Big Buck Bunny,24 帧):输入 token 13,312 → 1,905(−86%),耗时 66.0s → 26.9s(−59%),描述覆盖度与逐帧基本一致;12 帧时也分别是 −73% 和 −38%。
|
|
@@ -157,9 +157,9 @@ host 半零外部依赖(只用 Node 内置 `fetch`/`crypto`),不写锁文
|
|
|
157
157
|
| POST | `/dsh-video/api/transcribe` | SSE:`{url?/docId?, force?}`,帧 `phase` / `progress{percent,stage}` / `done{docId,transcript,cached?}` / `error{code,message}`;无语音时自动兜底 OCR / 画面理解(stage 含 `vision`) |
|
|
158
158
|
| POST | `/dsh-video/api/analyze` | SSE:`{mode:'summary'\|'ask', url?/docId?, question?, history?}`,帧 `phase` / `delta` / `done` / `error` |
|
|
159
159
|
| POST | `/dsh-video/api/chapters` | `{url?/docId?}` → `{ok,value:{chapters:[{start,title,gist}],duration}}` |
|
|
160
|
-
| GET | `/dsh-video/api/storyboard?docId=` | 取缓存的分镜:`{ok,value:{docId,url,title,platform,duration,generatedAt,tookMs,frameSampling,cuts,frames,cols,rows,cellWidth,cellHeight,described,cells:[{index,row,col,seconds,description?}]}}\|null` |
|
|
161
|
-
| GET | `/dsh-video/api/storyboard/image?docId=` |
|
|
162
|
-
| POST | `/dsh-video/api/storyboard` | SSE:`{docId?/url?, frames?:
|
|
160
|
+
| GET | `/dsh-video/api/storyboard?docId=` | 取缓存的分镜:`{ok,value:{docId,url,title,platform,duration,generatedAt,tookMs,frameSampling,cuts,frames,budget?,cols,rows,cellWidth,cellHeight,sheets:[{index,cols,rows,cellWidth,cellHeight,cells}],described,cells:[{index,sheet,row,col,seconds,description?}]}}\|null` |
|
|
161
|
+
| GET | `/dsh-video/api/storyboard/image?docId=&sheet=` | 第 N 张分镜拼图(`sheet` 从 0 开始,默认 0)`image/jpeg`(`no-store`);没生成过返回 `{code:'not-found'}` |
|
|
162
|
+
| POST | `/dsh-video/api/storyboard` | SSE:`{docId?/url?, frames?:0, cols?:3, describe?:boolean, force?:boolean}`(`frames: 0/缺省` = 按时长自动预算),帧 `phase` / `progress{stage:download\|scan\|frames,percent}` / `done{cached,docId,storyboard}` / `error{code,message}` |
|
|
163
163
|
| POST | `/dsh-video/api/storyboard/clear` | `{docId}` 删掉该视频的分镜(json + jpg) |
|
|
164
164
|
| POST | `/dsh-video/api/history/favorite` | `{docId, favorite}` 历史列表星标:返回 `{ok,value:{items}}`(重新解析不会冲掉) |
|
|
165
165
|
| GET | `/dsh-video/api/outline?docId=` | 章节/关键画面共用的大纲:`{ok,value:{docId,duration,source:'mindmap'\|'chapters'\|'uniform',signature,visuals,items:[{seconds,title,detail,description}]}}`;`visuals` 是已缓存的画面描述(没有则 null) |
|
|
@@ -239,10 +239,24 @@ host 半零外部依赖(只用 Node 内置 `fetch`/`crypto`),不写锁文
|
|
|
239
239
|
- 按镜头抽帧时,画面文字段的「结束时间」按相邻帧间隔的中位数估算(各帧自身的秒数是准的),时间轴粗一点但不会串行;
|
|
240
240
|
- 语音转写与画面分析**并行**:`crossCheck` 默认开启,`POST /transcribe` 一开始就同时下载音频和低清视频、跑 whisper 与抽帧识别,两路结果合并成一份稿子(画面描述带「画面:」前缀)互相印证;显式设 `crossCheck: false` 才回到「先语音、没有语音再看画面」的顺序。
|
|
241
241
|
- 画面文字识别:只在 macOS 可用(依赖系统 Vision 框架);识别质量取决于画面文字是否清晰,滚动弹幕/花哨字体容易出错;抽帧默认按镜头切点(找不到切点时均匀铺满),因此长视频的覆盖密度取决于镜头数量而不是固定间隔。
|
|
242
|
+
### 帧数是怎么定的
|
|
243
|
+
|
|
244
|
+
不再固定 24 帧。`lib/budget.js` 按「时长 + 剪辑密度」算预算(参考 dsh-video-storyboard 的窗口思路):
|
|
245
|
+
|
|
246
|
+
| 情况 | 密度 | 例 |
|
|
247
|
+
|---|---|---|
|
|
248
|
+
| 常规节奏 | 每 20 秒 1 帧 | 20 分钟 → 60 帧(分 3 张拼图,受上限裁到 48 帧 / 2 张) |
|
|
249
|
+
| 剪辑密集(平均 <8 秒一个镜头) | 每 12 秒 1 帧 | 3.5 分钟 MV → 28 帧 / 2 张 |
|
|
250
|
+
| 长镜头 / 口播(平均 >45 秒一个镜头) | 每 30 秒 1 帧 | 1 小时讲座 → 96 帧(上限) |
|
|
251
|
+
|
|
252
|
+
- 下限 6 帧、上限 `visionMaxFrames`(默认 96,可调到 240);
|
|
253
|
+
- 镜头数也参与:至少接近「一镜半帧」,碎剪的片子不会因为时长预算被削平;
|
|
254
|
+
- 一张拼图最多 24 格,超出的自动分第二张,视觉模型逐张描述后合并。
|
|
255
|
+
|
|
242
256
|
- 画面理解:需要当前(或设置里指定的)模型声明 `image` 输入能力;会把拼好的分镜图发给模型(与 OCR 的本机离线不同);默认一张图装 9–24 格,所以按「张」而不是按「帧」计费(比逐帧省很多),单次生成仍要 15–30s;画面里既没字也没可描述内容时仍会得到 `asr-empty`。
|
|
243
257
|
- 本地转写:Whisper 首次加载 574MB 模型约 6–12 秒(每次调用固定成本,长视频占比可忽略);SenseVoice 段级时间戳是按标点聚合的近似值;`threads: 0` 交给引擎自己决定(whisper.cpp 默认 4)。
|
|
244
258
|
- 双语对照:翻译按批调用当前模型(40 段 / 3200 字一批),长视频要十几批、每批几秒;译文质量和模型相关;「跟随播放」高亮依赖播放器外壳回传时间,目前只有 YouTube 内嵌播放器会发,B 站播放器拿不到播放进度(不报错、不高亮)。
|
|
245
|
-
-
|
|
259
|
+
- 分镜:最多 2 张拼图 / 48 格(20 分钟片子约 46 格),第一次生成要下 480p + 跑一遍切点扫描(10 分钟片子约 10–20 秒);低清视频与切点会按视频缓存(`~/.dsh/video-assets/`,最多 8 个 / 2GB / 30 天),之后转写、画面理解、分镜都复用,重新生成约 1 秒;「附画面描述」会调用视觉模型,图片会离开本机。
|
|
246
260
|
- 只支持单条链接,不做播放列表 / 合集递归。
|
|
247
261
|
|
|
248
262
|
## 开发与自测
|
package/client/client.js
CHANGED
|
@@ -1282,11 +1282,12 @@ window.__ModuleLoader__.load({
|
|
|
1282
1282
|
/* -------------------------------------------------------------- 分镜 */
|
|
1283
1283
|
|
|
1284
1284
|
/** 分镜图地址:优先 transport 的 http 起源,其次同源相对路径(与播放器外壳一致)。 */
|
|
1285
|
-
function storyboardImageSrc(storyboard, docId) {
|
|
1285
|
+
function storyboardImageSrc(storyboard, docId, sheetIndex) {
|
|
1286
1286
|
const id = String((storyboard && storyboard.docId) || docId || '').trim();
|
|
1287
1287
|
if (!id) return '';
|
|
1288
1288
|
const version = storyboard && storyboard.generatedAt ? String(storyboard.generatedAt) : '';
|
|
1289
1289
|
let path = `/dsh-video/api/storyboard/image?docId=${encodeURIComponent(id)}`;
|
|
1290
|
+
if (sheetIndex !== undefined && sheetIndex !== null) path += `&sheet=${Math.max(0, Math.floor(Number(sheetIndex) || 0))}`;
|
|
1290
1291
|
if (version) path += `&v=${encodeURIComponent(version)}`;
|
|
1291
1292
|
const transport = playerTransportBase();
|
|
1292
1293
|
return transport ? `${transport}${path}` : path;
|
|
@@ -1606,22 +1607,32 @@ window.__ModuleLoader__.load({
|
|
|
1606
1607
|
|
|
1607
1608
|
/* -------------------------------------------------------------- 分镜 */
|
|
1608
1609
|
|
|
1609
|
-
/** 分镜页签:host
|
|
1610
|
+
/** 分镜页签:host 拼好的分镜图(可能多张)+ 每格时间戳覆盖层(点击 seek)。 */
|
|
1610
1611
|
function StoryboardTab({ doc, storyboard, describe, onDescribe, onGenerate, onCancel, onSeek }) {
|
|
1611
|
-
const [
|
|
1612
|
+
const [brokenSheets, setBrokenSheets] = useState({});
|
|
1612
1613
|
const data = storyboard ? storyboard.data : null;
|
|
1613
1614
|
const loading = Boolean(storyboard && storyboard.loading);
|
|
1614
1615
|
const error = storyboard ? storyboard.error : '';
|
|
1615
|
-
useEffect(() => {
|
|
1616
|
+
useEffect(() => { setBrokenSheets({}); }, [data && data.generatedAt]);
|
|
1616
1617
|
if (!doc) {
|
|
1617
1618
|
return h('div', { className: 'dvv-tabpane dvv-sb-tab' },
|
|
1618
1619
|
h('div', { className: 'dvv-empty-note' }, '先解析一个视频,再来看它的分镜。'));
|
|
1619
1620
|
}
|
|
1621
|
+
const budget = data && data.budget ? data.budget : null;
|
|
1622
|
+
const sheets = data
|
|
1623
|
+
? (Array.isArray(data.sheets) && data.sheets.length ? data.sheets : [{ index: 0, cols: data.cols, rows: data.rows }])
|
|
1624
|
+
: [];
|
|
1625
|
+
const sheetCount = sheets.length || 1;
|
|
1626
|
+
const secondsPerFrame = budget ? Number(budget.secondsPerFrame) : 0;
|
|
1627
|
+
const budgetReason = budget && budget.reason ? String(budget.reason) : '';
|
|
1620
1628
|
const toolbar = h('div', { key: 'bar', className: 'dvv-sb-toolbar' }, [
|
|
1621
1629
|
data ? h('span', { key: 'cached', className: 'dvv-mini' },
|
|
1622
1630
|
storyboard.cached === false ? '刚刚生成' : `已缓存${data.generatedAt ? ` · ${fmtRelative(data.generatedAt)}` : ''}`) : null,
|
|
1623
1631
|
data && data.described ? h('span', { key: 'described', className: 'dvv-mini', 'data-ok': 'true' }, '含描述') : null,
|
|
1624
|
-
data && data.frames ? h('span', {
|
|
1632
|
+
data && data.frames ? h('span', {
|
|
1633
|
+
key: 'count', className: 'dvv-sb-meta',
|
|
1634
|
+
title: budgetReason || undefined,
|
|
1635
|
+
}, `${data.frames} 帧 · ${sheetCount} 张${secondsPerFrame > 0 ? ` · 每 ${secondsPerFrame}s 一帧` : ''}`) : null,
|
|
1625
1636
|
data && data.frames ? h('span', { key: 'sampling', className: 'dvv-sb-meta' },
|
|
1626
1637
|
data.frameSampling === 'interval' ? '按间隔抽帧' : '按镜头切点抽帧') : null,
|
|
1627
1638
|
h('span', { key: 'spacer', className: 'dvv-sb-spacer' }),
|
|
@@ -1634,7 +1645,7 @@ window.__ModuleLoader__.load({
|
|
|
1634
1645
|
]),
|
|
1635
1646
|
data ? h('a', {
|
|
1636
1647
|
key: 'big', className: 'dvv-btn dvv-link-btn',
|
|
1637
|
-
href: storyboardImageSrc(data, doc.docId), target: '_blank', rel: 'noreferrer',
|
|
1648
|
+
href: storyboardImageSrc(data, doc.docId, 0), target: '_blank', rel: 'noreferrer',
|
|
1638
1649
|
}, '打开大图') : null,
|
|
1639
1650
|
h('button', {
|
|
1640
1651
|
key: 'generate', type: 'button', className: 'dvv-btn', disabled: loading,
|
|
@@ -1658,40 +1669,52 @@ window.__ModuleLoader__.load({
|
|
|
1658
1669
|
body = h('div', { key: 'empty', className: 'dvv-empty-note' },
|
|
1659
1670
|
'这个视频还没有分镜。点「生成分镜」,把整段视频的画面拼成一张图。');
|
|
1660
1671
|
} else {
|
|
1661
|
-
const
|
|
1662
|
-
const
|
|
1663
|
-
|
|
1664
|
-
|
|
1665
|
-
|
|
1666
|
-
|
|
1667
|
-
|
|
1668
|
-
|
|
1669
|
-
|
|
1670
|
-
|
|
1671
|
-
|
|
1672
|
-
h('
|
|
1673
|
-
key: '
|
|
1674
|
-
|
|
1675
|
-
|
|
1676
|
-
}
|
|
1677
|
-
|
|
1678
|
-
|
|
1679
|
-
|
|
1680
|
-
|
|
1681
|
-
|
|
1682
|
-
|
|
1683
|
-
|
|
1684
|
-
|
|
1685
|
-
|
|
1686
|
-
|
|
1687
|
-
|
|
1688
|
-
|
|
1689
|
-
|
|
1690
|
-
|
|
1691
|
-
|
|
1692
|
-
|
|
1693
|
-
|
|
1694
|
-
|
|
1672
|
+
const allCells = Array.isArray(data.cells) ? data.cells : [];
|
|
1673
|
+
const sheetNodes = sheets.map((sheet, index) => {
|
|
1674
|
+
const sheetIndex = Number.isFinite(Number(sheet.index)) ? Math.max(0, Math.floor(Number(sheet.index))) : index;
|
|
1675
|
+
const own = Array.isArray(sheet.cells) && sheet.cells.length
|
|
1676
|
+
? sheet.cells
|
|
1677
|
+
: allCells.filter((cell) => Math.max(0, Math.floor(Number(cell.sheet) || 0)) === sheetIndex);
|
|
1678
|
+
const cols = Math.max(1, Number(sheet.cols) || Number(data.cols) || 1);
|
|
1679
|
+
const rows = Math.max(1, Number(sheet.rows) || Number(data.rows) || 1);
|
|
1680
|
+
const broken = Boolean(brokenSheets[sheetIndex]);
|
|
1681
|
+
return h('div', { key: `sheet-${sheetIndex}`, className: 'dvv-sb-sheet', 'data-sheet': String(sheetIndex) }, [
|
|
1682
|
+
broken
|
|
1683
|
+
? h('div', { key: 'broken', className: 'dvv-sb-broken' }, [
|
|
1684
|
+
h('div', { key: 'title', className: 'dvv-sb-broken-title' }, '分镜图加载失败'),
|
|
1685
|
+
h('button', { key: 'retry', type: 'button', className: 'dvv-btn', onClick: () => onGenerate() }, '重新生成'),
|
|
1686
|
+
])
|
|
1687
|
+
: h('div', { key: 'sb', className: 'dvv-sb' }, [
|
|
1688
|
+
h('img', {
|
|
1689
|
+
key: 'img', className: 'dvv-sb-img',
|
|
1690
|
+
src: storyboardImageSrc(data, doc.docId, sheetIndex),
|
|
1691
|
+
alt: `${doc.title || '视频'} 的分镜图(第 ${index + 1} 张)`, loading: 'lazy',
|
|
1692
|
+
onError: () => setBrokenSheets((prev) => ({ ...prev, [sheetIndex]: true })),
|
|
1693
|
+
}),
|
|
1694
|
+
...own.map((cell) => {
|
|
1695
|
+
const col = Math.max(0, Number(cell.col) || 0);
|
|
1696
|
+
const row = Math.max(0, Number(cell.row) || 0);
|
|
1697
|
+
const label = formatClock(cell.seconds);
|
|
1698
|
+
return h('button', {
|
|
1699
|
+
key: `cell-${sheetIndex}-${cell.index}-${row}-${col}`, type: 'button', className: 'dvv-sb-cell',
|
|
1700
|
+
style: {
|
|
1701
|
+
left: `${(col / cols) * 100}%`,
|
|
1702
|
+
top: `${(row / rows) * 100}%`,
|
|
1703
|
+
width: `${100 / cols}%`,
|
|
1704
|
+
height: `${100 / rows}%`,
|
|
1705
|
+
},
|
|
1706
|
+
title: cell.description || `跳到 ${label}`,
|
|
1707
|
+
'data-seconds': String(cell.seconds),
|
|
1708
|
+
'data-sheet': String(sheetIndex),
|
|
1709
|
+
onClick: () => onSeek(cell.seconds),
|
|
1710
|
+
}, h('span', { key: 'clock', className: 'dvv-sb-clock' }, label));
|
|
1711
|
+
}),
|
|
1712
|
+
]),
|
|
1713
|
+
h('div', { key: 'cap', className: 'dvv-sb-cap' }, `第 ${index + 1}/${sheetCount} 张 · ${cols}×${rows} 格`),
|
|
1714
|
+
]);
|
|
1715
|
+
});
|
|
1716
|
+
body = h('div', { key: 'sheets', className: 'dvv-sb-wrap' }, [
|
|
1717
|
+
...sheetNodes,
|
|
1695
1718
|
data.described ? h('div', { key: 'note', className: 'dvv-sb-note' },
|
|
1696
1719
|
'把鼠标放到格子上可以看到画面描述;点击格子会跳到对应画面。') : null,
|
|
1697
1720
|
].filter(Boolean));
|
|
@@ -1721,7 +1744,7 @@ window.__ModuleLoader__.load({
|
|
|
1721
1744
|
]);
|
|
1722
1745
|
}
|
|
1723
1746
|
|
|
1724
|
-
/* ---------------------------------------------------------- 历史列表 / 抽屉 */
|
|
1747
|
+
/* ---------------------------------------------------------- 历史列表 / 抽屉 */
|
|
1725
1748
|
|
|
1726
1749
|
/** 封面缩略图:没有 cover 或加载失败时用纯色占位。 */
|
|
1727
1750
|
function HistoryCover({ src, alt }) {
|
|
@@ -1807,7 +1830,7 @@ window.__ModuleLoader__.load({
|
|
|
1807
1830
|
.map((value) => String(value ?? '')).join(' ').toLowerCase();
|
|
1808
1831
|
return hay.includes(needle);
|
|
1809
1832
|
});
|
|
1810
|
-
return h('div', { className: 'dvv-overlay', onClick: onClose }, h('div', {
|
|
1833
|
+
return h('div', { className: 'dvv-overlay', 'data-quiet': 'true', onClick: onClose }, h('div', {
|
|
1811
1834
|
key: 'drawer', className: 'dvv-drawer', onClick: (event) => { if (event && typeof event.stopPropagation === 'function') event.stopPropagation(); },
|
|
1812
1835
|
}, [
|
|
1813
1836
|
h('div', { key: 'head', className: 'dvv-drawer-head' }, [
|
|
@@ -3612,7 +3635,8 @@ a.dvv-toolbar-title:hover{color:var(--dsw-alias-brand-primary)}
|
|
|
3612
3635
|
.dvv-textarea{flex:1;min-width:0;font:inherit;font-size:13px;line-height:21px;min-height:42px;max-height:140px;padding:8px 12px;border-radius:10px;border:1px solid var(--dsw-alias-border-l1,rgba(0,0,0,.14));background:var(--dsw-alias-bg-layer-1,transparent);color:inherit;outline:none;resize:vertical}
|
|
3613
3636
|
.dvv-textarea:focus{border-color:var(--dsw-alias-brand-primary)}
|
|
3614
3637
|
.dvv-overlay{position:absolute;inset:0;z-index:20;display:flex;justify-content:flex-end;background:var(--dsw-alias-bg-overlay,rgba(0,0,0,.32))}
|
|
3615
|
-
.dvv-
|
|
3638
|
+
.dvv-overlay[data-quiet="true"]{background:transparent}
|
|
3639
|
+
.dvv-drawer{display:flex;flex-direction:column;width:min(420px,92%);height:100%;background:var(--dsw-alias-bg-layer-1,Canvas);border-left:1px solid var(--dsw-alias-border-l1,rgba(0,0,0,.12));box-shadow:-10px 0 32px rgba(0,0,0,.18)}
|
|
3616
3640
|
.dvv-modal{display:flex;flex-direction:column;width:min(600px,94%);max-height:88%;margin:auto;background:var(--dsw-alias-bg-layer-1,Canvas);border:1px solid var(--dsw-alias-border-l1,rgba(0,0,0,.12));border-radius:14px;box-shadow:0 18px 48px rgba(0,0,0,.24);overflow:hidden}
|
|
3617
3641
|
.dvv-drawer-head{display:flex;align-items:center;gap:8px;padding:12px 16px;border-bottom:1px solid var(--dsw-alias-border-l1,rgba(0,0,0,.1));background:var(--dsw-specific-sidebar-fill,transparent)}
|
|
3618
3642
|
.dvv-drawer-title{flex:1;min-width:0;font-size:14px;line-height:21px;font-weight:600}
|
|
@@ -3737,7 +3761,9 @@ a.dvv-toolbar-title:hover{color:var(--dsw-alias-brand-primary)}
|
|
|
3737
3761
|
.dvv-sb-meta{font-size:11px;line-height:16px;color:var(--dsw-alias-label-secondary);font-variant-numeric:tabular-nums}
|
|
3738
3762
|
.dvv-sb-check{display:flex;align-items:center;gap:4px;font-size:11px;line-height:16px;color:var(--dsw-alias-label-secondary);cursor:pointer}
|
|
3739
3763
|
.dvv-sb-progress{display:flex;flex-direction:column;gap:8px;padding:12px 0}
|
|
3740
|
-
.dvv-sb-wrap{display:flex;flex-direction:column;gap:
|
|
3764
|
+
.dvv-sb-wrap{display:flex;flex-direction:column;gap:14px}
|
|
3765
|
+
.dvv-sb-sheet{display:flex;flex-direction:column;gap:4px;min-width:0}
|
|
3766
|
+
.dvv-sb-cap{font-size:11px;line-height:16px;color:var(--dsw-alias-label-secondary);font-variant-numeric:tabular-nums}
|
|
3741
3767
|
.dvv-sb{position:relative;width:100%;border-radius:12px;overflow:hidden;border:1px solid var(--dsw-alias-border-l1);background:var(--dsw-alias-bg-layer-2)}
|
|
3742
3768
|
.dvv-sb-img{display:block;width:100%;height:auto}
|
|
3743
3769
|
.dvv-sb-cell{position:absolute;appearance:none;margin:0;padding:0;border:0;background:transparent;cursor:pointer;outline:none}
|
package/lib/asr.js
CHANGED
|
@@ -47,7 +47,7 @@ export const ASR_DEFAULTS = {
|
|
|
47
47
|
// frameMode: auto(先 OCR,文字太少再用视觉模型)/ ocr(只用本机 OCR)/ vision(直接看画面)。
|
|
48
48
|
frameMode: 'auto',
|
|
49
49
|
visionModel: '',
|
|
50
|
-
visionMaxFrames:
|
|
50
|
+
visionMaxFrames: 96,
|
|
51
51
|
// 喂图方式:sheet = 多帧拼成一张分镜图(默认,省图省时)/ frames = 逐帧发。
|
|
52
52
|
visionPack: 'sheet',
|
|
53
53
|
sheetCols: 3,
|
package/lib/assets.js
CHANGED
|
@@ -68,7 +68,7 @@ export async function readVideoAsset(docId) {
|
|
|
68
68
|
if (!docId) return null;
|
|
69
69
|
const paths = assetPaths(docId);
|
|
70
70
|
const meta = await readJson(paths.meta);
|
|
71
|
-
if (!meta?.file) return null;
|
|
71
|
+
if (!meta?.file || /\.(json|tmp|part)$/i.test(meta.file)) return null;
|
|
72
72
|
try {
|
|
73
73
|
const info = await stat(meta.file);
|
|
74
74
|
if (!info.size) return null;
|
package/lib/budget.js
ADDED
|
@@ -0,0 +1,84 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* 抽帧预算:一帧多少秒、总共几帧、需要几张拼图。
|
|
3
|
+
*
|
|
4
|
+
* 参考 dsh-video-storyboard 的做法:按时长切窗口(5 分钟一段),窗口内按「每 20 秒一格」定格数,
|
|
5
|
+
* 格数吸附到 4/6/9/12/16;窗口数有上限,超出就把窗口拉长而不是砍掉尾巴。
|
|
6
|
+
* 这里把同样的思路压成一个纯函数,供三条画面管线共用:
|
|
7
|
+
* - 画面理解(frameTranscript / 视觉模型描述)
|
|
8
|
+
* - 分镜页签(storyboard 拼图)
|
|
9
|
+
* - 章节大纲配图(outline visuals,条数由大纲决定、不再二次抽帧)
|
|
10
|
+
*
|
|
11
|
+
* 与固定帧数相比的关键差别:
|
|
12
|
+
* 1. 帧数随时长增长(1 帧 / 20 秒),长视频不再永远只有 24 帧;
|
|
13
|
+
* 2. 剪辑密度参与计算:碎剪(≤8 秒一个镜头)提到 1 帧 / 12 秒,口播长镜头降到 1 帧 / 30 秒;
|
|
14
|
+
* 3. 拼图一张最多 24 格,超出的部分自动分成多张,而不是直接丢弃。
|
|
15
|
+
*/
|
|
16
|
+
|
|
17
|
+
/** 一张拼图最多几格(与 lib/sheet.js 的 SHEET_MAX_CELLS 保持一致)。 */
|
|
18
|
+
export const CELLS_PER_SHEET = 24;
|
|
19
|
+
|
|
20
|
+
export const BUDGET_DEFAULTS = Object.freeze({
|
|
21
|
+
secondsPerFrame: 20,
|
|
22
|
+
denseSecondsPerFrame: 12,
|
|
23
|
+
sparseSecondsPerFrame: 30,
|
|
24
|
+
minFrames: 6,
|
|
25
|
+
maxFrames: 96,
|
|
26
|
+
denseShotSeconds: 8,
|
|
27
|
+
sparseShotSeconds: 45,
|
|
28
|
+
});
|
|
29
|
+
|
|
30
|
+
function clamp(value, low, high) {
|
|
31
|
+
return Math.min(high, Math.max(low, value));
|
|
32
|
+
}
|
|
33
|
+
|
|
34
|
+
/**
|
|
35
|
+
* @param {{duration?: number, cuts?: number, ceiling?: number, mode?: 'vision'|'storyboard'}} [options]
|
|
36
|
+
* @returns {{frames:number, secondsPerFrame:number, sheets:number, cellsPerSheet:number, density:string, duration:number, ceiling:number, reason:string}}
|
|
37
|
+
*/
|
|
38
|
+
export function planFrameBudget({ duration = 0, cuts = 0, ceiling = 0, mode = 'vision' } = {}) {
|
|
39
|
+
const dur = Math.max(0, Number(duration) || 0);
|
|
40
|
+
const shots = Math.max(0, Number(cuts) || 0);
|
|
41
|
+
const cap = Math.round(clamp(Number(ceiling) || BUDGET_DEFAULTS.maxFrames, BUDGET_DEFAULTS.minFrames, 240));
|
|
42
|
+
const cellsPerSheet = CELLS_PER_SHEET;
|
|
43
|
+
|
|
44
|
+
let secondsPerFrame = BUDGET_DEFAULTS.secondsPerFrame;
|
|
45
|
+
let density = 'normal';
|
|
46
|
+
if (dur > 0 && shots > 0) {
|
|
47
|
+
const shotPerSecond = shots / dur;
|
|
48
|
+
if (shotPerSecond < 1 / BUDGET_DEFAULTS.sparseShotSeconds) {
|
|
49
|
+
secondsPerFrame = BUDGET_DEFAULTS.sparseSecondsPerFrame;
|
|
50
|
+
density = 'sparse';
|
|
51
|
+
} else if (shotPerSecond > 1 / BUDGET_DEFAULTS.denseShotSeconds) {
|
|
52
|
+
secondsPerFrame = BUDGET_DEFAULTS.denseSecondsPerFrame;
|
|
53
|
+
density = 'dense';
|
|
54
|
+
}
|
|
55
|
+
}
|
|
56
|
+
|
|
57
|
+
const byDuration = dur > 0 ? Math.ceil(dur / secondsPerFrame) : 0;
|
|
58
|
+
// 镜头本身也是信息载体:镜头比时长预算还多时,至少接近「一镜一帧」的一半。
|
|
59
|
+
const byShots = shots > 0 ? Math.ceil(shots * 0.5) : 0;
|
|
60
|
+
const wanted = Math.max(byDuration, byShots, BUDGET_DEFAULTS.minFrames);
|
|
61
|
+
const frames = clamp(wanted, BUDGET_DEFAULTS.minFrames, cap);
|
|
62
|
+
const sheets = Math.max(1, Math.ceil(frames / cellsPerSheet));
|
|
63
|
+
|
|
64
|
+
const reason = [
|
|
65
|
+
`${Math.round(dur)}s`,
|
|
66
|
+
shots > 0 ? `${shots} 个镜头` : '未扫切点',
|
|
67
|
+
`每 ${secondsPerFrame}s 一帧`,
|
|
68
|
+
density === 'dense' ? '剪辑密集' : density === 'sparse' ? '长镜头/口播' : '常规节奏',
|
|
69
|
+
`${frames} 帧 / ${sheets} 张图`,
|
|
70
|
+
].join(' · ');
|
|
71
|
+
|
|
72
|
+
return {
|
|
73
|
+
frames,
|
|
74
|
+
secondsPerFrame,
|
|
75
|
+
sheets,
|
|
76
|
+
cellsPerSheet,
|
|
77
|
+
density,
|
|
78
|
+
duration: Math.round(dur),
|
|
79
|
+
ceiling: cap,
|
|
80
|
+
reason,
|
|
81
|
+
};
|
|
82
|
+
}
|
|
83
|
+
|
|
84
|
+
export default planFrameBudget;
|
package/lib/frames.js
CHANGED
|
@@ -34,6 +34,14 @@ function subsample(frames, limit) {
|
|
|
34
34
|
return out;
|
|
35
35
|
}
|
|
36
36
|
|
|
37
|
+
/** 把列表按固定大小切成若干段(拼图一张最多装 sheet.js 的 SHEET_MAX_CELLS 格)。 */
|
|
38
|
+
function chunk(list, size) {
|
|
39
|
+
const width = Math.max(1, Math.round(Number(size) || 1));
|
|
40
|
+
const out = [];
|
|
41
|
+
for (let i = 0; i < list.length; i += width) out.push(list.slice(i, i + width));
|
|
42
|
+
return out;
|
|
43
|
+
}
|
|
44
|
+
|
|
37
45
|
/** 把逐帧文字/描述拼成有序、不重叠的段落。 */
|
|
38
46
|
function buildSegments(frames, texts, intervalSeconds) {
|
|
39
47
|
const span = Math.max(1, Math.round(Number(intervalSeconds) || 8));
|
|
@@ -141,11 +149,12 @@ export async function frameTranscript({
|
|
|
141
149
|
// 噪声文字,也照样让视觉模型描述画面——这类视频的信息几乎全在画面上。
|
|
142
150
|
const shouldDescribe = visionUsable && (visionCfg.frameMode === 'vision' || sparse || preferVision);
|
|
143
151
|
if (shouldDescribe) {
|
|
144
|
-
//
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
152
|
+
// 帧数预算已经由 captureFrames 按「时长 + 剪辑密度」算好(见 lib/budget.js)。
|
|
153
|
+
// 拼图模式一张最多装 SHEET_MAX_CELLS 格,超出的部分分成多张,而不是直接丢掉。
|
|
154
|
+
const sampled = subsample(frames, visionCfg.visionMaxFrames);
|
|
155
|
+
const sheetChunks = visionCfg.visionPack === 'sheet'
|
|
156
|
+
? chunk(sampled, SHEET_MAX_CELLS)
|
|
157
|
+
: [sampled];
|
|
149
158
|
onPhase?.(
|
|
150
159
|
`画面文字${ocrChars ? '不够多' : '没有识别到'},改为让视觉模型看画面(${sampled.length} 帧,图片会发给当前模型)…`,
|
|
151
160
|
);
|
|
@@ -158,23 +167,29 @@ export async function frameTranscript({
|
|
|
158
167
|
let describedFrames = [];
|
|
159
168
|
if (visionCfg.visionPack === 'sheet') {
|
|
160
169
|
try {
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
170
|
+
for (let i = 0; i < sheetChunks.length; i += 1) {
|
|
171
|
+
const part = sheetChunks[i];
|
|
172
|
+
const sheet = await buildContactSheet({
|
|
173
|
+
frames: part,
|
|
174
|
+
workdir,
|
|
175
|
+
ffmpeg: tools?.ffmpeg?.path || 'ffmpeg',
|
|
176
|
+
cols: visionCfg.sheetCols,
|
|
177
|
+
signal,
|
|
178
|
+
});
|
|
179
|
+
onPhase?.(
|
|
180
|
+
sheetChunks.length > 1
|
|
181
|
+
? `把第 ${i + 1}/${sheetChunks.length} 组 ${part.length} 帧拼成分镜图(${sheet.cols}×${sheet.rows}),交给视觉模型(图片会发给当前模型)…`
|
|
182
|
+
: `把 ${part.length} 帧拼成一张分镜图(${sheet.cols}×${sheet.rows}),交给视觉模型一次看完(图片会发给当前模型)…`,
|
|
183
|
+
);
|
|
184
|
+
const partCells = await describeSheet({
|
|
185
|
+
llm: ai.llm,
|
|
186
|
+
route,
|
|
187
|
+
attachments,
|
|
188
|
+
sheet,
|
|
189
|
+
signal,
|
|
190
|
+
});
|
|
191
|
+
describedFrames = describedFrames.concat(partCells);
|
|
192
|
+
}
|
|
178
193
|
} catch (error) {
|
|
179
194
|
if (signal?.aborted) throw error;
|
|
180
195
|
onPhase?.(`分镜拼图没成功(${error?.message ?? error}),改回逐帧交给模型…`);
|
package/lib/index.js
CHANGED
|
@@ -63,7 +63,7 @@ export const name = 'dsh-video-insight';
|
|
|
63
63
|
/** Route prefix the panel calls; same origin as the GUI, so no CORS preflight. */
|
|
64
64
|
export const API_PREFIX = '/dsh-video/api';
|
|
65
65
|
|
|
66
|
-
const VERSION = '0.8.
|
|
66
|
+
const VERSION = '0.8.5';
|
|
67
67
|
const CREDENTIALS_FILE = join(homedir(), '.dsh', 'video-credentials.json');
|
|
68
68
|
|
|
69
69
|
const DEFAULTS = {
|
|
@@ -112,7 +112,7 @@ function sanitizeAsr(raw, base) {
|
|
|
112
112
|
} else if (key === 'frameMode') {
|
|
113
113
|
out.frameMode = ['auto', 'ocr', 'vision'].includes(String(value)) ? String(value) : 'auto';
|
|
114
114
|
} else if (key === 'visionMaxFrames') {
|
|
115
|
-
out.visionMaxFrames = Math.min(Math.max(Math.round(Number(value) ||
|
|
115
|
+
out.visionMaxFrames = Math.min(Math.max(Math.round(Number(value) || 96), 8), 240);
|
|
116
116
|
} else if (key === 'visionPack') {
|
|
117
117
|
out.visionPack = String(value) === 'frames' ? 'frames' : 'sheet';
|
|
118
118
|
} else if (key === 'sheetCols') {
|
|
@@ -1118,7 +1118,8 @@ function createApiHandler(aiCtx, config) {
|
|
|
1118
1118
|
|
|
1119
1119
|
if (req.method === 'GET' && route === '/storyboard/image') {
|
|
1120
1120
|
const docId = String(url.searchParams.get('docId') ?? '').trim();
|
|
1121
|
-
const
|
|
1121
|
+
const sheetIndex = Math.max(0, Math.min(3, Math.round(Number(url.searchParams.get('sheet')) || 0)));
|
|
1122
|
+
const image = docId ? await readStoryboardImage(docId, sheetIndex) : null;
|
|
1122
1123
|
if (!image) {
|
|
1123
1124
|
json(res, 404, { ok: false, code: 'not-found', error: '这个视频还没有分镜图。' });
|
|
1124
1125
|
return;
|
package/lib/media.js
CHANGED
|
@@ -276,8 +276,11 @@ function ytdlpProgress(chunk, onProgress) {
|
|
|
276
276
|
}
|
|
277
277
|
|
|
278
278
|
async function pickDownloaded(workdir, prefix = 'audio.') {
|
|
279
|
-
|
|
280
|
-
const
|
|
279
|
+
// 同目录里还有 video.json / cuts.json 之类的元信息文件,前缀匹配会撞上它们;跳过非媒体文件。
|
|
280
|
+
const names = (await readdir(workdir))
|
|
281
|
+
.filter((item) => item.startsWith(prefix) && !/\.(json|tmp|part|ytdl|srt|vtt|description)$/i.test(item))
|
|
282
|
+
.sort();
|
|
283
|
+
const name = names[0];
|
|
281
284
|
return name ? join(workdir, name) : '';
|
|
282
285
|
}
|
|
283
286
|
|
package/lib/ocr.js
CHANGED
|
@@ -23,6 +23,7 @@ import {
|
|
|
23
23
|
planFrameTimes,
|
|
24
24
|
scanCuts,
|
|
25
25
|
} from './cuts.js';
|
|
26
|
+
import { planFrameBudget } from './budget.js';
|
|
26
27
|
import {
|
|
27
28
|
TOOLS_DIR,
|
|
28
29
|
runProcess,
|
|
@@ -319,7 +320,12 @@ async function cutFrames({
|
|
|
319
320
|
});
|
|
320
321
|
cutList = scanned.cuts;
|
|
321
322
|
}
|
|
322
|
-
const
|
|
323
|
+
const budget = planFrameBudget({
|
|
324
|
+
duration,
|
|
325
|
+
cuts: cutList.length,
|
|
326
|
+
ceiling: maxFrames > 0 ? maxFrames : undefined,
|
|
327
|
+
});
|
|
328
|
+
const planned = planFrameTimes({ duration, cuts: cutList, maxFrames: budget.frames, minGap });
|
|
323
329
|
const times = planned.map((item) => item.seconds);
|
|
324
330
|
if (!times.length) throw new VideoError('ocr-empty', '没有找到可用的画面时间点。');
|
|
325
331
|
onPhase?.(
|
|
@@ -335,7 +341,7 @@ async function cutFrames({
|
|
|
335
341
|
signal,
|
|
336
342
|
onProgress: (event) => onProgress?.({ percent: event.percent, stage: 'frames' }),
|
|
337
343
|
});
|
|
338
|
-
return { frames, intervalSeconds: medianGap(times), cuts: cutList.length };
|
|
344
|
+
return { frames, intervalSeconds: medianGap(times), cuts: cutList.length, budget };
|
|
339
345
|
}
|
|
340
346
|
|
|
341
347
|
/**
|
|
@@ -385,6 +391,7 @@ export async function captureFrames({
|
|
|
385
391
|
let frames = [];
|
|
386
392
|
let spanSeconds = intervalSeconds;
|
|
387
393
|
let cuts = 0;
|
|
394
|
+
let budget = null;
|
|
388
395
|
if (sampling === 'cuts') {
|
|
389
396
|
try {
|
|
390
397
|
const cutsAsset = await ensureCutsAsset({
|
|
@@ -414,6 +421,7 @@ export async function captureFrames({
|
|
|
414
421
|
frames = outcome.frames;
|
|
415
422
|
spanSeconds = outcome.intervalSeconds || intervalSeconds;
|
|
416
423
|
cuts = Number(outcome.cuts) || 0;
|
|
424
|
+
budget = outcome.budget ?? null;
|
|
417
425
|
} catch (error) {
|
|
418
426
|
if (error?.code === 'aborted' || signal?.aborted) throw error;
|
|
419
427
|
onPhase?.(`镜头切分没成功(${error?.message ?? error}),改回每 ${intervalSeconds} 秒抽一帧…`);
|
|
@@ -434,7 +442,15 @@ export async function captureFrames({
|
|
|
434
442
|
cuts = 0;
|
|
435
443
|
}
|
|
436
444
|
if (!frames.length) throw new VideoError('ocr-empty', '没有抽到任何画面帧(视频可能无法解码)。');
|
|
437
|
-
|
|
445
|
+
if (!budget) {
|
|
446
|
+
budget = planFrameBudget({
|
|
447
|
+
duration: Number(doc?.duration) || 0,
|
|
448
|
+
cuts,
|
|
449
|
+
ceiling: maxFrames,
|
|
450
|
+
});
|
|
451
|
+
onPhase?.(`按 ${budget.reason}抽帧…`);
|
|
452
|
+
}
|
|
453
|
+
return { video, frames, intervalSeconds: spanSeconds, cuts, budget };
|
|
438
454
|
}
|
|
439
455
|
|
|
440
456
|
/**
|
package/lib/storyboard.js
CHANGED
|
@@ -1,9 +1,12 @@
|
|
|
1
1
|
/**
|
|
2
|
-
* storyboard.js —
|
|
2
|
+
* storyboard.js — 「分镜」页签的后端:把视频切成镜头、按预算取帧、拼成分镜图,
|
|
3
3
|
* 附每个格子的秒数(客户端点格子即可 seek)。可选让视觉模型给每格补一句描述。
|
|
4
4
|
*
|
|
5
|
-
* 与「画面兜底」共用同一套抽帧(captureFrames:按镜头切点抽 + 480p
|
|
5
|
+
* 与「画面兜底」共用同一套抽帧(captureFrames:按镜头切点抽 + 480p 下载 + 预算),
|
|
6
6
|
* 区别只是这里把结果当「产物」持久化,而不是只在没字幕时临时用一下。
|
|
7
|
+
*
|
|
8
|
+
* 帧数不再固定:由 lib/budget.js 按「时长 + 剪辑密度」计算(默认约每 20 秒一帧),
|
|
9
|
+
* 一张拼图最多 24 格,超过就自动分成多张(最多 2 张 / 48 格),长视频不会只留 24 张代表帧。
|
|
7
10
|
*/
|
|
8
11
|
|
|
9
12
|
import { copyFile, mkdir, readFile, rename, unlink, writeFile } from 'node:fs/promises';
|
|
@@ -13,13 +16,14 @@ import { join } from 'node:path';
|
|
|
13
16
|
import { createWorkdir, cleanupWorkdir, detectToolchain } from './media.js';
|
|
14
17
|
import { captureFrames } from './ocr.js';
|
|
15
18
|
import { asrSettings } from './asr.js';
|
|
16
|
-
import { buildContactSheet } from './sheet.js';
|
|
19
|
+
import { buildContactSheet, SHEET_MAX_CELLS } from './sheet.js';
|
|
17
20
|
import { describeSheet, resolveVisionRoute, visionStatus } from './vision.js';
|
|
18
21
|
import { VideoError } from './util.js';
|
|
19
22
|
|
|
20
23
|
export const STORYBOARD_DIR = join(homedir(), '.dsh', 'video-storyboards');
|
|
21
|
-
export const STORYBOARD_DEFAULT_FRAMES =
|
|
22
|
-
export const STORYBOARD_MAX_FRAMES =
|
|
24
|
+
export const STORYBOARD_DEFAULT_FRAMES = 0; // 0 = 按预算自动
|
|
25
|
+
export const STORYBOARD_MAX_FRAMES = SHEET_MAX_CELLS;
|
|
26
|
+
export const STORYBOARD_MAX_CELLS = SHEET_MAX_CELLS * 2;
|
|
23
27
|
|
|
24
28
|
function safeId(docId) {
|
|
25
29
|
return String(docId ?? '')
|
|
@@ -32,7 +36,8 @@ export function storyboardPaths(docId) {
|
|
|
32
36
|
return {
|
|
33
37
|
dir: STORYBOARD_DIR,
|
|
34
38
|
meta: join(STORYBOARD_DIR, `${base}.json`),
|
|
35
|
-
|
|
39
|
+
legacyImage: join(STORYBOARD_DIR, `${base}.jpg`),
|
|
40
|
+
image: (index = 0) => join(STORYBOARD_DIR, `${base}-${Number(index) + 1}.jpg`),
|
|
36
41
|
};
|
|
37
42
|
}
|
|
38
43
|
|
|
@@ -40,6 +45,7 @@ function sanitizeCells(cells) {
|
|
|
40
45
|
return (Array.isArray(cells) ? cells : [])
|
|
41
46
|
.map((cell) => ({
|
|
42
47
|
index: Number(cell?.index) || 0,
|
|
48
|
+
sheet: Number(cell?.sheet) || 0,
|
|
43
49
|
row: Number(cell?.row) || 0,
|
|
44
50
|
col: Number(cell?.col) || 0,
|
|
45
51
|
seconds: Math.max(0, Math.round(Number(cell?.seconds) * 100) / 100),
|
|
@@ -48,18 +54,34 @@ function sanitizeCells(cells) {
|
|
|
48
54
|
.filter((cell) => Number.isFinite(cell.seconds));
|
|
49
55
|
}
|
|
50
56
|
|
|
51
|
-
/** 读取已生成的分镜(没有/损坏返回 null
|
|
57
|
+
/** 读取已生成的分镜(没有/损坏返回 null)。旧版单图数据会自动包成一张 sheets。 */
|
|
52
58
|
export async function readStoryboard(docId) {
|
|
53
59
|
if (!docId) return null;
|
|
54
60
|
try {
|
|
55
|
-
const
|
|
61
|
+
const paths = storyboardPaths(docId);
|
|
62
|
+
const raw = await readFile(paths.meta, 'utf8');
|
|
56
63
|
const data = JSON.parse(raw);
|
|
57
64
|
if (!data || !Array.isArray(data.cells) || !data.cells.length) return null;
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
65
|
+
const cells = sanitizeCells(data.cells);
|
|
66
|
+
const legacy = !Array.isArray(data.sheets);
|
|
67
|
+
const sheets = legacy
|
|
68
|
+
? [{
|
|
69
|
+
index: 0,
|
|
70
|
+
file: paths.legacyImage,
|
|
71
|
+
cols: Number(data.cols) || 3,
|
|
72
|
+
rows: Number(data.rows) || 1,
|
|
73
|
+
cellWidth: Number(data.cellWidth) || 512,
|
|
74
|
+
cellHeight: Number(data.cellHeight) || 288,
|
|
75
|
+
}]
|
|
76
|
+
: data.sheets.map((sheet, index) => ({
|
|
77
|
+
index,
|
|
78
|
+
file: paths.image(index),
|
|
79
|
+
cols: Number(sheet?.cols) || 3,
|
|
80
|
+
rows: Number(sheet?.rows) || 1,
|
|
81
|
+
cellWidth: Number(sheet?.cellWidth) || 512,
|
|
82
|
+
cellHeight: Number(sheet?.cellHeight) || 288,
|
|
83
|
+
}));
|
|
84
|
+
return { ...data, sheets, cells, imagePath: sheets[0]?.file ?? paths.legacyImage };
|
|
63
85
|
} catch {
|
|
64
86
|
return null;
|
|
65
87
|
}
|
|
@@ -73,6 +95,13 @@ async function writeStoryboard(docId, data) {
|
|
|
73
95
|
await rename(tmp, paths.meta);
|
|
74
96
|
}
|
|
75
97
|
|
|
98
|
+
function chunk(list, size) {
|
|
99
|
+
const width = Math.max(1, Math.round(Number(size) || 1));
|
|
100
|
+
const out = [];
|
|
101
|
+
for (let i = 0; i < list.length; i += width) out.push(list.slice(i, i + width));
|
|
102
|
+
return out;
|
|
103
|
+
}
|
|
104
|
+
|
|
76
105
|
/**
|
|
77
106
|
* 生成(或重新生成)一个视频的分镜图。
|
|
78
107
|
*
|
|
@@ -80,7 +109,7 @@ async function writeStoryboard(docId, data) {
|
|
|
80
109
|
* @param {object} options.doc 已解析的 VideoDoc(需要 url/duration)
|
|
81
110
|
* @param {object} options.settings 插件设置
|
|
82
111
|
* @param {object} options.aiCtx `{llm, attachments}`,仅在 describe=true 时用到
|
|
83
|
-
* @param {number} [options.frames] 最多多少格(
|
|
112
|
+
* @param {number} [options.frames] 最多多少格(0 = 自动预算,默认自动;上限 24 格/张、2 张)
|
|
84
113
|
* @param {number} [options.cols] 列数(2–6,默认 3)
|
|
85
114
|
* @param {boolean} [options.describe] 是否让视觉模型给每格补一句描述
|
|
86
115
|
* @param {AbortSignal} [options.signal]
|
|
@@ -91,7 +120,7 @@ export async function generateStoryboard({
|
|
|
91
120
|
doc,
|
|
92
121
|
settings = {},
|
|
93
122
|
aiCtx = null,
|
|
94
|
-
frames =
|
|
123
|
+
frames = 0,
|
|
95
124
|
cols = 3,
|
|
96
125
|
describe = false,
|
|
97
126
|
signal,
|
|
@@ -103,10 +132,13 @@ export async function generateStoryboard({
|
|
|
103
132
|
const tools = await detectToolchain(asr, { fresh: false });
|
|
104
133
|
const workdir = await createWorkdir();
|
|
105
134
|
const started = Date.now();
|
|
106
|
-
const
|
|
135
|
+
const requested = Math.round(Number(frames) || 0);
|
|
136
|
+
const ceiling = requested > 0
|
|
137
|
+
? Math.max(4, Math.min(STORYBOARD_MAX_CELLS, requested))
|
|
138
|
+
: STORYBOARD_MAX_CELLS;
|
|
107
139
|
const columns = Math.max(2, Math.min(6, Math.round(Number(cols) || 3)));
|
|
108
140
|
let cuts = 0;
|
|
109
|
-
|
|
141
|
+
const paths = storyboardPaths(doc.docId);
|
|
110
142
|
try {
|
|
111
143
|
const captured = await captureFrames({
|
|
112
144
|
doc,
|
|
@@ -117,55 +149,80 @@ export async function generateStoryboard({
|
|
|
117
149
|
onPhase,
|
|
118
150
|
onProgress,
|
|
119
151
|
signal,
|
|
120
|
-
maxFrames:
|
|
152
|
+
maxFrames: ceiling,
|
|
121
153
|
});
|
|
122
154
|
cuts = Number(captured.cuts) || 0;
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
signal,
|
|
130
|
-
});
|
|
155
|
+
const parts = chunk(captured.frames, SHEET_MAX_CELLS);
|
|
156
|
+
onPhase?.(
|
|
157
|
+
parts.length > 1
|
|
158
|
+
? `正在把 ${captured.frames.length} 帧拼成 ${parts.length} 张分镜图…(${captured.budget?.reason ?? ''})`
|
|
159
|
+
: `正在把 ${captured.frames.length} 帧拼成一张分镜图…(${captured.budget?.reason ?? ''})`,
|
|
160
|
+
);
|
|
131
161
|
|
|
132
|
-
const cells =
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
162
|
+
const cells = [];
|
|
163
|
+
const sheets = [];
|
|
164
|
+
for (let index = 0; index < parts.length; index += 1) {
|
|
165
|
+
const part = parts[index];
|
|
166
|
+
// eslint-disable-next-line no-await-in-loop
|
|
167
|
+
const sheet = await buildContactSheet({
|
|
168
|
+
frames: part,
|
|
169
|
+
workdir,
|
|
170
|
+
ffmpeg: tools.ffmpeg.path,
|
|
171
|
+
cols: columns,
|
|
172
|
+
signal,
|
|
173
|
+
outPath: join(workdir, `sheet-${index + 1}.jpg`),
|
|
174
|
+
});
|
|
175
|
+
const baseIndex = cells.length;
|
|
176
|
+
for (const cell of sheet.layout) {
|
|
177
|
+
const frame = part[cell.index] ?? part[0];
|
|
178
|
+
cells.push({
|
|
179
|
+
index: baseIndex + cell.index,
|
|
180
|
+
sheet: index,
|
|
181
|
+
row: cell.row,
|
|
182
|
+
col: cell.col,
|
|
183
|
+
seconds: Math.round(Number(frame?.seconds ?? cell.seconds ?? 0) * 100) / 100,
|
|
184
|
+
});
|
|
185
|
+
}
|
|
186
|
+
sheets.push({ sheet, file: paths.image(index), part });
|
|
187
|
+
}
|
|
141
188
|
|
|
142
189
|
// 可选:视觉模型给每格补一句描述(把已经拼好的图再发一次,不再抽帧)
|
|
143
190
|
if (describe) {
|
|
144
191
|
const vision = await visionStatus(aiCtx, asr, { fresh: true });
|
|
145
192
|
if (vision?.available) {
|
|
146
193
|
const route = await resolveVisionRoute(aiCtx, asr);
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
194
|
+
for (let index = 0; index < sheets.length; index += 1) {
|
|
195
|
+
onPhase?.(
|
|
196
|
+
sheets.length > 1
|
|
197
|
+
? `正在让视觉模型描述第 ${index + 1}/${sheets.length} 张分镜…`
|
|
198
|
+
: '正在让视觉模型给每一格写一句描述…',
|
|
199
|
+
);
|
|
200
|
+
// eslint-disable-next-line no-await-in-loop
|
|
201
|
+
const described = await describeSheet({
|
|
202
|
+
llm: aiCtx.llm,
|
|
203
|
+
route,
|
|
204
|
+
attachments: aiCtx.attachments,
|
|
205
|
+
sheet: sheets[index].sheet,
|
|
206
|
+
signal,
|
|
207
|
+
maxTokens: 2400,
|
|
208
|
+
});
|
|
209
|
+
const bySeconds = new Map((described ?? []).map((item) => [Math.round(Number(item.seconds) * 100) / 100, String(item.description ?? '')]));
|
|
210
|
+
for (const cell of cells) {
|
|
211
|
+
if (cell.sheet !== index) continue;
|
|
212
|
+
const text = bySeconds.get(cell.seconds);
|
|
213
|
+
if (text) cell.description = text.slice(0, 200);
|
|
214
|
+
}
|
|
160
215
|
}
|
|
161
216
|
} else {
|
|
162
217
|
onPhase?.('没有可用的视觉模型,分镜只保留画面和时间点。');
|
|
163
218
|
}
|
|
164
219
|
}
|
|
165
220
|
|
|
166
|
-
const paths = storyboardPaths(doc.docId);
|
|
167
221
|
await mkdir(paths.dir, { recursive: true });
|
|
168
|
-
|
|
222
|
+
for (const entry of sheets) {
|
|
223
|
+
// eslint-disable-next-line no-await-in-loop
|
|
224
|
+
await copyFile(entry.sheet.file, entry.file);
|
|
225
|
+
}
|
|
169
226
|
const data = {
|
|
170
227
|
docId: doc.docId,
|
|
171
228
|
url: doc.url,
|
|
@@ -177,34 +234,53 @@ export async function generateStoryboard({
|
|
|
177
234
|
frameSampling: String(asr.frameSampling ?? 'cuts'),
|
|
178
235
|
cuts,
|
|
179
236
|
frames: cells.length,
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
237
|
+
sheets: sheets.map((entry) => ({
|
|
238
|
+
cols: entry.sheet.cols,
|
|
239
|
+
rows: entry.sheet.rows,
|
|
240
|
+
cellWidth: entry.sheet.cellWidth,
|
|
241
|
+
cellHeight: entry.sheet.cellHeight,
|
|
242
|
+
cells: entry.part.length,
|
|
243
|
+
})),
|
|
244
|
+
budget: captured.budget ?? null,
|
|
245
|
+
// 兼容旧客户端:第一张图的尺寸也放在顶层
|
|
246
|
+
cols: sheets[0]?.sheet.cols ?? columns,
|
|
247
|
+
rows: sheets[0]?.sheet.rows ?? 1,
|
|
248
|
+
cellWidth: sheets[0]?.sheet.cellWidth ?? 512,
|
|
249
|
+
cellHeight: sheets[0]?.sheet.cellHeight ?? 288,
|
|
184
250
|
described: cells.some((cell) => cell.description),
|
|
185
251
|
cells,
|
|
186
252
|
};
|
|
187
253
|
await writeStoryboard(doc.docId, data);
|
|
188
|
-
return { ...data, imagePath: paths.image, cached: false };
|
|
254
|
+
return { ...data, imagePath: paths.image(0), cached: false };
|
|
189
255
|
} finally {
|
|
190
256
|
await cleanupWorkdir(workdir);
|
|
191
257
|
}
|
|
192
258
|
}
|
|
193
259
|
|
|
194
|
-
/**
|
|
195
|
-
export async function readStoryboardImage(docId) {
|
|
260
|
+
/** 某一张分镜图(供 HTTP 路由直接读)。index 从 0 开始;旧版单图数据回落 `<base>.jpg`。 */
|
|
261
|
+
export async function readStoryboardImage(docId, index = 0) {
|
|
196
262
|
const paths = storyboardPaths(docId);
|
|
263
|
+
const position = Number(index) || 0;
|
|
197
264
|
try {
|
|
198
|
-
return await readFile(paths.image);
|
|
265
|
+
return await readFile(paths.image(position));
|
|
199
266
|
} catch {
|
|
267
|
+
if (position === 0) {
|
|
268
|
+
try {
|
|
269
|
+
return await readFile(paths.legacyImage);
|
|
270
|
+
} catch {
|
|
271
|
+
return null;
|
|
272
|
+
}
|
|
273
|
+
}
|
|
200
274
|
return null;
|
|
201
275
|
}
|
|
202
276
|
}
|
|
203
277
|
|
|
204
|
-
/**
|
|
278
|
+
/** 删除某个视频的分镜缓存(含多张图)。 */
|
|
205
279
|
export async function clearStoryboard(docId) {
|
|
206
280
|
const paths = storyboardPaths(docId);
|
|
207
|
-
|
|
281
|
+
const files = [paths.meta, paths.legacyImage];
|
|
282
|
+
for (let index = 0; index < 4; index += 1) files.push(paths.image(index));
|
|
283
|
+
for (const file of files) {
|
|
208
284
|
// eslint-disable-next-line no-await-in-loop
|
|
209
285
|
await unlink(file).catch(() => {});
|
|
210
286
|
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "dsh-video-insight",
|
|
3
|
-
"version": "0.8.
|
|
3
|
+
"version": "0.8.5",
|
|
4
4
|
"description": "在 DeepSeek Harness 里快速分析视频:左侧栏「视频解析」面板,粘贴 YouTube / 哔哩哔哩链接,自动抓字幕并生成总结、章节、要点、原文与追问;无字幕时可用本机 Whisper / SenseVoice 转写兜底(音频不出本机),纯无声视频还能用画面文字识别(OCR)或视觉模型描述画面;右栏有 AI 思维导图;追问是基于转写稿的 RAG 问答(回答带「依据」时间戳);分析结果按视频缓存,历史记录可回看。",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"author": "Jerry (Azerz)",
|