dsh-daily-brief 2.0.3 → 2.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +16 -3
- package/lib/engine/ai.js +24 -5
- package/lib/engine/collect/bilibili-comments.js +229 -0
- package/lib/engine/collect/bilibili.js +126 -2
- package/lib/engine/collect/index.js +39 -2
- package/lib/engine/collect/x.js +331 -86
- package/lib/engine/collect/youtube-comments.js +313 -0
- package/lib/engine/collect/youtube.js +87 -2
- package/lib/engine/fetch.js +23 -2
- package/lib/engine/render/email.js +42 -0
- package/lib/engine/render/html.js +49 -2
- package/lib/engine/render/model.js +16 -0
- package/lib/engine/runner.js +11 -2
- package/lib/engine/text.js +17 -0
- package/package.json +5 -1
package/README.md
CHANGED
|
@@ -27,9 +27,10 @@
|
|
|
27
27
|
## 环境要求
|
|
28
28
|
|
|
29
29
|
- Windows 10/11(弹窗与计划任务依赖 Windows;其他平台只能生成 HTML)。
|
|
30
|
+
- DeepSeek Harness 核心 **≥ 0.2.1-rc.3**(已声明 `engines.dsh`,在 0.2.2-rc.1 上验证)。
|
|
30
31
|
- 已安装 DeepSeek Harness,并能用 `plugin_manager` 安装本地插件目录。
|
|
31
32
|
- 本机有 Microsoft Edge 或 Google Chrome(用于独立窗口)。
|
|
32
|
-
- Node.js
|
|
33
|
+
- Node.js 20+(随 DSH 提供;跑测试也用它)。
|
|
33
34
|
|
|
34
35
|
## 安装
|
|
35
36
|
|
|
@@ -83,8 +84,8 @@
|
|
|
83
84
|
| --- | --- |
|
|
84
85
|
| `you@gmail.com`、`me@school.edu` | 邮箱(IMAP)。Gmail 用应用专用密码;学校邮箱识别不出服务器时会多出一个 IMAP 服务器输入框 |
|
|
85
86
|
| `me@outlook.com`、`me@hotmail.com`、`me@live.com` | 不再要求 Azure:给出一张「转发到 Gmail / QQ」的三步指南;按指南把 Outlook 邮件转发过去,再添加接收邮箱即可。Outlook 日历则直接粘贴发布的 ICS 链接 |
|
|
86
|
-
| `https://youtube.com/@频道`、`https://youtu.be/视频ID`、播放列表 | YouTube 更新(频道 RSS /
|
|
87
|
-
| `https://space.bilibili.com/UP主ID`、`https://bilibili.com/video/BV…`、排行榜 | 哔哩哔哩更新(UP 主投稿 / 单个视频 /
|
|
87
|
+
| `https://youtube.com/@频道`、`https://youtu.be/视频ID`、播放列表 | YouTube 更新(频道 RSS / 视频页),不需要登录;同时读取每个视频的评论数,评论最多的那条会单独生成「今日热议」卡 |
|
|
88
|
+
| `https://space.bilibili.com/UP主ID`、`https://bilibili.com/video/BV…`、排行榜 | 哔哩哔哩更新(UP 主投稿 / 单个视频 / 排行榜),公开接口,无需登录;同样读每条视频的评论数,评论最多的那条生成「今日热议」卡 |
|
|
88
89
|
| `https://…/feed.xml`、`.ics`、任意网页 | 订阅源 / 日历 / 网页。普通网页会自动判断是「单篇文章」还是「列表页」:文章提取标题、正文摘要与发布时间;列表页提取最近条目,抓不到时提示设置 CSS 选择器 |
|
|
89
90
|
| `https://www.managebac.com/...`、`https://…managebac.cn/...` | ManageBac 通知(.com / .cn 及学校子域;需要登录 Cookie,或日历订阅链接) |
|
|
90
91
|
| `https://…feishu.cn/docx|wiki|calendar/...` | 飞书文档 / 知识库 / 日历;前端渲染抓不到时建议用 ICS 订阅 |
|
|
@@ -158,6 +159,12 @@ Outlook 日历 →「设置 → 共享日历 → 发布日历」→ 复制 ICS
|
|
|
158
159
|
|
|
159
160
|
- 默认「自动」:仅当本机 DSH 有可用登录时,用 headless 方式让模型写标题、一句引言和两段要点;失败自动回退本地文案。
|
|
160
161
|
- 只改文案,不改抓取结果、不增删条目;提示词里明确禁止编造与出现网址。
|
|
162
|
+
- 提示词里会带上视频的评论数与「今日热议」的三条热评,因此要点能写出「46 条评论都在吐槽片头」这类内容;评论数只由接口给出,模型不得自行推测。
|
|
163
|
+
- 以下任一条件不满足就会跳过,并在简报顶部写明原因(`logs\runner.log` 里是同一句):
|
|
164
|
+
1. `%(LOCALAPPDATA)%\Programs\DSH NEXT\DSH NEXT.exe` 存在,且 `resources\app.asar` 或解包后的 `resources\app` 里有 `lib\desktop-cli.js`;
|
|
165
|
+
2. headless profile 已登录(`…\daily-brief\headless-auth.yml`,跟随你桌面端当前账号);
|
|
166
|
+
3. 模型在 `ai.timeoutSec`(默认 120 秒)内返回可解析的 JSON。
|
|
167
|
+
- 关掉:`ai.polish` 设为 `off`,或面板里选择「不润色」。
|
|
161
168
|
|
|
162
169
|
## 数据与隐私
|
|
163
170
|
|
|
@@ -171,15 +178,21 @@ Outlook 日历 →「设置 → 共享日历 → 发布日历」→ 复制 ICS
|
|
|
171
178
|
| --- | --- |
|
|
172
179
|
| 面板提示「重启一次 DSH」 | 插件主进程仍是旧代码:完全退出并重开 DSH,面板会恢复可编辑 |
|
|
173
180
|
| X 新闻经常很少 | syndication 对同一出口 IP 限流很紧(约每 15 分钟 1 次),插件会自动用官方博客补位;可减少账号数量,或改用 RSS 订阅 |
|
|
181
|
+
| 简报有条目,但没有标题和要点 | AI 润色被跳过,简报顶部会写「AI 润色未生效:原因」。常见原因:headless 未登录、DSH 升级后目录结构变化(v2.1 起 `app.asar` 与解包后的 `app\` 都认)、模型超过 `ai.timeoutSec`、或 `ai.polish` 被设成 `off` |
|
|
182
|
+
| 某个来源长期「获取超时(35 秒)」 | 慢来源(X / YouTube / B 站 / Steam / 浏览器 / IMAP / Graph)每个 35 秒、普通来源 22 秒,超时即记 error 且当天 0 条。X 最容易吃满:syndication 限流按出口 IP 共享,账号多了第一轮就耗尽,之后还要等官方博客兜底 |
|
|
183
|
+
| B 站报 -352 / -412 | UP 主空间接口对冷请求很敏感。现在会自动改走「动态」接口(另一套风控,通常还能读到投稿);两条都被挡住时说明本机 IP 被临时限制,等 5–30 分钟再试,或改用单条视频 / 排行榜(这两个接口更宽松)。整轮里命中 -412 后评论数请求会直接短路,避免把封禁时间越拖越长 |
|
|
174
184
|
| Steam 显示「暂时连不上」 | 官方新闻接口偶发超时,会自动显示上次结果;稍后重跑即可 |
|
|
175
185
|
| 服务器一直离线 | 确认地址与端口、服务器是否开机;内网地址需允许私有网络(已默认允许) |
|
|
176
186
|
| 桌面窗口没出现 | 检查是否安装 Edge/Chrome;查看 `logs\runner.log` 的投递记录 |
|
|
177
187
|
| 飞书/ManageBac 抓不到条目 | 多为前端渲染或需要登录:改用日历 ICS 订阅,或在「添加来源」里填入 Cookie |
|
|
178
188
|
| Outlook 邮箱填密码后报错 / 连不上 | Microsoft 已停用 IMAP 密码登录,但不再需要 Azure:按上面「Outlook 怎么办」把邮件转发到 Gmail / QQ,再添加接收邮箱;或直接用 Outlook 日历的 ICS 链接 |
|
|
189
|
+
| 插件市场里安装失败,任务日志是 `pnpm` 且提到 `@napi-rs/canvas-*` / `@img/sharp-*` 或 `EPERM`/`EBUSY` | 失败发生在市场对整个 DSH profile 做 pnpm 重解析时(删宿主自带的原生可选包,被运行中的 DSH 占用),与插件包无关。完全退出 DSH(含托盘)后重装;仍失败就在终端执行 `dsh plugin add dsh-daily-brief` 取真正的错误行 |
|
|
179
190
|
|
|
180
191
|
## 配置参考(`config.json`)
|
|
181
192
|
|
|
182
193
|
- `sources[]`:`type`(x/steam/minecraft/http/rss/atom/json/ics/page/youtube/bilibili/imap/graph/browser/managebac)、`section`、`label`、`enabled`、`options`。
|
|
194
|
+
- YouTube 的 `options`:`limit`、`sinceHours`、`commentScan`(默认 6,最多 10,看前几条视频的评论数;`comments:false` 可整体关掉评论读取)、`commentsPerVideo`(默认 4,热议卡里带几条热评)。
|
|
195
|
+
- Bilibili 的 `options`:`limit`、`sinceHours`、`mid` / `bvid`(覆盖链接里的 ID)、`commentScan`、`commentsPerVideo`、`comments:false`,含义同上。
|
|
183
196
|
- `graph.clientId`:仅 To Do 等高级 Microsoft Graph 来源需要的 Azure 应用(客户端)ID;普通 Outlook 邮件/日历不用填。
|
|
184
197
|
- `persona`:`preset`(whale/assistant/butler/officer/custom 或自建卡片的 id)、`customName`、`customTagline`、`customAccent`、`customPrompt`,以及 `cards[]`(每张 `{ id, name, tagline, accent, prompt }`,最多 12 张)。
|
|
185
198
|
- `schedule`:`time`(HH:MM)、`enabled`、`wakeToRun`、`prepareLeadMin`(提前准备分钟数,默认 10)。
|
package/lib/engine/ai.js
CHANGED
|
@@ -32,10 +32,18 @@ export function resolveDesktopRuntime(env = process.env) {
|
|
|
32
32
|
const resources = process.versions.electron && process.resourcesPath
|
|
33
33
|
? process.resourcesPath
|
|
34
34
|
: path.join(path.dirname(exe), 'resources');
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
35
|
+
// Packaged builds ship app.asar; a patched/unpacked install keeps a plain
|
|
36
|
+
// `app` folder instead. The CLI lives at the same relative path in both.
|
|
37
|
+
const roots = [path.join(resources, 'app.asar'), path.join(resources, 'app')];
|
|
38
|
+
const root = roots.find((candidate) => fs.existsSync(path.join(candidate, 'lib', 'desktop-cli.js'))) || roots[0];
|
|
39
|
+
const cli = path.join(root, 'lib', 'desktop-cli.js');
|
|
40
|
+
return {
|
|
41
|
+
exe,
|
|
42
|
+
cli,
|
|
43
|
+
resources,
|
|
44
|
+
asar: path.join(resources, 'app.asar'),
|
|
45
|
+
available: fs.existsSync(exe) && fs.existsSync(cli),
|
|
46
|
+
};
|
|
39
47
|
}
|
|
40
48
|
|
|
41
49
|
function buildPrompt(briefing, config) {
|
|
@@ -57,9 +65,20 @@ function buildPrompt(briefing, config) {
|
|
|
57
65
|
lines.push(`\n【${section.zh}】共 ${section.items.length} 条`);
|
|
58
66
|
for (const item of section.items) {
|
|
59
67
|
const detail = String(item.detail || '').replace(/\s+/g, ' ').trim().slice(0, 110);
|
|
60
|
-
|
|
68
|
+
const hits = Number(item.commentCount) > 0 ? `(${item.commentCount} 条评论)` : '';
|
|
69
|
+
lines.push(`- ${item.title}${item.meta ? `(${item.meta})` : ''}${hits}${detail ? `:${detail}` : ''}`);
|
|
61
70
|
}
|
|
62
71
|
}
|
|
72
|
+
const featured = briefing.featured;
|
|
73
|
+
if (featured && featured.title) {
|
|
74
|
+
lines.push(`\n【今日热议】${featured.title}(${Number(featured.commentCount) || 0} 条评论)`);
|
|
75
|
+
for (const comment of (Array.isArray(featured.comments) ? featured.comments : []).slice(0, 3)) {
|
|
76
|
+
if (!comment?.text) continue;
|
|
77
|
+
const likes = Number(comment.likes) > 0 ? `,${comment.likes} 赞` : '';
|
|
78
|
+
lines.push(`- 热评${comment.author ? ` @${comment.author}` : ''}${likes}:${String(comment.text).replace(/\s+/g, ' ').slice(0, 90)}`);
|
|
79
|
+
}
|
|
80
|
+
lines.push('这条热议内容必须出现在对应板块(通常是游戏)的一条要点里:点出视频讲了什么、评论区在讨论什么,可以引用一句热评的意思,但不要照抄原句,也不要编造评论数以外的事实。');
|
|
81
|
+
}
|
|
63
82
|
lines.push('', 'points 只写 aiNews 与 gameNews 两个键,值是字符串数组;没有内容的板块给空数组。');
|
|
64
83
|
return lines.join('\n').slice(0, 12000);
|
|
65
84
|
}
|
|
@@ -0,0 +1,229 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Bilibili comments — two public reply endpoints, no login required.
|
|
3
|
+
*
|
|
4
|
+
* /x/v2/reply/count?type=1&oid=<aid> → { count }
|
|
5
|
+
* /x/v2/reply/main?type=1&oid=<aid>&mode=3&next=0 → { top_replies, replies, cursor }
|
|
6
|
+
*
|
|
7
|
+
* Counts are exact for every video. A logged-out client is only shown three
|
|
8
|
+
* replies per call (the web player behaves the same way), so the card shows at
|
|
9
|
+
* most a handful of hot comments; adding a Cookie to the source raises the cap.
|
|
10
|
+
* Replies come back already ordered by relevance, but we sort by likes anyway
|
|
11
|
+
* so the card is stable between runs.
|
|
12
|
+
*
|
|
13
|
+
* The module takes a `getJson(url, referer)` callback instead of fetching on its
|
|
14
|
+
* own: bilibili.js already owns the buvid cookie, the 400 ms gap and the saved
|
|
15
|
+
* per-source Cookie header, and both callers must share them.
|
|
16
|
+
*/
|
|
17
|
+
import fs from 'node:fs';
|
|
18
|
+
import path from 'node:path';
|
|
19
|
+
import { plainCommentText as stripEmotes, stripHtml, truncate } from '../text.js';
|
|
20
|
+
import { mapLimit } from './youtube-comments.js';
|
|
21
|
+
|
|
22
|
+
export { plainCommentText } from '../text.js';
|
|
23
|
+
|
|
24
|
+
export const COMMENT_CACHE_MS = 6 * 3600 * 1000;
|
|
25
|
+
export const COMMENT_SCAN_DEFAULT = 6;
|
|
26
|
+
export const COMMENT_SCAN_MAX = 10;
|
|
27
|
+
export const COMMENT_CONCURRENCY = 2;
|
|
28
|
+
|
|
29
|
+
function cachePath(storeDir, aid) {
|
|
30
|
+
if (!storeDir) return '';
|
|
31
|
+
return path.join(storeDir, '.cache', `bilibili-comments-${aid}.json`);
|
|
32
|
+
}
|
|
33
|
+
|
|
34
|
+
function readCache(file) {
|
|
35
|
+
if (!file) return null;
|
|
36
|
+
try {
|
|
37
|
+
return JSON.parse(fs.readFileSync(file, 'utf8'));
|
|
38
|
+
} catch {
|
|
39
|
+
return null;
|
|
40
|
+
}
|
|
41
|
+
}
|
|
42
|
+
|
|
43
|
+
function writeCache(file, value) {
|
|
44
|
+
if (!file) return;
|
|
45
|
+
try {
|
|
46
|
+
fs.mkdirSync(path.dirname(file), { recursive: true });
|
|
47
|
+
fs.writeFileSync(file, JSON.stringify(value));
|
|
48
|
+
} catch {
|
|
49
|
+
/* cache is best-effort */
|
|
50
|
+
}
|
|
51
|
+
}
|
|
52
|
+
|
|
53
|
+
/** Reply count already carried by a list payload (space/ranking/view). */
|
|
54
|
+
export function commentCountFromVideo(video) {
|
|
55
|
+
const candidates = [video?.stat?.reply, video?.comment, video?.reply_count, video?.rcount];
|
|
56
|
+
for (const raw of candidates) {
|
|
57
|
+
const value = Number(raw);
|
|
58
|
+
if (Number.isFinite(value) && value > 0) return Math.round(value);
|
|
59
|
+
}
|
|
60
|
+
return 0;
|
|
61
|
+
}
|
|
62
|
+
|
|
63
|
+
/** One API reply → the flat shape the briefing card renders. */
|
|
64
|
+
export function normalizeReply(reply) {
|
|
65
|
+
const ctime = Number(reply?.ctime);
|
|
66
|
+
return {
|
|
67
|
+
id: String(reply?.rpid_str || reply?.rpid || ''),
|
|
68
|
+
author: truncate(String(reply?.member?.uname || reply?.member?.name || '').trim(), 48),
|
|
69
|
+
text: truncate(stripEmotes(reply?.content?.message), 220),
|
|
70
|
+
likes: Math.max(0, Math.round(Number(reply?.like) || 0)),
|
|
71
|
+
replies: Math.max(0, Math.round(Number(reply?.rcount) || 0)),
|
|
72
|
+
published: Number.isFinite(ctime) && ctime > 0 ? new Date(ctime * 1000).toISOString() : '',
|
|
73
|
+
};
|
|
74
|
+
}
|
|
75
|
+
|
|
76
|
+
/** Merge pinned + hot replies, drop duplicates/empties, best-liked first. */
|
|
77
|
+
export function readReplyPage(data) {
|
|
78
|
+
const raw = [];
|
|
79
|
+
for (const bucket of [data?.top_replies, data?.top, data?.replies]) {
|
|
80
|
+
if (Array.isArray(bucket)) raw.push(...bucket);
|
|
81
|
+
}
|
|
82
|
+
const seen = new Set();
|
|
83
|
+
const out = [];
|
|
84
|
+
for (const reply of raw) {
|
|
85
|
+
const item = normalizeReply(reply);
|
|
86
|
+
if (item.text === '') continue;
|
|
87
|
+
const key = item.id || `${item.author}\u0000${item.text}`;
|
|
88
|
+
if (seen.has(key)) continue;
|
|
89
|
+
seen.add(key);
|
|
90
|
+
out.push(item);
|
|
91
|
+
}
|
|
92
|
+
return out.sort((a, b) => b.likes - a.likes);
|
|
93
|
+
}
|
|
94
|
+
|
|
95
|
+
export function parseReplyCount(data) {
|
|
96
|
+
const value = Number(data?.count);
|
|
97
|
+
return Number.isFinite(value) && value >= 0 ? Math.round(value) : null;
|
|
98
|
+
}
|
|
99
|
+
|
|
100
|
+
/**
|
|
101
|
+
* Comment count (and optionally hot comments) for one video.
|
|
102
|
+
* Never throws: a block (-352/-412), a deleted video or a network error comes
|
|
103
|
+
* back as `{ error }` and the caller keeps the title it already had.
|
|
104
|
+
*/
|
|
105
|
+
export async function fetchVideoComments(
|
|
106
|
+
aid,
|
|
107
|
+
{ getJson, storeDir, withComments = true, maxComments = 4, cacheMs = COMMENT_CACHE_MS, force = false } = {},
|
|
108
|
+
) {
|
|
109
|
+
const id = Number(aid);
|
|
110
|
+
if (!Number.isFinite(id) || id <= 0) return { aid, commentCount: null, comments: [], bytes: 0, elapsedMs: 0 };
|
|
111
|
+
const file = cachePath(storeDir, id);
|
|
112
|
+
const cached = force ? null : readCache(file);
|
|
113
|
+
const fresh = cached && Date.now() - Number(cached.at || 0) < cacheMs;
|
|
114
|
+
// A count-only lookup must not satisfy a later "give me the replies" call.
|
|
115
|
+
const hasComments = cached && (cached.full === true || (Array.isArray(cached.comments) && cached.comments.length > 0));
|
|
116
|
+
if (fresh && Number.isFinite(Number(cached.commentCount)) && (!withComments || hasComments)) {
|
|
117
|
+
const comments = withComments ? (cached.comments || []).slice(0, maxComments) : [];
|
|
118
|
+
return { aid: id, commentCount: Number(cached.commentCount), comments, bytes: 0, elapsedMs: 0, cached: true };
|
|
119
|
+
}
|
|
120
|
+
|
|
121
|
+
const referer = `https://www.bilibili.com/video/av${id}`;
|
|
122
|
+
let bytes = 0;
|
|
123
|
+
let elapsedMs = 0;
|
|
124
|
+
// A fresh cache still holding the count saves the count request when only the
|
|
125
|
+
// replies are missing.
|
|
126
|
+
let commentCount = fresh && Number.isFinite(Number(cached.commentCount)) ? Number(cached.commentCount) : null;
|
|
127
|
+
let comments = [];
|
|
128
|
+
|
|
129
|
+
if (commentCount === null) {
|
|
130
|
+
try {
|
|
131
|
+
const counted = await getJson(`https://api.bilibili.com/x/v2/reply/count?type=1&oid=${id}`, referer);
|
|
132
|
+
bytes += counted.bytes || 0;
|
|
133
|
+
elapsedMs += counted.elapsedMs || 0;
|
|
134
|
+
commentCount = parseReplyCount(counted.data);
|
|
135
|
+
} catch (error) {
|
|
136
|
+
return {
|
|
137
|
+
aid: id,
|
|
138
|
+
commentCount: cached ? Number(cached.commentCount) : null,
|
|
139
|
+
comments: cached?.comments || [],
|
|
140
|
+
bytes,
|
|
141
|
+
elapsedMs,
|
|
142
|
+
error: error?.message || '评论数读取失败',
|
|
143
|
+
};
|
|
144
|
+
}
|
|
145
|
+
}
|
|
146
|
+
|
|
147
|
+
if (withComments && commentCount > 0) {
|
|
148
|
+
try {
|
|
149
|
+
const page = await getJson(
|
|
150
|
+
`https://api.bilibili.com/x/v2/reply/main?type=1&oid=${id}&mode=3&next=0&ps=20`,
|
|
151
|
+
referer,
|
|
152
|
+
);
|
|
153
|
+
bytes += page.bytes || 0;
|
|
154
|
+
elapsedMs += page.elapsedMs || 0;
|
|
155
|
+
comments = readReplyPage(page.data).slice(0, Math.max(1, Math.min(6, maxComments)));
|
|
156
|
+
} catch {
|
|
157
|
+
/* count is what ranks the video; comments are decoration */
|
|
158
|
+
}
|
|
159
|
+
}
|
|
160
|
+
|
|
161
|
+
writeCache(file, { at: Date.now(), commentCount, comments, full: withComments && commentCount > 0 });
|
|
162
|
+
return { aid: id, commentCount, comments, bytes, elapsedMs };
|
|
163
|
+
}
|
|
164
|
+
|
|
165
|
+
/**
|
|
166
|
+
* Give a list of videos their comment counts and pick the most-discussed one.
|
|
167
|
+
* Counts that the list payload already carries (space/ranking/view) cost no
|
|
168
|
+
* request, so a ranking board usually needs exactly one extra call.
|
|
169
|
+
*/
|
|
170
|
+
export async function enrichBilibiliComments(list, options, context, getJson) {
|
|
171
|
+
const items = Array.isArray(list) ? list : [];
|
|
172
|
+
const requested = Number(options?.commentScan ?? COMMENT_SCAN_DEFAULT);
|
|
173
|
+
const scanLimit =
|
|
174
|
+
options?.comments === false
|
|
175
|
+
? 0
|
|
176
|
+
: Math.max(
|
|
177
|
+
0,
|
|
178
|
+
Math.min(items.length, Number.isFinite(requested) ? requested : COMMENT_SCAN_DEFAULT, COMMENT_SCAN_MAX),
|
|
179
|
+
);
|
|
180
|
+
if (scanLimit === 0) return { items, featured: null, bytes: 0, note: '' };
|
|
181
|
+
|
|
182
|
+
let bytes = 0;
|
|
183
|
+
let failures = 0;
|
|
184
|
+
const scanned = await mapLimit(items.slice(0, scanLimit), COMMENT_CONCURRENCY, async (item) => {
|
|
185
|
+
if (Number(item.commentCount) > 0) return item;
|
|
186
|
+
const aid = Number(item.aid);
|
|
187
|
+
if (!Number.isFinite(aid) || aid <= 0) return item;
|
|
188
|
+
const stats = await fetchVideoComments(aid, { getJson, storeDir: context.storeDir, withComments: false });
|
|
189
|
+
bytes += stats.bytes || 0;
|
|
190
|
+
if (stats.error) failures += 1;
|
|
191
|
+
if (Number.isFinite(stats.commentCount) && stats.commentCount > 0) {
|
|
192
|
+
item.commentCount = Math.round(stats.commentCount);
|
|
193
|
+
}
|
|
194
|
+
return item;
|
|
195
|
+
});
|
|
196
|
+
|
|
197
|
+
const ranked = scanned.filter((item) => Number(item.commentCount) > 0).sort((a, b) => b.commentCount - a.commentCount);
|
|
198
|
+
if (ranked.length === 0) {
|
|
199
|
+
return { items, featured: null, bytes, note: failures > 0 ? '评论数暂时读不到' : '' };
|
|
200
|
+
}
|
|
201
|
+
|
|
202
|
+
const best = ranked[0];
|
|
203
|
+
const requestedComments = Number(options?.commentsPerVideo ?? 4);
|
|
204
|
+
const perVideo = Math.max(1, Math.min(6, Number.isFinite(requestedComments) ? requestedComments : 4));
|
|
205
|
+
const full = await fetchVideoComments(Number(best.aid), {
|
|
206
|
+
getJson,
|
|
207
|
+
storeDir: context.storeDir,
|
|
208
|
+
withComments: true,
|
|
209
|
+
maxComments: perVideo,
|
|
210
|
+
});
|
|
211
|
+
bytes += full.bytes || 0;
|
|
212
|
+
const comments = (full.comments ?? []).slice(0, perVideo);
|
|
213
|
+
best.comments = comments;
|
|
214
|
+
|
|
215
|
+
return {
|
|
216
|
+
items,
|
|
217
|
+
featured: {
|
|
218
|
+
id: best.id,
|
|
219
|
+
title: best.title,
|
|
220
|
+
url: best.url,
|
|
221
|
+
date: best.date ?? '',
|
|
222
|
+
meta: best.meta ?? 'B 站',
|
|
223
|
+
commentCount: best.commentCount,
|
|
224
|
+
comments,
|
|
225
|
+
},
|
|
226
|
+
bytes,
|
|
227
|
+
note: `评论最多:${truncate(best.title, 60)}(${best.commentCount} 条)`,
|
|
228
|
+
};
|
|
229
|
+
}
|
|
@@ -13,6 +13,7 @@ import path from 'node:path';
|
|
|
13
13
|
import { safeFetch } from '../fetch.js';
|
|
14
14
|
import { readSourceHeaders } from '../secrets.js';
|
|
15
15
|
import { truncate, stripHtml } from '../text.js';
|
|
16
|
+
import { commentCountFromVideo, enrichBilibiliComments } from './bilibili-comments.js';
|
|
16
17
|
|
|
17
18
|
const NAV_CACHE_MS = 6 * 3600 * 1000;
|
|
18
19
|
const ITEMS_CACHE_MS = 8 * 60 * 1000;
|
|
@@ -82,7 +83,19 @@ function biliError(data) {
|
|
|
82
83
|
return `B 站接口返回 ${Number.isFinite(code) ? code : '未知'}:${data?.message || '未知错误'}`;
|
|
83
84
|
}
|
|
84
85
|
|
|
85
|
-
|
|
86
|
+
// B 站风控按 IP 计费:一旦答 -412,同一轮里再敲十几次只会把封禁时间拖长。
|
|
87
|
+
// 命中后进入冷却,后续请求直接短路;列表类请求可用 bypassRiskCooldown 再试一次。
|
|
88
|
+
let riskUntil = 0;
|
|
89
|
+
const RISK_COOLDOWN_MS = 10 * 60 * 1000;
|
|
90
|
+
|
|
91
|
+
function isRiskError(error) {
|
|
92
|
+
return /-(352|412|799)|风控|限流|冷却/.test(String(error?.message || ''));
|
|
93
|
+
}
|
|
94
|
+
|
|
95
|
+
async function biliJson(url, storeDir, referer, source, options = {}) {
|
|
96
|
+
if (options.bypassRiskCooldown !== true && Date.now() < riskUntil) {
|
|
97
|
+
throw new Error('B 站接口仍在风控冷却中,本次跳过');
|
|
98
|
+
}
|
|
86
99
|
await gap();
|
|
87
100
|
const cookie = await buvidCookie(storeDir);
|
|
88
101
|
// A saved Cookie request header lets users read spaces that would otherwise
|
|
@@ -100,6 +113,7 @@ async function biliJson(url, storeDir, referer, source) {
|
|
|
100
113
|
},
|
|
101
114
|
});
|
|
102
115
|
const data = parseJson(response.text);
|
|
116
|
+
if (Number(data?.code) === -412) riskUntil = Date.now() + RISK_COOLDOWN_MS;
|
|
103
117
|
if (Number(data?.code) !== 0) throw new Error(biliError(data));
|
|
104
118
|
return { data: data.data, bytes: response.bytes, elapsedMs: response.elapsedMs };
|
|
105
119
|
}
|
|
@@ -197,6 +211,8 @@ function itemFromVideo(video, authorFallback = '') {
|
|
|
197
211
|
const date = created > 0 ? new Date(created * 1000).toISOString() : '';
|
|
198
212
|
const author = String(video?.owner?.name || video?.author || authorFallback || '').trim();
|
|
199
213
|
const description = String(video?.desc || video?.description || '').trim();
|
|
214
|
+
const aid = Number(video?.aid || 0);
|
|
215
|
+
const commentCount = commentCountFromVideo(video);
|
|
200
216
|
return {
|
|
201
217
|
id: (`bilibili-${bvid || video?.aid || video?.title || ''}`).slice(0, 300),
|
|
202
218
|
title: truncate(stripHtml(video?.title || ''), 180),
|
|
@@ -206,10 +222,35 @@ function itemFromVideo(video, authorFallback = '') {
|
|
|
206
222
|
meta: author || 'B 站',
|
|
207
223
|
badge: 'B站',
|
|
208
224
|
tone: '',
|
|
225
|
+
// aid stays on the item so the comment pass can ask about exactly the
|
|
226
|
+
// videos it shows; normalizeItem() drops it again before rendering.
|
|
227
|
+
...(Number.isFinite(aid) && aid > 0 ? { aid } : {}),
|
|
228
|
+
...(commentCount > 0 ? { commentCount } : {}),
|
|
209
229
|
};
|
|
210
230
|
}
|
|
211
231
|
|
|
212
232
|
async function collectSpace(mid, source, context) {
|
|
233
|
+
try {
|
|
234
|
+
return await collectSpaceArchive(mid, source, context);
|
|
235
|
+
} catch (error) {
|
|
236
|
+
// 空间接口在限流时也会用「空列表」冒充成功,所以「没有公开视频」也要再试一次。
|
|
237
|
+
if (!isUnavailable(error)) throw error;
|
|
238
|
+
try {
|
|
239
|
+
return await collectSpaceDynamic(mid, source, context);
|
|
240
|
+
} catch (fallback) {
|
|
241
|
+
if (!isUnavailable(fallback)) throw fallback;
|
|
242
|
+
throw new Error('B 站空间暂时读不到(本机 IP 被限流 -412,或该 UP 没有公开投稿):限流通常 5–30 分钟恢复;也可以先改用单条视频链接,或给该来源补填浏览器 Cookie。');
|
|
243
|
+
}
|
|
244
|
+
}
|
|
245
|
+
}
|
|
246
|
+
|
|
247
|
+
/** 限流与「读不到」都值得换条路再试;真正的配置错误不该被吞掉。 */
|
|
248
|
+
function isUnavailable(error) {
|
|
249
|
+
const message = String(error?.message || '');
|
|
250
|
+
return isRiskError(error) || /没有公开视频|读不到/.test(message);
|
|
251
|
+
}
|
|
252
|
+
|
|
253
|
+
async function collectSpaceArchive(mid, source, context) {
|
|
213
254
|
const limit = source.options.limit ?? 8;
|
|
214
255
|
const keys = await wbiKeys(context.storeDir);
|
|
215
256
|
const query = signQuery(
|
|
@@ -221,6 +262,7 @@ async function collectSpace(mid, source, context) {
|
|
|
221
262
|
context.storeDir,
|
|
222
263
|
`https://space.bilibili.com/${mid}`,
|
|
223
264
|
source,
|
|
265
|
+
{ bypassRiskCooldown: true },
|
|
224
266
|
);
|
|
225
267
|
const list = data?.list?.vlist || [];
|
|
226
268
|
if (list.length === 0) throw new Error('这个 B 站空间暂时没有公开视频');
|
|
@@ -233,6 +275,59 @@ async function collectSpace(mid, source, context) {
|
|
|
233
275
|
};
|
|
234
276
|
}
|
|
235
277
|
|
|
278
|
+
/** 动态接口里的投稿(DYNAMIC_TYPE_AV)摊平成视频对象。 */
|
|
279
|
+
function dynamicVideos(data) {
|
|
280
|
+
const entries = Array.isArray(data?.items) ? data.items : [];
|
|
281
|
+
const videos = [];
|
|
282
|
+
for (const entry of entries) {
|
|
283
|
+
if (String(entry?.type || '') !== 'DYNAMIC_TYPE_AV') continue;
|
|
284
|
+
const archive = entry?.modules?.module_dynamic?.major?.archive;
|
|
285
|
+
if (!archive?.bvid) continue;
|
|
286
|
+
const published = Number(archive.pubdate || entry?.modules?.module_author?.pub_ts || 0);
|
|
287
|
+
videos.push({
|
|
288
|
+
...archive,
|
|
289
|
+
created: published,
|
|
290
|
+
owner: { name: String(entry?.modules?.module_author?.name || '') },
|
|
291
|
+
});
|
|
292
|
+
}
|
|
293
|
+
return videos;
|
|
294
|
+
}
|
|
295
|
+
|
|
296
|
+
async function collectSpaceDynamic(mid, source, context) {
|
|
297
|
+
const limit = Math.min(source.options.limit ?? 8, 12);
|
|
298
|
+
const cutoff = Date.now() - (source.options.sinceHours ?? 24) * 3600000;
|
|
299
|
+
let offset = '';
|
|
300
|
+
let bytes = 0;
|
|
301
|
+
let elapsedMs = 0;
|
|
302
|
+
const videos = [];
|
|
303
|
+
for (let page = 0; page < 3; page += 1) {
|
|
304
|
+
const suffix = offset === '' ? '' : `&offset=${encodeURIComponent(offset)}`;
|
|
305
|
+
const { data, bytes: size, elapsedMs: spent } = await biliJson(
|
|
306
|
+
`https://api.bilibili.com/x/polymer/web-dynamic/v1/feed/space?host_mid=${encodeURIComponent(mid)}&timezone_offset=-480&features=itemOpusStyle${suffix}`,
|
|
307
|
+
context.storeDir,
|
|
308
|
+
`https://space.bilibili.com/${mid}/dynamic`,
|
|
309
|
+
source,
|
|
310
|
+
{ bypassRiskCooldown: true },
|
|
311
|
+
);
|
|
312
|
+
bytes += size;
|
|
313
|
+
elapsedMs += spent;
|
|
314
|
+
videos.push(...dynamicVideos(data));
|
|
315
|
+
const next = data?.offset;
|
|
316
|
+
if (data?.has_more !== true || !next || String(next) === offset) break;
|
|
317
|
+
offset = String(next);
|
|
318
|
+
const oldest = videos.reduce((min, video) => Math.min(min, Number(video.created) * 1000 || Number.MAX_SAFE_INTEGER), Number.MAX_SAFE_INTEGER);
|
|
319
|
+
if (videos.length >= limit * 2 || oldest < cutoff) break;
|
|
320
|
+
}
|
|
321
|
+
if (videos.length === 0) throw new Error('这个 B 站空间暂时没有公开视频');
|
|
322
|
+
const name = String(videos[0]?.owner?.name || '').trim();
|
|
323
|
+
return {
|
|
324
|
+
items: videos.map((video) => itemFromVideo(video, name)).filter((item) => item.title !== ''),
|
|
325
|
+
note: name ? `来自 ${name} 的 B 站投稿` : 'B 站 UP 主更新',
|
|
326
|
+
bytes,
|
|
327
|
+
elapsedMs,
|
|
328
|
+
};
|
|
329
|
+
}
|
|
330
|
+
|
|
236
331
|
async function collectVideo(source, parsed, context) {
|
|
237
332
|
let bvid = String(source.options.bvid || parsed.bvid || '').trim();
|
|
238
333
|
let aid = String(parsed.aid || '').trim();
|
|
@@ -307,7 +402,20 @@ export async function collectBilibili(source, context = {}) {
|
|
|
307
402
|
|
|
308
403
|
let result;
|
|
309
404
|
if (mode === 'space') {
|
|
310
|
-
|
|
405
|
+
try {
|
|
406
|
+
result = await collectSpace(String(options.mid || parsed.mid), source, context);
|
|
407
|
+
} catch (error) {
|
|
408
|
+
// 空间接口限流时,宁可端出上次读到的投稿,也不要让整个来源当天消失。
|
|
409
|
+
const stale = readCache(file);
|
|
410
|
+
const items = stale?.value?.items || [];
|
|
411
|
+
if (isUnavailable(error) && items.length > 0) {
|
|
412
|
+
return {
|
|
413
|
+
...stale.value,
|
|
414
|
+
note: [stale.value.note, '接口限流,沿用上次缓存'].filter(Boolean).join(' · '),
|
|
415
|
+
};
|
|
416
|
+
}
|
|
417
|
+
throw error;
|
|
418
|
+
}
|
|
311
419
|
} else if (mode === 'ranking') {
|
|
312
420
|
result = await collectRanking(source, context);
|
|
313
421
|
} else {
|
|
@@ -319,6 +427,22 @@ export async function collectBilibili(source, context = {}) {
|
|
|
319
427
|
return new Date(item.date).getTime() >= cutoff;
|
|
320
428
|
});
|
|
321
429
|
if (fresh.length > 0 && mode === 'space') result = { ...result, items: fresh.slice(0, options.limit ?? 8) };
|
|
430
|
+
|
|
431
|
+
// Same treatment as YouTube: count every video on the card, then keep the
|
|
432
|
+
// most-discussed one as the day's featured card with its hottest replies.
|
|
433
|
+
const enriched = await enrichBilibiliComments(result.items, options, context, (url, referer) =>
|
|
434
|
+
biliJson(url, context.storeDir, referer, source),
|
|
435
|
+
);
|
|
436
|
+
if (enriched.featured || enriched.note) {
|
|
437
|
+
result = {
|
|
438
|
+
...result,
|
|
439
|
+
items: enriched.items,
|
|
440
|
+
bytes: (result.bytes || 0) + (enriched.bytes || 0),
|
|
441
|
+
...(enriched.featured ? { featured: enriched.featured } : {}),
|
|
442
|
+
...(enriched.note ? { note: [result.note, enriched.note].filter(Boolean).join(' · ') } : {}),
|
|
443
|
+
};
|
|
444
|
+
}
|
|
445
|
+
|
|
322
446
|
writeCache(file, { value: result, at: Date.now() });
|
|
323
447
|
return result;
|
|
324
448
|
}
|
|
@@ -17,7 +17,7 @@ import { collectBilibili } from './bilibili.js';
|
|
|
17
17
|
import { collectX } from './x.js';
|
|
18
18
|
import { collectSteam } from './steam.js';
|
|
19
19
|
import { collectMinecraft } from './minecraft.js';
|
|
20
|
-
import { truncate } from '../text.js';
|
|
20
|
+
import { plainCommentText, truncate } from '../text.js';
|
|
21
21
|
|
|
22
22
|
export const COLLECTORS = {
|
|
23
23
|
x: collectX,
|
|
@@ -41,8 +41,38 @@ export const COLLECTORS = {
|
|
|
41
41
|
// API hops (channel id, feed, WBI signature), so these need more headroom.
|
|
42
42
|
const SLOW_TYPES = new Set(['imap', 'graph', 'browser', 'x', 'steam', 'youtube', 'bilibili']);
|
|
43
43
|
|
|
44
|
-
function
|
|
44
|
+
function normalizeComments(value) {
|
|
45
|
+
if (!Array.isArray(value)) return [];
|
|
46
|
+
return value
|
|
47
|
+
.slice(0, 4)
|
|
48
|
+
.map((comment) => ({
|
|
49
|
+
author: truncate(String(comment?.author ?? '').replace(/\s+/g, ' ').trim(), 48),
|
|
50
|
+
text: truncate(plainCommentText(comment?.text), 220),
|
|
51
|
+
likes: Number(comment?.likes) > 0 ? Math.round(Number(comment.likes)) : 0,
|
|
52
|
+
replies: Number(comment?.replies) > 0 ? Math.round(Number(comment.replies)) : 0,
|
|
53
|
+
}))
|
|
54
|
+
.filter((comment) => comment.text !== '');
|
|
55
|
+
}
|
|
56
|
+
|
|
57
|
+
function normalizeFeatured(featured, source) {
|
|
58
|
+
if (!featured || typeof featured !== 'object') return null;
|
|
59
|
+
const commentCount = Number(featured.commentCount);
|
|
45
60
|
return {
|
|
61
|
+
id: truncate(String(featured.id ?? featured.url ?? ''), 300),
|
|
62
|
+
title: truncate(String(featured.title ?? ''), 200),
|
|
63
|
+
url: typeof featured.url === 'string' ? featured.url : '',
|
|
64
|
+
date: typeof featured.date === 'string' ? featured.date : '',
|
|
65
|
+
meta: truncate(String(featured.meta ?? ''), 80),
|
|
66
|
+
commentCount: Number.isFinite(commentCount) && commentCount > 0 ? Math.round(commentCount) : 0,
|
|
67
|
+
comments: normalizeComments(featured.comments),
|
|
68
|
+
sourceId: source.id,
|
|
69
|
+
sourceLabel: source.label,
|
|
70
|
+
section: source.section,
|
|
71
|
+
};
|
|
72
|
+
}
|
|
73
|
+
|
|
74
|
+
function normalizeItem(item, source) {
|
|
75
|
+
const out = {
|
|
46
76
|
id: truncate(String(item.id ?? item.url ?? item.title ?? `${Math.random()}`), 300),
|
|
47
77
|
title: truncate(String(item.title ?? ''), 200),
|
|
48
78
|
url: typeof item.url === 'string' ? item.url : '',
|
|
@@ -54,6 +84,11 @@ function normalizeItem(item, source) {
|
|
|
54
84
|
sourceId: source.id,
|
|
55
85
|
sourceLabel: source.label,
|
|
56
86
|
};
|
|
87
|
+
const commentCount = Number(item.commentCount);
|
|
88
|
+
if (Number.isFinite(commentCount) && commentCount > 0) out.commentCount = Math.round(commentCount);
|
|
89
|
+
const comments = normalizeComments(item.comments);
|
|
90
|
+
if (comments.length > 0) out.comments = comments;
|
|
91
|
+
return out;
|
|
57
92
|
}
|
|
58
93
|
|
|
59
94
|
export async function collectSource(source, context = {}) {
|
|
@@ -75,6 +110,7 @@ export async function collectSource(source, context = {}) {
|
|
|
75
110
|
}),
|
|
76
111
|
]);
|
|
77
112
|
const items = (result?.items ?? []).map((item) => normalizeItem(item, source)).filter((item) => item.title !== '');
|
|
113
|
+
const featured = normalizeFeatured(result?.featured, source);
|
|
78
114
|
return {
|
|
79
115
|
sourceId: source.id,
|
|
80
116
|
status: result?.status ?? (items.length > 0 ? 'ok' : 'skipped'),
|
|
@@ -82,6 +118,7 @@ export async function collectSource(source, context = {}) {
|
|
|
82
118
|
note: result?.note ?? '',
|
|
83
119
|
bytes: result?.bytes,
|
|
84
120
|
server: result?.server,
|
|
121
|
+
...(featured ? { featured } : {}),
|
|
85
122
|
elapsedMs: result?.elapsedMs ?? Date.now() - started,
|
|
86
123
|
};
|
|
87
124
|
} catch (error) {
|