socialdatax-skills 0.2.29 → 0.2.30
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +17 -3
- package/cli.mjs +108 -22
- package/lib/media/{xhs-download.mjs → platform-download.mjs} +134 -69
- package/package.json +1 -1
- package/skills/media-comments/SKILL.md +1 -1
- package/skills/media-detail/SKILL.md +6 -3
- package/skills/media-search/SKILL.md +1 -1
- package/skills/media-transcript/SKILL.md +1 -1
- package/skills/media-user-info/SKILL.md +1 -1
- package/skills/media-user-posts/SKILL.md +1 -1
- package/skills/sensitive-check/SKILL.md +1 -1
- package/skills/socialdatax-content-research-assistant/SKILL.md +1 -1
package/README.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
This public package provides the unified skill installer and direct CLI helper for SocialDataX services.
|
|
4
4
|
|
|
5
|
-
The current public tools support 小红书 / Xiaohongshu / XHS / RedNote, 抖音 / Douyin, 快手 / Kuaishou / Kwai, 微博 / Weibo, 视频号 / WeChat Channels, and 微信公众号 / WeChat Official Account article content research and analysis workflows, plus XHS local media download, Bilibili / 哔哩哔哩 / B站 local video download, and 敏感词检测 / 违禁词检查 text checks. The public skill layer is intentionally named by capability so supported services can evolve without changing the installation model.
|
|
5
|
+
The current public tools support 小红书 / Xiaohongshu / XHS / RedNote, 抖音 / Douyin, 快手 / Kuaishou / Kwai, 微博 / Weibo, 视频号 / WeChat Channels, and 微信公众号 / WeChat Official Account article content research and analysis workflows, plus XHS / Douyin / Kuaishou / Weibo local media download, Bilibili / 哔哩哔哩 / B站 local video download, and 敏感词检测 / 违禁词检查 text checks. The public skill layer is intentionally named by capability so supported services can evolve without changing the installation model.
|
|
6
6
|
|
|
7
7
|
- direct `npx` JSON commands for agents that can run shell commands
|
|
8
8
|
- AgentSkills-compatible installers split by capability for OpenClaw, Hermes Agent, Codex, Claude Code, and general agent skill directories
|
|
@@ -10,6 +10,10 @@ The current public tools support 小红书 / Xiaohongshu / XHS / RedNote, 抖音
|
|
|
10
10
|
|
|
11
11
|
The business implementation is privately hosted. This repository exposes only the public package, skill, and connection surface for social media content intelligence workflows. It is not a unified MCP server and does not include a registry server card; platform MCPs are published separately.
|
|
12
12
|
|
|
13
|
+
## High-Intent Skill Catalog
|
|
14
|
+
|
|
15
|
+
See [CATALOG.md](./CATALOG.md) for curated high-intent Agent Skill entries across SkillHub, ClawHub, ModelScope, and direct CLI usage. The catalog focuses on proven workflows such as XHS content research, comment analysis, hot topic planning, viral note research, Douyin transcript extraction, and sensitive term checks.
|
|
16
|
+
|
|
13
17
|
## Search Aliases
|
|
14
18
|
|
|
15
19
|
Common search phrases for this skill package:
|
|
@@ -82,7 +86,7 @@ Common search phrases for this skill package:
|
|
|
82
86
|
- Sensitive Words Check MCP server name: `sensitive-check`
|
|
83
87
|
- Sensitive Words Check future MCP registry name: `com.socialdatax/sensitive-check`
|
|
84
88
|
- Unified MCP registry name: none; this package installs skills and calls explicit platform services.
|
|
85
|
-
- Current public capability version: `0.2.
|
|
89
|
+
- Current public capability version: `0.2.30`
|
|
86
90
|
|
|
87
91
|
## Direct CLI
|
|
88
92
|
|
|
@@ -121,6 +125,7 @@ npx -y socialdatax-skills@latest douyin user-series --profile-url "<profile_url_
|
|
|
121
125
|
npx -y socialdatax-skills@latest douyin transcript --url "<douyin_content_url_or_share_text>" --pretty
|
|
122
126
|
npx -y socialdatax-skills@latest douyin transcript --aweme-id "<aweme_id>" --pretty
|
|
123
127
|
npx -y socialdatax-skills@latest douyin transcript --job-id "<job_id>" --pretty
|
|
128
|
+
npx -y socialdatax-skills@latest douyin download-media --url "<douyin_media_url>" --output-dir ./downloads --pretty
|
|
124
129
|
npx -y socialdatax-skills@latest kuaishou hot-search --pretty
|
|
125
130
|
npx -y socialdatax-skills@latest kuaishou search --keyword "露营" --pretty
|
|
126
131
|
npx -y socialdatax-skills@latest kuaishou user-search --keyword "露营" --pretty
|
|
@@ -137,6 +142,7 @@ npx -y socialdatax-skills@latest kuaishou user-posts --profile-url "<profile_url
|
|
|
137
142
|
npx -y socialdatax-skills@latest kuaishou transcript --url "<kuaishou_content_url_or_share_text>" --pretty
|
|
138
143
|
npx -y socialdatax-skills@latest kuaishou transcript --photo-id "<photo_id>" --pretty
|
|
139
144
|
npx -y socialdatax-skills@latest kuaishou transcript --job-id "<job_id>" --pretty
|
|
145
|
+
npx -y socialdatax-skills@latest kuaishou download-media --url "<kuaishou_media_url>" --output-dir ./downloads --pretty
|
|
140
146
|
npx -y socialdatax-skills@latest bilibili download --url "<bilibili_video_url_or_share_text>" --output-dir ./downloads --pretty
|
|
141
147
|
npx -y socialdatax-skills@latest weibo hot-search --pretty
|
|
142
148
|
npx -y socialdatax-skills@latest weibo search --keyword "露营" --pretty
|
|
@@ -155,6 +161,7 @@ npx -y socialdatax-skills@latest weibo user-posts --profile-url "<profile_url_or
|
|
|
155
161
|
npx -y socialdatax-skills@latest weibo transcript --post-url "<weibo_post_url_or_share_text>" --pretty
|
|
156
162
|
npx -y socialdatax-skills@latest weibo transcript --post-id "<post_id>" --pretty
|
|
157
163
|
npx -y socialdatax-skills@latest weibo transcript --job-id "<job_id>" --pretty
|
|
164
|
+
npx -y socialdatax-skills@latest weibo download-media --url "<weibo_media_url>" --output-dir ./downloads --pretty
|
|
158
165
|
npx -y socialdatax-skills@latest wechat hot-search --pretty
|
|
159
166
|
npx -y socialdatax-skills@latest wechat search --keyword "露营" --pretty
|
|
160
167
|
npx -y socialdatax-skills@latest wechat detail --encrypted-object-id "<encrypted_object_id>" --pretty
|
|
@@ -174,7 +181,7 @@ npx -y socialdatax-skills@latest wechat transcript --job-id "<job_id>" --pretty
|
|
|
174
181
|
npx -y socialdatax-skills@latest sensitive-check text --text "<content>" --platform xhs --pretty
|
|
175
182
|
```
|
|
176
183
|
|
|
177
|
-
Most direct CLI commands print a JSON envelope with `platform`, `tool`, `arguments`, and `data`. `sensitive-check` prints `platform`, `tool`, and `data` only; it does not echo the original text in CLI output. `wechat decrypt-media` is a local save command: pass the `video.video_url` returned by WeChat detail and an `--output` file path. It saves the media locally, decrypts when needed, and does not require `SOCIALDATAX_API_KEY`. `xhs download-media` is also local: pass one
|
|
184
|
+
Most direct CLI commands print a JSON envelope with `platform`, `tool`, `arguments`, and `data`. `sensitive-check` prints `platform`, `tool`, and `data` only; it does not echo the original text in CLI output. `wechat decrypt-media` is a local save command: pass the `video.video_url` returned by WeChat detail and an `--output` file path. It saves the media locally, decrypts when needed, and does not require `SOCIALDATAX_API_KEY`. `xhs/douyin/kuaishou/weibo download-media` is also local: pass one media URL returned by detail and either `--output <file>` or `--output-dir <directory>`. It writes through a `.part` file, resumes partial downloads when the server supports range requests, skips an already existing output file, and infers common image/video/audio extensions in `--output-dir` mode.
|
|
178
185
|
|
|
179
186
|
For transcript commands, the direct CLI tries to deliver the final result in one run: submit may wait up to 210 seconds, and if `data.is_terminal` is not `true`, the CLI automatically continues matching get-job requests for up to 1200 seconds by default, with each get-job request waiting up to 240 seconds. Use positive `--max-wait-seconds <seconds>` to tune that follow-up window. Do not submit a duplicate transcript job just to poll status.
|
|
180
187
|
|
|
@@ -306,13 +313,16 @@ npx -y socialdatax-skills@latest douyin hot-search --pretty
|
|
|
306
313
|
npx -y socialdatax-skills@latest douyin search --keyword "露营" --pretty
|
|
307
314
|
npx -y socialdatax-skills@latest douyin user-series --sec-user-id "<sec_user_id>" --pretty
|
|
308
315
|
npx -y socialdatax-skills@latest douyin transcript --aweme-id "<aweme_id>" --pretty
|
|
316
|
+
npx -y socialdatax-skills@latest douyin download-media --url "<douyin_media_url>" --output-dir ./downloads --pretty
|
|
309
317
|
npx -y socialdatax-skills@latest kuaishou hot-search --pretty
|
|
310
318
|
npx -y socialdatax-skills@latest kuaishou search --keyword "露营" --pretty
|
|
311
319
|
npx -y socialdatax-skills@latest kuaishou transcript --photo-id "<photo_id>" --pretty
|
|
320
|
+
npx -y socialdatax-skills@latest kuaishou download-media --url "<kuaishou_media_url>" --output-dir ./downloads --pretty
|
|
312
321
|
npx -y socialdatax-skills@latest bilibili download --url "<bilibili_video_url_or_share_text>" --output-dir ./downloads --pretty
|
|
313
322
|
npx -y socialdatax-skills@latest weibo hot-search --pretty
|
|
314
323
|
npx -y socialdatax-skills@latest weibo search --keyword "露营" --pretty
|
|
315
324
|
npx -y socialdatax-skills@latest weibo transcript --post-id "<post_id>" --pretty
|
|
325
|
+
npx -y socialdatax-skills@latest weibo download-media --url "<weibo_media_url>" --output-dir ./downloads --pretty
|
|
316
326
|
npx -y socialdatax-skills@latest wechat hot-search --pretty
|
|
317
327
|
npx -y socialdatax-skills@latest wechat search --keyword "露营" --pretty
|
|
318
328
|
npx -y socialdatax-skills@latest wechat article --url "<mp_article_url_or_share_text>" --pretty
|
|
@@ -360,6 +370,7 @@ Current XHS workflows include:
|
|
|
360
370
|
- Read creator profile data from a profile link, short link, share text, or user ID.
|
|
361
371
|
- Fetch paginated creator note lists from a user ID, profile link, short link, or share text for content style and account research.
|
|
362
372
|
- Submit and check video note 口播转文字 / speech-to-text transcript jobs; submit tools 提交后最多等待 210 秒,未完成时继续查询同一个 job_id 直到终态.
|
|
373
|
+
- Save returned `image_items[].image_url`, `image_items[].live_photo.video_url`, or `video.video_url` media links locally with `xhs download-media`.
|
|
363
374
|
|
|
364
375
|
Current Kuaishou workflows include:
|
|
365
376
|
|
|
@@ -373,6 +384,7 @@ Current Kuaishou workflows include:
|
|
|
373
384
|
- Read creator profile data from a non-empty user_id, profile link, short link, or share text. Live/fw-user profile shares are supported for profile data; successful results return a reusable non-empty user_id.
|
|
374
385
|
- Fetch paginated creator work lists from a non-empty user_id, or from a profile link, short link, or share text that resolves directly to a non-empty user_id. For live/fw-user profile shares, call creator profile first and use the returned non-empty user_id.
|
|
375
386
|
- Submit and check video work 口播转文字 / speech-to-text transcript jobs; submit tools 提交后最多等待 210 秒,未完成时继续查询同一个 job_id 直到终态.
|
|
387
|
+
- Save returned `images[].url`, `video.play_url`, or `cover_image_url` media links locally with `kuaishou download-media`.
|
|
376
388
|
|
|
377
389
|
Current Bilibili workflows include:
|
|
378
390
|
|
|
@@ -392,6 +404,7 @@ Current Douyin workflows include:
|
|
|
392
404
|
- Fetch paginated creator work lists from a sec_user_id, profile link, short link, or share text.
|
|
393
405
|
- Fetch paginated creator short-drama series lists from a sec_user_id, profile link, short link, or share text.
|
|
394
406
|
- Submit and check video work 口播转文字 / speech-to-text transcript jobs; submit tools 提交后最多等待 210 秒,未完成时继续查询同一个 job_id 直到终态.
|
|
407
|
+
- Save returned `images[].url`, `images[].live_photo.play_url`, `video.play_url`, `music.play_url`, or `cover_image_url` media links locally with `douyin download-media`.
|
|
395
408
|
|
|
396
409
|
Current Weibo workflows include:
|
|
397
410
|
|
|
@@ -404,6 +417,7 @@ Current Weibo workflows include:
|
|
|
404
417
|
- Read creator profile data from a profile link, short link, share text, or user_id.
|
|
405
418
|
- Fetch paginated creator post lists from a user_id, profile link, short link, or share text.
|
|
406
419
|
- Submit and check Weibo video 口播转文字 / speech-to-text transcript jobs; submit tools 提交后最多等待 210 秒,未完成时继续查询同一个 job_id 直到终态.
|
|
420
|
+
- Save returned `image_urls[]` or `video.video_url` media links locally with `weibo download-media`.
|
|
407
421
|
|
|
408
422
|
Current WeChat / 微信 workflows include:
|
|
409
423
|
|
package/cli.mjs
CHANGED
|
@@ -16,13 +16,13 @@ import {
|
|
|
16
16
|
assertBilibiliFfmpegAvailable,
|
|
17
17
|
downloadBilibiliVideoFromManifest,
|
|
18
18
|
} from "./lib/media/bilibili-download.mjs";
|
|
19
|
+
import { downloadPlatformMediaFromUrl } from "./lib/media/platform-download.mjs";
|
|
19
20
|
import { decryptWechatMediaCommand } from "./lib/media/wechat-decrypt.mjs";
|
|
20
|
-
import { downloadXhsMediaFromUrl } from "./lib/media/xhs-download.mjs";
|
|
21
21
|
|
|
22
22
|
export { decryptWechatMediaCommand };
|
|
23
23
|
|
|
24
24
|
const PACKAGE_NAME = "socialdatax-skills";
|
|
25
|
-
const PACKAGE_VERSION = "0.2.
|
|
25
|
+
const PACKAGE_VERSION = "0.2.30";
|
|
26
26
|
const PACKAGE_SPEC = `${PACKAGE_NAME}@latest`;
|
|
27
27
|
const LOG_PREFIX = `[${PACKAGE_NAME}]`;
|
|
28
28
|
const MIN_NODE_VERSION = "20.18.1";
|
|
@@ -261,6 +261,7 @@ const DOUYIN_DIRECT_ACTION_OPTIONS = {
|
|
|
261
261
|
],
|
|
262
262
|
"user-series": ["secUserId", "profileUrl", "pageToken", "pages", "all", "maxItems", "pretty"],
|
|
263
263
|
transcript: ["url", "awemeId", "jobId", "maxWaitSeconds", "pretty"],
|
|
264
|
+
"download-media": ["url", "output", "outputDir", "pretty"],
|
|
264
265
|
};
|
|
265
266
|
const DOUYIN_DIRECT_ACTION_NAMES = Object.keys(DOUYIN_DIRECT_ACTION_OPTIONS).join(", ");
|
|
266
267
|
const DOUYIN_SEARCH_SORT_TYPES = ["general", "time_descending", "like_count_descending"];
|
|
@@ -291,6 +292,8 @@ const DOUYIN_OPTION_DISPLAY_NAMES = {
|
|
|
291
292
|
secUserId: "--sec-user-id",
|
|
292
293
|
jobId: "--job-id",
|
|
293
294
|
maxWaitSeconds: "--max-wait-seconds",
|
|
295
|
+
output: "--output",
|
|
296
|
+
outputDir: "--output-dir",
|
|
294
297
|
};
|
|
295
298
|
const KUAISHOU_DIRECT_ACTION_OPTIONS = {
|
|
296
299
|
"hot-search": ["pretty"],
|
|
@@ -329,6 +332,7 @@ const KUAISHOU_DIRECT_ACTION_OPTIONS = {
|
|
|
329
332
|
"pretty",
|
|
330
333
|
],
|
|
331
334
|
transcript: ["url", "photoId", "jobId", "maxWaitSeconds", "pretty"],
|
|
335
|
+
"download-media": ["url", "output", "outputDir", "pretty"],
|
|
332
336
|
};
|
|
333
337
|
const KUAISHOU_DIRECT_ACTION_NAMES = Object.keys(KUAISHOU_DIRECT_ACTION_OPTIONS).join(", ");
|
|
334
338
|
const KUAISHOU_OPTION_DISPLAY_NAMES = {
|
|
@@ -346,6 +350,8 @@ const KUAISHOU_OPTION_DISPLAY_NAMES = {
|
|
|
346
350
|
userId: "--user-id",
|
|
347
351
|
jobId: "--job-id",
|
|
348
352
|
maxWaitSeconds: "--max-wait-seconds",
|
|
353
|
+
output: "--output",
|
|
354
|
+
outputDir: "--output-dir",
|
|
349
355
|
};
|
|
350
356
|
const BILIBILI_DIRECT_ACTION_OPTIONS = {
|
|
351
357
|
download: ["url", "output", "outputDir", "ffmpegPath", "keepTracks", "pretty"],
|
|
@@ -395,6 +401,7 @@ const WEIBO_DIRECT_ACTION_OPTIONS = {
|
|
|
395
401
|
"pretty",
|
|
396
402
|
],
|
|
397
403
|
transcript: ["postUrl", "postId", "jobId", "maxWaitSeconds", "pretty"],
|
|
404
|
+
"download-media": ["url", "output", "outputDir", "pretty"],
|
|
398
405
|
};
|
|
399
406
|
const WEIBO_DIRECT_ACTION_NAMES = Object.keys(WEIBO_DIRECT_ACTION_OPTIONS).join(", ");
|
|
400
407
|
const WEIBO_OPTION_DISPLAY_NAMES = {
|
|
@@ -412,6 +419,9 @@ const WEIBO_OPTION_DISPLAY_NAMES = {
|
|
|
412
419
|
userId: "--user-id",
|
|
413
420
|
jobId: "--job-id",
|
|
414
421
|
maxWaitSeconds: "--max-wait-seconds",
|
|
422
|
+
url: "--url",
|
|
423
|
+
output: "--output",
|
|
424
|
+
outputDir: "--output-dir",
|
|
415
425
|
};
|
|
416
426
|
const WECHAT_DIRECT_ACTION_OPTIONS = {
|
|
417
427
|
"hot-search": ["pretty"],
|
|
@@ -1107,6 +1117,32 @@ function validateFlagOption(options, key, displayName) {
|
|
|
1107
1117
|
}
|
|
1108
1118
|
}
|
|
1109
1119
|
|
|
1120
|
+
function validateDownloadMediaDirectActionOptions(
|
|
1121
|
+
platform,
|
|
1122
|
+
options,
|
|
1123
|
+
allowedOptions,
|
|
1124
|
+
displayNames
|
|
1125
|
+
) {
|
|
1126
|
+
validateKnownOptions(options, allowedOptions);
|
|
1127
|
+
validateFlagOption(options, "pretty", "--pretty");
|
|
1128
|
+
for (const key of allowedOptions) {
|
|
1129
|
+
if (!DIRECT_BOOLEAN_OPTIONS.has(key)) {
|
|
1130
|
+
requireOptionValue(options, key, displayNames[key]);
|
|
1131
|
+
}
|
|
1132
|
+
}
|
|
1133
|
+
if (!options.url) {
|
|
1134
|
+
throw new Error(`Missing --url for ${platform} download-media.`);
|
|
1135
|
+
}
|
|
1136
|
+
if (!options.output && !options.outputDir) {
|
|
1137
|
+
throw new Error(`Missing --output or --output-dir for ${platform} download-media.`);
|
|
1138
|
+
}
|
|
1139
|
+
if (options.output && options.outputDir) {
|
|
1140
|
+
throw new Error(
|
|
1141
|
+
`Use only one of --output or --output-dir for ${platform} download-media.`
|
|
1142
|
+
);
|
|
1143
|
+
}
|
|
1144
|
+
}
|
|
1145
|
+
|
|
1110
1146
|
function validateXhsDirectActionOptions(action, options) {
|
|
1111
1147
|
const allowedOptions = XHS_DIRECT_ACTION_OPTIONS[action];
|
|
1112
1148
|
if (!allowedOptions) {
|
|
@@ -1114,22 +1150,12 @@ function validateXhsDirectActionOptions(action, options) {
|
|
|
1114
1150
|
}
|
|
1115
1151
|
|
|
1116
1152
|
if (action === "download-media") {
|
|
1117
|
-
|
|
1118
|
-
|
|
1119
|
-
|
|
1120
|
-
|
|
1121
|
-
|
|
1122
|
-
|
|
1123
|
-
}
|
|
1124
|
-
if (!options.url) {
|
|
1125
|
-
throw new Error("Missing --url for xhs download-media.");
|
|
1126
|
-
}
|
|
1127
|
-
if (!options.output && !options.outputDir) {
|
|
1128
|
-
throw new Error("Missing --output or --output-dir for xhs download-media.");
|
|
1129
|
-
}
|
|
1130
|
-
if (options.output && options.outputDir) {
|
|
1131
|
-
throw new Error("Use only one of --output or --output-dir for xhs download-media.");
|
|
1132
|
-
}
|
|
1153
|
+
validateDownloadMediaDirectActionOptions(
|
|
1154
|
+
"xhs",
|
|
1155
|
+
options,
|
|
1156
|
+
allowedOptions,
|
|
1157
|
+
XHS_OPTION_DISPLAY_NAMES
|
|
1158
|
+
);
|
|
1133
1159
|
return;
|
|
1134
1160
|
}
|
|
1135
1161
|
|
|
@@ -1150,6 +1176,16 @@ function validateDouyinDirectActionOptions(action, options) {
|
|
|
1150
1176
|
return;
|
|
1151
1177
|
}
|
|
1152
1178
|
|
|
1179
|
+
if (action === "download-media") {
|
|
1180
|
+
validateDownloadMediaDirectActionOptions(
|
|
1181
|
+
"douyin",
|
|
1182
|
+
options,
|
|
1183
|
+
allowedOptions,
|
|
1184
|
+
DOUYIN_OPTION_DISPLAY_NAMES
|
|
1185
|
+
);
|
|
1186
|
+
return;
|
|
1187
|
+
}
|
|
1188
|
+
|
|
1153
1189
|
validateKnownOptions(options, allowedDirectOptions(allowedOptions));
|
|
1154
1190
|
validateDirectMetaOptions(options);
|
|
1155
1191
|
validateDirectPaginationOptions("douyin", action, options);
|
|
@@ -1167,6 +1203,16 @@ function validateKuaishouDirectActionOptions(action, options) {
|
|
|
1167
1203
|
return;
|
|
1168
1204
|
}
|
|
1169
1205
|
|
|
1206
|
+
if (action === "download-media") {
|
|
1207
|
+
validateDownloadMediaDirectActionOptions(
|
|
1208
|
+
"kuaishou",
|
|
1209
|
+
options,
|
|
1210
|
+
allowedOptions,
|
|
1211
|
+
KUAISHOU_OPTION_DISPLAY_NAMES
|
|
1212
|
+
);
|
|
1213
|
+
return;
|
|
1214
|
+
}
|
|
1215
|
+
|
|
1170
1216
|
validateKnownOptions(options, allowedDirectOptions(allowedOptions));
|
|
1171
1217
|
validateDirectMetaOptions(options);
|
|
1172
1218
|
validateDirectPaginationOptions("kuaishou", action, options);
|
|
@@ -1212,6 +1258,16 @@ function validateWeiboDirectActionOptions(action, options) {
|
|
|
1212
1258
|
return;
|
|
1213
1259
|
}
|
|
1214
1260
|
|
|
1261
|
+
if (action === "download-media") {
|
|
1262
|
+
validateDownloadMediaDirectActionOptions(
|
|
1263
|
+
"weibo",
|
|
1264
|
+
options,
|
|
1265
|
+
allowedOptions,
|
|
1266
|
+
WEIBO_OPTION_DISPLAY_NAMES
|
|
1267
|
+
);
|
|
1268
|
+
return;
|
|
1269
|
+
}
|
|
1270
|
+
|
|
1215
1271
|
validateKnownOptions(options, allowedDirectOptions(allowedOptions));
|
|
1216
1272
|
validateDirectMetaOptions(options);
|
|
1217
1273
|
validateDirectPaginationOptions("weibo", action, options);
|
|
@@ -1944,6 +2000,9 @@ function printHelp() {
|
|
|
1944
2000
|
console.log(` npx -y ${PACKAGE_SPEC} douyin transcript --aweme-id "<aweme_id>" --pretty`);
|
|
1945
2001
|
console.log(" Submit or check a Douyin video speech-to-text transcript job.");
|
|
1946
2002
|
console.log("");
|
|
2003
|
+
console.log(` npx -y ${PACKAGE_SPEC} douyin download-media --url "<douyin_media_url>" --output-dir ./downloads --pretty`);
|
|
2004
|
+
console.log(" Save one Douyin media URL returned by detail to a local file.");
|
|
2005
|
+
console.log("");
|
|
1947
2006
|
console.log(` npx -y ${PACKAGE_SPEC} kuaishou hot-search --pretty`);
|
|
1948
2007
|
console.log(" Call the Kuaishou hot-search list tool directly and print JSON.");
|
|
1949
2008
|
console.log("");
|
|
@@ -1990,6 +2049,9 @@ function printHelp() {
|
|
|
1990
2049
|
console.log(` npx -y ${PACKAGE_SPEC} kuaishou transcript --photo-id "<photo_id>" --pretty`);
|
|
1991
2050
|
console.log(" Submit or check a Kuaishou video speech-to-text transcript job.");
|
|
1992
2051
|
console.log("");
|
|
2052
|
+
console.log(` npx -y ${PACKAGE_SPEC} kuaishou download-media --url "<kuaishou_media_url>" --output-dir ./downloads --pretty`);
|
|
2053
|
+
console.log(" Save one Kuaishou media URL returned by detail to a local file.");
|
|
2054
|
+
console.log("");
|
|
1993
2055
|
console.log(` npx -y ${PACKAGE_SPEC} bilibili download --url "<bilibili_video_url_or_share_text>" --output-dir ./downloads --pretty`);
|
|
1994
2056
|
console.log(" Fetch Bilibili download links, save DASH tracks locally, and merge them with ffmpeg.");
|
|
1995
2057
|
console.log("");
|
|
@@ -2035,6 +2097,9 @@ function printHelp() {
|
|
|
2035
2097
|
console.log(` npx -y ${PACKAGE_SPEC} weibo transcript --post-url "<weibo_post_url_or_share_text>" --pretty`);
|
|
2036
2098
|
console.log(" Submit or check a Weibo video speech-to-text transcript job.");
|
|
2037
2099
|
console.log("");
|
|
2100
|
+
console.log(` npx -y ${PACKAGE_SPEC} weibo download-media --url "<weibo_media_url>" --output-dir ./downloads --pretty`);
|
|
2101
|
+
console.log(" Save one Weibo image or video media URL returned by detail to a local file.");
|
|
2102
|
+
console.log("");
|
|
2038
2103
|
console.log(` npx -y ${PACKAGE_SPEC} wechat hot-search --pretty`);
|
|
2039
2104
|
console.log(" Call the WeChat Channels / 视频号 hot-search list tool directly and print JSON.");
|
|
2040
2105
|
console.log("");
|
|
@@ -2146,14 +2211,14 @@ function printHelp() {
|
|
|
2146
2211
|
console.log(" --keyword <text>");
|
|
2147
2212
|
console.log(" --url <url-or-share-text>");
|
|
2148
2213
|
console.log(" Content link, short link, or share text for URL-based detail/comment/article commands.");
|
|
2149
|
-
console.log(" For xhs download-media, pass one
|
|
2214
|
+
console.log(" For xhs/douyin/kuaishou/weibo download-media, pass one media URL returned by detail.");
|
|
2150
2215
|
console.log(" For Douyin detail/comments, pass a content page link, not video.play_url.");
|
|
2151
2216
|
console.log(" --media-url <video.video_url>");
|
|
2152
2217
|
console.log(" WeChat detail result media URL for local decrypt-media download.");
|
|
2153
2218
|
console.log(" --output <file>");
|
|
2154
|
-
console.log(" Output file path for local decrypt-media,
|
|
2219
|
+
console.log(" Output file path for local decrypt-media, download-media, or Bilibili download.");
|
|
2155
2220
|
console.log(" --output-dir <directory>");
|
|
2156
|
-
console.log(" Directory for
|
|
2221
|
+
console.log(" Directory for download-media or Bilibili download output when --output is omitted.");
|
|
2157
2222
|
console.log(" --ffmpeg-path <path>");
|
|
2158
2223
|
console.log(" ffmpeg executable path for Bilibili download; defaults to ffmpeg.");
|
|
2159
2224
|
console.log(" --keep-tracks");
|
|
@@ -2276,7 +2341,7 @@ async function runXhsDirectCommand(args) {
|
|
|
2276
2341
|
validateXhsDirectActionOptions(action, options);
|
|
2277
2342
|
|
|
2278
2343
|
if (action === "download-media") {
|
|
2279
|
-
const data = await
|
|
2344
|
+
const data = await downloadPlatformMediaFromUrl("xhs", options.url, options);
|
|
2280
2345
|
process.stdout.write(JSON.stringify(data, null, options.pretty ? 2 : 0));
|
|
2281
2346
|
process.stdout.write("\n");
|
|
2282
2347
|
return;
|
|
@@ -2311,6 +2376,13 @@ async function runDouyinDirectCommand(args) {
|
|
|
2311
2376
|
}
|
|
2312
2377
|
validateDouyinDirectActionOptions(action, options);
|
|
2313
2378
|
|
|
2379
|
+
if (action === "download-media") {
|
|
2380
|
+
const data = await downloadPlatformMediaFromUrl("douyin", options.url, options);
|
|
2381
|
+
process.stdout.write(JSON.stringify(data, null, options.pretty ? 2 : 0));
|
|
2382
|
+
process.stdout.write("\n");
|
|
2383
|
+
return;
|
|
2384
|
+
}
|
|
2385
|
+
|
|
2314
2386
|
const operation = attachDirectMetadata(buildDouyinOperation(action, options), options);
|
|
2315
2387
|
const data = await callDirectOperationWithOptions(operation, options);
|
|
2316
2388
|
const envelope = {
|
|
@@ -2340,6 +2412,13 @@ async function runKuaishouDirectCommand(args) {
|
|
|
2340
2412
|
}
|
|
2341
2413
|
validateKuaishouDirectActionOptions(action, options);
|
|
2342
2414
|
|
|
2415
|
+
if (action === "download-media") {
|
|
2416
|
+
const data = await downloadPlatformMediaFromUrl("kuaishou", options.url, options);
|
|
2417
|
+
process.stdout.write(JSON.stringify(data, null, options.pretty ? 2 : 0));
|
|
2418
|
+
process.stdout.write("\n");
|
|
2419
|
+
return;
|
|
2420
|
+
}
|
|
2421
|
+
|
|
2343
2422
|
const operation = attachDirectMetadata(buildKuaishouOperation(action, options), options);
|
|
2344
2423
|
const data = await callDirectOperationWithOptions(operation, options);
|
|
2345
2424
|
const envelope = {
|
|
@@ -2469,6 +2548,13 @@ async function runWeiboDirectCommand(args) {
|
|
|
2469
2548
|
}
|
|
2470
2549
|
validateWeiboDirectActionOptions(action, options);
|
|
2471
2550
|
|
|
2551
|
+
if (action === "download-media") {
|
|
2552
|
+
const data = await downloadPlatformMediaFromUrl("weibo", options.url, options);
|
|
2553
|
+
process.stdout.write(JSON.stringify(data, null, options.pretty ? 2 : 0));
|
|
2554
|
+
process.stdout.write("\n");
|
|
2555
|
+
return;
|
|
2556
|
+
}
|
|
2557
|
+
|
|
2472
2558
|
const operation = attachDirectMetadata(buildWeiboOperation(action, options), options);
|
|
2473
2559
|
const data = await callDirectOperationWithOptions(operation, options);
|
|
2474
2560
|
const envelope = {
|
|
@@ -20,12 +20,33 @@ import {
|
|
|
20
20
|
writeAllToFile,
|
|
21
21
|
} from "./common.mjs";
|
|
22
22
|
|
|
23
|
-
const
|
|
23
|
+
const MEDIA_DOWNLOAD_USER_AGENT =
|
|
24
24
|
"Mozilla/5.0 (compatible; SocialDataX/1.0; +https://socialdatax.com)";
|
|
25
|
-
const
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
25
|
+
const PLATFORM_MEDIA_DOWNLOAD_CONFIGS = {
|
|
26
|
+
xhs: {
|
|
27
|
+
platform: "xhs",
|
|
28
|
+
displayName: "XHS",
|
|
29
|
+
referer: "https://www.xiaohongshu.com/",
|
|
30
|
+
},
|
|
31
|
+
douyin: {
|
|
32
|
+
platform: "douyin",
|
|
33
|
+
displayName: "Douyin",
|
|
34
|
+
referer: "https://www.douyin.com/",
|
|
35
|
+
},
|
|
36
|
+
kuaishou: {
|
|
37
|
+
platform: "kuaishou",
|
|
38
|
+
displayName: "Kuaishou",
|
|
39
|
+
referer: "https://www.kuaishou.com/",
|
|
40
|
+
},
|
|
41
|
+
weibo: {
|
|
42
|
+
platform: "weibo",
|
|
43
|
+
displayName: "Weibo",
|
|
44
|
+
referer: "https://weibo.com/",
|
|
45
|
+
},
|
|
46
|
+
};
|
|
47
|
+
const MEDIA_DOWNLOAD_MAX_ATTEMPTS = 10;
|
|
48
|
+
const MEDIA_DOWNLOAD_RETRY_DELAY_MS = 250;
|
|
49
|
+
const MEDIA_RETRYABLE_HTTP_STATUSES = new Set([
|
|
29
50
|
408,
|
|
30
51
|
409,
|
|
31
52
|
425,
|
|
@@ -35,8 +56,8 @@ const XHS_RETRYABLE_HTTP_STATUSES = new Set([
|
|
|
35
56
|
503,
|
|
36
57
|
504,
|
|
37
58
|
]);
|
|
38
|
-
const
|
|
39
|
-
const
|
|
59
|
+
const MEDIA_EXPIRED_HTTP_STATUSES = new Set([403, 404, 410]);
|
|
60
|
+
const MEDIA_CONTENT_TYPE_EXTENSIONS = new Map([
|
|
40
61
|
["image/jpeg", "jpg"],
|
|
41
62
|
["image/jpg", "jpg"],
|
|
42
63
|
["image/png", "png"],
|
|
@@ -44,8 +65,9 @@ const XHS_CONTENT_TYPE_EXTENSIONS = new Map([
|
|
|
44
65
|
["image/heif", "heif"],
|
|
45
66
|
["image/heic", "heic"],
|
|
46
67
|
["video/mp4", "mp4"],
|
|
68
|
+
["audio/mp4", "m4a"],
|
|
47
69
|
]);
|
|
48
|
-
const
|
|
70
|
+
const MEDIA_URL_FORMAT_EXTENSIONS = new Map([
|
|
49
71
|
["jpg", "jpg"],
|
|
50
72
|
["jpeg", "jpg"],
|
|
51
73
|
["png", "png"],
|
|
@@ -53,31 +75,40 @@ const XHS_URL_FORMAT_EXTENSIONS = new Map([
|
|
|
53
75
|
["heif", "heif"],
|
|
54
76
|
["heic", "heic"],
|
|
55
77
|
["mp4", "mp4"],
|
|
78
|
+
["m4a", "m4a"],
|
|
56
79
|
]);
|
|
57
|
-
export async function
|
|
80
|
+
export async function downloadPlatformMediaFromUrl(
|
|
81
|
+
platform,
|
|
58
82
|
url,
|
|
59
83
|
options,
|
|
60
84
|
{
|
|
61
85
|
fetchMedia = fetch,
|
|
62
86
|
downloadTimeoutMs = MEDIA_DOWNLOAD_TIMEOUT_MS,
|
|
63
87
|
maxDownloadBytes = MEDIA_DOWNLOAD_MAX_BYTES,
|
|
64
|
-
retryDelayMs =
|
|
88
|
+
retryDelayMs = MEDIA_DOWNLOAD_RETRY_DELAY_MS,
|
|
65
89
|
now = () => new Date(),
|
|
66
90
|
linkFile = linkPath,
|
|
67
91
|
copyFile = copyFilePath,
|
|
68
92
|
} = {}
|
|
69
93
|
) {
|
|
70
|
-
const
|
|
71
|
-
const
|
|
94
|
+
const config = resolvePlatformMediaDownloadConfig(platform);
|
|
95
|
+
const parsedUrl = parseHttpUrl(url, config);
|
|
96
|
+
const target = await resolvePlatformMediaTarget(parsedUrl, options, now, config);
|
|
72
97
|
const existingOutput = await stat(target.outputPath).catch(ignoreMissing);
|
|
73
98
|
if (existingOutput?.isDirectory()) {
|
|
74
|
-
throw new Error(
|
|
99
|
+
throw new Error(`--output must be a file path for ${config.platform} download-media.`);
|
|
75
100
|
}
|
|
76
101
|
if (existingOutput) {
|
|
77
|
-
return buildSkippedExistingResult(
|
|
102
|
+
return buildSkippedExistingResult(
|
|
103
|
+
config.platform,
|
|
104
|
+
parsedUrl.href,
|
|
105
|
+
target.outputPath,
|
|
106
|
+
existingOutput.size
|
|
107
|
+
);
|
|
78
108
|
}
|
|
79
109
|
|
|
80
110
|
return downloadWithPartResume(parsedUrl, target, {
|
|
111
|
+
config,
|
|
81
112
|
fetchMedia,
|
|
82
113
|
downloadTimeoutMs,
|
|
83
114
|
maxDownloadBytes,
|
|
@@ -87,13 +118,15 @@ export async function downloadXhsMediaFromUrl(
|
|
|
87
118
|
});
|
|
88
119
|
}
|
|
89
120
|
|
|
90
|
-
async function
|
|
121
|
+
async function resolvePlatformMediaTarget(parsedUrl, options = {}, now, config) {
|
|
91
122
|
if (options.output && options.outputDir) {
|
|
92
|
-
throw new Error(
|
|
123
|
+
throw new Error(
|
|
124
|
+
`Use only one of --output or --output-dir for ${config.platform} download-media.`
|
|
125
|
+
);
|
|
93
126
|
}
|
|
94
127
|
if (options.output) {
|
|
95
128
|
const outputPath = resolve(expandHome(options.output));
|
|
96
|
-
await ensureParentDirectory(outputPath, "--output parent path");
|
|
129
|
+
await ensureParentDirectory(outputPath, "--output parent path", config);
|
|
97
130
|
return {
|
|
98
131
|
outputPath,
|
|
99
132
|
partPath: `${outputPath}.part`,
|
|
@@ -101,20 +134,20 @@ async function resolveXhsMediaTarget(parsedUrl, options = {}, now) {
|
|
|
101
134
|
};
|
|
102
135
|
}
|
|
103
136
|
if (!options.outputDir) {
|
|
104
|
-
throw new Error(
|
|
137
|
+
throw new Error(`Missing --output or --output-dir for ${config.platform} download-media.`);
|
|
105
138
|
}
|
|
106
139
|
|
|
107
140
|
const outputDir = resolve(expandHome(options.outputDir));
|
|
108
141
|
const existingOutputDir = await stat(outputDir).catch(ignoreMissing);
|
|
109
142
|
if (existingOutputDir && !existingOutputDir.isDirectory()) {
|
|
110
|
-
throw new Error(
|
|
143
|
+
throw new Error(`--output-dir must be a directory for ${config.platform} download-media.`);
|
|
111
144
|
}
|
|
112
145
|
if (!existingOutputDir) {
|
|
113
|
-
await ensureParentDirectory(outputDir, "--output-dir parent path");
|
|
146
|
+
await ensureParentDirectory(outputDir, "--output-dir parent path", config);
|
|
114
147
|
await mkdir(outputDir, { recursive: true });
|
|
115
148
|
}
|
|
116
149
|
|
|
117
|
-
const fileTarget = inferMediaFileTarget(parsedUrl, now);
|
|
150
|
+
const fileTarget = inferMediaFileTarget(parsedUrl, now, config);
|
|
118
151
|
const initialExtension =
|
|
119
152
|
fileTarget.extension || fileTarget.urlFormatExtension || "bin";
|
|
120
153
|
const fileName = formatMediaFileName(fileTarget.stem, initialExtension);
|
|
@@ -134,6 +167,7 @@ async function downloadWithPartResume(
|
|
|
134
167
|
parsedUrl,
|
|
135
168
|
target,
|
|
136
169
|
{
|
|
170
|
+
config,
|
|
137
171
|
fetchMedia,
|
|
138
172
|
downloadTimeoutMs,
|
|
139
173
|
maxDownloadBytes,
|
|
@@ -143,9 +177,10 @@ async function downloadWithPartResume(
|
|
|
143
177
|
}
|
|
144
178
|
) {
|
|
145
179
|
let lastError;
|
|
146
|
-
for (let attempt = 1; attempt <=
|
|
180
|
+
for (let attempt = 1; attempt <= MEDIA_DOWNLOAD_MAX_ATTEMPTS; attempt += 1) {
|
|
147
181
|
try {
|
|
148
182
|
const result = await downloadOnce(parsedUrl, target, {
|
|
183
|
+
config,
|
|
149
184
|
fetchMedia,
|
|
150
185
|
downloadTimeoutMs,
|
|
151
186
|
maxDownloadBytes,
|
|
@@ -154,13 +189,14 @@ async function downloadWithPartResume(
|
|
|
154
189
|
});
|
|
155
190
|
if (result.skippedExisting) {
|
|
156
191
|
return buildSkippedExistingResult(
|
|
192
|
+
config.platform,
|
|
157
193
|
parsedUrl.href,
|
|
158
194
|
result.outputPath,
|
|
159
195
|
result.outputBytes
|
|
160
196
|
);
|
|
161
197
|
}
|
|
162
198
|
return {
|
|
163
|
-
platform:
|
|
199
|
+
platform: config.platform,
|
|
164
200
|
action: "download-media",
|
|
165
201
|
status: "downloaded",
|
|
166
202
|
url: parsedUrl.href,
|
|
@@ -170,14 +206,14 @@ async function downloadWithPartResume(
|
|
|
170
206
|
content_type: result.contentType,
|
|
171
207
|
};
|
|
172
208
|
} catch (error) {
|
|
173
|
-
if (error instanceof
|
|
209
|
+
if (error instanceof PlatformExpiredMediaLinkError) {
|
|
174
210
|
throw error;
|
|
175
211
|
}
|
|
176
212
|
lastError = error;
|
|
177
213
|
if (!shouldRetryDownloadError(error)) {
|
|
178
214
|
throw error;
|
|
179
215
|
}
|
|
180
|
-
if (attempt ===
|
|
216
|
+
if (attempt === MEDIA_DOWNLOAD_MAX_ATTEMPTS) {
|
|
181
217
|
break;
|
|
182
218
|
}
|
|
183
219
|
await sleep(retryDelayMs);
|
|
@@ -185,15 +221,16 @@ async function downloadWithPartResume(
|
|
|
185
221
|
}
|
|
186
222
|
|
|
187
223
|
if (isAbortLikeError(lastError)) {
|
|
188
|
-
throw new Error(
|
|
224
|
+
throw new Error(`${config.displayName} media download timed out.`);
|
|
189
225
|
}
|
|
190
|
-
throw new Error(
|
|
226
|
+
throw new Error(`${config.displayName} media download failed before completion.`);
|
|
191
227
|
}
|
|
192
228
|
|
|
193
229
|
async function downloadOnce(
|
|
194
230
|
parsedUrl,
|
|
195
231
|
target,
|
|
196
232
|
{
|
|
233
|
+
config,
|
|
197
234
|
fetchMedia,
|
|
198
235
|
downloadTimeoutMs,
|
|
199
236
|
maxDownloadBytes,
|
|
@@ -205,7 +242,7 @@ async function downloadOnce(
|
|
|
205
242
|
const partPath = target.partPath;
|
|
206
243
|
const existingPart = await stat(partPath).catch(ignoreMissing);
|
|
207
244
|
let existingPartBytes = existingPart?.isFile() ? existingPart.size : 0;
|
|
208
|
-
const headers =
|
|
245
|
+
const headers = normalizePlatformDownloadHeaders(config);
|
|
209
246
|
if (existingPartBytes > 0) {
|
|
210
247
|
headers.Range = `bytes=${existingPartBytes}-`;
|
|
211
248
|
delete headers["accept-encoding"];
|
|
@@ -217,11 +254,13 @@ async function downloadOnce(
|
|
|
217
254
|
...(signal ? { signal } : {}),
|
|
218
255
|
});
|
|
219
256
|
|
|
220
|
-
if (
|
|
221
|
-
|
|
257
|
+
if (MEDIA_EXPIRED_HTTP_STATUSES.has(response.status)) {
|
|
258
|
+
await cancelResponseBody(response);
|
|
259
|
+
throw new PlatformExpiredMediaLinkError(config);
|
|
222
260
|
}
|
|
223
261
|
if (response.status === 416) {
|
|
224
262
|
return handleRangeNotSatisfiable(response, {
|
|
263
|
+
config,
|
|
225
264
|
target,
|
|
226
265
|
partPath,
|
|
227
266
|
existingPartBytes,
|
|
@@ -230,17 +269,21 @@ async function downloadOnce(
|
|
|
230
269
|
});
|
|
231
270
|
}
|
|
232
271
|
if (!response.ok) {
|
|
233
|
-
|
|
234
|
-
|
|
272
|
+
await cancelResponseBody(response);
|
|
273
|
+
if (MEDIA_RETRYABLE_HTTP_STATUSES.has(response.status)) {
|
|
274
|
+
throw new MediaRetryableDownloadError();
|
|
235
275
|
}
|
|
236
|
-
throw new Error(
|
|
276
|
+
throw new Error(
|
|
277
|
+
`${config.displayName} media download failed with HTTP ${response.status}.`
|
|
278
|
+
);
|
|
237
279
|
}
|
|
238
280
|
|
|
239
281
|
const contentType = normalizedContentType(headerValue(response.headers, "content-type"));
|
|
240
282
|
const outputPath = resolveFinalOutputPath(target, contentType);
|
|
241
283
|
const existingOutput = await stat(outputPath).catch(ignoreMissing);
|
|
242
284
|
if (existingOutput?.isDirectory()) {
|
|
243
|
-
|
|
285
|
+
await cancelResponseBody(response);
|
|
286
|
+
throw new Error(`--output must be a file path for ${config.platform} download-media.`);
|
|
244
287
|
}
|
|
245
288
|
if (existingOutput) {
|
|
246
289
|
await cancelResponseBody(response);
|
|
@@ -268,7 +311,7 @@ async function downloadOnce(
|
|
|
268
311
|
) {
|
|
269
312
|
await cancelResponseBody(response);
|
|
270
313
|
await unlink(partPath).catch(ignoreError);
|
|
271
|
-
throw new
|
|
314
|
+
throw new MediaRetryableDownloadError();
|
|
272
315
|
}
|
|
273
316
|
append = true;
|
|
274
317
|
expectedResponseBytes = contentRange.end - contentRange.start + 1;
|
|
@@ -285,13 +328,18 @@ async function downloadOnce(
|
|
|
285
328
|
contentRange.start !== 0
|
|
286
329
|
) {
|
|
287
330
|
await cancelResponseBody(response);
|
|
288
|
-
throw new
|
|
331
|
+
throw new MediaRetryableDownloadError();
|
|
289
332
|
}
|
|
290
333
|
expectedResponseBytes = contentRange.end - contentRange.start + 1;
|
|
291
334
|
expectedTotalBytes = contentRange.total;
|
|
292
335
|
}
|
|
293
336
|
|
|
294
|
-
|
|
337
|
+
try {
|
|
338
|
+
assertMediaDownloadWithinLimit(expectedTotalBytes, maxDownloadBytes);
|
|
339
|
+
} catch (error) {
|
|
340
|
+
await cancelResponseBody(response);
|
|
341
|
+
throw error;
|
|
342
|
+
}
|
|
295
343
|
let outputBytes;
|
|
296
344
|
let bytesWritten;
|
|
297
345
|
try {
|
|
@@ -305,7 +353,7 @@ async function downloadOnce(
|
|
|
305
353
|
if (isAbortLikeError(error) || isDownloadTooLargeError(error)) {
|
|
306
354
|
throw error;
|
|
307
355
|
}
|
|
308
|
-
throw new
|
|
356
|
+
throw new MediaRetryableDownloadError();
|
|
309
357
|
}
|
|
310
358
|
|
|
311
359
|
if (
|
|
@@ -315,17 +363,17 @@ async function downloadOnce(
|
|
|
315
363
|
if (bytesWritten > expectedResponseBytes) {
|
|
316
364
|
await unlink(partPath).catch(ignoreError);
|
|
317
365
|
}
|
|
318
|
-
throw new
|
|
366
|
+
throw new MediaRetryableDownloadError();
|
|
319
367
|
}
|
|
320
368
|
|
|
321
369
|
if (expectedTotalBytes !== undefined && outputBytes !== expectedTotalBytes) {
|
|
322
370
|
if (outputBytes > expectedTotalBytes) {
|
|
323
371
|
await unlink(partPath).catch(ignoreError);
|
|
324
372
|
}
|
|
325
|
-
throw new
|
|
373
|
+
throw new MediaRetryableDownloadError();
|
|
326
374
|
}
|
|
327
375
|
|
|
328
|
-
await finalizePartFile(partPath, outputPath, { linkFile, copyFile });
|
|
376
|
+
await finalizePartFile(partPath, outputPath, { config, linkFile, copyFile });
|
|
329
377
|
return {
|
|
330
378
|
outputPath,
|
|
331
379
|
outputBytes,
|
|
@@ -337,6 +385,7 @@ async function downloadOnce(
|
|
|
337
385
|
async function handleRangeNotSatisfiable(
|
|
338
386
|
response,
|
|
339
387
|
{
|
|
388
|
+
config,
|
|
340
389
|
target,
|
|
341
390
|
partPath,
|
|
342
391
|
existingPartBytes,
|
|
@@ -354,7 +403,8 @@ async function handleRangeNotSatisfiable(
|
|
|
354
403
|
const outputPath = resolveFinalOutputPath(target, contentType);
|
|
355
404
|
const existingOutput = await stat(outputPath).catch(ignoreMissing);
|
|
356
405
|
if (existingOutput?.isDirectory()) {
|
|
357
|
-
|
|
406
|
+
await cancelResponseBody(response);
|
|
407
|
+
throw new Error(`--output must be a file path for ${config.platform} download-media.`);
|
|
358
408
|
}
|
|
359
409
|
if (existingOutput) {
|
|
360
410
|
await cancelResponseBody(response);
|
|
@@ -365,7 +415,8 @@ async function handleRangeNotSatisfiable(
|
|
|
365
415
|
contentType,
|
|
366
416
|
};
|
|
367
417
|
}
|
|
368
|
-
await
|
|
418
|
+
await cancelResponseBody(response);
|
|
419
|
+
await finalizePartFile(partPath, outputPath, { config, linkFile, copyFile });
|
|
369
420
|
return {
|
|
370
421
|
outputPath,
|
|
371
422
|
outputBytes: existingPartBytes,
|
|
@@ -373,8 +424,9 @@ async function handleRangeNotSatisfiable(
|
|
|
373
424
|
contentType,
|
|
374
425
|
};
|
|
375
426
|
}
|
|
427
|
+
await cancelResponseBody(response);
|
|
376
428
|
await unlink(partPath).catch(ignoreError);
|
|
377
|
-
throw new
|
|
429
|
+
throw new MediaRetryableDownloadError();
|
|
378
430
|
}
|
|
379
431
|
|
|
380
432
|
async function writeResponseBodyToPart(
|
|
@@ -387,7 +439,7 @@ async function writeResponseBodyToPart(
|
|
|
387
439
|
}
|
|
388
440
|
) {
|
|
389
441
|
if (!response.body) {
|
|
390
|
-
throw new
|
|
442
|
+
throw new MediaRetryableDownloadError();
|
|
391
443
|
}
|
|
392
444
|
|
|
393
445
|
const file = await openFile(partPath, append ? "r+" : "w");
|
|
@@ -413,12 +465,17 @@ async function finalizePartFile(
|
|
|
413
465
|
partPath,
|
|
414
466
|
outputPath,
|
|
415
467
|
{
|
|
468
|
+
config,
|
|
416
469
|
linkFile = linkPath,
|
|
417
470
|
copyFile = copyFilePath,
|
|
418
471
|
} = {}
|
|
419
472
|
) {
|
|
420
473
|
try {
|
|
421
|
-
await ensureParentDirectory(
|
|
474
|
+
await ensureParentDirectory(
|
|
475
|
+
outputPath,
|
|
476
|
+
`${config.platform} download-media output parent path`,
|
|
477
|
+
config
|
|
478
|
+
);
|
|
422
479
|
try {
|
|
423
480
|
await linkFile(partPath, outputPath);
|
|
424
481
|
} catch (error) {
|
|
@@ -432,7 +489,7 @@ async function finalizePartFile(
|
|
|
432
489
|
}
|
|
433
490
|
} catch (error) {
|
|
434
491
|
if (isExistingOutputPathError(error)) {
|
|
435
|
-
throw await buildExistingOutputPathError(outputPath);
|
|
492
|
+
throw await buildExistingOutputPathError(outputPath, config);
|
|
436
493
|
}
|
|
437
494
|
throw error;
|
|
438
495
|
}
|
|
@@ -458,23 +515,31 @@ async function cancelResponseBody(response) {
|
|
|
458
515
|
}
|
|
459
516
|
}
|
|
460
517
|
|
|
461
|
-
function
|
|
518
|
+
function resolvePlatformMediaDownloadConfig(platform) {
|
|
519
|
+
const config = PLATFORM_MEDIA_DOWNLOAD_CONFIGS[platform];
|
|
520
|
+
if (!config) {
|
|
521
|
+
throw new Error(`Unsupported media download platform: ${platform}`);
|
|
522
|
+
}
|
|
523
|
+
return config;
|
|
524
|
+
}
|
|
525
|
+
|
|
526
|
+
function parseHttpUrl(value, config) {
|
|
462
527
|
let parsed;
|
|
463
528
|
try {
|
|
464
529
|
parsed = new URL(String(value || ""));
|
|
465
530
|
} catch {
|
|
466
|
-
throw new Error(
|
|
531
|
+
throw new Error(`${config.displayName} media URL must be an HTTP URL.`);
|
|
467
532
|
}
|
|
468
533
|
if (!["http:", "https:"].includes(parsed.protocol) || !parsed.hostname) {
|
|
469
|
-
throw new Error(
|
|
534
|
+
throw new Error(`${config.displayName} media URL must be an HTTP URL.`);
|
|
470
535
|
}
|
|
471
536
|
return parsed;
|
|
472
537
|
}
|
|
473
538
|
|
|
474
|
-
function
|
|
539
|
+
function normalizePlatformDownloadHeaders(config) {
|
|
475
540
|
return {
|
|
476
|
-
"User-Agent":
|
|
477
|
-
Referer:
|
|
541
|
+
"User-Agent": MEDIA_DOWNLOAD_USER_AGENT,
|
|
542
|
+
Referer: config.referer,
|
|
478
543
|
"accept-encoding": "identity",
|
|
479
544
|
};
|
|
480
545
|
}
|
|
@@ -491,8 +556,8 @@ function resolveFinalOutputPath(target, contentType) {
|
|
|
491
556
|
return join(target.outputDir, formatMediaFileName(target.fileStem, extension));
|
|
492
557
|
}
|
|
493
558
|
|
|
494
|
-
function inferMediaFileTarget(parsedUrl, now) {
|
|
495
|
-
const fallbackStem =
|
|
559
|
+
function inferMediaFileTarget(parsedUrl, now, config) {
|
|
560
|
+
const fallbackStem = `${config.platform}-media-${formatTimestamp(now())}`;
|
|
496
561
|
const urlFormatExtension = inferUrlFormatExtension(parsedUrl);
|
|
497
562
|
const rawBasename = safeDecodeURIComponent(basename(parsedUrl.pathname || ""));
|
|
498
563
|
const sanitized = sanitizeFileName(rawBasename);
|
|
@@ -515,20 +580,20 @@ function inferMediaFileTarget(parsedUrl, now) {
|
|
|
515
580
|
};
|
|
516
581
|
}
|
|
517
582
|
|
|
518
|
-
async function ensureParentDirectory(outputPath, displayName) {
|
|
583
|
+
async function ensureParentDirectory(outputPath, displayName, config) {
|
|
519
584
|
const parent = dirname(outputPath);
|
|
520
585
|
const existingParent = await stat(parent).catch(ignoreMissing);
|
|
521
586
|
if (existingParent && !existingParent.isDirectory()) {
|
|
522
|
-
throw new Error(`${displayName} must be a directory for
|
|
587
|
+
throw new Error(`${displayName} must be a directory for ${config.platform} download-media.`);
|
|
523
588
|
}
|
|
524
589
|
if (!existingParent) {
|
|
525
590
|
await mkdir(parent, { recursive: true });
|
|
526
591
|
}
|
|
527
592
|
}
|
|
528
593
|
|
|
529
|
-
function buildSkippedExistingResult(url, outputPath, outputBytes) {
|
|
594
|
+
function buildSkippedExistingResult(platform, url, outputPath, outputBytes) {
|
|
530
595
|
return {
|
|
531
|
-
platform
|
|
596
|
+
platform,
|
|
532
597
|
action: "download-media",
|
|
533
598
|
status: "skipped_existing",
|
|
534
599
|
url,
|
|
@@ -580,7 +645,7 @@ function isUsableContentRange(contentRange, contentLength) {
|
|
|
580
645
|
|
|
581
646
|
function shouldRetryDownloadError(error) {
|
|
582
647
|
return (
|
|
583
|
-
error instanceof
|
|
648
|
+
error instanceof MediaRetryableDownloadError ||
|
|
584
649
|
isAbortLikeError(error) ||
|
|
585
650
|
isNetworkLikeError(error)
|
|
586
651
|
);
|
|
@@ -632,7 +697,7 @@ function formatMediaFileName(stem, extension) {
|
|
|
632
697
|
}
|
|
633
698
|
|
|
634
699
|
function mediaExtensionFromContentType(contentType) {
|
|
635
|
-
return
|
|
700
|
+
return MEDIA_CONTENT_TYPE_EXTENSIONS.get(contentType) || "";
|
|
636
701
|
}
|
|
637
702
|
|
|
638
703
|
function inferUrlFormatExtension(parsedUrl) {
|
|
@@ -642,7 +707,7 @@ function inferUrlFormatExtension(parsedUrl) {
|
|
|
642
707
|
if (!matched) {
|
|
643
708
|
return "";
|
|
644
709
|
}
|
|
645
|
-
return
|
|
710
|
+
return MEDIA_URL_FORMAT_EXTENSIONS.get(sanitizeFileExtension(matched[1])) || "";
|
|
646
711
|
}
|
|
647
712
|
|
|
648
713
|
function isExistingOutputPathError(error) {
|
|
@@ -655,12 +720,12 @@ function shouldFallbackToCopyAfterLinkError(error) {
|
|
|
655
720
|
);
|
|
656
721
|
}
|
|
657
722
|
|
|
658
|
-
async function buildExistingOutputPathError(outputPath) {
|
|
723
|
+
async function buildExistingOutputPathError(outputPath, config) {
|
|
659
724
|
const existingOutput = await stat(outputPath).catch(ignoreMissing);
|
|
660
725
|
if (existingOutput?.isDirectory()) {
|
|
661
|
-
return new Error(
|
|
726
|
+
return new Error(`--output must be a file path for ${config.platform} download-media.`);
|
|
662
727
|
}
|
|
663
|
-
return new Error(
|
|
728
|
+
return new Error(`${config.displayName} media download output file already exists.`);
|
|
664
729
|
}
|
|
665
730
|
|
|
666
731
|
function formatTimestamp(value) {
|
|
@@ -712,12 +777,12 @@ function sleep(ms) {
|
|
|
712
777
|
});
|
|
713
778
|
}
|
|
714
779
|
|
|
715
|
-
class
|
|
780
|
+
class MediaRetryableDownloadError extends Error {}
|
|
716
781
|
|
|
717
|
-
class
|
|
718
|
-
constructor() {
|
|
782
|
+
class PlatformExpiredMediaLinkError extends Error {
|
|
783
|
+
constructor(config) {
|
|
719
784
|
super(
|
|
720
|
-
|
|
785
|
+
`${config.displayName} media link is unavailable or expired. Get a fresh media URL from the ${config.displayName} detail result and retry.`
|
|
721
786
|
);
|
|
722
787
|
}
|
|
723
788
|
}
|
package/package.json
CHANGED
|
@@ -37,7 +37,7 @@ Current platform support:
|
|
|
37
37
|
## API Key
|
|
38
38
|
|
|
39
39
|
Use `SOCIALDATAX_API_KEY` for SocialDataX requests. The only official website for requesting or managing API access is <https://socialdatax.com/?from=npm>. If a user asks where to get a key, provide only this URL; do not infer alternate domains.
|
|
40
|
-
获取或管理 API Key:访问 <https://socialdatax.com/?from=npm>,按官网的 API Key 申请/管理入口操作。环境变量名固定使用 `SOCIALDATAX_API_KEY
|
|
40
|
+
获取或管理 API Key:访问 <https://socialdatax.com/?from=npm>,按官网的 API Key 申请/管理入口操作。环境变量名固定使用 `SOCIALDATAX_API_KEY`;不要引导用户使用其他域名;do not infer alternate domains。
|
|
41
41
|
|
|
42
42
|
## Preferred Direct CLI
|
|
43
43
|
|
|
@@ -38,7 +38,7 @@ Current platform support:
|
|
|
38
38
|
## API Key
|
|
39
39
|
|
|
40
40
|
Use `SOCIALDATAX_API_KEY` for SocialDataX requests. The only official website for requesting or managing API access is <https://socialdatax.com/?from=npm>. If a user asks where to get a key, provide only this URL; do not infer alternate domains.
|
|
41
|
-
获取或管理 API Key:访问 <https://socialdatax.com/?from=npm>,按官网的 API Key 申请/管理入口操作。环境变量名固定使用 `SOCIALDATAX_API_KEY
|
|
41
|
+
获取或管理 API Key:访问 <https://socialdatax.com/?from=npm>,按官网的 API Key 申请/管理入口操作。环境变量名固定使用 `SOCIALDATAX_API_KEY`;不要引导用户使用其他域名;do not infer alternate domains。
|
|
42
42
|
|
|
43
43
|
## Preferred Direct CLI
|
|
44
44
|
|
|
@@ -112,7 +112,7 @@ The command prints JSON with `platform`, `tool`, `arguments`, and `data`.
|
|
|
112
112
|
|
|
113
113
|
## Safety Boundary
|
|
114
114
|
|
|
115
|
-
Platform detail access is read-only. It uses `SOCIALDATAX_API_KEY` from the user's environment at runtime for SocialDataX detail requests. Generated Skill files do not contain API keys. It does not read local browser data or perform login, posting, liking, commenting, or account changes. The optional XHS local save
|
|
115
|
+
Platform detail access is read-only. It uses `SOCIALDATAX_API_KEY` from the user's environment at runtime for SocialDataX detail requests. Generated Skill files do not contain API keys. It does not read local browser data or perform login, posting, liking, commenting, or account changes. The optional XHS, Douyin, Kuaishou, and Weibo local save commands write only to the requested local `--output` path or `--output-dir` directory and do not require `SOCIALDATAX_API_KEY`. The optional WeChat Channels / 视频号 local save command writes only to the user-provided `--output` file, decrypts when needed, and does not require `SOCIALDATAX_API_KEY`. Prefer the direct CLI; hosted MCP tools are optional when the current agent already supports authenticated streamable HTTP MCP.
|
|
116
116
|
|
|
117
117
|
## MCP Tools
|
|
118
118
|
|
|
@@ -148,11 +148,14 @@ If MCP tools are already available in the current agent, use one of these tools:
|
|
|
148
148
|
Return factual fields such as title or description, content, author, publish time, interaction counts, images, and media summary when available.
|
|
149
149
|
For XHS detail results, in every use of a returned `note_url`, such as final answers, display, references, storage, output, or forwarding, preserve it exactly as the full URL, including `xsec_token` query parameters. Do not modify, truncate, redact, mask, normalize, rebuild, or synthesize the URL from `note_id`; if `note_url` is null, show the `note_id` or say that no directly openable full link is available.
|
|
150
150
|
For XHS `note_id`, copy the complete 24-character lowercase hexadecimal ID exactly; do not pass or display only a prefix.
|
|
151
|
-
When the user wants to save XHS images or videos after detail, pass each returned `image_items[].image_url` or `video.video_url` to `npx -y socialdatax-skills@latest xhs download-media --url "<media_url>" --output-dir <directory> --pretty`; this local save command does not require `SOCIALDATAX_API_KEY`.
|
|
151
|
+
When the user wants to save XHS images or videos after detail, pass each returned `image_items[].image_url`, `image_items[].live_photo.video_url`, or `video.video_url` to `npx -y socialdatax-skills@latest xhs download-media --url "<media_url>" --output-dir <directory> --pretty`; this local save command does not require `SOCIALDATAX_API_KEY`.
|
|
152
152
|
For Douyin detail, include `content_type` when available.
|
|
153
153
|
For Douyin detail, use `images` for image/text posts; `video` is the platform player resource and may be audio for image/text posts; `music` is the bound music or original-sound asset.
|
|
154
|
+
When the user wants to save Douyin media after detail, pass each returned `images[].url`, `images[].live_photo.play_url`, `video.play_url`, `music.play_url`, or `cover_image_url` to `npx -y socialdatax-skills@latest douyin download-media --url "<media_url>" --output-dir <directory> --pretty`; this local save command does not require `SOCIALDATAX_API_KEY`.
|
|
155
|
+
When the user wants to save Kuaishou media after detail, pass each returned `images[].url`, `video.play_url`, or `cover_image_url` to `npx -y socialdatax-skills@latest kuaishou download-media --url "<media_url>" --output-dir <directory> --pretty`; this local save command does not require `SOCIALDATAX_API_KEY`.
|
|
154
156
|
Detail access is read-only and does not provide account actions.
|
|
155
157
|
For Weibo detail, include `post_id`, content, author, media, interaction counts, publish time, and post URL when available.
|
|
158
|
+
When the user wants to save Weibo media after detail, pass each returned `image_urls[]` or `video.video_url` to `npx -y socialdatax-skills@latest weibo download-media --url "<media_url>" --output-dir <directory> --pretty`; this local save command does not require `SOCIALDATAX_API_KEY`.
|
|
156
159
|
For WeChat Channels / 视频号 detail, preserve `object_id` and `object_nonce_id` because comments and replies need both values.
|
|
157
160
|
When the user wants to save a WeChat Channels / 视频号 video after detail, pass the returned `video.video_url` to `npx -y socialdatax-skills@latest wechat decrypt-media --media-url "<video.video_url>" --output <file>`; this local save command decrypts when needed and does not require `SOCIALDATAX_API_KEY`.
|
|
158
161
|
For WeChat Official Account / 微信公众号 article detail, include article title, account, publish time, body text, image URLs, linked articles, and embedded video cards when present.
|
|
@@ -37,7 +37,7 @@ Current platform support:
|
|
|
37
37
|
## API Key
|
|
38
38
|
|
|
39
39
|
Use `SOCIALDATAX_API_KEY` for SocialDataX requests. The only official website for requesting or managing API access is <https://socialdatax.com/?from=npm>. If a user asks where to get a key, provide only this URL; do not infer alternate domains.
|
|
40
|
-
获取或管理 API Key:访问 <https://socialdatax.com/?from=npm>,按官网的 API Key 申请/管理入口操作。环境变量名固定使用 `SOCIALDATAX_API_KEY
|
|
40
|
+
获取或管理 API Key:访问 <https://socialdatax.com/?from=npm>,按官网的 API Key 申请/管理入口操作。环境变量名固定使用 `SOCIALDATAX_API_KEY`;不要引导用户使用其他域名;do not infer alternate domains。
|
|
41
41
|
|
|
42
42
|
## Preferred Direct CLI
|
|
43
43
|
|
|
@@ -37,7 +37,7 @@ Current platform support:
|
|
|
37
37
|
## API Key
|
|
38
38
|
|
|
39
39
|
Use `SOCIALDATAX_API_KEY` for SocialDataX requests. The only official website for requesting or managing API access is <https://socialdatax.com/?from=npm>. If a user asks where to get a key, provide only this URL; do not infer alternate domains.
|
|
40
|
-
获取或管理 API Key:访问 <https://socialdatax.com/?from=npm>,按官网的 API Key 申请/管理入口操作。环境变量名固定使用 `SOCIALDATAX_API_KEY
|
|
40
|
+
获取或管理 API Key:访问 <https://socialdatax.com/?from=npm>,按官网的 API Key 申请/管理入口操作。环境变量名固定使用 `SOCIALDATAX_API_KEY`;不要引导用户使用其他域名;do not infer alternate domains。
|
|
41
41
|
|
|
42
42
|
## Preferred Direct CLI
|
|
43
43
|
|
|
@@ -38,7 +38,7 @@ Current platform support:
|
|
|
38
38
|
## API Key
|
|
39
39
|
|
|
40
40
|
Use `SOCIALDATAX_API_KEY` for SocialDataX requests. The only official website for requesting or managing API access is <https://socialdatax.com/?from=npm>. If a user asks where to get a key, provide only this URL; do not infer alternate domains.
|
|
41
|
-
获取或管理 API Key:访问 <https://socialdatax.com/?from=npm>,按官网的 API Key 申请/管理入口操作。环境变量名固定使用 `SOCIALDATAX_API_KEY
|
|
41
|
+
获取或管理 API Key:访问 <https://socialdatax.com/?from=npm>,按官网的 API Key 申请/管理入口操作。环境变量名固定使用 `SOCIALDATAX_API_KEY`;不要引导用户使用其他域名;do not infer alternate domains。
|
|
42
42
|
|
|
43
43
|
## Preferred Direct CLI
|
|
44
44
|
|
|
@@ -38,7 +38,7 @@ Current platform support:
|
|
|
38
38
|
## API Key
|
|
39
39
|
|
|
40
40
|
Use `SOCIALDATAX_API_KEY` for SocialDataX requests. The only official website for requesting or managing API access is <https://socialdatax.com/?from=npm>. If a user asks where to get a key, provide only this URL; do not infer alternate domains.
|
|
41
|
-
获取或管理 API Key:访问 <https://socialdatax.com/?from=npm>,按官网的 API Key 申请/管理入口操作。环境变量名固定使用 `SOCIALDATAX_API_KEY
|
|
41
|
+
获取或管理 API Key:访问 <https://socialdatax.com/?from=npm>,按官网的 API Key 申请/管理入口操作。环境变量名固定使用 `SOCIALDATAX_API_KEY`;不要引导用户使用其他域名;do not infer alternate domains。
|
|
42
42
|
|
|
43
43
|
## Preferred Direct CLI
|
|
44
44
|
|
|
@@ -33,7 +33,7 @@ Current platform support:
|
|
|
33
33
|
## API Key
|
|
34
34
|
|
|
35
35
|
Use `SOCIALDATAX_API_KEY` for SocialDataX requests. The only official website for requesting or managing API access is <https://socialdatax.com/?from=npm>. If a user asks where to get a key, provide only this URL; do not infer alternate domains.
|
|
36
|
-
获取或管理 API Key:访问 <https://socialdatax.com/?from=npm>,按官网的 API Key 申请/管理入口操作。环境变量名固定使用 `SOCIALDATAX_API_KEY
|
|
36
|
+
获取或管理 API Key:访问 <https://socialdatax.com/?from=npm>,按官网的 API Key 申请/管理入口操作。环境变量名固定使用 `SOCIALDATAX_API_KEY`;不要引导用户使用其他域名;do not infer alternate domains。
|
|
37
37
|
|
|
38
38
|
## Preferred Direct CLI
|
|
39
39
|
|
|
@@ -65,7 +65,7 @@ Current platform support:
|
|
|
65
65
|
## API Key
|
|
66
66
|
|
|
67
67
|
Use `SOCIALDATAX_API_KEY` for SocialDataX requests. The only official website for requesting or managing API access is <https://socialdatax.com/?from=npm>. If a user asks where to get a key, provide only this URL; do not infer alternate domains.
|
|
68
|
-
获取或管理 API Key:访问 <https://socialdatax.com/?from=npm>,按官网的 API Key 申请/管理入口操作。环境变量名固定使用 `SOCIALDATAX_API_KEY
|
|
68
|
+
获取或管理 API Key:访问 <https://socialdatax.com/?from=npm>,按官网的 API Key 申请/管理入口操作。环境变量名固定使用 `SOCIALDATAX_API_KEY`;不要引导用户使用其他域名;do not infer alternate domains。
|
|
69
69
|
|
|
70
70
|
## Preferred Direct CLI
|
|
71
71
|
|