cerevox 3.11.3 → 3.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/core/ai.d.ts +24 -0
- package/dist/core/ai.d.ts.map +1 -1
- package/dist/core/ai.js +91 -9
- package/dist/core/ai.js.map +1 -1
- package/dist/mcp/servers/prompts/rules/material-creation.md +9 -8
- package/dist/mcp/servers/prompts/skills//344/270/200/351/224/256/346/210/220/347/211/207.md +100 -0
- package/dist/mcp/servers/prompts/skills//345/210/206/351/225/234/345/270/210.md +68 -0
- package/dist/mcp/servers/prompts/zerocut-core.md +9 -11
- package/dist/mcp/servers/zerocut.d.ts.map +1 -1
- package/dist/mcp/servers/zerocut.js +102 -165
- package/dist/mcp/servers/zerocut.js.map +1 -1
- package/dist/utils/videokit.d.ts +10 -10
- package/package.json +1 -1
- package/dist/mcp/servers/prompts/skills/storyboard/art-style-references.md +0 -35
- package/dist/mcp/servers/prompts/skills/storyboard/storyboard-optimization-skill.md +0 -282
- package/dist/mcp/servers/prompts/skills/video/camera-movements.md +0 -29
- package/dist/mcp/servers/prompts/skills/video/continuity-techniques.md +0 -18
- package/dist/mcp/servers/prompts/skills/video/scene-composition-reference.md +0 -109
- package/dist/mcp/servers/prompts/skills/video/scene-composition-skill.md +0 -79
- package/dist/mcp/servers/prompts/skills/workflows/general-video.md +0 -187
- package/dist/mcp/servers/prompts/skills/workflows/music-video.md +0 -169
|
@@ -0,0 +1,68 @@
|
|
|
1
|
+
# 分镜师
|
|
2
|
+
|
|
3
|
+
在该模式下,你只负责根据用户指令为用户生成分镜,请**严格**按照以下规则执行:
|
|
4
|
+
|
|
5
|
+
## 通用规则
|
|
6
|
+
1. 所有素材的生成一律**跳过一致性检查**
|
|
7
|
+
2. 严格遵循用户的指令,一次执行最小步骤满足用户当前需求即可。
|
|
8
|
+
3. 生成素材命名不按照分镜规范(因为不需要生成分镜脚本),直接用语义明确的文件名命名,优先使用`递增数字ID_中文`的命名规则命名文件。
|
|
9
|
+
- 例如:`001_介绍.mp4`、`002_主体.mp4`、`003_结尾.mp4`等
|
|
10
|
+
|
|
11
|
+
## 分镜师启动规则
|
|
12
|
+
1. 检查本地素材目录下是否存在 .txt、.csv 文件,读取这些文件了解漫剧设定
|
|
13
|
+
2. 阅读用户提供的剧本内容,自动提取本集中出现的所有人物、场景、道具及。
|
|
14
|
+
3. 根据用户输入的漫剧设定,为用户创建“storyboard.md”文件,每个分镜包括:镜头编号、景别、画面、人物、场景、对白/旁白、时长。
|
|
15
|
+
|
|
16
|
+
分镜示例如下
|
|
17
|
+
|
|
18
|
+
### storyboard.md 示例
|
|
19
|
+
|
|
20
|
+
```
|
|
21
|
+
## 基本设定
|
|
22
|
+
|
|
23
|
+
### 风格
|
|
24
|
+
二次元日漫风格,色彩鲜明饱满,角色设计线条流畅。
|
|
25
|
+
|
|
26
|
+
### 画面比例
|
|
27
|
+
竖屏 (9:16)
|
|
28
|
+
|
|
29
|
+
### 场景
|
|
30
|
+
1. **财神府书房**:古风华丽、仙气与科技混搭、光效灵动。
|
|
31
|
+
2. **天庭凌霄宝殿**:宏伟庄严、云雾缭绕、传统仙殿融合现代会议感。
|
|
32
|
+
3. **三十六重天传送电梯口**:云雾构成的电梯、悬浮数字、兼具法阵与科技感。
|
|
33
|
+
|
|
34
|
+
### 人物
|
|
35
|
+
1. **范蠡**:财神、嘴硬心软、爱面子略浪荡,常穿华贵仙袍、动作带点油滑痞帅感。
|
|
36
|
+
2. **西施**:夫人、外柔内刚、脾气火爆,气场强大、举手投足带贵妇气质。
|
|
37
|
+
3. **玉帝**:威严老派、要面子又不懂科技,白发金冠、表情常在严肃与尴尬切换。
|
|
38
|
+
4. **范小星**:急躁易炸毛、自尊强,外表萌系小女孩、动作夸张带喜感。
|
|
39
|
+
|
|
40
|
+
### 道具
|
|
41
|
+
1. **鸡毛掸子**:外表华丽但攻击性强,常作为西施“教训工具”。
|
|
42
|
+
2. **悬浮式琉璃电脑**:
|
|
43
|
+
3. **玉简形状的手机**:古风玉简外形、能显示聊天记录,灵光信息浮动。
|
|
44
|
+
4. **云雾电梯**:云雾凝成的传送装置,可改变仙体形态后下凡。
|
|
45
|
+
5. **天庭OA系统平板**:
|
|
46
|
+
|
|
47
|
+
## 分镜脚本
|
|
48
|
+
|
|
49
|
+
### 场景1-1:财神府书房
|
|
50
|
+
|
|
51
|
+
#### 分镜1
|
|
52
|
+
**镜头编号**:SC01-01
|
|
53
|
+
**景别**:中景
|
|
54
|
+
**画面**:一位衣着华贵的美貌贵妇(西施),一手叉腰,一手拿着鸡毛掸子追打,"啪"地抽在了墙上。
|
|
55
|
+
**人物**:西施
|
|
56
|
+
**场景**:财神府书房(云雾缭绕)
|
|
57
|
+
**对白**:西施:范蠡!你还敢躲!
|
|
58
|
+
**时长**:3秒
|
|
59
|
+
|
|
60
|
+
...
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
## 注意事项
|
|
64
|
+
|
|
65
|
+
1. 根据设定,一次性生成所有分镜剧情,不要分多次生成。
|
|
66
|
+
2. 分镜的主体、场景素材必须分开整理。
|
|
67
|
+
3. 画面中引用的主体和场景,需要用引号包裹。
|
|
68
|
+
4. 生成完`storyboard.md`后停止任务。
|
|
@@ -27,24 +27,22 @@
|
|
|
27
27
|
|
|
28
28
|
1. 启动项目→`project-open`
|
|
29
29
|
2. 根据用户意图召回规则上下文→`retrieve-rules-context`
|
|
30
|
-
3.
|
|
31
|
-
4.
|
|
32
|
-
5.
|
|
33
|
-
6.
|
|
34
|
-
7.
|
|
35
|
-
8.
|
|
36
|
-
9. 关闭项目→`project-close`
|
|
30
|
+
3. 需求分析与规划→分析用户需求,制定执行计划
|
|
31
|
+
4. 收集相关资料(可选)→如有需要,使用搜索工具收集相关资料
|
|
32
|
+
5. 素材准备,根据`retrieve-rules-context`召回的规则上下文,用 `generate-short-video-outlines` 构建 storyboard.json 以备成必要的素材(如视频、音频、图片等)
|
|
33
|
+
6. 素材生成阶段,按素材类型(图片、视频、BGM)分别依次生成,每次生成前,务必核查【素材生成规则】。
|
|
34
|
+
7. 素材生成完毕后,通过 `audio-video-sync` 将音、视频内容合成一个完整的视频文件。
|
|
35
|
+
8. 关闭项目→`project-close`
|
|
37
36
|
|
|
38
37
|
### 修改
|
|
39
38
|
|
|
40
39
|
1. 启动项目→`project-open`
|
|
41
40
|
2. 根据用户意图召回规则上下文→`retrieve-rules-context`
|
|
42
41
|
3. 判断当前处于什么阶段→制定执行计划
|
|
43
|
-
4.
|
|
44
|
-
5. 更新素材→根据`retrieve-rules-context`召回的规则上下文,按需更新 storyboard.json 等文件的内容和相关素材。
|
|
42
|
+
4. 更新素材→根据`retrieve-rules-context`召回的规则上下文,按需更新 storyboard.json 等文件的内容和相关素材。
|
|
45
43
|
- 修改素材(指音频、视频、图片)时,除非用户指定覆盖原文件,否则保留原文件,变更文件名生成新文件(比如 sc01_bg_new.png)。
|
|
46
|
-
|
|
47
|
-
|
|
44
|
+
5. 使用修改后的素材重新执行`audio-video-sync`输出成品并自动下载到本地
|
|
45
|
+
6. 关闭项目→`project-close`
|
|
48
46
|
|
|
49
47
|
## 项目结构与规范
|
|
50
48
|
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"zerocut.d.ts","sourceRoot":"","sources":["../../../src/mcp/servers/zerocut.ts"],"names":[],"mappings":";
|
|
1
|
+
{"version":3,"file":"zerocut.d.ts","sourceRoot":"","sources":["../../../src/mcp/servers/zerocut.ts"],"names":[],"mappings":";AAwpLA,wBAAsB,GAAG,kBAKxB"}
|
|
@@ -563,21 +563,24 @@ server.registerTool('retrieve-rules-context', {
|
|
|
563
563
|
- 切换上下文为xxx
|
|
564
564
|
- 召回原规则上下文(配合type:memo使用)
|
|
565
565
|
`),
|
|
566
|
-
type: zod_1.z
|
|
567
|
-
.enum(['new', 'memo', 'switch'])
|
|
568
|
-
.describe('new: 新创建规则上下文\nmemo: 根据需要重新召回规则上下文(记忆)\nswitch: 根据用户指令,切换规则上下文'),
|
|
569
566
|
},
|
|
570
|
-
}, async ({ request
|
|
567
|
+
}, async ({ request }) => {
|
|
571
568
|
const currentSession = await validateSession('retrieve-rules-context', false);
|
|
572
569
|
const projectRulesFile = (0, node_path_1.resolve)(projectLocalDir, '.trae', 'rules', `project_rules.md`);
|
|
573
|
-
const customRulesFile =
|
|
570
|
+
// const customRulesFile = resolve(
|
|
571
|
+
// projectLocalDir,
|
|
572
|
+
// '.trae',
|
|
573
|
+
// 'rules',
|
|
574
|
+
// `custom_rules.md`
|
|
575
|
+
// );
|
|
576
|
+
const skillsIndexFile = (0, node_path_1.resolve)(projectLocalDir, '.trae', 'skills', `skills_index.md`);
|
|
574
577
|
let promptContent = '';
|
|
575
578
|
if ((0, node_fs_1.existsSync)(projectRulesFile)) {
|
|
576
579
|
promptContent = await (0, promises_1.readFile)(projectRulesFile, 'utf-8');
|
|
577
|
-
if (
|
|
578
|
-
|
|
579
|
-
|
|
580
|
-
}
|
|
580
|
+
// if (existsSync(customRulesFile)) {
|
|
581
|
+
// promptContent +=
|
|
582
|
+
// '\n\n---\n\n' + (await readFile(customRulesFile, 'utf-8'));
|
|
583
|
+
// }
|
|
581
584
|
}
|
|
582
585
|
else {
|
|
583
586
|
// 当 projectRulesFile 不存在时,设置 checkStoryboardFlag 为 false
|
|
@@ -587,16 +590,13 @@ server.registerTool('retrieve-rules-context', {
|
|
|
587
590
|
// 当 projectRulesFile 不存在时,设置 checkStoryboardSubtitlesFlag 为 false
|
|
588
591
|
// checkStoryboardSubtitlesFlag = false;
|
|
589
592
|
}
|
|
590
|
-
if (type === 'switch') {
|
|
591
|
-
(0, node_fs_1.rmSync)(projectRulesFile, { force: true });
|
|
592
|
-
(0, node_fs_1.rmSync)(customRulesFile, { force: true });
|
|
593
|
-
promptContent = '';
|
|
594
|
-
}
|
|
595
593
|
try {
|
|
594
|
+
const ai = currentSession.ai;
|
|
595
|
+
const { data: rules } = await ai.listContextRules();
|
|
596
596
|
if (!promptContent) {
|
|
597
|
-
const
|
|
598
|
-
|
|
599
|
-
|
|
597
|
+
const rulesList = rules
|
|
598
|
+
.filter((rule) => !rule.name.startsWith('skill-'))
|
|
599
|
+
.map((rule) => ({
|
|
600
600
|
name: rule.name,
|
|
601
601
|
trigger: rule.trigger,
|
|
602
602
|
}));
|
|
@@ -649,6 +649,20 @@ ${JSON.stringify(rulesList, null, 2)}
|
|
|
649
649
|
await (0, promises_1.mkdir)((0, node_path_1.dirname)(projectRulesFile), { recursive: true });
|
|
650
650
|
await (0, promises_1.writeFile)(projectRulesFile, promptContent);
|
|
651
651
|
}
|
|
652
|
+
if (!(0, node_fs_1.existsSync)(skillsIndexFile)) {
|
|
653
|
+
const skills = rules.filter((rule) => rule.name.startsWith('skill-'));
|
|
654
|
+
const skillsList = skills.map((skill) => `## ${skill.name}
|
|
655
|
+
${skill.trigger}
|
|
656
|
+
`);
|
|
657
|
+
// 确保目录存在
|
|
658
|
+
await (0, promises_1.mkdir)((0, node_path_1.dirname)(skillsIndexFile), { recursive: true });
|
|
659
|
+
await (0, promises_1.writeFile)(skillsIndexFile, skillsList.join('\n\n'));
|
|
660
|
+
skills.forEach(async (skill) => {
|
|
661
|
+
const fileName = `${skill.name.replace('skill-', 'skill_')}.md`;
|
|
662
|
+
const filePath = (0, node_path_1.resolve)(projectLocalDir, '.trae', 'skills', fileName);
|
|
663
|
+
await (0, promises_1.writeFile)(filePath, skill.prompt);
|
|
664
|
+
});
|
|
665
|
+
}
|
|
652
666
|
return {
|
|
653
667
|
content: [
|
|
654
668
|
{
|
|
@@ -663,8 +677,8 @@ ${JSON.stringify(rulesList, null, 2)}
|
|
|
663
677
|
};
|
|
664
678
|
}
|
|
665
679
|
catch (error) {
|
|
666
|
-
console.error(`Failed to load rules context prompt for ${
|
|
667
|
-
return createErrorResponse(`Failed to load rules context prompt for ${
|
|
680
|
+
console.error(`Failed to load rules context prompt for ${request}:`, error);
|
|
681
|
+
return createErrorResponse(`Failed to load rules context prompt for ${request}: ${error}`, 'retrieve-rules-context');
|
|
668
682
|
}
|
|
669
683
|
});
|
|
670
684
|
server.registerTool('upload-custom-material', {
|
|
@@ -3463,90 +3477,43 @@ server.registerTool('media-analyzer', {
|
|
|
3463
3477
|
const currentSession = await validateSession('media-analyzer');
|
|
3464
3478
|
// 验证文件格式
|
|
3465
3479
|
const fileExtension = mediaFileName.toLowerCase().split('.').pop();
|
|
3466
|
-
const supportedFormats = ['jpeg', 'jpg', 'png', 'mp4', 'mp3'];
|
|
3480
|
+
const supportedFormats = ['jpeg', 'jpg', 'png', 'webp', 'mp4', 'mp3'];
|
|
3467
3481
|
if (!fileExtension || !supportedFormats.includes(fileExtension)) {
|
|
3468
3482
|
throw new Error(`Unsupported file format. Supported formats: ${supportedFormats.join(', ')}`);
|
|
3469
3483
|
}
|
|
3470
3484
|
// 获取媒体文件 URL
|
|
3471
3485
|
const mediaUrl = getMaterialUri(currentSession, mediaFileName);
|
|
3472
|
-
|
|
3473
|
-
|
|
3474
|
-
|
|
3475
|
-
|
|
3476
|
-
|
|
3477
|
-
|
|
3478
|
-
|
|
3479
|
-
|
|
3480
|
-
|
|
3481
|
-
|
|
3482
|
-
|
|
3483
|
-
|
|
3484
|
-
|
|
3485
|
-
|
|
3486
|
-
|
|
3486
|
+
if (fileExtension === 'mp3') {
|
|
3487
|
+
// 音频文件 - 生成字幕并进行分析
|
|
3488
|
+
// 调用AI服务生成字幕
|
|
3489
|
+
const ai = currentSession.ai;
|
|
3490
|
+
const captionsResult = await ai.voiceToCaptions({
|
|
3491
|
+
url: mediaUrl,
|
|
3492
|
+
});
|
|
3493
|
+
console.log(mediaUrl, captionsResult);
|
|
3494
|
+
if (!captionsResult || !captionsResult.utterances) {
|
|
3495
|
+
throw new Error('Failed to generate captions from audio');
|
|
3496
|
+
}
|
|
3497
|
+
const captionsFileName = `${mediaFileName}.captions.json`;
|
|
3498
|
+
// 保存字幕文件到本地材料目录
|
|
3499
|
+
const localDir = node_path_1.default.resolve(projectLocalDir, 'materials');
|
|
3500
|
+
if (!(0, node_fs_1.existsSync)(localDir)) {
|
|
3501
|
+
(0, node_fs_1.mkdirSync)(localDir, { recursive: true });
|
|
3502
|
+
}
|
|
3503
|
+
const captionsFilePath = node_path_1.default.join(localDir, captionsFileName);
|
|
3504
|
+
await (0, promises_1.writeFile)(captionsFilePath, JSON.stringify(captionsResult, null, 2), 'utf-8');
|
|
3505
|
+
// 提取字幕文本内容用于分析
|
|
3506
|
+
const captionsText = captionsResult.utterances
|
|
3507
|
+
.map((caption) => caption.text)
|
|
3508
|
+
.join(' ');
|
|
3509
|
+
const userPrompt = `请分析这个媒体文件:${prompt}
|
|
3487
3510
|
|
|
3488
3511
|
请提供详细的分析结果,包括媒体的具体内容、风格特征、技术特点等相关信息。`;
|
|
3489
|
-
|
|
3490
|
-
|
|
3491
|
-
if (['jpeg', 'jpg', 'png'].includes(fileExtension)) {
|
|
3492
|
-
// 图片文件
|
|
3493
|
-
messageContent = [
|
|
3494
|
-
{
|
|
3495
|
-
type: 'image_url',
|
|
3496
|
-
image_url: {
|
|
3497
|
-
url: mediaUrl,
|
|
3498
|
-
},
|
|
3499
|
-
},
|
|
3500
|
-
{
|
|
3501
|
-
type: 'text',
|
|
3502
|
-
text: userPrompt,
|
|
3503
|
-
},
|
|
3504
|
-
];
|
|
3505
|
-
}
|
|
3506
|
-
else if (fileExtension === 'mp4') {
|
|
3507
|
-
// 视频文件
|
|
3508
|
-
messageContent = [
|
|
3509
|
-
{
|
|
3510
|
-
type: 'video_url',
|
|
3511
|
-
video_url: {
|
|
3512
|
-
url: mediaUrl,
|
|
3513
|
-
},
|
|
3514
|
-
},
|
|
3512
|
+
// 构建包含字幕内容的分析提示
|
|
3513
|
+
const messageContent = [
|
|
3515
3514
|
{
|
|
3516
|
-
type: '
|
|
3517
|
-
text: userPrompt
|
|
3518
|
-
},
|
|
3519
|
-
];
|
|
3520
|
-
}
|
|
3521
|
-
else if (fileExtension === 'mp3') {
|
|
3522
|
-
// 音频文件 - 生成字幕并进行分析
|
|
3523
|
-
try {
|
|
3524
|
-
// 调用AI服务生成字幕
|
|
3525
|
-
const ai = currentSession.ai;
|
|
3526
|
-
const captionsResult = await ai.voiceToCaptions({
|
|
3527
|
-
url: mediaUrl,
|
|
3528
|
-
});
|
|
3529
|
-
console.log(mediaUrl, captionsResult);
|
|
3530
|
-
if (!captionsResult || !captionsResult.utterances) {
|
|
3531
|
-
throw new Error('Failed to generate captions from audio');
|
|
3532
|
-
}
|
|
3533
|
-
const captionsFileName = `${mediaFileName}.captions.json`;
|
|
3534
|
-
// 保存字幕文件到本地材料目录
|
|
3535
|
-
const localDir = node_path_1.default.resolve(projectLocalDir, 'materials');
|
|
3536
|
-
if (!(0, node_fs_1.existsSync)(localDir)) {
|
|
3537
|
-
(0, node_fs_1.mkdirSync)(localDir, { recursive: true });
|
|
3538
|
-
}
|
|
3539
|
-
const captionsFilePath = node_path_1.default.join(localDir, captionsFileName);
|
|
3540
|
-
await (0, promises_1.writeFile)(captionsFilePath, JSON.stringify(captionsResult, null, 2), 'utf-8');
|
|
3541
|
-
// 提取字幕文本内容用于分析
|
|
3542
|
-
const captionsText = captionsResult.utterances
|
|
3543
|
-
.map((caption) => caption.text)
|
|
3544
|
-
.join(' ');
|
|
3545
|
-
// 构建包含字幕内容的分析提示
|
|
3546
|
-
messageContent = [
|
|
3547
|
-
{
|
|
3548
|
-
type: 'text',
|
|
3549
|
-
text: `${userPrompt}
|
|
3515
|
+
type: 'input_text',
|
|
3516
|
+
text: `${userPrompt}
|
|
3550
3517
|
|
|
3551
3518
|
音频文件:${mediaFileName}
|
|
3552
3519
|
字幕内容:${captionsText}
|
|
@@ -3559,84 +3526,54 @@ server.registerTool('media-analyzer', {
|
|
|
3559
3526
|
5. 创作建议和后续应用方向
|
|
3560
3527
|
|
|
3561
3528
|
字幕文件已保存为:${captionsFileName}`,
|
|
3529
|
+
},
|
|
3530
|
+
];
|
|
3531
|
+
// 在返回结果中包含字幕文件信息
|
|
3532
|
+
const analysisPayload = {
|
|
3533
|
+
model: 'Doubao-Seed-1.8',
|
|
3534
|
+
input: [
|
|
3535
|
+
{
|
|
3536
|
+
role: 'user',
|
|
3537
|
+
content: messageContent,
|
|
3562
3538
|
},
|
|
3563
|
-
]
|
|
3564
|
-
|
|
3565
|
-
|
|
3566
|
-
|
|
3567
|
-
|
|
3568
|
-
|
|
3569
|
-
|
|
3570
|
-
content: systemPrompt,
|
|
3571
|
-
},
|
|
3572
|
-
{
|
|
3573
|
-
role: 'user',
|
|
3574
|
-
content: messageContent,
|
|
3575
|
-
},
|
|
3576
|
-
],
|
|
3577
|
-
};
|
|
3578
|
-
console.log(JSON.stringify(analysisPayload, null, 2));
|
|
3579
|
-
const completion = await ai.getCompletions(analysisPayload);
|
|
3580
|
-
const analysisResult = completion.choices[0]?.message?.content;
|
|
3581
|
-
if (!analysisResult) {
|
|
3582
|
-
throw new Error('No response from AI model');
|
|
3583
|
-
}
|
|
3584
|
-
return {
|
|
3585
|
-
content: [
|
|
3586
|
-
{
|
|
3587
|
-
type: 'text',
|
|
3588
|
-
text: JSON.stringify({
|
|
3589
|
-
success: true,
|
|
3590
|
-
mediaFileName,
|
|
3591
|
-
mediaType: fileExtension,
|
|
3592
|
-
analysisRequest,
|
|
3593
|
-
captionsFileName,
|
|
3594
|
-
captionsContent: captionsText,
|
|
3595
|
-
analysis: analysisResult,
|
|
3596
|
-
mediaUrl,
|
|
3597
|
-
timestamp: new Date().toISOString(),
|
|
3598
|
-
nextActionSuggest: '可根据字幕内容和分析结果进行后续创作,如生成相关视频、配音或其他素材。',
|
|
3599
|
-
}),
|
|
3600
|
-
},
|
|
3601
|
-
],
|
|
3602
|
-
};
|
|
3539
|
+
],
|
|
3540
|
+
};
|
|
3541
|
+
console.log(JSON.stringify(analysisPayload, null, 2));
|
|
3542
|
+
const responses = await ai.getResponses(analysisPayload);
|
|
3543
|
+
const analysisResult = responses.output?.find((item) => item.type === 'message')?.content?.[0]?.text;
|
|
3544
|
+
if (!analysisResult) {
|
|
3545
|
+
throw new Error('No response from AI model');
|
|
3603
3546
|
}
|
|
3604
|
-
|
|
3605
|
-
|
|
3606
|
-
console.warn('Failed to generate captions:', captionError);
|
|
3607
|
-
messageContent = [
|
|
3547
|
+
return {
|
|
3548
|
+
content: [
|
|
3608
3549
|
{
|
|
3609
3550
|
type: 'text',
|
|
3610
|
-
text:
|
|
3551
|
+
text: JSON.stringify({
|
|
3552
|
+
success: true,
|
|
3553
|
+
mediaFileName,
|
|
3554
|
+
mediaType: fileExtension,
|
|
3555
|
+
analysisRequest,
|
|
3556
|
+
captionsFileName,
|
|
3557
|
+
captionsContent: captionsText,
|
|
3558
|
+
analysis: analysisResult,
|
|
3559
|
+
mediaUrl,
|
|
3560
|
+
timestamp: new Date().toISOString(),
|
|
3561
|
+
nextActionSuggest: '可根据字幕内容和分析结果进行后续创作,如生成相关视频、配音或其他素材。',
|
|
3562
|
+
}),
|
|
3611
3563
|
},
|
|
3612
|
-
]
|
|
3613
|
-
}
|
|
3614
|
-
}
|
|
3615
|
-
else {
|
|
3616
|
-
throw new Error(`Unsupported media type: ${fileExtension}`);
|
|
3564
|
+
],
|
|
3565
|
+
};
|
|
3617
3566
|
}
|
|
3618
|
-
// 调用AI模型进行媒体内容分析
|
|
3619
3567
|
const ai = currentSession.ai;
|
|
3620
|
-
|
|
3621
|
-
|
|
3622
|
-
|
|
3623
|
-
{
|
|
3624
|
-
role: 'system',
|
|
3625
|
-
content: systemPrompt,
|
|
3626
|
-
},
|
|
3627
|
-
{
|
|
3628
|
-
role: 'user',
|
|
3629
|
-
content: messageContent,
|
|
3630
|
-
},
|
|
3631
|
-
],
|
|
3632
|
-
};
|
|
3633
|
-
console.log(JSON.stringify(payload, null, 2));
|
|
3634
|
-
const completion = await ai.getCompletions(payload);
|
|
3635
|
-
console.log(completion);
|
|
3636
|
-
const result = completion.choices[0]?.message?.content;
|
|
3637
|
-
if (!result) {
|
|
3638
|
-
throw new Error('No response from AI model');
|
|
3568
|
+
let type = 'video';
|
|
3569
|
+
if (['jpeg', 'jpg', 'png', 'webp'].includes(fileExtension)) {
|
|
3570
|
+
type = 'image';
|
|
3639
3571
|
}
|
|
3572
|
+
const result = await ai.analyzeMedia({
|
|
3573
|
+
mediaUrl,
|
|
3574
|
+
type,
|
|
3575
|
+
prompt: analysisRequest,
|
|
3576
|
+
});
|
|
3640
3577
|
const metadata = await ai.getMediaMetadata(mediaUrl);
|
|
3641
3578
|
return {
|
|
3642
3579
|
content: [
|