@ash_j1997/ffmpeg-mcp-server 0.4.2 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +34 -22
- package/dist/index.js +283 -91
- package/dist/tools/audio.js +24 -9
- package/dist/tools/burn_subtitles.js +92 -0
- package/dist/tools/compress.js +8 -4
- package/dist/tools/concat.js +14 -3
- package/dist/tools/convert.js +77 -42
- package/dist/tools/keyframes.js +75 -0
- package/dist/tools/normalize.js +75 -0
- package/dist/tools/overlay.js +115 -0
- package/dist/tools/probe.js +6 -2
- package/dist/tools/reformat.js +13 -7
- package/dist/tools/silence.js +81 -20
- package/dist/tools/speed.js +24 -3
- package/dist/tools/split.js +15 -9
- package/dist/tools/subtitles.js +5 -2
- package/dist/tools/thumbnail.js +24 -13
- package/dist/tools/trim.js +36 -14
- package/dist/tools/waveform.js +3 -3
- package/dist/utils/exec.js +139 -8
- package/dist/utils/hwaccel.js +148 -26
- package/package.json +2 -1
package/README.md
CHANGED
|
@@ -4,7 +4,7 @@
|
|
|
4
4
|
|
|
5
5
|
A Model Context Protocol (MCP) server that gives AI tools access to FFmpeg and FFprobe to process video and audio locally.
|
|
6
6
|
|
|
7
|
-
Supports
|
|
7
|
+
Supports cross-platform GPU hardware encoding (Apple Silicon, NVIDIA NVENC, Intel QuickSync, AMD AMF), live progress updates via MCP progress tokens, instant cancellation via `AbortSignal`, and inlines image previews directly for multimodal vision models.
|
|
8
8
|
|
|
9
9
|
---
|
|
10
10
|
|
|
@@ -12,46 +12,63 @@ Supports Apple Silicon hardware encoding and returns image previews directly to
|
|
|
12
12
|
|
|
13
13
|
When AI agents process media, they often hallucinate invalid flags, syntax errors, or incompatible pixel formats.
|
|
14
14
|
|
|
15
|
-
This server provides
|
|
15
|
+
This server provides **17 typed tools** using Zod schemas. The AI passes parameters (such as `{ inputPath: "video.mov", preset: "web_mp4" }`), and the server runs the valid FFmpeg command locally with proper error handling, hardware acceleration, and progress notifications.
|
|
16
16
|
|
|
17
17
|
---
|
|
18
18
|
|
|
19
19
|
## Key Features
|
|
20
20
|
|
|
21
|
-
* **Hardware
|
|
22
|
-
* **
|
|
23
|
-
* **
|
|
24
|
-
* **
|
|
25
|
-
* **
|
|
21
|
+
* **Universal Hardware Acceleration:** Automatically detects and leverages hardware encoders across macOS (VideoToolbox), Linux/Windows GPUs (NVIDIA NVENC, Intel QuickSync, AMD AMF, and VA-API) for 5x–20x faster exports with low CPU usage.
|
|
22
|
+
* **Real-time Progress Reporting:** Emits live percentage updates via standard MCP progress tokens (`notifications/progress`), enabling progress bars in modern AI clients.
|
|
23
|
+
* **Instant Cancellation:** Listens to client `AbortSignal` events to immediately terminate child FFmpeg processes when a generation is stopped.
|
|
24
|
+
* **Multimodal Vision Previews:** Inlines base64 image data for thumbnails, audio waveforms, and multi-frame contact sheets so vision models (Claude 3.7, GPT-4o, Gemini) can "see" media content.
|
|
25
|
+
* **Social Formats & Automation:** Reformats landscape video to 9:16 vertical video with blurred backgrounds, burns customizable hardcoded subtitles, and stacks top/bottom split-screen reels for TikTok and Instagram Reels.
|
|
26
|
+
* **Audio Engineering:** Normalizes loudness to broadcast/streaming standards (EBU R128 / -16 LUFS), cleans vocal audio with 80Hz rumble removal, and trims dead air while preserving lip-sync.
|
|
26
27
|
|
|
27
28
|
---
|
|
28
29
|
|
|
29
|
-
## Included Tools (
|
|
30
|
+
## Included Tools (17 Total)
|
|
30
31
|
|
|
31
32
|
| Tool | Description | Key Parameters |
|
|
32
33
|
| :--- | :--- | :--- |
|
|
33
34
|
| **`media_probe`** | Inspects media files with ffprobe. Returns metadata, streams, codecs, resolution, and duration as JSON. | `inputPath` |
|
|
34
|
-
| **`media_convert`** | Converts video and audio formats. Includes presets for web MP4, WebM, and animated GIF. | `inputPath`, `outputPath`, `preset`, `crf`, `resolution`, `fps`, `hardwareAcceleration` |
|
|
35
|
+
| **`media_convert`** | Converts video and audio formats. Includes presets for web MP4, WebM, and animated GIF with GPU acceleration. | `inputPath`, `outputPath`, `preset`, `crf`, `resolution`, `fps`, `hardwareAcceleration` |
|
|
35
36
|
| **`trim_media`** | Cuts a segment from a video or audio file using stream copy or re-encoding. | `inputPath`, `outputPath`, `startTime`, `endTime`, `duration`, `accurateReencode` |
|
|
36
37
|
| **`extract_audio`** | Rips audio tracks from video. Includes a 16kHz mono WAV preset for Whisper speech recognition. | `inputPath`, `outputPath`, `format`, `bitrate` |
|
|
37
38
|
| **`extract_thumbnail`** | Captures a still image at a timestamp or midpoint. Returns base64 image data for vision models. | `inputPath`, `outputPath`, `timestamp`, `resolution`, `includeBase64` |
|
|
38
39
|
| **`compress_video`** | Compresses video to fit target sizes (such as GitHub PRs under 10MB or Discord under 25MB). | `inputPath`, `outputPath`, `targetSizeMB`, `preset`, `crf`, `maxWidth`, `hardwareAcceleration` |
|
|
39
|
-
| **`reformat_aspect_ratio`** | Changes video ratio to 9:16 vertical, 1:1 square, or 4:5 portrait with blurred borders or center crop. | `inputPath`, `outputPath`, `targetRatio`, `mode`, `blurAmount`, `padColor` |
|
|
40
|
+
| **`reformat_aspect_ratio`** | Changes video ratio to 9:16 vertical, 1:1 square, or 4:5 portrait with blurred borders or center crop. | `inputPath`, `outputPath`, `targetRatio`, `mode`, `blurAmount`, `padColor`, `hardwareAcceleration` |
|
|
40
41
|
| **`create_split_screen`** | Stacks two videos top-and-bottom (9:16 Reels/TikTok) or side-by-side with audio mixing and subtitles. | `firstVideoPath`, `secondVideoPath`, `outputPath`, `layout`, `audioSource`, `subtitlesPath` |
|
|
41
|
-
| **`trim_silence`** | Detects and removes silent pauses from recordings
|
|
42
|
+
| **`trim_silence`** | Detects and removes silent pauses from recordings while preserving video & audio lip-sync. | `inputPath`, `outputPath`, `mode`, `thresholdDb`, `minDurationSeconds` |
|
|
42
43
|
| **`concat_media`** | Joins multiple video or audio files in order using stream copy or re-encoding. | `inputPaths`, `outputPath`, `mode` |
|
|
43
44
|
| **`generate_waveform`** | Generates a PNG waveform image. Returns base64 image data for vision models. | `inputPath`, `outputPath`, `width`, `height`, `color`, `scale`, `includeBase64` |
|
|
44
|
-
| **`add_subtitles`** | Embeds SRT or VTT subtitle tracks into video files without re-encoding video. | `videoPath`, `subtitlesPath`, `outputPath`, `language`, `title` |
|
|
45
|
+
| **`add_subtitles`** | Embeds SRT or VTT subtitle tracks into MP4/MKV video files without re-encoding video. | `videoPath`, `subtitlesPath`, `outputPath`, `language`, `title` |
|
|
46
|
+
| **`burn_subtitles`** *(New)* | Hardcodes subtitles directly onto video frames with customizable font size, colors, and margins. | `videoPath`, `subtitlesPath`, `outputPath`, `fontSize`, `fontColor`, `outlineColor`, `marginV` |
|
|
47
|
+
| **`extract_keyframes`** *(New)* | Extracts multiple representative frames across the video and generates a contact sheet grid for vision AI. | `inputPath`, `outputPath`, `count`, `layout`, `includeBase64` |
|
|
48
|
+
| **`normalize_audio`** *(New)* | Normalizes audio loudness using EBU R128 (default -16 LUFS) with optional speech clarity high-pass filter. | `inputPath`, `outputPath`, `targetLufs`, `targetPeakDb`, `voiceClarity` |
|
|
49
|
+
| **`add_overlay`** *(New)* | Places a watermark image or logo PNG over video with customizable corner position, scale, and opacity. | `videoPath`, `overlayPath`, `outputPath`, `position`, `padding`, `scaleWidth`, `opacity` |
|
|
45
50
|
| **`change_speed`** | Adjusts playback speed (such as 2x fast forward or 0.5x slow motion) with pitch correction. | `inputPath`, `outputPath`, `speed`, `muteAudio` |
|
|
46
51
|
|
|
47
52
|
---
|
|
48
53
|
|
|
54
|
+
## MCP Resources & Prompts
|
|
55
|
+
|
|
56
|
+
### Resources
|
|
57
|
+
* **`media://capabilities`**: Inspects local FFmpeg installation, active version, detected hardware accelerators (VideoToolbox, NVENC, QSV, AMF, VAAPI), and supported video encoders.
|
|
58
|
+
* **`media://presets`**: Standard reference table of social aspect ratios, platform upload file size limits, and broadcast audio loudness targets.
|
|
59
|
+
|
|
60
|
+
### Prompts
|
|
61
|
+
* **`social_clip_generator`**: Interactive workflow that turns raw landscape video into vertical shorts with blurred background padding, trimmed silence, and hardcoded captions.
|
|
62
|
+
* **`whisper_audio_prep`**: Standard pipeline for extracting 16kHz mono WAV audio and normalizing loudness for speech-to-text models.
|
|
63
|
+
|
|
64
|
+
---
|
|
65
|
+
|
|
49
66
|
## Requirements
|
|
50
67
|
|
|
51
68
|
* **Node.js** >= 18.0.0
|
|
52
69
|
* **FFmpeg & FFprobe** installed locally:
|
|
53
70
|
* **macOS:** `brew install ffmpeg`
|
|
54
|
-
* **Linux (Ubuntu/Debian):** `sudo apt update && sudo apt install ffmpeg`
|
|
71
|
+
* **Linux (Ubuntu/Debian):** `sudo apt update && sudo apt install ffmpeg libass-dev`
|
|
55
72
|
* **Windows:** `winget install Gyan.FFmpeg` or `choco install ffmpeg`
|
|
56
73
|
|
|
57
74
|
---
|
|
@@ -77,25 +94,20 @@ Run directly with `npx` (no manual cloning or building required):
|
|
|
77
94
|
* **Claude Desktop (macOS):** `~/Library/Application Support/Claude/claude_desktop_config.json`
|
|
78
95
|
* **Claude Desktop (Windows):** `%APPDATA%\Claude\claude_desktop_config.json`
|
|
79
96
|
* **Antigravity IDE:** `~/.gemini/config/mcp_config.json`
|
|
80
|
-
* **Cursor / Windsurf:** Add as a stdio server with command `npx` and args `["-y", "@ash_j1997/ffmpeg-mcp-server"]
|
|
97
|
+
* **Cursor / Windsurf:** Add as a stdio server with command `npx` and args `["-y", "@ash_j1997/ffmpeg-mcp-server"]`
|
|
81
98
|
|
|
82
99
|
---
|
|
83
100
|
|
|
84
|
-
## Local Development
|
|
101
|
+
## Local Development & Testing
|
|
85
102
|
|
|
86
103
|
1. **Install dependencies:**
|
|
87
104
|
```bash
|
|
88
105
|
npm install
|
|
89
106
|
```
|
|
90
107
|
|
|
91
|
-
2. **
|
|
92
|
-
```bash
|
|
93
|
-
npm run build
|
|
94
|
-
```
|
|
95
|
-
|
|
96
|
-
3. **Run tests:**
|
|
108
|
+
2. **Run build and automated test suite:**
|
|
97
109
|
```bash
|
|
98
|
-
|
|
110
|
+
npm test
|
|
99
111
|
```
|
|
100
112
|
|
|
101
113
|
---
|
package/dist/index.js
CHANGED
|
@@ -2,11 +2,11 @@
|
|
|
2
2
|
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
|
|
3
3
|
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
|
|
4
4
|
import { z } from "zod";
|
|
5
|
-
import { probeMedia } from "./tools/probe.js";
|
|
6
|
-
import { convertMedia } from "./tools/convert.js";
|
|
7
|
-
import { trimMedia } from "./tools/trim.js";
|
|
8
|
-
import { extractAudio } from "./tools/audio.js";
|
|
9
|
-
import { extractThumbnail } from "./tools/thumbnail.js";
|
|
5
|
+
import { probeMedia, probeMediaSchema } from "./tools/probe.js";
|
|
6
|
+
import { convertMedia, convertMediaSchema } from "./tools/convert.js";
|
|
7
|
+
import { trimMedia, trimMediaSchema } from "./tools/trim.js";
|
|
8
|
+
import { extractAudio, extractAudioSchema } from "./tools/audio.js";
|
|
9
|
+
import { extractThumbnail, extractThumbnailSchema } from "./tools/thumbnail.js";
|
|
10
10
|
import { compressMedia, compressMediaSchema } from "./tools/compress.js";
|
|
11
11
|
import { concatMedia, concatMediaSchema } from "./tools/concat.js";
|
|
12
12
|
import { generateWaveform, generateWaveformSchema } from "./tools/waveform.js";
|
|
@@ -15,16 +15,39 @@ import { changeSpeed, changeSpeedSchema } from "./tools/speed.js";
|
|
|
15
15
|
import { reformatAspectRatio, reformatAspectSchema } from "./tools/reformat.js";
|
|
16
16
|
import { trimSilence, trimSilenceSchema } from "./tools/silence.js";
|
|
17
17
|
import { createSplitScreen, splitScreenSchema } from "./tools/split.js";
|
|
18
|
+
import { burnSubtitles, burnSubtitlesSchema } from "./tools/burn_subtitles.js";
|
|
19
|
+
import { extractKeyframes, extractKeyframesSchema } from "./tools/keyframes.js";
|
|
20
|
+
import { normalizeAudio, normalizeAudioSchema } from "./tools/normalize.js";
|
|
21
|
+
import { addOverlay, addOverlaySchema } from "./tools/overlay.js";
|
|
22
|
+
import { detectHwAccel } from "./utils/hwaccel.js";
|
|
23
|
+
import { runCommand } from "./utils/exec.js";
|
|
18
24
|
const server = new McpServer({
|
|
19
25
|
name: "ffmpeg-mcp-server",
|
|
20
|
-
version: "0.
|
|
26
|
+
version: "0.5.0",
|
|
21
27
|
});
|
|
28
|
+
function createProgressCallback(extra) {
|
|
29
|
+
return (p) => {
|
|
30
|
+
if (extra._meta?.progressToken !== undefined && typeof extra.sendNotification === "function") {
|
|
31
|
+
extra
|
|
32
|
+
.sendNotification({
|
|
33
|
+
method: "notifications/progress",
|
|
34
|
+
params: {
|
|
35
|
+
progressToken: extra._meta.progressToken,
|
|
36
|
+
progress: p.percent ?? (p.timeSeconds ? Math.round(p.timeSeconds) : 0),
|
|
37
|
+
total: 100,
|
|
38
|
+
message: p.percent
|
|
39
|
+
? `Processing: ${p.percent}% (speed: ${p.speed || "N/A"})`
|
|
40
|
+
: `Processing media (${p.timeSeconds || 0}s rendered)...`,
|
|
41
|
+
},
|
|
42
|
+
})
|
|
43
|
+
.catch(() => { });
|
|
44
|
+
}
|
|
45
|
+
};
|
|
46
|
+
}
|
|
22
47
|
// Tool 1: Probe Media
|
|
23
|
-
server.tool("media_probe", "Inspect audio and video files with ffprobe. Returns container metadata, stream details, codecs, resolution, bitrate, and duration as JSON.", {
|
|
24
|
-
inputPath: z.string().describe("Path to the media file to inspect (e.g. './video.mp4')"),
|
|
25
|
-
}, async ({ inputPath }) => {
|
|
48
|
+
server.tool("media_probe", "Inspect audio and video files with ffprobe. Returns container metadata, stream details, codecs, resolution, bitrate, and duration as JSON.", probeMediaSchema.shape, async ({ inputPath }, extra) => {
|
|
26
49
|
try {
|
|
27
|
-
const result = await probeMedia(inputPath);
|
|
50
|
+
const result = await probeMedia(inputPath, { signal: extra.signal });
|
|
28
51
|
return {
|
|
29
52
|
content: [{ type: "text", text: JSON.stringify(result, null, 2) }],
|
|
30
53
|
};
|
|
@@ -37,40 +60,15 @@ server.tool("media_probe", "Inspect audio and video files with ffprobe. Returns
|
|
|
37
60
|
}
|
|
38
61
|
});
|
|
39
62
|
// Tool 2: Convert Media
|
|
40
|
-
server.tool("media_convert", "Convert video and audio files between formats. Includes presets for web MP4 (H.264/AAC), WebM (VP9/Opus), and animated GIF. Supports Apple Silicon
|
|
41
|
-
inputPath: z.string().describe("Path to the source media file to convert"),
|
|
42
|
-
outputPath: z.string().describe("Destination path for the converted file (e.g. ./output.mp4, ./output.webm, ./animation.gif)"),
|
|
43
|
-
preset: z
|
|
44
|
-
.enum(["web_mp4", "webm", "gif", "auto"])
|
|
45
|
-
.optional()
|
|
46
|
-
.describe("Encoding preset: 'web_mp4' for browser playback, 'webm' for WebM video, 'gif' for animated GIF, or 'auto'. Default: 'auto'."),
|
|
47
|
-
crf: z
|
|
48
|
-
.number()
|
|
49
|
-
.min(0)
|
|
50
|
-
.max(51)
|
|
51
|
-
.optional()
|
|
52
|
-
.describe("Quality factor from 0 to 51. Lower numbers mean higher quality. Default: 23 for MP4, 30 for WebM."),
|
|
53
|
-
resolution: z
|
|
54
|
-
.string()
|
|
55
|
-
.optional()
|
|
56
|
-
.describe("Target size formatted as 'WIDTHxHEIGHT', such as '1920x1080' or '1280x720'"),
|
|
57
|
-
fps: z
|
|
58
|
-
.number()
|
|
59
|
-
.positive()
|
|
60
|
-
.optional()
|
|
61
|
-
.describe("Target frame rate in frames per second, such as 30, 60, or 15"),
|
|
62
|
-
hardwareAcceleration: z
|
|
63
|
-
.boolean()
|
|
64
|
-
.optional()
|
|
65
|
-
.describe("Use Apple Silicon VideoToolbox encoding when available. Default: true."),
|
|
66
|
-
}, async (args) => {
|
|
63
|
+
server.tool("media_convert", "Convert video and audio files between formats. Includes presets for web MP4 (H.264/AAC), WebM (VP9/Opus), and animated GIF. Supports hardware acceleration on Apple Silicon, NVIDIA, Intel, and AMD.", convertMediaSchema.shape, async (args, extra) => {
|
|
67
64
|
try {
|
|
68
|
-
const
|
|
65
|
+
const onProgress = createProgressCallback(extra);
|
|
66
|
+
const result = await convertMedia(args, { signal: extra.signal, onProgress });
|
|
69
67
|
return {
|
|
70
68
|
content: [
|
|
71
69
|
{
|
|
72
70
|
type: "text",
|
|
73
|
-
text: `Successfully converted media:\nOutput: ${result.outputPath}\
|
|
71
|
+
text: `Successfully converted media:\nOutput: ${result.outputPath}\nEncoder: ${result.encoderUsed} (Hardware Accelerated: ${result.hardwareAccelerated})\nCommand: ${result.commandExecuted}`,
|
|
74
72
|
},
|
|
75
73
|
],
|
|
76
74
|
};
|
|
@@ -83,19 +81,10 @@ server.tool("media_convert", "Convert video and audio files between formats. Inc
|
|
|
83
81
|
}
|
|
84
82
|
});
|
|
85
83
|
// Tool 3: Trim Media
|
|
86
|
-
server.tool("trim_media", "Cut a segment from a video or audio file. Uses stream copy without re-encoding by default, or re-encodes when accurate cuts are needed.", {
|
|
87
|
-
inputPath: z.string().describe("Path to the source media file to cut."),
|
|
88
|
-
outputPath: z.string().describe("Destination path for the trimmed output file (e.g. './clip.mp4')."),
|
|
89
|
-
startTime: z.string().describe("Start timestamp formatted as 'HH:MM:SS' (e.g. '00:01:30') or seconds (e.g. '90')."),
|
|
90
|
-
endTime: z.string().optional().describe("Optional end timestamp formatted as 'HH:MM:SS' or seconds. Do not combine with duration."),
|
|
91
|
-
duration: z.string().optional().describe("Optional duration in seconds from startTime (e.g. '30'). If neither endTime nor duration is set, trims to end of file."),
|
|
92
|
-
accurateReencode: z
|
|
93
|
-
.boolean()
|
|
94
|
-
.optional()
|
|
95
|
-
.describe("Re-encode video for cuts between keyframes. Default: false (uses fast stream copy)."),
|
|
96
|
-
}, async (args) => {
|
|
84
|
+
server.tool("trim_media", "Cut a segment from a video or audio file. Uses stream copy without re-encoding by default, or re-encodes when accurate cuts are needed.", trimMediaSchema.shape, async (args, extra) => {
|
|
97
85
|
try {
|
|
98
|
-
const
|
|
86
|
+
const onProgress = createProgressCallback(extra);
|
|
87
|
+
const result = await trimMedia(args, { signal: extra.signal, onProgress });
|
|
99
88
|
return {
|
|
100
89
|
content: [
|
|
101
90
|
{
|
|
@@ -113,22 +102,15 @@ server.tool("trim_media", "Cut a segment from a video or audio file. Uses stream
|
|
|
113
102
|
}
|
|
114
103
|
});
|
|
115
104
|
// Tool 4: Extract Audio
|
|
116
|
-
server.tool("extract_audio", "Extract an audio track from a video file into a standalone audio file. Supports the 'whisper_wav' preset (16kHz mono WAV) for speech recognition models, plus MP3, AAC, FLAC, and stream copy.", {
|
|
117
|
-
inputPath: z.string().describe("Path to the source video or audio file."),
|
|
118
|
-
outputPath: z.string().describe("Destination path for the output audio file (e.g. './extracted.wav' or './audio.mp3')."),
|
|
119
|
-
format: z
|
|
120
|
-
.enum(["whisper_wav", "mp3", "aac", "flac", "copy"])
|
|
121
|
-
.optional()
|
|
122
|
-
.describe("Audio format: 'whisper_wav' (16kHz mono WAV for Whisper), 'mp3', 'aac', 'flac', or 'copy' (extract without re-encoding)."),
|
|
123
|
-
bitrate: z.string().optional().describe("Target audio bitrate for MP3 or AAC (e.g. '128k', '192k', '320k')."),
|
|
124
|
-
}, async (args) => {
|
|
105
|
+
server.tool("extract_audio", "Extract an audio track from a video file into a standalone audio file. Supports the 'whisper_wav' preset (16kHz mono WAV) for speech recognition models, plus MP3, AAC, FLAC, and stream copy.", extractAudioSchema.shape, async (args, extra) => {
|
|
125
106
|
try {
|
|
126
|
-
const
|
|
107
|
+
const onProgress = createProgressCallback(extra);
|
|
108
|
+
const result = await extractAudio(args, { signal: extra.signal, onProgress });
|
|
127
109
|
return {
|
|
128
110
|
content: [
|
|
129
111
|
{
|
|
130
112
|
type: "text",
|
|
131
|
-
text: `Successfully extracted audio:\nOutput: ${result.outputPath}\nCommand: ${result.commandExecuted}`,
|
|
113
|
+
text: `Successfully extracted audio:\nOutput: ${result.outputPath}\nFormat: ${result.format}\nCommand: ${result.commandExecuted}`,
|
|
132
114
|
},
|
|
133
115
|
],
|
|
134
116
|
};
|
|
@@ -141,18 +123,9 @@ server.tool("extract_audio", "Extract an audio track from a video file into a st
|
|
|
141
123
|
}
|
|
142
124
|
});
|
|
143
125
|
// Tool 5: Extract Thumbnail
|
|
144
|
-
server.tool("extract_thumbnail", "Capture a still image from a video. Returns the saved image path and inlined base64 image data so vision models can inspect the frame.", {
|
|
145
|
-
inputPath: z.string().describe("Path to the source video file."),
|
|
146
|
-
outputPath: z.string().describe("Destination image path ending in .png or .jpg (e.g. './thumb.png')."),
|
|
147
|
-
timestamp: z
|
|
148
|
-
.string()
|
|
149
|
-
.optional()
|
|
150
|
-
.describe("Timestamp to capture, such as '00:00:05', '10', or 'midpoint' for the middle frame. Default: '00:00:01'."),
|
|
151
|
-
resolution: z.string().optional().describe("Optional scaled size formatted as 'WIDTHxHEIGHT', such as '1280x720'. Defaults to source resolution."),
|
|
152
|
-
includeBase64: z.boolean().optional().describe("Return base64 image data in response for vision models. Default: true."),
|
|
153
|
-
}, async (args) => {
|
|
126
|
+
server.tool("extract_thumbnail", "Capture a single still image from a video. Returns the saved image path and inlined base64 image data so vision models can inspect the frame.", extractThumbnailSchema.shape, async (args, extra) => {
|
|
154
127
|
try {
|
|
155
|
-
const result = await extractThumbnail(args);
|
|
128
|
+
const result = await extractThumbnail(args, { signal: extra.signal });
|
|
156
129
|
const content = [];
|
|
157
130
|
if (result.base64 && result.mimeType) {
|
|
158
131
|
content.push({
|
|
@@ -175,9 +148,10 @@ server.tool("extract_thumbnail", "Capture a still image from a video. Returns th
|
|
|
175
148
|
}
|
|
176
149
|
});
|
|
177
150
|
// Tool 6: Compress Video
|
|
178
|
-
server.tool("compress_video", "Compress video files to fit upload limits (such as GitHub PRs under 10MB or Discord under 25MB) using target file sizes or CRF quality levels. Does not modify the original file.", compressMediaSchema.shape, async (args) => {
|
|
151
|
+
server.tool("compress_video", "Compress video files to fit upload limits (such as GitHub PRs under 10MB or Discord under 25MB) using target file sizes or CRF quality levels. Does not modify the original file.", compressMediaSchema.shape, async (args, extra) => {
|
|
179
152
|
try {
|
|
180
|
-
const
|
|
153
|
+
const onProgress = createProgressCallback(extra);
|
|
154
|
+
const result = await compressMedia(args, { signal: extra.signal, onProgress });
|
|
181
155
|
return {
|
|
182
156
|
content: [
|
|
183
157
|
{
|
|
@@ -195,9 +169,10 @@ server.tool("compress_video", "Compress video files to fit upload limits (such a
|
|
|
195
169
|
}
|
|
196
170
|
});
|
|
197
171
|
// Tool 7: Concat Media
|
|
198
|
-
server.tool("concat_media", "Join multiple video or audio files in order into a single file. Uses stream copy for matching files or re-encodes when formats differ.", concatMediaSchema.shape, async (args) => {
|
|
172
|
+
server.tool("concat_media", "Join multiple video or audio files in order into a single file. Uses stream copy for matching files or re-encodes when formats differ.", concatMediaSchema.shape, async (args, extra) => {
|
|
199
173
|
try {
|
|
200
|
-
const
|
|
174
|
+
const onProgress = createProgressCallback(extra);
|
|
175
|
+
const result = await concatMedia(args, { signal: extra.signal, onProgress });
|
|
201
176
|
return {
|
|
202
177
|
content: [
|
|
203
178
|
{
|
|
@@ -215,9 +190,9 @@ server.tool("concat_media", "Join multiple video or audio files in order into a
|
|
|
215
190
|
}
|
|
216
191
|
});
|
|
217
192
|
// Tool 8: Generate Waveform
|
|
218
|
-
server.tool("generate_waveform", "Generate a PNG waveform image from an audio or video file. Returns the file path and inlined base64 image data.", generateWaveformSchema.shape, async (args) => {
|
|
193
|
+
server.tool("generate_waveform", "Generate a PNG waveform image from an audio or video file. Returns the file path and inlined base64 image data.", generateWaveformSchema.shape, async (args, extra) => {
|
|
219
194
|
try {
|
|
220
|
-
const result = await generateWaveform(args);
|
|
195
|
+
const result = await generateWaveform(args, { signal: extra.signal });
|
|
221
196
|
const content = [];
|
|
222
197
|
if (result.base64) {
|
|
223
198
|
content.push({
|
|
@@ -239,10 +214,11 @@ server.tool("generate_waveform", "Generate a PNG waveform image from an audio or
|
|
|
239
214
|
};
|
|
240
215
|
}
|
|
241
216
|
});
|
|
242
|
-
// Tool 9: Add Subtitles
|
|
243
|
-
server.tool("add_subtitles", "Embed an SRT or VTT subtitle track into a video
|
|
217
|
+
// Tool 9: Add Subtitles (Soft Embedded)
|
|
218
|
+
server.tool("add_subtitles", "Embed an SRT or VTT subtitle track into a video container (mov_text for MP4, srt for MKV) without re-encoding the video stream.", addSubtitlesSchema.shape, async (args, extra) => {
|
|
244
219
|
try {
|
|
245
|
-
const
|
|
220
|
+
const onProgress = createProgressCallback(extra);
|
|
221
|
+
const result = await addSubtitles(args, { signal: extra.signal, onProgress });
|
|
246
222
|
return {
|
|
247
223
|
content: [
|
|
248
224
|
{
|
|
@@ -260,9 +236,10 @@ server.tool("add_subtitles", "Embed an SRT or VTT subtitle track into a video fi
|
|
|
260
236
|
}
|
|
261
237
|
});
|
|
262
238
|
// Tool 10: Change Speed
|
|
263
|
-
server.tool("change_speed", "Change video and audio playback speed (such as 2x for fast forward or 0.5x for slow motion) with pitch correction.", changeSpeedSchema.shape, async (args) => {
|
|
239
|
+
server.tool("change_speed", "Change video and audio playback speed (such as 2x for fast forward or 0.5x for slow motion) with pitch correction.", changeSpeedSchema.shape, async (args, extra) => {
|
|
264
240
|
try {
|
|
265
|
-
const
|
|
241
|
+
const onProgress = createProgressCallback(extra);
|
|
242
|
+
const result = await changeSpeed(args, { signal: extra.signal, onProgress });
|
|
266
243
|
return {
|
|
267
244
|
content: [
|
|
268
245
|
{
|
|
@@ -280,9 +257,10 @@ server.tool("change_speed", "Change video and audio playback speed (such as 2x f
|
|
|
280
257
|
}
|
|
281
258
|
});
|
|
282
259
|
// Tool 11: Reformat Aspect Ratio
|
|
283
|
-
server.tool("reformat_aspect_ratio", "Change video aspect ratio to 9:16 vertical, 1:1 square, or 4:5 portrait. Adds blurred video borders, solid padding, or center crops.", reformatAspectSchema.shape, async (args) => {
|
|
260
|
+
server.tool("reformat_aspect_ratio", "Change video aspect ratio to 9:16 vertical, 1:1 square, or 4:5 portrait. Adds blurred video borders, solid padding, or center crops.", reformatAspectSchema.shape, async (args, extra) => {
|
|
284
261
|
try {
|
|
285
|
-
const
|
|
262
|
+
const onProgress = createProgressCallback(extra);
|
|
263
|
+
const result = await reformatAspectRatio(args, { signal: extra.signal, onProgress });
|
|
286
264
|
return {
|
|
287
265
|
content: [
|
|
288
266
|
{
|
|
@@ -300,9 +278,10 @@ server.tool("reformat_aspect_ratio", "Change video aspect ratio to 9:16 vertical
|
|
|
300
278
|
}
|
|
301
279
|
});
|
|
302
280
|
// Tool 12: Trim Silence
|
|
303
|
-
server.tool("trim_silence", "Detect and remove silent pauses from audio or video files. Returns
|
|
281
|
+
server.tool("trim_silence", "Detect and remove silent pauses from audio or video files. Preserves video & audio lip-sync in 'start_end' mode. Returns original duration and seconds saved.", trimSilenceSchema.shape, async (args, extra) => {
|
|
304
282
|
try {
|
|
305
|
-
const
|
|
283
|
+
const onProgress = createProgressCallback(extra);
|
|
284
|
+
const result = await trimSilence(args, { signal: extra.signal, onProgress });
|
|
306
285
|
return {
|
|
307
286
|
content: [
|
|
308
287
|
{
|
|
@@ -320,9 +299,10 @@ server.tool("trim_silence", "Detect and remove silent pauses from audio or video
|
|
|
320
299
|
}
|
|
321
300
|
});
|
|
322
301
|
// Tool 13: Create Split Screen
|
|
323
|
-
server.tool("create_split_screen", "Combine two video files into a split-screen video. Stacks clips vertically (1080x1920 for Reels and TikTok) or places them side-by-side. Supports audio mixing and subtitle embedding.", splitScreenSchema.shape, async (args) => {
|
|
302
|
+
server.tool("create_split_screen", "Combine two video files into a split-screen video. Stacks clips vertically (1080x1920 for Reels and TikTok) or places them side-by-side. Supports audio mixing and subtitle embedding.", splitScreenSchema.shape, async (args, extra) => {
|
|
324
303
|
try {
|
|
325
|
-
const
|
|
304
|
+
const onProgress = createProgressCallback(extra);
|
|
305
|
+
const result = await createSplitScreen(args, { signal: extra.signal, onProgress });
|
|
326
306
|
return {
|
|
327
307
|
content: [
|
|
328
308
|
{
|
|
@@ -339,10 +319,222 @@ server.tool("create_split_screen", "Combine two video files into a split-screen
|
|
|
339
319
|
};
|
|
340
320
|
}
|
|
341
321
|
});
|
|
322
|
+
// Tool 14: Burn Subtitles (Hardcoded)
|
|
323
|
+
server.tool("burn_subtitles", "Hardcode/burn subtitles directly into video frames with customizable font size, colors, and positioning. Unlike soft subtitles, burned subtitles render on all platforms including TikTok, Instagram Reels, and YouTube Shorts.", burnSubtitlesSchema.shape, async (args, extra) => {
|
|
324
|
+
try {
|
|
325
|
+
const onProgress = createProgressCallback(extra);
|
|
326
|
+
const result = await burnSubtitles(args, { signal: extra.signal, onProgress });
|
|
327
|
+
return {
|
|
328
|
+
content: [
|
|
329
|
+
{
|
|
330
|
+
type: "text",
|
|
331
|
+
text: `Successfully burned subtitles into video:\nOutput: ${result.outputPath}\nSubtitles: ${result.subtitlesBurned}\nFont Size: ${result.fontSize}pt (${result.fontColor})\nEncoder: ${result.encoderUsed} (Hardware Accelerated: ${result.hardwareAccelerated})\nCommand: ${result.commandExecuted}`,
|
|
332
|
+
},
|
|
333
|
+
],
|
|
334
|
+
};
|
|
335
|
+
}
|
|
336
|
+
catch (err) {
|
|
337
|
+
return {
|
|
338
|
+
isError: true,
|
|
339
|
+
content: [{ type: "text", text: `Error burning subtitles: ${err.message}` }],
|
|
340
|
+
};
|
|
341
|
+
}
|
|
342
|
+
});
|
|
343
|
+
// Tool 15: Extract Keyframes (Multi-Frame / Contact Sheet)
|
|
344
|
+
server.tool("extract_keyframes", "Extract multiple representative frames across a video's timeline. In 'montage' mode, merges frames into a single contact-sheet grid image and inlines base64 for vision models to inspect the whole video at once.", extractKeyframesSchema.shape, async (args, extra) => {
|
|
345
|
+
try {
|
|
346
|
+
const result = await extractKeyframes(args, { signal: extra.signal });
|
|
347
|
+
const content = [];
|
|
348
|
+
if (result.base64 && result.mimeType) {
|
|
349
|
+
content.push({
|
|
350
|
+
type: "image",
|
|
351
|
+
data: result.base64,
|
|
352
|
+
mimeType: result.mimeType,
|
|
353
|
+
});
|
|
354
|
+
}
|
|
355
|
+
content.push({
|
|
356
|
+
type: "text",
|
|
357
|
+
text: `Successfully extracted ${result.frameCount} keyframes:\nOutput: ${result.outputPath}\nLayout: ${result.layout}${result.gridDimensions ? ` (${result.gridDimensions} grid)` : ""}\nCommand: ${result.commandExecuted}`,
|
|
358
|
+
});
|
|
359
|
+
return { content };
|
|
360
|
+
}
|
|
361
|
+
catch (err) {
|
|
362
|
+
return {
|
|
363
|
+
isError: true,
|
|
364
|
+
content: [{ type: "text", text: `Error extracting keyframes: ${err.message}` }],
|
|
365
|
+
};
|
|
366
|
+
}
|
|
367
|
+
});
|
|
368
|
+
// Tool 16: Normalize Audio (Loudnorm / EBU R128)
|
|
369
|
+
server.tool("normalize_audio", "Normalize audio loudness using the broadcast standard EBU R128 (loudnorm). Supports -16 LUFS (podcasts/Reels), -14 LUFS (YouTube), and optional voice clarity enhancement (80Hz high-pass to remove mic rumble). Works on audio or video files.", normalizeAudioSchema.shape, async (args, extra) => {
|
|
370
|
+
try {
|
|
371
|
+
const onProgress = createProgressCallback(extra);
|
|
372
|
+
const result = await normalizeAudio(args, { signal: extra.signal, onProgress });
|
|
373
|
+
return {
|
|
374
|
+
content: [
|
|
375
|
+
{
|
|
376
|
+
type: "text",
|
|
377
|
+
text: `Successfully normalized audio loudness:\nOutput: ${result.outputPath}\nTarget LUFS: ${result.targetLufs} LUFS (Peak: ${result.targetPeakDb} dBTP)\nVoice Clarity: ${result.voiceClarityEnabled ? "Enabled" : "Disabled"}\nMedia Type: ${result.isVideo ? "Video (lossless video stream copy)" : "Audio"}\nCommand: ${result.commandExecuted}`,
|
|
378
|
+
},
|
|
379
|
+
],
|
|
380
|
+
};
|
|
381
|
+
}
|
|
382
|
+
catch (err) {
|
|
383
|
+
return {
|
|
384
|
+
isError: true,
|
|
385
|
+
content: [{ type: "text", text: `Error normalizing audio: ${err.message}` }],
|
|
386
|
+
};
|
|
387
|
+
}
|
|
388
|
+
});
|
|
389
|
+
// Tool 17: Add Overlay (Watermark / Branding)
|
|
390
|
+
server.tool("add_overlay", "Place a watermark image, logo PNG, or branding graphic over a video with customizable position, padding, scaling, and opacity.", addOverlaySchema.shape, async (args, extra) => {
|
|
391
|
+
try {
|
|
392
|
+
const onProgress = createProgressCallback(extra);
|
|
393
|
+
const result = await addOverlay(args, { signal: extra.signal, onProgress });
|
|
394
|
+
return {
|
|
395
|
+
content: [
|
|
396
|
+
{
|
|
397
|
+
type: "text",
|
|
398
|
+
text: `Successfully added overlay:\nOutput: ${result.outputPath}\nPosition: ${result.position} (padding: ${result.padding}px)\nOpacity: ${result.opacity}\nEncoder: ${result.encoderUsed} (Hardware Accelerated: ${result.hardwareAccelerated})\nCommand: ${result.commandExecuted}`,
|
|
399
|
+
},
|
|
400
|
+
],
|
|
401
|
+
};
|
|
402
|
+
}
|
|
403
|
+
catch (err) {
|
|
404
|
+
return {
|
|
405
|
+
isError: true,
|
|
406
|
+
content: [{ type: "text", text: `Error adding overlay: ${err.message}` }],
|
|
407
|
+
};
|
|
408
|
+
}
|
|
409
|
+
});
|
|
410
|
+
// Resource 1: Capabilities & Hardware Info
|
|
411
|
+
server.registerResource("system_capabilities", "media://capabilities", {
|
|
412
|
+
title: "FFmpeg System Capabilities",
|
|
413
|
+
description: "Detected FFmpeg version, hardware acceleration devices, and available video encoders.",
|
|
414
|
+
mimeType: "application/json",
|
|
415
|
+
}, async () => {
|
|
416
|
+
const hwInfo = await detectHwAccel();
|
|
417
|
+
let versionStr = "unknown";
|
|
418
|
+
try {
|
|
419
|
+
const res = await runCommand("ffmpeg", ["-version"]);
|
|
420
|
+
const firstLine = res.stdout.split("\n")[0] || "";
|
|
421
|
+
versionStr = firstLine.replace(/^ffmpeg version\s*/, "").split(" ")[0];
|
|
422
|
+
}
|
|
423
|
+
catch { }
|
|
424
|
+
const data = {
|
|
425
|
+
ffmpegVersion: versionStr,
|
|
426
|
+
hardwareAcceleration: {
|
|
427
|
+
isAvailable: hwInfo.isAvailable,
|
|
428
|
+
primaryType: hwInfo.hardwareType,
|
|
429
|
+
supportedTypes: hwInfo.availableTypes,
|
|
430
|
+
h264Encoder: hwInfo.h264Encoder,
|
|
431
|
+
hevcEncoder: hwInfo.hevcEncoder,
|
|
432
|
+
},
|
|
433
|
+
availableEncoders: hwInfo.encoders,
|
|
434
|
+
totalToolsActive: 17,
|
|
435
|
+
};
|
|
436
|
+
return {
|
|
437
|
+
contents: [
|
|
438
|
+
{
|
|
439
|
+
uri: "media://capabilities",
|
|
440
|
+
mimeType: "application/json",
|
|
441
|
+
text: JSON.stringify(data, null, 2),
|
|
442
|
+
},
|
|
443
|
+
],
|
|
444
|
+
};
|
|
445
|
+
});
|
|
446
|
+
// Resource 2: Social Presets & Standards
|
|
447
|
+
server.registerResource("social_presets", "media://presets", {
|
|
448
|
+
title: "Media Presets and Guidelines",
|
|
449
|
+
description: "Aspect ratio standards, loudness limits, and upload specifications for social media platforms.",
|
|
450
|
+
mimeType: "application/json",
|
|
451
|
+
}, async () => {
|
|
452
|
+
const presets = {
|
|
453
|
+
aspectRatios: {
|
|
454
|
+
vertical_9_16: { resolution: "1080x1920", platforms: ["TikTok", "Instagram Reels", "YouTube Shorts"] },
|
|
455
|
+
horizontal_16_9: { resolution: "1920x1080", platforms: ["YouTube", "Twitter/X", "Vimeo"] },
|
|
456
|
+
square_1_1: { resolution: "1080x1080", platforms: ["Instagram Post", "LinkedIn"] },
|
|
457
|
+
portrait_4_5: { resolution: "1080x1350", platforms: ["Instagram Feed"] },
|
|
458
|
+
},
|
|
459
|
+
targetSizes: {
|
|
460
|
+
github_pr: { maxMB: 10, recommendedPreset: "github_pr" },
|
|
461
|
+
discord_free: { maxMB: 25, recommendedPreset: "discord" },
|
|
462
|
+
email_attachment: { maxMB: 15, recommendedPreset: "email" },
|
|
463
|
+
},
|
|
464
|
+
audioStandards: {
|
|
465
|
+
podcast_and_shorts: { lufs: -16, peakDb: -1.5 },
|
|
466
|
+
youtube: { lufs: -14, peakDb: -1.0 },
|
|
467
|
+
broadcast_ebu_r128: { lufs: -23, peakDb: -1.0 },
|
|
468
|
+
speech_recognition: { sampleRate: 16000, channels: 1, codec: "pcm_s16le" },
|
|
469
|
+
},
|
|
470
|
+
};
|
|
471
|
+
return {
|
|
472
|
+
contents: [
|
|
473
|
+
{
|
|
474
|
+
uri: "media://presets",
|
|
475
|
+
mimeType: "application/json",
|
|
476
|
+
text: JSON.stringify(presets, null, 2),
|
|
477
|
+
},
|
|
478
|
+
],
|
|
479
|
+
};
|
|
480
|
+
});
|
|
481
|
+
// Prompt 1: Social Media Clip Generator
|
|
482
|
+
server.registerPrompt("social_clip_generator", {
|
|
483
|
+
title: "Generate Viral Social Media Clip",
|
|
484
|
+
description: "Step-by-step workflow for turning landscape footage into a 9:16 vertical short with burned subtitles.",
|
|
485
|
+
argsSchema: {
|
|
486
|
+
inputPath: z.string().describe("Path to the source landscape video"),
|
|
487
|
+
subtitlesPath: z.string().optional().describe("Optional path to subtitles file (.srt)"),
|
|
488
|
+
},
|
|
489
|
+
}, async ({ inputPath, subtitlesPath }) => {
|
|
490
|
+
return {
|
|
491
|
+
messages: [
|
|
492
|
+
{
|
|
493
|
+
role: "user",
|
|
494
|
+
content: {
|
|
495
|
+
type: "text",
|
|
496
|
+
text: `Please convert this video into a 9:16 vertical reel suitable for TikTok, Instagram Reels, and YouTube Shorts:
|
|
497
|
+
Source Video: ${inputPath}
|
|
498
|
+
${subtitlesPath ? `Subtitles: ${subtitlesPath}` : "No separate subtitles provided."}
|
|
499
|
+
|
|
500
|
+
Recommended workflow:
|
|
501
|
+
1. Use 'media_probe' to check duration and resolution.
|
|
502
|
+
2. Use 'reformat_aspect_ratio' with targetRatio: '9:16' and mode: 'blur_background'.
|
|
503
|
+
3. Use 'trim_silence' with mode: 'start_end' to eliminate dead air at the beginning and end.
|
|
504
|
+
4. If subtitles are available, use 'burn_subtitles' to hardcode visible captions for mobile viewers.
|
|
505
|
+
5. Use 'normalize_audio' with targetLufs: -16 to ensure consistent loudness.`,
|
|
506
|
+
},
|
|
507
|
+
},
|
|
508
|
+
],
|
|
509
|
+
};
|
|
510
|
+
});
|
|
511
|
+
// Prompt 2: Whisper Transcription Prep
|
|
512
|
+
server.registerPrompt("whisper_audio_prep", {
|
|
513
|
+
title: "Prepare Media for Whisper Speech Recognition",
|
|
514
|
+
description: "Extract and normalize 16kHz mono audio optimized for Whisper and local STT models.",
|
|
515
|
+
argsSchema: {
|
|
516
|
+
inputPath: z.string().describe("Path to video or audio file"),
|
|
517
|
+
},
|
|
518
|
+
}, async ({ inputPath }) => {
|
|
519
|
+
return {
|
|
520
|
+
messages: [
|
|
521
|
+
{
|
|
522
|
+
role: "user",
|
|
523
|
+
content: {
|
|
524
|
+
type: "text",
|
|
525
|
+
text: `Extract the audio from '${inputPath}' for speech-to-text processing:
|
|
526
|
+
1. Use 'extract_audio' with format: 'whisper_wav' (16kHz mono PCM WAV).
|
|
527
|
+
2. If the audio is quiet, use 'normalize_audio' with voiceClarity: true.
|
|
528
|
+
3. Pass the resulting WAV to your speech recognition tool.`,
|
|
529
|
+
},
|
|
530
|
+
},
|
|
531
|
+
],
|
|
532
|
+
};
|
|
533
|
+
});
|
|
342
534
|
async function main() {
|
|
343
535
|
const transport = new StdioServerTransport();
|
|
344
536
|
await server.connect(transport);
|
|
345
|
-
console.error("FFmpeg MCP Server running on stdio (
|
|
537
|
+
console.error("FFmpeg MCP Server running on stdio (17 tools, 2 resources, 2 prompts active)");
|
|
346
538
|
}
|
|
347
539
|
main().catch((err) => {
|
|
348
540
|
console.error("Fatal error starting FFmpeg MCP server:", err);
|