@falkenslab/video-mcp 0.0.0-stage → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +87 -2
- package/dist/server.mjs +38091 -0
- package/package.json +60 -4
package/README.md
CHANGED
|
@@ -1,3 +1,88 @@
|
|
|
1
|
-
|
|
1
|
+
<div align="center">
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
<img src="https://raw.githubusercontent.com/falkenslab/agent-extensions/main/extensions/video/assets/logo.svg" alt="video-mcp logo: a video frame with a play button above two lines of subtitles" width="112"/>
|
|
4
|
+
|
|
5
|
+
# video-mcp
|
|
6
|
+
|
|
7
|
+
**Your AI agent understands a video and writes its subtitles, on your computer.**<br/>
|
|
8
|
+
Its captions read as a timed transcript, or the speech transcribed with faster-whisper when it has none, frames where the screen changes, and a WebVTT file ready to upload.
|
|
9
|
+
|
|
10
|
+

|
|
11
|
+

|
|
12
|
+
[](../../LICENSE)
|
|
13
|
+
|
|
14
|
+
</div>
|
|
15
|
+
|
|
16
|
+
video-mcp is an MCP server that lets an AI agent work with a video: a file on your computer, a YouTube video or another site's. It reads the captions the video already has (YouTube's, a `.vtt` or `.srt`, or a file's own), transcribes the speech when there are none, takes frames where the picture changes so the agent can look at the slides or the code on screen, and writes a `.vtt` subtitle file from a transcript, which Moodle and browsers take as a captions track. Everything runs on your computer with programs you install yourself (ffmpeg, faster-whisper, yt-dlp); nothing is uploaded. It's the server of Falken's Lab's [video extension](../../extensions/video/), which adds the skills that say how an agent uses it well.
|
|
17
|
+
|
|
18
|
+
## ✨ Why video-mcp
|
|
19
|
+
|
|
20
|
+
- 💬 **Captions first.** A video's own captions are instant and usually better than a transcription: the agent reads them before transcribing anything.
|
|
21
|
+
- 🎙️ **Transcription on your computer.** faster-whisper, with the model size you choose, on the CPU; the audio never leaves your computer.
|
|
22
|
+
- 🖼️ **What's on screen.** Frames where a slide changes, another window opens or code is written, each with its time, to match with what's said.
|
|
23
|
+
- 📝 **Subtitles a player takes.** WebVTT in order, never overlapping, two short lines each, with misheard names corrected throughout.
|
|
24
|
+
- 🧰 **Nothing installed for you.** `check_requirements` says what's missing and the command for your system; a tool that needs a missing program says the same.
|
|
25
|
+
|
|
26
|
+
## 🚀 Quick start
|
|
27
|
+
|
|
28
|
+
With [Node.js](https://nodejs.org) 20 or later and the programs you need ([ffmpeg](https://ffmpeg.org/download.html) for frames, [faster-whisper](https://github.com/SYSTRAN/faster-whisper#installation) for transcribing, [yt-dlp](https://github.com/yt-dlp/yt-dlp/wiki/Installation) for YouTube), build it here (`npm install`, `npm run build`) and add it to Claude Code:
|
|
29
|
+
|
|
30
|
+
```bash
|
|
31
|
+
claude mcp add video-mcp -- node /path/to/agent-extensions/mcps/video/dist/server.mjs
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
Then ask about a video. In an agent built on agent-kit, or in Claude Code with its skills, install the [video extension](../../extensions/video/) instead. Once it's on npm, `npx -y @falkenslab/video-mcp` will run it.
|
|
35
|
+
|
|
36
|
+
## 💬 What you can ask
|
|
37
|
+
|
|
38
|
+
👩🏫 As a teacher:
|
|
39
|
+
|
|
40
|
+
- "Make Spanish subtitles for my video lesson.mp4, ready to upload to Moodle."
|
|
41
|
+
- "Which of these YouTube videos have no subtitles?"
|
|
42
|
+
|
|
43
|
+
🧑🎓 As a student:
|
|
44
|
+
|
|
45
|
+
- "Summarize this video lecture with its chapters, and the code it shows."
|
|
46
|
+
- "What does the teacher say about static routes in this video, and when?"
|
|
47
|
+
|
|
48
|
+
## 🧰 What's inside
|
|
49
|
+
|
|
50
|
+
| Tool | What it does |
|
|
51
|
+
| --- | --- |
|
|
52
|
+
| `get_video_info` | A video's title, length, picture, sound, chapters and the captions it has (YouTube's tracks or a file's subtitle streams); only reads |
|
|
53
|
+
| `get_captions` | Its captions as a timed transcript, a page at a time: YouTube's (someone's, automatic or translated), a `.vtt` or `.srt` file or address, or a file's own; only reads |
|
|
54
|
+
| `transcribe_video` | The speech as a timed transcript when there are no captions: faster-whisper reads the video and transcribes it, without ffmpeg; kept as JSON for the next page and for subtitles |
|
|
55
|
+
| `extract_frames` | Frames where the picture changes, or at an interval, as JPEG files with their times |
|
|
56
|
+
| `write_subtitles` | A `.vtt` from a transcript, a `.srt` or `.vtt`, or cues, tidied as subtitles, with corrections |
|
|
57
|
+
| `clean_up_video` | Deletes what was kept for a video once it's done with: its downloads and frames, and its transcripts if asked; never the person's own file |
|
|
58
|
+
| `check_requirements` | ffmpeg, Python with faster-whisper and yt-dlp: found or not, their versions and how to install each; only reads |
|
|
59
|
+
| `server_info` | The version, the settings and the programs found; only reads |
|
|
60
|
+
|
|
61
|
+
## ⚙️ Settings
|
|
62
|
+
|
|
63
|
+
Environment variables, all optional; one left as an unexpanded `${user_config.*}` counts as not set:
|
|
64
|
+
|
|
65
|
+
| Variable | What it sets |
|
|
66
|
+
| --- | --- |
|
|
67
|
+
| `VIDEO_MCP_WORK_DIR` | Where downloads, transcripts and frames go, a folder per video (by default the extension's data folder, else `~/.video-mcp`) |
|
|
68
|
+
| `VIDEO_MCP_WHISPER_MODEL` | faster-whisper's model: `tiny`, `base`, `small` (the default), `medium`, `large-v3` or `turbo` |
|
|
69
|
+
| `VIDEO_MCP_LANGUAGE` | The videos' usual language (`es`, `en`…), for captions and transcription; detected when not set |
|
|
70
|
+
| `VIDEO_MCP_PYTHON` | The Python with faster-whisper, when it isn't the one on the PATH (a virtual environment's) |
|
|
71
|
+
| `VIDEO_MCP_WHISPER_DEVICE` | `cpu` (the default), `cuda` or `auto` |
|
|
72
|
+
|
|
73
|
+
## 🔧 How it works
|
|
74
|
+
|
|
75
|
+
- **Programs, never a shell.** ffmpeg, ffprobe and yt-dlp are found on the PATH (yt-dlp also as `python -m yt_dlp`), and run with an argument list. Python is `VIDEO_MCP_PYTHON`, else the first `python3` or `python` on the PATH (`python`, then `py -3` on Windows, skipping the Microsoft Store's stubs) that has faster-whisper.
|
|
76
|
+
- **Transcription.** A short Python script passed with `-c` has faster-whisper read the video (or its downloaded audio) itself, through the PyAV it brings, as padawan does, so ffmpeg isn't needed; it cuts the stretch asked for and runs faster-whisper (int8, voice activity detection to skip silences), printing a JSON line per segment, which the server reads as it comes and reports as progress. The transcript is kept as `transcript-<model>-<language>.json` in the video's folder. `clean_up_video` deletes the downloads and frames afterwards.
|
|
77
|
+
- **Frames.** One ffmpeg pass with a `select` filter: the first frame, a frame where the scene score passes the threshold (but not within 2 seconds of the last), and one at least every `every` seconds; `showinfo` gives each frame's time. Over `maxFrames`, an even spread is kept.
|
|
78
|
+
- **YouTube.** yt-dlp reads the video's details and captions tracks (`-J`) and downloads the chosen track as WebVTT; YouTube's rolling automatic captions are read once each. To transcribe or take frames, it downloads the audio alone, or the picture alone up to 1080p, into the video's folder, where they're reused.
|
|
79
|
+
|
|
80
|
+
## 🛠️ Developing
|
|
81
|
+
|
|
82
|
+
In this folder: `npm install`, then `npm run typecheck`, `npm run lint`, `npm test`, `npm run build` (one file, `dist/server.mjs`, with esbuild) and `npm run check` (the built server's tools, complete). The tests fake ffmpeg, ffprobe, yt-dlp and Python (`test/fake.ts`), so they need none of them. Until the package is on npm, the build also copies `dist/server.mjs` into `extensions/video/dist/`, which the extension runs and which is committed; CI fails if that copy is stale. `npm publish` runs all the checks first; releases are published to npm by the owner.
|
|
83
|
+
|
|
84
|
+
<div align="center">
|
|
85
|
+
|
|
86
|
+
Made with 💖 by Falken's Lab Team
|
|
87
|
+
|
|
88
|
+
</div>
|