vidclaude 0.2.0

package/README.md

@@ -0,0 +1,237 @@
+# vidclaude
+
+Multimodal video understanding for Claude Code. Extract frames, transcribe audio in 90+ languages, build temporal timelines — all from a single command. No API key needed.
+
+```bash
+pip install vidclaude
+vidclaude video.mp4 --mode standard --verbose
+```
+
+## What it does
+
+Drop a video in, get structured evidence out. Claude in your conversation does the thinking.
+
+```
+Video File
+  │
+  ├─ ffmpeg ──────────► Frames (adaptive, shot-aware sampling)
+  ├─ faster-whisper ──► Transcript with timestamps (large-v3, 90+ languages)
+  ├─ pytesseract ─────► On-screen text / OCR (optional)
+  ├─ scene detection ─► Shot boundaries
+  │
+  └─► Timeline ──► evidence.md ──► Claude reasons over it
+```
+
+Works with Hindi, English, Spanish, Japanese, Arabic — any of the 90+ languages Whisper supports. Language is auto-detected.
+
+## Prerequisites
+
+| Requirement | Install |
+|-------------|---------|
+| Python 3.10+ | [python.org](https://python.org) |
+| ffmpeg | Windows: `winget install ffmpeg` / macOS: `brew install ffmpeg` / Linux: `sudo apt install ffmpeg` |
+
+## Install
+
+```bash
+pip install vidclaude
+```
+
+That's it. First run downloads the Whisper model (~3GB, one time).
+
+## Usage
+
+### With Claude Code (recommended)
+
+Set up the skill once:
+
+```bash
+vidclaude --install-skill
+```
+
+Then in Claude Code, just say:
+
+> "analyze the video at C:/Users/me/Videos/meeting.mp4"
+>
+> "what does the speaker say about the budget in presentation.mp4?"
+>
+> "when does the logo appear in intro.mov?"
+
+Claude runs the extraction, reads the evidence report + frames, and answers your question. Your Max/Pro plan covers everything — no API key needed.
+
+Follow-up questions about the same video are instant (cached).
+
+### From the command line
+
+```bash
+# Standard analysis — good for most videos
+vidclaude video.mp4 --mode standard --verbose
+
+# Quick — fewer frames, faster
+vidclaude video.mp4 --mode quick
+
+# Deep — dense frames, full OCR, detailed
+vidclaude video.mp4 --mode deep --verbose
+
+# Batch process a folder
+vidclaude ./videos/ --verbose
+
+# Skip audio transcription
+vidclaude video.mp4 --no-audio
+
+# Force fresh extraction (ignore cache)
+vidclaude video.mp4 --no-cache
+```
+
+### Output
+
+Every run creates a `.vidcache/` directory next to the video:
+
+```
+.vidcache/a3f7b2c1/
+  evidence.md        ← Human-readable report (Claude reads this)
+  frames/            ← Extracted JPEG frames
+  transcript.json    ← Timestamped transcript
+  timeline.json      ← Unified event timeline
+  meta.json          ← Video metadata
+  shots.json         ← Shot boundaries
+  ocr.json           ← On-screen text
+  summaries.json     ← Scene/chapter summaries
+```
+
+## Modes
+
+| Mode | Frames | Whisper model | OCR | Best for |
+|------|--------|---------------|-----|----------|
+| `quick` | ~20, uniform | base | skip | Short clips, fast overview |
+| `standard` | ~60, shot-aware | large-v3 | keyframes | General use |
+| `deep` | ~150, burst sampling | large-v3 | all frames | Long videos, detailed analysis |
+
+**Smart frame budget**: If a video is too long for the frame limit, FPS is automatically reduced. Shot boundary frames are always prioritized.
+
+## CLI Reference
+
+```
+vidclaude [input] [options]
+
+positional:
+  input                    Video file or folder
+
+setup:
+  --install-skill          Copy SKILL.md to current dir for Claude Code
+
+processing:
+  --mode {quick,standard,deep}   Processing mode (default: standard)
+  -f, --fps N              Override frames per second
+  -m, --max-frames N       Override max frame count
+  --no-audio               Skip audio transcription
+  --no-ocr                 Skip OCR extraction
+  --no-cache               Force re-extraction
+
+output:
+  --extract                Print cache path summary
+  -o, --output FILE        Write output to file
+  --verbose                Show detailed progress
+  --batch-summary          Cross-video summary for folders
+```
+
+## How it works
+
+Built on a multi-layer video understanding architecture:
+
+- **Layer A — Ingestion**: Validates format (MP4, MOV, MKV, WebM, AVI), extracts metadata via ffprobe
+- **Layer B — Segmentation**: Detects shot boundaries using ffmpeg's scene change filter
+- **Layer C — Adaptive Sampling**: Content-aware frame selection — more frames at scene transitions, smart frame budgets per mode
+- **Layer D — Audio**: faster-whisper with large-v3 for multilingual transcription with word-level timestamps and VAD filtering
+- **Layer E — OCR**: pytesseract text extraction from key frames (optional)
+- **Layer G — Timeline**: Merges speech, OCR, and scene events into a single time-sorted list
+- **Layer I — Memory**: Hierarchical summaries for longer videos (scene → chapter → global)
+- **Layer J — Evidence Assembly**: Generates `evidence.md` with frame references, transcript, timeline for Claude to reason over
+
+## Intent-aware processing
+
+The tool classifies your question and adjusts the pipeline:
+
+| Question type | Example | What happens |
+|--------------|---------|-------------|
+| Description | "What happens in this video?" | Balanced extraction |
+| Moment retrieval | "When does the person stand up?" | Prioritizes transcript + timeline |
+| Temporal ordering | "Does X happen before Y?" | Prioritizes timeline events |
+| Counting | "How many cars appear?" | Denser frame sampling |
+| OCR / text | "What text is on the slide?" | Prioritizes OCR extraction |
+| Speech | "What did they say about revenue?" | Prioritizes transcript |
+
+## Optional extras
+
+```bash
+# OCR support (also needs Tesseract binary installed)
+pip install pytesseract
+
+# Standalone API mode (use outside Claude Code)
+pip install anthropic
+export ANTHROPIC_API_KEY=sk-...
+vidclaude video.mp4 --api -q "What happens in this video?"
+```
+
+## Examples
+
+**Analyze a meeting recording:**
+```bash
+vidclaude meeting.mp4 --mode standard --verbose
+# → 55 frames, full transcript, timeline
+# → In Claude Code: "summarize the key decisions from this meeting"
+```
+
+**Review security footage:**
+```bash
+vidclaude ./cameras/ --mode deep --verbose
+# → Batch processes all videos in the folder
+# → Dense frame extraction catches fast events
+```
+
+**Extract text from a lecture:**
+```bash
+pip install pytesseract
+vidclaude lecture.mp4 --mode standard --verbose
+# → Captures slide text via OCR + speaker transcript
+```
+
+**Quick check on a short clip:**
+```bash
+vidclaude clip.mp4 --mode quick
+# → 20 frames, fast whisper, done in seconds
+```
+
+## Troubleshooting
+
+| Problem | Fix |
+|---------|-----|
+| `ffmpeg not found` | Install: `winget install ffmpeg` (Windows) / `brew install ffmpeg` (Mac) |
+| `No module named 'faster_whisper'` | `pip install faster-whisper` |
+| Slow first run | Normal — downloading Whisper large-v3 model (~3GB, one time) |
+| Wrong language detected | Whisper auto-detects; usually correct. For edge cases, the transcript still captures phonetics |
+| Large `.vidcache` folder | Delete it to free space: `rm -rf .vidcache/` |
+| Want fresh extraction | Use `--no-cache` flag |
+| OCR not working | Install pytesseract + Tesseract binary |
+
+## Project structure
+
+```
+vidclaude/
+  cli.py          Argument parsing, orchestration, caching
+  models.py       Data model (VideoMeta, Frame, Shot, TranscriptChunk, etc.)
+  ingest.py       Video validation + ffprobe metadata
+  segment.py      Shot detection + adaptive frame sampling
+  audio.py        faster-whisper transcription (large-v3)
+  ocr.py          pytesseract text extraction
+  intent.py       Question intent classification
+  timeline.py     Temporal event merging
+  memory.py       Hierarchical summaries
+  reason.py       Evidence assembly + optional API mode
+  util.py         ffmpeg helpers, image encoding, caching
+  SKILL.md        Claude Code skill definition (bundled)
+```
+
+## License
+
+MIT

package/SKILL.md

@@ -0,0 +1,138 @@
+---
+name: video-understand
+description: >
+  Analyze video files using multimodal extraction (frames, audio transcript,
+  OCR, timeline) and Claude's reasoning. Use when the user asks to analyze,
+  understand, describe, or answer questions about a video file or folder of
+  videos. Triggers on: "analyze this video", "what happens in this video",
+  "describe the video", "video question", any path ending in
+  .mp4/.mov/.mkv/.webm followed by a question about it.
+tools: ["Bash", "Read", "Glob"]
+---
+
+# Video Understanding Skill
+
+Analyze videos by extracting visual frames, audio transcripts, OCR text, and
+temporal timelines, then reasoning over the combined evidence.
+
+## How It Works
+
+This skill uses a Python extraction pipeline (`video_understand.py`) that:
+1. Ingests the video and extracts metadata via ffprobe
+2. Detects shot boundaries using ffmpeg scene change detection
+3. Adaptively samples frames (more frames at scene transitions)
+4. Transcribes audio using OpenAI Whisper (if installed)
+5. Extracts on-screen text via OCR (if pytesseract installed)
+6. Builds a unified temporal timeline of all events
+7. Generates an `evidence.md` report + cached artifacts
+
+You (Claude) then read the evidence and frame images to reason over the video.
+
+## Step-by-Step Usage
+
+### Step 1: Extract evidence from the video
+
+Run the extraction pipeline:
+
+```bash
+python D:/ai_creative_stuff/claudevid/video_understand.py "<video_path>" --extract --mode standard --verbose
+```
+
+For quick analysis (fewer frames, faster):
+```bash
+python D:/ai_creative_stuff/claudevid/video_understand.py "<video_path>" --extract --mode quick --verbose
+```
+
+For deep analysis (more frames, OCR on all frames):
+```bash
+python D:/ai_creative_stuff/claudevid/video_understand.py "<video_path>" --extract --mode deep --verbose
+```
+
+### Step 2: Read the evidence report
+
+The script outputs a cache directory path. Read the evidence report:
+
+```bash
+# The cache path is printed by the script, e.g.:
+#   Cache: /path/to/.vidcache/a3f7b2c1
+# Read the evidence:
+cat /path/to/.vidcache/<hash>/evidence.md
+```
+
+Use the Read tool to read `evidence.md` from the cache directory.
+
+### Step 3: View key frames
+
+Read the frame images listed in evidence.md to see what's in the video.
+Select 5-10 representative frames spread across the video's duration.
+Use the Read tool on the frame image paths.
+
+### Step 4: Answer the question
+
+Reason over the combined evidence (timeline, transcript, OCR, frames) to
+answer the user's question. Ground claims in timestamps. Note uncertainties.
+
+### Follow-up Questions
+
+For follow-up questions about the same video, the cache already exists.
+Re-read the evidence.md and relevant frames — no need to re-extract.
+
+To force re-extraction: add `--no-cache` flag.
+
+## Batch Processing
+
+Process a folder of videos:
+```bash
+python D:/ai_creative_stuff/claudevid/video_understand.py "<folder_path>" --extract --mode standard --verbose
+```
+
+## CLI Reference
+
+| Flag | Default | Description |
+|------|---------|-------------|
+| `input` | required | Video file or folder path |
+| `--extract` | off | Extract only (skill mode, no API key needed) |
+| `-q "..."` | none | Question (for API standalone mode) |
+| `-f N` | mode default | Frames per second override |
+| `-m N` | mode default | Max frames override |
+| `--no-audio` | off | Skip audio transcription |
+| `--no-ocr` | off | Skip OCR extraction |
+| `--mode` | standard | quick / standard / deep |
+| `--verbose` | off | Detailed progress output |
+| `--no-cache` | off | Force re-extraction |
+| `--api` | off | Standalone mode (needs ANTHROPIC_API_KEY) |
+| `-o file` | stdout | Write output to file |
+
+## Modes Comparison
+
+| Aspect | quick | standard | deep |
+|--------|-------|----------|------|
+| Frame sampling | 0.2fps, max 20 | 0.5fps + shot boundaries, max 60 | 1.0fps + burst, max 150 |
+| Audio | whisper base | whisper base | whisper small |
+| OCR | skip | keyframes only | all frames |
+| Summaries | skip | for videos > 5min | always |
+
+## Prerequisites
+
+1. **Python 3.10+**
+2. **ffmpeg** on PATH — `winget install ffmpeg` or download from https://ffmpeg.org
+3. **Pillow**: `pip install Pillow`
+4. **Whisper** (optional): `pip install openai-whisper` (requires PyTorch)
+5. **Tesseract** (optional): Install Tesseract OCR + `pip install pytesseract`
+
+## Troubleshooting
+
+- **"ffmpeg not found"**: Ensure ffmpeg is on your PATH. Run `ffmpeg -version` to verify.
+- **"openai-whisper not installed"**: Audio will be skipped. Install with `pip install openai-whisper`.
+- **Slow frame extraction**: Use `--mode quick` or reduce with `-m 10`.
+- **Large cache**: Delete `.vidcache/` folders to free space.
+- **Re-analyze same video**: Use `--no-cache` to force fresh extraction.
+
+## Extension Ideas
+
+- **OmniVision offline footage review**: Process security/dashcam footage folders,
+  generate timeline reports, flag anomalous events
+- **Meeting summarization**: Extract slides (OCR) + transcript, generate meeting notes
+- **Content moderation**: Scan video batches for policy violations
+- **Sports analysis**: Dense frame extraction for play-by-play breakdown
+- **Accessibility**: Generate audio descriptions from visual content

package/bin/setup.js

@@ -0,0 +1,45 @@
+#!/usr/bin/env node
+
+const { execSync } = require("child_process");
+const fs = require("fs");
+const path = require("path");
+
+const reqPath = path.join(__dirname, "..", "requirements.txt");
+
+console.log("vidclaude: Installing Python dependencies...");
+
+// Check Python exists
+try {
+  execSync("python --version", { stdio: "pipe" });
+} catch {
+  console.warn(
+    "\n⚠ Python not found. You need Python 3.10+ to use vidclaude.\n" +
+      "  Install from: https://python.org\n" +
+      "  Then run: pip install -r requirements.txt\n"
+  );
+  process.exit(0); // Don't fail npm install
+}
+
+// Check ffmpeg exists
+try {
+  execSync("ffmpeg -version", { stdio: "pipe" });
+} catch {
+  console.warn(
+    "\n⚠ ffmpeg not found. You need ffmpeg to use vidclaude.\n" +
+      "  Windows: winget install ffmpeg\n" +
+      "  macOS:   brew install ffmpeg\n" +
+      "  Linux:   sudo apt install ffmpeg\n"
+  );
+}
+
+// Install Python deps
+try {
+  execSync(`pip install -r "${reqPath}"`, { stdio: "inherit" });
+  console.log("\nvidclaude: Setup complete!");
+  console.log("  Run: npx vidclaude video.mp4 --extract --mode standard --verbose\n");
+} catch {
+  console.warn(
+    "\n⚠ Failed to install Python dependencies.\n" +
+      "  Try manually: pip install -r requirements.txt\n"
+  );
+}

package/bin/vidclaude.js

@@ -0,0 +1,27 @@
+#!/usr/bin/env node
+
+const { spawn } = require("child_process");
+const path = require("path");
+
+const scriptPath = path.join(__dirname, "..", "video_understand.py");
+const args = process.argv.slice(2);
+
+// Pass all arguments through to the Python script
+const proc = spawn("python", [scriptPath, ...args], {
+  stdio: "inherit",
+});
+
+proc.on("error", (err) => {
+  if (err.code === "ENOENT") {
+    console.error(
+      "Error: Python not found. Install Python 3.10+ from https://python.org"
+    );
+    process.exit(1);
+  }
+  console.error(`Error: ${err.message}`);
+  process.exit(1);
+});
+
+proc.on("close", (code) => {
+  process.exit(code || 0);
+});

package/package.json

@@ -0,0 +1,31 @@
+{
+  "name": "vidclaude",
+  "version": "0.2.0",
+  "description": "Multimodal video understanding for Claude Code — extract frames, transcribe audio, build timelines from any video",
+  "bin": {
+    "vidclaude": "./bin/vidclaude.js"
+  },
+  "files": [
+    "bin/",
+    "vidclaude/",
+    "video_understand.py",
+    "requirements.txt",
+    "SKILL.md",
+    "README.md"
+  ],
+  "scripts": {
+    "postinstall": "node ./bin/setup.js"
+  },
+  "keywords": [
+    "video",
+    "claude",
+    "multimodal",
+    "whisper",
+    "analysis",
+    "claude-code"
+  ],
+  "license": "MIT",
+  "engines": {
+    "node": ">=16.0.0"
+  }
+}

package/requirements.txt

@@ -0,0 +1,2 @@
+Pillow>=10.0
+faster-whisper>=1.0

package/vidclaude/SKILL.md

@@ -0,0 +1,138 @@
+---
+name: video-understand
+description: >
+  Analyze video files using multimodal extraction (frames, audio transcript,
+  OCR, timeline) and Claude's reasoning. Use when the user asks to analyze,
+  understand, describe, or answer questions about a video file or folder of
+  videos. Triggers on: "analyze this video", "what happens in this video",
+  "describe the video", "video question", any path ending in
+  .mp4/.mov/.mkv/.webm followed by a question about it.
+tools: ["Bash", "Read", "Glob"]
+---
+
+# Video Understanding Skill
+
+Analyze videos by extracting visual frames, audio transcripts, OCR text, and
+temporal timelines, then reasoning over the combined evidence.
+
+## How It Works
+
+This skill uses a Python extraction pipeline (`video_understand.py`) that:
+1. Ingests the video and extracts metadata via ffprobe
+2. Detects shot boundaries using ffmpeg scene change detection
+3. Adaptively samples frames (more frames at scene transitions)
+4. Transcribes audio using OpenAI Whisper (if installed)
+5. Extracts on-screen text via OCR (if pytesseract installed)
+6. Builds a unified temporal timeline of all events
+7. Generates an `evidence.md` report + cached artifacts
+
+You (Claude) then read the evidence and frame images to reason over the video.
+
+## Step-by-Step Usage
+
+### Step 1: Extract evidence from the video
+
+Run the extraction pipeline:
+
+```bash
+python D:/ai_creative_stuff/claudevid/video_understand.py "<video_path>" --extract --mode standard --verbose
+```
+
+For quick analysis (fewer frames, faster):
+```bash
+python D:/ai_creative_stuff/claudevid/video_understand.py "<video_path>" --extract --mode quick --verbose
+```
+
+For deep analysis (more frames, OCR on all frames):
+```bash
+python D:/ai_creative_stuff/claudevid/video_understand.py "<video_path>" --extract --mode deep --verbose
+```
+
+### Step 2: Read the evidence report
+
+The script outputs a cache directory path. Read the evidence report:
+
+```bash
+# The cache path is printed by the script, e.g.:
+#   Cache: /path/to/.vidcache/a3f7b2c1
+# Read the evidence:
+cat /path/to/.vidcache/<hash>/evidence.md
+```
+
+Use the Read tool to read `evidence.md` from the cache directory.
+
+### Step 3: View key frames
+
+Read the frame images listed in evidence.md to see what's in the video.
+Select 5-10 representative frames spread across the video's duration.
+Use the Read tool on the frame image paths.
+
+### Step 4: Answer the question
+
+Reason over the combined evidence (timeline, transcript, OCR, frames) to
+answer the user's question. Ground claims in timestamps. Note uncertainties.
+
+### Follow-up Questions
+
+For follow-up questions about the same video, the cache already exists.
+Re-read the evidence.md and relevant frames — no need to re-extract.
+
+To force re-extraction: add `--no-cache` flag.
+
+## Batch Processing
+
+Process a folder of videos:
+```bash
+python D:/ai_creative_stuff/claudevid/video_understand.py "<folder_path>" --extract --mode standard --verbose
+```
+
+## CLI Reference
+
+| Flag | Default | Description |
+|------|---------|-------------|
+| `input` | required | Video file or folder path |
+| `--extract` | off | Extract only (skill mode, no API key needed) |
+| `-q "..."` | none | Question (for API standalone mode) |
+| `-f N` | mode default | Frames per second override |
+| `-m N` | mode default | Max frames override |
+| `--no-audio` | off | Skip audio transcription |
+| `--no-ocr` | off | Skip OCR extraction |
+| `--mode` | standard | quick / standard / deep |
+| `--verbose` | off | Detailed progress output |
+| `--no-cache` | off | Force re-extraction |
+| `--api` | off | Standalone mode (needs ANTHROPIC_API_KEY) |
+| `-o file` | stdout | Write output to file |
+
+## Modes Comparison
+
+| Aspect | quick | standard | deep |
+|--------|-------|----------|------|
+| Frame sampling | 0.2fps, max 20 | 0.5fps + shot boundaries, max 60 | 1.0fps + burst, max 150 |
+| Audio | whisper base | whisper base | whisper small |
+| OCR | skip | keyframes only | all frames |
+| Summaries | skip | for videos > 5min | always |
+
+## Prerequisites
+
+1. **Python 3.10+**
+2. **ffmpeg** on PATH — `winget install ffmpeg` or download from https://ffmpeg.org
+3. **Pillow**: `pip install Pillow`
+4. **Whisper** (optional): `pip install openai-whisper` (requires PyTorch)
+5. **Tesseract** (optional): Install Tesseract OCR + `pip install pytesseract`
+
+## Troubleshooting
+
+- **"ffmpeg not found"**: Ensure ffmpeg is on your PATH. Run `ffmpeg -version` to verify.
+- **"openai-whisper not installed"**: Audio will be skipped. Install with `pip install openai-whisper`.
+- **Slow frame extraction**: Use `--mode quick` or reduce with `-m 10`.
+- **Large cache**: Delete `.vidcache/` folders to free space.
+- **Re-analyze same video**: Use `--no-cache` to force fresh extraction.
+
+## Extension Ideas
+
+- **OmniVision offline footage review**: Process security/dashcam footage folders,
+  generate timeline reports, flag anomalous events
+- **Meeting summarization**: Extract slides (OCR) + transcript, generate meeting notes
+- **Content moderation**: Scan video batches for policy violations
+- **Sports analysis**: Dense frame extraction for play-by-play breakdown
+- **Accessibility**: Generate audio descriptions from visual content

package/vidclaude/__init__.py

@@ -0,0 +1,3 @@
+"""vidclaude — Multimodal video understanding powered by Claude."""
+
+__version__ = "0.2.0"