@vellumai/assistant 0.3.4 → 0.3.6

package/src/config/bundled-skills/media-processing/SKILL.md

@@ -0,0 +1,176 @@
+---
+name: "Media Processing"
+description: "Ingest and process media files (video, audio, image) through a 3-phase pipeline: preprocess, map (Gemini), and reduce (Claude)"
+metadata: {"vellum": {"emoji": "🎬"}}
+---
+
+Ingest and track processing of media files (video, audio, images) through a configurable 3-phase pipeline.
+
+## End-to-End Workflow
+
+The processing pipeline follows a sequential 3-phase flow:
+
+1. **Ingest** (`ingest_media`) — Register a media file, detect MIME type, extract duration, deduplicate by content hash.
+2. **Preprocess** (`extract_keyframes`) — Detect dead time, segment the video into windows, extract downscaled keyframes, build a subject registry, and write a pipeline manifest.
+3. **Map** (`analyze_keyframes`) — Send each segment's frames to Gemini 2.5 Flash with assistant-provided extraction instructions and a JSON Schema for guaranteed structured output. Supports concurrency pooling, cost tracking, resumability, and automatic retries.
+4. **Reduce / Query** (`query_media`) — Send all map output to Claude for intelligent analysis and Q&A. Supports arbitrary natural language queries about video content.
+5. **Clip** (`generate_clip`) — Extract video clips around specific moments.
+
+The processing pipeline service (`services/processing-pipeline.ts`) orchestrates phases 2-4 automatically with retries, resumability, and cancellation support.
+
+## Tools
+
+### ingest_media
+
+Register a media file for processing. Accepts an absolute file path, validates the file exists, detects MIME type, extracts duration (for video/audio via ffprobe), and registers the asset with content-hash deduplication.
+
+### media_status
+
+Query the processing status of a media asset. Returns the asset metadata along with per-stage progress details. Use this to monitor pipeline progress.
+
+### extract_keyframes
+
+Preprocess a video asset: detect dead time via mpdecimate, segment the video into windows, extract downscaled keyframes at regular intervals, build a subject registry, and write a pipeline manifest.
+
+Parameters:
+- `asset_id` (required) — ID of the media asset.
+- `interval_seconds` — Interval between keyframes (default: 3s).
+- `segment_duration` — Duration of each segment window (default: 20s).
+- `dead_time_threshold` — Sensitivity for dead-time detection (default: 0.02).
+- `section_config` — Path to a JSON file with manual section boundaries.
+- `skip_dead_time` — Whether to detect and skip dead time (default: true).
+- `short_edge` — Short edge resolution for downscaled frames in pixels (default: 480).
+
+### analyze_keyframes
+
+Map video segments through Gemini's structured output API. Reads frames from the preprocess manifest, sends each segment to Gemini with assistant-provided extraction instructions and a JSON Schema for guaranteed structured output. Supports concurrency pooling, cost tracking, resumability (skips segments with existing results), and automatic retries with exponential backoff.
+
+Parameters:
+- `asset_id` (required) — ID of the media asset.
+- `system_prompt` (required) — Extraction instructions for Gemini.
+- `output_schema` (required) — JSON Schema for structured output.
+- `context` — Additional context to include in the prompt.
+- `model` — Gemini model to use (default: `gemini-2.5-flash`).
+- `concurrency` — Maximum concurrent API requests (default: 10).
+- `max_retries` — Retry attempts per segment on failure (default: 3).
+
+### query_media
+
+Query video analysis data using natural language. Sends map output (from analyze_keyframes) to Claude for intelligent analysis and Q&A. Supports arbitrary questions about video content.
+
+Parameters:
+- `asset_id` (required) — ID of the media asset.
+- `query` (required) — Natural language query about the video data.
+- `system_prompt` — Optional system prompt for Claude.
+- `model` — LLM model to use (default: `claude-sonnet-4-6`).
+
+### generate_clip
+
+Extract a video clip from a media asset using ffmpeg. Applies configurable pre/post-roll padding (clamped to file boundaries), outputs the clip as a temporary file.
+
+### media_diagnostics
+
+Get a diagnostic report for a media asset. Returns:
+- **Processing stats**: total keyframes extracted.
+- **Per-stage status and timing**: which stages (preprocess, map, reduce) have run, how long each took, current progress.
+- **Failure reasons**: last error from any failed stage.
+- **Cost estimation**: based on segment count and Gemini 2.5 Flash pricing, plus a note about Claude reduce costs.
+
+## Services
+
+### Processing Pipeline (services/processing-pipeline.ts)
+
+Orchestrates the full processing pipeline with reliability features:
+- **Sequential execution**: preprocess, map, reduce.
+- **Retries**: Each stage is retried with exponential backoff and jitter (configurable max retries and base delay).
+- **Resumability**: Checks processing_stages to find the last completed stage and resumes from there. Safe to restart after crashes.
+- **Cancellation**: Cooperative cancellation via asset status. Set asset status to `cancelled` and the pipeline stops between stages.
+- **Idempotency**: Re-ingesting the same file hash is a no-op. Re-running a fully completed pipeline is also a no-op.
+- **Graceful degradation**: If a stage fails mid-batch (e.g., Gemini API errors), partial results are saved. The stage is marked as failed with the error details, and the pipeline stops without losing work.
+
+### Preprocess (services/preprocess.ts)
+
+Handles dead-time detection, video segmentation, keyframe extraction, and subject registry building. Writes a pipeline manifest consumed by the Map phase.
+
+### Gemini Map (services/gemini-map.ts)
+
+Sends video segments to Gemini 2.5 Flash with structured output schemas. Handles concurrency pooling, cost tracking, resumability, and retries.
+
+### Reduce (services/reduce.ts)
+
+Sends Map output to Claude as text for analysis. Two modes:
+- **One-shot merge**: assembles all Map results and sends to Claude with a system prompt.
+- **Interactive Q&A**: loads existing map output + user query, sends to Claude.
+
+### Concurrency Pool (services/concurrency-pool.ts)
+
+Limits concurrent API calls during the Map phase to avoid rate limiting.
+
+### Cost Tracker (services/cost-tracker.ts)
+
+Tracks estimated API costs during pipeline execution.
+
+## Operator Runbook
+
+### Monitoring Progress
+
+Use `media_status` to check the current state of any asset:
+- **registered** — Ingested but not yet processed.
+- **processing** — Pipeline is running.
+- **indexed** — All stages completed successfully.
+- **failed** — A stage failed. Check stage details for the error.
+
+The response includes per-stage progress (0-100%) so you can see exactly where processing stands.
+
+### Diagnosing Failures
+
+Use `media_diagnostics` to get a full diagnostic report:
+1. Check the `stages` array for any stage with `status: "failed"`.
+2. Read the `lastError` field for that stage to understand what went wrong.
+3. Check `durationMs` to see if a stage timed out or ran unusually long.
+4. Common failure causes:
+   - **preprocess**: ffmpeg not installed, corrupt video file, disk full.
+   - **map**: Gemini API key not configured, API rate limits, network errors.
+   - **reduce**: No LLM provider configured, no map output exists.
+
+After fixing the root cause, re-run the failed stage. The pipeline is resumable — it picks up from where it left off.
+
+### Cost Expectations
+
+The Map phase (Gemini 2.5 Flash) is the primary cost driver. Cost scales with video duration, keyframe interval, and segment size:
+
+| Video Duration | Interval | Keyframes | Segments (~10 frames each) | Estimated Map Cost |
+|----------------|----------|-----------|----------------------------|--------------------|
+| 30 min         | 3s       | ~600      | ~60                        | ~$0.06             |
+| 60 min         | 3s       | ~1,200    | ~120                       | ~$0.12             |
+| 90 min         | 3s       | ~1,800    | ~180                       | ~$0.18             |
+| 90 min         | 5s       | ~1,080    | ~108                       | ~$0.11             |
+
+The Reduce phase (Claude) adds a small additional cost per query. The `media_diagnostics` tool provides per-asset cost estimates.
+
+### Known Limitations
+
+- **ffmpeg required**: Keyframe extraction and clip generation require ffmpeg to be installed on the host.
+- **Single-file ingestion**: Each `ingest_media` call processes one file. Batch ingestion is not yet supported.
+- **Gemini rate limits**: The Map phase uses concurrency pooling (default 10) to stay within API limits. Reduce concurrency if you hit 429 errors.
+- **No real-time processing**: The pipeline processes pre-recorded media files. Live/streaming video is not supported.
+
+### Troubleshooting
+
+| Symptom | Likely Cause | Fix |
+|---------|-------------|-----|
+| "No keyframes found" | extract_keyframes not run or failed | Check preprocess stage status; re-run if needed |
+| "No map output found" | analyze_keyframes not run | Run analyze_keyframes with appropriate system_prompt and output_schema |
+| "No LLM provider available" | API key not configured | Add one in Settings |
+| Map phase slow | Large video, small interval | Increase interval_seconds or reduce concurrency |
+| Gemini returns errors | Rate limits or schema issues | Check max_retries setting; simplify output_schema if needed |
+| Pipeline stuck at "processing" | Stage crashed without updating status | Use `media_diagnostics` to find the stuck stage; re-run manually |
+
+## Usage Notes
+
+- The `ingest_media` tool requires an absolute path to a local file.
+- Supported media types: video (mp4, mov, avi, mkv, webm, etc.), audio (mp3, wav, m4a, etc.), and images (png, jpg, gif, webp, etc.).
+- For video and audio files, duration is automatically extracted via ffprobe (requires ffmpeg to be installed).
+- Duplicate files are detected by content hash and return the existing asset record.
+- The `analyze_keyframes` tool is marked as medium risk because it makes external API calls to Gemini, which incur costs.
+- All schema tables, services, and tool interfaces are media-generic. Domain-specific interpretation belongs in the system_prompt and output_schema parameters.

package/src/config/bundled-skills/media-processing/TOOLS.json

@@ -0,0 +1,230 @@
+{
+  "version": 1,
+  "tools": [
+    {
+      "name": "ingest_media",
+      "description": "Ingest a media file (video, audio, or image) for processing. Validates the file, detects MIME type, extracts duration for video/audio, registers the asset with content-hash dedup, and enqueues an initial processing job.",
+      "category": "media",
+      "risk": "low",
+      "input_schema": {
+        "type": "object",
+        "properties": {
+          "file_path": {
+            "type": "string",
+            "description": "Absolute path to a local media file (video, audio, or image)"
+          },
+          "title": {
+            "type": "string",
+            "description": "Optional human-readable title for the media asset. Defaults to the filename."
+          },
+          "metadata": {
+            "type": "object",
+            "description": "Optional JSON metadata to attach to the asset (e.g., pipeline config, source info)"
+          }
+        },
+        "required": ["file_path"]
+      },
+      "executor": "tools/ingest-media.ts",
+      "execution_target": "host"
+    },
+    {
+      "name": "media_status",
+      "description": "Query the processing status of one or more media assets, including per-stage progress details.",
+      "category": "media",
+      "risk": "low",
+      "input_schema": {
+        "type": "object",
+        "properties": {
+          "asset_id": {
+            "type": "string",
+            "description": "ID of a specific media asset to query"
+          },
+          "file_path": {
+            "type": "string",
+            "description": "File path to look up a media asset by its original path"
+          },
+          "status_filter": {
+            "type": "string",
+            "enum": ["registered", "processing", "indexed", "failed"],
+            "description": "Filter assets by processing status"
+          }
+        }
+      },
+      "executor": "tools/media-status.ts",
+      "execution_target": "host"
+    },
+    {
+      "name": "extract_keyframes",
+      "description": "Preprocess a video asset: detect dead time, segment into windows, extract downscaled keyframes, build a subject registry, and write a pipeline manifest. Replaces the old simple keyframe extraction with a full Phase 0 preprocess pipeline.",
+      "category": "media",
+      "risk": "low",
+      "input_schema": {
+        "type": "object",
+        "properties": {
+          "asset_id": {
+            "type": "string",
+            "description": "ID of the media asset (must be a video)"
+          },
+          "interval_seconds": {
+            "type": "number",
+            "description": "Interval between keyframes in seconds. Default: 3"
+          },
+          "segment_duration": {
+            "type": "number",
+            "description": "Duration of each segment window in seconds. Default: 20"
+          },
+          "dead_time_threshold": {
+            "type": "number",
+            "description": "Sensitivity threshold for mpdecimate dead-time detection. Default: 0.02"
+          },
+          "section_config": {
+            "type": "string",
+            "description": "Path to a JSON file with manual section boundary definitions"
+          },
+          "skip_dead_time": {
+            "type": "boolean",
+            "description": "Whether to detect and skip dead time. Default: true"
+          },
+          "short_edge": {
+            "type": "number",
+            "description": "Short edge resolution for downscaled frames in pixels. Default: 480"
+          }
+        },
+        "required": ["asset_id"]
+      },
+      "executor": "tools/extract-keyframes.ts",
+      "execution_target": "host"
+    },
+    {
+      "name": "analyze_keyframes",
+      "description": "Map video segments through Gemini's structured output API for vision-based analysis. Reads frames from the preprocess manifest, sends each segment to Gemini with assistant-provided extraction instructions and a JSON Schema for guaranteed structured output. Supports concurrency pooling, cost tracking, resumability (skips segments with existing results), and automatic retries with exponential backoff.",
+      "category": "media",
+      "risk": "medium",
+      "input_schema": {
+        "type": "object",
+        "properties": {
+          "asset_id": {
+            "type": "string",
+            "description": "ID of the media asset whose segments to analyze"
+          },
+          "system_prompt": {
+            "type": "string",
+            "description": "Assistant-provided extraction instructions for Gemini (e.g., what to look for in the frames)"
+          },
+          "output_schema": {
+            "type": "object",
+            "description": "JSON Schema for structured output — Gemini will enforce this schema on the response"
+          },
+          "context": {
+            "type": "object",
+            "description": "Additional context to include in the prompt (e.g., subject registry, domain-specific info)"
+          },
+          "model": {
+            "type": "string",
+            "description": "Gemini model to use. Default: 'gemini-2.5-flash'"
+          },
+          "concurrency": {
+            "type": "number",
+            "minimum": 1,
+            "description": "Maximum concurrent Gemini API requests. Default: 10"
+          },
+          "max_retries": {
+            "type": "number",
+            "minimum": 0,
+            "description": "Maximum retry attempts per segment on failure. Default: 3"
+          }
+        },
+        "required": ["asset_id", "system_prompt", "output_schema"]
+      },
+      "executor": "tools/analyze-keyframes.ts",
+      "execution_target": "host"
+    },
+    {
+      "name": "query_media",
+      "description": "Query video analysis data using natural language. Sends map output (from analyze_keyframes) to Claude for intelligent analysis and Q&A. Supports arbitrary questions about video content — Claude reads the full structured analysis and answers based on the data.",
+      "category": "media",
+      "risk": "low",
+      "input_schema": {
+        "type": "object",
+        "properties": {
+          "asset_id": {
+            "type": "string",
+            "description": "ID of the media asset to query"
+          },
+          "query": {
+            "type": "string",
+            "description": "Natural language query about the video data (e.g., 'What happens at the 5 minute mark?', 'Summarize the key events', 'Are there any scoring plays?')"
+          },
+          "system_prompt": {
+            "type": "string",
+            "description": "Optional system prompt for Claude to customize analysis behavior"
+          },
+          "model": {
+            "type": "string",
+            "description": "LLM model to use for analysis. Default: 'claude-sonnet-4-6'"
+          }
+        },
+        "required": ["asset_id", "query"]
+      },
+      "executor": "tools/query-media-events.ts",
+      "execution_target": "host"
+    },
+    {
+      "name": "generate_clip",
+      "description": "Extract a video clip from a media asset using ffmpeg. Applies configurable pre/post-roll padding (clamped to file boundaries), outputs the clip as a temporary file, and registers it as an attachment for in-chat delivery.",
+      "category": "media",
+      "risk": "low",
+      "input_schema": {
+        "type": "object",
+        "properties": {
+          "asset_id": {
+            "type": "string",
+            "description": "ID of the media asset (must be a video)"
+          },
+          "start_time": {
+            "type": "number",
+            "description": "Start time of the clip in seconds"
+          },
+          "end_time": {
+            "type": "number",
+            "description": "End time of the clip in seconds"
+          },
+          "pre_roll": {
+            "type": "number",
+            "description": "Seconds of padding before start_time. Default: 3"
+          },
+          "post_roll": {
+            "type": "number",
+            "description": "Seconds of padding after end_time. Default: 2"
+          },
+          "output_format": {
+            "type": "string",
+            "enum": ["mp4", "webm", "mov"],
+            "description": "Output video format. Default: 'mp4'"
+          }
+        },
+        "required": ["asset_id", "start_time", "end_time"]
+      },
+      "executor": "tools/generate-clip.ts",
+      "execution_target": "host"
+    },
+    {
+      "name": "media_diagnostics",
+      "description": "Get a diagnostic report for a media asset including processing stats, per-stage timing (preprocess/map/reduce), failure reasons, and Gemini-based cost estimation.",
+      "category": "media",
+      "risk": "low",
+      "input_schema": {
+        "type": "object",
+        "properties": {
+          "asset_id": {
+            "type": "string",
+            "description": "ID of the media asset to diagnose"
+          }
+        },
+        "required": ["asset_id"]
+      },
+      "executor": "tools/media-diagnostics.ts",
+      "execution_target": "host"
+    }
+  ]
+}

package/src/config/bundled-skills/media-processing/__tests__/concurrency-pool.test.ts

@@ -0,0 +1,77 @@
+import { describe, expect, it } from 'bun:test';
+import { ConcurrencyPool } from '../services/concurrency-pool.js';
+
+describe('ConcurrencyPool', () => {
+  it('allows up to maxConcurrency tasks to run simultaneously', async () => {
+    const pool = new ConcurrencyPool(2);
+
+    await pool.acquire();
+    await pool.acquire();
+
+    expect(pool.activeCount).toBe(2);
+    expect(pool.waitingCount).toBe(0);
+
+    // Third acquire should not resolve immediately
+    let thirdResolved = false;
+    const thirdPromise = pool.acquire().then(() => {
+      thirdResolved = true;
+    });
+
+    // Yield the microtask queue so the promise chain can settle if it were going to
+    await Promise.resolve();
+    expect(thirdResolved).toBe(false);
+    expect(pool.waitingCount).toBe(1);
+
+    // Release one slot — third should now resolve
+    pool.release();
+    await thirdPromise;
+    expect(thirdResolved).toBe(true);
+    expect(pool.activeCount).toBe(2);
+    expect(pool.waitingCount).toBe(0);
+  });
+
+  it('releases slots on error so the pool does not leak', async () => {
+    const pool = new ConcurrencyPool(1);
+
+    const runTask = async () => {
+      await pool.acquire();
+      try {
+        throw new Error('simulated failure');
+      } finally {
+        pool.release();
+      }
+    };
+
+    // First task errors and releases its slot
+    await expect(runTask()).rejects.toThrow('simulated failure');
+    expect(pool.activeCount).toBe(0);
+
+    // Pool should still be usable after the error
+    await pool.acquire();
+    expect(pool.activeCount).toBe(1);
+    pool.release();
+  });
+
+  it('rejects maxConcurrency less than 1', () => {
+    expect(() => new ConcurrencyPool(0)).toThrow('maxConcurrency must be at least 1');
+  });
+
+  it('defaults to maxConcurrency of 10', async () => {
+    const pool = new ConcurrencyPool();
+
+    // Acquire 10 slots — all should resolve immediately
+    for (let i = 0; i < 10; i++) {
+      await pool.acquire();
+    }
+    expect(pool.activeCount).toBe(10);
+
+    // 11th should queue
+    let eleventhResolved = false;
+    pool.acquire().then(() => {
+      eleventhResolved = true;
+    });
+    await Promise.resolve();
+    expect(eleventhResolved).toBe(false);
+    expect(pool.waitingCount).toBe(1);
+  });
+});

package/src/config/bundled-skills/media-processing/__tests__/cost-tracker.test.ts

@@ -0,0 +1,69 @@
+import { describe, expect, it } from 'bun:test';
+import { CostTracker } from '../services/cost-tracker.js';
+
+describe('CostTracker', () => {
+  it('accumulates costs across multiple segments', () => {
+    const tracker = new CostTracker();
+
+    tracker.record({
+      segmentId: 'seg-001',
+      model: 'gemini-2.5-flash',
+      inputTokens: 1_000_000,
+      outputTokens: 0,
+    });
+
+    tracker.record({
+      segmentId: 'seg-002',
+      model: 'gemini-2.5-flash',
+      inputTokens: 0,
+      outputTokens: 1_000_000,
+    });
+
+    const summary = tracker.getSummary();
+    expect(summary.segmentCount).toBe(2);
+    expect(summary.totalInputTokens).toBe(1_000_000);
+    expect(summary.totalOutputTokens).toBe(1_000_000);
+    // $0.15 for 1M input + $0.60 for 1M output = $0.75
+    expect(summary.totalEstimatedUSD).toBeCloseTo(0.75, 6);
+  });
+
+  it('computes per-entry costs using Gemini 2.5 Flash pricing', () => {
+    const tracker = new CostTracker();
+
+    const entry = tracker.record({
+      segmentId: 'seg-010',
+      model: 'gemini-2.5-flash',
+      inputTokens: 200_000,
+      outputTokens: 50_000,
+    });
+
+    // Input: 200k * ($0.15 / 1M) = $0.03
+    // Output: 50k * ($0.60 / 1M) = $0.03
+    // Total: $0.06
+    expect(entry.estimatedUSD).toBeCloseTo(0.06, 6);
+    expect(entry.segmentId).toBe('seg-010');
+    expect(entry.model).toBe('gemini-2.5-flash');
+  });
+
+  it('returns an empty summary when no entries have been recorded', () => {
+    const tracker = new CostTracker();
+    const summary = tracker.getSummary();
+
+    expect(summary.segmentCount).toBe(0);
+    expect(summary.totalInputTokens).toBe(0);
+    expect(summary.totalOutputTokens).toBe(0);
+    expect(summary.totalEstimatedUSD).toBe(0);
+    expect(summary.entries).toHaveLength(0);
+  });
+
+  it('preserves entry order in summary', () => {
+    const tracker = new CostTracker();
+
+    tracker.record({ segmentId: 'a', model: 'm', inputTokens: 100, outputTokens: 200 });
+    tracker.record({ segmentId: 'b', model: 'm', inputTokens: 300, outputTokens: 400 });
+    tracker.record({ segmentId: 'c', model: 'm', inputTokens: 500, outputTokens: 600 });
+
+    const ids = tracker.getSummary().entries.map((e) => e.segmentId);
+    expect(ids).toEqual(['a', 'b', 'c']);
+  });
+});