@vellumai/assistant 0.4.26 → 0.4.29

package/src/config/bundled-skills/mcp-setup/SKILL.md

@@ -0,0 +1,75 @@
+---
+name: "MCP Setup"
+description: "Add, authenticate, list, and remove MCP (Model Context Protocol) servers"
+user-invocable: false
+metadata: { "vellum": { "emoji": "🔌" } }
+---
+
+Help users configure MCP servers so external tools (e.g. Linear, GitHub, Notion) become available in conversations.
+
+## CLI Commands
+
+All commands use `vellum mcp`. Run them via `host_bash`.
+
+### List servers
+
+```
+vellum mcp list
+```
+
+Shows all configured servers with connection status, transport type, and URL.
+
+### Add a server
+
+```
+vellum mcp add <name> -t <transport-type> -u <url> [-r <risk>] [--disabled]
+```
+
+- `<name>` — unique identifier (e.g. `linear`, `github`)
+- `-t` — transport type: `stdio`, `sse`, or `streamable-http`
+- `-u` — server URL (required for `sse`/`streamable-http`)
+- `-c` — command (required for `stdio`), `-a` for args
+- `-r` — risk level: `low`, `medium`, or `high` (default: `high`)
+
+Examples:
+
+```
+vellum mcp add linear -t streamable-http -u https://mcp.linear.app/mcp
+vellum mcp add context7 -t streamable-http -u https://mcp.context7.com/mcp -r low
+vellum mcp add local-db -t stdio -c npx -a -y @my/mcp-server
+```
+
+### Authenticate (OAuth)
+
+```
+vellum mcp auth <name>
+```
+
+Opens the user's browser for OAuth authorization. Only works for `sse`/`streamable-http` servers that require authentication. After the user completes login in the browser, tokens are saved automatically.
+
+Use this when:
+
+- A server shows `! Needs authentication` in `vellum mcp list`
+- An MCP tool call fails with an auth/token error
+- Setting up a new OAuth-protected server for the first time
+
+### Remove a server
+
+```
+vellum mcp remove <name>
+```
+
+Removes the server config and cleans up any stored OAuth credentials.
+
+## After Changes
+
+After adding, removing, or authenticating a server, the user must **quit and relaunch the Vellum app** for changes to take effect. The app runs its own assistant process — `vellum daemon restart` only restarts the CLI assistant, which is a separate process.
+
+Tell the user: "Please quit and relaunch the Vellum app, then start a new conversation."
+
+## When to Use
+
+- User asks to connect/set up an external service via MCP
+- User asks "what MCP servers do I have?"
+- An MCP tool returns an auth error — offer to run `vellum mcp auth <name>`
+- User wants to remove an MCP integration

package/src/config/bundled-skills/media-processing/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: "Media Processing"
 description: "Ingest and process media files (video, audio, image) through a 3-phase pipeline: preprocess, map (Gemini), and reduce (Claude)"
-metadata: {"vellum": {"emoji": "🎬"}}
+metadata: { "vellum": { "emoji": "🎬" } }
 ---
 
 Ingest and track processing of media files (video, audio, images) through a configurable 3-phase pipeline.
@@ -33,25 +33,35 @@ Query the processing status of a media asset. Returns the asset metadata along w
 Preprocess a video asset: detect dead time via mpdecimate, segment the video into windows, extract downscaled keyframes at regular intervals, build a subject registry, and write a pipeline manifest.
 
 Parameters:
+
 - `asset_id` (required) — ID of the media asset.
 - `interval_seconds` — Interval between keyframes (default: 1s). Use 0.5s for sports/action content where frame density matters.
 - `segment_duration` — Duration of each segment window (default: 15s).
 - `dead_time_threshold` — Sensitivity for dead-time detection (default: 0.02).
 - `section_config` — Path to a JSON file with manual section boundaries.
-- `skip_dead_time` — Whether to detect and skip dead time (default: false). Dead-time detection can be too aggressive for continuous action video like sports — it may incorrectly skip live play. Enable only for content with clear idle periods (e.g., lectures, surveillance footage).
+- `detect_dead_time` — Whether to detect and skip dead time (default: false). Dead-time detection can be too aggressive for continuous action video like sports — it may incorrectly skip live play. Enable only for content with clear idle periods (e.g., lectures, surveillance footage).
 - `short_edge` — Short edge resolution for downscaled frames in pixels (default: 480).
+- `include_audio` — Whether to extract and transcribe audio for each segment (default: false). When enabled, each segment's audio is transcribed and stored alongside visual frames.
+- `transcription_mode` — Transcription backend: `'api'` (OpenAI Whisper cloud) or `'local'` (whisper.cpp on-device). Default: `'local'`. The `'api'` mode requires an OpenAI API key configured in settings.
 
 ### analyze_keyframes
 
-Map video segments through Gemini's structured output API. Reads frames from the preprocess manifest, sends each segment to Gemini with assistant-provided extraction instructions and a JSON Schema for guaranteed structured output. Supports concurrency pooling, cost tracking, resumability (skips segments with existing results), and automatic retries with exponential backoff.
+Map video segments through Gemini's structured output API. Supports two modes:
+
+- **`keyframes`** (default) — Reads frames from the preprocess manifest, sends each segment's images to Gemini. Requires `extract_keyframes` to be run first. Best for longer videos (> 1 hour) or when you need fine-grained control over frame selection (interval, segment duration, dead-time skipping).
+- **`direct_video`** — Uploads the video file directly to Gemini's Files API. Gemini sees actual motion and temporal context instead of static frames. Best for shorter videos (< 1 hour) where temporal context matters (detecting actions, transitions, motion patterns). Has a 2 GB file size limit. Does not require `extract_keyframes` preprocessing.
+
+Both modes produce the same `MapOutput` format, so `query_media` works identically regardless of which mode was used.
 
 Parameters:
+
 - `asset_id` (required) — ID of the media asset.
 - `system_prompt` (required) — Extraction instructions for Gemini.
 - `output_schema` (required) — JSON Schema for structured output.
+- `mode` — Analysis mode: `'keyframes'` (default) or `'direct_video'`.
 - `context` — Additional context to include in the prompt.
 - `model` — Gemini model to use (default: `gemini-2.5-flash`).
-- `concurrency` — Maximum concurrent API requests (default: 10).
+- `concurrency` — Maximum concurrent API requests (default: 10, keyframes mode only).
 - `max_retries` — Retry attempts per segment on failure (default: 3).
 
 ### query_media
@@ -59,6 +69,7 @@ Parameters:
 Query video analysis data using natural language. Sends map output (from analyze_keyframes) to Claude for intelligent analysis and Q&A. Supports arbitrary questions about video content.
 
 Parameters:
+
 - `asset_id` (required) — ID of the media asset.
 - `query` (required) — Natural language query about the video data.
 - `system_prompt` — Optional system prompt for Claude.
@@ -71,6 +82,7 @@ Extract a video clip from a media asset using ffmpeg. Applies configurable pre/p
 ### media_diagnostics
 
 Get a diagnostic report for a media asset. Returns:
+
 - **Processing stats**: total keyframes extracted.
 - **Per-stage status and timing**: which stages (preprocess, map, reduce) have run, how long each took, current progress.
 - **Failure reasons**: last error from any failed stage.
@@ -81,6 +93,7 @@ Get a diagnostic report for a media asset. Returns:
 ### Processing Pipeline (services/processing-pipeline.ts)
 
 Orchestrates the full processing pipeline with reliability features:
+
 - **Sequential execution**: preprocess, map, reduce.
 - **Retries**: Each stage is retried with exponential backoff and jitter (configurable max retries and base delay).
 - **Resumability**: Checks processing_stages to find the last completed stage and resumes from there. Safe to restart after crashes.
@@ -99,6 +112,7 @@ Sends video segments to Gemini 2.5 Flash with structured output schemas. Handles
 ### Reduce (services/reduce.ts)
 
 Sends Map output to Claude as text for analysis. Two modes:
+
 - **One-shot merge**: assembles all Map results and sends to Claude with a system prompt.
 - **Interactive Q&A**: loads existing map output + user query, sends to Claude.
 
@@ -110,6 +124,24 @@ Limits concurrent API calls during the Map phase to avoid rate limiting.
 
 Tracks estimated API costs during pipeline execution.
 
+## Audio + Vision Multimodal Analysis
+
+When `include_audio` is enabled on `extract_keyframes`, the pipeline transcribes each segment's audio track and attaches the transcript to the segment data. During the Map phase (`analyze_keyframes`), Gemini receives both the visual frames and the audio transcript for each segment, enabling multimodal analysis that combines what is seen with what is said.
+
+This is useful for:
+
+- **Lectures and presentations**: Correlate slide content (visual) with speaker narration (audio).
+- **Sports broadcasts**: Combine on-screen action with commentary for richer event detection.
+- **Meetings and interviews**: Pair facial expressions and gestures with spoken dialogue.
+- **Tutorials and demos**: Link on-screen actions with verbal instructions.
+
+Transcription modes:
+
+- **`local`** (default): Uses whisper.cpp for on-device transcription. Requires `whisper-cpp` to be installed (`brew install whisper-cpp`). No API costs, but slower.
+- **`api`**: Uses the OpenAI Whisper API for cloud-based transcription. Faster and more accurate, but requires an OpenAI API key and incurs per-minute costs.
+
+The audio transcription degrades gracefully — if transcription fails for a segment (missing tools, no audio track, API errors), the segment proceeds with visual-only analysis.
+
 ## Best Practices
 
 ### Map Prompt Strategy: Go Broad, Not Targeted
@@ -168,29 +200,36 @@ Be specific and factual. Describe what you see, not what you infer happened betw
 
 ### Clip Delivery
 
-The `generate_clip` tool outputs clips as temporary files. These may not deliver reliably via sandbox attachments. For reliable delivery, use `host_bash` + ffmpeg to save clips to a user-specified location as a fallback.
+The `generate_clip` tool automatically opens clips in the user's default video player after extraction. Clips are saved persistently in the asset's pipeline directory (`pipeline/<assetId>/clips/`). The `clipPath` field in the tool response contains the absolute file path.
+
+The tool handles high-bitrate and incompatible codec sources automatically — it tries stream copy first for speed, then falls back to H.264 re-encoding if needed. **Always use `generate_clip` rather than manual ffmpeg commands.**
+
+Always provide a descriptive `title` parameter (e.g. `"snow-dive-closeup"`, `"goal-celebration"`) so clips get meaningful filenames instead of timestamp-based names.
 
 ## Known Limitations — Vision Analysis
 
 Gemini performs well at **spatial/descriptive analysis** from static keyframes:
+
 - Player positions, formations, and spacing
 - Jersey numbers and identifying features
 - Ball location and which team has possession
 - Score and on-screen text
 - Camera angles and scene composition
 
-Gemini **hallucinates when asked to detect fast temporal events** from static frames, regardless of frame density:
+Gemini **hallucinates when asked to detect fast temporal events** from static frames (keyframes mode), regardless of frame density:
+
 - Turnovers, steals, fouls, and specific plays
 - Fast transitions and split-second actions
 - Causality between frames (what "happened" vs. what's visible)
 
-The model is good at describing **what is there** but bad at detecting **what happened**. Structure your map prompts and queries accordingly — ask the model to describe scenes, then use `query_media` (Claude) to reason about patterns and events across the descriptive data.
+The model is good at describing **what is there** but bad at detecting **what happened** from static frames. For content where temporal context matters, consider using `mode: 'direct_video'` which lets Gemini see actual motion. For keyframes mode, structure your map prompts and queries accordingly — ask the model to describe scenes, then use `query_media` (Claude) to reason about patterns and events across the descriptive data.
 
 ## Operator Runbook
 
 ### Monitoring Progress
 
 Use `media_status` to check the current state of any asset:
+
 - **registered** — Ingested but not yet processed.
 - **processing** — Pipeline is running.
 - **indexed** — All stages completed successfully.
@@ -201,6 +240,7 @@ The response includes per-stage progress (0-100%) so you can see exactly where p
 ### Diagnosing Failures
 
 Use `media_diagnostics` to get a full diagnostic report:
+
 1. Check the `stages` array for any stage with `status: "failed"`.
 2. Read the `lastError` field for that stage to understand what went wrong.
 3. Check `durationMs` to see if a stage timed out or ran unusually long.
@@ -224,14 +264,14 @@ Use `media_diagnostics` to get per-asset cost estimates. The Map phase (Gemini)
 
 ### Troubleshooting
 
-| Symptom | Likely Cause | Fix |
-|---------|-------------|-----|
-| "No keyframes found" | extract_keyframes not run or failed | Check preprocess stage status; re-run if needed |
-| "No map output found" | analyze_keyframes not run | Run analyze_keyframes with appropriate system_prompt and output_schema |
-| "No LLM provider available" | API key not configured | Add one in Settings |
-| Map phase slow | Large video, small interval | Increase interval_seconds or reduce concurrency |
-| Gemini returns errors | Rate limits or schema issues | Check max_retries setting; simplify output_schema if needed |
-| Pipeline stuck at "processing" | Stage crashed without updating status | Use `media_diagnostics` to find the stuck stage; re-run manually |
+| Symptom                        | Likely Cause                          | Fix                                                                    |
+| ------------------------------ | ------------------------------------- | ---------------------------------------------------------------------- |
+| "No keyframes found"           | extract_keyframes not run or failed   | Check preprocess stage status; re-run if needed                        |
+| "No map output found"          | analyze_keyframes not run             | Run analyze_keyframes with appropriate system_prompt and output_schema |
+| "No LLM provider available"    | API key not configured                | Add one in Settings                                                    |
+| Map phase slow                 | Large video, small interval           | Increase interval_seconds or reduce concurrency                        |
+| Gemini returns errors          | Rate limits or schema issues          | Check max_retries setting; simplify output_schema if needed            |
+| Pipeline stuck at "processing" | Stage crashed without updating status | Use `media_diagnostics` to find the stuck stage; re-run manually       |
 
 ## Usage Notes
 

package/src/config/bundled-skills/media-processing/TOOLS.json

@@ -81,13 +81,22 @@
             "type": "string",
             "description": "Path to a JSON file with manual section boundary definitions"
           },
-          "skip_dead_time": {
+          "detect_dead_time": {
             "type": "boolean",
-            "description": "Whether to detect and skip dead time. Default: false. Can be too aggressive for continuous action video like sports."
+            "description": "Whether to detect and skip dead time (static/idle frames). Default: false. Can be too aggressive for continuous action video like sports."
           },
           "short_edge": {
             "type": "number",
             "description": "Short edge resolution for downscaled frames in pixels. Default: 480"
+          },
+          "include_audio": {
+            "type": "boolean",
+            "description": "Whether to extract and transcribe audio for each segment. Default: false."
+          },
+          "transcription_mode": {
+            "type": "string",
+            "enum": ["api", "local"],
+            "description": "Transcription backend: 'api' (OpenAI Whisper cloud) or 'local' (whisper.cpp). Default: 'local'."
           }
         },
         "required": ["asset_id"]
@@ -107,6 +116,11 @@
             "type": "string",
             "description": "ID of the media asset whose segments to analyze"
           },
+          "mode": {
+            "type": "string",
+            "enum": ["keyframes", "direct_video"],
+            "description": "Analysis mode: 'keyframes' extracts frames and sends images (default), 'direct_video' uploads video directly to Gemini. Direct video lets Gemini see motion/temporal context but has a 2GB file size limit."
+          },
           "system_prompt": {
             "type": "string",
             "description": "Assistant-provided extraction instructions for Gemini (e.g., what to look for in the frames)"
@@ -201,6 +215,10 @@
             "type": "string",
             "enum": ["mp4", "webm", "mov"],
             "description": "Output video format. Default: 'mp4'"
+          },
+          "title": {
+            "type": "string",
+            "description": "Short descriptive title for the clip (e.g. 'snow-dive-closeup', 'goal-celebration'). Used as the filename. If omitted, falls back to timestamp-based naming."
           }
         },
         "required": ["asset_id", "start_time", "end_time"]

package/src/config/bundled-skills/media-processing/__tests__/concurrency-pool.test.ts

@@ -1,9 +1,9 @@
-import { describe, expect, it } from 'bun:test';
+import { describe, expect, it } from "bun:test";
 
-import { ConcurrencyPool } from '../services/concurrency-pool.js';
+import { ConcurrencyPool } from "../services/concurrency-pool.js";
 
-describe('ConcurrencyPool', () => {
-  it('allows up to maxConcurrency tasks to run simultaneously', async () => {
+describe("ConcurrencyPool", () => {
+  it("allows up to maxConcurrency tasks to run simultaneously", async () => {
     const pool = new ConcurrencyPool(2);
 
     await pool.acquire();
@@ -31,20 +31,20 @@ describe('ConcurrencyPool', () => {
     expect(pool.waitingCount).toBe(0);
   });
 
-  it('releases slots on error so the pool does not leak', async () => {
+  it("releases slots on error so the pool does not leak", async () => {
     const pool = new ConcurrencyPool(1);
 
     const runTask = async () => {
       await pool.acquire();
       try {
-        throw new Error('simulated failure');
+        throw new Error("simulated failure");
       } finally {
         pool.release();
       }
     };
 
     // First task errors and releases its slot
-    await expect(runTask()).rejects.toThrow('simulated failure');
+    await expect(runTask()).rejects.toThrow("simulated failure");
     expect(pool.activeCount).toBe(0);
 
     // Pool should still be usable after the error
@@ -53,11 +53,13 @@ describe('ConcurrencyPool', () => {
     pool.release();
   });
 
-  it('rejects maxConcurrency less than 1', () => {
-    expect(() => new ConcurrencyPool(0)).toThrow('maxConcurrency must be at least 1');
+  it("rejects maxConcurrency less than 1", () => {
+    expect(() => new ConcurrencyPool(0)).toThrow(
+      "maxConcurrency must be at least 1",
+    );
   });
 
-  it('defaults to maxConcurrency of 10', async () => {
+  it("defaults to maxConcurrency of 10", async () => {
     const pool = new ConcurrencyPool();
 
     // Acquire 10 slots — all should resolve immediately

package/src/config/bundled-skills/media-processing/__tests__/cost-tracker.test.ts

@@ -1,21 +1,21 @@
-import { describe, expect, it } from 'bun:test';
+import { describe, expect, it } from "bun:test";
 
-import { CostTracker } from '../services/cost-tracker.js';
+import { CostTracker } from "../services/cost-tracker.js";
 
-describe('CostTracker', () => {
-  it('accumulates costs across multiple segments', () => {
+describe("CostTracker", () => {
+  it("accumulates costs across multiple segments", () => {
     const tracker = new CostTracker();
 
     tracker.record({
-      segmentId: 'seg-001',
-      model: 'gemini-2.5-flash',
+      segmentId: "seg-001",
+      model: "gemini-2.5-flash",
       inputTokens: 1_000_000,
       outputTokens: 0,
     });
 
     tracker.record({
-      segmentId: 'seg-002',
-      model: 'gemini-2.5-flash',
+      segmentId: "seg-002",
+      model: "gemini-2.5-flash",
       inputTokens: 0,
       outputTokens: 1_000_000,
     });
@@ -28,12 +28,12 @@ describe('CostTracker', () => {
     expect(summary.totalEstimatedUSD).toBeCloseTo(0.75, 6);
   });
 
-  it('computes per-entry costs using Gemini 2.5 Flash pricing', () => {
+  it("computes per-entry costs using Gemini 2.5 Flash pricing", () => {
     const tracker = new CostTracker();
 
     const entry = tracker.record({
-      segmentId: 'seg-010',
-      model: 'gemini-2.5-flash',
+      segmentId: "seg-010",
+      model: "gemini-2.5-flash",
       inputTokens: 200_000,
       outputTokens: 50_000,
     });
@@ -42,11 +42,11 @@ describe('CostTracker', () => {
     // Output: 50k * ($0.60 / 1M) = $0.03
     // Total: $0.06
     expect(entry.estimatedUSD).toBeCloseTo(0.06, 6);
-    expect(entry.segmentId).toBe('seg-010');
-    expect(entry.model).toBe('gemini-2.5-flash');
+    expect(entry.segmentId).toBe("seg-010");
+    expect(entry.model).toBe("gemini-2.5-flash");
   });
 
-  it('returns an empty summary when no entries have been recorded', () => {
+  it("returns an empty summary when no entries have been recorded", () => {
     const tracker = new CostTracker();
     const summary = tracker.getSummary();
 
@@ -57,14 +57,29 @@ describe('CostTracker', () => {
     expect(summary.entries).toHaveLength(0);
   });
 
-  it('preserves entry order in summary', () => {
+  it("preserves entry order in summary", () => {
     const tracker = new CostTracker();
 
-    tracker.record({ segmentId: 'a', model: 'm', inputTokens: 100, outputTokens: 200 });
-    tracker.record({ segmentId: 'b', model: 'm', inputTokens: 300, outputTokens: 400 });
-    tracker.record({ segmentId: 'c', model: 'm', inputTokens: 500, outputTokens: 600 });
+    tracker.record({
+      segmentId: "a",
+      model: "m",
+      inputTokens: 100,
+      outputTokens: 200,
+    });
+    tracker.record({
+      segmentId: "b",
+      model: "m",
+      inputTokens: 300,
+      outputTokens: 400,
+    });
+    tracker.record({
+      segmentId: "c",
+      model: "m",
+      inputTokens: 500,
+      outputTokens: 600,
+    });
 
     const ids = tracker.getSummary().entries.map((e) => e.segmentId);
-    expect(ids).toEqual(['a', 'b', 'c']);
+    expect(ids).toEqual(["a", "b", "c"]);
   });
 });