@vellumai/assistant 0.3.5 → 0.3.7

package/src/config/bundled-skills/google-oauth-setup/SKILL.md

@@ -0,0 +1,183 @@
+---
+name: "Google OAuth Setup"
+description: "Set up Google Cloud OAuth credentials for Gmail and Calendar using browser automation"
+user-invocable: true
+includes: ["browser", "public-ingress"]
+metadata: {"vellum": {"emoji": "\ud83d\udd11"}}
+---
+
+You are helping your user set up Google Cloud OAuth credentials so Gmail and Google Calendar integrations can connect. You will automate the entire GCP setup via the browser while the user watches via screencast. The user's only manual action is signing in to their Google account — everything else is fully automated.
+
+## Client Check
+
+If the user is on Telegram (or any non-macOS client without browser automation):
+
+> "Gmail setup requires browser automation, which is available on the macOS app. Please open the Vellum app on your Mac and ask me to connect Gmail there — I'll handle the rest automatically."
+
+Stop here. Do not attempt a manual walkthrough.
+
+## Prerequisites
+
+Before starting, check that `ingress.publicBaseUrl` is configured (`INGRESS_PUBLIC_BASE_URL` env var or workspace config). If it is not set, load and execute the **public-ingress** skill first (`skill_load` with `skill: "public-ingress"`) to set up an ngrok tunnel and persist the public URL. The OAuth redirect URI depends on this value.
+
+## Step 1: Single Upfront Confirmation
+
+Use `ui_show` with `surface_type: "confirmation"` and this message:
+
+> **Set up Google Cloud for Gmail & Calendar**
+>
+> Here's what will happen:
+> 1. **A browser opens** — you sign in to your Google account
+> 2. **I automate everything** — project creation, APIs, OAuth config, credentials
+> 3. **You enter credentials** from a downloaded file (secure prompt — I never see them)
+> 4. **You authorize Vellum** with one click
+>
+> The whole thing takes 2-3 minutes. Ready?
+
+If the user declines, acknowledge and stop. No further confirmations are needed after this point.
+
+## Step 2: Open Google Cloud Console
+
+Use `browser_navigate` to go to `https://console.cloud.google.com/`.
+
+Take a `browser_screenshot` and `browser_snapshot` to check the page state:
+- **If a sign-in page appears:** Tell the user: "Please sign in to your Google account in the browser preview panel (or the Chrome window that just opened)." Then **auto-detect sign-in completion** by polling `browser_snapshot` every 5-10 seconds. Check if the current URL has moved away from `accounts.google.com` to `console.cloud.google.com`. Do NOT ask the user to "let me know when you're done" — detect it automatically. Once sign-in is detected, tell the user: "Signed in! Starting the automated setup now..."
+- **If already signed in** (URL is already `console.cloud.google.com`): Tell the user: "Already signed in — starting setup now..." and continue immediately.
+- **If a CAPTCHA appears:** The browser automation's built-in handoff will handle this. If it persists, tell the user: "There's a CAPTCHA in the browser — please complete it and I'll continue automatically."
+- **If the console dashboard loads:** Continue to Step 3.
+
+## Step 3: Create or Select a Project
+
+Tell the user: "Creating Google Cloud project 'Vellum Assistant'..."
+
+Navigate to `https://console.cloud.google.com/projectcreate`.
+
+Take a `browser_snapshot`. Fill in the project name:
+- Use `browser_type` to set the project name to "Vellum Assistant"
+- Use `browser_click` to submit the "Create" button
+
+Wait a few seconds, take a `browser_screenshot` and `browser_snapshot` to confirm. If the project already exists, navigate to its dashboard. Note the project ID for subsequent steps.
+
+Tell the user: "Project created!"
+
+## Step 4: Enable Gmail and Calendar APIs
+
+Tell the user: "Enabling Gmail and Calendar APIs..."
+
+Navigate to `https://console.cloud.google.com/apis/library/gmail.googleapis.com?project=PROJECT_ID` (substitute actual project ID).
+
+Take a `browser_snapshot`:
+- If already enabled (shows "API enabled" or "Manage" button): skip.
+- If not: click the "Enable" button and wait.
+
+Then navigate to `https://console.cloud.google.com/apis/library/calendar-json.googleapis.com?project=PROJECT_ID`.
+
+Same check — enable if needed.
+
+Take a `browser_screenshot` to show result. Tell the user: "APIs enabled!"
+
+## Step 5: Configure OAuth Consent Screen
+
+Tell the user: "Configuring OAuth consent screen — this is the longest step, but it's fully automated..."
+
+Navigate to `https://console.cloud.google.com/apis/credentials/consent?project=PROJECT_ID`.
+
+Take a `browser_snapshot`:
+- If consent screen is already configured: skip to Step 6.
+- If user type selection appears: select "External" and click "Create".
+
+Fill in the consent screen form:
+1. **App name:** "Vellum Assistant"
+2. **User support email:** Select the user's email from the dropdown
+3. **Developer contact email:** Type the user's email address
+4. Leave other fields as defaults
+
+Navigate through the wizard pages:
+- App information page: Fill fields, click "Save and Continue"
+- Scopes page: Click "Add or Remove Scopes", search for and select:
+  - `https://www.googleapis.com/auth/gmail.readonly`
+  - `https://www.googleapis.com/auth/gmail.modify`
+  - `https://www.googleapis.com/auth/gmail.send`
+  - `https://www.googleapis.com/auth/calendar.readonly`
+  - `https://www.googleapis.com/auth/calendar.events`
+  - `https://www.googleapis.com/auth/userinfo.email`
+  - Click "Update" then "Save and Continue"
+- Test users page: Add the user's email as a test user, click "Save and Continue"
+- Summary page: Click "Back to Dashboard"
+
+Tell the user: "Consent screen configured!"
+
+## Step 6: Create OAuth Credentials
+
+Tell the user: "Creating OAuth credentials..."
+
+Navigate to `https://console.cloud.google.com/apis/credentials?project=PROJECT_ID`.
+
+Click "+ Create Credentials" then select "OAuth client ID".
+
+Take a `browser_snapshot` and fill in:
+1. **Application type:** Select "Web application"
+2. **Name:** "Vellum Assistant"
+3. **Authorized redirect URIs:** Click "Add URI" and enter `${ingress.publicBaseUrl}/webhooks/oauth/callback`
+
+Click "Create".
+
+## Step 7: Download Credentials JSON
+
+Tell the user: "Almost done — downloading credentials..."
+
+After the credentials dialog appears, click the "Download JSON" button (it may say "DOWNLOAD JSON" or show a download icon).
+
+Use `browser_wait_for_download` to wait for the file to download.
+
+Tell the user: "Credentials downloaded!"
+
+## Step 8: Secure Credential Entry
+
+Tell the user: "I've downloaded the credentials file. Please open it and enter the values below. I won't see what you type — these go directly to secure storage."
+
+```
+credential_store prompt:
+  service: "integration:gmail"
+  field: "client_id"
+  label: "Google OAuth Client ID"
+  description: "Open the downloaded JSON file and copy the client_id value"
+  placeholder: "123456789.apps.googleusercontent.com"
+```
+
+```
+credential_store prompt:
+  service: "integration:gmail"
+  field: "client_secret"
+  label: "Google OAuth Client Secret"
+  description: "Copy the client_secret value from the same JSON file"
+  placeholder: "GOCSPX-..."
+```
+
+## Step 9: OAuth2 Authorization
+
+Tell the user: "Opening Google sign-in so you can authorize Vellum. Just click 'Allow' on the consent page."
+
+Use `credential_store` with:
+
+```
+action: "oauth2_connect"
+service: "integration:gmail"
+```
+
+This auto-reads client_id/client_secret from the secure store and auto-fills auth_url, token_url, scopes, and extra_params from well-known config.
+
+**If the user sees a "This app isn't verified" warning:** Tell them this is normal for apps in testing mode. Click "Advanced" then "Go to Vellum Assistant (unsafe)" to proceed.
+
+## Step 10: Done!
+
+"**Gmail and Calendar are connected!** You can now read, search, and send emails, plus view and manage your calendar. Try asking me to check your inbox or show your upcoming events!"
+
+## Error Handling
+
+- **Page load failures:** Retry navigation once. If it still fails, tell the user and ask them to check their internet connection.
+- **Permission errors in GCP:** The user may need billing enabled or organization-level permissions. Explain clearly and ask them to resolve it.
+- **Consent screen already configured:** Don't overwrite — skip to credential creation.
+- **Element not found:** Take a fresh `browser_snapshot` to re-assess. GCP UI may have changed. Tell the user what you're looking for if stuck.
+- **OAuth flow timeout or failure:** Offer to retry. The credentials are already stored, so reconnecting only requires re-running the authorization flow.
+- **Any unexpected state:** Take a `browser_screenshot` and `browser_snapshot`, describe what you see, and ask the user for guidance.

package/src/config/bundled-skills/influencer/SKILL.md

@@ -0,0 +1,144 @@
+---
+name: "Influencer Research"
+description: "Research influencers on Instagram, TikTok, and X/Twitter using the Chrome extension relay"
+user-invocable: true
+metadata: {"vellum": {"emoji": "🔍"}}
+---
+
+You can research and discover influencers across Instagram, TikTok, and X/Twitter using the `vellum influencer` CLI.
+
+## CLI Setup
+
+**IMPORTANT: Always use `host_bash` (not `bash`) for all `vellum influencer` commands.** The influencer CLI needs host access for the Chrome extension relay and the `vellum` binary, neither of which are available inside the sandbox.
+
+`vellum influencer` is a built-in subcommand of the Vellum assistant CLI. If `vellum` is not found, prepend `PATH="$HOME/.local/bin:$PATH"` to the command.
+
+## Prerequisites
+
+- The Chrome extension relay must be connected (user should have the Vellum extension loaded in Chrome)
+- The user must be **logged in** on each platform they want to search (Instagram, TikTok, X) in their Chrome browser
+- The extension MUST have the `debugger` permission (required to bypass CSP on Instagram and other Meta sites)
+- If the relay is not connected, tell the user: "Please open Chrome, click the Vellum extension icon, and click Connect — then I'll retry."
+
+## Platform-Specific Architecture
+
+### Instagram
+Instagram's search at `/explore/search/keyword/?q=...` shows a **grid of posts**, NOT profiles. The discovery flow is:
+1. Search by keyword → extract post links (`/p/` and `/reel/`)
+2. Visit each post → find the author username from page links
+3. Deduplicate usernames
+4. Visit each unique profile → scrape stats from `meta[name="description"]` (most reliable source, format: "49K Followers, 463 Following, 551 Posts - Display Name (@user)")
+5. Filter and rank by criteria
+
+**CSP Note:** Instagram blocks `eval()`, `new Function()`, inline scripts, and blob URLs via strict CSP. The extension uses `chrome.debugger` API (CDP Runtime.evaluate) as a fallback, which bypasses all CSP restrictions.
+
+### TikTok
+TikTok has a dedicated user search at `/search/user?q=...`. Each result card produces a predictable text pattern in `innerText`:
+```
+DisplayName
+username
+77.9K
+Followers
+·
+1.5M
+Likes
+Follow
+```
+We parse this pattern directly (DOM class selectors are obfuscated and unreliable on TikTok). After extracting usernames and follower counts, we visit each profile for bios.
+
+### X/Twitter
+X has a people search at `/search?q=...&f=user` with `[data-testid="UserCell"]` components containing username, display name, bio, and verified status.
+
+## Typical Flow
+
+When the user asks to find or research influencers:
+
+1. **Understand the criteria.** Ask about:
+   - **Niche/topic** — what kind of influencers? (fitness, beauty, tech, food, etc.)
+   - **Platforms** — Instagram, TikTok, X/Twitter, or all three?
+   - **Follower range** — micro (1K-10K), mid-tier (10K-100K), macro (100K-1M), mega (1M+)?
+   - **Verified only?** — do they need the blue checkmark?
+   - Don't over-ask. If the user says "find me fitness influencers on Instagram", that's enough to start.
+
+2. **Search** — run `vellum influencer search "<query>" --platforms <platforms> [options] --json`
+
+3. **Present results** — show a clean summary of each influencer found:
+   - Username and display name
+   - Platform
+   - Follower count
+   - Bio snippet
+   - Verified status
+   - Content themes detected
+   - Profile URL
+
+4. **Deep dive** (if needed) — run `vellum influencer profile <username> --platform <platform> --json` to get detailed data on a specific influencer.
+
+5. **Compare** (if needed) — run `vellum influencer compare instagram:user1 twitter:user2 tiktok:user3 --json` to compare influencers side by side.
+
+## Follower Range Shortcuts
+
+When the user describes influencer tiers, map to these ranges:
+- **Nano**: `--min-followers 1000 --max-followers 10000`
+- **Micro**: `--min-followers 10000 --max-followers 100000`
+- **Mid-tier**: `--min-followers 100000 --max-followers 500000`
+- **Macro**: `--min-followers 500000 --max-followers 1000000`
+- **Mega**: `--min-followers 1000000`
+
+Human-friendly numbers are supported: `10k`, `100k`, `1m`, etc.
+
+## Command Reference
+
+```
+vellum influencer search "<query>" [options] --json
+  --platforms <list>       Comma-separated: instagram,tiktok,twitter (default: all three)
+  --min-followers <n>      Minimum follower count (e.g. 10k, 100000)
+  --max-followers <n>      Maximum follower count (e.g. 1m, 500k)
+  --limit <n>              Max results per platform (default: 10)
+  --verified               Only return verified accounts
+
+vellum influencer profile <username> --platform <platform> --json
+  --platform <platform>    instagram, tiktok, or twitter (default: instagram)
+
+vellum influencer compare <platform:username ...> --json
+  Arguments are space-separated platform:username pairs
+  e.g. instagram:nike twitter:nike tiktok:nike
+```
+
+## Important Behavior
+
+- **Use `--json` flag** on all commands for reliable parsing.
+- **Always use `host_bash`** for these commands, never `bash`.
+- **Be patient with results.** The tool navigates actual browser tabs, so each platform search takes 10-30 seconds. Warn the user it may take a moment.
+- **Rate limiting.** Don't hammer the platforms. The tool has built-in delays, but avoid running many searches in rapid succession.
+- **Present results nicely.** Use tables or formatted lists. Group by platform. Highlight standout profiles.
+- **Offer next steps.** After showing results, ask if they want to:
+  - Get more details on specific profiles
+  - Compare top picks side by side
+  - Search with different criteria
+  - Export the results
+- **Handle errors gracefully.** If a platform fails (e.g. not logged in), show results from the platforms that worked and mention which one failed.
+- **Do NOT use the browser skill.** All influencer research goes through the CLI, not browser automation.
+
+## Example Interactions
+
+**User**: "Find me fitness influencers on Instagram and TikTok"
+
+1. `vellum influencer search "fitness coach workout" --platforms instagram,tiktok --limit 10 --json`
+2. Present results grouped by platform with follower counts and bios
+3. "I found 8 fitness influencers on Instagram and 6 on TikTok. Want me to dig deeper into any of these profiles?"
+
+**User**: "I need micro-influencers in the beauty niche, verified only"
+
+1. `vellum influencer search "beauty makeup skincare" --platforms instagram,tiktok,twitter --min-followers 10k --max-followers 100k --verified --limit 10 --json`
+2. Present filtered results
+3. Offer to compare top picks
+
+**User**: "Compare @username1 on Instagram with @username2 on TikTok"
+
+1. `vellum influencer compare instagram:username1 tiktok:username2 --json`
+2. Present side-by-side comparison with followers, engagement, bio, themes
+
+**User**: "Tell me more about @specificuser on Instagram"
+
+1. `vellum influencer profile specificuser --platform instagram --json`
+2. Show full profile details including bio, follower/following counts, verified status, content themes

package/src/config/bundled-skills/macos-automation/icon.svg

@@ -0,0 +1,12 @@
+<svg viewBox="0 0 16 16" xmlns="http://www.w3.org/2000/svg">
+<rect x="2" y="2" width="12" height="11" fill="#e8e8e8" stroke="#333" stroke-width="1"/>
+<rect x="3" y="3" width="10" height="9" fill="#f5f5f5"/>
+<circle cx="8" cy="7" r="2" fill="#0071e3"/>
+<rect x="5" y="5" width="1" height="1" fill="#333"/>
+<rect x="10" y="5" width="1" height="1" fill="#333"/>
+<rect x="5" y="9" width="1" height="1" fill="#333"/>
+<rect x="10" y="9" width="1" height="1" fill="#333"/>
+<path d="M 8 4 L 8 6 M 6 7 L 8 7 M 8 8 L 8 10 M 8 7 L 10 7" stroke="#0071e3" stroke-width="1" fill="none"/>
+<rect x="3" y="13" width="10" height="1" fill="#333"/>
+<rect x="4" y="14" width="8" height="1" fill="#0071e3"/>
+</svg>

package/src/config/bundled-skills/media-processing/SKILL.md

@@ -1,23 +1,22 @@
 ---
 name: "Media Processing"
-description: "Ingest and process media files (video, audio, image) through multi-stage pipelines including keyframe extraction, vision analysis, and timeline generation"
+description: "Ingest and process media files (video, audio, image) through a 3-phase pipeline: preprocess, map (Gemini), and reduce (Claude)"
 metadata: {"vellum": {"emoji": "🎬"}}
 ---
 
-Ingest and track processing of media files (video, audio, images) through configurable multi-stage pipelines.
+Ingest and track processing of media files (video, audio, images) through a configurable 3-phase pipeline.
 
 ## End-to-End Workflow
 
-The processing pipeline follows a sequential flow. Each stage depends on the output of the previous one:
+The processing pipeline follows a sequential 3-phase flow:
 
 1. **Ingest** (`ingest_media`) — Register a media file, detect MIME type, extract duration, deduplicate by content hash.
-2. **Extract Keyframes** (`extract_keyframes`) — Pull frames from video at regular intervals (default: every 3 seconds) using ffmpeg.
-3. **Analyze Keyframes** (`analyze_keyframes`) — Send each keyframe to Claude VLM for structured scene analysis (subjects, actions, context).
-4. **Generate Timeline** — Aggregate vision outputs into coherent timeline segments (called via `services/timeline-service.ts`).
-5. **Detect Events** (`detect_events`) — Apply configurable detection rules against timeline segments to find events of interest.
-6. **Query & Clip** — Use `query_media_events` to search events with natural language, and `generate_clip` to extract video clips around specific moments.
+2. **Preprocess** (`extract_keyframes`) — Detect dead time, segment the video into windows, extract downscaled keyframes, build a subject registry, and write a pipeline manifest.
+3. **Map** (`analyze_keyframes`) — Send each segment's frames to Gemini 2.5 Flash with assistant-provided extraction instructions and a JSON Schema for guaranteed structured output. Supports concurrency pooling, cost tracking, resumability, and automatic retries.
+4. **Reduce / Query** (`query_media`) — Send all map output to Claude for intelligent analysis and Q&A. Supports arbitrary natural language queries about video content.
+5. **Clip** (`generate_clip`) — Extract video clips around specific moments.
 
-The processing pipeline service (`services/processing-pipeline.ts`) can orchestrate stages 2-5 automatically with retries, resumability, and cancellation support.
+The processing pipeline service (`services/processing-pipeline.ts`) orchestrates phases 2-4 automatically with retries, resumability, and cancellation support.
 
 ## Tools
 
@@ -31,86 +30,85 @@ Query the processing status of a media asset. Returns the asset metadata along w
 
 ### extract_keyframes
 
-Extract keyframes from a video asset at regular intervals using ffmpeg. Frames are saved as JPEG images and registered in the database for subsequent vision analysis.
+Preprocess a video asset: detect dead time via mpdecimate, segment the video into windows, extract downscaled keyframes at regular intervals, build a subject registry, and write a pipeline manifest.
 
-### analyze_keyframes
+Parameters:
+- `asset_id` (required) — ID of the media asset.
+- `interval_seconds` — Interval between keyframes (default: 3s).
+- `segment_duration` — Duration of each segment window (default: 20s).
+- `dead_time_threshold` — Sensitivity for dead-time detection (default: 0.02).
+- `section_config` — Path to a JSON file with manual section boundaries.
+- `skip_dead_time` — Whether to detect and skip dead time (default: true).
+- `short_edge` — Short edge resolution for downscaled frames in pixels (default: 480).
 
-Analyze extracted keyframes using Claude VLM (vision language model). Produces structured JSON output with scene descriptions, subjects, actions, and context. Supports resumability by skipping already-analyzed frames.
+### analyze_keyframes
 
-### detect_events
+Map video segments through Gemini's structured output API. Reads frames from the preprocess manifest, sends each segment to Gemini with assistant-provided extraction instructions and a JSON Schema for guaranteed structured output. Supports concurrency pooling, cost tracking, resumability (skips segments with existing results), and automatic retries with exponential backoff.
 
-Detect events from timeline segments using configurable detection rules. Built-in rule types:
-- **segment_transition** — Fires when a specified field changes between adjacent segments.
-- **short_segment** — Fires when a segment's duration is below a threshold.
-- **attribute_match** — Fires when segment attribute values match a regex pattern.
+Parameters:
+- `asset_id` (required) — ID of the media asset.
+- `system_prompt` (required) — Extraction instructions for Gemini.
+- `output_schema` (required) — JSON Schema for structured output.
+- `context` — Additional context to include in the prompt.
+- `model` — Gemini model to use (default: `gemini-2.5-flash`).
+- `concurrency` — Maximum concurrent API requests (default: 10).
+- `max_retries` — Retry attempts per segment on failure (default: 3).
 
-If no rules are provided, sensible defaults are applied based on the event type.
+### query_media
 
-### query_media_events
+Query video analysis data using natural language. Sends map output (from analyze_keyframes) to Claude for intelligent analysis and Q&A. Supports arbitrary questions about video content.
 
-Query detected events using natural language. Parses the query into structured filters (event type, count, confidence threshold, time range) and returns matching events ranked by confidence.
+Parameters:
+- `asset_id` (required) — ID of the media asset.
+- `query` (required) — Natural language query about the video data.
+- `system_prompt` — Optional system prompt for Claude.
+- `model` — LLM model to use (default: `claude-sonnet-4-6`).
 
 ### generate_clip
 
 Extract a video clip from a media asset using ffmpeg. Applies configurable pre/post-roll padding (clamped to file boundaries), outputs the clip as a temporary file.
 
-### select_tracking_profile
-
-Configure which event capabilities are enabled for a media asset. Capabilities are organized into tiers:
-- **Ready**: Production-quality detection, included by default.
-- **Beta**: Functional but may have accuracy gaps. Results include a confidence disclaimer.
-- **Experimental**: Early-stage detection, expect noise. Results include a confidence disclaimer.
-
-Call without capabilities to see available options; call with a capabilities array to set the profile.
-
-### submit_feedback
-
-Submit feedback on a detected event. Supports four types:
-- **correct** — Confirms the event is accurate.
-- **incorrect** — Marks a false positive.
-- **boundary_edit** — Adjusts start/end times.
-- **missed** — Reports an event the system failed to detect.
-
-### recalibrate
-
-Re-rank existing events based on accumulated feedback. Adjusts confidence scores using correction patterns (false positive rates, missed events, boundary adjustments).
-
 ### media_diagnostics
 
 Get a diagnostic report for a media asset. Returns:
-- **Processing stats**: total keyframes, vision outputs, timeline segments, events detected.
-- **Per-stage status and timing**: which stages have run, how long each took, current progress.
+- **Processing stats**: total keyframes extracted.
+- **Per-stage status and timing**: which stages (preprocess, map, reduce) have run, how long each took, current progress.
 - **Failure reasons**: last error from any failed stage.
-- **Cost estimation**: based on keyframe count and estimated API cost per frame.
-- **Feedback summary**: precision/recall estimates per event type.
+- **Cost estimation**: based on segment count and Gemini 2.5 Flash pricing, plus a note about Claude reduce costs.
 
 ## Services
 
 ### Processing Pipeline (services/processing-pipeline.ts)
 
 Orchestrates the full processing pipeline with reliability features:
-- **Sequential execution**: keyframe_extraction, vision_analysis, timeline_generation, event_detection.
+- **Sequential execution**: preprocess, map, reduce.
 - **Retries**: Each stage is retried with exponential backoff and jitter (configurable max retries and base delay).
 - **Resumability**: Checks processing_stages to find the last completed stage and resumes from there. Safe to restart after crashes.
 - **Cancellation**: Cooperative cancellation via asset status. Set asset status to `cancelled` and the pipeline stops between stages.
 - **Idempotency**: Re-ingesting the same file hash is a no-op. Re-running a fully completed pipeline is also a no-op.
-- **Graceful degradation**: If a stage fails mid-batch (e.g., vision API errors), partial results are saved. The stage is marked as failed with the error details, and the pipeline stops without losing work.
+- **Graceful degradation**: If a stage fails mid-batch (e.g., Gemini API errors), partial results are saved. The stage is marked as failed with the error details, and the pipeline stops without losing work.
+
+### Preprocess (services/preprocess.ts)
 
-### Timeline Generation (services/timeline-service.ts)
+Handles dead-time detection, video segmentation, keyframe extraction, and subject registry building. Writes a pipeline manifest consumed by the Map phase.
 
-Aggregates vision analysis outputs into coherent timeline segments. Groups adjacent keyframes that share similar scene characteristics into time ranges with merged attributes.
+### Gemini Map (services/gemini-map.ts)
 
-### Event Detection (services/event-detection-service.ts)
+Sends video segments to Gemini 2.5 Flash with structured output schemas. Handles concurrency pooling, cost tracking, resumability, and retries.
 
-Evaluates configurable detection rules against timeline segments. Produces scored event candidates with weighted confidence.
+### Reduce (services/reduce.ts)
 
-### Feedback Aggregation (services/feedback-aggregation.ts)
+Sends Map output to Claude as text for analysis. Two modes:
+- **One-shot merge**: assembles all Map results and sends to Claude with a system prompt.
+- **Interactive Q&A**: loads existing map output + user query, sends to Claude.
 
-Computes precision/recall estimates per event type from user feedback. Provides structured JSON export for offline analysis.
+### Concurrency Pool (services/concurrency-pool.ts)
 
-### Capability Registry (services/capability-registry.ts)
+Limits concurrent API calls during the Map phase to avoid rate limiting.
 
-Maintains an extensible, domain-agnostic catalog of available tracking capabilities with tier classification. Other domains can register their own capabilities by calling `registerCapability()`.
+### Cost Tracker (services/cost-tracker.ts)
+
+Tracks estimated API costs during pipeline execution.
 
 ## Operator Runbook
 
@@ -131,62 +129,41 @@ Use `media_diagnostics` to get a full diagnostic report:
 2. Read the `lastError` field for that stage to understand what went wrong.
 3. Check `durationMs` to see if a stage timed out or ran unusually long.
 4. Common failure causes:
-   - **keyframe_extraction**: ffmpeg not installed, corrupt video file, disk full.
-   - **vision_analysis**: ANTHROPIC_API_KEY not set, API rate limits, network errors.
-   - **timeline_generation**: No keyframes or vision outputs exist (earlier stage skipped or failed).
-   - **event_detection**: No timeline segments exist.
+   - **preprocess**: ffmpeg not installed, corrupt video file, disk full.
+   - **map**: Gemini API key not configured, API rate limits, network errors.
+   - **reduce**: No LLM provider configured, no map output exists.
 
 After fixing the root cause, re-run the failed stage. The pipeline is resumable — it picks up from where it left off.
 
-### Configuring Tracking Profiles
-
-1. Call `select_tracking_profile` with just the `asset_id` to see available capabilities and their tiers.
-2. Call again with a `capabilities` array to enable the desired event types.
-3. Only enabled capabilities are returned by `query_media_events`.
-4. The capability registry is extensible — new domains can register capabilities via `registerCapability()` in `services/capability-registry.ts`.
-
-### Feedback and Recalibration
-
-1. Review detected events using `query_media_events`.
-2. For each event, submit feedback via `submit_feedback`:
-   - Mark correct detections as `correct` to build precision data.
-   - Mark false positives as `incorrect`.
-   - Adjust boundaries with `boundary_edit`.
-   - Report missed events with `missed` (creates a new event record).
-3. Run `recalibrate` to re-rank events based on accumulated feedback.
-4. Use `media_diagnostics` to check precision/recall estimates after feedback.
-
 ### Cost Expectations
 
-Vision analysis is the primary cost driver. Cost scales linearly with video duration and keyframe interval:
+The Map phase (Gemini 2.5 Flash) is the primary cost driver. Cost scales with video duration, keyframe interval, and segment size:
 
-| Video Duration | Interval | Keyframes | Estimated Cost |
-|----------------|----------|-----------|----------------|
-| 30 min         | 3s       | ~600      | ~$1.80         |
-| 60 min         | 3s       | ~1,200    | ~$3.60         |
-| 90 min         | 3s       | ~1,800    | ~$5.40         |
-| 90 min         | 5s       | ~1,080    | ~$3.24         |
+| Video Duration | Interval | Keyframes | Segments (~10 frames each) | Estimated Map Cost |
+|----------------|----------|-----------|----------------------------|--------------------|
+| 30 min         | 3s       | ~600      | ~60                        | ~$0.06             |
+| 60 min         | 3s       | ~1,200    | ~120                       | ~$0.12             |
+| 90 min         | 3s       | ~1,800    | ~180                       | ~$0.18             |
+| 90 min         | 5s       | ~1,080    | ~108                       | ~$0.11             |
 
-Increasing the keyframe interval reduces cost proportionally but may miss short-duration events. The `media_diagnostics` tool provides per-asset cost estimates.
+The Reduce phase (Claude) adds a small additional cost per query. The `media_diagnostics` tool provides per-asset cost estimates.
 
 ### Known Limitations
 
 - **ffmpeg required**: Keyframe extraction and clip generation require ffmpeg to be installed on the host.
 - **Single-file ingestion**: Each `ingest_media` call processes one file. Batch ingestion is not yet supported.
-- **Vision model latency**: Analyzing keyframes is the slowest stage. A 90-minute video at 3-second intervals requires ~1,800 API calls.
-- **Scene similarity heuristic**: Timeline segmentation uses Jaccard similarity on subjects — it works well for distinct scenes but may over-merge visually similar but semantically different moments.
-- **Detection rules are heuristic**: Event detection uses rule-based scoring, not ML. Accuracy depends on how well the rules match the target event patterns. Use feedback and recalibration to improve over time.
+- **Gemini rate limits**: The Map phase uses concurrency pooling (default 10) to stay within API limits. Reduce concurrency if you hit 429 errors.
 - **No real-time processing**: The pipeline processes pre-recorded media files. Live/streaming video is not supported.
 
 ### Troubleshooting
 
 | Symptom | Likely Cause | Fix |
 |---------|-------------|-----|
-| "No keyframes found" | extract_keyframes not run or failed | Check keyframe_extraction stage status; re-run if needed |
-| "ANTHROPIC_API_KEY not set" | Missing env var | Set ANTHROPIC_API_KEY in the environment |
-| Vision analysis very slow | Large video, small interval | Increase interval_seconds or use smaller batch_size |
-| Low event confidence | Detection rules too broad | Tune rules: increase weights on high-signal rules, use tighter regex patterns |
-| Many false positives | Rules overfitting on noise | Submit `incorrect` feedback, then run `recalibrate` |
+| "No keyframes found" | extract_keyframes not run or failed | Check preprocess stage status; re-run if needed |
+| "No map output found" | analyze_keyframes not run | Run analyze_keyframes with appropriate system_prompt and output_schema |
+| "No LLM provider available" | API key not configured | Add one in Settings |
+| Map phase slow | Large video, small interval | Increase interval_seconds or reduce concurrency |
+| Gemini returns errors | Rate limits or schema issues | Check max_retries setting; simplify output_schema if needed |
 | Pipeline stuck at "processing" | Stage crashed without updating status | Use `media_diagnostics` to find the stuck stage; re-run manually |
 
 ## Usage Notes
@@ -195,5 +172,5 @@ Increasing the keyframe interval reduces cost proportionally but may miss short-
 - Supported media types: video (mp4, mov, avi, mkv, webm, etc.), audio (mp3, wav, m4a, etc.), and images (png, jpg, gif, webp, etc.).
 - For video and audio files, duration is automatically extracted via ffprobe (requires ffmpeg to be installed).
 - Duplicate files are detected by content hash and return the existing asset record.
-- The `analyze_keyframes` tool is marked as medium risk because it makes external API calls to Claude VLM, which incur costs.
-- All schema tables, services, and tool interfaces are media-generic. Domain-specific interpretation belongs in VLM prompt templates.
+- The `analyze_keyframes` tool is marked as medium risk because it makes external API calls to Gemini, which incur costs.
+- All schema tables, services, and tool interfaces are media-generic. Domain-specific interpretation belongs in the system_prompt and output_schema parameters.