@sylphx/video-reader-mcp 0.1.0 → 0.1.2

package/README.md

@@ -1,86 +1,274 @@
-# Video Reader MCP
+<div align="center">
 
-> Evidence-first video reading for AI agents — ffprobe, subtitles, scenes, transcripts, and timelines without frame-by-frame LLM vision.
+# Cue
 
-**Status:** v0.1.0 shipped — `read_video` MCP tool available.
+### Timeline proof for agents 
 
-Orchestrated by [smart-reader-mcp](https://github.com/SylphxAI/smart-reader-mcp) — portfolio ADR lives there, not in pdf-reader-mcp.
+**Cue** (transitional package `@sylphx/video-reader-mcp`) — local video timeline evidence, not frame-by-frame VLM.
 
-| Repository | Role |
+<p align="center">
+  <img src="https://mark.sylphx.com/api/v1/banner?type=mesh&theme=tokyonight&text=video+reader+mcp&desc=Evidence-first+video+reading+for+AI+agents+%E2%80%94+ffprobe%2C+subtitles%2C+scenes%2C+transcr&height=200&animation=rise&credit=0" alt="video-reader-mcp — Sylphx Mark banner" width="100%" />
+</p>
+
+### Your agent watched the video. **Did it read the timeline?**
+
+Evidence-first video reading for AI agents. One call turns any local video into a
+**timeline document** — ffprobe streams, chapters, embedded subtitles, scene
+boundaries, and warnings you can cite without frame-by-frame vision LLM calls.
+
+[![npm version](https://img.shields.io/npm/v/@sylphx/video-reader-mcp?style=flat-square)](https://www.npmjs.com/package/@sylphx/video-reader-mcp)
+[![License](https://img.shields.io/badge/License-MIT-blue?style=flat-square)](https://opensource.org/licenses/MIT)
+[![CI/CD](https://img.shields.io/github/actions/workflow/status/SylphxAI/video-reader-mcp/ci.yml?style=flat-square&label=CI/CD)](https://github.com/SylphxAI/video-reader-mcp/actions/workflows/ci.yml)
+[![TypeScript](https://img.shields.io/badge/TypeScript-7.0-blue.svg?style=flat-square)](https://www.typescriptlang.org/)
+
+**Local-first** · **One smart `read_video` call** · **Timeline evidence + provenance** · **20 tests**
+
+SOTA family roadmap: [docs/roadmap/sota-family-roadmap.md](docs/roadmap/sota-family-roadmap.md).
+
+[⭐ Star this repo](https://github.com/SylphxAI/video-reader-mcp) if agents should read video timelines with proof, not sampled frame captions.
+· [Quick start](#quick-start) · [See it work](#see-it-work) · [Why not frame-by-frame vision?](#why-not-frame-by-frame-vision)
+
+This repository is product SSOT. Sibling agent tools live in separate repos.
+Each instrument is an independent repository (marketplace + stars).
+
+</div>
+
+---
+
+
+
+## Read video structure (not N-second frame spam)
+
+Cue is **local-first timeline architecture**: streams, dialogue, scene cuts, **structural keyframes**, and **agent_index** for text-only agents.
+
+Spec: [docs/specs/agent-video-read-contract.md](docs/specs/agent-video-read-contract.md)
+
+## Product docs
+
+| Doc | Purpose |
 | --- | --- |
-| [pdf-reader-mcp](https://github.com/SylphxAI/pdf-reader-mcp) | PDF (production) |
-| [image-reader-mcp](https://github.com/SylphxAI/image-reader-mcp) | Image |
-| **video-reader-mcp** (this repo) | Video |
-| [smart-reader-mcp](https://github.com/SylphxAI/smart-reader-mcp) | Unified read + delegate |
+| [docs/POSITIONING.md](docs/POSITIONING.md) | Strategic positioning |
+| [docs/COMPETITIVE.md](docs/COMPETITIVE.md) | Peer anchors and wedge |
+| [docs/EVIDENCE_CONTRACT.md](docs/EVIDENCE_CONTRACT.md) | Evidence = result contract |
+| [docs/TOOL_SURFACE.md](docs/TOOL_SURFACE.md) | Few clear tools policy |
+| [docs/PRODUCT_INDEPENDENCE.md](docs/PRODUCT_INDEPENDENCE.md) | This repo is SSOT |
+| [docs/IPPB.md](docs/IPPB.md) | Independent public product bar |
+| [docs/PUBLISH.md](docs/PUBLISH.md) | npm/git publish status |
+
+## The problem
 
-## Read vs interpret
+Videos are not a pile of frames. They are codecs, chapters, embedded subtitles,
+scene cuts, variable frame rates, and timelines measured in milliseconds.
 
-**Read** (this repo): extract facts, metadata, transcripts, regions, and timelines with provenance — **no generative LLM required**.
+Most agent stacks sample frames and ask a **vision LLM** what it sees. Subtitles
+get skipped. Scene boundaries vanish. Duration and stream metadata never reach
+context. Citations become "around minute two, I think." Then the agent
+hallucinates — confidently.
 
-**Interpret** (out of scope): summarize, classify, or answer open questions — belongs in the agent or an optional remote provider adapter.
+**Video Reader MCP is built for the moment your agent needs a citeable timeline,
+not a slideshow summary.**
 
-## MCP surface
+## Why not frame-by-frame vision?
 
-Primary tool: `read_video`
+| Typical vision path | Video Reader MCP |
+| --- | --- |
+| Sample N frames into a vision model | ffprobe format + stream metadata in one call |
+| Paraphrased "what happens" | Embedded subtitle cues with `start_ms`, `end_ms`, and provenance |
+| Scene changes guessed from captions | Optional ffmpeg scene filter with timestamp evidence |
+| Missing audio / VFR silently ignored | Warnings for missing ffmpeg/ffprobe, VFR, missing audio, skipped ASR |
+| Cloud API by default | **Local-first** — ffprobe + ffmpeg on your machine |
+| Ship and pray | **20** tests on parsers, fixture corpus, doctor, release gate, and integration |
 
-Returns a **timeline document** per source:
+## See it work
 
-- ffprobe format + stream metadata
-- chapter markers
-- embedded subtitle cues (`time_ms`, text, provenance)
-- optional scene boundaries (ffmpeg `scene` filter)
-- warnings (missing ffmpeg/ffprobe, VFR, missing audio, skipped ASR)
-- optional local ASR hook (skipped in v0.1 unless adapter is wired)
+### Install (30 seconds)
 
-No per-frame vision LLM calls. No cloud APIs by default.
+```bash
+npm install -g @sylphx/video-reader-mcp
+cue doctor
+claude mcp add cue -- npx @sylphx/video-reader-mcp
+```
+
+**Install once. Call once.**
+
+```bash
+claude mcp add video-reader -- npx @sylphx/video-reader-mcp
+```
+
+```json
+{
+  "sources": [{ "path": "/absolute/path/to/demo.mp4" }],
+  "include_subtitles": true,
+  "include_scenes": true
+}
+```
+
+`read_video` builds a timeline document per source — no per-frame vision LLM
+calls:
+
+```json
+{
+  "source": "/absolute/path/to/demo.mp4",
+  "success": true,
+  "data": {
+    "provenance": {
+      "source": "/absolute/path/to/demo.mp4",
+      "tool": "read_video",
+      "version": "0.1.0",
+      "extracted_at": "2026-07-09T12:00:00.000Z"
+    },
+    "format": {
+      "format_name": "mov,mp4,m4a,3gp,3g2,mj2",
+      "duration_ms": 125500
+    },
+    "streams": [
+      { "index": 0, "codec_type": "video", "width": 1920, "height": 1080 },
+      { "index": 1, "codec_type": "audio", "channels": 2, "sample_rate": 48000 }
+    ],
+    "chapters": [
+      { "id": 0, "start_ms": 0, "end_ms": 60250, "title": "Intro" }
+    ],
+    "subtitles": [
+      {
+        "index": 0,
+        "start_ms": 1200,
+        "end_ms": 3400,
+        "text": "Welcome to the demo.",
+        "provenance": { "method": "ffmpeg_extract", "format": "srt" }
+      }
+    ],
+    "scenes": [
+      {
+        "index": 0,
+        "time_ms": 45200,
+        "provenance": { "method": "ffmpeg_scene_filter", "threshold": 0.4 }
+      }
+    ],
+    "warnings": []
+  }
+}
+```
+
+Abbreviated shape — optional local ASR transcript hooks skip gracefully when no
+adapter is wired.
 
 ## Prerequisites
 
-- Node.js ≥ 22.13
+- Node.js `>=22.13`
 - **ffprobe** (required) and **ffmpeg** (recommended for subtitles + scenes) on `PATH`
 
-## Quick start
+## MCP Tool Surface
 
-```bash
-npx @sylphx/video-reader-mcp
-```
+| Tool | Use it when the agent needs to... |
+| --- | --- |
+| `read_video` | Read one or more local videos and return ffprobe metadata, chapters, subtitles, scenes, and timeline warnings. |
+
+Supported formats: MP4, M4V, MKV, MOV, WebM, and other formats ffprobe can inspect.
 
-From source:
+## Quick Start
+
+### Claude Code
 
 ```bash
-bun install
-bun run build
-bun run start
+claude mcp add video-reader -- npx @sylphx/video-reader-mcp
 ```
 
-### Example `read_video` input
+### Claude Desktop
+
+Add this to `claude_desktop_config.json`:
 
 ```json
 {
-  "sources": [{ "path": "./sample.mp4" }],
-  "include_subtitles": true,
-  "include_scenes": true,
-  "scene_threshold": 0.4
+  "mcpServers": {
+    "video-reader": {
+      "command": "npx",
+      "args": ["@sylphx/video-reader-mcp"]
+    }
+  }
 }
 ```
 
+### Any MCP Client
+
+```bash
+npx @sylphx/video-reader-mcp
+```
+
 ### HTTP transport (optional)
 
 ```bash
-MCP_TRANSPORT=http MCP_HTTP_PORT=8080 node dist/index.js
+MCP_TRANSPORT=http MCP_HTTP_PORT=8080 npx @sylphx/video-reader-mcp
+```
+
+## Security model
+
+- **Local-first** — `read_video` inspects local files; remote URLs are not fetched by default.
+- **ffprobe/ffmpeg boundary** — probe and frame tools shell out to configured binaries on PATH; missing tools return explicit errors.
+- **Fixture corpus** — CI validates parser and safety fixtures; corrupted inputs fail closed with structured diagnostics.
+- **Evidence envelope** — timestamps, frame indices, and extraction routes are preserved so agents can verify claims.
+
+## Release proof
+
+Claims are backed by CI `benchmark:release-gate`, fixture corpus checks, and the shipped-path matrix (Rust-default primary tools).
+
+```bash
+bun run benchmark:release-gate
 ```
 
+Artifact: `benchmark-artifacts/video_reader_release_gate.json` — must report `status: passed` before release.
+
 ## Development
 
 ```bash
+git clone https://github.com/SylphxAI/video-reader-mcp.git
+cd video-reader-mcp
 bun install
-bun run typecheck
-bun test
 bun run build
+bun test
+bun run doctor
+bun run benchmark:release-gate
+```
+
+Useful checks:
+
+```bash
+bun run check
+bun run typecheck
+bun run benchmark:release-gate
 ```
 
-Unit tests mock parsers and do not require ffmpeg in CI. Integration with real media is optional locally.
+Example `read_video` requests live in [`examples/`](examples/). CI runs parser,
+fixture corpus, doctor, and release-gate checks; integration tests exercise ffmpeg
+when available on the runner.
+
+## Support
+
+- [Issues](https://github.com/SylphxAI/video-reader-mcp/issues)
+- [npm package](https://www.npmjs.com/package/@sylphx/video-reader-mcp)
+- Portfolio orchestration: [smart-reader-mcp](https://github.com/SylphxAI/smart-reader-mcp)
+
+## Help this reach more builders
+
+If frame-by-frame vision guesses have wasted your context, your citations, or
+your trust in agent output, you are exactly who this project is for.
+
+**[⭐ Star the repo](https://github.com/SylphxAI/video-reader-mcp)** — it is the
+fastest way to help more agent builders find evidence-first video reading. Share
+it in your MCP client setup, team wiki, or agent stack README.
+
+### Discovery (in progress)
+
+| Channel | Status |
+| --- | --- |
+| [Glama MCP directory](https://glama.ai/mcp/servers/SylphxAI/video-reader-mcp) | Listed — [claim server](https://glama.ai/mcp/servers/SylphxAI/video-reader-mcp/admin) for full discoverability |
+| [Official MCP Registry](https://registry.modelcontextprotocol.io/v0.1/servers?search=io.github.SylphxAI/video-reader-mcp) | Listed — `io.github.SylphxAI/video-reader-mcp` @ v0.1.0 |
+| [TensorBlock MCP Index PR #1113](https://github.com/TensorBlock/awesome-mcp-servers/pull/1113) | Open — multimedia/document processing listing |
+| [MCP servers community issue #4500](https://github.com/modelcontextprotocol/servers/issues/4500) | Open — community server highlight |
+| [mcp.so listing issue #3068](https://github.com/chatmcp/mcpso/issues/3068) | Open — directory submission request |
+| [mcpservers.org submit](https://mcpservers.org/submit) | Not listed yet — free web-form submission |
+
+Know another MCP directory? [Open an issue](https://github.com/SylphxAI/video-reader-mcp/issues/new) with the link.
 
 ## License
 
-MIT © [SylphxAI](https://github.com/SylphxAI)
+MIT © [SylphxAI](https://github.com/SylphxAI)

package/bin/video-reader-mcp

@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+# Video Reader MCP launcher — Rust rmcp only (fail-closed, no TS stdio fallback).
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+
+resolve_rust_bin() {
+  if [[ -n "${VIDEO_READER_MCP_RUST_BIN:-}" && -x "${VIDEO_READER_MCP_RUST_BIN}" ]]; then
+    printf '%s\n' "${VIDEO_READER_MCP_RUST_BIN}"
+    return 0
+  fi
+
+  for candidate in \
+    "$ROOT/bin/native/video-reader-mcp-server" \
+    "$ROOT/target/release/video-reader-mcp-server" \
+    "$ROOT/target/debug/video-reader-mcp-server"; do
+    if [[ -x "$candidate" ]]; then
+      printf '%s\n' "$candidate"
+      return 0
+    fi
+  done
+
+  return 1
+}
+
+resolve_transport() {
+  if [[ -n "${VIDEO_READER_MCP_TRANSPORT:-}" ]]; then
+    printf '%s\n' "${VIDEO_READER_MCP_TRANSPORT}"
+    return 0
+  fi
+  if [[ -n "${MCP_TRANSPORT:-}" ]]; then
+    printf '%s\n' "${MCP_TRANSPORT}"
+    return 0
+  fi
+  printf '%s\n' "stdio"
+}
+
+if bin="$(resolve_rust_bin)"; then
+  transport="$(resolve_transport)"
+  if [[ "$transport" == "http" ]]; then
+    export MCP_TRANSPORT=http
+    export VIDEO_READER_MCP_TRANSPORT=http
+  fi
+  exec "$bin" "$@"
+fi
+
+echo "[video-reader-mcp] Rust MCP server (rmcp) is not built. Run: bun run build:rust" >&2
+exit 1

package/dist/doctor-cli.js

@@ -0,0 +1,167 @@
+#!/usr/bin/env node
+
+// src/doctor-cli.ts
+import { createRequire } from "node:module";
+
+// src/doctor.ts
+import { spawnSync } from "node:child_process";
+import { existsSync as existsSync2 } from "node:fs";
+import path2 from "node:path";
+import { fileURLToPath as fileURLToPath2 } from "node:url";
+
+// src/engine/rust-timeline.ts
+import { existsSync } from "node:fs";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+var here = path.dirname(fileURLToPath(import.meta.url));
+function resolveRustCliBinary() {
+  const env = process.env.VIDEO_READER_CLI;
+  if (env && existsSync(env)) {
+    return env;
+  }
+  const release = path.join(here, "../../target/release/video-reader-cli");
+  if (existsSync(release)) {
+    return release;
+  }
+  const debug = path.join(here, "../../target/debug/video-reader-cli");
+  if (existsSync(debug)) {
+    return debug;
+  }
+  return "video-reader-cli";
+}
+
+// src/utils/exec.ts
+import { execFile } from "node:child_process";
+import { promisify } from "node:util";
+var execFileAsync = promisify(execFile);
+var DEFAULT_MAX_BUFFER = 16 * 1024 * 1024;
+var binaryCache = new Map;
+var isBinaryAvailable = async (binary) => {
+  const cached = binaryCache.get(binary);
+  if (cached !== undefined)
+    return cached;
+  try {
+    await execFileAsync(binary, ["-version"], { timeout: 5000, encoding: "utf8" });
+    binaryCache.set(binary, true);
+    return true;
+  } catch {
+    try {
+      await execFileAsync(binary, ["--version"], { timeout: 5000, encoding: "utf8" });
+      binaryCache.set(binary, true);
+      return true;
+    } catch {
+      binaryCache.set(binary, false);
+      return false;
+    }
+  }
+};
+
+// src/doctor.ts
+var here2 = path2.dirname(fileURLToPath2(import.meta.url));
+var probeBinary = async (id, binary, required) => {
+  const available = await isBinaryAvailable(binary);
+  if (available) {
+    return {
+      id,
+      status: "ok",
+      message: `${binary} is installed and responds to -version.`
+    };
+  }
+  return {
+    id,
+    status: required ? "fail" : "warn",
+    message: required ? `${binary} is required for read_video but was not found on PATH.` : `${binary} is optional for some extraction paths.`
+  };
+};
+var probeRustTimelineCli = () => {
+  const binary = resolveRustCliBinary();
+  if (binary !== "video-reader-cli" && existsSync2(binary)) {
+    const probe = spawnSync(binary, [], {
+      input: JSON.stringify({
+        tool: "build_cache_key",
+        input: {
+          source_hash: "abc123",
+          options: { include_streams: true, include_chapters: true }
+        }
+      }),
+      encoding: "utf8",
+      timeout: 5000
+    });
+    if (probe.status === 0) {
+      return {
+        id: "rust_timeline_cli",
+        status: "ok",
+        message: `Rust timeline CLI is available at ${binary}.`
+      };
+    }
+  }
+  const release = path2.join(here2, "../target/release/video-reader-cli");
+  const debug = path2.join(here2, "../target/debug/video-reader-cli");
+  if (existsSync2(release) || existsSync2(debug)) {
+    return {
+      id: "rust_timeline_cli",
+      status: "ok",
+      message: "Rust timeline CLI is built locally."
+    };
+  }
+  return {
+    id: "rust_timeline_cli",
+    status: "warn",
+    message: "Rust timeline CLI is not built. Run `cargo build --release` to enable VIDEO_READER_USE_RUST_TIMELINE=1."
+  };
+};
+var probeNode = () => {
+  const version = process.versions.node;
+  const major = Number.parseInt(version.split(".")[0] ?? "0", 10);
+  if (major >= 22) {
+    return {
+      id: "node",
+      status: "ok",
+      message: `Node.js ${version} meets the >=22.13 requirement.`
+    };
+  }
+  return {
+    id: "node",
+    status: "warn",
+    message: `Node.js ${version} is below the recommended >=22.13 runtime.`
+  };
+};
+var aggregateStatus = (checks) => {
+  if (checks.some((check) => check.status === "fail")) {
+    return "unavailable";
+  }
+  if (checks.some((check) => check.status === "warn")) {
+    return "degraded";
+  }
+  return "ready";
+};
+async function runDoctor(version) {
+  const checks = [
+    probeNode(),
+    probeRustTimelineCli(),
+    await probeBinary("ffprobe", "ffprobe", true),
+    await probeBinary("ffmpeg", "ffmpeg", false)
+  ];
+  return {
+    profile: "video_reader_doctor",
+    version,
+    status: aggregateStatus(checks),
+    checks
+  };
+}
+function formatDoctorReport(report) {
+  return JSON.stringify(report, null, 2);
+}
+
+// src/doctor-cli.ts
+var require2 = createRequire(import.meta.url);
+var packageJson = require2("../package.json");
+async function main() {
+  const report = await runDoctor(packageJson.version);
+  console.log(formatDoctorReport(report));
+  process.exit(report.status === "unavailable" ? 1 : 0);
+}
+main().catch((error) => {
+  console.error("[Video Reader MCP] Doctor error:", error);
+  process.exit(1);
+});

package/package.json

@@ -1,23 +1,28 @@
 {
   "name": "@sylphx/video-reader-mcp",
-  "version": "0.1.0",
+  "version": "0.1.2",
   "mcpName": "io.github.SylphxAI/video-reader-mcp",
-  "description": "Evidence-first video reading for AI agents — ffprobe, subtitles, scenes, transcripts, and timelines without frame-by-frame LLM vision.",
+  "description": "Cue \u2014 Evidence-first video reading for AI agents \u2014 ffprobe, subtitles, scenes, transcripts, and timelines without frame-by-frame LLM vision.",
   "type": "module",
   "publishConfig": {
     "access": "public"
   },
   "license": "MIT",
   "bin": {
-    "video-reader-mcp": "./dist/index.js"
+    "video-reader-mcp": "./bin/video-reader-mcp",
+    "cue": "./bin/video-reader-mcp"
   },
   "files": [
+    "bin/",
     "dist/",
     "README.md",
-    "LICENSE"
+    "LICENSE",
+    "src/"
   ],
   "exports": {
-    ".": "./dist/index.js"
+    ".": "./dist/doctor-cli.js",
+    "./sdk": "./src/sdk.ts",
+    "./cue": "./src/sdk.ts"
   },
   "repository": {
     "type": "git",
@@ -39,17 +44,24 @@
     "node": ">=22.13.0"
   },
   "scripts": {
-    "build": "tsc && chmod +x dist/index.js",
+    "build:rust": "cargo build --release -p video-reader-core -p video-reader-cli -p video-reader-mcp-server && bun scripts/stage-rust-mcp.ts",
+    "test:rust": "cargo test",
+    "build": "bun build src/doctor-cli.ts --outdir dist --target node && chmod +x dist/doctor-cli.js",
     "typecheck": "tsc --noEmit -p tsconfig.json && tsc --noEmit -p tsconfig.test.json",
     "test": "bun test",
+    "test:rust-timeline": "cargo test && bun test test/rustTimeline.boundary.test.ts",
     "check": "biome check .",
     "check:fix": "biome check --write .",
     "sync:server-json": "bun scripts/sync-server-json.ts",
     "version-packages": "changeset version && bun run sync:server-json",
-    "start": "node dist/index.js",
+    "start": "./bin/video-reader-mcp",
     "clean": "rm -rf dist coverage",
-    "prepublishOnly": "bun run clean && bun run build",
-    "release": "bun run typecheck && bun run check && bun run build && bun test && changeset publish"
+    "prepublishOnly": "bun run clean && bun run build && bun run build:rust",
+    "release": "bun run typecheck && bun run check && bun run build && bun test && changeset publish",
+    "doctor": "bun run src/doctor-cli.ts",
+    "benchmark:release-gate": "bun scripts/release-gate.ts",
+    "benchmark:public-proof": "bun scripts/public-proof.ts",
+    "brand:pack-plan": "bun scripts/brand-pack-plan.ts"
   },
   "dependencies": {
     "@modelcontextprotocol/sdk": "^1.29.0",
@@ -63,5 +75,6 @@
     "@types/node": "^25.9.1",
     "typescript": "^7.0.2"
   },
-  "packageManager": "bun@1.3.12"
+  "packageManager": "bun@1.3.12",
+  "private": false
 }

package/src/doctor-cli.ts

@@ -0,0 +1,18 @@
+#!/usr/bin/env node
+
+import { createRequire } from 'node:module';
+import { formatDoctorReport, runDoctor } from './doctor.js';
+
+const require = createRequire(import.meta.url);
+const packageJson = require('../package.json') as { version: string };
+
+async function main(): Promise<void> {
+  const report = await runDoctor(packageJson.version);
+  console.log(formatDoctorReport(report));
+  process.exit(report.status === 'unavailable' ? 1 : 0);
+}
+
+main().catch((error: unknown) => {
+  console.error('[Video Reader MCP] Doctor error:', error);
+  process.exit(1);
+});