davinci-resolve-mcp 2.49.0 → 2.51.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -2,6 +2,48 @@
2
2
 
3
3
  Release history for the DaVinci Resolve MCP Server. The latest release is summarized in the root README; older entries live here to keep the README focused.
4
4
 
5
+ ## What's New in v2.51.0
6
+
7
+ CLAP audio embeddings — the final phase of the post-program improvements.
8
+ Detection-based like every other backend (never auto-installed); local
9
+ compute, so nothing touches the caps ledger.
10
+
11
+ - **Added** audio embedding backend detection: CLAP via `transformers`
12
+ (laion/clap-htsat-unfused, preferred) or the `laion_clap` package, plus
13
+ torch + ffmpeg. `capabilities` gains an `audio` block with install
14
+ guidance and a `clap_audio` Tools-page entry.
15
+ - **Added** `build_embeddings(kinds=["audio"])`: one CLAP window per shot
16
+ (center-cropped to ~10 s, piped from the source media as raw PCM —
17
+ read-only on source media, no temp files) plus a clip-level mean vector,
18
+ stored as `embedding_kind="audio"` rows. Idempotent via content hashes;
19
+ offline media is reported in `skipped_missing_media`; a missing backend
20
+ is a graceful skip with install guidance.
21
+ - **Added** `find_similar(kind="audio")`: shot/clip queries over the audio
22
+ vectors, and free-text queries ("engine revving") via the CLAP text
23
+ encoder.
24
+
25
+ ## What's New in v2.50.0
26
+
27
+ The last JSON-fed readers now source from the DB-canonical analysis store
28
+ (consistency/perf hygiene — the JSON export is lockstep with the DB, so this
29
+ is shape-preserving by construction).
30
+
31
+ - **Changed** `summarize_reports` aggregates from the DB (blob + human
32
+ overlay) when every report dir on disk is covered by an ingested clip row;
33
+ pre-v9 roots and MIXED roots (some clips not ingested) fall back WHOLESALE
34
+ to the JSON walk — a partial DB view would silently under-report. The
35
+ summary gains a `"source": "db"|"json"` key for observability; the F1
36
+ provenance map (source_reports / missing_reports) is unchanged.
37
+ - **Changed** `build_analysis_index` sources local reports from the DB
38
+ instead of re-parsing every `analysis.json`; job-linked EXTERNAL report
39
+ paths (their rows live under another project's DB) and pre-v9 dirs keep
40
+ the JSON read. The FTS schema and the query surface are identical; the
41
+ result gains `report_sources` counts.
42
+ - **Added** DB-vs-JSON parity tests (semantic equality with normalized
43
+ ordering) plus wholesale-fallback regressions for mixed and pre-v9 roots;
44
+ live-validated on the sample analysis root (summary, index counts, and
45
+ FTS query results identical on both paths).
46
+
5
47
  ## What's New in v2.49.0
6
48
 
7
49
  Cross-shot relationships (spec §4 — pattern recognition only): the shot
package/README.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # DaVinci Resolve MCP Server
2
2
 
3
- [![Version](https://img.shields.io/badge/version-2.49.0-blue.svg)](https://github.com/samuelgursky/davinci-resolve-mcp/releases)
3
+ [![Version](https://img.shields.io/badge/version-2.51.0-blue.svg)](https://github.com/samuelgursky/davinci-resolve-mcp/releases)
4
4
  [![npm](https://img.shields.io/npm/v/davinci-resolve-mcp.svg?label=npm&color=CB3837)](https://www.npmjs.com/package/davinci-resolve-mcp)
5
5
  [![API Coverage](https://img.shields.io/badge/API%20Coverage-100%25-brightgreen.svg)](docs/reference/api-coverage.md)
6
6
  [![Tools](https://img.shields.io/badge/MCP%20Tools-34%20(341%20full)-blue.svg)](#server-modes)
package/docs/SKILL.md CHANGED
@@ -547,15 +547,23 @@ no editorial suggestions): `same_setup_as` / `alt_take_of` (symmetric) and
547
547
  vendor tokens, so nothing here touches the caps ledger. Backends are
548
548
  detected, never installed (capabilities lists them with install guidance):
549
549
  text = ollama serving `nomic-embed-text` or sentence-transformers; visual =
550
- open_clip (ViT-B-32, needs torch).
551
- - `build_embeddings(kinds=["text","visual"]?, clip_id?)` idempotent;
552
- embeds clip summaries, shot descriptions (+ deep field groups), transcript
553
- segments, and sampled frames (per-shot visual vector = mean of its
554
- frames'). Only re-embeds entities whose content changed.
555
- - `find_similar(text=… | clip_id=… | clip_id+shot_index, kind="text"|"visual",
556
- entity_types?, limit?)` brute-force cosine over the project's vectors.
557
- Free-text visual queries use the CLIP text encoder ("cracked windshield"
558
- finds the frame). Results carry scores plus clip/shot/segment context.
550
+ open_clip (ViT-B-32, needs torch); audio (v2.51.0+) = CLAP via
551
+ `transformers` (laion/clap-htsat-unfused, preferred) or the `laion_clap`
552
+ package needs torch + ffmpeg.
553
+ - `build_embeddings(kinds=["text","visual","audio"]?, clip_id?)`
554
+ idempotent; embeds clip summaries, shot descriptions (+ deep field
555
+ groups), transcript segments, and sampled frames (per-shot visual vector
556
+ = mean of its frames'). `kinds=["audio"]` embeds one CLAP window per shot
557
+ (center-cropped to ~10s, piped from the source media as raw PCM —
558
+ read-only, no temp files) plus a clip-level mean vector; clips whose
559
+ media is offline are reported in `skipped_missing_media`. Only re-embeds
560
+ entities whose content changed.
561
+ - `find_similar(text=… | clip_id=… | clip_id+shot_index,
562
+ kind="text"|"visual"|"audio", entity_types?, limit?)` — brute-force
563
+ cosine over the project's vectors. Free-text visual queries use the CLIP
564
+ text encoder ("cracked windshield" finds the frame); free-text audio
565
+ queries use the CLAP text encoder ("engine revving" finds the shot).
566
+ Results carry scores plus clip/shot/segment context.
559
567
  The panel search box gains a `Semantic` toggle when a text backend is
560
568
  detected. Vectors live in the per-project DB (schema v10).
561
569
 
package/install.py CHANGED
@@ -35,7 +35,7 @@ from src.utils.update_check import (
35
35
 
36
36
  # ─── Version ──────────────────────────────────────────────────────────────────
37
37
 
38
- VERSION = "2.49.0"
38
+ VERSION = "2.51.0"
39
39
  # Only hard floor: mcp[cli] requires Python 3.10+. There is no upper bound —
40
40
  # Resolve's scripting bridge loads into newer interpreters on recent builds
41
41
  # (Python 3.14 verified against Resolve Studio 20.3.2). Older Resolve builds
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "davinci-resolve-mcp",
3
- "version": "2.49.0",
3
+ "version": "2.51.0",
4
4
  "description": "NPM bootstrapper for the DaVinci Resolve MCP Server.",
5
5
  "license": "MIT",
6
6
  "author": "Samuel Gursky <samgursky@gmail.com>",
@@ -80,7 +80,7 @@ if not logging.getLogger().handlers:
80
80
  handlers=[logging.StreamHandler()],
81
81
  )
82
82
 
83
- VERSION = "2.49.0"
83
+ VERSION = "2.51.0"
84
84
  logger = logging.getLogger("davinci-resolve-mcp")
85
85
  logger.info(f"Starting DaVinci Resolve MCP Server v{VERSION}")
86
86
  logger.info(f"Detected platform: {get_platform()}")
package/src/server.py CHANGED
@@ -11,7 +11,7 @@ Usage:
11
11
  python src/server.py --full # Start the 341-tool granular server instead
12
12
  """
13
13
 
14
- VERSION = "2.49.0"
14
+ VERSION = "2.51.0"
15
15
 
16
16
  import base64
17
17
  import os
@@ -1,12 +1,15 @@
1
1
  """Embeddings + similarity search (Phase C of the analysis program).
2
2
 
3
3
  Text vectors over clip/shot summaries and transcript segments, CLIP image
4
- vectors over sampled frames. Backends are detected, never installed (the
5
- whisper pattern):
4
+ vectors over sampled frames, CLAP audio vectors over per-shot audio windows.
5
+ Backends are detected, never installed (the whisper pattern):
6
6
 
7
7
  - text: ollama serving ``nomic-embed-text`` (preferred — works on Apple
8
8
  Silicon via Metal), or ``sentence_transformers`` when installed.
9
9
  - visual: ``open_clip_torch`` (ViT-B-32) when installed alongside torch.
10
+ - audio: ``transformers`` ClapModel (laion/clap-htsat-unfused; preferred) or
11
+ ``laion_clap`` when installed alongside torch; needs ffmpeg. Audio windows
12
+ are piped from the source media as raw PCM — read-only, no temp files.
10
13
 
11
14
  Vectors live in the per-project DB (schema v10 ``embeddings`` table) as
12
15
  float32 BLOBs, one row per (entity, kind, model). Similarity is brute-force
@@ -37,12 +40,17 @@ OLLAMA_TEXT_MODEL = os.environ.get("DAVINCI_RESOLVE_MCP_EMBED_MODEL", "nomic-emb
37
40
  SENTENCE_TRANSFORMERS_MODEL = "all-MiniLM-L6-v2"
38
41
  OPEN_CLIP_MODEL = ("ViT-B-32", "laion2b_s34b_b79k")
39
42
 
43
+ CLAP_HF_MODEL = "laion/clap-htsat-unfused"
44
+ CLAP_SAMPLE_RATE = 48000
45
+ AUDIO_WINDOW_SECONDS = 10.0 # CLAP works on ~10s windows; longer shots are center-cropped
46
+
40
47
  _PROBE_TIMEOUT_SECONDS = 2.0
41
48
  _EMBED_TIMEOUT_SECONDS = 120.0
42
49
 
43
50
  # Lazy singletons for heavyweight local models.
44
51
  _ST_MODEL = None
45
52
  _CLIP_STATE: Optional[Tuple[Any, Any, Any]] = None # (model, preprocess, torch)
53
+ _CLAP_STATE: Optional[Tuple[str, Any]] = None # (backend, state)
46
54
 
47
55
 
48
56
  def _now() -> str:
@@ -140,6 +148,26 @@ def detect_embedding_capabilities() -> Dict[str, Any]:
140
148
  else "pip install torch open_clip_torch"
141
149
  )
142
150
 
151
+ transformers_available = importlib.util.find_spec("transformers") is not None
152
+ laion_clap_available = importlib.util.find_spec("laion_clap") is not None
153
+ ffmpeg_available = bool(shutil.which("ffmpeg"))
154
+ audio_backends: List[str] = []
155
+ if torch_available and ffmpeg_available:
156
+ if transformers_available:
157
+ audio_backends.append("transformers_clap")
158
+ if laion_clap_available:
159
+ audio_backends.append("laion_clap")
160
+ if not audio_backends:
161
+ if not ffmpeg_available:
162
+ guidance["audio"] = "Install ffmpeg (audio windows are extracted with it)."
163
+ else:
164
+ guidance["audio"] = (
165
+ "pip install transformers (CLAP via laion/clap-htsat-unfused), "
166
+ "or pip install laion_clap"
167
+ if torch_available
168
+ else "pip install torch transformers"
169
+ )
170
+
143
171
  return {
144
172
  "success": True,
145
173
  "no_auto_install": True,
@@ -156,6 +184,12 @@ def detect_embedding_capabilities() -> Dict[str, Any]:
156
184
  "backends": ["open_clip"] if visual_available else [],
157
185
  "model": "-".join(OPEN_CLIP_MODEL) if visual_available else None,
158
186
  },
187
+ "audio": {
188
+ "available": bool(audio_backends),
189
+ "backends": audio_backends,
190
+ "model": CLAP_HF_MODEL if audio_backends else None,
191
+ "ffmpeg": ffmpeg_available,
192
+ },
159
193
  "install_guidance": guidance,
160
194
  }
161
195
 
@@ -279,6 +313,148 @@ def embed_text_for_visual_query(text: str) -> Dict[str, Any]:
279
313
  return {"success": False, "error": f"{type(exc).__name__}: {exc}"}
280
314
 
281
315
 
316
+ # ── audio (CLAP) ─────────────────────────────────────────────────────────────
317
+
318
+
319
+ def _clap_state() -> Tuple[str, Any]:
320
+ """Lazy-load the preferred CLAP backend: transformers (HF-cached, kept
321
+ current) over the laion_clap package."""
322
+ global _CLAP_STATE
323
+ if _CLAP_STATE is None:
324
+ caps = detect_embedding_capabilities()["audio"]
325
+ backends = caps["backends"]
326
+ if "transformers_clap" in backends:
327
+ from transformers import ClapModel, ClapProcessor
328
+
329
+ model = ClapModel.from_pretrained(CLAP_HF_MODEL)
330
+ model.eval()
331
+ processor = ClapProcessor.from_pretrained(CLAP_HF_MODEL)
332
+ _CLAP_STATE = ("transformers_clap", (model, processor))
333
+ elif "laion_clap" in backends:
334
+ import laion_clap
335
+
336
+ module = laion_clap.CLAP_Module(enable_fusion=False)
337
+ module.load_ckpt() # default pretrained checkpoint
338
+ _CLAP_STATE = ("laion_clap", module)
339
+ else:
340
+ raise RuntimeError("No audio-embedding backend available")
341
+ return _CLAP_STATE
342
+
343
+
344
+ def _extract_audio_window(file_path: str, start_seconds: float, duration_seconds: float):
345
+ """Mono 48 kHz float32 samples piped straight from ffmpeg — read-only on
346
+ the source media, no temp files. Returns None when the window is empty
347
+ (e.g. video-only media)."""
348
+ import subprocess
349
+
350
+ import numpy as np
351
+
352
+ command = [
353
+ "ffmpeg", "-v", "error",
354
+ "-ss", f"{max(0.0, float(start_seconds)):.3f}",
355
+ "-t", f"{max(0.1, float(duration_seconds)):.3f}",
356
+ "-i", file_path,
357
+ "-vn", "-ac", "1", "-ar", str(CLAP_SAMPLE_RATE),
358
+ "-f", "f32le", "-",
359
+ ]
360
+ try:
361
+ proc = subprocess.run(command, capture_output=True, timeout=120, check=False)
362
+ except (OSError, subprocess.TimeoutExpired):
363
+ return None
364
+ if proc.returncode != 0 or not proc.stdout:
365
+ return None
366
+ samples = np.frombuffer(proc.stdout, dtype=np.float32)
367
+ return samples if samples.size else None
368
+
369
+
370
+ def _clap_audio_vectors(waveforms: List[Any]) -> Tuple[List[List[float]], str]:
371
+ backend, state = _clap_state()
372
+ if backend == "transformers_clap":
373
+ import torch
374
+
375
+ model, processor = state
376
+ inputs = processor(audios=waveforms, sampling_rate=CLAP_SAMPLE_RATE, return_tensors="pt", padding=True)
377
+ with torch.no_grad():
378
+ features = model.get_audio_features(**inputs)
379
+ features = features / features.norm(dim=-1, keepdim=True)
380
+ return [[float(x) for x in row.tolist()] for row in features], f"transformers_clap:{CLAP_HF_MODEL}"
381
+ import numpy as np
382
+
383
+ module = state
384
+ vectors = module.get_audio_embedding_from_data(x=np.stack(waveforms), use_tensor=False)
385
+ return [[float(x) for x in row] for row in vectors], "laion_clap:default"
386
+
387
+
388
+ def embed_audio_windows(file_path: str, windows: List[Tuple[float, float]]) -> Dict[str, Any]:
389
+ """Embed (start_seconds, end_seconds) windows of one media file with CLAP.
390
+
391
+ Windows longer than AUDIO_WINDOW_SECONDS are center-cropped. Vector slots
392
+ are None for windows with no decodable audio.
393
+ """
394
+ caps = detect_embedding_capabilities()
395
+ if not caps["audio"]["available"]:
396
+ return {
397
+ "success": False,
398
+ "error": "No audio-embedding backend available",
399
+ "install_guidance": caps["install_guidance"].get("audio"),
400
+ }
401
+ if not os.path.isfile(file_path):
402
+ return {"success": False, "error": f"media not found: {file_path}"}
403
+ waveforms: List[Any] = []
404
+ slots: List[Optional[int]] = []
405
+ for start, end in windows:
406
+ duration = max(0.1, float(end) - float(start))
407
+ if duration > AUDIO_WINDOW_SECONDS:
408
+ start = float(start) + (duration - AUDIO_WINDOW_SECONDS) / 2.0
409
+ duration = AUDIO_WINDOW_SECONDS
410
+ samples = _extract_audio_window(file_path, float(start), duration)
411
+ if samples is None:
412
+ slots.append(None)
413
+ continue
414
+ slots.append(len(waveforms))
415
+ waveforms.append(samples)
416
+ if not waveforms:
417
+ return {"success": True, "vectors": [None] * len(windows), "model": None,
418
+ "note": "no decodable audio in any window"}
419
+ try:
420
+ vectors, model = _clap_audio_vectors(waveforms)
421
+ except Exception as exc: # noqa: BLE001 — backend failures surface as data
422
+ return {"success": False, "error": f"{type(exc).__name__}: {exc}"}
423
+ return {
424
+ "success": True,
425
+ "vectors": [vectors[slot] if slot is not None else None for slot in slots],
426
+ "model": model,
427
+ }
428
+
429
+
430
+ def embed_text_for_audio_query(text: str) -> Dict[str, Any]:
431
+ """CLAP text encoder — lets a free-text query ('engine revving') search
432
+ audio vectors."""
433
+ caps = detect_embedding_capabilities()
434
+ if not caps["audio"]["available"]:
435
+ return {
436
+ "success": False,
437
+ "error": "No audio-embedding backend available",
438
+ "install_guidance": caps["install_guidance"].get("audio"),
439
+ }
440
+ try:
441
+ backend, state = _clap_state()
442
+ if backend == "transformers_clap":
443
+ import torch
444
+
445
+ model, processor = state
446
+ inputs = processor(text=[str(text)], return_tensors="pt", padding=True)
447
+ with torch.no_grad():
448
+ features = model.get_text_features(**inputs)
449
+ features = features / features.norm(dim=-1, keepdim=True)
450
+ return {"success": True, "vector": [float(x) for x in features[0].tolist()],
451
+ "model": f"transformers_clap:{CLAP_HF_MODEL}"}
452
+ vectors = state.get_text_embedding([str(text), ""], use_tensor=False)
453
+ return {"success": True, "vector": [float(x) for x in vectors[0]], "model": "laion_clap:default"}
454
+ except Exception as exc: # noqa: BLE001
455
+ return {"success": False, "error": f"{type(exc).__name__}: {exc}"}
456
+
457
+
282
458
  # ── content builders ─────────────────────────────────────────────────────────
283
459
 
284
460
 
@@ -497,6 +673,78 @@ def build_embeddings(
497
673
  else:
498
674
  result["visual"] = {"success": True, "embedded": 0, "note": "all up to date"}
499
675
 
676
+ if "audio" in kinds:
677
+ caps = detect_embedding_capabilities()
678
+ if not caps["audio"]["available"]:
679
+ result["audio"] = {
680
+ "success": False,
681
+ "skipped": True,
682
+ "error": "No audio-embedding backend available",
683
+ "install_guidance": caps["install_guidance"].get("audio"),
684
+ }
685
+ result["success"] = False
686
+ return result
687
+ existing = _existing_rows(conn, "audio")
688
+ embedded_count = 0
689
+ missing_media: List[str] = []
690
+ audio_model: Optional[str] = None
691
+ for clip in conn.execute(f"SELECT * FROM clips{where}", args).fetchall():
692
+ clip = dict(clip)
693
+ file_path = str(clip.get("file_path") or "")
694
+ if not file_path or not os.path.isfile(file_path):
695
+ missing_media.append(str(clip.get("clip_name") or clip["clip_uuid"]))
696
+ continue
697
+ shots = [dict(s) for s in conn.execute(
698
+ "SELECT shot_uuid, time_seconds_start, time_seconds_end FROM shots "
699
+ "WHERE clip_uuid = ? ORDER BY shot_index",
700
+ (clip["clip_uuid"],),
701
+ ).fetchall()]
702
+ windows: List[Tuple[float, float]] = []
703
+ window_meta: List[Tuple[str, str, str]] = []
704
+ for shot in shots:
705
+ start, end = shot.get("time_seconds_start"), shot.get("time_seconds_end")
706
+ if start is None or end is None or float(end) - float(start) < 0.2:
707
+ continue
708
+ h = _content_hash(f"{file_path}|{start}|{end}")
709
+ if existing.get(("shot", str(shot["shot_uuid"]))) == h:
710
+ continue
711
+ windows.append((float(start), float(end)))
712
+ window_meta.append(("shot", str(shot["shot_uuid"]), h))
713
+ if not windows:
714
+ continue
715
+ embedded = embed_audio_windows(file_path, windows)
716
+ if not embedded.get("success"):
717
+ result["audio"] = embedded
718
+ result["success"] = False
719
+ return result
720
+ if not embedded.get("model"):
721
+ continue # no decodable audio in this clip (e.g. video-only)
722
+ audio_model = str(embedded["model"])
723
+ items: List[Tuple[str, str, str, List[float]]] = []
724
+ clip_vectors: List[List[float]] = []
725
+ for m, vector in zip(window_meta, embedded["vectors"]):
726
+ if vector is None:
727
+ continue
728
+ items.append((m[0], m[1], m[2], vector))
729
+ clip_vectors.append(vector)
730
+ # Clip-level audio vector = mean of its shot windows' (the
731
+ # per-shot-visual pattern).
732
+ if clip_vectors:
733
+ dim = len(clip_vectors[0])
734
+ mean = [sum(v[i] for v in clip_vectors) / len(clip_vectors) for i in range(dim)]
735
+ items.append((
736
+ "clip", str(clip["clip_uuid"]),
737
+ _content_hash(f"{file_path}|clip|{len(clip_vectors)}"), mean,
738
+ ))
739
+ embedded_count += _store_vectors(project_root, "audio", audio_model, items)
740
+ result["audio"] = {
741
+ "success": True,
742
+ "embedded": embedded_count,
743
+ "model": audio_model,
744
+ **({"skipped_missing_media": missing_media} if missing_media else {}),
745
+ **({"note": "all up to date"} if not embedded_count and not missing_media else {}),
746
+ }
747
+
500
748
  result["wall_clock_ms"] = int((time.time() - started) * 1000)
501
749
  counts = conn.execute(
502
750
  "SELECT embedding_kind, COUNT(*) AS n FROM embeddings GROUP BY embedding_kind"
@@ -569,8 +817,8 @@ def find_similar(
569
817
 
570
818
  conn = timeline_brain_db.connect(project_root)
571
819
  kind = (kind or "text").strip().lower()
572
- if kind not in ("text", "visual"):
573
- return {"success": False, "error": f"kind must be 'text' or 'visual', got {kind!r}"}
820
+ if kind not in ("text", "visual", "audio"):
821
+ return {"success": False, "error": f"kind must be 'text', 'visual', or 'audio', got {kind!r}"}
574
822
 
575
823
  exclude: Optional[Tuple[str, str]] = None
576
824
  query_vector: Optional[List[float]] = None
@@ -583,6 +831,12 @@ def find_similar(
583
831
  return embedded
584
832
  query_vector = embedded["vectors"][0]
585
833
  query_model = str(embedded["model"])
834
+ elif kind == "audio":
835
+ encoded = embed_text_for_audio_query(str(text))
836
+ if not encoded.get("success"):
837
+ return encoded
838
+ query_vector = encoded["vector"]
839
+ query_model = str(encoded["model"])
586
840
  else:
587
841
  encoded = embed_text_for_visual_query(str(text))
588
842
  if not encoded.get("success"):
@@ -1458,6 +1458,20 @@ TOOL_INSTALL: Dict[str, Dict[str, Any]] = {
1458
1458
  "verify": "python -c \"import open_clip\"",
1459
1459
  "notes": "Needs torch. Model weights (~350 MB) download on first use.",
1460
1460
  },
1461
+ "clap_audio": {
1462
+ "label": "CLAP (audio embeddings)",
1463
+ "bundle": "embeddings",
1464
+ "required_for": ["audio similarity (find_similar kind=audio)"],
1465
+ "commands": {
1466
+ "all": "pip install transformers",
1467
+ },
1468
+ "verify": "python -c \"import transformers\"",
1469
+ "notes": (
1470
+ "Needs torch + ffmpeg. Uses laion/clap-htsat-unfused (~600 MB, "
1471
+ "downloads on first use); the laion_clap package works as an "
1472
+ "alternative backend."
1473
+ ),
1474
+ },
1461
1475
  "whisper_cpp": {
1462
1476
  "label": "whisper.cpp",
1463
1477
  "bundle": "transcription",
@@ -1650,6 +1664,11 @@ def detect_capabilities(env: Optional[Dict[str, str]] = None) -> Dict[str, Any]:
1650
1664
  bool(embedding_caps.get("visual", {}).get("available")),
1651
1665
  {"python_module": "open_clip"},
1652
1666
  ),
1667
+ "clap_audio": _tool_entry(
1668
+ "clap_audio",
1669
+ bool(embedding_caps.get("audio", {}).get("available")),
1670
+ {"backends": embedding_caps.get("audio", {}).get("backends", [])},
1671
+ ),
1653
1672
  },
1654
1673
  "embeddings": embedding_caps,
1655
1674
  "transcription": {
@@ -5994,20 +6013,68 @@ def load_report(project_root: str, report_path: Optional[str] = None, clip_dir:
5994
6013
  return {"success": True, "path": path, "report": payload}
5995
6014
 
5996
6015
 
5997
- def summarize_reports(project_root: str) -> Dict[str, Any]:
5998
- root = normalize_path(project_root)
6016
+ def _collect_reports_for_summary(root: str) -> Tuple[List[Dict[str, Any]], List[str], str]:
6017
+ """(reports, report_paths, source) for summarize_reports.
6018
+
6019
+ DB-first: when every report dir on disk is covered by an ingested clip
6020
+ row, reports come from the DB-canonical store (blob + human overlay —
6021
+ identical content to the lockstep JSON export). Pre-v9 roots and MIXED
6022
+ roots (some clips not ingested) fall back WHOLESALE to the JSON walk —
6023
+ a partial DB view would silently under-report.
6024
+ """
5999
6025
  clips_root = os.path.join(root, "clips")
6000
- reports: List[Dict[str, Any]] = []
6001
- report_paths: List[str] = []
6026
+ disk_paths: List[str] = []
6002
6027
  if os.path.isdir(clips_root):
6003
6028
  for dirpath, _, filenames in os.walk(clips_root):
6004
6029
  if "analysis.json" in filenames:
6005
- report_path = os.path.join(dirpath, "analysis.json")
6030
+ disk_paths.append(os.path.join(dirpath, "analysis.json"))
6031
+ disk_paths.sort()
6032
+
6033
+ try:
6034
+ from src.utils import analysis_store, timeline_brain_db
6035
+
6036
+ conn = timeline_brain_db.connect(root)
6037
+ db_dirs = {
6038
+ str(r["clip_dir"]): str(r["clip_uuid"])
6039
+ for r in conn.execute(
6040
+ "SELECT clip_dir, clip_uuid FROM clips WHERE clip_dir IS NOT NULL"
6041
+ ).fetchall()
6042
+ }
6043
+ except Exception: # noqa: BLE001 — no DB (pre-v9) → JSON
6044
+ db_dirs = {}
6045
+ if disk_paths and db_dirs:
6046
+ dir_names = [os.path.basename(os.path.dirname(p)) for p in disk_paths]
6047
+ if all(name in db_dirs for name in dir_names):
6048
+ from src.utils import analysis_store
6049
+
6050
+ reports: List[Dict[str, Any]] = []
6051
+ complete = True
6052
+ for path, name in zip(disk_paths, dir_names):
6006
6053
  try:
6007
- reports.append(_read_json(report_path))
6008
- report_paths.append(report_path)
6009
- except (OSError, json.JSONDecodeError):
6010
- continue
6054
+ report = analysis_store.export_report(root, db_dirs[name])
6055
+ except Exception: # noqa: BLE001
6056
+ report = None
6057
+ if not isinstance(report, dict):
6058
+ complete = False
6059
+ break
6060
+ reports.append(report)
6061
+ if complete:
6062
+ return reports, disk_paths, "db"
6063
+
6064
+ reports = []
6065
+ report_paths: List[str] = []
6066
+ for path in disk_paths:
6067
+ try:
6068
+ reports.append(_read_json(path))
6069
+ report_paths.append(path)
6070
+ except (OSError, json.JSONDecodeError):
6071
+ continue
6072
+ return reports, report_paths, "json"
6073
+
6074
+
6075
+ def summarize_reports(project_root: str) -> Dict[str, Any]:
6076
+ root = normalize_path(project_root)
6077
+ reports, report_paths, reports_source = _collect_reports_for_summary(root)
6011
6078
  warnings = []
6012
6079
  motion_counts: Dict[str, int] = {}
6013
6080
  tags: Dict[str, int] = {}
@@ -6047,6 +6114,7 @@ def summarize_reports(project_root: str) -> Dict[str, Any]:
6047
6114
  summary = {
6048
6115
  "success": True,
6049
6116
  "project_root": root,
6117
+ "source": reports_source, # "db" (canonical store) | "json" (walk fallback)
6050
6118
  "clip_reports": len(reports),
6051
6119
  "motion_distribution": motion_counts,
6052
6120
  "technical_warning_count": len(warnings),
@@ -7202,9 +7270,42 @@ def build_analysis_index(project_root: str, *, index_path: Optional[Any] = None)
7202
7270
  conn.execute("INSERT INTO index_metadata (key, value) VALUES (?, ?)", ("fts_enabled", "1" if fts_enabled else "0"))
7203
7271
  conn.execute("INSERT INTO index_metadata (key, value) VALUES (?, ?)", ("image_blob_policy", "excluded"))
7204
7272
 
7273
+ # DB-first sourcing: local reports whose clip dir is ingested come from
7274
+ # the DB-canonical store (blob + human overlay — identical content to
7275
+ # the lockstep JSON export) instead of re-parsing every analysis.json.
7276
+ # Pre-v9 dirs and job-linked EXTERNAL report paths (their rows live
7277
+ # under another project's DB) keep the JSON read. The index schema and
7278
+ # the query surface are unchanged either way.
7279
+ clips_root_prefix = os.path.realpath(os.path.join(root, "clips")) + os.sep
7280
+ try:
7281
+ from src.utils import timeline_brain_db as _brain_db
7282
+
7283
+ _db_dirs = {
7284
+ str(r["clip_dir"]): str(r["clip_uuid"])
7285
+ for r in _brain_db.connect(root).execute(
7286
+ "SELECT clip_dir, clip_uuid FROM clips WHERE clip_dir IS NOT NULL"
7287
+ ).fetchall()
7288
+ }
7289
+ except Exception: # noqa: BLE001 — no DB (pre-v9) → JSON for everything
7290
+ _db_dirs = {}
7291
+ report_sources = {"db": 0, "json": 0}
7205
7292
  for report_path in sorted(_iter_analysis_report_files(root)):
7206
7293
  try:
7207
- report = _read_json(report_path)
7294
+ report = None
7295
+ if _db_dirs and os.path.realpath(report_path).startswith(clips_root_prefix):
7296
+ clip_uuid = _db_dirs.get(os.path.basename(os.path.dirname(report_path)))
7297
+ if clip_uuid:
7298
+ try:
7299
+ from src.utils import analysis_store as _analysis_store
7300
+
7301
+ report = _analysis_store.export_report(root, clip_uuid)
7302
+ except Exception: # noqa: BLE001 — fall back per-report
7303
+ report = None
7304
+ if isinstance(report, dict):
7305
+ report_sources["db"] += 1
7306
+ else:
7307
+ report = _read_json(report_path)
7308
+ report_sources["json"] += 1
7208
7309
  row_counts = _insert_analysis_report_into_index(conn, report_path, report, fts_enabled=fts_enabled)
7209
7310
  counts["clips"] += 1
7210
7311
  for key, value in row_counts.items():
@@ -7244,6 +7345,7 @@ def build_analysis_index(project_root: str, *, index_path: Optional[Any] = None)
7244
7345
  "image_blob_policy": "excluded",
7245
7346
  "fts_enabled": bool(counts["clips"]) and _sqlite_table_exists(db_path, "clips_fts"),
7246
7347
  "counts": counts,
7348
+ "report_sources": report_sources, # how many reports came from the DB vs JSON
7247
7349
  "failed_report_count": len(failed_reports),
7248
7350
  "failed_reports": failed_reports[:50],
7249
7351
  "size_bytes": os.path.getsize(db_path) if os.path.isfile(db_path) else 0,