davinci-resolve-mcp 2.49.0 → 2.51.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +42 -0
- package/README.md +1 -1
- package/docs/SKILL.md +17 -9
- package/install.py +1 -1
- package/package.json +1 -1
- package/src/granular/common.py +1 -1
- package/src/server.py +1 -1
- package/src/utils/embeddings.py +258 -4
- package/src/utils/media_analysis.py +112 -10
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,48 @@
|
|
|
2
2
|
|
|
3
3
|
Release history for the DaVinci Resolve MCP Server. The latest release is summarized in the root README; older entries live here to keep the README focused.
|
|
4
4
|
|
|
5
|
+
## What's New in v2.51.0
|
|
6
|
+
|
|
7
|
+
CLAP audio embeddings — the final phase of the post-program improvements.
|
|
8
|
+
Detection-based like every other backend (never auto-installed); local
|
|
9
|
+
compute, so nothing touches the caps ledger.
|
|
10
|
+
|
|
11
|
+
- **Added** audio embedding backend detection: CLAP via `transformers`
|
|
12
|
+
(laion/clap-htsat-unfused, preferred) or the `laion_clap` package, plus
|
|
13
|
+
torch + ffmpeg. `capabilities` gains an `audio` block with install
|
|
14
|
+
guidance and a `clap_audio` Tools-page entry.
|
|
15
|
+
- **Added** `build_embeddings(kinds=["audio"])`: one CLAP window per shot
|
|
16
|
+
(center-cropped to ~10 s, piped from the source media as raw PCM —
|
|
17
|
+
read-only on source media, no temp files) plus a clip-level mean vector,
|
|
18
|
+
stored as `embedding_kind="audio"` rows. Idempotent via content hashes;
|
|
19
|
+
offline media is reported in `skipped_missing_media`; a missing backend
|
|
20
|
+
is a graceful skip with install guidance.
|
|
21
|
+
- **Added** `find_similar(kind="audio")`: shot/clip queries over the audio
|
|
22
|
+
vectors, and free-text queries ("engine revving") via the CLAP text
|
|
23
|
+
encoder.
|
|
24
|
+
|
|
25
|
+
## What's New in v2.50.0
|
|
26
|
+
|
|
27
|
+
The last JSON-fed readers now source from the DB-canonical analysis store
|
|
28
|
+
(consistency/perf hygiene — the JSON export is lockstep with the DB, so this
|
|
29
|
+
is shape-preserving by construction).
|
|
30
|
+
|
|
31
|
+
- **Changed** `summarize_reports` aggregates from the DB (blob + human
|
|
32
|
+
overlay) when every report dir on disk is covered by an ingested clip row;
|
|
33
|
+
pre-v9 roots and MIXED roots (some clips not ingested) fall back WHOLESALE
|
|
34
|
+
to the JSON walk — a partial DB view would silently under-report. The
|
|
35
|
+
summary gains a `"source": "db"|"json"` key for observability; the F1
|
|
36
|
+
provenance map (source_reports / missing_reports) is unchanged.
|
|
37
|
+
- **Changed** `build_analysis_index` sources local reports from the DB
|
|
38
|
+
instead of re-parsing every `analysis.json`; job-linked EXTERNAL report
|
|
39
|
+
paths (their rows live under another project's DB) and pre-v9 dirs keep
|
|
40
|
+
the JSON read. The FTS schema and the query surface are identical; the
|
|
41
|
+
result gains `report_sources` counts.
|
|
42
|
+
- **Added** DB-vs-JSON parity tests (semantic equality with normalized
|
|
43
|
+
ordering) plus wholesale-fallback regressions for mixed and pre-v9 roots;
|
|
44
|
+
live-validated on the sample analysis root (summary, index counts, and
|
|
45
|
+
FTS query results identical on both paths).
|
|
46
|
+
|
|
5
47
|
## What's New in v2.49.0
|
|
6
48
|
|
|
7
49
|
Cross-shot relationships (spec §4 — pattern recognition only): the shot
|
package/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# DaVinci Resolve MCP Server
|
|
2
2
|
|
|
3
|
-
[](https://github.com/samuelgursky/davinci-resolve-mcp/releases)
|
|
4
4
|
[](https://www.npmjs.com/package/davinci-resolve-mcp)
|
|
5
5
|
[](docs/reference/api-coverage.md)
|
|
6
6
|
[-blue.svg)](#server-modes)
|
package/docs/SKILL.md
CHANGED
|
@@ -547,15 +547,23 @@ no editorial suggestions): `same_setup_as` / `alt_take_of` (symmetric) and
|
|
|
547
547
|
vendor tokens, so nothing here touches the caps ledger. Backends are
|
|
548
548
|
detected, never installed (capabilities lists them with install guidance):
|
|
549
549
|
text = ollama serving `nomic-embed-text` or sentence-transformers; visual =
|
|
550
|
-
open_clip (ViT-B-32, needs torch).
|
|
551
|
-
|
|
552
|
-
|
|
553
|
-
|
|
554
|
-
|
|
555
|
-
|
|
556
|
-
|
|
557
|
-
|
|
558
|
-
|
|
550
|
+
open_clip (ViT-B-32, needs torch); audio (v2.51.0+) = CLAP via
|
|
551
|
+
`transformers` (laion/clap-htsat-unfused, preferred) or the `laion_clap`
|
|
552
|
+
package — needs torch + ffmpeg.
|
|
553
|
+
- `build_embeddings(kinds=["text","visual","audio"]?, clip_id?)` —
|
|
554
|
+
idempotent; embeds clip summaries, shot descriptions (+ deep field
|
|
555
|
+
groups), transcript segments, and sampled frames (per-shot visual vector
|
|
556
|
+
= mean of its frames'). `kinds=["audio"]` embeds one CLAP window per shot
|
|
557
|
+
(center-cropped to ~10s, piped from the source media as raw PCM —
|
|
558
|
+
read-only, no temp files) plus a clip-level mean vector; clips whose
|
|
559
|
+
media is offline are reported in `skipped_missing_media`. Only re-embeds
|
|
560
|
+
entities whose content changed.
|
|
561
|
+
- `find_similar(text=… | clip_id=… | clip_id+shot_index,
|
|
562
|
+
kind="text"|"visual"|"audio", entity_types?, limit?)` — brute-force
|
|
563
|
+
cosine over the project's vectors. Free-text visual queries use the CLIP
|
|
564
|
+
text encoder ("cracked windshield" finds the frame); free-text audio
|
|
565
|
+
queries use the CLAP text encoder ("engine revving" finds the shot).
|
|
566
|
+
Results carry scores plus clip/shot/segment context.
|
|
559
567
|
The panel search box gains a `Semantic` toggle when a text backend is
|
|
560
568
|
detected. Vectors live in the per-project DB (schema v10).
|
|
561
569
|
|
package/install.py
CHANGED
|
@@ -35,7 +35,7 @@ from src.utils.update_check import (
|
|
|
35
35
|
|
|
36
36
|
# ─── Version ──────────────────────────────────────────────────────────────────
|
|
37
37
|
|
|
38
|
-
VERSION = "2.
|
|
38
|
+
VERSION = "2.51.0"
|
|
39
39
|
# Only hard floor: mcp[cli] requires Python 3.10+. There is no upper bound —
|
|
40
40
|
# Resolve's scripting bridge loads into newer interpreters on recent builds
|
|
41
41
|
# (Python 3.14 verified against Resolve Studio 20.3.2). Older Resolve builds
|
package/package.json
CHANGED
package/src/granular/common.py
CHANGED
|
@@ -80,7 +80,7 @@ if not logging.getLogger().handlers:
|
|
|
80
80
|
handlers=[logging.StreamHandler()],
|
|
81
81
|
)
|
|
82
82
|
|
|
83
|
-
VERSION = "2.
|
|
83
|
+
VERSION = "2.51.0"
|
|
84
84
|
logger = logging.getLogger("davinci-resolve-mcp")
|
|
85
85
|
logger.info(f"Starting DaVinci Resolve MCP Server v{VERSION}")
|
|
86
86
|
logger.info(f"Detected platform: {get_platform()}")
|
package/src/server.py
CHANGED
package/src/utils/embeddings.py
CHANGED
|
@@ -1,12 +1,15 @@
|
|
|
1
1
|
"""Embeddings + similarity search (Phase C of the analysis program).
|
|
2
2
|
|
|
3
3
|
Text vectors over clip/shot summaries and transcript segments, CLIP image
|
|
4
|
-
vectors over sampled frames
|
|
5
|
-
whisper pattern):
|
|
4
|
+
vectors over sampled frames, CLAP audio vectors over per-shot audio windows.
|
|
5
|
+
Backends are detected, never installed (the whisper pattern):
|
|
6
6
|
|
|
7
7
|
- text: ollama serving ``nomic-embed-text`` (preferred — works on Apple
|
|
8
8
|
Silicon via Metal), or ``sentence_transformers`` when installed.
|
|
9
9
|
- visual: ``open_clip_torch`` (ViT-B-32) when installed alongside torch.
|
|
10
|
+
- audio: ``transformers`` ClapModel (laion/clap-htsat-unfused; preferred) or
|
|
11
|
+
``laion_clap`` when installed alongside torch; needs ffmpeg. Audio windows
|
|
12
|
+
are piped from the source media as raw PCM — read-only, no temp files.
|
|
10
13
|
|
|
11
14
|
Vectors live in the per-project DB (schema v10 ``embeddings`` table) as
|
|
12
15
|
float32 BLOBs, one row per (entity, kind, model). Similarity is brute-force
|
|
@@ -37,12 +40,17 @@ OLLAMA_TEXT_MODEL = os.environ.get("DAVINCI_RESOLVE_MCP_EMBED_MODEL", "nomic-emb
|
|
|
37
40
|
SENTENCE_TRANSFORMERS_MODEL = "all-MiniLM-L6-v2"
|
|
38
41
|
OPEN_CLIP_MODEL = ("ViT-B-32", "laion2b_s34b_b79k")
|
|
39
42
|
|
|
43
|
+
CLAP_HF_MODEL = "laion/clap-htsat-unfused"
|
|
44
|
+
CLAP_SAMPLE_RATE = 48000
|
|
45
|
+
AUDIO_WINDOW_SECONDS = 10.0 # CLAP works on ~10s windows; longer shots are center-cropped
|
|
46
|
+
|
|
40
47
|
_PROBE_TIMEOUT_SECONDS = 2.0
|
|
41
48
|
_EMBED_TIMEOUT_SECONDS = 120.0
|
|
42
49
|
|
|
43
50
|
# Lazy singletons for heavyweight local models.
|
|
44
51
|
_ST_MODEL = None
|
|
45
52
|
_CLIP_STATE: Optional[Tuple[Any, Any, Any]] = None # (model, preprocess, torch)
|
|
53
|
+
_CLAP_STATE: Optional[Tuple[str, Any]] = None # (backend, state)
|
|
46
54
|
|
|
47
55
|
|
|
48
56
|
def _now() -> str:
|
|
@@ -140,6 +148,26 @@ def detect_embedding_capabilities() -> Dict[str, Any]:
|
|
|
140
148
|
else "pip install torch open_clip_torch"
|
|
141
149
|
)
|
|
142
150
|
|
|
151
|
+
transformers_available = importlib.util.find_spec("transformers") is not None
|
|
152
|
+
laion_clap_available = importlib.util.find_spec("laion_clap") is not None
|
|
153
|
+
ffmpeg_available = bool(shutil.which("ffmpeg"))
|
|
154
|
+
audio_backends: List[str] = []
|
|
155
|
+
if torch_available and ffmpeg_available:
|
|
156
|
+
if transformers_available:
|
|
157
|
+
audio_backends.append("transformers_clap")
|
|
158
|
+
if laion_clap_available:
|
|
159
|
+
audio_backends.append("laion_clap")
|
|
160
|
+
if not audio_backends:
|
|
161
|
+
if not ffmpeg_available:
|
|
162
|
+
guidance["audio"] = "Install ffmpeg (audio windows are extracted with it)."
|
|
163
|
+
else:
|
|
164
|
+
guidance["audio"] = (
|
|
165
|
+
"pip install transformers (CLAP via laion/clap-htsat-unfused), "
|
|
166
|
+
"or pip install laion_clap"
|
|
167
|
+
if torch_available
|
|
168
|
+
else "pip install torch transformers"
|
|
169
|
+
)
|
|
170
|
+
|
|
143
171
|
return {
|
|
144
172
|
"success": True,
|
|
145
173
|
"no_auto_install": True,
|
|
@@ -156,6 +184,12 @@ def detect_embedding_capabilities() -> Dict[str, Any]:
|
|
|
156
184
|
"backends": ["open_clip"] if visual_available else [],
|
|
157
185
|
"model": "-".join(OPEN_CLIP_MODEL) if visual_available else None,
|
|
158
186
|
},
|
|
187
|
+
"audio": {
|
|
188
|
+
"available": bool(audio_backends),
|
|
189
|
+
"backends": audio_backends,
|
|
190
|
+
"model": CLAP_HF_MODEL if audio_backends else None,
|
|
191
|
+
"ffmpeg": ffmpeg_available,
|
|
192
|
+
},
|
|
159
193
|
"install_guidance": guidance,
|
|
160
194
|
}
|
|
161
195
|
|
|
@@ -279,6 +313,148 @@ def embed_text_for_visual_query(text: str) -> Dict[str, Any]:
|
|
|
279
313
|
return {"success": False, "error": f"{type(exc).__name__}: {exc}"}
|
|
280
314
|
|
|
281
315
|
|
|
316
|
+
# ── audio (CLAP) ─────────────────────────────────────────────────────────────
|
|
317
|
+
|
|
318
|
+
|
|
319
|
+
def _clap_state() -> Tuple[str, Any]:
|
|
320
|
+
"""Lazy-load the preferred CLAP backend: transformers (HF-cached, kept
|
|
321
|
+
current) over the laion_clap package."""
|
|
322
|
+
global _CLAP_STATE
|
|
323
|
+
if _CLAP_STATE is None:
|
|
324
|
+
caps = detect_embedding_capabilities()["audio"]
|
|
325
|
+
backends = caps["backends"]
|
|
326
|
+
if "transformers_clap" in backends:
|
|
327
|
+
from transformers import ClapModel, ClapProcessor
|
|
328
|
+
|
|
329
|
+
model = ClapModel.from_pretrained(CLAP_HF_MODEL)
|
|
330
|
+
model.eval()
|
|
331
|
+
processor = ClapProcessor.from_pretrained(CLAP_HF_MODEL)
|
|
332
|
+
_CLAP_STATE = ("transformers_clap", (model, processor))
|
|
333
|
+
elif "laion_clap" in backends:
|
|
334
|
+
import laion_clap
|
|
335
|
+
|
|
336
|
+
module = laion_clap.CLAP_Module(enable_fusion=False)
|
|
337
|
+
module.load_ckpt() # default pretrained checkpoint
|
|
338
|
+
_CLAP_STATE = ("laion_clap", module)
|
|
339
|
+
else:
|
|
340
|
+
raise RuntimeError("No audio-embedding backend available")
|
|
341
|
+
return _CLAP_STATE
|
|
342
|
+
|
|
343
|
+
|
|
344
|
+
def _extract_audio_window(file_path: str, start_seconds: float, duration_seconds: float):
|
|
345
|
+
"""Mono 48 kHz float32 samples piped straight from ffmpeg — read-only on
|
|
346
|
+
the source media, no temp files. Returns None when the window is empty
|
|
347
|
+
(e.g. video-only media)."""
|
|
348
|
+
import subprocess
|
|
349
|
+
|
|
350
|
+
import numpy as np
|
|
351
|
+
|
|
352
|
+
command = [
|
|
353
|
+
"ffmpeg", "-v", "error",
|
|
354
|
+
"-ss", f"{max(0.0, float(start_seconds)):.3f}",
|
|
355
|
+
"-t", f"{max(0.1, float(duration_seconds)):.3f}",
|
|
356
|
+
"-i", file_path,
|
|
357
|
+
"-vn", "-ac", "1", "-ar", str(CLAP_SAMPLE_RATE),
|
|
358
|
+
"-f", "f32le", "-",
|
|
359
|
+
]
|
|
360
|
+
try:
|
|
361
|
+
proc = subprocess.run(command, capture_output=True, timeout=120, check=False)
|
|
362
|
+
except (OSError, subprocess.TimeoutExpired):
|
|
363
|
+
return None
|
|
364
|
+
if proc.returncode != 0 or not proc.stdout:
|
|
365
|
+
return None
|
|
366
|
+
samples = np.frombuffer(proc.stdout, dtype=np.float32)
|
|
367
|
+
return samples if samples.size else None
|
|
368
|
+
|
|
369
|
+
|
|
370
|
+
def _clap_audio_vectors(waveforms: List[Any]) -> Tuple[List[List[float]], str]:
|
|
371
|
+
backend, state = _clap_state()
|
|
372
|
+
if backend == "transformers_clap":
|
|
373
|
+
import torch
|
|
374
|
+
|
|
375
|
+
model, processor = state
|
|
376
|
+
inputs = processor(audios=waveforms, sampling_rate=CLAP_SAMPLE_RATE, return_tensors="pt", padding=True)
|
|
377
|
+
with torch.no_grad():
|
|
378
|
+
features = model.get_audio_features(**inputs)
|
|
379
|
+
features = features / features.norm(dim=-1, keepdim=True)
|
|
380
|
+
return [[float(x) for x in row.tolist()] for row in features], f"transformers_clap:{CLAP_HF_MODEL}"
|
|
381
|
+
import numpy as np
|
|
382
|
+
|
|
383
|
+
module = state
|
|
384
|
+
vectors = module.get_audio_embedding_from_data(x=np.stack(waveforms), use_tensor=False)
|
|
385
|
+
return [[float(x) for x in row] for row in vectors], "laion_clap:default"
|
|
386
|
+
|
|
387
|
+
|
|
388
|
+
def embed_audio_windows(file_path: str, windows: List[Tuple[float, float]]) -> Dict[str, Any]:
|
|
389
|
+
"""Embed (start_seconds, end_seconds) windows of one media file with CLAP.
|
|
390
|
+
|
|
391
|
+
Windows longer than AUDIO_WINDOW_SECONDS are center-cropped. Vector slots
|
|
392
|
+
are None for windows with no decodable audio.
|
|
393
|
+
"""
|
|
394
|
+
caps = detect_embedding_capabilities()
|
|
395
|
+
if not caps["audio"]["available"]:
|
|
396
|
+
return {
|
|
397
|
+
"success": False,
|
|
398
|
+
"error": "No audio-embedding backend available",
|
|
399
|
+
"install_guidance": caps["install_guidance"].get("audio"),
|
|
400
|
+
}
|
|
401
|
+
if not os.path.isfile(file_path):
|
|
402
|
+
return {"success": False, "error": f"media not found: {file_path}"}
|
|
403
|
+
waveforms: List[Any] = []
|
|
404
|
+
slots: List[Optional[int]] = []
|
|
405
|
+
for start, end in windows:
|
|
406
|
+
duration = max(0.1, float(end) - float(start))
|
|
407
|
+
if duration > AUDIO_WINDOW_SECONDS:
|
|
408
|
+
start = float(start) + (duration - AUDIO_WINDOW_SECONDS) / 2.0
|
|
409
|
+
duration = AUDIO_WINDOW_SECONDS
|
|
410
|
+
samples = _extract_audio_window(file_path, float(start), duration)
|
|
411
|
+
if samples is None:
|
|
412
|
+
slots.append(None)
|
|
413
|
+
continue
|
|
414
|
+
slots.append(len(waveforms))
|
|
415
|
+
waveforms.append(samples)
|
|
416
|
+
if not waveforms:
|
|
417
|
+
return {"success": True, "vectors": [None] * len(windows), "model": None,
|
|
418
|
+
"note": "no decodable audio in any window"}
|
|
419
|
+
try:
|
|
420
|
+
vectors, model = _clap_audio_vectors(waveforms)
|
|
421
|
+
except Exception as exc: # noqa: BLE001 — backend failures surface as data
|
|
422
|
+
return {"success": False, "error": f"{type(exc).__name__}: {exc}"}
|
|
423
|
+
return {
|
|
424
|
+
"success": True,
|
|
425
|
+
"vectors": [vectors[slot] if slot is not None else None for slot in slots],
|
|
426
|
+
"model": model,
|
|
427
|
+
}
|
|
428
|
+
|
|
429
|
+
|
|
430
|
+
def embed_text_for_audio_query(text: str) -> Dict[str, Any]:
|
|
431
|
+
"""CLAP text encoder — lets a free-text query ('engine revving') search
|
|
432
|
+
audio vectors."""
|
|
433
|
+
caps = detect_embedding_capabilities()
|
|
434
|
+
if not caps["audio"]["available"]:
|
|
435
|
+
return {
|
|
436
|
+
"success": False,
|
|
437
|
+
"error": "No audio-embedding backend available",
|
|
438
|
+
"install_guidance": caps["install_guidance"].get("audio"),
|
|
439
|
+
}
|
|
440
|
+
try:
|
|
441
|
+
backend, state = _clap_state()
|
|
442
|
+
if backend == "transformers_clap":
|
|
443
|
+
import torch
|
|
444
|
+
|
|
445
|
+
model, processor = state
|
|
446
|
+
inputs = processor(text=[str(text)], return_tensors="pt", padding=True)
|
|
447
|
+
with torch.no_grad():
|
|
448
|
+
features = model.get_text_features(**inputs)
|
|
449
|
+
features = features / features.norm(dim=-1, keepdim=True)
|
|
450
|
+
return {"success": True, "vector": [float(x) for x in features[0].tolist()],
|
|
451
|
+
"model": f"transformers_clap:{CLAP_HF_MODEL}"}
|
|
452
|
+
vectors = state.get_text_embedding([str(text), ""], use_tensor=False)
|
|
453
|
+
return {"success": True, "vector": [float(x) for x in vectors[0]], "model": "laion_clap:default"}
|
|
454
|
+
except Exception as exc: # noqa: BLE001
|
|
455
|
+
return {"success": False, "error": f"{type(exc).__name__}: {exc}"}
|
|
456
|
+
|
|
457
|
+
|
|
282
458
|
# ── content builders ─────────────────────────────────────────────────────────
|
|
283
459
|
|
|
284
460
|
|
|
@@ -497,6 +673,78 @@ def build_embeddings(
|
|
|
497
673
|
else:
|
|
498
674
|
result["visual"] = {"success": True, "embedded": 0, "note": "all up to date"}
|
|
499
675
|
|
|
676
|
+
if "audio" in kinds:
|
|
677
|
+
caps = detect_embedding_capabilities()
|
|
678
|
+
if not caps["audio"]["available"]:
|
|
679
|
+
result["audio"] = {
|
|
680
|
+
"success": False,
|
|
681
|
+
"skipped": True,
|
|
682
|
+
"error": "No audio-embedding backend available",
|
|
683
|
+
"install_guidance": caps["install_guidance"].get("audio"),
|
|
684
|
+
}
|
|
685
|
+
result["success"] = False
|
|
686
|
+
return result
|
|
687
|
+
existing = _existing_rows(conn, "audio")
|
|
688
|
+
embedded_count = 0
|
|
689
|
+
missing_media: List[str] = []
|
|
690
|
+
audio_model: Optional[str] = None
|
|
691
|
+
for clip in conn.execute(f"SELECT * FROM clips{where}", args).fetchall():
|
|
692
|
+
clip = dict(clip)
|
|
693
|
+
file_path = str(clip.get("file_path") or "")
|
|
694
|
+
if not file_path or not os.path.isfile(file_path):
|
|
695
|
+
missing_media.append(str(clip.get("clip_name") or clip["clip_uuid"]))
|
|
696
|
+
continue
|
|
697
|
+
shots = [dict(s) for s in conn.execute(
|
|
698
|
+
"SELECT shot_uuid, time_seconds_start, time_seconds_end FROM shots "
|
|
699
|
+
"WHERE clip_uuid = ? ORDER BY shot_index",
|
|
700
|
+
(clip["clip_uuid"],),
|
|
701
|
+
).fetchall()]
|
|
702
|
+
windows: List[Tuple[float, float]] = []
|
|
703
|
+
window_meta: List[Tuple[str, str, str]] = []
|
|
704
|
+
for shot in shots:
|
|
705
|
+
start, end = shot.get("time_seconds_start"), shot.get("time_seconds_end")
|
|
706
|
+
if start is None or end is None or float(end) - float(start) < 0.2:
|
|
707
|
+
continue
|
|
708
|
+
h = _content_hash(f"{file_path}|{start}|{end}")
|
|
709
|
+
if existing.get(("shot", str(shot["shot_uuid"]))) == h:
|
|
710
|
+
continue
|
|
711
|
+
windows.append((float(start), float(end)))
|
|
712
|
+
window_meta.append(("shot", str(shot["shot_uuid"]), h))
|
|
713
|
+
if not windows:
|
|
714
|
+
continue
|
|
715
|
+
embedded = embed_audio_windows(file_path, windows)
|
|
716
|
+
if not embedded.get("success"):
|
|
717
|
+
result["audio"] = embedded
|
|
718
|
+
result["success"] = False
|
|
719
|
+
return result
|
|
720
|
+
if not embedded.get("model"):
|
|
721
|
+
continue # no decodable audio in this clip (e.g. video-only)
|
|
722
|
+
audio_model = str(embedded["model"])
|
|
723
|
+
items: List[Tuple[str, str, str, List[float]]] = []
|
|
724
|
+
clip_vectors: List[List[float]] = []
|
|
725
|
+
for m, vector in zip(window_meta, embedded["vectors"]):
|
|
726
|
+
if vector is None:
|
|
727
|
+
continue
|
|
728
|
+
items.append((m[0], m[1], m[2], vector))
|
|
729
|
+
clip_vectors.append(vector)
|
|
730
|
+
# Clip-level audio vector = mean of its shot windows' (the
|
|
731
|
+
# per-shot-visual pattern).
|
|
732
|
+
if clip_vectors:
|
|
733
|
+
dim = len(clip_vectors[0])
|
|
734
|
+
mean = [sum(v[i] for v in clip_vectors) / len(clip_vectors) for i in range(dim)]
|
|
735
|
+
items.append((
|
|
736
|
+
"clip", str(clip["clip_uuid"]),
|
|
737
|
+
_content_hash(f"{file_path}|clip|{len(clip_vectors)}"), mean,
|
|
738
|
+
))
|
|
739
|
+
embedded_count += _store_vectors(project_root, "audio", audio_model, items)
|
|
740
|
+
result["audio"] = {
|
|
741
|
+
"success": True,
|
|
742
|
+
"embedded": embedded_count,
|
|
743
|
+
"model": audio_model,
|
|
744
|
+
**({"skipped_missing_media": missing_media} if missing_media else {}),
|
|
745
|
+
**({"note": "all up to date"} if not embedded_count and not missing_media else {}),
|
|
746
|
+
}
|
|
747
|
+
|
|
500
748
|
result["wall_clock_ms"] = int((time.time() - started) * 1000)
|
|
501
749
|
counts = conn.execute(
|
|
502
750
|
"SELECT embedding_kind, COUNT(*) AS n FROM embeddings GROUP BY embedding_kind"
|
|
@@ -569,8 +817,8 @@ def find_similar(
|
|
|
569
817
|
|
|
570
818
|
conn = timeline_brain_db.connect(project_root)
|
|
571
819
|
kind = (kind or "text").strip().lower()
|
|
572
|
-
if kind not in ("text", "visual"):
|
|
573
|
-
return {"success": False, "error": f"kind must be 'text' or '
|
|
820
|
+
if kind not in ("text", "visual", "audio"):
|
|
821
|
+
return {"success": False, "error": f"kind must be 'text', 'visual', or 'audio', got {kind!r}"}
|
|
574
822
|
|
|
575
823
|
exclude: Optional[Tuple[str, str]] = None
|
|
576
824
|
query_vector: Optional[List[float]] = None
|
|
@@ -583,6 +831,12 @@ def find_similar(
|
|
|
583
831
|
return embedded
|
|
584
832
|
query_vector = embedded["vectors"][0]
|
|
585
833
|
query_model = str(embedded["model"])
|
|
834
|
+
elif kind == "audio":
|
|
835
|
+
encoded = embed_text_for_audio_query(str(text))
|
|
836
|
+
if not encoded.get("success"):
|
|
837
|
+
return encoded
|
|
838
|
+
query_vector = encoded["vector"]
|
|
839
|
+
query_model = str(encoded["model"])
|
|
586
840
|
else:
|
|
587
841
|
encoded = embed_text_for_visual_query(str(text))
|
|
588
842
|
if not encoded.get("success"):
|
|
@@ -1458,6 +1458,20 @@ TOOL_INSTALL: Dict[str, Dict[str, Any]] = {
|
|
|
1458
1458
|
"verify": "python -c \"import open_clip\"",
|
|
1459
1459
|
"notes": "Needs torch. Model weights (~350 MB) download on first use.",
|
|
1460
1460
|
},
|
|
1461
|
+
"clap_audio": {
|
|
1462
|
+
"label": "CLAP (audio embeddings)",
|
|
1463
|
+
"bundle": "embeddings",
|
|
1464
|
+
"required_for": ["audio similarity (find_similar kind=audio)"],
|
|
1465
|
+
"commands": {
|
|
1466
|
+
"all": "pip install transformers",
|
|
1467
|
+
},
|
|
1468
|
+
"verify": "python -c \"import transformers\"",
|
|
1469
|
+
"notes": (
|
|
1470
|
+
"Needs torch + ffmpeg. Uses laion/clap-htsat-unfused (~600 MB, "
|
|
1471
|
+
"downloads on first use); the laion_clap package works as an "
|
|
1472
|
+
"alternative backend."
|
|
1473
|
+
),
|
|
1474
|
+
},
|
|
1461
1475
|
"whisper_cpp": {
|
|
1462
1476
|
"label": "whisper.cpp",
|
|
1463
1477
|
"bundle": "transcription",
|
|
@@ -1650,6 +1664,11 @@ def detect_capabilities(env: Optional[Dict[str, str]] = None) -> Dict[str, Any]:
|
|
|
1650
1664
|
bool(embedding_caps.get("visual", {}).get("available")),
|
|
1651
1665
|
{"python_module": "open_clip"},
|
|
1652
1666
|
),
|
|
1667
|
+
"clap_audio": _tool_entry(
|
|
1668
|
+
"clap_audio",
|
|
1669
|
+
bool(embedding_caps.get("audio", {}).get("available")),
|
|
1670
|
+
{"backends": embedding_caps.get("audio", {}).get("backends", [])},
|
|
1671
|
+
),
|
|
1653
1672
|
},
|
|
1654
1673
|
"embeddings": embedding_caps,
|
|
1655
1674
|
"transcription": {
|
|
@@ -5994,20 +6013,68 @@ def load_report(project_root: str, report_path: Optional[str] = None, clip_dir:
|
|
|
5994
6013
|
return {"success": True, "path": path, "report": payload}
|
|
5995
6014
|
|
|
5996
6015
|
|
|
5997
|
-
def
|
|
5998
|
-
|
|
6016
|
+
def _collect_reports_for_summary(root: str) -> Tuple[List[Dict[str, Any]], List[str], str]:
|
|
6017
|
+
"""(reports, report_paths, source) for summarize_reports.
|
|
6018
|
+
|
|
6019
|
+
DB-first: when every report dir on disk is covered by an ingested clip
|
|
6020
|
+
row, reports come from the DB-canonical store (blob + human overlay —
|
|
6021
|
+
identical content to the lockstep JSON export). Pre-v9 roots and MIXED
|
|
6022
|
+
roots (some clips not ingested) fall back WHOLESALE to the JSON walk —
|
|
6023
|
+
a partial DB view would silently under-report.
|
|
6024
|
+
"""
|
|
5999
6025
|
clips_root = os.path.join(root, "clips")
|
|
6000
|
-
|
|
6001
|
-
report_paths: List[str] = []
|
|
6026
|
+
disk_paths: List[str] = []
|
|
6002
6027
|
if os.path.isdir(clips_root):
|
|
6003
6028
|
for dirpath, _, filenames in os.walk(clips_root):
|
|
6004
6029
|
if "analysis.json" in filenames:
|
|
6005
|
-
|
|
6030
|
+
disk_paths.append(os.path.join(dirpath, "analysis.json"))
|
|
6031
|
+
disk_paths.sort()
|
|
6032
|
+
|
|
6033
|
+
try:
|
|
6034
|
+
from src.utils import analysis_store, timeline_brain_db
|
|
6035
|
+
|
|
6036
|
+
conn = timeline_brain_db.connect(root)
|
|
6037
|
+
db_dirs = {
|
|
6038
|
+
str(r["clip_dir"]): str(r["clip_uuid"])
|
|
6039
|
+
for r in conn.execute(
|
|
6040
|
+
"SELECT clip_dir, clip_uuid FROM clips WHERE clip_dir IS NOT NULL"
|
|
6041
|
+
).fetchall()
|
|
6042
|
+
}
|
|
6043
|
+
except Exception: # noqa: BLE001 — no DB (pre-v9) → JSON
|
|
6044
|
+
db_dirs = {}
|
|
6045
|
+
if disk_paths and db_dirs:
|
|
6046
|
+
dir_names = [os.path.basename(os.path.dirname(p)) for p in disk_paths]
|
|
6047
|
+
if all(name in db_dirs for name in dir_names):
|
|
6048
|
+
from src.utils import analysis_store
|
|
6049
|
+
|
|
6050
|
+
reports: List[Dict[str, Any]] = []
|
|
6051
|
+
complete = True
|
|
6052
|
+
for path, name in zip(disk_paths, dir_names):
|
|
6006
6053
|
try:
|
|
6007
|
-
|
|
6008
|
-
|
|
6009
|
-
|
|
6010
|
-
|
|
6054
|
+
report = analysis_store.export_report(root, db_dirs[name])
|
|
6055
|
+
except Exception: # noqa: BLE001
|
|
6056
|
+
report = None
|
|
6057
|
+
if not isinstance(report, dict):
|
|
6058
|
+
complete = False
|
|
6059
|
+
break
|
|
6060
|
+
reports.append(report)
|
|
6061
|
+
if complete:
|
|
6062
|
+
return reports, disk_paths, "db"
|
|
6063
|
+
|
|
6064
|
+
reports = []
|
|
6065
|
+
report_paths: List[str] = []
|
|
6066
|
+
for path in disk_paths:
|
|
6067
|
+
try:
|
|
6068
|
+
reports.append(_read_json(path))
|
|
6069
|
+
report_paths.append(path)
|
|
6070
|
+
except (OSError, json.JSONDecodeError):
|
|
6071
|
+
continue
|
|
6072
|
+
return reports, report_paths, "json"
|
|
6073
|
+
|
|
6074
|
+
|
|
6075
|
+
def summarize_reports(project_root: str) -> Dict[str, Any]:
|
|
6076
|
+
root = normalize_path(project_root)
|
|
6077
|
+
reports, report_paths, reports_source = _collect_reports_for_summary(root)
|
|
6011
6078
|
warnings = []
|
|
6012
6079
|
motion_counts: Dict[str, int] = {}
|
|
6013
6080
|
tags: Dict[str, int] = {}
|
|
@@ -6047,6 +6114,7 @@ def summarize_reports(project_root: str) -> Dict[str, Any]:
|
|
|
6047
6114
|
summary = {
|
|
6048
6115
|
"success": True,
|
|
6049
6116
|
"project_root": root,
|
|
6117
|
+
"source": reports_source, # "db" (canonical store) | "json" (walk fallback)
|
|
6050
6118
|
"clip_reports": len(reports),
|
|
6051
6119
|
"motion_distribution": motion_counts,
|
|
6052
6120
|
"technical_warning_count": len(warnings),
|
|
@@ -7202,9 +7270,42 @@ def build_analysis_index(project_root: str, *, index_path: Optional[Any] = None)
|
|
|
7202
7270
|
conn.execute("INSERT INTO index_metadata (key, value) VALUES (?, ?)", ("fts_enabled", "1" if fts_enabled else "0"))
|
|
7203
7271
|
conn.execute("INSERT INTO index_metadata (key, value) VALUES (?, ?)", ("image_blob_policy", "excluded"))
|
|
7204
7272
|
|
|
7273
|
+
# DB-first sourcing: local reports whose clip dir is ingested come from
|
|
7274
|
+
# the DB-canonical store (blob + human overlay — identical content to
|
|
7275
|
+
# the lockstep JSON export) instead of re-parsing every analysis.json.
|
|
7276
|
+
# Pre-v9 dirs and job-linked EXTERNAL report paths (their rows live
|
|
7277
|
+
# under another project's DB) keep the JSON read. The index schema and
|
|
7278
|
+
# the query surface are unchanged either way.
|
|
7279
|
+
clips_root_prefix = os.path.realpath(os.path.join(root, "clips")) + os.sep
|
|
7280
|
+
try:
|
|
7281
|
+
from src.utils import timeline_brain_db as _brain_db
|
|
7282
|
+
|
|
7283
|
+
_db_dirs = {
|
|
7284
|
+
str(r["clip_dir"]): str(r["clip_uuid"])
|
|
7285
|
+
for r in _brain_db.connect(root).execute(
|
|
7286
|
+
"SELECT clip_dir, clip_uuid FROM clips WHERE clip_dir IS NOT NULL"
|
|
7287
|
+
).fetchall()
|
|
7288
|
+
}
|
|
7289
|
+
except Exception: # noqa: BLE001 — no DB (pre-v9) → JSON for everything
|
|
7290
|
+
_db_dirs = {}
|
|
7291
|
+
report_sources = {"db": 0, "json": 0}
|
|
7205
7292
|
for report_path in sorted(_iter_analysis_report_files(root)):
|
|
7206
7293
|
try:
|
|
7207
|
-
report =
|
|
7294
|
+
report = None
|
|
7295
|
+
if _db_dirs and os.path.realpath(report_path).startswith(clips_root_prefix):
|
|
7296
|
+
clip_uuid = _db_dirs.get(os.path.basename(os.path.dirname(report_path)))
|
|
7297
|
+
if clip_uuid:
|
|
7298
|
+
try:
|
|
7299
|
+
from src.utils import analysis_store as _analysis_store
|
|
7300
|
+
|
|
7301
|
+
report = _analysis_store.export_report(root, clip_uuid)
|
|
7302
|
+
except Exception: # noqa: BLE001 — fall back per-report
|
|
7303
|
+
report = None
|
|
7304
|
+
if isinstance(report, dict):
|
|
7305
|
+
report_sources["db"] += 1
|
|
7306
|
+
else:
|
|
7307
|
+
report = _read_json(report_path)
|
|
7308
|
+
report_sources["json"] += 1
|
|
7208
7309
|
row_counts = _insert_analysis_report_into_index(conn, report_path, report, fts_enabled=fts_enabled)
|
|
7209
7310
|
counts["clips"] += 1
|
|
7210
7311
|
for key, value in row_counts.items():
|
|
@@ -7244,6 +7345,7 @@ def build_analysis_index(project_root: str, *, index_path: Optional[Any] = None)
|
|
|
7244
7345
|
"image_blob_policy": "excluded",
|
|
7245
7346
|
"fts_enabled": bool(counts["clips"]) and _sqlite_table_exists(db_path, "clips_fts"),
|
|
7246
7347
|
"counts": counts,
|
|
7348
|
+
"report_sources": report_sources, # how many reports came from the DB vs JSON
|
|
7247
7349
|
"failed_report_count": len(failed_reports),
|
|
7248
7350
|
"failed_reports": failed_reports[:50],
|
|
7249
7351
|
"size_bytes": os.path.getsize(db_path) if os.path.isfile(db_path) else 0,
|