@autoaiai/aicut 0.1.11 → 0.1.13

package/skills/videodb/reference/indexing.md

@@ -1,537 +0,0 @@
-# Understanding & Indexing Guide (v2)
-
-Turn a video into structured, retrievable data in three stages: **understand → index → retrieve**.
-
-For code-level details (signatures, class properties, validation errors), see [indexing-reference.md](indexing-reference.md). For retrieval, see [search.md](search.md). For the v1 scene-index API, see [legacy/index.md](legacy/index.md) and [migration.md](migration.md).
-
-Requires `videodb>=0.5.0`.
-
-Runnable notebooks covering every analyzer, segmentation and sampling, multi-analyzer pipelines, and custom indexes live in the [indexing-v2 cookbook](https://github.com/video-db/videodb-cookbook/tree/preview/guides/indexing-v2).
-
-## Contents
-
-- The three stages
-- Minimal end-to-end
-- Stage 1: Understanding (analyzers, segmentation, sampling, VLM config, structured output, chaining)
-- Stage 2: Indexing (source forms, use_for, fields, build status)
-- Index names across a collection
-- Inspecting an index
-- Managing runs and indexes
-- Scope: video vs collection
-- Cost and latency (frame count, interval, model tier)
-- Gotchas
-
-## The three stages
-
-| Stage | Call | Produces | Async? |
-|-------|------|----------|--------|
-| 1. Understand | `video.understand(analyzers=[...])` | `Understanding` holding one **artifact** per analyzer | yes — `wait_until_complete()` |
-| 2. Index | `video.index(source=analyzer)` | `Index` (retrievable) | validation sync, embedding async |
-| 3. Retrieve | `search` / `ask` / `semantic_search` / `query` / `aggregate` | shots, answers, rows | sync |
-
-An **analyzer** extracts one kind of signal. Its output is an **artifact**. Indexing an artifact declares what you can do with it later. Understanding and indexing are separate, so one artifact can back several indexes with different capabilities.
-
----
-
-## Minimal end-to-end
-
-```python
-import time
-
-import videodb
-
-conn = videodb.connect()
-coll = conn.get_collection()
-video = coll.get_video("your-video-id")
-
-understanding = video.understand(
-    analyzers=[
-        {"type": "spoken_words", "name": "transcript"},
-        {"type": "vlm", "name": "scene",
-         "config": {"prompt": "Describe the scene and any on-screen text."}},
-    ],
-    segmentation={"type": "shot", "threshold": 30},
-)
-
-# A `partial` run is not terminal to the SDK — poll the analyzers, not the run.
-# `analyzers and` is load-bearing: a refresh can transiently return an empty
-# list, and all([]) is True, which would exit while the run is still going.
-deadline = time.time() + 3600
-while time.time() < deadline:
-    analyzers = understanding.refresh().list_analyzers()
-    if analyzers and all(a.is_complete for a in analyzers):
-        break
-    time.sleep(15)
-
-for analyzer in understanding.list_analyzers():
-    if not analyzer.is_successful:
-        continue
-    video.index(source=analyzer, name=analyzer.name).wait_until_complete()
-
-response = video.search("discussion about pricing")
-for shot in response.shots:
-    print(f"[{shot.start:.1f}s - {shot.end:.1f}s] {shot.text}")
-if response.response_type in ("shots", "deepsearch") and len(response):
-    print(response.compile())
-```
-
-**Name your analyzers.** Index names default correctly either way — omit `name` on `index()` and the server derives `transcript`, `scene`, `objects` from the artifact type. The trap is passing `name=analyzer.name` for an analyzer you never named: the server assigns *analyzer* names like `vlm-3f2a91bc`, so that call creates an index called `vlm-3f2a91bc`, breaking `index_names=["scene"]` downstream. Naming the analyzer is also what makes `understanding.get_analyzer("scene")` resolve; without it that call raises `ValueError`.
-
-So: either name every analyzer (as above), or call `video.index(source=analyzer)` with no `name=` and take the derived default. Do not mix the two.
-
----
-
-## Stage 1: Understanding
-
-### Analyzers
-
-Each analyzer produces one artifact. Give every analyzer an explicit `name` — the conventional choice is the artifact name for its type:
-
-| `type` | Artifact name | Fields it emits |
-|--------|---------------|-----------------|
-| `spoken_words` | `transcript` | `text`, `chunks`, `words`, `language`, `speaker` |
-| `vlm` | `scene` | `scene_description`, `action`, `activity`, `location`, `setting`, `shot_type`, `emotion`, `topic`, `song_detection`, `character_description`, `object_description`, `brand_names`, `outputs`, `full_text` |
-| `object_detection` | `objects` | `summary`, `frames`, `detections`, `objects` |
-| `ocr` | `ocr` | `combined_text`, `text`, `words`, `language` |
-| `brand_detection` | `brands` | `brand_names`, `brands`, `summary`, `detections` |
-| `activity_recognition` | `activity` | `labels`, `activity`, `actions`, `detections` |
-| `location_detection` | `location` | `location_type`, `setting`, `time_of_day`, `location` |
-| `faces` | `faces` | `identities`, `detections`, `faces` |
-| `audio_event_detection` | `audio_events` | `events`, `labels`, `audio_events` |
-
-Those field names are what you reference in `fields` at index time and in `filter` at search time.
-
-Analyzer types are plain strings — there is no SDK enum. Do not write `from videodb import AnalyzerType`.
-
-These artifact names are what an index defaults to when you call `video.index(source=analyzer)` without a `name`. They are **not** what `analyzer.name` returns: an analyzer you didn't name gets a generated `{type}-{random}` value like `vlm-3f2a91bc`, which is also why `get_analyzer("scene")` would raise `ValueError`. Naming each analyzer keeps the two aligned.
-
-### Segmentation
-
-Run-level. Controls how the video is cut into timestamped scenes that analyzers operate on.
-
-```python
-video.understand(
-    segmentation={"type": "time", "seconds": 5},    # fixed-length scenes
-    analyzers=[{"type": "vlm"}],
-)
-
-video.understand(
-    # cut-based scenes; min_scene_len keeps very short cuts from becoming their own scene
-    segmentation={"type": "shot", "threshold": 30, "min_scene_len": 15},
-    analyzers=[{"type": "vlm"}],
-)
-```
-
-| Type | Keys | Use when |
-|------|------|----------|
-| `time` | `seconds` | Continuous or static footage, predictable cost, uniform sampling |
-| `shot` | `threshold`, optional `min_scene_len` | Edited content where cuts are meaningful boundaries |
-
-Segmentation is shared by every analyzer in the run.
-
-> **Two spellings exist for the time window.** `seconds` (an integer) is what the official cookbook uses for the video pipeline; `window` (a string with a unit, e.g. `"5s"`) appears in the RTStream API and in parts of the server. If `seconds` appears to be ignored, try `{"type": "time", "window": "5s"}`.
-
-### Sampling
-
-Per-analyzer. Controls which frames within each scene reach the model.
-
-```python
-{"type": "vlm", "sampling": {"strategy": "uniform", "frame_count": 8}}   # N frames per scene
-{"type": "object_detection", "sampling": {"strategy": "interval", "every": 1}}  # one frame per second
-```
-
-### Transform
-
-Run-level preprocessing. Lower resolution costs less and runs faster.
-
-```python
-video.understand(transform={"resolution": "480p"}, analyzers=[...])
-```
-
-### VLM configuration
-
-```python
-{
-    "type": "vlm",
-    "name": "scene",
-    "config": {
-        "model": "pro",
-        "prompt": "Describe the scene, the setting, and any visible text.",
-    },
-}
-```
-
-Managed model aliases, cheapest to most capable:
-
-| Alias | Use when |
-|-------|----------|
-| `mini` | Lowest-cost lightweight analysis |
-| `basic` | Routine extraction and descriptions |
-| `pro` | Balanced default |
-| `ultra` | Highest-quality complex reasoning |
-
-You can also name a provider model directly, e.g. `"google/gemini-2.5-flash"` or a supported `openai/...` model. Managed models do not take `sandbox_id`; self-hosted ones do.
-
-`config` is forwarded to the server untyped, so a misspelled tier fails server-side rather than at the call.
-
-### Structured output
-
-Give the VLM a `schema` to get typed fields instead of prose. This is what makes `query()` and `aggregate()` useful downstream — you can only filter and count on real fields.
-
-Simple form — a type name per field:
-
-```python
-"schema": {
-    "scene_description": "text",       # prose, embedded for semantic search
-    "activity": "string",              # short label, good for filter/aggregate
-    "visible_objects": ["string"],     # array shorthand
-    "confidence": {"type": "number", "min": 0, "max": 1},
-}
-```
-
-Richer form — enums, nested objects, and arrays of objects. Enums are worth the effort: a constrained vocabulary makes `aggregate(group_by=...)` produce clean buckets instead of free-text noise.
-
-```python
-"schema": {
-    "scene_description": "text",
-    "activity": {
-        "type": "enum",
-        "values": ["conversation", "using_device", "walking", "object_interaction", "other"],
-    },
-    "setting": {
-        "type": "object",
-        "fields": {
-            "location_type": {"type": "enum", "values": ["office", "home", "outdoor", "vehicle"]},
-            "environment": {"type": "enum", "values": ["indoor", "outdoor", "unknown"]},
-        },
-    },
-    "object_descriptions": {
-        "type": "array",
-        "min_items": 0,
-        "max_items": 4,
-        "items": {"name": "string", "description": "string"},
-    },
-}
-```
-
-A field can be made optional and self-describing: `{"type": "string", "required": False, "description": "Visible setting, if identifiable."}`.
-
-Two further `config` keys control how the schema is enforced:
-
-| Key | Values | Purpose |
-|-----|--------|---------|
-| `schema_mode` | `auto` (default), `native_required`, `prompt_only` | Whether to use the provider's native structured output, require it, or fall back to prompting |
-| `schema_max_retries` | int | Retries when the model returns output that doesn't satisfy the schema |
-
-Nested schema fields are addressed downstream with dotted paths — `setting.location_type`, `object_descriptions.description` — in `fields`, in `filter`, and in `index_names`.
-
-### Chaining analyzers
-
-An analyzer can read earlier artifacts through `inputs`, and interpolate them into its prompt with `{{inputs.<name>}}`. Order in the list does not matter; `inputs` declares the dependency.
-
-```python
-understanding = video.understand(
-    segmentation={"type": "shot", "threshold": 30},
-    analyzers=[
-        {"type": "object_detection", "name": "objects",
-         "sampling": {"strategy": "interval", "every": 1}},
-        {"type": "spoken_words", "name": "transcript"},
-        {
-            "type": "vlm",
-            "name": "scene",
-            "inputs": ["objects", "transcript"],
-            "sampling": {"strategy": "uniform", "frame_count": 8},
-            "config": {
-                "prompt": (
-                    "Describe the scene using the frames as primary evidence.\n\n"
-                    "Spoken words:\n{{inputs.transcript}}\n\n"
-                    "Detected objects:\n{{inputs.objects}}"
-                ),
-            },
-        },
-    ],
-)
-```
-
-### Waiting
-
-```python
-understanding.wait_until_complete(timeout=1800, poll_interval=10)  # raises TimeoutError
-```
-
-Or pass `callback_url` to `understand()` and skip polling entirely.
-
-> **A mixed run never satisfies `wait_until_complete()`.** Run statuses are `queued`, `running`, `done`, `partial`, `failed`. A run reports `done` only when *every* analyzer succeeded; if one fails or is skipped the run ends `partial` — and the SDK's terminal set is only `{done, failed}`, so polling continues until it raises `TimeoutError`. When some analyzers may legitimately fail, either use a short timeout and catch it, or poll the analyzers directly:
-
-```python
-import time
-
-deadline = time.time() + 1800
-while time.time() < deadline:
-    understanding.refresh()
-    if all(a.is_complete for a in understanding.list_analyzers()):
-        break
-    time.sleep(15)
-
-for analyzer in understanding.list_analyzers():
-    print(analyzer.name, analyzer.status)
-```
-
-### Reading analyzer output
-
-`get_output()` returns timestamped scenes. Each scene carries `start`, `end`, `scene_id`, and a `data` dict holding that analyzer's fields:
-
-```python
-output = understanding.get_analyzer("scene").get_output()
-
-# The payload is normally {"scenes": [...]} but can be a bare list — handle both.
-scenes = output.get("scenes", output) if isinstance(output, dict) else output
-
-for scene in scenes or []:
-    print(scene["start"], scene["end"], scene["data"])
-```
-
-A transcript scene's `data` has `text`, `language`, `words`; a schema'd VLM scene's `data` has exactly the fields you declared. You do not need to read the output at all to index it — indexing by reference is cheaper — but it is the fastest way to see what an analyzer actually produced before you decide which fields to index.
-
----
-
-## Stage 2: Indexing
-
-### Source forms
-
-```python
-# 1. Analyzer object — preferred. Indexed by reference; scene data never leaves the server.
-analyzer = understanding.get_analyzer("scene")
-video.index(source=analyzer, name="scene")
-
-# 2. That analyzer's output. Equivalent result, but the scenes round-trip through your process.
-video.index(source=analyzer.get_output(), name="scene")
-
-# 3. Reference dict, when you only have IDs.
-video.index(name="scene", source={"understanding_id": "und_123", "analyzer_id": "an_456"})
-
-# 4. Your own temporal records — no understanding run needed.
-video.index(name="chapters", source=[
-    {"start": 0.0, "end": 12.4, "summary": "Opening credits.", "chapter_type": "intro"},
-    {"start": 12.4, "end": 98.0, "summary": "Product walkthrough.", "chapter_type": "demo"},
-])
-```
-
-Forms 1 and 2 produce the same index. Prefer form 1 unless you already fetched the output for another reason.
-
-Custom records require `start` and `end`; `scene_id` and `metadata` are optional and any other keys are indexable. Use this to bring third-party analysis into VideoDB search.
-
-### Capabilities (`use_for`)
-
-```python
-video.index(source=analyzer, name="scene", use_for=["semantic", "query", "aggregate"])
-```
-
-| Capability | Unlocks |
-|-----------|---------|
-| `semantic` | `semantic_search()`, and `search()` over this index |
-| `query` | `query()` — exact filters |
-| `aggregate` | `aggregate()` — counts, grouping, facets |
-
-Omitting `use_for` defaults to all three, **and degrades gracefully**: if no scene holds embeddable top-level text, the index quietly drops `semantic` and keeps `query` + `aggregate`. Asking for `semantic` explicitly on the same artifact raises `use_for includes semantic but no scene has embeddable text` instead. Omitting is forgiving; requesting is strict.
-
-The trigger is the data, not the analyzer type. Artifacts whose text lives only in nested structures — object detection stores its labels under `frames.detections` — commonly degrade, but one that also emits a top-level `summary` will keep `semantic`. Check `index.use_for` after creation to see what you actually got.
-
-### Field configuration
-
-`fields` maps field groups to field names, and decides what is optimised for what. Unlisted fields are still stored and still returnable via `return_fields` — they just aren't searchable.
-
-```python
-scene = understanding.get_analyzer("scene")
-
-video.index(
-    source=scene,
-    name="scene",
-    use_for=["semantic", "query", "aggregate"],
-    fields={
-        "semantic": ["scene_description", "setting.location_type"],
-        "filter": ["activity", "setting.location_type", "setting.environment"],
-        "aggregate": ["activity", "setting.location_type"],
-    },
-)
-```
-
-Prose goes in `semantic`; enums and short labels go in `filter` and `aggregate`.
-
-| Group | Purpose |
-|-------|---------|
-| `semantic` | Vector search |
-| `filter` | `query()` conditions |
-| `aggregate` | Grouping and faceting |
-| `sort` | Result ordering |
-
-Use dotted paths for nested data. When a path crosses a list, values are collected from **every** element:
-
-```python
-objects = understanding.get_analyzer("objects")
-
-video.index(
-    source=objects,
-    name="objects",
-    use_for=["query", "aggregate"],
-    fields={
-        "filter": ["frames.detections.label", "frames.detections.score"],
-        "aggregate": ["frames.detections.label"],
-        "sort": ["frames.detections.score"],
-    },
-)
-```
-
-### Default field derivation
-
-Omitted groups are derived from your data. Well-known names get product defaults:
-
-| Field | Default groups |
-|-------|----------------|
-| `text` | `semantic`, `fts` |
-| `combined_text` | `semantic`, `fts` |
-| `scene_description` | `semantic` |
-| `language`, `brand_names` | `filter`, `aggregate` |
-| `activity`, `location` | `semantic`, `filter`, `aggregate` |
-| `frames.detections.label` | `filter`, `aggregate` |
-| `frames.detections.score` | `filter`, `sort` |
-
-Everything else is classified by shape: booleans → `filter`+`aggregate`; numbers → `filter`+`aggregate`+`sort`; strings → `filter`+`aggregate`, plus `semantic` when the value reads as prose; string arrays → `filter`+`aggregate`; nested objects and lists of objects get **no** group and are retrievable only through `return_fields`.
-
-Declaring a group wins verbatim. An explicit empty list opts out rather than triggering derivation:
-
-```python
-fields={"semantic": ["scene_description"], "filter": []}   # no filter fields at all
-```
-
-`fts` (full-text) is applied automatically to `text` and `combined_text`, so transcripts and OCR are keyword-searchable without asking.
-
-### Build status
-
-```python
-index.wait_until_complete(timeout=1800, poll_interval=10)
-
-if index.is_successful:
-    print(index.record_count, index.fields)
-else:
-    print(index.error)
-```
-
-| Status | Meaning |
-|--------|---------|
-| `building` | Ingest in progress, then embedding. Immediately after `index()` returns there may be **no rows yet**; once the ingest lands, `query()` and `aggregate()` work while embeddings finish |
-| `ready` | Fully operational, including semantic search |
-| `failed` | Build failed — read `index.error` |
-
-Do not treat `building` as "queryable" without checking. If you need results straight away, wait for `ready`.
-
-Field and source validation happens **synchronously**, on the `index()` call itself. A bad field path fails immediately and the error lists the field names that do exist.
-
----
-
-## Index names across a collection
-
-An index name is a schema contract within a collection. Reuse the same name across videos and you can retrieve over all of them at once:
-
-```python
-coll.semantic_search("a person holding a phone", index_names=["scene"])
-coll.query(index_name="objects",
-           filter=[{"field": "frames.detections.label", "op": "contains", "value": "car"}])
-coll.aggregate(index_name="brands", group_by="brand_names")
-```
-
-Indexes sharing a name must have identical field structures — a mismatch fails at create time.
-
-That cuts both ways. When you are iterating on a schema and do **not** want to fan out yet, give each run a unique name (`f"scene_{timestamp}"`) so a changed field layout doesn't collide with the previous attempt.
-
----
-
-## Inspecting an index
-
-```python
-index = video.get_index(name="scene")
-
-for field, schema in index.field_schema.items():
-    print(field, schema.type, schema.groups, schema.operators)
-
-page = index.records(limit=20)
-for record in page.records:
-    print(record.start, record.end, record.data)
-```
-
-Paginate with the cursor:
-
-```python
-cursor = None
-while True:
-    page = index.records(limit=100, cursor=cursor)
-    for record in page:
-        ...
-    cursor = page.next_cursor
-    if not cursor:
-        break
-```
-
-`field_schema[...].operators` tells you which filter operators a field actually supports — use it instead of guessing.
-
----
-
-## Managing runs and indexes
-
-```python
-video.list_understandings()
-video.get_understanding(understanding_id)
-video.delete_understanding(understanding_id)
-
-video.list_indexes()                      # all
-video.list_indexes(use_for="semantic")    # single string, not a list
-video.get_index(name="scene")
-video.get_index(index_id="idx_123")
-video.delete_index(index_id)
-index.delete()
-```
-
-Deleting an index removes its retrieval structures only — the video and the stored artifacts survive.
-
----
-
-## Scope: video vs collection
-
-Creation is **video-scoped**. `Collection` has no `understand()` and no `index()`; it is the fan-out read scope, exposing `search`, `ask`, `semantic_search`, `query`, and `aggregate` over every indexed video in it.
-
-For live streams, `RTStream` has its own `understand()` and `index()` — see [rtstream-reference.md](rtstream-reference.md).
-
----
-
-## Cost and latency
-
-Cost scales with the number of model calls, which is scenes × frames per scene.
-
-| Lever | Cheaper | More thorough |
-|-------|---------|---------------|
-| Segmentation interval | 30–60s scenes | 5s scenes |
-| Sampling | 1 frame per scene | 5–8 frames per scene |
-| Model tier | `mini` / `basic` | `pro` / `ultra` |
-| Transform | `{"resolution": "480p"}` | native resolution |
-
-A 60s interval at 1 frame costs roughly a thirtieth of a 5s interval at 5 frames over the same video. Start coarse, tighten only where retrieval quality demands it. For long videos, pass `callback_url` instead of blocking on `wait_until_complete()`.
-
-Prompt specificity matters more than frame count for retrieval quality. A prompt naming the dimensions you intend to search on ("count people, note clothing colour, note indoor/outdoor") beats "describe this scene" at the same cost.
-
----
-
-## Gotchas
-
-- **Terminal statuses differ.** `Understanding.is_successful` means `status == "done"`; `Index.is_successful` means `status == "ready"`. Use the properties rather than comparing strings.
-- **A run reports `done` only when every analyzer succeeded.** One failed or skipped analyzer makes the run `partial`, which the SDK does not treat as terminal — see the warning under Waiting. Gate on `analyzer.is_successful` before indexing each artifact.
-- **Never match on `analyzer.type`.** It echoes the server's internal name, not what you sent: a `spoken_words` analyzer reports `.type == "speech_transcription"`. Match on `analyzer.name`, which is yours.
-- **Guard the polling loop against an empty list.** `all([])` is `True`, so `all(a.is_complete for a in ...)` exits immediately if a refresh transiently returns no analyzers — and the next `index()` then fails with `Analyzer ... not done`. Check the list is non-empty first.
-- **Indexing is slower than you expect.** A 102-second clip with 5-second segmentation and three analyzers ran well past 30 minutes. `wait_until_complete()` defaults to a 1800-second timeout, which a modest video can exceed — prefer `callback_url` for anything long.
-- **`list_analyzers()` reads a local cache** and makes no network call. Call `understanding.refresh()` first if you didn't just wait on it.
-- **`get_analyzer()` raises `ValueError`** when the name doesn't match — it does not return `None`.
-- **`Understanding` has no `get_output()`.** Use `understanding.get_analyzer_output(name)` or `analyzer.get_output()`.
-- **The creator is `video.index()`.** There is no separate `create_index` method.
-- **`use_for` arity is asymmetric**: `index(use_for=["semantic", "query"])` takes a list; `list_indexes(use_for="semantic")` takes a single string.
-- **`Index` and friends are not top-level exports.** Use `from videodb.index import Index, IndexRecord, RecordPage, FieldSchema`. `IndexCapability`, `FieldGroup`, `Understanding`, and `UnderstandingAnalyzer` are importable from `videodb` directly.
-- **The server recognises more field groups than the SDK enum names.** `FieldGroup` lists four; `fts`, `hydrate`, and `return` also exist. Don't assert that `index.fields` only contains the four.
-- **Both stages are asynchronous but validation is not.** Malformed `fields` or `source` raise on the call; slow embedding shows up as a status.