@heyai-rules/pilo-masterkit 1.2.2 → 2.1.0

package/.agent/skills/videodb/reference/search.md

@@ -0,0 +1,230 @@
+# Search & Indexing Guide
+
+Search allows you to find specific moments inside videos using natural language queries, exact keywords, or visual scene descriptions.
+
+## Prerequisites
+
+Videos **must be indexed** before they can be searched. Indexing is a one-time operation per video per index type.
+
+## Indexing
+
+### Spoken Word Index
+
+Index the transcribed speech content of a video for semantic and keyword search:
+
+```python
+video = coll.get_video(video_id)
+
+# force=True makes indexing idempotent — skips if already indexed
+video.index_spoken_words(force=True)
+```
+
+This transcribes the audio track and builds a searchable index over the spoken content. Required for semantic search and keyword search.
+
+**Parameters:**
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `language_code` | `str\|None` | `None` | Language code of the video |
+| `segmentation_type` | `SegmentationType` | `SegmentationType.sentence` | Segmentation type (`sentence` or `llm`) |
+| `force` | `bool` | `False` | Set to `True` to skip if already indexed (avoids "already exists" error) |
+| `callback_url` | `str\|None` | `None` | Webhook URL for async notification |
+
+### Scene Index
+
+Index visual content by generating AI descriptions of scenes. Like spoken word indexing, this raises an error if a scene index already exists. Extract the existing `scene_index_id` from the error message.
+
+```python
+import re
+from videodb import SceneExtractionType
+
+try:
+    scene_index_id = video.index_scenes(
+        extraction_type=SceneExtractionType.shot_based,
+        prompt="Describe the visual content, objects, actions, and setting in this scene.",
+    )
+except Exception as e:
+    match = re.search(r"id\s+([a-f0-9]+)", str(e))
+    if match:
+        scene_index_id = match.group(1)
+    else:
+        raise
+```
+
+**Extraction types:**
+
+| Type | Description | Best For |
+|------|-------------|----------|
+| `SceneExtractionType.shot_based` | Splits on visual shot boundaries | General purpose, action content |
+| `SceneExtractionType.time_based` | Splits at fixed intervals | Uniform sampling, long static content |
+| `SceneExtractionType.transcript` | Splits based on transcript segments | Speech-driven scene boundaries |
+
+**Parameters for `time_based`:**
+
+```python
+video.index_scenes(
+    extraction_type=SceneExtractionType.time_based,
+    extraction_config={"time": 5, "select_frames": ["first", "last"]},
+    prompt="Describe what is happening in this scene.",
+)
+```
+
+## Search Types
+
+### Semantic Search
+
+Natural language queries matched against spoken content:
+
+```python
+from videodb import SearchType
+
+results = video.search(
+    query="explaining the benefits of machine learning",
+    search_type=SearchType.semantic,
+)
+```
+
+Returns ranked segments where the spoken content semantically matches the query.
+
+### Keyword Search
+
+Exact term matching in transcribed speech:
+
+```python
+results = video.search(
+    query="artificial intelligence",
+    search_type=SearchType.keyword,
+)
+```
+
+Returns segments containing the exact keyword or phrase.
+
+### Scene Search
+
+Visual content queries matched against indexed scene descriptions. Requires a prior `index_scenes()` call.
+
+`index_scenes()` returns a `scene_index_id`. Pass it to `video.search()` to target a specific scene index (especially important when a video has multiple scene indexes):
+
+```python
+from videodb import SearchType, IndexType
+from videodb.exceptions import InvalidRequestError
+
+# Search using semantic search against the scene index.
+# Use score_threshold to filter low-relevance noise (recommended: 0.3+).
+try:
+    results = video.search(
+        query="person writing on a whiteboard",
+        search_type=SearchType.semantic,
+        index_type=IndexType.scene,
+        scene_index_id=scene_index_id,
+        score_threshold=0.3,
+    )
+    shots = results.get_shots()
+except InvalidRequestError as e:
+    if "No results found" in str(e):
+        shots = []
+    else:
+        raise
+```
+
+**Important notes:**
+
+- Use `SearchType.semantic` with `index_type=IndexType.scene` — this is the most reliable combination and works on all plans.
+- `SearchType.scene` exists but may not be available on all plans (e.g. Free tier). Prefer `SearchType.semantic` with `IndexType.scene`.
+- The `scene_index_id` parameter is optional. If omitted, the search runs against all scene indexes on the video. Pass it to target a specific index.
+- You can create multiple scene indexes per video (with different prompts or extraction types) and search them independently using `scene_index_id`.
+
+### Scene Search with Metadata Filtering
+
+When indexing scenes with custom metadata, you can combine semantic search with metadata filters:
+
+```python
+from videodb import SearchType, IndexType
+
+results = video.search(
+    query="a skillful chasing scene",
+    search_type=SearchType.semantic,
+    index_type=IndexType.scene,
+    scene_index_id=scene_index_id,
+    filter=[{"camera_view": "road_ahead"}, {"action_type": "chasing"}],
+)
+```
+
+See the [scene_level_metadata_indexing cookbook](https://github.com/video-db/videodb-cookbook/blob/main/quickstart/scene_level_metadata_indexing.ipynb) for a full example of custom metadata indexing and filtered search.
+
+## Working with Results
+
+### Get Shots
+
+Access individual result segments:
+
+```python
+results = video.search("your query")
+
+for shot in results.get_shots():
+    print(f"Video: {shot.video_id}")
+    print(f"Start: {shot.start:.2f}s")
+    print(f"End: {shot.end:.2f}s")
+    print(f"Text: {shot.text}")
+    print("---")
+```
+
+### Play Compiled Results
+
+Stream all matching segments as a single compiled video:
+
+```python
+results = video.search("your query")
+stream_url = results.compile()
+results.play()  # opens compiled stream in browser
+```
+
+### Extract Clips
+
+Download or stream specific result segments:
+
+```python
+for shot in results.get_shots():
+    stream_url = shot.generate_stream()
+    print(f"Clip: {stream_url}")
+```
+
+## Cross-Collection Search
+
+Search across all videos in a collection:
+
+```python
+coll = conn.get_collection()
+
+# Search across all videos in the collection
+results = coll.search(
+    query="product demo",
+    search_type=SearchType.semantic,
+)
+
+for shot in results.get_shots():
+    print(f"Video: {shot.video_id} [{shot.start:.1f}s - {shot.end:.1f}s]")
+```
+
+> **Note:** Collection-level search only supports `SearchType.semantic`. Using `SearchType.keyword` or `SearchType.scene` with `coll.search()` will raise `NotImplementedError`. For keyword or scene search, use `video.search()` on individual videos instead.
+
+## Search + Compile
+
+Index, search, and compile matching segments into a single playable stream:
+
+```python
+video.index_spoken_words(force=True)
+results = video.search(query="your query", search_type=SearchType.semantic)
+stream_url = results.compile()
+print(stream_url)
+```
+
+## Tips
+
+- **Index once, search many times**: Indexing is the expensive operation. Once indexed, searches are fast.
+- **Combine index types**: Index both spoken words and scenes to enable all search types on the same video.
+- **Refine queries**: Semantic search works best with descriptive, natural language phrases rather than single keywords.
+- **Use keyword search for precision**: When you need exact term matches, keyword search avoids semantic drift.
+- **Handle "No results found"**: `video.search()` raises `InvalidRequestError` when no results match. Always wrap search calls in try/except and treat `"No results found"` as an empty result set.
+- **Filter scene search noise**: Semantic scene search can return low-relevance results for vague queries. Use `score_threshold=0.3` (or higher) to filter noise.
+- **Idempotent indexing**: Use `index_spoken_words(force=True)` to safely re-index. `index_scenes()` has no `force` parameter — wrap it in try/except and extract the existing `scene_index_id` from the error message with `re.search(r"id\s+([a-f0-9]+)", str(e))`.

package/.agent/skills/videodb/reference/streaming.md

@@ -0,0 +1,406 @@
+# Streaming & Playback
+
+VideoDB generates streams on-demand, returning HLS-compatible URLs that play instantly in any standard video player. No render times or export waits - edits, searches, and compositions stream immediately.
+
+## Prerequisites
+
+Videos **must be uploaded** to a collection before streams can be generated. For search-based streams, the video must also be **indexed** (spoken words and/or scenes). See [search.md](search.md) for indexing details.
+
+## Core Concepts
+
+### Stream Generation
+
+Every video, search result, and timeline in VideoDB can produce a **stream URL**. This URL points to an HLS (HTTP Live Streaming) manifest that is compiled on demand.
+
+```python
+# From a video
+stream_url = video.generate_stream()
+
+# From a timeline
+stream_url = timeline.generate_stream()
+
+# From search results
+stream_url = results.compile()
+```
+
+## Streaming a Single Video
+
+### Basic Playback
+
+```python
+import videodb
+
+conn = videodb.connect()
+coll = conn.get_collection()
+video = coll.get_video("your-video-id")
+
+# Generate stream URL
+stream_url = video.generate_stream()
+print(f"Stream: {stream_url}")
+
+# Open in default browser
+video.play()
+```
+
+### With Subtitles
+
+```python
+# Index and add subtitles first
+video.index_spoken_words(force=True)
+stream_url = video.add_subtitle()
+
+# Returned URL already includes subtitles
+print(f"Subtitled stream: {stream_url}")
+```
+
+### Specific Segments
+
+Stream only a portion of a video by passing a timeline of timestamp ranges:
+
+```python
+# Stream seconds 10-30 and 60-90
+stream_url = video.generate_stream(timeline=[(10, 30), (60, 90)])
+print(f"Segment stream: {stream_url}")
+```
+
+## Streaming Timeline Compositions
+
+Build a multi-asset composition and stream it in real time:
+
+```python
+import videodb
+from videodb.timeline import Timeline
+from videodb.asset import VideoAsset, AudioAsset, ImageAsset, TextAsset, TextStyle
+
+conn = videodb.connect()
+coll = conn.get_collection()
+
+video = coll.get_video(video_id)
+music = coll.get_audio(music_id)
+
+timeline = Timeline(conn)
+
+# Main video content
+timeline.add_inline(VideoAsset(asset_id=video.id))
+
+# Background music overlay (starts at second 0)
+timeline.add_overlay(0, AudioAsset(asset_id=music.id))
+
+# Text overlay at the beginning
+timeline.add_overlay(0, TextAsset(
+    text="Live Demo",
+    duration=3,
+    style=TextStyle(fontsize=48, fontcolor="white", boxcolor="#000000"),
+))
+
+# Generate the composed stream
+stream_url = timeline.generate_stream()
+print(f"Composed stream: {stream_url}")
+```
+
+**Important:** `add_inline()` only accepts `VideoAsset`. Use `add_overlay()` for `AudioAsset`, `ImageAsset`, and `TextAsset`.
+
+For detailed timeline editing, see [editor.md](editor.md).
+
+## Streaming Search Results
+
+Compile search results into a single stream of all matching segments:
+
+```python
+from videodb import SearchType
+from videodb.exceptions import InvalidRequestError
+
+video.index_spoken_words(force=True)
+try:
+    results = video.search("key announcement", search_type=SearchType.semantic)
+
+    # Compile all matching shots into one stream
+    stream_url = results.compile()
+    print(f"Search results stream: {stream_url}")
+
+    # Or play directly
+    results.play()
+except InvalidRequestError as exc:
+    if "No results found" in str(exc):
+        print("No matching announcement segments were found.")
+    else:
+        raise
+```
+
+### Stream Individual Search Hits
+
+```python
+from videodb.exceptions import InvalidRequestError
+
+try:
+    results = video.search("product demo", search_type=SearchType.semantic)
+    for i, shot in enumerate(results.get_shots()):
+        stream_url = shot.generate_stream()
+        print(f"Hit {i+1} [{shot.start:.1f}s-{shot.end:.1f}s]: {stream_url}")
+except InvalidRequestError as exc:
+    if "No results found" in str(exc):
+        print("No product demo segments matched the query.")
+    else:
+        raise
+```
+
+## Audio Playback
+
+Get a signed playback URL for audio content:
+
+```python
+audio = coll.get_audio(audio_id)
+playback_url = audio.generate_url()
+print(f"Audio URL: {playback_url}")
+```
+
+## Complete Workflow Examples
+
+### Search-to-Stream Pipeline
+
+Combine search, timeline composition, and streaming in one workflow:
+
+```python
+import videodb
+from videodb import SearchType
+from videodb.exceptions import InvalidRequestError
+from videodb.timeline import Timeline
+from videodb.asset import VideoAsset, TextAsset, TextStyle
+
+conn = videodb.connect()
+coll = conn.get_collection()
+video = coll.get_video("your-video-id")
+
+video.index_spoken_words(force=True)
+
+# Search for key moments
+queries = ["introduction", "main demo", "Q&A"]
+timeline = Timeline(conn)
+timeline_offset = 0.0
+
+for query in queries:
+    try:
+        results = video.search(query, search_type=SearchType.semantic)
+        shots = results.get_shots()
+    except InvalidRequestError as exc:
+        if "No results found" in str(exc):
+            shots = []
+        else:
+            raise
+
+    if not shots:
+        continue
+
+    # Add the section label where this batch starts in the compiled timeline
+    timeline.add_overlay(timeline_offset, TextAsset(
+        text=query.title(),
+        duration=2,
+        style=TextStyle(fontsize=36, fontcolor="white", boxcolor="#222222"),
+    ))
+
+    for shot in shots:
+        timeline.add_inline(
+            VideoAsset(asset_id=shot.video_id, start=shot.start, end=shot.end)
+        )
+        timeline_offset += shot.end - shot.start
+
+stream_url = timeline.generate_stream()
+print(f"Dynamic compilation: {stream_url}")
+```
+
+### Multi-Video Stream
+
+Combine clips from different videos into a single stream:
+
+```python
+import videodb
+from videodb.timeline import Timeline
+from videodb.asset import VideoAsset
+
+conn = videodb.connect()
+coll = conn.get_collection()
+
+video_clips = [
+    {"id": "vid_001", "start": 0, "end": 15},
+    {"id": "vid_002", "start": 10, "end": 30},
+    {"id": "vid_003", "start": 5, "end": 25},
+]
+
+timeline = Timeline(conn)
+for clip in video_clips:
+    timeline.add_inline(
+        VideoAsset(asset_id=clip["id"], start=clip["start"], end=clip["end"])
+    )
+
+stream_url = timeline.generate_stream()
+print(f"Multi-video stream: {stream_url}")
+```
+
+### Conditional Stream Assembly
+
+Build a stream dynamically based on search availability:
+
+```python
+import videodb
+from videodb import SearchType
+from videodb.exceptions import InvalidRequestError
+from videodb.timeline import Timeline
+from videodb.asset import VideoAsset, TextAsset, TextStyle
+
+conn = videodb.connect()
+coll = conn.get_collection()
+video = coll.get_video("your-video-id")
+
+video.index_spoken_words(force=True)
+
+timeline = Timeline(conn)
+
+# Try to find specific content; fall back to full video
+topics = ["opening remarks", "technical deep dive", "closing"]
+
+found_any = False
+timeline_offset = 0.0
+for topic in topics:
+    try:
+        results = video.search(topic, search_type=SearchType.semantic)
+        shots = results.get_shots()
+    except InvalidRequestError as exc:
+        if "No results found" in str(exc):
+            shots = []
+        else:
+            raise
+
+    if shots:
+        found_any = True
+        timeline.add_overlay(timeline_offset, TextAsset(
+            text=topic.title(),
+            duration=2,
+            style=TextStyle(fontsize=32, fontcolor="white", boxcolor="#1a1a2e"),
+        ))
+        for shot in shots:
+            timeline.add_inline(
+                VideoAsset(asset_id=shot.video_id, start=shot.start, end=shot.end)
+            )
+            timeline_offset += shot.end - shot.start
+
+if found_any:
+    stream_url = timeline.generate_stream()
+    print(f"Curated stream: {stream_url}")
+else:
+    # Fall back to full video stream
+    stream_url = video.generate_stream()
+    print(f"Full video stream: {stream_url}")
+```
+
+### Live Event Recap
+
+Process an event recording into a streamable recap with multiple sections:
+
+```python
+import videodb
+from videodb import SearchType
+from videodb.exceptions import InvalidRequestError
+from videodb.timeline import Timeline
+from videodb.asset import VideoAsset, AudioAsset, ImageAsset, TextAsset, TextStyle
+
+conn = videodb.connect()
+coll = conn.get_collection()
+
+# Upload event recording
+event = coll.upload(url="https://example.com/event-recording.mp4")
+event.index_spoken_words(force=True)
+
+# Generate background music
+music = coll.generate_music(
+    prompt="upbeat corporate background music",
+    duration=120,
+)
+
+# Generate title image
+title_img = coll.generate_image(
+    prompt="modern event recap title card, dark background, professional",
+    aspect_ratio="16:9",
+)
+
+# Build the recap timeline
+timeline = Timeline(conn)
+timeline_offset = 0.0
+
+# Main video segments from search
+try:
+    keynote = event.search("keynote announcement", search_type=SearchType.semantic)
+    keynote_shots = keynote.get_shots()[:5]
+except InvalidRequestError as exc:
+    if "No results found" in str(exc):
+        keynote_shots = []
+    else:
+        raise
+if keynote_shots:
+    keynote_start = timeline_offset
+    for shot in keynote_shots:
+        timeline.add_inline(
+            VideoAsset(asset_id=shot.video_id, start=shot.start, end=shot.end)
+        )
+        timeline_offset += shot.end - shot.start
+else:
+    keynote_start = None
+
+try:
+    demo = event.search("product demo", search_type=SearchType.semantic)
+    demo_shots = demo.get_shots()[:5]
+except InvalidRequestError as exc:
+    if "No results found" in str(exc):
+        demo_shots = []
+    else:
+        raise
+if demo_shots:
+    demo_start = timeline_offset
+    for shot in demo_shots:
+        timeline.add_inline(
+            VideoAsset(asset_id=shot.video_id, start=shot.start, end=shot.end)
+        )
+        timeline_offset += shot.end - shot.start
+else:
+    demo_start = None
+
+# Overlay title card image
+timeline.add_overlay(0, ImageAsset(
+    asset_id=title_img.id, width=100, height=100, x=80, y=20, duration=5
+))
+
+# Overlay section labels at the correct timeline offsets
+if keynote_start is not None:
+    timeline.add_overlay(max(5, keynote_start), TextAsset(
+        text="Keynote Highlights",
+        duration=3,
+        style=TextStyle(fontsize=40, fontcolor="white", boxcolor="#0d1117"),
+    ))
+if demo_start is not None:
+    timeline.add_overlay(max(5, demo_start), TextAsset(
+        text="Demo Highlights",
+        duration=3,
+        style=TextStyle(fontsize=36, fontcolor="white", boxcolor="#0d1117"),
+    ))
+
+# Overlay background music
+timeline.add_overlay(0, AudioAsset(
+    asset_id=music.id, fade_in_duration=3
+))
+
+# Stream the final recap
+stream_url = timeline.generate_stream()
+print(f"Event recap: {stream_url}")
+```
+
+---
+
+## Tips
+
+- **HLS compatibility**: Stream URLs return HLS manifests (`.m3u8`). They work in Safari natively, and in other browsers via hls.js or similar libraries.
+- **On-demand compilation**: Streams are compiled server-side when requested. The first play may have a brief compilation delay; subsequent plays of the same composition are cached.
+- **Caching**: Calling `video.generate_stream()` a second time without arguments returns the cached stream URL rather than recompiling.
+- **Segment streams**: `video.generate_stream(timeline=[(start, end)])` is the fastest way to stream a specific clip without building a full `Timeline` object.
+- **Inline vs overlay**: `add_inline()` only accepts `VideoAsset` and places assets sequentially on the main track. `add_overlay()` accepts `AudioAsset`, `ImageAsset`, and `TextAsset` and layers them on top at a given start time.
+- **TextStyle defaults**: `TextStyle` defaults to `font='Sans'`, `fontcolor='black'`. Use `boxcolor` (not `bgcolor`) for background color on text.
+- **Combine with generation**: Use `coll.generate_music(prompt, duration)` and `coll.generate_image(prompt, aspect_ratio)` to create assets for timeline compositions.
+- **Playback**: `.play()` opens the stream URL in the default system browser. For programmatic use, work with the URL string directly.