@mevdragon/vidfarm-devcli 0.20.12 → 0.20.13
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/skills/hyperframes-cli/SKILL.md +1 -1
- package/.agents/skills/vidfarm-director/SKILL.md +26 -0
- package/.agents/skills/vidfarm-director/references/assets-and-sourcing.md +1 -1
- package/.agents/skills/vidfarm-director/references/automation-and-local-dev.md +3 -1
- package/.agents/skills/vidfarm-director/references/core-workflows.md +2 -2
- package/SKILL.director.md +32 -4
- package/dist/src/cli.js +25 -33
- package/dist/src/devcli/clip-store.js +9 -3
- package/dist/src/devcli/clips.js +2 -3
- package/dist/src/services/clip-curation/ffmpeg.js +5 -1
- package/dist/src/services/clip-curation/refine.js +18 -10
- package/dist/src/services/clip-curation/scan.js +15 -16
- package/dist/src/services/provider-errors.js +51 -0
- package/package.json +6 -2
|
@@ -86,7 +86,7 @@ Add `inspect` for layout-sensitive work and `render --strict` in CI to fail on l
|
|
|
86
86
|
|
|
87
87
|
### Visual smoke test — required when the project uses sub-compositions
|
|
88
88
|
|
|
89
|
-
`lint` / `validate` / `inspect` evaluate each composition **in isolation**. They never load `index.html` and mount sub-compositions via `data-composition-src`, so they cannot catch cross-file mount failures
|
|
89
|
+
`lint` / `validate` / `inspect` evaluate each composition **in isolation**. They never load `index.html` and mount sub-compositions via `data-composition-src`, so they cannot catch cross-file mount failures. Load the `hyperframes-core` skill, then read its `references/sub-compositions.md` “Common pitfalls” section. The only gate that catches them is one that actually loads `index.html` and seeks the timeline.
|
|
90
90
|
|
|
91
91
|
Use `hyperframes snapshot` — it loads the project the same way `render` does (so it exercises the same mount path) but only captures the timestamps you request, so it's seconds instead of a full render:
|
|
92
92
|
|
|
@@ -50,11 +50,23 @@ Present both harnesses to the director, recommend (A) unless they've asked for p
|
|
|
50
50
|
|
|
51
51
|
## Web AI chat vs. local devcli — know your surface's limits
|
|
52
52
|
|
|
53
|
+
Choose the surface before choosing instructions:
|
|
54
|
+
|
|
55
|
+
| Surface | Live timeline | Shell / local files | Use it for |
|
|
56
|
+
|---|---:|---:|---|
|
|
57
|
+
| Web copilot | Yes, through `editor_action` | No | Short and medium edits on the open composition |
|
|
58
|
+
| Desktop coding agent | Through pulled files and devcli | Yes | Full rebuilds, scripts, batch work, and render/inspect loops |
|
|
59
|
+
| Devcli without an agent | Command-driven | Yes | Deterministic pull, edit, render, approve, and automation operations |
|
|
60
|
+
|
|
61
|
+
When this pack is installed locally, assume the desktop coding-agent surface unless trusted runtime context explicitly identifies the web editor. In web chat, declared tools and the latest `editor_context` override any capability described in this broad pack.
|
|
62
|
+
|
|
53
63
|
You may be running as the **in-web AI chat** (the /editor copilot, the chat dock, or the /chat and /library assistants) or as a **local coding agent** driving `vidfarm-devcli` (Claude Code / Codex on the user's machine). Same three paintbrushes, different reach — be honest about which surface you are and route heavy work accordingly:
|
|
54
64
|
|
|
65
|
+
- **Determine the surface before claiming capabilities.** The web chat has only its declared tools and REST routes. It cannot execute arbitrary JavaScript/Python, open a shell, create a local repository script, or use the user's filesystem. Never tell a web-chat user that you ran code or wrote a script unless a dedicated declared tool actually did so. A desktop coding agent has a real shell and filesystem and MAY write/run scripts, perform arbitrary local computations over paginated API results, create reports/CSVs/JSON, edit composition files, and orchestrate long devcli workflows within the user's authorization.
|
|
55
66
|
- **The web AI chat can do all three paintbrushes** — clip raws, author HTML/hyperframe motion, and generate AI media — and it drives edits directly on the live timeline. Keep small-to-medium jobs here: text/caption swaps, a scene or two replaced, single generations, captions, approve/schedule. Just do them.
|
|
56
67
|
- **Where the web chat struggles: complex, long, multi-step transformations.** A full multi-scene re-theme, an iterative render-critique-iterate loop, heavy scripted or batch work, or anything needing a real filesystem and many sequential tool calls will hit context limits, turn/timeout ceilings, and the web editor's constraints (CSS/declarative motion only — JS animation adapters are stripped on save). Don't grind a big transformation one layer at a time in a chat turn and stall.
|
|
57
68
|
- **Practical workaround — hand the heavy job to local devcli.** When a task is genuinely large or long-running, **proactively recommend the director run it locally with an AI coding agent** (Claude Code / OpenAI Codex / any capable agent): `vidfarm pull <forkId>` writes the composition + the `.harness/` grounding bundle to disk, the agent edits with the full devcli verb set and JS animation adapters, renders free with `vidfarm serve`, and `vidfarm publish` pushes it back. This is the **best-quality (B) harness's** natural home (adversarial grading with a coding agent). Frame it as "this is a big rebuild — you'll get a better, faster result running it locally with a coding agent; here's how," not as a dead end.
|
|
69
|
+
- **Offer a handoff, do not impersonate the desktop agent.** When web chat reaches that boundary, offer to save a Markdown handoff in My Files containing the objective, selected template/fork IDs, asset paths, grounding, constraints, completed work, and suggested devcli commands. Create it only after the user agrees. The desktop agent should read that document, pull the referenced fork, and then use its actual code/shell capabilities.
|
|
58
70
|
- **Never send the user away just to read knowledge.** Deeper skill knowledge is always a **tool call** away in-place: call `load_skill` (e.g. `load_skill('vidfarm-director', file='references/editor-workflows.md')`, or a craft pack like `editor-capabilities` / `hyperframes-animation`) to pull the exact reference you need mid-conversation. Only recommend switching surfaces for the WORK (a heavy transformation), never for the information.
|
|
59
71
|
|
|
60
72
|
## Read Only What You Need
|
|
@@ -68,6 +80,20 @@ Read only the relevant reference file for the current task.
|
|
|
68
80
|
- Getting-started interviews, strategy docs, onboarding flow: `references/onboarding.md`
|
|
69
81
|
- Primitive routes such as TTS, STT, music, overlays, background removal, product placement: `references/primitives.md`
|
|
70
82
|
|
|
83
|
+
## HyperFrames Skills — Load on Demand
|
|
84
|
+
|
|
85
|
+
Vidfarm ships a curated HyperFrames skill suite alongside this director pack. Use it for composition authoring and motion craft without loading the entire suite into context.
|
|
86
|
+
|
|
87
|
+
1. Route broad video-creation requests through `hyperframes` first. It selects the appropriate workflow skill.
|
|
88
|
+
2. Load only the selected workflow skill, then add the narrow domain skill required by the current step.
|
|
89
|
+
3. Load `hyperframes-core` before writing or restructuring composition HTML.
|
|
90
|
+
4. Load `hyperframes-animation`, `hyperframes-keyframes`, `hyperframes-creative`, or `hyperframes-cli` only when the task needs that specific capability.
|
|
91
|
+
5. Use `vidfarm-media` for narration, music, transcription, captions timing, background removal, and media sourcing. Prefer Vidfarm primitives and the user's existing library; when unavailable, use an equivalent capability already exposed by the user's desktop AI agent.
|
|
92
|
+
|
|
93
|
+
On the web copilot, call `load_skill('<name>')` and load referenced files only when the selected skill reaches that step. On desktop, inspect the agent's available-skill catalog and local `.agents/skills` / `.claude/skills` entries before declaring a skill missing. If needed, install only the selected bundled pack with `vidfarm skills add <name>`; do not bulk-install or bulk-read the suite. If the Vidfarm CLI is unavailable, use the desktop agent's native skill discovery or local-skill mechanism and continue with the closest installed equivalent.
|
|
94
|
+
|
|
95
|
+
HyperFrames authoring and rendering in this package are Vidfarm-native: local work uses the bundled composition toolchain and `vidfarm serve`; cloud work uses Vidfarm render routes. Do not require an external vendor account, repository, publish service, or telemetry endpoint. Keep `HYPERFRAMES_SKIP_SKILLS=1` and `HYPERFRAMES_NO_TELEMETRY=1` in Vidfarm-managed environments so the bundled skills stay pinned and local work does not phone home.
|
|
96
|
+
|
|
71
97
|
## Quick Router
|
|
72
98
|
|
|
73
99
|
Choose the narrowest path that satisfies the request.
|
|
@@ -27,7 +27,7 @@ Mine a **long-form** video (podcast, stream VOD, webinar, any YouTube/TikTok/IG/
|
|
|
27
27
|
|
|
28
28
|
- **Poll** `GET /raws/scan/:scanId` until the source `status` is `complete` (or `failed` — crashed pipelines are reconciled to `failed`, pollers never spin).
|
|
29
29
|
- **Results**: `GET /raws/feed?source=<source_video_id>` (or the whole library), hybrid search via `POST /raws/search` `{ query: "someone looks confused" }`, per-raw download at `GET /raws/:clipId/download`. Raws carry taxonomy tags, a description, a transcript, an `aspect` field when cropped, the hunt's `tracer`, and (with a gemini/openai key) a semantic embedding.
|
|
30
|
-
- **Billing** —
|
|
30
|
+
- **Billing** — scan compute (`clip_scan_lambda` GB-seconds + Step Functions transitions) and each persisted semantic embedding are wallet-metered; the 202 response includes a `compute_estimate`. AI tagging/refine runs on **your saved provider key (BYOK)** and is not wallet-billed. Submission is wallet-gated (402 when empty).
|
|
31
31
|
- **The original stays temporary** — URL-ingested and uploaded sources live ONLY in your temp folder (`users/…/temporary/clip-sources/…`) with a hard **30-day TTL** (auto-deleted from S3 + the temp-file list). The hunted raws themselves are durable.
|
|
32
32
|
- **Captions rule** — "no captions / no on-screen text" hunts are handled by scene SELECTION (the refine pass drops text-heavy scenes). **Never run GhostCut caption removal on a long-form source** (it is hard-capped at ~15 minutes); to actually erase burned-in text, apply `POST /api/v1/primitives/videos/remove-captions` to individual FINISHED raws afterwards.
|
|
33
33
|
- **Clip an EXACT subrange (no AI hunt)** — when you want one precise hand-picked in/out rather than an open-ended hunt, use the **Clipper**: the web page `https://vidfarm.cc/tools/clipper` (previews the source straight from its URL, HH:MM:SS:NNNN in/out, live length estimate, multi-clip session grouped by tracer), the endpoint `POST /raws/clip-range` `{ source_url|preview_url|temp_file_id, start_sec, end_sec, tracer?, folder_path?, name? }` (trims exactly that window into one raw — compute-only, no AI/key), or the devcli twin `vidfarm clipper <video-url|file> --start <t> --end <t> [--tracer NAME --folder NAME --name TXT]`. The devcli path is now **local-first**: local files clip into the local raws store by default, and URL sources are staged locally before trimming unless you pass `--cloud`. If you do **not** have a `VIDFARM_API_KEY`, URL clipping cannot bootstrap the remote source into the local backend; open the long-form raw video in the browser, save/extract the actual video file locally, then run `vidfarm clipper ./source.mp4 ...`. The Library "Import source" button opens the Clipper; each save lands in `/raws/<folder>` (defaults to the tracer). Reach for `/raws/scan` for multi-clip hunts, `/raws/clip-range` for surgical single cuts. (The masked-image and time-scoped-video editors live alongside it at `/tools/image` and `/tools/video`.)
|
|
@@ -28,6 +28,8 @@ Send a stable `tracer` on export so retries are traceable and filterable in job
|
|
|
28
28
|
|
|
29
29
|
**Scripting mode** is the recommended posture for repeatable template automation. Use it when a director wants to take a template they like, agree on a base fork, and then drive bulk or one-off edits entirely through REST or `vidfarm api` from a script, Lambda, or local machine.
|
|
30
30
|
|
|
31
|
+
This section is for a **desktop/local coding agent**, not the web copilot. A local Codex/Claude agent may use its shell and filesystem to write JavaScript/TypeScript/Python/shell scripts, fetch every API page, join and score catalog/library data, calculate statistics, emit CSV/JSON/Markdown reports, manipulate composition DOM files, and run iterative render/inspection loops. The web copilot cannot inherit those abilities from this document: it may only call its declared tools and bounded REST routes. If web chat prepares work for this flow, consume its My Files handoff document as input; do not claim the web chat itself executed the script.
|
|
32
|
+
|
|
31
33
|
Contract:
|
|
32
34
|
|
|
33
35
|
1. Pick the template once, then fork once and treat the resulting `forkId` as the stable base.
|
|
@@ -111,7 +113,7 @@ If a local AI script rewrites text or scenes without consuming those files first
|
|
|
111
113
|
|
|
112
114
|
| Command | REST route | Flow step |
|
|
113
115
|
|---|---|---|
|
|
114
|
-
| `vidfarm discover [query]` | `GET /discover/feed[?q=]` |
|
|
116
|
+
| `vidfarm discover [query] [--sort relevance\|wow\|automation\|recent] [--cursor <cursor>]` | `GET /discover/feed[?q=&sort=&cursor=]` | hybrid-search templates and page through the catalog |
|
|
115
117
|
| `vidfarm videos [query] [--mine]` | `GET /api/v1/videos[?q=&mine=]` | browse/search source inspirations |
|
|
116
118
|
| `vidfarm inspiration-add <url\|file>` | `POST /discover/templates` (files: `POST /discover/templates/upload/presign` + PUT first) | add your own source (URL or local video upload) |
|
|
117
119
|
| `vidfarm inspiration-rm <id>` | `DELETE /discover/templates/:id` | remove a private one |
|
|
@@ -51,7 +51,7 @@ Templates are listed on the Vidfarm homepage and `/discover`. Each has a `templa
|
|
|
51
51
|
|
|
52
52
|
- Browser: browse `/discover`
|
|
53
53
|
- API: `GET /discover/feed` — returns `{ templates: [{ templateId, slugId, title, previewUrl, viralDna, durationSeconds, sourceType, promotions, keywords, summary, ... }], next_cursor }`
|
|
54
|
-
- Search: `GET /discover/feed?q=<offer>&limit=20`
|
|
54
|
+
- Search: `GET /discover/feed?q=<offer>&limit=20&sort=relevance` hybrid-searches the eligible public catalog using semantic embeddings plus lexical matches. Semantic query embedding uses Vidfarm's canonical OpenRouter-routed model and bills the provider cost × the standard 1.2 markup to the user's wallet. The response's `search` block reports `mode`, `embedding_space`, and any `semantic_limitation`; when it says `lexical_structured`, disclose the limitation briefly and continue rather than refusing. Decomposition adds `promotions`, `keywords`, `summary`, `categoryTags`, and `catalogIntelligence` (`wowScore`, `wowReason`, `automationScore`, `automationReason`, `contentStyles`, `searchText`). Use `sort=wow` for highest-quality/client-impressing formats, `sort=automation` for cheap repeatable bulk formats, and `sort=recent` only when freshness is the intent. Follow `next_cursor` with `cursor=<value>`; never call page one the whole catalog. `GET /api/v1/videos?q=<offer>&limit=20[&mine=true]` searches source **inspirations**. Undecomposed inspirations have only sparse ingest metadata, so they are harder to retrieve semantically. Explain that somebody in the world needs to decompose one once and the shared enrichment then benefits everyone; the current user need not act unless they want that specific inspiration immediately.
|
|
55
55
|
|
|
56
56
|
Each template exposes a public preview:
|
|
57
57
|
|
|
@@ -73,7 +73,7 @@ To bring a new viral video into the catalog as a **private** template you own, i
|
|
|
73
73
|
- `DELETE /discover/templates/:entryId` — remove a private inspiration/template you own (accepts either the `inspiration_...` or minted `template_...` id).
|
|
74
74
|
- `POST /api/v1/inspirations/:inspirationId/decompose { user_prompt? }` — AI-decompose an inspiration's downloaded video into scenes (requires a saved provider key; same 120s source cap as auto-decompose).
|
|
75
75
|
|
|
76
|
-
devcli: `vidfarm inspiration-add <url|file.mp4>` (a local file path runs the presign→PUT→finalize flow for you), `vidfarm inspiration-rm <id>`, `vidfarm inspiration-decompose <id>`, `vidfarm discover [query]` to browse/search templates, `vidfarm public-raws [query] [--category <key>] [--type <raw_type>] [--bookmark <raw_id>]` to browse/save the public raws catalog, and `vidfarm videos [query] [--mine]` to browse/search source inspirations. On the web, the Discover page's **Add Template** modal accepts either a URL or a file upload.
|
|
76
|
+
devcli: `vidfarm inspiration-add <url|file.mp4>` (a local file path runs the presign→PUT→finalize flow for you), `vidfarm inspiration-rm <id>`, `vidfarm inspiration-decompose <id>`, `vidfarm discover [query] [--sort relevance|wow|automation|recent] [--cursor <next_cursor>]` to browse/search templates, `vidfarm public-raws [query] [--category <key>] [--type <raw_type>] [--bookmark <raw_id>]` to browse/save the public raws catalog, and `vidfarm videos [query] [--mine]` to browse/search source inspirations. On the web, the Discover page's **Add Template** modal accepts either a URL or a file upload.
|
|
77
77
|
|
|
78
78
|
## Fork a template
|
|
79
79
|
|
package/SKILL.director.md
CHANGED
|
@@ -50,11 +50,23 @@ Present both harnesses to the director, recommend (A) unless they've asked for p
|
|
|
50
50
|
|
|
51
51
|
## Web AI chat vs. local devcli — know your surface's limits
|
|
52
52
|
|
|
53
|
+
Choose the surface before choosing instructions:
|
|
54
|
+
|
|
55
|
+
| Surface | Live timeline | Shell / local files | Use it for |
|
|
56
|
+
|---|---:|---:|---|
|
|
57
|
+
| Web copilot | Yes, through `editor_action` | No | Short and medium edits on the open composition |
|
|
58
|
+
| Desktop coding agent | Through pulled files and devcli | Yes | Full rebuilds, scripts, batch work, and render/inspect loops |
|
|
59
|
+
| Devcli without an agent | Command-driven | Yes | Deterministic pull, edit, render, approve, and automation operations |
|
|
60
|
+
|
|
61
|
+
When this pack is installed locally, assume the desktop coding-agent surface unless trusted runtime context explicitly identifies the web editor. In web chat, declared tools and the latest `editor_context` override any capability described in this broad pack.
|
|
62
|
+
|
|
53
63
|
You may be running as the **in-web AI chat** (the /editor copilot, the chat dock, or the /chat and /library assistants) or as a **local coding agent** driving `vidfarm-devcli` (Claude Code / Codex on the user's machine). Same three paintbrushes, different reach — be honest about which surface you are and route heavy work accordingly:
|
|
54
64
|
|
|
65
|
+
- **Determine the surface before claiming capabilities.** The web chat has only its declared tools and REST routes. It cannot execute arbitrary JavaScript/Python, open a shell, create a local repository script, or use the user's filesystem. Never tell a web-chat user that you ran code or wrote a script unless a dedicated declared tool actually did so. A desktop coding agent has a real shell and filesystem and MAY write/run scripts, perform arbitrary local computations over paginated API results, create reports/CSVs/JSON, edit composition files, and orchestrate long devcli workflows within the user's authorization.
|
|
55
66
|
- **The web AI chat can do all three paintbrushes** — clip raws, author HTML/hyperframe motion, and generate AI media — and it drives edits directly on the live timeline. Keep small-to-medium jobs here: text/caption swaps, a scene or two replaced, single generations, captions, approve/schedule. Just do them.
|
|
56
67
|
- **Where the web chat struggles: complex, long, multi-step transformations.** A full multi-scene re-theme, an iterative render-critique-iterate loop, heavy scripted or batch work, or anything needing a real filesystem and many sequential tool calls will hit context limits, turn/timeout ceilings, and the web editor's constraints (CSS/declarative motion only — JS animation adapters are stripped on save). Don't grind a big transformation one layer at a time in a chat turn and stall.
|
|
57
68
|
- **Practical workaround — hand the heavy job to local devcli.** When a task is genuinely large or long-running, **proactively recommend the director run it locally with an AI coding agent** (Claude Code / OpenAI Codex / any capable agent): `vidfarm pull <forkId>` writes the composition + the `.harness/` grounding bundle to disk, the agent edits with the full devcli verb set and JS animation adapters, renders free with `vidfarm serve`, and `vidfarm publish` pushes it back. This is the **best-quality (B) harness's** natural home (adversarial grading with a coding agent). Frame it as "this is a big rebuild — you'll get a better, faster result running it locally with a coding agent; here's how," not as a dead end.
|
|
69
|
+
- **Offer a handoff, do not impersonate the desktop agent.** When web chat reaches that boundary, offer to save a Markdown handoff in My Files containing the objective, selected template/fork IDs, asset paths, grounding, constraints, completed work, and suggested devcli commands. Create it only after the user agrees. The desktop agent should read that document, pull the referenced fork, and then use its actual code/shell capabilities.
|
|
58
70
|
- **Never send the user away just to read knowledge.** Deeper skill knowledge is always a **tool call** away in-place: call `load_skill` (e.g. `load_skill('vidfarm-director', file='references/editor-workflows.md')`, or a craft pack like `editor-capabilities` / `hyperframes-animation`) to pull the exact reference you need mid-conversation. Only recommend switching surfaces for the WORK (a heavy transformation), never for the information.
|
|
59
71
|
|
|
60
72
|
## Read Only What You Need
|
|
@@ -68,6 +80,20 @@ Read only the relevant reference file for the current task.
|
|
|
68
80
|
- Getting-started interviews, strategy docs, onboarding flow: `references/onboarding.md`
|
|
69
81
|
- Primitive routes such as TTS, STT, music, overlays, background removal, product placement: `references/primitives.md`
|
|
70
82
|
|
|
83
|
+
## HyperFrames Skills — Load on Demand
|
|
84
|
+
|
|
85
|
+
Vidfarm ships a curated HyperFrames skill suite alongside this director pack. Use it for composition authoring and motion craft without loading the entire suite into context.
|
|
86
|
+
|
|
87
|
+
1. Route broad video-creation requests through `hyperframes` first. It selects the appropriate workflow skill.
|
|
88
|
+
2. Load only the selected workflow skill, then add the narrow domain skill required by the current step.
|
|
89
|
+
3. Load `hyperframes-core` before writing or restructuring composition HTML.
|
|
90
|
+
4. Load `hyperframes-animation`, `hyperframes-keyframes`, `hyperframes-creative`, or `hyperframes-cli` only when the task needs that specific capability.
|
|
91
|
+
5. Use `vidfarm-media` for narration, music, transcription, captions timing, background removal, and media sourcing. Prefer Vidfarm primitives and the user's existing library; when unavailable, use an equivalent capability already exposed by the user's desktop AI agent.
|
|
92
|
+
|
|
93
|
+
On the web copilot, call `load_skill('<name>')` and load referenced files only when the selected skill reaches that step. On desktop, inspect the agent's available-skill catalog and local `.agents/skills` / `.claude/skills` entries before declaring a skill missing. If needed, install only the selected bundled pack with `vidfarm skills add <name>`; do not bulk-install or bulk-read the suite. If the Vidfarm CLI is unavailable, use the desktop agent's native skill discovery or local-skill mechanism and continue with the closest installed equivalent.
|
|
94
|
+
|
|
95
|
+
HyperFrames authoring and rendering in this package are Vidfarm-native: local work uses the bundled composition toolchain and `vidfarm serve`; cloud work uses Vidfarm render routes. Do not require an external vendor account, repository, publish service, or telemetry endpoint. Keep `HYPERFRAMES_SKIP_SKILLS=1` and `HYPERFRAMES_NO_TELEMETRY=1` in Vidfarm-managed environments so the bundled skills stay pinned and local work does not phone home.
|
|
96
|
+
|
|
71
97
|
## Quick Router
|
|
72
98
|
|
|
73
99
|
Choose the narrowest path that satisfies the request.
|
|
@@ -158,7 +184,7 @@ Templates are listed on the Vidfarm homepage and `/discover`. Each has a `templa
|
|
|
158
184
|
|
|
159
185
|
- Browser: browse `/discover`
|
|
160
186
|
- API: `GET /discover/feed` — returns `{ templates: [{ templateId, slugId, title, previewUrl, viralDna, durationSeconds, sourceType, promotions, keywords, summary, ... }], next_cursor }`
|
|
161
|
-
- Search: `GET /discover/feed?q=<offer>&limit=20`
|
|
187
|
+
- Search: `GET /discover/feed?q=<offer>&limit=20&sort=relevance` hybrid-searches the eligible public catalog using semantic embeddings plus lexical matches. Semantic query embedding uses Vidfarm's canonical OpenRouter-routed model and bills the provider cost × the standard 1.2 markup to the user's wallet. The response's `search` block reports `mode`, `embedding_space`, and any `semantic_limitation`; when it says `lexical_structured`, disclose the limitation briefly and continue rather than refusing. Decomposition adds `promotions`, `keywords`, `summary`, `categoryTags`, and `catalogIntelligence` (`wowScore`, `wowReason`, `automationScore`, `automationReason`, `contentStyles`, `searchText`). Use `sort=wow` for highest-quality/client-impressing formats, `sort=automation` for cheap repeatable bulk formats, and `sort=recent` only when freshness is the intent. Follow `next_cursor` with `cursor=<value>`; never call page one the whole catalog. `GET /api/v1/videos?q=<offer>&limit=20[&mine=true]` searches source **inspirations**. Undecomposed inspirations have only sparse ingest metadata, so they are harder to retrieve semantically. Explain that somebody in the world needs to decompose one once and the shared enrichment then benefits everyone; the current user need not act unless they want that specific inspiration immediately.
|
|
162
188
|
|
|
163
189
|
Each template exposes a public preview:
|
|
164
190
|
|
|
@@ -180,7 +206,7 @@ To bring a new viral video into the catalog as a **private** template you own, i
|
|
|
180
206
|
- `DELETE /discover/templates/:entryId` — remove a private inspiration/template you own (accepts either the `inspiration_...` or minted `template_...` id).
|
|
181
207
|
- `POST /api/v1/inspirations/:inspirationId/decompose { user_prompt? }` — AI-decompose an inspiration's downloaded video into scenes (requires a saved provider key; same 120s source cap as auto-decompose).
|
|
182
208
|
|
|
183
|
-
devcli: `vidfarm inspiration-add <url|file.mp4>` (a local file path runs the presign→PUT→finalize flow for you), `vidfarm inspiration-rm <id>`, `vidfarm inspiration-decompose <id>`, `vidfarm discover [query]` to browse/search templates, `vidfarm public-raws [query] [--category <key>] [--type <raw_type>] [--bookmark <raw_id>]` to browse/save the public raws catalog, and `vidfarm videos [query] [--mine]` to browse/search source inspirations. On the web, the Discover page's **Add Template** modal accepts either a URL or a file upload.
|
|
209
|
+
devcli: `vidfarm inspiration-add <url|file.mp4>` (a local file path runs the presign→PUT→finalize flow for you), `vidfarm inspiration-rm <id>`, `vidfarm inspiration-decompose <id>`, `vidfarm discover [query] [--sort relevance|wow|automation|recent] [--cursor <next_cursor>]` to browse/search templates, `vidfarm public-raws [query] [--category <key>] [--type <raw_type>] [--bookmark <raw_id>]` to browse/save the public raws catalog, and `vidfarm videos [query] [--mine]` to browse/search source inspirations. On the web, the Discover page's **Add Template** modal accepts either a URL or a file upload.
|
|
184
210
|
|
|
185
211
|
## Fork a template
|
|
186
212
|
|
|
@@ -774,7 +800,7 @@ Mine a **long-form** video (podcast, stream VOD, webinar, any YouTube/TikTok/IG/
|
|
|
774
800
|
|
|
775
801
|
- **Poll** `GET /raws/scan/:scanId` until the source `status` is `complete` (or `failed` — crashed pipelines are reconciled to `failed`, pollers never spin).
|
|
776
802
|
- **Results**: `GET /raws/feed?source=<source_video_id>` (or the whole library), hybrid search via `POST /raws/search` `{ query: "someone looks confused" }`, per-raw download at `GET /raws/:clipId/download`. Raws carry taxonomy tags, a description, a transcript, an `aspect` field when cropped, the hunt's `tracer`, and (with a gemini/openai key) a semantic embedding.
|
|
777
|
-
- **Billing** —
|
|
803
|
+
- **Billing** — scan compute (`clip_scan_lambda` GB-seconds + Step Functions transitions) and each persisted semantic embedding are wallet-metered; the 202 response includes a `compute_estimate`. AI tagging/refine runs on **your saved provider key (BYOK)** and is not wallet-billed. Submission is wallet-gated (402 when empty).
|
|
778
804
|
- **The original stays temporary** — URL-ingested and uploaded sources live ONLY in your temp folder (`users/…/temporary/clip-sources/…`) with a hard **30-day TTL** (auto-deleted from S3 + the temp-file list). The hunted raws themselves are durable.
|
|
779
805
|
- **Captions rule** — "no captions / no on-screen text" hunts are handled by scene SELECTION (the refine pass drops text-heavy scenes). **Never run GhostCut caption removal on a long-form source** (it is hard-capped at ~15 minutes); to actually erase burned-in text, apply `POST /api/v1/primitives/videos/remove-captions` to individual FINISHED raws afterwards.
|
|
780
806
|
- **Clip an EXACT subrange (no AI hunt)** — when you want one precise hand-picked in/out rather than an open-ended hunt, use the **Clipper**: the web page `https://vidfarm.cc/tools/clipper` (previews the source straight from its URL, HH:MM:SS:NNNN in/out, live length estimate, multi-clip session grouped by tracer), the endpoint `POST /raws/clip-range` `{ source_url|preview_url|temp_file_id, start_sec, end_sec, tracer?, folder_path?, name? }` (trims exactly that window into one raw — compute-only, no AI/key), or the devcli twin `vidfarm clipper <video-url|file> --start <t> --end <t> [--tracer NAME --folder NAME --name TXT]`. The devcli path is now **local-first**: local files clip into the local raws store by default, and URL sources are staged locally before trimming unless you pass `--cloud`. If you do **not** have a `VIDFARM_API_KEY`, URL clipping cannot bootstrap the remote source into the local backend; open the long-form raw video in the browser, save/extract the actual video file locally, then run `vidfarm clipper ./source.mp4 ...`. The Library "Import source" button opens the Clipper; each save lands in `/raws/<folder>` (defaults to the tracer). Reach for `/raws/scan` for multi-clip hunts, `/raws/clip-range` for surgical single cuts. (The masked-image and time-scoped-video editors live alongside it at `/tools/image` and `/tools/video`.)
|
|
@@ -893,6 +919,8 @@ Send a stable `tracer` on export so retries are traceable and filterable in job
|
|
|
893
919
|
|
|
894
920
|
**Scripting mode** is the recommended posture for repeatable template automation. Use it when a director wants to take a template they like, agree on a base fork, and then drive bulk or one-off edits entirely through REST or `vidfarm api` from a script, Lambda, or local machine.
|
|
895
921
|
|
|
922
|
+
This section is for a **desktop/local coding agent**, not the web copilot. A local Codex/Claude agent may use its shell and filesystem to write JavaScript/TypeScript/Python/shell scripts, fetch every API page, join and score catalog/library data, calculate statistics, emit CSV/JSON/Markdown reports, manipulate composition DOM files, and run iterative render/inspection loops. The web copilot cannot inherit those abilities from this document: it may only call its declared tools and bounded REST routes. If web chat prepares work for this flow, consume its My Files handoff document as input; do not claim the web chat itself executed the script.
|
|
923
|
+
|
|
896
924
|
Contract:
|
|
897
925
|
|
|
898
926
|
1. Pick the template once, then fork once and treat the resulting `forkId` as the stable base.
|
|
@@ -976,7 +1004,7 @@ If a local AI script rewrites text or scenes without consuming those files first
|
|
|
976
1004
|
|
|
977
1005
|
| Command | REST route | Flow step |
|
|
978
1006
|
|---|---|---|
|
|
979
|
-
| `vidfarm discover [query]` | `GET /discover/feed[?q=]` |
|
|
1007
|
+
| `vidfarm discover [query] [--sort relevance\|wow\|automation\|recent] [--cursor <cursor>]` | `GET /discover/feed[?q=&sort=&cursor=]` | hybrid-search templates and page through the catalog |
|
|
980
1008
|
| `vidfarm videos [query] [--mine]` | `GET /api/v1/videos[?q=&mine=]` | browse/search source inspirations |
|
|
981
1009
|
| `vidfarm inspiration-add <url\|file>` | `POST /discover/templates` (files: `POST /discover/templates/upload/presign` + PUT first) | add your own source (URL or local video upload) |
|
|
982
1010
|
| `vidfarm inspiration-rm <id>` | `DELETE /discover/templates/:id` | remove a private one |
|
package/dist/src/cli.js
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
2
|
import { copyFileSync, createWriteStream, existsSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs";
|
|
3
3
|
import path from "node:path";
|
|
4
|
-
import {
|
|
4
|
+
import { tmpdir } from "node:os";
|
|
5
5
|
import { randomUUID } from "node:crypto";
|
|
6
6
|
import { fileURLToPath } from "node:url";
|
|
7
7
|
import { parseArgs } from "node:util";
|
|
@@ -83,8 +83,10 @@ Local editor loop:
|
|
|
83
83
|
|
|
84
84
|
Discover & inspiration (browse the viral-video catalog, add your own source):
|
|
85
85
|
discover [query...] List/search TEMPLATES (best for an offer) → GET /discover/feed
|
|
86
|
-
--query <text>
|
|
86
|
+
--query <text> Hybrid semantic + keyword search
|
|
87
87
|
--limit <n> Max results
|
|
88
|
+
--cursor <cursor> Continue from next_cursor
|
|
89
|
+
--sort <mode> relevance, wow, automation, or recent
|
|
88
90
|
public-raws [query...] Browse PUBLIC RAWS (feed / categories) → GET /api/v1/public-raws
|
|
89
91
|
--query <text> Keyword/vector query over title + metadata
|
|
90
92
|
--category <key> Filter one public-raw category (e.g. emotion:confused)
|
|
@@ -487,11 +489,12 @@ Account:
|
|
|
487
489
|
add-provider-key <provider> <secret> Save an AI provider key → POST /api/v1/user/me/provider-keys
|
|
488
490
|
|
|
489
491
|
Agent skill (install the latest director skill so your AI agent can act):
|
|
490
|
-
update-skill
|
|
491
|
-
|
|
492
|
-
|
|
493
|
-
--
|
|
494
|
-
--print Print the
|
|
492
|
+
update-skill DEPRECATED installer alias; installs the canonical
|
|
493
|
+
multi-file vidfarm-director pack via \`skills add\`.
|
|
494
|
+
Use \`vidfarm skills add vidfarm-director\` directly.
|
|
495
|
+
--dir <path> Project root for the canonical pack install
|
|
496
|
+
--print Print the generated single-file director rollup
|
|
497
|
+
(copy/share artifact; not the canonical install)
|
|
495
498
|
|
|
496
499
|
Files (multi-step flows the devcli handles for you):
|
|
497
500
|
upload <file> Upload to the EPHEMERAL temp store, print URL → POST /api/v1/user/me/temporary-files/upload
|
|
@@ -2184,14 +2187,14 @@ async function runTemplateCommand(argv) {
|
|
|
2184
2187
|
}
|
|
2185
2188
|
}
|
|
2186
2189
|
// ── Discover & inspiration ──────────────────────────────────────────────────
|
|
2187
|
-
// `discover` lists TEMPLATES; `--query`
|
|
2190
|
+
// `discover` lists TEMPLATES; `--query` hybrid-searches the catalog so an agent
|
|
2188
2191
|
// CLI (Claude Code / Codex) can answer "which templates suit my <offer>?". Each
|
|
2189
2192
|
// result carries decompose-derived promotions/keywords/summary to reason over.
|
|
2190
2193
|
async function runDiscoverCommand(argv) {
|
|
2191
|
-
const parsed = parseArgs({ args: argv, allowPositionals: true, options: { ...commonOptions(), limit: { type: "string" }, query: { type: "string" } } });
|
|
2194
|
+
const parsed = parseArgs({ args: argv, allowPositionals: true, options: { ...commonOptions(), limit: { type: "string" }, query: { type: "string" }, cursor: { type: "string" }, sort: { type: "string" } } });
|
|
2192
2195
|
const ctx = commonContext(parsed.values);
|
|
2193
2196
|
const query = (parsed.values.query ?? parsed.positionals.join(" ").trim()) || undefined;
|
|
2194
|
-
const result = await apiRequest({ method: "GET", host: ctx.host, path: "/discover/feed", auth: ctx.auth, query: { limit: parsed.values.limit, q: query } });
|
|
2197
|
+
const result = await apiRequest({ method: "GET", host: ctx.host, path: "/discover/feed", auth: ctx.auth, query: { limit: parsed.values.limit, q: query, cursor: parsed.values.cursor, sort: parsed.values.sort } });
|
|
2195
2198
|
assertApiOk(result, "discover");
|
|
2196
2199
|
emitResult(result, ctx.json, [["Browse ", discoverFrontendUrl(ctx.host)]]);
|
|
2197
2200
|
}
|
|
@@ -6250,29 +6253,18 @@ async function runUpdateSkillCommand(argv) {
|
|
|
6250
6253
|
}
|
|
6251
6254
|
return;
|
|
6252
6255
|
}
|
|
6253
|
-
|
|
6254
|
-
|
|
6255
|
-
: parsed.values.global
|
|
6256
|
-
? path.join(homedir(), ".claude", "skills")
|
|
6257
|
-
: path.resolve(process.cwd(), ".claude", "skills");
|
|
6258
|
-
const written = [];
|
|
6259
|
-
for (const name of skillNames) {
|
|
6260
|
-
const { contents, source } = await fetchSkillContents(ctx.host, SKILL_TARGETS[name]);
|
|
6261
|
-
const destDir = path.join(skillsRoot, name);
|
|
6262
|
-
mkdirSync(destDir, { recursive: true });
|
|
6263
|
-
const destPath = path.join(destDir, "SKILL.md");
|
|
6264
|
-
writeFileSync(destPath, contents, "utf8");
|
|
6265
|
-
written.push({ skill: name, path: destPath, source, bytes: Buffer.byteLength(contents, "utf8") });
|
|
6266
|
-
}
|
|
6267
|
-
if (ctx.json) {
|
|
6268
|
-
printJson({ ok: true, skillsRoot, installed: written });
|
|
6269
|
-
return;
|
|
6256
|
+
if (parsed.values.global || parsed.values.platform) {
|
|
6257
|
+
throw new Error("update-skill --global/--platform are retired. Use `vidfarm skills add vidfarm-director --dir <project>` for the canonical multi-file pack; `update-skill --print` remains available for the generated copy-friendly rollup.");
|
|
6270
6258
|
}
|
|
6271
|
-
|
|
6272
|
-
|
|
6273
|
-
|
|
6274
|
-
|
|
6275
|
-
|
|
6276
|
-
|
|
6259
|
+
if (!ctx.json) {
|
|
6260
|
+
console.warn(`Deprecated: update-skill now delegates to the canonical multi-file skill installer. Use ${BOLD}vidfarm skills add vidfarm-director${RESET}.`);
|
|
6261
|
+
}
|
|
6262
|
+
await runSkillsCommand([
|
|
6263
|
+
"add",
|
|
6264
|
+
"vidfarm-director",
|
|
6265
|
+
...(parsed.values.dir ? ["--dir", String(parsed.values.dir)] : []),
|
|
6266
|
+
...(parsed.values.host ? ["--host", String(parsed.values.host)] : []),
|
|
6267
|
+
...(ctx.json ? ["--json"] : [])
|
|
6268
|
+
]);
|
|
6277
6269
|
}
|
|
6278
6270
|
//# sourceMappingURL=cli.js.map
|
|
@@ -71,6 +71,7 @@ export class ClipStore {
|
|
|
71
71
|
thumbnail_path TEXT NOT NULL,
|
|
72
72
|
tags_json TEXT NOT NULL,
|
|
73
73
|
description TEXT NOT NULL DEFAULT '',
|
|
74
|
+
embedding_text TEXT,
|
|
74
75
|
taxonomy_version TEXT NOT NULL,
|
|
75
76
|
tier TEXT NOT NULL,
|
|
76
77
|
user_notes TEXT,
|
|
@@ -96,6 +97,10 @@ export class ClipStore {
|
|
|
96
97
|
scanned_at TEXT NOT NULL
|
|
97
98
|
);
|
|
98
99
|
`);
|
|
100
|
+
const clipColumns = this.db.prepare("PRAGMA table_info(clips)").all();
|
|
101
|
+
if (!clipColumns.some((column) => column.name === "embedding_text")) {
|
|
102
|
+
this.db.exec("ALTER TABLE clips ADD COLUMN embedding_text TEXT");
|
|
103
|
+
}
|
|
99
104
|
}
|
|
100
105
|
seedBuiltinPresets() {
|
|
101
106
|
const stmt = this.db.prepare("INSERT OR REPLACE INTO presets(preset_id, name, description, criteria_json, builtin, created_at) VALUES (?, ?, ?, ?, 1, ?)");
|
|
@@ -109,9 +114,9 @@ export class ClipStore {
|
|
|
109
114
|
this.db
|
|
110
115
|
.prepare(`INSERT OR REPLACE INTO clips
|
|
111
116
|
(clip_id, source_video_id, source_filename, start_time_sec, end_time_sec, duration_sec,
|
|
112
|
-
file_path, thumbnail_path, tags_json, description, taxonomy_version, tier, user_notes, created_at)
|
|
113
|
-
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`)
|
|
114
|
-
.run(c.clip_id, c.source_video_id, c.source_filename, c.start_time_sec, c.end_time_sec, c.duration_sec, c.file_path, c.thumbnail_path, JSON.stringify(c.tags), c.description ?? "", c.taxonomy_version, c.tier, c.user_notes ?? null, c.created_at);
|
|
117
|
+
file_path, thumbnail_path, tags_json, description, embedding_text, taxonomy_version, tier, user_notes, created_at)
|
|
118
|
+
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`)
|
|
119
|
+
.run(c.clip_id, c.source_video_id, c.source_filename, c.start_time_sec, c.end_time_sec, c.duration_sec, c.file_path, c.thumbnail_path, JSON.stringify(c.tags), c.description ?? "", c.embedding_text ?? null, c.taxonomy_version, c.tier, c.user_notes ?? null, c.created_at);
|
|
115
120
|
if (c.embedding && c.embedding.length === this.dim) {
|
|
116
121
|
this.db.prepare("DELETE FROM vec_clips WHERE clip_id = ?").run(c.clip_id);
|
|
117
122
|
this.db
|
|
@@ -331,6 +336,7 @@ function rowToClip(row) {
|
|
|
331
336
|
thumbnail_path: row.thumbnail_path,
|
|
332
337
|
tags: safeParse(row.tags_json, emptyTags()),
|
|
333
338
|
description: row.description ?? "",
|
|
339
|
+
embedding_text: row.embedding_text ?? undefined,
|
|
334
340
|
taxonomy_version: row.taxonomy_version,
|
|
335
341
|
tier: row.tier ?? "flash-lite",
|
|
336
342
|
user_notes: row.user_notes ?? undefined,
|
package/dist/src/devcli/clips.js
CHANGED
|
@@ -12,7 +12,7 @@ import { homedir } from "node:os";
|
|
|
12
12
|
import { parseArgs } from "node:util";
|
|
13
13
|
import { createInterface } from "node:readline/promises";
|
|
14
14
|
import { createIdV7 } from "../lib/ids.js";
|
|
15
|
-
import { buildEffectiveGuidance, buildPrivateUserNotesGuidance, ClipModelClient, DEFAULT_CLIP_ASPECT, DEFAULT_CLIP_DURATION_BAND, detectLocalAgent, detectScenes, effectiveHuntDurationSec, estimateScanCostFromScenes,
|
|
15
|
+
import { buildEffectiveGuidance, buildPrivateUserNotesGuidance, ClipModelClient, DEFAULT_CLIP_ASPECT, DEFAULT_CLIP_DURATION_BAND, detectLocalAgent, detectScenes, effectiveHuntDurationSec, estimateScanCostFromScenes, formatCostEstimate, hasFfmpeg, LocalAgentClipClient, normalizeAspect, normalizeCropFocus, normalizeWindows, parseClipHuntPrompt, parseTimeRanges, probeVideo, resolveDurationBand, resolveTargetClipCount, scanVideo, searchClips } from "../services/clip-curation/index.js";
|
|
16
16
|
import { ClipStore } from "./clip-store.js";
|
|
17
17
|
export const CLIPS_HELP = `vidfarm raws — build & search a local raws library (local ffmpeg + local agent by default)
|
|
18
18
|
|
|
@@ -269,8 +269,7 @@ async function runScan(argv) {
|
|
|
269
269
|
console.log(`[raws] preferring scenes without on-screen text/captions (selection only — GhostCut is never run on the source).`);
|
|
270
270
|
console.log(`[raws] aiming for ~${targetClipCount} distinct raw${targetClipCount === 1 ? "" : "s"}${maxClips ? " (--max-clips)" : " (~10 / 10 min)"}.`);
|
|
271
271
|
console.log(`[raws] detecting scenes …`);
|
|
272
|
-
const
|
|
273
|
-
const scenes = fitScenesToDurationBand(detected, durationBand);
|
|
272
|
+
const scenes = await detectScenes(videoPath, { durationSec: probe.duration_sec, windows, ...sceneOptions });
|
|
274
273
|
if (scenes.length === 0)
|
|
275
274
|
throw new Error("No scenes detected (is the video readable / non-empty?).");
|
|
276
275
|
const store = new ClipStore(values.home);
|
|
@@ -130,7 +130,11 @@ export async function detectScenes(videoPath, opts = {}) {
|
|
|
130
130
|
}
|
|
131
131
|
async function detectScenesInSpan(videoPath, opts) {
|
|
132
132
|
const threshold = opts.threshold ?? 0.3;
|
|
133
|
-
|
|
133
|
+
// Preserve very short detected shots as independent candidates. Absorbing a
|
|
134
|
+
// 2-10 frame flash into its neighbour makes it impossible for prompt/OCR
|
|
135
|
+
// refinement to reject that flash later. 50ms only de-duplicates effectively
|
|
136
|
+
// identical detector timestamps.
|
|
137
|
+
const minScene = opts.minSceneSec ?? 0.05;
|
|
134
138
|
const maxScene = opts.maxSceneSec ?? 8;
|
|
135
139
|
const { spanStart, spanEnd } = opts;
|
|
136
140
|
const spanDuration = spanEnd - spanStart;
|
|
@@ -14,10 +14,10 @@ import { extractKeyframes } from "./ffmpeg.js";
|
|
|
14
14
|
const MAX_SCENES_PER_BATCH = 60;
|
|
15
15
|
// Bound total cost on very long sources: past this many batches we keep the raw
|
|
16
16
|
// segmentation for the whole video (logged) instead of firing dozens of calls.
|
|
17
|
-
const MAX_REFINE_BATCHES =
|
|
17
|
+
const MAX_REFINE_BATCHES = 30;
|
|
18
18
|
export async function refineScenesWithGuidance(input) {
|
|
19
19
|
const { client, videoPath, scenes, guidancePrompt } = input;
|
|
20
|
-
if (!guidancePrompt.trim() || scenes.length
|
|
20
|
+
if (!guidancePrompt.trim() || scenes.length === 0) {
|
|
21
21
|
return { scenes, refined: false, kept: scenes.length, dropped: 0, merged: 0 };
|
|
22
22
|
}
|
|
23
23
|
const batchCount = Math.ceil(scenes.length / MAX_SCENES_PER_BATCH);
|
|
@@ -60,6 +60,8 @@ export async function refineScenesWithGuidance(input) {
|
|
|
60
60
|
? Math.max(1, Math.round((input.targetClipCount * batch.length) / scenes.length))
|
|
61
61
|
: undefined;
|
|
62
62
|
const batchDecisions = await askForDecisions(client, guidancePrompt, batch, frames, batchTarget);
|
|
63
|
+
if (!batchDecisions)
|
|
64
|
+
throw new Error("model returned no refinement plan");
|
|
63
65
|
for (let i = 0; i < batch.length; i++) {
|
|
64
66
|
decisions.push(batchDecisions?.[i] ?? keepDefault());
|
|
65
67
|
}
|
|
@@ -111,7 +113,7 @@ async function askForDecisions(client, guidancePrompt, scenes, frames, targetCli
|
|
|
111
113
|
maxOutputTokens: Math.max(1000, scenes.length * 40)
|
|
112
114
|
});
|
|
113
115
|
}
|
|
114
|
-
function rebuildScenes(scenes, decisions) {
|
|
116
|
+
export function rebuildScenes(scenes, decisions) {
|
|
115
117
|
const out = [];
|
|
116
118
|
let dropped = 0;
|
|
117
119
|
let merged = 0;
|
|
@@ -122,19 +124,25 @@ function rebuildScenes(scenes, decisions) {
|
|
|
122
124
|
dropped++;
|
|
123
125
|
continue;
|
|
124
126
|
}
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
127
|
+
const previous = out[out.length - 1];
|
|
128
|
+
const isAdjacent = previous
|
|
129
|
+
? Math.abs(previous.end_time_sec - scenes[i].start_time_sec) <= 0.05
|
|
130
|
+
: false;
|
|
131
|
+
if (d.merge_with_previous && previous && isAdjacent) {
|
|
132
|
+
// Extend only an ADJACENT previous kept clip. A dropped scene creates a
|
|
133
|
+
// real timeline gap; bridging that gap would silently put the rejected
|
|
134
|
+
// footage back into the exported raw.
|
|
135
|
+
previous.end_time_sec = scenes[i].end_time_sec;
|
|
136
|
+
previous.duration_sec = round3(previous.end_time_sec - previous.start_time_sec);
|
|
130
137
|
merged++;
|
|
131
138
|
continue;
|
|
132
139
|
}
|
|
133
140
|
out.push({ ...scenes[i], index: out.length });
|
|
134
141
|
}
|
|
135
|
-
//
|
|
142
|
+
// An empty plan is a valid strict-prompt result. Silently restoring every
|
|
143
|
+
// scene here would completely ignore the user's request.
|
|
136
144
|
if (out.length === 0) {
|
|
137
|
-
return { scenes, refined:
|
|
145
|
+
return { scenes: [], refined: true, kept: 0, dropped, merged };
|
|
138
146
|
}
|
|
139
147
|
return { scenes: out, refined: true, kept: out.length, dropped, merged };
|
|
140
148
|
}
|
|
@@ -76,10 +76,10 @@ export async function processSceneToClip(input) {
|
|
|
76
76
|
await extractThumbnail({ videoPath, scene, outPath: thumbLocal, cropFilter });
|
|
77
77
|
}
|
|
78
78
|
// 5. Embed the document text for semantic search (skipped when the client can't embed).
|
|
79
|
+
const embeddingText = buildClipEmbeddingText(tags, tagResult.description, ctx.userNotes);
|
|
79
80
|
let embedding;
|
|
80
81
|
if (ctx.client.canEmbed) {
|
|
81
|
-
|
|
82
|
-
[embedding] = await ctx.client.embedDocuments([embedText]);
|
|
82
|
+
[embedding] = await ctx.client.embedDocuments([embeddingText]);
|
|
83
83
|
}
|
|
84
84
|
const clip = {
|
|
85
85
|
clip_id: clipId,
|
|
@@ -92,6 +92,7 @@ export async function processSceneToClip(input) {
|
|
|
92
92
|
thumbnail_path: ctx.thumbStoredPath ? ctx.thumbStoredPath(clipId, thumbLocal) : thumbLocal,
|
|
93
93
|
tags,
|
|
94
94
|
description: tagResult.description,
|
|
95
|
+
embedding_text: embeddingText,
|
|
95
96
|
embedding,
|
|
96
97
|
taxonomy_version: ctx.client.taxonomyVersion ?? ACTIVE_TAXONOMY_VERSION,
|
|
97
98
|
tier: ctx.client.tier,
|
|
@@ -113,21 +114,15 @@ export async function scanVideo(opts) {
|
|
|
113
114
|
const probe = opts.probe ?? (await probeVideo(opts.videoPath));
|
|
114
115
|
const windows = normalizeWindows(opts.windows, probe.duration_sec);
|
|
115
116
|
const band = opts.durationBand ?? null;
|
|
116
|
-
//
|
|
117
|
-
//
|
|
118
|
-
//
|
|
119
|
-
// them — the fit is intentionally NOT re-run on supplied scenes since
|
|
120
|
-
// fitting is not idempotent (a fitted 20s scene could re-merge toward the
|
|
121
|
-
// target on a second pass).
|
|
117
|
+
// Keep detected shots separate until AFTER prompt refinement. This is
|
|
118
|
+
// important: judging a multi-shot duration span from one midpoint frame can
|
|
119
|
+
// approve unrelated opening/closing footage or miss text overlays in them.
|
|
122
120
|
let scenes = opts.scenes ??
|
|
123
|
-
|
|
121
|
+
await detectScenes(opts.videoPath, {
|
|
124
122
|
durationSec: probe.duration_sec,
|
|
125
123
|
windows,
|
|
126
|
-
// A duration band lifts the scene-split ceiling so raw cuts can merge
|
|
127
|
-
// up to the requested clip length instead of being pre-chopped at 8s.
|
|
128
|
-
...(band ? { maxSceneSec: band.max_sec } : {}),
|
|
129
124
|
...opts.sceneOptions
|
|
130
|
-
})
|
|
125
|
+
});
|
|
131
126
|
// Prompt-guided segmentation: let the guidance reshape which scenes become
|
|
132
127
|
// clips (keep/drop/merge/dedup) before we spend tagging tokens on them.
|
|
133
128
|
if (opts.ctx.guidancePrompt?.trim() && opts.ctx.refineSegmentation !== false) {
|
|
@@ -139,11 +134,15 @@ export async function scanVideo(opts) {
|
|
|
139
134
|
workDir: opts.ctx.workDir,
|
|
140
135
|
targetClipCount: opts.targetClipCount
|
|
141
136
|
});
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
137
|
+
if (!refined.refined && refined.note) {
|
|
138
|
+
throw new Error(`Prompt-guided raw selection failed: ${refined.note}`);
|
|
139
|
+
}
|
|
140
|
+
scenes = refined.scenes;
|
|
145
141
|
opts.onRefine?.(refined);
|
|
146
142
|
}
|
|
143
|
+
// Assemble prompt-approved, contiguous shots into useful clip durations.
|
|
144
|
+
// fitScenesToDurationBand never crosses a dropped gap.
|
|
145
|
+
scenes = fitScenesToDurationBand(scenes, band);
|
|
147
146
|
// Deterministic count/uniqueness cap: if more scenes survived than the target
|
|
148
147
|
// (refine skipped >60 scenes, or the model kept more than asked), keep an
|
|
149
148
|
// evenly-spaced — hence less duplicative — subset.
|
|
@@ -12,6 +12,9 @@ export class SpeechVoiceMismatchError extends Error {
|
|
|
12
12
|
}
|
|
13
13
|
export class ProviderQuotaExceededError extends Error {
|
|
14
14
|
}
|
|
15
|
+
/** The provider refused the request or output because of its content/safety policy. */
|
|
16
|
+
export class ProviderSafetyError extends Error {
|
|
17
|
+
}
|
|
15
18
|
export class ProviderWaitExceededError extends Error {
|
|
16
19
|
}
|
|
17
20
|
export class ProviderLeaseTimeoutError extends Error {
|
|
@@ -63,6 +66,54 @@ export function isHardQuotaError(details) {
|
|
|
63
66
|
"resource_exhausted"
|
|
64
67
|
].some((pattern) => normalized.includes(pattern));
|
|
65
68
|
}
|
|
69
|
+
export function isSafetyOrRefusalError(details) {
|
|
70
|
+
const normalized = details.toLowerCase();
|
|
71
|
+
return [
|
|
72
|
+
"content_policy_violation",
|
|
73
|
+
"content policy",
|
|
74
|
+
"safety policy",
|
|
75
|
+
"safety filter",
|
|
76
|
+
"safety_reason",
|
|
77
|
+
"finishreason\":\"safety",
|
|
78
|
+
"moderation_blocked",
|
|
79
|
+
"blocked_reason",
|
|
80
|
+
"blockedreason",
|
|
81
|
+
"prompt was blocked",
|
|
82
|
+
"moderation",
|
|
83
|
+
"moderated",
|
|
84
|
+
"responsible ai",
|
|
85
|
+
"refused",
|
|
86
|
+
"refusal",
|
|
87
|
+
"prohibited content",
|
|
88
|
+
"policy violation"
|
|
89
|
+
].some((pattern) => normalized.includes(pattern));
|
|
90
|
+
}
|
|
91
|
+
export function providerFailureFromDetails(provider, operation, status, details) {
|
|
92
|
+
if (isSafetyOrRefusalError(details)) {
|
|
93
|
+
return new ProviderSafetyError(`${provider} refused this ${operation} because its safety or content policy was triggered${conciseProviderDetails(details)}. ` +
|
|
94
|
+
"Try revising the prompt or choose another provider/model.");
|
|
95
|
+
}
|
|
96
|
+
return new Error(`${provider} ${operation} returned ${status}${conciseProviderDetails(details)}`);
|
|
97
|
+
}
|
|
98
|
+
export function serializeProviderFailure(error) {
|
|
99
|
+
const message = error instanceof Error ? error.message : "AI provider request failed";
|
|
100
|
+
if (error instanceof ProviderSafetyError) {
|
|
101
|
+
return { message, type: "provider_safety_refusal", retryable: false, user_action: "Revise the prompt or choose another provider/model." };
|
|
102
|
+
}
|
|
103
|
+
if (error instanceof ProviderRateLimitError) {
|
|
104
|
+
return { message, type: "provider_rate_limited", retryable: true, retry_after_seconds: error.retryAfterSeconds, user_action: "Wait, then retry or choose another provider." };
|
|
105
|
+
}
|
|
106
|
+
if (error instanceof ProviderQuotaExceededError) {
|
|
107
|
+
return { message, type: "provider_quota_exceeded", retryable: false, user_action: "Check provider billing/quota or choose another provider." };
|
|
108
|
+
}
|
|
109
|
+
if (error instanceof ProviderAuthError) {
|
|
110
|
+
return { message, type: "provider_auth_error", retryable: false, user_action: "Check the saved provider API key." };
|
|
111
|
+
}
|
|
112
|
+
if (error instanceof ProviderTransientError) {
|
|
113
|
+
return { message, type: "provider_temporarily_unavailable", retryable: true, retry_after_seconds: error.retryAfterSeconds, user_action: "Retry shortly or choose another provider." };
|
|
114
|
+
}
|
|
115
|
+
return { message, type: "provider_request_failed", retryable: false };
|
|
116
|
+
}
|
|
66
117
|
export function conciseProviderDetails(details) {
|
|
67
118
|
const compact = details.replace(/\s+/g, " ").trim();
|
|
68
119
|
return compact ? `: ${compact.slice(0, 500)}` : "";
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@mevdragon/vidfarm-devcli",
|
|
3
|
-
"version": "0.20.
|
|
3
|
+
"version": "0.20.13",
|
|
4
4
|
"description": "Local bridge for the Vidfarm Trackpad Editor. `vidfarm serve <template_id>` boots the FULL editor on localhost (disk-backed records/storage, free in-process render); edit composition.html on disk (Claude Code, Codex, etc.) and the browser live-morphs it.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
@@ -53,6 +53,7 @@
|
|
|
53
53
|
"dev:cli": "tsx src/cli.ts",
|
|
54
54
|
"build:frontend": "node scripts/build-homepage-client.mjs",
|
|
55
55
|
"build:hyperframes-editor": "cd demo && node scripts/build.mjs",
|
|
56
|
+
"seo:refresh": "tsx scripts/generate-route-seo.ts",
|
|
56
57
|
"operator:template-artifact": "node scripts/platform-operator-template-artifact.mjs",
|
|
57
58
|
"bootstrap:staging-templates": "tsx scripts/bootstrap-staging-template-releases.ts",
|
|
58
59
|
"migrate:sqlite-to-dynamodb": "tsx scripts/migrate-sqlite-to-dynamodb.ts",
|
|
@@ -60,7 +61,10 @@
|
|
|
60
61
|
"rollback:prod-inplace": "bash scripts/rollback-prod-inplace.sh",
|
|
61
62
|
"build": "npm run build:frontend && npm run build:hyperframes-editor && node -e \"require('node:fs').rmSync('dist', { recursive: true, force: true })\" && tsc -p tsconfig.json",
|
|
62
63
|
"start": "node --import ./dist/src/instrument.js --enable-source-maps dist/src/index.js",
|
|
63
|
-
"check": "tsc -p tsconfig.json --noEmit",
|
|
64
|
+
"check": "tsc -p tsconfig.json --noEmit && npm run check:skills",
|
|
65
|
+
"test:clips": "node --import tsx --test test/clip-curation.test.ts",
|
|
66
|
+
"check:skills": "node scripts/build-director-skill-rollup.mjs --check && node scripts/check-skill-routes.mjs",
|
|
67
|
+
"benchmark:editor-chat": "node --import tsx scripts/benchmark-editor-chat-harness.mjs",
|
|
64
68
|
"cdk:deploy:prod-serverless": "npm run build && dotenv -e .env.production -- npx aws-cdk deploy --app 'node dist/infra/cdk/bin/vidfarm-prod.js'",
|
|
65
69
|
"cdk:synth:staging-serverless": "npm run build && dotenv -e .env.staging -- npx aws-cdk synth --app 'node dist/infra/cdk/bin/vidfarm-serverless-staging.js'",
|
|
66
70
|
"cdk:deploy:staging-serverless": "npm run build && dotenv -e .env.staging -- npx aws-cdk deploy --app 'node dist/infra/cdk/bin/vidfarm-serverless-staging.js'",
|