@mevdragon/vidfarm-devcli 0.20.12 → 0.20.14
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/skills/hyperframes-cli/SKILL.md +1 -1
- package/.agents/skills/vidfarm-director/SKILL.md +29 -0
- package/.agents/skills/vidfarm-director/references/assets-and-sourcing.md +1 -1
- package/.agents/skills/vidfarm-director/references/automation-and-local-dev.md +3 -1
- package/.agents/skills/vidfarm-director/references/core-workflows.md +2 -2
- package/.agents/skills/vidfarm-director/references/rest-api.md +84 -0
- package/SKILL.director.md +35 -4
- package/dist/src/cli.js +25 -33
- package/dist/src/devcli/clip-store.js +9 -3
- package/dist/src/devcli/clips.js +2 -3
- package/dist/src/services/clip-curation/ffmpeg.js +5 -1
- package/dist/src/services/clip-curation/refine.js +18 -10
- package/dist/src/services/clip-curation/scan.js +15 -16
- package/dist/src/services/provider-errors.js +51 -0
- package/package.json +6 -2
|
@@ -86,7 +86,7 @@ Add `inspect` for layout-sensitive work and `render --strict` in CI to fail on l
|
|
|
86
86
|
|
|
87
87
|
### Visual smoke test — required when the project uses sub-compositions
|
|
88
88
|
|
|
89
|
-
`lint` / `validate` / `inspect` evaluate each composition **in isolation**. They never load `index.html` and mount sub-compositions via `data-composition-src`, so they cannot catch cross-file mount failures
|
|
89
|
+
`lint` / `validate` / `inspect` evaluate each composition **in isolation**. They never load `index.html` and mount sub-compositions via `data-composition-src`, so they cannot catch cross-file mount failures. Load the `hyperframes-core` skill, then read its `references/sub-compositions.md` “Common pitfalls” section. The only gate that catches them is one that actually loads `index.html` and seeks the timeline.
|
|
90
90
|
|
|
91
91
|
Use `hyperframes snapshot` — it loads the project the same way `render` does (so it exercises the same mount path) but only captures the timestamps you request, so it's seconds instead of a full render:
|
|
92
92
|
|
|
@@ -50,11 +50,25 @@ Present both harnesses to the director, recommend (A) unless they've asked for p
|
|
|
50
50
|
|
|
51
51
|
## Web AI chat vs. local devcli — know your surface's limits
|
|
52
52
|
|
|
53
|
+
**Desktop users: this is a skill, not an SDK or required runtime.** It gives an AI agent Vidfarm's operating knowledge; it does not proxy requests, add hidden capabilities, or require the agent to use a particular client. Every cloud workflow described here can be performed with ordinary HTTP against the Vidfarm REST API. For convenience and safer composition/file workflows, recommend the `vidfarm` command from `@mevdragon/vidfarm-devcli`: it wraps the same REST routes, handles auth, uploads, polling, pagination, local files, and pull/edit/render loops. Use raw REST whenever the user prefers it, the CLI lacks a new route, or the work belongs in an existing service or automation stack.
|
|
54
|
+
|
|
55
|
+
Choose the surface before choosing instructions:
|
|
56
|
+
|
|
57
|
+
| Surface | Live timeline | Shell / local files | Use it for |
|
|
58
|
+
|---|---:|---:|---|
|
|
59
|
+
| Web copilot | Yes, through `editor_action` | No | Short and medium edits on the open composition |
|
|
60
|
+
| Desktop coding agent | Through pulled files and devcli | Yes | Full rebuilds, scripts, batch work, and render/inspect loops |
|
|
61
|
+
| Devcli without an agent | Command-driven | Yes | Deterministic pull, edit, render, approve, and automation operations |
|
|
62
|
+
|
|
63
|
+
When this pack is installed locally, assume the desktop coding-agent surface unless trusted runtime context explicitly identifies the web editor. In web chat, declared tools and the latest `editor_context` override any capability described in this broad pack.
|
|
64
|
+
|
|
53
65
|
You may be running as the **in-web AI chat** (the /editor copilot, the chat dock, or the /chat and /library assistants) or as a **local coding agent** driving `vidfarm-devcli` (Claude Code / Codex on the user's machine). Same three paintbrushes, different reach — be honest about which surface you are and route heavy work accordingly:
|
|
54
66
|
|
|
67
|
+
- **Determine the surface before claiming capabilities.** The web chat has only its declared tools and REST routes. It cannot execute arbitrary JavaScript/Python, open a shell, create a local repository script, or use the user's filesystem. Never tell a web-chat user that you ran code or wrote a script unless a dedicated declared tool actually did so. A desktop coding agent has a real shell and filesystem and MAY write/run scripts, perform arbitrary local computations over paginated API results, create reports/CSVs/JSON, edit composition files, and orchestrate long devcli workflows within the user's authorization.
|
|
55
68
|
- **The web AI chat can do all three paintbrushes** — clip raws, author HTML/hyperframe motion, and generate AI media — and it drives edits directly on the live timeline. Keep small-to-medium jobs here: text/caption swaps, a scene or two replaced, single generations, captions, approve/schedule. Just do them.
|
|
56
69
|
- **Where the web chat struggles: complex, long, multi-step transformations.** A full multi-scene re-theme, an iterative render-critique-iterate loop, heavy scripted or batch work, or anything needing a real filesystem and many sequential tool calls will hit context limits, turn/timeout ceilings, and the web editor's constraints (CSS/declarative motion only — JS animation adapters are stripped on save). Don't grind a big transformation one layer at a time in a chat turn and stall.
|
|
57
70
|
- **Practical workaround — hand the heavy job to local devcli.** When a task is genuinely large or long-running, **proactively recommend the director run it locally with an AI coding agent** (Claude Code / OpenAI Codex / any capable agent): `vidfarm pull <forkId>` writes the composition + the `.harness/` grounding bundle to disk, the agent edits with the full devcli verb set and JS animation adapters, renders free with `vidfarm serve`, and `vidfarm publish` pushes it back. This is the **best-quality (B) harness's** natural home (adversarial grading with a coding agent). Frame it as "this is a big rebuild — you'll get a better, faster result running it locally with a coding agent; here's how," not as a dead end.
|
|
71
|
+
- **Offer a handoff, do not impersonate the desktop agent.** When web chat reaches that boundary, offer to save a Markdown handoff in My Files containing the objective, selected template/fork IDs, asset paths, grounding, constraints, completed work, and suggested devcli commands. Create it only after the user agrees. The desktop agent should read that document, pull the referenced fork, and then use its actual code/shell capabilities.
|
|
58
72
|
- **Never send the user away just to read knowledge.** Deeper skill knowledge is always a **tool call** away in-place: call `load_skill` (e.g. `load_skill('vidfarm-director', file='references/editor-workflows.md')`, or a craft pack like `editor-capabilities` / `hyperframes-animation`) to pull the exact reference you need mid-conversation. Only recommend switching surfaces for the WORK (a heavy transformation), never for the information.
|
|
59
73
|
|
|
60
74
|
## Read Only What You Need
|
|
@@ -67,6 +81,21 @@ Read only the relevant reference file for the current task.
|
|
|
67
81
|
- REST automation, `vidfarm` command surface, local serve loop, skill packs: `references/automation-and-local-dev.md`
|
|
68
82
|
- Getting-started interviews, strategy docs, onboarding flow: `references/onboarding.md`
|
|
69
83
|
- Primitive routes such as TTS, STT, music, overlays, background removal, product placement: `references/primitives.md`
|
|
84
|
+
- Complete REST API map and raw-HTTP conventions: `references/rest-api.md`. Load it only when the user asks for REST, an endpoint/schema, direct HTTP integration, or exhaustive API coverage. For the entire specification, follow its domain links and load every listed reference; do not preload them into ordinary director conversations.
|
|
85
|
+
|
|
86
|
+
## HyperFrames Skills — Load on Demand
|
|
87
|
+
|
|
88
|
+
Vidfarm ships a curated HyperFrames skill suite alongside this director pack. Use it for composition authoring and motion craft without loading the entire suite into context.
|
|
89
|
+
|
|
90
|
+
1. Route broad video-creation requests through `hyperframes` first. It selects the appropriate workflow skill.
|
|
91
|
+
2. Load only the selected workflow skill, then add the narrow domain skill required by the current step.
|
|
92
|
+
3. Load `hyperframes-core` before writing or restructuring composition HTML.
|
|
93
|
+
4. Load `hyperframes-animation`, `hyperframes-keyframes`, `hyperframes-creative`, or `hyperframes-cli` only when the task needs that specific capability.
|
|
94
|
+
5. Use `vidfarm-media` for narration, music, transcription, captions timing, background removal, and media sourcing. Prefer Vidfarm primitives and the user's existing library; when unavailable, use an equivalent capability already exposed by the user's desktop AI agent.
|
|
95
|
+
|
|
96
|
+
On the web copilot, call `load_skill('<name>')` and load referenced files only when the selected skill reaches that step. On desktop, inspect the agent's available-skill catalog and local `.agents/skills` / `.claude/skills` entries before declaring a skill missing. If needed, install only the selected bundled pack with `vidfarm skills add <name>`; do not bulk-install or bulk-read the suite. If the Vidfarm CLI is unavailable, use the desktop agent's native skill discovery or local-skill mechanism and continue with the closest installed equivalent.
|
|
97
|
+
|
|
98
|
+
HyperFrames authoring and rendering in this package are Vidfarm-native: local work uses the bundled composition toolchain and `vidfarm serve`; cloud work uses Vidfarm render routes. Do not require an external vendor account, repository, publish service, or telemetry endpoint. Keep `HYPERFRAMES_SKIP_SKILLS=1` and `HYPERFRAMES_NO_TELEMETRY=1` in Vidfarm-managed environments so the bundled skills stay pinned and local work does not phone home.
|
|
70
99
|
|
|
71
100
|
## Quick Router
|
|
72
101
|
|
|
@@ -27,7 +27,7 @@ Mine a **long-form** video (podcast, stream VOD, webinar, any YouTube/TikTok/IG/
|
|
|
27
27
|
|
|
28
28
|
- **Poll** `GET /raws/scan/:scanId` until the source `status` is `complete` (or `failed` — crashed pipelines are reconciled to `failed`, pollers never spin).
|
|
29
29
|
- **Results**: `GET /raws/feed?source=<source_video_id>` (or the whole library), hybrid search via `POST /raws/search` `{ query: "someone looks confused" }`, per-raw download at `GET /raws/:clipId/download`. Raws carry taxonomy tags, a description, a transcript, an `aspect` field when cropped, the hunt's `tracer`, and (with a gemini/openai key) a semantic embedding.
|
|
30
|
-
- **Billing** —
|
|
30
|
+
- **Billing** — scan compute (`clip_scan_lambda` GB-seconds + Step Functions transitions) and each persisted semantic embedding are wallet-metered; the 202 response includes a `compute_estimate`. AI tagging/refine runs on **your saved provider key (BYOK)** and is not wallet-billed. Submission is wallet-gated (402 when empty).
|
|
31
31
|
- **The original stays temporary** — URL-ingested and uploaded sources live ONLY in your temp folder (`users/…/temporary/clip-sources/…`) with a hard **30-day TTL** (auto-deleted from S3 + the temp-file list). The hunted raws themselves are durable.
|
|
32
32
|
- **Captions rule** — "no captions / no on-screen text" hunts are handled by scene SELECTION (the refine pass drops text-heavy scenes). **Never run GhostCut caption removal on a long-form source** (it is hard-capped at ~15 minutes); to actually erase burned-in text, apply `POST /api/v1/primitives/videos/remove-captions` to individual FINISHED raws afterwards.
|
|
33
33
|
- **Clip an EXACT subrange (no AI hunt)** — when you want one precise hand-picked in/out rather than an open-ended hunt, use the **Clipper**: the web page `https://vidfarm.cc/tools/clipper` (previews the source straight from its URL, HH:MM:SS:NNNN in/out, live length estimate, multi-clip session grouped by tracer), the endpoint `POST /raws/clip-range` `{ source_url|preview_url|temp_file_id, start_sec, end_sec, tracer?, folder_path?, name? }` (trims exactly that window into one raw — compute-only, no AI/key), or the devcli twin `vidfarm clipper <video-url|file> --start <t> --end <t> [--tracer NAME --folder NAME --name TXT]`. The devcli path is now **local-first**: local files clip into the local raws store by default, and URL sources are staged locally before trimming unless you pass `--cloud`. If you do **not** have a `VIDFARM_API_KEY`, URL clipping cannot bootstrap the remote source into the local backend; open the long-form raw video in the browser, save/extract the actual video file locally, then run `vidfarm clipper ./source.mp4 ...`. The Library "Import source" button opens the Clipper; each save lands in `/raws/<folder>` (defaults to the tracer). Reach for `/raws/scan` for multi-clip hunts, `/raws/clip-range` for surgical single cuts. (The masked-image and time-scoped-video editors live alongside it at `/tools/image` and `/tools/video`.)
|
|
@@ -28,6 +28,8 @@ Send a stable `tracer` on export so retries are traceable and filterable in job
|
|
|
28
28
|
|
|
29
29
|
**Scripting mode** is the recommended posture for repeatable template automation. Use it when a director wants to take a template they like, agree on a base fork, and then drive bulk or one-off edits entirely through REST or `vidfarm api` from a script, Lambda, or local machine.
|
|
30
30
|
|
|
31
|
+
This section is for a **desktop/local coding agent**, not the web copilot. A local Codex/Claude agent may use its shell and filesystem to write JavaScript/TypeScript/Python/shell scripts, fetch every API page, join and score catalog/library data, calculate statistics, emit CSV/JSON/Markdown reports, manipulate composition DOM files, and run iterative render/inspection loops. The web copilot cannot inherit those abilities from this document: it may only call its declared tools and bounded REST routes. If web chat prepares work for this flow, consume its My Files handoff document as input; do not claim the web chat itself executed the script.
|
|
32
|
+
|
|
31
33
|
Contract:
|
|
32
34
|
|
|
33
35
|
1. Pick the template once, then fork once and treat the resulting `forkId` as the stable base.
|
|
@@ -111,7 +113,7 @@ If a local AI script rewrites text or scenes without consuming those files first
|
|
|
111
113
|
|
|
112
114
|
| Command | REST route | Flow step |
|
|
113
115
|
|---|---|---|
|
|
114
|
-
| `vidfarm discover [query]` | `GET /discover/feed[?q=]` |
|
|
116
|
+
| `vidfarm discover [query] [--sort relevance\|wow\|automation\|recent] [--cursor <cursor>]` | `GET /discover/feed[?q=&sort=&cursor=]` | hybrid-search templates and page through the catalog |
|
|
115
117
|
| `vidfarm videos [query] [--mine]` | `GET /api/v1/videos[?q=&mine=]` | browse/search source inspirations |
|
|
116
118
|
| `vidfarm inspiration-add <url\|file>` | `POST /discover/templates` (files: `POST /discover/templates/upload/presign` + PUT first) | add your own source (URL or local video upload) |
|
|
117
119
|
| `vidfarm inspiration-rm <id>` | `DELETE /discover/templates/:id` | remove a private one |
|
|
@@ -51,7 +51,7 @@ Templates are listed on the Vidfarm homepage and `/discover`. Each has a `templa
|
|
|
51
51
|
|
|
52
52
|
- Browser: browse `/discover`
|
|
53
53
|
- API: `GET /discover/feed` — returns `{ templates: [{ templateId, slugId, title, previewUrl, viralDna, durationSeconds, sourceType, promotions, keywords, summary, ... }], next_cursor }`
|
|
54
|
-
- Search: `GET /discover/feed?q=<offer>&limit=20`
|
|
54
|
+
- Search: `GET /discover/feed?q=<offer>&limit=20&sort=relevance` hybrid-searches the eligible public catalog using semantic embeddings plus lexical matches. Semantic query embedding uses Vidfarm's canonical OpenRouter-routed model and bills the provider cost × the standard 1.2 markup to the user's wallet. The response's `search` block reports `mode`, `embedding_space`, and any `semantic_limitation`; when it says `lexical_structured`, disclose the limitation briefly and continue rather than refusing. Decomposition adds `promotions`, `keywords`, `summary`, `categoryTags`, and `catalogIntelligence` (`wowScore`, `wowReason`, `automationScore`, `automationReason`, `contentStyles`, `searchText`). Use `sort=wow` for highest-quality/client-impressing formats, `sort=automation` for cheap repeatable bulk formats, and `sort=recent` only when freshness is the intent. Follow `next_cursor` with `cursor=<value>`; never call page one the whole catalog. `GET /api/v1/videos?q=<offer>&limit=20[&mine=true]` searches source **inspirations**. Undecomposed inspirations have only sparse ingest metadata, so they are harder to retrieve semantically. Explain that somebody in the world needs to decompose one once and the shared enrichment then benefits everyone; the current user need not act unless they want that specific inspiration immediately.
|
|
55
55
|
|
|
56
56
|
Each template exposes a public preview:
|
|
57
57
|
|
|
@@ -73,7 +73,7 @@ To bring a new viral video into the catalog as a **private** template you own, i
|
|
|
73
73
|
- `DELETE /discover/templates/:entryId` — remove a private inspiration/template you own (accepts either the `inspiration_...` or minted `template_...` id).
|
|
74
74
|
- `POST /api/v1/inspirations/:inspirationId/decompose { user_prompt? }` — AI-decompose an inspiration's downloaded video into scenes (requires a saved provider key; same 120s source cap as auto-decompose).
|
|
75
75
|
|
|
76
|
-
devcli: `vidfarm inspiration-add <url|file.mp4>` (a local file path runs the presign→PUT→finalize flow for you), `vidfarm inspiration-rm <id>`, `vidfarm inspiration-decompose <id>`, `vidfarm discover [query]` to browse/search templates, `vidfarm public-raws [query] [--category <key>] [--type <raw_type>] [--bookmark <raw_id>]` to browse/save the public raws catalog, and `vidfarm videos [query] [--mine]` to browse/search source inspirations. On the web, the Discover page's **Add Template** modal accepts either a URL or a file upload.
|
|
76
|
+
devcli: `vidfarm inspiration-add <url|file.mp4>` (a local file path runs the presign→PUT→finalize flow for you), `vidfarm inspiration-rm <id>`, `vidfarm inspiration-decompose <id>`, `vidfarm discover [query] [--sort relevance|wow|automation|recent] [--cursor <next_cursor>]` to browse/search templates, `vidfarm public-raws [query] [--category <key>] [--type <raw_type>] [--bookmark <raw_id>]` to browse/save the public raws catalog, and `vidfarm videos [query] [--mine]` to browse/search source inspirations. On the web, the Discover page's **Add Template** modal accepts either a URL or a file upload.
|
|
77
77
|
|
|
78
78
|
## Fork a template
|
|
79
79
|
|
|
@@ -0,0 +1,84 @@
|
|
|
1
|
+
# Vidfarm REST API — On-Demand Map
|
|
2
|
+
|
|
3
|
+
Use this file when integrating through raw HTTP, looking up an endpoint, or auditing the complete public API. This is a navigation layer: the detailed request bodies, response shapes, state transitions, cost notes, and route-specific guardrails live in the linked domain references so the default skill context stays small.
|
|
4
|
+
|
|
5
|
+
## Contents
|
|
6
|
+
|
|
7
|
+
- [Client choice](#client-choice)
|
|
8
|
+
- [Base URL and authentication](#base-url-and-authentication)
|
|
9
|
+
- [Complete specification map](#complete-specification-map)
|
|
10
|
+
- [Raw HTTP pattern](#raw-http-pattern)
|
|
11
|
+
- [Operational rules](#operational-rules)
|
|
12
|
+
- [Keeping the reference complete](#keeping-the-reference-complete)
|
|
13
|
+
|
|
14
|
+
## Client choice
|
|
15
|
+
|
|
16
|
+
The skill is documentation for an AI agent. It is not a client library and does not execute requests.
|
|
17
|
+
|
|
18
|
+
- **Raw REST:** every cloud operation is available through HTTP and can be called from `curl`, `fetch`, Python, CI, a server, or any other HTTP client.
|
|
19
|
+
- **Recommended desktop convenience:** `npx @mevdragon/vidfarm-devcli ...` (or the installed `vidfarm` binary) wraps those same routes. Prefer it for interactive desktop work because it handles the `vidfarm-api-key` header, uploads, polling, pagination, local paths, composition pull/publish state, and readable errors.
|
|
20
|
+
- **Use raw REST** when building a durable integration, working in an existing application stack, calling a route the devcli has not wrapped yet, or when the user explicitly requests HTTP.
|
|
21
|
+
|
|
22
|
+
Do not imply that installing this skill installs the CLI. Do not imply that the CLI unlocks API capabilities unavailable through REST.
|
|
23
|
+
|
|
24
|
+
## Base URL and authentication
|
|
25
|
+
|
|
26
|
+
- Production base URL: `https://vidfarm.cc`
|
|
27
|
+
- API-key authentication: `vidfarm-api-key: <key>`
|
|
28
|
+
- Do not use `Authorization: Bearer` unless a specific non-API browser/login flow explicitly documents it.
|
|
29
|
+
- Send `content-type: application/json` for JSON bodies and `accept: application/json` when JSON is expected.
|
|
30
|
+
- Treat fork IDs, share tokens, upload URLs, and returned media URLs according to the route-specific guidance in the domain references.
|
|
31
|
+
|
|
32
|
+
## Complete specification map
|
|
33
|
+
|
|
34
|
+
For a focused task, load only the relevant file. If the user requests the **entire REST API specification**, load this file plus every file in the following table. Together they are the bundled director-facing REST specification.
|
|
35
|
+
|
|
36
|
+
| API area | Load | Coverage |
|
|
37
|
+
|---|---|---|
|
|
38
|
+
| Accounts, login, provider keys, discovery, inspirations, templates, forks, versions, render, approval, sharing, scheduling, billing | `core-workflows.md` | Customer and composition lifecycle |
|
|
39
|
+
| Composition HTML/JSON, timeline layers, editor actions, clone/fork, auto-decompose, video context, captions, transitions, speech regeneration | `editor-workflows.md` | Editing and decomposition |
|
|
40
|
+
| Public raws, clip hunts, exact-range clipping, My Files, temporary files, folders, annotations, semantic search, recurring-character assets | `assets-and-sourcing.md` | Media and library APIs |
|
|
41
|
+
| Direct automation, pagination, scripting, devcli-to-route parity, local/cloud boundaries, skill-pack distribution | `automation-and-local-dev.md` | Headless and desktop integration |
|
|
42
|
+
| Image, video, TTS, STT, music, captions, overlays, inpaint, background/caption removal, product placement, async primitive jobs | `primitives.md` | Generation and transformation primitives |
|
|
43
|
+
| Onboarding document persistence | `onboarding.md` | Durable strategy/context files |
|
|
44
|
+
|
|
45
|
+
The route and payload examples in those files are the public, customer-facing contract. Architecture-only internals are intentionally excluded. When exact behavior is ambiguous, inspect the live response or route implementation rather than inventing a field.
|
|
46
|
+
|
|
47
|
+
## Raw HTTP pattern
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
curl --fail-with-body \
|
|
51
|
+
-H "vidfarm-api-key: $VIDFARM_API_KEY" \
|
|
52
|
+
-H "accept: application/json" \
|
|
53
|
+
"https://vidfarm.cc/discover/feed?limit=20"
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
For JSON mutations:
|
|
57
|
+
|
|
58
|
+
```bash
|
|
59
|
+
curl --fail-with-body \
|
|
60
|
+
-X POST \
|
|
61
|
+
-H "vidfarm-api-key: $VIDFARM_API_KEY" \
|
|
62
|
+
-H "content-type: application/json" \
|
|
63
|
+
-H "accept: application/json" \
|
|
64
|
+
--data '{"template_id":"template_example"}' \
|
|
65
|
+
"https://vidfarm.cc/api/v1/compositions"
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
Use environment variables or a secret manager for keys. Never paste secrets into a skill, composition, source file, command transcript intended for sharing, or generated report.
|
|
69
|
+
|
|
70
|
+
## Operational rules
|
|
71
|
+
|
|
72
|
+
- Follow `next_cursor`/cursor fields until exhausted when the user asks for all records; do not call the first page the full dataset.
|
|
73
|
+
- Treat submission and generation routes as non-idempotent unless the route explicitly states otherwise. Check existing state before retrying.
|
|
74
|
+
- Poll asynchronous jobs at the documented status endpoint and stop on terminal success or failure. Do not fabricate completion from an accepted response.
|
|
75
|
+
- Upload local media through the documented presign/upload/finalize sequence. The devcli is recommended on desktop because it performs this sequence automatically.
|
|
76
|
+
- Parse and mutate composition HTML as a DOM. Never edit it by string concatenation.
|
|
77
|
+
- Render through `POST /api/v1/compositions/:forkId/render`; do not call the underlying renderer directly.
|
|
78
|
+
- Surface wallet/provider costs before expensive operations and preserve returned job IDs for audit and retries.
|
|
79
|
+
|
|
80
|
+
## Keeping the reference complete
|
|
81
|
+
|
|
82
|
+
The canonical skill sources live in `.agents/skills/vidfarm-director/`. `SKILL.director.md` is a generated copy-friendly rollup; this REST map is deliberately excluded from that immediate rollup so ordinary skill loads remain lean. It is distributed in the multi-file skill pack and fetched on demand through the public skill-pack routes.
|
|
83
|
+
|
|
84
|
+
When routes change, update the appropriate domain reference and run `npm run check:skills`. That check verifies every documented `/api/` and `/login/` method/path against the registered application routes. Add new public API areas to the table above so exhaustive API requests remain discoverable.
|
package/SKILL.director.md
CHANGED
|
@@ -50,11 +50,25 @@ Present both harnesses to the director, recommend (A) unless they've asked for p
|
|
|
50
50
|
|
|
51
51
|
## Web AI chat vs. local devcli — know your surface's limits
|
|
52
52
|
|
|
53
|
+
**Desktop users: this is a skill, not an SDK or required runtime.** It gives an AI agent Vidfarm's operating knowledge; it does not proxy requests, add hidden capabilities, or require the agent to use a particular client. Every cloud workflow described here can be performed with ordinary HTTP against the Vidfarm REST API. For convenience and safer composition/file workflows, recommend the `vidfarm` command from `@mevdragon/vidfarm-devcli`: it wraps the same REST routes, handles auth, uploads, polling, pagination, local files, and pull/edit/render loops. Use raw REST whenever the user prefers it, the CLI lacks a new route, or the work belongs in an existing service or automation stack.
|
|
54
|
+
|
|
55
|
+
Choose the surface before choosing instructions:
|
|
56
|
+
|
|
57
|
+
| Surface | Live timeline | Shell / local files | Use it for |
|
|
58
|
+
|---|---:|---:|---|
|
|
59
|
+
| Web copilot | Yes, through `editor_action` | No | Short and medium edits on the open composition |
|
|
60
|
+
| Desktop coding agent | Through pulled files and devcli | Yes | Full rebuilds, scripts, batch work, and render/inspect loops |
|
|
61
|
+
| Devcli without an agent | Command-driven | Yes | Deterministic pull, edit, render, approve, and automation operations |
|
|
62
|
+
|
|
63
|
+
When this pack is installed locally, assume the desktop coding-agent surface unless trusted runtime context explicitly identifies the web editor. In web chat, declared tools and the latest `editor_context` override any capability described in this broad pack.
|
|
64
|
+
|
|
53
65
|
You may be running as the **in-web AI chat** (the /editor copilot, the chat dock, or the /chat and /library assistants) or as a **local coding agent** driving `vidfarm-devcli` (Claude Code / Codex on the user's machine). Same three paintbrushes, different reach — be honest about which surface you are and route heavy work accordingly:
|
|
54
66
|
|
|
67
|
+
- **Determine the surface before claiming capabilities.** The web chat has only its declared tools and REST routes. It cannot execute arbitrary JavaScript/Python, open a shell, create a local repository script, or use the user's filesystem. Never tell a web-chat user that you ran code or wrote a script unless a dedicated declared tool actually did so. A desktop coding agent has a real shell and filesystem and MAY write/run scripts, perform arbitrary local computations over paginated API results, create reports/CSVs/JSON, edit composition files, and orchestrate long devcli workflows within the user's authorization.
|
|
55
68
|
- **The web AI chat can do all three paintbrushes** — clip raws, author HTML/hyperframe motion, and generate AI media — and it drives edits directly on the live timeline. Keep small-to-medium jobs here: text/caption swaps, a scene or two replaced, single generations, captions, approve/schedule. Just do them.
|
|
56
69
|
- **Where the web chat struggles: complex, long, multi-step transformations.** A full multi-scene re-theme, an iterative render-critique-iterate loop, heavy scripted or batch work, or anything needing a real filesystem and many sequential tool calls will hit context limits, turn/timeout ceilings, and the web editor's constraints (CSS/declarative motion only — JS animation adapters are stripped on save). Don't grind a big transformation one layer at a time in a chat turn and stall.
|
|
57
70
|
- **Practical workaround — hand the heavy job to local devcli.** When a task is genuinely large or long-running, **proactively recommend the director run it locally with an AI coding agent** (Claude Code / OpenAI Codex / any capable agent): `vidfarm pull <forkId>` writes the composition + the `.harness/` grounding bundle to disk, the agent edits with the full devcli verb set and JS animation adapters, renders free with `vidfarm serve`, and `vidfarm publish` pushes it back. This is the **best-quality (B) harness's** natural home (adversarial grading with a coding agent). Frame it as "this is a big rebuild — you'll get a better, faster result running it locally with a coding agent; here's how," not as a dead end.
|
|
71
|
+
- **Offer a handoff, do not impersonate the desktop agent.** When web chat reaches that boundary, offer to save a Markdown handoff in My Files containing the objective, selected template/fork IDs, asset paths, grounding, constraints, completed work, and suggested devcli commands. Create it only after the user agrees. The desktop agent should read that document, pull the referenced fork, and then use its actual code/shell capabilities.
|
|
58
72
|
- **Never send the user away just to read knowledge.** Deeper skill knowledge is always a **tool call** away in-place: call `load_skill` (e.g. `load_skill('vidfarm-director', file='references/editor-workflows.md')`, or a craft pack like `editor-capabilities` / `hyperframes-animation`) to pull the exact reference you need mid-conversation. Only recommend switching surfaces for the WORK (a heavy transformation), never for the information.
|
|
59
73
|
|
|
60
74
|
## Read Only What You Need
|
|
@@ -67,6 +81,21 @@ Read only the relevant reference file for the current task.
|
|
|
67
81
|
- REST automation, `vidfarm` command surface, local serve loop, skill packs: `references/automation-and-local-dev.md`
|
|
68
82
|
- Getting-started interviews, strategy docs, onboarding flow: `references/onboarding.md`
|
|
69
83
|
- Primitive routes such as TTS, STT, music, overlays, background removal, product placement: `references/primitives.md`
|
|
84
|
+
- Complete REST API map and raw-HTTP conventions: `references/rest-api.md`. Load it only when the user asks for REST, an endpoint/schema, direct HTTP integration, or exhaustive API coverage. For the entire specification, follow its domain links and load every listed reference; do not preload them into ordinary director conversations.
|
|
85
|
+
|
|
86
|
+
## HyperFrames Skills — Load on Demand
|
|
87
|
+
|
|
88
|
+
Vidfarm ships a curated HyperFrames skill suite alongside this director pack. Use it for composition authoring and motion craft without loading the entire suite into context.
|
|
89
|
+
|
|
90
|
+
1. Route broad video-creation requests through `hyperframes` first. It selects the appropriate workflow skill.
|
|
91
|
+
2. Load only the selected workflow skill, then add the narrow domain skill required by the current step.
|
|
92
|
+
3. Load `hyperframes-core` before writing or restructuring composition HTML.
|
|
93
|
+
4. Load `hyperframes-animation`, `hyperframes-keyframes`, `hyperframes-creative`, or `hyperframes-cli` only when the task needs that specific capability.
|
|
94
|
+
5. Use `vidfarm-media` for narration, music, transcription, captions timing, background removal, and media sourcing. Prefer Vidfarm primitives and the user's existing library; when unavailable, use an equivalent capability already exposed by the user's desktop AI agent.
|
|
95
|
+
|
|
96
|
+
On the web copilot, call `load_skill('<name>')` and load referenced files only when the selected skill reaches that step. On desktop, inspect the agent's available-skill catalog and local `.agents/skills` / `.claude/skills` entries before declaring a skill missing. If needed, install only the selected bundled pack with `vidfarm skills add <name>`; do not bulk-install or bulk-read the suite. If the Vidfarm CLI is unavailable, use the desktop agent's native skill discovery or local-skill mechanism and continue with the closest installed equivalent.
|
|
97
|
+
|
|
98
|
+
HyperFrames authoring and rendering in this package are Vidfarm-native: local work uses the bundled composition toolchain and `vidfarm serve`; cloud work uses Vidfarm render routes. Do not require an external vendor account, repository, publish service, or telemetry endpoint. Keep `HYPERFRAMES_SKIP_SKILLS=1` and `HYPERFRAMES_NO_TELEMETRY=1` in Vidfarm-managed environments so the bundled skills stay pinned and local work does not phone home.
|
|
70
99
|
|
|
71
100
|
## Quick Router
|
|
72
101
|
|
|
@@ -158,7 +187,7 @@ Templates are listed on the Vidfarm homepage and `/discover`. Each has a `templa
|
|
|
158
187
|
|
|
159
188
|
- Browser: browse `/discover`
|
|
160
189
|
- API: `GET /discover/feed` — returns `{ templates: [{ templateId, slugId, title, previewUrl, viralDna, durationSeconds, sourceType, promotions, keywords, summary, ... }], next_cursor }`
|
|
161
|
-
- Search: `GET /discover/feed?q=<offer>&limit=20`
|
|
190
|
+
- Search: `GET /discover/feed?q=<offer>&limit=20&sort=relevance` hybrid-searches the eligible public catalog using semantic embeddings plus lexical matches. Semantic query embedding uses Vidfarm's canonical OpenRouter-routed model and bills the provider cost × the standard 1.2 markup to the user's wallet. The response's `search` block reports `mode`, `embedding_space`, and any `semantic_limitation`; when it says `lexical_structured`, disclose the limitation briefly and continue rather than refusing. Decomposition adds `promotions`, `keywords`, `summary`, `categoryTags`, and `catalogIntelligence` (`wowScore`, `wowReason`, `automationScore`, `automationReason`, `contentStyles`, `searchText`). Use `sort=wow` for highest-quality/client-impressing formats, `sort=automation` for cheap repeatable bulk formats, and `sort=recent` only when freshness is the intent. Follow `next_cursor` with `cursor=<value>`; never call page one the whole catalog. `GET /api/v1/videos?q=<offer>&limit=20[&mine=true]` searches source **inspirations**. Undecomposed inspirations have only sparse ingest metadata, so they are harder to retrieve semantically. Explain that somebody in the world needs to decompose one once and the shared enrichment then benefits everyone; the current user need not act unless they want that specific inspiration immediately.
|
|
162
191
|
|
|
163
192
|
Each template exposes a public preview:
|
|
164
193
|
|
|
@@ -180,7 +209,7 @@ To bring a new viral video into the catalog as a **private** template you own, i
|
|
|
180
209
|
- `DELETE /discover/templates/:entryId` — remove a private inspiration/template you own (accepts either the `inspiration_...` or minted `template_...` id).
|
|
181
210
|
- `POST /api/v1/inspirations/:inspirationId/decompose { user_prompt? }` — AI-decompose an inspiration's downloaded video into scenes (requires a saved provider key; same 120s source cap as auto-decompose).
|
|
182
211
|
|
|
183
|
-
devcli: `vidfarm inspiration-add <url|file.mp4>` (a local file path runs the presign→PUT→finalize flow for you), `vidfarm inspiration-rm <id>`, `vidfarm inspiration-decompose <id>`, `vidfarm discover [query]` to browse/search templates, `vidfarm public-raws [query] [--category <key>] [--type <raw_type>] [--bookmark <raw_id>]` to browse/save the public raws catalog, and `vidfarm videos [query] [--mine]` to browse/search source inspirations. On the web, the Discover page's **Add Template** modal accepts either a URL or a file upload.
|
|
212
|
+
devcli: `vidfarm inspiration-add <url|file.mp4>` (a local file path runs the presign→PUT→finalize flow for you), `vidfarm inspiration-rm <id>`, `vidfarm inspiration-decompose <id>`, `vidfarm discover [query] [--sort relevance|wow|automation|recent] [--cursor <next_cursor>]` to browse/search templates, `vidfarm public-raws [query] [--category <key>] [--type <raw_type>] [--bookmark <raw_id>]` to browse/save the public raws catalog, and `vidfarm videos [query] [--mine]` to browse/search source inspirations. On the web, the Discover page's **Add Template** modal accepts either a URL or a file upload.
|
|
184
213
|
|
|
185
214
|
## Fork a template
|
|
186
215
|
|
|
@@ -774,7 +803,7 @@ Mine a **long-form** video (podcast, stream VOD, webinar, any YouTube/TikTok/IG/
|
|
|
774
803
|
|
|
775
804
|
- **Poll** `GET /raws/scan/:scanId` until the source `status` is `complete` (or `failed` — crashed pipelines are reconciled to `failed`, pollers never spin).
|
|
776
805
|
- **Results**: `GET /raws/feed?source=<source_video_id>` (or the whole library), hybrid search via `POST /raws/search` `{ query: "someone looks confused" }`, per-raw download at `GET /raws/:clipId/download`. Raws carry taxonomy tags, a description, a transcript, an `aspect` field when cropped, the hunt's `tracer`, and (with a gemini/openai key) a semantic embedding.
|
|
777
|
-
- **Billing** —
|
|
806
|
+
- **Billing** — scan compute (`clip_scan_lambda` GB-seconds + Step Functions transitions) and each persisted semantic embedding are wallet-metered; the 202 response includes a `compute_estimate`. AI tagging/refine runs on **your saved provider key (BYOK)** and is not wallet-billed. Submission is wallet-gated (402 when empty).
|
|
778
807
|
- **The original stays temporary** — URL-ingested and uploaded sources live ONLY in your temp folder (`users/…/temporary/clip-sources/…`) with a hard **30-day TTL** (auto-deleted from S3 + the temp-file list). The hunted raws themselves are durable.
|
|
779
808
|
- **Captions rule** — "no captions / no on-screen text" hunts are handled by scene SELECTION (the refine pass drops text-heavy scenes). **Never run GhostCut caption removal on a long-form source** (it is hard-capped at ~15 minutes); to actually erase burned-in text, apply `POST /api/v1/primitives/videos/remove-captions` to individual FINISHED raws afterwards.
|
|
780
809
|
- **Clip an EXACT subrange (no AI hunt)** — when you want one precise hand-picked in/out rather than an open-ended hunt, use the **Clipper**: the web page `https://vidfarm.cc/tools/clipper` (previews the source straight from its URL, HH:MM:SS:NNNN in/out, live length estimate, multi-clip session grouped by tracer), the endpoint `POST /raws/clip-range` `{ source_url|preview_url|temp_file_id, start_sec, end_sec, tracer?, folder_path?, name? }` (trims exactly that window into one raw — compute-only, no AI/key), or the devcli twin `vidfarm clipper <video-url|file> --start <t> --end <t> [--tracer NAME --folder NAME --name TXT]`. The devcli path is now **local-first**: local files clip into the local raws store by default, and URL sources are staged locally before trimming unless you pass `--cloud`. If you do **not** have a `VIDFARM_API_KEY`, URL clipping cannot bootstrap the remote source into the local backend; open the long-form raw video in the browser, save/extract the actual video file locally, then run `vidfarm clipper ./source.mp4 ...`. The Library "Import source" button opens the Clipper; each save lands in `/raws/<folder>` (defaults to the tracer). Reach for `/raws/scan` for multi-clip hunts, `/raws/clip-range` for surgical single cuts. (The masked-image and time-scoped-video editors live alongside it at `/tools/image` and `/tools/video`.)
|
|
@@ -893,6 +922,8 @@ Send a stable `tracer` on export so retries are traceable and filterable in job
|
|
|
893
922
|
|
|
894
923
|
**Scripting mode** is the recommended posture for repeatable template automation. Use it when a director wants to take a template they like, agree on a base fork, and then drive bulk or one-off edits entirely through REST or `vidfarm api` from a script, Lambda, or local machine.
|
|
895
924
|
|
|
925
|
+
This section is for a **desktop/local coding agent**, not the web copilot. A local Codex/Claude agent may use its shell and filesystem to write JavaScript/TypeScript/Python/shell scripts, fetch every API page, join and score catalog/library data, calculate statistics, emit CSV/JSON/Markdown reports, manipulate composition DOM files, and run iterative render/inspection loops. The web copilot cannot inherit those abilities from this document: it may only call its declared tools and bounded REST routes. If web chat prepares work for this flow, consume its My Files handoff document as input; do not claim the web chat itself executed the script.
|
|
926
|
+
|
|
896
927
|
Contract:
|
|
897
928
|
|
|
898
929
|
1. Pick the template once, then fork once and treat the resulting `forkId` as the stable base.
|
|
@@ -976,7 +1007,7 @@ If a local AI script rewrites text or scenes without consuming those files first
|
|
|
976
1007
|
|
|
977
1008
|
| Command | REST route | Flow step |
|
|
978
1009
|
|---|---|---|
|
|
979
|
-
| `vidfarm discover [query]` | `GET /discover/feed[?q=]` |
|
|
1010
|
+
| `vidfarm discover [query] [--sort relevance\|wow\|automation\|recent] [--cursor <cursor>]` | `GET /discover/feed[?q=&sort=&cursor=]` | hybrid-search templates and page through the catalog |
|
|
980
1011
|
| `vidfarm videos [query] [--mine]` | `GET /api/v1/videos[?q=&mine=]` | browse/search source inspirations |
|
|
981
1012
|
| `vidfarm inspiration-add <url\|file>` | `POST /discover/templates` (files: `POST /discover/templates/upload/presign` + PUT first) | add your own source (URL or local video upload) |
|
|
982
1013
|
| `vidfarm inspiration-rm <id>` | `DELETE /discover/templates/:id` | remove a private one |
|
package/dist/src/cli.js
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
2
|
import { copyFileSync, createWriteStream, existsSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs";
|
|
3
3
|
import path from "node:path";
|
|
4
|
-
import {
|
|
4
|
+
import { tmpdir } from "node:os";
|
|
5
5
|
import { randomUUID } from "node:crypto";
|
|
6
6
|
import { fileURLToPath } from "node:url";
|
|
7
7
|
import { parseArgs } from "node:util";
|
|
@@ -83,8 +83,10 @@ Local editor loop:
|
|
|
83
83
|
|
|
84
84
|
Discover & inspiration (browse the viral-video catalog, add your own source):
|
|
85
85
|
discover [query...] List/search TEMPLATES (best for an offer) → GET /discover/feed
|
|
86
|
-
--query <text>
|
|
86
|
+
--query <text> Hybrid semantic + keyword search
|
|
87
87
|
--limit <n> Max results
|
|
88
|
+
--cursor <cursor> Continue from next_cursor
|
|
89
|
+
--sort <mode> relevance, wow, automation, or recent
|
|
88
90
|
public-raws [query...] Browse PUBLIC RAWS (feed / categories) → GET /api/v1/public-raws
|
|
89
91
|
--query <text> Keyword/vector query over title + metadata
|
|
90
92
|
--category <key> Filter one public-raw category (e.g. emotion:confused)
|
|
@@ -487,11 +489,12 @@ Account:
|
|
|
487
489
|
add-provider-key <provider> <secret> Save an AI provider key → POST /api/v1/user/me/provider-keys
|
|
488
490
|
|
|
489
491
|
Agent skill (install the latest director skill so your AI agent can act):
|
|
490
|
-
update-skill
|
|
491
|
-
|
|
492
|
-
|
|
493
|
-
--
|
|
494
|
-
--print Print the
|
|
492
|
+
update-skill DEPRECATED installer alias; installs the canonical
|
|
493
|
+
multi-file vidfarm-director pack via \`skills add\`.
|
|
494
|
+
Use \`vidfarm skills add vidfarm-director\` directly.
|
|
495
|
+
--dir <path> Project root for the canonical pack install
|
|
496
|
+
--print Print the generated single-file director rollup
|
|
497
|
+
(copy/share artifact; not the canonical install)
|
|
495
498
|
|
|
496
499
|
Files (multi-step flows the devcli handles for you):
|
|
497
500
|
upload <file> Upload to the EPHEMERAL temp store, print URL → POST /api/v1/user/me/temporary-files/upload
|
|
@@ -2184,14 +2187,14 @@ async function runTemplateCommand(argv) {
|
|
|
2184
2187
|
}
|
|
2185
2188
|
}
|
|
2186
2189
|
// ── Discover & inspiration ──────────────────────────────────────────────────
|
|
2187
|
-
// `discover` lists TEMPLATES; `--query`
|
|
2190
|
+
// `discover` lists TEMPLATES; `--query` hybrid-searches the catalog so an agent
|
|
2188
2191
|
// CLI (Claude Code / Codex) can answer "which templates suit my <offer>?". Each
|
|
2189
2192
|
// result carries decompose-derived promotions/keywords/summary to reason over.
|
|
2190
2193
|
async function runDiscoverCommand(argv) {
|
|
2191
|
-
const parsed = parseArgs({ args: argv, allowPositionals: true, options: { ...commonOptions(), limit: { type: "string" }, query: { type: "string" } } });
|
|
2194
|
+
const parsed = parseArgs({ args: argv, allowPositionals: true, options: { ...commonOptions(), limit: { type: "string" }, query: { type: "string" }, cursor: { type: "string" }, sort: { type: "string" } } });
|
|
2192
2195
|
const ctx = commonContext(parsed.values);
|
|
2193
2196
|
const query = (parsed.values.query ?? parsed.positionals.join(" ").trim()) || undefined;
|
|
2194
|
-
const result = await apiRequest({ method: "GET", host: ctx.host, path: "/discover/feed", auth: ctx.auth, query: { limit: parsed.values.limit, q: query } });
|
|
2197
|
+
const result = await apiRequest({ method: "GET", host: ctx.host, path: "/discover/feed", auth: ctx.auth, query: { limit: parsed.values.limit, q: query, cursor: parsed.values.cursor, sort: parsed.values.sort } });
|
|
2195
2198
|
assertApiOk(result, "discover");
|
|
2196
2199
|
emitResult(result, ctx.json, [["Browse ", discoverFrontendUrl(ctx.host)]]);
|
|
2197
2200
|
}
|
|
@@ -6250,29 +6253,18 @@ async function runUpdateSkillCommand(argv) {
|
|
|
6250
6253
|
}
|
|
6251
6254
|
return;
|
|
6252
6255
|
}
|
|
6253
|
-
|
|
6254
|
-
|
|
6255
|
-
: parsed.values.global
|
|
6256
|
-
? path.join(homedir(), ".claude", "skills")
|
|
6257
|
-
: path.resolve(process.cwd(), ".claude", "skills");
|
|
6258
|
-
const written = [];
|
|
6259
|
-
for (const name of skillNames) {
|
|
6260
|
-
const { contents, source } = await fetchSkillContents(ctx.host, SKILL_TARGETS[name]);
|
|
6261
|
-
const destDir = path.join(skillsRoot, name);
|
|
6262
|
-
mkdirSync(destDir, { recursive: true });
|
|
6263
|
-
const destPath = path.join(destDir, "SKILL.md");
|
|
6264
|
-
writeFileSync(destPath, contents, "utf8");
|
|
6265
|
-
written.push({ skill: name, path: destPath, source, bytes: Buffer.byteLength(contents, "utf8") });
|
|
6266
|
-
}
|
|
6267
|
-
if (ctx.json) {
|
|
6268
|
-
printJson({ ok: true, skillsRoot, installed: written });
|
|
6269
|
-
return;
|
|
6256
|
+
if (parsed.values.global || parsed.values.platform) {
|
|
6257
|
+
throw new Error("update-skill --global/--platform are retired. Use `vidfarm skills add vidfarm-director --dir <project>` for the canonical multi-file pack; `update-skill --print` remains available for the generated copy-friendly rollup.");
|
|
6270
6258
|
}
|
|
6271
|
-
|
|
6272
|
-
|
|
6273
|
-
|
|
6274
|
-
|
|
6275
|
-
|
|
6276
|
-
|
|
6259
|
+
if (!ctx.json) {
|
|
6260
|
+
console.warn(`Deprecated: update-skill now delegates to the canonical multi-file skill installer. Use ${BOLD}vidfarm skills add vidfarm-director${RESET}.`);
|
|
6261
|
+
}
|
|
6262
|
+
await runSkillsCommand([
|
|
6263
|
+
"add",
|
|
6264
|
+
"vidfarm-director",
|
|
6265
|
+
...(parsed.values.dir ? ["--dir", String(parsed.values.dir)] : []),
|
|
6266
|
+
...(parsed.values.host ? ["--host", String(parsed.values.host)] : []),
|
|
6267
|
+
...(ctx.json ? ["--json"] : [])
|
|
6268
|
+
]);
|
|
6277
6269
|
}
|
|
6278
6270
|
//# sourceMappingURL=cli.js.map
|
|
@@ -71,6 +71,7 @@ export class ClipStore {
|
|
|
71
71
|
thumbnail_path TEXT NOT NULL,
|
|
72
72
|
tags_json TEXT NOT NULL,
|
|
73
73
|
description TEXT NOT NULL DEFAULT '',
|
|
74
|
+
embedding_text TEXT,
|
|
74
75
|
taxonomy_version TEXT NOT NULL,
|
|
75
76
|
tier TEXT NOT NULL,
|
|
76
77
|
user_notes TEXT,
|
|
@@ -96,6 +97,10 @@ export class ClipStore {
|
|
|
96
97
|
scanned_at TEXT NOT NULL
|
|
97
98
|
);
|
|
98
99
|
`);
|
|
100
|
+
const clipColumns = this.db.prepare("PRAGMA table_info(clips)").all();
|
|
101
|
+
if (!clipColumns.some((column) => column.name === "embedding_text")) {
|
|
102
|
+
this.db.exec("ALTER TABLE clips ADD COLUMN embedding_text TEXT");
|
|
103
|
+
}
|
|
99
104
|
}
|
|
100
105
|
seedBuiltinPresets() {
|
|
101
106
|
const stmt = this.db.prepare("INSERT OR REPLACE INTO presets(preset_id, name, description, criteria_json, builtin, created_at) VALUES (?, ?, ?, ?, 1, ?)");
|
|
@@ -109,9 +114,9 @@ export class ClipStore {
|
|
|
109
114
|
this.db
|
|
110
115
|
.prepare(`INSERT OR REPLACE INTO clips
|
|
111
116
|
(clip_id, source_video_id, source_filename, start_time_sec, end_time_sec, duration_sec,
|
|
112
|
-
file_path, thumbnail_path, tags_json, description, taxonomy_version, tier, user_notes, created_at)
|
|
113
|
-
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`)
|
|
114
|
-
.run(c.clip_id, c.source_video_id, c.source_filename, c.start_time_sec, c.end_time_sec, c.duration_sec, c.file_path, c.thumbnail_path, JSON.stringify(c.tags), c.description ?? "", c.taxonomy_version, c.tier, c.user_notes ?? null, c.created_at);
|
|
117
|
+
file_path, thumbnail_path, tags_json, description, embedding_text, taxonomy_version, tier, user_notes, created_at)
|
|
118
|
+
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`)
|
|
119
|
+
.run(c.clip_id, c.source_video_id, c.source_filename, c.start_time_sec, c.end_time_sec, c.duration_sec, c.file_path, c.thumbnail_path, JSON.stringify(c.tags), c.description ?? "", c.embedding_text ?? null, c.taxonomy_version, c.tier, c.user_notes ?? null, c.created_at);
|
|
115
120
|
if (c.embedding && c.embedding.length === this.dim) {
|
|
116
121
|
this.db.prepare("DELETE FROM vec_clips WHERE clip_id = ?").run(c.clip_id);
|
|
117
122
|
this.db
|
|
@@ -331,6 +336,7 @@ function rowToClip(row) {
|
|
|
331
336
|
thumbnail_path: row.thumbnail_path,
|
|
332
337
|
tags: safeParse(row.tags_json, emptyTags()),
|
|
333
338
|
description: row.description ?? "",
|
|
339
|
+
embedding_text: row.embedding_text ?? undefined,
|
|
334
340
|
taxonomy_version: row.taxonomy_version,
|
|
335
341
|
tier: row.tier ?? "flash-lite",
|
|
336
342
|
user_notes: row.user_notes ?? undefined,
|
package/dist/src/devcli/clips.js
CHANGED
|
@@ -12,7 +12,7 @@ import { homedir } from "node:os";
|
|
|
12
12
|
import { parseArgs } from "node:util";
|
|
13
13
|
import { createInterface } from "node:readline/promises";
|
|
14
14
|
import { createIdV7 } from "../lib/ids.js";
|
|
15
|
-
import { buildEffectiveGuidance, buildPrivateUserNotesGuidance, ClipModelClient, DEFAULT_CLIP_ASPECT, DEFAULT_CLIP_DURATION_BAND, detectLocalAgent, detectScenes, effectiveHuntDurationSec, estimateScanCostFromScenes,
|
|
15
|
+
import { buildEffectiveGuidance, buildPrivateUserNotesGuidance, ClipModelClient, DEFAULT_CLIP_ASPECT, DEFAULT_CLIP_DURATION_BAND, detectLocalAgent, detectScenes, effectiveHuntDurationSec, estimateScanCostFromScenes, formatCostEstimate, hasFfmpeg, LocalAgentClipClient, normalizeAspect, normalizeCropFocus, normalizeWindows, parseClipHuntPrompt, parseTimeRanges, probeVideo, resolveDurationBand, resolveTargetClipCount, scanVideo, searchClips } from "../services/clip-curation/index.js";
|
|
16
16
|
import { ClipStore } from "./clip-store.js";
|
|
17
17
|
export const CLIPS_HELP = `vidfarm raws — build & search a local raws library (local ffmpeg + local agent by default)
|
|
18
18
|
|
|
@@ -269,8 +269,7 @@ async function runScan(argv) {
|
|
|
269
269
|
console.log(`[raws] preferring scenes without on-screen text/captions (selection only — GhostCut is never run on the source).`);
|
|
270
270
|
console.log(`[raws] aiming for ~${targetClipCount} distinct raw${targetClipCount === 1 ? "" : "s"}${maxClips ? " (--max-clips)" : " (~10 / 10 min)"}.`);
|
|
271
271
|
console.log(`[raws] detecting scenes …`);
|
|
272
|
-
const
|
|
273
|
-
const scenes = fitScenesToDurationBand(detected, durationBand);
|
|
272
|
+
const scenes = await detectScenes(videoPath, { durationSec: probe.duration_sec, windows, ...sceneOptions });
|
|
274
273
|
if (scenes.length === 0)
|
|
275
274
|
throw new Error("No scenes detected (is the video readable / non-empty?).");
|
|
276
275
|
const store = new ClipStore(values.home);
|
|
@@ -130,7 +130,11 @@ export async function detectScenes(videoPath, opts = {}) {
|
|
|
130
130
|
}
|
|
131
131
|
async function detectScenesInSpan(videoPath, opts) {
|
|
132
132
|
const threshold = opts.threshold ?? 0.3;
|
|
133
|
-
|
|
133
|
+
// Preserve very short detected shots as independent candidates. Absorbing a
|
|
134
|
+
// 2-10 frame flash into its neighbour makes it impossible for prompt/OCR
|
|
135
|
+
// refinement to reject that flash later. 50ms only de-duplicates effectively
|
|
136
|
+
// identical detector timestamps.
|
|
137
|
+
const minScene = opts.minSceneSec ?? 0.05;
|
|
134
138
|
const maxScene = opts.maxSceneSec ?? 8;
|
|
135
139
|
const { spanStart, spanEnd } = opts;
|
|
136
140
|
const spanDuration = spanEnd - spanStart;
|
|
@@ -14,10 +14,10 @@ import { extractKeyframes } from "./ffmpeg.js";
|
|
|
14
14
|
const MAX_SCENES_PER_BATCH = 60;
|
|
15
15
|
// Bound total cost on very long sources: past this many batches we keep the raw
|
|
16
16
|
// segmentation for the whole video (logged) instead of firing dozens of calls.
|
|
17
|
-
const MAX_REFINE_BATCHES =
|
|
17
|
+
const MAX_REFINE_BATCHES = 30;
|
|
18
18
|
export async function refineScenesWithGuidance(input) {
|
|
19
19
|
const { client, videoPath, scenes, guidancePrompt } = input;
|
|
20
|
-
if (!guidancePrompt.trim() || scenes.length
|
|
20
|
+
if (!guidancePrompt.trim() || scenes.length === 0) {
|
|
21
21
|
return { scenes, refined: false, kept: scenes.length, dropped: 0, merged: 0 };
|
|
22
22
|
}
|
|
23
23
|
const batchCount = Math.ceil(scenes.length / MAX_SCENES_PER_BATCH);
|
|
@@ -60,6 +60,8 @@ export async function refineScenesWithGuidance(input) {
|
|
|
60
60
|
? Math.max(1, Math.round((input.targetClipCount * batch.length) / scenes.length))
|
|
61
61
|
: undefined;
|
|
62
62
|
const batchDecisions = await askForDecisions(client, guidancePrompt, batch, frames, batchTarget);
|
|
63
|
+
if (!batchDecisions)
|
|
64
|
+
throw new Error("model returned no refinement plan");
|
|
63
65
|
for (let i = 0; i < batch.length; i++) {
|
|
64
66
|
decisions.push(batchDecisions?.[i] ?? keepDefault());
|
|
65
67
|
}
|
|
@@ -111,7 +113,7 @@ async function askForDecisions(client, guidancePrompt, scenes, frames, targetCli
|
|
|
111
113
|
maxOutputTokens: Math.max(1000, scenes.length * 40)
|
|
112
114
|
});
|
|
113
115
|
}
|
|
114
|
-
function rebuildScenes(scenes, decisions) {
|
|
116
|
+
export function rebuildScenes(scenes, decisions) {
|
|
115
117
|
const out = [];
|
|
116
118
|
let dropped = 0;
|
|
117
119
|
let merged = 0;
|
|
@@ -122,19 +124,25 @@ function rebuildScenes(scenes, decisions) {
|
|
|
122
124
|
dropped++;
|
|
123
125
|
continue;
|
|
124
126
|
}
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
127
|
+
const previous = out[out.length - 1];
|
|
128
|
+
const isAdjacent = previous
|
|
129
|
+
? Math.abs(previous.end_time_sec - scenes[i].start_time_sec) <= 0.05
|
|
130
|
+
: false;
|
|
131
|
+
if (d.merge_with_previous && previous && isAdjacent) {
|
|
132
|
+
// Extend only an ADJACENT previous kept clip. A dropped scene creates a
|
|
133
|
+
// real timeline gap; bridging that gap would silently put the rejected
|
|
134
|
+
// footage back into the exported raw.
|
|
135
|
+
previous.end_time_sec = scenes[i].end_time_sec;
|
|
136
|
+
previous.duration_sec = round3(previous.end_time_sec - previous.start_time_sec);
|
|
130
137
|
merged++;
|
|
131
138
|
continue;
|
|
132
139
|
}
|
|
133
140
|
out.push({ ...scenes[i], index: out.length });
|
|
134
141
|
}
|
|
135
|
-
//
|
|
142
|
+
// An empty plan is a valid strict-prompt result. Silently restoring every
|
|
143
|
+
// scene here would completely ignore the user's request.
|
|
136
144
|
if (out.length === 0) {
|
|
137
|
-
return { scenes, refined:
|
|
145
|
+
return { scenes: [], refined: true, kept: 0, dropped, merged };
|
|
138
146
|
}
|
|
139
147
|
return { scenes: out, refined: true, kept: out.length, dropped, merged };
|
|
140
148
|
}
|
|
@@ -76,10 +76,10 @@ export async function processSceneToClip(input) {
|
|
|
76
76
|
await extractThumbnail({ videoPath, scene, outPath: thumbLocal, cropFilter });
|
|
77
77
|
}
|
|
78
78
|
// 5. Embed the document text for semantic search (skipped when the client can't embed).
|
|
79
|
+
const embeddingText = buildClipEmbeddingText(tags, tagResult.description, ctx.userNotes);
|
|
79
80
|
let embedding;
|
|
80
81
|
if (ctx.client.canEmbed) {
|
|
81
|
-
|
|
82
|
-
[embedding] = await ctx.client.embedDocuments([embedText]);
|
|
82
|
+
[embedding] = await ctx.client.embedDocuments([embeddingText]);
|
|
83
83
|
}
|
|
84
84
|
const clip = {
|
|
85
85
|
clip_id: clipId,
|
|
@@ -92,6 +92,7 @@ export async function processSceneToClip(input) {
|
|
|
92
92
|
thumbnail_path: ctx.thumbStoredPath ? ctx.thumbStoredPath(clipId, thumbLocal) : thumbLocal,
|
|
93
93
|
tags,
|
|
94
94
|
description: tagResult.description,
|
|
95
|
+
embedding_text: embeddingText,
|
|
95
96
|
embedding,
|
|
96
97
|
taxonomy_version: ctx.client.taxonomyVersion ?? ACTIVE_TAXONOMY_VERSION,
|
|
97
98
|
tier: ctx.client.tier,
|
|
@@ -113,21 +114,15 @@ export async function scanVideo(opts) {
|
|
|
113
114
|
const probe = opts.probe ?? (await probeVideo(opts.videoPath));
|
|
114
115
|
const windows = normalizeWindows(opts.windows, probe.duration_sec);
|
|
115
116
|
const band = opts.durationBand ?? null;
|
|
116
|
-
//
|
|
117
|
-
//
|
|
118
|
-
//
|
|
119
|
-
// them — the fit is intentionally NOT re-run on supplied scenes since
|
|
120
|
-
// fitting is not idempotent (a fitted 20s scene could re-merge toward the
|
|
121
|
-
// target on a second pass).
|
|
117
|
+
// Keep detected shots separate until AFTER prompt refinement. This is
|
|
118
|
+
// important: judging a multi-shot duration span from one midpoint frame can
|
|
119
|
+
// approve unrelated opening/closing footage or miss text overlays in them.
|
|
122
120
|
let scenes = opts.scenes ??
|
|
123
|
-
|
|
121
|
+
await detectScenes(opts.videoPath, {
|
|
124
122
|
durationSec: probe.duration_sec,
|
|
125
123
|
windows,
|
|
126
|
-
// A duration band lifts the scene-split ceiling so raw cuts can merge
|
|
127
|
-
// up to the requested clip length instead of being pre-chopped at 8s.
|
|
128
|
-
...(band ? { maxSceneSec: band.max_sec } : {}),
|
|
129
124
|
...opts.sceneOptions
|
|
130
|
-
})
|
|
125
|
+
});
|
|
131
126
|
// Prompt-guided segmentation: let the guidance reshape which scenes become
|
|
132
127
|
// clips (keep/drop/merge/dedup) before we spend tagging tokens on them.
|
|
133
128
|
if (opts.ctx.guidancePrompt?.trim() && opts.ctx.refineSegmentation !== false) {
|
|
@@ -139,11 +134,15 @@ export async function scanVideo(opts) {
|
|
|
139
134
|
workDir: opts.ctx.workDir,
|
|
140
135
|
targetClipCount: opts.targetClipCount
|
|
141
136
|
});
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
137
|
+
if (!refined.refined && refined.note) {
|
|
138
|
+
throw new Error(`Prompt-guided raw selection failed: ${refined.note}`);
|
|
139
|
+
}
|
|
140
|
+
scenes = refined.scenes;
|
|
145
141
|
opts.onRefine?.(refined);
|
|
146
142
|
}
|
|
143
|
+
// Assemble prompt-approved, contiguous shots into useful clip durations.
|
|
144
|
+
// fitScenesToDurationBand never crosses a dropped gap.
|
|
145
|
+
scenes = fitScenesToDurationBand(scenes, band);
|
|
147
146
|
// Deterministic count/uniqueness cap: if more scenes survived than the target
|
|
148
147
|
// (refine skipped >60 scenes, or the model kept more than asked), keep an
|
|
149
148
|
// evenly-spaced — hence less duplicative — subset.
|
|
@@ -12,6 +12,9 @@ export class SpeechVoiceMismatchError extends Error {
|
|
|
12
12
|
}
|
|
13
13
|
export class ProviderQuotaExceededError extends Error {
|
|
14
14
|
}
|
|
15
|
+
/** The provider refused the request or output because of its content/safety policy. */
|
|
16
|
+
export class ProviderSafetyError extends Error {
|
|
17
|
+
}
|
|
15
18
|
export class ProviderWaitExceededError extends Error {
|
|
16
19
|
}
|
|
17
20
|
export class ProviderLeaseTimeoutError extends Error {
|
|
@@ -63,6 +66,54 @@ export function isHardQuotaError(details) {
|
|
|
63
66
|
"resource_exhausted"
|
|
64
67
|
].some((pattern) => normalized.includes(pattern));
|
|
65
68
|
}
|
|
69
|
+
export function isSafetyOrRefusalError(details) {
|
|
70
|
+
const normalized = details.toLowerCase();
|
|
71
|
+
return [
|
|
72
|
+
"content_policy_violation",
|
|
73
|
+
"content policy",
|
|
74
|
+
"safety policy",
|
|
75
|
+
"safety filter",
|
|
76
|
+
"safety_reason",
|
|
77
|
+
"finishreason\":\"safety",
|
|
78
|
+
"moderation_blocked",
|
|
79
|
+
"blocked_reason",
|
|
80
|
+
"blockedreason",
|
|
81
|
+
"prompt was blocked",
|
|
82
|
+
"moderation",
|
|
83
|
+
"moderated",
|
|
84
|
+
"responsible ai",
|
|
85
|
+
"refused",
|
|
86
|
+
"refusal",
|
|
87
|
+
"prohibited content",
|
|
88
|
+
"policy violation"
|
|
89
|
+
].some((pattern) => normalized.includes(pattern));
|
|
90
|
+
}
|
|
91
|
+
export function providerFailureFromDetails(provider, operation, status, details) {
|
|
92
|
+
if (isSafetyOrRefusalError(details)) {
|
|
93
|
+
return new ProviderSafetyError(`${provider} refused this ${operation} because its safety or content policy was triggered${conciseProviderDetails(details)}. ` +
|
|
94
|
+
"Try revising the prompt or choose another provider/model.");
|
|
95
|
+
}
|
|
96
|
+
return new Error(`${provider} ${operation} returned ${status}${conciseProviderDetails(details)}`);
|
|
97
|
+
}
|
|
98
|
+
export function serializeProviderFailure(error) {
|
|
99
|
+
const message = error instanceof Error ? error.message : "AI provider request failed";
|
|
100
|
+
if (error instanceof ProviderSafetyError) {
|
|
101
|
+
return { message, type: "provider_safety_refusal", retryable: false, user_action: "Revise the prompt or choose another provider/model." };
|
|
102
|
+
}
|
|
103
|
+
if (error instanceof ProviderRateLimitError) {
|
|
104
|
+
return { message, type: "provider_rate_limited", retryable: true, retry_after_seconds: error.retryAfterSeconds, user_action: "Wait, then retry or choose another provider." };
|
|
105
|
+
}
|
|
106
|
+
if (error instanceof ProviderQuotaExceededError) {
|
|
107
|
+
return { message, type: "provider_quota_exceeded", retryable: false, user_action: "Check provider billing/quota or choose another provider." };
|
|
108
|
+
}
|
|
109
|
+
if (error instanceof ProviderAuthError) {
|
|
110
|
+
return { message, type: "provider_auth_error", retryable: false, user_action: "Check the saved provider API key." };
|
|
111
|
+
}
|
|
112
|
+
if (error instanceof ProviderTransientError) {
|
|
113
|
+
return { message, type: "provider_temporarily_unavailable", retryable: true, retry_after_seconds: error.retryAfterSeconds, user_action: "Retry shortly or choose another provider." };
|
|
114
|
+
}
|
|
115
|
+
return { message, type: "provider_request_failed", retryable: false };
|
|
116
|
+
}
|
|
66
117
|
export function conciseProviderDetails(details) {
|
|
67
118
|
const compact = details.replace(/\s+/g, " ").trim();
|
|
68
119
|
return compact ? `: ${compact.slice(0, 500)}` : "";
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@mevdragon/vidfarm-devcli",
|
|
3
|
-
"version": "0.20.
|
|
3
|
+
"version": "0.20.14",
|
|
4
4
|
"description": "Local bridge for the Vidfarm Trackpad Editor. `vidfarm serve <template_id>` boots the FULL editor on localhost (disk-backed records/storage, free in-process render); edit composition.html on disk (Claude Code, Codex, etc.) and the browser live-morphs it.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
@@ -53,6 +53,7 @@
|
|
|
53
53
|
"dev:cli": "tsx src/cli.ts",
|
|
54
54
|
"build:frontend": "node scripts/build-homepage-client.mjs",
|
|
55
55
|
"build:hyperframes-editor": "cd demo && node scripts/build.mjs",
|
|
56
|
+
"seo:refresh": "tsx scripts/generate-route-seo.ts",
|
|
56
57
|
"operator:template-artifact": "node scripts/platform-operator-template-artifact.mjs",
|
|
57
58
|
"bootstrap:staging-templates": "tsx scripts/bootstrap-staging-template-releases.ts",
|
|
58
59
|
"migrate:sqlite-to-dynamodb": "tsx scripts/migrate-sqlite-to-dynamodb.ts",
|
|
@@ -60,7 +61,10 @@
|
|
|
60
61
|
"rollback:prod-inplace": "bash scripts/rollback-prod-inplace.sh",
|
|
61
62
|
"build": "npm run build:frontend && npm run build:hyperframes-editor && node -e \"require('node:fs').rmSync('dist', { recursive: true, force: true })\" && tsc -p tsconfig.json",
|
|
62
63
|
"start": "node --import ./dist/src/instrument.js --enable-source-maps dist/src/index.js",
|
|
63
|
-
"check": "tsc -p tsconfig.json --noEmit",
|
|
64
|
+
"check": "tsc -p tsconfig.json --noEmit && npm run check:skills",
|
|
65
|
+
"test:clips": "node --import tsx --test test/clip-curation.test.ts",
|
|
66
|
+
"check:skills": "node scripts/build-director-skill-rollup.mjs --check && node scripts/check-skill-routes.mjs",
|
|
67
|
+
"benchmark:editor-chat": "node --import tsx scripts/benchmark-editor-chat-harness.mjs",
|
|
64
68
|
"cdk:deploy:prod-serverless": "npm run build && dotenv -e .env.production -- npx aws-cdk deploy --app 'node dist/infra/cdk/bin/vidfarm-prod.js'",
|
|
65
69
|
"cdk:synth:staging-serverless": "npm run build && dotenv -e .env.staging -- npx aws-cdk synth --app 'node dist/infra/cdk/bin/vidfarm-serverless-staging.js'",
|
|
66
70
|
"cdk:deploy:staging-serverless": "npm run build && dotenv -e .env.staging -- npx aws-cdk deploy --app 'node dist/infra/cdk/bin/vidfarm-serverless-staging.js'",
|