@officexapp/vidfarm-devcli 0.21.24 → 0.21.27
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/skills/vidfarm/SKILL.md +35 -7
- package/.agents/skills/vidfarm/references/assets-and-sourcing.md +26 -0
- package/.agents/skills/vidfarm/references/automation-and-local-dev.md +37 -6
- package/.agents/skills/vidfarm/references/core-workflows.md +1 -1
- package/.agents/skills/vidfarm/references/editor-workflows.md +44 -2
- package/.agents/skills/vidfarm/references/primitives.md +28 -0
- package/.agents/skills/vidfarm-media/SKILL.md +3 -1
- package/.agents/skills/vidfarm-media/references/tts.md +1 -1
- package/SKILL.director.md +171 -16
- package/SKILL.md +14 -2
- package/clipper.md +3 -2
- package/dist/src/cli.js +412 -17
- package/dist/src/devcli/cost-mode.js +66 -26
- package/dist/src/devcli/sequence.js +619 -0
- package/dist/src/services/sequence-prompts.js +463 -0
- package/package.json +4 -2
- package/public/assets/file-directory-app.js +26 -26
- package/public/assets/homepage-client-app.js +12 -12
- package/public/serve-shells/editor.html +66 -3
- package/public/serve-shells/library-files.html +60 -3
- package/public/serve-shells/library-raws.html +60 -3
- package/public/serve-shells/tools-clipper.html +60 -3
- package/public/serve-shells/tools-image.html +1069 -483
- package/public/serve-shells/tools-video.html +140 -14
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: vidfarm
|
|
3
|
-
description: Use Vidfarm as a director. Browse/add inspiration videos, browse
|
|
3
|
+
description: Use Vidfarm as a director. Browse/add inspiration videos, browse the free public raws catalog BY CATEGORY (curated shelves like scroll-stoppers/greenscreen/reaction — the cheapest way to source footage for one video, and a ready-made clip pool for bulk scripting N variants), fork a template into a composition, edit it in the Trackpad Editor (timeline-based like Premiere/DaVinci), auto-decompose source video into scenes, render to MP4, approve into a shareable post, and schedule it. Includes login, provider keys, discovery, versioning, uploads/downloads, and billing. Every step is available as raw REST; `vidfarm-devcli` wraps those routes and composes the file-backed scripting flows.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# Vidfarm Director
|
|
@@ -81,23 +81,49 @@ Do not use this skill to author new templates from scratch, deploy platform infr
|
|
|
81
81
|
|
|
82
82
|
## Cost mode — ask before you spend, then remember it
|
|
83
83
|
|
|
84
|
-
Vidfarm work can burn real AI credits on the user's wallet / provider keys. **Save them money by default.** Before the first billed step of a session, ask the user which spend posture they want, and
|
|
84
|
+
Vidfarm work can burn real AI credits on the user's wallet / provider keys. **Save them money by default.** Before the first billed step of a session, ask the user which spend posture they want, and quote the per-video cost in one plain line each. The first two are about *how much* AI you buy; the last two are two different *production methods* — rich-ai buys reusable greenscreen raws and animates them in HTML, pure-videogen generates every frame:
|
|
85
85
|
|
|
86
|
-
|
|
86
|
+
| Mode | Cost per video | What it means |
|
|
87
|
+
|---|---|---|
|
|
88
|
+
| **minimize** | **$0 — explicitly free** | free local compute + free stock media only; billed AI is refused without `--yes` |
|
|
89
|
+
| **hybrid** *(recommended)* | **~$0.01–$1** | free where it's free; pay AI only where it clearly wins |
|
|
90
|
+
| **rich-ai** | **$1+** | AI video gen mints **reusable greenscreen raws**, then hyperframes HTML/CSS motion remixes them with the rest of the library — the footage is saved and reused later |
|
|
91
|
+
| **pure-videogen** | **$5+** | every shot is generated: text script → image storyboard → frame-by-frame scene generation. No reuse, no HTML motion |
|
|
92
|
+
|
|
93
|
+
**Every dollar above is billed to the user's OWN AI provider keys (BYOK)** — the keys they save via `vidfarm add-provider-key <provider> <key>` or **Settings → Bring your own keys** (<https://vidfarm.cc/settings/developer>). Vidfarm isn't marking it up; the model providers charge it directly. (Vidfarm wallet credits are only spent when the user deliberately runs on the platform key instead of their own.) Say this out loud the first time you ask — it's why `minimize` really is $0: with no key spend, nothing can be charged.
|
|
94
|
+
|
|
95
|
+
- **minimize** — **$0 videos.** Stay on FREE local compute wherever possible (local render, local TTS — `vidfarm tts` already defaults to the free local Kokoro voice in this mode — `stt --engine whisper`, `remove-greenscreen --local`, reused raw clips + HTML hyperframes). For assets, reach for the **free stock catalog** before paying to generate anything — `vidfarm media search "<meaning>" --type <bgm|sfx|image|vector|icon|video>` pulls royalty-free, commercial-safe music/sound-effects/images/icons/stock-video from pixabay/openverse/iconify at $0 instead of billing AI music/image generation (see the `vidfarm-media` skill). No surprise AI spend.
|
|
87
96
|
- **Check the keyless sources first — Openverse and iconify.** Openverse (CC/CC0 **music, SFX, and images**) and iconify (**icons**) need **no account or key at all**, so they always work in `minimize` mode. Prefer them for BGM, sound effects, icons, and CC imagery before anything else.
|
|
88
97
|
- **Pixabay key** unlocks the photos/vectors/stock-video slots (music/SFX/icons/CC images are keyless). It's a **free** stock-media key, not an AI key. Don't assume it's missing when a search comes up short — it **may already be saved**: check `vidfarm provider-keys` (or the web app's **Settings → Bring your own keys** / <https://vidfarm.cc/settings/developer>). If it isn't, the user grabs a free one at <https://pixabay.com/api/docs/> and saves it once — `vidfarm add-provider-key pixabay <key>`, the Settings surface, or by handing the key to their desktop AI agent to run that command. After it's saved, cost-mode `minimize` sourcing works end-to-end at $0.
|
|
89
98
|
- **You can still get CUSTOM art in `minimize` — hand the prompt to the user and let a free image generator do it.** Stock and `mask` only cover art that already exists somewhere; when the video genuinely needs a bespoke graphic, **don't conclude "we can't" and don't quietly bill `generate`**. Write the prompt and ask the user to paste it into a **free** image generator — <https://meta.ai>, free-tier ChatGPT, or a free Hugging Face image Space (<https://huggingface.co/spaces>) — then hand the PNG back with `vidfarm put-file` (or drag it into **My Files** in the web app). $0, zero wallet spend. Full loop + the prompt template: **“Free manual image-gen”** below.
|
|
90
|
-
- **hybrid** *(recommend this)* —
|
|
99
|
+
- **hybrid** *(recommend this)* — **~$0.01–$1 per video, on their BYOK key.** Free where it's free; pay for AI only where it clearly wins (a hero shot, a voice you can't fake locally). A mostly-hyperframes video with one generated image lands near the low end; a few AI images plus premium narration approaches the high end.
|
|
91
100
|
- **Narration starts free here too.** In BOTH `minimize` and `hybrid`, a bare `vidfarm tts "…"` runs the keyless local Kokoro-82M voice — no flag needed. Ask for a premium voice explicitly (`--style`, `--voice <provider preset>`, `--provider`, `--own-key`, or `--cloud`) when the piece needs one, and say so out loud so the user knows what they're paying for. `rich-ai` skips the free default entirely. If the local engine isn't installed on their machine, the run falls back to their provider key / cloud automatically and prints why.
|
|
92
101
|
- **Free-TTS draft, premium-TTS final — now the default shape, so SAY it out loud.** Narration gets rewritten several times before the script locks, and paying for premium ElevenLabs TTS on every draft wastes money re-voicing lines that will change. The CLI defaults to the free local voice in `hybrid`, so drafts are free automatically — but a robotic draft voice reads as "broken" to a user who wasn't told. **Tell them the first time you generate narration**: the draft voice is the free local one, and you'll re-voice the approved final script in premium TTS (`--cloud` ElevenLabs, or `--style`/`--voice` on their own key) right before the last render. If they'd rather hear the real voice the whole way, pass the premium flags from the start — and if the piece is voice-led (a testimonial, a brand spot where the voice IS the product), offer premium up front instead of drafting free.
|
|
93
|
-
- **rich-ai** —
|
|
102
|
+
- **rich-ai** — **$1+ per video, on their BYOK key.** AI video generation is used, but **not to generate the finished video** — it's used to mint **reusable greenscreen raws**, which are keyed once and then animated/remixed in HTML. The shape of a rich-ai video:
|
|
103
|
+
1. **Generate the cast and props as greenscreen raws.** `vidfarm avatar "<who>" --say "<line>"` for a talking presenter, `vidfarm create-overlay "<subject>"` (or the free-keying `vidfarm cutout --generate "<prompt>"`) for illustrations and props, and `vidfarm generate video --prompt "<subject> … on a flat solid #00FF00 background"` for motion elements. Everything is generated on greenscreen *on purpose*, so it composites over any background later.
|
|
104
|
+
2. **Key it once, keep the plate.** The primitives key the plate in the same job (transparent WebM with audio for avatars, transparent PNG/WebP for overlays) and hand back `greenscreen_source_url` too — so you can re-key at a different tolerance without paying for a second generation. `--local` keys on the user's machine with free ffmpeg.
|
|
105
|
+
3. **Animate and remix in hyperframes, not in the generator.** Motion, camera moves, transitions, text, and staging are **HTML/CSS + hyperframes keyframes over the keyed raws**, combined with existing library raws and public raws. Generated seconds are expensive; HTML motion is free — so buy *assets* with AI and get *motion* for $0.
|
|
106
|
+
4. **SAVE every generated raw so the spend amortizes — this is the whole point of the mode.** A generated asset that only exists inside one composition was rented; one that's in the library was *bought*. Persist it the moment the job settles:
|
|
107
|
+
- **Foldered + annotated in the library** — `vidfarm put-file ./zara-keyed.webm --folder greenscreen/zara --notes "Zara, teal jacket, transparent WebM w/ audio, keyed from the green plate — reuse as the presenter in any 9:16 spot"`. The notes are vector-embedded, so `vidfarm files --search "presenter in a teal jacket"` finds it next month. Keep the un-keyed `greenscreen_source_url` plate too — re-keying it later is free, re-generating is not.
|
|
108
|
+
- **Footage-shaped raws** can go into the `/raws` library instead (`vidfarm clipper ./generated.mp4 --folder greenscreen-cast --name "shrug"`), where `vidfarm raws search "<meaning>"` / `raws list` / `raws export <ids…> --to ./dir` reach them.
|
|
109
|
+
- **Local by default; cloud when they want it.** Both stores live on the user's own machine under `~/.vidfarm` — free, offline, no account needed. If the user is on a **paid/Pro Vidfarm plan** and wants them everywhere (second machine, teammates, the web editor), mirror with **`vidfarm sync push /files`** and **`vidfarm sync push /raws`** (`sync pull` on the other side; `--dry-run` first). Offer the sync; don't assume it — a free-plan user keeps everything locally and loses nothing.
|
|
110
|
+
- **Say this out loud when you propose rich-ai**: the $1+ is a one-time purchase of a reusable cast/prop library, and the *next* video that reuses those raws can run in `hybrid` or even `minimize` for ~$0.
|
|
111
|
+
- Practical rule: before generating anything in rich-ai, **search what already exists** — `vidfarm raws search "<meaning>"` (their own library, free) then `vidfarm public-raws --category greenscreen --query "<meaning>"` (the free public shelf). Only generate the raws that genuinely don't exist yet.
|
|
112
|
+
- When a run is likely to go well past $1, say the number before you spend it.
|
|
113
|
+
- **pure-videogen** — **$5+ per video, on their BYOK key. The most expensive mode — never enter it by accident.** Here the *entire* video is generated footage: no raw reuse, no HTML motion, no compositing tricks. It buys cinematic continuity that keyed-raws-plus-HTML can't fake. Do it in this order — the planning is what keeps the cost from doubling:
|
|
114
|
+
1. **Script it in TEXT first.** Write the whole piece as a beat-by-beat script (shot list, on-screen action, dialogue/narration, duration per scene) and get the user to approve it *before* any generation call. Text iterations are free; regenerating a scene because the script changed is not.
|
|
115
|
+
2. **Storyboard it as IMAGES.** Generate a still keyframe per scene with `vidfarm generate image …` (cheap — cents, not dollars) and approve the look, cast, framing, and continuity on the stills. Lock characters with a reference/sprite card (`--ref`) so faces and wardrobe survive across scenes. Fix the storyboard until it's right; this is where art direction happens.
|
|
116
|
+
3. **Then generate the scenes frame-by-frame from those keyframes.** Drive each video generation off its approved storyboard still (start frame, and the previous scene's last frame as the next scene's start frame for continuity) — `frame_images` / `--ref` on the video primitives. Generate scene by scene and review each before generating the next; do not fan out the whole film in one shot.
|
|
117
|
+
4. **Assemble, don't re-render.** Cut the approved scenes onto the timeline and do captions, audio, and pacing in hyperframes/the editor as usual — never regenerate footage to fix a caption or a trim.
|
|
118
|
+
- Quote the real number before starting: N scenes × the provider's per-second video price, plus storyboard images. If the piece would survive as rich-ai (greenscreen cast on HTML backgrounds), **say so and offer that first** — it is typically 5× cheaper.
|
|
119
|
+
- **Don't hand-roll steps 1–4 — `vidfarm sequence` is that pipeline.** `vidfarm sequence init "<brief>" --dir ./x` scaffolds a plan (+ a `PLAN_TASK.md` you can direct yourself or hand to a coding agent), then `characters` → `boards` → `prompts` → `animate` → `assemble`, all resumable off one `sequence.json`. `vidfarm sequence run` walks it with a confirm gate before each spend; `--yes` runs it unattended. It writes the character-card and storyboard prompts in a block grammar that states reference priority (card = identity, board = staging) and forbids animating the board sheet itself — the two failure modes that otherwise eat a video credit each. Preview every prompt for $0 with `vidfarm sequence prompts --print board:part-01 | shot:part-01 | card:<slug>`. See `references/editor-workflows.md` → *`vidfarm sequence`*.
|
|
94
120
|
|
|
95
121
|
Then **ask whether to remember the choice**, and save it where it will actually be read next time:
|
|
96
122
|
|
|
97
|
-
- **Local coding agent (devcli):** run `vidfarm cost-mode <choice>` — it persists to `~/.vidfarm/cost-mode.json`, and every billed devcli command then respects it (in `minimize`, billed ops refuse to run without `--yes`; in `hybrid`/`rich-ai` they print each op's cost). Also offer to record it in the agent's own memory so a *future* session recalls it — but which memory differs by agent, so ask, don't assume: Claude Code → `CLAUDE.md` (or its memory dir); Codex / OpenCode / most others → `AGENTS.md`; or a plain note file the user names. A third-party memory tool the user already uses is fine too. Never write to a memory file the user didn't approve.
|
|
123
|
+
- **Local coding agent (devcli):** run `vidfarm cost-mode <choice>` — it persists to `~/.vidfarm/cost-mode.json`, and every billed devcli command then respects it (in `minimize`, billed ops refuse to run without `--yes`; in `hybrid`/`rich-ai`/`pure-videogen` they print each op's cost). Also offer to record it in the agent's own memory so a *future* session recalls it — but which memory differs by agent, so ask, don't assume: Claude Code → `CLAUDE.md` (or its memory dir); Codex / OpenCode / most others → `AGENTS.md`; or a plain note file the user names. A third-party memory tool the user already uses is fine too. Never write to a memory file the user didn't approve.
|
|
98
124
|
- **Web app UI (the /editor copilot, chat dock, /chat, /library):** there is no agent-memory file to write, so **do not** try to persist it — just ask each time, unless the user tells you their standing preference in this conversation (then honor it for the session).
|
|
99
125
|
|
|
100
|
-
If the user hasn't picked yet and you're about to spend, name the cheaper path and the cost, and ask. `vidfarm cost-mode` with no argument prints the current mode +
|
|
126
|
+
If the user hasn't picked yet and you're about to spend, name the cheaper path and the cost, and ask. `vidfarm cost-mode` with no argument prints the current mode + an explanation of each for you to relay.
|
|
101
127
|
|
|
102
128
|
## Default stance
|
|
103
129
|
|
|
@@ -139,6 +165,8 @@ Directors also accumulate a **reusable media asset library** — logos, stickers
|
|
|
139
165
|
|
|
140
166
|
Present both harnesses to the director, recommend (A) unless they've asked for premium or budget covers it, and explain the tradeoff in these terms. Full methodology: `references/editor-workflows.md` (“The three paintbrushes & two replication harnesses”); cost bands: `references/core-workflows.md` (Cost spectrum).
|
|
141
167
|
|
|
168
|
+
**"Create an avatar" = a TALKING-HEAD VIDEO with audio, keyed off a greenscreen.** When a director asks for an **avatar** — or a spokesperson, presenter, host, UGC creator, talking head — they never mean a still portrait and never a silent clip. Use **`vidfarm avatar "<who they are>" --say "<the line they speak>"`** (`POST /api/v1/primitives/videos/create-avatar`): it generates the talking head **with lip-synced spoken audio** on a **flat greenscreen plate at an exact known key color**, then keys that plate out in the same job, returning a **transparent presenter WebM** (audio intact) that composites over whatever background the scene needs — in the preview and the final render. Pass `--ref` (a headshot, or the character's `character_sprite_card.png`) to lock the likeness across videos, `--aspect-ratio`/`--duration` to fit the slot, and `--place ./dir --at <time>` to drop it straight onto a composition. **Never hand back the green plate as the finished avatar** — the green comes off before it goes on a timeline. `--local` keys the plate on your machine with free ffmpeg (generation still rides the user's own cloud video keys); `--keep-greenscreen` stops at the plate. Details: `references/primitives.md` → "Primitive: talking_avatar". Generation is AI video — the priciest thing here — so in `cost-mode minimize` offer a free talking-head clip (`vidfarm public-raws --category talking-head`) first.
|
|
169
|
+
|
|
142
170
|
**Explainer/cutout videos — the transparent-sticker workflow.** For explainers (a subject "on stage" while labels, arrows, and props pop in around it), the cheap workhorse is a **transparent cutout sticker**: `vidfarm cutout --generate "<subject>"` AI-generates the graphic on a chroma plate, keys it out, **and trims the canvas down to the subject's true min width/height** — one free local ffmpeg step, no mostly-empty PNG to fight with — then `vidfarm place` + `vidfarm keyframes` scale/position and animate it (zoom, grow, shake, drift). It's the same "generate a reusable element once, then reuse it" thrift as the cheap harness, tuned for stickers. Full guided harness: recipe `recipes/cutout-graphics-for-explainers.md`; placement + zoom/grow/shake/move motion recipes: `references/editor-workflows.md` (“Cutout graphics for explainers”).
|
|
143
171
|
|
|
144
172
|
**Explainer house style — the defaults to build with unless told otherwise.** **White background / light mode** (plain white stage, no gradients, no dark mode, no photo backdrop), **kinetic word-by-word captions** in dark ink on the light stage (`vidfarm captions generate --style word-pop --color "#111111" --active-color "#7C3AED" --background-style plain` — skip outlines/shadows, they're only needed over busy footage), and **female TTS narration** (`vidfarm tts --voice coral` on OpenAI — `nova` for energy, `sage` for calm; `Kore`/`Leda` on Gemini; any ElevenLabs voice via `vidfarm voices`). **Keep it clean and simple** — one idea on screen at a time, two or three cutouts per beat, one accent color, one font, lots of white space; remove before you add. **Illustrations default to simplicity**: flat vector, simple shapes, minimal detail, 2–3 flat colors, no baked-in text — simple art keys cleanly, trims tight, and stays on-style across the whole cast. State the defaults once so the director can override any of them. Full detail: recipe `recipes/cutout-graphics-for-explainers.md` (“House style — the explainer defaults”).
|
|
@@ -15,6 +15,30 @@ Both are async primitive jobs — poll `GET /api/v1/primitives/jobs/:jobId` unti
|
|
|
15
15
|
|
|
16
16
|
**Don't reach for this when the real goal is clips.** If the user wants short clips mined out of a long video, `POST /raws/scan` with `source_url` already downloads the source for you as part of the hunt (below) — a separate download call is wasted spend.
|
|
17
17
|
|
|
18
|
+
## Public raws catalog — browse BY CATEGORY (check here first)
|
|
19
|
+
|
|
20
|
+
Before you hunt footage and long before you generate it, check what the platform **already has**. The public raws catalog is a curated, pre-tagged, free-to-use pool of short clips, organised into **categories** (shelves) — and browsing it by category is one of the highest-leverage moves available to you, for two reasons:
|
|
21
|
+
|
|
22
|
+
1. **Making one video** — a category answers "what could go here?" faster than any search string. You don't have to describe the clip you want; you scan a shelf of clips that already fit the job.
|
|
23
|
+
2. **Bulk scripting** — a category IS a ready-made clip pool. Pull one shelf, and you have N interchangeable clips to fan a single composition across N variants (N hooks, N B-roll beds, N reaction cutaways) in one loop. This is the cheapest way to produce volume: no hunt, no downloads, no generation spend, no per-clip AI call.
|
|
24
|
+
|
|
25
|
+
**Always list the shelves before pulling clips** — it's one cheap call, it tells you what exists and how much of it, and it stops you inventing a query for footage that isn't there:
|
|
26
|
+
|
|
27
|
+
```bash
|
|
28
|
+
vidfarm public-raws --categories # shelves + live counts + what each is FOR
|
|
29
|
+
vidfarm public-raws --category scroll-stoppers --limit 40
|
|
30
|
+
vidfarm public-raws --category greenscreen --query "person shrugging"
|
|
31
|
+
vidfarm public-raws --bookmark <raw_id> # save one into /raws/public/* for reuse
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
REST twins: `GET /api/v1/public-raws/categories` (shelves + counts) and `GET /api/v1/public-raws?category=<shelf>&q=…`. Web: `https://vidfarm.cc/discover/raws/categories`.
|
|
35
|
+
|
|
36
|
+
`--category` takes a **curated shelf** (`scroll-stoppers`, `greenscreen`, `reaction`, `talking-head`, `b-roll`, `lifestyle`, `product-shots`, `demos`, `text-graphics`) or an exact taxonomy key (`emotion:confused`, `content_type:reaction`). A clip can sit on more than one shelf — a greenscreen reaction is honestly both — so shelves overlap by design. Don't hardcode the shelf list: read it from `--categories`, which is the live source of truth.
|
|
37
|
+
|
|
38
|
+
What each shelf is for, in practice: **Scroll Stoppers** are half-second pattern-breaks to cut over your first frame so the thumb stops; **Greenscreen** clips key straight onto any background via the greenscreen primitive; **Reaction** clips are the cutaway that tells the viewer how to feel about what you just showed them; **B-Roll** buys you a cut whenever narration has to keep going.
|
|
39
|
+
|
|
40
|
+
A bookmarked public raw lands in the director's own `/raws/public/*` folder, after which it behaves like any other library raw (`set_layer_media` / `vidfarm set-media` to drop it on the timeline).
|
|
41
|
+
|
|
18
42
|
## Raws (long-form → short-form raws)
|
|
19
43
|
|
|
20
44
|
Mine a **long-form** video (podcast, stream VOD, webinar, any YouTube/TikTok/IG/X URL, or an upload) into a library of tagged, searchable **raws**. This is the `/raws` surface — browse it at `https://vidfarm.cc/library/raws` (the Library page's "Approved / Raws" tabs).
|
|
@@ -114,6 +138,8 @@ Recurring characters that must look the same across videos have a **dedicated, b
|
|
|
114
138
|
|
|
115
139
|
**Consistency is then mechanical.** On every generation featuring the character, **pass the sprite card's `viewUrl` as the reference input** — `prompt_attachments` for image generation/edit, `input_references` for `videos/generate` and the editor's `generate_layer` — and lift wording from the manifest/`character_about.md` into the prompt.
|
|
116
140
|
|
|
141
|
+
**"Create an avatar" means a TALKING-HEAD VIDEO, not a portrait.** When a director asks for an avatar — or a spokesperson, presenter, host, UGC creator, talking head — generate a **video of that person speaking, with audio**, via `POST /api/v1/primitives/videos/create-avatar` (devcli: `vidfarm avatar "<who they are>" --say "<line>"`). By default it is shot on a **flat greenscreen plate at an exact key color** and the same job keys that plate out, handing back a **transparent presenter WebM** (audio preserved) you composite over whatever background the scene needs — the green never reaches the timeline. Pass `--ref` / `input_references` with the character's `character_sprite_card.png` (or the director's headshot) so the face stays on-model across videos, and `--say` with the exact line: a mute avatar is a broken deliverable. Full parameters, billing, and the `--local` free-keying path are in `references/primitives.md` → "Primitive: talking_avatar".
|
|
142
|
+
|
|
117
143
|
**Image shorthand — "character card" and "storyboard".** Two words the image generator understands as full layouts (spelled out automatically by the platform; you can also write them out yourself):
|
|
118
144
|
- **"character card"** (a.k.a. *character sheet / model sheet / sprite card*) → a **CHARACTER REFERENCE SHEET** on a neutral background: body shots (full / three-quarter / half / head-and-shoulders), a turnaround (front / side / back), an expression sheet of face close-ups (happy / sad / angry / blank-dazed), and a pose sheet (2–3 action poses), with small labels and the **same face / outfit / art style** across every panel. This is exactly the reusable identity reference above — generate it, then persist it as `character_sprite_card.png`.
|
|
119
145
|
- **"storyboard"** → a sheet of **sequential numbered panels** (a grid of frames), each a still of one shot with a short caption under it (action / camera move / voiceover), consistent characters + setting across panels. A storyboard is **also a style fit-check**, so render the panels in the **same visual style intended for the final video** (photoreal / 3D / anime / cinematic / etc.) — don't force a rough sketch unless no style was given.
|
|
@@ -88,6 +88,25 @@ vidfarm api POST "/api/v1/compositions/${FORK_ID}/render" --data '{"tracer":"scr
|
|
|
88
88
|
|
|
89
89
|
If you need many variants, keep the base fork fixed and fan out by cloning that fork or by reapplying the same edit function to multiple fork ids. Use the raw REST routes directly when you want maximum control; use `vidfarm-devcli` when you want auth, polling, and file helpers without writing the plumbing yourself.
|
|
90
90
|
|
|
91
|
+
**Where the N in "N variants" comes from: a public raws category.** The hard part of bulk scripting is usually not the loop, it's sourcing N distinct clips without N downloads or N generation calls. A curated public-raws shelf hands you that pool in one request — every clip pre-tagged, free, and already hosted, so the loop body is just a media swap:
|
|
92
|
+
|
|
93
|
+
```bash
|
|
94
|
+
# 1. See what shelves exist (never hardcode the list — this is the live vocabulary).
|
|
95
|
+
vidfarm public-raws --categories --json
|
|
96
|
+
|
|
97
|
+
# 2. Pull one shelf as your clip pool.
|
|
98
|
+
vidfarm public-raws --category scroll-stoppers --limit 20 --json > pool.json
|
|
99
|
+
|
|
100
|
+
# 3. Fan the SAME fork edit across the pool — one variant per clip.
|
|
101
|
+
# (jq over pool.json → previewUrl; swap the hook layer; render each.)
|
|
102
|
+
for URL in $(jq -r '.raws[].previewUrl' pool.json); do
|
|
103
|
+
vidfarm set-media ./work --layer hook --src "$URL"
|
|
104
|
+
vidfarm render "$FORK_ID" --dir ./work --tracer "hooks-$(basename "$URL" .mp4)"
|
|
105
|
+
done
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
That is the cheapest volume play available: no hunt, no per-clip AI call, no wallet spend on sourcing. Pick the shelf that matches the beat you're varying — `scroll-stoppers` for hook A/B tests, `reaction` for cutaways, `b-roll` for narration beds, `greenscreen` for keyed foregrounds. See [Public raws catalog](#public-raws-catalog--browse-by-category-check-here-first).
|
|
109
|
+
|
|
91
110
|
## `vidfarm-devcli` — full command surface
|
|
92
111
|
|
|
93
112
|
`@officexapp/vidfarm-devcli` wraps the **entire director REST flow** as CLI commands. It is a thin shell over the REST API, not a second implementation: most named commands map 1:1 to one REST route, and the file-backed commands compose the documented routes for upload/download + render-polling. Auth via `--api-key <key>` or `VIDFARM_API_KEY`.
|
|
@@ -179,7 +198,7 @@ The licensed harness also carries the **generative build workflow** guidance (ch
|
|
|
179
198
|
| `vidfarm schedules <postId>` | `GET /api/v1/approved/posts/:postId/schedules` | browse scheduled posts |
|
|
180
199
|
| `vidfarm login <email>` / `vidfarm verify <email> <code>` | `POST /api/v1/user/request-otp` · `verify-otp` | get an API key |
|
|
181
200
|
| `vidfarm whoami` | `GET /api/v1/user/me` | who am I (also prints the active cost mode) |
|
|
182
|
-
| `vidfarm cost-mode [minimize\|hybrid\|rich-ai]` | (local `~/.vidfarm/cost-mode.json`) | show/set the money-saving preference every billed command respects; no arg = show + explain
|
|
201
|
+
| `vidfarm cost-mode [minimize\|hybrid\|rich-ai\|pure-videogen]` | (local `~/.vidfarm/cost-mode.json`) | show/set the money-saving preference every billed command respects; no arg = show + explain each; `--clear` forgets it |
|
|
183
202
|
| `vidfarm wallet [--job <id>\|--tracer <t>] [--limit <n>]` | `GET /api/v1/user/me/wallet` | cost log: balance + lifetime spend + recent charges. `--job <renderJobId>` prints **what that one video cost** (sums its charges); `--tracer <t>` sums a tracer. Cloud-only; readable on the free plan too (shows $0.00). Aliases: `spend`, `costs` |
|
|
184
203
|
| `vidfarm provider-keys` / `vidfarm add-provider-key <p> <secret>` | `GET`·`POST /api/v1/user/me/provider-keys` | manage AI keys |
|
|
185
204
|
| `vidfarm upload <file> [--folder <path>]` | presign → S3 PUT → finalize (`.../temporary-files/presign` + `.../temporary-files`) | upload → durable URL (ephemeral, 30-day TTL; prefer `--folder temp` for scratch). Goes direct to S3, so large files (up to **200 MB**) bypass the ~6 MB Lambda body limit |
|
|
@@ -215,14 +234,26 @@ The licensed harness also carries the **generative build workflow** guidance (ch
|
|
|
215
234
|
|
|
216
235
|
## Cost mode — the devcli's money-saving guardrail
|
|
217
236
|
|
|
218
|
-
|
|
237
|
+
The four modes, quoted as **cost per finished video**. The first two are spend postures; the last two are also *production methods* — `rich-ai` buys reusable greenscreen raws and animates them in HTML, `pure-videogen` generates every frame:
|
|
238
|
+
|
|
239
|
+
| Mode | Per video | Posture |
|
|
240
|
+
|---|---|---|
|
|
241
|
+
| `minimize` | **$0 — explicitly free** | free local engines + free stock media only; billed ops refuse without `--yes` |
|
|
242
|
+
| `hybrid` *(default recommendation)* | **~$0.01–$1** | free where free, AI where it clearly wins; each billed op prints its cost |
|
|
243
|
+
| `rich-ai` | **$1+** | AI video gen mints **reusable greenscreen raws** (saved to the library, reused later); motion is hyperframes HTML/CSS over the keyed raws |
|
|
244
|
+
| `pure-videogen` | **$5+** | script in text → storyboard in images → frame-by-frame scene generation. No reuse, no HTML motion |
|
|
245
|
+
|
|
246
|
+
**All of it bills to the user's own AI provider keys (BYOK)** — the keys saved with `vidfarm add-provider-key <provider> <key>` or at **Settings → Bring your own keys** (<https://vidfarm.cc/settings/developer>). The model providers charge those keys directly; Vidfarm wallet credits only come into play when the user deliberately runs on the platform key instead of their own. So `minimize` isn't "cheap", it's **zero**: nothing reaches a paid key at all.
|
|
247
|
+
|
|
248
|
+
`vidfarm cost-mode <minimize|hybrid|rich-ai|pure-videogen>` records a single spend preference (in `~/.vidfarm/cost-mode.json`) that every **billed** command honors: `generate`, `music`, `decompose`, `inspiration-decompose`, `create`, `replicate`, `inpaint`, `create-overlay`, `cutout --generate` (only the generation step; a bare `cutout` on an existing graphic is free), and the cloud paths of `render --target cloud`, `tts --cloud`, `stt --cloud`, `remove-greenscreen` (non-`--local`). FREE local engines never gate (`render` local default, `tts --engine local`, `stt --engine whisper`, `remove-greenscreen --local`, `cutout` on a file/url, all the file-editing verbs). `vidfarm media search` (the free stock catalog — pixabay/openverse/iconify music, SFX, images, icons, stock video) is also free and never gates.
|
|
219
249
|
|
|
220
|
-
- **minimize** — a billed op is **refused** unless you add `--yes`; the error names the free local alternative (which now includes the matching `vidfarm media search` for music/SFX/image/video). Use this to guarantee no surprise AI spend. Before paying to generate music, sound effects, or images, try `vidfarm media search "<meaning>" --type <bgm|sfx|image|vector|icon|video>` first — free royalty-free assets instead of a billed `music`/`generate` call. **Check the keyless sources first — Openverse (CC/CC0 music, SFX, images) and iconify (icons) need no account at all**, so they always work in `minimize`. Photos/vectors/stock-video need a **free Pixabay key** that **may already be saved** — check `vidfarm provider-keys` (or web **Settings → Bring your own keys** / <https://vidfarm.cc/settings/developer>) before assuming a short result means "no key." If absent, save one once: `vidfarm add-provider-key pixabay <key>` (free key from <https://pixabay.com/api/docs/>), the Settings surface, or hand it to the desktop AI agent to run that command.
|
|
250
|
+
- **minimize ($0 videos)** — a billed op is **refused** unless you add `--yes`; the error names the free local alternative (which now includes the matching `vidfarm media search` for music/SFX/image/video). Use this to guarantee no surprise AI spend. Before paying to generate music, sound effects, or images, try `vidfarm media search "<meaning>" --type <bgm|sfx|image|vector|icon|video>` first — free royalty-free assets instead of a billed `music`/`generate` call. **Check the keyless sources first — Openverse (CC/CC0 music, SFX, images) and iconify (icons) need no account at all**, so they always work in `minimize`. Photos/vectors/stock-video need a **free Pixabay key** that **may already be saved** — check `vidfarm provider-keys` (or web **Settings → Bring your own keys** / <https://vidfarm.cc/settings/developer>) before assuming a short result means "no key." If absent, save one once: `vidfarm add-provider-key pixabay <key>` (free key from <https://pixabay.com/api/docs/>), the Settings surface, or hand it to the desktop AI agent to run that command.
|
|
221
251
|
- **minimize still gets CUSTOM images — via a free manual generator.** A refused `generate` is not the end of the road. Offer the user the manual loop (ask once, then make it the session default): **you write the prompt → they run it free in <https://meta.ai>, free-tier ChatGPT, or a free Hugging Face image Space (<https://huggingface.co/spaces>) → they hand the PNG back** via `vidfarm put-file ./sheet.png` or web **My Files**. Ask for **one sheet holding every graphic you need**, gridded on a **flat pure-green plate** (`#00FF00`), no text — one round trip instead of N, which saves the user's time and your tokens. Then split it locally for $0: `vidfarm mask ./sheet.png --crop x,y,w,h --flat "#00FF00" --out prop-a.png`, once per element (drop `--flat` and let local ONNX matting handle it if the tool ignored the green background). Full prompt template + loop: recipe `recipes/cutout-graphics-for-explainers.md` (“Free manual image-gen”).
|
|
222
|
-
- **hybrid** *(default recommendation)* — billed ops run but print a one-line cost notice each.
|
|
223
|
-
- **rich-ai** — billed ops run without gating; cost is still printed.
|
|
252
|
+
- **hybrid (~$0.01–$1 per video)** *(default recommendation)* — billed ops run but print a one-line cost notice each, charged to the user's BYOK key.
|
|
253
|
+
- **rich-ai ($1+ per video)** — billed ops run without gating; cost is still printed. AI *video* generation is the line item that pushes a video well past $1 — quote it before running. Spend it on **reusable greenscreen raws**, not on finished shots: `vidfarm avatar "<who>" --say "<line>"` for presenters, `vidfarm create-overlay "<subject>"` / `cutout --generate` for props and illustrations, or `generate video` prompted onto a flat key-color plate. The primitives key the plate in the same job and also hand back `greenscreen_source_url`, so re-keying at a different tolerance is free. Then **animate in hyperframes HTML/CSS over the keyed raws** — generated seconds cost money, motion doesn't — and **persist every asset**: `vidfarm put-file ./keyed.webm --folder greenscreen/<name> --notes "<what it is, when to reuse it>"` (notes are vector-embedded → `vidfarm files --search`), or `vidfarm clipper ./generated.mp4 --folder greenscreen-cast --name "<name>"` for footage-shaped raws in `/raws`. Both stores are **local by default** under `~/.vidfarm`; on a **paid/Pro plan** mirror them with `vidfarm sync push /files` and `vidfarm sync push /raws` (`sync pull` elsewhere, `--dry-run` first). Before generating, always search what already exists — `vidfarm raws search "<meaning>"` then `vidfarm public-raws --category greenscreen --query "<meaning>"`. That is what makes the mode amortize: the next video can reuse the same cast in `hybrid`/`minimize` for ~$0.
|
|
254
|
+
- **pure-videogen ($5+ per video)** — the most expensive mode; billed ops run without gating and a single run can be many dollars, so quote the number (N scenes × per-second video price + storyboard images) before starting. Everything is generated footage — no raw reuse, no HTML motion. Plan first so you don't pay twice: (1) write the whole piece as a **text script** (shot list, action, dialogue, per-scene duration) and get it approved; (2) **storyboard it as images** — one still per scene via `vidfarm generate image` (cents, not dollars), locking cast/wardrobe with `--ref` on a sprite card, and iterate on the stills until the look is right; (3) **generate scenes from those keyframes** — drive each `generate video` off its approved still (`--ref` / `frame_images`, chaining the previous scene's last frame into the next scene's start frame for continuity), one scene at a time with a review between; (4) **assemble on the timeline** — captions, audio, and pacing in hyperframes, never a regeneration to fix a trim. If the piece would survive as `rich-ai` (greenscreen cast over HTML backgrounds), offer that first — typically ~5× cheaper.
|
|
224
255
|
|
|
225
|
-
**Narration defaults to the FREE local voice in minimize AND hybrid.** A bare `vidfarm tts "…"` runs the keyless local Kokoro-82M engine in both of those modes — you no longer have to remember `--engine local`. A run **opts out** of that default by asking for a premium voice (`--style`, `--provider`, `--model`, `--own-key`, or a non-Kokoro `--voice` like `alloy`/`Kore`/an ElevenLabs id), by passing `--cloud`/`--engine byok`, or by being in `rich-ai`. If the local engine isn't installed on the machine (it needs `pip install kokoro-onnx soundfile` + a ~340MB model on first use), the run **falls back** to the user's provider key / cloud instead of failing — it prints the reason on stderr so you can tell the user why the voice changed.
|
|
256
|
+
**Narration defaults to the FREE local voice in minimize AND hybrid.** A bare `vidfarm tts "…"` runs the keyless local Kokoro-82M engine in both of those modes — you no longer have to remember `--engine local`. A run **opts out** of that default by asking for a premium voice (`--style`, `--provider`, `--model`, `--own-key`, or a non-Kokoro `--voice` like `alloy`/`Kore`/an ElevenLabs id), by passing `--cloud`/`--engine byok`, or by being in `rich-ai`/`pure-videogen`. If the local engine isn't installed on the machine (it needs `pip install kokoro-onnx soundfile` + a ~340MB model on first use), the run **falls back** to the user's provider key / cloud instead of failing — it prints the reason on stderr so you can tell the user why the voice changed.
|
|
226
257
|
|
|
227
258
|
Precedence: `--cost-mode <m>` flag → `VIDFARM_COST_MODE` env → the saved `cost-mode` → default (hybrid, flagged as "not set"). When nothing is saved and a billed op runs, the CLI prints a "no preference set — ask the user" nudge instead of silently spending, so the default posture really is *ask before you spend*.
|
|
228
259
|
|
|
@@ -73,7 +73,7 @@ To bring a new viral video into the catalog as a **private** template you own, i
|
|
|
73
73
|
- `DELETE /discover/templates/:entryId` — remove a private inspiration/template you own (accepts either the `inspiration_...` or minted `template_...` id).
|
|
74
74
|
- `POST /api/v1/inspirations/:inspirationId/decompose { user_prompt? }` — AI-decompose an inspiration's downloaded video into scenes (requires a saved provider key; same 120s source cap as auto-decompose).
|
|
75
75
|
|
|
76
|
-
devcli: `vidfarm inspiration-add <url|file.mp4>` (a local file path runs the presign→PUT→finalize flow for you), `vidfarm inspiration-rm <id>`, `vidfarm inspiration-decompose <id>`, `vidfarm discover [query] [--sort relevance|wow|automation|recent] [--cursor <next_cursor>]` to browse/search templates, `vidfarm public-raws [query] [--category <key>] [--type <raw_type>] [--bookmark <raw_id>]` to browse/save the public raws catalog, and `vidfarm videos [query] [--mine]` to browse/search source inspirations. On the web, the Discover page's **Add Template** modal accepts either a URL or a file upload.
|
|
76
|
+
devcli: `vidfarm inspiration-add <url|file.mp4>` (a local file path runs the presign→PUT→finalize flow for you), `vidfarm inspiration-rm <id>`, `vidfarm inspiration-decompose <id>`, `vidfarm discover [query] [--sort relevance|wow|automation|recent] [--cursor <next_cursor>]` to browse/search templates, `vidfarm public-raws [--categories | query] [--category <shelf|key>] [--type <raw_type>] [--bookmark <raw_id>]` to browse/save the public raws catalog — **start with `--categories`** to see the shelves and their counts, then pull one with `--category <shelf>`; a shelf doubles as a ready-made clip pool for bulk scripting (see [Public raws catalog](#public-raws-catalog--browse-by-category-check-here-first)), and `vidfarm videos [query] [--mine]` to browse/search source inspirations. On the web, the Discover page's **Add Template** modal accepts either a URL or a file upload.
|
|
77
77
|
|
|
78
78
|
## Fork a template
|
|
79
79
|
|
|
@@ -7,7 +7,7 @@ Almost every editor session is a director taking a template / fork / project and
|
|
|
7
7
|
|
|
8
8
|
Name the plan back in these terms ("I'll SWAP the captions and REPLACE the scenes"), then execute axis by axis. Be **proactive at the heavy end** — carry the whole transformation, don't wait to be micro-managed one layer at a time.
|
|
9
9
|
|
|
10
|
-
**Fuel a scene REPLACE with raw clips, not expensive AI video.** A heavy scenes-axis replace needs footage; sources in cost order: (1) the director's own library — search `/raws` and `/files` (`vidfarm raws search …`, `vidfarm files --search …` / `browse_files`); (2) **HUNT new raws** out of a long-form source (podcast/VOD/webinar or any YouTube/TikTok/IG/X URL) — `vidfarm raws scan <src> --prompt "<what the new scenes need>" --aspect <canvas> [--duration N --no-text --range …]` (local-first, free compute) or the async `POST /clips/scan` / `/raws/scan`; then reuse the picks (`set_layer_media` / `vidfarm set-media` to swap in place, `add_layer`/`vidfarm place` for net-new scenes); (3) `generate_layer` / `vidfarm generate` AI generation — the **expensive last resort**, only for scenes no real clip can cover. When a big scene re-work is asked for but no footage is given, **ask for a source to hunt (or point at the raws library) before AI-generating** — see [Raws](#raws-long-form--short-form-raws) and [Generate AI media …](#generate-ai-media-and-drop-it-on-the-timeline).
|
|
10
|
+
**Fuel a scene REPLACE with raw clips, not expensive AI video.** A heavy scenes-axis replace needs footage; sources in cost order: (1) the director's own library — search `/raws` and `/files` (`vidfarm raws search …`, `vidfarm files --search …` / `browse_files`); (1b) the **PUBLIC RAWS catalog, browsed by category** — free, pre-tagged, already on the platform, and no download or hunt required (`vidfarm public-raws --categories` to see the shelves, then `--category <shelf>`); (2) **HUNT new raws** out of a long-form source (podcast/VOD/webinar or any YouTube/TikTok/IG/X URL) — `vidfarm raws scan <src> --prompt "<what the new scenes need>" --aspect <canvas> [--duration N --no-text --range …]` (local-first, free compute) or the async `POST /clips/scan` / `/raws/scan`; then reuse the picks (`set_layer_media` / `vidfarm set-media` to swap in place, `add_layer`/`vidfarm place` for net-new scenes); (3) `generate_layer` / `vidfarm generate` AI generation — the **expensive last resort**, only for scenes no real clip can cover. When a big scene re-work is asked for but no footage is given, **ask for a source to hunt (or point at the raws library) before AI-generating** — see [Raws](#raws-long-form--short-form-raws) and [Generate AI media …](#generate-ai-media-and-drop-it-on-the-timeline).
|
|
11
11
|
|
|
12
12
|
**Audio is natively multi-track — overlay narration + music + SFX, each at its own volume.** A composition mixes UNLIMITED simultaneous `<audio>` layers; each sits on its own `data-track-index` and carries its own `data-volume` (0–2, default 1), and the runtime mixes them with per-track volume honored identically in the preview and the exported MP4 (a real ffmpeg `amix` of every audio layer at render). So you never need a pre-mixed file — lay **narration/voiceover at ~1.0 on one track, a music bed at ~0.1–0.2 on a separate track, and SFX on their own tracks**, each via `add_layer kind=audio` (web) / `vidfarm place --kind audio --volume …` (devcli), tuning levels later with the Inspector's Volume slider or `set_layer_media` (`volume`, `muted`). **The key move when recreating a template whose original baked music + narration into ONE audio track: rebuild it as TWO independent tracks** — a fresh narration track (`/audio/speech`, or same-voice reword via `/audio/regenerate-speech` / `vidfarm speech regenerate`) at ~1.0 and a separate real music track at ~0.1–0.2 — then mute/remove the original combined source-audio layer. This gives the director independent voice and music volume, and works around AI TTS being unable to emit narration+music in one file: you compose the mix on the timeline. Honesty: you can't un-mix / stem-separate the original's baked audio — the two tracks are a fresh narration track **plus** a real music file (owned / user-provided / `browse_files` / generated with `/api/v1/primitives/music/generate` (`vidfarm music`)), never a faked or duplicated voice layer.
|
|
13
13
|
|
|
@@ -17,7 +17,7 @@ Vidfarm is deliberately founder-friendly and thrift-first: **we do not spend exp
|
|
|
17
17
|
|
|
18
18
|
> **This decomposition is now a materialized artifact.** Every decomposed fork carries a **replication harness** (`replication-harness.json` / `editor_context.replication_harness` / `GET .../replication-harness.json`) that already contains BOTH plans as concrete per-beat brush assignments plus reusable-asset guidance and viral-DNA guardrails. When it's present, **execute its beats** instead of re-deriving them; the methodology below is how it was built and the fallback when it's absent (older forks / free tier). See the route section further down.
|
|
19
19
|
|
|
20
|
-
1. **Raw clips** — cut + remix existing long-form / short-form footage. The workhorse. Source in cost order: the director's own `/raws` + `/files` library → HUNT new raws out of a URL/VOD (`vidfarm raws scan`) → only then generate. Combining a **background video + a foreground video** (greenscreen / picture-in-picture) covers a huge share of "video meme" formats with zero generation.
|
|
20
|
+
1. **Raw clips** — cut + remix existing long-form / short-form footage. The workhorse. Source in cost order: the director's own `/raws` + `/files` library → the **public raws catalog by category** (`vidfarm public-raws --categories`, then `--category <shelf>`; free, pre-tagged, and the fastest way to fill a slot or fan one composition into N variants) → HUNT new raws out of a URL/VOD (`vidfarm raws scan`) → only then generate. Combining a **background video + a foreground video** (greenscreen / picture-in-picture) covers a huge share of "video meme" formats with zero generation.
|
|
21
21
|
2. **HTML/JS hyperframes** — video-from-HTML. Animate text, image elements, logos, stickers, charts and lower-thirds with CSS/declarative motion (web editor) or JS adapters (anime.js/GSAP/Lottie/Three, devcli only). Deterministic, cheap, re-themeable — the right brush for titles, kinetic captions, data beats, brand cards, and any "graphic" scene.
|
|
22
22
|
3. **Pure AI generation** — AI image / video / voice / music. The most expensive brush (AI **video** especially, $1–$10+); the last resort for a beat no clip or hyperframe can cover.
|
|
23
23
|
|
|
@@ -291,6 +291,46 @@ When `applies` is true, run the workflow **exactly in order, and honor the two c
|
|
|
291
291
|
|
|
292
292
|
When `applies` is **false** (talking-head, screen-recording, clip-remix, kinetic-text, meme-recaption, real b-roll), ignore this block and rebuild thrift-first from `harness.cheap.scenes[]` as above — do NOT reach for the generative pipeline. `status:"none"` → run `POST /auto-decompose` first.
|
|
293
293
|
|
|
294
|
+
#### `vidfarm sequence` — the generative workflow as one resumable command
|
|
295
|
+
|
|
296
|
+
The workflow above, run end-to-end from a desktop agent (or by hand) without gluing the steps together yourself. All state lives in a single `sequence.json`, so every step is resumable and rerunning a finished step is a no-op.
|
|
297
|
+
|
|
298
|
+
```bash
|
|
299
|
+
vidfarm sequence init "a lone windsurfer rides the wind across sunlit grass hills" \
|
|
300
|
+
--dir ./wind --parts 2 --panels 8 --duration 15 --aspect-ratio 16:9 \
|
|
301
|
+
--character Bree --character-description "athletic, platinum bob, cropped jacket, barefoot" \
|
|
302
|
+
--style "painterly anime illustration, crisp daylight, saturated blue sky"
|
|
303
|
+
|
|
304
|
+
vidfarm sequence run --dir ./wind # walks the whole pipeline, gate before each spend
|
|
305
|
+
vidfarm sequence run --dir ./wind --yes # fully autonomous, no prompts
|
|
306
|
+
```
|
|
307
|
+
|
|
308
|
+
`init` writes a **structurally complete but generically written** plan plus `PLAN_TASK.md`. The plan is what makes the output good, so the intended second step is to direct it — open `sequence.json`, or hand `PLAN_TASK.md` to a coding agent in that folder (`claude "read PLAN_TASK.md and do it"`), then `vidfarm sequence plan --dir ./wind --check`. Running without that step still works; it just produces generic direction.
|
|
309
|
+
|
|
310
|
+
| Step | Cost | What it does |
|
|
311
|
+
|---|---|---|
|
|
312
|
+
| `characters` | ~$0.05 each | One identity sheet per character — the identity authority for every later shot |
|
|
313
|
+
| `boards` | ~$0.05 each | One storyboard sheet per part, written in the block grammar below |
|
|
314
|
+
| `prompts` | free | Composes the video prompt per part, deterministically, from the plan |
|
|
315
|
+
| `animate` | **~$1+ each** | Generates the video per part, citing card + board as references |
|
|
316
|
+
| `assemble` | free | ffmpeg-concats the parts (`--music <file|url>`) into `renders/final.mp4` |
|
|
317
|
+
|
|
318
|
+
**The ordering is the point.** Boards cost cents and video costs dollars, so you read a board *before* you buy a shot. `sequence run` puts a confirm gate after `characters` and after `boards` for exactly that reason; `--yes` (or a non-TTY / `--json` run) skips them. Preview any prompt for free before spending:
|
|
319
|
+
|
|
320
|
+
```bash
|
|
321
|
+
vidfarm sequence prompts --dir ./wind --print board:part-01 # storyboard image prompt
|
|
322
|
+
vidfarm sequence prompts --dir ./wind --print shot:part-01 # video-model prompt
|
|
323
|
+
vidfarm sequence prompts --dir ./wind --print card:bree # character card prompt
|
|
324
|
+
```
|
|
325
|
+
|
|
326
|
+
Three rules are baked into the generated prompts and carry most of the quality:
|
|
327
|
+
|
|
328
|
+
1. **Reference priority is stated explicitly** — the character card controls identity; the storyboard controls staging, motion, geography and continuity. Without the split, video models copy faces off the board's rough panels.
|
|
329
|
+
2. **"Do not render the storyboard sheet itself"** — omit it and the model cheerfully animates the grid, borders and all.
|
|
330
|
+
3. **Panels are restated as `PANEL BEATS` in words**, so shot order survives even when the model reads the board loosely.
|
|
331
|
+
|
|
332
|
+
`parts[].panels[]` carries `lens`, `camera`, `action`, `state`, `style`, a `rhythm` (hold/build/burst/impact/…) and an `escalation` (L1 calm → L5 peak). Those feed the board's DIRECTOR STRIP — the rhythm/escalation tracks under the grid that make a board read as *timing* rather than as N unrelated stills. Set `parts[].extend_from: "part-01"` on part 2 and its video prompt opens with `extend` instead of re-establishing the world. `--board-style final` renders board panels in the final look (a style fit-check) instead of monochrome previz.
|
|
333
|
+
|
|
294
334
|
## Generate AI media and drop it on the timeline
|
|
295
335
|
|
|
296
336
|
Both surfaces can generate a NEW AI video/image (optionally conditioned on reference images) and place it into the timeline — filling a blank gap or replacing a scene — using the async generation primitives (`POST /api/v1/primitives/videos/generate`, `/images/generate`). Generation is a **queued job**: the POST returns `202 { job_id }`, and the finished media URL only appears after the job settles (`result.primary_file_url` / `video.file_url` / an artifact `public_url`). Poll `GET /api/v1/user/me/jobs/:jobId` until `status: "succeeded"`.
|
|
@@ -299,6 +339,8 @@ Both surfaces can generate a NEW AI video/image (optionally conditioned on refer
|
|
|
299
339
|
|
|
300
340
|
- **Editor web copilot**: use the `generate_layer` editor_action — it does the whole flow (submit job → placeholder clip → auto-swap on completion) and reads `pending_generations` from `editor_context`. Set `media_type`, `prompt`, `aspect_ratio` (match the canvas), and `intent` (`fill_gap` with `start`/`duration`, `replace_layer` with `replace_layer_key`, or `add`). For character consistency pass a cast `reference_url` — or the character's `character_sprite_card.png` viewUrl from `/files/characters/<slug>/` (see "Recurring characters are first-class") — in `input_references` (video) / `prompt_attachments` (image).
|
|
301
341
|
|
|
342
|
+
- **Avatars are a different route.** "Create an avatar" (spokesperson / presenter / host / UGC creator / talking head) is NOT plain `/videos/generate` or `generate_layer` — it's `POST /api/v1/primitives/videos/create-avatar` (devcli `vidfarm avatar "<who>" --say "<line>"`), which forces the spoken audio, the exact-key-color greenscreen plate, and the chroma key, and returns a transparent presenter WebM to layer over the scene. The web copilot calls it with `http_request`, then `add_layer`/`set_layer_media` the `primary_file_url` as a video layer. See `references/primitives.md` → "Primitive: talking_avatar".
|
|
343
|
+
|
|
302
344
|
- **Desktop agents (Claude Code / Codex)** — two devcli commands:
|
|
303
345
|
```bash
|
|
304
346
|
# 1) Sync the fork to disk and see where the blank space and scenes are
|
|
@@ -51,6 +51,34 @@ curl -X POST "$VIDFARM_BASE/api/v1/primitives/images/create-overlay" \
|
|
|
51
51
|
-d '{"tracer": "vox-overlay", "payload": {"prompt": "a friendly cartoon lightbulb mascot, flat vector illustration", "aspect_ratio": "1:1"}}'
|
|
52
52
|
```
|
|
53
53
|
|
|
54
|
+
## Primitive: talking_avatar ("create an avatar" — the talking-head avatar)
|
|
55
|
+
|
|
56
|
+
**When a user says "create an avatar" / "make me a spokesperson / presenter / host / UGC creator / talking head", this is what they mean — ALWAYS a VIDEO of a talking head with SPOKEN AUDIO, never a still portrait and never a silent clip.** The avatar is generated on a **flat greenscreen plate at an exact known key color** and then keyed off it in the same job, so what you get back is a **transparent presenter** you composite over whatever background the scene needs. Same fused shape as `media_overlay`, but for video-with-voice.
|
|
57
|
+
|
|
58
|
+
- `POST /api/v1/primitives/videos/create-avatar` (flat alias: `POST /api/v1/primitives/create-avatar`)
|
|
59
|
+
- Body: `{ "tracer": "...", "payload": { "prompt": "friendly 30-something founder in a grey hoodie, warm smile", "script": "Here's why we built this.", "voice"?: "warm, upbeat, mid-tempo", "input_references"?: ["https://.../headshot.png"], "frame_images"?: [...], "provider"?: "...", "model"?: "...", "duration"?: 8, "resolution"?: "720p", "aspect_ratio"?: "9:16", "preset"?: "green"|"digital-green"|"blue"|"white"|"black"|"magenta", "key_color"?: "#00FF00", "tolerance"?: 0.3, "softness"?: 0.1, "despill"?: true, "keep_greenscreen"?: false }, "webhook_url"?: "..." }`
|
|
60
|
+
- **`prompt` is WHO the avatar is** — face, age, wardrobe, vibe. Do NOT describe the background; the platform appends the talking-head framing plus the exact flat key-color plate requirement for you (that's why the key color is *expected* and the key comes out clean).
|
|
61
|
+
- **`script` is what they SAY, verbatim** — it is spoken out loud with lip-sync (audio generation is forced on; a mute talking head is a broken deliverable). Omit `script` only for a deliberate idle/listening loop.
|
|
62
|
+
- **`input_references` locks the likeness** — a headshot, brand photo, or a character's `character_sprite_card.png`. Pass it on *every* generation of a recurring spokesperson so the face doesn't drift (see "Recurring characters are first-class" in assets-and-sourcing).
|
|
63
|
+
- `key_color` / `preset` moves the plate off green when the avatar is green-heavy (green shirt, green hair) — key against magenta instead. The chroma knobs behave exactly as in `remove-greenscreen`.
|
|
64
|
+
- `keep_greenscreen: true` returns the raw plate un-keyed — for when you want to key it yourself locally with free ffmpeg.
|
|
65
|
+
- Response: `primary_file_url` is the **transparent WebM (VP9 alpha, audio preserved)** — the alpha survives into the `/editor` preview AND the final render. `greenscreen_source_url` is the raw plate, kept so you can re-key at a different tolerance without paying the provider for a second generation.
|
|
66
|
+
- Billing: the caller's **own BYOK video-provider key** for the generation leg, plus the real measured keying compute × 1.2 (`greenscreen_chroma_key`). AI video is the single most expensive thing on the platform — on a free plan or a "don't spend money" instruction, say so and offer a free talking-head clip from the public raws catalog (`vidfarm public-raws --category talking-head`) instead.
|
|
67
|
+
- **Never ship the green plate as the deliverable.** The green background must be gone before the avatar goes on a timeline; place `primary_file_url` as a video layer over the background scene (`add_layer` / `vidfarm place --kind video`).
|
|
68
|
+
|
|
69
|
+
devcli: `vidfarm avatar "<who they are>" --say "<line>" [--ref headshot.png] [--aspect-ratio 9:16] [--duration 8] [--local] [--place ./my-video]` — aliases `talking-head`, `spokesperson`. Generation always runs in the cloud on the user's own keys; `--local` brings the plate down and keys it on your machine with bundled ffmpeg for free, `--keep-greenscreen` stops at the plate.
|
|
70
|
+
|
|
71
|
+
```bash
|
|
72
|
+
# Cloud (default): transparent talking avatar, ready to drop on the timeline.
|
|
73
|
+
vidfarm avatar "friendly 30-something founder in a grey hoodie" \
|
|
74
|
+
--say "We built this because editing shouldn't take a weekend." \
|
|
75
|
+
--ref ./founder-headshot.png --aspect-ratio 9:16 --duration 8 \
|
|
76
|
+
--place ./my-video --at 0:03
|
|
77
|
+
|
|
78
|
+
# Same generation, keyed locally with free ffmpeg instead.
|
|
79
|
+
vidfarm avatar "cheerful barista in an apron" --say "One flat white, coming up." --local
|
|
80
|
+
```
|
|
81
|
+
|
|
54
82
|
## Primitive: videos/download — **Vidfarm downloads videos off websites for you**
|
|
55
83
|
|
|
56
84
|
**Capability statement (say this plainly when a user asks "can you download this video?"): yes — on a paid plan, Vidfarm downloads a video straight off a website URL for you.** Hand it a YouTube / TikTok / Instagram / X (Twitter) / other supported post URL and it returns a durable Vidfarm-hosted file. No local yt-dlp, no browser extension, no "I can't access that URL." The route is intentionally broader than its old name suggests: it returns either a normal MP4 for video posts, or a slideshow payload for photo/carousel posts.
|
|
@@ -14,7 +14,7 @@ Check which engines this run will use and say so in one line — then continue.
|
|
|
14
14
|
- Saved keys: `vidfarm provider-keys`, or env `OPENAI_API_KEY` / `GEMINI_API_KEY` / `OPENROUTER_API_KEY`.
|
|
15
15
|
- Key present → BYOK cloud speech: `vidfarm tts` (promptable voice via `--style`) and `vidfarm stt` (real word timestamps on an OpenAI whisper-1 key). **But note the cost-mode default below — with a key saved, plain `vidfarm tts` still uses the FREE local voice in `minimize`/`hybrid` unless you ask for a premium one.**
|
|
16
16
|
- No key → keyless local engines: Kokoro-82M TTS (`npx hyperframes tts`) and whisper.cpp timings (`npx hyperframes transcribe`). They download model weights on first use and never need an account.
|
|
17
|
-
- **Cost mode drives the TTS engine.** `minimize` and `hybrid` (and the unset default) → free local Kokoro; `rich-ai`, or any premium ask (`--style`/`--provider`/`--model`/`--own-key`/non-Kokoro `--voice`/`--cloud`) → BYOK or cloud. A missing local engine falls back to BYOK/cloud automatically. Say which voice tier the narration is on the first time you generate it.
|
|
17
|
+
- **Cost mode drives the TTS engine.** The four modes quote a per-video budget — `minimize` = **$0**, `hybrid` = **~$0.01–$1**, `rich-ai` = **$1+** (AI video gen spent on reusable greenscreen raws, animated in hyperframes HTML), `pure-videogen` = **$5+** (script → storyboard → frame-by-frame generated scenes), all billed to the user's own AI provider keys (BYOK), never marked up by Vidfarm. `minimize` and `hybrid` (and the unset default) → free local Kokoro; `rich-ai`/`pure-videogen`, or any premium ask (`--style`/`--provider`/`--model`/`--own-key`/non-Kokoro `--voice`/`--cloud`) → BYOK or cloud. A missing local engine falls back to BYOK/cloud automatically. Say which voice tier the narration is on the first time you generate it.
|
|
18
18
|
- Don't write keys into a per-repo `.env`. Details: `references/tts.md`.
|
|
19
19
|
|
|
20
20
|
## The audio engine
|
|
@@ -135,8 +135,10 @@ Legacy commands still work and map onto the same directory: `vidfarm files --sea
|
|
|
135
135
|
Same ownership rule, resolved through the file directory:
|
|
136
136
|
|
|
137
137
|
- **Search what the user has** — `vidfarm directory search "<meaning>"` (scope with `--path /raws` or `--path /files`); or the legacy `vidfarm files --search` / `vidfarm raws search`.
|
|
138
|
+
- **Browse the free PUBLIC RAWS catalog by category** — `vidfarm public-raws --categories` lists curated shelves (scroll-stoppers, greenscreen, reaction, b-roll, talking-head, …) with live counts and what each is for; `vidfarm public-raws --category <shelf> [--query "…"]` pulls that shelf; `--bookmark <raw_id>` saves one into the user's `/raws/public/*`. Free, pre-tagged, already hosted, and cleared for use — **check here before hunting or generating footage.** One shelf also doubles as a ready-made clip pool for fanning a composition into N variants (bulk scripting) with no download or generation spend. API: `GET /api/v1/public-raws/categories`, `GET /api/v1/public-raws?category=<shelf>`.
|
|
138
139
|
- **Populate raws** — `vidfarm raws scan <video>` hunts reusable raws out of the user's long-form footage into `/raws/<source>/`.
|
|
139
140
|
- **Generate on the user's keys** — `vidfarm generate image|video --prompt "…"` (BYOK primitives; `--place <dir>` drops the result straight into a composition).
|
|
141
|
+
- **"Create an avatar" — a talking head, on the user's keys** — `vidfarm avatar "<who they are>" --say "<their line>" [--ref headshot.png] [--aspect-ratio 9:16]` (`POST /api/v1/primitives/videos/create-avatar`). An avatar is always a **video of someone speaking, with lip-synced audio** — never a still portrait, never mute. It's shot on an exact-key-color **greenscreen** plate and keyed off it in the same job, so you get a **transparent presenter** (audio preserved) to composite over any background; `--ref` (headshot or `character_sprite_card.png`) keeps the face on-model across videos, `--local` keys the plate for free with ffmpeg. AI video is the priciest generation here — offer the free `talking-head` raws shelf first when the user is cost-conscious.
|
|
140
142
|
- **Search the free stock catalog** — `vidfarm media search "<meaning>" --type <image|vector|icon|video|bgm|sfx>` (shorthand: `vidfarm media icon "home"`). Returns royalty-free, commercial-safe results, each tagged with its license — honor `attribution` when `attribution_required` is true. Web/API equivalent: `GET /api/v1/primitives/media/search?type=&q=&limit=`.
|
|
141
143
|
- Never scrape or hotlink arbitrary third-party pages; the catalog above is the sanctioned licensed source, and a capture workflow's own screenshots + the user's brand assets are always fair game.
|
|
142
144
|
|
|
@@ -52,7 +52,7 @@ vidfarm voices --own-key # the customer's own ElevenLabs account voices
|
|
|
52
52
|
vidfarm tts "Welcome back to the channel" --style "calm, warm bedtime narrator" --voice alloy --out narration.mp3 --json
|
|
53
53
|
```
|
|
54
54
|
|
|
55
|
-
- **Cost-mode default: FREE local voice.** In cost mode `minimize` AND `hybrid` (including the unset default), a bare `vidfarm tts "…"` runs the keyless local Kokoro-82M engine — no `--engine local` needed. Opt out by asking for a premium voice (`--style`, `--provider`, `--model`, `--own-key`, or a non-Kokoro `--voice` such as `alloy`/`Kore`/an ElevenLabs id), by passing `--cloud` or `--engine byok`, or by being in `rich-ai`. If the local engine isn't installed the run **falls back** to the BYOK/cloud path below rather than failing, printing the reason on stderr.
|
|
55
|
+
- **Cost-mode default: FREE local voice.** In cost mode `minimize` AND `hybrid` (including the unset default), a bare `vidfarm tts "…"` runs the keyless local Kokoro-82M engine — no `--engine local` needed. Opt out by asking for a premium voice (`--style`, `--provider`, `--model`, `--own-key`, or a non-Kokoro `--voice` such as `alloy`/`Kore`/an ElevenLabs id), by passing `--cloud` or `--engine byok`, or by being in `rich-ai`/`pure-videogen`. If the local engine isn't installed the run **falls back** to the BYOK/cloud path below rather than failing, printing the reason on stderr.
|
|
56
56
|
- **Local-first**: runs directly on your env key (no cloud job, no wallet). `--cloud` is the explicit backup — an async platform job on your saved provider keys via `POST /api/v1/primitives/audio/speech` (poll the returned `job_id`).
|
|
57
57
|
- **Promptable voice**: `--style` (alias `--instructions`) directs tone/pacing/accent/emotion/persona — "excited sports announcer", "whispered documentary aside".
|
|
58
58
|
- **Voices** (provider inferred from the voice when `--provider` is omitted; wrong-family voices are rejected with a clear error):
|