@mevdragon/vidfarm-devcli 0.9.0 → 0.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +37 -1
- package/SKILL.director.md +123 -0
- package/SKILL.platform.md +15 -2
- package/dist/src/account-pages-legacy.js +15 -10
- package/dist/src/app.js +550 -184
- package/dist/src/cli.js +397 -14
- package/dist/src/config.js +5 -0
- package/dist/src/devcli/clips.js +312 -49
- package/dist/src/devcli/speech.js +175 -0
- package/dist/src/editor-chat.js +3 -2
- package/dist/src/frontend/homepage-view.js +0 -5
- package/dist/src/page-shell.js +47 -0
- package/dist/src/primitive-context.js +16 -0
- package/dist/src/primitive-registry.js +239 -0
- package/dist/src/services/billing.js +3 -0
- package/dist/src/services/clip-curation/ffmpeg.js +59 -17
- package/dist/src/services/clip-curation/gemini.js +72 -23
- package/dist/src/services/clip-curation/hunt.js +332 -0
- package/dist/src/services/clip-curation/index.js +3 -1
- package/dist/src/services/clip-curation/local-agent.js +247 -0
- package/dist/src/services/clip-curation/refine.js +7 -29
- package/dist/src/services/clip-curation/scan.js +37 -10
- package/dist/src/services/local-dynamo.js +0 -0
- package/dist/src/services/provider-errors.js +74 -0
- package/dist/src/services/providers.js +30 -429
- package/dist/src/services/serverless-records.js +9 -2
- package/dist/src/services/speech.js +467 -0
- package/dist/src/services/storage.js +16 -1
- package/package.json +1 -1
- package/public/assets/homepage-client-app.js +17 -17
package/README.md
CHANGED
|
@@ -50,7 +50,43 @@ Everything runs locally (`RECORDS_DRIVER=local`, `STORAGE_DRIVER=local`), includ
|
|
|
50
50
|
|
|
51
51
|
Flags: `--port` (default 3000), `--dir` (default `./.vidfarm-local`), `--key` (bootstrap/browser key, or `VIDFARM_API_KEY`), `--fork <id>`, `--no-open`. `vidfarm <template_id>` is an alias for `serve <template_id>`.
|
|
52
52
|
|
|
53
|
-
Beyond the local editor, the devcli also wraps the **entire director REST flow** as 1:1 commands — `discover`, `inspiration-add`, `fork`, `decompose`, `snapshot`, `render`, `approve`, `schedule`, `posts`, `upload`/`download`, plus a raw `vidfarm api <METHOD> <path>` passthrough. Each maps to exactly one REST route and prints the prod frontend URL to open. Run `vidfarm --help` for the full surface, or see `SKILL.director.md` § "`vidfarm-devcli` — full command surface".
|
|
53
|
+
Beyond the local editor, the devcli also wraps the **entire director REST flow** as 1:1 commands — `discover`, `inspiration-add`, `fork`, `decompose`, `snapshot`, `render`, `approve`, `schedule`, `posts`, `upload`/`download`, `clips` (see below), `tts`/`stt` (see below), plus a raw `vidfarm api <METHOD> <path>` passthrough. Each maps to exactly one REST route and prints the prod frontend URL to open. Run `vidfarm --help` for the full surface, or see `SKILL.director.md` § "`vidfarm-devcli` — full command surface".
|
|
54
|
+
|
|
55
|
+
## Clip hunting: `vidfarm clips`
|
|
56
|
+
|
|
57
|
+
Mine a long-form video (podcast, VOD, webinar, YouTube URL) into tagged, searchable short clips — **local-first**: local ffmpeg for compute and your local `claude`/`codex` CLI subscription for scene evaluation, so it works with zero API keys. Provider keys (`--provider gemini|openai|openrouter`) are the fallback; `--cloud` is the explicit backup that runs the hunt on the deployed pipeline instead (bills AWS compute only — the AI runs on your saved BYOK keys, and the uploaded original auto-deletes from your temp folder after 30 days).
|
|
58
|
+
|
|
59
|
+
```bash
|
|
60
|
+
# hunt only 12:30–15:45, ~30s clips (soft 20–40s band), cropped vertical, no on-screen text
|
|
61
|
+
vidfarm clips scan ./podcast.mp4 --range "12:30-15:45" --duration 30 --aspect vertical --no-text \
|
|
62
|
+
--prompt "guest reaction moments"
|
|
63
|
+
|
|
64
|
+
vidfarm clips search "confused reaction after reading a message"
|
|
65
|
+
vidfarm clips export <clip-ids…> --to ./picks
|
|
66
|
+
|
|
67
|
+
# backup: run it in the cloud (or hunt a URL without downloading anything)
|
|
68
|
+
vidfarm clips scan --cloud --url "https://youtube.com/watch?v=…" --duration 30 --aspect 9:16
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
Duration is a **soft range** (10→5–20s, 30→20–40s), `--range` means only those windows are decoded (the cost saver on long videos), and `--no-text` selects text-free scenes — caption *removal* (GhostCut) stays a per-finished-clip primitive, never a long-form pass. Run `vidfarm clips --help` for everything (`list`, `match`, `preset`, `sources`, `export`).
|
|
72
|
+
|
|
73
|
+
## Speech: `vidfarm tts` / `vidfarm stt`
|
|
74
|
+
|
|
75
|
+
Text-to-speech and speech-to-text, **local-first** like `clips`: both default to your own `OPENAI_API_KEY` / `GEMINI_API_KEY` / `OPENROUTER_API_KEY` (no cloud job, no wallet), with `--cloud` as the explicit backup that runs the platform primitive on your saved BYOK keys instead. (Speech can't ride the local `claude`/`codex` agent — no audio I/O — so the local path always needs a provider key.)
|
|
76
|
+
|
|
77
|
+
```bash
|
|
78
|
+
# text → narration audio; the VOICE STYLE is promptable
|
|
79
|
+
vidfarm tts "Welcome back. Today we test the three most viral hooks." \
|
|
80
|
+
--style "energetic UGC creator, conversational" --voice coral --out narration.mp3
|
|
81
|
+
|
|
82
|
+
# video or audio → transcript, BOTH formats in one pass:
|
|
83
|
+
# simple subtitles (transcript.txt + timed transcript.srt)
|
|
84
|
+
# + multi-speaker segments (transcript.json: {speaker, start_sec, end_sec, text})
|
|
85
|
+
vidfarm stt ./podcast.mp4 --out transcript # local ffmpeg demux + your key
|
|
86
|
+
vidfarm stt "https://cdn.example.com/clip.mp4" --cloud # backup: POST /api/v1/primitives/audio/transcribe
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
Speaker labels (diarization, on by default) need a Gemini key — OpenAI/OpenRouter keys degrade to a plain single-speaker transcript. The matching REST primitives are `POST /api/v1/primitives/audio/speech` and `/audio/transcribe` (aliases `/tts`, `/stt`).
|
|
54
90
|
|
|
55
91
|
### Crash telemetry & opt-out
|
|
56
92
|
|
package/SKILL.director.md
CHANGED
|
@@ -208,6 +208,58 @@ Use it whenever you need to know what the video says or shows: writing/translati
|
|
|
208
208
|
- **Editor chat (frontend AI)** exposes this as the optional `video_context` tool — the copilot calls it on demand.
|
|
209
209
|
- **Desktop agents (Claude Code / Codex)**: fetch the `video-context.json` route directly (e.g. `vidfarm api GET /api/v1/compositions/<forkId>/video-context.json`) to ground edits in what the video says and shows.
|
|
210
210
|
|
|
211
|
+
## Clip hunting (long-form → short-form clips)
|
|
212
|
+
|
|
213
|
+
Mine a **long-form** video (podcast, stream VOD, webinar, any YouTube/TikTok/IG/X URL, or an upload) into a library of tagged, searchable **short clips**. This is the `/clips` surface — browse it at `https://vidfarm.cc/clips` (the Library page's "Finished / Clips" toggle).
|
|
214
|
+
|
|
215
|
+
**Start a hunt** — `POST /clips/scan` (async: returns `202 { scan_id }` immediately):
|
|
216
|
+
|
|
217
|
+
```jsonc
|
|
218
|
+
{
|
|
219
|
+
// ONE source (pick one):
|
|
220
|
+
"source_url": "https://youtube.com/watch?v=…", // downloaded into your temp folder first
|
|
221
|
+
"temp_file_id": "…", // a video you uploaded to temporary-files
|
|
222
|
+
"attachment_id": "…", // a video already in My Files
|
|
223
|
+
"s3_key": "…" , // an already-staged object
|
|
224
|
+
|
|
225
|
+
// What to hunt for — free text; inline hints are parsed too
|
|
226
|
+
"prompt": "people holding food up to their face, no text on screen",
|
|
227
|
+
|
|
228
|
+
// Hunt controls (all optional; also expressible inline in the prompt):
|
|
229
|
+
"ranges": ["12:30-15:45", "20:00-22:10"], // ONLY hunt these source windows — big cost saver on long videos
|
|
230
|
+
"target_duration_sec": 30, // SOFT length band, not a hard cut: 10→5-20s, 30→20-40s, 60→40-80s, 120→80-160s
|
|
231
|
+
"aspect": "9:16", // crop every clip: 9:16 | 16:9 | 4:3 | 1:1 (synonyms: vertical/portrait, horizontal/landscape, square)
|
|
232
|
+
"crop_focus": "center", // center | top | bottom | left | right (top biases toward faces)
|
|
233
|
+
"avoid_text": true, // prefer scenes WITHOUT burned-in captions/on-screen text — a scene-SELECTION filter, NEVER GhostCut on the source
|
|
234
|
+
"tracer": "my-campaign" // rolls up all the hunt's billing/observability events
|
|
235
|
+
}
|
|
236
|
+
```
|
|
237
|
+
|
|
238
|
+
- **Poll** `GET /clips/scan/:scanId` until the source `status` is `complete` (or `failed` — crashed pipelines are reconciled to `failed`, pollers never spin).
|
|
239
|
+
- **Results**: `GET /clips/feed?source=<source_video_id>` (or the whole library), hybrid search via `POST /clips/search` `{ query: "someone looks confused" }`, per-clip download at `GET /clips/:clipId/download`. Clips carry taxonomy tags, a description, a transcript, an `aspect` field when cropped, and (with a gemini/openai key) a semantic embedding.
|
|
240
|
+
- **Billing** — **AWS compute only** (`clip_scan_lambda` GB-seconds + Step Functions transitions, a fraction of a cent for typical hunts; the 202 response includes a `compute_estimate`). The AI tagging/refine runs on **your saved provider key (BYOK)** and is never wallet-billed. Submission is wallet-gated (402 when empty).
|
|
241
|
+
- **The original stays temporary** — URL-ingested and uploaded sources live ONLY in your temp folder (`users/…/temporary/clip-sources/…`) with a hard **30-day TTL** (auto-deleted from S3 + the temp-file list). The hunted clips themselves are durable.
|
|
242
|
+
- **Captions rule** — "no captions / no on-screen text" hunts are handled by scene SELECTION (the refine pass drops text-heavy scenes). **Never run GhostCut caption removal on a long-form source** (it is hard-capped at ~15 minutes); to actually erase burned-in text, apply `POST /api/v1/primitives/videos/remove-captions` to individual FINISHED clips afterwards.
|
|
243
|
+
|
|
244
|
+
**devcli (local-first — this is the default way to hunt on your own machine):**
|
|
245
|
+
|
|
246
|
+
```bash
|
|
247
|
+
# Local machine power: local ffmpeg + your local claude/codex CLI subscription (no API key needed)
|
|
248
|
+
vidfarm clips scan ./podcast.mp4 --range "12:30-15:45" --duration 30 --aspect vertical --no-text \
|
|
249
|
+
--prompt "guest reaction moments"
|
|
250
|
+
|
|
251
|
+
# Provider keys are the fallback (--provider gemini|openai|openrouter); the cloud pipeline is the
|
|
252
|
+
# EXPLICIT backup, never the default:
|
|
253
|
+
vidfarm clips scan --cloud ./podcast.mp4 --duration 30 --aspect 9:16 --tracer my-campaign
|
|
254
|
+
vidfarm clips scan --cloud --url "https://youtube.com/watch?v=…" --range "0:00-10:00"
|
|
255
|
+
|
|
256
|
+
# Then search/reuse the library
|
|
257
|
+
vidfarm clips search "confused reaction after reading a message"
|
|
258
|
+
vidfarm clips export <clip-ids…> --to ./picks
|
|
259
|
+
```
|
|
260
|
+
|
|
261
|
+
Local scans persist to a SQLite library under `~/.vidfarm` (free compute, subscription-powered evaluation); `--cloud` uploads to your temp folder (30-day TTL) or passes `--url`, runs the deployed pipeline, and bills AWS compute only.
|
|
262
|
+
|
|
211
263
|
## Generate AI media and drop it on the timeline
|
|
212
264
|
|
|
213
265
|
Both surfaces can generate a NEW AI video/image (optionally conditioned on reference images) and place it into the timeline — filling a blank gap or replacing a scene — using the async generation primitives (`POST /api/v1/primitives/videos/generate`, `/images/generate`). Generation is a **queued job**: the POST returns `202 { job_id }`, and the finished media URL only appears after the job settles (`result.primary_file_url` / `video.file_url` / an artifact `public_url`). Poll `GET /api/v1/user/me/jobs/:jobId` until `status: "succeeded"`.
|
|
@@ -450,6 +502,8 @@ Send a stable `tracer` on export so retries are traceable and filterable in job
|
|
|
450
502
|
| `vidfarm fork <template_id>` | `POST /api/v1/compositions` | fork a template |
|
|
451
503
|
| `vidfarm pull <forkId> [--dir <p>]` | `GET .../compositions/:forkId/{composition.html,json,video-context.json,cast.json}` | sync a fork to disk + print gaps/scene keys |
|
|
452
504
|
| `vidfarm generate <image\|video> --prompt "…"` | `POST /api/v1/primitives/{images,videos}/generate` (polls job) | generate AI media → finished URL |
|
|
505
|
+
| `vidfarm tts "…" [--style "…"] [--voice <v>] [--out <file>]` | (LOCAL-FIRST: your own OPENAI/GEMINI/OPENROUTER_API_KEY → audio file on disk; `--cloud` = `POST /api/v1/primitives/audio/speech` + poll) | text → narration audio with a promptable voice style |
|
|
506
|
+
| `vidfarm stt <file\|url> [--out <base>] [--no-diarize]` (alias: `transcribe`) | (LOCAL-FIRST: local ffmpeg demux + your own key; `--cloud` = `POST /api/v1/primitives/audio/transcribe` + poll) | video/audio → transcript in BOTH formats: simple subtitles (txt + SRT) and multi-speaker segments (json) |
|
|
453
507
|
| `vidfarm place <dir> --src <url\|file> [--at\|--replace]` | (edits local composition.html; local files → serve disk store or temp upload) | drop media (URL **or local file**) into a gap / over a scene |
|
|
454
508
|
| `vidfarm decompose <forkId>` | `POST .../compositions/:forkId/auto-decompose` | split source into scenes |
|
|
455
509
|
| `vidfarm remove-video-captions <forkId>` (alias: `ghostcut`) | `GET .../compositions/:forkId/remove-video-captions` | subtitle-removal status |
|
|
@@ -472,6 +526,10 @@ Send a stable `tracer` on export so retries are traceable and filterable in job
|
|
|
472
526
|
| `vidfarm files [--folder <path>]` | `GET /api/v1/user/me/attachments` | list My Files assets + folders |
|
|
473
527
|
| `vidfarm get-file <id> [dest] [--print]` | (resolve id → view_url, then stream/print) | read one My Files asset |
|
|
474
528
|
| `vidfarm put-file <file> [--folder <p>] [--as <name>] [--content/--stdin]` | `POST /api/v1/user/me/attachments/upload` | write into My Files (persistent) |
|
|
529
|
+
| `vidfarm clips scan <video> [--range MM:SS-MM:SS] [--duration <s>] [--aspect 9:16] [--no-text] [--prompt "…"]` | (LOCAL: local ffmpeg + local claude/codex agent → `~/.vidfarm` SQLite) | hunt long-form video into short clips on this machine |
|
|
530
|
+
| `vidfarm clips scan --cloud <video\|--url <url>> [--tracer <id>]` | temp-file presign/PUT/finalize + `POST /clips/scan` + poll | BACKUP: run the hunt on the deployed pipeline (bills AWS compute only) |
|
|
531
|
+
| `vidfarm clips search "…"` / `clips match "…"` / `clips list` / `clips sources` | (local library; NL→criteria via local agent or provider key) | search/reuse the clip library |
|
|
532
|
+
| `vidfarm clips preset list\|run\|save` / `clips export <ids…> --to <dir>` | (local library) | saved queries; copy clip MP4s out |
|
|
475
533
|
|
|
476
534
|
**Frontend URLs are first-class output.** Every command that has a human-openable page (editor, discover, approved-post preview, share link) prints that prod frontend URL as a highlighted line. `render --wait` polls to completion and prints the final MP4 URL; `approve` prints the approved-post `share_url`. Add `--json` to any command for pure JSON (agent-friendly, no banners).
|
|
477
535
|
|
|
@@ -587,6 +645,7 @@ Remove burned-in captions/subtitles/on-screen text from any video URL (GhostCut-
|
|
|
587
645
|
- Response: standard primitive job (`202 { job_id }`). The job probes the source, submits to GhostCut, polls to completion server-side (typically a few minutes), and mirrors the result. Poll `GET /api/v1/primitives/jobs/:jobId` until `status: "succeeded"`, then read `primary_file_url` / `video.file_url` (`captionsRemovedVideoUrl` in the output).
|
|
588
646
|
- Billing: `~$0.10 per 30 seconds` of source video (per-chunk, 1-chunk minimum; `ghostcut_subtitle_removal` cost center), plus the small probe cost.
|
|
589
647
|
- For the CURRENT fork's decompose flow, don't re-run this primitive — the caption-free mirror already exists via `GET /api/v1/compositions/:forkId/remove-video-captions`.
|
|
648
|
+
- **Short clips only — hard cap ~15 minutes** (`GHOSTCUT_MAX_DURATION_SEC`, default 900s). Never point this at a long-form source: for "clips without captions" from a long video, run a clip hunt with `avoid_text: true` (scene selection — see "Clip hunting") and, if needed, apply this primitive to the individual FINISHED clips.
|
|
590
649
|
|
|
591
650
|
Example:
|
|
592
651
|
|
|
@@ -673,6 +732,70 @@ curl -X POST "$VIDFARM_BASE/api/v1/primitives/media/dedupe" \
|
|
|
673
732
|
}'
|
|
674
733
|
```
|
|
675
734
|
|
|
735
|
+
## Primitive: tts (text → speech)
|
|
736
|
+
|
|
737
|
+
Generate spoken narration audio from text on the caller's saved AI provider keys (OpenAI, Gemini, or OpenRouter — BYOK, never wallet-billed). The **voice style is promptable**: `instructions` is a free-form direction for tone, pacing, accent, emotion, and persona, on top of the provider's `voice` preset. Result is a durable Vidfarm audio URL (mp3/wav) ready to drop into a composition as an audio layer.
|
|
738
|
+
|
|
739
|
+
- `POST /api/v1/primitives/audio/speech` (alias: `POST /api/v1/primitives/tts`)
|
|
740
|
+
- Body: `{ "tracer": "...", "payload": { "text": "...", "voice"?: "...", "instructions"?: "...", "provider"?: "openai" | "gemini" | "openrouter", "model"?: "...", "output_format"?: "mp3" | "wav" }, "webhook_url"?: "..." }`
|
|
741
|
+
- `text` (required, ≤8000 chars) — the words to speak, verbatim. Aliases: `input`, `script`.
|
|
742
|
+
- `voice` (optional) — provider voice preset (OpenAI: `alloy`, `ash`, `coral`, …; Gemini: `Kore`, `Puck`, `Charon`, …).
|
|
743
|
+
- `instructions` (optional) — the voice-STYLE prompt, e.g. `"calm, warm bedtime narrator with a slight British accent"` or `"excited sports announcer, fast paced"`. Aliases: `style`, `style_instructions`, `voice_style`.
|
|
744
|
+
- `output_format` defaults to `mp3` (note: Gemini answers in wav regardless — read `audio.content_type`).
|
|
745
|
+
- Response: standard primitive job. Poll to completion, then read `primary_file_url` / `audio.file_url`.
|
|
746
|
+
- Billing: BYOK — the speech call runs on the customer's own provider key and is never wallet-billed.
|
|
747
|
+
- devcli: `vidfarm tts "…" --style "…"` runs LOCAL-FIRST on your own env key (no job, no cloud); add `--cloud` to run this route instead.
|
|
748
|
+
|
|
749
|
+
Example:
|
|
750
|
+
|
|
751
|
+
```bash
|
|
752
|
+
curl -X POST "$VIDFARM_BASE/api/v1/primitives/audio/speech" \
|
|
753
|
+
-H "vidfarm-api-key: $VIDFARM_API_KEY" \
|
|
754
|
+
-H "content-type: application/json" \
|
|
755
|
+
-d '{
|
|
756
|
+
"tracer": "demo-tts",
|
|
757
|
+
"payload": {
|
|
758
|
+
"text": "Welcome back. Today we are testing the three most viral hooks.",
|
|
759
|
+
"voice": "coral",
|
|
760
|
+
"instructions": "energetic UGC creator, conversational, slightly breathless"
|
|
761
|
+
}
|
|
762
|
+
}'
|
|
763
|
+
```
|
|
764
|
+
|
|
765
|
+
## Primitive: stt (video/audio → transcript)
|
|
766
|
+
|
|
767
|
+
Transcribe speech from **any video or audio URL** on the caller's saved AI provider keys (video sources are demuxed to audio automatically via the platform ffmpeg seam). One job returns **BOTH formats**:
|
|
768
|
+
|
|
769
|
+
1. **Simple subtitle version** — the plain transcript (`output.text`, `transcript.txt`) plus timed SRT cues (`output.subtitles.srt`, `transcript.srt`) with no speaker labels — drop-in captions.
|
|
770
|
+
2. **Advanced multi-speaker version** — speaker-attributed, timestamped segments (`output.segments`, each `{ speaker, start_sec, end_sec, text }`, with `output.speakers` listing the cast) for multi-narration sources like podcasts, interviews, and skits — stored durably as `transcript.json`.
|
|
771
|
+
|
|
772
|
+
- `POST /api/v1/primitives/audio/transcribe` (alias: `POST /api/v1/primitives/stt`)
|
|
773
|
+
- Body: `{ "tracer": "...", "payload": { "source_url": "https://...", "diarize"?: true, "language"?: "en", "prompt"?: "...", "provider"?: "gemini" | "openai" | "openrouter", "model"?: "...", "media_type"?: "auto" | "video" | "audio" }, "webhook_url"?: "..." }`
|
|
774
|
+
- `source_url` (required) — video OR audio URL. Aliases: `source_video_url`, `source_audio_url`, `video_url`, `audio_url`, `url`. For a fork's composition, pick the right source via `GET /api/v1/compositions/:forkId/remove-video-captions` first.
|
|
775
|
+
- `diarize` (default `true`) — attribute segments to speakers. **Speaker labels need a Gemini key** (audio-native model); OpenAI/OpenRouter keys degrade to a plain single-speaker transcript (`diarization: "none"`).
|
|
776
|
+
- `prompt` (optional) — domain/vocabulary hint; `language` (optional) — expected language code.
|
|
777
|
+
- Response: standard primitive job. Poll to completion; `output` carries both formats inline (very long transcripts set `text_truncated: true` and keep the full copy in the `transcript.json` artifact), and the artifacts `transcript.json` / `transcript.srt` / `transcript.txt` are durable URLs.
|
|
778
|
+
- Limits: ~40 minutes of speech per job (20MB extracted audio) — trim longer sources with `/videos/trim` or `/audio/trim` and transcribe in parts.
|
|
779
|
+
- Billing: the AI transcription is BYOK (never wallet-billed); only the tiny audio-demux compute step bills (`primitive_media_lambda`), like every media primitive.
|
|
780
|
+
- For the CURRENT fork, the decompose-time transcript may already exist via `video-context.json` — prefer it before paying for a new transcription; use this primitive for other URLs, multi-speaker attribution, or SRT output.
|
|
781
|
+
- devcli: `vidfarm stt <file|url>` runs LOCAL-FIRST (local ffmpeg + your own key); add `--cloud` to run this route instead.
|
|
782
|
+
|
|
783
|
+
Example:
|
|
784
|
+
|
|
785
|
+
```bash
|
|
786
|
+
curl -X POST "$VIDFARM_BASE/api/v1/primitives/audio/transcribe" \
|
|
787
|
+
-H "vidfarm-api-key: $VIDFARM_API_KEY" \
|
|
788
|
+
-H "content-type: application/json" \
|
|
789
|
+
-d '{
|
|
790
|
+
"tracer": "demo-stt",
|
|
791
|
+
"payload": {
|
|
792
|
+
"source_url": "https://cdn.example.com/podcast-clip.mp4",
|
|
793
|
+
"diarize": true,
|
|
794
|
+
"language": "en"
|
|
795
|
+
}
|
|
796
|
+
}'
|
|
797
|
+
```
|
|
798
|
+
|
|
676
799
|
## Brainstorm primitives
|
|
677
800
|
|
|
678
801
|
The `brainstorm/*` primitives are the strategy toolkit. They are reusable, billable AI reasoning steps — the same family the AI Copilot exposes as chip suggestions. Treat **product placement** as a first-class member of this family, right alongside angles and hooks:
|
package/SKILL.platform.md
CHANGED
|
@@ -229,7 +229,18 @@ Read-only companion: `GET /api/v1/compositions/:forkId/remove-video-captions` (l
|
|
|
229
229
|
|
|
230
230
|
Standalone primitive: `POST /api/v1/primitives/videos/remove-captions` (alias: `POST /api/v1/primitives/remove-video-captions`; id `primitive:video_remove_captions` in `src/primitive-registry.ts`) exposes the same GhostCut pipeline for ANY video URL, decoupled from forks and decompose. The job probes the source (duration prices the task), submits, polls GhostCut to completion inside the job runner (12-minute budget, vs the fork routes where the client drives completion), mirrors `captions-removed.mp4` into primitive storage, and bills the `ghostcut_subtitle_removal` cost center per 30s chunk.
|
|
231
231
|
|
|
232
|
-
Configuration: `GHOSTCUT_KEY` and `GHOSTCUT_SECRET` env vars per stage. Pass-through cost: `GHOSTCUT_USD_PER_30S` (default 0.10).
|
|
232
|
+
Configuration: `GHOSTCUT_KEY` and `GHOSTCUT_SECRET` env vars per stage. Pass-through cost: `GHOSTCUT_USD_PER_30S` (default 0.10). **Guard:** the primitive hard-fails sources longer than `GHOSTCUT_MAX_DURATION_SEC` (default 900s) — GhostCut is a per-finished-clip tool; long-form "no captions" requests are served by the clip-hunt `avoid_text` scene-selection filter instead, never by laundering a whole long video through GhostCut.
|
|
233
|
+
|
|
234
|
+
## Clip hunting pipeline (long-form → short-form)
|
|
235
|
+
|
|
236
|
+
`POST /clips/scan` starts a Step Functions STANDARD workflow (`…ClipScanWorkflow`): `probe_and_detect` → `Map(process_scene, maxConcurrency 5)` → `finalize`, all three states dispatching the same `infra/lambda/clip-scan.ts` on an `action` field. Business logic lives in the shared `src/services/clip-curation/` core (also consumed by devcli and the Hono API), with the hunt controls in `hunt.ts`:
|
|
237
|
+
|
|
238
|
+
- **`hunt_spec`** — windows (time ranges; ffmpeg input-seeks per window so only the requested footage is DECODED — the actual cost saving), a soft `duration_band` (`resolveDurationBand`: 10→5-20s, 30→20-40s via ±target/3), output `aspect` + `crop_focus` (smart-centre ffmpeg crop applied to clips, thumbnails, AND tagging keyframes so the model judges the framed output), `avoid_text` (rides the keep/drop/merge refine pass as a selection filter), and `source_url` (RapidAPI ingest → the user's temp folder). The spec travels as ONE opaque JsonPath object through the state machine, so new hunt fields are Lambda-code-only changes — no state-machine redeploy.
|
|
239
|
+
- **Sources are temporary** — URL ingests and uploads land at `users/{id}/temporary/clip-sources/…` with a hard 30-day TTL enforced three ways: tag-scoped S3 lifecycle (`vidfarm-temp=1`, tagged server-side at temp-file finalize/upload), native DynamoDB TTL (`expires_at_epoch_seconds`, top-level on the item), and the list-route sweep. Hunted clips (`clips/{owner}/…`) are durable.
|
|
240
|
+
- **Billing: AWS compute ONLY.** Every Lambda invocation post-hoc bills its GB-seconds to the `clip_scan_lambda` cost center (idempotent on `awsRequestId`); `finalize` adds `step_functions_standard` scaled by scene count (3 + N transitions); URL ingest adds the existing `rapidapi_video_download` pass-through. `jobId = scan_id`, and the caller's `tracer` rides every event (gsi2-tracer rollups). The BYOK tagging/embedding spend (gemini/openai/openrouter keys in the state input) is deliberately never wallet-billed. Submission is wallet-gated (`assertWalletCanQueueJobs` → 402); the 202 response carries a pre-scan `compute_estimate`.
|
|
241
|
+
- **Failure reconciliation** — the workflow has no failure hook into records, so `GET /clips/scan/:scanId` lazily `DescribeExecution`s a running scan and flips the scan+source records to `failed` (or `complete`) so pollers converge.
|
|
242
|
+
- **devcli is LOCAL-FIRST** — `vidfarm clips scan` runs local ffmpeg and, by default, the user's local claude/codex CLI subscription (`LocalAgentClipClient`, implementing the same `ClipHuntModelClient` interface as the API providers; refine's model call lives on the client as `decideScenes` to make them swappable). Provider keys are the fallback; `--cloud` is the explicit backup that presign-uploads to the temp folder and drives this pipeline.
|
|
243
|
+
- **Bundling gotcha** — the clip-scan and primitive-media Lambdas bundle `ffmpeg-static`/`ffprobe-static` with `forceDockerBundling: true`: ffmpeg-static downloads the binary for the platform it is installed on, so a mac-local bundle ships a darwin ffmpeg the Lambda cannot exec (`exit 126`). They also need `VIDFARM_DATA_DIR=/tmp/vidfarm` because `config.ts` mkdirs at import.
|
|
233
244
|
|
|
234
245
|
## Editor Chat
|
|
235
246
|
|
|
@@ -253,8 +264,9 @@ Primitives are hosted operations that are NOT templates — they're reusable bui
|
|
|
253
264
|
- `download_media_video` — social/media URL → durable MP4
|
|
254
265
|
- `hyperframes_render` — composition HTML → MP4 (the render primitive publishing uses)
|
|
255
266
|
- `brainstorm` — LLM brainstorm route
|
|
267
|
+
- `tts` / `stt` — speech, BYOK on the caller's saved provider keys. `POST /api/v1/primitives/audio/speech` (`primitive:tts`) turns text into narration audio with a promptable voice style (`instructions`); `POST /api/v1/primitives/audio/transcribe` (`primitive:stt`) transcribes any video or audio URL (video demuxes through the primitive-media ffmpeg seam first, 64kbps mono, ~40min/20MB cap) and returns **two formats**: a simple subtitle transcript (plain text + SRT cues) and an advanced multi-speaker version (speaker-attributed timed segments; diarization is Gemini-only — JSON-mode audio transcription in `src/services/speech.ts`). The raw provider speech calls live in `src/services/speech.ts`, shared by `ProviderService` (leased BYOK keys, usage metering) and the devcli `vidfarm tts`/`vidfarm stt` commands, which run LOCAL-FIRST on the user's own env key (local ffmpeg demux, no job, no wallet) with `--cloud` as the explicit backup route.
|
|
256
268
|
|
|
257
|
-
Directors call primitives when they want a one-off asset before feeding it into a template. Templates call primitives internally as workflow steps. All primitives are metered against the caller's wallet.
|
|
269
|
+
Directors call primitives when they want a one-off asset before feeding it into a template. Templates call primitives internally as workflow steps. All primitives are metered against the caller's wallet (BYOK AI calls excepted — e.g. tts/stt model spend rides the caller's own provider key; only compute like the stt demux step wallet-bills).
|
|
258
270
|
|
|
259
271
|
## Billing
|
|
260
272
|
|
|
@@ -265,6 +277,7 @@ Key mechanics:
|
|
|
265
277
|
- Every wallet mutation carries a `tracer` (idempotency key)
|
|
266
278
|
- Provider spend is passed through at cost — no markup on AI provider calls
|
|
267
279
|
- Platform infrastructure (cloud render, GhostCut, storage) has a small safety buffer applied
|
|
280
|
+
- Clip hunts bill **AWS compute only** (`clip_scan_lambda` GB-seconds + `step_functions_standard` transitions, keyed to `jobId = scan_id`); the hunt's AI tagging runs on the caller's own provider keys (BYOK) and never touches the wallet
|
|
268
281
|
- Failed jobs are refunded automatically when the failure is on the platform side
|
|
269
282
|
- **Local `vidfarm serve` render is free + unguarded** — `publishCompositionToRenderer` runs the render in-process (`runPrimitiveJobInProcess`) and skips the wallet/paid guard when no cloud state machine is wired; only cloud render/templates are paid + wallet-guarded
|
|
270
283
|
|
|
@@ -6834,7 +6834,7 @@ export function renderLibraryPage(input) {
|
|
|
6834
6834
|
.library-shell { display: grid; grid-template-rows: auto minmax(0, 1fr); gap: 12px; }
|
|
6835
6835
|
.library-toolbar {
|
|
6836
6836
|
display: grid;
|
|
6837
|
-
grid-template-columns: minmax(
|
|
6837
|
+
grid-template-columns: auto minmax(200px, 1fr) auto auto;
|
|
6838
6838
|
align-items: center;
|
|
6839
6839
|
gap: 10px;
|
|
6840
6840
|
padding: 10px 12px;
|
|
@@ -7031,8 +7031,7 @@ export function renderLibraryPage(input) {
|
|
|
7031
7031
|
text-overflow: ellipsis;
|
|
7032
7032
|
white-space: nowrap;
|
|
7033
7033
|
}
|
|
7034
|
-
.library-
|
|
7035
|
-
.library-history-short { display: none; }
|
|
7034
|
+
.library-toolbar .view-toggle { justify-self: start; }
|
|
7036
7035
|
.library-empty { display: grid; place-items: center; min-height: 260px; border: 1px dashed rgba(191, 164, 109, 0.36); border-radius: 18px; color: #617087; background: rgba(255,252,247,.62); }
|
|
7037
7036
|
${renderReadyPostScheduleStyles()}
|
|
7038
7037
|
@media (max-width: 1180px) {
|
|
@@ -7145,24 +7144,27 @@ export function renderLibraryPage(input) {
|
|
|
7145
7144
|
.library-body { padding-top: 12px; }
|
|
7146
7145
|
.library-shell { min-width: 0; }
|
|
7147
7146
|
.library-list-shell { overflow: visible; padding-right: 0; }
|
|
7148
|
-
.library-
|
|
7147
|
+
.library-options summary {
|
|
7149
7148
|
min-height: 38px;
|
|
7150
7149
|
width: 100%;
|
|
7151
7150
|
padding: 0 8px;
|
|
7152
7151
|
white-space: nowrap;
|
|
7153
7152
|
font-size: 0.86rem;
|
|
7154
7153
|
}
|
|
7155
|
-
.library-
|
|
7156
|
-
flex: 0 0
|
|
7157
|
-
|
|
7154
|
+
.library-toolbar .view-toggle {
|
|
7155
|
+
flex: 0 0 auto;
|
|
7156
|
+
order: -1;
|
|
7157
|
+
}
|
|
7158
|
+
.library-toolbar .view-toggle-tab {
|
|
7159
|
+
min-height: 38px;
|
|
7160
|
+
padding: 0 12px;
|
|
7161
|
+
font-size: 0.82rem;
|
|
7158
7162
|
}
|
|
7159
7163
|
.library-options {
|
|
7160
7164
|
flex: 0 0 76px;
|
|
7161
7165
|
width: 76px;
|
|
7162
7166
|
min-width: 0;
|
|
7163
7167
|
}
|
|
7164
|
-
.library-history-full { display: none; }
|
|
7165
|
-
.library-history-short { display: inline; }
|
|
7166
7168
|
.library-selection-count {
|
|
7167
7169
|
display: none;
|
|
7168
7170
|
min-width: 0;
|
|
@@ -7247,8 +7249,11 @@ export function renderLibraryPage(input) {
|
|
|
7247
7249
|
<div class="frame-body library-body">
|
|
7248
7250
|
<section class="library-shell">
|
|
7249
7251
|
<div class="library-toolbar">
|
|
7252
|
+
<div class="view-toggle" role="tablist" aria-label="Library view">
|
|
7253
|
+
<a class="view-toggle-tab is-active" role="tab" aria-current="page" href="${escapeAttribute(withAccountQuery("/library", input.currentAccountId))}">Finished</a>
|
|
7254
|
+
<a class="view-toggle-tab" role="tab" href="${escapeAttribute(withAccountQuery("/clips", input.currentAccountId))}">Clips</a>
|
|
7255
|
+
</div>
|
|
7250
7256
|
<input class="library-search" data-library-search type="search" placeholder="Search published renders" autocomplete="off">
|
|
7251
|
-
<a class="button secondary library-history-link" href="${escapeAttribute(withAccountQuery("/job-runs", input.currentAccountId))}"><span class="library-history-full">Render History</span><span class="library-history-short">Runs</span></a>
|
|
7252
7257
|
<details class="library-options">
|
|
7253
7258
|
<summary>Options</summary>
|
|
7254
7259
|
<div class="library-options-panel">
|