@mevdragon/vidfarm-devcli 0.20.5 → 0.20.8

package/.agents/skills/vidfarm-director/references/primitives.md

@@ -0,0 +1,265 @@
+## Primitive: image_remove_background
+
+Remove the background from any image URL. Result is a transparent PNG/WebP stored at a durable Vidfarm URL you can reuse in compositions or downstream primitives.
+
+- `POST /api/v1/primitives/images/remove-background`
+- Body: `{ "source_image_url": "https://...", "output_format": "webp" | "png" | "jpeg" }` (`output_format` defaults to `webp`)
+- Response: standard primitive job; poll for completion, then read `image.file_url` / `primary_file_url`
+- Billing: flat pass-through fee via RapidAPI (`rapidapi_remove_background` cost center)
+- Reach for this only for an **arbitrary photo with a messy background**. If the image already sits on a **flat solid background** (anything you generated on a controlled backdrop), use the far cheaper `image_remove_background_greenscreen` below instead.
+
+Example:
+
+```bash
+curl -X POST "$VIDFARM_BASE/api/v1/primitives/images/remove-background" \
+  -H "vidfarm-api-key: $VIDFARM_API_KEY" \
+  -H "content-type: application/json" \
+  -d '{"source_image_url": "https://cdn.example.com/photo.jpg"}'
+```
+
+## Primitive: image_remove_background_greenscreen
+
+Cheap **local chroma-key** background removal — no third-party API, just `sharp` doing per-pixel color-distance keying, so it costs a fraction of `image_remove_background`. Use it whenever the source sits on a **flat, solid, evenly-lit background**: a green screen, or any single fill color you pass via `key_color` (e.g. `#FFFFFF` for a white background, `#0047BB` for blue). The result is a transparent PNG/WebP at a durable Vidfarm URL.
+
+- `POST /api/v1/primitives/images/remove-background-greenscreen` (flat alias: `POST /api/v1/primitives/remove-background-greenscreen`)
+- Body: `{ "tracer": "...", "payload": { "source_image_url": "https://...", "key_color"?: "#00FF00", "tolerance"?: 0.3, "softness"?: 0.1, "despill"?: true, "output_format"?: "png" | "webp" }, "webhook_url"?: "..." }`
+  - `key_color` — the background color to key out (hex / `rgb(...)` / named `green`/`white`/`blue`/`black`). Default `#00FF00`.
+  - `tolerance` (0–1, default `0.3`) — how close a pixel must be to `key_color` to be removed. Raise it if fringe survives, lower it if the subject is being eaten.
+  - `softness` (0–1, default `0.1`) — feather band beyond `tolerance` for anti-aliased edges.
+  - `despill` (default `true`) — suppresses the residual key-color rim on the subject's edges (only applied for chromatic keys like green/blue).
+- Response: standard primitive job; poll for completion, then read `image.file_url` / `primary_file_url`. `keyed_fraction` in the output tells you what share of pixels were removed (near-0 usually means the wrong `key_color` or too-low `tolerance`).
+- Billing: small compute-only wallet fee (`greenscreen_chroma_key` cost center) — cheaper than the RapidAPI route.
+
+## Primitive: media_overlay ("create media overlay" — Vox-style)
+
+**The go-to way to mint a transparent overlay graphic to float over a composition.** So-called "Vox-style" animations are really just AI-generated illustrations with transparent backgrounds animated in HTML over the timeline — this primitive is that two-step process fused into one job: it (1) generates an AI image of your subject on a **forced flat key-color background** (using the caller's own BYOK image keys, with the greenscreen instruction injected into the prompt automatically) and (2) chroma-keys that background out, returning a ready-to-composite transparent PNG/WebP. **Prefer this over generate-then-manually-remove**, and reach for it *frequently* whenever a scene wants a floating illustration, prop, icon, sticker, or cut-out character.
+
+- `POST /api/v1/primitives/images/create-overlay` (flat alias: `POST /api/v1/primitives/create-media-overlay`)
+- Body: `{ "tracer": "...", "payload": { "prompt": "a cartoon rocket ship, flat vector illustration", "provider"?: "...", "model"?: "...", "prompt_attachments"?: ["https://..."], "aspect_ratio"?: "1:1", "image_size"?: "1K"|"2K"|"4K", "key_color"?: "#00FF00", "tolerance"?: 0.3, "softness"?: 0.1, "despill"?: true, "output_format"?: "png"|"webp" }, "webhook_url"?: "..." }`
+  - Just describe the **subject** in `prompt` — do NOT describe the background; the primitive appends the flat key-color background requirement for you.
+  - `key_color` lets you key against something other than green if your subject is itself green-heavy (e.g. key against `#FF00FF` magenta for a green frog).
+  - The chroma-key params (`tolerance`/`softness`/`despill`) behave exactly as in `image_remove_background_greenscreen`.
+- Response: `primary_file_url` is the finished transparent overlay; `greenscreen_source_url` is the raw pre-key frame (kept so you can re-key at a different tolerance without paying for another generation).
+- Billing: small platform wallet fee (`media_overlay` cost center) **plus** the BYOK image-generation cost on the caller's own provider key.
+- Placement: once the job settles, `add_layer` (or the editor's `generate_layer`/`add_layer`) the `primary_file_url` as an image layer over the video and animate it (Ken Burns, keyframes, entrance transitions) like any other overlay.
+
+Example:
+
+```bash
+curl -X POST "$VIDFARM_BASE/api/v1/primitives/images/create-overlay" \
+  -H "vidfarm-api-key: $VIDFARM_API_KEY" \
+  -H "content-type: application/json" \
+  -d '{"tracer": "vox-overlay", "payload": {"prompt": "a friendly cartoon lightbulb mascot, flat vector illustration", "aspect_ratio": "1:1"}}'
+```
+
+## Primitive: video_remove_captions
+
+Remove burned-in captions/subtitles/on-screen text from any video URL (GhostCut-powered — the same pipeline auto-decompose uses for its caption-free mirror, exposed as a standalone reusable primitive). Result is a durable caption-free MP4 stored at a Vidfarm URL.
+
+- `POST /api/v1/primitives/videos/remove-captions` (alias: `POST /api/v1/primitives/remove-video-captions`)
+- Body: `{ "tracer": "...", "payload": { "source_video_url": "https://..." }, "webhook_url"?: "..." }`
+- Response: standard primitive job (`202 { job_id }`). The job probes the source, submits to GhostCut, polls to completion server-side (typically a few minutes), and mirrors the result. Poll `GET /api/v1/primitives/jobs/:jobId` until `status: "succeeded"`, then read `primary_file_url` / `video.file_url` (`captionsRemovedVideoUrl` in the output).
+- Billing: `~$0.10 per 30 seconds` of source video (per-chunk, 1-chunk minimum; `ghostcut_subtitle_removal` cost center), plus the small probe cost.
+- For the CURRENT fork's decompose flow, don't re-run this primitive — the caption-free mirror already exists via `GET /api/v1/compositions/:forkId/remove-video-captions`.
+- **Short clips only — hard cap ~15 minutes** (`GHOSTCUT_MAX_DURATION_SEC`, default 900s). Never point this at a long-form source: for "clips without captions" from a long video, run a raws hunt with `avoid_text: true` (scene selection — see "Raws") and, if needed, apply this primitive to the individual FINISHED raws.
+
+Example:
+
+```bash
+curl -X POST "$VIDFARM_BASE/api/v1/primitives/videos/remove-captions" \
+  -H "vidfarm-api-key: $VIDFARM_API_KEY" \
+  -H "content-type: application/json" \
+  -d '{"tracer": "demo-remove-captions", "payload": {"source_video_url": "https://cdn.example.com/clip.mp4"}}'
+```
+
+## Primitive: media_dedupe
+
+Apply subtle camouflage transforms (zoom, tilt, rotate, saturation, playback speed, contrast, brightness, hue rotate, blur, horizontal flip, tint overlay) to any source **image or video** URL. Useful when reposting existing media and you want small perceptual differences to reduce duplicate-detection.
+
+- `POST /api/v1/primitives/media/dedupe`
+- Body: `{ "tracer": "...", "payload": { ...fields... }, "webhook_url"?: "..." }`
+  - Note: webhook delivery is not yet active — `webhook_url` is accepted and persisted on the job but never fired. Poll the job endpoints (`GET /api/v1/primitives/jobs/:jobId`) for completion.
+- Response: standard primitive job. Poll to completion, then read `primary_file_url` (also `video.file_url` for MP4 or `image.file_url` for stills)
+- Billing: metered as a HyperFrames render (via `hyperframes_lambda` on prod cloud render; free on a local serve box)
+
+Payload fields:
+
+- `source_media_url` (required, URL) — the image or video to transform
+- `media_type` (`"image" | "video"`, optional) — auto-detected from URL extension if omitted (`.mp4/.mov/.webm/.m4v` → video, else image)
+- `effects` (optional object). All fields optional; defaults camouflage lightly:
+  - `zoom` (default `1.04`) — scale factor
+  - `tilt` (default `3`) — degrees of 3D X-axis tilt (perspective 1000px)
+  - `rotate` (default `3`) — degrees of 2D rotation
+  - `saturation` (default `1.05`)
+  - `speed` (default `1.05`, video only) — playback rate, also compresses output duration accordingly
+  - `horizontal_flip` (default `false`)
+  - `contrast` (default `1.05`)
+  - `brightness` (default `1.05`)
+  - `hue_rotate` (default `0`) — degrees
+  - `blur` (default `0`) — pixels
+- `tint_color` (default `"#FF8C00"`), `tint_opacity` (default `0.08`) — subtle color overlay
+- `width` (default `1080`), `height` (default `1920`) — output canvas
+- `duration_ms` — output duration for video; if omitted, `fallback_duration_ms` (default `5000`) is used
+- `object_fit` (`"cover" | "contain" | "fill" | "none" | "scale-down"`, default `"cover"`)
+- `background_color` (default `"#000000"`) — visible when `object_fit` leaves letterboxing
+- `muted` (default `false`), `volume` (default `1`) — audio pass-through on video
+- `output_format` (`"png" | "jpeg" | "webp"`, default `"png"`) — image mode only; video mode always outputs MP4
+
+Video example (camouflage a reused clip):
+
+```bash
+curl -X POST "$VIDFARM_BASE/api/v1/primitives/media/dedupe" \
+  -H "vidfarm-api-key: $VIDFARM_API_KEY" \
+  -H "content-type: application/json" \
+  -d '{
+    "tracer": "dedupe-2026-07-01-abc",
+    "payload": {
+      "source_media_url": "https://cdn.example.com/reel.mp4",
+      "media_type": "video",
+      "duration_ms": 8500,
+      "effects": {
+        "zoom": 1.05,
+        "tilt": 2,
+        "rotate": -2,
+        "speed": 1.03,
+        "hue_rotate": 4,
+        "horizontal_flip": true
+      },
+      "tint_color": "#00A3FF",
+      "tint_opacity": 0.06
+    }
+  }'
+```
+
+Image example:
+
+```bash
+curl -X POST "$VIDFARM_BASE/api/v1/primitives/media/dedupe" \
+  -H "vidfarm-api-key: $VIDFARM_API_KEY" \
+  -H "content-type: application/json" \
+  -d '{
+    "tracer": "dedupe-still-01",
+    "payload": {
+      "source_media_url": "https://cdn.example.com/photo.jpg",
+      "media_type": "image",
+      "effects": { "zoom": 1.06, "rotate": -1, "hue_rotate": 8 },
+      "output_format": "webp"
+    }
+  }'
+```
+
+## Primitive: music (text → music)
+
+Generate music (instrumental, songs with lyrics, background beds, jingles, scores) via ElevenLabs. **Default this on freely — music is a core primitive.** `use_wallet_credits` defaults **true**: it runs on vidfarm's platform ElevenLabs key and bills the customer's wallet. Recommend keeping it on; set it false only to save wallet credits or to use the customer's OWN saved ElevenLabs key.
+
+- `POST /api/v1/primitives/music/generate` (alias: `POST /api/v1/primitives/music`)
+- Body: `{ "tracer": "...", "payload": { "prompt": "...", "music_length_ms"?: 30000, "composition_plan"?: {...}, "model"?: "...", "use_wallet_credits"?: true }, "webhook_url"?: "..." }`
+  - `prompt` — describe the vibe/genre/instrumentation. `music_length_ms` ≤ 300000 (5 min), default 30000.
+  - `composition_plan` (optional) — ElevenLabs section-by-section plan for granular lyric/style control (see the loadable `music` skill pack).
+- Result: durable mp3 (`primary_file_url` / `audio.file_url`). Mount as its own `<audio>` layer ~0.1–0.2 under narration.
+- devcli: `vidfarm music "<prompt>" --length 30` (`--own-key` to use your ElevenLabs key).
+
+## Primitive: tts (text → speech)
+
+Generate spoken narration audio from text. **`use_wallet_credits` defaults true** → high-quality ElevenLabs narration on vidfarm's platform key, billed to the wallet — the recommended default. Set it false to use the customer's OWN ElevenLabs key or a BYOK OpenAI/Gemini/OpenRouter key (BYOK is never wallet-billed). The **voice style is promptable** via `instructions` (tone/pacing/accent/emotion/persona). Result is a durable Vidfarm audio URL (mp3/wav) ready to drop into a composition as an audio layer.
+
+- `POST /api/v1/primitives/audio/speech` (alias: `POST /api/v1/primitives/tts`)
+- Body: `{ "tracer": "...", "payload": { "text": "...", "voice"?: "...", "instructions"?: "...", "use_wallet_credits"?: true, "provider"?: "openai" | "gemini" | "openrouter", "model"?: "...", "output_format"?: "mp3" | "wav" }, "webhook_url"?: "..." }`
+  - `text` (required, ≤8000 chars) — the words to speak, verbatim. Aliases: `input`, `script`.
+  - `voice` (optional) — on the ElevenLabs path, an ElevenLabs **voice_id** (or friendly name george/sarah/…); on the BYOK path a provider preset (OpenAI `alloy`/…; Gemini `Kore`/…). **Default a sensible voice and tell the user they can pick from many** — list them with `GET /api/v1/primitives/audio/voices` (or `vidfarm voices`).
+  - `instructions` (optional) — the voice-STYLE prompt. Aliases: `style`, `style_instructions`, `voice_style`.
+  - `output_format` defaults to `mp3` (Gemini/ElevenLabs may answer in wav — read `audio.content_type`).
+- Response: standard primitive job. Poll to completion, then read `primary_file_url` / `audio.file_url`.
+- Billing: wallet-billed on the platform ElevenLabs key (default); free/BYOK when `use_wallet_credits: false` and the caller's own key serves it.
+- devcli: `vidfarm tts "…" --style "…"` runs LOCAL-FIRST on your own env key (no job); add `--cloud` for the ElevenLabs platform job (`--own-key` = your key). `vidfarm voices` lists voices.
+
+## Primitive: audio/voices (list ElevenLabs voices)
+
+`GET /api/v1/primitives/audio/voices` returns the ElevenLabs voice catalog — `{ scope, default_voice_id, voice_library_url, voices: [{ voice_id, name, category, labels, description, preview_url }] }`. Default scope is vidfarm's platform account; add `?use_wallet_credits=false` to list the voices on the customer's OWN saved ElevenLabs key. Pick a `voice_id` and pass it as `voice` to `/audio/speech`. devcli: `vidfarm voices` (`--own-key` for the user's account).
+
+Example:
+
+```bash
+curl -X POST "$VIDFARM_BASE/api/v1/primitives/audio/speech" \
+  -H "vidfarm-api-key: $VIDFARM_API_KEY" \
+  -H "content-type: application/json" \
+  -d '{
+    "tracer": "demo-tts",
+    "payload": {
+      "text": "Welcome back. Today we are testing the three most viral hooks.",
+      "voice": "coral",
+      "instructions": "energetic UGC creator, conversational, slightly breathless"
+    }
+  }'
+```
+
+## Primitive: stt (video/audio → transcript)
+
+Transcribe speech from **any video or audio URL** on the caller's saved AI provider keys (video sources are demuxed to audio automatically via the platform ffmpeg seam). One job returns **BOTH formats**:
+
+1. **Simple subtitle version** — the plain transcript (`output.text`, `transcript.txt`) plus timed SRT cues (`output.subtitles.srt`, `transcript.srt`) with no speaker labels — drop-in captions.
+2. **Advanced multi-speaker version** — speaker-attributed, timestamped segments (`output.segments`, each `{ speaker, start_sec, end_sec, text }`, with `output.speakers` listing the cast) for multi-narration sources like podcasts, interviews, and skits — stored durably as `transcript.json`.
+
+- `POST /api/v1/primitives/audio/transcribe` (alias: `POST /api/v1/primitives/stt`)
+- Body: `{ "tracer": "...", "payload": { "source_url": "https://...", "diarize"?: true, "word_timestamps"?: false, "use_wallet_credits"?: true, "language"?: "en", "prompt"?: "...", "provider"?: "gemini" | "openai" | "openrouter", "model"?: "...", "media_type"?: "auto" | "video" | "audio" }, "webhook_url"?: "..." }`
+  - `source_url` (required) — video OR audio URL. Aliases: `source_video_url`, `source_audio_url`, `video_url`, `audio_url`, `url`. For a fork's composition, pick the right source via `GET /api/v1/compositions/:forkId/remove-video-captions` first.
+  - `use_wallet_credits` (default `true`) — ElevenLabs Scribe on the platform key (native diarization + real word timestamps), billed to the wallet. Set false to use the customer's OWN ElevenLabs key or a BYOK gemini/openai/openrouter key.
+  - `diarize` (default `true`) — attribute segments to speakers. On the BYOK path speaker labels need a **Gemini key**; OpenAI/OpenRouter degrade to a single-speaker transcript. ElevenLabs Scribe (the default) diarizes natively.
+  - `prompt` (optional) — domain/vocabulary hint; `language` (optional) — expected language code.
+- Response: standard primitive job. Poll to completion; `output` carries both formats inline (very long transcripts set `text_truncated: true` and keep the full copy in the `transcript.json` artifact), and the artifacts `transcript.json` / `transcript.srt` / `transcript.txt` are durable URLs.
+- Limits: ~40 minutes of speech per job (20MB extracted audio) — trim longer sources with `/videos/trim` or `/audio/trim` and transcribe in parts.
+- Billing: wallet-billed on the platform ElevenLabs key (default); free/BYOK when `use_wallet_credits: false`. The tiny audio-demux compute step always bills (`primitive_media_lambda`), like every media primitive.
+- For the CURRENT fork, the decompose-time transcript may already exist via `video-context.json` — prefer it before paying for a new transcription; use this primitive for other URLs, multi-speaker attribution, or SRT output.
+- devcli: `vidfarm stt <file|url>` runs LOCAL-FIRST (local ffmpeg + your own key); add `--cloud` to run this route instead.
+
+Example:
+
+```bash
+curl -X POST "$VIDFARM_BASE/api/v1/primitives/audio/transcribe" \
+  -H "vidfarm-api-key: $VIDFARM_API_KEY" \
+  -H "content-type: application/json" \
+  -d '{
+    "tracer": "demo-stt",
+    "payload": {
+      "source_url": "https://cdn.example.com/podcast-clip.mp4",
+      "diarize": true,
+      "language": "en"
+    }
+  }'
+```
+
+## Brainstorm primitives
+
+The `brainstorm/*` primitives are the strategy toolkit. They are reusable, billable AI reasoning steps — the same family the AI Copilot exposes as chip suggestions. Treat **product placement** as a first-class member of this family, right alongside angles and hooks:
+
+- `POST /api/v1/primitives/brainstorm/coldstart` — `{ payload: { user_message } }` → foundational questionnaire for a customer starting from zero.
+- `POST /api/v1/primitives/brainstorm/awareness_stages` — `{ payload: { offer_description } }` → which Eugene-Schwartz awareness stages to target first.
+- `POST /api/v1/primitives/brainstorm/angles` — `{ payload: { offer_description, problem_awareness, solution_awareness } }` → persuasive angles.
+- `POST /api/v1/primitives/brainstorm/hooks` — `{ payload: { offer_description } }` → many TikTok-native hooks.
+- `POST /api/v1/primitives/brainstorm/product_placement` — `{ payload: { source_video_url, offer_description } }` → **watches the video** and returns concrete, timestamped opportunities to natively place the product.
+
+### Primitive: brainstorm_product_placement
+
+Analyze a source video and identify where a product can be organically placed so it feels native rather than bolted on. This is a **multimodal** primitive: it feeds the actual video to a vision-capable model, so prefer a saved **Gemini** key (native video understanding). OpenRouter multimodal models also work; plain OpenAI chat keys cannot watch video.
+
+- `POST /api/v1/primitives/brainstorm/product_placement`
+- Body: `{ "tracer": "...", "payload": { "source_video_url": "...", "offer_description": "...", "count"?: 8, "provider"?: "gemini" }, "webhook_url"?: "..." }`
+  - `source_video_url` (required, URL) — the exact durable/public video to analyze. Aliases accepted: `video_url`, `source_url`, `url`. For a fork's composition, call `GET /api/v1/compositions/:forkId/remove-video-captions` first and pass the correct URL — the **original video** (`original_source_url`) or the **decomposed video** (`mirrored_url`).
+  - `offer_description` (required) — the product/offer to place. Aliases: `offer`, `description`, `details`.
+  - `count` (optional, 3–30, default 8) — number of opportunities; only send when the user asks for a specific number.
+- Response: standard primitive job. Poll `GET /api/v1/primitives/jobs/:jobId` to completion, then read `result.json` (also surfaced on the job output as `opportunities[]`), where each entry has `timestamp`, `scene`, `placement_type`, `placement_idea`, and `why_it_works`.
+
+You can also satisfy a quick, conversational product-placement question by reasoning over an attached video directly instead of calling this primitive — use the primitive when you want a durable, structured, billable artifact the customer can save and reuse.
+
+```bash
+curl -X POST "$VIDFARM_BASE/api/v1/primitives/brainstorm/product_placement" \
+  -H "vidfarm-api-key: $VIDFARM_API_KEY" \
+  -H "content-type: application/json" \
+  -d '{
+    "tracer": "product-placement-01",
+    "payload": {
+      "source_video_url": "https://cdn.example.com/source.mp4",
+      "offer_description": "A $29/mo AI meal-planning app for busy parents."
+    }
+  }'
+```