@mevdragon/vidfarm-devcli 0.19.0 → 0.20.0

package/.agents/skills/editor-capabilities/SKILL.md

@@ -63,7 +63,7 @@ A full worked example is in `references/re-theme-walkthrough.md`.
 
 A heavy scenes-axis replace needs footage. Source it in **cost order**, and be proactive: when a big scene re-work is asked for but no footage is given, offer to hunt a source *before* defaulting to expensive AI generation.
 
-1. **The user's own library** — `browse_files` search/list over `/raws` (footage) and `/files` (durable assets). Free.
+1. **The user's own library** — `browse_files` search/list over `/raws` (footage) and `/files` (durable assets). Free. **You do not need the user to paste a URL.** When they say "use my raws" / "it's in my file directory" / name a clip, `browse_files action=search` (or `list path='/raws'`) to find it, then `action=read` with its `file_id` — the read's **`view_url` IS the public, ready media URL**. Drop that `view_url` straight into `set_layer_media` `src`. Never stall asking for a URL you can look up, and never pass a placeholder `src` like `"please-ignore"`.
 2. **HUNT new raws** out of a long-form source (podcast/VOD/webinar or any YouTube/TikTok/IG/X URL): `POST /clips/scan` (aka `/raws/scan`) with `{ source_url|temp_file_id|s3_key, prompt, aspect, target_duration_sec?, ranges?, avoid_text? }` — async (202 + scan_id), bills **AWS compute only** (the AI tagging runs on the user's own keys, never billed). Then browse `/raws/feed?source=<id>` and reuse the picks (`set_layer_media` in place, `add_layer`/`generate_layer intent:"add"` for net-new scenes).
 3. **`generate_layer` AI generation** — the expensive last resort, only for scenes no real clip can cover. AI *image* gen is cheap (use freely); AI *video* gen is expensive — ask permission / confirm budget first.
 
@@ -115,7 +115,7 @@ You usually already know a clip's aspect from how you sourced it (a hunted raw's
 **Generate & export**
 - `generate_layer` — submit an AI generate job, drop a placeholder in the target slot, auto-swap the finished media (no manual poll). `intent`: `fill_gap` (+ start/duration on a `timeline_gaps` span), `replace_layer` (+ `replace_layer_key` — timing/geometry copied), or `add`. Set `aspect_ratio` to the canvas.
 - **Transparent overlays ("Vox-style") — `POST /api/v1/primitives/images/create-overlay`** via `http_request`. This is the go-to way to mint a floating illustration / prop / icon / cut-out character to animate over the video: one job generates an AI image on a forced flat key-color background and chroma-keys it out to a transparent PNG (`result.primary_file_url`). Body `{ tracer, payload: { prompt, aspect_ratio?, key_color?, prompt_attachments? } }` — describe only the subject (the background instruction is injected for you). Then `add_layer` the finished URL as an overlay image and animate it (`ken_burns`/`set_layer_keyframes`/`set_transitions`). Reach for it *often* when a scene wants a transparent graphic element rather than a full-frame still. Cheap-remove sibling for an image already on a flat background: `POST /api/v1/primitives/images/remove-background-greenscreen` (`key_color`/`tolerance`/`softness`); use RapidAPI `images/remove-background` only for messy-background photos.
-- `export_composition` — same as the Export button; queues the render on AWS Lambda. The MP4 appears as `last_export_url` in a later `editor_context`; never fabricate the URL.
+- `export_composition` — same as the Export button; queues the render on AWS Lambda. The MP4 appears as `last_export_url` in a later `editor_context`; never fabricate the URL. **Confirm first — never auto-render.** Rendering is slow, costs money, and pops a status modal, so by default STOP after an edit, let the user review the live preview, and *ask* ("Happy with the preview? Render the final MP4?") before firing this. Only export unprompted when the user pre-authorized it ("edit and just render it"). A render started, then the user can keep working — multiple renders are tracked in the Viral DNA panel's "N Rendering…" badge and its render-history modal.
 - `fork_composition` — branch a NEW editable copy and open it (only on an explicit user branch/duplicate/"save as" request — see Forking). `fork_from:"current"` keeps all current edits; `fork_from:"default"` starts fresh from the template's original default. Same as the ⋯ menu's **New fork** and `POST /api/v1/compositions/:forkId/fork { from }`.
 
 ## Reading `editor_context` — the fields that matter
@@ -125,7 +125,7 @@ You usually already know a clip's aspect from how you sourced it (a hunted raw's
 - `composition_context` (the viral DNA — read before writing any copy).
 - `editor_harness` (the **style-execution brief** — read before any nontrivial edit; see below).
 - `composition_width`/`_height`/`aspect_ratio` (request matching aspect on generate so media fills the canvas), `composition_duration_seconds`.
-- `layers[]` — each with `layer_key`/`element_id`/`slug`, `track` (= z-index), start/duration/geometry, current `object_fit`/`object_position` (the media's fit framing — read it before re-cropping an aspect-mismatched clip), `transition`/`transition_out`/`ken_burns`/`animation`, and `is_full_canvas`/`is_timeline_proxy` (a proxy renders 100%×100% even if shown tiny — trust the flag on replacement).
+- `layers[]` — each with `layer_key`/`element_id`/`slug`, `kind` (video/image/audio/caption/text), current `src`, `track` (= z-index), start/duration/geometry, current `object_fit`/`object_position` (the media's fit framing — read it before re-cropping an aspect-mismatched clip), `transition`/`transition_out`/`ken_burns`/`animation`, and `is_full_canvas`/`is_timeline_proxy` (a proxy renders 100%×100% even if shown tiny — trust the flag on replacement). **This is the ONLY valid source of layer keys.** To swap a clip's media, copy the EXACT `layer_key` (or `slug`) of the *video/image* layer from here — do not invent a semantic scene name (`growth_examples`, `mistakes_reflection`) unless that exact string is a key/slug in this array, and don't target a scene label/group when you mean the video inside it. If `set_layer_media` returns "No change applied" or "Layer not found", the error lists the real media layers — retry with one of those exact keys, and never report a swap done until the next `editor_context` shows the layer's `src` changed.
 - `timeline_gaps[]` — precomputed blank/black spans; map a user's timestamp to the covering gap.
 - `pending_generations[]` — AI gens still rendering (don't double-fire; report progress; `status:"error"` means it failed).
 - `cast[]` — recurring people for character consistency (use `reference_url` on generate).

package/.agents/skills/music/SKILL.md

@@ -0,0 +1,416 @@
+---
+name: music
+description: Generate music using ElevenLabs Music API. Use when creating instrumental tracks, songs with lyrics, background music, jingles, or any AI-generated music composition. Supports prompt-based generation, composition plans for granular control, and detailed output with metadata.
+license: MIT
+compatibility: Requires internet access and an ElevenLabs API key (ELEVENLABS_API_KEY).
+metadata: {"openclaw": {"requires": {"env": ["ELEVENLABS_API_KEY"]}, "primaryEnv": "ELEVENLABS_API_KEY"}}
+---
+
+# ElevenLabs Music Generation
+
+Generate music from text prompts - supports instrumental tracks, songs with lyrics, and fine-grained control via composition plans.
+
+> **Setup:** See [Installation Guide](references/installation.md). For JavaScript, use `@elevenlabs/*` packages only.
+
+All examples below default to `music_v2`, the current generation model. Pass `model_id="music_v1"` only when explicitly requested to.
+
+## Quick Start
+
+### Python
+
+```python
+from elevenlabs import ElevenLabs
+
+client = ElevenLabs()
+
+audio = client.music.compose(
+    prompt="A chill lo-fi hip hop beat with jazzy piano chords",
+    music_length_ms=30000,
+    model_id="music_v2",
+)
+
+with open("output.mp3", "wb") as f:
+    for chunk in audio:
+        f.write(chunk)
+```
+
+### TypeScript
+
+```typescript
+import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js";
+import { createWriteStream } from "fs";
+
+const client = new ElevenLabsClient();
+const audio = await client.music.compose({
+  prompt: "A chill lo-fi hip hop beat with jazzy piano chords",
+  musicLengthMs: 30000,
+  modelId: "music_v2",
+});
+audio.pipe(createWriteStream("output.mp3"));
+```
+
+### cURL
+
+```bash
+curl -X POST "https://api.elevenlabs.io/v1/music" \
+  -H "xi-api-key: $ELEVENLABS_API_KEY" -H "Content-Type: application/json" \
+  -d '{"prompt": "A chill lo-fi beat", "music_length_ms": 30000, "model_id": "music_v2"}' \
+  --output output.mp3
+```
+
+## Methods
+
+| Method | Description |
+|--------|-------------|
+| `music.compose` | Generate audio from a prompt or composition plan |
+| `music.stream` | Stream audio chunks as they are generated (paid plans) |
+| `music.composition_plan.create` | Generate a structured plan for fine-grained control |
+| `music.compose_detailed` | Generate audio + composition plan + metadata; pass `store_for_inpainting=True` to enable inpainting |
+| `music.compose_detailed_stream` | Stream audio plus composition plan, metadata, and optional word timestamps as Server-Sent Events |
+| `music.video_to_music` | Generate background music from one or more uploaded video files |
+| `music.upload` | Upload an audio file for later inpainting workflows, optionally extracting its composition plan or word-level timestamps |
+
+See [API Reference](references/api_reference.md) for full parameter details.
+
+`music.upload` is available to enterprise clients with access to the inpainting feature.
+
+## Video to Music
+
+Generate background music from uploaded video clips via
+[`POST /v1/music/video-to-music`](https://elevenlabs.io/docs/api-reference/music/video-to-music)
+(`client.music.video_to_music`). This is separate from prompt-based
+[`music.compose`](https://elevenlabs.io/docs/api-reference/music/compose) (`POST /v1/music`).
+
+The API combines videos in order, accepts an optional natural-language description, and lets you
+steer style with up to 10 tags such as `upbeat` or `cinematic`. This endpoint still defaults to
+`music_v1`; pass `model_id="music_v2"` to use the newer model.
+
+### Python
+
+```python
+from elevenlabs import ElevenLabs
+
+client = ElevenLabs()
+
+audio = client.music.video_to_music(
+    videos=["trailer.mp4"],
+    description="Build suspense, then resolve with a warm cinematic finish.",
+    tags=["cinematic", "suspenseful", "uplifting"],
+    model_id="music_v2",
+)
+
+with open("video-score.mp3", "wb") as f:
+    for chunk in audio:
+        f.write(chunk)
+```
+
+### TypeScript
+
+```typescript
+import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js";
+import { createReadStream, createWriteStream } from "fs";
+
+const client = new ElevenLabsClient();
+
+const audio = await client.music.videoToMusic({
+  videos: [createReadStream("trailer.mp4")],
+  description: "Build suspense, then resolve with a warm cinematic finish.",
+  tags: ["cinematic", "suspenseful", "uplifting"],
+  modelId: "music_v2",
+});
+
+audio.pipe(createWriteStream("video-score.mp3"));
+```
+
+### cURL
+
+```bash
+curl -X POST "https://api.elevenlabs.io/v1/music/video-to-music" \
+  -H "xi-api-key: $ELEVENLABS_API_KEY" \
+  -F "videos=@trailer.mp4" \
+  -F "description=Build suspense, then resolve with a warm cinematic finish." \
+  -F "tags=cinematic" \
+  -F "tags=suspenseful" \
+  -F "tags=uplifting" \
+  -F "model_id=music_v2" \
+  --output video-score.mp3
+```
+
+Constraints from the current API schema:
+
+- Upload 1-10 video files per request
+- Keep total combined upload size at or below 200 MB
+- Keep total combined video duration at or below 600 seconds
+- Use `description` for high-level musical direction and `tags` for concise style cues
+
+## Composition Plans
+
+`music_v2` composition plans are an ordered list of `chunks`. Each chunk specifies its own
+`text` (section label, lyrics, inline cues), `duration_ms`, `positive_styles`, `negative_styles`,
+and `context_adherence` (`low`, `medium`, or `high`, default `high`). Up to 30 chunks per plan,
+each 3,000–120,000 ms, total length 3 s to 10 minutes.
+
+Generate a plan first, edit it, then compose:
+
+```python
+plan = client.music.composition_plan.create(
+    prompt="An epic orchestral piece building to a climax",
+    music_length_ms=60000,
+    model_id="music_v2",
+)
+
+# Edit chunks in place
+plan["chunks"][0]["text"] = "[Intro]\nQuiet strings rising"
+
+audio = client.music.compose(
+    composition_plan=plan,
+    model_id="music_v2",
+)
+```
+
+```typescript
+const plan = await client.music.compositionPlan.create({
+  prompt: "An epic orchestral piece building to a climax",
+  musicLengthMs: 60000,
+  modelId: "music_v2",
+});
+
+plan.chunks[0].text = "[Intro]\nQuiet strings rising";
+
+const audio = await client.music.compose({
+  compositionPlan: plan,
+  modelId: "music_v2",
+});
+```
+
+Or hand-build a plan to control lyrics and style per section:
+
+```python
+composition_plan = {
+    "chunks": [
+        {
+            "text": "[Verse]\nWalking down an empty street",
+            "duration_ms": 15000,
+            "positive_styles": ["pop", "upbeat", "female vocals", "acoustic guitar"],
+            "negative_styles": ["dark", "slow"],
+            "context_adherence": "high",
+        },
+        {
+            "text": "[Chorus]\nThis is my moment",
+            "duration_ms": 15000,
+            "positive_styles": ["powerful vocals", "full band"],
+            "negative_styles": [],
+            "context_adherence": "high",
+        },
+    ]
+}
+
+audio = client.music.compose(composition_plan=composition_plan, model_id="music_v2")
+```
+
+```typescript
+const compositionPlan = {
+  chunks: [
+    {
+      text: "[Verse]\nWalking down an empty street",
+      durationMs: 15000,
+      positiveStyles: ["pop", "upbeat", "female vocals", "acoustic guitar"],
+      negativeStyles: ["dark", "slow"],
+      contextAdherence: "high",
+    },
+    {
+      text: "[Chorus]\nThis is my moment",
+      durationMs: 15000,
+      positiveStyles: ["powerful vocals", "full band"],
+      negativeStyles: [],
+      contextAdherence: "high",
+    },
+  ],
+};
+
+const audio = await client.music.compose({
+  compositionPlan,
+  modelId: "music_v2",
+});
+```
+
+Put broader characteristics (genre, instrumentation, vocal style) in `positive_styles`, not in
+`text`. The first chunk's styles set the overall tone — include 6–7 styles there.
+
+## Output Formats
+
+Use the `output_format` query parameter on compose, detailed compose, or stream requests to select
+the generated audio format. `auto` chooses a model-appropriate MP3 format; for `music_v2`, it
+selects `mp3_48000_192`. Higher-bitrate MP3 options include `mp3_48000_240` and `mp3_48000_320`.
+
+## Streaming
+
+For paid plans, stream audio chunks as they are generated instead of waiting for the full file:
+
+```python
+from io import BytesIO
+
+stream = client.music.stream(
+    prompt="A driving synthwave track with arpeggiated leads",
+    music_length_ms=30000,
+    model_id="music_v2",
+)
+
+buffer = BytesIO()
+for chunk in stream:
+    if chunk:
+        buffer.write(chunk)
+```
+
+```typescript
+const stream = await client.music.stream({
+  prompt: "A driving synthwave track with arpeggiated leads",
+  musicLengthMs: 30000,
+  modelId: "music_v2",
+});
+
+const chunks: Buffer[] = [];
+for await (const chunk of stream) {
+  chunks.push(chunk);
+}
+```
+
+### Detailed streaming
+
+Use detailed streaming when the application needs generated music metadata while audio is still
+arriving. `POST /v1/music/detailed/stream` accepts the same prompt or composition-plan body as
+detailed compose, streams `text/event-stream`, and can include word timestamps with
+`with_timestamps`.
+
+```bash
+curl -N -X POST "https://api.elevenlabs.io/v1/music/detailed/stream?output_format=auto" \
+  -H "xi-api-key: $ELEVENLABS_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{"prompt": "A bright indie pop hook with warm guitars", "music_length_ms": 30000, "model_id": "music_v2", "with_timestamps": true}'
+```
+
+## Inpainting
+
+Inpainting edits or extends a stored song by mixing **audio reference chunks** (unchanged slices
+of a stored song) with new **generation chunks** in a single composition plan. 
+
+Step 1 — get a `song_id`, either by storing a fresh generation or uploading existing audio:
+
+```python
+# Option A: keep a generation for later editing
+result = client.music.compose_detailed(
+    prompt="An upbeat pop song with verse and chorus",
+    music_length_ms=60000,
+    model_id="music_v2",
+    store_for_inpainting=True,
+)
+song_id = result.song_id
+
+# Option B: upload an existing track and extract its plan
+uploaded = client.music.upload(
+    file=open("my-song.mp3", "rb"),
+    extract_composition_plan="music_v2",
+)
+song_id = uploaded.song_id
+composition_plan = uploaded.composition_plan
+```
+
+```typescript
+import { createReadStream } from "fs";
+
+// Option A: keep a generation for later editing
+const result = await client.music.composeDetailed({
+  prompt: "An upbeat pop song with verse and chorus",
+  musicLengthMs: 60000,
+  modelId: "music_v2",
+  storeForInpainting: true,
+});
+let songId = result.songId;
+
+// Option B: upload an existing track and extract its plan
+const uploaded = await client.music.upload({
+  file: createReadStream("my-song.mp3"),
+  extractCompositionPlan: "music_v2",
+});
+songId = uploaded.songId;
+const compositionPlan = uploaded.compositionPlan;
+```
+
+Step 2 — compose a plan that references the stored audio and regenerates the part you want to
+change:
+
+```python
+plan = {
+    "chunks": [
+        {"song_id": song_id, "range": {"start_ms": 0, "end_ms": 30000}},
+        {
+            "text": "[Chorus]\nWe're rising up tonight",
+            "duration_ms": 30000,
+            "positive_styles": ["bigger drums", "layered vocals", "anthemic"],
+            "negative_styles": ["sparse"],
+            "context_adherence": "high",
+        },
+    ]
+}
+
+audio = client.music.compose(composition_plan=plan, model_id="music_v2")
+```
+
+```typescript
+const plan = {
+  chunks: [
+    { songId, range: { startMs: 0, endMs: 30000 } },
+    {
+      text: "[Chorus]\nWe're rising up tonight",
+      durationMs: 30000,
+      positiveStyles: ["bigger drums", "layered vocals", "anthemic"],
+      negativeStyles: ["sparse"],
+      contextAdherence: "high",
+    },
+  ],
+};
+
+const audio = await client.music.compose({
+  compositionPlan: plan,
+  modelId: "music_v2",
+});
+```
+
+To match the feel of a stored slice without copying it, attach a `conditioning_ref` (up to
+30,000 ms) plus a `condition_strength` of `low`, `medium`, `high`, or `xhigh` to a generation
+chunk. Conditioning placed on the first chunk influences every later chunk.
+
+See [API Reference](references/api_reference.md) for the full inpainting parameter list.
+
+## Content Restrictions
+
+- Cannot reference specific artists, bands, or copyrighted lyrics
+- `bad_prompt` errors include a `prompt_suggestion` with alternative phrasing
+- `bad_composition_plan` errors include a `composition_plan_suggestion`
+
+## Error Handling
+
+```python
+try:
+    audio = client.music.compose(prompt="...", music_length_ms=30000)
+except Exception as e:
+    print(f"API error: {e}")
+```
+
+```typescript
+try {
+  const audio = await client.music.compose({
+    prompt: "...",
+    musicLengthMs: 30000,
+  });
+} catch (err) {
+  console.error("API error:", err);
+}
+```
+
+Common errors: 401 (invalid key), 422 (invalid params), 429 (rate limit).
+
+## References
+
+- [Installation Guide](references/installation.md)
+- [API Reference](references/api_reference.md)