@kolbo/mcp 1.71.4 → 1.72.2

package/README.md

@@ -132,6 +132,9 @@ Without the optional skill, the config block alone already exposes every tool
 | `generate_sound` | Text → sound effect. Duration, prompt influence, and per-provider controls (Stable Audio guidance, Kie loop/tempo/key, Seed-Audio voice/speed/volume/pitch + reference audio/image) |
 | `generate_3d` | Text or reference images → 3D model (GLB/FBX/OBJ/USDZ) |
 | `transcribe_audio` | Audio/video URL or file → text + SRT subtitles. Language, speaker diarization, audio-event tagging, and SRT formatting (words/line, lines/subtitle, caption stretch) |
+| `separate_audio_stems` | Audio/video URL (or a Kolbo `generation_id`) → Dialogue / Music / Effects / without-dialogue (M&E) layers. Kolbo's own masking pipeline with a speech classifier on top, so a centred engine or ambience is not handed back mislabelled as dialogue. 5 credits, runs inline |
+| `clean_dialogue_leftovers` | Strip voices still faintly audible in an M&E layer. 17 credits — escalation only, it trades bed fidelity to remove the leak |
+| `separate_ambience` | Pull room tone / atmosphere out of the Effects (or Music) bed as its own layer. 17 credits |
 
 Every image/video/creative-director tool accepts `visual_dna_ids` and `moodboard_id` for character/style consistency across outputs — you can compose `create_visual_dna` → `generate_image` (with the DNA applied server-side) in a single agent turn. `generate_creative_director` also accepts `moodboard_ids` plural for blending.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@kolbo/mcp",
-  "version": "1.71.4",
+  "version": "1.72.2",
   "description": "Kolbo AI MCP Server - Generate images, videos, music, speech, and sound effects from Claude Code",
   "main": "src/index.js",
   "bin": {

package/skill/GENERATED.md

@@ -1,6 +1,6 @@
 # AUTO-GENERATED — do not edit
 
-This tree is mirrored from kolbo-code@c5b2132, the single source of truth.
+This tree is mirrored from kolbo-code@94bfcae, the single source of truth.
 Canonical source: packages/opencode/skills/kolbo/
 Distribution: .github/workflows/sync-skill-to-plugin.yml
 

package/skill/SKILL.md

@@ -76,6 +76,7 @@ For multi-scene / batch work this pairs with `generate_creative_director` (see b
 | Use **Visual DNA** / character consistency / `@name` syntax | `references/workflows/visual-dna.md` |
 | Start or continue a **multi-step production** (storyboard → scenes → final cut) | `references/workflows/production-log.md` |
 | **Transcribe** or **analyze** audio/video | `references/workflows/transcription.md` |
+| **Split a soundtrack into layers** — remove/isolate speech, strip narration, instrumental bed, stems for dubbing | `references/workflows/audio-stems.md` |
 | **Scrape brand/product info** before generating + persist as `.kolbo/brand-kits/<slug>.md` | `references/workflows/research-first.md` |
 | Browse, manage, or present existing **media library** items | `references/workflows/media-library.md` |
 | Confirm **cost** or validate **resolution / aspect / duration** against model caps | `references/workflows/cost-and-validation.md` |
@@ -101,6 +102,9 @@ Each `references/models/*.md` mirrors the matching skill prompt in `kolbo-api/sr
 | `generate_speech` | TTS. Use `list_voices` to pick a voice. |
 | `generate_sound` | Sound effects. |
 | `generate_3d` | 3D models from text / single image / multi-view. Returns GLB/FBX/OBJ/USDZ. |
+| `separate_audio_stems` | Split a soundtrack into Dialogue / Music / Effects / without-dialogue (M&E). The route for removing or isolating speech, instrumental beds, and stems for dubbing. 5cr, inline. See `workflows/audio-stems.md`. |
+| `clean_dialogue_leftovers` | Strip voices still faintly audible in an M&E layer. 17cr — only when the user reports the leak, it trades fidelity. |
+| `separate_ambience` | Pull room tone out of the Effects bed as its own lane. 17cr. |
 
 ### Discovery, Library, Visual DNA, Moodboards, Chat, Publishing
 | Tool | Purpose |

package/skill/references/workflows/audio-stems.md

@@ -0,0 +1,81 @@
+# Audio Stem Separation — splitting a soundtrack into layers
+
+Load this file when the user wants to **pull a soundtrack apart**: remove or isolate speech,
+strip narration, mute the music, get a clean instrumental / M&E bed, prepare a track for
+dubbing or localisation, clean up a podcast, or hand an editor stems.
+
+This is Kolbo's own pipeline, not a vendor passthrough. It is a **masking** separation — the
+layers sum back to the original at −30 to −37 dB, so nothing is resynthesised — with a speech
+classifier layered on top. That classifier is the part that matters: a "vocals" mask is not
+the same thing as dialogue. Engines, impacts and centred ambience land in it at full volume,
+and a plain vocal remover hands those back labelled as speech. When no words are actually
+spoken, Kolbo folds that mask into Effects instead.
+
+## Decision tree
+
+```
+"Remove the vocals / speech / narration"            → separate_audio_stems, hand back the `me` layer
+"Isolate just the dialogue"                          → separate_audio_stems, hand back the `dialogue` layer
+"Get me the instrumental / music bed"                → separate_audio_stems, hand back the `music` layer
+"Give the editor stems" / dubbing / localisation     → separate_audio_stems, hand back ALL layers
+"I can still hear voices in the clean track"         → clean_dialogue_leftovers on the `me` URL
+"I want the room tone / atmosphere on its own"       → separate_ambience on the `sfx` URL
+Just want the words as text, not the audio           → transcribe_audio (references/workflows/transcription.md)
+```
+
+Never reach for `edit_video({ operation: "generate_audio" })` for any of these — that ADDS a
+generated audio layer, it does not take one away.
+
+## The layers
+
+`separate_audio_stems` returns whichever of these the material actually contains:
+
+| type | What it is |
+|---|---|
+| `dialogue` | Speech only. Absent when the clip has no spoken words. |
+| `music` | Score / song bed. Absent when the clip has no music. |
+| `sfx` | Effects and Foley. |
+| `me` | Everything except dialogue ("M&E") — the track you dub over. |
+| `original` | The untouched mix, for reference. |
+
+A missing lane is a real finding, not an error: a talking-head clip with no score genuinely
+has no Music layer, and shipping an empty row plus an `sfx` byte-identical to `me` would be
+two lies instead of one honest pair. Say what came back rather than implying a lane failed.
+
+## Two inputs, one tool
+
+- **`audio_url`** — any public media URL, audio or video: an upload, a Kolbo generation, a
+  direct link. Stateless: nothing is stored against it, there is **no cache**, so calling
+  twice on the same file separates twice and bills twice. Sources up to **15 minutes**; trim
+  longer files (`trim_video`, or ffmpeg locally) and separate each section.
+- **`generation_id`** — a Kolbo video you already generated. The layers are saved onto that
+  generation, so a repeat call returns them free and the web app's mixer sees the same thing.
+
+Prefer `generation_id` whenever the source IS a Kolbo generation — it is cacheable and it
+keeps the API and the web app in agreement.
+
+## Cost and when NOT to escalate
+
+| Tool | Credits | When |
+|---|---|---|
+| `separate_audio_stems` | 5 | Always the first call. |
+| `clean_dialogue_leftovers` | 17 | ONLY when the user actually hears voice bleeding through `me`. |
+| `separate_ambience` | 17 | ONLY when the user wants room tone as its own lane. |
+
+`clean_dialogue_leftovers` is deliberately not automatic. It trades fidelity to do its job —
+the model that removes the leak reconstructs the bed less cleanly (~−12 dB) than the masking
+model that produced it (~−30 dB) — so running it by reflex makes the common case worse to fix
+a rare one. Typical on dense crowd scenes; unnecessary on clean dialogue. Ask, or wait for the
+user to report the leak. Both 17-credit tools bill even when the analysis finds nothing to
+remove (the pass still runs); the response reports `already_clean` / `skipped` when that
+happens, so say so rather than presenting a no-op as a result.
+
+## Practical notes
+
+- These tools run **inline** — no job id, nothing to poll. Each call holds for 20–90 seconds
+  on a typical clip. Don't call `get_generation_status` on the result.
+- Layer URLs are WAV on the Kolbo CDN. Hand the user the URLs directly.
+- A file whose audio track is digital silence is refused **before** any charge (`NO_AUDIO`).
+  That is the system working, not a failure to retry.
+- Local file on the user's machine? Same rule as everywhere else — `upload_media` /
+  `media_upload_widget` / `create_upload_ticket` first, then pass the returned https:// URL.

package/src/index.js

@@ -76,6 +76,7 @@ const { registerReviewTools } = require('./tools/review');
 const { registerVoiceTools } = require('./tools/voices');
 const { registerMusicLibraryTools } = require('./tools/music_library');
 const { registerStockLibraryTools } = require('./tools/stock_library');
+const { registerAudioStemTools } = require('./tools/audio_stems');
 const { registerApps, attachToolWidgetMeta } = require('./apps');
 const { attachToolAnnotations } = require('./toolAnnotations');
 
@@ -162,6 +163,7 @@ function createServer(opts = {}) {
   registerReviewTools(server, client, toolOptions);
   registerMusicLibraryTools(server, client, toolOptions);
   registerStockLibraryTools(server, client, toolOptions);
+  registerAudioStemTools(server, client, toolOptions);
 
   // MCP Apps widget resources (ui://kolbo/*). Registering resources is inert
   // for text-only hosts — they never fetch them.

package/src/toolAnnotations.js

@@ -66,6 +66,7 @@ const DESTRUCTIVE_WRITE = [
   'edit_image', 'edit_video', 'trim_video', 'clone_voice',
   'chat_send_message', 'generate_character_sheet',
   'acquire_clean_music_track', 'import_music_track_to_library',
+  'separate_audio_stems', 'clean_dialogue_leftovers', 'separate_ambience',
 
   // Deletes and whole-value replacement updates are conservatively destructive.
   'delete_voice', 'delete_visual_dna', 'delete_visual_dna_folder',

package/src/tools/_shared.js

@@ -54,7 +54,8 @@ const FILE_INPUT_TOOLS = [
   'generate_elements', 'generate_first_last_frame', 'generate_lipsync',
   'generate_3d', 'edit_image', 'edit_video', 'transcribe_audio',
   'create_visual_dna', 'generate_character_sheet', 'clone_voice',
-  'chat_send_message', 'create_moodboard'
+  'chat_send_message', 'create_moodboard',
+  'separate_audio_stems', 'clean_dialogue_leftovers', 'separate_ambience'
 ];
 
 const REMOTE_FILE_HINT =

package/src/tools/audio_stems.js

@@ -0,0 +1,159 @@
+/* ⛔ BACKWARD COMPATIBILITY: Tool names and arg names below are a PUBLIC
+ * CONTRACT. Never rename, remove, or break an existing tool/arg. Full rules: ../index.js top-of-file.
+ *
+ * Audio stem separation — Kolbo's own pipeline, not a passthrough to one vendor.
+ *
+ * Demucs supplies the masks (its stems sum back to the original at −30 to −37 dB, so nothing
+ * is resynthesised) and a speech classifier supplies the semantic gate Demucs lacks: a
+ * "vocals" mask is NOT the same thing as dialogue, because engines, impacts and centred
+ * ambience land there at full volume. When no words are actually spoken, that mask is folded
+ * back into Effects rather than handed over mislabeled as Dialogue. That gate is why these
+ * tools exist as their own surface instead of a `separate` operation on edit_video.
+ *
+ * These three routes answer INLINE — there is no job id and nothing to poll. Each call holds
+ * its connection for the whole run, so they carry an explicit long request timeout rather
+ * than the client default.
+ */
+
+const { z } = require('zod');
+const { projectIdField } = require('./_shared');
+
+// The API answers inline and its own socket ceiling is 10 minutes. Sit just above that so a
+// server-side failure surfaces as the server's message instead of a client-side abort with
+// no explanation. Matches the pattern used for other long single-request tools.
+const STEMS_TIMEOUT_MS = 11 * 60 * 1000;
+
+const SOURCE_HINT =
+  'Either a public media URL (audio or video — a Kolbo generation, an upload_media result, '
+  + 'or any direct link) OR `generation_id` of a Kolbo video you already generated. '
+  + 'Pass exactly one.';
+
+function summarize(result) {
+  return {
+    layers: (result.layers || []).map((l) => ({ type: l.type, label: l.label, url: l.url })),
+    ...(result.generation_id ? { generation_id: result.generation_id } : {}),
+    ...(typeof result.credits_used === 'number' ? { credits_used: result.credits_used } : {}),
+  };
+}
+
+function textResult(payload) {
+  return { content: [{ type: 'text', text: JSON.stringify(payload, null, 2) }] };
+}
+
+/** One source, two accepted shapes. Reject "both" rather than silently preferring one. */
+function buildBody({ audio_url, generation_id, project_id }, label) {
+  if (!audio_url && !generation_id) {
+    throw new Error(`${label}: provide audio_url (a media URL) or generation_id (a Kolbo video).`);
+  }
+  if (audio_url && generation_id) {
+    throw new Error(`${label}: pass audio_url OR generation_id, not both — they are different sources and the result would be ambiguous.`);
+  }
+  const body = generation_id ? { generation_id } : { audio_url };
+  if (project_id) body.project_id = project_id;
+  return body;
+}
+
+function registerAudioStemTools(server, client) {
+  // ─── separate_audio_stems ─────────────────────────────────────
+  server.tool(
+    'separate_audio_stems',
+    'Split a soundtrack into separate audio layers: Dialogue, Music, Effects, and a '
+    + '"without dialogue" (M&E) track — Kolbo\'s own separation pipeline, built for dubbing, '
+    + 'localisation, podcast cleanup and re-scoring. Use this whenever someone wants to '
+    + 'remove/isolate speech, strip narration, mute the music, get a clean instrumental bed, '
+    + 'or hand an editor stems. A speech classifier decides whether the vocal mask is really '
+    + 'dialogue, so an engine roar or centred ambience is not mislabelled as speech — that is '
+    + 'what makes this better than a plain vocal remover. Costs 5 credits. Runs inline and '
+    + 'returns the finished layer URLs (usually 20–90 seconds); there is nothing to poll. '
+    + 'Sources up to 15 minutes long — trim longer files first. NOTE: there is no cache on the '
+    + 'URL form, so calling it twice on the same file separates twice and bills twice.',
+    {
+      audio_url: z.string().optional()
+        .describe('Public URL of the audio or video to separate. ' + SOURCE_HINT),
+      generation_id: z.string().optional()
+        .describe('ID of a Kolbo video generation to separate instead of a URL. Layers are also saved onto that generation, so a repeat call is free.'),
+      project_id: projectIdField,
+    },
+    async ({ audio_url, generation_id, project_id }) => {
+      const body = buildBody({ audio_url, generation_id, project_id }, 'separate_audio_stems');
+      const result = await client.post('/v1/audio/separate', body, { timeoutMs: STEMS_TIMEOUT_MS });
+      return textResult({
+        ...summarize(result),
+        has_dialogue: result.has_dialogue,
+        has_music: result.has_music,
+        cached: result.cached || false,
+        // The lane names are not self-explanatory to a caller seeing them for the first time.
+        layer_guide: {
+          dialogue: 'Speech only.',
+          music: 'Score / song bed. Absent when the clip has no music.',
+          sfx: 'Effects and Foley.',
+          me: 'Everything except dialogue (M&E) — the track to dub over.',
+          original: 'The untouched mix, for reference.',
+        },
+        next_steps: 'If voices are still faintly audible in the "me" layer, run clean_dialogue_leftovers on its URL. To pull room tone out of the Effects bed, run separate_ambience on it.',
+      });
+    },
+  );
+
+  // ─── clean_dialogue_leftovers ─────────────────────────────────
+  server.tool(
+    'clean_dialogue_leftovers',
+    'Strip voices that are still faintly audible in a "without dialogue" (M&E) track produced '
+    + 'by separate_audio_stems. Use it only when the user actually hears speech bleeding '
+    + 'through — it is not part of the normal flow, and it trades fidelity to do its job '
+    + '(the generative model that removes the leak reconstructs the bed less cleanly than the '
+    + 'masking model that made it). Typical on dense crowd scenes; unnecessary on clean '
+    + 'dialogue. Costs 17 credits, charged even when the track turns out to be already clean '
+    + '(the analysis pass still runs) — the response says which happened. Runs inline.',
+    {
+      audio_url: z.string().optional()
+        .describe('URL of the "me" / without-dialogue layer returned by separate_audio_stems.'),
+      generation_id: z.string().optional()
+        .describe('ID of a Kolbo video whose stems were already separated — cleans that generation\'s M&E layer in place.'),
+      project_id: projectIdField,
+    },
+    async ({ audio_url, generation_id, project_id }) => {
+      const body = buildBody({ audio_url, generation_id, project_id }, 'clean_dialogue_leftovers');
+      const result = await client.post('/v1/audio/clean-dialogue', body, { timeoutMs: STEMS_TIMEOUT_MS });
+      return textResult({
+        ...summarize(result),
+        ...(result.passes !== undefined && { cleanup_passes: result.passes }),
+        ...(result.already_clean !== undefined && { already_clean: result.already_clean }),
+        cached: result.cached || false,
+      });
+    },
+  );
+
+  // ─── separate_ambience ────────────────────────────────────────
+  server.tool(
+    'separate_ambience',
+    'Pull room tone / atmosphere out of an Effects or Music bed, leaving the hard effects '
+    + 'behind — a fourth lane on top of separate_audio_stems, for when someone wants the '
+    + 'ambience of a location on its own, or wants it gone. Run separate_audio_stems first '
+    + 'and pass the resulting "sfx" (or "music") layer URL. Costs 17 credits, charged even '
+    + 'when the bed turns out to have no separable ambience — the response reports '
+    + 'skipped: true in that case. Runs inline.',
+    {
+      audio_url: z.string().optional()
+        .describe('URL of the Effects ("sfx") or Music layer returned by separate_audio_stems.'),
+      source_type: z.enum(['sfx', 'music', 'me']).optional()
+        .describe('Which lane audio_url came from, so the leftover is labelled correctly. Default: "sfx".'),
+      generation_id: z.string().optional()
+        .describe('ID of a Kolbo video whose stems were already separated — picks the right source lane automatically.'),
+      project_id: projectIdField,
+    },
+    async ({ audio_url, source_type, generation_id, project_id }) => {
+      const body = buildBody({ audio_url, generation_id, project_id }, 'separate_ambience');
+      if (source_type && !generation_id) body.source_type = source_type;
+      const result = await client.post('/v1/audio/ambience', body, { timeoutMs: STEMS_TIMEOUT_MS });
+      return textResult({
+        ...summarize(result),
+        ...(result.skipped !== undefined && { skipped: result.skipped }),
+        ...(result.levels && { levels_db: result.levels }),
+        cached: result.cached || false,
+      });
+    },
+  );
+}
+
+module.exports = { registerAudioStemTools };