@kolbo/mcp 1.43.0 → 1.44.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +4 -4
- package/package.json +1 -1
- package/skill/SKILL.md +4 -3
- package/skill/references/models/voice-tts.md +85 -0
- package/src/tools/generate.js +91 -15
package/README.md
CHANGED
|
@@ -127,11 +127,11 @@ Without the optional skill, the config block alone already exposes every tool
|
|
|
127
127
|
| `generate_first_last_frame` | First frame + last frame → interpolated video |
|
|
128
128
|
| `generate_lipsync` | Source image/video + audio → lipsynced video (Sync-3 adds active-speaker selection, emotion, model mode, temperature) |
|
|
129
129
|
| `generate_creative_director` | One brief → N coordinated scenes (image or video) |
|
|
130
|
-
| `generate_music` | Text (+ optional lyrics) → song |
|
|
131
|
-
| `generate_speech` | Text + voice → spoken audio |
|
|
132
|
-
| `generate_sound` | Text → sound effect |
|
|
130
|
+
| `generate_music` | Text (+ optional lyrics) → song. Style, title, negative tags, length, and Suno fine-controls (style weight, weirdness, audio weight, persona / singing voice) |
|
|
131
|
+
| `generate_speech` | Text + voice → spoken audio. Full expressive/style control: free-form voice direction (Google/Gemini), preset styles + emotions (DeepDub / MiniMax), speed, and per-provider voice settings (ElevenLabs stability/similarity/style, DeepDub accent/variance/tempo, MiniMax pitch/volume/intensity/timbre) |
|
|
132
|
+
| `generate_sound` | Text → sound effect. Duration, prompt influence, and per-provider controls (Stable Audio guidance, Kie loop/tempo/key, Seed-Audio voice/speed/volume/pitch + reference audio/image) |
|
|
133
133
|
| `generate_3d` | Text or reference images → 3D model (GLB/FBX/OBJ/USDZ) |
|
|
134
|
-
| `transcribe_audio` | Audio/video URL or file → text + SRT subtitles |
|
|
134
|
+
| `transcribe_audio` | Audio/video URL or file → text + SRT subtitles. Language, speaker diarization, audio-event tagging, and SRT formatting (words/line, lines/subtitle, caption stretch) |
|
|
135
135
|
|
|
136
136
|
Every image/video/creative-director tool accepts `visual_dna_ids` and `moodboard_id` for character/style consistency across outputs — you can compose `create_visual_dna` → `generate_image` (with the DNA applied server-side) in a single agent turn. `generate_creative_director` also accepts `moodboard_ids` plural for blending.
|
|
137
137
|
|
package/package.json
CHANGED
package/skill/SKILL.md
CHANGED
|
@@ -103,9 +103,9 @@ Each `references/models/*.md` mirrors the matching skill prompt in `kolbo-api/sr
|
|
|
103
103
|
| `generate_elements` | Reference-driven video. **Primary route for DNA → video.** |
|
|
104
104
|
| `generate_first_last_frame` | Keyframe interpolation between two frames. |
|
|
105
105
|
| `generate_lipsync` | Lipsync audio to an image or video face. |
|
|
106
|
-
| `generate_music` | Music generation (Suno + variants). |
|
|
107
|
-
| `generate_speech` | TTS. Use `list_voices` to pick a voice. |
|
|
108
|
-
| `generate_sound` | Sound effects. |
|
|
106
|
+
| `generate_music` | Music generation (Suno + variants). Controls: `style`, `title`, `lyrics`, `instrumental`, `vocal_gender`, `negative_tags`, `duration_seconds`, plus Suno fine-controls (`style_weight`, `weirdness`, `audio_weight`, `persona_id`, `singing_dna_id`/`singing_voice_id`). |
|
|
107
|
+
| `generate_speech` | TTS. Use `list_voices` to pick a voice, then steer delivery with per-provider style/emotion controls — see `references/models/voice-tts.md`. |
|
|
108
|
+
| `generate_sound` | Sound effects. Controls: `duration`, `prompt_influence`, plus provider-specific `cfg_strength` (Stable Audio), `sound_loop`/`sound_tempo`/`sound_key` (Kie), and Seed-Audio `seed_*` params. |
|
|
109
109
|
| `generate_3d` | 3D models from text / single image / multi-view. Returns GLB/FBX/OBJ/USDZ. |
|
|
110
110
|
|
|
111
111
|
### Discovery, Library, Visual DNA, Moodboards, Chat, App Builder, Publishing
|
|
@@ -162,6 +162,7 @@ A user-named tool — in any language — overrides every other rule. Recognized
|
|
|
162
162
|
- User named one → use it. Model identifiers resolve leniently — shorthand like `"z-image"` or `"nano banana 2"` auto-resolves to the exact identifier, so don't over-engineer exact-id lookups (`list_models` is still authoritative for constraints, caps, and pricing).
|
|
163
163
|
- Auto-select → only from "Auto-selectable" section (models with a `summary`). Cheapest fit. Prefer `[RECOMMENDED]` when cost is similar.
|
|
164
164
|
- Never auto-select from "Named-only" section.
|
|
165
|
+
- **ALWAYS pass an explicit `model` — never omit it.** Omitting routes to the backend "Smart Select" auto-router, which we deliberately avoid: it hides the model choice and the generation used to show just "Auto" instead of the model that ran. YOU pick the specific model (steps above); only omit `model` when the user explicitly asks for auto-pick, or for chat media analysis (video/audio `media_urls`, which routes to Gemini vision).
|
|
165
166
|
4. **Validate inputs** against model caps — see `references/workflows/cost-and-validation.md`.
|
|
166
167
|
5. **How calls work**: each tool blocks until generation is fully complete. Images: seconds. Video: minutes. Multiple tool calls in one response run concurrently. On hosts with live widgets the tool instead returns `submitted` instantly — the card updates on its own; you only need `get_generation_status` when a follow-up step needs the output URLs.
|
|
167
168
|
6. **Checking status — NEVER poll in a loop**: `get_generation_status` takes `wait=true` (blocks server-side until done, ~3 min) and `generation_ids` (check MANY generations in ONE call — returns `all_done` + which are still running). One `wait=true` call replaces any polling loop. If it comes back with some still processing, call it ONCE more with `wait=true` and the remaining ids.
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
# Voice / Text-to-Speech (`generate_speech`) — full style & option reference
|
|
2
|
+
|
|
3
|
+
`generate_speech` turns text into spoken audio. Every voice belongs to a
|
|
4
|
+
**provider** (ElevenLabs, DeepDub, MiniMax, Google/Gemini, OpenAI, Zonos). Each
|
|
5
|
+
provider exposes its own expressive controls. You may pass any control on any
|
|
6
|
+
call — **the engine silently ignores controls that don't apply to the chosen
|
|
7
|
+
voice's provider**, so you never need to branch on provider yourself.
|
|
8
|
+
|
|
9
|
+
## Pick the voice first
|
|
10
|
+
Call `list_voices` (filter by `provider`, `language`, `gender`) and pass the
|
|
11
|
+
returned `voice_id` — or a display name like `"Rachel"`. Cloned/custom voices
|
|
12
|
+
resolve by name too. The voice determines the provider, which determines which
|
|
13
|
+
controls below take effect.
|
|
14
|
+
|
|
15
|
+
## Core params (all providers)
|
|
16
|
+
| Param | Type | Notes |
|
|
17
|
+
|---|---|---|
|
|
18
|
+
| `text` | string (required) | The words to speak. |
|
|
19
|
+
| `voice` | string | Voice id or display name. Default `"Rachel"`. |
|
|
20
|
+
| `model` | string | From `list_models type="text_to_speech"`. Default `eleven_v3`. Usually inferred from the voice — only needed to force a specific engine. |
|
|
21
|
+
| `language` | string | BCP-47 code, e.g. `"en-US"`, `"he-IL"`, `"es-ES"`. |
|
|
22
|
+
| `speaking_speed` | number | `0.5` (slow) – `2.0` (fast). Default `1.0`. Applies to ElevenLabs / OpenAI / Google. |
|
|
23
|
+
| `project_id` | string | Scope into a project (see Projects rules). |
|
|
24
|
+
|
|
25
|
+
## Expressive style / emotion (provider-specific)
|
|
26
|
+
| Param | Provider(s) | Values / notes |
|
|
27
|
+
|---|---|---|
|
|
28
|
+
| `style_instructions` | **Google / Gemini** | Free-form natural-language direction, e.g. `"whisper conspiratorially, slightly amused"`, `"excited sports announcer"`. Max 500 chars. |
|
|
29
|
+
| `selected_style` | **DeepDub**, MiniMax | Preset style. DeepDub: `reading`, `conversational`, `angry`, `breathy`, `panic`, `amused`, `sad`, `whisper`, `singing`, `shout`, `scream`, `mumbling`, `excited`. |
|
|
30
|
+
| `emotion` | **MiniMax** | `happy`, `sad`, `angry`, `fearful`, `disgusted`, `surprised`, `calm`, `fluent`, `whisper`. |
|
|
31
|
+
|
|
32
|
+
## ElevenLabs voice settings
|
|
33
|
+
| Param | Range | Default | Effect |
|
|
34
|
+
|---|---|---|---|
|
|
35
|
+
| `similarity_boost` | 0–1 | 0.75 | Higher hews closer to the source voice. |
|
|
36
|
+
| `style` | 0–1 | 0.5 | Style exaggeration — higher is more expressive/dramatic. |
|
|
37
|
+
| `use_speaker_boost` | bool | true | Speaker-clarity boost. |
|
|
38
|
+
|
|
39
|
+
## DeepDub controls
|
|
40
|
+
| Param | Range | Default | Effect |
|
|
41
|
+
|---|---|---|---|
|
|
42
|
+
| `variance` | 0–1 | 0.2 | More variation / takes. |
|
|
43
|
+
| `tempo` | 0–2 | 1.0 | Pacing multiplier. |
|
|
44
|
+
| `promptBoost` | bool | true | Higher fidelity to the text. |
|
|
45
|
+
| `seed` | int | — | Reproducibility (same seed + inputs → same output). Also honored by Zonos. |
|
|
46
|
+
| `accentControl` | object | — | `{ accentBaseLocale, accentLocale, accentRatio }` — blend an accent. Provide BOTH `accentBaseLocale` (e.g. `"en-US"`) and `accentLocale` (e.g. `"en-GB"`); `accentRatio` 0–1 (default 0.5). |
|
|
47
|
+
| `voiceTitle` | string | — | Display title for a custom/cloned voice. |
|
|
48
|
+
|
|
49
|
+
## MiniMax fine controls
|
|
50
|
+
| Param | Range | Default | Effect |
|
|
51
|
+
|---|---|---|---|
|
|
52
|
+
| `minimax_pitch` | −12 … 12 | 0 | Pitch shift. |
|
|
53
|
+
| `minimax_vol` | 0–10 | 1 | Volume. |
|
|
54
|
+
| `minimax_intensity` | — | — | Voice intensity. |
|
|
55
|
+
| `minimax_timbre` | — | — | Voice timbre. |
|
|
56
|
+
|
|
57
|
+
## Examples
|
|
58
|
+
Neutral ElevenLabs read:
|
|
59
|
+
```
|
|
60
|
+
generate_speech(text="Welcome to Kolbo.", voice="Rachel")
|
|
61
|
+
```
|
|
62
|
+
Whispered, conspiratorial Gemini delivery:
|
|
63
|
+
```
|
|
64
|
+
generate_speech(text="Meet me at midnight.", voice="Kore",
|
|
65
|
+
style_instructions="whisper conspiratorially, slow and breathy")
|
|
66
|
+
```
|
|
67
|
+
Angry DeepDub take, faster:
|
|
68
|
+
```
|
|
69
|
+
generate_speech(text="Get out of my house!", voice="<deepdub voice>",
|
|
70
|
+
selected_style="angry", tempo=1.2)
|
|
71
|
+
```
|
|
72
|
+
Excited MiniMax with pitch/volume tweaks:
|
|
73
|
+
```
|
|
74
|
+
generate_speech(text="We won the championship!", voice="<minimax voice>",
|
|
75
|
+
emotion="happy", minimax_pitch=3, minimax_vol=6)
|
|
76
|
+
```
|
|
77
|
+
British-accented DeepDub blend:
|
|
78
|
+
```
|
|
79
|
+
generate_speech(text="Good evening.", voice="<deepdub voice>",
|
|
80
|
+
accentControl={ accentBaseLocale: "en-US", accentLocale: "en-GB", accentRatio: 0.7 })
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
## Credits
|
|
84
|
+
~5 credits per 100 characters for most TTS models (Zonos ~3; voice design/clone
|
|
85
|
+
~30 flat). Charged only on success. Use `check_credits` once per conversation.
|
package/src/tools/generate.js
CHANGED
|
@@ -391,22 +391,36 @@ function registerGenerateTools(server, client, options = {}) {
|
|
|
391
391
|
// ─── generate_music ────────────────────────────────────────
|
|
392
392
|
server.tool(
|
|
393
393
|
'generate_music',
|
|
394
|
-
'Generate music from a text description using Kolbo AI. Supports instrumental mode, custom lyrics, style direction, and
|
|
394
|
+
'Generate music from a text description using Kolbo AI. Supports instrumental mode, custom lyrics, style direction, vocal gender, negative tags, song length, and Suno fine-controls (style weight, weirdness, audio weight, persona/singing voice). Default model is Suno. Some controls are Suno-only; the engine ignores controls that do not apply to the chosen model. Returns the final audio URL when complete.',
|
|
395
395
|
{
|
|
396
396
|
prompt: z.string().describe('Text description of the music to generate (e.g., "upbeat electronic dance track with synthesizers")'),
|
|
397
397
|
model: z.string().optional().describe('Model identifier. Use list_models type="music_gen" to see options. Omit for Suno (default).'),
|
|
398
398
|
style: z.string().optional().describe('Music style / genre (e.g., "pop", "rock", "lo-fi", "electronic", "jazz")'),
|
|
399
|
+
title: z.string().optional().describe('Song title. If omitted, one is generated.'),
|
|
399
400
|
instrumental: z.boolean().optional().describe('Generate instrumental only, no vocals. Default: false'),
|
|
400
401
|
lyrics: z.string().optional().describe('Custom lyrics for the song. If omitted, lyrics are generated automatically from the prompt unless instrumental is true.'),
|
|
401
402
|
vocal_gender: z.string().optional().describe('Preferred vocal gender: "male" or "female". Only applies when instrumental is false.'),
|
|
403
|
+
negative_tags: z.string().optional().describe('Styles / sounds to EXCLUDE, comma-separated (e.g. "heavy metal, screaming, distortion"). Suno.'),
|
|
404
|
+
duration_seconds: z.number().optional().describe('Target song length in seconds (length-capable models like ElevenLabs Music). Clamped 5–300. Omit for the model default.'),
|
|
402
405
|
enhance_prompt: z.boolean().optional().describe('Enhance the prompt. Default: true'),
|
|
403
406
|
preset_id: z.string().optional().describe('Preset ID from list_presets type="music" to apply a saved music style preset.'),
|
|
407
|
+
// ── Suno fine controls ──
|
|
408
|
+
style_weight: z.number().optional().describe('Suno: how strongly the style/genre is applied, 0–1.'),
|
|
409
|
+
weirdness: z.number().optional().describe('Suno: creativity / weirdness constraint, 0–1. Higher = more experimental.'),
|
|
410
|
+
audio_weight: z.number().optional().describe('Suno: influence of an audio/persona reference, 0–1.'),
|
|
411
|
+
persona_id: z.string().optional().describe('Suno persona id — reuse a saved singing voice/persona.'),
|
|
412
|
+
use_composition_plan: z.boolean().optional().describe('Suno: enable structured composition planning (verse/chorus structure).'),
|
|
413
|
+
singing_dna_id: z.string().optional().describe('Visual DNA character id whose singing voice to use (must be owned by the caller).'),
|
|
414
|
+
singing_voice_id: z.string().optional().describe('Custom cloned singing-voice id (must be owned by the caller).'),
|
|
404
415
|
project_id: projectIdField
|
|
405
416
|
},
|
|
406
|
-
async ({ prompt, model, style, instrumental, lyrics, vocal_gender, enhance_prompt, preset_id, project_id }) => {
|
|
417
|
+
async ({ prompt, model, style, title, instrumental, lyrics, vocal_gender, negative_tags, duration_seconds, enhance_prompt, preset_id, style_weight, weirdness, audio_weight, persona_id, use_composition_plan, singing_dna_id, singing_voice_id, project_id }) => {
|
|
407
418
|
model = await canonicalModelId(client, model); // lenient id resolution ("z-image" → "z-image/turbo")
|
|
408
419
|
const gen = await client.post('/v1/generate/music', {
|
|
409
|
-
prompt, model, style, instrumental, lyrics, vocal_gender,
|
|
420
|
+
prompt, model, style, title, instrumental, lyrics, vocal_gender, negative_tags,
|
|
421
|
+
duration_seconds, enhance_prompt, preset_id,
|
|
422
|
+
style_weight, weirdness, audio_weight, persona_id, use_composition_plan,
|
|
423
|
+
singing_dna_id, singing_voice_id, project_id
|
|
410
424
|
});
|
|
411
425
|
|
|
412
426
|
if (ui()) return uiGenerating({
|
|
@@ -437,23 +451,53 @@ function registerGenerateTools(server, client, options = {}) {
|
|
|
437
451
|
// ─── generate_speech ───────────────────────────────────────
|
|
438
452
|
server.tool(
|
|
439
453
|
'generate_speech',
|
|
440
|
-
'Convert text to speech using Kolbo AI. Default provider is ElevenLabs. To pick a specific voice by language/gender, call list_voices first and pass the returned voice_id (or a voice display name — both work). Returns the final audio URL when complete.',
|
|
454
|
+
'Convert text to speech using Kolbo AI. Default provider is ElevenLabs. To pick a specific voice by language/gender, call list_voices first and pass the returned voice_id (or a voice display name — both work). Every voice belongs to a provider (ElevenLabs, DeepDub, MiniMax, Google/Gemini, OpenAI, Zonos) and each provider exposes its own expressive/style controls below — the engine ignores any control that does not apply to the chosen voice\'s provider, so it is safe to pass only what you need. Returns the final audio URL when complete.',
|
|
441
455
|
{
|
|
442
456
|
text: z.string().describe('The text to convert to speech'),
|
|
443
457
|
voice: z.string().optional().describe('Voice ID (from list_voices) or voice display name (e.g., "Rachel", "Adam"). Default: "Rachel"'),
|
|
444
458
|
model: z.string().optional().describe('Model identifier. Use list_models type="text_to_speech" to see options. Default: eleven_v3'),
|
|
445
459
|
language: z.string().optional().describe('Language code (e.g., "en-US", "he-IL", "es-ES"). Default: "en-US"'),
|
|
460
|
+
// ── Expressive style / emotion (provider-specific) ──
|
|
461
|
+
style_instructions: z.string().optional().describe('Google/Gemini voices ONLY. Free-form natural-language voice direction, e.g. "whisper conspiratorially, slightly amused" or "excited sports announcer". Max 500 chars. Ignored by other providers.'),
|
|
462
|
+
selected_style: z.string().optional().describe('DeepDub & MiniMax voices. Preset expressive style/emotion. DeepDub supports: reading, conversational, angry, breathy, panic, amused, sad, whisper, singing, shout, scream, mumbling, excited. Ignored by other providers.'),
|
|
463
|
+
emotion: z.string().optional().describe('MiniMax voices. Emotion: happy, sad, angry, fearful, disgusted, surprised, calm, fluent, whisper.'),
|
|
464
|
+
speaking_speed: z.number().optional().describe('Speech speed 0.5 (slow) – 2.0 (fast). Default 1.0. Applies to ElevenLabs / OpenAI / Google.'),
|
|
465
|
+
// ── ElevenLabs voice settings ──
|
|
466
|
+
similarity_boost: z.number().optional().describe('ElevenLabs voice similarity, 0–1. Default 0.75. Higher hews closer to the original voice.'),
|
|
467
|
+
style: z.number().optional().describe('ElevenLabs style exaggeration, 0–1. Default 0.5. Higher = more expressive/dramatic.'),
|
|
468
|
+
use_speaker_boost: z.boolean().optional().describe('ElevenLabs speaker boost. Default true.'),
|
|
469
|
+
// ── DeepDub controls ──
|
|
470
|
+
variance: z.number().optional().describe('DeepDub voice variance, 0–1. Default 0.2. Higher = more takes/variation.'),
|
|
471
|
+
tempo: z.number().optional().describe('DeepDub tempo multiplier, 0–2. Default 1.0.'),
|
|
472
|
+
promptBoost: z.boolean().optional().describe('DeepDub prompt-fidelity boost. Default true.'),
|
|
473
|
+
seed: z.number().optional().describe('Reproducibility seed (DeepDub / Zonos). Same seed + inputs → same output.'),
|
|
474
|
+
accentControl: z.object({
|
|
475
|
+
accentBaseLocale: z.string().describe('Base accent locale, e.g. "en-US".'),
|
|
476
|
+
accentLocale: z.string().describe('Target accent locale, e.g. "en-GB".'),
|
|
477
|
+
accentRatio: z.number().optional().describe('Blend ratio 0–1. Default 0.5.')
|
|
478
|
+
}).optional().describe('DeepDub accent steering. Provide both base and target locale to blend an accent.'),
|
|
479
|
+
voiceTitle: z.string().optional().describe('DeepDub display title for a custom/cloned voice.'),
|
|
480
|
+
// ── MiniMax fine controls ──
|
|
481
|
+
minimax_pitch: z.number().optional().describe('MiniMax pitch, −12 to 12. Default 0.'),
|
|
482
|
+
minimax_vol: z.number().optional().describe('MiniMax volume, 0–10. Default 1.'),
|
|
483
|
+
minimax_intensity: z.number().optional().describe('MiniMax voice intensity.'),
|
|
484
|
+
minimax_timbre: z.number().optional().describe('MiniMax voice timbre.'),
|
|
446
485
|
project_id: projectIdField
|
|
447
486
|
},
|
|
448
|
-
async ({ text, voice, model, language, project_id }) => {
|
|
487
|
+
async ({ text, voice, model, language, style_instructions, selected_style, emotion, speaking_speed, similarity_boost, style, use_speaker_boost, variance, tempo, promptBoost, seed, accentControl, voiceTitle, minimax_pitch, minimax_vol, minimax_intensity, minimax_timbre, project_id }) => {
|
|
449
488
|
model = await canonicalModelId(client, model); // lenient id resolution ("z-image" → "z-image/turbo")
|
|
450
489
|
const gen = await client.post('/v1/generate/speech', {
|
|
451
|
-
text, voice, model, language,
|
|
490
|
+
text, voice, model, language,
|
|
491
|
+
style_instructions, selected_style, emotion, speaking_speed,
|
|
492
|
+
similarity_boost, style, use_speaker_boost,
|
|
493
|
+
variance, tempo, promptBoost, seed, accentControl, voiceTitle,
|
|
494
|
+
minimax_pitch, minimax_vol, minimax_intensity, minimax_timbre,
|
|
495
|
+
project_id
|
|
452
496
|
});
|
|
453
497
|
|
|
454
498
|
if (ui()) return uiGenerating({
|
|
455
499
|
tool: 'generate_speech', kind: 'audio', gen, client, model, prompt: text,
|
|
456
|
-
settings: { voice: voice || 'Rachel' }
|
|
500
|
+
settings: { voice: voice || 'Rachel', style: selected_style || emotion || style_instructions }
|
|
457
501
|
});
|
|
458
502
|
|
|
459
503
|
const result = await pollUntilDone(client, gen.generation_id, {
|
|
@@ -478,18 +522,34 @@ function registerGenerateTools(server, client, options = {}) {
|
|
|
478
522
|
// ─── generate_sound ────────────────────────────────────────
|
|
479
523
|
server.tool(
|
|
480
524
|
'generate_sound',
|
|
481
|
-
'Generate sound effects (not music, not speech) from a text description using Kolbo AI. Use this for ambient sounds, foley, impacts, atmospheres, UI sounds, etc. For music use generate_music; for voice use generate_speech. Returns the final audio URL when complete.',
|
|
525
|
+
'Generate sound effects (not music, not speech) from a text description using Kolbo AI. Use this for ambient sounds, foley, impacts, atmospheres, UI sounds, etc. For music use generate_music; for voice use generate_speech. Beyond the core prompt/duration, per-provider controls are available (Stable Audio guidance, Kie loop/tempo/key, Seed-Audio voice/speed/volume/pitch + reference audio/image); the engine ignores controls that do not apply to the chosen model. Returns the final audio URL when complete.',
|
|
482
526
|
{
|
|
483
527
|
prompt: z.string().describe('Text description of the sound effect (e.g., "thunder clap with rain", "door creaking open", "futuristic UI beep")'),
|
|
484
528
|
model: z.string().optional().describe('Model identifier. Use list_models type="text_to_sound" to see options. Default: elevenlabs-sound-effects-v1'),
|
|
485
529
|
duration: z.number().optional().describe('Duration in seconds. Omit for automatic duration.'),
|
|
486
|
-
prompt_influence: z.number().optional().describe('
|
|
530
|
+
prompt_influence: z.number().optional().describe('ElevenLabs: how strongly the prompt guides the generation (0–1). Default: 0.5. Lower = more creative freedom; higher = more literal.'),
|
|
531
|
+
// ── FAL Stable Audio / mmaudio ──
|
|
532
|
+
cfg_strength: z.number().optional().describe('FAL (Stable Audio 3 / mmaudio): classifier-free guidance strength. Higher hews closer to the prompt.'),
|
|
533
|
+
// ── Kie ──
|
|
534
|
+
sound_loop: z.boolean().optional().describe('Kie: generate a seamlessly looping sound.'),
|
|
535
|
+
sound_tempo: z.number().optional().describe('Kie: tempo control.'),
|
|
536
|
+
sound_key: z.string().optional().describe('Kie: musical key / scale.'),
|
|
537
|
+
// ── FAL Seed Audio ──
|
|
538
|
+
seed_voice: z.string().optional().describe('FAL Seed-Audio: voice to use.'),
|
|
539
|
+
seed_speed: z.number().optional().describe('FAL Seed-Audio: speed multiplier, 0.5–2.0.'),
|
|
540
|
+
seed_volume: z.number().optional().describe('FAL Seed-Audio: volume, 0–1.'),
|
|
541
|
+
seed_pitch: z.number().optional().describe('FAL Seed-Audio: pitch shift in semitones.'),
|
|
542
|
+
seed_reference_audio_urls: z.array(z.string()).optional().describe('FAL Seed-Audio: up to 3 reference audio URLs to condition the sound.'),
|
|
543
|
+
seed_reference_image_url: z.string().optional().describe('FAL Seed-Audio: a reference image URL to condition the sound.'),
|
|
487
544
|
project_id: projectIdField
|
|
488
545
|
},
|
|
489
|
-
async ({ prompt, model, duration, prompt_influence, project_id }) => {
|
|
546
|
+
async ({ prompt, model, duration, prompt_influence, cfg_strength, sound_loop, sound_tempo, sound_key, seed_voice, seed_speed, seed_volume, seed_pitch, seed_reference_audio_urls, seed_reference_image_url, project_id }) => {
|
|
490
547
|
model = await canonicalModelId(client, model); // lenient id resolution ("z-image" → "z-image/turbo")
|
|
491
548
|
const gen = await client.post('/v1/generate/sound', {
|
|
492
|
-
prompt, model, duration, prompt_influence,
|
|
549
|
+
prompt, model, duration, prompt_influence,
|
|
550
|
+
cfg_strength, sound_loop, sound_tempo, sound_key,
|
|
551
|
+
seed_voice, seed_speed, seed_volume, seed_pitch,
|
|
552
|
+
seed_reference_audio_urls, seed_reference_image_url, project_id
|
|
493
553
|
});
|
|
494
554
|
|
|
495
555
|
if (ui()) return uiGenerating({
|
|
@@ -963,23 +1023,39 @@ function registerGenerateTools(server, client, options = {}) {
|
|
|
963
1023
|
// ─── transcribe_audio ──────────────────────────────────────
|
|
964
1024
|
server.tool(
|
|
965
1025
|
'transcribe_audio',
|
|
966
|
-
'Transcribe audio or video into text + SRT subtitles. Source can be a URL or an absolute local file path. Returns the full text, SRT content, duration, and download URLs for .srt/.txt files. Works on both audio-only files (mp3, wav, m4a) and videos with audio tracks (mp4, mov, webm).',
|
|
1026
|
+
'Transcribe audio or video into text + SRT subtitles. Source can be a URL or an absolute local file path. Returns the full text, SRT content, duration, and download URLs for .srt/.txt files. Works on both audio-only files (mp3, wav, m4a) and videos with audio tracks (mp4, mov, webm). Supports language selection, speaker diarization, audio-event tagging, and SRT subtitle formatting controls.',
|
|
967
1027
|
{
|
|
968
1028
|
source: z.string().describe('URL or absolute local path to the audio / video file to transcribe'),
|
|
1029
|
+
language: z.string().optional().describe('Language code of the speech (e.g. "en", "he", "es"). Omit to auto-detect.'),
|
|
1030
|
+
diarize: z.boolean().optional().describe('Detect and label distinct speakers. Default: false.'),
|
|
1031
|
+
tag_audio_events: z.boolean().optional().describe('Tag non-speech audio events (laughter, applause, music) in the transcript. Default: false.'),
|
|
1032
|
+
remove_punctuation: z.boolean().optional().describe('Strip punctuation from the transcript. Default: false.'),
|
|
1033
|
+
generate_srt: z.boolean().optional().describe('Produce SRT + word-by-word SRT subtitle files. Default: true.'),
|
|
1034
|
+
words_per_line: z.number().optional().describe('SRT: max words per subtitle line, 1–18. Default: 12.'),
|
|
1035
|
+
lines_per_subtitle: z.number().optional().describe('SRT: max lines per subtitle cue, 1–4. Default: 2.'),
|
|
1036
|
+
stretch_captions: z.boolean().optional().describe('SRT: extend each cue\'s end time to the next cue\'s start (gap-free subtitles). Default: true.'),
|
|
969
1037
|
project_id: projectIdField
|
|
970
1038
|
},
|
|
971
|
-
async ({ source, project_id }) => {
|
|
1039
|
+
async ({ source, language, diarize, tag_audio_events, remove_punctuation, generate_srt, words_per_line, lines_per_subtitle, stretch_captions, project_id }) => {
|
|
972
1040
|
if (!source) throw new Error('source is required (URL or absolute local path)');
|
|
973
1041
|
|
|
1042
|
+
// Advanced transcription controls forwarded when provided (undefined keys are dropped by the client).
|
|
1043
|
+
const opts = {
|
|
1044
|
+
language, diarize, tag_audio_events, remove_punctuation,
|
|
1045
|
+
generate_srt, words_per_line, lines_per_subtitle, stretch_captions, project_id
|
|
1046
|
+
};
|
|
1047
|
+
|
|
974
1048
|
const isUrl = /^https?:\/\//i.test(source);
|
|
975
1049
|
let startResponse;
|
|
976
1050
|
if (isUrl) {
|
|
977
|
-
startResponse = await client.post('/v1/transcribe', { audio_url: source,
|
|
1051
|
+
startResponse = await client.post('/v1/transcribe', { audio_url: source, ...opts });
|
|
978
1052
|
} else {
|
|
979
1053
|
const resolved = await resolveToBuffer(source, 'audio');
|
|
980
1054
|
const form = new FormData();
|
|
981
1055
|
form.append('file', resolved.buffer, { filename: resolved.filename, contentType: resolved.contentType });
|
|
982
|
-
|
|
1056
|
+
for (const [k, v] of Object.entries(opts)) {
|
|
1057
|
+
if (v !== undefined && v !== null) form.append(k, typeof v === 'boolean' ? String(v) : v);
|
|
1058
|
+
}
|
|
983
1059
|
startResponse = await client.postMultipart('/v1/transcribe', form);
|
|
984
1060
|
}
|
|
985
1061
|
|