@sogni-ai/sogni-protocol 1.0.0-alpha.31 → 1.0.0-alpha.33
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/manifests/generation-tools.json +90 -4
- package/manifests/openai-tools.json +89 -4
- package/package.json +1 -1
- package/schemas/tools/animate_photo.schema.json +4 -2
- package/schemas/tools/generate_speech.schema.json +81 -0
- package/schemas/tools/generate_video.schema.json +3 -2
- package/version.json +1 -1
|
@@ -5,6 +5,7 @@
|
|
|
5
5
|
"generate_image": "../schemas/tools/generate_image.schema.json",
|
|
6
6
|
"generate_video": "../schemas/tools/generate_video.schema.json",
|
|
7
7
|
"generate_music": "../schemas/tools/generate_music.schema.json",
|
|
8
|
+
"generate_speech": "../schemas/tools/generate_speech.schema.json",
|
|
8
9
|
"edit_image": "../schemas/tools/edit_image.schema.json",
|
|
9
10
|
"apply_style": "../schemas/tools/apply_style.schema.json",
|
|
10
11
|
"restore_photo": "../schemas/tools/restore_photo.schema.json",
|
|
@@ -218,6 +219,7 @@
|
|
|
218
219
|
"seedance2-5",
|
|
219
220
|
"minimax-h3-t2v",
|
|
220
221
|
"minimax-h3-t2v-turbo",
|
|
222
|
+
"minimax-h3-fasth3-t2v-turbo",
|
|
221
223
|
"happyhorse-1.1-t2v",
|
|
222
224
|
"happyhorse-1.1-i2v",
|
|
223
225
|
"happyhorse-1.1-r2v",
|
|
@@ -226,7 +228,7 @@
|
|
|
226
228
|
"wan3.0-video",
|
|
227
229
|
"wan3.0-spicy-video"
|
|
228
230
|
],
|
|
229
|
-
"description": "\"ltx25\" (default): LTX 2.5 with native audio; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflow. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. Video model. \"ltx23\": LTX 2.3 rollback with native audio. \"wan22\": quick simple motion without audio. \"minimax-h3-t2v\": standard 20-step MiniMax H3 text-to-video; \"minimax-h3-t2v-turbo\": 4-step Turbo text-to-video.
|
|
231
|
+
"description": "\"ltx25\" (default): LTX 2.5 with native audio; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflow. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. Video model. \"ltx23\": LTX 2.3 rollback with native audio. \"wan22\": quick simple motion without audio. \"minimax-h3-t2v\": standard 20-step MiniMax H3 text-to-video; \"minimax-h3-t2v-turbo\": the existing 4-step LightX2V Turbo text-to-video; \"minimax-h3-fasth3-t2v-turbo\": the separate FastVideo VSA four-step FastH3 engine, about 2x faster and fixed to Euler/simple. All use native audio, fixed 24fps, 5.17-15.08s, and a 768p-class 32px-grid canvas; use animate_photo for H3 image-conditioned modes. Base and Turbo T2V/I2V/FLF2V prompts use the exact ordered fields integrated_multimodal_description, overall_soundscape, and non_diegetic_music; I2V/FLF2V prepend the official alignment line. \"minimax-h3-r2v\": standard 20-step MiniMax H3 reference-to-video; \"minimax-h3-r2v-turbo\": the dedicated LightX2V 4-step Ref2VA Turbo workflow using Euler/simple and a 960x544 default. FastH3 has no R2V mode. Both R2V selectors accept up to 9 images, 3 videos, and 3 audios (12 files total); at least one visual reference (image or video) is required and audio alone is invalid. Select references with referenceImageIndices/referenceVideoIndices/referenceAudioIndices and address them with the official <Subject N>/<Picture N>/<Video N>/<Audio N> semantics. Seedance quality is selected only by model: use \"seedance2-mini\" for Seedance 2.0 Mini or faster/lower-cost 720p iteration, and use \"seedance2\" for the full Seedance 2.0 model, explicit full-quality requests, 1080p/4K requests, or generated/uploaded storyboard images unless the user explicitly asks for a draft or Mini. Do not use Default Media Quality Fast/HQ/Pro or targetResolution to represent Seedance quality. \"seedance2-5\": Seedance 2.5, the newest Seedance generation — 480p and 720p ONLY (it cannot render 1080p or 4K), 4-30s per clip at a fixed 24 fps, native audio, first-and-last-frame conditioning, and a much larger reference budget than the 2.0 family: up to 30 images, 10 videos, and 10 audios, with up to 50 reference media files total, subject to those per-modality caps. Choose \"seedance2-5\" when the user asks for Seedance 2.5, wants a single continuous Seedance clip longer than 15s (2.5 renders up to 30s in one call instead of being split and stitched), or wants a first-and-last-frame Seedance transition. Keep \"seedance2\" for 1080p/4K requests, which Seedance 2.5 cannot satisfy. Seedance supports multimodal loose reference assets: images (up to 9), videos (up to 3), and audios (up to 3), with no more than 12 asset files total. Use @Image1/@Video1/@Audio1 style references in creative briefs when assigning roles. Assign every useful reference asset a role and prefer positive preservation constraints. If an uploaded video is the source clip to transform, upscale, enhance, restyle, or remaster, use video_to_video with controlMode=\"seedance-v2v\" instead of generate_video referenceVideoIndices. Alibaba HappyHorse 1.1 video models (third-party vendor — requires Premium Spark). Select by mode: \"happyhorse-1.1-t2v\" for text-to-video, \"happyhorse-1.1-i2v\" for image-to-video from one first-frame image, and \"happyhorse-1.1-r2v\" for reference-to-video with up to 9 reference images. Resolutions 720P and 1080P; duration 3-15 seconds at 24 fps; native synchronized audio is always generated (do not set generateAudio or negativePrompt). Supported aspect ratios: 16:9, 9:16, 1:1, 4:3, 3:4, 4:5, 5:4, 9:21, 21:9. HappyHorse 1.1 takes image references only and renders a native synchronized audio track (always on; do not set generateAudio or a negative prompt). Pick the model by mode: happyhorse-1.1-t2v for text-to-video (no reference image), happyhorse-1.1-i2v for image-to-video from a single first frame, and happyhorse-1.1-r2v for reference-to-video with 1 to 9 reference images. For r2v, tag the images in the prompt as [Image 1]…[Image 9] and assign each a clear role. HappyHorse does not accept reference videos or reference audios. \"wan3.0-video\" is Alibaba Wan 3 and \"wan3.0-spicy-video\" is MuleRouter w3.0-video. Both render 2-30s or smart duration at fixed 30 fps with optional native audio, provider prompt expansion, 480p/720p/1080p, adaptive/fixed ratios, first/last frames, and up to 10 image/5 video/5 audio references. Only Alibaba wan3.0-video accepts document/web context and watermark. Frame anchors and loose references are mutually exclusive. Do not send negativePrompt; video references are loose conditioning for a new result, not source-video editing or extension."
|
|
230
232
|
},
|
|
231
233
|
"generateAudio": {
|
|
232
234
|
"type": "boolean",
|
|
@@ -287,7 +289,7 @@
|
|
|
287
289
|
"type": "string",
|
|
288
290
|
"minLength": 1
|
|
289
291
|
},
|
|
290
|
-
"description": "Ordered LoRA IDs to apply to a MiniMax H3 render. Use only when the user explicitly asks for a LoRA or for an effect one of these names describes. Stack up to 8 in one request; order matters because the adapters apply in sequence and do not commute. Keep this array positionally aligned with loraStrengths. The first render with an uncached LoRA takes longer to start while the worker downloads it.\n\nAccepted only when videoModel is one of \"minimax-h3-t2v\", \"minimax-h3-t2v-turbo\", \"minimax-h3-r2v\", \"minimax-h3-r2v-turbo\". Every other video model on this tool loads no LoRAs and silently ignores these arrays, so set videoModel to an H3 mode in the same call when the user asks for one.\n\nOne LoRA is published for MiniMax H3 today: h3-realism-people (fal), a realism pass trained on live-action footage of people. It restores skin texture and pores, stray hairs, fabric weave and a fine sensor grain that the base model smooths away, and holds up in close-up. It needs its trigger word: put r34l1sm near the FRONT of the prompt, or the render comes back as ordinary H3 with no error. Exact ranges and any LoRA published since: GET /v1/loras/comfy?modelId=<model>. Do not invent ids."
|
|
292
|
+
"description": "Ordered LoRA IDs to apply to a MiniMax H3 render. Use only when the user explicitly asks for a LoRA or for an effect one of these names describes. Stack up to 8 in one request; order matters because the adapters apply in sequence and do not commute. Keep this array positionally aligned with loraStrengths. The first render with an uncached LoRA takes longer to start while the worker downloads it.\n\nAccepted only when videoModel is one of \"minimax-h3-t2v\", \"minimax-h3-t2v-turbo\", \"minimax-h3-fasth3-t2v-turbo\", \"minimax-h3-r2v\", \"minimax-h3-r2v-turbo\". Every other video model on this tool loads no LoRAs and silently ignores these arrays, so set videoModel to an H3 mode in the same call when the user asks for one.\n\nOne LoRA is published for MiniMax H3 today: h3-realism-people (fal), a realism pass trained on live-action footage of people. It restores skin texture and pores, stray hairs, fabric weave and a fine sensor grain that the base model smooths away, and holds up in close-up. It needs its trigger word: put r34l1sm near the FRONT of the prompt, or the render comes back as ordinary H3 with no error. Exact ranges and any LoRA published since: GET /v1/loras/comfy?modelId=<model>. Do not invent ids."
|
|
291
293
|
},
|
|
292
294
|
"loraStrengths": {
|
|
293
295
|
"type": "array",
|
|
@@ -369,6 +371,88 @@
|
|
|
369
371
|
}
|
|
370
372
|
}
|
|
371
373
|
},
|
|
374
|
+
{
|
|
375
|
+
"type": "function",
|
|
376
|
+
"function": {
|
|
377
|
+
"name": "generate_speech",
|
|
378
|
+
"description": "Turn written words into spoken audio. Use when the user wants something read aloud, a narration, a voiceover, a line of dialogue, an audiobook passage, or a voice cloned from a recording they uploaded. This tool speaks text; it does not compose music — use generate_music for songs and instrumentals.",
|
|
379
|
+
"parameters": {
|
|
380
|
+
"type": "object",
|
|
381
|
+
"properties": {
|
|
382
|
+
"prompt": {
|
|
383
|
+
"type": "string",
|
|
384
|
+
"description": "The exact words to be spoken, verbatim. This is NOT a description of the audio: whatever is written here is read aloud character for character. \"a calm woman reading the news\" would be spoken as those seven words — put that in voiceDescription instead and write the actual news copy here.\n\nLITERAL PROMPT OVERRIDE: If the user explicitly says not to modify the prompt, or to use it exactly/verbatim/as-is, copy the identified prompt text verbatim instead of applying these construction rules unless a hard requirement is missing.\n\nPunctuation is prosody: full stops, commas, question marks and ellipses control pauses and intonation, so keep them. Write numbers, dates, currency and abbreviations the way they should be said (\"nineteen eighty-four\", \"twelve dollars fifty\", \"Doctor Chen\") when the plain form would be ambiguous. Line breaks are not pauses; use punctuation.\n\nLimit 4096 characters, roughly five minutes of speech. A longer script is refused rather than cut off mid-sentence, so split it across several calls at a natural break.",
|
|
385
|
+
"maxLength": 4096
|
|
386
|
+
},
|
|
387
|
+
"model": {
|
|
388
|
+
"type": "string",
|
|
389
|
+
"enum": [
|
|
390
|
+
"voice",
|
|
391
|
+
"clone",
|
|
392
|
+
"design"
|
|
393
|
+
],
|
|
394
|
+
"description": "Which speech model to use, chosen by what the user gave you. \"voice\" (default): one of nine studio voices, optionally restyled by voiceDescription — the right choice for narration, voiceover and dialogue when no particular person is being imitated. \"clone\": reproduces a specific voice from a recording the user uploaded; requires voiceSourceIndex. \"design\": invents a speaker who does not exist from a written description; requires voiceDescription and takes no recording. Pick \"clone\" whenever the user uploads a voice clip and asks for that person, and \"design\" when they describe a speaker instead of choosing one."
|
|
395
|
+
},
|
|
396
|
+
"voice": {
|
|
397
|
+
"type": "string",
|
|
398
|
+
"enum": [
|
|
399
|
+
"serena",
|
|
400
|
+
"vivian",
|
|
401
|
+
"uncle_fu",
|
|
402
|
+
"ryan",
|
|
403
|
+
"aiden",
|
|
404
|
+
"ono_anna",
|
|
405
|
+
"sohee",
|
|
406
|
+
"eric",
|
|
407
|
+
"dylan"
|
|
408
|
+
],
|
|
409
|
+
"description": "Studio voice to speak in. Only used when model=\"voice\". Female: serena (English), vivian (Chinese), ono_anna (Japanese), sohee (Korean). Male: ryan, aiden, eric, dylan (English), uncle_fu (older, Chinese). Every voice speaks all supported languages — the note above describes the accent it carries, not a limit on what it can read. Default: serena."
|
|
410
|
+
},
|
|
411
|
+
"voiceDescription": {
|
|
412
|
+
"type": "string",
|
|
413
|
+
"description": "How the line should be delivered, or who should deliver it. With model=\"voice\" this restyles the chosen voice without changing who it is: \"whispering, close to the microphone\", \"furious\", \"reading a bedtime story\", \"like a sports commentator\". With model=\"design\" this is required and describes a speaker to invent: age, gender, accent, timbre, pace, mood, and recording space — \"a warm, unhurried narrator in her forties with a faint Scottish lilt, close-miked in a quiet room\". Describe one coherent person; contradictory directions produce a voice that shifts mid-sentence. Not accepted with model=\"clone\", where the recording defines the voice. Limit 512 characters.",
|
|
414
|
+
"maxLength": 512
|
|
415
|
+
},
|
|
416
|
+
"voiceSourceIndex": {
|
|
417
|
+
"type": "number",
|
|
418
|
+
"description": "Index of the uploaded audio file holding the voice to clone (0-based, from the uploaded files list, the same numbering other tools use for audio). Required when model=\"clone\" and ignored otherwise. The clip should be three to thirty seconds of one person speaking cleanly, with no music, no second speaker and no heavy room echo; anything past thirty seconds is trimmed.",
|
|
419
|
+
"minimum": 0
|
|
420
|
+
},
|
|
421
|
+
"voiceTranscript": {
|
|
422
|
+
"type": "string",
|
|
423
|
+
"description": "The exact words spoken in the uploaded clip, when model=\"clone\". Supplying it lets the model condition on the recording itself rather than on a speaker fingerprint alone, which is markedly closer to the source — set it whenever the user tells you what the clip says or the transcript is otherwise known. Omitting it still produces a recognisable clone, just a looser one. Limit 1024 characters.",
|
|
424
|
+
"maxLength": 1024
|
|
425
|
+
},
|
|
426
|
+
"language": {
|
|
427
|
+
"type": "string",
|
|
428
|
+
"enum": [
|
|
429
|
+
"auto",
|
|
430
|
+
"english",
|
|
431
|
+
"chinese",
|
|
432
|
+
"japanese",
|
|
433
|
+
"korean",
|
|
434
|
+
"german",
|
|
435
|
+
"french",
|
|
436
|
+
"russian",
|
|
437
|
+
"portuguese",
|
|
438
|
+
"spanish",
|
|
439
|
+
"italian"
|
|
440
|
+
],
|
|
441
|
+
"description": "Language of the script. Default \"auto\", which infers it from the text and is the only setting that reads a code-switched line correctly. Pin a language only when auto mis-reads a name, a loanword, or a passage that is ambiguous between two of them. Cloning is cross-lingual: an English reference clip can read Japanese in the same voice."
|
|
442
|
+
},
|
|
443
|
+
"numberOfVariations": {
|
|
444
|
+
"type": "number",
|
|
445
|
+
"description": "Number of takes (1-16). Each take is a separate read of the same script with the same voice, differing only in delivery. Use more than 1 when the user asks for options or alternate reads. Default: 1.",
|
|
446
|
+
"minimum": 1,
|
|
447
|
+
"maximum": 16
|
|
448
|
+
}
|
|
449
|
+
},
|
|
450
|
+
"required": [
|
|
451
|
+
"prompt"
|
|
452
|
+
]
|
|
453
|
+
}
|
|
454
|
+
}
|
|
455
|
+
},
|
|
372
456
|
{
|
|
373
457
|
"type": "function",
|
|
374
458
|
"function": {
|
|
@@ -655,12 +739,14 @@
|
|
|
655
739
|
"happyhorse-1.1-r2v",
|
|
656
740
|
"minimax-h3-i2v",
|
|
657
741
|
"minimax-h3-i2v-turbo",
|
|
742
|
+
"minimax-h3-fasth3-i2v-turbo",
|
|
658
743
|
"minimax-h3-flf2v",
|
|
659
744
|
"minimax-h3-flf2v-turbo",
|
|
745
|
+
"minimax-h3-fasth3-flf2v-turbo",
|
|
660
746
|
"wan3.0-video",
|
|
661
747
|
"wan3.0-spicy-video"
|
|
662
748
|
],
|
|
663
|
-
"description": "\"ltx25\" (default): LTX 2.5 I2V or first/last-frame video with native audio; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflow. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. Which video model to use. \"ltx23\": LTX 2.3 rollback with native audio. \"wan22\": quick simple motion without audio, up to 10s. \"minimax-h3-i2v\"
|
|
749
|
+
"description": "\"ltx25\" (default): LTX 2.5 I2V or first/last-frame video with native audio; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflow. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. Which video model to use. \"ltx23\": LTX 2.3 rollback with native audio. \"wan22\": quick simple motion without audio, up to 10s. \"minimax-h3-i2v\" is standard MiniMax H3 from one first frame; \"minimax-h3-i2v-turbo\" is the existing 4-step LightX2V Turbo engine; \"minimax-h3-fasth3-i2v-turbo\" is the separate FastVideo VSA four-step FastH3 engine, about 2x faster and fixed to Euler/simple. The matching FLF2V selectors provide standard, LightX2V Turbo, and FastH3 Turbo first/last-frame generation; FastH3 has no R2V mode; use frameRole=\"both\" and provide the end frame. H3 generates native audio at fixed 24fps for 5.17-15.08s and has no negative-prompt input. H3 Base and Turbo prompts use the exact three-field contract and the official mode-specific alignment line. Do not set Seedance here; use generate_video with Seedance references. \"wan3.0-video\" is Alibaba Wan 3 and \"wan3.0-spicy-video\" is MuleRouter w3.0-video. Both render 2-30s or smart duration at fixed 30 fps with optional native audio, provider prompt expansion, 480p/720p/1080p, adaptive/fixed ratios, first/last frames, and up to 10 image/5 video/5 audio references. Only Alibaba wan3.0-video accepts document/web context and watermark. Frame anchors and loose references are mutually exclusive. Do not send negativePrompt; video references are loose conditioning for a new result, not source-video editing or extension."
|
|
664
750
|
},
|
|
665
751
|
"negativePrompt": {
|
|
666
752
|
"type": "string",
|
|
@@ -764,7 +850,7 @@
|
|
|
764
850
|
"type": "string",
|
|
765
851
|
"minLength": 1
|
|
766
852
|
},
|
|
767
|
-
"description": "Ordered LoRA IDs to apply to a MiniMax H3 render. Use only when the user explicitly asks for a LoRA or for an effect one of these names describes. Stack up to 8 in one request; order matters because the adapters apply in sequence and do not commute. Keep this array positionally aligned with loraStrengths. The first render with an uncached LoRA takes longer to start while the worker downloads it.\n\nAccepted only when videoModel is one of \"minimax-h3-i2v\", \"minimax-h3-i2v-turbo\", \"minimax-h3-flf2v\", \"minimax-h3-flf2v-turbo\". Every other video model on this tool loads no LoRAs and silently ignores these arrays, so set videoModel to an H3 mode in the same call when the user asks for one.\n\nOne LoRA is published for MiniMax H3 today: h3-realism-people (fal), a realism pass trained on live-action footage of people. It restores skin texture and pores, stray hairs, fabric weave and a fine sensor grain that the base model smooths away, and holds up in close-up. It needs its trigger word: put r34l1sm near the FRONT of the prompt, or the render comes back as ordinary H3 with no error. Exact ranges and any LoRA published since: GET /v1/loras/comfy?modelId=<model>. Do not invent ids."
|
|
853
|
+
"description": "Ordered LoRA IDs to apply to a MiniMax H3 render. Use only when the user explicitly asks for a LoRA or for an effect one of these names describes. Stack up to 8 in one request; order matters because the adapters apply in sequence and do not commute. Keep this array positionally aligned with loraStrengths. The first render with an uncached LoRA takes longer to start while the worker downloads it.\n\nAccepted only when videoModel is one of \"minimax-h3-i2v\", \"minimax-h3-i2v-turbo\", \"minimax-h3-fasth3-i2v-turbo\", \"minimax-h3-flf2v\", \"minimax-h3-flf2v-turbo\", \"minimax-h3-fasth3-flf2v-turbo\". Every other video model on this tool loads no LoRAs and silently ignores these arrays, so set videoModel to an H3 mode in the same call when the user asks for one.\n\nOne LoRA is published for MiniMax H3 today: h3-realism-people (fal), a realism pass trained on live-action footage of people. It restores skin texture and pores, stray hairs, fabric weave and a fine sensor grain that the base model smooths away, and holds up in close-up. It needs its trigger word: put r34l1sm near the FRONT of the prompt, or the render comes back as ordinary H3 with no error. Exact ranges and any LoRA published since: GET /v1/loras/comfy?modelId=<model>. Do not invent ids."
|
|
768
854
|
},
|
|
769
855
|
"loraStrengths": {
|
|
770
856
|
"type": "array",
|
|
@@ -180,6 +180,7 @@
|
|
|
180
180
|
"seedance2-5",
|
|
181
181
|
"minimax-h3-t2v",
|
|
182
182
|
"minimax-h3-t2v-turbo",
|
|
183
|
+
"minimax-h3-fasth3-t2v-turbo",
|
|
183
184
|
"happyhorse-1.1-t2v",
|
|
184
185
|
"happyhorse-1.1-i2v",
|
|
185
186
|
"happyhorse-1.1-r2v",
|
|
@@ -188,7 +189,7 @@
|
|
|
188
189
|
"wan3.0-video",
|
|
189
190
|
"wan3.0-spicy-video"
|
|
190
191
|
],
|
|
191
|
-
"description": "\"ltx25\" (default): LTX 2.5 with native audio; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflow. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. Video model. \"ltx23\": LTX 2.3 rollback with native audio. \"wan22\": quick simple motion without audio. \"minimax-h3-t2v\": standard 20-step MiniMax H3 text-to-video; \"minimax-h3-t2v-turbo\": 4-step Turbo text-to-video.
|
|
192
|
+
"description": "\"ltx25\" (default): LTX 2.5 with native audio; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflow. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. Video model. \"ltx23\": LTX 2.3 rollback with native audio. \"wan22\": quick simple motion without audio. \"minimax-h3-t2v\": standard 20-step MiniMax H3 text-to-video; \"minimax-h3-t2v-turbo\": the existing 4-step LightX2V Turbo text-to-video; \"minimax-h3-fasth3-t2v-turbo\": the separate FastVideo VSA four-step FastH3 engine, about 2x faster and fixed to Euler/simple. All use native audio, fixed 24fps, 5.17-15.08s, and a 768p-class 32px-grid canvas; use animate_photo for H3 image-conditioned modes. Base and Turbo T2V/I2V/FLF2V prompts use the exact ordered fields integrated_multimodal_description, overall_soundscape, and non_diegetic_music; I2V/FLF2V prepend the official alignment line. \"minimax-h3-r2v\": standard 20-step MiniMax H3 reference-to-video; \"minimax-h3-r2v-turbo\": the dedicated LightX2V 4-step Ref2VA Turbo workflow using Euler/simple and a 960x544 default. FastH3 has no R2V mode. Both R2V selectors accept up to 9 images, 3 videos, and 3 audios (12 files total); at least one visual reference (image or video) is required and audio alone is invalid. Select references with referenceImageIndices/referenceVideoIndices/referenceAudioIndices and address them with the official <Subject N>/<Picture N>/<Video N>/<Audio N> semantics. Seedance quality is selected only by model: use \"seedance2-mini\" for Seedance 2.0 Mini or faster/lower-cost 720p iteration, and use \"seedance2\" for the full Seedance 2.0 model, explicit full-quality requests, 1080p/4K requests, or generated/uploaded storyboard images unless the user explicitly asks for a draft or Mini. Do not use Default Media Quality Fast/HQ/Pro or targetResolution to represent Seedance quality. \"seedance2-5\": Seedance 2.5, the newest Seedance generation — 480p and 720p ONLY (it cannot render 1080p or 4K), 4-30s per clip at a fixed 24 fps, native audio, first-and-last-frame conditioning, and a much larger reference budget than the 2.0 family: up to 30 images, 10 videos, and 10 audios, with up to 50 reference media files total, subject to those per-modality caps. Choose \"seedance2-5\" when the user asks for Seedance 2.5, wants a single continuous Seedance clip longer than 15s (2.5 renders up to 30s in one call instead of being split and stitched), or wants a first-and-last-frame Seedance transition. Keep \"seedance2\" for 1080p/4K requests, which Seedance 2.5 cannot satisfy. Seedance supports multimodal loose reference assets: images (up to 9), videos (up to 3), and audios (up to 3), with no more than 12 asset files total. Use @Image1/@Video1/@Audio1 style references in creative briefs when assigning roles. Assign every useful reference asset a role and prefer positive preservation constraints. If an uploaded video is the source clip to transform, upscale, enhance, restyle, or remaster, use video_to_video with controlMode=\"seedance-v2v\" instead of generate_video referenceVideoIndices. Alibaba HappyHorse 1.1 video models (third-party vendor — requires Premium Spark). Select by mode: \"happyhorse-1.1-t2v\" for text-to-video, \"happyhorse-1.1-i2v\" for image-to-video from one first-frame image, and \"happyhorse-1.1-r2v\" for reference-to-video with up to 9 reference images. Resolutions 720P and 1080P; duration 3-15 seconds at 24 fps; native synchronized audio is always generated (do not set generateAudio or negativePrompt). Supported aspect ratios: 16:9, 9:16, 1:1, 4:3, 3:4, 4:5, 5:4, 9:21, 21:9. HappyHorse 1.1 takes image references only and renders a native synchronized audio track (always on; do not set generateAudio or a negative prompt). Pick the model by mode: happyhorse-1.1-t2v for text-to-video (no reference image), happyhorse-1.1-i2v for image-to-video from a single first frame, and happyhorse-1.1-r2v for reference-to-video with 1 to 9 reference images. For r2v, tag the images in the prompt as [Image 1]…[Image 9] and assign each a clear role. HappyHorse does not accept reference videos or reference audios. \"wan3.0-video\" is Alibaba Wan 3 and \"wan3.0-spicy-video\" is MuleRouter w3.0-video. Both render 2-30s or smart duration at fixed 30 fps with optional native audio, provider prompt expansion, 480p/720p/1080p, adaptive/fixed ratios, first/last frames, and up to 10 image/5 video/5 audio references. Only Alibaba wan3.0-video accepts document/web context and watermark. Frame anchors and loose references are mutually exclusive. Do not send negativePrompt; video references are loose conditioning for a new result, not source-video editing or extension."
|
|
192
193
|
},
|
|
193
194
|
"generateAudio": {
|
|
194
195
|
"type": "boolean",
|
|
@@ -249,7 +250,7 @@
|
|
|
249
250
|
"type": "string",
|
|
250
251
|
"minLength": 1
|
|
251
252
|
},
|
|
252
|
-
"description": "Ordered LoRA IDs to apply to a MiniMax H3 render. Use only when the user explicitly asks for a LoRA or for an effect one of these names describes. Stack up to 8 in one request; order matters because the adapters apply in sequence and do not commute. Keep this array positionally aligned with loraStrengths. The first render with an uncached LoRA takes longer to start while the worker downloads it.\n\nAccepted only when videoModel is one of \"minimax-h3-t2v\", \"minimax-h3-t2v-turbo\", \"minimax-h3-r2v\", \"minimax-h3-r2v-turbo\". Every other video model on this tool loads no LoRAs and silently ignores these arrays, so set videoModel to an H3 mode in the same call when the user asks for one.\n\nOne LoRA is published for MiniMax H3 today: h3-realism-people (fal), a realism pass trained on live-action footage of people. It restores skin texture and pores, stray hairs, fabric weave and a fine sensor grain that the base model smooths away, and holds up in close-up. It needs its trigger word: put r34l1sm near the FRONT of the prompt, or the render comes back as ordinary H3 with no error. Exact ranges and any LoRA published since: GET /v1/loras/comfy?modelId=<model>. Do not invent ids."
|
|
253
|
+
"description": "Ordered LoRA IDs to apply to a MiniMax H3 render. Use only when the user explicitly asks for a LoRA or for an effect one of these names describes. Stack up to 8 in one request; order matters because the adapters apply in sequence and do not commute. Keep this array positionally aligned with loraStrengths. The first render with an uncached LoRA takes longer to start while the worker downloads it.\n\nAccepted only when videoModel is one of \"minimax-h3-t2v\", \"minimax-h3-t2v-turbo\", \"minimax-h3-fasth3-t2v-turbo\", \"minimax-h3-r2v\", \"minimax-h3-r2v-turbo\". Every other video model on this tool loads no LoRAs and silently ignores these arrays, so set videoModel to an H3 mode in the same call when the user asks for one.\n\nOne LoRA is published for MiniMax H3 today: h3-realism-people (fal), a realism pass trained on live-action footage of people. It restores skin texture and pores, stray hairs, fabric weave and a fine sensor grain that the base model smooths away, and holds up in close-up. It needs its trigger word: put r34l1sm near the FRONT of the prompt, or the render comes back as ordinary H3 with no error. Exact ranges and any LoRA published since: GET /v1/loras/comfy?modelId=<model>. Do not invent ids."
|
|
253
254
|
},
|
|
254
255
|
"loraStrengths": {
|
|
255
256
|
"type": "array",
|
|
@@ -331,6 +332,88 @@
|
|
|
331
332
|
}
|
|
332
333
|
}
|
|
333
334
|
},
|
|
335
|
+
{
|
|
336
|
+
"type": "function",
|
|
337
|
+
"function": {
|
|
338
|
+
"name": "generate_speech",
|
|
339
|
+
"description": "Turn written words into spoken audio. Use when the user wants something read aloud, a narration, a voiceover, a line of dialogue, an audiobook passage, or a voice cloned from a recording they uploaded. This tool speaks text; it does not compose music — use generate_music for songs and instrumentals.",
|
|
340
|
+
"parameters": {
|
|
341
|
+
"type": "object",
|
|
342
|
+
"properties": {
|
|
343
|
+
"prompt": {
|
|
344
|
+
"type": "string",
|
|
345
|
+
"description": "The exact words to be spoken, verbatim. This is NOT a description of the audio: whatever is written here is read aloud character for character. \"a calm woman reading the news\" would be spoken as those seven words — put that in voiceDescription instead and write the actual news copy here.\n\nLITERAL PROMPT OVERRIDE: If the user explicitly says not to modify the prompt, or to use it exactly/verbatim/as-is, copy the identified prompt text verbatim instead of applying these construction rules unless a hard requirement is missing.\n\nPunctuation is prosody: full stops, commas, question marks and ellipses control pauses and intonation, so keep them. Write numbers, dates, currency and abbreviations the way they should be said (\"nineteen eighty-four\", \"twelve dollars fifty\", \"Doctor Chen\") when the plain form would be ambiguous. Line breaks are not pauses; use punctuation.\n\nLimit 4096 characters, roughly five minutes of speech. A longer script is refused rather than cut off mid-sentence, so split it across several calls at a natural break.",
|
|
346
|
+
"maxLength": 4096
|
|
347
|
+
},
|
|
348
|
+
"model": {
|
|
349
|
+
"type": "string",
|
|
350
|
+
"enum": [
|
|
351
|
+
"voice",
|
|
352
|
+
"clone",
|
|
353
|
+
"design"
|
|
354
|
+
],
|
|
355
|
+
"description": "Which speech model to use, chosen by what the user gave you. \"voice\" (default): one of nine studio voices, optionally restyled by voiceDescription — the right choice for narration, voiceover and dialogue when no particular person is being imitated. \"clone\": reproduces a specific voice from a recording the user uploaded; requires voiceSourceIndex. \"design\": invents a speaker who does not exist from a written description; requires voiceDescription and takes no recording. Pick \"clone\" whenever the user uploads a voice clip and asks for that person, and \"design\" when they describe a speaker instead of choosing one."
|
|
356
|
+
},
|
|
357
|
+
"voice": {
|
|
358
|
+
"type": "string",
|
|
359
|
+
"enum": [
|
|
360
|
+
"serena",
|
|
361
|
+
"vivian",
|
|
362
|
+
"uncle_fu",
|
|
363
|
+
"ryan",
|
|
364
|
+
"aiden",
|
|
365
|
+
"ono_anna",
|
|
366
|
+
"sohee",
|
|
367
|
+
"eric",
|
|
368
|
+
"dylan"
|
|
369
|
+
],
|
|
370
|
+
"description": "Studio voice to speak in. Only used when model=\"voice\". Female: serena (English), vivian (Chinese), ono_anna (Japanese), sohee (Korean). Male: ryan, aiden, eric, dylan (English), uncle_fu (older, Chinese). Every voice speaks all supported languages — the note above describes the accent it carries, not a limit on what it can read. Default: serena."
|
|
371
|
+
},
|
|
372
|
+
"voiceDescription": {
|
|
373
|
+
"type": "string",
|
|
374
|
+
"description": "How the line should be delivered, or who should deliver it. With model=\"voice\" this restyles the chosen voice without changing who it is: \"whispering, close to the microphone\", \"furious\", \"reading a bedtime story\", \"like a sports commentator\". With model=\"design\" this is required and describes a speaker to invent: age, gender, accent, timbre, pace, mood, and recording space — \"a warm, unhurried narrator in her forties with a faint Scottish lilt, close-miked in a quiet room\". Describe one coherent person; contradictory directions produce a voice that shifts mid-sentence. Not accepted with model=\"clone\", where the recording defines the voice. Limit 512 characters.",
|
|
375
|
+
"maxLength": 512
|
|
376
|
+
},
|
|
377
|
+
"voiceSourceIndex": {
|
|
378
|
+
"type": "number",
|
|
379
|
+
"description": "Index of the uploaded audio file holding the voice to clone (0-based, from the uploaded files list, the same numbering other tools use for audio). Required when model=\"clone\" and ignored otherwise. The clip should be three to thirty seconds of one person speaking cleanly, with no music, no second speaker and no heavy room echo; anything past thirty seconds is trimmed.",
|
|
380
|
+
"minimum": 0
|
|
381
|
+
},
|
|
382
|
+
"voiceTranscript": {
|
|
383
|
+
"type": "string",
|
|
384
|
+
"description": "The exact words spoken in the uploaded clip, when model=\"clone\". Supplying it lets the model condition on the recording itself rather than on a speaker fingerprint alone, which is markedly closer to the source — set it whenever the user tells you what the clip says or the transcript is otherwise known. Omitting it still produces a recognisable clone, just a looser one. Limit 1024 characters.",
|
|
385
|
+
"maxLength": 1024
|
|
386
|
+
},
|
|
387
|
+
"language": {
|
|
388
|
+
"type": "string",
|
|
389
|
+
"enum": [
|
|
390
|
+
"auto",
|
|
391
|
+
"english",
|
|
392
|
+
"chinese",
|
|
393
|
+
"japanese",
|
|
394
|
+
"korean",
|
|
395
|
+
"german",
|
|
396
|
+
"french",
|
|
397
|
+
"russian",
|
|
398
|
+
"portuguese",
|
|
399
|
+
"spanish",
|
|
400
|
+
"italian"
|
|
401
|
+
],
|
|
402
|
+
"description": "Language of the script. Default \"auto\", which infers it from the text and is the only setting that reads a code-switched line correctly. Pin a language only when auto mis-reads a name, a loanword, or a passage that is ambiguous between two of them. Cloning is cross-lingual: an English reference clip can read Japanese in the same voice."
|
|
403
|
+
},
|
|
404
|
+
"numberOfVariations": {
|
|
405
|
+
"type": "number",
|
|
406
|
+
"description": "Number of takes (1-16). Each take is a separate read of the same script with the same voice, differing only in delivery. Use more than 1 when the user asks for options or alternate reads. Default: 1.",
|
|
407
|
+
"minimum": 1,
|
|
408
|
+
"maximum": 16
|
|
409
|
+
}
|
|
410
|
+
},
|
|
411
|
+
"required": [
|
|
412
|
+
"prompt"
|
|
413
|
+
]
|
|
414
|
+
}
|
|
415
|
+
}
|
|
416
|
+
},
|
|
334
417
|
{
|
|
335
418
|
"type": "function",
|
|
336
419
|
"function": {
|
|
@@ -599,12 +682,14 @@
|
|
|
599
682
|
"happyhorse-1.1-r2v",
|
|
600
683
|
"minimax-h3-i2v",
|
|
601
684
|
"minimax-h3-i2v-turbo",
|
|
685
|
+
"minimax-h3-fasth3-i2v-turbo",
|
|
602
686
|
"minimax-h3-flf2v",
|
|
603
687
|
"minimax-h3-flf2v-turbo",
|
|
688
|
+
"minimax-h3-fasth3-flf2v-turbo",
|
|
604
689
|
"wan3.0-video",
|
|
605
690
|
"wan3.0-spicy-video"
|
|
606
691
|
],
|
|
607
|
-
"description": "\"ltx25\" (default): LTX 2.5 I2V or first/last-frame video with native audio; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflow. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. Which video model to use. \"ltx23\": LTX 2.3 rollback with native audio. \"wan22\": quick simple motion without audio, up to 10s. \"minimax-h3-i2v\"
|
|
692
|
+
"description": "\"ltx25\" (default): LTX 2.5 I2V or first/last-frame video with native audio; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflow. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. Which video model to use. \"ltx23\": LTX 2.3 rollback with native audio. \"wan22\": quick simple motion without audio, up to 10s. \"minimax-h3-i2v\" is standard MiniMax H3 from one first frame; \"minimax-h3-i2v-turbo\" is the existing 4-step LightX2V Turbo engine; \"minimax-h3-fasth3-i2v-turbo\" is the separate FastVideo VSA four-step FastH3 engine, about 2x faster and fixed to Euler/simple. The matching FLF2V selectors provide standard, LightX2V Turbo, and FastH3 Turbo first/last-frame generation; FastH3 has no R2V mode; use frameRole=\"both\" and provide the end frame. H3 generates native audio at fixed 24fps for 5.17-15.08s and has no negative-prompt input. H3 Base and Turbo prompts use the exact three-field contract and the official mode-specific alignment line. Do not set Seedance here; use generate_video with Seedance references. \"wan3.0-video\" is Alibaba Wan 3 and \"wan3.0-spicy-video\" is MuleRouter w3.0-video. Both render 2-30s or smart duration at fixed 30 fps with optional native audio, provider prompt expansion, 480p/720p/1080p, adaptive/fixed ratios, first/last frames, and up to 10 image/5 video/5 audio references. Only Alibaba wan3.0-video accepts document/web context and watermark. Frame anchors and loose references are mutually exclusive. Do not send negativePrompt; video references are loose conditioning for a new result, not source-video editing or extension."
|
|
608
693
|
},
|
|
609
694
|
"negativePrompt": {
|
|
610
695
|
"type": "string",
|
|
@@ -708,7 +793,7 @@
|
|
|
708
793
|
"type": "string",
|
|
709
794
|
"minLength": 1
|
|
710
795
|
},
|
|
711
|
-
"description": "Ordered LoRA IDs to apply to a MiniMax H3 render. Use only when the user explicitly asks for a LoRA or for an effect one of these names describes. Stack up to 8 in one request; order matters because the adapters apply in sequence and do not commute. Keep this array positionally aligned with loraStrengths. The first render with an uncached LoRA takes longer to start while the worker downloads it.\n\nAccepted only when videoModel is one of \"minimax-h3-i2v\", \"minimax-h3-i2v-turbo\", \"minimax-h3-flf2v\", \"minimax-h3-flf2v-turbo\". Every other video model on this tool loads no LoRAs and silently ignores these arrays, so set videoModel to an H3 mode in the same call when the user asks for one.\n\nOne LoRA is published for MiniMax H3 today: h3-realism-people (fal), a realism pass trained on live-action footage of people. It restores skin texture and pores, stray hairs, fabric weave and a fine sensor grain that the base model smooths away, and holds up in close-up. It needs its trigger word: put r34l1sm near the FRONT of the prompt, or the render comes back as ordinary H3 with no error. Exact ranges and any LoRA published since: GET /v1/loras/comfy?modelId=<model>. Do not invent ids."
|
|
796
|
+
"description": "Ordered LoRA IDs to apply to a MiniMax H3 render. Use only when the user explicitly asks for a LoRA or for an effect one of these names describes. Stack up to 8 in one request; order matters because the adapters apply in sequence and do not commute. Keep this array positionally aligned with loraStrengths. The first render with an uncached LoRA takes longer to start while the worker downloads it.\n\nAccepted only when videoModel is one of \"minimax-h3-i2v\", \"minimax-h3-i2v-turbo\", \"minimax-h3-fasth3-i2v-turbo\", \"minimax-h3-flf2v\", \"minimax-h3-flf2v-turbo\", \"minimax-h3-fasth3-flf2v-turbo\". Every other video model on this tool loads no LoRAs and silently ignores these arrays, so set videoModel to an H3 mode in the same call when the user asks for one.\n\nOne LoRA is published for MiniMax H3 today: h3-realism-people (fal), a realism pass trained on live-action footage of people. It restores skin texture and pores, stray hairs, fabric weave and a fine sensor grain that the base model smooths away, and holds up in close-up. It needs its trigger word: put r34l1sm near the FRONT of the prompt, or the render comes back as ordinary H3 with no error. Exact ranges and any LoRA published since: GET /v1/loras/comfy?modelId=<model>. Do not invent ids."
|
|
712
797
|
},
|
|
713
798
|
"loraStrengths": {
|
|
714
799
|
"type": "array",
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@sogni-ai/sogni-protocol",
|
|
3
|
-
"version": "1.0.0-alpha.
|
|
3
|
+
"version": "1.0.0-alpha.33",
|
|
4
4
|
"description": "Language-neutral protocol artifacts for the Sogni ecosystem: tool schemas, prompts, OpenAI tool manifests, and enums. Consumed by every Sogni SDK (TypeScript, Swift, and future Python/Kotlin/Rust SDKs) so contracts stay in lockstep across languages.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"sogni",
|
|
@@ -29,12 +29,14 @@
|
|
|
29
29
|
"happyhorse-1.1-r2v",
|
|
30
30
|
"minimax-h3-i2v",
|
|
31
31
|
"minimax-h3-i2v-turbo",
|
|
32
|
+
"minimax-h3-fasth3-i2v-turbo",
|
|
32
33
|
"minimax-h3-flf2v",
|
|
33
34
|
"minimax-h3-flf2v-turbo",
|
|
35
|
+
"minimax-h3-fasth3-flf2v-turbo",
|
|
34
36
|
"wan3.0-video",
|
|
35
37
|
"wan3.0-spicy-video"
|
|
36
38
|
],
|
|
37
|
-
"description": "\"ltx25\" (default): LTX 2.5 I2V or first/last-frame video with native audio; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflow. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. Which video model to use. \"ltx23\": LTX 2.3 rollback with native audio. \"wan22\": quick simple motion without audio, up to 10s. \"minimax-h3-i2v\"
|
|
39
|
+
"description": "\"ltx25\" (default): LTX 2.5 I2V or first/last-frame video with native audio; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflow. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. Which video model to use. \"ltx23\": LTX 2.3 rollback with native audio. \"wan22\": quick simple motion without audio, up to 10s. \"minimax-h3-i2v\" is standard MiniMax H3 from one first frame; \"minimax-h3-i2v-turbo\" is the existing 4-step LightX2V Turbo engine; \"minimax-h3-fasth3-i2v-turbo\" is the separate FastVideo VSA four-step FastH3 engine, about 2x faster and fixed to Euler/simple. The matching FLF2V selectors provide standard, LightX2V Turbo, and FastH3 Turbo first/last-frame generation; FastH3 has no R2V mode; use frameRole=\"both\" and provide the end frame. H3 generates native audio at fixed 24fps for 5.17-15.08s and has no negative-prompt input. H3 Base and Turbo prompts use the exact three-field contract and the official mode-specific alignment line. Do not set Seedance here; use generate_video with Seedance references. \"wan3.0-video\" is Alibaba Wan 3 and \"wan3.0-spicy-video\" is MuleRouter w3.0-video. Both render 2-30s or smart duration at fixed 30 fps with optional native audio, provider prompt expansion, 480p/720p/1080p, adaptive/fixed ratios, first/last frames, and up to 10 image/5 video/5 audio references. Only Alibaba wan3.0-video accepts document/web context and watermark. Frame anchors and loose references are mutually exclusive. Do not send negativePrompt; video references are loose conditioning for a new result, not source-video editing or extension."
|
|
38
40
|
},
|
|
39
41
|
"negativePrompt": {
|
|
40
42
|
"type": "string",
|
|
@@ -138,7 +140,7 @@
|
|
|
138
140
|
"type": "string",
|
|
139
141
|
"minLength": 1
|
|
140
142
|
},
|
|
141
|
-
"description": "Ordered LoRA IDs to apply to a MiniMax H3 render. Use only when the user explicitly asks for a LoRA or for an effect one of these names describes. Stack up to 8 in one request; order matters because the adapters apply in sequence and do not commute. Keep this array positionally aligned with loraStrengths. The first render with an uncached LoRA takes longer to start while the worker downloads it.\n\nAccepted only when videoModel is one of \"minimax-h3-i2v\", \"minimax-h3-i2v-turbo\", \"minimax-h3-flf2v\", \"minimax-h3-flf2v-turbo\". Every other video model on this tool loads no LoRAs and silently ignores these arrays, so set videoModel to an H3 mode in the same call when the user asks for one.\n\nOne LoRA is published for MiniMax H3 today: h3-realism-people (fal), a realism pass trained on live-action footage of people. It restores skin texture and pores, stray hairs, fabric weave and a fine sensor grain that the base model smooths away, and holds up in close-up. It needs its trigger word: put r34l1sm near the FRONT of the prompt, or the render comes back as ordinary H3 with no error. Exact ranges and any LoRA published since: GET /v1/loras/comfy?modelId=<model>. Do not invent ids."
|
|
143
|
+
"description": "Ordered LoRA IDs to apply to a MiniMax H3 render. Use only when the user explicitly asks for a LoRA or for an effect one of these names describes. Stack up to 8 in one request; order matters because the adapters apply in sequence and do not commute. Keep this array positionally aligned with loraStrengths. The first render with an uncached LoRA takes longer to start while the worker downloads it.\n\nAccepted only when videoModel is one of \"minimax-h3-i2v\", \"minimax-h3-i2v-turbo\", \"minimax-h3-fasth3-i2v-turbo\", \"minimax-h3-flf2v\", \"minimax-h3-flf2v-turbo\", \"minimax-h3-fasth3-flf2v-turbo\". Every other video model on this tool loads no LoRAs and silently ignores these arrays, so set videoModel to an H3 mode in the same call when the user asks for one.\n\nOne LoRA is published for MiniMax H3 today: h3-realism-people (fal), a realism pass trained on live-action footage of people. It restores skin texture and pores, stray hairs, fabric weave and a fine sensor grain that the base model smooths away, and holds up in close-up. It needs its trigger word: put r34l1sm near the FRONT of the prompt, or the render comes back as ordinary H3 with no error. Exact ranges and any LoRA published since: GET /v1/loras/comfy?modelId=<model>. Do not invent ids."
|
|
142
144
|
},
|
|
143
145
|
"loraStrengths": {
|
|
144
146
|
"type": "array",
|
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
{
|
|
2
|
+
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
3
|
+
"$id": "https://schemas.sogni.ai/creative-agent/2026-09-07.1/tools/generate_speech.schema.json",
|
|
4
|
+
"title": "generate_speech arguments",
|
|
5
|
+
"schemaVersion": "2026-09-07.1",
|
|
6
|
+
"description": "Turn written words into spoken audio. Use when the user wants something read aloud, a narration, a voiceover, a line of dialogue, an audiobook passage, or a voice cloned from a recording they uploaded. This tool speaks text; it does not compose music — use generate_music for songs and instrumentals.",
|
|
7
|
+
"type": "object",
|
|
8
|
+
"additionalProperties": false,
|
|
9
|
+
"properties": {
|
|
10
|
+
"prompt": {
|
|
11
|
+
"type": "string",
|
|
12
|
+
"description": "The exact words to be spoken, verbatim. This is NOT a description of the audio: whatever is written here is read aloud character for character. \"a calm woman reading the news\" would be spoken as those seven words — put that in voiceDescription instead and write the actual news copy here.\n\nLITERAL PROMPT OVERRIDE: If the user explicitly says not to modify the prompt, or to use it exactly/verbatim/as-is, copy the identified prompt text verbatim instead of applying these construction rules unless a hard requirement is missing.\n\nPunctuation is prosody: full stops, commas, question marks and ellipses control pauses and intonation, so keep them. Write numbers, dates, currency and abbreviations the way they should be said (\"nineteen eighty-four\", \"twelve dollars fifty\", \"Doctor Chen\") when the plain form would be ambiguous. Line breaks are not pauses; use punctuation.\n\nLimit 4096 characters, roughly five minutes of speech. A longer script is refused rather than cut off mid-sentence, so split it across several calls at a natural break.",
|
|
13
|
+
"maxLength": 4096
|
|
14
|
+
},
|
|
15
|
+
"model": {
|
|
16
|
+
"type": "string",
|
|
17
|
+
"enum": [
|
|
18
|
+
"voice",
|
|
19
|
+
"clone",
|
|
20
|
+
"design"
|
|
21
|
+
],
|
|
22
|
+
"description": "Which speech model to use, chosen by what the user gave you. \"voice\" (default): one of nine studio voices, optionally restyled by voiceDescription — the right choice for narration, voiceover and dialogue when no particular person is being imitated. \"clone\": reproduces a specific voice from a recording the user uploaded; requires voiceSourceIndex. \"design\": invents a speaker who does not exist from a written description; requires voiceDescription and takes no recording. Pick \"clone\" whenever the user uploads a voice clip and asks for that person, and \"design\" when they describe a speaker instead of choosing one."
|
|
23
|
+
},
|
|
24
|
+
"voice": {
|
|
25
|
+
"type": "string",
|
|
26
|
+
"enum": [
|
|
27
|
+
"serena",
|
|
28
|
+
"vivian",
|
|
29
|
+
"uncle_fu",
|
|
30
|
+
"ryan",
|
|
31
|
+
"aiden",
|
|
32
|
+
"ono_anna",
|
|
33
|
+
"sohee",
|
|
34
|
+
"eric",
|
|
35
|
+
"dylan"
|
|
36
|
+
],
|
|
37
|
+
"description": "Studio voice to speak in. Only used when model=\"voice\". Female: serena (English), vivian (Chinese), ono_anna (Japanese), sohee (Korean). Male: ryan, aiden, eric, dylan (English), uncle_fu (older, Chinese). Every voice speaks all supported languages — the note above describes the accent it carries, not a limit on what it can read. Default: serena."
|
|
38
|
+
},
|
|
39
|
+
"voiceDescription": {
|
|
40
|
+
"type": "string",
|
|
41
|
+
"description": "How the line should be delivered, or who should deliver it. With model=\"voice\" this restyles the chosen voice without changing who it is: \"whispering, close to the microphone\", \"furious\", \"reading a bedtime story\", \"like a sports commentator\". With model=\"design\" this is required and describes a speaker to invent: age, gender, accent, timbre, pace, mood, and recording space — \"a warm, unhurried narrator in her forties with a faint Scottish lilt, close-miked in a quiet room\". Describe one coherent person; contradictory directions produce a voice that shifts mid-sentence. Not accepted with model=\"clone\", where the recording defines the voice. Limit 512 characters.",
|
|
42
|
+
"maxLength": 512
|
|
43
|
+
},
|
|
44
|
+
"voiceSourceIndex": {
|
|
45
|
+
"type": "number",
|
|
46
|
+
"description": "Index of the uploaded audio file holding the voice to clone (0-based, from the uploaded files list, the same numbering other tools use for audio). Required when model=\"clone\" and ignored otherwise. The clip should be three to thirty seconds of one person speaking cleanly, with no music, no second speaker and no heavy room echo; anything past thirty seconds is trimmed.",
|
|
47
|
+
"minimum": 0
|
|
48
|
+
},
|
|
49
|
+
"voiceTranscript": {
|
|
50
|
+
"type": "string",
|
|
51
|
+
"description": "The exact words spoken in the uploaded clip, when model=\"clone\". Supplying it lets the model condition on the recording itself rather than on a speaker fingerprint alone, which is markedly closer to the source — set it whenever the user tells you what the clip says or the transcript is otherwise known. Omitting it still produces a recognisable clone, just a looser one. Limit 1024 characters.",
|
|
52
|
+
"maxLength": 1024
|
|
53
|
+
},
|
|
54
|
+
"language": {
|
|
55
|
+
"type": "string",
|
|
56
|
+
"enum": [
|
|
57
|
+
"auto",
|
|
58
|
+
"english",
|
|
59
|
+
"chinese",
|
|
60
|
+
"japanese",
|
|
61
|
+
"korean",
|
|
62
|
+
"german",
|
|
63
|
+
"french",
|
|
64
|
+
"russian",
|
|
65
|
+
"portuguese",
|
|
66
|
+
"spanish",
|
|
67
|
+
"italian"
|
|
68
|
+
],
|
|
69
|
+
"description": "Language of the script. Default \"auto\", which infers it from the text and is the only setting that reads a code-switched line correctly. Pin a language only when auto mis-reads a name, a loanword, or a passage that is ambiguous between two of them. Cloning is cross-lingual: an English reference clip can read Japanese in the same voice."
|
|
70
|
+
},
|
|
71
|
+
"numberOfVariations": {
|
|
72
|
+
"type": "number",
|
|
73
|
+
"description": "Number of takes (1-16). Each take is a separate read of the same script with the same voice, differing only in delivery. Use more than 1 when the user asks for options or alternate reads. Default: 1.",
|
|
74
|
+
"minimum": 1,
|
|
75
|
+
"maximum": 16
|
|
76
|
+
}
|
|
77
|
+
},
|
|
78
|
+
"required": [
|
|
79
|
+
"prompt"
|
|
80
|
+
]
|
|
81
|
+
}
|
|
@@ -68,6 +68,7 @@
|
|
|
68
68
|
"seedance2-5",
|
|
69
69
|
"minimax-h3-t2v",
|
|
70
70
|
"minimax-h3-t2v-turbo",
|
|
71
|
+
"minimax-h3-fasth3-t2v-turbo",
|
|
71
72
|
"happyhorse-1.1-t2v",
|
|
72
73
|
"happyhorse-1.1-i2v",
|
|
73
74
|
"happyhorse-1.1-r2v",
|
|
@@ -76,7 +77,7 @@
|
|
|
76
77
|
"wan3.0-video",
|
|
77
78
|
"wan3.0-spicy-video"
|
|
78
79
|
],
|
|
79
|
-
"description": "\"ltx25\" (default): LTX 2.5 with native audio; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflow. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. Video model. \"ltx23\": LTX 2.3 rollback with native audio. \"wan22\": quick simple motion without audio. \"minimax-h3-t2v\": standard 20-step MiniMax H3 text-to-video; \"minimax-h3-t2v-turbo\": 4-step Turbo text-to-video.
|
|
80
|
+
"description": "\"ltx25\" (default): LTX 2.5 with native audio; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflow. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. Video model. \"ltx23\": LTX 2.3 rollback with native audio. \"wan22\": quick simple motion without audio. \"minimax-h3-t2v\": standard 20-step MiniMax H3 text-to-video; \"minimax-h3-t2v-turbo\": the existing 4-step LightX2V Turbo text-to-video; \"minimax-h3-fasth3-t2v-turbo\": the separate FastVideo VSA four-step FastH3 engine, about 2x faster and fixed to Euler/simple. All use native audio, fixed 24fps, 5.17-15.08s, and a 768p-class 32px-grid canvas; use animate_photo for H3 image-conditioned modes. Base and Turbo T2V/I2V/FLF2V prompts use the exact ordered fields integrated_multimodal_description, overall_soundscape, and non_diegetic_music; I2V/FLF2V prepend the official alignment line. \"minimax-h3-r2v\": standard 20-step MiniMax H3 reference-to-video; \"minimax-h3-r2v-turbo\": the dedicated LightX2V 4-step Ref2VA Turbo workflow using Euler/simple and a 960x544 default. FastH3 has no R2V mode. Both R2V selectors accept up to 9 images, 3 videos, and 3 audios (12 files total); at least one visual reference (image or video) is required and audio alone is invalid. Select references with referenceImageIndices/referenceVideoIndices/referenceAudioIndices and address them with the official <Subject N>/<Picture N>/<Video N>/<Audio N> semantics. Seedance quality is selected only by model: use \"seedance2-mini\" for Seedance 2.0 Mini or faster/lower-cost 720p iteration, and use \"seedance2\" for the full Seedance 2.0 model, explicit full-quality requests, 1080p/4K requests, or generated/uploaded storyboard images unless the user explicitly asks for a draft or Mini. Do not use Default Media Quality Fast/HQ/Pro or targetResolution to represent Seedance quality. \"seedance2-5\": Seedance 2.5, the newest Seedance generation — 480p and 720p ONLY (it cannot render 1080p or 4K), 4-30s per clip at a fixed 24 fps, native audio, first-and-last-frame conditioning, and a much larger reference budget than the 2.0 family: up to 30 images, 10 videos, and 10 audios, with up to 50 reference media files total, subject to those per-modality caps. Choose \"seedance2-5\" when the user asks for Seedance 2.5, wants a single continuous Seedance clip longer than 15s (2.5 renders up to 30s in one call instead of being split and stitched), or wants a first-and-last-frame Seedance transition. Keep \"seedance2\" for 1080p/4K requests, which Seedance 2.5 cannot satisfy. Seedance supports multimodal loose reference assets: images (up to 9), videos (up to 3), and audios (up to 3), with no more than 12 asset files total. Use @Image1/@Video1/@Audio1 style references in creative briefs when assigning roles. Assign every useful reference asset a role and prefer positive preservation constraints. If an uploaded video is the source clip to transform, upscale, enhance, restyle, or remaster, use video_to_video with controlMode=\"seedance-v2v\" instead of generate_video referenceVideoIndices. Alibaba HappyHorse 1.1 video models (third-party vendor — requires Premium Spark). Select by mode: \"happyhorse-1.1-t2v\" for text-to-video, \"happyhorse-1.1-i2v\" for image-to-video from one first-frame image, and \"happyhorse-1.1-r2v\" for reference-to-video with up to 9 reference images. Resolutions 720P and 1080P; duration 3-15 seconds at 24 fps; native synchronized audio is always generated (do not set generateAudio or negativePrompt). Supported aspect ratios: 16:9, 9:16, 1:1, 4:3, 3:4, 4:5, 5:4, 9:21, 21:9. HappyHorse 1.1 takes image references only and renders a native synchronized audio track (always on; do not set generateAudio or a negative prompt). Pick the model by mode: happyhorse-1.1-t2v for text-to-video (no reference image), happyhorse-1.1-i2v for image-to-video from a single first frame, and happyhorse-1.1-r2v for reference-to-video with 1 to 9 reference images. For r2v, tag the images in the prompt as [Image 1]…[Image 9] and assign each a clear role. HappyHorse does not accept reference videos or reference audios. \"wan3.0-video\" is Alibaba Wan 3 and \"wan3.0-spicy-video\" is MuleRouter w3.0-video. Both render 2-30s or smart duration at fixed 30 fps with optional native audio, provider prompt expansion, 480p/720p/1080p, adaptive/fixed ratios, first/last frames, and up to 10 image/5 video/5 audio references. Only Alibaba wan3.0-video accepts document/web context and watermark. Frame anchors and loose references are mutually exclusive. Do not send negativePrompt; video references are loose conditioning for a new result, not source-video editing or extension."
|
|
80
81
|
},
|
|
81
82
|
"generateAudio": {
|
|
82
83
|
"type": "boolean",
|
|
@@ -137,7 +138,7 @@
|
|
|
137
138
|
"type": "string",
|
|
138
139
|
"minLength": 1
|
|
139
140
|
},
|
|
140
|
-
"description": "Ordered LoRA IDs to apply to a MiniMax H3 render. Use only when the user explicitly asks for a LoRA or for an effect one of these names describes. Stack up to 8 in one request; order matters because the adapters apply in sequence and do not commute. Keep this array positionally aligned with loraStrengths. The first render with an uncached LoRA takes longer to start while the worker downloads it.\n\nAccepted only when videoModel is one of \"minimax-h3-t2v\", \"minimax-h3-t2v-turbo\", \"minimax-h3-r2v\", \"minimax-h3-r2v-turbo\". Every other video model on this tool loads no LoRAs and silently ignores these arrays, so set videoModel to an H3 mode in the same call when the user asks for one.\n\nOne LoRA is published for MiniMax H3 today: h3-realism-people (fal), a realism pass trained on live-action footage of people. It restores skin texture and pores, stray hairs, fabric weave and a fine sensor grain that the base model smooths away, and holds up in close-up. It needs its trigger word: put r34l1sm near the FRONT of the prompt, or the render comes back as ordinary H3 with no error. Exact ranges and any LoRA published since: GET /v1/loras/comfy?modelId=<model>. Do not invent ids."
|
|
141
|
+
"description": "Ordered LoRA IDs to apply to a MiniMax H3 render. Use only when the user explicitly asks for a LoRA or for an effect one of these names describes. Stack up to 8 in one request; order matters because the adapters apply in sequence and do not commute. Keep this array positionally aligned with loraStrengths. The first render with an uncached LoRA takes longer to start while the worker downloads it.\n\nAccepted only when videoModel is one of \"minimax-h3-t2v\", \"minimax-h3-t2v-turbo\", \"minimax-h3-fasth3-t2v-turbo\", \"minimax-h3-r2v\", \"minimax-h3-r2v-turbo\". Every other video model on this tool loads no LoRAs and silently ignores these arrays, so set videoModel to an H3 mode in the same call when the user asks for one.\n\nOne LoRA is published for MiniMax H3 today: h3-realism-people (fal), a realism pass trained on live-action footage of people. It restores skin texture and pores, stray hairs, fabric weave and a fine sensor grain that the base model smooths away, and holds up in close-up. It needs its trigger word: put r34l1sm near the FRONT of the prompt, or the render comes back as ordinary H3 with no error. Exact ranges and any LoRA published since: GET /v1/loras/comfy?modelId=<model>. Do not invent ids."
|
|
141
142
|
},
|
|
142
143
|
"loraStrengths": {
|
|
143
144
|
"type": "array",
|
package/version.json
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
1
|
{
|
|
2
|
-
"protocolVersion": "4.
|
|
2
|
+
"protocolVersion": "4.3.0",
|
|
3
3
|
"description": "Sogni protocol artifact version. SDKs may refuse to operate against a protocolVersion they were not built for. Bump the major when removing or renaming any schema / enum / manifest field; bump the minor when adding new optional fields or new tools; bump the patch for description / prose changes only."
|
|
4
4
|
}
|