@sogni-ai/sogni-protocol 1.0.0-alpha.40 → 1.0.0-alpha.42
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/manifests/generation-tools.json +60 -14
- package/manifests/openai-tools.json +60 -14
- package/package.json +1 -1
- package/prompts/tools/animate_photo.json +1 -1
- package/prompts/tools/generate_video.json +1 -1
- package/prompts/tools/sound_to_video.json +4 -3
- package/schemas/tools/animate_photo.schema.json +1 -1
- package/schemas/tools/generate_video.schema.json +14 -2
- package/schemas/tools/sound_to_video.schema.json +31 -9
- package/schemas/tools/video_to_video.schema.json +14 -2
- package/version.json +1 -1
|
@@ -245,7 +245,7 @@
|
|
|
245
245
|
"wan3.0-video",
|
|
246
246
|
"wan3.0-spicy-video"
|
|
247
247
|
],
|
|
248
|
-
"description": "\"ltx25\" (default): LTX 2.5 with native audio; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflow. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. Video model. \"ltx23\": LTX 2.3 rollback with native audio. \"wan22\": quick simple motion without audio. \"minimax-h3-t2v\": standard 20-step MiniMax H3 text-to-video; \"minimax-h3-t2v-turbo\": the existing 4-step LightX2V Turbo text-to-video; \"minimax-h3-fasth3-t2v-turbo\": the separate FastVideo VSA four-step FastH3 engine, about 2x faster and fixed to Euler/simple; \"minimax-h3-fasth3-t2v-turbo-2stage\": the two-stage FastH3 engine: FastH3 renders the canvas, then the worker enlarges it 2x and refines it, so the clip is delivered at twice the canvas width and height with the same length and audio. targetResolution picks the delivered class: 1080 renders a 544px short-edge canvas (960x544 is delivered at 1920x1088) for
|
|
248
|
+
"description": "\"ltx25\" (default): LTX 2.5 with native audio; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflow. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. Video model. \"ltx23\": LTX 2.3 rollback with native audio. \"wan22\": quick simple motion without audio. \"minimax-h3-t2v\": standard 20-step MiniMax H3 text-to-video; \"minimax-h3-t2v-turbo\": the existing 4-step LightX2V Turbo text-to-video; \"minimax-h3-fasth3-t2v-turbo\": the separate FastVideo VSA four-step FastH3 engine, about 2x faster and fixed to Euler/simple; \"minimax-h3-fasth3-t2v-turbo-2stage\": the two-stage FastH3 engine: FastH3 renders the canvas, then the worker enlarges it 2x and refines it, so the clip is delivered at twice the canvas width and height with the same length and audio. targetResolution picks the delivered class: 1080 renders a 544px short-edge canvas (960x544 is delivered at 1920x1088) for 10 Spark per second, 1440 or omitted renders the 768p canvas for 2K (1344x768 is delivered at 2688x1536) for 16 Spark per second, and 720 renders a 384px canvas (672x384 is delivered at 1344x768) for the regular FastH3 price of 4 Spark per second. The estimate prices every request. It takes the same inputs, durations and LoRAs as \"minimax-h3-fasth3-t2v-turbo\". Choose it when the user asks for 1080p, 1440p or 2K MiniMax H3 output, for two-stage output, or for the sharpest/best H3 quality; for ordinary 768p FastH3 output keep the regular FastH3 selector at targetResolution 768. All use native audio, fixed 24fps, 5.17-15.08s, and a 768p-class 32px-grid canvas; use animate_photo for H3 image-conditioned modes. Base and Turbo T2V/I2V/FLF2V prompts use the exact ordered fields integrated_multimodal_description, overall_soundscape, and non_diegetic_music; I2V/FLF2V prepend the official alignment line. \"minimax-h3-r2v\": standard 20-step MiniMax H3 reference-to-video; \"minimax-h3-r2v-turbo\": the dedicated LightX2V 4-step Ref2VA Turbo workflow using Euler/simple and a 960x544 default. FastH3 has no R2V mode. Both R2V selectors accept up to 9 images, 3 videos, and 3 audios (12 files total); at least one visual reference (image or video) is required and audio alone is invalid. Select references with referenceImageIndices/referenceVideoIndices/referenceAudioIndices and address them with the official <Subject N>/<Picture N>/<Video N>/<Audio N> semantics. Seedance quality is selected only by model: use \"seedance2-mini\" for Seedance 2.0 Mini or faster/lower-cost 720p iteration, and use \"seedance2\" for the full Seedance 2.0 model, explicit full-quality requests, 1080p/4K requests, or generated/uploaded storyboard images unless the user explicitly asks for a draft or Mini. Do not use Default Media Quality Fast/HQ/Pro or targetResolution to represent Seedance quality. \"seedance2-5\": Seedance 2.5, the newest Seedance generation — 480p, 720p, and 1080p (4K is unsupported), 4-30s per clip at a fixed 24 fps, native audio, first-and-last-frame conditioning, and a much larger reference budget than the 2.0 family: up to 30 images, 10 videos, and 10 audios, with up to 50 reference media files total, subject to those per-modality caps. Choose \"seedance2-5\" when the user asks for Seedance 2.5, wants a single continuous Seedance clip longer than 15s (2.5 renders up to 30s in one call instead of being split and stitched), or wants a first-and-last-frame Seedance transition. Keep \"seedance2\" for 4K requests; Seedance 2.5 supports up to 1080p. Seedance supports multimodal loose reference assets: images (up to 9), videos (up to 3), and audios (up to 3), with no more than 12 asset files total. Use @Image1/@Video1/@Audio1 style references in creative briefs when assigning roles. Assign every useful reference asset a role and prefer positive preservation constraints. If an uploaded video is the source clip to transform, upscale, enhance, restyle, or remaster, use video_to_video with controlMode=\"seedance-v2v\" instead of generate_video referenceVideoIndices. Alibaba HappyHorse 1.1 video models (third-party vendor — requires Premium Spark). Select by mode: \"happyhorse-1.1-t2v\" for text-to-video, \"happyhorse-1.1-i2v\" for image-to-video from one first-frame image, and \"happyhorse-1.1-r2v\" for reference-to-video with up to 9 reference images. Resolutions 720P and 1080P; duration 3-15 seconds at 24 fps; native synchronized audio is always generated (do not set generateAudio or negativePrompt). Supported aspect ratios: 16:9, 9:16, 1:1, 4:3, 3:4, 4:5, 5:4, 9:21, 21:9. HappyHorse 1.1 takes image references only and renders a native synchronized audio track (always on; do not set generateAudio or a negative prompt). Pick the model by mode: happyhorse-1.1-t2v for text-to-video (no reference image), happyhorse-1.1-i2v for image-to-video from a single first frame, and happyhorse-1.1-r2v for reference-to-video with 1 to 9 reference images. For r2v, tag the images in the prompt as [Image 1]…[Image 9] and assign each a clear role. HappyHorse does not accept reference videos or reference audios. \"wan3.0-video\" is Alibaba Wan 3 and \"wan3.0-spicy-video\" is MuleRouter w3.0-video. Both render 2-30s at fixed 30 fps with optional native audio, provider prompt expansion, 480p/720p/1080p, adaptive/fixed ratios, first/last frames, and up to 10 image/5 video/5 audio references. Only Alibaba wan3.0-video accepts document/web context and watermark. Frame anchors and loose references are mutually exclusive. Do not send negativePrompt; video references are loose conditioning for a new result, not source-video editing or extension."
|
|
249
249
|
},
|
|
250
250
|
"generateAudio": {
|
|
251
251
|
"type": "boolean",
|
|
@@ -282,7 +282,7 @@
|
|
|
282
282
|
},
|
|
283
283
|
"targetResolution": {
|
|
284
284
|
"type": "number",
|
|
285
|
-
"description": "Short-side video resolution target in pixels. Use when the user asks for a bare named resolution such as \"480p\", \"720p\", \"1080p\", \"2160p\", or \"4K\" without exact pixels or an output orientation. This is resolution only, not a Seedance quality tier: Seedance quality is selected by videoModel (\"seedance2\" vs \"seedance2-mini\" vs \"seedance2-5\"). Seedance 2.0 full supports 4K; Seedance Mini
|
|
285
|
+
"description": "Short-side video resolution target in pixels. Use when the user asks for a bare named resolution such as \"480p\", \"720p\", \"1080p\", \"2160p\", or \"4K\" without exact pixels or an output orientation. This is resolution only, not a Seedance quality tier: Seedance quality is selected by videoModel (\"seedance2\" vs \"seedance2-mini\" vs \"seedance2-5\"). Seedance 2.0 full supports 4K; Seedance Mini supports 480p/720p; Seedance 2.5 supports 480p/720p/1080p, so never set 4K for \"seedance2-5\". Wan 3 supports exactly 480p, 720p, and 1080p. HappyHorse supports only 720p and 1080p. Never set 4K for Wan 3 or HappyHorse. MiniMax H3 renders inside a 1344x768 pixel budget on a 32px grid, so use 768 for the regular H3 selectors and never 1080p or 4K. The two-stage H3 selector \"minimax-h3-fasth3-t2v-turbo-2stage\" delivers twice the canvas, so there targetResolution names the delivered short-edge class: 1080 (544px canvas short edge: 960x544 delivered at 1920x1088), 1440 for 2K (the 1344x768 canvas delivered at 2688x1536), or 720 (384px canvas: 672x384 delivered at 1344x768); omit it for 2K. Never set 4K for H3. Do not set targetResolution from Default Media Quality Fast/HQ/Pro. If omitted for Seedance, Wan 3, HappyHorse, or MiniMax H3, the host uses the selected model default. This preserves/inherits the current video shape instead of forcing landscape. Do NOT set width, height, or exact-pixel aspectRatio for bare named resolution requests. If the user says \"720p portrait\", \"720p landscape\", \"4K portrait\", or \"4K landscape\", use exact width/height/aspectRatio instead."
|
|
286
286
|
},
|
|
287
287
|
"numberOfVariations": {
|
|
288
288
|
"type": "number",
|
|
@@ -316,6 +316,18 @@
|
|
|
316
316
|
"type": "number"
|
|
317
317
|
},
|
|
318
318
|
"description": "Strength for each LoRA in loras, in the same order. Omitting the array applies 1.0 to every LoRA, which is NOT the catalog default and for h3-realism-people is already at the top of its band, so send explicit values. Video LoRAs are positive-only — unlike the bipolar Krea 2 image sliders, a negative value is not an inverse effect and 0 is off. h3-realism-people takes 0-2 and its catalog default is 0.8; 0.6-1 is the usable band. It also pulls the camera in as it climbs: at 1.5 and above the shot reliably recomposes and the grade darkens, which on an image-conditioned mode can crop the subject out of the frame the user supplied. Raise it above 1 only when the user asks for more, and prefer the default when they supplied a first or last frame. h3-vbvr-video-reasoning and h3-mystic-xxx-v4 both take 0-1 and do default to 1.0, with usable bands of 0.7-1 and 0.2-1. h3-natural-face-speech and h3-better-motion take 0-1.5 and default to 0.6; their usable band is 0.4-0.8."
|
|
319
|
+
},
|
|
320
|
+
"outputFormat": {
|
|
321
|
+
"type": "string",
|
|
322
|
+
"enum": [
|
|
323
|
+
"mp4",
|
|
324
|
+
"mov"
|
|
325
|
+
],
|
|
326
|
+
"description": "Video container. Defaults to mp4. MOV is supported only by Seedance 2.5; choose it when the user requests MOV for editing."
|
|
327
|
+
},
|
|
328
|
+
"returnLastFrame": {
|
|
329
|
+
"type": "boolean",
|
|
330
|
+
"description": "Seedance 2.5 only. Set true to export a separate image of the final frame alongside the video. The result includes lastFrameUrl, which can be used as the first-frame image for a subsequent clip. Defaults to false; this does not extend the video automatically."
|
|
319
331
|
}
|
|
320
332
|
},
|
|
321
333
|
"required": [
|
|
@@ -834,7 +846,7 @@
|
|
|
834
846
|
"wan3.0-video",
|
|
835
847
|
"wan3.0-spicy-video"
|
|
836
848
|
],
|
|
837
|
-
"description": "\"ltx25\" (default): LTX 2.5 I2V or first/last-frame video with native audio; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflow. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. Which video model to use. \"ltx23\": LTX 2.3 rollback with native audio. \"wan22\": quick simple motion without audio, up to 10s. \"minimax-h3-i2v\" is standard MiniMax H3 from one first frame; \"minimax-h3-i2v-turbo\" is the existing 4-step LightX2V Turbo engine; \"minimax-h3-fasth3-i2v-turbo\" is the separate FastVideo VSA four-step FastH3 engine, about 2x faster and fixed to Euler/simple. \"minimax-h3-fasth3-i2v-turbo-2stage\" (first frame) and \"minimax-h3-fasth3-flf2v-turbo-2stage\" (first and last frame) are the two-stage FastH3 engine: FastH3 renders the canvas, then the worker enlarges it 2x and refines it, so the clip is delivered at twice the canvas width and height with the same length and audio. targetResolution picks the delivered class: 1080 renders a 544px short-edge canvas (960x544 is delivered at 1920x1088) for
|
|
849
|
+
"description": "\"ltx25\" (default): LTX 2.5 I2V or first/last-frame video with native audio; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflow. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. Which video model to use. \"ltx23\": LTX 2.3 rollback with native audio. \"wan22\": quick simple motion without audio, up to 10s. \"minimax-h3-i2v\" is standard MiniMax H3 from one first frame; \"minimax-h3-i2v-turbo\" is the existing 4-step LightX2V Turbo engine; \"minimax-h3-fasth3-i2v-turbo\" is the separate FastVideo VSA four-step FastH3 engine, about 2x faster and fixed to Euler/simple. \"minimax-h3-fasth3-i2v-turbo-2stage\" (first frame) and \"minimax-h3-fasth3-flf2v-turbo-2stage\" (first and last frame) are the two-stage FastH3 engine: FastH3 renders the canvas, then the worker enlarges it 2x and refines it, so the clip is delivered at twice the canvas width and height with the same length and audio. targetResolution picks the delivered class: 1080 renders a 544px short-edge canvas (960x544 is delivered at 1920x1088) for 10 Spark per second, 1440 or omitted renders the 768p canvas for 2K (1344x768 is delivered at 2688x1536) for 16 Spark per second, and 720 renders a 384px canvas (672x384 is delivered at 1344x768) for the regular FastH3 price of 4 Spark per second. The estimate prices every request. They take the same inputs, durations and LoRAs as their FastH3 selectors. Choose them when the user asks for 1080p, 1440p or 2K MiniMax H3 output, for two-stage output, or for the sharpest/best H3 quality; for ordinary 768p FastH3 output keep the regular FastH3 selector at targetResolution 768. The matching FLF2V selectors provide standard, LightX2V Turbo, FastH3 Turbo, and two-stage FastH3 first/last-frame generation; FastH3 has no R2V mode; use frameRole=\"both\" and provide the end frame. H3 generates native audio at fixed 24fps for 5.17-15.08s and has no negative-prompt input. H3 Base and Turbo prompts use the exact three-field contract and the official mode-specific alignment line. Do not set Seedance here; use generate_video with Seedance references. \"wan3.0-video\" is Alibaba Wan 3 and \"wan3.0-spicy-video\" is MuleRouter w3.0-video. Both render 2-30s at fixed 30 fps with optional native audio, provider prompt expansion, 480p/720p/1080p, adaptive/fixed ratios, first/last frames, and up to 10 image/5 video/5 audio references. Only Alibaba wan3.0-video accepts document/web context and watermark. Frame anchors and loose references are mutually exclusive. Do not send negativePrompt; video references are loose conditioning for a new result, not source-video editing or extension."
|
|
838
850
|
},
|
|
839
851
|
"negativePrompt": {
|
|
840
852
|
"type": "string",
|
|
@@ -1032,7 +1044,7 @@
|
|
|
1032
1044
|
"seedance2-mini",
|
|
1033
1045
|
"seedance2-5"
|
|
1034
1046
|
],
|
|
1035
|
-
"description": "Model selector for this video-to-video request. Usually omit: non-Seedance controls default to \"ltx25-v2v\"; use \"ltx23-v2v\" only for rollback. LTX 2.5 Fast, HQ, and Pro currently use the release-validated official Distilled workflow for canny, pose, depth, detailer, inpaint, and outpaint. Dev is not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. For controlMode=\"seedance-v2v\", Seedance quality is selected only by model: use \"seedance2-mini\" for faster/lower-cost drafts and use \"seedance2\" for full-quality Seedance or 1080p/4K. \"seedance2-5\" supports 480p/720p, 4-30s at 24 fps, native audio, and first/last-frame conditioning; keep \"seedance2\" for
|
|
1047
|
+
"description": "Model selector for this video-to-video request. Usually omit: non-Seedance controls default to \"ltx25-v2v\"; use \"ltx23-v2v\" only for rollback. LTX 2.5 Fast, HQ, and Pro currently use the release-validated official Distilled workflow for canny, pose, depth, detailer, inpaint, and outpaint. Dev is not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. For controlMode=\"seedance-v2v\", Seedance quality is selected only by model: use \"seedance2-mini\" for faster/lower-cost drafts and use \"seedance2\" for full-quality Seedance or 1080p/4K. \"seedance2-5\" supports 480p/720p/1080p, 4-30s at 24 fps, native audio, and first/last-frame conditioning; keep \"seedance2\" for 4K."
|
|
1036
1048
|
},
|
|
1037
1049
|
"generateAudio": {
|
|
1038
1050
|
"type": "boolean",
|
|
@@ -1040,7 +1052,7 @@
|
|
|
1040
1052
|
},
|
|
1041
1053
|
"targetResolution": {
|
|
1042
1054
|
"type": "number",
|
|
1043
|
-
"description": "Seedance V2V only. Short-side output resolution target in pixels. Use when the user asks for a bare named resolution such as \"480p\", \"720p\", \"1080p\", \"2160p\", or \"4K\" without exact dimensions. Seedance V2V full supports 4K; Seedance V2V Mini
|
|
1055
|
+
"description": "Seedance V2V only. Short-side output resolution target in pixels. Use when the user asks for a bare named resolution such as \"480p\", \"720p\", \"1080p\", \"2160p\", or \"4K\" without exact dimensions. Seedance V2V full supports 4K; Seedance V2V Mini and Fast support 480p and 720p only; Seedance 2.5 also supports 1080p, so never set 4K for \"seedance2-5\". Preserve the source video shape instead of forcing landscape pixels."
|
|
1044
1056
|
},
|
|
1045
1057
|
"sourceImageIndex": {
|
|
1046
1058
|
"type": "number",
|
|
@@ -1084,6 +1096,18 @@
|
|
|
1084
1096
|
"description": "Number of video variations to generate (1-16). Default: 1.",
|
|
1085
1097
|
"minimum": 1,
|
|
1086
1098
|
"maximum": 16
|
|
1099
|
+
},
|
|
1100
|
+
"outputFormat": {
|
|
1101
|
+
"type": "string",
|
|
1102
|
+
"enum": [
|
|
1103
|
+
"mp4",
|
|
1104
|
+
"mov"
|
|
1105
|
+
],
|
|
1106
|
+
"description": "Video container. Defaults to mp4. MOV is supported only by Seedance 2.5; choose it when the user requests MOV for editing."
|
|
1107
|
+
},
|
|
1108
|
+
"returnLastFrame": {
|
|
1109
|
+
"type": "boolean",
|
|
1110
|
+
"description": "Seedance 2.5 only. Set true to export a separate image of the final frame alongside the video. The result includes lastFrameUrl, which can be used as the first-frame image for a subsequent clip. Defaults to false; this does not extend the video automatically."
|
|
1087
1111
|
}
|
|
1088
1112
|
},
|
|
1089
1113
|
"required": [
|
|
@@ -1268,13 +1292,13 @@
|
|
|
1268
1292
|
"type": "function",
|
|
1269
1293
|
"function": {
|
|
1270
1294
|
"name": "sound_to_video",
|
|
1271
|
-
"description": "Generate video synchronized to audio. Use when the user has uploaded an audio file (mp3, wav, m4a, flac) and the audio is the primary sync target, especially uploaded-audio-only workflows. Also use after generate_music (\"turn that song into a video\", \"make a music video from that\"). Auto-detects generated audio from generate_music if no audio file is uploaded. Seedance animate_photo/generate_video can also attach uploaded audio as a loose @Audio reference when an image or video reference anchors the request; use this tool instead when the soundtrack itself should drive the video. If the user provides a reference image, use ltx25-ia2v by default (ltx23-ia2v is rollback); for lip-sync with a face image, use wan-s2v; if no image, use ltx25-a2v by default (ltx23-a2v is rollback). If the user wants dialogue/audio WITHOUT pre-existing audio, use animate_photo instead (LTX 2.5 and LTX 2.3 generate audio natively). Note: Persona voice clips from resolve_personas are NOT used by this tool — for persona voice identity in video, use animate_photo or generate_video with videoModel=\"ltx23\" because LTX 2.5 has no compatible ID-LoRA. LONG AUDIO ON SEEDANCE: Seedance 2.0 and Mini cap each clip at 15s; Seedance 2.5 renders up to 30s in one call, so prefer seedance2-5 for 16-30s audio instead of splitting. When the user uploads audio longer than the per-clip cap of the selected model and Seedance is selected (seedance2, seedance2-mini, or seedance2-5), do NOT clamp to 15s and drop the rest — split the run into multiple sound_to_video calls in the same turn (one per 15s segment, so a 20s audio becomes two clips: audioStart=0 duration=15, then audioStart=15 duration=5) and finish with a single stitch_video call referencing the resulting clip indices in order with audioIndex pointing at the same uploaded audio so the stitched output carries the full original soundtrack. LTX/WAN models accept up to 20s per clip, so single-call is fine for them. Use videoModel=\"wan3.0-video\" when the user explicitly requests Wan 3 audio-driven video. Use videoModel=\"wan3.0-spicy-video\" for Wan 3.0 Enhanced audio-driven video through MuleRouter provider ID w3.0-video; it supports adaptive ratios and provider prompt expansion.",
|
|
1295
|
+
"description": "Generate video synchronized to audio. Use when the user has uploaded an audio file (mp3, wav, m4a, flac) and the audio is the primary sync target, especially uploaded-audio-only workflows. Also use after generate_music (\"turn that song into a video\", \"make a music video from that\"). Auto-detects generated audio from generate_music if no audio file is uploaded. Seedance animate_photo/generate_video can also attach uploaded audio as a loose @Audio reference when an image or video reference anchors the request; use this tool instead when the soundtrack itself should drive the video. If the user provides a reference image, use ltx25-ia2v by default (ltx23-ia2v is rollback); for lip-sync with a face image, use wan-s2v; if no image, use ltx25-a2v by default (ltx23-a2v is rollback). If the user wants dialogue/audio WITHOUT pre-existing audio, use animate_photo instead (LTX 2.5 and LTX 2.3 generate audio natively). Note: Persona voice clips from resolve_personas are NOT used by this tool — for persona voice identity in video, use animate_photo or generate_video with videoModel=\"ltx23\" because LTX 2.5 has no compatible ID-LoRA. LONG AUDIO ON SEEDANCE: Seedance 2.0 and Mini cap each clip at 15s; Seedance 2.5 renders up to 30s in one call, so prefer seedance2-5 for 16-30s audio instead of splitting. When the user uploads audio longer than the per-clip cap of the selected model and Seedance is selected (seedance2, seedance2-mini, or seedance2-5), do NOT clamp to 15s and drop the rest — split the run into multiple sound_to_video calls in the same turn (one per 15s segment, so a 20s audio becomes two clips: audioStart=0 duration=15, then audioStart=15 duration=5) and finish with a single stitch_video call referencing the resulting clip indices in order with audioIndex pointing at the same uploaded audio so the stitched output carries the full original soundtrack. LTX/WAN models accept up to 20s per clip, so single-call is fine for them. Use videoModel=\"wan3.0-video\" when the user explicitly requests Wan 3 audio-driven video. Use videoModel=\"wan3.0-spicy-video\" for Wan 3.0 Enhanced audio-driven video through MuleRouter provider ID w3.0-video; it supports adaptive ratios and provider prompt expansion. MINIMAX H3 AUDIO: only when the user asks for MiniMax H3 or FastH3 audio-driven video, use videoModel=\"minimax-h3-fasth3-ia2v-turbo\" with a first-frame image (sourceImageIndex), \"minimax-h3-fasth3-flfa2v-turbo\" with a first and a last frame (sourceImageIndex and endImageIndex), or \"minimax-h3-fasth3-a2v-turbo\" with no image; add \"-2stage\" only for 1080p, 1440p or 2K H3 output. Without an explicit MiniMax H3 or FastH3 request keep the LTX 2.5 defaults. The MiniMax H3 selectors on generate_video and animate_photo cannot take an uploaded audio track; send uploaded-audio H3 requests here.",
|
|
1272
1296
|
"parameters": {
|
|
1273
1297
|
"type": "object",
|
|
1274
1298
|
"properties": {
|
|
1275
1299
|
"prompt": {
|
|
1276
1300
|
"type": "string",
|
|
1277
|
-
"description": "Describe the video like a cinematographer. Let the audio define timing — use the prompt for visual interpretation. One flowing paragraph, present tense, specific natural language.\n\nLITERAL PROMPT OVERRIDE: If the user explicitly says not to modify the prompt, or to use it exactly/verbatim/as-is, copy the identified prompt text verbatim instead of applying these construction rules unless a hard requirement is missing. For Seedance or Wan 3, set expandPrompt=false.\n\nSTRUCTURE: shot/style and scale → subject → environment, lighting, color, texture, atmosphere → visual action synced to audio → camera movement. For LTX 2.3 image+audio mode, do not re-describe static details already visible in the reference image; focus on motion, action, camera, and how the image responds to the audio.\n\nMOTION PACING: Scale complexity to duration. <=6s: 1 main visual beat + 1 simple camera move. Around 10s: 2-3 clear beats + 1 camera move. >10s: up to 4 beats in clear sequence. Let the audio define timing, but avoid stacking subject, camera, and environment motion in short clips.\n\nBLOCKING: Direct layout when it affects the shot: left/right placement, foreground/background, facing direction, and relative distance between subjects.\n\nLIP-SYNC: Shot framing, speaker's appearance and setting, physical performance synced to audio — gestures, expressions, jaw movement between phrases. Include acting beats.\n\nMUSIC VISUALIZATION: Visual style, environment, and how elements react to rhythm and energy.\n\nAUDIO-REACTIVE: Motion and visual changes that correspond to sounds in the track.\n\nLTX VOCABULARY: camera (tracking, dolly, pan, tilt, handheld, static frame), lighting/atmosphere (golden hour, neon glow, dramatic shadows, fog, rain, smoke, reflections), scale/pacing (expansive, epic, intimate, claustrophobic, slow motion, time-lapse, lingering shot, continuous shot), style/genre (film noir, painterly, cyberpunk, stop-motion, claymation, 2D/3D animation, hand-drawn, fantasy, thriller, experimental film).\n\nAVOID: Vague prompts, too many competing visual elements, abstract descriptions without visible behavior, rigid numeric constraints, readable text or logos. QUOTING RULE: ONLY use double quotes for spoken dialogue. Never quote on-screen text, overlay text, titles, captions, signs, or any visual text — describe them without quotes.\n\nNON-SEEDANCE POSITIVE CONSTRAINTS: For ltx25-ia2v, ltx25-a2v, ltx23-ia2v, ltx23-a2v, and wan-s2v, prompt is a positive prompt. Translate user avoid/no/don't constraints into affirmative production constraints instead of copying negative phrasing. Preserve exact quoted visible text or dialogue when the user explicitly requests it; keep surrounding surfaces blank.\n\nBATCH VARIATIONS: When numberOfVariations > 1, use Dynamic Prompt syntax to vary the visual interpretation while keeping audio sync intent consistent. This is one Sogni project with multiple jobs, so prefer it when all outputs share the same audio source/window, image source, model, duration, dimensions, and parameters and only prompt text varies. Example: \"{abstract neon visualization|nature scene with swaying trees|urban street with rain} synced to the beat\"."
|
|
1301
|
+
"description": "Describe the video like a cinematographer. Let the audio define timing — use the prompt for visual interpretation. One flowing paragraph, present tense, specific natural language.\n\nLITERAL PROMPT OVERRIDE: If the user explicitly says not to modify the prompt, or to use it exactly/verbatim/as-is, copy the identified prompt text verbatim instead of applying these construction rules unless a hard requirement is missing. For Seedance or Wan 3, set expandPrompt=false.\n\nSTRUCTURE: shot/style and scale → subject → environment, lighting, color, texture, atmosphere → visual action synced to audio → camera movement. For LTX 2.3 image+audio mode, do not re-describe static details already visible in the reference image; focus on motion, action, camera, and how the image responds to the audio.\n\nMOTION PACING: Scale complexity to duration. <=6s: 1 main visual beat + 1 simple camera move. Around 10s: 2-3 clear beats + 1 camera move. >10s: up to 4 beats in clear sequence. Let the audio define timing, but avoid stacking subject, camera, and environment motion in short clips.\n\nBLOCKING: Direct layout when it affects the shot: left/right placement, foreground/background, facing direction, and relative distance between subjects.\n\nLIP-SYNC: Shot framing, speaker's appearance and setting, physical performance synced to audio — gestures, expressions, jaw movement between phrases. Include acting beats.\n\nMUSIC VISUALIZATION: Visual style, environment, and how elements react to rhythm and energy.\n\nAUDIO-REACTIVE: Motion and visual changes that correspond to sounds in the track.\n\nLTX VOCABULARY: camera (tracking, dolly, pan, tilt, handheld, static frame), lighting/atmosphere (golden hour, neon glow, dramatic shadows, fog, rain, smoke, reflections), scale/pacing (expansive, epic, intimate, claustrophobic, slow motion, time-lapse, lingering shot, continuous shot), style/genre (film noir, painterly, cyberpunk, stop-motion, claymation, 2D/3D animation, hand-drawn, fantasy, thriller, experimental film).\n\nAVOID: Vague prompts, too many competing visual elements, abstract descriptions without visible behavior, rigid numeric constraints, readable text or logos. QUOTING RULE: ONLY use double quotes for spoken dialogue. Never quote on-screen text, overlay text, titles, captions, signs, or any visual text — describe them without quotes.\n\nNON-SEEDANCE POSITIVE CONSTRAINTS: For ltx25-ia2v, ltx25-a2v, ltx23-ia2v, ltx23-a2v, and wan-s2v, prompt is a positive prompt. Translate user avoid/no/don't constraints into affirmative production constraints instead of copying negative phrasing. Preserve exact quoted visible text or dialogue when the user explicitly requests it; keep surrounding surfaces blank.\n\nBATCH VARIATIONS: When numberOfVariations > 1, use Dynamic Prompt syntax to vary the visual interpretation while keeping audio sync intent consistent. This is one Sogni project with multiple jobs, so prefer it when all outputs share the same audio source/window, image source, model, duration, dimensions, and parameters and only prompt text varies. Example: \"{abstract neon visualization|nature scene with swaying trees|urban street with rain} synced to the beat\".\n\nMINIMAX H3 AUDIO SELECTORS: write the request plainly (subject, action, camera, the voice or sound heard in the upload and any exact words spoken in it); the MiniMax H3 prompt shaper turns it into the H3 contract for the matching image-to-video, first-and-last-frame or text-to-video mode. Describe the uploaded audio as it is, and do not invent other dialogue or music."
|
|
1278
1302
|
},
|
|
1279
1303
|
"expandPrompt": {
|
|
1280
1304
|
"type": "boolean",
|
|
@@ -1282,7 +1306,7 @@
|
|
|
1282
1306
|
},
|
|
1283
1307
|
"negativePrompt": {
|
|
1284
1308
|
"type": "string",
|
|
1285
|
-
"description": "Advanced LTX 2.5/LTX 2.3/WAN only. The LTX A2V and IA2V workflows accept this separate negative prompt. Use it only when the user explicitly asks to set one. Do not set for Seedance.\n\nWan 3 has no negativePrompt request field; do not set this for wan3.0-video."
|
|
1309
|
+
"description": "Advanced LTX 2.5/LTX 2.3/WAN only. The LTX A2V and IA2V workflows accept this separate negative prompt. Use it only when the user explicitly asks to set one. Do not set for Seedance.\n\nWan 3 has no negativePrompt request field; do not set this for wan3.0-video.\n\nMiniMax H3 has no negative-prompt input; do not set this for the MiniMax H3 FastH3 audio selectors and state exclusions positively in prompt."
|
|
1286
1310
|
},
|
|
1287
1311
|
"audioSourceIndex": {
|
|
1288
1312
|
"type": "number",
|
|
@@ -1290,16 +1314,20 @@
|
|
|
1290
1314
|
},
|
|
1291
1315
|
"sourceImageIndex": {
|
|
1292
1316
|
"type": "number",
|
|
1293
|
-
"description": "Optional index of an uploaded image to use as the starting frame (0-based). Required for lip-sync models (WAN S2V). For audio-only-to-video models (LTX 2.3 A2V), this is optional — omit it to generate video purely from text + audio."
|
|
1317
|
+
"description": "Optional index of an uploaded image to use as the starting frame (0-based). Required for lip-sync models (WAN S2V). For audio-only-to-video models (LTX 2.3 A2V), this is optional — omit it to generate video purely from text + audio. MiniMax H3 FastH3 audio guide: \"minimax-h3-fasth3-ia2v-turbo\" and \"minimax-h3-fasth3-flfa2v-turbo\" (and their -2stage forms) require this first frame; \"minimax-h3-fasth3-a2v-turbo\" and its -2stage form take no image, so omit it for them."
|
|
1318
|
+
},
|
|
1319
|
+
"endImageIndex": {
|
|
1320
|
+
"type": "number",
|
|
1321
|
+
"description": "Which image to use as the LAST frame, indexed like animate_photo's endImageIndex: negative indices for uploaded images (-1 = first upload, -2 = second upload) and 0-based non-negative indices for generated results. Only \"minimax-h3-fasth3-flfa2v-turbo\" and \"minimax-h3-fasth3-flfa2v-turbo-2stage\" take it, and they require it together with sourceImageIndex (for two uploaded images use sourceImageIndex=-1 and endImageIndex=-2). Omit it for every other videoModel."
|
|
1294
1322
|
},
|
|
1295
1323
|
"audioStart": {
|
|
1296
1324
|
"type": "number",
|
|
1297
|
-
"description": "Start offset in seconds into the audio track. Use when the user says \"start 20 seconds in\", \"skip the intro\", \"use the chorus at 1:30\", etc. Default: 0 (beginning of audio). The video will be synced to the audio starting from this point.",
|
|
1325
|
+
"description": "Start offset in seconds into the audio track. Use when the user says \"start 20 seconds in\", \"skip the intro\", \"use the chorus at 1:30\", etc. Default: 0 (beginning of audio). The video will be synced to the audio starting from this point. The MiniMax H3 FastH3 audio selectors take audioStart too: the clip uses the upload from audioStart for its own length.",
|
|
1298
1326
|
"minimum": 0
|
|
1299
1327
|
},
|
|
1300
1328
|
"duration": {
|
|
1301
1329
|
"type": "number",
|
|
1302
|
-
"description": "Video duration in seconds. Default: 5. Per-model range: LTX/WAN 2.2 = 2-20s; Wan 3 = 2-30s; Seedance 2.0 and Mini = 4-15s; Seedance 2.5 = 4-30s. For music videos, use the maximum duration the selected model allows because the audio is usually longer than the video limit. Use when the user explicitly requests a specific length.",
|
|
1330
|
+
"description": "Video duration in seconds. Default: 5. Per-model range: LTX/WAN 2.2 = 2-20s; Wan 3 = 2-30s; Seedance 2.0 and Mini = 4-15s; Seedance 2.5 = 4-30s. For music videos, use the maximum duration the selected model allows because the audio is usually longer than the video limit. Use when the user explicitly requests a specific length. MiniMax H3 FastH3 audio selectors render 124-362 frames on a 17-frame grid at a fixed 24 fps, so the clip runs about 5.2-15.1 seconds and a length outside that window snaps to it; for longer audio pick the window with audioStart.",
|
|
1303
1331
|
"minimum": 2,
|
|
1304
1332
|
"maximum": 30
|
|
1305
1333
|
},
|
|
@@ -1339,13 +1367,19 @@
|
|
|
1339
1367
|
"ltx23-ia2v",
|
|
1340
1368
|
"ltx23-a2v",
|
|
1341
1369
|
"wan3.0-video",
|
|
1342
|
-
"wan3.0-spicy-video"
|
|
1370
|
+
"wan3.0-spicy-video",
|
|
1371
|
+
"minimax-h3-fasth3-ia2v-turbo",
|
|
1372
|
+
"minimax-h3-fasth3-ia2v-turbo-2stage",
|
|
1373
|
+
"minimax-h3-fasth3-flfa2v-turbo",
|
|
1374
|
+
"minimax-h3-fasth3-flfa2v-turbo-2stage",
|
|
1375
|
+
"minimax-h3-fasth3-a2v-turbo",
|
|
1376
|
+
"minimax-h3-fasth3-a2v-turbo-2stage"
|
|
1343
1377
|
],
|
|
1344
|
-
"description": "\"ltx25-ia2v\" (default with image) and \"ltx25-a2v\" (default without image): LTX 2.5 image+audio and audio-only modes; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflows. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates official ComfyUI Dev recipes. Video model. \"ltx23-ia2v\" (rollback with image): LTX 2.3 image+audio to video, audio-reactive with a reference image; Fast/HQ use the distilled 8-step worker and Default Media Quality Pro uses the non-distilled dev worker. \"ltx23-a2v\" (rollback without image): LTX 2.3 audio-only to video, no image needed, creates video purely from text prompt + audio with the same quality-tier routing. \"wan-s2v\": WAN 2.2 sound-to-video, best for lip-sync with a face image, fast 4-step. \"seedance2\": full Seedance 2.0 audio-reference video, 4-15s. \"seedance2-mini\": Seedance 2.0 Mini, 720p cap, fastest/lower-cost Seedance option. Seedance quality is selected only by this model value: pick \"seedance2-mini\" for faster/lower-cost drafts or explicit Mini requests, and pick \"seedance2\" for full-quality Seedance or 1080p/4K. Do not infer the Seedance model from Default Media Quality Fast/HQ/Pro. \"seedance2-5\": Seedance 2.5, the newest Seedance generation — 480p and
|
|
1378
|
+
"description": "\"ltx25-ia2v\" (default with image) and \"ltx25-a2v\" (default without image): LTX 2.5 image+audio and audio-only modes; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflows. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates official ComfyUI Dev recipes. Video model. \"ltx23-ia2v\" (rollback with image): LTX 2.3 image+audio to video, audio-reactive with a reference image; Fast/HQ use the distilled 8-step worker and Default Media Quality Pro uses the non-distilled dev worker. \"ltx23-a2v\" (rollback without image): LTX 2.3 audio-only to video, no image needed, creates video purely from text prompt + audio with the same quality-tier routing. \"wan-s2v\": WAN 2.2 sound-to-video, best for lip-sync with a face image, fast 4-step. \"seedance2\": full Seedance 2.0 audio-reference video, 4-15s. \"seedance2-mini\": Seedance 2.0 Mini, 720p cap, fastest/lower-cost Seedance option. Seedance quality is selected only by this model value: pick \"seedance2-mini\" for faster/lower-cost drafts or explicit Mini requests, and pick \"seedance2\" for full-quality Seedance or 1080p/4K. Do not infer the Seedance model from Default Media Quality Fast/HQ/Pro. \"seedance2-5\": Seedance 2.5, the newest Seedance generation — 480p, 720p, and 1080p (4K is unsupported), 4-30s per clip at a fixed 24 fps, native audio, first-and-last-frame conditioning, and a much larger reference budget than the 2.0 family: up to 30 images, 10 videos, and 10 audios, with up to 50 reference media files total, subject to those per-modality caps. Choose \"seedance2-5\" when the user asks for Seedance 2.5, wants a single continuous Seedance clip longer than 15s (2.5 renders up to 30s in one call instead of being split and stitched), or wants a first-and-last-frame Seedance transition. Keep \"seedance2\" for 4K requests; Seedance 2.5 supports up to 1080p. For Seedance audio-reference prompts, preserve exact spoken dialogue when the user supplied it, and assign @Image1/@Audio1 roles. If the user asks for speech without words, describe the vocal performance without inventing quoted dialogue. Treat lip-sync, voice cloning, and real-human reference behavior as provider-sensitive rather than guaranteed. Omit to auto-select based on whether an image is present. \"wan3.0-video\" is Alibaba Wan 3 and \"wan3.0-spicy-video\" is MuleRouter w3.0-video. Both render 2-30s at fixed 30 fps with optional native audio, provider prompt expansion, 480p/720p/1080p, adaptive/fixed ratios, first/last frames, and up to 10 image/5 video/5 audio references. Only Alibaba wan3.0-video accepts document/web context and watermark. Frame anchors and loose references are mutually exclusive. Do not send negativePrompt; video references are loose conditioning for a new result, not source-video editing or extension. MiniMax H3 FastH3 audio guide: \"minimax-h3-fasth3-ia2v-turbo\" (first-frame image from sourceImageIndex plus the audio), \"minimax-h3-fasth3-flfa2v-turbo\" (first frame from sourceImageIndex, last frame from endImageIndex, plus the audio) and \"minimax-h3-fasth3-a2v-turbo\" (audio only; set neither sourceImageIndex nor endImageIndex) run the four-step FastH3 engine with the uploaded audio driving the picture from frame 0, and the output keeps that audio as its soundtrack. Choose them only when the user asks for MiniMax H3 or FastH3; never pick them in place of the LTX 2.5 defaults. They render 124-362 frames on the H3 17-frame grid at a fixed 24 fps (about 5.2-15.1 seconds) on a 32px grid within 1344x768, so use targetResolution 768 or omit it. audioStart picks the window of the upload. generateAudio=false, LoRAs and negativePrompt are not supported. Each costs the FastH3 price of its image-to-video, first-and-last-frame or text-to-video mode, 4 Spark per second. \"minimax-h3-fasth3-ia2v-turbo-2stage\", \"minimax-h3-fasth3-flfa2v-turbo-2stage\" and \"minimax-h3-fasth3-a2v-turbo-2stage\" are the two-stage forms: the same inputs rendered on the FastH3 canvas, then enlarged 2x and refined, delivered at twice the canvas with the same length and audio. For them targetResolution names the delivered class: 1080 renders a 544px short-edge canvas (960x544 is delivered at 1920x1088) for 10 Spark per second, and 1440 or omitted renders the 768p canvas for 2K (1344x768 is delivered at 2688x1536) for 16 Spark per second. The audio two-stage selectors have no 720p price class (720 would render a 384px canvas at the 1080p rate), so for 720p or 768p audio-guided H3 output use the regular selector. The estimate prices every request."
|
|
1345
1379
|
},
|
|
1346
1380
|
"generateAudio": {
|
|
1347
1381
|
"type": "boolean",
|
|
1348
|
-
"description": "Whether the returned video should include audio. Omit to include audio by default; set false when the user asks for silent output or no audio. The reference audio is still required and still drives generation even when the returned video has no audio track."
|
|
1382
|
+
"description": "Whether the returned video should include audio. Omit to include audio by default; set false when the user asks for silent output or no audio. The reference audio is still required and still drives generation even when the returned video has no audio track. MiniMax H3 FastH3 audio selectors always deliver the uploaded audio: omit generateAudio for them (false is refused)."
|
|
1349
1383
|
},
|
|
1350
1384
|
"numberOfVariations": {
|
|
1351
1385
|
"type": "number",
|
|
@@ -1360,6 +1394,18 @@
|
|
|
1360
1394
|
"aspectRatio": {
|
|
1361
1395
|
"type": "string",
|
|
1362
1396
|
"description": "Do NOT set unless the user explicitly requests an aspect ratio, format, orientation, or exact pixel dimensions. When a reference/source image is used and the user did not ask to change its shape, omit this field so the handler preserves the selected source image's own ratio.\n\nFormats: \"16:9\", \"9:16\", \"4:5\", \"1:1\", \"4:3\", \"3:2\", \"21:9\", or exact pixels like \"1920x1080\".\n\nCRITICAL: When the user specifies exact pixel dimensions (e.g., \"1280x720\", \"1080x1920\", \"1920x1080\", \"3840x2160\") or an orientation-qualified named resolution (e.g., \"720p landscape\", \"720p portrait\"), use the exact pixel format, NOT a ratio like \"16:9\" or \"9:16\". Exact user-requested dimensions override the selected default media quality, including Pro/HQ defaults. A bare named video resolution like \"720p resolution\" is only a resolution tier/short-side request; do not turn it into landscape pixels and do not set aspectRatio unless the user also states landscape, portrait, vertical, horizontal, or exact pixels. If requested pixels are in bounds but not on the model's pixel step, still pass the user's exact pixel request; the handler snaps to the nearest supported size internally. Only use ratio format when the user says a generic format name without pixel dimensions.\n\nMappings (use ONLY when user does NOT specify pixel dimensions): landscape/widescreen/YouTube/cinematic → \"16:9\". portrait → \"9:16\". TikTok/Reels/IG Reels → \"1080x1920\". ultrawide/cinema scope → \"21:9\". Instagram post → \"4:5\". square → \"1:1\". standard/TV → \"4:3\". 720p landscape → \"1280x720\". 720p portrait → \"720x1280\". 1080p landscape → \"1920x1080\". 1080p portrait/HD portrait → \"1080x1920\". 4K landscape → \"3840x2160\". 4K portrait → \"2160x3840\". Never set for generic requests like \"make a video\"."
|
|
1397
|
+
},
|
|
1398
|
+
"outputFormat": {
|
|
1399
|
+
"type": "string",
|
|
1400
|
+
"enum": [
|
|
1401
|
+
"mp4",
|
|
1402
|
+
"mov"
|
|
1403
|
+
],
|
|
1404
|
+
"description": "Video container. Defaults to mp4. MOV is supported only by Seedance 2.5; choose it when the user requests MOV for editing."
|
|
1405
|
+
},
|
|
1406
|
+
"returnLastFrame": {
|
|
1407
|
+
"type": "boolean",
|
|
1408
|
+
"description": "Seedance 2.5 only. Set true to export a separate image of the final frame alongside the video. The result includes lastFrameUrl, which can be used as the first-frame image for a subsequent clip. Defaults to false; this does not extend the video automatically."
|
|
1363
1409
|
}
|
|
1364
1410
|
},
|
|
1365
1411
|
"required": [
|
|
@@ -204,7 +204,7 @@
|
|
|
204
204
|
"wan3.0-video",
|
|
205
205
|
"wan3.0-spicy-video"
|
|
206
206
|
],
|
|
207
|
-
"description": "\"ltx25\" (default): LTX 2.5 with native audio; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflow. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. Video model. \"ltx23\": LTX 2.3 rollback with native audio. \"wan22\": quick simple motion without audio. \"minimax-h3-t2v\": standard 20-step MiniMax H3 text-to-video; \"minimax-h3-t2v-turbo\": the existing 4-step LightX2V Turbo text-to-video; \"minimax-h3-fasth3-t2v-turbo\": the separate FastVideo VSA four-step FastH3 engine, about 2x faster and fixed to Euler/simple; \"minimax-h3-fasth3-t2v-turbo-2stage\": the two-stage FastH3 engine: FastH3 renders the canvas, then the worker enlarges it 2x and refines it, so the clip is delivered at twice the canvas width and height with the same length and audio. targetResolution picks the delivered class: 1080 renders a 544px short-edge canvas (960x544 is delivered at 1920x1088) for
|
|
207
|
+
"description": "\"ltx25\" (default): LTX 2.5 with native audio; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflow. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. Video model. \"ltx23\": LTX 2.3 rollback with native audio. \"wan22\": quick simple motion without audio. \"minimax-h3-t2v\": standard 20-step MiniMax H3 text-to-video; \"minimax-h3-t2v-turbo\": the existing 4-step LightX2V Turbo text-to-video; \"minimax-h3-fasth3-t2v-turbo\": the separate FastVideo VSA four-step FastH3 engine, about 2x faster and fixed to Euler/simple; \"minimax-h3-fasth3-t2v-turbo-2stage\": the two-stage FastH3 engine: FastH3 renders the canvas, then the worker enlarges it 2x and refines it, so the clip is delivered at twice the canvas width and height with the same length and audio. targetResolution picks the delivered class: 1080 renders a 544px short-edge canvas (960x544 is delivered at 1920x1088) for 10 Spark per second, 1440 or omitted renders the 768p canvas for 2K (1344x768 is delivered at 2688x1536) for 16 Spark per second, and 720 renders a 384px canvas (672x384 is delivered at 1344x768) for the regular FastH3 price of 4 Spark per second. The estimate prices every request. It takes the same inputs, durations and LoRAs as \"minimax-h3-fasth3-t2v-turbo\". Choose it when the user asks for 1080p, 1440p or 2K MiniMax H3 output, for two-stage output, or for the sharpest/best H3 quality; for ordinary 768p FastH3 output keep the regular FastH3 selector at targetResolution 768. All use native audio, fixed 24fps, 5.17-15.08s, and a 768p-class 32px-grid canvas; use animate_photo for H3 image-conditioned modes. Base and Turbo T2V/I2V/FLF2V prompts use the exact ordered fields integrated_multimodal_description, overall_soundscape, and non_diegetic_music; I2V/FLF2V prepend the official alignment line. \"minimax-h3-r2v\": standard 20-step MiniMax H3 reference-to-video; \"minimax-h3-r2v-turbo\": the dedicated LightX2V 4-step Ref2VA Turbo workflow using Euler/simple and a 960x544 default. FastH3 has no R2V mode. Both R2V selectors accept up to 9 images, 3 videos, and 3 audios (12 files total); at least one visual reference (image or video) is required and audio alone is invalid. Select references with referenceImageIndices/referenceVideoIndices/referenceAudioIndices and address them with the official <Subject N>/<Picture N>/<Video N>/<Audio N> semantics. Seedance quality is selected only by model: use \"seedance2-mini\" for Seedance 2.0 Mini or faster/lower-cost 720p iteration, and use \"seedance2\" for the full Seedance 2.0 model, explicit full-quality requests, 1080p/4K requests, or generated/uploaded storyboard images unless the user explicitly asks for a draft or Mini. Do not use Default Media Quality Fast/HQ/Pro or targetResolution to represent Seedance quality. \"seedance2-5\": Seedance 2.5, the newest Seedance generation — 480p, 720p, and 1080p (4K is unsupported), 4-30s per clip at a fixed 24 fps, native audio, first-and-last-frame conditioning, and a much larger reference budget than the 2.0 family: up to 30 images, 10 videos, and 10 audios, with up to 50 reference media files total, subject to those per-modality caps. Choose \"seedance2-5\" when the user asks for Seedance 2.5, wants a single continuous Seedance clip longer than 15s (2.5 renders up to 30s in one call instead of being split and stitched), or wants a first-and-last-frame Seedance transition. Keep \"seedance2\" for 4K requests; Seedance 2.5 supports up to 1080p. Seedance supports multimodal loose reference assets: images (up to 9), videos (up to 3), and audios (up to 3), with no more than 12 asset files total. Use @Image1/@Video1/@Audio1 style references in creative briefs when assigning roles. Assign every useful reference asset a role and prefer positive preservation constraints. If an uploaded video is the source clip to transform, upscale, enhance, restyle, or remaster, use video_to_video with controlMode=\"seedance-v2v\" instead of generate_video referenceVideoIndices. Alibaba HappyHorse 1.1 video models (third-party vendor — requires Premium Spark). Select by mode: \"happyhorse-1.1-t2v\" for text-to-video, \"happyhorse-1.1-i2v\" for image-to-video from one first-frame image, and \"happyhorse-1.1-r2v\" for reference-to-video with up to 9 reference images. Resolutions 720P and 1080P; duration 3-15 seconds at 24 fps; native synchronized audio is always generated (do not set generateAudio or negativePrompt). Supported aspect ratios: 16:9, 9:16, 1:1, 4:3, 3:4, 4:5, 5:4, 9:21, 21:9. HappyHorse 1.1 takes image references only and renders a native synchronized audio track (always on; do not set generateAudio or a negative prompt). Pick the model by mode: happyhorse-1.1-t2v for text-to-video (no reference image), happyhorse-1.1-i2v for image-to-video from a single first frame, and happyhorse-1.1-r2v for reference-to-video with 1 to 9 reference images. For r2v, tag the images in the prompt as [Image 1]…[Image 9] and assign each a clear role. HappyHorse does not accept reference videos or reference audios. \"wan3.0-video\" is Alibaba Wan 3 and \"wan3.0-spicy-video\" is MuleRouter w3.0-video. Both render 2-30s at fixed 30 fps with optional native audio, provider prompt expansion, 480p/720p/1080p, adaptive/fixed ratios, first/last frames, and up to 10 image/5 video/5 audio references. Only Alibaba wan3.0-video accepts document/web context and watermark. Frame anchors and loose references are mutually exclusive. Do not send negativePrompt; video references are loose conditioning for a new result, not source-video editing or extension."
|
|
208
208
|
},
|
|
209
209
|
"generateAudio": {
|
|
210
210
|
"type": "boolean",
|
|
@@ -241,7 +241,7 @@
|
|
|
241
241
|
},
|
|
242
242
|
"targetResolution": {
|
|
243
243
|
"type": "number",
|
|
244
|
-
"description": "Short-side video resolution target in pixels. Use when the user asks for a bare named resolution such as \"480p\", \"720p\", \"1080p\", \"2160p\", or \"4K\" without exact pixels or an output orientation. This is resolution only, not a Seedance quality tier: Seedance quality is selected by videoModel (\"seedance2\" vs \"seedance2-mini\" vs \"seedance2-5\"). Seedance 2.0 full supports 4K; Seedance Mini
|
|
244
|
+
"description": "Short-side video resolution target in pixels. Use when the user asks for a bare named resolution such as \"480p\", \"720p\", \"1080p\", \"2160p\", or \"4K\" without exact pixels or an output orientation. This is resolution only, not a Seedance quality tier: Seedance quality is selected by videoModel (\"seedance2\" vs \"seedance2-mini\" vs \"seedance2-5\"). Seedance 2.0 full supports 4K; Seedance Mini supports 480p/720p; Seedance 2.5 supports 480p/720p/1080p, so never set 4K for \"seedance2-5\". Wan 3 supports exactly 480p, 720p, and 1080p. HappyHorse supports only 720p and 1080p. Never set 4K for Wan 3 or HappyHorse. MiniMax H3 renders inside a 1344x768 pixel budget on a 32px grid, so use 768 for the regular H3 selectors and never 1080p or 4K. The two-stage H3 selector \"minimax-h3-fasth3-t2v-turbo-2stage\" delivers twice the canvas, so there targetResolution names the delivered short-edge class: 1080 (544px canvas short edge: 960x544 delivered at 1920x1088), 1440 for 2K (the 1344x768 canvas delivered at 2688x1536), or 720 (384px canvas: 672x384 delivered at 1344x768); omit it for 2K. Never set 4K for H3. Do not set targetResolution from Default Media Quality Fast/HQ/Pro. If omitted for Seedance, Wan 3, HappyHorse, or MiniMax H3, the host uses the selected model default. This preserves/inherits the current video shape instead of forcing landscape. Do NOT set width, height, or exact-pixel aspectRatio for bare named resolution requests. If the user says \"720p portrait\", \"720p landscape\", \"4K portrait\", or \"4K landscape\", use exact width/height/aspectRatio instead."
|
|
245
245
|
},
|
|
246
246
|
"numberOfVariations": {
|
|
247
247
|
"type": "number",
|
|
@@ -275,6 +275,18 @@
|
|
|
275
275
|
"type": "number"
|
|
276
276
|
},
|
|
277
277
|
"description": "Strength for each LoRA in loras, in the same order. Omitting the array applies 1.0 to every LoRA, which is NOT the catalog default and for h3-realism-people is already at the top of its band, so send explicit values. Video LoRAs are positive-only — unlike the bipolar Krea 2 image sliders, a negative value is not an inverse effect and 0 is off. h3-realism-people takes 0-2 and its catalog default is 0.8; 0.6-1 is the usable band. It also pulls the camera in as it climbs: at 1.5 and above the shot reliably recomposes and the grade darkens, which on an image-conditioned mode can crop the subject out of the frame the user supplied. Raise it above 1 only when the user asks for more, and prefer the default when they supplied a first or last frame. h3-vbvr-video-reasoning and h3-mystic-xxx-v4 both take 0-1 and do default to 1.0, with usable bands of 0.7-1 and 0.2-1. h3-natural-face-speech and h3-better-motion take 0-1.5 and default to 0.6; their usable band is 0.4-0.8."
|
|
278
|
+
},
|
|
279
|
+
"outputFormat": {
|
|
280
|
+
"type": "string",
|
|
281
|
+
"enum": [
|
|
282
|
+
"mp4",
|
|
283
|
+
"mov"
|
|
284
|
+
],
|
|
285
|
+
"description": "Video container. Defaults to mp4. MOV is supported only by Seedance 2.5; choose it when the user requests MOV for editing."
|
|
286
|
+
},
|
|
287
|
+
"returnLastFrame": {
|
|
288
|
+
"type": "boolean",
|
|
289
|
+
"description": "Seedance 2.5 only. Set true to export a separate image of the final frame alongside the video. The result includes lastFrameUrl, which can be used as the first-frame image for a subsequent clip. Defaults to false; this does not extend the video automatically."
|
|
278
290
|
}
|
|
279
291
|
},
|
|
280
292
|
"required": [
|
|
@@ -774,7 +786,7 @@
|
|
|
774
786
|
"wan3.0-video",
|
|
775
787
|
"wan3.0-spicy-video"
|
|
776
788
|
],
|
|
777
|
-
"description": "\"ltx25\" (default): LTX 2.5 I2V or first/last-frame video with native audio; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflow. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. Which video model to use. \"ltx23\": LTX 2.3 rollback with native audio. \"wan22\": quick simple motion without audio, up to 10s. \"minimax-h3-i2v\" is standard MiniMax H3 from one first frame; \"minimax-h3-i2v-turbo\" is the existing 4-step LightX2V Turbo engine; \"minimax-h3-fasth3-i2v-turbo\" is the separate FastVideo VSA four-step FastH3 engine, about 2x faster and fixed to Euler/simple. \"minimax-h3-fasth3-i2v-turbo-2stage\" (first frame) and \"minimax-h3-fasth3-flf2v-turbo-2stage\" (first and last frame) are the two-stage FastH3 engine: FastH3 renders the canvas, then the worker enlarges it 2x and refines it, so the clip is delivered at twice the canvas width and height with the same length and audio. targetResolution picks the delivered class: 1080 renders a 544px short-edge canvas (960x544 is delivered at 1920x1088) for
|
|
789
|
+
"description": "\"ltx25\" (default): LTX 2.5 I2V or first/last-frame video with native audio; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflow. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. Which video model to use. \"ltx23\": LTX 2.3 rollback with native audio. \"wan22\": quick simple motion without audio, up to 10s. \"minimax-h3-i2v\" is standard MiniMax H3 from one first frame; \"minimax-h3-i2v-turbo\" is the existing 4-step LightX2V Turbo engine; \"minimax-h3-fasth3-i2v-turbo\" is the separate FastVideo VSA four-step FastH3 engine, about 2x faster and fixed to Euler/simple. \"minimax-h3-fasth3-i2v-turbo-2stage\" (first frame) and \"minimax-h3-fasth3-flf2v-turbo-2stage\" (first and last frame) are the two-stage FastH3 engine: FastH3 renders the canvas, then the worker enlarges it 2x and refines it, so the clip is delivered at twice the canvas width and height with the same length and audio. targetResolution picks the delivered class: 1080 renders a 544px short-edge canvas (960x544 is delivered at 1920x1088) for 10 Spark per second, 1440 or omitted renders the 768p canvas for 2K (1344x768 is delivered at 2688x1536) for 16 Spark per second, and 720 renders a 384px canvas (672x384 is delivered at 1344x768) for the regular FastH3 price of 4 Spark per second. The estimate prices every request. They take the same inputs, durations and LoRAs as their FastH3 selectors. Choose them when the user asks for 1080p, 1440p or 2K MiniMax H3 output, for two-stage output, or for the sharpest/best H3 quality; for ordinary 768p FastH3 output keep the regular FastH3 selector at targetResolution 768. The matching FLF2V selectors provide standard, LightX2V Turbo, FastH3 Turbo, and two-stage FastH3 first/last-frame generation; FastH3 has no R2V mode; use frameRole=\"both\" and provide the end frame. H3 generates native audio at fixed 24fps for 5.17-15.08s and has no negative-prompt input. H3 Base and Turbo prompts use the exact three-field contract and the official mode-specific alignment line. Do not set Seedance here; use generate_video with Seedance references. \"wan3.0-video\" is Alibaba Wan 3 and \"wan3.0-spicy-video\" is MuleRouter w3.0-video. Both render 2-30s at fixed 30 fps with optional native audio, provider prompt expansion, 480p/720p/1080p, adaptive/fixed ratios, first/last frames, and up to 10 image/5 video/5 audio references. Only Alibaba wan3.0-video accepts document/web context and watermark. Frame anchors and loose references are mutually exclusive. Do not send negativePrompt; video references are loose conditioning for a new result, not source-video editing or extension."
|
|
778
790
|
},
|
|
779
791
|
"negativePrompt": {
|
|
780
792
|
"type": "string",
|
|
@@ -972,7 +984,7 @@
|
|
|
972
984
|
"seedance2-mini",
|
|
973
985
|
"seedance2-5"
|
|
974
986
|
],
|
|
975
|
-
"description": "Model selector for this video-to-video request. Usually omit: non-Seedance controls default to \"ltx25-v2v\"; use \"ltx23-v2v\" only for rollback. LTX 2.5 Fast, HQ, and Pro currently use the release-validated official Distilled workflow for canny, pose, depth, detailer, inpaint, and outpaint. Dev is not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. For controlMode=\"seedance-v2v\", Seedance quality is selected only by model: use \"seedance2-mini\" for faster/lower-cost drafts and use \"seedance2\" for full-quality Seedance or 1080p/4K. \"seedance2-5\" supports 480p/720p, 4-30s at 24 fps, native audio, and first/last-frame conditioning; keep \"seedance2\" for
|
|
987
|
+
"description": "Model selector for this video-to-video request. Usually omit: non-Seedance controls default to \"ltx25-v2v\"; use \"ltx23-v2v\" only for rollback. LTX 2.5 Fast, HQ, and Pro currently use the release-validated official Distilled workflow for canny, pose, depth, detailer, inpaint, and outpaint. Dev is not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. For controlMode=\"seedance-v2v\", Seedance quality is selected only by model: use \"seedance2-mini\" for faster/lower-cost drafts and use \"seedance2\" for full-quality Seedance or 1080p/4K. \"seedance2-5\" supports 480p/720p/1080p, 4-30s at 24 fps, native audio, and first/last-frame conditioning; keep \"seedance2\" for 4K."
|
|
976
988
|
},
|
|
977
989
|
"generateAudio": {
|
|
978
990
|
"type": "boolean",
|
|
@@ -980,7 +992,7 @@
|
|
|
980
992
|
},
|
|
981
993
|
"targetResolution": {
|
|
982
994
|
"type": "number",
|
|
983
|
-
"description": "Seedance V2V only. Short-side output resolution target in pixels. Use when the user asks for a bare named resolution such as \"480p\", \"720p\", \"1080p\", \"2160p\", or \"4K\" without exact dimensions. Seedance V2V full supports 4K; Seedance V2V Mini
|
|
995
|
+
"description": "Seedance V2V only. Short-side output resolution target in pixels. Use when the user asks for a bare named resolution such as \"480p\", \"720p\", \"1080p\", \"2160p\", or \"4K\" without exact dimensions. Seedance V2V full supports 4K; Seedance V2V Mini and Fast support 480p and 720p only; Seedance 2.5 also supports 1080p, so never set 4K for \"seedance2-5\". Preserve the source video shape instead of forcing landscape pixels."
|
|
984
996
|
},
|
|
985
997
|
"sourceImageIndex": {
|
|
986
998
|
"type": "number",
|
|
@@ -1024,6 +1036,18 @@
|
|
|
1024
1036
|
"description": "Number of video variations to generate (1-16). Default: 1.",
|
|
1025
1037
|
"minimum": 1,
|
|
1026
1038
|
"maximum": 16
|
|
1039
|
+
},
|
|
1040
|
+
"outputFormat": {
|
|
1041
|
+
"type": "string",
|
|
1042
|
+
"enum": [
|
|
1043
|
+
"mp4",
|
|
1044
|
+
"mov"
|
|
1045
|
+
],
|
|
1046
|
+
"description": "Video container. Defaults to mp4. MOV is supported only by Seedance 2.5; choose it when the user requests MOV for editing."
|
|
1047
|
+
},
|
|
1048
|
+
"returnLastFrame": {
|
|
1049
|
+
"type": "boolean",
|
|
1050
|
+
"description": "Seedance 2.5 only. Set true to export a separate image of the final frame alongside the video. The result includes lastFrameUrl, which can be used as the first-frame image for a subsequent clip. Defaults to false; this does not extend the video automatically."
|
|
1027
1051
|
}
|
|
1028
1052
|
},
|
|
1029
1053
|
"required": [
|
|
@@ -1208,13 +1232,13 @@
|
|
|
1208
1232
|
"type": "function",
|
|
1209
1233
|
"function": {
|
|
1210
1234
|
"name": "sound_to_video",
|
|
1211
|
-
"description": "Generate video synchronized to audio. Use when the user has uploaded an audio file (mp3, wav, m4a, flac) and the audio is the primary sync target, especially uploaded-audio-only workflows. Also use after generate_music (\"turn that song into a video\", \"make a music video from that\"). Auto-detects generated audio from generate_music if no audio file is uploaded. Seedance animate_photo/generate_video can also attach uploaded audio as a loose @Audio reference when an image or video reference anchors the request; use this tool instead when the soundtrack itself should drive the video. If the user provides a reference image, use ltx25-ia2v by default (ltx23-ia2v is rollback); for lip-sync with a face image, use wan-s2v; if no image, use ltx25-a2v by default (ltx23-a2v is rollback). If the user wants dialogue/audio WITHOUT pre-existing audio, use animate_photo instead (LTX 2.5 and LTX 2.3 generate audio natively). Note: Persona voice clips from resolve_personas are NOT used by this tool — for persona voice identity in video, use animate_photo or generate_video with videoModel=\"ltx23\" because LTX 2.5 has no compatible ID-LoRA. LONG AUDIO ON SEEDANCE: Seedance 2.0 and Mini cap each clip at 15s; Seedance 2.5 renders up to 30s in one call, so prefer seedance2-5 for 16-30s audio instead of splitting. When the user uploads audio longer than the per-clip cap of the selected model and Seedance is selected (seedance2, seedance2-mini, or seedance2-5), do NOT clamp to 15s and drop the rest — split the run into multiple sound_to_video calls in the same turn (one per 15s segment, so a 20s audio becomes two clips: audioStart=0 duration=15, then audioStart=15 duration=5) and finish with a single stitch_video call referencing the resulting clip indices in order with audioIndex pointing at the same uploaded audio so the stitched output carries the full original soundtrack. LTX/WAN models accept up to 20s per clip, so single-call is fine for them. Use videoModel=\"wan3.0-video\" when the user explicitly requests Wan 3 audio-driven video. Use videoModel=\"wan3.0-spicy-video\" for Wan 3.0 Enhanced audio-driven video through MuleRouter provider ID w3.0-video; it supports adaptive ratios and provider prompt expansion.",
|
|
1235
|
+
"description": "Generate video synchronized to audio. Use when the user has uploaded an audio file (mp3, wav, m4a, flac) and the audio is the primary sync target, especially uploaded-audio-only workflows. Also use after generate_music (\"turn that song into a video\", \"make a music video from that\"). Auto-detects generated audio from generate_music if no audio file is uploaded. Seedance animate_photo/generate_video can also attach uploaded audio as a loose @Audio reference when an image or video reference anchors the request; use this tool instead when the soundtrack itself should drive the video. If the user provides a reference image, use ltx25-ia2v by default (ltx23-ia2v is rollback); for lip-sync with a face image, use wan-s2v; if no image, use ltx25-a2v by default (ltx23-a2v is rollback). If the user wants dialogue/audio WITHOUT pre-existing audio, use animate_photo instead (LTX 2.5 and LTX 2.3 generate audio natively). Note: Persona voice clips from resolve_personas are NOT used by this tool — for persona voice identity in video, use animate_photo or generate_video with videoModel=\"ltx23\" because LTX 2.5 has no compatible ID-LoRA. LONG AUDIO ON SEEDANCE: Seedance 2.0 and Mini cap each clip at 15s; Seedance 2.5 renders up to 30s in one call, so prefer seedance2-5 for 16-30s audio instead of splitting. When the user uploads audio longer than the per-clip cap of the selected model and Seedance is selected (seedance2, seedance2-mini, or seedance2-5), do NOT clamp to 15s and drop the rest — split the run into multiple sound_to_video calls in the same turn (one per 15s segment, so a 20s audio becomes two clips: audioStart=0 duration=15, then audioStart=15 duration=5) and finish with a single stitch_video call referencing the resulting clip indices in order with audioIndex pointing at the same uploaded audio so the stitched output carries the full original soundtrack. LTX/WAN models accept up to 20s per clip, so single-call is fine for them. Use videoModel=\"wan3.0-video\" when the user explicitly requests Wan 3 audio-driven video. Use videoModel=\"wan3.0-spicy-video\" for Wan 3.0 Enhanced audio-driven video through MuleRouter provider ID w3.0-video; it supports adaptive ratios and provider prompt expansion. MINIMAX H3 AUDIO: only when the user asks for MiniMax H3 or FastH3 audio-driven video, use videoModel=\"minimax-h3-fasth3-ia2v-turbo\" with a first-frame image (sourceImageIndex), \"minimax-h3-fasth3-flfa2v-turbo\" with a first and a last frame (sourceImageIndex and endImageIndex), or \"minimax-h3-fasth3-a2v-turbo\" with no image; add \"-2stage\" only for 1080p, 1440p or 2K H3 output. Without an explicit MiniMax H3 or FastH3 request keep the LTX 2.5 defaults. The MiniMax H3 selectors on generate_video and animate_photo cannot take an uploaded audio track; send uploaded-audio H3 requests here.",
|
|
1212
1236
|
"parameters": {
|
|
1213
1237
|
"type": "object",
|
|
1214
1238
|
"properties": {
|
|
1215
1239
|
"prompt": {
|
|
1216
1240
|
"type": "string",
|
|
1217
|
-
"description": "Describe the video like a cinematographer. Let the audio define timing — use the prompt for visual interpretation. One flowing paragraph, present tense, specific natural language.\n\nLITERAL PROMPT OVERRIDE: If the user explicitly says not to modify the prompt, or to use it exactly/verbatim/as-is, copy the identified prompt text verbatim instead of applying these construction rules unless a hard requirement is missing. For Seedance or Wan 3, set expandPrompt=false.\n\nSTRUCTURE: shot/style and scale → subject → environment, lighting, color, texture, atmosphere → visual action synced to audio → camera movement. For LTX 2.3 image+audio mode, do not re-describe static details already visible in the reference image; focus on motion, action, camera, and how the image responds to the audio.\n\nMOTION PACING: Scale complexity to duration. <=6s: 1 main visual beat + 1 simple camera move. Around 10s: 2-3 clear beats + 1 camera move. >10s: up to 4 beats in clear sequence. Let the audio define timing, but avoid stacking subject, camera, and environment motion in short clips.\n\nBLOCKING: Direct layout when it affects the shot: left/right placement, foreground/background, facing direction, and relative distance between subjects.\n\nLIP-SYNC: Shot framing, speaker's appearance and setting, physical performance synced to audio — gestures, expressions, jaw movement between phrases. Include acting beats.\n\nMUSIC VISUALIZATION: Visual style, environment, and how elements react to rhythm and energy.\n\nAUDIO-REACTIVE: Motion and visual changes that correspond to sounds in the track.\n\nLTX VOCABULARY: camera (tracking, dolly, pan, tilt, handheld, static frame), lighting/atmosphere (golden hour, neon glow, dramatic shadows, fog, rain, smoke, reflections), scale/pacing (expansive, epic, intimate, claustrophobic, slow motion, time-lapse, lingering shot, continuous shot), style/genre (film noir, painterly, cyberpunk, stop-motion, claymation, 2D/3D animation, hand-drawn, fantasy, thriller, experimental film).\n\nAVOID: Vague prompts, too many competing visual elements, abstract descriptions without visible behavior, rigid numeric constraints, readable text or logos. QUOTING RULE: ONLY use double quotes for spoken dialogue. Never quote on-screen text, overlay text, titles, captions, signs, or any visual text — describe them without quotes.\n\nNON-SEEDANCE POSITIVE CONSTRAINTS: For ltx25-ia2v, ltx25-a2v, ltx23-ia2v, ltx23-a2v, and wan-s2v, prompt is a positive prompt. Translate user avoid/no/don't constraints into affirmative production constraints instead of copying negative phrasing. Preserve exact quoted visible text or dialogue when the user explicitly requests it; keep surrounding surfaces blank.\n\nBATCH VARIATIONS: When numberOfVariations > 1, use Dynamic Prompt syntax to vary the visual interpretation while keeping audio sync intent consistent. This is one Sogni project with multiple jobs, so prefer it when all outputs share the same audio source/window, image source, model, duration, dimensions, and parameters and only prompt text varies. Example: \"{abstract neon visualization|nature scene with swaying trees|urban street with rain} synced to the beat\"."
|
|
1241
|
+
"description": "Describe the video like a cinematographer. Let the audio define timing — use the prompt for visual interpretation. One flowing paragraph, present tense, specific natural language.\n\nLITERAL PROMPT OVERRIDE: If the user explicitly says not to modify the prompt, or to use it exactly/verbatim/as-is, copy the identified prompt text verbatim instead of applying these construction rules unless a hard requirement is missing. For Seedance or Wan 3, set expandPrompt=false.\n\nSTRUCTURE: shot/style and scale → subject → environment, lighting, color, texture, atmosphere → visual action synced to audio → camera movement. For LTX 2.3 image+audio mode, do not re-describe static details already visible in the reference image; focus on motion, action, camera, and how the image responds to the audio.\n\nMOTION PACING: Scale complexity to duration. <=6s: 1 main visual beat + 1 simple camera move. Around 10s: 2-3 clear beats + 1 camera move. >10s: up to 4 beats in clear sequence. Let the audio define timing, but avoid stacking subject, camera, and environment motion in short clips.\n\nBLOCKING: Direct layout when it affects the shot: left/right placement, foreground/background, facing direction, and relative distance between subjects.\n\nLIP-SYNC: Shot framing, speaker's appearance and setting, physical performance synced to audio — gestures, expressions, jaw movement between phrases. Include acting beats.\n\nMUSIC VISUALIZATION: Visual style, environment, and how elements react to rhythm and energy.\n\nAUDIO-REACTIVE: Motion and visual changes that correspond to sounds in the track.\n\nLTX VOCABULARY: camera (tracking, dolly, pan, tilt, handheld, static frame), lighting/atmosphere (golden hour, neon glow, dramatic shadows, fog, rain, smoke, reflections), scale/pacing (expansive, epic, intimate, claustrophobic, slow motion, time-lapse, lingering shot, continuous shot), style/genre (film noir, painterly, cyberpunk, stop-motion, claymation, 2D/3D animation, hand-drawn, fantasy, thriller, experimental film).\n\nAVOID: Vague prompts, too many competing visual elements, abstract descriptions without visible behavior, rigid numeric constraints, readable text or logos. QUOTING RULE: ONLY use double quotes for spoken dialogue. Never quote on-screen text, overlay text, titles, captions, signs, or any visual text — describe them without quotes.\n\nNON-SEEDANCE POSITIVE CONSTRAINTS: For ltx25-ia2v, ltx25-a2v, ltx23-ia2v, ltx23-a2v, and wan-s2v, prompt is a positive prompt. Translate user avoid/no/don't constraints into affirmative production constraints instead of copying negative phrasing. Preserve exact quoted visible text or dialogue when the user explicitly requests it; keep surrounding surfaces blank.\n\nBATCH VARIATIONS: When numberOfVariations > 1, use Dynamic Prompt syntax to vary the visual interpretation while keeping audio sync intent consistent. This is one Sogni project with multiple jobs, so prefer it when all outputs share the same audio source/window, image source, model, duration, dimensions, and parameters and only prompt text varies. Example: \"{abstract neon visualization|nature scene with swaying trees|urban street with rain} synced to the beat\".\n\nMINIMAX H3 AUDIO SELECTORS: write the request plainly (subject, action, camera, the voice or sound heard in the upload and any exact words spoken in it); the MiniMax H3 prompt shaper turns it into the H3 contract for the matching image-to-video, first-and-last-frame or text-to-video mode. Describe the uploaded audio as it is, and do not invent other dialogue or music."
|
|
1218
1242
|
},
|
|
1219
1243
|
"expandPrompt": {
|
|
1220
1244
|
"type": "boolean",
|
|
@@ -1222,7 +1246,7 @@
|
|
|
1222
1246
|
},
|
|
1223
1247
|
"negativePrompt": {
|
|
1224
1248
|
"type": "string",
|
|
1225
|
-
"description": "Advanced LTX 2.5/LTX 2.3/WAN only. The LTX A2V and IA2V workflows accept this separate negative prompt. Use it only when the user explicitly asks to set one. Do not set for Seedance.\n\nWan 3 has no negativePrompt request field; do not set this for wan3.0-video."
|
|
1249
|
+
"description": "Advanced LTX 2.5/LTX 2.3/WAN only. The LTX A2V and IA2V workflows accept this separate negative prompt. Use it only when the user explicitly asks to set one. Do not set for Seedance.\n\nWan 3 has no negativePrompt request field; do not set this for wan3.0-video.\n\nMiniMax H3 has no negative-prompt input; do not set this for the MiniMax H3 FastH3 audio selectors and state exclusions positively in prompt."
|
|
1226
1250
|
},
|
|
1227
1251
|
"audioSourceIndex": {
|
|
1228
1252
|
"type": "number",
|
|
@@ -1230,16 +1254,20 @@
|
|
|
1230
1254
|
},
|
|
1231
1255
|
"sourceImageIndex": {
|
|
1232
1256
|
"type": "number",
|
|
1233
|
-
"description": "Optional index of an uploaded image to use as the starting frame (0-based). Required for lip-sync models (WAN S2V). For audio-only-to-video models (LTX 2.3 A2V), this is optional — omit it to generate video purely from text + audio."
|
|
1257
|
+
"description": "Optional index of an uploaded image to use as the starting frame (0-based). Required for lip-sync models (WAN S2V). For audio-only-to-video models (LTX 2.3 A2V), this is optional — omit it to generate video purely from text + audio. MiniMax H3 FastH3 audio guide: \"minimax-h3-fasth3-ia2v-turbo\" and \"minimax-h3-fasth3-flfa2v-turbo\" (and their -2stage forms) require this first frame; \"minimax-h3-fasth3-a2v-turbo\" and its -2stage form take no image, so omit it for them."
|
|
1258
|
+
},
|
|
1259
|
+
"endImageIndex": {
|
|
1260
|
+
"type": "number",
|
|
1261
|
+
"description": "Which image to use as the LAST frame, indexed like animate_photo's endImageIndex: negative indices for uploaded images (-1 = first upload, -2 = second upload) and 0-based non-negative indices for generated results. Only \"minimax-h3-fasth3-flfa2v-turbo\" and \"minimax-h3-fasth3-flfa2v-turbo-2stage\" take it, and they require it together with sourceImageIndex (for two uploaded images use sourceImageIndex=-1 and endImageIndex=-2). Omit it for every other videoModel."
|
|
1234
1262
|
},
|
|
1235
1263
|
"audioStart": {
|
|
1236
1264
|
"type": "number",
|
|
1237
|
-
"description": "Start offset in seconds into the audio track. Use when the user says \"start 20 seconds in\", \"skip the intro\", \"use the chorus at 1:30\", etc. Default: 0 (beginning of audio). The video will be synced to the audio starting from this point.",
|
|
1265
|
+
"description": "Start offset in seconds into the audio track. Use when the user says \"start 20 seconds in\", \"skip the intro\", \"use the chorus at 1:30\", etc. Default: 0 (beginning of audio). The video will be synced to the audio starting from this point. The MiniMax H3 FastH3 audio selectors take audioStart too: the clip uses the upload from audioStart for its own length.",
|
|
1238
1266
|
"minimum": 0
|
|
1239
1267
|
},
|
|
1240
1268
|
"duration": {
|
|
1241
1269
|
"type": "number",
|
|
1242
|
-
"description": "Video duration in seconds. Default: 5. Per-model range: LTX/WAN 2.2 = 2-20s; Wan 3 = 2-30s; Seedance 2.0 and Mini = 4-15s; Seedance 2.5 = 4-30s. For music videos, use the maximum duration the selected model allows because the audio is usually longer than the video limit. Use when the user explicitly requests a specific length.",
|
|
1270
|
+
"description": "Video duration in seconds. Default: 5. Per-model range: LTX/WAN 2.2 = 2-20s; Wan 3 = 2-30s; Seedance 2.0 and Mini = 4-15s; Seedance 2.5 = 4-30s. For music videos, use the maximum duration the selected model allows because the audio is usually longer than the video limit. Use when the user explicitly requests a specific length. MiniMax H3 FastH3 audio selectors render 124-362 frames on a 17-frame grid at a fixed 24 fps, so the clip runs about 5.2-15.1 seconds and a length outside that window snaps to it; for longer audio pick the window with audioStart.",
|
|
1243
1271
|
"minimum": 2,
|
|
1244
1272
|
"maximum": 30
|
|
1245
1273
|
},
|
|
@@ -1279,13 +1307,19 @@
|
|
|
1279
1307
|
"ltx23-ia2v",
|
|
1280
1308
|
"ltx23-a2v",
|
|
1281
1309
|
"wan3.0-video",
|
|
1282
|
-
"wan3.0-spicy-video"
|
|
1310
|
+
"wan3.0-spicy-video",
|
|
1311
|
+
"minimax-h3-fasth3-ia2v-turbo",
|
|
1312
|
+
"minimax-h3-fasth3-ia2v-turbo-2stage",
|
|
1313
|
+
"minimax-h3-fasth3-flfa2v-turbo",
|
|
1314
|
+
"minimax-h3-fasth3-flfa2v-turbo-2stage",
|
|
1315
|
+
"minimax-h3-fasth3-a2v-turbo",
|
|
1316
|
+
"minimax-h3-fasth3-a2v-turbo-2stage"
|
|
1283
1317
|
],
|
|
1284
|
-
"description": "\"ltx25-ia2v\" (default with image) and \"ltx25-a2v\" (default without image): LTX 2.5 image+audio and audio-only modes; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflows. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates official ComfyUI Dev recipes. Video model. \"ltx23-ia2v\" (rollback with image): LTX 2.3 image+audio to video, audio-reactive with a reference image; Fast/HQ use the distilled 8-step worker and Default Media Quality Pro uses the non-distilled dev worker. \"ltx23-a2v\" (rollback without image): LTX 2.3 audio-only to video, no image needed, creates video purely from text prompt + audio with the same quality-tier routing. \"wan-s2v\": WAN 2.2 sound-to-video, best for lip-sync with a face image, fast 4-step. \"seedance2\": full Seedance 2.0 audio-reference video, 4-15s. \"seedance2-mini\": Seedance 2.0 Mini, 720p cap, fastest/lower-cost Seedance option. Seedance quality is selected only by this model value: pick \"seedance2-mini\" for faster/lower-cost drafts or explicit Mini requests, and pick \"seedance2\" for full-quality Seedance or 1080p/4K. Do not infer the Seedance model from Default Media Quality Fast/HQ/Pro. \"seedance2-5\": Seedance 2.5, the newest Seedance generation — 480p and
|
|
1318
|
+
"description": "\"ltx25-ia2v\" (default with image) and \"ltx25-a2v\" (default without image): LTX 2.5 image+audio and audio-only modes; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflows. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates official ComfyUI Dev recipes. Video model. \"ltx23-ia2v\" (rollback with image): LTX 2.3 image+audio to video, audio-reactive with a reference image; Fast/HQ use the distilled 8-step worker and Default Media Quality Pro uses the non-distilled dev worker. \"ltx23-a2v\" (rollback without image): LTX 2.3 audio-only to video, no image needed, creates video purely from text prompt + audio with the same quality-tier routing. \"wan-s2v\": WAN 2.2 sound-to-video, best for lip-sync with a face image, fast 4-step. \"seedance2\": full Seedance 2.0 audio-reference video, 4-15s. \"seedance2-mini\": Seedance 2.0 Mini, 720p cap, fastest/lower-cost Seedance option. Seedance quality is selected only by this model value: pick \"seedance2-mini\" for faster/lower-cost drafts or explicit Mini requests, and pick \"seedance2\" for full-quality Seedance or 1080p/4K. Do not infer the Seedance model from Default Media Quality Fast/HQ/Pro. \"seedance2-5\": Seedance 2.5, the newest Seedance generation — 480p, 720p, and 1080p (4K is unsupported), 4-30s per clip at a fixed 24 fps, native audio, first-and-last-frame conditioning, and a much larger reference budget than the 2.0 family: up to 30 images, 10 videos, and 10 audios, with up to 50 reference media files total, subject to those per-modality caps. Choose \"seedance2-5\" when the user asks for Seedance 2.5, wants a single continuous Seedance clip longer than 15s (2.5 renders up to 30s in one call instead of being split and stitched), or wants a first-and-last-frame Seedance transition. Keep \"seedance2\" for 4K requests; Seedance 2.5 supports up to 1080p. For Seedance audio-reference prompts, preserve exact spoken dialogue when the user supplied it, and assign @Image1/@Audio1 roles. If the user asks for speech without words, describe the vocal performance without inventing quoted dialogue. Treat lip-sync, voice cloning, and real-human reference behavior as provider-sensitive rather than guaranteed. Omit to auto-select based on whether an image is present. \"wan3.0-video\" is Alibaba Wan 3 and \"wan3.0-spicy-video\" is MuleRouter w3.0-video. Both render 2-30s at fixed 30 fps with optional native audio, provider prompt expansion, 480p/720p/1080p, adaptive/fixed ratios, first/last frames, and up to 10 image/5 video/5 audio references. Only Alibaba wan3.0-video accepts document/web context and watermark. Frame anchors and loose references are mutually exclusive. Do not send negativePrompt; video references are loose conditioning for a new result, not source-video editing or extension. MiniMax H3 FastH3 audio guide: \"minimax-h3-fasth3-ia2v-turbo\" (first-frame image from sourceImageIndex plus the audio), \"minimax-h3-fasth3-flfa2v-turbo\" (first frame from sourceImageIndex, last frame from endImageIndex, plus the audio) and \"minimax-h3-fasth3-a2v-turbo\" (audio only; set neither sourceImageIndex nor endImageIndex) run the four-step FastH3 engine with the uploaded audio driving the picture from frame 0, and the output keeps that audio as its soundtrack. Choose them only when the user asks for MiniMax H3 or FastH3; never pick them in place of the LTX 2.5 defaults. They render 124-362 frames on the H3 17-frame grid at a fixed 24 fps (about 5.2-15.1 seconds) on a 32px grid within 1344x768, so use targetResolution 768 or omit it. audioStart picks the window of the upload. generateAudio=false, LoRAs and negativePrompt are not supported. Each costs the FastH3 price of its image-to-video, first-and-last-frame or text-to-video mode, 4 Spark per second. \"minimax-h3-fasth3-ia2v-turbo-2stage\", \"minimax-h3-fasth3-flfa2v-turbo-2stage\" and \"minimax-h3-fasth3-a2v-turbo-2stage\" are the two-stage forms: the same inputs rendered on the FastH3 canvas, then enlarged 2x and refined, delivered at twice the canvas with the same length and audio. For them targetResolution names the delivered class: 1080 renders a 544px short-edge canvas (960x544 is delivered at 1920x1088) for 10 Spark per second, and 1440 or omitted renders the 768p canvas for 2K (1344x768 is delivered at 2688x1536) for 16 Spark per second. The audio two-stage selectors have no 720p price class (720 would render a 384px canvas at the 1080p rate), so for 720p or 768p audio-guided H3 output use the regular selector. The estimate prices every request."
|
|
1285
1319
|
},
|
|
1286
1320
|
"generateAudio": {
|
|
1287
1321
|
"type": "boolean",
|
|
1288
|
-
"description": "Whether the returned video should include audio. Omit to include audio by default; set false when the user asks for silent output or no audio. The reference audio is still required and still drives generation even when the returned video has no audio track."
|
|
1322
|
+
"description": "Whether the returned video should include audio. Omit to include audio by default; set false when the user asks for silent output or no audio. The reference audio is still required and still drives generation even when the returned video has no audio track. MiniMax H3 FastH3 audio selectors always deliver the uploaded audio: omit generateAudio for them (false is refused)."
|
|
1289
1323
|
},
|
|
1290
1324
|
"numberOfVariations": {
|
|
1291
1325
|
"type": "number",
|
|
@@ -1300,6 +1334,18 @@
|
|
|
1300
1334
|
"aspectRatio": {
|
|
1301
1335
|
"type": "string",
|
|
1302
1336
|
"description": "Do NOT set unless the user explicitly requests an aspect ratio, format, orientation, or exact pixel dimensions. When a reference/source image is used and the user did not ask to change its shape, omit this field so the handler preserves the selected source image's own ratio.\n\nFormats: \"16:9\", \"9:16\", \"4:5\", \"1:1\", \"4:3\", \"3:2\", \"21:9\", or exact pixels like \"1920x1080\".\n\nCRITICAL: When the user specifies exact pixel dimensions (e.g., \"1280x720\", \"1080x1920\", \"1920x1080\", \"3840x2160\") or an orientation-qualified named resolution (e.g., \"720p landscape\", \"720p portrait\"), use the exact pixel format, NOT a ratio like \"16:9\" or \"9:16\". Exact user-requested dimensions override the selected default media quality, including Pro/HQ defaults. A bare named video resolution like \"720p resolution\" is only a resolution tier/short-side request; do not turn it into landscape pixels and do not set aspectRatio unless the user also states landscape, portrait, vertical, horizontal, or exact pixels. If requested pixels are in bounds but not on the model's pixel step, still pass the user's exact pixel request; the handler snaps to the nearest supported size internally. Only use ratio format when the user says a generic format name without pixel dimensions.\n\nMappings (use ONLY when user does NOT specify pixel dimensions): landscape/widescreen/YouTube/cinematic → \"16:9\". portrait → \"9:16\". TikTok/Reels/IG Reels → \"1080x1920\". ultrawide/cinema scope → \"21:9\". Instagram post → \"4:5\". square → \"1:1\". standard/TV → \"4:3\". 720p landscape → \"1280x720\". 720p portrait → \"720x1280\". 1080p landscape → \"1920x1080\". 1080p portrait/HD portrait → \"1080x1920\". 4K landscape → \"3840x2160\". 4K portrait → \"2160x3840\". Never set for generic requests like \"make a video\"."
|
|
1337
|
+
},
|
|
1338
|
+
"outputFormat": {
|
|
1339
|
+
"type": "string",
|
|
1340
|
+
"enum": [
|
|
1341
|
+
"mp4",
|
|
1342
|
+
"mov"
|
|
1343
|
+
],
|
|
1344
|
+
"description": "Video container. Defaults to mp4. MOV is supported only by Seedance 2.5; choose it when the user requests MOV for editing."
|
|
1345
|
+
},
|
|
1346
|
+
"returnLastFrame": {
|
|
1347
|
+
"type": "boolean",
|
|
1348
|
+
"description": "Seedance 2.5 only. Set true to export a separate image of the final frame alongside the video. The result includes lastFrameUrl, which can be used as the first-frame image for a subsequent clip. Defaults to false; this does not extend the video automatically."
|
|
1303
1349
|
}
|
|
1304
1350
|
},
|
|
1305
1351
|
"required": [
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@sogni-ai/sogni-protocol",
|
|
3
|
-
"version": "1.0.0-alpha.
|
|
3
|
+
"version": "1.0.0-alpha.42",
|
|
4
4
|
"description": "Language-neutral protocol artifacts for the Sogni ecosystem: tool schemas, prompts, OpenAI tool manifests, and enums. Consumed by every Sogni SDK (TypeScript, Swift, and future Python/Kotlin/Rust SDKs) so contracts stay in lockstep across languages.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"sogni",
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
"contractId": "animate_photo_v1",
|
|
3
3
|
"version": "1.2.0",
|
|
4
4
|
"toolName": "animate_photo",
|
|
5
|
-
"baseDescription": "animate_photo produces video from one or more source images using LTX 2.5 by default, with LTX 2.3 retained as a rollback path.\n\nVIDEO PROMPT QUOTING: In video prompts, ONLY use double quotes for spoken dialogue.\nSpeaker tags are allowed outside the quotes for screenplay-style dialogue, e.g.\nCHARACTER: \"We made it.\" Never put on-screen text, overlay text, titles, captions, signs,\nwatermarks, or any visual text in quotes — describe them without quotes (e.g. bold white text\nreading CONGRATULATIONS overlays the lower third). Quotes signal speech to the model;\nquoting non-speech text confuses audio generation.\n\nDIALOGUE DURATION: Spoken dialogue in video prompts must fit the clip duration. Estimate\nat 2.5 words per second for natural cinematic delivery, plus ~1 second per acting beat\n(pauses, gestures, glances between lines). If the user did NOT explicitly request a specific\nduration (using default 5s), extend the duration to fit the dialogue (max 20s). If the user\nexplicitly requested a specific duration, condense the dialogue to fit while preserving meaning.\nAlways check: total dialogue words ÷ 2.5 + beat count ≤ clip duration.\n\nLATEST GENERATED IMAGE FOLLOW-UP: When the newest user turn asks to animate, make a video,\nor make a clip from a generated image/result (for example \"the apple\", \"this one\",\n\"the latest image\"), use animate_photo with that latest generated image. Do not inherit an\nolder Seedance model, resolution, or duration from an unrelated prior turn unless the newest\nuser turn explicitly says Seedance or confirms an immediately suggested Seedance video stage.\nLTX supports exact 2-20s durations, so honor requests like 3s exactly.\n\nWORD BUDGET PER CLIP: The handler REJECTS clips whose spoken dialogue exceeds the budget\n— there is NO auto-trim, so plan dialogue lengths up-front. Hard maximum is 3.75 spoken\nwords per second. Ceilings: 5s = 18 words, 6s = 22 words, 8s = 30 words, 10s = 37 words,\n15s = 56 words, 20s = 75 words. Aim below these ceilings. If a scene's dialogue won't fit,\ntighten the lines, raise the per-clip duration, or split into two segments — do NOT submit\nand hope it works. Spoken words inside double quotes count toward the budget; speaker tags\nand visual/action prose are free.\n\nBATCH VIDEO PER-CLIP DURATION: For a multi-segment animate_photo batch\n(sourceImageIndices + prompts) when the user states a TOTAL video length but NO per-clip\nlength, target 15 seconds per clip when dialogue is involved, and pass that duration\nexplicitly. Example: 60s total → 4 segments × 15s, NOT 6×10s or 12×5s. There is NO 3-clip\nbatch cap: sourceImageIndices supports up to 16 clips, so never split one planned batch into\n\"first 3\" and \"remaining clips\" calls. Do NOT split a planned 15s dialogue scene into multiple\nshorter clips just because a retry complains about word budget; keep duration=15 and tighten\nthe line. Use 5s clips only for single short motion beats or one very short spoken phrase.\nIf the user explicitly specifies a per-clip duration, honor that instead.\n\nN-VERSIONS-OF-A-VIDEO PATTERN: NEVER call animate_photo N times sequentially — ALWAYS\nuse sourceImageIndices in ONE call so all N projects run in parallel. Two flavors:\n(A) SHARED CONTENT — one edit_image/generate_image call with numberOfVariations=N + {|}\nDynamic Prompts to make N distinct source images, then ONE animate_photo call with\nsourceImageIndices=[start..start+N-1] and a single shared prompt.\n(B) PER-CLIP CONTENT — when each clip has DIFFERENT dialogue, jokes, narration, or motion,\npass BOTH sourceImageIndices AND prompts (array of N strings, one per clip) in the SAME\nsingle animate_photo call. The top-level prompt is still required — pass a brief batch summary.\n\nCRITICAL: sourceImageIndices values MUST be read from the latest edit_image/generate_image\ntool result's startIndex field — if startIndex=3 and 4 images were generated, pass\nsourceImageIndices=[3,4,5,6], NOT [0,1,2,3]. Negative indices refer to uploaded images:\n-1 first upload, -2 second upload, -3 third upload. Use repeated -1 entries only when\nintentionally reusing the primary uploaded image. When prompts is supplied, prompts.length\nMUST equal sourceImageIndices.length.\n\nSEEDANCE UPLOADED STORYBOARD DEFAULT: If the user uploaded a storyboard, shot sheet,\nor visual trailer board and asks to make a trailer/video/movie/clip from it, do NOT use\nanimate_photo on the board image and do NOT split it into four LTX clips. Use generate_video\nwith Seedance referenceImageIndices for one continuous clip unless the user explicitly asks\nfor separate LTX clips or first-frame/last-frame animation.\n\nSCREENPLAY / STORYBOARD ANIMATE RULE: For full storyboard projects, use one\nanimate_photo batch with sourceImageIndices + prompts so each clip keeps its own exact\nscene text, stable cast anchors, and screenplay-style speaker-tagged dialogue, and all video\nclips render in parallel. Every speaking clip's video prompt must include that clip's actual\nquoted dialogue, not placeholders such as \"while speaking\", \"dialogue begins\", \"explaining\",\nor \"final line lands\". If each generated scene keyframe should be both the first and last frame\nof its own stitched segment, call animate_photo with sourceImageIndices=[start..end],\nframeRole=\"both\", prompts=[...], and OMIT endImageIndex/endImageIndices so the handler\nuses each source as its own end frame.\n\nUPLOADED REFERENCE LOOPED SKITS: When the user supplies one uploaded reference image and\nasks for several scripted/storyboard/dialogue segments to reuse that same image as BOTH the\nfirst frame and last frame of each segment before stitching, do it in ONE animate_photo call:\nsourceImageIndices=[-1,-1,...], frameRole=\"both\", endImageIndex=-1 (or matching\nendImageIndices=[-1,-1,...]), duration equal to the requested per-segment duration, and\nprompts=[one full scene prompt per segment]. Each prompt must preserve the exact screenplay\nspeaker tags and quoted dialogue from that scene, e.g. HOST: \"...\" GUEST: \"...\". Do not\ndrop speaker tags, convert them to generic narration, omit the last-frame contract, analyze\nthe image first, generate new keyframes first, or split the batch into serial calls. After\nthe single animate_photo batch completes, call stitch_video with the returned video indices.\n\nFor adjacent transition chains: N images create N-1 clips — call animate_photo with\nframeRole=\"both\", sourceImageIndices=[start..end-1], endImageIndices=[start+1..end],\nprompts=[one transition prompt per adjacent pair], then stitch_video. If 5 uploaded images\nare the keyframe sequence, use sourceImageIndices=[-1,-2,-3,-4],\nendImageIndices=[-2,-3,-4,-5], frameRole=\"both\", prompts length 4, then stitch_video.\nDo NOT set endImageIndex=-1 in generated-keyframe patterns — that means every clip ends\non the primary uploaded image.\n\nUPLOADED FIRST-FRAME/LAST-FRAME TRANSITION CHAINS: If the user uploads multiple images\nand asks for a video that transitions from image to image, changes country/version every\nN seconds, or says to use first-frame/last-frame for each pair, call animate_photo directly.\nDo not call edit_image, generate_image, analyze_image, or map_assets_for_model first — the\nuploaded images are already the keyframes. For N uploaded images, create N-1 adjacent clips\nunless the user explicitly asks for a loop back to the first image. Use per-clip duration\nfrom \"every N seconds\" when present; otherwise divide the requested total by the number of\nadjacent clips. After animate_photo returns the batch videos, always call stitch_video with\nthose video indices before finalizing.\n\nMINIMAX H3 TWO-STAGE OUTPUT: 1080p and 2K MiniMax H3 delivery is its own selector, not an option. \"minimax-h3-fasth3-i2v-turbo-2stage\" (first frame) and \"minimax-h3-fasth3-flf2v-turbo-2stage\" (first and last frame, frameRole=\"both\" with an end frame) are the two-stage FastH3 engine: FastH3 renders the canvas, then the worker enlarges it 2x and refines it, so the clip is delivered at twice the canvas width and height with the same length and audio. targetResolution names the delivered class: 1080 renders a 544px short-edge canvas (960x544 becomes 1920x1088) for
|
|
5
|
+
"baseDescription": "animate_photo produces video from one or more source images using LTX 2.5 by default, with LTX 2.3 retained as a rollback path.\n\nVIDEO PROMPT QUOTING: In video prompts, ONLY use double quotes for spoken dialogue.\nSpeaker tags are allowed outside the quotes for screenplay-style dialogue, e.g.\nCHARACTER: \"We made it.\" Never put on-screen text, overlay text, titles, captions, signs,\nwatermarks, or any visual text in quotes — describe them without quotes (e.g. bold white text\nreading CONGRATULATIONS overlays the lower third). Quotes signal speech to the model;\nquoting non-speech text confuses audio generation.\n\nDIALOGUE DURATION: Spoken dialogue in video prompts must fit the clip duration. Estimate\nat 2.5 words per second for natural cinematic delivery, plus ~1 second per acting beat\n(pauses, gestures, glances between lines). If the user did NOT explicitly request a specific\nduration (using default 5s), extend the duration to fit the dialogue (max 20s). If the user\nexplicitly requested a specific duration, condense the dialogue to fit while preserving meaning.\nAlways check: total dialogue words ÷ 2.5 + beat count ≤ clip duration.\n\nLATEST GENERATED IMAGE FOLLOW-UP: When the newest user turn asks to animate, make a video,\nor make a clip from a generated image/result (for example \"the apple\", \"this one\",\n\"the latest image\"), use animate_photo with that latest generated image. Do not inherit an\nolder Seedance model, resolution, or duration from an unrelated prior turn unless the newest\nuser turn explicitly says Seedance or confirms an immediately suggested Seedance video stage.\nLTX supports exact 2-20s durations, so honor requests like 3s exactly.\n\nWORD BUDGET PER CLIP: The handler REJECTS clips whose spoken dialogue exceeds the budget\n— there is NO auto-trim, so plan dialogue lengths up-front. Hard maximum is 3.75 spoken\nwords per second. Ceilings: 5s = 18 words, 6s = 22 words, 8s = 30 words, 10s = 37 words,\n15s = 56 words, 20s = 75 words. Aim below these ceilings. If a scene's dialogue won't fit,\ntighten the lines, raise the per-clip duration, or split into two segments — do NOT submit\nand hope it works. Spoken words inside double quotes count toward the budget; speaker tags\nand visual/action prose are free.\n\nBATCH VIDEO PER-CLIP DURATION: For a multi-segment animate_photo batch\n(sourceImageIndices + prompts) when the user states a TOTAL video length but NO per-clip\nlength, target 15 seconds per clip when dialogue is involved, and pass that duration\nexplicitly. Example: 60s total → 4 segments × 15s, NOT 6×10s or 12×5s. There is NO 3-clip\nbatch cap: sourceImageIndices supports up to 16 clips, so never split one planned batch into\n\"first 3\" and \"remaining clips\" calls. Do NOT split a planned 15s dialogue scene into multiple\nshorter clips just because a retry complains about word budget; keep duration=15 and tighten\nthe line. Use 5s clips only for single short motion beats or one very short spoken phrase.\nIf the user explicitly specifies a per-clip duration, honor that instead.\n\nN-VERSIONS-OF-A-VIDEO PATTERN: NEVER call animate_photo N times sequentially — ALWAYS\nuse sourceImageIndices in ONE call so all N projects run in parallel. Two flavors:\n(A) SHARED CONTENT — one edit_image/generate_image call with numberOfVariations=N + {|}\nDynamic Prompts to make N distinct source images, then ONE animate_photo call with\nsourceImageIndices=[start..start+N-1] and a single shared prompt.\n(B) PER-CLIP CONTENT — when each clip has DIFFERENT dialogue, jokes, narration, or motion,\npass BOTH sourceImageIndices AND prompts (array of N strings, one per clip) in the SAME\nsingle animate_photo call. The top-level prompt is still required — pass a brief batch summary.\n\nCRITICAL: sourceImageIndices values MUST be read from the latest edit_image/generate_image\ntool result's startIndex field — if startIndex=3 and 4 images were generated, pass\nsourceImageIndices=[3,4,5,6], NOT [0,1,2,3]. Negative indices refer to uploaded images:\n-1 first upload, -2 second upload, -3 third upload. Use repeated -1 entries only when\nintentionally reusing the primary uploaded image. When prompts is supplied, prompts.length\nMUST equal sourceImageIndices.length.\n\nSEEDANCE UPLOADED STORYBOARD DEFAULT: If the user uploaded a storyboard, shot sheet,\nor visual trailer board and asks to make a trailer/video/movie/clip from it, do NOT use\nanimate_photo on the board image and do NOT split it into four LTX clips. Use generate_video\nwith Seedance referenceImageIndices for one continuous clip unless the user explicitly asks\nfor separate LTX clips or first-frame/last-frame animation.\n\nSCREENPLAY / STORYBOARD ANIMATE RULE: For full storyboard projects, use one\nanimate_photo batch with sourceImageIndices + prompts so each clip keeps its own exact\nscene text, stable cast anchors, and screenplay-style speaker-tagged dialogue, and all video\nclips render in parallel. Every speaking clip's video prompt must include that clip's actual\nquoted dialogue, not placeholders such as \"while speaking\", \"dialogue begins\", \"explaining\",\nor \"final line lands\". If each generated scene keyframe should be both the first and last frame\nof its own stitched segment, call animate_photo with sourceImageIndices=[start..end],\nframeRole=\"both\", prompts=[...], and OMIT endImageIndex/endImageIndices so the handler\nuses each source as its own end frame.\n\nUPLOADED REFERENCE LOOPED SKITS: When the user supplies one uploaded reference image and\nasks for several scripted/storyboard/dialogue segments to reuse that same image as BOTH the\nfirst frame and last frame of each segment before stitching, do it in ONE animate_photo call:\nsourceImageIndices=[-1,-1,...], frameRole=\"both\", endImageIndex=-1 (or matching\nendImageIndices=[-1,-1,...]), duration equal to the requested per-segment duration, and\nprompts=[one full scene prompt per segment]. Each prompt must preserve the exact screenplay\nspeaker tags and quoted dialogue from that scene, e.g. HOST: \"...\" GUEST: \"...\". Do not\ndrop speaker tags, convert them to generic narration, omit the last-frame contract, analyze\nthe image first, generate new keyframes first, or split the batch into serial calls. After\nthe single animate_photo batch completes, call stitch_video with the returned video indices.\n\nFor adjacent transition chains: N images create N-1 clips — call animate_photo with\nframeRole=\"both\", sourceImageIndices=[start..end-1], endImageIndices=[start+1..end],\nprompts=[one transition prompt per adjacent pair], then stitch_video. If 5 uploaded images\nare the keyframe sequence, use sourceImageIndices=[-1,-2,-3,-4],\nendImageIndices=[-2,-3,-4,-5], frameRole=\"both\", prompts length 4, then stitch_video.\nDo NOT set endImageIndex=-1 in generated-keyframe patterns — that means every clip ends\non the primary uploaded image.\n\nUPLOADED FIRST-FRAME/LAST-FRAME TRANSITION CHAINS: If the user uploads multiple images\nand asks for a video that transitions from image to image, changes country/version every\nN seconds, or says to use first-frame/last-frame for each pair, call animate_photo directly.\nDo not call edit_image, generate_image, analyze_image, or map_assets_for_model first — the\nuploaded images are already the keyframes. For N uploaded images, create N-1 adjacent clips\nunless the user explicitly asks for a loop back to the first image. Use per-clip duration\nfrom \"every N seconds\" when present; otherwise divide the requested total by the number of\nadjacent clips. After animate_photo returns the batch videos, always call stitch_video with\nthose video indices before finalizing.\n\nMINIMAX H3 TWO-STAGE OUTPUT: 1080p and 2K MiniMax H3 delivery is its own selector, not an option. \"minimax-h3-fasth3-i2v-turbo-2stage\" (first frame) and \"minimax-h3-fasth3-flf2v-turbo-2stage\" (first and last frame, frameRole=\"both\" with an end frame) are the two-stage FastH3 engine: FastH3 renders the canvas, then the worker enlarges it 2x and refines it, so the clip is delivered at twice the canvas width and height with the same length and audio. targetResolution names the delivered class: 1080 renders a 544px short-edge canvas (960x544 becomes 1920x1088) for 10 Spark per second, 1440 or omitted renders the 768p canvas for 2K (1344x768 becomes 2688x1536) for 16 Spark per second, and 720 renders a 384px canvas (672x384 becomes 1344x768) for the regular FastH3 price of 4 Spark per second. The estimate prices every request. The source aspect is kept (a portrait source at 1080 renders 544x960 and is delivered at 1088x1920). They take the same prompt contract, durations and LoRAs as their FastH3 selectors. Choose them when the user asks for 1080p, 1440p or 2K MiniMax H3 output, for two-stage output, or for the sharpest/best H3 quality; for ordinary 768p FastH3 output keep the regular FastH3 selector at targetResolution 768. Never set 4K for H3.",
|
|
6
6
|
"parameterDocs": {
|
|
7
7
|
"sourceImageIndices": "Batch source image indices. Read startIndex from prior generate_image/edit_image result. Negative = uploaded images (-1 = first upload).",
|
|
8
8
|
"prompts": "Per-clip prompt array. Length MUST equal sourceImageIndices.length when both are set.",
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
"contractId": "generate_video_v1",
|
|
3
3
|
"version": "1.3.0",
|
|
4
4
|
"toolName": "generate_video",
|
|
5
|
-
"baseDescription": "generate_video produces text-to-video clips and Seedance multimodal reference videos.\nUse for text-only video generation with no source image input. For Seedance, also use this\ntool when uploaded/generated images, videos, or audio are loose references. Use animate_photo\nonly when a non-Seedance source image must become the first frame of an LTX/WAN animation.\n\nSEEDANCE UPLOADED STORYBOARD DEFAULT: When the user uploads a storyboard, shot sheet,\nmood board, or trailer concept image and asks to make a movie trailer/video/clip from it,\ndefault to one Seedance generate_video call with referenceImageIndices=[-1]. Do not first\nextract panels with edit_image, do not generate replacement keyframes, and do not make four\nseparate LTX animate_photo clips unless the user explicitly asks for separate clips or LTX.\nUse seedance2 when premium Spark access is available; if premium access is unavailable,\nexplain the limitation or use the best non-Seedance fallback the user accepts.\n\nSTORYTELLING / COMMERCIAL / TRAILER PROMPTS: For creative video requests, turn the brief\ninto timed, causally connected visual beats before writing the final prompt. Default social\nvideo is 15s 9:16 with a strong first 1-2s, visible escalation, payoff, and brand/CTA/final\nimage. Commercials should show audience desire/problem, transformation, proof/benefit, and\nCTA. Trailers should follow hook → world → disruption → escalation → reveal → title/CTA.\nEvery beat must be generatable: subject, setting, action, camera, lighting, audio, and text\nrole where relevant. Avoid vague \"cinematic\" filler, feature dumps, and beautiful images with\nno visible change.\n\nVIDEO PROMPT QUOTING: ONLY use double quotes for spoken dialogue in video prompts. Never\nquote on-screen text, titles, captions, or visual text elements — describe them without\nquotes. Quotes signal speech to the model and confuse audio generation.\n\nSTORYBOARD TEXT: Structural headings, section numbers, slide titles, panel titles, and\ncaptions in storyboard references may become short audio-only narration/VO or\nkey-message beats, but they are not subtitles, title cards, lower thirds, or visible\noverlays unless the user explicitly asks for visible text, on-screen text, a title\ncard, subtitle, lower third, signage, or CTA. Keep narration as separate brief phrases\nwith pauses; do not concatenate storyboard labels into run-on voiceover.\n\nDIALOGUE DURATION: Spoken dialogue must fit the clip. Estimate 2.5 words per second\nnatural delivery plus ~1s per acting beat. Hard maximum 3.75 words/second.\nCheck: dialogue words ÷ 2.5 + beats ≤ duration. Do not submit oversized dialogue.\n\nLATEST USER DURATION WINS: In follow-up turns, use the newest duration the user states,\neven if a previous assistant message mentioned a longer script/runtime. For example, if\nhistory says \"the full script is 66 seconds\" but the user now says \"do a 30 second version\",\ngenerate the 30 second version. Do not ask a clarification question just because history\ncontains another duration; treat the latest user request as the override.\n\nSEEDANCE DURATION LIMITS: Seedance 2.0 and Mini support 4-15s clips; Seedance 2.5 supports 4-30s clips. If the user explicitly asks\nfor Seedance below 4s, do not silently round up. Ask whether they prefer a 4s Seedance clip\nor an exact-duration LTX clip. If the user did not explicitly ask for Seedance, choose the\nmodel/tool that can satisfy the requested duration exactly.\n\nMINIMAX H3 TWO-STAGE OUTPUT: 1080p and 2K MiniMax H3 delivery is its own selector, not an option. \"minimax-h3-fasth3-t2v-turbo-2stage\" is the two-stage FastH3 engine: FastH3 renders the canvas, then the worker enlarges it 2x and refines it, so the clip is delivered at twice the canvas width and height with the same length and audio. targetResolution names the delivered class: 1080 renders a 544px short-edge canvas (960x544 becomes 1920x1088) for
|
|
5
|
+
"baseDescription": "generate_video produces text-to-video clips and Seedance multimodal reference videos.\nUse for text-only video generation with no source image input. For Seedance, also use this\ntool when uploaded/generated images, videos, or audio are loose references. Use animate_photo\nonly when a non-Seedance source image must become the first frame of an LTX/WAN animation.\n\nSEEDANCE UPLOADED STORYBOARD DEFAULT: When the user uploads a storyboard, shot sheet,\nmood board, or trailer concept image and asks to make a movie trailer/video/clip from it,\ndefault to one Seedance generate_video call with referenceImageIndices=[-1]. Do not first\nextract panels with edit_image, do not generate replacement keyframes, and do not make four\nseparate LTX animate_photo clips unless the user explicitly asks for separate clips or LTX.\nUse seedance2 when premium Spark access is available; if premium access is unavailable,\nexplain the limitation or use the best non-Seedance fallback the user accepts.\n\nSTORYTELLING / COMMERCIAL / TRAILER PROMPTS: For creative video requests, turn the brief\ninto timed, causally connected visual beats before writing the final prompt. Default social\nvideo is 15s 9:16 with a strong first 1-2s, visible escalation, payoff, and brand/CTA/final\nimage. Commercials should show audience desire/problem, transformation, proof/benefit, and\nCTA. Trailers should follow hook → world → disruption → escalation → reveal → title/CTA.\nEvery beat must be generatable: subject, setting, action, camera, lighting, audio, and text\nrole where relevant. Avoid vague \"cinematic\" filler, feature dumps, and beautiful images with\nno visible change.\n\nVIDEO PROMPT QUOTING: ONLY use double quotes for spoken dialogue in video prompts. Never\nquote on-screen text, titles, captions, or visual text elements — describe them without\nquotes. Quotes signal speech to the model and confuse audio generation.\n\nSTORYBOARD TEXT: Structural headings, section numbers, slide titles, panel titles, and\ncaptions in storyboard references may become short audio-only narration/VO or\nkey-message beats, but they are not subtitles, title cards, lower thirds, or visible\noverlays unless the user explicitly asks for visible text, on-screen text, a title\ncard, subtitle, lower third, signage, or CTA. Keep narration as separate brief phrases\nwith pauses; do not concatenate storyboard labels into run-on voiceover.\n\nDIALOGUE DURATION: Spoken dialogue must fit the clip. Estimate 2.5 words per second\nnatural delivery plus ~1s per acting beat. Hard maximum 3.75 words/second.\nCheck: dialogue words ÷ 2.5 + beats ≤ duration. Do not submit oversized dialogue.\n\nLATEST USER DURATION WINS: In follow-up turns, use the newest duration the user states,\neven if a previous assistant message mentioned a longer script/runtime. For example, if\nhistory says \"the full script is 66 seconds\" but the user now says \"do a 30 second version\",\ngenerate the 30 second version. Do not ask a clarification question just because history\ncontains another duration; treat the latest user request as the override.\n\nSEEDANCE DURATION LIMITS: Seedance 2.0 and Mini support 4-15s clips; Seedance 2.5 supports 4-30s clips. If the user explicitly asks\nfor Seedance below 4s, do not silently round up. Ask whether they prefer a 4s Seedance clip\nor an exact-duration LTX clip. If the user did not explicitly ask for Seedance, choose the\nmodel/tool that can satisfy the requested duration exactly.\n\nMINIMAX H3 TWO-STAGE OUTPUT: 1080p and 2K MiniMax H3 delivery is its own selector, not an option. \"minimax-h3-fasth3-t2v-turbo-2stage\" is the two-stage FastH3 engine: FastH3 renders the canvas, then the worker enlarges it 2x and refines it, so the clip is delivered at twice the canvas width and height with the same length and audio. targetResolution names the delivered class: 1080 renders a 544px short-edge canvas (960x544 becomes 1920x1088) for 10 Spark per second, 1440 or omitted renders the 768p canvas for 2K (1344x768 becomes 2688x1536) for 16 Spark per second, and 720 renders a 384px canvas (672x384 becomes 1344x768) for the regular FastH3 price of 4 Spark per second. The estimate prices every request. It takes the same prompt contract, durations and LoRAs as \"minimax-h3-fasth3-t2v-turbo\". Choose it when the user asks for 1080p, 1440p or 2K MiniMax H3 output, for two-stage output, or for the sharpest/best H3 quality; for ordinary 768p FastH3 output keep the regular FastH3 selector at targetResolution 768. Never set 4K for H3.",
|
|
6
6
|
"parameterDocs": {
|
|
7
7
|
"prompt": "Video prompt. Use double quotes ONLY for spoken dialogue. Describe visual text without quotes.",
|
|
8
8
|
"duration": "Clip duration in seconds. Plan dialogue word count against the 3.75 words/second ceiling."
|
|
@@ -1,10 +1,11 @@
|
|
|
1
1
|
{
|
|
2
2
|
"contractId": "sound_to_video_v1",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.1.0",
|
|
4
4
|
"toolName": "sound_to_video",
|
|
5
|
-
"baseDescription": "sound_to_video creates audio-synced video from an audio source. Works with uploaded audio\nfiles (mp3, m4a, wav) OR previously generated music from generate_music (auto-detected).\n\nWhen the user asks to \"turn that song/music into a video\" after generate_music, use\nsound_to_video — it will automatically find the generated audio.\n\nFor music visualization (syncing video to a specific song or audio track), use the\ngenerate_music → sound_to_video pipeline. Do NOT use animate_photo or generate_video for\naudio-driven visualization.\n\nanimate_photo and generate_video produce audio natively via LTX 2.5 by default, with LTX 2.3 retained as rollback — never pre-generate\naudio for those tools. sound_to_video is only for when the audio IS the primary creative\ninput driving the video output.",
|
|
5
|
+
"baseDescription": "sound_to_video creates audio-synced video from an audio source. Works with uploaded audio\nfiles (mp3, m4a, wav) OR previously generated music from generate_music (auto-detected).\n\nWhen the user asks to \"turn that song/music into a video\" after generate_music, use\nsound_to_video — it will automatically find the generated audio.\n\nFor music visualization (syncing video to a specific song or audio track), use the\ngenerate_music → sound_to_video pipeline. Do NOT use animate_photo or generate_video for\naudio-driven visualization.\n\nanimate_photo and generate_video produce audio natively via LTX 2.5 by default, with LTX 2.3 retained as rollback — never pre-generate\naudio for those tools. sound_to_video is only for when the audio IS the primary creative\ninput driving the video output.\n\nMINIMAX H3 AUDIO GUIDE: only when the user asks for MiniMax H3 or FastH3 with an uploaded audio\ntrack, use sound_to_video with videoModel \"minimax-h3-fasth3-ia2v-turbo\" (first-frame image via\nsourceImageIndex), \"minimax-h3-fasth3-flfa2v-turbo\" (first frame via sourceImageIndex and last\nframe via endImageIndex) or \"minimax-h3-fasth3-a2v-turbo\" (audio only, no image). The audio\ndrives the picture from frame 0 and stays in the output. Clips are 124-362 frames at a fixed\n24 fps (about 5.2-15.1 seconds); audioStart picks the window. generateAudio=false, LoRAs and\nnegativePrompt are not supported. Add \"-2stage\" only for 1080p, 1440p or 2K H3 output (1080 or\n1440 targetResolution); 720p-class audio output stays on the regular selector. Never use these\nin place of the LTX 2.5 defaults when the user did not ask for MiniMax H3 or FastH3.",
|
|
6
6
|
"parameterDocs": {
|
|
7
7
|
"audioSource": "Uploaded audio file or reference to a prior generate_music result. Auto-detected when omitted after generate_music.",
|
|
8
|
-
"negativePrompt": "Advanced LTX 2.5/LTX 2.3/WAN only. The LTX A2V and IA2V workflows accept a separate negative prompt; do not set it for Seedance."
|
|
8
|
+
"negativePrompt": "Advanced LTX 2.5/LTX 2.3/WAN only. The LTX A2V and IA2V workflows accept a separate negative prompt; do not set it for Seedance.",
|
|
9
|
+
"endImageIndex": "Last frame for the MiniMax H3 FastH3 first/last frame + audio selectors only, indexed like animate_photo (-1 first upload, -2 second upload, non-negative generated results)."
|
|
9
10
|
}
|
|
10
11
|
}
|
|
@@ -38,7 +38,7 @@
|
|
|
38
38
|
"wan3.0-video",
|
|
39
39
|
"wan3.0-spicy-video"
|
|
40
40
|
],
|
|
41
|
-
"description": "\"ltx25\" (default): LTX 2.5 I2V or first/last-frame video with native audio; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflow. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. Which video model to use. \"ltx23\": LTX 2.3 rollback with native audio. \"wan22\": quick simple motion without audio, up to 10s. \"minimax-h3-i2v\" is standard MiniMax H3 from one first frame; \"minimax-h3-i2v-turbo\" is the existing 4-step LightX2V Turbo engine; \"minimax-h3-fasth3-i2v-turbo\" is the separate FastVideo VSA four-step FastH3 engine, about 2x faster and fixed to Euler/simple. \"minimax-h3-fasth3-i2v-turbo-2stage\" (first frame) and \"minimax-h3-fasth3-flf2v-turbo-2stage\" (first and last frame) are the two-stage FastH3 engine: FastH3 renders the canvas, then the worker enlarges it 2x and refines it, so the clip is delivered at twice the canvas width and height with the same length and audio. targetResolution picks the delivered class: 1080 renders a 544px short-edge canvas (960x544 is delivered at 1920x1088) for
|
|
41
|
+
"description": "\"ltx25\" (default): LTX 2.5 I2V or first/last-frame video with native audio; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflow. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. Which video model to use. \"ltx23\": LTX 2.3 rollback with native audio. \"wan22\": quick simple motion without audio, up to 10s. \"minimax-h3-i2v\" is standard MiniMax H3 from one first frame; \"minimax-h3-i2v-turbo\" is the existing 4-step LightX2V Turbo engine; \"minimax-h3-fasth3-i2v-turbo\" is the separate FastVideo VSA four-step FastH3 engine, about 2x faster and fixed to Euler/simple. \"minimax-h3-fasth3-i2v-turbo-2stage\" (first frame) and \"minimax-h3-fasth3-flf2v-turbo-2stage\" (first and last frame) are the two-stage FastH3 engine: FastH3 renders the canvas, then the worker enlarges it 2x and refines it, so the clip is delivered at twice the canvas width and height with the same length and audio. targetResolution picks the delivered class: 1080 renders a 544px short-edge canvas (960x544 is delivered at 1920x1088) for 10 Spark per second, 1440 or omitted renders the 768p canvas for 2K (1344x768 is delivered at 2688x1536) for 16 Spark per second, and 720 renders a 384px canvas (672x384 is delivered at 1344x768) for the regular FastH3 price of 4 Spark per second. The estimate prices every request. They take the same inputs, durations and LoRAs as their FastH3 selectors. Choose them when the user asks for 1080p, 1440p or 2K MiniMax H3 output, for two-stage output, or for the sharpest/best H3 quality; for ordinary 768p FastH3 output keep the regular FastH3 selector at targetResolution 768. The matching FLF2V selectors provide standard, LightX2V Turbo, FastH3 Turbo, and two-stage FastH3 first/last-frame generation; FastH3 has no R2V mode; use frameRole=\"both\" and provide the end frame. H3 generates native audio at fixed 24fps for 5.17-15.08s and has no negative-prompt input. H3 Base and Turbo prompts use the exact three-field contract and the official mode-specific alignment line. Do not set Seedance here; use generate_video with Seedance references. \"wan3.0-video\" is Alibaba Wan 3 and \"wan3.0-spicy-video\" is MuleRouter w3.0-video. Both render 2-30s at fixed 30 fps with optional native audio, provider prompt expansion, 480p/720p/1080p, adaptive/fixed ratios, first/last frames, and up to 10 image/5 video/5 audio references. Only Alibaba wan3.0-video accepts document/web context and watermark. Frame anchors and loose references are mutually exclusive. Do not send negativePrompt; video references are loose conditioning for a new result, not source-video editing or extension."
|
|
42
42
|
},
|
|
43
43
|
"negativePrompt": {
|
|
44
44
|
"type": "string",
|
|
@@ -74,7 +74,7 @@
|
|
|
74
74
|
"wan3.0-video",
|
|
75
75
|
"wan3.0-spicy-video"
|
|
76
76
|
],
|
|
77
|
-
"description": "\"ltx25\" (default): LTX 2.5 with native audio; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflow. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. Video model. \"ltx23\": LTX 2.3 rollback with native audio. \"wan22\": quick simple motion without audio. \"minimax-h3-t2v\": standard 20-step MiniMax H3 text-to-video; \"minimax-h3-t2v-turbo\": the existing 4-step LightX2V Turbo text-to-video; \"minimax-h3-fasth3-t2v-turbo\": the separate FastVideo VSA four-step FastH3 engine, about 2x faster and fixed to Euler/simple; \"minimax-h3-fasth3-t2v-turbo-2stage\": the two-stage FastH3 engine: FastH3 renders the canvas, then the worker enlarges it 2x and refines it, so the clip is delivered at twice the canvas width and height with the same length and audio. targetResolution picks the delivered class: 1080 renders a 544px short-edge canvas (960x544 is delivered at 1920x1088) for
|
|
77
|
+
"description": "\"ltx25\" (default): LTX 2.5 with native audio; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflow. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. Video model. \"ltx23\": LTX 2.3 rollback with native audio. \"wan22\": quick simple motion without audio. \"minimax-h3-t2v\": standard 20-step MiniMax H3 text-to-video; \"minimax-h3-t2v-turbo\": the existing 4-step LightX2V Turbo text-to-video; \"minimax-h3-fasth3-t2v-turbo\": the separate FastVideo VSA four-step FastH3 engine, about 2x faster and fixed to Euler/simple; \"minimax-h3-fasth3-t2v-turbo-2stage\": the two-stage FastH3 engine: FastH3 renders the canvas, then the worker enlarges it 2x and refines it, so the clip is delivered at twice the canvas width and height with the same length and audio. targetResolution picks the delivered class: 1080 renders a 544px short-edge canvas (960x544 is delivered at 1920x1088) for 10 Spark per second, 1440 or omitted renders the 768p canvas for 2K (1344x768 is delivered at 2688x1536) for 16 Spark per second, and 720 renders a 384px canvas (672x384 is delivered at 1344x768) for the regular FastH3 price of 4 Spark per second. The estimate prices every request. It takes the same inputs, durations and LoRAs as \"minimax-h3-fasth3-t2v-turbo\". Choose it when the user asks for 1080p, 1440p or 2K MiniMax H3 output, for two-stage output, or for the sharpest/best H3 quality; for ordinary 768p FastH3 output keep the regular FastH3 selector at targetResolution 768. All use native audio, fixed 24fps, 5.17-15.08s, and a 768p-class 32px-grid canvas; use animate_photo for H3 image-conditioned modes. Base and Turbo T2V/I2V/FLF2V prompts use the exact ordered fields integrated_multimodal_description, overall_soundscape, and non_diegetic_music; I2V/FLF2V prepend the official alignment line. \"minimax-h3-r2v\": standard 20-step MiniMax H3 reference-to-video; \"minimax-h3-r2v-turbo\": the dedicated LightX2V 4-step Ref2VA Turbo workflow using Euler/simple and a 960x544 default. FastH3 has no R2V mode. Both R2V selectors accept up to 9 images, 3 videos, and 3 audios (12 files total); at least one visual reference (image or video) is required and audio alone is invalid. Select references with referenceImageIndices/referenceVideoIndices/referenceAudioIndices and address them with the official <Subject N>/<Picture N>/<Video N>/<Audio N> semantics. Seedance quality is selected only by model: use \"seedance2-mini\" for Seedance 2.0 Mini or faster/lower-cost 720p iteration, and use \"seedance2\" for the full Seedance 2.0 model, explicit full-quality requests, 1080p/4K requests, or generated/uploaded storyboard images unless the user explicitly asks for a draft or Mini. Do not use Default Media Quality Fast/HQ/Pro or targetResolution to represent Seedance quality. \"seedance2-5\": Seedance 2.5, the newest Seedance generation — 480p, 720p, and 1080p (4K is unsupported), 4-30s per clip at a fixed 24 fps, native audio, first-and-last-frame conditioning, and a much larger reference budget than the 2.0 family: up to 30 images, 10 videos, and 10 audios, with up to 50 reference media files total, subject to those per-modality caps. Choose \"seedance2-5\" when the user asks for Seedance 2.5, wants a single continuous Seedance clip longer than 15s (2.5 renders up to 30s in one call instead of being split and stitched), or wants a first-and-last-frame Seedance transition. Keep \"seedance2\" for 4K requests; Seedance 2.5 supports up to 1080p. Seedance supports multimodal loose reference assets: images (up to 9), videos (up to 3), and audios (up to 3), with no more than 12 asset files total. Use @Image1/@Video1/@Audio1 style references in creative briefs when assigning roles. Assign every useful reference asset a role and prefer positive preservation constraints. If an uploaded video is the source clip to transform, upscale, enhance, restyle, or remaster, use video_to_video with controlMode=\"seedance-v2v\" instead of generate_video referenceVideoIndices. Alibaba HappyHorse 1.1 video models (third-party vendor — requires Premium Spark). Select by mode: \"happyhorse-1.1-t2v\" for text-to-video, \"happyhorse-1.1-i2v\" for image-to-video from one first-frame image, and \"happyhorse-1.1-r2v\" for reference-to-video with up to 9 reference images. Resolutions 720P and 1080P; duration 3-15 seconds at 24 fps; native synchronized audio is always generated (do not set generateAudio or negativePrompt). Supported aspect ratios: 16:9, 9:16, 1:1, 4:3, 3:4, 4:5, 5:4, 9:21, 21:9. HappyHorse 1.1 takes image references only and renders a native synchronized audio track (always on; do not set generateAudio or a negative prompt). Pick the model by mode: happyhorse-1.1-t2v for text-to-video (no reference image), happyhorse-1.1-i2v for image-to-video from a single first frame, and happyhorse-1.1-r2v for reference-to-video with 1 to 9 reference images. For r2v, tag the images in the prompt as [Image 1]…[Image 9] and assign each a clear role. HappyHorse does not accept reference videos or reference audios. \"wan3.0-video\" is Alibaba Wan 3 and \"wan3.0-spicy-video\" is MuleRouter w3.0-video. Both render 2-30s at fixed 30 fps with optional native audio, provider prompt expansion, 480p/720p/1080p, adaptive/fixed ratios, first/last frames, and up to 10 image/5 video/5 audio references. Only Alibaba wan3.0-video accepts document/web context and watermark. Frame anchors and loose references are mutually exclusive. Do not send negativePrompt; video references are loose conditioning for a new result, not source-video editing or extension."
|
|
78
78
|
},
|
|
79
79
|
"generateAudio": {
|
|
80
80
|
"type": "boolean",
|
|
@@ -111,7 +111,7 @@
|
|
|
111
111
|
},
|
|
112
112
|
"targetResolution": {
|
|
113
113
|
"type": "number",
|
|
114
|
-
"description": "Short-side video resolution target in pixels. Use when the user asks for a bare named resolution such as \"480p\", \"720p\", \"1080p\", \"2160p\", or \"4K\" without exact pixels or an output orientation. This is resolution only, not a Seedance quality tier: Seedance quality is selected by videoModel (\"seedance2\" vs \"seedance2-mini\" vs \"seedance2-5\"). Seedance 2.0 full supports 4K; Seedance Mini
|
|
114
|
+
"description": "Short-side video resolution target in pixels. Use when the user asks for a bare named resolution such as \"480p\", \"720p\", \"1080p\", \"2160p\", or \"4K\" without exact pixels or an output orientation. This is resolution only, not a Seedance quality tier: Seedance quality is selected by videoModel (\"seedance2\" vs \"seedance2-mini\" vs \"seedance2-5\"). Seedance 2.0 full supports 4K; Seedance Mini supports 480p/720p; Seedance 2.5 supports 480p/720p/1080p, so never set 4K for \"seedance2-5\". Wan 3 supports exactly 480p, 720p, and 1080p. HappyHorse supports only 720p and 1080p. Never set 4K for Wan 3 or HappyHorse. MiniMax H3 renders inside a 1344x768 pixel budget on a 32px grid, so use 768 for the regular H3 selectors and never 1080p or 4K. The two-stage H3 selector \"minimax-h3-fasth3-t2v-turbo-2stage\" delivers twice the canvas, so there targetResolution names the delivered short-edge class: 1080 (544px canvas short edge: 960x544 delivered at 1920x1088), 1440 for 2K (the 1344x768 canvas delivered at 2688x1536), or 720 (384px canvas: 672x384 delivered at 1344x768); omit it for 2K. Never set 4K for H3. Do not set targetResolution from Default Media Quality Fast/HQ/Pro. If omitted for Seedance, Wan 3, HappyHorse, or MiniMax H3, the host uses the selected model default. This preserves/inherits the current video shape instead of forcing landscape. Do NOT set width, height, or exact-pixel aspectRatio for bare named resolution requests. If the user says \"720p portrait\", \"720p landscape\", \"4K portrait\", or \"4K landscape\", use exact width/height/aspectRatio instead."
|
|
115
115
|
},
|
|
116
116
|
"numberOfVariations": {
|
|
117
117
|
"type": "number",
|
|
@@ -145,6 +145,18 @@
|
|
|
145
145
|
"type": "number"
|
|
146
146
|
},
|
|
147
147
|
"description": "Strength for each LoRA in loras, in the same order. Omitting the array applies 1.0 to every LoRA, which is NOT the catalog default and for h3-realism-people is already at the top of its band, so send explicit values. Video LoRAs are positive-only — unlike the bipolar Krea 2 image sliders, a negative value is not an inverse effect and 0 is off. h3-realism-people takes 0-2 and its catalog default is 0.8; 0.6-1 is the usable band. It also pulls the camera in as it climbs: at 1.5 and above the shot reliably recomposes and the grade darkens, which on an image-conditioned mode can crop the subject out of the frame the user supplied. Raise it above 1 only when the user asks for more, and prefer the default when they supplied a first or last frame. h3-vbvr-video-reasoning and h3-mystic-xxx-v4 both take 0-1 and do default to 1.0, with usable bands of 0.7-1 and 0.2-1. h3-natural-face-speech and h3-better-motion take 0-1.5 and default to 0.6; their usable band is 0.4-0.8."
|
|
148
|
+
},
|
|
149
|
+
"outputFormat": {
|
|
150
|
+
"type": "string",
|
|
151
|
+
"enum": [
|
|
152
|
+
"mp4",
|
|
153
|
+
"mov"
|
|
154
|
+
],
|
|
155
|
+
"description": "Video container. Defaults to mp4. MOV is supported only by Seedance 2.5; choose it when the user requests MOV for editing."
|
|
156
|
+
},
|
|
157
|
+
"returnLastFrame": {
|
|
158
|
+
"type": "boolean",
|
|
159
|
+
"description": "Seedance 2.5 only. Set true to export a separate image of the final frame alongside the video. The result includes lastFrameUrl, which can be used as the first-frame image for a subsequent clip. Defaults to false; this does not extend the video automatically."
|
|
148
160
|
}
|
|
149
161
|
},
|
|
150
162
|
"required": [
|
|
@@ -3,13 +3,13 @@
|
|
|
3
3
|
"$id": "https://schemas.sogni.ai/creative-agent/2026-07-18.1/tools/sound_to_video.schema.json",
|
|
4
4
|
"title": "sound_to_video arguments",
|
|
5
5
|
"schemaVersion": "2026-07-18.1",
|
|
6
|
-
"description": "Generate video synchronized to audio. Use when the user has uploaded an audio file (mp3, wav, m4a, flac) and the audio is the primary sync target, especially uploaded-audio-only workflows. Also use after generate_music (\"turn that song into a video\", \"make a music video from that\"). Auto-detects generated audio from generate_music if no audio file is uploaded. Seedance animate_photo/generate_video can also attach uploaded audio as a loose @Audio reference when an image or video reference anchors the request; use this tool instead when the soundtrack itself should drive the video. If the user provides a reference image, use ltx25-ia2v by default (ltx23-ia2v is rollback); for lip-sync with a face image, use wan-s2v; if no image, use ltx25-a2v by default (ltx23-a2v is rollback). If the user wants dialogue/audio WITHOUT pre-existing audio, use animate_photo instead (LTX 2.5 and LTX 2.3 generate audio natively). Note: Persona voice clips from resolve_personas are NOT used by this tool — for persona voice identity in video, use animate_photo or generate_video with videoModel=\"ltx23\" because LTX 2.5 has no compatible ID-LoRA. LONG AUDIO ON SEEDANCE: Seedance 2.0 and Mini cap each clip at 15s; Seedance 2.5 renders up to 30s in one call, so prefer seedance2-5 for 16-30s audio instead of splitting. When the user uploads audio longer than the per-clip cap of the selected model and Seedance is selected (seedance2, seedance2-mini, or seedance2-5), do NOT clamp to 15s and drop the rest — split the run into multiple sound_to_video calls in the same turn (one per 15s segment, so a 20s audio becomes two clips: audioStart=0 duration=15, then audioStart=15 duration=5) and finish with a single stitch_video call referencing the resulting clip indices in order with audioIndex pointing at the same uploaded audio so the stitched output carries the full original soundtrack. LTX/WAN models accept up to 20s per clip, so single-call is fine for them. Use videoModel=\"wan3.0-video\" when the user explicitly requests Wan 3 audio-driven video. Use videoModel=\"wan3.0-spicy-video\" for Wan 3.0 Enhanced audio-driven video through MuleRouter provider ID w3.0-video; it supports adaptive ratios and provider prompt expansion.",
|
|
6
|
+
"description": "Generate video synchronized to audio. Use when the user has uploaded an audio file (mp3, wav, m4a, flac) and the audio is the primary sync target, especially uploaded-audio-only workflows. Also use after generate_music (\"turn that song into a video\", \"make a music video from that\"). Auto-detects generated audio from generate_music if no audio file is uploaded. Seedance animate_photo/generate_video can also attach uploaded audio as a loose @Audio reference when an image or video reference anchors the request; use this tool instead when the soundtrack itself should drive the video. If the user provides a reference image, use ltx25-ia2v by default (ltx23-ia2v is rollback); for lip-sync with a face image, use wan-s2v; if no image, use ltx25-a2v by default (ltx23-a2v is rollback). If the user wants dialogue/audio WITHOUT pre-existing audio, use animate_photo instead (LTX 2.5 and LTX 2.3 generate audio natively). Note: Persona voice clips from resolve_personas are NOT used by this tool — for persona voice identity in video, use animate_photo or generate_video with videoModel=\"ltx23\" because LTX 2.5 has no compatible ID-LoRA. LONG AUDIO ON SEEDANCE: Seedance 2.0 and Mini cap each clip at 15s; Seedance 2.5 renders up to 30s in one call, so prefer seedance2-5 for 16-30s audio instead of splitting. When the user uploads audio longer than the per-clip cap of the selected model and Seedance is selected (seedance2, seedance2-mini, or seedance2-5), do NOT clamp to 15s and drop the rest — split the run into multiple sound_to_video calls in the same turn (one per 15s segment, so a 20s audio becomes two clips: audioStart=0 duration=15, then audioStart=15 duration=5) and finish with a single stitch_video call referencing the resulting clip indices in order with audioIndex pointing at the same uploaded audio so the stitched output carries the full original soundtrack. LTX/WAN models accept up to 20s per clip, so single-call is fine for them. Use videoModel=\"wan3.0-video\" when the user explicitly requests Wan 3 audio-driven video. Use videoModel=\"wan3.0-spicy-video\" for Wan 3.0 Enhanced audio-driven video through MuleRouter provider ID w3.0-video; it supports adaptive ratios and provider prompt expansion. MINIMAX H3 AUDIO: only when the user asks for MiniMax H3 or FastH3 audio-driven video, use videoModel=\"minimax-h3-fasth3-ia2v-turbo\" with a first-frame image (sourceImageIndex), \"minimax-h3-fasth3-flfa2v-turbo\" with a first and a last frame (sourceImageIndex and endImageIndex), or \"minimax-h3-fasth3-a2v-turbo\" with no image; add \"-2stage\" only for 1080p, 1440p or 2K H3 output. Without an explicit MiniMax H3 or FastH3 request keep the LTX 2.5 defaults. The MiniMax H3 selectors on generate_video and animate_photo cannot take an uploaded audio track; send uploaded-audio H3 requests here.",
|
|
7
7
|
"type": "object",
|
|
8
8
|
"additionalProperties": false,
|
|
9
9
|
"properties": {
|
|
10
10
|
"prompt": {
|
|
11
11
|
"type": "string",
|
|
12
|
-
"description": "Describe the video like a cinematographer. Let the audio define timing — use the prompt for visual interpretation. One flowing paragraph, present tense, specific natural language.\n\nLITERAL PROMPT OVERRIDE: If the user explicitly says not to modify the prompt, or to use it exactly/verbatim/as-is, copy the identified prompt text verbatim instead of applying these construction rules unless a hard requirement is missing. For Seedance or Wan 3, set expandPrompt=false.\n\nSTRUCTURE: shot/style and scale → subject → environment, lighting, color, texture, atmosphere → visual action synced to audio → camera movement. For LTX 2.3 image+audio mode, do not re-describe static details already visible in the reference image; focus on motion, action, camera, and how the image responds to the audio.\n\nMOTION PACING: Scale complexity to duration. <=6s: 1 main visual beat + 1 simple camera move. Around 10s: 2-3 clear beats + 1 camera move. >10s: up to 4 beats in clear sequence. Let the audio define timing, but avoid stacking subject, camera, and environment motion in short clips.\n\nBLOCKING: Direct layout when it affects the shot: left/right placement, foreground/background, facing direction, and relative distance between subjects.\n\nLIP-SYNC: Shot framing, speaker's appearance and setting, physical performance synced to audio — gestures, expressions, jaw movement between phrases. Include acting beats.\n\nMUSIC VISUALIZATION: Visual style, environment, and how elements react to rhythm and energy.\n\nAUDIO-REACTIVE: Motion and visual changes that correspond to sounds in the track.\n\nLTX VOCABULARY: camera (tracking, dolly, pan, tilt, handheld, static frame), lighting/atmosphere (golden hour, neon glow, dramatic shadows, fog, rain, smoke, reflections), scale/pacing (expansive, epic, intimate, claustrophobic, slow motion, time-lapse, lingering shot, continuous shot), style/genre (film noir, painterly, cyberpunk, stop-motion, claymation, 2D/3D animation, hand-drawn, fantasy, thriller, experimental film).\n\nAVOID: Vague prompts, too many competing visual elements, abstract descriptions without visible behavior, rigid numeric constraints, readable text or logos. QUOTING RULE: ONLY use double quotes for spoken dialogue. Never quote on-screen text, overlay text, titles, captions, signs, or any visual text — describe them without quotes.\n\nNON-SEEDANCE POSITIVE CONSTRAINTS: For ltx25-ia2v, ltx25-a2v, ltx23-ia2v, ltx23-a2v, and wan-s2v, prompt is a positive prompt. Translate user avoid/no/don't constraints into affirmative production constraints instead of copying negative phrasing. Preserve exact quoted visible text or dialogue when the user explicitly requests it; keep surrounding surfaces blank.\n\nBATCH VARIATIONS: When numberOfVariations > 1, use Dynamic Prompt syntax to vary the visual interpretation while keeping audio sync intent consistent. This is one Sogni project with multiple jobs, so prefer it when all outputs share the same audio source/window, image source, model, duration, dimensions, and parameters and only prompt text varies. Example: \"{abstract neon visualization|nature scene with swaying trees|urban street with rain} synced to the beat\"."
|
|
12
|
+
"description": "Describe the video like a cinematographer. Let the audio define timing — use the prompt for visual interpretation. One flowing paragraph, present tense, specific natural language.\n\nLITERAL PROMPT OVERRIDE: If the user explicitly says not to modify the prompt, or to use it exactly/verbatim/as-is, copy the identified prompt text verbatim instead of applying these construction rules unless a hard requirement is missing. For Seedance or Wan 3, set expandPrompt=false.\n\nSTRUCTURE: shot/style and scale → subject → environment, lighting, color, texture, atmosphere → visual action synced to audio → camera movement. For LTX 2.3 image+audio mode, do not re-describe static details already visible in the reference image; focus on motion, action, camera, and how the image responds to the audio.\n\nMOTION PACING: Scale complexity to duration. <=6s: 1 main visual beat + 1 simple camera move. Around 10s: 2-3 clear beats + 1 camera move. >10s: up to 4 beats in clear sequence. Let the audio define timing, but avoid stacking subject, camera, and environment motion in short clips.\n\nBLOCKING: Direct layout when it affects the shot: left/right placement, foreground/background, facing direction, and relative distance between subjects.\n\nLIP-SYNC: Shot framing, speaker's appearance and setting, physical performance synced to audio — gestures, expressions, jaw movement between phrases. Include acting beats.\n\nMUSIC VISUALIZATION: Visual style, environment, and how elements react to rhythm and energy.\n\nAUDIO-REACTIVE: Motion and visual changes that correspond to sounds in the track.\n\nLTX VOCABULARY: camera (tracking, dolly, pan, tilt, handheld, static frame), lighting/atmosphere (golden hour, neon glow, dramatic shadows, fog, rain, smoke, reflections), scale/pacing (expansive, epic, intimate, claustrophobic, slow motion, time-lapse, lingering shot, continuous shot), style/genre (film noir, painterly, cyberpunk, stop-motion, claymation, 2D/3D animation, hand-drawn, fantasy, thriller, experimental film).\n\nAVOID: Vague prompts, too many competing visual elements, abstract descriptions without visible behavior, rigid numeric constraints, readable text or logos. QUOTING RULE: ONLY use double quotes for spoken dialogue. Never quote on-screen text, overlay text, titles, captions, signs, or any visual text — describe them without quotes.\n\nNON-SEEDANCE POSITIVE CONSTRAINTS: For ltx25-ia2v, ltx25-a2v, ltx23-ia2v, ltx23-a2v, and wan-s2v, prompt is a positive prompt. Translate user avoid/no/don't constraints into affirmative production constraints instead of copying negative phrasing. Preserve exact quoted visible text or dialogue when the user explicitly requests it; keep surrounding surfaces blank.\n\nBATCH VARIATIONS: When numberOfVariations > 1, use Dynamic Prompt syntax to vary the visual interpretation while keeping audio sync intent consistent. This is one Sogni project with multiple jobs, so prefer it when all outputs share the same audio source/window, image source, model, duration, dimensions, and parameters and only prompt text varies. Example: \"{abstract neon visualization|nature scene with swaying trees|urban street with rain} synced to the beat\".\n\nMINIMAX H3 AUDIO SELECTORS: write the request plainly (subject, action, camera, the voice or sound heard in the upload and any exact words spoken in it); the MiniMax H3 prompt shaper turns it into the H3 contract for the matching image-to-video, first-and-last-frame or text-to-video mode. Describe the uploaded audio as it is, and do not invent other dialogue or music."
|
|
13
13
|
},
|
|
14
14
|
"expandPrompt": {
|
|
15
15
|
"type": "boolean",
|
|
@@ -17,7 +17,7 @@
|
|
|
17
17
|
},
|
|
18
18
|
"negativePrompt": {
|
|
19
19
|
"type": "string",
|
|
20
|
-
"description": "Advanced LTX 2.5/LTX 2.3/WAN only. The LTX A2V and IA2V workflows accept this separate negative prompt. Use it only when the user explicitly asks to set one. Do not set for Seedance.\n\nWan 3 has no negativePrompt request field; do not set this for wan3.0-video."
|
|
20
|
+
"description": "Advanced LTX 2.5/LTX 2.3/WAN only. The LTX A2V and IA2V workflows accept this separate negative prompt. Use it only when the user explicitly asks to set one. Do not set for Seedance.\n\nWan 3 has no negativePrompt request field; do not set this for wan3.0-video.\n\nMiniMax H3 has no negative-prompt input; do not set this for the MiniMax H3 FastH3 audio selectors and state exclusions positively in prompt."
|
|
21
21
|
},
|
|
22
22
|
"audioSourceIndex": {
|
|
23
23
|
"type": "number",
|
|
@@ -25,16 +25,20 @@
|
|
|
25
25
|
},
|
|
26
26
|
"sourceImageIndex": {
|
|
27
27
|
"type": "number",
|
|
28
|
-
"description": "Optional index of an uploaded image to use as the starting frame (0-based). Required for lip-sync models (WAN S2V). For audio-only-to-video models (LTX 2.3 A2V), this is optional — omit it to generate video purely from text + audio."
|
|
28
|
+
"description": "Optional index of an uploaded image to use as the starting frame (0-based). Required for lip-sync models (WAN S2V). For audio-only-to-video models (LTX 2.3 A2V), this is optional — omit it to generate video purely from text + audio. MiniMax H3 FastH3 audio guide: \"minimax-h3-fasth3-ia2v-turbo\" and \"minimax-h3-fasth3-flfa2v-turbo\" (and their -2stage forms) require this first frame; \"minimax-h3-fasth3-a2v-turbo\" and its -2stage form take no image, so omit it for them."
|
|
29
|
+
},
|
|
30
|
+
"endImageIndex": {
|
|
31
|
+
"type": "number",
|
|
32
|
+
"description": "Which image to use as the LAST frame, indexed like animate_photo's endImageIndex: negative indices for uploaded images (-1 = first upload, -2 = second upload) and 0-based non-negative indices for generated results. Only \"minimax-h3-fasth3-flfa2v-turbo\" and \"minimax-h3-fasth3-flfa2v-turbo-2stage\" take it, and they require it together with sourceImageIndex (for two uploaded images use sourceImageIndex=-1 and endImageIndex=-2). Omit it for every other videoModel."
|
|
29
33
|
},
|
|
30
34
|
"audioStart": {
|
|
31
35
|
"type": "number",
|
|
32
|
-
"description": "Start offset in seconds into the audio track. Use when the user says \"start 20 seconds in\", \"skip the intro\", \"use the chorus at 1:30\", etc. Default: 0 (beginning of audio). The video will be synced to the audio starting from this point.",
|
|
36
|
+
"description": "Start offset in seconds into the audio track. Use when the user says \"start 20 seconds in\", \"skip the intro\", \"use the chorus at 1:30\", etc. Default: 0 (beginning of audio). The video will be synced to the audio starting from this point. The MiniMax H3 FastH3 audio selectors take audioStart too: the clip uses the upload from audioStart for its own length.",
|
|
33
37
|
"minimum": 0
|
|
34
38
|
},
|
|
35
39
|
"duration": {
|
|
36
40
|
"type": "number",
|
|
37
|
-
"description": "Video duration in seconds. Default: 5. Per-model range: LTX/WAN 2.2 = 2-20s; Wan 3 = 2-30s; Seedance 2.0 and Mini = 4-15s; Seedance 2.5 = 4-30s. For music videos, use the maximum duration the selected model allows because the audio is usually longer than the video limit. Use when the user explicitly requests a specific length.",
|
|
41
|
+
"description": "Video duration in seconds. Default: 5. Per-model range: LTX/WAN 2.2 = 2-20s; Wan 3 = 2-30s; Seedance 2.0 and Mini = 4-15s; Seedance 2.5 = 4-30s. For music videos, use the maximum duration the selected model allows because the audio is usually longer than the video limit. Use when the user explicitly requests a specific length. MiniMax H3 FastH3 audio selectors render 124-362 frames on a 17-frame grid at a fixed 24 fps, so the clip runs about 5.2-15.1 seconds and a length outside that window snaps to it; for longer audio pick the window with audioStart.",
|
|
38
42
|
"minimum": 2,
|
|
39
43
|
"maximum": 30
|
|
40
44
|
},
|
|
@@ -74,13 +78,19 @@
|
|
|
74
78
|
"ltx23-ia2v",
|
|
75
79
|
"ltx23-a2v",
|
|
76
80
|
"wan3.0-video",
|
|
77
|
-
"wan3.0-spicy-video"
|
|
81
|
+
"wan3.0-spicy-video",
|
|
82
|
+
"minimax-h3-fasth3-ia2v-turbo",
|
|
83
|
+
"minimax-h3-fasth3-ia2v-turbo-2stage",
|
|
84
|
+
"minimax-h3-fasth3-flfa2v-turbo",
|
|
85
|
+
"minimax-h3-fasth3-flfa2v-turbo-2stage",
|
|
86
|
+
"minimax-h3-fasth3-a2v-turbo",
|
|
87
|
+
"minimax-h3-fasth3-a2v-turbo-2stage"
|
|
78
88
|
],
|
|
79
|
-
"description": "\"ltx25-ia2v\" (default with image) and \"ltx25-a2v\" (default without image): LTX 2.5 image+audio and audio-only modes; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflows. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates official ComfyUI Dev recipes. Video model. \"ltx23-ia2v\" (rollback with image): LTX 2.3 image+audio to video, audio-reactive with a reference image; Fast/HQ use the distilled 8-step worker and Default Media Quality Pro uses the non-distilled dev worker. \"ltx23-a2v\" (rollback without image): LTX 2.3 audio-only to video, no image needed, creates video purely from text prompt + audio with the same quality-tier routing. \"wan-s2v\": WAN 2.2 sound-to-video, best for lip-sync with a face image, fast 4-step. \"seedance2\": full Seedance 2.0 audio-reference video, 4-15s. \"seedance2-mini\": Seedance 2.0 Mini, 720p cap, fastest/lower-cost Seedance option. Seedance quality is selected only by this model value: pick \"seedance2-mini\" for faster/lower-cost drafts or explicit Mini requests, and pick \"seedance2\" for full-quality Seedance or 1080p/4K. Do not infer the Seedance model from Default Media Quality Fast/HQ/Pro. \"seedance2-5\": Seedance 2.5, the newest Seedance generation — 480p and
|
|
89
|
+
"description": "\"ltx25-ia2v\" (default with image) and \"ltx25-a2v\" (default without image): LTX 2.5 image+audio and audio-only modes; Fast, HQ, and Pro currently use the release-validated official Distilled INT8 workflows. The Dev checkpoints are not publicly routed until upstream publishes and Sogni validates official ComfyUI Dev recipes. Video model. \"ltx23-ia2v\" (rollback with image): LTX 2.3 image+audio to video, audio-reactive with a reference image; Fast/HQ use the distilled 8-step worker and Default Media Quality Pro uses the non-distilled dev worker. \"ltx23-a2v\" (rollback without image): LTX 2.3 audio-only to video, no image needed, creates video purely from text prompt + audio with the same quality-tier routing. \"wan-s2v\": WAN 2.2 sound-to-video, best for lip-sync with a face image, fast 4-step. \"seedance2\": full Seedance 2.0 audio-reference video, 4-15s. \"seedance2-mini\": Seedance 2.0 Mini, 720p cap, fastest/lower-cost Seedance option. Seedance quality is selected only by this model value: pick \"seedance2-mini\" for faster/lower-cost drafts or explicit Mini requests, and pick \"seedance2\" for full-quality Seedance or 1080p/4K. Do not infer the Seedance model from Default Media Quality Fast/HQ/Pro. \"seedance2-5\": Seedance 2.5, the newest Seedance generation — 480p, 720p, and 1080p (4K is unsupported), 4-30s per clip at a fixed 24 fps, native audio, first-and-last-frame conditioning, and a much larger reference budget than the 2.0 family: up to 30 images, 10 videos, and 10 audios, with up to 50 reference media files total, subject to those per-modality caps. Choose \"seedance2-5\" when the user asks for Seedance 2.5, wants a single continuous Seedance clip longer than 15s (2.5 renders up to 30s in one call instead of being split and stitched), or wants a first-and-last-frame Seedance transition. Keep \"seedance2\" for 4K requests; Seedance 2.5 supports up to 1080p. For Seedance audio-reference prompts, preserve exact spoken dialogue when the user supplied it, and assign @Image1/@Audio1 roles. If the user asks for speech without words, describe the vocal performance without inventing quoted dialogue. Treat lip-sync, voice cloning, and real-human reference behavior as provider-sensitive rather than guaranteed. Omit to auto-select based on whether an image is present. \"wan3.0-video\" is Alibaba Wan 3 and \"wan3.0-spicy-video\" is MuleRouter w3.0-video. Both render 2-30s at fixed 30 fps with optional native audio, provider prompt expansion, 480p/720p/1080p, adaptive/fixed ratios, first/last frames, and up to 10 image/5 video/5 audio references. Only Alibaba wan3.0-video accepts document/web context and watermark. Frame anchors and loose references are mutually exclusive. Do not send negativePrompt; video references are loose conditioning for a new result, not source-video editing or extension. MiniMax H3 FastH3 audio guide: \"minimax-h3-fasth3-ia2v-turbo\" (first-frame image from sourceImageIndex plus the audio), \"minimax-h3-fasth3-flfa2v-turbo\" (first frame from sourceImageIndex, last frame from endImageIndex, plus the audio) and \"minimax-h3-fasth3-a2v-turbo\" (audio only; set neither sourceImageIndex nor endImageIndex) run the four-step FastH3 engine with the uploaded audio driving the picture from frame 0, and the output keeps that audio as its soundtrack. Choose them only when the user asks for MiniMax H3 or FastH3; never pick them in place of the LTX 2.5 defaults. They render 124-362 frames on the H3 17-frame grid at a fixed 24 fps (about 5.2-15.1 seconds) on a 32px grid within 1344x768, so use targetResolution 768 or omit it. audioStart picks the window of the upload. generateAudio=false, LoRAs and negativePrompt are not supported. Each costs the FastH3 price of its image-to-video, first-and-last-frame or text-to-video mode, 4 Spark per second. \"minimax-h3-fasth3-ia2v-turbo-2stage\", \"minimax-h3-fasth3-flfa2v-turbo-2stage\" and \"minimax-h3-fasth3-a2v-turbo-2stage\" are the two-stage forms: the same inputs rendered on the FastH3 canvas, then enlarged 2x and refined, delivered at twice the canvas with the same length and audio. For them targetResolution names the delivered class: 1080 renders a 544px short-edge canvas (960x544 is delivered at 1920x1088) for 10 Spark per second, and 1440 or omitted renders the 768p canvas for 2K (1344x768 is delivered at 2688x1536) for 16 Spark per second. The audio two-stage selectors have no 720p price class (720 would render a 384px canvas at the 1080p rate), so for 720p or 768p audio-guided H3 output use the regular selector. The estimate prices every request."
|
|
80
90
|
},
|
|
81
91
|
"generateAudio": {
|
|
82
92
|
"type": "boolean",
|
|
83
|
-
"description": "Whether the returned video should include audio. Omit to include audio by default; set false when the user asks for silent output or no audio. The reference audio is still required and still drives generation even when the returned video has no audio track."
|
|
93
|
+
"description": "Whether the returned video should include audio. Omit to include audio by default; set false when the user asks for silent output or no audio. The reference audio is still required and still drives generation even when the returned video has no audio track. MiniMax H3 FastH3 audio selectors always deliver the uploaded audio: omit generateAudio for them (false is refused)."
|
|
84
94
|
},
|
|
85
95
|
"numberOfVariations": {
|
|
86
96
|
"type": "number",
|
|
@@ -95,6 +105,18 @@
|
|
|
95
105
|
"aspectRatio": {
|
|
96
106
|
"type": "string",
|
|
97
107
|
"description": "Do NOT set unless the user explicitly requests an aspect ratio, format, orientation, or exact pixel dimensions. When a reference/source image is used and the user did not ask to change its shape, omit this field so the handler preserves the selected source image's own ratio.\n\nFormats: \"16:9\", \"9:16\", \"4:5\", \"1:1\", \"4:3\", \"3:2\", \"21:9\", or exact pixels like \"1920x1080\".\n\nCRITICAL: When the user specifies exact pixel dimensions (e.g., \"1280x720\", \"1080x1920\", \"1920x1080\", \"3840x2160\") or an orientation-qualified named resolution (e.g., \"720p landscape\", \"720p portrait\"), use the exact pixel format, NOT a ratio like \"16:9\" or \"9:16\". Exact user-requested dimensions override the selected default media quality, including Pro/HQ defaults. A bare named video resolution like \"720p resolution\" is only a resolution tier/short-side request; do not turn it into landscape pixels and do not set aspectRatio unless the user also states landscape, portrait, vertical, horizontal, or exact pixels. If requested pixels are in bounds but not on the model's pixel step, still pass the user's exact pixel request; the handler snaps to the nearest supported size internally. Only use ratio format when the user says a generic format name without pixel dimensions.\n\nMappings (use ONLY when user does NOT specify pixel dimensions): landscape/widescreen/YouTube/cinematic → \"16:9\". portrait → \"9:16\". TikTok/Reels/IG Reels → \"1080x1920\". ultrawide/cinema scope → \"21:9\". Instagram post → \"4:5\". square → \"1:1\". standard/TV → \"4:3\". 720p landscape → \"1280x720\". 720p portrait → \"720x1280\". 1080p landscape → \"1920x1080\". 1080p portrait/HD portrait → \"1080x1920\". 4K landscape → \"3840x2160\". 4K portrait → \"2160x3840\". Never set for generic requests like \"make a video\"."
|
|
108
|
+
},
|
|
109
|
+
"outputFormat": {
|
|
110
|
+
"type": "string",
|
|
111
|
+
"enum": [
|
|
112
|
+
"mp4",
|
|
113
|
+
"mov"
|
|
114
|
+
],
|
|
115
|
+
"description": "Video container. Defaults to mp4. MOV is supported only by Seedance 2.5; choose it when the user requests MOV for editing."
|
|
116
|
+
},
|
|
117
|
+
"returnLastFrame": {
|
|
118
|
+
"type": "boolean",
|
|
119
|
+
"description": "Seedance 2.5 only. Set true to export a separate image of the final frame alongside the video. The result includes lastFrameUrl, which can be used as the first-frame image for a subsequent clip. Defaults to false; this does not extend the video automatically."
|
|
98
120
|
}
|
|
99
121
|
},
|
|
100
122
|
"required": [
|
|
@@ -48,7 +48,7 @@
|
|
|
48
48
|
"seedance2-mini",
|
|
49
49
|
"seedance2-5"
|
|
50
50
|
],
|
|
51
|
-
"description": "Model selector for this video-to-video request. Usually omit: non-Seedance controls default to \"ltx25-v2v\"; use \"ltx23-v2v\" only for rollback. LTX 2.5 Fast, HQ, and Pro currently use the release-validated official Distilled workflow for canny, pose, depth, detailer, inpaint, and outpaint. Dev is not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. For controlMode=\"seedance-v2v\", Seedance quality is selected only by model: use \"seedance2-mini\" for faster/lower-cost drafts and use \"seedance2\" for full-quality Seedance or 1080p/4K. \"seedance2-5\" supports 480p/720p, 4-30s at 24 fps, native audio, and first/last-frame conditioning; keep \"seedance2\" for
|
|
51
|
+
"description": "Model selector for this video-to-video request. Usually omit: non-Seedance controls default to \"ltx25-v2v\"; use \"ltx23-v2v\" only for rollback. LTX 2.5 Fast, HQ, and Pro currently use the release-validated official Distilled workflow for canny, pose, depth, detailer, inpaint, and outpaint. Dev is not publicly routed until upstream publishes and Sogni validates an official ComfyUI Dev recipe. For controlMode=\"seedance-v2v\", Seedance quality is selected only by model: use \"seedance2-mini\" for faster/lower-cost drafts and use \"seedance2\" for full-quality Seedance or 1080p/4K. \"seedance2-5\" supports 480p/720p/1080p, 4-30s at 24 fps, native audio, and first/last-frame conditioning; keep \"seedance2\" for 4K."
|
|
52
52
|
},
|
|
53
53
|
"generateAudio": {
|
|
54
54
|
"type": "boolean",
|
|
@@ -56,7 +56,7 @@
|
|
|
56
56
|
},
|
|
57
57
|
"targetResolution": {
|
|
58
58
|
"type": "number",
|
|
59
|
-
"description": "Seedance V2V only. Short-side output resolution target in pixels. Use when the user asks for a bare named resolution such as \"480p\", \"720p\", \"1080p\", \"2160p\", or \"4K\" without exact dimensions. Seedance V2V full supports 4K; Seedance V2V Mini
|
|
59
|
+
"description": "Seedance V2V only. Short-side output resolution target in pixels. Use when the user asks for a bare named resolution such as \"480p\", \"720p\", \"1080p\", \"2160p\", or \"4K\" without exact dimensions. Seedance V2V full supports 4K; Seedance V2V Mini and Fast support 480p and 720p only; Seedance 2.5 also supports 1080p, so never set 4K for \"seedance2-5\". Preserve the source video shape instead of forcing landscape pixels."
|
|
60
60
|
},
|
|
61
61
|
"sourceImageIndex": {
|
|
62
62
|
"type": "number",
|
|
@@ -100,6 +100,18 @@
|
|
|
100
100
|
"description": "Number of video variations to generate (1-16). Default: 1.",
|
|
101
101
|
"minimum": 1,
|
|
102
102
|
"maximum": 16
|
|
103
|
+
},
|
|
104
|
+
"outputFormat": {
|
|
105
|
+
"type": "string",
|
|
106
|
+
"enum": [
|
|
107
|
+
"mp4",
|
|
108
|
+
"mov"
|
|
109
|
+
],
|
|
110
|
+
"description": "Video container. Defaults to mp4. MOV is supported only by Seedance 2.5; choose it when the user requests MOV for editing."
|
|
111
|
+
},
|
|
112
|
+
"returnLastFrame": {
|
|
113
|
+
"type": "boolean",
|
|
114
|
+
"description": "Seedance 2.5 only. Set true to export a separate image of the final frame alongside the video. The result includes lastFrameUrl, which can be used as the first-frame image for a subsequent clip. Defaults to false; this does not extend the video automatically."
|
|
103
115
|
}
|
|
104
116
|
},
|
|
105
117
|
"required": [
|
package/version.json
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
1
|
{
|
|
2
|
-
"protocolVersion": "7.
|
|
2
|
+
"protocolVersion": "7.2.0",
|
|
3
3
|
"description": "Sogni protocol artifact version. SDKs may refuse to operate against a protocolVersion they were not built for. Bump the major when removing or renaming any schema / enum / manifest field; bump the minor when adding new optional fields or new tools; bump the patch for description / prose changes only."
|
|
4
4
|
}
|