@shop-os/modelkit 0.4.7 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +71 -0
- package/README.md +43 -5
- package/dist/catalog.cjs +6291 -882
- package/dist/catalog.cjs.map +1 -1
- package/dist/catalog.d.cts +660 -34
- package/dist/catalog.d.ts +660 -34
- package/dist/catalog.js +6291 -882
- package/dist/catalog.js.map +1 -1
- package/dist/index.cjs +11226 -3910
- package/dist/index.cjs.map +1 -1
- package/dist/index.d.cts +654 -25
- package/dist/index.d.ts +654 -25
- package/dist/index.js +11211 -3910
- package/dist/index.js.map +1 -1
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -9,6 +9,77 @@ header in the release PR.
|
|
|
9
9
|
|
|
10
10
|
## [Unreleased]
|
|
11
11
|
|
|
12
|
+
## [0.6.0] - Unreleased
|
|
13
|
+
|
|
14
|
+
### Added
|
|
15
|
+
- `model3d` model type: a prompt and/or photos of an object in, a GLB out. Seven fal rows (`trellis-2`, `hunyuan3d-3.1-pro`, `tripo-h3.1`, `rodin-gen-2.5`, `meshy-7.1`, `hi3d-3.0`, `sam-3d-objects`), `exposure: ["space_builder"]`. What each takes is data on `constraints.model3d` (family, endpoints, text, required image or prompt, views, texture, PBR, seed, resolutions); the catalog schema requires it on every `model3d` row and rejects a spec its family's request mapping could not run or price (a missing endpoint for a request it accepts, views, texture, PBR, seed or resolution options the family does not read or needs).
|
|
16
|
+
- `ModelKit.submitModel3d` / `model3dStatus` / `model3dResult` / `cancelModel3d` (fal adapter): a resumable queue job like the video tools, with the same pinned queue origin, webhook and key tiers. Inputs a row cannot run are refused before submit.
|
|
17
|
+
- `model3dInputRefusal`, `model3dPricingSku`, `model3dSkus`, `model3dTexture`, `model3dResolution`, `model3dConstraints`, `MODEL3D_IMAGE_SLOTS` and `MODEL3D_MODELS`, also on the browser-safe `@shop-os/modelkit/catalog` entry. A 3D request bills one flat price per option combination (its SKU), not per model id.
|
|
18
|
+
- `Model3dRequest.views.top` (and `Model3dWired.top`): a fifth photo, sent only by `unordered` rows, so Rodin Gen-2.5 takes the five images fal's `image_urls` allows (Meshy sends the first four set; named and ordered rows have no slot for it). `MODEL3D_IMAGE_SLOTS` lists the slots the request mapping fills per views kind, and the catalog schema refuses a `maxImages` above that count.
|
|
19
|
+
- `UsageUnit` `"requests"`: one call billed at a flat price, the usage sink's `REQUESTS`. The `model3d` rows and their usage events use it (a 3D job is one flat-priced request, not an image). Kits before 0.6.0 quarantine rows with this unit.
|
|
20
|
+
- `constraints.model3d.seedMax` and `model3dSeedFits(spec, seed)`: the largest seed a row's provider takes (Rodin Gen-2.5: 65535, fal's limit; absent means any non-negative integer). `submitModel3d` refuses a seed outside it as `unsupported_option` before submit, and the catalog schema refuses `seedMax` on a row without `seed`.
|
|
21
|
+
- The 3D adapter accepts only https GLB and thumbnail URLs in a result (stored or webhook); any other scheme is a `provider_rejected` `ModelKitError`, never passed through.
|
|
22
|
+
|
|
23
|
+
### Changed
|
|
24
|
+
- The fal queue helpers moved from `fal-video-tools.ts` to `fal-queue.ts`, shared by video tools and 3D. `falWebhookBodySchema` and `FalWebhookBody` are still exported from the package root.
|
|
25
|
+
|
|
26
|
+
## [0.5.0] - Unreleased
|
|
27
|
+
|
|
28
|
+
ENG-3271: the latest fal / Prodia models for the catalog and the Sloosh canvas.
|
|
29
|
+
|
|
30
|
+
Flat-priced media rows cap their lengths at what the price covers: ElevenLabs Music v2.5 60 s, Mirelo SFX 2.0 30 s, ElevenLabs Dubbing 60 s of source, Scribe v2 10 minutes, VEED Lipsync v2 30 s of video. VEED Subtitles is priced per source second like the other video tools and offers no translation (VEED bills it extra and it has no price row); a subtitles row without `translationLanguages` now refuses a `translationLanguage`.
|
|
31
|
+
|
|
32
|
+
### Added
|
|
33
|
+
- Image rows on fal: Krea 3 (`krea-3`), Grok Imagine Image 2.0 (`grok-imagine-image-2.0`), Qwen Image 3 (`qwen-image-3`), Kling Image O3 (`kling-image-o3`), Kling Image 3.0 (`kling-image-3.0`), Recraft V4.1 Flash (`recraft-v4.1-flash`) and SenseNova U1 Infographic (`sensenova-u1-infographic`). One row per model: references switch to the model's edit endpoint; Recraft and SenseNova generate from a prompt only.
|
|
34
|
+
- Upscale row SeedVR2 (`seedvr2-upscale`, fal `fal-ai/seedvr/upscale/image`), offered at 2x and 4x through the new `constraints.scaleFactors`; `resolveImageOptions` refuses any other factor (under `coerce` the row default applies).
|
|
35
|
+
- Video rows: Kling 3.0 Motion Control (pro, standard), Kling O3 and Kling O3 Edit, MiniMax H3 Max Turbo / Insert / Recast / Relight, Happy Horse 1.1 and Happy Horse Video Edit, Vidu Q4 and OmniHuman 2 on fal; Grok Imagine 1.5 on xai-video. Constraints come from each endpoint's published schema; requests the endpoints cannot take are refused before the call.
|
|
36
|
+
- fal video families live in their own mapper files behind exact id sets (`fal-video-families.ts`); `isFalVideoKnown` accepts the new ids.
|
|
37
|
+
- Video tools: FlashVSR upscale and LTX 2.3 SDR to HDR. New `VideoToolOutputFormat` `"exr"`: `VideoToolResult.frames` carries the EXR frame ZIP and `video` the MP4 preview; new tool constraints `maxOutputShortEdge`, `maxOutputFrames`, `minOutputFrames` and `maxOutputFps`. `planVideoTool` applies the rate cap and frame bounds once: `VideoToolOutputSpec.fps` and the new `frames` are what the HDR request sends and what usage reports (a 120 fps source renders on LTX 2.3 HDR at 60 fps, and the 481-frame cap counts frames at that rate).
|
|
38
|
+
- New `VideoRequest.insertWindow` and supported option `"insertWindow"` (H3 Max Insert).
|
|
39
|
+
- New `constraints.requiresOneOf` (video rows): at least one of the listed inputs must be present; `resolveVideoOptions` refuses the request otherwise (`missing_input`). Vidu Q4 sets `["firstFrame", "references"]`, so a prompt-only Vidu request is refused before the call. Loose schema: kits before 0.5.0 ignore it.
|
|
40
|
+
- New `constraints.noPrompt` (video rows whose endpoint has no prompt field): the request needs no prompt; `resolveVideoOptions` and `resolveOptions` accept an empty one and drop a non-empty one with an `option_ignored` warning (`field: "prompt"`) in both `error` and `coerce` mode. MiniMax H3 Max Relight sets it, the only 0.5.0 video row whose saved fal schema has no `prompt`; rows without it are unchanged. Loose schema: kits before 0.5.0 ignore it.
|
|
41
|
+
- Audio rows: Lyria 3.5 (google-genai, Interactions API), ElevenLabs Music v2.5 and MiniMax Music 3 (`audio`), and on fal the new model type `sound-effect`: ElevenLabs Sound Effects v2, Mirelo SFX 2.0 (`mirelo-sfx-2.0`, text to audio) and Mirelo SFX 2.0 video (`mirelo-sfx-2.0-video`, `requires: ["sourceVideo"]`, scores a video). Music stays `audio`; `generateAudio` serves both types and takes `lyrics` (rows with `supports: "lyrics"`, MiniMax Music 3). Data-only `sound-effect` rows are accepted only on fal.
|
|
42
|
+
- New model types and kit methods for the single-purpose canvas nodes, in the shapes the shopos-ui audio nodes call (`ProposedAudioKit`, shopos-ui #3673):
|
|
43
|
+
- `speech` → `generateSpeech({ model, text, voice?, language?, style?, outputFormat? })`: ElevenLabs v4, v4 Turbo, Multilingual v2, Gemini 3.8 Flash TTS and Flash Lite TTS, and MiniMax Speech 2.8 HD (`minimax-speech-2.8-hd`, fal `fal-ai/minimax/speech-2.8-hd`, `voice_setting.voice_id`, default `Wise_Woman`).
|
|
44
|
+
- `voice-change` → `changeVoice({ model, sourceAudio, voice?, outputFormat? })`: ElevenLabs voice changer.
|
|
45
|
+
- `lipsync` → `lipsync({ model, sourceVideo?, image?, audio, resolution? })` → `{ video, usage }`: Sync Lipsync 2, 2 Pro, 3, VEED Lipsync v2 (`requires: ["sourceVideo", ...]`, a video port) and MiniMax H3 Max lip sync (`requires: ["firstFrame", ...]`, an image port). Runs on fal's queue video-job lifecycle (direct or Cloudflare passthrough) and waits for the result; `call.wait` tunes the poll interval and timeout.
|
|
46
|
+
- `dubbing` → `dub({ model, sourceVideo? | sourceAudio?, targetLanguage, outputFormat? })` → `{ video? | audio?, usage }`: ElevenLabs dubbing; a video dubs on the queue job lifecycle, an audio file in one call. Both sources at once are refused.
|
|
47
|
+
- `transcription` → `transcribe({ model, source, language?, timestamps? })` → `{ text, subtitles?, language?, words, usage }`: ElevenLabs Scribe v2; `timestamps: true` adds SubRip `subtitles` built from the word timings.
|
|
48
|
+
- Each method resolves through `MEDIA_METHODS` and its own resolver (`resolveSpeechOptions`, `resolveVoiceChangeOptions`, `resolveLipsyncOptions`, `resolveDubbingOptions`, `resolveTranscriptionOptions`), refuses before any provider call (`missing_input` / `unsupported_option`), validates against the saved fal schema in tests, and emits one usage event. New `Adapter` methods `generateSpeech`, `changeVoice`, `lipsync`, `dub` (`DubbingOutcome`) back them; `adapterServesType` checks them. Data-only rows of these types are accepted only on fal.
|
|
49
|
+
- Video-tool kind `bg-remove` (VEED video background removal, VP9 WebM output). A video-tool result file that fal labels `application/octet-stream` (or not at all) reports its mime from the URL extension (`.webm`, `.mp4`, `.mov`, `.zip`), else from the requested format, so VEED's WebM is `video/webm`; a real provider type is kept.
|
|
50
|
+
- Model type `mask` (`generateImage`): SAM 3 Segmentation (`sam-3-image`, fal `fal-ai/sam-3/image`) takes one image and a prompt, optionally refined by `ImageRequest.points` / `boxes` (new `SupportedOption` `points`, `boxes`; types `MaskPoint`, `MaskBox`), and returns one PNG mask per match (up to 32), billed as one request. A request without the image or the prompt, or with an aspect ratio, size or exact size (dropped with a warning under `coerce`), or with points or boxes that are not whole non-negative pixels, is refused before the call. Data-only mask rows are accepted only on fal's SAM 3 endpoint.
|
|
51
|
+
- Video-tool kind `subtitles`: VEED Subtitles (`veed-subtitles`, fal `veed/subtitles`) burns captions into a clip through `submitVideoTool`. New `VideoToolRequest.subtitles` (`SubtitleOptions`: `preset`, `language`, `translationLanguage`, `position`, `shadow`, `baseline` / `highlighted` font, weight, color) and tool constraints `presets` (first is the default, `simple`), `languages`, `translationLanguages` and `maxSecondsAbove1080p`, all from the endpoint schema. `planVideoTool` refuses an unknown preset or language, a weight outside 100 to 900, a color that is not hex, and sources over 2 hours (1 hour above 1080p). Unset style fields stay out of the body, so the preset keeps its own. Output MP4 (`video/mp4` also when fal labels it octet-stream). Data-only subtitles rows are accepted only on `veed/subtitles`.
|
|
52
|
+
- Lipsync row VEED Lipsync v2 (`veed-lipsync-v2`, fal `veed/lipsync/v2`): source video plus audio, on the same queue job lifecycle as the other lipsync rows.
|
|
53
|
+
- Voices: `constraints.voices: VoiceOption[]` (`{ id, name, gender?, accent?, languages?, description?, previewUrl? }`, `id` is the value sent to the provider, `gender` is `male` / `female` / `neutral`, `previewUrl` https only) and `defaults.voice`. Data in `catalog/voices-*.ts`: ElevenLabs, the 21 names fal's schemas list (default Rachel), with labels and ElevenLabs' public preview URLs joined by name (18 of 21 match); Gemini 3.8 TTS, the schema's 30-voice enum (Google labels for 6); MiniMax Speech 2.8 HD, fal's 17 documented voices and 297 MiniMax system voices whose ids the canvas option format accepts, no previews.
|
|
54
|
+
- Languages: `constraints.languages: LanguageOption[]` (`{ code, name }`, `code` sent to the provider) and `defaults.language` on the ElevenLabs and MiniMax speech rows, Scribe v2 and dubbing (the target language). Data in `catalog/languages-audio.ts`.
|
|
55
|
+
- Output formats: `constraints.outputFormats` and `defaults.outputFormat` on the audio rows whose endpoint lets the caller pick (ElevenLabs v4 / v4 Turbo / voice changer / sound effects / Music v2.5 MP3 formats, Mirelo `wav` / `mp3`); `AudioRequest.outputFormat`. Only MP3 and WAV are listed.
|
|
56
|
+
- New `constraints.references.min` (video rows): the fewest references of each kind a request needs, `{ images?, videos?, audios? }`. `resolveVideoOptions` refuses fewer after trimming (`missing_input`), and the write schema refuses a minimum over the kind's cap; `requires: ["references"]` still means one reference of any kind. Kling 3.0 Motion Control (pro, standard) sets `{ images: 1, videos: 1 }`, OmniHuman 2 `{ images: 1, audios: 1 }`, MiniMax H3 Max Recast and Relight `{ images: 1 }`. Loose schema: kits before 0.5.0 ignore it.
|
|
57
|
+
- `SupportedOption` `style` (Gemini TTS delivery instruction) and `lyrics`; `RequiredInput` `sourceAudio`. Speech rows carry `constraints.prompt.maxChars`; lipsync and voice-change rows carry `constraints.references.audioSeconds` (30 s on the flat lipsync rows, 60 s on the voice changer: the capped inputs their flat prices assume; 5 to 900 s on MiniMax H3). The write schema refuses a default voice, language or format the row does not list, a repeated id, and a non-https preview. A consumer on an older kit quarantines rows that use the new types or vocabulary.
|
|
58
|
+
|
|
59
|
+
### Changed
|
|
60
|
+
- `resolveOptions` (`@shop-os/modelkit/catalog`) resolves every model type the client executes, through the resolver the client runs: speech, voice-change, dubbing, lipsync, transcription, sound-effect, mask and every video-tool kind (`planVideoTool`, refusing what the adapter would) as well as image, upscale, bg-remove, video and audio. Breaking: it takes `{ method, request }` (the client method: `generateSpeech`, `changeVoice`, `lipsync`, `dub`, `transcribe`, ...) and returns `{ method, request, warnings }`, plus the planned `output` for `submitVideoTool`; a row the method does not execute (a text row, a speech row sent to `generateAudio`) is the client's `unsupported_option` error, no longer a plain `Error`. New `MEDIA_METHODS` maps each `ModelType` to its client method; the client and `resolveOptions` both route through it. `generateAudio` and `generateVideo` no longer take speech, voice-change, lipsync or dubbing rows.
|
|
61
|
+
- The exported per-type collections hold every row of their types, and `MODELS` is only their union: `VIDEO_MODELS` now includes the ENG-3271 video families (Vidu Q4, Kling O3, MiniMax H3 Max, Happy Horse, OmniHuman 2, Grok Imagine 1.5 ...), `VIDEO_TOOL_MODELS` includes VEED video background removal and VEED Subtitles (out of `MEDIA_NODE_MODELS`), and `MEDIA_NODE_MODELS` includes SAM 3 (`mask`). `MODELS` keeps the same rows and the same order within each type; only those three rows move in the cross-type order.
|
|
62
|
+
- Lyria cancellation: the call's abort signal reaches the provider request on both Lyria paths whatever `maxRetries` is. Before, Lyria 3.5 (Interactions API) dropped it unless `maxRetries` was set, and the Lyria 3 previews (`generateContent`) never sent it, so aborting only took effect between retry attempts.
|
|
63
|
+
- fal Cloudflare passthrough: the gateway's `sync_mode: true` (inline outputs) no longer overwrites an endpoint's own `sync_mode` field, so sync-lipsync keeps its `"cut_off"` mode instead of sending a value its schema rejects.
|
|
64
|
+
- fal video jobs: `result({ signal })` aborts together with the call's own signal (before, the result signal replaced it), and passthrough jobs pass it to every status and result read.
|
|
65
|
+
- fal video jobs: `poll()` reports `succeeded` only after it has read and decoded the result (direct and Cloudflare passthrough). A job fal marks COMPLETED whose result read fails or holds no output now polls as `failed` (non-retryable error) or `pending` (retryable), and `result()` returns the result `poll()` already decoded instead of fetching it again.
|
|
66
|
+
- `resolveVideoOptions` and `resolveAudioOptions` check `requires` / `requiresOneOf` on the resolved request, after reference trimming and conditioning coercion, so an input the row cannot take no longer satisfies them: Vidu Q4 with only reference videos (cap 0) is now refused with `missing_input` instead of resolving to a request with neither a first frame nor a usable reference. A request missing a required input can now report another refusal first when it has one.
|
|
67
|
+
- fal image families: `exactSize` (or `size: "WxH"`) is no longer dropped. Krea 3 and Qwen Image 3 send it as `image_size` `{ width, height }` within each schema's bounds (Krea: sides 256 to 3200 px in multiples of 16, at most 4,220,160 px; Qwen: 512² to 2048² px in total) and bill it at the smallest tier that holds it (1K up to 1024², else 2K); a size outside those bounds is refused before the call (`unsupported_option`). Grok Imagine Image 2.0, Kling Image O3 / 3.0, Recraft V4.1 Flash, SenseNova U1 Infographic and SeedVR2 take tiers or ratios only and refuse an `exactSize`.
|
|
68
|
+
- H3 Max Insert: the insert window follows the saved fal schema, `start_time` 1.625 to 60 s and `resume_time` over 1.625 (exclusive) up to 60 s, resuming at or after the start. Before, the planner required a resume time of at least 2.625 s and refused valid windows such as 1.625 to 2 s.
|
|
69
|
+
- LTX 2.3 HDR: `planVideoTool` reports the rendered length, `frames / fps`, as `VideoToolOutputSpec.durationSeconds`, next to the `fps` and `frames` the request sends, and the usage event bills that length. Rounding to whole frames and the 9-frame minimum now reach the metadata and the billed seconds (a 0.2 s clip at 30 fps renders and bills 0.3 s); before, both kept the source length. Rows without frame limits keep the source length.
|
|
70
|
+
- `resolveOptions` and the per-type resolvers apply the endpoint rules the adapters used to apply alone (`resolve-provider-rules.ts`), so preflight refuses or coerces exactly what the client does: SAM 3 (`mask`) needs a prompt, takes no aspect ratio / size / exact size (`unsupported_option`; dropped with an `option_ignored` warning under `coerce`) and only whole non-negative pixel points and boxes; MiniMax Music 3 needs `lyrics` or `instrumental: true` (`missing_input`); xai-video rows refuse 1080p with references or a last frame (lowered to 720p with a `resolution_coerced` warning under `coerce`). The adapters keep the same checks for callers that skip the resolver.
|
|
71
|
+
- `resolveOptions` and the image / video resolvers also apply the Krea 3 and Qwen Image 3 `exactSize` bounds and the H3 Max Insert window rules, through the same functions the fal adapter calls (`resolve-image-rules.ts`, `resolve-provider-rules.ts`): an out-of-bounds exact size or a missing or out-of-range insert window is refused at preflight with the client's error, and an accepted exact size sets the resolved `size` to the tier it bills at (`1536x1024` resolves to `2K`, as the usage event reports), with a `size_coerced` warning when the caller's own `size` differs.
|
|
72
|
+
- `resolveOptions` refuses an `exactSize` on the tier-only fal image families (Grok Imagine Image 2.0, Kling Image O3 / 3.0, Recraft V4.1 Flash, SenseNova U1 Infographic, SeedVR2) as the adapter does (`unsupported_option`, both modes): neither of Grok's saved schemas (text-to-image, edit) has an `image_size` field. Before, `1536x1024` resolved and only the call failed. The family specs, exact-size bounds included, live in `resolve-image-rules.ts`, which the resolver and the fal image planner both read.
|
|
73
|
+
- Kling 3.0 Motion Control (pro, standard): `resolveOptions` refuses a request without both the character image and the motion video (`missing_input`, through `references.min`) and one with a start or end frame (`unsupported_option`), as the adapter does. Before, one reference of either kind passed preflight.
|
|
74
|
+
- Kling O3: the cap of 7 images, start and end frames included, is checked in `resolveOptions` (`unsupported_option`, both modes) by the rule the adapter calls (`resolve-video-rules.ts`). Before, 6 reference images and both frames passed preflight and failed at the call.
|
|
75
|
+
- `resolveOptions` applies every 0.5.0 fal family planner's acceptance rules through the functions the planners call (`resolve-image-rules.ts`, `resolve-video-rules.ts`, `resolve-provider-rules.ts`), so preflight refuses, in both modes and before any provider call, what execution refuses: the fal image families check the output format, the image count, a required prompt and its length, `aspectRatio: "adaptive"` where the endpoints have no auto size (Krea 3, Qwen Image 3, Kling Image 3.0, Recraft, SenseNova) and Grok's quality; the video families check a start or end frame on endpoints without one (Kling O3 Edit, H3 Max Insert / Recast / Relight, Happy Horse Video Edit, OmniHuman 2), a required prompt (Kling O3 and O3 Edit, H3 Max Turbo, Happy Horse text and reference modes, Happy Horse Video Edit, Vidu Q4 with references), H3 Max Turbo's 2 s soundtrack minimums and OmniHuman 2's 4 to 300 s audio; fal music and sound-effect rows refuse references, a source video on text rows and an empty prompt (Mirelo video excepted). Vidu Q4's planner reports audio-only references as `missing_input`, as the resolver does (was `unsupported_option`). Two adjustments the adapters made alone run in the resolver too, with the same warnings: Vidu Q4 image-to-video turns `generateAudio` off (`option_ignored`), and H3 Max Turbo with an end frame alone drops the ratio (`ratio_dropped` only for the caller's own ratio; before, the adapter also warned for the row default).
|
|
76
|
+
- `CatalogModelSchema` (and the read path's per-row parse) validate `defaults.points`, `defaults.boxes` and `defaults.insertWindow`: points and box corners must be whole, non-negative pixels, boxes need `xMax > xMin` and `yMax > yMin` (the SAM 3 request rule, checked by the same functions), and the insert window must be `{ startSeconds: number, resumeSeconds: number }`. Its bounds stay with the request path (H3 Max Insert's window check), so the row schema refuses no window a request could use. Before, a malformed default such as `points: "invalid"` passed validation and every mask request then threw a raw `TypeError`.
|
|
77
|
+
- `resolveImageOptions` refuses references on an image row whose `references.images` is 0 (`unsupported_option`); under `coerce` they are dropped with a `references_trimmed` warning. Recraft V4.1 Flash and SenseNova U1 Infographic declare `references: { images: 0 }`, so consumers cap them at zero instead of treating them as uncapped.
|
|
78
|
+
- xai-video: up to the row's reference cap (14 on 1.5; past the SDK's 7 through the REST body), a pinned `last_frame` on rows that support `lastFrame`, and 1080p refused (coerced to 720p under `coerce`) with references or a last frame.
|
|
79
|
+
|
|
80
|
+
### Upgrade notes
|
|
81
|
+
- Bump every consumer to 0.5.0 before pro-backend seeds these rows: an older kit sends unknown fal image ids through the Muse fallback with the wrong body, and quarantines rows that carry the new vocabulary.
|
|
82
|
+
|
|
12
83
|
## [0.4.7] - Unreleased
|
|
13
84
|
|
|
14
85
|
### Fixed
|
package/README.md
CHANGED
|
@@ -39,6 +39,14 @@ const song = await kit.generateAudio({
|
|
|
39
39
|
});
|
|
40
40
|
const score = await kit.generateAudio({ model: "mmaudio", prompt, sourceVideo: { url } });
|
|
41
41
|
|
|
42
|
+
// Single-purpose audio nodes: voices, languages and formats are ids the row lists
|
|
43
|
+
// (`constraints.voices`, `constraints.languages`, `constraints.outputFormats`).
|
|
44
|
+
const speech = await kit.generateSpeech({ model: "elevenlabs-v4", text, voice: "Aria" });
|
|
45
|
+
const changed = await kit.changeVoice({ model: "elevenlabs-voice-changer", sourceAudio: { url } });
|
|
46
|
+
const synced = await kit.lipsync({ model: "lipsync-2", sourceVideo: { url }, audio: { url } });
|
|
47
|
+
const dubbed = await kit.dub({ model: "elevenlabs-dubbing", sourceVideo: { url }, targetLanguage: "es" });
|
|
48
|
+
const transcript = await kit.transcribe({ model: "elevenlabs-scribe-v2", source: { url }, timestamps: true });
|
|
49
|
+
|
|
42
50
|
const clip = await kit.prepareMediaPart(
|
|
43
51
|
"gemini-2.5-flash",
|
|
44
52
|
{ bytes, mime: "video/mp4" },
|
|
@@ -57,8 +65,9 @@ ark adapter (Seedance) supports it today; every other adapter throws `unsupporte
|
|
|
57
65
|
|
|
58
66
|
### Video tools
|
|
59
67
|
|
|
60
|
-
`video-tool` rows (Topaz upscale, frame interpolation, SDR to HDR on fal) take
|
|
61
|
-
return a clip with no prompt.
|
|
68
|
+
`video-tool` rows (Topaz, FLUX and FlashVSR upscale, frame interpolation, SDR to HDR on fal) take
|
|
69
|
+
a clip and return a clip with no prompt. An `exr` HDR row (LTX 2.3) returns a ZIP of 16-bit EXR
|
|
70
|
+
frames as `result.frames` and an MP4 preview as `result.video`. They run as provider jobs the caller tracks, because a 30 s
|
|
62
71
|
4K job can take over 40 minutes:
|
|
63
72
|
|
|
64
73
|
```ts
|
|
@@ -74,6 +83,31 @@ const out = await kit.videoToolResult(sub, { webhook: verifiedBody }); // or pol
|
|
|
74
83
|
`source` carries the caller's measured `width`, `height`, `durationSeconds` and `fps`; the kit
|
|
75
84
|
never measures. Verifying the webhook signature is the caller's job.
|
|
76
85
|
|
|
86
|
+
### 3D models
|
|
87
|
+
|
|
88
|
+
`model3d` rows (Trellis 2, Hunyuan3D, Tripo, Rodin, Meshy, Hi3D, SAM 3D on fal) take a prompt
|
|
89
|
+
and/or photos of an object and return a GLB. They run as provider jobs too, the same way:
|
|
90
|
+
|
|
91
|
+
```ts
|
|
92
|
+
const sub = await kit.submitModel3d({ model: "tripo-h3.1", image, views: { left, back, right } });
|
|
93
|
+
// store `sub` (plain JSON); later: model3dStatus(sub), model3dResult(sub), cancelModel3d(sub)
|
|
94
|
+
const out = await kit.model3dResult(sub); // out.glb.url, out.thumbnail?.url, out.sku
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
What a row takes (text, required image or prompt, views, texture, PBR, seed, a priced
|
|
98
|
+
resolution) is data on `constraints.model3d`; the adapter picks the fal endpoint and builds the
|
|
99
|
+
body per `family`. The catalog schema checks a spec against what its family's mapping reads
|
|
100
|
+
(the endpoints a request it accepts goes to, its views, texture, PBR, seed and resolution
|
|
101
|
+
options, and a `maxImages` no larger than the image slots it sends: `MODEL3D_IMAGE_SLOTS`, 5 for
|
|
102
|
+
`unordered` views with `views.top` as the fifth, 4 for `named` and `ordered`), so a row the
|
|
103
|
+
adapter could not run or price is rejected on load and on write. The result's GLB and thumbnail
|
|
104
|
+
URLs are https or the call fails (`provider_rejected`), and the usage event is one request
|
|
105
|
+
(`unit: "requests"`), billed at the flat price of its SKU.
|
|
106
|
+
`model3dInputRefusal(spec, wired)` says why inputs cannot run (also checked
|
|
107
|
+
before submit), and `model3dPricingSku(row, options, inputs)` names the flat price a request
|
|
108
|
+
bills under; `model3dSkus(row)` lists them all, and pro-backend keeps a `credit_pricing_flat`
|
|
109
|
+
row for each. All three are in the browser-safe catalog entry.
|
|
110
|
+
|
|
77
111
|
Keys come from `process.env` by default: bare names for pro (`OPENAI_API_KEY`, `FAL_KEY`, `ARK_API_KEY` ...), the `ENTERPRISE_` prefix for enterprise and the `SLOOSH_` prefix for sloosh. Pass `keys: { pro, enterprise, sloosh }` to override. `tier: "sloosh"` uses the sloosh keys only: a missing key is a `ModelKitError` with code `auth`, never a fallback to pro.
|
|
78
112
|
|
|
79
113
|
## Browser-safe catalog
|
|
@@ -84,7 +118,9 @@ import { Catalog, MODELS, resolveOptions } from "@shop-os/modelkit/catalog";
|
|
|
84
118
|
|
|
85
119
|
The catalog subpath carries the model rows (`MODELS` plus the per-type
|
|
86
120
|
arrays), `Catalog`, the option resolvers (`resolveOptions` and the per-type
|
|
87
|
-
resolvers), `ModelKitError` and the catalog types.
|
|
121
|
+
resolvers), `ModelKitError` and the catalog types. `resolveOptions(row, { method, request })`
|
|
122
|
+
resolves a request exactly as that client method would (`MEDIA_METHODS` lists the methods
|
|
123
|
+
each model type runs on), so a picker can validate before it calls the kit. It includes no adapters, no
|
|
88
124
|
client, no keys, no text helpers and no provider SDKs, and builds for
|
|
89
125
|
`es2022` / neutral so it bundles straight into the browser.
|
|
90
126
|
|
|
@@ -267,7 +303,7 @@ const defaultKit = createDefaultModelKit({ keys: { pro: keysFromEnv(process.env)
|
|
|
267
303
|
| ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
268
304
|
| Display | every active row carries `display` (provider, description, logo) and `capabilities`; UIs derive their pickers from `kit.catalog.list()` and keep only an allowlist and ordering of their own. |
|
|
269
305
|
| Ids | `id` is the pro-backend pricing key. Every historical id (`nb-pro`, `veo-3.1-generate-001`, `kling-v2.5-turbo-pro`) is an alias. |
|
|
270
|
-
| Options | Image tiers are `512` (Gemini 3.1 Flash only), `1K`, `2K`, `4K`. `resolveOptions` fills catalog defaults, errors on an unsupported aspect ratio, size, resolution or duration (`onUnsupported: "coerce"` snaps to the nearest supported value with a warning instead; since 0.1.6 ratio and duration honour this too, so pass `"coerce"` where the old code snapped silently), trims references (audio first, then video), and drops options the model ignores with a warning. Audio requests go through the same resolver and take `abortSignal` (a row without `supports` supports nothing). `aspectRatio: "adaptive"` means "send no ratio": it is never coerced or replaced by the row default, and Gemini, Muse, Kling and Kling O1 omit the field so the model infers it from the references. Every coercion is also logged once per call through `logger.warn`. Gemini image `safetySettings` and `temperature` are sent only when the caller passes them. A reference with a `label` gets a text part with that label right before the image on Gemini image and Omni ("Image 2:", "Here is the brand logo:"). Topaz takes `upscale` (`enhanceModel`, `subjectDetection`, `faceEnhancement` and its creativity/strength) next to `scaleFactor`/`outputFormat`; Seedance takes `watermark`; MiniMax takes `lastFrame` and `safetyChecker`; Seedream 4.5 goes to `4K` and `9:21`. |
|
|
306
|
+
| Options | Image tiers are `512` (Gemini 3.1 Flash only), `1K`, `2K`, `4K`. `resolveOptions` fills catalog defaults, errors on an unsupported aspect ratio, size, resolution or duration (`onUnsupported: "coerce"` snaps to the nearest supported value with a warning instead; since 0.1.6 ratio and duration honour this too, so pass `"coerce"` where the old code snapped silently), trims references (audio first, then video), and drops options the model ignores with a warning. Audio requests go through the same resolver and take `abortSignal` (a row without `supports` supports nothing). `aspectRatio: "adaptive"` means "send no ratio": it is never coerced or replaced by the row default, and Gemini, Muse, Kling and Kling O1 omit the field so the model infers it from the references. Every coercion is also logged once per call through `logger.warn`. The fal family rows of 0.5.0 also run the endpoint rules their planners run (`resolve-image-rules.ts`, `resolve-video-rules.ts`, `resolve-provider-rules.ts`): what an endpoint cannot take is refused in both modes, before the call. A video row's `references.min` (`{ images?, videos?, audios? }`) names the references of each kind a request needs. Gemini image `safetySettings` and `temperature` are sent only when the caller passes them. A reference with a `label` gets a text part with that label right before the image on Gemini image and Omni ("Image 2:", "Here is the brand logo:"). Topaz takes `upscale` (`enhanceModel`, `subjectDetection`, `faceEnhancement` and its creativity/strength) next to `scaleFactor`/`outputFormat`; Seedance takes `watermark`; MiniMax takes `lastFrame` and `safetyChecker`; Seedream 4.5 goes to `4K` and `9:21`. |
|
|
271
307
|
| Keys | Enterprise keys win only when every key the adapter (and its route, e.g. the cloudflare token for a vendor that really goes through the gateway) needs is set on the enterprise tier; a fallback to pro keys is logged once per call. |
|
|
272
308
|
| Routing | Cloudflare AI Gateway for text only. Never for Gemini images or Omni. fal passthrough for geo-blocked regions; the fal direct route submits to the queue and polls, so `job.result({ timeoutMs, signal })` is live. |
|
|
273
309
|
| Retry | One classifier, one policy. Never on content blocks, aborts, or 4xx except 429; SDKs with their own retry (Replicate) run under a single kit attempt; a 429 on a status poll keeps polling unless `CallContext.maxRetries` is `0`. See durable media execution above. |
|
|
@@ -277,7 +313,9 @@ const defaultKit = createDefaultModelKit({ keys: { pro: keysFromEnv(process.env)
|
|
|
277
313
|
|
|
278
314
|
## Add a model
|
|
279
315
|
|
|
280
|
-
1. Add a row to `src/catalog/{image,video,video-tools,text,other}.ts`.
|
|
316
|
+
1. Add a row to `src/catalog/{image,video,video-families,video-tools,model3d,text,other}.ts`. A new fal
|
|
317
|
+
video family gets its own mapper file (`src/adapters/fal-<family>.ts`) listed in
|
|
318
|
+
`fal-video-families.ts`, which is what makes its exact ids pass the data-only check. Copy `constraints` from the provider docs.
|
|
281
319
|
2. Run `pnpm test`. The catalog test checks id and alias uniqueness.
|
|
282
320
|
3. Bump `version` in `package.json` in a PR that touches only `packages/modelkit/**`.
|
|
283
321
|
Merging it to `dev` runs `modelkit-publish.yml`, which publishes through the npm
|