@mindstudio-ai/agent 0.1.150 → 0.1.152

package/llms.txt

@@ -285,8 +285,10 @@ Analyze an image using a vision model based on a text prompt.
 
 #### analyzeVideo
 Analyze a video using a video analysis model based on a text prompt.
-- Uses the configured video analysis model to generate a text analysis of the video.
-- The prompt should describe what to look for or extract from the video.
+- The prompt describes what to look for or extract from the video.
+- The default model is gemini-3.8-flash. Any Gemini chat model reads video (the 3.x Flash and Flash Lite models, gemini-3.1-pro, and the 2.5 family), as does TwelveLabs' pegasus-1.6; set one with `videoAnalysisModelOverride`, with `temperature` and `maxResponseTokens` beside `model`.
+- `videoUrl` is a video file's URL (an app file, a Google Drive share link, or any public URL). Gemini also reads a public YouTube link; Pegasus doesn't.
+- Length: Pegasus takes 1 second to 1 hour. Gemini takes about 3 hours at its default resolution; the 3.x models read a second of video as about 66 tokens and the 2.5 models as about 258, which is what a long video costs.
 - Input: `{ prompt: string, videoUrl: string, videoAnalysisModelOverride?: { model: string, config?: object } | { model: string, temperature: number, maxResponseTokens: number, ignorePreamble?: boolean, userMessagePreprocessor?: { dataSource?: string, messageTemplate?: string, maxResults?: number, enabled?: boolean, shouldInherit?: boolean }, preamble?: string, multiModelEnabled?: boolean, editResponseEnabled?: boolean, promptCaching?: boolean, config?: object } }`
 - Output: `{ analysis: string }`
 
@@ -332,10 +334,11 @@ Create a draft email in the connected Gmail account.
 
 #### decomposeImage
 Split an image into a base image and transparent layers.
-- Uses Seedream 5.0 Flash layer decomposition. There is no model picker.
-- An optional prompt can name the objects or groupings to separate.
-- The output is an array of hosted image URLs. PNG output keeps layer transparency.
-- Input: `{ image: string, prompt?: string, intermediateAsset?: boolean }`
+- Uses Seedream 5.0 Flash layer decomposition by default, and Seedream 5.0 Pro when type is "advanced", which separates more cleanly and costs more (and twice as much at 2K).
+- `resolution` is "1k", "1.5k" or "2k" (the default).
+- An optional prompt names the objects or groupings to separate. Left out, the model separates the main elements itself: text, subjects, decorations and background.
+- The output is an array of hosted PNG image URLs: first the base image at full size, then each separated element as a transparent layer cropped to the element. The layers come with no position, so placing one back over the base means finding where it sat. Flash returned 13 layers for one photo and Pro 16.
+- Input: `{ image: string, prompt?: string, type?: "standard" | "advanced", resolution?: "1k" | "1.5k" | "2k", intermediateAsset?: boolean }`
 - Output: `{ imageUrls: string[] }`
 
 #### deleteDataSource
@@ -395,6 +398,27 @@ Generate or enhance a video generation prompt using a language model. Optionally
 - Input: `{ initialPrompt: string, includeNegativePrompt: boolean, negativePromptDestinationVariableName?: string, systemPrompt: string, modelOverride?: unknown }`
 - Output: `{ prompt: string, negativePrompt?: string }`
 
+#### evaluateDecision
+Ask structured questions about text, JSON, or images and get calibrated answers from a decision model. The model returns a probability, a selected option, or a score. It does not write a reply. Each question is a noul (yes or no, as a probability from 0 to 1), a choice (one option from a set you define, with a probability for each), or a score (a rating against ordered levels). Put every question in the same call. The question id is yours: answers come back under the same ids. The model defaults to Jev (jev-latest).
+- state is the content to judge: a string, or a JSON object or array (a record, a chat log, application state).
+- questions is a map. The key is an id you choose. It is not shown to the model. The answer comes back under the same id.
+- A noul asks a yes/no question and returns noul, a probability from 0 (no) to 1 (yes). Optional criteria.true and criteria.false say what a yes and a no mean.
+- A choice picks one option. criteria is a map of option id to a short description (or null). At most 255 options. The answer includes the winning choice, a probability for every option, and confidence.
+- A score rates the state on an ordered rubric. criteria is 2 to 10 level descriptions, low to high. The answer's score can land between levels, and includes a probability per level plus confidence.
+- An answer of type refusal means the model declined that question (GPT-6 Luna Decisions can).
+- instructions may be a string, or an object that keeps the question in one field and reference data in the others. Refer to those fields by name in backticks.
+- Send every question you need in one call. The model reads the state once and answers them together.
+- Probabilities from different models are not comparable: tune a threshold against the model, and the version, it will run on.
+- decisionModelOverride.model picks the model:
+- jev-latest (the default) is TypeSafe's current stable release, text only, up to 64k tokens. jev-preview tracks TypeSafe's newest release. Pin a version such as jev-1.13.0 when a threshold was tuned against that version.
+- clef and clef-flash run Cloudflare Clef, which also reads up to 4 PNG, JPEG, or WebP images, each at most 4 MiB and 8 MiB together, and takes at most 64 questions, each id using only letters, digits, underscore, dot, and hyphen, up to 100 characters. clef is the larger model, clef-flash the faster one. @cf/cloudflare/clef and @cf/cloudflare/clef-flash name the same two.
+- gpt-6-luna-decisions runs OpenAI's Decisions API (in beta), which reads up to 922k tokens and PNG, JPEG, WebP, or GIF images of up to 20 MB each. It takes up to 200 questions, and a choice needs at least 2 options. gpt-6-luna names it too.
+- images is for a model that reads them. Pass URLs or data URLs. A model that reads none refuses the call.
+- The response model field is the version that answered.
+- Billed per input token. Output tokens are free. Each model has its own rate, and GPT-6 Luna Decisions bills a prompt over 272k tokens at twice its rate.
+- Input: `{ state: string | object | unknown[], questions: object, images?: string[], decisionModelOverride?: { model: string, config?: object } }`
+- Output: `{ model: string, answers: object, usage: { input_tokens: number, output_tokens: number } }`
+
 #### extractAudioFromVideo
 Extract audio MP3 from a video file
 - Input: `{ videoUrl: string }`
@@ -404,12 +428,19 @@ Extract audio MP3 from a video file
 Download a file from a URL and extract its text content. Supports PDFs (including scanned/image-based PDFs via OCR), plain text files, and other document formats.
 - Best suited for PDFs and raw text/document files. For web pages, use the scrapeUrl step instead.
 - Handles both text-layer PDFs and image-based/scanned PDFs (e.g. Figma/Canva exports, scanned documents). Image-based PDFs are processed with OCR automatically — there is no need to convert PDF pages to images first.
+- Plain text, code, HTML, spreadsheets and Office documents (docx, pptx, odt, rtf, epub) are read without a model and cost nothing. PDFs and images go to the extraction model, billed per page.
 - Accepts a single URL, a comma-separated list of URLs, or a JSON array of URLs.
 - Nothing is published. One of the app's own files — including a private one, or a share link — is read where it sits; any other URL is downloaded and staged privately for the extraction model, then discarded when extraction finishes.
-- Optionally set `model` to a specific document-extraction model (`mistral-ocr-latest`, `llamaparse`, `google-document-ai`); omit to use the platform default.
+- `text` is each document's text: a string for one URL, an array for several. `documents` has each document page by page, with its page numbers and the pages billed. Set `includeBlocks` to add each page's typed blocks (title, text, table, image, header, footer…) with their bounding boxes, and the page's size.
+- A file that can't be read (unreachable, an unsupported type, past the model's limits, or refused by the model) fails the step with the reason.
+- `extractionModelOverride.model` picks the model; omit it to use the workspace's default:
+- `mistral-ocr-4.1` (the default): PDFs and images up to 50 MB and 1,000 pages, with typed blocks.
+- `mistral-document-ai-2512`: Mistral Document AI on Microsoft Foundry, up to 30 MB and 30 pages, with typed blocks.
+- `llamaparse`: LlamaParse's agentic mode, up to 100 MB and 1,000 pages, markdown without blocks.
+- `google-document-ai`: Google Document AI OCR, up to 500 pages, where the deployment has set it up.
 - Maximum file size is 50MB per URL.
-- Input: `{ url: string | string[], model?: string }`
-- Output: `{ text: string | string[] }`
+- Input: `{ url: string | string[], extractionModelOverride?: { model: string, config?: object }, model?: string, includeBlocks?: boolean }`
+- Output: `{ text: string | string[], documents: { url: string, text: string, pages: { pageNumber: number, markdown: string, blocks?: unknown[], dimensions?: { width: number, height: number, dpi?: number } }[], pagesProcessed: number }[] }`
 
 #### fetchDataSourceDocument
 Legacy v1 data sources. v2 apps should use `dataSources.defineDataSource()` from @mindstudio-ai/agent instead, which owns parsing, chunking, embedding and citations. Fetch the full extracted text contents of a document in a data source.
@@ -427,8 +458,12 @@ Fetch recent message history from a Slack channel.
 
 #### generate3dModel
 Generate a 3D model using a 3D generation model.
-- Text-to-3D models use the prompt field.
-- Image-to-3D and multi-view models take image URLs through the selected model's configuration inputs.
+- The default model is wavespeed-tripo3d-h3.1 (Tripo v3.1), which makes a model from the prompt, from one `image`, or from `images`: 2–4 views of the same object, in order front, left, back, right.
+- Models and what they take, set in `threeDModelOverride.config`:
+- From text or an image: wavespeed-tripo3d-h3.1, wavespeed-meshy7.1 (also `images`, 1–4 views), wavespeed-hunyuan3d-v3 (an `image`, plus optional `back_image_url`, `left_image_url`, `right_image_url`), wavespeed-hunyuan-3d-v3.1-rapid, wavespeed-tripo3d-p2, wavespeed-rodin-v2.5 (`images`, 1–5).
+- From an image only: trellis-2, wavespeed-meshy7 (or `images`, 1–4 views), wavespeed-tripo3d-v2.5, seed3d-2.0, wavespeed-sam-3d-objects (with optional `mask_images`).
+- From views: wavespeed-hi3d-v3.0 (`front_image`, and optional `back_image`, `left_image`, `right_image`).
+- `$mode` in the config picks a model's mode (`text-to-3d`, `image-to-3d`, or its multi-view mode). Left out, `images` picks the multi-view mode and `image` picks image-to-3d; with neither, the model's default mode. A text mode needs the prompt; in the others the prompt is optional guidance. A prompt longer than the model reads is refused, not cut.
 - The output is standardized as a GLB URL plus optional FBX/OBJ/USDZ, thumbnail, texture maps, and provider task ID.
 - Input: `{ prompt?: string, intermediateAsset?: boolean, threeDModelOverride?: { model: string, config?: object } }`
 - Output: `{ prompt?: string, glbUrl: string, fbxUrl?: string, objUrl?: string, usdzUrl?: string, thumbnailUrl?: string, textureUrls?: object[], providerTaskId?: string, resolvedConfig?: object }`
@@ -461,14 +496,23 @@ Generate a lip sync video from audio and an image or video.
 
 #### generateMusic
 Generate an audio file from provided instructions (text) using a music model.
-- The text field contains the instructions (prompt) for the music generation.
+- `text` describes the song: genre, mood, instruments, tempo, vocals, and for Lyria its structure (`[Verse]`, `[Chorus]`, timestamps such as `[0:00-0:20] Intro`). A text longer than the model reads is refused, not cut.
+- The default model is elevenlabs-music_v2.5. Options go in `musicModelOverride.config`, and differ by model:
+- elevenlabs-music_v2.5, elevenlabs-music_v2: `musicLengthMs`, the song's length from 3,000 to 600,000 (the model chooses one from the text otherwise, and holds a length it's given), and `forceInstrumental: true` for no vocals.
+- lyria-3.5: songs of up to about three minutes, their length steered from the text; `image`, the URL of an image for the song to take its mood and subject from. lyria-3 makes 30-second clips, lyria-3-pro songs of up to about three minutes.
+- minimax-music-3.0: `lyrics` (written from the text when left out), `is_instrumental: true`, `bitrate`, `sample_rate`. minimax-music-2.5 requires `lyrics`.
 - In foreground mode, the audio is displayed to the user. In background mode, the URL is saved to a variable.
 - Input: `{ text: string, intermediateAsset?: boolean, musicModelOverride?: { model: string, config?: object } }`
 - Output: `{ audioUrl: string }`
 
 #### generateSound
 Generate a sound effect from a text description using a sound model.
-- The text field contains the description of the sound effect to generate.
+- `text` describes the sound effect to generate.
+- The default model is elevenlabs-sound_v2. Options go in `soundModelOverride.config`:
+- `durationSeconds` — the length, from 0.5 to 30. Left out, the model picks one from the text, and the effect is billed as 5 seconds whatever its length.
+- `loop: true` — a sound that repeats with no audible start or end, for ambience and backgrounds.
+- `promptInfluence` — how literally the text is followed, from 0 to 1 (default 0.3).
+- `outputFormat` — MP3 (`mp3_44100_128` by default, up to `mp3_44100_192`) or Opus (`opus_48000_32` to `opus_48000_192`).
 - In foreground mode, the audio is displayed to the user. In background mode, the URL is saved to a variable.
 - Input: `{ text: string, intermediateAsset?: boolean, soundModelOverride?: { model: string, config?: object } }`
 - Output: `{ audioUrl: string }`
@@ -1432,17 +1476,8 @@ Call any QuickBooks Online Accounting API route using a connected company.
 ### TypeSafe
 
 #### typesafeEvaluate
-Ask structured questions about text, JSON, or images and get calibrated answers. The model returns a probability, a selected option, or a score. It does not write a reply. Each question is a noul (yes or no, as a probability from 0 to 1), a choice (one option from a set you define, with a probability for each), or a score (a rating against ordered levels). Put every question in the same call. The question id is yours: answers come back under the same ids. The model defaults to Jev (jev-latest). Pass clef or clef-flash to run Cloudflare Clef with the same questions and answers. Clef and Clef Flash can also read up to 4 images.
-- state is the content to judge: a string, or a JSON object or array (a record, a chat log, application state).
-- images is for Clef and Clef Flash. Pass up to 4 PNG, JPEG, or WebP images as URLs. Each file is at most 4 MiB, and 8 MiB together. A call on Jev that includes images is rejected.
-- questions is a map. The key is an id you choose. It is not shown to the model. The answer comes back under the same id.
-- A noul asks a yes/no question and returns noul, a probability from 0 (no) to 1 (yes). Optional criteria.true and criteria.false say what a yes and a no mean.
-- A choice picks one option. criteria is a map of option id to a short description (or null). At most 255 options. The answer includes the winning choice, a probability for every option, and confidence.
-- A score rates the state on an ordered rubric. criteria is 2 to 10 level descriptions, low to high. The answer's score can land between levels, and includes a probability per level plus confidence.
-- instructions may be a string, or an object that keeps the question in one field and reference data in the others. Refer to those fields by name in backticks.
-- Send every question you need in one call. The model reads the state once and answers them together.
-- model defaults to jev-latest, the current stable TypeSafe release. jev-preview tracks the newest TypeSafe release. Pin a version such as jev-1.13.0 when a threshold was tuned against that version. clef and clef-flash run Cloudflare Clef on the same questions and answers. clef is the larger model. clef-flash is the faster one. @cf/cloudflare/clef and @cf/cloudflare/clef-flash name the same two. Clef accepts at most 64 questions, and a question id may use only letters, digits, underscore, dot, and hyphen, up to 100 characters. The response model field is the version that answered.
-- Billed per input token. Output tokens are free. Clef and Clef-flash have their own rates.
+Use evaluateDecision: the same questions, answers and models, with decisionModelOverride in place of model. evaluateDecision's earlier name, which still runs: ask structured questions about text, JSON, or images and get calibrated answers from a decision model.
+- Runs exactly as evaluateDecision does, with the same questions, answers, models and limits (see its notes), and model in place of decisionModelOverride.model.
 - Input: `{ state: string | object | unknown[], questions: object, model?: string, images?: string[] }`
 - Output: `{ model: string, answers: object, usage: { input_tokens: number, output_tokens: number } }`
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@mindstudio-ai/agent",
-  "version": "0.1.150",
+  "version": "0.1.152",
   "description": "TypeScript SDK for MindStudio direct step execution",
   "type": "module",
   "main": "./dist/index.js",