@mindstudio-ai/agent 0.1.148 → 0.1.150

package/llms.txt

@@ -320,7 +320,7 @@ Legacy v1 data sources. v2 apps should use `dataSources.defineDataSource()` from
 - The data source is created empty — use the "Upload Data Source Document" block to add documents.
 - Returns the new data source ID which can be used in subsequent blocks.
 - Input: `{ name: string }`
-- Output: `unknown`
+- Output: `{ dataSourceId: string, name: string }`
 
 #### createGmailDraft
 Create a draft email in the connected Gmail account.
@@ -344,7 +344,7 @@ Legacy v1 data sources. v2 apps should use `dataSources.defineDataSource()` from
 - The Milvus partition is cleaned up asynchronously by a background cron job.
 - The data source must belong to the current app version.
 - Input: `{ dataSourceId: string }`
-- Output: `unknown`
+- Output: `{ dataSourceId: string }`
 
 #### deleteDataSourceDocument
 Legacy v1 data sources. v2 apps should use `dataSources.defineDataSource()` from @mindstudio-ai/agent instead, which owns parsing, chunking, embedding and citations. Delete a single document from a data source.
@@ -352,7 +352,7 @@ Legacy v1 data sources. v2 apps should use `dataSources.defineDataSource()` from
 - Requires both the data source ID and document ID.
 - After deletion, reloads vectors into Milvus so the data source reflects the change immediately.
 - Input: `{ dataSourceId: string, documentId: string }`
-- Output: `unknown`
+- Output: `{ dataSourceId: string, documentId: string }`
 
 #### detectChanges
 Detect changes between runs by comparing current input against previously stored state. Routes execution based on whether a change occurred.
@@ -417,7 +417,7 @@ Legacy v1 data sources. v2 apps should use `dataSources.defineDataSource()` from
 - The document must have been successfully processed (status "done").
 - Also returns document metadata (name, summary, word count).
 - Input: `{ dataSourceId: string, documentId: string }`
-- Output: `unknown`
+- Output: `{ documentId: string, name: string, content: string, summary: string, numWords: number }`
 
 #### fetchSlackChannelHistory
 Fetch recent message history from a Slack channel.
@@ -454,17 +454,17 @@ Generate an image from a text prompt using an AI model.
 - Output: `{ imageUrl: string | string[] }`
 
 #### generateLipsync
-Generate a lip sync video from provided audio and image.
+Generate a lip sync video from audio and an image or video.
 - In foreground mode, the video is displayed to the user. In background mode, the URL is saved to a variable.
 - Input: `{ intermediateAsset?: boolean, addWatermark?: boolean, lipsyncModelOverride?: { model: string, config?: object } }`
-- Output: `unknown`
+- Output: `{ videoUrl: string }`
 
 #### generateMusic
 Generate an audio file from provided instructions (text) using a music model.
 - The text field contains the instructions (prompt) for the music generation.
 - In foreground mode, the audio is displayed to the user. In background mode, the URL is saved to a variable.
 - Input: `{ text: string, intermediateAsset?: boolean, musicModelOverride?: { model: string, config?: object } }`
-- Output: `unknown`
+- Output: `{ audioUrl: string }`
 
 #### generateSound
 Generate a sound effect from a text description using a sound model.
@@ -513,7 +513,7 @@ Download attachments from a Gmail email and store them on the app's own storage.
 - Attachments larger than 25MB are skipped.
 - Use the message ID from Search Gmail Emails, List Recent Gmail Emails, or Get Gmail Email steps.
 - Input: `{ messageId: string, connectionId?: string }`
-- Output: `unknown`
+- Output: `{ attachments: { filename: string, mimeType: string, size: number, url: string }[] }`
 
 #### getGmailUnreadCount
 Get the number of unread emails in the connected Gmail inbox.
@@ -521,7 +521,7 @@ Get the number of unread emails in the connected Gmail inbox.
 - Returns the unread message count for the inbox label.
 - This is a lightweight call that does not fetch any email content.
 - Input: `{ connectionId?: string }`
-- Output: `unknown`
+- Output: `{ unreadCount: number }`
 
 #### getMediaMetadata
 Get info about a media file
@@ -559,7 +559,7 @@ Legacy v1 data sources. v2 apps should use `dataSources.defineDataSource()` from
 - Returns metadata for every data source associated with the current app version.
 - Each entry includes the data source ID, name, description, status, and document list.
 - Input: `object`
-- Output: `unknown`
+- Output: `{ dataSources: ({ id: string, name: string, description: string, status: "loaded" | "hibernating", dateCreated: string, documents: ({ id: string, name: string, status: "processing" | "done" | "error", numWords?: number, dateUploaded: string })[] })[] }`
 
 #### listGmailLabels
 List all labels in the connected Gmail account. Use these label IDs or names with the Update Gmail Labels step.
@@ -567,7 +567,7 @@ List all labels in the connected Gmail account. Use these label IDs or names wit
 - Returns both system labels (INBOX, SENT, TRASH, etc.) and user-created labels.
 - Label type is "system" for built-in labels or "user" for custom labels.
 - Input: `{ connectionId?: string }`
-- Output: `unknown`
+- Output: `{ labels: { id: string, name: string, type: string }[] }`
 
 #### listRecentGmailEmails
 List recent emails from the connected Gmail inbox.
@@ -575,7 +575,7 @@ List recent emails from the connected Gmail inbox.
 - Returns up to 100 emails (default 5), ordered by most recent first.
 - Functionally equivalent to Search Gmail Emails with an "in:inbox" query.
 - Input: `{ connectionId?: string, exportType: "json" | "text", limit: string }`
-- Output: `unknown`
+- Output: `{ emails: { id: string, subject: string, from: string, to: string, date: string, plainBody: string, htmlBody: string, labels: string }[] }`
 
 #### logic
 Route execution to different branches based on AI evaluation, comparison operators, or workflow jumps.
@@ -839,7 +839,7 @@ Send an existing draft from the connected Gmail account.
 - The draft is sent and removed from the Drafts folder.
 - Use the draft ID returned by the Create Gmail Draft or List Gmail Drafts steps.
 - Input: `{ draftId: string, connectionId?: string }`
-- Output: `unknown`
+- Output: `{ messageId: string }`
 
 #### sendGmailMessage
 Send an email from the connected Gmail account.
@@ -872,7 +872,7 @@ Mark one or more Gmail emails as read or unread.
 - Accepts one or more message IDs as a comma-separated string or array.
 - Set markAsRead to true to mark as read, false to mark as unread.
 - Input: `{ messageIds: string, markAsRead: boolean, connectionId?: string }`
-- Output: `unknown`
+- Output: `{ messageIds: string[] }`
 
 #### setRunTitle
 Set the title of the agent run for the user's history
@@ -890,21 +890,33 @@ Explicitly set a variable to a given value.
 #### textToSpeech
 Generate an audio file from provided text using a speech model.
 - The text field contains the exact words to be spoken (not instructions).
+- Voice, delivery and format are set through `speechModelOverride.config`, whose keys are the model's own (look a model up with its id). Delivery direction, where a model takes one, is never spoken: `styleInstruction` on gemini-3.8-flash-tts and gemini-3.8-flash-lite-tts, `instructions` on gpt-realtime-2.1-mini-tts, `voice_instruction` on seed-speech-tts-2.0, `emotion` on cartesia-sonic-3.6 and the MiniMax models.
+- Tags in the text act at that point, in each vendor's own syntax: `[laughs]` or `[whispers]` on elevenlabs-tts with Eleven v4 or v3; `<laugh>`, `<sigh>` or `<whispers>` on the Gemini 3.8 models; `[laugh]`, `[pause]` or `<whisper>…</whisper>` on grok-voice-tts; `[laughter]` or `<emotion value="excited"/>` on cartesia-sonic-3.6; `[be happy and upbeat]` on inworld-tts-2.
+- A dialogue between voices: set `$mode: 'dialogue'` and name each speaker (`speaker1_name`, `speaker1_voice`, `speaker2_name`, …), then write the text as a script, one `Name: line` per turn. Supported by elevenlabs-tts (Eleven v4 or v3, up to 4 speakers) and the Gemini 3.8 models (exactly 2).
+- A text longer than a model takes in one request is spoken in pieces and joined, and then comes back as MP3 whatever format was asked for.
 - In foreground mode, the audio is displayed to the user. In background mode, the URL is saved to a variable.
 - Input: `{ text: string, intermediateAsset?: boolean, speechModelOverride?: { model: string, config?: object } }`
 - Output: `{ audioUrl: string }`
 
 #### transcribeAudio
 Convert an audio file to text using a transcription model.
-- The prompt field provides optional context to improve transcription accuracy (e.g. language, speaker names, domain).
-- Detail beyond the plain transcript is requested through `transcriptionModelOverride.config`, and support varies by model:
-- `includeSpeakers: 'yes'` — label each passage with a speaker. Supported by elevenlabs-scribe-v1, elevenlabs-scribe-v2, deepgram-nova-3 and grok-stt. Also switches `text` to a timestamped, speaker-labelled block format.
-- `includeWordTimings: 'yes'` — add per-word timings and confidence inside each segment. Supported by elevenlabs-scribe-v1, elevenlabs-scribe-v2, deepgram-nova-3 and whisper-1. Off by default because an hour of speech is roughly ten thousand words.
-- `detectLanguage: 'yes'` — detect the spoken language instead of assuming English (deepgram-nova-3 only; Scribe and the whisper models always report it).
-- The `segments` output carries timings, confidence and speakers as structured data, so there is no need to slice the audio up to recover them. It is populated for elevenlabs-scribe-v1/v2, deepgram-nova-3, whisper-1 and the DeepInfra-served whisper/voxtral/qwen models, and absent for models that return nothing but text (gpt-transcribe, gpt-4o-transcribe, grok-stt).
+- The default model is elevenlabs-scribe-v2: speakers, word timings, key terms and redaction, at any length.
+- The prompt field gives context (speaker names, domain) to OpenAI's and DeepInfra's models. Other models ignore it; give them `keyterms` instead.
+- Options go in `transcriptionModelOverride.config`, and support varies by model:
+- `language` — the spoken language as a code such as `en` or `es`; left out, the model detects it. Every model takes it. deepgram-nova-3 assumes `en` unless told, and also takes `multi` for speech that switches language (billed at its multilingual rate).
+- `detectLanguage: 'yes'` — detect the language instead of assuming English (deepgram-nova-3 only).
+- `includeSpeakers: 'yes'` — label each passage with a speaker. Supported by elevenlabs-scribe-v2, elevenlabs-scribe-v2-medical, deepgram-nova-3, grok-stt, voxtral-mini-transcribe-2 and gemini-3.5-transcribe; gpt-4o-transcribe-diarize always does. Scribe and Voxtral also switch `text` to a timestamped, speaker-labelled block format; Deepgram, Gemini and gpt-4o-transcribe-diarize to `Speaker N:` lines.
+- `numSpeakers` — the most people talking, to help tell them apart (Scribe).
+- `keyterms` — a list of names, products and terms the audio contains, so they're spelled right. Supported by gpt-transcribe, Scribe, deepgram-nova-3, grok-stt, voxtral-mini-transcribe-2 and gemini-3.5-transcribe. Billed as an add-on on Scribe and Deepgram.
+- `redactPii: 'yes'` — replace names, phone numbers, addresses and other personal information with a label of what was there (Scribe, deepgram-nova-3). Billed as an add-on.
+- `includeWordTimings: 'yes'` — add per-word timings and confidence inside each segment. Supported by Scribe, deepgram-nova-3, whisper-1, grok-stt, voxtral-mini-transcribe-2 and gemini-3.5-transcribe. Off by default because an hour of speech is roughly ten thousand words.
+- `subtitles: 'srt' | 'vtt'` — also return the transcript as a subtitle file, in the `subtitles` output, timed by the segments. Any model that returns segments.
+- gemini-3.5-transcribe can't take `keyterms` together with speakers, word timings or subtitles, and voxtral-mini-transcribe-2 can't take `language` together with word timings or subtitles; either asked for both is refused.
+- The `segments` output carries timings, confidence and speakers as structured data, so there is no need to slice the audio up to recover them. It is populated by Scribe, deepgram-nova-3, grok-stt, whisper-1, gpt-4o-transcribe-diarize, voxtral-mini-transcribe-2 (unless given a language), gemini-3.5-transcribe (with word timings or subtitles) and the DeepInfra-served whisper and qwen models, and absent for models that return nothing but text (gpt-transcribe, gpt-4o-transcribe, gpt-4o-mini-transcribe, the DeepInfra-served voxtral models).
+- A recording of any length can be transcribed. One longer than a model takes at once (OpenAI's and DeepInfra's models, Gemini) is transcribed in pieces and joined, its timings running from the start of the recording. Speakers can't be followed from one piece to the next, so with speakers on, such a recording is refused; Scribe and Deepgram label speakers at any length.
 - `confidence` is always 0..1 with higher meaning more confident, whichever model produced it.
 - Input: `{ audioUrl: string, prompt: string, transcriptionModelOverride?: { model: string, config?: object } }`
-- Output: `{ text: string, segments?: { start: number, end: number, text: string, confidence?: number, speaker?: string, language?: string, words?: { text: string, start?: number, end?: number, confidence?: number, speaker?: string }[] }[], language?: string, durationSeconds?: number }`
+- Output: `{ text: string, segments?: { start: number, end: number, text: string, confidence?: number, speaker?: string, language?: string, words?: { text: string, start?: number, end?: number, confidence?: number, speaker?: string }[] }[], language?: string, durationSeconds?: number, subtitles?: string }`
 
 #### trimMedia
 Trim an audio or video clip
@@ -927,7 +939,7 @@ Legacy v1 data sources. v2 apps should use `dataSources.defineDataSource()` from
 - Once processing finishes, vectors are loaded into Milvus so the data source is immediately queryable.
 - Supported file types (when using a URL) are the same as the data source upload UI (PDF, DOCX, TXT, etc.).
 - Input: `{ dataSourceId: string, file: string, fileName: string }`
-- Output: `unknown`
+- Output: `{ documentId: string, dataSourceId: string, fileName: string }`
 
 #### upscaleImage
 Increase the resolution of an image using AI upscaling.
@@ -1420,17 +1432,18 @@ Call any QuickBooks Online Accounting API route using a connected company.
 ### TypeSafe
 
 #### typesafeEvaluate
-Ask TypeSafe structured questions about text or JSON and get calibrated answers. The model returns a probability, a selected option, or a score. It does not write a reply. Each question is a noul (yes or no, as a probability from 0 to 1), a choice (one option from a set you define, with a probability for each), or a score (a rating against ordered levels). Put every question in the same call. The question id is yours: answers come back under the same ids. The model defaults to Jev (jev-latest).
-- state is the content to judge: a string, or a JSON object or array (a record, a chat log, application state). Text only. Turn images, audio, and video into text first.
+Ask structured questions about text, JSON, or images and get calibrated answers. The model returns a probability, a selected option, or a score. It does not write a reply. Each question is a noul (yes or no, as a probability from 0 to 1), a choice (one option from a set you define, with a probability for each), or a score (a rating against ordered levels). Put every question in the same call. The question id is yours: answers come back under the same ids. The model defaults to Jev (jev-latest). Pass clef or clef-flash to run Cloudflare Clef with the same questions and answers. Clef and Clef Flash can also read up to 4 images.
+- state is the content to judge: a string, or a JSON object or array (a record, a chat log, application state).
+- images is for Clef and Clef Flash. Pass up to 4 PNG, JPEG, or WebP images as URLs. Each file is at most 4 MiB, and 8 MiB together. A call on Jev that includes images is rejected.
 - questions is a map. The key is an id you choose. It is not shown to the model. The answer comes back under the same id.
 - A noul asks a yes/no question and returns noul, a probability from 0 (no) to 1 (yes). Optional criteria.true and criteria.false say what a yes and a no mean.
 - A choice picks one option. criteria is a map of option id to a short description (or null). At most 255 options. The answer includes the winning choice, a probability for every option, and confidence.
 - A score rates the state on an ordered rubric. criteria is 2 to 10 level descriptions, low to high. The answer's score can land between levels, and includes a probability per level plus confidence.
 - instructions may be a string, or an object that keeps the question in one field and reference data in the others. Refer to those fields by name in backticks.
 - Send every question you need in one call. The model reads the state once and answers them together.
-- model defaults to jev-latest, the current stable release. jev-preview tracks the newest release. Pin a version such as jev-1.13.0 when a threshold was tuned against that version. The response model field is the version that answered.
-- Billed per input token. Output tokens are free.
-- Input: `{ state: string | object | unknown[], questions: object, model?: string }`
+- model defaults to jev-latest, the current stable TypeSafe release. jev-preview tracks the newest TypeSafe release. Pin a version such as jev-1.13.0 when a threshold was tuned against that version. clef and clef-flash run Cloudflare Clef on the same questions and answers. clef is the larger model. clef-flash is the faster one. @cf/cloudflare/clef and @cf/cloudflare/clef-flash name the same two. Clef accepts at most 64 questions, and a question id may use only letters, digits, underscore, dot, and hyphen, up to 100 characters. The response model field is the version that answered.
+- Billed per input token. Output tokens are free. Clef and Clef-flash have their own rates.
+- Input: `{ state: string | object | unknown[], questions: object, model?: string, images?: string[] }`
 - Output: `{ model: string, answers: object, usage: { input_tokens: number, output_tokens: number } }`
 
 ### X

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@mindstudio-ai/agent",
-  "version": "0.1.148",
+  "version": "0.1.150",
   "description": "TypeScript SDK for MindStudio direct step execution",
   "type": "module",
   "main": "./dist/index.js",