@juspay/neurolink 9.8.0 → 9.10.0

package/CHANGELOG.md

@@ -1,3 +1,15 @@
+## [9.10.0](https://github.com/juspay/neurolink/compare/v9.9.0...v9.10.0) (2026-02-20)
+
+### Features
+
+- **(generateText):** add prepareStep and toolChoice passthrough support for multi-step agentic generation ([4cd340a](https://github.com/juspay/neurolink/commit/4cd340af7d39f72006d09fe86569232d751dcd8d))
+
+## [9.9.0](https://github.com/juspay/neurolink/compare/v9.8.0...v9.9.0) (2026-02-17)
+
+### Features
+
+- **(video-analysis):** add video-analysis support in neurolink ([c35f8a8](https://github.com/juspay/neurolink/commit/c35f8a8d52cc1366e10b8701285e1bec52e27d98))
+
 ## [9.8.0](https://github.com/juspay/neurolink/compare/v9.7.0...v9.8.0) (2026-02-17)
 
 ### Features

package/README.md

@@ -35,17 +35,18 @@ Extracted from production systems at Juspay and battle-tested at enterprise scal
 
 ## What's New (Q1 2026)
 
-| Feature                             | Version | Description                                                                                                                                                   | Guide                                                           |
-| ----------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- |
-| **Context Window Management**       | v9.2.0  | 4-stage compaction pipeline with auto-detection, budget gate at 80% usage, per-provider token estimation                                                      | [Context Compaction Guide](docs/features/context-compaction.md) |
-| **File Processor System**           | v9.1.0  | 17+ file type processors with ProcessorRegistry, security sanitization, SVG text injection                                                                    | [File Processors Guide](docs/features/file-processors.md)       |
-| **RAG with generate()/stream()**    | v9.2.0  | Pass `rag: { files }` to generate/stream for automatic document chunking, embedding, and AI-powered search. 10 chunking strategies, hybrid search, reranking. | [RAG Guide](docs/features/rag.md)                               |
-| **External TracerProvider Support** | v8.43.0 | Integrate NeuroLink with existing OpenTelemetry instrumentation. Prevents duplicate registration conflicts.                                                   | [Observability Guide](docs/features/observability.md)           |
-| **Server Adapters**                 | v8.43.0 | Multi-framework HTTP server with Hono, Express, Fastify, Koa support. Full CLI for server management with foreground/background modes.                        | [Server Adapters Guide](docs/guides/server-adapters/index.md)   |
-| **Title Generation Events**         | v8.38.0 | Emit `conversation:titleGenerated` event when conversation title is generated. Supports custom title prompts via `NEUROLINK_TITLE_PROMPT`.                    | [Conversation Memory Guide](docs/conversation-memory.md)        |
-| **Video Generation with Veo**       | v8.32.0 | Video generation using Veo 3.1 (`veo-3.1`). Realistic video generation with many parameter options                                                            | [Video Generation Guide](docs/features/video-generation.md)     |
-| **Image Generation with Gemini**    | v8.31.0 | Native image generation using Gemini 2.0 Flash Experimental (`imagen-3.0-generate-002`). High-quality image synthesis directly from Google AI.                | [Image Generation Guide](docs/image-generation-streaming.md)    |
-| **HTTP/Streamable HTTP Transport**  | v8.29.0 | Connect to remote MCP servers via HTTP with authentication headers, automatic retry with exponential backoff, and configurable rate limiting.                 | [HTTP Transport Guide](docs/mcp-http-transport.md)              |
+| Feature                             | Version | Description                                                                                                                                                   | Guide                                                                 |
+| ----------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
+| **Context Window Management**       | v9.2.0  | 4-stage compaction pipeline with auto-detection, budget gate at 80% usage, per-provider token estimation                                                      | [Context Compaction Guide](docs/features/context-compaction.md)       |
+| **Tool Execution Control**          | v9.3.0  | `prepareStep` and `toolChoice` support for per-step tool enforcement in multi-step agentic loops. API-level control over tool calls.                          | [API Reference](docs/api/type-aliases/GenerateOptions.md#preparestep) |
+| **File Processor System**           | v9.1.0  | 17+ file type processors with ProcessorRegistry, security sanitization, SVG text injection                                                                    | [File Processors Guide](docs/features/file-processors.md)             |
+| **RAG with generate()/stream()**    | v9.2.0  | Pass `rag: { files }` to generate/stream for automatic document chunking, embedding, and AI-powered search. 10 chunking strategies, hybrid search, reranking. | [RAG Guide](docs/features/rag.md)                                     |
+| **External TracerProvider Support** | v8.43.0 | Integrate NeuroLink with existing OpenTelemetry instrumentation. Prevents duplicate registration conflicts.                                                   | [Observability Guide](docs/features/observability.md)                 |
+| **Server Adapters**                 | v8.43.0 | Multi-framework HTTP server with Hono, Express, Fastify, Koa support. Full CLI for server management with foreground/background modes.                        | [Server Adapters Guide](docs/guides/server-adapters/index.md)         |
+| **Title Generation Events**         | v8.38.0 | Emit `conversation:titleGenerated` event when conversation title is generated. Supports custom title prompts via `NEUROLINK_TITLE_PROMPT`.                    | [Conversation Memory Guide](docs/conversation-memory.md)              |
+| **Video Generation with Veo**       | v8.32.0 | Video generation using Veo 3.1 (`veo-3.1`). Realistic video generation with many parameter options                                                            | [Video Generation Guide](docs/features/video-generation.md)           |
+| **Image Generation with Gemini**    | v8.31.0 | Native image generation using Gemini 2.0 Flash Experimental (`imagen-3.0-generate-002`). High-quality image synthesis directly from Google AI.                | [Image Generation Guide](docs/image-generation-streaming.md)          |
+| **HTTP/Streamable HTTP Transport**  | v8.29.0 | Connect to remote MCP servers via HTTP with authentication headers, automatic retry with exponential backoff, and configurable rate limiting.                 | [HTTP Transport Guide](docs/mcp-http-transport.md)                    |
 
 - **External TracerProvider Support** – Integrate NeuroLink with applications that already have OpenTelemetry instrumentation. Supports auto-detection and manual configuration. → [Observability Guide](docs/features/observability.md)
 - **Server Adapters** – Deploy NeuroLink as an HTTP API server with your framework of choice (Hono, Express, Fastify, Koa). Full CLI support with `serve` and `server` commands for foreground/background modes, route management, and OpenAPI generation. → [Server Adapters Guide](docs/guides/server-adapters/index.md)
@@ -56,6 +57,7 @@ Extracted from production systems at Juspay and battle-tested at enterprise scal
 - **RAG with generate()/stream()** – Just pass `rag: { files: ["./docs/guide.md"] }` to `generate()` or `stream()`. NeuroLink auto-chunks, embeds, and creates a search tool the AI can invoke. 10 chunking strategies, hybrid search, 5 reranker types. → [RAG Guide](docs/features/rag.md)
 - **HTTP/Streamable HTTP Transport for MCP** – Connect to remote MCP servers via HTTP with authentication headers, retry logic, and rate limiting. → [HTTP Transport Guide](docs/mcp-http-transport.md)
 - 🧠 **Gemini 3 Preview Support** - Full support for gemini-3-flash-preview and gemini-3-pro-preview with extended thinking capabilities
+- 🎯 **Tool Execution Control** – Use `prepareStep` to enforce specific tool calls, change the LLM models per step in multi-step agentic executions. Prevents LLMs from skipping required tools. Use `toolChoice` for static control, or `prepareStep` for dynamic per-step logic. → [GenerateOptions Reference](docs/api/type-aliases/GenerateOptions.md#preparestep)
 - **Structured Output with Zod Schemas** – Type-safe JSON generation with automatic validation using `schema` + `output.format: "json"` in `generate()`. → [Structured Output Guide](docs/features/structured-output.md)
 - **CSV File Support** – Attach CSV files to prompts for AI-powered data analysis with auto-detection. → [CSV Guide](docs/features/multimodal-chat.md#csv-file-support)
 - **PDF File Support** – Process PDF documents with native visual analysis for Vertex AI, Anthropic, Bedrock, AI Studio. → [PDF Guide](docs/features/pdf-support.md)

package/dist/adapters/video/videoAnalyzer.d.ts

@@ -0,0 +1,26 @@
+/**
+ * Video Analysis Handler
+ *
+ * Provides video analysis using Google's Gemini 2.0 Flash model.
+ * Supports both Vertex AI and Gemini API providers.
+ *
+ * @module adapters/video/geminiVideoAnalyzer
+ */
+import { AIProviderName } from "../../constants/enums.js";
+import type { CoreMessage } from "ai";
+export declare function analyzeVideoWithVertexAI(frames: CoreMessage, options?: {
+    project?: string;
+    location?: string;
+    model?: string;
+}): Promise<string>;
+export declare function analyzeVideoWithGeminiAPI(frames: CoreMessage, options?: {
+    apiKey?: string;
+    model?: string;
+}): Promise<string>;
+export declare function analyzeVideo(frames: CoreMessage, options?: {
+    provider?: AIProviderName;
+    project?: string;
+    location?: string;
+    apiKey?: string;
+    model?: string;
+}): Promise<string>;

package/dist/adapters/video/videoAnalyzer.js

@@ -0,0 +1,222 @@
+/**
+ * Video Analysis Handler
+ *
+ * Provides video analysis using Google's Gemini 2.0 Flash model.
+ * Supports both Vertex AI and Gemini API providers.
+ *
+ * @module adapters/video/geminiVideoAnalyzer
+ */
+import { AIProviderName, ErrorSeverity, ErrorCategory, } from "../../constants/enums.js";
+import { logger } from "../../utils/logger.js";
+import { readFile } from "node:fs/promises";
+import { NeuroLinkError } from "../../utils/errorHandling.js";
+// ---------------------------------------------------------------------------
+// Shared config
+// ---------------------------------------------------------------------------
+const DEFAULT_MODEL = "gemini-2.0-flash";
+const DEFAULT_LOCATION = "us-central1";
+/**
+ * Convert CoreMessage content array to Gemini parts format
+ *
+ * @param contentArray - Array of content items from CoreMessage
+ * @returns Array of parts in Gemini API format
+ */
+function buildContentParts(frames) {
+    const contentArray = Array.isArray(frames.content) ? frames.content : [];
+    return contentArray.map((item) => {
+        if (item.type === "text" && item.text) {
+            return { text: item.text };
+        }
+        else if (item.type === "image" && item.image) {
+            let base64Data;
+            // Handle Buffer or Uint8Array
+            if (Buffer.isBuffer(item.image) || item.image instanceof Uint8Array) {
+                base64Data = Buffer.from(item.image).toString("base64");
+            }
+            else if (typeof item.image === "string") {
+                // Strip data URI prefix if present (e.g., "data:image/jpeg;base64,")
+                base64Data = item.image.replace(/^data:image\/[a-z]+;base64,/, "");
+            }
+            else {
+                throw new Error(`Invalid image data type: expected string, Buffer, or Uint8Array, got ${typeof item.image}`);
+            }
+            return {
+                inlineData: {
+                    mimeType: "image/jpeg",
+                    data: base64Data,
+                },
+            };
+        }
+        throw new Error(`Invalid content type: ${item.type}`);
+    });
+}
+/**
+ * Configuration for video frame analysis.
+ * Generic prompt that handles both general content and technical bug reporting.
+ */
+function buildConfig() {
+    return {
+        systemInstruction: `You are a Visual Analysis Assistant.
+Your task is to analyze images or video frames provided by the user and extract structured visual features. The user may or may not provide an issue description. Your role is to understand the visual content, optionally correlate it with the provided issue, and produce a structured output that can be directly consumed by another LLM for analysis, debugging, or decision-making.
+
+Follow these rules strictly:
+- The analysis must be generic and applicable to any domain (UI, dashboards, video frames, animations, charts, documents, etc.).
+- Support both images and videos (single frame or multiple frames).
+- Extract only what is visually observable; do not assume backend behavior unless supported by visuals.
+- The JSON must be structured, consistent, and machine-readable.
+- Logs are optional and should only be included if explicitly provided.
+- The final output must be clear, concise, and actionable for an LLM.
+
+Always produce the output in the following format:
+
+Issue:
+<Refined issue description if provided, otherwise a clear description of the observed visual situation>
+
+Image/Video Patterns:
+<Structured JSON describing extracted visual features and anomalies>
+
+Steps to Reproduce:
+<Ordered steps that reliably reproduce the issue based on the visual context>
+
+[Logs: Include ONLY if provided by the user]
+
+Proof:
+<Visual evidence explaining how the image/video confirms the issue>
+Ensure the final response is fully self-sufficient and does not reference external context.`,
+    };
+}
+// ---------------------------------------------------------------------------
+// Vertex AI
+// ---------------------------------------------------------------------------
+export async function analyzeVideoWithVertexAI(frames, options = {}) {
+    const startTime = Date.now();
+    const { GoogleGenAI } = await import("@google/genai");
+    // Get default config and merge with provided options
+    const config = await getVertexConfig();
+    const project = options.project ?? config.project;
+    const location = options.location ?? config.location;
+    const model = options.model || DEFAULT_MODEL;
+    // Extract content array from CoreMessage
+    const contentArray = Array.isArray(frames.content) ? frames.content : [];
+    const frameCount = contentArray.filter((item) => item.type === "image").length;
+    logger.debug("[GeminiVideoAnalyzer] Analyzing video with Vertex AI", {
+        project,
+        location,
+        model,
+        frameCount,
+    });
+    const ai = new GoogleGenAI({ vertexai: true, project, location });
+    // Convert frames content to parts array for Gemini
+    const parts = buildContentParts(frames);
+    const response = await ai.models.generateContent({
+        model,
+        config: buildConfig(),
+        contents: [
+            {
+                role: "user",
+                parts,
+            },
+        ],
+    });
+    const responseText = response.text || "";
+    const processingTime = Date.now() - startTime;
+    logger.debug("[GeminiVideoAnalyzer] Vertex response received", {
+        responseLength: responseText.length,
+        processingTime,
+    });
+    return responseText;
+}
+// ---------------------------------------------------------------------------
+// Gemini API (Google AI)
+// ---------------------------------------------------------------------------
+export async function analyzeVideoWithGeminiAPI(frames, options = {}) {
+    const startTime = Date.now();
+    const { GoogleGenAI } = await import("@google/genai");
+    const apiKey = options.apiKey || process.env.GOOGLE_AI_API_KEY;
+    const model = options.model || DEFAULT_MODEL;
+    if (!apiKey) {
+        throw new Error("GOOGLE_AI_API_KEY environment variable is required for Gemini API video analysis");
+    }
+    // Extract content array from CoreMessage
+    const contentArray = Array.isArray(frames.content) ? frames.content : [];
+    const frameCount = contentArray.filter((item) => item.type === "image").length;
+    logger.debug("[GeminiVideoAnalyzer] Analyzing video with Gemini API", {
+        model,
+        frameCount,
+    });
+    const ai = new GoogleGenAI({ apiKey });
+    // Convert frames content to parts array for Gemini
+    const parts = buildContentParts(frames);
+    logger.debug("[GeminiVideoAnalyzer] Generating analysis with frames");
+    const response = await ai.models.generateContent({
+        model,
+        config: buildConfig(),
+        contents: [
+            {
+                role: "user",
+                parts,
+            },
+        ],
+    });
+    const responseText = response.text || "";
+    const processingTime = Date.now() - startTime;
+    logger.debug("[GeminiVideoAnalyzer] Gemini API response received", {
+        responseLength: responseText.length,
+        processingTime,
+    });
+    return responseText;
+}
+async function getVertexConfig() {
+    const location = process.env.GOOGLE_VERTEX_LOCATION || DEFAULT_LOCATION;
+    // Try environment variables first
+    let project = process.env.GOOGLE_VERTEX_PROJECT ||
+        process.env.GOOGLE_CLOUD_PROJECT ||
+        process.env.GOOGLE_CLOUD_PROJECT_ID ||
+        process.env.VERTEX_PROJECT_ID;
+    // Fallback: read from ADC credentials file
+    if (!project && process.env.GOOGLE_APPLICATION_CREDENTIALS) {
+        try {
+            const credData = JSON.parse(await readFile(process.env.GOOGLE_APPLICATION_CREDENTIALS, "utf-8"));
+            project = credData.quota_project_id || credData.project_id;
+        }
+        catch (e) {
+            // Ignore read errors, will throw below if project still not found
+            logger.debug("Failed to read project from credentials file", {
+                error: e instanceof Error ? e.message : String(e),
+            });
+        }
+    }
+    if (!project) {
+        throw new NeuroLinkError({
+            code: "PROVIDER_NOT_CONFIGURED",
+            message: "Google Cloud project not found. Set GOOGLE_VERTEX_PROJECT or GOOGLE_CLOUD_PROJECT environment variable, or ensure ADC credentials contain project_id",
+            category: ErrorCategory.CONFIGURATION,
+            severity: ErrorSeverity.HIGH,
+            retriable: false,
+            context: {
+                missingVar: "GOOGLE_VERTEX_PROJECT",
+                feature: "video-generation",
+                checkedEnvVars: [
+                    "GOOGLE_VERTEX_PROJECT",
+                    "GOOGLE_CLOUD_PROJECT",
+                    "GOOGLE_CLOUD_PROJECT_ID",
+                    "VERTEX_PROJECT_ID",
+                ],
+            },
+        });
+    }
+    return { project, location };
+}
+export async function analyzeVideo(frames, options = {}) {
+    const provider = options.provider || AIProviderName.AUTO;
+    // Vertex — only when GOOGLE_VERTEX_PROJECT is explicitly set
+    if (provider === AIProviderName.VERTEX || provider === AIProviderName.AUTO) {
+        return analyzeVideoWithVertexAI(frames, options);
+    }
+    // Gemini API — when GOOGLE_AI_API_KEY is set
+    if (provider === AIProviderName.GOOGLE_AI && process.env.GOOGLE_AI_API_KEY) {
+        return analyzeVideoWithGeminiAPI(frames, options);
+    }
+    throw new Error("No valid provider configuration found. " +
+        "Set GOOGLE_VERTEX_PROJECT for Vertex AI or GOOGLE_AI_API_KEY for Gemini API.");
+}

package/dist/cli/loop/optionsSchema.d.ts

@@ -5,4 +5,4 @@ import type { TextGenerationOptions } from "../../lib/types/generateTypes.js";
  * This object provides metadata for validation and help text in the CLI loop.
  * It is derived from the main TextGenerationOptions interface to ensure consistency.
  */
-export declare const textGenerationOptionsSchema: Record<keyof Omit<TextGenerationOptions, "prompt" | "input" | "schema" | "tools" | "context" | "conversationHistory" | "conversationMessages" | "conversationMemoryConfig" | "originalPrompt" | "middleware" | "expectedOutcome" | "evaluationCriteria" | "region" | "csvOptions" | "tts" | "thinkingConfig" | "fileRegistry" | "abortSignal" | "toolFilter" | "excludeTools">, OptionSchema>;
+export declare const textGenerationOptionsSchema: Record<keyof Omit<TextGenerationOptions, "prompt" | "input" | "schema" | "tools" | "context" | "conversationHistory" | "conversationMessages" | "conversationMemoryConfig" | "originalPrompt" | "middleware" | "expectedOutcome" | "evaluationCriteria" | "region" | "csvOptions" | "tts" | "thinkingConfig" | "fileRegistry" | "abortSignal" | "toolFilter" | "excludeTools" | "toolChoice" | "prepareStep">, OptionSchema>;

package/dist/core/baseProvider.js

@@ -7,6 +7,7 @@ import { composeAbortSignals, createTimeoutController, TimeoutError, } from "../
 import { shouldDisableBuiltinTools } from "../utils/toolUtils.js";
 import { getKeyCount, getKeysAsString } from "../utils/transformationUtils.js";
 import { TTSProcessor } from "../utils/ttsProcessor.js";
+import { hasVideoFrames, executeVideoAnalysis, } from "../utils/videoAnalysisProcessor.js";
 import { GenerationHandler } from "./modules/GenerationHandler.js";
 // Import modules for composition
 import { MessageBuilder } from "./modules/MessageBuilder.js";
@@ -473,6 +474,25 @@ export class BaseProvider {
             // ===== Normal AI Generation Flow =====
             const { tools, model } = await this.prepareGenerationContext(options);
             const messages = await this.buildMessages(options);
+            // ===== VIDEO ANALYSIS FROM MESSAGES CONTENT =====
+            // Check if video files are present in messages content array
+            // If video analysis is needed, perform it and return early to avoid running generation
+            if (hasVideoFrames(messages)) {
+                const videoAnalysisResult = await executeVideoAnalysis(messages, {
+                    provider: options.provider,
+                    providerName: this.providerName,
+                    region: options.region,
+                    model: options.model,
+                });
+                // Return video analysis result directly without running generation
+                const videoResult = {
+                    content: videoAnalysisResult,
+                    provider: options.provider ?? this.providerName,
+                    model: this.modelName,
+                    usage: { input: 0, output: 0, total: 0 }, // Video analysis doesn't use standard token counting
+                };
+                return await this.enhanceResult(videoResult, options, startTime);
+            }
             // Compose timeout signal with user-provided abort signal (mirrors stream path)
             const timeoutController = createTimeoutController(options.timeout, this.providerName, "generate");
             const composedSignal = composeAbortSignals(options.abortSignal, timeoutController?.controller.signal);

package/dist/core/modules/GenerationHandler.js

@@ -49,8 +49,12 @@ export class GenerationHandler {
             model,
             messages,
             ...(shouldUseTools && Object.keys(tools).length > 0 && { tools }),
-            maxSteps: options.maxSteps || DEFAULT_MAX_STEPS,
-            ...(shouldUseTools && { toolChoice: "auto" }),
+            maxSteps: options.maxSteps ?? DEFAULT_MAX_STEPS,
+            ...(shouldUseTools &&
+                options.toolChoice && { toolChoice: options.toolChoice }),
+            ...(options.prepareStep && {
+                experimental_prepareStep: options.prepareStep,
+            }),
             temperature: options.temperature,
             maxTokens: options.maxTokens,
             abortSignal: options.abortSignal,

package/dist/lib/adapters/video/videoAnalyzer.d.ts

@@ -0,0 +1,26 @@
+/**
+ * Video Analysis Handler
+ *
+ * Provides video analysis using Google's Gemini 2.0 Flash model.
+ * Supports both Vertex AI and Gemini API providers.
+ *
+ * @module adapters/video/geminiVideoAnalyzer
+ */
+import { AIProviderName } from "../../constants/enums.js";
+import type { CoreMessage } from "ai";
+export declare function analyzeVideoWithVertexAI(frames: CoreMessage, options?: {
+    project?: string;
+    location?: string;
+    model?: string;
+}): Promise<string>;
+export declare function analyzeVideoWithGeminiAPI(frames: CoreMessage, options?: {
+    apiKey?: string;
+    model?: string;
+}): Promise<string>;
+export declare function analyzeVideo(frames: CoreMessage, options?: {
+    provider?: AIProviderName;
+    project?: string;
+    location?: string;
+    apiKey?: string;
+    model?: string;
+}): Promise<string>;

package/dist/lib/adapters/video/videoAnalyzer.js

@@ -0,0 +1,223 @@
+/**
+ * Video Analysis Handler
+ *
+ * Provides video analysis using Google's Gemini 2.0 Flash model.
+ * Supports both Vertex AI and Gemini API providers.
+ *
+ * @module adapters/video/geminiVideoAnalyzer
+ */
+import { AIProviderName, ErrorSeverity, ErrorCategory, } from "../../constants/enums.js";
+import { logger } from "../../utils/logger.js";
+import { readFile } from "node:fs/promises";
+import { NeuroLinkError } from "../../utils/errorHandling.js";
+// ---------------------------------------------------------------------------
+// Shared config
+// ---------------------------------------------------------------------------
+const DEFAULT_MODEL = "gemini-2.0-flash";
+const DEFAULT_LOCATION = "us-central1";
+/**
+ * Convert CoreMessage content array to Gemini parts format
+ *
+ * @param contentArray - Array of content items from CoreMessage
+ * @returns Array of parts in Gemini API format
+ */
+function buildContentParts(frames) {
+    const contentArray = Array.isArray(frames.content) ? frames.content : [];
+    return contentArray.map((item) => {
+        if (item.type === "text" && item.text) {
+            return { text: item.text };
+        }
+        else if (item.type === "image" && item.image) {
+            let base64Data;
+            // Handle Buffer or Uint8Array
+            if (Buffer.isBuffer(item.image) || item.image instanceof Uint8Array) {
+                base64Data = Buffer.from(item.image).toString("base64");
+            }
+            else if (typeof item.image === "string") {
+                // Strip data URI prefix if present (e.g., "data:image/jpeg;base64,")
+                base64Data = item.image.replace(/^data:image\/[a-z]+;base64,/, "");
+            }
+            else {
+                throw new Error(`Invalid image data type: expected string, Buffer, or Uint8Array, got ${typeof item.image}`);
+            }
+            return {
+                inlineData: {
+                    mimeType: "image/jpeg",
+                    data: base64Data,
+                },
+            };
+        }
+        throw new Error(`Invalid content type: ${item.type}`);
+    });
+}
+/**
+ * Configuration for video frame analysis.
+ * Generic prompt that handles both general content and technical bug reporting.
+ */
+function buildConfig() {
+    return {
+        systemInstruction: `You are a Visual Analysis Assistant.
+Your task is to analyze images or video frames provided by the user and extract structured visual features. The user may or may not provide an issue description. Your role is to understand the visual content, optionally correlate it with the provided issue, and produce a structured output that can be directly consumed by another LLM for analysis, debugging, or decision-making.
+
+Follow these rules strictly:
+- The analysis must be generic and applicable to any domain (UI, dashboards, video frames, animations, charts, documents, etc.).
+- Support both images and videos (single frame or multiple frames).
+- Extract only what is visually observable; do not assume backend behavior unless supported by visuals.
+- The JSON must be structured, consistent, and machine-readable.
+- Logs are optional and should only be included if explicitly provided.
+- The final output must be clear, concise, and actionable for an LLM.
+
+Always produce the output in the following format:
+
+Issue:
+<Refined issue description if provided, otherwise a clear description of the observed visual situation>
+
+Image/Video Patterns:
+<Structured JSON describing extracted visual features and anomalies>
+
+Steps to Reproduce:
+<Ordered steps that reliably reproduce the issue based on the visual context>
+
+[Logs: Include ONLY if provided by the user]
+
+Proof:
+<Visual evidence explaining how the image/video confirms the issue>
+Ensure the final response is fully self-sufficient and does not reference external context.`,
+    };
+}
+// ---------------------------------------------------------------------------
+// Vertex AI
+// ---------------------------------------------------------------------------
+export async function analyzeVideoWithVertexAI(frames, options = {}) {
+    const startTime = Date.now();
+    const { GoogleGenAI } = await import("@google/genai");
+    // Get default config and merge with provided options
+    const config = await getVertexConfig();
+    const project = options.project ?? config.project;
+    const location = options.location ?? config.location;
+    const model = options.model || DEFAULT_MODEL;
+    // Extract content array from CoreMessage
+    const contentArray = Array.isArray(frames.content) ? frames.content : [];
+    const frameCount = contentArray.filter((item) => item.type === "image").length;
+    logger.debug("[GeminiVideoAnalyzer] Analyzing video with Vertex AI", {
+        project,
+        location,
+        model,
+        frameCount,
+    });
+    const ai = new GoogleGenAI({ vertexai: true, project, location });
+    // Convert frames content to parts array for Gemini
+    const parts = buildContentParts(frames);
+    const response = await ai.models.generateContent({
+        model,
+        config: buildConfig(),
+        contents: [
+            {
+                role: "user",
+                parts,
+            },
+        ],
+    });
+    const responseText = response.text || "";
+    const processingTime = Date.now() - startTime;
+    logger.debug("[GeminiVideoAnalyzer] Vertex response received", {
+        responseLength: responseText.length,
+        processingTime,
+    });
+    return responseText;
+}
+// ---------------------------------------------------------------------------
+// Gemini API (Google AI)
+// ---------------------------------------------------------------------------
+export async function analyzeVideoWithGeminiAPI(frames, options = {}) {
+    const startTime = Date.now();
+    const { GoogleGenAI } = await import("@google/genai");
+    const apiKey = options.apiKey || process.env.GOOGLE_AI_API_KEY;
+    const model = options.model || DEFAULT_MODEL;
+    if (!apiKey) {
+        throw new Error("GOOGLE_AI_API_KEY environment variable is required for Gemini API video analysis");
+    }
+    // Extract content array from CoreMessage
+    const contentArray = Array.isArray(frames.content) ? frames.content : [];
+    const frameCount = contentArray.filter((item) => item.type === "image").length;
+    logger.debug("[GeminiVideoAnalyzer] Analyzing video with Gemini API", {
+        model,
+        frameCount,
+    });
+    const ai = new GoogleGenAI({ apiKey });
+    // Convert frames content to parts array for Gemini
+    const parts = buildContentParts(frames);
+    logger.debug("[GeminiVideoAnalyzer] Generating analysis with frames");
+    const response = await ai.models.generateContent({
+        model,
+        config: buildConfig(),
+        contents: [
+            {
+                role: "user",
+                parts,
+            },
+        ],
+    });
+    const responseText = response.text || "";
+    const processingTime = Date.now() - startTime;
+    logger.debug("[GeminiVideoAnalyzer] Gemini API response received", {
+        responseLength: responseText.length,
+        processingTime,
+    });
+    return responseText;
+}
+async function getVertexConfig() {
+    const location = process.env.GOOGLE_VERTEX_LOCATION || DEFAULT_LOCATION;
+    // Try environment variables first
+    let project = process.env.GOOGLE_VERTEX_PROJECT ||
+        process.env.GOOGLE_CLOUD_PROJECT ||
+        process.env.GOOGLE_CLOUD_PROJECT_ID ||
+        process.env.VERTEX_PROJECT_ID;
+    // Fallback: read from ADC credentials file
+    if (!project && process.env.GOOGLE_APPLICATION_CREDENTIALS) {
+        try {
+            const credData = JSON.parse(await readFile(process.env.GOOGLE_APPLICATION_CREDENTIALS, "utf-8"));
+            project = credData.quota_project_id || credData.project_id;
+        }
+        catch (e) {
+            // Ignore read errors, will throw below if project still not found
+            logger.debug("Failed to read project from credentials file", {
+                error: e instanceof Error ? e.message : String(e),
+            });
+        }
+    }
+    if (!project) {
+        throw new NeuroLinkError({
+            code: "PROVIDER_NOT_CONFIGURED",
+            message: "Google Cloud project not found. Set GOOGLE_VERTEX_PROJECT or GOOGLE_CLOUD_PROJECT environment variable, or ensure ADC credentials contain project_id",
+            category: ErrorCategory.CONFIGURATION,
+            severity: ErrorSeverity.HIGH,
+            retriable: false,
+            context: {
+                missingVar: "GOOGLE_VERTEX_PROJECT",
+                feature: "video-generation",
+                checkedEnvVars: [
+                    "GOOGLE_VERTEX_PROJECT",
+                    "GOOGLE_CLOUD_PROJECT",
+                    "GOOGLE_CLOUD_PROJECT_ID",
+                    "VERTEX_PROJECT_ID",
+                ],
+            },
+        });
+    }
+    return { project, location };
+}
+export async function analyzeVideo(frames, options = {}) {
+    const provider = options.provider || AIProviderName.AUTO;
+    // Vertex — only when GOOGLE_VERTEX_PROJECT is explicitly set
+    if (provider === AIProviderName.VERTEX || provider === AIProviderName.AUTO) {
+        return analyzeVideoWithVertexAI(frames, options);
+    }
+    // Gemini API — when GOOGLE_AI_API_KEY is set
+    if (provider === AIProviderName.GOOGLE_AI && process.env.GOOGLE_AI_API_KEY) {
+        return analyzeVideoWithGeminiAPI(frames, options);
+    }
+    throw new Error("No valid provider configuration found. " +
+        "Set GOOGLE_VERTEX_PROJECT for Vertex AI or GOOGLE_AI_API_KEY for Gemini API.");
+}
+//# sourceMappingURL=videoAnalyzer.js.map

package/dist/lib/core/baseProvider.js

@@ -7,6 +7,7 @@ import { composeAbortSignals, createTimeoutController, TimeoutError, } from "../
 import { shouldDisableBuiltinTools } from "../utils/toolUtils.js";
 import { getKeyCount, getKeysAsString } from "../utils/transformationUtils.js";
 import { TTSProcessor } from "../utils/ttsProcessor.js";
+import { hasVideoFrames, executeVideoAnalysis, } from "../utils/videoAnalysisProcessor.js";
 import { GenerationHandler } from "./modules/GenerationHandler.js";
 // Import modules for composition
 import { MessageBuilder } from "./modules/MessageBuilder.js";
@@ -473,6 +474,25 @@ export class BaseProvider {
             // ===== Normal AI Generation Flow =====
             const { tools, model } = await this.prepareGenerationContext(options);
             const messages = await this.buildMessages(options);
+            // ===== VIDEO ANALYSIS FROM MESSAGES CONTENT =====
+            // Check if video files are present in messages content array
+            // If video analysis is needed, perform it and return early to avoid running generation
+            if (hasVideoFrames(messages)) {
+                const videoAnalysisResult = await executeVideoAnalysis(messages, {
+                    provider: options.provider,
+                    providerName: this.providerName,
+                    region: options.region,
+                    model: options.model,
+                });
+                // Return video analysis result directly without running generation
+                const videoResult = {
+                    content: videoAnalysisResult,
+                    provider: options.provider ?? this.providerName,
+                    model: this.modelName,
+                    usage: { input: 0, output: 0, total: 0 }, // Video analysis doesn't use standard token counting
+                };
+                return await this.enhanceResult(videoResult, options, startTime);
+            }
             // Compose timeout signal with user-provided abort signal (mirrors stream path)
             const timeoutController = createTimeoutController(options.timeout, this.providerName, "generate");
             const composedSignal = composeAbortSignals(options.abortSignal, timeoutController?.controller.signal);

package/dist/lib/core/modules/GenerationHandler.js

@@ -49,8 +49,12 @@ export class GenerationHandler {
             model,
             messages,
             ...(shouldUseTools && Object.keys(tools).length > 0 && { tools }),
-            maxSteps: options.maxSteps || DEFAULT_MAX_STEPS,
-            ...(shouldUseTools && { toolChoice: "auto" }),
+            maxSteps: options.maxSteps ?? DEFAULT_MAX_STEPS,
+            ...(shouldUseTools &&
+                options.toolChoice && { toolChoice: options.toolChoice }),
+            ...(options.prepareStep && {
+                experimental_prepareStep: options.prepareStep,
+            }),
             temperature: options.temperature,
             maxTokens: options.maxTokens,
             abortSignal: options.abortSignal,

package/dist/lib/neurolink.js

@@ -1570,6 +1570,9 @@ Current user's request: ${currentInput}`;
                 disableTools: options.disableTools,
                 toolFilter: options.toolFilter,
                 excludeTools: options.excludeTools,
+                maxSteps: options.maxSteps,
+                toolChoice: options.toolChoice,
+                prepareStep: options.prepareStep,
                 enableAnalytics: options.enableAnalytics,
                 enableEvaluation: options.enableEvaluation,
                 context: options.context,

package/dist/lib/processors/media/VideoProcessor.js

@@ -529,7 +529,7 @@ export class VideoProcessor extends BaseFileProcessor {
         // Extract frames using ffmpeg
         const framesDir = join(tempDir, "frames");
         await fs.mkdir(framesDir, { recursive: true });
-        await this.runFfmpegFrameExtraction(videoPath, framesDir, timestamps);
+        await this.runFfmpegFrameExtraction(videoPath, framesDir, timestamps, intervalSec);
         // Read extracted frames and resize with sharp
         const keyframes = [];
         for (let i = 0; i < timestamps.length; i++) {
@@ -563,15 +563,11 @@ export class VideoProcessor extends BaseFileProcessor {
      * @param outputDir - Directory to write frame files
      * @param timestamps - Array of timestamps in seconds
      */
-    runFfmpegFrameExtraction(videoPath, outputDir, timestamps) {
+    runFfmpegFrameExtraction(videoPath, outputDir, timestamps, intervalSec) {
         return new Promise((resolve, reject) => {
-            // Build select filter expression: select='eq(n,0)+eq(n,250)+...'
-            // Instead, use fps filter for simpler approach - extract at regular intervals
-            // using -vf fps=1/interval approach which is more reliable
-            // Build timestamp-based filter
-            const selectExpr = timestamps
-                .map((t) => `gte(t\\,${t})*lt(t\\,${t + 0.5})`)
-                .join("+");
+            // Improved select expression to pick exactly one frame per interval
+            // instead of multiple frames within a 0.5s window.
+            const selectExpr = `isnan(prev_selected_t)+gte(t-prev_selected_t,${intervalSec}-0.001)`;
             const timeoutId = setTimeout(() => {
                 reject(new Error(`ffmpeg frame extraction timed out after ${VIDEO_CONFIG.FFMPEG_TIMEOUT_MS}ms`));
             }, VIDEO_CONFIG.FFMPEG_TIMEOUT_MS);
@@ -861,19 +857,20 @@ export class VideoProcessor extends BaseFileProcessor {
             }
             const clampedCount = Math.min(frameCount, VIDEO_CONFIG.MAX_FRAMES);
             const timestamps = [];
+            let interval = duration;
             if (clampedCount === 1) {
                 timestamps.push(startSec);
             }
             else {
-                const step = duration / (clampedCount - 1);
+                interval = duration / (clampedCount - 1);
                 for (let i = 0; i < clampedCount; i++) {
-                    timestamps.push(startSec + step * i);
+                    timestamps.push(startSec + interval * i);
                 }
             }
             // Extract frames
             const framesDir = join(tempDir, "frames");
             await fs.mkdir(framesDir, { recursive: true });
-            await this.runFfmpegFrameExtraction(tempVideoPath, framesDir, timestamps);
+            await this.runFfmpegFrameExtraction(tempVideoPath, framesDir, timestamps, interval);
             // Read and resize frames
             const keyframes = [];
             for (let i = 0; i < timestamps.length; i++) {

package/dist/lib/types/generateTypes.d.ts

@@ -1,4 +1,4 @@
-import type { Schema, Tool } from "ai";
+import type { Schema, Tool, ToolChoice, StepResult, LanguageModel } from "ai";
 import type { AIProviderName } from "../constants/enums.js";
 import type { RAGConfig } from "../rag/types.js";
 import type { AnalyticsData, TokenUsage } from "./analytics.js";
@@ -248,6 +248,55 @@ export type GenerateOptions = {
      * Default: false (backward compatible — tool schemas are injected into system prompt).
      */
     skipToolPromptInjection?: boolean;
+    /** Maximum number of tool execution steps (default: 200) */
+    maxSteps?: number;
+    /**
+     * Tool choice configuration for the generation.
+     * Controls whether and which tools the model must call.
+     *
+     * - `"auto"` (default): the model can choose whether and which tools to call
+     * - `"none"`: no tool calls allowed
+     * - `"required"`: the model must call at least one tool
+     * - `{ type: "tool", toolName: string }`: the model must call the specified tool
+     *
+     * Note: When used without `prepareStep`, this applies to **every step** in the
+     * `maxSteps` loop. Using `"required"` or `{ type: "tool" }` without `prepareStep`
+     * will cause infinite tool calls until `maxSteps` is exhausted.
+     */
+    toolChoice?: ToolChoice<Record<string, Tool>>;
+    /**
+     * Optional callback that runs before each step in a multi-step generation.
+     * Allows dynamically changing `toolChoice` and available tools per step.
+     *
+     * This is the recommended way to enforce specific tool calls on certain steps
+     * while allowing the model freedom on others.
+     *
+     * Maps to Vercel AI SDK's `experimental_prepareStep`.
+     *
+     * @example Force a specific tool on step 0, then switch to auto:
+     * ```typescript
+     * prepareStep: ({ stepNumber, steps }) => {
+     *   if (stepNumber === 0) {
+     *     return {
+     *       toolChoice: { type: 'tool', toolName: 'myTool' }
+     *     };
+     *   }
+     *   return { toolChoice: 'auto' };
+     * }
+     * ```
+     *
+     * @see https://ai-sdk.dev/docs/reference/ai-sdk-core/generate-text#parameters
+     */
+    prepareStep?: (options: {
+        steps: StepResult<Record<string, Tool>>[];
+        stepNumber: number;
+        maxSteps: number;
+        model: LanguageModel;
+    }) => PromiseLike<{
+        model?: LanguageModel;
+        toolChoice?: ToolChoice<Record<string, Tool>>;
+        experimental_activeTools?: string[];
+    } | undefined>;
     enableEvaluation?: boolean;
     enableAnalytics?: boolean;
     context?: StandardRecord;
@@ -521,6 +570,7 @@ export type TextGenerationOptions = {
          */
         images?: Array<Buffer | string | import("./content.js").ImageWithAltText>;
         pdfFiles?: Array<Buffer | string>;
+        files?: Array<Buffer | string | import("./fileTypes.js").FileWithMetadata>;
     };
     provider?: AIProviderName;
     model?: string;
@@ -568,6 +618,53 @@ export type TextGenerationOptions = {
     toolFilter?: string[];
     /** Exclude these tools by name (blacklist). Applied after toolFilter. */
     excludeTools?: string[];
+    /**
+     * Tool choice configuration for the generation.
+     * Controls whether and which tools the model must call.
+     *
+     * - `"auto"` (default): the model can choose whether and which tools to call
+     * - `"none"`: no tool calls allowed
+     * - `"required"`: the model must call at least one tool
+     * - `{ type: "tool", toolName: string }`: the model must call the specified tool
+     *
+     * Note: When used without `prepareStep`, this applies to **every step** in the
+     * `maxSteps` loop. Using `"required"` or `{ type: "tool" }` without `prepareStep`
+     * will cause infinite tool calls until `maxSteps` is exhausted.
+     */
+    toolChoice?: ToolChoice<Record<string, Tool>>;
+    /**
+     * Optional callback that runs before each step in a multi-step generation.
+     * Allows dynamically changing `toolChoice` and available tools per step.
+     *
+     * This is the recommended way to enforce specific tool calls on certain steps
+     * while allowing the model freedom on others.
+     *
+     * Maps to Vercel AI SDK's `experimental_prepareStep`.
+     *
+     * @example Force a specific tool on step 0, then switch to auto:
+     * ```typescript
+     * prepareStep: ({ stepNumber, steps }) => {
+     *   if (stepNumber === 0) {
+     *     return {
+     *       toolChoice: { type: 'tool', toolName: 'myTool' }
+     *     };
+     *   }
+     *   return { toolChoice: 'auto' };
+     * }
+     * ```
+     *
+     * @see https://ai-sdk.dev/docs/reference/ai-sdk-core/generate-text#parameters
+     */
+    prepareStep?: (options: {
+        steps: StepResult<Record<string, Tool>>[];
+        stepNumber: number;
+        maxSteps: number;
+        model: LanguageModel;
+    }) => PromiseLike<{
+        model?: LanguageModel;
+        toolChoice?: ToolChoice<Record<string, Tool>>;
+        experimental_activeTools?: string[];
+    } | undefined>;
     /**
      * Text-to-Speech (TTS) configuration
      *

package/dist/lib/utils/videoAnalysisProcessor.d.ts

@@ -0,0 +1,30 @@
+/**
+ * Video Analysis Processor
+ *
+ * Formats video analysis results into human-readable text
+ *
+ * @module utils/videoAnalysisProcessor
+ */
+import type { CoreMessage } from "ai";
+import { AIProviderName } from "../constants/enums.js";
+/**
+ * Check if messages contain video frames (images)
+ *
+ * @param messages - Array of CoreMessage objects
+ * @returns true if video frames are present
+ */
+export declare function hasVideoFrames(messages: CoreMessage[]): boolean;
+/**
+ * Execute video analysis on messages containing video frames
+ *
+ * @param messages - Array of CoreMessage objects with video frames
+ * @param options - Video analysis options
+ * @returns Video analysis text result
+ * @throws Error if analysis fails
+ */
+export declare function executeVideoAnalysis(messages: CoreMessage[], options: {
+    provider?: AIProviderName | string;
+    providerName?: AIProviderName;
+    region?: string;
+    model?: string;
+}): Promise<string>;

package/dist/lib/utils/videoAnalysisProcessor.js

@@ -0,0 +1,59 @@
+/**
+ * Video Analysis Processor
+ *
+ * Formats video analysis results into human-readable text
+ *
+ * @module utils/videoAnalysisProcessor
+ */
+import { AIProviderName } from "../constants/enums.js";
+import { logger } from "./logger.js";
+/**
+ * Check if messages contain video frames (images)
+ *
+ * @param messages - Array of CoreMessage objects
+ * @returns true if video frames are present
+ */
+export function hasVideoFrames(messages) {
+    return messages.some((msg) => {
+        if (Array.isArray(msg.content)) {
+            return msg.content.some((part) => typeof part === "object" &&
+                part !== null &&
+                "type" in part &&
+                part.type === "image");
+        }
+        return false;
+    });
+}
+/**
+ * Execute video analysis on messages containing video frames
+ *
+ * @param messages - Array of CoreMessage objects with video frames
+ * @param options - Video analysis options
+ * @returns Video analysis text result
+ * @throws Error if analysis fails
+ */
+export async function executeVideoAnalysis(messages, options) {
+    logger.debug("[VideoAnalysisProcessor] Video frames detected, triggering analysis");
+    const { analyzeVideo } = await import("../adapters/video/videoAnalyzer.js");
+    const provider = options.provider === AIProviderName.GOOGLE_AI ||
+        (options.provider === AIProviderName.AUTO && process.env.GOOGLE_AI_API_KEY)
+        ? AIProviderName.GOOGLE_AI
+        : options.provider === AIProviderName.VERTEX ||
+            options.providerName === AIProviderName.VERTEX
+            ? AIProviderName.VERTEX
+            : AIProviderName.AUTO;
+    const videoAnalysisText = await analyzeVideo(messages[0], {
+        provider: provider,
+        project: options.region
+            ? undefined
+            : process.env.GOOGLE_VERTEX_PROJECT || process.env.GOOGLE_CLOUD_PROJECT,
+        location: options.region || process.env.GOOGLE_VERTEX_LOCATION,
+        model: options.model || "gemini-2.0-flash",
+    });
+    logger.debug("[VideoAnalysisProcessor] Video analysis completed", {
+        hasResult: !!videoAnalysisText,
+        resultLength: videoAnalysisText?.length,
+    });
+    return videoAnalysisText;
+}
+//# sourceMappingURL=videoAnalysisProcessor.js.map

package/dist/neurolink.js

@@ -1570,6 +1570,9 @@ Current user's request: ${currentInput}`;
                 disableTools: options.disableTools,
                 toolFilter: options.toolFilter,
                 excludeTools: options.excludeTools,
+                maxSteps: options.maxSteps,
+                toolChoice: options.toolChoice,
+                prepareStep: options.prepareStep,
                 enableAnalytics: options.enableAnalytics,
                 enableEvaluation: options.enableEvaluation,
                 context: options.context,

package/dist/processors/media/VideoProcessor.js

@@ -529,7 +529,7 @@ export class VideoProcessor extends BaseFileProcessor {
         // Extract frames using ffmpeg
         const framesDir = join(tempDir, "frames");
         await fs.mkdir(framesDir, { recursive: true });
-        await this.runFfmpegFrameExtraction(videoPath, framesDir, timestamps);
+        await this.runFfmpegFrameExtraction(videoPath, framesDir, timestamps, intervalSec);
         // Read extracted frames and resize with sharp
         const keyframes = [];
         for (let i = 0; i < timestamps.length; i++) {
@@ -563,15 +563,11 @@ export class VideoProcessor extends BaseFileProcessor {
      * @param outputDir - Directory to write frame files
      * @param timestamps - Array of timestamps in seconds
      */
-    runFfmpegFrameExtraction(videoPath, outputDir, timestamps) {
+    runFfmpegFrameExtraction(videoPath, outputDir, timestamps, intervalSec) {
         return new Promise((resolve, reject) => {
-            // Build select filter expression: select='eq(n,0)+eq(n,250)+...'
-            // Instead, use fps filter for simpler approach - extract at regular intervals
-            // using -vf fps=1/interval approach which is more reliable
-            // Build timestamp-based filter
-            const selectExpr = timestamps
-                .map((t) => `gte(t\\,${t})*lt(t\\,${t + 0.5})`)
-                .join("+");
+            // Improved select expression to pick exactly one frame per interval
+            // instead of multiple frames within a 0.5s window.
+            const selectExpr = `isnan(prev_selected_t)+gte(t-prev_selected_t,${intervalSec}-0.001)`;
             const timeoutId = setTimeout(() => {
                 reject(new Error(`ffmpeg frame extraction timed out after ${VIDEO_CONFIG.FFMPEG_TIMEOUT_MS}ms`));
             }, VIDEO_CONFIG.FFMPEG_TIMEOUT_MS);
@@ -861,19 +857,20 @@ export class VideoProcessor extends BaseFileProcessor {
             }
             const clampedCount = Math.min(frameCount, VIDEO_CONFIG.MAX_FRAMES);
             const timestamps = [];
+            let interval = duration;
             if (clampedCount === 1) {
                 timestamps.push(startSec);
             }
             else {
-                const step = duration / (clampedCount - 1);
+                interval = duration / (clampedCount - 1);
                 for (let i = 0; i < clampedCount; i++) {
-                    timestamps.push(startSec + step * i);
+                    timestamps.push(startSec + interval * i);
                 }
             }
             // Extract frames
             const framesDir = join(tempDir, "frames");
             await fs.mkdir(framesDir, { recursive: true });
-            await this.runFfmpegFrameExtraction(tempVideoPath, framesDir, timestamps);
+            await this.runFfmpegFrameExtraction(tempVideoPath, framesDir, timestamps, interval);
             // Read and resize frames
             const keyframes = [];
             for (let i = 0; i < timestamps.length; i++) {

package/dist/types/generateTypes.d.ts

@@ -1,4 +1,4 @@
-import type { Schema, Tool } from "ai";
+import type { Schema, Tool, ToolChoice, StepResult, LanguageModel } from "ai";
 import type { AIProviderName } from "../constants/enums.js";
 import type { RAGConfig } from "../rag/types.js";
 import type { AnalyticsData, TokenUsage } from "./analytics.js";
@@ -248,6 +248,55 @@ export type GenerateOptions = {
      * Default: false (backward compatible — tool schemas are injected into system prompt).
      */
     skipToolPromptInjection?: boolean;
+    /** Maximum number of tool execution steps (default: 200) */
+    maxSteps?: number;
+    /**
+     * Tool choice configuration for the generation.
+     * Controls whether and which tools the model must call.
+     *
+     * - `"auto"` (default): the model can choose whether and which tools to call
+     * - `"none"`: no tool calls allowed
+     * - `"required"`: the model must call at least one tool
+     * - `{ type: "tool", toolName: string }`: the model must call the specified tool
+     *
+     * Note: When used without `prepareStep`, this applies to **every step** in the
+     * `maxSteps` loop. Using `"required"` or `{ type: "tool" }` without `prepareStep`
+     * will cause infinite tool calls until `maxSteps` is exhausted.
+     */
+    toolChoice?: ToolChoice<Record<string, Tool>>;
+    /**
+     * Optional callback that runs before each step in a multi-step generation.
+     * Allows dynamically changing `toolChoice` and available tools per step.
+     *
+     * This is the recommended way to enforce specific tool calls on certain steps
+     * while allowing the model freedom on others.
+     *
+     * Maps to Vercel AI SDK's `experimental_prepareStep`.
+     *
+     * @example Force a specific tool on step 0, then switch to auto:
+     * ```typescript
+     * prepareStep: ({ stepNumber, steps }) => {
+     *   if (stepNumber === 0) {
+     *     return {
+     *       toolChoice: { type: 'tool', toolName: 'myTool' }
+     *     };
+     *   }
+     *   return { toolChoice: 'auto' };
+     * }
+     * ```
+     *
+     * @see https://ai-sdk.dev/docs/reference/ai-sdk-core/generate-text#parameters
+     */
+    prepareStep?: (options: {
+        steps: StepResult<Record<string, Tool>>[];
+        stepNumber: number;
+        maxSteps: number;
+        model: LanguageModel;
+    }) => PromiseLike<{
+        model?: LanguageModel;
+        toolChoice?: ToolChoice<Record<string, Tool>>;
+        experimental_activeTools?: string[];
+    } | undefined>;
     enableEvaluation?: boolean;
     enableAnalytics?: boolean;
     context?: StandardRecord;
@@ -521,6 +570,7 @@ export type TextGenerationOptions = {
          */
         images?: Array<Buffer | string | import("./content.js").ImageWithAltText>;
         pdfFiles?: Array<Buffer | string>;
+        files?: Array<Buffer | string | import("./fileTypes.js").FileWithMetadata>;
     };
     provider?: AIProviderName;
     model?: string;
@@ -568,6 +618,53 @@ export type TextGenerationOptions = {
     toolFilter?: string[];
     /** Exclude these tools by name (blacklist). Applied after toolFilter. */
     excludeTools?: string[];
+    /**
+     * Tool choice configuration for the generation.
+     * Controls whether and which tools the model must call.
+     *
+     * - `"auto"` (default): the model can choose whether and which tools to call
+     * - `"none"`: no tool calls allowed
+     * - `"required"`: the model must call at least one tool
+     * - `{ type: "tool", toolName: string }`: the model must call the specified tool
+     *
+     * Note: When used without `prepareStep`, this applies to **every step** in the
+     * `maxSteps` loop. Using `"required"` or `{ type: "tool" }` without `prepareStep`
+     * will cause infinite tool calls until `maxSteps` is exhausted.
+     */
+    toolChoice?: ToolChoice<Record<string, Tool>>;
+    /**
+     * Optional callback that runs before each step in a multi-step generation.
+     * Allows dynamically changing `toolChoice` and available tools per step.
+     *
+     * This is the recommended way to enforce specific tool calls on certain steps
+     * while allowing the model freedom on others.
+     *
+     * Maps to Vercel AI SDK's `experimental_prepareStep`.
+     *
+     * @example Force a specific tool on step 0, then switch to auto:
+     * ```typescript
+     * prepareStep: ({ stepNumber, steps }) => {
+     *   if (stepNumber === 0) {
+     *     return {
+     *       toolChoice: { type: 'tool', toolName: 'myTool' }
+     *     };
+     *   }
+     *   return { toolChoice: 'auto' };
+     * }
+     * ```
+     *
+     * @see https://ai-sdk.dev/docs/reference/ai-sdk-core/generate-text#parameters
+     */
+    prepareStep?: (options: {
+        steps: StepResult<Record<string, Tool>>[];
+        stepNumber: number;
+        maxSteps: number;
+        model: LanguageModel;
+    }) => PromiseLike<{
+        model?: LanguageModel;
+        toolChoice?: ToolChoice<Record<string, Tool>>;
+        experimental_activeTools?: string[];
+    } | undefined>;
     /**
      * Text-to-Speech (TTS) configuration
      *

package/dist/utils/videoAnalysisProcessor.d.ts

@@ -0,0 +1,30 @@
+/**
+ * Video Analysis Processor
+ *
+ * Formats video analysis results into human-readable text
+ *
+ * @module utils/videoAnalysisProcessor
+ */
+import type { CoreMessage } from "ai";
+import { AIProviderName } from "../constants/enums.js";
+/**
+ * Check if messages contain video frames (images)
+ *
+ * @param messages - Array of CoreMessage objects
+ * @returns true if video frames are present
+ */
+export declare function hasVideoFrames(messages: CoreMessage[]): boolean;
+/**
+ * Execute video analysis on messages containing video frames
+ *
+ * @param messages - Array of CoreMessage objects with video frames
+ * @param options - Video analysis options
+ * @returns Video analysis text result
+ * @throws Error if analysis fails
+ */
+export declare function executeVideoAnalysis(messages: CoreMessage[], options: {
+    provider?: AIProviderName | string;
+    providerName?: AIProviderName;
+    region?: string;
+    model?: string;
+}): Promise<string>;

package/dist/utils/videoAnalysisProcessor.js

@@ -0,0 +1,58 @@
+/**
+ * Video Analysis Processor
+ *
+ * Formats video analysis results into human-readable text
+ *
+ * @module utils/videoAnalysisProcessor
+ */
+import { AIProviderName } from "../constants/enums.js";
+import { logger } from "./logger.js";
+/**
+ * Check if messages contain video frames (images)
+ *
+ * @param messages - Array of CoreMessage objects
+ * @returns true if video frames are present
+ */
+export function hasVideoFrames(messages) {
+    return messages.some((msg) => {
+        if (Array.isArray(msg.content)) {
+            return msg.content.some((part) => typeof part === "object" &&
+                part !== null &&
+                "type" in part &&
+                part.type === "image");
+        }
+        return false;
+    });
+}
+/**
+ * Execute video analysis on messages containing video frames
+ *
+ * @param messages - Array of CoreMessage objects with video frames
+ * @param options - Video analysis options
+ * @returns Video analysis text result
+ * @throws Error if analysis fails
+ */
+export async function executeVideoAnalysis(messages, options) {
+    logger.debug("[VideoAnalysisProcessor] Video frames detected, triggering analysis");
+    const { analyzeVideo } = await import("../adapters/video/videoAnalyzer.js");
+    const provider = options.provider === AIProviderName.GOOGLE_AI ||
+        (options.provider === AIProviderName.AUTO && process.env.GOOGLE_AI_API_KEY)
+        ? AIProviderName.GOOGLE_AI
+        : options.provider === AIProviderName.VERTEX ||
+            options.providerName === AIProviderName.VERTEX
+            ? AIProviderName.VERTEX
+            : AIProviderName.AUTO;
+    const videoAnalysisText = await analyzeVideo(messages[0], {
+        provider: provider,
+        project: options.region
+            ? undefined
+            : process.env.GOOGLE_VERTEX_PROJECT || process.env.GOOGLE_CLOUD_PROJECT,
+        location: options.region || process.env.GOOGLE_VERTEX_LOCATION,
+        model: options.model || "gemini-2.0-flash",
+    });
+    logger.debug("[VideoAnalysisProcessor] Video analysis completed", {
+        hasResult: !!videoAnalysisText,
+        resultLength: videoAnalysisText?.length,
+    });
+    return videoAnalysisText;
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@juspay/neurolink",
-  "version": "9.8.0",
+  "version": "9.10.0",
   "description": "Universal AI Development Platform with working MCP integration, multi-provider support, and professional CLI. Built-in tools operational, 58+ external MCP servers discoverable. Connect to filesystem, GitHub, database operations, and more. Build, test, and deploy AI applications with 13 providers: OpenAI, Anthropic, Google AI, AWS Bedrock, Azure, Hugging Face, Ollama, and Mistral AI.",
   "author": {
     "name": "Juspay Technologies",