@r16t/multimodal-mcp 1.0.0

package/README.md

@@ -0,0 +1,178 @@
+# multimodal-mcp
+
+Multi-provider media generation MCP server. Generate images, videos, and audio from text prompts using OpenAI, xAI, and Google through a single unified interface.
+
+## Features
+
+- 🎨 **Image Generation** — Generate images via OpenAI (gpt-image-1), xAI (grok-imagine-image), or Google (imagen-4)
+- 🎬 **Video Generation** — Generate videos via OpenAI (sora-2), xAI (grok-imagine-video), or Google (veo-3.1)
+- 🔊 **Audio Generation** — Text-to-speech via OpenAI (tts-1) or Google (gemini-2.5-flash-preview-tts)
+- 🔄 **Auto-Discovery** — Automatically detects configured providers from environment variables
+- 🎯 **Provider Selection** — Auto-selects or explicitly choose a provider per request
+- 📁 **File Output** — Saves all generated media to disk with descriptive filenames
+
+## Quick Start
+
+### Claude Desktop
+
+Add to your Claude Desktop config (`~/Library/Application Support/Claude/claude_desktop_config.json`):
+
+```json
+{
+  "mcpServers": {
+    "multimodal-mcp": {
+      "command": "npx",
+      "args": ["@r16t/multimodal-mcp"],
+      "env": {
+        "OPENAI_API_KEY": "sk-...",
+        "XAI_API_KEY": "xai-...",
+        "GOOGLE_API_KEY": "AIza...",
+        "MEDIA_OUTPUT_DIR": "/tmp/media"
+      }
+    }
+  }
+}
+```
+
+You only need to set keys for the providers you want to use. At least one is required.
+
+### Cursor / Other MCP Clients
+
+```json
+{
+  "mcpServers": {
+    "multimodal-mcp": {
+      "command": "npx",
+      "args": ["@r16t/multimodal-mcp"],
+      "env": {
+        "OPENAI_API_KEY": "sk-..."
+      }
+    }
+  }
+}
+```
+
+## Environment Variables
+
+| Variable | Required | Description |
+|----------|----------|-------------|
+| `OPENAI_API_KEY` | At least one provider key | OpenAI API key — enables image, video, and audio generation via gpt-image-1, sora-2, and tts-1 |
+| `XAI_API_KEY` | At least one provider key | xAI API key — enables image and video generation via grok-imagine-image and grok-imagine-video |
+| `GOOGLE_API_KEY` | At least one provider key | Google API key — enables image, video, and audio generation via imagen-4, veo-3.1, and gemini-2.5-flash-preview-tts |
+| `GEMINI_API_KEY` | — | Alias for `GOOGLE_API_KEY`; either name is accepted |
+| `MEDIA_OUTPUT_DIR` | No | Directory for saved media files. Defaults to the system temp directory |
+
+## Available Tools
+
+### `generate_image`
+
+Generate an image from a text prompt.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `prompt` | string | Yes | Text description of the image to generate |
+| `provider` | string | No | Provider to use: `openai`, `xai`, `google`. Auto-selects if omitted |
+| `aspectRatio` | string | No | Aspect ratio: `1:1`, `16:9`, `9:16`, `4:3`, `3:4` |
+| `quality` | string | No | Quality level: `low`, `standard`, `high` |
+| `providerOptions` | object | No | Provider-specific parameters passed through directly |
+
+### `generate_video`
+
+Generate a video from a text prompt. Video generation is asynchronous and may take several minutes.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `prompt` | string | Yes | Text description of the video to generate |
+| `provider` | string | No | Provider to use: `openai`, `xai`, `google`. Auto-selects if omitted |
+| `duration` | number | No | Video duration in seconds (provider limits apply) |
+| `aspectRatio` | string | No | Aspect ratio: `16:9`, `9:16`, `1:1` |
+| `resolution` | string | No | Resolution: `480p`, `720p`, `1080p` |
+| `providerOptions` | object | No | Provider-specific parameters passed through directly |
+
+### `generate_audio`
+
+Generate audio (text-to-speech) from text. Audio generation is synchronous.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `text` | string | Yes | Text to convert to speech |
+| `provider` | string | No | Provider to use: `openai`, `google`. Auto-selects if omitted |
+| `voice` | string | No | Voice name (provider-specific). OpenAI: `alloy`, `ash`, `coral`, `echo`, `fable`, `nova`, `onyx`, `sage`, `shimmer`. Google: `Kore`, `Charon`, `Fenrir`, `Aoede`, `Puck`, etc. |
+| `speed` | number | No | Speech speed multiplier (OpenAI only): `0.25` to `4.0` |
+| `format` | string | No | Output format (OpenAI only): `mp3`, `opus`, `aac`, `flac`, `wav`, `pcm` |
+| `providerOptions` | object | No | Provider-specific parameters passed through directly |
+
+### `list_providers`
+
+List all configured media generation providers and their capabilities. Takes no parameters.
+
+## Provider Capabilities
+
+| Provider | Image | Video | Audio | Image Model | Video Model | Audio Model |
+|----------|:-----:|:-----:|:-----:|-------------|-------------|-------------|
+| OpenAI | ✅ | ✅ | ✅ | gpt-image-1 | sora-2 | tts-1 |
+| xAI | ✅ | ✅ | — | grok-imagine-image | grok-imagine-video | — |
+| Google | ✅ | ✅ | ✅ | imagen-4 | veo-3.1 | gemini-2.5-flash-preview-tts |
+
+### Image Aspect Ratios
+
+| Provider | 1:1 | 16:9 | 9:16 | 4:3 | 3:4 |
+|----------|:---:|:----:|:----:|:---:|:---:|
+| OpenAI | ✅ | ✅ | ✅ | ✅ | ✅ |
+| xAI | ✅ | ✅ | ✅ | ✅ | ✅ |
+| Google | ✅ | ✅ | ✅ | ✅ | ✅ |
+
+### Video Aspect Ratios & Resolutions
+
+| Provider | 16:9 | 9:16 | 1:1 | 480p | 720p | 1080p |
+|----------|:----:|:----:|:---:|:----:|:----:|:-----:|
+| OpenAI | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+| xAI | ✅ | ✅ | ✅ | — | ✅ | ✅ |
+| Google | ✅ | ✅ | — | — | ✅ | ✅ |
+
+### Audio Formats
+
+| Provider | mp3 | opus | aac | flac | wav | pcm |
+|----------|:---:|:----:|:---:|:----:|:---:|:---:|
+| OpenAI | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+| Google | — | — | — | — | ✅ | — |
+
+## Troubleshooting
+
+### No providers configured
+
+```
+[config] No provider API keys detected
+```
+
+Set at least one of `OPENAI_API_KEY`, `XAI_API_KEY`, or `GOOGLE_API_KEY` in the MCP server's `env` block.
+
+### Provider not available for requested media type
+
+All three providers support image and video generation. Audio generation (text-to-speech) is supported by OpenAI and Google. xAI does not currently offer a standalone TTS API. If you specify a `provider` that isn't configured (no API key) or doesn't support the requested media type, you'll receive an error. Omit the `provider` parameter to auto-select from configured providers.
+
+### Video generation timeout
+
+Video generation polls for up to 10 minutes. If your video hasn't completed in that window, the request will fail with a timeout error. Try a shorter `duration` or a simpler `prompt`.
+
+### xAI image generation returned no data
+
+This indicates the xAI API returned an empty response. Check that your `XAI_API_KEY` is valid and that your prompt does not violate xAI content policies.
+
+### Google image/video generation failed: 403
+
+Verify your `GOOGLE_API_KEY` has the Generative Language API enabled in Google Cloud Console.
+
+## Development
+
+```bash
+npm run build      # Compile TypeScript to build/
+npm test           # Run tests with Vitest
+npm run lint       # Lint and auto-fix with ESLint
+npm run typecheck  # Type-check without emitting
+npm run dev        # Watch mode for TypeScript compilation
+```
+
+## License
+
+MIT

package/build/config.d.ts

@@ -0,0 +1,10 @@
+import { z } from "zod";
+declare const configSchema: z.ZodObject<{
+    openaiApiKey: z.ZodOptional<z.ZodString>;
+    xaiApiKey: z.ZodOptional<z.ZodString>;
+    googleApiKey: z.ZodOptional<z.ZodString>;
+    outputDirectory: z.ZodString;
+}, z.core.$strip>;
+export type Config = z.infer<typeof configSchema>;
+export declare function loadConfig(): Config;
+export {};

package/build/config.js

@@ -0,0 +1,33 @@
+import { z } from "zod";
+import { tmpdir } from "node:os";
+const configSchema = z.object({
+    openaiApiKey: z.string().optional(),
+    xaiApiKey: z.string().optional(),
+    googleApiKey: z.string().optional(),
+    outputDirectory: z.string(),
+});
+function resolveGoogleKey() {
+    return process.env.GOOGLE_API_KEY || process.env.GEMINI_API_KEY || undefined;
+}
+export function loadConfig() {
+    const config = configSchema.parse({
+        openaiApiKey: process.env.OPENAI_API_KEY || undefined,
+        xaiApiKey: process.env.XAI_API_KEY || undefined,
+        googleApiKey: resolveGoogleKey(),
+        outputDirectory: process.env.MEDIA_OUTPUT_DIR || tmpdir(),
+    });
+    const detected = [];
+    if (config.openaiApiKey)
+        detected.push("OpenAI");
+    if (config.xaiApiKey)
+        detected.push("xAI");
+    if (config.googleApiKey)
+        detected.push("Google");
+    if (detected.length > 0) {
+        console.error(`[config] Detected providers: ${detected.join(", ")}`);
+    }
+    else {
+        console.error("[config] No provider API keys detected");
+    }
+    return config;
+}

package/build/errors.d.ts

@@ -0,0 +1 @@
+export declare function sanitizeError(error: unknown): string;

package/build/errors.js

@@ -0,0 +1,24 @@
+const API_KEY_PATTERNS = [
+    /sk-[a-zA-Z0-9_-]{10,}/g,
+    /xai-[a-zA-Z0-9_-]{10,}/g,
+    /AIzaSy[a-zA-Z0-9_-]{10,}/g,
+    /key=[a-zA-Z0-9_-]{20,}/g,
+];
+export function sanitizeError(error) {
+    let message;
+    if (error instanceof Error) {
+        message = error.message;
+    }
+    else if (typeof error === "string" && error.length > 0) {
+        message = error;
+    }
+    else {
+        return "Unknown error";
+    }
+    for (const pattern of API_KEY_PATTERNS) {
+        message = message.replace(pattern, "[REDACTED]");
+    }
+    message = message.replace(/\n\s+at .+/g, "");
+    message = message.replace(/\/[^\s:]+\.[tj]s(:\d+)?(:\d+)?/g, "[internal]");
+    return message.trim() || "Unknown error";
+}

package/build/file-manager.d.ts

@@ -0,0 +1,7 @@
+import type { GeneratedMedia } from "./providers/types.js";
+export declare class FileManager {
+    private readonly outputDirectory;
+    constructor(outputDirectory: string);
+    save(media: GeneratedMedia, type: "image" | "video" | "audio"): Promise<string>;
+    private getExtension;
+}

package/build/file-manager.js

@@ -0,0 +1,37 @@
+import { mkdir, writeFile } from "node:fs/promises";
+import { join, resolve } from "node:path";
+import { randomBytes } from "node:crypto";
+export class FileManager {
+    outputDirectory;
+    constructor(outputDirectory) {
+        this.outputDirectory = resolve(outputDirectory);
+    }
+    async save(media, type) {
+        await mkdir(this.outputDirectory, { recursive: true });
+        const extension = this.getExtension(type, media.mimeType);
+        const provider = media.metadata.provider || "unknown";
+        const timestamp = Date.now();
+        const random = randomBytes(4).toString("hex");
+        const filename = `${type}-${timestamp}-${provider}-${random}.${extension}`;
+        const filePath = join(this.outputDirectory, filename);
+        await writeFile(filePath, media.data);
+        return filePath;
+    }
+    getExtension(type, mimeType) {
+        if (type === "video")
+            return "mp4";
+        if (type === "audio") {
+            const audioExtensions = {
+                "audio/mpeg": "mp3",
+                "audio/opus": "opus",
+                "audio/aac": "aac",
+                "audio/flac": "flac",
+                "audio/wav": "wav",
+                "audio/pcm": "pcm",
+            };
+            return audioExtensions[mimeType] ?? "mp3";
+        }
+        const match = mimeType.match(/image\/(\w+)/);
+        return match ? match[1] : "png";
+    }
+}

package/build/index.d.ts

@@ -0,0 +1,2 @@
+#!/usr/bin/env node
+export {};

package/build/index.js

@@ -0,0 +1,16 @@
+#!/usr/bin/env node
+import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
+import { loadConfig } from "./config.js";
+import { createServer } from "./server.js";
+async function main() {
+    console.error("[multimodal-mcp] Starting server...");
+    const config = loadConfig();
+    const server = createServer(config);
+    const transport = new StdioServerTransport();
+    await server.connect(transport);
+    console.error("[multimodal-mcp] Server connected and ready");
+}
+main().catch((error) => {
+    console.error("[multimodal-mcp] Fatal error:", error);
+    process.exit(1);
+});

package/build/providers/google.d.ts

@@ -0,0 +1,10 @@
+import type { MediaProvider, ProviderCapabilities, ImageParams, VideoParams, AudioParams, GeneratedMedia } from "./types.js";
+export declare class GoogleProvider implements MediaProvider {
+    readonly name = "google";
+    readonly capabilities: ProviderCapabilities;
+    private apiKey;
+    constructor(apiKey: string);
+    generateImage(params: ImageParams): Promise<GeneratedMedia>;
+    generateVideo(params: VideoParams): Promise<GeneratedMedia>;
+    generateAudio(params: AudioParams): Promise<GeneratedMedia>;
+}

package/build/providers/google.js

@@ -0,0 +1,105 @@
+import { pollForCompletion } from "./polling.js";
+const GEMINI_BASE_URL = "https://generativelanguage.googleapis.com/v1beta";
+export class GoogleProvider {
+    name = "google";
+    capabilities = {
+        supportsImageGeneration: true,
+        supportsVideoGeneration: true,
+        supportsAudioGeneration: true,
+        supportedImageAspectRatios: ["1:1", "16:9", "9:16", "4:3", "3:4"],
+        supportedVideoAspectRatios: ["16:9", "9:16"],
+        supportedVideoResolutions: ["720p", "1080p"],
+        supportedAudioFormats: ["wav"],
+        maxVideoDurationSeconds: 8,
+    };
+    apiKey;
+    constructor(apiKey) {
+        this.apiKey = apiKey;
+    }
+    async generateImage(params) {
+        const response = await fetch(`${GEMINI_BASE_URL}/models/imagen-4:generateImages?key=${this.apiKey}`, {
+            method: "POST",
+            headers: { "Content-Type": "application/json" },
+            body: JSON.stringify({
+                prompt: params.prompt,
+                config: {
+                    aspectRatio: params.aspectRatio,
+                    ...params.providerOptions,
+                },
+            }),
+        });
+        if (!response.ok) {
+            throw new Error(`Google image generation failed: ${response.status}`);
+        }
+        const result = (await response.json());
+        const base64 = result.generatedImages[0].image.bytesBase64Encoded;
+        return {
+            data: Buffer.from(base64, "base64"),
+            mimeType: "image/png",
+            metadata: { model: "imagen-4", provider: "google" },
+        };
+    }
+    async generateVideo(params) {
+        const submitResponse = await fetch(`${GEMINI_BASE_URL}/models/veo-3.1:predictLongRunning?key=${this.apiKey}`, {
+            method: "POST",
+            headers: { "Content-Type": "application/json" },
+            body: JSON.stringify({
+                prompt: params.prompt,
+                config: {
+                    aspectRatio: params.aspectRatio,
+                    durationSeconds: params.duration,
+                    ...params.providerOptions,
+                },
+            }),
+        });
+        if (!submitResponse.ok) {
+            throw new Error(`Google video generation failed: ${submitResponse.status}`);
+        }
+        const operation = (await submitResponse.json());
+        const result = await pollForCompletion(async () => {
+            const statusResponse = await fetch(`${GEMINI_BASE_URL}/${operation.name}?key=${this.apiKey}`);
+            return statusResponse.json();
+        }, (status) => status.done === true, { timeoutMs: 600_000, intervalMs: 5_000 });
+        const videoUri = result.response.videos[0].uri;
+        const videoResponse = await fetch(videoUri);
+        const data = Buffer.from(await videoResponse.arrayBuffer());
+        return {
+            data,
+            mimeType: "video/mp4",
+            metadata: { model: "veo-3.1", provider: "google", operationName: operation.name },
+        };
+    }
+    async generateAudio(params) {
+        const voice = params.voice ?? "Kore";
+        const model = "gemini-2.5-flash-preview-tts";
+        const response = await fetch(`${GEMINI_BASE_URL}/models/${model}:generateContent?key=${this.apiKey}`, {
+            method: "POST",
+            headers: { "Content-Type": "application/json" },
+            body: JSON.stringify({
+                contents: [{ parts: [{ text: params.text }] }],
+                generationConfig: {
+                    response_modalities: ["AUDIO"],
+                    speech_config: {
+                        voiceConfig: {
+                            prebuiltVoiceConfig: { voiceName: voice },
+                        },
+                    },
+                    ...params.providerOptions,
+                },
+            }),
+        });
+        if (!response.ok) {
+            throw new Error(`Google audio generation failed: ${response.status}`);
+        }
+        const result = (await response.json());
+        const audioPart = result.candidates[0]?.content?.parts?.find((part) => part.inlineData !== undefined);
+        if (!audioPart?.inlineData) {
+            throw new Error("Google audio generation returned no audio data");
+        }
+        return {
+            data: Buffer.from(audioPart.inlineData.data, "base64"),
+            mimeType: audioPart.inlineData.mimeType || "audio/wav",
+            metadata: { model, provider: "google", voice },
+        };
+    }
+}

package/build/providers/openai.d.ts

@@ -0,0 +1,12 @@
+import type { MediaProvider, ProviderCapabilities, ImageParams, VideoParams, AudioParams, GeneratedMedia } from "./types.js";
+export declare class OpenAIProvider implements MediaProvider {
+    readonly name = "openai";
+    readonly capabilities: ProviderCapabilities;
+    private client;
+    constructor(apiKey: string);
+    generateImage(params: ImageParams): Promise<GeneratedMedia>;
+    generateVideo(params: VideoParams): Promise<GeneratedMedia>;
+    generateAudio(params: AudioParams): Promise<GeneratedMedia>;
+    private audioFormatToMimeType;
+    private mapAspectRatioToSize;
+}

package/build/providers/openai.js

@@ -0,0 +1,92 @@
+import OpenAI from "openai";
+import { pollForCompletion } from "./polling.js";
+const ASPECT_RATIO_TO_SIZE = {
+    "1:1": "1024x1024",
+    "16:9": "1536x1024",
+    "9:16": "1024x1536",
+    "4:3": "1024x768",
+    "3:4": "768x1024",
+};
+export class OpenAIProvider {
+    name = "openai";
+    capabilities = {
+        supportsImageGeneration: true,
+        supportsVideoGeneration: true,
+        supportsAudioGeneration: true,
+        supportedImageAspectRatios: ["1:1", "16:9", "9:16", "4:3", "3:4"],
+        supportedVideoAspectRatios: ["16:9", "9:16", "1:1"],
+        supportedVideoResolutions: ["480p", "720p", "1080p"],
+        supportedAudioFormats: ["mp3", "opus", "aac", "flac", "wav", "pcm"],
+        maxVideoDurationSeconds: 20,
+    };
+    client;
+    constructor(apiKey) {
+        this.client = new OpenAI({ apiKey });
+    }
+    async generateImage(params) {
+        const response = await this.client.images.generate({
+            model: "gpt-image-1",
+            prompt: params.prompt,
+            size: this.mapAspectRatioToSize(params.aspectRatio),
+            quality: params.quality === "high" ? "hd" : "standard",
+            response_format: "b64_json",
+            ...params.providerOptions,
+        });
+        const base64Data = response.data[0].b64_json;
+        return {
+            data: Buffer.from(base64Data, "base64"),
+            mimeType: "image/png",
+            metadata: { model: "gpt-image-1", provider: "openai" },
+        };
+    }
+    async generateVideo(params) {
+        const videos = this.client.videos;
+        const job = await videos.create({
+            model: "sora-2",
+            prompt: params.prompt,
+            duration: params.duration,
+            ...params.providerOptions,
+        });
+        const result = await pollForCompletion(() => videos.retrieve(job.id), (status) => status.status === "completed", { timeoutMs: 600_000, intervalMs: 5_000 });
+        const videoUrl = result.url;
+        const videoResponse = await fetch(videoUrl);
+        const data = Buffer.from(await videoResponse.arrayBuffer());
+        return {
+            data,
+            mimeType: "video/mp4",
+            metadata: { model: "sora-2", provider: "openai", jobId: job.id },
+        };
+    }
+    async generateAudio(params) {
+        const format = params.format ?? "mp3";
+        const voice = params.voice ?? "alloy";
+        const response = await this.client.audio.speech.create({
+            model: "tts-1",
+            input: params.text,
+            voice: voice,
+            response_format: format,
+            speed: params.speed ?? 1.0,
+            ...params.providerOptions,
+        });
+        const data = Buffer.from(await response.arrayBuffer());
+        return {
+            data,
+            mimeType: this.audioFormatToMimeType(format),
+            metadata: { model: "tts-1", provider: "openai", voice, format },
+        };
+    }
+    audioFormatToMimeType(format) {
+        const mimeTypes = {
+            mp3: "audio/mpeg",
+            opus: "audio/opus",
+            aac: "audio/aac",
+            flac: "audio/flac",
+            wav: "audio/wav",
+            pcm: "audio/pcm",
+        };
+        return mimeTypes[format] ?? "audio/mpeg";
+    }
+    mapAspectRatioToSize(aspectRatio) {
+        return ASPECT_RATIO_TO_SIZE[aspectRatio] ?? "1024x1024";
+    }
+}

package/build/providers/polling.d.ts

@@ -0,0 +1,6 @@
+export interface PollOptions {
+    timeoutMs: number;
+    intervalMs: number;
+    onPoll?: (attempt: number) => void;
+}
+export declare function pollForCompletion<T>(checkStatus: () => Promise<T>, isComplete: (result: T) => boolean, options: PollOptions): Promise<T>;

package/build/providers/polling.js

@@ -0,0 +1,14 @@
+export async function pollForCompletion(checkStatus, isComplete, options) {
+    const startTime = Date.now();
+    let attempt = 0;
+    while (Date.now() - startTime < options.timeoutMs) {
+        const result = await checkStatus();
+        attempt++;
+        options.onPoll?.(attempt);
+        if (isComplete(result)) {
+            return result;
+        }
+        await new Promise((resolve) => setTimeout(resolve, options.intervalMs));
+    }
+    throw new Error(`Generation timed out after ${options.timeoutMs / 1000} seconds`);
+}

package/build/providers/registry.d.ts

@@ -0,0 +1,10 @@
+import type { MediaProvider, ProviderInfo } from "./types.js";
+export declare class ProviderRegistry {
+    private providers;
+    register(provider: MediaProvider): void;
+    getProvider(name?: string): MediaProvider | undefined;
+    getImageProviders(): MediaProvider[];
+    getVideoProviders(): MediaProvider[];
+    getAudioProviders(): MediaProvider[];
+    listCapabilities(): ProviderInfo[];
+}

package/build/providers/registry.js

@@ -0,0 +1,28 @@
+export class ProviderRegistry {
+    providers = new Map();
+    register(provider) {
+        this.providers.set(provider.name, provider);
+    }
+    getProvider(name) {
+        if (name) {
+            return this.providers.get(name);
+        }
+        const first = this.providers.values().next();
+        return first.done ? undefined : first.value;
+    }
+    getImageProviders() {
+        return [...this.providers.values()].filter((p) => p.capabilities.supportsImageGeneration);
+    }
+    getVideoProviders() {
+        return [...this.providers.values()].filter((p) => p.capabilities.supportsVideoGeneration);
+    }
+    getAudioProviders() {
+        return [...this.providers.values()].filter((p) => p.capabilities.supportsAudioGeneration);
+    }
+    listCapabilities() {
+        return [...this.providers.values()].map((p) => ({
+            name: p.name,
+            capabilities: p.capabilities,
+        }));
+    }
+}

package/build/providers/types.d.ts

@@ -0,0 +1,46 @@
+export interface MediaProvider {
+    readonly name: string;
+    readonly capabilities: ProviderCapabilities;
+    generateImage(params: ImageParams): Promise<GeneratedMedia>;
+    generateVideo(params: VideoParams): Promise<GeneratedMedia>;
+    generateAudio(params: AudioParams): Promise<GeneratedMedia>;
+}
+export interface ProviderCapabilities {
+    supportsImageGeneration: boolean;
+    supportsVideoGeneration: boolean;
+    supportsAudioGeneration: boolean;
+    supportedImageAspectRatios: string[];
+    supportedVideoAspectRatios: string[];
+    supportedVideoResolutions: string[];
+    supportedAudioFormats: string[];
+    maxVideoDurationSeconds: number;
+}
+export interface ImageParams {
+    prompt: string;
+    aspectRatio: string;
+    quality: string;
+    providerOptions?: Record<string, unknown>;
+}
+export interface VideoParams {
+    prompt: string;
+    duration: number;
+    aspectRatio: string;
+    resolution: string;
+    providerOptions?: Record<string, unknown>;
+}
+export interface AudioParams {
+    text: string;
+    voice?: string;
+    speed?: number;
+    format?: string;
+    providerOptions?: Record<string, unknown>;
+}
+export interface GeneratedMedia {
+    data: Buffer;
+    mimeType: string;
+    metadata: Record<string, unknown>;
+}
+export interface ProviderInfo {
+    name: string;
+    capabilities: ProviderCapabilities;
+}

package/build/providers/types.js

@@ -0,0 +1 @@
+export {};

package/build/providers/xai.d.ts

@@ -0,0 +1,11 @@
+import type { MediaProvider, ProviderCapabilities, ImageParams, VideoParams, AudioParams, GeneratedMedia } from "./types.js";
+export declare class XAIProvider implements MediaProvider {
+    readonly name = "xai";
+    readonly capabilities: ProviderCapabilities;
+    private client;
+    private apiKey;
+    constructor(apiKey: string);
+    generateImage(params: ImageParams): Promise<GeneratedMedia>;
+    generateVideo(params: VideoParams): Promise<GeneratedMedia>;
+    generateAudio(_params: AudioParams): Promise<GeneratedMedia>;
+}

package/build/providers/xai.js

@@ -0,0 +1,80 @@
+import OpenAI from "openai";
+import { pollForCompletion } from "./polling.js";
+const XAI_BASE_URL = "https://api.x.ai/v1";
+const IMAGE_MODEL = "grok-imagine-image";
+const VIDEO_MODEL = "grok-imagine-video";
+export class XAIProvider {
+    name = "xai";
+    capabilities = {
+        supportsImageGeneration: true,
+        supportsVideoGeneration: true,
+        supportsAudioGeneration: false,
+        supportedImageAspectRatios: ["1:1", "16:9", "9:16", "4:3", "3:4"],
+        supportedVideoAspectRatios: ["16:9", "9:16", "1:1"],
+        supportedVideoResolutions: ["720p", "1080p"],
+        supportedAudioFormats: [],
+        maxVideoDurationSeconds: 15,
+    };
+    client;
+    apiKey;
+    constructor(apiKey) {
+        this.apiKey = apiKey;
+        this.client = new OpenAI({ apiKey, baseURL: XAI_BASE_URL });
+    }
+    async generateImage(params) {
+        const response = await this.client.images.generate({
+            model: IMAGE_MODEL,
+            prompt: params.prompt,
+            response_format: "b64_json",
+            ...params.providerOptions,
+        });
+        const imageData = response.data?.[0];
+        if (!imageData?.b64_json) {
+            throw new Error("xAI image generation returned no data");
+        }
+        const base64Data = imageData.b64_json;
+        return {
+            data: Buffer.from(base64Data, "base64"),
+            mimeType: "image/png",
+            metadata: { model: IMAGE_MODEL, provider: "xai" },
+        };
+    }
+    async generateVideo(params) {
+        const submitResponse = await fetch(`${XAI_BASE_URL}/videos/generations`, {
+            method: "POST",
+            headers: {
+                "Content-Type": "application/json",
+                Authorization: `Bearer ${this.apiKey}`,
+            },
+            body: JSON.stringify({
+                model: VIDEO_MODEL,
+                prompt: params.prompt,
+                duration: params.duration,
+                aspect_ratio: params.aspectRatio,
+                ...params.providerOptions,
+            }),
+        });
+        if (!submitResponse.ok) {
+            throw new Error(`xAI video generation failed: ${submitResponse.status}`);
+        }
+        const submitResult = (await submitResponse.json());
+        const statusResult = await pollForCompletion(async () => {
+            const statusResponse = await fetch(`${XAI_BASE_URL}/videos/generations/${submitResult.request_id}`, { headers: { Authorization: `Bearer ${this.apiKey}` } });
+            return statusResponse.json();
+        }, (result) => result.status === "done", { timeoutMs: 600_000, intervalMs: 5_000 });
+        const videoResponse = await fetch(statusResult.video_url);
+        const data = Buffer.from(await videoResponse.arrayBuffer());
+        return {
+            data,
+            mimeType: "video/mp4",
+            metadata: {
+                model: VIDEO_MODEL,
+                provider: "xai",
+                requestId: submitResult.request_id,
+            },
+        };
+    }
+    async generateAudio(_params) {
+        throw new Error("xAI does not support audio generation");
+    }
+}

package/build/server.d.ts

@@ -0,0 +1,3 @@
+import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
+import type { Config } from "./config.js";
+export declare function createServer(config: Config): McpServer;

package/build/server.js

@@ -0,0 +1,58 @@
+import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
+import { z } from "zod";
+import { ProviderRegistry } from "./providers/registry.js";
+import { OpenAIProvider } from "./providers/openai.js";
+import { XAIProvider } from "./providers/xai.js";
+import { GoogleProvider } from "./providers/google.js";
+import { FileManager } from "./file-manager.js";
+import { buildGenerateImageHandler } from "./tools/generate-image.js";
+import { buildGenerateVideoHandler } from "./tools/generate-video.js";
+import { buildGenerateAudioHandler } from "./tools/generate-audio.js";
+import { buildListProvidersHandler } from "./tools/list-providers.js";
+export function createServer(config) {
+    const registry = new ProviderRegistry();
+    const fileManager = new FileManager(config.outputDirectory);
+    if (config.openaiApiKey) {
+        registry.register(new OpenAIProvider(config.openaiApiKey));
+        console.error("[server] Registered OpenAI provider");
+    }
+    if (config.xaiApiKey) {
+        registry.register(new XAIProvider(config.xaiApiKey));
+        console.error("[server] Registered xAI provider");
+    }
+    if (config.googleApiKey) {
+        registry.register(new GoogleProvider(config.googleApiKey));
+        console.error("[server] Registered Google provider");
+    }
+    const generateImageHandler = buildGenerateImageHandler(registry, fileManager);
+    const generateVideoHandler = buildGenerateVideoHandler(registry, fileManager);
+    const generateAudioHandler = buildGenerateAudioHandler(registry, fileManager);
+    const listProvidersHandler = buildListProvidersHandler(registry);
+    const providerNames = registry.listCapabilities().map((p) => p.name).join(", ") || "none configured";
+    const server = new McpServer({ name: "multimodal-mcp", version: "1.0.0" });
+    server.tool("generate_image", `Generate an image from a text prompt using AI. Available providers: ${providerNames}`, {
+        prompt: z.string().describe("Text description of the image to generate"),
+        provider: z.string().optional().describe("Provider to use: openai, xai, google. Auto-selects if omitted."),
+        aspectRatio: z.string().optional().describe("Aspect ratio: 1:1, 16:9, 9:16, 4:3, 3:4"),
+        quality: z.string().optional().describe("Quality level: low, standard, high"),
+        providerOptions: z.record(z.string(), z.unknown()).optional().describe("Provider-specific parameters passed through directly"),
+    }, async (params) => generateImageHandler(params));
+    server.tool("generate_video", `Generate a video from a text prompt using AI. Available providers: ${providerNames}`, {
+        prompt: z.string().describe("Text description of the video to generate"),
+        provider: z.string().optional().describe("Provider to use: openai, xai, google. Auto-selects if omitted."),
+        duration: z.number().optional().describe("Video duration in seconds (provider limits apply)"),
+        aspectRatio: z.string().optional().describe("Aspect ratio: 16:9, 9:16, 1:1"),
+        resolution: z.string().optional().describe("Resolution: 480p, 720p, 1080p"),
+        providerOptions: z.record(z.string(), z.unknown()).optional().describe("Provider-specific parameters passed through directly"),
+    }, async (params) => generateVideoHandler(params));
+    server.tool("generate_audio", `Generate audio (text-to-speech) from text using AI. Available providers: ${providerNames}`, {
+        text: z.string().describe("Text to convert to speech"),
+        provider: z.string().optional().describe("Provider to use: openai, google. Auto-selects if omitted."),
+        voice: z.string().optional().describe("Voice name (provider-specific). OpenAI: alloy, ash, coral, echo, fable, nova, onyx, sage, shimmer. Google: Kore, Charon, Fenrir, Aoede, Puck, etc."),
+        speed: z.number().optional().describe("Speech speed multiplier (OpenAI only): 0.25 to 4.0"),
+        format: z.string().optional().describe("Output format (OpenAI only): mp3, opus, aac, flac, wav, pcm"),
+        providerOptions: z.record(z.string(), z.unknown()).optional().describe("Provider-specific parameters passed through directly"),
+    }, async (params) => generateAudioHandler(params));
+    server.tool("list_providers", "List all configured media generation providers and their capabilities", async () => listProvidersHandler());
+    return server;
+}

package/build/tools/generate-audio.d.ts

@@ -0,0 +1,22 @@
+import type { ProviderRegistry } from "../providers/registry.js";
+import type { FileManager } from "../file-manager.js";
+export declare function buildGenerateAudioHandler(registry: ProviderRegistry, fileManager: FileManager): (params: {
+    text: string;
+    provider?: string;
+    voice?: string;
+    speed?: number;
+    format?: string;
+    providerOptions?: Record<string, unknown>;
+}) => Promise<{
+    isError: true;
+    content: {
+        type: "text";
+        text: string;
+    }[];
+} | {
+    content: {
+        type: "text";
+        text: string;
+    }[];
+    isError?: undefined;
+}>;

package/build/tools/generate-audio.js

@@ -0,0 +1,48 @@
+import { sanitizeError } from "../errors.js";
+export function buildGenerateAudioHandler(registry, fileManager) {
+    return async (params) => {
+        const provider = params.provider
+            ? registry.getProvider(params.provider)
+            : registry.getAudioProviders()[0];
+        if (!provider) {
+            const available = registry.getAudioProviders().map((p) => p.name).join(", ") || "none";
+            const text = params.provider
+                ? `Provider "${params.provider}" is not configured or does not support audio. Available audio providers: ${available}`
+                : "No audio provider available. Configure one of: OPENAI_API_KEY, GOOGLE_API_KEY";
+            return {
+                isError: true,
+                content: [{ type: "text", text }],
+            };
+        }
+        if (!provider.capabilities.supportsAudioGeneration) {
+            const available = registry.getAudioProviders().map((p) => p.name).join(", ") || "none";
+            return {
+                isError: true,
+                content: [{
+                        type: "text",
+                        text: `Provider "${provider.name}" does not support audio generation. Available audio providers: ${available}`,
+                    }],
+            };
+        }
+        try {
+            const media = await provider.generateAudio({
+                text: params.text,
+                voice: params.voice,
+                speed: params.speed,
+                format: params.format,
+                providerOptions: params.providerOptions,
+            });
+            const filePath = await fileManager.save(media, "audio");
+            return {
+                content: [{ type: "text", text: `Audio saved to ${filePath}` }],
+            };
+        }
+        catch (error) {
+            const message = sanitizeError(error);
+            return {
+                isError: true,
+                content: [{ type: "text", text: `Audio generation failed: ${message}` }],
+            };
+        }
+    };
+}

package/build/tools/generate-image.d.ts

@@ -0,0 +1,21 @@
+import type { ProviderRegistry } from "../providers/registry.js";
+import type { FileManager } from "../file-manager.js";
+export declare function buildGenerateImageHandler(registry: ProviderRegistry, fileManager: FileManager): (params: {
+    prompt: string;
+    provider?: string;
+    aspectRatio?: string;
+    quality?: string;
+    providerOptions?: Record<string, unknown>;
+}) => Promise<{
+    isError: true;
+    content: {
+        type: "text";
+        text: string;
+    }[];
+} | {
+    content: {
+        type: "text";
+        text: string;
+    }[];
+    isError?: undefined;
+}>;

package/build/tools/generate-image.js

@@ -0,0 +1,35 @@
+import { sanitizeError } from "../errors.js";
+export function buildGenerateImageHandler(registry, fileManager) {
+    return async (params) => {
+        const provider = registry.getProvider(params.provider);
+        if (!provider) {
+            const availableNames = registry.getImageProviders().map((p) => p.name).join(", ");
+            const text = params.provider
+                ? `Provider "${params.provider}" is not configured. Available providers: ${availableNames || "none"}`
+                : "No image provider available. Configure one of: OPENAI_API_KEY, XAI_API_KEY, GOOGLE_API_KEY";
+            return {
+                isError: true,
+                content: [{ type: "text", text }],
+            };
+        }
+        try {
+            const media = await provider.generateImage({
+                prompt: params.prompt,
+                aspectRatio: params.aspectRatio ?? "1:1",
+                quality: params.quality ?? "standard",
+                providerOptions: params.providerOptions,
+            });
+            const filePath = await fileManager.save(media, "image");
+            return {
+                content: [{ type: "text", text: `Image saved to ${filePath}` }],
+            };
+        }
+        catch (error) {
+            const message = sanitizeError(error);
+            return {
+                isError: true,
+                content: [{ type: "text", text: `Image generation failed: ${message}` }],
+            };
+        }
+    };
+}

package/build/tools/generate-video.d.ts

@@ -0,0 +1,22 @@
+import type { ProviderRegistry } from "../providers/registry.js";
+import type { FileManager } from "../file-manager.js";
+export declare function buildGenerateVideoHandler(registry: ProviderRegistry, fileManager: FileManager): (params: {
+    prompt: string;
+    provider?: string;
+    duration?: number;
+    aspectRatio?: string;
+    resolution?: string;
+    providerOptions?: Record<string, unknown>;
+}) => Promise<{
+    isError: true;
+    content: {
+        type: "text";
+        text: string;
+    }[];
+} | {
+    content: {
+        type: "text";
+        text: string;
+    }[];
+    isError?: undefined;
+}>;

package/build/tools/generate-video.js

@@ -0,0 +1,36 @@
+import { sanitizeError } from "../errors.js";
+export function buildGenerateVideoHandler(registry, fileManager) {
+    return async (params) => {
+        const provider = registry.getProvider(params.provider);
+        if (!provider) {
+            const available = registry.getVideoProviders().map((p) => p.name).join(", ") || "none";
+            const text = params.provider
+                ? `Provider "${params.provider}" is not configured. Available providers: ${available}`
+                : "No video provider available. Configure one of: OPENAI_API_KEY, XAI_API_KEY, GOOGLE_API_KEY";
+            return {
+                isError: true,
+                content: [{ type: "text", text }],
+            };
+        }
+        try {
+            const media = await provider.generateVideo({
+                prompt: params.prompt,
+                duration: params.duration ?? 5,
+                aspectRatio: params.aspectRatio ?? "16:9",
+                resolution: params.resolution ?? "720p",
+                providerOptions: params.providerOptions,
+            });
+            const filePath = await fileManager.save(media, "video");
+            return {
+                content: [{ type: "text", text: `Video saved to ${filePath}` }],
+            };
+        }
+        catch (error) {
+            const message = sanitizeError(error);
+            return {
+                isError: true,
+                content: [{ type: "text", text: `Video generation failed: ${message}` }],
+            };
+        }
+    };
+}

package/build/tools/list-providers.d.ts

@@ -0,0 +1,7 @@
+import type { ProviderRegistry } from "../providers/registry.js";
+export declare function buildListProvidersHandler(registry: ProviderRegistry): () => Promise<{
+    content: {
+        type: "text";
+        text: string;
+    }[];
+}>;

package/build/tools/list-providers.js

@@ -0,0 +1,27 @@
+export function buildListProvidersHandler(registry) {
+    return async () => {
+        const providers = registry.listCapabilities();
+        if (providers.length === 0) {
+            return {
+                content: [{
+                        type: "text",
+                        text: "No providers configured. Set one or more API keys: OPENAI_API_KEY, XAI_API_KEY, GOOGLE_API_KEY",
+                    }],
+            };
+        }
+        const lines = providers.map((p) => {
+            const caps = [];
+            if (p.capabilities.supportsImageGeneration)
+                caps.push("image");
+            if (p.capabilities.supportsVideoGeneration)
+                caps.push("video");
+            return `- ${p.name}: ${caps.join(", ")}`;
+        });
+        return {
+            content: [{
+                    type: "text",
+                    text: `Configured providers:\n${lines.join("\n")}`,
+                }],
+        };
+    };
+}

package/package.json

@@ -0,0 +1,53 @@
+{
+  "name": "@r16t/multimodal-mcp",
+  "version": "1.0.0",
+  "description": "Multi-provider media generation MCP server",
+  "type": "module",
+  "main": "build/index.js",
+  "bin": {
+    "multimodal-mcp": "build/index.js",
+    "@r16t/multimodal-mcp": "build/index.js"
+  },
+  "files": [
+    "build/"
+  ],
+  "scripts": {
+    "build": "tsc && chmod +x build/index.js",
+    "prepare": "npm run build",
+    "dev": "tsc --watch",
+    "test": "vitest",
+    "lint": "eslint src --fix",
+    "typecheck": "tsc --noEmit"
+  },
+  "license": "MIT",
+  "keywords": [
+    "mcp",
+    "media",
+    "image-generation",
+    "video-generation",
+    "audio-generation",
+    "text-to-speech",
+    "openai",
+    "xai",
+    "google",
+    "claude"
+  ],
+  "engines": {
+    "node": ">=18"
+  },
+  "dependencies": {
+    "@google/generative-ai": "^0.24.1",
+    "@modelcontextprotocol/sdk": "^1.27.1",
+    "openai": "^6.25.0",
+    "zod": "^4.3.6"
+  },
+  "devDependencies": {
+    "@eslint/js": "^10.0.1",
+    "@types/node": "^25.3.3",
+    "eslint": "^10.0.2",
+    "globals": "^17.4.0",
+    "typescript": "^5.9.3",
+    "typescript-eslint": "^8.56.1",
+    "vitest": "^4.0.18"
+  }
+}