pi-media-models 0.1.4 → 0.1.6

package/README.md

@@ -3,109 +3,97 @@
 [![npm version](https://img.shields.io/npm/v/pi-media-models.svg)](https://www.npmjs.com/package/pi-media-models)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
-Provider-neutral multimodal generation extension for [Pi Coding Agent](https://pi.dev). Generate images, video, audio, and speech across 9 providers with a single unified interface.
+**Provider-neutral multimodal generation extension for Pi Coding Agent.**
 
----
+This extension seamlessly bridges Pi's reasoning capabilities with top-tier AI media generation platforms. It abstracts away complex multi-part uploads, background task polling, CDN hosting, and API differences, exposing exactly **6 unified tools** for the agent.
 
-## Installation
+## ✨ Features
 
-Run this once in any terminal:
+- **Unified Interface**: One request format (`provider`, `model`, `prompt`, `referenceImages`, etc.) maps automatically to the correct capability across providers.
+- **Config-First Authentication**: Manage all API keys and credentials directly in a single `media-models.json` configuration file — no cluttered environment variables needed.
+- **Auto-Download**: Output media (images, videos, audio) is automatically downloaded and saved to a local directory (`~/.pi/agent/media/outputs/`) using atomic `.part` renames. LLM context remains pristine and only receives local file paths.
+- **Smart Input Resolution**: Pass local paths (`C:/...` or `/path/...`), file URIs (`file://`), standard URLs (`http(s)://`), or base64 (`data:...`). The router transparently handles multipart uploads, base64 encoding, or CDN pre-uploading (e.g., for `fal.ai`).
+- **Resilient Polling**: Advanced `MediaJob` processing handles asynchronous Long-Running Operations (LROs), 429 rate limits (respecting `Retry-After`), and timeouts. Supports remote job cancellation where supported by the provider.
 
-```bash
-pi install npm:pi-media-models
-```
+## 📦 Installation
 
-That's it. Pi downloads the package, registers the extension, and loads the skill guide automatically. No manual configuration needed to get started.
+This extension is built for the **Pi Coding Agent**. It bundles both the executable extension logic and the LLM `SKILL.md` prompt context.
 
-If Pi is already running, type `/reload` in the chat to activate it in the current session.
+Install natively inside your Pi environment:
 
-> **Alternative:** Install directly from source
-> ```bash
-> pi install git:github.com/luffysolution-svg/pi-media-models
-> ```
+```bash
+# Install via NPM (Recommended)
+pi install npm:pi-media-models
 
----
+# Or install directly from GitHub
+pi install git:github.com/luffysolution-svg/pi-media-models
+```
 
-## Quick Start
+Once installed, restart Pi or type `/reload` in your active session.
 
-After installation, open Pi and start chatting. No extra steps required if you have environment variables set for your chosen provider.
+## 🚀 Quick Usage
 
-**Example prompts:**
+After installation, the extension and its companion skill are active immediately. You can talk to Pi naturally — Pi knows how to query available models and route tasks to the appropriate tool.
 
-```
-Generate an image of a misty mountain at sunrise using fal.ai
-```
+**Example Prompts:**
 
 ```
-Use xAI to create a 5-second video of ocean waves with audio
+帮我用 fal.ai 画一张赛博朋克风格的雨夜街景，比例 16:9
 ```
 
 ```
-Convert this text to speech using OpenAI: "Hello, world"
+Use xAI to generate a 5-second video of ocean waves with audio
 ```
 
 ```
-What image/video providers do I have configured?
+使用 Atlas 编辑这张图片，把背景换成雪山：C:/assets/photo.jpg
 ```
 
-Pi will automatically call the right tool, handle uploads and polling, and save the result to your local disk. It reports the saved file path when done.
-
----
-
-## API Key Setup
-
-The extension reads keys from environment variables. Set whichever providers you want to use:
-
-| Provider | Environment Variable |
-|---|---|
-| fal.ai | `FAL_KEY` |
-| OpenAI | `OPENAI_API_KEY` |
-| xAI (Grok) | `XAI_API_KEY` |
-| Google Gemini | `GEMINI_API_KEY` |
-| DashScope / QwenCloud | `DASHSCOPE_API_KEY` |
-| OpenRouter | `OPENROUTER_API_KEY` |
-| Atlas | `ATLAS_API_KEY` |
-| Vertex AI | `GOOGLE_APPLICATION_CREDENTIALS` (ADC) |
-
-**macOS / Linux** — add to `~/.zshrc` or `~/.bashrc`:
-
-```bash
-export FAL_KEY="your-key"
-export XAI_API_KEY="your-key"
 ```
-
-**Windows** — run in PowerShell (persists across reboots):
-
-```powershell
-[System.Environment]::SetEnvironmentVariable("FAL_KEY", "your-key", "User")
-[System.Environment]::SetEnvironmentVariable("XAI_API_KEY", "your-key", "User")
+帮我查一下当前已配置好可用的多模态模型有哪些？
 ```
 
-Then restart Pi (or your terminal) for the variables to take effect.
+Pi 会自动调取对应工具、完成排队轮询与文件下载，并直接返回本地媒体文件的保存路径。
 
-### Alternative: Config File
+## ⚙️ Configuration
 
-If you prefer not to use environment variables, create the file `~/.pi/agent/media-models.json`:
+All API keys, custom endpoints, and output paths are configured directly in a single JSON file.
 
-```json
-{
-  "providerOptions": {
-    "fal": { "apiKey": "your-fal-key" },
-    "xai": { "apiKey": "xai-your-key" },
-    "openai": { "apiKey": "sk-your-key" }
-  }
-}
-```
+**Configuration File Location:**
+- **Global (Recommended):** `~/.pi/agent/media-models.json`
+- **Project-Specific:** `<project_root>/.pi/media-models.json`
 
-Keys in the config file take precedence over environment variables for that provider.
+### Example `media-models.json`
 
-**Vertex AI** requires a service account JSON file path instead of an API key:
+Create or edit `~/.pi/agent/media-models.json`:
 
 ```json
 {
+  "outputDir": "~/.pi/agent/media/outputs",
   "providerOptions": {
+    "fal": {
+      "apiKey": "fal-xxxxxxxxxxxxxxxxxxxx"
+    },
+    "xai": {
+      "apiKey": "xai-xxxxxxxxxxxxxxxxxxxx"
+    },
+    "atlas": {
+      "apiKey": "sk-xxxxxxxxxxxxxxxxxxxx"
+    },
+    "dashscope": {
+      "apiKey": "sk-xxxxxxxxxxxxxxxxxxxx"
+    },
+    "openai": {
+      "apiKey": "sk-xxxxxxxxxxxxxxxxxxxx"
+    },
+    "gemini": {
+      "apiKey": "AIzaxxxxxxxxxxxxxxxxxxxx"
+    },
+    "openrouter": {
+      "apiKey": "sk-or-xxxxxxxxxxxxxxxxxxxx"
+    },
     "vertex": {
-      "credentialsFile": "/path/to/service-account.json",
+      "credentialsFile": "/path/to/vertex-service-account.json",
       "project": "my-gcp-project",
       "location": "us-central1"
     }
@@ -113,83 +101,27 @@ Keys in the config file take precedence over environment variables for that prov
 }
 ```
 
----
-
-## Output Files
-
-All generated media is automatically downloaded and saved to:
-
-```
-~/.pi/agent/media/outputs/
-```
-
-To use a different directory, add `outputDir` to the config file:
-
-```json
-{
-  "outputDir": "/Users/alice/Pictures/ai-output"
-}
-```
-
----
-
-## Supported Providers
-
-| Provider | Capabilities | Env Var |
-|---|---|---|
-| **fal.ai** | Images, Video, Audio, TTS, STT | `FAL_KEY` |
-| **xAI (Grok Imagine)** | Image gen/edit, T2V, I2V, video extend | `XAI_API_KEY` |
-| **OpenAI** | Image gen/edit (DALL-E), TTS, STT (Whisper) | `OPENAI_API_KEY` |
-| **Google Gemini** | Imagen, Gemini image, Veo video, Lyria audio, TTS | `GEMINI_API_KEY` |
-| **Google Vertex AI** | Same as Gemini, enterprise ADC auth | ADC JSON |
-| **DashScope / 百炼** | Qwen image, Wan video, music, TTS/STT | `DASHSCOPE_API_KEY` |
-| **QwenCloud** | Same as DashScope (international endpoints) | `DASHSCOPE_API_KEY` |
-| **OpenRouter** | Image gen, async video, TTS | `OPENROUTER_API_KEY` |
-| **Atlas** | Image gen/edit, video (sync + async) | `ATLAS_API_KEY` |
-
-To see which providers are active in your current setup, ask Pi:
-
-```
-What media providers do I have configured?
-```
-
----
-
-## Available Tools
-
-The extension registers exactly 6 tools. Pi selects the correct one automatically based on your request.
-
-| Tool | Purpose |
-|---|---|
-| `media_models` | List providers, models, and capabilities |
-| `image_generate` | Generate images from text or reference images |
-| `image_edit` | Edit an existing image (inpainting, style, background) |
-| `video_generate` | Text-to-video, image-to-video, video extend/edit |
-| `audio_generate` | Generate music or raw audio |
-| `speech_generate` | TTS (text → audio file) or STT (audio → transcript) |
-
----
+> **Note**: You only need to fill in the providers you plan to use. Unused providers can simply be omitted.
 
-## Advanced: Custom OpenAI-Compatible Providers
+### Custom OpenAI-Compatible Providers
 
-You can add any OpenAI-compatible API as a custom provider in the config file. Capabilities and endpoints must be declared explicitly — the extension does not probe `/models` to infer them.
+You can connect any third-party or internal OpenAI-compatible media gateway by declaring it in `customProviders`:
 
 ```json
 {
   "customProviders": [
     {
-      "id": "my-gateway",
+      "id": "my-custom-ai",
       "name": "Internal AI Gateway",
-      "baseUrl": "https://api.internal.example.com/v1",
-      "apiKeyEnv": "MY_INTERNAL_KEY",
+      "baseUrl": "https://api.internal.com/v1",
       "auth": "bearer",
       "models": [
         {
-          "id": "internal-image-v2",
+          "id": "internal-video-pro",
           "vendor": "internal",
-          "capabilities": ["image.text_to_image"],
+          "capabilities": ["video.text_to_video"],
           "endpoints": {
-            "image.text_to_image": "/images/generations"
+            "video.text_to_video": "/videos/generations"
           }
         }
       ]
@@ -198,24 +130,45 @@ You can add any OpenAI-compatible API as a custom provider in the config file. C
 }
 ```
 
----
+## 🔌 Supported Providers
+
+| Provider | Provider ID | Supported Capabilities | Config Option |
+|---|---|---|---|
+| **fal.ai** | `fal` | Images, Video, Audio, TTS, STT (Queue/CDN integration) | `providerOptions.fal.apiKey` |
+| **xAI (Grok Imagine)** | `xai` | Image Gen/Edit, T2V/I2V, Reference-to-Video, Video Extend | `providerOptions.xai.apiKey` |
+| **Atlas API** | `atlas` | Sync/Async Image Gen & Edit, Video LROs | `providerOptions.atlas.apiKey` |
+| **DashScope / 百炼** | `dashscope` | Qwen/Wan Images, Wan Video, Fun-Music, TTS/STT | `providerOptions.dashscope.apiKey` |
+| **QwenCloud** | `qwencloud` | Same as DashScope (International endpoints) | `providerOptions.qwencloud.apiKey` |
+| **OpenAI API** | `openai` | Image Gen/Edit (DALL-E), TTS, STT (Whisper) | `providerOptions.openai.apiKey` |
+| **Google Gemini API** | `gemini` | Gemini/Imagen, Veo, Lyria, TTS, STT | `providerOptions.gemini.apiKey` |
+| **Google Vertex AI** | `vertex` | ADC, Imagen/Gemini, Veo, Lyria, TTS, STT | `providerOptions.vertex.credentialsFile` |
+| **OpenRouter** | `openrouter` | Image Generation, Async Video, TTS | `providerOptions.openrouter.apiKey` |
+
+## 🛠️ Exposed Tools
+
+The extension registers exactly 6 unified tools for the reasoning agent:
 
-## Security
+1. `media_models`: Lists providers, configured models, and supported capabilities.
+2. `image_generate`: Generate images from text, image, or multiple reference inputs.
+3. `image_edit`: Edit existing images (supports masks and multiple references).
+4. `video_generate`: Generates, edits, or extends videos. Automatically maps inputs (`referenceImages`, `inputVideo`, `duration`, `generateAudio`, etc.) to the provider's exact capability.
+5. `audio_generate`: Generate music or raw audio (separate from TTS).
+6. `speech_generate`: Handle TTS (Text-to-Speech) and STT (Speech-to-Text).
 
-- API keys and Bearer tokens are automatically redacted (`[REDACTED]`) from all error messages and logs before being returned to the LLM context.
-- The config file (`media-models.json`) is listed in `.gitignore` and is never included in the npm package.
-- No telemetry. All requests go directly from your machine to the provider API.
+## 🔒 Security & Privacy
+
+- **No Key Logging**: API keys and Bearer tokens are redacted (`[REDACTED]`) from all error logs and HTTP outputs before being returned to the LLM.
+- **Local Downloads**: Media assets are fetched directly by your local client into your configured `outputDir` without third-party proxies.
+- **Git Ignored**: `media-models.json` is automatically ignored from git repositories to prevent accidental credential commits.
 
 ---
 
-## Updating
+## 🔄 Updating
 
 ```bash
 pi update npm:pi-media-models
 ```
 
----
-
-## License
+## 📄 License
 
 MIT

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-media-models",
-  "version": "0.1.4",
+  "version": "0.1.6",
   "type": "module",
   "description": "Provider-neutral multimodal generation tools for Pi Coding Agent",
   "keywords": [

package/skills/pi-media/SKILL.md

@@ -82,34 +82,28 @@ All generated media is automatically downloaded to `~/.pi/agent/media/outputs/`.
 
 ## Key Configuration
 
-If the user asks how to set up API keys, there are two methods:
-
-**Environment variables** (any shell or system env):
-
-| Provider | Variable |
-|---|---|
-| OpenAI | `OPENAI_API_KEY` |
-| fal.ai | `FAL_KEY` |
-| Gemini | `GEMINI_API_KEY` |
-| Vertex AI | `GOOGLE_APPLICATION_CREDENTIALS` (ADC) |
-| xAI | `XAI_API_KEY` |
-| DashScope / QwenCloud | `DASHSCOPE_API_KEY` |
-| OpenRouter | `OPENROUTER_API_KEY` |
-| Atlas | `ATLAS_API_KEY` |
-
-**Config file** (`~/.pi/agent/media-models.json` or `.pi/media-models.json` in the project root):
+All API keys and provider options are configured in `~/.pi/agent/media-models.json` (or `.pi/media-models.json` in the workspace root):
 
 ```json
 {
-  "outputDir": "/path/to/output",
+  "outputDir": "~/.pi/agent/media/outputs",
   "providerOptions": {
+    "fal": { "apiKey": "fal-..." },
     "xai": { "apiKey": "xai-..." },
+    "atlas": { "apiKey": "sk-..." },
+    "dashscope": { "apiKey": "sk-..." },
+    "openai": { "apiKey": "sk-..." },
+    "gemini": { "apiKey": "AIza..." },
+    "openrouter": { "apiKey": "sk-or-..." },
     "vertex": {
       "credentialsFile": "/path/to/service-account.json",
-      "project": "my-gcp-project"
+      "project": "my-gcp-project",
+      "location": "us-central1"
     }
   }
 }
 ```
 
-Keys in `providerOptions` take precedence over environment variables for that provider.
+Tell the user to edit `~/.pi/agent/media-models.json` directly to add or update API keys.
+
+*(Environment variables such as `FAL_KEY`, `OPENAI_API_KEY`, `XAI_API_KEY` are also checked as a fallback).*

package/src/adapters/base.ts

@@ -30,7 +30,7 @@ export abstract class BaseAdapter implements ProviderAdapter {
 
   protected key(request: MediaRequest): string {
     const configKey = request.providerOptions?.apiKey;
-    const value = (typeof configKey === "string" ? configKey : undefined) ?? (this.envKey ? this.env[this.envKey] : undefined)
+    const value = (typeof configKey === "string" && configKey.trim() ? configKey.trim() : undefined) ?? (this.envKey ? this.env[this.envKey] : undefined)
     if (!value) throw new MediaError('AUTH', `${this.envKey ?? `${this.id} API key`} is not set in environment or config`, { provider: this.id })
     return value
   }

package/src/adapters/google.ts

@@ -3,6 +3,7 @@ import { fileURLToPath } from 'node:url'
 import { MediaError } from '../errors.js'
 import { MediaJob } from '../media-job.js'
 import { BaseAdapter, artifactsOrThrow, makeModel } from './base.js'
+import { expandHomePath } from '../config.js'
 import type { AdapterContext, AdapterResult, Capability, JobStatus, JsonObject, MediaRequest, ModelDescriptor } from '../types.js'
 import type { AdapterDependencies } from './base.js'
 
@@ -164,7 +165,8 @@ export class GoogleMediaAdapter extends BaseAdapter {
     const options = request.providerOptions;
     const configuredFile = typeof options?.credentialsFile === "string" ? options.credentialsFile : undefined
     const rawKeyFilename = configuredFile ?? this.env.VERTEX_CREDENTIALS_FILE ?? this.env.GOOGLE_APPLICATION_CREDENTIALS
-    const keyFilename = rawKeyFilename?.startsWith('file://') ? fileURLToPath(rawKeyFilename) : rawKeyFilename
+    const expandedKeyFile = expandHomePath(rawKeyFilename)
+    const keyFilename = expandedKeyFile?.startsWith('file://') ? fileURLToPath(expandedKeyFile) : expandedKeyFile
     const auth = new GoogleAuth({
       scopes: ['https://www.googleapis.com/auth/cloud-platform'],
       ...(keyFilename ? { keyFilename } : {}),

package/src/config.ts

@@ -1,9 +1,18 @@
 import { readFile } from 'node:fs/promises'
 import { homedir } from 'node:os'
-import { join } from 'node:path'
+import { join, resolve } from 'node:path'
 import { CAPABILITIES, type Capability, type JsonObject } from './types.js'
 import { MediaError } from './errors.js'
 
+export function expandHomePath(rawPath: string | undefined): string | undefined {
+  if (!rawPath) return undefined
+  if (rawPath === '~') return homedir()
+  if (rawPath.startsWith('~/') || rawPath.startsWith('~\\')) {
+    return join(homedir(), rawPath.slice(2))
+  }
+  return rawPath
+}
+
 export interface CustomAsyncConfig {
   idPath: string
   statusPath: string
@@ -61,8 +70,11 @@ export async function loadMediaConfig(cwd: string, allowProjectConfig: boolean):
   const globalPath = join(homedir(), '.pi', 'agent', 'media-models.json')
   const global = await parseFile(globalPath) ?? EMPTY_CONFIG
   const project = allowProjectConfig ? await parseFile(join(cwd, '.pi', 'media-models.json')) : undefined
+  const rawOutputDir = project?.outputDir ?? global.outputDir
+  const expandedOutputDir = expandHomePath(rawOutputDir)
+  const outputDir = expandedOutputDir ? resolve(cwd, expandedOutputDir) : undefined
   const merged: MediaConfig = {
-    outputDir: project?.outputDir ?? global.outputDir,
+    outputDir,
     customProviders: project?.customProviders ?? global.customProviders ?? [],
     providerOptions: { ...(global.providerOptions ?? {}), ...(project?.providerOptions ?? {}) },
   }

package/src/router.ts

@@ -49,9 +49,15 @@ export class CapabilityRouter {
   }
 
   private isConfigured(adapter: ProviderAdapter): boolean {
-    if (!adapter.envKey) return true
-    if (this.env[adapter.envKey]) return true
-    if (this.providerDefaults[adapter.id]?.apiKey) return true
+    if (adapter.id === 'vertex') {
+      const vertexOpts = this.providerDefaults['vertex']
+      if (typeof vertexOpts?.credentialsFile === 'string' && vertexOpts.credentialsFile.trim()) return true
+      if (this.env.GOOGLE_APPLICATION_CREDENTIALS || this.env.VERTEX_CREDENTIALS_FILE) return true
+      return false
+    }
+    const configKey = this.providerDefaults[adapter.id]?.apiKey
+    if (typeof configKey === 'string' && configKey.trim()) return true
+    if (adapter.envKey && this.env[adapter.envKey]) return true
     return false
   }