pi-media-models 0.1.0

package/README.md

@@ -0,0 +1,174 @@
+# pi-media-models
+
+Pi Coding Agent 的 Provider-neutral 多模态生成 Extension。只注册 6 个稳定 Tool；模型和 Provider 的变化被隔离在 Capability Router 与 Adapter 内。
+
+## 架构
+
+```text
+Pi Tool
+  → Capability Router
+    → Provider Adapter (Provider 与模型厂商分离)
+      → MediaJob (poll/backoff/timeout/AbortSignal/cancel-if-supported)
+        → Normalized Result
+          → Download (~/.pi/agent/media/outputs, .part + atomic rename)
+```
+
+Provider 原始大 JSON 不进入 LLM 上下文。Tool 只返回 Provider/模型、capability、任务 ID（如有）、本地文件路径或 STT 文本。
+
+## Tools
+
+- `media_models`：列出 Provider、已知模型/厂商和显式 capability。
+- `image_generate`：文生图、图生图、多参考图。
+- `image_edit`：图片编辑、多图编辑、mask（按 Provider 支持情况）。
+- `video_generate`：统一参数 `prompt`、`provider`、`model`、`inputImage`、`endImage`、`referenceImages`、`referenceVideos`、`referenceAudios`、`inputVideo`、`duration`、`resolution`、`aspectRatio`、`seed`、`generateAudio`、`operation`、`providerOptions`；自动映射 T2V/I2V/首尾帧/reference/edit/extend。
+- `audio_generate`：音乐或模型原生音频生成（不是 TTS）。
+- `speech_generate`：`operation=tts|stt`。
+
+所有输入文件字段接受本地路径、`file://`、`http(s)://` 和 data URI。Adapter 会按接口转为 data URI、base64、multipart 或上传 URL。fal 本地输入先上传 fal CDN；需要公网 URL 且没有文档化上传接口的 Provider 会使用 data URI，最终是否接受仍取决于具体模型。
+
+## Provider
+
+| Provider | Adapter | 能力摘要 | Key |
+|---|---|---|---|
+| OpenRouter | `OpenRouterAdapter` | Images API、视频异步任务、TTS | `OPENROUTER_API_KEY` |
+| fal.ai | `FalAdapter` | 任意 endpoint 的图片/视频/音频/TTS/STT；Queue + CDN upload | `FAL_KEY` |
+| 百炼/DashScope | `DashScopeAdapter` | Qwen/Wan 图片、Wan 视频参考/编辑/延长/原生音频、Fun-Music、TTS/STT | `DASHSCOPE_API_KEY` |
+| QwenCloud | `DashScopeAdapter`（国际 endpoint） | 同 DashScope 协议 | `DASHSCOPE_API_KEY` |
+| OpenAI API | `OpenAIAdapter` | 图片生成/编辑、TTS/STT；**不实现 Sora/OpenAI Video** | `OPENAI_API_KEY` |
+| Gemini API | `GoogleMediaAdapter` | Gemini/Imagen 图片、Veo、Lyria、TTS/STT | `GEMINI_API_KEY` |
+| Vertex AI | `GoogleMediaAdapter` | ADC、Imagen/Gemini、Veo、Lyria、TTS/STT | ADC |
+| xAI | 独立 `XAIAdapter` | Grok Imagine 图片生成/多图编辑、T2V/I2V/reference-to-video、video edit/extend | `XAI_API_KEY` |
+| Atlas | 独立 `AtlasAdapter` | 文档化图片同步/异步/编辑、视频任务及 reference image/video/audio、原生音频 | `ATLAS_API_KEY` |
+| 自定义 OpenAI-compatible | `CustomOpenAICompatibleAdapter` | 仅用户显式声明的 model/capability/endpoint | 用户声明的 env 名 |
+
+## 安装
+
+本目录已位于 Pi 全局自动发现位置：
+
+```bash
+cd ~/.pi/agent/extensions/pi-media
+npm install
+npm run check
+```
+
+在 Pi 中运行 `/reload`，或启动时显式加载：
+
+```bash
+pi -e ~/.pi/agent/extensions/pi-media/index.ts
+```
+
+生成结果默认立即下载到：
+
+```text
+~/.pi/agent/media/outputs/
+```
+
+## 配置
+
+API Key 只从环境变量读取，不写 JSON。可选配置：
+
+- 全局：`~/.pi/agent/media-models.json`
+- 项目：`<repo>/.pi/media-models.json`（只有 Pi 信任项目后才读取）
+
+```json
+{
+  "outputDir": "D:/media-output",
+  "providerOptions": {
+    "dashscope": {
+      "baseUrl": "https://WORKSPACE_ID.cn-beijing.maas.aliyuncs.com"
+    }
+  },
+  "customProviders": [
+    {
+      "id": "my-media",
+      "name": "My explicit media gateway",
+      "baseUrl": "https://media.example.com/v1",
+      "apiKeyEnv": "MY_MEDIA_API_KEY",
+      "auth": "bearer",
+      "models": [
+        {
+          "id": "vendor/image-model",
+          "vendor": "vendor",
+          "capabilities": ["image.text_to_image"],
+          "endpoints": {
+            "image.text_to_image": "/images/generations"
+          }
+        },
+        {
+          "id": "vendor/video-model",
+          "vendor": "vendor",
+          "capabilities": ["video.text_to_video"],
+          "endpoints": {
+            "video.text_to_video": {
+              "path": "/videos",
+              "format": "json",
+              "async": {
+                "idPath": "id",
+                "statusPath": "status",
+                "pollEndpoint": "/videos/{id}",
+                "resultPath": "result",
+                "cancelEndpoint": "/videos/{id}/cancel",
+                "successValues": ["completed"],
+                "failureValues": ["failed", "cancelled"]
+              }
+            }
+          }
+        }
+      ]
+    }
+  ]
+}
+```
+
+自定义 Provider **不会**请求或信任 `GET /models` 来推断能力；每个模型必须同时声明 `capabilities` 与对应 `endpoints`。
+
+## 使用示例
+
+先调用 `media_models` 查看 capability，再调用统一 Tool。例如：
+
+```json
+{
+  "provider": "xai",
+  "model": "grok-imagine-video-1.5",
+  "prompt": "A paper boat drifting down a rainy street",
+  "inputImage": "C:/assets/boat.png",
+  "duration": 8,
+  "resolution": "720p",
+  "aspectRatio": "16:9",
+  "generateAudio": true
+}
+```
+
+Atlas 多模态 reference-to-video：
+
+```json
+{
+  "provider": "atlas",
+  "model": "bytedance/seedance-2.0/text-to-video",
+  "prompt": "Product launch film",
+  "referenceImages": ["https://example.com/product.png"],
+  "referenceVideos": ["https://example.com/motion.mp4"],
+  "referenceAudios": ["https://example.com/voice.mp3"],
+  "generateAudio": true
+}
+```
+
+## 安全与可靠性
+
+- Key 仅取环境变量；错误消息自动脱敏，不记录请求头或完整 Provider JSON。
+- HTTP 请求有超时；幂等请求对 429/5xx 退避重试并尊重 `Retry-After`。为避免重复计费，非幂等生成 POST 默认不自动重试。
+- 异步任务统一支持 polling、backoff、总超时和 `AbortSignal`；仅在 Provider 文档明确提供 cancel 时调用远端取消（fal、DashScope、自定义显式 cancel）。
+- 测试进程设置 `PI_MEDIA_TEST_MODE=1`，未注入 mock fetch 时真实网络请求会直接失败，避免付费误调用。
+
+## 已知限制
+
+- 媒体模型与参数变化很快；`media_models` 中内置列表是已知入口，不是实时价格/可用性保证。具体模型、区域和账户权限仍由 Provider 校验。
+- fal 的输入/输出 schema 按 endpoint 变化，通用字段可通过 `providerOptions` 覆盖；应按所选 endpoint 文档传原生字段。
+- xAI 官方未公开视频任务 cancel；中止只停止本地 polling。自定义音频 reference 可能要求 trusted-partner 权限。
+- Gemini/Veo 和 Atlas 未公开视频任务 cancel。Atlas 文档未声明独立 TTS/STT、视频 edit/extend，因此不虚构这些 capability。
+- Vertex Veo REST 使用 `:fetchPredictOperation`；GCS 输出需要调用身份有对象读取权限。建议配置 Provider 原生输出到可读 GCS 或返回 base64。
+- 本仓库测试不进行真实付费生成；真实 Key、配额、内容策略和临时 URL 生命周期需在用户明确授权后做 smoke test。
+
+## 文档基线
+
+Atlas 以用户指定的 <https://doc.aixoras.com/jieruwendang/1-jiekouwendang.html> 为准。其余实现基于 OpenAI、OpenRouter、fal.ai、Alibaba/QwenCloud、Google/Vertex 和 xAI 官方文档（检查日期：2026-08-31）。

package/SKILL.md

@@ -0,0 +1 @@
+---\nname: pi-media\ndescription: 统一的多模态媒体生成能力（文生图、图生图、图片编辑、文生视频、图生视频、视频延长、原声音频、TTS、STT等）。支持 OpenAI, Gemini, Vertex, xAI, Atlas, DashScope, QwenCloud, fal.ai, OpenRouter 等。当用户要求生成、编辑媒体内容时，使用该技能提供的媒体生成工具。\n---\n\n# Pi Media Models (Multimodal Generation)\n\nThis extension provides a unified, provider-neutral set of tools to generate, edit, and manipulate multimodal content (images, videos, audio, speech) across leading AI API providers.\n\n## 核心工作流 (Core Workflow)\n\n当你收到用户的多模态生成、编辑需求（如“画一张图”、“生成一段视频”、“把这段话读出来”等）时，请遵循以下步骤：\n\n1. **确认模型和能力**：\n   如果用户没有指定具体的模型，或者你不确定哪个平台支持当前操作，请先调用 `media_models` 工具，查看当前已配置（`configured: true`）的 Provider 以及它们支持的 capabilities。\n   *提示：绝不要凭空捏造模型名称或能力，一定要查阅 `media_models` 返回的支持列表。*\n\n2. **调用生成工具**：\n   根据任务类型，调用对应的生成工具：\n   - `image_generate`：用于生成全新的图片、或者带有参考图的生成。\n   - `image_edit`：用于编辑现有图片（支持局部重绘、背景替换等，取决于平台）。\n   - `video_generate`：用于文生视频、图生视频、首尾帧视频、参考视频、视频编辑、视频延长。\n   - `audio_generate`：用于生成音乐、音效或模型原生的声音表达。\n   - `speech_generate`：用于传统的 TTS（文字转语音）或 STT（语音转文字）。\n\n3. **处理输入文件 (Input Files)**：\n   如果用户提供了参考图片、音频或视频的**本地绝对路径**，或者是公网 URL，请直接将它们填入 `inputImage`、`referenceImages`、`inputVideo` 等参数中。底层的媒体路由会自动处理路径解析、文件读取、Multipart 构建甚至 CDN 预先上传（例如针对 fal.ai）。你不需要自己去读文件内容并转换为 Base64。\n\n4. **输出产物 (Output)**：\n   所有的工具调用都会**自动下载**生成的媒体文件，并将其保存在本地磁盘（通常在 `~/.pi/agent/media/outputs/`）。工具的返回结果会包含这些绝对路径。你只需要在回复中清晰地将这个路径展示给用户即可，不需要做额外的文件提取操作。\n\n## 可用参数指南\n\n对于 `video_generate` 或其他多模态生成任务，你可能需要使用丰富的控制参数：\n*   `prompt`: 必需。描述你想要的画面或声音。\n*   `aspectRatio`: 画面比例，如 \"16:9\", \"9:16\", \"1:1\" 等。\n*   `resolution`: 画面分辨率或尺寸，如 \"720p\", \"1080p\", \"1024x1024\"。\n*   `duration`: 视频或音频的时长（秒），如 5, 8。\n*   `generateAudio`: 布尔值，用于那些同时支持生成画面的伴随音效的模型（如 xAI, Veo, Atlas）。\n*   `providerOptions`: 如果某个特定 Provider 有其独占的高级参数（例如 fal.ai 的 endpoints 专属参数，或者设置单独的 `apiKey`），可以放在这里。例如 `{\"apiKey\": \"sk-...\"}`。\n\n## API Key 配置说明\n\n如果用户询问如何配置密钥，请告诉他们有两种方式：\n\n1. **系统环境变量**：直接 export 对应的 KEY，如 `OPENAI_API_KEY`, `FAL_KEY`, `XAI_API_KEY`, `GEMINI_API_KEY`, `DASHSCOPE_API_KEY`, `ATLAS_API_KEY`，或者用于 Vertex 的 `GOOGLE_APPLICATION_CREDENTIALS`。\n2. **配置文件 (media-models.json)**：在 `~/.pi/agent/media-models.json` (或项目下的 `.pi/media-models.json`) 中，可以直接配置特定厂商的 `apiKey`。\n   ```json\n   {\n     \"providerOptions\": {\n       \"xai\": {\n         \"apiKey\": \"xai-xxxxxxxx\"\n       },\n       \"vertex\": {\n         \"credentialsFile\": \"C:/path/to/vertex-service-account.json\",\n         \"project\": \"my-gcp-project\"\n       }\n     }\n   }\n   ```\n   此方法方便在项目中进行独立配置，而不会污染全局环境变量。\n

package/index.ts

@@ -0,0 +1,11 @@
+import type { ExtensionAPI } from '@earendil-works/pi-coding-agent'
+import { registerMediaTools } from './src/tools.js'
+
+export { CapabilityRouter } from './src/router.js'
+export { MediaJob } from './src/media-job.js'
+export { MediaError, redactSecrets } from './src/errors.js'
+export * from './src/types.js'
+
+export default function piMediaExtension(pi: ExtensionAPI): void {
+  registerMediaTools(pi)
+}

package/package.json

@@ -0,0 +1,40 @@
+{
+  "name": "pi-media-models",
+  "version": "0.1.0",
+  "type": "module",
+  "description": "Provider-neutral multimodal generation tools for Pi Coding Agent",
+  "pi": {
+    "extensions": [
+      "./index.ts"
+    ],
+    "skills": [
+      "./SKILL.md"
+    ]
+  },
+  "scripts": {
+    "typecheck": "tsc --noEmit",
+    "test": "cross-env PI_MEDIA_TEST_MODE=1 tsx --test test/**/*.test.ts",
+    "check": "npm run typecheck && npm test",
+    "smoke:live": "tsx scripts/live-smoke.ts"
+  },
+  "dependencies": {
+    "google-auth-library": "^10.5.0",
+    "typebox": "^1.0.62"
+  },
+  "devDependencies": {
+    "@earendil-works/pi-coding-agent": "0.84.4",
+    "@types/node": "^24.10.0",
+    "cross-env": "^10.1.0",
+    "tsx": "^4.21.0",
+    "typescript": "^5.9.3"
+  },
+  "engines": {
+    "node": ">=20"
+  },
+  "files": [
+    "src",
+    "index.ts",
+    "SKILL.md",
+    "README.md"
+  ]
+}

package/src/adapters/atlas.ts

@@ -0,0 +1,151 @@
+import { MediaError } from '../errors.js'
+import { MediaJob, mapJobState } from '../media-job.js'
+import { BaseAdapter, artifactsOrThrow, bearerHeaders, dataUris, makeModel, mergeOptions, requirePrompt } from './base.js'
+import type { AdapterContext, AdapterResult, Capability, JobStatus, JsonObject, MediaRequest, ModelDescriptor } from '../types.js'
+
+const ATLAS_CAPS: Capability[] = [
+  'image.text_to_image', 'image.image_to_image', 'image.edit',
+  'video.text_to_video', 'video.image_to_video', 'video.first_last_frame', 'video.reference',
+  'video.native_audio',
+]
+
+export class AtlasAdapter extends BaseAdapter {
+  readonly id = 'atlas'
+  readonly displayName = 'Atlas API (aixoras.com)'
+  readonly envKey = 'ATLAS_API_KEY'
+  private readonly baseUrl = 'https://api.aixoras.com/v1'
+
+  models(): ModelDescriptor[] {
+    return [
+      makeModel(this.id, 'openai', 'gpt-image-2', ['image.text_to_image', 'image.image_to_image', 'image.edit'], 'Use the exact model id returned for your Atlas account'),
+      makeModel(this.id, 'bytedance', 'bytedance/seedance-2.0/text-to-video', ['video.text_to_video', 'video.reference', 'video.native_audio']),
+      makeModel(this.id, 'bytedance', 'bytedance/seedance-2.0/image-to-video', ['video.image_to_video', 'video.first_last_frame', 'video.reference', 'video.native_audio']),
+    ]
+  }
+
+  supports(capability: Capability): boolean { return ATLAS_CAPS.includes(capability) }
+
+  async execute(request: MediaRequest, context: AdapterContext): Promise<AdapterResult> {
+    this.assertSupport(request)
+    if (request.capability.startsWith('video.')) return this.video(request, context)
+    if (request.capability === 'image.text_to_image' && !request.inputImage && !request.referenceImages?.length) return this.imageGenerate(request, context)
+    return this.imageEdit(request, context)
+  }
+
+  private async imageGenerate(request: MediaRequest, context: AdapterContext): Promise<AdapterResult> {
+    const key = this.key(request)
+    const asyncMode = request.providerOptions?.async !== false
+    const { async: _async, timeoutMs: _timeoutMs, ...nativeOptions } = request.providerOptions ?? {}
+    const payload = mergeOptions({
+      model: request.model, prompt: requirePrompt(request), n: request.count ?? 1,
+      ...(request.aspectRatio ? { aspect_ratio: request.aspectRatio } : {}),
+      ...(request.resolution ? { size: request.resolution } : {}),
+      response_format: 'url',
+      ...(request.seed !== undefined ? { extra_fields: { seed: request.seed } } : {}),
+    }, nativeOptions)
+    const submitted = await this.http.json<Record<string, unknown>>(`${this.baseUrl}/images/generations${asyncMode ? '/async' : ''}`, {
+      method: 'POST', headers: bearerHeaders(key, { 'Content-Type': 'application/json' }), body: JSON.stringify(payload),
+      signal: context.signal, provider: this.id, secrets: [key], timeoutMs: 120_000,
+    })
+    return asyncMode ? this.waitImage(request, submitted, context) : artifactsOrThrow(this.result(request, submitted, 'image'))
+  }
+
+  private async imageEdit(request: MediaRequest, context: AdapterContext): Promise<AdapterResult> {
+    const key = this.key(request)
+    const sources = [request.inputImage, ...(request.referenceImages ?? [])].filter((value): value is string => Boolean(value))
+    if (!sources.length) throw new MediaError('INPUT', 'Atlas image edit requires inputImage or referenceImages', { provider: this.id })
+    if (sources.length > 1) throw new MediaError('CAPABILITY_UNSUPPORTED', 'Atlas documentation defines one image upload for image edits', { provider: this.id })
+    const asyncMode = request.providerOptions?.async === true
+    const form = new FormData()
+    form.set('model', request.model)
+    form.set('prompt', requirePrompt(request))
+    form.set('n', String(request.count ?? 1))
+    form.set('response_format', 'url')
+    for (const source of sources) {
+      const image = await this.input.asBlob(source, context.signal)
+      form.append(sources.length === 1 ? 'image' : 'image[]', image.blob, image.fileName)
+    }
+    for (const [name, value] of Object.entries(request.providerOptions ?? {})) {
+      if (!['async', 'timeoutMs'].includes(name) && value !== undefined) form.set(name, typeof value === 'string' ? value : JSON.stringify(value))
+    }
+    const submitted = await this.http.json<Record<string, unknown>>(`${this.baseUrl}/images/edits${asyncMode ? '/async' : ''}`, {
+      method: 'POST', headers: bearerHeaders(key), body: form, signal: context.signal,
+      provider: this.id, secrets: [key], timeoutMs: 120_000,
+    })
+    return asyncMode ? this.waitImage(request, submitted, context) : artifactsOrThrow(this.result(request, submitted, 'image'))
+  }
+
+  private async waitImage(request: MediaRequest, submitted: Record<string, unknown>, context: AdapterContext): Promise<AdapterResult> {
+    const key = this.key(request)
+    const taskId = stringId(submitted)
+    if (!taskId) return artifactsOrThrow(this.result(request, submitted, 'image'))
+    const completed = await this.pollTask<Record<string, unknown>>(
+      taskId, `${this.baseUrl}/images/tasks/${encodeURIComponent(taskId)}`, key, request, context,
+    )
+    return artifactsOrThrow(this.result(request, completed, 'image', { jobId: taskId }))
+  }
+
+  private async video(request: MediaRequest, context: AdapterContext): Promise<AdapterResult> {
+    const key = this.key(request)
+    const inputImage = request.inputImage ? await this.input.asDataUri(request.inputImage, context.signal) : undefined
+    const endImage = request.endImage ? await this.input.asDataUri(request.endImage, context.signal) : undefined
+    const references = await dataUris(this.input, request.referenceImages, context.signal)
+    const referenceVideos = await dataUris(this.input, request.referenceVideos, context.signal)
+    const referenceAudios = await dataUris(this.input, request.referenceAudios, context.signal)
+    const inputVideo = request.inputVideo ? await this.input.asDataUri(request.inputVideo, context.signal) : undefined
+    const images = [...new Set([inputImage, endImage, ...references].filter((value): value is string => Boolean(value)))]
+    const metadata: JsonObject = {
+      ...(references.length ? { reference_images: references } : {}),
+      ...(referenceVideos.length ? { reference_videos: referenceVideos } : {}),
+      ...(referenceAudios.length ? { reference_audios: referenceAudios } : {}),
+      ...(request.aspectRatio ? { aspect_ratio: request.aspectRatio } : {}),
+      ...(request.generateAudio !== undefined ? { generate_audio: request.generateAudio } : {}),
+      ...(inputVideo ? { input_video: inputVideo } : {}),
+      ...(request.operation ? { operation: request.operation } : {}),
+      ...((request.providerOptions?.metadata && typeof request.providerOptions.metadata === 'object') ? request.providerOptions.metadata as JsonObject : {}),
+    }
+    const { metadata: _metadata, timeoutMs: _timeoutMs, ...nativeOptions } = request.providerOptions ?? {}
+    const payload = mergeOptions({
+      model: request.model, prompt: requirePrompt(request),
+      ...(request.duration ? { duration: request.duration, seconds: String(request.duration) } : {}),
+      ...(request.resolution ? { resolution: request.resolution, size: request.resolution } : {}),
+      ...(images.length ? { images, image: images[0], input_reference: images[0] } : {}),
+      metadata,
+    }, nativeOptions)
+    const submitted = await this.http.json<Record<string, unknown>>(`${this.baseUrl}/video/generations`, {
+      method: 'POST', headers: bearerHeaders(key, { 'Content-Type': 'application/json' }), body: JSON.stringify(payload),
+      signal: context.signal, provider: this.id, secrets: [key], timeoutMs: 60_000,
+    })
+    const taskId = stringId(submitted)
+    if (!taskId) return artifactsOrThrow(this.result(request, submitted, 'video'))
+    const completed = await this.pollTask<Record<string, unknown>>(
+      taskId, `${this.baseUrl}/video/generations/${encodeURIComponent(taskId)}`, key, request, context,
+    )
+    return artifactsOrThrow(this.result(request, completed, 'video', { jobId: taskId }))
+  }
+
+  private async pollTask<T extends Record<string, unknown>>(taskId: string, url: string, key: string, request: MediaRequest, context: AdapterContext): Promise<T> {
+    const job = new MediaJob<T>({
+      id: taskId, provider: this.id, signal: context.signal, timeoutMs: timeout(request), minDelayMs: 3_000, maxDelayMs: 5_000,
+      onProgress: status => context.onProgress?.(`Atlas ${taskId}: ${status.state}`),
+      poll: async signal => {
+        const status = await this.http.json<T>(url, {
+          headers: bearerHeaders(key), signal, provider: this.id, secrets: [key], timeoutMs: 30_000,
+        })
+        const state = mapJobState(status.status ?? status.raw_status)
+        const message = typeof status.fail_reason === 'string' ? status.fail_reason : undefined
+        return { state, ...(state === 'succeeded' ? { result: status } : {}), ...(message ? { message } : {}) } satisfies JobStatus<T>
+      },
+    })
+    return job.wait()
+  }
+}
+
+function stringId(payload: Record<string, unknown>): string | undefined {
+  return typeof payload.task_id === 'string' ? payload.task_id : typeof payload.id === 'string' ? payload.id : undefined
+}
+
+function timeout(request: MediaRequest): number {
+  const value = request.providerOptions?.timeoutMs
+  return typeof value === 'number' && value > 0 ? value : 30 * 60_000
+}

package/src/adapters/base.ts

@@ -0,0 +1,107 @@
+import { MediaError } from '../errors.js'
+import { extractArtifacts } from '../artifacts.js'
+import type { Capability, MediaKind, MediaRequest, ModelDescriptor, ProviderAdapter, AdapterContext, AdapterResult, JsonObject, RemoteArtifact } from '../types.js'
+import type { HttpClient } from '../http.js'
+import type { InputResolver } from '../input.js'
+
+export interface AdapterDependencies {
+  http: HttpClient
+  input: InputResolver
+  env?: NodeJS.ProcessEnv
+}
+
+export abstract class BaseAdapter implements ProviderAdapter {
+  abstract readonly id: string
+  abstract readonly displayName: string
+  abstract readonly envKey?: string
+  protected readonly http: HttpClient
+  protected readonly input: InputResolver
+  protected readonly env: NodeJS.ProcessEnv
+
+  constructor(dependencies: AdapterDependencies) {
+    this.http = dependencies.http
+    this.input = dependencies.input
+    this.env = dependencies.env ?? process.env
+  }
+
+  abstract models(): ModelDescriptor[]
+  abstract supports(capability: Capability, model: string): boolean
+  abstract execute(request: MediaRequest, context: AdapterContext): Promise<AdapterResult>
+
+  protected key(request: MediaRequest): string {
+    const configKey = request.providerOptions?.apiKey;
+    const value = (typeof configKey === "string" ? configKey : undefined) ?? (this.envKey ? this.env[this.envKey] : undefined)
+    if (!value) throw new MediaError('AUTH', `${this.envKey ?? `${this.id} API key`} is not set in environment or config`, { provider: this.id })
+    return value
+  }
+
+  protected assertSupport(request: MediaRequest): void {
+    if (!this.supports(request.capability, request.model)) {
+      throw new MediaError('CAPABILITY_UNSUPPORTED', `${this.displayName} model ${request.model} does not declare ${request.capability}`, { provider: this.id })
+    }
+  }
+
+  protected result(request: MediaRequest, payload: unknown, fallback: MediaKind, options: { jobId?: string; text?: string; warnings?: string[]; headers?: Record<string, string> } = {}): AdapterResult {
+    const artifacts = extractArtifacts(payload, fallback).map(artifact => options.headers ? { ...artifact, headers: options.headers } : artifact)
+    return {
+      provider: this.id,
+      model: request.model,
+      capability: request.capability,
+      artifacts,
+      ...(options.jobId ? { jobId: options.jobId } : {}),
+      ...(options.text ? { text: options.text } : {}),
+      ...(options.warnings?.length ? { warnings: options.warnings } : {}),
+    }
+  }
+}
+
+export function mergeOptions(base: JsonObject, options?: JsonObject): JsonObject {
+  return options ? { ...base, ...options } : base
+}
+
+export function bearerHeaders(key: string, extra: Record<string, string> = {}): Record<string, string> {
+  return { Authorization: `Bearer ${key}`, ...extra }
+}
+
+export function requirePrompt(request: MediaRequest): string {
+  if (!request.prompt?.trim()) throw new MediaError('INPUT', `${request.capability} requires prompt`, { provider: request.provider })
+  return request.prompt
+}
+
+export async function dataUris(input: InputResolver, sources: readonly string[] | undefined, signal?: AbortSignal): Promise<string[]> {
+  if (!sources) return []
+  return Promise.all(sources.map(source => input.asDataUri(source, signal)))
+}
+
+export function artifactsOrThrow(result: AdapterResult): AdapterResult {
+  if (result.artifacts.length === 0 && !result.text) {
+    throw new MediaError('PROVIDER', `${result.provider} returned no media artifact`, { provider: result.provider })
+  }
+  return result
+}
+
+export function makeModel(provider: string, vendor: string, id: string, capabilities: Capability[], notes?: string): ModelDescriptor {
+  return { provider, vendor, id, capabilities, ...(notes ? { notes } : {}) }
+}
+
+export function withArtifactHeaders(artifacts: RemoteArtifact[], headers: Record<string, string>): RemoteArtifact[] {
+  return artifacts.map(artifact => ({ ...artifact, headers }))
+}
+
+export function extractText(payload: unknown): string | undefined {
+  const candidates: string[] = []
+  function walk(value: unknown, key = ''): void {
+    if (typeof value === 'string' && /^(?:text|transcript|transcription|output_text)$/i.test(key)) {
+      candidates.push(value)
+      return
+    }
+    if (Array.isArray(value)) {
+      for (const item of value) walk(item, key)
+      return
+    }
+    if (!value || typeof value !== 'object') return
+    for (const [childKey, child] of Object.entries(value as Record<string, unknown>)) walk(child, childKey)
+  }
+  walk(payload)
+  return candidates.length ? [...new Set(candidates)].join('\n') : undefined
+}

package/src/adapters/custom.ts

@@ -0,0 +1,154 @@
+import { MediaError } from '../errors.js'
+import { MediaJob, mapJobState } from '../media-job.js'
+import type { CustomEndpointConfig, CustomModelConfig, CustomProviderConfig } from '../config.js'
+import type { AdapterContext, AdapterResult, Capability, JobStatus, JsonObject, MediaRequest, ModelDescriptor } from '../types.js'
+import type { AdapterDependencies } from './base.js'
+import { BaseAdapter, artifactsOrThrow, dataUris, makeModel, mergeOptions } from './base.js'
+
+export class CustomOpenAICompatibleAdapter extends BaseAdapter {
+  readonly id: string
+  readonly displayName: string
+  readonly envKey: string
+
+  constructor(private readonly config: CustomProviderConfig, dependencies: AdapterDependencies) {
+    super(dependencies)
+    this.id = config.id
+    this.displayName = config.name ?? config.id
+    this.envKey = config.apiKeyEnv
+  }
+
+  models(): ModelDescriptor[] {
+    return this.config.models.map(model => makeModel(this.id, model.vendor, model.id, model.capabilities, 'Explicit custom-provider declaration; no /models capability inference'))
+  }
+
+  supports(capability: Capability, model: string): boolean {
+    return Boolean(this.model(model)?.capabilities.includes(capability))
+  }
+
+  async execute(request: MediaRequest, context: AdapterContext): Promise<AdapterResult> {
+    this.assertSupport(request)
+    const model = this.model(request.model)
+    if (!model) throw new MediaError('CONFIG', `Unknown custom model ${this.id}/${request.model}`, { provider: this.id })
+    const declared = model.endpoints[request.capability]
+    if (!declared) throw new MediaError('CONFIG', `No endpoint declared for ${request.capability}`, { provider: this.id })
+    const endpoint: CustomEndpointConfig = typeof declared === 'string' ? { path: declared } : declared
+    const key = this.key(request)
+    const headers = { ...this.authHeaders(key), ...(this.config.headers ?? {}) }
+    const { body, contentType } = await this.body(request, endpoint.format ?? 'json', context.signal)
+    const submitted = await this.http.json<Record<string, unknown>>(this.url(endpoint.path), {
+      method: endpoint.method ?? 'POST', headers: { ...headers, ...(contentType ? { 'Content-Type': contentType } : {}) }, body,
+      signal: context.signal, provider: this.id, secrets: [key], timeoutMs: 120_000,
+    })
+    if (!endpoint.async) return artifactsOrThrow(this.result(request, submitted, kind(request), { text: textResult(submitted) }))
+    const asyncConfig = endpoint.async
+    const jobId = getPath(submitted, asyncConfig.idPath)
+    if (typeof jobId !== 'string') throw new MediaError('PROVIDER', `${this.id} async response did not contain ${asyncConfig.idPath}`, { provider: this.id })
+    const pollUrl = this.url(asyncConfig.pollEndpoint.replace('{id}', encodeURIComponent(jobId)))
+    const job = new MediaJob<Record<string, unknown>>({
+      id: jobId, provider: this.id, signal: context.signal, timeoutMs: timeout(request),
+      onProgress: status => context.onProgress?.(`${this.id} ${jobId}: ${status.state}`),
+      poll: async signal => {
+        const status = await this.http.json<Record<string, unknown>>(pollUrl, {
+          headers, signal, provider: this.id, secrets: [key], timeoutMs: 30_000,
+        })
+        const raw = String(getPath(status, asyncConfig.statusPath) ?? '')
+        const state = customState(raw, asyncConfig)
+        const selected = asyncConfig.resultPath ? getPath(status, asyncConfig.resultPath) : status
+        return {
+          state,
+          ...(state === 'succeeded' ? { result: (selected && typeof selected === 'object' ? selected : status) as Record<string, unknown> } : {}),
+        } satisfies JobStatus<Record<string, unknown>>
+      },
+      ...(asyncConfig.cancelEndpoint ? { cancel: async (signal: AbortSignal) => {
+        await this.http.request(this.url(asyncConfig.cancelEndpoint?.replace('{id}', encodeURIComponent(jobId)) ?? ''), {
+          method: 'POST', headers, signal, provider: this.id, secrets: [key], timeoutMs: 10_000,
+        })
+      } } : {}),
+    })
+    const completed = await job.wait()
+    return artifactsOrThrow(this.result(request, completed, kind(request), { jobId, text: textResult(completed) }))
+  }
+
+  private model(id: string): CustomModelConfig | undefined { return this.config.models.find(model => model.id === id) }
+
+  private authHeaders(key: string): Record<string, string> {
+    if (this.config.auth === 'none') return {}
+    if (this.config.auth === 'x-api-key') return { 'x-api-key': key }
+    return { Authorization: `Bearer ${key}` }
+  }
+
+  private url(path: string): string {
+    if (/^https?:\/\//i.test(path)) return path
+    return `${this.config.baseUrl.replace(/\/+$/, '')}/${path.replace(/^\/+/, '')}`
+  }
+
+  private async body(request: MediaRequest, format: 'json' | 'multipart', signal?: AbortSignal): Promise<{ body: BodyInit; contentType?: string }> {
+    const sources = [request.inputImage, ...(request.referenceImages ?? [])].filter((value): value is string => Boolean(value))
+    if (format === 'multipart') {
+      const form = new FormData()
+      form.set('model', request.model)
+      if (request.prompt) form.set('prompt', request.prompt)
+      if (request.text) form.set('input', request.text)
+      for (const source of sources) {
+        const file = await this.input.asBlob(source, signal)
+        form.append(sources.length > 1 ? 'image[]' : 'image', file.blob, file.fileName)
+      }
+      if (request.inputAudio) {
+        const file = await this.input.asBlob(request.inputAudio, signal)
+        form.set('file', file.blob, file.fileName)
+      }
+      if (request.inputVideo) {
+        const file = await this.input.asBlob(request.inputVideo, signal)
+        form.set('video', file.blob, file.fileName)
+      }
+      for (const [key, value] of Object.entries(request.providerOptions ?? {})) if (value !== undefined) form.set(key, typeof value === 'string' ? value : JSON.stringify(value))
+      return { body: form }
+    }
+    const references = await dataUris(this.input, sources, signal)
+    const payload = mergeOptions({
+      model: request.model,
+      ...(request.prompt ? { prompt: request.prompt } : {}),
+      ...(request.text ? { input: request.text } : {}),
+      ...(references[0] ? { image: references[0] } : {}),
+      ...(references.length ? { images: references } : {}),
+      ...(request.inputVideo ? { video: await this.input.asDataUri(request.inputVideo, signal) } : {}),
+      ...(request.inputAudio ? { audio: await this.input.asDataUri(request.inputAudio, signal) } : {}),
+      ...(request.endImage ? { end_image: await this.input.asDataUri(request.endImage, signal) } : {}),
+      ...(request.duration ? { duration: request.duration } : {}),
+      ...(request.resolution ? { size: request.resolution } : {}),
+      ...(request.aspectRatio ? { aspect_ratio: request.aspectRatio } : {}),
+      ...(request.seed !== undefined ? { seed: request.seed } : {}),
+      ...(request.generateAudio !== undefined ? { generate_audio: request.generateAudio } : {}),
+      ...(request.voice ? { voice: request.voice } : {}),
+      ...(request.language ? { language: request.language } : {}),
+      ...(request.responseFormat ? { response_format: request.responseFormat } : {}),
+      ...(request.operation ? { operation: request.operation } : {}),
+    }, request.providerOptions)
+    return { body: JSON.stringify(payload), contentType: 'application/json' }
+  }
+}
+
+function getPath(value: unknown, path: string): unknown {
+  return path.split('.').filter(Boolean).reduce<unknown>((current, key) => current && typeof current === 'object' ? (current as Record<string, unknown>)[key] : undefined, value)
+}
+
+function customState(raw: string, config: CustomEndpointConfig['async']): JobStatus<never>['state'] {
+  const normalized = raw.toLowerCase()
+  if (config?.successValues?.some(value => value.toLowerCase() === normalized)) return 'succeeded'
+  if (config?.failureValues?.some(value => value.toLowerCase() === normalized)) return 'failed'
+  return mapJobState(raw)
+}
+
+function kind(request: MediaRequest): 'image' | 'video' | 'audio' | 'text' {
+  return request.capability.startsWith('image.') ? 'image' : request.capability.startsWith('video.') ? 'video' : request.capability === 'speech.stt' ? 'text' : 'audio'
+}
+
+function textResult(payload: unknown): string | undefined {
+  const text = getPath(payload, 'text') ?? getPath(payload, 'data.text')
+  return typeof text === 'string' ? text : undefined
+}
+
+function timeout(request: MediaRequest): number {
+  const value = request.providerOptions?.timeoutMs
+  return typeof value === 'number' && value > 0 ? value : 30 * 60_000
+}