pi-media-models 0.1.0 → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +62 -128
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -1,117 +1,83 @@
|
|
|
1
1
|
# pi-media-models
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
[](https://www.npmjs.com/package/pi-media-models)
|
|
4
|
+
[](https://opensource.org/licenses/MIT)
|
|
4
5
|
|
|
5
|
-
|
|
6
|
+
**Provider-neutral multimodal generation extension for Pi Coding Agent.**
|
|
6
7
|
|
|
7
|
-
|
|
8
|
-
Pi Tool
|
|
9
|
-
→ Capability Router
|
|
10
|
-
→ Provider Adapter (Provider 与模型厂商分离)
|
|
11
|
-
→ MediaJob (poll/backoff/timeout/AbortSignal/cancel-if-supported)
|
|
12
|
-
→ Normalized Result
|
|
13
|
-
→ Download (~/.pi/agent/media/outputs, .part + atomic rename)
|
|
14
|
-
```
|
|
15
|
-
|
|
16
|
-
Provider 原始大 JSON 不进入 LLM 上下文。Tool 只返回 Provider/模型、capability、任务 ID（如有）、本地文件路径或 STT 文本。
|
|
8
|
+
This extension seamlessly bridges Pi's reasoning capabilities with top-tier AI media generation platforms. It abstracts away complex multi-part uploads, background task polling, CDN hosting, and API differences, exposing exactly **6 unified tools** for the agent.
|
|
17
9
|
|
|
18
|
-
##
|
|
10
|
+
## ✨ Features
|
|
19
11
|
|
|
20
|
-
- `
|
|
21
|
-
- `
|
|
22
|
-
- `
|
|
23
|
-
- `
|
|
24
|
-
- `audio_generate`：音乐或模型原生音频生成（不是 TTS）。
|
|
25
|
-
- `speech_generate`：`operation=tts|stt`。
|
|
12
|
+
- **Unified Interface**: One request format (`provider`, `model`, `prompt`, `referenceImages`, etc.) maps automatically to the correct capability across providers.
|
|
13
|
+
- **Auto-Download**: Output media (images, videos, audio) is automatically downloaded and saved to a local directory (`~/.pi/agent/media/outputs/`) using atomic `.part` renames. LLM context remains pristine and only receives local file paths.
|
|
14
|
+
- **Smart Input Resolution**: Pass local paths (`C:/...`), file URIs (`file://`), standard URLs (`http(s)://`), or base64 (`data:...`). The router transparently handles multipart uploads, base64 encoding, or CDN pre-uploading (e.g., for `fal.ai`).
|
|
15
|
+
- **Resilient Polling**: Advanced `MediaJob` processing handles asynchronous Long-Running Operations (LROs), 429 rate limits (respecting `Retry-After`), and timeouts. Supports remote job cancellation where supported by the provider.
|
|
26
16
|
|
|
27
|
-
|
|
17
|
+
## 📦 Installation
|
|
28
18
|
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
| Provider | Adapter | 能力摘要 | Key |
|
|
32
|
-
|---|---|---|---|
|
|
33
|
-
| OpenRouter | `OpenRouterAdapter` | Images API、视频异步任务、TTS | `OPENROUTER_API_KEY` |
|
|
34
|
-
| fal.ai | `FalAdapter` | 任意 endpoint 的图片/视频/音频/TTS/STT；Queue + CDN upload | `FAL_KEY` |
|
|
35
|
-
| 百炼/DashScope | `DashScopeAdapter` | Qwen/Wan 图片、Wan 视频参考/编辑/延长/原生音频、Fun-Music、TTS/STT | `DASHSCOPE_API_KEY` |
|
|
36
|
-
| QwenCloud | `DashScopeAdapter`（国际 endpoint） | 同 DashScope 协议 | `DASHSCOPE_API_KEY` |
|
|
37
|
-
| OpenAI API | `OpenAIAdapter` | 图片生成/编辑、TTS/STT；**不实现 Sora/OpenAI Video** | `OPENAI_API_KEY` |
|
|
38
|
-
| Gemini API | `GoogleMediaAdapter` | Gemini/Imagen 图片、Veo、Lyria、TTS/STT | `GEMINI_API_KEY` |
|
|
39
|
-
| Vertex AI | `GoogleMediaAdapter` | ADC、Imagen/Gemini、Veo、Lyria、TTS/STT | ADC |
|
|
40
|
-
| xAI | 独立 `XAIAdapter` | Grok Imagine 图片生成/多图编辑、T2V/I2V/reference-to-video、video edit/extend | `XAI_API_KEY` |
|
|
41
|
-
| Atlas | 独立 `AtlasAdapter` | 文档化图片同步/异步/编辑、视频任务及 reference image/video/audio、原生音频 | `ATLAS_API_KEY` |
|
|
42
|
-
| 自定义 OpenAI-compatible | `CustomOpenAICompatibleAdapter` | 仅用户显式声明的 model/capability/endpoint | 用户声明的 env 名 |
|
|
19
|
+
This extension is built for the **Pi Coding Agent**. It includes both the executable extension logic and the LLM `SKILL.md` prompt context.
|
|
43
20
|
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
本目录已位于 Pi 全局自动发现位置：
|
|
21
|
+
Install natively inside your Pi environment:
|
|
47
22
|
|
|
48
23
|
```bash
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
npm run check
|
|
52
|
-
```
|
|
53
|
-
|
|
54
|
-
在 Pi 中运行 `/reload`，或启动时显式加载：
|
|
24
|
+
# Install via NPM (Recommended)
|
|
25
|
+
pi install npm:pi-media-models
|
|
55
26
|
|
|
56
|
-
|
|
57
|
-
pi
|
|
27
|
+
# Or install directly from GitHub
|
|
28
|
+
pi install git:github.com/luffysolution-svg/pi-media-models
|
|
58
29
|
```
|
|
59
30
|
|
|
60
|
-
|
|
31
|
+
Once installed, simply type `/reload` in your active Pi session to apply the extension.
|
|
61
32
|
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
33
|
+
## 🔌 Supported Providers
|
|
34
|
+
|
|
35
|
+
| Provider | Adapter | Supported Capabilities | Required Env Var |
|
|
36
|
+
|---|---|---|---|
|
|
37
|
+
| **OpenRouter** | `OpenRouterAdapter` | Image Generation, Async Video, TTS | `OPENROUTER_API_KEY` |
|
|
38
|
+
| **fal.ai** | `FalAdapter` | Images, Video, Audio, TTS, STT (Queue/CDN integration) | `FAL_KEY` |
|
|
39
|
+
| **Google Gemini API** | `GoogleMediaAdapter` | Gemini/Imagen, Veo, Lyria, TTS, STT | `GEMINI_API_KEY` |
|
|
40
|
+
| **Google Vertex AI** | `GoogleMediaAdapter` | ADC, Imagen/Gemini, Veo, Lyria, TTS, STT | ADC credentials |
|
|
41
|
+
| **DashScope / 百炼** | `DashScopeAdapter` | Qwen/Wan Images, Wan Video, Fun-Music, TTS/STT | `DASHSCOPE_API_KEY` |
|
|
42
|
+
| **QwenCloud** | `DashScopeAdapter` | Same as DashScope (International endpoints) | `DASHSCOPE_API_KEY` |
|
|
43
|
+
| **OpenAI API** | `OpenAIAdapter` | Image Gen/Edit (DALL-E), TTS, STT (Whisper) | `OPENAI_API_KEY` |
|
|
44
|
+
| **xAI (Grok Imagine)** | `XAIAdapter` | Image Gen/Edit, T2V/I2V, Reference-to-Video, Edit, Extend | `XAI_API_KEY` |
|
|
45
|
+
| **Atlas API** | `AtlasAdapter` | Sync/Async Image Gen & Edit, Video LROs | `ATLAS_API_KEY` |
|
|
65
46
|
|
|
66
|
-
##
|
|
47
|
+
## ⚙️ Configuration
|
|
67
48
|
|
|
68
|
-
API
|
|
49
|
+
API Keys can be provided as standard environment variables. Alternatively, you can configure them (along with advanced options) via JSON configuration.
|
|
69
50
|
|
|
70
|
-
|
|
71
|
-
-
|
|
51
|
+
**Configuration Path**:
|
|
52
|
+
`~/.pi/agent/media-models.json` (Global) or `<project_root>/.pi/media-models.json` (Project-specific).
|
|
72
53
|
|
|
73
54
|
```json
|
|
74
55
|
{
|
|
75
|
-
"outputDir": "
|
|
56
|
+
"outputDir": "/path/to/custom/output/directory",
|
|
76
57
|
"providerOptions": {
|
|
58
|
+
"vertex": {
|
|
59
|
+
"credentialsFile": "/path/to/vertex-service-account.json",
|
|
60
|
+
"project": "my-gcp-project",
|
|
61
|
+
"location": "us-central1"
|
|
62
|
+
},
|
|
77
63
|
"dashscope": {
|
|
78
|
-
"baseUrl": "https
|
|
64
|
+
"baseUrl": "https://<workspace_id>.cn-beijing.maas.aliyuncs.com"
|
|
79
65
|
}
|
|
80
66
|
},
|
|
81
67
|
"customProviders": [
|
|
82
68
|
{
|
|
83
|
-
"id": "my-
|
|
84
|
-
"name": "
|
|
85
|
-
"baseUrl": "https://
|
|
86
|
-
"apiKeyEnv": "
|
|
69
|
+
"id": "my-custom-ai",
|
|
70
|
+
"name": "Internal AI Gateway",
|
|
71
|
+
"baseUrl": "https://api.internal.com/v1",
|
|
72
|
+
"apiKeyEnv": "MY_INTERNAL_KEY",
|
|
87
73
|
"auth": "bearer",
|
|
88
74
|
"models": [
|
|
89
75
|
{
|
|
90
|
-
"id": "
|
|
91
|
-
"vendor": "
|
|
92
|
-
"capabilities": ["image.text_to_image"],
|
|
93
|
-
"endpoints": {
|
|
94
|
-
"image.text_to_image": "/images/generations"
|
|
95
|
-
}
|
|
96
|
-
},
|
|
97
|
-
{
|
|
98
|
-
"id": "vendor/video-model",
|
|
99
|
-
"vendor": "vendor",
|
|
76
|
+
"id": "internal-video-pro",
|
|
77
|
+
"vendor": "internal",
|
|
100
78
|
"capabilities": ["video.text_to_video"],
|
|
101
79
|
"endpoints": {
|
|
102
|
-
"video.text_to_video":
|
|
103
|
-
"path": "/videos",
|
|
104
|
-
"format": "json",
|
|
105
|
-
"async": {
|
|
106
|
-
"idPath": "id",
|
|
107
|
-
"statusPath": "status",
|
|
108
|
-
"pollEndpoint": "/videos/{id}",
|
|
109
|
-
"resultPath": "result",
|
|
110
|
-
"cancelEndpoint": "/videos/{id}/cancel",
|
|
111
|
-
"successValues": ["completed"],
|
|
112
|
-
"failureValues": ["failed", "cancelled"]
|
|
113
|
-
}
|
|
114
|
-
}
|
|
80
|
+
"video.text_to_video": "/videos/generations"
|
|
115
81
|
}
|
|
116
82
|
}
|
|
117
83
|
]
|
|
@@ -120,55 +86,23 @@ API Key 只从环境变量读取，不写 JSON。可选配置：
|
|
|
120
86
|
}
|
|
121
87
|
```
|
|
122
88
|
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
## 使用示例
|
|
126
|
-
|
|
127
|
-
先调用 `media_models` 查看 capability，再调用统一 Tool。例如：
|
|
128
|
-
|
|
129
|
-
```json
|
|
130
|
-
{
|
|
131
|
-
"provider": "xai",
|
|
132
|
-
"model": "grok-imagine-video-1.5",
|
|
133
|
-
"prompt": "A paper boat drifting down a rainy street",
|
|
134
|
-
"inputImage": "C:/assets/boat.png",
|
|
135
|
-
"duration": 8,
|
|
136
|
-
"resolution": "720p",
|
|
137
|
-
"aspectRatio": "16:9",
|
|
138
|
-
"generateAudio": true
|
|
139
|
-
}
|
|
140
|
-
```
|
|
141
|
-
|
|
142
|
-
Atlas 多模态 reference-to-video：
|
|
143
|
-
|
|
144
|
-
```json
|
|
145
|
-
{
|
|
146
|
-
"provider": "atlas",
|
|
147
|
-
"model": "bytedance/seedance-2.0/text-to-video",
|
|
148
|
-
"prompt": "Product launch film",
|
|
149
|
-
"referenceImages": ["https://example.com/product.png"],
|
|
150
|
-
"referenceVideos": ["https://example.com/motion.mp4"],
|
|
151
|
-
"referenceAudios": ["https://example.com/voice.mp3"],
|
|
152
|
-
"generateAudio": true
|
|
153
|
-
}
|
|
154
|
-
```
|
|
89
|
+
*Note: Custom OpenAI-compatible endpoints require explicit capability mapping in the configuration, avoiding hallucinated unsupported paths from standard `/models` probing.*
|
|
155
90
|
|
|
156
|
-
##
|
|
91
|
+
## 🛠️ Exposed Tools
|
|
157
92
|
|
|
158
|
-
|
|
159
|
-
- HTTP 请求有超时；幂等请求对 429/5xx 退避重试并尊重 `Retry-After`。为避免重复计费，非幂等生成 POST 默认不自动重试。
|
|
160
|
-
- 异步任务统一支持 polling、backoff、总超时和 `AbortSignal`；仅在 Provider 文档明确提供 cancel 时调用远端取消（fal、DashScope、自定义显式 cancel）。
|
|
161
|
-
- 测试进程设置 `PI_MEDIA_TEST_MODE=1`，未注入 mock fetch 时真实网络请求会直接失败，避免付费误调用。
|
|
93
|
+
The extension registers the following unified tools for the reasoning agent:
|
|
162
94
|
|
|
163
|
-
|
|
95
|
+
1. `media_models`: Lists providers, configured models, and capabilities.
|
|
96
|
+
2. `image_generate`: Generate images from text, image, or multiple reference inputs.
|
|
97
|
+
3. `image_edit`: Edit existing images (supports masks and multiple references).
|
|
98
|
+
4. `video_generate`: Generates, edits, or extends videos. Automatically maps inputs (`referenceImages`, `inputVideo`, `duration`, `generateAudio`, etc.) to the provider's exact capability.
|
|
99
|
+
5. `audio_generate`: Generate music or raw audio (separate from TTS).
|
|
100
|
+
6. `speech_generate`: Handle TTS (Text-to-Speech) and STT (Speech-to-Text).
|
|
164
101
|
|
|
165
|
-
|
|
166
|
-
- fal 的输入/输出 schema 按 endpoint 变化，通用字段可通过 `providerOptions` 覆盖；应按所选 endpoint 文档传原生字段。
|
|
167
|
-
- xAI 官方未公开视频任务 cancel；中止只停止本地 polling。自定义音频 reference 可能要求 trusted-partner 权限。
|
|
168
|
-
- Gemini/Veo 和 Atlas 未公开视频任务 cancel。Atlas 文档未声明独立 TTS/STT、视频 edit/extend，因此不虚构这些 capability。
|
|
169
|
-
- Vertex Veo REST 使用 `:fetchPredictOperation`；GCS 输出需要调用身份有对象读取权限。建议配置 Provider 原生输出到可读 GCS 或返回 base64。
|
|
170
|
-
- 本仓库测试不进行真实付费生成；真实 Key、配额、内容策略和临时 URL 生命周期需在用户明确授权后做 smoke test。
|
|
102
|
+
## 🔒 Security & Privacy
|
|
171
103
|
|
|
172
|
-
|
|
104
|
+
- **No Key Logging**: API keys and Bearer tokens are redacted (`[REDACTED]`) from all error logs and HTTP outputs before being returned to the LLM.
|
|
105
|
+
- **Stateless Configuration**: Configuration does not hardcode user secrets if initialized via environment variables.
|
|
173
106
|
|
|
174
|
-
|
|
107
|
+
---
|
|
108
|
+
*Built for production multimodal orchestration inside Pi.*
|