pi-cliproxyapi-native 0.3.3 → 0.3.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +50 -41
- package/package.json +8 -6
- package/src/catalog.ts +30 -15
- package/src/media-defaults.ts +1 -1
- package/src/media.ts +208 -51
package/README.md
CHANGED
|
@@ -31,15 +31,15 @@ Chat uses Pi's native API adapters and model metadata, with availability from yo
|
|
|
31
31
|
|
|
32
32
|
## Install
|
|
33
33
|
|
|
34
|
-
The extension targets Pi 1.
|
|
34
|
+
The extension targets Pi 1.5.0 and CLIProxyAPI 8.0.57, with Node.js 22.19 or newer. Run CLIProxyAPI with your upstream accounts configured before connecting Pi. The extension uses Pi's host-provided packages, adds no runtime dependencies, and needs no compilation.
|
|
35
35
|
|
|
36
36
|
If you installed `pi-cliproxy-native`, remove it before installing the renamed package. Both register the same `cliproxyapi` provider.
|
|
37
37
|
|
|
38
38
|
```sh
|
|
39
|
-
pi install npm:pi-cliproxyapi-native@0.3.
|
|
39
|
+
pi install npm:pi-cliproxyapi-native@0.3.5
|
|
40
40
|
```
|
|
41
41
|
|
|
42
|
-
For Git installation, use `pi install git:github.com/hawkff/pi-cliproxyapi-native@v0.3.
|
|
42
|
+
For Git installation, use `pi install git:github.com/hawkff/pi-cliproxyapi-native@v0.3.5`. For a local checkout, use `pi install /path/to/pi-cliproxyapi-native`.
|
|
43
43
|
|
|
44
44
|
The default proxy address is `http://localhost:8317`. For another address, set the [connection](#connection) before logging in. Restart Pi, then run:
|
|
45
45
|
|
|
@@ -117,7 +117,7 @@ Check video status for request_id <returned-id>.
|
|
|
117
117
|
| `cliproxyapi_generate_video` | `prompt`; optional `model`, `resolution`, `aspect_ratio`, and integer `duration` from 1 to 15 seconds | A `request_id` for the submitted video. |
|
|
118
118
|
| `cliproxyapi_video_status` | `request_id` | One status check: pending, completed with a URL, or failed. |
|
|
119
119
|
|
|
120
|
-
Pass `model` to choose an exact ID for one tool call. Otherwise, the tool uses its saved `/cli:model` selection. Without either, eligible OpenAI/GPT chats default to `gpt-image-2.
|
|
120
|
+
Pass `model` to choose an exact ID for one tool call. Otherwise, the tool uses its saved `/cli:model` selection. Without either, eligible OpenAI/GPT chats default to `gpt-image-2.6-sunburst` for images, which needs CLIProxyAPI 8.0.56 or newer. Other chats need an image selection or explicit ID. Videos need a selection or explicit ID.
|
|
121
121
|
|
|
122
122
|
The automatic image default applies to native OpenAI/Codex chats, recognized GPT chat IDs through other providers, and configured OpenAI metadata aliases. An OpenAI-compatible API alone does not qualify.
|
|
123
123
|
|
|
@@ -129,21 +129,21 @@ Invalid or disabled saved selections block automatic fallback until you clear or
|
|
|
129
129
|
|
|
130
130
|
Pass output controls for each generation, or omit them to keep the backend defaults. `cliproxyapi_media_models` lists each model's `controls.size`, `controls.resolution`, and `controls.aspect_ratio`. Empty lists mean the control is unsupported; `size_limits` describes custom pixel sizes beyond the listed presets.
|
|
131
131
|
|
|
132
|
-
| Models | Controls
|
|
133
|
-
| ------------- |
|
|
134
|
-
| OpenAI images | `size`: `auto` or `WIDTHxHEIGHT`. No separate `resolution` or `aspect_ratio`.
|
|
135
|
-
| Gemini images | `aspect_ratio` and model-specific `resolution` tiers.
|
|
136
|
-
| xAI images | `resolution`: `1k` or `2k`, plus `aspect_ratio`.
|
|
137
|
-
| xAI videos | `resolution`: `480p` or `720p`, plus `aspect_ratio`. The 1.5 model
|
|
132
|
+
| Models | Controls |
|
|
133
|
+
| ------------- | ------------------------------------------------------------------------------------------------------------------------ |
|
|
134
|
+
| OpenAI images | `size`: `auto` or `WIDTHxHEIGHT`. No separate `resolution` or `aspect_ratio`. |
|
|
135
|
+
| Gemini images | `aspect_ratio` and model-specific `resolution` tiers. |
|
|
136
|
+
| xAI images | `resolution`: `1k` or `2k`, plus `aspect_ratio`. |
|
|
137
|
+
| xAI videos | `resolution`: `480p` or `720p`, plus `aspect_ratio`. The 1.5 model, its preview alias, and 1.5 Lite also accept `1080p`. |
|
|
138
138
|
|
|
139
|
-
GPT Image 1.5 accepts `auto`, `1024x1024`, `1536x1024`, and `1024x1536`. GPT Image 2 and 2.
|
|
139
|
+
GPT Image 1.5 accepts `auto`, `1024x1024`, `1536x1024`, and `1024x1536`. GPT Image 2, 2.5, and 2.6 also accept custom dimensions: both edges must be multiples of 16 and at most 3840 pixels. The longer edge cannot exceed three times the shorter one. Total pixels must be between 655,360 and 8,294,400. OpenAI marks resolutions above `2560x1440` as experimental. For GIF output, `size` sets each frame and can go below the 655,360-pixel minimum. OpenAI checks the frame size, frame count, and image count against its request budget.
|
|
140
140
|
|
|
141
141
|
Gemini 3 Pro Image supports `1K`, `2K`, and `4K`. Gemini 3.1 Flash Image also supports `512`. Flash Lite Image supports `1K` only. Gemini 2.5 Flash Image has no resolution selector. Use the exact case shown in discovery, including uppercase `K` for Gemini and lowercase `k` for xAI.
|
|
142
142
|
|
|
143
143
|
Aspect ratios vary by model and proxy route. For example, CLIProxyAPI 8.0.7 accepts `20:9` for xAI images but drops xAI's `21:9` and `auto` values. The extension rejects unsupported controls before authentication or network access instead of substituting another size or shape.
|
|
144
144
|
|
|
145
145
|
```text
|
|
146
|
-
Generate an image with gpt-image-2.
|
|
146
|
+
Generate an image with gpt-image-2.6-sunburst at size 2048x2048.
|
|
147
147
|
Generate a 2K image at 16:9 with gemini-3-pro-image.
|
|
148
148
|
Generate a 1-second 1080p video at 9:16 with grok-imagine-video-1.5.
|
|
149
149
|
```
|
|
@@ -156,53 +156,62 @@ Control values follow the [OpenAI image guide](https://developers.openai.com/api
|
|
|
156
156
|
|
|
157
157
|
Use `cliproxyapi_media_models` to inspect controls for the exact advertised model ID. Omitted options keep the backend defaults. The extension rejects unsupported values before contacting the proxy.
|
|
158
158
|
|
|
159
|
-
| Option | Supported values
|
|
160
|
-
| -------------------- |
|
|
161
|
-
| `quality` | `auto`, `low`, `medium`, `high`. Sunburst and Flare also accept `xhigh` and `max`.
|
|
162
|
-
| `
|
|
163
|
-
| `
|
|
164
|
-
| `
|
|
165
|
-
| `
|
|
166
|
-
| `
|
|
167
|
-
| `
|
|
168
|
-
| `
|
|
169
|
-
| `
|
|
170
|
-
| `
|
|
171
|
-
|
|
172
|
-
|
|
159
|
+
| Option | Supported values |
|
|
160
|
+
| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
161
|
+
| `quality` | `auto`, `low`, `medium`, `high`. Sunburst and Flare also accept `xhigh` and `max`. |
|
|
162
|
+
| `complexity` | `low`, `medium`, `high` for GPT Image 2.6 Sunburst and Flare. It sets how much the model plans before rendering, and OpenAI defaults to `high`. Use `low` for quick edits. |
|
|
163
|
+
| `background` | `auto`, `opaque`, `transparent`. Transparent output requires PNG or WebP; GPT Image 2 transparency is in preview upstream. |
|
|
164
|
+
| `output_format` | `png`, `jpeg`, `webp`. GPT Image 2.6 Sunburst and Flare also generate `gif` and `svg`. JPEG and WebP can reduce file size and help keep results within the response limit. |
|
|
165
|
+
| `output_compression` | Integer from 0 to 100, with an explicit `jpeg` or `webp` format. Zero is a valid value. |
|
|
166
|
+
| `frame_count` | GIF output only. Integer from 2 to 16. OpenAI defaults to 9. |
|
|
167
|
+
| `frame_duration_ms` | GIF output only. Milliseconds per frame from 20 to 2000, rounded to 10. OpenAI defaults to 200. |
|
|
168
|
+
| `moderation` | `auto` or `low`. OpenAI still applies its content policies. |
|
|
169
|
+
| `n` | Integer from 1 to 10 on an explicit `api/` image route. Other routes allow only 1 because subscription translation can drop the count. |
|
|
170
|
+
| `partial_images` | Integer from 0 to 3. The proxy streams up to that many draft frames while the image renders, and Pi shows them as tool progress. Requires `n` of 1 and PNG, JPEG, or WebP output. Each draft adds about 100 image output tokens. |
|
|
171
|
+
| `images` | One to 16 local PNG, JPEG, or WebP paths. Supplying these selects `/v1/images/edits` instead of generation. |
|
|
172
|
+
| `mask` | Optional local PNG mask. The first image must also be PNG with matching dimensions. The mask needs a grayscale-alpha or RGBA alpha channel; transparent regions mark edits. |
|
|
173
|
+
| `markup` | Optional local PNG annotation layer for GPT Image 2.5 and 2.6 Sunburst and Flare. It needs an alpha channel, the first image's dimensions, and transparency outside the marks. It cannot be combined with `mask`. |
|
|
174
|
+
| `input_fidelity` | `low` or `high`, for GPT Image 1.5 editing only. Omit it for other models. |
|
|
175
|
+
|
|
176
|
+
Editing uploads the files you name to the configured proxy. Paths can be absolute or relative to the working directory. The extension accepts regular files, checks file signatures, and limits uploads to 14 MiB per file and 20 MiB total including the mask or markup layer. It compares markup dimensions only with a PNG first image; OpenAI checks JPEG and WebP inputs. It does not fetch input URLs, resize inputs, or modify source files. OpenAI performs full image decoding and validates the edit. These local limits keep base64 uploads bounded and may be lower than upstream limits.
|
|
173
177
|
|
|
174
178
|
```text
|
|
175
|
-
Generate two images with api/gpt-image-2.
|
|
176
|
-
Edit ./photo.png with api/gpt-image-2.
|
|
179
|
+
Generate two images with api/gpt-image-2.6-sunburst at 1536x864, quality max, output_format webp, output_compression 85.
|
|
180
|
+
Edit ./photo.png with api/gpt-image-2.6-flare: replace the background with a garden, using ./mask.png as the mask.
|
|
181
|
+
Edit ./kitchen.webp with gpt-image-2.6-flare and complexity low, using ./circle.png as markup: recolor the circled vase cobalt blue.
|
|
182
|
+
Generate a looping GIF of a waving robot with gpt-image-2.6-flare, output_format gif, frame_count 8, size 512x512.
|
|
183
|
+
Generate an SVG logo for a bakery with gpt-image-2.6-sunburst and output_format svg.
|
|
177
184
|
```
|
|
178
185
|
|
|
186
|
+
GIF and SVG output work only for generation, without streaming, `partial_images`, or `output_compression`. SVG also needs `n` of 1. OpenAI renders a reference image at the chosen `quality`, and a second model writes the SVG from it. The extension saves `.gif` and `.svg` files without inline previews and checks the GIF signature. SVG must be UTF-8 static drawing markup with presentation attributes and references only inside the file, which matches OpenAI's documented output. The extension rejects anything else before saving, including scripts, CSS, comments, animation, event handlers, and external references.
|
|
187
|
+
|
|
179
188
|
All returned images share the existing 32 MiB response limit and 4 MiB preview budget. Streamed requests allow another 32 MiB for each draft frame. The extension saves only the final image and drops drafts it cannot decode. Larger sizes, higher quality, and multiple images can increase cost and latency. Use fewer images or JPEG/WebP if the response exceeds the limit. Generation and editing do not retry after a failure.
|
|
180
189
|
|
|
181
|
-
The extension supports the verified options above, not a general OpenAI API passthrough. Remote file IDs,
|
|
190
|
+
The extension supports the verified options above, not a general OpenAI API passthrough. Remote file IDs, SVG author model selection, audio, batches, and realtime APIs are not exposed. Adding an upstream API key does not enable endpoints that the proxy or extension cannot handle.
|
|
182
191
|
|
|
183
192
|
### Image usage and cost
|
|
184
193
|
|
|
185
194
|
OpenAI image responses report token usage. The image tool returns it to Pi, so image calls count toward the session totals in the footer and `/session`.
|
|
186
195
|
|
|
187
|
-
Costs are estimates at OpenAI list prices per million tokens: $5 for text input, $8 for image input, and $30 for image output on GPT Image 2 and 2.5 Sunburst and Flare. GPT Image 1.5 output costs $32. The plain `gpt-image-2.5` subscription
|
|
196
|
+
Costs are estimates at OpenAI list prices per million tokens: $5 for text input, $8 for image input, and $30 for image output on GPT Image 2, and on 2.5 and 2.6 Sunburst and Flare. GPT Image 1.5 output costs $32. The plain `gpt-image-2.5` and `gpt-image-2.6` subscription models have no published price, so they report tokens at zero cost. SVG output also reports the tokens of the model that writes the SVG. The extension adds them to the totals and prices them from Pi's OpenAI model metadata, or at zero when Pi doesn't know that model. Subscription routes show the same estimates when the proxy reports usage, even though the plan covers them. Google and xAI image results report no usage.
|
|
188
197
|
|
|
189
198
|
### Pi image models
|
|
190
199
|
|
|
191
200
|
When Pi refreshes the catalog, at interactive startup or through `/cliproxyapi-refresh`, the advertised image models also register as Pi image models. Codemode scripts list them with `models.getAvailableOfType("image", "cliproxyapi")` and call `models.generateImages()`, and other extensions can use `ctx.modelRegistry.generateImages()`.
|
|
192
201
|
|
|
193
|
-
These calls send the prompt with backend defaults. Pi's image interface has no size, quality, count, mask, or preview options. Image blocks in the input select OpenAI editing, under the same 16-image, 14 MiB per image, and 20 MiB total limits. Google and xAI models accept text only. Results arrive as image blocks with usage and are not saved to disk. Use `cliproxyapi_generate_image` when you need controls, masks, or saved files.
|
|
202
|
+
These calls send the prompt with backend defaults. Pi's image interface has no size, quality, count, format, mask, markup, or preview options. Image blocks in the input select OpenAI editing, under the same 16-image, 14 MiB per image, and 20 MiB total limits. Google and xAI models accept text only. Results arrive as image blocks with usage and are not saved to disk. Use `cliproxyapi_generate_image` when you need controls, masks, markup, GIF or SVG output, or saved files.
|
|
194
203
|
|
|
195
204
|
### Supported media models
|
|
196
205
|
|
|
197
206
|
Your proxy must advertise the exact ID for a model to be available.
|
|
198
207
|
|
|
199
|
-
| Provider | Models
|
|
200
|
-
| ------------- |
|
|
201
|
-
| OpenAI models | `gpt-image-2.5-flare`, `gpt-image-2.5-sunburst`, `gpt-image-2.5`, `gpt-image-2`, `gpt-image-1.5`
|
|
202
|
-
| Google models | `gemini-2.5-flash-image`, `gemini-3.1-flash-image`, `gemini-3-pro-image`, `gemini-3.1-flash-lite-image`
|
|
203
|
-
| xAI models | `grok-imagine-image`, `grok-imagine-image-quality`, `grok-imagine-image-2.0`, `grok-imagine-video`, `grok-imagine-video-1.5`, `grok-imagine-video-1.5-preview` |
|
|
208
|
+
| Provider | Models |
|
|
209
|
+
| ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
210
|
+
| OpenAI models | `gpt-image-2.6-flare`, `gpt-image-2.6-sunburst`, `gpt-image-2.6`, `gpt-image-2.5-flare`, `gpt-image-2.5-sunburst`, `gpt-image-2.5`, `gpt-image-2`, `gpt-image-1.5` |
|
|
211
|
+
| Google models | `gemini-2.5-flash-image`, `gemini-3.1-flash-image`, `gemini-3-pro-image`, `gemini-3.1-flash-lite-image` |
|
|
212
|
+
| xAI models | `grok-imagine-image`, `grok-imagine-image-quality`, `grok-imagine-image-2.0`, `grok-imagine-video`, `grok-imagine-video-1.5`, `grok-imagine-video-1.5-lite`, `grok-imagine-video-1.5-preview` |
|
|
204
213
|
|
|
205
|
-
Supported media IDs can also use advertised `api/` and `oauth/` credential routes. OpenAI's API has no plain `gpt-image-2.5`, so the picker disables `api
|
|
214
|
+
Supported media IDs can also use advertised `api/` and `oauth/` credential routes. OpenAI's API has no plain `gpt-image-2.5` or `gpt-image-2.6`, so the picker disables their `api/` routes and points to Sunburst or Flare; subscription routes keep them. Gemini image models also support `vertex/` and `antigravity/` backend routes. Those backend prefixes do not enable OpenAI or xAI media execution; the picker shows those entries as disabled. Other prefixes, nested prefixes, media aliases, unknown media IDs, Google Veo, and non-OpenAI editing are unsupported.
|
|
206
215
|
|
|
207
216
|
The extension disables these retired Imagen IDs, including their supported prefixed forms:
|
|
208
217
|
|
|
@@ -216,9 +225,9 @@ The picker explains their retirement, and generation rejects them before network
|
|
|
216
225
|
|
|
217
226
|
### Files, status, and limits
|
|
218
227
|
|
|
219
|
-
Image generation saves JPEG, PNG, or
|
|
228
|
+
Image generation saves JPEG, PNG, WebP, GIF, or SVG files under a new `.pi/cliproxyapi-image-*/` directory in your working directory. The extension checks base64 and file signatures, uses private permissions, and does not overwrite existing files. Keep `.pi/` out of version control.
|
|
220
229
|
|
|
221
|
-
Inline previews require a vision-capable chat model and share a 4 MiB base64 budget per result. Larger images return file paths without previews. Use Pi's `read` tool to inspect saved images. URL-only image responses fail without a download.
|
|
230
|
+
Inline previews require a vision-capable chat model and share a 4 MiB base64 budget per result. Larger images return file paths without previews, and GIF and SVG results never include one. Use Pi's `read` tool to inspect saved images. URL-only image responses fail without a download.
|
|
222
231
|
|
|
223
232
|
Gemini responses must contain one completed candidate. The extension skips thought parts and saves up to 16 final images within a 256-part response limit. Safety/refusal signals, incomplete or text-only results, malformed data, MIME/signature mismatches, and excess images or parts fail before saving.
|
|
224
233
|
|
|
@@ -384,11 +393,11 @@ On `api/` routes, Pi can send thinking effort as a `configuration_update` item.
|
|
|
384
393
|
|
|
385
394
|
When Pi omits strictness in the top-level Responses tool list, the extension sends `strict: null`, so optional arguments remain optional. Tools declared inside the conversation keep Pi's strictness values. Explicit payload hooks retain final control.
|
|
386
395
|
|
|
387
|
-
OpenAI images POST `model`, `prompt`, `n`, and supplied controls to `/v1/images/generations`, without `response_format`. Editing uses `/v1/images/edits` with JSON `images` and optional `mask` data URLs. A nonzero `partial_images` adds `stream: true`, and the extension reads server-sent events. Each `*.partial_image` frame becomes a progress preview, and the `*.completed` frame carries the final image and usage. xAI uses the same route with `response_format: "b64_json"`. Gemini uses `/v1beta/models/{id}:generateContent`. Media generation sends the prompt and generation options without chat history, system instructions, or tools.
|
|
396
|
+
OpenAI images POST `model`, `prompt`, `n`, and supplied controls to `/v1/images/generations`, without `response_format`. Editing uses `/v1/images/edits` with JSON `images` and optional `mask` or `markup` data URLs. A nonzero `partial_images` adds `stream: true`, and the extension reads server-sent events. Each `*.partial_image` frame becomes a progress preview, and the `*.completed` frame carries the final image and usage. xAI uses the same route with `response_format: "b64_json"`. Gemini uses `/v1beta/models/{id}:generateContent`. Media generation sends the prompt and generation options without chat history, system instructions, or tools.
|
|
388
397
|
|
|
389
398
|
OpenAI image POSTs use a dedicated HTTP connection. It follows `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` from the environment, where Pi also places its `httpProxy` setting, on Node 22.21+ and 24.5+. Older Node versions connect directly. The connection disables socket timeouts so the ten-minute deadline controls the request, requests identity encoding, and rejects compressed responses.
|
|
390
399
|
|
|
391
|
-
The OpenAI image implementation follows CLIProxyAPI v8.0.
|
|
400
|
+
The OpenAI image implementation follows CLIProxyAPI v8.0.57's [image handler](https://github.com/router-for-me/CLIProxyAPI/blob/v8.0.57/sdk/api/handlers/openai/openai_images_handlers.go) and [Codex image executor](https://github.com/router-for-me/CLIProxyAPI/blob/v8.0.57/internal/runtime/executor/codex_openai_images.go), plus the OpenAI [generation](https://developers.openai.com/api/reference/resources/images/methods/generate) and [editing](https://developers.openai.com/api/reference/resources/images/methods/edit) references. The xAI and Google routes follow v7.2.158's [video handler](https://github.com/router-for-me/CLIProxyAPI/blob/v7.2.158/sdk/api/handlers/openai/openai_videos_handlers.go), [Gemini handler](https://github.com/router-for-me/CLIProxyAPI/blob/v7.2.158/sdk/api/handlers/gemini/gemini_handlers.go), and [Vertex executor](https://github.com/router-for-me/CLIProxyAPI/blob/v7.2.158/internal/runtime/executor/gemini_vertex_executor.go). See [xAI's video documentation](https://docs.x.ai/developers/model-capabilities/video/generation) for duration and status semantics.
|
|
392
401
|
|
|
393
402
|
## License
|
|
394
403
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-cliproxyapi-native",
|
|
3
|
-
"version": "0.3.
|
|
3
|
+
"version": "0.3.5",
|
|
4
4
|
"description": "CLIProxyAPI chat discovery, image generation and editing, and video generation with Pi's native adapters",
|
|
5
5
|
"publishConfig": {
|
|
6
6
|
"access": "public"
|
|
@@ -38,16 +38,18 @@
|
|
|
38
38
|
"check": "pnpm typecheck && pnpm lint && pnpm test"
|
|
39
39
|
},
|
|
40
40
|
"devDependencies": {
|
|
41
|
-
"@biomejs/biome": "^2.6.
|
|
42
|
-
"@earendil-works/pi-ai": "^1.
|
|
43
|
-
"@earendil-works/pi-coding-agent": "^1.
|
|
44
|
-
"@earendil-works/pi-tui": "^1.
|
|
41
|
+
"@biomejs/biome": "^2.6.5",
|
|
42
|
+
"@earendil-works/pi-ai": "^1.5.0",
|
|
43
|
+
"@earendil-works/pi-coding-agent": "^1.5.0",
|
|
44
|
+
"@earendil-works/pi-tui": "^1.5.0",
|
|
45
45
|
"@types/node": "^24.19.2",
|
|
46
|
+
"typebox": "1.3.27",
|
|
46
47
|
"typescript": "7.0.2"
|
|
47
48
|
},
|
|
48
49
|
"peerDependencies": {
|
|
49
50
|
"@earendil-works/pi-ai": "*",
|
|
50
51
|
"@earendil-works/pi-coding-agent": "*",
|
|
51
|
-
"@earendil-works/pi-tui": "*"
|
|
52
|
+
"@earendil-works/pi-tui": "*",
|
|
53
|
+
"typebox": "*"
|
|
52
54
|
}
|
|
53
55
|
}
|
package/src/catalog.ts
CHANGED
|
@@ -118,6 +118,15 @@ const mediaModels = new Map<
|
|
|
118
118
|
price?: readonly [number, number, number];
|
|
119
119
|
}
|
|
120
120
|
>([
|
|
121
|
+
[
|
|
122
|
+
"gpt-image-2.6-flare",
|
|
123
|
+
{ name: "GPT Image 2.6 Flare", purpose: "image", route: "openai", price: gptImage2Price },
|
|
124
|
+
],
|
|
125
|
+
[
|
|
126
|
+
"gpt-image-2.6-sunburst",
|
|
127
|
+
{ name: "GPT Image 2.6 Sunburst", purpose: "image", route: "openai", price: gptImage2Price },
|
|
128
|
+
],
|
|
129
|
+
["gpt-image-2.6", { name: "GPT Image 2.6", purpose: "image", route: "openai" }],
|
|
121
130
|
[
|
|
122
131
|
"gpt-image-2.5-flare",
|
|
123
132
|
{ name: "GPT Image 2.5 Flare", purpose: "image", route: "openai", price: gptImage2Price },
|
|
@@ -134,6 +143,7 @@ const mediaModels = new Map<
|
|
|
134
143
|
["grok-imagine-image-2.0", { name: "Grok Imagine Image 2.0", purpose: "image", route: "xai" }],
|
|
135
144
|
["grok-imagine-video", { name: "Grok Imagine Video", purpose: "video", route: "xai" }],
|
|
136
145
|
["grok-imagine-video-1.5", { name: "Grok Imagine Video 1.5", purpose: "video", route: "xai" }],
|
|
146
|
+
["grok-imagine-video-1.5-lite", { name: "Grok Imagine Video 1.5 Lite", purpose: "video", route: "xai" }],
|
|
137
147
|
[
|
|
138
148
|
"grok-imagine-video-1.5-preview",
|
|
139
149
|
{ name: "Grok Imagine Video 1.5 Preview", purpose: "video", route: "xai" },
|
|
@@ -164,13 +174,12 @@ export function mediaCapability(id: string) {
|
|
|
164
174
|
if (backend && (capability?.route === "xai" || capability?.route === "openai")) {
|
|
165
175
|
return { ...capability, route: undefined, disabledReason: `Unsupported media execution on ${backend}.` };
|
|
166
176
|
}
|
|
167
|
-
// CLIProxyAPI
|
|
168
|
-
if (capability && authType === "api-key" &&
|
|
177
|
+
// CLIProxyAPI advertises the plain names for API keys, but OpenAI's Images API has only Sunburst and Flare.
|
|
178
|
+
if (capability && authType === "api-key" && /^gpt-image-2\.[56]$/.test(metadataId)) {
|
|
169
179
|
return {
|
|
170
180
|
...capability,
|
|
171
181
|
route: undefined,
|
|
172
|
-
disabledReason:
|
|
173
|
-
"OpenAI's API has no gpt-image-2.5 model. Use api/gpt-image-2.5-sunburst or api/gpt-image-2.5-flare.",
|
|
182
|
+
disabledReason: `OpenAI's API has no ${metadataId} model. Use api/${metadataId}-sunburst or api/${metadataId}-flare.`,
|
|
174
183
|
};
|
|
175
184
|
}
|
|
176
185
|
return capability;
|
|
@@ -184,6 +193,9 @@ export function mediaControls(id: string) {
|
|
|
184
193
|
const capability = mediaCapability(id);
|
|
185
194
|
const { metadataId, authType } = modelRoute(id);
|
|
186
195
|
const openai = capability?.route === "openai";
|
|
196
|
+
// Sunburst and Flare add quality tiers and markup edits; 2.6 adds complexity plus GIF and SVG output.
|
|
197
|
+
const tiered = openai && /^gpt-image-2\.[56]-(sunburst|flare)$/.test(metadataId);
|
|
198
|
+
const image26 = tiered && metadataId.startsWith("gpt-image-2.6-");
|
|
187
199
|
const commonRatios = ["1:1", "16:9", "9:16", "4:3", "3:4", "3:2", "2:3"];
|
|
188
200
|
let resolution: string[] = [];
|
|
189
201
|
let aspect_ratio: string[] = [];
|
|
@@ -213,24 +225,27 @@ export function mediaControls(id: string) {
|
|
|
213
225
|
: undefined,
|
|
214
226
|
resolution,
|
|
215
227
|
aspect_ratio,
|
|
216
|
-
quality: openai
|
|
217
|
-
|
|
218
|
-
"auto",
|
|
219
|
-
"low",
|
|
220
|
-
"medium",
|
|
221
|
-
"high",
|
|
222
|
-
...(/^gpt-image-2\.5-(sunburst|flare)$/.test(metadataId) ? ["xhigh", "max"] : []),
|
|
223
|
-
]
|
|
224
|
-
: [],
|
|
228
|
+
quality: openai ? ["auto", "low", "medium", "high", ...(tiered ? ["xhigh", "max"] : [])] : [],
|
|
229
|
+
complexity: image26 ? ["low", "medium", "high"] : [],
|
|
225
230
|
background: openai ? ["auto", "opaque", "transparent"] : [],
|
|
226
|
-
output_format: openai ? ["png", "jpeg", "webp"] : [],
|
|
231
|
+
output_format: openai ? ["png", "jpeg", "webp", ...(image26 ? ["gif", "svg"] : [])] : [],
|
|
232
|
+
// GIF sizes are per frame and skip min_pixels; OpenAI checks the animation against its pixel budget.
|
|
233
|
+
gif: image26
|
|
234
|
+
? { frame_count: { min: 2, max: 16 }, frame_duration_ms: { min: 20, max: 2000 } }
|
|
235
|
+
: undefined,
|
|
227
236
|
output_compression: openai ? { min: 0, max: 100, formats: ["jpeg", "webp"] } : undefined,
|
|
228
237
|
moderation: openai ? ["auto", "low"] : [],
|
|
229
238
|
// Subscription image-tool translation does not forward n.
|
|
230
239
|
n: openai ? { min: 1, max: authType === "api-key" ? 10 : 1 } : undefined,
|
|
231
240
|
partial_images: openai ? { min: 0, max: 3 } : undefined,
|
|
232
241
|
editing: openai
|
|
233
|
-
? {
|
|
242
|
+
? {
|
|
243
|
+
max_images: 16,
|
|
244
|
+
max_file_bytes: 14 * 1024 * 1024,
|
|
245
|
+
max_total_bytes: 20 * 1024 * 1024,
|
|
246
|
+
mask: "png",
|
|
247
|
+
...(tiered ? { markup: "png" } : {}),
|
|
248
|
+
}
|
|
234
249
|
: undefined,
|
|
235
250
|
input_fidelity: openai && metadataId === "gpt-image-1.5" ? ["low", "high"] : [],
|
|
236
251
|
};
|
package/src/media-defaults.ts
CHANGED
|
@@ -3,7 +3,7 @@ import { type MediaPurpose, mediaCapability, mediaPurpose, modelRoute } from "./
|
|
|
3
3
|
import { type Config, isModelId, isRecord, PROVIDER_ID } from "./config.ts";
|
|
4
4
|
|
|
5
5
|
export const MEDIA_DEFAULTS_ENTRY = "cliproxyapi-media-defaults";
|
|
6
|
-
export const OPENAI_IMAGE_DEFAULT = "gpt-image-2.
|
|
6
|
+
export const OPENAI_IMAGE_DEFAULT = "gpt-image-2.6-sunburst";
|
|
7
7
|
// null blocks fallback for invalid selections, including after unrelated picker saves.
|
|
8
8
|
export type MediaDefaults = Partial<Record<MediaPurpose, string | null>>;
|
|
9
9
|
export type DefaultsContext = {
|
package/src/media.ts
CHANGED
|
@@ -4,19 +4,19 @@ import * as http from "node:http";
|
|
|
4
4
|
import * as https from "node:https";
|
|
5
5
|
import { join, resolve } from "node:path";
|
|
6
6
|
import { addAbortSignal, Readable } from "node:stream";
|
|
7
|
-
import {
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
Type,
|
|
17
|
-
type Usage,
|
|
7
|
+
import type {
|
|
8
|
+
AssistantImages,
|
|
9
|
+
ImageApi,
|
|
10
|
+
ImageContent,
|
|
11
|
+
ImageModel,
|
|
12
|
+
ImagesContext,
|
|
13
|
+
ImagesOptions,
|
|
14
|
+
TextContent,
|
|
15
|
+
Usage,
|
|
18
16
|
} from "@earendil-works/pi-ai";
|
|
17
|
+
import { getBuiltinModels } from "@earendil-works/pi-ai/providers/all";
|
|
19
18
|
import type { ExtensionAPI, ExtensionContext } from "@earendil-works/pi-coding-agent";
|
|
19
|
+
import { type Static, Type } from "typebox";
|
|
20
20
|
import { fetchCatalog, mapMediaCatalog, mediaCapability, mediaControls } from "./catalog.ts";
|
|
21
21
|
import { type Config, deadline, isRecord, PROVIDER_ID, validateKey } from "./config.ts";
|
|
22
22
|
import { type DefaultsContext, effectiveMediaDefaults, resolveMediaModel } from "./media-defaults.ts";
|
|
@@ -66,13 +66,22 @@ const imageParameters = Type.Object({
|
|
|
66
66
|
description: "OpenAI quality from controls.quality; Sunburst and Flare also support xhigh and max.",
|
|
67
67
|
}),
|
|
68
68
|
),
|
|
69
|
+
complexity: Type.Optional(
|
|
70
|
+
Type.String({
|
|
71
|
+
description:
|
|
72
|
+
"GPT Image 2.6 Sunburst/Flare planning effort: low, medium, or high (upstream default high). Low suits quick edits.",
|
|
73
|
+
}),
|
|
74
|
+
),
|
|
69
75
|
background: Type.Optional(
|
|
70
76
|
Type.String({
|
|
71
77
|
description: "OpenAI background: auto, opaque, or transparent. Transparency requires PNG or WebP.",
|
|
72
78
|
}),
|
|
73
79
|
),
|
|
74
80
|
output_format: Type.Optional(
|
|
75
|
-
Type.String({
|
|
81
|
+
Type.String({
|
|
82
|
+
description:
|
|
83
|
+
"OpenAI output format: png, jpeg, or webp. GPT Image 2.6 Sunburst/Flare also generate gif and svg (svg needs n of 1). Omit for the backend default.",
|
|
84
|
+
}),
|
|
76
85
|
),
|
|
77
86
|
output_compression: Type.Optional(
|
|
78
87
|
Type.Integer({
|
|
@@ -81,6 +90,20 @@ const imageParameters = Type.Object({
|
|
|
81
90
|
description: "OpenAI JPEG/WebP compression, 0–100. Requires an explicit jpeg or webp output_format.",
|
|
82
91
|
}),
|
|
83
92
|
),
|
|
93
|
+
frame_count: Type.Optional(
|
|
94
|
+
Type.Integer({
|
|
95
|
+
minimum: 2,
|
|
96
|
+
maximum: 16,
|
|
97
|
+
description: "GIF output only: number of frames, 2–16 (upstream default 9).",
|
|
98
|
+
}),
|
|
99
|
+
),
|
|
100
|
+
frame_duration_ms: Type.Optional(
|
|
101
|
+
Type.Integer({
|
|
102
|
+
minimum: 20,
|
|
103
|
+
maximum: 2000,
|
|
104
|
+
description: "GIF output only: milliseconds per frame, 20–2000, rounded to 10 (upstream default 200).",
|
|
105
|
+
}),
|
|
106
|
+
),
|
|
84
107
|
moderation: Type.Optional(Type.String({ description: "OpenAI moderation: auto or low." })),
|
|
85
108
|
n: Type.Optional(
|
|
86
109
|
Type.Integer({
|
|
@@ -113,6 +136,13 @@ const imageParameters = Type.Object({
|
|
|
113
136
|
"Optional local PNG mask for images[0]. The first image must also be PNG with matching dimensions; mask must have an alpha channel. Transparent regions are edited.",
|
|
114
137
|
}),
|
|
115
138
|
),
|
|
139
|
+
markup: Type.Optional(
|
|
140
|
+
Type.String({
|
|
141
|
+
minLength: 1,
|
|
142
|
+
description:
|
|
143
|
+
"Optional local PNG annotation layer for images[0] with GPT Image 2.5/2.6 Sunburst or Flare: transparent except for marks showing where to edit. Same size as images[0]; cannot be combined with mask.",
|
|
144
|
+
}),
|
|
145
|
+
),
|
|
116
146
|
input_fidelity: Type.Optional(
|
|
117
147
|
Type.String({
|
|
118
148
|
description: "Editing fidelity for GPT Image 1.5 only: low or high. Omit for other models.",
|
|
@@ -123,7 +153,7 @@ const imageParameters = Type.Object({
|
|
|
123
153
|
minLength: 1,
|
|
124
154
|
maxLength: 32,
|
|
125
155
|
description:
|
|
126
|
-
"OpenAI image size: auto or WIDTHxHEIGHT. Use controls.size and controls.size_limits from cliproxyapi_media_models. Omit to preserve the backend default.",
|
|
156
|
+
"OpenAI image size: auto or WIDTHxHEIGHT. Use controls.size and controls.size_limits from cliproxyapi_media_models; GIF sizes are per frame and may go below min_pixels. Omit to preserve the backend default.",
|
|
127
157
|
}),
|
|
128
158
|
),
|
|
129
159
|
});
|
|
@@ -142,6 +172,7 @@ function mediaOutputOptions(model: string, params: Partial<Static<typeof imagePa
|
|
|
142
172
|
resolution: params.resolution,
|
|
143
173
|
aspect_ratio: params.aspect_ratio,
|
|
144
174
|
quality: params.quality,
|
|
175
|
+
complexity: params.complexity,
|
|
145
176
|
background: params.background,
|
|
146
177
|
output_format: params.output_format,
|
|
147
178
|
output_compression: params.output_compression,
|
|
@@ -149,13 +180,19 @@ function mediaOutputOptions(model: string, params: Partial<Static<typeof imagePa
|
|
|
149
180
|
n: params.n,
|
|
150
181
|
input_fidelity: params.input_fidelity,
|
|
151
182
|
partial_images: params.partial_images,
|
|
183
|
+
frame_count: params.frame_count,
|
|
184
|
+
frame_duration_ms: params.frame_duration_ms,
|
|
152
185
|
};
|
|
153
186
|
const controls = mediaControls(model);
|
|
187
|
+
const gif = options.output_format === "gif";
|
|
188
|
+
// OpenAI generates GIF and SVG without streaming and only on the generation endpoint.
|
|
189
|
+
const generationOnly = gif || options.output_format === "svg";
|
|
154
190
|
for (const field of [
|
|
155
191
|
"size",
|
|
156
192
|
"resolution",
|
|
157
193
|
"aspect_ratio",
|
|
158
194
|
"quality",
|
|
195
|
+
"complexity",
|
|
159
196
|
"background",
|
|
160
197
|
"output_format",
|
|
161
198
|
"moderation",
|
|
@@ -176,7 +213,7 @@ function mediaOutputOptions(model: string, params: Partial<Static<typeof imagePa
|
|
|
176
213
|
height % limits.multiple_of === 0 &&
|
|
177
214
|
Math.max(width, height) <= limits.max_edge &&
|
|
178
215
|
Math.max(width, height) / Math.min(width, height) <= limits.max_aspect_ratio &&
|
|
179
|
-
pixels >= limits.min_pixels &&
|
|
216
|
+
(gif || pixels >= limits.min_pixels) &&
|
|
180
217
|
pixels <= limits.max_pixels
|
|
181
218
|
)
|
|
182
219
|
continue;
|
|
@@ -197,11 +234,24 @@ function mediaOutputOptions(model: string, params: Partial<Static<typeof imagePa
|
|
|
197
234
|
!Number.isInteger(options.partial_images) ||
|
|
198
235
|
options.partial_images < controls.partial_images.min ||
|
|
199
236
|
options.partial_images > controls.partial_images.max ||
|
|
200
|
-
(options.partial_images > 0 && (options.n ?? 1) !== 1))
|
|
237
|
+
(options.partial_images > 0 && ((options.n ?? 1) !== 1 || generationOnly)))
|
|
201
238
|
)
|
|
202
239
|
throw new Error(
|
|
203
|
-
"CLIProxyAPI partial_images requires an OpenAI image model, n of 1, and an integer from 0 to 3.",
|
|
240
|
+
"CLIProxyAPI partial_images requires an OpenAI image model, n of 1, PNG, JPEG, or WebP output, and an integer from 0 to 3.",
|
|
204
241
|
);
|
|
242
|
+
for (const field of ["frame_count", "frame_duration_ms"] as const) {
|
|
243
|
+
const value = options[field];
|
|
244
|
+
const range = controls.gif?.[field];
|
|
245
|
+
if (
|
|
246
|
+
value !== undefined &&
|
|
247
|
+
(!gif || !range || !Number.isInteger(value) || value < range.min || value > range.max)
|
|
248
|
+
)
|
|
249
|
+
throw new Error(`CLIProxyAPI ${field} requires gif output_format and a value within controls.gif.`);
|
|
250
|
+
}
|
|
251
|
+
if (options.output_format === "svg" && (options.n ?? 1) !== 1)
|
|
252
|
+
throw new Error("CLIProxyAPI svg output_format requires n of 1.");
|
|
253
|
+
if (generationOnly && params.images !== undefined)
|
|
254
|
+
throw new Error("CLIProxyAPI gif and svg output_format are for generation, not editing.");
|
|
205
255
|
if (
|
|
206
256
|
options.output_compression !== undefined &&
|
|
207
257
|
(!controls.output_compression ||
|
|
@@ -213,7 +263,11 @@ function mediaOutputOptions(model: string, params: Partial<Static<typeof imagePa
|
|
|
213
263
|
throw new Error(
|
|
214
264
|
"CLIProxyAPI output_compression requires jpeg or webp output_format and an integer from 0 to 100.",
|
|
215
265
|
);
|
|
216
|
-
if (
|
|
266
|
+
if (
|
|
267
|
+
options.background === "transparent" &&
|
|
268
|
+
options.output_format !== undefined &&
|
|
269
|
+
!["png", "webp"].includes(options.output_format)
|
|
270
|
+
)
|
|
217
271
|
throw new Error("CLIProxyAPI transparent backgrounds require png or webp output_format.");
|
|
218
272
|
if (options.input_fidelity !== undefined && params.images === undefined)
|
|
219
273
|
throw new Error("CLIProxyAPI input_fidelity requires editing images.");
|
|
@@ -227,7 +281,8 @@ async function imageInputs(
|
|
|
227
281
|
signal: AbortSignal,
|
|
228
282
|
) {
|
|
229
283
|
if (params.images === undefined) {
|
|
230
|
-
if (params.mask !== undefined
|
|
284
|
+
if (params.mask !== undefined || params.markup !== undefined)
|
|
285
|
+
throw new Error("CLIProxyAPI mask and markup require editing images.");
|
|
231
286
|
return {};
|
|
232
287
|
}
|
|
233
288
|
const limits = mediaControls(model).editing;
|
|
@@ -238,14 +293,18 @@ async function imageInputs(
|
|
|
238
293
|
params.images.length > limits.max_images
|
|
239
294
|
)
|
|
240
295
|
throw new Error("CLIProxyAPI editing requires 1–16 local images and a supported OpenAI image model.");
|
|
296
|
+
if (params.markup !== undefined && (params.mask !== undefined || !limits.markup))
|
|
297
|
+
throw new Error(
|
|
298
|
+
"CLIProxyAPI markup requires GPT Image 2.5 or 2.6 Sunburst or Flare and cannot be combined with mask.",
|
|
299
|
+
);
|
|
300
|
+
// The mask or markup layer follows the input images.
|
|
301
|
+
const layer = params.mask ?? params.markup;
|
|
241
302
|
const images: { image_url: string }[] = [];
|
|
242
303
|
let mask: { image_url: string } | undefined;
|
|
304
|
+
let markup: { image_url: string } | undefined;
|
|
243
305
|
let firstPngDimensions: Buffer | undefined;
|
|
244
306
|
let total = 0;
|
|
245
|
-
for (const [index, path] of [
|
|
246
|
-
...params.images,
|
|
247
|
-
...(params.mask === undefined ? [] : [params.mask]),
|
|
248
|
-
].entries()) {
|
|
307
|
+
for (const [index, path] of [...params.images, ...(layer === undefined ? [] : [layer])].entries()) {
|
|
249
308
|
signal.throwIfAborted();
|
|
250
309
|
if (
|
|
251
310
|
typeof path !== "string" ||
|
|
@@ -286,22 +345,25 @@ async function imageInputs(
|
|
|
286
345
|
} catch {
|
|
287
346
|
throw new Error(`CLIProxyAPI editing input "${path}" is not a supported PNG, JPEG, or WebP image.`);
|
|
288
347
|
}
|
|
289
|
-
const
|
|
348
|
+
const isLayer = index === params.images.length;
|
|
290
349
|
if (index === 0 && mimeType === "image/png") firstPngDimensions = bytes.subarray(16, 24);
|
|
350
|
+
// Masks need a matching PNG first image; OpenAI checks markup on JPEG or WebP images after orientation.
|
|
351
|
+
const sameSize = firstPngDimensions?.equals(bytes.subarray(16, 24));
|
|
291
352
|
if (
|
|
292
|
-
|
|
353
|
+
isLayer &&
|
|
293
354
|
(mimeType !== "image/png" ||
|
|
294
|
-
!
|
|
295
|
-
|
|
355
|
+
![4, 6].includes(bytes[25]) ||
|
|
356
|
+
(sameSize === undefined ? params.mask !== undefined : !sameSize))
|
|
296
357
|
)
|
|
297
358
|
throw new Error(
|
|
298
|
-
"CLIProxyAPI mask and
|
|
359
|
+
"CLIProxyAPI mask and markup layers must be alpha-channel PNGs matching a PNG first image's size; masks need a PNG first image.",
|
|
299
360
|
);
|
|
300
361
|
const input = { image_url: `data:${mimeType};base64,${bytes.toString("base64")}` };
|
|
301
|
-
if (
|
|
302
|
-
else
|
|
362
|
+
if (!isLayer) images.push(input);
|
|
363
|
+
else if (params.mask !== undefined) mask = input;
|
|
364
|
+
else markup = input;
|
|
303
365
|
}
|
|
304
|
-
return { images, mask };
|
|
366
|
+
return { images, mask, markup };
|
|
305
367
|
}
|
|
306
368
|
|
|
307
369
|
export async function mediaKey(ctx: Pick<MediaContext, "modelRegistry">, signal: AbortSignal) {
|
|
@@ -604,7 +666,9 @@ async function generationKey(
|
|
|
604
666
|
const key = await mediaKey(ctx, signal);
|
|
605
667
|
const available = mapMediaCatalog(await fetchCatalog(config, key, signal));
|
|
606
668
|
if (!available.some(({ id }) => id === params.model)) {
|
|
607
|
-
throw new Error(
|
|
669
|
+
throw new Error(
|
|
670
|
+
"The selected media model is not available in the live CLIProxyAPI catalog. Update CLIProxyAPI, or choose an advertised model with /cli:model or an explicit model.",
|
|
671
|
+
);
|
|
608
672
|
}
|
|
609
673
|
return key;
|
|
610
674
|
}
|
|
@@ -615,7 +679,7 @@ function imageType(bytes: Buffer) {
|
|
|
615
679
|
if (
|
|
616
680
|
bytes.length > 24 &&
|
|
617
681
|
bytes.subarray(0, 8).equals(Buffer.from("89504e470d0a1a0a", "hex")) &&
|
|
618
|
-
bytes.toString("
|
|
682
|
+
bytes.toString("latin1", 12, 16) === "IHDR"
|
|
619
683
|
) {
|
|
620
684
|
extension = "png";
|
|
621
685
|
mimeType = "image/png";
|
|
@@ -631,8 +695,8 @@ function imageType(bytes: Buffer) {
|
|
|
631
695
|
mimeType = "image/jpeg";
|
|
632
696
|
} else if (
|
|
633
697
|
bytes.length > 20 &&
|
|
634
|
-
bytes.toString("
|
|
635
|
-
bytes.toString("
|
|
698
|
+
bytes.toString("latin1", 0, 4) === "RIFF" &&
|
|
699
|
+
bytes.toString("latin1", 8, 12) === "WEBP" &&
|
|
636
700
|
bytes.readUInt32LE(4) + 8 === bytes.length
|
|
637
701
|
) {
|
|
638
702
|
extension = "webp";
|
|
@@ -643,7 +707,71 @@ function imageType(bytes: Buffer) {
|
|
|
643
707
|
return { extension, mimeType };
|
|
644
708
|
}
|
|
645
709
|
|
|
646
|
-
|
|
710
|
+
// OpenAI documents static SVG without scripts, external resources, or raster images. Only static drawing
|
|
711
|
+
// elements with presentation attributes and same-document references pass; anything else fails closed.
|
|
712
|
+
const svgElements = new Set(
|
|
713
|
+
`svg g defs title desc symbol use path rect circle ellipse line polyline polygon text tspan textPath
|
|
714
|
+
linearGradient radialGradient stop pattern clipPath mask marker filter feBlend feColorMatrix feComponentTransfer
|
|
715
|
+
feComposite feConvolveMatrix feDiffuseLighting feDisplacementMap feDistantLight feDropShadow feFlood feFuncA
|
|
716
|
+
feFuncB feFuncG feFuncR feGaussianBlur feMerge feMergeNode feMorphology feOffset fePointLight
|
|
717
|
+
feSpecularLighting feSpotLight feTile feTurbulence`.split(/\s+/),
|
|
718
|
+
);
|
|
719
|
+
const xmlEntities: Record<string, string> = { amp: "&", lt: "<", gt: ">", quot: '"', apos: "'" };
|
|
720
|
+
// Checked after decoding references: event handlers and style attributes after any separator HTML accepts,
|
|
721
|
+
// CSS escapes and image functions, xml:base, foreign namespaces, and references outside the document. The scan
|
|
722
|
+
// covers text too, so a label that looks like an assignment fails closed. CSS whitespace is ASCII only.
|
|
723
|
+
const unsafeSvg =
|
|
724
|
+
/[^\w.:-](?:on[a-z]+|style)\s*=|\\|(?:image-set|image|src)\(|xml:base|xmlns(?::[\w.-]+)?\s*=\s*["'](?!http:\/\/www\.w3\.org\/(?:2000\/svg|1999\/xlink)["'])|(?:href|src)\s*=\s*(?:"(?!#)|'(?!#)|[^"'\s#>])|url\([ \t\n\r\f]*(?:"(?!#)|'(?!#)|[^"'#) \t\n\r\f])/i;
|
|
725
|
+
|
|
726
|
+
// Accept GIF or SVG only when the request asked GPT Image 2.6 for that format.
|
|
727
|
+
function outputType(bytes: Buffer, format?: string) {
|
|
728
|
+
if (format === "gif") {
|
|
729
|
+
if (bytes.length > 13 && /^GIF8[79]a$/.test(bytes.toString("latin1", 0, 6)) && bytes.at(-1) === 0x3b)
|
|
730
|
+
return { extension: "gif", mimeType: "image/gif" };
|
|
731
|
+
throw new Error("CLIProxyAPI returned a malformed GIF.");
|
|
732
|
+
}
|
|
733
|
+
if (format !== "svg") return imageType(bytes);
|
|
734
|
+
let text: string;
|
|
735
|
+
try {
|
|
736
|
+
text = new TextDecoder("utf-8", { fatal: true }).decode(bytes);
|
|
737
|
+
} catch {
|
|
738
|
+
throw new Error("CLIProxyAPI returned an SVG that is not valid UTF-8.");
|
|
739
|
+
}
|
|
740
|
+
// Only a UTF-8 XML declaration is removed: another encoding, such as ISO-2022-JP, makes renderers read
|
|
741
|
+
// different markup. Tags are checked before decoding because a decoded "<" in text is not markup.
|
|
742
|
+
const markup = text
|
|
743
|
+
.replace(
|
|
744
|
+
/^\s*<\?xml\s+version\s*=\s*(["'])1\.\d+\1(?:\s+encoding\s*=\s*(["'])utf-8\2)?(?:\s+standalone\s*=\s*(["'])(?:yes|no)\3)?\s*\?>/i,
|
|
745
|
+
"",
|
|
746
|
+
)
|
|
747
|
+
.trim();
|
|
748
|
+
// One decoding pass, so an escaped ampersand cannot start a second reference.
|
|
749
|
+
const decoded = markup.replace(
|
|
750
|
+
/&(?:#(x[\da-f]+|\d+)|(amp|lt|gt|quot|apos));?/gi,
|
|
751
|
+
(_, code?: string, name?: string) => {
|
|
752
|
+
if (code === undefined) return xmlEntities[name?.toLowerCase() ?? ""] ?? "\\";
|
|
753
|
+
const point = /^x/i.test(code) ? Number.parseInt(code.slice(1), 16) : Number(code);
|
|
754
|
+
return point <= 0x10ffff ? String.fromCodePoint(point) : "\\";
|
|
755
|
+
},
|
|
756
|
+
);
|
|
757
|
+
if (
|
|
758
|
+
!/^<svg[\s>]/.test(markup) ||
|
|
759
|
+
!markup.endsWith("</svg>") ||
|
|
760
|
+
// XML forbids these controls, and ESC can switch an encoding in the middle of an attribute name.
|
|
761
|
+
/(?![\t\n\r])\p{Cc}/u.test(text) ||
|
|
762
|
+
// Comments, CDATA, declarations, and processing instructions. HTML parses comments inside SVG
|
|
763
|
+
// integration points such as desc and title differently from XML, so none are accepted.
|
|
764
|
+
/<[!?]/.test(markup) ||
|
|
765
|
+
// XML defines only these named references; HTML would decode others such as ( into markup.
|
|
766
|
+
/&(?!(?:amp|lt|gt|quot|apos);|#)/.test(markup) ||
|
|
767
|
+
[...markup.matchAll(/<\/?([^\s/>]*)/g)].some(([, name]) => !svgElements.has(name)) ||
|
|
768
|
+
unsafeSvg.test(decoded)
|
|
769
|
+
)
|
|
770
|
+
throw new Error("CLIProxyAPI returned a malformed SVG or one with active or external content.");
|
|
771
|
+
return { extension: "svg", mimeType: "image/svg+xml" };
|
|
772
|
+
}
|
|
773
|
+
|
|
774
|
+
function parseImage(entry: unknown, format?: string) {
|
|
647
775
|
if (!isRecord(entry) || typeof entry.b64_json !== "string") {
|
|
648
776
|
throw new Error("CLIProxyAPI did not return b64_json image data. URL-only images are not downloaded.");
|
|
649
777
|
}
|
|
@@ -652,12 +780,12 @@ function parseImage(entry: unknown) {
|
|
|
652
780
|
if (!encoded || bytes.toString("base64") !== encoded) {
|
|
653
781
|
throw new Error("CLIProxyAPI returned invalid base64 image data.");
|
|
654
782
|
}
|
|
655
|
-
const { extension, mimeType } =
|
|
783
|
+
const { extension, mimeType } = outputType(bytes, format);
|
|
656
784
|
const image: ImageContent = { type: "image", data: encoded, mimeType };
|
|
657
785
|
return { bytes, extension, image };
|
|
658
786
|
}
|
|
659
787
|
|
|
660
|
-
function parseImages(value: unknown, count: number) {
|
|
788
|
+
function parseImages(value: unknown, count: number, format?: string) {
|
|
661
789
|
if (
|
|
662
790
|
!isRecord(value) ||
|
|
663
791
|
value.error !== undefined ||
|
|
@@ -665,7 +793,7 @@ function parseImages(value: unknown, count: number) {
|
|
|
665
793
|
value.data.length !== count
|
|
666
794
|
)
|
|
667
795
|
throw new Error(`CLIProxyAPI image response must contain ${count} image(s).`);
|
|
668
|
-
return value.data.map(parseImage);
|
|
796
|
+
return value.data.map((entry: unknown) => parseImage(entry, format));
|
|
669
797
|
}
|
|
670
798
|
|
|
671
799
|
function parseGoogleImages(value: unknown) {
|
|
@@ -728,7 +856,7 @@ function parseGoogleImages(value: unknown) {
|
|
|
728
856
|
}
|
|
729
857
|
|
|
730
858
|
// OpenAI reports image token usage. Costs use list prices and stay zero for unpriced models.
|
|
731
|
-
function imageUsage(model: string, value: unknown) {
|
|
859
|
+
function imageUsage(model: string, value: unknown, svg?: unknown) {
|
|
732
860
|
const tokens = (field: unknown) =>
|
|
733
861
|
typeof field === "number" && Number.isSafeInteger(field) && field >= 0 ? field : undefined;
|
|
734
862
|
if (!isRecord(value)) return undefined;
|
|
@@ -747,8 +875,22 @@ function imageUsage(model: string, value: unknown) {
|
|
|
747
875
|
cacheWrite: 0,
|
|
748
876
|
total: 0,
|
|
749
877
|
};
|
|
750
|
-
cost.total = cost.input + cost.output;
|
|
751
878
|
const usage: Usage = { input, output, cacheRead: 0, cacheWrite: 0, totalTokens: input + output, cost };
|
|
879
|
+
// SVG output adds the authoring model's tokens, priced from Pi's OpenAI metadata when Pi knows it.
|
|
880
|
+
if (isRecord(svg)) {
|
|
881
|
+
const author = svg.model;
|
|
882
|
+
const svgInput = tokens(svg.input_tokens);
|
|
883
|
+
const svgOutput = tokens(svg.output_tokens);
|
|
884
|
+
if (svgInput !== undefined && svgOutput !== undefined) {
|
|
885
|
+
const price = getBuiltinModels("openai").find(({ id }) => id === author)?.cost;
|
|
886
|
+
usage.input += svgInput;
|
|
887
|
+
usage.output += svgOutput;
|
|
888
|
+
usage.totalTokens += svgInput + svgOutput;
|
|
889
|
+
cost.input += (svgInput * (price?.input ?? 0)) / 1e6;
|
|
890
|
+
cost.output += (svgOutput * (price?.output ?? 0)) / 1e6;
|
|
891
|
+
}
|
|
892
|
+
}
|
|
893
|
+
cost.total = cost.input + cost.output;
|
|
752
894
|
return usage;
|
|
753
895
|
}
|
|
754
896
|
|
|
@@ -762,13 +904,14 @@ async function requestImages(
|
|
|
762
904
|
output: Partial<ReturnType<typeof mediaOutputOptions>>;
|
|
763
905
|
images?: { image_url: string }[];
|
|
764
906
|
mask?: { image_url: string };
|
|
907
|
+
markup?: { image_url: string };
|
|
765
908
|
},
|
|
766
909
|
signal: AbortSignal,
|
|
767
910
|
onPartial?: (image: ImageContent, index: number) => void,
|
|
768
911
|
{ onPayload, ...transport }: TransportOptions & { onPayload?: (payload: unknown) => unknown } = {},
|
|
769
912
|
) {
|
|
770
913
|
const route = mediaCapability(model)?.route;
|
|
771
|
-
const { prompt, images, mask } = request;
|
|
914
|
+
const { prompt, images, mask, markup } = request;
|
|
772
915
|
const { partial_images = 0, ...output } = request.output;
|
|
773
916
|
const n = output.n ?? 1;
|
|
774
917
|
const stream = route === "openai" && partial_images > 0;
|
|
@@ -783,7 +926,7 @@ async function requestImages(
|
|
|
783
926
|
aspect_ratio: output.aspect_ratio,
|
|
784
927
|
}
|
|
785
928
|
: route === "openai"
|
|
786
|
-
? { model, prompt, ...output, n, images, mask, ...(stream ? { stream, partial_images } : {}) }
|
|
929
|
+
? { model, prompt, ...output, n, images, mask, markup, ...(stream ? { stream, partial_images } : {}) }
|
|
787
930
|
: {
|
|
788
931
|
contents: [{ role: "user", parts: [{ text: prompt }] }],
|
|
789
932
|
generationConfig: {
|
|
@@ -808,9 +951,17 @@ async function requestImages(
|
|
|
808
951
|
body: payload,
|
|
809
952
|
openaiImage: route === "openai",
|
|
810
953
|
});
|
|
954
|
+
// Decode the format actually requested, which a native payload hook may have changed.
|
|
955
|
+
const format =
|
|
956
|
+
route === "openai" && isRecord(payload) && typeof payload.output_format === "string"
|
|
957
|
+
? payload.output_format
|
|
958
|
+
: undefined;
|
|
811
959
|
return {
|
|
812
|
-
images: route === "gemini" ? parseGoogleImages(result) : parseImages(result, n),
|
|
813
|
-
usage:
|
|
960
|
+
images: route === "gemini" ? parseGoogleImages(result) : parseImages(result, n, format),
|
|
961
|
+
usage:
|
|
962
|
+
route === "openai" && isRecord(result)
|
|
963
|
+
? imageUsage(model, result.usage, result.svg_usage)
|
|
964
|
+
: undefined,
|
|
814
965
|
};
|
|
815
966
|
}
|
|
816
967
|
// Each frame carries one full image, so the stream may hold the final image plus every preview.
|
|
@@ -840,7 +991,10 @@ async function requestImages(
|
|
|
840
991
|
(event) => isRecord(event) && typeof event.type === "string" && event.type.endsWith(".completed"),
|
|
841
992
|
);
|
|
842
993
|
if (completed.length !== n) throw new Error(`CLIProxyAPI image stream must complete ${n} image(s).`);
|
|
843
|
-
return {
|
|
994
|
+
return {
|
|
995
|
+
images: completed.map((event) => parseImage(event)),
|
|
996
|
+
usage: imageUsage(model, completed[0].usage),
|
|
997
|
+
};
|
|
844
998
|
}
|
|
845
999
|
|
|
846
1000
|
export async function generateNativeImages(
|
|
@@ -968,11 +1122,14 @@ export async function generateImage(
|
|
|
968
1122
|
let previewBudget = MAX_INLINE_IMAGE_BASE64_BYTES;
|
|
969
1123
|
const content: (TextContent | ImageContent)[] = [];
|
|
970
1124
|
for (const [index, { image }] of images.entries()) {
|
|
971
|
-
|
|
972
|
-
|
|
973
|
-
:
|
|
974
|
-
|
|
975
|
-
:
|
|
1125
|
+
// Chat providers disagree on animated GIF input and reject SVG, so both skip inline previews.
|
|
1126
|
+
const previewOmitted = ["image/gif", "image/svg+xml"].includes(image.mimeType)
|
|
1127
|
+
? "Preview omitted: GIF and SVG results are saved without an inline preview."
|
|
1128
|
+
: !ctx.model?.input.includes("image")
|
|
1129
|
+
? "Preview omitted: the current chat model does not support images."
|
|
1130
|
+
: image.data.length > previewBudget
|
|
1131
|
+
? "Preview omitted: image base64 exceeds the 4 MiB inline budget. Use read on the saved file."
|
|
1132
|
+
: undefined;
|
|
976
1133
|
content.push({
|
|
977
1134
|
type: "text",
|
|
978
1135
|
text: `Saved image: ${files[index].path}${previewOmitted ? `\n${previewOmitted}` : ""}`,
|
|
@@ -1077,7 +1234,7 @@ export function registerMediaTools(pi: ExtensionAPI, config: Config) {
|
|
|
1077
1234
|
name: "cliproxyapi_generate_image",
|
|
1078
1235
|
label: "CLIProxyAPI image",
|
|
1079
1236
|
description:
|
|
1080
|
-
"Generate images, or edit local images with OpenAI models by supplying images and an optional mask. Supports OpenAI size, quality, background, format/compression, image count and streamed partial previews
|
|
1237
|
+
"Generate images, or edit local images with OpenAI models by supplying images and an optional mask or markup layer. Supports OpenAI size, quality, complexity, background, format/compression, GIF/SVG output, image count and streamed partial previews; check cliproxyapi_media_models for each model's controls. Uses the selected model or automatic OpenAI/GPT default without changing chat. Saves final images under a new .pi/ directory with optional previews. Maximum response: 32 MiB in total, plus 32 MiB per streamed preview; inline base64: 4 MiB. No automatic retries.",
|
|
1081
1238
|
promptSnippet: "Generate requested raster images through CLIProxyAPI using the effective image default.",
|
|
1082
1239
|
promptGuidelines: [
|
|
1083
1240
|
"Use cliproxyapi_generate_image for requested raster images. Omit model to use the effective default unless a specific image model is requested; keep the current chat model.",
|