pi-cliproxyapi-native 0.3.4 → 0.3.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +40 -31
- package/package.json +2 -2
- package/src/catalog.ts +29 -15
- package/src/media-defaults.ts +1 -1
- package/src/media.ts +198 -40
package/README.md
CHANGED
|
@@ -31,15 +31,15 @@ Chat uses Pi's native API adapters and model metadata, with availability from yo
|
|
|
31
31
|
|
|
32
32
|
## Install
|
|
33
33
|
|
|
34
|
-
The extension targets Pi 1.5.0 and CLIProxyAPI 8.0.
|
|
34
|
+
The extension targets Pi 1.5.0 and CLIProxyAPI 8.0.57, with Node.js 22.19 or newer. Run CLIProxyAPI with your upstream accounts configured before connecting Pi. The extension uses Pi's host-provided packages, adds no runtime dependencies, and needs no compilation.
|
|
35
35
|
|
|
36
36
|
If you installed `pi-cliproxy-native`, remove it before installing the renamed package. Both register the same `cliproxyapi` provider.
|
|
37
37
|
|
|
38
38
|
```sh
|
|
39
|
-
pi install npm:pi-cliproxyapi-native@0.3.
|
|
39
|
+
pi install npm:pi-cliproxyapi-native@0.3.5
|
|
40
40
|
```
|
|
41
41
|
|
|
42
|
-
For Git installation, use `pi install git:github.com/hawkff/pi-cliproxyapi-native@v0.3.
|
|
42
|
+
For Git installation, use `pi install git:github.com/hawkff/pi-cliproxyapi-native@v0.3.5`. For a local checkout, use `pi install /path/to/pi-cliproxyapi-native`.
|
|
43
43
|
|
|
44
44
|
The default proxy address is `http://localhost:8317`. For another address, set the [connection](#connection) before logging in. Restart Pi, then run:
|
|
45
45
|
|
|
@@ -117,7 +117,7 @@ Check video status for request_id <returned-id>.
|
|
|
117
117
|
| `cliproxyapi_generate_video` | `prompt`; optional `model`, `resolution`, `aspect_ratio`, and integer `duration` from 1 to 15 seconds | A `request_id` for the submitted video. |
|
|
118
118
|
| `cliproxyapi_video_status` | `request_id` | One status check: pending, completed with a URL, or failed. |
|
|
119
119
|
|
|
120
|
-
Pass `model` to choose an exact ID for one tool call. Otherwise, the tool uses its saved `/cli:model` selection. Without either, eligible OpenAI/GPT chats default to `gpt-image-2.
|
|
120
|
+
Pass `model` to choose an exact ID for one tool call. Otherwise, the tool uses its saved `/cli:model` selection. Without either, eligible OpenAI/GPT chats default to `gpt-image-2.6-sunburst` for images, which needs CLIProxyAPI 8.0.56 or newer. Other chats need an image selection or explicit ID. Videos need a selection or explicit ID.
|
|
121
121
|
|
|
122
122
|
The automatic image default applies to native OpenAI/Codex chats, recognized GPT chat IDs through other providers, and configured OpenAI metadata aliases. An OpenAI-compatible API alone does not qualify.
|
|
123
123
|
|
|
@@ -136,14 +136,14 @@ Pass output controls for each generation, or omit them to keep the backend defau
|
|
|
136
136
|
| xAI images | `resolution`: `1k` or `2k`, plus `aspect_ratio`. |
|
|
137
137
|
| xAI videos | `resolution`: `480p` or `720p`, plus `aspect_ratio`. The 1.5 model, its preview alias, and 1.5 Lite also accept `1080p`. |
|
|
138
138
|
|
|
139
|
-
GPT Image 1.5 accepts `auto`, `1024x1024`, `1536x1024`, and `1024x1536`. GPT Image 2 and 2.
|
|
139
|
+
GPT Image 1.5 accepts `auto`, `1024x1024`, `1536x1024`, and `1024x1536`. GPT Image 2, 2.5, and 2.6 also accept custom dimensions: both edges must be multiples of 16 and at most 3840 pixels. The longer edge cannot exceed three times the shorter one. Total pixels must be between 655,360 and 8,294,400. OpenAI marks resolutions above `2560x1440` as experimental. For GIF output, `size` sets each frame and can go below the 655,360-pixel minimum. OpenAI checks the frame size, frame count, and image count against its request budget.
|
|
140
140
|
|
|
141
141
|
Gemini 3 Pro Image supports `1K`, `2K`, and `4K`. Gemini 3.1 Flash Image also supports `512`. Flash Lite Image supports `1K` only. Gemini 2.5 Flash Image has no resolution selector. Use the exact case shown in discovery, including uppercase `K` for Gemini and lowercase `k` for xAI.
|
|
142
142
|
|
|
143
143
|
Aspect ratios vary by model and proxy route. For example, CLIProxyAPI 8.0.7 accepts `20:9` for xAI images but drops xAI's `21:9` and `auto` values. The extension rejects unsupported controls before authentication or network access instead of substituting another size or shape.
|
|
144
144
|
|
|
145
145
|
```text
|
|
146
|
-
Generate an image with gpt-image-2.
|
|
146
|
+
Generate an image with gpt-image-2.6-sunburst at size 2048x2048.
|
|
147
147
|
Generate a 2K image at 16:9 with gemini-3-pro-image.
|
|
148
148
|
Generate a 1-second 1080p video at 9:16 with grok-imagine-video-1.5.
|
|
149
149
|
```
|
|
@@ -156,41 +156,50 @@ Control values follow the [OpenAI image guide](https://developers.openai.com/api
|
|
|
156
156
|
|
|
157
157
|
Use `cliproxyapi_media_models` to inspect controls for the exact advertised model ID. Omitted options keep the backend defaults. The extension rejects unsupported values before contacting the proxy.
|
|
158
158
|
|
|
159
|
-
| Option | Supported values
|
|
160
|
-
| -------------------- |
|
|
161
|
-
| `quality` | `auto`, `low`, `medium`, `high`. Sunburst and Flare also accept `xhigh` and `max`.
|
|
162
|
-
| `
|
|
163
|
-
| `
|
|
164
|
-
| `
|
|
165
|
-
| `
|
|
166
|
-
| `
|
|
167
|
-
| `
|
|
168
|
-
| `
|
|
169
|
-
| `
|
|
170
|
-
| `
|
|
171
|
-
|
|
172
|
-
|
|
159
|
+
| Option | Supported values |
|
|
160
|
+
| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
161
|
+
| `quality` | `auto`, `low`, `medium`, `high`. Sunburst and Flare also accept `xhigh` and `max`. |
|
|
162
|
+
| `complexity` | `low`, `medium`, `high` for GPT Image 2.6 Sunburst and Flare. It sets how much the model plans before rendering, and OpenAI defaults to `high`. Use `low` for quick edits. |
|
|
163
|
+
| `background` | `auto`, `opaque`, `transparent`. Transparent output requires PNG or WebP; GPT Image 2 transparency is in preview upstream. |
|
|
164
|
+
| `output_format` | `png`, `jpeg`, `webp`. GPT Image 2.6 Sunburst and Flare also generate `gif` and `svg`. JPEG and WebP can reduce file size and help keep results within the response limit. |
|
|
165
|
+
| `output_compression` | Integer from 0 to 100, with an explicit `jpeg` or `webp` format. Zero is a valid value. |
|
|
166
|
+
| `frame_count` | GIF output only. Integer from 2 to 16. OpenAI defaults to 9. |
|
|
167
|
+
| `frame_duration_ms` | GIF output only. Milliseconds per frame from 20 to 2000, rounded to 10. OpenAI defaults to 200. |
|
|
168
|
+
| `moderation` | `auto` or `low`. OpenAI still applies its content policies. |
|
|
169
|
+
| `n` | Integer from 1 to 10 on an explicit `api/` image route. Other routes allow only 1 because subscription translation can drop the count. |
|
|
170
|
+
| `partial_images` | Integer from 0 to 3. The proxy streams up to that many draft frames while the image renders, and Pi shows them as tool progress. Requires `n` of 1 and PNG, JPEG, or WebP output. Each draft adds about 100 image output tokens. |
|
|
171
|
+
| `images` | One to 16 local PNG, JPEG, or WebP paths. Supplying these selects `/v1/images/edits` instead of generation. |
|
|
172
|
+
| `mask` | Optional local PNG mask. The first image must also be PNG with matching dimensions. The mask needs a grayscale-alpha or RGBA alpha channel; transparent regions mark edits. |
|
|
173
|
+
| `markup` | Optional local PNG annotation layer for GPT Image 2.5 and 2.6 Sunburst and Flare. It needs an alpha channel, the first image's dimensions, and transparency outside the marks. It cannot be combined with `mask`. |
|
|
174
|
+
| `input_fidelity` | `low` or `high`, for GPT Image 1.5 editing only. Omit it for other models. |
|
|
175
|
+
|
|
176
|
+
Editing uploads the files you name to the configured proxy. Paths can be absolute or relative to the working directory. The extension accepts regular files, checks file signatures, and limits uploads to 14 MiB per file and 20 MiB total including the mask or markup layer. It compares markup dimensions only with a PNG first image; OpenAI checks JPEG and WebP inputs. It does not fetch input URLs, resize inputs, or modify source files. OpenAI performs full image decoding and validates the edit. These local limits keep base64 uploads bounded and may be lower than upstream limits.
|
|
173
177
|
|
|
174
178
|
```text
|
|
175
|
-
Generate two images with api/gpt-image-2.
|
|
176
|
-
Edit ./photo.png with api/gpt-image-2.
|
|
179
|
+
Generate two images with api/gpt-image-2.6-sunburst at 1536x864, quality max, output_format webp, output_compression 85.
|
|
180
|
+
Edit ./photo.png with api/gpt-image-2.6-flare: replace the background with a garden, using ./mask.png as the mask.
|
|
181
|
+
Edit ./kitchen.webp with gpt-image-2.6-flare and complexity low, using ./circle.png as markup: recolor the circled vase cobalt blue.
|
|
182
|
+
Generate a looping GIF of a waving robot with gpt-image-2.6-flare, output_format gif, frame_count 8, size 512x512.
|
|
183
|
+
Generate an SVG logo for a bakery with gpt-image-2.6-sunburst and output_format svg.
|
|
177
184
|
```
|
|
178
185
|
|
|
186
|
+
GIF and SVG output work only for generation, without streaming, `partial_images`, or `output_compression`. SVG also needs `n` of 1. OpenAI renders a reference image at the chosen `quality`, and a second model writes the SVG from it. The extension saves `.gif` and `.svg` files without inline previews and checks the GIF signature. SVG must be UTF-8 static drawing markup with presentation attributes and references only inside the file, which matches OpenAI's documented output. The extension rejects anything else before saving, including scripts, CSS, comments, animation, event handlers, and external references.
|
|
187
|
+
|
|
179
188
|
All returned images share the existing 32 MiB response limit and 4 MiB preview budget. Streamed requests allow another 32 MiB for each draft frame. The extension saves only the final image and drops drafts it cannot decode. Larger sizes, higher quality, and multiple images can increase cost and latency. Use fewer images or JPEG/WebP if the response exceeds the limit. Generation and editing do not retry after a failure.
|
|
180
189
|
|
|
181
|
-
The extension supports the verified options above, not a general OpenAI API passthrough. Remote file IDs,
|
|
190
|
+
The extension supports the verified options above, not a general OpenAI API passthrough. Remote file IDs, SVG author model selection, audio, batches, and realtime APIs are not exposed. Adding an upstream API key does not enable endpoints that the proxy or extension cannot handle.
|
|
182
191
|
|
|
183
192
|
### Image usage and cost
|
|
184
193
|
|
|
185
194
|
OpenAI image responses report token usage. The image tool returns it to Pi, so image calls count toward the session totals in the footer and `/session`.
|
|
186
195
|
|
|
187
|
-
Costs are estimates at OpenAI list prices per million tokens: $5 for text input, $8 for image input, and $30 for image output on GPT Image 2 and 2.5 Sunburst and Flare. GPT Image 1.5 output costs $32. The plain `gpt-image-2.5` subscription
|
|
196
|
+
Costs are estimates at OpenAI list prices per million tokens: $5 for text input, $8 for image input, and $30 for image output on GPT Image 2, and on 2.5 and 2.6 Sunburst and Flare. GPT Image 1.5 output costs $32. The plain `gpt-image-2.5` and `gpt-image-2.6` subscription models have no published price, so they report tokens at zero cost. SVG output also reports the tokens of the model that writes the SVG. The extension adds them to the totals and prices them from Pi's OpenAI model metadata, or at zero when Pi doesn't know that model. Subscription routes show the same estimates when the proxy reports usage, even though the plan covers them. Google and xAI image results report no usage.
|
|
188
197
|
|
|
189
198
|
### Pi image models
|
|
190
199
|
|
|
191
200
|
When Pi refreshes the catalog, at interactive startup or through `/cliproxyapi-refresh`, the advertised image models also register as Pi image models. Codemode scripts list them with `models.getAvailableOfType("image", "cliproxyapi")` and call `models.generateImages()`, and other extensions can use `ctx.modelRegistry.generateImages()`.
|
|
192
201
|
|
|
193
|
-
These calls send the prompt with backend defaults. Pi's image interface has no size, quality, count, mask, or preview options. Image blocks in the input select OpenAI editing, under the same 16-image, 14 MiB per image, and 20 MiB total limits. Google and xAI models accept text only. Results arrive as image blocks with usage and are not saved to disk. Use `cliproxyapi_generate_image` when you need controls, masks, or saved files.
|
|
202
|
+
These calls send the prompt with backend defaults. Pi's image interface has no size, quality, count, format, mask, markup, or preview options. Image blocks in the input select OpenAI editing, under the same 16-image, 14 MiB per image, and 20 MiB total limits. Google and xAI models accept text only. Results arrive as image blocks with usage and are not saved to disk. Use `cliproxyapi_generate_image` when you need controls, masks, markup, GIF or SVG output, or saved files.
|
|
194
203
|
|
|
195
204
|
### Supported media models
|
|
196
205
|
|
|
@@ -198,11 +207,11 @@ Your proxy must advertise the exact ID for a model to be available.
|
|
|
198
207
|
|
|
199
208
|
| Provider | Models |
|
|
200
209
|
| ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
201
|
-
| OpenAI models | `gpt-image-2.5-flare`, `gpt-image-2.5-sunburst`, `gpt-image-2.5`, `gpt-image-2`, `gpt-image-1.5`
|
|
210
|
+
| OpenAI models | `gpt-image-2.6-flare`, `gpt-image-2.6-sunburst`, `gpt-image-2.6`, `gpt-image-2.5-flare`, `gpt-image-2.5-sunburst`, `gpt-image-2.5`, `gpt-image-2`, `gpt-image-1.5` |
|
|
202
211
|
| Google models | `gemini-2.5-flash-image`, `gemini-3.1-flash-image`, `gemini-3-pro-image`, `gemini-3.1-flash-lite-image` |
|
|
203
212
|
| xAI models | `grok-imagine-image`, `grok-imagine-image-quality`, `grok-imagine-image-2.0`, `grok-imagine-video`, `grok-imagine-video-1.5`, `grok-imagine-video-1.5-lite`, `grok-imagine-video-1.5-preview` |
|
|
204
213
|
|
|
205
|
-
Supported media IDs can also use advertised `api/` and `oauth/` credential routes. OpenAI's API has no plain `gpt-image-2.5`, so the picker disables `api
|
|
214
|
+
Supported media IDs can also use advertised `api/` and `oauth/` credential routes. OpenAI's API has no plain `gpt-image-2.5` or `gpt-image-2.6`, so the picker disables their `api/` routes and points to Sunburst or Flare; subscription routes keep them. Gemini image models also support `vertex/` and `antigravity/` backend routes. Those backend prefixes do not enable OpenAI or xAI media execution; the picker shows those entries as disabled. Other prefixes, nested prefixes, media aliases, unknown media IDs, Google Veo, and non-OpenAI editing are unsupported.
|
|
206
215
|
|
|
207
216
|
The extension disables these retired Imagen IDs, including their supported prefixed forms:
|
|
208
217
|
|
|
@@ -216,9 +225,9 @@ The picker explains their retirement, and generation rejects them before network
|
|
|
216
225
|
|
|
217
226
|
### Files, status, and limits
|
|
218
227
|
|
|
219
|
-
Image generation saves JPEG, PNG, or
|
|
228
|
+
Image generation saves JPEG, PNG, WebP, GIF, or SVG files under a new `.pi/cliproxyapi-image-*/` directory in your working directory. The extension checks base64 and file signatures, uses private permissions, and does not overwrite existing files. Keep `.pi/` out of version control.
|
|
220
229
|
|
|
221
|
-
Inline previews require a vision-capable chat model and share a 4 MiB base64 budget per result. Larger images return file paths without previews. Use Pi's `read` tool to inspect saved images. URL-only image responses fail without a download.
|
|
230
|
+
Inline previews require a vision-capable chat model and share a 4 MiB base64 budget per result. Larger images return file paths without previews, and GIF and SVG results never include one. Use Pi's `read` tool to inspect saved images. URL-only image responses fail without a download.
|
|
222
231
|
|
|
223
232
|
Gemini responses must contain one completed candidate. The extension skips thought parts and saves up to 16 final images within a 256-part response limit. Safety/refusal signals, incomplete or text-only results, malformed data, MIME/signature mismatches, and excess images or parts fail before saving.
|
|
224
233
|
|
|
@@ -384,11 +393,11 @@ On `api/` routes, Pi can send thinking effort as a `configuration_update` item.
|
|
|
384
393
|
|
|
385
394
|
When Pi omits strictness in the top-level Responses tool list, the extension sends `strict: null`, so optional arguments remain optional. Tools declared inside the conversation keep Pi's strictness values. Explicit payload hooks retain final control.
|
|
386
395
|
|
|
387
|
-
OpenAI images POST `model`, `prompt`, `n`, and supplied controls to `/v1/images/generations`, without `response_format`. Editing uses `/v1/images/edits` with JSON `images` and optional `mask` data URLs. A nonzero `partial_images` adds `stream: true`, and the extension reads server-sent events. Each `*.partial_image` frame becomes a progress preview, and the `*.completed` frame carries the final image and usage. xAI uses the same route with `response_format: "b64_json"`. Gemini uses `/v1beta/models/{id}:generateContent`. Media generation sends the prompt and generation options without chat history, system instructions, or tools.
|
|
396
|
+
OpenAI images POST `model`, `prompt`, `n`, and supplied controls to `/v1/images/generations`, without `response_format`. Editing uses `/v1/images/edits` with JSON `images` and optional `mask` or `markup` data URLs. A nonzero `partial_images` adds `stream: true`, and the extension reads server-sent events. Each `*.partial_image` frame becomes a progress preview, and the `*.completed` frame carries the final image and usage. xAI uses the same route with `response_format: "b64_json"`. Gemini uses `/v1beta/models/{id}:generateContent`. Media generation sends the prompt and generation options without chat history, system instructions, or tools.
|
|
388
397
|
|
|
389
398
|
OpenAI image POSTs use a dedicated HTTP connection. It follows `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` from the environment, where Pi also places its `httpProxy` setting, on Node 22.21+ and 24.5+. Older Node versions connect directly. The connection disables socket timeouts so the ten-minute deadline controls the request, requests identity encoding, and rejects compressed responses.
|
|
390
399
|
|
|
391
|
-
The OpenAI image implementation follows CLIProxyAPI v8.0.
|
|
400
|
+
The OpenAI image implementation follows CLIProxyAPI v8.0.57's [image handler](https://github.com/router-for-me/CLIProxyAPI/blob/v8.0.57/sdk/api/handlers/openai/openai_images_handlers.go) and [Codex image executor](https://github.com/router-for-me/CLIProxyAPI/blob/v8.0.57/internal/runtime/executor/codex_openai_images.go), plus the OpenAI [generation](https://developers.openai.com/api/reference/resources/images/methods/generate) and [editing](https://developers.openai.com/api/reference/resources/images/methods/edit) references. The xAI and Google routes follow v7.2.158's [video handler](https://github.com/router-for-me/CLIProxyAPI/blob/v7.2.158/sdk/api/handlers/openai/openai_videos_handlers.go), [Gemini handler](https://github.com/router-for-me/CLIProxyAPI/blob/v7.2.158/sdk/api/handlers/gemini/gemini_handlers.go), and [Vertex executor](https://github.com/router-for-me/CLIProxyAPI/blob/v7.2.158/internal/runtime/executor/gemini_vertex_executor.go). See [xAI's video documentation](https://docs.x.ai/developers/model-capabilities/video/generation) for duration and status semantics.
|
|
392
401
|
|
|
393
402
|
## License
|
|
394
403
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-cliproxyapi-native",
|
|
3
|
-
"version": "0.3.
|
|
3
|
+
"version": "0.3.5",
|
|
4
4
|
"description": "CLIProxyAPI chat discovery, image generation and editing, and video generation with Pi's native adapters",
|
|
5
5
|
"publishConfig": {
|
|
6
6
|
"access": "public"
|
|
@@ -38,7 +38,7 @@
|
|
|
38
38
|
"check": "pnpm typecheck && pnpm lint && pnpm test"
|
|
39
39
|
},
|
|
40
40
|
"devDependencies": {
|
|
41
|
-
"@biomejs/biome": "^2.6.
|
|
41
|
+
"@biomejs/biome": "^2.6.5",
|
|
42
42
|
"@earendil-works/pi-ai": "^1.5.0",
|
|
43
43
|
"@earendil-works/pi-coding-agent": "^1.5.0",
|
|
44
44
|
"@earendil-works/pi-tui": "^1.5.0",
|
package/src/catalog.ts
CHANGED
|
@@ -118,6 +118,15 @@ const mediaModels = new Map<
|
|
|
118
118
|
price?: readonly [number, number, number];
|
|
119
119
|
}
|
|
120
120
|
>([
|
|
121
|
+
[
|
|
122
|
+
"gpt-image-2.6-flare",
|
|
123
|
+
{ name: "GPT Image 2.6 Flare", purpose: "image", route: "openai", price: gptImage2Price },
|
|
124
|
+
],
|
|
125
|
+
[
|
|
126
|
+
"gpt-image-2.6-sunburst",
|
|
127
|
+
{ name: "GPT Image 2.6 Sunburst", purpose: "image", route: "openai", price: gptImage2Price },
|
|
128
|
+
],
|
|
129
|
+
["gpt-image-2.6", { name: "GPT Image 2.6", purpose: "image", route: "openai" }],
|
|
121
130
|
[
|
|
122
131
|
"gpt-image-2.5-flare",
|
|
123
132
|
{ name: "GPT Image 2.5 Flare", purpose: "image", route: "openai", price: gptImage2Price },
|
|
@@ -165,13 +174,12 @@ export function mediaCapability(id: string) {
|
|
|
165
174
|
if (backend && (capability?.route === "xai" || capability?.route === "openai")) {
|
|
166
175
|
return { ...capability, route: undefined, disabledReason: `Unsupported media execution on ${backend}.` };
|
|
167
176
|
}
|
|
168
|
-
// CLIProxyAPI
|
|
169
|
-
if (capability && authType === "api-key" &&
|
|
177
|
+
// CLIProxyAPI advertises the plain names for API keys, but OpenAI's Images API has only Sunburst and Flare.
|
|
178
|
+
if (capability && authType === "api-key" && /^gpt-image-2\.[56]$/.test(metadataId)) {
|
|
170
179
|
return {
|
|
171
180
|
...capability,
|
|
172
181
|
route: undefined,
|
|
173
|
-
disabledReason:
|
|
174
|
-
"OpenAI's API has no gpt-image-2.5 model. Use api/gpt-image-2.5-sunburst or api/gpt-image-2.5-flare.",
|
|
182
|
+
disabledReason: `OpenAI's API has no ${metadataId} model. Use api/${metadataId}-sunburst or api/${metadataId}-flare.`,
|
|
175
183
|
};
|
|
176
184
|
}
|
|
177
185
|
return capability;
|
|
@@ -185,6 +193,9 @@ export function mediaControls(id: string) {
|
|
|
185
193
|
const capability = mediaCapability(id);
|
|
186
194
|
const { metadataId, authType } = modelRoute(id);
|
|
187
195
|
const openai = capability?.route === "openai";
|
|
196
|
+
// Sunburst and Flare add quality tiers and markup edits; 2.6 adds complexity plus GIF and SVG output.
|
|
197
|
+
const tiered = openai && /^gpt-image-2\.[56]-(sunburst|flare)$/.test(metadataId);
|
|
198
|
+
const image26 = tiered && metadataId.startsWith("gpt-image-2.6-");
|
|
188
199
|
const commonRatios = ["1:1", "16:9", "9:16", "4:3", "3:4", "3:2", "2:3"];
|
|
189
200
|
let resolution: string[] = [];
|
|
190
201
|
let aspect_ratio: string[] = [];
|
|
@@ -214,24 +225,27 @@ export function mediaControls(id: string) {
|
|
|
214
225
|
: undefined,
|
|
215
226
|
resolution,
|
|
216
227
|
aspect_ratio,
|
|
217
|
-
quality: openai
|
|
218
|
-
|
|
219
|
-
"auto",
|
|
220
|
-
"low",
|
|
221
|
-
"medium",
|
|
222
|
-
"high",
|
|
223
|
-
...(/^gpt-image-2\.5-(sunburst|flare)$/.test(metadataId) ? ["xhigh", "max"] : []),
|
|
224
|
-
]
|
|
225
|
-
: [],
|
|
228
|
+
quality: openai ? ["auto", "low", "medium", "high", ...(tiered ? ["xhigh", "max"] : [])] : [],
|
|
229
|
+
complexity: image26 ? ["low", "medium", "high"] : [],
|
|
226
230
|
background: openai ? ["auto", "opaque", "transparent"] : [],
|
|
227
|
-
output_format: openai ? ["png", "jpeg", "webp"] : [],
|
|
231
|
+
output_format: openai ? ["png", "jpeg", "webp", ...(image26 ? ["gif", "svg"] : [])] : [],
|
|
232
|
+
// GIF sizes are per frame and skip min_pixels; OpenAI checks the animation against its pixel budget.
|
|
233
|
+
gif: image26
|
|
234
|
+
? { frame_count: { min: 2, max: 16 }, frame_duration_ms: { min: 20, max: 2000 } }
|
|
235
|
+
: undefined,
|
|
228
236
|
output_compression: openai ? { min: 0, max: 100, formats: ["jpeg", "webp"] } : undefined,
|
|
229
237
|
moderation: openai ? ["auto", "low"] : [],
|
|
230
238
|
// Subscription image-tool translation does not forward n.
|
|
231
239
|
n: openai ? { min: 1, max: authType === "api-key" ? 10 : 1 } : undefined,
|
|
232
240
|
partial_images: openai ? { min: 0, max: 3 } : undefined,
|
|
233
241
|
editing: openai
|
|
234
|
-
? {
|
|
242
|
+
? {
|
|
243
|
+
max_images: 16,
|
|
244
|
+
max_file_bytes: 14 * 1024 * 1024,
|
|
245
|
+
max_total_bytes: 20 * 1024 * 1024,
|
|
246
|
+
mask: "png",
|
|
247
|
+
...(tiered ? { markup: "png" } : {}),
|
|
248
|
+
}
|
|
235
249
|
: undefined,
|
|
236
250
|
input_fidelity: openai && metadataId === "gpt-image-1.5" ? ["low", "high"] : [],
|
|
237
251
|
};
|
package/src/media-defaults.ts
CHANGED
|
@@ -3,7 +3,7 @@ import { type MediaPurpose, mediaCapability, mediaPurpose, modelRoute } from "./
|
|
|
3
3
|
import { type Config, isModelId, isRecord, PROVIDER_ID } from "./config.ts";
|
|
4
4
|
|
|
5
5
|
export const MEDIA_DEFAULTS_ENTRY = "cliproxyapi-media-defaults";
|
|
6
|
-
export const OPENAI_IMAGE_DEFAULT = "gpt-image-2.
|
|
6
|
+
export const OPENAI_IMAGE_DEFAULT = "gpt-image-2.6-sunburst";
|
|
7
7
|
// null blocks fallback for invalid selections, including after unrelated picker saves.
|
|
8
8
|
export type MediaDefaults = Partial<Record<MediaPurpose, string | null>>;
|
|
9
9
|
export type DefaultsContext = {
|
package/src/media.ts
CHANGED
|
@@ -14,6 +14,7 @@ import type {
|
|
|
14
14
|
TextContent,
|
|
15
15
|
Usage,
|
|
16
16
|
} from "@earendil-works/pi-ai";
|
|
17
|
+
import { getBuiltinModels } from "@earendil-works/pi-ai/providers/all";
|
|
17
18
|
import type { ExtensionAPI, ExtensionContext } from "@earendil-works/pi-coding-agent";
|
|
18
19
|
import { type Static, Type } from "typebox";
|
|
19
20
|
import { fetchCatalog, mapMediaCatalog, mediaCapability, mediaControls } from "./catalog.ts";
|
|
@@ -65,13 +66,22 @@ const imageParameters = Type.Object({
|
|
|
65
66
|
description: "OpenAI quality from controls.quality; Sunburst and Flare also support xhigh and max.",
|
|
66
67
|
}),
|
|
67
68
|
),
|
|
69
|
+
complexity: Type.Optional(
|
|
70
|
+
Type.String({
|
|
71
|
+
description:
|
|
72
|
+
"GPT Image 2.6 Sunburst/Flare planning effort: low, medium, or high (upstream default high). Low suits quick edits.",
|
|
73
|
+
}),
|
|
74
|
+
),
|
|
68
75
|
background: Type.Optional(
|
|
69
76
|
Type.String({
|
|
70
77
|
description: "OpenAI background: auto, opaque, or transparent. Transparency requires PNG or WebP.",
|
|
71
78
|
}),
|
|
72
79
|
),
|
|
73
80
|
output_format: Type.Optional(
|
|
74
|
-
Type.String({
|
|
81
|
+
Type.String({
|
|
82
|
+
description:
|
|
83
|
+
"OpenAI output format: png, jpeg, or webp. GPT Image 2.6 Sunburst/Flare also generate gif and svg (svg needs n of 1). Omit for the backend default.",
|
|
84
|
+
}),
|
|
75
85
|
),
|
|
76
86
|
output_compression: Type.Optional(
|
|
77
87
|
Type.Integer({
|
|
@@ -80,6 +90,20 @@ const imageParameters = Type.Object({
|
|
|
80
90
|
description: "OpenAI JPEG/WebP compression, 0–100. Requires an explicit jpeg or webp output_format.",
|
|
81
91
|
}),
|
|
82
92
|
),
|
|
93
|
+
frame_count: Type.Optional(
|
|
94
|
+
Type.Integer({
|
|
95
|
+
minimum: 2,
|
|
96
|
+
maximum: 16,
|
|
97
|
+
description: "GIF output only: number of frames, 2–16 (upstream default 9).",
|
|
98
|
+
}),
|
|
99
|
+
),
|
|
100
|
+
frame_duration_ms: Type.Optional(
|
|
101
|
+
Type.Integer({
|
|
102
|
+
minimum: 20,
|
|
103
|
+
maximum: 2000,
|
|
104
|
+
description: "GIF output only: milliseconds per frame, 20–2000, rounded to 10 (upstream default 200).",
|
|
105
|
+
}),
|
|
106
|
+
),
|
|
83
107
|
moderation: Type.Optional(Type.String({ description: "OpenAI moderation: auto or low." })),
|
|
84
108
|
n: Type.Optional(
|
|
85
109
|
Type.Integer({
|
|
@@ -112,6 +136,13 @@ const imageParameters = Type.Object({
|
|
|
112
136
|
"Optional local PNG mask for images[0]. The first image must also be PNG with matching dimensions; mask must have an alpha channel. Transparent regions are edited.",
|
|
113
137
|
}),
|
|
114
138
|
),
|
|
139
|
+
markup: Type.Optional(
|
|
140
|
+
Type.String({
|
|
141
|
+
minLength: 1,
|
|
142
|
+
description:
|
|
143
|
+
"Optional local PNG annotation layer for images[0] with GPT Image 2.5/2.6 Sunburst or Flare: transparent except for marks showing where to edit. Same size as images[0]; cannot be combined with mask.",
|
|
144
|
+
}),
|
|
145
|
+
),
|
|
115
146
|
input_fidelity: Type.Optional(
|
|
116
147
|
Type.String({
|
|
117
148
|
description: "Editing fidelity for GPT Image 1.5 only: low or high. Omit for other models.",
|
|
@@ -122,7 +153,7 @@ const imageParameters = Type.Object({
|
|
|
122
153
|
minLength: 1,
|
|
123
154
|
maxLength: 32,
|
|
124
155
|
description:
|
|
125
|
-
"OpenAI image size: auto or WIDTHxHEIGHT. Use controls.size and controls.size_limits from cliproxyapi_media_models. Omit to preserve the backend default.",
|
|
156
|
+
"OpenAI image size: auto or WIDTHxHEIGHT. Use controls.size and controls.size_limits from cliproxyapi_media_models; GIF sizes are per frame and may go below min_pixels. Omit to preserve the backend default.",
|
|
126
157
|
}),
|
|
127
158
|
),
|
|
128
159
|
});
|
|
@@ -141,6 +172,7 @@ function mediaOutputOptions(model: string, params: Partial<Static<typeof imagePa
|
|
|
141
172
|
resolution: params.resolution,
|
|
142
173
|
aspect_ratio: params.aspect_ratio,
|
|
143
174
|
quality: params.quality,
|
|
175
|
+
complexity: params.complexity,
|
|
144
176
|
background: params.background,
|
|
145
177
|
output_format: params.output_format,
|
|
146
178
|
output_compression: params.output_compression,
|
|
@@ -148,13 +180,19 @@ function mediaOutputOptions(model: string, params: Partial<Static<typeof imagePa
|
|
|
148
180
|
n: params.n,
|
|
149
181
|
input_fidelity: params.input_fidelity,
|
|
150
182
|
partial_images: params.partial_images,
|
|
183
|
+
frame_count: params.frame_count,
|
|
184
|
+
frame_duration_ms: params.frame_duration_ms,
|
|
151
185
|
};
|
|
152
186
|
const controls = mediaControls(model);
|
|
187
|
+
const gif = options.output_format === "gif";
|
|
188
|
+
// OpenAI generates GIF and SVG without streaming and only on the generation endpoint.
|
|
189
|
+
const generationOnly = gif || options.output_format === "svg";
|
|
153
190
|
for (const field of [
|
|
154
191
|
"size",
|
|
155
192
|
"resolution",
|
|
156
193
|
"aspect_ratio",
|
|
157
194
|
"quality",
|
|
195
|
+
"complexity",
|
|
158
196
|
"background",
|
|
159
197
|
"output_format",
|
|
160
198
|
"moderation",
|
|
@@ -175,7 +213,7 @@ function mediaOutputOptions(model: string, params: Partial<Static<typeof imagePa
|
|
|
175
213
|
height % limits.multiple_of === 0 &&
|
|
176
214
|
Math.max(width, height) <= limits.max_edge &&
|
|
177
215
|
Math.max(width, height) / Math.min(width, height) <= limits.max_aspect_ratio &&
|
|
178
|
-
pixels >= limits.min_pixels &&
|
|
216
|
+
(gif || pixels >= limits.min_pixels) &&
|
|
179
217
|
pixels <= limits.max_pixels
|
|
180
218
|
)
|
|
181
219
|
continue;
|
|
@@ -196,11 +234,24 @@ function mediaOutputOptions(model: string, params: Partial<Static<typeof imagePa
|
|
|
196
234
|
!Number.isInteger(options.partial_images) ||
|
|
197
235
|
options.partial_images < controls.partial_images.min ||
|
|
198
236
|
options.partial_images > controls.partial_images.max ||
|
|
199
|
-
(options.partial_images > 0 && (options.n ?? 1) !== 1))
|
|
237
|
+
(options.partial_images > 0 && ((options.n ?? 1) !== 1 || generationOnly)))
|
|
200
238
|
)
|
|
201
239
|
throw new Error(
|
|
202
|
-
"CLIProxyAPI partial_images requires an OpenAI image model, n of 1, and an integer from 0 to 3.",
|
|
240
|
+
"CLIProxyAPI partial_images requires an OpenAI image model, n of 1, PNG, JPEG, or WebP output, and an integer from 0 to 3.",
|
|
203
241
|
);
|
|
242
|
+
for (const field of ["frame_count", "frame_duration_ms"] as const) {
|
|
243
|
+
const value = options[field];
|
|
244
|
+
const range = controls.gif?.[field];
|
|
245
|
+
if (
|
|
246
|
+
value !== undefined &&
|
|
247
|
+
(!gif || !range || !Number.isInteger(value) || value < range.min || value > range.max)
|
|
248
|
+
)
|
|
249
|
+
throw new Error(`CLIProxyAPI ${field} requires gif output_format and a value within controls.gif.`);
|
|
250
|
+
}
|
|
251
|
+
if (options.output_format === "svg" && (options.n ?? 1) !== 1)
|
|
252
|
+
throw new Error("CLIProxyAPI svg output_format requires n of 1.");
|
|
253
|
+
if (generationOnly && params.images !== undefined)
|
|
254
|
+
throw new Error("CLIProxyAPI gif and svg output_format are for generation, not editing.");
|
|
204
255
|
if (
|
|
205
256
|
options.output_compression !== undefined &&
|
|
206
257
|
(!controls.output_compression ||
|
|
@@ -212,7 +263,11 @@ function mediaOutputOptions(model: string, params: Partial<Static<typeof imagePa
|
|
|
212
263
|
throw new Error(
|
|
213
264
|
"CLIProxyAPI output_compression requires jpeg or webp output_format and an integer from 0 to 100.",
|
|
214
265
|
);
|
|
215
|
-
if (
|
|
266
|
+
if (
|
|
267
|
+
options.background === "transparent" &&
|
|
268
|
+
options.output_format !== undefined &&
|
|
269
|
+
!["png", "webp"].includes(options.output_format)
|
|
270
|
+
)
|
|
216
271
|
throw new Error("CLIProxyAPI transparent backgrounds require png or webp output_format.");
|
|
217
272
|
if (options.input_fidelity !== undefined && params.images === undefined)
|
|
218
273
|
throw new Error("CLIProxyAPI input_fidelity requires editing images.");
|
|
@@ -226,7 +281,8 @@ async function imageInputs(
|
|
|
226
281
|
signal: AbortSignal,
|
|
227
282
|
) {
|
|
228
283
|
if (params.images === undefined) {
|
|
229
|
-
if (params.mask !== undefined
|
|
284
|
+
if (params.mask !== undefined || params.markup !== undefined)
|
|
285
|
+
throw new Error("CLIProxyAPI mask and markup require editing images.");
|
|
230
286
|
return {};
|
|
231
287
|
}
|
|
232
288
|
const limits = mediaControls(model).editing;
|
|
@@ -237,14 +293,18 @@ async function imageInputs(
|
|
|
237
293
|
params.images.length > limits.max_images
|
|
238
294
|
)
|
|
239
295
|
throw new Error("CLIProxyAPI editing requires 1–16 local images and a supported OpenAI image model.");
|
|
296
|
+
if (params.markup !== undefined && (params.mask !== undefined || !limits.markup))
|
|
297
|
+
throw new Error(
|
|
298
|
+
"CLIProxyAPI markup requires GPT Image 2.5 or 2.6 Sunburst or Flare and cannot be combined with mask.",
|
|
299
|
+
);
|
|
300
|
+
// The mask or markup layer follows the input images.
|
|
301
|
+
const layer = params.mask ?? params.markup;
|
|
240
302
|
const images: { image_url: string }[] = [];
|
|
241
303
|
let mask: { image_url: string } | undefined;
|
|
304
|
+
let markup: { image_url: string } | undefined;
|
|
242
305
|
let firstPngDimensions: Buffer | undefined;
|
|
243
306
|
let total = 0;
|
|
244
|
-
for (const [index, path] of [
|
|
245
|
-
...params.images,
|
|
246
|
-
...(params.mask === undefined ? [] : [params.mask]),
|
|
247
|
-
].entries()) {
|
|
307
|
+
for (const [index, path] of [...params.images, ...(layer === undefined ? [] : [layer])].entries()) {
|
|
248
308
|
signal.throwIfAborted();
|
|
249
309
|
if (
|
|
250
310
|
typeof path !== "string" ||
|
|
@@ -285,22 +345,25 @@ async function imageInputs(
|
|
|
285
345
|
} catch {
|
|
286
346
|
throw new Error(`CLIProxyAPI editing input "${path}" is not a supported PNG, JPEG, or WebP image.`);
|
|
287
347
|
}
|
|
288
|
-
const
|
|
348
|
+
const isLayer = index === params.images.length;
|
|
289
349
|
if (index === 0 && mimeType === "image/png") firstPngDimensions = bytes.subarray(16, 24);
|
|
350
|
+
// Masks need a matching PNG first image; OpenAI checks markup on JPEG or WebP images after orientation.
|
|
351
|
+
const sameSize = firstPngDimensions?.equals(bytes.subarray(16, 24));
|
|
290
352
|
if (
|
|
291
|
-
|
|
353
|
+
isLayer &&
|
|
292
354
|
(mimeType !== "image/png" ||
|
|
293
|
-
!
|
|
294
|
-
|
|
355
|
+
![4, 6].includes(bytes[25]) ||
|
|
356
|
+
(sameSize === undefined ? params.mask !== undefined : !sameSize))
|
|
295
357
|
)
|
|
296
358
|
throw new Error(
|
|
297
|
-
"CLIProxyAPI mask and
|
|
359
|
+
"CLIProxyAPI mask and markup layers must be alpha-channel PNGs matching a PNG first image's size; masks need a PNG first image.",
|
|
298
360
|
);
|
|
299
361
|
const input = { image_url: `data:${mimeType};base64,${bytes.toString("base64")}` };
|
|
300
|
-
if (
|
|
301
|
-
else
|
|
362
|
+
if (!isLayer) images.push(input);
|
|
363
|
+
else if (params.mask !== undefined) mask = input;
|
|
364
|
+
else markup = input;
|
|
302
365
|
}
|
|
303
|
-
return { images, mask };
|
|
366
|
+
return { images, mask, markup };
|
|
304
367
|
}
|
|
305
368
|
|
|
306
369
|
export async function mediaKey(ctx: Pick<MediaContext, "modelRegistry">, signal: AbortSignal) {
|
|
@@ -603,7 +666,9 @@ async function generationKey(
|
|
|
603
666
|
const key = await mediaKey(ctx, signal);
|
|
604
667
|
const available = mapMediaCatalog(await fetchCatalog(config, key, signal));
|
|
605
668
|
if (!available.some(({ id }) => id === params.model)) {
|
|
606
|
-
throw new Error(
|
|
669
|
+
throw new Error(
|
|
670
|
+
"The selected media model is not available in the live CLIProxyAPI catalog. Update CLIProxyAPI, or choose an advertised model with /cli:model or an explicit model.",
|
|
671
|
+
);
|
|
607
672
|
}
|
|
608
673
|
return key;
|
|
609
674
|
}
|
|
@@ -614,7 +679,7 @@ function imageType(bytes: Buffer) {
|
|
|
614
679
|
if (
|
|
615
680
|
bytes.length > 24 &&
|
|
616
681
|
bytes.subarray(0, 8).equals(Buffer.from("89504e470d0a1a0a", "hex")) &&
|
|
617
|
-
bytes.toString("
|
|
682
|
+
bytes.toString("latin1", 12, 16) === "IHDR"
|
|
618
683
|
) {
|
|
619
684
|
extension = "png";
|
|
620
685
|
mimeType = "image/png";
|
|
@@ -630,8 +695,8 @@ function imageType(bytes: Buffer) {
|
|
|
630
695
|
mimeType = "image/jpeg";
|
|
631
696
|
} else if (
|
|
632
697
|
bytes.length > 20 &&
|
|
633
|
-
bytes.toString("
|
|
634
|
-
bytes.toString("
|
|
698
|
+
bytes.toString("latin1", 0, 4) === "RIFF" &&
|
|
699
|
+
bytes.toString("latin1", 8, 12) === "WEBP" &&
|
|
635
700
|
bytes.readUInt32LE(4) + 8 === bytes.length
|
|
636
701
|
) {
|
|
637
702
|
extension = "webp";
|
|
@@ -642,7 +707,71 @@ function imageType(bytes: Buffer) {
|
|
|
642
707
|
return { extension, mimeType };
|
|
643
708
|
}
|
|
644
709
|
|
|
645
|
-
|
|
710
|
+
// OpenAI documents static SVG without scripts, external resources, or raster images. Only static drawing
|
|
711
|
+
// elements with presentation attributes and same-document references pass; anything else fails closed.
|
|
712
|
+
const svgElements = new Set(
|
|
713
|
+
`svg g defs title desc symbol use path rect circle ellipse line polyline polygon text tspan textPath
|
|
714
|
+
linearGradient radialGradient stop pattern clipPath mask marker filter feBlend feColorMatrix feComponentTransfer
|
|
715
|
+
feComposite feConvolveMatrix feDiffuseLighting feDisplacementMap feDistantLight feDropShadow feFlood feFuncA
|
|
716
|
+
feFuncB feFuncG feFuncR feGaussianBlur feMerge feMergeNode feMorphology feOffset fePointLight
|
|
717
|
+
feSpecularLighting feSpotLight feTile feTurbulence`.split(/\s+/),
|
|
718
|
+
);
|
|
719
|
+
const xmlEntities: Record<string, string> = { amp: "&", lt: "<", gt: ">", quot: '"', apos: "'" };
|
|
720
|
+
// Checked after decoding references: event handlers and style attributes after any separator HTML accepts,
|
|
721
|
+
// CSS escapes and image functions, xml:base, foreign namespaces, and references outside the document. The scan
|
|
722
|
+
// covers text too, so a label that looks like an assignment fails closed. CSS whitespace is ASCII only.
|
|
723
|
+
const unsafeSvg =
|
|
724
|
+
/[^\w.:-](?:on[a-z]+|style)\s*=|\\|(?:image-set|image|src)\(|xml:base|xmlns(?::[\w.-]+)?\s*=\s*["'](?!http:\/\/www\.w3\.org\/(?:2000\/svg|1999\/xlink)["'])|(?:href|src)\s*=\s*(?:"(?!#)|'(?!#)|[^"'\s#>])|url\([ \t\n\r\f]*(?:"(?!#)|'(?!#)|[^"'#) \t\n\r\f])/i;
|
|
725
|
+
|
|
726
|
+
// Accept GIF or SVG only when the request asked GPT Image 2.6 for that format.
|
|
727
|
+
function outputType(bytes: Buffer, format?: string) {
|
|
728
|
+
if (format === "gif") {
|
|
729
|
+
if (bytes.length > 13 && /^GIF8[79]a$/.test(bytes.toString("latin1", 0, 6)) && bytes.at(-1) === 0x3b)
|
|
730
|
+
return { extension: "gif", mimeType: "image/gif" };
|
|
731
|
+
throw new Error("CLIProxyAPI returned a malformed GIF.");
|
|
732
|
+
}
|
|
733
|
+
if (format !== "svg") return imageType(bytes);
|
|
734
|
+
let text: string;
|
|
735
|
+
try {
|
|
736
|
+
text = new TextDecoder("utf-8", { fatal: true }).decode(bytes);
|
|
737
|
+
} catch {
|
|
738
|
+
throw new Error("CLIProxyAPI returned an SVG that is not valid UTF-8.");
|
|
739
|
+
}
|
|
740
|
+
// Only a UTF-8 XML declaration is removed: another encoding, such as ISO-2022-JP, makes renderers read
|
|
741
|
+
// different markup. Tags are checked before decoding because a decoded "<" in text is not markup.
|
|
742
|
+
const markup = text
|
|
743
|
+
.replace(
|
|
744
|
+
/^\s*<\?xml\s+version\s*=\s*(["'])1\.\d+\1(?:\s+encoding\s*=\s*(["'])utf-8\2)?(?:\s+standalone\s*=\s*(["'])(?:yes|no)\3)?\s*\?>/i,
|
|
745
|
+
"",
|
|
746
|
+
)
|
|
747
|
+
.trim();
|
|
748
|
+
// One decoding pass, so an escaped ampersand cannot start a second reference.
|
|
749
|
+
const decoded = markup.replace(
|
|
750
|
+
/&(?:#(x[\da-f]+|\d+)|(amp|lt|gt|quot|apos));?/gi,
|
|
751
|
+
(_, code?: string, name?: string) => {
|
|
752
|
+
if (code === undefined) return xmlEntities[name?.toLowerCase() ?? ""] ?? "\\";
|
|
753
|
+
const point = /^x/i.test(code) ? Number.parseInt(code.slice(1), 16) : Number(code);
|
|
754
|
+
return point <= 0x10ffff ? String.fromCodePoint(point) : "\\";
|
|
755
|
+
},
|
|
756
|
+
);
|
|
757
|
+
if (
|
|
758
|
+
!/^<svg[\s>]/.test(markup) ||
|
|
759
|
+
!markup.endsWith("</svg>") ||
|
|
760
|
+
// XML forbids these controls, and ESC can switch an encoding in the middle of an attribute name.
|
|
761
|
+
/(?![\t\n\r])\p{Cc}/u.test(text) ||
|
|
762
|
+
// Comments, CDATA, declarations, and processing instructions. HTML parses comments inside SVG
|
|
763
|
+
// integration points such as desc and title differently from XML, so none are accepted.
|
|
764
|
+
/<[!?]/.test(markup) ||
|
|
765
|
+
// XML defines only these named references; HTML would decode others such as ( into markup.
|
|
766
|
+
/&(?!(?:amp|lt|gt|quot|apos);|#)/.test(markup) ||
|
|
767
|
+
[...markup.matchAll(/<\/?([^\s/>]*)/g)].some(([, name]) => !svgElements.has(name)) ||
|
|
768
|
+
unsafeSvg.test(decoded)
|
|
769
|
+
)
|
|
770
|
+
throw new Error("CLIProxyAPI returned a malformed SVG or one with active or external content.");
|
|
771
|
+
return { extension: "svg", mimeType: "image/svg+xml" };
|
|
772
|
+
}
|
|
773
|
+
|
|
774
|
+
function parseImage(entry: unknown, format?: string) {
|
|
646
775
|
if (!isRecord(entry) || typeof entry.b64_json !== "string") {
|
|
647
776
|
throw new Error("CLIProxyAPI did not return b64_json image data. URL-only images are not downloaded.");
|
|
648
777
|
}
|
|
@@ -651,12 +780,12 @@ function parseImage(entry: unknown) {
|
|
|
651
780
|
if (!encoded || bytes.toString("base64") !== encoded) {
|
|
652
781
|
throw new Error("CLIProxyAPI returned invalid base64 image data.");
|
|
653
782
|
}
|
|
654
|
-
const { extension, mimeType } =
|
|
783
|
+
const { extension, mimeType } = outputType(bytes, format);
|
|
655
784
|
const image: ImageContent = { type: "image", data: encoded, mimeType };
|
|
656
785
|
return { bytes, extension, image };
|
|
657
786
|
}
|
|
658
787
|
|
|
659
|
-
function parseImages(value: unknown, count: number) {
|
|
788
|
+
function parseImages(value: unknown, count: number, format?: string) {
|
|
660
789
|
if (
|
|
661
790
|
!isRecord(value) ||
|
|
662
791
|
value.error !== undefined ||
|
|
@@ -664,7 +793,7 @@ function parseImages(value: unknown, count: number) {
|
|
|
664
793
|
value.data.length !== count
|
|
665
794
|
)
|
|
666
795
|
throw new Error(`CLIProxyAPI image response must contain ${count} image(s).`);
|
|
667
|
-
return value.data.map(parseImage);
|
|
796
|
+
return value.data.map((entry: unknown) => parseImage(entry, format));
|
|
668
797
|
}
|
|
669
798
|
|
|
670
799
|
function parseGoogleImages(value: unknown) {
|
|
@@ -727,7 +856,7 @@ function parseGoogleImages(value: unknown) {
|
|
|
727
856
|
}
|
|
728
857
|
|
|
729
858
|
// OpenAI reports image token usage. Costs use list prices and stay zero for unpriced models.
|
|
730
|
-
function imageUsage(model: string, value: unknown) {
|
|
859
|
+
function imageUsage(model: string, value: unknown, svg?: unknown) {
|
|
731
860
|
const tokens = (field: unknown) =>
|
|
732
861
|
typeof field === "number" && Number.isSafeInteger(field) && field >= 0 ? field : undefined;
|
|
733
862
|
if (!isRecord(value)) return undefined;
|
|
@@ -746,8 +875,22 @@ function imageUsage(model: string, value: unknown) {
|
|
|
746
875
|
cacheWrite: 0,
|
|
747
876
|
total: 0,
|
|
748
877
|
};
|
|
749
|
-
cost.total = cost.input + cost.output;
|
|
750
878
|
const usage: Usage = { input, output, cacheRead: 0, cacheWrite: 0, totalTokens: input + output, cost };
|
|
879
|
+
// SVG output adds the authoring model's tokens, priced from Pi's OpenAI metadata when Pi knows it.
|
|
880
|
+
if (isRecord(svg)) {
|
|
881
|
+
const author = svg.model;
|
|
882
|
+
const svgInput = tokens(svg.input_tokens);
|
|
883
|
+
const svgOutput = tokens(svg.output_tokens);
|
|
884
|
+
if (svgInput !== undefined && svgOutput !== undefined) {
|
|
885
|
+
const price = getBuiltinModels("openai").find(({ id }) => id === author)?.cost;
|
|
886
|
+
usage.input += svgInput;
|
|
887
|
+
usage.output += svgOutput;
|
|
888
|
+
usage.totalTokens += svgInput + svgOutput;
|
|
889
|
+
cost.input += (svgInput * (price?.input ?? 0)) / 1e6;
|
|
890
|
+
cost.output += (svgOutput * (price?.output ?? 0)) / 1e6;
|
|
891
|
+
}
|
|
892
|
+
}
|
|
893
|
+
cost.total = cost.input + cost.output;
|
|
751
894
|
return usage;
|
|
752
895
|
}
|
|
753
896
|
|
|
@@ -761,13 +904,14 @@ async function requestImages(
|
|
|
761
904
|
output: Partial<ReturnType<typeof mediaOutputOptions>>;
|
|
762
905
|
images?: { image_url: string }[];
|
|
763
906
|
mask?: { image_url: string };
|
|
907
|
+
markup?: { image_url: string };
|
|
764
908
|
},
|
|
765
909
|
signal: AbortSignal,
|
|
766
910
|
onPartial?: (image: ImageContent, index: number) => void,
|
|
767
911
|
{ onPayload, ...transport }: TransportOptions & { onPayload?: (payload: unknown) => unknown } = {},
|
|
768
912
|
) {
|
|
769
913
|
const route = mediaCapability(model)?.route;
|
|
770
|
-
const { prompt, images, mask } = request;
|
|
914
|
+
const { prompt, images, mask, markup } = request;
|
|
771
915
|
const { partial_images = 0, ...output } = request.output;
|
|
772
916
|
const n = output.n ?? 1;
|
|
773
917
|
const stream = route === "openai" && partial_images > 0;
|
|
@@ -782,7 +926,7 @@ async function requestImages(
|
|
|
782
926
|
aspect_ratio: output.aspect_ratio,
|
|
783
927
|
}
|
|
784
928
|
: route === "openai"
|
|
785
|
-
? { model, prompt, ...output, n, images, mask, ...(stream ? { stream, partial_images } : {}) }
|
|
929
|
+
? { model, prompt, ...output, n, images, mask, markup, ...(stream ? { stream, partial_images } : {}) }
|
|
786
930
|
: {
|
|
787
931
|
contents: [{ role: "user", parts: [{ text: prompt }] }],
|
|
788
932
|
generationConfig: {
|
|
@@ -807,9 +951,17 @@ async function requestImages(
|
|
|
807
951
|
body: payload,
|
|
808
952
|
openaiImage: route === "openai",
|
|
809
953
|
});
|
|
954
|
+
// Decode the format actually requested, which a native payload hook may have changed.
|
|
955
|
+
const format =
|
|
956
|
+
route === "openai" && isRecord(payload) && typeof payload.output_format === "string"
|
|
957
|
+
? payload.output_format
|
|
958
|
+
: undefined;
|
|
810
959
|
return {
|
|
811
|
-
images: route === "gemini" ? parseGoogleImages(result) : parseImages(result, n),
|
|
812
|
-
usage:
|
|
960
|
+
images: route === "gemini" ? parseGoogleImages(result) : parseImages(result, n, format),
|
|
961
|
+
usage:
|
|
962
|
+
route === "openai" && isRecord(result)
|
|
963
|
+
? imageUsage(model, result.usage, result.svg_usage)
|
|
964
|
+
: undefined,
|
|
813
965
|
};
|
|
814
966
|
}
|
|
815
967
|
// Each frame carries one full image, so the stream may hold the final image plus every preview.
|
|
@@ -839,7 +991,10 @@ async function requestImages(
|
|
|
839
991
|
(event) => isRecord(event) && typeof event.type === "string" && event.type.endsWith(".completed"),
|
|
840
992
|
);
|
|
841
993
|
if (completed.length !== n) throw new Error(`CLIProxyAPI image stream must complete ${n} image(s).`);
|
|
842
|
-
return {
|
|
994
|
+
return {
|
|
995
|
+
images: completed.map((event) => parseImage(event)),
|
|
996
|
+
usage: imageUsage(model, completed[0].usage),
|
|
997
|
+
};
|
|
843
998
|
}
|
|
844
999
|
|
|
845
1000
|
export async function generateNativeImages(
|
|
@@ -967,11 +1122,14 @@ export async function generateImage(
|
|
|
967
1122
|
let previewBudget = MAX_INLINE_IMAGE_BASE64_BYTES;
|
|
968
1123
|
const content: (TextContent | ImageContent)[] = [];
|
|
969
1124
|
for (const [index, { image }] of images.entries()) {
|
|
970
|
-
|
|
971
|
-
|
|
972
|
-
:
|
|
973
|
-
|
|
974
|
-
:
|
|
1125
|
+
// Chat providers disagree on animated GIF input and reject SVG, so both skip inline previews.
|
|
1126
|
+
const previewOmitted = ["image/gif", "image/svg+xml"].includes(image.mimeType)
|
|
1127
|
+
? "Preview omitted: GIF and SVG results are saved without an inline preview."
|
|
1128
|
+
: !ctx.model?.input.includes("image")
|
|
1129
|
+
? "Preview omitted: the current chat model does not support images."
|
|
1130
|
+
: image.data.length > previewBudget
|
|
1131
|
+
? "Preview omitted: image base64 exceeds the 4 MiB inline budget. Use read on the saved file."
|
|
1132
|
+
: undefined;
|
|
975
1133
|
content.push({
|
|
976
1134
|
type: "text",
|
|
977
1135
|
text: `Saved image: ${files[index].path}${previewOmitted ? `\n${previewOmitted}` : ""}`,
|
|
@@ -1076,7 +1234,7 @@ export function registerMediaTools(pi: ExtensionAPI, config: Config) {
|
|
|
1076
1234
|
name: "cliproxyapi_generate_image",
|
|
1077
1235
|
label: "CLIProxyAPI image",
|
|
1078
1236
|
description:
|
|
1079
|
-
"Generate images, or edit local images with OpenAI models by supplying images and an optional mask. Supports OpenAI size, quality, background, format/compression, image count and streamed partial previews
|
|
1237
|
+
"Generate images, or edit local images with OpenAI models by supplying images and an optional mask or markup layer. Supports OpenAI size, quality, complexity, background, format/compression, GIF/SVG output, image count and streamed partial previews; check cliproxyapi_media_models for each model's controls. Uses the selected model or automatic OpenAI/GPT default without changing chat. Saves final images under a new .pi/ directory with optional previews. Maximum response: 32 MiB in total, plus 32 MiB per streamed preview; inline base64: 4 MiB. No automatic retries.",
|
|
1080
1238
|
promptSnippet: "Generate requested raster images through CLIProxyAPI using the effective image default.",
|
|
1081
1239
|
promptGuidelines: [
|
|
1082
1240
|
"Use cliproxyapi_generate_image for requested raster images. Omit model to use the effective default unless a specific image model is requested; keep the current chat model.",
|