@yanickxia/agent-vision 0.2.0 → 0.3.0

package/skills/agent-vision/references/configuration.md

@@ -27,12 +27,20 @@ The first location that exists wins:
       "models": ["gpt-4o"]
     }
   ],
+  "generationProviders": [
+    {
+      "name": "image-api",
+      "baseUrl": "https://your-image-api.example/v1",
+      "apiKey": "sk-...",
+      "models": ["your-image-generation-model"]
+    }
+  ],
   "maxTokens": 4096,
   "timeoutMs": 120000
 }
 ```
 
-- Every provider × model combination is a target. With several targets, all
+- Each vision provider (`providers`) × model combination is a target. With several targets, all
   of them are requested **concurrently** and the first successful answer
   wins; the remaining requests are aborted. Note the cost: N targets means
   up to N paid completions per analysis.
@@ -47,9 +55,65 @@ The first location that exists wins:
 - The file may contain API keys: keep it out of version control and run
   `chmod 600` on it.
 
+## Image generation
+
+`generationProviders` uses the same connection fields as `providers`:
+required `baseUrl` and non-empty `models`, optional `name`, `apiKey`, and
+`headers`, plus a generation-only `type`: `openai-images` (default for old
+configs) or `volcengine-ark`. Unknown types are rejected; protocol selection
+is explicit, not inferred from the hostname or model name. You may configure
+either capability alone or both in one file. Both adapters use synchronous
+`POST /images/generations` responses with `data[0].b64_json` or `data[0].url`.
+
+Generation targets are tried **sequentially in file order**, stopping at the
+first valid image. They are not raced because multiple successful generations
+would waste money. Failure fallback can still incur multiple charges, especially
+after timeouts. Configure one target to avoid automatic fallback.
+
+Each call requests one image. `size` is optional; `quality` is optional and
+only supported by `openai-images`. Omitting them keeps provider defaults.
+The OpenAI adapter omits `response_format`, because some models reject it.
+This supports text-to-image, not image edits or async jobs.
+
+Generation shares `timeoutMs`, `maxImageMb`, `headers`, and `allowPrivateUrls`
+with analysis. Increase `timeoutMs` for slower generation models. API-returned
+image URLs and redirects are subject to the same private-URL restrictions as
+input images; API credentials are not forwarded to the download host.
+
+### Volcengine Ark (Seedream)
+
+```json
+{
+  "generationProviders": [
+    {
+      "type": "volcengine-ark",
+      "name": "volcengine",
+      "baseUrl": "https://ark.cn-beijing.volces.com/api/v3",
+      "models": ["doubao-seedream-5-0-lite-260128"]
+    }
+  ]
+}
+```
+
+Supply a provider `apiKey`, or set `ARK_API_KEY`; an explicit
+`AGENT_VISION_GENERATION_API_KEY` overrides the latter. Model IDs must be
+enabled for the account; `ep-...` IDs pointing to image models also work.
+
+The Ark adapter requests `response_format: "b64_json"` (still accepts URL
+responses), never sends OpenAI's `n` or `quality`, and uses the server's
+single-image/non-streaming defaults. It omits model-specific group/stream flags
+so the same payload also works for opaque endpoint IDs. Watermark and output
+format remain at provider defaults. `--quality` is rejected before the request;
+use a supported `--size`, such as `2K`, or leave it unset. Images commonly use
+JPEG; choosing a `.png` filename does not convert the bytes.
+
+This is the [Ark Seedream API](https://ark.volcengine.com/region:cn-beijing/docs/image-generation-api.md),
+not the separate Volcengine Visual Intelligence/Jimeng AK/SK-signed API.
+Image editing, image groups and layer decomposition are not implemented.
+
 ## Environment variables (fallback)
 
-Required when no settings file is used:
+For image/video understanding when no settings file providers are used:
 
 ```bash
 export AGENT_VISION_BASE_URL="https://your-openai-compatible-endpoint/v1"
@@ -61,15 +125,53 @@ export AGENT_VISION_MODEL="your-vision-model"
 not require authentication. The server also recognizes `VISION_*` and
 `OPENAI_*` aliases.
 
+For generation when no `generationProviders` are defined:
+
+```bash
+export AGENT_VISION_GENERATION_BASE_URL="https://your-image-api.example/v1"
+export AGENT_VISION_GENERATION_API_KEY="your-key"
+export AGENT_VISION_GENERATION_MODEL="your-image-generation-model"
+```
+
+`AGENT_VISION_GENERATION_TYPE` defaults to `openai-images`. In that mode, the
+generation URL and key fall back to the corresponding `AGENT_VISION_*` values
+and their aliases. With `volcengine-ark`, the URL defaults to
+`https://ark.cn-beijing.volces.com/api/v3` and the key falls back to `ARK_API_KEY`;
+neither inherits the vision/OpenAI URL or key, to avoid sending another
+provider's credentials to Ark. Explicit `AGENT_VISION_GENERATION_BASE_URL` and
+`AGENT_VISION_GENERATION_API_KEY` override these defaults.
+
+For an environment-only Ark setup:
+
+```bash
+export AGENT_VISION_GENERATION_TYPE="volcengine-ark"
+export AGENT_VISION_GENERATION_MODEL="doubao-seedream-5-0-lite-260128"
+export ARK_API_KEY="your-ark-key"
+```
+
+The generation model must be explicit: it never falls back to a vision/chat
+model. File provider keys use their own type's fallback; an explicit
+`"apiKey": ""` disables key fallback for an unauthenticated local provider.
+File providers default to `openai-images` independently of the environment's
+generation type.
+
+`AGENT_VISION_GENERATION_HEADERS` adds generation-only headers. Header precedence
+(last wins): `AGENT_VISION_HEADERS` → `AGENT_VISION_GENERATION_HEADERS` →
+file top-level `headers` → provider `headers`. File `generationProviders` take
+precedence over the environment's generation target.
+
 Optional variables:
 
 - `AGENT_VISION_MAX_TOKENS` (default `4096`)
 - `AGENT_VISION_TIMEOUT_MS` (default `120000`)
-- `AGENT_VISION_MAX_IMAGE_MB` (default `20`)
+- `AGENT_VISION_MAX_IMAGE_MB` (default `20`, also applies to generated images)
 - `AGENT_VISION_MAX_VIDEO_MB` (default `200`)
 - `AGENT_VISION_HEADERS` (JSON object of extra HTTP headers)
-- `AGENT_VISION_ALLOW_PRIVATE_URLS=true` (allow private image/video source URLs)
+- `AGENT_VISION_ALLOW_PRIVATE_URLS=true` (allow private input and generated-image URLs)
 - `AGENT_VISION_FFMPEG_PATH` (override ffmpeg; otherwise PATH, then bundled binary)
 
 The configured API endpoint itself may be private; the private-URL guard only
-applies to user-provided image and video source URLs.
+applies to image/video source URLs and API-returned generated-image URLs.
+
+`agent-vision doctor` lists both capabilities without sending API requests.
+Generation-only setups do not need a vision target or ffmpeg.