@yanickxia/agent-vision 0.1.0 → 0.3.0

package/skills/agent-vision/SKILL.md

@@ -1,64 +1,91 @@
 ---
 name: agent-vision
-description: Analyze images, screenshots, diagrams, and videos with the agent-vision MCP tools. Use for image questions, OCR, UI inspection, visual debugging, screen recordings, video summaries, and timeline analysis.
+description: Generate images from text using configured image APIs; analyze videos through sampled frames; provide fallback image understanding when native vision is unavailable or failed. For image analysis only, if you can see the image natively, look directly instead of using this skill. Native vision does not replace image generation.
 license: MIT
-compatibility: Requires the agent-vision MCP server or Node.js 20+ for the npx fallback, plus an OpenAI-compatible vision API.
+compatibility: Node.js 20+ for the CLI; the MCP server is optional.
 ---
 
 # agent-vision
 
-Use this skill when the user asks you to inspect an image or video. The MCP
-server is the execution layer; this skill explains when and how to call it.
+The Skill runs the standalone CLI; no MCP server is required. CLI and MCP read
+the same `agent-vision/settings.json` (`AGENT_VISION_CONFIG` can override its
+location) and use the same provider adapters, including Volcengine Ark.
 
-## Choose the tool
+## Choose the capability
 
-| Input | Tool |
-|---|---|
-| Screenshot, photo, diagram, UI, scanned text | `analyze_image` |
-| Screen recording, demo, clip, animation timeline | `analyze_video` |
+For **image generation**, skip the vision gate and use `generate` below.
+This creates a new bitmap from text; it does not edit an existing image.
 
-Do not call a vision tool for a text-only task.
+For **image analysis**, apply the following gate:
 
-## Calling rules
+If you can read images natively (for example the Read tool in Claude Code or
+built-in multimodal input), look at the image yourself and answer directly.
+Do not load or run anything from this skill for an image you can already see.
+Image analysis through this skill is a fallback for agents without native
+vision or when native image reading failed.
 
-1. Pass the user's original question as `prompt`. A focused question is better
-   than a generic request for a description.
+Videos are the exception: no agent can watch a video natively. Video analysis
+always goes through the CLI below.
+
+## Analyze with the CLI
+
+When you cannot see the input yourself (or it is a video), run the analysis
+directly and read the plain-text output:
+
+```bash
+npx -y @yanickxia/agent-vision image "/absolute/path/to/error.png" --prompt "Read the error message and identify the likely cause."
+npx -y @yanickxia/agent-vision video "/absolute/path/to/demo.mp4" --prompt "Summarize the UI flow in chronological order." --frames 8
+```
+
+If the package is installed locally, run `agent-vision image ...` instead of
+`npx -y @yanickxia/agent-vision ...`.
+For a source checkout, run `npm run build` in that checkout once, then use
+`node /absolute/path/to/agent-vision/dist/cli.js ...` for any CLI command.
+This uses the checkout's implementation rather than an older published package.
+
+## Analysis calling rules
+
+1. Pass the user's original question as `--prompt`. A focused question is
+   better than a generic request for a description.
 2. Prefer an absolute local path. HTTP(S) URLs are also supported. Images may
    additionally be supplied as a `data:image/...;base64,...` URI.
-3. For videos, keep the default `max_frames=8` unless the user needs a faster
+3. For videos, keep the default `--frames 8` unless the user needs a faster
    result or a denser timeline. The allowed range is 1–16.
 4. If the result is uncertain, say so. Never invent visual details after a
-   failed tool call.
+   failed run.
 5. Retry at most once for a transient API or timeout error. Configuration,
    unsupported-format, and missing-file errors require correction, not retries.
 
-## Examples
-
-```json
-analyze_image({
-  "source": "/absolute/path/to/error.png",
-  "prompt": "Read the error message and identify the likely cause."
-})
-```
+## Generate an image
 
-```json
-analyze_video({
-  "source": "/absolute/path/to/demo.mp4",
-  "prompt": "Summarize the UI flow in chronological order.",
-  "max_frames": 8
-})
+```bash
+npx -y @yanickxia/agent-vision generate --prompt "A watercolor orange cat sitting by a window" --output "/absolute/path/to/cat.png"
 ```
 
-## npx fallback
-
-If the MCP tools are not available but shell execution is allowed, run:
+With a configured Volcengine Ark provider:
 
 ```bash
-npx -y @yanickxia/agent-vision image "/absolute/path/to/image.png" --prompt "user question"
-npx -y @yanickxia/agent-vision video "/absolute/path/to/video.mp4" --prompt "user question" --frames 8
+npx -y @yanickxia/agent-vision generate --prompt "A watercolor orange cat sitting by a window" --output "/absolute/path/to/cat.jpg" --size 2K
 ```
 
-Before the npm package is available, replace the package name with
-`github:yanickxia/agent-vision`.
+- Pass the requested image description as `--prompt` (or `-p`).
+- Choose a new local file via `--output` (or `-o`); its parent directory must
+  exist. Existing files are never overwritten. Successful stdout is the saved
+  absolute path. The file keeps the provider's actual image format; the filename
+  extension does not convert it.
+- `--size` and `--quality` are optional provider-specific values. Omit them
+  unless the task or model requires them.
+- For Volcengine Ark Seedream, configure `type: "volcengine-ark"`; omit
+  `--quality` (unsupported). Use a supported size such as `2K`, or omit size.
+  See the configuration reference for its endpoint and `ARK_API_KEY` fallback.
+- Generation providers are independent from vision providers and are tried
+  sequentially, stopping after the first success. Do not blindly repeat a
+  failed generation: a timeout may still incur a charge, and configured
+  fallback targets may already have been attempted.
+- If using MCP instead, call `generate_image` with `prompt`, optional
+  `output_path`, `size`, and `quality`. It returns image content and metadata;
+  without `output_path` it does not save a local file.
 
-Configuration details are in `references/configuration.md`.
+Read [references/configuration.md](references/configuration.md) when configuring
+providers or diagnosing a configuration error. Run `agent-vision doctor` to
+inspect configured vision and generation targets without calling either API.

package/skills/agent-vision/references/configuration.md

@@ -1,6 +1,119 @@
 # Configuration
 
-Required for normal use:
+agent-vision reads its configuration from a JSON settings file. Environment
+variables still work as a fallback.
+
+## Settings file
+
+The first location that exists wins:
+
+1. `$AGENT_VISION_CONFIG` — an explicit file path (an error if it is missing)
+2. `$XDG_CONFIG_HOME/agent-vision/settings.json`
+   (default: `~/.config/agent-vision/settings.json`)
+
+```json
+{
+  "providers": [
+    {
+      "name": "dashscope",
+      "baseUrl": "https://dashscope.aliyuncs.com/compatible-mode/v1",
+      "apiKey": "sk-...",
+      "models": ["qwen-vl-max", "qwen-vl-plus"]
+    },
+    {
+      "name": "openai",
+      "baseUrl": "https://api.openai.com/v1",
+      "apiKey": "sk-...",
+      "models": ["gpt-4o"]
+    }
+  ],
+  "generationProviders": [
+    {
+      "name": "image-api",
+      "baseUrl": "https://your-image-api.example/v1",
+      "apiKey": "sk-...",
+      "models": ["your-image-generation-model"]
+    }
+  ],
+  "maxTokens": 4096,
+  "timeoutMs": 120000
+}
+```
+
+- Each vision provider (`providers`) × model combination is a target. With several targets, all
+  of them are requested **concurrently** and the first successful answer
+  wins; the remaining requests are aborted. Note the cost: N targets means
+  up to N paid completions per analysis.
+- `name` is optional and only used in error messages; it defaults to the
+  baseUrl host.
+- `apiKey` is optional per provider (for local endpoints without auth) and
+  falls back to the environment key.
+- `headers` per provider merges over the top-level `headers`.
+- Top-level knobs (all optional): `maxTokens`, `timeoutMs`, `headers`,
+  `maxImageMb`, `maxVideoMb`, `allowPrivateUrls`, `ffmpegPath`.
+- Fields defined in the file take precedence over environment variables.
+- The file may contain API keys: keep it out of version control and run
+  `chmod 600` on it.
+
+## Image generation
+
+`generationProviders` uses the same connection fields as `providers`:
+required `baseUrl` and non-empty `models`, optional `name`, `apiKey`, and
+`headers`, plus a generation-only `type`: `openai-images` (default for old
+configs) or `volcengine-ark`. Unknown types are rejected; protocol selection
+is explicit, not inferred from the hostname or model name. You may configure
+either capability alone or both in one file. Both adapters use synchronous
+`POST /images/generations` responses with `data[0].b64_json` or `data[0].url`.
+
+Generation targets are tried **sequentially in file order**, stopping at the
+first valid image. They are not raced because multiple successful generations
+would waste money. Failure fallback can still incur multiple charges, especially
+after timeouts. Configure one target to avoid automatic fallback.
+
+Each call requests one image. `size` is optional; `quality` is optional and
+only supported by `openai-images`. Omitting them keeps provider defaults.
+The OpenAI adapter omits `response_format`, because some models reject it.
+This supports text-to-image, not image edits or async jobs.
+
+Generation shares `timeoutMs`, `maxImageMb`, `headers`, and `allowPrivateUrls`
+with analysis. Increase `timeoutMs` for slower generation models. API-returned
+image URLs and redirects are subject to the same private-URL restrictions as
+input images; API credentials are not forwarded to the download host.
+
+### Volcengine Ark (Seedream)
+
+```json
+{
+  "generationProviders": [
+    {
+      "type": "volcengine-ark",
+      "name": "volcengine",
+      "baseUrl": "https://ark.cn-beijing.volces.com/api/v3",
+      "models": ["doubao-seedream-5-0-lite-260128"]
+    }
+  ]
+}
+```
+
+Supply a provider `apiKey`, or set `ARK_API_KEY`; an explicit
+`AGENT_VISION_GENERATION_API_KEY` overrides the latter. Model IDs must be
+enabled for the account; `ep-...` IDs pointing to image models also work.
+
+The Ark adapter requests `response_format: "b64_json"` (still accepts URL
+responses), never sends OpenAI's `n` or `quality`, and uses the server's
+single-image/non-streaming defaults. It omits model-specific group/stream flags
+so the same payload also works for opaque endpoint IDs. Watermark and output
+format remain at provider defaults. `--quality` is rejected before the request;
+use a supported `--size`, such as `2K`, or leave it unset. Images commonly use
+JPEG; choosing a `.png` filename does not convert the bytes.
+
+This is the [Ark Seedream API](https://ark.volcengine.com/region:cn-beijing/docs/image-generation-api.md),
+not the separate Volcengine Visual Intelligence/Jimeng AK/SK-signed API.
+Image editing, image groups and layer decomposition are not implemented.
+
+## Environment variables (fallback)
+
+For image/video understanding when no settings file providers are used:
 
 ```bash
 export AGENT_VISION_BASE_URL="https://your-openai-compatible-endpoint/v1"
@@ -12,15 +125,53 @@ export AGENT_VISION_MODEL="your-vision-model"
 not require authentication. The server also recognizes `VISION_*` and
 `OPENAI_*` aliases.
 
-Useful optional variables:
+For generation when no `generationProviders` are defined:
+
+```bash
+export AGENT_VISION_GENERATION_BASE_URL="https://your-image-api.example/v1"
+export AGENT_VISION_GENERATION_API_KEY="your-key"
+export AGENT_VISION_GENERATION_MODEL="your-image-generation-model"
+```
+
+`AGENT_VISION_GENERATION_TYPE` defaults to `openai-images`. In that mode, the
+generation URL and key fall back to the corresponding `AGENT_VISION_*` values
+and their aliases. With `volcengine-ark`, the URL defaults to
+`https://ark.cn-beijing.volces.com/api/v3` and the key falls back to `ARK_API_KEY`;
+neither inherits the vision/OpenAI URL or key, to avoid sending another
+provider's credentials to Ark. Explicit `AGENT_VISION_GENERATION_BASE_URL` and
+`AGENT_VISION_GENERATION_API_KEY` override these defaults.
+
+For an environment-only Ark setup:
+
+```bash
+export AGENT_VISION_GENERATION_TYPE="volcengine-ark"
+export AGENT_VISION_GENERATION_MODEL="doubao-seedream-5-0-lite-260128"
+export ARK_API_KEY="your-ark-key"
+```
+
+The generation model must be explicit: it never falls back to a vision/chat
+model. File provider keys use their own type's fallback; an explicit
+`"apiKey": ""` disables key fallback for an unauthenticated local provider.
+File providers default to `openai-images` independently of the environment's
+generation type.
+
+`AGENT_VISION_GENERATION_HEADERS` adds generation-only headers. Header precedence
+(last wins): `AGENT_VISION_HEADERS` → `AGENT_VISION_GENERATION_HEADERS` →
+file top-level `headers` → provider `headers`. File `generationProviders` take
+precedence over the environment's generation target.
+
+Optional variables:
 
 - `AGENT_VISION_MAX_TOKENS` (default `4096`)
 - `AGENT_VISION_TIMEOUT_MS` (default `120000`)
-- `AGENT_VISION_MAX_IMAGE_MB` (default `20`)
+- `AGENT_VISION_MAX_IMAGE_MB` (default `20`, also applies to generated images)
 - `AGENT_VISION_MAX_VIDEO_MB` (default `200`)
 - `AGENT_VISION_HEADERS` (JSON object of extra HTTP headers)
-- `AGENT_VISION_ALLOW_PRIVATE_URLS=true` (allow private image/video source URLs)
+- `AGENT_VISION_ALLOW_PRIVATE_URLS=true` (allow private input and generated-image URLs)
 - `AGENT_VISION_FFMPEG_PATH` (override ffmpeg; otherwise PATH, then bundled binary)
 
 The configured API endpoint itself may be private; the private-URL guard only
-applies to user-provided image and video source URLs.
+applies to image/video source URLs and API-returned generated-image URLs.
+
+`agent-vision doctor` lists both capabilities without sending API requests.
+Generation-only setups do not need a vision target or ffmpeg.