picturereader-zcode 1.0.3 → 2.0.1

package/skills/vision-analyze.md

@@ -0,0 +1,256 @@
+---
+name: vision-analyze
+description: Unified image understanding tool that combines pixel scan, OCR, and optional VLM for complete image analysis. Use when you need one call to both verify what is in the image and get a natural-language interpretation.
+whenToUse: 需要一次性获取图片的多种证据（像素扫描 + OCR + VLM 描述）时使用。
+---
+
+# vision_analyze 统一识图工具
+
+## 工具概述
+
+`vision_analyze` 是 picturereader 的统一识图入口，一次调用可获取：
+- **像素扫描证据**（image_scan）
+- **OCR 文字识别**（image_ocr）
+- **VLM 语义描述**（可选，需配置 SEE_BASE）
+
+## 核心特性
+
+### 1. 低信息量拦截
+自动检测空白/简单图片，防止 VLM 幻觉：
+- 颜色种类太少（≤8 种）
+- 单一颜色占比过高（≥90%）
+- 主导颜色且边缘稀少
+- 亮度方差太小
+
+### 2. 证据交叉验证
+- VLM 描述与像素/OCR 冲突时，以像素/OCR 实测为准
+- 所有证据以文本形式返回，供主模型推理
+
+### 3. VLM 可选配置
+- 默认不配置 VLM，只返回像素扫描和 OCR 证据
+- 需要 VLM 时，设置 `SEE_BASE` 环境变量
+
+### 4. 智能调用策略
+- **简单图片**：不调用外部 API，使用像素扫描 + OCR 即可
+- **复杂/精密图片**：调用外部 API 获取语义理解
+- **多次提问**：支持对同一张图进行多次不同角度的提问
+
+## 使用建议
+
+### 推荐的工作流程
+
+建议先用 `image_scan` 自己看，了解图片内容后再决定是否需要调用 VLM：
+
+```
+# 先看图片内容
+image_scan(file_path="C:/shot.png", size=32)
+
+# 根据结果决定下一步
+# - 简单图片 → 直接描述，不需要 VLM
+# - 需要文字 → image_ocr
+# - 复杂场景 → vision_analyze（含 VLM）
+```
+
+### 何时调用 VLM
+
+简单图片用像素扫描就够了，复杂场景可以调用 VLM 获取语义理解。具体由你根据图片内容判断。
+
+### 交叉验证
+
+主模型需要对 VLM 结果进行交叉验证：
+
+1. **像素证据优先**：VLM 描述与像素扫描冲突时，以像素证据为准
+2. **OCR 优先**：VLM 识别的文字与 OCR 冲突时，以 OCR 为准
+3. **逻辑验证**：VLM 描述不符合逻辑时（如"天空是绿色的"），标记为幻觉
+4. **多次提问验证**：对同一张图进行多次不同角度的提问，验证一致性
+
+### 多次提问策略
+
+对同一张图可以进行多次不同角度的提问，以获取更全面的理解：
+
+```
+# 第一次：整体描述
+vision_analyze(
+  file_path="C:/shot.png",
+  prompt="描述这个图片的整体内容",
+  include_scan=true,
+  include_ocr=true,
+  include_vlm=true
+)
+
+# 第二次：细节询问
+vision_analyze(
+  file_path="C:/shot.png",
+  prompt="图片中有哪些文字？请详细列出",
+  include_scan=false,
+  include_ocr=false,
+  include_vlm=true
+)
+
+# 第三次：推理判断
+vision_analyze(
+  file_path="C:/shot.png",
+  prompt="这个界面设计是否合理？有哪些问题？",
+  include_scan=false,
+  include_ocr=false,
+  include_vlm=true
+)
+```
+
+## 使用方法
+
+### 基本用法（无 VLM）
+```
+vision_analyze(
+  file_path="C:/shot.png",
+  include_scan=true,
+  include_ocr=false,
+  include_vlm=false
+)
+```
+
+### 完整用法（含 VLM）
+```
+vision_analyze(
+  file_path="C:/shot.png",
+  prompt="描述这个界面，有哪些元素？布局是否正常？",
+  include_scan=true,
+  include_ocr=true,
+  include_vlm=true,
+  allow_low_info=false,
+  stop_after=false
+)
+```
+
+### 多次提问用法
+```
+# 对同一张图进行多次不同角度的提问
+vision_analyze(file_path="C:/shot.png", prompt="整体描述", include_vlm=true)
+vision_analyze(file_path="C:/shot.png", prompt="有哪些文字？", include_vlm=true)
+vision_analyze(file_path="C:/shot.png", prompt="设计是否合理？", include_vlm=true)
+```
+
+## 参数说明
+
+| 参数 | 类型 | 默认值 | 说明 |
+|------|------|--------|------|
+| `file_path` | string | 必需 | 图片路径（PNG/JPEG/GIF/BMP） |
+| `prompt` | string | "Describe this image in detail." | VLM 提示词 |
+| `include_scan` | boolean | true | 是否包含像素扫描证据 |
+| `include_ocr` | boolean | false | 是否包含 OCR 文字识别 |
+| `ocr_engine` | string | "windows" | OCR 引擎：windows 或 paddle |
+| `include_vlm` | boolean | true | 是否包含 VLM 描述（需配置 SEE_BASE） |
+| `allow_low_info` | boolean | false | 是否允许低信息量图片调用 VLM |
+| `stop_after` | boolean | false | 调用后是否停止本地 llama-server |
+
+## 输出格式
+
+```json
+{
+  "path": "C:/shot.png",
+  "lowInformation": false,
+  "scan": "[scan]\nimage: C:/shot.png (1920x1080 -> 32x18 cells, ...)\n...",
+  "ocr": "[ocr]\nocr: C:/shot.png (1920x1080, region=full, engine=windows)\n...",
+  "vlm": "[vlm]\n这是一个桌面应用程序的截图，包含...",
+  "combined": "[scan]\n...\n\n---\n\n[ocr]\n...\n\n---\n\n[vlm]\n..."
+}
+```
+
+## 使用场景
+
+### 1. UI/界面验证
+```
+vision_analyze(
+  file_path="C:/ui_screenshot.png",
+  prompt="这个界面有哪些按钮？布局是否正常？有没有错位？",
+  include_scan=true,
+  include_ocr=true,
+  include_vlm=true
+)
+```
+
+### 2. 游戏截图分析
+```
+vision_analyze(
+  file_path="C:/game_screenshot.png",
+  prompt="这是什么游戏？画面中有什么角色/物体？",
+  include_scan=true,
+  include_ocr=true,
+  include_vlm=true
+)
+```
+
+### 3. 文档/图片 OCR
+```
+vision_analyze(
+  file_path="C:/document.png",
+  include_scan=false,
+  include_ocr=true,
+  include_vlm=false,
+  ocr_engine="paddle"
+)
+```
+
+### 4. 长任务视觉验证
+```
+# 循环验证流程
+1. 截图
+2. vision_analyze(file_path="C:/step1.png", include_scan=true, include_ocr=true)
+3. 与预期比对
+4. 不一致则修正
+5. 再截图验证
+```
+
+## 配置说明
+
+### PaddleOCR（可选）
+```bash
+# 安装 PaddleOCR
+node scripts/setup-ocr.mjs
+
+# 环境变量
+DSH_PADDLE_PYTHON=C:\Users\Administrator\paddle_venv\Scripts\python.exe
+DSH_PADDLE_CACHE=<插件目录>\.paddlex-cache
+```
+
+### VLM（可选，默认不配置）
+```bash
+# 本地 llama-server
+SEE_BASE=http://127.0.0.1:8080/v1
+SEE_MODEL=Huihui-Qwen3-VL-4B-Instruct-abliterated
+SEE_SERVER_EXE=E:\llama\llama-server.exe
+SEE_SERVER_MODEL=E:\llama\models\model.f16.gguf
+SEE_SERVER_MMPROJ=E:\llama\models\mmproj-f16.gguf
+SEE_SERVER_PORT=8080
+SEE_SERVER_NGL=20
+SEE_SERVER_CTX=16384
+
+# 远程 API
+SEE_BASE=https://api.openai.com/v1
+SEE_MODEL=gpt-4-vision-preview
+SEE_API_KEY=sk-xxx
+```
+
+## 注意事项
+
+1. **VLM 可选**：默认不配置 VLM，`vision_analyze` 会跳过 VLM 调用，只返回像素扫描和 OCR 证据
+2. **低信息量拦截**：空白/简单图片会自动拦截，不调用 VLM（防止幻觉）
+3. **证据优先级**：像素扫描 > OCR > VLM（冲突时以实测为准）
+4. **性能考虑**：VLM 调用需要 2-5 秒，建议在需要语义理解时才启用
+5. **WebP 不支持**：需要先转换为 PNG/JPEG
+
+## 与其他工具的关系
+
+- **image_scan**：像素级扫描，返回详细的颜色/结构证据
+- **image_ocr**：文字识别，返回 OCR 文本
+- **image_sample**：材质/纹理取样
+- **vision_analyze**：统一入口，组合以上证据 + 可选 VLM
+
+**必须遵守的工作流程**：
+1. **先用 `image_scan` 自己看**（像素扫描）→ 了解图片内容和复杂度
+2. 根据扫描结果判断：
+   - 简单图片（颜色单一、结构简单）→ 不需要 VLM，直接描述
+   - 需要文字 → 用 `image_ocr`
+   - 复杂场景（多人物、多物体、复杂背景）→ 用 `vision_analyze`（含 VLM）
+   - 需要材质细节 → 用 `image_sample`
+3. **不要直接调用 VLM**，先自己看再决定是否需要外部 API

package/src/guard.js

@@ -0,0 +1,101 @@
+/**
+ * Low-information image guard.
+ *
+ * Small local VLMs tend to hallucinate on blank / very simple images. Before
+ * sending an image to a VLM we can cheaply measure color diversity, dominant
+ * color coverage, edge density and brightness variance, then decide whether
+ * the image is too empty to be worth a VLM call.
+ *
+ * @module picturereader/guard
+ */
+
+const SAMPLE = 64;
+
+/**
+ * Calculate luminance (Rec.601).
+ * @param {number} r - red channel 0..255.
+ * @param {number} g - green channel 0..255.
+ * @param {number} b - blue channel 0..255.
+ * @returns {number} luminance 0..255.
+ */
+function gray(r, g, b) {
+  return 0.299 * r + 0.587 * g + 0.114 * b;
+}
+
+/**
+ * Detect low-information images (blank, very simple, or unrendered).
+ *
+ * The guard checks four heuristics:
+ * 1. Color diversity: unique color buckets <= 8
+ * 2. Dominant color coverage: top color >= 90%
+ * 3. Dominant color with low edge density: top >= 60% AND edges < 8%
+ * 4. Low brightness variance: standard deviation < 8
+ *
+ * @param {Uint8ClampedArray|Buffer} rgba - RGBA pixel data.
+ * @param {number} width - image width in pixels.
+ * @param {number} height - image height in pixels.
+ * @returns {boolean} true when the image looks blank / very low-information.
+ */
+export function isLowInformationImage(rgba, width, height) {
+  if (width <= 0 || height <= 0 || rgba.length < 4) return true;
+
+  // Downsample to SAMPLE x SAMPLE (nearest neighbor is fine for a guard).
+  const cells = [];
+  const cellSizeX = Math.max(1, Math.floor(width / SAMPLE));
+  const cellSizeY = Math.max(1, Math.floor(height / SAMPLE));
+  const gridW = Math.min(SAMPLE, width);
+  const gridH = Math.min(SAMPLE, height);
+
+  for (let gy = 0; gy < gridH; gy++) {
+    for (let gx = 0; gx < gridW; gx++) {
+      const px = Math.min(width - 1, gx * cellSizeX + Math.floor(cellSizeX / 2));
+      const py = Math.min(height - 1, gy * cellSizeY + Math.floor(cellSizeY / 2));
+      const i = (py * width + px) * 4;
+      cells.push([rgba[i], rgba[i + 1], rgba[i + 2], rgba[i + 3]]);
+    }
+  }
+
+  const buckets = new Map();
+  let total = 0;
+  let sum = 0;
+  let sumSq = 0;
+  let edgeCount = 0;
+  let edgePairs = 0;
+
+  for (let y = 0; y < gridH; y++) {
+    for (let x = 0; x < gridW; x++) {
+      const [r, g, b] = cells[y * gridW + x];
+      // Quantize to 3 bits per channel for bucketing
+      const key = ((r & 0xe0) << 10) | ((g & 0xe0) << 5) | (b & 0xe0);
+      buckets.set(key, (buckets.get(key) ?? 0) + 1);
+
+      const lum = gray(r, g, b);
+      sum += lum;
+      sumSq += lum * lum;
+      total++;
+
+      // Check horizontal edge (luminance difference > 20)
+      if (x + 1 < gridW) {
+        const [r2, g2, b2] = cells[y * gridW + x + 1];
+        const lum2 = gray(r2, g2, b2);
+        if (Math.abs(lum - lum2) > 20) edgeCount++;
+        edgePairs++;
+      }
+    }
+  }
+
+  const unique = buckets.size;
+  const top = Math.max(...buckets.values());
+  const topRatio = total > 0 ? top / total : 1;
+  const edgeRatio = edgePairs > 0 ? edgeCount / edgePairs : 0;
+  const mean = total > 0 ? sum / total : 0;
+  const variance = total > 0 ? Math.max(0, sumSq / total - mean * mean) : 0;
+  const stdDev = Math.sqrt(variance);
+
+  return (
+    unique <= 8 ||
+    topRatio >= 0.9 ||
+    (topRatio >= 0.6 && edgeRatio < 0.08) ||
+    stdDev < 8
+  );
+}

package/src/index.js

@@ -15,6 +15,7 @@
  */
 
 import { createImageScanTool, createImageOcrTool, createImageSampleTool } from './tool.js';
+import { createVisionAnalyzeTool } from './vision-analyze.js';
 
 export const name = 'picturereader';
 
@@ -26,5 +27,6 @@ export function apply(ctx) {
     ctx.tools.register(createImageScanTool(ctx));
     ctx.tools.register(createImageOcrTool(ctx));
     ctx.tools.register(createImageSampleTool(ctx));
+    ctx.tools.register(createVisionAnalyzeTool(ctx));
   });
 }

package/src/vision-analyze.js

@@ -0,0 +1,260 @@
+/**
+ * vision_analyze — unified image understanding tool.
+ *
+ * Runs the full local vision pipeline:
+ *   1. decode + low-information guard
+ *   2. optional pixel scan (image_scan)
+ *   3. optional OCR (image_ocr)
+ *   4. optional local/remote VLM description
+ *
+ * All evidence is returned as text so a text-only model can reason about the
+ * image without trusting any single source blindly.
+ *
+ * Key features:
+ * - Smart API calling: simple images don't call external APIs
+ * - Cross-validation: main model verifies VLM results against pixel/OCR evidence
+ * - Multiple questions: support asking different questions about the same image
+ *
+ * @module picturereader/vision-analyze
+ */
+
+import { extname } from 'node:path';
+import { BYTE_CAP, MAX_PIXELS } from './tool.js';
+import { isLowInformationImage } from './guard.js';
+import { ensureServer, stopServer, sendVisionRequest, defaultVlmConfig, isVlmConfigured } from './vlm.js';
+
+const CORE_URL = new URL('./core.js', import.meta.url).href;
+let coreCache = { url: null, mtime: -1, module: null };
+
+/**
+ * Load the latest core.js module with cache-busting.
+ * @returns {Promise<object>} the core module namespace.
+ */
+async function importCore() {
+  const { stat } = await import('node:fs/promises');
+  const url = new URL(CORE_URL);
+  const info = await stat(url);
+  if (coreCache.module !== null && coreCache.url === CORE_URL && info.mtimeMs === coreCache.mtime) {
+    return coreCache.module;
+  }
+  const module = await import(`${url.href}?t=${info.mtimeMs}`);
+  coreCache = { url: CORE_URL, mtime: info.mtimeMs, module };
+  return module;
+}
+
+/**
+ * Parse a boolean argument with fallback.
+ * @param {any} value - the argument value.
+ * @param {boolean} fallback - default value.
+ * @returns {boolean} parsed boolean.
+ */
+function boolArg(value, fallback = false) {
+  if (value === undefined || value === null) return fallback;
+  if (typeof value === 'boolean') return value;
+  return String(value) === 'true' || String(value) === '1';
+}
+
+/**
+ * Build the vision_analyze tool.
+ * @param {object} ctx - the Cordis context.
+ * @returns {object} the tool definition.
+ */
+export function createVisionAnalyzeTool(ctx) {
+  return {
+    name: 'vision_analyze',
+    description: [
+      'Unified image understanding: decode an image, run a low-information guard, optionally scan pixels, OCR text, and/or ask the VLM for a semantic description.',
+      'Use this when you need one call to both verify what is in the image and get a natural-language interpretation.',
+      'Returns evidence blocks: scan (pixel stats), ocr (real text), vlm (model description). If low-information guard triggers and allow_low_info is false, it will not call the VLM.',
+      'Supported formats: PNG, JPEG, GIF (first frame), BMP. WebP is not supported yet.',
+      'VLM is optional: if SEE_BASE is not configured, VLM calls are skipped automatically.',
+      'Smart API calling: simple images (low color diversity, high dominant color coverage) skip VLM automatically.',
+      'Multiple questions: call this tool multiple times with different prompts on the same image for comprehensive analysis.',
+      'Cross-validation: main model should verify VLM results against pixel scan and OCR evidence.'
+    ].join(' '),
+    parameters: {
+      type: 'object',
+      additionalProperties: true,
+      properties: {
+        file_path: {
+          type: 'string',
+          description: 'Path to the image file (PNG/JPEG/GIF/BMP).'
+        },
+        prompt: {
+          type: 'string',
+          description: 'Question/instruction for the VLM, e.g. "Describe this UI" or "What is wrong with this map rendering?"'
+        },
+        include_scan: {
+          type: 'boolean',
+          description: 'Include pixel scan evidence (default true).'
+        },
+        include_ocr: {
+          type: 'boolean',
+          description: 'Include OCR text evidence (default false; set true when text matters).'
+        },
+        ocr_engine: {
+          type: 'string',
+          enum: ['windows', 'paddle'],
+          description: 'OCR engine: windows (default) or paddle (better for glowing/curved/game text).'
+        },
+        include_vlm: {
+          type: 'boolean',
+          description: 'Include VLM description (default true, but skipped if SEE_BASE not configured).'
+        },
+        allow_low_info: {
+          type: 'boolean',
+          description: 'Skip the low-information guard and force VLM even on blank/simple images (default false).'
+        },
+        stop_after: {
+          type: 'boolean',
+          description: 'Stop the local llama-server after this call if this plugin started it (default false).'
+        }
+      },
+      required: ['file_path']
+    },
+    output: {
+      schema: {
+        type: 'object',
+        additionalProperties: true,
+        properties: {
+          path: { type: 'string' },
+          lowInformation: { type: 'boolean' },
+          message: { type: 'string' },
+          scan: { type: 'string' },
+          ocr: { type: 'string' },
+          vlm: { type: 'string' },
+          combined: { type: 'string' }
+        },
+        required: ['path']
+      },
+      render: (_args, value) => {
+        const text = value.combined ?? value.message ?? JSON.stringify(value);
+        return [{ type: 'text', text }];
+      }
+    },
+    isConcurrencySafe: () => true,
+    async execute(args, exec) {
+      if (exec.signal?.aborted) throw new Error('vision_analyze: cancelled');
+      const filePath = String(args.file_path ?? '').trim();
+      if (!filePath) throw new Error('vision_analyze: file_path must be a non-empty string');
+
+      const ext = extname(filePath).toLowerCase();
+      const core = await importCore();
+      if (core.UNSUPPORTED_EXTENSIONS.has(ext)) {
+        throw new Error('vision_analyze: WebP is not supported yet — convert to PNG or JPEG first');
+      }
+      if (!core.IMAGE_EXTENSIONS.has(ext)) {
+        throw new Error(`vision_analyze: unsupported image type "${ext}" (supported: PNG, JPEG, GIF, BMP)`);
+      }
+
+      const cwd = exec.agent?.session?.header?.cwd;
+      const target = await ctx.fs.resolve(filePath, {
+        ...(cwd !== undefined ? { cwd } : {}),
+        signal: exec.signal
+      });
+      const info = await ctx.fs.stat(target, exec.signal);
+      if (!info) {
+        throw new Error(`vision_analyze: cannot read "${target.displayPath}": file not found`);
+      }
+      if (info.type !== 'file') {
+        throw new Error(`vision_analyze: cannot read "${target.displayPath}": not a regular file`);
+      }
+      const data = await ctx.fs.readBytes(target, exec.signal, BYTE_CAP);
+      const buf = Buffer.isBuffer(data) ? data : Buffer.from(data);
+      const image = core.decodeImage(buf, ext);
+      if (image.width * image.height > MAX_PIXELS) {
+        throw new Error(
+          `vision_analyze: ${image.width}x${image.height} exceeds the ${MAX_PIXELS}-pixel decode limit — downscale or crop first`
+        );
+      }
+
+      const includeScan = args.include_scan === undefined ? true : boolArg(args.include_scan, true);
+      const includeOcr = args.include_ocr === undefined ? false : boolArg(args.include_ocr, false);
+      const includeVlm = args.include_vlm === undefined ? true : boolArg(args.include_vlm, true);
+      const allowLowInfo = boolArg(args.allow_low_info, false);
+      const stopAfter = boolArg(args.stop_after, false);
+      const prompt = args.prompt ?? 'Describe this image in detail.';
+
+      // Check if VLM is configured
+      const vlmAvailable = isVlmConfigured();
+      const shouldCallVlm = includeVlm && vlmAvailable;
+
+      const lowInfo = isLowInformationImage(image.data, image.width, image.height);
+      const blocks = [];
+      let ocrText = '';
+      let scanText = '';
+      let vlmText = '';
+
+      if (lowInfo && !allowLowInfo) {
+        const message =
+          '[vision_analyze] 低信息量拦截：图片空白或内容极少，为避免 VLM 幻觉，未调用 VLM。' +
+          '请检查截图是否空白/未渲染/窗口在屏幕外；如确需识别请设置 allow_low_info=true。';
+        ctx.emit('fs/observed', target, { kind: 'present', version: info.version }, exec);
+        return { path: target.displayPath, lowInformation: true, message, combined: message };
+      }
+
+      if (includeScan) {
+        const analysis = core.analyzeImage(image.data, image.width, image.height, {
+          size: 32,
+          mode: 'auto',
+          region: undefined,
+          palette: 'auto'
+        });
+        scanText = core.renderImageScan({
+          path: target.displayPath,
+          width: image.width,
+          height: image.height,
+          ...analysis
+        });
+        blocks.push(`[scan]\n${scanText}`);
+      }
+
+      if (includeOcr) {
+        const engine = args.ocr_engine ?? 'windows';
+        const ocr = await core.ocrImage(buf, ext, { engine });
+        ocrText = core.renderOcr({
+          path: target.displayPath,
+          width: ocr.width,
+          height: ocr.height,
+          region: 'full',
+          engine: ocr.engine,
+          lines: ocr.lines
+        });
+        blocks.push(`[ocr]\n${ocrText}`);
+      }
+
+      if (shouldCallVlm) {
+        const config = defaultVlmConfig();
+        let startedByUs = false;
+        try {
+          const child = await ensureServer(config);
+          startedByUs = child !== null;
+          const base64 = buf.toString('base64');
+          const mime = ext === '.jpg' || ext === '.jpeg' ? 'image/jpeg' : ext === '.png' ? 'image/png' : ext === '.gif' ? 'image/gif' : 'image/bmp';
+          const safePrompt =
+            prompt +
+            '\n\n重要：只描述图中明确可见的内容。如果图中没有明显物体/文字/界面元素，请直接回答：画面空白或内容极少。不要推测、不要脑补不存在的角色/场景/文字。';
+          vlmText = await sendVisionRequest(config, [{ mime, base64 }], safePrompt);
+          blocks.push(`[vlm]\n${vlmText}`);
+        } finally {
+          if (stopAfter && startedByUs) {
+            await stopServer();
+          }
+        }
+      } else if (includeVlm && !vlmAvailable) {
+        blocks.push('[vlm] VLM 未配置（SEE_BASE 环境变量为空）');
+      }
+
+      ctx.emit('fs/observed', target, { kind: 'present', version: info.version }, exec);
+      const combined = blocks.join('\n\n---\n\n');
+      return {
+        path: target.displayPath,
+        lowInformation: false,
+        ...(scanText ? { scan: scanText } : {}),
+        ...(ocrText ? { ocr: ocrText } : {}),
+        ...(vlmText ? { vlm: vlmText } : {}),
+        combined
+      };
+    }
+  };
+}