@liustack/modlens 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Leon Liu (liustack)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,77 @@
+# ModLens
+
+A CLI toolkit for AI agents that converts image sources (local path or remote URL) into structured text evidence, bridging the vision gap for text-only LLM workflows.
+
+[中文说明](README.zh-CN.md)
+
+## Features
+
+- Built for non-vision LLM setups (text-only models + external vision bridge)
+- Supports local image paths and remote image URLs
+- Pluggable vision backend — ships with Gemini CLI; more engines (PaddleOCR, DeepSeek, etc.) planned
+- Outputs machine-consumable JSON (OCR + layout + semantics + visual clues)
+- Designed to be called from Agent Skills (Claude Code, Codex, Cursor, etc.)
+
+## Install
+
+```bash
+npm install -g @liustack/modlens
+```
+
+The default backend requires Gemini CLI to be installed and authenticated:
+
+```bash
+npm install -g @google/gemini-cli
+gemini
+```
+
+Or run with `npx`:
+
+```bash
+npx @liustack/modlens [options]
+```
+
+## Usage
+
+```bash
+# Print JSON result to stdout
+modlens -i screenshot.png
+
+# Save to file
+modlens -i screenshot.png -o lens.json
+
+# Specify model + extra prompt constraints
+modlens -i screenshot.png -m gemini-2.5-flash --prompt "Focus on table structure"
+```
+
+## Options
+
+| Flag | Description |
+|------|-------------|
+| `-i, --input <path>` | Input image path (required) |
+| `-o, --output <path>` | Write result JSON to a file |
+| `-m, --model <name>` | Vision model name (backend-specific) |
+| `--prompt <text>` | Extra extraction constraints |
+| `--timeout <ms>` | Timeout in milliseconds (default: `180000`) |
+| `--gemini-bin <path>` | Gemini CLI binary path (default: `gemini`) |
+
+## Vision Backends
+
+ModLens uses a pluggable architecture for vision recognition. The current v1 ships with **Gemini CLI** as the default backend. Future versions will support additional engines such as PaddleOCR, DeepSeek OCR, and other multimodal/vision-capable models.
+
+## Agent Skill
+
+- [modlens/SKILL.md](skills/modlens/SKILL.md)
+
+## Notes
+
+- `modlens` focuses on visual parsing only.
+- `modsearch` and `modfetch` belong to separate projects and are intentionally out of scope.
+
+## Disclaimer
+
+This project is for **personal learning and experimentation only**. It is not intended for commercial use.
+
+## License
+
+MIT

package/README.zh-CN.md

@@ -0,0 +1,69 @@
+# ModLens
+
+面向 AI Agent 的视觉外挂 CLI，用于把图片来源（本地路径或远程 URL）转成结构化文本证据，补齐纯文本 LLM 的“看图”能力。
+
+## 特性
+
+- 面向“无视觉能力模型”场景（文本模型 + 外挂视觉）
+- 支持本地图片路径与远程图片 URL
+- 基于 Gemini CLI 非交互调用（`gemini -p`）
+- 输出结构化 JSON（OCR + 布局 + 语义 + 视觉线索）
+- 适合作为 Agent Skill 工具被 Claude Code / Codex 等调用
+
+## 安装
+
+```bash
+npm install -g @liustack/modlens
+```
+
+需要先安装并认证 Gemini CLI：
+
+```bash
+npm install -g @google/gemini-cli
+gemini
+```
+
+或直接用 `npx`：
+
+```bash
+npx @liustack/modlens [options]
+```
+
+## 用法
+
+```bash
+# 标准输出 JSON
+modlens -i screenshot.png
+
+# 落盘到文件
+modlens -i screenshot.png -o lens.json
+
+# 指定模型和额外解析要求
+modlens -i screenshot.png -m gemini-2.5-flash --prompt "重点提取表格结构"
+```
+
+## 参数
+
+- `-i, --input <path>` 输入图片路径（必填）
+- `-o, --output <path>` 可选输出 JSON 路径
+- `-m, --model <name>` Gemini 模型名
+- `--prompt <text>` 额外解析约束
+- `--timeout <ms>` 超时毫秒（默认 `180000`）
+- `--gemini-bin <path>` Gemini 可执行路径（默认 `gemini`）
+
+## Agent Skill
+
+- [modlens/SKILL.md](skills/modlens/SKILL.md)
+
+## 说明
+
+- `modlens` 只做视觉解析。
+- `modsearch` / `modfetch` 是其他独立项目，不在本仓库实现。
+
+## 免责声明
+
+本项目仅供**个人学习与实验**使用，请勿用于商业用途。
+
+## License
+
+MIT

package/dist/main.js

@@ -0,0 +1,275 @@
+#!/usr/bin/env node
+import { Command } from "commander";
+import * as fs from "fs";
+import * as path from "path";
+import { spawn } from "child_process";
+function buildVisionPrompt(imageAbsolutePath, extraPrompt) {
+  const basePrompt = `Analyze this image: ${imageAbsolutePath}
+
+You are an image parsing engine.
+Analyze the provided image.
+
+Goal: Convert all image information into structured results consumable by a text-only LLM.
+
+Strict requirements:
+1. Output JSON only. No Markdown, no explanatory text.
+2. Cover all visible text, semantics, structure, layout, and visual clues as thoroughly as possible.
+3. If any information is uncertain, note it in the uncertainty field.
+
+Output JSON structure:
+{
+  "summary": "",
+  "ocr": {
+    "full_text": "",
+    "lines": [
+      {
+        "text": "",
+        "language": "",
+        "confidence": 0
+      }
+    ]
+  },
+  "layout": {
+    "regions": [
+      {
+        "id": "",
+        "type": "title|subtitle|paragraph|list|table|chart|form|image|icon|other",
+        "bbox": { "x": 0, "y": 0, "w": 0, "h": 0 },
+        "reading_order": 1,
+        "text": ""
+      }
+    ]
+  },
+  "semantics": {
+    "scene": "",
+    "intent": "",
+    "entities": [
+      {
+        "name": "",
+        "type": "",
+        "evidence": ""
+      }
+    ],
+    "relations": [
+      {
+        "subject": "",
+        "predicate": "",
+        "object": ""
+      }
+    ]
+  },
+  "visual": {
+    "dominant_colors": [""],
+    "style": "",
+    "notes": [""]
+  },
+  "uncertainty": [""]
+}`;
+  if (!extraPrompt || !extraPrompt.trim()) {
+    return basePrompt;
+  }
+  return `${basePrompt}
+
+Additional requirements:
+${extraPrompt.trim()}`;
+}
+const DEFAULT_TIMEOUT_MS = 18e4;
+function buildGeminiInvocation(options) {
+  const isRemote = isRemoteSource(options.imagePath);
+  const imageSource = isRemote ? options.imagePath.trim() : path.resolve(options.imagePath);
+  const imageDir = path.dirname(imageSource);
+  const prompt = buildVisionPrompt(imageSource, options.extraPrompt);
+  const args = ["-p", prompt, "--output-format", "json"];
+  if (options.model) {
+    args.push("-m", options.model);
+  }
+  return {
+    command: options.geminiBin || "gemini",
+    args,
+    cwd: options.workspaceDir || (isRemote ? process.cwd() : imageDir)
+  };
+}
+function parseGeminiCliJsonOutput(stdout) {
+  let parsed;
+  try {
+    parsed = JSON.parse(stdout.trim());
+  } catch (error) {
+    throw new Error(`Failed to parse Gemini JSON output: ${error.message}`);
+  }
+  if (!parsed || typeof parsed !== "object" || typeof parsed.response !== "string") {
+    throw new Error("Gemini JSON output is missing a string `response` field.");
+  }
+  return parsed;
+}
+function extractStructuredResponse(text) {
+  const rawText = text.trim();
+  const direct = tryParseJson(rawText);
+  if (direct !== null) {
+    return {
+      structured: direct,
+      rawText
+    };
+  }
+  const fencedMatch = /```(?:json)?\s*([\s\S]*?)```/i.exec(rawText);
+  if (fencedMatch) {
+    const parsedFenced = tryParseJson(fencedMatch[1].trim());
+    if (parsedFenced !== null) {
+      return {
+        structured: parsedFenced,
+        rawText
+      };
+    }
+  }
+  const firstBrace = rawText.indexOf("{");
+  const lastBrace = rawText.lastIndexOf("}");
+  if (firstBrace >= 0 && lastBrace > firstBrace) {
+    const possibleJson = rawText.slice(firstBrace, lastBrace + 1);
+    const parsedObject = tryParseJson(possibleJson);
+    if (parsedObject !== null) {
+      return {
+        structured: parsedObject,
+        rawText
+      };
+    }
+  }
+  return {
+    structured: null,
+    rawText
+  };
+}
+async function analyzeImage(options) {
+  const resolvedInput = resolveInput(options.input);
+  if (resolvedInput.kind === "local") {
+    validateInputFile(resolvedInput.source);
+  }
+  const invocation = buildGeminiInvocation({
+    imagePath: resolvedInput.source,
+    model: options.model,
+    geminiBin: options.geminiBin,
+    workspaceDir: options.workspaceDir,
+    extraPrompt: options.prompt
+  });
+  const commandResult = await runCommand(invocation, options.timeoutMs ?? DEFAULT_TIMEOUT_MS);
+  const geminiOutput = parseGeminiCliJsonOutput(commandResult.stdout);
+  const extracted = extractStructuredResponse(geminiOutput.response);
+  return {
+    image: resolvedInput.source,
+    structured: extracted.structured,
+    rawText: extracted.rawText,
+    meta: {
+      generatedAt: (/* @__PURE__ */ new Date()).toISOString(),
+      model: options.model ?? null,
+      geminiSessionId: geminiOutput.session_id ?? null,
+      geminiStats: geminiOutput.stats ?? null
+    }
+  };
+}
+function resolveInput(input) {
+  const trimmed = input.trim();
+  if (!trimmed) {
+    throw new Error("Input path is required.");
+  }
+  if (isRemoteSource(trimmed)) {
+    return { source: trimmed, kind: "remote" };
+  }
+  if (/^file:\/\//i.test(trimmed)) {
+    const localPath = decodeURI(trimmed.replace(/^file:\/\//i, ""));
+    return { source: path.resolve(localPath), kind: "local" };
+  }
+  return { source: path.resolve(trimmed), kind: "local" };
+}
+function isRemoteSource(value) {
+  return /^https?:\/\//i.test(value.trim());
+}
+function validateInputFile(filePath) {
+  if (!fs.existsSync(filePath)) {
+    throw new Error(`Input image not found: ${filePath}`);
+  }
+  const stat = fs.statSync(filePath);
+  if (!stat.isFile()) {
+    throw new Error(`Input is not a file: ${filePath}`);
+  }
+}
+function tryParseJson(text) {
+  try {
+    return JSON.parse(text);
+  } catch {
+    return null;
+  }
+}
+function runCommand(invocation, timeoutMs) {
+  return new Promise((resolve, reject) => {
+    const child = spawn(invocation.command, invocation.args, {
+      cwd: invocation.cwd,
+      stdio: ["ignore", "pipe", "pipe"]
+    });
+    let stdout = "";
+    let stderr = "";
+    let timedOut = false;
+    const timer = setTimeout(() => {
+      timedOut = true;
+      child.kill("SIGTERM");
+    }, timeoutMs);
+    child.stdout.on("data", (chunk) => {
+      stdout += chunk.toString();
+    });
+    child.stderr.on("data", (chunk) => {
+      stderr += chunk.toString();
+    });
+    child.on("error", (error) => {
+      clearTimeout(timer);
+      if (error.code === "ENOENT") {
+        reject(new Error(`Gemini CLI not found: ${invocation.command}`));
+        return;
+      }
+      reject(error);
+    });
+    child.on("close", (code) => {
+      clearTimeout(timer);
+      if (timedOut) {
+        reject(new Error(`Gemini CLI timed out after ${timeoutMs} ms.`));
+        return;
+      }
+      if (code !== 0) {
+        reject(
+          new Error(
+            `Gemini CLI failed with code ${code}.${stderr ? ` stderr: ${stderr.trim()}` : ""}`
+          )
+        );
+        return;
+      }
+      resolve({ stdout, stderr });
+    });
+  });
+}
+const program = new Command();
+program.name("modlens").description("Image-to-text visual bridge for non-vision LLM workflows").version("0.1.0").requiredOption("-i, --input <path>", "Input image path").option("-o, --output <path>", "Write result JSON to a file").option("-m, --model <name>", "Gemini model name").option("--prompt <text>", "Extra prompt constraints for this image").option("--timeout <ms>", "Command timeout in milliseconds", "180000").option("--gemini-bin <path>", "Gemini CLI binary path", "gemini").action(async (options) => {
+  try {
+    const timeoutMs = Number.parseInt(options.timeout, 10);
+    if (!Number.isFinite(timeoutMs) || timeoutMs <= 0) {
+      throw new Error("Invalid --timeout. Use a positive integer in milliseconds.");
+    }
+    const result = await analyzeImage({
+      input: options.input,
+      model: options.model,
+      prompt: options.prompt,
+      timeoutMs,
+      geminiBin: options.geminiBin
+    });
+    const output = JSON.stringify(result, null, 2);
+    if (options.output) {
+      const outputPath = path.resolve(options.output);
+      fs.mkdirSync(path.dirname(outputPath), { recursive: true });
+      fs.writeFileSync(outputPath, output, "utf-8");
+    }
+    process.stdout.write(`${output}
+`);
+  } catch (error) {
+    process.stderr.write(
+      `Error: ${error instanceof Error ? error.message : String(error)}
+`
+    );
+    process.exit(1);
+  }
+});
+program.parse();

package/package.json

@@ -0,0 +1,54 @@
+{
+  "name": "@liustack/modlens",
+  "version": "0.1.0",
+  "description": "CLI tool to provide visual understanding for non-vision LLMs",
+  "type": "module",
+  "bin": {
+    "modlens": "./dist/main.js"
+  },
+  "files": [
+    "dist",
+    "skills/modlens/SKILL.md",
+    "skills/modlens/references"
+  ],
+  "keywords": [
+    "cli",
+    "vision",
+    "ocr",
+    "gemini",
+    "claude-code",
+    "skill",
+    "image-to-text",
+    "multimodal",
+    "modlens"
+  ],
+  "author": "Leon Liu",
+  "license": "MIT",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/liustack/modlens.git"
+  },
+  "bugs": {
+    "url": "https://github.com/liustack/modlens/issues"
+  },
+  "homepage": "https://github.com/liustack/modlens#readme",
+  "engines": {
+    "node": ">=18"
+  },
+  "dependencies": {
+    "commander": "^13.1.0"
+  },
+  "devDependencies": {
+    "@types/node": "^22.19.7",
+    "typescript": "^5.9.3",
+    "vite": "^6.4.1",
+    "vitest": "^3.2.4"
+  },
+  "scripts": {
+    "dev": "vite build --watch",
+    "build": "vite build",
+    "typecheck": "tsc --noEmit",
+    "test": "vitest run",
+    "docs:list": "node scripts/docs-list.js"
+  }
+}

package/skills/modlens/SKILL.md

@@ -0,0 +1,75 @@
+---
+name: modlens
+description: "Bridge image understanding for non-vision LLM workflows. Use when user provides screenshots/photos/charts/doc images and the active model lacks multimodal vision. Call modlens to extract OCR text, semantics, structure, and layout as JSON evidence."
+allowed-tools:
+  - Bash
+---
+
+# ModLens — Vision Bridge Skill
+
+Use this skill when:
+- User asks to analyze an image/screenshot/chart/document photo
+- Current model cannot directly read images
+- You need structured visual evidence before downstream reasoning
+
+Do not use this skill for:
+- Web search (`modsearch`)
+- Web fetch (`modfetch`)
+
+## Prerequisites
+
+```bash
+modlens --version
+```
+
+The default vision backend (Gemini CLI) must also be installed and authenticated:
+
+```bash
+gemini --version
+```
+
+If `gemini` is missing:
+
+```bash
+npm install -g @google/gemini-cli
+gemini
+```
+
+## Command
+
+```bash
+modlens -i <image-path>
+```
+
+Optional:
+
+```bash
+modlens -i <image-path> -o <output-json-path> -m <model-name> --prompt "<extra constraints>"
+```
+
+## Workflow
+
+1. If user includes one or more images, run `modlens` for each image.
+2. Parse returned JSON.
+3. Feed `summary`, `ocr`, `layout`, and `semantics` back into your reasoning context.
+4. If confidence is low or uncertainty is high, tell user what is ambiguous.
+
+## Output Contract
+
+- `summary`: high-level description
+- `ocr.full_text` + `ocr.lines`: extracted text evidence
+- `layout.regions`: structural/layout blocks with reading order
+- `semantics`: entities, scene, intent, relations
+- `visual`: color/style clues
+- `uncertainty`: uncertain points
+
+Detailed schema: `references/output-schema.md`
+
+## Failure Handling
+
+- If command fails due to missing auth or quota, report exact error and ask user to check backend setup (e.g., run `gemini` for Gemini CLI login).
+- If JSON is partially malformed, keep raw text and continue with best-effort extraction.
+
+## Implementation Note
+
+v1 uses Gemini CLI as the default vision backend (`gemini -p` with JSON output mode). The architecture is designed to support additional vision engines (PaddleOCR, DeepSeek OCR, etc.) in future versions.

package/skills/modlens/references/output-schema.md

@@ -0,0 +1,57 @@
+# ModLens Output Schema (v1)
+
+```json
+{
+  "summary": "string",
+  "ocr": {
+    "full_text": "string",
+    "lines": [
+      {
+        "text": "string",
+        "language": "string",
+        "confidence": 0
+      }
+    ]
+  },
+  "layout": {
+    "regions": [
+      {
+        "id": "string",
+        "type": "title|subtitle|paragraph|list|table|chart|form|image|icon|other",
+        "bbox": { "x": 0, "y": 0, "w": 0, "h": 0 },
+        "reading_order": 1,
+        "text": "string"
+      }
+    ]
+  },
+  "semantics": {
+    "scene": "string",
+    "intent": "string",
+    "entities": [
+      {
+        "name": "string",
+        "type": "string",
+        "evidence": "string"
+      }
+    ],
+    "relations": [
+      {
+        "subject": "string",
+        "predicate": "string",
+        "object": "string"
+      }
+    ]
+  },
+  "visual": {
+    "dominant_colors": ["string"],
+    "style": "string",
+    "notes": ["string"]
+  },
+  "uncertainty": ["string"]
+}
+```
+
+Notes:
+- `confidence` is 0-1 numeric estimate.
+- `bbox` uses image-relative coordinates.
+- If a field is unavailable, return empty string/array and explain in `uncertainty`.