mimo2codex 0.1.15 → 0.1.16

package/mimoskill/references/ocr_workflow.md

@@ -0,0 +1,216 @@
+# OCR / image recognition workflow
+
+`mimoskill/scripts/ocr.py` is the fallback path for reading or describing
+images when the surrounding chat model can't see them. It always calls
+`mimo-v2.5` (MiMo's vision-capable model) internally, regardless of which
+model the rest of the conversation is using.
+
+## TL;DR
+
+```bash
+export MIMO_API_KEY=sk-xxxxxxxxxxxxxxxx
+
+# default mode (text) — verbatim OCR
+python3 mimoskill/scripts/ocr.py path/to/image.png
+
+# describe the image in 2-4 sentences
+python3 mimoskill/scripts/ocr.py --mode describe path/to/image.png
+
+# structured JSON (text + regions + language + summary)
+python3 mimoskill/scripts/ocr.py --mode structured a.png b.jpg
+
+# re-render as GitHub-flavored Markdown
+python3 mimoskill/scripts/ocr.py --mode markdown form.png
+```
+
+## Why this skill exists
+
+The proxy strips image attachments when the active chat model can't accept
+them (`src/translate/reqToChat.ts:48-72`). Non-vision MiMo variants —
+`mimo-v2.5-pro`, `mimo-v2.5-pro[1m]`, `mimo-v2-flash` — return 404
+"No endpoints found that support image input" if images are forwarded.
+The proxy drops the images and leaves an `[N image attachment(s) omitted: …]`
+placeholder so the conversation doesn't crash.
+
+`ocr.py` is the recommended way to recover that content **without changing
+the chat model**: it independently calls `mimo-v2.5`, returns text, and the
+caller pipes that text back into the conversation as a normal user message.
+
+## Input modes
+
+The positional `IMAGE` args (0 or more) accept:
+
+| Form | Example | What ocr.py does |
+|---|---|---|
+| Local path | `./scan.png`, `C:\foo.jpg` | reads bytes, magic-byte sniffs MIME, base64-encodes to a `data:` URL |
+| `http(s)://` URL | `https://example.com/x.png` | forwarded as-is; MiMo fetches server-side |
+| `data:` URL | `data:image/png;base64,…` | forwarded as-is |
+| `-` (single dash) | piped from stdin | reads one image's bytes from stdin |
+| nothing + non-TTY stdin | `cat x.png \| ocr.py` | same as `-` |
+
+Magic-byte table (file extension is **not** trusted):
+
+| Bytes | MIME |
+|---|---|
+| `89 50 4E 47 0D 0A 1A 0A` | `image/png` |
+| `FF D8 FF` | `image/jpeg` |
+| `47 49 46 38 37 61` / `…39 61` | `image/gif` |
+| `52 49 46 46 …. 57 45 42 50` | `image/webp` |
+| `42 4D` | `image/bmp` |
+| (anything else) | falls back to `image/png` |
+
+## Output modes (`--mode`)
+
+### `text` (default) — verbatim OCR
+
+```bash
+python3 mimoskill/scripts/ocr.py invoice.png
+```
+
+Stdout is the raw extracted text. Line breaks, reading order, and rough
+column/table layout (whitespace + pipes) are preserved. No commentary, no
+translation, no summary. Unreadable spans become `[unreadable]`. Image with
+no text returns the single line `[no text detected]`.
+
+### `describe` — short prose description
+
+```bash
+python3 mimoskill/scripts/ocr.py --mode describe screenshot.png
+```
+
+2-4 sentences covering layout, key elements, visible text (quoted), and
+notable colors. No invented details.
+
+### `structured` — JSON
+
+```bash
+python3 mimoskill/scripts/ocr.py --mode structured form.png
+```
+
+Stdout is a single JSON object:
+
+```json
+{
+  "text": "...",
+  "language": "zh-Hans",
+  "regions": [
+    {"label": "title", "text": "增值税电子发票", "role": "title"},
+    {"label": "buyer", "text": "...", "role": "paragraph"},
+    {"label": "items", "text": "...", "role": "table"}
+  ],
+  "summary": "A Chinese VAT e-invoice with buyer/seller and four line items."
+}
+```
+
+`regions[].role` is one of `title`, `paragraph`, `list`, `table`, `caption`,
+`ui`, `handwriting`, `other`.
+
+**Note**: `structured` returns **logical regions** (role classification),
+not pixel bounding boxes. MiMo does not currently expose grounded pixel
+coordinates the way some other vision models do; this skill won't pretend
+to. If you need pixel boxes, use a model that does (e.g. Gemini grounding,
+Tesseract with `--psm 6` + position data).
+
+### `markdown` — re-render as GFM
+
+```bash
+python3 mimoskill/scripts/ocr.py --mode markdown spec.png
+```
+
+Headings become `#`/`##`, tables become pipe tables, code-like text becomes
+fenced blocks, lists become `-`. Reading order preserved. Output is the
+Markdown body only — no preamble, no outer fence.
+
+## Batch (multi-image) calls
+
+Pass multiple positional args:
+
+```bash
+python3 mimoskill/scripts/ocr.py page1.png page2.png page3.png
+```
+
+All images go to MiMo in a **single** chat completion (one billable call).
+The model can cross-reference (e.g. ID front + back). Output is a single
+text body in reading order across the images.
+
+When you need a different prompt per image, run `ocr.py` N times instead.
+
+## `--lang` and `--prompt`
+
+- `--lang LANG` appends `Primary language: <LANG>.` to the prompt. Useful
+  for CJK to prevent the model from outputting Pinyin transliteration:
+  `ocr.py --lang Chinese scan.png` or `--lang zh` or `--lang 日本語`.
+
+- `--prompt EXTRA` appends a free-text instruction:
+  `ocr.py --mode text --prompt "Only handwriting, ignore printed text." form.png`
+
+## Model selection
+
+| You pass | ocr.py uses |
+|---|---|
+| nothing | `$MIMO_OCR_MODEL` → `$MIMO_MODEL` (if vision-capable) → `mimo-v2.5` |
+| `--model mimo-v2.5` | `mimo-v2.5` |
+| `--model mimo-v2.5[1m]` | `mimo-v2.5[1m]` |
+| `--model mimo-v2-omni` | `mimo-v2-omni` |
+| `--model mimo-v2.5-pro` | **switches to `mimo-v2.5`** (stderr note) |
+| `--model mimo-v2.5-pro[1m]` | **switches to `mimo-v2.5`** |
+| `--model mimo-v2-flash` | **switches to `mimo-v2.5`** |
+
+Non-vision models would return 404 from MiMo, so the script coerces them
+silently (one stderr line) rather than failing.
+
+## When `MIMO_API_KEY` isn't set
+
+`ocr.py` exits with code `3` and this stderr message:
+
+```
+error: MIMO_API_KEY is not set; ocr.py needs MiMo V2.5 vision to read images.
+  set one at https://platform.xiaomimimo.com/#/console/api-keys
+  OR if you want fully-local OCR with no API key, install tesseract:
+      macOS:    brew install tesseract tesseract-lang
+      Ubuntu:   sudo apt install tesseract-ocr tesseract-ocr-chi-sim
+      Windows:  https://github.com/UB-Mannheim/tesseract/wiki
+    then run: tesseract <image> - -l eng+chi_sim
+  (tesseract is NOT installed or invoked by this skill; this is just a pointer.)
+```
+
+The tesseract pointer is **just a pointer** — this skill never auto-shells
+to it. Keeps the dependency surface predictable.
+
+## Common pitfalls
+
+- **PDFs are not supported directly.** Rasterize first with one of:
+  - `pdftoppm -png input.pdf out` (Poppler)
+  - `mutool draw -o out-%d.png input.pdf` (MuPDF)
+  - macOS: `sips -s format png input.pdf --out out.png`
+- **Multi-image batches share one prompt.** If you need different modes /
+  languages per image, invoke `ocr.py` once per image.
+- **`structured` mode is logical regions, not pixel boxes.** See above.
+- **`--stream` + `structured`**: the streamed body is still a single JSON
+  object; buffer it before parsing.
+
+## Exit codes
+
+| Code | Meaning |
+|---|---|
+| 0 | Success |
+| 1 | MiMo HTTP error (error body printed to stderr) |
+| 2 | argv / usage error (no image, mutually exclusive flags, etc.) |
+| 3 | `MIMO_API_KEY` not set |
+| 4 | Local image file not found / unreadable |
+
+## Composing with `mimo_chat.py`
+
+OCR + downstream LLM call is a common pattern:
+
+```bash
+TEXT=$(python3 mimoskill/scripts/ocr.py invoice.png)
+python3 mimoskill/scripts/mimo_chat.py "Summarize this invoice:\n$TEXT"
+```
+
+Or structured + parse:
+
+```bash
+JSON=$(python3 mimoskill/scripts/ocr.py --mode structured invoice.png)
+echo "$JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['summary'])"
+```

package/mimoskill/scripts/generate_image.py

@@ -0,0 +1,163 @@
+#!/usr/bin/env python3
+"""
+generate_image.py — general (non-pet) image generation.
+
+Thin wrapper over generate_pet.py: same providers (auto / pollinations /
+gpt-image-1 / replicate / local-sd), no chibi-pet prompt boilerplate,
+plus an optional --style for common looks.
+
+For Codex /hatch pets, keep using generate_pet.py — it has pet-tuned prompt
+prefixes and the --bundle (idle/working/done) state machine.
+
+Usage:
+    # free, no key
+    python3 generate_image.py --prompt "isometric cyberpunk city at dusk" --out out.png
+
+    # styled
+    python3 generate_image.py --style pixel-art --prompt "a brave knight" --out k.png
+
+    # best quality (needs PET_OPENAI_API_KEY — same env var as the pet flow)
+    python3 generate_image.py --provider gpt-image-1 --prompt "..." --out out.png
+
+    # multiple variants
+    python3 generate_image.py --n 4 --prompt "..." --out img.png
+    # produces img-1.png, img-2.png, img-3.png, img-4.png
+
+Only depends on the standard library.
+"""
+from __future__ import annotations
+
+import argparse
+import importlib.util
+import sys
+from pathlib import Path
+
+
+# Load generate_pet.py as a module by absolute path (not `import generate_pet`)
+# — the skill is invoked from arbitrary cwd, and we don't ship an __init__.py.
+_HERE = Path(__file__).resolve().parent
+_GP_PATH = _HERE / "generate_pet.py"
+_spec = importlib.util.spec_from_file_location("_generate_pet", _GP_PATH)
+if _spec is None or _spec.loader is None:
+    sys.stderr.write(f"error: cannot load {_GP_PATH}\n")
+    sys.exit(2)
+_gp = importlib.util.module_from_spec(_spec)
+_spec.loader.exec_module(_gp)
+
+
+# --- style presets ----------------------------------------------------------
+
+STYLES: dict[str, tuple[str, str]] = {
+    "plain": ("", ""),
+    "pixel-art": (
+        "Retro 16-bit pixel art sprite of ",
+        ", transparent background, single sprite, nearest-neighbor",
+    ),
+    "photo": (
+        "Photorealistic photograph of ",
+        ", natural lighting, sharp focus, shallow depth of field",
+    ),
+    "3d-render": (
+        "Cute 3D render of ",
+        ", soft global illumination, octane render",
+    ),
+    "line-art": (
+        "Black ink line art of ",
+        ", clean linework, white background, no shading",
+    ),
+    "watercolor": (
+        "Hand-drawn ink and watercolor of ",
+        ", loose linework, watercolor wash",
+    ),
+    "sticker": (
+        "Chibi sticker mascot of ",
+        ", transparent background, soft cel-shading, single character",
+    ),
+}
+
+
+def apply_style(prompt: str, style: str) -> str:
+    prefix, suffix = STYLES[style]
+    if not prefix and not suffix:
+        return prompt
+    body = prompt.strip().rstrip(".,;")
+    return f"{prefix}{body}{suffix}"
+
+
+# --- main -------------------------------------------------------------------
+
+def main() -> None:
+    p = argparse.ArgumentParser(
+        description=__doc__.split("\n", 1)[0],
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+    )
+    p.add_argument("--prompt", required=True, help="what to draw (used verbatim by default)")
+    p.add_argument(
+        "--style",
+        choices=list(STYLES),
+        default="plain",
+        help="optional prompt preset (default: plain — no prefix/suffix)",
+    )
+    p.add_argument(
+        "--provider",
+        choices=["auto"] + list(_gp.PROVIDERS),
+        default="auto",
+        help="image gen backend (same as generate_pet.py)",
+    )
+    p.add_argument("--reference", type=Path, help="reference image (gpt-image-1 only)")
+    p.add_argument("--quality", default="medium", choices=["low", "medium", "high", "hd"])
+    p.add_argument("--out", type=Path, required=True, help="output path (PNG)")
+    p.add_argument("--n", type=int, default=1, help="number of variants to generate")
+    p.add_argument("--size", default=None, help="forwarded where supported (e.g. 1024x1024)")
+    p.add_argument("--seed", type=int, default=None, help="forwarded where supported")
+    args = p.parse_args()
+
+    if args.n < 1:
+        sys.stderr.write("error: --n must be >= 1\n")
+        sys.exit(2)
+
+    # Resolve auto provider with the same status line generate_pet.py emits.
+    if args.provider == "auto":
+        chosen = _gp.resolve_auto_provider()
+        if chosen == "pollinations":
+            sys.stderr.write(
+                "[provider] auto -> pollinations (free, no key required).\n"
+                "           For higher quality, set PET_OPENAI_API_KEY (real OpenAI key)\n"
+                "           and rerun, or pass --provider replicate / local-sd.\n\n"
+            )
+        else:
+            sys.stderr.write(f"[provider] auto -> {chosen}\n\n")
+        args.provider = chosen
+
+    final_prompt = apply_style(args.prompt, args.style)
+    sys.stderr.write(f"prompt: {final_prompt}\n")
+
+    # --size / --seed: emit a note where the underlying provider doesn't
+    # plumb them through. v1 forwards nothing (generate_pet.py hard-codes
+    # 1024x1024 / no seed); future versions can extend per-provider.
+    if args.size and args.size != "1024x1024":
+        sys.stderr.write(
+            f"note: --size {args.size} ignored in v1 (providers run at 1024x1024).\n"
+        )
+    if args.seed is not None:
+        sys.stderr.write(
+            "note: --seed ignored in v1 (not plumbed through to providers yet).\n"
+        )
+
+    def out_path_for(i: int) -> Path:
+        if args.n == 1:
+            return args.out
+        stem = args.out.stem
+        suffix = args.out.suffix or ".png"
+        return args.out.parent / f"{stem}-{i + 1}{suffix}"
+
+    for i in range(args.n):
+        out = out_path_for(i)
+        sys.stderr.write(f"generating ({i + 1}/{args.n}) -> {out}\n")
+        _gp.generate_one(args.provider, final_prompt, args.reference, args.quality, out)
+
+    sys.stderr.write(f"\n[ok] wrote {args.n} image(s)\n")
+
+
+if __name__ == "__main__":
+    main()