pi-simocracy 0.3.0 → 0.4.1

package/README.md

@@ -1,18 +1,28 @@
 # pi-simocracy
 
 Load a [Simocracy](https://simocracy.org) sim into your [`pi`](https://github.com/mariozechner/pi-coding-agent) chat — see its
-pixel-art sprite render in the terminal and chat with the agent **as that sim**.
+sprite render in the terminal and chat with the agent **as that sim**.
 
 ```
 /sim mr meow
 ```
 
 …fetches Mr Meow from Simocracy's ATProto indexer, renders his 32×32
-sprite as colored ANSI half-blocks directly in the chat, and pushes
-his constitution + speaking style into pi's system prompt so pi
+pixel-art sprite as colored ANSI half-blocks directly in the chat, and
+pushes his constitution + speaking style into pi's system prompt so pi
 roleplays as Mr Meow until you `/sim unload`.
 
-![Mr Meow loaded inline in pi](demo/sim-load.gif)
+<p align="center">
+  <img src="https://raw.githubusercontent.com/GainForest/pi-simocracy/main/demo/sim-load.gif" alt="Mr Meow (a pipoya pixel-art cat) loaded inline in pi's chat" width="760">
+</p>
+
+As of v0.4.0, codex pet sims (OpenAI hatch-pet skill output) render too —
+load **Einstein** with `/sim einstein` and the WebP atlas decodes, the
+idle frame crops, and the half-block ANSI render fires inline:
+
+<p align="center">
+  <img src="https://raw.githubusercontent.com/GainForest/pi-simocracy/main/demo/codex-pet-load.gif" alt="Einstein (a codex pet sim with a WebP petSheet) loaded inline in pi's chat" width="760">
+</p>
 
 ---
 
@@ -109,11 +119,18 @@ The same actions are exposed to pi as tools, so the model can drive them itself:
    - `org.simocracy.sim`     — display name + sprite + avatar blob refs
    - `org.simocracy.agents`  — short description + full constitution
    - `org.simocracy.style`   — speaking style / mannerisms
-4. **Render.** Fetch the sprite-sheet blob (128×128 PNG, 4×4 of 32×32
-   walking frames) via `com.atproto.sync.getBlob`, decode with `pngjs`,
-   crop the front-facing walk-1 frame, emit as 24-bit ANSI using the
-   upper-half-block character `▀` so each terminal cell paints two
-   pixels. Transparent regions show pi's background through.
+4. **Render.** Fetch the sprite blob via `com.atproto.sync.getBlob`,
+   decode it, crop the front-facing idle frame, and emit as 24-bit ANSI
+   using the upper-half-block character `▀` so each terminal cell paints
+   two pixels. Two render paths depending on the sim's `spriteKind`:
+   - **`pipoya`** (legacy + default): 128×128 PNG, 4×4 of 32×32 walking
+     frames; decode with `pngjs`, take row 0 col 0 at native size.
+   - **`codexPet`** (OpenAI hatch-pet output): 1536×1872 atlas, 8×9 of
+     192×208 cells. PNG sheets decode through `pngjs`; WebP sheets
+     decode through `@jsquash/webp` (wasm, lazy-init). The idle cell
+     (row 0 col 0) is box-downscaled to ~32 wide so the inline render
+     stays comparable in height to a pipoya sprite.
+   Transparent regions show pi's background through.
 5. **Inject.** A `before_agent_start` event handler appends the sim's
    identity + constitution + speaking style to pi's system prompt **every
    turn**. After `/sim unload`, a one-shot override fires on the next
@@ -129,15 +146,17 @@ keeps the terminal it's already running in.
 
 ```
 src/
-├── index.ts        # extension entry: slash command, tools, persona injection
-├── persona.ts      # buildSimPrompt(sim) — the system-prompt fragment
-├── simocracy.ts    # indexer + PDS client (read-only fetchers)
-├── writes.ts       # PDS writers + ownership / sign-in preconditions
-├── png-to-ansi.ts  # RGBA half-block ANSI renderer
-├── openrouter.ts   # minimal OpenRouter client (only used by simocracy_chat)
-└── auth/           # ATProto OAuth loopback flow + session storage
+├── index.ts          # extension entry: slash command, tools, persona injection
+├── persona.ts        # buildSimPrompt(sim) — the system-prompt fragment
+├── simocracy.ts      # indexer + PDS client (read-only fetchers)
+├── writes.ts         # PDS writers + ownership / sign-in preconditions
+├── png-to-ansi.ts    # RGBA half-block ANSI renderer + downscalers
+├── webp-to-rgba.ts   # @jsquash/webp wrapper for codex pet WebP sheets
+├── openrouter.ts     # minimal OpenRouter client (only used by simocracy_chat)
+└── auth/             # ATProto OAuth loopback flow + session storage
 demo/
-└── sim-load.tape   # vhs tape — render with `vhs demo/sim-load.tape`
+├── sim-load.tape       # vhs tape — Mr Meow (pipoya)
+└── codex-pet-load.tape # vhs tape — Einstein (codex pet)
 ```
 
 ---
@@ -153,11 +172,12 @@ pi -e $(pwd)/src/index.ts -ne -ns  # load the extension directly
 
 Then in `pi`: `/sim mr meow`.
 
-To rebuild the demo recording:
+To rebuild the demo recordings:
 
 ```bash
 brew install vhs                   # one-time
-vhs demo/sim-load.tape             # writes demo/sim-load.{webm,gif}
+vhs demo/sim-load.tape             # Mr Meow (pipoya) — demo/sim-load.{webm,gif}
+vhs demo/codex-pet-load.tape       # Einstein (codex pet) — demo/codex-pet-load.{webm,gif}
 ```
 
 ---
@@ -172,7 +192,11 @@ These come bundled with `pi` itself, so installing pi-simocracy via
 
 Direct npm dependencies (auto-installed):
 
-- `pngjs` — PNG decoder for sprite blobs
+- `pngjs` — PNG decoder for pipoya sprite blobs and codex pet PNG sheets
+- `@jsquash/webp` — wasm WebP decoder for codex pet WebP sheets
+  (lazy-init, no native bindings)
+- `@atproto/api` + `@atproto/oauth-client-node` — ATProto loopback OAuth
+  for `/sim login` and PDS writes via `simocracy_update_sim`
 - `typebox` — tool parameter schemas
 
 ---
@@ -192,4 +216,4 @@ Direct npm dependencies (auto-installed):
 
 ## License
 
-MIT — see [LICENSE](./LICENSE).
+MIT — see [LICENSE](https://github.com/GainForest/pi-simocracy/blob/main/LICENSE).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-simocracy",
-  "version": "0.3.0",
+  "version": "0.4.1",
   "description": "Pi extension: load a Simocracy sim into your chat — see its pixel-art sprite render inline in the terminal and roleplay with it.",
   "type": "module",
   "author": "David Dao <david@gainforest.earth> (https://github.com/daviddao)",
@@ -45,6 +45,7 @@
   "dependencies": {
     "@atproto/api": "^0.19.11",
     "@atproto/oauth-client-node": "^0.3.17",
+    "@jsquash/webp": "^1.5.0",
     "pngjs": "^7.0.0",
     "typebox": "^1.1.24"
   },

package/src/index.ts

@@ -81,7 +81,9 @@ import {
   cropRgba,
   detectPixelArtScale,
   downscaleRgbaNearest,
+  boxDownscaleRgba,
 } from "./png-to-ansi.ts";
+import { decodeWebp } from "./webp-to-rgba.ts";
 import { openRouterComplete, type ChatMessage } from "./openrouter.ts";
 import { buildSimPrompt, type LoadedSim } from "./persona.ts";
 import { runLogin, runLogout, runWhoami } from "./auth/commands.ts";
@@ -124,19 +126,43 @@ function blobLink(ref: unknown): string | null {
 }
 
 /**
- * Render a sim's sprite at its native 32×32 size as colored ANSI
- * half-block art — 16 cells tall, 32 cells wide. Compact enough to fit
- * comfortably in a terminal alongside the loaded-sim message.
+ * Maximum long-edge of a non-pixel-art avatar before we box-downscale it.
+ * Picked so codex pet idle frames render at roughly the same on-screen
+ * height as a 32×32 pipoya sprite (~17 terminal lines) instead of
+ * ballooning to 60+ lines and pushing the rest of chat off-screen.
+ */
+const NON_PIXEL_ART_TARGET_LONG_EDGE = 40;
+
+/**
+ * Render a sim's avatar as colored ANSI half-block art, sized to fit
+ * comfortably alongside the loaded-sim message in a typical terminal.
+ *
+ * Branches on `spriteKind` (lexicon discriminator):
+ *
+ *   pipoya   — fetch the 4×4 walk-sheet PNG, crop the front-facing
+ *              walk-1 frame (32×32, row 0 col 0), render at native
+ *              resolution. ~16 lines tall.
  *
- * Pulls the front-facing walk1 frame (row 0, col 0) from the 128×128
- * sprite-sheet blob. Falls back to the static avatar PNG if no sheet
- * is published for this sim.
+ *   codexPet — prefer the 8×9 atlas (`petSheet`, 1536×1872, 192×208
+ *              cells) when it's PNG: crop the idle cell (row 0 col 0)
+ *              and box-downscale to ~32 wide. The atlas is usually WebP
+ *              (the OpenAI hatch-pet skill emits WebP and `pngjs`
+ *              doesn't speak WebP), so we fall through to the rendered
+ *              `image` thumbnail — a PNG that the simocracy.org client
+ *              generates at 128×128 from the idle frame. That gets
+ *              box-downscaled to a comparable terminal size.
+ *
+ *   absent   — treated as legacy 'pipoya' for back-compat.
  */
 async function renderSpriteAnsi(sim: SimMatch): Promise<string | null> {
+  const spriteKind = sim.sim.spriteKind ?? "pipoya";
   const spriteLink = blobLink(sim.sim.sprite?.ref);
+  const petSheetLink = blobLink(sim.sim.petSheet?.ref);
+  const petSheetMime = sim.sim.petSheet?.mimeType;
   const imageLink = blobLink(sim.sim.image?.ref);
 
-  if (spriteLink) {
+  // Pipoya 4×4 walk sheet — the legacy/default path.
+  if (spriteKind !== "codexPet" && spriteLink) {
     try {
       const buf = await fetchBlob(sim.did, spriteLink);
       const { width, height, data } = decodePng(buf);
@@ -158,23 +184,74 @@ async function renderSpriteAnsi(sim: SimMatch): Promise<string | null> {
         alphaThreshold: 16,
       });
     } catch {
-      /* fall through to avatar */
+      /* fall through to image fallback */
     }
   }
 
+  // Codex pet atlas — the canonical asset for codex pets. Sheets are
+  // 1536×1872 with 192×208 cells laid out 8 cols × 9 rows; row 0 col 0
+  // is the idle frame. Both PNG and WebP are valid in the lexicon (the
+  // hatch-pet skill emits WebP, the dropzone preserves PNG when the
+  // user drops a PNG sheet) so we pick the right decoder by mimeType.
+  // We crop the idle cell first thing and box-downscale to ~32 wide,
+  // so the inline render is similar in height to a pipoya sprite
+  // (~17 lines).
+  if (spriteKind === "codexPet" && petSheetLink) {
+    try {
+      const buf = await fetchBlob(sim.did, petSheetLink);
+      const { width, height, data } =
+        petSheetMime === "image/webp" ? await decodeWebp(buf) : decodePng(buf);
+      const CELL_W = 192;
+      const CELL_H = 208;
+      if (width >= CELL_W && height >= CELL_H) {
+        const cell = cropRgba(data, width, height, 0, 0, CELL_W, CELL_H);
+        const targetW = 32;
+        const targetH = Math.round((CELL_H / CELL_W) * targetW); // ~35
+        const scaled = boxDownscaleRgba(cell, CELL_W, CELL_H, targetW, targetH);
+        return renderRgbaToAnsi(scaled.data, scaled.width, scaled.height, {
+          cropToContent: true,
+          cropPad: 1,
+          indent: 2,
+          alphaThreshold: 16,
+        });
+      }
+    } catch {
+      /* fall through to image fallback */
+    }
+  }
+
+  // Image fallback — used when a sim has no walk sheet (legacy pipoya
+  // sims that pre-date `org.simocracy.sim.sprite`) or when the codex
+  // pet atlas decode failed for any reason. The simocracy.org client
+  // always generates a 128×128 PNG idle thumbnail and uploads it as
+  // `image` for codex pets, so even when this path runs the right pose
+  // shows up.
   if (imageLink) {
     try {
       const buf = await fetchBlob(sim.did, imageLink);
       const { width, height, data } = decodePng(buf);
-      // Old sims (pre-sprite-sheet) only have the avatar PNG, which
-      // simocracy.org renders by 4×-upscaling a native 32×32 sprite into
-      // a 128×128 image with nearest-neighbour. Detect that and downsample
-      // back to the original size so the inline render is the same
-      // ~13-line height as a sprite-sheet-equipped sim instead of
-      // ballooning to ~22 lines and pushing chat off-screen.
+      // Old pipoya sims publish a 128×128 PNG that's a 4×-upscaled 32×32
+      // pixel-art sprite. Detect and undo that with nearest-neighbour
+      // downsampling.
       const scale = detectPixelArtScale(data, width, height, 8);
-      const native =
-        scale > 1 ? downscaleRgbaNearest(data, width, height, scale) : { data, width, height };
+      let native: { data: Buffer; width: number; height: number } =
+        scale > 1
+          ? downscaleRgbaNearest(data, width, height, scale)
+          : { data, width, height };
+      // Non-pixel-art images (codex pet thumbnails, avatars from other
+      // pipelines) come through at full resolution — box-downscale them
+      // so the ANSI render fits in a typical terminal row budget.
+      const longEdge = Math.max(native.width, native.height);
+      if (scale === 1 && longEdge > NON_PIXEL_ART_TARGET_LONG_EDGE) {
+        const k = NON_PIXEL_ART_TARGET_LONG_EDGE / longEdge;
+        native = boxDownscaleRgba(
+          native.data,
+          native.width,
+          native.height,
+          Math.max(1, Math.round(native.width * k)),
+          Math.max(1, Math.round(native.height * k)),
+        );
+      }
       return renderRgbaToAnsi(native.data, native.width, native.height, {
         cropToContent: true,
         cropPad: 1,
@@ -942,8 +1019,11 @@ async function tryLoadFromQuery(query: string): Promise<LoadedSim | null> {
       const { getRecordFromPds } = await import("./simocracy.ts");
       const sim = await getRecordFromPds<{
         name: string;
-        image?: { ref: unknown };
-        sprite?: { ref: unknown };
+        spriteKind?: "pipoya" | "codexPet";
+        image?: { ref: unknown; mimeType: string; size: number };
+        sprite?: { ref: unknown; mimeType: string; size: number };
+        petSheet?: { ref: unknown; mimeType: string; size: number };
+        petManifest?: { id?: string; displayName?: string; description?: string };
         $type?: string;
       }>(did, "org.simocracy.sim", rkey);
       const match: SimMatch = {
@@ -954,9 +1034,12 @@ async function tryLoadFromQuery(query: string): Promise<LoadedSim | null> {
         sim: {
           $type: "org.simocracy.sim",
           name: sim.name,
+          spriteKind: sim.spriteKind,
           settings: { selectedOptions: {} },
           image: sim.image as never,
           sprite: sim.sprite as never,
+          petSheet: sim.petSheet as never,
+          petManifest: sim.petManifest,
           createdAt: "",
         },
       };

package/src/png-to-ansi.ts

@@ -235,6 +235,85 @@ export function downscaleRgbaNearest(
   return { data: out, width: newW, height: newH };
 }
 
+/**
+ * Box-average downscale an RGBA buffer to arbitrary target dimensions.
+ *
+ * Uses straight area-weighted averaging on premultiplied alpha so partly
+ * transparent edge pixels don't bleed black into the result. Designed for
+ * non-pixel-art inputs (codex pet idle thumbnails, codex pet sheet cells)
+ * where we want a smaller display size at non-integer ratios.
+ *
+ * If the target equals the source size, returns the input untouched.
+ */
+export function boxDownscaleRgba(
+  data: Buffer,
+  width: number,
+  height: number,
+  targetW: number,
+  targetH: number,
+): { data: Buffer; width: number; height: number } {
+  if (targetW < 1 || targetH < 1) {
+    throw new Error(`boxDownscaleRgba target must be ≥1×1, got ${targetW}×${targetH}`);
+  }
+  if (targetW === width && targetH === height) {
+    return { data, width, height };
+  }
+  const out = Buffer.alloc(targetW * targetH * 4);
+  // For each output pixel, average the source rectangle that maps to it.
+  // Edge cells are clamped to the source extent.
+  for (let oy = 0; oy < targetH; oy++) {
+    const sy0 = (oy * height) / targetH;
+    const sy1 = ((oy + 1) * height) / targetH;
+    const y0 = Math.floor(sy0);
+    const y1 = Math.min(height, Math.ceil(sy1));
+    for (let ox = 0; ox < targetW; ox++) {
+      const sx0 = (ox * width) / targetW;
+      const sx1 = ((ox + 1) * width) / targetW;
+      const x0 = Math.floor(sx0);
+      const x1 = Math.min(width, Math.ceil(sx1));
+
+      // Premultiplied accumulation so transparent pixels don't smear
+      // black RGB values into mostly-opaque neighbours.
+      let rSum = 0,
+        gSum = 0,
+        bSum = 0,
+        aSum = 0,
+        wSum = 0;
+      for (let y = y0; y < y1; y++) {
+        // Vertical coverage of this source row inside the output cell.
+        const wy = Math.min(y + 1, sy1) - Math.max(y, sy0);
+        if (wy <= 0) continue;
+        for (let x = x0; x < x1; x++) {
+          const wx = Math.min(x + 1, sx1) - Math.max(x, sx0);
+          if (wx <= 0) continue;
+          const w = wx * wy;
+          const i = (y * width + x) * 4;
+          const a = data[i + 3];
+          const aw = a * w;
+          rSum += data[i] * aw;
+          gSum += data[i + 1] * aw;
+          bSum += data[i + 2] * aw;
+          aSum += aw;
+          wSum += w;
+        }
+      }
+      const di = (oy * targetW + ox) * 4;
+      if (aSum > 0) {
+        out[di] = Math.min(255, Math.round(rSum / aSum));
+        out[di + 1] = Math.min(255, Math.round(gSum / aSum));
+        out[di + 2] = Math.min(255, Math.round(bSum / aSum));
+        out[di + 3] = Math.min(255, Math.round(aSum / wSum));
+      } else {
+        out[di] = 0;
+        out[di + 1] = 0;
+        out[di + 2] = 0;
+        out[di + 3] = 0;
+      }
+    }
+  }
+  return { data: out, width: targetW, height: targetH };
+}
+
 /**
  * Detect the integer upscale factor of a pixel-art image by scanning
  * for the largest factor F where every F×F block has uniform colour.

package/src/simocracy.ts

@@ -24,13 +24,33 @@ export interface BlobRef {
   size: number;
 }
 
+/** Which sprite system a sim uses. Absent = legacy 'pipoya'. */
+export type SpriteKind = "pipoya" | "codexPet";
+
+/** Subset of pet.json from the OpenAI hatch-pet skill output. */
+export interface CodexPetManifest {
+  id?: string;
+  displayName?: string;
+  description?: string;
+}
+
 export interface SimRecord {
   $type: "org.simocracy.sim";
   name: string;
-  settings: SpriteSettings;
+  /** Discriminator for sprite system. Absent = 'pipoya'. */
+  spriteKind?: SpriteKind;
+  /** Pipoya appearance settings (only used when spriteKind = 'pipoya'). */
+  settings?: SpriteSettings;
+  /** Rendered avatar PNG/JPEG/WebP thumbnail (always present for codex pets, generated client-side as a 128×128 PNG). */
   image?: BlobRef;
+  /** Pipoya 4×4 walk sheet (128×128 PNG). Only present when spriteKind = 'pipoya'. */
   sprite?: BlobRef;
+  /** Codex pet 8×9 atlas (1536×1872, 192×208 cells, PNG or WebP). Only present when spriteKind = 'codexPet'. */
+  petSheet?: BlobRef;
+  /** Subset of pet.json from the hatch-pet skill output. Only present when spriteKind = 'codexPet'. */
+  petManifest?: CodexPetManifest;
   createdAt: string;
+  updatedAt?: string;
 }
 
 export interface AgentsRecord {

package/src/webp-to-rgba.ts

@@ -0,0 +1,70 @@
+/**
+ * WebP → RGBA decoder for Node.
+ *
+ * `pngjs` covers the PNG side of pi-simocracy, but the OpenAI hatch-pet
+ * skill (and therefore most `org.simocracy.sim` records with
+ * `spriteKind = "codexPet"`) ships its 1536×1872 atlas as WebP. We need
+ * a WebP decoder to render the idle frame.
+ *
+ * @jsquash/webp's `decode()` is browser-shaped: in Node its wasm glue
+ * tries to `fetch()` its own `.wasm` URL, which Undici rejects for
+ * `file://`. Workaround: read the wasm bytes from disk via
+ * `createRequire().resolve()`, compile to a `WebAssembly.Module`, and
+ * feed it to the package's documented `init()` escape hatch. Fully ESM,
+ * no native bindings.
+ *
+ * The wasm module is initialised lazily on the first `decodeWebp()`
+ * call so loading pi-simocracy stays cheap when no one ever loads a
+ * codex pet sim. Subsequent calls reuse the same module.
+ */
+
+import { createRequire } from "node:module";
+import { readFile } from "node:fs/promises";
+import decode, { init as initWebpDecode } from "@jsquash/webp/decode.js";
+
+let wasmInitPromise: Promise<void> | null = null;
+
+async function ensureWasmInit(): Promise<void> {
+  if (wasmInitPromise) return wasmInitPromise;
+  wasmInitPromise = (async () => {
+    const require = createRequire(import.meta.url);
+    // @jsquash/webp ships the decoder wasm next to its JS glue; the
+    // package only re-exports JS files, so we have to resolve the wasm
+    // path manually. Resolving against the JS entry guarantees we hit
+    // the same install of the package the bundler/linker picked.
+    const decodeJs = require.resolve("@jsquash/webp/decode.js");
+    const wasmPath = decodeJs.replace(/decode\.js$/, "codec/dec/webp_dec.wasm");
+    const bytes = await readFile(wasmPath);
+    const mod = await WebAssembly.compile(bytes);
+    await initWebpDecode(mod);
+  })().catch((err) => {
+    // Reset so the next caller can retry; otherwise a transient FS
+    // error would permanently break codex-pet rendering for the session.
+    wasmInitPromise = null;
+    throw new Error(`Failed to init @jsquash/webp wasm: ${(err as Error).message}`);
+  });
+  return wasmInitPromise;
+}
+
+/**
+ * Decode a WebP buffer into the same flat-RGBA shape `decodePng()`
+ * returns, so call sites can treat the two interchangeably.
+ *
+ * Allocates a fresh `Buffer` rather than aliasing `Uint8ClampedArray`
+ * so downstream `Buffer`-only helpers (`pixelAt`, `cropRgba`,
+ * `boxDownscaleRgba`) keep working without coercion.
+ */
+export async function decodeWebp(
+  buf: Buffer,
+): Promise<{ width: number; height: number; data: Buffer }> {
+  await ensureWasmInit();
+  // jSquash wants an ArrayBuffer view of just the relevant bytes —
+  // passing the whole underlying buffer mis-decodes if `buf` is a slice.
+  const ab = buf.buffer.slice(buf.byteOffset, buf.byteOffset + buf.byteLength);
+  const img = await decode(ab as ArrayBuffer);
+  return {
+    width: img.width,
+    height: img.height,
+    data: Buffer.from(img.data.buffer, img.data.byteOffset, img.data.byteLength),
+  };
+}