pi-simocracy 0.3.0 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +65 -21
- package/package.json +2 -1
- package/src/index.ts +507 -58
- package/src/persona.ts +37 -0
- package/src/png-encode.ts +29 -0
- package/src/png-to-ansi.ts +79 -0
- package/src/simocracy.ts +21 -1
- package/src/webp-to-rgba.ts +70 -0
package/README.md
CHANGED
|
@@ -1,18 +1,28 @@
|
|
|
1
1
|
# pi-simocracy
|
|
2
2
|
|
|
3
3
|
Load a [Simocracy](https://simocracy.org) sim into your [`pi`](https://github.com/mariozechner/pi-coding-agent) chat — see its
|
|
4
|
-
|
|
4
|
+
sprite render in the terminal and chat with the agent **as that sim**.
|
|
5
5
|
|
|
6
6
|
```
|
|
7
7
|
/sim mr meow
|
|
8
8
|
```
|
|
9
9
|
|
|
10
10
|
…fetches Mr Meow from Simocracy's ATProto indexer, renders his 32×32
|
|
11
|
-
sprite as colored ANSI half-blocks directly in the chat, and
|
|
12
|
-
his constitution + speaking style into pi's system prompt so pi
|
|
11
|
+
pixel-art sprite as colored ANSI half-blocks directly in the chat, and
|
|
12
|
+
pushes his constitution + speaking style into pi's system prompt so pi
|
|
13
13
|
roleplays as Mr Meow until you `/sim unload`.
|
|
14
14
|
|
|
15
|
-
|
|
15
|
+
<p align="center">
|
|
16
|
+
<img src="https://raw.githubusercontent.com/GainForest/pi-simocracy/main/demo/sim-load.gif" alt="Mr Meow (a pipoya pixel-art cat) loaded inline in pi's chat" width="760">
|
|
17
|
+
</p>
|
|
18
|
+
|
|
19
|
+
As of v0.4.0, codex pet sims (OpenAI hatch-pet skill output) render too —
|
|
20
|
+
load **Einstein** with `/sim einstein` and the WebP atlas decodes, the
|
|
21
|
+
idle frame crops, and the half-block ANSI render fires inline:
|
|
22
|
+
|
|
23
|
+
<p align="center">
|
|
24
|
+
<img src="https://raw.githubusercontent.com/GainForest/pi-simocracy/main/demo/codex-pet-load.gif" alt="Einstein (a codex pet sim with a WebP petSheet) loaded inline in pi's chat" width="760">
|
|
25
|
+
</p>
|
|
16
26
|
|
|
17
27
|
---
|
|
18
28
|
|
|
@@ -109,11 +119,37 @@ The same actions are exposed to pi as tools, so the model can drive them itself:
|
|
|
109
119
|
- `org.simocracy.sim` — display name + sprite + avatar blob refs
|
|
110
120
|
- `org.simocracy.agents` — short description + full constitution
|
|
111
121
|
- `org.simocracy.style` — speaking style / mannerisms
|
|
112
|
-
4. **Render.** Fetch the sprite
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
122
|
+
4. **Render.** Fetch the sprite blob via `com.atproto.sync.getBlob`,
|
|
123
|
+
decode it, crop the front-facing idle frame. Two render paths
|
|
124
|
+
depending on the sim's `spriteKind`:
|
|
125
|
+
- **`pipoya`** (legacy + default): 128×128 PNG, 4×4 of 32×32 walking
|
|
126
|
+
frames; decode with `pngjs`, take row 0 col 0 at native size.
|
|
127
|
+
- **`codexPet`** (OpenAI hatch-pet output): 1536×1872 atlas, 8×9 of
|
|
128
|
+
192×208 cells. PNG sheets decode through `pngjs`; WebP sheets
|
|
129
|
+
decode through `@jsquash/webp` (wasm, lazy-init). The idle cell
|
|
130
|
+
(row 0 col 0) is the source for both render paths below.
|
|
131
|
+
|
|
132
|
+
Then emit through one of two terminal output paths, picked
|
|
133
|
+
automatically per-terminal:
|
|
134
|
+
- **Inline graphics** (Kitty graphics protocol or iTerm2 inline
|
|
135
|
+
images). The cropped RGBA cell is re-encoded to PNG via `pngjs`
|
|
136
|
+
and handed to pi-tui's `Image` component, which transmits it as a
|
|
137
|
+
true-color bitmap. Used in Kitty, Ghostty, WezTerm, Konsole, and
|
|
138
|
+
iTerm2 — the terminal does its own scaling, so pixel-art sprites
|
|
139
|
+
stay crisp and codex pets render at full fidelity.
|
|
140
|
+
- **24-bit ANSI half-blocks** (universal fallback). Emits the
|
|
141
|
+
upper/lower half-block characters `▀`/`▄` with `\x1b[38;2;…m`
|
|
142
|
+
true-color escapes so each terminal cell paints two pixels. Used
|
|
143
|
+
in Apple Terminal, plain SSH, tmux without passthrough, and
|
|
144
|
+
anywhere that doesn't advertise inline-image support. Transparent
|
|
145
|
+
regions show pi's background through.
|
|
146
|
+
|
|
147
|
+
To force the half-block path even on a graphics-capable terminal
|
|
148
|
+
(handy for screenshots and demo recordings), set
|
|
149
|
+
`SIMOCRACY_INLINE_GRAPHICS=ansi`. The default is `auto`.
|
|
150
|
+
|
|
151
|
+
Both paths use the same upstream RGBA buffer — swapping between
|
|
152
|
+
them changes only the final encoding step, never the source pixels.
|
|
117
153
|
5. **Inject.** A `before_agent_start` event handler appends the sim's
|
|
118
154
|
identity + constitution + speaking style to pi's system prompt **every
|
|
119
155
|
turn**. After `/sim unload`, a one-shot override fires on the next
|
|
@@ -129,15 +165,18 @@ keeps the terminal it's already running in.
|
|
|
129
165
|
|
|
130
166
|
```
|
|
131
167
|
src/
|
|
132
|
-
├── index.ts
|
|
133
|
-
├── persona.ts
|
|
134
|
-
├── simocracy.ts
|
|
135
|
-
├── writes.ts
|
|
136
|
-
├── png-to-ansi.ts
|
|
137
|
-
├──
|
|
138
|
-
|
|
168
|
+
├── index.ts # extension entry: slash command, tools, persona injection
|
|
169
|
+
├── persona.ts # buildSimPrompt(sim) — the system-prompt fragment
|
|
170
|
+
├── simocracy.ts # indexer + PDS client (read-only fetchers)
|
|
171
|
+
├── writes.ts # PDS writers + ownership / sign-in preconditions
|
|
172
|
+
├── png-to-ansi.ts # RGBA half-block ANSI renderer + downscalers
|
|
173
|
+
├── png-encode.ts # RGBA → PNG encoder for inline-graphics protocols
|
|
174
|
+
├── webp-to-rgba.ts # @jsquash/webp wrapper for codex pet WebP sheets
|
|
175
|
+
├── openrouter.ts # minimal OpenRouter client (only used by simocracy_chat)
|
|
176
|
+
└── auth/ # ATProto OAuth loopback flow + session storage
|
|
139
177
|
demo/
|
|
140
|
-
|
|
178
|
+
├── sim-load.tape # vhs tape — Mr Meow (pipoya)
|
|
179
|
+
└── codex-pet-load.tape # vhs tape — Einstein (codex pet)
|
|
141
180
|
```
|
|
142
181
|
|
|
143
182
|
---
|
|
@@ -153,11 +192,12 @@ pi -e $(pwd)/src/index.ts -ne -ns # load the extension directly
|
|
|
153
192
|
|
|
154
193
|
Then in `pi`: `/sim mr meow`.
|
|
155
194
|
|
|
156
|
-
To rebuild the demo
|
|
195
|
+
To rebuild the demo recordings:
|
|
157
196
|
|
|
158
197
|
```bash
|
|
159
198
|
brew install vhs # one-time
|
|
160
|
-
vhs demo/sim-load.tape #
|
|
199
|
+
vhs demo/sim-load.tape # Mr Meow (pipoya) — demo/sim-load.{webm,gif}
|
|
200
|
+
vhs demo/codex-pet-load.tape # Einstein (codex pet) — demo/codex-pet-load.{webm,gif}
|
|
161
201
|
```
|
|
162
202
|
|
|
163
203
|
---
|
|
@@ -172,7 +212,11 @@ These come bundled with `pi` itself, so installing pi-simocracy via
|
|
|
172
212
|
|
|
173
213
|
Direct npm dependencies (auto-installed):
|
|
174
214
|
|
|
175
|
-
- `pngjs` — PNG decoder for sprite blobs
|
|
215
|
+
- `pngjs` — PNG decoder for pipoya sprite blobs and codex pet PNG sheets
|
|
216
|
+
- `@jsquash/webp` — wasm WebP decoder for codex pet WebP sheets
|
|
217
|
+
(lazy-init, no native bindings)
|
|
218
|
+
- `@atproto/api` + `@atproto/oauth-client-node` — ATProto loopback OAuth
|
|
219
|
+
for `/sim login` and PDS writes via `simocracy_update_sim`
|
|
176
220
|
- `typebox` — tool parameter schemas
|
|
177
221
|
|
|
178
222
|
---
|
|
@@ -192,4 +236,4 @@ Direct npm dependencies (auto-installed):
|
|
|
192
236
|
|
|
193
237
|
## License
|
|
194
238
|
|
|
195
|
-
MIT — see [LICENSE](
|
|
239
|
+
MIT — see [LICENSE](https://github.com/GainForest/pi-simocracy/blob/main/LICENSE).
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-simocracy",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.5.0",
|
|
4
4
|
"description": "Pi extension: load a Simocracy sim into your chat — see its pixel-art sprite render inline in the terminal and roleplay with it.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"author": "David Dao <david@gainforest.earth> (https://github.com/daviddao)",
|
|
@@ -45,6 +45,7 @@
|
|
|
45
45
|
"dependencies": {
|
|
46
46
|
"@atproto/api": "^0.19.11",
|
|
47
47
|
"@atproto/oauth-client-node": "^0.3.17",
|
|
48
|
+
"@jsquash/webp": "^1.5.0",
|
|
48
49
|
"pngjs": "^7.0.0",
|
|
49
50
|
"typebox": "^1.1.24"
|
|
50
51
|
},
|
package/src/index.ts
CHANGED
|
@@ -60,7 +60,15 @@ import type {
|
|
|
60
60
|
ExtensionContext,
|
|
61
61
|
ExtensionCommandContext,
|
|
62
62
|
} from "@mariozechner/pi-coding-agent";
|
|
63
|
-
import {
|
|
63
|
+
import {
|
|
64
|
+
Box,
|
|
65
|
+
Image,
|
|
66
|
+
Text,
|
|
67
|
+
allocateImageId,
|
|
68
|
+
getCapabilities,
|
|
69
|
+
type ImageTheme,
|
|
70
|
+
type TUI,
|
|
71
|
+
} from "@mariozechner/pi-tui";
|
|
64
72
|
import { Type } from "typebox";
|
|
65
73
|
|
|
66
74
|
import {
|
|
@@ -81,7 +89,10 @@ import {
|
|
|
81
89
|
cropRgba,
|
|
82
90
|
detectPixelArtScale,
|
|
83
91
|
downscaleRgbaNearest,
|
|
92
|
+
boxDownscaleRgba,
|
|
84
93
|
} from "./png-to-ansi.ts";
|
|
94
|
+
import { encodeRgbaToPng } from "./png-encode.ts";
|
|
95
|
+
import { decodeWebp } from "./webp-to-rgba.ts";
|
|
85
96
|
import { openRouterComplete, type ChatMessage } from "./openrouter.ts";
|
|
86
97
|
import { buildSimPrompt, type LoadedSim } from "./persona.ts";
|
|
87
98
|
import { runLogin, runLogout, runWhoami } from "./auth/commands.ts";
|
|
@@ -111,6 +122,100 @@ let loadedSim: LoadedSim | null = null;
|
|
|
111
122
|
*/
|
|
112
123
|
let justUnloaded: string | null = null;
|
|
113
124
|
|
|
125
|
+
// ---------------------------------------------------------------------------
|
|
126
|
+
// Animation state
|
|
127
|
+
//
|
|
128
|
+
// We animate the most recently loaded codex pet's idle row inline in chat,
|
|
129
|
+
// using the same pattern pi-tui's spinner uses: a setInterval ticks a frame
|
|
130
|
+
// counter and calls `tui.requestRender()`, which makes pi-tui re-run the
|
|
131
|
+
// message renderer. Each render returns a new `Image` keyed by a stable
|
|
132
|
+
// Kitty image ID, so the terminal swaps the displayed frame in place.
|
|
133
|
+
//
|
|
134
|
+
// Typing keeps working during animation — input handling and rendering are
|
|
135
|
+
// independent code paths in pi-tui (verified by inspection of `Loader`).
|
|
136
|
+
//
|
|
137
|
+
// Only ONE sim animates at a time. When a new sim is loaded, the previous
|
|
138
|
+
// timer stops and the previous message freezes on its idle frame.
|
|
139
|
+
// ---------------------------------------------------------------------------
|
|
140
|
+
|
|
141
|
+
/** Captured pi-tui handle. Set by the `simocracy` widget factory the first
|
|
142
|
+
* time a sim is loaded; reused for every subsequent animation tick. Lazy
|
|
143
|
+
* because the TUI doesn't exist when the extension first imports. */
|
|
144
|
+
let capturedTui: TUI | null = null;
|
|
145
|
+
|
|
146
|
+
interface ActiveAnimation {
|
|
147
|
+
/** Identifies which loaded-sim message owns this animation. The renderer
|
|
148
|
+
* compares against `details.animationKey` to decide whether to use the
|
|
149
|
+
* current frame or the static idle PNG. */
|
|
150
|
+
key: string;
|
|
151
|
+
/** Stable Kitty image ID so frame transmissions replace the previous one
|
|
152
|
+
* instead of stacking. Allocated once per sim load. */
|
|
153
|
+
imageId: number;
|
|
154
|
+
/** Pre-encoded base64 PNGs in playback order. */
|
|
155
|
+
frames: string[];
|
|
156
|
+
/** Frame width / height in pixels (uniform across frames). */
|
|
157
|
+
widthPx: number;
|
|
158
|
+
heightPx: number;
|
|
159
|
+
/** Currently displayed frame index. */
|
|
160
|
+
currentFrame: number;
|
|
161
|
+
/** Active setInterval handle so we can clear it on unload / reload. */
|
|
162
|
+
intervalId: ReturnType<typeof setInterval> | null;
|
|
163
|
+
}
|
|
164
|
+
let currentAnimation: ActiveAnimation | null = null;
|
|
165
|
+
|
|
166
|
+
/** Default-on; set `SIMOCRACY_ANIMATION=off` to freeze on idle frame 0. */
|
|
167
|
+
const animationEnabled =
|
|
168
|
+
(process.env.SIMOCRACY_ANIMATION ?? "on").toLowerCase() !== "off";
|
|
169
|
+
|
|
170
|
+
/**
|
|
171
|
+
* Width of the inline sprite render in terminal cells. The source PNG
|
|
172
|
+
* is always transmitted at native resolution (192×208 for codex pets,
|
|
173
|
+
* 32×32 for pipoya — see `renderSprite`); this number controls only
|
|
174
|
+
* how many cells the terminal uses to display it. Aspect ratio is
|
|
175
|
+
* preserved by pi-tui's `calculateImageRows`. Override with the
|
|
176
|
+
* `SIMOCRACY_SPRITE_WIDTH` env var.
|
|
177
|
+
*
|
|
178
|
+
* 10 cells wide gives a compact ≈6-row inline render that sits
|
|
179
|
+
* comfortably alongside one or two paragraphs of bio text without
|
|
180
|
+
* dominating the chat. Bump it to 20 or 32 if you want the sprite
|
|
181
|
+
* to read at a glance.
|
|
182
|
+
*/
|
|
183
|
+
const spriteWidthCells = (() => {
|
|
184
|
+
const raw = process.env.SIMOCRACY_SPRITE_WIDTH;
|
|
185
|
+
const parsed = raw ? Number.parseInt(raw, 10) : NaN;
|
|
186
|
+
return Number.isFinite(parsed) && parsed >= 4 && parsed <= 120 ? parsed : 10;
|
|
187
|
+
})();
|
|
188
|
+
|
|
189
|
+
function stopCurrentAnimation(): void {
|
|
190
|
+
if (currentAnimation?.intervalId !== null && currentAnimation?.intervalId !== undefined) {
|
|
191
|
+
clearInterval(currentAnimation.intervalId);
|
|
192
|
+
}
|
|
193
|
+
currentAnimation = null;
|
|
194
|
+
}
|
|
195
|
+
|
|
196
|
+
function startAnimationFor(key: string, frames: { pngBase64: string[]; fps: number; widthPx: number; heightPx: number }): void {
|
|
197
|
+
// Always replace any prior animation — only the most recent loaded-sim
|
|
198
|
+
// message animates.
|
|
199
|
+
stopCurrentAnimation();
|
|
200
|
+
if (!animationEnabled || frames.pngBase64.length < 2) return;
|
|
201
|
+
currentAnimation = {
|
|
202
|
+
key,
|
|
203
|
+
imageId: allocateImageId(),
|
|
204
|
+
frames: frames.pngBase64,
|
|
205
|
+
widthPx: frames.widthPx,
|
|
206
|
+
heightPx: frames.heightPx,
|
|
207
|
+
currentFrame: 0,
|
|
208
|
+
intervalId: null,
|
|
209
|
+
};
|
|
210
|
+
const intervalMs = Math.max(1000 / frames.fps, 60); // clamp to ≆16 fps max
|
|
211
|
+
currentAnimation.intervalId = setInterval(() => {
|
|
212
|
+
if (!currentAnimation) return;
|
|
213
|
+
currentAnimation.currentFrame =
|
|
214
|
+
(currentAnimation.currentFrame + 1) % currentAnimation.frames.length;
|
|
215
|
+
capturedTui?.requestRender();
|
|
216
|
+
}, intervalMs);
|
|
217
|
+
}
|
|
218
|
+
|
|
114
219
|
// ---------------------------------------------------------------------------
|
|
115
220
|
// Helpers
|
|
116
221
|
// ---------------------------------------------------------------------------
|
|
@@ -124,19 +229,104 @@ function blobLink(ref: unknown): string | null {
|
|
|
124
229
|
}
|
|
125
230
|
|
|
126
231
|
/**
|
|
127
|
-
*
|
|
128
|
-
*
|
|
129
|
-
*
|
|
232
|
+
* Maximum long-edge of a non-pixel-art avatar before we box-downscale it.
|
|
233
|
+
* Picked so codex pet idle frames render at roughly the same on-screen
|
|
234
|
+
* height as a 32×32 pipoya sprite (~17 terminal lines) instead of
|
|
235
|
+
* ballooning to 60+ lines and pushing the rest of chat off-screen.
|
|
236
|
+
*/
|
|
237
|
+
const NON_PIXEL_ART_TARGET_LONG_EDGE = 40;
|
|
238
|
+
|
|
239
|
+
/**
|
|
240
|
+
* Render a sim's avatar as colored ANSI half-block art, sized to fit
|
|
241
|
+
* comfortably alongside the loaded-sim message in a typical terminal.
|
|
242
|
+
*
|
|
243
|
+
* Branches on `spriteKind` (lexicon discriminator):
|
|
130
244
|
*
|
|
131
|
-
*
|
|
132
|
-
*
|
|
133
|
-
*
|
|
245
|
+
* pipoya — fetch the 4×4 walk-sheet PNG, crop the front-facing
|
|
246
|
+
* walk-1 frame (32×32, row 0 col 0), render at native
|
|
247
|
+
* resolution. ~16 lines tall.
|
|
248
|
+
*
|
|
249
|
+
* codexPet — prefer the 8×9 atlas (`petSheet`, 1536×1872, 192×208
|
|
250
|
+
* cells) when it's PNG: crop the idle cell (row 0 col 0)
|
|
251
|
+
* and box-downscale to ~32 wide. The atlas is usually WebP
|
|
252
|
+
* (the OpenAI hatch-pet skill emits WebP and `pngjs`
|
|
253
|
+
* doesn't speak WebP), so we fall through to the rendered
|
|
254
|
+
* `image` thumbnail — a PNG that the simocracy.org client
|
|
255
|
+
* generates at 128×128 from the idle frame. That gets
|
|
256
|
+
* box-downscaled to a comparable terminal size.
|
|
257
|
+
*
|
|
258
|
+
* absent — treated as legacy 'pipoya' for back-compat.
|
|
259
|
+
*
|
|
260
|
+
* Returns BOTH the ANSI half-block render (always — it's our universal
|
|
261
|
+
* fallback) AND, when possible, a PNG of the same RGBA cell for inline
|
|
262
|
+
* terminal-graphics protocols (Kitty / iTerm2). The renderer chooses
|
|
263
|
+
* which to display based on the host terminal's capabilities.
|
|
264
|
+
*
|
|
265
|
+
* The PNG is encoded at the *native* resolution we cropped to (32×32
|
|
266
|
+
* for pipoya, 192×208 for codexPet, the post-downscale size for the
|
|
267
|
+
* image fallback). Kitty / iTerm2 do their own scaling to the target
|
|
268
|
+
* cell box, so passing the native pixels gives the terminal the most
|
|
269
|
+
* information to work with — pixel art scales up crisply with
|
|
270
|
+
* nearest-neighbour, codex pet thumbnails scale down cleanly.
|
|
271
|
+
*/
|
|
272
|
+
export interface SpriteRender {
|
|
273
|
+
ansi: string;
|
|
274
|
+
png?: { data: Buffer; widthPx: number; heightPx: number };
|
|
275
|
+
/**
|
|
276
|
+
* Optional animation frames. Each entry in `pngs` is a PNG of one
|
|
277
|
+
* cell. Set for codex pets (idle row of their atlas); absent for
|
|
278
|
+
* everything else (pipoya sprites and image fallbacks render as a
|
|
279
|
+
* single static frame). The renderer plays these in a loop using
|
|
280
|
+
* the Kitty in-place image-swap protocol when animation is enabled.
|
|
281
|
+
*/
|
|
282
|
+
frames?: { pngs: Buffer[]; widthPx: number; heightPx: number; fps: number };
|
|
283
|
+
}
|
|
284
|
+
|
|
285
|
+
/**
|
|
286
|
+
* Codex pet atlas constants — keep in sync with the simocracy-v2 hatch-pet
|
|
287
|
+
* skill that produces these sheets. The atlas is 8 cols × 9 rows of
|
|
288
|
+
* 192×208 cells; the idle animation lives on row 0, frames 0–5 (the same
|
|
289
|
+
* default tui-pets ships when a pet.json doesn't override it).
|
|
134
290
|
*/
|
|
135
|
-
|
|
291
|
+
const CODEX_PET_CELL_W = 192;
|
|
292
|
+
const CODEX_PET_CELL_H = 208;
|
|
293
|
+
const CODEX_PET_COLS = 8;
|
|
294
|
+
const CODEX_PET_IDLE_FRAMES = [0, 1, 2, 3, 4, 5];
|
|
295
|
+
const CODEX_PET_IDLE_FPS = 5;
|
|
296
|
+
|
|
297
|
+
async function renderSprite(sim: SimMatch): Promise<SpriteRender | null> {
|
|
298
|
+
const spriteKind = sim.sim.spriteKind ?? "pipoya";
|
|
136
299
|
const spriteLink = blobLink(sim.sim.sprite?.ref);
|
|
300
|
+
const petSheetLink = blobLink(sim.sim.petSheet?.ref);
|
|
301
|
+
const petSheetMime = sim.sim.petSheet?.mimeType;
|
|
137
302
|
const imageLink = blobLink(sim.sim.image?.ref);
|
|
138
303
|
|
|
139
|
-
|
|
304
|
+
/** Render an RGBA region to ANSI half-blocks (shared options). */
|
|
305
|
+
const toAnsi = (data: Buffer, width: number, height: number) =>
|
|
306
|
+
renderRgbaToAnsi(data, width, height, {
|
|
307
|
+
cropToContent: true,
|
|
308
|
+
cropPad: 1,
|
|
309
|
+
indent: 2,
|
|
310
|
+
alphaThreshold: 16,
|
|
311
|
+
});
|
|
312
|
+
|
|
313
|
+
/** Bundle ANSI + PNG of the same RGBA region. PNG-encode is a tiny
|
|
314
|
+
* cost and is wrapped in try/catch — if encoding ever fails we
|
|
315
|
+
* still return the ANSI render (lossless fallback). */
|
|
316
|
+
const bundle = (data: Buffer, width: number, height: number): SpriteRender => {
|
|
317
|
+
const ansi = toAnsi(data, width, height);
|
|
318
|
+
let png: SpriteRender["png"];
|
|
319
|
+
try {
|
|
320
|
+
const pngBytes = encodeRgbaToPng(data, width, height);
|
|
321
|
+
png = { data: pngBytes, widthPx: width, heightPx: height };
|
|
322
|
+
} catch {
|
|
323
|
+
png = undefined;
|
|
324
|
+
}
|
|
325
|
+
return { ansi, png };
|
|
326
|
+
};
|
|
327
|
+
|
|
328
|
+
// Pipoya 4×4 walk sheet — the legacy/default path.
|
|
329
|
+
if (spriteKind !== "codexPet" && spriteLink) {
|
|
140
330
|
try {
|
|
141
331
|
const buf = await fetchBlob(sim.did, spriteLink);
|
|
142
332
|
const { width, height, data } = decodePng(buf);
|
|
@@ -144,43 +334,116 @@ async function renderSpriteAnsi(sim: SimMatch): Promise<string | null> {
|
|
|
144
334
|
if (width >= FRAME && height >= FRAME) {
|
|
145
335
|
// Sheets are 4×4 of 32×32 frames — row 0 col 0 = front-facing walk1.
|
|
146
336
|
const frame = cropRgba(data, width, height, 0, 0, FRAME, FRAME);
|
|
147
|
-
return
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
337
|
+
return bundle(frame, FRAME, FRAME);
|
|
338
|
+
}
|
|
339
|
+
return bundle(data, width, height);
|
|
340
|
+
} catch {
|
|
341
|
+
/* fall through to image fallback */
|
|
342
|
+
}
|
|
343
|
+
}
|
|
344
|
+
|
|
345
|
+
// Codex pet atlas — the canonical asset for codex pets. Sheets are
|
|
346
|
+
// 1536×1872 with 192×208 cells laid out 8 cols × 9 rows; row 0 col 0
|
|
347
|
+
// is the idle frame. Both PNG and WebP are valid in the lexicon (the
|
|
348
|
+
// hatch-pet skill emits WebP, the dropzone preserves PNG when the
|
|
349
|
+
// user drops a PNG sheet) so we pick the right decoder by mimeType.
|
|
350
|
+
// We crop the idle cell first thing. For the ANSI render we
|
|
351
|
+
// box-downscale to ~32 wide so the inline render is similar in
|
|
352
|
+
// height to a pipoya sprite (~17 lines). For the inline-graphics
|
|
353
|
+
// PNG we keep the native 192×208 resolution — Kitty / iTerm2 will
|
|
354
|
+
// scale it down at display time, which preserves more detail than
|
|
355
|
+
// pre-downscaling here.
|
|
356
|
+
if (spriteKind === "codexPet" && petSheetLink) {
|
|
357
|
+
try {
|
|
358
|
+
const buf = await fetchBlob(sim.did, petSheetLink);
|
|
359
|
+
const { width, height, data } =
|
|
360
|
+
petSheetMime === "image/webp" ? await decodeWebp(buf) : decodePng(buf);
|
|
361
|
+
const CELL_W = 192;
|
|
362
|
+
const CELL_H = 208;
|
|
363
|
+
if (width >= CELL_W && height >= CELL_H) {
|
|
364
|
+
const cell = cropRgba(data, width, height, 0, 0, CELL_W, CELL_H);
|
|
365
|
+
const targetW = 32;
|
|
366
|
+
const targetH = Math.round((CELL_H / CELL_W) * targetW); // ~35
|
|
367
|
+
const scaled = boxDownscaleRgba(cell, CELL_W, CELL_H, targetW, targetH);
|
|
368
|
+
const ansi = toAnsi(scaled.data, scaled.width, scaled.height);
|
|
369
|
+
let png: SpriteRender["png"];
|
|
370
|
+
try {
|
|
371
|
+
const pngBytes = encodeRgbaToPng(cell, CELL_W, CELL_H);
|
|
372
|
+
png = { data: pngBytes, widthPx: CELL_W, heightPx: CELL_H };
|
|
373
|
+
} catch {
|
|
374
|
+
png = undefined;
|
|
375
|
+
}
|
|
376
|
+
|
|
377
|
+
// Idle animation frames — same atlas, different cell offsets.
|
|
378
|
+
// Encoded eagerly here so the message renderer can flip
|
|
379
|
+
// between them at ~5 FPS without re-decoding the WebP. Wrapped
|
|
380
|
+
// in its own try/catch — a frame-encoding failure shouldn't
|
|
381
|
+
// disable the static render.
|
|
382
|
+
let frames: SpriteRender["frames"];
|
|
383
|
+
try {
|
|
384
|
+
const framePngs: Buffer[] = [];
|
|
385
|
+
for (const idx of CODEX_PET_IDLE_FRAMES) {
|
|
386
|
+
const cx = (idx % CODEX_PET_COLS) * CELL_W;
|
|
387
|
+
const cy = Math.floor(idx / CODEX_PET_COLS) * CELL_H;
|
|
388
|
+
// Skip frames that fall outside the actual atlas extent.
|
|
389
|
+
if (cx + CELL_W > width || cy + CELL_H > height) continue;
|
|
390
|
+
const frameCell = cropRgba(data, width, height, cx, cy, CELL_W, CELL_H);
|
|
391
|
+
framePngs.push(encodeRgbaToPng(frameCell, CELL_W, CELL_H));
|
|
392
|
+
}
|
|
393
|
+
// Single-frame "animations" are pointless — leave `frames`
|
|
394
|
+
// unset so the renderer takes the static path. Two or more
|
|
395
|
+
// = a real loop.
|
|
396
|
+
if (framePngs.length >= 2) {
|
|
397
|
+
frames = {
|
|
398
|
+
pngs: framePngs,
|
|
399
|
+
widthPx: CELL_W,
|
|
400
|
+
heightPx: CELL_H,
|
|
401
|
+
fps: CODEX_PET_IDLE_FPS,
|
|
402
|
+
};
|
|
403
|
+
}
|
|
404
|
+
} catch {
|
|
405
|
+
frames = undefined;
|
|
406
|
+
}
|
|
407
|
+
return { ansi, png, frames };
|
|
153
408
|
}
|
|
154
|
-
return renderRgbaToAnsi(data, width, height, {
|
|
155
|
-
cropToContent: true,
|
|
156
|
-
cropPad: 1,
|
|
157
|
-
indent: 2,
|
|
158
|
-
alphaThreshold: 16,
|
|
159
|
-
});
|
|
160
409
|
} catch {
|
|
161
|
-
/* fall through to
|
|
410
|
+
/* fall through to image fallback */
|
|
162
411
|
}
|
|
163
412
|
}
|
|
164
413
|
|
|
414
|
+
// Image fallback — used when a sim has no walk sheet (legacy pipoya
|
|
415
|
+
// sims that pre-date `org.simocracy.sim.sprite`) or when the codex
|
|
416
|
+
// pet atlas decode failed for any reason. The simocracy.org client
|
|
417
|
+
// always generates a 128×128 PNG idle thumbnail and uploads it as
|
|
418
|
+
// `image` for codex pets, so even when this path runs the right pose
|
|
419
|
+
// shows up.
|
|
165
420
|
if (imageLink) {
|
|
166
421
|
try {
|
|
167
422
|
const buf = await fetchBlob(sim.did, imageLink);
|
|
168
423
|
const { width, height, data } = decodePng(buf);
|
|
169
|
-
// Old sims
|
|
170
|
-
//
|
|
171
|
-
//
|
|
172
|
-
// back to the original size so the inline render is the same
|
|
173
|
-
// ~13-line height as a sprite-sheet-equipped sim instead of
|
|
174
|
-
// ballooning to ~22 lines and pushing chat off-screen.
|
|
424
|
+
// Old pipoya sims publish a 128×128 PNG that's a 4×-upscaled 32×32
|
|
425
|
+
// pixel-art sprite. Detect and undo that with nearest-neighbour
|
|
426
|
+
// downsampling.
|
|
175
427
|
const scale = detectPixelArtScale(data, width, height, 8);
|
|
176
|
-
|
|
177
|
-
scale > 1
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
428
|
+
let native: { data: Buffer; width: number; height: number } =
|
|
429
|
+
scale > 1
|
|
430
|
+
? downscaleRgbaNearest(data, width, height, scale)
|
|
431
|
+
: { data, width, height };
|
|
432
|
+
// Non-pixel-art images (codex pet thumbnails, avatars from other
|
|
433
|
+
// pipelines) come through at full resolution — box-downscale them
|
|
434
|
+
// so the ANSI render fits in a typical terminal row budget.
|
|
435
|
+
const longEdge = Math.max(native.width, native.height);
|
|
436
|
+
if (scale === 1 && longEdge > NON_PIXEL_ART_TARGET_LONG_EDGE) {
|
|
437
|
+
const k = NON_PIXEL_ART_TARGET_LONG_EDGE / longEdge;
|
|
438
|
+
native = boxDownscaleRgba(
|
|
439
|
+
native.data,
|
|
440
|
+
native.width,
|
|
441
|
+
native.height,
|
|
442
|
+
Math.max(1, Math.round(native.width * k)),
|
|
443
|
+
Math.max(1, Math.round(native.height * k)),
|
|
444
|
+
);
|
|
445
|
+
}
|
|
446
|
+
return bundle(native.data, native.width, native.height);
|
|
184
447
|
} catch {
|
|
185
448
|
/* fall through */
|
|
186
449
|
}
|
|
@@ -207,10 +470,10 @@ async function loadSimByName(query: string): Promise<{
|
|
|
207
470
|
|
|
208
471
|
async function hydrateLoadedSim(match: SimMatch): Promise<LoadedSim> {
|
|
209
472
|
// Fetch agents (constitution), style, sprite ANSI + handle in parallel.
|
|
210
|
-
const [agents, style,
|
|
473
|
+
const [agents, style, sprite, handle] = await Promise.all([
|
|
211
474
|
fetchAgentsForSim(match.uri).catch(() => null) as Promise<AgentsRecord | null>,
|
|
212
475
|
fetchStyleForSim(match.uri).catch(() => null) as Promise<StyleRecord | null>,
|
|
213
|
-
|
|
476
|
+
renderSprite(match).catch(() => null),
|
|
214
477
|
resolveHandle(match.did).catch(() => null),
|
|
215
478
|
]);
|
|
216
479
|
|
|
@@ -220,14 +483,36 @@ async function hydrateLoadedSim(match: SimMatch): Promise<LoadedSim> {
|
|
|
220
483
|
rkey: match.rkey,
|
|
221
484
|
name: match.sim.name,
|
|
222
485
|
handle,
|
|
223
|
-
spriteAnsi:
|
|
486
|
+
spriteAnsi: sprite?.ansi,
|
|
487
|
+
spritePng: sprite?.png
|
|
488
|
+
? {
|
|
489
|
+
base64: sprite.png.data.toString("base64"),
|
|
490
|
+
widthPx: sprite.png.widthPx,
|
|
491
|
+
heightPx: sprite.png.heightPx,
|
|
492
|
+
}
|
|
493
|
+
: undefined,
|
|
494
|
+
spriteFrames: sprite?.frames
|
|
495
|
+
? {
|
|
496
|
+
pngBase64: sprite.frames.pngs.map((b) => b.toString("base64")),
|
|
497
|
+
fps: sprite.frames.fps,
|
|
498
|
+
widthPx: sprite.frames.widthPx,
|
|
499
|
+
heightPx: sprite.frames.heightPx,
|
|
500
|
+
}
|
|
501
|
+
: undefined,
|
|
224
502
|
shortDescription: agents?.shortDescription,
|
|
225
503
|
description: agents?.description,
|
|
226
504
|
style: style?.description,
|
|
227
505
|
};
|
|
228
506
|
}
|
|
229
507
|
|
|
230
|
-
|
|
508
|
+
/**
|
|
509
|
+
* Build the bio text block that appears alongside (or below) the
|
|
510
|
+
* sprite in the loaded-sim message: name + handle + AT-URI +
|
|
511
|
+
* shortDescription. Indented two spaces so it lines up with the ANSI
|
|
512
|
+
* sprite render. Used both as a standalone block (Image+Text
|
|
513
|
+
* variant) and as the trailing portion of `formatSimSummary`.
|
|
514
|
+
*/
|
|
515
|
+
function formatSimBio(
|
|
231
516
|
sim: LoadedSim,
|
|
232
517
|
theme?: ExtensionContext["ui"]["theme"],
|
|
233
518
|
): string {
|
|
@@ -236,10 +521,6 @@ function formatSimSummary(
|
|
|
236
521
|
? (s: string) => theme.fg("accent", s)
|
|
237
522
|
: (s: string) => s;
|
|
238
523
|
const lines: string[] = [];
|
|
239
|
-
if (sim.spriteAnsi) {
|
|
240
|
-
lines.push(sim.spriteAnsi);
|
|
241
|
-
lines.push("");
|
|
242
|
-
}
|
|
243
524
|
lines.push(` 🐾 ${accent(sim.name)}${sim.handle ? dim(` @${sim.handle}`) : ""} loaded—pi is now in character.`);
|
|
244
525
|
lines.push(dim(` ${sim.uri}`));
|
|
245
526
|
if (sim.shortDescription) {
|
|
@@ -249,6 +530,19 @@ function formatSimSummary(
|
|
|
249
530
|
return lines.join("\n");
|
|
250
531
|
}
|
|
251
532
|
|
|
533
|
+
function formatSimSummary(
|
|
534
|
+
sim: LoadedSim,
|
|
535
|
+
theme?: ExtensionContext["ui"]["theme"],
|
|
536
|
+
): string {
|
|
537
|
+
const lines: string[] = [];
|
|
538
|
+
if (sim.spriteAnsi) {
|
|
539
|
+
lines.push(sim.spriteAnsi);
|
|
540
|
+
lines.push("");
|
|
541
|
+
}
|
|
542
|
+
lines.push(formatSimBio(sim, theme));
|
|
543
|
+
return lines.join("\n");
|
|
544
|
+
}
|
|
545
|
+
|
|
252
546
|
// The OpenTUI standalone animated viewer used to live here. It now ships
|
|
253
547
|
// alongside this file as `viewer.ts` for anyone who wants the full-window
|
|
254
548
|
// experience — run it manually with:
|
|
@@ -334,11 +628,90 @@ export default async function simocracy(pi: ExtensionAPI) {
|
|
|
334
628
|
|
|
335
629
|
// -------------------------------------------------------------------------
|
|
336
630
|
// Custom message renderer — shows the sprite + bio inline in the chat.
|
|
631
|
+
//
|
|
632
|
+
// Two render paths, picked per-call:
|
|
633
|
+
//
|
|
634
|
+
// 1. Inline graphics (preferred when supported). Terminals that
|
|
635
|
+
// advertise the Kitty graphics protocol (Kitty, Ghostty, WezTerm,
|
|
636
|
+
// Konsole) or iTerm2's inline-image protocol get a real PNG of
|
|
637
|
+
// the sprite via pi-tui's `Image` component, stacked above the
|
|
638
|
+
// bio text in a `Box`. Pixels are crisp, scaling is the
|
|
639
|
+
// terminal's job.
|
|
640
|
+
//
|
|
641
|
+
// 2. ANSI half-blocks (universal fallback). Everything else —
|
|
642
|
+
// Apple Terminal, tmux without passthrough, plain SSH, dumb
|
|
643
|
+
// pipes — falls back to the existing `▀`/`▄` half-block art.
|
|
644
|
+
//
|
|
645
|
+
// Override with `SIMOCRACY_INLINE_GRAPHICS=ansi` to force the
|
|
646
|
+
// half-block path even when the terminal supports inline graphics
|
|
647
|
+
// (handy for screenshots, demo recordings, or terminals that
|
|
648
|
+
// *advertise* support but render glitchily).
|
|
337
649
|
// -------------------------------------------------------------------------
|
|
338
|
-
|
|
650
|
+
const inlineGraphicsMode =
|
|
651
|
+
(process.env.SIMOCRACY_INLINE_GRAPHICS ?? "auto").toLowerCase();
|
|
652
|
+
pi.registerMessageRenderer<SimLoadedDetails>("simocracy_sim_loaded", (message, _opts, theme) => {
|
|
653
|
+
const details = (message.details as SimLoadedDetails | undefined) ?? {};
|
|
339
654
|
const body =
|
|
340
|
-
(message.
|
|
341
|
-
|
|
655
|
+
details.body ?? (typeof message.content === "string" ? message.content : "");
|
|
656
|
+
|
|
657
|
+
// Decide whether to use the inline-graphics path. Auto-detect by
|
|
658
|
+
// default; honour the env-var override either direction.
|
|
659
|
+
const caps = getCapabilities();
|
|
660
|
+
const wantGraphics =
|
|
661
|
+
inlineGraphicsMode === "auto"
|
|
662
|
+
? caps.images !== null
|
|
663
|
+
: inlineGraphicsMode === "kitty" || inlineGraphicsMode === "iterm2";
|
|
664
|
+
if (wantGraphics && details.spritePngBase64 && details.bioText) {
|
|
665
|
+
// Pi-tui's Image needs a fallback colour for terminals that
|
|
666
|
+
// claim image support but later fail to render — we use the
|
|
667
|
+
// theme's `dim` so the placeholder text is unobtrusive.
|
|
668
|
+
const imageTheme: ImageTheme = {
|
|
669
|
+
fallbackColor: theme.fg("dim", "") ? (s: string) => theme.fg("dim", s) : (s: string) => s,
|
|
670
|
+
};
|
|
671
|
+
// Animation handoff: if there's an active animation AND it's
|
|
672
|
+
// for this exact message (matching animationKey), pull the
|
|
673
|
+
// current frame's PNG + the stable Kitty image ID. The image
|
|
674
|
+
// ID makes successive frame transmissions replace each other
|
|
675
|
+
// in place rather than stacking. For everything else (older
|
|
676
|
+
// loaded-sim messages, sims without animation frames) we use
|
|
677
|
+
// the static idle PNG with no image ID, so they freeze
|
|
678
|
+
// gracefully on the first frame.
|
|
679
|
+
let pngBase64 = details.spritePngBase64;
|
|
680
|
+
let imageOptions: { maxWidthCells: number; imageId?: number } = {
|
|
681
|
+
maxWidthCells: spriteWidthCells,
|
|
682
|
+
};
|
|
683
|
+
let imageDims = {
|
|
684
|
+
widthPx: details.spritePngWidth ?? 0,
|
|
685
|
+
heightPx: details.spritePngHeight ?? 0,
|
|
686
|
+
};
|
|
687
|
+
if (
|
|
688
|
+
currentAnimation &&
|
|
689
|
+
details.animationKey &&
|
|
690
|
+
currentAnimation.key === details.animationKey
|
|
691
|
+
) {
|
|
692
|
+
pngBase64 = currentAnimation.frames[currentAnimation.currentFrame] ?? pngBase64;
|
|
693
|
+
imageOptions = {
|
|
694
|
+
maxWidthCells: spriteWidthCells,
|
|
695
|
+
imageId: currentAnimation.imageId,
|
|
696
|
+
};
|
|
697
|
+
imageDims = { widthPx: currentAnimation.widthPx, heightPx: currentAnimation.heightPx };
|
|
698
|
+
}
|
|
699
|
+
// Source PNG is at full native resolution (192×208 for codex
|
|
700
|
+
// pets, 32×32 for pipoya). The terminal scales it down to
|
|
701
|
+
// `spriteWidthCells` cells wide on display — we never
|
|
702
|
+
// pre-downsample on our side, so quality is preserved.
|
|
703
|
+
const image = new Image(
|
|
704
|
+
pngBase64,
|
|
705
|
+
"image/png",
|
|
706
|
+
imageTheme,
|
|
707
|
+
imageOptions,
|
|
708
|
+
imageDims,
|
|
709
|
+
);
|
|
710
|
+
const box = new Box(0, 0);
|
|
711
|
+
box.addChild(image);
|
|
712
|
+
box.addChild(new Text(details.bioText, 0, 0));
|
|
713
|
+
return box;
|
|
714
|
+
}
|
|
342
715
|
return new Text(body, 0, 0);
|
|
343
716
|
});
|
|
344
717
|
|
|
@@ -411,6 +784,7 @@ export default async function simocracy(pi: ExtensionAPI) {
|
|
|
411
784
|
const name = loadedSim.name;
|
|
412
785
|
loadedSim = null;
|
|
413
786
|
justUnloaded = name;
|
|
787
|
+
stopCurrentAnimation();
|
|
414
788
|
ctx.ui.setStatus("simocracy", undefined);
|
|
415
789
|
ctx.ui.setWidget("simocracy", undefined);
|
|
416
790
|
ctx.ui.notify(`Unloaded ${name}. Pi will break character on the next reply.`, "info");
|
|
@@ -500,6 +874,7 @@ export default async function simocracy(pi: ExtensionAPI) {
|
|
|
500
874
|
const name = loadedSim.name;
|
|
501
875
|
loadedSim = null;
|
|
502
876
|
justUnloaded = name;
|
|
877
|
+
stopCurrentAnimation();
|
|
503
878
|
if (ctx.hasUI) {
|
|
504
879
|
ctx.ui.setStatus("simocracy", undefined);
|
|
505
880
|
ctx.ui.setWidget("simocracy", undefined);
|
|
@@ -942,8 +1317,11 @@ async function tryLoadFromQuery(query: string): Promise<LoadedSim | null> {
|
|
|
942
1317
|
const { getRecordFromPds } = await import("./simocracy.ts");
|
|
943
1318
|
const sim = await getRecordFromPds<{
|
|
944
1319
|
name: string;
|
|
945
|
-
|
|
946
|
-
|
|
1320
|
+
spriteKind?: "pipoya" | "codexPet";
|
|
1321
|
+
image?: { ref: unknown; mimeType: string; size: number };
|
|
1322
|
+
sprite?: { ref: unknown; mimeType: string; size: number };
|
|
1323
|
+
petSheet?: { ref: unknown; mimeType: string; size: number };
|
|
1324
|
+
petManifest?: { id?: string; displayName?: string; description?: string };
|
|
947
1325
|
$type?: string;
|
|
948
1326
|
}>(did, "org.simocracy.sim", rkey);
|
|
949
1327
|
const match: SimMatch = {
|
|
@@ -954,9 +1332,12 @@ async function tryLoadFromQuery(query: string): Promise<LoadedSim | null> {
|
|
|
954
1332
|
sim: {
|
|
955
1333
|
$type: "org.simocracy.sim",
|
|
956
1334
|
name: sim.name,
|
|
1335
|
+
spriteKind: sim.spriteKind,
|
|
957
1336
|
settings: { selectedOptions: {} },
|
|
958
1337
|
image: sim.image as never,
|
|
959
1338
|
sprite: sim.sprite as never,
|
|
1339
|
+
petSheet: sim.petSheet as never,
|
|
1340
|
+
petManifest: sim.petManifest,
|
|
960
1341
|
createdAt: "",
|
|
961
1342
|
},
|
|
962
1343
|
};
|
|
@@ -970,6 +1351,41 @@ async function tryLoadFromQuery(query: string): Promise<LoadedSim | null> {
|
|
|
970
1351
|
return await hydrateLoadedSim(result.matches[0]);
|
|
971
1352
|
}
|
|
972
1353
|
|
|
1354
|
+
/**
|
|
1355
|
+
* Shape of `details` on the `simocracy_sim_loaded` custom message.
|
|
1356
|
+
* The renderer reads this to choose between the inline-graphics and
|
|
1357
|
+
* ANSI-half-block render paths; both fields are best-effort — the
|
|
1358
|
+
* renderer falls back to the combined `body` string if anything is
|
|
1359
|
+
* missing.
|
|
1360
|
+
*/
|
|
1361
|
+
interface SimLoadedDetails {
|
|
1362
|
+
uri?: string;
|
|
1363
|
+
did?: string;
|
|
1364
|
+
rkey?: string;
|
|
1365
|
+
name?: string;
|
|
1366
|
+
/** Combined ANSI sprite + bio, used by the half-block fallback path
|
|
1367
|
+
* and as the textual log content. */
|
|
1368
|
+
body?: string;
|
|
1369
|
+
/** Bio text only (no sprite), used alongside the Image component
|
|
1370
|
+
* on the inline-graphics path. */
|
|
1371
|
+
bioText?: string;
|
|
1372
|
+
/** base64-encoded PNG of the sprite cell. Triggers the inline-graphics
|
|
1373
|
+
* path when set + the terminal advertises image support. */
|
|
1374
|
+
spritePngBase64?: string;
|
|
1375
|
+
/** Native PNG width in pixels (aspect ratio for Image scaling). */
|
|
1376
|
+
spritePngWidth?: number;
|
|
1377
|
+
/** Native PNG height in pixels. */
|
|
1378
|
+
spritePngHeight?: number;
|
|
1379
|
+
/**
|
|
1380
|
+
* Identity tag for the active animation, if any. The message renderer
|
|
1381
|
+
* compares against `currentAnimation.key` to decide whether to swap
|
|
1382
|
+
* in the current animation frame or freeze on the static idle PNG.
|
|
1383
|
+
* Stable across re-renders of the same message; differs across
|
|
1384
|
+
* separate sim loads.
|
|
1385
|
+
*/
|
|
1386
|
+
animationKey?: string;
|
|
1387
|
+
}
|
|
1388
|
+
|
|
973
1389
|
async function postSimToChat(
|
|
974
1390
|
pi: ExtensionAPI,
|
|
975
1391
|
ctx: ExtensionContext,
|
|
@@ -977,21 +1393,54 @@ async function postSimToChat(
|
|
|
977
1393
|
_reload: boolean,
|
|
978
1394
|
) {
|
|
979
1395
|
ctx.ui.setStatus("simocracy", `🐾 ${sim.name}`);
|
|
980
|
-
|
|
981
|
-
|
|
1396
|
+
// Use the factory form of setWidget so we can capture pi-tui's TUI
|
|
1397
|
+
// handle. We need it to call `requestRender()` from the animation
|
|
1398
|
+
// setInterval (the message renderer doesn't get a TUI reference).
|
|
1399
|
+
// The factory itself returns a tiny static Text widget — the TUI
|
|
1400
|
+
// capture is the actual purpose.
|
|
1401
|
+
const headerText = `Simocracy: ${sim.name}${sim.handle ? ` (@${sim.handle})` : ""}`;
|
|
1402
|
+
ctx.ui.setWidget(
|
|
1403
|
+
"simocracy",
|
|
1404
|
+
(tui) => {
|
|
1405
|
+
capturedTui = tui;
|
|
1406
|
+
return new Text(headerText, 0, 0);
|
|
1407
|
+
},
|
|
1408
|
+
{ placement: "aboveEditor" },
|
|
1409
|
+
);
|
|
982
1410
|
const body = formatSimSummary(sim, ctx.ui.theme);
|
|
1411
|
+
const bioText = formatSimBio(sim, ctx.ui.theme);
|
|
1412
|
+
// Unique key per load so the renderer can tell which message owns
|
|
1413
|
+
// the active animation. AT-URI + timestamp protects against
|
|
1414
|
+
// re-loading the same sim twice (each load starts its own loop).
|
|
1415
|
+
const animationKey = `${sim.uri}#${Date.now()}`;
|
|
1416
|
+
const details: SimLoadedDetails = {
|
|
1417
|
+
uri: sim.uri,
|
|
1418
|
+
did: sim.did,
|
|
1419
|
+
rkey: sim.rkey,
|
|
1420
|
+
name: sim.name,
|
|
1421
|
+
body,
|
|
1422
|
+
bioText,
|
|
1423
|
+
animationKey,
|
|
1424
|
+
};
|
|
1425
|
+
if (sim.spritePng) {
|
|
1426
|
+
details.spritePngBase64 = sim.spritePng.base64;
|
|
1427
|
+
details.spritePngWidth = sim.spritePng.widthPx;
|
|
1428
|
+
details.spritePngHeight = sim.spritePng.heightPx;
|
|
1429
|
+
}
|
|
983
1430
|
pi.sendMessage({
|
|
984
1431
|
customType: "simocracy_sim_loaded",
|
|
985
1432
|
content: stripAnsiForLog(body),
|
|
986
1433
|
display: true,
|
|
987
|
-
details
|
|
988
|
-
uri: sim.uri,
|
|
989
|
-
did: sim.did,
|
|
990
|
-
rkey: sim.rkey,
|
|
991
|
-
name: sim.name,
|
|
992
|
-
body,
|
|
993
|
-
},
|
|
1434
|
+
details,
|
|
994
1435
|
});
|
|
1436
|
+
// Kick off (or replace) the animation loop. Only one loop runs at a
|
|
1437
|
+
// time — the last loaded sim animates, earlier messages freeze on
|
|
1438
|
+
// their idle frame.
|
|
1439
|
+
if (sim.spriteFrames) {
|
|
1440
|
+
startAnimationFor(animationKey, sim.spriteFrames);
|
|
1441
|
+
} else {
|
|
1442
|
+
stopCurrentAnimation();
|
|
1443
|
+
}
|
|
995
1444
|
}
|
|
996
1445
|
|
|
997
1446
|
/** Strip ANSI escapes for the textual log copy (the renderer uses details.body). */
|
package/src/persona.ts
CHANGED
|
@@ -18,6 +18,43 @@ export interface LoadedSim {
|
|
|
18
18
|
style?: string;
|
|
19
19
|
/** Pre-rendered colored ANSI art of the sim's sprite. */
|
|
20
20
|
spriteAnsi?: string;
|
|
21
|
+
/**
|
|
22
|
+
* Pre-encoded PNG of the sim's sprite (idle frame), for inline
|
|
23
|
+
* terminal graphics protocols (Kitty, iTerm2). Present when sprite
|
|
24
|
+
* rendering succeeded and the source could be encoded; the renderer
|
|
25
|
+
* uses this when the host terminal advertises image support and
|
|
26
|
+
* falls back to `spriteAnsi` otherwise.
|
|
27
|
+
*/
|
|
28
|
+
spritePng?: {
|
|
29
|
+
/** base64-encoded PNG bytes (no `data:` prefix). */
|
|
30
|
+
base64: string;
|
|
31
|
+
/** Native PNG width in pixels (used for aspect ratio). */
|
|
32
|
+
widthPx: number;
|
|
33
|
+
/** Native PNG height in pixels. */
|
|
34
|
+
heightPx: number;
|
|
35
|
+
};
|
|
36
|
+
/**
|
|
37
|
+
* Pre-encoded PNG frames of the sim's idle animation, for terminals
|
|
38
|
+
* that support inline graphics. When present **and** the renderer's
|
|
39
|
+
* animation timer is active for this sim, the message renderer
|
|
40
|
+
* cycles through these instead of repeating `spritePng` — producing
|
|
41
|
+
* a gentle in-chat idle loop using the same Kitty-protocol
|
|
42
|
+
* in-place image swap that pi-tui's spinner uses to animate.
|
|
43
|
+
*
|
|
44
|
+
* Currently populated only for `codexPet` sims (their atlas defines
|
|
45
|
+
* an explicit 6-frame idle row); pipoya and image-fallback sims
|
|
46
|
+
* stay static.
|
|
47
|
+
*/
|
|
48
|
+
spriteFrames?: {
|
|
49
|
+
/** base64-encoded PNG bytes per frame, in display order. */
|
|
50
|
+
pngBase64: string[];
|
|
51
|
+
/** Display rate in frames-per-second. */
|
|
52
|
+
fps: number;
|
|
53
|
+
/** Native PNG width in pixels (uniform across all frames). */
|
|
54
|
+
widthPx: number;
|
|
55
|
+
/** Native PNG height in pixels. */
|
|
56
|
+
heightPx: number;
|
|
57
|
+
};
|
|
21
58
|
}
|
|
22
59
|
|
|
23
60
|
/**
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Encode an RGBA pixel buffer to a PNG (`Buffer`) for inline terminal
|
|
3
|
+
* graphics protocols (Kitty, iTerm2). Both protocols accept base64-PNG
|
|
4
|
+
* payloads — Kitty supports it via `f=100`, iTerm2 via `inline=1`.
|
|
5
|
+
*
|
|
6
|
+
* Sized to be tiny: the only operation here is wrapping `pngjs`'s
|
|
7
|
+
* synchronous packer in a typed helper that matches the
|
|
8
|
+
* decoder/cropper interfaces in `png-to-ansi.ts`. We never need
|
|
9
|
+
* streaming or async — the largest image we encode is a 192×208 codex
|
|
10
|
+
* pet idle cell (~30 KB PNG).
|
|
11
|
+
*/
|
|
12
|
+
|
|
13
|
+
import { PNG } from "pngjs";
|
|
14
|
+
|
|
15
|
+
/**
|
|
16
|
+
* Encode a flat RGBA8 buffer to PNG. `data.length` must equal
|
|
17
|
+
* `width * height * 4`. Returns a fresh `Buffer` containing the PNG
|
|
18
|
+
* bytes (signature `89 50 4e 47 …`).
|
|
19
|
+
*/
|
|
20
|
+
export function encodeRgbaToPng(data: Buffer, width: number, height: number): Buffer {
|
|
21
|
+
if (data.length !== width * height * 4) {
|
|
22
|
+
throw new Error(
|
|
23
|
+
`encodeRgbaToPng: expected ${width * height * 4} bytes for ${width}×${height}, got ${data.length}`,
|
|
24
|
+
);
|
|
25
|
+
}
|
|
26
|
+
const png = new PNG({ width, height });
|
|
27
|
+
data.copy(png.data);
|
|
28
|
+
return PNG.sync.write(png);
|
|
29
|
+
}
|
package/src/png-to-ansi.ts
CHANGED
|
@@ -235,6 +235,85 @@ export function downscaleRgbaNearest(
|
|
|
235
235
|
return { data: out, width: newW, height: newH };
|
|
236
236
|
}
|
|
237
237
|
|
|
238
|
+
/**
|
|
239
|
+
* Box-average downscale an RGBA buffer to arbitrary target dimensions.
|
|
240
|
+
*
|
|
241
|
+
* Uses straight area-weighted averaging on premultiplied alpha so partly
|
|
242
|
+
* transparent edge pixels don't bleed black into the result. Designed for
|
|
243
|
+
* non-pixel-art inputs (codex pet idle thumbnails, codex pet sheet cells)
|
|
244
|
+
* where we want a smaller display size at non-integer ratios.
|
|
245
|
+
*
|
|
246
|
+
* If the target equals the source size, returns the input untouched.
|
|
247
|
+
*/
|
|
248
|
+
export function boxDownscaleRgba(
|
|
249
|
+
data: Buffer,
|
|
250
|
+
width: number,
|
|
251
|
+
height: number,
|
|
252
|
+
targetW: number,
|
|
253
|
+
targetH: number,
|
|
254
|
+
): { data: Buffer; width: number; height: number } {
|
|
255
|
+
if (targetW < 1 || targetH < 1) {
|
|
256
|
+
throw new Error(`boxDownscaleRgba target must be ≥1×1, got ${targetW}×${targetH}`);
|
|
257
|
+
}
|
|
258
|
+
if (targetW === width && targetH === height) {
|
|
259
|
+
return { data, width, height };
|
|
260
|
+
}
|
|
261
|
+
const out = Buffer.alloc(targetW * targetH * 4);
|
|
262
|
+
// For each output pixel, average the source rectangle that maps to it.
|
|
263
|
+
// Edge cells are clamped to the source extent.
|
|
264
|
+
for (let oy = 0; oy < targetH; oy++) {
|
|
265
|
+
const sy0 = (oy * height) / targetH;
|
|
266
|
+
const sy1 = ((oy + 1) * height) / targetH;
|
|
267
|
+
const y0 = Math.floor(sy0);
|
|
268
|
+
const y1 = Math.min(height, Math.ceil(sy1));
|
|
269
|
+
for (let ox = 0; ox < targetW; ox++) {
|
|
270
|
+
const sx0 = (ox * width) / targetW;
|
|
271
|
+
const sx1 = ((ox + 1) * width) / targetW;
|
|
272
|
+
const x0 = Math.floor(sx0);
|
|
273
|
+
const x1 = Math.min(width, Math.ceil(sx1));
|
|
274
|
+
|
|
275
|
+
// Premultiplied accumulation so transparent pixels don't smear
|
|
276
|
+
// black RGB values into mostly-opaque neighbours.
|
|
277
|
+
let rSum = 0,
|
|
278
|
+
gSum = 0,
|
|
279
|
+
bSum = 0,
|
|
280
|
+
aSum = 0,
|
|
281
|
+
wSum = 0;
|
|
282
|
+
for (let y = y0; y < y1; y++) {
|
|
283
|
+
// Vertical coverage of this source row inside the output cell.
|
|
284
|
+
const wy = Math.min(y + 1, sy1) - Math.max(y, sy0);
|
|
285
|
+
if (wy <= 0) continue;
|
|
286
|
+
for (let x = x0; x < x1; x++) {
|
|
287
|
+
const wx = Math.min(x + 1, sx1) - Math.max(x, sx0);
|
|
288
|
+
if (wx <= 0) continue;
|
|
289
|
+
const w = wx * wy;
|
|
290
|
+
const i = (y * width + x) * 4;
|
|
291
|
+
const a = data[i + 3];
|
|
292
|
+
const aw = a * w;
|
|
293
|
+
rSum += data[i] * aw;
|
|
294
|
+
gSum += data[i + 1] * aw;
|
|
295
|
+
bSum += data[i + 2] * aw;
|
|
296
|
+
aSum += aw;
|
|
297
|
+
wSum += w;
|
|
298
|
+
}
|
|
299
|
+
}
|
|
300
|
+
const di = (oy * targetW + ox) * 4;
|
|
301
|
+
if (aSum > 0) {
|
|
302
|
+
out[di] = Math.min(255, Math.round(rSum / aSum));
|
|
303
|
+
out[di + 1] = Math.min(255, Math.round(gSum / aSum));
|
|
304
|
+
out[di + 2] = Math.min(255, Math.round(bSum / aSum));
|
|
305
|
+
out[di + 3] = Math.min(255, Math.round(aSum / wSum));
|
|
306
|
+
} else {
|
|
307
|
+
out[di] = 0;
|
|
308
|
+
out[di + 1] = 0;
|
|
309
|
+
out[di + 2] = 0;
|
|
310
|
+
out[di + 3] = 0;
|
|
311
|
+
}
|
|
312
|
+
}
|
|
313
|
+
}
|
|
314
|
+
return { data: out, width: targetW, height: targetH };
|
|
315
|
+
}
|
|
316
|
+
|
|
238
317
|
/**
|
|
239
318
|
* Detect the integer upscale factor of a pixel-art image by scanning
|
|
240
319
|
* for the largest factor F where every F×F block has uniform colour.
|
package/src/simocracy.ts
CHANGED
|
@@ -24,13 +24,33 @@ export interface BlobRef {
|
|
|
24
24
|
size: number;
|
|
25
25
|
}
|
|
26
26
|
|
|
27
|
+
/** Which sprite system a sim uses. Absent = legacy 'pipoya'. */
|
|
28
|
+
export type SpriteKind = "pipoya" | "codexPet";
|
|
29
|
+
|
|
30
|
+
/** Subset of pet.json from the OpenAI hatch-pet skill output. */
|
|
31
|
+
export interface CodexPetManifest {
|
|
32
|
+
id?: string;
|
|
33
|
+
displayName?: string;
|
|
34
|
+
description?: string;
|
|
35
|
+
}
|
|
36
|
+
|
|
27
37
|
export interface SimRecord {
|
|
28
38
|
$type: "org.simocracy.sim";
|
|
29
39
|
name: string;
|
|
30
|
-
|
|
40
|
+
/** Discriminator for sprite system. Absent = 'pipoya'. */
|
|
41
|
+
spriteKind?: SpriteKind;
|
|
42
|
+
/** Pipoya appearance settings (only used when spriteKind = 'pipoya'). */
|
|
43
|
+
settings?: SpriteSettings;
|
|
44
|
+
/** Rendered avatar PNG/JPEG/WebP thumbnail (always present for codex pets, generated client-side as a 128×128 PNG). */
|
|
31
45
|
image?: BlobRef;
|
|
46
|
+
/** Pipoya 4×4 walk sheet (128×128 PNG). Only present when spriteKind = 'pipoya'. */
|
|
32
47
|
sprite?: BlobRef;
|
|
48
|
+
/** Codex pet 8×9 atlas (1536×1872, 192×208 cells, PNG or WebP). Only present when spriteKind = 'codexPet'. */
|
|
49
|
+
petSheet?: BlobRef;
|
|
50
|
+
/** Subset of pet.json from the hatch-pet skill output. Only present when spriteKind = 'codexPet'. */
|
|
51
|
+
petManifest?: CodexPetManifest;
|
|
33
52
|
createdAt: string;
|
|
53
|
+
updatedAt?: string;
|
|
34
54
|
}
|
|
35
55
|
|
|
36
56
|
export interface AgentsRecord {
|
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* WebP → RGBA decoder for Node.
|
|
3
|
+
*
|
|
4
|
+
* `pngjs` covers the PNG side of pi-simocracy, but the OpenAI hatch-pet
|
|
5
|
+
* skill (and therefore most `org.simocracy.sim` records with
|
|
6
|
+
* `spriteKind = "codexPet"`) ships its 1536×1872 atlas as WebP. We need
|
|
7
|
+
* a WebP decoder to render the idle frame.
|
|
8
|
+
*
|
|
9
|
+
* @jsquash/webp's `decode()` is browser-shaped: in Node its wasm glue
|
|
10
|
+
* tries to `fetch()` its own `.wasm` URL, which Undici rejects for
|
|
11
|
+
* `file://`. Workaround: read the wasm bytes from disk via
|
|
12
|
+
* `createRequire().resolve()`, compile to a `WebAssembly.Module`, and
|
|
13
|
+
* feed it to the package's documented `init()` escape hatch. Fully ESM,
|
|
14
|
+
* no native bindings.
|
|
15
|
+
*
|
|
16
|
+
* The wasm module is initialised lazily on the first `decodeWebp()`
|
|
17
|
+
* call so loading pi-simocracy stays cheap when no one ever loads a
|
|
18
|
+
* codex pet sim. Subsequent calls reuse the same module.
|
|
19
|
+
*/
|
|
20
|
+
|
|
21
|
+
import { createRequire } from "node:module";
|
|
22
|
+
import { readFile } from "node:fs/promises";
|
|
23
|
+
import decode, { init as initWebpDecode } from "@jsquash/webp/decode.js";
|
|
24
|
+
|
|
25
|
+
let wasmInitPromise: Promise<void> | null = null;
|
|
26
|
+
|
|
27
|
+
async function ensureWasmInit(): Promise<void> {
|
|
28
|
+
if (wasmInitPromise) return wasmInitPromise;
|
|
29
|
+
wasmInitPromise = (async () => {
|
|
30
|
+
const require = createRequire(import.meta.url);
|
|
31
|
+
// @jsquash/webp ships the decoder wasm next to its JS glue; the
|
|
32
|
+
// package only re-exports JS files, so we have to resolve the wasm
|
|
33
|
+
// path manually. Resolving against the JS entry guarantees we hit
|
|
34
|
+
// the same install of the package the bundler/linker picked.
|
|
35
|
+
const decodeJs = require.resolve("@jsquash/webp/decode.js");
|
|
36
|
+
const wasmPath = decodeJs.replace(/decode\.js$/, "codec/dec/webp_dec.wasm");
|
|
37
|
+
const bytes = await readFile(wasmPath);
|
|
38
|
+
const mod = await WebAssembly.compile(bytes);
|
|
39
|
+
await initWebpDecode(mod);
|
|
40
|
+
})().catch((err) => {
|
|
41
|
+
// Reset so the next caller can retry; otherwise a transient FS
|
|
42
|
+
// error would permanently break codex-pet rendering for the session.
|
|
43
|
+
wasmInitPromise = null;
|
|
44
|
+
throw new Error(`Failed to init @jsquash/webp wasm: ${(err as Error).message}`);
|
|
45
|
+
});
|
|
46
|
+
return wasmInitPromise;
|
|
47
|
+
}
|
|
48
|
+
|
|
49
|
+
/**
|
|
50
|
+
* Decode a WebP buffer into the same flat-RGBA shape `decodePng()`
|
|
51
|
+
* returns, so call sites can treat the two interchangeably.
|
|
52
|
+
*
|
|
53
|
+
* Allocates a fresh `Buffer` rather than aliasing `Uint8ClampedArray`
|
|
54
|
+
* so downstream `Buffer`-only helpers (`pixelAt`, `cropRgba`,
|
|
55
|
+
* `boxDownscaleRgba`) keep working without coercion.
|
|
56
|
+
*/
|
|
57
|
+
export async function decodeWebp(
|
|
58
|
+
buf: Buffer,
|
|
59
|
+
): Promise<{ width: number; height: number; data: Buffer }> {
|
|
60
|
+
await ensureWasmInit();
|
|
61
|
+
// jSquash wants an ArrayBuffer view of just the relevant bytes —
|
|
62
|
+
// passing the whole underlying buffer mis-decodes if `buf` is a slice.
|
|
63
|
+
const ab = buf.buffer.slice(buf.byteOffset, buf.byteOffset + buf.byteLength);
|
|
64
|
+
const img = await decode(ab as ArrayBuffer);
|
|
65
|
+
return {
|
|
66
|
+
width: img.width,
|
|
67
|
+
height: img.height,
|
|
68
|
+
data: Buffer.from(img.data.buffer, img.data.byteOffset, img.data.byteLength),
|
|
69
|
+
};
|
|
70
|
+
}
|