@oh-my-pi/snapcompact 16.2.6 → 16.2.8

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -2,6 +2,25 @@
2
2
 
3
3
  ## [Unreleased]
4
4
 
5
+ ## [16.2.8] - 2026-06-30
6
+
7
+ ### Fixed
8
+
9
+ - Fixed large snapcompact archives being reconstructed into unbounded per-request image payloads by adding a frame base64 byte budget and omitting over-budget archive frames from prompt blocks. ([#3792](https://github.com/can1357/oh-my-pi/issues/3792))
10
+
11
+ ## [16.2.7] - 2026-06-30
12
+
13
+ ### Added
14
+
15
+ - Added the `silver16-bw` shape backed by an embedded Silver TrueType font to support CJK and other non-Latin text.
16
+ - Added `resolveShapeForText` to support font-aware shape resolution.
17
+
18
+ ### Changed
19
+
20
+ - Improved non-ASCII text normalization by folding semantic emojis to ASCII labels (e.g., `[OK]`, `[WARN]`), dropping decorative emojis, and folding box-drawing symbols to ASCII skeletons.
21
+ - Enhanced missing glyph rendering to use the embedded Silver TrueType fallback per-character, including support for East Asian wide characters across two grid cells.
22
+ - Updated text wrapping, pagination, and provider shape geometries to support wide character footprints and updated X.org 8x13 font metrics.
23
+
5
24
  ## [16.1.23] - 2026-06-26
6
25
 
7
26
  ### Added
package/README.md CHANGED
@@ -9,7 +9,7 @@ Built for [oh-my-pi](https://github.com/can1357/oh-my-pi)'s compaction pipeline,
9
9
  ## How it works
10
10
 
11
11
  1. Discarded history is serialized to compact text (`serializeConversation`), with per-tool-result and per-argument character caps.
12
- 2. Text is normalized for the bundled bitmap fonts (`normalize`): ANSI sequences stripped, whitespace collapsed, newline runs folded into a single full-block glyph so line structure survives.
12
+ 2. Text is normalized for the selected native font (`normalize`): ANSI sequences stripped, whitespace collapsed, newline runs folded into a single full-block glyph, box drawing and compatibility symbols folded to ASCII, semantic emoji folded to ASCII labels, decorative emoji dropped, and non-Latin glyphs preserved when either the selected font or the embedded Silver fallback can render them.
13
13
  3. Pages of text are rasterized into PNG frames (`render` / `renderMany`). Frame width is fixed per shape; height hugs the rows actually printed, so a partially filled frame never bills blank pixel rows.
14
14
  4. Frames persist in the compaction entry's `preserveData` and are re-attached to the summary message on every context rebuild.
15
15
 
@@ -17,13 +17,15 @@ Frame shapes are provider-aware, chosen by SQuAD recall evals (see `research/`)
17
17
 
18
18
  | Reader | Default shape | Notes |
19
19
  | --- | --- | --- |
20
- | Anthropic | `6x12-dim` | X.org 6x12 glyphs, stopwords dimmed gray; high-res Claude lines get 1932px frames |
21
- | Google | `doc-8on16-sent-dim` @2048 | Two newspaper columns, sentence-hue ink; Gemini bills a fixed per-image budget, so larger frames are free chars |
22
- | OpenAI | `8on16-bw` | 8x13 glyphs on a patch-aligned 16px pitch, sent at `detail: "original"` |
20
+ | Anthropic | `11on16-bw` | X.org 8x13 glyphs on an 11px advance; high-res Claude lines get 1932px frames |
21
+ | Google | `8on22-bw` @2048 | X.org 8x13 glyphs on a 22px pitch; Gemini bills a fixed per-image budget, so larger frames are free chars |
22
+ | OpenAI | `8on22-bw` | X.org 8x13 glyphs on a 22px pitch, sent at `detail: "original"` |
23
23
  | Unknown | Anthropic shape | Per-provider image-count budgets guard against gateways that silently drop frames |
24
24
 
25
25
  `resolveShape({ api, id })` matches the model id, not just the wire API — a Claude routed through Vertex or OpenRouter keeps its Claude shape, priced for the gateway actually carrying the request.
26
26
 
27
+ Bitmap shapes keep their provider-tuned geometry and draw missing glyphs through the embedded Silver TrueType fallback one character at a time; East Asian (CJK/Kana/Hangul) glyphs render full-width across two cells so they stay legible in the narrow ASCII grid. Selecting `silver16-bw` uses Silver for the whole frame.
28
+
27
29
  ## Install
28
30
 
29
31
  ```sh
@@ -58,8 +60,8 @@ const result = await compact(preparation, { model });
58
60
 
59
61
  - **Compaction**: `compact`, `CompactionPreparation`, `CompactionResult`, `getPreservedArchive`, `images`, `historyBlocks`
60
62
  - **Rendering**: `render`, `renderMany`, `frames`, `geometry`
61
- - **Shapes**: `SHAPES`, `SHAPE_VARIANTS`, `resolveShape`, `idealShapeVariant`, `isShape`, `isShapeVariantName`
62
- - **Text**: `serializeConversation`, `normalize`, `dimStopwords`, `wrap`
63
+ - **Shapes**: `SHAPES`, `SHAPE_VARIANTS`, `resolveShape`, `resolveShapeForText`, `idealShapeVariant`, `isShape`, `isShapeVariantName`
64
+ - **Text**: `serializeConversation`, `normalize`, `scanRenderability`, `renderabilityProbeText`, `dimStopwords`, `wrap`
63
65
  - **Budgets**: `providerImageBudget`, `MAX_FRAMES_DEFAULT`, `FRAME_TOKEN_ESTIMATE`, `HQ_EDGE_FRAMES`
64
66
  - **File ops**: `createFileOps`, `computeFileLists`, `upsertFileOperations`
65
67
 
@@ -43,11 +43,11 @@
43
43
  * Frames persist in the compaction entry's `preserveData` and are
44
44
  * re-attached to the compaction summary message on every context rebuild.
45
45
  */
46
- import type { Api, ImageContent, Message, Model, TextContent } from "@oh-my-pi/pi-ai";
46
+ import type { Api, ImageContent, Message, TextContent } from "@oh-my-pi/pi-ai";
47
47
  /** One eval-validated frame shape: font, cell, ink, repetition, and size. */
48
48
  export interface Shape {
49
49
  /** Bundled font in the native renderer. */
50
- font: "5x8" | "8x8" | "6x12" | "8x13";
50
+ font: "5x8" | "8x8" | "6x12" | "8x13" | "silver";
51
51
  /** Target cell advance in pixels; differing from the font's natural cell
52
52
  * renders via Lanczos stretch (anti-aliased RGB frame). */
53
53
  cellWidth: number;
@@ -84,9 +84,10 @@ export type ShapeGeometry = Omit<Shape, "frameTokenEstimate" | "imageDetail">;
84
84
  * and `8x13` the X.org misc fonts, `8on16` 8x13 glyphs on an 8x16 cell pitch
85
85
  * (no stretch, extra leading), `8on22` the same glyphs on a 22px pitch (more
86
86
  * leading), `11on16` the same glyphs on an 11px advance (more tracking),
87
- * `doc-` prefixed shapes a two-column word-wrapped newspaper layout. Ink:
88
- * `sent` cycles six hues at sentence boundaries, `bw` is plain black, `-dim`
89
- * suffix prints stopwords in gray.
87
+ * `silver16` the embedded Silver TrueType font on a 16px grid for CJK and
88
+ * other non-Latin text, and `doc-` prefixed shapes a two-column word-wrapped
89
+ * newspaper layout. Ink: `sent` cycles six hues at sentence boundaries, `bw`
90
+ * is plain black, `-dim` suffix prints stopwords in gray.
90
91
  */
91
92
  export declare const SHAPE_VARIANTS: {
92
93
  readonly "8x8r-bw": {
@@ -197,6 +198,14 @@ export declare const SHAPE_VARIANTS: {
197
198
  readonly lineRepeat: 1;
198
199
  readonly frameSize: 1568;
199
200
  };
201
+ readonly "silver16-bw": {
202
+ readonly font: "silver";
203
+ readonly cellWidth: 16;
204
+ readonly cellHeight: 16;
205
+ readonly variant: "bw";
206
+ readonly lineRepeat: 1;
207
+ readonly frameSize: 1568;
208
+ };
200
209
  readonly "doc-8on16-bw": {
201
210
  readonly font: "8x13";
202
211
  readonly cellWidth: 8;
@@ -280,6 +289,14 @@ export interface ShapeTarget {
280
289
  * `Model` or any `{ api, id }` subset.
281
290
  */
282
291
  export declare function resolveShape(model?: ShapeTarget, variant?: ShapeVariantName | "auto"): Shape;
292
+ /**
293
+ * Pick the frame shape for `text` without changing the selected shape.
294
+ *
295
+ * Glyph-level Silver fallback happens during normalization/rendering, so this
296
+ * helper exists for callers that need a text-aware API name while preserving
297
+ * explicit and provider-selected shapes.
298
+ */
299
+ export declare function resolveShapeForText(_text: string, model?: ShapeTarget, variant?: ShapeVariantName | "auto"): Shape;
283
300
  /** Legacy frame edge in pixels (the 5x8 shape's eval-validated size). New
284
301
  * shapes carry their own `frameSize`. */
285
302
  export declare const FRAME_SIZE = 2576;
@@ -300,6 +317,21 @@ export declare const HQ_EDGE_FRAMES = 3;
300
317
  * cap, billed at +5% margin (ceil(4784 * 1.05)). Keeps the overflow guard from
301
318
  * undercounting a high-res archive at the raised {@link MAX_FRAMES_DEFAULT}. */
302
319
  export declare const FRAME_TOKEN_ESTIMATE = 5024;
320
+ /** Conservative upper bound for one persisted frame's base64 payload. The
321
+ * measured high-res Anthropic `8x13`/`11on16` PNG frames sit around 159 KB;
322
+ * 170 KB leaves margin for denser glyph pages without permitting multi-MB
323
+ * standing request bodies at large context windows. */
324
+ export declare const FRAME_DATA_BYTES_ESTIMATE = 170000;
325
+ /** Maximum snapcompact image base64 carried in every rebuilt provider request.
326
+ * Above this, provider backends can accept the HTTP body but fail mid-stream
327
+ * with opaque 5xx errors. Keep this independent from visual-token budgeting:
328
+ * a 1M-token model can afford 70 images on paper, but not the resulting
329
+ * ~11 MB JSON payload on every turn. */
330
+ export declare const FRAME_DATA_BYTES_BUDGET = 3000000;
331
+ /** Frame-count cap implied by {@link FRAME_DATA_BYTES_BUDGET}. */
332
+ export declare function maxFramesForDataBudget(maxFrameDataBytes?: number): number;
333
+ /** Base64 byte length for persisted snapcompact frames. */
334
+ export declare function frameDataBytes(frames: readonly Pick<Frame, "data">[]): number;
303
335
  /**
304
336
  * Per-request image-count budgets by provider id. These cap how many images an
305
337
  * entire request may carry (archive/system-prompt/tool-result imaging combined).
@@ -365,8 +397,8 @@ export interface Geometry {
365
397
  export interface Options<TMessage = Message> extends SerializeOptions {
366
398
  /** App-level message transformer (same contract as agent-core's `SummaryOptions.convertToLlm`). */
367
399
  convertToLlm?: ConvertToLlm<TMessage>;
368
- /** Model whose provider API selects the frame shape. */
369
- model?: Pick<Model, "api">;
400
+ /** Model whose provider API and id select the frame shape. */
401
+ model?: ShapeTarget;
370
402
  /** Explicit shape override; wins over `model`. */
371
403
  shape?: Shape;
372
404
  /** Frame edge in pixels. Defaults to the shape's `frameSize`. */
@@ -454,27 +486,24 @@ export declare function serializeConversation(messages: Message[], options?: Ser
454
486
  * entirely with pitch-black ink, so line structure survives whitespace
455
487
  * collapsing at a one-cell cost. */
456
488
  export declare const NEWLINE_GLYPH = "\u2588";
489
+ export interface NormalizeOptions {
490
+ /** Shape whose font is tried before the embedded Silver fallback. */
491
+ shape?: Pick<Shape, "font">;
492
+ /** Native font name when a full shape is not available. */
493
+ font?: Shape["font"];
494
+ }
457
495
  /**
458
496
  * Prepare text for printing: strip ANSI escape sequences, collapse horizontal
459
- * whitespace runs to single spaces and newline-bearing runs to one
460
- * {@link NEWLINE_GLYPH} (drawn as a pitch-black cell), then fold everything
461
- * outside the fonts' ASCII + Latin-1 coverage to ASCII approximations — first
462
- * through the {@link CHAR_FOLD} punctuation table, then via an NFKD
463
- * decomposition that recovers the ASCII skeleton of compatibility characters
464
- * (fullwidth, super/subscripts, ligatures, circled/math-styled alphanumerics,
465
- * Roman numerals, vulgar fractions). Unrenderable control/format/combining
466
- * characters are dropped without occupying a cell; `?` remains the fallback
467
- * for unsupported graphic characters. The zero-width ink toggles
468
- * {@link DIM_ON}/{@link DIM_OFF} pass through untouched.
497
+ * whitespace runs, fold unsupported symbols (including box drawing to ASCII),
498
+ * preserve Unicode glyphs that either the selected font or embedded Silver
499
+ * fallback can render, and drop decorative emoji instead of printing `?`.
469
500
  */
470
- export declare function normalize(text: string): string;
501
+ export declare function normalize(text: string, options?: NormalizeOptions): string;
471
502
  /**
472
- * Scan text to determine the proportion of graphic characters that will hit the
473
- * `?` fallback during {@link normalize}. Used as a preflight check to abort
474
- * snapcompact and fall back to the text summarizer when the input is heavily
475
- * non-renderable (e.g., CJK).
503
+ * Scan text with the same font-aware path as {@link normalize}; unsafe means
504
+ * more than 5% of graphic characters would hit the `?` fallback.
476
505
  */
477
- export declare function scanRenderability(text: string): {
506
+ export declare function scanRenderability(text: string, options?: NormalizeOptions): {
478
507
  isSafe: boolean;
479
508
  unrenderableRatio: number;
480
509
  };
@@ -492,7 +521,7 @@ export declare function dimStopwords(text: string): string;
492
521
  * markers count toward word length here; serialized history places them at
493
522
  * word boundaries, so the drift is at most one cell per affected line.
494
523
  */
495
- export declare function wrap(text: string, width: number): string[];
524
+ export declare function wrap(text: string, width: number, wideCells?: boolean): string[];
496
525
  export declare function geometry(shape: Shape, size?: number): Geometry;
497
526
  /** Render one snapcompact frame from already-normalized text. Doc shapes
498
527
  * (`columns === 2`) expect one page of `\n`-joined pre-wrapped lines. */
@@ -501,8 +530,8 @@ export declare function render(text: string, shape: Shape, size?: number): Promi
501
530
  export interface RenderManyOptions {
502
531
  /** Explicit shape; wins over `model`. */
503
532
  shape?: Shape;
504
- /** Model whose `api` selects the eval-optimal shape. */
505
- model?: Pick<Model, "api">;
533
+ /** Model whose provider API and id select the frame shape. */
534
+ model?: ShapeTarget;
506
535
  /** Frame edge in px; defaults to the shape's `frameSize`. */
507
536
  frameSize?: number;
508
537
  /** Hard cap on frames produced; omit for unbounded (caller decides usage). */
@@ -527,13 +556,20 @@ export declare function getPreservedArchive(preserveData: Record<string, unknown
527
556
  export declare function stripPreservedArchive(preserveData: Record<string, unknown> | undefined): Record<string, unknown> | undefined;
528
557
  /** Extract persisted archive source text as plain text for LLM summarization. */
529
558
  export declare function archiveSourceText(archive: Archive): string | undefined;
559
+ /** Build the text used to choose and preflight a font-aware snapcompact shape. */
560
+ export declare function renderabilityProbeText(serialized: string, previousPreserveData?: Record<string, unknown>, previousSummary?: string): string;
561
+ /** Options for reconstructing a persisted snapcompact archive into prompt blocks. */
562
+ export interface HistoryBlockOptions {
563
+ /** Hard cap on image base64 bytes attached to one rebuilt provider request. */
564
+ maxFrameDataBytes?: number;
565
+ }
530
566
  /** Convert archive frames into LLM image blocks (oldest first). */
531
567
  export declare function images(archive: Archive): ImageContent[];
532
568
  /** Ordered archive blocks for a compaction summary message, oldest to newest:
533
569
  * the oldest text region, the imaged middle, then the newest text region.
534
570
  * Runtime-only; reconstructed from {@link Archive} on each context rebuild
535
571
  * instead of persisted on the session entry. */
536
- export declare function historyBlocks(archive: Archive): (TextContent | ImageContent)[];
572
+ export declare function historyBlocks(archive: Archive, options?: HistoryBlockOptions): (TextContent | ImageContent)[];
537
573
  /**
538
574
  * Run a snapcompact compaction over prepared messages. Fully local: serializes
539
575
  * the discarded history, appends it to the accumulated archive source text, and
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "type": "module",
3
3
  "name": "@oh-my-pi/snapcompact",
4
- "version": "16.2.6",
4
+ "version": "16.2.8",
5
5
  "description": "Bitmap-frame context compression for vision-capable LLMs",
6
6
  "homepage": "https://omp.sh",
7
7
  "author": "Can Boluk",
@@ -31,10 +31,10 @@
31
31
  "fmt": "biome format --write ."
32
32
  },
33
33
  "dependencies": {
34
- "@oh-my-pi/pi-ai": "16.2.6",
35
- "@oh-my-pi/pi-natives": "16.2.6",
36
- "@oh-my-pi/pi-utils": "16.2.6",
37
- "@oh-my-pi/pi-wire": "16.2.6"
34
+ "@oh-my-pi/pi-ai": "16.2.8",
35
+ "@oh-my-pi/pi-natives": "16.2.8",
36
+ "@oh-my-pi/pi-utils": "16.2.8",
37
+ "@oh-my-pi/pi-wire": "16.2.8"
38
38
  },
39
39
  "devDependencies": {
40
40
  "@types/bun": "^1.3.14"
@@ -44,8 +44,8 @@
44
44
  * re-attached to the compaction summary message on every context rebuild.
45
45
  */
46
46
 
47
- import type { Api, ImageContent, Message, Model, TextContent } from "@oh-my-pi/pi-ai";
48
- import { renderSnapcompactPng } from "@oh-my-pi/pi-natives";
47
+ import type { Api, ImageContent, Message, TextContent } from "@oh-my-pi/pi-ai";
48
+ import { renderSnapcompactPng, snapcompactSupportedChars } from "@oh-my-pi/pi-natives";
49
49
  import { formatGroupedPaths, prompt } from "@oh-my-pi/pi-utils";
50
50
  import { INTENT_FIELD } from "@oh-my-pi/pi-wire";
51
51
  import fileOperationsTemplate from "./prompts/file-operations.md" with { type: "text" };
@@ -58,7 +58,7 @@ import snapcompactSummaryPrompt from "./prompts/snapcompact-summary.md" with { t
58
58
  /** One eval-validated frame shape: font, cell, ink, repetition, and size. */
59
59
  export interface Shape {
60
60
  /** Bundled font in the native renderer. */
61
- font: "5x8" | "8x8" | "6x12" | "8x13";
61
+ font: "5x8" | "8x8" | "6x12" | "8x13" | "silver";
62
62
  /** Target cell advance in pixels; differing from the font's natural cell
63
63
  * renders via Lanczos stretch (anti-aliased RGB frame). */
64
64
  cellWidth: number;
@@ -97,9 +97,10 @@ export type ShapeGeometry = Omit<Shape, "frameTokenEstimate" | "imageDetail">;
97
97
  * and `8x13` the X.org misc fonts, `8on16` 8x13 glyphs on an 8x16 cell pitch
98
98
  * (no stretch, extra leading), `8on22` the same glyphs on a 22px pitch (more
99
99
  * leading), `11on16` the same glyphs on an 11px advance (more tracking),
100
- * `doc-` prefixed shapes a two-column word-wrapped newspaper layout. Ink:
101
- * `sent` cycles six hues at sentence boundaries, `bw` is plain black, `-dim`
102
- * suffix prints stopwords in gray.
100
+ * `silver16` the embedded Silver TrueType font on a 16px grid for CJK and
101
+ * other non-Latin text, and `doc-` prefixed shapes a two-column word-wrapped
102
+ * newspaper layout. Ink: `sent` cycles six hues at sentence boundaries, `bw`
103
+ * is plain black, `-dim` suffix prints stopwords in gray.
103
104
  */
104
105
  export const SHAPE_VARIANTS = {
105
106
  "8x8r-bw": { font: "8x8", cellWidth: 8, cellHeight: 8, variant: "bw", lineRepeat: 2, frameSize: 1568 },
@@ -147,6 +148,14 @@ export const SHAPE_VARIANTS = {
147
148
  lineRepeat: 1,
148
149
  frameSize: 1568,
149
150
  },
151
+ "silver16-bw": {
152
+ font: "silver",
153
+ cellWidth: 16,
154
+ cellHeight: 16,
155
+ variant: "bw",
156
+ lineRepeat: 1,
157
+ frameSize: 1568,
158
+ },
150
159
  "doc-8on16-bw": {
151
160
  font: "8x13",
152
161
  cellWidth: 8,
@@ -271,7 +280,7 @@ export function isShape(value: unknown): value is Shape {
271
280
  const variant = shape.variant;
272
281
  const detail = shape.imageDetail;
273
282
  return (
274
- (font === "5x8" || font === "8x8" || font === "6x12" || font === "8x13") &&
283
+ (font === "5x8" || font === "8x8" || font === "6x12" || font === "8x13" || font === "silver") &&
275
284
  typeof shape.cellWidth === "number" &&
276
285
  shape.cellWidth > 0 &&
277
286
  typeof shape.cellHeight === "number" &&
@@ -376,6 +385,17 @@ export function resolveShape(model?: ShapeTarget, variant?: ShapeVariantName | "
376
385
  return priceShape(ideal?.frameSize ? { ...base, frameSize: ideal.frameSize } : base, family);
377
386
  }
378
387
 
388
+ /**
389
+ * Pick the frame shape for `text` without changing the selected shape.
390
+ *
391
+ * Glyph-level Silver fallback happens during normalization/rendering, so this
392
+ * helper exists for callers that need a text-aware API name while preserving
393
+ * explicit and provider-selected shapes.
394
+ */
395
+ export function resolveShapeForText(_text: string, model?: ShapeTarget, variant?: ShapeVariantName | "auto"): Shape {
396
+ return resolveShape(model, variant);
397
+ }
398
+
379
399
  // ============================================================================
380
400
  // Constants
381
401
  // ============================================================================
@@ -404,6 +424,29 @@ export const HQ_EDGE_FRAMES = 3;
404
424
  * undercounting a high-res archive at the raised {@link MAX_FRAMES_DEFAULT}. */
405
425
  export const FRAME_TOKEN_ESTIMATE = 5024;
406
426
 
427
+ /** Conservative upper bound for one persisted frame's base64 payload. The
428
+ * measured high-res Anthropic `8x13`/`11on16` PNG frames sit around 159 KB;
429
+ * 170 KB leaves margin for denser glyph pages without permitting multi-MB
430
+ * standing request bodies at large context windows. */
431
+ export const FRAME_DATA_BYTES_ESTIMATE = 170_000;
432
+
433
+ /** Maximum snapcompact image base64 carried in every rebuilt provider request.
434
+ * Above this, provider backends can accept the HTTP body but fail mid-stream
435
+ * with opaque 5xx errors. Keep this independent from visual-token budgeting:
436
+ * a 1M-token model can afford 70 images on paper, but not the resulting
437
+ * ~11 MB JSON payload on every turn. */
438
+ export const FRAME_DATA_BYTES_BUDGET = 3_000_000;
439
+
440
+ /** Frame-count cap implied by {@link FRAME_DATA_BYTES_BUDGET}. */
441
+ export function maxFramesForDataBudget(maxFrameDataBytes: number = FRAME_DATA_BYTES_BUDGET): number {
442
+ return Math.max(1, Math.floor(maxFrameDataBytes / FRAME_DATA_BYTES_ESTIMATE));
443
+ }
444
+
445
+ /** Base64 byte length for persisted snapcompact frames. */
446
+ export function frameDataBytes(frames: readonly Pick<Frame, "data">[]): number {
447
+ return frames.reduce((sum, frame) => sum + frame.data.length, 0);
448
+ }
449
+
407
450
  /**
408
451
  * Per-request image-count budgets by provider id. These cap how many images an
409
452
  * entire request may carry (archive/system-prompt/tool-result imaging combined).
@@ -492,8 +535,8 @@ export interface Geometry {
492
535
  export interface Options<TMessage = Message> extends SerializeOptions {
493
536
  /** App-level message transformer (same contract as agent-core's `SummaryOptions.convertToLlm`). */
494
537
  convertToLlm?: ConvertToLlm<TMessage>;
495
- /** Model whose provider API selects the frame shape. */
496
- model?: Pick<Model, "api">;
538
+ /** Model whose provider API and id select the frame shape. */
539
+ model?: ShapeTarget;
497
540
  /** Explicit shape override; wins over `model`. */
498
541
  shape?: Shape;
499
542
  /** Frame edge in pixels. Defaults to the shape's `frameSize`. */
@@ -918,6 +961,48 @@ const UNRENDERABLE = /[\p{Cc}\p{Mn}\p{Me}\p{Cs}]/u;
918
961
  * letter prints without the diacritic the bundled fonts cannot compose. */
919
962
  const COMBINING_MARKS = /\p{M}+/gu;
920
963
 
964
+ /** Status-like pictographs that carry meaning in tool output; all other emoji
965
+ * pictographs drop instead of burning cells as `?`. */
966
+ const EMOJI_FOLD: Record<string, string> = {
967
+ "✅": "[OK]",
968
+ "☑": "[OK]",
969
+ "✔": "[OK]",
970
+ "❌": "[FAIL]",
971
+ "❎": "[FAIL]",
972
+ "✖": "[FAIL]",
973
+ "⚠": "[WARN]",
974
+ "🚨": "[ALERT]",
975
+ ℹ: "[INFO]",
976
+ "🐛": "[BUG]",
977
+ "💥": "[CRASH]",
978
+ "🔥": "[HOT]",
979
+ "🔒": "[LOCK]",
980
+ "🔓": "[UNLOCK]",
981
+ "📁": "[DIR]",
982
+ "📂": "[DIR]",
983
+ "📄": "[FILE]",
984
+ "📝": "[NOTE]",
985
+ "🧪": "[TEST]",
986
+ "⏳": "[WAIT]",
987
+ "⌛": "[WAIT]",
988
+ "🚀": "[RUN]",
989
+ };
990
+
991
+ const EMOJI_PICTOGRAPH = /\p{Extended_Pictographic}/u;
992
+
993
+ export interface NormalizeOptions {
994
+ /** Shape whose font is tried before the embedded Silver fallback. */
995
+ shape?: Pick<Shape, "font">;
996
+ /** Native font name when a full shape is not available. */
997
+ font?: Shape["font"];
998
+ }
999
+
1000
+ interface NormalizedText {
1001
+ text: string;
1002
+ totalGraphics: number;
1003
+ fallbackCount: number;
1004
+ }
1005
+
921
1006
  /**
922
1007
  * Aggressive single-code-point ASCII fold via Unicode NFKD: decompose the
923
1008
  * compatibility form (fullwidth, super/subscripts, ligatures, circled and
@@ -927,13 +1012,17 @@ const COMBINING_MARKS = /\p{M}+/gu;
927
1012
  * point has no decomposition or still leaves an undrawable glyph, so the
928
1013
  * caller falls back to `?`.
929
1014
  */
1015
+ function isAsciiOrLatin1(cp: number): boolean {
1016
+ return (cp >= 0x20 && cp < 0x7f) || (cp >= 0xa0 && cp <= 0xff);
1017
+ }
1018
+
930
1019
  function foldToAscii(ch: string): string | undefined {
931
1020
  const decomposed = ch.normalize("NFKD").replace(COMBINING_MARKS, "");
932
1021
  if (decomposed === ch) return undefined;
933
1022
  let out = "";
934
1023
  for (const part of decomposed) {
935
- const cp = part.codePointAt(0) as number;
936
- if ((cp >= 0x20 && cp < 0x7f) || (cp >= 0xa0 && cp <= 0xff)) {
1024
+ const cp = part.codePointAt(0);
1025
+ if (cp !== undefined && isAsciiOrLatin1(cp)) {
937
1026
  out += part;
938
1027
  continue;
939
1028
  }
@@ -944,90 +1033,126 @@ function foldToAscii(ch: string): string | undefined {
944
1033
  return out;
945
1034
  }
946
1035
 
947
- /**
948
- * Prepare text for printing: strip ANSI escape sequences, collapse horizontal
949
- * whitespace runs to single spaces and newline-bearing runs to one
950
- * {@link NEWLINE_GLYPH} (drawn as a pitch-black cell), then fold everything
951
- * outside the fonts' ASCII + Latin-1 coverage to ASCII approximations — first
952
- * through the {@link CHAR_FOLD} punctuation table, then via an NFKD
953
- * decomposition that recovers the ASCII skeleton of compatibility characters
954
- * (fullwidth, super/subscripts, ligatures, circled/math-styled alphanumerics,
955
- * Roman numerals, vulgar fractions). Unrenderable control/format/combining
956
- * characters are dropped without occupying a cell; `?` remains the fallback
957
- * for unsupported graphic characters. The zero-width ink toggles
958
- * {@link DIM_ON}/{@link DIM_OFF} pass through untouched.
959
- */
960
- export function normalize(text: string): string {
1036
+ function renderableUnicodeChars(chars: readonly string[], font: Shape["font"] | undefined): ReadonlySet<string> {
1037
+ if (chars.length === 0) return new Set();
1038
+ const text = chars.join("");
1039
+ const primaryFont = font ?? "5x8";
1040
+ const supported = new Set(snapcompactSupportedChars(primaryFont, text));
1041
+ if (primaryFont !== "silver") {
1042
+ for (const ch of snapcompactSupportedChars("silver", text)) supported.add(ch);
1043
+ }
1044
+ return supported;
1045
+ }
1046
+
1047
+ function normalizedInputChars(text: string): string[] {
961
1048
  const stripped = text.includes("\u001b") ? Bun.stripANSI(text) : text;
962
1049
  const collapsed = stripped
963
1050
  // A run of pure format chars (BOM is both \s and Cf) vanishes; only a
964
1051
  // run containing genuine whitespace separates words.
965
1052
  .replace(COLLAPSIBLE, run => (LINE_BREAK.test(run) ? NEWLINE_GLYPH : /[^\p{Cf}]/u.test(run) ? " " : ""))
966
1053
  .replace(EDGE_RUNS, "");
967
- let out = "";
968
- for (const ch of collapsed) {
969
- const cp = ch.codePointAt(0) as number;
970
- if ((cp >= 0x20 && cp < 0x7f) || (cp >= 0xa0 && cp <= 0xff)) {
971
- out += ch;
1054
+ return [...collapsed];
1055
+ }
1056
+
1057
+ function candidateUnicodeChars(chars: readonly string[]): string[] {
1058
+ const unique = new Set<string>();
1059
+ for (const ch of chars) {
1060
+ const cp = ch.codePointAt(0);
1061
+ if (cp === undefined || isAsciiOrLatin1(cp) || ch === DIM_ON || ch === DIM_OFF || ch === NEWLINE_GLYPH) {
972
1062
  continue;
973
1063
  }
974
- if (ch === DIM_ON || ch === DIM_OFF || ch === NEWLINE_GLYPH) {
975
- out += ch;
1064
+ if (
1065
+ CHAR_FOLD[ch] !== undefined ||
1066
+ (cp >= 0x2500 && cp <= 0x257f) ||
1067
+ EMOJI_FOLD[ch] !== undefined ||
1068
+ EMOJI_PICTOGRAPH.test(ch) ||
1069
+ foldToAscii(ch) !== undefined ||
1070
+ UNRENDERABLE.test(ch)
1071
+ ) {
976
1072
  continue;
977
1073
  }
978
- const fold = CHAR_FOLD[ch];
979
- if (fold !== undefined) {
980
- out += fold;
981
- } else if (cp >= 0x2500 && cp <= 0x257f) {
982
- // Box drawing: keep table skeletons legible.
983
- out += cp === 0x2502 || cp === 0x2503 ? "|" : cp === 0x2500 || cp === 0x2501 ? "-" : "+";
984
- } else {
985
- const folded = foldToAscii(ch);
986
- if (folded !== undefined) out += folded;
987
- else if (!UNRENDERABLE.test(ch)) out += "?";
988
- }
1074
+ unique.add(ch);
989
1075
  }
990
- return out;
1076
+ return [...unique];
991
1077
  }
992
1078
 
993
- /**
994
- * Scan text to determine the proportion of graphic characters that will hit the
995
- * `?` fallback during {@link normalize}. Used as a preflight check to abort
996
- * snapcompact and fall back to the text summarizer when the input is heavily
997
- * non-renderable (e.g., CJK).
998
- */
999
- export function scanRenderability(text: string): { isSafe: boolean; unrenderableRatio: number } {
1000
- const stripped = text.includes("\u001b") ? Bun.stripANSI(text) : text;
1001
- const collapsed = stripped
1002
- .replace(COLLAPSIBLE, run => (LINE_BREAK.test(run) ? NEWLINE_GLYPH : /[^\p{Cf}]/u.test(run) ? " " : ""))
1003
- .replace(EDGE_RUNS, "");
1079
+ function normalizeWithStats(text: string, options?: NormalizeOptions): NormalizedText {
1080
+ const chars = normalizedInputChars(text);
1081
+ const font = options?.font ?? options?.shape?.font;
1082
+ const supported = renderableUnicodeChars(candidateUnicodeChars(chars), font);
1083
+ const out: string[] = [];
1004
1084
  let totalGraphics = 0;
1005
1085
  let fallbackCount = 0;
1006
- for (const ch of collapsed) {
1007
- const cp = ch.codePointAt(0) as number;
1008
- if ((cp >= 0x20 && cp < 0x7f) || (cp >= 0xa0 && cp <= 0xff)) {
1086
+
1087
+ for (const ch of chars) {
1088
+ const cp = ch.codePointAt(0);
1089
+ if (cp === undefined) continue;
1090
+ if (isAsciiOrLatin1(cp)) {
1091
+ out.push(ch);
1009
1092
  totalGraphics++;
1010
1093
  continue;
1011
1094
  }
1012
1095
  if (ch === DIM_ON || ch === DIM_OFF || ch === NEWLINE_GLYPH) {
1096
+ out.push(ch);
1097
+ continue;
1098
+ }
1099
+ const emoji = EMOJI_FOLD[ch];
1100
+ if (emoji !== undefined) {
1101
+ out.push(emoji);
1102
+ totalGraphics++;
1013
1103
  continue;
1014
1104
  }
1015
1105
  const fold = CHAR_FOLD[ch];
1016
1106
  if (fold !== undefined) {
1107
+ out.push(fold);
1017
1108
  totalGraphics++;
1018
- } else if (cp >= 0x2500 && cp <= 0x257f) {
1109
+ continue;
1110
+ }
1111
+ if (cp >= 0x2500 && cp <= 0x257f) {
1112
+ out.push(cp === 0x2502 || cp === 0x2503 ? "|" : cp === 0x2500 || cp === 0x2501 ? "-" : "+");
1019
1113
  totalGraphics++;
1020
- } else {
1021
- const folded = foldToAscii(ch);
1022
- if (folded !== undefined) {
1023
- totalGraphics++;
1024
- } else if (!UNRENDERABLE.test(ch)) {
1025
- totalGraphics++;
1026
- fallbackCount++;
1027
- }
1114
+ continue;
1115
+ }
1116
+ if (!EMOJI_PICTOGRAPH.test(ch) && supported.has(ch)) {
1117
+ out.push(ch);
1118
+ totalGraphics++;
1119
+ continue;
1120
+ }
1121
+ const folded = foldToAscii(ch);
1122
+ if (folded !== undefined) {
1123
+ out.push(folded);
1124
+ totalGraphics++;
1125
+ } else if (EMOJI_PICTOGRAPH.test(ch)) {
1126
+ } else if (!UNRENDERABLE.test(ch)) {
1127
+ out.push("?");
1128
+ totalGraphics++;
1129
+ fallbackCount++;
1028
1130
  }
1029
1131
  }
1030
- const unrenderableRatio = totalGraphics > 0 ? fallbackCount / totalGraphics : 0;
1132
+
1133
+ return { text: out.join("").replace(/ +/g, " ").replace(EDGE_RUNS, ""), totalGraphics, fallbackCount };
1134
+ }
1135
+
1136
+ /**
1137
+ * Prepare text for printing: strip ANSI escape sequences, collapse horizontal
1138
+ * whitespace runs, fold unsupported symbols (including box drawing to ASCII),
1139
+ * preserve Unicode glyphs that either the selected font or embedded Silver
1140
+ * fallback can render, and drop decorative emoji instead of printing `?`.
1141
+ */
1142
+ export function normalize(text: string, options?: NormalizeOptions): string {
1143
+ return normalizeWithStats(text, options).text;
1144
+ }
1145
+
1146
+ /**
1147
+ * Scan text with the same font-aware path as {@link normalize}; unsafe means
1148
+ * more than 5% of graphic characters would hit the `?` fallback.
1149
+ */
1150
+ export function scanRenderability(
1151
+ text: string,
1152
+ options?: NormalizeOptions,
1153
+ ): { isSafe: boolean; unrenderableRatio: number } {
1154
+ const normalized = normalizeWithStats(text, options);
1155
+ const unrenderableRatio = normalized.totalGraphics > 0 ? normalized.fallbackCount / normalized.totalGraphics : 0;
1031
1156
  return { isSafe: unrenderableRatio <= 0.05, unrenderableRatio };
1032
1157
  }
1033
1158
 
@@ -1086,34 +1211,131 @@ export function dimStopwords(text: string): string {
1086
1211
  /** Char cells between the two doc columns (research exp14 `GUTTER`). */
1087
1212
  const DOC_GUTTER = 3;
1088
1213
 
1214
+ /** East Asian Wide / Fullwidth code points that occupy two grid cells when a
1215
+ * narrow bitmap shape draws them through the Silver fallback. Mirrors
1216
+ * `is_wide` in `crates/pi-natives/src/snapcompact.rs`; the two MUST stay in
1217
+ * sync or native layout and this capacity math disagree on cell counts. */
1218
+ function isWideCodePoint(cp: number): boolean {
1219
+ return (
1220
+ (cp >= 0x1100 && cp <= 0x115f) ||
1221
+ (cp >= 0x2e80 && cp <= 0x2eff) ||
1222
+ (cp >= 0x2f00 && cp <= 0x2fdf) ||
1223
+ (cp >= 0x3000 && cp <= 0x303e) ||
1224
+ (cp >= 0x3041 && cp <= 0x33ff) ||
1225
+ (cp >= 0x3400 && cp <= 0x4dbf) ||
1226
+ (cp >= 0x4e00 && cp <= 0x9fff) ||
1227
+ (cp >= 0xa000 && cp <= 0xa4cf) ||
1228
+ (cp >= 0xac00 && cp <= 0xd7a3) ||
1229
+ (cp >= 0xf900 && cp <= 0xfaff) ||
1230
+ (cp >= 0xfe30 && cp <= 0xfe4f) ||
1231
+ (cp >= 0xff00 && cp <= 0xff60) ||
1232
+ (cp >= 0xffe0 && cp <= 0xffe6) ||
1233
+ (cp >= 0x20000 && cp <= 0x2fffd) ||
1234
+ (cp >= 0x30000 && cp <= 0x3fffd)
1235
+ );
1236
+ }
1237
+
1238
+ /** Cells one character occupies: 0 for the zero-width dim toggles, 2 for wide
1239
+ * code points in narrow bitmap shapes, 1 otherwise. Mirrors native
1240
+ * `cell_units`. */
1241
+ function charCells(ch: string, wideCells: boolean): number {
1242
+ if (ch === DIM_ON || ch === DIM_OFF) return 0;
1243
+ const cp = ch.codePointAt(0);
1244
+ return wideCells && cp !== undefined && isWideCodePoint(cp) ? 2 : 1;
1245
+ }
1246
+
1247
+ /** Wide code points span two cells in every shape except the square-celled
1248
+ * Silver shape, which sizes each cell for a full-width glyph already. */
1249
+ function usesWideCells(shape: Pick<Shape, "font">): boolean {
1250
+ return shape.font !== "silver";
1251
+ }
1252
+
1253
+ /** Total grid cells a string occupies (ignoring row wrapping/pads). */
1254
+ function cellLength(text: string, wideCells: boolean): number {
1255
+ let cells = 0;
1256
+ for (const ch of text) cells += charCells(ch, wideCells);
1257
+ return cells;
1258
+ }
1259
+
1260
+ /** Longest prefix of `text` that fits `width` cells (at least one char). */
1261
+ function sliceCells(text: string, width: number, wideCells: boolean): string {
1262
+ let cells = 0;
1263
+ let out = "";
1264
+ let placed = false;
1265
+ for (const ch of text) {
1266
+ const w = charCells(ch, wideCells);
1267
+ if (placed && cells + w > width) break;
1268
+ out += ch;
1269
+ cells += w;
1270
+ if (w > 0) placed = true;
1271
+ }
1272
+ return out;
1273
+ }
1274
+
1275
+ /** Split `text` into pages that each fill at most `capacity` grid cells,
1276
+ * inserting a one-cell pad before a wide glyph that would straddle the right
1277
+ * edge (mirrors native `place_cell`). Pages are contiguous substrings, so each
1278
+ * renders independently starting at cell 0. A single char wider than the whole
1279
+ * budget still rides its page; the native renderer clips it. */
1280
+ function paginateCells(text: string, capacity: number, cols: number, wideCells: boolean): string[] {
1281
+ const chars = [...text];
1282
+ const pages: string[] = [];
1283
+ let start = 0;
1284
+ let cell = 0;
1285
+ let hasCell = false;
1286
+ for (let i = 0; i < chars.length; i++) {
1287
+ const w = charCells(chars[i] ?? "", wideCells);
1288
+ if (w === 0) continue;
1289
+ let at = cell;
1290
+ if (w === 2 && cols >= 2 && at % cols === cols - 1) at += 1;
1291
+ if (hasCell && at + w > capacity) {
1292
+ pages.push(chars.slice(start, i).join(""));
1293
+ start = i;
1294
+ at = 0;
1295
+ }
1296
+ cell = at + w;
1297
+ hasCell = true;
1298
+ }
1299
+ if (hasCell) pages.push(chars.slice(start).join(""));
1300
+ return pages;
1301
+ }
1302
+
1089
1303
  /**
1090
1304
  * Greedy word-wrap, no mid-word breaks (hard split only for width+ words) —
1091
1305
  * ported verbatim from `research/exp14_bestgpt.py` `wrap()`. Zero-width dim
1092
1306
  * markers count toward word length here; serialized history places them at
1093
1307
  * word boundaries, so the drift is at most one cell per affected line.
1094
1308
  */
1095
- export function wrap(text: string, width: number): string[] {
1309
+ export function wrap(text: string, width: number, wideCells = false): string[] {
1096
1310
  const lines: string[] = [];
1097
1311
  let cur = "";
1312
+ let curCells = 0;
1098
1313
  for (const token of text.split(/\s+/)) {
1099
1314
  if (token.length === 0) continue;
1100
1315
  let word = token;
1101
- while (word.length > width) {
1316
+ let wordCells = cellLength(word, wideCells);
1317
+ while (wordCells > width) {
1102
1318
  // Pathological; never hit on prose.
1103
1319
  if (cur) {
1104
1320
  lines.push(cur);
1105
1321
  cur = "";
1322
+ curCells = 0;
1106
1323
  }
1107
- lines.push(word.slice(0, width));
1108
- word = word.slice(width);
1324
+ const head = sliceCells(word, width, wideCells);
1325
+ lines.push(head);
1326
+ word = word.slice(head.length);
1327
+ wordCells = cellLength(word, wideCells);
1109
1328
  }
1110
1329
  if (!cur) {
1111
1330
  cur = word;
1112
- } else if (cur.length + 1 + word.length <= width) {
1331
+ curCells = wordCells;
1332
+ } else if (curCells + 1 + wordCells <= width) {
1113
1333
  cur += ` ${word}`;
1334
+ curCells += 1 + wordCells;
1114
1335
  } else {
1115
1336
  lines.push(cur);
1116
1337
  cur = word;
1338
+ curCells = wordCells;
1117
1339
  }
1118
1340
  }
1119
1341
  if (cur) lines.push(cur);
@@ -1126,8 +1348,8 @@ export function wrap(text: string, width: number): string[] {
1126
1348
  * Every input character lands on exactly one page (whitespace becomes the
1127
1349
  * wrap points).
1128
1350
  */
1129
- function docPages(normalized: string, geo: Geometry): string[] {
1130
- const lines = wrap(normalized, geo.cols);
1351
+ function docPages(normalized: string, geo: Geometry, wideCells: boolean): string[] {
1352
+ const lines = wrap(normalized, geo.cols, wideCells);
1131
1353
  const perPage = 2 * geo.rows;
1132
1354
  const pages: string[] = [];
1133
1355
  for (let offset = 0; offset < lines.length; offset += perPage) {
@@ -1165,18 +1387,35 @@ function nativeRenderOptions(shape: Shape, size: number) {
1165
1387
  };
1166
1388
  }
1167
1389
 
1168
- function renderedChars(text: string, shape: Shape, capacity: number): number {
1169
- let visible = text.length - (text.match(DIM_MARKERS)?.length ?? 0);
1170
- // Doc line separators consume no cell; in the grid they print as a blank.
1171
- if (shape.columns === 2) visible -= text.match(NEWLINES)?.length ?? 0;
1172
- return Math.min(visible, capacity);
1390
+ function renderedChars(text: string, shape: Shape, geo: Geometry): number {
1391
+ if (shape.columns === 2) {
1392
+ let visible = [...text].length - (text.match(DIM_MARKERS)?.length ?? 0);
1393
+ visible -= text.match(NEWLINES)?.length ?? 0;
1394
+ return Math.min(visible, geo.capacity);
1395
+ }
1396
+ // Grid: count visible chars that fit within the frame's cell budget, with
1397
+ // wide glyphs taking two cells (and a straddle pad) exactly as the renderer.
1398
+ const wideCells = usesWideCells(shape);
1399
+ let cell = 0;
1400
+ let count = 0;
1401
+ for (const ch of text) {
1402
+ const w = charCells(ch, wideCells);
1403
+ if (w === 0) continue;
1404
+ let at = cell;
1405
+ if (w === 2 && geo.cols >= 2 && at % geo.cols === geo.cols - 1) at += 1;
1406
+ if (at + w > geo.capacity) break;
1407
+ cell = at + w;
1408
+ count++;
1409
+ }
1410
+ return count;
1173
1411
  }
1174
1412
 
1175
1413
  /** Render one snapcompact frame from already-normalized text. Doc shapes
1176
1414
  * (`columns === 2`) expect one page of `\n`-joined pre-wrapped lines. */
1177
1415
  export async function render(text: string, shape: Shape, size: number = shape.frameSize): Promise<RenderedFrame> {
1178
- const { cols, rows, capacity } = geometry(shape, size);
1179
- const chars = renderedChars(text, shape, capacity);
1416
+ const geo = geometry(shape, size);
1417
+ const { cols, rows } = geo;
1418
+ const chars = renderedChars(text, shape, geo);
1180
1419
  const data = await renderSnapcompactPng(text, nativeRenderOptions(shape, size));
1181
1420
  return { data, cols, rows, chars };
1182
1421
  }
@@ -1197,8 +1436,8 @@ function pageFinisher(shape: Shape): (page: string) => string {
1197
1436
  export interface RenderManyOptions {
1198
1437
  /** Explicit shape; wins over `model`. */
1199
1438
  shape?: Shape;
1200
- /** Model whose `api` selects the eval-optimal shape. */
1201
- model?: Pick<Model, "api">;
1439
+ /** Model whose provider API and id select the frame shape. */
1440
+ model?: ShapeTarget;
1202
1441
  /** Frame edge in px; defaults to the shape's `frameSize`. */
1203
1442
  frameSize?: number;
1204
1443
  /** Hard cap on frames produced; omit for unbounded (caller decides usage). */
@@ -1210,27 +1449,26 @@ export interface RenderManyOptions {
1210
1449
  * (first page first). Empty/whitespace-only input yields no frames.
1211
1450
  */
1212
1451
  export async function renderMany(text: string, options?: RenderManyOptions): Promise<ImageContent[]> {
1213
- const shape = options?.shape ?? resolveShape(options?.model);
1452
+ const shape = options?.shape ?? resolveShapeForText(text, options?.model);
1214
1453
  const frameSize = options?.frameSize ?? shape.frameSize;
1215
1454
  const geo = geometry(shape, frameSize);
1216
- const normalized = normalize(text);
1455
+ const normalized = normalize(text, { shape });
1217
1456
  const cap = options?.maxFrames;
1218
1457
  // Build the per-frame texts in order first (cheap, synchronous), then fan
1219
1458
  // the native PNG renders out concurrently — render() is async/off-thread,
1220
1459
  // so awaiting each before starting the next leaves throughput on the table.
1221
1460
  const pageTexts: string[] = [];
1461
+ const wideCells = usesWideCells(shape);
1222
1462
  if (shape.columns === 2) {
1223
1463
  const finish = pageFinisher(shape);
1224
- for (const page of docPages(normalized, geo)) {
1464
+ for (const page of docPages(normalized, geo, wideCells)) {
1225
1465
  if (cap !== undefined && pageTexts.length >= cap) break;
1226
1466
  pageTexts.push(finish(page));
1227
1467
  }
1228
1468
  } else {
1229
- for (let offset = 0; offset < normalized.length; offset += geo.capacity) {
1469
+ for (const page of paginateCells(normalized, geo.capacity, geo.cols, wideCells)) {
1230
1470
  if (cap !== undefined && pageTexts.length >= cap) break;
1231
- let chunk = normalized.slice(offset, offset + geo.capacity);
1232
- if (shape.stopwordDim) chunk = dimStopwords(chunk);
1233
- pageTexts.push(chunk);
1471
+ pageTexts.push(shape.stopwordDim ? dimStopwords(page) : page);
1234
1472
  }
1235
1473
  }
1236
1474
  const rendered = await Promise.all(pageTexts.map(page => render(page, shape, frameSize)));
@@ -1246,11 +1484,12 @@ export async function renderMany(text: string, options?: RenderManyOptions): Pro
1246
1484
  * For doc shapes this wraps the text once and counts pages of `2 * rows`
1247
1485
  * lines; for grid shapes it divides by the frame capacity. */
1248
1486
  export function frames(text: string, options?: Pick<RenderManyOptions, "shape" | "model" | "frameSize">): number {
1249
- const shape = options?.shape ?? resolveShape(options?.model);
1487
+ const shape = options?.shape ?? resolveShapeForText(text, options?.model);
1250
1488
  const geo = geometry(shape, options?.frameSize ?? shape.frameSize);
1251
- const normalized = normalize(text);
1252
- if (shape.columns === 2) return Math.ceil(wrap(normalized, geo.cols).length / (2 * geo.rows));
1253
- return Math.ceil(normalized.length / geo.capacity);
1489
+ const normalized = normalize(text, { shape });
1490
+ const wideCells = usesWideCells(shape);
1491
+ if (shape.columns === 2) return Math.ceil(wrap(normalized, geo.cols, wideCells).length / (2 * geo.rows));
1492
+ return paginateCells(normalized, geo.capacity, geo.cols, wideCells).length;
1254
1493
  }
1255
1494
 
1256
1495
  // ============================================================================
@@ -1313,6 +1552,67 @@ export function archiveSourceText(archive: Archive): string | undefined {
1313
1552
  return text.length > 0 ? toPlainText(text) : undefined;
1314
1553
  }
1315
1554
 
1555
+ /** Build the text used to choose and preflight a font-aware snapcompact shape. */
1556
+ export function renderabilityProbeText(
1557
+ serialized: string,
1558
+ previousPreserveData?: Record<string, unknown>,
1559
+ previousSummary?: string,
1560
+ ): string {
1561
+ const previousArchive = getPreservedArchive(previousPreserveData);
1562
+ const previousText = previousArchive ? (archiveSourceText(previousArchive) ?? "") : "";
1563
+ if (previousText.length > 0) return `${previousText}${NEWLINE_GLYPH}${serialized}`;
1564
+ if (previousSummary) return `${previousSummary}${NEWLINE_GLYPH}${serialized}`;
1565
+ return serialized;
1566
+ }
1567
+
1568
+ /** Options for reconstructing a persisted snapcompact archive into prompt blocks. */
1569
+ export interface HistoryBlockOptions {
1570
+ /** Hard cap on image base64 bytes attached to one rebuilt provider request. */
1571
+ maxFrameDataBytes?: number;
1572
+ }
1573
+
1574
+ function formatFrameDataBytes(bytes: number): string {
1575
+ if (bytes >= 1_000_000) return `${(bytes / 1_000_000).toFixed(1)} MB`;
1576
+ if (bytes >= 1_000) return `${(bytes / 1_000).toFixed(1)} KB`;
1577
+ return `${bytes} B`;
1578
+ }
1579
+
1580
+ function imagesWithinBudget(
1581
+ archive: Archive,
1582
+ maxFrameDataBytes: number | undefined,
1583
+ ): { images: ImageContent[]; omittedFrames: number; omittedBytes: number } {
1584
+ if (maxFrameDataBytes === undefined) {
1585
+ return { images: images(archive), omittedFrames: 0, omittedBytes: 0 };
1586
+ }
1587
+
1588
+ let usedBytes = 0;
1589
+ let omittedFrames = 0;
1590
+ let omittedBytes = 0;
1591
+ const keptNewestFirst: Frame[] = [];
1592
+ for (let index = archive.frames.length - 1; index >= 0; index--) {
1593
+ const frame = archive.frames[index];
1594
+ if (!frame) continue;
1595
+ const bytes = frame.data.length;
1596
+ if (usedBytes + bytes > maxFrameDataBytes) {
1597
+ omittedFrames++;
1598
+ omittedBytes += bytes;
1599
+ continue;
1600
+ }
1601
+ usedBytes += bytes;
1602
+ keptNewestFirst.push(frame);
1603
+ }
1604
+ keptNewestFirst.reverse();
1605
+ return { images: images({ ...archive, frames: keptNewestFirst }), omittedFrames, omittedBytes };
1606
+ }
1607
+
1608
+ function omittedFrameNotice(omittedFrames: number, omittedBytes: number): string {
1609
+ return [
1610
+ "-------------- snapcompact image middle omitted",
1611
+ `${omittedFrames.toLocaleString()} archived image frame${omittedFrames === 1 ? "" : "s"} (${formatFrameDataBytes(omittedBytes)} base64) exceeded the per-request snapcompact payload budget. The compacted summary and visible text edges remain available.`,
1612
+ "--------------",
1613
+ ].join("\n");
1614
+ }
1615
+
1316
1616
  /** Convert archive frames into LLM image blocks (oldest first). */
1317
1617
  export function images(archive: Archive): ImageContent[] {
1318
1618
  return archive.frames.map(frame => ({
@@ -1326,23 +1626,38 @@ export function images(archive: Archive): ImageContent[] {
1326
1626
  * the oldest text region, the imaged middle, then the newest text region.
1327
1627
  * Runtime-only; reconstructed from {@link Archive} on each context rebuild
1328
1628
  * instead of persisted on the session entry. */
1329
- export function historyBlocks(archive: Archive): (TextContent | ImageContent)[] {
1629
+ export function historyBlocks(archive: Archive, options: HistoryBlockOptions = {}): (TextContent | ImageContent)[] {
1330
1630
  const blocks: (TextContent | ImageContent)[] = [];
1331
- const hasImages = archive.frames.length > 0;
1631
+ const budgeted = imagesWithinBudget(archive, options.maxFrameDataBytes);
1632
+ const hasImages = budgeted.images.length > 0;
1633
+ const hasOmittedImages = budgeted.omittedFrames > 0;
1332
1634
  if (archive.textHead) {
1333
- const suffix = hasImages ? "\n-------------- imaged middle below\n" : "";
1635
+ const suffix = hasImages
1636
+ ? "\n-------------- imaged middle below\n"
1637
+ : hasOmittedImages
1638
+ ? `\n${omittedFrameNotice(budgeted.omittedFrames, budgeted.omittedBytes)}\n`
1639
+ : "";
1334
1640
  blocks.push({ type: "text", text: toPlainText(archive.textHead) + suffix });
1641
+ } else if (hasOmittedImages && !hasImages) {
1642
+ blocks.push({ type: "text", text: omittedFrameNotice(budgeted.omittedFrames, budgeted.omittedBytes) });
1643
+ }
1644
+ // Omitted frames are the OLDEST archived images: the byte budget keeps the
1645
+ // newest tail frames, so the gap notice precedes the kept images to keep the
1646
+ // reconstructed blocks oldest-to-newest.
1647
+ if (hasImages && hasOmittedImages) {
1648
+ blocks.push({ type: "text", text: omittedFrameNotice(budgeted.omittedFrames, budgeted.omittedBytes) });
1335
1649
  }
1336
- blocks.push(...images(archive));
1650
+ blocks.push(...budgeted.images);
1337
1651
  if (archive.textTail) {
1338
1652
  const prefix = hasImages
1339
1653
  ? "-------------- imaged middle above\n"
1340
- : archive.truncatedChars > 0
1654
+ : archive.truncatedChars > 0 || hasOmittedImages
1341
1655
  ? "\n-------------- middle history omitted above\n"
1342
1656
  : "";
1343
1657
  const tail = prefix + toPlainText(archive.textTail);
1344
- if (blocks.length > 0 && blocks[blocks.length - 1]?.type === "text") {
1345
- (blocks[blocks.length - 1] as TextContent).text += tail;
1658
+ const lastBlock = blocks[blocks.length - 1];
1659
+ if (lastBlock?.type === "text") {
1660
+ lastBlock.text += tail;
1346
1661
  } else {
1347
1662
  blocks.push({ type: "text", text: tail });
1348
1663
  }
@@ -1356,9 +1671,10 @@ export function historyBlocks(archive: Archive): (TextContent | ImageContent)[]
1356
1671
 
1357
1672
  /** Denser companion of `high` for the foveated archive middle: same family and
1358
1673
  * frame size (identical per-frame bill) but a tighter cell. Returns `high`
1359
- * unchanged for doc layouts or when no denser variant exists (foveation off). */
1674
+ * unchanged for doc layouts, TrueType Unicode shapes, or when no denser
1675
+ * variant exists (foveation off). */
1360
1676
  function denseCompanion(high: Shape, api: Api | undefined): Shape {
1361
- if (high.columns === 2) return high;
1677
+ if (high.columns === 2 || high.font === "silver") return high;
1362
1678
  const family = billingFamily(api);
1363
1679
  const low = priceShape({ ...SHAPE_VARIANTS[FAMILY_VARIANT_LOW[family]], frameSize: high.frameSize }, family);
1364
1680
  return geometry(low).capacity > geometry(high).capacity ? low : high;
@@ -1381,13 +1697,9 @@ interface ArchiveLayout {
1381
1697
  truncatedChars: number;
1382
1698
  }
1383
1699
 
1384
- /** Slice `text` into `capacity`-char frames at one shape (tier). */
1385
- function sliceFrames(text: string, capacity: number, shape: Shape): PlanFrame[] {
1386
- const out: PlanFrame[] = [];
1387
- for (let offset = 0; offset < text.length; offset += capacity) {
1388
- out.push({ text: text.slice(offset, offset + capacity), shape });
1389
- }
1390
- return out;
1700
+ /** Wrap each page string as a planned frame at one shape (tier). */
1701
+ function planFrames(pages: readonly string[], shape: Shape): PlanFrame[] {
1702
+ return pages.map(text => ({ text, shape }));
1391
1703
  }
1392
1704
 
1393
1705
  /**
@@ -1425,7 +1737,7 @@ function planArchive(text: string, high: Shape, low: Shape, maxFrames: number):
1425
1737
  // Doc layouts wrap (no char-slicing) and don't foveate: one tier, keep the
1426
1738
  // newest pages with the session head pinned, drop the oldest middle.
1427
1739
  if (high.columns === 2) {
1428
- const pages = docPages(imageText, geometry(high));
1740
+ const pages = docPages(imageText, geometry(high), usesWideCells(high));
1429
1741
  let kept = pages;
1430
1742
  let truncatedChars = 0;
1431
1743
  if (pages.length > maxFrames) {
@@ -1435,7 +1747,7 @@ function planArchive(text: string, high: Shape, low: Shape, maxFrames: number):
1435
1747
  }
1436
1748
  const flat = kept.map(page => page.replaceAll("\n", " ")).join(" ");
1437
1749
  return {
1438
- frames: kept.map(page => ({ text: page, shape: high })),
1750
+ frames: planFrames(kept, high),
1439
1751
  textHead,
1440
1752
  textTail,
1441
1753
  keptText: textHead + flat + textTail,
@@ -1443,10 +1755,12 @@ function planArchive(text: string, high: Shape, low: Shape, maxFrames: number):
1443
1755
  };
1444
1756
  }
1445
1757
 
1446
- // Grid: render all-HQ when the image region fits the budget outright.
1447
- if (Math.ceil(imageText.length / capHi) <= maxFrames) {
1758
+ // Grid: paginate the imaged region into HQ frames (cell-aware, so wide CJK
1759
+ // glyphs spanning two cells never overflow a frame's capacity).
1760
+ const hiPages = paginateCells(imageText, capHi, geometry(high).cols, usesWideCells(high));
1761
+ if (hiPages.length <= maxFrames) {
1448
1762
  return {
1449
- frames: sliceFrames(imageText, capHi, high),
1763
+ frames: planFrames(hiPages, high),
1450
1764
  textHead,
1451
1765
  textTail,
1452
1766
  keptText: textHead + imageText + textTail,
@@ -1457,22 +1771,23 @@ function planArchive(text: string, high: Shape, low: Shape, maxFrames: number):
1457
1771
  // Foveate the imaged middle: HQ edges, dense center, drop the oldest dense slice.
1458
1772
  const capLo = geometry(low).capacity;
1459
1773
  const imageEdgeFrames = Math.min(HQ_EDGE_FRAMES, Math.floor((maxFrames - 1) / 2));
1460
- const imageEdgeCap = imageEdgeFrames * capHi;
1461
- const imageHead = imageText.slice(0, imageEdgeCap);
1462
- const imageTail = imageEdgeCap > 0 ? imageText.slice(imageText.length - imageEdgeCap) : "";
1463
- let middleText = imageText.slice(imageEdgeCap, imageText.length - imageEdgeCap);
1774
+ const headPages = hiPages.slice(0, imageEdgeFrames);
1775
+ const tailPages = imageEdgeFrames > 0 ? hiPages.slice(hiPages.length - imageEdgeFrames) : [];
1776
+ const imageHead = headPages.join("");
1777
+ const imageTail = tailPages.join("");
1778
+ const middleSource = imageText.slice(imageHead.length, imageText.length - imageTail.length);
1779
+ let middlePages = paginateCells(middleSource, capLo, geometry(low).cols, usesWideCells(low));
1780
+ const middleBudget = maxFrames - 2 * imageEdgeFrames;
1464
1781
  let truncatedChars = 0;
1465
- const middleCap = (maxFrames - 2 * imageEdgeFrames) * capLo;
1466
- if (middleText.length > middleCap) {
1467
- truncatedChars = middleText.length - middleCap;
1468
- middleText = middleText.slice(truncatedChars);
1782
+ let middleText = middleSource;
1783
+ if (middlePages.length > middleBudget) {
1784
+ const dropped = middlePages.slice(0, middlePages.length - middleBudget).join("");
1785
+ truncatedChars = dropped.length;
1786
+ middleText = middleSource.slice(dropped.length);
1787
+ middlePages = middlePages.slice(middlePages.length - middleBudget);
1469
1788
  }
1470
1789
  return {
1471
- frames: [
1472
- ...sliceFrames(imageHead, capHi, high),
1473
- ...sliceFrames(middleText, capLo, low),
1474
- ...sliceFrames(imageTail, capHi, high),
1475
- ],
1790
+ frames: [...planFrames(headPages, high), ...planFrames(middlePages, low), ...planFrames(tailPages, high)],
1476
1791
  textHead,
1477
1792
  textTail,
1478
1793
  keptText: textHead + imageHead + middleText + imageTail + textTail,
@@ -1501,19 +1816,9 @@ export async function compact<TMessage = Message>(
1501
1816
  if (!firstKeptEntryId) {
1502
1817
  throw new Error("First kept entry has no ID - session may need migration");
1503
1818
  }
1504
- const baseShape = options?.shape ?? resolveShape(options?.model);
1505
- const frameSize = options?.frameSize ?? baseShape.frameSize;
1506
- const high = frameSize === baseShape.frameSize ? baseShape : { ...baseShape, frameSize };
1507
- const low = denseCompanion(high, options?.model?.api);
1508
- const geo = geometry(high);
1509
- // The engine default caps archive growth; a caller-supplied maxFrames only
1510
- // lowers it further (an upper limit), never raising it past the default.
1511
- const maxFrames = Math.max(1, Math.min(options?.maxFrames ?? MAX_FRAMES_DEFAULT, MAX_FRAMES_DEFAULT));
1512
-
1513
1819
  const messages = preparation.messagesToSummarize.concat(preparation.turnPrefixMessages);
1514
1820
  const llmMessages = (options?.convertToLlm ?? defaultConvertToLlm)(messages);
1515
- let archiveText = normalize(serializeConversation(llmMessages, options));
1516
-
1821
+ const serialized = serializeConversation(llmMessages, options);
1517
1822
  const previousArchive = getPreservedArchive(previousPreserveData);
1518
1823
  const previousText =
1519
1824
  previousArchive?.text ??
@@ -1522,8 +1827,20 @@ export async function compact<TMessage = Message>(
1522
1827
  .join(NEWLINE_GLYPH);
1523
1828
  const hasPreviousText = previousText.length > 0;
1524
1829
  const includedPreviousSummary = !hasPreviousText && !!previousSummary;
1830
+ const shapeProbeText = renderabilityProbeText(serialized, previousPreserveData, previousSummary);
1831
+ const baseShape = options?.shape ?? resolveShapeForText(shapeProbeText, options?.model);
1832
+ const frameSize = options?.frameSize ?? baseShape.frameSize;
1833
+ const high = frameSize === baseShape.frameSize ? baseShape : { ...baseShape, frameSize };
1834
+ const low = denseCompanion(high, options?.model?.api);
1835
+ const geo = geometry(high);
1836
+ // The engine default caps archive growth; a caller-supplied maxFrames only
1837
+ // lowers it further (an upper limit), never raising it past the default.
1838
+ const maxFrames = Math.max(1, Math.min(options?.maxFrames ?? MAX_FRAMES_DEFAULT, MAX_FRAMES_DEFAULT));
1839
+
1840
+ let archiveText = normalize(serialized, { shape: high });
1841
+
1525
1842
  if (includedPreviousSummary && previousSummary) {
1526
- const head = `[Summary of earlier history] ${normalize(previousSummary)}`;
1843
+ const head = `[Summary of earlier history] ${normalize(previousSummary, { shape: high })}`;
1527
1844
  archiveText = archiveText.length > 0 ? `${head} [Recent conversation] ${archiveText}` : head;
1528
1845
  }
1529
1846