@zenera/cli 1.1.23 → 1.1.25

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (184) hide show
  1. package/README.md +88 -2
  2. package/dist/ask.d.ts +51 -0
  3. package/dist/ask.js +166 -0
  4. package/dist/banner.js +2 -2
  5. package/dist/batch.d.ts +83 -0
  6. package/dist/batch.js +413 -0
  7. package/dist/catalog.d.ts +11 -1
  8. package/dist/catalog.js +80 -7
  9. package/dist/command.d.ts +6 -0
  10. package/dist/commands/inspect.d.ts +57 -0
  11. package/dist/commands/inspect.js +539 -91
  12. package/dist/commands/list.js +128 -19
  13. package/dist/commands/memory.js +461 -32
  14. package/dist/commands/meta.js +311 -21
  15. package/dist/commands/models.js +110 -14
  16. package/dist/commands/open.js +4 -4
  17. package/dist/commands/run.js +175 -36
  18. package/dist/commands/sandbox.js +148 -15
  19. package/dist/engine.d.ts +24 -0
  20. package/dist/engine.js +61 -8
  21. package/dist/eventlog.d.ts +59 -0
  22. package/dist/eventlog.js +146 -0
  23. package/dist/liveness.d.ts +7 -0
  24. package/dist/liveness.js +20 -3
  25. package/dist/main.js +5 -2
  26. package/dist/meta/editor.d.ts +9 -0
  27. package/dist/meta/editor.js +60 -0
  28. package/dist/meta/finetune/children.d.ts +8 -0
  29. package/dist/meta/finetune/children.js +76 -0
  30. package/dist/meta/finetune/command.d.ts +30 -0
  31. package/dist/meta/finetune/command.js +338 -0
  32. package/dist/meta/finetune/dataset/anchors.d.ts +41 -0
  33. package/dist/meta/finetune/dataset/anchors.js +239 -0
  34. package/dist/meta/finetune/dataset/changes.d.ts +36 -0
  35. package/dist/meta/finetune/dataset/changes.js +319 -0
  36. package/dist/meta/finetune/dataset/command.d.ts +17 -0
  37. package/dist/meta/finetune/dataset/command.js +1011 -0
  38. package/dist/meta/finetune/dataset/drift.d.ts +33 -0
  39. package/dist/meta/finetune/dataset/drift.js +71 -0
  40. package/dist/meta/finetune/dataset/export.d.ts +26 -0
  41. package/dist/meta/finetune/dataset/export.js +51 -0
  42. package/dist/meta/finetune/dataset/index.d.ts +2 -0
  43. package/dist/meta/finetune/dataset/index.js +2 -0
  44. package/dist/meta/finetune/dataset/sample.d.ts +50 -0
  45. package/dist/meta/finetune/dataset/sample.js +197 -0
  46. package/dist/meta/finetune/dataset/schema.d.ts +33 -0
  47. package/dist/meta/finetune/dataset/schema.js +216 -0
  48. package/dist/meta/finetune/dataset/store.d.ts +35 -0
  49. package/dist/meta/finetune/dataset/store.js +177 -0
  50. package/dist/meta/finetune/dataset/types.d.ts +125 -0
  51. package/dist/meta/finetune/dataset/types.js +39 -0
  52. package/dist/meta/finetune/failure.d.ts +15 -0
  53. package/dist/meta/finetune/failure.js +26 -0
  54. package/dist/meta/finetune/feedback.d.ts +45 -0
  55. package/dist/meta/finetune/feedback.js +85 -0
  56. package/dist/meta/finetune/improvements.d.ts +51 -0
  57. package/dist/meta/finetune/improvements.js +244 -0
  58. package/dist/meta/finetune/index.d.ts +2 -0
  59. package/dist/meta/finetune/index.js +2 -0
  60. package/dist/meta/finetune/journal.d.ts +35 -0
  61. package/dist/meta/finetune/journal.js +135 -0
  62. package/dist/meta/finetune/live.d.ts +74 -0
  63. package/dist/meta/finetune/live.js +226 -0
  64. package/dist/meta/finetune/loop.d.ts +3 -0
  65. package/dist/meta/finetune/loop.js +114 -0
  66. package/dist/meta/finetune/memories.d.ts +47 -0
  67. package/dist/meta/finetune/memories.js +154 -0
  68. package/dist/meta/finetune/report.d.ts +33 -0
  69. package/dist/meta/finetune/report.js +1284 -0
  70. package/dist/meta/finetune/retry.d.ts +24 -0
  71. package/dist/meta/finetune/retry.js +96 -0
  72. package/dist/meta/finetune/sampler.d.ts +18 -0
  73. package/dist/meta/finetune/sampler.js +41 -0
  74. package/dist/meta/finetune/system.d.ts +27 -0
  75. package/dist/meta/finetune/system.js +132 -0
  76. package/dist/meta/finetune/train.d.ts +39 -0
  77. package/dist/meta/finetune/train.js +391 -0
  78. package/dist/meta/finetune/tui.d.ts +16 -0
  79. package/dist/meta/finetune/tui.js +291 -0
  80. package/dist/meta/finetune/tuning.d.ts +167 -0
  81. package/dist/meta/finetune/tuning.js +289 -0
  82. package/dist/meta/finetune/usage.d.ts +62 -0
  83. package/dist/meta/finetune/usage.js +225 -0
  84. package/dist/meta/host.d.ts +14 -0
  85. package/dist/meta/host.js +20 -0
  86. package/dist/meta/index.d.ts +11 -0
  87. package/dist/meta/index.js +11 -0
  88. package/dist/meta/launch.d.ts +81 -0
  89. package/dist/meta/launch.js +286 -0
  90. package/dist/meta/model.d.ts +44 -0
  91. package/dist/meta/model.js +119 -0
  92. package/dist/meta/prompts.d.ts +18 -0
  93. package/dist/meta/prompts.js +61 -0
  94. package/dist/meta/relay.d.ts +20 -0
  95. package/dist/meta/relay.js +123 -0
  96. package/dist/meta/tokens.d.ts +52 -0
  97. package/dist/meta/tokens.js +226 -0
  98. package/dist/meta/window.d.ts +61 -0
  99. package/dist/meta/window.js +198 -0
  100. package/dist/meta/wire.d.ts +42 -0
  101. package/dist/meta/wire.js +129 -0
  102. package/dist/modelref.d.ts +16 -0
  103. package/dist/modelref.js +41 -0
  104. package/dist/narrate.js +5 -2
  105. package/dist/podman.d.ts +42 -0
  106. package/dist/podman.js +198 -5
  107. package/dist/progress.d.ts +33 -0
  108. package/dist/progress.js +402 -0
  109. package/dist/projects.d.ts +0 -5
  110. package/dist/projects.js +2 -15
  111. package/dist/request.d.ts +45 -0
  112. package/dist/request.js +300 -0
  113. package/dist/sandbox.d.ts +2 -0
  114. package/dist/sandbox.js +20 -3
  115. package/dist/scaffold.d.ts +15 -6
  116. package/dist/scaffold.js +5 -60
  117. package/dist/session.d.ts +26 -1
  118. package/dist/session.js +46 -32
  119. package/dist/term.d.ts +5 -0
  120. package/dist/term.js +28 -3
  121. package/dist/trace.d.ts +84 -0
  122. package/dist/trace.js +737 -0
  123. package/dist/tui/models.d.ts +33 -0
  124. package/dist/tui/models.js +389 -0
  125. package/dist/tui/wrap.d.ts +5 -0
  126. package/dist/tui/wrap.js +12 -0
  127. package/dist/usage.d.ts +78 -0
  128. package/dist/usage.js +51 -0
  129. package/dist/validate.js +25 -3
  130. package/package.json +3 -2
  131. package/templates/editor/.github/copilot-instructions.md +41 -107
  132. package/templates/editor/.github/prompts/analyze.prompt.md +31 -0
  133. package/templates/editor/.github/prompts/dataset.prompt.md +108 -0
  134. package/templates/editor/.github/prompts/finetune-apply.prompt.md +91 -0
  135. package/templates/editor/.github/prompts/finetune.prompt.md +744 -0
  136. package/templates/editor/.github/prompts/new-agent.prompt.md +3 -1
  137. package/templates/editor/.github/prompts/new-skill.prompt.md +2 -0
  138. package/templates/editor/.github/prompts/project-review.prompt.md +4 -3
  139. package/templates/editor/.github/skills/zen-analyze-run/SKILL.md +483 -0
  140. package/templates/editor/.github/skills/zen-cli/SKILL.md +18 -4
  141. package/templates/editor/.github/skills/zen-cli/references/check.md +7 -5
  142. package/templates/editor/.github/skills/zen-cli/references/dataset.md +34 -0
  143. package/templates/editor/.github/skills/zen-cli/references/finetune.md +77 -0
  144. package/templates/editor/.github/skills/zen-cli/references/inspect.md +300 -21
  145. package/templates/editor/.github/skills/zen-cli/references/memory.md +121 -28
  146. package/templates/editor/.github/skills/zen-cli/references/meta.md +61 -0
  147. package/templates/editor/.github/skills/zen-cli/references/models.md +6 -2
  148. package/templates/editor/.github/skills/zen-cli/references/projects.md +12 -5
  149. package/templates/editor/.github/skills/zen-cli/references/rag.md +24 -6
  150. package/templates/editor/.github/skills/zen-cli/references/run.md +201 -4
  151. package/templates/editor/.github/skills/zen-cli/references/sandbox.md +10 -1
  152. package/templates/editor/.github/skills/zen-code-python/SKILL.md +123 -0
  153. package/templates/editor/.github/skills/zen-code-python/references/code_python.md +48 -0
  154. package/templates/editor/.github/skills/zen-dataset/SKILL.md +253 -0
  155. package/templates/editor/.github/skills/zen-finetune/SKILL.md +1791 -0
  156. package/templates/editor/.github/skills/zen-finetune/references/readme-template.md +268 -0
  157. package/templates/editor/.github/skills/zen-finetune/scripts/batch.mjs +390 -0
  158. package/templates/editor/.github/skills/zen-finetune/scripts/difficult.mjs +243 -0
  159. package/templates/editor/.github/skills/zen-finetune/scripts/memory.mjs +280 -0
  160. package/templates/editor/.github/skills/zen-finetune/scripts/next.mjs +398 -0
  161. package/templates/editor/.github/skills/zen-finetune/scripts/report.mjs +797 -0
  162. package/templates/editor/.github/skills/zen-finetune/scripts/select.mjs +354 -0
  163. package/templates/editor/.github/skills/zen-finetune/scripts/usage.mjs +722 -0
  164. package/templates/editor/.github/skills/zen-inspect/SKILL.md +431 -0
  165. package/templates/editor/.github/skills/zen-inspect-ask/SKILL.md +270 -0
  166. package/templates/editor/.github/skills/zen-memory/SKILL.md +78 -29
  167. package/templates/editor/.github/skills/zen-memory-warmup/SKILL.md +363 -531
  168. package/templates/editor/.github/skills/zen-rag-docs/SKILL.md +20 -8
  169. package/templates/editor/.github/skills/zen-review/SKILL.md +5 -3
  170. package/templates/editor/.github/skills/zen-review/scripts/check-paths.sh +5 -4
  171. package/templates/editor/.github/skills/zen-sandbox/SKILL.md +25 -2
  172. package/templates/editor/.github/skills/zen-sandbox-capacity/SKILL.md +334 -0
  173. package/templates/editor/.github/skills/zen-sandbox-capacity/references/loadtest.md +181 -0
  174. package/templates/editor/.github/skills/zen-sandbox-capacity/scripts/preflight_sandbox.sh +457 -0
  175. package/templates/editor/.github/skills/zen-topology/SKILL.md +283 -0
  176. package/templates/editor/.vscode/agents.schema.json +1 -1
  177. package/templates/editor/.vscode/settings.json +2 -0
  178. package/templates/project/agents/files-instructions.md +35 -13
  179. package/templates/project/agents/fork-instructions.md +27 -0
  180. package/templates/project/agents/memory-instructions.md +106 -34
  181. package/templates/project/agents/tools-instructions.md +7 -4
  182. package/templates/project/gitignore +5 -0
  183. package/dist/meta.d.ts +0 -221
  184. package/dist/meta.js +0 -684
@@ -387,6 +387,7 @@ outline and the copies on disk.
387
387
  zen rag docs list <files|sections|tables> [-d <dir>] [filters…]
388
388
  zen rag docs grep <pattern> [-d <dir>] [filters…]
389
389
  zen rag docs show <file> [--section <name>] [--lines <from-to>]
390
+ zen rag docs show <target>... [--max-lines <n>]
390
391
  zen rag docs stats
391
392
  ```
392
393
 
@@ -411,6 +412,7 @@ zen rag docs grep "Retry-After" # every matching line, and its sec
411
412
  zen rag docs grep "X-RateLimit-\w+" --regex --file "api/**"
412
413
  zen rag docs show api/routing.md --section "Rate limits"
413
414
  zen rag docs show api/routing.md --lines 40-80
415
+ zen rag docs show "api/routing.md:24-31" "api/errors.md#Retries" # several, one call
414
416
  zen rag docs stats
415
417
  ```
416
418
 
@@ -431,6 +433,14 @@ the whole point of it.
431
433
  line-number gutter (`--no-numbers` to drop it). It is the natural follow-up to
432
434
  every search: find the passage, then read what is actually around it.
433
435
 
436
+ **Read every passage you will cite in one `show`.** It takes any number of
437
+ targets written the way a citation is - `<document>:<a>-<b>`, `<document>:<a>`,
438
+ `<document>#<heading>`, or a bare name - and prints each under a header that is
439
+ itself a target, in the order given. Overlapping ranges print once; a missing
440
+ document or heading is reported in place while the rest still print. The
441
+ `--max-lines` budget (default 400) is shared fairly, and a cut block ends with
442
+ the target that continues it. One call per passage costs a whole turn each.
443
+
434
444
  `stats` says what is in an index and what built it - the documents, the counts,
435
445
  the embedding model. It is the fastest way to answer "is this index the one I
436
446
  think it is?".
@@ -443,6 +453,7 @@ think it is?".
443
453
  | _anything you would have run `grep` for_ | `grep` / `list` - never the shell |
444
454
  | "does `X-Request-Id` appear anywhere?" | `grep X-Request-Id` |
445
455
  | "what does the rate-limits section actually say?" | `show <file> --section "Rate limits"` |
456
+ | "read the four passages the search turned up" | `show <doc>:<a>-<b> <doc>:<a>-<b> ...` - once |
446
457
  | "which documents are even in here?" | `list files` |
447
458
  | "what is the shape of this manual?" | `list sections --depth 2` |
448
459
  | "where is the table of per-route limits?" | `list tables --file "api/**"` |
@@ -470,6 +481,7 @@ Add `-d <dir>` when the index is not the nearest one, or name it once with
470
481
  | `grep '^#' file.md` | `zen rag docs list sections --file file.md` |
471
482
  | `sed -n '40,80p' file.md` | `zen rag docs show file.md --lines 40-80` |
472
483
  | `cat file.md` (to find one section) | `zen rag docs show file.md --section "Rate limits"` |
484
+ | a `sed`/`show` per passage | `zen rag docs show a.md:40-80 b.md:12-30` |
473
485
  | `ls` the index directory | `zen rag docs stats` |
474
486
 
475
487
  ```sh
@@ -497,12 +509,12 @@ const index = await docs.DocsIndex.open(
497
509
  const project = await loadProject('./my-project', { tools: docs.docsTools(index) });
498
510
  ```
499
511
 
500
- | Tool | For |
501
- | ------------- | ------------------------------------------------------------ |
502
- | `search_docs` | the passages that match, quoted with their line numbers |
503
- | `list_docs` | the documents, their headings, or their tables - no search |
504
- | `grep_docs` | every matching line, counted in full - no search |
505
- | `read_docs` | a section or a line range, verbatim and with nothing omitted |
512
+ | Tool | For |
513
+ | ------------- | ------------------------------------------------------------------------------------------ |
514
+ | `search_docs` | the passages that match, quoted with their line numbers |
515
+ | `list_docs` | the documents, their headings, or their tables - no search |
516
+ | `grep_docs` | every matching line, counted in full - no search |
517
+ | `read_docs` | a section or a line range, verbatim and with nothing omitted; several at once with `reads` |
506
518
 
507
519
  Only `search_docs` ranks; the other three are exact, because a model told "no
508
520
  results" by a vector search has learned nothing - a ranking returns the top of a
@@ -602,8 +614,8 @@ reference and `guides/` for the task documentation.
602
614
  - Does X exist, how is it spelled, how many are there → `grep_docs`. It is
603
615
  complete; a search is not, and cannot prove absence.
604
616
  - Limits and defaults are in tables → add `kind: ["table_row"]`.
605
- - Then `read_docs` the lines around the hit before answering. Never answer from
606
- the excerpt alone.
617
+ - Then `read_docs` the lines around every hit before answering, all of them in
618
+ one call with `reads`. Never answer from the excerpt alone.
607
619
  - Never `grep`/`rg`/`cat` the tree - the tools above are local and exact.
608
620
 
609
621
  Worked: per-route rate limits are the table under "Rate limits" in
@@ -58,9 +58,11 @@ to open a file that is not there.
58
58
 
59
59
  It exits `1` whenever it prints a candidate, because the list is meant to be
60
60
  emptied or explained rather than skimmed. It strips urls before tokenising, skips
61
- a handful of English constructions that contain a slash, and leaves
62
- `agents/memory-instructions.md` out entirely - that file is worked examples by
63
- design, and `check-instructions.sh` is the check it actually needs.
61
+ a handful of English constructions that contain a slash, and leaves the four
62
+ copies `zen` ships (`agents/{files,fork,memory,tools}-instructions.md`) out
63
+ entirely - they are worked examples by design, `zen check` keeps them
64
+ byte-identical (`rules.stale`), and `check-instructions.sh` is the check memory
65
+ actually needs.
64
66
 
65
67
  Everything it does print is either noise the script has not learned yet or a
66
68
  real finding. There is no third case. A candidate that is genuinely noise is a
@@ -43,9 +43,10 @@ strip_urls() {
43
43
  # The three trees §2.5 names, prose only: a skill's own script is code the
44
44
  # sandbox runs, not text rendered into a prompt, and its paths are its own.
45
45
  #
46
- # `agents/memory-instructions.md` is left out on purpose. It is a copy of the
47
- # reference, full of worked examples, so it fails this sweep by design and
48
- # every run of it — and it has a check of its own that a sweep cannot make:
46
+ # The four copies `zen` ships (files-, fork-, memory-, tools-instructions.md)
47
+ # are left out on purpose. They are full of worked examples, so they fail this
48
+ # sweep by design, and the project cannot edit them anyway: `zen check` reports
49
+ # any byte that differs as `rules.stale`. Memory has a check of its own too:
49
50
  # .github/skills/zen-memory/scripts/check-instructions.sh
50
51
  files() {
51
52
  {
@@ -54,7 +55,7 @@ files() {
54
55
  for file in agents/*instructions.md; do
55
56
  if [ -f "$file" ]; then echo "$file"; fi
56
57
  done
57
- } | grep -v '^agents/memory-instructions\.md$' | sort -u
58
+ } | grep -Ev '^agents/(files|fork|memory|tools)-instructions\.md$' | sort -u
58
59
  }
59
60
 
60
61
  # Every token in those files, urls removed first, and a sentence's full stop
@@ -178,6 +178,26 @@ agents:
178
178
  | `env` | string[] | `[]` | Host environment variables to pass by name. |
179
179
  | `keys` | boolean | `true` | Forward model credentials to the container. |
180
180
 
181
+ ### `memory` and `cpus` divide a machine they cannot enlarge
182
+
183
+ On macOS and Windows these are shares of the podman VM, not of the host. The VM
184
+ is sized **once**, by whichever project first ran `zen sandbox up` on this
185
+ machine, and it defaults to **2048 MiB and 2 CPUs** however large the host is.
186
+ Raising `sandbox.memory` afterwards never resizes it — the init flags are only
187
+ read when no machine exists yet.
188
+
189
+ So the scaffolded default, `memory: 4096`, promises each container twice the
190
+ whole default VM, and podman will admit as many of them as you ask for without
191
+ complaining. Nothing reports this: `zen check` does not, and `zen sandbox
192
+ status` prints no memory figure at all. It surfaces later as an exit 137, or as
193
+ an exit 124 that looks like a slow index.
194
+
195
+ One container is rarely the problem. Before running several at once — a
196
+ `zen run batch` with `--concurrency` above 1 — load **zen-sandbox-capacity** and
197
+ run its `preflight_sandbox.sh`, which measures the VM, the container cap and the
198
+ index and says whether the arithmetic fits. A `zen-finetune` run is the
199
+ exception: it starts at `min(maxBatch + recheck, 16)` and halves on OOM instead.
200
+
181
201
  ---
182
202
 
183
203
  ## 6. Hardening Postures: `standard` vs `strict`
@@ -242,7 +262,8 @@ Use the `zen sandbox` commands via terminal to inspect and manage container stat
242
262
  zen sandbox status # Check podman version, machine status, and active containers
243
263
  zen sandbox up # Start VM (macOS) and pull or build the required image
244
264
  zen sandbox pull # Re-pull base image or force build of sandbox/Dockerfile
245
- zen sandbox clean # Force remove all zn-* containers created by this project
265
+ zen sandbox clean # Force remove all zn-* containers created by this CLI
266
+ zen sandbox clean --idle # Only what no live run uses: stopped and orphaned ones
246
267
  zen sandbox disk # Inspect disk space consumed by images, containers, and volumes
247
268
  ```
248
269
 
@@ -254,4 +275,6 @@ zen sandbox disk # Inspect disk space consumed by images, containers,
254
275
  - **Permission Denied in `/workspace`:**
255
276
  - In `strict` mode without `user:`, rootless Podman uses `keep-id` to preserve host UID. If using custom `user:`, verify host directory permissions allow that UID.
256
277
  - **Disk Space Pressure:**
257
- - Run `zen sandbox clean` to prune stopped session containers.
278
+ - Run `zen sandbox clean --idle` to prune stopped and orphaned session containers.
279
+ - **`could not create container … exceeded num_locks`:**
280
+ - Every container holds one engine lock (2048 by default), stopped ones included. Run `zen sandbox clean --idle`.
@@ -0,0 +1,334 @@
1
+ ---
2
+ name: zen-sandbox-capacity
3
+ description: Whether the machine can actually run the batch you are about to start. The podman VM is the ceiling and it is sized once, at 2 GiB by default, whatever the host has - so N concurrent containers over a multi-gigabyte index thrash and are OOM-killed while the host sits idle. Covers the three layers (host, VM, container) and which one can add memory, the exit-code table (137 is always capacity, 124 is usually 137 in disguise, and raising `timeout:` converts one into the other and hides the fault), the memory budget and how to measure its terms instead of assuming them, separating `local` probes from `embed` ones so a starved index is never mistaken for a throttled provider, recording the machine spec so two rounds' token counts are comparable at all, and `scripts/preflight_sandbox.sh` as the gate that answers it in under a minute. Load before any `zen run batch`, before raising `--concurrency` above 1, on exit 137, on a `killed` message from the shell, on an exit 124 from a `zen rag` call, when a batch is slow or timed out, and before comparing tokens or wall-clock between two rounds. Not for a single interactive `zen run` - one container is not a capacity problem.
4
+ ---
5
+
6
+ # Capacity
7
+
8
+ A container is not a machine. On macOS every container `zen` starts lives inside
9
+ one Linux virtual machine, that VM has a fixed amount of memory, and **the
10
+ number is not the host's**. A laptop with 36 GiB routinely runs a 2 GiB VM, and
11
+ the host being nearly empty is no comfort at all to the process being killed
12
+ inside it.
13
+
14
+ That gap is what this skill exists for. It reads as a provider problem, a
15
+ network problem, a slow-index problem and a flaky-model problem long before it
16
+ reads as what it is, because the symptom arrives as a timeout and timeouts are
17
+ the one error everybody already has a theory about.
18
+
19
+ The evidence is at the bottom. The short version: five tuning rounds, eighty
20
+ minutes of wall clock and six prose edits were spent on an exit 124 produced by
21
+ a VM smaller than the index it was asked to read.
22
+
23
+ ## Does this apply here
24
+
25
+ Three things, and if any is absent say so and stop rather than guessing:
26
+
27
+ | Needs | Without it |
28
+ | --------------------------------------- | --------------------------------------------------------- |
29
+ | `podman` on `PATH` | Nothing below is measurable. Report it and stop |
30
+ | A VM — i.e. macOS or Windows, not Linux | Linux podman is native: no VM, no ceiling, not applicable |
31
+ | An agent holding `sandbox:*` | No container is ever started; capacity is not the fault |
32
+
33
+ The check reports all three and exits `3` for any of them. Docker, Colima and
34
+ Lima are out of scope on purpose: they have their own VM with its own controls,
35
+ and a number read out of podman would be about a machine that is not running the
36
+ work.
37
+
38
+ ## The three layers
39
+
40
+ Routinely confused, and only one of them can add memory.
41
+
42
+ | Layer | Read with | Set with | Can add memory? |
43
+ | ------------- | ---------------------------------------------- | ---------------------------------- | -------------------------------------- |
44
+ | The host | `sysctl -n hw.memsize` | — | No. It bounds what the VM may be given |
45
+ | **The VM** | `podman info --format json` → `.host.memTotal` | `podman machine set --memory` | **Yes. This is the ceiling** |
46
+ | One container | `podman inspect … .HostConfig.Memory` | `sandbox.memory:` in `agents.yaml` | No. It only divides the VM up |
47
+
48
+ `sandbox.memory: 8192` does not give a container 8 GiB. It gives it permission
49
+ to use 8 GiB of whatever the VM has, and a 2 GiB VM cannot honour that promise to
50
+ one container, let alone to eight.
51
+
52
+ ## The machine is sized once, by whichever project created it
53
+
54
+ This is the fact that makes the whole failure mode invisible, and it is worth
55
+ stating on its own.
56
+
57
+ `zen` creates the podman machine the first time any project needs a container,
58
+ with `machine init --cpus <n> --memory <m>` taken from **that project's**
59
+ `sandbox:` block, falling back to **2 CPUs and 2048 MiB**. From then on the
60
+ machine exists, so the code path that sizes it is never reached again.
61
+
62
+ Three consequences:
63
+
64
+ 1. **Editing `sandbox.memory:` afterwards resizes nothing.** It changes the
65
+ per-container cap inside an unchanged VM. The `agents.yaml` diff looks like a
66
+ capacity fix and is not one.
67
+ 2. **The VM may be sized by a project you have forgotten about.** The first
68
+ project to start a container on this host wins, permanently.
69
+ 3. **The scaffolded default is already oversubscribed.** `zen init` writes
70
+ `memory: 4096` per container, and the default VM is 2048 MiB — so one
71
+ container is permitted twice the whole machine, and podman will admit eight of
72
+ them without a word of complaint. Nothing refuses an impossible sum; it is
73
+ only ever discovered by being killed.
74
+
75
+ Resizing is the operator's to do, never this skill's and never the check's:
76
+
77
+ ```sh
78
+ podman machine stop
79
+ podman machine set --memory 12288 --cpus 6
80
+ podman machine start
81
+ ```
82
+
83
+ `machine stop` kills every running container, so it does not happen in the
84
+ middle of somebody's batch. Print the command and hand it over.
85
+
86
+ ## Reading a failure: the exit codes
87
+
88
+ The single most useful table here, because the whole misdiagnosis turns on one
89
+ row of it.
90
+
91
+ | Exit | Means | Points at |
92
+ | ----------: | ---------------------------- | ------------------------------------------------------------- |
93
+ | `124` | `timeout` fired | The work was too slow — **often a symptom of 137 conditions** |
94
+ | `137` | SIGKILL — the **OOM killer** | Capacity. Always. There is no other reading |
95
+ | `143` | SIGTERM | Shutdown or `stop_command`, not capacity |
96
+ | `0`, slowly | Thrash | Capacity, near the edge and not yet over it |
97
+
98
+ **124 and 137 co-occur under memory pressure**, and that is the trap. A VM under
99
+ pressure swaps (or, with no swap, spends the time in reclaim), so everything runs
100
+ an order of magnitude slower, so commands hit their timeout — and the ones that
101
+ do not hit it get killed outright. A run shows both, and 124 is the more common
102
+ of the two, so the 124 gets investigated.
103
+
104
+ Then the obvious repair makes it worse:
105
+
106
+ ```yaml
107
+ sandbox:
108
+ timeout: 600 # was 120 — "the commands just need longer"
109
+ ```
110
+
111
+ Raising the timeout converts 137 into 124 and converts 124 into a very long
112
+ success. It has removed the only unambiguous signal in the system. If a project
113
+ has a `timeout:` well above the 120-second default, treat it as evidence that
114
+ this diagnosis was already made once and got it wrong.
115
+
116
+ ## The budget
117
+
118
+ Two bounds. Report both, always, because they fail for different reasons and
119
+ take different fixes.
120
+
121
+ ### 1. Admission — free, static, and usually the finding
122
+
123
+ ```
124
+ concurrency × sandbox.memory ≤ vm_memory
125
+ ```
126
+
127
+ This is what podman will _permit_ to be allocated. It needs no probe, no
128
+ container and no timing: it is two numbers out of `agents.yaml` and one out of
129
+ `podman info`. When it fails, nothing in the system will refuse the sum — the
130
+ only thing keeping the machine alive is that the containers do not all peak at
131
+ once, which is luck rather than configuration.
132
+
133
+ It is a warning and not the verdict, because a container permitted 4 GiB that
134
+ only ever wants 900 MiB does no harm. The exception is not a warning at all:
135
+ when `sandbox.memory` alone exceeds the VM, the promise is impossible at
136
+ `--concurrency 1` and the batch width has nothing to do with it. That is the
137
+ scaffolded default on an unresized machine — `memory: 4096` against 2048 MiB.
138
+
139
+ ### 2. Working set — measured, and the one that names a number
140
+
141
+ ```
142
+ required_vm_memory ≥ largest_index_size
143
+ + (concurrency × per_container_peak_rss)
144
+ + 1 GiB headroom
145
+ ```
146
+
147
+ Every term is measured, never assumed:
148
+
149
+ - **`largest_index_size`** — `du -sk` over each index under `assets/`. An index
150
+ is read through the page cache, and the cache lives in the VM's memory. A 3.2
151
+ GiB index inside a 2 GiB VM cannot be cached at all, which is why the same
152
+ `grep` takes 5 seconds on the host and 110 inside the container.
153
+ - **`per_container_peak_rss`** — **sampled at `--concurrency 1`**, with
154
+ `podman stats`, running the command the batch will actually run. Never
155
+ estimated from the image size, and never divided out of a loaded run, because a
156
+ run under pressure reports the RSS it was _allowed_, not the one it wanted.
157
+ - **`1 GiB headroom`** — not optional. Check `swapTotal` first: a podman machine
158
+ normally reports **`SwapTotal 0`**, so there is no pressure valve. The kernel
159
+ does not slow down and then recover; it kills something immediately. Headroom
160
+ is the whole of the margin.
161
+
162
+ ## The check
163
+
164
+ ```sh
165
+ .github/skills/zen-sandbox-capacity/scripts/preflight_sandbox.sh --concurrency 8
166
+ ```
167
+
168
+ | Flag | Effect |
169
+ | ------------------- | ------------------------------------------------------ |
170
+ | `--concurrency <n>` | The batch width to answer for. Default 1 |
171
+ | `--assets <dir>` | Where the indexes are. Default `assets` |
172
+ | `--skip-probe` | Admission bound only — instant, and needs no container |
173
+ | `--peak-rss <MiB>` | Use a known figure instead of sampling one |
174
+ | `--json` | The same answer as data |
175
+
176
+ | Exit | Means |
177
+ | ---: | --------------------------------------------------------- |
178
+ | `0` | Sufficient. Start the batch |
179
+ | `1` | Insufficient. The output names by how much, and two fixes |
180
+ | `3` | Not applicable — no podman, no VM, or no sandboxed agent |
181
+
182
+ Non-zero is usable as a gate, which is the point of it being a script:
183
+
184
+ ```sh
185
+ .github/skills/zen-sandbox-capacity/scripts/preflight_sandbox.sh --concurrency 8 \
186
+ && zen run batch --input cases.json --concurrency 8
187
+ ```
188
+
189
+ It **mutates nothing**. It starts one short-lived container with `--rm`, reads,
190
+ and prints; it never stops, resizes or reconfigures the machine.
191
+
192
+ What it prints is a verdict with the numbers that produced it and the exact
193
+ command that fixes it — the shape `zen models test` uses:
194
+
195
+ ```
196
+ host 36.0 GiB total
197
+ vm 2.0 GiB total · 0 swap · 2 cpus podman-machine-default
198
+ container 4096 MiB cap · 4 cpus agents.yaml
199
+ index 3.2 GiB assets/docs-db (largest of 2)
200
+ peak rss 912 MiB sampled at concurrency 1
201
+
202
+ admission 8 x 4096 MiB = 32.0 GiB permitted against a 2.0 GiB vm OVERSUBSCRIBED
203
+ concurrency 8 needs ~11.4 GiB vm has 2.0 GiB INSUFFICIENT
204
+ fix: podman machine stop && podman machine set --memory 12288 && podman machine start
205
+ or: run the batch at --concurrency 1
206
+ ```
207
+
208
+ **Both remedies, every time.** Resizing is not available on every machine — a
209
+ shared host, a CI runner, a laptop with 8 GiB — and lowering concurrency always
210
+ is. A check that offers only the fix the reader cannot apply has told them
211
+ nothing.
212
+
213
+ Target runtime is under 60 seconds so that running it is never a decision. A
214
+ full concurrency sweep would be better evidence and would not get run; that
215
+ sweep is the _diagnostic_ and this is the _gate_. `--skip-probe` drops it to
216
+ about a second when the admission bound is all that is in question.
217
+
218
+ ## Separating the index from the provider
219
+
220
+ Any probe of a capacity problem must label each command as one of two kinds:
221
+
222
+ | Kind | Makes a network call | Examples |
223
+ | ------- | ------------------------------ | ---------------------------------------------- |
224
+ | `local` | No | `zen rag docs stats`, `grep`, `list`, `show` |
225
+ | `embed` | Yes, one, before anything else | `zen rag docs search`, `zen rag schema search` |
226
+
227
+ Without the split, a slow `search` is indistinguishable from a throttled one,
228
+ and the operator is sent to the wrong vendor. With it, the reading is mechanical:
229
+
230
+ | Observed | Verdict |
231
+ | ----------------------------------------------- | -------------------------------------- |
232
+ | Container slow, host fast | The VM. Capacity, and only capacity |
233
+ | Both slow, `embed` only | The embedding endpoint. Not this skill |
234
+ | Both slow, `local` too | Neither — the index itself |
235
+ | Container slow on `local`, host slow on `embed` | Both, independently |
236
+
237
+ This is also why the check probes with a `local` command. A gate that needed a
238
+ credential and a network round trip would fail for reasons that have nothing to
239
+ do with the machine.
240
+
241
+ ## A machine spec is part of a round
242
+
243
+ Record it into the run directory, next to the findings:
244
+
245
+ ```sh
246
+ {
247
+ echo "host $(($(sysctl -n hw.memsize) / 1073741824)) GiB"
248
+ podman info --format json | jq -r '.host | "vm \(.memTotal / 1073741824 | floor) GiB · swap \(.swapTotal) · \(.cpus) cpus"'
249
+ echo "batch --concurrency 8"
250
+ } > finetune/runs/batch01-nomem-run1/machine.txt
251
+ ```
252
+
253
+ The consequence is not bookkeeping, it is validity: **token counts and
254
+ wall-clock are only comparable within one machine spec.** A round that ran at 2
255
+ GiB and a round that ran at 12 GiB differ in trajectory length for reasons that
256
+ have nothing to do with the prose — retries, timeouts, abandoned branches and
257
+ compactions all move with pressure. Resizing the VM invalidates a baseline
258
+ exactly as editing a prompt does, and for the same reason.
259
+
260
+ And the harder rule: **a round containing any exit 137 is void, not graded.** Its
261
+ trajectories are of a system being killed, not of a system following
262
+ instructions. No prose may be changed on its evidence. Re-run it on a machine
263
+ that fits and grade that.
264
+
265
+ ## The host side too
266
+
267
+ The containers are not the only process. `zen run batch` is itself a Node
268
+ process on the host, holding every item's envelope, and it has been OOM-killed
269
+ in its own right — the shell reports `zsh: killed` and there is no run directory
270
+ to inspect, which looks like nothing happened at all.
271
+
272
+ Two remedies, in order:
273
+
274
+ 1. Lower `--concurrency`. It reduces both sides at once.
275
+ 2. `NODE_OPTIONS=--max-old-space-size=8192 zen run batch …`, when the host
276
+ genuinely has the memory and Node's default heap is the binding constraint.
277
+
278
+ The check reports host total and free alongside the VM's, so the two are never
279
+ confused for each other.
280
+
281
+ ## When to escalate
282
+
283
+ The check answers _is there enough_. When the answer is yes and it is still
284
+ slow, the question has become _what is it spending the time on_, and that wants
285
+ a sweep rather than a gate. The method is in
286
+ [references/loadtest.md](./references/loadtest.md): a 2×2 of host against
287
+ container and `local` against `embed`, at rising concurrency, written to CSV.
288
+
289
+ Escalate when, and only when:
290
+
291
+ - The check passes and a batch is still timing out.
292
+ - The verdict is "both slow", so the split has not settled which side it is.
293
+ - A resize did not move the numbers — that refutes the capacity hypothesis and
294
+ something else is wrong.
295
+
296
+ Do not escalate to establish that 8 concurrent containers do not fit in 2 GiB.
297
+ The check already said so, and a sweep that confirms it has spent twenty minutes
298
+ agreeing.
299
+
300
+ ## The evidence this skill was written from
301
+
302
+ One project, one afternoon, measured rather than reasoned:
303
+
304
+ | Fact | Measured |
305
+ | --------------------------- | ------------------------------------------ |
306
+ | podman VM default on macOS | 2 GiB, on a host with 36 GiB |
307
+ | `docs` index | 3.2 GiB — larger than the whole VM |
308
+ | `docs grep`, host, c=8 | 5.5 s |
309
+ | `docs grep`, container, c=8 | 105–118 s, and **3 of 8 exited 137** |
310
+ | Embedding calls, c=8 | ~2× latency, and **zero 429s** |
311
+ | VM swap | **0** — no valve, so the kill is immediate |
312
+
313
+ Read the last two rows together. The provider was never throttling; the
314
+ hypothesis that it was survived five rounds because the only visible symptom was
315
+ a timeout, and the one measurement that would have refuted it — the same command
316
+ run on the host — was never made.
317
+
318
+ ## What this does not cover
319
+
320
+ - **Sandbox configuration generally.** `persist`, mounts, `keys`, hardening,
321
+ images: load **zen-sandbox**.
322
+ - **Docker, Colima, Lima, or Linux-native podman.** Detected and reported as not
323
+ applicable, never attempted.
324
+ - **Provider rate limits.** A probe that finds throttling hands off; diagnosing
325
+ a quota is the provider's documentation and **zen-cli**'s `zen models test`.
326
+ - **Index construction and sizing.** How large an index ought to be, and how to
327
+ make it smaller, is **zen-rag-docs**.
328
+
329
+ ## Related
330
+
331
+ - **zen-sandbox** — what a container is, how to configure it, and the mounts
332
+ - **zen-finetune** — the tuning loop this is a precondition of; its Phase 3
333
+ - **zen-cli** — `zen sandbox status|up|disk`, and what each exit code means
334
+ - **zen-inspect** — finding the failed node once the round has been re-run