@zenera/cli 1.1.23 → 1.1.25
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +88 -2
- package/dist/ask.d.ts +51 -0
- package/dist/ask.js +166 -0
- package/dist/banner.js +2 -2
- package/dist/batch.d.ts +83 -0
- package/dist/batch.js +413 -0
- package/dist/catalog.d.ts +11 -1
- package/dist/catalog.js +80 -7
- package/dist/command.d.ts +6 -0
- package/dist/commands/inspect.d.ts +57 -0
- package/dist/commands/inspect.js +539 -91
- package/dist/commands/list.js +128 -19
- package/dist/commands/memory.js +461 -32
- package/dist/commands/meta.js +311 -21
- package/dist/commands/models.js +110 -14
- package/dist/commands/open.js +4 -4
- package/dist/commands/run.js +175 -36
- package/dist/commands/sandbox.js +148 -15
- package/dist/engine.d.ts +24 -0
- package/dist/engine.js +61 -8
- package/dist/eventlog.d.ts +59 -0
- package/dist/eventlog.js +146 -0
- package/dist/liveness.d.ts +7 -0
- package/dist/liveness.js +20 -3
- package/dist/main.js +5 -2
- package/dist/meta/editor.d.ts +9 -0
- package/dist/meta/editor.js +60 -0
- package/dist/meta/finetune/children.d.ts +8 -0
- package/dist/meta/finetune/children.js +76 -0
- package/dist/meta/finetune/command.d.ts +30 -0
- package/dist/meta/finetune/command.js +338 -0
- package/dist/meta/finetune/dataset/anchors.d.ts +41 -0
- package/dist/meta/finetune/dataset/anchors.js +239 -0
- package/dist/meta/finetune/dataset/changes.d.ts +36 -0
- package/dist/meta/finetune/dataset/changes.js +319 -0
- package/dist/meta/finetune/dataset/command.d.ts +17 -0
- package/dist/meta/finetune/dataset/command.js +1011 -0
- package/dist/meta/finetune/dataset/drift.d.ts +33 -0
- package/dist/meta/finetune/dataset/drift.js +71 -0
- package/dist/meta/finetune/dataset/export.d.ts +26 -0
- package/dist/meta/finetune/dataset/export.js +51 -0
- package/dist/meta/finetune/dataset/index.d.ts +2 -0
- package/dist/meta/finetune/dataset/index.js +2 -0
- package/dist/meta/finetune/dataset/sample.d.ts +50 -0
- package/dist/meta/finetune/dataset/sample.js +197 -0
- package/dist/meta/finetune/dataset/schema.d.ts +33 -0
- package/dist/meta/finetune/dataset/schema.js +216 -0
- package/dist/meta/finetune/dataset/store.d.ts +35 -0
- package/dist/meta/finetune/dataset/store.js +177 -0
- package/dist/meta/finetune/dataset/types.d.ts +125 -0
- package/dist/meta/finetune/dataset/types.js +39 -0
- package/dist/meta/finetune/failure.d.ts +15 -0
- package/dist/meta/finetune/failure.js +26 -0
- package/dist/meta/finetune/feedback.d.ts +45 -0
- package/dist/meta/finetune/feedback.js +85 -0
- package/dist/meta/finetune/improvements.d.ts +51 -0
- package/dist/meta/finetune/improvements.js +244 -0
- package/dist/meta/finetune/index.d.ts +2 -0
- package/dist/meta/finetune/index.js +2 -0
- package/dist/meta/finetune/journal.d.ts +35 -0
- package/dist/meta/finetune/journal.js +135 -0
- package/dist/meta/finetune/live.d.ts +74 -0
- package/dist/meta/finetune/live.js +226 -0
- package/dist/meta/finetune/loop.d.ts +3 -0
- package/dist/meta/finetune/loop.js +114 -0
- package/dist/meta/finetune/memories.d.ts +47 -0
- package/dist/meta/finetune/memories.js +154 -0
- package/dist/meta/finetune/report.d.ts +33 -0
- package/dist/meta/finetune/report.js +1284 -0
- package/dist/meta/finetune/retry.d.ts +24 -0
- package/dist/meta/finetune/retry.js +96 -0
- package/dist/meta/finetune/sampler.d.ts +18 -0
- package/dist/meta/finetune/sampler.js +41 -0
- package/dist/meta/finetune/system.d.ts +27 -0
- package/dist/meta/finetune/system.js +132 -0
- package/dist/meta/finetune/train.d.ts +39 -0
- package/dist/meta/finetune/train.js +391 -0
- package/dist/meta/finetune/tui.d.ts +16 -0
- package/dist/meta/finetune/tui.js +291 -0
- package/dist/meta/finetune/tuning.d.ts +167 -0
- package/dist/meta/finetune/tuning.js +289 -0
- package/dist/meta/finetune/usage.d.ts +62 -0
- package/dist/meta/finetune/usage.js +225 -0
- package/dist/meta/host.d.ts +14 -0
- package/dist/meta/host.js +20 -0
- package/dist/meta/index.d.ts +11 -0
- package/dist/meta/index.js +11 -0
- package/dist/meta/launch.d.ts +81 -0
- package/dist/meta/launch.js +286 -0
- package/dist/meta/model.d.ts +44 -0
- package/dist/meta/model.js +119 -0
- package/dist/meta/prompts.d.ts +18 -0
- package/dist/meta/prompts.js +61 -0
- package/dist/meta/relay.d.ts +20 -0
- package/dist/meta/relay.js +123 -0
- package/dist/meta/tokens.d.ts +52 -0
- package/dist/meta/tokens.js +226 -0
- package/dist/meta/window.d.ts +61 -0
- package/dist/meta/window.js +198 -0
- package/dist/meta/wire.d.ts +42 -0
- package/dist/meta/wire.js +129 -0
- package/dist/modelref.d.ts +16 -0
- package/dist/modelref.js +41 -0
- package/dist/narrate.js +5 -2
- package/dist/podman.d.ts +42 -0
- package/dist/podman.js +198 -5
- package/dist/progress.d.ts +33 -0
- package/dist/progress.js +402 -0
- package/dist/projects.d.ts +0 -5
- package/dist/projects.js +2 -15
- package/dist/request.d.ts +45 -0
- package/dist/request.js +300 -0
- package/dist/sandbox.d.ts +2 -0
- package/dist/sandbox.js +20 -3
- package/dist/scaffold.d.ts +15 -6
- package/dist/scaffold.js +5 -60
- package/dist/session.d.ts +26 -1
- package/dist/session.js +46 -32
- package/dist/term.d.ts +5 -0
- package/dist/term.js +28 -3
- package/dist/trace.d.ts +84 -0
- package/dist/trace.js +737 -0
- package/dist/tui/models.d.ts +33 -0
- package/dist/tui/models.js +389 -0
- package/dist/tui/wrap.d.ts +5 -0
- package/dist/tui/wrap.js +12 -0
- package/dist/usage.d.ts +78 -0
- package/dist/usage.js +51 -0
- package/dist/validate.js +25 -3
- package/package.json +3 -2
- package/templates/editor/.github/copilot-instructions.md +41 -107
- package/templates/editor/.github/prompts/analyze.prompt.md +31 -0
- package/templates/editor/.github/prompts/dataset.prompt.md +108 -0
- package/templates/editor/.github/prompts/finetune-apply.prompt.md +91 -0
- package/templates/editor/.github/prompts/finetune.prompt.md +744 -0
- package/templates/editor/.github/prompts/new-agent.prompt.md +3 -1
- package/templates/editor/.github/prompts/new-skill.prompt.md +2 -0
- package/templates/editor/.github/prompts/project-review.prompt.md +4 -3
- package/templates/editor/.github/skills/zen-analyze-run/SKILL.md +483 -0
- package/templates/editor/.github/skills/zen-cli/SKILL.md +18 -4
- package/templates/editor/.github/skills/zen-cli/references/check.md +7 -5
- package/templates/editor/.github/skills/zen-cli/references/dataset.md +34 -0
- package/templates/editor/.github/skills/zen-cli/references/finetune.md +77 -0
- package/templates/editor/.github/skills/zen-cli/references/inspect.md +300 -21
- package/templates/editor/.github/skills/zen-cli/references/memory.md +121 -28
- package/templates/editor/.github/skills/zen-cli/references/meta.md +61 -0
- package/templates/editor/.github/skills/zen-cli/references/models.md +6 -2
- package/templates/editor/.github/skills/zen-cli/references/projects.md +12 -5
- package/templates/editor/.github/skills/zen-cli/references/rag.md +24 -6
- package/templates/editor/.github/skills/zen-cli/references/run.md +201 -4
- package/templates/editor/.github/skills/zen-cli/references/sandbox.md +10 -1
- package/templates/editor/.github/skills/zen-code-python/SKILL.md +123 -0
- package/templates/editor/.github/skills/zen-code-python/references/code_python.md +48 -0
- package/templates/editor/.github/skills/zen-dataset/SKILL.md +253 -0
- package/templates/editor/.github/skills/zen-finetune/SKILL.md +1791 -0
- package/templates/editor/.github/skills/zen-finetune/references/readme-template.md +268 -0
- package/templates/editor/.github/skills/zen-finetune/scripts/batch.mjs +390 -0
- package/templates/editor/.github/skills/zen-finetune/scripts/difficult.mjs +243 -0
- package/templates/editor/.github/skills/zen-finetune/scripts/memory.mjs +280 -0
- package/templates/editor/.github/skills/zen-finetune/scripts/next.mjs +398 -0
- package/templates/editor/.github/skills/zen-finetune/scripts/report.mjs +797 -0
- package/templates/editor/.github/skills/zen-finetune/scripts/select.mjs +354 -0
- package/templates/editor/.github/skills/zen-finetune/scripts/usage.mjs +722 -0
- package/templates/editor/.github/skills/zen-inspect/SKILL.md +431 -0
- package/templates/editor/.github/skills/zen-inspect-ask/SKILL.md +270 -0
- package/templates/editor/.github/skills/zen-memory/SKILL.md +78 -29
- package/templates/editor/.github/skills/zen-memory-warmup/SKILL.md +363 -531
- package/templates/editor/.github/skills/zen-rag-docs/SKILL.md +20 -8
- package/templates/editor/.github/skills/zen-review/SKILL.md +5 -3
- package/templates/editor/.github/skills/zen-review/scripts/check-paths.sh +5 -4
- package/templates/editor/.github/skills/zen-sandbox/SKILL.md +25 -2
- package/templates/editor/.github/skills/zen-sandbox-capacity/SKILL.md +334 -0
- package/templates/editor/.github/skills/zen-sandbox-capacity/references/loadtest.md +181 -0
- package/templates/editor/.github/skills/zen-sandbox-capacity/scripts/preflight_sandbox.sh +457 -0
- package/templates/editor/.github/skills/zen-topology/SKILL.md +283 -0
- package/templates/editor/.vscode/agents.schema.json +1 -1
- package/templates/editor/.vscode/settings.json +2 -0
- package/templates/project/agents/files-instructions.md +35 -13
- package/templates/project/agents/fork-instructions.md +27 -0
- package/templates/project/agents/memory-instructions.md +106 -34
- package/templates/project/agents/tools-instructions.md +7 -4
- package/templates/project/gitignore +5 -0
- package/dist/meta.d.ts +0 -221
- package/dist/meta.js +0 -684
|
@@ -387,6 +387,7 @@ outline and the copies on disk.
|
|
|
387
387
|
zen rag docs list <files|sections|tables> [-d <dir>] [filters…]
|
|
388
388
|
zen rag docs grep <pattern> [-d <dir>] [filters…]
|
|
389
389
|
zen rag docs show <file> [--section <name>] [--lines <from-to>]
|
|
390
|
+
zen rag docs show <target>... [--max-lines <n>]
|
|
390
391
|
zen rag docs stats
|
|
391
392
|
```
|
|
392
393
|
|
|
@@ -411,6 +412,7 @@ zen rag docs grep "Retry-After" # every matching line, and its sec
|
|
|
411
412
|
zen rag docs grep "X-RateLimit-\w+" --regex --file "api/**"
|
|
412
413
|
zen rag docs show api/routing.md --section "Rate limits"
|
|
413
414
|
zen rag docs show api/routing.md --lines 40-80
|
|
415
|
+
zen rag docs show "api/routing.md:24-31" "api/errors.md#Retries" # several, one call
|
|
414
416
|
zen rag docs stats
|
|
415
417
|
```
|
|
416
418
|
|
|
@@ -431,6 +433,14 @@ the whole point of it.
|
|
|
431
433
|
line-number gutter (`--no-numbers` to drop it). It is the natural follow-up to
|
|
432
434
|
every search: find the passage, then read what is actually around it.
|
|
433
435
|
|
|
436
|
+
**Read every passage you will cite in one `show`.** It takes any number of
|
|
437
|
+
targets written the way a citation is - `<document>:<a>-<b>`, `<document>:<a>`,
|
|
438
|
+
`<document>#<heading>`, or a bare name - and prints each under a header that is
|
|
439
|
+
itself a target, in the order given. Overlapping ranges print once; a missing
|
|
440
|
+
document or heading is reported in place while the rest still print. The
|
|
441
|
+
`--max-lines` budget (default 400) is shared fairly, and a cut block ends with
|
|
442
|
+
the target that continues it. One call per passage costs a whole turn each.
|
|
443
|
+
|
|
434
444
|
`stats` says what is in an index and what built it - the documents, the counts,
|
|
435
445
|
the embedding model. It is the fastest way to answer "is this index the one I
|
|
436
446
|
think it is?".
|
|
@@ -443,6 +453,7 @@ think it is?".
|
|
|
443
453
|
| _anything you would have run `grep` for_ | `grep` / `list` - never the shell |
|
|
444
454
|
| "does `X-Request-Id` appear anywhere?" | `grep X-Request-Id` |
|
|
445
455
|
| "what does the rate-limits section actually say?" | `show <file> --section "Rate limits"` |
|
|
456
|
+
| "read the four passages the search turned up" | `show <doc>:<a>-<b> <doc>:<a>-<b> ...` - once |
|
|
446
457
|
| "which documents are even in here?" | `list files` |
|
|
447
458
|
| "what is the shape of this manual?" | `list sections --depth 2` |
|
|
448
459
|
| "where is the table of per-route limits?" | `list tables --file "api/**"` |
|
|
@@ -470,6 +481,7 @@ Add `-d <dir>` when the index is not the nearest one, or name it once with
|
|
|
470
481
|
| `grep '^#' file.md` | `zen rag docs list sections --file file.md` |
|
|
471
482
|
| `sed -n '40,80p' file.md` | `zen rag docs show file.md --lines 40-80` |
|
|
472
483
|
| `cat file.md` (to find one section) | `zen rag docs show file.md --section "Rate limits"` |
|
|
484
|
+
| a `sed`/`show` per passage | `zen rag docs show a.md:40-80 b.md:12-30` |
|
|
473
485
|
| `ls` the index directory | `zen rag docs stats` |
|
|
474
486
|
|
|
475
487
|
```sh
|
|
@@ -497,12 +509,12 @@ const index = await docs.DocsIndex.open(
|
|
|
497
509
|
const project = await loadProject('./my-project', { tools: docs.docsTools(index) });
|
|
498
510
|
```
|
|
499
511
|
|
|
500
|
-
| Tool | For
|
|
501
|
-
| ------------- |
|
|
502
|
-
| `search_docs` | the passages that match, quoted with their line numbers
|
|
503
|
-
| `list_docs` | the documents, their headings, or their tables - no search
|
|
504
|
-
| `grep_docs` | every matching line, counted in full - no search
|
|
505
|
-
| `read_docs` | a section or a line range, verbatim and with nothing omitted |
|
|
512
|
+
| Tool | For |
|
|
513
|
+
| ------------- | ------------------------------------------------------------------------------------------ |
|
|
514
|
+
| `search_docs` | the passages that match, quoted with their line numbers |
|
|
515
|
+
| `list_docs` | the documents, their headings, or their tables - no search |
|
|
516
|
+
| `grep_docs` | every matching line, counted in full - no search |
|
|
517
|
+
| `read_docs` | a section or a line range, verbatim and with nothing omitted; several at once with `reads` |
|
|
506
518
|
|
|
507
519
|
Only `search_docs` ranks; the other three are exact, because a model told "no
|
|
508
520
|
results" by a vector search has learned nothing - a ranking returns the top of a
|
|
@@ -602,8 +614,8 @@ reference and `guides/` for the task documentation.
|
|
|
602
614
|
- Does X exist, how is it spelled, how many are there → `grep_docs`. It is
|
|
603
615
|
complete; a search is not, and cannot prove absence.
|
|
604
616
|
- Limits and defaults are in tables → add `kind: ["table_row"]`.
|
|
605
|
-
- Then `read_docs` the lines around
|
|
606
|
-
the excerpt alone.
|
|
617
|
+
- Then `read_docs` the lines around every hit before answering, all of them in
|
|
618
|
+
one call with `reads`. Never answer from the excerpt alone.
|
|
607
619
|
- Never `grep`/`rg`/`cat` the tree - the tools above are local and exact.
|
|
608
620
|
|
|
609
621
|
Worked: per-route rate limits are the table under "Rate limits" in
|
|
@@ -58,9 +58,11 @@ to open a file that is not there.
|
|
|
58
58
|
|
|
59
59
|
It exits `1` whenever it prints a candidate, because the list is meant to be
|
|
60
60
|
emptied or explained rather than skimmed. It strips urls before tokenising, skips
|
|
61
|
-
a handful of English constructions that contain a slash, and leaves
|
|
62
|
-
`agents/memory-instructions.md` out
|
|
63
|
-
design,
|
|
61
|
+
a handful of English constructions that contain a slash, and leaves the four
|
|
62
|
+
copies `zen` ships (`agents/{files,fork,memory,tools}-instructions.md`) out
|
|
63
|
+
entirely - they are worked examples by design, `zen check` keeps them
|
|
64
|
+
byte-identical (`rules.stale`), and `check-instructions.sh` is the check memory
|
|
65
|
+
actually needs.
|
|
64
66
|
|
|
65
67
|
Everything it does print is either noise the script has not learned yet or a
|
|
66
68
|
real finding. There is no third case. A candidate that is genuinely noise is a
|
|
@@ -43,9 +43,10 @@ strip_urls() {
|
|
|
43
43
|
# The three trees §2.5 names, prose only: a skill's own script is code the
|
|
44
44
|
# sandbox runs, not text rendered into a prompt, and its paths are its own.
|
|
45
45
|
#
|
|
46
|
-
# `
|
|
47
|
-
#
|
|
48
|
-
#
|
|
46
|
+
# The four copies `zen` ships (files-, fork-, memory-, tools-instructions.md)
|
|
47
|
+
# are left out on purpose. They are full of worked examples, so they fail this
|
|
48
|
+
# sweep by design, and the project cannot edit them anyway: `zen check` reports
|
|
49
|
+
# any byte that differs as `rules.stale`. Memory has a check of its own too:
|
|
49
50
|
# .github/skills/zen-memory/scripts/check-instructions.sh
|
|
50
51
|
files() {
|
|
51
52
|
{
|
|
@@ -54,7 +55,7 @@ files() {
|
|
|
54
55
|
for file in agents/*instructions.md; do
|
|
55
56
|
if [ -f "$file" ]; then echo "$file"; fi
|
|
56
57
|
done
|
|
57
|
-
} | grep -
|
|
58
|
+
} | grep -Ev '^agents/(files|fork|memory|tools)-instructions\.md$' | sort -u
|
|
58
59
|
}
|
|
59
60
|
|
|
60
61
|
# Every token in those files, urls removed first, and a sentence's full stop
|
|
@@ -178,6 +178,26 @@ agents:
|
|
|
178
178
|
| `env` | string[] | `[]` | Host environment variables to pass by name. |
|
|
179
179
|
| `keys` | boolean | `true` | Forward model credentials to the container. |
|
|
180
180
|
|
|
181
|
+
### `memory` and `cpus` divide a machine they cannot enlarge
|
|
182
|
+
|
|
183
|
+
On macOS and Windows these are shares of the podman VM, not of the host. The VM
|
|
184
|
+
is sized **once**, by whichever project first ran `zen sandbox up` on this
|
|
185
|
+
machine, and it defaults to **2048 MiB and 2 CPUs** however large the host is.
|
|
186
|
+
Raising `sandbox.memory` afterwards never resizes it — the init flags are only
|
|
187
|
+
read when no machine exists yet.
|
|
188
|
+
|
|
189
|
+
So the scaffolded default, `memory: 4096`, promises each container twice the
|
|
190
|
+
whole default VM, and podman will admit as many of them as you ask for without
|
|
191
|
+
complaining. Nothing reports this: `zen check` does not, and `zen sandbox
|
|
192
|
+
status` prints no memory figure at all. It surfaces later as an exit 137, or as
|
|
193
|
+
an exit 124 that looks like a slow index.
|
|
194
|
+
|
|
195
|
+
One container is rarely the problem. Before running several at once — a
|
|
196
|
+
`zen run batch` with `--concurrency` above 1 — load **zen-sandbox-capacity** and
|
|
197
|
+
run its `preflight_sandbox.sh`, which measures the VM, the container cap and the
|
|
198
|
+
index and says whether the arithmetic fits. A `zen-finetune` run is the
|
|
199
|
+
exception: it starts at `min(maxBatch + recheck, 16)` and halves on OOM instead.
|
|
200
|
+
|
|
181
201
|
---
|
|
182
202
|
|
|
183
203
|
## 6. Hardening Postures: `standard` vs `strict`
|
|
@@ -242,7 +262,8 @@ Use the `zen sandbox` commands via terminal to inspect and manage container stat
|
|
|
242
262
|
zen sandbox status # Check podman version, machine status, and active containers
|
|
243
263
|
zen sandbox up # Start VM (macOS) and pull or build the required image
|
|
244
264
|
zen sandbox pull # Re-pull base image or force build of sandbox/Dockerfile
|
|
245
|
-
zen sandbox clean # Force remove all zn-* containers created by this
|
|
265
|
+
zen sandbox clean # Force remove all zn-* containers created by this CLI
|
|
266
|
+
zen sandbox clean --idle # Only what no live run uses: stopped and orphaned ones
|
|
246
267
|
zen sandbox disk # Inspect disk space consumed by images, containers, and volumes
|
|
247
268
|
```
|
|
248
269
|
|
|
@@ -254,4 +275,6 @@ zen sandbox disk # Inspect disk space consumed by images, containers,
|
|
|
254
275
|
- **Permission Denied in `/workspace`:**
|
|
255
276
|
- In `strict` mode without `user:`, rootless Podman uses `keep-id` to preserve host UID. If using custom `user:`, verify host directory permissions allow that UID.
|
|
256
277
|
- **Disk Space Pressure:**
|
|
257
|
-
- Run `zen sandbox clean` to prune stopped session containers.
|
|
278
|
+
- Run `zen sandbox clean --idle` to prune stopped and orphaned session containers.
|
|
279
|
+
- **`could not create container … exceeded num_locks`:**
|
|
280
|
+
- Every container holds one engine lock (2048 by default), stopped ones included. Run `zen sandbox clean --idle`.
|
|
@@ -0,0 +1,334 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: zen-sandbox-capacity
|
|
3
|
+
description: Whether the machine can actually run the batch you are about to start. The podman VM is the ceiling and it is sized once, at 2 GiB by default, whatever the host has - so N concurrent containers over a multi-gigabyte index thrash and are OOM-killed while the host sits idle. Covers the three layers (host, VM, container) and which one can add memory, the exit-code table (137 is always capacity, 124 is usually 137 in disguise, and raising `timeout:` converts one into the other and hides the fault), the memory budget and how to measure its terms instead of assuming them, separating `local` probes from `embed` ones so a starved index is never mistaken for a throttled provider, recording the machine spec so two rounds' token counts are comparable at all, and `scripts/preflight_sandbox.sh` as the gate that answers it in under a minute. Load before any `zen run batch`, before raising `--concurrency` above 1, on exit 137, on a `killed` message from the shell, on an exit 124 from a `zen rag` call, when a batch is slow or timed out, and before comparing tokens or wall-clock between two rounds. Not for a single interactive `zen run` - one container is not a capacity problem.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Capacity
|
|
7
|
+
|
|
8
|
+
A container is not a machine. On macOS every container `zen` starts lives inside
|
|
9
|
+
one Linux virtual machine, that VM has a fixed amount of memory, and **the
|
|
10
|
+
number is not the host's**. A laptop with 36 GiB routinely runs a 2 GiB VM, and
|
|
11
|
+
the host being nearly empty is no comfort at all to the process being killed
|
|
12
|
+
inside it.
|
|
13
|
+
|
|
14
|
+
That gap is what this skill exists for. It reads as a provider problem, a
|
|
15
|
+
network problem, a slow-index problem and a flaky-model problem long before it
|
|
16
|
+
reads as what it is, because the symptom arrives as a timeout and timeouts are
|
|
17
|
+
the one error everybody already has a theory about.
|
|
18
|
+
|
|
19
|
+
The evidence is at the bottom. The short version: five tuning rounds, eighty
|
|
20
|
+
minutes of wall clock and six prose edits were spent on an exit 124 produced by
|
|
21
|
+
a VM smaller than the index it was asked to read.
|
|
22
|
+
|
|
23
|
+
## Does this apply here
|
|
24
|
+
|
|
25
|
+
Three things, and if any is absent say so and stop rather than guessing:
|
|
26
|
+
|
|
27
|
+
| Needs | Without it |
|
|
28
|
+
| --------------------------------------- | --------------------------------------------------------- |
|
|
29
|
+
| `podman` on `PATH` | Nothing below is measurable. Report it and stop |
|
|
30
|
+
| A VM — i.e. macOS or Windows, not Linux | Linux podman is native: no VM, no ceiling, not applicable |
|
|
31
|
+
| An agent holding `sandbox:*` | No container is ever started; capacity is not the fault |
|
|
32
|
+
|
|
33
|
+
The check reports all three and exits `3` for any of them. Docker, Colima and
|
|
34
|
+
Lima are out of scope on purpose: they have their own VM with its own controls,
|
|
35
|
+
and a number read out of podman would be about a machine that is not running the
|
|
36
|
+
work.
|
|
37
|
+
|
|
38
|
+
## The three layers
|
|
39
|
+
|
|
40
|
+
Routinely confused, and only one of them can add memory.
|
|
41
|
+
|
|
42
|
+
| Layer | Read with | Set with | Can add memory? |
|
|
43
|
+
| ------------- | ---------------------------------------------- | ---------------------------------- | -------------------------------------- |
|
|
44
|
+
| The host | `sysctl -n hw.memsize` | — | No. It bounds what the VM may be given |
|
|
45
|
+
| **The VM** | `podman info --format json` → `.host.memTotal` | `podman machine set --memory` | **Yes. This is the ceiling** |
|
|
46
|
+
| One container | `podman inspect … .HostConfig.Memory` | `sandbox.memory:` in `agents.yaml` | No. It only divides the VM up |
|
|
47
|
+
|
|
48
|
+
`sandbox.memory: 8192` does not give a container 8 GiB. It gives it permission
|
|
49
|
+
to use 8 GiB of whatever the VM has, and a 2 GiB VM cannot honour that promise to
|
|
50
|
+
one container, let alone to eight.
|
|
51
|
+
|
|
52
|
+
## The machine is sized once, by whichever project created it
|
|
53
|
+
|
|
54
|
+
This is the fact that makes the whole failure mode invisible, and it is worth
|
|
55
|
+
stating on its own.
|
|
56
|
+
|
|
57
|
+
`zen` creates the podman machine the first time any project needs a container,
|
|
58
|
+
with `machine init --cpus <n> --memory <m>` taken from **that project's**
|
|
59
|
+
`sandbox:` block, falling back to **2 CPUs and 2048 MiB**. From then on the
|
|
60
|
+
machine exists, so the code path that sizes it is never reached again.
|
|
61
|
+
|
|
62
|
+
Three consequences:
|
|
63
|
+
|
|
64
|
+
1. **Editing `sandbox.memory:` afterwards resizes nothing.** It changes the
|
|
65
|
+
per-container cap inside an unchanged VM. The `agents.yaml` diff looks like a
|
|
66
|
+
capacity fix and is not one.
|
|
67
|
+
2. **The VM may be sized by a project you have forgotten about.** The first
|
|
68
|
+
project to start a container on this host wins, permanently.
|
|
69
|
+
3. **The scaffolded default is already oversubscribed.** `zen init` writes
|
|
70
|
+
`memory: 4096` per container, and the default VM is 2048 MiB — so one
|
|
71
|
+
container is permitted twice the whole machine, and podman will admit eight of
|
|
72
|
+
them without a word of complaint. Nothing refuses an impossible sum; it is
|
|
73
|
+
only ever discovered by being killed.
|
|
74
|
+
|
|
75
|
+
Resizing is the operator's to do, never this skill's and never the check's:
|
|
76
|
+
|
|
77
|
+
```sh
|
|
78
|
+
podman machine stop
|
|
79
|
+
podman machine set --memory 12288 --cpus 6
|
|
80
|
+
podman machine start
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
`machine stop` kills every running container, so it does not happen in the
|
|
84
|
+
middle of somebody's batch. Print the command and hand it over.
|
|
85
|
+
|
|
86
|
+
## Reading a failure: the exit codes
|
|
87
|
+
|
|
88
|
+
The single most useful table here, because the whole misdiagnosis turns on one
|
|
89
|
+
row of it.
|
|
90
|
+
|
|
91
|
+
| Exit | Means | Points at |
|
|
92
|
+
| ----------: | ---------------------------- | ------------------------------------------------------------- |
|
|
93
|
+
| `124` | `timeout` fired | The work was too slow — **often a symptom of 137 conditions** |
|
|
94
|
+
| `137` | SIGKILL — the **OOM killer** | Capacity. Always. There is no other reading |
|
|
95
|
+
| `143` | SIGTERM | Shutdown or `stop_command`, not capacity |
|
|
96
|
+
| `0`, slowly | Thrash | Capacity, near the edge and not yet over it |
|
|
97
|
+
|
|
98
|
+
**124 and 137 co-occur under memory pressure**, and that is the trap. A VM under
|
|
99
|
+
pressure swaps (or, with no swap, spends the time in reclaim), so everything runs
|
|
100
|
+
an order of magnitude slower, so commands hit their timeout — and the ones that
|
|
101
|
+
do not hit it get killed outright. A run shows both, and 124 is the more common
|
|
102
|
+
of the two, so the 124 gets investigated.
|
|
103
|
+
|
|
104
|
+
Then the obvious repair makes it worse:
|
|
105
|
+
|
|
106
|
+
```yaml
|
|
107
|
+
sandbox:
|
|
108
|
+
timeout: 600 # was 120 — "the commands just need longer"
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
Raising the timeout converts 137 into 124 and converts 124 into a very long
|
|
112
|
+
success. It has removed the only unambiguous signal in the system. If a project
|
|
113
|
+
has a `timeout:` well above the 120-second default, treat it as evidence that
|
|
114
|
+
this diagnosis was already made once and got it wrong.
|
|
115
|
+
|
|
116
|
+
## The budget
|
|
117
|
+
|
|
118
|
+
Two bounds. Report both, always, because they fail for different reasons and
|
|
119
|
+
take different fixes.
|
|
120
|
+
|
|
121
|
+
### 1. Admission — free, static, and usually the finding
|
|
122
|
+
|
|
123
|
+
```
|
|
124
|
+
concurrency × sandbox.memory ≤ vm_memory
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
This is what podman will _permit_ to be allocated. It needs no probe, no
|
|
128
|
+
container and no timing: it is two numbers out of `agents.yaml` and one out of
|
|
129
|
+
`podman info`. When it fails, nothing in the system will refuse the sum — the
|
|
130
|
+
only thing keeping the machine alive is that the containers do not all peak at
|
|
131
|
+
once, which is luck rather than configuration.
|
|
132
|
+
|
|
133
|
+
It is a warning and not the verdict, because a container permitted 4 GiB that
|
|
134
|
+
only ever wants 900 MiB does no harm. The exception is not a warning at all:
|
|
135
|
+
when `sandbox.memory` alone exceeds the VM, the promise is impossible at
|
|
136
|
+
`--concurrency 1` and the batch width has nothing to do with it. That is the
|
|
137
|
+
scaffolded default on an unresized machine — `memory: 4096` against 2048 MiB.
|
|
138
|
+
|
|
139
|
+
### 2. Working set — measured, and the one that names a number
|
|
140
|
+
|
|
141
|
+
```
|
|
142
|
+
required_vm_memory ≥ largest_index_size
|
|
143
|
+
+ (concurrency × per_container_peak_rss)
|
|
144
|
+
+ 1 GiB headroom
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
Every term is measured, never assumed:
|
|
148
|
+
|
|
149
|
+
- **`largest_index_size`** — `du -sk` over each index under `assets/`. An index
|
|
150
|
+
is read through the page cache, and the cache lives in the VM's memory. A 3.2
|
|
151
|
+
GiB index inside a 2 GiB VM cannot be cached at all, which is why the same
|
|
152
|
+
`grep` takes 5 seconds on the host and 110 inside the container.
|
|
153
|
+
- **`per_container_peak_rss`** — **sampled at `--concurrency 1`**, with
|
|
154
|
+
`podman stats`, running the command the batch will actually run. Never
|
|
155
|
+
estimated from the image size, and never divided out of a loaded run, because a
|
|
156
|
+
run under pressure reports the RSS it was _allowed_, not the one it wanted.
|
|
157
|
+
- **`1 GiB headroom`** — not optional. Check `swapTotal` first: a podman machine
|
|
158
|
+
normally reports **`SwapTotal 0`**, so there is no pressure valve. The kernel
|
|
159
|
+
does not slow down and then recover; it kills something immediately. Headroom
|
|
160
|
+
is the whole of the margin.
|
|
161
|
+
|
|
162
|
+
## The check
|
|
163
|
+
|
|
164
|
+
```sh
|
|
165
|
+
.github/skills/zen-sandbox-capacity/scripts/preflight_sandbox.sh --concurrency 8
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
| Flag | Effect |
|
|
169
|
+
| ------------------- | ------------------------------------------------------ |
|
|
170
|
+
| `--concurrency <n>` | The batch width to answer for. Default 1 |
|
|
171
|
+
| `--assets <dir>` | Where the indexes are. Default `assets` |
|
|
172
|
+
| `--skip-probe` | Admission bound only — instant, and needs no container |
|
|
173
|
+
| `--peak-rss <MiB>` | Use a known figure instead of sampling one |
|
|
174
|
+
| `--json` | The same answer as data |
|
|
175
|
+
|
|
176
|
+
| Exit | Means |
|
|
177
|
+
| ---: | --------------------------------------------------------- |
|
|
178
|
+
| `0` | Sufficient. Start the batch |
|
|
179
|
+
| `1` | Insufficient. The output names by how much, and two fixes |
|
|
180
|
+
| `3` | Not applicable — no podman, no VM, or no sandboxed agent |
|
|
181
|
+
|
|
182
|
+
Non-zero is usable as a gate, which is the point of it being a script:
|
|
183
|
+
|
|
184
|
+
```sh
|
|
185
|
+
.github/skills/zen-sandbox-capacity/scripts/preflight_sandbox.sh --concurrency 8 \
|
|
186
|
+
&& zen run batch --input cases.json --concurrency 8
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
It **mutates nothing**. It starts one short-lived container with `--rm`, reads,
|
|
190
|
+
and prints; it never stops, resizes or reconfigures the machine.
|
|
191
|
+
|
|
192
|
+
What it prints is a verdict with the numbers that produced it and the exact
|
|
193
|
+
command that fixes it — the shape `zen models test` uses:
|
|
194
|
+
|
|
195
|
+
```
|
|
196
|
+
host 36.0 GiB total
|
|
197
|
+
vm 2.0 GiB total · 0 swap · 2 cpus podman-machine-default
|
|
198
|
+
container 4096 MiB cap · 4 cpus agents.yaml
|
|
199
|
+
index 3.2 GiB assets/docs-db (largest of 2)
|
|
200
|
+
peak rss 912 MiB sampled at concurrency 1
|
|
201
|
+
|
|
202
|
+
admission 8 x 4096 MiB = 32.0 GiB permitted against a 2.0 GiB vm OVERSUBSCRIBED
|
|
203
|
+
concurrency 8 needs ~11.4 GiB vm has 2.0 GiB INSUFFICIENT
|
|
204
|
+
fix: podman machine stop && podman machine set --memory 12288 && podman machine start
|
|
205
|
+
or: run the batch at --concurrency 1
|
|
206
|
+
```
|
|
207
|
+
|
|
208
|
+
**Both remedies, every time.** Resizing is not available on every machine — a
|
|
209
|
+
shared host, a CI runner, a laptop with 8 GiB — and lowering concurrency always
|
|
210
|
+
is. A check that offers only the fix the reader cannot apply has told them
|
|
211
|
+
nothing.
|
|
212
|
+
|
|
213
|
+
Target runtime is under 60 seconds so that running it is never a decision. A
|
|
214
|
+
full concurrency sweep would be better evidence and would not get run; that
|
|
215
|
+
sweep is the _diagnostic_ and this is the _gate_. `--skip-probe` drops it to
|
|
216
|
+
about a second when the admission bound is all that is in question.
|
|
217
|
+
|
|
218
|
+
## Separating the index from the provider
|
|
219
|
+
|
|
220
|
+
Any probe of a capacity problem must label each command as one of two kinds:
|
|
221
|
+
|
|
222
|
+
| Kind | Makes a network call | Examples |
|
|
223
|
+
| ------- | ------------------------------ | ---------------------------------------------- |
|
|
224
|
+
| `local` | No | `zen rag docs stats`, `grep`, `list`, `show` |
|
|
225
|
+
| `embed` | Yes, one, before anything else | `zen rag docs search`, `zen rag schema search` |
|
|
226
|
+
|
|
227
|
+
Without the split, a slow `search` is indistinguishable from a throttled one,
|
|
228
|
+
and the operator is sent to the wrong vendor. With it, the reading is mechanical:
|
|
229
|
+
|
|
230
|
+
| Observed | Verdict |
|
|
231
|
+
| ----------------------------------------------- | -------------------------------------- |
|
|
232
|
+
| Container slow, host fast | The VM. Capacity, and only capacity |
|
|
233
|
+
| Both slow, `embed` only | The embedding endpoint. Not this skill |
|
|
234
|
+
| Both slow, `local` too | Neither — the index itself |
|
|
235
|
+
| Container slow on `local`, host slow on `embed` | Both, independently |
|
|
236
|
+
|
|
237
|
+
This is also why the check probes with a `local` command. A gate that needed a
|
|
238
|
+
credential and a network round trip would fail for reasons that have nothing to
|
|
239
|
+
do with the machine.
|
|
240
|
+
|
|
241
|
+
## A machine spec is part of a round
|
|
242
|
+
|
|
243
|
+
Record it into the run directory, next to the findings:
|
|
244
|
+
|
|
245
|
+
```sh
|
|
246
|
+
{
|
|
247
|
+
echo "host $(($(sysctl -n hw.memsize) / 1073741824)) GiB"
|
|
248
|
+
podman info --format json | jq -r '.host | "vm \(.memTotal / 1073741824 | floor) GiB · swap \(.swapTotal) · \(.cpus) cpus"'
|
|
249
|
+
echo "batch --concurrency 8"
|
|
250
|
+
} > finetune/runs/batch01-nomem-run1/machine.txt
|
|
251
|
+
```
|
|
252
|
+
|
|
253
|
+
The consequence is not bookkeeping, it is validity: **token counts and
|
|
254
|
+
wall-clock are only comparable within one machine spec.** A round that ran at 2
|
|
255
|
+
GiB and a round that ran at 12 GiB differ in trajectory length for reasons that
|
|
256
|
+
have nothing to do with the prose — retries, timeouts, abandoned branches and
|
|
257
|
+
compactions all move with pressure. Resizing the VM invalidates a baseline
|
|
258
|
+
exactly as editing a prompt does, and for the same reason.
|
|
259
|
+
|
|
260
|
+
And the harder rule: **a round containing any exit 137 is void, not graded.** Its
|
|
261
|
+
trajectories are of a system being killed, not of a system following
|
|
262
|
+
instructions. No prose may be changed on its evidence. Re-run it on a machine
|
|
263
|
+
that fits and grade that.
|
|
264
|
+
|
|
265
|
+
## The host side too
|
|
266
|
+
|
|
267
|
+
The containers are not the only process. `zen run batch` is itself a Node
|
|
268
|
+
process on the host, holding every item's envelope, and it has been OOM-killed
|
|
269
|
+
in its own right — the shell reports `zsh: killed` and there is no run directory
|
|
270
|
+
to inspect, which looks like nothing happened at all.
|
|
271
|
+
|
|
272
|
+
Two remedies, in order:
|
|
273
|
+
|
|
274
|
+
1. Lower `--concurrency`. It reduces both sides at once.
|
|
275
|
+
2. `NODE_OPTIONS=--max-old-space-size=8192 zen run batch …`, when the host
|
|
276
|
+
genuinely has the memory and Node's default heap is the binding constraint.
|
|
277
|
+
|
|
278
|
+
The check reports host total and free alongside the VM's, so the two are never
|
|
279
|
+
confused for each other.
|
|
280
|
+
|
|
281
|
+
## When to escalate
|
|
282
|
+
|
|
283
|
+
The check answers _is there enough_. When the answer is yes and it is still
|
|
284
|
+
slow, the question has become _what is it spending the time on_, and that wants
|
|
285
|
+
a sweep rather than a gate. The method is in
|
|
286
|
+
[references/loadtest.md](./references/loadtest.md): a 2×2 of host against
|
|
287
|
+
container and `local` against `embed`, at rising concurrency, written to CSV.
|
|
288
|
+
|
|
289
|
+
Escalate when, and only when:
|
|
290
|
+
|
|
291
|
+
- The check passes and a batch is still timing out.
|
|
292
|
+
- The verdict is "both slow", so the split has not settled which side it is.
|
|
293
|
+
- A resize did not move the numbers — that refutes the capacity hypothesis and
|
|
294
|
+
something else is wrong.
|
|
295
|
+
|
|
296
|
+
Do not escalate to establish that 8 concurrent containers do not fit in 2 GiB.
|
|
297
|
+
The check already said so, and a sweep that confirms it has spent twenty minutes
|
|
298
|
+
agreeing.
|
|
299
|
+
|
|
300
|
+
## The evidence this skill was written from
|
|
301
|
+
|
|
302
|
+
One project, one afternoon, measured rather than reasoned:
|
|
303
|
+
|
|
304
|
+
| Fact | Measured |
|
|
305
|
+
| --------------------------- | ------------------------------------------ |
|
|
306
|
+
| podman VM default on macOS | 2 GiB, on a host with 36 GiB |
|
|
307
|
+
| `docs` index | 3.2 GiB — larger than the whole VM |
|
|
308
|
+
| `docs grep`, host, c=8 | 5.5 s |
|
|
309
|
+
| `docs grep`, container, c=8 | 105–118 s, and **3 of 8 exited 137** |
|
|
310
|
+
| Embedding calls, c=8 | ~2× latency, and **zero 429s** |
|
|
311
|
+
| VM swap | **0** — no valve, so the kill is immediate |
|
|
312
|
+
|
|
313
|
+
Read the last two rows together. The provider was never throttling; the
|
|
314
|
+
hypothesis that it was survived five rounds because the only visible symptom was
|
|
315
|
+
a timeout, and the one measurement that would have refuted it — the same command
|
|
316
|
+
run on the host — was never made.
|
|
317
|
+
|
|
318
|
+
## What this does not cover
|
|
319
|
+
|
|
320
|
+
- **Sandbox configuration generally.** `persist`, mounts, `keys`, hardening,
|
|
321
|
+
images: load **zen-sandbox**.
|
|
322
|
+
- **Docker, Colima, Lima, or Linux-native podman.** Detected and reported as not
|
|
323
|
+
applicable, never attempted.
|
|
324
|
+
- **Provider rate limits.** A probe that finds throttling hands off; diagnosing
|
|
325
|
+
a quota is the provider's documentation and **zen-cli**'s `zen models test`.
|
|
326
|
+
- **Index construction and sizing.** How large an index ought to be, and how to
|
|
327
|
+
make it smaller, is **zen-rag-docs**.
|
|
328
|
+
|
|
329
|
+
## Related
|
|
330
|
+
|
|
331
|
+
- **zen-sandbox** — what a container is, how to configure it, and the mounts
|
|
332
|
+
- **zen-finetune** — the tuning loop this is a precondition of; its Phase 3
|
|
333
|
+
- **zen-cli** — `zen sandbox status|up|disk`, and what each exit code means
|
|
334
|
+
- **zen-inspect** — finding the failed node once the round has been re-run
|