@cad0p/pi-memory-budget 0.0.0-stage β†’ 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md ADDED
@@ -0,0 +1,19 @@
1
+ # Changelog
2
+
3
+ All notable changes to this project will be documented in this file.
4
+
5
+ ## [0.1.0] - 2026-11-19
6
+
7
+ <!-- USER-EDITABLE SECTION START -->
8
+ **First release: memory budgets for pi extensions.**
9
+
10
+ - `pi-memory-budget run` measures how much RSS an extension adds on top of vanilla `pi -ne` on a fresh session and on synthetic long sessions (default 200 MB and 2x), and exits 1 when it is over budget.
11
+ - Budgets: steady +20 MB (fresh and long), growth with session size +5 MB, transient spike +100 MB and back under the steady budget within 30 s, regression vs a baseline build +5 MB. Each budget can be overridden or waived with a written reason in `budget.json`; waivers are reported, never silent.
12
+ - `pi-memory-budget fixture` writes the deterministic synthetic session (pi session format v3) used for the long cases.
13
+ - Markdown report for PR bodies and JSON for CI artefacts; RSS from `/proc` on Linux and `ps` on macOS, child processes of pi reported separately, event-loop responsiveness via RPC `get_state` pings.
14
+ - Works as a git dependency (`github:cad0p/pi-memory-budget#<sha>`) before the first npm publish: plain ESM, no build step.
15
+ <!-- USER-EDITABLE SECTION END -->
16
+
17
+ ### πŸš€ Features
18
+
19
+ - Memory budget CLI for pi extensions (closes #1)
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Pier Carlo Cadoppi
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md CHANGED
@@ -1,3 +1,143 @@
1
- # Temporary Holding Version
1
+ # pi-memory-budget
2
2
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
3
+ Memory budgets for [pi](https://github.com/earendil-works/pi) extensions: how much RSS an extension adds on top of vanilla pi on a fresh session and on long sessions, as a CI gate.
4
+
5
+ Extensions are loaded in-process, so whatever an extension allocates is added to every pi process. An audit on 2026-11-19 found napkin-context at +810 MB before fixes, pi-devin at about +275 MB on a 420 MB session and pi-steering at about +100 MB. Nothing in review caught it. This package measures it and fails the build.
6
+
7
+ ## How it measures
8
+
9
+ For every case it starts `pi --mode rpc -ne -ns -np --no-themes --no-mcp -nc -e <extension>…` (the candidate) and the same command without `-e` (vanilla, same pi, same cwd, same env), one process at a time, and samples RSS every 200 ms for at least a minimum window (default 30 s). The isolation flags are always passed so that the only difference between the two is the extension under test: no skills, prompt templates, themes, MCP servers or context files.
10
+
11
+ A run first **waits for quiescence**, but never before `--settle-min` (default 30 s) has passed β€” pi does small amounts of work in its first half minute, and the GC harness has forced full GCs by then, identically in every configuration. Quiescence means the first point at which the trailing 10 s of total RSS has a range under 5 MB, capped by `--settle-max` (default 120 s). If the process never settles, steady is reported as *not settled* and the steady check fails with that reason. The run keeps sampling until the release after the peak is observable (peak + `spikeReleaseSeconds`, bounded), so a release is never failed just because the window ended:
12
+
13
+ - **Offline**: every pi run gets `PI_OFFLINE=1`, `PI_SKIP_VERSION_CHECK=1`, `PI_TELEMETRY=0` β€” pi's startup network work (model catalog refresh, version check, telemetry) allocates tens of MB at an unpredictable time and made the vanilla level bimodal (120–178 MB on a quiet box). With it off, vanilla settles within a few MB. `--env` overrides these if a different value is genuinely needed. Limitation: an extension whose startup itself needs the network is measured without that traffic β€” if its cost matters, measure it on a longer window and read the transient spike, not steady.
14
+ - **GC-normalized**: an idle pi allocates almost nothing, so left to itself V8 never runs a major GC and the settled level is whatever garbage was uncollected when allocation stopped β€” run-to-run noise of tens of MB. Every pi run (vanilla and candidate alike, so it cancels out) is therefore started with `--expose-gc` β€” passed through `NODE_OPTIONS`, which reaches node whether pi is a global launcher, a `node_modules/.bin` shim or a plain `node cli.js` β€” plus a tiny harness extension (`src/gc-harness.js`) that forces major GCs at start, whenever the heap has grown by ~4 MB, and periodically. *steady* measures retained memory, not GC timing. `--no-gc` disables it. Garbage only collects in the pi process itself; children are measured exactly as before. On glibc, freed V8 pages and large buffers go back to the OS promptly, so the post-GC level is observable directly; anything malloc keeps is stable across runs and simply part of the measured level.
15
+
16
+ - **total RSS** = the pi pid plus every descendant process (recursive) β€” an extension cannot hide memory in a spawned helper; **main** and **children** are still reported separately for diagnosis.
17
+ - **steady** = median total RSS of the settled 10 s window; **spike** = max; **released** = last sample.
18
+ - Linux reads `/proc/<pid>/status` (`VmRSS`), macOS uses `ps`. Descendant processes of pi are reported separately as *children*.
19
+ - It sends RPC `get_state` every 100 ms and reports the largest response gap, as a check on event-loop blocking (reported, not budgeted).
20
+ - Each configuration runs N times (default 3) and the **median** is used. RSS is noisy by a few MB, so the vanilla baseline always runs in the same job, interleaved with the candidate. The report also shows the run-to-run **steady spread** (maxβˆ’min) per configuration: when the spread of the reference configuration alone exceeds a check's budget, that check is **INCONCLUSIVE** β€” a distinct non-zero verdict (exit code 3 when nothing else failed) β€” because no delta can be trusted against a baseline noisier than the budget. Rerun on a quieter machine or with more `--runs`.
21
+
22
+ Cases:
23
+
24
+ | Case | Session |
25
+ |---|---|
26
+ | `fresh` | `--no-session` |
27
+ | `long` | synthetic session of `--session-mb` MiB (default 200) |
28
+ | `long2x` | synthetic session of twice that size |
29
+
30
+ The long sessions are generated, never copied from real sessions: deterministic for a seed and size, in pi's session format v3 (user and assistant messages with thinking blocks, tool calls and tool results from a few bytes to about 50 KB, branches, custom entries, labels). They are cached under `--fixtures-dir` (default `<tmpdir>/pi-memory-budget`) and copied for every run, because pi may write to them. `pi-memory-budget fixture --mb 200 --out f.jsonl --seed 1` writes one by hand.
31
+
32
+ ## Budgets
33
+
34
+ Approved defaults; all in MB except the release time. Every value is "candidate minus vanilla" (medians).
35
+
36
+ | Budget | Default | Check |
37
+ |---|---:|---|
38
+ | `steadyFresh` | 20 | steady, fresh |
39
+ | `steadyLong` | 20 | steady, long |
40
+ | `growth2x` | 5 | delta(long2x) - delta(long): growth proportional to the session size |
41
+ | `spike` | 100 | max RSS of the candidate minus vanilla steady, every case |
42
+ | `spikeReleaseSeconds` | 30 | seconds from the peak until the delta is back under the steady budget for 3 consecutive samples; the run extends past the window to observe it. If it still cannot be observed (process died, caps), the check is INCONCLUSIVE β€” a distinct non-zero verdict, never a plain FAIL |
43
+ | `regression` | 5 | candidate minus a baseline build (`--baseline-extension`), every case, only when given |
44
+
45
+ If the candidate's own settled level is over the steady budget, the release check measures against that level (+5 MB noise) so one steady problem is not reported twice.
46
+
47
+ ## Usage
48
+
49
+ ```bash
50
+ pnpm add -D github:cad0p/pi-memory-budget#<sha> @earendil-works/pi-coding-agent # before the first npm publish
51
+ pnpm add -D @cad0p/pi-memory-budget @earendil-works/pi-coding-agent # after
52
+
53
+ pnpm exec pi-memory-budget run \
54
+ --extension ./extensions/my-ext/index.ts \
55
+ --baseline-extension ./main/extensions/my-ext/index.ts \
56
+ --config budget.json --report memory-report.md --json memory-report.json
57
+ ```
58
+
59
+ pi is resolved from `--pi <cmd>`, then `node_modules/.bin/pi` of `--cwd`, then `PATH`. Exit code: 0 everything passes or is waived, 1 a budget FAILs, 2 usage or runtime error, 3 no FAIL but at least one check is INCONCLUSIVE.
60
+
61
+ | Option | Meaning |
62
+ |---|---|
63
+ | `--extension <path>` | extension under test, repeatable (several are measured together) |
64
+ | `--baseline-extension <path>` | baseline build, e.g. checked out from `main`, repeatable |
65
+ | `--config <budget.json>` | budgets, waivers, env, setup hooks |
66
+ | `--cwd <dir>` | working directory of pi (a workspace or vault the extension needs) |
67
+ | `--env K=V` | environment variable for every pi process, repeatable |
68
+ | `--runs <n>` | runs per configuration and case (default 3) |
69
+ | `--settle-min <s>` | no settle before this time; the GC harness has collected by then (default 30) |
70
+ | `--window <s>` | minimum window per run (default 30), `--steady-seconds` (default 10), `--settle-max` (default 120) |
71
+ | `--session-mb <n>` | long session size in MiB (default 200) |
72
+ | `--cases <list>` | `fresh,long,long2x` subset |
73
+ | `--seed <n>` | fixture seed (default 1) |
74
+ | `--pi <cmd>`, `--pi-arg <arg>` | pi command and extra arguments for every process |
75
+ | `--no-gc` | do not start pi with `--expose-gc` and the GC harness (comparison runs, non-node pi) |
76
+ | `--report <md>`, `--json <json>` | outputs |
77
+ | `--fixtures-dir <dir>` | fixture cache |
78
+
79
+ A single run needs about `runs x cases x configurations x window` seconds (defaults: 3 x 3 x 2 x 30 s = 9 min). Never more than one pi process runs at a time, and every pi process (and its children) is killed on exit, SIGINT and errors.
80
+
81
+ ### budget.json
82
+
83
+ ```json
84
+ {
85
+ "extensions": ["extensions/napkin-context/index.ts"],
86
+ "env": { "NAPKIN_VAULT": "fixtures/vault" },
87
+ "setup": { "*": "scripts/prepare-workspace.sh" },
88
+ "budgets": { "steadyLong": 30 },
89
+ "waivers": { "growth2x": "the index is linear in the session by design, tracked in #42" }
90
+ }
91
+ ```
92
+
93
+ - Relative paths resolve against the config file; CLI options override the file.
94
+ - `setup` (a command, or case name or `*` mapped to a command) runs with `sh -c` in `--cwd` before every pi process, with `PMB_CASE`, `PMB_CONFIG` (vanilla, candidate, baseline), `PMB_RUN` and `PMB_SESSION_FILE` in its environment.
95
+ - A **waiver** marks a budget as justified. It needs a non-empty reason and is reported as `WAIVED (would FAIL): <reason>` with the measured value, never skipped silently. Raising a limit in `budgets` is the other option; a waiver keeps the original limit visible.
96
+
97
+ ## Reading the report
98
+
99
+ ```
100
+ ## Memory budget: FAIL
101
+
102
+ ### Case: long
103
+
104
+ | Check | Vanilla | Candidate | Delta | Budget | Result |
105
+ |---|---:|---:|---:|---:|---|
106
+ | steady added RSS, long | 262.5 | 305.3 | +42.8 | <= +20 MB | FAIL |
107
+ | transient spike over vanilla steady, long | 262.5 | 305.3 | +42.8 | <= +100 MB | PASS |
108
+ | back under the steadyLong budget after the spike, long | | | 0.0 s | <= 30 s | PASS |
109
+ ```
110
+
111
+ - *Vanilla* and *Candidate* are medians in MB (steady for the steady check, candidate max for the spike); *Delta* is what the extension adds.
112
+ - A second table per case lists steady, the run-to-run steady spread, spike, released, children steady and the max ping gap of every configuration, plus post-GC heap/external/arrayBuffers numbers from the GC harness (diagnostic only β€” budgets stay on total RSS). A large *children* value means the extension spawns processes that the budget on the pi process does not see. A large vanilla *spread* explains an INCONCLUSIVE verdict.
113
+ - *Growth with session size* compares the long and long2x deltas. A FAIL means the extension's memory scales with the session: fix it, or justify it in the PR.
114
+ - JSON (`schema: pi-memory-budget/1`) has the same checks plus aggregates and the raw RSS samples of every run (`[t seconds, rss MB, children MB]`).
115
+
116
+ ## CI
117
+
118
+ ```yaml
119
+ - run: pnpm install --frozen-lockfile
120
+ - name: Memory budget
121
+ run: >
122
+ pnpm exec pi-memory-budget run
123
+ --extension ./extensions/my-ext/index.ts
124
+ --config budget.json
125
+ --report memory-report.md --json memory-report.json
126
+ - if: always()
127
+ run: cat memory-report.md >> "$GITHUB_STEP_SUMMARY"
128
+ - if: always()
129
+ uses: actions/upload-artifact@v4
130
+ with: { name: memory-budget, path: "memory-report.*" }
131
+ ```
132
+
133
+ Run it on `ubuntu-latest` (stable numbers). Use the same pi version for the candidate and the baseline: pi is a devDependency, so both are covered by the lockfile. Copy the "Memory budget" section from [docs/consumer-verification-section.md](docs/consumer-verification-section.md) into your `docs/verification.md`.
134
+
135
+ ## Development
136
+
137
+ ```bash
138
+ pnpm install
139
+ pnpm lint && pnpm typecheck && pnpm test
140
+ PMB_E2E=1 pnpm test:e2e # real pi, a few minutes; CI runs it on ubuntu and macOS
141
+ ```
142
+
143
+ Plain ESM with JSDoc types, no build step. See [docs/verification.md](docs/verification.md) for the checklist. License: MIT.
@@ -0,0 +1,28 @@
1
+ <!-- Paste this section into your repo's docs/verification.md. Replace the extension path. -->
2
+
3
+ ## Memory budget
4
+
5
+ Every pi extension must stay within strict memory budgets compared with vanilla pi (`pi -ne`), on a fresh session and on long sessions. The check is [`@cad0p/pi-memory-budget`](https://github.com/cad0p/pi-memory-budget); do not copy the harness into this repo.
6
+
7
+ | Check | Budget (added RSS vs vanilla) |
8
+ |---|---|
9
+ | Steady, fresh session | <= +20 MB |
10
+ | Steady, long session (200 MB synthetic) | <= +20 MB |
11
+ | Growth with session size (2x vs 1x) | <= +5 MB more; growth proportional to the session needs a justification in the PR |
12
+ | Transient spike | <= +100 MB, and back under the steady budget within 30 s of the peak (held for 3 samples) |
13
+ | Regression vs `main` | <= +5 MB, otherwise justified in the PR |
14
+
15
+ - **CI:** the `memory-budget` step runs `pnpm exec pi-memory-budget run --extension <path> --config budget.json` on `ubuntu-latest` and fails the job on any FAIL. The report is in the job summary and the `memory-budget` artifact.
16
+ - **Before a PR that touches code loaded at startup or per session entry:** run it locally, with the `main` build as `--baseline-extension`, and paste the markdown report under `## Memory budget` in the PR body:
17
+
18
+ ```bash
19
+ pnpm exec pi-memory-budget run --extension ./extensions/<name>/index.ts \
20
+ --baseline-extension <path to main checkout>/extensions/<name>/index.ts \
21
+ --config budget.json --report memory-report.md
22
+ ```
23
+
24
+ - **Regression tests:** a fix for a memory problem lands with a test that fails on the old behaviour (for example a bound on what is retained per session entry), not only with the budget run.
25
+ - **Waivers:** a budget that cannot be met is never skipped. Put it in `budget.json` under `waivers` with a reason that links the issue; the report prints it as WAIVED with the measured value, and the PR body must say why.
26
+ - **Needs a workspace** (a vault, a repo)? Pass `--cwd` and `--env`, or a `setup` command, in `budget.json`.
27
+ - **What is measured:** total RSS of the pi process plus every descendant (an extension cannot hide memory in a spawned helper), after a quiescence wait (RSS stable for 10 s, capped at `--settle-max` 120 s), with every pi process running `--expose-gc` and a GC harness so steady is retained memory, not uncollected garbage. An INCONCLUSIVE verdict (release could not be observed, never-settled reference, or vanilla's own run-to-run spread larger than the budget) fails the job like a FAIL and needs a human look, not a waiver.
28
+ - **Reading the numbers:** medians of 3 runs; RSS noise is a few MB but tens of MB on big sessions. A result within 5 MB of a budget deserves a rerun with `--runs 5` before a verdict.
@@ -0,0 +1,55 @@
1
+ # Verification checklist: pi-memory-budget
2
+
3
+ What must be true before a change to this repo merges, and how to prove it.
4
+
5
+ ## CI gates
6
+
7
+ CI (`.github/workflows/ci.yml`) runs the `test` job on `ubuntu-latest` and `macos-latest`; the contexts are `test (ubuntu-latest)` and `test (macos-latest)`. The job runs, in order:
8
+
9
+ 1. `pnpm install --frozen-lockfile`
10
+ 2. `pnpm lint` (biome)
11
+ 3. `pnpm typecheck` (`tsc --noEmit`, `checkJs`)
12
+ 4. `pnpm test` (unit tests, offline, a few seconds)
13
+ 5. `PMB_E2E=1 pnpm test:e2e`: the real CLI against a real pi (devDependency) and the fixture extensions in `test/fixtures`.
14
+
15
+ The two validators (`validate-package-version`, `validate-release-pr`) only check version and changelog conventions. Correctness lives in the `test` matrix, so both `test` contexts are required checks.
16
+
17
+ **Green means:**
18
+
19
+ - lint, typecheck and unit tests pass on both OSes;
20
+ - the e2e run shows that `ext-pass.js` (about 5 MB) passes with exit 0, `ext-leak.js` (about 60 MB) fails the steady budget with exit 1 and is WAIVED (exit 0) when the config carries a reason, `ext-grow.js` (memory proportional to the session) fails `growth2x`, pi loads the generated fixture with the same entry count, and no pi process is left behind.
21
+
22
+ Every measured pi runs with `PI_OFFLINE=1`, `PI_SKIP_VERSION_CHECK=1` and `PI_TELEMETRY=0` (startup network work made the vanilla level bimodal), and no settle is accepted before `--settle-min` (default 30 s). Steady is GC-normalized since #3: every pi process runs with `--expose-gc` and the harness extension in `src/gc-harness.js`, so the margin being measured is retained memory, not uncollected garbage. On top of that, a check whose reference configuration (vanilla, or the baseline extension for `regression`) has a run-to-run steady spread larger than the check's own budget is INCONCLUSIVE β€” the CLI then exits 3 (no FAIL) β€” instead of turning baseline noise into a red-herring PASS or FAIL. A runner flake should therefore show as INCONCLUSIVE, never as a wrong verdict.
23
+
24
+ If an e2e assertion flakes on a runner, do not widen it silently: the margins are 5 MB (pass) against 20 MB budget and 60 MB (fail) against it, and the GC harness keeps steady noise in the low MB range. A flake means the measurement got noisier than that: check the run-to-run spread in the JSON report (`aggregates.<case>.<config>.steadySpreadMb`) and investigate the sampler or the window first.
25
+
26
+ ## Offline gates (before pushing)
27
+
28
+ ```bash
29
+ pnpm lint && pnpm typecheck && pnpm test
30
+ ```
31
+
32
+ Every commit must pass these three. Do not edit `CHANGELOG.md` or the version on feature branches (the validator rejects it); user-facing notes go in the PR body under `## Release notes`.
33
+
34
+ ## Independent review
35
+
36
+ A second reviewer (human or agent) who did not write the change checks:
37
+
38
+ - the budget numbers against the approved defaults (README table, `DEFAULT_BUDGETS`, the vault note `open-source/github/pi-memory-budget/_about`);
39
+ - that waivers can never be silent: a waiver without a reason is refused, and a waived check is still printed with its measured value;
40
+ - that no pi process can be orphaned (exit, SIGINT, errors) and that at most one pi process runs at a time;
41
+ - that fixtures contain no real session content.
42
+
43
+ ## Live proof, by change type
44
+
45
+ | Change | Proof |
46
+ |---|---|
47
+ | Sampler, runner, settle/window logic | run `pi-memory-budget run` with `--runs 1` on a known extension, compare against `grep VmRSS /proc/<pid>/status` plus the `task/*/children` descendants, taken by hand at the same moment; check `pgrep -f -- "--mode rpc"` is empty afterwards |
48
+ | Fixture generator | `pi-memory-budget fixture --mb 5 --out f.jsonl`, open it with `pi --mode rpc --session f.jsonl -ne` and RPC `get_entries`: the count must equal the generator's entry count (the e2e test automates this); two runs with the same seed must be byte-identical (`cmp`) |
49
+ | Budgets, evaluation, report | unit tests plus one real run with a waiver and one without, pasting both reports in the PR |
50
+ | pi version bump (devDependency) | full e2e on both OSes; if the session format changed, update `src/fixture.mjs` against pi's `docs/session-format.md` |
51
+ | Before the first publish | `npm pack --dry-run --json` lists `src`, `docs`, `README.md`, `CHANGELOG.md`, `LICENSE`; install the git dependency into a scratch project and run the CLI from `node_modules/.bin` |
52
+
53
+ ## Memory budget of this package
54
+
55
+ The CLI itself is not loaded into pi. It is a separate Node process and adds nothing to the pi processes it measures.
package/package.json CHANGED
@@ -1,6 +1,65 @@
1
1
  {
2
2
  "name": "@cad0p/pi-memory-budget",
3
- "version": "0.0.0-stage",
4
- "stub": true,
5
- "description": "Temporary package placeholder for staged publishing"
6
- }
3
+ "version": "0.1.0",
4
+ "description": "Memory budgets for pi extensions: added RSS vs vanilla pi on fresh and long sessions, as a CI gate",
5
+ "type": "module",
6
+ "bin": {
7
+ "pi-memory-budget": "./src/cli.mjs"
8
+ },
9
+ "exports": {
10
+ ".": "./src/index.mjs"
11
+ },
12
+ "private": false,
13
+ "publishConfig": {
14
+ "access": "public",
15
+ "registry": "https://registry.npmjs.org"
16
+ },
17
+ "keywords": [
18
+ "pi",
19
+ "pi-extension",
20
+ "memory",
21
+ "rss",
22
+ "budget",
23
+ "ci"
24
+ ],
25
+ "repository": {
26
+ "type": "git",
27
+ "url": "git+https://github.com/cad0p/pi-memory-budget.git"
28
+ },
29
+ "homepage": "https://github.com/cad0p/pi-memory-budget#readme",
30
+ "bugs": {
31
+ "url": "https://github.com/cad0p/pi-memory-budget/issues"
32
+ },
33
+ "engines": {
34
+ "node": ">=22.19.0"
35
+ },
36
+ "files": [
37
+ "src",
38
+ "README.md",
39
+ "CHANGELOG.md",
40
+ "LICENSE",
41
+ "docs"
42
+ ],
43
+ "scripts": {
44
+ "lint": "biome check src test",
45
+ "lint:fix": "biome check --write src test",
46
+ "typecheck": "tsc --noEmit",
47
+ "test": "node --test test/*.test.mjs",
48
+ "test:e2e": "PMB_E2E=1 node --test test/e2e.test.mjs"
49
+ },
50
+ "devDependencies": {
51
+ "@biomejs/biome": "2.3.14",
52
+ "@earendil-works/pi-coding-agent": "1.5.2",
53
+ "@types/node": "^22.0.0",
54
+ "typescript": "^5.8.0"
55
+ },
56
+ "peerDependencies": {
57
+ "@earendil-works/pi-coding-agent": ">=1.5.0"
58
+ },
59
+ "peerDependenciesMeta": {
60
+ "@earendil-works/pi-coding-agent": {
61
+ "optional": true
62
+ }
63
+ },
64
+ "license": "MIT"
65
+ }
package/src/args.mjs ADDED
@@ -0,0 +1,199 @@
1
+ // Command line parsing for the pi-memory-budget CLI.
2
+
3
+ import { parseArgs } from "node:util";
4
+
5
+ export const USAGE = `pi-memory-budget: memory budgets for pi extensions
6
+
7
+ Usage:
8
+ pi-memory-budget run --extension <path> [options]
9
+ pi-memory-budget fixture --mb <n> --out <file.jsonl> [--seed <n>]
10
+
11
+ run options:
12
+ --extension <path> extension under test (repeatable, passed to pi as -e)
13
+ --baseline-extension <path> baseline build of the extension, e.g. from main (repeatable)
14
+ --config <budget.json> budgets, waivers, env, setup hooks (see README)
15
+ --cwd <dir> working directory of pi (default: current directory)
16
+ --env K=V extra environment variable for every pi process (repeatable)
17
+ --runs <n> runs per configuration and case, median is used (default 3)
18
+ --window <s> measurement window per run in seconds (default 30)
19
+ --steady-seconds <s> settled-window length for steady RSS (default 10)
20
+ --settle-max <s> cap on the quiescence wait; "not settled" past it (default 120)
21
+ --settle-min <s> no settle before this time; every run is forced-GC'd by then (default 30)
22
+ --session-mb <n> size of the long session fixture in MiB (default 200)
23
+ --cases <list> comma-separated: fresh,long,long2x (default all)
24
+ --seed <n> fixture seed (default 1)
25
+ --pi <cmd> pi command (default: ./node_modules/.bin/pi of --cwd, then PATH)
26
+ --pi-arg <arg> extra argument for every pi process (repeatable)
27
+ --no-gc do not start pi with --expose-gc and the GC harness
28
+ (default: on β€” steady is measured after forced GCs)
29
+ --report <file.md> write the markdown report
30
+ --json <file.json> write the JSON report
31
+ --fixtures-dir <dir> fixture cache (default: <tmpdir>/pi-memory-budget)
32
+ --quiet no progress output on stderr
33
+
34
+ Exit code: 0 all budgets pass or are waived, 1 a budget FAILs, 2 usage or runtime error,
35
+ 3 no FAIL but at least one check is INCONCLUSIVE (e.g. vanilla's own run spread
36
+ exceeds the budget, or a release could not be observed).
37
+ `;
38
+
39
+ /**
40
+ * @typedef {object} RunOptions
41
+ * @property {string[]} extensions
42
+ * @property {string[]} baselineExtensions
43
+ * @property {string | undefined} config
44
+ * @property {string | undefined} cwd
45
+ * @property {string[]} env
46
+ * @property {number | undefined} runs
47
+ * @property {number | undefined} window
48
+ * @property {number | undefined} steadySeconds
49
+ * @property {number | undefined} settleMax
50
+ * @property {number | undefined} settleMin
51
+ * @property {number | undefined} sessionMb
52
+ * @property {string[] | undefined} cases
53
+ * @property {number | undefined} seed
54
+ * @property {string | undefined} pi
55
+ * @property {string[]} piArgs
56
+ * @property {string | undefined} report
57
+ * @property {string | undefined} json
58
+ * @property {string | undefined} fixturesDir
59
+ * @property {boolean} noGc
60
+ * @property {boolean} quiet
61
+ */
62
+
63
+ /**
64
+ * @param {string | undefined} v
65
+ * @param {string} name
66
+ * @param {{ min?: number, integer?: boolean }} [rule]
67
+ */
68
+ function num(v, name, rule = {}) {
69
+ if (v === undefined) return undefined;
70
+ const n = Number(v);
71
+ if (
72
+ !Number.isFinite(n) ||
73
+ (rule.integer && !Number.isInteger(n)) ||
74
+ n < (rule.min ?? 0)
75
+ ) {
76
+ throw new UsageError(
77
+ `--${name} must be ${rule.integer ? "an integer" : "a number"} >= ${rule.min ?? 0}, got "${v}"`,
78
+ );
79
+ }
80
+ return n;
81
+ }
82
+
83
+ export class UsageError extends Error {}
84
+
85
+ export const CASES = ["fresh", "long", "long2x"];
86
+
87
+ /**
88
+ * @param {string[]} argv arguments after the "run" subcommand
89
+ * @returns {RunOptions & { help: boolean }}
90
+ */
91
+ export function parseRunArgs(argv) {
92
+ let parsed;
93
+ try {
94
+ parsed = parseArgs({
95
+ args: argv,
96
+ allowPositionals: false,
97
+ options: {
98
+ extension: { type: "string", multiple: true },
99
+ "baseline-extension": { type: "string", multiple: true },
100
+ config: { type: "string" },
101
+ cwd: { type: "string" },
102
+ env: { type: "string", multiple: true },
103
+ runs: { type: "string" },
104
+ window: { type: "string" },
105
+ "steady-seconds": { type: "string" },
106
+ "settle-max": { type: "string" },
107
+ "settle-min": { type: "string" },
108
+ "session-mb": { type: "string" },
109
+ cases: { type: "string" },
110
+ seed: { type: "string" },
111
+ pi: { type: "string" },
112
+ "pi-arg": { type: "string", multiple: true },
113
+ report: { type: "string" },
114
+ json: { type: "string" },
115
+ "fixtures-dir": { type: "string" },
116
+ "no-gc": { type: "boolean" },
117
+ quiet: { type: "boolean" },
118
+ help: { type: "boolean", short: "h" },
119
+ },
120
+ });
121
+ } catch (e) {
122
+ throw new UsageError(e instanceof Error ? e.message : String(e));
123
+ }
124
+ const v = parsed.values;
125
+ let cases;
126
+ if (v.cases !== undefined) {
127
+ cases = v.cases
128
+ .split(",")
129
+ .map((s) => s.trim())
130
+ .filter(Boolean);
131
+ const bad = cases.filter((c) => !CASES.includes(c));
132
+ if (bad.length || cases.length === 0) {
133
+ throw new UsageError(
134
+ `--cases must be a comma-separated subset of ${CASES.join(",")}, got "${v.cases}"`,
135
+ );
136
+ }
137
+ }
138
+ for (const e of v.env ?? []) {
139
+ if (!/^[A-Za-z_][A-Za-z0-9_]*=/.test(e))
140
+ throw new UsageError(`--env expects K=V, got "${e}"`);
141
+ }
142
+ return {
143
+ extensions: v.extension ?? [],
144
+ baselineExtensions: v["baseline-extension"] ?? [],
145
+ config: v.config,
146
+ cwd: v.cwd,
147
+ env: v.env ?? [],
148
+ runs: num(v.runs, "runs", { min: 1, integer: true }),
149
+ window: num(v.window, "window", { min: 1 }),
150
+ steadySeconds: num(v["steady-seconds"], "steady-seconds", { min: 1 }),
151
+ settleMax: num(v["settle-max"], "settle-max", { min: 1 }),
152
+ settleMin: num(v["settle-min"], "settle-min", { min: 0 }),
153
+ sessionMb: num(v["session-mb"], "session-mb", { min: 1 }),
154
+ cases,
155
+ seed: num(v.seed, "seed", { integer: true }),
156
+ pi: v.pi,
157
+ piArgs: v["pi-arg"] ?? [],
158
+ report: v.report,
159
+ json: v.json,
160
+ fixturesDir: v["fixtures-dir"],
161
+ noGc: v["no-gc"] ?? false,
162
+ quiet: v.quiet ?? false,
163
+ help: v.help ?? false,
164
+ };
165
+ }
166
+
167
+ /**
168
+ * @param {string[]} argv arguments after the "fixture" subcommand
169
+ * @returns {{ mb: number, out: string, seed: number, cwd: string | undefined, help: boolean }}
170
+ */
171
+ export function parseFixtureArgs(argv) {
172
+ let parsed;
173
+ try {
174
+ parsed = parseArgs({
175
+ args: argv,
176
+ allowPositionals: false,
177
+ options: {
178
+ mb: { type: "string" },
179
+ out: { type: "string" },
180
+ seed: { type: "string" },
181
+ cwd: { type: "string" },
182
+ help: { type: "boolean", short: "h" },
183
+ },
184
+ });
185
+ } catch (e) {
186
+ throw new UsageError(e instanceof Error ? e.message : String(e));
187
+ }
188
+ const v = parsed.values;
189
+ if (!v.help && (v.mb === undefined || v.out === undefined)) {
190
+ throw new UsageError("fixture needs --mb <n> and --out <file.jsonl>");
191
+ }
192
+ return {
193
+ mb: num(v.mb, "mb", { min: 0.01 }) ?? 0,
194
+ out: v.out ?? "",
195
+ seed: num(v.seed, "seed", { integer: true }) ?? 1,
196
+ cwd: v.cwd,
197
+ help: v.help ?? false,
198
+ };
199
+ }