@ngockhoale/ukit 3.5.10 → 3.5.13

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -2,6 +2,43 @@
2
2
 
3
3
  All notable changes to UKit are documented here.
4
4
 
5
+ ## 3.5.13 - 2026-11-20
6
+
7
+ Memory-safe test design and execution. A field Python/Sanic backend (~790 files, ~10k tests) grew one pytest process to ~5 GB RSS and swapped for hours; adding pytest-xdist workers made it worse. The cause was a per-test leak in app fixtures, not runner speed. UKit now helps agents write tests that do not leak and run large suites without swapping. No new command: the skills route to the runner. This release also ships the 3.5.12 Stop-gate fix, which reached git and GitHub but not npm.
8
+
9
+ - **Full-suite runner `.claude/ukit/index/test-batches.mjs`** (Node builtins only; pytest, vitest and jest adapters in `test-batches-adapters.mjs`, with a documented adapter contract for other frameworks).
10
+ - `run` discovers tests natively (pytest `--collect-only`, `vitest list`, `jest --listTests`) and writes a deterministic manifest. It then runs whole-file batches one at a time, each in a fresh process (`--batch-files`, `--batch-size` cases).
11
+ - Every file runs. RAM thresholds only flag heavy batches. When free memory is critically low, it waits a bounded time and then stops as INCOMPLETE.
12
+ - Verdicts: PASS 0, FAIL 1, INCOMPLETE 2 (crash, timeout, collection error, unaccounted ids, deferred batches), UNSAFE 3 (a `--guard` real-data path changed). Only PASS satisfies a release gate.
13
+ - Each batch records a log, duration, ms/test and peak RSS (`/usr/bin/time`: bytes on macOS, KiB on Linux; pytest `ru_maxrss` fallback; unknown is never 0). A multi-file batch peak is never attributed to one file; measured-heavy files run solo in later plans.
14
+ - Leftover child processes are reaped. Temp dirs and the injected plugin are removed, and only the newest 5 runs are kept.
15
+ - **Other modes.**
16
+ - `plan` shows the inventory and batches without running anything.
17
+ - `audit [--changed]` checks case budgets, including parametrize expansion (unit 40 · integration 20 · app-heavy 10 · e2e 10 per file, configurable in `.ukit/storage/test-batches.json`). It also flags fixtures without teardown, app-per-test registries, real data paths and timer leaks. These are errors only on new files; legacy files are reported, never rewritten.
18
+ - `detect-leaks --file` reports in-process RSS/heap growth, and `--resource-warnings` turns on pytest `-W error::ResourceWarning`.
19
+ - **Skills and agents.**
20
+ - testing-quality gains "Writing Memory-Safe Tests" and "Memory-Safe Full Suite", plus a `memory-safe-tests.md` reference covering layout planning, fixture lifecycle, test data safety, leak diagnosis and cleanup. The skill now installs as a directory.
21
+ - code-review gains test-diff rules: missing fixture teardown or per-test app registries are P1; reachable real data is P0/P1; skipping tests to save RAM is P1.
22
+ - The Claude and omp `code-reviewer` and `feature-implementer` agents carry the same rules.
23
+ - **Memory.** A seed-once `docs/memory/testing.md` (never overwritten; added to existing projects by `ukit update`) lets `ukit memory recall` surface these patterns on test tasks.
24
+ - Discovery failures name a missing local framework install instead of a bare npx error.
25
+
26
+ ## 3.5.12 - 2026-11-20
27
+
28
+ The Stop gate blocked every stop on a heavily loaded machine. On a field host with load average 22-40, starting any process took about 500ms. The completion evaluator's 6 git calls therefore took about 6s, against a fixed 3s slice. Every Stop was reported as "evaluator could not finish" until the breaker released it after 7 tries. No new command.
29
+
30
+ - **Evaluator slice scales with the deadline.** `resolveLedgerBudgetMs` now gives the evaluator `deadline - 3000ms` (at least 3000ms, at most 8000ms), so the shipped deadline gives 7000ms. Short deadlines keep their old slices.
31
+ - **Longer Stop wrapper deadline.** `completion-gate.sh` arms 10000ms (was 5000ms); the chain-runner slot is `:16` (was `:12`), and the host Stop timeout is 26s (was 22s) so it keeps the 10s margin over the chain. omp is unaffected: its bridge runs the evaluator in-process with no time slice.
32
+ - Still fail-closed: a real overrun still blocks and still feeds the same breaker.
33
+
34
+ ## 3.5.11 - 2026-11-19
35
+
36
+ Hotfix for 3.5.10 output compression. Since 3.5.10 the summary replaces the Bash output on Claude Code, and the replacement also hit commands whose output *is* the content: `sed -n 25,120p file` returned ten mangled lines instead of the code, and chained status commands (`node -p …; npm view …; git status`) lost whole sections.
37
+
38
+ - **Content reads pass through raw.** `cat`, `sed`, `head`, `tail`, `nl`, `bat`, `less`, `more`, `jq`, `diff`, `awk`, `cut`, `git show`/`diff`/`blame`/`cat-file` and `git log -p` are never summarized. Listing-only `git diff --stat`/`--name-only` still compress.
39
+ - **Bounded and chained commands pass through raw.** A pipeline that already ends in `head`/`tail`/`sed`, or several commands chained with `;`/`&&`/`||` (a leading `cd …` is ignored), keeps its full output.
40
+ - Test runs, builds, installs, searches and long listings compress exactly as in 3.5.10.
41
+
5
42
  ## 3.5.10 - 2026-11-19
6
43
 
7
44
  Sessions compacted after two or three prompts. Three causes, all fixed at the source; no new command. No host was run live; the checks are unit, parity and hook-process tests.
@@ -914,6 +914,17 @@ entries:
914
914
  load_policy: on-demand
915
915
  validation: [manual]
916
916
  archive_policy: never
917
+ - id: tpl-docs-memory-testing
918
+ path: template_project/docs/memory/testing.md
919
+ class: canonical
920
+ audience: [agent, maintainer]
921
+ owner: product
922
+ source_of_truth: template_project/docs/memory/testing.md
923
+ merge_strategy: skip
924
+ load_policy: on-demand
925
+ validation: [manual]
926
+ archive_policy: never
927
+ notes: '3.5.13: seed-once default testing patterns (memory-safe tests) so memory recall surfaces them on test tasks; user-owned after install'
917
928
  - id: tpl-docs-worklog
918
929
  path: template_project/docs/WORKLOG.md
919
930
  class: canonical
@@ -44,6 +44,19 @@ items:
44
44
  packs:
45
45
  - core
46
46
 
47
+ # Default testing patterns for memory recall (docs/memory/*.md is a recall source). Seed-once:
48
+ # created only when absent (new installs AND updates of existing projects), never overwritten.
49
+ - id: docs-memory-testing
50
+ type: config
51
+ sourceTemplate: docs/memory/testing.md
52
+ targetPath: docs/memory/testing.md
53
+ requires: []
54
+ mergeStrategy: skip
55
+ variables: []
56
+ enabledByDefault: true
57
+ packs:
58
+ - core
59
+
47
60
  - id: runtime-readme
48
61
  type: config
49
62
  sourceTemplate: ukit/README.md
@@ -474,8 +487,8 @@ items:
474
487
 
475
488
  - id: testing-quality-skill
476
489
  type: skill
477
- sourceTemplate: .claude/skills/testing-quality/SKILL.md
478
- targetPath: .claude/skills/testing-quality/SKILL.md
490
+ sourceTemplate: .claude/skills/testing-quality
491
+ targetPath: .claude/skills/testing-quality
479
492
  requires:
480
493
  - core-skill-delivery
481
494
  mergeStrategy: overwrite_with_backup
@@ -1701,6 +1714,30 @@ items:
1701
1714
  packs:
1702
1715
  - core
1703
1716
 
1717
+ # Memory-safe test execution + test-design audit (testing-quality skill § Memory-Safe Full Suite).
1718
+ - id: ukit-test-batches-script
1719
+ type: config
1720
+ sourceTemplate: .claude/ukit/index/test-batches.mjs
1721
+ targetPath: .claude/ukit/index/test-batches.mjs
1722
+ requires:
1723
+ - ukit-test-batches-adapters-script
1724
+ mergeStrategy: overwrite_with_backup
1725
+ variables: []
1726
+ enabledByDefault: true
1727
+ packs:
1728
+ - core
1729
+
1730
+ - id: ukit-test-batches-adapters-script
1731
+ type: config
1732
+ sourceTemplate: .claude/ukit/index/test-batches-adapters.mjs
1733
+ targetPath: .claude/ukit/index/test-batches-adapters.mjs
1734
+ requires: []
1735
+ mergeStrategy: overwrite_with_backup
1736
+ variables: []
1737
+ enabledByDefault: true
1738
+ packs:
1739
+ - core
1740
+
1704
1741
  - id: ukit-ci-check-script
1705
1742
  type: config
1706
1743
  sourceTemplate: .claude/ukit/index/ci-check.mjs
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@ngockhoale/ukit",
3
- "version": "3.5.10",
3
+ "version": "3.5.13",
4
4
  "description": "Install/update an index-first AI workspace for Claude Code, OpenAI Codex and omp (Oh My Pi).",
5
5
  "license": "MIT",
6
6
  "type": "module",
@@ -67,6 +67,7 @@ If any input is missing, return `CHANGES-REQUESTED` with reason "incomplete hand
67
67
  4. **Safety / security / data loss** — Destructive actions, auth/permission, path handling, unsafe shell/DB/file ops.
68
68
  5. **Performance / scale** — Accidental N+1, repeated I/O, large scans inside hot paths.
69
69
  6. **Maintainability** — Duplicated logic, dead branches, misleading naming, drift between docs/tests/source.
70
+ 6a. **Test hygiene (test diffs)** — Follow `.claude/skills/code-review/SKILL.md` § Test Diffs: app/server/client/DB fixtures without teardown, real-data reachability, per-test app registries, oversized new test files (count parametrize expansion), and tests skipped to save time/RAM. Run `node .claude/ukit/index/test-batches.mjs audit --changed`; an enforced ERROR on a new test file is `CHANGES-REQUESTED` unless the project records an exception.
70
71
  7. **Solution fit** — For each added block, ask with evidence and cite the construct + why (never the prose length):
71
72
  - Does new code duplicate an existing helper with the same semantics? A same-semantics duplicate is `CHANGES-REQUESTED` — reuse the existing helper.
72
73
  - Would the standard library or a native platform API satisfy every stated requirement? If yes, the custom implementation is avoidable.
@@ -43,6 +43,7 @@ and even then, report it, don't ask about it.
43
43
  ### 2. Plan Approach (< 1 minute)
44
44
 
45
45
  - List files to create/modify (max diff).
46
+ - **Tests you add:** plan the layout first (feature, category, expected case count incl. parametrize, target files, fixture teardown) and follow `.claude/skills/testing-quality/SKILL.md` § Writing Memory-Safe Tests — small focused files, every app/client/DB fixture torn down, test paths isolated before app import. Full-suite verification uses `node .claude/ukit/index/test-batches.mjs run`.
46
47
  - **Handoff mode only:** if no Test Plan exists in the task file and task is not `trivial`, write one inline before implementing (happy + ≥2 edge cases of different kinds; regression test if fixing a bug). In daily mode, skip this step.
47
48
  - **Choose the smallest solution that fully meets the stated requirements and project conventions.** When more than one option qualifies, prefer, in order: (1) reuse an existing codebase helper with the same semantics; (2) standard library, native platform API, or an already-installed dependency; (3) custom code or a new dependency (last resort). Never drop validation, security, accessibility, error handling, compatibility, or required tests to reduce code. A bugfix fixes the root cause and the affected callers. If the small option is insufficient, build the full option — never ship a weaker version and ask. No speculative abstraction for imagined needs.
48
49
 
@@ -86,10 +86,10 @@ fi
86
86
  __ukit_coord_out="$(mktemp "${TMPDIR:-/tmp}/ukit-coord-out.XXXXXX")"
87
87
  __ukit_coord_err="$(mktemp "${TMPDIR:-/tmp}/ukit-coord-err.XXXXXX" 2>/dev/null)" || __ukit_coord_err=/dev/null
88
88
  __ukit_coord_t0=$SECONDS
89
- printf '%s' "$INPUT" | UKIT_HARNESS="${UKIT_HARNESS:-claude-code}" UKIT_HOOK_DEADLINE_MS="${UKIT_HOOK_DEADLINE_MS:-5000}" node "$SCRIPT" --evaluate-stop > "$__ukit_coord_out" 2>"$__ukit_coord_err" &
89
+ printf '%s' "$INPUT" | UKIT_HARNESS="${UKIT_HARNESS:-claude-code}" UKIT_HOOK_DEADLINE_MS="${UKIT_HOOK_DEADLINE_MS:-10000}" node "$SCRIPT" --evaluate-stop > "$__ukit_coord_out" 2>"$__ukit_coord_err" &
90
90
  __ukit_coord_pid=$!
91
- __ukit_coord_ms="${UKIT_HOOK_DEADLINE_MS:-5000}"
92
- case "$__ukit_coord_ms" in ''|*[!0-9]*) __ukit_coord_ms=5000 ;; esac
91
+ __ukit_coord_ms="${UKIT_HOOK_DEADLINE_MS:-10000}"
92
+ case "$__ukit_coord_ms" in ''|*[!0-9]*) __ukit_coord_ms=10000 ;; esac
93
93
  __ukit_coord_ms=$((__ukit_coord_ms + 1500))
94
94
  printf -v __ukit_coord_s '%d.%03d' $((__ukit_coord_ms / 1000)) $((__ukit_coord_ms % 1000))
95
95
  ( sleep "$__ukit_coord_s" 2>/dev/null; kill -9 "$__ukit_coord_pid" 2>/dev/null ) <&- >/dev/null 2>&1 &
@@ -213,8 +213,8 @@
213
213
  "hooks": [
214
214
  {
215
215
  "type": "command",
216
- "command": "node \"$CLAUDE_PROJECT_DIR/.claude/ukit/runtime/hook-chain-runner.mjs\" --emit-verdict - \"$CLAUDE_PROJECT_DIR/.claude/hooks/completion-gate.sh\":12",
217
- "timeout": 22
216
+ "command": "node \"$CLAUDE_PROJECT_DIR/.claude/ukit/runtime/hook-chain-runner.mjs\" --emit-verdict - \"$CLAUDE_PROJECT_DIR/.claude/hooks/completion-gate.sh\":16",
217
+ "timeout": 26
218
218
  }
219
219
  ]
220
220
  }
@@ -46,6 +46,20 @@ Check in this order:
46
46
  - misleading naming
47
47
  - drift between docs/tests/source
48
48
 
49
+ ## Test Diffs (when the change adds or edits tests)
50
+
51
+ Run `node .claude/ukit/index/test-batches.mjs audit --changed` first. It statically enforces case budgets and fixture teardown on new test files. Then verify by reading:
52
+
53
+ - **Fixture builds an app/server/client/DB/browser without teardown** → **P1**. Flag `return app` fixtures, `yield` with nothing after it, missing `afterAll`/`afterEach` close, and uncancelled background tasks. These leak per test and caused multi-GB suites in the field.
54
+ - **App per test with unique names / global registries** (`Sanic(f"app_{uuid}")`, OpenAPI/docs singletons, module-level routers keeping app refs) → **P1**, unless teardown unregisters and clears them.
55
+ - **Real data reachable** → **P0** if it writes, P1 if it reads. Examples: hardcoded user/project paths, app paths captured at import time, test env set after the app import, background sweeps/fetch left enabled.
56
+ - **Oversized new test file** over the category budget (count parametrize expansion) without a recorded exception → **P2**. Suggest a split by behaviour that keeps fixture lifecycles intact.
57
+ - **Shared mutable state / order dependence / unclosed files, sessions or subprocesses** → P2 (P1 if it already flakes).
58
+ - **Tests skipped, deselected or xfailed to save time or RAM**, or a full-suite claim backed only by the last batch → **P1**.
59
+ - **Legacy test files**: report violations for refactoring. Never ask to delete, exclude or bulk-rewrite their tests.
60
+
61
+ Details and remediation patterns: `.claude/skills/testing-quality/memory-safe-tests.md`.
62
+
49
63
  ## Workflow
50
64
 
51
65
  ### 1. Gather minimal context
@@ -1,6 +1,6 @@
1
1
  ---
2
2
  name: testing-quality
3
- description: Quality-driven testing methodology combining Test-Driven Development (TDD) principles with common anti-pattern prevention - enforces test-first implementation and clean, reliable test practices
3
+ description: Quality-driven testing methodology combining Test-Driven Development (TDD) principles with common anti-pattern prevention - enforces test-first implementation and clean, reliable test practices. Also use when writing or planning tests (small focused files, fixture teardown, isolated test data) and when running a full or large test suite, or when tests are slow, swap, or grow in RAM (memory-safe sequential batches).
4
4
  ---
5
5
 
6
6
  # Testing Quality & TDD
@@ -77,6 +77,33 @@ After green only:
77
77
  **Good:** TDD cycle includes integration tests.
78
78
  **Fix:** Write failing integration test first.
79
79
 
80
+ ## Writing Memory-Safe Tests
81
+
82
+ Applies whenever you create or substantially change tests. Full rules + examples: `memory-safe-tests.md` (same folder).
83
+
84
+ 1. **Plan the test layout before writing.** Note the feature, category (unit / integration / app-heavy / e2e), expected case count *including parametrize expansion*, the files you will create, and how fixtures are torn down. Default per-file budgets: unit 40 · integration 20 · app-heavy 10 · e2e 10. Over budget → split by behaviour into focused files. Never write one 200-case file when the cases group into smaller modules. Do not split if it breaks a fixture lifecycle.
85
+ 2. **Every fixture that builds something releases it.** App, server, client, DB, browser, thread, subprocess, async task: use `yield` + `try/finally` (pytest) or `afterEach`/`afterAll` (JS). Cancel background tasks, close connections, unregister the app from global registries, clear docs/OpenAPI singletons. Prefer one app per module (`scope="module"`) when tests do not mutate app state. Never give each test a uniquely named app without unregistering it.
86
+ 3. **Isolate data before import.** Point data, DB, log and telemetry paths to a temp dir *before* the app module is imported. Disable schedulers, sweeps and auto-fetch. App code reads paths from config at call time, not at import. Tests never touch real project data, real auth/system DBs or shared logs.
87
+ 4. **Tests stay independent.** No shared mutable state, no order dependence, and each test passes when run alone.
88
+
89
+ Check your work: `node .claude/ukit/index/test-batches.mjs audit --changed`. It enforces budgets and fixture teardown on **new** test files only. Legacy files are reported, never rewritten, deleted or excluded.
90
+
91
+ ## Memory-Safe Full Suite
92
+
93
+ When the whole suite must run (release gate, "run all tests", large suites, or any suite that has swapped or grown in RAM):
94
+
95
+ ```
96
+ node .claude/ukit/index/test-batches.mjs run --guard <real-data-dir>
97
+ ```
98
+
99
+ - Runs **sequentially in small batches, one fresh process per batch**, so the OS gets RAM back after each batch. Do not add parallel workers (`-n`, `--maxWorkers`) as a memory fix — workers multiply RAM.
100
+ - **Every file always runs.** The RAM threshold only flags heavy batches/files (MB) in the report. It never skips or kills tests. If free RAM is critically low, the runner waits a bounded time and then stops as `INCOMPLETE`. It never reports green with batches left.
101
+ - The aggregated report gives total time, max batch RSS, pass/fail/skip/xfail/error totals, every failed id, unaccounted ids, heavy files and isolation findings. Exit codes: `0 PASS · 1 FAIL · 2 INCOMPLETE · 3 UNSAFE`. Only `PASS` satisfies a release gate.
102
+ - Suspected leak → `detect-leaks --file <heavy file>` for in-process growth evidence, plus `--resource-warnings` to surface unclosed DB connections. Fix the fixture teardown; smaller batches alone do not fix a per-test leak.
103
+ - After the work: delete probe scripts, temp logs and superseded run logs, and remove tools you tried and abandoned (for example xdist). Nothing left over.
104
+
105
+ Never skip, deselect or `xfail` tests to save RAM or time. Never loosen trust/permission settings to make tests pass.
106
+
80
107
  ## Quick Reference
81
108
 
82
109
  | Quality | Good | Bad |
@@ -95,3 +122,6 @@ Before marking work complete:
95
122
  - [ ] All tests pass (green)
96
123
  - [ ] No test-only methods in production code
97
124
  - [ ] Mocks mirror real API structure
125
+ - [ ] New test files within the case budget (or a recorded exception); app/client fixtures torn down
126
+ - [ ] Test paths/env isolated before app import; no real data touched
127
+ - [ ] Full-suite claims backed by a `test-batches.mjs run` verdict of `PASS` (not "the last batch passed")
@@ -0,0 +1,133 @@
1
+ # Memory-Safe Tests — reference
2
+
3
+ Read on demand from `SKILL.md`. Origin: a Python/Sanic backend (~790 files, ~10k tests) where one pytest
4
+ process grew to ~5 GB RSS and swapped for hours; `pytest-xdist -n4` was worse (4 × 1.2–1.9 GB). Batches near
5
+ 750 MB ran ~0.16 s/test, batches near 2.4 GB ran 0.66–1.2 s/test. The leak was per **test**, not per file, so
6
+ smaller files alone did not fix it. Those numbers are that project's incident evidence, not UKit benchmarks.
7
+
8
+ ## 1. Plan the test layout before writing (lightweight)
9
+
10
+ Before creating or substantially changing tests, write a 3–6 line plan in your working notes:
11
+
12
+ ```
13
+ feature: <name> category: unit | integration | app-heavy | e2e
14
+ cases: ~<N incl. parametrize expansion> → files: <file A (cases)>, <file B (cases)>
15
+ fixtures: <what is built, scope, how it is torn down>
16
+ isolation: <temp dirs / env set before app import; background jobs disabled>
17
+ ```
18
+
19
+ Starting budgets (defaults, override in `.ukit/storage/test-batches.json` → `budgets`):
20
+
21
+ | Category | max cases / file | max cases / batch |
22
+ |---|---:|---:|
23
+ | unit | 40 | 100 |
24
+ | integration | 20 | 40 |
25
+ | app-heavy (builds an app/server/client) | 10 | 20 |
26
+ | e2e (browser/full stack) | 10 | 10 |
27
+
28
+ - Count **expanded** cases: `parametrize([...30 items])` is 30 cases, not 1.
29
+ - Split by feature/behaviour (`test_orders_create.py`, `test_orders_refund.py`), never by arbitrary halves.
30
+ - Do not split when it would break a module-scoped fixture's lifecycle or scatter one behaviour across files.
31
+ - A justified exception goes in config: `"exceptions": { "<test file>": "<reason>" }`.
32
+
33
+ ## 2. Fixture lifecycle rules
34
+
35
+ Any fixture that creates an app, server, client, DB connection/pool, browser, worker, thread, subprocess or
36
+ async task MUST release it, including on failure and partial setup:
37
+
38
+ **pytest**
39
+ ```python
40
+ @pytest.fixture(scope="module") # one app per module when tests do not mutate app state
41
+ def app(tmp_path_factory):
42
+ app = create_app(config={"DATA_DIR": tmp_path_factory.mktemp("data"), "SCHEDULER": False})
43
+ try:
44
+ yield app
45
+ finally:
46
+ for task in list(getattr(app, "background_tasks", [])): # framework-specific task handles
47
+ task.cancel()
48
+ app.ctx.db.close()
49
+ Sanic._app_registry.pop(app.name, None) # global app registry (Sanic); clear other singletons too
50
+ openapi_registry.clear() # docs/OpenAPI singletons that accumulate per app
51
+ gc.collect()
52
+ ```
53
+ - Module-level blueprints/routers that remember every app they were attached to: build them inside a
54
+ factory, or detach/reset them in teardown.
55
+ - Unique app names per test (`Sanic(f"app_{uuid}")`) are a leak signal — every name stays registered.
56
+ - Use `request.addfinalizer` or `try/finally` when setup has several steps, so partial setup is cleaned.
57
+
58
+ **vitest / jest**
59
+ ```js
60
+ let server;
61
+ beforeAll(async () => { server = await buildApp({ dataDir: tmpDir }); await server.listen({ port: 0 }); });
62
+ afterAll(async () => { await server?.close(); }); // also clearInterval, await pool.end(), browser.close()
63
+ ```
64
+ - `vi.useFakeTimers()` must be paired with `vi.useRealTimers()`; timers need `clearInterval`/`unref`.
65
+
66
+ Do not promote every fixture to module/session scope — only when reuse cannot leak state between tests.
67
+ Every test must still pass alone and in any order.
68
+
69
+ ## 3. Test data safety (mandatory)
70
+
71
+ - Set the test environment **before the app module is imported** (top of `conftest.py`, vitest
72
+ `setupFiles`/`globalSetup`): data, DB, log, cache, upload and telemetry paths → a temp dir.
73
+ - App code reads paths from config **at call time**. A module-level `DATA_DIR = os.environ[...]` captured at
74
+ import silently points tests at real data; refactor it to read from app config.
75
+ - Disable schedulers, sweeps, auto-fetch, polling and startup migrations unless the test is about them.
76
+ - Never point tests at the real project data dir, real auth/system DB or the shared log file.
77
+ - Prove it: run the suite with `--guard <real data dir>` (repeatable). Any change there makes the run `UNSAFE`.
78
+
79
+ ## 4. Running the full suite: `test-batches.mjs`
80
+
81
+ ```
82
+ node .claude/ukit/index/test-batches.mjs plan # inventory + batches, runs nothing
83
+ node .claude/ukit/index/test-batches.mjs run --guard data # full suite, sequential fresh-process batches
84
+ node .claude/ukit/index/test-batches.mjs audit [--changed] # design-time budgets + fixture teardown + real paths
85
+ node .claude/ukit/index/test-batches.mjs detect-leaks --file <heavy test file>
86
+ ```
87
+ Options: `--framework pytest|vitest|jest`, `--batch-files N` (default 10), `--batch-size <cases>`,
88
+ `--heavy-mb N` (flag only, default 1024), `--critical-free-mb N` (default 200), `--timeout-ms N`,
89
+ `--resource-warnings` (pytest `-W error::ResourceWarning`), `--python <path>`, `--out <dir>`, `--json`,
90
+ `-- <framework args>`. Project defaults: `.ukit/storage/test-batches.json`.
91
+
92
+ Guarantees:
93
+ - Native discovery (pytest `--collect-only` node ids, `vitest list`, `jest --listTests`); batches are whole
94
+ files sorted deterministically; no name-regex filtering. One batch at a time, one fresh process each, no
95
+ parallel workers added; leftover child processes of a batch are reaped before the next one.
96
+ - Every file runs. RAM thresholds only **flag** heavy batches; they never skip, filter or kill tests.
97
+ - Low memory before a batch → bounded wait (2 min), then the run stops as `INCOMPLETE` (never green).
98
+ - Verdicts / exit codes: `PASS` 0 · `FAIL` 1 (test failures/errors, file import errors) · `INCOMPLETE` 2
99
+ (crash, timeout, collection error, unaccounted ids, deferred batches) · `UNSAFE` 3 (guarded path changed).
100
+ `PASS` requires every discovered id (pytest/vitest) or file (jest) reported by a completed batch.
101
+ - Report: `.ukit/storage/test-batches/runs/<run>/summary.json` + `manifest.json` + per-batch `output.log`
102
+ and `batch.json` (duration, ms/test, peak RSS + its source). Only the newest 5 runs are kept; temp files
103
+ and the injected plugin are removed after every run.
104
+ - Peak RSS: macOS `/usr/bin/time -l` (bytes), Linux `/usr/bin/time -v` (KiB), else pytest `ru_maxrss`
105
+ in-process; Windows without these → `unknown`, never 0. A multi-file batch peak is never attributed to one
106
+ file; single-file batches are stored in `history.json` and later plans run measured-heavy files solo.
107
+ - Each batch gets `UKIT_TEST_TMPDIR` (removed afterwards) — conftest/setup files may use it as the root for
108
+ data/log/DB paths.
109
+
110
+ Release gates and "run the whole suite" requests use `run`. Targeted checks while coding still run the one
111
+ relevant test file directly — no batching needed for that.
112
+
113
+ ## 5. Leak diagnosis
114
+
115
+ `detect-leaks --file <f>` runs the file **twice in one process** (pytest `--keep-duplicates`) or samples
116
+ heap every 250 ms (vitest/jest) and classifies `warmup-then-flat` / `some-growth-recheck` /
117
+ `likely-unbounded-growth`. It also lists threads/child processes alive at session end and
118
+ `ResourceWarning`s. It is evidence, not a root cause: confirm with `tracemalloc` snapshots (Python) or
119
+ `--expose-gc` heap snapshots (Node) before changing code. Comparing the peak RSS of two fresh processes does
120
+ not prove a leak.
121
+
122
+ ## 6. Other frameworks — adapter contract
123
+
124
+ Add an adapter object to `test-batches-adapters.mjs` exposing `discover`, `batchCommand`, `parseReport` and
125
+ optionally `leakCommand` (contract documented at the top of that file). Discovery must use the framework's
126
+ own listing; if it cannot list case ids, return `tests: null` (file-level accounting). Never parse
127
+ human-readable output when a machine-readable reporter exists.
128
+
129
+ ## 7. Cleanup after investigation
130
+
131
+ Delete probe scripts, temp logs and superseded run logs you created. Remove tools you tried and abandoned
132
+ (e.g. a `pytest-xdist` dependency or `-n` flags added only for the experiment). Leave the project as you found
133
+ it plus the intended change.