@stablekernel/pi-background-run 0.4.0 → 0.6.0

package/package.json

@@ -1,8 +1,11 @@
 {
   "name": "@stablekernel/pi-background-run",
-  "version": "0.4.0",
+  "version": "0.6.0",
   "description": "Run long shell commands detached in the background for pi; get woken on completion. Output lands in a file; context stays clean.",
   "type": "module",
+  "engines": {
+    "node": ">=20"
+  },
   "license": "MIT",
   "publishConfig": {
     "access": "public"
@@ -15,15 +18,18 @@
       "./extension/index.ts"
     ],
     "skills": [
-      "./skill/run-bg"
+      "./skill/run-bg",
+      "./skill/digest-config"
     ]
   },
   "scripts": {
     "test": "bun test extension/index.test.ts",
-    "lint": "tsc --noEmit"
+    "lint": "tsc --noEmit",
+    "lint:pr-title": "bun scripts/check-conventional-commit.ts"
   },
   "files": [
     "extension/",
+    "!extension/index.test.ts",
     "skill/",
     "README.md",
     "LICENSE"
@@ -56,6 +62,7 @@
   "devDependencies": {
     "@earendil-works/pi-coding-agent": "*",
     "@earendil-works/pi-tui": "*",
+    "@types/node": "^24.0.0",
     "typebox": "^1.3.0",
     "typescript": "^5.7.0"
   }

package/skill/digest-config/SKILL.md

@@ -0,0 +1,117 @@
+---
+name: digest-config
+description: Set up the pi-bgrun digest scorecard for this project. Use when the user asks to configure a digest, enable digest heuristics, or when a pi-bgrun nudge points at this skill. Samples the project's real job logs, picks a shipped preset (go-test, jest, pytest, junit-xml) or drafts a custom digest command, validates it against green AND red logs, then writes the digest section into .pi/pi-bgrun.json.
+---
+
+# Configure a project digest scorecard
+
+Goal: a `digest` section in `<project>/.pi/pi-bgrun.json` whose command turns a
+job log into a short pass/fail scorecard, appended to every `bgrun` wake as
+`digest (<label>): ...`. The scorecard must be reliable on both green and red
+logs — a wrong scorecard is worse than none. Most projects run more than one
+kind of job (unit tests, a build, e2e); configure one entry per job type rather
+than one command that guesses.
+
+Config shapes — a single object (legacy) or an ordered **list** of entries; if
+both `preset` and `command` are set within one entry, the preset wins:
+
+```json
+{ "digest": { "preset": "go-test" } }
+```
+
+```json
+{ "digest": { "command": "grep -E 'FAIL|ok  ' \"$1\" | head -5" } }
+```
+
+```json
+{
+  "digest": [
+    { "type": "test",  "preset": "go-test" },
+    { "type": "build", "label": "build",
+      "command": "grep -E '^error' \"$1\" | head -5" },
+    { "match": { "command": "*cargo*" }, "label": "cargo", "preset": "go-test" },
+    { "preset": "go-test" }
+  ]
+}
+```
+
+Prefer a `type` on each entry: it is matched exactly (case-insensitive) against
+the `type:` the agent passes to `bgrun`, so it does not depend on job names or
+command lines staying stable. When you configure a `type`, tell the agent to
+pass it: `bgrun(command: …, name: …, type: "test")`. If a job's `type`/name
+selects no entry, pi-bgrun logs a one-line diagnostic naming the job and the
+configured types — check it when a scorecard is expected but absent.
+
+Selection (exactly one entry, or none):
+
+1. Entries with a `type` are checked **first**, in config order, and match only
+   a job declaring that exact type that also satisfies the entry's `match` if
+   it has one. First type match wins.
+2. Otherwise the entries **without** a `type` are scanned in config order:
+   `match.name` / `match.command` globs and no-`match` defaults.
+3. No match → no digest.
+
+- `type` and `match` compose (AND): with both present, only a job of that type
+  that also satisfies the glob matches.
+- `match.name` / `match.command` are **globs** tested against the job's
+  `name` and command line; both present → both must match. They are
+  **case-insensitive and whole-string** (`*` any run, `?` one character;
+  write `*text*` for a substring; `\` escapes the next character).
+- An entry with no `match` (or an empty `match`) matches every job — put it
+  **last** as the default. Include one so jobs you did not anticipate still
+  get a scorecard.
+- `label` sets the wake tag; without it a type entry uses its `type` string, a
+  glob entry uses the matched `match.name`, else the preset id (or `command`).
+- An invalid `match` (non-string), an invalid `type`, or an entry with no valid
+  `preset`/`command` is dropped silently; an empty/all-invalid list counts as
+  unconfigured.
+
+Shipped presets: `go-test` (package ok/FAIL counts + failing test names),
+`jest` (Tests/Test Suites summary + failed test names), `pytest` (final
+passed/failed/error summary line + FAILED test ids), `junit-xml`
+(`<failure>`/`<error>` counts + failing testcase names). Each preset also
+carries a `suggestedType` (all `test`) — use it as the `type` when scaffolding
+an entry, e.g. `{ "type": "test", "preset": "go-test" }`. The suggestion is
+advisory; a preset entry with no `type` still applies to every job.
+
+## Procedure
+
+1. **Find done-job logs.** Locate the project's jobsDir from the config
+   layering: in a project root it defaults to `<project>/.pi-bgrun/jobs`;
+   otherwise to the machine-global `~/.pi-bgrun/jobs`. An explicit `jobsDir`
+   (any config layer) or `PI_BGRUN_DIR` overrides it; `PI_BGRUN_GLOBAL_DIR`
+   retargets the machine-global base. The resolved path is printed as
+   `log: <path>` by `bgrun` and by `bgstatus <id>` — read it back there if
+   unsure. List the `*.log` files of finished jobs.
+2. **Sample the formats across job types.** Group the logs by job type using
+   each job's `name` and command line (from `bgstatus`); most projects have at
+   least a test job and a build job. Pick 2-3 logs per type — at least one
+   green and one red run each — and inspect them with `ctx_execute_file`
+   (context-mode sandbox, so only your printed summary enters context).
+   Identify the runner / output format for each type, and name the type with a
+   short token (`test`, `build`, `lint`, `e2e`).
+3. **Try a preset first, per type.** Run each shipped preset's command against
+   a sample log (`sh -c '<preset command>' sh <logpath>`). Preset commands are
+   data in the package's `extension/digestPresets.ts`. Clean scorecard on green
+   AND red samples → use that preset for that type, seeding the entry's
+   `type` from the preset's `suggestedType` (all shipped presets suggest
+   `test`). Repeat for each type.
+4. **Draft custom commands** for types no preset fits. Use awk/sed/grep/jq; the
+   command receives the log path as `$1` and MUST end in `head -N` so output is
+   bounded. Keep it to a count line plus failed-item names.
+5. **Validate each entry on green AND red.** Run every drafted command against
+   every sample for its type. Each must produce a correct scorecard on both: no
+   phantom failures on green logs, no missing failures on red ones. If a type
+   has no reliable command, omit that entry (or leave the digest unconfigured)
+   rather than shipping a wrong scorecard — say why.
+6. **Write the config, one `type` entry per job type.** Merge the entries into
+   `<project>/.pi/pi-bgrun.json`, preserving any existing keys, giving each
+   entry the `type` you identified in step 2, and putting the no-`match`
+   default entry **last**. Use `match.name` / `match.command` globs only for
+   jobs that will not pass a `type`. Create the file if absent. Tell the user
+   (or the agent driving `bgrun`) which `type:` value to pass for each job.
+7. **Smoke-test each entry.** Start a real `bgrun` job of each configured type
+   (e.g. the test command AND the build command), passing the matching
+   `type:`, and check that its wake carries a correct `digest (<label>):` block
+   for the right entry. If a block is empty, wrong, or comes from the wrong
+   entry, fix the command / type / ordering and repeat step 7.

package/skill/run-bg/SKILL.md

@@ -28,15 +28,18 @@ no polling.
 
 | Action | Tool |
 |---|---|
-| Start  | `bgrun(command: "make test-short", name: "unit-tests")` → `started: <job-id>` (name is an optional short label; use it so jobs are recognizable in `bgstatus`, the status widget, and wake messages) |
+| Start  | `bgrun(command: "make test-short", name: "unit-tests", type: "test")` → `started: <job-id>` (name is an optional short label; use it so jobs are recognizable in `bgstatus`, the status widget, and wake messages) |
 | Status | `bgstatus(<job-id>)` for one job, or `bgstatus()` for this session's running jobs — finished jobs are hidden by default; pass `includeDone: true` to list them |
-| Tail   | `bgtail(<job-id>, 40)` |
+| Tail   | `bgtail(<job-id>, 40)` — first read: last-40 tail; later reads: only lines appended since (delta tailing) |
+| Grep   | `bggrep(<job-id>, "pattern", context?)` — line-numbered matches, capped and condensed; default pattern = generic failure signatures (override when you know the format) |
 | Clean  | `bgclean()` for this session's old logs; `bgclean all` to sweep every session's (default 7-day retention) |
 
 ## Workflow
 
 1. **Start:** call `bgrun` with the command (and a short `name`, e.g. `name: "unit-tests"`).
-   Note the returned job-id. Continue other
+   When the project's digest config defines `type` entries, also pass the
+   matching `type` (e.g. `type: "test"`) — like `name`, it helps the wake
+   select the right digest scorecard. Note the returned job-id. Continue other
    work; you will be woken automatically when the job finishes.
 2. **On wake:** check the exit status in the wake message first.
    - `exit: 0` → success. `bgtail` to confirm.
@@ -47,28 +50,76 @@ no polling.
    - `done exit=<non-zero>` → failure; analyze the log.
    - `running` but the job should have finished long ago → likely crashed (the
      process died without writing the exit marker). Analyze the log with
-     `ctx_execute_file`.
+     `bggrep` (any jobs dir, last 2 MB) or `ctx_execute_file` on the absolute path
+     (whole file — needed for logs bigger than 2 MB).
 
 ### Reading results without flooding context
 
-- **Quick peek (≤40 lines):** call `bgtail` with the job id and `lines: 40` — strips the `__BGRUN_EXIT__` marker.
-- **Whole-log failure analysis:** `ctx_execute_file` on the log path:
-
-  ```
+If the wake message carries a `digest (<label>):` block (the label is the
+entry's `label`, its type, a matched `match.name`, or the preset id /
+`command`), read that
+first — it is a short pass/fail scorecard configured for this project and
+usually answers "what failed" without any follow-up read. `bgtail` stays the
+positional-peek tool for everything else.
+
+- **Quick peek (≤40 lines):** call `bgtail` with the job id and `lines: 40` — strips the `__BGRUN_EXIT__` marker. The first read returns the last-40 tail; repeat reads return only lines appended since your last read (delta tailing) — polling a running job is nearly free.
+- **Failure extraction:** `bggrep(<job-id>, "pattern")` — line-numbered matches with optional context lines, capped and condensed. Resolves the job id to the configured jobs dir itself — no path to reconstruct. (`ctx_execute_file` can read the same file given its absolute path.) Searches the last 2 MiB by default; `bytes: 67108864` widens it to the whole capped log — more scanning costs latency and memory, **not context**, since the returned matches stay capped. Pass your own pattern whenever you know the tool's output format; the default only catches common failure signatures.
+- **Whole-log failure analysis:** `ctx_execute_file` on the log's **absolute
+  path**. Unlike `bgtail`/`bggrep` (bounded to the last 2 MB), this reads the
+  whole file — the only way to cover a log bigger than 2 MB, e.g. one that hit
+  the size ceiling. Copy the `log:` path from `bgrun`'s `started:` line and
+  expand `~` yourself (it is not expanded for you; the tool takes an absolute
+  path or one relative to the project root). Otherwise it is an ordinary tool
+  call: your normal Read-deny rules still apply.
+
+  ```javascript
   ctx_execute_file(
-    path: "~/.pi-bgrun/jobs/<JOB>.log",
+    path: "/Users/me/project/.pi-bgrun/jobs/<JOB>.log",
     language: "javascript",
     code: "const L=FILE_CONTENT.split('\\n'); \
            const fails=L.filter(l=>/(--- FAIL|FAIL|panic:|Error:)/.test(l)); \
            console.log(`lines: ${L.length}, failures: ${fails.length}`); \
            console.log(fails.slice(0,40).join('\\n'));"
-  )
+  })
   ```
 
   A 10 000-line `make test` log collapses to a ~30-line summary in context.
 
-**Never `cat`, `Read`, or `bash cat` a full bgrun log.** Always `bgtail` or
-`ctx_execute_file`.
+**Why `bggrep` instead of `bash grep` on the log?**
+
+- `bash grep` output is uncapped — a retry-storm log can dump thousands of
+  matching lines (megabytes) straight into context, and staying safe depends
+  on remembering `| head` on every single call. `bggrep` is bounded by design
+  (last 2 MB of the log, per-line 10 000-char pre-truncation, ~50 matches,
+  ~8KB, plus a wall-clock match budget so a runaway regex errors instead of
+  hanging).
+- It takes the job id — no log-path reconstruction, no shell-quoting of the
+  regex, and no reliance on the agent getting `~` expansion right.
+- Output is self-describing: match count, line numbers, `…[N skipped]…` gap
+  markers, `— none` for no-match.
+
+Plain `grep` via bash is fine only for a one-off search you know is tiny.
+
+**Never `cat`, `Read`, `bash cat`, or `bash grep` a full bgrun log.** Always
+`bgtail`, `bggrep`, or (for project-local logs) `ctx_execute_file`.
+
+**Order of preference, cheapest first: `bgtail` → `bggrep` → `ctx_execute_file`.**
+Reach for the sandbox only when you need something a regex over lines cannot
+express — totals, dedup, grouping, joining the log against another file.
+
+`ctx_execute_file` is not itself a context dump: the file's bytes never enter
+context, only your script's **stdout** does ("raw content never leaves"). So the
+cost is exactly what you print — which makes `console.log(FILE_CONTENT)` (or
+`print(open(path).read())`, or a big unbounded slice) the one way a whole-log
+analysis turns into a context dump, and a capped-by-default 64 MiB log makes
+that expensive rather than merely rude. Aggregate, then cap what you print:
+
+- print counts / grouped summaries / the first N matches — not the content;
+- keep a `.slice(0, 40)` / `[:40]` on anything you echo;
+- for many different questions about one big log, index it once (`ctx_index`)
+  and `ctx_search` it, instead of re-scanning the file per call;
+- `bgtail` with a larger `lines`, or a tighter `bggrep` pattern, is usually the
+  cheaper answer to "I need to see more".
 
 ## After a pi restart or session switch
 
@@ -77,20 +128,45 @@ no polling.
 - After a restart/switch, run `bgstatus(<job-id>)` — the id still resolves via the
   log's `__BGRUN_EXIT__=N` marker. To browse everything on disk, use
   `bgstatus(includeDone: true)`.
-- Each session only tracks its own jobs by default. Jobs from other sessions
-  appear only when `adoptForeignJobs` is enabled in `~/.pi/agent/pi-bgrun.json`
-  (or `PI_BGRUN_FOREIGN_JOBS=1`).
+- Each session only tracks its own jobs by default. Running jobs from other
+  sessions appear only when `adoptForeignJobs` is enabled in
+  `~/.pi/agent/pi-bgrun.json` (or `PI_BGRUN_FOREIGN_JOBS=1`); finished foreign
+  logs appear with `bgstatus(includeDone: true)` regardless.
 
 ## Rules
 
 - Call the tools; never hand-roll `nohup … &` inline.
 - One job = one id. Multiple concurrent jobs are fine — each has its own log.
-- Logs live in `~/.pi-bgrun/jobs` (override with `PI_BGRUN_DIR`).
+- Job logs are capped by default (`maxLogBytes` / `PI_BGRUN_MAX_LOG_BYTES`,
+  64 MiB; `0` = unlimited) and the cap keeps the **first** bytes. A log that
+  ends with `__BGRUN_TRUNC__ output truncated: kept the first <N> bytes` (or
+  `__BGRUN_NOCAP__ log ceiling unavailable`, when the ceiling could not be
+  installed and the job ran uncapped) hit that ceiling: output past it was dropped, not lost to a failure — the job
+  still ran to completion with its real exit code, and readers (`bgtail`,
+  `bggrep`, the wake's line count/last line) filter the notice out. The wake's
+  Stats line, `bgtail` and `bggrep` all say when a log was capped (and report
+  `truncatedAtBytes` in their details), and a configured digest scorecard is
+  skipped rather than scored against an incomplete log — so on a capped job,
+  read a missing digest as "unknown", **not** as "no failures", and do not
+  re-run the command to see the missing tail; raise the ceiling if you need the
+  whole log. The flag lives in the exit marker (`__BGRUN_EXIT__=0
+  truncated=<N>`, or `nocap=1`), never in printable text — the `__BGRUN_*__`
+  lines are reserved, so a notice-looking line printed by the command itself is
+  content, not a signal. A search window is also limited to its last
+  500 000 lines: when that bites, `bgtail`/`bggrep` say so — a "none" from a
+  trimmed window means the head was not searched.
+- Logs default to `<project>/.pi-bgrun/jobs` in a repo — project-scoped is the
+  model (`~/.pi-bgrun/jobs` is a deprecated fallback for a cwd with no project
+  root; an absolute `PI_BGRUN_DIR`/`jobsDir` still works but is legacy). Project-local dirs are
+  auto-ignored via `.git/info/exclude`, which keeps `git status` clean; the
+  logs stay reachable for project-sandboxed analysis tools like
+  `ctx_execute_file` because they live inside the project.
 - Cleanup: `bgclean` removes only THIS session's old logs; `bgclean all`
   sweeps every session's. Auto-sweeps at session start/shutdown are
-  session-scoped plus a global orphan pass (default on — removes finished
-  week-old logs from crashed/abandoned sessions; disable with
-  `globalAutoClean: false` / `PI_BGRUN_GLOBAL_AUTO_CLEAN=0`). Retention is
-  `cleanupDays` (default 7, configurable).
-- To stop a running job, use `bash` with `kill <pid>` (the pid is in the `bgstatus`
-  output). There is no `bgkill` tool.
+  session-scoped plus an orphan pass (default on — removes finished week-old
+  logs from crashed/abandoned sessions; disable with `globalAutoClean: false`
+  / `PI_BGRUN_GLOBAL_AUTO_CLEAN=0`). Under the project-local default the orphan
+  pass covers the current project's jobs dir AND the machine-global
+  `~/.pi-bgrun/jobs`; an explicit absolute `jobsDir` is swept alone. Retention
+  is `cleanupDays` (default 7, configurable).
+- To stop a running job, use `bash` with `kill -- -<pid>` (process group — required because the child is spawned detached). The pid is the last `--`-separated segment of the job id; it is not shown as a separate field in `bgstatus` output. There is no `bgkill` tool.