@profoundry-us/highball 0.6.0 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +246 -0
- package/ONBOARDING.md +30 -4
- package/README.md +69 -1
- package/assets/dashboard.html +19 -4
- package/lib/config.js +48 -0
- package/lib/init.js +15 -1
- package/lib/journal.js +16 -1
- package/lib/judge.js +15 -1
- package/lib/mcp.js +14 -2
- package/lib/run.js +238 -80
- package/lib/runs.js +17 -3
- package/package.json +4 -3
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,246 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project are documented in this file.
|
|
4
|
+
|
|
5
|
+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
6
|
+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
|
+
Release dates are the day the version was published to npm.
|
|
8
|
+
|
|
9
|
+
## [Unreleased]
|
|
10
|
+
|
|
11
|
+
## [0.7.0] - 2026-09-10
|
|
12
|
+
|
|
13
|
+
### Added
|
|
14
|
+
|
|
15
|
+
- Per-rule timeouts. Every rule runs under a wall-clock budget: 8s for a
|
|
16
|
+
`fast` rule, 60s otherwise, 300s for an AI-judged rule. Raise a kind for the
|
|
17
|
+
whole repo with a top-level `timeouts:` block (`fast:`, `full:`, `judge:`),
|
|
18
|
+
or one rule with `timeout:`; both are seconds. A rule past its budget is
|
|
19
|
+
killed together with every process it started and fails with a message
|
|
20
|
+
naming the budget it hit. (#21)
|
|
21
|
+
- The journal records a run from the moment it starts and rewrites the record
|
|
22
|
+
after every rule, so a run that dies mid-way still shows how far it got. A
|
|
23
|
+
runner stopped by a signal (a hook timeout, a closed terminal, Ctrl-C) kills
|
|
24
|
+
the rule it was in, journals the run as `killed`, and exits 1. The CLI, the
|
|
25
|
+
MCP text and the dashboard widget label `running`, `killed` and `timed out`.
|
|
26
|
+
(#21)
|
|
27
|
+
- Publishing a tag now also creates its GitHub release, with notes generated
|
|
28
|
+
from the merged pull requests. 0.7.0 is the first release with a page. (#22)
|
|
29
|
+
- This changelog.
|
|
30
|
+
|
|
31
|
+
### Changed
|
|
32
|
+
|
|
33
|
+
- `init` writes a 120-second timeout on the fast hook it scaffolds; Claude
|
|
34
|
+
Code's default was ten minutes. (#21)
|
|
35
|
+
- The check loop is asynchronous, which is what lets the runner enforce its
|
|
36
|
+
own budgets and respond to signals. Rule output is still captured the same
|
|
37
|
+
way; the AI judge keeps its synchronous spawn and receives the budget as a
|
|
38
|
+
timeout. (#21)
|
|
39
|
+
- The onboarding guide tells the agent to install with the package manager
|
|
40
|
+
the repo's lockfile names (yarn, pnpm or npm), and what to do when yarn
|
|
41
|
+
refuses on the repo's own `engines` field. The README lists all three
|
|
42
|
+
install commands. (#18)
|
|
43
|
+
- Highball's own fast rule is scoped to changed files: it parses every changed
|
|
44
|
+
JavaScript file and runs the changed modules' unit tests, leaving the suites
|
|
45
|
+
that spawn the CLI to turn end. The README states the targets the budgets
|
|
46
|
+
serve: fast lane under 2s, full suite under 30s. (#21)
|
|
47
|
+
|
|
48
|
+
### Fixed
|
|
49
|
+
|
|
50
|
+
- The test suite no longer reports fixture runs to the PostHog project named in
|
|
51
|
+
the developer's environment. (#21)
|
|
52
|
+
|
|
53
|
+
**Behaviour change for consuming repos:** a `fast: true` rule that takes
|
|
54
|
+
longer than 8 seconds now fails as timed out. Give it its own `timeout:`, or
|
|
55
|
+
scope it to `HIGHBALL_CHANGED_FILES`.
|
|
56
|
+
|
|
57
|
+
## [0.6.1] - 2026-09-08
|
|
58
|
+
|
|
59
|
+
### Changed
|
|
60
|
+
|
|
61
|
+
- `package.json` carries the homepage, https://highball.profoundry.us, so the
|
|
62
|
+
npm page links to the site. The README links there too. (#16)
|
|
63
|
+
|
|
64
|
+
## [0.6.0] - 2026-09-04
|
|
65
|
+
|
|
66
|
+
### Added
|
|
67
|
+
|
|
68
|
+
- Three ways to switch a repo's checks off, none of them silent: `enabled:
|
|
69
|
+
false` in `checks.yml` for everyone who clones the repo; `HIGHBALL_DISABLED`
|
|
70
|
+
for one machine; and a `.highball/disabled` marker file for one checkout,
|
|
71
|
+
gitignored by the `.highball/.gitignore` that `init` now writes and re-read
|
|
72
|
+
on every run, so it toggles inside a live agent session with no restart.
|
|
73
|
+
Text in the marker comes back as the reason on every run. (#10, #13)
|
|
74
|
+
- Continuous integration runs the test suite on pull requests and pushes to
|
|
75
|
+
`main`, on Node 18 and 22. (#11)
|
|
76
|
+
|
|
77
|
+
### Changed
|
|
78
|
+
|
|
79
|
+
- `--if-changed` stamps are per checkout rather than per project, so a git
|
|
80
|
+
worktree or a second clone never skips a run because the other one passed.
|
|
81
|
+
(#9)
|
|
82
|
+
- `enabled:` accepts only a real boolean; `enabled: no` and `enabled: "false"`
|
|
83
|
+
are errors rather than checks quietly left on. `HIGHBALL_DISABLED=0`,
|
|
84
|
+
`false`, `no`, `off` and empty mean not disabled. (#10)
|
|
85
|
+
- The README says when each off switch takes effect, and documents keeping the
|
|
86
|
+
PostHog key in the environment instead of the repo. (#8, #12)
|
|
87
|
+
|
|
88
|
+
## [0.5.0] - 2026-09-03
|
|
89
|
+
|
|
90
|
+
### Added
|
|
91
|
+
|
|
92
|
+
- Runs report to PostHog: `reporting.posthog` in `checks.yml` (or
|
|
93
|
+
`HIGHBALL_POSTHOG_KEY` in the environment) sends one `highball_run` event
|
|
94
|
+
per run and one `highball_check` event per rule, in a single batch after
|
|
95
|
+
the checks finish. The project key is write-only by design, so it is
|
|
96
|
+
committed config with no login step. Dashboard queries ship in
|
|
97
|
+
`docs/posthog-queries.sql`. (#4)
|
|
98
|
+
- The fast hook `init` scaffolds matches `Bash` as well as `Write|Edit`, since
|
|
99
|
+
agents in auto mode edit through the shell, and runs with `--if-changed`,
|
|
100
|
+
which fingerprints the working tree and exits at once when nothing moved
|
|
101
|
+
since the last run. (#7)
|
|
102
|
+
- The MCP server resolves the current project from its `project` or `dir`
|
|
103
|
+
argument, a `checks.yml` at its working directory, or the client's roots,
|
|
104
|
+
and never falls back to another repo's journal. When none names a repo, the
|
|
105
|
+
widget offers the journaled projects as a picker. (#7)
|
|
106
|
+
- A regression test for the stdin deadline, timing the runner process rather
|
|
107
|
+
than the pipeline it sits in. (#3)
|
|
108
|
+
|
|
109
|
+
### Removed
|
|
110
|
+
|
|
111
|
+
- The hosted dashboard sink and everything that served it: `highball login`,
|
|
112
|
+
`~/.highball/credentials.json`, `reporting.url`, `HIGHBALL_URL` and
|
|
113
|
+
`HIGHBALL_TOKEN`. Enforcement was always local; the witness half is now a
|
|
114
|
+
PostHog project the team owns. A `checks.yml` still carrying
|
|
115
|
+
`reporting.url` prints a warning on every run and reports nowhere. (#5)
|
|
116
|
+
|
|
117
|
+
### Changed
|
|
118
|
+
|
|
119
|
+
- Package metadata points at the renamed repository, profoundry-us/highball.
|
|
120
|
+
(#6)
|
|
121
|
+
|
|
122
|
+
## [0.4.1] - 2026-08-19
|
|
123
|
+
|
|
124
|
+
### Fixed
|
|
125
|
+
|
|
126
|
+
- `zod` is a declared dependency. `lib/mcp.js` imported it while it was only
|
|
127
|
+
present by hoisting, so `highball mcp` failed to start under pnpm, Yarn PnP
|
|
128
|
+
and nohoist layouts. (#2)
|
|
129
|
+
|
|
130
|
+
## [0.4.0] - 2026-08-15
|
|
131
|
+
|
|
132
|
+
### Added
|
|
133
|
+
|
|
134
|
+
- AI-judged rules are a first-class rule type. A rule declares `rubric:`
|
|
135
|
+
instead of `run:`; the runner bundles the changed files the rubric's front
|
|
136
|
+
matter selects, applies it through headless Claude, and returns the same
|
|
137
|
+
pass/fail contract as any other rule. Rubric rules never join a `--fast`
|
|
138
|
+
run, never pass through `exec.via`, and never spawn the model when no
|
|
139
|
+
changed file matches. Packs keep the rubrics; the engine lives here. (#1)
|
|
140
|
+
|
|
141
|
+
## [0.3.2] - 2026-08-15
|
|
142
|
+
|
|
143
|
+
### Fixed
|
|
144
|
+
|
|
145
|
+
- The runner no longer hangs when stdin is open but silent. It read hook
|
|
146
|
+
payloads to EOF, which never returned when invoked from a pipeline that
|
|
147
|
+
neither wrote nor closed; the read now races a 400ms deadline and degrades
|
|
148
|
+
to a run with no session context.
|
|
149
|
+
- `bin/highball.js` is marked executable in git, so directory installs work
|
|
150
|
+
as well as tarball installs.
|
|
151
|
+
|
|
152
|
+
## [0.3.1] - 2026-08-15
|
|
153
|
+
|
|
154
|
+
### Changed
|
|
155
|
+
|
|
156
|
+
- `init` scaffolds the scoped command, `npx @profoundry-us/highball`, into
|
|
157
|
+
`.claude/settings.json`. The unscoped name belongs to an unrelated package,
|
|
158
|
+
and a committed hook is the wrong place to leave that ambiguity. Every
|
|
159
|
+
example in the docs is scoped as well, including the MCP registration.
|
|
160
|
+
- The onboarding guide's container guidance is its own section, with the
|
|
161
|
+
probes to run and the four traps that had each cost a real onboarding:
|
|
162
|
+
`-T`, `--workdir`, self-orchestrating targets, and host-only tools.
|
|
163
|
+
|
|
164
|
+
## [0.3.0] - 2026-08-14
|
|
165
|
+
|
|
166
|
+
### Added
|
|
167
|
+
|
|
168
|
+
- `highball mcp`: an MCP server over stdio exposing `list_runs`, `get_run` and
|
|
169
|
+
`run_checks`, each carrying an MCP Apps dashboard widget. Hosts that render
|
|
170
|
+
Apps get an interactive run list with click-through detail, expandable
|
|
171
|
+
per-rule output, and re-run buttons; every other host gets the same picture
|
|
172
|
+
as aligned plain text, decided by the client's declared capabilities.
|
|
173
|
+
- A widget development harness, `npm run harness`, serving the real dashboard
|
|
174
|
+
against live journal data.
|
|
175
|
+
- Every list view shows each run's total duration, and runs are grouped under
|
|
176
|
+
the agent prompt that produced them, read from the hook's transcript.
|
|
177
|
+
- The journal records each rule's command, so quiet rules still have
|
|
178
|
+
something to show; rows with nothing to reveal are no longer disclosures.
|
|
179
|
+
- Publishing from a pushed `v*` tag through npm trusted publishing (OIDC),
|
|
180
|
+
gated on the tag matching `package.json`, the tests passing, and the
|
|
181
|
+
tarball carrying the widget.
|
|
182
|
+
|
|
183
|
+
## [0.2.2] - 2026-08-11
|
|
184
|
+
|
|
185
|
+
Not published to npm.
|
|
186
|
+
|
|
187
|
+
### Changed
|
|
188
|
+
|
|
189
|
+
- The run detail view uses the same column formatter as the list.
|
|
190
|
+
|
|
191
|
+
## [0.2.1] - 2026-08-11
|
|
192
|
+
|
|
193
|
+
Not published to npm.
|
|
194
|
+
|
|
195
|
+
### Added
|
|
196
|
+
|
|
197
|
+
- `highball runs <n> --logs` prints every rule's captured output; the journal
|
|
198
|
+
keeps the last 10KB per rule, pass or fail.
|
|
199
|
+
|
|
200
|
+
### Changed
|
|
201
|
+
|
|
202
|
+
- `highball runs` output is colored when writing to a terminal (`NO_COLOR`
|
|
203
|
+
respected, `FORCE_COLOR` overrides), with status columns right-aligned.
|
|
204
|
+
|
|
205
|
+
## [0.2.0] - 2026-08-11
|
|
206
|
+
|
|
207
|
+
Not published to npm.
|
|
208
|
+
|
|
209
|
+
### Added
|
|
210
|
+
|
|
211
|
+
- A local run journal at `~/.highball/runs/<project>.jsonl`, written on every
|
|
212
|
+
run whether or not reporting is configured and pruned to the last 200 runs,
|
|
213
|
+
and `highball runs` to list recent runs and show one run's detail.
|
|
214
|
+
|
|
215
|
+
## [0.1.0] - 2026-08-11
|
|
216
|
+
|
|
217
|
+
### Added
|
|
218
|
+
|
|
219
|
+
- The runner: `highball run [--fast]` executes the rules in
|
|
220
|
+
`.highball/checks.yml`, prints progress, and exits 2 with the failures on
|
|
221
|
+
stderr, the contract a Claude Code Stop hook reads as "block the agent and
|
|
222
|
+
hand it the output". Rules marked `fast: true` run on every edit; `todo:
|
|
223
|
+
true` declares a rule that is tracked but never fails.
|
|
224
|
+
- The execution context: `exec.via` wraps every rule in a declared command
|
|
225
|
+
(a container, say) and a rule opts out with `exec: host`. The runner owns
|
|
226
|
+
git and hands every rule the changed-file list in `HIGHBALL_CHANGED_FILES`.
|
|
227
|
+
- `highball init` scaffolds `checks.yml` and the Claude Code hooks, never
|
|
228
|
+
overwriting what exists.
|
|
229
|
+
- `highball onboard` prints a setup guide written for the repo's own AI agent.
|
|
230
|
+
- `highball login`, for the hosted dashboard of the time.
|
|
231
|
+
- MIT license and registry metadata.
|
|
232
|
+
|
|
233
|
+
[Unreleased]: https://github.com/profoundry-us/highball/compare/v0.7.0...HEAD
|
|
234
|
+
[0.7.0]: https://github.com/profoundry-us/highball/compare/v0.6.1...v0.7.0
|
|
235
|
+
[0.6.1]: https://github.com/profoundry-us/highball/compare/v0.6.0...v0.6.1
|
|
236
|
+
[0.6.0]: https://github.com/profoundry-us/highball/compare/v0.5.0...v0.6.0
|
|
237
|
+
[0.5.0]: https://github.com/profoundry-us/highball/compare/v0.4.1...v0.5.0
|
|
238
|
+
[0.4.1]: https://github.com/profoundry-us/highball/compare/v0.4.0...v0.4.1
|
|
239
|
+
[0.4.0]: https://github.com/profoundry-us/highball/compare/de0d5a8...v0.4.0
|
|
240
|
+
[0.3.2]: https://github.com/profoundry-us/highball/compare/aa148c1...de0d5a8
|
|
241
|
+
[0.3.1]: https://github.com/profoundry-us/highball/compare/e0e54d4...aa148c1
|
|
242
|
+
[0.3.0]: https://github.com/profoundry-us/highball/compare/292dbbd...e0e54d4
|
|
243
|
+
[0.2.2]: https://github.com/profoundry-us/highball/compare/99d20c1...292dbbd
|
|
244
|
+
[0.2.1]: https://github.com/profoundry-us/highball/compare/e123bc7...99d20c1
|
|
245
|
+
[0.2.0]: https://github.com/profoundry-us/highball/compare/91c6b6d...e123bc7
|
|
246
|
+
[0.1.0]: https://github.com/profoundry-us/highball/commits/91c6b6d
|
package/ONBOARDING.md
CHANGED
|
@@ -16,10 +16,27 @@ yourself.
|
|
|
16
16
|
|
|
17
17
|
`npx @profoundry-us/highball --help` must work (Node >= 18, package installed as a dev
|
|
18
18
|
dependency). If it doesn't, ask your human whether to install from the npm
|
|
19
|
-
registry
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
19
|
+
registry or from a local tarball path they provide. Install with the package
|
|
20
|
+
manager the repo already uses — the lockfile tells you which, and mixing
|
|
21
|
+
managers leaves two lockfiles that disagree:
|
|
22
|
+
|
|
23
|
+
| Lockfile present | Install command |
|
|
24
|
+
|---|---|
|
|
25
|
+
| `yarn.lock` | `yarn add -D @profoundry-us/highball` |
|
|
26
|
+
| `pnpm-lock.yaml` | `pnpm add -D @profoundry-us/highball` |
|
|
27
|
+
| `package-lock.json`, or none | `npm install --save-dev @profoundry-us/highball` |
|
|
28
|
+
|
|
29
|
+
If the install stops with `The engine "node" is incompatible with this
|
|
30
|
+
module`, the repo pins a Node version in its own `engines` field and you are
|
|
31
|
+
running a different one (yarn treats that as a hard error; npm only warns).
|
|
32
|
+
Switch to the pinned version with whatever the repo already uses (`.nvmrc`,
|
|
33
|
+
`.node-version`, `.tool-versions`, `volta`) and retry. Don't pass
|
|
34
|
+
`--ignore-engines` and don't edit the `engines` field — if no version manager
|
|
35
|
+
is set up, ask your human.
|
|
36
|
+
|
|
37
|
+
In a repo with no `package.json`, create a minimal private one first
|
|
38
|
+
(`{ "name": "<repo>", "private": true }`) and gitignore `node_modules/` if it
|
|
39
|
+
isn't already.
|
|
23
40
|
|
|
24
41
|
**Never run bare `npx highball` where the package is NOT installed**: the
|
|
25
42
|
unscoped npm name `highball` belongs to an unrelated package, and npx
|
|
@@ -127,6 +144,7 @@ checks:
|
|
|
127
144
|
name: Unit tests (offline)
|
|
128
145
|
run: just test-unit # orchestrates its own docker exec
|
|
129
146
|
exec: host # → must run on the host
|
|
147
|
+
timeout: 300 # seconds; the default for a full rule is 60
|
|
130
148
|
|
|
131
149
|
- id: coverage-ratchet
|
|
132
150
|
name: Coverage never decreases
|
|
@@ -145,6 +163,14 @@ Decision rules:
|
|
|
145
163
|
- The runner exports `HIGHBALL_CHANGED_FILES` (newline-separated,
|
|
146
164
|
repo-relative) to every rule — scripts that want changed-only behavior
|
|
147
165
|
can read it instead of shelling out to git.
|
|
166
|
+
- Every rule runs under a budget and is killed past it: 8s for a `fast`
|
|
167
|
+
rule, 60s otherwise, 300s for an AI-judged one. Time the repo's test
|
|
168
|
+
command before you write its rule, and give it an honest `timeout:`
|
|
169
|
+
(seconds) if it needs more — a suite marked `fast` needs one for sure.
|
|
170
|
+
Raise a whole kind with a top-level `timeouts:` block (`fast:`,
|
|
171
|
+
`full:`, `judge:`). A rule that times out fails and says which budget
|
|
172
|
+
it hit; do not answer that by raising the number without asking your
|
|
173
|
+
human whether the rule hung.
|
|
148
174
|
|
|
149
175
|
## 4. Telemetry — **human**, and optional
|
|
150
176
|
|
package/README.md
CHANGED
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
# @profoundry-us/highball
|
|
2
2
|
|
|
3
|
+
**Website:** [highball.profoundry.us](https://highball.profoundry.us) · [Getting started](https://highball.profoundry.us/#getting-started)
|
|
4
|
+
|
|
3
5
|
The Highball runner: executes a repo's `.highball/checks.yml` rules and blocks
|
|
4
6
|
AI coding agents on failure (exit 2, the Claude Code hook contract) — "local
|
|
5
7
|
CI for AI agents". Enforcement is entirely local and needs no account, no
|
|
@@ -14,7 +16,14 @@ endpoint means checks still run and still block, they just aren't recorded.
|
|
|
14
16
|
|
|
15
17
|
## Install
|
|
16
18
|
|
|
17
|
-
Published releases
|
|
19
|
+
Published releases, with whichever package manager the repo's lockfile says
|
|
20
|
+
it uses:
|
|
21
|
+
|
|
22
|
+
```bash
|
|
23
|
+
yarn add -D @profoundry-us/highball
|
|
24
|
+
npm install --save-dev @profoundry-us/highball
|
|
25
|
+
pnpm add -D @profoundry-us/highball
|
|
26
|
+
```
|
|
18
27
|
|
|
19
28
|
**Always use the scoped name.** The unscoped npm name `highball` belongs to
|
|
20
29
|
an unrelated package, so a bare `npx highball` — in a committed hook, a
|
|
@@ -35,6 +44,9 @@ installing the package, tell the repo's Claude Code agent:
|
|
|
35
44
|
|
|
36
45
|
> Run `npx @profoundry-us/highball onboard` and follow the instructions.
|
|
37
46
|
|
|
47
|
+
The site's [Getting started](https://highball.profoundry.us/#getting-started)
|
|
48
|
+
section shows this route and the manual one side by side.
|
|
49
|
+
|
|
38
50
|
[ONBOARDING.md](ONBOARDING.md) (which that command prints) walks the agent
|
|
39
51
|
through surveying the repo's real toolchain, scaffolding, writing rules that
|
|
40
52
|
reflect what the repo already trusts, and verifying all four proofs —
|
|
@@ -79,10 +91,16 @@ reporting:
|
|
|
79
91
|
exec:
|
|
80
92
|
via: docker compose exec -T app
|
|
81
93
|
|
|
94
|
+
# Wall-clock budgets in seconds; a rule past its budget is killed and
|
|
95
|
+
# fails as "timed out". Defaults: fast 8, full 60, judge 300.
|
|
96
|
+
timeouts:
|
|
97
|
+
full: 120
|
|
98
|
+
|
|
82
99
|
checks:
|
|
83
100
|
- id: unit-tests
|
|
84
101
|
name: Unit tests
|
|
85
102
|
run: bundle exec rspec spec # runs through exec.via
|
|
103
|
+
timeout: 300 # this rule's own budget
|
|
86
104
|
|
|
87
105
|
- id: js-syntax
|
|
88
106
|
name: Playback JS parses
|
|
@@ -103,6 +121,46 @@ The runner computes the branch's changed-file list once (it owns git) and
|
|
|
103
121
|
hands it to every rule via `HIGHBALL_CHANGED_FILES` — check scripts stay pure
|
|
104
122
|
analyzers and need no git in their execution context.
|
|
105
123
|
|
|
124
|
+
## Timeouts
|
|
125
|
+
|
|
126
|
+
Every rule runs under a wall-clock budget. A rule that runs past it is
|
|
127
|
+
killed — the whole process tree, so a hung test runner started by `npm`
|
|
128
|
+
started by a shell doesn't live on after the run has given up on it — and
|
|
129
|
+
fails with the reason:
|
|
130
|
+
|
|
131
|
+
```
|
|
132
|
+
→ Unit tests ... TIMED OUT (8.0s)
|
|
133
|
+
|
|
134
|
+
### Unit tests (unit-tests) failed. Fix before finishing:
|
|
135
|
+
…
|
|
136
|
+
highball: timed out after 8s and was killed. Its budget is `timeouts.fast: 8`
|
|
137
|
+
in .highball/checks.yml — raise it there if this rule legitimately needs
|
|
138
|
+
longer, or find what hung.
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
The defaults are deliberately tight: **8s** for a rule marked `fast: true`,
|
|
142
|
+
**60s** for everything else, **300s** for an AI-judged rule. A fast rule runs
|
|
143
|
+
after every agent edit, and a rule worth that is one that finishes before the
|
|
144
|
+
agent's next thought — 8s is generous for a lint or a grep and far too short
|
|
145
|
+
for a suite, which is the point: a suite marked fast has to earn it with its
|
|
146
|
+
own `timeout:`. Raise a kind for the whole repo with `timeouts:`, or give
|
|
147
|
+
one rule its own `timeout:`; both are seconds. Either way the number is a
|
|
148
|
+
decision in `checks.yml`, not a mystery.
|
|
149
|
+
|
|
150
|
+
The budgets are kill switches, not targets. The targets, for a small project:
|
|
151
|
+
the fast lane under **2s**, the full suite under **30s**. A fast rule should
|
|
152
|
+
look only at what changed — the runner hands every rule the changed-file
|
|
153
|
+
list in `HIGHBALL_CHANGED_FILES`, so "parse the changed files and run the
|
|
154
|
+
changed modules' own tests" is a short script — and a full suite that can't
|
|
155
|
+
make 30s should scope itself the same way rather than run long. Highball's
|
|
156
|
+
own `.highball/checks/tests-for-changed` is the worked example.
|
|
157
|
+
|
|
158
|
+
The budget is what turns a hang into evidence. Without one, a rule that never
|
|
159
|
+
returns holds the hook open until Claude Code kills it, and (before 0.7) the
|
|
160
|
+
run was journaled nowhere — "the tests hang sometimes" had nothing to point
|
|
161
|
+
at. The hook timeouts `init` writes are backstops for the runner itself, not
|
|
162
|
+
budgets for rules: they should only ever fire if the runner is wedged.
|
|
163
|
+
|
|
106
164
|
## Turning it off
|
|
107
165
|
|
|
108
166
|
Three switches, differing in who they affect and — the part that usually
|
|
@@ -316,6 +374,16 @@ pass or fail, while PostHog gets a one-line summary and no logs at all. So the
|
|
|
316
374
|
runner is fully self-sufficient with no `reporting:` block — PostHog adds
|
|
317
375
|
cross-developer trends, not visibility you'd otherwise lack.
|
|
318
376
|
|
|
377
|
+
A run is journaled from the moment it starts, and rewritten after every rule,
|
|
378
|
+
so the record is always as current as the run. A run that ends the ordinary
|
|
379
|
+
way is `passed` or `failed`. One still marked `running` is either live
|
|
380
|
+
right now or died without getting to say so — a crash, a `kill -9`. One
|
|
381
|
+
marked `killed` was stopped in the middle of a rule (Claude Code giving up
|
|
382
|
+
on the hook, a closed terminal, Ctrl-C); the runner took the rule's process
|
|
383
|
+
tree down with it and journaled which rule it was in. A rule that ran past
|
|
384
|
+
its budget is `failed` with `timed_out`. None of these used to leave a
|
|
385
|
+
trace, which is exactly when a trace was needed.
|
|
386
|
+
|
|
319
387
|
## Roadmap
|
|
320
388
|
|
|
321
389
|
Built-in generic rules (spec pairing, focused-spec detection, diff budgets)
|
package/assets/dashboard.html
CHANGED
|
@@ -52,6 +52,8 @@
|
|
|
52
52
|
.chip.passed { color: var(--green); background: var(--green-bg); }
|
|
53
53
|
.chip.failed { color: var(--red); background: var(--red-bg); }
|
|
54
54
|
.chip.todo { color: var(--muted); background: var(--gray-bg); }
|
|
55
|
+
.chip.running { color: var(--muted); background: var(--gray-bg); }
|
|
56
|
+
.chip.killed { color: var(--red); background: var(--red-bg); }
|
|
55
57
|
.mono { font-family: ui-monospace, Menlo, monospace; font-size: 12px; }
|
|
56
58
|
details.rule {
|
|
57
59
|
border: 1px solid var(--line); border-radius: 8px; background: var(--card);
|
|
@@ -158,8 +160,21 @@
|
|
|
158
160
|
if (s < 86400) return `${Math.round(s / 3600)}h ago`;
|
|
159
161
|
return `${Math.round(s / 86400)}d ago`;
|
|
160
162
|
}
|
|
161
|
-
function chip(status) {
|
|
162
|
-
return `<span class="chip ${esc(status)}">${esc(
|
|
163
|
+
function chip(status, label = status) {
|
|
164
|
+
return `<span class="chip ${esc(status)}">${esc(label)}</span>`;
|
|
165
|
+
}
|
|
166
|
+
// "running" is a run the journal saw start and not finish — live right
|
|
167
|
+
// now, or one that died without saying so. "killed" is one the runner
|
|
168
|
+
// was stopped in the middle of.
|
|
169
|
+
function runLabel(status) {
|
|
170
|
+
return status === "passed" ? "✓ passed"
|
|
171
|
+
: status === "running" ? "… running"
|
|
172
|
+
: status === "killed" ? "✗ killed"
|
|
173
|
+
: "✗ failed";
|
|
174
|
+
}
|
|
175
|
+
// A failure the runner caused, rather than the rule, says so.
|
|
176
|
+
function ruleLabel(result) {
|
|
177
|
+
return result.timed_out ? "timed out" : result.killed ? "killed" : result.status;
|
|
163
178
|
}
|
|
164
179
|
function tally(results) {
|
|
165
180
|
const counts = {};
|
|
@@ -217,7 +232,7 @@
|
|
|
217
232
|
lastGroup = group;
|
|
218
233
|
return heading + `
|
|
219
234
|
<div class="row" data-index="${run.index}">
|
|
220
|
-
<span class="chip ${esc(run.status)}">${run.status
|
|
235
|
+
<span class="chip ${esc(run.status)}">${runLabel(run.status)}</span>
|
|
221
236
|
<span class="when">${timeAgo(run.started_at)}</span>
|
|
222
237
|
<span class="meta">${run.trigger === "edit" ? "fast" : "full"}</span>
|
|
223
238
|
<span class="mono">${esc(run.branch || "-")}</span>
|
|
@@ -252,7 +267,7 @@
|
|
|
252
267
|
`<span class="glyph ${esc(result.status)}">${result.status === "passed" ? "✓" : result.status === "todo" ? "•" : "✗"}</span>` +
|
|
253
268
|
`<span>${esc(result.name)}</span>` +
|
|
254
269
|
`<span class="spacer"></span>` +
|
|
255
|
-
chip(result.status) +
|
|
270
|
+
chip(result.status, ruleLabel(result)) +
|
|
256
271
|
`<span class="meta">${duration}</span>`;
|
|
257
272
|
|
|
258
273
|
// What the panel can actually show. Quiet rules (the AI judges print
|
package/lib/config.js
CHANGED
|
@@ -37,6 +37,23 @@ export function loadConfig(root = process.cwd()) {
|
|
|
37
37
|
"`run:`, `rubric:`, or `todo: true`."
|
|
38
38
|
);
|
|
39
39
|
}
|
|
40
|
+
if (rule.timeout !== undefined) {
|
|
41
|
+
assertSeconds(rule.timeout, `rule \`${rule.id ?? "(unnamed)"}\` has \`timeout:\``);
|
|
42
|
+
}
|
|
43
|
+
}
|
|
44
|
+
if (config.timeouts !== undefined) {
|
|
45
|
+
if (typeof config.timeouts !== "object" || config.timeouts === null) {
|
|
46
|
+
throw new Error(`${CONFIG_PATH}: \`timeouts:\` must be a block of fast/full/judge seconds.`);
|
|
47
|
+
}
|
|
48
|
+
for (const [ kind, value ] of Object.entries(config.timeouts)) {
|
|
49
|
+
if (!(kind in DEFAULT_TIMEOUTS)) {
|
|
50
|
+
throw new Error(
|
|
51
|
+
`${CONFIG_PATH}: \`timeouts.${kind}\` is not a thing — ` +
|
|
52
|
+
`the kinds are ${Object.keys(DEFAULT_TIMEOUTS).join(", ")}.`
|
|
53
|
+
);
|
|
54
|
+
}
|
|
55
|
+
assertSeconds(value, `\`timeouts.${kind}:\``);
|
|
56
|
+
}
|
|
40
57
|
}
|
|
41
58
|
// The dashboard sink is gone; PostHog is the only telemetry path. A repo
|
|
42
59
|
// carrying the old block would otherwise report nowhere and say nothing
|
|
@@ -115,6 +132,37 @@ export function resolvePosthog(config, env = process.env) {
|
|
|
115
132
|
return { host, key };
|
|
116
133
|
}
|
|
117
134
|
|
|
135
|
+
// Wall-clock budgets per rule, in seconds. A fast rule is one worth running
|
|
136
|
+
// after every edit, and a rule worth that is one that finishes before the
|
|
137
|
+
// agent's next thought — 8s is generous for a lint or a grep and far too
|
|
138
|
+
// short for a suite, which is the point: a suite marked fast has to earn
|
|
139
|
+
// it with its own `timeout:`. Full rules get a minute by default; teams
|
|
140
|
+
// with a long suite raise `timeouts.full` and own the number. AI judges
|
|
141
|
+
// wait on a model, so they get their own, longer default.
|
|
142
|
+
//
|
|
143
|
+
// The budget is what turns a hang into evidence. Without one, a rule that
|
|
144
|
+
// never returns holds the hook until Claude Code kills it, and the run is
|
|
145
|
+
// journaled nowhere.
|
|
146
|
+
export const DEFAULT_TIMEOUTS = { fast: 8, full: 60, judge: 300 };
|
|
147
|
+
|
|
148
|
+
function assertSeconds(value, where) {
|
|
149
|
+
if (typeof value !== "number" || !Number.isFinite(value) || value <= 0) {
|
|
150
|
+
throw new Error(
|
|
151
|
+
`${CONFIG_PATH}: ${where} must be a positive number of seconds, not ` +
|
|
152
|
+
`\`${value}\` (${typeof value}).`
|
|
153
|
+
);
|
|
154
|
+
}
|
|
155
|
+
}
|
|
156
|
+
|
|
157
|
+
// Milliseconds a rule may run before the runner kills it: the rule's own
|
|
158
|
+
// `timeout:` if it has one, else the default for its kind.
|
|
159
|
+
export function timeoutFor(rule, config = {}) {
|
|
160
|
+
const timeouts = { ...DEFAULT_TIMEOUTS, ...(config.timeouts ?? {}) };
|
|
161
|
+
const seconds = rule.timeout ??
|
|
162
|
+
(rule.rubric ? timeouts.judge : rule.fast ? timeouts.fast : timeouts.full);
|
|
163
|
+
return seconds * 1000;
|
|
164
|
+
}
|
|
165
|
+
|
|
118
166
|
// The execution-context decision (ADR 202608): rule definitions stay
|
|
119
167
|
// environment-agnostic; the checkout declares `exec.via` once and every
|
|
120
168
|
// rule runs through it unless it opts out with `exec: host`. No declared
|
package/lib/init.js
CHANGED
|
@@ -28,6 +28,13 @@ project: ${project}
|
|
|
28
28
|
# host: https://us.i.posthog.com
|
|
29
29
|
# project_key: phc_your_key
|
|
30
30
|
|
|
31
|
+
# Wall-clock budgets, in seconds. A rule past its budget is killed along
|
|
32
|
+
# with everything it started, fails with "timed out", and is journaled
|
|
33
|
+
# that way. Defaults: fast 8, full 60, judge 300. Raise a kind here, or
|
|
34
|
+
# give one rule its own \`timeout:\` below.
|
|
35
|
+
# timeouts:
|
|
36
|
+
# full: 120
|
|
37
|
+
|
|
31
38
|
# If this repo's toolchain lives in a container, declare the wrapper once
|
|
32
39
|
# and every rule runs through it; rules that belong on the host opt out
|
|
33
40
|
# with \`exec: host\`. Omit entirely for host-based setups.
|
|
@@ -38,6 +45,7 @@ checks:
|
|
|
38
45
|
# - id: unit-tests
|
|
39
46
|
# name: Unit tests
|
|
40
47
|
# run: npm test
|
|
48
|
+
# timeout: 300
|
|
41
49
|
|
|
42
50
|
# - id: lint
|
|
43
51
|
# name: Lint & formatting
|
|
@@ -52,6 +60,11 @@ checks:
|
|
|
52
60
|
// The fast hook matches Bash as well as the edit tools: agents in auto mode
|
|
53
61
|
// edit through Bash, and a hook on Write|Edit alone never fires for them.
|
|
54
62
|
// --if-changed keeps the Bash firings free when the tree hasn't moved.
|
|
63
|
+
//
|
|
64
|
+
// The hook timeouts are backstops, not budgets: the runner times out each
|
|
65
|
+
// rule itself (`timeouts:` in checks.yml) and journals the result, so the
|
|
66
|
+
// hook's own limit should only ever fire if the runner is somehow wedged.
|
|
67
|
+
// Without one, Claude Code's default for a fast hook is ten minutes.
|
|
55
68
|
const HOOKS_JSON = {
|
|
56
69
|
hooks: {
|
|
57
70
|
PostToolUse: [
|
|
@@ -59,7 +72,8 @@ const HOOKS_JSON = {
|
|
|
59
72
|
matcher: "Write|Edit|Bash",
|
|
60
73
|
hooks: [{
|
|
61
74
|
type: "command",
|
|
62
|
-
command: "npx @profoundry-us/highball run --fast --if-changed"
|
|
75
|
+
command: "npx @profoundry-us/highball run --fast --if-changed",
|
|
76
|
+
timeout: 120
|
|
63
77
|
}]
|
|
64
78
|
}
|
|
65
79
|
],
|
package/lib/journal.js
CHANGED
|
@@ -21,13 +21,28 @@ export function journalPath(project, dir = journalDir()) {
|
|
|
21
21
|
return join(dir, `${project}.jsonl`);
|
|
22
22
|
}
|
|
23
23
|
|
|
24
|
+
// A record carrying an `id` replaces its earlier self rather than adding a
|
|
25
|
+
// line: the runner writes a run when it starts and again after every rule,
|
|
26
|
+
// so a run cut off mid-way still shows how far it got — one record per run
|
|
27
|
+
// either way. Records without an id (older runners, tests) simply append.
|
|
24
28
|
export function appendRun(project, record, dir = journalDir()) {
|
|
25
29
|
mkdirSync(dir, { recursive: true, mode: 0o700 });
|
|
26
30
|
const path = journalPath(project, dir);
|
|
27
31
|
const lines = existsSync(path)
|
|
28
32
|
? readFileSync(path, "utf8").split("\n").filter(Boolean)
|
|
29
33
|
: [];
|
|
30
|
-
|
|
34
|
+
const line = JSON.stringify(record);
|
|
35
|
+
const at = record.id === undefined ? -1 : lines.findIndex((existing) => {
|
|
36
|
+
// The substring test keeps this cheap; the parse keeps it honest.
|
|
37
|
+
if (!existing.includes(record.id)) return false;
|
|
38
|
+
try {
|
|
39
|
+
return JSON.parse(existing).id === record.id;
|
|
40
|
+
} catch {
|
|
41
|
+
return false;
|
|
42
|
+
}
|
|
43
|
+
});
|
|
44
|
+
if (at === -1) lines.push(line);
|
|
45
|
+
else lines[at] = line;
|
|
31
46
|
writeFileSync(path, lines.slice(-MAX_RUNS).join("\n") + "\n");
|
|
32
47
|
}
|
|
33
48
|
|
package/lib/judge.js
CHANGED
|
@@ -137,6 +137,7 @@ export function judge(options) {
|
|
|
137
137
|
const {
|
|
138
138
|
rubricPath,
|
|
139
139
|
changed,
|
|
140
|
+
timeoutMs,
|
|
140
141
|
spawn = spawnSync,
|
|
141
142
|
exists = existsSync,
|
|
142
143
|
read = (path) => readFileSync(path, "utf8")
|
|
@@ -166,10 +167,23 @@ export function judge(options) {
|
|
|
166
167
|
input: buildPrompt(body, bundle),
|
|
167
168
|
encoding: "utf8",
|
|
168
169
|
env: { ...process.env, HIGHBALL_JUDGE: "1" },
|
|
169
|
-
maxBuffer: 32 * 1024 * 1024
|
|
170
|
+
maxBuffer: 32 * 1024 * 1024,
|
|
171
|
+
// The runner's budget for this rule. A judge that never answers —
|
|
172
|
+
// a stalled model, a CLI waiting on a login prompt — is a failed
|
|
173
|
+
// rule with a reason, not a hook held open until something else
|
|
174
|
+
// kills it.
|
|
175
|
+
timeout: timeoutMs,
|
|
176
|
+
killSignal: "SIGKILL"
|
|
170
177
|
}
|
|
171
178
|
);
|
|
172
179
|
|
|
180
|
+
if (child.error?.code === "ETIMEDOUT") {
|
|
181
|
+
return {
|
|
182
|
+
passed: false,
|
|
183
|
+
timedOut: true,
|
|
184
|
+
output: `AI judge gave no verdict within ${timeoutMs / 1000}s.`
|
|
185
|
+
};
|
|
186
|
+
}
|
|
173
187
|
if (child.error?.code === "ENOENT") {
|
|
174
188
|
return {
|
|
175
189
|
passed: false,
|
package/lib/mcp.js
CHANGED
|
@@ -109,6 +109,18 @@ function reply(text, structuredContent) {
|
|
|
109
109
|
const glyphFor = (status) =>
|
|
110
110
|
status === "passed" ? "✓" : status === "todo" ? "•" : "✗";
|
|
111
111
|
|
|
112
|
+
const runLabel = (status) =>
|
|
113
|
+
status === "passed" ? "✓ passed"
|
|
114
|
+
: status === "running" ? "… running"
|
|
115
|
+
: status === "killed" ? "✗ KILLED"
|
|
116
|
+
: "✗ FAILED";
|
|
117
|
+
|
|
118
|
+
// A failure the runner caused, rather than the rule, says so.
|
|
119
|
+
const ruleLabel = (result) =>
|
|
120
|
+
result.timed_out ? "failed (timed out)"
|
|
121
|
+
: result.killed ? "failed (killed)"
|
|
122
|
+
: result.status;
|
|
123
|
+
|
|
112
124
|
export function listText(project, runs, total = runs.length) {
|
|
113
125
|
if (runs.length === 0) return `No runs recorded for ${project} yet.`;
|
|
114
126
|
const scope = total > runs.length
|
|
@@ -117,7 +129,7 @@ export function listText(project, runs, total = runs.length) {
|
|
|
117
129
|
: "";
|
|
118
130
|
const rows = runs.map((run) => [
|
|
119
131
|
`#${run.index}`,
|
|
120
|
-
run.status
|
|
132
|
+
runLabel(run.status),
|
|
121
133
|
run.trigger === "edit" ? "fast" : "full",
|
|
122
134
|
run.branch || "-",
|
|
123
135
|
run.duration_ms != null ? `${(run.duration_ms / 1000).toFixed(1)}s` : "-",
|
|
@@ -148,7 +160,7 @@ export function detailText(project, index, run) {
|
|
|
148
160
|
const rules = run.results.map((result) => {
|
|
149
161
|
const duration = result.duration_ms != null
|
|
150
162
|
? ` (${(result.duration_ms / 1000).toFixed(1)}s)` : "";
|
|
151
|
-
let line = ` ${glyphFor(result.status)} ${result.name} — ${result
|
|
163
|
+
let line = ` ${glyphFor(result.status)} ${result.name} — ${ruleLabel(result)}${duration}`;
|
|
152
164
|
if (result.status === "failed" && result.output_tail) {
|
|
153
165
|
line += "\n" + result.output_tail.split("\n").map((l) => ` ${l}`).join("\n");
|
|
154
166
|
}
|
package/lib/run.js
CHANGED
|
@@ -3,11 +3,12 @@
|
|
|
3
3
|
// contract: a Stop hook reading exit 2 blocks the agent and feeds the
|
|
4
4
|
// output back). Reporting is the witness half and is best-effort: an
|
|
5
5
|
// unreachable PostHog must never block the agent.
|
|
6
|
-
import { execSync,
|
|
6
|
+
import { execSync, spawn } from "node:child_process";
|
|
7
|
+
import { randomUUID } from "node:crypto";
|
|
7
8
|
import { readFileSync } from "node:fs";
|
|
8
9
|
import {
|
|
9
10
|
CONFIG_PATH, DISABLED_MARKER, disabledByEnv, disabledByMarker,
|
|
10
|
-
loadConfig, resolvePosthog, commandFor
|
|
11
|
+
loadConfig, resolvePosthog, commandFor, timeoutFor
|
|
11
12
|
} from "./config.js";
|
|
12
13
|
import { appendRun } from "./journal.js";
|
|
13
14
|
import { readStamp, treeFingerprint, writeStamp } from "./stamp.js";
|
|
@@ -96,48 +97,121 @@ export async function run(args) {
|
|
|
96
97
|
const startedAt = new Date();
|
|
97
98
|
const results = [];
|
|
98
99
|
|
|
99
|
-
|
|
100
|
-
|
|
100
|
+
// Everything the journal record needs that doesn't change during the
|
|
101
|
+
// run, gathered once: the record is written many times (below), and the
|
|
102
|
+
// transcript read behind `work` is not free.
|
|
103
|
+
const context = {
|
|
104
|
+
id: randomUUID(),
|
|
105
|
+
project: config.project,
|
|
106
|
+
startedAt,
|
|
107
|
+
fastOnly,
|
|
108
|
+
session: hook.session_id || null,
|
|
109
|
+
work: latestUserPrompt(hook.transcript_path),
|
|
110
|
+
branch: git("git branch --show-current"),
|
|
111
|
+
commit: git("git rev-parse HEAD")
|
|
112
|
+
};
|
|
101
113
|
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
114
|
+
// The journal sees the run from its first moment, not its last. One
|
|
115
|
+
// record per run, written when the run starts and rewritten after every
|
|
116
|
+
// rule (same id, so it replaces itself), so a run that never reaches the
|
|
117
|
+
// end — a hook timeout, a killed terminal, a crash — still leaves behind
|
|
118
|
+
// exactly how far it got and which rule it was in. Before this, a run
|
|
119
|
+
// that died mid-rule was journaled nowhere at all, and "npm test hangs
|
|
120
|
+
// sometimes" had no evidence to point at. Failures to write never fail
|
|
121
|
+
// the checks, same policy as reporting.
|
|
122
|
+
const journal = (status) => {
|
|
123
|
+
try {
|
|
124
|
+
appendRun(config.project, journalRecord(context, status, results));
|
|
125
|
+
} catch (error) {
|
|
126
|
+
console.error(`highball journal skipped: ${error.message}`);
|
|
109
127
|
}
|
|
128
|
+
};
|
|
129
|
+
journal("running");
|
|
110
130
|
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
131
|
+
// A stopped runner — Claude Code giving up on the hook, a Ctrl-C, a
|
|
132
|
+
// closed terminal — takes the rule it was in down with it and says so in
|
|
133
|
+
// the journal, instead of leaving a hung suite running on unowned and a
|
|
134
|
+
// run that looks like it never happened. `current` is the child of the
|
|
135
|
+
// rule executing right now; `stopped` remembers the signal so the loop
|
|
136
|
+
// can wind down rather than start the next rule.
|
|
137
|
+
let current = null;
|
|
138
|
+
let stopped = null;
|
|
139
|
+
const onSignal = (signal) => {
|
|
140
|
+
if (stopped) return;
|
|
141
|
+
stopped = signal;
|
|
142
|
+
if (current) killTree(current);
|
|
143
|
+
};
|
|
144
|
+
const SIGNALS = [ "SIGTERM", "SIGINT", "SIGHUP" ];
|
|
145
|
+
for (const signal of SIGNALS) process.on(signal, onSignal);
|
|
146
|
+
|
|
147
|
+
try {
|
|
148
|
+
for (const rule of rules) {
|
|
149
|
+
process.stdout.write(`→ ${rule.name} ... `);
|
|
150
|
+
|
|
151
|
+
// Placeholder rules are tracked, not run: they report as "todo" so
|
|
152
|
+
// the widget and PostHog show the full intended ruleset, and they can never
|
|
153
|
+
// fail a run — an aspiration shouldn't block anyone.
|
|
154
|
+
if (rule.todo) {
|
|
155
|
+
console.log("todo (not implemented yet)");
|
|
156
|
+
results.push({ rule, passed: true, todo: true, durationMs: null, output: "" });
|
|
157
|
+
journal("running");
|
|
158
|
+
continue;
|
|
159
|
+
}
|
|
160
|
+
|
|
161
|
+
const timeoutMs = timeoutFor(rule, config);
|
|
115
162
|
const t0 = process.hrtime.bigint();
|
|
116
|
-
|
|
163
|
+
let outcome;
|
|
164
|
+
|
|
165
|
+
// Rubric rules run in-process instead of shelling out: the judge needs
|
|
166
|
+
// the `claude` CLI, which lives on the host, so it bypasses `exec.via`
|
|
167
|
+
// by construction rather than by annotation.
|
|
168
|
+
if (rule.rubric) {
|
|
169
|
+
outcome = judge({ rubricPath: rule.rubric, changed, timeoutMs });
|
|
170
|
+
} else {
|
|
171
|
+
// The runner owns git (ADR 202608): check scripts get the changed
|
|
172
|
+
// list handed to them and stay pure analyzers — no git, no network
|
|
173
|
+
// required in their execution context (which may be a container).
|
|
174
|
+
const child = await runCommand(commandFor(rule, config), {
|
|
175
|
+
env: { ...process.env, HIGHBALL_CHANGED_FILES: changed },
|
|
176
|
+
timeoutMs,
|
|
177
|
+
onSpawn: (spawned) => { current = spawned; }
|
|
178
|
+
});
|
|
179
|
+
current = null;
|
|
180
|
+
outcome = { passed: child.status === 0, output: child.output, timedOut: child.timedOut };
|
|
181
|
+
}
|
|
182
|
+
|
|
117
183
|
const durationMs = Math.round(Number(process.hrtime.bigint() - t0) / 1e6);
|
|
184
|
+
const result = { rule, passed: outcome.passed, todo: false, durationMs, output: outcome.output };
|
|
118
185
|
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
186
|
+
if (stopped) {
|
|
187
|
+
// Whatever the child reported, it reported because we killed it.
|
|
188
|
+
result.passed = false;
|
|
189
|
+
result.killed = true;
|
|
190
|
+
result.output = `${outcome.output}\n\nhighball was stopped (${stopped}) while this rule was running.`;
|
|
191
|
+
console.log(`KILLED (${(durationMs / 1000).toFixed(1)}s)`);
|
|
192
|
+
results.push(result);
|
|
193
|
+
break;
|
|
194
|
+
}
|
|
123
195
|
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
}
|
|
135
|
-
|
|
136
|
-
const
|
|
137
|
-
|
|
196
|
+
if (outcome.timedOut) {
|
|
197
|
+
result.passed = false;
|
|
198
|
+
result.timedOut = true;
|
|
199
|
+
result.output = `${outcome.output}\n\n${timeoutAdvice(rule, config, timeoutMs)}`;
|
|
200
|
+
console.log(`TIMED OUT (${(durationMs / 1000).toFixed(1)}s)`);
|
|
201
|
+
} else {
|
|
202
|
+
console.log(`${result.passed ? "passed" : "FAILED"} (${(durationMs / 1000).toFixed(1)}s)`);
|
|
203
|
+
}
|
|
204
|
+
results.push(result);
|
|
205
|
+
journal("running");
|
|
206
|
+
}
|
|
207
|
+
} finally {
|
|
208
|
+
for (const signal of SIGNALS) process.off(signal, onSignal);
|
|
209
|
+
}
|
|
138
210
|
|
|
139
|
-
|
|
140
|
-
|
|
211
|
+
if (stopped) {
|
|
212
|
+
journal("killed");
|
|
213
|
+
console.error(`\nhighball: stopped by ${stopped} — run journaled as killed`);
|
|
214
|
+
return 1;
|
|
141
215
|
}
|
|
142
216
|
|
|
143
217
|
// Stamped pass or fail: after a failure the agent's next reads must not
|
|
@@ -146,59 +220,18 @@ export async function run(args) {
|
|
|
146
220
|
|
|
147
221
|
const failures = results.filter((result) => !result.passed);
|
|
148
222
|
const durationMs = Date.now() - startedAt.getTime();
|
|
149
|
-
const branch = git("git branch --show-current");
|
|
150
|
-
const commitSha = git("git rev-parse HEAD");
|
|
151
223
|
|
|
152
224
|
const { host, key } = resolvePosthog(config);
|
|
153
225
|
if (host && key) {
|
|
154
226
|
await reportPosthog({
|
|
155
227
|
host, key, project: config.project, results, hook, fastOnly, startedAt,
|
|
156
|
-
durationMs, branch, commitSha, version: VERSION
|
|
228
|
+
durationMs, branch: context.branch, commitSha: context.commit, version: VERSION
|
|
157
229
|
});
|
|
158
230
|
}
|
|
159
231
|
|
|
160
232
|
// The local journal is unconditional — `highball runs` works with no
|
|
161
|
-
// reporting configured at all.
|
|
162
|
-
|
|
163
|
-
try {
|
|
164
|
-
appendRun(config.project, {
|
|
165
|
-
started_at: startedAt.toISOString(),
|
|
166
|
-
// The repo this run happened in — how the MCP server later resolves
|
|
167
|
-
// "the current project" and grounds the widget's re-run buttons.
|
|
168
|
-
dir: process.cwd(),
|
|
169
|
-
// Which agent session, and what it was working on — the grouping
|
|
170
|
-
// key and label for run history views. The hook payload points at
|
|
171
|
-
// the session transcript; its last real user prompt is the work.
|
|
172
|
-
session: hook.session_id || null,
|
|
173
|
-
work: latestUserPrompt(hook.transcript_path),
|
|
174
|
-
duration_ms: durationMs,
|
|
175
|
-
trigger: fastOnly ? "edit" : "stop",
|
|
176
|
-
branch,
|
|
177
|
-
commit: commitSha,
|
|
178
|
-
status: failures.length === 0 ? "passed" : "failed",
|
|
179
|
-
results: results.map((result) => ({
|
|
180
|
-
id: result.rule.id,
|
|
181
|
-
name: result.rule.name,
|
|
182
|
-
status: result.todo ? "todo" : result.passed ? "passed" : "failed",
|
|
183
|
-
duration_ms: result.durationMs,
|
|
184
|
-
// The command that produced this result. Quiet rules journal no
|
|
185
|
-
// output at all (the AI judges print nothing when they pass), which
|
|
186
|
-
// left viewers with an expandable row wrapping an empty panel; the
|
|
187
|
-
// command is the one detail every real rule can always show. todo
|
|
188
|
-
// rules have no command — nothing ran — which is exactly what makes
|
|
189
|
-
// them inert rather than falsely clickable.
|
|
190
|
-
command:
|
|
191
|
-
result.rule.run ??
|
|
192
|
-
(result.rule.rubric ? `judge ${result.rule.rubric}` : null),
|
|
193
|
-
// Unlike PostHog (one-line summaries only), the journal keeps
|
|
194
|
-
// every rule's output GitHub-Actions-style — it's the user's own
|
|
195
|
-
// disk, and `highball runs <n> --logs` is the payoff.
|
|
196
|
-
output_tail: result.output ? result.output.slice(-10_000) : null
|
|
197
|
-
}))
|
|
198
|
-
});
|
|
199
|
-
} catch (error) {
|
|
200
|
-
console.error(`highball journal skipped: ${error.message}`);
|
|
201
|
-
}
|
|
233
|
+
// reporting configured at all.
|
|
234
|
+
journal(failures.length === 0 ? "passed" : "failed");
|
|
202
235
|
|
|
203
236
|
if (failures.length === 0) return 0;
|
|
204
237
|
|
|
@@ -211,6 +244,131 @@ export async function run(args) {
|
|
|
211
244
|
return 2;
|
|
212
245
|
}
|
|
213
246
|
|
|
247
|
+
// The journal's view of a run at some moment: the same shape whether the
|
|
248
|
+
// run is still going, finished, or was stopped, so every reader handles
|
|
249
|
+
// one record type and `status` alone says which.
|
|
250
|
+
function journalRecord(context, status, results) {
|
|
251
|
+
return {
|
|
252
|
+
id: context.id,
|
|
253
|
+
started_at: context.startedAt.toISOString(),
|
|
254
|
+
// The repo this run happened in — how the MCP server later resolves
|
|
255
|
+
// "the current project" and grounds the widget's re-run buttons.
|
|
256
|
+
dir: process.cwd(),
|
|
257
|
+
// Which agent session, and what it was working on — the grouping
|
|
258
|
+
// key and label for run history views. The hook payload points at
|
|
259
|
+
// the session transcript; its last real user prompt is the work.
|
|
260
|
+
session: context.session,
|
|
261
|
+
work: context.work,
|
|
262
|
+
duration_ms: Date.now() - context.startedAt.getTime(),
|
|
263
|
+
trigger: context.fastOnly ? "edit" : "stop",
|
|
264
|
+
branch: context.branch,
|
|
265
|
+
commit: context.commit,
|
|
266
|
+
status,
|
|
267
|
+
results: results.map((result) => ({
|
|
268
|
+
id: result.rule.id,
|
|
269
|
+
name: result.rule.name,
|
|
270
|
+
status: result.todo ? "todo" : result.passed ? "passed" : "failed",
|
|
271
|
+
duration_ms: result.durationMs,
|
|
272
|
+
// Why a failed rule failed, when the reason was the runner's and not
|
|
273
|
+
// the rule's. Absent otherwise, so ordinary records stay as they were.
|
|
274
|
+
...(result.timedOut ? { timed_out: true } : {}),
|
|
275
|
+
...(result.killed ? { killed: true } : {}),
|
|
276
|
+
// The command that produced this result. Quiet rules journal no
|
|
277
|
+
// output at all (the AI judges print nothing when they pass), which
|
|
278
|
+
// left viewers with an expandable row wrapping an empty panel; the
|
|
279
|
+
// command is the one detail every real rule can always show. todo
|
|
280
|
+
// rules have no command — nothing ran — which is exactly what makes
|
|
281
|
+
// them inert rather than falsely clickable.
|
|
282
|
+
command:
|
|
283
|
+
result.rule.run ??
|
|
284
|
+
(result.rule.rubric ? `judge ${result.rule.rubric}` : null),
|
|
285
|
+
// Unlike PostHog (one-line summaries only), the journal keeps
|
|
286
|
+
// every rule's output GitHub-Actions-style — it's the user's own
|
|
287
|
+
// disk, and `highball runs <n> --logs` is the payoff.
|
|
288
|
+
output_tail: result.output ? result.output.slice(-10_000) : null
|
|
289
|
+
}))
|
|
290
|
+
};
|
|
291
|
+
}
|
|
292
|
+
|
|
293
|
+
// What the agent reads after a rule ran out of time. It names the knob,
|
|
294
|
+
// because the right fix is sometimes a slower rule's honest budget and
|
|
295
|
+
// sometimes a hang — and either way the number should be a decision
|
|
296
|
+
// someone made in checks.yml, not a mystery.
|
|
297
|
+
function timeoutAdvice(rule, config, timeoutMs) {
|
|
298
|
+
const seconds = timeoutMs / 1000;
|
|
299
|
+
const knob = rule.timeout != null
|
|
300
|
+
? `\`timeout: ${seconds}\` on this rule`
|
|
301
|
+
: `\`timeouts.${rule.rubric ? "judge" : rule.fast ? "fast" : "full"}: ${seconds}\``;
|
|
302
|
+
return `highball: timed out after ${seconds}s and was killed. Its budget is ${knob} ` +
|
|
303
|
+
`in ${CONFIG_PATH} — raise it there if this rule legitimately needs longer, ` +
|
|
304
|
+
"or find what hung.";
|
|
305
|
+
}
|
|
306
|
+
|
|
307
|
+
// Output kept per rule. Past this the rest is dropped rather than buffered:
|
|
308
|
+
// a runaway logger must not turn into a runaway runner.
|
|
309
|
+
const MAX_OUTPUT = 32 * 1024 * 1024;
|
|
310
|
+
|
|
311
|
+
// Runs one rule's command with a wall-clock budget. Resolves rather than
|
|
312
|
+
// rejects in every case — a rule that can't even start is a failed rule
|
|
313
|
+
// with the error as its output, not a crashed runner.
|
|
314
|
+
function runCommand(command, { env, timeoutMs, onSpawn }) {
|
|
315
|
+
return new Promise((resolve) => {
|
|
316
|
+
const child = spawn(`${command} 2>&1`, {
|
|
317
|
+
shell: true,
|
|
318
|
+
// Its own process group (POSIX), so a timeout can kill the rule's
|
|
319
|
+
// whole tree — the shell, the npm it started, the test runner npm
|
|
320
|
+
// started — and not just the shell, which would leave a hung suite
|
|
321
|
+
// running on after the run had already given up on it.
|
|
322
|
+
detached: process.platform !== "win32",
|
|
323
|
+
env,
|
|
324
|
+
stdio: [ "ignore", "pipe", "pipe" ]
|
|
325
|
+
});
|
|
326
|
+
onSpawn?.(child);
|
|
327
|
+
|
|
328
|
+
let output = "";
|
|
329
|
+
const collect = (chunk) => {
|
|
330
|
+
if (output.length < MAX_OUTPUT) output += chunk;
|
|
331
|
+
};
|
|
332
|
+
child.stdout.setEncoding("utf8");
|
|
333
|
+
child.stdout.on("data", collect);
|
|
334
|
+
child.stderr.setEncoding("utf8");
|
|
335
|
+
child.stderr.on("data", collect);
|
|
336
|
+
|
|
337
|
+
let timedOut = false;
|
|
338
|
+
const timer = setTimeout(() => {
|
|
339
|
+
timedOut = true;
|
|
340
|
+
killTree(child);
|
|
341
|
+
}, timeoutMs);
|
|
342
|
+
|
|
343
|
+
child.on("error", (error) => {
|
|
344
|
+
clearTimeout(timer);
|
|
345
|
+
resolve({ status: null, output: `${output}${error.message}\n`, timedOut });
|
|
346
|
+
});
|
|
347
|
+
child.on("close", (status) => {
|
|
348
|
+
clearTimeout(timer);
|
|
349
|
+
resolve({ status, output, timedOut });
|
|
350
|
+
});
|
|
351
|
+
});
|
|
352
|
+
}
|
|
353
|
+
|
|
354
|
+
// Stops a rule's whole process tree: SIGTERM to the group first so
|
|
355
|
+
// anything listening can clean up, SIGKILL a moment later for whatever
|
|
356
|
+
// didn't. Exported for tests.
|
|
357
|
+
export function killTree(child) {
|
|
358
|
+
const signal = (name) => {
|
|
359
|
+
try {
|
|
360
|
+
if (process.platform === "win32" || !child.pid) child.kill(name);
|
|
361
|
+
else process.kill(-child.pid, name);
|
|
362
|
+
} catch {
|
|
363
|
+
// already gone
|
|
364
|
+
}
|
|
365
|
+
};
|
|
366
|
+
signal("SIGTERM");
|
|
367
|
+
const escalate = setTimeout(() => signal("SIGKILL"), 1000);
|
|
368
|
+
escalate.unref();
|
|
369
|
+
child.once("close", () => clearTimeout(escalate));
|
|
370
|
+
}
|
|
371
|
+
|
|
214
372
|
// Claude Code hooks pass a JSON payload on stdin (session_id and
|
|
215
373
|
// friends); that id groups this run with the rest of the agent's session
|
|
216
374
|
// in the journal and in PostHog. A TTY means a human at a terminal — don't block on
|
package/lib/runs.js
CHANGED
|
@@ -15,6 +15,20 @@ const green = (text) => paint("32", text);
|
|
|
15
15
|
const red = (text) => paint("31;1", text);
|
|
16
16
|
const dim = (text) => paint("2", text);
|
|
17
17
|
|
|
18
|
+
// A run's status, as a label. "running" is a run the journal has seen
|
|
19
|
+
// start and not finish — live, or one that died without saying so.
|
|
20
|
+
const runLabel = (status) =>
|
|
21
|
+
status === "passed" ? green("✓ passed")
|
|
22
|
+
: status === "running" ? dim("… running")
|
|
23
|
+
: status === "killed" ? red("✗ KILLED")
|
|
24
|
+
: red("✗ FAILED");
|
|
25
|
+
|
|
26
|
+
// A rule's status, as a label: a failure the runner caused says why.
|
|
27
|
+
const ruleLabel = (result) =>
|
|
28
|
+
result.status === "passed" ? green(result.status)
|
|
29
|
+
: result.status === "todo" ? dim(result.status)
|
|
30
|
+
: red(result.timed_out ? "TIMED OUT" : result.killed ? "KILLED" : result.status.toUpperCase());
|
|
31
|
+
|
|
18
32
|
// Column math must use what the eye sees, not what the terminal parses.
|
|
19
33
|
const visible = (text) => text.replace(/\x1b\[[0-9;]*m/g, "");
|
|
20
34
|
const padEnd = (text, width) => text + " ".repeat(Math.max(0, width - visible(text).length));
|
|
@@ -70,7 +84,7 @@ function list(project, history) {
|
|
|
70
84
|
run.trigger === "edit" ? "fast" : "full",
|
|
71
85
|
run.branch || "-",
|
|
72
86
|
dim(run.duration_ms != null ? `${(run.duration_ms / 1000).toFixed(1)}s` : "-"),
|
|
73
|
-
run.status
|
|
87
|
+
runLabel(run.status),
|
|
74
88
|
tally(run.results)
|
|
75
89
|
]);
|
|
76
90
|
const lines = formatRows(rows, new Set([ 4, 5, 6 ]));
|
|
@@ -98,7 +112,7 @@ function detail(project, history, number, showLogs) {
|
|
|
98
112
|
}
|
|
99
113
|
|
|
100
114
|
const took = run.duration_ms != null ? ` · took ${(run.duration_ms / 1000).toFixed(1)}s` : "";
|
|
101
|
-
const status = run.status
|
|
115
|
+
const status = runLabel(run.status);
|
|
102
116
|
console.log(
|
|
103
117
|
`Run #${number} — ${project} · ${run.trigger === "edit" ? "fast checks" : "full suite"}` +
|
|
104
118
|
` · ${run.branch || "-"} · ${(run.commit || "").slice(0, 7)}` +
|
|
@@ -112,7 +126,7 @@ function detail(project, history, number, showLogs) {
|
|
|
112
126
|
const rows = run.results.map((result) => [
|
|
113
127
|
result.status === "passed" ? green("✓") : result.status === "todo" ? dim("•") : red("✗"),
|
|
114
128
|
result.status === "failed" ? red(result.name) : result.name,
|
|
115
|
-
|
|
129
|
+
ruleLabel(result),
|
|
116
130
|
result.duration_ms != null ? dim(`${(result.duration_ms / 1000).toFixed(1)}s`) : ""
|
|
117
131
|
]);
|
|
118
132
|
const lines = formatRows(rows, new Set([ 2, 3 ]));
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@profoundry-us/highball",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.7.0",
|
|
4
4
|
"description": "Highball runner — local CI for AI coding agents: runs a repo's .highball/checks.yml rules, blocks the agent on failure, and optionally reports runs to PostHog.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"ai",
|
|
@@ -14,7 +14,7 @@
|
|
|
14
14
|
"type": "git",
|
|
15
15
|
"url": "git+https://github.com/profoundry-us/highball.git"
|
|
16
16
|
},
|
|
17
|
-
"homepage": "https://
|
|
17
|
+
"homepage": "https://highball.profoundry.us",
|
|
18
18
|
"bugs": {
|
|
19
19
|
"url": "https://github.com/profoundry-us/highball/issues"
|
|
20
20
|
},
|
|
@@ -27,7 +27,8 @@
|
|
|
27
27
|
"lib",
|
|
28
28
|
"assets",
|
|
29
29
|
"README.md",
|
|
30
|
-
"ONBOARDING.md"
|
|
30
|
+
"ONBOARDING.md",
|
|
31
|
+
"CHANGELOG.md"
|
|
31
32
|
],
|
|
32
33
|
"engines": {
|
|
33
34
|
"node": ">=18"
|