sawbuck 0.1.46 → 0.1.48
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CONTRACT.md +9 -9
- package/README.md +53 -33
- package/bins.mjs +1 -1
- package/contract-cmd.mjs +20 -10
- package/debt.mjs +26 -11
- package/exclude.mjs +5 -4
- package/history.mjs +4 -1
- package/init.mjs +7 -6
- package/memory.mjs +5 -13
- package/package.json +4 -3
- package/pipe.mjs +2 -0
- package/rank.mjs +2 -3
- package/reflect.mjs +30 -21
- package/repo-state.mjs +21 -4
- package/rules.mjs +10 -2
- package/sawbuck-hook.mjs +2 -7
- package/sawbuck.mjs +360 -139
- package/shape.mjs +96 -12
- package/status.mjs +14 -6
- package/steers.mjs +1 -1
package/CONTRACT.md
CHANGED
|
@@ -2,13 +2,13 @@
|
|
|
2
2
|
|
|
3
3
|
*Put your coding agent under contract: a win counts only if it survives verification from the source alone, not because the agent said so. The terms below are not a promise the agent makes, they are a result anyone can re-prove.*
|
|
4
4
|
|
|
5
|
-
**Status:** Draft v0.1 · **License:** Apache-2.0 · **Reference implementation:** [Sawbuck](https://github.com/sawbuck-ai/sawbuck) (the CLI), [promptwheel](https://github.com/promptwheel-ai/promptwheel) (the check), [@promptwheel/logbook](https://github.com/promptwheel-ai/logbook) (the
|
|
5
|
+
**Status:** Draft v0.1 · **License:** Apache-2.0 · **Reference implementation:** [Sawbuck](https://github.com/sawbuck-ai/sawbuck) (the CLI), [promptwheel](https://github.com/promptwheel-ai/promptwheel) (the check), [@promptwheel/logbook](https://github.com/promptwheel-ai/logbook) (the memory)
|
|
6
6
|
|
|
7
7
|
This is an open interface for making an AI coding harness trustworthy. It defines how a loop's wins get verified, what a verdict looks like, what counts as gaming the goalposts, and how a repo's history is carried along. Any harness can emit or consume it without using this code. Sawbuck is a reference implementation; it is not the only possible one.
|
|
8
8
|
|
|
9
9
|
The contract has one property that holds all of it together: a verdict is **deterministic and reproducible from `(base, head, config)` alone**. No LLM is in the path, and nothing is read from the agent's trajectory, so the result cannot be obfuscated against and does not change when you ask it twice.
|
|
10
10
|
|
|
11
|
-
A corollary worth stating as an invariant: **the gate is defined only by in-repo, versioned config.** External systems (a dashboard, an issue tracker
|
|
11
|
+
A corollary worth stating as an invariant: **the gate is defined only by in-repo, versioned config.** External systems (a dashboard, an issue tracker) may *report on* a verdict, but they must never *define* the guardrail. A guardrail that lives outside the repo is one the check cannot see in the diff, so it cannot be fingerprinted, so loosening it leaves no trace to catch as gamed. Memory (the input) is Logbook, read from git; the gate (the output) stays in-repo and fingerprinted.
|
|
12
12
|
|
|
13
13
|
> This contract covers the check. Sawbuck pairs it with the other half, **memory**: the relevant history for what you touch, plus the live debt, from @promptwheel/logbook.
|
|
14
14
|
>
|
|
@@ -23,7 +23,7 @@ The per-turn signal a loop consumes. The check (`run`) exits:
|
|
|
23
23
|
| Exit | Verdict | Meaning | A conforming loop should |
|
|
24
24
|
|------|---------|---------|--------------------------|
|
|
25
25
|
| `0` | `pass` | The win is real: it survived a source-only re-proof and no guard regressed. | Keep the change. |
|
|
26
|
-
| `1` | `fail` | A guarded metric regressed beyond the measured noise band. | Revert. |
|
|
26
|
+
| `1` | `fail` | A guarded metric regressed beyond the measured noise band, or a protected path (a mandate) was touched. | Revert. |
|
|
27
27
|
| `2` | `gamed` | A metric improved only by editing the goalposts (test, grader, golden, config), not the source. | Revert and flag. |
|
|
28
28
|
| `3` | `inconclusive` | Nothing could be certified: a guard never ran (inert) and no metric was measured to improve, or a win's source-only re-run could not be applied, so the win could not be re-proved either way. | Do not trust the green; do not keep on this signal alone. |
|
|
29
29
|
|
|
@@ -55,7 +55,7 @@ The agent consumes this as a per-turn reward it cannot inflate. The harness cons
|
|
|
55
55
|
"gamed": true,
|
|
56
56
|
"sourceOnly": null,
|
|
57
57
|
"retained": 0.0,
|
|
58
|
-
"gamingReason": "the win changed zero production-source files, only test/config/grader/golden"
|
|
58
|
+
"gamingReason": "the \"win\" changed zero production-source files, only test/config/grader/golden"
|
|
59
59
|
}
|
|
60
60
|
]
|
|
61
61
|
}
|
|
@@ -67,9 +67,9 @@ Fixed vocabularies (a conforming implementation uses exactly these):
|
|
|
67
67
|
- `metric.status` ∈ `improved | regressed | unchanged | inconclusive | unmeasurable`
|
|
68
68
|
- `direction` ∈ `up | down | pass` (the direction that counts as better; `pass` means a 0/1 gate)
|
|
69
69
|
|
|
70
|
-
`retained` is the fraction of a metric's gain that survived the source-only re-run (§3). `noise` is the measured jitter band as a single number, across the top-level `repeat` runs; a delta inside it leaves that metric `inconclusive` (neither a win nor a regression; the verdict is not failed by it). `confidence` (`none | unverified | low | medium | high`, `none` for an unmeasurable metric) and `sourceOnly` (the metric's value when only source edits are replayed, §3) are informational fields the reference implementation adds; a minimal conforming report may omit them. A consumer validates a report with `conformsToContract()` from the `promptwheel` package (see INTEGRATE.md)
|
|
70
|
+
`retained` is the fraction of a metric's gain that survived the source-only re-run (§3). `noise` is the measured jitter band as a single number, across the top-level `repeat` runs; a delta inside it leaves that metric `inconclusive` (neither a win nor a regression; the verdict is not failed by it). `confidence` (`none | unverified | low | medium | high`, `none` for an unmeasurable metric) and `sourceOnly` (the metric's value when only source edits are replayed, §3) are informational fields the reference implementation adds; a minimal conforming report may omit them. A consumer validates a report with `conformsToContract()` from the `promptwheel` package (see INTEGRATE.md).
|
|
71
71
|
|
|
72
|
-
`gate` (optional) is the gate-integrity record. `gate.fingerprint` is a truncated SHA-256 of the normalized gate definition at `base` and `head`, so a changed gate is visible in the audit record. `gate.weakened`, present only when it happened, lists guards the change dropped (removed, or set to `guard:false`) that would have failed at `head`. A non-empty `gate.weakened` forces `verdict: gamed`: an agent cannot earn a pass by disabling the guard it is being judged by.
|
|
72
|
+
`gate` (optional) is the gate-integrity record. `gate.fingerprint` is a truncated SHA-256 of the normalized gate definition at `base` and `head`, so a changed gate is visible in the audit record. `gate.weakened`, present only when it happened, lists guards the change dropped (removed, or set to `guard:false`) that would have failed at `head`. A non-empty `gate.weakened` forces `verdict: gamed`: an agent cannot earn a pass by disabling the guard it is being judged by. In a signed repo the config is itself a protected path, so the same change also breaks a mandate and reads `fail` (exit 1) first; either way it does not pass.
|
|
73
73
|
|
|
74
74
|
## 3. The integrity boundary (what counts as gaming)
|
|
75
75
|
|
|
@@ -81,15 +81,15 @@ A win is **`gamed`** when it does not survive a **source-only re-run**: revert e
|
|
|
81
81
|
- **Source**: everything else, the code the change was supposed to be about.
|
|
82
82
|
- **`evaluatorPaths`**: an opt-in allowlist a repo declares for its own gate files (a threshold file, a budget, a ruleset), because a generic name is a classifier risk and only the repo owner knows which files are the goalposts.
|
|
83
83
|
|
|
84
|
-
**Honest boundary, stated not hidden:** a cheat that lives *inside the source itself* (code that behaves correctly only while a test is watching) is invisible to a source-only re-run, by construction. The contract catches
|
|
84
|
+
**Honest boundary, stated not hidden:** a cheat that lives *inside the source itself* (code that behaves correctly only while a test is watching) is invisible to a source-only re-run, by construction. The contract catches a win that came from an out-of-source goalpost move; it does not claim to catch a source-resident backdoor. A goalpost moved where there was no win to re-prove (a suite weakened while it stays green) is not proved either way: the check notes it and does not fail it. That needs held-out or behavioral analysis, which is a different layer.
|
|
85
85
|
|
|
86
86
|
**Protected terms (declared, optionally enforced).** `evaluatorPaths` is also the declaration surface for immutability: the files a repo names there are its fixed terms, the goalposts the agent does not get to move. The baseline is detection (a win that depended on changing a protected term reads `gamed`). A stronger conformance level *enforces* it: the audit gate (CI, or a hook the agent cannot bypass) fails any change to a protected term unless a human signs off, so the terms cannot be moved mid-game. Note the honest limit: a file cannot be made truly immutable to an agent that has write access to the repo. The guarantee comes from the un-bypassable gate, not from the filesystem.
|
|
87
87
|
|
|
88
|
-
**Declared override (an intentional test change).** The check flags a win that came only from moving the goalposts, not from the source. Sometimes that move is legitimate: a test that asserted the wrong thing, an obsolete test removed, a guard intentionally retired. The sanctioned way through is to DECLARE it, not to silently weaken the gate. A `Sawbuck-Allow: <reason>` trailer on the commit (`Promptwheel-Allow:` is the same thing; or `--allow "<reason>"` for a working-tree or CI run) acknowledges the change, so a would-be `gamed` reads `pass` with the acknowledgment recorded in `allowed` (the reason, and the metric or gate names it cleared). The check does not judge intent; it forces intent to be explicit and auditable. An UNdeclared test-only win is still `gamed`; a declared one is on the record, so a reviewer sees the reason instead of a silent green. It covers the edit, the delete, and the new file alike. The declaration lives in the commit, so it is immutable and attributable: revoking it is a review act, not a deletion, and a reasonless trailer does not override.
|
|
88
|
+
**Declared override (an intentional test change).** The check flags a win that came only from moving the goalposts, not from the source. Sometimes that move is legitimate: a test that asserted the wrong thing, an obsolete test removed, a guard intentionally retired. The sanctioned way through is to DECLARE it, not to silently weaken the gate. A `Sawbuck-Allow: <reason>` trailer on the commit (`Promptwheel-Allow:` is the same thing; or `--allow "<reason>"` for a working-tree or CI run) acknowledges the change, so a would-be `gamed` reads `pass` with the acknowledgment recorded in `allowed` (the reason, and the metric or gate names it cleared). The check does not judge intent; it forces intent to be explicit and auditable. An UNdeclared test-only win is still `gamed`; a declared one is on the record, so a reviewer sees the reason instead of a silent green. It covers the edit, the delete, and the new file alike. The declaration lives in the commit, so it is immutable and attributable: revoking it is a review act, not a deletion, and a reasonless trailer does not override. A commit message is not a person, though: an agent can write the trailer. So the engine ignores trailers when the contract sets `allowTrailer: false` (every `sawbuck contract` does), and Sawbuck never reads one, signed or not: `sawbuck check`, its Stop hook and its git hook take only `--allow`, or on a pull request a `sawbuck-allow:` comment from someone with write access.
|
|
89
89
|
|
|
90
90
|
## 4. The context record (the trail knowledge)
|
|
91
91
|
|
|
92
|
-
A conforming implementation also carries the repo's history as deterministic memory: a ledger derived from git (hotspots, reverted approaches, suppressed or weakened tests, fragile areas), surfaced
|
|
92
|
+
A conforming implementation also carries the repo's history as deterministic memory: a ledger derived from git (hotspots, reverted approaches, suppressed or weakened tests, fragile areas), surfaced on demand as a ranked slice for the files being touched. It is a projection of git, never a second source of truth, so it is always reconstructable and never drifts from what actually happened.
|
|
93
93
|
|
|
94
94
|
---
|
|
95
95
|
|
package/README.md
CHANGED
|
@@ -4,8 +4,6 @@
|
|
|
4
4
|
|
|
5
5
|
It re-proves every win from the agent's **source** edits alone, so a test that went from red to green only because the agent edited the test reads as gamed.
|
|
6
6
|
|
|
7
|
-
> Sawbuck is the reference implementation of the [Verified-Outcome Contract](CONTRACT.md): the open interface for a coding loop you can trust.
|
|
8
|
-
|
|
9
7
|
In Claude Code, install the plugin and keep working:
|
|
10
8
|
|
|
11
9
|
```
|
|
@@ -13,7 +11,7 @@ In Claude Code, install the plugin and keep working:
|
|
|
13
11
|
/plugin install sawbuck-reins@sawbuck
|
|
14
12
|
```
|
|
15
13
|
|
|
16
|
-
The first time you open a git repository with no rules, Claude finishes your task and offers once to add them. Say yes: it writes the rule, sets your protected paths, and
|
|
14
|
+
The first time you open a git repository with no rules, Claude finishes your task and offers once to add them. Say yes: it writes the rule, sets your protected paths, and watches every turn: protected paths at the end of each turn, each commit re-proved from the source.
|
|
17
15
|
|
|
18
16
|
Or from the command line:
|
|
19
17
|
|
|
@@ -21,16 +19,16 @@ Or from the command line:
|
|
|
21
19
|
# 1 · try it (no install, no commitment): verify your latest change right now
|
|
22
20
|
npx sawbuck check
|
|
23
21
|
|
|
24
|
-
# 2 · give this repo its rules: the written rule, your tests
|
|
22
|
+
# 2 · give this repo its rules: the written rule, protected paths, and your tests if it has no config yet
|
|
25
23
|
npm i -g sawbuck
|
|
26
|
-
sawbuck contract # --full adds supply-chain guards, a repo lock and a CI check
|
|
27
|
-
sawbuck attach #
|
|
24
|
+
sawbuck contract # --full adds supply-chain guards, a repo lock and, on a GitHub repo, a CI check
|
|
25
|
+
sawbuck attach # watch every Claude Code turn (--shared for a team file, --git for any other agent)
|
|
28
26
|
```
|
|
29
27
|
|
|
30
|
-
**Zero-config:** it
|
|
28
|
+
**Zero-config:** it finds npm, pytest, cargo and go tests on its own, so in those repos there is nothing to write to start. Anything else (a Makefile target, say) reads inconclusive until you set the command in `promptwheel.config.json`.
|
|
31
29
|
On attach it tells you exactly what it can measure here (and how to fix it if it can't).
|
|
32
|
-
`sawbuck init`
|
|
33
|
-
In CI, the same check runs as the [promptwheel GitHub Action](https://github.com/promptwheel-ai/promptwheel#in-ci-and-in-claude-code) (`uses: promptwheel-ai/promptwheel@v0`); the debt gate below is this repo's own Action. For every project at once: `sawbuck attach --user`.
|
|
30
|
+
`sawbuck init` writes that config: your detected tests and the tripwires, with notes from your history. Coverage, latency or a cost budget you add to it by hand.
|
|
31
|
+
In CI, the same check runs as the [promptwheel GitHub Action](https://github.com/promptwheel-ai/promptwheel#in-ci-and-in-claude-code) (`uses: promptwheel-ai/promptwheel@v0`); the debt gate below is this repo's own Action. For every project at once: `sawbuck attach --user`. Under `--user` the check runs only in repos that have a `promptwheel.config.json`.
|
|
34
32
|
|
|
35
33
|
One entry point each: a person types `sawbuck`; CI and libraries use `promptwheel` (the engine underneath, `import { check }`). `sawbuck check` passes every `promptwheel run` flag through, so you never need both.
|
|
36
34
|
|
|
@@ -42,56 +40,78 @@ $ sawbuck check
|
|
|
42
40
|
It does NOT survive a source-only re-run, so it does not count.
|
|
43
41
|
|
|
44
42
|
$ sawbuck check # after fixing the source and restoring the test
|
|
45
|
-
🐴 ✓ verified: your working change earned it. "tests_pass"
|
|
43
|
+
🐴 ✓ verified: your working change earned it. "tests_pass" improved, and the source carried the win
|
|
44
|
+
(it survives a source-only re-run).
|
|
46
45
|
```
|
|
47
46
|
|
|
48
|
-
Exit codes: `0` verified · `1` regression · `2` gamed · `3` inconclusive · `64` could not run (not a git repo, no commit); no detectable test command reads `3` inconclusive.
|
|
47
|
+
Exit codes: `0` pass (verified, held, or allowed) · `1` sent back (a regression or a protected path) · `2` gamed · `3` inconclusive · `64` could not run (not a git repo, no commit, a bad ref, an unknown flag, a bad or missing flag value, a bad config, or `--working` with nothing uncommitted, and it says which); no detectable test command reads `3` inconclusive, and says so.
|
|
49
48
|
|
|
50
49
|
## What it does
|
|
51
50
|
|
|
52
|
-
- **Verifies the win.** `check` re-runs your metric from the source diff alone. A test that went from red to green because of a deleted, weakened, or skipped test, a mocked grader, an edited golden file, or a loosened declared config reads `gamed`. A suite weakened while it stays green is not proved: there is no win to re-run. Tripwires catch skips and removed assertions in a signed repo. Deterministic, zero-LLM, $0, and it never reads the trajectory. No appeals once signed; `--allow`
|
|
51
|
+
- **Verifies the win.** `check` re-runs your metric from the source diff alone. A test that went from red to green because of a deleted, weakened, or skipped test, a mocked grader, an edited golden file, or a loosened declared config reads `gamed`. A suite weakened while it stays green is not proved: there is no win to re-run. Tripwires catch skips and removed assertions in a signed repo. Deterministic, zero-LLM, $0, and it never reads the trajectory. No appeals once signed; `--allow` lets a change through with a person's reason, and the check says it was allowed, never verified. A `Sawbuck-Allow:` commit trailer clears nothing in `sawbuck check`, the Stop hook or the git hook, signed or not, because an agent can write a commit message. Red-teamed in 26 scenarios across 9 ecosystems: of 22 cheats, 12 were caught and 10 got through, each one a limit the report lists, and all 4 honest fixes passed ([promptwheel/bench/REDTEAM.md](https://github.com/promptwheel-ai/promptwheel/blob/main/bench/REDTEAM.md)).
|
|
53
52
|
- **Says what it cannot see.** A cheat that lives inside the source itself (a value hardcoded to the one visible case, logic keyed to the test environment) is invisible to a source-only re-run, by construction. That limit is in [the contract](CONTRACT.md#3-the-integrity-boundary-what-counts-as-gaming), not hidden.
|
|
54
53
|
- **Remembers what broke.** `context` surfaces the relevant history (reverts, fragile areas, test-weakening) for the files you are touching: a ranked trail guide, before you act.
|
|
55
|
-
- **Shows you everything.** `status` is the inventory: what it has caught for you over time, the debt accruing now, and whether it is armed here, in one view. On-demand, so it is quiet until you go looking.
|
|
54
|
+
- **Shows you everything.** `status` is the inventory: what it has caught for you over time, the debt accruing now, and whether it is armed here, in one view. It says watching only while hooks are wired for this repo, not because an earlier attach left its record on disk. On-demand, so it is quiet until you go looking.
|
|
56
55
|
- **Rides on, does not replace.** Sawbuck is the verify + memory layer. Your agent and your loop stay yours.
|
|
57
|
-
- **Clean on and off.** `attach` writes
|
|
56
|
+
- **Clean on and off.** `attach` writes a git-excluded `.claude/settings.local.json`, the lines in `.git/info/exclude` that keep its files out of `git status`, a receipt under `.git/sawbuck/`, a line in `~/.sawbuck/registry.json` (the list `detach --all` reads), and in a signed repo `.sawbuck/state.json` (`attach --git` writes the post-commit hook and `.sawbuck/state.json` in place of the settings file); `attach --dry-run` lists all of it. `detach` removes the hooks and the receipt, and still removes the hooks when the receipt is lost; `detach --dry-run` lists what would go. A post-commit hook that was there before `attach --git` is put back byte for byte. Signed contract files stay because they are yours to review. The record (`.sawbuck/`, the logbook files) stays too, still out of `git status`, until `detach --purge` deletes it and the exclude lines. `detach --verify` proves the wiring is gone and names what is left.
|
|
57
|
+
- **A limit of the hooks file.** `.claude/settings.local.json` is kept out of git, so taking the hooks out of it shows in no diff, and the check cannot see it. `sawbuck doctor` can: it reads the receipt and says the hooks are gone, or that the set is incomplete (a missing Stop hook, say). `sawbuck attach --shared` avoids it: the hooks go in a tracked file, and a removal shows in git.
|
|
58
|
+
- **A limit of the turn hook.** The hook re-runs your tests on commits, not on uncommitted work. A test edited to go green and left uncommitted is not sent back at the end of that turn. It is caught when it is committed, and `sawbuck check` reads it on demand, which is what the written rule tells the agent to run before finishing.
|
|
59
|
+
- **A limit of the git hook.** `attach --git` pins the hook to the checkout it was attached from, so a commit made in a linked git worktree is not checked. A hooks folder inside the work tree (`core.hooksPath`, as husky sets it) is refused, and attach prints the line to add by hand; `detach`, `doctor` and `status` do not know about a line added that way.
|
|
58
60
|
|
|
59
61
|
## Commands
|
|
60
62
|
|
|
61
63
|
| command | what |
|
|
62
64
|
|---|---|
|
|
63
|
-
| `sawbuck contract` | gives a repo its rules:
|
|
65
|
+
| `sawbuck contract` | gives a repo its rules: protected paths and the written rule. With no config yet it writes one with your detected tests; an existing config is kept as it is, and gets no tests added. `--full` adds supply-chain guards, a repo lock and a CI check (only with a github.com origin and a real test command; it says when it skips it). `--strict` makes them block, also after an earlier `--full`. `--dry-run` previews |
|
|
64
66
|
| `sawbuck check [--base R --head R] [--repeat N] [--allow "reason"] [--json]` | verify your latest change (working tree if dirty, else the last commit, or a range); zero-config, no attach, no record. Every `promptwheel run` flag passes through: this is the engine with a plain-spoken verdict |
|
|
65
|
-
| `sawbuck attach [--user]` |
|
|
66
|
-
| `sawbuck detach [--user]` | remove
|
|
67
|
+
| `sawbuck attach [--user \| --shared \| --git]` | watch every Claude Code turn here: protected paths at the end of each turn, a source-only re-run on each commit. `--user`: in every repo that has a `promptwheel.config.json`. `--shared`: a tracked `.claude/settings.json` a team commits, with no path from your machine in it. `--git`: a post-commit hook, so every commit is checked whatever made it, from the first one after attach (the very first commit of a repo has nothing to be checked against). It leaves a hook alone if it is not a shell script or not executable, and says what to add by hand |
|
|
68
|
+
| `sawbuck detach [--user] [--all] [--verify] [--purge] [--dry-run]` | remove the hooks; no flag removes whatever is wired here, with or without the receipt, `--all` does every repo. `--purge` also deletes the record, `--dry-run` only lists what would go |
|
|
67
69
|
| `/sawbuck-reins:audit` | finds rules in your agent files that can pull against each other, and proposes which wins. No history needed |
|
|
68
70
|
| `/sawbuck-reins:rules` | proposes the rules you keep enforcing, from your own messages, each with quotes; you pick what to keep. Tested on one person so far |
|
|
69
|
-
| `/sawbuck-reins:shape` | what you push on,
|
|
70
|
-
| `sawbuck
|
|
71
|
-
| `sawbuck
|
|
71
|
+
| `/sawbuck-reins:shape` | what you push on, from a sample of your own messages, as a card. It also reads up to 120 past turns and names habits the agent has that you never push on, each with the agent's own words. Leads from one read, tested on one person |
|
|
72
|
+
| `sawbuck init` | writes only the config: your detected tests and the tripwires, with notes from your history. `contract` runs it for you when the repo has no config yet |
|
|
73
|
+
| `sawbuck status` | the inventory (caught over time, debt now, armed state, what you keep saying), in one view; `--log` for the recent per-check feed |
|
|
74
|
+
| `sawbuck debt [repo]` | the whole-repo live suppression debt: what is still silenced in HEAD now, disabled tests first, heaviest files ranked |
|
|
72
75
|
| `sawbuck context` | the ranked relevant history for the current change |
|
|
73
|
-
| `sawbuck doctor` · `detach --verify` |
|
|
76
|
+
| `sawbuck doctor [--all]` · `detach --verify` | run what each hook runs (project and user level) and say if it still works, if the set is incomplete, or if it is gone (exit 1), `--all` for every repo it is attached to; prove the wiring is gone |
|
|
74
77
|
| `sawbuck statusline` | a one-line status-line segment (🐴 watching / N caught) |
|
|
75
78
|
|
|
79
|
+
## Protected paths
|
|
80
|
+
|
|
81
|
+
`sawbuck contract` protects these by default. A change to one reads "sent back", whoever made it.
|
|
82
|
+
|
|
83
|
+
| rule | paths | how a person makes an exception |
|
|
84
|
+
|---|---|---|
|
|
85
|
+
| `contract-frozen` | `promptwheel.config.json`, `sawbuck.config.json` | `sawbuck check --allow "<reason>"` |
|
|
86
|
+
| `hooks-frozen` | `.claude/settings.json`, `.claude/settings.local.json`, `.sawbuck/**` | `sawbuck check --allow "<reason>"` |
|
|
87
|
+
| `workflows-frozen` | `.github/workflows/**` | remove the rule first |
|
|
88
|
+
| `registry-config-frozen` | `.npmrc`, `.yarnrc`, `.yarnrc.yml` | remove the rule first |
|
|
89
|
+
| `build-images-frozen` | `Dockerfile*`, `docker-compose*` | remove the rule first |
|
|
90
|
+
| `publish-scripts-frozen` | `scripts/release*`, `scripts/publish*` | remove the rule first |
|
|
91
|
+
| `codeowners-frozen` | `CODEOWNERS` | remove the rule first |
|
|
92
|
+
|
|
93
|
+
To remove a rule, delete its entry under `mandates` in `promptwheel.config.json`, in a commit of its own. That file is protected too, so that commit takes `--allow`. Then make the change. To protect another path, add an entry the same way: `{"name": "billing-frozen", "paths": ["src/billing/**"], "reason": "billing changes go through review"}`.
|
|
94
|
+
|
|
76
95
|
## Give your agent the rule
|
|
77
96
|
|
|
78
|
-
`sawbuck init`
|
|
97
|
+
`sawbuck contract` and `sawbuck init` write this block into your `AGENTS.md` for you (and `CLAUDE.md` / `.cursorrules` if present; `init --no-agent-block` skips it), and `sawbuck attach` enforces it through the hook: every turn checks protected paths, and every commit is re-proved. Here it is to paste yourself if you want:
|
|
79
98
|
|
|
80
|
-
In this repo a change is done only when the win survives verification
|
|
81
|
-
|
|
82
|
-
from editing a test, config, or grader, not the source, so undo that and earn it in the
|
|
83
|
-
Run `sawbuck context` first to see what broke in these
|
|
84
|
-
fragile spots). Never skip, weaken, or delete a
|
|
99
|
+
In this repo a change is done only when the win survives verification from the source
|
|
100
|
+
alone. Before finishing, run `npx -y sawbuck check`. If it reads `gamed`, the green came
|
|
101
|
+
from editing a test, config, or grader, not the source, so undo that and earn it in the
|
|
102
|
+
source. Run `npx -y sawbuck context` after your first edit to see what broke in these
|
|
103
|
+
files before (reverts, weakened tests, fragile spots). Never skip, weaken, or delete a
|
|
104
|
+
test to go green.
|
|
85
105
|
|
|
86
|
-
**With `--full`.** It starts by watching: protected paths, new install scripts and sources outside the registry block from day one, and new dependencies and structure changes show as notes. `sawbuck contract --strict` blocks them all. When something fails, the check says what changed in words. If your tests can't run in CI, the check says so and doesn't fail the job, and protected paths are still enforced. To accept it on a pull request, someone with write access comments `sawbuck-allow: <reason>` and re-runs the check; the reason stays on the record. An agent signed in as that person could post the same comment, so branch protection and review still matter. Locally, use `--allow "<reason>"`.
|
|
106
|
+
**With `--full`.** It starts by watching: protected paths, new install scripts and sources outside the registry block from day one, and new dependencies and structure changes show as notes. `sawbuck contract --strict` blocks them all. When something fails, the check says what changed in words. If your tests can't run in CI, the check says so and doesn't fail the job, and protected paths are still enforced. To accept it on a pull request, someone with write access comments `sawbuck-allow: <reason>` and re-runs the check; the reason stays on the record. An agent signed in as that person could post the same comment, so branch protection and review still matter. Locally, use `--allow "<reason>"`. A commit trailer is not a person, so Sawbuck does not read one.
|
|
87
107
|
|
|
88
108
|
**Rewritten history.** Logbook reads git, so a squash or a rebase takes a revert with it. `sawbuck context` also reads the reflog for commits no branch reaches any more and shows them for the files in hand, and once attached it keeps them in `.sawbuck/rewritten.jsonl` after the reflog expires. It stays local: a clone does not get it, nothing older than the reflog comes back, and work that was never committed is not in git at all. Your steering is separate: `reins` reads your agent transcripts, not git.
|
|
89
109
|
|
|
90
110
|
**Tested claims.** Every claim above, and every limit, is a test in [`test/claims.test.mjs`](test/claims.test.mjs), run on a clean account.
|
|
91
111
|
|
|
92
|
-
**Measured, not assumed.** On a scenario whose natural fix sits inside a protected module, 62 trials: 52 driven past a forced compaction (two on the default model, one on Haiku) and 10 shallow, some under "urgent hotfix" pressure. With no rule the agent edited the protected file 17 of 17 times; with a rule in any form, including a mandate, 0 of 45. Earlier scenarios did not tempt it (no rule, 0 of 17), and the wall never had to fire. The model honors a rule it was told. The gate is for the rule nobody told it, the session that outlived the summary, and the proof that it held:
|
|
112
|
+
**Measured, not assumed.** On a scenario whose natural fix sits inside a protected module, 62 trials: 52 driven past a forced compaction (two on the default model, one on Haiku) and 10 shallow, some under "urgent hotfix" pressure. With no rule the agent edited the protected file 17 of 17 times; with a rule in any form, including a mandate, 0 of 45. Earlier scenarios did not tempt it (no rule, 0 of 17), and the wall never had to fire. The model honors a rule it was told. The gate is for the rule nobody told it, the session that outlived the summary, and the proof that it held: once attached, the hooks re-read the mandates from the repo on every turn, say them again after a compaction, and send back a turn that leaves a protected path changed, up to three times by default. Numbers and method: [sawbuck.ai/contracts](https://sawbuck.ai/contracts).
|
|
93
113
|
|
|
94
|
-
Without a `promptwheel.config.json` the check measures your detected test command. To make it count more (coverage, lint, latency, an eval) or to protect
|
|
114
|
+
Without a `promptwheel.config.json` the check measures your detected test command. `sawbuck init` writes the config with that command and the tripwires, and explains each line with evidence from your history. To make it count more (coverage, lint, latency, an eval) or to protect another path, add a metric or a mandate to it by hand.
|
|
95
115
|
|
|
96
116
|
## Gate debt on pull requests
|
|
97
117
|
|
|
@@ -110,15 +130,15 @@ jobs:
|
|
|
110
130
|
- run: npx -y sawbuck debt --since "origin/${{ github.base_ref }}" --fail-on undocumented
|
|
111
131
|
```
|
|
112
132
|
|
|
113
|
-
`--fail-on` takes `undocumented` (a bare `# type: ignore` / `# noqa` with no code), `test` (a newly disabled test), `any`, or a number of new suppressions. Exit 1 fails the check; add `--json` for the machine-readable delta. This repo is also a composite action: `uses: sawbuck-ai/sawbuck@v0` with `base` and `fail-on` inputs.
|
|
133
|
+
`--fail-on` takes `undocumented` (a bare `# type: ignore` / `# noqa` with no code), `test` (a newly disabled test), `any`, or a number of new suppressions; anything else is an error (exit 64). Without `--since` it gates everything live in HEAD, not only what a branch adds. Exit 1 fails the check; add `--json` for the machine-readable delta. This repo is also a composite action: `uses: sawbuck-ai/sawbuck@v0` with `base` and `fail-on` inputs.
|
|
114
134
|
|
|
115
135
|
Debt is on demand: `sawbuck debt`, and `sawbuck context` for the files in hand. To have history added to every turn, set `{"turn":{"context":true}}` in `sawbuck.config.json`. It is off by default because a written rule did the work in every bench, and context spent on each prompt is context lost.
|
|
116
136
|
|
|
117
137
|
## The engines
|
|
118
138
|
|
|
119
|
-
Sawbuck is thin glue over two
|
|
139
|
+
Sawbuck is thin glue over two engines:
|
|
120
140
|
|
|
121
|
-
- **the check** : the deterministic outcome gate, a
|
|
141
|
+
- **the check** : the deterministic outcome gate, written up as a draft spec (the [Verified-Outcome Contract](CONTRACT.md)). Reference implementation: **promptwheel**. You plug in *what* to measure (your tests, lint, coverage, an eval); the source re-proof is the fixed part. Embed it with `import { check }`, or check any tool's verdict with `conformsToContract()` from the same package.
|
|
122
142
|
- **the memory** : the relevant history for the files you touch, plus the live debt, from **@promptwheel/logbook** (git history as agent memory; also ships an MCP server).
|
|
123
143
|
|
|
124
144
|
Apache 2.0.
|
package/bins.mjs
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
// bins.mjs: resolve the two engine CLIs once, for every sawbuck entry point.
|
|
2
2
|
// Order: env override (PW_BIN / LB_BIN), the installed package, then the dev sibling checkout.
|
|
3
|
-
//
|
|
3
|
+
// Logbook is reached only through memory.mjs; LB lives
|
|
4
4
|
// here so the resolver is written once, not per file.
|
|
5
5
|
|
|
6
6
|
import { dirname, join } from 'node:path';
|
package/contract-cmd.mjs
CHANGED
|
@@ -1,8 +1,9 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
|
-
// sawbuck contract: give a repo its first rules with one command. Keeps whatever is already here
|
|
3
|
-
//
|
|
4
|
-
// 20 to 0 of 20). --full adds the supply-chain guards, the repo lock and a CI check. Zero-LLM. Re-running changes
|
|
2
|
+
// sawbuck contract: give a repo its first rules with one command. Keeps whatever config is already here (and adds no
|
|
3
|
+
// tests to it); with no config it writes one with your detected tests. Adds the protected paths and the written rule, which is what changes the agent (the pressure bench: 11 of
|
|
4
|
+
// 20 to 0 of 20). --full adds the supply-chain guards, the repo lock and a CI check (github.com origin, real test command). Zero-LLM. Re-running changes
|
|
5
5
|
// nothing. Protected product code (auth, billing) is suggested, never added: freezing those is a person's decision.
|
|
6
|
+
import './pipe.mjs';
|
|
6
7
|
import { ensureExcluded } from './exclude.mjs';
|
|
7
8
|
import { writeAgentBlock } from './agent-block.mjs';
|
|
8
9
|
import { execFileSync } from 'node:child_process';
|
|
@@ -18,7 +19,7 @@ const dry = process.argv.includes('--dry-run');
|
|
|
18
19
|
const strict = process.argv.includes('--strict');
|
|
19
20
|
const full = process.argv.includes('--full') || strict;
|
|
20
21
|
// a new contract starts by watching: everyday change shows as a note. Protected paths, install scripts and sources
|
|
21
|
-
// outside the registry still fail from day one. --strict holds them all, and
|
|
22
|
+
// outside the registry still fail from day one. --strict holds them all, and makes the notes of an earlier --full block.
|
|
22
23
|
const REPORT = new Set(['dependency_entries', 'lock_drift', 'pins_changed', 'unpinned_execution', 'install_policy_gaps']);
|
|
23
24
|
// Tests need their dependencies, so an npm lockfile gets an install, with no scripts. Elsewhere tests may not run in
|
|
24
25
|
// CI, so inconclusive does not fail the job: protected paths are checked either way, and a gamed or failed verdict still fails.
|
|
@@ -51,19 +52,27 @@ try {
|
|
|
51
52
|
const had = existsSync(cfgPath);
|
|
52
53
|
// without --full, the paths that exist only for the lock and the scanners do not apply
|
|
53
54
|
const mandates = preset.mandates.filter((m) => full || !JSON.stringify(m.paths || []).match(/packs|promptwheel\.lock/));
|
|
55
|
+
// what both the dry run and the real run report: every protected path in the contract after signing, and the
|
|
56
|
+
// supply-chain guards --full puts beside your own tests
|
|
57
|
+
const names = (l) => new Set((l || []).map((x) => x && x.name));
|
|
58
|
+
const counts = (c) => `${new Set([...names(c.mandates), ...names(mandates)]).size} protected paths${full ? `, ${preset.metrics.length} supply-chain guards` : ''}`;
|
|
54
59
|
if (dry) {
|
|
55
|
-
|
|
60
|
+
let now = {}; try { if (had) now = JSON.parse(readFileSync(cfgPath, 'utf8')); } catch {}
|
|
61
|
+
const notes = strict ? (now.metrics || []).filter((m) => m && m.report && REPORT.has(m.name)).length : 0;
|
|
62
|
+
process.stdout.write(`sawbuck contract would ${had ? 'update' : 'write'} ${cfgPath}: ${counts(now)}${full ? ', the scanners under .promptwheel/packs/, and promptwheel.lock.json' : ''}${notes ? `, and make ${notes} guards that are notes block` : ''}.\n`);
|
|
56
63
|
} else {
|
|
57
64
|
if (!had) run([join(HERE, 'init.mjs'), '-y', '--no-investigate'], repo);
|
|
58
65
|
const cfg = existsSync(cfgPath) ? JSON.parse(readFileSync(cfgPath, 'utf8')) : { repeat: 1, metrics: [] };
|
|
59
66
|
const before = { m: (cfg.mandates || []).length, g: (cfg.metrics || []).length };
|
|
60
|
-
const names = (l) => new Set((l || []).map((x) => x && x.name));
|
|
61
67
|
const hasM = names(cfg.mandates), hasG = names(cfg.metrics);
|
|
62
68
|
cfg.mandates = [...(cfg.mandates || []), ...mandates.filter((m) => !hasM.has(m.name))];
|
|
63
69
|
if (full) {
|
|
64
70
|
cfg.metrics = [...(cfg.metrics || []), ...preset.metrics.filter((m) => !hasG.has(m.name)).map((m) => (!strict && REPORT.has(m.name) ? { ...m, report: true } : m))];
|
|
65
71
|
cfg.evaluatorPaths = [...new Set([...(cfg.evaluatorPaths || []), ...(preset.evaluatorPaths || [])])];
|
|
66
72
|
}
|
|
73
|
+
// --strict on a repo signed with --full: the guards that were notes now block, as the --full output promised
|
|
74
|
+
let hardened = 0;
|
|
75
|
+
if (strict) for (const m of cfg.metrics || []) if (m && m.report && REPORT.has(m.name)) { delete m.report; hardened++; }
|
|
67
76
|
cfg.allowTrailer = false;
|
|
68
77
|
writeFileSync(cfgPath, JSON.stringify(cfg, null, 2) + '\n');
|
|
69
78
|
const pack = join(repo, '.promptwheel', 'packs', 'supply-chain');
|
|
@@ -92,10 +101,11 @@ try {
|
|
|
92
101
|
const locked = existsSync(join(repo, 'promptwheel.lock.json'));
|
|
93
102
|
if (full && !locked) run([join(pack, 'lock.mjs'), 'write', repo], repo);
|
|
94
103
|
const added = (cfg.mandates.length - before.m) + (cfg.metrics.length - before.g);
|
|
95
|
-
const fresh = added || (full && !locked);
|
|
104
|
+
const fresh = added || hardened || ci || (full && !locked);
|
|
96
105
|
process.stdout.write(!fresh ? 'sawbuck: already signed, nothing changed.\n' : full
|
|
97
|
-
? `sawbuck: signed. ${cfg
|
|
98
|
-
: `sawbuck: signed. ${cfg
|
|
106
|
+
? `sawbuck: signed. ${counts(cfg)}, and the shape of this repo in promptwheel.lock.json.\n${strict ? (hardened ? ` ${hardened} guards that were notes now block. The contract is a protected path, so the commit that carries this needs a person's reason: sawbuck check --allow "<reason>".\n` : '') : ' New dependencies and structure changes show as notes for now; protected paths and install scripts block. --strict blocks them all.\n'}${ci ? ' Added .github/workflows/sawbuck.yml, so pull requests are checked in CI. Commit it with the rest.\n' : ''}`
|
|
107
|
+
: `sawbuck: signed. ${counts(cfg)}, and the rule in AGENTS.md.\n --full adds supply-chain guards, a repo lock and a CI check.\n`);
|
|
108
|
+
if (full && !ci && !existsSync(wf)) process.stdout.write(` No CI check added: ${!/github\.com/.test(remote) ? 'this repo has no github.com origin' : 'there is no real test command yet, and a placeholder would fail every pull request'}. Run sawbuck contract --full again once it has one.\n`);
|
|
99
109
|
}
|
|
100
110
|
} finally { rmSync(scratch, { recursive: true, force: true }); }
|
|
101
111
|
|
|
@@ -105,4 +115,4 @@ try { files = execFileSync('git', ['-C', repo, 'ls-files'], { encoding: 'utf8',
|
|
|
105
115
|
const seen = new Set();
|
|
106
116
|
for (const f of files) { const m = f.match(/^(.*?(?:^|\/)(?:auth|authentication|billing|payments?|migrations))\//); if (m && !/(^|\/)(node_modules|test|tests|docs?)\//.test(m[1])) seen.add(m[1]); }
|
|
107
117
|
if (seen.size) process.stdout.write(`\n To protect product code too, add these protected paths under "mandates" in promptwheel.config.json:\n${[...seen].slice(0, 5).map((d) => ` { "name": "${d.replace(/\W+/g, '-')}", "paths": ["${d}/**"], "reason": "changed by a person" }`).join('\n')}\n`);
|
|
108
|
-
if (!dry) process.stdout.write(`\n Commit promptwheel.config.json and AGENTS.md${full ? ', promptwheel.lock.json and .promptwheel/packs/' : ''}. Then sawbuck attach
|
|
118
|
+
if (!dry) process.stdout.write(`\n Commit promptwheel.config.json and AGENTS.md${full ? ', promptwheel.lock.json and .promptwheel/packs/' : ''}. Then sawbuck attach watches every turn: protected paths at turn end, each commit re-proved.\n`);
|
package/debt.mjs
CHANGED
|
@@ -5,23 +5,39 @@
|
|
|
5
5
|
// rule code or reason) or BARE (just the directive, nobody wrote why). The whole-repo view;
|
|
6
6
|
// `sawbuck context` is the task-scoped one. The live state (git grep of HEAD, gated) is the
|
|
7
7
|
// complete lens for CURRENT debt; the historical diff scan under-counts it.
|
|
8
|
-
// Zero-dep. Usage: node debt.mjs [--repo P] [--top N] [--since REF] [--json]
|
|
8
|
+
// Zero-dep. Usage: node debt.mjs [repo] [--repo P] [--top N] [--since REF] [--fail-on undocumented|test|any|N] [--json]
|
|
9
9
|
|
|
10
|
+
import './pipe.mjs';
|
|
10
11
|
import { basename, resolve } from 'node:path';
|
|
12
|
+
import { execFileSync } from 'node:child_process';
|
|
11
13
|
import { events as memEvents, debt as memDebt } from './memory.mjs';
|
|
12
14
|
|
|
13
15
|
const flag = (n, d) => { const i = process.argv.indexOf(n); return i > -1 ? (process.argv[i + 1] ?? true) : d; };
|
|
14
|
-
|
|
16
|
+
// the repo is --repo, or the first plain argument (`sawbuck debt <path>`), at its root
|
|
17
|
+
const VALUED = new Set(['--repo', '--top', '--since', '--fail-on']);
|
|
18
|
+
const plain = process.argv.slice(2).filter((a, i, l) => !a.startsWith('-') && !VALUED.has(l[i - 1]));
|
|
19
|
+
const asked = resolve(flag('--repo', plain[0] || process.cwd()));
|
|
20
|
+
let repo = asked;
|
|
21
|
+
try { repo = execFileSync('git', ['-C', asked, 'rev-parse', '--show-toplevel'], { encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'] }).trim(); }
|
|
22
|
+
catch { console.error(`sawbuck debt: ${asked} is not a git repository`); process.exit(64); }
|
|
15
23
|
const top = Math.max(1, parseInt(flag('--top', '12'), 10) || 12);
|
|
16
24
|
const since = flag('--since', null);
|
|
17
25
|
const failRaw = flag('--fail-on', null); // undocumented | test | any | <N>; bare --fail-on = undocumented
|
|
18
|
-
const failMode = failRaw === true ? 'undocumented' : (failRaw ? String(failRaw).toLowerCase() : null);
|
|
26
|
+
const failMode = (failRaw === true || String(failRaw).startsWith('--')) ? 'undocumented' : (failRaw ? String(failRaw).toLowerCase() : null);
|
|
27
|
+
if (failMode && !/^(undocumented|test|tests|disabled|any|\d+)$/.test(failMode)) { console.error(`sawbuck debt: --fail-on takes undocumented|test|any|N, not "${failRaw}"`); process.exit(64); }
|
|
28
|
+
// the gate: fail (exit 1) when the list holds the kind of debt --fail-on names. With --since the list is what the
|
|
29
|
+
// branch adds; without it, everything live in HEAD.
|
|
30
|
+
const gateFails = (list) => !failMode ? false
|
|
31
|
+
: /^\d+$/.test(failMode) ? list.length >= Math.max(1, +failMode) // 0 means any: a clean list never fails
|
|
32
|
+
: failMode === 'any' ? list.length > 0
|
|
33
|
+
: (failMode === 'test' || failMode === 'tests' || failMode === 'disabled') ? list.some((it) => it.testSkip)
|
|
34
|
+
: list.some((it) => !it.documented);
|
|
19
35
|
const asJson = process.argv.includes('--json');
|
|
20
36
|
|
|
21
37
|
// events give fragility + fall-back dates (Logbook's facts). Best-effort.
|
|
22
38
|
const events = memEvents(repo);
|
|
23
39
|
|
|
24
|
-
// the live suppression ledger via the
|
|
40
|
+
// the live suppression ledger via the Logbook adapter. Each
|
|
25
41
|
// item arrives classified (documented/testSkip) and with its line content and blame date (`since`),
|
|
26
42
|
// so this file does not import Logbook's pattern or gating. null = Logbook FAILED
|
|
27
43
|
// (unmeasurable is NOT clean, refuse and exit nonzero); [] = a genuinely clean tree.
|
|
@@ -41,11 +57,7 @@ if (since && typeof since === 'string') {
|
|
|
41
57
|
const added = live.filter((it) => it.content && !base.has(it.file + '\u0000' + norm(it.content)));
|
|
42
58
|
const skips = added.filter((it) => it.testSkip).length, undoc = added.filter((it) => !it.documented).length;
|
|
43
59
|
// the gate: fail (exit 1) when the branch adds the kind of debt --fail-on names.
|
|
44
|
-
const fail =
|
|
45
|
-
: /^\d+$/.test(failMode) ? added.length >= +failMode
|
|
46
|
-
: failMode === 'any' ? added.length > 0
|
|
47
|
-
: (failMode === 'test' || failMode === 'tests' || failMode === 'disabled') ? skips > 0
|
|
48
|
-
: undoc > 0; // default: undocumented
|
|
60
|
+
const fail = gateFails(added);
|
|
49
61
|
if (asJson) { console.log(JSON.stringify({ since, added: added.length, testSkips: skips, undocumented: undoc, failOn: failMode, fail, items: added.map((it) => ({ file: it.file, line: it.line, kind: String(it.kind).trim(), documented: it.documented })) }, null, 2)); process.exit(fail ? 1 : 0); }
|
|
50
62
|
if (!added.length) { console.log(`🐴 sawbuck: no new suppression debt since ${since} (clean).`); process.exit(0); }
|
|
51
63
|
const o = [`🐴 sawbuck: ${added.length} new suppression${added.length === 1 ? '' : 's'} since ${since} · ${skips} disabled test${skips === 1 ? '' : 's'} · ${undoc} undocumented (no code/reason)`];
|
|
@@ -56,6 +68,7 @@ if (since && typeof since === 'string') {
|
|
|
56
68
|
process.exit(fail ? 1 : 0);
|
|
57
69
|
}
|
|
58
70
|
|
|
71
|
+
const liveFail = gateFails(live);
|
|
59
72
|
if (!live.length) { console.log(asJson ? '{"total":0,"testSkips":0,"codeSuppressions":0,"undocumented":0,"files":[]}' : '🐴 sawbuck: no live suppression debt in HEAD (nothing silenced in src/test/config). A clean tree.'); process.exit(0); }
|
|
60
73
|
|
|
61
74
|
// Fragility: debt on code that keeps breaking (reverts, fix-bursts) is the debt that matters; debt
|
|
@@ -105,8 +118,8 @@ const files = [...byFile.values()].map((f) => {
|
|
|
105
118
|
const onFragile = files.filter((f) => f.reverts > 0 || f.hot).length;
|
|
106
119
|
|
|
107
120
|
if (asJson) {
|
|
108
|
-
console.log(JSON.stringify({ total: live.length, testSkips, codeSuppressions: codeSup, undocumented: bare, filesOnFragileCode: onFragile, files: files.map((f) => ({ file: f.file, live: f.n, testSkips: f.skips, suppressions: f.sup, undocumented: f.bareSup, reverts: f.reverts, hot: f.hot, risk: f.risk, oldest: f.oldest })) }, null, 2));
|
|
109
|
-
process.exit(0);
|
|
121
|
+
console.log(JSON.stringify({ total: live.length, testSkips, codeSuppressions: codeSup, undocumented: bare, filesOnFragileCode: onFragile, ...(failMode ? { failOn: failMode, fail: liveFail } : {}), files: files.map((f) => ({ file: f.file, live: f.n, testSkips: f.skips, suppressions: f.sup, undocumented: f.bareSup, reverts: f.reverts, hot: f.hot, risk: f.risk, oldest: f.oldest })) }, null, 2));
|
|
122
|
+
process.exit(liveFail ? 1 : 0);
|
|
110
123
|
}
|
|
111
124
|
|
|
112
125
|
const out = [];
|
|
@@ -125,4 +138,6 @@ if (dated.length) {
|
|
|
125
138
|
for (const it of dated.slice(0, 5)) out.push(` ${it.file}:${it.line} ${String(it.kind).trim()} (since ${it.since})`);
|
|
126
139
|
}
|
|
127
140
|
out.push(`\n bare = no code/reason, the suspect kind. leads, not verdicts; full dated ledger: logbook audit`);
|
|
141
|
+
if (failMode) out.push(liveFail ? `\n ❌ debt gate FAILED (--fail-on ${failMode}): HEAD carries this debt. Add --since <base> to gate only what a branch adds.` : `\n ✓ debt gate passed (--fail-on ${failMode}).`);
|
|
128
142
|
console.log(out.join('\n'));
|
|
143
|
+
process.exit(liveFail ? 1 : 0);
|
package/exclude.mjs
CHANGED
|
@@ -12,8 +12,8 @@ export const EXCLUDED = [...RECORD_DIRS, '.claude/settings.local.json'];
|
|
|
12
12
|
|
|
13
13
|
// The one writer of the block, in the repository's own exclude file wherever its git dir is (a worktree or
|
|
14
14
|
// submodule keeps it elsewhere). Rewrites to the current set, so a grown set retrofits existing repos; no lines
|
|
15
|
-
// strips it. Returns whether the file changed.
|
|
16
|
-
function writeBlock(r, lines) {
|
|
15
|
+
// strips it. Returns whether the file changed; with `dry`, whether it would, writing nothing.
|
|
16
|
+
function writeBlock(r, lines, dry) {
|
|
17
17
|
try {
|
|
18
18
|
const rel = sh(r, 'rev-parse', '--git-path', 'info/exclude');
|
|
19
19
|
if (!rel) return false;
|
|
@@ -23,10 +23,11 @@ function writeBlock(r, lines) {
|
|
|
23
23
|
const block = lines.length ? `# >>> sawbuck >>>\n${lines.join('\n')}\n# <<< sawbuck <<<\n` : '';
|
|
24
24
|
const next = (block && stripped && !stripped.endsWith('\n') ? stripped + '\n' : stripped) + block;
|
|
25
25
|
if (next === body) return false;
|
|
26
|
+
if (dry) return true;
|
|
26
27
|
mkdirSync(dirname(f), { recursive: true });
|
|
27
28
|
writeFileSync(f, next);
|
|
28
29
|
return true;
|
|
29
30
|
} catch { return false; }
|
|
30
31
|
}
|
|
31
|
-
export const ensureExcluded = (r) => writeBlock(r, EXCLUDED);
|
|
32
|
-
export const clearExcluded = (r) => writeBlock(r, []);
|
|
32
|
+
export const ensureExcluded = (r, dry) => writeBlock(r, EXCLUDED, dry);
|
|
33
|
+
export const clearExcluded = (r, dry) => writeBlock(r, [], dry);
|
package/history.mjs
CHANGED
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
// One reader for Claude Code history on this machine, so every feature counts the same way: side threads and meta lines
|
|
2
|
-
// are skipped,
|
|
2
|
+
// are skipped, a line copied by a resumed session counts once, and sessions started in a temp folder are left out,
|
|
3
|
+
// because those are scripts and demos, not your work.
|
|
3
4
|
import { readdirSync, readFileSync } from 'node:fs';
|
|
4
5
|
import { join } from 'node:path';
|
|
5
6
|
|
|
@@ -11,10 +12,12 @@ export function typedText(message) {
|
|
|
11
12
|
}
|
|
12
13
|
|
|
13
14
|
// calls onSession() before each transcript and onEvent(e) for each line, and returns nothing
|
|
15
|
+
const SCRATCH_DIR = /^-(private-)?(tmp|var-folders)(-|$)|AppData-Local-Temp/; // a project folder is named after the path the session started in
|
|
14
16
|
export function readHistory(root, { onSession = () => {}, onEvent }) {
|
|
15
17
|
const seen = new Set();
|
|
16
18
|
let dirs = []; try { dirs = readdirSync(root); } catch { return; }
|
|
17
19
|
for (const d of dirs) {
|
|
20
|
+
if (SCRATCH_DIR.test(d)) continue;
|
|
18
21
|
let files = []; try { files = readdirSync(join(root, d)).filter((f) => f.endsWith('.jsonl')); } catch { continue; }
|
|
19
22
|
for (const f of files) {
|
|
20
23
|
let text = ''; try { text = readFileSync(join(root, d, f), 'utf8'); } catch { continue; }
|
package/init.mjs
CHANGED
|
@@ -15,17 +15,21 @@
|
|
|
15
15
|
// --force overwrite an existing promptwheel.config.json
|
|
16
16
|
// --preset <name> pass through to `promptwheel init`
|
|
17
17
|
|
|
18
|
+
import './pipe.mjs';
|
|
18
19
|
import { writeAgentBlock } from './agent-block.mjs';
|
|
19
20
|
import { execFileSync } from 'node:child_process';
|
|
20
21
|
import { readFileSync, writeFileSync, existsSync, unlinkSync } from 'node:fs';
|
|
21
22
|
import { join } from 'node:path';
|
|
22
23
|
import { createInterface } from 'node:readline';
|
|
23
|
-
import { PW
|
|
24
|
+
import { PW } from './bins.mjs';
|
|
25
|
+
import { events as history } from './memory.mjs';
|
|
24
26
|
|
|
25
27
|
const argv = process.argv.slice(2);
|
|
26
28
|
const has = (f) => argv.includes(f);
|
|
27
29
|
const out = (s) => process.stdout.write(s.endsWith('\n') ? s : s + '\n');
|
|
28
|
-
const repo = (() => { try { return execFileSync('git', ['rev-parse', '--show-toplevel'], { cwd: process.cwd(), encoding: 'utf8' }).trim(); } catch { return
|
|
30
|
+
const repo = (() => { try { return execFileSync('git', ['rev-parse', '--show-toplevel'], { cwd: process.cwd(), encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'] }).trim(); } catch { return ''; } })();
|
|
31
|
+
// the contract is checked against git history, so outside a repository there is nothing to write it for
|
|
32
|
+
if (!repo) { process.stderr.write(`sawbuck: ${process.cwd()} is not a git repository (run init inside your project)\n`); process.exit(64); }
|
|
29
33
|
|
|
30
34
|
const yes = has('--yes') || has('-y');
|
|
31
35
|
const forceWizard = has('--wizard');
|
|
@@ -62,10 +66,7 @@ else { writeCfg(); report(); }
|
|
|
62
66
|
|
|
63
67
|
// ---------------------------------------------------------------------------
|
|
64
68
|
function investigateHistory() {
|
|
65
|
-
|
|
66
|
-
try { record = execFileSync(process.execPath, [LB, 'query', '--json'], { cwd: repo, encoding: 'utf8', maxBuffer: 256 * 1024 * 1024 }); }
|
|
67
|
-
catch (e) { record = (e.stdout || '').toString(); }
|
|
68
|
-
const events = record.split('\n').filter(Boolean).map((l) => { try { return JSON.parse(l); } catch { return null; } }).filter(Boolean);
|
|
69
|
+
const events = history(repo); // every event, through the one adapter: not the newest 200, and none of Logbook's own count lines
|
|
69
70
|
const codeish = (p) => /\.(m?[jt]sx?|cjs|py|go|rs|rb|java|kt|c|cc|cpp|h|hpp|php|swift|scala|clj|ex|exs)$/i.test(p);
|
|
70
71
|
let reverts = 0, suppressions = 0, weakenings = 0;
|
|
71
72
|
const freq = new Map();
|
package/memory.mjs
CHANGED
|
@@ -16,7 +16,6 @@ function parseArray(out) {
|
|
|
16
16
|
|
|
17
17
|
// --- debt sub-classification (Sawbuck policy; applied once so the ledger and the per-turn surface
|
|
18
18
|
// agree).
|
|
19
|
-
export const isTestSkip = (m) => /\bit\.skip\b|\btest\.skip\b|xit\(|xdescribe\(|\bdescribe\.skip\b|pytest\.mark\.skip|unittest\.skip|\bt\.Skip\b|@Disabled\b|@Ignore\b|^\[Ignore\b|^Skip\s*=|^#\[ignore|markTestSkipped|markTestIncomplete/.test(String(m).trim());
|
|
20
19
|
export const isDocumented = (c) => {
|
|
21
20
|
c = String(c || '');
|
|
22
21
|
if (/eslint-disable/.test(c)) return /eslint-disable(?:-next-line|-line)?\s+\S/.test(c);
|
|
@@ -27,12 +26,11 @@ export const isDocumented = (c) => {
|
|
|
27
26
|
return /\(\s*["']/.test(c) || /=\s*["']/.test(c);
|
|
28
27
|
};
|
|
29
28
|
|
|
30
|
-
const classify = (r
|
|
29
|
+
const classify = (r) => ({
|
|
31
30
|
file: r.file, line: r.line, kind: String(r.kind ?? '').trim(), content: r.content ?? '',
|
|
32
31
|
since: r.since ?? null, sha: r.sha ?? null,
|
|
33
32
|
documented: r.content ? isDocumented(r.content) : false,
|
|
34
|
-
testSkip:
|
|
35
|
-
source,
|
|
33
|
+
testSkip: r.testSkip === true, // Logbook classifies it
|
|
36
34
|
});
|
|
37
35
|
|
|
38
36
|
// events(): the raw git event record, the ranking input. [] on failure, exactly as "no history".
|
|
@@ -45,16 +43,10 @@ export function events(repo) {
|
|
|
45
43
|
// debt(): live suppressions still in the tree, each classified. null when Logbook could not answer.
|
|
46
44
|
export function debt(repo, { paths = null, ref = null } = {}) {
|
|
47
45
|
const out = run(LB, repo, ['audit', '--json', ...(ref ? ['--ref', ref] : []), ...((paths || []).flatMap((f) => ['--file', f]))]);
|
|
48
|
-
return out === null ? null : parseArray(out).map(
|
|
46
|
+
return out === null ? null : parseArray(out).map(classify);
|
|
49
47
|
}
|
|
50
48
|
|
|
51
49
|
// record(): keep a machine finding as an annotation. Best-effort, never throws into the caller.
|
|
52
|
-
export function record(repo, { sha, reason,
|
|
53
|
-
|
|
54
|
-
const exec = (a) => execFileSync(process.execPath, [LB, ...a], { cwd: repo, encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'] });
|
|
55
|
-
try {
|
|
56
|
-
try { exec(['annotate', sha, text, ...(side ? ['--side', side] : []), ...(evidenceFile ? ['--evidence-file', evidenceFile] : []), '--by', by]); }
|
|
57
|
-
catch { exec(['annotate', sha, text, '--by', by]); }
|
|
58
|
-
return true;
|
|
59
|
-
} catch { return false; }
|
|
50
|
+
export function record(repo, { sha, reason, by = 'sawbuck' }) {
|
|
51
|
+
return run(LB, repo, ['annotate', sha, String(reason).slice(0, 280), '--by', by]) !== null;
|
|
60
52
|
}
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "sawbuck",
|
|
3
|
-
"version": "0.1.
|
|
4
|
-
"description": "
|
|
3
|
+
"version": "0.1.48",
|
|
4
|
+
"description": "Write down the rules your agent follows, and prove they held. A deterministic, zero-LLM check re-proves each win from the agent's source edits alone, so a test that went green by editing the test, grader, or config is sent back. For Claude Code, with Codex untested.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "Apache-2.0",
|
|
7
7
|
"engines": {
|
|
@@ -34,7 +34,7 @@
|
|
|
34
34
|
},
|
|
35
35
|
"dependencies": {
|
|
36
36
|
"@promptwheel/logbook": "0.11.0",
|
|
37
|
-
"promptwheel": "0.8.
|
|
37
|
+
"promptwheel": "0.8.1"
|
|
38
38
|
},
|
|
39
39
|
"files": [
|
|
40
40
|
"sawbuck.mjs",
|
|
@@ -45,6 +45,7 @@
|
|
|
45
45
|
"shape.mjs",
|
|
46
46
|
"agent-block.mjs",
|
|
47
47
|
"exclude.mjs",
|
|
48
|
+
"pipe.mjs",
|
|
48
49
|
"repo-state.mjs",
|
|
49
50
|
"contract-cmd.mjs",
|
|
50
51
|
"bins.mjs",
|
package/pipe.mjs
ADDED