@tduyng/pi-harness-loop 0.0.3 → 0.0.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +73 -119
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -1,138 +1,123 @@
|
|
|
1
1
|
# @tduyng/pi-harness-loop
|
|
2
2
|
|
|
3
|
-
The harness fixes itself.
|
|
3
|
+
The harness fixes itself. It reads what a project's sessions repeated, writes the skill or rule that would have prevented it, installs it, and removes it when the failure rate does not fall.
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
A correction typed in every session lives in a chat message that dies with the session. A rule written by hand is usually worse, because nothing says which rule earns its place and the front of every request grows either way.
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
A session pays nothing until the loop acts. The report is a slash command rather than a tool. A tool would put its schema in the system prompt of every session, and that schema is the tax this package exists to reduce. `lesson` is the one tool, 252 tokens, and it arms only on a request naming a trigger of a lesson still in force.
|
|
8
8
|
|
|
9
|
-
|
|
9
|
+
## Measure the rate, not the story
|
|
10
10
|
|
|
11
|
-
|
|
11
|
+
The loop installs a part, runs the project's check, and counts the sessions after the install against the sessions before it. Every session counts, including the ones that did not hit the failure, because a session that did not fail is the evidence.
|
|
12
12
|
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
The session goes idle. The loop reads the sessions of this project, and if a failure recurred across them it writes a brief with the evidence. One model call turns that brief into the part, and the part is installed. Nothing has to be typed, and the loop never takes the session over.
|
|
13
|
+
The loop takes a part back out at the next idle if its failure rate did not fall, without anyone typing a command.
|
|
16
14
|
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
`PI_HARNESS_LOOP=off` stops all of it.
|
|
20
|
-
|
|
21
|
-
## Commands
|
|
22
|
-
|
|
23
|
-
| Command | Effect |
|
|
15
|
+
| Verdict | Meaning |
|
|
24
16
|
| --- | --- |
|
|
25
|
-
|
|
|
26
|
-
|
|
|
27
|
-
|
|
|
28
|
-
|
|
|
29
|
-
|
|
|
30
|
-
|
|
|
31
|
-
|
|
32
|
-
A report is cleared by the next prompt as well, so one you have read does not sit above the editor for the rest of the session.
|
|
33
|
-
|
|
34
|
-
## What the report shows
|
|
35
|
-
|
|
36
|
-
The harness tree: the skills and agent profiles pi loads, the problems in them, and what the skill block costs on every request. The rules come from pi's own loader, so they do not drift from it.
|
|
17
|
+
| `held` | The rate fell to three quarters of what it was, and sessions still call tools at more than half their old rate |
|
|
18
|
+
| `flat` | The rate stayed within a quarter either way |
|
|
19
|
+
| `worse` | The rate rose by a quarter or more |
|
|
20
|
+
| `avoided` | The rate fell but the work did too, so the failure stopped because the agent did less |
|
|
21
|
+
| `confounded` | A commit touched the failure in the window, so the rate is not the part's |
|
|
22
|
+
| `early` | Fewer than three sessions have run since the install |
|
|
23
|
+
| `out-of-window` | The corpus holds no session from before the install |
|
|
37
24
|
|
|
38
|
-
|
|
39
|
-
- A skill whose frontmatter name differs from its directory is advertised under one name and looked for under another.
|
|
40
|
-
- An agent profile with no description is listed as "no description", so the model cannot choose it.
|
|
41
|
-
- A skill that tells the agent to run a script it does not ship sends it after a file that is not there.
|
|
25
|
+
A `flat`, `worse` or `avoided` install is not the end of the finding. The next brief says the earlier form did not carry and shows both rates, and the part's lesson stops matching. A finding whose part was all markdown and still did not carry is marked `needs code`, and the loop stops proposing prose for it, because the failure lives in tool behaviour and a rule cannot reach it. A finding comes back when the failure grows past the count at the last proposal, and it stops after three.
|
|
42
26
|
|
|
43
|
-
|
|
27
|
+
## The loop
|
|
44
28
|
|
|
45
|
-
The
|
|
29
|
+
The session goes idle. The loop reads the sessions of this project, and a failure that recurred gets a brief with the evidence. One model call turns that brief into the part, the loop checks it against the rules pi's loader applies, and installs it. The loop never takes the session over.
|
|
46
30
|
|
|
47
|
-
|
|
31
|
+
The writer has no tools. It carries the brief, returns files as text, and the loop parses, checks and installs them, so nothing the writer says is executed. A proposal costs one completion, and a part adds to the front of every later request, so both have a ceiling. The scan is cached in `.pi/harness/scan.json` behind a fingerprint of the session files.
|
|
48
32
|
|
|
49
|
-
|
|
33
|
+
`PI_HARNESS_LOOP=off` stops all of it.
|
|
50
34
|
|
|
51
|
-
|
|
35
|
+
```
|
|
36
|
+
/harness-loop the setup, the findings, the cost, the lessons
|
|
37
|
+
/harness-loop propose [id] run a proposal now
|
|
38
|
+
/harness-loop lint only the problems
|
|
39
|
+
/harness-loop revert <id> put the files back by hand
|
|
40
|
+
/harness-loop install <id> install a part already on disk
|
|
41
|
+
/harness-loop clear take the report off the screen
|
|
42
|
+
```
|
|
52
43
|
|
|
53
|
-
|
|
44
|
+
The next prompt clears a report too, so one already read does not sit above the editor.
|
|
54
45
|
|
|
55
|
-
|
|
46
|
+
## What the report covers
|
|
56
47
|
|
|
57
|
-
The
|
|
48
|
+
The report checks the harness tree against pi's own loader rules: a skill with no `description` is dropped at load, a frontmatter name that differs from its directory is advertised under one name and looked for under another, a profile with no description cannot be chosen by the model, a skill that runs a script it does not ship sends the agent after a missing file, and an `extensions/` package that declares an entry the root manifest never loads is a package nobody gets.
|
|
58
49
|
|
|
59
|
-
|
|
50
|
+
Pi advertises every skill by name and description in the system prompt, so a skill nobody opens is a tax on every request. The report lists the advertised ones with the characters they cost and how many are not advertised at all.
|
|
60
51
|
|
|
61
|
-
|
|
52
|
+
The findings are what recurred in this session: the same failure three times, the same command three times, the same file read three times, a run that hit a failure and changed nothing, a tool call with no result. The recurrences are the same shapes across the sessions of this project.
|
|
62
53
|
|
|
63
|
-
|
|
54
|
+
The lessons are the ones whose cue matches the request in the editor, read from `.pi/harness/lessons.jsonl`.
|
|
64
55
|
|
|
65
|
-
|
|
56
|
+
## Three checks before a part lands
|
|
66
57
|
|
|
67
|
-
|
|
58
|
+
Each one is a false positive that would otherwise get installed.
|
|
68
59
|
|
|
69
|
-
|
|
60
|
+
- **Replay**: the failure's own command runs again. When it passes, the failure no longer reproduces and nothing is written. The same command runs after an install and its result stays with the install.
|
|
70
61
|
|
|
71
|
-
|
|
62
|
+
- **Producer**: a failure whose message the tree no longer carries is not a property of the harness. The loop cuts ids, paths and numbers out of the message, searches the tree then the dependencies, and when neither has it and git shows a commit that touched that text after the last session that hit it, the finding is skipped.
|
|
72
63
|
|
|
73
|
-
|
|
64
|
+
- **Variant**: a new error string that is a smaller variant of an edit failure already tried is not a new finding. The edit tool has one matching mechanic, so the 2-session "oldText must be unique" is the 9-session "oldText must match" with a different error string.
|
|
74
65
|
|
|
75
|
-
A
|
|
66
|
+
A defect in one run is noise, a defect in every run is the harness. A signature that reduces to `<path>` or a bare `Script failed` is not a finding, because it names nothing a part can change. A failure also needs independent evidence: when every session that hit it edited a file and each pair shares one, they are one piece of work, and a rule written from it would encode that task rather than the harness.
|
|
76
67
|
|
|
77
|
-
|
|
68
|
+
The loop stands out of the way when something else drives the session: an active goal, or the day's budget of five proposals, stops it.
|
|
78
69
|
|
|
79
|
-
##
|
|
70
|
+
## What a part may be
|
|
80
71
|
|
|
81
|
-
|
|
72
|
+
The brief asks for a skill, an agent profile or a rule, and the writer puts the files in `.pi/harness/<id>/`. The part is read back, checked against the same rules the tree is checked against, and installed.
|
|
82
73
|
|
|
83
|
-
|
|
74
|
+
A part under `global/` holds in every project, and it is installed only when `global` is set, because one project's failure does not get to become every project's rule. Without it the part lands in the project: `.agents/skills/<name>/`, `.pi/agents/<name>.md`, or the project's context file for a rule.
|
|
84
75
|
|
|
85
|
-
A
|
|
76
|
+
The prompt says what each form costs. A skill is paid for by its name and description, about 300 characters on every request, and its body costs nothing until opened. A rule is paid for by its length and applies always. The rule wins when the behaviour has to hold every time, and the skill wins when the situation is specific enough to route on.
|
|
86
77
|
|
|
87
|
-
A
|
|
78
|
+
A part may carry `CONFIRM.md`: one shell command that exits 0 only when the finding can no longer happen, with no file writes and no network. It runs after the project check, and a failing confirmation puts the part back. Without one the part installs on the project check alone, recorded as unconfirmed.
|
|
88
79
|
|
|
89
|
-
## The
|
|
80
|
+
## The check behind an install
|
|
90
81
|
|
|
91
|
-
|
|
82
|
+
The check is the first of `check` in `.pi/harness.json`, the `test` script in `package.json`, or the `test` recipe in `justfile`. The loop installs the part first, so the common case costs one run: when the check passes, the part stays. When it fails, the loop puts the part back and runs the check once more without it, so the verdict is about the difference rather than about a suite that was already red. A rule installs without a check, since no test reads a markdown file.
|
|
92
83
|
|
|
93
|
-
|
|
84
|
+
`maxStandingTokens` is the second ceiling, 1,500 tokens by default. An install that would cross it stays on disk with its brief, and the message names the current total, so the way through is to revert a part that did not carry rather than to raise the cap. A rule is charged in full, a skill only for its name and description. `0` turns the ceiling off.
|
|
94
85
|
|
|
95
|
-
|
|
86
|
+
`/harness-loop revert <install>` puts the files back by hand, newest install first when two touched the same file. A file whose content changed after the install is left alone and named, because an edit is not the loop's to overwrite.
|
|
96
87
|
|
|
97
|
-
## The
|
|
88
|
+
## The lesson that transfers
|
|
98
89
|
|
|
99
|
-
|
|
90
|
+
When a part passes, the session records the cue, the lesson, how to confirm it worked, the evidence, and a confidence. The model may decline with `NO_ENTRY`, and a block with no confirm line is dropped, because a rule nobody can check is a rule of thumb.
|
|
100
91
|
|
|
101
|
-
|
|
92
|
+
A request matches a lesson when it carries two triggers, or one trigger of six characters or more that the project rarely uses. One short word is a coincidence, and `key` is in half the requests of a project that never talks about credentials. A match records a use, so a lesson nobody recalls is visible in the record.
|
|
102
93
|
|
|
103
|
-
|
|
94
|
+
## With the rest of the suite
|
|
104
95
|
|
|
105
|
-
`
|
|
96
|
+
`pi-goal` owns the turns of a session with a goal, so the loop does not propose while one is active and waits to install while one is, and it uses the same check runner so the two never disagree about what a check said. `pi-context` prunes what the model sees, not the session, so findings still see the whole branch. `pi-guard` blocks the shell from reading credentials, and the rule it needs is in the global context, saying what to do instead of what is blocked.
|
|
106
97
|
|
|
107
|
-
##
|
|
98
|
+
## Findings from this machine
|
|
108
99
|
|
|
109
|
-
|
|
100
|
+
- 12 of 19 skills are not advertised at all, because their frontmatter says `disable-model-invocation`, so they cost nothing. The other 7 make a block of 2,652 characters, about 663 tokens on every request.
|
|
110
101
|
|
|
111
|
-
|
|
102
|
+
- 5 of those 7 were opened in none of 26 sessions scanned, costing 362 tokens per request. `tdd` was the sixth until the skill stopped being advertised on 2026-10-26, from a proposal and nothing else.
|
|
112
103
|
|
|
113
|
-
|
|
114
|
-
| --- | --- |
|
|
115
|
-
| `held` | The rate fell to three quarters of what it was, and sessions still call tools at more than half their old rate |
|
|
116
|
-
| `flat` | The rate stayed within a quarter either way |
|
|
117
|
-
| `worse` | The rate rose by a quarter or more |
|
|
118
|
-
| `avoided` | The rate fell but the work did too, so the failure stopped because the agent did less |
|
|
119
|
-
| `confounded` | A commit touched the failure in the window, so the rate is not the part's |
|
|
120
|
-
| `early` | Fewer than three sessions have run since the install |
|
|
121
|
-
| `out-of-window` | The corpus holds no session from before the install |
|
|
104
|
+
- The shell blocked 14 reads of credential files across 2 sessions, and every retry with another shell reached the same block. The rule from it says what to do instead.
|
|
122
105
|
|
|
123
|
-
The
|
|
106
|
+
- The edit tool rejected 9 old texts across 4 sessions, and each rejection reported a closest match the model had paraphrased. The rule from it says to copy old text out of the read that preceded the edit.
|
|
124
107
|
|
|
125
|
-
|
|
108
|
+
- 4 reads were refused for being over 350 lines across 3 sessions, each a file a grep would have narrowed. The rule from it says to grep first.
|
|
126
109
|
|
|
127
|
-
|
|
110
|
+
## Install
|
|
128
111
|
|
|
129
|
-
|
|
112
|
+
```bash
|
|
113
|
+
pi install npm:@tduyng/pi-harness-loop
|
|
114
|
+
```
|
|
130
115
|
|
|
131
|
-
|
|
116
|
+
Local checkout: `pi install /path/to/dotfiles/pi/agent/extensions/pi-harness-loop`
|
|
132
117
|
|
|
133
|
-
|
|
118
|
+
The scans use `rg`, so it has to be on PATH. Sessions of the project must exist on disk, which they do after one run.
|
|
134
119
|
|
|
135
|
-
`.pi/harness.json
|
|
120
|
+
`.pi/harness.json`, every key optional:
|
|
136
121
|
|
|
137
122
|
```json
|
|
138
123
|
{
|
|
@@ -146,42 +131,11 @@ A finding comes back when the failure grows past the count it had at the last pr
|
|
|
146
131
|
}
|
|
147
132
|
```
|
|
148
133
|
|
|
149
|
-
`writerModel`
|
|
150
|
-
|
|
151
|
-
`auto` is read when the session goes idle and stops the loop writing a part on its own, while the manual command still works. `autoInstall` is read when the part is ready, so a project that wants every change by hand sets it to `false` and the part stays on disk.
|
|
152
|
-
|
|
153
|
-
The tokens the provider reported are written to `.pi/harness/spend.jsonl` with the outcome, so a budget of five proposals a day is five completions rather than five sessions. Every install records the characters it adds to the front of the prompt, and the report sums the active ones, so a new proposal knows what the project already pays before choosing a rule over a skill.
|
|
154
|
-
|
|
155
|
-
## What it works with
|
|
156
|
-
|
|
157
|
-
The other extensions drive the same session, so the loop reads their state instead of guessing at it.
|
|
158
|
-
|
|
159
|
-
`pi-goal` owns the turns of a session with a goal, so the loop does not propose while one is active and waits to install while one is. The check runner in `pi-goal` is the loop's runner too, so the two do not disagree about what a check said.
|
|
134
|
+
`writerModel` resolves from the project file, then `PI_HARNESS_WRITER_MODEL`, then the `model` in `subagent.json`, which is the model pi already runs chores on. With none of them the session's own model writes it, and a session on an expensive model pays that model for a short chore. `auto` is read on idle and stops the loop writing a part on its own while the manual command still works. `autoInstall` is read when the part is ready, so a project that wants every change by hand sets it to `false` and the part stays on disk.
|
|
160
135
|
|
|
161
|
-
|
|
136
|
+
## Limits
|
|
162
137
|
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
The measurements come from the sessions on this machine, so they describe this setup.
|
|
168
|
-
|
|
169
|
-
- 12 of 19 skills are not advertised at all, because their frontmatter says `disable-model-invocation`, so they cost nothing. The other 7 make a block of 2,652 characters, about 663 tokens on every request of every project.
|
|
170
|
-
- 5 of those 7 were opened in none of 26 sessions scanned. They cost 1,447 of the 2,652 characters, about 362 tokens per request. `tdd` was the sixth: 39 sessions with no open, and the skill stopped being advertised on 2026-10-26, from a proposal and nothing else. An open is a read of a `SKILL.md` or a `/skill:<name>` command, so a skill reached by hand is not called dead.
|
|
171
|
-
- The shell blocked 14 reads of credential files across 2 sessions, and every retry with another shell reached the same block. The rule that came out of it says what to do instead.
|
|
172
|
-
- The edit tool rejected 9 old texts across 4 sessions, and each rejection reported a closest match the model had paraphrased. The rule that came out of it says to copy old text out of the read that preceded the edit, and to read the file again when an edit is rejected.
|
|
173
|
-
- 4 reads were refused for being over 350 lines across 3 sessions, and each was a file a grep would have narrowed. The rule that came out of it says to grep first.
|
|
174
|
-
|
|
175
|
-
## What it is not
|
|
176
|
-
|
|
177
|
-
It does not score the harness. It measures the rate of the failure a part was written for, which is not the same as the work being better: a session ends, and nothing says whether the work was good. The findings are the part that can be read without a grader, and the rate is the part that says whether the part held.
|
|
178
|
-
|
|
179
|
-
It does not compare two runs of the same task. The comparison it makes is across the sessions of this project, which are different tasks. The same task run twice is what a benchmark does, and what a working session does not.
|
|
180
|
-
|
|
181
|
-
## Install
|
|
182
|
-
|
|
183
|
-
```bash
|
|
184
|
-
pi install npm:@tduyng/pi-harness-loop
|
|
185
|
-
```
|
|
186
|
-
|
|
187
|
-
Local checkout: `pi install /path/to/dotfiles/pi/agent/extensions/pi-harness-loop`
|
|
138
|
+
- Five proposals a day, three installs a day, six turns per proposal, three proposals per finding.
|
|
139
|
+
- It does not score the harness. It measures the rate of the failure a part was written for. Whether the work got better is a different question.
|
|
140
|
+
- It does not compare two runs of the same task. The comparison is across the sessions of this project, which are different tasks.
|
|
141
|
+
- A rule is markdown in a file no test reads, so `/harness-loop lint` is the only gate it passes.
|
package/package.json
CHANGED