@tduyng/pi-autoresearch 0.0.3 → 0.0.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (2) hide show
  1. package/README.md +42 -28
  2. package/package.json +1 -1
package/README.md CHANGED
@@ -1,12 +1,20 @@
1
1
  # @tduyng/pi-autoresearch
2
2
 
3
- An experiment loop you turn on with one command. Measure, read the number, keep or discard, and let the tools handle git, scope, and the noise floor.
3
+ An experiment loop turned on with one command: measure, read the number, keep or discard. The tools handle git, scope and the noise floor.
4
4
 
5
- ## What it costs
5
+ An optimization loop run by hand fails in three predictable ways. The benchmark gets edited until it shows a win. A regression gets committed because the number moved. The log is re-read whole every iteration, at more cost than the change.
6
6
 
7
- Nothing at cold start. No tool, no skill, no system prompt block: the extension adds one slash command, and that is free until typed. The two tools are declared to the model only while `/autoresearch` runs, and removed on `off`, `clear`, or when `max_runs` is reached.
7
+ Nothing at cold start. The extension adds one slash command, and the two tools are declared to the model only while `/autoresearch` runs. The extension removes them on `off`, `clear`, or when `max_runs` is reached.
8
8
 
9
- The saving is the second half of every optimization loop. Editing a benchmark, committing a regression, and re-reading a whole log by hand are the parts that cost turns, and all three are one line of bookkeeping here.
9
+ ## Keep or discard stays with the model
10
+
11
+ No rule can tell whether 1 percent came from the diff, whether the code is now simpler, or whether a clean run was jitter. The tools measure and the model decides. A rule that decided would be wrong in exactly the interesting cases.
12
+
13
+ The measurement is crash-safe on disk: `run_experiment` writes a pending measurement before anything else, so a restart can still log it and a crash mid-iteration loses nothing.
14
+
15
+ The benchmark cannot move under the comparison. The script is hashed, editing it voids the comparison, and `/autoresearch rebaseline` is the deliberate way to accept a new baseline.
16
+
17
+ A keep cannot escape its scope. `scope` globs in the frontmatter bound the files a keep may commit, so an experiment cannot quietly rewrite a file it was not measuring.
10
18
 
11
19
  ## The loop
12
20
 
@@ -14,26 +22,23 @@ The saving is the second half of every optimization loop. Editing a benchmark, c
14
22
  edit -> run_experiment -> read the output -> log_experiment -> repeat
15
23
  ```
16
24
 
17
- `run_experiment` snapshots the working tree, runs `.auto/measure.sh`, parses `METRIC name=value` lines, runs the optional checks, and returns the output, the metrics, the changed files, the noise advice, and the scope violations. It does not touch git history.
18
-
19
- `log_experiment` takes `status: keep | discard | crash | checks_failed` and the model's reasoning in `asi`. A `keep` commits the files the run touched with a conventional subject built from the description. Every other status restores those files. A run inside the noise band, a crash, or a scope violation is reported rather than decided, because a rule cannot know whether the diff caused the gain.
25
+ `run_experiment` snapshots the working tree, runs `.auto/measure.sh`, parses `METRIC name=value` lines, runs the optional checks, and returns the output, the metrics, the changed files, the noise advice and any scope violations. It does not touch git history.
20
26
 
21
- ## What it adds over the usual loop
27
+ `log_experiment` takes `status: keep | discard | crash | checks_failed` and the model's reasoning in `asi`. A `keep` commits the files the run touched with a conventional subject built from the description. Every other status restores those files.
22
28
 
23
- | Mechanism | Why |
29
+ | Mechanism | Effect |
24
30
  | --- | --- |
25
- | Model owns keep or discard | A rule cannot tell whether 1% came from the diff, or whether simpler code is worth 1% |
26
31
  | Full benchmark output | Phase timings and errors reach the model, so the next hypothesis comes from data |
27
- | Noise floor from median absolute deviation | Advice that separates jitter from a real win, and confidence never overrides the model |
28
- | Pending measurement | One run between the tools, crash-safe on disk, so a restart can still log it |
29
- | Scope globs in frontmatter | A keep cannot touch files outside the declared scope |
30
- | Benchmark script hash | Editing the benchmark to move the number voids the comparison |
32
+ | Noise floor from median absolute deviation | Separates jitter from a real win, without overriding the model |
33
+ | Pending measurement | One run between the tools, crash-safe, so a restart can still log it |
34
+ | Scope globs | A keep cannot touch files outside the declared scope |
35
+ | Benchmark hash | Editing the benchmark to move the number voids the comparison |
31
36
  | Worktree lock and dirty baseline | One loop per worktree, and files another session left dirty are never committed |
32
37
  | Finalize in temporary worktrees | Kept runs become branches without moving HEAD under another session |
33
- | Compaction digest from disk | Prompt, recent runs, dead ends, and ideas survive a reset with no model call |
34
- | Conventional commits | `type(scope): summary`, at most 72 characters, one `Metric:` trailer, built from validated parts |
38
+ | Compaction digest from disk | Prompt, recent runs, dead ends and ideas survive a reset with no model call |
39
+ | Conventional commits | `type(scope): summary`, at most 72 characters, one `Metric:` trailer |
35
40
 
36
- ## Files
41
+ ## The project files
37
42
 
38
43
  `.auto/prompt.md` carries the frontmatter and the objective:
39
44
 
@@ -52,21 +57,19 @@ min_gain: 0.5
52
57
  What is optimized, the constraints, and what a good change looks like.
53
58
  ```
54
59
 
55
- `min_gain` is the smallest relative improvement the advice treats as real, in percent. It never blocks a keep.
60
+ `min_gain` is the smallest relative improvement the advice calls real, in percent. It never blocks a keep.
56
61
 
57
62
  | File | Purpose |
58
63
  | --- | --- |
59
- | `.auto/prompt.md` | Frontmatter config, the objective, and what has been tried |
64
+ | `.auto/prompt.md` | Frontmatter config, the objective, what has been tried |
60
65
  | `.auto/measure.sh` | The benchmark, printing `METRIC name=value` |
61
66
  | `.auto/log.jsonl` | Append-only record |
62
67
  | `.auto/checks.sh` | Optional correctness checks, run after a passing benchmark |
63
- | `.auto/ideas.md` | Optional backlog of ideas not tried |
68
+ | `.auto/ideas.md` | Optional backlog |
64
69
  | `.auto/pending.json` | The run waiting for a decision |
65
70
  | `.auto/lock.json` | The session that owns this worktree |
66
71
 
67
- `.auto/` is added to the repository's `.git/info/exclude` on the first run, including from a linked worktree, so none of it lands in a commit.
68
-
69
- Run it in an interactive session or through RPC. `pi -p` exits as soon as the slash command returns, before the loop's first turn, so a print run cannot drive it.
72
+ The first run adds `.auto/` to `.git/info/exclude`, including from a linked worktree, so none of it lands in a commit.
70
73
 
71
74
  ## Commands
72
75
 
@@ -76,20 +79,20 @@ Run it in an interactive session or through RPC. `pi -p` exits as soon as the sl
76
79
  | `/autoresearch status` | Print the digest |
77
80
  | `/autoresearch off` | Stop and remove the tools |
78
81
  | `/autoresearch rebaseline` | Accept a changed benchmark script as a new baseline |
79
- | `/autoresearch finalize` | Split kept runs into independent branches from the merge-base |
82
+ | `/autoresearch finalize` | Split kept runs into independent branches from the merge base |
80
83
  | `/autoresearch clear` | Delete the log and stop |
81
84
 
82
- `finalize` groups kept runs by file overlap and creates one branch per group from the merge-base in temporary worktrees. The current worktree is never checked out to another branch, and the union of the branches is verified against the original before it reports.
85
+ `finalize` groups kept runs by file overlap and creates one branch per group in temporary worktrees. It never checks the current worktree out to another branch, and it verifies the union of the branches against the original before it reports.
83
86
 
84
87
  ## Worktrees and other sessions
85
88
 
86
- One loop owns a worktree at a time: `/autoresearch` takes `.auto/lock.json` and refuses a second session before it can run anything. Files that were already dirty when the loop started are recorded in the lock and never committed or reverted.
89
+ One loop owns a worktree at a time: `/autoresearch` takes `.auto/lock.json` and refuses a second session before it runs anything. Files that were already dirty when the loop started are recorded in the lock and never committed or reverted.
87
90
 
88
- For a second loop, use a separate git worktree, which gets its own `.auto/`, branch, and log. The first run on trunk creates `autoresearch/<name>-<timestamp>` in a worktree at `~/worktrees/<repo>/autoresearch-<name>-<timestamp>` and leaves the shared checkout on trunk. An existing feature branch is used as it is.
91
+ For a second loop, use a separate git worktree, which gets its own `.auto/`, branch and log. The first run on trunk creates `autoresearch/<name>-<timestamp>` in a worktree at `~/worktrees/<repo>/autoresearch-<name>-<timestamp>` and leaves the shared checkout on trunk. An existing feature branch is used as it is.
89
92
 
90
93
  ## Compaction and restarts
91
94
 
92
- While the loop is active, `session_before_compact` replaces pi's model summary with a deterministic digest built from `.auto/`: the prompt body, recent runs, kept runs, dead ends, ideas, and any pending measurement. No model call, no lost history.
95
+ While the loop is active, `session_before_compact` replaces pi's model summary with a deterministic digest built from `.auto/`: the prompt body, recent runs, kept runs, dead ends, ideas and any pending measurement. No model call, and no history lost to a summary that forgot the metric.
93
96
 
94
97
  After a restart of the same session the loop re-arms from the session's activation entry and sends one short resume message. A fresh session never auto-activates.
95
98
 
@@ -100,3 +103,14 @@ pi install npm:@tduyng/pi-autoresearch
100
103
  ```
101
104
 
102
105
  Local checkout: `pi install /path/to/dotfiles/pi/agent/extensions/pi-autoresearch`
106
+
107
+ It needs a git repository, since a keep commits, the lock is a file, and finalize uses worktrees. The measure script runs through `bash -c`.
108
+
109
+ Run it in an interactive session or through RPC. `pi -p` exits as soon as the slash command returns, before the loop's first turn, so a print run cannot drive it.
110
+
111
+ ## Limits
112
+
113
+ - A run has a default timeout of 600 seconds, and the measure script's own timeout is the one that applies.
114
+ - Auto-resume gives up after 200 turns, and an idle resume after 40.
115
+ - Output is trimmed to the last 40 lines, with 128 KB retained and a 20 MB cap on the temp log.
116
+ - `finalize` reports a conflict instead of resolving one. Splitting the kept runs is a bookkeeping job, not a merge.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@tduyng/pi-autoresearch",
3
- "version": "0.0.3",
3
+ "version": "0.0.4",
4
4
  "description": "Autonomous experiment loop that costs nothing until you type /autoresearch: measure, decide, commit or revert, with scope and benchmark guards.",
5
5
  "keywords": [
6
6
  "pi",