@tduyng/pi-autoresearch 0.0.2 → 0.0.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (2) hide show
  1. package/README.md +68 -64
  2. package/package.json +2 -2
package/README.md CHANGED
@@ -1,47 +1,44 @@
1
1
  # @tduyng/pi-autoresearch
2
2
 
3
- An autonomous experiment loop for pi that costs nothing until you type `/autoresearch`.
4
- Two tools and one log. The model reads the benchmark, decides keep or discard, and the tools do the
5
- bookkeeping: commit, revert, scope, script hash, noise floor, and one line of context back.
3
+ An experiment loop turned on with one command: measure, read the number, keep or discard. The tools handle git, scope and the noise floor.
6
4
 
7
- ## Cold start
5
+ An optimization loop run by hand fails in three predictable ways. The benchmark gets edited until it shows a win. A regression gets committed because the number moved. The log is re-read whole every iteration, at more cost than the change.
8
6
 
9
- Nothing is registered at load: no tool, no skill, no system prompt block. The extension adds one slash
10
- command. The tools are registered and declared to the model only when `/autoresearch` runs, and removed on
11
- `off`, `clear`, or when `max_runs` is reached.
7
+ Nothing at cold start. The extension adds one slash command, and the two tools are declared to the model only while `/autoresearch` runs. The extension removes them on `off`, `clear`, or when `max_runs` is reached.
8
+
9
+ ## Keep or discard stays with the model
10
+
11
+ No rule can tell whether 1 percent came from the diff, whether the code is now simpler, or whether a clean run was jitter. The tools measure and the model decides. A rule that decided would be wrong in exactly the interesting cases.
12
+
13
+ The measurement is crash-safe on disk: `run_experiment` writes a pending measurement before anything else, so a restart can still log it and a crash mid-iteration loses nothing.
14
+
15
+ The benchmark cannot move under the comparison. The script is hashed, editing it voids the comparison, and `/autoresearch rebaseline` is the deliberate way to accept a new baseline.
16
+
17
+ A keep cannot escape its scope. `scope` globs in the frontmatter bound the files a keep may commit, so an experiment cannot quietly rewrite a file it was not measuring.
12
18
 
13
19
  ## The loop
14
20
 
15
- ```
21
+ ```text
16
22
  edit -> run_experiment -> read the output -> log_experiment -> repeat
17
23
  ```
18
24
 
19
- `run_experiment` snapshots the working tree, runs `.auto/measure.sh`, parses `METRIC name=value` lines, runs
20
- the optional checks, and returns the output, the metrics, the changed files, the noise advice, and the
21
- scope violations. It does not touch git history.
22
-
23
- `log_experiment` takes `status: keep | discard | crash | checks_failed` and the model's reasoning in `asi`.
24
- A `keep` commits the files the run touched with a conventional subject built from the description. Every
25
- other status restores those files. A run inside the noise band, a crash, or a scope violation is reported,
26
- not auto-decided: the model chooses.
25
+ `run_experiment` snapshots the working tree, runs `.auto/measure.sh`, parses `METRIC name=value` lines, runs the optional checks, and returns the output, the metrics, the changed files, the noise advice and any scope violations. It does not touch git history.
27
26
 
28
- ## What it adds over the usual loop
27
+ `log_experiment` takes `status: keep | discard | crash | checks_failed` and the model's reasoning in `asi`. A `keep` commits the files the run touched with a conventional subject built from the description. Every other status restores those files.
29
28
 
30
- | Mechanism | Why |
29
+ | Mechanism | Effect |
31
30
  | --- | --- |
32
- | Model owns keep or discard | a rule cannot know whether the diff caused the gain, or whether simpler code is worth 1% |
33
- | Full benchmark output | phase timings and errors reach the model, so the next hypothesis comes from data |
34
- | Noise floor from median absolute deviation | advice that separates jitter from a real win; confidence never overrides the model |
35
- | Pending measurement | one run between the two tools, crash-safe on disk, so a restart can still log it |
36
- | Scope globs in frontmatter | a keep cannot touch files outside the declared scope |
37
- | Benchmark script hash | editing the benchmark to move the number voids the comparison; `/autoresearch rebaseline` starts a new segment |
38
- | Worktree lock and dirty baseline | one loop per worktree, and files another session left dirty are never committed or reverted |
39
- | Finalize in temporary worktrees | kept runs become branches without moving HEAD under another session |
40
- | Compaction digest from disk | the prompt body, recent runs, dead ends and ideas survive a context reset, no LLM call |
41
- | Conventional commits | `type(scope): summary`, at most 72 characters, one `Metric:` trailer, built by the tool from validated parts |
42
- | Lazy tool registration | zero tokens at cold start |
43
-
44
- ## Files
31
+ | Full benchmark output | Phase timings and errors reach the model, so the next hypothesis comes from data |
32
+ | Noise floor from median absolute deviation | Separates jitter from a real win, without overriding the model |
33
+ | Pending measurement | One run between the tools, crash-safe, so a restart can still log it |
34
+ | Scope globs | A keep cannot touch files outside the declared scope |
35
+ | Benchmark hash | Editing the benchmark to move the number voids the comparison |
36
+ | Worktree lock and dirty baseline | One loop per worktree, and files another session left dirty are never committed |
37
+ | Finalize in temporary worktrees | Kept runs become branches without moving HEAD under another session |
38
+ | Compaction digest from disk | Prompt, recent runs, dead ends and ideas survive a reset with no model call |
39
+ | Conventional commits | `type(scope): summary`, at most 72 characters, one `Metric:` trailer |
40
+
41
+ ## The project files
45
42
 
46
43
  `.auto/prompt.md` carries the frontmatter and the objective:
47
44
 
@@ -60,53 +57,60 @@ min_gain: 0.5
60
57
  What is optimized, the constraints, and what a good change looks like.
61
58
  ```
62
59
 
63
- `min_gain` is the smallest relative improvement the advice treats as real, in percent. It never blocks a keep.
60
+ `min_gain` is the smallest relative improvement the advice calls real, in percent. It never blocks a keep.
64
61
 
65
62
  | File | Purpose |
66
63
  | --- | --- |
67
- | `.auto/prompt.md` | frontmatter config plus the objective and what has been tried |
68
- | `.auto/measure.sh` | benchmark, prints `METRIC name=value` |
69
- | `.auto/log.jsonl` | append-only record |
70
- | `.auto/checks.sh` | optional correctness checks, run after a passing benchmark |
71
- | `.auto/ideas.md` | optional backlog of ideas not tried yet |
72
- | `.auto/pending.json` | the run waiting for a decision |
73
- | `.auto/lock.json` | the session that owns this worktree |
64
+ | `.auto/prompt.md` | Frontmatter config, the objective, what has been tried |
65
+ | `.auto/measure.sh` | The benchmark, printing `METRIC name=value` |
66
+ | `.auto/log.jsonl` | Append-only record |
67
+ | `.auto/checks.sh` | Optional correctness checks, run after a passing benchmark |
68
+ | `.auto/ideas.md` | Optional backlog |
69
+ | `.auto/pending.json` | The run waiting for a decision |
70
+ | `.auto/lock.json` | The session that owns this worktree |
74
71
 
75
- `.auto/` is added to the repository's common `.git/info/exclude` on the first run, so none of it lands in a
76
- commit, including from a linked worktree.
77
-
78
- Run it in an interactive pi session or through RPC. `pi -p` exits as soon as the slash command returns,
79
- before the loop's first turn, so a print run cannot drive the loop.
72
+ The first run adds `.auto/` to `.git/info/exclude`, including from a linked worktree, so none of it lands in a commit.
80
73
 
81
74
  ## Commands
82
75
 
83
76
  | Command | Effect |
84
77
  | --- | --- |
85
- | `/autoresearch <goal>` | start a session, or resume the one in `.auto/` |
86
- | `/autoresearch status` | print the digest |
87
- | `/autoresearch off` | stop and remove the tools |
88
- | `/autoresearch rebaseline` | accept a changed benchmark script as a new baseline |
89
- | `/autoresearch finalize` | split kept runs into independent branches from the merge-base |
90
- | `/autoresearch clear` | delete the log and stop |
78
+ | `/autoresearch <goal>` | Start a session, or resume the one in `.auto/` |
79
+ | `/autoresearch status` | Print the digest |
80
+ | `/autoresearch off` | Stop and remove the tools |
81
+ | `/autoresearch rebaseline` | Accept a changed benchmark script as a new baseline |
82
+ | `/autoresearch finalize` | Split kept runs into independent branches from the merge base |
83
+ | `/autoresearch clear` | Delete the log and stop |
91
84
 
92
- `finalize` groups kept runs by file overlap and creates one branch per group from the merge-base in
93
- temporary worktrees. The current worktree is never checked out to another branch. It verifies that the
94
- union of the branches matches the original branch before reporting.
85
+ `finalize` groups kept runs by file overlap and creates one branch per group in temporary worktrees. It never checks the current worktree out to another branch, and it verifies the union of the branches against the original before it reports.
95
86
 
96
87
  ## Worktrees and other sessions
97
88
 
98
- One loop owns a worktree at a time. `/autoresearch` takes `.auto/lock.json`, and a second session in the same
99
- worktree is refused before it can run anything. Files that were already dirty when the loop started are
100
- recorded in the lock and never committed or reverted.
89
+ One loop owns a worktree at a time: `/autoresearch` takes `.auto/lock.json` and refuses a second session before it runs anything. Files that were already dirty when the loop started are recorded in the lock and never committed or reverted.
101
90
 
102
- For a second loop on the same repository, use a separate git worktree. Each worktree has its own `.auto/`,
103
- its own branch, and its own log. The first run on trunk creates `autoresearch/<name>-<timestamp>` in a git
104
- worktree at `~/worktrees/<repo>/autoresearch-<name>-<timestamp>`, and the shared checkout stays on trunk. Start
105
- pi in that directory to run the loop. An existing feature branch is used as it is.
91
+ For a second loop, use a separate git worktree, which gets its own `.auto/`, branch and log. The first run on trunk creates `autoresearch/<name>-<timestamp>` in a worktree at `~/worktrees/<repo>/autoresearch-<name>-<timestamp>` and leaves the shared checkout on trunk. An existing feature branch is used as it is.
106
92
 
107
93
  ## Compaction and restarts
108
94
 
109
- While the loop is active, `session_before_compact` replaces pi's LLM summary with a deterministic digest
110
- built from `.auto/`: the prompt body, the recent runs, kept runs, dead ends, ideas, and any pending
111
- measurement. After a restart of the same session, the loop re-arms from the session's activation entry and
112
- sends one short resume message. A fresh session never auto-activates.
95
+ While the loop is active, `session_before_compact` replaces pi's model summary with a deterministic digest built from `.auto/`: the prompt body, recent runs, kept runs, dead ends, ideas and any pending measurement. No model call, and no history lost to a summary that forgot the metric.
96
+
97
+ After a restart of the same session the loop re-arms from the session's activation entry and sends one short resume message. A fresh session never auto-activates.
98
+
99
+ ## Install
100
+
101
+ ```bash
102
+ pi install npm:@tduyng/pi-autoresearch
103
+ ```
104
+
105
+ Local checkout: `pi install /path/to/dotfiles/pi/agent/extensions/pi-autoresearch`
106
+
107
+ It needs a git repository, since a keep commits, the lock is a file, and finalize uses worktrees. The measure script runs through `bash -c`.
108
+
109
+ Run it in an interactive session or through RPC. `pi -p` exits as soon as the slash command returns, before the loop's first turn, so a print run cannot drive it.
110
+
111
+ ## Limits
112
+
113
+ - A run has a default timeout of 600 seconds, and the measure script's own timeout is the one that applies.
114
+ - Auto-resume gives up after 200 turns, and an idle resume after 40.
115
+ - Output is trimmed to the last 40 lines, with 128 KB retained and a 20 MB cap on the temp log.
116
+ - `finalize` reports a conflict instead of resolving one. Splitting the kept runs is a bookkeeping job, not a merge.
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "@tduyng/pi-autoresearch",
3
- "version": "0.0.2",
4
- "description": "Autonomous experiment loop with zero cold-start cost: measure, decide, commit or revert, with scope and benchmark guards.",
3
+ "version": "0.0.4",
4
+ "description": "Autonomous experiment loop that costs nothing until you type /autoresearch: measure, decide, commit or revert, with scope and benchmark guards.",
5
5
  "keywords": [
6
6
  "pi",
7
7
  "pi-package",