@alphacoder-v0/pi-loops 0.17.5 → 0.18.0

package/recipes/autoresearch/README.md

@@ -0,0 +1,27 @@
+# autoresearch — one experiment per run, a ledger, held-out evidence before promotion
+
+The shape is the one Karpathy's autoresearch and Arbor made familiar: a **research contract** a
+person writes (objective, metric, which files may change, which are protected, dev and held-out
+commands, a promotion rule), a **ledger** of every attempt including the failed ones, an isolated
+worktree per experiment, and a rule that the held-out number decides — never the dev number a run
+was tuning against. What pi-loops adds is that nothing here needs a kernel: the contract and the
+ledger are files in the repository, each run is one stateful loop run, and the checker of
+`--verify` is the evaluator that re-measures before a promotion reaches you.
+
+| file | what it is |
+|---|---|
+| [recipe.toml](recipe.toml) | one job, `every 1h`, `--verify`, 40-minute timeout |
+| [experiment.md](experiment.md) | the playbook: contract → ledger → one hypothesis → worktree → measure → row → maybe promote |
+| [RESEARCH.template.md](RESEARCH.template.md) | copy to the project root as `RESEARCH.md` and fill in; the loop stops until it exists |
+| [example/](example/) | a filled-in contract with a naive exact k-NN in pure Python: `bash eval.sh dev` prints `metric=<speedup>` |
+
+Try it on the example first:
+
+```sh
+cp -r .agents/skills/autoresearch/example /tmp/knn && cd /tmp/knn && git init -q && git add -A && git commit -qm "baseline"
+pi            # then: /recipe add autoresearch   and   /cron run autoresearch
+```
+
+The first run measures the baseline and writes `results.tsv`; the next ones each try one idea.
+Under `propose` a promotable branch arrives in `/inbox` with both numbers; under `act` it is
+merged into the checkout and the finding says so.

package/recipes/autoresearch/RESEARCH.template.md

@@ -0,0 +1,37 @@
+# Research contract
+
+Copy this file to the project root as `RESEARCH.md` and fill it in. The autoresearch loop reads it
+on every run and never changes it; a run that finds it missing or its protected files altered stops
+with a finding. The example in `.agents/skills/autoresearch/example/` is a filled-in one.
+
+## Objective
+
+One sentence a person can check: what better means. *Make the exact k-nearest-neighbour search in
+`solution.py` faster while keeping its output identical.*
+
+## Metric
+
+- **name:** `speedup` — **higher is better** (or: `val_loss`, lower is better)
+- **dev command:** `bash eval.sh dev` — prints one line `metric=<number>`; used to iterate
+- **held-out command:** `bash eval.sh test` — same shape, other data; used only to decide promotion
+- **baseline:** measured by the first run on the untouched code and written to the ledger; leave blank
+
+## Files
+
+- **editable:** `solution.py` (one path per line; nothing else may change)
+- **protected:** `eval.sh`, `eval.py`, `task.py` — the loop records their `sha256sum` in the ledger's
+  header on the first run and refuses to work if any differs since
+
+## Promotion rule
+
+A branch is promotable when its **held-out** metric beats the best promoted (or the baseline) by at
+least **5%**, and the dev metric agreed. Under `propose` the loop files a finding and a person
+merges; under `act` the loop merges the branch into this checkout's current branch itself.
+
+## Budget and layout
+
+- **per experiment:** at most 10 minutes of evaluation; a run that exceeds it retires the experiment
+- **ledger:** `research/results.tsv` (created by the loop, never committed by it)
+- **worktrees:** `../<repo>-research/<NNN>`, branch `research/<NNN>-<slug>`; retired branches are deleted, the ledger row stays
+- **directions worth trying first (optional):** …
+- **directions not to try (optional):** …

package/recipes/autoresearch/example/RESEARCH.md

@@ -0,0 +1,31 @@
+# Research contract
+
+## Objective
+
+Make the exact k-nearest-neighbour search in `solution.py` faster while keeping its output
+identical to the reference.
+
+## Metric
+
+- **name:** `speedup` — higher is better (wall-clock of the reference divided by wall-clock of `solution.py`, measured in the same process, median of 3)
+- **dev command:** `bash eval.sh dev`
+- **held-out command:** `bash eval.sh test`
+- **baseline:** (written by the first run)
+
+## Files
+
+- **editable:** `solution.py`
+- **protected:** `eval.sh`, `eval.py`
+
+## Promotion rule
+
+Held-out `speedup` at least 5% above the best promoted row (or the baseline), with the dev run
+agreeing and the output check passing.
+
+## Budget and layout
+
+- **per experiment:** at most 5 minutes of evaluation
+- **ledger:** `results.tsv`
+- **worktrees:** `../<repo>-research/<NNN>`, branch `research/<NNN>-<slug>`
+- **directions worth trying first:** partial selection instead of a full sort; avoid recomputing the query's coordinates in the inner loop; batch the distance computation
+- **directions not to try:** approximate search (the output must stay exact); C extensions or third-party packages (standard library only)

package/recipes/autoresearch/example/eval.py

@@ -0,0 +1,60 @@
+"""Protected: generates the data, checks exactness, measures speedup. Do not edit in an experiment.
+
+Usage: python3 eval.py dev|test   → prints `metric=<speedup>` (or `metric=0 error=...`).
+
+The reference is the naive algorithm frozen here, so a speedup is relative to the starting point
+and independent of the machine's absolute speed. Timings are the median of three runs of each.
+"""
+import random
+import statistics
+import sys
+import time
+
+import solution
+
+SPLITS = {"dev": (1, 2500, 80, 8, 4), "test": (2, 3000, 90, 8, 5)}  # seed, n points, n queries, dims, k
+
+
+def reference(points, queries, k):
+    # Deliberately the same loops as the untouched solution.py, so the baseline speedup is 1.0.
+    out = []
+    for q in queries:
+        dists = []
+        for i, p in enumerate(points):
+            d = 0.0
+            for a, b in zip(p, q):
+                d += (a - b) * (a - b)
+            dists.append((d, i))
+        dists.sort()
+        out.append([i for _, i in dists[:k]])
+    return out
+
+
+def timed(fn, *args):
+    runs = []
+    for _ in range(3):
+        t0 = time.perf_counter()
+        fn(*args)
+        runs.append(time.perf_counter() - t0)
+    return statistics.median(runs)
+
+
+def main():
+    split = sys.argv[1] if len(sys.argv) > 1 else "dev"
+    seed, n, m, dims, k = SPLITS[split]
+    rng = random.Random(seed)
+    points = [[rng.random() for _ in range(dims)] for _ in range(n)]
+    queries = [[rng.random() for _ in range(dims)] for _ in range(m)]
+    expected = reference(points, queries, k)
+    got = solution.knn(points, queries, k)
+    if got != expected:
+        print("metric=0 error=output differs from the reference")
+        return 1
+    ref_t = timed(reference, points, queries, k)
+    sol_t = timed(solution.knn, points, queries, k)
+    print(f"metric={ref_t / sol_t:.3f} split={split} ref_s={ref_t:.3f} sol_s={sol_t:.3f}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

package/recipes/autoresearch/example/eval.sh

@@ -0,0 +1,3 @@
+#!/bin/sh
+# Protected. `bash eval.sh dev` iterates; `bash eval.sh test` decides promotion.
+cd "$(dirname "$0")" && exec python3 eval.py "${1:-dev}"

package/recipes/autoresearch/example/solution.py

@@ -0,0 +1,20 @@
+"""The editable file: exact k-nearest neighbours, deliberately naive.
+
+`knn(points, queries, k)` returns, for each query, the indices of the k closest points by squared
+Euclidean distance, ties broken by lower index. The evaluator checks the output against a
+reference and times this function; only this file may change.
+"""
+
+
+def knn(points, queries, k):
+    out = []
+    for q in queries:
+        dists = []
+        for i, p in enumerate(points):
+            d = 0.0
+            for a, b in zip(p, q):
+                d += (a - b) * (a - b)
+            dists.append((d, i))
+        dists.sort()
+        out.append([i for _, i in dists[:k]])
+    return out

package/recipes/autoresearch/experiment.md

@@ -0,0 +1,82 @@
+---
+name: autoresearch-experiment
+description: One run of the autoresearch loop. Reads the research contract and the ledger, runs one experiment in a worktree, measures it, writes the row. Proposes promotion only on held-out evidence; retired experiments stay in the ledger as negative evidence.
+---
+
+# Experiment: one run
+
+Autonomy: propose
+
+The line above is the installed level. `propose`: a promotable branch is a finding a person merges.
+`act`: you merge it into this checkout's current branch yourself (`git merge --no-ff`), then file
+the finding saying so. Nothing else differs between the levels.
+
+You are the autoresearch loop. Nobody is watching. One run is **one experiment**: a hypothesis,
+a branch, a measurement, a row in the ledger. The run has a timeout; commit and write the row
+before anything slow. Your final reply is read by a program.
+
+## The contract and the ledger are the truth
+
+1. Read `RESEARCH.md` at the project root. Missing → one finding: `autoresearch: no RESEARCH.md —
+   copy .agents/skills/autoresearch/RESEARCH.template.md to the project root and fill it in`, notes
+   unchanged, stop. Never write the contract yourself.
+2. Check the protected files: `sha256sum` each, compare with the hashes in the ledger header. A
+   difference → one finding naming the file, stop. (First run: record them.)
+3. Read the ledger at the path the contract names. Missing → create it:
+
+   ```text
+   # autoresearch ledger — protected: eval.sh=<sha256> eval.py=<sha256> …
+   id	started_at	branch	hypothesis	status	dev	heldout	note
+   ```
+
+   then measure the **baseline**: run the dev and the held-out commands on the untouched code and
+   write row `000` with status `baseline`. That is the whole first run.
+
+4. **Resume.** If your notes name an experiment in progress and its worktree exists, continue it
+   from the branch's `git log` and `git status` instead of starting another.
+
+## One experiment
+
+5. **Choose a hypothesis.** Read every ledger row: `retired` rows are negative evidence and are not
+   retried in the same form; `kept` rows are the frontier to build on; the contract's "worth
+   trying first" and "not to try" lists bound the space. Write one sentence: what you will change
+   and why it should move the metric. Append the row now with status `running` — a crash leaves a
+   record, not a mystery.
+6. **Branch.** `git worktree add <worktrees>/<NNN> -b research/<NNN>-<slug> <current branch>`
+   (create the worktrees directory first if it is missing).
+   Change only the editable files. Commit in the worktree before measuring.
+7. **Measure.** Run the dev command in the worktree, within the contract's time budget. Parse
+   `metric=<number>`. Compare with the best `kept`/`promoted` row (or the baseline):
+   - **not better** → status `retired`, note why in ten words, `git worktree remove` and delete
+     the branch. No finding: a retired experiment is normal, and the row is what the next run needs.
+   - **better on dev** → run the held-out command. Write both numbers. If held-out also passes the
+     contract's promotion rule → status `kept` and go to 8; otherwise `retired` with note
+     `dev-only gain` (the classic overfit; still negative evidence) and remove the worktree.
+8. **Promote.** Under `propose`: one finding —
+   `promote research/<NNN>-<slug>: dev <x> → <y>, held-out <a> → <b> (<hypothesis>)` — and leave
+   the worktree and branch for the person. Under `act`: in the project checkout,
+   `git merge --no-ff research/<NNN>-<slug>`, set status `promoted`, remove the worktree, keep the
+   branch, and file the finding as `promoted …`.
+
+Never edit a protected file, never push, never merge under `propose`, never run two experiments
+in one run. If the eval commands fail for a reason that is not your change (missing interpreter,
+data not generated), that is one finding and a stop, not a retry.
+
+## For the checker
+
+A second sub-agent reviews the findings of this run before they reach the inbox. For a `promote`
+or `promoted` finding: check out the named branch in its worktree (or the merge in the checkout),
+run the contract's **held-out** command yourself, and keep the finding only if the number you get
+satisfies the promotion rule against the ledger's best row; drop it with the number otherwise. A
+`no RESEARCH.md` or protected-file finding is kept if the file state is as described.
+
+## Notes for the next run
+
+`best=<id> <heldout>` · `next=<NNN>` · `in-progress: <id> worktree=<path>` or `none` ·
+`tried: <one line per idea class you have exhausted>`. Under 2000 characters; the ledger holds
+the rest.
+
+## Inbox
+
+Only: a promotion (proposed or done), a contract problem, an eval that cannot run. Never a retired
+experiment — the ledger is where those live, and a person reads the ledger when they want to.

package/recipes/autoresearch/recipe.toml

@@ -0,0 +1,15 @@
+# autoresearch: a person writes the research contract, a loop runs one experiment per run and keeps
+# a ledger, a checker re-measures on held-out data before anything is proposed. docs/recipes.md.
+name = "autoresearch"
+summary = "One experiment per run against a research contract you wrote; a ledger of what was tried; promotion only on held-out evidence."
+needs_tracker = false
+levels = ["propose", "act"]
+files = ["RESEARCH.template.md", "example/RESEARCH.md", "example/solution.py", "example/eval.py", "example/eval.sh"]
+budget_hint_usd = 15
+
+[[job]]
+name = "autoresearch"
+schedule = "every 1h"
+playbook = "experiment.md"
+verify = true
+timeout = "40m"

package/recipes/changelog-draft/recipe.toml

@@ -0,0 +1,9 @@
+name = "changelog-draft"
+summary = "When the default branch is ahead of the last tag, draft the changelog entry and propose a version; a person tags."
+needs_tracker = false
+levels = ["report"]
+
+[[job]]
+name = "changelog-draft"
+schedule = "0 18 * * 1-5"
+playbook = "release.md"

package/recipes/changelog-draft/release.md

@@ -0,0 +1,35 @@
+---
+name: issue-release
+description: One run of the release loop. Compares the default branch with the last tag and, when there is something worth shipping, drafts the changelog entry and proposes a version to the inbox. It never tags or publishes.
+---
+
+# Release: one run
+
+Autonomy: report
+
+You are the release loop. You **propose**; a person releases. Tagging and publishing need things
+an unattended run does not have (a signing key, a one-time password, a decision about the version
+number), so the checkpoint is placed here, after everything that can be prepared has been.
+
+1. Establish the facts:
+
+   ```sh
+   git fetch origin --tags
+   git describe --tags --abbrev=0 origin/<default branch>      # last release
+   git log <last tag>..origin/<default branch> --oneline       # what has landed since
+   ```
+
+2. If the head is the same commit you last reported (in your notes), or there is nothing but
+   `chore:` / `docs:` / `ci:` commits since the tag, stop: notes unchanged, no inbox line.
+
+3. Otherwise draft the changelog entry in the repository's own changelog style (read the top of
+   its `CHANGELOG.md`): one line per `feat:` / `fix:` / `perf:` commit, grouped the way earlier
+   entries are, each line saying what a user notices rather than what a file does. Pick the
+   version the way the project's history does (a `feat:` bumps the minor of a 0.x project; only
+   fixes bump the patch). Do not edit any file; the draft goes in the inbox line.
+
+4. Report: `release proposal: v<next> — <k> commits since v<last> (<f> feat, <b> fix); draft: <the
+   entry, condensed to one line per change>`. Keep it under 500 characters; if the draft is
+   longer, write it to a comment on a `release` issue instead and link it.
+
+Notes: `last-reported-head=<sha> last-tag=<tag>`.

package/recipes/issue-loop/README.md

@@ -0,0 +1,16 @@
+# issue-loop — an issue-driven iteration loop for a repository
+
+Three files a stateful loop reads, one script that creates the labels they move issues through.
+The design, the state machine and how to install it in a project are in
+[docs/recipes.md](../../docs/recipes.md#issue-loop).
+
+| file | read by | what one run does |
+|---|---|---|
+| [triage.md](triage.md) | the `issue-triage` loop | evaluates new and replied-to issues, verifies claims, writes agent briefs, recommends the next state |
+| [implement.md](implement.md) | the `issue-implement` loop | resumes cut-off work, addresses review feedback, claims one `ready-for-agent` issue, opens a pull request |
+| [labels.sh](labels.sh) | you, once per repository | creates the category and state labels (idempotent) |
+
+Install it with `/recipe add issue-loop` in a pi opened in the repository: the wizard asks for the
+autonomy level, copies the playbooks to `.agents/skills/issue-loop/`, runs `labels.sh` after
+showing it, and creates the jobs. Each job's prompt is a pointer to its playbook, so the procedure
+is read fresh on every run and a person can edit it in place.

package/recipes/issue-loop/implement.md

@@ -0,0 +1,135 @@
+---
+name: issue-implement
+description: One run of the implement loop. Finishes what it was doing, addresses review feedback on open pull requests, then claims at most one ready-for-agent issue, builds it in a worktree from its Agent Brief, and opens a pull request. Reports each state change to the inbox.
+---
+
+# Implement: one run
+
+Autonomy: propose
+
+The line above is the installed autonomy level. At every level you claim, build, push and open the
+pull request; **merging stays with a person** — the tracker's review is the checkpoint `act` does
+not remove. The level changes only the triage side of this recipe.
+
+You are the implement loop. Nobody is watching. The run has a timeout; work so that being cut off
+loses nothing (commit early, push before you stop). Your final reply is read by a program.
+
+The tracker is the truth; your notes are a cache. The labels you own are `agent-working`,
+`in-review` and `agent-blocked`; the transitions you may make are the ones below and no other.
+Every comment you post starts with `> *This was generated by AI.*` on its own line.
+
+## Order of work
+
+**First, read `docs/agents/issue-tracker.md`.** It says which tracker this repository uses and how
+to list, read, comment, change state, claim and close. The `gh` commands in this playbook are the
+GitHub spelling; if the file describes local Markdown files, do each step the way it describes
+instead (the states and the checkpoints are the same).
+
+Do these in order; stop after the first that produces a full run's work.
+
+1. **Resume.** If your notes name an issue in progress and its worktree exists, continue it from
+   the branch's `git log` and `git status`. A previous run was cut off; do not start over.
+2. **Review feedback.** For each open pull request you opened (issue label `in-review`) whose
+   review state is `CHANGES_REQUESTED`, or that has a comment by a person newer than the
+   timestamp in your notes: check out its worktree, address every point (reply to each comment
+   saying what you did, or why not), run the project's checks, push, and re-request review.
+   Report: `#<n> review feedback addressed — PR <url>`.
+3. **One new issue.** The frontier is the open issues labeled `ready-for-agent` with no assignee
+   whose blockers are all closed (a `Blocked by: #m` line in the body or brief, or the tracker's
+   native dependency). Take the **oldest one**. One per run — a second one waits for the next run.
+4. **Housekeeping.** For every worktree under the worktree directory whose issue is closed and
+   whose pull request is merged or closed: `git worktree remove <path>` and delete the local
+   branch. Drop the issue from your notes. This step runs in any run that reaches it.
+
+```sh
+gh issue list --state open --label ready-for-agent --json number,title,body,assignees,createdAt
+gh issue view <n> --comments --json body,comments,labels
+gh pr list --state open --author @me --json number,url,headRefName,reviewDecision,comments,updatedAt
+```
+
+## Claiming
+
+The claim is the first write, and it is what keeps two runs (or two machines) off the same issue:
+
+```sh
+gh issue edit <n> --add-label agent-working --remove-label ready-for-agent --add-assignee @me
+```
+
+If the edit fails because the label is already gone, someone else has it: stop and pick the next.
+Then the worktree, beside the repository so nothing lands in the checkout the loop runs in:
+
+```sh
+git fetch origin <default branch>
+git worktree add ../<repo>-issues/<n> -b issue/<n>-<slug> origin/<default branch>
+```
+
+Comment on the issue: which branch, and that a pull request will follow. Write the issue number,
+branch and worktree path to your notes **now**, before any implementation, so a cut-off run resumes.
+
+## Building it
+
+- The **latest `## Agent Brief` comment is the contract**; the issue body and discussion are
+  context. If the brief and the code disagree in a way the brief did not foresee, that is a block
+  (below), not a judgement call.
+- Work in the worktree. Follow the repository's `AGENTS.md` / `CLAUDE.md` and its conventions.
+  For a bug, write the failing test first. Run the project's check command (the one its
+  `AGENTS.md` names — for pi-loops, `npm run ci`) before every commit that claims to be done.
+- Commit as `<type>: <description>`; push with `-u` to a branch named after the issue. Never
+  push to the default branch, never force-push, never rebase a branch someone has reviewed.
+- Open the pull request when every acceptance criterion in the brief is checked:
+
+  ```sh
+  gh pr create --base <default branch> --head issue/<n>-<slug> --title "<type>: <description>" --body "$(cat <<'MD'
+  > *This was generated by AI.*
+
+  Closes #<n>
+
+  ## What changed
+  …
+
+  ## Test plan
+  - [x] <the check command> passes
+  - [x] each acceptance criterion, and how it was checked
+  MD
+  )"
+  gh issue edit <n> --add-label in-review --remove-label agent-working
+  ```
+
+  Report: `#<n> PR opened — <url> (<k> commits, checks green)`. The maintainer merges; the
+  `Closes` line closes the issue.
+
+## Blocked
+
+When you cannot finish — the brief contradicts the code, a check fails in a way that is not yours
+to change, a decision is needed, a credential is missing — commit and push what exists, then:
+
+```sh
+gh issue edit <n> --add-label agent-blocked --remove-label agent-working
+```
+
+Comment with the specific question or the specific failure (command and output), the branch, and
+what you would do next given an answer. Report: `#<n> blocked — <one-line reason>`. The
+maintainer answers, edits the brief if needed, and moves the issue back to `ready-for-agent`; do
+not touch an `agent-blocked` issue again until they do.
+
+Two consecutive runs failing the check command on the same issue is a block, not a third attempt.
+
+## Out of time
+
+If the run is close to its timeout with the work unfinished: commit as a work-in-progress, push,
+leave the issue in `agent-working`, and write in your notes what is done and what is next. No
+inbox line — resuming is your job, not the maintainer's.
+
+## Notes for the next run
+
+- `in-progress: #<n> branch=<name> worktree=<path> done=<short list> next=<short list>` or `in-progress: none`
+- one line per open PR: `pr #<pr> issue=#<n> last-handled=<ISO timestamp with offset>`
+- `default-branch=<name> worktree-dir=<path>` once you have established them
+
+Under 2000 characters. If you must cut, keep `in-progress` and cut the oldest PR lines; the
+tracker has the rest.
+
+## Inbox
+
+One line per state change you made, in the shapes above. Housekeeping produces no inbox lines. A
+run that only resumed and is still working produces none either.

package/recipes/issue-loop/labels.sh

@@ -0,0 +1,21 @@
+#!/bin/sh
+# The labels the issue loop moves issues through. Idempotent: --force updates the colour and
+# description of a label that already exists and never touches the issues carrying it.
+# Run inside a clone of the repository (gh infers the repository from the remote).
+set -e
+
+lab() { gh label create "$1" --description "$2" --color "$3" --force >/dev/null && echo "  $1"; }
+
+echo "category labels"
+lab bug          "Something is broken"                                          d73a4a
+lab enhancement  "New feature or improvement"                                    a2eeef
+
+echo "state labels (exactly one per open issue)"
+lab needs-triage    "Awaiting evaluation; the triage loop posts a brief or questions here"     e4e669
+lab needs-info      "Waiting on the reporter; returns to needs-triage when they reply"          d876e3
+lab ready-for-agent "Brief accepted by a maintainer; the implement loop may claim it"           0e8a16
+lab ready-for-human "Needs a person: judgement, access, design, manual testing"                 fbca04
+lab agent-working   "Claimed by the implement loop; a worktree and branch exist for it"         1d76db
+lab in-review       "A pull request is open; waiting for a person to review and merge"          5319e7
+lab agent-blocked   "The implement loop stopped; its last comment says what it needs"           b60205
+lab wontfix         "Will not be actioned"                                                      ffffff

package/recipes/issue-loop/recipe.toml

@@ -0,0 +1,18 @@
+# The issue loop: the tracker as a state machine that three loops turn. docs/recipes.md explains it.
+name = "issue-loop"
+summary = "Triage new issues into agent briefs, build the accepted ones in a worktree, open pull requests; a person promotes and merges."
+needs_tracker = true
+levels = ["propose", "act"]
+setup = "labels.sh"
+budget_hint_usd = 10
+
+[[job]]
+name = "issue-triage"
+schedule = "*/30 * * * *"
+playbook = "triage.md"
+
+[[job]]
+name = "issue-implement"
+schedule = "15,45 * * * *"
+playbook = "implement.md"
+timeout = "45m"