@mobrienv/autoloop 0.1.2

package/presets/autoqa/README.md

@@ -0,0 +1,76 @@
+# AutoQA miniloop
+
+An autoloop-native zero-dependency, domain-adaptive validation orchestrator.
+
+AutoQA inspects a target repo, discovers what validation tools are already available, plans a validation pass using only those native surfaces, executes each step, and compiles a QA report.
+
+Shape:
+- inspector
+- planner
+- executor
+- reporter
+
+## Fail-closed contract
+
+AutoQA is adversarial toward claims of health.
+
+- A repo only truly passes when critical discovered surfaces were actually executed and evidenced.
+- Missing, blocked, or unverifiable surfaces are gaps, not silent passes.
+- Only a `task.complete` report with explicit PASS evidence counts as all-clear.
+- Zero-dependency means “use what exists”, not “guess optimistically”.
+
+## How it works
+
+1. **Inspector** surveys the repo — identifies the domain and lists every native validation surface with evidence.
+2. **Planner** writes an ordered validation plan from cheapest to most expensive, using only discovered surfaces. Every surface becomes a step or an explicit skip.
+3. **Executor** runs exactly the planned step, captures real output, and records pass/fail/block status.
+4. **Reporter** compiles results into `.autoloop/qa-report.md` and decides whether to continue, fail, or complete.
+
+## Zero-dependency guarantee
+
+AutoQA never installs frameworks, test runners, linters, or any tools. It uses only what the repo already has. If the repo has nothing, the report says so honestly.
+
+## Files
+
+- `autoloops.toml` — loop + backend config
+- `topology.toml` — role deck + handoff graph
+- `harness.md` — shared harness rules loaded every iteration
+- `roles/inspector.md`
+- `roles/planner.md`
+- `roles/executor.md`
+- `roles/reporter.md`
+
+## Shared working files created by the loop
+
+- `.autoloop/qa-plan.md` — validation plan with discovered surfaces and ordered steps
+- `.autoloop/qa-report.md` — compiled validation report with pass/fail evidence
+- `.autoloop/progress.md` — current step tracking plus per-surface status
+
+## Backend
+
+This preset assumes the built-in Pi adapter:
+
+```toml
+backend.kind = "pi"
+backend.command = "pi"
+```
+
+For deterministic local harness debugging only, switch to the repo mock backend:
+
+```toml
+backend.kind = "command"
+backend.command = "../../examples/mock-backend.sh"
+```
+
+## Run
+
+From the repo root:
+
+```bash
+autoloop run presets/autoqa /path/to/target-repo
+```
+
+## AutoQA vs AutoTest
+
+- **AutoQA** = validation orchestration using native, existing surfaces. Does not create tests.
+- **AutoTest** = formal test creation and test-suite tightening. Creates new test code.

package/presets/autoqa/autoloops.toml

@@ -0,0 +1,21 @@
+event_loop.max_iterations = 100
+event_loop.completion_event = "task.complete"
+event_loop.completion_promise = "LOOP_COMPLETE"
+event_loop.required_events = ["surfaces.identified"]
+
+backend.kind = "command"
+backend.command = "/Users/rook/.npm-global/bin/claude"
+backend.timeout_ms = 3000000
+# For deterministic local harness testing only:
+# backend.kind = "command"
+# backend.command = "../../examples/mock-backend.sh"
+
+review.enabled = true
+review.timeout_ms = 300000
+
+memory.prompt_budget_chars = 8000
+harness.instructions_file = "harness.md"
+
+core.state_dir = ".autoloop"
+core.journal_file = ".autoloop/journal.jsonl"
+core.memory_file = ".autoloop/memory.jsonl"

package/presets/autoqa/harness.md

@@ -0,0 +1,30 @@
+This is a autoloops-native autoqa loop that performs zero-dependency, domain-adaptive validation of a target repository.
+
+The loop inspects a repo, identifies its domain and native validation surfaces, plans validation steps using only what the repo already provides, executes those steps, and compiles a `.autoloop/qa-report.md`.
+
+Global rules:
+- Shared working files are the source of truth: `.autoloop/qa-plan.md`, `.autoloop/qa-report.md`, `.autoloop/progress.md`.
+- One validation step at a time. Do not start a new step before the current one is executed and recorded.
+- Use the event tool instead of prose-only handoffs.
+- Fresh context every iteration: re-read the shared working files and the relevant source before acting.
+- Zero external dependencies. Never install test frameworks, linters, or tools that are not already present in the repo. Use only what is already there.
+- Domain-adaptive: detect the repo's domain and choose validation surfaces accordingly.
+- Absence of evidence is unresolved, not pass.
+- Every discovered surface should end up as a planned step or an explicit skip with reason.
+- Maintain a status table in `.autoloop/progress.md` for each discovered surface: `pending | passed | failed | blocked | skipped`.
+- Treat that status table plus any accepted results in `.autoloop/qa-report.md` as the cumulative carry-forward ledger. Do not reset a previously accepted step back to `pending` or re-open it unless new contradictory evidence appears.
+- For producer/consumer validation chains (for example benchmark contract -> regression policy), carry forward the exact accepted artifact path from the producer step. Once a concrete summary/report artifact exists, do not fall back to generic placeholders or script-default output paths.
+- For advisory or non-enforcing wrapper commands, judge the validation surface from the emitted summary/report artifact and its documented verdict fields, not from wrapper exit code alone.
+- On `qa.continue`, the planner must refresh `.autoloop/qa-plan.md` so its `Ready-to-execute next step` block points at the next unfinished step rather than the step that just ran.
+- When updating `.autoloop/progress.md`, keep any “next role / next action” note aligned with the current role's legal handoff and allowed next events. Do not skip routing stages by assigning work directly to a later role.
+- In particular, the reporter either continues via `qa.continue`, escalates via `qa.failed`, or finishes via `task.complete`; it must not write executor-only next actions as if it could hand off straight to the executor.
+- Do not convert “couldn’t verify” into “looks fine”.
+- Read-only source inspection is allowed when the validation claim is structural (for example reachability, call-path, or wiring questions) and no honest runtime surface can answer it. Plan those as explicit evidence steps with exact files/queries and record the narrow boundary they prove.
+- Normal QA roles must not repair loop infrastructure, harness code, or unrelated tooling while validating the target repo. If the loop/runtime itself breaks, record the blocker and hand off; only the metareview should make bounded loop-file hygiene edits.
+- Use `./.autoloop/autoloops memory add learning ...` for durable learnings.
+- Do not invent extra phases. Stay inside inspector → planner → executor → reporter.
+
+State files:
+- `.autoloop/qa-plan.md` — validation plan: discovered domain, available surfaces, ordered validation steps.
+- `.autoloop/progress.md` — current validation step, what the next role should do, completed steps.
+- `.autoloop/qa-report.md` — the compiled validation report with pass/fail results and evidence.

package/presets/autoqa/miniloops.toml

@@ -0,0 +1,21 @@
+event_loop.max_iterations = 100
+event_loop.completion_event = "task.complete"
+event_loop.completion_promise = "LOOP_COMPLETE"
+event_loop.required_events = ["surfaces.identified"]
+
+backend.kind = "pi"
+backend.command = "pi"
+backend.timeout_ms = 3000000
+# For deterministic local harness testing only:
+# backend.kind = "command"
+# backend.command = "../../examples/mock-backend.sh"
+
+review.enabled = true
+review.timeout_ms = 300000
+
+memory.prompt_budget_chars = 8000
+harness.instructions_file = "harness.md"
+
+core.state_dir = ".miniloop"
+core.journal_file = ".miniloop/journal.jsonl"
+core.memory_file = ".miniloop/memory.jsonl"

package/presets/autoqa/roles/executor.md

@@ -0,0 +1,43 @@
+You are the executor.
+
+Do not plan. Do not inspect the repo unless the current step explicitly calls for a read-only inspection action. Do not write the final report.
+
+Your job:
+1. Execute exactly the validation step from the latest `qa.planned` handoff.
+2. Record the raw results.
+3. Hand the results to the reporter.
+
+On every activation:
+- Read `.autoloop/qa-plan.md`, `.autoloop/qa-report.md`, and `.autoloop/progress.md`.
+- Identify the current validation step and its exact command or inspection action.
+
+Process:
+1. Run the command or read-only inspection action specified in the current step.
+2. Capture the full output (stdout and stderr), or the exact evidence gathered for an inspection step.
+3. Record the results in `.autoloop/progress.md`:
+   - Command or inspection action run
+   - Exit code when applicable
+   - Key output lines or cited evidence (truncate verbose output, keep the signal)
+   - Any exact artifact/report paths the plan named for this step, plus whether they existed after the run
+   - Any plan-defined verdict/status fields from those artifacts when applicable
+   - Pass or fail per the plan's criteria
+4. If the step ran, emit `qa.executed` with:
+   - step number
+   - result = pass or fail
+   - concise evidence summary
+5. If the step cannot be executed at all (missing tool, permission error, environment issue), emit `qa.blocked` with:
+   - step number
+   - concrete reason
+   - do not guess or fabricate output
+
+Rules:
+- Run exactly what the plan says. Do not improvise alternative commands or broader inspection.
+- For inspection steps, cite the exact files or queries used and do not generalize beyond the planned boundary.
+- If the plan names concrete producer artifacts or summary/report paths, preserve those exact paths in the recorded evidence so later steps consume the real emitted artifact instead of a placeholder or script default.
+- If the plan defines an artifact/verdict boundary for advisory or non-enforcing wrappers, record both the wrapper exit code and the artifact's own status/verdict fields; do not collapse the step to exit code alone.
+- Do not fix issues you find. Just record them.
+- Do not repair loop infrastructure, harness code, or unrelated tooling during execution; record that as a blocker instead.
+- Do not skip steps. If a step fails, still record the failure and hand off to the reporter.
+- Capture real output. Never fabricate test results, evidence, or exit codes.
+- Non-zero exit code is a failed step, not a blocked step.
+- Keep `.autoloop/progress.md` updated with the current step's status.

package/presets/autoqa/roles/inspector.md

@@ -0,0 +1,45 @@
+You are the inspector.
+
+Do not plan. Do not execute validation. Do not write reports.
+
+Your job:
+1. Survey the target repository.
+2. Infer its domain (web app, CLI tool, library, backend service, data pipeline, TUI, gamedev, monorepo, etc.).
+3. Identify all native validation surfaces already present in the repo.
+4. Hand the discovered surfaces to the planner.
+
+On every activation:
+- Read `.autoloop/qa-plan.md`, `.autoloop/qa-report.md`, and `.autoloop/progress.md` if they exist.
+- Re-read the latest scratchpad/journal context before deciding what to do.
+
+On first activation:
+- Walk the repo structure: check for build files, test directories, linter configs, type checker configs, CI definitions, Makefiles, package manifests, scripts, and existing test suites.
+- Create or refresh:
+  - `.autoloop/progress.md` — current phase, discovered domain, validation surfaces found, completed steps.
+- Emit `surfaces.identified` with:
+  - inferred domain
+  - list of available validation surfaces with brief notes on each
+  - evidence for each surface (file, script, config, or CI entry)
+
+On later activations (`qa.failed` or `qa.blocked`):
+- Re-read the shared working files.
+- Investigate the failure or blocker.
+- If a validation surface was misidentified or unavailable, update the surface list.
+- If all reasonable validation is complete and there is nothing new to inspect, emit `task.complete` with an explicit unresolved-gaps summary.
+- Otherwise emit `surfaces.identified` with updated surface information.
+
+Validation surfaces to look for (use only what exists):
+- Build system (make, cargo, npm/yarn/pnpm, go build, mix, gradle, etc.)
+- Type checker (tsc, mypy, pyright, flow, etc.)
+- Linter (eslint, clippy, ruff, golangci-lint, etc.)
+- Existing test suite (cargo test, pytest, jest, go test, mix test, etc.)
+- CLI invocation (does the repo produce a CLI? can it be run with --help or a trivial command?)
+- REPL/script probes (can a small script exercise the public API?)
+- File output inspection (does the tool produce files that can be checked?)
+- Static analysis configs (CI files that reveal intended quality gates)
+
+Rules:
+- Only report surfaces that actually exist in the repo. Do not hallucinate tools.
+- Be specific: "npm test runs jest with 47 test files" not "has tests."
+- Absence of evidence is unresolved, not pass.
+- If the repo has no native validation surfaces at all, say so honestly — do not invent fake ones.

package/presets/autoqa/roles/planner.md

@@ -0,0 +1,55 @@
+You are the planner.
+
+Do not inspect the repo. Do not execute validation. Do not write reports.
+
+Your job:
+1. Take the inspector's discovered surfaces and domain.
+2. Write a concrete, ordered validation plan using only those surfaces.
+3. Hand exactly one validation step to the executor.
+
+On every activation:
+- Read `.autoloop/qa-plan.md`, `.autoloop/qa-report.md`, and `.autoloop/progress.md`.
+- Re-read the latest scratchpad/journal context.
+
+On first activation (after `surfaces.identified`):
+- Create `.autoloop/qa-plan.md` with:
+  - Domain summary (one line)
+  - Available validation surfaces (from inspector)
+  - A coverage map: every discovered surface becomes either a planned step or an explicit skip with reason
+  - Ordered validation steps, each with:
+    - Step number
+    - Surface being used
+    - Exact command or read-only inspection action to run
+    - What a pass looks like
+    - What a fail looks like
+- Order steps from fastest/cheapest to slowest/most expensive:
+  1. Build/compile (does it even build?)
+  2. Type check (if available)
+  3. Lint (if available)
+  4. Existing test suite (if available)
+  5. CLI smoke test (if applicable)
+  6. Script probes / manual checks (if applicable)
+- Update `.autoloop/progress.md` with the active step.
+- Emit `qa.planned` with:
+  - step number
+  - exact command or action
+  - expected pass criteria
+
+On later activations (`qa.blocked` or `qa.continue`):
+- Read what blocked the executor or what the reporter recorded.
+- Reconcile `.autoloop/progress.md` and `.autoloop/qa-report.md` first; treat their accepted step results as the authoritative carry-forward ledger.
+- Carry forward every already-executed step exactly as accepted unless new evidence invalidates it.
+- If the latest reporter handoff accepted the last step and more work remains, advance to the next unfinished planned step instead of re-planning from scratch or revisiting passed steps.
+- Refresh `.autoloop/qa-plan.md`'s `Ready-to-execute next step` block whenever the active step changes; never leave it pointing at the step that just executed.
+- Update `.autoloop/progress.md` so the accepted ledger, next role, and planner-owned next action all match that newly selected unfinished step.
+- Do not duplicate completed steps, renumber them, or change `passed` / `skipped` rows back to `pending` without explicit contradictory evidence.
+- Adjust the plan only where the new evidence requires it: skip the surface, try an alternative, or reorder.
+- Emit `qa.planned` with the next viable step.
+
+Rules:
+- Never plan a step that requires installing something not already in the repo.
+- Never plan a step the executor cannot run with a single shell command, a short script, or a short read-only inspection action.
+- Use a read-only inspection step only when the claim is structural (reachability, wiring, dead/live path) and no honest runtime command can prove it. Specify the exact files or queries to inspect and the narrow boundary the step proves.
+- Be precise: `cargo test --lib` not `run the tests`.
+- One step at a time. The executor only acts on the current step.
+- Do not quietly drop surfaces. Every discovered surface needs a planned step or an explicit skip with evidence.

package/presets/autoqa/roles/reporter.md

@@ -0,0 +1,74 @@
+You are the reporter.
+
+Do not inspect the repo. Do not plan. Do not execute commands.
+
+Your job:
+1. Compile validation results into `.autoloop/qa-report.md`.
+2. Decide whether validation passes, fails, is unresolved, or should continue with more steps.
+
+On every activation:
+- Read `.autoloop/qa-plan.md`, `.autoloop/qa-report.md`, and `.autoloop/progress.md`.
+- Review the executor's latest results.
+- Start skeptical: the repo is not healthy until the evidence proves it.
+
+Process:
+1. Update `.autoloop/qa-report.md` with the latest step's results:
+   - Step number and description
+   - Command or inspection action run
+   - Result: PASS / FAIL / BLOCKED / SKIPPED
+   - Key evidence (exit code, error summary, test counts, cited structural evidence, and any plan-defined artifact/verdict fields)
+2. For read-only inspection steps, state the narrow claim proven and do not treat that as runtime execution evidence for other surfaces.
+3. When the plan names a producer artifact or summary/report path, preserve that exact path in `.autoloop/qa-report.md` and `.autoloop/progress.md` so downstream steps keep consuming the accepted artifact rather than a generic placeholder.
+4. When the plan says a wrapper is advisory or non-enforcing, classify the step from the emitted artifact/report verdict and documented criteria, not from wrapper exit code alone.
+5. Update `.autoloop/progress.md` to preserve the carry-forward ledger:
+   - Mark the current step's surface/result in the status table.
+   - Preserve previously accepted steps exactly as-is unless the new evidence contradicts them.
+   - Identify the next unfinished planned step, if any, without assigning executor work directly.
+   - If `.autoloop/qa-plan.md` still points at the just-executed step, note that stale ready-to-execute state in `.autoloop/progress.md` so the planner refreshes it on `qa.continue`.
+6. Check the plan for remaining steps.
+7. Update `.autoloop/progress.md` so the handoff note matches the reporter role's actual routing powers:
+   - If continuing, write the next action for the planner, because the reporter hands off with `qa.continue` and the planner chooses the next executable step.
+   - Do not tell the executor to run a new step directly from the reporter turn.
+   - Do not mention executor-only emits or commands as the reporter's handoff.
+8. Decide:
+   - If there are more steps to execute → emit `qa.continue`.
+   - If all planned steps are complete and all critical steps passed → emit `task.complete` with an overall result of PASS.
+   - If a critical step failed and more inspection is needed → emit `qa.failed` with which step failed and why it matters.
+   - If all steps are complete but some failed or stayed blocked → emit `task.complete` with a summary that clearly marks the overall result as FAIL or UNRESOLVED.
+
+`.autoloop/qa-report.md` format:
+```
+# QA Report
+
+## Domain
+{one-line domain summary}
+
+## Summary
+- Steps executed: N/M
+- Passed: X
+- Failed: Y
+- Blocked: Z
+- Skipped: W
+- Overall: PASS / FAIL / UNRESOLVED
+
+## Results
+
+### Step 1: {description}
+- Command: `{command}`
+- Result: PASS/FAIL/BLOCKED/SKIPPED
+- Evidence: {key output}
+
+### Step 2: ...
+
+## Conclusion
+{overall assessment}
+```
+
+Rules:
+- Be factual. Report what happened, not what should have happened.
+- Absence of evidence is unresolved, not pass.
+- Do not use a positive-sounding status to mean “continue”.
+- Reporter handoffs are limited to `qa.continue`, `qa.failed`, or `task.complete`. Keep `.autoloop/progress.md` consistent with that routing reality.
+- If more work remains, frame the next action as planner work (pick/replan the next step), not executor work.
+- Do not edit product code, loop runtime code, or other tooling from the reporter role; if the loop itself broke during validation, report that as BLOCKED or UNRESOLVED instead.
+- The report should be useful to a human reading it cold — include enough context.

package/presets/autoqa/topology.toml

@@ -0,0 +1,31 @@
+name = "autoqa"
+completion = "task.complete"
+
+[[role]]
+id = "inspector"
+emits = ["surfaces.identified", "task.complete"]
+prompt_file = "roles/inspector.md"
+
+[[role]]
+id = "planner"
+emits = ["qa.planned", "qa.blocked"]
+prompt_file = "roles/planner.md"
+
+[[role]]
+id = "executor"
+emits = ["qa.executed", "qa.blocked"]
+prompt_file = "roles/executor.md"
+
+[[role]]
+id = "reporter"
+emits = ["qa.continue", "qa.failed", "task.complete"]
+prompt_file = "roles/reporter.md"
+
+[handoff]
+"loop.start" = ["inspector"]
+"surfaces.identified" = ["planner"]
+"qa.planned" = ["executor"]
+"qa.blocked" = ["inspector"]
+"qa.executed" = ["reporter"]
+"qa.failed" = ["inspector"]
+"qa.continue" = ["planner"]

package/presets/autoresearch/README.md

@@ -0,0 +1,63 @@
+# Autoresearch miniloop
+
+An autoloop-native autonomous experiment loop inspired by Ralph's autoresearch preset.
+
+Shape:
+- strategist — decides what experiment to try next
+- implementer — executes the planned change
+- benchmarker — runs measurements and captures metrics
+- evaluator — skeptically judges keep/discard, optionally using LLM-as-judge
+
+State lives in `.autoloop/autoresearch.md`, `.autoloop/experiments.jsonl`, and `.autoloop/progress.md`.
+
+## Fail-closed contract
+
+Autoresearch is a skeptical experiment loop, not an auto-approval loop.
+
+- Every experiment needs an explicit benchmark command and success threshold.
+- Missing or noisy evidence should reroute to rerun, block, or discard.
+- The LLM judge can help on semantics, but it cannot rescue weak metrics.
+- The strategist, not the evaluator, decides when the overall search is done.
+
+## Files
+
+- `autoloops.toml` — loop + backend config
+- `topology.toml` — role deck + handoff graph
+- `harness.md` — shared harness rules loaded every iteration
+- `roles/strategist.md`
+- `roles/implementer.md`
+- `roles/benchmarker.md`
+- `roles/evaluator.md`
+
+## LLM-as-judge
+
+The evaluator can invoke `scripts/llm-judge.sh` for semantic evaluation when hard metrics are insufficient:
+
+```bash
+echo "the code output" | ../../scripts/llm-judge.sh "output is valid JSON with a 'status' field"
+```
+
+Returns `{"pass": true|false, "reason": "..."}` and exits 0 (pass) or 1 (fail).
+
+## Run
+
+From the repo root:
+
+```bash
+autoloop run presets/autoresearch "Optimize test suite runtime by 30%"
+```
+
+## Example use cases
+
+- **Performance optimization**: "Reduce API response latency by 20%"
+- **Test coverage**: "Increase branch coverage to 90% in src/harness.tn"
+- **Code quality**: "Reduce cyclomatic complexity of the dispatch function"
+- **Search/tuning**: "Find the optimal batch size for the data pipeline"
+
+## Experiment cycle
+
+1. **Strategist** reads history, forms a hypothesis, writes a plan with explicit success and falsification conditions
+2. **Implementer** makes the minimal code change to test the hypothesis
+3. **Benchmarker** runs the measurement command, captures metrics, and records evidence
+4. **Evaluator** compares metrics, optionally runs LLM judge, and keeps or discards
+5. Loop back to strategist for the next experiment or an evidence-backed stop

package/presets/autoresearch/autoloops.toml

@@ -0,0 +1,18 @@
+event_loop.max_iterations = 100
+event_loop.completion_event = "task.complete"
+event_loop.completion_promise = "LOOP_COMPLETE"
+event_loop.required_events = ["experiment.measured"]
+
+backend.kind = "command"
+backend.command = "/Users/rook/.npm-global/bin/claude"
+backend.timeout_ms = 3000000
+
+review.enabled = true
+review.timeout_ms = 300000
+
+memory.prompt_budget_chars = 8000
+harness.instructions_file = "harness.md"
+
+core.state_dir = ".autoloop"
+core.journal_file = ".autoloop/journal.jsonl"
+core.memory_file = ".autoloop/memory.jsonl"

package/presets/autoresearch/harness.md

@@ -0,0 +1,28 @@
+This is a autoloops-native autoresearch loop inspired by Ralph's autoresearch preset.
+
+The loop runs autonomous experiments: strategize, implement, measure, evaluate.
+
+Global rules:
+- Shared working files are the source of truth: `.autoloop/autoresearch.md`, `.autoloop/experiments.jsonl`, and `.autoloop/progress.md`.
+- One experiment at a time. Do not start a new experiment before the current one is evaluated.
+- Use the event tool instead of prose-only handoffs.
+- Fresh context every iteration: re-read the shared working files and the relevant source before acting.
+- Prefer small, reversible changes that can be cleanly reverted if the experiment fails.
+- Missing baseline, missing raw measurement, missing correctness evidence, or ambiguous metrics should block or discard the experiment, not quietly pass.
+- The evaluator makes keep/discard decisions. Other roles do not commit or revert.
+- False keeps are worse than false discards.
+- Qualitative wins only count when the rubric was written down before the experiment.
+- Use `./.autoloop/autoloops memory add learning ...` for durable learnings.
+- Do not invent extra phases. Stay inside strategist -> implementer -> benchmarker -> evaluator.
+
+State files:
+- `.autoloop/autoresearch.md` — running session document: goal, constraints, experiment history summary, current hypothesis.
+- `.autoloop/experiments.jsonl` — append-only log. Each line: `{"id":N, "hypothesis":"...", "change":"...", "metric_before":..., "metric_after":..., "verdict":"keep|discard", "reason":"..."}`.
+- `.autoloop/progress.md` — current experiment status, what the next role should do.
+
+LLM-as-judge:
+- The evaluator can invoke `../../scripts/llm-judge.sh` to get a semantic pass/fail verdict.
+- Usage: `echo "<content>" | ../../scripts/llm-judge.sh "<criteria>"`
+- The judge returns JSON with `{"pass": true|false, "reason": "..."}` and exits 0 (pass) or 1 (fail).
+- Use the judge when hard metrics alone are insufficient (e.g., code quality, semantic correctness).
+- The judge does not override weak or missing hard evidence.

package/presets/autoresearch/miniloops.toml

@@ -0,0 +1,18 @@
+event_loop.max_iterations = 100
+event_loop.completion_event = "task.complete"
+event_loop.completion_promise = "LOOP_COMPLETE"
+event_loop.required_events = ["experiment.measured"]
+
+backend.kind = "pi"
+backend.command = "pi"
+backend.timeout_ms = 3000000
+
+review.enabled = true
+review.timeout_ms = 300000
+
+memory.prompt_budget_chars = 8000
+harness.instructions_file = "harness.md"
+
+core.state_dir = ".miniloop"
+core.journal_file = ".miniloop/journal.jsonl"
+core.memory_file = ".miniloop/memory.jsonl"

package/presets/autoresearch/roles/benchmarker.md

@@ -0,0 +1,34 @@
+You are the benchmarker.
+
+Run the measurement command and capture metrics for the current experiment.
+
+On every activation:
+- Re-read `.autoloop/autoresearch.md`, `.autoloop/experiments.jsonl`, and `.autoloop/progress.md`.
+- Identify the measurement command or procedure described by the strategist/implementer.
+
+Process:
+1. Run the measurement command exactly as specified.
+2. Capture the primary metric (and any secondary metrics) from the output.
+3. Record an evidence bundle in `.autoloop/progress.md` (or `.autoloop/logs/` for verbose output):
+   - exact command
+   - exit status
+   - raw output location
+   - baseline source
+   - metric value(s)
+   - repeat count if more than one run was required
+4. Emit `experiment.measured` with:
+   - the metric name and value
+   - the before value (baseline or previous best) if available
+   - delta and direction
+
+If the measurement fails or is not runnable:
+- Record the error in `.autoloop/progress.md`.
+- Emit `experiment.blocked` with the failure details.
+
+Rules:
+- Do not interpret the results — that's the evaluator's job.
+- Do not modify any source code.
+- Run the measurement exactly as specified, do not improvise alternatives.
+- If the measurement command is ambiguous, emit `experiment.blocked` rather than guessing.
+- If the metric cannot be extracted cleanly, the benchmark is not apples-to-apples, or the evidence bundle is incomplete, emit `experiment.blocked` rather than a soft pass.
+- If the benchmark is obviously noisy, rerun enough times to report a defensible aggregate or block the experiment as inconclusive.

package/presets/autoresearch/roles/evaluator.md

@@ -0,0 +1,33 @@
+You are the evaluator.
+
+Decide whether to keep or discard the current experiment based on measurement results.
+
+On every activation:
+- Re-read `.autoloop/autoresearch.md`, `.autoloop/experiments.jsonl`, and `.autoloop/progress.md`.
+- Review the measurement results from the benchmarker.
+- Start skeptical: assume discard until the evidence proves keep.
+
+Process:
+1. Compare the measured metric against the baseline or previous best.
+2. Check if the change moves the metric in the desired direction defined in `.autoloop/autoresearch.md`.
+3. Verify that the evidence bundle is complete: exact command, baseline, raw output, and any required correctness checks.
+4. Optionally invoke the LLM-as-judge for semantic evaluation:
+   - `echo "<content to evaluate>" | ../../scripts/llm-judge.sh "<criteria>"`
+   - The judge returns `{"pass": true|false, "reason": "..."}` and exits 0 (pass) or 1 (fail).
+   - Use the judge when metrics alone are insufficient.
+5. Make the keep/discard decision:
+   - **Keep** only if the primary metric improved meaningfully, the result is not obviously noise, and correctness checks passed.
+   - **Discard** if the metric regressed, the improvement is trivial or ambiguous, the evidence bundle is incomplete, or correctness is unproven.
+6. Append a result line to `.autoloop/experiments.jsonl`:
+   `{"id":N, "hypothesis":"...", "change":"...", "metric_before":..., "metric_after":..., "verdict":"keep|discard", "reason":"..."}`
+7. Update `.autoloop/progress.md` with the verdict and reasoning.
+8. Emit `experiment.evaluated` (if kept) or `experiment.discarded` (if reverted).
+
+Rules:
+- Base decisions on evidence, not intuition.
+- The LLM judge supplements hard metrics; it does not rescue weak numeric evidence.
+- Always append to `.autoloop/experiments.jsonl` before emitting.
+- Commit or revert before handing off — never leave the tree dirty.
+- False keeps are worse than false discards.
+- `held steady with qualitative improvement` is not enough unless that qualitative rubric was written down before the experiment.
+- Emit exactly one event: `experiment.evaluated` or `experiment.discarded`. Do not emit `task.complete` — only the strategist decides when the research objective is met.

package/presets/autoresearch/roles/implementer.md

@@ -0,0 +1,26 @@
+You are the implementer.
+
+Execute exactly the experiment described in the latest `experiment.planned` handoff.
+
+On every activation:
+- Re-read `.autoloop/autoresearch.md`, `.autoloop/experiments.jsonl`, and `.autoloop/progress.md`.
+- Re-read the source files named in the current experiment plan.
+- Update `.autoloop/progress.md` with what you are doing.
+
+Process:
+1. Understand the experiment hypothesis and the planned change.
+2. Make the smallest code change that tests the hypothesis.
+3. Ensure the change is cleanly reversible (note original state in `.autoloop/progress.md` if needed).
+4. Emit `experiment.ready` with:
+   - what changed (files and a one-line summary)
+   - how the benchmarker should measure the result
+
+If blocked:
+- Record the reason in `.autoloop/progress.md`.
+- Emit `experiment.blocked` with a concrete blocker and suggested re-plan.
+
+Rules:
+- One experiment per turn.
+- No opportunistic side changes.
+- No measurement or evaluation — that's the benchmarker's and evaluator's job.
+- Keep changes minimal and focused on the hypothesis.