ai-test-loop 0.0.0-stage → 1.0.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Samuel Medeiros
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,145 @@
-# Temporary Holding Version
+# ai-test-loop
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+**Your agent is grading its own homework.** This makes it impossible: a critic script runs real
+checks, scores the artifact 0–100, loops until the threshold is met, and a fail-closed gate blocks
+delivery on security findings.
+
+![loop](https://github.com/Samuelfmedeiros/ai-test-loop/actions/workflows/loop.yml/badge.svg)
+
+No API keys, no dependencies, no model calls. `loop_runner.py` is deliberately dumb: exit codes and
+optional output patterns decide pass/fail. The model never gets to *say* the work is good — it has
+to *show* it.
+
+> **Quer isso pronto para o seu time?** Os packs com entrega por e-mail, o gate com relatório
+> e o guia de verificação estão em **https://skills.seu.pet/gratis/ai-test-loop**
+
+---
+
+## The problem
+
+Ask an agent "is this done?" and it answers about its own reasoning. The same context that produced
+the artifact also decides it is correct, so real defects survive to delivery:
+
+- a handler that is never called (tests pass, feature dead)
+- a test that passes because it asserts nothing
+- a page that breaks only at mobile width
+- a migration that works on the author's data and nowhere else
+- a "fix" that reverted a config someone changed on purpose
+
+Running "one more review pass" does not help, because the reviewer and the builder are the same
+process. Separating them is the whole trick.
+
+## Quickstart
+
+```bash
+git clone https://github.com/Samuelfmedeiros/ai-test-loop
+cd ai-test-loop
+
+# 1. describe your checks (name, command, weight — weights sum to 100)
+cat > loop.json <<'JSON'
+{
+  "name": "my-project",
+  "threshold": 85,
+  "max_rounds": 8,
+  "checks": [
+    {"name": "unit tests", "cmd": "pytest -q", "weight": 35, "timeout_s": 900},
+    {"name": "type check", "cmd": "mypy src", "weight": 10},
+    {"name": "build",      "cmd": "npm run build", "weight": 20},
+    {"name": "api up",     "cmd": "curl -sS 127.0.0.1:8000/health", "weight": 15,
+     "must_match": "\"status\"\\s*:\\s*\"ok\""},
+    {"name": "no secrets", "cmd": "gitleaks detect --no-banner", "weight": 20,
+     "fail_if_output_matches": "leaks found: [1-9]", "required": true}
+  ]
+}
+JSON
+
+# 2. run the critic
+python3 scripts/loop_runner.py --config loop.json
+```
+
+Output is machine-readable, so the *agent* can act on it and a CI job can gate on it:
+
+```
+SCORE=72
+GAPS=["build: command exited 1", "api up: output did not match pattern"]
+EVIDENCE=.loop/report-1.json
+```
+
+Fix the **single largest** gap, run `--round 2`, repeat. Cap the rounds (8 is plenty): if the score
+never converges, the check set is wrong, not the code.
+
+A self-contained demo lives in `examples/demo` and runs in CI on every push, so you can see a real
+pass and a real fail without configuring anything.
+
+## The four roles (never merge them)
+
+| Role | Who | Job |
+|---|---|---|
+| Orchestrator | the agent running the task | picks checks, runs rounds, owns the report |
+| Builder | the agent/model that wrote the artifact | fixes one class of gap per pass |
+| Critic | this script + your checks | runs real commands, returns score + gaps |
+| Reviewer | a **separate** context/model | audits critic and builder, can reopen the loop |
+
+Rule zero: builder ≠ critic ≠ reviewer. If one context does all three, the loop is theatre.
+
+## Three rules that make it honest
+
+1. **Every point is earned by a command that actually ran.** No points for "read the code". Weights
+   sum to 100 and the report stores the raw output next to each score.
+2. **`NOT VERIFIABLE` is not a pass.** If a check cannot run (missing device, no network, jail), it
+   scores 0 and the report says why. Silent skips are how a green board lies.
+3. **The security gate is fail-closed.** A secret in the diff or history, a high-severity SAST
+   finding, or an auth/path-traversal/injection issue found in review blocks delivery **even with a
+   perfect score**. An inconclusive answer counts as blocked, never as pass.
+
+## Reviewer panel
+
+For anything other people will use, ask 2–3 specialists instead of one generalist — quality and
+security always, performance only if there is a build or runtime cost, UX only if there is a
+user-facing surface. Each returns strict JSON:
+
+```json
+{"parecer":"CONCORDO|DISCORDO","nota":0-100,"gaps":["..."],"texto":"one paragraph with evidence"}
+```
+
+Consensus → deliver. Majority disagree → reopen the loop with the combined gaps. Prompt in
+`references/reviewer-prompt.md`, starting rubric in `references/scoring-rubric.md`.
+
+## When not to use it
+
+Throwaway exploration, spike branches, and one-line answers where the cost of the loop exceeds the
+cost of being wrong. It is for artifacts a human will rely on.
+
+## Files
+
+| Path | What |
+|---|---|
+| `SKILL.md` | the loop, the scoring rules, the fail-closed gate — drop it in your agent's skills dir |
+| `scripts/loop_runner.py` | the deterministic critic (stdlib only) |
+| `references/reviewer-prompt.md` | independent-review prompt, strict JSON output |
+| `references/scoring-rubric.md` | starting weights |
+| `templates/loop.example.json` | config with the usual check set |
+| `examples/demo/` | self-contained example used by CI |
+
+## Get the full pack
+
+This repo is the free, complete tool. The store has the ready-made packs for the cases where
+writing the checks is the slow part — CI gates, visual regression, MCP servers, MLOps guards:
+
+- Free download / e-mail delivery: **https://skills.seu.pet/gratis/ai-test-loop**
+- Catalog: **https://skills.seu.pet**
+- Want it installed and wired into your repo: **https://skills.seu.pet/servico**
+
+## License
+
+MIT — see `LICENSE`. Use it commercially, fork it, ship it.
+
+---
+
+### Em português
+
+O loop de teste para agentes: o construtor não assina o próprio trabalho. Um crítico roda checagens
+de verdade, dá nota 0–100, repete até o limite e barra a entrega se houver achado de segurança.
+`python3 scripts/loop_runner.py --config loop.json` — sem chave de API e sem dependência.
+
+Versão com entrega por e-mail e os packs prontos: https://skills.seu.pet/gratis/ai-test-loop

package/SKILL.md

@@ -0,0 +1,111 @@
+---
+name: ai-test-loop
+description: "Use before declaring any work done or shipped, when the agent must verify its own output, when a deliverable needs a score before release, when the builder keeps approving its own work, or when repeated review rounds never converge. Runs an independent critic that scores the artifact 0-100 with real checks, loops until the threshold, and blocks delivery on a fail-closed security gate."
+---
+
+# AI Test Loop — never ship unverified work
+
+The builder is not allowed to grade its own homework. This skill turns "looks done" into
+"verified, scored, and independently reviewed" — the loop either reaches the threshold with
+evidence, or it reports exactly which gaps remain.
+
+Use it whenever an agent produced something a human will rely on: code, UI, a pipeline, a
+document, a data migration. Not for throwaway exploration.
+
+## The four roles (never merge them)
+
+| Role | Who | Job |
+|---|---|---|
+| Orchestrator | the agent running the task | picks checks, runs rounds, fixes infrastructure, owns the report |
+| Builder | the agent/model that wrote the artifact | fixes gaps, one class of gap per pass |
+| Critic | a **script you write** (`loop_runner.py` + your checks) | runs real commands, returns a 0-100 score and a gap list |
+| Reviewer | a **separate** agent/model context | audits critic + builder, votes, can reopen the loop |
+
+Rule zero: builder ≠ critic ≠ reviewer. If the same context does all three, the loop is theatre.
+Run steps 1-3 below in the working session, then drive the script for the machine part.
+
+## The loop
+
+1. **Declare the checks before writing code.** For each check: name, command, weight, and what
+   "pass" means. Weights must sum to 100.
+2. **Run the critic** — `python scripts/loop_runner.py --config loop.json --round N`. It executes
+   every command for real, records raw output as evidence, and prints `SCORE=<0-100>` and
+   `GAPS=[...]`. It never asks a model whether the work is good; it measures.
+3. **Score, compare, decide.** `SCORE >= threshold` → go to the reviewer. Otherwise pick the
+   **single largest** gap, fix it, and re-run. Cap the rounds (8 is plenty); if it never
+   converges, the check set is wrong, not the code.
+4. **Independent review** before delivery — same artifact, fresh context, no justification from
+   the builder: `references/reviewer-prompt.md`. The reviewer returns strict JSON.
+5. **Security gate, fail-closed.** Any secret in the diff/history, any high-severity SAST finding,
+   any auth/path-traversal/injection issue found by review blocks delivery even with perfect
+   tests. Treat an inconclusive answer as blocked, never as pass.
+6. **Report** with the score, the evidence, the reviewer verdict, and what was *not* verifiable.
+
+## Scoring
+
+Weights are yours to set; start from `references/scoring-rubric.md`. Non-negotiables:
+
+- Every point must be earned by a command that actually ran. No points for "read the code".
+- Evidence is captured, not summarised: keep the raw output next to the score.
+- If a check cannot run (no environment, no device, dependency missing), it scores 0 and the
+  report says `NOT VERIFIABLE: <reason>` — never silently skipped, never counted as passing.
+
+## Reviewer panel (for anything that will be used by others)
+
+Ask 2-3 specialists instead of one generalist, each returning strict JSON:
+`{"parecer":"CONCORDO|DISCORDO","nota":0-100,"gaps":["..."],"texto":"one paragraph with evidence"}`.
+
+- Profiles by nature of the work: quality (always), security (always), performance (only if the
+  artifact has a build or a runtime cost), UX (only if there is a user-facing surface).
+- **Consensus** → deliver. **Majority disagree** → reopen the loop with the combined gaps.
+  **Security disagrees or is inconclusive** → block. **Tie 1-1** → a third specialist decides.
+- Do not send an empty `gaps` array as a formality: a reviewer that finds nothing must still cite
+  the evidence it inspected.
+
+## Pitfalls (each one measured in production)
+
+- **Template echo.** A model asked to answer with an enum often echoes the enum back
+  (`"CONCORDO|DISCORDO"`). Reject any value that matches the literal template, treat it as
+  inconclusive, retry once, then fail closed. A parser that accepts the echo reopens the loop
+  forever with a false gap.
+- **Piping hides the exit code.** `pytest | tail` returns the exit code of `tail` (0) — the critic
+  scores 100 with a failing test hidden in the output. Use `set -o pipefail` (or read exit codes
+  without a pipe) inside every check.
+- **A timeout shorter than the check is a silent zero.** When the critic is killed by its own
+  timeout, the orchestrator sees a score of 0 with an empty gap list and loops with nothing to fix.
+  Set the timeout above the real duration and always keep the critic's raw log.
+- **A short-circuiting "pass" is not a pass.** A server that responds 200 may be the wrong
+  server: on a port collision the check tests a different app and "fails" for the wrong reason.
+  Assert identity, not reachability (title, version, a known payload).
+- **Stale baselines.** A visual or snapshot baseline regenerated "to make it green" erases the
+  regression. Regenerating is allowed only when the change was intended — and it must be stated
+  in the report, with the diff before and after.
+- **Cold start.** First call to a cold service can take seconds; re-test before declaring a
+  timeout, and record that you retested.
+- **Fixing many gap classes at once.** One worker per gap class keeps the next verdict readable.
+  Batch everything and you cannot tell which change moved the score.
+
+## Output format
+
+```
+ROUND: <n>   SCORE: <0-100>   THRESHOLD: <t>   GATE: <fail-closed | passed>
+CHECKS: <name> PASS|FAIL <weight> — <one line of real evidence>
+GAPS: <sorted by weight, biggest first>
+REVIEWER: <CONCORDO|DISCORDO> <nota> — <evidence cited> | panel: q=… s=… p=… u=…
+SECURITY: <clean | BLOCKED: <finding>> | INCONCLUSIVE → blocked
+NOT VERIFIABLE: <check — reason>
+NEXT: <single next action>
+```
+
+## Scope and network access
+
+Runs only the commands you put in the config, on your machine. No network calls of its own, no
+telemetry, no uploads; the security step greps the repo locally. It never deploys and never
+merges: delivery decisions stay with the human owner.
+
+## Files
+
+- `scripts/loop_runner.py` — the critic: runs weighed checks, scores 0-100, prints gaps, keeps state.
+- `templates/loop.example.json` — a ready-to-edit config; copy it next to the project and adjust.
+- `references/scoring-rubric.md` — starting weights per artifact type.
+- `references/reviewer-prompt.md` — the strict-JSON reviewer prompt plus the anti-echo parser rules.

package/bin/cli.js

@@ -0,0 +1,29 @@
+#!/usr/bin/env node
+// Wrapper do critic determinístico (Python) para uso via `npx ai-test-loop`.
+// Por que existe: o runner é Python puro (stdlib) e a porta de entrada que dev
+// usa é `npx`. O wrapper só acha o interpretador e repassa os argumentos.
+const { spawn } = require("node:child_process");
+const path = require("node:path");
+const fs = require("node:fs");
+
+const script = path.join(__dirname, "..", "scripts", "loop_runner.py");
+if (!fs.existsSync(script)) {
+  console.error("ai-test-loop: runner não encontrado em " + script);
+  process.exit(2);
+}
+
+const candidatos = process.env.AI_TEST_LOOP_PYTHON
+  ? [process.env.AI_TEST_LOOP_PYTHON]
+  : ["python3", "python"];
+
+function tentar(i) {
+  if (i >= candidatos.length) {
+    console.error("ai-test-loop: Python 3 não encontrado (tente AI_TEST_LOOP_PYTHON=/caminho/python3)");
+    process.exit(2);
+  }
+  const p = spawn(candidatos[i], [script, ...process.argv.slice(2)], { stdio: "inherit" });
+  p.on("error", () => tentar(i + 1));
+  p.on("exit", (code, signal) => process.exit(signal ? 1 : code ?? 0));
+}
+
+tentar(0);

package/package.json

@@ -1,6 +1,39 @@
 {
   "name": "ai-test-loop",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
-}
+  "version": "1.0.0",
+  "description": "Deterministic critic for AI agents: runs weighed checks for real, scores 0-100 and blocks delivery below the threshold. Stdlib-only runner.",
+  "license": "MIT",
+  "author": "Samuel Medeiros",
+  "homepage": "https://skills.seu.pet/en/p/ai-test-loop?utm_source=npm&utm_medium=readme&utm_campaign=ai-test-loop",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/Samuelfmedeiros/ai-test-loop.git"
+  },
+  "keywords": [
+    "ai-agents",
+    "test-loop",
+    "quality-gate",
+    "critic",
+    "llm",
+    "claude-code",
+    "codex"
+  ],
+  "bin": {
+    "ai-test-loop": "bin/cli.js"
+  },
+  "files": [
+    "bin/",
+    "scripts/",
+    "templates/",
+    "references/",
+    "SKILL.md",
+    "README.md",
+    "LICENSE"
+  ],
+  "engines": {
+    "node": ">=16"
+  },
+  "scripts": {
+    "smoke": "node bin/cli.js --config templates/loop.example.json --dry-run"
+  }
+}

package/references/reviewer-prompt.md

@@ -0,0 +1,61 @@
+# Reviewer prompt (independent critic) — strict JSON, fail-closed
+
+Send this to a **separate** model context that did not write the artifact. Never paste the
+builder's justification: the reviewer must see the artifact and the evidence, not the defence.
+Include: the artifact (diff or files), the critic report (`loop-report-rN.json`), the acceptance
+criteria, and the raw evidence of the checks that failed.
+
+---
+
+You are the independent reviewer of a deliverable you did not write. You have no stake in it
+being accepted. Your job is to find what is wrong, not to be agreeable.
+
+Artifact: <paste diff / file list / content>
+Critic report: <paste SCORE, GAPS, evidence>
+Acceptance criteria: <paste>
+
+Answer with ONE JSON object and nothing else:
+
+{
+  "parecer": "CONCORDO" | "DISCORDO",
+  "nota": <integer 0-100>,
+  "gaps": ["<gap 1>", "<gap 2>"],
+  "texto": "<one paragraph citing concrete evidence: file, line, command output>"
+}
+
+Rules:
+
+1. `CONCORDO` means you inspected the artifact and the evidence and believe it is safe to deliver
+   at the stated threshold. It does not mean the work is perfect.
+2. `DISCORDO` requires at least one concrete gap with evidence. A gap is actionable and specific
+   ("handler at src/api.py:88 returns 500 when payload is empty"), never vague ("could be cleaner").
+3. Check these in order, and stop at the first blocking finding:
+   a. Do the checks actually exercise the deliverable, or do they test a different process,
+      an old build, or a mock of the very thing under test?
+   b. Is any security-relevant change present (secrets, auth, crypto, injection, path traversal,
+      deserialisation, dependency additions)? If yes, judge it explicitly.
+   c. Are there irreversible or destructive operations (DELETE, DROP, TRUNCATE, force push,
+      overwrite of production data) without a stated backup and rollback?
+   d. Does the score come from real commands, with raw evidence, or from a summary someone wrote?
+4. If you cannot decide because information is missing, answer `DISCORDO` with
+   `"gaps": ["INCONCLUSIVE: <what is missing>"]` and `"nota": 0`. Inconclusive blocks.
+5. Do not reward length, style, or effort. Only correctness, evidence, and safety.
+6. Cite at least one piece of inspected evidence in `texto`, even when you agree.
+
+---
+
+## Parser rules (put these in the orchestrator, not in the model)
+
+- **Anti-echo.** Models frequently return the enum template literally: `"parecer": "CONCORDO|DISCORDO"`
+  or `nota: <integer 0-100>`. If the value equals the template, contains `|`, or is not one of the
+  allowed literals, treat the answer as inconclusive: retry once with the template restated as a
+  plain sentence, then fail closed (block).
+- **Strict parse.** Extract the first balanced `{...}` block; `json.loads` it. Anything else (prose,
+  markdown fences, trailing commentary) is ignored, not interpreted.
+- **Missing keys.** A missing `gaps` list is treated as `[]`; a missing `nota` is treated as 0.
+- **Numeric bounds.** Clamp `nota` to 0-100; a float is rounded down.
+- **Consensus.** Panel of 2-3 reviewers, all `CONCORDO` → deliver. Any `DISCORDO` → reopen the loop
+  with the union of gaps. Tie → a third reviewer decides. Security `DISCORDO` or inconclusive →
+  block, regardless of the other votes and of the score.
+- **Cost control.** Reviewer calls cost tokens: send the diff and the failing evidence, not the
+  whole repository, and keep a per-round cap on reviewer invocations.

package/references/scoring-rubric.md

@@ -0,0 +1,60 @@
+# Starting weights per artifact type
+
+Weights are normalised by `loop_runner.py`, so they need not sum to 100 — but keep them readable.
+A check with weight 0 is useless (it can never move the score), and a check with weight 80 makes
+every other check decoration. Copy the row that matches the artifact and tune after one round.
+
+## Backend / API service
+
+| Check | Command example | Weight |
+|---|---|---|
+| unit tests | `pytest -q` | 35 |
+| integration / e2e | `pytest -q tests/e2e` | 20 |
+| type check | `mypy src` | 10 |
+| real HTTP call | `curl -sS localhost:8000/health` with `must_match` on a known payload | 15 |
+| no secrets | `gitleaks detect --no-banner` with `fail_if_output_matches` | 20 |
+
+## Web app / UI
+
+| Check | Command example | Weight |
+|---|---|---|
+| build | `npm run build` | 25 |
+| unit tests | `npm test -- --run` | 20 |
+| e2e smoke | `npx playwright test --grep @smoke` | 25 |
+| desktop + mobile screenshot | screenshot script, `must_match` on a non-empty page title | 15 |
+| accessibility | `npx axe` or Lighthouse CI budget | 15 |
+
+Visual baselines: a snapshot diff is a check, never a pass by regeneration. If the baseline was
+regenerated, say so in the report and attach the before/after diff.
+
+## Data pipeline / migration
+
+| Check | Command example | Weight |
+|---|---|---|
+| row counts before vs after | script printing `src=… dst=…` with `must_match` on equality | 30 |
+| schema matches | `alembic check` / `pg_dump --schema-only` diff | 25 |
+| dry run on a copy | pipeline run against a scratch database | 25 |
+| no destructive statements | `grep -nE 'DROP|TRUNCATE'` with `fail_if_output_matches` | 20 |
+
+## Document / report
+
+| Check | Command example | Weight |
+|---|---|---|
+| required sections present | grep for each heading, one check per group | 30 |
+| links resolve | `curl -sI` each URL, `expect_exit: 0` | 20 |
+| numbers traceable to source | command re-deriving the totals | 30 |
+| no placeholders left | `grep -nE 'TODO|TBD|lorem'` with `fail_if_output_matches` | 20 |
+
+## Threshold
+
+- 85-90 for anything user-facing or public.
+- 95-100 for money, auth, migrations, or irreversible actions — and the security check must be
+  `required: true`.
+- Below 85 the loop keeps going; if three rounds move the score by less than 2 points, the checks
+  are measuring noise — fix the checks before fixing the code.
+
+## Non-blocking checks
+
+Mark `required: false` for quality-nice-to-haves (coverage delta, bundle-size budget, docs polish).
+They still score 0 when they fail and they are printed as `NON_BLOCKING_GAPS`, but they do not gate
+delivery. Never mark a security, data-integrity or destructive-operation check as non-blocking.

package/scripts/loop_runner.py

@@ -0,0 +1,266 @@
+#!/usr/bin/env python3
+"""loop_runner — the deterministic critic of the AI Test Loop.
+
+Runs a list of weighed checks for real, captures their raw output as evidence, and prints:
+
+    SCORE=<0-100>
+    GAPS=["<check>: <why it failed>", ...]
+    EVIDENCE=<path to the JSON report>
+
+The agent (builder) is not allowed to grade its own work; this script measures. It is
+deliberately dumb: exit codes + optional output patterns decide pass/fail. No model calls,
+no network, no telemetry.
+
+Config (JSON):
+
+{
+  "name": "my-project",
+  "threshold": 85,
+  "max_rounds": 8,
+  "checks": [
+    {"name": "unit tests",   "cmd": "pytest -q",           "weight": 40, "timeout_s": 900},
+    {"name": "build",        "cmd": "npm run build",       "weight": 25, "timeout_s": 600},
+    {"name": "lint",         "cmd": "ruff check .",        "weight": 10},
+    {"name": "no secrets",   "cmd": "gitleaks detect --no-banner", "weight": 15,
+     "fail_if_output_matches": "leaks found: [1-9]"},
+    {"name": "api responds", "cmd": "curl -sS localhost:8000/health", "weight": 10,
+     "must_match": "\"status\"\\s*:\\s*\"ok\""}
+  ]
+}
+
+Check keys:
+  cmd                      (required) shell command; runs with pipefail semantics via bash -o pipefail -c
+  weight                   (required) points; all weights are normalised, so they need not sum to 100
+  timeout_s                (optional, default 900) hard timeout — a timeout is a FAIL, never a skip
+  expect_exit              (optional, default 0) exit code that means pass
+  must_match               (optional) regex that MUST appear in combined output
+  fail_if_output_matches   (optional) regex that fails the check when it appears
+  required                 (optional, default true) when false, a FAIL scores 0 but is reported as a
+                           non-blocking gap (useful for "nice to have" checks)
+  cwd                      (optional) working directory for this check
+
+Usage:
+    python loop_runner.py --config loop.json [--round 2] [--json] [--quiet]
+    python loop_runner.py --config loop.json --show-state      # print round history and stop
+    python loop_runner.py --config loop.json --dry-run         # list checks, run nothing
+
+Exit code: 0 when SCORE >= threshold (gate passed), 1 when below, 2 on usage/config error.
+
+State: <config dir>/.<config-stem>-state.json accumulates every round, so a crash or a restart
+never loses the evidence trail. The report of the latest round is written to
+<config dir>/<config-stem>-report-r<round>.json. Names are derived from the config file so two
+projects in the same folder never share a history.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import pathlib
+import re
+import shlex
+import subprocess
+import sys
+import time
+from datetime import datetime, timezone
+
+DEFAULT_TIMEOUT = 900
+MAX_OUTPUT_CHARS = 8000
+
+
+def load_config(path: str) -> dict:
+    try:
+        with open(path, "r", encoding="utf-8") as fh:
+            cfg = json.load(fh)
+    except FileNotFoundError:
+        sys.exit(f"[loop_runner] config not found: {path}")
+    except json.JSONDecodeError as exc:
+        sys.exit(f"[loop_runner] config is not valid JSON: {exc}")
+    checks = cfg.get("checks")
+    if not isinstance(checks, list) or not checks:
+        sys.exit("[loop_runner] config needs a non-empty 'checks' list")
+    for i, chk in enumerate(checks):
+        if not chk.get("cmd"):
+            sys.exit(f"[loop_runner] check #{i} has no 'cmd'")
+        if not isinstance(chk.get("weight", None), (int, float)) or chk["weight"] <= 0:
+            sys.exit(f"[loop_runner] check {chk.get('name', i)!r} needs a positive 'weight'")
+    cfg.setdefault("name", os.path.basename(os.path.dirname(os.path.abspath(path))) or "loop")
+    cfg.setdefault("threshold", 85)
+    cfg.setdefault("max_rounds", 8)
+    return cfg
+
+
+def run_check(chk: dict) -> dict:
+    """Run one check for real, return its result record with raw evidence."""
+    name = chk.get("name") or chk["cmd"][:40]
+    timeout = int(chk.get("timeout_s", DEFAULT_TIMEOUT))
+    cwd = chk.get("cwd") or None
+    expect_exit = int(chk.get("expect_exit", 0))
+    # bash -o pipefail: without it, `pytest | tail` returns tail's exit code (0) and a failing
+    # check scores as passing — the single most common false green in this kind of runner.
+    wrapped = ["bash", "-o", "pipefail", "-c", chk["cmd"]]
+    started = time.time()
+    timed_out = False
+    try:
+        proc = subprocess.run(
+            wrapped, cwd=cwd, capture_output=True, text=True, timeout=timeout,
+            stdin=subprocess.DEVNULL,
+        )
+        rc, out, err = proc.returncode, proc.stdout or "", proc.stderr or ""
+    except subprocess.TimeoutExpired as exc:
+        timed_out = True
+        rc = None
+        out = (exc.stdout or b"").decode("utf-8", "replace") if isinstance(exc.stdout, bytes) else (exc.stdout or "")
+        err = (exc.stderr or b"").decode("utf-8", "replace") if isinstance(exc.stderr, bytes) else (exc.stderr or "")
+    duration = round(time.time() - started, 2)
+    combined = (out or "") + ("\n" + err if err else "")
+
+    reasons: list[str] = []
+    if timed_out:
+        reasons.append(f"timed out after {timeout}s")
+    elif rc != expect_exit:
+        reasons.append(f"exit code {rc}, expected {expect_exit}")
+
+    must = chk.get("must_match")
+    if must and not re.search(must, combined):
+        reasons.append(f"output does not match required pattern /{must}/")
+
+    fail_pat = chk.get("fail_if_output_matches")
+    if fail_pat:
+        hit = re.search(fail_pat, combined)
+        if hit:
+            reasons.append(f"output matched failure pattern /{fail_pat}/: {hit.group(0)!r}")
+
+    passed = not reasons
+    evidence = combined.strip()
+    if len(evidence) > MAX_OUTPUT_CHARS:
+        evidence = evidence[: MAX_OUTPUT_CHARS // 2] + "\n...[truncated]...\n" + evidence[-MAX_OUTPUT_CHARS // 2:]
+    return {
+        "name": name,
+        "cmd": chk["cmd"],
+        "weight": float(chk["weight"]),
+        "required": bool(chk.get("required", True)),
+        "passed": passed,
+        "reasons": reasons,
+        "exit_code": rc,
+        "duration_s": duration,
+        "evidence": evidence,
+    }
+
+
+def score(checks: list[dict]) -> float:
+    total = sum(c["weight"] for c in checks) or 1.0
+    earned = sum(c["weight"] for c in checks if c["passed"])
+    return round(earned / total * 100, 1)
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(add_help=True, description="Deterministic critic for the AI Test Loop")
+    ap.add_argument("--config", required=True)
+    ap.add_argument("--round", type=int)
+    ap.add_argument("--json", action="store_true", help="also dump the full report as JSON to stdout")
+    ap.add_argument("--quiet", action="store_true", help="only SCORE/GAPS/EVIDENCE lines")
+    ap.add_argument("--dry-run", action="store_true")
+    ap.add_argument("--show-state", action="store_true")
+    args = ap.parse_args()
+
+    cfg_path = os.path.abspath(args.config)
+    cfg = load_config(cfg_path)
+    base = os.path.dirname(cfg_path)
+    # Per-config filenames: two configs in the same folder must not share one state file,
+    # otherwise rounds from project A pollute project B's history and round numbers.
+    stem = os.path.splitext(os.path.basename(cfg_path))[0]
+    state_path = os.path.join(base, f".{stem}-state.json")
+
+    if args.show_state:
+        if not os.path.exists(state_path):
+            print("[loop_runner] no state file yet")
+            return 0
+        with open(state_path, "r", encoding="utf-8") as fh:
+            st = json.load(fh)
+        for r in st.get("rounds", []):
+            print(f"round {r['round']}: SCORE={r['score']} threshold={r['threshold']} "
+                  f"failed={[c['name'] for c in r['checks'] if not c['passed']]}")
+        return 0
+
+    if args.dry_run:
+        for c in cfg["checks"]:
+            print(f"[dry-run] {c.get('name', c['cmd'][:40])}  weight={c['weight']}  cmd={c['cmd']}")
+        print(f"[dry-run] threshold={cfg['threshold']} max_rounds={cfg['max_rounds']}")
+        return 0
+
+    round_no = args.round
+    if round_no is None:
+        prev = 0
+        if os.path.exists(state_path):
+            with open(state_path, "r", encoding="utf-8") as fh:
+                prev = len(json.load(fh).get("rounds", []))
+        round_no = prev + 1
+
+    if round_no > int(cfg["max_rounds"]):
+        print(f"ROUND: {round_no}  ABORT: exceeded max_rounds={cfg['max_rounds']}")
+        print("GAPS=[\"round budget exhausted — the check set or the threshold needs revisiting\"]")
+        return 1
+
+    results = []
+    for chk in cfg["checks"]:
+        if not args.quiet:
+            print(f"[loop_runner] round {round_no}: running {chk.get('name', chk['cmd'][:40])!r} ...",
+                  file=sys.stderr, flush=True)
+        results.append(run_check(chk))
+
+    value = score(results)
+    threshold = float(cfg["threshold"])
+    gate = value >= threshold
+    gaps = [f"{c['name']}: {'; '.join(c['reasons'])}" for c in results if not c["passed"] and c["required"]]
+    soft = [f"{c['name']}: {'; '.join(c['reasons'])}" for c in results if not c["passed"] and not c["required"]]
+
+    report = {
+        "project": cfg["name"],
+        "round": round_no,
+        "ts": datetime.now(timezone.utc).isoformat(timespec="seconds"),
+        "threshold": threshold,
+        "score": value,
+        "gate": "passed" if gate else "failed",
+        "checks": results,
+        "gaps": gaps,
+        "non_blocking_gaps": soft,
+    }
+    report_path = os.path.join(base, f"{stem}-report-r{round_no}.json")
+    pathlib.Path(report_path).write_text(
+        json.dumps(report, indent=2, ensure_ascii=False), encoding="utf-8")
+
+    state = {"project": cfg["name"], "rounds": []}
+    if os.path.exists(state_path):
+        try:
+            with open(state_path, "r", encoding="utf-8") as fh:
+                state = json.load(fh)
+            state.setdefault("rounds", [])
+        except json.JSONDecodeError:
+            pass
+    state["rounds"].append({"round": round_no, "score": value, "threshold": threshold,
+                            "gate": report["gate"], "ts": report["ts"], "report": report_path,
+                            "checks": results})
+    pathlib.Path(state_path).write_text(
+        json.dumps(state, indent=2, ensure_ascii=False), encoding="utf-8")
+
+    if not args.quiet:
+        for c in results:
+            status = "PASS" if c["passed"] else "FAIL"
+            detail = c["evidence"].strip().splitlines()[-1][:160] if c["evidence"].strip() else "(no output)"
+            print(f"  {status}  {c['name']:<28} w={c['weight']:<5} {c['duration_s']:>7}s  {detail}")
+
+    print(f"SCORE={value}")
+    print(f"THRESHOLD={threshold}")
+    print(f"GATE={'passed' if gate else 'failed'} (round {round_no}/{cfg['max_rounds']})")
+    print(f"GAPS={json.dumps(gaps, ensure_ascii=False)}")
+    if soft:
+        print(f"NON_BLOCKING_GAPS={json.dumps(soft, ensure_ascii=False)}")
+    print(f"EVIDENCE={report_path}")
+    if args.json:
+        print(json.dumps(report, ensure_ascii=False, indent=2))
+    return 0 if gate else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

package/templates/loop.example.json

@@ -0,0 +1,15 @@
+{
+  "name": "my-project",
+  "threshold": 85,
+  "max_rounds": 8,
+  "checks": [
+    {"name": "unit tests",   "cmd": "pytest -q",              "weight": 35, "timeout_s": 900},
+    {"name": "type check",   "cmd": "mypy src",               "weight": 10, "timeout_s": 300},
+    {"name": "build",        "cmd": "npm run build",          "weight": 20, "timeout_s": 600},
+    {"name": "api responds", "cmd": "curl -sS 127.0.0.1:8000/health", "weight": 15,
+     "must_match": "\"status\"\\s*:\\s*\"ok\""},
+    {"name": "no secrets",   "cmd": "gitleaks detect --no-banner", "weight": 20,
+     "fail_if_output_matches": "leaks found: [1-9]",
+     "required": true}
+  ]
+}