selfheal-api 1.0.0rc1__tar.gz

selfheal_api-1.0.0rc1/.dockerignore

@@ -0,0 +1,13 @@
+# Keep the build context small and keep secrets/evidence out of images.
+.venv
+.git
+.pytest_cache
+__pycache__
+*.pyc
+.env
+results
+results.bak
+logs
+logs.bak
+traces
+data

selfheal_api-1.0.0rc1/.python-version

@@ -0,0 +1 @@
+3.14.6

selfheal_api-1.0.0rc1/Containerfile.pipeline

@@ -0,0 +1,39 @@
+# Service C: the pipeline -- tails service A's log, runs the stages on alerts, serves the API/UI.
+FROM python:3.13-slim
+WORKDIR /app
+
+# `openssl dgst -sign` signs the GitHub App's JWT (selfheal/app_check.py): the key file is passed by path and never read by Python.
+RUN apt-get update && apt-get install -y --no-install-recommends openssl \
+    && rm -rf /var/lib/apt/lists/*
+
+COPY pyproject.toml README.md constraints.txt constraints-image.txt ./
+COPY selfheal/ selfheal/
+# ONE pip call with both constraint files, so the resolver sees every requirement at once (installing the OpenTelemetry
+# instrumentation in a second call downgraded api/semantic-conventions under an sdk that needs the newer ones). The
+# pipeline imports the customer's app, so that app's own imports (opentelemetry-instrumentation-fastapi) must resolve here too.
+# SELFHEAL_VERSION: the build context has no git history, so `selfheal --version` would say 0.0.0+unknown; pass the real version
+# (docker-compose.yml forwards $SELFHEAL_VERSION and $SELFHEAL_COMMIT; `selfheal runner-install --print-version` prints the version).
+# `pip check` fails the build on any remaining conflict; pip itself only warns about conflicts in already-installed packages.
+ARG SELFHEAL_VERSION=0.0.0+unknown
+ARG SELFHEAL_COMMIT=unknown
+RUN SETUPTOOLS_SCM_PRETEND_VERSION_FOR_SELFHEAL_API=$SELFHEAL_VERSION \
+    pip install --no-cache-dir -c constraints.txt -c constraints-image.txt . opentelemetry-instrumentation-fastapi \
+    && pip check
+# The same install record `runner-install` writes, so `selfheal --version` names the commit the image was built from.
+RUN python3 -c "import json, os, sys, datetime; c = os.environ['SELFHEAL_COMMIT']; \
+c != 'unknown' and open(os.path.join(sys.prefix, 'selfheal-install.json'), 'w').write(json.dumps({'commit': c, \
+'version': os.environ['SELFHEAL_VERSION'], 'installed_at': datetime.datetime.now(datetime.timezone.utc).isoformat(timespec='seconds'), \
+'installed_by': 'image build'}))"
+
+ENV PYTHONDONTWRITEBYTECODE=1 \
+    PIPELINE_APP_CHECKOUT=/demo-api \
+    OTEL_LOG_PATH=/tmp/pipeline-imports-app-but-never-logs.jsonl \
+    PIPELINE_LOG_PATH=/app/logs/otel-logs.jsonl \
+    PIPELINE_DATA_DIR=/app/data \
+    PIPELINE_DB_JOURNAL=DELETE
+EXPOSE 8100
+
+HEALTHCHECK --interval=5s --timeout=2s --start-period=10s --retries=5 \
+    CMD python3 -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8100/health', timeout=2)"
+
+CMD ["selfheal", "serve"]

selfheal_api-1.0.0rc1/FINDINGS.md

@@ -0,0 +1,157 @@
+# jevbrief-eval: Findings and Final Thresholds
+
+The "decide" half of the self-healing-APIs pipeline (collect → classify → second
+opinion → gate) as evaluated against 6 scenarios, 6 goal-wording variants, and ~280
+real Jev calls across this evaluation phase. This is the write-up requested at the end
+of that work: final thresholds, what each fix actually changed, the one open item, and
+the one thing this data cannot tell us.
+
+## Final thresholds
+
+```json
+{"T_incident": 0.8, "T_class": 0.8, "M": 0.2}
+```
+
+**Unchanged from the original provisional guess.** A full sweep (`tune_thresholds.py`)
+across both axes from 0.50 to 0.95 found **unsafe stays at 0 everywhere in that range**
+— see "The limitation" below for why that means these numbers aren't actually
+*validated*, just *not contradicted*. Lowering them bought measurably more automation
+coverage in the sweep (0.8/0.8 → 7/100 rows proceed+recommend; 0.5/0.5 → 20/100) but
+with no evidence it was buying real safety margin to spend, so they were left alone
+rather than moved for a speculative gain.
+
+## What each fix actually changed
+
+In the order they were built, with the concrete before/after:
+
+1. **Clean, seeded fixtures** (`traffic.py --clean`, `run_scenarios.py --clean <name>`).
+   `baseline()` incidentally trips the null-field and slow-query bugs in every
+   scenario's fixture (ids 4/8 and 5/10 in its loop), so every scenario's log file
+   contained noise from bugs it wasn't testing. `baseline_clean()` fixtures now exist
+   for all 5 scenarios with a real injected failure; the originals are kept as an
+   untouched, checksum-verified archive. This directly fixed a false "wrong pick" on
+   `two_incidents/misleading`, where Jev had confidently (0.96-0.97) picked real
+   leftover baseline noise that the answer key didn't know to expect — not a model
+   error, an incomplete fixture. After the fix: `two_incidents` incident accuracy
+   8/8 (was 6/8).
+
+2. **`two_incidents` class scoring** (`ground_truth.score_class`). Previously scored
+   against a fixed set regardless of which of the two real incidents was actually
+   picked. Now defers to whichever incident's own ground truth applies (null_field's if
+   the `AttributeError` was picked, deliberately unscored if the `KeyError` was, same as
+   `missing_input` alone).
+
+3. **Safety net** (`safety_net.py`). Plain code, independent of the goal and of Jev:
+   scans the raw log for ERROR/5xx records regardless of what jevbrief concluded. Across
+   the final 100-row stress run, it would open an "unexplained errors" ticket on 40/100
+   rows where the goal-driven question alone said `none`/`low_confidence` — including
+   both `load_spike/vague` rows sitting on 15 real 503 rejections that would otherwise
+   have disappeared with no ticket at all. **Silent misses (real error, Jev said none,
+   backstop didn't fire): 0/100** across every run checked. **False fires on `healthy`:
+   0/15.**
+
+4. **Route/status claim check** (`goals.ALERT_CLAIMS` + `report.check_claim_mismatch` +
+   `gate.py`'s `claim_mismatch` field). When the goal makes an explicit route/status
+   claim (`alert`/`misleading` variants), does the picked fact actually match it? A
+   mismatch escalates regardless of confidence. 20/100 rows in the final run made a
+   checkable claim; **0/20 mismatches** — the clean fixtures already removed the one
+   case (the `two_incidents` near-miss above) this would otherwise have caught.
+
+5. **Straddle logic fix** (`gate.py`, `STRADDLE_MIN_RUNNER_UP = 0.10`). This was a real
+   logic bug, not a tuning question: the straddle check ("escalate if the top two
+   classes span agent-fixable and human-only, regardless of confidence") fired on *any*
+   nonzero runner-up probability — so `slow_query`, which classified as `infrastructure`
+   at ~0.999 probability with `bug_fix` at ~0.001, was permanently stuck at `escalate`
+   via straddle no matter how confident or consistent the model was. Now a runner-up
+   needs ≥0.10 probability to count as genuine indecision. Both directions tested
+   (fixable-winner-with-small-human-only-runner-up, and the reverse), plus the boundary
+   case at exactly 0.10. After the fix: **`slow_query` reaches `recommend` in all 10
+   tested `alert` runs** (was stuck at `escalate` in 100% of runs before).
+
+6. **Alert goal template** (`goals.py`, `ALERT_SIGNALS`). The original `alert` goals
+   were terse (`"elevated 5xx on GET /users/{user_id}"`). Rewritten to what a real
+   alerting rule actually emits: service/route, status class or latency, an observed
+   rate, and a time window (`"5xx rate on GET /users/{user_id} rose from 0% to 9% at
+   14:02"`) — confirmed to contain no exception type and no answer (see audit below).
+   This alone, with no threshold change, moved `null_field`'s ticket-class confidence
+   from 0.61-0.70 to **0.80-0.85**, flipping it from permanently `escalate` to
+   `proceed` in all 5 tested runs at the unchanged `T_class=0.8`.
+
+**Audit of the alert templates** (`ALERT_SIGNALS`, checked programmatically against a
+list of leak terms: exception names, "crash", "queue", "saturate", "reject",
+"unhandled", etc.): all 5 templates plus `healthy`'s "all services nominal" are clean —
+each contains only service/route/status-class-or-latency/rate/time, nothing only
+visible by having already solved the incident.
+
+## Open item: `load_spike`
+
+Per-fact inspection of 5 real `load_spike/alert` traces: **every run has exactly one
+kept fact** (the checkout-saturation message) — there is nothing to merge in grouping,
+jevbrief already collapsed all 15 rejection log lines into one group. The probability
+split is consistently `{real fact: 0.76-0.82, "none": 0.18-0.24}` — one fact clearly
+wins every time. Per the branching rule this called for a wording pass, not a grouping
+fix, so one was tried: adding a duration clause (`"...sustained for 6 minutes"`, a
+legitimate real-alert field). It didn't help — 5 fresh calls came back *worse* (0.44-
+0.54, 2/5 dropped to `low_confidence` entirely) — so it was reverted. `goals.py`
+currently matches exactly what the final 100-row run used.
+
+Worth noting separately: **Jev's reported `confidence` and the raw top-choice
+probability are not the same number.** `load_spike/alert` consistently shows
+`probabilities: {real_fact: ~0.78, none: ~0.22}` alongside a *separately reported*
+`confidence` of only 0.52-0.64 for that same choice — a real, non-obvious gap between
+"how the model split its answer across options" and "how sure it says it is," which
+`gate.py` correctly reads from (`incident_confidence` is the latter, not the former),
+but is worth remembering when eyeballing a trace: the probability dict alone can look
+more confident than the number that actually gates the decision.
+
+`load_spike` remains stuck at `escalate` under `alert` goals. This is a genuine open
+item, not a thresholds question — the sweep showed no T_incident value in the tested
+range was unsafe for it, because it's never confident enough to reach `applied` at all,
+let alone cross into unsafe territory.
+
+## The limitation: this data cannot locate a threshold floor
+
+The sweep (`tune_thresholds.py results/eval_20261018T042738Z`, T_incident and T_class
+both from 0.50 to 0.95) found **unsafe = 0 at every single tested combination**. That
+is not the same as "0.8/0.8 is correct" — it means the straddle fix, the route/status
+check, and the rules-vs-Jev agreement check are currently carrying all of the measured
+safety margin in this dataset, and none of the 100 rows in the final run ever gave the
+confidence thresholds themselves a chance to be the thing that mattered. Any specific
+number between 0.5 and 0.95 would have been equally "validated" by this data, which
+means none of them actually are. Moving the thresholds now, in either direction, would
+be a guess dressed up as a measurement. The honest statement is: **thresholds are
+unchanged because nothing in ~280 real calls across two sessions ever produced a
+genuinely unsafe decision to calibrate against** — which is a good sign for the gate's
+other safeguards, and a real gap in what this evaluation can currently prove about the
+confidence numbers specifically. More scenarios with a wider range of genuine ambiguity
+(not just the deliberately-ambiguous `missing_input`/`two_incidents`) would be needed
+before trusting any specific threshold value.
+
+## Test commands
+
+```bash
+cd "/Users/ruchiradnaik/code/self-maintaining-apis/jevbrief-eval"
+source .venv/bin/activate
+
+python3 -m pytest tests/ -v                                          # 54 tests, zero cost
+python3 report.py results/eval_20261018T042738Z                      # the final 100-row run
+python3 tune_thresholds.py results/eval_20261018T042738Z             # reproduce the sweep, zero cost
+```
+
+## Addendum (2026-10-18): final-outcome layer and the infrastructure relaxation
+
+- `gate.py` now lets a low-confidence incident pick through to `recommend` when Jev and the
+  rules both say `infrastructure` (reason carries a `[low confidence: ...]` flag). Re-scored
+  on `results/eval_20261018T042738Z`: gate actions went from 93 escalate / 5 proceed /
+  2 recommend to 90 / 5 / 5 (three `slow_query` rows), vs-ground-truth overcautious 38 -> 35,
+  unsafe still 0. The "recommend in all 10 alert runs" figure above predates this and was not
+  reproduced on the live run on 2026-10-18 (0.64 incident confidence -> escalate).
+- `outcome.py` adds `no_action` and `unexplained_errors_ticket` as final actions.
+- **Known hazard of `no_action` -- FIXED (see IMPLEMENTATION.md, Phase 9):** the paragraph below describes the
+  original problem. An alert-triggered run can no longer end as `no_action` (it ends `alert_unexplained`), the
+  safety net now counts WARN and slow-response records, and a manual run over a log with any of them escalates.
+  Original text: **Known hazard of `no_action`:** the safety net only counts ERROR/5xx records, so a log whose
+  only problem is a WARN-level slow query looks "clean" to it. In the 100-row run all 5
+  `slow_query/misleading` rows (goal about the wrong route; 4 real slow-query WARNs in the log)
+  get Jev `none` at 0.80-0.90 and therefore `no_action` -- previously `escalate`. See the
+  open decision in the session notes before relying on `no_action`.

selfheal_api-1.0.0rc1/FINDINGS_V1.md

@@ -0,0 +1,156 @@
+# FINDINGS_V1
+
+> The first sections below are the record of the original build (plan-then-approve flow) and the first real run; they are kept as written. The second live run, the design changes it led to, and the empty tables for the final measurements are at the end of the file.
+
+Status of the V1 task list (`V1_TASKS.md`) after the build session of 2026-10-18/19. Written to be honest about what has
+and has not been proven, not to look finished.
+
+## What was built and is covered by tests (863 tests, no network, no real budget)
+
+| Task | State |
+|---|---|
+| T0 verify | The three decision fixes and the UI polish are on `main` (PRs #2, #3). **Real-Jev check, 1 call (throwaway DB):** `replay null_field` -> incident pick confidence **0.96**, class `bug_fix` confidence **0.82**, threshold used **`T_class_agreed` 0.7** (rules' specific rule agrees with Jev), final action **proceed**; pre-flight passed (check 3 `unknown`: no stack trace in the frozen fixture). Evaluation report on the latest 40-row run: **unsafe 0 / 34**. Real `data/pipeline.db`: backed up to `data/backups/`, then the 36 manual replay runs (and their 1,116 node events) were deleted; 107 watcher + 19 webhook runs remain. |
+| T1 | `demo-api` repo created locally and on GitHub (private); app, traffic generator (`traffic/`), own compose, CI, pre-commit, `scripts/reset-bugs.sh`, `selfheal.yml`. Pipeline loads the app via `appsource.py` (`PIPELINE_APP_CHECKOUT`); frozen copies of app + traffic under `eval/demo_app_frozen/`; compose runs only the pipeline. |
+| T1b | Stack traces: `stacktrace.py` + demo-api `feature/otel-stacktraces`. Check 3 names the field (`email`) and shows `app.py:182`. |
+| T2 | `TicketSink` / `GitHubIssuesSink` / `FakeTicketSink`, one issue per incident, idempotent create, 30-minute pinned status comment, redacted untrusted samples, `ticket` graph node. |
+| T3 | `GET /api/evidence/{issue}` (workspace token), deterministic endpoint brief, `schema_version` + sha256 stored. |
+| T4 | Plan schema, `one_obvious_fix`, `plan_id`, plan / design-decision comments, every failure path escalates. |
+| T5 | Approval rules (permission, stale plan, wrong option, `/reject`), approvals recorded (who, when, plan_id, option) and shown on the incident. |
+| T6 | Implement step with scope enforcement in code, local commit only. |
+| T7 | `selfheal guard`: spec diff, tests, red-to-green, scope, replay. |
+| T8 | `Reviewer` interface, Claude reviewer (blind then compare), verdict downgrade in code. |
+| T9 | Draft-PR gate, GitHub webhook (signature-verified) -> resolve / return to owner, fix-flow panel, live `plan`/`implement`/`guardrails`/`reviewer`/`draft_pr` nodes. |
+| T10 | `pip install selfheal`, `selfheal init|doctor|serve|guard`; `init` dogfooded on demo-api at `v0-buggy` (output identical to the hand-written `selfheal.yml`). |
+| T11 | An **offline** end-to-end test (`tests/test_end_to_end_offline.py`) drives incident -> issue -> evidence -> plan -> approval -> implement -> guardrails -> reviewer -> draft PR -> merge webhook -> resolved with fakes. |
+
+## What is NOT proven yet
+
+- **No real end-to-end run.** The "twice in a row from a reset" demo, screenshots, timings and cost per fix need real
+  secrets (Claude auth, a GitHub token for the pipeline, the workspace token, a bot login) and a running service. None of these were available
+  to the build session, so there are **no timings or costs** to report. Do not read the offline test as a measurement.
+- **The workflows have never run on GitHub.** The Claude Code CLI flags in `ClaudeCodeRunner` (`--allowed-tools`,
+  `--max-budget-usd`, `--output-format json`), `oasdiff breaking --fail-on WARN`, the Schemathesis invocation and the
+  `npm`/`go install` steps are untested against the real tools.
+- **Nothing is merged.** `demo-api` `main` (tagged `v0-buggy`) and `feature/otel-stacktraces` are on GitHub; the pipeline work is split into seven stacked PRs for you to review and merge in order.
+- **Required check on PRs.** Events caused by the default `GITHUB_TOKEN` do not trigger workflows, so every workflow acts as one fine-grained PAT (`SELFHEAL_GITHUB_TOKEN`; exact permissions in `SETUP.md`), which also makes the bot identity the PAT's user. Not yet exercised on GitHub.
+- **Local connectivity.** Workflows run on a self-hosted runner labelled `selfheal-local` on this Mac, and the pipeline polls GitHub every 60 s instead of relying on webhooks (the webhook handler is kept). `selfheal doctor` checks the runner, the poller/webhook, the secrets, the required `guard` check and the evidence API.
+- **`v0-buggy` predates the stack-trace instrumentation.** `reset-bugs.sh` now resets to `v0-buggy-otel` (tagged after `feature/otel-stacktraces` is merged); `v0-buggy` is unchanged.
+- **Namespace and pins.** `pip install selfheal` uses `>=` ranges, not the pinned `constraints.txt`.
+- **Real DB.** The 36 manual replay runs were removed (backup in `data/backups/`); their trace and slice files under `data/` were left in place.
+
+## Things worth deciding
+
+1. Grant the `workflow` scope, then push `demo-api` and open the PRs (the branches are ready locally).
+2. Provide the secrets listed by `selfheal init`, then run the demo twice and fill in timings and cost here.
+3. Whether the bot identity should be a GitHub App rather than your user (the plan workflow trusts only `SELFHEAL_BOT_LOGIN`).
+
+## The first real run (2026-10-19 onward): what worked and what broke
+
+The first run on real GitHub, a real self-hosted runner, real Claude and real Jev against `demo-api` at `v0-buggy-otel`.
+Everything below was found by that run, not by the offline tests (which is the point of the section above: they could not
+see these). The symptoms are summarised from the commit and PR descriptions, not from saved run logs. Each item lists the symptom, the cause and the fix; PR numbers refer to `ruchiradnaik/self-maintaining-apis`.
+
+### What worked
+
+- Detection to decision on real telemetry: the watcher found the `/users/{user_id}` null-field incident, Jev picked and
+  classified it, the gate said `proceed`, and the pre-flight named the field from the stack trace.
+- The ticket flow end to end up to a human-approved plan: issue created with the evidence link, plan comment from the bot,
+  `/approve` read from a person with write access, the implement step committing only to the approved files.
+- The safety layers held: the bot could not approve, the Claude steps ran in the podman sandbox, the guardrails ran in
+  the network-less container, and the scope check refused anything outside the plan. Nothing unsafe reached a PR.
+- The guard, once fixed (below), told a real fix from a failing one using its per-check table on the issue.
+
+### Problems the run exposed, and the fixes
+
+| # | Problem (symptom) | Cause | Fix |
+|---|---|---|---|
+| 1 | The plan comment was ignored as untrusted / the draft PR did not trigger the required check. | The PAT belonged to a different GitHub account than `SELFHEAL_BOT_LOGIN`, and only the bot's plan comments are trusted. | `selfheal doctor` now calls `GET /user` with `SELFHEAL_GITHUB_TOKEN` and fails unless the real login equals `SELFHEAL_BOT_LOGIN` (and the bot differs from the approver). SETUP.md says the PAT is created while signed in as the bot (PR #13). |
+| 2 | An issue was created and no plan ever appeared. | The issue was created with three labels, which fired three `labeled` events in one concurrency group; GitHub cancelled the `selfheal:plan` run in favour of a later irrelevant one. | The pipeline creates the issue with informational labels, then adds `selfheal:plan` in a separate call. The plan and implement concurrency groups are keyed on whether the event is the trigger: only the trigger shares `selfheal-<plan\|implement>-<issue>-run`, every other event gets a unique group and cannot cancel it. Issue bodies also say "incident 4" instead of "#4" (GitHub autolinked it to issue 4) (PR #13). |
+| 3 | Workflow jobs hung forever on the self-hosted Mac. | `actions/setup-python` runs `sudo installer` and waits for a password; `PATH` was unreliable (conda base was the first interpreter). | Workflows no longer use setup-python or `PATH`. They run only `"$SELFHEAL_BIN"`, the CLI from a dedicated runtime venv that `selfheal runner-install` builds from a commit on `origin/main`; a job fails in its first step if the variable is unset, relative, missing or broken (PR #13, `f9488dd`). |
+| 4 | `/approve` was silently ignored. | GitHub's editor added leading whitespace / zero-width characters; the strict parser saw a different first line. | The approval is read after stripping whitespace and zero-width characters, still strictly (first line, nothing else on it). The workflow starts on `contains('/approve')` because expressions cannot trim, and a near-miss gets one reply with the reason and the exact syntax (`b73ace2`). |
+| 5 | Guard failed a correct fix on the spec diff. | oasdiff's output was read as text, and "No changes detected" was treated as a breaking change. | `spec_diff` parses `--format json` and fails only on actual breaking entries; "No changes detected" is a pass. oasdiff is pinned to v1.33.0 and taken from `OASDIFF_BIN` (`b73ace2`). |
+| 6 | Guard blamed the fix for failures that were already on `main` (an undocumented 404, a KeyError on one POST). | The Schemathesis replay was absolute. | It now runs the same command, seed (20261019) and settings on the base and on the branch, keys failures by (operation, check), and fails only on new ones, plus a 5xx on a touched operation (`--touched-policy`). Pre-existing failures are informational in the report, issue comment and PR body (`b73ace2`). |
+| 7 | The reviewer flagged the approved design as an unexpected change. | It was shown the diff and the plan summary but not the option the human actually approved, nor the guardrail results. | The reviewer now receives the full approved option (summary, behaviours, spec change, new test, files) verbatim plus the guard table, and plan, implement and guard share one scope rule, including the exact `new_test_file` the option names (`c6dd25b`, `4e5e704`). |
+| 8 | 42 issues for 4 incidents after the laptop slept. | The watcher resolved every incident while the machine slept; the next detection reopened it as a new episode, and the issue marker included the episode. | One incident = one issue (PR #16): a stable episode-free `<!-- selfheal-key:... -->` marker with GitHub as the source of truth; one status comment edited in place, new comments only for meaningful changes (24 h limit); a regression grace period (30 min, `PIPELINE_TICKET_REGRESSION_GRACE_S`, or sooner on `POST /api/deploy-finished`, which `scripts/deploy.sh` calls) before a recurrence reopens the same issue; an issue a person closed is muted; poll gaps restart the quiet clock instead of mass-resolving. `selfheal tickets dedupe [--close]` lists and, after confirming, closes the leftovers. |
+| 9 | The guard's oasdiff check suddenly started failing on a different version. | Older workflows ran `go install oasdiff@latest` into `~/go/bin`, overwriting the pinned binary. | No workflow installs anything now; `tests/test_workflow_templates.py` fails if a template runs `go install` or uses `@latest`. Install the pin into `~/.selfheal/bin` (SETUP.md); doctor checks it and flags workflows that differ from the current template (`c6dd25b`). |
+| 10 | The pipeline image failed to build / run: an OpenTelemetry dependency conflict. | pip resolved the pipeline's packages and the customer app's `opentelemetry-instrumentation-fastapi` in separate calls, and pip only warns about conflicts. | One `pip install` call with both constraint files, then `pip check`, so the build fails on any conflict (PR #19, `5c3f22d`). `dedupe` also reads the repo from `--dir` / `selfheal.yml` there. |
+| 11 | `selfheal tickets dedupe` listed duplicates but `--close` found none (or the reverse). | List and close ran separate discoveries, and GitHub intermittently returned empty issue lists. | They share one discovery that retries around empty lists, and closing a duplicate never mutes the incident (PR #20, `65e1126`). |
+
+Related hardening from the same run: the guard skips any `selfheal/*` branch that is not `selfheal/<number>` opened by the
+bot instead of crashing on the integer parse (PR #18), and `runner-install` refuses a versioned Homebrew `/Cellar/` interpreter.
+
+## The second live run (W0): what broke in App mode
+
+Found by the owner's run against `demo-api` with `github_auth: app` (the `selfhealapi-bot` GitHub App), fixed in PR #36. As in the first run the symptoms are from the PR description and the plan, not from saved logs, and no numbers are claimed.
+
+| # | Problem (symptom) | Cause | Fix |
+|---|---|---|---|
+| 12 | App mode silently stayed in PAT mode: everything worked, but as the PAT user. | Most likely: the service reads `selfheal.yml` (and so `github_auth`) once, at startup, from the checkout the container mounts, and the file was pulled after the container started, so the running service never saw `github_auth: app` (the code path was checked by tests; the live log was not kept). Nothing said which mode was running. | The service logs one startup line (`github auth: app (App <id>, bot <login>, key <path>)` or `github auth: pat (<login>)`); `GET /api/status` reports `auth_mode`, `bot_login`, `app_id`; `selfheal doctor` compares the config with the running service and fails on a mismatch ("running auth"), telling you to restart the pipeline when it is stale. Tests spy that in app mode the sink, poller, labeler, dedupe and `/api/github-token` never touch `SELFHEAL_GITHUB_TOKEN`, and the env fallback in `github.py` is gone in app mode. |
+| 13 | The guard workflow skipped the bot's PR. | The guard compared the PR author's login (or the actor) with the accepted bot logins only, and the template gave it no numeric id; a login spelled differently from the configured one (an App's `<slug>[bot]`) made it skip, and the skip did not say what had been compared. | `actions.guard_scope` / `bot_actor` accept the configured login, the App's `<slug>[bot]`, the legacy PAT user, **or the numeric user id** (`SELFHEAL_BOT_USER_ID`, or looked up for the App bot); the template passes `github.actor_id`; the skip notice and the log say what was compared (actor, id, accepted logins and ids). |
+| 14 | Incidents were never analysed: the hourly cap of 6 was used up by the first incidents and the rest starved. | `max_runs_per_hour` was 6 (compose hardcoded `PIPELINE_MAX_RUNS_PER_HOUR=6`, cooldown 600); an incident over the budget had no queue and the next one to be picked was the one seen last (`last_seen DESC`). | `max_runs_per_hour` and `cooldown_s` are read from `selfheal.yml` (env overrides; compose no longer hardcodes them; the demo values are in `SETUP.md`). An incident waiting for budget is explicit: `queued_position` in `/api/status` and `/api/incidents`, "Queued (#N, waiting for run budget)" in the UI, and the oldest unanalysed incident starts first when budget frees. |
+| 15 | Incidents on routes that fail 2-4% of requests flapped: resolved, reopened as a new episode, resolved again. | One global quiet time (window + 10 s) resolved an incident whenever the gap between two failures was longer than that. | Resolution is route-aware: after the quiet window the route must also have served at least `resolve_min_successes` (30, scaled up by failure rate) successful requests since the last failure, or stay quiet for `resolve_quiet_low_rate_s` (1800, scaled by the route's mean gap, capped). Tests use 2-4% failure-rate fixtures. |
+| 16 | Wording told people to "restart the runner service", which does not exist when the runner is started with `./run.sh`. | Text in `setup_cmds.py`, `runtime.py` and the workflow templates assumed the service form. | It now says to restart the runner (the service, or Ctrl+C then `./run.sh`); the asserting test was updated. |
+
+## Design decisions made after the first runs
+
+These are decisions, not results; each is implemented and covered by offline tests only. None has been measured in a live run yet.
+
+| Decision | PR | Why | What it costs |
+|---|---|---|---|
+| **No plan approval for fixes; server-side scope instead.** The fix intent is written inside the implement session and never posted for approval; the limits come from the pipeline: stack-trace files of the failing route from the issue evidence, exactly one new test, file and line caps, never protected / workflow / dependency / spec / docstring / existing-test files. The agent may narrow the limits, never widen them. The reviewer also checks the diff against the issue evidence. | #39 | The plan step was the slowest human wait and the plan approval was a weak gate (a person approving text, not code). A limit the agent cannot set protects more than a plan the agent wrote. | Ambiguous fixes are decided by the agent (most conservative reading, the other listed as "Alternative behaviour" in the PR); the human gate moves to the draft PR. The plan flow stays for a future feature flow (`flow: feature`). |
+| **Trigger safety.** `selfheal:implement` is added by the bot, alone, in its own API call after the issue and its informational labels exist; implement starts only for that label from the bot (login or id); the concurrency group is keyed on whether the event is the trigger. | #39 | Repeats the lesson of finding 2 (three labels, three events, one cancelled run) and keeps comments and people's labels from starting a run. | |
+| **Retry with the failure fed back** (`max_implement_retries`, default 2, hard cap 5) plus the repo's own pre-commit hooks as a guardrail. | #40 | A failed guardrail is usually a small mistake the agent can fix when told which check failed. | More Claude cost per incident (one session per attempt, recorded per attempt). The fix intent, the scope, the guard plan and every check are identical across attempts; nothing is loosened. |
+| **`/revise` on the PR** (a write-access human, `max_revisions` 3, hard cap 5). Guidance is trusted intent from that person; everything else stays untrusted. | #41 | A person reviewing the draft PR can steer it without a plan step or editing code. | The same scope, guardrails and reviewer apply to the whole PR; a revision that leaves scope is discarded. |
+| **Setup wizard** (`selfheal setup`, owner or self-host mode, `--dry-run`, App manifest flow). | #38 | The manual setup took many ordered steps across GitHub, podman and the runner; finding 12 showed that configuration drift is easy to miss. | Built on `main`, not on this stack; not yet run live (`WIZARD_TEST.md` lists what was and was not checked). |
+| **Regression policy.** A merged fix that fails again after the grace period (or a deploy signal) reopens the **same issue** with a "Regressed" comment for a person; no automatic re-fix. | existing behaviour, documented | One incident is one issue; an automatic second fix of a fix a person just merged is the loop the design avoids. | A regression needs a person to look at it. |
+
+## Not measured yet (live-run items)
+
+Nothing in this section has been observed: the final live run on the stacked branches has not happened when this file was written. In particular the following are
+unverified: App mode actually running as `selfhealapi-bot[bot]` end to end, the guard accepting the bot by id on a real PR, the run-budget queue under real traffic, flapping routes no longer flapping, a real retry, a real `/revise`, and the wizard on a real repo. `eval/score_demo.py` now scores the direct flow (outcomes `auto_fix`, `declined`, `escalate`, `recommendation`; retries and revisions from the export) but has only run on synthetic data.
+
+## Results (empty until the final run)
+
+Fill from `eval/score_demo.py`, `selfheal export --format csv` (or `GET /api/export?format=json`) after the owner's run. Nothing here is an estimate.
+
+Scores (the many-bug run):
+
+| Outcome | Expected (`eval/demo_bugs.yml`) | Observed | Notes |
+|---|---|---|---|
+| Auto-fixed and merged correctly | _TBD_ | _TBD_ | |
+| Escalated | _TBD_ | _TBD_ | |
+| Recommendations | _TBD_ | _TBD_ | |
+| Wrongly auto-fixed (must be 0) | 0 | _TBD_ | |
+
+Timings (seconds from detection in the export; median / max over the incidents that reached the stage):
+
+| Stage (export column) | Median | Max | n |
+|---|---|---|---|
+| detected -> issue created (`issue_created_s`) | _TBD_ | _TBD_ | _TBD_ |
+| issue created -> fix intent recorded (`fix_intent_s`) | _TBD_ | _TBD_ | _TBD_ |
+| implement done (`implement_done_s`) | _TBD_ | _TBD_ | _TBD_ |
+| guardrails done (`guard_done_s`) | _TBD_ | _TBD_ | _TBD_ |
+| reviewer done (`reviewer_done_s`) | _TBD_ | _TBD_ | _TBD_ |
+| PR opened (`pr_opened_s`) | _TBD_ | _TBD_ | _TBD_ |
+| merged by a person (`merged_s`) | _TBD_ | _TBD_ | _TBD_ |
+| resolved (`resolved_s`) | _TBD_ | _TBD_ | _TBD_ |
+
+Attempts and revisions (from `implement_attempts`, `failed_attempts`, `revisions`):
+
+| Measure | Value |
+|---|---|
+| Incidents that needed a retry | _TBD_ |
+| Attempts per incident (median / max) | _TBD_ |
+| Incidents that failed after the last attempt | _TBD_ |
+| `/revise` rounds used | _TBD_ |
+
+Cost (USD from the Claude CLI's `total_cost_usd`, per the export's `implement_*`, `review_*`, `total_*` columns and `revision_cost_usd`):
+
+| Claude step | Cost per session (USD) | Tokens in / out | Sessions |
+|---|---|---|---|
+| implement (all attempts) | _TBD_ | _TBD_ | _TBD_ |
+| reviewer | _TBD_ | _TBD_ | _TBD_ |
+| `/revise` | _TBD_ | _TBD_ | _TBD_ |
+| **Total per incident** | _TBD_ | _TBD_ | _TBD_ |
+
+(The export also has `plan_*` columns; they are empty for fixes now that there is no plan step.)