@andwhy/e2e-pipeline 0.12.1 → 0.12.2

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@andwhy/e2e-pipeline",
-  "version": "0.12.1",
+  "version": "0.12.2",
   "description": "Scaffolds the full e2e testing pipeline: agent-assisted Playwright tests, CI runs against Vercel previews, and Checkly monitors that follow production deploys",
   "type": "module",
   "license": "MIT",

package/templates/github/prompts/checkly-heal.md

@@ -43,6 +43,12 @@ negotiable:
    workflow reads your outcome from it after you finish, and destroying it
    turns your entire run into an unexplained failure. No `rm -rf`, no
    `git clean`. Only add files to it.
+6. **Runtime is metered — stop the moment the classification is certain.**
+   You run under a hard 35-minute step timeout, and every minute is paid CI
+   and API time. Run the narrowest thing that answers the current question,
+   never the full suite, and never re-run a test whose answer you already
+   have. If the deadline nears, write an honest partial outcome instead of
+   pushing on — a run killed by the timeout reports nothing.
 
 ## Procedure
 
@@ -60,17 +66,20 @@ run "wasn't real".
 
 ### 2. Reproduce
 
-Run only the failing spec against production:
+Identify the narrowest reproduction: the failing spec and test named by
+`result.json` when present; without a result file, the spec files belonging
+to the failing check. Run that once against production, stopping at the
+first failure:
 
 ```bash
-BASE_URL=<production-url> pnpm playwright test <failing-spec> --project=Chrome
+BASE_URL=<production-url> pnpm playwright test <failing-spec> --project=Chrome --max-failures=1
 ```
 
 - **Fails** → continue to triage.
-- **Passes** → run it again with `--repeat-each=3`. All green means the
-  failure was transient on Checkly's side: classification `not-reproducible`,
-  stop, change nothing. Intermittent failures mean the test is **flaky** —
-  continue to triage with that hypothesis.
+- **Passes** → classification `not-reproducible`. Write the outcome and stop
+  immediately — no repeat runs, no wider investigation. Checkly saw it fail
+  and it passes now, so the blip was transient; a genuinely unstable test
+  will alert again, and that repetition is the signal a human acts on.
 
 ### 3. Triage — before any edit
 
@@ -84,9 +93,10 @@ the live page:
   behavior hides an outage.
 - **The test is flaky.** The outcome works, but the test fails
   intermittently — races, missing waits for dynamic content, animation
-  timing, order-dependent assertions. Classification `flaky` — proceed to
-  step 4 and fix the root cause per the robustness rules; blanket timeout
-  increases are not a fix.
+  timing, order-dependent assertions. When the trace alone doesn't settle
+  it, repeating the one failing test is the only permitted repeat run.
+  Classification `flaky` — proceed to step 4 and fix the root cause per the
+  robustness rules; blanket timeout increases are not a fix.
 - **The test is stale.** Production changed legitimately (new markup, changed
   copy, moved element) and the test's selectors or assertions no longer
   describe it, while the actual user outcome still works. Classification
@@ -113,10 +123,11 @@ that make the test robust.
 
 ### 5. Verify
 
-- **Test-only fixes:** re-run the spec against production until green — for a
-  flaky fix, prove stability with `--repeat-each=5`, not a single pass. Then
-  run the full e2e suite once the same way to catch collateral damage, and
-  validate on Checkly's infrastructure:
+- **Test-only fixes:** re-run the fixed spec against production until green —
+  for a flaky fix, prove stability with `--repeat-each=5` on that one spec,
+  not a single pass. Do not run the full suite (override 6): the heal PR gets
+  its own `pr-playwright-tests` preview run, and that is where collateral
+  damage surfaces. Validate the fixed spec on Checkly's infrastructure:
 
   ```bash
   pnpm checkly pw-test --env BASE_URL="<production-url>" -- <failing-spec>

package/templates/github/workflows/checkly-heal.yml

@@ -165,36 +165,55 @@ jobs:
           npm install -g @anthropic-ai/claude-code@2
           # Transcript lives OUTSIDE .checkly-heal so it survives whatever the
           # agent does to its working directory, and outside the repo tree so
-          # the diff guard never counts it.
+          # the diff guard never counts it. stream-json writes each event as it
+          # happens — a run killed by the timeout still leaves the full history
+          # (plain json buffers everything and a kill leaves an empty file).
           claude -p "$(cat .github/prompts/checkly-heal.md)" \
             --model claude-opus-5 \
             --allowedTools "Bash,Read,Edit,Write,Glob,Grep" \
             --permission-mode dontAsk \
             --max-turns 150 \
-            --output-format json \
-            > /tmp/checkly-heal-agent-output.json || echo "claude exited non-zero: $?" >> "$GITHUB_STEP_SUMMARY"
+            --output-format stream-json --verbose \
+            > /tmp/checkly-heal-agent-output.json || echo "claude exited non-zero: $?" | tee -a "$GITHUB_STEP_SUMMARY"
+          # Diagnostics go to the job log AND the summary: the log is readable
+          # over the API, the summary only in the browser.
           {
             echo "### Agent step diagnostics"
             echo '```'
             ls -la .checkly-heal 2>&1 || echo ".checkly-heal missing"
             tail -c 2000 /tmp/checkly-heal-agent-output.json 2>/dev/null || echo "no agent output"
             echo '```'
-          } >> "$GITHUB_STEP_SUMMARY"
+          } | tee -a "$GITHUB_STEP_SUMMARY"
 
       # The guard is deterministic on purpose: the PR path requires an outcome
       # file claiming a fix AND a diff clear of protected paths. App-source
       # changes (test hooks) are allowed but derived from the diff, not the
       # agent's claim, and flagged downstream. Everything else — protected
       # paths, missing outcome, other classifications — escalates.
+      # always(): a killed agent step (timeout, cancel) must still produce a
+      # classification, or Slack reports an unexplained failure.
       - name: Read the outcome and guard the diff
-        if: steps.guard.outputs.mode == 'heal'
+        if: always() && steps.guard.outputs.mode == 'heal'
         id: outcome
+        env:
+          AGENT_CONCLUSION: ${{ steps.agent.conclusion }}
         run: |
           classification="agent-failed"
           summary="The heal agent did not produce an outcome."
-          if [ -f .checkly-heal/outcome.json ] && jq -e . .checkly-heal/outcome.json > /dev/null 2>&1; then
+          if [ "$AGENT_CONCLUSION" = "success" ] && [ -f .checkly-heal/outcome.json ] && jq -e . .checkly-heal/outcome.json > /dev/null 2>&1; then
             classification=$(jq -r '.classification // "agent-failed"' .checkly-heal/outcome.json)
             summary=$(jq -r '.summary // ""' .checkly-heal/outcome.json)
+          else
+            # No trustworthy outcome — say WHY, so the Slack message diagnoses
+            # itself. An outcome written before a kill is unverified: escalate
+            # rather than open a PR for it.
+            if [ "$AGENT_CONCLUSION" = "skipped" ]; then
+              summary="The heal agent never ran — a setup step failed before it."
+            elif [ "$AGENT_CONCLUSION" != "success" ]; then
+              summary="The heal agent was killed before finishing (step conclusion: ${AGENT_CONCLUSION:-unknown} — usually the 35-minute timeout)."
+            fi
+            agent_error=$(jq -r 'select(.is_error == true) | .result // empty' /tmp/checkly-heal-agent-output.json 2>/dev/null | head -c 300)
+            [ -n "$agent_error" ] && summary="$summary Agent error: $agent_error"
           fi
 
           files=$(git status --porcelain | awk '{print $NF}' | grep -v "^\.checkly-heal/" || true)
@@ -332,5 +351,8 @@ jobs:
           path: |
             .checkly-heal/
             /tmp/checkly-heal-agent-output.json
+          # .checkly-heal is a dot-directory; without this it is silently
+          # excluded and the debug artifact loses context/outcome/result.
+          include-hidden-files: true
           if-no-files-found: ignore
           retention-days: 14