@andwhy/e2e-pipeline 0.12.0 → 0.12.2

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@andwhy/e2e-pipeline",
-  "version": "0.12.0",
+  "version": "0.12.2",
   "description": "Scaffolds the full e2e testing pipeline: agent-assisted Playwright tests, CI runs against Vercel previews, and Checkly monitors that follow production deploys",
   "type": "module",
   "license": "MIT",

package/templates/github/prompts/checkly-heal.md

@@ -39,6 +39,16 @@ negotiable:
    touch `.github/`, `.claude/`, `package.json`, lockfiles, or any
    `*.config.*` file.
 4. **Never run git commit, push, or any GitHub CLI/API command.**
+5. **Never delete, clean, or recreate the `.checkly-heal/` directory** — the
+   workflow reads your outcome from it after you finish, and destroying it
+   turns your entire run into an unexplained failure. No `rm -rf`, no
+   `git clean`. Only add files to it.
+6. **Runtime is metered — stop the moment the classification is certain.**
+   You run under a hard 35-minute step timeout, and every minute is paid CI
+   and API time. Run the narrowest thing that answers the current question,
+   never the full suite, and never re-run a test whose answer you already
+   have. If the deadline nears, write an honest partial outcome instead of
+   pushing on — a run killed by the timeout reports nothing.
 
 ## Procedure
 
@@ -47,19 +57,29 @@ negotiable:
 Read the context and result files. Identify the failing spec file, test, and
 step. Read the spec. Resolve the production `BASE_URL` (override 1).
 
+If the check name from the context matches **nothing** in
+`checkly.config.{ts,mjs}` — a webhook test, a renamed check, a synthetic
+dispatch — there is nothing to reproduce: write the outcome immediately with
+classification `not-reproducible` and a summary saying the check name is
+unknown to this repo, then stop. Do not skip the outcome file because the
+run "wasn't real".
+
 ### 2. Reproduce
 
-Run only the failing spec against production:
+Identify the narrowest reproduction: the failing spec and test named by
+`result.json` when present; without a result file, the spec files belonging
+to the failing check. Run that once against production, stopping at the
+first failure:
 
 ```bash
-BASE_URL=<production-url> pnpm playwright test <failing-spec> --project=Chrome
+BASE_URL=<production-url> pnpm playwright test <failing-spec> --project=Chrome --max-failures=1
 ```
 
 - **Fails** → continue to triage.
-- **Passes** → run it again with `--repeat-each=3`. All green means the
-  failure was transient on Checkly's side: classification `not-reproducible`,
-  stop, change nothing. Intermittent failures mean the test is **flaky** —
-  continue to triage with that hypothesis.
+- **Passes** → classification `not-reproducible`. Write the outcome and stop
+  immediately — no repeat runs, no wider investigation. Checkly saw it fail
+  and it passes now, so the blip was transient; a genuinely unstable test
+  will alert again, and that repetition is the signal a human acts on.
 
 ### 3. Triage — before any edit
 
@@ -73,9 +93,10 @@ the live page:
   behavior hides an outage.
 - **The test is flaky.** The outcome works, but the test fails
   intermittently — races, missing waits for dynamic content, animation
-  timing, order-dependent assertions. Classification `flaky` — proceed to
-  step 4 and fix the root cause per the robustness rules; blanket timeout
-  increases are not a fix.
+  timing, order-dependent assertions. When the trace alone doesn't settle
+  it, repeating the one failing test is the only permitted repeat run.
+  Classification `flaky` — proceed to step 4 and fix the root cause per the
+  robustness rules; blanket timeout increases are not a fix.
 - **The test is stale.** Production changed legitimately (new markup, changed
   copy, moved element) and the test's selectors or assertions no longer
   describe it, while the actual user outcome still works. Classification
@@ -102,10 +123,11 @@ that make the test robust.
 
 ### 5. Verify
 
-- **Test-only fixes:** re-run the spec against production until green — for a
-  flaky fix, prove stability with `--repeat-each=5`, not a single pass. Then
-  run the full e2e suite once the same way to catch collateral damage, and
-  validate on Checkly's infrastructure:
+- **Test-only fixes:** re-run the fixed spec against production until green —
+  for a flaky fix, prove stability with `--repeat-each=5` on that one spec,
+  not a single pass. Do not run the full suite (override 6): the heal PR gets
+  its own `pr-playwright-tests` preview run, and that is where collateral
+  damage surfaces. Validate the fixed spec on Checkly's infrastructure:
 
   ```bash
   pnpm checkly pw-test --env BASE_URL="<production-url>" -- <failing-spec>

package/templates/github/workflows/checkly-heal.yml

@@ -163,28 +163,57 @@ jobs:
           CHECKLY_ACCOUNT_ID: ${{ secrets.CHECKLY_ACCOUNT_ID }}
         run: |
           npm install -g @anthropic-ai/claude-code@2
+          # Transcript lives OUTSIDE .checkly-heal so it survives whatever the
+          # agent does to its working directory, and outside the repo tree so
+          # the diff guard never counts it. stream-json writes each event as it
+          # happens — a run killed by the timeout still leaves the full history
+          # (plain json buffers everything and a kill leaves an empty file).
           claude -p "$(cat .github/prompts/checkly-heal.md)" \
             --model claude-opus-5 \
             --allowedTools "Bash,Read,Edit,Write,Glob,Grep" \
             --permission-mode dontAsk \
             --max-turns 150 \
-            --output-format json \
-            > .checkly-heal/agent-output.json || true
+            --output-format stream-json --verbose \
+            > /tmp/checkly-heal-agent-output.json || echo "claude exited non-zero: $?" | tee -a "$GITHUB_STEP_SUMMARY"
+          # Diagnostics go to the job log AND the summary: the log is readable
+          # over the API, the summary only in the browser.
+          {
+            echo "### Agent step diagnostics"
+            echo '```'
+            ls -la .checkly-heal 2>&1 || echo ".checkly-heal missing"
+            tail -c 2000 /tmp/checkly-heal-agent-output.json 2>/dev/null || echo "no agent output"
+            echo '```'
+          } | tee -a "$GITHUB_STEP_SUMMARY"
 
       # The guard is deterministic on purpose: the PR path requires an outcome
       # file claiming a fix AND a diff clear of protected paths. App-source
       # changes (test hooks) are allowed but derived from the diff, not the
       # agent's claim, and flagged downstream. Everything else — protected
       # paths, missing outcome, other classifications — escalates.
+      # always(): a killed agent step (timeout, cancel) must still produce a
+      # classification, or Slack reports an unexplained failure.
       - name: Read the outcome and guard the diff
-        if: steps.guard.outputs.mode == 'heal'
+        if: always() && steps.guard.outputs.mode == 'heal'
         id: outcome
+        env:
+          AGENT_CONCLUSION: ${{ steps.agent.conclusion }}
         run: |
           classification="agent-failed"
           summary="The heal agent did not produce an outcome."
-          if [ -f .checkly-heal/outcome.json ] && jq -e . .checkly-heal/outcome.json > /dev/null 2>&1; then
+          if [ "$AGENT_CONCLUSION" = "success" ] && [ -f .checkly-heal/outcome.json ] && jq -e . .checkly-heal/outcome.json > /dev/null 2>&1; then
             classification=$(jq -r '.classification // "agent-failed"' .checkly-heal/outcome.json)
             summary=$(jq -r '.summary // ""' .checkly-heal/outcome.json)
+          else
+            # No trustworthy outcome — say WHY, so the Slack message diagnoses
+            # itself. An outcome written before a kill is unverified: escalate
+            # rather than open a PR for it.
+            if [ "$AGENT_CONCLUSION" = "skipped" ]; then
+              summary="The heal agent never ran — a setup step failed before it."
+            elif [ "$AGENT_CONCLUSION" != "success" ]; then
+              summary="The heal agent was killed before finishing (step conclusion: ${AGENT_CONCLUSION:-unknown} — usually the 35-minute timeout)."
+            fi
+            agent_error=$(jq -r 'select(.is_error == true) | .result // empty' /tmp/checkly-heal-agent-output.json 2>/dev/null | head -c 300)
+            [ -n "$agent_error" ] && summary="$summary Agent error: $agent_error"
           fi
 
           files=$(git status --porcelain | awk '{print $NF}' | grep -v "^\.checkly-heal/" || true)
@@ -319,6 +348,11 @@ jobs:
         uses: actions/upload-artifact@v4
         with:
           name: checkly-heal-${{ github.run_id }}
-          path: .checkly-heal/
+          path: |
+            .checkly-heal/
+            /tmp/checkly-heal-agent-output.json
+          # .checkly-heal is a dot-directory; without this it is silently
+          # excluded and the debug artifact loses context/outcome/result.
+          include-hidden-files: true
           if-no-files-found: ignore
           retention-days: 14