@andwhy/e2e-pipeline 0.11.1 → 0.12.1

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@andwhy/e2e-pipeline",
-  "version": "0.11.1",
+  "version": "0.12.1",
   "description": "Scaffolds the full e2e testing pipeline: agent-assisted Playwright tests, CI runs against Vercel previews, and Checkly monitors that follow production deploys",
   "type": "module",
   "license": "MIT",

package/templates/claude/skills/playwright-test-pipeline/SKILL.md

@@ -88,9 +88,10 @@ The scaffold writes templates, not a working setup. Walk these with the user:
    - `VERCEL_AUTOMATION_BYPASS_SECRET` — Vercel → project → Settings →
      Deployment Protection → Protection Bypass for Automation. Per project —
      never copied from another repo.
-   - `SLACK_WEBHOOK_URL` — a Slack incoming webhook for the project's
-     notification channel; the auto-heal workflow posts its alerts there.
-     Ask which channel before creating one.
+   - `SLACK_WEBHOOK_URL` — the **org-shared** e2e-channel webhook: paste the
+     value from 1Password ("e2e Slack webhook"). Same value in every repo;
+     do not walk the user through creating a Slack app or webhook — routing
+     to the project channel happens via a Slack rule in Phase 5.
    - `ANTHROPIC_API_KEY` needs **no action**: it is an organization-level
      secret inherited by every repo. Do not tell the user to add it.
 5. **Existing tests under `tests/`** (unit, a11y — the scaffold lists them as
@@ -242,10 +243,21 @@ Set up once per project:
 5. **Do not also subscribe a Slack alert channel to these suites.** The
    pipeline is the single alert path; a parallel raw Checkly alert double-pings
    the channel and trains people to ignore one of them.
-6. Dry run: GitHub → Actions → "Checkly auto-heal" → Run workflow, with the
-   suite's check name. Confirm the run completes and the Slack message
-   arrives (a passing check should end as "not reproducible" — summary only,
-   no ping — so a real dry run is easiest right after any genuine failure).
+6. **One Slack Workflow Builder rule** routes this project's messages from
+   the shared e2e channel to the project channel. All notifications post to
+   the e2e channel tagged `[e2e:<repo-name>]`; the rule matches the **full
+   bracketed tag** — never a short name like `gbk`, which collides with
+   other text — and sends the message to the project channel. The sent
+   message's template is just the message-text variable (plus the
+   link-to-original variable when available, so threads happen on the
+   original) — no static prefix text. Have the user add a **second
+   manager** to the rule and record it in the rule registry in the e2e
+   channel's canvas.
+7. Dry run: GitHub → Actions → "Checkly auto-heal" → Run workflow, with the
+   suite's check name. Confirm the run completes and the message shows up in
+   the e2e channel and, via the rule, in the project channel (a passing
+   check should end as "not reproducible" — summary only, no message — so a
+   real dry run is easiest right after any genuine failure).
 
 What the user should expect afterwards: a failing monitor produces either a
 `checkly-heal` PR plus a Slack note (stale or flaky test — **review

package/templates/github/prompts/checkly-heal.md

@@ -39,6 +39,10 @@ negotiable:
    touch `.github/`, `.claude/`, `package.json`, lockfiles, or any
    `*.config.*` file.
 4. **Never run git commit, push, or any GitHub CLI/API command.**
+5. **Never delete, clean, or recreate the `.checkly-heal/` directory** — the
+   workflow reads your outcome from it after you finish, and destroying it
+   turns your entire run into an unexplained failure. No `rm -rf`, no
+   `git clean`. Only add files to it.
 
 ## Procedure
 
@@ -47,6 +51,13 @@ negotiable:
 Read the context and result files. Identify the failing spec file, test, and
 step. Read the spec. Resolve the production `BASE_URL` (override 1).
 
+If the check name from the context matches **nothing** in
+`checkly.config.{ts,mjs}` — a webhook test, a renamed check, a synthetic
+dispatch — there is nothing to reproduce: write the outcome immediately with
+classification `not-reproducible` and a summary saying the check name is
+unknown to this repo, then stop. Do not skip the outcome file because the
+run "wasn't real".
+
 ### 2. Reproduce
 
 Run only the failing spec against production:

package/templates/github/workflows/checkly-heal.yml

@@ -10,9 +10,11 @@ name: Checkly auto-heal
 # (.github, .claude, configs, lockfiles). The agent never pushes: branch,
 # commit, PR, and Slack are workflow steps.
 # Needs repo secrets CHECKLY_API_KEY, CHECKLY_ACCOUNT_ID, SLACK_WEBHOOK_URL
-# (values in 1Password); ANTHROPIC_API_KEY is inherited from the organization.
-# Setup docs: the playwright-test-pipeline skill walks through the Checkly
-# webhook channel.
+# (the org-shared e2e-channel webhook; values in 1Password); ANTHROPIC_API_KEY
+# is inherited from the organization. Slack messages carry an [e2e:<repo>]
+# tag so Slack Workflow Builder rules route them from the shared channel to
+# project channels. Setup docs: the playwright-test-pipeline skill walks
+# through the Checkly webhook channel and the Slack rule.
 on:
   repository_dispatch:
     types: [checkly-alert]
@@ -161,13 +163,23 @@ jobs:
           CHECKLY_ACCOUNT_ID: ${{ secrets.CHECKLY_ACCOUNT_ID }}
         run: |
           npm install -g @anthropic-ai/claude-code@2
+          # Transcript lives OUTSIDE .checkly-heal so it survives whatever the
+          # agent does to its working directory, and outside the repo tree so
+          # the diff guard never counts it.
           claude -p "$(cat .github/prompts/checkly-heal.md)" \
             --model claude-opus-5 \
             --allowedTools "Bash,Read,Edit,Write,Glob,Grep" \
             --permission-mode dontAsk \
             --max-turns 150 \
             --output-format json \
-            > .checkly-heal/agent-output.json || true
+            > /tmp/checkly-heal-agent-output.json || echo "claude exited non-zero: $?" >> "$GITHUB_STEP_SUMMARY"
+          {
+            echo "### Agent step diagnostics"
+            echo '```'
+            ls -la .checkly-heal 2>&1 || echo ".checkly-heal missing"
+            tail -c 2000 /tmp/checkly-heal-agent-output.json 2>/dev/null || echo "no agent output"
+            echo '```'
+          } >> "$GITHUB_STEP_SUMMARY"
 
       # The guard is deterministic on purpose: the PR path requires an outcome
       # file claiming a fix AND a diff clear of protected paths. App-source
@@ -295,6 +307,10 @@ jobs:
             *)
               text=":rotating_light: *Checkly: \`$CHECK_NAME\` failed and the auto-heal run did not finish cleanly.* ${SUMMARY:-Check the run logs.} This needs a developer." ;;
           esac
+          # Routing tag: Slack Workflow Builder rules match the full bracketed
+          # tag ([e2e:<repo>]) to fan messages out from the shared e2e channel
+          # to project channels without substring collisions.
+          text="[e2e:${GITHUB_REPOSITORY##*/}] $text"
           [ "$APP_CHANGE" = "true" ] && text="$text\nIncludes app-side test hooks — the PR's preview run validates them."
           [ -n "$RESULT_LINK" ] && text="$text\nCheckly result: $RESULT_LINK"
           text="$text\nRun: $RUN_URL"
@@ -313,6 +329,8 @@ jobs:
         uses: actions/upload-artifact@v4
         with:
           name: checkly-heal-${{ github.run_id }}
-          path: .checkly-heal/
+          path: |
+            .checkly-heal/
+            /tmp/checkly-heal-agent-output.json
           if-no-files-found: ignore
           retention-days: 14