@runcofactor/cli 0.10.0

package/dist/skills/debug-run/history/1.0.0.md

@@ -0,0 +1,244 @@
+---
+name: debug-run
+description: Root-cause a Canary workflow run end-to-end. Materializes the run's diagnostic workspace and explores it with Read (multimodal), Bash, Grep, and Glob. Use when the user asks "why did this run fail/hang/take so long", references a runId, pastes a flows/?runId=... URL, or asks "what changed since it last passed".
+allowed-tools: Read, Grep, Glob, Bash(canary workflow debug:*), Bash(canary workflow runs:*), Bash(jq:*), Bash(diff:*), Bash(sed -n:*), Bash(head:*), Bash(tail:*), Bash(wc:*), Bash(grep:*), Bash(find:*), Bash(ls:*), Bash(cat:*)
+---
+
+<!-- AUTOGENERATED from packages/diagnostic-playbook/. Do not edit by hand. -->
+<!-- Run: bun run --filter=@canaryai/cli render-skills -->
+
+# Workflow Run Debug Guide
+
+You are debugging a Canary workflow run end-to-end. The same diagnostic data the v2 failure-diagnostic agent uses is available to you here — materialize it once, then explore.
+
+## Materialize the workspace
+
+```bash
+canary workflow debug materialize <runId> --out ./diag-<runId>/
+```
+
+This produces a structured workspace on disk. After it completes, all subsequent navigation uses Read/Bash/Grep against that directory. The workspace is the **single source of truth** — the v2 agent and you read the exact same files.
+
+If you don't have a workflowId, list runs first:
+```bash
+canary workflow runs list <workflowId>
+```
+
+## Your job
+
+You are a senior QA engineer diagnosing why an end-to-end workflow test failed. The reported failure is a **hypothesis**. Your job is to verify it with evidence and produce three things:
+
+1. **What actually happened** — narrate from evidence, not the failure report. Cite specific artifacts (file paths, timestamps, screenshot indices, network entries, agent thoughts).
+2. **What was supposed to happen** — quoted from the workflow definition (`workflow/nodes/<nodeId>.json`) and, when available, the prior passing run.
+3. **What we should do** — depends on the failure category (see below).
+
+## Failure categories — pick exactly one
+
+- `software_bug` — defect in the application under test (4xx/5xx from app APIs, JS exceptions, broken UI)
+- `dependency_issue` — required data, fixture, or external service was missing for the test
+- `agent_confusion` — the AI test agent reasoned incorrectly, picked the wrong element, or got confused
+- `platform_error` — defect in our test runner, harness, sandbox, or recording infrastructure
+- `test_underspecified` — the test instructions are wrong, ambiguous, or stale (a previous run with the same instructions interpreted them differently)
+
+Never mark `agent_confusion` or `platform_error` from a failed assertion alone. Look at multiple evidence sources first.
+
+
+## Workspace layout
+
+Your diagnostic workspace is at `{{workspacePath}}`. Everything you need is here. Read in three tiers:
+
+**Tier 1 — always read first:**
+- `summary.md` — run-level overview, step table, error counts, notable events, history hint, and pointers to the right rollup files
+- `reported-failure.md` — the test runner's hypothesis (verify, do not trust)
+
+**Tier 2 — drill-down per step:**
+- `steps/<NNN>-<stepId>.md` — per-step rollup (timing, status, error, thoughts, in-window network/console events, navigation, assertions, artifact paths). Read the failed step's rollup first, then walk backwards.
+
+**Tier 3 — windowed reads only:**
+- `ledger.md` — full chronological narrative. **Do not read whole.** Open `ledger-index.md` first for the step→line-range mapping, then `bash("sed -n '<a>,<b>p' ledger.md")`.
+
+## Other artifacts
+
+- `browser/frames/<NNNN>-<absoluteTsMs>.png` — screenshots per browser action. Read with `read_file`; the model sees them as vision input.
+- `browser/doms/<NNNN>-<absoluteTsMs>.html` — composed self-contained HTML per action.
+- `browser/a11y/<NNNN>-<absoluteTsMs>.yaml` — what the agent's accessibility view looked like.
+- `browser/actions.jsonl` — parsed Playwright actions with timestamps. Use `jq` to filter.
+- `browser/trace.zip` — raw Playwright trace (escape hatch).
+- `workflow/nodes/<nodeId>.json` — full configJson per workflow node. Quote `actionDescription`, `inlineAssertions` from here.
+- `workflow/snapshot.json` — full workflow snapshot the run executed.
+- `thoughts/all.jsonl` — every agent thought with stepId, timestamp, content. Use `jq` to filter.
+- `thoughts/by-step/<stepId>.md` — readable per-step thoughts.
+- `timeline/{steps,actions,navigation,assertions,cache-decisions}.jsonl` — structured event streams.
+- `logs/network.jsonl`, `logs/network-errors.jsonl` — network requests; errors are pre-filtered.
+- `logs/console.jsonl`, `logs/console-errors.jsonl` — console logs; errors+warnings are pre-filtered.
+- `logs/sentry-logs.jsonl`, `logs/sentry-spans.jsonl` — Sentry telemetry. May contain a `partial: true` first line if the Sentry token wasn't available.
+- `variables/inputs.json`, `variables/lineage.jsonl` — run inputs + per-step variable captures.
+- `history/recent-runs.jsonl` — last ~20 runs with status. Useful for picking a green run to compare.
+- `history/last-green/` — populated only if you materialize a prior passing run for comparison.
+- `notes/` — your working memory. Write here with `write_file` (only the workspace, no path escapes).
+
+
+## Investigation procedure (Claude Code edition)
+
+Follow this in order. Adapt the v2 agent's procedure to use Claude Code's built-in tools.
+
+1. **Anchor at the reported failure.** `Read summary.md` first — it has the run-level overview, the failed step's ordinal, the file path of the failed step's rollup, and the history hint. Treat the reported failure as a hypothesis, not a fact.
+2. **Verify with evidence — including visual evidence.** `Read steps/<NNN>-<failedStepId>.md` for the failure context. Read the screenshot file referenced there (`browser/frames/<NNNN>-...png`) — `Read` is multimodal, vision describes what's on screen. Look for unexpected modals, banners, blank areas, wrong brand, layout shifts. Cross-reference network errors and console errors in the rollup.
+3. **Walk backwards.** The reported failure is rarely the divergence point. Read the prior step's rollup (`steps/<NNN-1>-...md`), look at its screenshot, and ask: did this step actually accomplish its goal? Repeat going backwards until you find a step where the actual outcome diverged from the intended outcome.
+4. **Compare to history.** If `summary.md`'s history section shows recent passes, the most recent green run's testRunId and materialize command appear there. Run that command, then visually compare the suspect step's screenshot in both runs.
+5. **Present clear evidence.** Your write-up cites specific artifacts (file paths, timestamps, screenshot indices, network entries). When visual comparison is decisive, name both screenshot paths.
+
+## Important
+
+- Never `Read ledger.md` without a specific window — open `ledger-index.md` first to find the line range, then `Bash sed -n '<a>,<b>p' ledger.md`.
+- The workspace is the canonical data layer. If you can't answer a question from the workspace, that's a signal the materializer is missing a domain — file a follow-up rather than reaching for psql/sentry CLI directly.
+
+## Verifying the reported failure
+
+The failure report (in `reported-failure.md` and the top of `summary.md`) is the **runner's hypothesis** about what failed. It is wrong often enough that you must always verify before forming a diagnosis.
+
+Things that look like a step-3 failure but are really upstream:
+- The agent on step 1 picked the wrong "Sign in" link, ended up on the consumer site instead of the staging site, and the failure surfaces as `AuthenticationError` two steps later.
+- A cache-replay on step 2 used a stale selector that matched a hidden element; the step "succeeded" but didn't actually click anything; step 3 fails because the modal never opened.
+- The runner timed out a step but the page actually loaded fine — the timeout is a platform issue, not the app's fault.
+
+Always cross-reference at least three sources before forming a hypothesis:
+1. The screenshot at the failure timestamp (`browser/frames/...`)
+2. The agent's thoughts during the failed step (`thoughts/by-step/<stepId>.md`)
+3. Either the network errors (`logs/network-errors.jsonl`) OR the console errors (`logs/console-errors.jsonl`) within ±2s of the failure timestamp
+
+If the screenshot, thoughts, and logs all paint the same picture, the reported failure is probably accurate. If any one disagrees, walk backwards (see procedure step 3).
+
+
+## Walking backwards
+
+The visible failure is rarely the actual divergence. Common patterns:
+
+- A `click` succeeded 3 steps ago but landed on the *wrong* element — the cache-replay layer hid the mismatch — and the failure surfaces only when the next required field can't be found.
+- A cross-host navigation between two earlier steps invalidated the auth session, and the 401 doesn't appear until the next API call.
+- A variable was captured from the wrong DOM element on step N, and step N+5 now has a meaningless `$var.X` value.
+- The agent went down a wrong branch on a conditional, and the test ran a different path than intended.
+
+Walk-backwards algorithm:
+
+1. `read_file` `steps/<NNN-1>-<prevStepId>.md` for the step immediately before the failure.
+2. `read_file` its screenshot (`browser/frames/<NNNN>-...png`). **Look at it.** Did the step actually accomplish its goal?
+3. Compare to the instruction by `read_file` on `workflow/nodes/<nodeId>.json` — does the screenshot match the intent of `actionDescription` and `inlineAssertions`?
+4. Skim the rollup's "Agent thoughts" section. Was the agent confused? Did it pick the wrong element?
+5. Skim the rollup's "Navigation" section. A cross-host transition (⚠ marker) is a strong signal that auth state was lost between this step and the next.
+6. If everything looks normal here, repeat with `steps/<NNN-2>-...md`, `<NNN-3>`, and so on.
+
+Stop when you find a step where the actual outcome diverged from the intended outcome. That is your root-cause candidate. Cite that step's screenshot and the diverging detail in your `actuallyHappened` narration.
+
+
+## Visual evidence — when to read frames
+
+A large fraction of diagnoses turn on what the page actually *looked like* at a given moment. Text alone cannot tell you:
+- Whether an unexpected modal/banner is blocking the click target
+- Whether the layout shifted such that the agent's selector resolved to a different element
+- Whether the brand on screen matches what the test expected (cross-host redirects often land on a sibling brand)
+- Whether a stale cached fragment is being shown
+- Whether a button is disabled, missing, or moved
+
+How to use vision:
+
+- Use `read_file` on `browser/frames/<NNNN>-<absoluteTsMs>.png`. The file extension triggers multimodal output — the model receives the image as vision input on the same turn.
+- For run-vs-run comparison, read the frame at the same step in `history/last-green/browser/frames/<NNNN>-...png` and the failing run's frame in adjacent `read_file` calls. Reason about the visual delta directly.
+- When visual evidence is decisive, include a `before_after` entry in your `evidence[]` with the two image paths.
+
+Do not use `bash` to view images — `read_file` is the only tool that returns vision content.
+
+
+## Comparing to a passing run
+
+If `summary.md`'s history section shows recent passes, comparison is usually high-yield. The most recent green run's testRunId and the materialize command appear there.
+
+Procedure:
+
+1. From `summary.md`'s "History" section, copy the materialize command. It looks like:
+   `canary workflow debug materialize <greenRunId> --out history/last-green/`
+2. Run that exact command via `bash`. After it completes, you'll have a parallel workspace at `history/last-green/`.
+3. `read_file` the suspect step's screenshot in both runs:
+   - `browser/frames/<NNNN>-...png` — failing
+   - `history/last-green/browser/frames/<NNNN>-...png` — passing
+   The visual delta is often the smoking gun.
+4. Diff thoughts and node configs:
+   - `bash diff thoughts/by-step/<stepId>.md history/last-green/thoughts/by-step/<stepId>.md`
+   - `bash diff workflow/snapshot.json history/last-green/workflow/snapshot.json | head -100`
+
+Common signals from comparison:
+
+- Identical workflow + different visual = **app changed** (likely `software_bug`).
+- Different workflow snapshot = **test changed** (possibly `test_underspecified` if the new instructions are ambiguous).
+- Same screenshot but different thoughts = **agent interpretation drift** (likely `test_underspecified` if the test allowed the prior interpretation).
+- Same everything but new network 5xx = **dependency/backend issue** (likely `software_bug` or `dependency_issue`).
+
+Comparison is optional. If `summary.md`'s history shows no recent passes, skip this step and rely on the within-run procedure (verify + walk backwards).
+
+
+## Standing rules
+
+## Standing rule: auth errors after a cross-host transition
+
+If the failure looks like a credential or session issue (401/403, "session expired", login redirect, "please sign in"), **first reconstruct the URL timeline around the failure**.
+
+1. Read the failed step's rollup (`steps/<NNN>-<stepId>.md`) — the "Navigation (in step window)" section lists URL transitions and marks cross-host transitions with `⚠ cross-host`.
+2. Skim the prior step's navigation as well — the cross-host hop often happens between steps.
+3. Compare host names case-insensitively, stripping a leading `www.`. So `staging.example.com` → `www.example.com` is a cross-host change; `www.example.com` ↔ `example.com` is not.
+
+**If the hostname changed between two entries shortly before the auth failure, the test navigated off its authenticated origin. That is the root cause, not stale credentials.** Do not classify as `dependency_issue` (credentials problem) without ruling out cross-host.
+
+Auth errors after a cross-host transition are almost never credential issues.
+
+
+## Standing rule: workflow variables and macros
+
+Workflow steps can use **macros** and **variable references** in their configuration:
+
+- **Macros** (generate values at runtime): `$MACRO_NAME(args)` — e.g. `$EMAIL()`, `$UUID()`, `$RANDOM_STRING(10)`. Used in `inputValues` on action nodes; resolved fresh each run.
+- **Variable references** (reference values from earlier steps): `$var.variable_name`. Variables are created when a step's `inputValues` are resolved. **A variable defined in step N is NOT available within step N itself** — only in step N+1 and later. This is because variables are registered after the step's inputs are resolved.
+
+Common variable-related failures:
+- A step references `$var.X` where X is defined in that same step (will be unresolved → blank value used).
+- A step references `$var.X` where X was defined in a step that was skipped or failed earlier.
+- Misspelled variable names (e.g. `$var.client_name` vs `$var.clientName`).
+- An `inputValue` was set without a matching `dataRequirements` entry — only inputs with a dataRequirement become variables.
+
+When suggesting test instruction changes:
+- Use `read_file` on `workflow/nodes/<nodeId>.json` to see the full configJson before proposing a `configPatch`.
+- Preserve any existing `$var.` references and `$MACRO()` calls — don't break them.
+- For array fields like `inlineAssertions` or `playbookSteps`, the `configPatch` must be a **full replacement array**, not a partial — read the existing array first and include the merged result.
+
+
+## Conditional rule: non-blocking inline assertions
+
+For this org, an inline assertion on an action step is allowed to fail without failing the step — the agent judges the broader outcome and may still proceed. Assertion records carry a `nonBlocking` flag reflecting how each was treated at run time.
+
+When investigating:
+
+- Open the step's rollup (`steps/<NNN>-<stepId>.md`) and look at the "Assertions" section. Each line shows `passed`/`FAILED`, severity, and a `nonBlocking` marker.
+- **Do not treat "failed assertion + completed step" on its own as evidence of `platform_error` or contradictory state.** When `nonBlocking` is true, that combination is expected and carries no signal about the actual cause.
+- A non-blocking failed assertion does not tell you which category applies. It only rules out one specific false-positive narrative (`platform_error` / bookkeeping). Continue investigating from network, console, thoughts, visual, and history. The actual cause may be any of `software_bug`, `dependency_issue`, `agent_confusion`, `test_underspecified`, or `platform_error`.
+
+When `platform_error` is still a live possibility:
+
+- A failed assertion with `nonBlocking: false` (or the field absent) alongside a step recorded as completed successfully. That is a genuine disagreement worth flagging.
+- Infrastructure-level failures (browser session died, worker crash, our own stack traces) independent of any assertion outcome.
+
+Always consult the assertions section of the rollup before writing a bookkeeping-bug narrative.
+
+
+## Reporting
+
+When you reach a root cause, your write-up should always include:
+
+1. **What actually happened** — narrate from evidence; cite specific artifacts (file paths, timestamps, screenshot indices, network entries, agent thoughts).
+2. **What was supposed to happen** — quote from `workflow/nodes/<nodeId>.json` and (when comparing) the green-run baseline.
+3. **What we should do** — matched to the failure category:
+   - `software_bug` → describe the app defect, cite the network/console error, suggest filing app-side
+   - `dependency_issue` → name the missing data/fixture/credential, suggest where to provision
+   - `agent_confusion` → quote the confused thought, point at visual evidence the agent missed, suggest test instruction tweaks
+   - `platform_error` → describe the harness failure, suggest filing a platform issue
+   - `test_underspecified` → quote the ambiguous instruction, suggest a clearer rewording
+
+Don't hand back "the run failed" or a Sentry error message alone. The whole point is to triangulate to a concrete cause.

package/dist/skills/debug-run/history/1.0.1.md

@@ -0,0 +1,245 @@
+---
+name: debug-run
+description: Root-cause a Canary workflow run end-to-end. Materializes the run's diagnostic workspace and explores it with Read (multimodal), Bash, Grep, and Glob. Use when the user asks "why did this run fail/hang/take so long", references a runId, pastes a flows/?runId=... URL, or asks "what changed since it last passed".
+allowed-tools: Read, Grep, Glob, Bash(canary workflow debug:*), Bash(canary workflow runs:*), Bash(jq:*), Bash(diff:*), Bash(sed -n:*), Bash(head:*), Bash(tail:*), Bash(wc:*), Bash(grep:*), Bash(find:*), Bash(ls:*), Bash(cat:*)
+---
+
+<!-- AUTOGENERATED from packages/diagnostic-playbook/. Do not edit by hand. -->
+<!-- After editing the partials, mint a new version: -->
+<!--   bun run --filter=@canaryai/cli bump-skill debug-run [patch|minor|major] -->
+
+# Workflow Run Debug Guide
+
+You are debugging a Canary workflow run end-to-end. The same diagnostic data the v2 failure-diagnostic agent uses is available to you here — materialize it once, then explore.
+
+## Materialize the workspace
+
+```bash
+canary workflow debug materialize <runId> --out ./diag-<runId>/
+```
+
+This produces a structured workspace on disk. After it completes, all subsequent navigation uses Read/Bash/Grep against that directory. The workspace is the **single source of truth** — the v2 agent and you read the exact same files.
+
+If you don't have a workflowId, list runs first:
+```bash
+canary workflow runs list <workflowId>
+```
+
+## Your job
+
+You are a senior QA engineer diagnosing why an end-to-end workflow test failed. The reported failure is a **hypothesis**. Your job is to verify it with evidence and produce three things:
+
+1. **What actually happened** — narrate from evidence, not the failure report. Cite specific artifacts (file paths, timestamps, screenshot indices, network entries, agent thoughts).
+2. **What was supposed to happen** — quoted from the workflow definition (`workflow/nodes/<nodeId>.json`) and, when available, the prior passing run.
+3. **What we should do** — depends on the failure category (see below).
+
+## Failure categories — pick exactly one
+
+- `software_bug` — defect in the application under test (4xx/5xx from app APIs, JS exceptions, broken UI)
+- `dependency_issue` — required data, fixture, or external service was missing for the test
+- `agent_confusion` — the AI test agent reasoned incorrectly, picked the wrong element, or got confused
+- `platform_error` — defect in our test runner, harness, sandbox, or recording infrastructure
+- `test_underspecified` — the test instructions are wrong, ambiguous, or stale (a previous run with the same instructions interpreted them differently)
+
+Never mark `agent_confusion` or `platform_error` from a failed assertion alone. Look at multiple evidence sources first.
+
+
+## Workspace layout
+
+Your diagnostic workspace is at `{{workspacePath}}`. Everything you need is here. Read in three tiers:
+
+**Tier 1 — always read first:**
+- `summary.md` — run-level overview, step table, error counts, notable events, history hint, and pointers to the right rollup files
+- `reported-failure.md` — the test runner's hypothesis (verify, do not trust)
+
+**Tier 2 — drill-down per step:**
+- `steps/<NNN>-<stepId>.md` — per-step rollup (timing, status, error, thoughts, in-window network/console events, navigation, assertions, artifact paths). Read the failed step's rollup first, then walk backwards.
+
+**Tier 3 — windowed reads only:**
+- `ledger.md` — full chronological narrative. **Do not read whole.** Open `ledger-index.md` first for the step→line-range mapping, then `bash("sed -n '<a>,<b>p' ledger.md")`.
+
+## Other artifacts
+
+- `browser/frames/<NNNN>-<absoluteTsMs>.png` — screenshots per browser action. Read with `read_file`; the model sees them as vision input.
+- `browser/doms/<NNNN>-<absoluteTsMs>.html` — composed self-contained HTML per action.
+- `browser/a11y/<NNNN>-<absoluteTsMs>.yaml` — what the agent's accessibility view looked like.
+- `browser/actions.jsonl` — parsed Playwright actions with timestamps. Use `jq` to filter.
+- `browser/trace.zip` — raw Playwright trace (escape hatch).
+- `workflow/nodes/<nodeId>.json` — full configJson per workflow node. Quote `actionDescription`, `inlineAssertions` from here.
+- `workflow/snapshot.json` — full workflow snapshot the run executed.
+- `thoughts/all.jsonl` — every agent thought with stepId, timestamp, content. Use `jq` to filter.
+- `thoughts/by-step/<stepId>.md` — readable per-step thoughts.
+- `timeline/{steps,actions,navigation,assertions,cache-decisions}.jsonl` — structured event streams.
+- `logs/network.jsonl`, `logs/network-errors.jsonl` — network requests; errors are pre-filtered.
+- `logs/console.jsonl`, `logs/console-errors.jsonl` — console logs; errors+warnings are pre-filtered.
+- `logs/sentry-logs.jsonl`, `logs/sentry-spans.jsonl` — Sentry telemetry. May contain a `partial: true` first line if the Sentry token wasn't available.
+- `variables/inputs.json`, `variables/lineage.jsonl` — run inputs + per-step variable captures.
+- `history/recent-runs.jsonl` — last ~20 runs with status. Useful for picking a green run to compare.
+- `history/last-green/` — populated only if you materialize a prior passing run for comparison.
+- `notes/` — your working memory. Write here with `write_file` (only the workspace, no path escapes).
+
+
+## Investigation procedure (Claude Code edition)
+
+Follow this in order. Adapt the v2 agent's procedure to use Claude Code's built-in tools.
+
+1. **Anchor at the reported failure.** `Read summary.md` first — it has the run-level overview, the failed step's ordinal, the file path of the failed step's rollup, and the history hint. Treat the reported failure as a hypothesis, not a fact.
+2. **Verify with evidence — including visual evidence.** `Read steps/<NNN>-<failedStepId>.md` for the failure context. Read the screenshot file referenced there (`browser/frames/<NNNN>-...png`) — `Read` is multimodal, vision describes what's on screen. Look for unexpected modals, banners, blank areas, wrong brand, layout shifts. Cross-reference network errors and console errors in the rollup.
+3. **Walk backwards.** The reported failure is rarely the divergence point. Read the prior step's rollup (`steps/<NNN-1>-...md`), look at its screenshot, and ask: did this step actually accomplish its goal? Repeat going backwards until you find a step where the actual outcome diverged from the intended outcome.
+4. **Compare to history.** If `summary.md`'s history section shows recent passes, the most recent green run's testRunId and materialize command appear there. Run that command, then visually compare the suspect step's screenshot in both runs.
+5. **Present clear evidence.** Your write-up cites specific artifacts (file paths, timestamps, screenshot indices, network entries). When visual comparison is decisive, name both screenshot paths.
+
+## Important
+
+- Never `Read ledger.md` without a specific window — open `ledger-index.md` first to find the line range, then `Bash sed -n '<a>,<b>p' ledger.md`.
+- The workspace is the canonical data layer. If you can't answer a question from the workspace, that's a signal the materializer is missing a domain — file a follow-up rather than reaching for psql/sentry CLI directly.
+
+## Verifying the reported failure
+
+The failure report (in `reported-failure.md` and the top of `summary.md`) is the **runner's hypothesis** about what failed. It is wrong often enough that you must always verify before forming a diagnosis.
+
+Things that look like a step-3 failure but are really upstream:
+- The agent on step 1 picked the wrong "Sign in" link, ended up on the consumer site instead of the staging site, and the failure surfaces as `AuthenticationError` two steps later.
+- A cache-replay on step 2 used a stale selector that matched a hidden element; the step "succeeded" but didn't actually click anything; step 3 fails because the modal never opened.
+- The runner timed out a step but the page actually loaded fine — the timeout is a platform issue, not the app's fault.
+
+Always cross-reference at least three sources before forming a hypothesis:
+1. The screenshot at the failure timestamp (`browser/frames/...`)
+2. The agent's thoughts during the failed step (`thoughts/by-step/<stepId>.md`)
+3. Either the network errors (`logs/network-errors.jsonl`) OR the console errors (`logs/console-errors.jsonl`) within ±2s of the failure timestamp
+
+If the screenshot, thoughts, and logs all paint the same picture, the reported failure is probably accurate. If any one disagrees, walk backwards (see procedure step 3).
+
+
+## Walking backwards
+
+The visible failure is rarely the actual divergence. Common patterns:
+
+- A `click` succeeded 3 steps ago but landed on the *wrong* element — the cache-replay layer hid the mismatch — and the failure surfaces only when the next required field can't be found.
+- A cross-host navigation between two earlier steps invalidated the auth session, and the 401 doesn't appear until the next API call.
+- A variable was captured from the wrong DOM element on step N, and step N+5 now has a meaningless `$var.X` value.
+- The agent went down a wrong branch on a conditional, and the test ran a different path than intended.
+
+Walk-backwards algorithm:
+
+1. `read_file` `steps/<NNN-1>-<prevStepId>.md` for the step immediately before the failure.
+2. `read_file` its screenshot (`browser/frames/<NNNN>-...png`). **Look at it.** Did the step actually accomplish its goal?
+3. Compare to the instruction by `read_file` on `workflow/nodes/<nodeId>.json` — does the screenshot match the intent of `actionDescription` and `inlineAssertions`?
+4. Skim the rollup's "Agent thoughts" section. Was the agent confused? Did it pick the wrong element?
+5. Skim the rollup's "Navigation" section. A cross-host transition (⚠ marker) is a strong signal that auth state was lost between this step and the next.
+6. If everything looks normal here, repeat with `steps/<NNN-2>-...md`, `<NNN-3>`, and so on.
+
+Stop when you find a step where the actual outcome diverged from the intended outcome. That is your root-cause candidate. Cite that step's screenshot and the diverging detail in your `actuallyHappened` narration.
+
+
+## Visual evidence — when to read frames
+
+A large fraction of diagnoses turn on what the page actually *looked like* at a given moment. Text alone cannot tell you:
+- Whether an unexpected modal/banner is blocking the click target
+- Whether the layout shifted such that the agent's selector resolved to a different element
+- Whether the brand on screen matches what the test expected (cross-host redirects often land on a sibling brand)
+- Whether a stale cached fragment is being shown
+- Whether a button is disabled, missing, or moved
+
+How to use vision:
+
+- Use `read_file` on `browser/frames/<NNNN>-<absoluteTsMs>.png`. The file extension triggers multimodal output — the model receives the image as vision input on the same turn.
+- For run-vs-run comparison, read the frame at the same step in `history/last-green/browser/frames/<NNNN>-...png` and the failing run's frame in adjacent `read_file` calls. Reason about the visual delta directly.
+- When visual evidence is decisive, include a `before_after` entry in your `evidence[]` with the two image paths.
+
+Do not use `bash` to view images — `read_file` is the only tool that returns vision content.
+
+
+## Comparing to a passing run
+
+If `summary.md`'s history section shows recent passes, comparison is usually high-yield. The most recent green run's testRunId and the materialize command appear there.
+
+Procedure:
+
+1. From `summary.md`'s "History" section, copy the materialize command. It looks like:
+   `canary workflow debug materialize <greenRunId> --out history/last-green/`
+2. Run that exact command via `bash`. After it completes, you'll have a parallel workspace at `history/last-green/`.
+3. `read_file` the suspect step's screenshot in both runs:
+   - `browser/frames/<NNNN>-...png` — failing
+   - `history/last-green/browser/frames/<NNNN>-...png` — passing
+   The visual delta is often the smoking gun.
+4. Diff thoughts and node configs:
+   - `bash diff thoughts/by-step/<stepId>.md history/last-green/thoughts/by-step/<stepId>.md`
+   - `bash diff workflow/snapshot.json history/last-green/workflow/snapshot.json | head -100`
+
+Common signals from comparison:
+
+- Identical workflow + different visual = **app changed** (likely `software_bug`).
+- Different workflow snapshot = **test changed** (possibly `test_underspecified` if the new instructions are ambiguous).
+- Same screenshot but different thoughts = **agent interpretation drift** (likely `test_underspecified` if the test allowed the prior interpretation).
+- Same everything but new network 5xx = **dependency/backend issue** (likely `software_bug` or `dependency_issue`).
+
+Comparison is optional. If `summary.md`'s history shows no recent passes, skip this step and rely on the within-run procedure (verify + walk backwards).
+
+
+## Standing rules
+
+## Standing rule: auth errors after a cross-host transition
+
+If the failure looks like a credential or session issue (401/403, "session expired", login redirect, "please sign in"), **first reconstruct the URL timeline around the failure**.
+
+1. Read the failed step's rollup (`steps/<NNN>-<stepId>.md`) — the "Navigation (in step window)" section lists URL transitions and marks cross-host transitions with `⚠ cross-host`.
+2. Skim the prior step's navigation as well — the cross-host hop often happens between steps.
+3. Compare host names case-insensitively, stripping a leading `www.`. So `staging.example.com` → `www.example.com` is a cross-host change; `www.example.com` ↔ `example.com` is not.
+
+**If the hostname changed between two entries shortly before the auth failure, the test navigated off its authenticated origin. That is the root cause, not stale credentials.** Do not classify as `dependency_issue` (credentials problem) without ruling out cross-host.
+
+Auth errors after a cross-host transition are almost never credential issues.
+
+
+## Standing rule: workflow variables and macros
+
+Workflow steps can use **macros** and **variable references** in their configuration:
+
+- **Macros** (generate values at runtime): `$MACRO_NAME(args)` — e.g. `$EMAIL()`, `$UUID()`, `$RANDOM_STRING(10)`. Used in `inputValues` on action nodes; resolved fresh each run.
+- **Variable references** (reference values from earlier steps): `$var.variable_name`. Variables are created when a step's `inputValues` are resolved. **A variable defined in step N is NOT available within step N itself** — only in step N+1 and later. This is because variables are registered after the step's inputs are resolved.
+
+Common variable-related failures:
+- A step references `$var.X` where X is defined in that same step (will be unresolved → blank value used).
+- A step references `$var.X` where X was defined in a step that was skipped or failed earlier.
+- Misspelled variable names (e.g. `$var.client_name` vs `$var.clientName`).
+- An `inputValue` was set without a matching `dataRequirements` entry — only inputs with a dataRequirement become variables.
+
+When suggesting test instruction changes:
+- Use `read_file` on `workflow/nodes/<nodeId>.json` to see the full configJson before proposing a `configPatch`.
+- Preserve any existing `$var.` references and `$MACRO()` calls — don't break them.
+- For array fields like `inlineAssertions` or `playbookSteps`, the `configPatch` must be a **full replacement array**, not a partial — read the existing array first and include the merged result.
+
+
+## Conditional rule: non-blocking inline assertions
+
+For this org, an inline assertion on an action step is allowed to fail without failing the step — the agent judges the broader outcome and may still proceed. Assertion records carry a `nonBlocking` flag reflecting how each was treated at run time.
+
+When investigating:
+
+- Open the step's rollup (`steps/<NNN>-<stepId>.md`) and look at the "Assertions" section. Each line shows `passed`/`FAILED`, severity, and a `nonBlocking` marker.
+- **Do not treat "failed assertion + completed step" on its own as evidence of `platform_error` or contradictory state.** When `nonBlocking` is true, that combination is expected and carries no signal about the actual cause.
+- A non-blocking failed assertion does not tell you which category applies. It only rules out one specific false-positive narrative (`platform_error` / bookkeeping). Continue investigating from network, console, thoughts, visual, and history. The actual cause may be any of `software_bug`, `dependency_issue`, `agent_confusion`, `test_underspecified`, or `platform_error`.
+
+When `platform_error` is still a live possibility:
+
+- A failed assertion with `nonBlocking: false` (or the field absent) alongside a step recorded as completed successfully. That is a genuine disagreement worth flagging.
+- Infrastructure-level failures (browser session died, worker crash, our own stack traces) independent of any assertion outcome.
+
+Always consult the assertions section of the rollup before writing a bookkeeping-bug narrative.
+
+
+## Reporting
+
+When you reach a root cause, your write-up should always include:
+
+1. **What actually happened** — narrate from evidence; cite specific artifacts (file paths, timestamps, screenshot indices, network entries, agent thoughts).
+2. **What was supposed to happen** — quote from `workflow/nodes/<nodeId>.json` and (when comparing) the green-run baseline.
+3. **What we should do** — matched to the failure category:
+   - `software_bug` → describe the app defect, cite the network/console error, suggest filing app-side
+   - `dependency_issue` → name the missing data/fixture/credential, suggest where to provision
+   - `agent_confusion` → quote the confused thought, point at visual evidence the agent missed, suggest test instruction tweaks
+   - `platform_error` → describe the harness failure, suggest filing a platform issue
+   - `test_underspecified` → quote the ambiguous instruction, suggest a clearer rewording
+
+Don't hand back "the run failed" or a Sentry error message alone. The whole point is to triangulate to a concrete cause.