@rallycry/conveyor-skills 1.0.11 → 1.0.13

package/skills/conveyor-release-review/SKILL.md

@@ -0,0 +1,399 @@
+---
+name: conveyor-release-review
+description: Audit a pending release before it ships — evaluate every change in it, back each finding with evidence (production or dev logs, a local reproduction, or a deterministic proof), grade findings by the project's own Priority levels, strike what an adversarial pass finds over-built, and file the survivors as two packs, release blockers and release suggestions. Use when the user says "/conveyor-release-review [release]", "review this release", "is this release safe to ship", "audit what's going out", or asks for a pre-release check of the dev-to-main gap. Local/MCP surface only; inside a pod the skill says so instead of failing. Never fixes, never starts a build, never merges.
+---
+
+# Conveyor Release Review
+
+Every change in a release already passed its own review. This audit exists for
+what that process cannot see: changes that are fine alone and break **in
+combination**, risk that only exists at release scale, and what happens at
+deploy time.
+
+It has two ways to fail and treats them as equal. Missing a real break ships
+it. Manufacturing work from speculation buries the real break under cards
+nobody will build, and teaches the team to ignore the next review. So nothing
+here is a finding until evidence says so, and nothing is filed until a
+reviewer whose job is to strike it has tried.
+
+## Goal and finish line
+
+**Goal:** every change in the release evaluated; every surviving finding
+carrying its evidence and one of the project's Priority levels, filed in at
+most two packs.
+
+**Finish line:** the report — verdict on its first line — posted to the
+conversation, and to the release card's chat when a release card exists.
+
+**A turn may end only when:** (a) the finish line is true, (b) an
+`AskUserQuestion` is pending because the release scope is genuinely ambiguous,
+or (c) agents you launched are still running; their completion notices resume
+the run. Anything else — a ledger with blank rows, findings listed but not
+filed, packs created with no report — is a stalled run, not a paused one.
+
+**Runtime:** *Codex CLI* — before step 0, create a thread goal with your goal
+tool: "Post a release verdict for <release> with every change evaluated and
+every surviving finding filed." Mark it complete only when the finish line is
+true, never because the budget is low. *Claude Code* — there is no goal tool;
+the finish line above is the loop.
+
+> **Environment — this skill does not apply inside a pod.** Filing the packs
+> needs `create_task`, `manage_priorities`, `add_dependency` against another
+> card, and `update_task` with `priorityValue` — all of which exist only on
+> the external `conveyor-mcp` surface. If
+> `mcp__conveyor__get_connection_context` reports a task binding and no user
+> account, you are in a pod: say so and stop. Do not hunt for an equivalent.
+
+## Ground rules
+
+- **All Conveyor tools fully-qualified** — `mcp__conveyor__get_task`, not
+  `get_task`. Most are deferred; load every one you expect to need in a single
+  ToolSearch `select:` up front.
+- **Pass `projectId` explicitly on every call.** The connection's default
+  project may not be the repo you are standing in.
+- **Findings become cards; fixes get their own PRs.** Never fix anything inside
+  the review, however small. A fix made here ships unreviewed in the release
+  you were asked to judge.
+- **This skill never moves a release.** No `start_task`, `create_release`,
+  `add_to_release`, `approve_and_merge_pr`, or verdict tool. Holding or
+  shipping is a human's call; your job is to make the right call obvious.
+- **Production is read-only.** Logs, and SELECT-only data reads through the
+  project's own sanctioned path. Never a write, never a hand-exported
+  credential.
+- **The project's rules outrank this skill.** Gate commands, log queries, data
+  access and any release checklist live in the host repo's CLAUDE.md. A wrapper
+  skill may add dimensions (see the end of this file); it cannot lower the
+  evidence bar.
+- **The fan-out rules are not project policy.** They follow from how the
+  harness behaves and from account limits, so no project can relax them. A
+  wrapper may LOWER the concurrency cap; it may never raise it or lift a rule.
+
+## 0 — Discover the project
+
+Which levels, branches and evidence sources exist is a per-project fact.
+Guessing wastes the review either way.
+
+1. `mcp__conveyor__get_connection_context`, then match the project to the
+   repo's `git remote` (`mcp__conveyor__list_projects` when there is no
+   default). Copy the project id exactly.
+2. `mcp__conveyor__manage_priorities { action: "list" }` — the priority
+   ladder. The level with the **lowest `value`** is the **blocker level**;
+   every other level is a **suggestion level**. Use the project's own names in
+   every card and in the report. The seeded defaults are High, Medium and Low,
+   and the rest of this skill uses those names for readability. A project with
+   no priorities configured gets High / Medium / Low as text labels and no
+   priority writes.
+3. The dev branch and the default branch, from the repo and its CLAUDE.md.
+4. **Evidence sources — record what exists AND what does not:**
+   `mcp__conveyor__list_project_integrations`, whether
+   `mcp__conveyor__query_gcp_logs` / `mcp__conveyor__query_grafana_logs`
+   answer, the repo's local stack or mock scripts (`package.json` scripts such
+   as `dev:mock` or `dev`, or a repo skill that stands the app up), its scoped
+   test commands, and its sanctioned read-only data path.
+5. `mcp__conveyor__list_tags` once — the vocabulary, and whether a `release`
+   tag exists (step 6 uses it, and never creates it).
+
+Announce one line naming the release, the priority ladder, and the evidence
+sources you have and lack. Then start.
+
+## 1 — Scope the release
+
+Anchor to a concrete diff before reading any code.
+
+1. **Find the release.** An explicit version, branch or range in the argument
+   wins. Otherwise `mcp__conveyor__search_tasks` for the release card
+   (`Release <version>`, or the `release` tag, in ReviewPR or InProgress);
+   `mcp__conveyor__get_task` on it gives the branch and PR number. No release
+   cut → review the dev-to-default gap and label the run **pre-release**.
+2. **Fetch with explicit refspecs.** Many checkouts only track the dev branch,
+   so an unfetched ref silently reviews the wrong range.
+
+   ```bash
+   git ls-remote --heads origin 'release/*'
+   git fetch origin <default>:refs/remotes/origin/<default> \
+     <dev>:refs/remotes/origin/<dev> \
+     'refs/heads/release/*:refs/remotes/origin/release/*'
+   TIP=origin/release/<version>          # or origin/<dev> when pre-release
+   git rev-parse "$TIP"                  # the SHA this verdict is for
+   git log --merges --oneline origin/<default>.."$TIP"
+   git diff origin/<default>..."$TIP" --name-only
+   gh pr list --base <dev> --state merged \
+     --search "merged:>$(git log -1 --format=%cs origin/<default>)"
+   ```
+
+   The `gh pr list` cross-check catches squash and rebase merges the merge log
+   misses. Use `--name-only`, never `--stat`: truncated paths hide files.
+3. **Record the tip SHA.** A full release keeps absorbing the dev branch until
+   it merges, so the tip moves under you. The verdict is for the SHA you
+   recorded. If it moved by report time, review the delta or list the commits
+   you did not review.
+4. **Map every PR to its card.** `conveyor/<slug>-<id>` branch names embed the
+   card slug; otherwise `search_tasks` by PR title. `get_task` gives the plan,
+   which is the intent. Reviewing a diff without its intent is how a
+   single-interpretation solution gets waved through twice. A PR with no card
+   is reviewed on its diff alone and noted as such.
+5. **Build the coverage ledger** — one row per PR or card, one row per
+   dimension in step 2. Every row ends `clean`, `finding`, or
+   `not reviewable (why)`. "Evaluate everything" means the ledger has no blank
+   rows when you finish.
+
+**Fanning out.** On a large release, split the ledger across subagents per
+area. Give each the ground rules and the evidence bar verbatim. Three rules
+hold, and the mechanics are in [references/fan-out.md](references/fan-out.md):
+
+1. **The hand-back is the only report channel.** The harness refuses report
+   files written by subagents. Each agent's final message is its report, and
+   you save each one to the run directory the moment it arrives.
+2. **Only you spawn agents.** Every agent prompt forbids the Agent, Task and
+   Workflow tools. Nested agents share your session limit.
+3. **At most four agents in flight**, the adversarial agent included. Launch
+   the next slice when one hands back, never in fixed waves.
+
+The ledger is still the completeness check, and it stays with you.
+
+## 2 — Review every dimension
+
+Work all of them. Each gets a coverage note in the report even when the note
+is "nothing found", so coverage is auditable. What to look for, the cheapest
+evidence, and a worked example of a real finding next to one that should be
+struck are in [references/dimensions.md](references/dimensions.md).
+
+| # | Dimension | The question |
+| --- | --- | --- |
+| 1 | **Cross-PR interaction** | Which files, symbols, tables and shared packages did two or more PRs touch, and what does their combined diff do? This is the headline check. |
+| 2 | **Data and migrations** | Destructive steps, locks on large tables, ordering between migrations, and the window where old code runs against the new schema. |
+| 3 | **Rollback and off-switches** | Which behavior changes have no flag, and what is the rollback for each risky change? |
+| 4 | **Deploy and config** | New env vars or secrets production lacks, CI or workflow changes, infra manifests, dependency major bumps. |
+| 5 | **Contracts and compatibility** | API, socket, MCP or published-package contract changes, against clients that are already deployed. |
+| 6 | **Security and access** | New methods or endpoints without an access check, secrets or personal data in logs, unbounded queries. |
+| 7 | **Stubs and cleanup** | In the diff only: TODO / FIXME / HACK, debug logging, `.only` or skipped tests, commented-out code, placeholder copy, orphaned flags. |
+| 8 | **Documented-contract drift** | `mcp__conveyor__get_tag` on each touched subsystem: did the release make an overview or a pinned invariant false? |
+| 9 | **Verification gaps** | Risky changed paths with no test, manual tests on member cards left unapproved or rejected (`mcp__conveyor__list_manual_tests`), CI state on the release PR. |
+| 10 | **Open signals** | Follow-ups promised in member-card chat that never became a card; incidents filed since the merge that implicate a release change. |
+| + | **Project dimensions** | Whatever a wrapper skill or the invoking prompt added. Same bar. |
+
+What comes out of this step is a list of **candidates**. None of them is a
+finding yet.
+
+## 3 — Evidence, or it is not a finding
+
+A candidate becomes a finding only with at least one of these. Record the
+exact query or command beside the result, so anyone can re-run it.
+
+| Grade | What it is |
+| --- | --- |
+| **Observed** | Logs or data. `env: "dev"` logs show the release's own code running, because its changes already merged to the dev branch. `env: "prod"` logs and read-only data show current traffic, data shape and row counts — what the release will meet. |
+| **Reproduced** | The project's local stack or mocks booted and the failure shown, or a scoped test or script that demonstrates the break. The steps go on the card. Nothing is committed. |
+| **Proven** | A deterministic static proof with both ends cited as `file:line` — a migration drops a column that code still reads. Allowed only when the failure needs no runtime assumption. "If a user happens to…" is not a proof. |
+
+- **Cheapest first:** logs, then a static proof, then a scoped test, then the
+  full local stack.
+- **Evidence downgrades too.** A path with no traffic in thirty days is a
+  reason to lower a finding or strike it.
+- **Time-box it.** A candidate that has cost fifteen minutes with no evidence
+  either way is Unverified. Move on.
+- **Unverified candidates are never filed.** They go in the report's
+  Unverified list, each with the evidence that would settle it. One that would
+  be blocker-level if true is named in the verdict line as an open question,
+  so a human decides with their eyes open.
+- **A missing source is reported, not substituted.** "No log access on this
+  project, so dimensions 4 and 10 rest on static reading" is a caveat the
+  reader needs.
+
+## 4 — Priority
+
+Grade by **consequence and evidence**, never by which dimension surfaced it.
+
+| Level | Test | Filed in |
+| --- | --- | --- |
+| **High** (the blocker level) | Shipping as-is causes data loss or corruption, a security or access hole, a broken live flow with no workaround, or a deploy that fails or cannot be rolled back. Observed, Reproduced or Proven only. | Release blockers |
+| **Medium** | A real user-facing defect or regression with a workaround or limited reach, or a gap that will bite within a release or two. | Release suggestions |
+| **Low** | Real but minor: cleanup, polish, small debt this release introduced. | Release suggestions |
+
+**Priority is not risk.** Priority is how urgently the fix is needed. Risk is
+how much important surface a change touches; identification and review set
+it, and this skill never does.
+
+The test for High is one question: *would I actually hold the release for
+this?* If the honest answer is "no, but it should be fixed soon", it is
+Medium. A blocker pack of twelve gets ignored, and then so does the one card
+in it that mattered.
+
+## 5 — The adversarial pass
+
+Every candidate that reached step 4, **together with the fix you would
+propose**, now faces a reviewer whose only job is to strike it. Use one fresh
+subagent for all candidates, under the fan-out rules. Write the candidates,
+with their evidence, priority and proposed fix, to `candidates.md` in the run
+directory; with no fan-out, put them in the prompt instead. The agent argues
+for striking each against the tests below and hands back one verdict per
+candidate. Where subagents are unavailable, switch stance explicitly and
+write the case against each finding before deciding.
+
+Strike when:
+
+1. **Over-engineering.** The fix is bigger than the problem: a new
+   abstraction, option or framework for something a few lines solve, or a
+   defense against a future nobody has asked for.
+2. **A niche path that already fails gracefully.** The path is rare — the
+   evidence shows little or no traffic — and its failure is already handled:
+   the error surfaces, no data is damaged, a retry works.
+3. **A redundant code path.** The fix adds a second guard, fallback, retry or
+   validation for something already guaranteed upstream. Cite the guard that
+   already exists.
+4. **The evidence does not support the claim.** It shows something nearby, or
+   something smaller.
+5. **The release neither caused nor worsened it.** Pre-existing problems are
+   out of scope. Worth recording → `mcp__conveyor__create_suggestion`, never a
+   release-pack card.
+6. **An open card already tracks it.** `mcp__conveyor__search_tasks`, two or
+   three keyword variants, all statuses. Name that card in the report instead
+   of filing a second one.
+
+Four outcomes: **Keep**, **Downgrade**, **Shrink** — keep the finding, replace
+the fix with the smallest change that removes the consequence — or **Strike**.
+Every struck finding is listed in the report with its reason. Nothing
+disappears silently, and a reader who disagrees with a strike can say so.
+
+## 6 — File the packs
+
+Survivors only. Created without asking, never started. Templates for the pack
+parent and the finding card are in
+[references/report-and-cards.md](references/report-and-cards.md).
+
+**The bar for every card is the `/conveyor-plan` plan format**
+([../conveyor-plan/references/plan-format.md](../conveyor-plan/references/plan-format.md)),
+Builder briefing included, because `/conveyor-build` is what runs these cards.
+Cite that format; do **not** invoke `/conveyor-plan`. You have already done
+the research, and its create-and-promote steps would file each card twice.
+
+`<label>` is `Release <version>` when a release is cut, and
+`Pre-release <YYYY-MM-DD>` otherwise.
+
+| | Release blockers | Release suggestions |
+| --- | --- | --- |
+| Holds | High findings | Medium and Low findings |
+| Parent title | `<label> blockers` | `<label> suggestions` |
+| Parent priority | the blocker level | the highest level among its children |
+| Lands in | **Open** — the release is waiting on it | **Planning** — a human promotes what they want built |
+
+In this order:
+
+1. `mcp__conveyor__create_task` the parent in Planning, with `priorityValue`,
+   and `tags: ["release"]` when step 0 found that tag.
+2. `mcp__conveyor__create_subtask` once per finding, full plan included, with
+   `dependsOn` only where one fix genuinely needs another to land first.
+3. `mcp__conveyor__update_task { taskId: <child>, priorityValue }` per child.
+   `create_subtask` carries no priority field; this is the sanctioned route.
+4. `mcp__conveyor__post_to_chat` on the parent — the sizing recommendation
+   and what the pack is for. Chat goes **before** any status move, because
+   identification reads it when the card leaves Planning.
+5. Blockers only: `mcp__conveyor__update_subtask { status: "Open" }` per
+   child, then `mcp__conveyor__update_task { status: "Open" }` on the parent.
+
+Then:
+
+- **A pack of one is a card.** One survivor at a level → a single card titled
+  `<label> blocker: <finding>` or `<label> suggestion: <finding>`.
+- **No survivors at a level → no pack at that level.** A clean release creates
+  nothing, and saying so is a complete result.
+- **Blocking is advisory.** Nothing in Conveyor stops a release PR from
+  merging. With a release card present, make the hold impossible to miss:
+  `mcp__conveyor__add_dependency { taskId: <release card>, dependsOnSlugOrId: <blocker pack or card> }`,
+  which shows the release as blocked until the fixes merge, and a
+  `post_to_chat` on the release card whose **first line** is the hold notice.
+  If `add_dependency` is refused, report the error once and carry on.
+- **Say how the fixes reach the release.** A full release absorbs whatever
+  merges to the dev branch, so merged fixes ship with it. A cherry-pick
+  release does not: a human adds them with Add to Release. Put the right
+  sentence in the blocker pack's plan.
+- **No `priorityValue` on `update_task`?** The connected MCP server predates
+  the field. Prefix each title with its level — `[High] …` — and say in the
+  report that priorities need setting by hand.
+- **On a tool error, stop and report the card rather than retrying it.** A
+  create that "failed" with a timeout may have landed; check with
+  `search_tasks` before creating it again.
+
+## 7 — Verdict and report
+
+The first line is the verdict, and it names the SHA:
+
+- **`HOLD`** — one or more High findings.
+- **`SHIP`** — none.
+- **`SHIP WITH OPEN QUESTIONS`** — no confirmed High, but at least one
+  Unverified candidate that would be High if true.
+
+Below it, in this order: the reviewed range and tip SHA; the packs, linked,
+with a findings table; the **deploy checklist**; the ledger summary with a
+note per dimension; struck findings with reasons; the Unverified list;
+evidence sources used and unavailable; anything not reviewed.
+
+The **deploy checklist** is every step a human must take at deploy — set a
+secret, run a backfill, flip a flag, apply a manifest — whether or not it is a
+finding. A release that is fine in code and fails because nobody knew to set a
+variable is the most avoidable outage there is.
+
+Post the report to the conversation, and with
+`mcp__conveyor__post_to_chat` to the release card when one exists. Never
+create a release card to hold it.
+
+## Running it again
+
+A release gets reviewed more than once: after the fixes land, or after the tip
+moves.
+
+- Find the existing packs by title (`search_tasks "<label> blockers"`) and add
+  new findings to them. Never create a second pack for the same release.
+- Read the last reviewed SHA from the release card's chat and review the
+  delta, not the whole release again.
+- Post the new verdict with the new SHA. The dependency on the release card
+  clears itself once the blocker pack merges to the dev branch.
+
+## Common mistakes
+
+- Reviewing the hunks. Combination bugs live in the overlap map, not in any
+  one diff.
+- Skipping migrations because CI passed. CI ran them against a schema no
+  production data ever stressed.
+- Grading by dimension. A cleanup item is not High because it touched a
+  migration, and a broken flow is not Low because it surfaced under "stubs".
+- One mega-card of findings. One card per finding, or nothing is
+  independently schedulable.
+- Re-litigating a merged PR. Per-PR review is `/conveyor-review`, and it
+  already happened.
+- Diagnosing an incident here. An incident the release implicates is
+  evidence; its root cause belongs to `/conveyor-triage`.
+- Telling subagents to write findings files. The harness refuses them. The
+  hand-back is the report, and saving it is your job.
+- Letting a reviewer fan out. Its agents share your session limit, and a limit
+  hit kills every agent in flight with its unsaved work.
+
+## Wrapping this skill for one project
+
+A repo with its own release concerns keeps them in a thin project skill that
+invokes this one and names its extra dimensions and evidence tools, rather
+than in a fork. The wrapper's shape is at the end of
+[references/dimensions.md](references/dimensions.md). Project dimensions
+become extra ledger rows, graded by the same rubric and held to the same
+evidence bar and adversarial pass.
+
+## Environment notes
+
+- All Conveyor tools are **fully qualified** and mostly **deferred** — one
+  ToolSearch `select:` with every name you expect to need, up front.
+- `search_tasks` returns 20 rows by default — pass `limit`.
+- `gh` reads its own auth. An empty result with exit 0 from `gh pr list` or
+  `gh pr checks` can mean an expired token rather than "nothing there";
+  re-check before concluding.
+- The log tools appear only for a project with the matching integration. If
+  they are absent, the project has none; say so instead of hunting.
+- `You've hit your session limit · resets <time>` (HTTP 429) is the account's
+  five-hour usage window, shared by every agent and by you. Nothing runs until
+  the reset, so do not retry. Resume per
+  [references/fan-out.md](references/fan-out.md).
+
+## Improve This Skill
+
+If this skill was insufficient or slowed the work down, file it with
+`mcp__conveyor__create_suggestion` on the Conveyor project: the issue,
+evidence, and proposed fix.