ductwork 1.0.0 → 1.1.1

data/.claude/skills/audit-database-indexes/SKILL.md

@@ -1,11 +1,188 @@
 ---
 name: audit-database-indexes
-description: Audit ductwork for missing database indexes
-allowed-tools: Read, Grep, Glob
+description: Audit the OSS ductwork codebase for missing, unusable, or redundant database indexes — hot claim-path queries with no supporting index, composite indexes in the wrong column order, partial-index predicates that break on MySQL, foreign keys and polymorphic columns without coverage, and indexes that duplicate a prefix of another. Use when the user asks about missing indexes, slow queries, query plans, index coverage, schema performance, or scaling a table.
+allowed-tools: Read, Grep, Glob, Bash
 ---
 
-# Audit Missing Database Indexes
+# Database Index Audit
 
-Audit the entire OSS ductwork codebase for queries that are missing a database index. All migrations live as templates under `lib/generators/ductwork/install/templates/db/**.rb`. Be sure to check all queries and determine if it is a hot path that needs an index. For example, reading next-to-be-claimed IDs, associations, etc. Ensure that suggestions work across at least PostgreSQL, MySQL, and SQLite.
+Audit the OSS ductwork codebase for queries whose access pattern is not
+supported by an index, and for indexes that cost writes without earning it.
 
-For each finding: file:line, severity, why it matters, suggested fix. Do not modify files.
+Bash is available for **read-only** inspection. Do not modify files, do not
+run migrations, and do not write to any database.
+
+## Step 0 — Load shared context
+
+- `.claude/skills/audit-common/method.md`
+- `.claude/skills/audit-common/severity.md`
+- `.claude/skills/audit-common/scope-boundaries.md`
+- `.claude/skills/audit-common/accepted-tradeoffs.md`
+
+**Both migration directories are in scope:**
+
+- `lib/generators/ductwork/install/templates/db/` — fresh installs
+- `lib/generators/ductwork/update/templates/db/` — existing installations
+
+An index added only to the install template never reaches anyone who already
+runs Ductwork. **Every index recommendation must specify both**: the change
+to the install template *and* a new upgrade migration. A recommendation that
+names only one is incomplete and should be reported as such.
+
+## Step 1 — Build two inventories
+
+### 1a. The index inventory
+
+Read every migration in both directories. Produce a table of what an
+already-migrated installation actually has: table, columns in order, unique?,
+partial predicate?, and which adapters receive it (several indexes are inside
+`if mysql?` / `else` branches and differ per adapter).
+
+Watch for indexes added, renamed, or dropped by later upgrade migrations —
+the install template alone does not tell you the current shape.
+
+### 1b. The query inventory
+
+Enumerate every query in `lib/` and `app/`. Do not rely on grep for `where`
+alone; include:
+
+- Named scopes and class-method finders on the models
+- `where`, `order`, `limit`, `pluck`, `exists?`, `find_each`, `update_all`,
+  `delete_all`, `count`
+- The claim paths in `branch_claim.rb`, `row_locking_execution_claim.rb`,
+  `optimistic_locking_execution_claim.rb`
+- Raw SQL fragments, including those built by `DatabaseClock`
+- Association traversals that emit a query per parent
+- Dashboard queries in `app/` — different profile from the hot path, still
+  real
+
+For each query record: table, `WHERE` columns, `ORDER BY`, `LIMIT`, and
+whether it runs on a hot path.
+
+## Step 2 — Classify by heat
+
+Severity depends far more on call frequency than on query shape.
+
+| Class | Meaning |
+|---|---|
+| **Hot** | Runs on every claim attempt, every poll tick, or per unit of work — by every worker and advancer thread simultaneously. Missing index here degrades non-linearly with table size. |
+| **Warm** | Runs per pipeline advancement, per job completion, or per reap sweep. |
+| **Cold** | Dashboard, CLI, health check, migrations, one-off maintenance. |
+
+The claim paths are the hottest queries in the system and the ones where a
+missing or unusable index matters most. Start there.
+
+Cold queries earn a finding only when the table grows unboundedly and the
+query is a full scan. Do not file an index for a dashboard query that filters
+an already-indexed column.
+
+## Step 3 — Checklist
+
+### 3.1 Missing coverage
+
+- Every `WHERE` column combination on a hot or warm query — is there an index
+  whose **leading columns** match? An index on `(a, b)` does not serve a
+  query filtering only on `b`.
+- Foreign keys and `belongs_to` columns used in lookups.
+- Columns backing uniqueness guarantees — is the constraint enforced by a
+  **unique index** in the database, or only by an ActiveRecord validation?
+  Validation-only uniqueness is a correctness finding, not just performance,
+  because concurrent claimers bypass it.
+- Columns used in `ORDER BY ... LIMIT 1` — the claim pattern. Without an
+  index providing the order, the database sorts the entire candidate set to
+  return one row.
+- Polymorphic association pairs (`*_type`, `*_id`) indexed together.
+
+### 3.2 Column order and usability
+
+An index can exist and still not be usable for a query:
+
+- **Equality columns first, then range/sort columns.** An index on
+  `(status, created_at)` serves `WHERE status = ? ORDER BY created_at`;
+  `(created_at, status)` does not.
+- Does the `ORDER BY` direction match, and are mixed `ASC`/`DESC` orders
+  supported by the index as declared?
+- Is a leading column wrapped in a function or cast in the query? That makes
+  the index unusable — including implicit casts from a type mismatch between
+  the bind and the column.
+- Are near-duplicate indexes (`(a, b, c)` and `(a, c, b)`) both actually
+  needed by distinct queries, or is one dead weight?
+
+### 3.3 The `IS NULL` ordering trap (PostgreSQL)
+
+**A known and previously diagnosed issue in this codebase — check for new
+instances.** A composite index `(a, b, c)` queried as
+`WHERE a = ? AND b IS NULL ORDER BY c` will **not** be used for ordering on
+PostgreSQL. PostgreSQL does not treat `IS NULL` as an equality constraint for
+index-ordering purposes, so you get a bitmap scan plus a sort instead of an
+ordered index scan — which defeats the `LIMIT 1` entirely.
+
+The fix is a partial index moving the null test into the predicate:
+`ON table (a, c) WHERE b IS NULL`.
+
+This is exactly the shape the claim paths use (`claimed_for_advancing_at IS
+NULL`, `completed_at IS NULL`), so check every one of them.
+
+### 3.4 Partial indexes and adapter portability
+
+Partial indexes (`where:`) work on PostgreSQL and SQLite. **MySQL does not
+support them at all.**
+
+The codebase already handles this with `if mysql?` / `else` branches that
+substitute a full index. For every partial index:
+
+- Is there a MySQL fallback branch, or does that adapter silently get no
+  index?
+- Is the fallback actually useful for the query, or is it a full index whose
+  leading column has terrible selectivity?
+- If a fix requires a partial index, state explicitly what MySQL gets
+  instead. A recommendation that only works on Postgres is incomplete.
+
+Also confirm the predicate is expressible: Cockroach and SQLite accept
+partial indexes but have their own restrictions on the predicate expression.
+
+### 3.5 Redundant and over-indexed
+
+Indexes are not free — every one is written on every insert and update, and
+these tables are write-heavy by nature.
+
+- Is any index a **strict prefix** of another? `(a)` is redundant when
+  `(a, b)` exists and nothing needs `(a)` alone for uniqueness.
+- Are there indexes no query in the inventory uses? Cross-reference inventory
+  1a against 1b and list the unmatched ones.
+- Does a table have so many indexes that insert cost is a concern? Call out
+  the write amplification on the hot insert paths.
+
+Report these as Low or Medium — real, but the risk of removal is nonzero and
+the reader should decide.
+
+### 3.6 Adapter-specific limits
+
+- **MySQL index key length.** Indexed string columns count toward a byte
+  limit (3072 bytes with InnoDB/DYNAMIC, and 767 in older configurations).
+  Non-Postgres installs store UUIDs as `string(36)` rather than a native
+  `uuid` type, so composite indexes over several ID columns are far wider
+  there than on Postgres. Check the widest composite indexes.
+- **Identifier length.** Oracle historically caps identifiers at 30
+  characters (128 in 12.2+). Auto-generated index names over long
+  `ductwork_*` table and column names can exceed it. Check the longest
+  generated names and recommend an explicit `name:` where they are at risk.
+- Does `MigrationHelper` gate the index correctly for every supported
+  adapter, or does an adapter fall through to a branch meant for another?
+
+## Step 4 — Verify and report
+
+Follow `method.md`. Additionally, every index finding must state:
+
+1. **The exact query** — file:line — that is unserved.
+2. **The access pattern** — `WHERE` columns, `ORDER BY`, `LIMIT`.
+3. **What the database does today** without the index (full scan, sort of the
+   candidate set, bitmap plus sort) and how that scales.
+4. **The proposed index**, in migration form, with column order justified.
+5. **The MySQL story** if the proposal is partial.
+6. **Both landing sites** — install template and a new upgrade migration.
+
+Do not claim a query plan you have not verified. If you are inferring the
+plan from the index shape rather than from an `EXPLAIN`, say so and mark the
+finding medium confidence. `EXPLAIN` requires a live database and is out of
+scope for this audit unless the user supplies one.

data/.claude/skills/audit-database-support/SKILL.md

@@ -1,18 +1,216 @@
 ---
 name: audit-database-support
-description: Audit ductwork for what database adapters and technologies are supported
-allowed-tools: Read, Grep, Glob
+description: Audit the OSS ductwork codebase for code, SQL, and migrations that break on a supported database adapter — PostgreSQL, CockroachDB, MySQL 8+, Trilogy, SQLite, or Oracle. Covers SKIP LOCKED and locking-mode differences, RETURNING, upsert syntax, partial indexes, isolation-level divergence, adapter_name detection bugs, type and identifier limits, and support claimed without CI coverage. Use when the user asks about database compatibility, adapter support, cross-database SQL, portability, or whether something works on a specific database.
+allowed-tools: Read, Grep, Glob, Bash
 ---
 
-# Audit Clock Drift
+# Database Support Audit
 
-Audit the entire OSS ductwork codebase for code and queries that do not support a certain database adapters or technology. Ensure support for:
+Audit the OSS ductwork codebase for code and queries that do not work
+correctly on every claimed-supported database.
 
-* PostgreSQL
-* CockroachDB
-* MySQL 8+ (adapter and technology)
-* Trilogy (adapter)
-* SQLite
-* Oracle
+Bash is available for **read-only** inspection. Do not modify files, do not
+run migrations, and do not write to any database.
 
-For each finding: file:line, severity, why it matters, suggested fix. Do not modify files.
+## Step 0 — Load shared context
+
+- `.claude/skills/audit-common/method.md`
+- `.claude/skills/audit-common/severity.md`
+- `.claude/skills/audit-common/scope-boundaries.md`
+- `.claude/skills/audit-common/accepted-tradeoffs.md`
+
+## Supported targets
+
+| Target | Kind | Notes |
+|---|---|---|
+| PostgreSQL | technology + adapter | Primary target |
+| CockroachDB | technology | PG wire protocol, **different semantics** |
+| MySQL 8+ | technology | |
+| `mysql2` | adapter | |
+| Trilogy | adapter | MySQL-compatible, different adapter name |
+| SQLite | technology + adapter | Weakest feature set; drives uniformity constraints |
+| Oracle | technology + adapter | |
+
+"Supported" means the feature works correctly, not that it fails loudly.
+Silent divergence is worse than an exception.
+
+**CI covers only postgres, mysql, trilogy, and sqlite**
+(`.github/workflows/main.yml`). Oracle and CockroachDB are claimed with no
+automated verification. Auditing that gap is this skill's job — see 3.8.
+
+## Step 1 — Inventory every divergence point
+
+Enumerate and count:
+
+1. **Adapter conditionals** — every `adapter_name` reference in `lib/` and
+   `app/`, plus every `postgresql?` / `mysql?` helper call in the migration
+   templates.
+2. **Raw SQL** — every string passed to `where`, `select_value`, `execute`,
+   `lock`, `order`, or built by `DatabaseClock`.
+3. **Locking calls** — `lock!`, `with_lock`, `.lock(...)`, any `FOR UPDATE`
+   variant.
+4. **Upserts** — `upsert`, `upsert_all`, `insert_all`, `unique_by`,
+   `RecordNotUnique` rescues.
+5. **Migration DDL** — both template directories, especially column types,
+   partial indexes, and anything inside an adapter conditional.
+6. **Type usage** — UUID columns, JSON columns, boolean columns, text vs
+   string, timestamp precision.
+
+## Step 2 — The divergence matrix
+
+Check every inventoried construct against this. This is the substance of the
+audit — do not audit from memory of what databases support.
+
+| Feature | PG | Cockroach | MySQL 8+ | SQLite | Oracle |
+|---|---|---|---|---|---|
+| `FOR UPDATE SKIP LOCKED` | Yes | **Parses, semantics differ** | Yes | **No row locks at all** | Yes |
+| `FOR NO KEY UPDATE` | Yes | Yes | **No equivalent** | n/a | **No equivalent** |
+| `FOR UPDATE NOWAIT` | Yes | Yes | Yes | n/a | Yes |
+| `RETURNING` | Yes | Yes | **No** | 3.35+ | Via different syntax |
+| Partial indexes (`WHERE`) | Yes | Yes | **No** | Yes | Via function-based index |
+| `ON CONFLICT` target (`unique_by`) | Yes | Yes | **No** (`ON DUPLICATE KEY`, no target) | Yes | **No** (`MERGE`) |
+| Native UUID type | Yes | Yes | **No** (string/binary) | **No** | **No** |
+| Transactional DDL | Yes | Yes | **No** (implicit commit) | Yes | **No** |
+| Default isolation | READ COMMITTED | **SERIALIZABLE** | **REPEATABLE READ** | SERIALIZABLE | READ COMMITTED |
+| Advisory locks | Yes | **No** | Yes (`GET_LOCK`) | No | Via `DBMS_LOCK` |
+| `LIMIT` in `UPDATE`/`DELETE` | **No** | No | Yes | Compile-flag | **No** |
+| Identifier length | 63 | 63+ | 64 | Generous | **30** (128 in 12.2+) |
+| Sub-second timestamps | Yes | Yes | Only if precision declared | Text-based | Yes |
+| Boolean type | Native | Native | `TINYINT(1)` | Integer | **No** (`NUMBER(1)`) |
+
+Two entries deserve special attention because they fail *silently*:
+
+- **CockroachDB `SKIP LOCKED`.** It is accepted syntactically. Do not
+  conclude "it parses, therefore it works." Cockroach's serializable
+  isolation and contention handling mean claim behavior is not equivalent to
+  Postgres, and its retryable-transaction errors surface differently.
+- **MySQL REPEATABLE READ.** A `SELECT` inside a transaction reads a
+  snapshot from the transaction's start, not from statement start. A
+  select-then-guarded-update CAS therefore behaves differently on MySQL than
+  on Postgres — the candidate read can be stale in a way READ COMMITTED
+  would not produce. Every CAS claim path needs checking against this.
+
+## Step 3 — Checklist
+
+### 3.1 Adapter detection consistency
+
+The most likely source of real bugs. Every conditional must classify all six
+targets correctly.
+
+- **Does the regex cover Cockroach?** The Cockroach adapter reports
+  `CockroachDB`, which does **not** match `/postgresql/i`. A helper testing
+  `/postgresql/i` sends Cockroach down the non-Postgres branch, while
+  `/postgresql|cockroach/` sends it down the Postgres one. Both patterns
+  exist in this codebase — that inconsistency means Cockroach gets Postgres
+  locking modes but non-Postgres column types. Verify each site and report
+  every divergence.
+- Does every regex cover both `mysql2` and `Trilogy`?
+- Is `downcase` applied consistently, or does one site rely on case-sensitive
+  matching that a differently-cased adapter name would miss?
+- Does every `case`/`if` chain over adapters have an `else` that **raises**
+  rather than silently falling through to a default meant for one adapter?
+  Compare against `DatabaseClock`, which raises `NotImplementedError` — that
+  is the pattern to hold others to.
+- Is detection based on `adapter_name` where a **capability** check would be
+  more honest (e.g. SQLite version for `RETURNING`)?
+
+### 3.2 Locking and claiming
+
+- Does each locking construct have a path for every adapter, including SQLite
+  where `lock!` is a no-op and row-level locking does not exist?
+- Where SQLite has no row locks, what provides the guarantee instead — a CAS
+  predicate, or nothing? "SQLite is single-writer so it's fine" is only true
+  within one process.
+- `FOR NO KEY UPDATE` exists on PG/Cockroach only. Is the fallback for
+  MySQL/Oracle correct, and is the deadlock consequence handled (retry) or
+  merely accepted?
+- Are Cockroach's retryable serialization errors (`40001`) handled anywhere,
+  or would they surface as unhandled failures?
+
+### 3.3 Upsert and conflict handling
+
+- `unique_by` requires a conflict target, which MySQL does not accept. Where
+  the code branches to `{}` for MySQL, does the resulting
+  `ON DUPLICATE KEY UPDATE` match against the intended unique index, or
+  against whatever unique key it happens to hit first? This is a silent
+  wrong-row hazard, not a syntax error.
+- Oracle has neither; does anything reach it?
+- Are `RecordNotUnique` rescues adapter-portable — does every adapter raise
+  the same ActiveRecord error class for a violated unique index?
+
+### 3.4 Types and identifiers
+
+- UUID: PG gets native `uuid`, others get `string(36)`. Are joins and
+  comparisons type-consistent, and does Cockroach land in the right branch?
+- Does any code assume the ID column is a native UUID?
+- JSON columns: are JSON *operators* used anywhere, or only whole-value
+  read/write? Operators are not portable.
+- Booleans: does any raw SQL compare against `TRUE`/`FALSE` literals?
+- Are any generated identifiers at risk of Oracle's limit? Check the longest
+  index and constraint names against the `ductwork_*` table names.
+
+### 3.5 Migrations
+
+- Does every migration run correctly on all six, including the adapter
+  conditionals?
+- Non-transactional DDL on MySQL and Oracle means a migration that fails
+  partway **cannot roll back**. Are multi-statement migrations written so a
+  partial application is recoverable?
+- Is the `up`/`down` (or `change`) reversible on every adapter?
+- Does any migration reference a model class? That breaks when the model
+  changes later — flag regardless of adapter.
+- Are the install and update templates consistent with each other per
+  adapter, or does one branch where the other doesn't?
+
+### 3.6 Raw SQL construction
+
+- Every raw fragment: is the syntax valid on all six, or gated?
+- Interpolated values — are they bound parameters or string-interpolated? An
+  interpolated integer is portable but still worth flagging if it is ever
+  attacker-influenced.
+- Are quoting and identifier-escaping done through the connection's quoting
+  methods rather than hardcoded backticks or double quotes? Backticks are
+  MySQL-only; double quotes mean identifiers on PG and can mean strings
+  elsewhere.
+- Are functions used that do not exist everywhere (`julianday`,
+  `clock_timestamp`, `NUMTODSINTERVAL`, `strftime`, `NOW()`, `GREATEST`)?
+
+### 3.7 ActiveRecord behavior that differs underneath
+
+- Does anything rely on the ordering of an unordered query? Only PG's heap
+  ordering makes this appear to work.
+- `update_all` / `delete_all` with `LIMIT` — not portable to PG or Oracle.
+- Does any code rely on `insert_all` returning IDs? That needs `RETURNING`.
+- Does anything depend on autoincrement semantics or last-insert-id?
+
+### 3.8 Claimed support without verification
+
+For each of the six targets, determine and report:
+
+- Is it exercised in CI? Name the workflow job, or state that none exists.
+- Is there any spec that exercises the adapter-specific branches written for
+  it?
+- Are there code paths written for it that no test covers?
+
+**Oracle and CockroachDB currently have no CI coverage.** Report this once,
+as a single finding, with the specific untested branches enumerated —
+`DatabaseClock`'s Oracle SQL, `run.rb`'s Cockroach locking branch, and any
+others found in Step 1. Do not file it separately per branch.
+
+Grade by consequence: an untested branch that would raise on first use is
+High; one that merely lacks a regression test is Medium.
+
+## Step 4 — Verify and report
+
+Follow `method.md`. Additionally, every finding must state:
+
+1. **Which adapters are affected** and which are fine.
+2. **The failure mode** — does it raise, silently return wrong results, or
+   silently degrade? Silent wrongness outranks a raise; say which it is.
+3. **Whether the code is reachable** on that adapter, or dead there.
+4. **The portable fix**, or an explicitly gated one covering every target.
+
+Do not report a divergence the code already handles correctly — check for an
+existing adapter conditional before filing. Per `method.md`, dedupe to the
+deepest owning layer: if one helper is wrong for Cockroach, that is one
+finding listing its call sites, not one finding per call site.

data/.claude/skills/audit-durability/SKILL.md

@@ -1,17 +1,234 @@
 ---
 name: audit-durability
-description: Audit ductwork for durability gaps
-allowed-tools: Read, Grep, Glob
+description: Audit the OSS ductwork codebase for durability gaps — crash windows where a process or thread death leaks a claim, loses work, or stalls a pipeline; incorrect transaction boundaries; incomplete claim fencing; reaper races; error-routing mistakes; and liveness holes where a run can never reach a terminal state. Use when the user asks to audit durability, crash safety, reliability, at-least-once behavior, reap correctness, claim leaks, stuck pipelines, or "what happens if this crashes here".
+allowed-tools: Read, Grep, Glob, Bash
 ---
 
-# Durability Audit Gap
+# Durability Audit
 
-Audit the entire OSS ductwork codebase for durability gaps.
+Audit the OSS ductwork codebase for durability gaps.
 
-Check for:
-1. **Stuck pipelines**: claims without transition records, advancements without completion, missing reaper coverage
-2. **Lost data**: writes after observable side effects, missing "write before you act" ordering, places where partial failure is not handled
-3. **Double execution**: missing fencing on claim token or process ID, missing idempotency on transitions, gaps in two-phase commit
-4. **Reaper clobbering**: heartbeat updates racing reaper swwps, stale claim token assumptions, missing recoery count increments
+Bash is available for **read-only** inspection (`grep`, `git log`, `git
+blame`, listing files). Do not modify any file, do not run the test suite,
+and do not run migrations.
 
-For each finding: file:line, severity, why it matters, suggested fix. Do not modify files.
+## Step 0 — Load shared context
+
+Read these first. They are not optional; they define what counts as a
+finding.
+
+- `.claude/skills/audit-common/method.md` — inventory, verification, output
+- `.claude/skills/audit-common/severity.md` — how to grade
+- `.claude/skills/audit-common/scope-boundaries.md` — OSS/Pro line, layout
+- `.claude/skills/audit-common/accepted-tradeoffs.md` — do not re-report
+
+**The Pro boundary matters most in this audit.** Step timeouts, interruptible
+advancement, and restarting threads stuck *inside job execution* are Pro
+features. They are the natural-sounding fix for several things you will find.
+They are out of scope. See `scope-boundaries.md`.
+
+## Step 1 — Inventory the durability surface
+
+Enumerate and count each of these before analyzing. State the counts in the
+report.
+
+1. **Claim paths** — `lib/ductwork/branch_claim.rb`,
+   `execution_claim.rb`, `row_locking_execution_claim.rb`,
+   `optimistic_locking_execution_claim.rb`.
+2. **Multi-step state mutations** — every method that writes more than one
+   row, or writes a row and then acts on it. Concentrated in
+   `lib/ductwork/models/` (`branch.rb`, `advancement.rb`, `execution.rb`,
+   `job.rb`, `run.rb`, `step.rb`, `process.rb`).
+3. **Transaction boundaries** — every `transaction do`, `after_commit`,
+   `lock!`, and `with_lock`.
+4. **Rescue and ensure sites** — every `rescue` and `ensure` in `lib/`.
+5. **Process and thread lifecycle** — all of `lib/ductwork/processes/`:
+   supervisors, runners, `job_worker.rb`, `pipeline_advancer.rb`, and their
+   start/restart/kill/shutdown paths.
+6. **Heartbeat and reap paths** — `Process.report_heartbeat!`,
+   `Process#reap!`, and each `reap_process_record!` across the runners and
+   supervisors.
+7. **Existing fault-injection checkpoints** — `grep -rn
+   "FaultInjection.checkpoint" lib/` and the specs in
+   `spec/integration/durability/`.
+
+## Step 2 — Crash-window analysis (the core of this audit)
+
+This is the primary technique. Do this before the checklist in Step 3.
+
+For every multi-step operation on the inventory, enumerate the points between
+its steps and answer, at each point:
+
+> If the process is SIGKILLed here — or the thread is killed here, or the
+> database connection drops here — what state is left behind, and what
+> recovers it?
+
+For each window, name the recovery mechanism explicitly:
+
+- An `ensure` block (does it actually run for this kind of death? `ensure`
+  runs on `Thread#kill` and `Interrupt`; it does **not** run on `SIGKILL`,
+  `exit!`, or a segfault)
+- Transaction rollback (only if the writes are genuinely in one transaction)
+- The heartbeat reaper (recovers on the *next sweep* — note the latency)
+- A CAS predicate that makes a stale write a no-op
+- The claim fence token (`Branch#claim_fence_token`, checked in
+  `Branch#release!`)
+- `BranchClaim#fail_abandoned_advancement`, reactively on next claim
+- Nothing — **this is the finding**
+
+A window whose only recovery is "the reaper eventually" is not automatically
+a bug; that is the design for process death. It *is* a bug when the reaper
+cannot see the leaked state, when recovery requires the global timeout for
+something that should be caught promptly, or when the leaked state blocks
+other work in the meantime.
+
+Report the specific window, not the general concern:
+
+> Between the `UPDATE` at `branch_claim.rb:NN` and the advancement `INSERT`
+> at `:MM`, a SIGKILL leaves the branch claimed with no advancement row.
+> Recovery is X, which takes Y.
+
+## Step 3 — Checklist
+
+Work these in order. For each, cite file:line or state that the category came
+back clean.
+
+### 3.1 Write ordering and observable side effects
+
+- Is state written durably **before** the action it authorizes, or after?
+- Does anything become visible to another process before the transaction that
+  makes it correct has committed?
+- Are results written before the record that says the work is done?
+- Is a job enqueued or a branch made claimable inside a transaction that can
+  still roll back?
+
+### 3.2 Transaction boundaries
+
+- Does each `transaction do` span exactly the writes that must be atomic —
+  no more, no less?
+- Are there nested transactions where an inner rollback silently becomes a
+  savepoint rollback, leaving the outer one committed?
+- Is there external or long-running work (a network call, user code, a sleep)
+  inside a transaction, holding locks?
+- Does anything rely on `after_commit` ordering that is not guaranteed?
+- Does a `lock!` happen *before* the read whose value it protects, or after
+  (a lock taken after reading protects nothing — check for a `reload`)?
+
+### 3.3 Claim integrity and fencing
+
+- **The re-assertion rule:** whatever the candidate `SELECT` filters on, the
+  claiming `UPDATE` must re-assert. A divergence between the two predicates
+  is a real finding. `Ductwork::Step::ADVANCEABLE_STATUSES` is the shared
+  constant; check both sides still use it.
+- Is `rows_updated` checked after every CAS `UPDATE`, and is losing the CAS
+  handled as a normal outcome rather than an error?
+- Can a claim be released by someone who no longer owns it — is the fence
+  token compared on every release path?
+- Can the same unit of work be claimed twice concurrently?
+- Is `process_id` (or the fence token) verified before a terminal write, or
+  is the write blind?
+
+### 3.4 Reaper correctness
+
+- Can the reaper release a claim that a healthy worker legitimately re-took
+  between the staleness check and the release? (Race 1 — active concern.)
+- Does every reap path go through the same guarded release, or does one
+  open-code it?
+- Is the heartbeat written on a schedule that cannot be starved by the work
+  loop itself — can a long unit of work delay the heartbeat past the timeout
+  and cause a self-reap?
+- Are reap sweeps idempotent if two supervisors sweep concurrently?
+- Are crash/recovery counters incremented exactly once per event?
+
+Race 2 (zombie worker overwriting reaper state) is deferred by decision — see
+`accepted-tradeoffs.md`.
+
+### 3.5 Thread and process lifecycle
+
+- Does every thread body have an outer `ensure` that abandons in-flight work?
+- Does thread *restart* clean up what the dead thread held, for deaths that
+  bypass `ensure`?
+- Is `restart` genuinely distinct from `start`, or aliased such that cleanup
+  is skipped?
+- On SIGTERM, is there a drain path — does in-flight work finish or get
+  cleanly abandoned, or is it simply dropped?
+- Does a `thread.kill` risk interrupting a query mid-flight and poisoning the
+  connection? (Known open for `PipelineAdvancer#kill` — new information only.)
+- If the supervisor restarts, does the sweep catch records orphaned by the
+  children it lost?
+
+### 3.6 Error routing and rescue layering
+
+The three legitimate layers, each owning exactly one semantic class:
+
+| Layer | Owns |
+|---|---|
+| `Job#execute` | user code raising → errored (retry) |
+| `JobWorker#work_loop` | anything else raising → crashed (keep thread alive) |
+| `Branch.with_latest_claimed` | token-conditional release on any exit |
+
+- Does any `rescue` extend past its layer's boundary call to cover framework
+  bookkeeping? A DB error while *recording success* must not route to the
+  user-code-failed path — it belongs to the outer backstop.
+- Can one logical failure produce two retry records by passing through two
+  layers?
+- Does any `rescue` swallow an error without recording it, leaving a claim
+  held and a record neither completed nor failed?
+- Are retry and crash counters attributed to the right layer?
+
+### 3.7 Liveness and terminal-state resolution
+
+- Can a run sit `in_progress` forever with no work left? Check fan-in
+  (`combine`, `collapse`, `converge`) where the last sibling to finish must
+  trigger resolution.
+- Is `resolve_terminal_state!` reached on **both** the halt path and the
+  completion path?
+- Are terminal transitions guarded by a run lock plus an already-terminal
+  check, so a completed run cannot be overwritten as halted or vice versa?
+- Can a fan-in wait on a sibling that will never reach a terminal state?
+- Do claims exclude halted and completed runs, so orphaned work cannot
+  resurrect a finished pipeline?
+- Is every `halt_reason` path reachable, and does each caller put the step in
+  its correct terminal state *before* halting?
+
+### 3.8 Connection and database failure
+
+- What happens if the connection drops mid-transaction — is the in-memory
+  record state now a lie about what is committed?
+- Does a failed heartbeat write retry, or silently pass and let the process
+  be reaped while healthy?
+- Are claim queries safe to retry, or would a retry after an ambiguous
+  timeout double-claim?
+
+## Step 4 — Fault-injection coverage cross-check
+
+Ductwork has a named-checkpoint fault harness
+(`lib/ductwork/fault_injection.rb`, driven by `DUCTWORK_FAULT`, supporting
+`kill`, `raise`, `sleep`, `exit`), with crash-window specs in
+`spec/integration/durability/`.
+
+Cross-reference the crash windows found in Step 2 against the checkpoints
+that exist and the specs that exercise them. Report:
+
+- **Windows with no checkpoint** — a durability-critical window that cannot
+  currently be tested. Recommend the checkpoint name and where it goes.
+- **Checkpoints with no spec** — injection points nothing exercises.
+- **Windows covered only for one failure mode** — e.g. a `raise` spec but no
+  `kill` spec, when `ensure` behavior differs between them. This distinction
+  is the whole point of the harness.
+
+Coverage gaps are usually Medium — real, but not themselves a live bug.
+Grade Higher only when the uncovered window is one you independently found a
+correctness problem in.
+
+## Step 5 — Verify and report
+
+Follow `method.md`: re-read every cited `file:line`, drop what does not hold
+or is already accepted, deduplicate to the deepest owning layer, and report
+inline in the format specified there.
+
+Findings in this audit must name the interleaving. "This is not locked" is
+not a finding; "advancer A passes the check at :NN while advancer B is
+between :MM and :QQ, so both create an advancement and the step runs twice"
+is a finding. Per `severity.md`, a finding with no describable interleaving
+caps at Medium.