mustflow 2.116.1 → 2.116.2

package/templates/default/locales/en/.mustflow/skills/fuzz-harness-review/references/fuzz-oracle-triage-ci-checklist.md

@@ -0,0 +1,307 @@
+# Fuzz Oracle, Triage, and CI Checklist
+
+Use this reference when coverage-guided fuzzing is combined with property-based generation,
+differential or metamorphic checking, crash minimization, sanitizer lanes, or CI corpus management.
+Coverage finds unfamiliar execution. The oracle decides whether the behavior is wrong.
+
+## Contents
+
+1. [Evidence model](#1-evidence-model)
+2. [Property-based entropy and abstract inputs](#2-property-based-entropy-and-abstract-inputs)
+3. [Semantic result envelopes](#3-semantic-result-envelopes)
+4. [Differential and metamorphic oracle ledger](#4-differential-and-metamorphic-oracle-ledger)
+5. [Stable property and crash identities](#5-stable-property-and-crash-identities)
+6. [Reproduction capsule and flaky classification](#6-reproduction-capsule-and-flaky-classification)
+7. [Exact minimization and shrinking](#7-exact-minimization-and-shrinking)
+8. [Trigger, detection, and root cause](#8-trigger-detection-and-root-cause)
+9. [Sanitizer and assertion lanes](#9-sanitizer-and-assertion-lanes)
+10. [CI cadence and asset classes](#10-ci-cadence-and-asset-classes)
+11. [Regression promotion and negative controls](#11-regression-promotion-and-negative-controls)
+12. [Primary reference anchors](#12-primary-reference-anchors)
+
+## 1. Evidence model
+
+Keep these roles separate:
+
+| Surface | Role | Does not prove |
+| --- | --- | --- |
+| Edge or feature coverage | Search guidance | Correct result |
+| Value, comparison, or distance signal | Proximity guidance | Failure |
+| Corpus | Search cache and replay input | Permanent regression coverage |
+| Sanitizer | Named runtime defect class | Semantic correctness |
+| Property or invariant | Contract violation | Root cause by itself |
+| Differential mismatch | Candidate disagreement | Which implementation is wrong |
+| Crash artifact | Trigger evidence | Reproducibility or uniqueness |
+| Explicit regression | Durable fixed contract | Current campaign breadth |
+
+Do not fail an input because its semantic distance score is large. Distance may guide mutation, but
+only a named oracle changes the outcome to failure.
+
+## 2. Property-based entropy and abstract inputs
+
+When a fuzzer drives a property-based generator:
+
+- consume artifact bytes through one deterministic entropy provider;
+- remove secondary calls to random APIs, current time, process-specific hash seeds, locale, or
+  scheduler choices unless those facts are encoded in the artifact;
+- make byte exhaustion deterministic and bounded;
+- decode lengths from remaining capacity instead of allocating from untrusted declarations;
+- select generated references from the current object or handle pool;
+- bias reducible values toward zero, one, empty, singleton, and named boundaries; and
+- reject only inputs outside the declared property domain, not inputs that are merely inconvenient.
+
+Generate one abstract object or command sequence and derive compared representations from it.
+Examples include old and new wire formats, JSON and binary encodings, or two backend inputs. State
+whether the parser is under test: a shared parser or shared AST can intentionally remove parser
+differences from the comparison.
+
+Assign an `origin_id` to the abstract input and retain it on every metamorphic follow-up, result,
+artifact, and minimized regression.
+
+## 3. Semantic result envelopes
+
+Normalize results into a versioned envelope before comparing implementations or builds:
+
+| Field | Example meaning |
+| --- | --- |
+| `outcome` | accepted, rejected, property failure, runtime failure |
+| `failure_class` | stable domain category, not renderer text |
+| `value` | canonical semantic value or digest |
+| `consumed` | input units consumed or trailing-data policy |
+| `state_delta` | durable or in-memory state transition |
+| `effects` | writes, events, messages, external calls, cleanup |
+| `checkpoint` | earliest normalized stage reached or diverged |
+| `property_id` | stable violated property or invariant |
+
+Preserve raw stdout, stderr, exceptions, and values separately for triage. Normalize only declared
+differences such as temporary paths, addresses, timestamps, map ordering, platform messages, locale,
+NaN spelling, or generated identifiers. Over-normalization can erase the defect.
+
+Use checkpoint digests for hot comparison only when the minimized artifact is later checked against
+the full canonical checkpoint state.
+
+## 4. Differential and metamorphic oracle ledger
+
+For every differential comparison, record:
+
+- the shared specified domain and excluded undefined or implementation-defined behavior;
+- implementation independence, including shared parsers, libraries, generators, and helpers;
+- compared envelope fields and explicitly ignored differences;
+- reference model, third implementation, specification rule, or metamorphic relation used to
+  assign fault; and
+- version, optimization, feature-flag, integer-width, architecture, or compatibility axes.
+
+Two implementations can share one bug. A mismatch is a candidate until independent evidence names
+the violated contract.
+
+For metamorphic testing, derive follow-ups from one origin and encode the relation's precondition.
+Useful relations include:
+
+- parse and canonical serialize round trip;
+- normalization idempotence;
+- equivalent chunk partitions;
+- permutation of fields whose order is declared irrelevant;
+- compression followed by decompression;
+- failed transaction leaves the prior state unchanged; and
+- monotonic or commutative behavior within a bounded declared domain.
+
+Do not compare independently generated inputs and call the relation metamorphic.
+
+## 5. Stable property and crash identities
+
+Give every property a durable identifier such as `P037_ROUNDTRIP_LENGTH`. Include domain or tolerance
+versions when they change meaning.
+
+Construct a canonical failure key from applicable stable fields:
+
+- runtime or sanitizer kind;
+- access or operation type;
+- property identifier;
+- earliest divergence checkpoint;
+- stable application frames after symbolization and runtime-frame removal; and
+- target, build, architecture, or decoder-schema identity when required.
+
+Do not route every property through one indistinguishable assertion site. Do not deduplicate by raw
+program counter, complete stack text, allocator frames, temporary paths, or rendered messages.
+
+## 6. Reproduction capsule and flaky classification
+
+Bind each artifact to a reproduction capsule:
+
+- binary and debug-symbol hashes;
+- compiler, linker, runtime, and dependency versions;
+- source revision and complete build flags;
+- sanitizer, fuzzer, seed, timeout, memory, and campaign options;
+- environment variables, locale, timezone, working directory, and filesystem fixture;
+- dynamic libraries, operating system, kernel-relevant behavior, CPU, and architecture;
+- thread count, scheduling or fault seed, forced yield points, and repetition count;
+- network response ordering, delays, and close points when applicable; and
+- raw artifact, decoder schema, canonical form, and expected failure key.
+
+Before minimizing, replay the capsule several times and record every observed failure key. Classify:
+
+- `deterministic`: the expected failure is stable;
+- `probabilistic`: the expected failure appears with a measured frequency; or
+- `multimodal`: multiple failure classes or checkpoints appear.
+
+A mix of use-after-free, timeout, and success is not one ordinary crash. Preserve the distribution
+and investigate state, scheduling, or undefined behavior before shrinking.
+
+Do not derive a gate such as “N failures in M attempts” from one observed sample. Repetition count,
+confidence rule, acceptable failure rate, and promotion threshold belong to a repository-owned,
+versioned flaky-test policy. Without that policy, report the measured distribution, retain the
+artifact in triage, and do not promote it to a deterministic regression gate.
+
+## 7. Exact minimization and shrinking
+
+The minimizer predicate must require the same failure, not merely a non-zero exit. Use the canonical
+failure key or a documented subset that remains stable across reduction.
+
+Alternate these reducers to a fixed point when both representations matter:
+
+1. byte reduction removes padding, duplicate fragments, or irrelevant fields;
+2. decode under the versioned harness schema;
+3. semantic shrinking removes nodes, operations, branches, and references while preserving
+   structural validity;
+4. deterministic serialization rebuilds dependent lengths, checksums, or offsets; and
+5. byte reduction runs again under the exact predicate.
+
+Shrink every causal dimension. For concurrency, reduce thread count, schedule seed, yields, and
+repetitions. For fault injection, reduce failing call index and one-shot or persistent mode. For
+network behavior, reduce response order, delay, chunking, and close point.
+
+Reject a reduction if it changes use-after-free into stack overflow, property failure into timeout,
+or one divergence checkpoint into another.
+
+## 8. Trigger, detection, and root cause
+
+Record three different coordinates:
+
+| Coordinate | Meaning |
+| --- | --- |
+| Trigger | Input and environment tuple that exposes the event |
+| Detection point | Sanitizer, assertion, property, or timeout observation |
+| Root cause | Earliest violated contract that created the invalid state |
+
+Use state checkpoints, allocation and free histories, uninitialized-value origins, and per-command
+invariants to move from detection toward creation. The top sanitizer frame is not automatically the
+fix location.
+
+For long sequences, find the last valid and first invalid checkpoint before bisecting source
+history. Otherwise allocator or optimization changes can identify the first visible symptom rather
+than the change that introduced the cause.
+
+Separate timeout, OOM, stack overflow, leak, and infinite progress failure. Compare multiple limits
+and input sizes, operation counts, CPU, maximum RSS, retained resources, and work counters to
+distinguish a fixed threshold from quadratic, exponential, or cumulative behavior.
+
+## 9. Sanitizer and assertion lanes
+
+Use purpose-specific builds when compatibility or cost differs:
+
+| Lane | Primary role |
+| --- | --- |
+| Fast coverage | Search volume and reachability |
+| Address and leak | Bounds, lifetime, double free, retained allocations |
+| Undefined and integer | UB, truncation, sign change, unintended overflow |
+| Uninitialized read | Uninitialized data; origin tracking during focused triage |
+| Race | Stateful sequence and replay under measured scheduling |
+| Assertion-heavy | Internal preconditions, postconditions, ownership, locks, invariants |
+| Production-equivalent replay | Confirm fuzz-friendly gates did not create a test-only result |
+
+Terminate on undefined behavior in triage so corrupted-state follow-on symptoms do not become new
+bugs. Scope integer checks explicitly; a generic undefined bundle is not proof against every
+conversion or unsigned arithmetic defect.
+
+Assertions should monitor internal contracts after external validation. Malformed input that is
+supposed to be rejected should not be converted into a harness assertion crash.
+
+Run invariants after each state transition, not only at the end. Prefer independent shadow checks:
+stored count versus traversal count, index result versus full scan, free versus used set, parent
+list versus child backlink, or actual resources versus tracked totals.
+
+## 10. CI cadence and asset classes
+
+Run bounded CI in this order:
+
+1. explicit semantic regressions;
+2. reviewed seed and minimized regression corpus replay; and
+3. new mutation with the remaining budget.
+
+Use separate time horizons:
+
+- PR: changed-code targets, explicit regressions, bounded replay, short mutation;
+- batch: periodic corpus growth and merge;
+- deep: slower sanitizer, stateful, cross-build, and cross-platform campaigns.
+
+Keep these asset classes separate:
+
+| Asset | Owner and lifecycle |
+| --- | --- |
+| Seed set | Small, reviewed, mostly stable, checked in when appropriate |
+| Coverage corpus | Evolving instrumentation-dependent search cache |
+| Crash artifacts | Original untriaged evidence, immutable until classified |
+| Semantic regressions | Minimized, explained, property-bound durable tests |
+
+Version every corpus bundle with target, harness schema, engine and options, build and instrumentation
+identity, sanitizer, creation time, parent corpus, and coverage fingerprint. On schema change, choose
+`migrate`, `replay-only`, or `invalidate` explicitly.
+
+Merge candidate corpora in staging, validate under the target decoder and instrumentation, then
+promote atomically while preserving the previous snapshot. Do not let multiple engines write one
+directory concurrently. Coverage pruning cannot delete semantic regressions it cannot observe.
+
+Track executions per second, calibration or load time, valid-input ratio, feature yield, stable
+unique failures, flaky rate, minimization success, and regression replay time. Line coverage alone
+cannot distinguish deep semantic exploration from shallow early rejection.
+
+## 11. Regression promotion and negative controls
+
+Promote a discovery only when it:
+
+- reproduces under its capsule;
+- has an exact minimized failure predicate;
+- represents a distinct contract or entry path rather than a duplicate symptom;
+- has a canonical human-readable structure when the byte encoding is opaque;
+- names the stable property or invariant; and
+- can run in a bounded ordinary regression path.
+
+Retain three related assets when valuable: original artifact bytes, canonical structured input, and
+an explicit property-bound regression. A coverage corpus or property-based example database remains
+a search cache, not the only permanent correctness record.
+
+Prove the fix with the original, minimized, and nearby inputs plus applicable sanitizer lanes. Then
+use a bounded negative control when practical: reintroduce the repaired violation or mutate the
+guarded condition and require the regression to fail. If the negative control passes, the test may
+only be hiding the detection point or avoiding the path.
+
+Delete an old regression only when a stronger property demonstrably covers it and catches a
+negative-control reintroduction of the historical defect.
+
+Keep flaky artifacts in a measured reproduction lane with owner and deadline. Do not silently
+delete them, and do not make an unstable artifact an endlessly ignored main-gate failure.
+
+## 12. Primary reference anchors
+
+Refresh tool-specific behavior from current primary documentation:
+
+- [LLVM libFuzzer](https://llvm.org/docs/LibFuzzer.html) for deterministic narrow targets, corpus,
+  replay, merge, minimization, value profiles, limits, artifacts, and fuzz-friendly builds.
+- [Hypothesis external fuzzers](https://hypothesis.readthedocs.io/en/latest/how-to/external-fuzzers.html)
+  for mapping external fuzzer bytes into one generated example.
+- [Hypothesis stateful testing](https://hypothesis.readthedocs.io/en/latest/stateful.html) for
+  command sequences, bundles, rules, and invariants.
+- [Clang AddressSanitizer](https://clang.llvm.org/docs/AddressSanitizer.html),
+  [UndefinedBehaviorSanitizer](https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html),
+  [MemorySanitizer](https://clang.llvm.org/docs/MemorySanitizer.html), and
+  [ThreadSanitizer](https://clang.llvm.org/docs/ThreadSanitizer.html) for current detector scope,
+  options, compatibility, symbolization, and runtime cost guidance.
+- [AFL++ fuzzing in depth](https://aflplus.plus/docs/fuzzing_in_depth/) for campaign setup,
+  instrumentation, sanitizer instances, corpus minimization, dictionaries, analysis, and triage.
+- [ClusterFuzzLite running guide](https://google.github.io/clusterfuzzlite/running-clusterfuzzlite/)
+  for pull-request, batch, coverage, corpus pruning, and scheduled workflow roles.
+- [SQLite testing](https://sqlite.org/testing.html) for independent harnesses, fault injection,
+  assertions, multiple builds, regression tests, and mutation-based test validation.
+
+Tool flags and cost ratios drift. Keep the skill's invariant stable and confirm exact options against
+the repository's pinned engine, compiler, sanitizer, and CI version.

package/templates/default/locales/en/.mustflow/skills/idempotency-integrity-review/SKILL.md

@@ -2,11 +2,11 @@
 mustflow_doc: skill.idempotency-integrity-review
 locale: en
 canonical: true
-revision: 2
+revision: 3
 lifecycle: mustflow-owned
 authority: procedure
 name: idempotency-integrity-review
-description: Apply this skill when code is created, changed, reviewed, or reported and repeated requests, retries, webhooks, queue redelivery, schedulers, batch replays, double clicks, external callbacks, timeout recovery, or duplicate business commands could make the same logical operation move money, inventory, credits, coupons, notifications, shipments, state transitions, durable records, or external side effects more than once.
+description: Apply this skill when code is created, changed, reviewed, or reported and repeated or out-of-order requests, retries, webhooks, queue delivery, schedulers, batch replays, callbacks, timeouts, or stale asynchronous completions could apply one logical business operation more than once, regress aggregate state, or let an old decision overwrite newer authority.
 metadata:
   mustflow_schema: "1"
   mustflow_kind: procedure
@@ -29,16 +29,19 @@ metadata:
 <!-- mustflow-section: purpose -->
 ## Purpose
 
-Review idempotency as duplicate intent integrity, not as an HTTP method checklist.
+Review idempotency as operation identity plus state-validity integrity, not as an HTTP method
+checklist.
 
-The review question is not "is this POST protected?" It is "if the same business intent is delivered twice, late, concurrently, after a timeout, or after the caller missed the first response, what internal state or external side effect moves twice?"
+The review question is not only "is this POST protected?" It is "if the same business intent is
+delivered twice, completes out of order, or acts on authority that has changed, which durable gate
+deduplicates the operation or rejects the stale result?"
 
 <!-- mustflow-section: use-when -->
 ## Use When
 
-- Code creates, changes, reviews, or reports POST, PATCH, DELETE, webhook, callback, queue consumer, scheduler, batch, retry, timeout recovery, mobile app retry, browser refresh, double-click, provider callback, external API call, or background job behavior.
+- Code creates, changes, reviews, or reports POST, PATCH, DELETE, webhook, callback, queue consumer, scheduler, batch, retry, timeout recovery, mobile app retry, browser refresh, double-click, provider callback, external API call, background job, or late asynchronous completion behavior.
 - The operation can charge, capture, refund, withdraw, grant points, deduct credits, deduct stock, issue coupons, send email, create shipments, grant entitlements, change status, publish events, acknowledge messages, or call providers.
-- Idempotency keys, event IDs, message IDs, request IDs, operation IDs, provider object IDs, inbox or outbox records, dedupe tables, processing states, retry libraries, locks, or duplicate request responses are introduced, changed, reviewed, or claimed safe.
+- Idempotency keys, event IDs, message IDs, request IDs, operation or attempt IDs, provider object IDs, aggregate versions, generation or fencing tokens, inbox or outbox records, dedupe tables, processing states, retry libraries, locks, ordering groups, or duplicate request responses are introduced, changed, reviewed, or claimed safe.
 - A review or final report claims an operation is idempotent, retry-safe, duplicate-safe, exactly-once, safe to replay, safe to reprocess, safe after timeout, protected by frontend disabling, protected by Redis, protected by a lock, or protected by provider idempotency.
 
 <!-- mustflow-section: do-not-use-when -->
@@ -56,12 +59,13 @@ The review question is not "is this POST protected?" It is "if the same business
 <!-- mustflow-section: required-inputs -->
 ## Required Inputs
 
-- Operation identity ledger: logical operation, actor, tenant, target resource, business operation type, request body hash, idempotency key, event ID, message ID, provider object ID, batch key, scheduler run key, and retry source.
+- Operation identity ledger: logical operation, attempt, effect, actor, tenant, target resource, business operation type, canonical payload hash, idempotency key, event ID, message ID, provider object ID, batch key, scheduler run key, and retry source.
+- Ordering and authority ledger: aggregate key, ordering scope, observed state, expected state, aggregate version, sequence, generation, fencing token, gap policy, stale-result decision, and authoritative conditional write.
 - Side-effect ledger: every charge, refund, balance change, stock change, coupon issue, shipment, email, notification, entitlement, status transition, file write, queue publish, cache change, provider call, and audit entry.
 - Durable dedupe evidence: unique constraints, idempotency table, inbox table, outbox table, applied event table, ledger source key, conditional update, state guard, provider idempotency key, and response record.
 - Duplicate response policy: cached success response, duplicate conflict response, in-progress response, payload mismatch response, failed validation response, unknown outcome recovery, and caller-visible retry contract.
 - Concurrency and recovery evidence: simultaneous duplicate requests, process restart, TTL expiry, `PROCESSING` stuck state, lease, heartbeat, retry budget, timeout classification, provider lookup, reconciliation, and lock fallback.
-- Queue, webhook, scheduler, and batch evidence: ack timing, redelivery semantics, event ordering, signature checks, event storage, per-aggregate ordering, reprocessing rules, and run-level uniqueness.
+- Queue, webhook, scheduler, and batch evidence: ack timing, redelivery semantics, event ordering scope, partition or group key derivation, missing-sequence handling, signature checks, event storage, per-aggregate ordering, reprocessing rules, and run-level uniqueness.
 - Test evidence: duplicate request tests, concurrent duplicate tests, timeout-then-retry tests, external-success-internal-failure tests, DB-success-response-failure tests, duplicate webhook tests, stale event tests, ack-failure tests, and configured command intents.
 
 <!-- mustflow-section: preconditions -->
@@ -76,10 +80,11 @@ The review question is not "is this POST protected?" It is "if the same business
 <!-- mustflow-section: allowed-edits -->
 ## Allowed Edits
 
-- Add or tighten durable idempotency records, request body hash comparison, user and tenant binding, operation-type binding, target-resource binding, unique constraints, atomic insert-or-return behavior, state guards, affected-row checks, inbox records, outbox records, applied-event ledgers, provider result lookup, response replay, processing lease recovery, and focused duplicate tests.
+- Add or tighten durable idempotency records, canonical payload hash comparison, user and tenant binding, operation-type binding, target-resource binding, unique non-null constraints, atomic insert-or-return behavior, aggregate versions, generation or fencing tokens, gap handling, conditional state guards, affected-row checks, inbox records, outbox records, applied-event ledgers, provider result lookup, response replay, processing lease recovery, and focused duplicate or stale-order tests.
 - Move external side effects behind durable operation records or outbox records when local style supports it.
 - Make retry classification explicit for validation failure, transient failure, timeout with unknown outcome, provider success with local failure, local success with missing response, duplicate in progress, and duplicate completed.
 - Do not treat frontend button disabling, debounce, in-memory flags, Redis TTL alone, provider idempotency alone, or distributed locks alone as final proof of idempotency.
+- Do not treat cancellation, process-local single-flight, FIFO naming, timestamps, arrival order, or auto-increment identifiers as durable proof that a late result is current.
 - Do not make duplicate protection by returning a different success shape unless the caller contract can safely reconcile it.
 
 <!-- mustflow-section: procedure -->
@@ -88,6 +93,7 @@ The review question is not "is this POST protected?" It is "if the same business
 1. Name the logical operation, not only the endpoint.
    - POST, PATCH, DELETE, webhook, queue consumer, scheduler, batch replay, retry wrapper, and callback handlers can all carry the same duplicate-intent risk.
    - Ask what "same intent" means in business terms: same order payment attempt, same refund, same point grant, same coupon redemption, same shipment, same notification, same state transition, same imported row, or same provider event.
+   - Separate the stable operation ID from attempt, message, event, effect, trace, worker, and provider IDs. Generate a new attempt ID for a retry without generating a new business operation.
 2. Start at the side effect.
    - Find calls and writes named like `charge`, `capture`, `refund`, `withdraw`, `grantPoint`, `deductStock`, `issueCoupon`, `sendEmail`, `createShipment`, `publish`, `ack`, `markPaid`, `fulfill`, and local equivalents.
    - For each side effect, ask what happens if the exact surrounding handler runs twice, runs concurrently, or runs after the first response is lost.
@@ -95,9 +101,11 @@ The review question is not "is this POST protected?" It is "if the same business
    - A key is not enough unless it is bound to actor, tenant, operation type, target resource, and request body hash or equivalent semantic fingerprint.
    - Duplicate key plus changed amount, resource, user, tenant, product, operation, or payload should become a stable mismatch response, not a new operation or silent old response.
    - Do not accept memory-only stores, process-local maps, or Redis TTL alone for operations that can be retried after restart, failover, delayed callback, or TTL expiry.
+   - Keep the operation key stable across request, durable operation record, inbox or outbox, external effect, result lookup, and response replay.
 4. Require durable uniqueness at the operation boundary.
    - App-only `if exists return` followed by `insert` is a race. Use a unique constraint, atomic insert, upsert, conditional write, or durable ledger key.
    - Review migrations or schema definitions, not just service code, for keys such as `order_id`, `payment_id`, `refund_id`, `event_id`, `message_id`, `idempotency_key`, `campaign_id + user_id`, and `settlement_date + merchant_id`.
+   - Where the key is required, verify that the authoritative uniqueness field cannot be null and that a matching key with a different canonical payload or binding is rejected.
 5. Check response replay.
    - Idempotency is often needed because the caller did not receive the first response.
    - A duplicate completed request should usually return the original response or an equivalent stable result the caller can reconcile.
@@ -111,52 +119,64 @@ The review question is not "is this POST protected?" It is "if the same business
 8. Check external API ordering.
    - External API call before durable local operation record creates duplicate risk when the local process times out or crashes before saving success.
    - Provider idempotency can protect the provider call while internal follow-up work still duplicates. Internal idempotency remains required.
-9. Review state transitions as conditional operations.
+9. Define the ordering scope and authority token.
+   - Name the smallest aggregate whose operations require order. Transport grouping or partitioning proves order only inside its documented scope and does not prove that application or external effects complete in that order.
+   - Prefer an aggregate version, store revision, generation, or fencing token created for authority. Do not infer causality from wall-clock time, retry time, arrival time, or auto-increment identifiers.
+   - Define older-or-equal, exactly-next, and ahead-with-a-gap branches. Bound buffering or retry, name the resynchronization owner, and keep stuck gaps observable.
+10. Review state transitions as conditional operations.
    - `UPDATE ... SET status = PAID WHERE id = ?` is weaker than `WHERE id = ? AND status = PENDING` or an equivalent state-machine guard.
-   - Late duplicate events should not overwrite terminal, newer, or higher-order states without version, event time, sequence, or transition-order checks.
-10. Review increments and ledger entries.
+   - Carry expected state, version, generation, or fence to the authoritative write. Treat zero affected rows or a failed compare-and-swap as a stale decision that requires re-reading authority, not blind replay of the old calculation.
+   - Late duplicate events should not overwrite terminal, newer, or higher-order states. Event time alone is not an authority token.
+11. Separate cancellation and request coalescing from correctness.
+   - Cancellation requests cleanup; it does not revoke callbacks, cache writes, durable commits, or external effects that already completed.
+   - Process-local single-flight reduces duplicate load but loses authority across instances, restart, failover, and leader change. Keep durable operation identity and storage guards behind it.
+12. Prefer order-insensitive mutations when the domain permits them.
+   - An immutable operation or ledger entry with a stable operation ID can be safer than replacing a full snapshot.
+   - Commutative application does not remove duplicate risk. Deduplicate operation identity and model reversal or compensation as another identified operation.
+13. Review increments and ledger entries.
     - `+=`, balance increases, point grants, stock deductions, usage counters, retry counters, and mutable totals duplicate naturally.
     - Require an applied-event table, source ledger key, unique entry identity, conditional update, or state transition that records the event already moved the number.
-11. Review deletion and "read" endpoints for hidden side effects.
+14. Review deletion and "read" endpoints for hidden side effects.
     - DELETE may be final-state idempotent while refund, notification, inventory restoration, audit, or permission side effects still duplicate.
     - GET endpoints that mark read, accept invites, decrement quota, create download usage, or mutate analytics are not harmless reads under crawler, preview, refresh, or retry behavior.
-12. Review queues and consumers as at-least-once by default.
+15. Review queues and consumers as at-least-once by default.
     - Consumer code should assume duplicate messages, delayed redelivery, concurrent workers, ack failure after commit, and process death after side effects.
     - Store message ID, event ID, business key, or source ledger key in an inbox or applied-event table before applying irreversible effects.
-13. Review ack timing.
+16. Review ack timing.
     - Ack before durable commit can lose work.
     - Ack after durable commit can redeliver work.
     - Prefer accepting redelivery and making the handler idempotent, with dead-letter and retry evidence for poison messages.
-14. Review webhook handlers as replay surfaces.
+17. Review webhook handlers as replay surfaces.
     - External providers resend callbacks when they miss acceptable responses.
     - Store event IDs or provider object/type pairs, verify signatures, bind the event to the expected business object, and guard state transitions against duplicates and stale event order.
-15. Review schedulers and batches.
+18. Review schedulers and batches.
     - Jobs are rerun after deploys, crashes, partial failures, and manual operator actions.
     - Use business keys such as `settlement_date + merchant_id`, `campaign_id + user_id`, import file row identity, period key, or run source ID instead of loose "already processed yesterday" checks.
-16. Review outbox and inbox pairing.
+19. Review outbox and inbox pairing.
     - DB change plus message publish can split when either side succeeds alone.
     - Route the convergence protocol for independently committed effects to `dual-write-consistency`. Keep inbox, applied-event, and duplicate-delivery handling in this skill.
-17. Review saga compensation.
+20. Review saga compensation.
     - Route compensation order and resumable saga state to `durable-workflow-orchestration`. Compensation actions such as cancel, refund, release stock, revoke entitlement, or restore credits still need idempotency keys tied to the original action here.
     - Double compensation can be worse than the original failure.
-18. Review `PROCESSING` and lease recovery.
+21. Review `PROCESSING` and lease recovery.
     - A `PROCESSING` row without lease, heartbeat, timeout, owner, or reconciliation can permanently block retries after a crash.
     - A stale processing record should recover by verifying side effects and either completing, failing safely, or allowing one fenced owner to continue.
-19. Review locks as helpers, not proof.
+22. Review locks as helpers, not proof.
     - Distributed locks can expire, split ownership, or pause through stop-the-world events.
     - Keep durable uniqueness, conditional writes, state guards, idempotency records, or fencing tokens as the final defense.
-20. Check grep bait near the operation.
-    - Review code around `Idempotency-Key`, `request_id`, `event_id`, `message_id`, `operation_id`, `dedupe`, `inbox`, `outbox`, `retry`, `timeout`, `Promise.all`, `ack`, `nack`, `visibility timeout`, `webhook`, `scheduler`, `batch`, `cron`, `PROCESSING`, `SUCCEEDED`, `FAILED`, `lock`, `Redis`, `send`, `publish`, `charge`, `refund`, `grant`, `deduct`, `createShipment`, and `cache`.
+23. Read [Operation Identity, Ordering, and Stale-Write Checklist](references/operation-ordering-stale-write-checklist.md) when duplicate delivery, out-of-order completion, aggregate sequencing, or stale writes intersect.
+24. Check grep bait near the operation.
+    - Review code around `Idempotency-Key`, `request_id`, `event_id`, `message_id`, `operation_id`, `attempt_id`, `effect_id`, `aggregate_version`, `generation`, `fencing`, `dedupe`, `inbox`, `outbox`, `retry`, `timeout`, `Promise.all`, `ack`, `nack`, `visibility timeout`, `webhook`, `scheduler`, `batch`, `cron`, `PROCESSING`, `SUCCEEDED`, `FAILED`, `lock`, `singleflight`, `Redis`, `send`, `publish`, `charge`, `refund`, `grant`, `deduct`, `createShipment`, and `cache`.
     - These terms are anchors, not automatic bugs.
-21. Require duplicate-path evidence.
-    - Good tests include same request twice, concurrent duplicate requests, first response lost, external success followed by DB failure, DB success followed by response failure, webhook replay, stale event replay, consumer commit followed by ack failure, batch rerun, processing-stuck recovery, and duplicate key with changed payload.
+25. Require duplicate, ordering, and stale-write evidence.
+    - Good tests include same request twice, concurrent duplicate requests, duplicate key with changed payload, later aggregate version before an earlier version, a missing-version gap, two attempts reading one authority, stale completion after a new generation or owner, first response lost, external success followed by DB failure, DB success followed by response failure, webhook replay, consumer commit followed by ack failure, batch rerun, and processing-stuck recovery.
     - If deterministic proof is not configured, report static risk and missing manual or integration evidence instead of approving the path.
 
 <!-- mustflow-section: postconditions -->
 ## Postconditions
 
-- The logical operation, duplicate sources, side effects, durable operation key, payload binding, response replay contract, timeout recovery, processing recovery, queue or webhook dedupe, scheduler or batch dedupe, outbox or inbox boundary, and duplicate-path tests are explicit.
-- Duplicate business requests, changed-payload key reuse, concurrent `exists` then `insert`, duplicate increments, stale state overwrites, provider timeout retries, pre-record external calls, duplicate webhook delivery, queue redelivery, scheduler reruns, double compensation, stuck processing rows, weak locks, and frontend-only guards are fixed or reported.
+- The logical operation and attempt identities, duplicate sources, ordering aggregate, authority token, gap policy, stale-result decision, side effects, durable operation key, payload binding, response replay contract, timeout recovery, processing recovery, queue or webhook dedupe, scheduler or batch dedupe, outbox or inbox boundary, and adversarial tests are explicit.
+- Duplicate business requests, changed-payload key reuse, concurrent `exists` then `insert`, per-retry operation IDs, duplicate increments, timestamp ordering, sequence gaps, stale state overwrites, cancellation-as-rollback, single-flight-as-proof, provider timeout retries, pre-record external calls, duplicate webhook delivery, queue redelivery, scheduler reruns, double compensation, stuck processing rows, weak locks, and frontend-only guards are fixed or reported.
 - Duplicate safety claims are backed by configured tests, schema evidence, framework evidence, provider documentation matched to current code, or labeled as static review risk.
 
 <!-- mustflow-section: verification -->
@@ -189,7 +209,7 @@ Prefer the narrowest configured test, build, docs, release, or mustflow intent t
 ## Output Format
 
 - Idempotency boundary reviewed
-- Logical operation, duplicate sources, operation identity, side effects, durable dedupe, payload binding, response replay, failure caching, timeout recovery, external provider, state-transition, increment or ledger, DELETE or hidden-read side-effect, queue, webhook, scheduler or batch, outbox/inbox, compensation, processing lease, lock, and test evidence findings
+- Logical operation and attempt identity, duplicate sources, ordering aggregate, authority token, gap policy, stale-result decision, side effects, durable dedupe, payload binding, response replay, failure caching, timeout recovery, external provider, state-transition, increment or ledger, DELETE or hidden-read side-effect, queue, webhook, scheduler or batch, outbox/inbox, compensation, processing lease, lock, and test evidence findings
 - Idempotency fixes made or recommended
 - Evidence level: configured-test evidence, schema evidence, provider or framework evidence, static review risk, manual-only, missing, or not applicable
 - Command intents run