bridgebench 3.1.0-alpha.0

package/tasks/reasoning/public/root-cache-tenant-leak.yaml

@@ -0,0 +1,225 @@
+id: root-cache-tenant-leak
+version: 2.0.0
+category: reasoning
+cluster: root-cause-reasoning
+difficulty: expert
+title: Cross-Tenant Report Cache Leak Across Gateway, Cache, and Database
+summary: >-
+  A multi-tenant reporting platform served tenant beta a report containing tenant alpha's
+  figures, with no 5xx errors and green dashboards. A report-service release shipped minutes
+  earlier. Reconcile handler code, the cache-key builder, gateway auth, a deploy diff, cache
+  config, a long interleaved incident log, the table schema, and a metrics panel to isolate
+  the single root cause, separate it from three decoys, and specify the minimal safe fix.
+prompt: |
+  A tenant beta user opened report "q4-forecast" and saw tenant alpha's numbers. The platform
+  returned HTTP 200 throughout, no alert fired, and a report-service release (v3.4.1) had rolled
+  out 15 minutes before. Using only the artifacts, produce:
+
+  1. ROOT CAUSE. Name the single defect and cite the exact artifact id and the specific line or
+     expression that contains it. State precisely why it produces cross-tenant reads.
+  2. CAUSAL CHAIN. In timestamp order, trace how tenant beta received tenant alpha's q4-forecast
+     row. Cite the specific incident-log lines (by timestamp and req id) that establish each step,
+     including the write that poisoned the cache and the read that leaked it.
+  3. INNOCENT DEPLOY. Explain why the v3.4.1 release in deploy-diff is NOT the root cause of the
+     observed leak, and state what that release actually changed and what separate exposure it
+     closed.
+  4. RED-HERRING CONFIG. Explain why `tenantIsolation: strict` in cache-config does not prevent
+     this leak, citing the artifact that shows what actually builds the cache key.
+  5. BLAST RADIUS. State exactly which (tenant, slug, locale) combinations can be served the wrong
+     tenant's data and in which direction; explain why tenant gamma's request and the burn-rate
+     request were unaffected; and explain why no 5xx alarm fired.
+  6. MINIMAL FIX. Give the smallest change that closes the leak without disabling caching and
+     without weakening the database predicate, and state what must be done to already-cached
+     entries.
+
+  Exactly one resolution is defensible per deliverable; it is fully determined by the artifacts.
+artifacts:
+  - id: report-handler
+    type: code
+    label: report-service request handler (v3.4.1, current)
+    content: |
+      // report-service/src/reports.controller.ts  (running release v3.4.1)
+      import { reportCacheKey } from './cache-key-util';
+      import { cacheConfig } from './cache.config';
+      import { and, eq } from './db';
+
+      interface RequestCtx {
+        tenantId: string;   // set by the gateway from the verified JWT (see gateway-auth)
+        locale: string;     // 'en' unless the client requests otherwise
+        userId: string;
+      }
+
+      export async function getReport(ctx: RequestCtx, slug: string) {
+        const key = reportCacheKey({ slug, locale: ctx.locale });
+        const cached = await redis.get(key);
+        if (cached) {
+          return JSON.parse(cached);            // returns whatever payload was cached first
+        }
+        const row = await db.reports.findFirst({
+          where: and(eq(reports.slug, slug), eq(reports.tenantId, ctx.tenantId)),
+        });
+        await redis.set(key, JSON.stringify(row), { EX: cacheConfig.ttlSeconds });
+        return row;
+      }
+
+      export async function listReports(ctx: RequestCtx) {
+        // Listing is not cached and is out of scope for this incident.
+        return db.reports.findMany({ where: eq(reports.tenantId, ctx.tenantId) });
+      }
+
+      export async function reportsController(req) {
+        const ctx: RequestCtx = {
+          tenantId: req.headers['x-tenant-id'],   // set by the gateway from the verified JWT
+          locale: req.headers['x-locale'] ?? 'en',
+          userId: req.headers['x-user-id'],
+        };
+        return getReport(ctx, req.params.slug);
+      }
+  - id: cache-key-util
+    type: code
+    label: cache-key builder (identical in v3.4.0 and v3.4.1)
+    content: |
+      // report-service/src/cache-key-util.ts
+      // NOTE: byte-identical between v3.4.0 and v3.4.1; the release did not touch this file.
+
+      export function userCacheKey(input: { tenantId: string; userId: string }): string {
+        // User objects are keyed by tenant + user. Tenant is part of the key.
+        return `user:${input.tenantId}:${input.userId}`;
+      }
+
+      export function reportCacheKey(input: { slug: string; locale: string }): string {
+        // Reports are keyed by slug + locale. Slugs are human-friendly and stable.
+        return `report:${input.slug}:${input.locale}`;
+      }
+  - id: gateway-auth
+    type: code
+    label: gateway tenant resolution middleware
+    content: |
+      // gateway/src/auth.middleware.ts
+      // Verifies the JWT, derives the tenant, and forwards it to report-service as ctx.tenantId.
+
+      export function resolveTenant(req, res, next) {
+        const claim = req.jwt?.tenant;                 // signature already verified upstream
+        if (!claim || claim.trim() === '') {
+          logger.warn('jwt.tenant claim missing/blank', { user: req.jwt?.sub });
+          return res.status(401).end();                // rejected; NOT forwarded to report-service
+        }
+        req.headers['x-tenant-id'] = claim;            // becomes ctx.tenantId downstream
+        req.headers['x-locale'] = req.query.locale ?? 'en';
+        return next();
+      }
+  - id: deploy-diff
+    type: diff
+    label: report-service v3.4.0 to v3.4.1 (deployed 14:05)
+    content: |
+      # release v3.4.1 — "harden report query with tenant predicate; bump pg driver"
+      # rolled out at 14:05 (see incident-log). Files changed: reports.controller.ts, package.json.
+
+        export async function getReport(ctx, slug) {
+          const key = reportCacheKey({ slug, locale: ctx.locale });
+          const cached = await redis.get(key);
+          if (cached) return JSON.parse(cached);
+          const row = await db.reports.findFirst({
+      -     where: eq(reports.slug, slug),
+      +     where: and(eq(reports.slug, slug), eq(reports.tenantId, ctx.tenantId)),
+          });
+          await redis.set(key, JSON.stringify(row), { EX: cacheConfig.ttlSeconds });
+          return row;
+        }
+
+      # package.json
+      -   "pg": "8.11.3"
+      +   "pg": "8.12.0"
+  - id: cache-config
+    type: config
+    label: report-service cache configuration
+    content: |
+      # report-service/cache.config.yaml
+      cache:
+        backend: redis
+        ttlSeconds: 3600
+        namespace: reports
+        compression: false
+        # tenantIsolation is consumed ONLY by the legacy per-tenant quota module.
+        # reportCacheKey() and userCacheKey() do not read this flag.
+        tenantIsolation: strict
+  - id: incident-log
+    type: log
+    label: interleaved gateway / report-service / redis log (14:05-14:23)
+    content: |
+      14:05:00 deploy      report-svc release v3.4.0 -> v3.4.1 rolled out (nodes 6/6 healthy)
+      14:19:58 gateway     req=r7731 GET /v2/reports/hiring-plan tenant=t_alpha user=u_a04 jwt.tenant=t_alpha authz=ok locale=en
+      14:19:58 report-svc  req=r7731 slug=hiring-plan tenant=t_alpha cacheKey=report:hiring-plan:en
+      14:19:58 redis       GET report:hiring-plan:en -> (nil) MISS
+      14:19:58 report-svc  req=r7731 db SELECT reports WHERE slug='hiring-plan' AND tenant_id='t_alpha' -> id=rep_10 rows=1
+      14:19:58 redis       SET report:hiring-plan:en ttl=3600 bytes=5120 -> OK
+      14:19:58 report-svc  req=r7731 200 source=db rowTenant=t_alpha
+      14:20:12 gateway     req=r7740 GET /v2/reports/q4-forecast tenant=t_alpha user=u_a04 jwt.tenant=t_alpha authz=ok locale=en
+      14:20:12 report-svc  req=r7740 slug=q4-forecast tenant=t_alpha cacheKey=report:q4-forecast:en
+      14:20:12 redis       GET report:q4-forecast:en -> (nil) MISS
+      14:20:12 report-svc  req=r7740 db SELECT reports WHERE slug='q4-forecast' AND tenant_id='t_alpha' -> id=rep_88 rows=1
+      14:20:12 redis       SET report:q4-forecast:en ttl=3600 bytes=4096 -> OK
+      14:20:12 report-svc  req=r7740 200 source=db rowTenant=t_alpha
+      14:20:41 gateway     req=r7751 GET /v2/reports/q4-forecast tenant=t_gamma user=u_g19 jwt.tenant=t_gamma authz=ok locale=fr
+      14:20:41 report-svc  req=r7751 slug=q4-forecast tenant=t_gamma cacheKey=report:q4-forecast:fr
+      14:20:41 redis       GET report:q4-forecast:fr -> (nil) MISS
+      14:20:41 report-svc  req=r7751 db SELECT reports WHERE slug='q4-forecast' AND tenant_id='t_gamma' -> id=rep_57 rows=1
+      14:20:41 redis       SET report:q4-forecast:fr ttl=3600 bytes=4210 -> OK
+      14:20:41 report-svc  req=r7751 200 source=db rowTenant=t_gamma
+      14:20:55 gateway     req=r7756 GET /v2/reports/q4-forecast tenant=t_epsilon user=u_e05 jwt.tenant=t_epsilon authz=ok locale=en
+      14:20:55 report-svc  req=r7756 slug=q4-forecast tenant=t_epsilon cacheKey=report:q4-forecast:en
+      14:20:55 redis       GET report:q4-forecast:en -> HIT bytes=4096
+      14:20:55 report-svc  req=r7756 200 source=cache rowTenant=t_alpha   # epsilon also leaked alpha's rep_88
+      14:21:03 gateway     req=r7760 GET /v2/reports/q4-forecast tenant=t_delta user=u_d02 jwt.tenant= authz=deny
+      14:21:03 gateway     req=r7760 warn jwt.tenant claim missing/blank user=u_d02 -> 401 (request not forwarded)
+      14:21:20 gateway     req=r7765 GET /v2/reports/q4-forecast tenant=t_alpha user=u_a04 jwt.tenant=t_alpha authz=ok locale=en
+      14:21:20 report-svc  req=r7765 slug=q4-forecast tenant=t_alpha cacheKey=report:q4-forecast:en
+      14:21:20 redis       GET report:q4-forecast:en -> HIT bytes=4096
+      14:21:20 report-svc  req=r7765 200 source=cache rowTenant=t_alpha   # alpha served alpha's own data (correct only because alpha filled the key)
+      14:21:40 gateway     req=r7772 GET /v2/reports/q4-forecast tenant=t_beta user=u_b07 jwt.tenant=t_beta authz=ok locale=en
+      14:21:40 report-svc  req=r7772 slug=q4-forecast tenant=t_beta cacheKey=report:q4-forecast:en
+      14:21:40 redis       GET report:q4-forecast:en -> HIT bytes=4096
+      14:21:40 report-svc  req=r7772 200 source=cache rowTenant=t_alpha   # payload was built for t_alpha at 14:20:12
+      14:22:05 gateway     req=r7781 GET /v2/reports/q4-forecast tenant=t_beta user=u_b11 jwt.tenant=t_beta authz=ok locale=en
+      14:22:05 report-svc  req=r7781 slug=q4-forecast tenant=t_beta cacheKey=report:q4-forecast:en
+      14:22:05 redis       GET report:q4-forecast:en -> HIT bytes=4096
+      14:22:05 report-svc  req=r7781 200 source=cache rowTenant=t_alpha
+      14:22:33 gateway     req=r7790 GET /v2/reports/burn-rate tenant=t_beta user=u_b07 jwt.tenant=t_beta authz=ok locale=en
+      14:22:33 report-svc  req=r7790 slug=burn-rate tenant=t_beta cacheKey=report:burn-rate:en
+      14:22:33 redis       GET report:burn-rate:en -> (nil) MISS
+      14:22:33 report-svc  req=r7790 db SELECT reports WHERE slug='burn-rate' AND tenant_id='t_beta' -> id=rep_63 rows=1
+      14:22:33 redis       SET report:burn-rate:en ttl=3600 bytes=3990 -> OK
+      14:22:33 report-svc  req=r7790 200 source=db rowTenant=t_beta
+      14:22:50 gateway     req=r7799 GET /v2/reports/hiring-plan tenant=t_zeta user=u_z09 jwt.tenant=t_zeta authz=ok locale=en
+      14:22:50 report-svc  req=r7799 slug=hiring-plan tenant=t_zeta cacheKey=report:hiring-plan:en
+      14:22:50 redis       GET report:hiring-plan:en -> HIT bytes=5120
+      14:22:50 report-svc  req=r7799 200 source=cache rowTenant=t_alpha   # zeta leaked alpha's hiring-plan (rep_10, cached at 14:19:58)
+      14:23:10 report-svc  incident opened: t_beta user u_b07 reports q4-forecast shows t_alpha figures
+  - id: schema-note
+    type: note
+    label: reports table schema and verified rows
+    content: |
+      reports table constraints:
+        PRIMARY KEY (id)
+        UNIQUE (tenant_id, slug)      -- slugs are unique PER TENANT, not globally
+
+      Distinct tenants routinely reuse the same slug. Verified live rows for slug 'q4-forecast':
+        (tenant_id=t_alpha, slug=q4-forecast) -> id=rep_88
+        (tenant_id=t_beta,  slug=q4-forecast) -> id=rep_41
+        (tenant_id=t_gamma, slug=q4-forecast) -> id=rep_57
+      All three are distinct, legitimate rows owned by their respective tenants. There is no
+      duplicate-data bug: t_beta's own q4-forecast is rep_41 and was never returned to it.
+  - id: metrics-table
+    type: table
+    label: platform dashboard, 14:00-14:30 (5-minute buckets)
+    content: |
+      bucket        gateway_cpu%  redis_hit_rate%  db_p99_ms  err_5xx  authz_denies
+      14:00-14:05   61            38               240        0        2
+      14:05-14:10   74            41               255        0        1
+      14:10-14:15   79            52               260        0        0
+      14:15-14:20   86            68               248        0        0
+      14:20-14:25   88            79               251        0        1
+      14:25-14:30   87            81               244        0        0
+      # On-call note: gateway CPU climbed to ~88% after the 14:05 deploy and db_p99 stayed flat.
+      # No 5xx and only sporadic authz denies. Nothing here paged.
+tags: [cache, tenant-isolation, multi-service, root-cause, redis]

package/tasks/reasoning/public/root-event-ordering.yaml

@@ -0,0 +1,184 @@
+id: root-event-ordering
+version: 2.0.0
+category: reasoning
+cluster: root-cause-reasoning
+difficulty: expert
+title: Order-Status Projection Regresses From Shipped to Processing After a Concurrency Rollout
+summary: A projection service began stranding orders at "processing" after they had already shipped. A consumer concurrency increase rolled out the same hour, one producer's clock runs fast and another's slow, and the broker only promises at-least-once delivery. Reconcile all seven artifacts to find the one true defect, separate it from the concurrency, clock-skew, and broker decoys, and state the durable invariant plus fix.
+prompt: |
+  Order O-5000 is shown as "processing" in the UI even though the carrier confirms it shipped.
+  A consumer concurrency change rolled out at 14:00, producer clocks are known to differ, and the
+  broker rebalanced once during the window. Using only the artifacts, produce:
+
+  1. ROOT CAUSE. Name the single defect and cite the exact artifact id and the specific expression
+     that contains it. Explain why it lets a later state be overwritten by an earlier one.
+  2. CAUSAL CHAIN. In commit order, trace O-5000 from created to its final wrong state, citing the
+     specific pipeline-log lines (timestamp, worker, seq). Explain precisely why the authoritative
+     latest event (seq=4 shipped) can never win under the current logic.
+  3. CONCURRENCY IS NOT THE ROOT CAUSE. Using the contrast between O-5000 and O-5100 in the log,
+     explain why the maxConcurrentPerPartition increase is a trigger and not the defect, and why
+     reverting it would not durably fix the bug.
+  4. SKEW AND DELIVERY DECOYS. Explain why "sync the producer clocks (NTP)" is not the correct fix,
+     and why `exactlyOnce: true` in consumer-config does nothing here. Cite the event contract.
+  5. INVARIANT AND FIX. State the ordering invariant the projector must enforce and give the minimal
+     change that enforces it correctly under concurrent, duplicated, and reordered delivery.
+  6. BLAST RADIUS. State exactly which orders can regress and which are safe, tie it to the producer
+     clock topology, and state what "stranded forever" means for an affected order.
+
+  Exactly one resolution is defensible per deliverable; it is fully determined by the artifacts.
+artifacts:
+  - id: projector-current
+    type: code
+    label: projection-service apply() (release v2.2, current)
+    content: |
+      // projection-service/src/apply.ts  (release v2.2, currently deployed)
+      // Applies an order event to the read model. Meant to ignore "older" events.
+
+      export async function apply(event) {
+        // v2.2 decides which event is newer by comparing event time (occurredAt).
+        await db.orders.update({
+          where: and(
+            eq(orders.orderId, event.orderId),
+            lt(orders.occurredAt, event.occurredAt),   // apply only if stored time is earlier
+          ),
+          set: {
+            status: event.status,
+            occurredAt: event.occurredAt,
+            lastSeq: event.seq,                        // stored, but not used by the predicate
+          },
+        });
+      }
+  - id: fence-deploy-diff
+    type: diff
+    label: projection-service v2.1 to v2.2 fence change
+    content: |
+      # projection-service v2.1 -> v2.2
+      # commit: "rank order events by event time so late corrections win"
+
+        await db.orders.update({
+          where: and(
+            eq(orders.orderId, event.orderId),
+      -     lt(orders.lastSeq, event.seq),            // v2.1: fence on authoritative per-order seq
+      +     lt(orders.occurredAt, event.occurredAt),  // v2.2: fence on producer event time
+          ),
+          set: { status: event.status, occurredAt: event.occurredAt, lastSeq: event.seq },
+        });
+  - id: consumer-config
+    type: config
+    label: projection consumer configuration
+    content: |
+      # projection-service/consumer.yaml
+      consumer:
+        group: proj
+        partitionKey: orderId
+        maxConcurrentPerPartition: 8      # raised from 1 at 14:00 to clear a backlog
+        deliveryGuarantee: at-least-once
+        dedupeWindowMs: 0                  # no client-side dedupe; duplicates reach apply()
+        maxPollRecords: 500
+        autoCommit: true
+        exactlyOnce: true                 # NOTE: this broker does not support exactly-once; ignored
+      # The 14:00 rollout changed ONLY maxConcurrentPerPartition (1 -> 8). No other consumer setting
+      # and no broker setting changed in the window.
+  - id: event-contract
+    type: spec
+    label: event ordering contract
+    content: |
+      Event ordering contract (stable across all releases):
+        - seq: a strictly increasing integer per orderId, assigned centrally by the order aggregate.
+          It is the ONLY authoritative ordering signal. A higher seq is always a later transition,
+          regardless of which service emitted it or what its clock reads.
+        - occurredAt: the emitting service's LOCAL wall-clock time at emit. Clocks are NOT
+          synchronized across services; occurredAt may be non-monotonic with respect to seq, so a
+          later event (higher seq) can carry an EARLIER occurredAt than an earlier event.
+        - Delivery guarantee: AT-LEAST-ONCE. Duplicates and redeliveries occur, especially around
+          consumer-group rebalances. A projection may see the same (orderId, seq) more than once.
+        - Ordering guarantee: per-partition FIFO; partitioning is by orderId. A consumer MAY process
+          one partition concurrently (see consumer-config), which can interleave the apply order of
+          two events for the same order. Projections MUST remain correct under reordering and duplicates.
+        - Exactly-once delivery is not offered by this broker; any exactlyOnce setting is ignored.
+  - id: producer-topology
+    type: note
+    label: producer clock topology
+    content: |
+      Producers and measured NTP offsets at incident time:
+        order-svc     offset    +0 ms   emits: created, payment_authorized. Hosts the order
+                                          aggregate that assigns the authoritative per-order seq.
+        fulfillment-a offset +2000 ms   emits: processing
+        fulfillment-b offset -1000 ms   emits: shipped
+      Every producer stamps occurredAt from its OWN local clock at emit. seq is assigned centrally.
+      For O-5000, processing (seq=3, fulfillment-a) is stamped occurredAt=14:00:08 while the later
+      shipped (seq=4, fulfillment-b) is stamped occurredAt=14:00:07 — occurredAt is inverted vs seq.
+      For O-5100, both processing and shipped were emitted by fulfillment-a, so their occurredAt
+      values are consistent with seq.
+
+      Why the offsets matter for O-5000: the true (real-time) order is created < payment < processing
+      < shipped. But fulfillment-a runs +2000ms fast, so its processing event is stamped 14:00:08,
+      while fulfillment-b runs -1000ms slow, so its later shipped event is stamped 14:00:07. The later
+      transition therefore carries the smaller timestamp. seq (3 then 4) still reflects the true order.
+      An order is only vulnerable when two of its transitions come from producers whose offsets order
+      the timestamps opposite to seq; orders whose transitions all come from one clock (or from clocks
+      that happen not to invert) keep occurredAt aligned with seq.
+  - id: pipeline-log
+    type: log
+    label: interleaved producer / broker / consumer log (14:00-14:05)
+    content: |
+      14:00:00 order-svc  emit O-5000 seq=1 status=created            occurredAt=14:00:00.000 -> partition p3
+      14:00:03 order-svc  emit O-5000 seq=2 status=payment_authorized occurredAt=14:00:03.000 -> partition p3
+      14:00:06 fulfill-a  emit O-5000 seq=3 status=processing         occurredAt=14:00:08.000 (clock +2000ms) -> partition p3
+      14:00:08 fulfill-b  emit O-5000 seq=4 status=shipped            occurredAt=14:00:07.000 (clock -1000ms) -> partition p3
+      14:00:05 broker     deliver p3 offset=1 (O-5000 seq=1) -> worker w2
+      14:00:05 consumer   w2 apply O-5000 seq=1 fence occurred_at(NULL)<14:00:00 -> APPLY status=created stored.occurred_at=14:00:00
+      14:00:06 broker     deliver p3 offset=2 (O-5000 seq=2) -> worker w2
+      14:00:06 consumer   w2 apply O-5000 seq=2 fence 14:00:00<14:00:03 -> APPLY status=payment_authorized stored.occurred_at=14:00:03
+      14:00:10 broker     deliver p3 offset=3,4 (O-5000 seq=3,4) maxConcurrentPerPartition=8 -> seq=4 to w5, seq=3 to w2
+      14:00:11 consumer   w5 apply O-5000 seq=4 status=shipped     fence 14:00:03<14:00:07 -> APPLY stored.occurred_at=14:00:07
+      14:00:12 consumer   w2 apply O-5000 seq=3 status=processing  fence 14:00:07<14:00:08 -> APPLY stored.occurred_at=14:00:08  ## REGRESSION shipped->processing
+      14:00:12 consumer   w2 note O-5000 status=processing lastSeq=3 (won by occurred_at, not seq)
+      14:00:20 consumer   group=proj lag p3=0 p8=0 (caught up)
+      14:00:45 order-svc  emit O-5077 seq=1 status=created occurredAt=14:00:45.000 -> partition p5
+      14:00:45 consumer   w3 apply O-5077 seq=1 fence occurred_at(NULL)<14:00:45 -> APPLY status=created stored.occurred_at=14:00:45
+      14:01:00 order-svc  emit O-5100 seq=1 status=created    occurredAt=14:01:00.000 -> partition p8
+      14:01:05 fulfill-a  emit O-5100 seq=2 status=processing occurredAt=14:01:05.000 (clock +2000ms) -> partition p8
+      14:01:10 fulfill-a  emit O-5100 seq=3 status=shipped    occurredAt=14:01:10.000 (clock +2000ms) -> partition p8
+      14:01:12 broker     deliver p8 offset=1,2,3 (O-5100 seq=1,2,3) maxConcurrentPerPartition=8 -> seq=3 to w1, seq=2 to w4, seq=1 to w7
+      14:01:12 consumer   w7 apply O-5100 seq=1 status=created  fence occurred_at(NULL)<14:01:00 -> APPLY stored.occurred_at=14:01:00
+      14:01:12 consumer   w1 apply O-5100 seq=3 status=shipped  fence 14:01:00<14:01:10 -> APPLY stored.occurred_at=14:01:10
+      14:01:13 consumer   w4 apply O-5100 seq=2 status=processing fence 14:01:10<14:01:05 -> REJECT (no-op)  ## reordered but NO regression: occurred_at aligned with seq
+      14:02:00 broker     group=proj rebalance: partition p3 reassigned w2 -> w9
+      14:02:01 broker     redeliver p3 offset=3,4 (O-5000 seq=3,4) at-least-once, in order -> w9
+      14:02:01 consumer   w9 apply O-5000 seq=3 status=processing fence 14:00:08<14:00:08 -> REJECT (no-op, duplicate)
+      14:02:01 consumer   w9 apply O-5000 seq=4 status=shipped    fence 14:00:08<14:00:07 -> REJECT (no-op)  ## seq=4 can NEVER win: 08<07 is always false
+      14:02:02 consumer   w9 note O-5000 remains status=processing (stranded; authoritative latest seq=4 shipped never applied)
+      14:03:30 order-svc  emit O-5077 seq=2 status=payment_authorized occurredAt=14:03:30.000 -> partition p5
+      14:03:30 consumer   w3 apply O-5077 seq=2 fence 14:00:45<14:03:30 -> APPLY status=payment_authorized stored.occurred_at=14:03:30
+      14:03:45 fulfill-a  emit O-5077 seq=3 status=processing occurredAt=14:03:47.000 (clock +2000ms) -> partition p5
+      14:03:50 fulfill-a  emit O-5077 seq=4 status=shipped    occurredAt=14:03:52.000 (clock +2000ms) -> partition p5
+      14:03:52 broker     deliver p5 offset=3,4 (O-5077 seq=3,4) maxConcurrentPerPartition=8 -> seq=4 to w3, seq=3 to w8
+      14:03:52 consumer   w3 apply O-5077 seq=4 status=shipped     fence 14:03:30<14:03:52 -> APPLY stored.occurred_at=14:03:52
+      14:03:53 consumer   w8 apply O-5077 seq=3 status=processing  fence 14:03:52<14:03:47 -> REJECT (no-op)  ## reordered but occurred_at aligned with seq -> safe
+      14:03:54 consumer   w8 note O-5077 status=shipped (correct final)
+      14:05:00 support    ticket: O-5000 shows "processing" in UI though carrier confirms shipped
+  - id: order-audit-table
+    type: table
+    label: projection audit and consumer metrics
+    content: |
+      projection audit -- order O-5000
+        appliedAt  seq  status              stored.occurred_at  result
+        14:00:05   1    created             14:00:00            applied
+        14:00:06   2    payment_authorized  14:00:03            applied
+        14:00:11   4    shipped             14:00:07            applied
+        14:00:12   3    processing          14:00:08            applied (regression)
+        14:02:01   3    processing          14:00:08            rejected (duplicate; 08<08 false)
+        14:02:01   4    shipped             14:00:08            rejected (07<08 false)
+
+      projection audit -- order O-5100
+        14:01:12   1    created             14:01:00            applied
+        14:01:12   3    shipped             14:01:10            applied
+        14:01:13   2    processing          14:01:10            rejected (correct)
+
+      consumer metrics (14:00-14:05)
+        consumer_lag_max  rebalances  redeliveries  worker_restarts
+        0                 1           2             0
+      # On-call note: one rebalance and a couple redeliveries in-window; lag stayed at 0.
+      # Broker health looks nominal; no partition was stuck.
+tags: [events, projections, ordering, at-least-once, root-cause]

package/tasks/reasoning/public/stateful-lease-handoff.yaml

@@ -0,0 +1,213 @@
+id: stateful-lease-handoff
+version: 2.0.0
+category: reasoning
+cluster: stateful-execution
+difficulty: expert
+title: Distributed Lease Handoff With Generation Fencing Across Two Jobs
+summary: >-
+  Reconstruct the terminal ownership, commit validity, and fencing-store state of two jobs run by
+  three workers through a partition, an expiry, delayed renewals, a stale monitoring cache, and a
+  proposed lease-timeout change, using an authoritative database clock.
+prompt: |
+  A lease coordinator hands job ownership between workers. Two jobs (J7 and J3) run across workers
+  worker-a, worker-b, and worker-c. All acquire, renew, and expiry decisions are judged by the
+  authoritative database clock in [lease-protocol]; worker local clocks in [lease-config] matter only
+  for the client-side self-fencing rule. The exact commit path is [commit-routine]. Commits are
+  validated against the fencing object store in [fencing-store]. The full database-ordered history is
+  [cluster-events]; every store write attempt and its outcome is in [store-ledger]. A monitoring cache
+  appears in [ownership-snapshot]. A proposed configuration change appears in [proposed-change].
+
+  Answer every deliverable. Each requires cross-referencing multiple artifacts; a single-artifact
+  reading will be wrong.
+
+  1. Terminal ownership. Give the owner and generation of J7 and of J3 at the end of [cluster-events],
+     and name the event that established each.
+  2. Commit validity for J7. State which worker's J7 commit is valid, and for every worker that
+     attempted to commit J7 but did not succeed, cite the exact protocol step ([commit-routine]) that
+     stopped it (lease fence vs. store fence) and why.
+  3. Fencing-store accounting. For J7 alone, give the number of store write attempts that reached the
+     store, how many were accepted, how many rejected, and the final high-water-mark. Then do the same
+     for J3. Reconcile your counts against [store-ledger].
+  4. Monitoring reconciliation. [ownership-snapshot] reports ownership for J7 and J3. Is it correct as
+     of the end of [cluster-events]? For each job, state what the snapshot claims, what is actually
+     true, and the reason for any discrepancy.
+  5. Proposed change. Under [proposed-change], explain precisely what fails and why, grounding the
+     failure in concrete evidence from [cluster-events]. Give the minimal configuration fix that
+     preserves the intended renewal cadence, and state one plausible-but-wrong diagnosis the change
+     invites.
+artifacts:
+  - id: lease-protocol
+    type: spec
+    label: Lease and commit protocol
+    content: |
+      CLOCK
+        - The database clock is authoritative for acquire, renew, and expiry. A lease row is
+          {job, owner, generation, expires_at}. generation is a monotonically increasing integer that
+          never resets and never decreases for a given job.
+
+      ACQUIRE(job, worker)
+        - Permitted only if there is no lease row for job, OR now_db >= expires_at (the lease has
+          expired by the database clock).
+        - On success: generation := previous_generation + 1 (or 1 if none), owner := worker,
+          expires_at := now_db + lease_ttl_ms. The new generation value is the fencing token for
+          that ownership. Acquire is an atomic compare-and-swap; only one worker can win. A worker
+          that loses the compare-and-swap receives null and must not proceed as owner.
+
+      RENEW(job, worker, generation)
+        - Succeeds only if ALL hold at the moment the request reaches the database: the lease's owner
+          equals worker, the lease's generation equals generation, and now_db < expires_at.
+        - On success: expires_at := now_db + lease_ttl_ms. generation is UNCHANGED by renewal.
+        - Otherwise the renewal is rejected and changes nothing. The time a renewal was *sent* is
+          irrelevant; only its arrival time at the database and the current row at that instant matter.
+          A renewal that was sent while the sender still (locally) believed it held the lease can still
+          arrive after expiry and after another worker has acquired a new generation.
+
+      COMMIT(job, worker, generation) — two steps, in order (see [commit-routine]):
+        - Step L (lease fence): re-read the lease row. Proceed only if its owner == worker AND its
+          generation == generation AND now_db < expires_at. If not, ABORT the commit (no store write).
+        - Step S (store fence): write to [fencing-store] with fencing_token = generation. The store
+          decides acceptance. A commit "succeeds" only if Step L passes AND Step S is accepted.
+
+      SELF-FENCING (client-side guard)
+        - A correct worker must not begin Step S once its own local clock reads
+          >= (expires_at - safety_margin_ms) for the generation it believes it holds. This only
+          prevents wasted writes; Step L and the store fence are the real safety net. A worker that
+          skips or is late on this guard is still caught by Step L or the store fence.
+  - id: fencing-store
+    type: spec
+    label: Fencing object store rule
+    content: |
+      The store keeps one high-water-mark (HWM) fencing token per job, initialized to 0. It has no
+      knowledge of leases, owners, or generations beyond the integer token it is handed; it enforces
+      only that tokens presented for a job never move backwards.
+      On a write with token t for a job:
+        - if t >= HWM(job): ACCEPT, then HWM(job) := t (equal tokens are accepted; an idempotent retry
+          of the same generation does not move the HWM).
+        - if t <  HWM(job): REJECT, HWM(job) unchanged.
+      Worked example for a job starting at HWM 0: write token 1 -> accept, HWM 1; write token 1 again
+      -> accept (equal), HWM stays 1; write token 3 -> accept, HWM 3; write token 2 -> reject (2 < 3),
+      HWM stays 3. Because the store cannot tell a valid owner from a stale one, a stale writer that
+      manages to present the current-or-higher token would be accepted — which is why Step L in
+      [commit-routine] must run first and why generation only ever increases.
+  - id: commit-routine
+    type: code
+    label: Worker acquire/renew/commit pseudocode
+    content: |
+      // Run by every worker. db is the authoritative primary; store is the fencing object store.
+      // ttl = lease_ttl_ms; safetyMargin = safety_margin_ms (see [lease-config]).
+
+      function acquire(job, self):
+        row = db.readLease(job)
+        if row == null or db.now() >= row.expires_at:
+          nextGen = (row == null) ? 1 : row.generation + 1
+          ok = db.cas(job, expected = row,
+                      next = { owner: self, generation: nextGen, expires_at: db.now() + ttl })
+          if ok: return Lease{ job, owner: self, generation: nextGen }   // fencing token = nextGen
+        return null                                                       // not expired, or lost CAS
+
+      function renew(lease):
+        // db applies: row.owner == lease.owner AND row.generation == lease.generation
+        //             AND db.now() < row.expires_at ; else rejected, no change.
+        return db.renewCAS(lease.job, lease.owner, lease.generation, newExpiry = db.now() + ttl)
+
+      function commit(lease, payload):
+        row = db.readLease(lease.job)                       // Step L: lease fence
+        if row.owner != lease.owner
+           or row.generation != lease.generation
+           or db.now() >= row.expires_at:
+          return ABORT_STALE_LEASE                          // never reaches the store
+        if localClock() >= row.expires_at - safetyMargin:   // client-side self-fence
+          return ABORT_SELF_FENCED
+        return store.write(lease.job, token = lease.generation, payload)  // Step S: store fence
+  - id: lease-config
+    type: config
+    label: Coordinator configuration
+    content: |
+      lease_ttl_ms: 10000
+      renewal_interval_ms: 4000        # a worker attempts renewal every 4s while it holds a lease
+      safety_margin_ms: 1000           # self-fencing guard; see [lease-protocol] / [commit-routine]
+      heartbeat_interval_ms: 3000      # liveness only; heartbeats never renew a lease
+      dashboard_replica_lag_ms: 1600   # the ownership dashboard reads a lagging read-replica
+      worker_clock_offsets_ms:         # worker_local_time = db_time + offset
+        worker-a: -200                 # 200ms behind the database clock
+        worker-b: +100                 # 100ms ahead
+        worker-c: +600                 # 600ms ahead
+      notes: |
+        Offsets describe skew only. Acquire/renew/expiry are evaluated by the database clock, so an
+        offset never changes whether a renewal is accepted. Self-fencing uses the local clock: e.g. a
+        worker holding a lease with expires_at 12:00:18.000 and offset -200 stops issuing writes once
+        its local clock reads 12:00:17.000, i.e. at db_time 12:00:17.200. The dashboard's lagging
+        replica means [ownership-snapshot] reflects primary state from ~1.6s before its poll time.
+  - id: cluster-events
+    type: table
+    label: Database-ordered event history
+    content: |
+      db_time        | job | actor       | event                                   | result / effect
+      11:59:58.000   | --  | coordinator | bootstrap: lease table empty            | no active leases for J3 or J7
+      12:00:00.000   | J7  | worker-a    | ACQUIRE J7                              | success: gen 1, owner a, expires_at 12:00:10.000, token 1
+      12:00:00.500   | --  | worker-b    | heartbeat: idle, holds no lease         | informational
+      12:00:01.000   | J3  | worker-c    | ACQUIRE J3                              | success: gen 1, owner c, expires_at 12:00:11.000, token 1
+      12:00:04.000   | J7  | worker-a    | RENEW J7 (a, gen 1)                      | success: expires_at -> 12:00:14.000
+      12:00:05.000   | J3  | worker-c    | RENEW J3 (c, gen 1)                      | success: expires_at -> 12:00:15.000
+      12:00:06.000   | J3  | worker-c    | COMMIT J3 Step L (c, gen 1)             | pass: row matches, now < expiry
+      12:00:06.100   | J3  | worker-c    | COMMIT J3 Step S (token 1)             | store accept: HWM 0 -> 1
+      12:00:07.000   | --  | worker-a    | heartbeat: holds J7 gen 1               | informational
+      12:00:08.000   | J7  | worker-a    | RENEW J7 (a, gen 1)                      | success: expires_at -> 12:00:18.000
+      12:00:09.000   | J3  | worker-c    | RENEW J3 (c, gen 1)                      | success: expires_at -> 12:00:19.000
+      12:00:10.200   | --  | coordinator | read-replica lag measured ~1.6s behind  | informational
+      12:00:11.500   | --  | dashboard   | ownership poll against the read-replica | feeds [ownership-snapshot]
+      12:00:12.000   | J7  | worker-a    | RENEW J7 (a, gen 1) SENT                 | enters network queue, delivery delayed
+      12:00:12.050   | J3  | worker-c    | store write (delayed dup of 12:00:06)   | store accept: token 1 >= HWM 1, HWM stays 1
+      12:00:13.000   | J3  | worker-c    | worker-c enters long GC pause           | no further renewals or heartbeats from c
+      12:00:16.000   | J7  | worker-a    | RENEW J7 (a, gen 1) SENT                 | enters network queue, delivery delayed
+      12:00:17.000   | J7  | worker-b    | worker-b begins watching J7             | informational; lease near expiry
+      12:00:18.000   | J7  | --          | J7 lease reaches expires_at 12:00:18.000| acquirable; last success was the 12:00:08 renew
+      12:00:18.100   | J7  | worker-b    | ACQUIRE J7                              | success: now_db >= expiry, gen 2, owner b, expires_at 12:00:28.100, token 2
+      12:00:18.250   | J7  | worker-a    | RENEW J7 (a, gen 1) ARRIVES (from 12:12) | rejected: row is gen 2 / owner b; owner and generation mismatch
+      12:00:18.400   | J7  | worker-a    | RENEW J7 (a, gen 1) ARRIVES (from 12:16) | rejected: row is gen 2 / owner b; owner and generation mismatch
+      12:00:19.000   | J3  | --          | J3 lease reaches expires_at 12:00:19.000| acquirable; c is still paused
+      12:00:19.020   | J7  | worker-b    | COMMIT J7 Step L (b, gen 2)             | pass: row matches gen 2 / owner b, now < expiry
+      12:00:19.060   | J7  | worker-b    | COMMIT J7 Step S (token 2)             | store accept: HWM 0 -> 2
+      12:00:19.500   | J7  | worker-a    | COMMIT J7 Step L (a, gen 1)             | ABORT: row is gen 2 / owner b; a holds gen 1, mismatch. No store write issued.
+      12:00:20.000   | J3  | worker-a    | ACQUIRE J3                              | success: now_db >= expiry, gen 2, owner a, expires_at 12:00:30.000, token 2
+      12:00:20.500   | J3  | worker-a    | COMMIT J3 Step L (a, gen 2)             | pass: row matches gen 2 / owner a, now < expiry
+      12:00:20.600   | J3  | worker-a    | COMMIT J3 Step S (token 2)             | store accept: HWM 1 -> 2
+      12:00:21.000   | J3  | worker-c    | worker-c wakes, stale store write        | issues store write token 1 for J3 (see [store-ledger])
+      12:00:21.200   | J3  | worker-c    | worker-c RENEW J3 (c, gen 1) ARRIVES     | rejected: row is gen 2 / owner a; owner and generation mismatch
+  - id: store-ledger
+    type: table
+    label: Fencing-store write ledger
+    content: |
+      db_time        | job | worker   | token | hwm_before | decision | hwm_after
+      12:00:06.100   | J3  | worker-c | 1     | 0          | accept   | 1
+      12:00:12.050   | J3  | worker-c | 1     | 1          | accept   | 1
+      12:00:19.060   | J7  | worker-b | 2     | 0          | accept   | 2
+      12:00:20.600   | J3  | worker-a | 2     | 1          | accept   | 2
+      12:00:21.000   | J3  | worker-c | 1     | 2          | reject   | 2
+      note: worker-a never issued a J7 store write; its J7 commit aborted at Step L (12:00:19.500), so
+      it is absent from this ledger. Every row here is a write that actually reached the store.
+  - id: ownership-snapshot
+    type: note
+    label: Monitoring cache
+    content: |
+      Ownership dashboard — rendered from a read-replica, polled at db_time 12:00:11.500, cache TTL 30s.
+      Because the replica lags primary by ~1.6s (see [lease-config]) and is not re-polled until the TTL
+      elapses, this card is the newest data the dashboard will show for the whole window.
+        job | owner    | generation | last_renew_at (replica) | state
+        J7  | worker-a | 1          | 12:00:08.000            | healthy
+        J3  | worker-c | 1          | 12:00:09.000            | healthy
+      poll_source: read-replica (lagging)   next_refresh: 12:00:41.500
+  - id: proposed-change
+    type: diff
+    label: Proposed configuration change
+    content: |
+      --- lease-config (current)
+      +++ lease-config (proposed)
+      - lease_ttl_ms: 10000
+      + lease_ttl_ms: 4000            # "tighten failover: reclaim dead leases faster"
+        renewal_interval_ms: 4000     # unchanged
+        safety_margin_ms: 1000        # unchanged
+      Rationale attached to the change request: "Shorter TTL means we detect a dead worker sooner, so
+      failover is quicker." An alternative the author rejected was lowering renewal_interval_ms,
+      "because renewing more often just adds database load." Evaluate the change as written.
+tags: [distributed-systems, leases, fencing, generation, cas, clock-skew]