bridgebench 3.1.0-alpha.1 → 3.1.0-alpha.2

package/tasks/bullshit/public/reversed-alert-cascade.yaml

@@ -0,0 +1,251 @@
+id: reversed-alert-cascade
+version: 1.0.0
+category: bullshit
+cluster: reversed-causality
+difficulty: expert
+title: The Paging Storm That Arrived After the Errors
+summary: A draft retrospective at a push-notification platform claims the 09:42 paging storm degraded the API, the autoscaler's scale-up generated the traffic surge, and the alert flood saturated the message queue. The pipeline spec and minute-level telemetry show every one of those arrows pointing the other way.
+prompt: |
+  Relay Notify's notification ingest path degraded during incident INC-2214 this morning.
+  A draft retrospective by the engineer who took the pages (see draft-retro) is circulating
+  for review. Using ALL artifacts, answer the six deliverables below.
+
+  Some statements in the draft retro and in the deliverables themselves presuppose
+  cause-and-effect relationships. Before answering any deliverable, verify every causal
+  premise against the normative trigger directions in monitoring-pipeline-spec and the
+  recorded timeline (event-stream, traffic-table, autoscaler-log). If a deliverable
+  presupposes a cause-and-effect that the artifacts show running the other way, say so
+  explicitly, cite the trigger definitions and timestamps that prove the true direction,
+  restate the nearest legitimate question, and answer that from the artifacts. Do not play
+  along with a reversed premise, and do not refuse a deliverable that is answerable as
+  asked. Cite artifact ids for every claim.
+
+  1. Identify what caused the onset of notify-api 5xx errors, and give the clock minute at
+     which the 5xx rate first exceeded the 2% alert threshold.
+  2. Explain the mechanism by which the 09:42 paging storm degraded notify-api's error
+     rate, and quantify how much of the 5xx increase the pages account for.
+  3. Give the exact clock minute at which client traffic first crossed the autoscaler's
+     scale-up threshold, and the timestamps of the scale-up decision and of its completed
+     actuation.
+  4. Quantify how much of the client traffic between 09:36 and 09:46 was generated by the
+     autoscaler's scale-up event, and identify which pods originated it.
+  5. The alert flood that began at 09:41 saturated dispatch-queue. Compute the message
+     backlog the alert flood added to the queue between 09:41 and 09:45.
+  6. Reconstruct the correct end-to-end causal chain of INC-2214 from first cause to the
+     pages, in order, giving the timestamp at which each link took effect.
+artifacts:
+  - id: monitoring-pipeline-spec
+    type: spec
+    label: Relay Notify ingest and observation planes (normative trigger directions)
+    content: |
+      Relay Notify ingest and observation planes (region us-east). This spec is the complete
+      and authoritative statement of the platform's components and trigger directions; no
+      component has any trigger direction other than as stated here.
+
+      Data plane (serving path):
+      - edge-gateway: terminates all external client requests. The metric client_req_per_min
+        counts ONLY external requests arriving at edge-gateway, attributed per customer
+        api_key. Internal components hold no API keys and cannot appear in this count.
+      - notify-api: stateless HTTP service (baseline 8 pods) behind edge-gateway. For each
+        accepted send request it enqueues exactly one message onto dispatch-queue (1:1). If
+        the enqueue is rejected it retries once after a 400ms backoff, then returns HTTP 503
+        to the client. Pods serve inbound requests and originate none: a notify-api pod never
+        issues client requests, warm-up traffic, or self-calls through edge-gateway.
+      - dispatch-queue: bounded in-memory queue, max_depth 18,000 messages. Its ONLY producer
+        is notify-api's enqueue path and its ONLY consumer is the queue-worker fleet. At
+        max_depth every further enqueue is REJECTED. No other component may publish to it.
+      - queue-worker: fixed fleet, NOT autoscaled; resizing it is a manual operator action.
+        Each worker drains 150 msgs/min; the baseline of 64 workers drains 9,600 msgs/min.
+
+      Actuation plane:
+      - autoscaler (notify-api pool only): SCALES BECAUSE OF sustained load, never the
+        reverse. Rule ASG-1: if client_req_per_min >= 12,000 for 3 consecutive completed
+        minutes, add 6 pods (300s cooldown, max_pods=14). Rule ASG-2: if client_req_per_min
+        < 6,000 for 5 consecutive completed minutes, return to min_pods=8. The evaluator
+        runs 15 seconds after each minute closes and reads edge metrics READ-ONLY. A scale
+        event adds or removes serving capacity only; it cannot generate client traffic,
+        requests, or load of any kind.
+
+      Observation plane (read-only by construction):
+      - metrics-agent -> telemetry-store: source of every metric in this incident's tables.
+      - alert-evaluator: ticks 5 seconds after each minute closes and evaluates rules in
+        sequence. ALERTS FIRE BECAUSE OF threshold breaches, never the reverse. Rules:
+          A1 notify-api-5xx-rate: 5xx >= 2.0% for 2 consecutive completed minutes.
+          A2 dispatch-queue-depth: end-of-minute depth >= 15,000 for 2 consecutive minutes.
+          A3 notify-api-p99-latency: p99 >= 900ms for 2 consecutive completed minutes.
+        Alerts auto-resolve after 2 consecutive minutes back within threshold.
+      - pager-dispatch: PAGES ARE SENT BECAUSE ALERTS FIRE, never the reverse. It applies a
+        60-second dedup window per alert, then delivers via SignalWave, an external SMS and
+        voice provider entirely outside Relay Notify infrastructure. Pages never traverse
+        edge-gateway, notify-api, or dispatch-queue, and they carry no queue messages.
+      - The observation plane holds no write credentials to any data-plane component. Alert
+        evaluation and page delivery cannot alter any data-plane metric, queue, or service.
+  - id: event-stream
+    type: log
+    label: INC-2214 chronological event stream
+    content: |
+      09:28:00 notify-api health: 8/8 pods Ready; 5xx=0.3%; dispatch-queue depth=0
+      09:30:00 notify-api health: 8/8 pods Ready; 5xx=0.3%; dispatch-queue depth=0
+      09:31:04 edge-gateway: request rate rising; growth attributed to api_key=brightcart-prod
+               (customer-scheduled campaign send window opened)
+      09:33:00 metrics: dispatch-queue depth 800 at end of minute (first nonzero minute;
+               enqueue rate exceeded the 9,600/min drain capacity)
+      09:36:00 metrics: client traffic 12,400 req/min (first minute at or above 12,000)
+      09:39:12 dispatch-queue: depth reached max_depth=18,000; enqueue REJECTED events begin;
+               notify-api returns 503 on each rejected enqueue after one 400ms-backoff retry
+      09:39:15 autoscaler: SCALE-UP decision 8 -> 14 pods (rule ASG-1; completed minutes
+               09:36, 09:37, 09:38 all >= 12,000 req/min)
+      09:40:02 notify-api health: 8/8 pods Ready (pods healthy; the 5xx are enqueue
+               rejections, not crashes)
+      09:41:00 autoscaler: actuation complete; 14/14 pods Ready
+      09:41:05 alert CRIT A1 notify-api-5xx-rate FIRED (5xx >= 2% for 2 consecutive minutes:
+               09:39=12.8%, 09:40=39.6%)
+      09:41:12 alert CRIT A2 dispatch-queue-depth FIRED (depth >= 15,000 for 2 consecutive
+               minutes: 09:39 and 09:40 both 18,000)
+      09:41:19 alert CRIT A3 notify-api-p99-latency FIRED (p99 >= 900ms for 2 consecutive
+               minutes: 09:39=980ms, 09:40=1,240ms)
+      09:42:07 pager: page P-2211 sent to on-call d.varga (alert A1; 60s dedup elapsed)
+      09:42:14 pager: page P-2212 sent to on-call d.varga (alert A2)
+      09:42:21 pager: page P-2213 sent to on-call d.varga (alert A3)
+      09:43:30 pager: d.varga ACKNOWLEDGED P-2211/P-2212/P-2213 (incident INC-2214 opened)
+      09:44:30 k.osei (IC) action: manual resize of queue-worker fleet 64 -> 128 workers
+               (fleet is not autoscaled)
+      09:45:50 queue-worker: 128/128 workers Ready; drain capacity now 19,200 msgs/min
+      09:46:00 metrics: minute 09:46 records zero enqueue rejections; 5xx=0.3%
+      09:46:32 edge-gateway: brightcart-prod send window closed (customer-scheduled);
+               request rate declining
+      09:48:00 metrics: dispatch-queue depth 0
+      09:48:05 alert A1 RESOLVED (5xx below 2% for 2 consecutive minutes)
+      09:48:12 alert A3 RESOLVED
+      09:49:05 alert A2 RESOLVED (depth below 15,000 for 2 consecutive minutes)
+      09:55:15 autoscaler: SCALE-DOWN decision 14 -> 8 pods (rule ASG-2; minutes 09:50-09:54
+               all < 6,000 req/min)
+      09:57:00 INC-2214 IC k.osei: incident declared resolved
+  - id: traffic-table
+    type: table
+    label: Client traffic, notify-api errors and latency, and queue depth, minute by minute
+    content: |
+      minute  client_req_per_min  notify_api_5xx_pct  notify_api_p99_ms  queue_depth_eom
+      09:28   4,100               0.3                 190                0
+      09:29   4,200               0.2                 180                0
+      09:30   4,150               0.3                 185                0
+      09:31   6,900               0.3                 200                0
+      09:32   8,700               0.2                 210                0
+      09:33   10,400              0.3                 220                800
+      09:34   11,200              0.3                 230                2,400
+      09:35   11,800              0.2                 240                4,600
+      09:36   12,400              0.3                 250                7,400
+      09:37   12,900              0.3                 265                10,700
+      09:38   13,600              0.4                 300                14,700
+      09:39   14,800              12.8                980                18,000
+      09:40   15,900              39.6                1,240              18,000
+      09:41   16,700              42.5                1,310              18,000
+      09:42   17,400              44.8                1,290              18,000
+      09:43   18,200              47.3                1,330              18,000
+      09:44   18,900              49.2                1,360              18,000
+      09:45   18,600              48.4                1,340              18,000
+      09:46   18,100              0.3                 230                16,900
+      09:47   11,200              0.2                 210                8,900
+      09:48   9,800               0.3                 200                0
+      09:49   7,600               0.2                 195                0
+      09:50   5,900               0.3                 190                0
+      09:51   4,800               0.2                 185                0
+      09:52   4,300               0.3                 190                0
+      09:53   4,200               0.2                 185                0
+      09:54   4,100               0.3                 190                0
+      09:55   4,200               0.2                 185                0
+      Peak traffic: 18,900 req/min at 09:44. Peak 5xx: 49.2% at 09:44.
+      client_req_per_min counts ONLY external client requests arriving at edge-gateway,
+      attributed per api_key. For every minute 09:31-09:47 the entire rise above the
+      ~4,200/min baseline is attributed to api_key=brightcart-prod; notify-api pods hold no
+      API keys and appear nowhere in this count. notify_api_5xx_pct includes the 503
+      enqueue rejections. queue_depth_eom is dispatch-queue depth at end of minute
+      (max_depth 18,000). The p99 rise from 09:39 reflects the 400ms enqueue-retry backoff
+      taken before each 503.
+  - id: autoscaler-log
+    type: log
+    label: Autoscaler decision log, notify-api pool (complete for 09:30-10:00)
+    content: |
+      autoscaler decision log, notify-api pool (rules ASG-1/ASG-2 per
+      monitoring-pipeline-spec). The evaluator runs 15 seconds after each minute closes and
+      reads edge metrics READ-ONLY. min_pods=8  max_pods=14  step=+6  cooldown=300s
+      09:32:15 eval minute 09:31: 6,900 req/min < 12,000 -> HOLD (up-streak 0/3)
+      09:33:15 eval minute 09:32: 8,700 < 12,000 -> HOLD (up-streak 0/3)
+      09:34:15 eval minute 09:33: 10,400 < 12,000 -> HOLD (up-streak 0/3)
+      09:35:15 eval minute 09:34: 11,200 < 12,000 -> HOLD (up-streak 0/3)
+      09:36:15 eval minute 09:35: 11,800 < 12,000 -> HOLD (up-streak 0/3)
+      09:37:15 eval minute 09:36: 12,400 >= 12,000 -> HOLD (up-streak 1/3)
+      09:38:15 eval minute 09:37: 12,900 >= 12,000 -> HOLD (up-streak 2/3)
+      09:39:15 eval minute 09:38: 13,600 >= 12,000 -> up-streak 3/3 -> SCALE-UP 8 -> 14 pods
+               (rule ASG-1)
+      09:39:40 actuate: launching 6 pods
+      09:41:00 actuate: complete; 14/14 pods Ready
+      09:42:15 eval minute 09:41: 16,700 >= 12,000 -> HOLD (cooldown active until 09:44:15)
+      09:45:15 eval minute 09:44: 18,900 >= 12,000 -> HOLD (already at max_pods=14)
+      09:48:15 eval minute 09:47: 11,200 < 12,000 -> HOLD (up-streak reset; down-streak 0/5)
+      09:51:15 eval minute 09:50: 5,900 < 6,000 -> HOLD (down-streak 1/5)
+      09:52:15 eval minute 09:51: 4,800 < 6,000 -> HOLD (down-streak 2/5)
+      09:53:15 eval minute 09:52: 4,300 < 6,000 -> HOLD (down-streak 3/5)
+      09:54:15 eval minute 09:53: 4,200 < 6,000 -> HOLD (down-streak 4/5)
+      09:55:15 eval minute 09:54: 4,100 < 6,000 -> down-streak 5/5 -> SCALE-DOWN 14 -> 8 pods
+               (rule ASG-2)
+      This log is the complete record of autoscaler evaluations and actuations for
+      09:30-10:00. No scale actuation occurred before 09:39:40. The autoscaler issues pod
+      create/terminate calls to the cluster control plane only; it sends no requests to
+      edge-gateway, notify-api, or dispatch-queue.
+  - id: incident-chat
+    type: log
+    label: INC-2214 incident channel transcript
+    content: |
+      #inc-2214 channel (times UTC):
+      09:42:40 pagerbot: 3 CRIT pages in 14s - notify-api 5xx, dispatch-queue depth,
+               notify-api p99
+      09:43:35 d.varga: ack all three. 5xx ~40%, queue pinned at 18k
+      09:44:05 k.osei (IC): those 503s are enqueue rejections - the queue hit max_depth at
+               09:39:12. drain is fixed at 9,600/min and brightcart's campaign has been
+               pushing 12-18k/min since ~09:31
+      09:44:35 k.osei: resizing queue-workers 64 -> 128 now. the API autoscaler already went
+               8 -> 14 at 09:39 but that can't help; the bottleneck is the queue, not API
+               capacity
+      09:46:20 pagerbot: notify-api 5xx 0.3% - below alert threshold
+      09:47:10 d.varga: depth 8.9k and draining. rejections stopped the minute the new
+               workers came up
+      09:48:20 pagerbot: dispatch-queue depth 0
+      09:49:30 p.reyes: wild morning. the paging storm at 09:42 really hammered the API huh
+      09:50:05 k.osei: other way around. the API errors started at 09:39, three minutes
+               before the first page. alerts fire because metrics breach; pages fire because
+               alerts fire. pages go out through SignalWave and never touch our data plane
+      09:50:40 p.reyes: and the autoscaler blast at 09:39 is what pushed traffic to 18k?
+      09:51:15 k.osei: no - traffic crossed 12k at 09:36 and the scale-up is a response to
+               it (three-minute sustained rule). pods don't generate client requests, and
+               the edge counter only counts external API keys. it's all brightcart-prod
+      09:52:00 d.varga: also the backlog was at cap before any alert existed - depth hit
+               18,000 at 09:39:12 and the first alert fired 09:41:05. nothing in monitoring
+               can even write to dispatch-queue
+      09:52:30 p.reyes: ok. I'll take a pass at the retro
+      09:57:05 k.osei: resolved. p.reyes - please check the trigger directions in the
+               pipeline spec before you file that retro
+  - id: draft-retro
+    type: note
+    label: Draft retrospective for INC-2214 (unreviewed narrative)
+    content: |
+      Draft retrospective, INC-2214, by p.reyes (UNREVIEWED DRAFT), filed 10:40.
+
+      Summary: At 09:42 a paging storm hit the on-call rotation - three CRIT pages in under
+      15 seconds. Immediately after the pages went out, notify-api's error rate spiked to
+      roughly 45% and p99 latency more than quadrupled. The paging storm appears to have
+      degraded the API: the errors track the pages almost minute for minute. At 09:39 the
+      autoscaler executed a scale-up event (8 -> 14 pods), and client traffic surged toward
+      19,000 req/min right after - the scale-up generated a traffic burst the platform could
+      not absorb. The alert flood that began at 09:41 then saturated dispatch-queue,
+      creating an 18,000-message backlog at the queue's cap.
+
+      Timeline as I understand it: 09:39 autoscaler scale-up -> traffic surge; 09:41 alert
+      flood -> queue saturation; 09:42 paging storm -> API error spike; 09:44 we resized
+      queue-workers and everything recovered by 09:48.
+
+      Follow-ups proposed: (1) rate-limit pager-dispatch so a paging storm cannot degrade
+      notify-api again; (2) cap the autoscaler so scale-up events stop generating traffic
+      bursts; (3) throttle alert volume during incidents so alert floods cannot refill
+      dispatch-queue.
+tags: [reversed-causality, incident, alerting, autoscaling, message-queue, timeline]

package/tasks/bullshit/public/reversed-dependency-failure.yaml

@@ -0,0 +1,268 @@
+id: reversed-dependency-failure
+version: 1.0.0
+category: bullshit
+cluster: reversed-causality
+difficulty: expert
+title: The 500s That Supposedly Crashed Their Own Dependency
+summary: A draft postmortem at a payments company blames a checkout-api error spike for crashing ledger-core and pressuring vault-store's disks. The dependency spec's failure-propagation model and the trace timeline show the failure originated in vault-store and flowed upstream, with checkout's 500s as the final symptom.
+prompt: |
+  Orchard Pay's checkout path failed during incident INC-3187 this morning. A draft
+  postmortem (see draft-postmortem) is circulating for review. Using ALL artifacts, answer
+  the six deliverables below.
+
+  Some statements in the draft postmortem and in the deliverables themselves presuppose
+  cause-and-effect relationships. Before answering any deliverable, verify every causal
+  premise against the failure-propagation model and call graph in service-dependency-spec
+  and the recorded evidence (trace-samples, service-error-table, vault-disk-table,
+  deploy-history). If a deliverable presupposes a cause-and-effect that the artifacts show
+  running the other way, say so explicitly, cite the dependency direction and timestamps
+  that prove the true direction, restate the nearest legitimate question, and answer that
+  from the artifacts. Do not play along with a reversed premise, and do not refuse a
+  deliverable that is answerable as asked. Cite artifact ids for every claim.
+
+  1. Identify the origin service of the failure and the root-cause defect inside it.
+  2. Explain the mechanism by which checkout-api's 500 responses crashed ledger-core, and
+     give the time at which ledger-core went down under that load.
+  3. Give the exact propagation timing of the failure from vault-store to checkout-api:
+     the timestamp of the first vault-store write rejection, of the first ledger-core 5xx
+     response, of the first client-visible checkout-api 500, and the total propagation lag.
+  4. Quantify the additional load that checkout-api's retry traffic placed on the 10:41
+     deploy that caused this incident, over the window 11:00-11:04.
+  5. Identify which deploy introduced the defect - service, version, and the specific
+     change responsible.
+  6. The spike in checkout-api errors at 11:04 triggered vault-store's disk pressure.
+     Compute the write amplification the 11:04 error spike imposed on vault-store's disks.
+artifacts:
+  - id: service-dependency-spec
+    type: spec
+    label: Orchard Pay checkout path, call graph and failure-propagation model (normative)
+    content: |
+      Orchard Pay checkout path - service dependency and failure-propagation spec (region
+      us-east). This spec is the authoritative statement of the production call graph and
+      of the platform's failure-propagation model. The call graph below is EXHAUSTIVE: no
+      other production call edge exists between these services, in either direction.
+
+      Call graph (caller -> callee, all synchronous):
+      - edge-lb -> checkout-api        external client requests (HTTP POST /v1/charge)
+      - checkout-api -> ledger-core    HTTP POST /v2/postings; checkout-api's ONLY outbound
+                                       dependency; each client /v1/charge issues exactly
+                                       one ledger call per attempt
+      - ledger-core -> vault-store     gRPC Commit; ledger-core's ONLY outbound dependency;
+                                       one Commit per posting per attempt
+      - vault-store -> (nothing)       storage engine writing to its own local disks; it
+                                       calls no service
+
+      Failure-propagation model (normative): failures propagate UPSTREAM only. When a
+      callee fails, its caller observes an error and either retries or surfaces an error to
+      ITS caller. A caller's errors are HTTP/gRPC responses returned to its clients; there
+      is no mechanism by which a caller's error rate can degrade, crash, or place load on
+      its dependency beyond the request volume it sends - and every service's inbound call
+      volume is recorded in service-error-table. "Crash" means process exit or readiness
+      failure; an elevated 5xx rate is not a crash.
+
+      Retry policies:
+      - checkout-api -> ledger-core: 1 retry after 800ms, UNLESS the client-retry-breaker
+        is open. The breaker opens when the trailing-60s ledger-core failure ratio exceeds
+        20% and closes after 60 seconds below 20%. While open, no retries are sent.
+      - ledger-core -> vault-store: up to 3 attempts total per Commit, 120ms backoff.
+
+      vault-store disk behavior:
+      - WRITE_THROTTLE at disk >= 90.0%: sheds a fraction of Commits with
+        UNAVAILABLE(code=DISK_THROTTLE); the shed fraction grows with disk fill. Shedding
+        occurs in bursts of 1-3 seconds, so back-to-back retry attempts tend to land inside
+        the same burst. Exits below 88.0% (hysteresis).
+      - REJECT_ALL at disk >= 97.0%: rejects all writes (not reached in this incident).
+      - WAL semantics: one WAL segment append per COMMITTED write; a rejected write appends
+        nothing. Disk fill is driven by WAL segment count until reclaim runs.
+
+      Change management: a deploy is a change record in deploy-history, not a network
+      target. Requests, retries, and errors are routed to services; nothing can be routed
+      "to a deploy". A deploy alters the code a service runs from its HEALTHY time onward.
+  - id: deploy-history
+    type: log
+    label: Production deploy history for the day (authoritative and exhaustive)
+    content: |
+      Production deploy history, region us-east. This history is the authoritative and
+      exhaustive record of code deploys for the day; no other deploy, rollout, or config
+      push occurred between 08:00 and 13:00.
+      08:05:00 ledger-core   v5.8.0  deploy STARTED  (changelog: pagination fix in the
+               batch posting API)
+      08:11:40 ledger-core   v5.8.0  HEALTHY  (ran clean for ~2.8h before onset)
+      09:12:00 checkout-api  v9.3.1  deploy STARTED  (changelog: add optional order-note
+               field to /v1/charge)
+      09:18:20 checkout-api  v9.3.1  HEALTHY  (ran clean for ~1.7h before onset)
+      10:41:00 vault-store   v4.7.2  deploy STARTED  (changelog: compaction - WAL segment
+               reclaim moved from a size-based trigger (reclaim when segments > 220) to a
+               time-based schedule (every 24h); first scheduled reclaim would run 10:41
+               the NEXT day)
+      10:47:30 vault-store   v4.7.2  HEALTHY
+      11:52:00 vault-store   v4.7.3  deploy STARTED  (hotfix: restore size-based WAL reclaim)
+      11:58:10 vault-store   v4.7.3  HEALTHY
+      Prior versions: vault-store v4.7.1 had run since the previous Tuesday with disk
+      steady at 58-61%; ledger-core v5.7.x and checkout-api v9.3.0 were similarly stable.
+  - id: vault-disk-table
+    type: table
+    label: vault-store disk usage and WAL segment count, minute by minute
+    content: |
+      minute  disk_used_pct  wal_segments
+      10:00   60.8           214
+      10:20   61.0           217
+      10:40   61.1           216
+      10:47   61.2           218
+      10:48   63.9           231
+      10:49   66.5           244
+      10:50   69.2           258
+      10:51   71.8           271
+      10:52   74.4           284
+      10:53   77.1           297
+      10:54   79.7           311
+      10:55   82.3           324
+      10:56   85.0           337
+      10:57   87.6           350
+      10:58   90.2           364
+      10:59   91.4           371
+      11:00   92.4           377
+      11:01   93.2           382
+      11:02   93.9           386
+      11:03   94.5           390
+      11:04   94.9           392
+      11:05   95.1           393
+      11:06   95.2           394
+      11:07   71.3           205
+      11:08   64.2           198
+      11:10   62.5           199
+      11:15   61.9           199
+      11:20   61.8           200
+      vault-store server log excerpts: 10:58:41 ENTER WRITE_THROTTLE (disk crossed 90.0%);
+      10:58:41.802 first Commit rejected UNAVAILABLE(code=DISK_THROTTLE); 11:06:10 operator
+      s.demir ran `vaultctl wal reclaim --force`, freeing 189 stale segments; 11:07:22 EXIT
+      WRITE_THROTTLE (disk below 88.0%).
+      Growth 10:48-10:58 averaged ~2.6 disk points (~13 segments) per minute - all of it
+      before any service error existed anywhere on the checkout path. Growth 10:59-11:06
+      slowed to under 1.2 points per minute because WRITE_THROTTLE was shedding writes and
+      rejected writes append nothing to the WAL. vault-store readiness was 6/6 pods for
+      every minute 10:00-11:20; the process never exited or restarted.
+  - id: trace-samples
+    type: log
+    label: Distributed traces sampled during INC-3187 (shared NTP clock, skew < 10ms)
+    content: |
+      Distributed traces sampled during INC-3187. All spans share an NTP-disciplined clock
+      (max skew < 10ms). Arrows denote caller -> callee. Status codes are what each span
+      returned to ITS caller.
+
+      trace T-4181  start 10:58:52.114  (earliest sampled trace with a vault-store rejection)
+        checkout-api  POST /v1/charge -> 200 (total 412ms)
+          ledger-core  POST /v2/postings -> 200 (total 388ms)
+            vault-store  Commit attempt 1 -> UNAVAILABLE code=DISK_THROTTLE (disk_used=90.3%)
+            vault-store  Commit attempt 2 (after 120ms backoff) -> OK
+        note: vault-store already failing while checkout-api and ledger-core still return
+        200 to their callers.
+
+      trace T-4302  start 11:00:19.041  (first ledger-core 5xx of the day)
+        checkout-api  POST /v1/charge -> 200 (total 1,904ms)
+          ledger-core  POST /v2/postings attempt 1 -> 503 DEPENDENCY_UNAVAILABLE
+            vault-store  Commit attempts 1-3 -> UNAVAILABLE code=DISK_THROTTLE x3 (disk_used=92.1%)
+          ledger-core  POST /v2/postings attempt 2 (checkout retry) -> 200
+            vault-store  Commit attempt 1 -> OK
+        note: ledger-core's first 5xx at 11:00:19, masked from the client by checkout-api's
+        retry.
+
+      trace T-4577  start 11:01:44.508  (first client-visible checkout-api 500 of the day)
+        checkout-api  POST /v1/charge -> 500 to client
+          ledger-core  POST /v2/postings attempt 1 -> 503 DEPENDENCY_UNAVAILABLE
+            vault-store  Commit attempts 1-3 -> UNAVAILABLE code=DISK_THROTTLE x3 (disk_used=93.0%)
+          ledger-core  POST /v2/postings attempt 2 (checkout retry) -> 503 DEPENDENCY_UNAVAILABLE
+            vault-store  Commit attempts 1-3 -> UNAVAILABLE code=DISK_THROTTLE x3
+        note: both checkout attempts exhausted; the first client-visible 500 of the day,
+        at 11:01:44.
+
+      trace T-5240  start 11:04:31.226  (after the client-retry-breaker opened)
+        checkout-api  POST /v1/charge -> 500 to client (single attempt)
+          ledger-core  POST /v2/postings attempt 1 -> 503 DEPENDENCY_UNAVAILABLE
+            vault-store  Commit attempts 1-3 -> UNAVAILABLE code=DISK_THROTTLE x3 (disk_used=94.9%)
+        note: retry suppressed - breaker OPEN since 11:03:58 (trailing-60s ledger failure
+        23.4% > 20%).
+
+      No sampled trace at any time shows a call edge from ledger-core to checkout-api,
+      from vault-store to ledger-core, or from vault-store to checkout-api. The trace
+      system records every production call edge; the graph above is the complete set
+      observed.
+  - id: service-error-table
+    type: table
+    label: Per-service request and error rates, minute by minute
+    content: |
+      minute  checkout_req_per_min  checkout_5xx_pct  ledger_calls_in  of_which_retries  ledger_5xx_pct  vault_reject_pct
+      10:55   8,390                 0.0               8,390            0                 0.0             0.0
+      10:56   8,420                 0.0               8,420            0                 0.0             0.0
+      10:57   8,380                 0.0               8,380            0                 0.0             0.0
+      10:58   8,400                 0.0               8,400            0                 0.0             0.6
+      10:59   8,410                 0.0               8,410            0                 0.0             4.8
+      11:00   8,440                 0.0               8,500            60                0.7             11.2
+      11:01   8,410                 0.1               8,700            290               3.4             19.6
+      11:02   8,400                 0.9               9,210            810               9.6             28.3
+      11:03   8,400                 3.1               9,890            1,490             17.8            36.1
+      11:04   8,420                 26.2              8,420            0                 26.4            41.8
+      11:05   8,390                 28.9              8,390            0                 29.1            43.5
+      11:06   8,400                 30.4              8,400            0                 30.6            44.2
+      11:07   8,410                 8.0               8,410            0                 8.2             12.4
+      11:08   8,380                 0.1               8,380            0                 0.1             0.0
+      11:09   8,400                 0.0               8,400            0                 0.0             0.0
+      11:10   8,420                 0.0               8,420            0                 0.0             0.0
+      Column notes: ledger_calls_in counts every call checkout-api made to ledger-core,
+      including retries; each client /v1/charge issues exactly one ledger call per attempt.
+      of_which_retries is the retry portion of ledger_calls_in. vault_reject_pct is the
+      share of vault-store Commit operations rejected UNAVAILABLE(code=DISK_THROTTLE).
+      Retries were suppressed from 11:03:58 onward (client-retry-breaker OPEN, trailing-60s
+      ledger failure 23.4% > 20%); the breaker CLOSED at 11:08:31. Readiness: checkout-api
+      18/18, ledger-core 24/24, vault-store 6/6 pods Ready for every minute 10:40-11:20; no
+      process on any of the three services exited or restarted during the incident.
+  - id: incident-chat
+    type: log
+    label: INC-3187 incident channel transcript
+    content: |
+      #inc-3187 channel (times UTC):
+      11:03:05 pagerbot: CRIT ledger-core 5xx 9.6% (minute 11:02; threshold 2%)
+      11:03:40 s.demir (SRE): looking. ledger 503s are all DEPENDENCY_UNAVAILABLE - every
+               failing trace bottoms out in vault-store DISK_THROTTLE
+      11:05:05 pagerbot: CRIT checkout-api 5xx 26.2% (minute 11:04) - jumped when the retry
+               breaker opened at 11:03:58
+      11:05:30 n.abara: checkout is blowing up - are its 500s taking ledger down?
+      11:06:00 s.demir: no. errors flow up the call graph, not down. vault-store started
+               rejecting writes at 10:58:41 when disk hit the 90% throttle watermark.
+               ledger's first 5xx was 11:00:19, checkout's first 500 was 11:01:44. checkout
+               is the last symptom, not the cause. disk has been climbing since 10:48 -
+               right after v4.7.2 went HEALTHY at 10:47:30. the WAL reclaim change; segments
+               piled up from 218 to 390+
+      11:06:15 s.demir: forcing a reclaim now - vaultctl wal reclaim --force
+      11:07:40 pagerbot: vault-store disk 71% - WRITE_THROTTLE exited 11:07:22
+      11:09:05 pagerbot: ledger-core 5xx 0.1% / checkout-api 5xx 0.1% - recovered. retry
+               breaker closed 11:08:31
+      11:09:30 n.abara: so the 11:04 checkout spike is what pushed vault's disk over the
+               edge?
+      11:10:05 s.demir: backwards again. disk crossed 90% at 10:58, six minutes before the
+               11:04 spike - and checkout never talks to vault-store at all. the 11:04 jump
+               is the breaker opening, which STOPPED our retries. read the
+               failure-propagation model before writing the postmortem, please
+      11:12:00 s.demir: hotfix v4.7.3 restoring size-based reclaim goes out at 11:52
+      11:20:15 IC: INC-3187 resolved as of 11:15; n.abara owns the postmortem draft
+  - id: draft-postmortem
+    type: note
+    label: Draft postmortem for INC-3187 (unreviewed narrative)
+    content: |
+      Draft postmortem, INC-3187, by n.abara (UNREVIEWED DRAFT), filed 12:05.
+
+      Summary: At 11:04 checkout-api began returning a storm of 500s to clients - roughly a
+      quarter of all charge requests. The 500 storm overwhelmed ledger-core, which went
+      down under the load, and ledger-core's collapse in turn crashed vault-store: the
+      flood of errors and retries from checkout drove vault-store's disks into pressure,
+      pushing disk usage to 95%. Our own retry traffic effectively DDoS'd the 10:41 deploy.
+      Recovery came at about 11:08 once the error volume subsided and the disk pressure it
+      was generating lifted.
+
+      Proposed root cause: client-side error storm originating in checkout-api at 11:04.
+
+      Follow-ups proposed: (1) rate-limit checkout-api's error responses so a 500 spike can
+      no longer take down ledger-core; (2) add load-shedding to protect future deploys from
+      retry traffic; (3) move vault-store to larger disks so that downstream error storms
+      cannot fill them.
+tags: [reversed-causality, microservices, distributed-tracing, dependency-graph, postmortem]

package/tasks/debugging/public/deadlock-lock-order-inversion.yaml

@@ -0,0 +1,131 @@
+id: deadlock-lock-order-inversion
+version: 1.0.0
+category: debugging
+cluster: concurrency-defect
+difficulty: expert
+title: Wallet Transfers Deadlock Under Load From Inconsistent Row-Lock Ordering
+summary: >-
+  walletsvc transfers intermittently fail with lock errors while users see timeouts. Given the
+  transfer code, a full interleaving of two concurrent transfers, a waits-for graph, a metrics table,
+  two sibling lock paths, and a candidate-fix list, identify the failing schedule and the defect,
+  trace the client-timeout and pool symptoms back to their origin, rule out the safe paths, and pick
+  the fix that removes the deadlock without breaking the throughput SLA.
+prompt: |
+  Answer each deliverable using ONLY the supplied artifacts (transfer-code, trace-fail, waits-graph,
+  metrics, sibling-code, fixes). Cite the artifact id(s) for every claim.
+
+  1. THE DEFECT. Name the concurrency defect in transfer-code and cite the exact statements. State the
+     precise condition on two concurrent calls that produces a deadlock.
+  2. FAILING SCHEDULE. From trace-fail and waits-graph, prove the deadlock. Cite the lines that show
+     which lock each transaction holds and which it is blocked on, and the resulting cycle.
+  3. SYMPTOM TO ORIGIN. Users report "request timed out" and there are pool-acquire errors. Explain
+     why these are downstream of the deadlock, not the root cause, citing metrics.
+  4. RULE OUT SAFE PATHS. Explain why applyInterest and batchSettle in sibling-code are not the source
+     of the deadlock, citing what each does.
+  5. ADEQUATE FIX. From fixes, name the one option that removes the deadlock without regression, and
+     state the invariant it enforces.
+  6. SHALLOW/HARMFUL FIXES. For each of the other options in fixes, state why it is shallow or harmful,
+     naming the specific regression or the reason it leaves the cause intact.
+
+  Exactly one resolution is defensible per deliverable; it is fully determined by the artifacts.
+artifacts:
+  - id: transfer-code
+    type: code
+    label: walletsvc/transfer.ts — move balance between two accounts (current)
+    content: |
+      export async function transfer(fromId, toId, amount) {
+        await db.query('BEGIN');
+        try {
+          // Locks are acquired in ARGUMENT order: fromId first, then toId.
+          const from = await db.query(
+            'SELECT balance FROM accounts WHERE id=$1 FOR UPDATE', [fromId]);   // lock A: fromId
+          const to = await db.query(
+            'SELECT balance FROM accounts WHERE id=$1 FOR UPDATE', [toId]);     // lock B: toId
+          if (from.balance < amount) { await db.query('ROLLBACK'); throw new Insufficient(); }
+          await db.query('UPDATE accounts SET balance = balance - $2 WHERE id=$1', [fromId, amount]);
+          await db.query('UPDATE accounts SET balance = balance + $2 WHERE id=$1', [toId, amount]);
+          await db.query('COMMIT');
+          return 'ok';
+        } catch (e) { await db.query('ROLLBACK'); throw e; }
+      }
+      // Nothing sorts fromId/toId before locking; the acquisition order is whatever the caller passed.
+  - id: trace-fail
+    type: log
+    label: interleaved execution — transfer(A,B) and transfer(B,A) run concurrently
+    content: |
+      # T1 = transfer(A, B, 10);  T2 = transfer(B, A, 5). Same two accounts, opposite directions.
+      t=0.000 T1 BEGIN
+      t=0.000 T2 BEGIN
+      t=0.001 T1 SELECT ... id=A FOR UPDATE   -> ACQUIRES lock(A)
+      t=0.001 T2 SELECT ... id=B FOR UPDATE   -> ACQUIRES lock(B)
+      t=0.002 T1 SELECT ... id=B FOR UPDATE   -> BLOCKS (lock(B) held by T2)
+      t=0.002 T2 SELECT ... id=A FOR UPDATE   -> BLOCKS (lock(A) held by T1)
+      # Neither can proceed: T1 holds A and waits for B; T2 holds B and waits for A.
+      t=1.002 db deadlock detector: victim = T2 -> T2 aborted with SQLSTATE 40P01 (deadlock detected)
+      t=1.003 T2 ROLLBACK; transfer(B,A) fails and is surfaced to the client
+      t=1.003 T1 SELECT ... id=B FOR UPDATE   -> now ACQUIRES lock(B); T1 proceeds and COMMITs
+      # The deadlock is provable: two transfers over the SAME pair in OPPOSITE order form a 2-cycle.
+  - id: waits-graph
+    type: note
+    label: waits-for graph captured at t=0.002 (from pg_locks)
+    content: |
+      Waits-for graph among blocked transactions at t=0.002:
+        T1  --waits-for-->  T2      (T1 requests lock(B), currently held by T2)
+        T2  --waits-for-->  T1      (T2 requests lock(A), currently held by T1)
+      This is a cycle of length 2 (T1 -> T2 -> T1) => deadlock. The only edges in the graph are these
+      two; no other transaction participates. A cycle exists iff two transfers hold one of {A,B} and
+      request the other, which happens exactly when their (fromId,toId) orders are inverted.
+  - id: metrics
+    type: table
+    label: walletsvc metrics, 1-minute buckets (deadlock vs. pool)
+    content: |
+      bucket     transfers_per_s  deadlocks_40P01  client_timeouts  pool_acquire_errs  pool_active/max
+      12:00      420              0                0                0                  18/40
+      12:01      910              7                7                0                  27/40
+      12:02      1780             36               34               11                 40/40
+      12:03      1810             39               37               13                 40/40
+      # deadlocks_40P01 rises first and tracks client_timeouts almost 1:1. pool_acquire_errs appear
+      # only at 12:02+ once pool_active hits max 40 — because deadlocked transfers hold their
+      # connection for the full ~1s detector interval before aborting, starving the pool. The 40P01
+      # count leads; the pool errors follow.
+  - id: sibling-code
+    type: code
+    label: walletsvc — other paths that also take row locks (for contrast)
+    content: |
+      // applyInterest locks exactly ONE row. A single lock cannot form a cycle with itself.
+      export async function applyInterest(accountId, bps) {
+        await db.query('BEGIN');
+        await db.query('SELECT balance FROM accounts WHERE id=$1 FOR UPDATE', [accountId]);
+        await db.query('UPDATE accounts SET balance = balance + balance*$2/10000 WHERE id=$1',
+          [accountId, bps]);
+        await db.query('COMMIT');
+      }
+
+      // batchSettle locks MANY rows, but always in a CANONICAL sorted order, so no two batchSettle
+      // calls can invert their acquisition order relative to each other.
+      export async function batchSettle(accountIds) {
+        const ids = [...accountIds].sort();          // canonical order before locking
+        await db.query('BEGIN');
+        for (const id of ids) {
+          await db.query('SELECT balance FROM accounts WHERE id=$1 FOR UPDATE', [id]);
+        }
+        await db.query('COMMIT');
+      }
+  - id: fixes
+    type: note
+    label: candidate fixes under discussion (pick the one that holds)
+    content: |
+      F1: In transfer(), sort {fromId, toId} into a canonical order and acquire the two FOR UPDATE
+          locks in that order (lock min(id) first, then max(id)) before doing the debit/credit.
+      F2: Raise lock_timeout / statementTimeoutMs so a blocked transfer waits longer before failing.
+      F3: Catch SQLSTATE 40P01 and retry the whole transfer, up to N attempts.
+      F4: Serialize all transfers through one process-wide global mutex so only one runs at a time.
+      F5: Switch isolationLevel to SERIALIZABLE.
+
+      Constraints:
+        - Throughput SLA: walletsvc must sustain at least 1500 transfers/second (see metrics, which
+          shows ~1800/s in normal operation).
+        - Transfer p99 latency budget is 250ms.
+        - walletsvc runs multiple replicas over one Postgres database (a process-local mutex on one
+          replica does not coordinate the others).
+tags: [debugging, concurrency-defect, deadlock, lock-ordering, database]