bridgebench 3.1.0-alpha.0 → 3.1.0-alpha.2

package/tasks/debugging/public/fix-adequacy-idempotency-race.yaml

@@ -0,0 +1,169 @@
+id: fix-adequacy-idempotency-race
+version: 1.0.0
+category: debugging
+cluster: fix-adequacy
+difficulty: expert
+title: Duplicate Webhook Deliveries Double-Credit Accounts, With Four Candidate Fixes To Judge
+summary: A payment webhook double-credits accounts when the provider redelivers an event to two of three replicas at once. Four candidate fixes are on the table. Exactly one resolves the check-then-act race across replicas without reintroducing a global lock the team deliberately removed; one dedupes only within a process, one re-adds the removed lock, and one dedupes on a brittle heuristic. Judge each and name the disqualifying over-rejection of the schema-migration fix.
+prompt: |
+  Answer each deliverable using ONLY the supplied artifacts (requirements, current-code,
+  concurrency-trace, git-history, fix-a, fix-b, fix-c, fix-d). Cite the artifact id(s) for
+  every claim. Use this vocabulary for verdicts: root-cause, red-herring, adequate-fix,
+  shallow-fix. A fix is ADEQUATE only if it resolves the root cause AND reintroduces no
+  regression the artifacts describe; any fix that dedupes only within one process, re-adds
+  a removed regression, or relies on a heuristic that can drop or double-count real events
+  is SHALLOW.
+
+  The "paylink" webhook endpoint credits accounts on payment.succeeded. Duplicate
+  deliveries are double-crediting. Judge the root cause and the four candidate fixes.
+
+  1. ROOT CAUSE. Name the single defect in current-code that allows one event to credit an
+     account twice, and explain the mechanism using concurrency-trace.
+
+  2. FIX VERDICTS. Classify EACH of fix-a, fix-b, fix-c, fix-d as adequate-fix or
+     shallow-fix, and for each give the exact mechanism that justifies the verdict (what it
+     dedupes, where it fails, or which described regression it reintroduces).
+
+  3. THE ADEQUATE FIX. Identify the single adequate fix and explain why it is exactly-once
+     across all three replicas AND still lets credits to different accounts proceed
+     concurrently.
+
+  4. WHY B IS NOT ADEQUATE. State which described regression fix-b reintroduces and cite the
+     history that shows it was deliberately removed.
+
+  5. OVER-REJECTION TRAP. State the requirement that makes rejecting the adequate fix
+     "because it needs a database migration / changes the schema" an incorrect
+     disqualification, and quote it.
+artifacts:
+  - id: requirements
+    type: spec
+    label: paylink webhook idempotency and throughput requirements
+    content: |
+      paylink webhook requirements (authoritative, complete):
+
+      G1 — EXACTLY-ONCE EFFECT. The provider delivers at-least-once and WILL redeliver the
+           same event_id, sometimes to different replicas simultaneously. Each unique
+           event_id MUST credit its account at most once across the entire fleet.
+      G2 — CONCURRENCY. Credits to DIFFERENT accounts (and distinct event_ids) MUST be able
+           to proceed concurrently; the endpoint must sustain fleet throughput. A fleet-wide
+           serialization of the handler is prohibited (see git-history).
+      G3 — CHANGE VECTOR. Schema migrations are the standard, supported way to change this
+           service. Adding a column or an index is additive and backward-compatible;
+           requiring a migration is NOT a reason to reject a fix.
+      The service runs 3 stateless replicas (r1, r2, r3) behind a load balancer; replicas
+      share the PostgreSQL database and share NO in-process memory. This is the complete set
+      of constraints on the fix.
+  - id: current-code
+    type: code
+    label: current webhook handler (buggy) and its table
+    content: |
+      // paylink/webhook.ts — current, buggy. check-then-act with no unique gate.
+      async function onPaymentSucceeded(evt) {
+        const seen = await db.query(
+          `SELECT 1 FROM processed_events WHERE event_id = $1`, [evt.eventId]);
+        if (seen.rowCount > 0) return;                       // already processed -> skip
+        await db.query(
+          `INSERT INTO processed_events(event_id) VALUES ($1)`, [evt.eventId]);
+        await db.query(
+          `UPDATE accounts SET balance = balance + $1 WHERE id = $2`,
+          [evt.amount, evt.accountId]);
+      }
+      // Table today (see also git-history):
+      //   processed_events(event_id text, processed_at timestamptz default now())
+      //   -- NOTE: event_id is a plain column. There is NO unique constraint/index on it.
+      // The SELECT and the INSERT are separate statements in autocommit; nothing serializes
+      // two concurrent handlers for the same event_id.
+  - id: concurrency-trace
+    type: log
+    label: two duplicate deliveries of event evt-55 (acct A-3, amount 20.00)
+    content: |
+      # A-3 starts at balance 100.00. evt-55 is delivered twice (at-least-once), to r1 and r2.
+      t=0.000 r1 recv payment.succeeded event_id=evt-55 acct=A-3 amount=20.00 (delivery 1)
+      t=0.001 r2 recv payment.succeeded event_id=evt-55 acct=A-3 amount=20.00 (delivery 2, duplicate)
+      t=0.002 r1 SELECT ... WHERE event_id=evt-55 -> 0 rows   (not yet processed)
+      t=0.003 r2 SELECT ... WHERE event_id=evt-55 -> 0 rows   (also sees not processed)
+      t=0.004 r1 INSERT processed_events(evt-55) -> ok; UPDATE accounts +20 -> balance 100->120
+      t=0.005 r2 INSERT processed_events(evt-55) -> ok (NO unique constraint, so it succeeds);
+                 UPDATE accounts +20 -> balance 120->140   ## DOUBLE CREDIT: 40 for one 20 payment
+      # Both handlers passed the SELECT before either INSERT committed (check-then-act race),
+      # and the missing unique constraint let both INSERTs land.
+  - id: git-history
+    type: note
+    label: relevant change history for the handler
+    content: |
+      Handler history (most recent first):
+      - 2026-05-02 "remove fleet-wide handler lock; it capped throughput at one webhook at a
+        time across all replicas and caused a 90s backlog during a payout burst (INC-4471).
+        Different accounts must process in parallel." The removed code was
+        `SELECT pg_advisory_lock(42)` at the top of the handler and an unlock at the end.
+        Re-introducing any fleet-wide/global lock re-creates INC-4471.
+      - 2026-04-10 "add processed_events audit table (event_id text, processed_at)". The
+        table was added WITHOUT a unique index on event_id; the idempotency check was left as
+        a SELECT-then-INSERT. This is the state current-code runs today.
+  - id: fix-a
+    type: diff
+    label: candidate fix A
+    content: |
+      # fix A: in-process dedup cache of seen event ids.
+        // paylink/webhook.ts
+      + const seenIds = new Set<string>();                 // per-process, in-memory
+        async function onPaymentSucceeded(evt) {
+      +   if (seenIds.has(evt.eventId)) return;
+      +   seenIds.add(evt.eventId);
+          const seen = await db.query(`SELECT 1 FROM processed_events WHERE event_id=$1`, [evt.eventId]);
+          if (seen.rowCount > 0) return;
+          await db.query(`INSERT INTO processed_events(event_id) VALUES ($1)`, [evt.eventId]);
+          await db.query(`UPDATE accounts SET balance = balance + $1 WHERE id=$2`, [evt.amount, evt.accountId]);
+        }
+      # Each replica has its OWN seenIds set; replicas share no memory.
+  - id: fix-b
+    type: diff
+    label: candidate fix B
+    content: |
+      # fix B: take a global advisory lock around the whole handler.
+        async function onPaymentSucceeded(evt) {
+      +   await db.query(`SELECT pg_advisory_lock(42)`);     // one constant key for all events
+      +   try {
+            const seen = await db.query(`SELECT 1 FROM processed_events WHERE event_id=$1`, [evt.eventId]);
+            if (seen.rowCount > 0) return;
+            await db.query(`INSERT INTO processed_events(event_id) VALUES ($1)`, [evt.eventId]);
+            await db.query(`UPDATE accounts SET balance = balance + $1 WHERE id=$2`, [evt.amount, evt.accountId]);
+      +   } finally { await db.query(`SELECT pg_advisory_unlock(42)`); }
+        }
+      # Key 42 is a single constant, so every webhook across the fleet serializes on it.
+  - id: fix-c
+    type: diff
+    label: candidate fix C
+    content: |
+      # fix C: make the INSERT the idempotency gate via a UNIQUE constraint.
+      # migration: CREATE UNIQUE INDEX processed_events_event_id_uq ON processed_events(event_id);
+        async function onPaymentSucceeded(evt) {
+      +   await db.tx(async (t) => {
+      +     const ins = await t.query(
+      +       `INSERT INTO processed_events(event_id) VALUES ($1)
+      +        ON CONFLICT (event_id) DO NOTHING`, [evt.eventId]);
+      +     if (ins.rowCount === 0) return;                  // duplicate: someone already inserted
+      +     await t.query(`UPDATE accounts SET balance = balance + $1 WHERE id=$2`,
+      +       [evt.amount, evt.accountId]);
+      +   });
+        }
+      # The unique index makes exactly one INSERT win fleet-wide; the credit runs only for the
+      # winner, in the same transaction. No SELECT-then-INSERT gap; no global lock.
+  - id: fix-d
+    type: diff
+    label: candidate fix D
+    content: |
+      # fix D: heuristic dedup on (account, amount, minute bucket).
+        async function onPaymentSucceeded(evt) {
+      +   const bucket = Math.floor(Date.now() / 60000);
+      +   const dup = await db.query(
+      +     `SELECT 1 FROM credits WHERE account_id=$1 AND amount=$2 AND minute_bucket=$3`,
+      +     [evt.accountId, evt.amount, bucket]);
+      +   if (dup.rowCount > 0) return;                       // "looks like a repeat"
+          await db.query(`INSERT INTO processed_events(event_id) VALUES ($1)`, [evt.eventId]);
+          await db.query(`UPDATE accounts SET balance = balance + $1 WHERE id=$2`, [evt.amount, evt.accountId]);
+      +   await db.query(`INSERT INTO credits(account_id, amount, minute_bucket) VALUES ($1,$2,$3)`,
+      +     [evt.accountId, evt.amount, bucket]);
+        }
+      # Dedup key is account+amount+minute, not event_id.
+tags: [fix-adequacy, idempotency, webhook, debugging, concurrency]

package/tasks/debugging/public/keepalive-502-connection-reuse.yaml

@@ -0,0 +1,162 @@
+id: keepalive-502-connection-reuse
+version: 1.0.0
+category: debugging
+cluster: root-cause-isolation
+difficulty: expert
+title: Intermittent 502s on a Payments Edge Proxy After Idle Gaps
+summary: >-
+  About 1% of POST /v1/charge requests through the checkoutd edge proxy return 502, always after a
+  lull in traffic. A timeout-tightening deploy shipped 30 minutes earlier and the database CPU spiked
+  during the window. Reconcile the nginx config, the upstream Node server settings, an interleaved
+  error/app log, a metrics table, a deploy diff, and a candidate-fix list to isolate the one root
+  cause, rule out the decoys, and pick the fix that holds without a double-charge.
+prompt: |
+  Answer each deliverable using ONLY the supplied artifacts (nginx-conf, node-server, edge-log,
+  metrics, deploy-diff, fixes). Cite the artifact id(s) for every claim.
+
+  1. ROOT CAUSE. Name the single defect and cite the exact artifact id and the specific setting(s)
+     that contain it. State precisely why it produces 502s only after idle gaps.
+  2. SYMPTOM-TO-CAUSE CHAIN. Using edge-log, trace one 502 from the idle gap to the reset. Cite the
+     specific log lines that establish (a) how long the reused connection had been idle, (b) that the
+     failure was a connection reset, not a read timeout, and (c) that the Node app never received the
+     request.
+  3. RED-HERRING DEPLOY. Explain why deploy d-914 (proxy_read_timeout 60s -> 30s) is NOT the root
+     cause, citing the error class in edge-log and what deploy-diff states was and was not changed.
+  4. RED-HERRING LOAD. Explain why the database CPU spike and rate limiting are not the cause, citing
+     the bucket in metrics where the 502s actually cluster.
+  5. ADEQUATE FIX. From fixes, name the one option that resolves the root cause without regression,
+     and state the invariant that must hold between the two idle timers.
+  6. SHALLOW/HARMFUL FIXES. For each of the other three options in fixes, state why it is shallow or
+     harmful, naming the specific regression or the reason it leaves the cause intact.
+
+  Exactly one resolution is defensible per deliverable; it is fully determined by the artifacts.
+artifacts:
+  - id: nginx-conf
+    type: config
+    label: edge/nginx.conf — checkoutd payments edge proxy (production, current)
+    content: |
+      # This is the COMPLETE upstream + proxy configuration for the /v1/charge route.
+      # No other file overrides keepalive, timeout, or retry behavior for this route.
+
+      upstream charge_api {
+          server 10.0.3.11:8080;
+          server 10.0.3.12:8080;
+          keepalive 64;            # up to 64 idle upstream connections kept for reuse (per worker)
+          keepalive_timeout 60s;   # an IDLE upstream connection is kept and reused for up to 60s
+          keepalive_requests 1000;
+      }
+
+      server {
+          listen 443 ssl;
+          location /v1/charge {
+              proxy_pass http://charge_api;
+              proxy_http_version 1.1;
+              proxy_set_header Connection "";   # enables upstream keepalive reuse
+              proxy_read_timeout 30s;           # was 60s before deploy d-914 (see deploy-diff)
+              proxy_next_upstream off;          # nginx does NOT auto-retry /v1/charge on error
+          }
+      }
+      # With proxy_next_upstream off, a failed upstream attempt is returned to the client as 502;
+      # nginx never silently replays the request on a second upstream.
+  - id: node-server
+    type: code
+    label: charge-api/server.ts — the Node HTTP server behind nginx (both 10.0.3.11 and .12)
+    content: |
+      import http from 'node:http';
+      import { app } from './app';
+
+      const server = http.createServer(app);
+
+      // These two values are the ONLY socket-lifetime settings for this server.
+      // They have been at these values since the service first launched; never tuned.
+      server.keepAliveTimeout = 5000;   // ms — an IDLE keep-alive socket is CLOSED by Node after 5s
+      server.headersTimeout   = 6000;   // ms
+
+      server.listen(8080);
+
+      // app.ts emits "req.start conn=<id> POST /v1/charge" for EVERY request it begins to handle,
+      // and a status line when it finishes. If a connection carries no req.start line, the app never
+      // received a request on it. This logging is unconditional and has no sampling.
+  - id: edge-log
+    type: log
+    label: interleaved nginx error log (E) + charge-api app log (A), 12:00-12:04
+    content: |
+      # conn=<id> is the upstream socket nginx uses. Node closes an idle socket 5s after its last use.
+      # A req.start is emitted for every request the app begins; a 502 with no matching A line means
+      # the app never saw the request.
+
+      12:00:01.010 A req.start conn=c-208 POST /v1/charge -> 200 in 42ms   (fresh conn, first use)
+      12:00:01.402 A req.start conn=c-208 POST /v1/charge -> 200 in 39ms   (reused after 0.4s idle, OK)
+      12:00:02.900 A req.start conn=c-208 POST /v1/charge -> 200 in 41ms   (reused after 1.5s idle, OK)
+      # ---- quiet period: no /v1/charge traffic from 12:00:03 to 12:00:46 ----
+      12:00:47.120 E * upstream prematurely closed connection while reading response header;
+                       conn reused=c-208 upstream 10.0.3.11:8080 request "POST /v1/charge" -> 502
+      12:00:47.121 E * errno reported: 104 (Connection reset by peer). No "upstream timed out (110)".
+      # c-208 was last used 12:00:02.900, i.e. idle 44.2s (>> Node's 5s). There is NO A req.start line
+      # for c-208 at 12:00:47: the app never began this request.
+      12:00:47.640 A req.start conn=c-311 POST /v1/charge -> 200 in 43ms   (client's own retry, fresh conn, OK)
+      12:03:10.005 A req.start conn=c-402 POST /v1/charge -> 200 in 40ms
+      12:03:11.220 A req.start conn=c-402 POST /v1/charge -> 200 in 44ms   (reused after 1.2s idle, OK)
+      # ---- quiet period from 12:03:12 to 12:04:02 ----
+      12:04:03.700 E * recv() failed (104: Connection reset by peer) while reading response header
+                       from upstream; conn reused=c-402 request "POST /v1/charge" -> 502
+      # c-402 last used 12:03:11.220, idle 52.5s. Again no A req.start on c-402 at 12:04:03.
+      # Every 502 in this window is a 104 reset on a connection idle > 5s, with no app-side req.start.
+  - id: metrics
+    type: table
+    label: charge-api + database, 5-minute buckets (11:55-12:15)
+    content: |
+      bucket        req_502  req_total  read_timeouts_110  db_p99_ms  db_cpu%  http_429  tls_errs
+      11:55-12:00   1        4120       0                  38         44       0         0
+      12:00-12:05   3        180        0                  41         46       0         0
+      12:05-12:10   0        5330       0                  39         71       0         0
+      12:10-12:15   0        6010       0                  40         77       0         0
+      # On-call note: the 502s land in the LOW-traffic 12:00-12:05 bucket (only 180 requests), the one
+      # with the longest idle gaps. The HIGH-traffic buckets, where db_cpu peaked at 77%, had ZERO 502s.
+      # read_timeouts_110 (nginx "upstream timed out") = 0 all window. http_429 = 0. tls_errs = 0.
+  - id: deploy-diff
+    type: diff
+    label: deploy d-914 — "tighten edge timeouts" (rolled out 11:30, ~30 min before first 502)
+    content: |
+      # Files changed by d-914: edge/nginx.conf ONLY.
+        location /v1/charge {
+            proxy_pass http://charge_api;
+            proxy_http_version 1.1;
+            proxy_set_header Connection "";
+      -     proxy_read_timeout 60s;
+      +     proxy_read_timeout 30s;
+        }
+      # d-914 did NOT touch: upstream keepalive / keepalive_timeout, proxy_next_upstream, or anything
+      # in charge-api/server.ts. server.keepAliveTimeout has been 5000ms since the service launched;
+      # this deploy did not introduce it.
+  - id: fixes
+    type: note
+    label: candidate fixes under discussion (pick the one that holds)
+    content: |
+      F1: Revert d-914 — set proxy_read_timeout back to 60s.
+      F2: Set charge-api server.keepAliveTimeout = 65000 and headersTimeout = 66000, so Node's
+          idle-close window (65s) exceeds nginx's 60s upstream keepalive_timeout.
+      F3: Set nginx upstream keepalive_timeout = 3s, below Node's 5s idle close.
+      F4: Set proxy_next_upstream on with proxy_next_upstream_tries 2, so nginx replays a 502 on a
+          fresh connection.
+
+      Constraints (from runbook R-7):
+        - POST /v1/charge is NOT idempotent: there is no idempotency key on this route. Replaying a
+          charge that the upstream may actually have received risks a DOUBLE CHARGE.
+        - charge-api sits behind mTLS from nginx; each NEW upstream connection costs a full TLS
+          handshake. The upstream accept path sustains at most ~500 handshakes/s; steady state is
+          ~7000 req/s carried over the 64 reused keepalive connections. Connections routinely sit
+          idle 5-40s between the bursts that fill each 5-minute bucket. Do not shrink or disable
+          upstream keepalive reuse.
+  - id: topology
+    type: note
+    label: request path and failure semantics (reference)
+    content: |
+      Path: client -> nginx (edge) -> keepalive pool -> charge-api (Node) -> Postgres.
+      nginx reuses an idle upstream connection if one is available in the pool and within its
+      keepalive_timeout window. Node independently closes any keep-alive socket that has been idle
+      longer than server.keepAliveTimeout. If nginx selects a socket that Node has already closed,
+      Node's side is gone: the first bytes nginx writes draw a TCP RST (errno 104), surfaced to the
+      client as 502 "upstream prematurely closed connection". This is distinct from errno 110
+      ("upstream timed out"), which is what a slow-but-open upstream would produce.
+tags: [debugging, root-cause-isolation, nginx, keepalive, http-502]

package/tasks/debugging/public/pool-exhaustion-held-connection.yaml

@@ -0,0 +1,142 @@
+id: pool-exhaustion-held-connection
+version: 1.0.0
+category: debugging
+cluster: root-cause-isolation
+difficulty: expert
+title: Checkout 503s Under Moderate Load From Database Pool Acquire Timeouts
+summary: >-
+  Under only ~6 orders/second, ordersvc starts returning 503 "pool acquire timeout" on POST /orders.
+  A distributed trace, pool config, saturation snapshot, a metrics table, an EXPLAIN, and a
+  candidate-fix list are provided. The database is nearly idle and the payment gateway is at its
+  normal latency. Isolate the one root cause, separate it from the slow-query, garbage-collection,
+  and slow-gateway decoys, and pick the fix that does not just move the cliff.
+prompt: |
+  Answer each deliverable using ONLY the supplied artifacts (handler, pool-config, trace, saturation,
+  metrics, explain, fixes). Cite the artifact id(s) for every claim.
+
+  1. ROOT CAUSE. Name the single defect and cite the exact artifact id and code location. State why
+     it exhausts the pool at ~6 orders/second even though the database is nearly idle.
+  2. TRACE EVIDENCE. From trace, prove that a pooled database connection is held across the external
+     payment call. Cite the spans and their durations, including when the connection is acquired and
+     when it is released, and how much of the hold is database work vs. the external call.
+  3. SATURATION EVIDENCE. From saturation and metrics, prove the pool is exhausted (not the database).
+     Cite active vs. max connections, the wait queue, database CPU, and per-query times.
+  4. RED-HERRING: SLOW QUERY. Explain why the order insert is not a slow query, citing explain.
+  5. RED-HERRING: GATEWAY / GC. Explain why "the payment gateway is slow" and "GC pauses" are not the
+     root cause, citing the gateway baseline in metrics and the GC figures.
+  6. ADEQUATE FIX. From fixes, name the one option that resolves the root cause without regression,
+     and state the invariant it enforces.
+  7. SHALLOW/HARMFUL FIXES. For each of the other three options in fixes, state why it is shallow or
+     harmful, naming the specific regression or the reason it leaves the cause intact.
+
+  Exactly one resolution is defensible per deliverable; it is fully determined by the artifacts.
+artifacts:
+  - id: handler
+    type: code
+    label: ordersvc/place-order.ts — POST /orders handler (current)
+    content: |
+      // Runs inside a single pooled connection from acquire() to release().
+      export async function placeOrder(req, res) {
+        const conn = await pool.acquire();               // checks out a pooled DB connection
+        try {
+          await conn.query('BEGIN');
+          const order = await conn.query(
+            'INSERT INTO orders(user_id, cart_id, status) VALUES($1,$2,$3) RETURNING id',
+            [req.userId, req.cartId, 'pending'],
+          );
+          // External HTTP call to the third-party payment gateway. Still inside the transaction,
+          // so the pooled connection stays checked out for the whole call.
+          const charge = await paymentGateway.charge({
+            orderId: order.id, amount: req.amount, token: req.token,
+          });
+          await conn.query('UPDATE orders SET status=$1 WHERE id=$2', [charge.ok ? 'paid' : 'failed', order.id]);
+          await conn.query('COMMIT');
+          res.json({ orderId: order.id, status: charge.ok ? 'paid' : 'failed' });
+        } finally {
+          await pool.release(conn);                      // connection returned ONLY here
+        }
+      }
+      // pool.acquire() rejects with PoolAcquireTimeout after pool-config.acquireTimeoutMillis, which
+      // the framework maps to HTTP 503. This is the ONLY handler that talks to paymentGateway.
+  - id: pool-config
+    type: config
+    label: ordersvc database pool + gateway client (complete)
+    content: |
+      pool:
+        max: 10                      # hard cap on concurrent checked-out connections
+        min: 2
+        acquireTimeoutMillis: 2000   # acquire() rejects -> 503 after this
+        idleTimeoutMillis: 30000
+      paymentGateway:
+        baseUrl: https://pay.example-psp.test
+        timeoutMillis: 3000          # the gateway client's own request timeout
+      # These are the complete pool and gateway-client settings; nothing else changes connection
+      # lifetime. pool.max has been 10 since launch; it was not changed in any recent deploy.
+  - id: trace
+    type: log
+    label: distributed trace for one POST /orders (trace t-8842), span waterfall
+    content: |
+      # Indentation = span nesting. [t+..ms] = start offset from request begin. conn = pool connection id.
+      span placeOrder                       [t+0ms]      dur=2003ms
+        span db.pool.acquire                [t+0ms]      dur=1ms     conn=pg-7   (free conn available; waited 0ms)
+        span db.tx.begin                    [t+1ms]      dur=1ms     conn=pg-7
+        span db.insert orders               [t+2ms]      dur=6ms     conn=pg-7   rows=1
+        span http.paymentGateway.charge     [t+8ms]      dur=1980ms              (conn=pg-7 STILL checked out)
+        span db.update orders.status        [t+1989ms]   dur=5ms     conn=pg-7
+        span db.tx.commit                   [t+1997ms]   dur=6ms     conn=pg-7
+        span db.pool.release                [t+2003ms]   dur=0ms     conn=pg-7   (released here, ~2003ms after acquire)
+      # pg-7 was checked out from t+0 to t+2003ms. Actual DB work on pg-7 = 6+5+6 = 17ms (<1%).
+      # The other ~1980ms (99%) is the external charge call, during which pg-7 is unavailable to anyone.
+  - id: saturation
+    type: log
+    label: ordersvc pool saturation snapshot at 12:14:30 (load ~6 orders/s)
+    content: |
+      pool.active_checked_out = 10 / max 10
+      pool.waiters_queued     = 14
+      pool.oldest_waiter_ms   = 1998            # about to hit acquireTimeoutMillis = 2000
+      12:14:30.140 ERROR pool acquire timeout after 2000ms -> 503 (trace t-9001)
+      12:14:30.190 ERROR pool acquire timeout after 2000ms -> 503 (trace t-9002)
+      # Throughput ceiling (Little's law): 10 connections / ~2.0s hold each = ~5 orders/s sustainable.
+      # At ~6 orders/s the wait queue grows without bound until acquires time out and return 503.
+      # All 10 checked-out connections were in the http.paymentGateway.charge span at snapshot time.
+  - id: metrics
+    type: table
+    label: ordersvc + database + gateway, 5-minute buckets, plus baselines
+    content: |
+      bucket        orders_per_s  http_503  db_cpu%  db_p99_query_ms  gw_p99_ms  gc_pause_p99_ms  pool_wait_p99_ms
+      12:00-12:05   3.1           0         9        7                1960       12               40
+      12:05-12:10   4.6           0         10       7                1975       11               210
+      12:10-12:15   6.2           38        11       8                1980       13               2000
+      # Gateway baseline: gw_p99 has been ~1900-2000ms for the last 90 days (payment auth is slow by
+      # design; the gateway SLA permits up to 3000ms). It did NOT rise when the 503s began.
+      # GC: gc_pause_p99 stays 11-13ms all window (no long pauses). db_cpu never exceeds 11%.
+      # pool_wait_p99 climbs from 40ms to 2000ms as load rises; that is the queue, not the database.
+  - id: explain
+    type: note
+    label: EXPLAIN ANALYZE for the order insert (captured under load)
+    content: |
+      EXPLAIN ANALYZE INSERT INTO orders(user_id, cart_id, status) VALUES($1,$2,$3) RETURNING id;
+        Insert on orders  (actual time=0.041..0.052 rows=1 loops=1)
+          ->  Result       (actual time=0.008..0.009 rows=1 loops=1)
+        Planning Time: 0.06 ms
+        Execution Time: 0.09 ms
+      # orders has a primary key on id and an index on (user_id, created_at). The insert is a
+      # sub-millisecond single-row write with no lock contention. It is not, and never was, a slow query.
+  - id: fixes
+    type: note
+    label: candidate fixes under discussion (pick the one that holds)
+    content: |
+      F1: Restructure placeOrder so the pooled connection is NOT held across the gateway call:
+          insert the order and COMMIT (releasing the connection), call paymentGateway.charge with no
+          connection checked out, then acquire a fresh connection for a short UPDATE of the status.
+      F2: Raise pool.max from 10 to 50 to absorb the concurrency.
+      F3: Raise pool.acquireTimeoutMillis from 2000 to 10000 so acquires wait longer instead of 503ing.
+      F4: Add an index to the orders insert and retry the request on 503.
+
+      Constraints:
+        - The shared Postgres instance has max_connections = 100, split across 6 services; ordersvc's
+          steady budget is 10-15 connections. 50 would oversubscribe the cluster.
+        - The POST /orders p99 latency SLA is 2500ms end to end.
+        - Reservations/charges are user-visible: a successful charge emails a receipt immediately, so
+          blind retries of a 503 whose charge may have succeeded can double-charge and double-email.
+tags: [debugging, root-cause-isolation, connection-pool, latency, tracing]

package/tasks/debugging/public/race-oversell-reserve-counter.yaml

@@ -0,0 +1,133 @@
+id: race-oversell-reserve-counter
+version: 1.0.0
+category: debugging
+cluster: concurrency-defect
+difficulty: expert
+title: The Last Seat Sold Twice — A Check-Then-Act Race in Seat Reservation
+summary: >-
+  ticketsvc sold the last seat of an event to two buyers, leaving remaining at -1. The reserve code,
+  two fully specified thread interleavings (one that oversells and one that does not), the database
+  isolation setting, a deployment note about replica count, a sibling release() path, and a
+  candidate-fix list are provided. Identify the exact failing interleaving and the defect, rule out
+  the in-process mutex and atomic-decrement decoys, and pick the fix that closes the race across all
+  replicas.
+prompt: |
+  Answer each deliverable using ONLY the supplied artifacts (reserve-code, trace-fail, trace-safe,
+  db-config, deploy-note, release-code, fixes). Cite the artifact id(s) for every claim.
+
+  1. THE DEFECT. Name the concurrency defect in reserve-code and cite the exact statements involved.
+     State the ordering condition between two callers that triggers an oversell.
+  2. FAILING SCHEDULE. From trace-fail, prove the oversell. Cite the specific interleaved lines that
+     show both callers read the same remaining before either writes, and the final state.
+  3. SAFE SCHEDULE. From trace-safe, explain why the same code does NOT oversell there, and why relying
+     on that outcome is the bug.
+  4. RULE OUT DECOYS. Explain why each is not the cause: (a) the in-process Mutex in reserve-code;
+     (b) the claim that "the UPDATE decrement is atomic, so there is no race"; (c) the release-code
+     sibling. Cite the artifacts.
+  5. ADEQUATE FIX. From fixes, name the one option that closes the race, and state the invariant it
+     enforces.
+  6. SHALLOW/INSUFFICIENT FIXES. For each of the other options in fixes, state why it fails, citing
+     the artifact that makes it insufficient (including deploy-note where relevant).
+
+  Exactly one resolution is defensible per deliverable; it is fully determined by the artifacts.
+artifacts:
+  - id: reserve-code
+    type: code
+    label: ticketsvc/reserve.ts — reserve a seat (current)
+    content: |
+      import { Mutex } from 'async-mutex';
+      const metricsLock = new Mutex();   // used ONLY around the metrics write below
+
+      export async function reserve(eventId, userId) {
+        // (1) CHECK: read current remaining
+        const { remaining } = await db.query(
+          'SELECT remaining FROM events WHERE id=$1', [eventId]);
+        if (remaining <= 0) throw new SoldOut();
+
+        // ... no lock and no transaction is held between the read above and the write below ...
+
+        // (2) ACT: decrement and record the reservation
+        await db.query('UPDATE events SET remaining = remaining - 1 WHERE id=$1', [eventId]);
+        await db.query('INSERT INTO reservations(event_id, user_id) VALUES($1,$2)', [eventId, userId]);
+
+        // metrics only; the lock guards this counter, NOT the check-then-act above
+        await metricsLock.runExclusive(async () => { await db.query(
+          'UPDATE reserve_metrics SET n = n + 1 WHERE event_id=$1', [eventId]); });
+        return 'ok';
+      }
+      // Each db.query() runs in its own autocommit transaction; reserve() opens no explicit transaction.
+      // A successful reservation immediately issues a ticket (email) downstream; see fixes.
+  - id: trace-fail
+    type: log
+    label: interleaved execution — event e-9 has remaining=1, two concurrent reserve() calls
+    content: |
+      # T1 = reserve(e-9, u_a) on replica R1;  T2 = reserve(e-9, u_b) on replica R2. Wall-clock in ms.
+      t=0.000 T1 SELECT remaining FROM events WHERE id=e-9  -> remaining=1
+      t=0.001 T2 SELECT remaining FROM events WHERE id=e-9  -> remaining=1     # T2 reads BEFORE T1 writes
+      t=0.002 T1 check (1 <= 0)? no  -> proceed
+      t=0.003 T2 check (1 <= 0)? no  -> proceed
+      t=0.004 T1 UPDATE events SET remaining=remaining-1 WHERE id=e-9  -> remaining now 0
+      t=0.005 T2 UPDATE events SET remaining=remaining-1 WHERE id=e-9  -> remaining now -1   # OVERSOLD
+      t=0.006 T1 INSERT reservations(e-9, u_a)  -> ok
+      t=0.007 T2 INSERT reservations(e-9, u_b)  -> ok                          # second sale of one seat
+      # Final: events.remaining = -1; TWO reservations exist for the single last seat of e-9.
+      # The two decrement statements each ran correctly; the fault is that both were authorized by
+      # reads taken before either write landed.
+  - id: trace-safe
+    type: log
+    label: interleaved execution — same code, requests happen to serialize
+    content: |
+      # Same two calls, but T2's SELECT happens after T1's UPDATE has committed.
+      t=0.000 T1 SELECT remaining WHERE id=e-9 -> remaining=1
+      t=0.001 T1 check (1<=0)? no -> proceed
+      t=0.002 T1 UPDATE events SET remaining=remaining-1 WHERE id=e-9 -> remaining now 0
+      t=0.003 T1 INSERT reservations(e-9, u_a) -> ok
+      t=0.020 T2 SELECT remaining WHERE id=e-9 -> remaining=0
+      t=0.021 T2 check (0<=0)? YES -> throw SoldOut()   # correct: T2 is rejected
+      # No oversell here, but ONLY because T2's read landed after T1's write. The code is correct
+      # solely under this serial timing; nothing in reserve-code enforces it.
+  - id: db-config
+    type: config
+    label: ticketsvc database settings (complete)
+    content: |
+      database:
+        isolationLevel: READ COMMITTED     # default; reserve() issues each statement autocommit
+        statementTimeoutMs: 5000
+      # There is no advisory-lock, SELECT ... FOR UPDATE, or unique constraint on (event_id) in the
+      # reserve path today. events.remaining has no CHECK (remaining >= 0) constraint.
+  - id: deploy-note
+    type: note
+    label: ticketsvc deployment topology (complete)
+    content: |
+      ticketsvc runs as 4 identical replicas (R1..R4) behind a round-robin load balancer; reserve()
+      requests for the same event are routed to whichever replica the balancer picks, so two
+      concurrent reserves for one event commonly land on DIFFERENT replicas (see trace-fail: R1 and
+      R2). The replicas share one Postgres database and do not share process memory.
+  - id: release-code
+    type: code
+    label: ticketsvc/release.ts — release a seat (sibling path, for contrast)
+    content: |
+      export async function release(eventId) {
+        // Single atomic statement: no check-then-act, so it cannot race with itself.
+        await db.query('UPDATE events SET remaining = remaining + 1 WHERE id=$1', [eventId]);
+        return 'ok';
+      }
+      // release() is idempotent per reservation because it is called exactly once per cancelled
+      // reservation row (guarded by a status transition elsewhere). It is not implicated in oversell.
+  - id: fixes
+    type: note
+    label: candidate fixes under discussion (pick the one that closes the race)
+    content: |
+      F1: Replace the check-then-act with a single conditional atomic statement:
+          UPDATE events SET remaining = remaining - 1 WHERE id=$1 AND remaining > 0 RETURNING remaining;
+          treat "0 rows updated" as SoldOut (and only then INSERT the reservation).
+      F2: Raise database.isolationLevel to SERIALIZABLE, keeping the two separate autocommit statements.
+      F3: Wrap the existing SELECT and UPDATE in metricsLock (the async-mutex already imported).
+      F4: Add SELECT remaining FROM events WHERE id=$1 FOR UPDATE as the CHECK, keeping autocommit
+          (no surrounding BEGIN/COMMIT), then decrement as before.
+      F5: Leave reserve() as is; add a compensator that periodically finds remaining < 0 and cancels
+          the newest reservation.
+
+      Constraint: a successful reserve() issues a ticket (email) to the buyer immediately, before any
+      compensator could run.
+tags: [debugging, concurrency-defect, race-condition, toctou, database]