bridgebench 3.1.0-alpha.0 → 3.1.0-alpha.2

package/tasks/debugging/public/error-propagation-config-swallow.yaml

@@ -0,0 +1,174 @@
+id: error-propagation-config-swallow
+version: 1.0.0
+category: debugging
+cluster: error-propagation
+difficulty: expert
+title: Shipgate Throws Undefined-Multiplier Deep in Rate Calculation, Far From Its Origin
+summary: A shipping-rate service throws a TypeError reading a multiplier deep inside rate calculation, and a retry layer relabels it as ServiceUnavailable. A recent unrelated caching change and a suspected downstream outage are in play. Trace past the throw site and the relabelers to the true origin, distinguish the masking mechanism that hides it, and pick the fix that addresses the origin without masking bad config.
+prompt: |
+  Answer each deliverable using ONLY the supplied artifacts (stacktrace, rate-code,
+  config-loader, zones-diff, retry-mw, error-policy, oncall-note). Cite the artifact id(s)
+  for every claim. Use this vocabulary for verdicts: root-cause, red-herring, adequate-fix,
+  shallow-fix.
+
+  Production alerts show "ServiceUnavailable: rate service degraded" but the underlying
+  crash is "TypeError: Cannot read properties of undefined (reading 'multiplier')" in
+  RateCalculator.apply. Do not stop at the throw site; trace to the true origin.
+
+  1. THROW SITE vs ORIGIN. Identify the exact line that throws and the value that is
+     undefined there, then trace up the call chain to the TRUE origin of the failure —
+     the earliest defect without which the throw could not occur. Name the origin and cite
+     it.
+
+  2. MASKING MECHANISM. Explain precisely why an operator sees a generic error far from the
+     origin rather than a clear one at the origin. Identify every layer that transforms or
+     hides the real error and what each does to it.
+
+  3. RED HERRINGS. Classify (a) the PricingCache change in zones-diff's neighboring commit,
+     (b) the suspected downstream geo-service outage, and (c) the RetryMiddleware. For each,
+     explain why it is or is not the root cause, citing evidence.
+
+  4. POLICY VIOLATION. State which artifact defines how configuration-load failures must be
+     handled, and which component violates that policy. Quote the rule.
+
+  5. FIX ADEQUACY. Three fixes are proposed: (a) add `zoneTable[zone]?.multiplier ?? 1`
+     at the throw site; (b) raise the RetryMiddleware attempt count and the alert threshold;
+     (c) make ConfigLoader stop swallowing parse errors so a malformed config aborts startup,
+     and fix the malformed zones.yaml. Classify each as adequate-fix or shallow-fix and
+     justify which resolves the origin without masking bad configuration.
+artifacts:
+  - id: stacktrace
+    type: log
+    label: production error with surrounding request log
+    content: |
+      12:04:01 INFO  boot: ConfigLoader.load('zones.yaml') ...
+      12:04:01 WARN  ConfigLoader: parse failed for zones.yaml, using defaults ({})   <-- only warning emitted
+      12:04:01 INFO  boot: PricingContext built (zones loaded: 0)
+      12:04:09 INFO  POST /v2/rates zone=EU-WEST weight=2.4kg
+      12:04:09 ERROR RateCalculator.apply: TypeError: Cannot read properties of undefined (reading 'multiplier')
+          at RateCalculator.apply (rate.ts:41)          zoneTable['EU-WEST'] is undefined
+          at PricingContext.quote (pricing-context.ts:88)
+          at RetryMiddleware.run (retry.ts:33)
+      12:04:09 WARN  RetryMiddleware: attempt 1 failed, retrying
+      12:04:39 WARN  RetryMiddleware: attempt 2 failed, retrying
+      12:05:09 ERROR RetryMiddleware: exhausted 3 attempts -> throw ServiceUnavailable('rate service degraded')
+      12:05:09 ERROR POST /v2/rates -> 503 ServiceUnavailable
+      # every retry re-runs the SAME calculation against the SAME empty zoneTable and fails identically.
+  - id: rate-code
+    type: code
+    label: RateCalculator.apply (throw site)
+    content: |
+      // shipgate/rate.ts
+      export class RateCalculator {
+        // zoneTable is injected by PricingContext; keyed by zone code.
+        constructor(private zoneTable: Record<string, { multiplier: number }>) {}
+
+        apply(zone: string, weightKg: number): number {
+          const base = weightKg * BASE_PER_KG;
+          const mult = this.zoneTable[zone].multiplier;   // rate.ts:41 — throws when zoneTable[zone] is undefined
+          return base * mult;
+        }
+      }
+      // NOTE: apply assumes every zone it is asked about exists in zoneTable. It does not
+      // validate zoneTable; it is only ever as correct as the table PricingContext gives it.
+  - id: config-loader
+    type: code
+    label: ConfigLoader.load
+    content: |
+      // shipgate/config-loader.ts
+      export class ConfigLoader {
+        static load(file: string): Record<string, unknown> {
+          try {
+            const text = readFileSync(configPath(file), 'utf8');
+            return parseYaml(text);            // throws on malformed YAML
+          } catch (e) {
+            // Swallow: log a warning and hand back an empty object so boot can continue.
+            logger.warn(`ConfigLoader: parse failed for ${file}, using defaults ({})`);
+            return {};                         // <-- real parse error is discarded here
+          }
+        }
+      }
+  - id: zones-diff
+    type: diff
+    label: recent commits to config and pricing
+    content: |
+      # commit A "add priority tier to EU zones" — shipgate/config/zones.yaml
+        zones:
+          EU-WEST:
+            multiplier: 1.35
+      -     priority: false
+      +     priority: false
+      +      express: true          # <-- over-indented by one space; sibling key becomes a
+                                     #     mapping value, making zones.yaml malformed YAML
+          US-EAST:
+            multiplier: 1.10
+
+      # commit B (unrelated, same day) "cache PricingContext for 60s" — pricing-context.ts
+        export class PricingContext {
+      +   private static cache?: { ctx: PricingContext; at: number };
+          static build() {
+      +     if (this.cache && Date.now() - this.cache.at < 60_000) return this.cache.ctx;
+            const zones = ConfigLoader.load('zones.yaml') as ZoneTable;
+      -     return new PricingContext(zones);
+      +     const ctx = new PricingContext(zones);
+      +     this.cache = { ctx, at: Date.now() };
+      +     return ctx;
+          }
+        }
+      # commit B changes only caching; it passes whatever ConfigLoader.load returns straight through.
+  - id: retry-mw
+    type: code
+    label: RetryMiddleware.run
+    content: |
+      // shipgate/retry.ts
+      export class RetryMiddleware {
+        async run<T>(fn: () => T): Promise<T> {
+          let attempt = 0;
+          for (;;) {
+            try { return fn(); }
+            catch (e) {
+              attempt++;
+              if (attempt >= 3) {
+                // relabels ANY exhausted failure as a generic 503, discarding the cause.
+                throw new ServiceUnavailable('rate service degraded');
+              }
+              await sleep(30_000);
+            }
+          }
+        }
+      }
+      // Retries a pure recomputation; it does not reload config between attempts.
+  - id: error-policy
+    type: spec
+    label: shipgate configuration error-handling policy
+    content: |
+      shipgate error-handling policy (authoritative, applies to all services):
+
+      P1 — Configuration load failures MUST abort process startup with the underlying
+           parse error surfaced. A service MUST NOT boot with partial or empty configuration
+           and MUST NOT substitute silent defaults for a config file that failed to parse.
+      P2 — Retry layers MAY retry transient I/O failures. They MUST NOT retry deterministic
+           errors (a recomputation over unchanged inputs), and MUST preserve the original
+           error cause when surfacing a failure.
+      P3 — geo-service degradation is reported via the geo-health probe and a
+           GeoUnavailable error; it never manifests as an undefined-multiplier TypeError.
+      This policy is the complete contract for config and error handling.
+  - id: oncall-note
+    type: note
+    label: on-call triage thread
+    content: |
+      Triage guesses during the incident (record only; not all correct):
+      - "The 60s PricingContext cache (commit B) is serving a corrupt object." — commit B
+        only adds caching and passes ConfigLoader.load's result through unchanged
+        (zones-diff); with an empty table it caches an empty table, but it did not create
+        the emptiness.
+      - "geo-service is down and that is cascading into rate errors." — per error-policy P3,
+        geo degradation surfaces as GeoUnavailable via the geo-health probe, never as an
+        undefined-multiplier TypeError; the geo-health probe was green all window.
+      - "RetryMiddleware is the problem — it keeps failing." — RetryMiddleware only relabels
+        and re-runs the same deterministic computation; it neither caused nor could fix the
+        empty table (retry-mw).
+      - "Just null-check the multiplier and move on." — that would return a wrong price for
+        every EU-WEST quote silently instead of surfacing that zones.yaml did not load.
+      Reminder: boot logged exactly one WARN — ConfigLoader parse failed, zones loaded: 0.
+tags: [error-propagation, config, swallowed-error, debugging, shipping]

package/tasks/debugging/public/error-propagation-retry-mask.yaml

@@ -0,0 +1,167 @@
+id: error-propagation-retry-mask
+version: 1.0.0
+category: debugging
+cluster: error-propagation
+difficulty: expert
+title: Batchmill Uploads Fail As Timeouts While The Real Error Is A Stale-Signature 403
+summary: An upload worker alerts on UploadTimeout for a third of jobs and on-call blames slow storage. But the retry layer relabels every exhausted failure as a timeout, hiding a 403 SignatureExpired on each attempt, and a recent change pre-signs URLs at enqueue so queue wait eats the signature's 60s validity. Storage is healthy and the client timeout is generous. Trace past the relabeled symptom to the true origin and pick the fix that addresses it.
+prompt: |
+  Answer each deliverable using ONLY the supplied artifacts (alert-log, upload-code,
+  signer-code, signer-diff, storage-metrics, queue-metrics, http-config, error-taxonomy).
+  Cite the artifact id(s) for every claim. Use this vocabulary for verdicts: root-cause,
+  red-herring, adequate-fix, shallow-fix.
+
+  The "batchmill" worker uploads files to object storage. About 30% of jobs alert as
+  "UploadTimeout: upstream did not respond in 30s". Do not stop at the alert's label;
+  trace to the true origin.
+
+  1. RELABELED SYMPTOM vs ORIGIN. State what the alert claims and what actually failed on
+     each attempt. Identify the TRUE origin — the earliest defect without which the failure
+     could not occur — and cite it.
+
+  2. MASKING MECHANISM. Explain precisely how a 403 SignatureExpired becomes an
+     "UploadTimeout" alert. Identify the layer that performs the relabel and why the retries
+     do not help.
+
+  3. WHICH JOBS FAIL. Using the numbers in the artifacts, state the precise condition that
+     separates the ~30% that fail from the ~70% that succeed, and show the arithmetic for
+     job u-9931.
+
+  4. RED HERRINGS. Classify (a) the storage backend latency, (b) the 30s HTTP client
+     timeout, and (c) the retry wrapper. For each, explain with evidence why it is or is not
+     the root cause.
+
+  5. FIX ADEQUACY. Three fixes are proposed: (a) raise HTTP client timeout to 120s and
+     retries to 10; (b) provision a larger storage instance to cut latency; (c) mint the
+     signature at the worker just before the put (or re-sign per attempt) so no put uses a
+     signature older than its 60s validity. Classify each as adequate-fix or shallow-fix and
+     justify which resolves the origin.
+artifacts:
+  - id: alert-log
+    type: log
+    label: worker log for job u-9931 (representative failing job)
+    content: |
+      # u-9931 was pre-signed at enqueue, then waited in the queue before a worker ran it.
+      09:58:40 INFO  enqueue u-9931: signRequest minted url, signature valid 09:58:40..09:59:40 (60s)
+      10:00:00 INFO  worker dequeued u-9931 (queue wait 80s); using the url minted at enqueue
+      10:00:00 WARN  storage.put attempt 1 -> HTTP 403 SignatureExpired (signature age 80s > 60s)  [12ms]
+      10:00:30 WARN  storage.put attempt 2 -> HTTP 403 SignatureExpired (age 110s; url NOT re-signed) [11ms]
+      10:01:00 WARN  storage.put attempt 3 -> HTTP 403 SignatureExpired (age 140s) [13ms]
+      10:01:00 ERROR withRetry exhausted 3 attempts -> throw UploadTimeout('upstream did not respond in 30s')
+      10:01:00 ERROR job u-9931 failed: UploadTimeout
+      # Every attempt returns 403 in ~12ms; none times out. The final error type is fixed by code, not measured.
+      # Contrast u-9930: enqueued 10:00:10, dequeued 10:00:25 (wait 15s), put at age 15s -> 200 OK.
+  - id: upload-code
+    type: code
+    label: enqueue and worker paths with retry wrapper
+    content: |
+      // batchmill/upload.ts
+      // PRODUCER (enqueue): pre-signs the URL now and stores it on the job.
+      function enqueue(file) {
+        const url = signRequest(bucketPathFor(file));   // signed at ENQUEUE (see signer-diff)
+        queue.push({ id: file.id, url, body: file.bytes });
+      }
+
+      // WORKER (dequeue): uses the URL minted at enqueue; does not sign.
+      async function uploadJob(job) {
+        return withRetry(3, () => storageClient.put(job.url, job.body));
+      }
+
+      // withRetry: retries on ANY thrown error; after N attempts throws a FIXED UploadTimeout.
+      async function withRetry(n, fn) {
+        let attempt = 0;
+        for (;;) {
+          try { return await fn(); }
+          catch (e) {
+            attempt++;
+            if (attempt >= n) {
+              // relabels the last failure as a timeout regardless of what actually happened.
+              throw new UploadTimeout('upstream did not respond in 30s');
+            }
+            await sleep(30_000);          // wait between attempts; job.url is NOT re-signed
+          }
+        }
+      }
+  - id: signer-code
+    type: code
+    label: signRequest (current)
+    content: |
+      // batchmill/signer.ts
+      export function signRequest(bucketPath: string): string {
+        // Mints a presigned URL whose signature the backend accepts for 60 seconds.
+        return presign(bucketPath, { expiresInSec: 60 });
+      }
+      // The backend returns HTTP 403 SignatureExpired for any put received more than
+      // expiresInSec seconds after the signature was minted. Signing is cheap (~0.3ms) and
+      // has no per-call side effects; a fresh signature can be minted at any point.
+  - id: signer-diff
+    type: diff
+    label: signing move shipped 2026-07-05
+    content: |
+      # commit "pre-sign at enqueue to shave worker dequeue latency" — batchmill/upload.ts
+        function enqueue(file) {
+      +   const url = signRequest(bucketPathFor(file));   // NEW: sign now, at enqueue
+      +   queue.push({ id: file.id, url, body: file.bytes });
+        }
+
+        async function uploadJob(job) {
+      -   const url = signRequest(job.bucketPath);         // BEFORE: signed at dequeue, right before put
+      -   return withRetry(3, () => storageClient.put(url, job.body));
+      +   return withRetry(3, () => storageClient.put(job.url, job.body));  // AFTER: reuse enqueue-time url
+        }
+      # BEFORE: the worker minted a fresh 60s signature immediately before the put, so queue
+      # wait did not count against it. AFTER: the signature is minted at enqueue, so the full
+      # queue wait (plus retry backoff) is charged against the 60s validity. The 60s expiry
+      # itself was not changed.
+  - id: storage-metrics
+    type: table
+    label: object-storage backend metrics (incident window)
+    content: |
+      metric                         value
+      -----------------------------  -----------
+      put p50 latency                12 ms
+      put p99 latency                41 ms
+      5xx rate                       0.00%
+      403 SignatureExpired rate      31.2%      <-- matches the ~30% failing jobs
+      throttling (503 SlowDown)      0.00%
+      # The backend is healthy and fast; it REJECTS ~31% of puts for expired signatures.
+      # It is not timing out, throttling, or erroring on the server side.
+  - id: queue-metrics
+    type: table
+    label: worker queue wait distribution (incident window)
+    content: |
+      percentile   queue_wait
+      ----------   ----------
+      p50          22 s
+      p70          60 s          <-- 70% of jobs wait <= 60s; ~30% wait longer
+      p90          145 s
+      p99          410 s
+      # queue wait is the time from enqueue (when the URL is now signed) to dequeue.
+      # Share of jobs with queue_wait > 60s ~= 30%, matching the 403 rate in storage-metrics.
+  - id: http-config
+    type: config
+    label: storage HTTP client configuration
+    content: |
+      # batchmill/http.yaml
+      storageClient:
+        connectTimeoutMs: 5000
+        requestTimeoutMs: 30000     # 30s; the "30s" quoted in the UploadTimeout string
+        maxSockets: 64
+      # The 30s in the alert text is copied from this setting; it is NOT a measured elapsed
+      # time. Observed put responses complete in ~12ms (storage-metrics, alert-log).
+  - id: error-taxonomy
+    type: spec
+    label: batchmill upload error taxonomy
+    content: |
+      batchmill error taxonomy (authoritative, complete for the upload path):
+
+      - UploadTimeout: the storage endpoint accepted the connection but sent no response
+        within requestTimeoutMs. Emitted ONLY on a genuine socket/read timeout.
+      - SignatureExpired (HTTP 403): the presigned URL's signature was older than its
+        expiresInSec when the backend received it. A retry with the SAME url cannot succeed;
+        the url must be re-signed to recover.
+      - Queue wait is unbounded and independent of signing; a job may sit in the queue for an
+        arbitrary time before a worker dequeues it.
+      There are no other upload error classes. withRetry's thrown type is a label chosen by
+      code, not a classification of the underlying HTTP status.
+tags: [error-propagation, retry, presigned-url, debugging, storage]

package/tasks/debugging/public/fix-adequacy-cursor-pagination.yaml

@@ -0,0 +1,166 @@
+id: fix-adequacy-cursor-pagination
+version: 1.0.0
+category: debugging
+cluster: fix-adequacy
+difficulty: expert
+title: Feed Pagination Duplicates And Skips Items, With Four Candidate Fixes To Judge
+summary: A feed paginated by OFFSET over created_at DESC returns duplicate and skipped items when rows are inserted or deleted between page requests. Four candidate fix diffs are on the table. Exactly one resolves the root cause without reintroducing a regression the contract describes; one masks, one is per-request only, and one switches to keyset on a non-unique column and drops rows at a tie boundary. Judge each and name the disqualifying over-rejection.
+prompt: |
+  Answer each deliverable using ONLY the supplied artifacts (contract, current-code,
+  bug-report, boundary-data, fix-a, fix-b, fix-c, fix-d). Cite the artifact id(s) for every
+  claim. Use this vocabulary for verdicts: root-cause, red-herring, adequate-fix,
+  shallow-fix. A fix is ADEQUATE only if it resolves the root cause AND reintroduces no
+  regression the artifacts describe; any fix that merely masks the symptom, fixes only one
+  request in isolation, or reintroduces a described regression is SHALLOW.
+
+  The "streamly" feed at GET /v2/feed returns some items twice and skips others while a
+  user scrolls. Judge the root cause and the four candidate fixes.
+
+  1. ROOT CAUSE. Name the single defect in current-code that lets items be duplicated or
+     skipped across page requests, and explain the mechanism using bug-report's scenario.
+
+  2. FIX VERDICTS. Classify EACH of fix-a, fix-b, fix-c, fix-d as adequate-fix or
+     shallow-fix, and for each give the exact mechanism that justifies the verdict (what it
+     masks, what it leaves unfixed, or what described regression it reintroduces).
+
+  3. THE ADEQUATE FIX. Identify the single adequate fix and show, using boundary-data, that
+     it produces no duplicate and no skip across the page boundary, including at the tie in
+     created_at.
+
+  4. WHY C IS NOT ADEQUATE. Using boundary-data, show concretely which item(s) fix-c drops
+     or repeats at the created_at tie boundary, and which contract rule it violates.
+
+  5. OVER-REJECTION TRAP. State the contract rule that makes rejecting the adequate fix
+     "because it changes the cursor format" an incorrect disqualification, and quote it.
+artifacts:
+  - id: contract
+    type: spec
+    label: streamly feed pagination contract
+    content: |
+      streamly GET /v2/feed pagination contract (authoritative, complete):
+
+      R1 — CURSOR IS OPAQUE. The `cursor` query parameter is an opaque token. Clients MUST
+           NOT parse it. Its internal encoding is an implementation detail and MAY change
+           between releases without a version bump. Changing what the cursor encodes is
+           explicitly permitted.
+      R2 — NO DUPLICATES OR SKIPS. Within a single scroll session, an item MUST NOT be
+           returned in more than one page, and no item that existed for the whole session
+           and matches the filter MUST be skipped — even when other rows are inserted or
+           deleted concurrently between page requests.
+      R3 — TOTAL ORDER. Results are ordered created_at DESC. created_at is NOT unique (many
+           rows share the same millisecond). Ties on created_at MUST be broken by id DESC to
+           give a deterministic total order. A pagination scheme that cannot distinguish two
+           rows with equal created_at is non-conforming.
+      This contract is the complete set of pagination requirements.
+  - id: current-code
+    type: code
+    label: current pagination query (buggy)
+    content: |
+      // streamly/feed.ts — current, buggy.
+      // cursor encodes a numeric OFFSET into the created_at DESC ordering.
+      async function page(cursor?: string) {
+        const offset = cursor ? decodeOffset(cursor) : 0;   // e.g. "o:3" -> 3
+        const rows = await db.query(`
+          SELECT id, created_at, body FROM posts
+          WHERE deleted = false
+          ORDER BY created_at DESC
+          LIMIT 3 OFFSET ${offset}
+        `);
+        return { rows, cursor: encodeOffset(offset + rows.length) };
+      }
+      // The next page is "the same ORDER BY, skip N rows". N is a position, not an anchor to
+      // a specific row.
+  - id: bug-report
+    type: note
+    label: reproduction of the duplicate and the skip
+    content: |
+      Repro (page size 3), newest first.
+      Initial feed:            [I10, I9, I8, I7, I6, I5, ...]
+      Page 1 (offset 0):       returns I10, I9, I8  -> next cursor offset=3
+
+      DUPLICATE case: before page 2, two new posts I11, I12 are inserted at the head.
+      Feed now:                [I12, I11, I10, I9, I8, I7, I6, ...]
+      Page 2 (offset 3):       returns I9, I8, I7  -> I9 and I8 were ALREADY shown on page 1
+                               (duplicates), because inserting 2 rows above shifted every
+                               row's offset by +2.
+
+      SKIP case: instead, before page 2, two head posts I10, I9 are deleted.
+      Feed now:                [I8, I7, I6, I5, ...]
+      Page 2 (offset 3):       returns I5, I4, I3  -> I8, I7, I6 were never shown (skipped),
+                               because removing 2 rows above pulled every row's offset by -2.
+      Both come from the same cause: OFFSET counts positions in a list that changes between
+      requests.
+  - id: boundary-data
+    type: table
+    label: rows around the page-1/page-2 boundary (note the created_at tie)
+    content: |
+      id   created_at (UTC)            note
+      ---  --------------------------  -------------------------------
+      I10  2026-07-10T10:00:00.002Z    page 1, row 1
+      I9   2026-07-10T10:00:00.001Z    page 1, row 2
+      I8   2026-07-10T10:00:00.000Z    page 1, row 3  <-- LAST row of page 1 (the anchor)
+      I7   2026-07-10T10:00:00.000Z    page 2, row 1  <-- SAME created_at as I8 (tie)
+      I6   2026-07-10T10:00:00.000Z    page 2, row 2  <-- SAME created_at as I8 (tie)
+      I5   2026-07-09T09:59:59.998Z    page 2, row 3
+      # Under the R3 total order (created_at DESC, id DESC) the correct sequence is
+      # I8, I7, I6 (10:00:00.000, ids 8 > 7 > 6), so page 2 must start at I7.
+  - id: fix-a
+    type: diff
+    label: candidate fix A
+    content: |
+      # fix A: keep OFFSET; dedup by id in the client before rendering.
+        // streamly-web/feed-list.ts
+      +   const seen = new Set(renderedIds);
+      +   const deduped = page.rows.filter(r => !seen.has(r.id));   // drop ids already shown
+          render(deduped);
+      # Server query is unchanged (still OFFSET). Only the client hides repeats.
+  - id: fix-b
+    type: diff
+    label: candidate fix B
+    content: |
+      # fix B: keep OFFSET; wrap the single page query in a REPEATABLE READ transaction.
+        async function page(cursor?) {
+          const offset = cursor ? decodeOffset(cursor) : 0;
+      +   return db.tx({ isolation: 'repeatable read' }, async (t) => {
+            const rows = await t.query(`SELECT ... ORDER BY created_at DESC LIMIT 3 OFFSET ${offset}`);
+            return { rows, cursor: encodeOffset(offset + rows.length) };
+      +   });
+        }
+      # Each page request is its own transaction/snapshot; there is no shared snapshot across
+      # the separate page-1 and page-2 requests.
+  - id: fix-c
+    type: diff
+    label: candidate fix C
+    content: |
+      # fix C: keyset pagination anchored on created_at only.
+        async function page(cursor?) {
+      -   const offset = cursor ? decodeOffset(cursor) : 0;
+      -   const rows = await db.query(`... ORDER BY created_at DESC LIMIT 3 OFFSET ${offset}`);
+      -   return { rows, cursor: encodeOffset(offset + rows.length) };
+      +   const ts = cursor ? decodeTs(cursor) : MAX_TS;              // last row's created_at
+      +   const rows = await db.query(`
+      +     SELECT id, created_at, body FROM posts
+      +     WHERE deleted = false AND created_at < '${ts}'
+      +     ORDER BY created_at DESC LIMIT 3`);
+      +   return { rows, cursor: encodeTs(rows.at(-1)?.created_at) };
+        }
+      # cursor now encodes only created_at.
+  - id: fix-d
+    type: diff
+    label: candidate fix D
+    content: |
+      # fix D: keyset pagination anchored on the (created_at, id) tuple.
+        async function page(cursor?) {
+      -   const offset = cursor ? decodeOffset(cursor) : 0;
+      -   const rows = await db.query(`... ORDER BY created_at DESC LIMIT 3 OFFSET ${offset}`);
+      -   return { rows, cursor: encodeOffset(offset + rows.length) };
+      +   const { ts, id } = cursor ? decodeKey(cursor) : { ts: MAX_TS, id: MAX_ID };
+      +   const rows = await db.query(`
+      +     SELECT id, created_at, body FROM posts
+      +     WHERE deleted = false AND (created_at, id) < ('${ts}', ${id})
+      +     ORDER BY created_at DESC, id DESC LIMIT 3`);
+      +   const last = rows.at(-1);
+      +   return { rows, cursor: encodeKey({ ts: last?.created_at, id: last?.id }) };
+        }
+      # cursor now encodes the (created_at, id) of the last returned row.
+tags: [fix-adequacy, pagination, keyset, debugging, cursor]