vouchington-tooling 2.0.2 → 3.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (68) hide show
  1. package/package.json +2 -2
  2. package/scripts/__pycache__/run_bounded_process.cpython-314.pyc +0 -0
  3. package/skills/manifest.json +15 -202
  4. package/skills/organize-github-issues/SKILL.md +20 -3
  5. package/skills/pr-description/SKILL.md +9 -65
  6. package/skills/retrospective/SKILL.md +15 -0
  7. package/skills/retrospective-distill/SKILL.md +1 -1
  8. package/skills/retrospective-distill/references/issue-body.md +6 -1
  9. package/skills/review-github-issue-taxonomy/SKILL.md +3 -4
  10. package/skills/triage-loop/SKILL.md +107 -26
  11. package/skills/triage-prs/SKILL.md +25 -27
  12. package/skills/triage-prs/references/feedback.md +14 -2
  13. package/skills/triage-prs/references/rubric.md +4 -4
  14. package/skills/triage-sweep/SKILL.md +26 -23
  15. package/skills/agent-workflow/SKILL.md +0 -72
  16. package/skills/agent-workflow/references/evidence-sweep.md +0 -12
  17. package/skills/agent-workflow/references/four-refs.md +0 -74
  18. package/skills/agent-workflow/references/implementation-and-review.md +0 -4
  19. package/skills/agent-workflow/references/implementation.md +0 -20
  20. package/skills/agent-workflow/references/review-response.md +0 -49
  21. package/skills/agent-workflow/references/review.md +0 -10
  22. package/skills/backend-vitest-test-authoring/SKILL.md +0 -22
  23. package/skills/backend-vitest-test-authoring/references/integration-boundaries.md +0 -10
  24. package/skills/bounded-iteration/SKILL.md +0 -92
  25. package/skills/bounded-iteration/references/sweep-patterns.md +0 -719
  26. package/skills/dependabot/SKILL.md +0 -117
  27. package/skills/dotnet-test-authoring/SKILL.md +0 -24
  28. package/skills/git-commit-checklist/SKILL.md +0 -29
  29. package/skills/github-actions-authoring/SKILL.md +0 -63
  30. package/skills/github-actions-checklist/SKILL.md +0 -118
  31. package/skills/github-issue/SKILL.md +0 -121
  32. package/skills/glide-mq/SKILL.md +0 -52
  33. package/skills/glide-mq/references/lease-patterns.md +0 -44
  34. package/skills/nextjs-vitest-test-authoring/SKILL.md +0 -21
  35. package/skills/nextjs-vitest-test-authoring/references/framework-boundaries.md +0 -9
  36. package/skills/npm-publishing/SKILL.md +0 -72
  37. package/skills/package-json-checklist/SKILL.md +0 -31
  38. package/skills/planning/SKILL.md +0 -37
  39. package/skills/planning/references/impact-discovery.md +0 -11
  40. package/skills/playwright-authoring/SKILL.md +0 -23
  41. package/skills/playwright-authoring/references/browser-reliability.md +0 -9
  42. package/skills/postgres-node-performance-tuning/SKILL.md +0 -35
  43. package/skills/postgres-node-performance-tuning/references/performance-patterns.md +0 -175
  44. package/skills/postgres-node-performance-tuning/references/reading-plans.md +0 -111
  45. package/skills/postgres-partitioning-uuid-v7/SKILL.md +0 -35
  46. package/skills/postgres-partitioning-uuid-v7/references/partition-lifecycle.md +0 -53
  47. package/skills/postgres-schema-design/SKILL.md +0 -31
  48. package/skills/postgres-schema-design/references/naming.md +0 -34
  49. package/skills/postgres-schema-design/references/relations.md +0 -66
  50. package/skills/postgres-schema-design/references/table-shapes.md +0 -78
  51. package/skills/pr-description/references/examples.md +0 -189
  52. package/skills/review-ci-logs/SKILL.md +0 -44
  53. package/skills/revisit-followups/SKILL.md +0 -47
  54. package/skills/stacked-prs/SKILL.md +0 -98
  55. package/skills/stacked-prs/references/drain.md +0 -177
  56. package/skills/stacked-prs/references/gh-stack.md +0 -147
  57. package/skills/static-analysis-checklist/SKILL.md +0 -32
  58. package/skills/storybook-authoring/SKILL.md +0 -22
  59. package/skills/storybook-authoring/references/component-coverage.md +0 -9
  60. package/skills/swift-test-authoring/SKILL.md +0 -23
  61. package/skills/swift-test-authoring/references/network-test-doubles.md +0 -9
  62. package/skills/test-authoring/SKILL.md +0 -27
  63. package/skills/test-authoring/references/core-practice.md +0 -16
  64. package/skills/test-authoring/references/dependency-boundaries.md +0 -38
  65. package/skills/test-authoring/references/tautological-tests.md +0 -47
  66. package/skills/triage-loop/references/shepherd.md +0 -64
  67. package/skills/vitest-test-authoring/SKILL.md +0 -22
  68. package/skills/vitest-test-authoring/references/mock-boundaries.md +0 -9
@@ -1,719 +0,0 @@
1
- # Sweep patterns
2
-
3
- Synthetic examples for [bounded iteration](../SKILL.md). `db.query` stands for the consumer's
4
- executor and `config` for its runtime configuration; every cap in SQL is a bound parameter.
5
- `min_uuidv7(timestamptz)` stands for the consumer's helper that returns the smallest UUIDv7 for a
6
- timestamp, like the lower-bound helper in
7
- [performance patterns](../../postgres-node-performance-tuning/references/performance-patterns.md).
8
- Comments on tables and columns are omitted for space.
9
-
10
- ## Due timestamp, partial index, capped claim
11
-
12
- A work-item row exists only while work is pending, so the index covers only rows a claim can pick:
13
-
14
- ```sql
15
- CREATE INDEX idx_invoice_delivery_work_items__available_at
16
- ON invoice_delivery_work_items (available_at, id)
17
- WHERE completed_at IS NULL;
18
- ```
19
-
20
- One claim takes at most `$1` due rows. Competing workers skip rows another worker holds, and the
21
- lease moves `available_at` to its expiry, so an abandoned claim is due again through the same index
22
- without an `OR` in the predicate:
23
-
24
- ```sql
25
- WITH due AS (
26
- SELECT id
27
- FROM invoice_delivery_work_items
28
- WHERE completed_at IS NULL AND available_at <= now()
29
- ORDER BY available_at, id
30
- LIMIT $1
31
- FOR UPDATE SKIP LOCKED
32
- )
33
- UPDATE invoice_delivery_work_items AS item
34
- SET lease_token = gen_random_uuid(),
35
- leased_at = now(),
36
- lease_expires_at = now() + $2::interval,
37
- available_at = now() + $2::interval,
38
- attempt_count = item.attempt_count + 1
39
- FROM due
40
- WHERE item.id = due.id
41
- RETURNING item.id, item.invoice_id, item.lease_token;
42
- ```
43
-
44
- Renewal moves `lease_expires_at` and `available_at` to the same new deadline, so a renewed lease
45
- is not due. Renewal, completion, and failure all add `AND lease_token = $mine`; see
46
- [table shapes](../../postgres-schema-design/references/table-shapes.md).
47
-
48
- ## Dirty marker or work item
49
-
50
- Do not walk every order to find the few that need reconciling. The writer that changes an order
51
- marks it in the same transaction, and the reconciler claims only marks, as above:
52
-
53
- ```sql
54
- CREATE TABLE order_reconcile_work_items (
55
- id uuid PRIMARY KEY DEFAULT uuidv7(),
56
- order_id uuid NOT NULL UNIQUE REFERENCES orders (id) ON DELETE CASCADE,
57
- available_at timestamptz NOT NULL DEFAULT now(),
58
- lease_token uuid,
59
- leased_at timestamptz,
60
- lease_expires_at timestamptz,
61
- attempt_count integer NOT NULL DEFAULT 0,
62
- generation integer NOT NULL DEFAULT 0,
63
- completed_at timestamptz
64
- );
65
-
66
- INSERT INTO order_reconcile_work_items (order_id)
67
- VALUES ($1)
68
- ON CONFLICT (order_id) DO UPDATE
69
- SET available_at = GREATEST(order_reconcile_work_items.available_at, now()),
70
- completed_at = NULL,
71
- generation = order_reconcile_work_items.generation + 1;
72
- ```
73
-
74
- The claim above applies with `order_id` for `invoice_id`, and its `RETURNING` also includes
75
- `generation`, so `$claimed` is captured in the same statement as the lease. Completion adds
76
- `AND generation = $claimed`. A re-mark never makes a leased row due early, since
77
- the claim moved `available_at` to the lease expiry. A completion that finds a newer generation
78
- releases the lease with `available_at = now()`, and the next claim takes the row again. Run cost
79
- follows the number of changes, not the number of orders. A job fanned out per account reads by
80
- parent id, `WHERE account_id = $1`, for the one account it was started for.
81
-
82
- ## High-water-mark window with an overlap
83
-
84
- Prefer keeping the position and the sweep's fixed end in job data for a single-writer continuation.
85
- Use a persisted cursor table only when the position must be transactional with domain rows, as
86
- here.
87
-
88
- An export reads `invoices` added since the last sweep. The cursor row holds the position and the
89
- sweep's fixed end, and the job seeds its own row:
90
-
91
- ```sql
92
- CREATE TABLE invoice_export_cursors (
93
- is_singleton boolean PRIMARY KEY DEFAULT true CHECK (is_singleton),
94
- cursor_invoice_id uuid,
95
- sweep_upper_bound_invoice_id uuid
96
- );
97
-
98
- INSERT INTO invoice_export_cursors DEFAULT VALUES ON CONFLICT DO NOTHING;
99
- ```
100
-
101
- A UUIDv7 is assigned at insert, not commit, so a row below the last sweep's end can commit after
102
- that sweep passed it. A new sweep starts an overlap before the previous end and fixes its own end
103
- now, only when the previous sweep reached its end. `$1` is the overlap, and NULL means the start,
104
- so the first sweep begins at the start. The overlap must exceed the longest insert-to-commit
105
- delay, which the consumer enforces, for example with a transaction timeout; a row that commits
106
- after the overlap has passed is missed for good. Work that cannot tolerate a miss writes a
107
- work-item row in the same transaction instead:
108
-
109
- ```sql
110
- UPDATE invoice_export_cursors
111
- SET cursor_invoice_id =
112
- min_uuidv7(uuid_extract_timestamp(sweep_upper_bound_invoice_id) - $1::interval),
113
- sweep_upper_bound_invoice_id = min_uuidv7(now())
114
- WHERE is_singleton
115
- AND cursor_invoice_id IS NOT DISTINCT FROM sweep_upper_bound_invoice_id;
116
- ```
117
-
118
- Each batch moves strictly forward inside the sweep, where `$1` is the cursor (the nil UUID for the
119
- start), `$2` the sweep's end, and `$3` the batch size:
120
-
121
- ```sql
122
- SELECT id, account_id, total_cents
123
- FROM invoices
124
- WHERE id > $1 AND id <= $2
125
- ORDER BY id
126
- LIMIT $3;
127
- ```
128
-
129
- Persist the last `id` as the cursor after each batch with a compare-and-set, so a stale or
130
- overlapping worker cannot move the position backward or into another sweep. `$1` is the new
131
- position, `$2` the cursor this batch started from (NULL at the start), and `$3` the sweep's end.
132
- Zero updated rows means another worker moved on, so the run stops:
133
-
134
- ```sql
135
- UPDATE invoice_export_cursors
136
- SET cursor_invoice_id = $1
137
- WHERE is_singleton
138
- AND cursor_invoice_id IS NOT DISTINCT FROM $2::uuid
139
- AND sweep_upper_bound_invoice_id = $3;
140
- ```
141
-
142
- A short batch ends the sweep: set the cursor to the sweep's end the same way, and the next run
143
- starts a new sweep. Rows in the overlap are read twice, so the export must be idempotent.
144
-
145
- A time position works the same way on a column the database sets at insert, such as
146
- `received_at timestamptz NOT NULL DEFAULT clock_timestamp()`, with the same overlap rule. Never use a business
147
- time such as `occurred_at`: a late-arriving row carries an old timestamp and falls behind every
148
- later window. Avoid `now()` too, because it records the transaction's start, not the insert. Fix the window's end when the window opens, keep a composite `(time, id)` keyset
149
- position inside it, and read
150
- `WHERE (received_at, id) > ($1, $2) AND received_at <= $3 ORDER BY received_at, id`, where `$3` is
151
- the window's end. Rows that share a timestamp are neither skipped nor repeated, and new arrivals
152
- wait for the next window. Advance `<verb>_through_at` to the window's end only after the window is
153
- drained, and start the next window an overlap earlier.
154
-
155
- ## Capped batch loop with `hasMore`
156
-
157
- The run stops at a batch cap or a deadline and says whether work remains. A short batch is the
158
- only proof that nothing is left. Rows a `SKIP LOCKED` claim passed over are held by another
159
- claimer, who owns them; if that claim rolls back, they are due again on the next scheduled run,
160
- so a short batch still ends this run's backlog. A cap that stops the run reports
161
- `hasMore: true`, even when the last batch happened to drain the work; the continuation finds
162
- nothing and stops:
163
-
164
- ```ts
165
- interface RunResult {
166
- processed: number
167
- hasMore: boolean
168
- }
169
-
170
- export async function runOrderReconcile(config: ReconcileConfig): Promise<RunResult> {
171
- const deadline = Date.now() + config.runDeadlineMs
172
- let processed = 0
173
- for (let batch = 0; batch < config.maxBatchesPerRun; batch += 1) {
174
- if (Date.now() >= deadline) return { processed, hasMore: true }
175
- const items = await claimDueItems(config.batchSize, config.leaseInterval)
176
- await reconcileOrders(items)
177
- processed += items.length
178
- if (items.length < config.batchSize) return { processed, hasMore: false }
179
- }
180
- return { processed, hasMore: true }
181
- }
182
-
183
- export async function orderReconcileJob(config: ReconcileConfig): Promise<void> {
184
- const { hasMore } = await runOrderReconcile(config)
185
- if (hasMore) await enqueueContinuation('order-reconcile')
186
- }
187
- ```
188
-
189
- The persisted position is the work-item table itself: claimed and completed rows leave the due
190
- set. A cursor sweep persists its position after each batch. The job asks for a continuation
191
- instead of looping on.
192
-
193
- ## Tunables
194
-
195
- Read each cap from runtime configuration with a validated default and a hard maximum. A bad value
196
- fails at startup, and no cap is unset or infinite:
197
-
198
- ```ts
199
- interface Bound {
200
- default: number
201
- max: number
202
- }
203
-
204
- export function readBound(name: string, raw: string | undefined, bound: Bound): number {
205
- const value = raw === undefined ? bound.default : Number(raw)
206
- if (!Number.isInteger(value) || value < 1 || value > bound.max) {
207
- throw new RangeError(`${name} must be an integer from 1 to ${bound.max}`)
208
- }
209
- return value
210
- }
211
-
212
- const batchSize = readBound('ORDER_RECONCILE_BATCH_SIZE', env.ORDER_RECONCILE_BATCH_SIZE, {
213
- default: 200,
214
- max: 1000,
215
- })
216
- ```
217
-
218
- Bind the value as a parameter, `LIMIT $1`, never a literal in SQL text. `LIMIT 1` for a single-row
219
- lookup and an external API's documented page maximum are contracts, not tunables.
220
-
221
- ## List endpoint
222
-
223
- The API schema keeps a static ceiling. The runtime default and effective maximum are tunable but
224
- never above it, and the server rejects an invalid size and clamps a large one:
225
-
226
- ```ts
227
- const API_MAX_PAGE_SIZE = 100
228
-
229
- export function pageSize(requested: number | undefined, config: PageConfig): number {
230
- const ceiling = Math.min(config.maxPageSize, API_MAX_PAGE_SIZE)
231
- const value = requested ?? config.defaultPageSize
232
- if (!Number.isInteger(value) || value < 1) {
233
- throw new RangeError('Page size must be a positive integer')
234
- }
235
- return Math.min(value, ceiling)
236
- }
237
- ```
238
-
239
- Page by keyset on the key the index serves. Fetch `pageSize + 1` rows to learn whether another page
240
- exists, return `pageSize`, and encode the last returned `id` as the next cursor. The first page
241
- omits `id < $2`:
242
-
243
- ```sql
244
- SELECT id, total_cents
245
- FROM orders
246
- WHERE account_id = $1 AND id < $2
247
- ORDER BY id DESC
248
- LIMIT $3;
249
- ```
250
-
251
- ## Bounded DELETE through a CTE
252
-
253
- DELETE has no `LIMIT`, so select the ids in a CTE and delete by join. The partial index serves the
254
- retention predicate, and `$1` is the cutoff and `$2` the batch size:
255
-
256
- ```sql
257
- CREATE INDEX idx_invoice_delivery_work_items__completed_at
258
- ON invoice_delivery_work_items (completed_at, id)
259
- WHERE completed_at IS NOT NULL;
260
-
261
- WITH expired AS (
262
- SELECT id
263
- FROM invoice_delivery_work_items
264
- WHERE completed_at IS NOT NULL AND completed_at < $1
265
- ORDER BY completed_at, id
266
- LIMIT $2
267
- FOR UPDATE SKIP LOCKED
268
- )
269
- DELETE FROM invoice_delivery_work_items AS item
270
- USING expired
271
- WHERE item.id = expired.id;
272
- ```
273
-
274
- Report `hasMore` when the deleted row count equals the batch size, and let the next run continue.
275
-
276
- ## Deduplicate against a large table
277
-
278
- Do not load every existing key to diff against a candidate list. Ask the table about the
279
- candidates only, in chunks whose size is a bounded tunable. The candidates themselves come from a
280
- capped page:
281
-
282
- ```sql
283
- SELECT external_ref
284
- FROM order_imports
285
- WHERE account_id = $1 AND external_ref = ANY($2::text[]);
286
- ```
287
-
288
- ```ts
289
- const known = new Set<string>()
290
- for (const chunk of chunks(candidates, config.lookupChunkSize)) {
291
- const rows = await db.query(findKnownRefsSql, [accountId, chunk])
292
- for (const row of rows) known.add(row.external_ref)
293
- }
294
- const fresh = candidates.filter((ref) => !known.has(ref))
295
- ```
296
-
297
- When the goal is only to insert the new rows, `INSERT ... ON CONFLICT DO NOTHING RETURNING` skips
298
- the lookup.
299
-
300
- ## Per-row interval: one bucket per threshold
301
-
302
- When the rule lives on the row, such as a per-site refresh interval, compare the stored fact with
303
- a bound per distinct threshold value, never with another column. `swept_at` stores when the last
304
- _completed_ sweep started; set it from the sweep's own start time when the sweep reports no more
305
- work. Each interval value is a bucket the index can seek into. A recursive CTE walks the leading
306
- column of the index one distinct value at a time (a loose index scan, or skip scan), so finding
307
- the buckets reads one index entry per bucket, not every enabled site. Each bucket is capped on its
308
- own, so one busy bucket cannot starve the others. Each branch inside a bucket has its own cap too,
309
- so a steady stream of never-swept sites cannot starve overdue rows. A run returns at most
310
- 2 times `$1` rows per bucket:
311
-
312
- ```sql
313
- CREATE INDEX idx_sites__refresh_due
314
- ON sites (refresh_interval_days, swept_at NULLS FIRST, id)
315
- WHERE is_enabled;
316
-
317
- WITH RECURSIVE bucket AS (
318
- (SELECT refresh_interval_days FROM sites
319
- WHERE is_enabled
320
- ORDER BY refresh_interval_days LIMIT 1)
321
- UNION ALL
322
- SELECT next_bucket.refresh_interval_days
323
- FROM bucket
324
- CROSS JOIN LATERAL (
325
- SELECT refresh_interval_days FROM sites
326
- WHERE is_enabled AND refresh_interval_days > bucket.refresh_interval_days
327
- ORDER BY refresh_interval_days LIMIT 1
328
- ) next_bucket
329
- )
330
- SELECT due.id, bucket.refresh_interval_days
331
- FROM bucket
332
- CROSS JOIN LATERAL (
333
- (SELECT id FROM sites
334
- WHERE is_enabled AND refresh_interval_days = bucket.refresh_interval_days
335
- AND swept_at IS NULL
336
- ORDER BY id LIMIT $1)
337
- UNION ALL
338
- (SELECT id FROM sites
339
- WHERE is_enabled AND refresh_interval_days = bucket.refresh_interval_days
340
- AND swept_at < now() - bucket.refresh_interval_days * interval '1 day'
341
- ORDER BY swept_at, id LIMIT $1)
342
- ) due;
343
- ```
344
-
345
- This is a complex shape: keep it only with `EXPLAIN ANALYZE` plans showing it beats the plain
346
- query.
347
-
348
- The loose scan beats a small thresholds table here because the intervals are chosen per row and
349
- there is no second source of truth to drift. If the intervals are a fixed, reviewed set, a
350
- thresholds table is just as bounded. Either way the bucket count must itself be bounded, for
351
- example by a `CHECK` that restricts `refresh_interval_days` to the allowed values. Declare the
352
- column `NOT NULL` with a default as well: a NULL interval is never enumerated by the loose scan
353
- (`>` skips NULL) and never compares as due, so that site would never be swept.
354
-
355
- Comparing `swept_at < now() - refresh_interval_days * interval '1 day'` alone cannot use an index:
356
- the right side depends on the row's own column, and `timestamptz + interval` is not immutable, so
357
- the sum cannot be indexed or stored as a generated column. Inside a bucket the interval is a
358
- constant, so the bound is a plain range on `swept_at`.
359
-
360
- Each condition is its own branch. `swept_at IS NULL OR swept_at < $bound` is not one index range:
361
- the planner makes `refresh_interval_days = k` the index condition and applies the `OR` as a filter.
362
- When fewer rows are due than the cap, that scan reads every row in the bucket. Verify by rows, not
363
- scan type: the plan's actual rows read stay close to the rows returned, and `Rows Removed by
364
- Filter` is near zero.
365
-
366
- ## Age milestones from the UUIDv7 key
367
-
368
- An account crosses age `M` when `now - M` passes its creation time, which the key encodes. So
369
- "crossed `M` since the last run" is a primary-key range with no stored column. `$1` is the previous
370
- run's time minus an overlap (the window start), `$2` the window end, fixed when the window opens,
371
- `$3` the milestone interval, `$4` the last `id` already read (the nil UUID at the start of a
372
- window), and `$5` the cap. This holds only when ids are minted at creation and never for imports
373
- or backfills; where that is not guaranteed, drive milestones from an indexed immutable `created_at`
374
- range instead (see the [parent lower bound](../../postgres-partitioning-uuid-v7/references/partition-lifecycle.md#parent-lower-bound)
375
- conditions on historical ids):
376
-
377
- ```sql
378
- CREATE INDEX idx_accounts__open ON accounts (id) WHERE closed_at IS NULL;
379
-
380
- SELECT id FROM accounts
381
- WHERE id >= min_uuidv7($1::timestamptz - $3::interval)
382
- AND id < min_uuidv7($2::timestamptz - $3::interval)
383
- AND id > $4
384
- AND closed_at IS NULL
385
- ORDER BY id
386
- LIMIT $5;
387
- ```
388
-
389
- Without the partial index, `closed_at IS NULL` is a filter on the primary-key range, and a window
390
- where most accounts are closed reads every one of them. Verify by rows: the plan's rows read stay
391
- close to the rows returned.
392
-
393
- A window can hold more accounts than the cap, so persist the last `id` returned after each batch
394
- and page inside the fixed window with `id > $4`. Advance the window only after a short batch: the
395
- next window starts an overlap before this one's end (`$2`), with the nil UUID as `$4`. Moving the
396
- start to the window's end after a capped batch would skip the accounts not yet read. This is the
397
- `age_milestone` signal's `last_id` in the dispatcher cursor table below.
398
-
399
- ## One dispatcher per signal
400
-
401
- The anti-pattern is one keyset walk over every account with an `OR` of unrelated reasons. No index
402
- serves the `OR`, so every account is read to find the few that need work:
403
-
404
- ```sql
405
- SELECT a.id
406
- FROM accounts a
407
- WHERE a.id > $1
408
- AND (a.id < min_uuidv7(now() - $2::interval)
409
- OR EXISTS (SELECT 1 FROM subscriptions s
410
- WHERE s.account_id = a.id AND s.expires_at < now())
411
- OR EXISTS (SELECT 1 FROM plan_changes c
412
- WHERE c.account_id = a.id AND c.id > $3))
413
- ORDER BY a.id
414
- LIMIT $4;
415
- ```
416
-
417
- Replace it with one dispatcher per signal, each with its own index-served predicate, position, and
418
- schedule. A signal backed by several timestamp columns reads one indexed range per column, joined
419
- with `UNION ALL`, not an `OR` across columns. A window can hold more rows than the cap, so each
420
- dispatcher pages inside it with a composite keyset position, as in the high-water-mark window.
421
- `$1` is the window end, `$2` and `$3` the position (`last_at`, `last_id`), and `$4` the cap:
422
-
423
- ```sql
424
- -- age milestones: the accounts range above, with `id > $3` as the position and
425
- -- `id < min_uuidv7($1 - milestone)` as the window end
426
-
427
- -- subscription expirations: one indexed range per timestamp column, merged by (at, id)
428
- -- (catches only expiries that pass in real time; see the note below)
429
- SELECT at, id, account_id FROM (
430
- (SELECT expires_at AS at, id, account_id FROM subscriptions
431
- WHERE (expires_at, id) > ($2, $3) AND expires_at < $1
432
- ORDER BY expires_at, id LIMIT $4)
433
- UNION ALL
434
- (SELECT grace_ends_at AS at, id, account_id FROM subscriptions
435
- WHERE (grace_ends_at, id) > ($2, $3) AND grace_ends_at < $1
436
- ORDER BY grace_ends_at, id LIMIT $4)
437
- ) either_column
438
- ORDER BY at, id
439
- LIMIT $4;
440
-
441
- -- plan changes: a UUIDv7 window, with `last_id` (seeded at the overlap boundary) as the position
442
- SELECT id, account_id FROM plan_changes
443
- WHERE id > $3 AND id < min_uuidv7($1)
444
- ORDER BY id
445
- LIMIT $4;
446
- ```
447
-
448
- Each dispatcher keeps its window end and position in one cursor table keyed by an enum:
449
-
450
- ```sql
451
- CREATE TYPE account_recalc_signals AS ENUM ('age_milestone', 'subscription_expiry', 'plan_change');
452
-
453
- CREATE TABLE account_recalc_cursors (
454
- signal account_recalc_signals PRIMARY KEY,
455
- changes_through_at timestamptz NOT NULL,
456
- last_at timestamptz NOT NULL,
457
- last_id uuid NOT NULL DEFAULT '00000000-0000-0000-0000-000000000000'
458
- );
459
- ```
460
-
461
- `changes_through_at` is the window end, fixed when the window opens. Advance `(last_at, last_id)`
462
- after each batch with a compare-and-set. When a batch is short, the window is drained: the next
463
- window opens with a new `changes_through_at`, and its position restarts an overlap before the
464
- previous end. A UUIDv7-keyed signal (`plan_change`) seeds `last_id` with
465
- `min_uuidv7(previous end - overlap)`, never the nil UUID, which would rescan from the start of
466
- the table; a timestamp-keyed signal sets `last_at` to the same boundary with the nil `last_id`.
467
-
468
- `expires_at` and `grace_ends_at` are mutable business timestamps, the same trap as using
469
- `occurred_at` as a high-water mark in the
470
- [high-water-mark window](#high-water-mark-window-with-an-overlap). A row updated or imported with
471
- an expiry older than the cursor position is never selected, so the dispatcher catches only
472
- expiries that pass in real time. Any write that sets an expiry at or before now, or before the
473
- cursor, enqueues the job directly in the write path or records an immutable change event
474
- (`plan_changes` is one) that a UUIDv7-keyed dispatcher reads.
475
-
476
- All of them enqueue the same `recalculateAccount` job with the per-event ids below, so a repeat
477
- read is harmless. Writes that cause a signal, such as recording a plan change, enqueue the job in
478
- the write path. The dispatchers are the backstop for a missed enqueue.
479
-
480
- ## Queue job ids for deduplication
481
-
482
- A queue that deduplicates by job id ignores an add whose id already exists, including a job that is
483
- running or retained after completion. An id keyed only by the entity drops a new trigger:
484
-
485
- ```ts
486
- // Account A's job is running when A's subscription expires: this second add is ignored.
487
- await queue.add('recalculateAccount', { accountId }, { jobId: `recalc__${accountId}` })
488
- ```
489
-
490
- Key the id by the entity and the event instead. Re-reading the same event in an overlap gives the
491
- same id, and a new event gets a new id:
492
-
493
- ```ts
494
- await queue.add(
495
- 'recalculateAccount',
496
- { accountId, changeId },
497
- { jobId: `recalc__${accountId}__planChanged__${changeId}` },
498
- )
499
- ```
500
-
501
- While the job record is retained, each trigger is enqueued at most once; that does not mean it
502
- runs. Deduplication lasts only as long as the record: retention must cover the longest overlap or
503
- replay interval, or a re-read event enqueues again after the record is removed. A requirement that
504
- a trigger take effect exactly once needs a durable event ledger, not job ids. If a job fails for
505
- good and its record is retained, the same id is still claimed, so retrying it needs an explicit
506
- reactivate-or-remove step before the next add.
507
-
508
- ## Repair through an existing reconciler
509
-
510
- When a side effect after commit is fire-and-forget (a cache fill, a filter add, an enqueue), repair
511
- it in the reconciler that already reads a `changed_through_at` window of recently changed rows. Do
512
- not create a new sweeper. `$1` and `$2` are the position (`changed_at`, `id`), seeded an overlap
513
- before the previous window's end, `$3` the window end fixed when the window opens, and `$4` the cap:
514
-
515
- ```sql
516
- SELECT id, handle, version
517
- FROM accounts
518
- WHERE (changed_at, id) > ($1, $2) AND changed_at <= $3
519
- ORDER BY changed_at, id
520
- LIMIT $4;
521
- ```
522
-
523
- Advance the position after each batch with a compare-and-set, and open the next window only after a
524
- short batch, as in the dispatcher cursor table.
525
-
526
- ```ts
527
- for (const account of await db.query(recentlyChangedAccountsSql, [
528
- ...position,
529
- to,
530
- config.batchSize,
531
- ])) {
532
- // Monotonic: the write compares `version`, so a delayed repair never overwrites a newer value.
533
- await accountCache.fillIfNotOlder(account.id, account, account.version)
534
- await addHandle(account.handle) // idempotent; the invalidating helper from membership filters
535
- }
536
- ```
537
-
538
- Idempotence alone is not enough, because the overlap reads rows twice and a delayed repair can land
539
- after a newer write. Make each repair monotonic: compare an integer `version` that every update
540
- increments (`version = version + 1`) on write, or invalidate the cache entry and let the next read
541
- load the current row. Do not compare `changed_at = now()`: it is the transaction's start time and
542
- is not monotonic across concurrent transactions.
543
-
544
- ## Membership filters
545
-
546
- A membership filter needs every key, but never on a schedule. The write path adds immediately after
547
- commit. A failed add stops trusting the filter, by clearing its ready marker so reads fall back to
548
- the source of truth, and enqueues a rebuild. The job id includes the filter generation: a fixed id
549
- would stay claimed once its record is retained, so a later failure could never enqueue. Each
550
- invalidation bumps the generation, so concurrent enqueues for one failure share an id and a new
551
- failure gets a new one. The reconciler repairs missed adds from its window, and a read that finds
552
- the filter missing enqueues the same rebuild:
553
-
554
- ```ts
555
- async function enqueueRebuild(generation: number, afterId = 'start'): Promise<void> {
556
- try {
557
- const jobId = `rebuildFilter__handles__${generation}__${afterId}`
558
- await queue.add('rebuildFilter', { generation, afterId }, { jobId })
559
- } catch (error) {
560
- logger.warn({ error }, 'could not enqueue the filter rebuild') // best effort
561
- }
562
- }
563
-
564
- export async function addHandle(handle: string): Promise<void> {
565
- try {
566
- await handleFilter.add(handle)
567
- } catch {
568
- await enqueueRebuild(await handleFilter.invalidate()) // clears ready, bumps the generation
569
- }
570
- }
571
-
572
- export async function hasHandle(handle: string): Promise<boolean> {
573
- if (!(await handleFilter.isReady())) {
574
- await enqueueRebuild(await handleFilter.generation())
575
- return handleExistsInDatabase(handle) // the read falls back even if the enqueue failed
576
- }
577
- // Hint only: a negative means "definitely absent" for keys committed before the last
578
- // successful add or rebuild. A positive may be a false positive, so verify it at the source.
579
- try {
580
- if (!(await handleFilter.has(handle))) return false
581
- } catch {
582
- await enqueueRebuild(await handleFilter.invalidate()) // a failing filter is not trusted
583
- }
584
- return handleExistsInDatabase(handle)
585
- }
586
- ```
587
-
588
- The add happens after commit, so while it is in flight a committed handle can be missing from a
589
- ready filter. The filter is therefore only a hint for callers that tolerate a just-committed miss,
590
- such as an availability pre-check. Correctness-critical paths, such as a uniqueness check or a
591
- read-your-writes lookup, use the database or its unique constraint and never trust a negative.
592
-
593
- The rebuild is an explicit, capped, resumable backfill with a keyset cursor, as in the
594
- high-water-mark window. Each capped page enqueues its continuation with the cursor in the job id
595
- (`rebuildFilter__handles__${generation}__${afterId}`), because a generation-only id would block the
596
- next page of the same generation. Each page checks that its generation is still current before
597
- scanning and again before enqueueing a continuation, and a stale page exits without doing
598
- anything. The rebuild captures its generation when it starts and sets the
599
- ready marker last, by a compare-and-set that succeeds only while that generation is still current
600
- (`markReady(generation)`), so a stale rebuild never overwrites a newer invalidation. Deleted keys and growth past
601
- capacity only cost false positives, so they are an operator's rebuild, not a schedule.
602
-
603
- ## Global ranking
604
-
605
- A result that depends on every row, such as a percentile tier, needs every row's score whatever the
606
- design. Keep one periodic aggregate. `$1` and `$2` are the tier cutoffs:
607
-
608
- ```sql
609
- WITH per_plan AS (
610
- SELECT s.channel_id, s.plan_id, count(*) AS subscribers
611
- FROM subscriptions s
612
- WHERE s.canceled_at IS NULL
613
- GROUP BY s.channel_id, s.plan_id
614
- ), weighted AS (
615
- SELECT p.channel_id, sum(p.subscribers * plan.subscriber_weight) AS score
616
- FROM per_plan p
617
- JOIN plans plan ON plan.id = p.plan_id
618
- GROUP BY p.channel_id
619
- ), scores AS (
620
- -- Start from channels so one with no active subscribers scores 0 instead of dropping out.
621
- SELECT c.id AS channel_id, coalesce(w.score, 0) AS score
622
- FROM channels c
623
- LEFT JOIN weighted w ON w.channel_id = c.id
624
- ), ranked AS (
625
- SELECT channel_id, PERCENT_RANK() OVER (ORDER BY score) AS percentile, score
626
- FROM scores
627
- )
628
- SELECT channel_id,
629
- CASE WHEN score = 0 THEN 'low'
630
- WHEN percentile >= $1 THEN 'top'
631
- WHEN percentile >= $2 THEN 'mid'
632
- ELSE 'low' END AS tier
633
- FROM ranked;
634
- ```
635
-
636
- `PERCENT_RANK()` gives tied scores the lowest rank, so a large tie at score 0 stays at percentile 0
637
- instead of reading as a high cumulative fraction, and score 0 is forced into the low tier. Write
638
- back only the changed tiers, in capped keyset batches over the result. Keeping a subscriber
639
- count current on every subscribe and unsubscribe is more total work: it makes one hot `channels`
640
- row take every write, and a plan change fans out to every channel the account follows. It does not
641
- make tiers fresher either, because the percentile still needs every channel's score. Give the one
642
- aggregate pass an EXPLAIN budget at representative size.
643
-
644
- ## External listing with a continuation token
645
-
646
- An object-storage or API listing loop caps the pages per run. When more remain, it enqueues a
647
- continuation job carrying the provider's continuation token. Restarting from page one each run is
648
- not a bound: items that keep failing at the front starve the rest.
649
-
650
- ```ts
651
- export async function listingJob(
652
- { token, traversalId = randomUUID() }: { token?: string; traversalId?: string },
653
- config: ListingConfig,
654
- ): Promise<void> {
655
- let next = token // traversalId is minted when a traversal starts at page one
656
- for (let page = 0; page < config.maxPagesPerRun; page += 1) {
657
- const result = await provider.list({ continuationToken: next, maxKeys: config.pageSize })
658
- await enqueueItems(result.items)
659
- if (result.nextToken !== undefined && result.nextToken === next) {
660
- throw new Error('Provider repeated its continuation token') // never loop or re-enqueue it
661
- }
662
- next = result.nextToken
663
- if (!next) return
664
- }
665
- // The traversal id keeps a provider that reuses a token across traversals from colliding.
666
- await queue.add(
667
- 'listing',
668
- { token: next, traversalId },
669
- { jobId: `listing__${traversalId}__${next}` },
670
- )
671
- }
672
- ```
673
-
674
- ## Verify with a plan
675
-
676
- Run each statement with representative parameters against representative table sizes, because a
677
- small table scans sequentially for good reason. The `due` query of the claim above should plan as
678
- a `Limit` over a `LockRows` over an index scan on the partial index, with no `Sort` between the
679
- scan and the `Limit`. Read the rows each node returns, not only the scan type: an index scan that
680
- feeds a `Sort` below the `Limit` still reads every match. A `Seq Scan` on a table that grows is
681
- also a finding:
682
-
683
- ```sql
684
- PREPARE claim_due (integer) AS
685
- SELECT id
686
- FROM invoice_delivery_work_items
687
- WHERE completed_at IS NULL AND available_at <= now()
688
- ORDER BY available_at, id
689
- LIMIT $1
690
- FOR UPDATE SKIP LOCKED;
691
-
692
- BEGIN;
693
- EXPLAIN (ANALYZE, BUFFERS) EXECUTE claim_due (200);
694
- ROLLBACK;
695
- ```
696
-
697
- `EXPLAIN ANALYZE` runs the statement, so check the claim `UPDATE` or the `DELETE` inside a
698
- transaction that you roll back, or on a disposable copy. Under contention, `SKIP LOCKED` passes over rows other claimers hold, and those rows sit under the
699
- `Limit`, so the scan reads the cap plus the rows concurrent claims hold. That stays bounded by the
700
- number of workers times the batch size when claims are short, so keep the claim transaction to the
701
- claim statement, and check the index scan's actual rows while several workers claim at once.
702
-
703
- ## Anti-patterns
704
-
705
- | Anti-pattern | Why it is unbounded | Bounded shape |
706
- | ----------------------------------------------------------- | ------------------------------------------------------------------------ | ------------------------------------------------------------ |
707
- | Loop until empty: `while (true)` claim, process, break on 0 | New and failing rows keep the run alive; the run has no end | A batch or deadline cap, `hasMore`, and a continuation |
708
- | A stream with no cap: `for await` over every account | Memory stays flat while run time and database work grow with the table | Stream a keyset window and cap the rows per run |
709
- | A key-only walk: select every account id, then check each | Touches every entity to find the few with work | Read work items, dirty markers, or a due timestamp |
710
- | A literal `LIMIT 500` in SQL text | Cannot be tuned per environment or during an incident | A validated tunable with a hard maximum, bound as `LIMIT $1` |
711
- | `Promise.all(accounts.map(sync))` over a result of any size | One task per row exhausts the pool or the upstream API | A bounded batch with bounded concurrency |
712
- | An `OR` across unrelated signals in one walk | No index serves the `OR`, so every account is read | One dispatcher per signal, each with its own range |
713
- | Comparing a fact column with a per-row interval column | The sum is not indexable, so every row is read | One bucket per interval value, bounded by a plain range |
714
- | Job ids keyed only by entity | A running or retained job silently drops a new trigger | An id keyed by entity and triggering event |
715
- | A new sweeper beside an existing reconciler | A second pass over rows the reconciler already reads | Add the idempotent repair to the existing reconciler |
716
- | A scheduled membership filter rebuild | Reads every key on every run | Rebuild on operator action or detected failure only |
717
- | Restarting an external listing at page one | Failing items at the front starve the rest | A continuation job carrying the provider's token |
718
- | `OFFSET` pagination | Every page rescans and discards the rows before it | Keyset on the indexed key |
719
- | `UPDATE` or `DELETE` with a predicate but no `LIMIT` | One statement locks and rewrites an unbounded row set in one transaction | A CTE that selects a capped, `SKIP LOCKED` id set |