vouchington-tooling 2.0.2 → 3.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/package.json +2 -2
- package/scripts/__pycache__/run_bounded_process.cpython-314.pyc +0 -0
- package/skills/manifest.json +15 -202
- package/skills/organize-github-issues/SKILL.md +20 -3
- package/skills/pr-description/SKILL.md +9 -65
- package/skills/retrospective/SKILL.md +15 -0
- package/skills/retrospective-distill/SKILL.md +1 -1
- package/skills/retrospective-distill/references/issue-body.md +6 -1
- package/skills/review-github-issue-taxonomy/SKILL.md +3 -4
- package/skills/triage-loop/SKILL.md +107 -26
- package/skills/triage-prs/SKILL.md +25 -27
- package/skills/triage-prs/references/feedback.md +14 -2
- package/skills/triage-prs/references/rubric.md +4 -4
- package/skills/triage-sweep/SKILL.md +26 -23
- package/skills/agent-workflow/SKILL.md +0 -72
- package/skills/agent-workflow/references/evidence-sweep.md +0 -12
- package/skills/agent-workflow/references/four-refs.md +0 -74
- package/skills/agent-workflow/references/implementation-and-review.md +0 -4
- package/skills/agent-workflow/references/implementation.md +0 -20
- package/skills/agent-workflow/references/review-response.md +0 -49
- package/skills/agent-workflow/references/review.md +0 -10
- package/skills/backend-vitest-test-authoring/SKILL.md +0 -22
- package/skills/backend-vitest-test-authoring/references/integration-boundaries.md +0 -10
- package/skills/bounded-iteration/SKILL.md +0 -92
- package/skills/bounded-iteration/references/sweep-patterns.md +0 -719
- package/skills/dependabot/SKILL.md +0 -117
- package/skills/dotnet-test-authoring/SKILL.md +0 -24
- package/skills/git-commit-checklist/SKILL.md +0 -29
- package/skills/github-actions-authoring/SKILL.md +0 -63
- package/skills/github-actions-checklist/SKILL.md +0 -118
- package/skills/github-issue/SKILL.md +0 -121
- package/skills/glide-mq/SKILL.md +0 -52
- package/skills/glide-mq/references/lease-patterns.md +0 -44
- package/skills/nextjs-vitest-test-authoring/SKILL.md +0 -21
- package/skills/nextjs-vitest-test-authoring/references/framework-boundaries.md +0 -9
- package/skills/npm-publishing/SKILL.md +0 -72
- package/skills/package-json-checklist/SKILL.md +0 -31
- package/skills/planning/SKILL.md +0 -37
- package/skills/planning/references/impact-discovery.md +0 -11
- package/skills/playwright-authoring/SKILL.md +0 -23
- package/skills/playwright-authoring/references/browser-reliability.md +0 -9
- package/skills/postgres-node-performance-tuning/SKILL.md +0 -35
- package/skills/postgres-node-performance-tuning/references/performance-patterns.md +0 -175
- package/skills/postgres-node-performance-tuning/references/reading-plans.md +0 -111
- package/skills/postgres-partitioning-uuid-v7/SKILL.md +0 -35
- package/skills/postgres-partitioning-uuid-v7/references/partition-lifecycle.md +0 -53
- package/skills/postgres-schema-design/SKILL.md +0 -31
- package/skills/postgres-schema-design/references/naming.md +0 -34
- package/skills/postgres-schema-design/references/relations.md +0 -66
- package/skills/postgres-schema-design/references/table-shapes.md +0 -78
- package/skills/pr-description/references/examples.md +0 -189
- package/skills/review-ci-logs/SKILL.md +0 -44
- package/skills/revisit-followups/SKILL.md +0 -47
- package/skills/stacked-prs/SKILL.md +0 -98
- package/skills/stacked-prs/references/drain.md +0 -177
- package/skills/stacked-prs/references/gh-stack.md +0 -147
- package/skills/static-analysis-checklist/SKILL.md +0 -32
- package/skills/storybook-authoring/SKILL.md +0 -22
- package/skills/storybook-authoring/references/component-coverage.md +0 -9
- package/skills/swift-test-authoring/SKILL.md +0 -23
- package/skills/swift-test-authoring/references/network-test-doubles.md +0 -9
- package/skills/test-authoring/SKILL.md +0 -27
- package/skills/test-authoring/references/core-practice.md +0 -16
- package/skills/test-authoring/references/dependency-boundaries.md +0 -38
- package/skills/test-authoring/references/tautological-tests.md +0 -47
- package/skills/triage-loop/references/shepherd.md +0 -64
- package/skills/vitest-test-authoring/SKILL.md +0 -22
- package/skills/vitest-test-authoring/references/mock-boundaries.md +0 -9
|
@@ -1,719 +0,0 @@
|
|
|
1
|
-
# Sweep patterns
|
|
2
|
-
|
|
3
|
-
Synthetic examples for [bounded iteration](../SKILL.md). `db.query` stands for the consumer's
|
|
4
|
-
executor and `config` for its runtime configuration; every cap in SQL is a bound parameter.
|
|
5
|
-
`min_uuidv7(timestamptz)` stands for the consumer's helper that returns the smallest UUIDv7 for a
|
|
6
|
-
timestamp, like the lower-bound helper in
|
|
7
|
-
[performance patterns](../../postgres-node-performance-tuning/references/performance-patterns.md).
|
|
8
|
-
Comments on tables and columns are omitted for space.
|
|
9
|
-
|
|
10
|
-
## Due timestamp, partial index, capped claim
|
|
11
|
-
|
|
12
|
-
A work-item row exists only while work is pending, so the index covers only rows a claim can pick:
|
|
13
|
-
|
|
14
|
-
```sql
|
|
15
|
-
CREATE INDEX idx_invoice_delivery_work_items__available_at
|
|
16
|
-
ON invoice_delivery_work_items (available_at, id)
|
|
17
|
-
WHERE completed_at IS NULL;
|
|
18
|
-
```
|
|
19
|
-
|
|
20
|
-
One claim takes at most `$1` due rows. Competing workers skip rows another worker holds, and the
|
|
21
|
-
lease moves `available_at` to its expiry, so an abandoned claim is due again through the same index
|
|
22
|
-
without an `OR` in the predicate:
|
|
23
|
-
|
|
24
|
-
```sql
|
|
25
|
-
WITH due AS (
|
|
26
|
-
SELECT id
|
|
27
|
-
FROM invoice_delivery_work_items
|
|
28
|
-
WHERE completed_at IS NULL AND available_at <= now()
|
|
29
|
-
ORDER BY available_at, id
|
|
30
|
-
LIMIT $1
|
|
31
|
-
FOR UPDATE SKIP LOCKED
|
|
32
|
-
)
|
|
33
|
-
UPDATE invoice_delivery_work_items AS item
|
|
34
|
-
SET lease_token = gen_random_uuid(),
|
|
35
|
-
leased_at = now(),
|
|
36
|
-
lease_expires_at = now() + $2::interval,
|
|
37
|
-
available_at = now() + $2::interval,
|
|
38
|
-
attempt_count = item.attempt_count + 1
|
|
39
|
-
FROM due
|
|
40
|
-
WHERE item.id = due.id
|
|
41
|
-
RETURNING item.id, item.invoice_id, item.lease_token;
|
|
42
|
-
```
|
|
43
|
-
|
|
44
|
-
Renewal moves `lease_expires_at` and `available_at` to the same new deadline, so a renewed lease
|
|
45
|
-
is not due. Renewal, completion, and failure all add `AND lease_token = $mine`; see
|
|
46
|
-
[table shapes](../../postgres-schema-design/references/table-shapes.md).
|
|
47
|
-
|
|
48
|
-
## Dirty marker or work item
|
|
49
|
-
|
|
50
|
-
Do not walk every order to find the few that need reconciling. The writer that changes an order
|
|
51
|
-
marks it in the same transaction, and the reconciler claims only marks, as above:
|
|
52
|
-
|
|
53
|
-
```sql
|
|
54
|
-
CREATE TABLE order_reconcile_work_items (
|
|
55
|
-
id uuid PRIMARY KEY DEFAULT uuidv7(),
|
|
56
|
-
order_id uuid NOT NULL UNIQUE REFERENCES orders (id) ON DELETE CASCADE,
|
|
57
|
-
available_at timestamptz NOT NULL DEFAULT now(),
|
|
58
|
-
lease_token uuid,
|
|
59
|
-
leased_at timestamptz,
|
|
60
|
-
lease_expires_at timestamptz,
|
|
61
|
-
attempt_count integer NOT NULL DEFAULT 0,
|
|
62
|
-
generation integer NOT NULL DEFAULT 0,
|
|
63
|
-
completed_at timestamptz
|
|
64
|
-
);
|
|
65
|
-
|
|
66
|
-
INSERT INTO order_reconcile_work_items (order_id)
|
|
67
|
-
VALUES ($1)
|
|
68
|
-
ON CONFLICT (order_id) DO UPDATE
|
|
69
|
-
SET available_at = GREATEST(order_reconcile_work_items.available_at, now()),
|
|
70
|
-
completed_at = NULL,
|
|
71
|
-
generation = order_reconcile_work_items.generation + 1;
|
|
72
|
-
```
|
|
73
|
-
|
|
74
|
-
The claim above applies with `order_id` for `invoice_id`, and its `RETURNING` also includes
|
|
75
|
-
`generation`, so `$claimed` is captured in the same statement as the lease. Completion adds
|
|
76
|
-
`AND generation = $claimed`. A re-mark never makes a leased row due early, since
|
|
77
|
-
the claim moved `available_at` to the lease expiry. A completion that finds a newer generation
|
|
78
|
-
releases the lease with `available_at = now()`, and the next claim takes the row again. Run cost
|
|
79
|
-
follows the number of changes, not the number of orders. A job fanned out per account reads by
|
|
80
|
-
parent id, `WHERE account_id = $1`, for the one account it was started for.
|
|
81
|
-
|
|
82
|
-
## High-water-mark window with an overlap
|
|
83
|
-
|
|
84
|
-
Prefer keeping the position and the sweep's fixed end in job data for a single-writer continuation.
|
|
85
|
-
Use a persisted cursor table only when the position must be transactional with domain rows, as
|
|
86
|
-
here.
|
|
87
|
-
|
|
88
|
-
An export reads `invoices` added since the last sweep. The cursor row holds the position and the
|
|
89
|
-
sweep's fixed end, and the job seeds its own row:
|
|
90
|
-
|
|
91
|
-
```sql
|
|
92
|
-
CREATE TABLE invoice_export_cursors (
|
|
93
|
-
is_singleton boolean PRIMARY KEY DEFAULT true CHECK (is_singleton),
|
|
94
|
-
cursor_invoice_id uuid,
|
|
95
|
-
sweep_upper_bound_invoice_id uuid
|
|
96
|
-
);
|
|
97
|
-
|
|
98
|
-
INSERT INTO invoice_export_cursors DEFAULT VALUES ON CONFLICT DO NOTHING;
|
|
99
|
-
```
|
|
100
|
-
|
|
101
|
-
A UUIDv7 is assigned at insert, not commit, so a row below the last sweep's end can commit after
|
|
102
|
-
that sweep passed it. A new sweep starts an overlap before the previous end and fixes its own end
|
|
103
|
-
now, only when the previous sweep reached its end. `$1` is the overlap, and NULL means the start,
|
|
104
|
-
so the first sweep begins at the start. The overlap must exceed the longest insert-to-commit
|
|
105
|
-
delay, which the consumer enforces, for example with a transaction timeout; a row that commits
|
|
106
|
-
after the overlap has passed is missed for good. Work that cannot tolerate a miss writes a
|
|
107
|
-
work-item row in the same transaction instead:
|
|
108
|
-
|
|
109
|
-
```sql
|
|
110
|
-
UPDATE invoice_export_cursors
|
|
111
|
-
SET cursor_invoice_id =
|
|
112
|
-
min_uuidv7(uuid_extract_timestamp(sweep_upper_bound_invoice_id) - $1::interval),
|
|
113
|
-
sweep_upper_bound_invoice_id = min_uuidv7(now())
|
|
114
|
-
WHERE is_singleton
|
|
115
|
-
AND cursor_invoice_id IS NOT DISTINCT FROM sweep_upper_bound_invoice_id;
|
|
116
|
-
```
|
|
117
|
-
|
|
118
|
-
Each batch moves strictly forward inside the sweep, where `$1` is the cursor (the nil UUID for the
|
|
119
|
-
start), `$2` the sweep's end, and `$3` the batch size:
|
|
120
|
-
|
|
121
|
-
```sql
|
|
122
|
-
SELECT id, account_id, total_cents
|
|
123
|
-
FROM invoices
|
|
124
|
-
WHERE id > $1 AND id <= $2
|
|
125
|
-
ORDER BY id
|
|
126
|
-
LIMIT $3;
|
|
127
|
-
```
|
|
128
|
-
|
|
129
|
-
Persist the last `id` as the cursor after each batch with a compare-and-set, so a stale or
|
|
130
|
-
overlapping worker cannot move the position backward or into another sweep. `$1` is the new
|
|
131
|
-
position, `$2` the cursor this batch started from (NULL at the start), and `$3` the sweep's end.
|
|
132
|
-
Zero updated rows means another worker moved on, so the run stops:
|
|
133
|
-
|
|
134
|
-
```sql
|
|
135
|
-
UPDATE invoice_export_cursors
|
|
136
|
-
SET cursor_invoice_id = $1
|
|
137
|
-
WHERE is_singleton
|
|
138
|
-
AND cursor_invoice_id IS NOT DISTINCT FROM $2::uuid
|
|
139
|
-
AND sweep_upper_bound_invoice_id = $3;
|
|
140
|
-
```
|
|
141
|
-
|
|
142
|
-
A short batch ends the sweep: set the cursor to the sweep's end the same way, and the next run
|
|
143
|
-
starts a new sweep. Rows in the overlap are read twice, so the export must be idempotent.
|
|
144
|
-
|
|
145
|
-
A time position works the same way on a column the database sets at insert, such as
|
|
146
|
-
`received_at timestamptz NOT NULL DEFAULT clock_timestamp()`, with the same overlap rule. Never use a business
|
|
147
|
-
time such as `occurred_at`: a late-arriving row carries an old timestamp and falls behind every
|
|
148
|
-
later window. Avoid `now()` too, because it records the transaction's start, not the insert. Fix the window's end when the window opens, keep a composite `(time, id)` keyset
|
|
149
|
-
position inside it, and read
|
|
150
|
-
`WHERE (received_at, id) > ($1, $2) AND received_at <= $3 ORDER BY received_at, id`, where `$3` is
|
|
151
|
-
the window's end. Rows that share a timestamp are neither skipped nor repeated, and new arrivals
|
|
152
|
-
wait for the next window. Advance `<verb>_through_at` to the window's end only after the window is
|
|
153
|
-
drained, and start the next window an overlap earlier.
|
|
154
|
-
|
|
155
|
-
## Capped batch loop with `hasMore`
|
|
156
|
-
|
|
157
|
-
The run stops at a batch cap or a deadline and says whether work remains. A short batch is the
|
|
158
|
-
only proof that nothing is left. Rows a `SKIP LOCKED` claim passed over are held by another
|
|
159
|
-
claimer, who owns them; if that claim rolls back, they are due again on the next scheduled run,
|
|
160
|
-
so a short batch still ends this run's backlog. A cap that stops the run reports
|
|
161
|
-
`hasMore: true`, even when the last batch happened to drain the work; the continuation finds
|
|
162
|
-
nothing and stops:
|
|
163
|
-
|
|
164
|
-
```ts
|
|
165
|
-
interface RunResult {
|
|
166
|
-
processed: number
|
|
167
|
-
hasMore: boolean
|
|
168
|
-
}
|
|
169
|
-
|
|
170
|
-
export async function runOrderReconcile(config: ReconcileConfig): Promise<RunResult> {
|
|
171
|
-
const deadline = Date.now() + config.runDeadlineMs
|
|
172
|
-
let processed = 0
|
|
173
|
-
for (let batch = 0; batch < config.maxBatchesPerRun; batch += 1) {
|
|
174
|
-
if (Date.now() >= deadline) return { processed, hasMore: true }
|
|
175
|
-
const items = await claimDueItems(config.batchSize, config.leaseInterval)
|
|
176
|
-
await reconcileOrders(items)
|
|
177
|
-
processed += items.length
|
|
178
|
-
if (items.length < config.batchSize) return { processed, hasMore: false }
|
|
179
|
-
}
|
|
180
|
-
return { processed, hasMore: true }
|
|
181
|
-
}
|
|
182
|
-
|
|
183
|
-
export async function orderReconcileJob(config: ReconcileConfig): Promise<void> {
|
|
184
|
-
const { hasMore } = await runOrderReconcile(config)
|
|
185
|
-
if (hasMore) await enqueueContinuation('order-reconcile')
|
|
186
|
-
}
|
|
187
|
-
```
|
|
188
|
-
|
|
189
|
-
The persisted position is the work-item table itself: claimed and completed rows leave the due
|
|
190
|
-
set. A cursor sweep persists its position after each batch. The job asks for a continuation
|
|
191
|
-
instead of looping on.
|
|
192
|
-
|
|
193
|
-
## Tunables
|
|
194
|
-
|
|
195
|
-
Read each cap from runtime configuration with a validated default and a hard maximum. A bad value
|
|
196
|
-
fails at startup, and no cap is unset or infinite:
|
|
197
|
-
|
|
198
|
-
```ts
|
|
199
|
-
interface Bound {
|
|
200
|
-
default: number
|
|
201
|
-
max: number
|
|
202
|
-
}
|
|
203
|
-
|
|
204
|
-
export function readBound(name: string, raw: string | undefined, bound: Bound): number {
|
|
205
|
-
const value = raw === undefined ? bound.default : Number(raw)
|
|
206
|
-
if (!Number.isInteger(value) || value < 1 || value > bound.max) {
|
|
207
|
-
throw new RangeError(`${name} must be an integer from 1 to ${bound.max}`)
|
|
208
|
-
}
|
|
209
|
-
return value
|
|
210
|
-
}
|
|
211
|
-
|
|
212
|
-
const batchSize = readBound('ORDER_RECONCILE_BATCH_SIZE', env.ORDER_RECONCILE_BATCH_SIZE, {
|
|
213
|
-
default: 200,
|
|
214
|
-
max: 1000,
|
|
215
|
-
})
|
|
216
|
-
```
|
|
217
|
-
|
|
218
|
-
Bind the value as a parameter, `LIMIT $1`, never a literal in SQL text. `LIMIT 1` for a single-row
|
|
219
|
-
lookup and an external API's documented page maximum are contracts, not tunables.
|
|
220
|
-
|
|
221
|
-
## List endpoint
|
|
222
|
-
|
|
223
|
-
The API schema keeps a static ceiling. The runtime default and effective maximum are tunable but
|
|
224
|
-
never above it, and the server rejects an invalid size and clamps a large one:
|
|
225
|
-
|
|
226
|
-
```ts
|
|
227
|
-
const API_MAX_PAGE_SIZE = 100
|
|
228
|
-
|
|
229
|
-
export function pageSize(requested: number | undefined, config: PageConfig): number {
|
|
230
|
-
const ceiling = Math.min(config.maxPageSize, API_MAX_PAGE_SIZE)
|
|
231
|
-
const value = requested ?? config.defaultPageSize
|
|
232
|
-
if (!Number.isInteger(value) || value < 1) {
|
|
233
|
-
throw new RangeError('Page size must be a positive integer')
|
|
234
|
-
}
|
|
235
|
-
return Math.min(value, ceiling)
|
|
236
|
-
}
|
|
237
|
-
```
|
|
238
|
-
|
|
239
|
-
Page by keyset on the key the index serves. Fetch `pageSize + 1` rows to learn whether another page
|
|
240
|
-
exists, return `pageSize`, and encode the last returned `id` as the next cursor. The first page
|
|
241
|
-
omits `id < $2`:
|
|
242
|
-
|
|
243
|
-
```sql
|
|
244
|
-
SELECT id, total_cents
|
|
245
|
-
FROM orders
|
|
246
|
-
WHERE account_id = $1 AND id < $2
|
|
247
|
-
ORDER BY id DESC
|
|
248
|
-
LIMIT $3;
|
|
249
|
-
```
|
|
250
|
-
|
|
251
|
-
## Bounded DELETE through a CTE
|
|
252
|
-
|
|
253
|
-
DELETE has no `LIMIT`, so select the ids in a CTE and delete by join. The partial index serves the
|
|
254
|
-
retention predicate, and `$1` is the cutoff and `$2` the batch size:
|
|
255
|
-
|
|
256
|
-
```sql
|
|
257
|
-
CREATE INDEX idx_invoice_delivery_work_items__completed_at
|
|
258
|
-
ON invoice_delivery_work_items (completed_at, id)
|
|
259
|
-
WHERE completed_at IS NOT NULL;
|
|
260
|
-
|
|
261
|
-
WITH expired AS (
|
|
262
|
-
SELECT id
|
|
263
|
-
FROM invoice_delivery_work_items
|
|
264
|
-
WHERE completed_at IS NOT NULL AND completed_at < $1
|
|
265
|
-
ORDER BY completed_at, id
|
|
266
|
-
LIMIT $2
|
|
267
|
-
FOR UPDATE SKIP LOCKED
|
|
268
|
-
)
|
|
269
|
-
DELETE FROM invoice_delivery_work_items AS item
|
|
270
|
-
USING expired
|
|
271
|
-
WHERE item.id = expired.id;
|
|
272
|
-
```
|
|
273
|
-
|
|
274
|
-
Report `hasMore` when the deleted row count equals the batch size, and let the next run continue.
|
|
275
|
-
|
|
276
|
-
## Deduplicate against a large table
|
|
277
|
-
|
|
278
|
-
Do not load every existing key to diff against a candidate list. Ask the table about the
|
|
279
|
-
candidates only, in chunks whose size is a bounded tunable. The candidates themselves come from a
|
|
280
|
-
capped page:
|
|
281
|
-
|
|
282
|
-
```sql
|
|
283
|
-
SELECT external_ref
|
|
284
|
-
FROM order_imports
|
|
285
|
-
WHERE account_id = $1 AND external_ref = ANY($2::text[]);
|
|
286
|
-
```
|
|
287
|
-
|
|
288
|
-
```ts
|
|
289
|
-
const known = new Set<string>()
|
|
290
|
-
for (const chunk of chunks(candidates, config.lookupChunkSize)) {
|
|
291
|
-
const rows = await db.query(findKnownRefsSql, [accountId, chunk])
|
|
292
|
-
for (const row of rows) known.add(row.external_ref)
|
|
293
|
-
}
|
|
294
|
-
const fresh = candidates.filter((ref) => !known.has(ref))
|
|
295
|
-
```
|
|
296
|
-
|
|
297
|
-
When the goal is only to insert the new rows, `INSERT ... ON CONFLICT DO NOTHING RETURNING` skips
|
|
298
|
-
the lookup.
|
|
299
|
-
|
|
300
|
-
## Per-row interval: one bucket per threshold
|
|
301
|
-
|
|
302
|
-
When the rule lives on the row, such as a per-site refresh interval, compare the stored fact with
|
|
303
|
-
a bound per distinct threshold value, never with another column. `swept_at` stores when the last
|
|
304
|
-
_completed_ sweep started; set it from the sweep's own start time when the sweep reports no more
|
|
305
|
-
work. Each interval value is a bucket the index can seek into. A recursive CTE walks the leading
|
|
306
|
-
column of the index one distinct value at a time (a loose index scan, or skip scan), so finding
|
|
307
|
-
the buckets reads one index entry per bucket, not every enabled site. Each bucket is capped on its
|
|
308
|
-
own, so one busy bucket cannot starve the others. Each branch inside a bucket has its own cap too,
|
|
309
|
-
so a steady stream of never-swept sites cannot starve overdue rows. A run returns at most
|
|
310
|
-
2 times `$1` rows per bucket:
|
|
311
|
-
|
|
312
|
-
```sql
|
|
313
|
-
CREATE INDEX idx_sites__refresh_due
|
|
314
|
-
ON sites (refresh_interval_days, swept_at NULLS FIRST, id)
|
|
315
|
-
WHERE is_enabled;
|
|
316
|
-
|
|
317
|
-
WITH RECURSIVE bucket AS (
|
|
318
|
-
(SELECT refresh_interval_days FROM sites
|
|
319
|
-
WHERE is_enabled
|
|
320
|
-
ORDER BY refresh_interval_days LIMIT 1)
|
|
321
|
-
UNION ALL
|
|
322
|
-
SELECT next_bucket.refresh_interval_days
|
|
323
|
-
FROM bucket
|
|
324
|
-
CROSS JOIN LATERAL (
|
|
325
|
-
SELECT refresh_interval_days FROM sites
|
|
326
|
-
WHERE is_enabled AND refresh_interval_days > bucket.refresh_interval_days
|
|
327
|
-
ORDER BY refresh_interval_days LIMIT 1
|
|
328
|
-
) next_bucket
|
|
329
|
-
)
|
|
330
|
-
SELECT due.id, bucket.refresh_interval_days
|
|
331
|
-
FROM bucket
|
|
332
|
-
CROSS JOIN LATERAL (
|
|
333
|
-
(SELECT id FROM sites
|
|
334
|
-
WHERE is_enabled AND refresh_interval_days = bucket.refresh_interval_days
|
|
335
|
-
AND swept_at IS NULL
|
|
336
|
-
ORDER BY id LIMIT $1)
|
|
337
|
-
UNION ALL
|
|
338
|
-
(SELECT id FROM sites
|
|
339
|
-
WHERE is_enabled AND refresh_interval_days = bucket.refresh_interval_days
|
|
340
|
-
AND swept_at < now() - bucket.refresh_interval_days * interval '1 day'
|
|
341
|
-
ORDER BY swept_at, id LIMIT $1)
|
|
342
|
-
) due;
|
|
343
|
-
```
|
|
344
|
-
|
|
345
|
-
This is a complex shape: keep it only with `EXPLAIN ANALYZE` plans showing it beats the plain
|
|
346
|
-
query.
|
|
347
|
-
|
|
348
|
-
The loose scan beats a small thresholds table here because the intervals are chosen per row and
|
|
349
|
-
there is no second source of truth to drift. If the intervals are a fixed, reviewed set, a
|
|
350
|
-
thresholds table is just as bounded. Either way the bucket count must itself be bounded, for
|
|
351
|
-
example by a `CHECK` that restricts `refresh_interval_days` to the allowed values. Declare the
|
|
352
|
-
column `NOT NULL` with a default as well: a NULL interval is never enumerated by the loose scan
|
|
353
|
-
(`>` skips NULL) and never compares as due, so that site would never be swept.
|
|
354
|
-
|
|
355
|
-
Comparing `swept_at < now() - refresh_interval_days * interval '1 day'` alone cannot use an index:
|
|
356
|
-
the right side depends on the row's own column, and `timestamptz + interval` is not immutable, so
|
|
357
|
-
the sum cannot be indexed or stored as a generated column. Inside a bucket the interval is a
|
|
358
|
-
constant, so the bound is a plain range on `swept_at`.
|
|
359
|
-
|
|
360
|
-
Each condition is its own branch. `swept_at IS NULL OR swept_at < $bound` is not one index range:
|
|
361
|
-
the planner makes `refresh_interval_days = k` the index condition and applies the `OR` as a filter.
|
|
362
|
-
When fewer rows are due than the cap, that scan reads every row in the bucket. Verify by rows, not
|
|
363
|
-
scan type: the plan's actual rows read stay close to the rows returned, and `Rows Removed by
|
|
364
|
-
Filter` is near zero.
|
|
365
|
-
|
|
366
|
-
## Age milestones from the UUIDv7 key
|
|
367
|
-
|
|
368
|
-
An account crosses age `M` when `now - M` passes its creation time, which the key encodes. So
|
|
369
|
-
"crossed `M` since the last run" is a primary-key range with no stored column. `$1` is the previous
|
|
370
|
-
run's time minus an overlap (the window start), `$2` the window end, fixed when the window opens,
|
|
371
|
-
`$3` the milestone interval, `$4` the last `id` already read (the nil UUID at the start of a
|
|
372
|
-
window), and `$5` the cap. This holds only when ids are minted at creation and never for imports
|
|
373
|
-
or backfills; where that is not guaranteed, drive milestones from an indexed immutable `created_at`
|
|
374
|
-
range instead (see the [parent lower bound](../../postgres-partitioning-uuid-v7/references/partition-lifecycle.md#parent-lower-bound)
|
|
375
|
-
conditions on historical ids):
|
|
376
|
-
|
|
377
|
-
```sql
|
|
378
|
-
CREATE INDEX idx_accounts__open ON accounts (id) WHERE closed_at IS NULL;
|
|
379
|
-
|
|
380
|
-
SELECT id FROM accounts
|
|
381
|
-
WHERE id >= min_uuidv7($1::timestamptz - $3::interval)
|
|
382
|
-
AND id < min_uuidv7($2::timestamptz - $3::interval)
|
|
383
|
-
AND id > $4
|
|
384
|
-
AND closed_at IS NULL
|
|
385
|
-
ORDER BY id
|
|
386
|
-
LIMIT $5;
|
|
387
|
-
```
|
|
388
|
-
|
|
389
|
-
Without the partial index, `closed_at IS NULL` is a filter on the primary-key range, and a window
|
|
390
|
-
where most accounts are closed reads every one of them. Verify by rows: the plan's rows read stay
|
|
391
|
-
close to the rows returned.
|
|
392
|
-
|
|
393
|
-
A window can hold more accounts than the cap, so persist the last `id` returned after each batch
|
|
394
|
-
and page inside the fixed window with `id > $4`. Advance the window only after a short batch: the
|
|
395
|
-
next window starts an overlap before this one's end (`$2`), with the nil UUID as `$4`. Moving the
|
|
396
|
-
start to the window's end after a capped batch would skip the accounts not yet read. This is the
|
|
397
|
-
`age_milestone` signal's `last_id` in the dispatcher cursor table below.
|
|
398
|
-
|
|
399
|
-
## One dispatcher per signal
|
|
400
|
-
|
|
401
|
-
The anti-pattern is one keyset walk over every account with an `OR` of unrelated reasons. No index
|
|
402
|
-
serves the `OR`, so every account is read to find the few that need work:
|
|
403
|
-
|
|
404
|
-
```sql
|
|
405
|
-
SELECT a.id
|
|
406
|
-
FROM accounts a
|
|
407
|
-
WHERE a.id > $1
|
|
408
|
-
AND (a.id < min_uuidv7(now() - $2::interval)
|
|
409
|
-
OR EXISTS (SELECT 1 FROM subscriptions s
|
|
410
|
-
WHERE s.account_id = a.id AND s.expires_at < now())
|
|
411
|
-
OR EXISTS (SELECT 1 FROM plan_changes c
|
|
412
|
-
WHERE c.account_id = a.id AND c.id > $3))
|
|
413
|
-
ORDER BY a.id
|
|
414
|
-
LIMIT $4;
|
|
415
|
-
```
|
|
416
|
-
|
|
417
|
-
Replace it with one dispatcher per signal, each with its own index-served predicate, position, and
|
|
418
|
-
schedule. A signal backed by several timestamp columns reads one indexed range per column, joined
|
|
419
|
-
with `UNION ALL`, not an `OR` across columns. A window can hold more rows than the cap, so each
|
|
420
|
-
dispatcher pages inside it with a composite keyset position, as in the high-water-mark window.
|
|
421
|
-
`$1` is the window end, `$2` and `$3` the position (`last_at`, `last_id`), and `$4` the cap:
|
|
422
|
-
|
|
423
|
-
```sql
|
|
424
|
-
-- age milestones: the accounts range above, with `id > $3` as the position and
|
|
425
|
-
-- `id < min_uuidv7($1 - milestone)` as the window end
|
|
426
|
-
|
|
427
|
-
-- subscription expirations: one indexed range per timestamp column, merged by (at, id)
|
|
428
|
-
-- (catches only expiries that pass in real time; see the note below)
|
|
429
|
-
SELECT at, id, account_id FROM (
|
|
430
|
-
(SELECT expires_at AS at, id, account_id FROM subscriptions
|
|
431
|
-
WHERE (expires_at, id) > ($2, $3) AND expires_at < $1
|
|
432
|
-
ORDER BY expires_at, id LIMIT $4)
|
|
433
|
-
UNION ALL
|
|
434
|
-
(SELECT grace_ends_at AS at, id, account_id FROM subscriptions
|
|
435
|
-
WHERE (grace_ends_at, id) > ($2, $3) AND grace_ends_at < $1
|
|
436
|
-
ORDER BY grace_ends_at, id LIMIT $4)
|
|
437
|
-
) either_column
|
|
438
|
-
ORDER BY at, id
|
|
439
|
-
LIMIT $4;
|
|
440
|
-
|
|
441
|
-
-- plan changes: a UUIDv7 window, with `last_id` (seeded at the overlap boundary) as the position
|
|
442
|
-
SELECT id, account_id FROM plan_changes
|
|
443
|
-
WHERE id > $3 AND id < min_uuidv7($1)
|
|
444
|
-
ORDER BY id
|
|
445
|
-
LIMIT $4;
|
|
446
|
-
```
|
|
447
|
-
|
|
448
|
-
Each dispatcher keeps its window end and position in one cursor table keyed by an enum:
|
|
449
|
-
|
|
450
|
-
```sql
|
|
451
|
-
CREATE TYPE account_recalc_signals AS ENUM ('age_milestone', 'subscription_expiry', 'plan_change');
|
|
452
|
-
|
|
453
|
-
CREATE TABLE account_recalc_cursors (
|
|
454
|
-
signal account_recalc_signals PRIMARY KEY,
|
|
455
|
-
changes_through_at timestamptz NOT NULL,
|
|
456
|
-
last_at timestamptz NOT NULL,
|
|
457
|
-
last_id uuid NOT NULL DEFAULT '00000000-0000-0000-0000-000000000000'
|
|
458
|
-
);
|
|
459
|
-
```
|
|
460
|
-
|
|
461
|
-
`changes_through_at` is the window end, fixed when the window opens. Advance `(last_at, last_id)`
|
|
462
|
-
after each batch with a compare-and-set. When a batch is short, the window is drained: the next
|
|
463
|
-
window opens with a new `changes_through_at`, and its position restarts an overlap before the
|
|
464
|
-
previous end. A UUIDv7-keyed signal (`plan_change`) seeds `last_id` with
|
|
465
|
-
`min_uuidv7(previous end - overlap)`, never the nil UUID, which would rescan from the start of
|
|
466
|
-
the table; a timestamp-keyed signal sets `last_at` to the same boundary with the nil `last_id`.
|
|
467
|
-
|
|
468
|
-
`expires_at` and `grace_ends_at` are mutable business timestamps, the same trap as using
|
|
469
|
-
`occurred_at` as a high-water mark in the
|
|
470
|
-
[high-water-mark window](#high-water-mark-window-with-an-overlap). A row updated or imported with
|
|
471
|
-
an expiry older than the cursor position is never selected, so the dispatcher catches only
|
|
472
|
-
expiries that pass in real time. Any write that sets an expiry at or before now, or before the
|
|
473
|
-
cursor, enqueues the job directly in the write path or records an immutable change event
|
|
474
|
-
(`plan_changes` is one) that a UUIDv7-keyed dispatcher reads.
|
|
475
|
-
|
|
476
|
-
All of them enqueue the same `recalculateAccount` job with the per-event ids below, so a repeat
|
|
477
|
-
read is harmless. Writes that cause a signal, such as recording a plan change, enqueue the job in
|
|
478
|
-
the write path. The dispatchers are the backstop for a missed enqueue.
|
|
479
|
-
|
|
480
|
-
## Queue job ids for deduplication
|
|
481
|
-
|
|
482
|
-
A queue that deduplicates by job id ignores an add whose id already exists, including a job that is
|
|
483
|
-
running or retained after completion. An id keyed only by the entity drops a new trigger:
|
|
484
|
-
|
|
485
|
-
```ts
|
|
486
|
-
// Account A's job is running when A's subscription expires: this second add is ignored.
|
|
487
|
-
await queue.add('recalculateAccount', { accountId }, { jobId: `recalc__${accountId}` })
|
|
488
|
-
```
|
|
489
|
-
|
|
490
|
-
Key the id by the entity and the event instead. Re-reading the same event in an overlap gives the
|
|
491
|
-
same id, and a new event gets a new id:
|
|
492
|
-
|
|
493
|
-
```ts
|
|
494
|
-
await queue.add(
|
|
495
|
-
'recalculateAccount',
|
|
496
|
-
{ accountId, changeId },
|
|
497
|
-
{ jobId: `recalc__${accountId}__planChanged__${changeId}` },
|
|
498
|
-
)
|
|
499
|
-
```
|
|
500
|
-
|
|
501
|
-
While the job record is retained, each trigger is enqueued at most once; that does not mean it
|
|
502
|
-
runs. Deduplication lasts only as long as the record: retention must cover the longest overlap or
|
|
503
|
-
replay interval, or a re-read event enqueues again after the record is removed. A requirement that
|
|
504
|
-
a trigger take effect exactly once needs a durable event ledger, not job ids. If a job fails for
|
|
505
|
-
good and its record is retained, the same id is still claimed, so retrying it needs an explicit
|
|
506
|
-
reactivate-or-remove step before the next add.
|
|
507
|
-
|
|
508
|
-
## Repair through an existing reconciler
|
|
509
|
-
|
|
510
|
-
When a side effect after commit is fire-and-forget (a cache fill, a filter add, an enqueue), repair
|
|
511
|
-
it in the reconciler that already reads a `changed_through_at` window of recently changed rows. Do
|
|
512
|
-
not create a new sweeper. `$1` and `$2` are the position (`changed_at`, `id`), seeded an overlap
|
|
513
|
-
before the previous window's end, `$3` the window end fixed when the window opens, and `$4` the cap:
|
|
514
|
-
|
|
515
|
-
```sql
|
|
516
|
-
SELECT id, handle, version
|
|
517
|
-
FROM accounts
|
|
518
|
-
WHERE (changed_at, id) > ($1, $2) AND changed_at <= $3
|
|
519
|
-
ORDER BY changed_at, id
|
|
520
|
-
LIMIT $4;
|
|
521
|
-
```
|
|
522
|
-
|
|
523
|
-
Advance the position after each batch with a compare-and-set, and open the next window only after a
|
|
524
|
-
short batch, as in the dispatcher cursor table.
|
|
525
|
-
|
|
526
|
-
```ts
|
|
527
|
-
for (const account of await db.query(recentlyChangedAccountsSql, [
|
|
528
|
-
...position,
|
|
529
|
-
to,
|
|
530
|
-
config.batchSize,
|
|
531
|
-
])) {
|
|
532
|
-
// Monotonic: the write compares `version`, so a delayed repair never overwrites a newer value.
|
|
533
|
-
await accountCache.fillIfNotOlder(account.id, account, account.version)
|
|
534
|
-
await addHandle(account.handle) // idempotent; the invalidating helper from membership filters
|
|
535
|
-
}
|
|
536
|
-
```
|
|
537
|
-
|
|
538
|
-
Idempotence alone is not enough, because the overlap reads rows twice and a delayed repair can land
|
|
539
|
-
after a newer write. Make each repair monotonic: compare an integer `version` that every update
|
|
540
|
-
increments (`version = version + 1`) on write, or invalidate the cache entry and let the next read
|
|
541
|
-
load the current row. Do not compare `changed_at = now()`: it is the transaction's start time and
|
|
542
|
-
is not monotonic across concurrent transactions.
|
|
543
|
-
|
|
544
|
-
## Membership filters
|
|
545
|
-
|
|
546
|
-
A membership filter needs every key, but never on a schedule. The write path adds immediately after
|
|
547
|
-
commit. A failed add stops trusting the filter, by clearing its ready marker so reads fall back to
|
|
548
|
-
the source of truth, and enqueues a rebuild. The job id includes the filter generation: a fixed id
|
|
549
|
-
would stay claimed once its record is retained, so a later failure could never enqueue. Each
|
|
550
|
-
invalidation bumps the generation, so concurrent enqueues for one failure share an id and a new
|
|
551
|
-
failure gets a new one. The reconciler repairs missed adds from its window, and a read that finds
|
|
552
|
-
the filter missing enqueues the same rebuild:
|
|
553
|
-
|
|
554
|
-
```ts
|
|
555
|
-
async function enqueueRebuild(generation: number, afterId = 'start'): Promise<void> {
|
|
556
|
-
try {
|
|
557
|
-
const jobId = `rebuildFilter__handles__${generation}__${afterId}`
|
|
558
|
-
await queue.add('rebuildFilter', { generation, afterId }, { jobId })
|
|
559
|
-
} catch (error) {
|
|
560
|
-
logger.warn({ error }, 'could not enqueue the filter rebuild') // best effort
|
|
561
|
-
}
|
|
562
|
-
}
|
|
563
|
-
|
|
564
|
-
export async function addHandle(handle: string): Promise<void> {
|
|
565
|
-
try {
|
|
566
|
-
await handleFilter.add(handle)
|
|
567
|
-
} catch {
|
|
568
|
-
await enqueueRebuild(await handleFilter.invalidate()) // clears ready, bumps the generation
|
|
569
|
-
}
|
|
570
|
-
}
|
|
571
|
-
|
|
572
|
-
export async function hasHandle(handle: string): Promise<boolean> {
|
|
573
|
-
if (!(await handleFilter.isReady())) {
|
|
574
|
-
await enqueueRebuild(await handleFilter.generation())
|
|
575
|
-
return handleExistsInDatabase(handle) // the read falls back even if the enqueue failed
|
|
576
|
-
}
|
|
577
|
-
// Hint only: a negative means "definitely absent" for keys committed before the last
|
|
578
|
-
// successful add or rebuild. A positive may be a false positive, so verify it at the source.
|
|
579
|
-
try {
|
|
580
|
-
if (!(await handleFilter.has(handle))) return false
|
|
581
|
-
} catch {
|
|
582
|
-
await enqueueRebuild(await handleFilter.invalidate()) // a failing filter is not trusted
|
|
583
|
-
}
|
|
584
|
-
return handleExistsInDatabase(handle)
|
|
585
|
-
}
|
|
586
|
-
```
|
|
587
|
-
|
|
588
|
-
The add happens after commit, so while it is in flight a committed handle can be missing from a
|
|
589
|
-
ready filter. The filter is therefore only a hint for callers that tolerate a just-committed miss,
|
|
590
|
-
such as an availability pre-check. Correctness-critical paths, such as a uniqueness check or a
|
|
591
|
-
read-your-writes lookup, use the database or its unique constraint and never trust a negative.
|
|
592
|
-
|
|
593
|
-
The rebuild is an explicit, capped, resumable backfill with a keyset cursor, as in the
|
|
594
|
-
high-water-mark window. Each capped page enqueues its continuation with the cursor in the job id
|
|
595
|
-
(`rebuildFilter__handles__${generation}__${afterId}`), because a generation-only id would block the
|
|
596
|
-
next page of the same generation. Each page checks that its generation is still current before
|
|
597
|
-
scanning and again before enqueueing a continuation, and a stale page exits without doing
|
|
598
|
-
anything. The rebuild captures its generation when it starts and sets the
|
|
599
|
-
ready marker last, by a compare-and-set that succeeds only while that generation is still current
|
|
600
|
-
(`markReady(generation)`), so a stale rebuild never overwrites a newer invalidation. Deleted keys and growth past
|
|
601
|
-
capacity only cost false positives, so they are an operator's rebuild, not a schedule.
|
|
602
|
-
|
|
603
|
-
## Global ranking
|
|
604
|
-
|
|
605
|
-
A result that depends on every row, such as a percentile tier, needs every row's score whatever the
|
|
606
|
-
design. Keep one periodic aggregate. `$1` and `$2` are the tier cutoffs:
|
|
607
|
-
|
|
608
|
-
```sql
|
|
609
|
-
WITH per_plan AS (
|
|
610
|
-
SELECT s.channel_id, s.plan_id, count(*) AS subscribers
|
|
611
|
-
FROM subscriptions s
|
|
612
|
-
WHERE s.canceled_at IS NULL
|
|
613
|
-
GROUP BY s.channel_id, s.plan_id
|
|
614
|
-
), weighted AS (
|
|
615
|
-
SELECT p.channel_id, sum(p.subscribers * plan.subscriber_weight) AS score
|
|
616
|
-
FROM per_plan p
|
|
617
|
-
JOIN plans plan ON plan.id = p.plan_id
|
|
618
|
-
GROUP BY p.channel_id
|
|
619
|
-
), scores AS (
|
|
620
|
-
-- Start from channels so one with no active subscribers scores 0 instead of dropping out.
|
|
621
|
-
SELECT c.id AS channel_id, coalesce(w.score, 0) AS score
|
|
622
|
-
FROM channels c
|
|
623
|
-
LEFT JOIN weighted w ON w.channel_id = c.id
|
|
624
|
-
), ranked AS (
|
|
625
|
-
SELECT channel_id, PERCENT_RANK() OVER (ORDER BY score) AS percentile, score
|
|
626
|
-
FROM scores
|
|
627
|
-
)
|
|
628
|
-
SELECT channel_id,
|
|
629
|
-
CASE WHEN score = 0 THEN 'low'
|
|
630
|
-
WHEN percentile >= $1 THEN 'top'
|
|
631
|
-
WHEN percentile >= $2 THEN 'mid'
|
|
632
|
-
ELSE 'low' END AS tier
|
|
633
|
-
FROM ranked;
|
|
634
|
-
```
|
|
635
|
-
|
|
636
|
-
`PERCENT_RANK()` gives tied scores the lowest rank, so a large tie at score 0 stays at percentile 0
|
|
637
|
-
instead of reading as a high cumulative fraction, and score 0 is forced into the low tier. Write
|
|
638
|
-
back only the changed tiers, in capped keyset batches over the result. Keeping a subscriber
|
|
639
|
-
count current on every subscribe and unsubscribe is more total work: it makes one hot `channels`
|
|
640
|
-
row take every write, and a plan change fans out to every channel the account follows. It does not
|
|
641
|
-
make tiers fresher either, because the percentile still needs every channel's score. Give the one
|
|
642
|
-
aggregate pass an EXPLAIN budget at representative size.
|
|
643
|
-
|
|
644
|
-
## External listing with a continuation token
|
|
645
|
-
|
|
646
|
-
An object-storage or API listing loop caps the pages per run. When more remain, it enqueues a
|
|
647
|
-
continuation job carrying the provider's continuation token. Restarting from page one each run is
|
|
648
|
-
not a bound: items that keep failing at the front starve the rest.
|
|
649
|
-
|
|
650
|
-
```ts
|
|
651
|
-
export async function listingJob(
|
|
652
|
-
{ token, traversalId = randomUUID() }: { token?: string; traversalId?: string },
|
|
653
|
-
config: ListingConfig,
|
|
654
|
-
): Promise<void> {
|
|
655
|
-
let next = token // traversalId is minted when a traversal starts at page one
|
|
656
|
-
for (let page = 0; page < config.maxPagesPerRun; page += 1) {
|
|
657
|
-
const result = await provider.list({ continuationToken: next, maxKeys: config.pageSize })
|
|
658
|
-
await enqueueItems(result.items)
|
|
659
|
-
if (result.nextToken !== undefined && result.nextToken === next) {
|
|
660
|
-
throw new Error('Provider repeated its continuation token') // never loop or re-enqueue it
|
|
661
|
-
}
|
|
662
|
-
next = result.nextToken
|
|
663
|
-
if (!next) return
|
|
664
|
-
}
|
|
665
|
-
// The traversal id keeps a provider that reuses a token across traversals from colliding.
|
|
666
|
-
await queue.add(
|
|
667
|
-
'listing',
|
|
668
|
-
{ token: next, traversalId },
|
|
669
|
-
{ jobId: `listing__${traversalId}__${next}` },
|
|
670
|
-
)
|
|
671
|
-
}
|
|
672
|
-
```
|
|
673
|
-
|
|
674
|
-
## Verify with a plan
|
|
675
|
-
|
|
676
|
-
Run each statement with representative parameters against representative table sizes, because a
|
|
677
|
-
small table scans sequentially for good reason. The `due` query of the claim above should plan as
|
|
678
|
-
a `Limit` over a `LockRows` over an index scan on the partial index, with no `Sort` between the
|
|
679
|
-
scan and the `Limit`. Read the rows each node returns, not only the scan type: an index scan that
|
|
680
|
-
feeds a `Sort` below the `Limit` still reads every match. A `Seq Scan` on a table that grows is
|
|
681
|
-
also a finding:
|
|
682
|
-
|
|
683
|
-
```sql
|
|
684
|
-
PREPARE claim_due (integer) AS
|
|
685
|
-
SELECT id
|
|
686
|
-
FROM invoice_delivery_work_items
|
|
687
|
-
WHERE completed_at IS NULL AND available_at <= now()
|
|
688
|
-
ORDER BY available_at, id
|
|
689
|
-
LIMIT $1
|
|
690
|
-
FOR UPDATE SKIP LOCKED;
|
|
691
|
-
|
|
692
|
-
BEGIN;
|
|
693
|
-
EXPLAIN (ANALYZE, BUFFERS) EXECUTE claim_due (200);
|
|
694
|
-
ROLLBACK;
|
|
695
|
-
```
|
|
696
|
-
|
|
697
|
-
`EXPLAIN ANALYZE` runs the statement, so check the claim `UPDATE` or the `DELETE` inside a
|
|
698
|
-
transaction that you roll back, or on a disposable copy. Under contention, `SKIP LOCKED` passes over rows other claimers hold, and those rows sit under the
|
|
699
|
-
`Limit`, so the scan reads the cap plus the rows concurrent claims hold. That stays bounded by the
|
|
700
|
-
number of workers times the batch size when claims are short, so keep the claim transaction to the
|
|
701
|
-
claim statement, and check the index scan's actual rows while several workers claim at once.
|
|
702
|
-
|
|
703
|
-
## Anti-patterns
|
|
704
|
-
|
|
705
|
-
| Anti-pattern | Why it is unbounded | Bounded shape |
|
|
706
|
-
| ----------------------------------------------------------- | ------------------------------------------------------------------------ | ------------------------------------------------------------ |
|
|
707
|
-
| Loop until empty: `while (true)` claim, process, break on 0 | New and failing rows keep the run alive; the run has no end | A batch or deadline cap, `hasMore`, and a continuation |
|
|
708
|
-
| A stream with no cap: `for await` over every account | Memory stays flat while run time and database work grow with the table | Stream a keyset window and cap the rows per run |
|
|
709
|
-
| A key-only walk: select every account id, then check each | Touches every entity to find the few with work | Read work items, dirty markers, or a due timestamp |
|
|
710
|
-
| A literal `LIMIT 500` in SQL text | Cannot be tuned per environment or during an incident | A validated tunable with a hard maximum, bound as `LIMIT $1` |
|
|
711
|
-
| `Promise.all(accounts.map(sync))` over a result of any size | One task per row exhausts the pool or the upstream API | A bounded batch with bounded concurrency |
|
|
712
|
-
| An `OR` across unrelated signals in one walk | No index serves the `OR`, so every account is read | One dispatcher per signal, each with its own range |
|
|
713
|
-
| Comparing a fact column with a per-row interval column | The sum is not indexable, so every row is read | One bucket per interval value, bounded by a plain range |
|
|
714
|
-
| Job ids keyed only by entity | A running or retained job silently drops a new trigger | An id keyed by entity and triggering event |
|
|
715
|
-
| A new sweeper beside an existing reconciler | A second pass over rows the reconciler already reads | Add the idempotent repair to the existing reconciler |
|
|
716
|
-
| A scheduled membership filter rebuild | Reads every key on every run | Rebuild on operator action or detected failure only |
|
|
717
|
-
| Restarting an external listing at page one | Failing items at the front starve the rest | A continuation job carrying the provider's token |
|
|
718
|
-
| `OFFSET` pagination | Every page rescans and discards the rows before it | Keyset on the indexed key |
|
|
719
|
-
| `UPDATE` or `DELETE` with a predicate but no `LIMIT` | One statement locks and rewrites an unbounded row set in one transaction | A CTE that selects a capped, `SKIP LOCKED` id set |
|