vouchington-tooling 0.72.2 → 0.72.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/package.json +1 -1
- package/scripts/__pycache__/run_bounded_process.cpython-312.pyc +0 -0
- package/skills/bounded-iteration/SKILL.md +2 -1
- package/skills/bounded-iteration/references/sweep-patterns.md +4 -0
- package/skills/manifest.json +4 -4
- package/skills/postgres-node-performance-tuning/SKILL.md +2 -1
- package/skills/postgres-node-performance-tuning/references/performance-patterns.md +22 -6
- package/skills/postgres-partitioning-uuid-v7/references/partition-lifecycle.md +5 -2
- package/skills/postgres-schema-design/SKILL.md +1 -0
- package/skills/postgres-schema-design/references/relations.md +3 -1
- package/skills/postgres-schema-design/references/table-shapes.md +31 -11
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "vouchington-tooling",
|
|
3
|
-
"version": "0.72.
|
|
3
|
+
"version": "0.72.3",
|
|
4
4
|
"description": "Vouchington CLI and extractable tooling libraries.",
|
|
5
5
|
"homepage": "https://github.com/vouchington/vouchington-tooling/tree/main/packages/vouchington-tooling#readme",
|
|
6
6
|
"bugs": {
|
|
Binary file
|
|
@@ -29,7 +29,8 @@ A run's cost is the rows it reads and the work it starts, not the memory it hold
|
|
|
29
29
|
per run.
|
|
30
30
|
5. **Resume without restarting.** Keep a keyset position on the UUIDv7 key or a
|
|
31
31
|
`<verb>_through_at` time position, fix the sweep's end when it starts, and rescan an overlap
|
|
32
|
-
when insert order differs from commit order.
|
|
32
|
+
when insert order differs from commit order. Prefer the position in job data; a cursor table is
|
|
33
|
+
for a position that must be transactional with domain rows, and follows
|
|
33
34
|
[table shapes](../postgres-schema-design/references/table-shapes.md).
|
|
34
35
|
(`postgres-generated-column-predicates`)
|
|
35
36
|
6. **No unbounded fan-out.** Use `Promise.all` only over a bounded result, with bounded
|
|
@@ -81,6 +81,10 @@ parent id, `WHERE account_id = $1`, for the one account it was started for.
|
|
|
81
81
|
|
|
82
82
|
## High-water-mark window with an overlap
|
|
83
83
|
|
|
84
|
+
Prefer keeping the position and the sweep's fixed end in job data for a single-writer continuation.
|
|
85
|
+
Use a persisted cursor table only when the position must be transactional with domain rows, as
|
|
86
|
+
here.
|
|
87
|
+
|
|
84
88
|
An export reads `invoices` added since the last sweep. The cursor row holds the position and the
|
|
85
89
|
sweep's fixed end, and the job seeds its own row:
|
|
86
90
|
|
package/skills/manifest.json
CHANGED
|
@@ -30,7 +30,7 @@
|
|
|
30
30
|
{
|
|
31
31
|
"name": "bounded-iteration",
|
|
32
32
|
"plugin": "vouchington-database",
|
|
33
|
-
"pluginVersion": "0.
|
|
33
|
+
"pluginVersion": "0.8.0",
|
|
34
34
|
"path": "bounded-iteration/SKILL.md"
|
|
35
35
|
},
|
|
36
36
|
{
|
|
@@ -115,19 +115,19 @@
|
|
|
115
115
|
{
|
|
116
116
|
"name": "postgres-node-performance-tuning",
|
|
117
117
|
"plugin": "vouchington-database",
|
|
118
|
-
"pluginVersion": "0.
|
|
118
|
+
"pluginVersion": "0.8.0",
|
|
119
119
|
"path": "postgres-node-performance-tuning/SKILL.md"
|
|
120
120
|
},
|
|
121
121
|
{
|
|
122
122
|
"name": "postgres-partitioning-uuid-v7",
|
|
123
123
|
"plugin": "vouchington-database",
|
|
124
|
-
"pluginVersion": "0.
|
|
124
|
+
"pluginVersion": "0.8.0",
|
|
125
125
|
"path": "postgres-partitioning-uuid-v7/SKILL.md"
|
|
126
126
|
},
|
|
127
127
|
{
|
|
128
128
|
"name": "postgres-schema-design",
|
|
129
129
|
"plugin": "vouchington-database",
|
|
130
|
-
"pluginVersion": "0.
|
|
130
|
+
"pluginVersion": "0.8.0",
|
|
131
131
|
"path": "postgres-schema-design/SKILL.md"
|
|
132
132
|
},
|
|
133
133
|
{
|
|
@@ -18,7 +18,8 @@ Measure the query plan and the workload shape before changing code.
|
|
|
18
18
|
- Keep connection-pool usage bounded.
|
|
19
19
|
- Validate a query change at representative cardinality. Watch latency, memory, lock time, and
|
|
20
20
|
connection pressure together.
|
|
21
|
-
- Cut round trips: fold dependent reads, batch per-item lookups, and
|
|
21
|
+
- Cut round trips without weakening correctness: fold dependent reads, batch per-item lookups, and
|
|
22
|
+
overlap independent reads; see
|
|
22
23
|
[round trips](references/performance-patterns.md#round-trips).
|
|
23
24
|
- Read the plan for hidden per-row work before trusting a change; see
|
|
24
25
|
[reading plans](references/reading-plans.md).
|
|
@@ -3,10 +3,16 @@
|
|
|
3
3
|
Use pools and release checked-out clients in `finally`, especially for cursors, streams, and COPY.
|
|
4
4
|
Route ordinary reads to a replica when its lag is acceptable; route read-after-write, locking, and
|
|
5
5
|
transaction-consistent reads to the writer. Do not hold a client across unrelated application work.
|
|
6
|
+
Never hold a transaction, connection, or advisory lock across an external call; take a token lease
|
|
7
|
+
in a shared store instead, with a TTL, renewal, and compare-and-delete release. Stop work when
|
|
8
|
+
renewal fails, and guard any write a stale owner could make with the token.
|
|
6
9
|
|
|
7
10
|
For large reads, use keyset pagination, cursors, or streams with cancellation and bounded batches.
|
|
8
11
|
For writes, prefer set-based batches such as UNNEST or COPY when they preserve validation and error
|
|
9
|
-
handling.
|
|
12
|
+
handling. Do large list diffs and dedupes in the worker as a streaming comparison of sorted
|
|
13
|
+
inputs, not as writer-side set operations. Measure query plans with representative cardinality
|
|
14
|
+
before changing an index or query. Adding an index for a measured plan is a default: do not ask
|
|
15
|
+
first and do not make it a spec stop condition.
|
|
10
16
|
|
|
11
17
|
Check query predicates, joins, ordering, and selected columns against index shape. Use EXPLAIN
|
|
12
18
|
evidence to confirm planner behavior. Add extended statistics only when observed estimates show a
|
|
@@ -126,13 +132,23 @@ lateral subquery, rather than reading the parent and then the relation.
|
|
|
126
132
|
`Promise.all` over statements on one transaction client is still serial, because one connection
|
|
127
133
|
runs one statement at a time. Merge them instead: CTEs, `UNNEST`, or one multi-row statement.
|
|
128
134
|
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
135
|
+
Write the batch implementation first: it takes an array of keys and returns a keyed map. The
|
|
136
|
+
single-item function is a thin wrapper over it (`addUrl(url)` calls `addUrls([url])`), with no
|
|
137
|
+
parallel per-item SQL path, for lookups and mutations alike. The batch must give a one-item array the
|
|
138
|
+
same atomicity, partial-failure, ordering, duplicate-key, and result-shape semantics as the single
|
|
139
|
+
call; if it doesn't, fix the batch rather than keep a per-item path. Call the batch from loops, `.map`
|
|
140
|
+
callbacks, and `Promise.all` fan-outs. `no-await-in-loop` misses a helper that queries internally
|
|
141
|
+
and `Promise.all(items.map(helper))`; review those call sites by hand.
|
|
133
142
|
|
|
134
143
|
Overlap independent cache and PostgreSQL reads with `Promise.all`, such as a rate-limiter check
|
|
135
|
-
alongside the user fetch. Serialize only true dependencies, where one result feeds the next call.
|
|
144
|
+
alongside the user fetch. Serialize only true dependencies, where one result feeds the next call. Suppress the
|
|
145
|
+
sequential-await lint only when each step consumes the previous step's result; a shared
|
|
146
|
+
transaction client or lock ordering is not a data dependency. A lock that must be taken first
|
|
147
|
+
stays a separate, earlier statement; never reorder or parallelize it. Audit every added suppression.
|
|
148
|
+
|
|
149
|
+
Correctness comes before round trips. Before folding a statement away, state the guarantee it
|
|
150
|
+
gave (what it locked, which snapshot it read) and reject the fold if that weakens. An `EXISTS`
|
|
151
|
+
inside a `FOR UPDATE` statement reads a snapshot taken before the lock wait, so it can be stale.
|
|
136
152
|
|
|
137
153
|
Run a symmetric check, such as "is either party blocking the other", as one query with an `OR`
|
|
138
154
|
or both-direction predicate, not twice with the arguments swapped.
|
|
@@ -4,8 +4,11 @@ UUIDv7 ordering supports range bounds and index-friendly time windows. Generate
|
|
|
4
4
|
instead of extracting timestamps in predicates; use a generated timestamp only when application
|
|
5
5
|
semantics require one. Partition on the range key that queries and retention actually constrain.
|
|
6
6
|
|
|
7
|
-
|
|
8
|
-
|
|
7
|
+
Choose a lifecycle. Pre-create future ranges before writes need them, or use on-demand ranges:
|
|
8
|
+
`RANGE (id)` with a DEFAULT partition, split into explicit ranges on demand, and a guarded drop
|
|
9
|
+
that refuses drops inside the retention window or under legal hold. Move or check rows in the
|
|
10
|
+
DEFAULT partition that belong to the range before dropping it. Maintain a default partition
|
|
11
|
+
only with an explicit attachment plan. Before attaching a populated range, prove the default partition excludes that range
|
|
9
12
|
or move conflicting rows; otherwise attachment can scan and lock the default partition.
|
|
10
13
|
|
|
11
14
|
Every primary or unique constraint on a partitioned table must include its partition key. Verify
|
|
@@ -13,6 +13,7 @@ Local policy overrides these portable conventions.
|
|
|
13
13
|
- Joined identities use foreign keys; read [relations](references/relations.md) before modeling references.
|
|
14
14
|
- Split differing shapes, share identical values, and keep history documents as JSON.
|
|
15
15
|
- Use canonical [table shapes](references/table-shapes.md) for revisions, changes, work, and cursors.
|
|
16
|
+
- Default to finite retention for everything except user content; see [table shapes](references/table-shapes.md#retention).
|
|
16
17
|
- Reuse parameterized trigger functions instead of copying bodies.
|
|
17
18
|
- Comment every table, column, and view; document reviewed exceptions explicitly.
|
|
18
19
|
|
|
@@ -2,7 +2,9 @@
|
|
|
2
2
|
|
|
3
3
|
An identity that readers join is an FK column or an FK child row. JSON holds structured,
|
|
4
4
|
schemaless data and history snapshots, including identities recorded as they were at the time.
|
|
5
|
-
Do not use UUID arrays, JSON ids, or type/id pairs for live relations. A
|
|
5
|
+
Do not use UUID arrays, JSON ids, or type/id pairs for live relations. A relation id that
|
|
6
|
+
appears in JSON or in a hashed key also gets an FK column on the same row; the JSON copy may stay,
|
|
7
|
+
and dedup keys on the FK columns, plus any other identity fields, rather than a hash of the document. A column never names a
|
|
6
8
|
table: use one table per target and generate any per-relation progress enum from the same metadata.
|
|
7
9
|
Cursor positions and random fencing tokens are intentionally not live entity references; record
|
|
8
10
|
their reviewed exception instead of inventing a foreign key.
|
|
@@ -7,14 +7,14 @@ explicit sweeper-written timestamp; `now()` cannot appear in an immutable genera
|
|
|
7
7
|
When state keeps returning to earlier states, record append-only attempts or changes and derive
|
|
8
8
|
current state from the latest row rather than erasing the previous lifecycle.
|
|
9
9
|
|
|
10
|
-
| Concept | Shape
|
|
11
|
-
| -------------------------- |
|
|
12
|
-
| Field-diff history | `<entity>_revisions`: UUIDv7 `id`, entity FK, nullable revision actor FK with `ON DELETE SET NULL`, revision-type enum, `changes jsonb`.
|
|
13
|
-
| Typed lifecycle ledger | `<entity>_changes`: UUIDv7 `id`, entity FK, change-type enum, retained actor identity FK with `ON DELETE RESTRICT`, typed joined payload and JSON for unjoined data. Time comes from `id`.
|
|
14
|
-
| Current state from a log | Latest change by `id DESC`, projected through a shared trigger or read through a view.
|
|
15
|
-
| Per-entity work | `<entity>_<job>_work_items`: entity FK, lease and final-outcome columns, claim index, retention.
|
|
16
|
-
| Job progress | `<job>_cursors
|
|
17
|
-
| Inbound or protocol events | Append-only `<source>_events` with `occurred_at` recording the event's business time.
|
|
10
|
+
| Concept | Shape |
|
|
11
|
+
| -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
12
|
+
| Field-diff history | `<entity>_revisions`: UUIDv7 `id`, entity FK, nullable revision actor FK with `ON DELETE SET NULL`, revision-type enum, `changes jsonb`. |
|
|
13
|
+
| Typed lifecycle ledger | `<entity>_changes`: UUIDv7 `id`, entity FK, change-type enum, retained actor identity FK with `ON DELETE RESTRICT`, typed joined payload and JSON for unjoined data. Time comes from `id`. |
|
|
14
|
+
| Current state from a log | Latest change by `id DESC`, projected through a shared trigger or read through a view. |
|
|
15
|
+
| Per-entity work | Default to a queue job id plus a bounded re-enqueue sweep. When justified, `<entity>_<job>_work_items`: entity FK, lease and final-outcome columns, claim index, retention. |
|
|
16
|
+
| Job progress | Prefer the position in job data. Use `<job>_cursors` only when it must be transactional with domain rows: finite enum key or checked singleton, keyset positions without FKs, optional sweep upper bound. |
|
|
17
|
+
| Inbound or protocol events | Append-only `<source>_events` with `occurred_at` recording the event's business time. |
|
|
18
18
|
|
|
19
19
|
Revisions and changes are append-only through a generic `fn_reject_mutation`, parameterized by
|
|
20
20
|
actor columns that may be erased to NULL. Permit only those erasures; parent-cascade deletion
|
|
@@ -25,20 +25,40 @@ A system change has a NULL actor. Retained actors remain historical identities,
|
|
|
25
25
|
|
|
26
26
|
## Work leases
|
|
27
27
|
|
|
28
|
-
|
|
28
|
+
Default to a queue job whose job id is the durable row id, enqueued after commit, plus a bounded
|
|
29
|
+
sweep that re-enqueues rows with no recorded outcome. The queue already supplies attempts,
|
|
30
|
+
backoff, locking, stall recovery and unique ids. A lease or outbox work-item table needs a stated
|
|
31
|
+
reason: legal or payment records, old values that vanish at commit, or an outcome that must
|
|
32
|
+
commit with domain rows.
|
|
33
|
+
|
|
34
|
+
When a table is justified, fence with one compare-and-set on the outcome or a generation
|
|
35
|
+
(`WHERE completed_at IS NULL` / `AND generation = $n`) instead of a full set of lease columns.
|
|
36
|
+
Idempotent database-only work needs no fence. A unique job id stops duplicate enqueues, not
|
|
37
|
+
duplicate runs.
|
|
38
|
+
|
|
39
|
+
When a full lease is needed, the lease lives on a row that exists only for the work. If an invoice outlives a delivery job,
|
|
29
40
|
use `invoice_delivery_work_items`, with an invoice FK and `ON DELETE CASCADE`, rather than putting
|
|
30
41
|
the lease on the invoice. Do not create a shared polymorphic lease table.
|
|
31
42
|
|
|
32
|
-
|
|
43
|
+
A full lease uses `lease_token uuid`, `leased_at`, `lease_expires_at`, `attempt_count`, `available_at`, and
|
|
33
44
|
`completed_at` plus timestamps for final outcomes. Add `generation` only when re-enqueue can race
|
|
34
45
|
a running worker. Each claim or reclaim gets a fresh opaque token. Every renewal, completion,
|
|
35
46
|
and failure must include `WHERE lease_token = $mine`; zero affected rows means ownership was lost.
|
|
36
47
|
Keep retries in an optional `_attempts` ledger with `attempt_number` when their lifecycle must
|
|
37
48
|
survive. One-shot compare-and-set claims and human assignments need their own shapes.
|
|
38
49
|
|
|
50
|
+
## Retention
|
|
51
|
+
|
|
52
|
+
Give every durable row a finite retention window, except user content. Prefer a compact per-entity
|
|
53
|
+
summary over raw event rows when no reader needs the individual events; append-only ledgers still
|
|
54
|
+
get a retention window. Delete a cache's older rows when a new version is written.
|
|
55
|
+
A bounded post-commit enqueue-loss window is acceptable when the stale result heals itself through
|
|
56
|
+
a TTL; no transactional dirty-work table is needed.
|
|
57
|
+
|
|
39
58
|
## Cursors and shared functions
|
|
40
59
|
|
|
41
|
-
|
|
60
|
+
Prefer keeping the position in job data; a cursor table is for a sweep whose position must commit with
|
|
61
|
+
its domain rows. A cursor key names the finite thing being swept. A singleton uses
|
|
42
62
|
`is_singleton boolean PRIMARY KEY DEFAULT true CHECK (is_singleton)`; the job seeds its own row
|
|
43
63
|
with `ON CONFLICT DO NOTHING`. `cursor_<thing>_id` and optional `sweep_upper_bound_<thing>_id` are
|
|
44
64
|
FK-free UUIDv7 positions; NULL means the start. Per-unit work progress belongs in work items.
|