vouchington-tooling 0.72.3 → 0.73.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "vouchington-tooling",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.73.0",
|
|
4
4
|
"description": "Vouchington CLI and extractable tooling libraries.",
|
|
5
5
|
"homepage": "https://github.com/vouchington/vouchington-tooling/tree/main/packages/vouchington-tooling#readme",
|
|
6
6
|
"bugs": {
|
|
Binary file
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: glide-mq
|
|
3
|
+
description: Design glide-mq queue jobs, ordering lanes, worker concurrency, and cross-job caps when writing or reviewing queue producers, workers, and dispatchers.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# glide-mq queues
|
|
7
|
+
|
|
8
|
+
Read applicable `AGENTS.md` and the repository's local queue adapter when present.
|
|
9
|
+
Local policy overrides these portable conventions.
|
|
10
|
+
|
|
11
|
+
Names below are the glide-mq API: the job option `ordering: { key, concurrency }`, the worker
|
|
12
|
+
option `concurrency`, `lockDuration`, and `job.moveToDelayed(timestamp)`. Read
|
|
13
|
+
[lease patterns](references/lease-patterns.md) for the lease shape.
|
|
14
|
+
|
|
15
|
+
1. **Every shared-queue job sets `ordering`.** An ungrouped job in a shared queue competes for
|
|
16
|
+
every worker slot, so a flood of it starves other families. Give each job an
|
|
17
|
+
`ordering.key` prefixed with its domain name and an explicit `ordering.concurrency`.
|
|
18
|
+
2. **`ordering.concurrency` is a global cap; the worker `concurrency` is per process.** The key's
|
|
19
|
+
cap holds across all workers and processes. The worker option only bounds one process's
|
|
20
|
+
active jobs. Never size lane caps to sum to the worker option. Take caps from runtime
|
|
21
|
+
configuration, and give batched families a low cap.
|
|
22
|
+
3. **Group FIFO ignores `priority`.** Jobs sharing an ordering key run in enqueue order whatever
|
|
23
|
+
their priority. Work that must not wait behind a flood, and dispatchers that rely on
|
|
24
|
+
priority, get their own lane. Put work that outsiders can flood in a lane of its own.
|
|
25
|
+
Ordering keys are the orchestration layer; add none on top.
|
|
26
|
+
4. **`globalConcurrency` caps the whole queue.** The worker option `globalConcurrency` limits
|
|
27
|
+
every job on the queue together, so on a shared queue it throttles all families at once. Use
|
|
28
|
+
ordering lanes for per-family caps, and use `globalConcurrency` only when one resource bounds
|
|
29
|
+
the whole queue.
|
|
30
|
+
5. **A job moved to delayed keeps its ordering slot until it completes.** A deferred job still
|
|
31
|
+
counts against `ordering.concurrency`, so deferred jobs can fill a key and block ready work
|
|
32
|
+
on it. A cap that must survive deferral belongs in a token-owned lease, released before
|
|
33
|
+
`moveToDelayed`, not in the ordering key. Never fix starvation by raising the key's
|
|
34
|
+
concurrency: overlap grows and the same number of deferred jobs still fill the slots.
|
|
35
|
+
6. **Lease mechanics.** Acquire a token-owned lease with `SET key token NX PX ttl` before the
|
|
36
|
+
guarded work. Set the TTL at or above the job's `lockDuration`, renew it while the job runs,
|
|
37
|
+
and release it with compare-and-delete in `finally`. On contention, `moveToDelayed` only this
|
|
38
|
+
job by a short, bounded, jittered delay so it keeps its id, cursor, and dedup. Release the
|
|
39
|
+
lease before any `moveToDelayed` for quota deferral. Reuse the repository's lease helper
|
|
40
|
+
rather than adding a second. See [lease patterns](references/lease-patterns.md).
|
|
41
|
+
7. **Job count is not a KPI.** Many small jobs with a dispatcher and a sweep are fine. Do not
|
|
42
|
+
merge or remove jobs to cut the count. Judge a design by model calls or spend per item and by
|
|
43
|
+
how simply the jobs share a lifecycle.
|
|
44
|
+
|
|
45
|
+
Dispatchers, paged fan-out, and job-id deduplication follow
|
|
46
|
+
[bounded iteration](../bounded-iteration/SKILL.md). A lease for work that spans an external call
|
|
47
|
+
replaces holding a PostgreSQL transaction or advisory lock open across it; see
|
|
48
|
+
[performance patterns](../postgres-node-performance-tuning/references/performance-patterns.md).
|
|
49
|
+
|
|
50
|
+
Consumer policy owns queue names, lane keys, caps and their configuration source, the lease helper,
|
|
51
|
+
and enforcement configuration.
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
# Lease patterns
|
|
2
|
+
|
|
3
|
+
A lease caps work across jobs and survives deferral, which an `ordering` slot does not: a job
|
|
4
|
+
moved to delayed keeps its slot until it completes.
|
|
5
|
+
|
|
6
|
+
## Acquire, renew, release
|
|
7
|
+
|
|
8
|
+
```ts
|
|
9
|
+
const token = randomUUID()
|
|
10
|
+
// SET leaseKey token NX PX ttlMs -> null when another job holds the lease
|
|
11
|
+
if (!(await setIfAbsent(leaseKey, token, ttlMs))) {
|
|
12
|
+
return job.moveToDelayed(Date.now() + jitteredDelayMs) // this job only; never falls through
|
|
13
|
+
}
|
|
14
|
+
try {
|
|
15
|
+
// renew while running, e.g. every ttlMs / 3, only while the stored value equals token
|
|
16
|
+
await doGuardedWork()
|
|
17
|
+
} finally {
|
|
18
|
+
await compareAndDelete(leaseKey, token) // script: delete only when the value equals token
|
|
19
|
+
}
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
- The TTL is at least the job's `lockDuration`. A shorter lease can expire while the job is
|
|
23
|
+
still active, and a second job then enters the guarded section.
|
|
24
|
+
- Renewal and release are scripts that compare the stored value with the token. A plain `DEL`
|
|
25
|
+
can remove a lease another job has since acquired.
|
|
26
|
+
- The delay on contention is short, bounded, and jittered so deferred jobs do not wake together.
|
|
27
|
+
`moveToDelayed` defers this job only and keeps its id, cursor, and dedup.
|
|
28
|
+
|
|
29
|
+
## Quota deferral
|
|
30
|
+
|
|
31
|
+
When the work defers itself for a quota or rate, release the lease first, then call
|
|
32
|
+
`moveToDelayed`. `moveToDelayed` never returns, so a release placed after it does not run, and a
|
|
33
|
+
deferred job that keeps the lease blocks the jobs the lease exists to admit.
|
|
34
|
+
|
|
35
|
+
```ts
|
|
36
|
+
await compareAndDelete(leaseKey, token)
|
|
37
|
+
await job.moveToDelayed(resumeAt)
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
## Why not the ordering key
|
|
41
|
+
|
|
42
|
+
Concurrency 1 on a key serializes ready work but lets deferred jobs hold the slot. Raising it to
|
|
43
|
+
2 lets two guarded sections overlap, and two deferred jobs still fill both slots. A lease
|
|
44
|
+
separates the cap on active work from the queue-order slot.
|
package/skills/manifest.json
CHANGED
|
@@ -30,7 +30,7 @@
|
|
|
30
30
|
{
|
|
31
31
|
"name": "bounded-iteration",
|
|
32
32
|
"plugin": "vouchington-database",
|
|
33
|
-
"pluginVersion": "0.
|
|
33
|
+
"pluginVersion": "0.9.0",
|
|
34
34
|
"path": "bounded-iteration/SKILL.md"
|
|
35
35
|
},
|
|
36
36
|
{
|
|
@@ -72,6 +72,12 @@
|
|
|
72
72
|
"pluginVersion": "0.14.2",
|
|
73
73
|
"path": "github-issue/SKILL.md"
|
|
74
74
|
},
|
|
75
|
+
{
|
|
76
|
+
"name": "glide-mq",
|
|
77
|
+
"plugin": "vouchington-database",
|
|
78
|
+
"pluginVersion": "0.9.0",
|
|
79
|
+
"path": "glide-mq/SKILL.md"
|
|
80
|
+
},
|
|
75
81
|
{
|
|
76
82
|
"name": "nextjs-vitest-test-authoring",
|
|
77
83
|
"plugin": "vouchington-testing",
|
|
@@ -115,19 +121,19 @@
|
|
|
115
121
|
{
|
|
116
122
|
"name": "postgres-node-performance-tuning",
|
|
117
123
|
"plugin": "vouchington-database",
|
|
118
|
-
"pluginVersion": "0.
|
|
124
|
+
"pluginVersion": "0.9.0",
|
|
119
125
|
"path": "postgres-node-performance-tuning/SKILL.md"
|
|
120
126
|
},
|
|
121
127
|
{
|
|
122
128
|
"name": "postgres-partitioning-uuid-v7",
|
|
123
129
|
"plugin": "vouchington-database",
|
|
124
|
-
"pluginVersion": "0.
|
|
130
|
+
"pluginVersion": "0.9.0",
|
|
125
131
|
"path": "postgres-partitioning-uuid-v7/SKILL.md"
|
|
126
132
|
},
|
|
127
133
|
{
|
|
128
134
|
"name": "postgres-schema-design",
|
|
129
135
|
"plugin": "vouchington-database",
|
|
130
|
-
"pluginVersion": "0.
|
|
136
|
+
"pluginVersion": "0.9.0",
|
|
131
137
|
"path": "postgres-schema-design/SKILL.md"
|
|
132
138
|
},
|
|
133
139
|
{
|