@eventferry/postgres 3.3.0 → 3.3.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (2) hide show
  1. package/CHANGELOG.md +269 -0
  2. package/package.json +9 -5
package/CHANGELOG.md ADDED
@@ -0,0 +1,269 @@
1
+ # @eventferry/postgres
2
+
3
+ ## 3.3.2
4
+
5
+ ### Patch Changes
6
+
7
+ - Updated dependencies [715523f]
8
+ - Updated dependencies [fb0549d]
9
+ - @eventferry/core@3.4.0
10
+
11
+ ## 3.3.1
12
+
13
+ ### Patch Changes
14
+
15
+ - 3c33f71: **chore: ship `CHANGELOG.md` inside the npm tarball**
16
+
17
+ Previously, each package's `files` allowlist contained only `"dist"` (and `"sql"` for `@eventferry/postgres`), so the auto-generated `CHANGELOG.md` was never published. Users browsing the package on npmjs.com or unpacking the tarball couldn't see release notes — they had to navigate to the GitHub repo.
18
+
19
+ This release adds `"CHANGELOG.md"` to the `files` array of every publishable package. Starting with this version, the per-version release notes are accessible:
20
+
21
+ - Directly in `node_modules/@eventferry/<pkg>/CHANGELOG.md` after `npm install`
22
+ - In the file listing on npmjs.com (under the "Code" / "Files" tab, depending on the npm UI)
23
+ - Inside the tarball downloaded from `https://registry.npmjs.org/...`
24
+
25
+ No code or API surface changes.
26
+
27
+ - Updated dependencies [3c33f71]
28
+ - @eventferry/core@3.3.1
29
+
30
+ ## 3.3.0
31
+
32
+ ### Minor Changes
33
+
34
+ - cdc20cf: **feat: DLQ enrichment + backpressure runtime + quota multiplier — Tier 1 of the reliability gap closed**
35
+
36
+ ### DLQ enrichment
37
+
38
+ Records routed to the dead-letter queue now carry the full context an operator needs to triage:
39
+
40
+ | Header | Set by | Note |
41
+ | --------------------------- | --------- | ------------------------------------------------------------------------------------------------ |
42
+ | `original-topic` | relay | already existed |
43
+ | `dlq-reason` | publisher | already existed (`error.message`) |
44
+ | `dlq-failed-at` | publisher | already existed (ISO timestamp) |
45
+ | `dlq-error-class` | publisher | **new** — `error.name` / constructor name |
46
+ | `dlq-attempts` | relay | **new** — string-encoded `attempts` count |
47
+ | `dlq-original-aggregate-id` | relay | **new** — for joining with business state |
48
+ | `dlq-original-message-id` | relay | **new** — for dedup / idempotency lookups |
49
+ | `dlq-error-stack` | relay | **new** — opt-in via `DlqConfig.includeStackTraces`, truncated to `maxStackBytes` (default 4 KB) |
50
+
51
+ ```ts
52
+ new Relay({
53
+ store,
54
+ publisher,
55
+ dlq: { topic: "orders.dlq", includeStackTraces: true, maxStackBytes: 4096 },
56
+ });
57
+ ```
58
+
59
+ ### Backpressure runtime behavior
60
+
61
+ When the driver classifies a failure as `errorKind: "backpressure"` (client-side producer queue full), the relay no longer treats it like a regular retriable failure. Instead:
62
+
63
+ - The record is re-queued via the new `OutboxStore.requeue(id, retryAt)` method,
64
+ - `attempts` is **not incremented** — the buffer being full is a "slow down" signal, not the record's fault,
65
+ - The retry is scheduled `RetryConfig.backpressureDelayMs` ms ahead (default 1000 ms).
66
+
67
+ Stores that don't implement `requeue` fall back to `markFailed` (with attempts++); both `@eventferry/postgres` and `@eventferry/mysql` ship a real implementation.
68
+
69
+ ### Quota multiplier
70
+
71
+ When the driver classifies a failure as `errorKind: "quota"` (broker `THROTTLING_QUOTA_EXCEEDED`), the scheduled retry delay is multiplied by `RetryConfig.quotaMultiplier` (default 5) so the producer gives the broker breathing room. Quota failures DO count as attempts — after the budget is exhausted the record routes to DLQ + `dead`.
72
+
73
+ ### New / changed types
74
+
75
+ - `RetryConfig` gains `backpressureDelayMs?` and `quotaMultiplier?`.
76
+ - `DlqConfig` gains `includeStackTraces?` and `maxStackBytes?`.
77
+ - `OutboxStore.requeue?(recordId, retryAt)` is a new **optional** method. Stores without it fall through to `markFailed`.
78
+
79
+ ### Backward compatibility
80
+
81
+ Pure-additive everywhere. Default behavior matches the prior release:
82
+
83
+ - A `RetryConfig` without `backpressureDelayMs` uses 1000 ms (sensible default).
84
+ - A `DlqConfig` without `includeStackTraces` keeps DLQ messages small (default off).
85
+ - An `OutboxStore` without `requeue` falls back to `markFailed` — same as before, just with a documented quirk.
86
+
87
+ This closes the last three Tier 1 items in `docs/kafka-gap-analysis/reliability.md`. Phase A reliability surface is now ~100% complete.
88
+
89
+ ### Patch Changes
90
+
91
+ - Updated dependencies [cdc20cf]
92
+ - @eventferry/core@3.3.0
93
+
94
+ ## 3.2.1
95
+
96
+ ### Patch Changes
97
+
98
+ - 9beb3e2: **chore: migrate to independent versioning (Astro pattern)**
99
+
100
+ Fixes the major-version inflation that produced four consecutive surprise majors (`1.0.4 → 2.0.0`, `2.0.0 → 3.0.0`, `3.0.0 → 4.0.0 corrected to 3.1.0`, `3.1.0 → 4.0.0 corrected to 3.2.0`) from changesets whose frontmatter only asked for `minor`.
101
+
102
+ **Root cause** (cited in [changesets/changesets#1759](https://github.com/changesets/changesets/issues/1759) and [docs/decisions.md](https://github.com/changesets/changesets/blob/main/docs/decisions.md)): the adapters listed `@eventferry/core` as a `peerDependency` with `workspace:*`. Changesets' documented rule is that an internal bump of a peer forces a major bump on the dependent — and the `fixed: [["@eventferry/*"]]` group reconciler then propagated that major across every package in the group.
103
+
104
+ **Fix** (exactly the [Astro config](https://github.com/withastro/astro/blob/main/.changeset/config.json)):
105
+
106
+ 1. `.changeset/config.json` — drop `fixed`, set `linked: []`, enable
107
+ `___experimentalUnsafeOptions_WILL_CHANGE_IN_PATCH.onlyUpdatePeerDependentsWhenOutOfRange: true`.
108
+ 2. Move `@eventferry/core` from `peerDependencies` to `dependencies` in
109
+ `@eventferry/postgres`, `@eventferry/mysql`, `@eventferry/kafka`, and
110
+ `@eventferry/schema-registry`. External user-facing peers (`pg`,
111
+ `mysql2`, `kafkajs`, `@confluentinc/kafka-javascript`,
112
+ `@kafkajs/confluent-schema-registry`) stay unchanged.
113
+
114
+ **Effect on releases.** Packages now evolve at independent semver tempos: a `core: minor` changeset produces `core@3.3.0` alongside `postgres@3.2.1` (patch, from "Updated dependencies"). No more major surprises. No more manual force-push corrections.
115
+
116
+ **Effect on consumers.** Pure-additive at the install boundary: `npm i @eventferry/kafka` now resolves `@eventferry/core` automatically (it's a regular dep). Previously consumers had to install it themselves as a peer; the typical flow already did this. No source-code changes required.
117
+
118
+ - Updated dependencies [9beb3e2]
119
+ - @eventferry/core@3.2.1
120
+
121
+ ## 3.2.0
122
+
123
+ ### Patch Changes
124
+
125
+ - @eventferry/core@3.2.0
126
+
127
+ ## 3.1.0
128
+
129
+ ### Patch Changes
130
+
131
+ - Updated dependencies [da39b08]
132
+ - @eventferry/core@3.1.0
133
+
134
+ ## 3.0.0
135
+
136
+ ### Patch Changes
137
+
138
+ - Updated dependencies [f0c7483]
139
+ - @eventferry/core@3.0.0
140
+
141
+ ## 2.0.0
142
+
143
+ ### Patch Changes
144
+
145
+ - @eventferry/core@2.0.0
146
+
147
+ ## 1.0.4
148
+
149
+ ### Patch Changes
150
+
151
+ - Updated dependencies [64d115d]
152
+ - @eventferry/core@1.0.4
153
+
154
+ ## 1.0.3
155
+
156
+ ### Patch Changes
157
+
158
+ - Updated dependencies [aaca9a2]
159
+ - @eventferry/core@1.0.3
160
+
161
+ ## 1.0.2
162
+
163
+ ### Patch Changes
164
+
165
+ - 89f1867: Declare `engines.node` (>=18) so npm shows the supported Node version and tooling can warn on unsupported runtimes.
166
+ - Updated dependencies [89f1867]
167
+ - @eventferry/core@1.0.2
168
+
169
+ ## 1.0.1
170
+
171
+ ### Patch Changes
172
+
173
+ - docs: polish per-package READMEs (npm page content). No code changes.
174
+ - Updated dependencies
175
+ - @eventferry/core@1.0.1
176
+
177
+ ## 1.0.0
178
+
179
+ ### Minor Changes
180
+
181
+ - b06f8ec: Add a low-latency notify-driven relay (Postgres `LISTEN`/`NOTIFY`).
182
+
183
+ - **core:** new `Waker` interface and an optional `Relay({ waker })`. The relay's
184
+ idle wait is now interruptible — when the waker signals, it claims immediately
185
+ instead of sleeping out `pollIntervalMs`. With no waker, behavior is unchanged.
186
+ - **postgres:** `PostgresNotifyWaker` holds a dedicated `LISTEN` connection and
187
+ wakes the relay on each notification, reconnecting with backoff if it drops.
188
+ `createNotifyTriggerSql(table, channel)` emits an `AFTER INSERT FOR EACH STATEMENT`
189
+ trigger that `pg_notify`s on commit (empty payload — the relay re-claims).
190
+ - Polling remains the safety net: a missed notification is caught by the next poll,
191
+ so no event is lost. All ordering/retry/DLQ/crash-recovery guarantees are unchanged.
192
+ - No new dependencies (`LISTEN`/`NOTIFY` is native to `pg`).
193
+
194
+ - b06f8ec: Add a retention helper: `PostgresStore.purgeDone({ olderThanMs, batchSize?, maxRows? })`.
195
+
196
+ Batch-deletes `done` rows whose `processed_at` is older than the cutoff and returns the
197
+ total removed, keeping the outbox table from growing unbounded. Run it periodically
198
+ (your own scheduler). Only `done` rows are purged.
199
+
200
+ Also adds `createRetentionIndexSql(table)` — an optional partial index over done rows
201
+ (`WHERE status = 2`) that speeds up the purge scan on high-volume tables; the default
202
+ indexes intentionally exclude done rows, so add this only if retention scans get slow.
203
+
204
+ - b06f8ec: Add a streaming relay that publishes straight from the Postgres WAL (logical replication).
205
+
206
+ - **postgres:** `PostgresStreamingRelay` consumes INSERTs on the outbox table via
207
+ `pg-logical-replication` + `pgoutput` (built-in, no DB extension) and publishes them
208
+ with no claim query on the happy path — lower DB load than the notify waker. A failed
209
+ publish is demoted to `failed`; an internal claim-based retry loop drains it with the
210
+ existing backoff / DLQ / dead handling. `pg-logical-replication` is a new **optional**
211
+ peer dependency, loaded only in streaming mode.
212
+ - **postgres:** `PostgresStore` gains `claimFailedOnly` (claims only `failed`/timed-out
213
+ `processing` rows, never `pending`) so the stream owns pending rows with no duplication.
214
+ `createPublicationSql(table, publication)` emits an idempotent insert-only publication.
215
+ - **core:** the record→message builder is extracted as `buildPublishable(record,
216
+ serializer)` and shared by `Relay` and the streaming relay (no behavior change).
217
+ - **At-least-once:** the slot's LSN is acknowledged only after a batch's side effects
218
+ commit; a crash re-streams and re-publishes (idempotent consumers absorb the duplicate).
219
+ - **Ordering:** streaming is best-effort per-aggregate (a retried failure may land after
220
+ later same-aggregate rows). Use the polling relay for the strict head-of-aggregate
221
+ guarantee. Requires `wal_level = logical`.
222
+
223
+ - b06f8ec: Strict per-aggregate ordering, crash recovery, and driver/packaging fixes.
224
+
225
+ - **postgres:** the claim query now enforces strict per-aggregate ordering by
226
+ only taking the _head_ of each aggregate (no earlier unfinished row for the
227
+ same `aggregateId`). At most one in-flight message per aggregate; failed
228
+ messages block their successors until resolved.
229
+ - **postgres:** added a `claimed_at` column and a visibility-timeout reaper
230
+ (`claimTimeoutMs`, default 60s) so rows orphaned by a crashed relay are
231
+ reclaimed instead of stuck in `processing` forever. Migration is upgrade-safe
232
+ (`ADD COLUMN IF NOT EXISTS`); the partial indexes were retuned for the new
233
+ ordered, reaper-aware claim.
234
+ - **core:** dead-lettered messages now carry the real `original-topic` header
235
+ (previously always empty); `ConsoleLogger` routes warn/error to the matching
236
+ `console` methods.
237
+ - **kafka:** the confluent driver now honors `acks` and `compression` (it
238
+ silently ignored them before), matching the kafkajs driver.
239
+ - **packaging:** the `@eventferry/postgres/migrations` subpath export now
240
+ advertises its types; `pnpm-workspace.yaml` dropped an invalid placeholder
241
+ block.
242
+
243
+ Note: `claimTimeoutMs` should exceed your worst-case publish latency. This is
244
+ an at-least-once system — pair it with idempotent producers/consumers.
245
+
246
+ - b06f8ec: Add W3C trace propagation (OpenTelemetry-compatible), dependency-free.
247
+
248
+ - **core:** new `Tracing` interface (`inject(carrier)`), the shape of an OpenTelemetry
249
+ `TextMapPropagator` — the library depends on no tracing package.
250
+ - **postgres:** `PostgresStore({ tracing })` captures the active W3C
251
+ `traceparent`/`tracestate` into the row's headers at `enqueue`, so it rides along to
252
+ the published message (on every path — polling, notify, streaming — since headers
253
+ already pass through) and the consumer can continue the trace.
254
+ - The caller's `headers` object is never mutated. With no `tracing` configured,
255
+ behavior is unchanged. The existing `trace-id` header stays for simple correlation.
256
+ - OpenTelemetry/Datadog/custom integrate via a ~5-line adapter (documented, not bundled).
257
+
258
+ ### Patch Changes
259
+
260
+ - b06f8ec: The streaming relay now creates its persistent logical replication slot on start if
261
+ it does not already exist (via `pg_create_logical_replication_slot`). Previously it
262
+ assumed the slot was pre-created and `subscribe` would fail otherwise. (Caught by the
263
+ new integration suite against real Postgres.)
264
+ - Updated dependencies [b06f8ec]
265
+ - Updated dependencies [b06f8ec]
266
+ - Updated dependencies [b06f8ec]
267
+ - Updated dependencies [b06f8ec]
268
+ - Updated dependencies [b06f8ec]
269
+ - @eventferry/core@1.0.0
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@eventferry/postgres",
3
- "version": "3.3.0",
3
+ "version": "3.3.2",
4
4
  "description": "PostgreSQL store for @eventferry (polling + lock-free claim)",
5
5
  "type": "module",
6
6
  "main": "./dist/index.cjs",
@@ -20,8 +20,12 @@
20
20
  },
21
21
  "files": [
22
22
  "dist",
23
- "sql"
23
+ "sql",
24
+ "CHANGELOG.md"
24
25
  ],
26
+ "publishConfig": {
27
+ "access": "public"
28
+ },
25
29
  "keywords": [
26
30
  "outbox",
27
31
  "outbox-pattern",
@@ -53,7 +57,7 @@
53
57
  "node": ">=18"
54
58
  },
55
59
  "dependencies": {
56
- "@eventferry/core": "3.3.0"
60
+ "@eventferry/core": "3.4.0"
57
61
  },
58
62
  "peerDependencies": {
59
63
  "pg": "^8.0.0",
@@ -70,8 +74,8 @@
70
74
  "pg-logical-replication": "^2.0.0",
71
75
  "tsup": "^8.3.5",
72
76
  "typescript": "^5.7.2",
73
- "vitest": "^2.1.8",
74
- "@eventferry/core": "3.3.0"
77
+ "vitest": "^3.2.6",
78
+ "@eventferry/core": "3.4.0"
75
79
  },
76
80
  "scripts": {
77
81
  "build": "tsup",