@gnldev/events 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/LICENSE ADDED
@@ -0,0 +1,201 @@
1
+ Apache License
2
+ Version 2.0, January 2004
3
+ http://www.apache.org/licenses/
4
+
5
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6
+
7
+ 1. Definitions.
8
+
9
+ "License" shall mean the terms and conditions for use, reproduction,
10
+ and distribution as defined by Sections 1 through 9 of this document.
11
+
12
+ "Licensor" shall mean the copyright owner or entity authorized by
13
+ the copyright owner that is granting the License.
14
+
15
+ "Legal Entity" shall mean the union of the acting entity and all
16
+ other entities that control, are controlled by, or are under common
17
+ control with that entity. For the purposes of this definition,
18
+ "control" means (i) the power, direct or indirect, to cause the
19
+ direction or management of such entity, whether by contract or
20
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
21
+ outstanding shares, or (iii) beneficial ownership of such entity.
22
+
23
+ "You" (or "Your") shall mean an individual or Legal Entity
24
+ exercising permissions granted by this License.
25
+
26
+ "Source" form shall mean the preferred form for making modifications,
27
+ including but not limited to software source code, documentation
28
+ source, and configuration files.
29
+
30
+ "Object" form shall mean any form resulting from mechanical
31
+ transformation or translation of a Source form, including but
32
+ not limited to compiled object code, generated documentation,
33
+ and conversions to other media types.
34
+
35
+ "Work" shall mean the work of authorship, whether in Source or
36
+ Object form, made available under the License, as indicated by a
37
+ copyright notice that is included in or attached to the work
38
+ (an example is provided in the Appendix below).
39
+
40
+ "Derivative Works" shall mean any work, whether in Source or Object
41
+ form, that is based on (or derived from) the Work and for which the
42
+ editorial revisions, annotations, elaborations, or other modifications
43
+ represent, as a whole, an original work of authorship. For the purposes
44
+ of this License, Derivative Works shall not include works that remain
45
+ separable from, or merely link (or bind by name) to the interfaces of,
46
+ the Work and Derivative Works thereof.
47
+
48
+ "Contribution" shall mean any work of authorship, including
49
+ the original version of the Work and any modifications or additions
50
+ to that Work or Derivative Works thereof, that is intentionally
51
+ submitted to Licensor for inclusion in the Work by the copyright owner
52
+ or by an individual or Legal Entity authorized to submit on behalf of
53
+ the copyright owner. For the purposes of this definition, "submitted"
54
+ means any form of electronic, verbal, or written communication sent
55
+ to the Licensor or its representatives, including but not limited to
56
+ communication on electronic mailing lists, source code control systems,
57
+ and issue tracking systems that are managed by, or on behalf of, the
58
+ Licensor for the purpose of discussing and improving the Work, but
59
+ excluding communication that is conspicuously marked or otherwise
60
+ designated in writing by the copyright owner as "Not a Contribution."
61
+
62
+ "Contributor" shall mean Licensor and any individual or Legal Entity
63
+ on behalf of whom a Contribution has been received by Licensor and
64
+ subsequently incorporated within the Work.
65
+
66
+ 2. Grant of Copyright License. Subject to the terms and conditions of
67
+ this License, each Contributor hereby grants to You a perpetual,
68
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
69
+ copyright license to reproduce, prepare Derivative Works of,
70
+ publicly display, publicly perform, sublicense, and distribute the
71
+ Work and such Derivative Works in Source or Object form.
72
+
73
+ 3. Grant of Patent License. Subject to the terms and conditions of
74
+ this License, each Contributor hereby grants to You a perpetual,
75
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
76
+ (except as stated in this section) patent license to make, have made,
77
+ use, offer to sell, sell, import, and otherwise transfer the Work,
78
+ where such license applies only to those patent claims licensable
79
+ by such Contributor that are necessarily infringed by their
80
+ Contribution(s) alone or by combination of their Contribution(s)
81
+ with the Work to which such Contribution(s) was submitted. If You
82
+ institute patent litigation against any entity (including a
83
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
84
+ or a Contribution incorporated within the Work constitutes direct
85
+ or contributory patent infringement, then any patent licenses
86
+ granted to You under this License for that Work shall terminate
87
+ as of the date such litigation is filed.
88
+
89
+ 4. Redistribution. You may reproduce and distribute copies of the
90
+ Work or Derivative Works thereof in any medium, with or without
91
+ modifications, and in Source or Object form, provided that You
92
+ meet the following conditions:
93
+
94
+ (a) You must give any other recipients of the Work or
95
+ Derivative Works a copy of this License; and
96
+
97
+ (b) You must cause any modified files to carry prominent notices
98
+ stating that You changed the files; and
99
+
100
+ (c) You must retain, in the Source form of any Derivative Works
101
+ that You distribute, all copyright, patent, trademark, and
102
+ attribution notices from the Source form of the Work,
103
+ excluding those notices that do not pertain to any part of
104
+ the Derivative Works; and
105
+
106
+ (d) If the Work includes a "NOTICE" text file as part of its
107
+ distribution, then any Derivative Works that You distribute must
108
+ include a readable copy of the attribution notices contained
109
+ within such NOTICE file, excluding those notices that do not
110
+ pertain to any part of the Derivative Works, in at least one
111
+ of the following places: within a NOTICE text file distributed
112
+ as part of the Derivative Works; within the Source form or
113
+ documentation, if provided along with the Derivative Works; or,
114
+ within a display generated by the Derivative Works, if and
115
+ wherever such third-party notices normally appear. The contents
116
+ of the NOTICE file are for informational purposes only and
117
+ do not modify the License. You may add Your own attribution
118
+ notices within Derivative Works that You distribute, alongside
119
+ or as an addendum to the NOTICE text from the Work, provided
120
+ that such additional attribution notices cannot be construed
121
+ as modifying the License.
122
+
123
+ You may add Your own copyright statement to Your modifications and
124
+ may provide additional or different license terms and conditions
125
+ for use, reproduction, or distribution of Your modifications, or
126
+ for any such Derivative Works as a whole, provided Your use,
127
+ reproduction, and distribution of the Work otherwise complies with
128
+ the conditions stated in this License.
129
+
130
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
131
+ any Contribution intentionally submitted for inclusion in the Work
132
+ by You to the Licensor shall be under the terms and conditions of
133
+ this License, without any additional terms or conditions.
134
+ Notwithstanding the above, nothing herein shall supersede or modify
135
+ the terms of any separate license agreement you may have executed
136
+ with Licensor regarding such Contributions.
137
+
138
+ 6. Trademarks. This License does not grant permission to use the trade
139
+ names, trademarks, service marks, or product names of the Licensor,
140
+ except as required for reasonable and customary use in describing the
141
+ origin of the Work and reproducing the content of the NOTICE file.
142
+
143
+ 7. Disclaimer of Warranty. Unless required by applicable law or
144
+ agreed to in writing, Licensor provides the Work (and each
145
+ Contributor provides its Contributions) on an "AS IS" BASIS,
146
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
147
+ implied, including, without limitation, any warranties or conditions
148
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
149
+ PARTICULAR PURPOSE. You are solely responsible for determining the
150
+ appropriateness of using or redistributing the Work and assume any
151
+ risks associated with Your exercise of permissions under this License.
152
+
153
+ 8. Limitation of Liability. In no event and under no legal theory,
154
+ whether in tort (including negligence), contract, or otherwise,
155
+ unless required by applicable law (such as deliberate and grossly
156
+ negligent acts) or agreed to in writing, shall any Contributor be
157
+ liable to You for damages, including any direct, indirect, special,
158
+ incidental, or consequential damages of any character arising as a
159
+ result of this License or out of the use or inability to use the
160
+ Work (including but not limited to damages for loss of goodwill,
161
+ work stoppage, computer failure or malfunction, or any and all
162
+ other commercial damages or losses), even if such Contributor
163
+ has been advised of the possibility of such damages.
164
+
165
+ 9. Accepting Warranty or Additional Liability. While redistributing
166
+ the Work or Derivative Works thereof, You may choose to offer,
167
+ and charge a fee for, acceptance of support, warranty, indemnity,
168
+ or other liability obligations and/or rights consistent with this
169
+ License. However, in accepting such obligations, You may act only
170
+ on Your own behalf and on Your sole responsibility, not on behalf
171
+ of any other Contributor, and only if You agree to indemnify,
172
+ defend, and hold each Contributor harmless for any liability
173
+ incurred by, or claims asserted against, such Contributor by reason
174
+ of your accepting any such warranty or additional liability.
175
+
176
+ END OF TERMS AND CONDITIONS
177
+
178
+ APPENDIX: How to apply the Apache License to your work.
179
+
180
+ To apply the Apache License to your work, attach the following
181
+ boilerplate notice, with the fields enclosed by brackets "[]"
182
+ replaced with your own identifying information. (Don't include
183
+ the brackets!) The text should be enclosed in the appropriate
184
+ comment syntax for the file format. We also recommend that a
185
+ file or class name and description of purpose be included on the
186
+ same "printed page" as the copyright notice for easier
187
+ identification within third-party archives.
188
+
189
+ Copyright 2026 Karaca Yılmaz (https://gnl.dev)
190
+
191
+ Licensed under the Apache License, Version 2.0 (the "License");
192
+ you may not use this file except in compliance with the License.
193
+ You may obtain a copy of the License at
194
+
195
+ http://www.apache.org/licenses/LICENSE-2.0
196
+
197
+ Unless required by applicable law or agreed to in writing, software
198
+ distributed under the License is distributed on an "AS IS" BASIS,
199
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200
+ See the License for the specific language governing permissions and
201
+ limitations under the License.
package/README.md ADDED
@@ -0,0 +1,183 @@
1
+ # @gnldev/events
2
+
3
+ A **durable event/notification bus** on top of the journal. `emit` writes to an append-only log; each
4
+ consumer processes events with its own ack marker: **exactly-once marking + at-least-once delivery**. The
5
+ marker is written **after** the handler succeeds — if the handler throws or the process dies, the event
6
+ isn't lost, a later poll retries it (on the backoff schedule below, until `maxAttempts`). The cost: between handler success and marking, a crash (or a
7
+ concurrent poll race) can cause redelivery → **write the handler as idempotent**, or use something durable
8
+ inside it (`runDurable`/`claim`). Fan-out: N consumers → each gets every event at least once, via its own
9
+ marker stream.
10
+
11
+ > Install: `pnpm add @gnldev/events` — or use it from a [repo clone](https://github.com/Karaca7/gnldev): `pnpm install && pnpm -r build`.
12
+
13
+ ```bash
14
+ npm i @gnldev/events # peer: @gnldev/durable
15
+ ```
16
+
17
+ ```ts
18
+ import { emit, createConsumer } from '@gnldev/events';
19
+ import { SqliteStorage } from '@gnldev/durable/sqlite';
20
+
21
+ const storage = new SqliteStorage('runs.db'); // emit/createConsumer take the WORK store
22
+
23
+ // Publish (idempotent: same id → a single event).
24
+ await emit(storage.work, 'refunds', { orderId: 'o1', amount: 50 }, { id: 'refund-o1' });
25
+
26
+ // Consume (each consumer name gets its own marker stream; the handler must be idempotent).
27
+ const consumer = createConsumer(storage.work, 'refunds', (payload, meta) => notify(payload), { name: 'emailer' });
28
+ await consumer.poll(); // or consumer.start()
29
+ ```
30
+
31
+ ## API
32
+ - `emit(work, topic, payload, { id?, maxDepth? }) → eventId` — `work` is `storage.work`
33
+ - `createConsumer(work, topic, handler, { name, pollMs?, backoff?, maxPollMs?, maxAttempts?, retryDelayMs? }) → { poll, start, stop }`
34
+ — `name` is required to separate fan-out acks; `maxAttempts`/`retryDelayMs` govern the dead-letter below.
35
+ - `listDeadEvents(work, topic, consumer) → DeadEvent[]` — everything quarantined for that consumer,
36
+ with `status` (`quarantined` | `released` | `delivered`), `attempts`, `error`, `releases` and the
37
+ original `payload`. A management call: it reads the whole topic log, so don't put it in a poll loop.
38
+ - `retryDeadEvent(work, topic, consumer, eventId) → boolean` — hands a quarantined event back for
39
+ delivery to that one consumer (fresh attempt budget, no backoff wait). Calling it again on an event
40
+ that is still `released` is safe and re-asserts the release. `false` has **three** causes, and the
41
+ third is not "nothing to do":
42
+ 1. the event was never quarantined;
43
+ 2. it has already been delivered (the ack marker is the single source of truth for that);
44
+ 3. **the release was abandoned** — the dead-letter record kept being rewritten underneath the
45
+ read-modify-write faster than it could complete (5 lost compare-and-swaps in a row: two
46
+ operators, two tabs, a retry script racing a human). Nothing was overwritten, it is logged on
47
+ `console.warn`, and the remedy is to **call it again**. Silently clobbering the other writer is
48
+ the alternative, and the compare-and-swap exists to refuse it: without it, two releases landing
49
+ together both read the same record and both wrote `releases: n + 1` from it — both returned
50
+ `true` and the record showed `releases: 1` (measured). Distinguish it from (1) and (2) by the
51
+ record itself: it is still there and still `quarantined`.
52
+
53
+ ## How it works
54
+ An append-only log per topic; consumer ack markers are separated by `name` → the same event flows to
55
+ multiple consumers. Since the marker is written after handler success, the delivery guarantee is
56
+ at-least-once: on a handler error the marker isn't written, that consumer's page cursor doesn't advance
57
+ past the event, and a later poll retries it; marking itself is exactly-once via CAS (`ackOnce`).
58
+
59
+ One failing event does **not** hold back the ones behind it: the pass keeps going through the rest of
60
+ the log, only the persisted bookmark stays parked. That parking is what makes retries safe, and it is
61
+ also why retries are finite — see the dead-letter below.
62
+
63
+ **A `:` in a topic, a consumer name or an event id is fine.** The bookkeeping keys are built by
64
+ joining those three on `:` (`evtack:<topic>:<consumer>:<id>`, and the same shape for `evtatt:`,
65
+ `evtdead:`, `evtcursor:`, `evtrescan:`), which is also a perfectly ordinary character in all three —
66
+ `billing:eu`, `orders:created`, a URN as an event id. Each part is therefore escaped before it is
67
+ joined: `:` → `%3A`, `%` → `%25`. Without that, topic `a` + consumer `b:c` and topic `a:b` + consumer
68
+ `c` produced the *same* ack key, so one consumer's marker made the other's event invisible — `poll()`
69
+ returned 0 forever, `listDeadEvents` was empty, nothing was logged. If you inspect the store by hand,
70
+ look for the escaped form (`evtack:refunds:billing%3Aeu:…`), not the literal name.
71
+
72
+ **The append-log namespace `evt:<topic>` is escaped too**, and that one is not belt-and-braces.
73
+ `WorkStore.list(ns)` is specified as a whole-value match — but the key a store *derives* from a
74
+ namespace is not something this package can see. The shipped Redis adapter stored records at
75
+ `wl:<ns>:<id>` and read a namespace back with a `SCAN MATCH wl:<ns>:*` prefix scan, so on a real
76
+ Redis 7 a consumer of topic `orders` was delivered topic `orders:eu`'s events (it never subscribed),
77
+ and `emit(topic: 'inv', { id: 'eu:x' })` returned an event id for a record `SET NX` had silently
78
+ refused, because topic `inv:eu` + id `x` had already claimed the same key. That adapter is fixed at
79
+ its own layer, which is the load-bearing repair; the topic is escaped here as well because
80
+ `WorkStore` is a public port, addressing a record by a concatenated key is an ordinary way to
81
+ implement it, and an escaped namespace is self-disambiguating under any implementation.
82
+
83
+ ## Retries + dead-letter (`maxAttempts`, `retryDelayMs`)
84
+ A handler that keeps throwing is retried on a spaced-out schedule, and after `maxAttempts` failed
85
+ attempts the event is **quarantined** (dead-lettered) for that consumer instead of retried forever.
86
+
87
+ - `maxAttempts` (default **8**) — the same `qatt → qfail` dead-letter shape as
88
+ [@gnldev/queue](../queue/README.md)'s `maxAttempts`, here `evtatt → evtdead`. `Infinity` disables
89
+ quarantine (retry forever) — delivery of other events is still unblocked, but this consumer's
90
+ bookmark stays parked behind the poison event, so every poll rescans the log from that point.
91
+ - `retryDelayMs` (default: exponential, 60s doubling to a 1h cap; a number = fixed spacing; `0` =
92
+ retry on the very next poll) — the wait before an event may be handed to the handler again. Without
93
+ it `maxAttempts` would count polls rather than time: at `pollMs: 200` the whole budget burns in about
94
+ a second, so a one-second downstream blip would dead-letter everything in flight. With the defaults
95
+ a failing event is given up on after **~2 hours**, not one second. A waiting event is skipped, never
96
+ lost: the bookmark stays frozen behind it exactly as for a failing one — which is what the
97
+ [load profile](#load-profile-while-an-event-is-failing-or-waiting) below is about.
98
+ The function form must return a finite number of milliseconds; if it throws or returns
99
+ `Infinity`/`NaN`/a non-number, that attempt falls back to the default schedule and logs a warning
100
+ (one consumer's broken schedule must not stop the topic's other deliveries). "Retry forever" is
101
+ `maxAttempts: Infinity`, not a delay of `Infinity`.
102
+ - A quarantined event is **not** acked — quarantine records that the consumer never saw it. It is
103
+ logged at `console.error` when it happens, listed by `listDeadEvents`, and comes back only through
104
+ `retryDeadEvent`, which is per-consumer (re-emitting would fan out to healthy consumers too).
105
+ Dead-letter records are kept as history after a successful retry, like queue's `qfail`.
106
+
107
+ ```ts
108
+ import { emit, createConsumer, listDeadEvents, retryDeadEvent } from '@gnldev/events';
109
+ import { SqliteStorage } from '@gnldev/durable/sqlite';
110
+
111
+ const storage = new SqliteStorage('runs.db');
112
+ await emit(storage.work, 'refunds', { orderId: 'o1' }, { id: 'refund-o1' });
113
+
114
+ const consumer = createConsumer(
115
+ storage.work,
116
+ 'refunds',
117
+ async (payload: { orderId: string }) => {
118
+ await paymentApi.refund(payload.orderId); // must be idempotent — delivery is at-least-once
119
+ },
120
+ {
121
+ name: 'refunder',
122
+ maxAttempts: 6, // quarantine after 6 failed attempts (default 8)
123
+ retryDelayMs: (attempt) => Math.min(30_000 * 2 ** (attempt - 1), 15 * 60_000),
124
+ },
125
+ );
126
+ await consumer.poll();
127
+
128
+ // The operator surface: what was given up on, and how to hand it back once the cause is fixed.
129
+ for (const dead of await listDeadEvents(storage.work, 'refunds', 'refunder')) {
130
+ if (dead.status !== 'quarantined') continue; // 'released' / 'delivered' are history, not a backlog
131
+ console.error(`refund ${dead.id} failed ${dead.attempts}x: ${dead.error}`);
132
+ await retryDeadEvent(storage.work, 'refunds', 'refunder', dead.id);
133
+ }
134
+ ```
135
+
136
+ ## Load profile while an event is failing or waiting
137
+
138
+ A consumer's persisted bookmark cannot move past an event that is still retryable — that is what
139
+ makes a retry safe rather than a silent skip. The price is that **every poll re-scans the log from
140
+ that point to the end**, and this is the *default* path, not something only `maxAttempts: Infinity`
141
+ opts into. Plan capacity for it.
142
+
143
+ Measured (real SQLite, 5 000 events, one failing event at index 10, default page size 50):
144
+
145
+ | poll | `work.list` | `work.get` | note |
146
+ |---|---|---|---|
147
+ | pass 1, delivering | 100 | 15 003 | 3 gets per undelivered event (ack + dead-letter + attempt) |
148
+ | pass 2–3, idle, **bookmark frozen** | 100 | 5 004 | 1 get per already-acked event, every poll |
149
+ | after the event is quarantined | 1 | 52 | only the last partial page — the healthy steady state |
150
+
151
+ That is a **~96×** difference in reads per poll between a frozen consumer and a healthy one, and it
152
+ scales with the length of the topic log, not with the number of events actually waiting. On Postgres
153
+ each of those `get`s is a network round-trip.
154
+
155
+ In wall clock, with two consumers on the same 50 000-event log (file-backed SQLite, one machine): the
156
+ healthy consumer's idle poll takes **56–67 ms**, the frozen one's **8.9–10.3 s**. Same log, same
157
+ store — the only difference is where the bookmark sits.
158
+
159
+ Two things make the window longer than you might expect:
160
+
161
+ - **`retryDelayMs` widened it from ~1 s to ~2 h.** The bookmark stays frozen for the whole retry
162
+ schedule, not just while the handler is actually failing. Before spaced retries the budget burned
163
+ out in about a second; the default now spends 8 attempts over roughly two hours, so a single poison
164
+ event holds the bookmark for that long before quarantine releases it.
165
+ - **Empty-poll backoff does not help a busy topic.** It only grows the interval when a poll delivers
166
+ *nothing* (`delivered > 0` resets it), so on a topic that is otherwise healthy the consumer keeps
167
+ polling at `pollMs` — 5 polls per second at the default 200 ms — and pays the full rescan each time.
168
+ On a fully idle topic the interval does reach `pollMs * 32` (6.4 s by default), which does help.
169
+
170
+ This is not a regression: the old behavior was worse (one poison event blocked every event behind it
171
+ indefinitely). It is a documented change in cost. If it matters for your topic: lower `maxAttempts` or
172
+ `retryDelayMs` so quarantine arrives sooner, raise `pollMs`, or keep topics short with retention
173
+ sweeps so "the rest of the log" stays small.
174
+
175
+ ## Empty-poll backoff (on by default)
176
+ If `start()` delivers no events on a given `poll()` call, the next wait starts at `pollMs` and doubles
177
+ (cap: `maxPollMs ?? pollMs*32`); it resets to `pollMs` as soon as an event is delivered. This prevents a
178
+ large number of consumers on an empty topic from generating tens of thousands of empty queries per second
179
+ (a poll storm). `backoff: false` reverts to the old fixed-interval behavior.
180
+
181
+ ## License
182
+
183
+ Apache-2.0 — see [LICENSE](./LICENSE).
@@ -0,0 +1,127 @@
1
+ import type { WorkStore } from '@gnldev/durable';
2
+ export interface EventMeta {
3
+ id: string;
4
+ topic: string;
5
+ }
6
+ export type EventHandler = (payload: any, meta: EventMeta) => Promise<void> | void;
7
+ export interface ConsumerOptions {
8
+ /** Consumer identity — ack markers are separated by this (fan-out). */
9
+ name: string;
10
+ /** start() poll interval (also the base for backoff growth on an empty queue). */
11
+ pollMs?: number;
12
+ /**
13
+ * Empty-poll exponential backoff (default ON): if poll() delivers 0 events, the next poll
14
+ * interval grows ×2 (ceiling: `maxPollMs ?? pollMs*32`) → prevents tens of thousands of empty
15
+ * queries per second on an empty topic with many consumers (poll storm, audit finding). Once
16
+ * something is delivered, the interval resets to `pollMs`. `false` → old behavior (fixed `pollMs` interval).
17
+ */
18
+ backoff?: boolean;
19
+ /** Backoff ceiling (default `pollMs*32`). Only meaningful when `backoff !== false`. */
20
+ maxPollMs?: number;
21
+ /**
22
+ * How many FAILED delivery attempts before an event is quarantined (dead-lettered) for THIS
23
+ * consumer. Default 8, spread over ~2 hours by `retryDelayMs` — the same evtatt→evtdead shape as
24
+ * @gnldev/queue's qatt→qfail, so an operator learns one dead-letter ontology, not two. (The number
25
+ * is 8 rather than queue's 5 because these attempts are TIME-spaced: see `retryDelayMs`.)
26
+ * `Infinity` = never quarantine (retry forever): delivery of OTHER events is still not blocked
27
+ * (see the `frozen` cursor below), but this consumer's cursor stays parked behind the poison event
28
+ * forever, so every poll re-scans the whole log from that point (the O(n) marker-check cost this
29
+ * package spent 5.1 removing). Opt into that knowingly.
30
+ */
31
+ maxAttempts?: number;
32
+ /**
33
+ * How long to WAIT after a failed attempt before the event may be handed to the handler again.
34
+ * Default: exponential from 60s, ×2 per attempt, capped at 1h — the same
35
+ * `Math.min(base * 2 ** (attempt - 1), cap)` shape @gnldev/scheduler uses for its retries, with an
36
+ * events-sized base/cap. A number = fixed spacing; `0` = retry on the very next poll.
37
+ *
38
+ * Why this exists: `maxAttempts` counts POLLS, not time. With the default 200ms poll interval a
39
+ * 5-attempt budget burned out in about one second, so a one-second downstream blip quarantined
40
+ * every in-flight event — permanently, needing an operator's `retryDeadEvent`. A retry budget is
41
+ * only a real budget if the attempts are spread over the kind of outage it is meant to survive:
42
+ * 8 attempts × this schedule = ~2h (60+120+240+480+960+1920+3600 seconds of waiting).
43
+ *
44
+ * A not-yet-due event is SKIPPED (not handed to the handler) but still FREEZES the consumer's
45
+ * cursor — it is retryable, so nothing behind it may be marked as passed. Waiting is not giving up.
46
+ *
47
+ * CONTRACT for the function form: it must return a FINITE number of milliseconds. If it throws, or
48
+ * returns Infinity/NaN/a non-number, the default schedule is used for that attempt and a warning is
49
+ * logged — the delivery of every OTHER event on the topic is not the place to pay for a broken
50
+ * schedule (a throw here used to escape poll() and stop the whole pass), and Infinity is not
51
+ * storable: it survives in memory as "never due again" but a SQLite/Postgres WorkStore round-trips
52
+ * it through JSON to null → 0 → due immediately, i.e. the same code behaving oppositely per adapter.
53
+ * "Retry forever" is `maxAttempts: Infinity`, not a delay of Infinity.
54
+ */
55
+ retryDelayMs?: number | ((attempt: number) => number);
56
+ }
57
+ /** A delivery that reached `maxAttempts` failures for one consumer, as stored under `evtdead:*`. */
58
+ interface StoredDead {
59
+ error: string;
60
+ attempts: number;
61
+ at: number;
62
+ /** Set by `retryDeadEvent` — released back for delivery, not yet succeeded. */
63
+ releasedAt?: number;
64
+ /** How many times it has been released (quarantine → release → quarantine again). */
65
+ releases?: number;
66
+ }
67
+ /** An event that has been quarantined for a consumer at least once — the operator-facing view. */
68
+ export interface DeadEvent extends StoredDead {
69
+ id: string;
70
+ topic: string;
71
+ consumer: string;
72
+ payload: unknown;
73
+ /**
74
+ * `quarantined` = parked, will NOT be delivered until released. `released` = `retryDeadEvent` has
75
+ * handed it back, awaiting the next poll. `delivered` = it eventually succeeded; the record is kept
76
+ * as history (same choice as queue's qfail: an append-only log, dead-letter history is permanent for audit).
77
+ */
78
+ status: 'quarantined' | 'released' | 'delivered';
79
+ }
80
+ export interface Consumer {
81
+ /**
82
+ * Deliver this consumer's not-yet-marked events to the handler. Returns the count of
83
+ * successfully marked (ackOnce won) deliveries. Delivery is at-least-once: if the handler
84
+ * throws, the event is skipped (marker not written) and retried on a later poll, once its
85
+ * `retryDelayMs` backoff has elapsed → the handler should be idempotent. Until then the event is
86
+ * skipped but the cursor stays frozen behind it (waiting is not giving up, and not loss either).
87
+ * After `maxAttempts` failures an event is QUARANTINED (`evtdead:*`,
88
+ * visible via `listDeadEvents`) and stops being redelivered until `retryDeadEvent` releases it.
89
+ */
90
+ poll(): Promise<number>;
91
+ start(): void;
92
+ stop(): void;
93
+ }
94
+ /**
95
+ * Phase 8 (audit finding: unbounded accumulation): if `maxDepth` is given — throws if, before
96
+ * publishing, the topic depth (the TOTAL record count in the `evt:<topic>` namespace: delivered +
97
+ * undelivered, an append-only log can't distinguish these without pruning) has reached/exceeded `maxDepth`.
98
+ */
99
+ export declare class EventDepthExceededError extends Error {
100
+ readonly detail: {
101
+ topic: string;
102
+ depth: number;
103
+ maxDepth: number;
104
+ };
105
+ constructor(message: string, detail: {
106
+ topic: string;
107
+ depth: number;
108
+ maxDepth: number;
109
+ });
110
+ }
111
+ /** Publish an event to a topic (idempotent: same id → a single event again). Returns eventId. */
112
+ export declare function emit(work: WorkStore, topic: string, payload: unknown, opts?: {
113
+ id?: string;
114
+ maxDepth?: number;
115
+ }): Promise<string>;
116
+ export declare function createConsumer(work: WorkStore, topic: string, handler: EventHandler, opts: ConsumerOptions): Consumer;
117
+ /**
118
+ * Every event that has been quarantined for `consumer` on `topic` — including ones later released
119
+ * and delivered (`status`), because dead-letter history is permanent for audit (same choice as
120
+ * queue's qfail: nothing is deleted from an append-only log).
121
+ *
122
+ * MANAGEMENT function: it reads the WHOLE topic log and does one `get` per event. Do NOT call it
123
+ * from a poll loop — the loop reads only the pages it needs, via cursor (5.1).
124
+ */
125
+ export declare function listDeadEvents(work: WorkStore, topic: string, consumer: string): Promise<DeadEvent[]>;
126
+ export declare function retryDeadEvent(work: WorkStore, topic: string, consumer: string, eventId: string): Promise<boolean>;
127
+ export {};
package/dist/index.js ADDED
@@ -0,0 +1,517 @@
1
+ // @gnldev/events — durable event/notification bus on top of WorkStore. emit writes to an append-only log;
2
+ // each consumer processes an event with its own ack marker (ackOnce=CAS): exactly-once MARKING +
3
+ // at-least-once DELIVERY. The marker is written AFTER the handler SUCCEEDS → if the handler throws/the
4
+ // process dies, the event isn't lost — a later poll retries it, spaced out by `retryDelayMs` (default
5
+ // exponential 60s→1h) so a short downstream outage can't burn the whole attempt budget in a second.
6
+ // Cost: a crash between handler success
7
+ // and ackOnce (or a concurrent poll race) → redelivery is possible. Write the handler idempotently, or
8
+ // use durable (runDurable/claim) inside the handler. Fan-out: N consumers → each gets every event at least once.
9
+ // (WorkStore keeps it in its own namespace → doesn't pollute the RunJournal/replay reader.)
10
+ // A handler that keeps throwing does NOT hold the topic hostage: other events keep being delivered
11
+ // in the same pass, and after `maxAttempts` the event is QUARANTINED (dead-letter) rather than
12
+ // retried forever — listDeadEvents() shows it, retryDeadEvent() hands it back. Quarantine is not an
13
+ // ack: a quarantined event is never counted as delivered, because the consumer never saw it.
14
+ import { createPollLoop } from '@gnldev/durable';
15
+ /**
16
+ * Phase 8 (audit finding: unbounded accumulation): if `maxDepth` is given — throws if, before
17
+ * publishing, the topic depth (the TOTAL record count in the `evt:<topic>` namespace: delivered +
18
+ * undelivered, an append-only log can't distinguish these without pruning) has reached/exceeded `maxDepth`.
19
+ */
20
+ export class EventDepthExceededError extends Error {
21
+ detail;
22
+ constructor(message, detail) {
23
+ super(message);
24
+ this.detail = detail;
25
+ this.name = 'EventDepthExceededError';
26
+ }
27
+ }
28
+ /**
29
+ * Counts records in the `ns` namespace only UP TO `limit` (early exit). WorkStore.list is paged
30
+ * (default page size e.g. 50) — an exact count would read O(depth/pageSize) pages; here it's enough
31
+ * to know "was the limit exceeded", so it stops once it reaches `limit` → cost is
32
+ * O(min(actual depth, maxDepth)) pages, NOT the ENTIRE log. Unless `maxDepth` is given (default
33
+ * behavior), this function is NEVER called → existing unbounded-topic behavior is preserved.
34
+ */
35
+ async function countUpTo(work, ns, limit) {
36
+ let count = 0;
37
+ let cursor;
38
+ for (;;) {
39
+ const page = await work.list(ns, { cursor });
40
+ count += page.items.length;
41
+ if (count >= limit || !page.nextCursor)
42
+ return count;
43
+ cursor = page.nextCursor;
44
+ }
45
+ }
46
+ /** Publish an event to a topic (idempotent: same id → a single event again). Returns eventId. */
47
+ export async function emit(work, topic, payload, opts = {}) {
48
+ if (opts.maxDepth != null) {
49
+ const depth = await countUpTo(work, logNsOf(topic), opts.maxDepth);
50
+ if (depth >= opts.maxDepth) {
51
+ throw new EventDepthExceededError(`@gnldev/events: topic depth limit exceeded (${depth} >= ${opts.maxDepth}) — event rejected (topic='${topic}').`, { topic, depth, maxDepth: opts.maxDepth });
52
+ }
53
+ }
54
+ return work.append(logNsOf(topic), payload, opts.id);
55
+ }
56
+ /**
57
+ * Escapes ONE component of a `:`-delimited marker key, so that different components can never
58
+ * produce the same key.
59
+ *
60
+ * WHY: the keys below are built by concatenation, and every component — topic, consumer name, event
61
+ * id — is a string the CALLER chose. `:` is a perfectly ordinary character in all three
62
+ * (`billing:eu`, `orders:created`, an URN as an event id), and it is also the delimiter. Measured
63
+ * without this: topic `a` + consumer `b:c` and topic `a:b` + consumer `c` both produce
64
+ * `evtack:a:b:c:order-42`, so the first consumer's ack marker made the second one's event
65
+ * invisible — `poll()` returned 0 forever, `listDeadEvents` was empty, nothing was logged. Silent,
66
+ * permanent loss of exactly the at-least-once delivery this package is for, from two name pairs
67
+ * that are each entirely reasonable on their own. (Pinned per key family in key-ambiguity.test.ts.)
68
+ *
69
+ * Only `:` (the delimiter) and `%` (the escape character itself) are rewritten. That makes the
70
+ * escape INJECTIVE — the two escape sequences are the only way a `%` can appear in the output, so a
71
+ * key decomposes back to exactly one component triple — and injective is the whole requirement.
72
+ *
73
+ * NOT `encodeURIComponent`: it throws `URIError` on a lone surrogate, which a truncated UTF-16
74
+ * string yields, and these key builders run inside `poll()` where `createPollLoop` swallows throws.
75
+ * That would make a key escape a new way to silently stop the poll loop — the same failure
76
+ * `delayFor` exists to keep caller input from causing. This escape is TOTAL: every string has one.
77
+ *
78
+ * THE `evt:<topic>` LOG NAMESPACE IS ESCAPED TOO, and this note used to say the opposite ("topic is
79
+ * its last and only variable component there, so that name is already unambiguous... a collision that
80
+ * cannot happen"). The name is indeed unambiguous; the reasoning was still wrong, because it argued
81
+ * about the NAME while the failure is in the KEY THE STORE DERIVES FROM IT, which this package cannot
82
+ * see. `WorkStore.list(ns)` is specified as a whole-value match, but RedisWorkStore had no ns column:
83
+ * it stored records at `wl:<ns>:<id>` and read a namespace back with a `SCAN MATCH wl:<ns>:*` PREFIX
84
+ * scan. Measured on a real Redis 7: a consumer of topic `orders` was delivered topic `orders:eu`'s
85
+ * events (it never subscribed), and `emit(topic='inv', id='eu:x')` returned an event id for a record
86
+ * `SET NX` had silently refused, because topic `inv:eu` + id `x` had already taken the same key — no
87
+ * dead-letter row, no log line, the loss class this escape exists to remove. (Pinned in
88
+ * log-namespace.test.ts.)
89
+ *
90
+ * The adapter is fixed at its own layer (redis-storage.ts `encNs`) and that fix is the load-bearing
91
+ * one — it restores the port contract for EVERY caller, not just this package. Escaping here changes
92
+ * no behavior on any of the four shipped adapters once it has landed. It stays because `WorkStore` is
93
+ * a PUBLIC port: a store that addresses a record by a concatenated key is an ordinary way to
94
+ * implement it, ours did it for months, and an escaped namespace is self-disambiguating under any of
95
+ * them. A rule with an exception clause is what failed here; this package now has the rule without it.
96
+ *
97
+ * `encNs` is a SECOND IMPLEMENTATION of this same three-line escape, and deliberately so: one escape
98
+ * SHAPE (`%3A`/`%25`), two layers that must be able to fail independently. Sharing a symbol would
99
+ * mean deleting either one silently reconfigures the other — the opposite of the defence in depth
100
+ * the paragraph above argues for. The two are pinned separately, and the `%` half of each is what
101
+ * distinguishes them: log-namespace.test.ts asserts it once through this package (a topic literally
102
+ * named `inv%3Aeu`) and once straight against the adapter (a namespace named `a%3Ab`).
103
+ *
104
+ * COST: a name (or event id) that actually contains `:` or `%` is now stored under a different key
105
+ * than it was, and — since the namespace is escaped as well — a topic containing one is stored in a
106
+ * different NAMESPACE than it was (`evt:orders:eu` → `evt:orders%3Aeu`), which on SQLite/Postgres is
107
+ * a `gnl_work_log.ns` value, i.e. the events themselves move, not just their markers. Nothing is
108
+ * published, so no store holds one yet; on a live deployment the old markers would be orphaned and
109
+ * every event redelivered once, and a colon-topic's existing log would go unread until its `ns` was
110
+ * renamed. Names without `:`/`%` — every test, every example, every documented key in the README —
111
+ * are byte-identical before and after.
112
+ */
113
+ const enc = (part) => part.replace(/%/g, '%25').replace(/:/g, '%3A');
114
+ // Marker keys. Inlined at two places before (the consumer and nothing else); now the management
115
+ // functions (listDeadEvents/retryDeadEvent) address the SAME keys, and a key format that two call
116
+ // sites have to agree on is a key format that must exist in exactly one place.
117
+ /** The append-log namespace an event lives in. The sixth key family — see `enc`. */
118
+ const logNsOf = (topic) => `evt:${enc(topic)}`;
119
+ const ackKey = (topic, consumer, id) => `evtack:${enc(topic)}:${enc(consumer)}:${enc(id)}`;
120
+ const attKey = (topic, consumer, id) => `evtatt:${enc(topic)}:${enc(consumer)}:${enc(id)}`;
121
+ const deadKey = (topic, consumer, id) => `evtdead:${enc(topic)}:${enc(consumer)}:${enc(id)}`;
122
+ const cursorKeyOf = (topic, consumer) => `evtcursor:${enc(topic)}:${enc(consumer)}`;
123
+ const rescanKeyOf = (topic, consumer) => `evtrescan:${enc(topic)}:${enc(consumer)}`;
124
+ /** Reads an `evtatt:*` value in either shape (see StoredAttempts). `undefined` = never failed. */
125
+ function readAttempts(raw) {
126
+ if (raw == null)
127
+ return undefined;
128
+ if (typeof raw === 'number')
129
+ return { n: raw, firstAt: 0, nextAt: 0 }; // pre-spacing bare count → due now
130
+ const r = raw;
131
+ return {
132
+ n: r.n ?? 0, firstAt: r.firstAt ?? 0, nextAt: r.nextAt ?? 0,
133
+ ...(r.gen !== undefined ? { gen: r.gen } : {}),
134
+ };
135
+ }
136
+ /**
137
+ * The identity of a RELEASE GENERATION. poll() reads the dead record before calling the handler and
138
+ * writes the quarantine record after — an arbitrarily long window in which `retryDeadEvent` can land.
139
+ * Comparing this stamp before/after is how an intervening release is detected, and it has to be a
140
+ * pair: `releasedAt` alone is a wall clock (two releases in the same millisecond compare equal) and
141
+ * `releases` alone doesn't move on the very first release. `0` for "no record / never released".
142
+ */
143
+ const releaseStamp = (rec) => `${rec?.releases ?? 0}@${rec?.releasedAt ?? 0}`;
144
+ /** The default retry schedule: @gnldev/scheduler's backoff shape (src/index.ts:104), events-sized. */
145
+ const defaultRetryDelay = (attempt) => Math.min(60_000 * 2 ** (attempt - 1), 3_600_000);
146
+ export function createConsumer(work, topic, handler, opts) {
147
+ const ns = logNsOf(topic);
148
+ const pollMs = opts.pollMs ?? 200;
149
+ const backoffOn = opts.backoff ?? true;
150
+ const maxPollMs = opts.maxPollMs ?? pollMs * 32;
151
+ const maxAttempts = opts.maxAttempts ?? 8;
152
+ const retryDelay = typeof opts.retryDelayMs === 'function' ? opts.retryDelayMs
153
+ : opts.retryDelayMs != null ? () => opts.retryDelayMs
154
+ : defaultRetryDelay;
155
+ // Read cursor specific to this consumer (WorkStore KV): persistently holds the position "all
156
+ // events before this are a fully-scanned page for this consumer" → subsequent polls won't list
157
+ // previously (fully processed) pages again (5.1: fixes starvation + O(n)-per-poll at 50+ events).
158
+ // Fan-out is unaffected: each consumer keeps its own cursor + its own ackOnce marker based on its
159
+ // `opts.name`. Old (cursor-less) event logs also stream from the start (cursor=undefined).
160
+ const cursorKey = cursorKeyOf(topic, opts.name);
161
+ const rescanKey = rescanKeyOf(topic, opts.name);
162
+ /**
163
+ * `retryDelayMs` is CALLER CODE called from inside the catch block, where nothing was catching it.
164
+ * A throwing schedule (a config read that blows up, an off-by-one on an array lookup) escaped
165
+ * poll() entirely: measured on a 10-event topic with a throwing delay fn — 1 event delivered, the
166
+ * attempt counter never written, and every later poll dying at the same line. Under `start()`
167
+ * createPollLoop swallows it, so that is a SILENT full stop — the exact head-of-line blocking this
168
+ * package exists to remove, re-entering through the caller's own hook. A bad schedule may cost the
169
+ * caller its schedule; it may not cost the topic its delivery.
170
+ *
171
+ * Non-finite is rejected for a different reason: it is not STORABLE. `Infinity` survives in an
172
+ * InMemory WorkStore (the event is never due again — a permanent freeze that isn't quarantine and
173
+ * isn't visible to listDeadEvents), but a SQLite/Postgres WorkStore round-trips it through JSON to
174
+ * `null` → read back as `0` → due IMMEDIATELY. Same code, same options, opposite behavior per
175
+ * adapter. There is no honest clamp for "infinity" either, so it falls back like a throw does:
176
+ * `maxAttempts: Infinity` is how "retry forever" is spelled here, not a delay of Infinity.
177
+ */
178
+ function delayFor(attempt, eventId) {
179
+ let raw;
180
+ try {
181
+ raw = retryDelay(attempt);
182
+ }
183
+ catch (e) {
184
+ console.warn(`@gnldev/events: retryDelayMs threw (topic=${topic}, consumer=${opts.name}, event=${eventId}, attempt=${attempt}) — falling back to the default 60s→1h schedule:`, e);
185
+ return defaultRetryDelay(attempt);
186
+ }
187
+ if (typeof raw !== 'number' || !Number.isFinite(raw)) {
188
+ console.warn(`@gnldev/events: retryDelayMs returned ${String(raw)} (topic=${topic}, consumer=${opts.name}, event=${eventId}, attempt=${attempt}) — a delay must be a finite number of milliseconds (it has to survive the store's JSON round-trip); falling back to the default 60s→1h schedule.`);
189
+ return defaultRetryDelay(attempt);
190
+ }
191
+ return Math.max(0, raw);
192
+ }
193
+ async function poll() {
194
+ // A release (retryDeadEvent) hands an event back that sits BEHIND the persisted cursor — the
195
+ // cursor moved past it precisely because quarantine made it terminal. One flagged full pass is
196
+ // how it gets back in view. It costs one extra scan of already-acked ids (cheap `get`s, the
197
+ // same cost the old locked cursor paid on EVERY poll) and it is idempotent: the flag is cleared
198
+ // only after a pass completes, so a crash mid-pass just rescans again.
199
+ const rescan = (await work.get(rescanKey)) === true;
200
+ // The stored cursor is dropped, not just ignored for one pass. Quarantining the event is what
201
+ // moved the cursor PAST it, so a pass that merely starts from the beginning would re-deliver it
202
+ // and then — because a failed release freezes the cursor and never persists anything — fall back
203
+ // to the same stale forward position on the next poll, leaving the event `released` and
204
+ // unreachable forever. (Found by mutation: no test failed when retryDeadEvent stopped resetting
205
+ // the attempt counter, and writing that test surfaced this instead.) `''` is the "start of log"
206
+ // Sentinel — WorkStore KV has no delete, so a key cannot be returned to absent.
207
+ //
208
+ // This rewind is NOT protected from concurrent writers, and the comment here used to claim it
209
+ // was ("only poll() writes the cursor"). poll() is the only cursor writer, but running two
210
+ // processes under the SAME consumer name is a supported way to scale a consumer (see the
211
+ // "already DELIVERED by another process" branch below), so there are as many cursor writers as
212
+ // there are pollers: another process's pass can persist a forward cursor right after this
213
+ // rewind and put the released event back out of view. That failure mode is the one
214
+ // retryDeadEvent already documents — the release "silently didn't take", the remedy is to
215
+ // release again, and `listDeadEvents` keeps showing it as `released` and undelivered, so it is
216
+ // observable rather than lost. What IS safe here is crashing right after the rewind: a rescan
217
+ // is a superset of a normal pass and skips acked ids by marker.
218
+ if (rescan)
219
+ await work.put(cursorKey, '');
220
+ let cursor = rescan ? undefined : (await work.get(cursorKey)) || undefined;
221
+ let delivered = 0;
222
+ // Has a RETRYABLE (failed, not yet quarantined) event been seen in this pass? Once true the
223
+ // persisted cursor stops moving — but the pass KEEPS GOING through the remaining pages.
224
+ let frozen = false;
225
+ for (;;) {
226
+ const page = await work.list(ns, { cursor });
227
+ for (const e of page.items) {
228
+ const ack = ackKey(topic, opts.name, e.id);
229
+ // Cheap "already marked?" check (ackOnce markers live in the same work-KV space) →
230
+ // events completed in previous polls don't go to the handler again.
231
+ if ((await work.get(ack)) !== undefined)
232
+ continue;
233
+ // Quarantined (and not released): TERMINAL for this consumer — not delivered, and
234
+ // deliberately NOT ack-marked, because "we gave up on it" is not "the consumer saw it".
235
+ // It doesn't freeze the cursor either; that is the whole point of quarantining. It is not
236
+ // silent: it was logged at console.error when it happened and it is listed by listDeadEvents.
237
+ const dk = deadKey(topic, opts.name, e.id);
238
+ const dead = await work.get(dk);
239
+ if (dead && !dead.releasedAt)
240
+ continue;
241
+ // Failed before and the backoff hasn't elapsed → NOT handed to the handler. The bookmark is
242
+ // frozen all the same: a waiting event is still retryable, so advancing past it would be the
243
+ // same silent loss as advancing past a failing one. (Without this, `maxAttempts` counted
244
+ // polls: at pollMs=200 the whole budget burned in ~1s and a one-second outage dead-lettered
245
+ // everything in flight. See ConsumerOptions.retryDelayMs.) COST: one extra KV `get` per
246
+ // not-yet-acked event per pass (2 → 3, alongside the ack and dead-letter checks). It is paid
247
+ // once per event on a healthy topic — an acked event never reaches this line — and there is
248
+ // no cheaper place to keep it: the due time has to survive a restart, so it lives in the store.
249
+ const attRaw = await work.get(attKey(topic, opts.name, e.id));
250
+ const att = readAttempts(attRaw);
251
+ if (att && att.nextAt > Date.now()) {
252
+ frozen = true;
253
+ continue;
254
+ }
255
+ // Contract: exactly-once MARKING + at-least-once DELIVERY. The marker is written AFTER
256
+ // the handler; a crash between handler success and ackOnce (or a concurrent poll race) →
257
+ // redelivery is possible. Write the handler idempotently, or use durable (runDurable/claim) inside it.
258
+ try {
259
+ await handler(e.payload, { id: e.id, topic });
260
+ }
261
+ catch (err) {
262
+ // Handler threw → ack marker NOT WRITTEN → the event is not lost. The poll loop doesn't
263
+ // die: this event is skipped, the rest of the page keeps processing.
264
+ //
265
+ // FRESH read of the dead record first. `dead` above was read BEFORE the handler ran, and
266
+ // the handler can take arbitrarily long — long enough for an operator's retryDeadEvent to
267
+ // land. Writing this attempt's bookkeeping from the stale read would undo that release
268
+ // (drop `releasedAt` → back to `quarantined`, rewind `releases`, overwrite the attempt
269
+ // reset) AFTER retryDeadEvent had already returned `true` to the operator. An attempt that
270
+ // belongs to the previous release generation may not touch the new one at all.
271
+ const fresh = await work.get(dk);
272
+ if (releaseStamp(fresh) !== releaseStamp(dead)) {
273
+ frozen = true; // released mid-flight → still live, and it gets the release's fresh budget
274
+ console.warn(`@gnldev/events: handler errored (topic=${topic}, consumer=${opts.name}, event=${e.id}) but the event was RELEASED while it ran — this attempt is discarded, the release stands:`, err);
275
+ continue;
276
+ }
277
+ const now = Date.now();
278
+ const n = (att?.n ?? 0) + 1;
279
+ // An event that reached its LAST attempt is about to become an operator's problem, so it
280
+ // is worth one extra `get` (only on this branch — never on the healthy or the still-
281
+ // retrying path) to check it is still ours to give up on. Two processes under the SAME
282
+ // consumer name is the supported way to scale a consumer: the other one can have delivered
283
+ // and ACKED this event while our attempt was in flight, and our late failure would then
284
+ // print "event QUARANTINED" and file a permanent dead-letter record for a delivery that
285
+ // SUCCEEDED. listDeadEvents reports it honestly as `delivered` and retryDeadEvent correctly
286
+ // refuses it, so nothing is lost — but an operator with an alarm on that line is paged for
287
+ // a non-event, and an alarm that cries wolf is worse than no alarm.
288
+ if (n >= maxAttempts && (await work.get(ack)) !== undefined) {
289
+ console.warn(`@gnldev/events: handler errored (topic=${topic}, consumer=${opts.name}, event=${e.id}) but the event was already DELIVERED by another process under the same consumer name — this attempt is discarded, nothing is quarantined:`, err);
290
+ continue; // NOT frozen: it is acked, so the cursor may pass it
291
+ }
292
+ const delay = delayFor(n, e.id); // called ONCE per failure; caller code, may throw (see delayFor)
293
+ // The attempt counter USED TO BE a get→put pair with the comment "quarantine happens a poll
294
+ // or two late, NEVER EARLY, and never turns into loss". The middle claim was false, and
295
+ // measured: `attRaw` is read BEFORE the handler runs, so an operator's retryDeadEvent
296
+ // landing anywhere in the handler's lifetime — a window the stamp check above cannot see,
297
+ // because it compares the dead record read before the release, not this key — was
298
+ // OVERWRITTEN by this put. The release had already returned `true`. Its effects both died:
299
+ // the fresh budget (measured: an 8-attempt release turned into 1 attempt, so the very next
300
+ // failure quarantined — early, not late) and the cleared backoff (measured: `nextAt` put an
301
+ // hour into the future on an event the operator had just said "now" about).
302
+ // So this is a CAS on its own key, exactly as the dead record's write below is — same
303
+ // window, same technique, same standard. Absent key → nothing to race with (a release
304
+ // implies a dead record implies a previous failure implies this key exists) and putIfMatch
305
+ // is false-on-absent, so that case is a plain put. Losing the CAS means someone else moved
306
+ // this event on: their record stands, this attempt is discarded, and nothing is quarantined
307
+ // off a counter we no longer own. What remains from the old comment is the benign half —
308
+ // two concurrent polls can still make an attempt go uncounted, which only ever DELAYS
309
+ // quarantine.
310
+ const ak = attKey(topic, opts.name, e.id);
311
+ const next = {
312
+ n,
313
+ firstAt: att?.firstAt || now,
314
+ nextAt: now + delay,
315
+ ...(att?.gen !== undefined ? { gen: att.gen } : {}), // the streak keeps its generation
316
+ };
317
+ const wroteAtt = attRaw === undefined || !work.putIfMatch
318
+ ? (await work.put(ak, next), true)
319
+ : await work.putIfMatch(ak, attRaw, next);
320
+ if (!wroteAtt) {
321
+ frozen = true; // someone else changed the counter under us → theirs stands, not ours
322
+ console.warn(`@gnldev/events: attempt write skipped — the retry record changed concurrently (topic=${topic}, consumer=${opts.name}, event=${e.id}); this attempt is discarded and the event stays live:`, err);
323
+ continue;
324
+ }
325
+ if (n >= maxAttempts) {
326
+ const rec = {
327
+ error: String(err?.message ?? err),
328
+ attempts: n,
329
+ at: now,
330
+ ...(fresh?.releases != null ? { releases: fresh.releases } : {}),
331
+ };
332
+ // Conditional write where the store supports it (8.2 WorkStore.putIfMatch): the fresh
333
+ // read above closes the handler-long window, this closes the sliver after it. If the key
334
+ // doesn't exist yet there is nothing to race with — a release can only exist once a dead
335
+ // record does — and putIfMatch is false-on-absent, so that case is a plain put.
336
+ const wrote = fresh === undefined || !work.putIfMatch
337
+ ? (await work.put(dk, rec), true)
338
+ : await work.putIfMatch(dk, fresh, rec);
339
+ if (!wrote) {
340
+ frozen = true; // someone else changed the record under us → theirs stands, not ours
341
+ console.warn(`@gnldev/events: quarantine write skipped — the dead-letter record changed concurrently (topic=${topic}, consumer=${opts.name}, event=${e.id}); the event stays live.`);
342
+ continue;
343
+ }
344
+ console.error(`@gnldev/events: event QUARANTINED after ${n} failed attempts (topic=${topic}, consumer=${opts.name}, event=${e.id}) — it will NOT be redelivered until retryDeadEvent(); inspect with listDeadEvents():`, err);
345
+ }
346
+ else {
347
+ frozen = true; // still retryable → the cursor must not move past it
348
+ console.warn(`@gnldev/events: handler errored (topic=${topic}, consumer=${opts.name}, event=${e.id}, attempt=${n}/${maxAttempts}) — retrying in ${delay}ms:`, err);
349
+ }
350
+ continue;
351
+ }
352
+ // Handler SUCCEEDED → mark it now. false = another poll/instance finished the race first
353
+ // (the handler may have run twice — idempotency is the handler's job); don't double-COUNT.
354
+ if (await work.ackOnce(ack))
355
+ delivered++;
356
+ }
357
+ if (!page.nextCursor) {
358
+ // End of the log. The last (partial) page's cursor is never persisted — it must be
359
+ // rescanned for new events.
360
+ if (rescan)
361
+ await work.put(rescanKey, false); // the released event has been back in view for a full pass
362
+ return delivered;
363
+ }
364
+ cursor = page.nextCursor;
365
+ // A retryable (unmarked, un-quarantined) event is still behind us → the PERSISTED cursor
366
+ // cannot advance past it: if it did, that event would never be scanned again = silent loss.
367
+ // But the SCAN continues — freezing the bookmark is not a reason to stop delivering. That
368
+ // conflation was the bug: `if (pageHasFailure) return delivered` stopped the whole pass, so
369
+ // one poison event on page 1 held back every event behind it forever. Measured (real SQLite,
370
+ // 120 events, the 4th always throwing): poll1=49, poll2=0, poll3=0 — events 50..119 were
371
+ // never delivered at all. With the split: poll1=119, and after maxAttempts the poison event
372
+ // is quarantined, which unfreezes the bookmark too.
373
+ // Rejected: advancing the cursor past the failure anyway (a one-line fix) — that is exactly
374
+ // the silent data loss this package refuses; the consumer never saw the event and nothing
375
+ // would record that it had been skipped.
376
+ if (!frozen)
377
+ await work.put(cursorKey, cursor);
378
+ }
379
+ }
380
+ // Phase 8.1: the tick/backoff/"polling" flag loop now lives in @gnldev/durable's shared
381
+ // createPollLoop (was a triplicate copy across queue/events/scheduler) — behavior is identical:
382
+ // if poll() delivers 0 events, the interval grows ×2 while backoffOn (ceiling maxPollMs); it
383
+ // resets to pollMs once something is delivered. poll() catches handler errors internally (above);
384
+ // remaining errors (store I/O etc.) are logged and swallowed by createPollLoop — the chain doesn't
385
+ // die (an unhandled rejection doesn't crash the process).
386
+ const loop = createPollLoop(async () => (await poll()) > 0, { pollMs, backoff: backoffOn, maxPollMs });
387
+ return {
388
+ poll,
389
+ start: loop.start,
390
+ stop: loop.stop,
391
+ };
392
+ }
393
+ // ── Dead-letter (quarantine) inspection + release ─────────────────────────────
394
+ // The operator-facing half of the fix. A poison event no longer blocks the topic, but "doesn't
395
+ // block" is only acceptable if "what happened to it" is answerable. These two functions are that
396
+ // answer, and they are deliberately the same pair @gnldev/queue exposes for jobs (listJobs /
397
+ // retryJob) — one dead-letter vocabulary across the two packages, not two.
398
+ /**
399
+ * Every event that has been quarantined for `consumer` on `topic` — including ones later released
400
+ * and delivered (`status`), because dead-letter history is permanent for audit (same choice as
401
+ * queue's qfail: nothing is deleted from an append-only log).
402
+ *
403
+ * MANAGEMENT function: it reads the WHOLE topic log and does one `get` per event. Do NOT call it
404
+ * from a poll loop — the loop reads only the pages it needs, via cursor (5.1).
405
+ */
406
+ export async function listDeadEvents(work, topic, consumer) {
407
+ const out = [];
408
+ let cursor;
409
+ for (;;) {
410
+ const page = await work.list(logNsOf(topic), { cursor });
411
+ for (const e of page.items) {
412
+ const rec = await work.get(deadKey(topic, consumer, e.id));
413
+ if (!rec)
414
+ continue;
415
+ const acked = (await work.get(ackKey(topic, consumer, e.id))) !== undefined;
416
+ out.push({
417
+ ...rec,
418
+ id: e.id,
419
+ topic,
420
+ consumer,
421
+ payload: e.payload,
422
+ status: acked ? 'delivered' : rec.releasedAt ? 'released' : 'quarantined',
423
+ });
424
+ }
425
+ if (!page.nextCursor)
426
+ return out;
427
+ cursor = page.nextCursor;
428
+ }
429
+ }
430
+ /**
431
+ * Releases a quarantined event back for delivery to `consumer` (attempt counter reset to 0, backoff
432
+ * cleared → due immediately). The next `poll()` re-scans the log from the start once — the consumer's
433
+ * cursor had already moved past the event, which is why a flag rather than the cursor is what has to
434
+ * change — and hands it to the handler again. Returns `false` (no-op) if the event was never
435
+ * quarantined or has since been DELIVERED, mirroring `retryJob`, which only acts on jobs that
436
+ * actually reached the dead-letter state — plus the contended case at the bottom of this note.
437
+ *
438
+ * IDEMPOTENT while `released`: calling it again on an already-released, not-yet-delivered event
439
+ * re-asserts the release (re-arms the rescan flag, hands back a fresh attempt budget) and returns
440
+ * `true`. That is not cosmetic — it is the ONLY way out of the race documented below. This function
441
+ * used to return `false` there, which meant a release whose flag was swallowed by an in-flight poll
442
+ * left the event `released` and unscanned forever: never delivered, never failing, never
443
+ * re-quarantined, and refused by the very call its own docstring prescribed as the remedy. A
444
+ * dead-letter that the package's API cannot get an event out of is not a dead-letter, it is a leak.
445
+ * (`releases` counts every release that TOOK EFFECT, so a re-assertion increments it too — the
446
+ * counter answers "how many times was this handed back", not "how many quarantine cycles".)
447
+ *
448
+ * Rejected alternative: re-EMIT the event under a new id (which is literally what queue's retryJob
449
+ * does). It is simpler and needs no rescan flag, but a topic is FAN-OUT: a re-emitted copy is
450
+ * delivered to EVERY consumer, so healing consumer A would force redelivery on healthy consumer B.
451
+ * B's handler is contractually idempotent, so it would not corrupt anything — but manufacturing
452
+ * redelivery for a consumer that never failed is not a repair, it's collateral. Release is
453
+ * per-consumer for the same reason ack markers and cursors are per-consumer.
454
+ *
455
+ * Rejected alternative: have this function rewind the persisted cursor itself, to the released
456
+ * event's page (storing that page cursor in the dead record). Cheaper than a full pass, but it adds
457
+ * another writer to the cursor — one that writes it BACKWARDS, which is the direction that can be
458
+ * lost: a poll already in flight (this consumer name may be run by several processes, see poll())
459
+ * writes it forwards afterwards and the rewind is gone, with the failure mode "the release silently
460
+ * didn't take". A flag is re-assertable and a cursor position is not, which is the whole difference:
461
+ * The flag is not atomic against an in-flight poll either (one that started before the release can
462
+ * clear it at the end of its own pass), but the cost is only that the release needs calling
463
+ * again — which the idempotent re-release above makes possible — and it is observable rather than
464
+ * silent: `listDeadEvents` keeps showing the event as `released` and undelivered.
465
+ *
466
+ * Returns `false` WITHOUT releasing in one further case: the dead record is being rewritten
467
+ * concurrently faster than this function can read-modify-write it (`RELEASE_CAS_TRIES` lost
468
+ * compare-and-swaps in a row). It is logged, and the remedy is to call again — the same remedy as
469
+ * the lost-flag race. A silent overwrite would be the alternative, and that is what the CAS is here
470
+ * to stop.
471
+ */
472
+ const RELEASE_CAS_TRIES = 5;
473
+ export async function retryDeadEvent(work, topic, consumer, eventId) {
474
+ const dk = deadKey(topic, consumer, eventId);
475
+ const ak = attKey(topic, consumer, eventId);
476
+ // Read → modify → write on a record other writers touch, so it is a compare-and-swap for the same
477
+ // reason poll()'s two writes are: measured without it, two releases landing together (an operator
478
+ // double-click, two panels, a retry script racing a human) both read the same record and both
479
+ // wrote `releases: n + 1` from it — `both returned true? true true`, and the record showed
480
+ // `releases: 1`. `releases` is an operator-facing number that answers "how many times was this
481
+ // handed back"; answering it wrongly is worse than not answering. It is also the ABA input `gen`
482
+ // exists to separate: equal `releases` plus (in the same millisecond) equal `releasedAt` is an
483
+ // equal `releaseStamp`, which is precisely what a value CAS cannot see through. Losing the CAS
484
+ // means someone else moved the record on, so this pass re-reads and re-applies on top of theirs
485
+ // rather than clobbering it — a lost release must become a LATER release, not a vanished one.
486
+ for (let tries = 0; tries < RELEASE_CAS_TRIES; tries++) {
487
+ const rec = await work.get(dk);
488
+ if (!rec)
489
+ return false; // never quarantined — a still-retrying or never-failed event needs no release
490
+ // DELIVERED is the only terminal state: the ack marker is the single source of truth for "the
491
+ // consumer saw it" (the same marker listDeadEvents reports as `status: 'delivered'`). `releasedAt`
492
+ // being set is NOT terminal — see the idempotency note above. Re-read each pass: a concurrent
493
+ // delivery is exactly the kind of thing that can land while a CAS is being retried.
494
+ if ((await work.get(ackKey(topic, consumer, eventId))) !== undefined)
495
+ return false;
496
+ const released = { ...rec, releasedAt: Date.now(), releases: (rec.releases ?? 0) + 1 };
497
+ // Order matters: the attempt reset and the dead record are written BEFORE the rescan flag, so a
498
+ // poll that reacts to the flag can never see a half-applied release.
499
+ //
500
+ // The reset carries this release's STAMP. Without it the reset is `{n:0, firstAt:0, nextAt:0}` for
501
+ // every release, byte-identical each time, and the poll's compare-and-swap on this key cannot tell
502
+ // "nobody touched it" from "a second release put it back to the same value" — so a poll holding the
503
+ // previous release's record would overwrite this one and its CAS would report success. See StoredAttempts.gen.
504
+ await work.put(ak, { n: 0, firstAt: 0, nextAt: 0, gen: releaseStamp(released) });
505
+ // The record always exists here (it was just read), so unlike poll()'s writes there is no
506
+ // absent-key branch — only the documented drop for a store predating WorkStore.putIfMatch,
507
+ // which keeps the old unconditional put and therefore the old race.
508
+ const wrote = !work.putIfMatch ? (await work.put(dk, released), true) : await work.putIfMatch(dk, rec, released);
509
+ if (!wrote)
510
+ continue; // someone else wrote it between the read and here → re-read and re-apply
511
+ await work.put(rescanKeyOf(topic, consumer), true);
512
+ return true;
513
+ }
514
+ console.warn(`@gnldev/events: release abandoned — the dead-letter record kept changing under it (topic=${topic}, consumer=${consumer}, event=${eventId}, tries=${RELEASE_CAS_TRIES}); nothing was overwritten, call retryDeadEvent() again.`);
515
+ return false;
516
+ }
517
+ //# sourceMappingURL=index.js.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,0GAA0G;AAC1G,iGAAiG;AACjG,uGAAuG;AACvG,sGAAsG;AACtG,oGAAoG;AACpG,wCAAwC;AACxC,uGAAuG;AACvG,iHAAiH;AACjH,4FAA4F;AAC5F,mGAAmG;AACnG,+FAA+F;AAC/F,oGAAoG;AACpG,6FAA6F;AAC7F,OAAO,EAAE,cAAc,EAAE,MAAM,iBAAiB,CAAC;AAiIjD;;;;GAIG;AACH,MAAM,OAAO,uBAAwB,SAAQ,KAAK;IAG9B;IAFlB,YACE,OAAe,EACC,MAA0D;QAE1E,KAAK,CAAC,OAAO,CAAC,CAAC;QAFC,WAAM,GAAN,MAAM,CAAoD;QAG1E,IAAI,CAAC,IAAI,GAAG,yBAAyB,CAAC;IACxC,CAAC;CACF;AAED;;;;;;GAMG;AACH,KAAK,UAAU,SAAS,CAAC,IAAe,EAAE,EAAU,EAAE,KAAa;IACjE,IAAI,KAAK,GAAG,CAAC,CAAC;IACd,IAAI,MAA0B,CAAC;IAC/B,SAAS,CAAC;QACR,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,IAAI,CAAC,EAAE,EAAE,EAAE,MAAM,EAAE,CAAC,CAAC;QAC7C,KAAK,IAAI,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC;QAC3B,IAAI,KAAK,IAAI,KAAK,IAAI,CAAC,IAAI,CAAC,UAAU;YAAE,OAAO,KAAK,CAAC;QACrD,MAAM,GAAG,IAAI,CAAC,UAAU,CAAC;IAC3B,CAAC;AACH,CAAC;AAED,iGAAiG;AACjG,MAAM,CAAC,KAAK,UAAU,IAAI,CACxB,IAAe,EACf,KAAa,EACb,OAAgB,EAChB,OAA2C,EAAE;IAE7C,IAAI,IAAI,CAAC,QAAQ,IAAI,IAAI,EAAE,CAAC;QAC1B,MAAM,KAAK,GAAG,MAAM,SAAS,CAAC,IAAI,EAAE,OAAO,CAAC,KAAK,CAAC,EAAE,IAAI,CAAC,QAAQ,CAAC,CAAC;QACnE,IAAI,KAAK,IAAI,IAAI,CAAC,QAAQ,EAAE,CAAC;YAC3B,MAAM,IAAI,uBAAuB,CAC/B,+CAA+C,KAAK,OAAO,IAAI,CAAC,QAAQ,8BAA8B,KAAK,KAAK,EAChH,EAAE,KAAK,EAAE,KAAK,EAAE,QAAQ,EAAE,IAAI,CAAC,QAAQ,EAAE,CAC1C,CAAC;QACJ,CAAC;IACH,CAAC;IACD,OAAO,IAAI,CAAC,MAAM,CAAC,OAAO,CAAC,KAAK,CAAC,EAAE,OAAO,EAAE,IAAI,CAAC,EAAE,CAAC,CAAC;AACvD,CAAC;AAED;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAwDG;AACH,MAAM,GAAG,GAAG,CAAC,IAAY,EAAE,EAAE,CAAC,IAAI,CAAC,OAAO,CAAC,IAAI,EAAE,KAAK,CAAC,CAAC,OAAO,CAAC,IAAI,EAAE,KAAK,CAAC,CAAC;AAE7E,gGAAgG;AAChG,kGAAkG;AAClG,+EAA+E;AAC/E,oFAAoF;AACpF,MAAM,OAAO,GAAG,CAAC,KAAa,EAAE,EAAE,CAAC,OAAO,GAAG,CAAC,KAAK,CAAC,EAAE,CAAC;AACvD,MAAM,MAAM,GAAG,CAAC,KAAa,EAAE,QAAgB,EAAE,EAAU,EAAE,EAAE,CAAC,UAAU,GAAG,CAAC,KAAK,CAAC,IAAI,GAAG,CAAC,QAAQ,CAAC,IAAI,GAAG,CAAC,EAAE,CAAC,EAAE,CAAC;AACnH,MAAM,MAAM,GAAG,CAAC,KAAa,EAAE,QAAgB,EAAE,EAAU,EAAE,EAAE,CAAC,UAAU,GAAG,CAAC,KAAK,CAAC,IAAI,GAAG,CAAC,QAAQ,CAAC,IAAI,GAAG,CAAC,EAAE,CAAC,EAAE,CAAC;AACnH,MAAM,OAAO,GAAG,CAAC,KAAa,EAAE,QAAgB,EAAE,EAAU,EAAE,EAAE,CAAC,WAAW,GAAG,CAAC,KAAK,CAAC,IAAI,GAAG,CAAC,QAAQ,CAAC,IAAI,GAAG,CAAC,EAAE,CAAC,EAAE,CAAC;AACrH,MAAM,WAAW,GAAG,CAAC,KAAa,EAAE,QAAgB,EAAE,EAAE,CAAC,aAAa,GAAG,CAAC,KAAK,CAAC,IAAI,GAAG,CAAC,QAAQ,CAAC,EAAE,CAAC;AACpG,MAAM,WAAW,GAAG,CAAC,KAAa,EAAE,QAAgB,EAAE,EAAE,CAAC,aAAa,GAAG,CAAC,KAAK,CAAC,IAAI,GAAG,CAAC,QAAQ,CAAC,EAAE,CAAC;AAEpG,kGAAkG;AAClG,SAAS,YAAY,CAAC,GAAY;IAChC,IAAI,GAAG,IAAI,IAAI;QAAE,OAAO,SAAS,CAAC;IAClC,IAAI,OAAO,GAAG,KAAK,QAAQ;QAAE,OAAO,EAAE,CAAC,EAAE,GAAG,EAAE,OAAO,EAAE,CAAC,EAAE,MAAM,EAAE,CAAC,EAAE,CAAC,CAAC,mCAAmC;IAC1G,MAAM,CAAC,GAAG,GAA8B,CAAC;IACzC,OAAO;QACL,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,EAAE,OAAO,EAAE,CAAC,CAAC,OAAO,IAAI,CAAC,EAAE,MAAM,EAAE,CAAC,CAAC,MAAM,IAAI,CAAC;QAC3D,GAAG,CAAC,CAAC,CAAC,GAAG,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,GAAG,EAAE,CAAC,CAAC,GAAG,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;KAC/C,CAAC;AACJ,CAAC;AAED;;;;;;GAMG;AACH,MAAM,YAAY,GAAG,CAAC,GAA2B,EAAE,EAAE,CAAC,GAAG,GAAG,EAAE,QAAQ,IAAI,CAAC,IAAI,GAAG,EAAE,UAAU,IAAI,CAAC,EAAE,CAAC;AAEtG,sGAAsG;AACtG,MAAM,iBAAiB,GAAG,CAAC,OAAe,EAAE,EAAE,CAAC,IAAI,CAAC,GAAG,CAAC,MAAM,GAAG,CAAC,IAAI,CAAC,OAAO,GAAG,CAAC,CAAC,EAAE,SAAS,CAAC,CAAC;AAEhG,MAAM,UAAU,cAAc,CAC5B,IAAe,EACf,KAAa,EACb,OAAqB,EACrB,IAAqB;IAErB,MAAM,EAAE,GAAG,OAAO,CAAC,KAAK,CAAC,CAAC;IAC1B,MAAM,MAAM,GAAG,IAAI,CAAC,MAAM,IAAI,GAAG,CAAC;IAClC,MAAM,SAAS,GAAG,IAAI,CAAC,OAAO,IAAI,IAAI,CAAC;IACvC,MAAM,SAAS,GAAG,IAAI,CAAC,SAAS,IAAI,MAAM,GAAG,EAAE,CAAC;IAChD,MAAM,WAAW,GAAG,IAAI,CAAC,WAAW,IAAI,CAAC,CAAC;IAC1C,MAAM,UAAU,GACd,OAAO,IAAI,CAAC,YAAY,KAAK,UAAU,CAAC,CAAC,CAAC,IAAI,CAAC,YAAY;QACzD,CAAC,CAAC,IAAI,CAAC,YAAY,IAAI,IAAI,CAAC,CAAC,CAAC,GAAG,EAAE,CAAC,IAAI,CAAC,YAAsB;YAC7D,CAAC,CAAC,iBAAiB,CAAC;IAC1B,6FAA6F;IAC7F,+FAA+F;IAC/F,kGAAkG;IAClG,kGAAkG;IAClG,2FAA2F;IAC3F,MAAM,SAAS,GAAG,WAAW,CAAC,KAAK,EAAE,IAAI,CAAC,IAAI,CAAC,CAAC;IAChD,MAAM,SAAS,GAAG,WAAW,CAAC,KAAK,EAAE,IAAI,CAAC,IAAI,CAAC,CAAC;IAEhD;;;;;;;;;;;;;;;OAeG;IACH,SAAS,QAAQ,CAAC,OAAe,EAAE,OAAe;QAChD,IAAI,GAAY,CAAC;QACjB,IAAI,CAAC;YACH,GAAG,GAAG,UAAU,CAAC,OAAO,CAAC,CAAC;QAC5B,CAAC;QAAC,OAAO,CAAC,EAAE,CAAC;YACX,OAAO,CAAC,IAAI,CAAC,6CAA6C,KAAK,cAAc,IAAI,CAAC,IAAI,WAAW,OAAO,aAAa,OAAO,kDAAkD,EAAE,CAAC,CAAC,CAAC;YACnL,OAAO,iBAAiB,CAAC,OAAO,CAAC,CAAC;QACpC,CAAC;QACD,IAAI,OAAO,GAAG,KAAK,QAAQ,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,GAAG,CAAC,EAAE,CAAC;YACrD,OAAO,CAAC,IAAI,CAAC,yCAAyC,MAAM,CAAC,GAAG,CAAC,WAAW,KAAK,cAAc,IAAI,CAAC,IAAI,WAAW,OAAO,aAAa,OAAO,mJAAmJ,CAAC,CAAC;YACnS,OAAO,iBAAiB,CAAC,OAAO,CAAC,CAAC;QACpC,CAAC;QACD,OAAO,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC;IAC1B,CAAC;IAED,KAAK,UAAU,IAAI;QACjB,6FAA6F;QAC7F,+FAA+F;QAC/F,4FAA4F;QAC5F,gGAAgG;QAChG,uEAAuE;QACvE,MAAM,MAAM,GAAG,CAAC,MAAM,IAAI,CAAC,GAAG,CAAU,SAAS,CAAC,CAAC,KAAK,IAAI,CAAC;QAC7D,8FAA8F;QAC9F,gGAAgG;QAChG,iGAAiG;QACjG,wFAAwF;QACxF,gGAAgG;QAChG,gGAAgG;QAChG,gFAAgF;QAChF,EAAE;QACF,8FAA8F;QAC9F,2FAA2F;QAC3F,yFAAyF;QACzF,+FAA+F;QAC/F,0FAA0F;QAC1F,mFAAmF;QACnF,0FAA0F;QAC1F,+FAA+F;QAC/F,8FAA8F;QAC9F,gEAAgE;QAChE,IAAI,MAAM;YAAE,MAAM,IAAI,CAAC,GAAG,CAAC,SAAS,EAAE,EAAE,CAAC,CAAC;QAC1C,IAAI,MAAM,GAAG,MAAM,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC,MAAM,IAAI,CAAC,GAAG,CAAS,SAAS,CAAC,CAAC,IAAI,SAAS,CAAC;QACnF,IAAI,SAAS,GAAG,CAAC,CAAC;QAClB,4FAA4F;QAC5F,wFAAwF;QACxF,IAAI,MAAM,GAAG,KAAK,CAAC;QACnB,SAAS,CAAC;YACR,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,IAAI,CAAC,EAAE,EAAE,EAAE,MAAM,EAAE,CAAC,CAAC;YAC7C,KAAK,MAAM,CAAC,IAAI,IAAI,CAAC,KAAK,EAAE,CAAC;gBAC3B,MAAM,GAAG,GAAG,MAAM,CAAC,KAAK,EAAE,IAAI,CAAC,IAAI,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC;gBAC3C,mFAAmF;gBACnF,oEAAoE;gBACpE,IAAI,CAAC,MAAM,IAAI,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,KAAK,SAAS;oBAAE,SAAS;gBAClD,kFAAkF;gBAClF,wFAAwF;gBACxF,0FAA0F;gBAC1F,8FAA8F;gBAC9F,MAAM,EAAE,GAAG,OAAO,CAAC,KAAK,EAAE,IAAI,CAAC,IAAI,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC;gBAC3C,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,GAAG,CAAa,EAAE,CAAC,CAAC;gBAC5C,IAAI,IAAI,IAAI,CAAC,IAAI,CAAC,UAAU;oBAAE,SAAS;gBACvC,4FAA4F;gBAC5F,6FAA6F;gBAC7F,yFAAyF;gBACzF,4FAA4F;gBAC5F,wFAAwF;gBACxF,6FAA6F;gBAC7F,4FAA4F;gBAC5F,gGAAgG;gBAChG,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,GAAG,CAAC,MAAM,CAAC,KAAK,EAAE,IAAI,CAAC,IAAI,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;gBAC9D,MAAM,GAAG,GAAG,YAAY,CAAC,MAAM,CAAC,CAAC;gBACjC,IAAI,GAAG,IAAI,GAAG,CAAC,MAAM,GAAG,IAAI,CAAC,GAAG,EAAE,EAAE,CAAC;oBAAC,MAAM,GAAG,IAAI,CAAC;oBAAC,SAAS;gBAAC,CAAC;gBAChE,uFAAuF;gBACvF,yFAAyF;gBACzF,uGAAuG;gBACvG,IAAI,CAAC;oBACH,MAAM,OAAO,CAAC,CAAC,CAAC,OAAO,EAAE,EAAE,EAAE,EAAE,CAAC,CAAC,EAAE,EAAE,KAAK,EAAE,CAAC,CAAC;gBAChD,CAAC;gBAAC,OAAO,GAAG,EAAE,CAAC;oBACb,wFAAwF;oBACxF,qEAAqE;oBACrE,EAAE;oBACF,yFAAyF;oBACzF,0FAA0F;oBAC1F,uFAAuF;oBACvF,uFAAuF;oBACvF,2FAA2F;oBAC3F,+EAA+E;oBAC/E,MAAM,KAAK,GAAG,MAAM,IAAI,CAAC,GAAG,CAAa,EAAE,CAAC,CAAC;oBAC7C,IAAI,YAAY,CAAC,KAAK,CAAC,KAAK,YAAY,CAAC,IAAI,CAAC,EAAE,CAAC;wBAC/C,MAAM,GAAG,IAAI,CAAC,CAAC,2EAA2E;wBAC1F,OAAO,CAAC,IAAI,CAAC,0CAA0C,KAAK,cAAc,IAAI,CAAC,IAAI,WAAW,CAAC,CAAC,EAAE,4FAA4F,EAAE,GAAG,CAAC,CAAC;wBACrM,SAAS;oBACX,CAAC;oBACD,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;oBACvB,MAAM,CAAC,GAAG,CAAC,GAAG,EAAE,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;oBAC5B,yFAAyF;oBACzF,qFAAqF;oBACrF,uFAAuF;oBACvF,2FAA2F;oBAC3F,wFAAwF;oBACxF,wFAAwF;oBACxF,4FAA4F;oBAC5F,2FAA2F;oBAC3F,oEAAoE;oBACpE,IAAI,CAAC,IAAI,WAAW,IAAI,CAAC,MAAM,IAAI,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,KAAK,SAAS,EAAE,CAAC;wBAC5D,OAAO,CAAC,IAAI,CAAC,0CAA0C,KAAK,cAAc,IAAI,CAAC,IAAI,WAAW,CAAC,CAAC,EAAE,4IAA4I,EAAE,GAAG,CAAC,CAAC;wBACrP,SAAS,CAAC,qDAAqD;oBACjE,CAAC;oBACD,MAAM,KAAK,GAAG,QAAQ,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,iEAAiE;oBAClG,4FAA4F;oBAC5F,wFAAwF;oBACxF,sFAAsF;oBACtF,0FAA0F;oBAC1F,kFAAkF;oBAClF,2FAA2F;oBAC3F,2FAA2F;oBAC3F,4FAA4F;oBAC5F,4EAA4E;oBAC5E,sFAAsF;oBACtF,sFAAsF;oBACtF,2FAA2F;oBAC3F,2FAA2F;oBAC3F,4FAA4F;oBAC5F,yFAAyF;oBACzF,sFAAsF;oBACtF,cAAc;oBACd,MAAM,EAAE,GAAG,MAAM,CAAC,KAAK,EAAE,IAAI,CAAC,IAAI,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC;oBAC1C,MAAM,IAAI,GAAmB;wBAC3B,CAAC;wBACD,OAAO,EAAE,GAAG,EAAE,OAAO,IAAI,GAAG;wBAC5B,MAAM,EAAE,GAAG,GAAG,KAAK;wBACnB,GAAG,CAAC,GAAG,EAAE,GAAG,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,GAAG,EAAE,GAAG,CAAC,GAAG,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,kCAAkC;qBACxF,CAAC;oBACF,MAAM,QAAQ,GAAG,MAAM,KAAK,SAAS,IAAI,CAAC,IAAI,CAAC,UAAU;wBACvD,CAAC,CAAC,CAAC,MAAM,IAAI,CAAC,GAAG,CAAC,EAAE,EAAE,IAAI,CAAC,EAAE,IAAI,CAAC;wBAClC,CAAC,CAAC,MAAM,IAAI,CAAC,UAAU,CAAC,EAAE,EAAE,MAAM,EAAE,IAAI,CAAC,CAAC;oBAC5C,IAAI,CAAC,QAAQ,EAAE,CAAC;wBACd,MAAM,GAAG,IAAI,CAAC,CAAC,sEAAsE;wBACrF,OAAO,CAAC,IAAI,CAAC,wFAAwF,KAAK,cAAc,IAAI,CAAC,IAAI,WAAW,CAAC,CAAC,EAAE,wDAAwD,EAAE,GAAG,CAAC,CAAC;wBAC/M,SAAS;oBACX,CAAC;oBACD,IAAI,CAAC,IAAI,WAAW,EAAE,CAAC;wBACrB,MAAM,GAAG,GAAe;4BACtB,KAAK,EAAE,MAAM,CAAE,GAAW,EAAE,OAAO,IAAI,GAAG,CAAC;4BAC3C,QAAQ,EAAE,CAAC;4BACX,EAAE,EAAE,GAAG;4BACP,GAAG,CAAC,KAAK,EAAE,QAAQ,IAAI,IAAI,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,KAAK,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;yBACjE,CAAC;wBACF,sFAAsF;wBACtF,yFAAyF;wBACzF,yFAAyF;wBACzF,gFAAgF;wBAChF,MAAM,KAAK,GAAG,KAAK,KAAK,SAAS,IAAI,CAAC,IAAI,CAAC,UAAU;4BACnD,CAAC,CAAC,CAAC,MAAM,IAAI,CAAC,GAAG,CAAC,EAAE,EAAE,GAAG,CAAC,EAAE,IAAI,CAAC;4BACjC,CAAC,CAAC,MAAM,IAAI,CAAC,UAAU,CAAC,EAAE,EAAE,KAAK,EAAE,GAAG,CAAC,CAAC;wBAC1C,IAAI,CAAC,KAAK,EAAE,CAAC;4BACX,MAAM,GAAG,IAAI,CAAC,CAAC,qEAAqE;4BACpF,OAAO,CAAC,IAAI,CAAC,iGAAiG,KAAK,cAAc,IAAI,CAAC,IAAI,WAAW,CAAC,CAAC,EAAE,0BAA0B,CAAC,CAAC;4BACrL,SAAS;wBACX,CAAC;wBACD,OAAO,CAAC,KAAK,CAAC,2CAA2C,CAAC,2BAA2B,KAAK,cAAc,IAAI,CAAC,IAAI,WAAW,CAAC,CAAC,EAAE,uFAAuF,EAAE,GAAG,CAAC,CAAC;oBAChO,CAAC;yBAAM,CAAC;wBACN,MAAM,GAAG,IAAI,CAAC,CAAC,qDAAqD;wBACpE,OAAO,CAAC,IAAI,CAAC,0CAA0C,KAAK,cAAc,IAAI,CAAC,IAAI,WAAW,CAAC,CAAC,EAAE,aAAa,CAAC,IAAI,WAAW,mBAAmB,KAAK,KAAK,EAAE,GAAG,CAAC,CAAC;oBACrK,CAAC;oBACD,SAAS;gBACX,CAAC;gBACD,yFAAyF;gBACzF,2FAA2F;gBAC3F,IAAI,MAAM,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC;oBAAE,SAAS,EAAE,CAAC;YAC3C,CAAC;YACD,IAAI,CAAC,IAAI,CAAC,UAAU,EAAE,CAAC;gBACrB,mFAAmF;gBACnF,4BAA4B;gBAC5B,IAAI,MAAM;oBAAE,MAAM,IAAI,CAAC,GAAG,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC,CAAC,2DAA2D;gBACzG,OAAO,SAAS,CAAC;YACnB,CAAC;YACD,MAAM,GAAG,IAAI,CAAC,UAAU,CAAC;YACzB,yFAAyF;YACzF,4FAA4F;YAC5F,0FAA0F;YAC1F,4FAA4F;YAC5F,6FAA6F;YAC7F,yFAAyF;YACzF,4FAA4F;YAC5F,oDAAoD;YACpD,4FAA4F;YAC5F,0FAA0F;YAC1F,yCAAyC;YACzC,IAAI,CAAC,MAAM;gBAAE,MAAM,IAAI,CAAC,GAAG,CAAC,SAAS,EAAE,MAAM,CAAC,CAAC;QACjD,CAAC;IACH,CAAC;IAED,wFAAwF;IACxF,gGAAgG;IAChG,6FAA6F;IAC7F,kGAAkG;IAClG,mGAAmG;IACnG,0DAA0D;IAC1D,MAAM,IAAI,GAAG,cAAc,CAAC,KAAK,IAAI,EAAE,CAAC,CAAC,MAAM,IAAI,EAAE,CAAC,GAAG,CAAC,EAAE,EAAE,MAAM,EAAE,OAAO,EAAE,SAAS,EAAE,SAAS,EAAE,CAAC,CAAC;IAEvG,OAAO;QACL,IAAI;QACJ,KAAK,EAAE,IAAI,CAAC,KAAK;QACjB,IAAI,EAAE,IAAI,CAAC,IAAI;KAChB,CAAC;AACJ,CAAC;AAED,iFAAiF;AACjF,+FAA+F;AAC/F,iGAAiG;AACjG,6FAA6F;AAC7F,2EAA2E;AAE3E;;;;;;;GAOG;AACH,MAAM,CAAC,KAAK,UAAU,cAAc,CAAC,IAAe,EAAE,KAAa,EAAE,QAAgB;IACnF,MAAM,GAAG,GAAgB,EAAE,CAAC;IAC5B,IAAI,MAA0B,CAAC;IAC/B,SAAS,CAAC;QACR,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,KAAK,CAAC,EAAE,EAAE,MAAM,EAAE,CAAC,CAAC;QACzD,KAAK,MAAM,CAAC,IAAI,IAAI,CAAC,KAAK,EAAE,CAAC;YAC3B,MAAM,GAAG,GAAG,MAAM,IAAI,CAAC,GAAG,CAAa,OAAO,CAAC,KAAK,EAAE,QAAQ,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;YACvE,IAAI,CAAC,GAAG;gBAAE,SAAS;YACnB,MAAM,KAAK,GAAG,CAAC,MAAM,IAAI,CAAC,GAAG,CAAC,MAAM,CAAC,KAAK,EAAE,QAAQ,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,KAAK,SAAS,CAAC;YAC5E,GAAG,CAAC,IAAI,CAAC;gBACP,GAAG,GAAG;gBACN,EAAE,EAAE,CAAC,CAAC,EAAE;gBACR,KAAK;gBACL,QAAQ;gBACR,OAAO,EAAE,CAAC,CAAC,OAAO;gBAClB,MAAM,EAAE,KAAK,CAAC,CAAC,CAAC,WAAW,CAAC,CAAC,CAAC,GAAG,CAAC,UAAU,CAAC,CAAC,CAAC,UAAU,CAAC,CAAC,CAAC,aAAa;aAC1E,CAAC,CAAC;QACL,CAAC;QACD,IAAI,CAAC,IAAI,CAAC,UAAU;YAAE,OAAO,GAAG,CAAC;QACjC,MAAM,GAAG,IAAI,CAAC,UAAU,CAAC;IAC3B,CAAC;AACH,CAAC;AAED;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAyCG;AACH,MAAM,iBAAiB,GAAG,CAAC,CAAC;AAE5B,MAAM,CAAC,KAAK,UAAU,cAAc,CAAC,IAAe,EAAE,KAAa,EAAE,QAAgB,EAAE,OAAe;IACpG,MAAM,EAAE,GAAG,OAAO,CAAC,KAAK,EAAE,QAAQ,EAAE,OAAO,CAAC,CAAC;IAC7C,MAAM,EAAE,GAAG,MAAM,CAAC,KAAK,EAAE,QAAQ,EAAE,OAAO,CAAC,CAAC;IAC5C,kGAAkG;IAClG,kGAAkG;IAClG,8FAA8F;IAC9F,2FAA2F;IAC3F,+FAA+F;IAC/F,iGAAiG;IACjG,+FAA+F;IAC/F,+FAA+F;IAC/F,gGAAgG;IAChG,8FAA8F;IAC9F,KAAK,IAAI,KAAK,GAAG,CAAC,EAAE,KAAK,GAAG,iBAAiB,EAAE,KAAK,EAAE,EAAE,CAAC;QACvD,MAAM,GAAG,GAAG,MAAM,IAAI,CAAC,GAAG,CAAa,EAAE,CAAC,CAAC;QAC3C,IAAI,CAAC,GAAG;YAAE,OAAO,KAAK,CAAC,CAAC,8EAA8E;QACtG,8FAA8F;QAC9F,mGAAmG;QACnG,8FAA8F;QAC9F,oFAAoF;QACpF,IAAI,CAAC,MAAM,IAAI,CAAC,GAAG,CAAC,MAAM,CAAC,KAAK,EAAE,QAAQ,EAAE,OAAO,CAAC,CAAC,CAAC,KAAK,SAAS;YAAE,OAAO,KAAK,CAAC;QACnF,MAAM,QAAQ,GAAe,EAAE,GAAG,GAAG,EAAE,UAAU,EAAE,IAAI,CAAC,GAAG,EAAE,EAAE,QAAQ,EAAE,CAAC,GAAG,CAAC,QAAQ,IAAI,CAAC,CAAC,GAAG,CAAC,EAAE,CAAC;QACnG,gGAAgG;QAChG,qEAAqE;QACrE,EAAE;QACF,mGAAmG;QACnG,mGAAmG;QACnG,oGAAoG;QACpG,+GAA+G;QAC/G,MAAM,IAAI,CAAC,GAAG,CAAC,EAAE,EAAE,EAAE,CAAC,EAAE,CAAC,EAAE,OAAO,EAAE,CAAC,EAAE,MAAM,EAAE,CAAC,EAAE,GAAG,EAAE,YAAY,CAAC,QAAQ,CAAC,EAA2B,CAAC,CAAC;QAC1G,0FAA0F;QAC1F,2FAA2F;QAC3F,oEAAoE;QACpE,MAAM,KAAK,GAAG,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC,CAAC,CAAC,MAAM,IAAI,CAAC,GAAG,CAAC,EAAE,EAAE,QAAQ,CAAC,EAAE,IAAI,CAAC,CAAC,CAAC,CAAC,MAAM,IAAI,CAAC,UAAU,CAAC,EAAE,EAAE,GAAG,EAAE,QAAQ,CAAC,CAAC;QACjH,IAAI,CAAC,KAAK;YAAE,SAAS,CAAC,yEAAyE;QAC/F,MAAM,IAAI,CAAC,GAAG,CAAC,WAAW,CAAC,KAAK,EAAE,QAAQ,CAAC,EAAE,IAAI,CAAC,CAAC;QACnD,OAAO,IAAI,CAAC;IACd,CAAC;IACD,OAAO,CAAC,IAAI,CAAC,4FAA4F,KAAK,cAAc,QAAQ,WAAW,OAAO,WAAW,iBAAiB,0DAA0D,CAAC,CAAC;IAC9O,OAAO,KAAK,CAAC;AACf,CAAC","sourcesContent":["// @gnldev/events — durable event/notification bus on top of WorkStore. emit writes to an append-only log;\n// each consumer processes an event with its own ack marker (ackOnce=CAS): exactly-once MARKING +\n// at-least-once DELIVERY. The marker is written AFTER the handler SUCCEEDS → if the handler throws/the\n// process dies, the event isn't lost — a later poll retries it, spaced out by `retryDelayMs` (default\n// exponential 60s→1h) so a short downstream outage can't burn the whole attempt budget in a second.\n// Cost: a crash between handler success\n// and ackOnce (or a concurrent poll race) → redelivery is possible. Write the handler idempotently, or\n// use durable (runDurable/claim) inside the handler. Fan-out: N consumers → each gets every event at least once.\n// (WorkStore keeps it in its own namespace → doesn't pollute the RunJournal/replay reader.)\n// A handler that keeps throwing does NOT hold the topic hostage: other events keep being delivered\n// in the same pass, and after `maxAttempts` the event is QUARANTINED (dead-letter) rather than\n// retried forever — listDeadEvents() shows it, retryDeadEvent() hands it back. Quarantine is not an\n// ack: a quarantined event is never counted as delivered, because the consumer never saw it.\nimport { createPollLoop } from '@gnldev/durable';\nimport type { WorkStore } from '@gnldev/durable';\n\nexport interface EventMeta {\n id: string;\n topic: string;\n}\n\nexport type EventHandler = (payload: any, meta: EventMeta) => Promise<void> | void;\n\nexport interface ConsumerOptions {\n /** Consumer identity — ack markers are separated by this (fan-out). */\n name: string;\n /** start() poll interval (also the base for backoff growth on an empty queue). */\n pollMs?: number;\n /**\n * Empty-poll exponential backoff (default ON): if poll() delivers 0 events, the next poll\n * interval grows ×2 (ceiling: `maxPollMs ?? pollMs*32`) → prevents tens of thousands of empty\n * queries per second on an empty topic with many consumers (poll storm, audit finding). Once\n * something is delivered, the interval resets to `pollMs`. `false` → old behavior (fixed `pollMs` interval).\n */\n backoff?: boolean;\n /** Backoff ceiling (default `pollMs*32`). Only meaningful when `backoff !== false`. */\n maxPollMs?: number;\n /**\n * How many FAILED delivery attempts before an event is quarantined (dead-lettered) for THIS\n * consumer. Default 8, spread over ~2 hours by `retryDelayMs` — the same evtatt→evtdead shape as\n * @gnldev/queue's qatt→qfail, so an operator learns one dead-letter ontology, not two. (The number\n * is 8 rather than queue's 5 because these attempts are TIME-spaced: see `retryDelayMs`.)\n * `Infinity` = never quarantine (retry forever): delivery of OTHER events is still not blocked\n * (see the `frozen` cursor below), but this consumer's cursor stays parked behind the poison event\n * forever, so every poll re-scans the whole log from that point (the O(n) marker-check cost this\n * package spent 5.1 removing). Opt into that knowingly.\n */\n maxAttempts?: number;\n /**\n * How long to WAIT after a failed attempt before the event may be handed to the handler again.\n * Default: exponential from 60s, ×2 per attempt, capped at 1h — the same\n * `Math.min(base * 2 ** (attempt - 1), cap)` shape @gnldev/scheduler uses for its retries, with an\n * events-sized base/cap. A number = fixed spacing; `0` = retry on the very next poll.\n *\n * Why this exists: `maxAttempts` counts POLLS, not time. With the default 200ms poll interval a\n * 5-attempt budget burned out in about one second, so a one-second downstream blip quarantined\n * every in-flight event — permanently, needing an operator's `retryDeadEvent`. A retry budget is\n * only a real budget if the attempts are spread over the kind of outage it is meant to survive:\n * 8 attempts × this schedule = ~2h (60+120+240+480+960+1920+3600 seconds of waiting).\n *\n * A not-yet-due event is SKIPPED (not handed to the handler) but still FREEZES the consumer's\n * cursor — it is retryable, so nothing behind it may be marked as passed. Waiting is not giving up.\n *\n * CONTRACT for the function form: it must return a FINITE number of milliseconds. If it throws, or\n * returns Infinity/NaN/a non-number, the default schedule is used for that attempt and a warning is\n * logged — the delivery of every OTHER event on the topic is not the place to pay for a broken\n * schedule (a throw here used to escape poll() and stop the whole pass), and Infinity is not\n * storable: it survives in memory as \"never due again\" but a SQLite/Postgres WorkStore round-trips\n * it through JSON to null → 0 → due immediately, i.e. the same code behaving oppositely per adapter.\n * \"Retry forever\" is `maxAttempts: Infinity`, not a delay of Infinity.\n */\n retryDelayMs?: number | ((attempt: number) => number);\n}\n\n/**\n * The `evtatt:*` record. Was a bare `number` before retry spacing existed; a bare number is still\n * READ (treated as \"n failures, due now\") because a store written by an older build costs one branch\n * to keep readable — nothing is published yet, so this is courtesy, not a compatibility contract.\n */\ninterface StoredAttempts {\n /** Failed attempts in the CURRENT streak (a release resets this to 0). */\n n: number;\n /** When the streak started — \"how long has this been failing\", which `n` alone can't answer. */\n firstAt: number;\n /** Earliest time the next attempt may run. `0` = due now. */\n nextAt: number;\n /**\n * The RELEASE GENERATION this streak belongs to (`releaseStamp` of the dead record that handed it\n * back). Absent = never released, which is why it is omitted rather than written as the `0@0`\n * default: a first-failure record then keeps the exact shape it always had.\n *\n * It exists because the write below is a compare-and-swap on the record's VALUE, and value CAS is\n * ABA-blind: `retryDeadEvent` resets the counter to `{n:0, firstAt:0, nextAt:0}` EVERY time, so two\n * successive releases produce byte-identical records. A poll that read the first release's record,\n * ran the handler, and had the second release land underneath it would find its \"expected\" value\n * still matching and overwrite the new release anyway — measured: the CAS alone left the\n * \"a release means NOW\" case red. The stamp makes each release's record distinguishable, which is\n * the only thing a value CAS can detect.\n */\n gen?: string;\n}\n\n/** A delivery that reached `maxAttempts` failures for one consumer, as stored under `evtdead:*`. */\ninterface StoredDead {\n error: string;\n attempts: number;\n at: number;\n /** Set by `retryDeadEvent` — released back for delivery, not yet succeeded. */\n releasedAt?: number;\n /** How many times it has been released (quarantine → release → quarantine again). */\n releases?: number;\n}\n\n/** An event that has been quarantined for a consumer at least once — the operator-facing view. */\nexport interface DeadEvent extends StoredDead {\n id: string;\n topic: string;\n consumer: string;\n payload: unknown;\n /**\n * `quarantined` = parked, will NOT be delivered until released. `released` = `retryDeadEvent` has\n * handed it back, awaiting the next poll. `delivered` = it eventually succeeded; the record is kept\n * as history (same choice as queue's qfail: an append-only log, dead-letter history is permanent for audit).\n */\n status: 'quarantined' | 'released' | 'delivered';\n}\n\nexport interface Consumer {\n /**\n * Deliver this consumer's not-yet-marked events to the handler. Returns the count of\n * successfully marked (ackOnce won) deliveries. Delivery is at-least-once: if the handler\n * throws, the event is skipped (marker not written) and retried on a later poll, once its\n * `retryDelayMs` backoff has elapsed → the handler should be idempotent. Until then the event is\n * skipped but the cursor stays frozen behind it (waiting is not giving up, and not loss either).\n * After `maxAttempts` failures an event is QUARANTINED (`evtdead:*`,\n * visible via `listDeadEvents`) and stops being redelivered until `retryDeadEvent` releases it.\n */\n poll(): Promise<number>;\n start(): void;\n stop(): void;\n}\n\n/**\n * Phase 8 (audit finding: unbounded accumulation): if `maxDepth` is given — throws if, before\n * publishing, the topic depth (the TOTAL record count in the `evt:<topic>` namespace: delivered +\n * undelivered, an append-only log can't distinguish these without pruning) has reached/exceeded `maxDepth`.\n */\nexport class EventDepthExceededError extends Error {\n constructor(\n message: string,\n public readonly detail: { topic: string; depth: number; maxDepth: number },\n ) {\n super(message);\n this.name = 'EventDepthExceededError';\n }\n}\n\n/**\n * Counts records in the `ns` namespace only UP TO `limit` (early exit). WorkStore.list is paged\n * (default page size e.g. 50) — an exact count would read O(depth/pageSize) pages; here it's enough\n * to know \"was the limit exceeded\", so it stops once it reaches `limit` → cost is\n * O(min(actual depth, maxDepth)) pages, NOT the ENTIRE log. Unless `maxDepth` is given (default\n * behavior), this function is NEVER called → existing unbounded-topic behavior is preserved.\n */\nasync function countUpTo(work: WorkStore, ns: string, limit: number): Promise<number> {\n let count = 0;\n let cursor: string | undefined;\n for (;;) {\n const page = await work.list(ns, { cursor });\n count += page.items.length;\n if (count >= limit || !page.nextCursor) return count;\n cursor = page.nextCursor;\n }\n}\n\n/** Publish an event to a topic (idempotent: same id → a single event again). Returns eventId. */\nexport async function emit(\n work: WorkStore,\n topic: string,\n payload: unknown,\n opts: { id?: string; maxDepth?: number } = {},\n): Promise<string> {\n if (opts.maxDepth != null) {\n const depth = await countUpTo(work, logNsOf(topic), opts.maxDepth);\n if (depth >= opts.maxDepth) {\n throw new EventDepthExceededError(\n `@gnldev/events: topic depth limit exceeded (${depth} >= ${opts.maxDepth}) — event rejected (topic='${topic}').`,\n { topic, depth, maxDepth: opts.maxDepth },\n );\n }\n }\n return work.append(logNsOf(topic), payload, opts.id);\n}\n\n/**\n * Escapes ONE component of a `:`-delimited marker key, so that different components can never\n * produce the same key.\n *\n * WHY: the keys below are built by concatenation, and every component — topic, consumer name, event\n * id — is a string the CALLER chose. `:` is a perfectly ordinary character in all three\n * (`billing:eu`, `orders:created`, an URN as an event id), and it is also the delimiter. Measured\n * without this: topic `a` + consumer `b:c` and topic `a:b` + consumer `c` both produce\n * `evtack:a:b:c:order-42`, so the first consumer's ack marker made the second one's event\n * invisible — `poll()` returned 0 forever, `listDeadEvents` was empty, nothing was logged. Silent,\n * permanent loss of exactly the at-least-once delivery this package is for, from two name pairs\n * that are each entirely reasonable on their own. (Pinned per key family in key-ambiguity.test.ts.)\n *\n * Only `:` (the delimiter) and `%` (the escape character itself) are rewritten. That makes the\n * escape INJECTIVE — the two escape sequences are the only way a `%` can appear in the output, so a\n * key decomposes back to exactly one component triple — and injective is the whole requirement.\n *\n * NOT `encodeURIComponent`: it throws `URIError` on a lone surrogate, which a truncated UTF-16\n * string yields, and these key builders run inside `poll()` where `createPollLoop` swallows throws.\n * That would make a key escape a new way to silently stop the poll loop — the same failure\n * `delayFor` exists to keep caller input from causing. This escape is TOTAL: every string has one.\n *\n * THE `evt:<topic>` LOG NAMESPACE IS ESCAPED TOO, and this note used to say the opposite (\"topic is\n * its last and only variable component there, so that name is already unambiguous... a collision that\n * cannot happen\"). The name is indeed unambiguous; the reasoning was still wrong, because it argued\n * about the NAME while the failure is in the KEY THE STORE DERIVES FROM IT, which this package cannot\n * see. `WorkStore.list(ns)` is specified as a whole-value match, but RedisWorkStore had no ns column:\n * it stored records at `wl:<ns>:<id>` and read a namespace back with a `SCAN MATCH wl:<ns>:*` PREFIX\n * scan. Measured on a real Redis 7: a consumer of topic `orders` was delivered topic `orders:eu`'s\n * events (it never subscribed), and `emit(topic='inv', id='eu:x')` returned an event id for a record\n * `SET NX` had silently refused, because topic `inv:eu` + id `x` had already taken the same key — no\n * dead-letter row, no log line, the loss class this escape exists to remove. (Pinned in\n * log-namespace.test.ts.)\n *\n * The adapter is fixed at its own layer (redis-storage.ts `encNs`) and that fix is the load-bearing\n * one — it restores the port contract for EVERY caller, not just this package. Escaping here changes\n * no behavior on any of the four shipped adapters once it has landed. It stays because `WorkStore` is\n * a PUBLIC port: a store that addresses a record by a concatenated key is an ordinary way to\n * implement it, ours did it for months, and an escaped namespace is self-disambiguating under any of\n * them. A rule with an exception clause is what failed here; this package now has the rule without it.\n *\n * `encNs` is a SECOND IMPLEMENTATION of this same three-line escape, and deliberately so: one escape\n * SHAPE (`%3A`/`%25`), two layers that must be able to fail independently. Sharing a symbol would\n * mean deleting either one silently reconfigures the other — the opposite of the defence in depth\n * the paragraph above argues for. The two are pinned separately, and the `%` half of each is what\n * distinguishes them: log-namespace.test.ts asserts it once through this package (a topic literally\n * named `inv%3Aeu`) and once straight against the adapter (a namespace named `a%3Ab`).\n *\n * COST: a name (or event id) that actually contains `:` or `%` is now stored under a different key\n * than it was, and — since the namespace is escaped as well — a topic containing one is stored in a\n * different NAMESPACE than it was (`evt:orders:eu` → `evt:orders%3Aeu`), which on SQLite/Postgres is\n * a `gnl_work_log.ns` value, i.e. the events themselves move, not just their markers. Nothing is\n * published, so no store holds one yet; on a live deployment the old markers would be orphaned and\n * every event redelivered once, and a colon-topic's existing log would go unread until its `ns` was\n * renamed. Names without `:`/`%` — every test, every example, every documented key in the README —\n * are byte-identical before and after.\n */\nconst enc = (part: string) => part.replace(/%/g, '%25').replace(/:/g, '%3A');\n\n// Marker keys. Inlined at two places before (the consumer and nothing else); now the management\n// functions (listDeadEvents/retryDeadEvent) address the SAME keys, and a key format that two call\n// sites have to agree on is a key format that must exist in exactly one place.\n/** The append-log namespace an event lives in. The sixth key family — see `enc`. */\nconst logNsOf = (topic: string) => `evt:${enc(topic)}`;\nconst ackKey = (topic: string, consumer: string, id: string) => `evtack:${enc(topic)}:${enc(consumer)}:${enc(id)}`;\nconst attKey = (topic: string, consumer: string, id: string) => `evtatt:${enc(topic)}:${enc(consumer)}:${enc(id)}`;\nconst deadKey = (topic: string, consumer: string, id: string) => `evtdead:${enc(topic)}:${enc(consumer)}:${enc(id)}`;\nconst cursorKeyOf = (topic: string, consumer: string) => `evtcursor:${enc(topic)}:${enc(consumer)}`;\nconst rescanKeyOf = (topic: string, consumer: string) => `evtrescan:${enc(topic)}:${enc(consumer)}`;\n\n/** Reads an `evtatt:*` value in either shape (see StoredAttempts). `undefined` = never failed. */\nfunction readAttempts(raw: unknown): StoredAttempts | undefined {\n if (raw == null) return undefined;\n if (typeof raw === 'number') return { n: raw, firstAt: 0, nextAt: 0 }; // pre-spacing bare count → due now\n const r = raw as Partial<StoredAttempts>;\n return {\n n: r.n ?? 0, firstAt: r.firstAt ?? 0, nextAt: r.nextAt ?? 0,\n ...(r.gen !== undefined ? { gen: r.gen } : {}),\n };\n}\n\n/**\n * The identity of a RELEASE GENERATION. poll() reads the dead record before calling the handler and\n * writes the quarantine record after — an arbitrarily long window in which `retryDeadEvent` can land.\n * Comparing this stamp before/after is how an intervening release is detected, and it has to be a\n * pair: `releasedAt` alone is a wall clock (two releases in the same millisecond compare equal) and\n * `releases` alone doesn't move on the very first release. `0` for \"no record / never released\".\n */\nconst releaseStamp = (rec: StoredDead | undefined) => `${rec?.releases ?? 0}@${rec?.releasedAt ?? 0}`;\n\n/** The default retry schedule: @gnldev/scheduler's backoff shape (src/index.ts:104), events-sized. */\nconst defaultRetryDelay = (attempt: number) => Math.min(60_000 * 2 ** (attempt - 1), 3_600_000);\n\nexport function createConsumer(\n work: WorkStore,\n topic: string,\n handler: EventHandler,\n opts: ConsumerOptions,\n): Consumer {\n const ns = logNsOf(topic);\n const pollMs = opts.pollMs ?? 200;\n const backoffOn = opts.backoff ?? true;\n const maxPollMs = opts.maxPollMs ?? pollMs * 32;\n const maxAttempts = opts.maxAttempts ?? 8;\n const retryDelay: (attempt: number) => number =\n typeof opts.retryDelayMs === 'function' ? opts.retryDelayMs\n : opts.retryDelayMs != null ? () => opts.retryDelayMs as number\n : defaultRetryDelay;\n // Read cursor specific to this consumer (WorkStore KV): persistently holds the position \"all\n // events before this are a fully-scanned page for this consumer\" → subsequent polls won't list\n // previously (fully processed) pages again (5.1: fixes starvation + O(n)-per-poll at 50+ events).\n // Fan-out is unaffected: each consumer keeps its own cursor + its own ackOnce marker based on its\n // `opts.name`. Old (cursor-less) event logs also stream from the start (cursor=undefined).\n const cursorKey = cursorKeyOf(topic, opts.name);\n const rescanKey = rescanKeyOf(topic, opts.name);\n\n /**\n * `retryDelayMs` is CALLER CODE called from inside the catch block, where nothing was catching it.\n * A throwing schedule (a config read that blows up, an off-by-one on an array lookup) escaped\n * poll() entirely: measured on a 10-event topic with a throwing delay fn — 1 event delivered, the\n * attempt counter never written, and every later poll dying at the same line. Under `start()`\n * createPollLoop swallows it, so that is a SILENT full stop — the exact head-of-line blocking this\n * package exists to remove, re-entering through the caller's own hook. A bad schedule may cost the\n * caller its schedule; it may not cost the topic its delivery.\n *\n * Non-finite is rejected for a different reason: it is not STORABLE. `Infinity` survives in an\n * InMemory WorkStore (the event is never due again — a permanent freeze that isn't quarantine and\n * isn't visible to listDeadEvents), but a SQLite/Postgres WorkStore round-trips it through JSON to\n * `null` → read back as `0` → due IMMEDIATELY. Same code, same options, opposite behavior per\n * adapter. There is no honest clamp for \"infinity\" either, so it falls back like a throw does:\n * `maxAttempts: Infinity` is how \"retry forever\" is spelled here, not a delay of Infinity.\n */\n function delayFor(attempt: number, eventId: string): number {\n let raw: unknown;\n try {\n raw = retryDelay(attempt);\n } catch (e) {\n console.warn(`@gnldev/events: retryDelayMs threw (topic=${topic}, consumer=${opts.name}, event=${eventId}, attempt=${attempt}) — falling back to the default 60s→1h schedule:`, e);\n return defaultRetryDelay(attempt);\n }\n if (typeof raw !== 'number' || !Number.isFinite(raw)) {\n console.warn(`@gnldev/events: retryDelayMs returned ${String(raw)} (topic=${topic}, consumer=${opts.name}, event=${eventId}, attempt=${attempt}) — a delay must be a finite number of milliseconds (it has to survive the store's JSON round-trip); falling back to the default 60s→1h schedule.`);\n return defaultRetryDelay(attempt);\n }\n return Math.max(0, raw);\n }\n\n async function poll(): Promise<number> {\n // A release (retryDeadEvent) hands an event back that sits BEHIND the persisted cursor — the\n // cursor moved past it precisely because quarantine made it terminal. One flagged full pass is\n // how it gets back in view. It costs one extra scan of already-acked ids (cheap `get`s, the\n // same cost the old locked cursor paid on EVERY poll) and it is idempotent: the flag is cleared\n // only after a pass completes, so a crash mid-pass just rescans again.\n const rescan = (await work.get<boolean>(rescanKey)) === true;\n // The stored cursor is dropped, not just ignored for one pass. Quarantining the event is what\n // moved the cursor PAST it, so a pass that merely starts from the beginning would re-deliver it\n // and then — because a failed release freezes the cursor and never persists anything — fall back\n // to the same stale forward position on the next poll, leaving the event `released` and\n // unreachable forever. (Found by mutation: no test failed when retryDeadEvent stopped resetting\n // the attempt counter, and writing that test surfaced this instead.) `''` is the \"start of log\"\n // Sentinel — WorkStore KV has no delete, so a key cannot be returned to absent.\n //\n // This rewind is NOT protected from concurrent writers, and the comment here used to claim it\n // was (\"only poll() writes the cursor\"). poll() is the only cursor writer, but running two\n // processes under the SAME consumer name is a supported way to scale a consumer (see the\n // \"already DELIVERED by another process\" branch below), so there are as many cursor writers as\n // there are pollers: another process's pass can persist a forward cursor right after this\n // rewind and put the released event back out of view. That failure mode is the one\n // retryDeadEvent already documents — the release \"silently didn't take\", the remedy is to\n // release again, and `listDeadEvents` keeps showing it as `released` and undelivered, so it is\n // observable rather than lost. What IS safe here is crashing right after the rewind: a rescan\n // is a superset of a normal pass and skips acked ids by marker.\n if (rescan) await work.put(cursorKey, '');\n let cursor = rescan ? undefined : (await work.get<string>(cursorKey)) || undefined;\n let delivered = 0;\n // Has a RETRYABLE (failed, not yet quarantined) event been seen in this pass? Once true the\n // persisted cursor stops moving — but the pass KEEPS GOING through the remaining pages.\n let frozen = false;\n for (;;) {\n const page = await work.list(ns, { cursor });\n for (const e of page.items) {\n const ack = ackKey(topic, opts.name, e.id);\n // Cheap \"already marked?\" check (ackOnce markers live in the same work-KV space) →\n // events completed in previous polls don't go to the handler again.\n if ((await work.get(ack)) !== undefined) continue;\n // Quarantined (and not released): TERMINAL for this consumer — not delivered, and\n // deliberately NOT ack-marked, because \"we gave up on it\" is not \"the consumer saw it\".\n // It doesn't freeze the cursor either; that is the whole point of quarantining. It is not\n // silent: it was logged at console.error when it happened and it is listed by listDeadEvents.\n const dk = deadKey(topic, opts.name, e.id);\n const dead = await work.get<StoredDead>(dk);\n if (dead && !dead.releasedAt) continue;\n // Failed before and the backoff hasn't elapsed → NOT handed to the handler. The bookmark is\n // frozen all the same: a waiting event is still retryable, so advancing past it would be the\n // same silent loss as advancing past a failing one. (Without this, `maxAttempts` counted\n // polls: at pollMs=200 the whole budget burned in ~1s and a one-second outage dead-lettered\n // everything in flight. See ConsumerOptions.retryDelayMs.) COST: one extra KV `get` per\n // not-yet-acked event per pass (2 → 3, alongside the ack and dead-letter checks). It is paid\n // once per event on a healthy topic — an acked event never reaches this line — and there is\n // no cheaper place to keep it: the due time has to survive a restart, so it lives in the store.\n const attRaw = await work.get(attKey(topic, opts.name, e.id));\n const att = readAttempts(attRaw);\n if (att && att.nextAt > Date.now()) { frozen = true; continue; }\n // Contract: exactly-once MARKING + at-least-once DELIVERY. The marker is written AFTER\n // the handler; a crash between handler success and ackOnce (or a concurrent poll race) →\n // redelivery is possible. Write the handler idempotently, or use durable (runDurable/claim) inside it.\n try {\n await handler(e.payload, { id: e.id, topic });\n } catch (err) {\n // Handler threw → ack marker NOT WRITTEN → the event is not lost. The poll loop doesn't\n // die: this event is skipped, the rest of the page keeps processing.\n //\n // FRESH read of the dead record first. `dead` above was read BEFORE the handler ran, and\n // the handler can take arbitrarily long — long enough for an operator's retryDeadEvent to\n // land. Writing this attempt's bookkeeping from the stale read would undo that release\n // (drop `releasedAt` → back to `quarantined`, rewind `releases`, overwrite the attempt\n // reset) AFTER retryDeadEvent had already returned `true` to the operator. An attempt that\n // belongs to the previous release generation may not touch the new one at all.\n const fresh = await work.get<StoredDead>(dk);\n if (releaseStamp(fresh) !== releaseStamp(dead)) {\n frozen = true; // released mid-flight → still live, and it gets the release's fresh budget\n console.warn(`@gnldev/events: handler errored (topic=${topic}, consumer=${opts.name}, event=${e.id}) but the event was RELEASED while it ran — this attempt is discarded, the release stands:`, err);\n continue;\n }\n const now = Date.now();\n const n = (att?.n ?? 0) + 1;\n // An event that reached its LAST attempt is about to become an operator's problem, so it\n // is worth one extra `get` (only on this branch — never on the healthy or the still-\n // retrying path) to check it is still ours to give up on. Two processes under the SAME\n // consumer name is the supported way to scale a consumer: the other one can have delivered\n // and ACKED this event while our attempt was in flight, and our late failure would then\n // print \"event QUARANTINED\" and file a permanent dead-letter record for a delivery that\n // SUCCEEDED. listDeadEvents reports it honestly as `delivered` and retryDeadEvent correctly\n // refuses it, so nothing is lost — but an operator with an alarm on that line is paged for\n // a non-event, and an alarm that cries wolf is worse than no alarm.\n if (n >= maxAttempts && (await work.get(ack)) !== undefined) {\n console.warn(`@gnldev/events: handler errored (topic=${topic}, consumer=${opts.name}, event=${e.id}) but the event was already DELIVERED by another process under the same consumer name — this attempt is discarded, nothing is quarantined:`, err);\n continue; // NOT frozen: it is acked, so the cursor may pass it\n }\n const delay = delayFor(n, e.id); // called ONCE per failure; caller code, may throw (see delayFor)\n // The attempt counter USED TO BE a get→put pair with the comment \"quarantine happens a poll\n // or two late, NEVER EARLY, and never turns into loss\". The middle claim was false, and\n // measured: `attRaw` is read BEFORE the handler runs, so an operator's retryDeadEvent\n // landing anywhere in the handler's lifetime — a window the stamp check above cannot see,\n // because it compares the dead record read before the release, not this key — was\n // OVERWRITTEN by this put. The release had already returned `true`. Its effects both died:\n // the fresh budget (measured: an 8-attempt release turned into 1 attempt, so the very next\n // failure quarantined — early, not late) and the cleared backoff (measured: `nextAt` put an\n // hour into the future on an event the operator had just said \"now\" about).\n // So this is a CAS on its own key, exactly as the dead record's write below is — same\n // window, same technique, same standard. Absent key → nothing to race with (a release\n // implies a dead record implies a previous failure implies this key exists) and putIfMatch\n // is false-on-absent, so that case is a plain put. Losing the CAS means someone else moved\n // this event on: their record stands, this attempt is discarded, and nothing is quarantined\n // off a counter we no longer own. What remains from the old comment is the benign half —\n // two concurrent polls can still make an attempt go uncounted, which only ever DELAYS\n // quarantine.\n const ak = attKey(topic, opts.name, e.id);\n const next: StoredAttempts = {\n n,\n firstAt: att?.firstAt || now,\n nextAt: now + delay,\n ...(att?.gen !== undefined ? { gen: att.gen } : {}), // the streak keeps its generation\n };\n const wroteAtt = attRaw === undefined || !work.putIfMatch\n ? (await work.put(ak, next), true)\n : await work.putIfMatch(ak, attRaw, next);\n if (!wroteAtt) {\n frozen = true; // someone else changed the counter under us → theirs stands, not ours\n console.warn(`@gnldev/events: attempt write skipped — the retry record changed concurrently (topic=${topic}, consumer=${opts.name}, event=${e.id}); this attempt is discarded and the event stays live:`, err);\n continue;\n }\n if (n >= maxAttempts) {\n const rec: StoredDead = {\n error: String((err as any)?.message ?? err),\n attempts: n,\n at: now,\n ...(fresh?.releases != null ? { releases: fresh.releases } : {}),\n };\n // Conditional write where the store supports it (8.2 WorkStore.putIfMatch): the fresh\n // read above closes the handler-long window, this closes the sliver after it. If the key\n // doesn't exist yet there is nothing to race with — a release can only exist once a dead\n // record does — and putIfMatch is false-on-absent, so that case is a plain put.\n const wrote = fresh === undefined || !work.putIfMatch\n ? (await work.put(dk, rec), true)\n : await work.putIfMatch(dk, fresh, rec);\n if (!wrote) {\n frozen = true; // someone else changed the record under us → theirs stands, not ours\n console.warn(`@gnldev/events: quarantine write skipped — the dead-letter record changed concurrently (topic=${topic}, consumer=${opts.name}, event=${e.id}); the event stays live.`);\n continue;\n }\n console.error(`@gnldev/events: event QUARANTINED after ${n} failed attempts (topic=${topic}, consumer=${opts.name}, event=${e.id}) — it will NOT be redelivered until retryDeadEvent(); inspect with listDeadEvents():`, err);\n } else {\n frozen = true; // still retryable → the cursor must not move past it\n console.warn(`@gnldev/events: handler errored (topic=${topic}, consumer=${opts.name}, event=${e.id}, attempt=${n}/${maxAttempts}) — retrying in ${delay}ms:`, err);\n }\n continue;\n }\n // Handler SUCCEEDED → mark it now. false = another poll/instance finished the race first\n // (the handler may have run twice — idempotency is the handler's job); don't double-COUNT.\n if (await work.ackOnce(ack)) delivered++;\n }\n if (!page.nextCursor) {\n // End of the log. The last (partial) page's cursor is never persisted — it must be\n // rescanned for new events.\n if (rescan) await work.put(rescanKey, false); // the released event has been back in view for a full pass\n return delivered;\n }\n cursor = page.nextCursor;\n // A retryable (unmarked, un-quarantined) event is still behind us → the PERSISTED cursor\n // cannot advance past it: if it did, that event would never be scanned again = silent loss.\n // But the SCAN continues — freezing the bookmark is not a reason to stop delivering. That\n // conflation was the bug: `if (pageHasFailure) return delivered` stopped the whole pass, so\n // one poison event on page 1 held back every event behind it forever. Measured (real SQLite,\n // 120 events, the 4th always throwing): poll1=49, poll2=0, poll3=0 — events 50..119 were\n // never delivered at all. With the split: poll1=119, and after maxAttempts the poison event\n // is quarantined, which unfreezes the bookmark too.\n // Rejected: advancing the cursor past the failure anyway (a one-line fix) — that is exactly\n // the silent data loss this package refuses; the consumer never saw the event and nothing\n // would record that it had been skipped.\n if (!frozen) await work.put(cursorKey, cursor);\n }\n }\n\n // Phase 8.1: the tick/backoff/\"polling\" flag loop now lives in @gnldev/durable's shared\n // createPollLoop (was a triplicate copy across queue/events/scheduler) — behavior is identical:\n // if poll() delivers 0 events, the interval grows ×2 while backoffOn (ceiling maxPollMs); it\n // resets to pollMs once something is delivered. poll() catches handler errors internally (above);\n // remaining errors (store I/O etc.) are logged and swallowed by createPollLoop — the chain doesn't\n // die (an unhandled rejection doesn't crash the process).\n const loop = createPollLoop(async () => (await poll()) > 0, { pollMs, backoff: backoffOn, maxPollMs });\n\n return {\n poll,\n start: loop.start,\n stop: loop.stop,\n };\n}\n\n// ── Dead-letter (quarantine) inspection + release ─────────────────────────────\n// The operator-facing half of the fix. A poison event no longer blocks the topic, but \"doesn't\n// block\" is only acceptable if \"what happened to it\" is answerable. These two functions are that\n// answer, and they are deliberately the same pair @gnldev/queue exposes for jobs (listJobs /\n// retryJob) — one dead-letter vocabulary across the two packages, not two.\n\n/**\n * Every event that has been quarantined for `consumer` on `topic` — including ones later released\n * and delivered (`status`), because dead-letter history is permanent for audit (same choice as\n * queue's qfail: nothing is deleted from an append-only log).\n *\n * MANAGEMENT function: it reads the WHOLE topic log and does one `get` per event. Do NOT call it\n * from a poll loop — the loop reads only the pages it needs, via cursor (5.1).\n */\nexport async function listDeadEvents(work: WorkStore, topic: string, consumer: string): Promise<DeadEvent[]> {\n const out: DeadEvent[] = [];\n let cursor: string | undefined;\n for (;;) {\n const page = await work.list(logNsOf(topic), { cursor });\n for (const e of page.items) {\n const rec = await work.get<StoredDead>(deadKey(topic, consumer, e.id));\n if (!rec) continue;\n const acked = (await work.get(ackKey(topic, consumer, e.id))) !== undefined;\n out.push({\n ...rec,\n id: e.id,\n topic,\n consumer,\n payload: e.payload,\n status: acked ? 'delivered' : rec.releasedAt ? 'released' : 'quarantined',\n });\n }\n if (!page.nextCursor) return out;\n cursor = page.nextCursor;\n }\n}\n\n/**\n * Releases a quarantined event back for delivery to `consumer` (attempt counter reset to 0, backoff\n * cleared → due immediately). The next `poll()` re-scans the log from the start once — the consumer's\n * cursor had already moved past the event, which is why a flag rather than the cursor is what has to\n * change — and hands it to the handler again. Returns `false` (no-op) if the event was never\n * quarantined or has since been DELIVERED, mirroring `retryJob`, which only acts on jobs that\n * actually reached the dead-letter state — plus the contended case at the bottom of this note.\n *\n * IDEMPOTENT while `released`: calling it again on an already-released, not-yet-delivered event\n * re-asserts the release (re-arms the rescan flag, hands back a fresh attempt budget) and returns\n * `true`. That is not cosmetic — it is the ONLY way out of the race documented below. This function\n * used to return `false` there, which meant a release whose flag was swallowed by an in-flight poll\n * left the event `released` and unscanned forever: never delivered, never failing, never\n * re-quarantined, and refused by the very call its own docstring prescribed as the remedy. A\n * dead-letter that the package's API cannot get an event out of is not a dead-letter, it is a leak.\n * (`releases` counts every release that TOOK EFFECT, so a re-assertion increments it too — the\n * counter answers \"how many times was this handed back\", not \"how many quarantine cycles\".)\n *\n * Rejected alternative: re-EMIT the event under a new id (which is literally what queue's retryJob\n * does). It is simpler and needs no rescan flag, but a topic is FAN-OUT: a re-emitted copy is\n * delivered to EVERY consumer, so healing consumer A would force redelivery on healthy consumer B.\n * B's handler is contractually idempotent, so it would not corrupt anything — but manufacturing\n * redelivery for a consumer that never failed is not a repair, it's collateral. Release is\n * per-consumer for the same reason ack markers and cursors are per-consumer.\n *\n * Rejected alternative: have this function rewind the persisted cursor itself, to the released\n * event's page (storing that page cursor in the dead record). Cheaper than a full pass, but it adds\n * another writer to the cursor — one that writes it BACKWARDS, which is the direction that can be\n * lost: a poll already in flight (this consumer name may be run by several processes, see poll())\n * writes it forwards afterwards and the rewind is gone, with the failure mode \"the release silently\n * didn't take\". A flag is re-assertable and a cursor position is not, which is the whole difference:\n * The flag is not atomic against an in-flight poll either (one that started before the release can\n * clear it at the end of its own pass), but the cost is only that the release needs calling\n * again — which the idempotent re-release above makes possible — and it is observable rather than\n * silent: `listDeadEvents` keeps showing the event as `released` and undelivered.\n *\n * Returns `false` WITHOUT releasing in one further case: the dead record is being rewritten\n * concurrently faster than this function can read-modify-write it (`RELEASE_CAS_TRIES` lost\n * compare-and-swaps in a row). It is logged, and the remedy is to call again — the same remedy as\n * the lost-flag race. A silent overwrite would be the alternative, and that is what the CAS is here\n * to stop.\n */\nconst RELEASE_CAS_TRIES = 5;\n\nexport async function retryDeadEvent(work: WorkStore, topic: string, consumer: string, eventId: string): Promise<boolean> {\n const dk = deadKey(topic, consumer, eventId);\n const ak = attKey(topic, consumer, eventId);\n // Read → modify → write on a record other writers touch, so it is a compare-and-swap for the same\n // reason poll()'s two writes are: measured without it, two releases landing together (an operator\n // double-click, two panels, a retry script racing a human) both read the same record and both\n // wrote `releases: n + 1` from it — `both returned true? true true`, and the record showed\n // `releases: 1`. `releases` is an operator-facing number that answers \"how many times was this\n // handed back\"; answering it wrongly is worse than not answering. It is also the ABA input `gen`\n // exists to separate: equal `releases` plus (in the same millisecond) equal `releasedAt` is an\n // equal `releaseStamp`, which is precisely what a value CAS cannot see through. Losing the CAS\n // means someone else moved the record on, so this pass re-reads and re-applies on top of theirs\n // rather than clobbering it — a lost release must become a LATER release, not a vanished one.\n for (let tries = 0; tries < RELEASE_CAS_TRIES; tries++) {\n const rec = await work.get<StoredDead>(dk);\n if (!rec) return false; // never quarantined — a still-retrying or never-failed event needs no release\n // DELIVERED is the only terminal state: the ack marker is the single source of truth for \"the\n // consumer saw it\" (the same marker listDeadEvents reports as `status: 'delivered'`). `releasedAt`\n // being set is NOT terminal — see the idempotency note above. Re-read each pass: a concurrent\n // delivery is exactly the kind of thing that can land while a CAS is being retried.\n if ((await work.get(ackKey(topic, consumer, eventId))) !== undefined) return false;\n const released: StoredDead = { ...rec, releasedAt: Date.now(), releases: (rec.releases ?? 0) + 1 };\n // Order matters: the attempt reset and the dead record are written BEFORE the rescan flag, so a\n // poll that reacts to the flag can never see a half-applied release.\n //\n // The reset carries this release's STAMP. Without it the reset is `{n:0, firstAt:0, nextAt:0}` for\n // every release, byte-identical each time, and the poll's compare-and-swap on this key cannot tell\n // \"nobody touched it\" from \"a second release put it back to the same value\" — so a poll holding the\n // previous release's record would overwrite this one and its CAS would report success. See StoredAttempts.gen.\n await work.put(ak, { n: 0, firstAt: 0, nextAt: 0, gen: releaseStamp(released) } satisfies StoredAttempts);\n // The record always exists here (it was just read), so unlike poll()'s writes there is no\n // absent-key branch — only the documented drop for a store predating WorkStore.putIfMatch,\n // which keeps the old unconditional put and therefore the old race.\n const wrote = !work.putIfMatch ? (await work.put(dk, released), true) : await work.putIfMatch(dk, rec, released);\n if (!wrote) continue; // someone else wrote it between the read and here → re-read and re-apply\n await work.put(rescanKeyOf(topic, consumer), true);\n return true;\n }\n console.warn(`@gnldev/events: release abandoned — the dead-letter record kept changing under it (topic=${topic}, consumer=${consumer}, event=${eventId}, tries=${RELEASE_CAS_TRIES}); nothing was overwritten, call retryDeadEvent() again.`);\n return false;\n}\n"]}
package/package.json ADDED
@@ -0,0 +1,57 @@
1
+ {
2
+ "name": "@gnldev/events",
3
+ "version": "0.1.0",
4
+ "license": "Apache-2.0",
5
+ "engines": {
6
+ "node": ">=22.13.0"
7
+ },
8
+ "description": "Durable event/notification bus on @gnldev/durable. Journal-backed exactly-once delivery per consumer; fan-out.",
9
+ "keywords": [
10
+ "ai",
11
+ "agent",
12
+ "llm",
13
+ "typescript",
14
+ "ai-sdk",
15
+ "durable",
16
+ "exactly-once",
17
+ "pubsub",
18
+ "streaming"
19
+ ],
20
+ "type": "module",
21
+ "main": "./dist/index.js",
22
+ "types": "./dist/index.d.ts",
23
+ "exports": {
24
+ ".": {
25
+ "types": "./dist/index.d.ts",
26
+ "default": "./dist/index.js"
27
+ },
28
+ "./package.json": "./package.json"
29
+ },
30
+ "files": [
31
+ "dist"
32
+ ],
33
+ "peerDependencies": {
34
+ "@gnldev/durable": "^0.1.0"
35
+ },
36
+ "devDependencies": {
37
+ "@gnldev/durable": "0.1.0"
38
+ },
39
+ "author": "Karaca Yılmaz (https://gnl.dev)",
40
+ "homepage": "https://gnl.dev",
41
+ "bugs": {
42
+ "url": "https://github.com/Karaca7/gnldev/issues"
43
+ },
44
+ "repository": {
45
+ "type": "git",
46
+ "url": "git+https://github.com/Karaca7/gnldev.git",
47
+ "directory": "packages/events"
48
+ },
49
+ "publishConfig": {
50
+ "access": "public"
51
+ },
52
+ "scripts": {
53
+ "build": "tsc -p tsconfig.json",
54
+ "typecheck": "tsc -p tsconfig.json --noEmit",
55
+ "test": "vitest run"
56
+ }
57
+ }