@h-ai/reach 1.0.0-beta.2 → 1.0.0-beta.21
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.en.md +189 -97
- package/README.ja.md +188 -96
- package/README.md +181 -89
- package/dist/index.d.ts +57 -3
- package/dist/index.js +664 -301
- package/dist/index.js.map +1 -1
- package/package.json +11 -11
package/README.en.md
CHANGED
|
@@ -4,30 +4,15 @@
|
|
|
4
4
|
|
|
5
5
|
[Framework home](../../README.en.md) · Related modules: [@h-ai/cache](../cache/README.en.md) · [@h-ai/core](../core/README.en.md) · [@h-ai/reldb](../reldb/README.en.md)
|
|
6
6
|
|
|
7
|
-
## Introduction
|
|
8
|
-
|
|
9
7
|
A notification module supporting multiple providers (email, SMS, API callbacks) at once. Templates bind to providers for scenario-based routing. Includes do-not-disturb (DND) policies and configuration-defined templates.
|
|
10
8
|
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
## Core features
|
|
14
|
-
|
|
15
|
-
| Provider | Channel | Description |
|
|
16
|
-
| ------------ | ------- | -------------------------------------------------- |
|
|
17
|
-
| `console` | General | Console output (development/tests) |
|
|
18
|
-
| `smtp` | Email | SMTP email (requires nodemailer) |
|
|
19
|
-
| `aliyun-sms` | SMS | Alibaba Cloud SMS through HTTP API, without an SDK |
|
|
20
|
-
| `api` | General | HTTP API callback (generic webhook) |
|
|
21
|
-
|
|
22
|
-
## Getting started
|
|
9
|
+
## Quick Start
|
|
23
10
|
|
|
24
11
|
### Dependencies
|
|
25
12
|
|
|
26
|
-
- `@h-ai/reldb`: persistence for delivery logs and templates.
|
|
13
|
+
- `@h-ai/reldb`: persistence for delivery logs and templates. Initialize it before configuring nonempty `templates` or DND delay. Failed template persistence or required table preparation makes `reach.init()` return `CONFIG_ERROR` and close candidate providers. Optional for immediate sends without templates or delayed queues.
|
|
27
14
|
- `@h-ai/cache`: optional distributed locks for DND delay; enabled automatically when initialized.
|
|
28
15
|
|
|
29
|
-
### Quick start
|
|
30
|
-
|
|
31
16
|
#### Installation
|
|
32
17
|
|
|
33
18
|
```bash
|
|
@@ -39,35 +24,174 @@ Install `nodemailer` for email delivery.
|
|
|
39
24
|
#### Initialize multiple providers
|
|
40
25
|
|
|
41
26
|
```ts
|
|
27
|
+
import { core } from '@h-ai/core'
|
|
42
28
|
import { reach } from '@h-ai/reach'
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
29
|
+
import { reldb } from '@h-ai/reldb'
|
|
30
|
+
|
|
31
|
+
const runtime = core.module.createRuntime([
|
|
32
|
+
{
|
|
33
|
+
name: 'reldb',
|
|
34
|
+
start: () => reldb.init({ type: 'sqlite', database: ':memory:' }),
|
|
35
|
+
close: () => reldb.close()
|
|
36
|
+
},
|
|
37
|
+
{
|
|
38
|
+
name: 'reach',
|
|
39
|
+
start: () => reach.init({
|
|
40
|
+
providers: [
|
|
41
|
+
{ name: 'local', type: 'console' }
|
|
42
|
+
],
|
|
43
|
+
templates: [
|
|
44
|
+
{
|
|
45
|
+
name: 'welcome',
|
|
46
|
+
provider: 'local',
|
|
47
|
+
subject: 'Welcome',
|
|
48
|
+
body: 'Hello {userName}'
|
|
49
|
+
}
|
|
50
|
+
],
|
|
51
|
+
}),
|
|
52
|
+
close: () => reach.close()
|
|
53
|
+
},
|
|
54
|
+
])
|
|
55
|
+
const initialized = await runtime.start()
|
|
48
56
|
if (!initialized.success)
|
|
49
57
|
throw new Error(initialized.error.message)
|
|
58
|
+
|
|
59
|
+
let result: Awaited<ReturnType<typeof reach.send>>
|
|
50
60
|
try {
|
|
51
|
-
|
|
61
|
+
result = await reach.send({
|
|
52
62
|
to: 'user@example.com',
|
|
53
63
|
template: 'welcome',
|
|
54
|
-
vars: { userName: 'Alice' }
|
|
64
|
+
vars: { userName: 'Alice' }
|
|
55
65
|
})
|
|
56
|
-
if (!result.success)
|
|
57
|
-
throw new Error(result.error.message)
|
|
58
66
|
}
|
|
59
|
-
|
|
60
|
-
await
|
|
67
|
+
catch (error) {
|
|
68
|
+
const closed = await runtime.close()
|
|
69
|
+
throw new AggregateError(closed.success
|
|
70
|
+
? [error]
|
|
71
|
+
: [error, closed.error], 'Reach operation failed')
|
|
72
|
+
}
|
|
73
|
+
const closed = await runtime.close()
|
|
74
|
+
if (!closed.success) {
|
|
75
|
+
throw new AggregateError(result.success
|
|
76
|
+
? [closed.error]
|
|
77
|
+
: [result.error, closed.error], 'Reach cleanup failed')
|
|
61
78
|
}
|
|
79
|
+
if (!result.success)
|
|
80
|
+
throw new Error(result.error.message)
|
|
62
81
|
```
|
|
63
82
|
|
|
64
83
|
This example uses the development Console provider. It records delivery metadata and does not send email or SMS. Configure SMTP, SMS or API providers below for real delivery; initialize optional persistence/shared cache when needed.
|
|
65
84
|
|
|
66
|
-
`reach.config` returns a **redacted configuration snapshot**. Embedded credentials in `api.url` / `endpoint`,
|
|
85
|
+
`reach.config` returns a **redacted configuration snapshot**. Embedded credentials in `api.url` / `endpoint`, every custom header value, SMTP `pass` and SMS keys are hidden. Gateway bodies are omitted. Deployment stores API headers entirely in private environment files, regardless of header names.
|
|
86
|
+
|
|
87
|
+
Provider failures retain the complete module error definition, including HTTP status. Raw SMTP/SMS responses are excluded from returned messages and causes; clients receive localized guidance while server logs retain diagnostic context.
|
|
88
|
+
|
|
89
|
+
Each template send uses one template snapshot for its subject, body and default provider, keeping concurrent edits from mixing versions. An explicit `provider` still overrides the template route.
|
|
90
|
+
|
|
91
|
+
## Features
|
|
92
|
+
|
|
93
|
+
Template sends need only `template`, `to` and `vars`; the provider comes from the template. An explicit `provider` still overrides it. Direct sends require a provider; omitting it returns a failed HaiResult.
|
|
94
|
+
|
|
95
|
+
### Operations and results
|
|
96
|
+
|
|
97
|
+
Use the complete initialization flow in this section first. The host validates and authorizes example business values; check success before reading data from asynchronous domain results. Here void means completion without data, null means absent; synchronous factories/tools return the listed type directly.
|
|
67
98
|
|
|
68
|
-
|
|
99
|
+
name identifies a template, vars replacement values, template its provider/body/subject definition. queueId is the persisted UUID; recovery/resolution includes the queried version and verified operator decision, never an untrusted claim.
|
|
69
100
|
|
|
70
|
-
####
|
|
101
|
+
#### reach
|
|
102
|
+
|
|
103
|
+
Send takes provider/template, recipients and vars and yields SendResult (sent/queued); acceptance is not final delivery, reconcile unknown outcomes first.
|
|
104
|
+
|
|
105
|
+
| Entry | Input parameters | Successful result | Call example |
|
|
106
|
+
| ------------ | ---------------- | ----------------- | -------------------------------------------------------------------------------------------- |
|
|
107
|
+
| `reach.send` | message | `SendResult` | `await reach.send({ template: 'welcome', to: 'user@example.com', vars: { name: 'Alice' } })` |
|
|
108
|
+
|
|
109
|
+
#### reach.template
|
|
110
|
+
|
|
111
|
+
Templates: resolve name yields a template; save object/array or remove name yields void; list yields templates; render name/vars yields body, subject and provider.
|
|
112
|
+
|
|
113
|
+
| Entry | Input parameters | Successful result | Call example |
|
|
114
|
+
| -------------------------- | ---------------- | ------------------ | -------------------------------------------------------------------------------------------------------------- |
|
|
115
|
+
| `reach.template.resolve` | name | `ReachTemplate` | `await reach.template.resolve('welcome')` |
|
|
116
|
+
| `reach.template.save` | template | `void` | `await reach.template.save({ name: 'welcome', provider: 'mail', body: 'Hello {{name}}', subject: 'Welcome' })` |
|
|
117
|
+
| `reach.template.saveBatch` | templates | `void` | `await reach.template.saveBatch([{ name: 'welcome', provider: 'mail', body: 'Hello {{name}}' }])` |
|
|
118
|
+
| `reach.template.remove` | name | `void` | `await reach.template.remove('welcome')` |
|
|
119
|
+
| `reach.template.list` | No arguments | `ReachTemplate[]` | `await reach.template.list()` |
|
|
120
|
+
| `reach.template.render` | name, vars | `RenderedTemplate` | `await reach.template.render('welcome', { name: 'Alice' })` |
|
|
121
|
+
|
|
122
|
+
#### reach.queue
|
|
123
|
+
|
|
124
|
+
Operations query queueId or filtered states: get yields entry/null, list entries. Requeue dead requires expected version/policy; resolve unknown requires a verified decision; success is void, not automatic replay.
|
|
125
|
+
|
|
126
|
+
| Entry | Input parameters | Successful result | Call example |
|
|
127
|
+
| ---------------------------- | ------------------- | ------------------------- | ------------------------------------------------------------- |
|
|
128
|
+
| `reach.queue.get` | queueId | `ReachQueueEntry \| null` | `await reach.queue.get(queueId)` |
|
|
129
|
+
| `reach.queue.list` | options | `ReachQueueEntry[]` | `await reach.queue.list({ status: 'dead', limit: 20 })` |
|
|
130
|
+
| `reach.queue.requeueDead` | queueId, options | `void` | `await reach.queue.requeueDead(queueId, requeueDecision)` |
|
|
131
|
+
| `reach.queue.resolveUnknown` | queueId, resolution | `void` | `await reach.queue.resolveUnknown(queueId, verifiedDecision)` |
|
|
132
|
+
|
|
133
|
+
### Distributed locks
|
|
134
|
+
|
|
135
|
+
The optional cache lock reduces duplicate scans; nodes can scan concurrently after its 300-second TTL. Database delivery states provide correctness. A round claims at most 100 messages, one immediately before each delivery. An atomic owner/lease check persists `sending` before the external call; in-flight leases renew every third of their duration. Expired owners cannot revive leases, start another delivery, or commit results. Recovery continues every minute outside DND.
|
|
136
|
+
|
|
137
|
+
- States follow `pending → processing → sending → sent`. Expired processing records can be claimed safely.
|
|
138
|
+
- Only API providers configured with `idempotent: true` can automatically retry in-flight requests. This declares receiver capability. Every attempt uses the persistent queueId as both the `Idempotency-Key` header and `idempotencyKey` payload field. The receiver must atomically commit deduplication and side effects. Retention must cover late requests from resumed old workers; if lateness is unbounded, deduplication records must not expire. The same Provider name across nodes must use the same target and deduplication store.
|
|
139
|
+
- SMTP, Aliyun SMS, and default API providers have no external idempotency guarantee. Failure, exceptions, or lease loss after sending begins produce an unknown outcome. Expired non-idempotent sends become `unknown` and are never automatically resent. Disconnected providers use persisted backoff; invalid stored JSON goes directly to dead before any external call. Console produces development metadata only.
|
|
140
|
+
- Lease renewal, owner checks, and cancellation cannot undo an email, SMS, or HTTP side effect. The module provides no general exactly-once guarantee.
|
|
141
|
+
|
|
142
|
+
### Reviewing unknown outcomes
|
|
143
|
+
|
|
144
|
+
Trusted server administration can call `reach.queue.get(queueId)` to inspect `pending / processing / sending / unknown / sent / discarded / dead` without recipient or body data. For unknown outcomes, stop or isolate the old worker and verify Provider delivery records first. Then resolve to sent or discarded. UUID/status validation and a conditional update allow only one concurrent resolution; this never sends again. Discarding cannot cancel requests already started. Create a replacement only after confirming the old request can no longer take effect.
|
|
145
|
+
|
|
146
|
+
```ts
|
|
147
|
+
const state = await reach.queue.get(queueId)
|
|
148
|
+
const resolved = await reach.queue.resolveUnknown(queueId, { status: 'sent', messageId: verifiedMessageId })
|
|
149
|
+
if (!resolved.success)
|
|
150
|
+
throw new Error(resolved.error.message)
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
### Inspecting and restoring dead messages
|
|
154
|
+
|
|
155
|
+
Trusted server administration can call `reach.queue.list({ status: 'dead', limit: 50, offset: 0 })`, or list `unknown`. Pages contain at most 100 records and offset is bounded by 1000000. Get/list return status, attemptCount, maxAttempts, nextAttemptAt, lastFailure, and updatedAt without recipients, bodies, or owner credentials. Fix the root cause, read the latest dead version, and conditionally restore it:
|
|
156
|
+
|
|
157
|
+
```ts
|
|
158
|
+
const state = await reach.queue.get(queueId)
|
|
159
|
+
if (!state.success)
|
|
160
|
+
throw new Error(state.error.message)
|
|
161
|
+
if (state.data?.status === 'dead' && state.data.updatedAt !== undefined) {
|
|
162
|
+
const restored = await reach.queue.requeueDead(queueId, { expectedUpdatedAt: state.data.updatedAt })
|
|
163
|
+
if (!restored.success)
|
|
164
|
+
throw new Error(restored.error.message)
|
|
165
|
+
}
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
Only the matching dead version is restored; concurrent/stale requests fail, and unknown cannot be requeued. Restoration resets attempts while retaining queueId, the original policy, and receiver idempotency requirements. Dead means automatic processing stopped, not that no external effect occurred: late requests from an earlier idempotent API worker may still arrive. Confirm durable receiver deduplication before restoring. Leave irreparable payloads in dead for investigation; restoration does not repair their content.
|
|
169
|
+
|
|
170
|
+
### Production shared state
|
|
171
|
+
|
|
172
|
+
Production rejects console providers and requires SMTP, SMS or an actual API service. A log write is not a delivery receipt. Reconcile unknown outcomes from non-idempotent providers rather than claiming exactly-once delivery. Initialization checks apply when NODE_ENV or HAI_CORE_ENV is production; only explicit HAI_E2E=1/HAI_E2E_MOCK=1 tests are exempt. [State, concurrency and recovery (Chinese)](../../.agents/instructions/distributed-state.md).
|
|
173
|
+
|
|
174
|
+
### Credential notifications and persistence setup
|
|
175
|
+
|
|
176
|
+
Use `reach.send({ ..., sensitive: true, expiresAt, idempotencyKey })` for passwords, OTPs and reset tokens. expiresAt is Unix milliseconds; stable operation keys must contain no credentials. A shared ledger stores only metadata receipts/status and redacted destinations, never bodies, variables or credential hashes. There is no delayed credential queue. Success returns receiptId; DND/expired messages are not sent, and uncertain delivery is never automatically repeated. Explicit development console output is isolated from persistent observers. Production Console restrictions remain. Normal messages may use deferPolicy: reject to avoid a second queue.
|
|
177
|
+
|
|
178
|
+
`migrateReach({ dryRun: true })` previews persistence preparation performed during repository initialization. Stop on failure, back up and stop conflicting writers first, preserve the sending ledger and never treat schema preparation as permission to replay credentials.
|
|
179
|
+
|
|
180
|
+
### API acknowledgements and destination boundaries
|
|
181
|
+
|
|
182
|
+
An empty successful 2xx response, including 204/205, means receiver acceptance; messageId is optional and acceptance does not prove final delivery. Nonempty responses must be JSON objects with an optional messageId string of at most 255 characters. Responses are bounded to 64 KiB. Invalid responses, timeouts and network failures never become false success or expose receiver content.
|
|
183
|
+
|
|
184
|
+
API `redirect` defaults to `error`; ordinary messages may explicitly use `follow`. Sensitive messages always reject redirects without forwarding payloads or private headers. `requireHttps:true` enforces initial HTTPS and cannot be combined with follow. When omitted, production sensitive messages require HTTPS; explicit false is only for an operator-approved trusted internal connection. Unknown source-side effects still require reconciliation and sensitive messages are never retried automatically. Application allowlists and network egress controls remain necessary.
|
|
185
|
+
|
|
186
|
+
PostgreSQL/MySQL queue claims, renewals and final acknowledgements use the database execution clock inside SQL; batch cutoffs also come from the database. Node clock skew and row-lock waits cannot revive an expired owner. Local SQLite remains a development/test backend with a controllable test clock.
|
|
187
|
+
|
|
188
|
+
## Configuration
|
|
189
|
+
|
|
190
|
+
Lifecycle: init takes backend/business configuration; use the module only after success. Await close before releasing borrowed dependencies. Follow the quick-start example and public signatures for exact return values.
|
|
191
|
+
|
|
192
|
+
Provider names must be unique; duplicates fail with CONFIG_ERROR before connections open. `close()` waits for initialization and releases its providers, preventing late publication after shutdown.
|
|
193
|
+
|
|
194
|
+
### Console (development/tests)
|
|
71
195
|
|
|
72
196
|
```ts
|
|
73
197
|
const config = { name: 'dev', type: 'console' }
|
|
@@ -88,14 +212,31 @@ const config = { name: 'dev', type: 'console' }
|
|
|
88
212
|
|
|
89
213
|
#### Alibaba Cloud SMS
|
|
90
214
|
|
|
91
|
-
| Field | Type | Required | Default | Description
|
|
92
|
-
| ----------------- | -------- | -------- | ----------------------- |
|
|
93
|
-
| `name` | `string` | ✅ | — | Provider name
|
|
94
|
-
| `type` | `string` | ✅ | — | Fixed `'aliyun-sms'`
|
|
95
|
-
| `accessKeyId` | `string` | ✅ | — | AccessKey ID
|
|
96
|
-
| `accessKeySecret` | `string` | ✅ | — | AccessKey Secret
|
|
97
|
-
| `signName` | `string` | ✅ | — | SMS signature
|
|
98
|
-
| `endpoint` | `string` | — | `dysmsapi.aliyuncs.com` |
|
|
215
|
+
| Field | Type | Required | Default | Description |
|
|
216
|
+
| ----------------- | -------- | -------- | ----------------------- | -------------------------- |
|
|
217
|
+
| `name` | `string` | ✅ | — | Provider name |
|
|
218
|
+
| `type` | `string` | ✅ | — | Fixed `'aliyun-sms'` |
|
|
219
|
+
| `accessKeyId` | `string` | ✅ | — | AccessKey ID |
|
|
220
|
+
| `accessKeySecret` | `string` | ✅ | — | AccessKey Secret |
|
|
221
|
+
| `signName` | `string` | ✅ | — | SMS signature |
|
|
222
|
+
| `endpoint` | `string` | — | `dysmsapi.aliyuncs.com` | Direct HTTPS GET endpoint |
|
|
223
|
+
| `gateway` | `object` | — | — | Optional isolation gateway |
|
|
224
|
+
|
|
225
|
+
Add `gateway` to an `aliyun-sms` provider to send a POST request to the full `url`. Alibaba Cloud signature input uses `POST`, the RPC path `/`, and the query parameters. The gateway path, headers, and body apply to the gateway hop.
|
|
226
|
+
|
|
227
|
+
```yaml
|
|
228
|
+
gateway:
|
|
229
|
+
url: https://sms-gateway.example.com/dx
|
|
230
|
+
headers:
|
|
231
|
+
X-Gateway-Client: delivery
|
|
232
|
+
body: {}
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
`headers` accepts string values. `body` accepts a JSON object, an array, or raw text. Omit `body` for a request without content. Objects and arrays default to `application/json`, and text defaults to `text/plain; charset=utf-8`. Set `Content-Type` to override the default. HTTP on a trusted private network requires `allowInsecureHttp: true`. The signed query, phone numbers, and template variables travel in plaintext over HTTP. Redact query strings in gateway and proxy access logs.
|
|
236
|
+
|
|
237
|
+
Bind a synchronous or asynchronous server-side function with `reach.init(config, { gatewayHooks: { sms: buildGatewayFields } })`, using the configured provider name as the key. The hook receives `{ method, url, body, signal }`, with the static body already serialized, and can return `{ headers, body }`. Its headers override static values by case-insensitive name, and its body replaces the entire static body. Omitted fields inherit the static values. The HTTP client manages transport headers such as `Host` and `Content-Length`. Treat the signed URL as sensitive. The gateway snapshot in `reach.config` includes the URL and HTTP opt-in flag.
|
|
238
|
+
|
|
239
|
+
The hook and request share a 10-second deadline. Redirects are rejected. HTTP 2xx with Alibaba Cloud `Code: OK` confirms acceptance, and `BizId` can be used to check delivery. Uncertain outcomes follow the existing delivery policy, without falling back to direct delivery.
|
|
99
240
|
|
|
100
241
|
#### API callbacks
|
|
101
242
|
|
|
@@ -132,7 +273,7 @@ if (result.data.deferred) {
|
|
|
132
273
|
}
|
|
133
274
|
```
|
|
134
275
|
|
|
135
|
-
Queue columns `created_at`, `updated_at`, `locked_until`, and `next_attempt_at` store Unix milliseconds as `BIGINT` (64-bit `INTEGER` on SQLite). The Reach repository normalizes integer strings returned by drivers such as PostgreSQL into safe integer `number` values. Negative, fractional, or unsafe stored integers return `SEND_FAILED` instead of being silently rounded. Template creation/update `TIMESTAMP` fields map to `Date`. Update the complete table schema during deployment, including attempt counters, backoff policy, and failure classification; module-owned
|
|
276
|
+
Queue columns `created_at`, `updated_at`, `locked_until`, and `next_attempt_at` store Unix milliseconds as `BIGINT` (64-bit `INTEGER` on SQLite). The Reach repository normalizes integer strings returned by drivers such as PostgreSQL into safe integer `number` values. Negative, fractional, or unsafe stored integers return `SEND_FAILED` instead of being silently rounded. Template creation/update `TIMESTAMP` fields map to `Date`. Update the complete table schema during deployment, including attempt counters, backoff policy, and failure classification; module-owned persistence preparation maintains that structure (below).
|
|
136
277
|
|
|
137
278
|
#### Durable retry policy
|
|
138
279
|
|
|
@@ -146,7 +287,13 @@ The `queue` policy is persisted with each queued message. A restart or another n
|
|
|
146
287
|
|
|
147
288
|
Every claim atomically increments the stored `attemptCount`. Transient failures become eligible at `now + min(maxRetryDelayMs, retryDelayMs × 2^(attemptCount−1))`; the minute-based scan outside DND can deliver later. A round handles at most 100 messages using a fixed eligibility cutoff, so failed messages cannot reenter that round. Missing/disconnected providers, network errors, and HTTP 408/425/429/5xx allow bounded retry. Invalid stored JSON and other HTTP 4xx from an idempotent API go directly to `dead`. Exhaustion, including a crash after the last claim, also produces dead. In-flight non-idempotent requests still become `unknown` regardless of remaining attempts.
|
|
148
289
|
|
|
149
|
-
|
|
290
|
+
The host loads and validates configuration before passing it to this module. Keep secrets in private server environment files. Select the backend, capacity, deadlines and logging for the actual use case; reinitialize and check the result after changes.
|
|
291
|
+
|
|
292
|
+
[Per-option purpose, use cases and examples](../../docs/usage/configuration/operations.md).
|
|
293
|
+
|
|
294
|
+
## Error Handling
|
|
295
|
+
|
|
296
|
+
Check result.success for domain operations. Handle failures with the module error definitions and safe messages; retain codes, correlation IDs and sanitized causes in server logs. Synchronous tools, factories, stream iteration and lifecycle calls follow their own signatures. A completed Promise alone is not business success. See each feature for recovery limits.
|
|
150
297
|
|
|
151
298
|
All operations return `HaiResult<T>`. Error codes are defined in `HaiReachError`:
|
|
152
299
|
|
|
@@ -170,62 +317,7 @@ if (!result.success) {
|
|
|
170
317
|
}
|
|
171
318
|
```
|
|
172
319
|
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
The optional cache lock reduces duplicate scans; nodes can scan concurrently after its 300-second TTL. Database delivery states provide correctness. A round claims at most 100 messages, one immediately before each delivery. An atomic owner/lease check persists `sending` before the external call; in-flight leases renew every third of their duration. Expired owners cannot revive leases, start another delivery, or commit results. Recovery continues every minute outside DND.
|
|
176
|
-
|
|
177
|
-
- States follow `pending → processing → sending → sent`. Expired processing records can be claimed safely.
|
|
178
|
-
- Only API providers configured with `idempotent: true` can automatically retry in-flight requests. This declares receiver capability. Every attempt uses the persistent queueId as both the `Idempotency-Key` header and `idempotencyKey` payload field. The receiver must atomically commit deduplication and side effects. Retention must cover late requests from resumed old workers; if lateness is unbounded, deduplication records must not expire. The same Provider name across nodes must use the same target and deduplication store.
|
|
179
|
-
- SMTP, Aliyun SMS, and default API providers have no external idempotency guarantee. Failure, exceptions, or lease loss after sending begins produce an unknown outcome. Expired non-idempotent sends become `unknown` and are never automatically resent. Disconnected providers use persisted backoff; invalid stored JSON goes directly to dead before any external call. Console produces development metadata only.
|
|
180
|
-
- Lease renewal, owner checks, and cancellation cannot undo an email, SMS, or HTTP side effect. The module provides no general exactly-once guarantee.
|
|
181
|
-
|
|
182
|
-
### Reviewing unknown outcomes
|
|
183
|
-
|
|
184
|
-
Trusted server administration can call `reach.queue.get(queueId)` to inspect `pending / processing / sending / unknown / sent / discarded / dead` without recipient or body data. For unknown outcomes, stop or isolate the old worker and verify Provider delivery records first. Then resolve to sent or discarded. UUID/status validation and a conditional update allow only one concurrent resolution; this never sends again. Discarding cannot cancel requests already started. Create a replacement only after confirming the old request can no longer take effect.
|
|
185
|
-
|
|
186
|
-
```ts
|
|
187
|
-
const state = await reach.queue.get(queueId)
|
|
188
|
-
const resolved = await reach.queue.resolveUnknown(queueId, { status: 'sent', messageId: verifiedMessageId })
|
|
189
|
-
if (!resolved.success)
|
|
190
|
-
throw new Error(resolved.error.message)
|
|
191
|
-
```
|
|
192
|
-
|
|
193
|
-
### Inspecting and restoring dead messages
|
|
194
|
-
|
|
195
|
-
Trusted server administration can call `reach.queue.list({ status: 'dead', limit: 50, offset: 0 })`, or list `unknown`. Pages contain at most 100 records and offset is bounded by 1000000. Get/list return status, attemptCount, maxAttempts, nextAttemptAt, lastFailure, and updatedAt without recipients, bodies, or owner credentials. Fix the root cause, read the latest dead version, and conditionally restore it:
|
|
196
|
-
|
|
197
|
-
```ts
|
|
198
|
-
const state = await reach.queue.get(queueId)
|
|
199
|
-
if (!state.success)
|
|
200
|
-
throw new Error(state.error.message)
|
|
201
|
-
if (state.data?.status === 'dead' && state.data.updatedAt !== undefined) {
|
|
202
|
-
const restored = await reach.queue.requeueDead(queueId, { expectedUpdatedAt: state.data.updatedAt })
|
|
203
|
-
if (!restored.success)
|
|
204
|
-
throw new Error(restored.error.message)
|
|
205
|
-
}
|
|
206
|
-
```
|
|
207
|
-
|
|
208
|
-
Only the matching dead version is restored; concurrent/stale requests fail, and unknown cannot be requeued. Restoration resets attempts while retaining queueId, the original policy, and receiver idempotency requirements. Dead means automatic processing stopped, not that no external effect occurred: late requests from an earlier idempotent API worker may still arrive. Confirm durable receiver deduplication before restoring. Leave irreparable payloads in dead for investigation; restoration does not repair their content.
|
|
209
|
-
|
|
210
|
-
## Production shared state
|
|
211
|
-
|
|
212
|
-
Production rejects console providers and requires SMTP, SMS or an actual API service. A log write is not a delivery receipt. Reconcile unknown outcomes from non-idempotent providers rather than claiming exactly-once delivery. Initialization checks apply when NODE_ENV or HAI_CORE_ENV is production; only explicit HAI_E2E=1/HAI_E2E_MOCK=1 tests are exempt. [State, concurrency and recovery (Chinese)](../../.agents/instructions/distributed-state.md)。
|
|
213
|
-
|
|
214
|
-
## Credential notifications and upgrades
|
|
215
|
-
|
|
216
|
-
Use `reach.send({ ..., sensitive: true, expiresAt, idempotencyKey })` for passwords, OTPs and reset tokens. expiresAt is Unix milliseconds; stable operation keys must contain no credentials. A shared ledger stores only metadata receipts/status and redacted destinations, never bodies, variables or credential hashes. There is no delayed credential queue. Success returns receiptId; DND/expired messages are not sent, and uncertain delivery is never automatically repeated. Explicit development console output is isolated from persistent observers. Production Console restrictions remain. Normal messages may use deferPolicy: reject to avoid a second queue.
|
|
217
|
-
|
|
218
|
-
`migrateReach({ dryRun: true })` previews supported migrations, applied during repository initialization. Statuses/identifiers remain; legacy in-flight work becomes unknown. Legacy payloads are retired once and pending old payloads discarded. Back up and stop old writers before upgrading; never replay old credentials.
|
|
219
|
-
|
|
220
|
-
## API acknowledgements and destination boundaries
|
|
221
|
-
|
|
222
|
-
An empty successful 2xx response, including 204/205, means receiver acceptance; messageId is optional and acceptance does not prove final delivery. Nonempty responses must be JSON objects with an optional messageId string of at most 255 characters. Responses are bounded to 64 KiB. Invalid responses, timeouts and network failures never become false success or expose receiver content.
|
|
223
|
-
|
|
224
|
-
API `redirect` defaults to `error`; ordinary messages may explicitly use `follow`. Sensitive messages always reject redirects without forwarding payloads or private headers. `requireHttps:true` enforces initial HTTPS and cannot be combined with follow. When omitted, production sensitive messages require HTTPS; explicit false is only for an operator-approved trusted internal connection. Unknown source-side effects still require reconciliation and sensitive messages are never retried automatically. Application allowlists and network egress controls remain necessary.
|
|
225
|
-
|
|
226
|
-
PostgreSQL/MySQL queue claims, renewals and final acknowledgements use the database execution clock inside SQL; batch cutoffs also come from the database. Node clock skew and row-lock waits cannot revive an expired owner. Local SQLite remains a development/test backend with a controllable test clock.
|
|
227
|
-
|
|
228
|
-
## Development and testing
|
|
320
|
+
## Development
|
|
229
321
|
|
|
230
322
|
```bash
|
|
231
323
|
pnpm --filter @h-ai/reach test
|