@onlineapps/error-handler-core 1.1.15 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +36 -0
- package/README.md +46 -0
- package/package.json +12 -7
- package/src/CircuitBreaker.js +138 -13
- package/src/CircuitOpenError.js +30 -0
- package/src/CompensationHandler.js +9 -2
- package/src/DLQRouter.js +131 -24
- package/src/ErrorClassifier.js +17 -3
- package/src/ErrorHandlerCore.js +154 -47
- package/src/RetryHandler.js +88 -14
- package/src/UnifiedLogSchema.js +159 -38
- package/src/configRanges.js +92 -0
- package/src/index.js +4 -0
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# Changelog — @onlineapps/error-handler-core
|
|
2
|
+
|
|
3
|
+
All notable changes to this package. Follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) format.
|
|
4
|
+
|
|
5
|
+
## [Unreleased]
|
|
6
|
+
|
|
7
|
+
### Added
|
|
8
|
+
|
|
9
|
+
- **Regrese pro tři řádky tabulky, které ji neměly (d.331).** `401` → `BUSINESS` (neopakuje se), `408` → `TIMEOUT` a `429` → `RATE_LIMIT` (obojí se opakuje) — u každého se tvrdí verdikt I `shouldRetry`, protože klasifikace bez opakovatelnosti neříká nic o osudu zprávy. Právě těmhle třem statusům chyběl záznam v druhé, dnes už smazané mapě orchestrátoru, takže se rate-limitovaný krok neopakoval proti řádku `RATE_LIMIT` v `docs/standards/ERROR_HANDLING.md`. Tabulka je měřením v pořádku — chyběl jí hlas, kterým se ozve, až se pohne (`automation-gates.md` §5). Kontrolní případ pojmenuje důsledek jediného jména: chyba značkovaná vyřazeným `statusCode` propadne na hláškové vzory.
|
|
10
|
+
|
|
11
|
+
- `CircuitOpenError` (code `CIRCUIT_OPEN`): an open circuit is a typed error with a code, not a message string — thrown by `CircuitBreaker.execute()`, exported from the package index, classified by `ErrorClassifier` (`b1849b6b`, d.153d-2; entry written 2026-09-12 — the change had shipped to the tree without one).
|
|
12
|
+
|
|
13
|
+
### Removed
|
|
14
|
+
|
|
15
|
+
- `circuitBreakerOptions` on the `CircuitBreakerManager` constructor (d.309). d.188 had turned it into a declared, validated door, and that made visible what it actually was: opossum's spelling of the five options the constructor already names, i.e. a second rail for one concern with no justification on record (`.claude/rules/change-discipline.md` § One rail per concern). Nothing outside this package's own tests ever passed it. Use the named options.
|
|
16
|
+
|
|
17
|
+
### Changed
|
|
18
|
+
|
|
19
|
+
- **BREAKING** `UnifiedLogSchema.create()` accepts only a category `ErrorClassifier` declares as `errorType`, and `_determineSeverity()` has no `|| 'error'` left (d.310). The severity map is keyed off `ErrorTypes` rather than repeating the names, and a declared category with no severity is refused at `require()` time — the second list used to drift silently, filing a new category under a severity nobody chose. A missing value and an invented category are now one refusal with one message.
|
|
20
|
+
- **BREAKING** `DLQRouter._determineRoutingReason()` reads the same resolved `retryCount` the DLQ message records (d.310). It used to read `context.retryCount` on its own, so a message carrying the number disagreed with its own reason: three retries recorded, and a reason saying the budget played no part.
|
|
21
|
+
- `UnifiedLogSchema.create()` uses `??` for every field where `null` states an absence (d.310). `handling.circuitBreakerState`, `handling.circuitName` and `handling.maxRetries` kept a present falsy value instead of filing it as "we do not know"; the three `context` keys behave as before (the `...context` spread already re-applied them) and now simply match the file's one shape.
|
|
22
|
+
- **BREAKING** `UnifiedLogSchema.create()` requires `error` to be an `Error` instance and `errorType` to be a non-empty string (d.309). `|| 'Unknown error'` could only fire for something that is not an error, and it turned that caller defect into an entry reading like a real failure with a real message; `|| 'UNKNOWN'` let the schema speak a classification nobody made.
|
|
23
|
+
- **BREAKING** `CircuitBreakerManager.getCircuitBreaker()` / `executeWithCircuitBreaker()` check their per-call `options` against the same table of ranges as the constructor (d.309), before the circuit is created and regardless of whether the breaker is enabled. This was the second unchecked way into opossum: `executeWithCircuitBreaker(name, fn, { timeout: 0 })` expired every call before its first byte. An undeclared key or a non-object is refused with the accepted keys listed.
|
|
24
|
+
- **BREAKING** `DLQRouter.routeToDLQ()` checks `retryCount` (integer >= 0) and reads it through `??` (d.309). `message.retryCount || context.retryCount || 0` skipped a `retryCount: 0` on the message in favour of the context, and `_determineRoutingReason()` compares that number with `maxRetries` — so a dead message could be filed under a reason nobody produced. Both records of one dead message (the DLQ message and the `message_dlq` event) now state the same resolved number.
|
|
25
|
+
- `UnifiedLogSchema.create()` writes `handling.action` and `handling.retryDelay` as they arrive (`??`): `'throw'` recorded a decision nobody had taken yet — `logError()` passes `handling: {}` precisely because the decision comes later — and `retryDelay: 0` is a documented valid value that `||` filed as "we do not know" (d.309). `DLQRouter` keeps a falsy-but-present `error.code` for the same reason.
|
|
26
|
+
- **BREAKING** `UnifiedLogSchema.create()` requires `moduleName` and `operation` (d.187). `|| 'unknown-module'` / `|| 'unknown-operation'` filed the entry under a module and an operation of those names, and downstream that reads exactly like a real one — the defect d.178 removed for `serviceName`. A caller that omits either is refused with the key named.
|
|
27
|
+
- **BREAKING** `maxRetries` means a budget of **attempts** everywhere (d.189). `ErrorHandlerCore._determineAction()` read the same key as a budget of retries and granted one call more than `RetryHandler.executeWithRetry()` ever ran: `maxRetries: 3` meant three calls on one path and four on the other. A caller driving the loop from `result.shouldRetry` now stops one attempt earlier.
|
|
28
|
+
- **BREAKING** `circuitBreakerOptions` is a declared, validated door (d.188). The raw spread reached opossum after the d.177 range checks, so `{ circuitBreakerOptions: { timeout: 0 } }` expired every call before its first byte. Only the five declared opossum options are accepted, each against the same range; an undeclared key or a non-object is refused with the accepted keys listed.
|
|
29
|
+
- **BREAKING** `ErrorHandlerCore` validates `context.retryCount` (integer >= 0) instead of reading every value through `|| 0` (d.187/d.189); it now decides whether an attempt is left, so a nonsense value is refused rather than read as "none yet".
|
|
30
|
+
- `UnifiedLogSchema.create()` keeps a falsy-but-present `error.code` and `handling.retryCount` (`??`, not `||`): zero retries so far is a measurement, and `null` is reserved for the absence (d.187).
|
|
31
|
+
- `ErrorHandlerCore` resolves `serviceVersion` and `environment` through `runtimeCfg.resolve()` alone (d.187). The `config.x || resolved.x` beside it asked the same question a second time — one rail per concern; no behaviour change, the resolver already preferred the explicit value.
|
|
32
|
+
- **BREAKING** `ErrorClassifier.classify()` reads the HTTP status from `error.status`, the field the error contract names, instead of `error.statusCode`. A branded business error (`code` string + `status` number, `api/docs/biz/70-contracts/error-handling.md` §1) was missing both of the first two lookups and fell through to the message-pattern rules, so a 404 whose message contained the word "invalid" was filed as VALIDATION. No `status || statusCode` chain: one name only (`architecture-principles.md` §3). A caller that attaches `statusCode` alone is no longer classified by it.
|
|
33
|
+
|
|
34
|
+
## [1.1.16] — 2026-08-29
|
|
35
|
+
|
|
36
|
+
- Changelog started with this release; the history before it is in git (`git log -- shared/error-handler-core`).
|
package/README.md
CHANGED
|
@@ -1,3 +1,15 @@
|
|
|
1
|
+
> Status: current
|
|
2
|
+
> Owns: the platform's error handling itself — classification, retry, circuit breaker, DLQ routing and compensation
|
|
3
|
+
|
|
4
|
+
<!-- BEGIN GENERATED: library-uniform — regenerate: npx oa-sync-template readme-uniform --all -->
|
|
5
|
+
Uniform: [library/connector](../connector/conn-orch-validator/manifests/library.manifest.json)
|
|
6
|
+
|
|
7
|
+
Duty sections that apply:
|
|
8
|
+
|
|
9
|
+
- `all`: L-MAIN, L-ENGINES, L-TESTS, L-TEST-SCRIPT, L-PACK-TESTS, L-PINS, L-NO-FILE-RANGE, L-CHANGELOG, L-README, L-README-REGION, L-CONSUMER
|
|
10
|
+
- `connector`: L-CONNECTOR-ENV
|
|
11
|
+
<!-- END GENERATED: library-uniform -->
|
|
12
|
+
|
|
1
13
|
# @onlineapps/error-handler-core
|
|
2
14
|
|
|
3
15
|
Core error handling functionality for OA Drive microservices. Used by both business services (via `conn-infra-error-handler` wrapper) and infrastructure services (directly).
|
|
@@ -48,6 +60,14 @@ const errorHandler = new ErrorHandlerCore({
|
|
|
48
60
|
});
|
|
49
61
|
```
|
|
50
62
|
|
|
63
|
+
`maxRetries` is a budget of **attempts**, not of retries on top of a first call:
|
|
64
|
+
`maxRetries: 3` means at most three calls, whichever path drives them —
|
|
65
|
+
`executeWithRetry()` loops three times, and a caller driving the loop itself from
|
|
66
|
+
`handleError()`'s `result.shouldRetry` stops after the third as well. The other
|
|
67
|
+
`handling` keys, their defaults and their accepted ranges belong to the component
|
|
68
|
+
that reads each key (`RetryHandler`, `CircuitBreakerManager`, `DLQRouter`); a
|
|
69
|
+
value outside its range is refused in the constructor.
|
|
70
|
+
|
|
51
71
|
### Log Error
|
|
52
72
|
|
|
53
73
|
```javascript
|
|
@@ -59,6 +79,12 @@ await errorHandler.logError({
|
|
|
59
79
|
});
|
|
60
80
|
```
|
|
61
81
|
|
|
82
|
+
`serviceName` (on the handler), `moduleName`, `operation` and `error` are required
|
|
83
|
+
inputs, and `error` must be an `Error` instance. A missing one is refused with a
|
|
84
|
+
message naming it — never renamed to `unknown-service` / `unknown-module` /
|
|
85
|
+
`unknown-operation` and never given a substitute message, all of which downstream
|
|
86
|
+
reads exactly like the real thing.
|
|
87
|
+
|
|
62
88
|
### Handle Error
|
|
63
89
|
|
|
64
90
|
```javascript
|
|
@@ -103,6 +129,26 @@ const result = await errorHandler.executeWithCircuitBreaker({
|
|
|
103
129
|
});
|
|
104
130
|
```
|
|
105
131
|
|
|
132
|
+
An open circuit refuses with `CircuitOpenError` — a type, not a sentence, because
|
|
133
|
+
this is the one refusal a client sees verbatim (INFRA-DOCS audit P5, point 7):
|
|
134
|
+
|
|
135
|
+
```javascript
|
|
136
|
+
const { CircuitOpenError } = require('@onlineapps/error-handler-core');
|
|
137
|
+
|
|
138
|
+
try {
|
|
139
|
+
await errorHandler.executeWithCircuitBreaker('user-api', fn);
|
|
140
|
+
} catch (error) {
|
|
141
|
+
if (error.code === 'CIRCUIT_OPEN') {
|
|
142
|
+
// error.details.name === 'user-api'; error instanceof CircuitOpenError
|
|
143
|
+
}
|
|
144
|
+
throw error;
|
|
145
|
+
}
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
Branch on `error.code`, never on `error.message`: the wording follows
|
|
149
|
+
`.claude/rules/architecture-principles.md` §5 and may be improved, the code and
|
|
150
|
+
`details.name` are the contract.
|
|
151
|
+
|
|
106
152
|
## API Reference
|
|
107
153
|
|
|
108
154
|
See [ERROR_HANDLING.md](/docs/standards/ERROR_HANDLING.md) for the cross-cutting standard (classification table, fail-loudly rules).
|
package/package.json
CHANGED
|
@@ -1,14 +1,19 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@onlineapps/error-handler-core",
|
|
3
|
-
"version": "
|
|
3
|
+
"version": "2.0.0",
|
|
4
4
|
"description": "Core error handling functionality for OA Drive microservices - classification, retry, circuit breaker, DLQ routing",
|
|
5
|
+
"oa": {
|
|
6
|
+
"category": "connector"
|
|
7
|
+
},
|
|
8
|
+
"engines": {
|
|
9
|
+
"node": ">=24.0.0 <25"
|
|
10
|
+
},
|
|
5
11
|
"main": "src/index.js",
|
|
6
12
|
"scripts": {
|
|
7
|
-
"test": "
|
|
13
|
+
"test": "npm run test:unit",
|
|
8
14
|
"test:coverage": "jest --coverage",
|
|
9
15
|
"test:watch": "jest --watch",
|
|
10
|
-
"test:unit": "jest tests/unit"
|
|
11
|
-
"test:component": "jest tests/component"
|
|
16
|
+
"test:unit": "jest tests/unit"
|
|
12
17
|
},
|
|
13
18
|
"keywords": [
|
|
14
19
|
"error-handling",
|
|
@@ -20,9 +25,9 @@
|
|
|
20
25
|
"author": "OnlineApps",
|
|
21
26
|
"license": "MIT",
|
|
22
27
|
"dependencies": {
|
|
23
|
-
"@onlineapps/
|
|
24
|
-
"@onlineapps/runtime-config": "1.0
|
|
25
|
-
"opossum": "
|
|
28
|
+
"@onlineapps/logger-contract": "1.1.0",
|
|
29
|
+
"@onlineapps/runtime-config": "1.1.0",
|
|
30
|
+
"opossum": "10.0.0"
|
|
26
31
|
},
|
|
27
32
|
"devDependencies": {
|
|
28
33
|
"jest": "^29.7.0"
|
package/src/CircuitBreaker.js
CHANGED
|
@@ -4,17 +4,132 @@
|
|
|
4
4
|
*/
|
|
5
5
|
|
|
6
6
|
const CircuitBreaker = require('opossum');
|
|
7
|
+
const CircuitOpenError = require('./CircuitOpenError');
|
|
8
|
+
const { assertNumberInRange, describeValue } = require('./configRanges');
|
|
9
|
+
|
|
10
|
+
const POSITIVE_MS = 'a number > 0 (milliseconds)';
|
|
11
|
+
|
|
12
|
+
/**
|
|
13
|
+
* The opossum options this class accepts, spelled as opossum spells them, each
|
|
14
|
+
* with the range it must satisfy. Declared ONCE and used by both ways in: the
|
|
15
|
+
* named constructor options, and the per-call options of `getCircuitBreaker()` —
|
|
16
|
+
* so a value cannot be valid on one path and broken on the other.
|
|
17
|
+
*/
|
|
18
|
+
const CIRCUIT_OPTION_RULES = {
|
|
19
|
+
timeout: { exclusiveMin: 0, expected: POSITIVE_MS },
|
|
20
|
+
errorThresholdPercentage: { min: 1, max: 100, expected: 'a number between 1 and 100 (percent)' },
|
|
21
|
+
resetTimeout: { exclusiveMin: 0, expected: POSITIVE_MS },
|
|
22
|
+
rollingCountTimeout: { exclusiveMin: 0, expected: POSITIVE_MS },
|
|
23
|
+
rollingCountBuckets: { integer: true, min: 1, expected: 'an integer >= 1' }
|
|
24
|
+
};
|
|
25
|
+
|
|
26
|
+
const ACCEPTED_OPTIONS = Object.keys(CIRCUIT_OPTION_RULES);
|
|
27
|
+
|
|
28
|
+
/**
|
|
29
|
+
* Accept per-call `options` only as declared, validated options.
|
|
30
|
+
*
|
|
31
|
+
* Two ways led into opossum and only one of them was checked. d.188 closed the
|
|
32
|
+
* first (`circuitBreakerOptions`, since retired: it carried opossum's spelling of
|
|
33
|
+
* the five options the constructor already names, which is a second rail for one
|
|
34
|
+
* concern — `.claude/rules/change-discipline.md` § One rail per concern). This is
|
|
35
|
+
* the second: `getCircuitBreaker()` spread the caller's per-call options straight
|
|
36
|
+
* over `this.options`, so `executeWithCircuitBreaker(name, fn, { timeout: 0 })`
|
|
37
|
+
* expired every call before its first byte. One table of rules, both ways in —
|
|
38
|
+
* a value cannot be valid on one path and broken on the other
|
|
39
|
+
* (`.claude/rules/architecture-principles.md` §4 fail-fast, §8 explicit).
|
|
40
|
+
*
|
|
41
|
+
* @param {*} options - Value as received
|
|
42
|
+
* @returns {Object} The accepted options
|
|
43
|
+
* @throws {Error} If the value is not an object, carries an undeclared key, or a
|
|
44
|
+
* declared key outside its range
|
|
45
|
+
*/
|
|
46
|
+
function assertCircuitOptions(options) {
|
|
47
|
+
if (options === undefined) return {};
|
|
48
|
+
|
|
49
|
+
if (typeof options !== 'object' || options === null || Array.isArray(options)) {
|
|
50
|
+
throw new Error(
|
|
51
|
+
'[CircuitBreakerManager] Invalid configuration value - "options" must be an object '
|
|
52
|
+
+ `of declared opossum options, got ${describeValue(options)}. `
|
|
53
|
+
+ 'Fix: pass an object such as { resetTimeout: 5000 }, or omit it.'
|
|
54
|
+
);
|
|
55
|
+
}
|
|
56
|
+
|
|
57
|
+
for (const [key, value] of Object.entries(options)) {
|
|
58
|
+
const rule = CIRCUIT_OPTION_RULES[key];
|
|
59
|
+
if (rule === undefined) {
|
|
60
|
+
throw new Error(
|
|
61
|
+
`[CircuitBreakerManager] Unknown circuit-breaker option - "options.${key}" `
|
|
62
|
+
+ `is not a declared option. Expected one of: ${ACCEPTED_OPTIONS.join(', ')}. `
|
|
63
|
+
+ 'Fix: use a declared option, or drop the key.'
|
|
64
|
+
);
|
|
65
|
+
}
|
|
66
|
+
assertNumberInRange('CircuitBreakerManager', `options.${key}`, value, rule);
|
|
67
|
+
}
|
|
68
|
+
|
|
69
|
+
return options;
|
|
70
|
+
}
|
|
7
71
|
|
|
8
72
|
class CircuitBreakerManager {
|
|
9
|
-
|
|
10
|
-
|
|
73
|
+
/**
|
|
74
|
+
* Every option is optional and its default is declared HERE, once — this
|
|
75
|
+
* class owns them. Written as destructuring defaults, not `||`: opossum threw
|
|
76
|
+
* its own `||` out in 10.0.0 exactly so an explicit 0 survives (see the test
|
|
77
|
+
* file's header), and this constructor used to undo that one line earlier —
|
|
78
|
+
* `resetTimeout: 0` never reached opossum at all. See
|
|
79
|
+
* `.claude/rules/architecture-principles.md` §3.
|
|
80
|
+
*
|
|
81
|
+
* Each numeric option is checked against its range before it reaches opossum
|
|
82
|
+
* (`.claude/rules/architecture-principles.md` §4): once d.173 let an explicit
|
|
83
|
+
* 0 through, `timeout: 0` expired every call before it started and
|
|
84
|
+
* `rollingCountBuckets: 0` divided the statistics window by zero.
|
|
85
|
+
*
|
|
86
|
+
* @param {Object} [config] - Circuit breaker configuration
|
|
87
|
+
* @param {boolean} [config.circuitBreakerEnabled=true] - Enable the breakers
|
|
88
|
+
* @param {number} [config.timeout=10000] - Call timeout in ms; > 0
|
|
89
|
+
* @param {number} [config.errorThreshold=50] - Error threshold in percent; 1–100
|
|
90
|
+
* @param {number} [config.resetTimeout=30000] - Open → half-open delay in ms; > 0
|
|
91
|
+
* @param {number} [config.rollingCountTimeout=10000] - Stats window in ms; > 0
|
|
92
|
+
* @param {number} [config.rollingCountBuckets=10] - Stats buckets; integer >= 1
|
|
93
|
+
* @throws {Error} If a named numeric option is outside its range
|
|
94
|
+
*/
|
|
95
|
+
constructor({
|
|
96
|
+
circuitBreakerEnabled,
|
|
97
|
+
timeout = 10000,
|
|
98
|
+
errorThreshold = 50,
|
|
99
|
+
resetTimeout = 30000,
|
|
100
|
+
rollingCountTimeout = 10000,
|
|
101
|
+
rollingCountBuckets = 10
|
|
102
|
+
} = {}) {
|
|
103
|
+
// The rule for each option is spelled once, in CIRCUIT_OPTION_RULES; the key
|
|
104
|
+
// passed here is the one the CALLER wrote, so the message names their key
|
|
105
|
+
// (`errorThreshold` is this class's name for opossum's
|
|
106
|
+
// `errorThresholdPercentage`).
|
|
107
|
+
assertNumberInRange(
|
|
108
|
+
'CircuitBreakerManager', 'timeout', timeout, CIRCUIT_OPTION_RULES.timeout
|
|
109
|
+
);
|
|
110
|
+
assertNumberInRange(
|
|
111
|
+
'CircuitBreakerManager', 'errorThreshold', errorThreshold,
|
|
112
|
+
CIRCUIT_OPTION_RULES.errorThresholdPercentage
|
|
113
|
+
);
|
|
114
|
+
assertNumberInRange(
|
|
115
|
+
'CircuitBreakerManager', 'resetTimeout', resetTimeout, CIRCUIT_OPTION_RULES.resetTimeout
|
|
116
|
+
);
|
|
117
|
+
assertNumberInRange(
|
|
118
|
+
'CircuitBreakerManager', 'rollingCountTimeout', rollingCountTimeout,
|
|
119
|
+
CIRCUIT_OPTION_RULES.rollingCountTimeout
|
|
120
|
+
);
|
|
121
|
+
assertNumberInRange(
|
|
122
|
+
'CircuitBreakerManager', 'rollingCountBuckets', rollingCountBuckets,
|
|
123
|
+
CIRCUIT_OPTION_RULES.rollingCountBuckets
|
|
124
|
+
);
|
|
125
|
+
|
|
126
|
+
this.enabled = circuitBreakerEnabled !== false;
|
|
11
127
|
this.options = {
|
|
12
|
-
timeout
|
|
13
|
-
errorThresholdPercentage:
|
|
14
|
-
resetTimeout
|
|
15
|
-
rollingCountTimeout
|
|
16
|
-
rollingCountBuckets
|
|
17
|
-
...config.circuitBreakerOptions
|
|
128
|
+
timeout,
|
|
129
|
+
errorThresholdPercentage: errorThreshold,
|
|
130
|
+
resetTimeout,
|
|
131
|
+
rollingCountTimeout,
|
|
132
|
+
rollingCountBuckets
|
|
18
133
|
};
|
|
19
134
|
|
|
20
135
|
// Registry of circuit breakers
|
|
@@ -26,10 +141,17 @@ class CircuitBreakerManager {
|
|
|
26
141
|
*
|
|
27
142
|
* @param {string} name - Operation name
|
|
28
143
|
* @param {Function} fn - Function to protect
|
|
29
|
-
* @param {Object} options -
|
|
144
|
+
* @param {Object} [options] - Declared opossum options for this call; they
|
|
145
|
+
* override the instance ones and are checked against the same ranges
|
|
30
146
|
* @returns {CircuitBreaker} Circuit breaker instance
|
|
147
|
+
* @throws {Error} If options is not an object of declared, in-range options
|
|
31
148
|
*/
|
|
32
149
|
getCircuitBreaker(name, fn, options = {}) {
|
|
150
|
+
// Before anything else, including the disabled pass-through: an option out
|
|
151
|
+
// of range is the caller's defect whether or not this manager acts on it,
|
|
152
|
+
// and a defect reported only in some configurations is a defect that hides.
|
|
153
|
+
const accepted = assertCircuitOptions(options);
|
|
154
|
+
|
|
33
155
|
if (!this.enabled) {
|
|
34
156
|
// Return pass-through if disabled
|
|
35
157
|
return {
|
|
@@ -42,7 +164,7 @@ class CircuitBreakerManager {
|
|
|
42
164
|
if (!this.breakers.has(name)) {
|
|
43
165
|
const breaker = new CircuitBreaker(fn, {
|
|
44
166
|
...this.options,
|
|
45
|
-
...
|
|
167
|
+
...accepted
|
|
46
168
|
});
|
|
47
169
|
|
|
48
170
|
this.breakers.set(name, breaker);
|
|
@@ -56,9 +178,12 @@ class CircuitBreakerManager {
|
|
|
56
178
|
*
|
|
57
179
|
* @param {string} name - Operation name
|
|
58
180
|
* @param {Function} fn - Async function to execute
|
|
59
|
-
* @param {Object} options -
|
|
181
|
+
* @param {Object} [options] - Declared opossum options for this call
|
|
60
182
|
* @returns {Promise<*>} Function result
|
|
61
|
-
* @throws {
|
|
183
|
+
* @throws {CircuitOpenError} If the circuit is open (code 'CIRCUIT_OPEN')
|
|
184
|
+
* @throws {Error} If the operation itself fails
|
|
185
|
+
* @throws {Error} If options is not an object of declared, in-range options —
|
|
186
|
+
* before the circuit is created
|
|
62
187
|
*/
|
|
63
188
|
async executeWithCircuitBreaker(name, fn, options = {}) {
|
|
64
189
|
const breaker = this.getCircuitBreaker(name, fn, options);
|
|
@@ -67,7 +192,7 @@ class CircuitBreakerManager {
|
|
|
67
192
|
return await breaker.fire();
|
|
68
193
|
} catch (error) {
|
|
69
194
|
if (error.code === 'EOPENBREAKER') {
|
|
70
|
-
throw new
|
|
195
|
+
throw new CircuitOpenError(name);
|
|
71
196
|
}
|
|
72
197
|
throw error;
|
|
73
198
|
}
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* CircuitOpenError
|
|
3
|
+
*
|
|
4
|
+
* The one error of this platform a client sees verbatim: when a circuit is open
|
|
5
|
+
* the caller gets no upstream answer at all, only this. That makes the refusal a
|
|
6
|
+
* contract, not an internal note — and a contract is a type with a code, never a
|
|
7
|
+
* sentence somebody may reword (INFRA-DOCS audit P5, point 7).
|
|
8
|
+
*
|
|
9
|
+
* Branch on `code === 'CIRCUIT_OPEN'`; `details.name` carries the circuit whose
|
|
10
|
+
* breaker refused. The message follows
|
|
11
|
+
* `.claude/rules/architecture-principles.md` §5 and is owned here, so every
|
|
12
|
+
* raise site produces exactly the same text.
|
|
13
|
+
*/
|
|
14
|
+
|
|
15
|
+
class CircuitOpenError extends Error {
|
|
16
|
+
/**
|
|
17
|
+
* @param {string} circuitName - Name of the circuit whose breaker is open
|
|
18
|
+
*/
|
|
19
|
+
constructor(circuitName) {
|
|
20
|
+
super(
|
|
21
|
+
`[CircuitBreaker] Service unavailable - circuit "${circuitName}" is open. `
|
|
22
|
+
+ 'Expected: retry after resetTimeout, or check the upstream.'
|
|
23
|
+
);
|
|
24
|
+
this.name = 'CircuitOpenError';
|
|
25
|
+
this.code = 'CIRCUIT_OPEN';
|
|
26
|
+
this.details = { name: circuitName };
|
|
27
|
+
}
|
|
28
|
+
}
|
|
29
|
+
|
|
30
|
+
module.exports = CircuitOpenError;
|
|
@@ -17,7 +17,10 @@ class CompensationHandler {
|
|
|
17
17
|
*/
|
|
18
18
|
register(operation, handler) {
|
|
19
19
|
if (typeof handler !== 'function') {
|
|
20
|
-
throw new Error(
|
|
20
|
+
throw new Error(
|
|
21
|
+
`[CompensationHandler] Compensation handler for ${operation} must be a function - `
|
|
22
|
+
+ `got ${typeof handler}. Fix: register(operation, async (context) => { … }).`
|
|
23
|
+
);
|
|
21
24
|
}
|
|
22
25
|
|
|
23
26
|
this.handlers.set(operation, handler);
|
|
@@ -45,7 +48,11 @@ class CompensationHandler {
|
|
|
45
48
|
try {
|
|
46
49
|
return await handler(context);
|
|
47
50
|
} catch (error) {
|
|
48
|
-
throw new Error(
|
|
51
|
+
throw new Error(
|
|
52
|
+
`[CompensationHandler] Compensation failed for ${operation} - ${error.message}. `
|
|
53
|
+
+ 'Fix: the rollback of this operation did not complete; the state it was undoing may be half-written.',
|
|
54
|
+
{ cause: error }
|
|
55
|
+
);
|
|
49
56
|
}
|
|
50
57
|
}
|
|
51
58
|
|
package/src/DLQRouter.js
CHANGED
|
@@ -3,29 +3,100 @@
|
|
|
3
3
|
* Routes failed messages to DLQ for manual inspection
|
|
4
4
|
*/
|
|
5
5
|
|
|
6
|
+
const { assertLogger } = require('@onlineapps/logger-contract');
|
|
7
|
+
const { assertNumberInRange, assertQueuePattern } = require('./configRanges');
|
|
8
|
+
|
|
9
|
+
/**
|
|
10
|
+
* Retries already made before the message was dead-lettered. Read in two places
|
|
11
|
+
* (the DLQ message and the monitoring event), so the rule that reads it lives in
|
|
12
|
+
* one (`.claude/rules/change-discipline.md` § One rail per concern).
|
|
13
|
+
*
|
|
14
|
+
* `??`, not `||`: `retryCount: 0` on the message is a measurement, and `||`
|
|
15
|
+
* skipped it in favour of the context — which is not merely untidy, because
|
|
16
|
+
* `_determineRoutingReason()` compares that number with `maxRetries` and would
|
|
17
|
+
* file the message under a reason nobody produced.
|
|
18
|
+
*
|
|
19
|
+
* @param {Object} message - Original message
|
|
20
|
+
* @param {Object} context - Additional context
|
|
21
|
+
* @returns {number} Accepted retry count
|
|
22
|
+
* @throws {Error} If the value present is not an integer >= 0
|
|
23
|
+
*/
|
|
24
|
+
function resolveRetryCount(message, context) {
|
|
25
|
+
const retryCount = message.retryCount ?? context.retryCount ?? 0;
|
|
26
|
+
assertNumberInRange('DLQRouter', 'retryCount', retryCount, {
|
|
27
|
+
integer: true,
|
|
28
|
+
min: 0,
|
|
29
|
+
expected: 'an integer >= 0 (the number of retries already made)'
|
|
30
|
+
});
|
|
31
|
+
return retryCount;
|
|
32
|
+
}
|
|
33
|
+
|
|
6
34
|
class DLQRouter {
|
|
35
|
+
/**
|
|
36
|
+
* @param {Object} config - Configuration
|
|
37
|
+
* @param {Object} config.logger - Logger with info/warn/error/debug (required)
|
|
38
|
+
* @param {boolean} [config.dlqEnabled=true] - Enable DLQ routing
|
|
39
|
+
* @param {Object} [config.mqClient] - MQ client used to publish to the DLQ
|
|
40
|
+
* @param {string} [config.dlqNamePattern='{queue}.dlq'] - DLQ naming pattern;
|
|
41
|
+
* a non-empty string containing the `{queue}` placeholder
|
|
42
|
+
* @param {boolean} [config.emitMonitoringEvents=true] - Emit message_dlq events
|
|
43
|
+
* @throws {Error} If dlqNamePattern cannot name a per-origin queue
|
|
44
|
+
*/
|
|
7
45
|
constructor(config = {}) {
|
|
46
|
+
// `dlqNamePattern` and `mqClient` carry their documented defaults as
|
|
47
|
+
// destructuring defaults, declared here once: a default stands in for a
|
|
48
|
+
// MISSING key, it never overrules what the caller passed
|
|
49
|
+
// (`.claude/rules/architecture-principles.md` §3). `logger` is read from
|
|
50
|
+
// `config` below because assertLogger owns its (absent) default: it throws.
|
|
51
|
+
const { mqClient = null, dlqNamePattern = '{queue}.dlq' } = config;
|
|
52
|
+
assertQueuePattern('DLQRouter', 'dlqNamePattern', dlqNamePattern);
|
|
53
|
+
|
|
8
54
|
this.enabled = config.dlqEnabled !== false;
|
|
9
|
-
this.mqClient =
|
|
10
|
-
this.dlqNamePattern =
|
|
55
|
+
this.mqClient = mqClient;
|
|
56
|
+
this.dlqNamePattern = dlqNamePattern;
|
|
11
57
|
this.emitMonitoringEvents = config.emitMonitoringEvents !== false;
|
|
58
|
+
// Both failure paths below deliberately swallow their error so that DLQ
|
|
59
|
+
// routing never breaks error handling; the logger is what keeps them visible.
|
|
60
|
+
this.logger = assertLogger(
|
|
61
|
+
'DLQRouter',
|
|
62
|
+
config.logger,
|
|
63
|
+
'a DLQ routing failure this router deliberately swallows still reaches the structured log'
|
|
64
|
+
);
|
|
12
65
|
}
|
|
13
66
|
|
|
14
67
|
/**
|
|
15
68
|
* Route message to DLQ
|
|
16
69
|
*
|
|
17
70
|
* @param {Object} message - Original message
|
|
71
|
+
* @param {string} [message.queue] - Queue the message came from; required
|
|
72
|
+
* unless `context.queue` carries it
|
|
18
73
|
* @param {Error} error - Error that occurred
|
|
19
74
|
* @param {string} errorType - Classified error type
|
|
20
75
|
* @param {Object} context - Additional context
|
|
76
|
+
* @param {string} [context.queue] - Queue of origin, when the message body
|
|
77
|
+
* does not carry it
|
|
21
78
|
* @returns {Promise<void>}
|
|
79
|
+
* @throws {Error} If neither the message nor the context names the queue
|
|
22
80
|
*/
|
|
23
81
|
async routeToDLQ(message, error, errorType, context = {}) {
|
|
24
82
|
if (!this.enabled || !this.mqClient) {
|
|
25
83
|
return; // DLQ disabled or no MQ client
|
|
26
84
|
}
|
|
27
|
-
|
|
28
|
-
|
|
85
|
+
|
|
86
|
+
// The queue of origin is the only thing a dead message can be found by, so a
|
|
87
|
+
// missing one is refused here — before anything is published — rather than
|
|
88
|
+
// renamed. `… || 'unknown'` used to file every such message in `unknown.dlq`,
|
|
89
|
+
// a queue nobody is looking in, and said nothing about it. Owner's rule:
|
|
90
|
+
// no invented value (`api/docs/governance/confirmations/dlq-purge.md` 002).
|
|
91
|
+
const originalQueue = message.queue ?? context.queue;
|
|
92
|
+
if (typeof originalQueue !== 'string' || originalQueue.trim() === '') {
|
|
93
|
+
throw new Error(
|
|
94
|
+
'[DLQRouter] Missing required field - "queue" must be a non-empty string naming the queue the '
|
|
95
|
+
+ 'message came from; a dead message is found by nothing else. '
|
|
96
|
+
+ 'Fix: pass message.queue, or context.queue when the body does not carry it.'
|
|
97
|
+
);
|
|
98
|
+
}
|
|
99
|
+
const retryCount = resolveRetryCount(message, context);
|
|
29
100
|
const dlqName = this._generateDLQName(originalQueue);
|
|
30
101
|
|
|
31
102
|
const dlqMessage = {
|
|
@@ -36,16 +107,18 @@ class DLQRouter {
|
|
|
36
107
|
error: {
|
|
37
108
|
message: error.message,
|
|
38
109
|
stack: error.stack,
|
|
39
|
-
|
|
110
|
+
// `??`: a code that is present but falsy is a value; `null` is reserved
|
|
111
|
+
// for the absence of one.
|
|
112
|
+
code: error.code ?? null,
|
|
40
113
|
type: errorType,
|
|
41
114
|
timestamp: new Date().toISOString()
|
|
42
115
|
},
|
|
43
116
|
|
|
44
117
|
// Routing metadata
|
|
45
118
|
originalQueue: originalQueue,
|
|
46
|
-
retryCount
|
|
119
|
+
retryCount,
|
|
47
120
|
routedAt: new Date().toISOString(),
|
|
48
|
-
routingReason: this._determineRoutingReason(errorType, context)
|
|
121
|
+
routingReason: this._determineRoutingReason(errorType, context, retryCount)
|
|
49
122
|
};
|
|
50
123
|
|
|
51
124
|
try {
|
|
@@ -55,25 +128,43 @@ class DLQRouter {
|
|
|
55
128
|
|
|
56
129
|
// Emit message_dlq monitoring event
|
|
57
130
|
if (this.emitMonitoringEvents) {
|
|
58
|
-
await this._emitDLQEvent(message, dlqName, originalQueue, error, errorType, context);
|
|
131
|
+
await this._emitDLQEvent(message, dlqName, originalQueue, error, errorType, context, retryCount);
|
|
59
132
|
}
|
|
60
133
|
} catch (dlqError) {
|
|
61
134
|
// If DLQ routing fails, log but don't throw
|
|
62
135
|
// (we don't want DLQ failures to break error handling)
|
|
63
|
-
|
|
136
|
+
this.logger.error('[DLQRouter] DLQ routing failed', {
|
|
137
|
+
dlqName,
|
|
138
|
+
originalQueue,
|
|
139
|
+
errorType,
|
|
140
|
+
reason: dlqError.message
|
|
141
|
+
});
|
|
64
142
|
}
|
|
65
143
|
}
|
|
66
144
|
|
|
67
145
|
/**
|
|
68
146
|
* Emit message_dlq event to monitoring
|
|
147
|
+
*
|
|
148
|
+
* @param {number} retryCount - Already resolved and checked by the caller, so
|
|
149
|
+
* both records of one dead message state the same number
|
|
69
150
|
* @private
|
|
70
151
|
*/
|
|
71
|
-
async _emitDLQEvent(message, dlqName, originalQueue, error, errorType, context) {
|
|
152
|
+
async _emitDLQEvent(message, dlqName, originalQueue, error, errorType, context, retryCount) {
|
|
72
153
|
if (!this.mqClient) return;
|
|
73
|
-
|
|
154
|
+
|
|
155
|
+
// Resolved before the try so the catch below can name the workflow it failed on.
|
|
156
|
+
//
|
|
157
|
+
// `null`, not `'unknown'`: a dead-lettered message need not belong to a
|
|
158
|
+
// workflow, but an absent id is never renamed into one. The consumer of
|
|
159
|
+
// `monitoring.workflow` takes any non-empty string as a real id
|
|
160
|
+
// (`infra/api_monitoring/src/consumer/apiflow-extension.js`, readReported +
|
|
161
|
+
// the MISSING_WORKFLOW_ID gate), so `'unknown'` wrote `workflow_traces` rows
|
|
162
|
+
// claiming a workflow by that name; `null` is the absence the message
|
|
163
|
+
// actually states, and that gate counts it as faulty telemetry instead.
|
|
164
|
+
// Owner: `api/docs/governance/confirmations/dlq-purge.md` 002.
|
|
165
|
+
const workflowId = message.workflow_id ?? context.workflow_id ?? null;
|
|
166
|
+
|
|
74
167
|
try {
|
|
75
|
-
const workflowId = message.workflow_id || context.workflow_id || 'unknown';
|
|
76
|
-
|
|
77
168
|
await this.mqClient.publish('monitoring.workflow', {
|
|
78
169
|
event_type: 'message_dlq',
|
|
79
170
|
workflow_id: workflowId,
|
|
@@ -86,7 +177,7 @@ class DLQRouter {
|
|
|
86
177
|
originalQueue: originalQueue,
|
|
87
178
|
reason: error.message,
|
|
88
179
|
errorType: errorType,
|
|
89
|
-
retryCount
|
|
180
|
+
retryCount,
|
|
90
181
|
routedAt: new Date().toISOString()
|
|
91
182
|
}
|
|
92
183
|
},
|
|
@@ -97,30 +188,46 @@ class DLQRouter {
|
|
|
97
188
|
}
|
|
98
189
|
}, { persistent: true });
|
|
99
190
|
|
|
100
|
-
|
|
191
|
+
// One per dead-lettered message: a trace of the side effect, not an event
|
|
192
|
+
// of its own — the published message_dlq event is the record.
|
|
193
|
+
this.logger.debug('[DLQRouter] message_dlq event emitted', { workflowId, dlqName });
|
|
101
194
|
} catch (emitError) {
|
|
102
195
|
// Silent fail - don't break DLQ routing
|
|
103
|
-
|
|
196
|
+
this.logger.warn('[DLQRouter] monitoring event emit failed', {
|
|
197
|
+
workflowId,
|
|
198
|
+
dlqName,
|
|
199
|
+
reason: emitError.message
|
|
200
|
+
});
|
|
104
201
|
}
|
|
105
202
|
}
|
|
106
203
|
|
|
107
204
|
/**
|
|
108
|
-
* Generate DLQ name from original queue name
|
|
205
|
+
* Generate DLQ name from original queue name. The constructor has already
|
|
206
|
+
* refused a pattern without `{queue}`, so there is no second branch here: the
|
|
207
|
+
* one this method used to carry returned a pattern-shaped name for every
|
|
208
|
+
* origin, which is the collapse the check now prevents.
|
|
109
209
|
* @private
|
|
110
210
|
*/
|
|
111
211
|
_generateDLQName(originalQueue) {
|
|
112
|
-
|
|
113
|
-
return this.dlqNamePattern.replace('{queue}', originalQueue);
|
|
114
|
-
}
|
|
115
|
-
return this.dlqNamePattern;
|
|
212
|
+
return this.dlqNamePattern.replace('{queue}', originalQueue);
|
|
116
213
|
}
|
|
117
214
|
|
|
118
215
|
/**
|
|
119
|
-
* Determine routing reason
|
|
216
|
+
* Determine routing reason.
|
|
217
|
+
*
|
|
218
|
+
* Takes the SAME resolved `retryCount` the message records. It used to read
|
|
219
|
+
* `context.retryCount` on its own, so when the number came on the message the
|
|
220
|
+
* two disagreed: the entry said three retries and the reason said the budget
|
|
221
|
+
* had nothing to do with it.
|
|
222
|
+
*
|
|
223
|
+
* @param {string} errorType - Classified error type
|
|
224
|
+
* @param {Object} context - Additional context
|
|
225
|
+
* @param {number} retryCount - Resolved by `resolveRetryCount()`
|
|
226
|
+
* @returns {string} Routing reason
|
|
120
227
|
* @private
|
|
121
228
|
*/
|
|
122
|
-
_determineRoutingReason(errorType, context) {
|
|
123
|
-
if (
|
|
229
|
+
_determineRoutingReason(errorType, context, retryCount) {
|
|
230
|
+
if (retryCount >= context.maxRetries) {
|
|
124
231
|
return 'max_retries_exceeded';
|
|
125
232
|
}
|
|
126
233
|
|