@onlineapps/mq-client-core 4.0.0 → 4.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +147 -0
- package/README.md +84 -1
- package/package.json +1 -1
- package/src/BaseClient.js +7 -2
- package/src/config/queueConfig.js +205 -6
- package/src/index.js +21 -0
- package/src/monitoring-initiated.js +187 -0
- package/src/transports/rabbitmqClient.js +212 -49
package/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,153 @@ All notable changes to this package. Follows [Keep a Changelog](https://keepacha
|
|
|
4
4
|
|
|
5
5
|
## [Unreleased]
|
|
6
6
|
|
|
7
|
+
## [4.1.0] — 2026-09-23
|
|
8
|
+
|
|
9
|
+
### Added — plánovač se dozví KONEC běhu z vlastní fronty (d.818)
|
|
10
|
+
|
|
11
|
+
`queueConfig.scheduler` nese třetí položku, `workflow.events`. Je to fronta,
|
|
12
|
+
kterou si plánovač sám naváže na existující výměnu `monitoring.workflow.fanout`
|
|
13
|
+
a čte si z ní terminální událost běhu (`completed` / `failed` / `discarded`) —
|
|
14
|
+
místo aby se ptal pozorovatele přes HTTP nebo četl databázi, která není jeho
|
|
15
|
+
(rozhodnutí vlastníka, `api/docs/governance/confirmations/scheduler.md` 007).
|
|
16
|
+
Žádný producent se nemění; od knihovny je jen deklarace.
|
|
17
|
+
|
|
18
|
+
TTL je **vyslovená, ne zděděná**: `x-message-ttl: 300000`, stejně jako u
|
|
19
|
+
sourozence `monitoring.workflow` na téže výměně. Fanout neumí filtrovat, takže
|
|
20
|
+
fronta dostane všechny workflow události platformy a plánovač většinu zahodí; co
|
|
21
|
+
stojí výpadek delší než TTL, je zapsané v 007 (události se ztratí a běh uzavře
|
|
22
|
+
per-schedule timeout, který zůstává pojistkou). Dál `x-max-length: 10000` a
|
|
23
|
+
dead-letter přes VÝCHOZÍ výměnu do `scheduler.dlq` — bez deklarované cesty by
|
|
24
|
+
`consume()` frontu vůbec nepřipojil (conf `mq-consumer-contract` 002).
|
|
25
|
+
|
|
26
|
+
Import: `queueConfig.queueName('scheduler', 'workflow.events')` složí jméno,
|
|
27
|
+
`queueConfig.declarationOptions('scheduler.workflow.events')` vrátí argumenty,
|
|
28
|
+
kterými ji služba asserteuje — druhá kopie těch čísel u služby by se s touhle
|
|
29
|
+
potkala jako 406 v den, kdy se rozejdou.
|
|
30
|
+
|
|
31
|
+
### Added — `workflow_initiated` má jednu kolej: stavitel události vedle jejího přenosu (d.819)
|
|
32
|
+
|
|
33
|
+
`monitoring.buildWorkflowInitiatedEvent()` a `monitoring.publishWorkflowInitiated()`
|
|
34
|
+
— jediný stavitel události, ze které se monitoring dozví, že běh existuje (čte
|
|
35
|
+
z ní tenanta, štítky i spouštěč a teprve jí označí běh za `active`). Dosud ji
|
|
36
|
+
stavěly dvě služby, každá po svém; rozhodnutí vlastníka
|
|
37
|
+
(`api/docs/governance/confirmations/scheduler.md` 009) dalo kolej sem, do
|
|
38
|
+
knihovny, která už vlastní přenosovou půlku téhož drátu.
|
|
39
|
+
|
|
40
|
+
Vstup: `{ workflowId, redactedCookbook, serviceName, timestamp }`. Modul
|
|
41
|
+
**neredaguje** — co je v receptu citlivé, ví jedině spec služby v registru, a
|
|
42
|
+
číst ho by knihovně přidalo závislost na specech operací; volající předává
|
|
43
|
+
observability kopii, kterou si sám postavil. A **nic nevymýšlí**: každý klíč
|
|
44
|
+
`_system` musí existovat (`hasOwnProperty`, ne pravdivost), takže
|
|
45
|
+
`tenant_uuid: null` projde jako výslovná hodnota (conf
|
|
46
|
+
`run-envelope-tenant-uuid` 001) a chybějící klíč je hlášená chyba, ne tiché
|
|
47
|
+
`null`.
|
|
48
|
+
|
|
49
|
+
Publikace jde stávající `publishToMonitoringWorkflow`, tedy fail-safe: ztracená
|
|
50
|
+
kopie se hlásí, běh se kvůli ní neodmítá. Jediná výjimka je chybějící logger —
|
|
51
|
+
bez něj by slib „každé selhání se ohlásí" neplatil.
|
|
52
|
+
|
|
53
|
+
Import: `const { monitoring } = require('@onlineapps/mq-client-core');`
|
|
54
|
+
|
|
55
|
+
### Added — jméno `monitoring.resource` má vlastníka v jádru (d.619)
|
|
56
|
+
|
|
57
|
+
Topic exchange, po které jezdí události `resource.changed.*`, byla poslední
|
|
58
|
+
stranou téhle linky bez vlastníka jména: publikující strana ji nesla jako
|
|
59
|
+
konstantu modulu (`api/infra/api_monitoring/src/consumer/watchers/
|
|
60
|
+
resourceChangedPublisher.js`, `EXCHANGE_NAME`), bindující strana jako **povinný
|
|
61
|
+
env klíč** (`MONITORING_RESOURCE_EXCHANGE` v delivery-endpointu), zatímco fronta
|
|
62
|
+
na ni navázaná (`delivery.resource.events`) má vlastníka tady od d.270/d.278.
|
|
63
|
+
Jméno, na kterém se musí shodnout tři strany jedné linky, má mít jednoho
|
|
64
|
+
vlastníka — jinak jich má tolik, kolik je míst, která ho píší, a překlep nebo
|
|
65
|
+
override na jedné straně ztrácí události tiše (týž důvod jako u fanout výměny
|
|
66
|
+
`infrastructure.health.events`, d.270).
|
|
67
|
+
|
|
68
|
+
Nově: `queueConfig.monitoring.resource` deklaruje `type: 'exchange'`,
|
|
69
|
+
`exchangeType: 'topic'`, `durable: true` — tvar, který publikující strana
|
|
70
|
+
asserteuje a který vrací i běžící broker — a `topology.monitoringResourceExchange`
|
|
71
|
+
to jméno publikuje, složené z té deklarace (`queueConfig.queueName()`), ne
|
|
72
|
+
přepsané. Import: `const { topology } = require('@onlineapps/mq-client-core');`
|
|
73
|
+
|
|
74
|
+
Doklad proti realitě, ne proti sobě: integrační tier jméno ověřuje na živém
|
|
75
|
+
brokeru (`checkExchange`) a deklarovaný typ `assertExchange`em, s kontrolním
|
|
76
|
+
případem, že tatáž výměna asserteovaná jako `fanout` končí 406 — typ je tedy
|
|
77
|
+
změřený, ne předpokládaný.
|
|
78
|
+
|
|
79
|
+
Beze změny na drátě: žádné jméno se nepřejmenovává, výměna se nezakládá (
|
|
80
|
+
deklarace není pokyn k vytvoření — zakládá ji dál vlastník prefixu). Pro INFRA
|
|
81
|
+
z toho plyne, že povinný klíč `MONITORING_RESOURCE_EXCHANGE` může po re-pinu
|
|
82
|
+
zmizet stejně jako pět jmen před ním.
|
|
83
|
+
|
|
84
|
+
## [4.0.1] — 2026-09-17
|
|
85
|
+
|
|
86
|
+
### Fixed — zrušení konzumenta brokerem přestalo být tiché (d.605)
|
|
87
|
+
|
|
88
|
+
`basic.cancel` od brokera — smazaná fronta, přesun fronty při HA failoveru
|
|
89
|
+
(`x-cancel-on-ha-failover`) — doručí amqplib do callbacku `consume()` jako
|
|
90
|
+
zprávu `null`. Klient na ni odpovídal holým `return`: žádný záznam, žádná změna
|
|
91
|
+
stavu, žádné znovupřipojení. Značka konzumenta zůstala nastavená, a protože
|
|
92
|
+
právě ona je jediný fakt, kterým `performHealthCheck()` odpovídá „je konzument
|
|
93
|
+
TOHOTO klienta připojený", hlásil klient `healthy: true` o frontě, ze které už
|
|
94
|
+
nic nečetl. Ticho je vada stejné závažnosti jako špatný výsledek
|
|
95
|
+
(`automation-gates.md` §5).
|
|
96
|
+
|
|
97
|
+
Nově v tomto pořadí: značka se zahodí (týž fakt, který d.386b zavedl pro
|
|
98
|
+
odmítnutou re-registraci — od té chvíle je klient `unhealthy` a jmenuje frontu),
|
|
99
|
+
ztráta se zapíše na úrovni **error** se jménem fronty i zrušené značky, a
|
|
100
|
+
konzument se znovu připojí **kolejí, kterou už používá obnova kanálu**: tělo
|
|
101
|
+
smyčky `_reRegisterConsumers()` dostalo jméno (`_reRegisterConsumer()`) a cancel
|
|
102
|
+
volá tutéž funkci pro jednu frontu. Celá smyčka by znovu připojila i konzumenty,
|
|
103
|
+
kteří o své místo nepřišli.
|
|
104
|
+
|
|
105
|
+
Klasifikaci, deklaraci a prefetch tedy dělá `_prepareQueueForConsume()` —
|
|
106
|
+
vlastněná business fronta se znovu deklaruje, nevlastněná infrastrukturní se jen
|
|
107
|
+
`checkQueue` a její absence je odmítnutí se značkou `CONSUMER_QUEUE_MISSING`,
|
|
108
|
+
nikdy tiché založení. Odmítnuté znovupřipojení nechá konzumenta sledovaného bez
|
|
109
|
+
značky a důvod jede na události, kterou balíček už má —
|
|
110
|
+
`consumer:re-registration:failed`.
|
|
111
|
+
|
|
112
|
+
**Žádný nový veřejný kontrakt**: žádná nová metoda, žádná nová událost, žádná
|
|
113
|
+
změna podpisu. Každý stav, do kterého se cesta může dostat, už balíček hlásí.
|
|
114
|
+
|
|
115
|
+
Ticho zůstává tam, kde zrušení není vada: vlastní `cancelConsumer()` callback
|
|
116
|
+
vůbec nevolá (amqplib odregistruje konzumenta na `basic.cancel-ok`; broker
|
|
117
|
+
neposílá cancel), a cancel, který dorazí během `disconnect()`, je zapsán na
|
|
118
|
+
úrovni debug — úmyslné zavření není porucha.
|
|
119
|
+
|
|
120
|
+
Kdo z toho co má: každá služba, která konzumuje frontu, se po re-pinu na tuto
|
|
121
|
+
verzi dozví, že o konzumenta přišla — dnes to nepozná ani z logu, ani ze
|
|
122
|
+
zdravotní kontroly.
|
|
123
|
+
|
|
124
|
+
### Fixed — „připojený konzument" má jednu definici, a čte ji i levný reportér (d.613)
|
|
125
|
+
|
|
126
|
+
`getConsumerState()` vracelo jako `active` klíče `_activeConsumers`, tedy
|
|
127
|
+
konzumenty, které klient **sleduje**. `_performHealthCheck()` odpovídá na tutéž
|
|
128
|
+
otázku podle značky (`consumerInfo.consumerTag`), kterou d.386b a d.605 nechávají
|
|
129
|
+
nastavenou jen po dobu skutečného připojení. Jeden soubor, dvě definice slova
|
|
130
|
+
„aktivní" — a rozcházely se přesně tam, kde na tom záleží: konzument, jemuž
|
|
131
|
+
někdo smazal frontu a jehož znovupřipojení broker odmítl (406), zůstává
|
|
132
|
+
sledovaný bez značky, takže zdravotní kontrola hlásila 0 připojených, kdežto
|
|
133
|
+
`getConsumerState()` frontu dál jmenovalo jako aktivní.
|
|
134
|
+
|
|
135
|
+
To není kosmetika. `getConsumerState()` je ta **levná synchronní** odpověď, na
|
|
136
|
+
kterou se čtenář dostane: `docs/standards/rabbitmq-channel-best-practices.md`
|
|
137
|
+
§ „Consumers stop receiving after a reconnect" i
|
|
138
|
+
`docs/operations/rabbitmq-connection-recovery-troubleshooting.md` posílají
|
|
139
|
+
operátora právě sem pro tento případ — a dostal odpověď, že konzument je aktivní.
|
|
140
|
+
|
|
141
|
+
Nově obě čtení volají jednu privátní funkci `_isConsumerAttached(consumerInfo)`;
|
|
142
|
+
`active` jmenuje **připojené** fronty a `tracked` zůstává počtem sledovaných,
|
|
143
|
+
takže `tracked > active.length` je celá věta „tento klient z některé fronty nic
|
|
144
|
+
nebere". Tvar návratové hodnoty se nemění (žádný klíč nepřibyl ani nezmizel),
|
|
145
|
+
mění se pravdivost hodnoty `active`; mimo tento balíček ji dnes nečte žádný kód
|
|
146
|
+
(změřeno přes `api`, `api_biz/*`, `fe_adminui`), takže se nic dalšího
|
|
147
|
+
nepřepisuje.
|
|
148
|
+
|
|
149
|
+
Kdo z toho co má: `@onlineapps/conn-infra-mq` to po re-pinu pošle dál ve
|
|
150
|
+
`getHealth().consumers` a `@onlineapps/service-wrapper` z toho odvodí
|
|
151
|
+
`components.mq` — služba se ztraceným konzumentem přestane být v heartbeatu
|
|
152
|
+
`healthy`.
|
|
153
|
+
|
|
7
154
|
## [4.0.0] — 2026-09-16
|
|
8
155
|
|
|
9
156
|
### Changed — značku odmítnutí nese i deklarační rail, jednou funkcí pro obě cesty (d.582)
|
package/README.md
CHANGED
|
@@ -187,7 +187,10 @@ Owner decision: `api/docs/governance/confirmations/connector-logger-contract.md`
|
|
|
187
187
|
three channels, the consumers this client holds, and what is waiting in the
|
|
188
188
|
publish buffer (`{ size, inMemory }` — there is one buffer). All five refuse
|
|
189
189
|
before `connect()` has built a transport, rather than answering with an invented
|
|
190
|
-
zero.
|
|
190
|
+
zero. `getConsumerState()` answers `{ tracked, active }`: `tracked` counts the
|
|
191
|
+
consumers this client would restore, `active` names the ones ATTACHED right now
|
|
192
|
+
— the same question the health check asks — so `tracked > active.length` says
|
|
193
|
+
this client takes no message off some queue it registered.
|
|
191
194
|
- `disconnectAll()` (static, on the class and on the package export) - Close every
|
|
192
195
|
client this module holds; see § Shutdown.
|
|
193
196
|
- `onError(callback)` - Register error handler
|
|
@@ -328,6 +331,39 @@ There is no third meaning. `requeueOnError: false` together with `maxAttempts`
|
|
|
328
331
|
above 1 is a contradiction and is **refused where it was written**, rather than
|
|
329
332
|
one of the two being silently ignored.
|
|
330
333
|
|
|
334
|
+
#### A cancel from the BROKER — the consumer that was stopped without asking
|
|
335
|
+
|
|
336
|
+
`basic.cancel` sent by the broker is how a consumer learns its attachment is gone
|
|
337
|
+
while the channel and the connection stay up: the queue was deleted, or the broker
|
|
338
|
+
moved it (`x-cancel-on-ha-failover`). amqplib delivers it to the consume callback
|
|
339
|
+
as a `null` message, and it reaches nothing else in this client — the channel
|
|
340
|
+
handlers listen for `error` and `close`, and both stay silent.
|
|
341
|
+
|
|
342
|
+
It is **not** a silent event. In this order:
|
|
343
|
+
|
|
344
|
+
- the consumer's tag is dropped, because it names an attachment the broker no
|
|
345
|
+
longer holds — so `performHealthCheck()` stops counting that consumer as
|
|
346
|
+
attached the moment the cancel arrives, rather than one recovery later;
|
|
347
|
+
- the loss is logged at **error** level, naming the queue and the tag the broker
|
|
348
|
+
cancelled;
|
|
349
|
+
- the consumer is re-attached on the rail a channel recreate uses, for that queue
|
|
350
|
+
alone: the queue is classified, declared or checked and the prefetch re-applied
|
|
351
|
+
exactly as `consume()` does it. So a business queue the service owns is
|
|
352
|
+
re-declared, and an infrastructure queue this client does not own is only
|
|
353
|
+
checked — a deleted one is refused with `CONSUMER_QUEUE_MISSING`, never
|
|
354
|
+
silently recreated.
|
|
355
|
+
|
|
356
|
+
If that re-attachment is refused, the consumer stays **tracked with no tag** — the
|
|
357
|
+
same state a refused re-registration leaves — so the client reports itself
|
|
358
|
+
unhealthy and names the queue, `consumer:re-registration:failed` carries the
|
|
359
|
+
broker's reason and code, and the next channel recreate tries again.
|
|
360
|
+
|
|
361
|
+
Two cancels that are NOT this: `cancelConsumer()` on this client (the broker
|
|
362
|
+
answers `basic.cancel-ok`, the callback is never invoked, and the consumer is
|
|
363
|
+
dropped from the registry on purpose), and a cancel arriving while
|
|
364
|
+
`disconnect()` is running — an intentional close is not a fault, and it is
|
|
365
|
+
recorded at debug level.
|
|
366
|
+
|
|
331
367
|
### Connection loss — the contract
|
|
332
368
|
|
|
333
369
|
When the broker goes away, the transport runs connection-level recovery with
|
|
@@ -637,6 +673,53 @@ topology.telemetryLogsQueue = 'mine'; // TypeError — the object is froz
|
|
|
637
673
|
- A service that still carries one of these names as a private literal is carrying the
|
|
638
674
|
spelling of a fact declared here; importing it changes the wire by nothing at all.
|
|
639
675
|
|
|
676
|
+
### monitoring — `workflow_initiated`, the event a run is known by
|
|
677
|
+
|
|
678
|
+
Monitoring learns that a run exists from ONE event and from nowhere else: it reads the run's
|
|
679
|
+
tenancy, its labels and its trigger out of `workflow_initiated` and marks the run active on it.
|
|
680
|
+
A service that starts runs and does not file it starts runs that are invisible for their whole
|
|
681
|
+
life. So the event has one builder, here, beside the transport half that sends it
|
|
682
|
+
([`scheduler` 009](/docs/governance/confirmations/scheduler.md)).
|
|
683
|
+
|
|
684
|
+
```javascript
|
|
685
|
+
const { monitoring } = require('@onlineapps/mq-client-core');
|
|
686
|
+
|
|
687
|
+
await monitoring.publishWorkflowInitiated({
|
|
688
|
+
mqClient,
|
|
689
|
+
logger,
|
|
690
|
+
workflowId,
|
|
691
|
+
redactedCookbook, // already redacted by the caller
|
|
692
|
+
serviceName: 'api_gateway', // who files the event
|
|
693
|
+
timestamp: startedAt, // the run's instant, not a second reading of the clock
|
|
694
|
+
context: { correlationId } // optional, for the log lines
|
|
695
|
+
});
|
|
696
|
+
```
|
|
697
|
+
|
|
698
|
+
`monitoring.buildWorkflowInitiatedEvent({ workflowId, redactedCookbook, serviceName, timestamp })`
|
|
699
|
+
composes the body alone, so a suite can state it without a broker. It carries exactly these ten
|
|
700
|
+
keys: `event_type` (`initiated`), `workflow_id`, `service_name`, `step_index` (`-1`), `step_id`
|
|
701
|
+
(`workflow_start`), `cookbook`, `status` (`initiated`), `tenant_uuid`, `tenant_id`, `timestamp`.
|
|
702
|
+
|
|
703
|
+
What it does NOT do, and both halves are rules rather than omissions:
|
|
704
|
+
|
|
705
|
+
- **It does not redact.** What is sensitive in a cookbook is known only to the service spec in
|
|
706
|
+
the registry, and reading that spec would give this package a dependency on the operation
|
|
707
|
+
specs. The caller hands over the observability copy it built; a path that cannot redact does
|
|
708
|
+
not file the event at all, because an empty set of sensitive fields is silent consent to an
|
|
709
|
+
unredacted copy.
|
|
710
|
+
- **It invents no value.** `service_name` and the instant are the caller's, and every key it
|
|
711
|
+
reads out of `_system` must EXIST — `hasOwnProperty`, not truthiness. `tenant_uuid: null`
|
|
712
|
+
therefore passes through as the explicit value it is (a webhook run proves itself by
|
|
713
|
+
signature and has no personal identity), while a MISSING `_system.tenant_id` or
|
|
714
|
+
`_system.tenant_uuid` is a reported error. A `|| null` here would make the two cases
|
|
715
|
+
identical and hide a broken envelope.
|
|
716
|
+
|
|
717
|
+
`publishWorkflowInitiated()` is **fail-safe**: it never throws. It is observability about a run
|
|
718
|
+
that has already happened, so refusing the run because the observer could not be told would
|
|
719
|
+
trade a real effect for a record of it — a lost copy is reported through the logger and the call
|
|
720
|
+
answers `false`. The one exception is a missing logger, which is the transport half's rule:
|
|
721
|
+
without it the promise "every failure is reported" cannot hold.
|
|
722
|
+
|
|
640
723
|
### redactUrl — a connection URL that is safe to log
|
|
641
724
|
|
|
642
725
|
`RABBITMQ_URL` carries the broker account, so anything that renders it verbatim
|
package/package.json
CHANGED
package/src/BaseClient.js
CHANGED
|
@@ -1125,9 +1125,14 @@ class BaseClient {
|
|
|
1125
1125
|
}
|
|
1126
1126
|
|
|
1127
1127
|
/**
|
|
1128
|
-
* The consumers this client currently holds: how many, and
|
|
1128
|
+
* The consumers this client currently holds: how many it TRACKS, and which of
|
|
1129
|
+
* them are ATTACHED right now. A consumer whose queue was deleted, or whose
|
|
1130
|
+
* re-registration the broker refused, stays tracked without its tag — so
|
|
1131
|
+
* `tracked > active.length` says this client takes no message off some queue
|
|
1132
|
+
* it registered (d.613; one definition of "attached",
|
|
1133
|
+
* `RabbitMQClient._isConsumerAttached()`).
|
|
1129
1134
|
*
|
|
1130
|
-
* @returns {Object} `{ tracked: <count>, active: [<queue names>] }`
|
|
1135
|
+
* @returns {Object} `{ tracked: <count>, active: [<attached queue names>] }`
|
|
1131
1136
|
* @throws {ConnectionError} Before `connect()` has built a transport.
|
|
1132
1137
|
*/
|
|
1133
1138
|
getConsumerState() {
|
|
@@ -60,7 +60,8 @@ const SECTION_PREFIXES = Object.freeze({
|
|
|
60
60
|
monitoring: 'monitoring',
|
|
61
61
|
telemetry: 'telemetry',
|
|
62
62
|
alerts: 'alert',
|
|
63
|
-
registry: 'registry'
|
|
63
|
+
registry: 'registry',
|
|
64
|
+
scheduler: 'scheduler'
|
|
64
65
|
});
|
|
65
66
|
|
|
66
67
|
/**
|
|
@@ -81,7 +82,8 @@ const INFRASTRUCTURE_PREFIX_LOOKUPS = Object.freeze({
|
|
|
81
82
|
validation: 'getValidationQueueConfig',
|
|
82
83
|
monitoring: 'getMonitoringQueueConfig',
|
|
83
84
|
telemetry: 'getTelemetryQueueConfig',
|
|
84
|
-
delivery: 'getDeliveryQueueConfig'
|
|
85
|
+
delivery: 'getDeliveryQueueConfig',
|
|
86
|
+
scheduler: 'getSchedulerQueueConfig'
|
|
85
87
|
});
|
|
86
88
|
|
|
87
89
|
const INFRASTRUCTURE_PREFIXES = Object.freeze(Object.keys(INFRASTRUCTURE_PREFIX_LOOKUPS));
|
|
@@ -689,6 +691,33 @@ module.exports = {
|
|
|
689
691
|
durable: true
|
|
690
692
|
},
|
|
691
693
|
|
|
694
|
+
/**
|
|
695
|
+
* monitoring.resource - Topic exchange for resource-changed events
|
|
696
|
+
*
|
|
697
|
+
* The monitoring consumer's resource-changed watcher publishes here with a routing
|
|
698
|
+
* key of the shape `resource.changed.<resource_type>`; the delivery endpoint binds
|
|
699
|
+
* `delivery.resource.events` to it by the pattern `resource.changed.#` for WS push.
|
|
700
|
+
* Three sides of ONE wire — the publisher, this exchange and the bound queue — and
|
|
701
|
+
* until d.619 the name had an owner on none of them: a module constant on the
|
|
702
|
+
* publishing side and a required env key on the binding side, so an override or a
|
|
703
|
+
* typo on either side loses every event silently. That is the same reasoning d.270
|
|
704
|
+
* settled for the health-events fanout, and the reason the name lives here and is
|
|
705
|
+
* published by `src/index.js` § topology.
|
|
706
|
+
*
|
|
707
|
+
* `topic`, `durable`, no arguments: what the publisher asserts, and what the running
|
|
708
|
+
* broker answers for it. An exchange holds nothing to dead-letter, so — like every
|
|
709
|
+
* other exchange in this file — it carries no route, even though every QUEUE of this
|
|
710
|
+
* section routes to `monitoring.dlq`.
|
|
711
|
+
*
|
|
712
|
+
* @see api/infra/api_monitoring/src/consumer/watchers/resourceChangedPublisher.js
|
|
713
|
+
* @see api/docs/biz/70-contracts/resource-changed.md
|
|
714
|
+
*/
|
|
715
|
+
resource: {
|
|
716
|
+
type: 'exchange',
|
|
717
|
+
exchangeType: 'topic',
|
|
718
|
+
durable: true
|
|
719
|
+
},
|
|
720
|
+
|
|
692
721
|
/**
|
|
693
722
|
* monitoring.infrastructure.health.events - the monitoring consumer's own copy of
|
|
694
723
|
* the infrastructure health events, bound to the infrastructure.health.events
|
|
@@ -743,8 +772,8 @@ module.exports = {
|
|
|
743
772
|
* Telemetry queue configurations
|
|
744
773
|
*
|
|
745
774
|
* The OpenTelemetry intake of the monitoring consumer: every service publishes
|
|
746
|
-
* logs, metrics and spans to `telemetry.exchange` with a routing
|
|
747
|
-
* own name, and these three queues are bound to it
|
|
775
|
+
* logs, metrics and spans to the `telemetry.exchange` declared below, with a routing
|
|
776
|
+
* key carrying its own name, and these three queues are bound to it
|
|
748
777
|
* (`docs/standards/monitoring-queues.md`, `docs/architecture/monitoring.md`).
|
|
749
778
|
*
|
|
750
779
|
* The TTL and the cap are the ones the queues are created with today; this config is
|
|
@@ -758,6 +787,45 @@ module.exports = {
|
|
|
758
787
|
* @see api/docs/governance/confirmations/mq-consumer-contract.md 003
|
|
759
788
|
*/
|
|
760
789
|
telemetry: {
|
|
790
|
+
/**
|
|
791
|
+
* telemetry.exchange - the topic exchange every telemetry producer on the platform
|
|
792
|
+
* publishes to, and the one the three queues below are bound to.
|
|
793
|
+
*
|
|
794
|
+
* The widest wire there is: every biz service through `@onlineapps/service-wrapper`,
|
|
795
|
+
* every infrastructure service through `@onlineapps/monitoring-core` (its logger and
|
|
796
|
+
* its OpenTelemetry exporter), and the monitoring consumer, which asserts the
|
|
797
|
+
* exchange, binds the three queues and drains them — and since I16 publishes its own
|
|
798
|
+
* lines onto it as well.
|
|
799
|
+
*
|
|
800
|
+
* Until d.708 the name appeared HERE only in the prose above, while the other three
|
|
801
|
+
* platform exchanges of this file (`infrastructure.health.events`,
|
|
802
|
+
* `monitoring.workflow.fanout`, `monitoring.resource`) each had a template. So one
|
|
803
|
+
* name had three owners — a module constant in the monitoring consumer
|
|
804
|
+
* (`infra/api_monitoring/src/consumer/topology.js`) and a config default in each of
|
|
805
|
+
* monitoring-core's two publishers — which is exactly the shape d.270 gave an owner
|
|
806
|
+
* for the health-events fanout and d.619 for the resource-changed topic
|
|
807
|
+
* (`change-discipline.md` § One rail per concern).
|
|
808
|
+
*
|
|
809
|
+
* `topic`, `durable`, no arguments: what all three of those sides assert today
|
|
810
|
+
* (`assertExchange(<name>, 'topic', { durable: true })`), measured before this entry
|
|
811
|
+
* was written. The broker compares an exchange's type and durability at EVERY
|
|
812
|
+
* declaration, so a value invented here would be `406 PRECONDITION-FAILED` for
|
|
813
|
+
* whichever side booted second. An exchange holds nothing to dead-letter, so — like
|
|
814
|
+
* every other exchange in this file — it carries no route, even though every QUEUE
|
|
815
|
+
* of this section routes to `telemetry.dlq`.
|
|
816
|
+
*
|
|
817
|
+
* Declaring it is not creating it: the exchange is brought into being by the service
|
|
818
|
+
* that asserts it, the same rule as `telemetry.dlq` below.
|
|
819
|
+
*
|
|
820
|
+
* @see api/infra/api_monitoring/src/consumer/index.js — assertMonitoringTopology()
|
|
821
|
+
* @see api/shared/monitoring/monitoring-core/src/exporters/rabbitmq.js
|
|
822
|
+
*/
|
|
823
|
+
exchange: {
|
|
824
|
+
type: 'exchange',
|
|
825
|
+
exchangeType: 'topic',
|
|
826
|
+
durable: true
|
|
827
|
+
},
|
|
828
|
+
|
|
761
829
|
/** telemetry.logs.queue - OpenTelemetry log records, drained into Loki. */
|
|
762
830
|
'logs.queue': {
|
|
763
831
|
durable: true,
|
|
@@ -932,15 +1000,129 @@ module.exports = {
|
|
|
932
1000
|
},
|
|
933
1001
|
|
|
934
1002
|
/**
|
|
935
|
-
* registry.changes -
|
|
936
|
-
*
|
|
1003
|
+
* registry.changes - the fanout exchange registry index changes travel on.
|
|
1004
|
+
*
|
|
1005
|
+
* One module is every side of this wire: `@onlineapps/conn-orch-registry`,
|
|
1006
|
+
* `src/registryEventConsumer.js`. It holds the only spelling of the name (`:92`),
|
|
1007
|
+
* it is the only party that ASSERTS the exchange (`:117`,
|
|
1008
|
+
* `assertExchange(name, 'fanout', { durable: true })`), it binds each service's
|
|
1009
|
+
* `{service}.registry.events` to it with the pattern `''` (`:133`), and it publishes
|
|
1010
|
+
* the snapshot request onto it (`:331`, `routingKey: ''`). The publish path declares
|
|
1011
|
+
* nothing since d.342, so there is no second founder.
|
|
1012
|
+
*
|
|
1013
|
+
* Until d.712 this was the ONE exchange of this file with no `exchangeType`, so the
|
|
1014
|
+
* founder had to decide the kind on its own while the template only said "an
|
|
1015
|
+
* exchange, durable". `type` is one of the two properties the broker compares at
|
|
1016
|
+
* every declaration, which makes it exactly the half that cannot live somewhere
|
|
1017
|
+
* else (`change-discipline.md` § One rail per concern; the same step d.619 made for
|
|
1018
|
+
* `monitoring.resource` and d.708 for `telemetry.exchange`).
|
|
1019
|
+
*
|
|
1020
|
+
* `fanout`, `durable`, no arguments: measured on that founder, not chosen. The
|
|
1021
|
+
* binding pattern `''` only routes on a fanout, so the type is load-bearing rather
|
|
1022
|
+
* than decorative. An exchange holds nothing to dead-letter, so — like every other
|
|
1023
|
+
* exchange in this file — it carries no route.
|
|
1024
|
+
*
|
|
1025
|
+
* Declaring it is not creating it: the exchange is brought into being by the service
|
|
1026
|
+
* whose registry client subscribes to changes.
|
|
1027
|
+
*
|
|
1028
|
+
* @see api/shared/connector/conn-orch-registry/src/registryEventConsumer.js
|
|
937
1029
|
*/
|
|
938
1030
|
changes: {
|
|
939
1031
|
type: 'exchange',
|
|
1032
|
+
exchangeType: 'fanout',
|
|
940
1033
|
durable: true
|
|
941
1034
|
}
|
|
942
1035
|
},
|
|
943
1036
|
|
|
1037
|
+
/**
|
|
1038
|
+
* Scheduler queue configurations — the platform's one holder of time.
|
|
1039
|
+
*
|
|
1040
|
+
* The scheduler consumes requests for a run: a recurrence that came due, or a deferred
|
|
1041
|
+
* request somebody filed. Both are published by other services and both are the ONLY
|
|
1042
|
+
* record that the run was owed, so the family is shaped for loss-aversion rather than
|
|
1043
|
+
* for freshness (`docs/governance/confirmations/scheduler.md` 001, 004).
|
|
1044
|
+
*
|
|
1045
|
+
* @see api/infra/api_scheduler/README.md
|
|
1046
|
+
*/
|
|
1047
|
+
scheduler: {
|
|
1048
|
+
/**
|
|
1049
|
+
* scheduler.request - every request for a run, from every producer.
|
|
1050
|
+
*
|
|
1051
|
+
* NO `x-message-ttl`, deliberately, and it is the one property that differs from the
|
|
1052
|
+
* shape of a workflow queue: a request that waited out a consumer outage is still the
|
|
1053
|
+
* run that was due, and a TTL would turn that outage into runs which silently never
|
|
1054
|
+
* happened. What bounds the queue is its length, and what catches a request the
|
|
1055
|
+
* scheduler could not process is the family's dead-letter queue below — over the
|
|
1056
|
+
* DEFAULT exchange, where the routing key IS the destination name, the same way every
|
|
1057
|
+
* other infrastructure family here routes (the `dlx` exchange belongs to the business
|
|
1058
|
+
* templates, whose queues one service owns).
|
|
1059
|
+
*
|
|
1060
|
+
* Without this declaration `consume()` refuses to attach a consumer at all — a queue
|
|
1061
|
+
* with no declared route would have the broker DROP what the policy rejects
|
|
1062
|
+
* (`docs/governance/confirmations/mq-consumer-contract.md` 002).
|
|
1063
|
+
*/
|
|
1064
|
+
request: {
|
|
1065
|
+
durable: true,
|
|
1066
|
+
arguments: {
|
|
1067
|
+
'x-max-length': 10000,
|
|
1068
|
+
'x-dead-letter-exchange': '', // Default exchange: routing key IS the queue name
|
|
1069
|
+
'x-dead-letter-routing-key': 'scheduler.dlq'
|
|
1070
|
+
}
|
|
1071
|
+
},
|
|
1072
|
+
|
|
1073
|
+
/**
|
|
1074
|
+
* scheduler.workflow.events - where the scheduler learns that a run it started ENDED.
|
|
1075
|
+
*
|
|
1076
|
+
* The third queue bound to `monitoring.workflow.fanout`, beside `monitoring.workflow`
|
|
1077
|
+
* and `delivery.workflow.events`. The owner settled it in
|
|
1078
|
+
* `docs/governance/confirmations/scheduler.md` 007: the terminal event of a run
|
|
1079
|
+
* (`completed` / `failed` / `discarded`) already travels on that exchange, so the
|
|
1080
|
+
* scheduler reads it there rather than asking the observer over HTTP or reading a
|
|
1081
|
+
* database that is not its own. No producer changes; the binding is the scheduler's own
|
|
1082
|
+
* act at boot (the pattern `infra/api_delivery_endpoint/src/index.js:164` already uses),
|
|
1083
|
+
* and this file only DECLARES what that queue is.
|
|
1084
|
+
*
|
|
1085
|
+
* THE TTL IS STATED, NOT INHERITED, and it is a property of that decision rather than a
|
|
1086
|
+
* default: a fanout cannot filter, so this queue receives EVERY workflow event of the
|
|
1087
|
+
* platform and the scheduler discards most of them. 300 000 ms is what
|
|
1088
|
+
* `monitoring.workflow` carries on the same exchange - the sibling whose readings this
|
|
1089
|
+
* one follows. What an outage longer than the TTL costs is on record: those events are
|
|
1090
|
+
* lost and the run is closed by the per-schedule timeout, which 007 keeps as the
|
|
1091
|
+
* backstop rather than replacing it.
|
|
1092
|
+
*
|
|
1093
|
+
* The route goes over the DEFAULT exchange, like every other infrastructure family
|
|
1094
|
+
* here, and ends in this family's own `scheduler.dlq`. Without a declared route
|
|
1095
|
+
* `consume()` refuses to attach at all (`BaseClient.js`,
|
|
1096
|
+
* `CONSUMER_DEAD_LETTER_ROUTE_MISSING`; `mq-consumer-contract` 002).
|
|
1097
|
+
*
|
|
1098
|
+
* @see api/docs/governance/confirmations/scheduler.md 007
|
|
1099
|
+
*/
|
|
1100
|
+
'workflow.events': {
|
|
1101
|
+
durable: true,
|
|
1102
|
+
arguments: {
|
|
1103
|
+
'x-message-ttl': 300000, // Stated by confirmation 007, like monitoring.workflow
|
|
1104
|
+
'x-max-length': 10000,
|
|
1105
|
+
'x-dead-letter-exchange': '', // Default exchange: routing key IS the queue name
|
|
1106
|
+
'x-dead-letter-routing-key': 'scheduler.dlq'
|
|
1107
|
+
}
|
|
1108
|
+
},
|
|
1109
|
+
|
|
1110
|
+
/**
|
|
1111
|
+
* scheduler.dlq - the END of this family's chain.
|
|
1112
|
+
*
|
|
1113
|
+
* Durable, capped higher, no TTL and no onward route — the three properties every
|
|
1114
|
+
* dead-letter queue in this file carries, and the reason the operator's `*.dlq`
|
|
1115
|
+
* dashboard finds it (`mq-consumer-contract` 003).
|
|
1116
|
+
*/
|
|
1117
|
+
dlq: {
|
|
1118
|
+
durable: true,
|
|
1119
|
+
arguments: {
|
|
1120
|
+
// No TTL for DLQ - messages should persist
|
|
1121
|
+
'x-max-length': 50000 // Higher limit for DLQ
|
|
1122
|
+
}
|
|
1123
|
+
}
|
|
1124
|
+
},
|
|
1125
|
+
|
|
944
1126
|
/**
|
|
945
1127
|
* The full wire name of a declared entry: the section's prefix + the entry's key.
|
|
946
1128
|
*
|
|
@@ -1219,6 +1401,23 @@ module.exports = {
|
|
|
1219
1401
|
return this.getQueueConfig('infrastructure', name);
|
|
1220
1402
|
},
|
|
1221
1403
|
|
|
1404
|
+
/**
|
|
1405
|
+
* Get scheduler queue configuration
|
|
1406
|
+
* @param {string} queueName - Queue name (e.g., 'scheduler.request')
|
|
1407
|
+
* @returns {Object} Queue configuration
|
|
1408
|
+
* @throws {ValidationError} If the name carries another prefix, or nothing declares it.
|
|
1409
|
+
*/
|
|
1410
|
+
getSchedulerQueueConfig(queueName) {
|
|
1411
|
+
if (!queueName.startsWith('scheduler.')) {
|
|
1412
|
+
throw new ValidationError(
|
|
1413
|
+
`[queueConfig] Queue ${queueName} is not a scheduler queue - Expected: a name starting with "scheduler.". `
|
|
1414
|
+
+ 'Fix: call getInfrastructureQueueConfig(), which routes each prefix to the right lookup.'
|
|
1415
|
+
);
|
|
1416
|
+
}
|
|
1417
|
+
const name = queueName.slice('scheduler.'.length);
|
|
1418
|
+
return this.getQueueConfig('scheduler', name);
|
|
1419
|
+
},
|
|
1420
|
+
|
|
1222
1421
|
/**
|
|
1223
1422
|
* Get monitoring queue configuration
|
|
1224
1423
|
* @param {string} queueName - Queue name (e.g., 'monitoring.workflow')
|
package/src/index.js
CHANGED
|
@@ -36,6 +36,10 @@ const {
|
|
|
36
36
|
publishToMonitoringServices,
|
|
37
37
|
isQueueUnavailableError,
|
|
38
38
|
} = require('./monitoring-publish');
|
|
39
|
+
const {
|
|
40
|
+
buildWorkflowInitiatedEvent,
|
|
41
|
+
publishWorkflowInitiated,
|
|
42
|
+
} = require('./monitoring-initiated');
|
|
39
43
|
const queueConfig = require('./config/queueConfig');
|
|
40
44
|
const deliveryPolicy = require('./config/deliveryPolicy');
|
|
41
45
|
const { redactUrl, UNPARSEABLE_PLACEHOLDER } = require('@onlineapps/logger-contract');
|
|
@@ -84,6 +88,11 @@ module.exports.monitoring = {
|
|
|
84
88
|
publishToMonitoringWorkflow,
|
|
85
89
|
publishToMonitoringServices,
|
|
86
90
|
isQueueUnavailableError,
|
|
91
|
+
// d.819 - the one builder of `workflow_initiated`, beside the transport half it uses
|
|
92
|
+
// (`docs/governance/confirmations/scheduler.md` 009). The gateway and the scheduler each
|
|
93
|
+
// built that event in their own words; two rails over one event is a defect by default.
|
|
94
|
+
buildWorkflowInitiatedEvent,
|
|
95
|
+
publishWorkflowInitiated,
|
|
87
96
|
};
|
|
88
97
|
/** The only public way to reach the central queue classification; `src/config/queueConfig` is an internal path, not a contract. */
|
|
89
98
|
module.exports.queueConfig = queueConfig;
|
|
@@ -109,6 +118,18 @@ module.exports.topology = Object.freeze({
|
|
|
109
118
|
monitoringInfrastructureHealthEventsQueue: queueConfig.queueName('monitoring', 'infrastructure.health.events'),
|
|
110
119
|
/** The delivery endpoint's queue bound to that exchange, for WS push. */
|
|
111
120
|
deliveryHealthEventsQueue: queueConfig.queueName('deliveryEvents', 'health.events'),
|
|
121
|
+
/**
|
|
122
|
+
* Topic exchange: the monitoring consumer's resource-changed watcher publishes here,
|
|
123
|
+
* with a routing key of the shape `resource.changed.<resource_type>`.
|
|
124
|
+
*
|
|
125
|
+
* Declared in `queueConfig` since d.619. Until then the name was a module constant on
|
|
126
|
+
* the publishing side and a required env key on the binding side
|
|
127
|
+
* (`MONITORING_RESOURCE_EXCHANGE`) — the third side of this wire with no owner, while
|
|
128
|
+
* the queue bound to it had one here since d.270.
|
|
129
|
+
*
|
|
130
|
+
* @see api/infra/api_monitoring/src/consumer/watchers/resourceChangedPublisher.js
|
|
131
|
+
*/
|
|
132
|
+
monitoringResourceExchange: queueConfig.queueName('monitoring', 'resource'),
|
|
112
133
|
/**
|
|
113
134
|
* The delivery endpoint's queue for resource-changed events, bound to the
|
|
114
135
|
* `monitoring.resource` exchange by the routing pattern `resource.changed.#`.
|
|
@@ -0,0 +1,187 @@
|
|
|
1
|
+
'use strict';
|
|
2
|
+
|
|
3
|
+
/**
|
|
4
|
+
* The `workflow_initiated` event — ONE rail for every service that starts a run.
|
|
5
|
+
*
|
|
6
|
+
* Monitoring learns that a run exists from this event and from nothing else: its intake
|
|
7
|
+
* reads the run's tenancy, labels and trigger out of `workflow_initiated` and marks the
|
|
8
|
+
* run `active` on it (`api/infra/api_monitoring/src/consumer/index.js`). A service that
|
|
9
|
+
* does not file it starts runs that are invisible for their whole life.
|
|
10
|
+
*
|
|
11
|
+
* Until now two services built that event, each in its own words: the gateway
|
|
12
|
+
* (`infra/api_gateway/shared/monitoringInitiatedCopy.js`, with a redacted cookbook) and
|
|
13
|
+
* the scheduler (`infra/api_scheduler/src/publisher/workflowPublisher.js`, inline, the
|
|
14
|
+
* cookbook verbatim and no `tenant_uuid` at all). Two implementations of one concern are
|
|
15
|
+
* a defect by default (`change-discipline.md` § One rail per concern), and the owner
|
|
16
|
+
* settled which rail survives: this one, in the library that already owns the transport
|
|
17
|
+
* half of the same wire (`docs/governance/confirmations/scheduler.md` 009). Both copies
|
|
18
|
+
* are retired by their own services in the change that adopts this module — no period
|
|
19
|
+
* with both (`architecture-principles.md` §11).
|
|
20
|
+
*
|
|
21
|
+
* WHAT THIS MODULE DOES NOT DO, deliberately:
|
|
22
|
+
*
|
|
23
|
+
* 1. **It does not redact.** What is sensitive in a cookbook is known only to the
|
|
24
|
+
* service spec in the registry, and reading that spec would give this library a
|
|
25
|
+
* dependency on the operation specs and on `infrastructure-tools`. The caller passes
|
|
26
|
+
* the observability copy it already built (`cookbookRedaction.js` in the gateway); a
|
|
27
|
+
* path that cannot redact does not file the event at all, because an empty set of
|
|
28
|
+
* sensitive fields is silent consent to an unredacted copy.
|
|
29
|
+
* 2. **It invents no value.** `service_name` and the instant are the caller's, and every
|
|
30
|
+
* key it reads out of `_system` must EXIST — `hasOwnProperty`, not truthiness, so
|
|
31
|
+
* `tenant_uuid: null` passes through as the explicit value it is (a webhook run has
|
|
32
|
+
* no personal identity, and a scheduled run of the pilot writes `null` on purpose —
|
|
33
|
+
* `docs/governance/confirmations/run-envelope-tenant-uuid.md` 001). A `|| null` here
|
|
34
|
+
* would hide a broken envelope instead of reporting it (`architecture-principles.md`
|
|
35
|
+
* §3).
|
|
36
|
+
*
|
|
37
|
+
* @see api/docs/governance/confirmations/scheduler.md 009
|
|
38
|
+
* @see api/docs/biz/70-contracts/monitoring-events.md
|
|
39
|
+
* @see ./monitoring-publish.js — the transport half: the queue, the fanout, the fail-safe
|
|
40
|
+
*/
|
|
41
|
+
|
|
42
|
+
const { assertLogger } = require('@onlineapps/logger-contract');
|
|
43
|
+
const { publishToMonitoringWorkflow } = require('./monitoring-publish');
|
|
44
|
+
|
|
45
|
+
/** Starting a run is not a step of the cookbook; it has its own constant coordinates. */
|
|
46
|
+
const STEP_ID = 'workflow_start';
|
|
47
|
+
const STEP_INDEX = -1;
|
|
48
|
+
/** The event's own two words, which the monitoring intake matches on. */
|
|
49
|
+
const EVENT_TYPE = 'initiated';
|
|
50
|
+
const STATUS = 'initiated';
|
|
51
|
+
|
|
52
|
+
/** The keys of `_system` this event carries, each of which must exist on the envelope. */
|
|
53
|
+
const REQUIRED_SYSTEM_KEYS = Object.freeze(['tenant_id', 'tenant_uuid']);
|
|
54
|
+
|
|
55
|
+
/**
|
|
56
|
+
* The body of the `initiated` event — exactly these keys and no others.
|
|
57
|
+
*
|
|
58
|
+
* @param {Object} args
|
|
59
|
+
* @param {string} args.workflowId - the id of the run this event reports.
|
|
60
|
+
* @param {Object} args.redactedCookbook - the observability copy of the unified cookbook,
|
|
61
|
+
* ALREADY redacted by the caller; it travels as it arrived.
|
|
62
|
+
* @param {string} args.serviceName - the platform name of the service filing the event;
|
|
63
|
+
* the monitoring intake refuses an event that names no publisher (`service_name_missing`).
|
|
64
|
+
* @param {string} args.timestamp - the instant the run started, ISO-8601. The caller's,
|
|
65
|
+
* not a second reading of the clock here: the run and the row observing it then say the
|
|
66
|
+
* same thing.
|
|
67
|
+
* @returns {Object} the message for `monitoring.workflow`.
|
|
68
|
+
* @throws {Error} when an argument is missing, or the envelope does not carry what this
|
|
69
|
+
* event states about it.
|
|
70
|
+
*/
|
|
71
|
+
function buildWorkflowInitiatedEvent({ workflowId, redactedCookbook, serviceName, timestamp } = {}) {
|
|
72
|
+
if (typeof workflowId !== 'string' || workflowId.trim() === '') {
|
|
73
|
+
throw new Error(
|
|
74
|
+
'[monitoring-initiated] Missing workflow_id - Expected: the id of the run this event '
|
|
75
|
+
+ 'reports, as a non-empty string. Fix: pass workflowId of the run being published.'
|
|
76
|
+
);
|
|
77
|
+
}
|
|
78
|
+
if (typeof serviceName !== 'string' || serviceName.trim() === '') {
|
|
79
|
+
throw new Error(
|
|
80
|
+
'[monitoring-initiated] Missing service_name - Expected: the platform name of the service '
|
|
81
|
+
+ 'filing this event; the monitoring intake refuses an event that names no publisher '
|
|
82
|
+
+ '(service_name_missing). Fix: pass config.serviceName.'
|
|
83
|
+
);
|
|
84
|
+
}
|
|
85
|
+
if (typeof timestamp !== 'string' || timestamp.trim() === '') {
|
|
86
|
+
throw new Error(
|
|
87
|
+
'[monitoring-initiated] Missing timestamp - Expected: the instant the run started, as an '
|
|
88
|
+
+ 'ISO-8601 string taken from the run itself, so the run and the row observing it say the '
|
|
89
|
+
+ 'same thing. Fix: pass the envelope\'s own instant, not a new Date() read here.'
|
|
90
|
+
);
|
|
91
|
+
}
|
|
92
|
+
if (!redactedCookbook || typeof redactedCookbook !== 'object') {
|
|
93
|
+
throw new Error(
|
|
94
|
+
'[monitoring-initiated] Missing cookbook - Expected: the redacted observability copy of the '
|
|
95
|
+
+ 'unified cookbook. Fix: pass the copy the caller built with its own redaction '
|
|
96
|
+
+ '(a path that cannot redact does not file this event at all).'
|
|
97
|
+
);
|
|
98
|
+
}
|
|
99
|
+
|
|
100
|
+
const system = redactedCookbook._system;
|
|
101
|
+
if (!system || typeof system !== 'object') {
|
|
102
|
+
throw new Error(
|
|
103
|
+
'[monitoring-initiated] Cookbook carries no _system - Expected: the unified cookbook with '
|
|
104
|
+
+ 'its system metadata, which is where the intake reads the run\'s tenancy, labels and '
|
|
105
|
+
+ 'trigger. Fix: build the copy from the unified cookbook the run was published with.'
|
|
106
|
+
);
|
|
107
|
+
}
|
|
108
|
+
|
|
109
|
+
for (const key of REQUIRED_SYSTEM_KEYS) {
|
|
110
|
+
if (!Object.prototype.hasOwnProperty.call(system, key)) {
|
|
111
|
+
throw new Error(
|
|
112
|
+
`[monitoring-initiated] Cookbook _system carries no ${key} - Expected: every entry path `
|
|
113
|
+
+ 'writes it explicitly (a value, or null where the path has none - a webhook run proves '
|
|
114
|
+
+ 'itself by signature, a scheduled run of the pilot has no personal identity). '
|
|
115
|
+
+ `Fix: set _system.${key} where the unified cookbook is built.`
|
|
116
|
+
);
|
|
117
|
+
}
|
|
118
|
+
}
|
|
119
|
+
|
|
120
|
+
return {
|
|
121
|
+
event_type: EVENT_TYPE,
|
|
122
|
+
workflow_id: workflowId,
|
|
123
|
+
service_name: serviceName,
|
|
124
|
+
step_index: STEP_INDEX,
|
|
125
|
+
step_id: STEP_ID,
|
|
126
|
+
cookbook: redactedCookbook,
|
|
127
|
+
status: STATUS,
|
|
128
|
+
tenant_uuid: system.tenant_uuid,
|
|
129
|
+
tenant_id: system.tenant_id,
|
|
130
|
+
timestamp
|
|
131
|
+
};
|
|
132
|
+
}
|
|
133
|
+
|
|
134
|
+
/**
|
|
135
|
+
* File the `initiated` event. NEVER throws — a lost copy is reported, not raised.
|
|
136
|
+
*
|
|
137
|
+
* It is observability about a run that has already happened: refusing the run because the
|
|
138
|
+
* observer could not be told would trade a real effect for a record of it. Both services
|
|
139
|
+
* that build this event today made the same decision in their own words, and it moves here
|
|
140
|
+
* with the event. A missing logger is the one exception, and it is the transport half's
|
|
141
|
+
* rule (`monitoring-publish.js`): without a logger the promise "every failure is reported"
|
|
142
|
+
* cannot hold, so the call refuses instead of dropping events in silence.
|
|
143
|
+
*
|
|
144
|
+
* @param {Object} args
|
|
145
|
+
* @param {Object} args.mqClient - the connected `BaseClient`.
|
|
146
|
+
* @param {Object} args.logger - the service logger; REQUIRED.
|
|
147
|
+
* @param {string} args.workflowId
|
|
148
|
+
* @param {Object} args.redactedCookbook
|
|
149
|
+
* @param {string} args.serviceName
|
|
150
|
+
* @param {string} args.timestamp
|
|
151
|
+
* @param {Object} [args.context] - extra context for the log lines.
|
|
152
|
+
* @returns {Promise<boolean>} whether the event reached the broker.
|
|
153
|
+
*/
|
|
154
|
+
async function publishWorkflowInitiated({
|
|
155
|
+
mqClient,
|
|
156
|
+
logger,
|
|
157
|
+
workflowId,
|
|
158
|
+
redactedCookbook,
|
|
159
|
+
serviceName,
|
|
160
|
+
timestamp,
|
|
161
|
+
context = {}
|
|
162
|
+
} = {}) {
|
|
163
|
+
assertLogger(
|
|
164
|
+
'publishWorkflowInitiated',
|
|
165
|
+
logger,
|
|
166
|
+
'a copy monitoring never received is reported instead of silently returning false',
|
|
167
|
+
'pass the logger your service already built'
|
|
168
|
+
);
|
|
169
|
+
|
|
170
|
+
try {
|
|
171
|
+
const event = buildWorkflowInitiatedEvent({ workflowId, redactedCookbook, serviceName, timestamp });
|
|
172
|
+
return await publishToMonitoringWorkflow(mqClient, event, logger, { workflowId, ...context });
|
|
173
|
+
} catch (error) {
|
|
174
|
+
logger.warn('[monitoring-initiated] Failed to publish workflow_initiated to monitoring', {
|
|
175
|
+
...context,
|
|
176
|
+
workflow_id: workflowId,
|
|
177
|
+
service_name: serviceName,
|
|
178
|
+
error: error.message
|
|
179
|
+
});
|
|
180
|
+
return false;
|
|
181
|
+
}
|
|
182
|
+
}
|
|
183
|
+
|
|
184
|
+
module.exports = {
|
|
185
|
+
buildWorkflowInitiatedEvent,
|
|
186
|
+
publishWorkflowInitiated
|
|
187
|
+
};
|
|
@@ -525,13 +525,52 @@ class RabbitMQClient extends EventEmitter {
|
|
|
525
525
|
}
|
|
526
526
|
|
|
527
527
|
/**
|
|
528
|
-
*
|
|
529
|
-
*
|
|
528
|
+
* THE definition of "this client's consumer is attached to that queue".
|
|
529
|
+
*
|
|
530
|
+
* The tag is the one fact that is about THIS client: the broker gives it to
|
|
531
|
+
* the attachment it created, and since d.386b / d.605 the entry carries it
|
|
532
|
+
* only while that attachment exists — a refused re-registration and a cancel
|
|
533
|
+
* sent by the broker both drop it while the consumer stays TRACKED, because
|
|
534
|
+
* the next recovery must find it on the list.
|
|
535
|
+
*
|
|
536
|
+
* It is a method rather than an expression repeated twice for the reason
|
|
537
|
+
* `change-discipline.md` § One rail per concern states: until d.613
|
|
538
|
+
* `getConsumerState()` answered from `_activeConsumers.keys()` and
|
|
539
|
+
* `_performHealthCheck()` from the tag, so one file held two definitions of
|
|
540
|
+
* "active" — and the cheap synchronous reader, the one a service and an
|
|
541
|
+
* operator actually read, was the one that answered about a consumer the
|
|
542
|
+
* broker had already cancelled.
|
|
543
|
+
*
|
|
544
|
+
* @param {Object|undefined} consumerInfo - The entry `_activeConsumers` holds
|
|
545
|
+
* @returns {boolean}
|
|
546
|
+
* @private
|
|
547
|
+
*/
|
|
548
|
+
_isConsumerAttached(consumerInfo) {
|
|
549
|
+
return typeof consumerInfo?.consumerTag === 'string' && consumerInfo.consumerTag !== '';
|
|
550
|
+
}
|
|
551
|
+
|
|
552
|
+
/**
|
|
553
|
+
* Get current consumer state.
|
|
554
|
+
*
|
|
555
|
+
* `tracked` counts the consumers this client would restore; `active` names the
|
|
556
|
+
* ones that are ATTACHED right now (`_isConsumerAttached()`, the same question
|
|
557
|
+
* `_performHealthCheck()` asks). `tracked > active.length` is therefore the
|
|
558
|
+
* cheap, synchronous statement that this client takes no message off some
|
|
559
|
+
* queue it registered — the state
|
|
560
|
+
* `docs/operations/rabbitmq-connection-recovery-troubleshooting.md` sends an
|
|
561
|
+
* operator here to read, and which the queue-name list could not carry.
|
|
562
|
+
*
|
|
563
|
+
* @returns {{tracked: number, active: string[]}} Consumer state information
|
|
530
564
|
*/
|
|
531
565
|
getConsumerState() {
|
|
566
|
+
const active = [];
|
|
567
|
+
for (const [queue, consumerInfo] of this._activeConsumers.entries()) {
|
|
568
|
+
if (this._isConsumerAttached(consumerInfo)) active.push(queue);
|
|
569
|
+
}
|
|
570
|
+
|
|
532
571
|
return {
|
|
533
572
|
tracked: this._activeConsumers.size,
|
|
534
|
-
active
|
|
573
|
+
active
|
|
535
574
|
};
|
|
536
575
|
}
|
|
537
576
|
|
|
@@ -1858,54 +1897,160 @@ class RabbitMQClient extends EventEmitter {
|
|
|
1858
1897
|
if (this._activeConsumers.size > 0) {
|
|
1859
1898
|
this._logger.info(`[RabbitMQClient] [mq-client-core] Re-registering ${this._activeConsumers.size} consumers...`);
|
|
1860
1899
|
for (const [queue, consumerInfo] of this._activeConsumers.entries()) {
|
|
1861
|
-
|
|
1862
|
-
await this._prepareQueueForConsume(queue, {
|
|
1863
|
-
durable: consumerInfo.options.durable,
|
|
1864
|
-
queueOptions: consumerInfo.options.queueOptions
|
|
1865
|
-
});
|
|
1866
|
-
|
|
1867
|
-
// The consumer's prefetch is re-applied here for the same reason the
|
|
1868
|
-
// queue step is: the channel is new, and QoS lives on the channel.
|
|
1869
|
-
await this._applyConsumerPrefetch(queue, consumerInfo.options.prefetch);
|
|
1870
|
-
|
|
1871
|
-
const consumeResult = await this._consumerChannel.consume(
|
|
1872
|
-
queue,
|
|
1873
|
-
consumerInfo.handler,
|
|
1874
|
-
{ noAck: consumerInfo.options.noAck }
|
|
1875
|
-
);
|
|
1876
|
-
consumerInfo.consumerTag = consumeResult.consumerTag;
|
|
1900
|
+
if (await this._reRegisterConsumer(queue, consumerInfo)) {
|
|
1877
1901
|
reRegistered++;
|
|
1878
|
-
|
|
1879
|
-
} catch (err) {
|
|
1902
|
+
} else {
|
|
1880
1903
|
failed++;
|
|
1881
|
-
this._logger.error(`[RabbitMQClient] [mq-client-core] Failed to re-register consumer for queue ${queue}:`, err.message);
|
|
1882
|
-
// The consumer STAYS tracked. Whatever refused it — a 406 against the
|
|
1883
|
-
// owner's declaration, a missing infrastructure queue, a channel that
|
|
1884
|
-
// went away mid-flight — is a condition somebody fixes at its source,
|
|
1885
|
-
// and the next recreate must find the consumer still on the list.
|
|
1886
|
-
//
|
|
1887
|
-
// Its TAG does not stay, and the two are not the same fact (d.386b). A
|
|
1888
|
-
// consumer tag names an attachment on ONE channel; the channel that
|
|
1889
|
-
// issued this one is gone and the new one refused to re-attach, so the
|
|
1890
|
-
// string the entry carried names nothing on the broker. Two readers
|
|
1891
|
-
// took it for an attachment that exists: `_performHealthCheck()` counts
|
|
1892
|
-
// `active` by "is the tag set", so a client whose only consumer had
|
|
1893
|
-
// just been refused reported `healthy: true` and `1/1 consumers active`
|
|
1894
|
-
// — the false OK d.386 exists to end, one layer below where d.386 could
|
|
1895
|
-
// reach it; and `cancelConsumer()` sent `basic.cancel` with it, asking
|
|
1896
|
-
// the broker to stop a consumer it does not hold.
|
|
1897
|
-
consumerInfo.consumerTag = null;
|
|
1898
|
-
// The broker's own code is what a listener acts on (406 = the queue
|
|
1899
|
-
// exists with different arguments), and the rail wraps it in a typed
|
|
1900
|
-
// error, so it is read from the cause when the wrapper carries none.
|
|
1901
|
-
const code = err.code !== undefined ? err.code : (err.cause ? err.cause.code : undefined);
|
|
1902
|
-
this.emit('consumer:re-registration:failed', { queue, error: err.message, code });
|
|
1903
1904
|
}
|
|
1904
1905
|
}
|
|
1905
1906
|
}
|
|
1906
1907
|
return { reRegistered, failed };
|
|
1907
1908
|
}
|
|
1908
1909
|
|
|
1910
|
+
/**
|
|
1911
|
+
* Re-attach ONE tracked consumer on the current consumer channel.
|
|
1912
|
+
*
|
|
1913
|
+
* This is the body of the loop above, given a name in d.605 because a second
|
|
1914
|
+
* event needs exactly it: `basic.cancel` from the broker takes ONE consumer
|
|
1915
|
+
* away while every other one stays attached, so re-running the whole loop
|
|
1916
|
+
* would re-attach consumers that never lost their place — a second
|
|
1917
|
+
* `basic.consume` for each of them on the broker. One rail, one set of
|
|
1918
|
+
* consequences (`change-discipline.md` § One rail per concern).
|
|
1919
|
+
*
|
|
1920
|
+
* Everything a first `consume()` does is done here: the queue is classified,
|
|
1921
|
+
* declared or checked by `_prepareQueueForConsume()` — so an owned business
|
|
1922
|
+
* queue is re-declared and an unowned infrastructure queue is only checked,
|
|
1923
|
+
* its absence answered with `CONSUMER_QUEUE_MISSING` rather than a silent
|
|
1924
|
+
* creation — and the prefetch is re-applied, because QoS lives on the channel.
|
|
1925
|
+
*
|
|
1926
|
+
* @param {string} queue - Queue name
|
|
1927
|
+
* @param {Object} consumerInfo - The entry `_activeConsumers` holds for it
|
|
1928
|
+
* @returns {Promise<boolean>} `true` when the consumer is attached again
|
|
1929
|
+
* @private
|
|
1930
|
+
*/
|
|
1931
|
+
async _reRegisterConsumer(queue, consumerInfo) {
|
|
1932
|
+
try {
|
|
1933
|
+
await this._prepareQueueForConsume(queue, {
|
|
1934
|
+
durable: consumerInfo.options.durable,
|
|
1935
|
+
queueOptions: consumerInfo.options.queueOptions
|
|
1936
|
+
});
|
|
1937
|
+
|
|
1938
|
+
// The consumer's prefetch is re-applied here for the same reason the
|
|
1939
|
+
// queue step is: the channel is new, and QoS lives on the channel.
|
|
1940
|
+
await this._applyConsumerPrefetch(queue, consumerInfo.options.prefetch);
|
|
1941
|
+
|
|
1942
|
+
const consumeResult = await this._consumerChannel.consume(
|
|
1943
|
+
queue,
|
|
1944
|
+
consumerInfo.handler,
|
|
1945
|
+
{ noAck: consumerInfo.options.noAck }
|
|
1946
|
+
);
|
|
1947
|
+
consumerInfo.consumerTag = consumeResult.consumerTag;
|
|
1948
|
+
this._logger.info(`[RabbitMQClient] [mq-client-core] ✓ Re-registered consumer for queue: ${queue} (consumerTag: ${consumeResult.consumerTag})`);
|
|
1949
|
+
return true;
|
|
1950
|
+
} catch (err) {
|
|
1951
|
+
this._logger.error(`[RabbitMQClient] [mq-client-core] Failed to re-register consumer for queue ${queue}:`, err.message);
|
|
1952
|
+
// The consumer STAYS tracked. Whatever refused it — a 406 against the
|
|
1953
|
+
// owner's declaration, a missing infrastructure queue, a channel that
|
|
1954
|
+
// went away mid-flight — is a condition somebody fixes at its source,
|
|
1955
|
+
// and the next recreate must find the consumer still on the list.
|
|
1956
|
+
//
|
|
1957
|
+
// Its TAG does not stay, and the two are not the same fact (d.386b). A
|
|
1958
|
+
// consumer tag names an attachment on ONE channel; the channel that
|
|
1959
|
+
// issued this one is gone and the new one refused to re-attach, so the
|
|
1960
|
+
// string the entry carried names nothing on the broker. Two readers
|
|
1961
|
+
// took it for an attachment that exists: `_performHealthCheck()` counts
|
|
1962
|
+
// `active` by "is the tag set", so a client whose only consumer had
|
|
1963
|
+
// just been refused reported `healthy: true` and `1/1 consumers active`
|
|
1964
|
+
// — the false OK d.386 exists to end, one layer below where d.386 could
|
|
1965
|
+
// reach it; and `cancelConsumer()` sent `basic.cancel` with it, asking
|
|
1966
|
+
// the broker to stop a consumer it does not hold.
|
|
1967
|
+
consumerInfo.consumerTag = null;
|
|
1968
|
+
// The broker's own code is what a listener acts on (406 = the queue
|
|
1969
|
+
// exists with different arguments), and the rail wraps it in a typed
|
|
1970
|
+
// error, so it is read from the cause when the wrapper carries none.
|
|
1971
|
+
const code = err.code !== undefined ? err.code : (err.cause ? err.cause.code : undefined);
|
|
1972
|
+
this.emit('consumer:re-registration:failed', { queue, error: err.message, code });
|
|
1973
|
+
return false;
|
|
1974
|
+
}
|
|
1975
|
+
}
|
|
1976
|
+
|
|
1977
|
+
/**
|
|
1978
|
+
* The broker cancelled one of this client's consumers.
|
|
1979
|
+
*
|
|
1980
|
+
* `basic.cancel` sent BY THE BROKER is not an error frame and does not touch
|
|
1981
|
+
* the channel or the connection: the queue was deleted, or the broker moved it
|
|
1982
|
+
* (`x-cancel-on-ha-failover`). Nothing else in this client hears it — the
|
|
1983
|
+
* channel handlers listen for `error` and `close`, and both stay silent — so
|
|
1984
|
+
* this method is the whole of the client's answer, and until d.605 that answer
|
|
1985
|
+
* was nothing at all.
|
|
1986
|
+
*
|
|
1987
|
+
* Three things happen, in this order:
|
|
1988
|
+
*
|
|
1989
|
+
* 1. the TAG is dropped, first, because it is the one fact
|
|
1990
|
+
* `_performHealthCheck()` reads to answer "is THIS client's consumer
|
|
1991
|
+
* attached" (d.386b). Dropping it before anything else leaves no window in
|
|
1992
|
+
* which the check reports an attachment the broker has already cancelled;
|
|
1993
|
+
* 2. the loss is stated at ERROR level, naming the queue and the cancelled
|
|
1994
|
+
* tag. A consumer that stops reading a queue is not a debug-level event,
|
|
1995
|
+
* and the record is what tells an operator which queue went quiet;
|
|
1996
|
+
* 3. the consumer is re-attached through `_reRegisterConsumer()`, the rail a
|
|
1997
|
+
* channel recreate already uses. A refusal there keeps the consumer
|
|
1998
|
+
* tracked with no tag and travels on `consumer:re-registration:failed` —
|
|
1999
|
+
* which is why this method introduces no event and no public method of its
|
|
2000
|
+
* own: every state it can reach is a state the package already reports.
|
|
2001
|
+
*
|
|
2002
|
+
* @param {string} queue - The queue whose consumer the broker cancelled
|
|
2003
|
+
* @returns {Promise<void>}
|
|
2004
|
+
* @private
|
|
2005
|
+
*/
|
|
2006
|
+
async _onBrokerCancel(queue) {
|
|
2007
|
+
if (this._closedByCaller) {
|
|
2008
|
+
// An intentional close is not a fault — the doctrine `disconnect()` states
|
|
2009
|
+
// and both channel handlers already follow.
|
|
2010
|
+
this._logger.debug(
|
|
2011
|
+
`[RabbitMQClient] [mq-client-core] [CONSUMER] Broker cancelled the consumer for queue "${queue}" while the client was being closed by its caller - no alarm, no re-attachment`
|
|
2012
|
+
);
|
|
2013
|
+
return;
|
|
2014
|
+
}
|
|
2015
|
+
|
|
2016
|
+
const consumerInfo = this._activeConsumers.get(queue);
|
|
2017
|
+
const cancelledTag = consumerInfo ? consumerInfo.consumerTag : null;
|
|
2018
|
+
|
|
2019
|
+
if (consumerInfo) {
|
|
2020
|
+
consumerInfo.consumerTag = null;
|
|
2021
|
+
}
|
|
2022
|
+
|
|
2023
|
+
this._logger.error(
|
|
2024
|
+
`[RabbitMQClient] Consumer cancelled by the broker - the consumer on queue "${queue}" `
|
|
2025
|
+
+ `(consumerTag: ${cancelledTag}) was stopped by the BROKER, not by this client: the queue `
|
|
2026
|
+
+ 'was deleted, or the broker moved it (x-cancel-on-ha-failover). '
|
|
2027
|
+
+ 'Expected: a consumer stays attached until cancelConsumer() or disconnect(). '
|
|
2028
|
+
+ 'Fix: none here - the client re-attaches it on the rail a channel recreate uses; if that '
|
|
2029
|
+
+ 'is refused, the consumer stays tracked with no tag, performHealthCheck() reports this '
|
|
2030
|
+
+ 'client unhealthy and names the queue, and consumer:re-registration:failed carries the '
|
|
2031
|
+
+ 'broker reason.',
|
|
2032
|
+
{ queue, consumerTag: cancelledTag }
|
|
2033
|
+
);
|
|
2034
|
+
|
|
2035
|
+
if (!consumerInfo) {
|
|
2036
|
+
// The client no longer holds this consumer, so there is nothing to
|
|
2037
|
+
// re-attach and no state left to correct. The loss is still stated above.
|
|
2038
|
+
return;
|
|
2039
|
+
}
|
|
2040
|
+
|
|
2041
|
+
if (!this._isChannelAlive(this._consumerChannel)) {
|
|
2042
|
+
// Re-attaching on a dead channel is not this method's job: the channel
|
|
2043
|
+
// recovery re-registers every tracked consumer when it recreates the
|
|
2044
|
+
// channel (d.262), and the consumer stays tracked for exactly that.
|
|
2045
|
+
this._logger.warn(
|
|
2046
|
+
`[RabbitMQClient] [mq-client-core] [CONSUMER] Consumer for queue "${queue}" stays tracked and unattached - the consumer channel is not alive, so the channel recovery re-registers it`
|
|
2047
|
+
);
|
|
2048
|
+
return;
|
|
2049
|
+
}
|
|
2050
|
+
|
|
2051
|
+
await this._reRegisterConsumer(queue, consumerInfo);
|
|
2052
|
+
}
|
|
2053
|
+
|
|
1909
2054
|
/**
|
|
1910
2055
|
* Disconnects: closes channel and connection.
|
|
1911
2056
|
* @returns {Promise<void>}
|
|
@@ -2061,8 +2206,10 @@ class RabbitMQClient extends EventEmitter {
|
|
|
2061
2206
|
// this client's and everybody else's alike, so it cannot say whether
|
|
2062
2207
|
// the one asking is among them. The tag can: it is the name the
|
|
2063
2208
|
// broker gave THIS client's attachment, and since d.386b it is set
|
|
2064
|
-
// only while that attachment exists.
|
|
2065
|
-
|
|
2209
|
+
// only while that attachment exists. The question is asked through
|
|
2210
|
+
// `_isConsumerAttached()` — the one definition `getConsumerState()`
|
|
2211
|
+
// now reads too (d.613).
|
|
2212
|
+
if (this._isConsumerAttached(consumerInfo)) {
|
|
2066
2213
|
health.consumers.active++;
|
|
2067
2214
|
} else {
|
|
2068
2215
|
health.consumers.inactive.push(queue);
|
|
@@ -3290,9 +3437,11 @@ class RabbitMQClient extends EventEmitter {
|
|
|
3290
3437
|
// leaves exactly those without a route of their own, because a dead-letter
|
|
3291
3438
|
// route on a dead-letter queue is a loop or a chain with no end
|
|
3292
3439
|
// (`mq-consumer-contract` 002 point 2, 003 point 2). Until d.452 the gate
|
|
3293
|
-
// asked only the first question, so the operator's own queue
|
|
3294
|
-
//
|
|
3295
|
-
//
|
|
3440
|
+
// asked only the first question, so the operator's own queue could not be
|
|
3441
|
+
// consumed through this library at all. Who reads such a queue is that
|
|
3442
|
+
// service's own fact, not this file's —
|
|
3443
|
+
// @see api/infra/api_monitoring/src/consumer/deadLetterPeek.js
|
|
3444
|
+
// The second question is derived from the templates
|
|
3296
3445
|
// (`queueConfig.isDeadLetterTarget()`), never from a list of names, so it
|
|
3297
3446
|
// cannot fall behind them; every other queue with no route is refused
|
|
3298
3447
|
// exactly as before, with the same sentence.
|
|
@@ -3532,7 +3681,21 @@ class RabbitMQClient extends EventEmitter {
|
|
|
3532
3681
|
// Wrap handler to track message processing and prefetch utilization
|
|
3533
3682
|
const messageHandler = async (msg) => {
|
|
3534
3683
|
if (msg === null) {
|
|
3535
|
-
|
|
3684
|
+
// `null` is not a message. It is `basic.cancel` FROM THE BROKER,
|
|
3685
|
+
// handed to this callback by amqplib (`handleCancel()` →
|
|
3686
|
+
// `dispatchMessage(fields, null)`), and it is the only way a consumer
|
|
3687
|
+
// learns that its attachment is gone while the channel and the
|
|
3688
|
+
// connection stay up: the queue was deleted, or the broker moved it
|
|
3689
|
+
// (`x-cancel-on-ha-failover`).
|
|
3690
|
+
//
|
|
3691
|
+
// Until d.605 the answer was this bare `return`. Nothing was logged,
|
|
3692
|
+
// the tag stayed set — so `_performHealthCheck()` went on counting the
|
|
3693
|
+
// consumer as attached — and nothing re-attached it. The service
|
|
3694
|
+
// reported itself healthy while no message was taken off the queue,
|
|
3695
|
+
// which is the failure `automation-gates.md` §5 calls a defect of the
|
|
3696
|
+
// same severity as a wrong result.
|
|
3697
|
+
await this._onBrokerCancel(queue);
|
|
3698
|
+
return;
|
|
3536
3699
|
}
|
|
3537
3700
|
|
|
3538
3701
|
// IMPORTANT:
|