@blamejs/core 0.6.32 → 0.6.34
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +2 -0
- package/README.md +2 -2
- package/index.js +2 -0
- package/lib/cache.js +61 -0
- package/lib/db.js +9 -8
- package/lib/framework-schema.js +11 -10
- package/lib/log-stream-cloudwatch.js +64 -5
- package/lib/log-stream-syslog.js +297 -0
- package/lib/log-stream.js +12 -12
- package/lib/protocol-dispatcher.js +1 -1
- package/lib/pubsub-cluster.js +154 -0
- package/lib/pubsub-redis.js +160 -0
- package/lib/pubsub.js +357 -0
- package/lib/redis-client.js +46 -1
- package/lib/websocket-channels.js +131 -217
- package/package.json +1 -1
- package/sbom.cyclonedx.json +6 -6
package/CHANGELOG.md
CHANGED
|
@@ -8,6 +8,8 @@ upgrading across more than a few patches at a time.
|
|
|
8
8
|
|
|
9
9
|
## v0.6.x
|
|
10
10
|
|
|
11
|
+
- **0.6.34** (2026-05-02) — distributed pub/sub primitive. **`b.pubsub`** — single API for cross-node fan-out across the framework, with three backends. `local` dispatches in-process to registered handlers (zero coordination overhead for single-node deploys); `cluster` polls a shared `_blamejs_pubsub_messages` table at `pollIntervalMs` (default 100ms) and dispatches new rows past `lastSeenId` from other nodes (publishedBy=self filter prevents loopback); `redis` opens a SUBSCRIBE-mode connection on `lib/redis-client.js` (now with new push-message demultiplexing — server-pushed `["message", channel, payload]` and `["pmessage", pattern, channel, payload]` arrays route through `setOnPushMessage` instead of consuming a pending request slot, while SUBSCRIBE / UNSUBSCRIBE acks still flow through the normal command pipeline). Per-instance nonce stamped on outgoing redis payloads so the SUBSCRIBE socket recognizes its own publishes and skips the loopback (without it every same-instance publish would fire local handlers twice). Operator API: `ps.subscribe(channel, handler) → token`, `ps.subscribePattern(pattern, handler) → token` (glob-style on local + cluster, native PSUBSCRIBE on redis), `ps.unsubscribe(token)`, `await ps.publish(channel, payload) → { local, remote }`, `await ps.close()`. `topicPrefix` opt scopes channel names so independent pubsub instances sharing a backend (cache invalidation + websocket channels + custom) don't collide. Handler errors are caught + logged via the framework's boot logger; they never abort dispatch to other handlers on the same channel. **`lib/websocket-channels.js`** — replaces the inline cluster-poll-and-fan-out logic with `b.pubsub` consumption. The hub now owns one pubsub instance per primitive; cross-node delivery is `pubsub.subscribe` per channel the hub joins, with the hub's `_localDispatch` as the handler. Per-channel pubsub subscription is refcounted by local conn count (subscribe on first conn, unsubscribe on last). The `_blamejs_ws_messages` table is renamed to `_blamejs_pubsub_messages` (column `channel` → `topic`) reflecting the generalization; pre-v1, no compat shim — operators upgrading wipe the previous table (which carried only ephemeral fan-out rows with default 60s retention). **`b.cache.create({ invalidationPubsub })`** — passing a `b.pubsub.create()` instance auto-publishes on every successful `del` / `clear` / `invalidateTag`, and subscribes for the same events so other cache instances on other nodes (or processes sharing the pubsub backend) react locally — primarily useful for the memory backend so stale per-node entries don't survive a global tag wipe. Re-entrancy guard prevents inbound invalidation events from re-publishing (no fan-out loops). Tests: 29 layer-0 assertions covering local + cluster behavior, topicPrefix isolation, pattern subscribe, handler error isolation, post-close error path, end-to-end cache invalidationPubsub fan-out via local pubsub. 10 live integration assertions against the docker redis container covering single-instance round-trip, PSUBSCRIBE pattern, cross-instance fan-out, and cache invalidation through redis PUB/SUB. Smoke 6817 / wiki e2e 178 / per-primitive integration 15 files / wiki integration 32 / shellcheck clean / eslint clean.
|
|
12
|
+
- **0.6.33** (2026-05-02) — log-stream sinks parity sweep. **`b.logStream` syslog sink (RFC 5424)** — first-class `protocol: "syslog"` for `b.logStream.init({ sinks: { ... } })`; new module `lib/log-stream-syslog.js`. URL-driven transport selection: `udp://host:514` / `tcp://host:514` / `tls://host:6514`. UDP is best-effort one-datagram-per-record; TCP / TLS use RFC 6587 octet-counting framing (`<length> <message>`) so collectors that prefer it over the older non-transparent newline framing parse cleanly. TLS is TLS 1.3 minimum; operators with private CAs pass `ca` (PEM string or array) for trust pinning, `servername` for SNI override (auto-suppressed on IP literals per the v0.6.28 redis-client convention). Outgoing records are formatted with PRI = `(facility << 3) | severity`, default facility `local0` (16), severity mapped from the framework's level field (debug=7 / info=6 / warn=4 / error=3); operators override via `facility`, `appName` (default `blamejs`), `procId` (default `process.pid`), `hostname` (default `os.hostname()`), `structuredData` (default `-`). Meta is JSON-encoded into the MSG body so the structured payload survives the wire as a single token. TCP / TLS reconnect with exponential backoff (default 250ms→30s); records buffer during the down window with `bufferLimit`-bounded oldest-drop semantics, replay on reconnect. `close()` waits up to 3s for an in-flight (re)connect to drain the buffer before tearing down — without this the slower TLS handshake raced fire-and-forget shutdown emits and silently dropped records. Operator-supplied `onDrop({reason, batch, error})` surfaces every drop class (`overflow` / `udp-send-error` / `write-error` / `sink-closed`). Removed `syslog` from `DEFERRED_PROTOCOLS`. **`b.logStream` cloudwatch sink: `autoCreate`** — pass `{ autoCreate: true }` to have the framework issue `CreateLogGroup` + `CreateLogStream` on first emit so operators provisioning collectors via env vars / runbooks don't need a separate aws-cli step. Idempotent: AWS's `ResourceAlreadyExistsException` is treated as success on both calls. Hard failures (5xx, AccessDenied, etc.) drop the batch with `onDrop` reason `autocreate-failed`. Default remains `autoCreate: false` so the AWS posture (operator pre-creates via aws / CDK / Terraform) stays the recommended path; `autoCreate` exists for ephemeral / dynamic-stream deployments where pre-provisioning is impractical. **Test suite**: 19 new framework-level cloudwatch assertions cover autoCreate fires both Create calls in order before PutLogEvents, ResourceAlreadyExists doesn't abort the post-create PutLogEvents, hard 5xx during CreateLogGroup drops the batch via `onDrop`, autoCreate=false skips Create calls entirely. Live syslog integration covers UDP / TCP wire delivery against the docker syslog-ng container (`/var/log/blamejs-test.log`) plus an in-process `tls.createServer` receiver that asserts the on-the-wire RFC 6587 octet-counting framing + RFC 5424 PRI / timestamp / structured-data slot the framework emits over TLS. **Test infrastructure fix**: `docker/init/generate-certs.sh` was overwriting an existing CA when re-run with `.complete` removed, invalidating every previously-issued leaf cert; now reuses an existing `(ca.crt, ca.key)` pair across `.complete`-only resets and only generates a fresh CA when neither file is present. Smoke 6790 / wiki e2e 178 / per-primitive integration 14 files (log-stream up from 16→18 checks, cloudwatch suite up from 39→58 assertions) / wiki integration 32 / shellcheck clean / eslint clean.
|
|
11
13
|
- **0.6.32** (2026-05-02) — E1 from the v0.6.x scope plan. **`b.cache` Redis backend** — first-class `backend: "redis"` for `b.cache.create({...})`, no operator-supplied glue needed. New module `lib/cache-redis.js` consumes `lib/redis-client.js` directly; storage layout: `<namespace>:e:<key>` (STRING, JSON-encoded value, PEXPIREAT-bounded), `<namespace>:t:<tag>` (SET, cacheKeys carrying that tag — powers `invalidateTag` fan-out), `<namespace>:k:<key>:tags` (SET, tags this key carries — powers per-key tag cleanup on `del`/`set`-overwrite, expires alongside the entry). TTL is enforced by Redis itself (PEXPIREAT) so the framework's sweeper is a no-op for this backend; sliding TTL bumps the entry's expiry on every read when `slidingTtl: true` AND `ttlMs` is finite. New opts on `cache.create`: `redisUrl` (required when `backend: "redis"`), `redisPassword`, `redisUsername`, `redisTls`, `redisCa` (private-CA trust pinning per v0.6.28), `redisServername`, `redisConnectTimeoutMs`, `redisCommandTimeoutMs`, `redisMaxReconnectAttempts`. Concurrency: invalidateTag filters out ghost entries (key already PEXPIRE'd from the keyspace but lingering in a tag SET) by EXISTS-checking before del. Lazy connect — `cache.create({backend:"redis"})` stays sync-safe; first op opens the socket. Tests: 10 new live integration assertions on top of the existing memory + custom-backend coverage (set+get round-trip, has, del, complex JSON value round-trip, short-TTL Redis-side expiry, multi-tag invalidateTag fan-out + preservation of untagged entries, wrap() single-flight memoization through Redis). Smoke 6780 / wiki e2e 178 / per-primitive integration 14 files (cache up from 15→25 checks) / wiki integration green.
|
|
12
14
|
- **0.6.31** (2026-05-02) — lazy-deferral cleanup wave from v0.6.27. **A1: queue-redis priority ordering** — `b.queue.enqueue({ priority })` on the Redis backend now matches queue-local's `ORDER BY priority DESC, availableAt ASC, enqueuedAt ASC` semantics. The Redis ZSET stays scored by availableAt only (so ZRANGEBYSCORE 0..nowMs cleanly filters ready vs not-yet-ready jobs), but `LEASE_LUA` now over-fetches `maxRows*5` candidates by score, HMGETs priority + availableAt + enqueuedAt for each, sorts server-side in Lua by the same triple queue-local sorts on, and leases the top `maxRows`. Closes the v0.6.27 deferral that punted priority ordering as "ZSET-incompatible". **A2: queue-redis flow `dependsOn` cascade** — `b.queue.enqueue({ flowId, flowChildName, dependsOn })` on the Redis backend now releases dependent jobs when their parents complete, mirroring queue-local's `_maybeReleaseFlowChildren`. A per-flow `<prefix>:flow:<flowId>` Redis SET tracks every job in the flow; `complete()` walks the set, looks up siblings whose `dependsOn` includes the just-completed job (by id OR `flowChildName`) AND every other dep is also satisfied, and HSET-bumps their `availableAt` to now + ZADDs them into the ready zset. The flow set persists across `complete()` calls so a later sibling's deps check can still find earlier-done parents by name; `purge()` cleans the set on queue teardown. Closes the v0.6.27 deferral. **A4: WebSocket per-message-deflate (RFC 7692)** — `b.websocket.handleUpgrade` negotiates the `permessage-deflate` extension when the client offers it, accepting `client_max_window_bits` / `server_max_window_bits` constraints (8-15 range, default 15) and always asserting `client_no_context_takeover` + `server_no_context_takeover` so every message uses a fresh zlib state. The connection's send path compresses TEXT/BINARY frames via `zlib.deflateRawSync`, strips the 4-byte `0x00 0x00 0xff 0xff` trailer per RFC 7692 §7.2.1, and sets RSV1 on the first frame of each compressed message. The receive path appends the trailer back and inflates via `zlib.inflateRawSync`. RSV1 on a continuation frame, RSV1 without negotiated extension, RSV2/RSV3 set, and decompressed payload exceeding `maxMessageBytes` all close with the appropriate RFC 6455 status (`PROTOCOL_ERROR`, `INVALID_PAYLOAD`, `MESSAGE_TOO_BIG`). Operator opt-out: pass `permessageDeflate: false` to `handleUpgrade` to refuse the extension even when offered. Tests cover the 101-handshake echo of negotiated params, real compression on a redundant payload (4000 bytes → ~30 bytes on the wire), uncompressed-frame round-trip on the same connection (mixed-mode per RFC 7692 §6), and unknown-extension graceful ignore. v0.6.27's third lazy deferral A3 (TLS-redis live test) was already covered by `test/integration/redis-client-tls.test.js` shipped in v0.6.28. Smoke 6780 / wiki e2e 178 / per-primitive integration 14 files green / wiki integration green.
|
|
13
15
|
- **0.6.30** (2026-05-02) — wiki-app integration gate + framework follow-ups surfaced by it. **`scripts/test-wiki-integration.js`** boots `examples/wiki` against the docker-compose fixture stack (real Redis / MinIO / Mailpit / CoreDNS / NTP / mtls-ca) and drives every backend through the wiki's HTTP surface AND the underlying framework primitives — validates that each primitive routes through the configured backend (cache → custom backend, queue → Redis Streams, mail → Mailpit SMTP, object-store → MinIO sigv4, log-stream → webhook receiver) AND that every fix shipped in v0.6.28 holds in a real-app context, not just in unit-test isolation. **Wiki gains test-only routes** under `/test/*` (gated by `WIKI_INTEGRATION_TEST=1`, mounted before CSRF/staticServe so test POSTs don't have to round-trip a token cookie) covering cache get/set/del, queue enqueue/size, mail send, object-store put/get, http-client fetch, log-stream emit, mtls-ca issue, ntp query, dns lookup, ssrf classify+check, plus `/test/diagnostic` that surfaces the active backend posture (network snapshot, queue backends, log sinks, mtls algorithm). **External-integration two-gate rule** — when a release diff touches a primitive that talks to an external service (`lib/redis-client.js`, `lib/queue-redis.js`, `lib/mail.js`, `lib/network-dns.js`, `lib/object-store/*`, `lib/log-stream*.js`, `lib/external-db.js`, `lib/cluster-*.js`, `lib/mtls-ca.js`, `lib/ssrf-guard.js`, `lib/http-client.js`, `lib/ntp-check.js`, `lib/cache.js`, `lib/webhook.js`), operators MUST run BOTH `scripts/test-integration.js` (per-primitive against real backends) AND `scripts/test-wiki-integration.js` (wiki app exercising the same backends end-to-end) before pushing. Documented in CONTRIBUTING.md's pre-push gate list and CLAUDE.md's release workflow as step 4a. The smoke + wiki-e2e gates above stay PURE (no docker dependency, runs in CI / on a developer laptop / inside prepack-guard); the two-gate is operator-run, opt-in, and surfaces bugs that mocks miss — fire-and-forget races, shutdown drains, TLS pinning, real DNS resolution, real protocol handshakes. **Framework fixes surfaced by the new gate**: `b.mtlsCa.create` now auto-creates `opts.dataDir` with `mode: 0o700` if it doesn't exist (matches `b.logStream`'s local sink, `b.backup`, `b.restoreBundle` — without it the first `initCA()` call hit `ENOENT` writing `ca.key.tmp`); `b.logStream` webhook-sink `close()` now waits for the in-flight `_flush()` before draining the buffer (the v0.6.28 drain fix only tracked emit-time wrapper promises, not the actual HTTP POST in flight; a record arriving mid-flush would buffer + the emit-time `_flush` early-returned on `if (inFlight) return`, and shutdown would then strand it). `b.queue.bootFromEnv()` is now wired in the wiki app's `build-app.js` unconditionally so the wiki picks up `BLAMEJS_QUEUE_PROTOCOL=redis` from env without code changes (was previously a documented-but-commented call site). **Test architecture rule (extends the v0.6.28 rule)** — live tests for the framework's primitives go under `test/integration/`; live tests for the wiki app exercising the framework end-to-end go under `examples/wiki/test/integration.js`. Smoke (`test/smoke.js`) and the existing wiki e2e (`examples/wiki/test/e2e.js`) stay pure. Smoke / wiki-e2e green; per-primitive integration suite green; wiki integration green. Eslint pin alignment + shellcheck added to the documented gate list (carry-over from v0.6.29).
|
package/README.md
CHANGED
|
@@ -46,8 +46,8 @@ The framework bundles the surface a typical Node app reaches for. Every primitiv
|
|
|
46
46
|
- **Crypto** — envelope-versioned PQC at rest (ML-KEM-1024 + P-384 hybrid, XChaCha20-Poly1305, SHAKE256), vault sealing, field-level crypto + cryptographic erasure (`b.cryptoField.eraseRow`), signed webhooks (SLH-DSA-SHAKE-256f), ECIES API encryption (`b.crypto`, `b.vault`, `b.webhook`); pure-JS mTLS CA that issues clientAuth / serverAuth / dual-EKU certs with SAN entries and auto-detects the highest-PQC signature algorithm the vendored x509 library accepts (today: ECDSA-P384-SHA384 bridge; self-upgrades to SLH-DSA / ML-DSA when the X.509 ecosystem catches up), PQC TLS gates inbound + outbound (`b.mtlsCa`, `b.pqcGate`, `b.pqcAgent`).
|
|
47
47
|
- **HTTP** — router with schema-validated routes + OpenAPI publication; full middleware stack (CSRF, CORS, rate-limit, security headers, CSP nonce, body parser, compression, SSE, request log, request-time DB role binding via `b.middleware.dbRoleFor`, in-process CIDR fence via `b.middleware.networkAllowlist`) wired by `createApp`; HTTP/1.1 + HTTP/2 outbound client with SSRF gate (cloud-metadata IPs hard-denied unconditionally; private / loopback / link-local overridable per call), scheme + userinfo + per-host (wildcard / per-method) destination allowlist, redirects, multipart, interceptors, progress, encrypted cookie jar (`b.httpClient`, `b.ssrfGuard`, `b.safeUrl`); operator-tunable network configurability — env-driven NTP / NTS (RFC 8915 authenticated time), IPv4-or-IPv6 NTP servers, DNS with IPv6 / DoH / DoT (private-CA trust pinning via `opts.ca`) / cache / lookup timeout, outbound HTTP proxy (`HTTP_PROXY` / `HTTPS_PROXY` / `NO_PROXY`), runtime DPI trust-store CA additions, application-level heartbeats, TCP socket defaults (`b.network`).
|
|
48
48
|
- **Defensive parsers** — `b.safeJson`, `b.safeBuffer`, `b.safeSql`, `b.safeSchema`, `b.parsers` (XML / TOML / YAML / .env), `b.config` (schema-validated env), `b.fileType` magic-byte content classification with deny-on-upload categories (image / document / archive / executable / etc.).
|
|
49
|
-
- **Communication** — WebSockets with channel/room fan-out across cluster replicas (`b.websocket`, `b.websocketChannels`); mail with multipart + attachments + DKIM + calendar invites + bounce intake (`b.mail`, `b.mailBounce`); generic notification dispatcher with operator-supplied transports (`b.notify`).
|
|
50
|
-
- **Observability** — tamper-evident audit chain with SLH-DSA-signed checkpoints, metrics, tracing (OTel pass-through when wired), PII redaction, log-stream sinks (local file rotation, generic webhook, OTLP/HTTP-JSON to an OTel collector, AWS CloudWatch Logs via SigV4), OTLP/HTTP-JSON exporter for traces + metrics (`b.audit`, `b.metrics`, `b.tracing`, `b.redact`, `b.logStream`, `b.otelExport`); operator-callable boot-time security policy assertions (`b.security.assertProduction`) and tamper-evident config-baseline drift detection signed with the audit-signing key (`b.configDrift`).
|
|
49
|
+
- **Communication** — WebSockets with channel/room fan-out across cluster replicas (`b.websocket`, `b.websocketChannels`); generic distributed pub/sub with cluster-table / Redis PUB/SUB / custom backends (`b.pubsub`); mail with multipart + attachments + DKIM + calendar invites + bounce intake (`b.mail`, `b.mailBounce`); generic notification dispatcher with operator-supplied transports (`b.notify`).
|
|
50
|
+
- **Observability** — tamper-evident audit chain with SLH-DSA-signed checkpoints, metrics, tracing (OTel pass-through when wired), PII redaction, log-stream sinks (local file rotation, generic webhook, OTLP/HTTP-JSON to an OTel collector, AWS CloudWatch Logs via SigV4 with optional autoCreate, RFC 5424 syslog over UDP/TCP/TLS), OTLP/HTTP-JSON exporter for traces + metrics (`b.audit`, `b.metrics`, `b.tracing`, `b.redact`, `b.logStream`, `b.otelExport`); operator-callable boot-time security policy assertions (`b.security.assertProduction`) and tamper-evident config-baseline drift detection signed with the audit-signing key (`b.configDrift`).
|
|
51
51
|
- **i18n** — CLDR plural rules, Accept-Language negotiation, Intl formatters, RTL (`b.i18n`).
|
|
52
52
|
- **Format helpers** — RFC 4180 CSV with Excel formula-injection prevention (`b.csv`), RFC 9562 UUID v4 + v7 (`b.uuid`), URL-safe slugs (`b.slug`), TZ-aware datetime (`b.time`), ZIP creation (`b.archive`), HMAC-signed cursor pagination (`b.pagination`), HTML form rendering + validation + CSRF (`b.forms`).
|
|
53
53
|
- **Production** — cluster leader election with fenced leases over Postgres/SQLite (`b.cluster`); cron + interval scheduler that runs exactly-once globally (`b.scheduler`); retry with full-jitter backoff + circuit breaker (`b.retry`); graceful shutdown (`b.appShutdown`); NTP boot check (`b.ntpCheck`); end-to-end-encrypted backup bundles with pre-flush fail-closed mode (`b.backup`); restore with pulled-bundle footprint preflight (`b.restore`); GDPR / PCI / HIPAA-shaped retention rules with multi-stage warn → archive → erase, legal-hold exemptions, dry-run preview, cross-table cascade (`b.retention`).
|
package/index.js
CHANGED
|
@@ -113,6 +113,7 @@ var time = require("./lib/time");
|
|
|
113
113
|
var uuid = require("./lib/uuid");
|
|
114
114
|
var mail = require("./lib/mail");
|
|
115
115
|
var mailBounce = require("./lib/mail-bounce");
|
|
116
|
+
var pubsub = require("./lib/pubsub");
|
|
116
117
|
var websocketChannels = require("./lib/websocket-channels");
|
|
117
118
|
var nonceStore = require("./lib/nonce-store");
|
|
118
119
|
var scheduler = require("./lib/scheduler");
|
|
@@ -227,6 +228,7 @@ module.exports = {
|
|
|
227
228
|
uuid: uuid,
|
|
228
229
|
mail: mail,
|
|
229
230
|
mailBounce: mailBounce,
|
|
231
|
+
pubsub: pubsub,
|
|
230
232
|
websocketChannels: websocketChannels,
|
|
231
233
|
nonceStore: nonceStore,
|
|
232
234
|
scheduler: scheduler,
|
package/lib/cache.js
CHANGED
|
@@ -739,6 +739,15 @@ function create(opts) {
|
|
|
739
739
|
"redisUrl", "redisPassword", "redisUsername", "redisTls", "redisCa",
|
|
740
740
|
"redisServername", "redisConnectTimeoutMs", "redisCommandTimeoutMs",
|
|
741
741
|
"redisMaxReconnectAttempts",
|
|
742
|
+
// Cross-node invalidation: when set, every successful
|
|
743
|
+
// del/clear/invalidateTag publishes an event on the supplied
|
|
744
|
+
// pubsub instance. Other cache instances on other nodes (or in
|
|
745
|
+
// other processes sharing the pubsub backend) react locally —
|
|
746
|
+
// mostly useful for the memory backend so stale per-node entries
|
|
747
|
+
// don't survive a global tag wipe. The cluster + redis backends
|
|
748
|
+
// are coherent by virtue of their shared store, but a hot
|
|
749
|
+
// memory-tier on top of either still benefits.
|
|
750
|
+
"invalidationPubsub",
|
|
742
751
|
], "cache");
|
|
743
752
|
_validateCreateOpts(opts);
|
|
744
753
|
|
|
@@ -756,6 +765,19 @@ function create(opts) {
|
|
|
756
765
|
var audit = opts.audit || null;
|
|
757
766
|
var operatorObs = opts.observability || null;
|
|
758
767
|
var clock = opts.clock || function () { return Date.now(); };
|
|
768
|
+
var invalidationPubsub = opts.invalidationPubsub || null;
|
|
769
|
+
if (invalidationPubsub && (
|
|
770
|
+
typeof invalidationPubsub.publish !== "function" ||
|
|
771
|
+
typeof invalidationPubsub.subscribe !== "function" ||
|
|
772
|
+
typeof invalidationPubsub.unsubscribe !== "function")) {
|
|
773
|
+
throw _err("BAD_OPT",
|
|
774
|
+
"cache.create: invalidationPubsub must implement { publish, subscribe, unsubscribe } (b.pubsub.create instance)");
|
|
775
|
+
}
|
|
776
|
+
var invalidationChannel = "cache:" + namespace + ":invalidate";
|
|
777
|
+
var invalidationToken = null;
|
|
778
|
+
// Re-entrancy guard — when we receive an invalidation event from
|
|
779
|
+
// another node we MUST NOT re-publish it (infinite fan-out loop).
|
|
780
|
+
var inboundInvalidation = false;
|
|
759
781
|
|
|
760
782
|
function emitObs(name, labels) {
|
|
761
783
|
try {
|
|
@@ -908,6 +930,7 @@ function create(opts) {
|
|
|
908
930
|
}
|
|
909
931
|
if (existed) emitObs("cache.del", { namespace: namespace });
|
|
910
932
|
softExpiry.delete(key);
|
|
933
|
+
_publishInvalidation({ kind: "del", key: key });
|
|
911
934
|
return existed;
|
|
912
935
|
}
|
|
913
936
|
|
|
@@ -946,6 +969,7 @@ function create(opts) {
|
|
|
946
969
|
inflight.clear();
|
|
947
970
|
swrInflight.clear();
|
|
948
971
|
softExpiry.clear();
|
|
972
|
+
_publishInvalidation({ kind: "clear" });
|
|
949
973
|
return purged;
|
|
950
974
|
}
|
|
951
975
|
|
|
@@ -1003,6 +1027,7 @@ function create(opts) {
|
|
|
1003
1027
|
// drop matches clear()'s safer-than-stale posture.
|
|
1004
1028
|
inflight.clear();
|
|
1005
1029
|
swrInflight.clear();
|
|
1030
|
+
_publishInvalidation({ kind: "tag", tag: tag });
|
|
1006
1031
|
return purged;
|
|
1007
1032
|
}
|
|
1008
1033
|
|
|
@@ -1134,9 +1159,45 @@ function create(opts) {
|
|
|
1134
1159
|
return promise;
|
|
1135
1160
|
}
|
|
1136
1161
|
|
|
1162
|
+
function _publishInvalidation(ev) {
|
|
1163
|
+
if (!invalidationPubsub || inboundInvalidation) return;
|
|
1164
|
+
try { invalidationPubsub.publish(invalidationChannel, ev); }
|
|
1165
|
+
catch (_e) { /* publish best-effort — local invalidation already happened */ }
|
|
1166
|
+
}
|
|
1167
|
+
|
|
1168
|
+
async function _onInboundInvalidation(ev /*, meta */) {
|
|
1169
|
+
if (!ev || closed) return;
|
|
1170
|
+
inboundInvalidation = true;
|
|
1171
|
+
try {
|
|
1172
|
+
if (ev.kind === "tag" && typeof ev.tag === "string" &&
|
|
1173
|
+
typeof backend.invalidateTag === "function") {
|
|
1174
|
+
try { await backend.invalidateTag(ev.tag); } catch (_e) {}
|
|
1175
|
+
} else if (ev.kind === "del" && typeof ev.key === "string") {
|
|
1176
|
+
try { await backend.del(ev.key); } catch (_e) {}
|
|
1177
|
+
} else if (ev.kind === "clear") {
|
|
1178
|
+
try { await backend.clear(); } catch (_e) {}
|
|
1179
|
+
}
|
|
1180
|
+
// Wipe local in-flight memoization so a freshly-invalidated key
|
|
1181
|
+
// can't resolve from a still-pending fetch on this node.
|
|
1182
|
+
inflight.clear();
|
|
1183
|
+
swrInflight.clear();
|
|
1184
|
+
softExpiry.clear();
|
|
1185
|
+
} finally {
|
|
1186
|
+
inboundInvalidation = false;
|
|
1187
|
+
}
|
|
1188
|
+
}
|
|
1189
|
+
|
|
1190
|
+
if (invalidationPubsub) {
|
|
1191
|
+
invalidationToken = invalidationPubsub.subscribe(invalidationChannel, _onInboundInvalidation);
|
|
1192
|
+
}
|
|
1193
|
+
|
|
1137
1194
|
async function close() {
|
|
1138
1195
|
if (closed) return;
|
|
1139
1196
|
closed = true;
|
|
1197
|
+
if (invalidationPubsub && invalidationToken) {
|
|
1198
|
+
try { invalidationPubsub.unsubscribe(invalidationToken); } catch (_e) {}
|
|
1199
|
+
invalidationToken = null;
|
|
1200
|
+
}
|
|
1140
1201
|
inflight.clear();
|
|
1141
1202
|
swrInflight.clear();
|
|
1142
1203
|
softExpiry.clear();
|
package/lib/db.js
CHANGED
|
@@ -110,7 +110,7 @@ var RESERVED_TABLE_NAMES = new Set([
|
|
|
110
110
|
"_blamejs_audit_purge_anchor",
|
|
111
111
|
"_blamejs_scheduler_ticks",
|
|
112
112
|
"_blamejs_rate_limit_counters",
|
|
113
|
-
"
|
|
113
|
+
"_blamejs_pubsub_messages",
|
|
114
114
|
"_blamejs_api_encrypt_nonces",
|
|
115
115
|
"_blamejs_api_keys",
|
|
116
116
|
"_blamejs_cache",
|
|
@@ -260,15 +260,16 @@ var FRAMEWORK_SCHEMA = [
|
|
|
260
260
|
sealedFields: [],
|
|
261
261
|
},
|
|
262
262
|
{
|
|
263
|
-
//
|
|
264
|
-
//
|
|
265
|
-
//
|
|
266
|
-
//
|
|
267
|
-
//
|
|
268
|
-
|
|
263
|
+
// _blamejs_pubsub_messages — cluster fan-out for `b.pubsub` (the
|
|
264
|
+
// generalization of the previous WebSocket-specific table). Any
|
|
265
|
+
// pubsub instance using the `cluster` backend writes a row on
|
|
266
|
+
// publish; other nodes poll for new ids and dispatch to their
|
|
267
|
+
// local subscribers. Rows older than the configured retention
|
|
268
|
+
// window are pruned by the backend on a rate-limited basis.
|
|
269
|
+
name: "_blamejs_pubsub_messages",
|
|
269
270
|
columns: {
|
|
270
271
|
id: "INTEGER PRIMARY KEY AUTOINCREMENT",
|
|
271
|
-
|
|
272
|
+
topic: "TEXT NOT NULL",
|
|
272
273
|
payload: "TEXT NOT NULL",
|
|
273
274
|
publishedAt: "INTEGER NOT NULL",
|
|
274
275
|
publishedBy: "TEXT NOT NULL",
|
package/lib/framework-schema.js
CHANGED
|
@@ -94,7 +94,7 @@ var LOCAL_TO_EXTERNAL = Object.freeze({
|
|
|
94
94
|
// WebSocket channel-hub cluster fan-out — publish() writes a row,
|
|
95
95
|
// other nodes poll for new ids and dispatch to their local
|
|
96
96
|
// subscribers. Same dual-storage shape as sessions / jobs / etc.
|
|
97
|
-
|
|
97
|
+
_blamejs_pubsub_messages: "_blamejs_pubsub_messages",
|
|
98
98
|
_blamejs_api_encrypt_nonces: "_blamejs_api_encrypt_nonces",
|
|
99
99
|
// _blamejs_api_keys — operator-facing API-key registry table for the
|
|
100
100
|
// b.apiKey primitive. PRIMARY KEY is namespace-scoped id (so multiple
|
|
@@ -368,14 +368,15 @@ function _rateLimitCountersDDL(dialect) {
|
|
|
368
368
|
};
|
|
369
369
|
}
|
|
370
370
|
|
|
371
|
-
//
|
|
372
|
-
//
|
|
373
|
-
//
|
|
374
|
-
//
|
|
375
|
-
//
|
|
376
|
-
|
|
371
|
+
// _blamejs_pubsub_messages — cluster fan-out for `b.pubsub` (the
|
|
372
|
+
// generalization of the previous WebSocket-specific table). publish()
|
|
373
|
+
// on any node writes a row; other nodes poll for new ids past their
|
|
374
|
+
// last seen and dispatch to local subscribers. Auto-incrementing id
|
|
375
|
+
// is essential — postgres needs BIGSERIAL, sqlite gets INTEGER
|
|
376
|
+
// PRIMARY KEY (which auto-increments implicitly).
|
|
377
|
+
function _pubsubMessagesDDL(dialect) {
|
|
377
378
|
var t = _types(dialect);
|
|
378
|
-
var name = LOCAL_TO_EXTERNAL.
|
|
379
|
+
var name = LOCAL_TO_EXTERNAL._blamejs_pubsub_messages;
|
|
379
380
|
var idCol = dialect === "postgres"
|
|
380
381
|
? "id BIGSERIAL PRIMARY KEY"
|
|
381
382
|
: "id INTEGER PRIMARY KEY AUTOINCREMENT";
|
|
@@ -383,7 +384,7 @@ function _wsMessagesDDL(dialect) {
|
|
|
383
384
|
create:
|
|
384
385
|
"CREATE TABLE IF NOT EXISTS " + name + " (" +
|
|
385
386
|
" " + idCol + "," +
|
|
386
|
-
"
|
|
387
|
+
" topic TEXT NOT NULL," +
|
|
387
388
|
" payload TEXT NOT NULL," +
|
|
388
389
|
" publishedAt " + t.INT + " NOT NULL," +
|
|
389
390
|
" publishedBy TEXT NOT NULL" +
|
|
@@ -683,7 +684,7 @@ async function ensureSchema(opts) {
|
|
|
683
684
|
_auditPurgeAnchorDDL(dialect),
|
|
684
685
|
_schedulerTicksDDL(dialect),
|
|
685
686
|
_rateLimitCountersDDL(dialect),
|
|
686
|
-
|
|
687
|
+
_pubsubMessagesDDL(dialect),
|
|
687
688
|
_apiEncryptNoncesDDL(dialect),
|
|
688
689
|
_apiKeysDDL(dialect),
|
|
689
690
|
_sessionsDDL(dialect),
|
|
@@ -87,12 +87,13 @@ function _serializeBatch(events, cfg, sequenceToken) {
|
|
|
87
87
|
return Buffer.from(JSON.stringify(body), "utf8");
|
|
88
88
|
}
|
|
89
89
|
|
|
90
|
-
function _signedHeaders(cfg, body) {
|
|
90
|
+
function _signedHeaders(cfg, body, target) {
|
|
91
|
+
target = target || "Logs_20140328.PutLogEvents";
|
|
91
92
|
var url = _resolveEndpoint(cfg);
|
|
92
93
|
var payloadHash = nodeCrypto.createHash("sha256").update(body).digest("hex");
|
|
93
94
|
var unsigned = {
|
|
94
95
|
"Content-Type": "application/x-amz-json-1.1",
|
|
95
|
-
"X-Amz-Target":
|
|
96
|
+
"X-Amz-Target": target,
|
|
96
97
|
};
|
|
97
98
|
var signed = sigv4.signRequest({
|
|
98
99
|
method: "POST",
|
|
@@ -108,6 +109,41 @@ function _signedHeaders(cfg, body) {
|
|
|
108
109
|
return signed.headers;
|
|
109
110
|
}
|
|
110
111
|
|
|
112
|
+
// CloudWatch CreateLogGroup / CreateLogStream wrappers. Operator opts in
|
|
113
|
+
// via cfg.autoCreate = true; default stays "operator pre-creates via
|
|
114
|
+
// Terraform / CDK / aws cli" so the framework doesn't paper over an
|
|
115
|
+
// IAM-misconfigured deployment with surprising side effects.
|
|
116
|
+
async function _ensureLogGroupAndStream(cfg) {
|
|
117
|
+
// CreateLogGroup — idempotent on the wire (we treat
|
|
118
|
+
// ResourceAlreadyExistsException as success).
|
|
119
|
+
var groupBody = Buffer.from(JSON.stringify({ logGroupName: cfg.logGroupName }), "utf8");
|
|
120
|
+
var groupHeaders = _signedHeaders(cfg, groupBody, "Logs_20140328.CreateLogGroup");
|
|
121
|
+
try {
|
|
122
|
+
await _post(cfg, groupBody, groupHeaders);
|
|
123
|
+
} catch (e) {
|
|
124
|
+
var msg = (e && e.message) || "";
|
|
125
|
+
if (!/ResourceAlreadyExistsException/.test(msg)) {
|
|
126
|
+
throw _err("AUTOCREATE_FAILED",
|
|
127
|
+
"log-stream cloudwatch autoCreate: CreateLogGroup failed: " + msg);
|
|
128
|
+
}
|
|
129
|
+
}
|
|
130
|
+
// CreateLogStream — same idempotency treatment.
|
|
131
|
+
var streamBody = Buffer.from(JSON.stringify({
|
|
132
|
+
logGroupName: cfg.logGroupName,
|
|
133
|
+
logStreamName: cfg.logStreamName,
|
|
134
|
+
}), "utf8");
|
|
135
|
+
var streamHeaders = _signedHeaders(cfg, streamBody, "Logs_20140328.CreateLogStream");
|
|
136
|
+
try {
|
|
137
|
+
await _post(cfg, streamBody, streamHeaders);
|
|
138
|
+
} catch (e) {
|
|
139
|
+
var msg2 = (e && e.message) || "";
|
|
140
|
+
if (!/ResourceAlreadyExistsException/.test(msg2)) {
|
|
141
|
+
throw _err("AUTOCREATE_FAILED",
|
|
142
|
+
"log-stream cloudwatch autoCreate: CreateLogStream failed: " + msg2);
|
|
143
|
+
}
|
|
144
|
+
}
|
|
145
|
+
}
|
|
146
|
+
|
|
111
147
|
function _post(cfg, body, headers) {
|
|
112
148
|
return httpClient.request({
|
|
113
149
|
method: "POST",
|
|
@@ -144,9 +180,11 @@ function create(config) {
|
|
|
144
180
|
}
|
|
145
181
|
if (!config.logGroupName || !config.logStreamName) {
|
|
146
182
|
throw _err("BAD_OPT",
|
|
147
|
-
"log-stream cloudwatch requires { logGroupName, logStreamName } " +
|
|
148
|
-
"
|
|
149
|
-
"
|
|
183
|
+
"log-stream cloudwatch requires { logGroupName, logStreamName }. " +
|
|
184
|
+
"Operator pre-creates both via aws / CDK / Terraform by default; " +
|
|
185
|
+
"pass { autoCreate: true } to have the framework issue " +
|
|
186
|
+
"CreateLogGroup + CreateLogStream on first emit (idempotent — " +
|
|
187
|
+
"ResourceAlreadyExistsException treated as success).");
|
|
150
188
|
}
|
|
151
189
|
var cfg = Object.assign({}, DEFAULTS, config);
|
|
152
190
|
var onDrop = typeof cfg.onDrop === "function" ? cfg.onDrop : null;
|
|
@@ -186,11 +224,32 @@ function create(config) {
|
|
|
186
224
|
return batch;
|
|
187
225
|
}
|
|
188
226
|
|
|
227
|
+
// autoCreate handshake — runs once per process before the first
|
|
228
|
+
// PutLogEvents. Promise cached so concurrent emits don't fire
|
|
229
|
+
// duplicate CreateLogGroup / CreateLogStream calls.
|
|
230
|
+
var autoCreatePromise = null;
|
|
231
|
+
function _ensureAutoCreated() {
|
|
232
|
+
if (!cfg.autoCreate) return Promise.resolve();
|
|
233
|
+
if (!autoCreatePromise) autoCreatePromise = _ensureLogGroupAndStream(cfg);
|
|
234
|
+
return autoCreatePromise;
|
|
235
|
+
}
|
|
236
|
+
|
|
189
237
|
async function _flush() {
|
|
190
238
|
if (inFlight) return;
|
|
191
239
|
if (buffer.length === 0) return;
|
|
192
240
|
inFlight = true;
|
|
193
241
|
try {
|
|
242
|
+
try { await _ensureAutoCreated(); }
|
|
243
|
+
catch (acErr) {
|
|
244
|
+
// autoCreate failure is permanent — every subsequent batch
|
|
245
|
+
// would hit the same error. Drop the queue with the
|
|
246
|
+
// operator-supplied onDrop callback so they see exactly which
|
|
247
|
+
// events were lost AND why, then bail.
|
|
248
|
+
var allBuffered = buffer.splice(0, buffer.length);
|
|
249
|
+
dropCount += allBuffered.length;
|
|
250
|
+
_emitDrop("autocreate-failed", allBuffered, acErr);
|
|
251
|
+
return;
|
|
252
|
+
}
|
|
194
253
|
while (buffer.length > 0 && !closed) {
|
|
195
254
|
var batch = _takeBatch();
|
|
196
255
|
if (batch.length === 0) break;
|