@blamejs/core 0.6.23 → 0.6.25
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +2 -0
- package/README.md +1 -1
- package/lib/log-stream-cloudwatch.js +309 -0
- package/lib/log-stream-otlp.js +292 -0
- package/lib/log-stream.js +98 -13
- package/lib/object-store/sigv4.js +8 -2
- package/package.json +1 -1
- package/sbom.cyclonedx.json +6 -6
package/CHANGELOG.md
CHANGED
|
@@ -8,6 +8,8 @@ upgrading across more than a few patches at a time.
|
|
|
8
8
|
|
|
9
9
|
## v0.6.x
|
|
10
10
|
|
|
11
|
+
- **0.6.25** (2026-05-02) — `b.logStream` gains an AWS CloudWatch Logs sink AND framework-level env wiring. **CloudWatch sink**: `protocol: "cloudwatch"` POSTs `PutLogEvents` over HTTPS with SigV4 signing (service `logs`); operator pre-creates the log group + log stream (the framework does NOT auto-create). Honors IAM role + STS session-token credentials. Respects all three CloudWatch caps automatically: 10,000 events / 1 MiB total payload / 256 KiB-per-event. Per-event oversize dropped at `emit()`-time with `onDrop` fired (truncated message in the drop notification). Per-batch oversize split mid-flush. Permanent AWS errors (`ResourceNotFoundException` / `AccessDeniedException` / `InvalidParameterException` / `UnrecognizedClientException` / `SerializationException`) skip the retry budget. `InvalidSequenceTokenException` (legacy CW accounts) extracts the expected token from the error message and retries with it once. `lib/object-store/sigv4.js` `signRequest()` is now service-agnostic — accepts `opts.service` (default still `"s3"` for back-compat). Removed `cloudwatch` from `DEFERRED_PROTOCOLS`. **`b.logStream.bootFromEnv({ env })`**: framework-level env-driven init mirroring `b.network.bootFromEnv`. Reads `BLAMEJS_LOG_STREAM_PROTOCOL` (`local`/`webhook`/`otlp`/`cloudwatch`), `BLAMEJS_LOG_STREAM_URL`, `BLAMEJS_LOG_STREAM_TOKEN`, `BLAMEJS_LOG_STREAM_SERVICE_NAME`, `BLAMEJS_LOG_STREAM_SERVICE_VERSION`, `BLAMEJS_LOG_STREAM_CLOUDWATCH_LOG_GROUP`, `BLAMEJS_LOG_STREAM_CLOUDWATCH_LOG_STREAM`, `BLAMEJS_LOG_STREAM_PATH`, plus standard AWS_*. Operators get a working log-stream sink without writing build-app code. Wiki app's `build-app.js` replaced its inline env-reading with one `b.logStream.bootFromEnv()` call; both docker-compose configs declare the new env knobs. **Wiki env-snapshot test**: parallel to api-snapshot.json — walks `process.env.X` / `env.X` / `safeEnv.readVar("X")` reads in the wiki app + framework `lib/`, walks docker-compose env declarations, captures the union as `examples/wiki/env-snapshot.json`, fails the e2e gate when env vars are added/removed without updating the snapshot OR when source-only / compose-only gaps appear (env knob declared but unread, env read but undocumented). The validator immediately surfaced 13 real gaps in the wiki app: 5 `WIKI_*` env vars read by source but missing from docker-compose (`WIKI_VAULT_MODE`, `WIKI_DB_AT_REST`, `WIKI_AUDIT_SIGNING_MODE`, `WIKI_BIND`, `WIKI_SITE_URL`), 2 framework env vars (`BLAMEJS_AUDIT_SIGNING_MODE`, `BLAMEJS_TMPDIR`) read by `lib/db.js` via `safeEnv.readVar` but never declared in the wiki's compose configs, and 6 dead env knobs in compose that nothing read. All 13 fixed. Update workflow: `BLAMEJS_UPDATE_ENV_SNAPSHOT=1 node examples/wiki/test/validate-env-snapshot.js` (mirrors the api-snapshot UX). Wiki observability page documents the new sink alongside webhook + otlp; README "What ships in the box" calls out all four log-stream sinks. Tests cover endpoint resolution, event-byte accounting, batch sorting + sequence-token round-trip, permanent-error classifier, validation, round-trip via mock CloudWatch, STS session-token propagation, ResourceNotFoundException no-retry path, 256 KiB per-event hard cap, dispatcher integration, AND batch splitting on the 1-MiB cap (5 quarter-MB events POST as 4 + 1 batches).
|
|
12
|
+
- **0.6.24** (2026-05-02) — `b.logStream` gains an OTLP/HTTP-JSON sink. `protocol: "otlp"` now forwards log records to any OpenTelemetry collector via the OTel Logs Data Model — `resourceLogs` → `scopeLogs` → `logRecords` envelope with `severityNumber` (debug=5/info=9/warn=13/error=17), `severityText`, `timeUnixNano` (string-encoded for JSON-safe 64-bit), `body.stringValue`, and OTel-typed attributes. Operator config: `{ url, serviceName, serviceVersion, resourceAttributes, auth, headers, batchSize, retry, onDrop, ... }` — same back-pressure semantics as the webhook sink (per-sink ring buffer, batched flush on size or maxBatchAgeMs, exponential-backoff retry, drop-on-overflow with operator-supplied onDrop callback). URL convention: `/v1/logs` is auto-appended when the operator passes the collector root. JSON not protobuf — operators benchmarking >100K logs/s ship the OTel Collector locally so the framework hands JSON to a sidecar that forwards via gRPC. Removed `otlp` from `DEFERRED_PROTOCOLS`. Wiki observability page documents the new sink with a `b.logStream.init` example pointing at an OTel collector. Tests cover URL resolution, attribute encoding (string / int / float / bool / array / nested object), severity mapping, round-trip via mock collector, auth header pass-through, retry on 5xx + drop on retry-exhaustion, buffer overflow, dispatcher integration. Verified 0 leaks across 291 commits via `gitleaks`.
|
|
11
13
|
- **0.6.23** (2026-05-02) — v0.6.22 follow-up cleanup. The cron-repeat call site in queue-local.js was passing both `availableAt` AND `delaySeconds` (the former was the precise next-fire ms; the latter was a redundant `Math.floor((nextMs - nowMs) / 1000)` computation that only existed to work around the bug v0.6.22 fixed). Dropped — the cron repeat now passes `availableAt` alone, matching the queue's documented precedence rule. The enqueue() docstring gains a 20-line "SCHEDULING PRECEDENCE" header documenting that `opts.availableAt` wins over `opts.delaySeconds` when both are passed, why the framework chose that direction, and which callers should use which form. New round-trip preservation regression test (`testEnqueueRoundTripsAvailableAt`) covers three precise targets, the delaySeconds-only path, and the both-opts-set case — gates against any future "I'll just rederive it from the floored seconds" mistake. Audited the rest of the framework for the same `(absolute-time, relative-time)` opt-overlap shape (cache.set, session.rotate, apiKey, dualControl): queue is the only primitive carrying both forms, so a generalized `b.time.resolveTimePoint` primitive would be premature with one call site.
|
|
12
14
|
- **0.6.22** (2026-05-02) — `b.queue.enqueue({ availableAt })` is now honoured. Previously the local-protocol enqueue() ignored opts.availableAt entirely, recomputing from `Date.now() + delaySeconds*1000`. The cron-repeat path passes both fields (the exact next-fire ms in availableAt + the floored seconds in delaySeconds), and the enqueue's recomputation lost sub-second precision plus drifted on the internal clock-vs-caller delta. Symptoms: cron-scheduled jobs landed up to 999ms off the intended boundary; the queue-flow-repeat smoke test was intermittently flaky on slow CI runners (caught by ubuntu-latest on the v0.6.21 commit). Fix: enqueue() honours opts.availableAt directly when finite; falls back to delaySeconds-based shorthand otherwise. Operators relying on `enqueue({ availableAt: T })` for non-cron scheduled jobs (e.g. "deliver this notification at exactly 09:00 tomorrow") now get the requested time instead of nowMs+0.
|
|
13
15
|
- **0.6.21** (2026-05-02) — closes the medium / low audit findings flagged after v0.6.18. **Cluster cache `invalidateTag`** now actually works — the cluster backend gained a `_blamejs_cache_tags` junction table (`(cacheKey, tag)` PK + index on `tag`), tag-aware `set` / `del` / `clear` / `_sweep`, plus `getTags(key)`. The old NOT_SUPPORTED-on-cluster path is gone; multi-tag rotation, mid-flight tag replacement on update, and namespace-scoped sweeps all covered. **Multi-column cursor pagination** — `b.pagination.cursor({ orderBy: [{column,direction},...] })` accepts a string (single column), an array of strings (multi, all using `opts.direction`), or an array of `{column,direction}` objects (mixed directions). The keyset WHERE expands to the standard OR cascade so successive pages can't skip or repeat rows when ties on the leading columns are broken by trailing ones. `_id` is appended as a tiebreaker if not in the chain. The Query class also gained chained `orderBy(col, dir)` calls — second-and-later calls extend a multi-column ORDER BY in the SQL. Cursor format is bumped to encode `{ orderKey, vals, forward }` instead of the old `{ orderBy, dir, orderByVal, id, forward }` — pre-1.0 break, no compat shim. **DoH POST mode** (RFC 8484 §4.1) — `b.network.dns.useDnsOverHttps({ method })` accepts `"GET" | "POST" | undefined` (auto). Auto switches to POST when the GET URL would exceed 2048 bytes (long DNS names). **DoT connection pooling** — per-`(host:port)` cached TLS socket with a 2-minute idle timeout, serialized in-flight queries per socket. Eliminates the per-query handshake. **INI parser** shipped as `b.parsers.ini` — covers Windows .ini / .gitconfig / systemd-unit / php.ini / tox.ini shape: sections (incl. `[parent.child]` / `[parent "child"]` nesting), `;` and `#` comments (inline + leading), single + double quoting with `\n` `\t` `\\` `\"` `\'` escapes, boolean coercion (`true`/`false`/`yes`/`no`/`on`/`off`), decimal + hex integers + floats. Prototype-pollution defense (`__proto__` / `constructor` / `prototype` rejected). Duplicate-key policy throws by default; `onDuplicate: "first" | "last"` opts in to silent shadowing. Section / per-section key / value-bytes caps configurable. **Cookie-jar file persistence** — `b.httpClient.cookieJar.create({ persist: "file", file: "/abs/path", vault: b.vault })` loads at construct, debounce-flushes on every set/clear, plus `flush()` and `close()` for explicit lifecycle. With `vault`, on-disk bytes are sealed; without, plaintext JSON (operator chooses). **Comment cleanups per rule §4** — `vault/index.js` stale `// later` removed, `mail.js` "future patch" wording rewritten as scope, `bundler.js` "What it does NOT do today (deliberately deferred)" rewritten as "Out of scope", `archive.js` "v1 scope cuts (deferred)" rewritten as "Out of scope", `framework-schema.js` "next release" wording rewritten to describe what's actually shipped, `http-client.js` "out of reach today" / "we'll plumb it through when h3 lands" rewritten as scope statements. **README CLI section** updated to reflect the v0.6.17 + v0.6.18 + v0.6.19 + v0.6.20 additions (security / config-drift / file-type / password / erase / retention) — was missing 6 subcommands.
|
package/README.md
CHANGED
|
@@ -47,7 +47,7 @@ The framework bundles the surface a typical Node app reaches for. Every primitiv
|
|
|
47
47
|
- **HTTP** — router with schema-validated routes + OpenAPI publication; full middleware stack (CSRF, CORS, rate-limit, security headers, CSP nonce, body parser, compression, SSE, request log, request-time DB role binding via `b.middleware.dbRoleFor`, in-process CIDR fence via `b.middleware.networkAllowlist`) wired by `createApp`; HTTP/1.1 + HTTP/2 outbound client with SSRF gate, scheme + userinfo + per-host (wildcard / per-method) destination allowlist, redirects, multipart, interceptors, progress, encrypted cookie jar (`b.httpClient`, `b.ssrfGuard`, `b.safeUrl`); operator-tunable network configurability — env-driven NTP / NTS (RFC 8915 authenticated time), DNS with IPv6 / DoH / DoT / cache / lookup timeout, outbound HTTP proxy (`HTTP_PROXY` / `HTTPS_PROXY` / `NO_PROXY`), runtime DPI trust-store CA additions, application-level heartbeats, TCP socket defaults (`b.network`).
|
|
48
48
|
- **Defensive parsers** — `b.safeJson`, `b.safeBuffer`, `b.safeSql`, `b.safeSchema`, `b.parsers` (XML / TOML / YAML / .env), `b.config` (schema-validated env), `b.fileType` magic-byte content classification with deny-on-upload categories (image / document / archive / executable / etc.).
|
|
49
49
|
- **Communication** — WebSockets with channel/room fan-out across cluster replicas (`b.websocket`, `b.websocketChannels`); mail with multipart + attachments + DKIM + calendar invites + bounce intake (`b.mail`, `b.mailBounce`); generic notification dispatcher with operator-supplied transports (`b.notify`).
|
|
50
|
-
- **Observability** — tamper-evident audit chain with SLH-DSA-signed checkpoints, metrics, tracing (OTel pass-through when wired), PII redaction, log-stream sinks, OTLP/HTTP-JSON exporter for
|
|
50
|
+
- **Observability** — tamper-evident audit chain with SLH-DSA-signed checkpoints, metrics, tracing (OTel pass-through when wired), PII redaction, log-stream sinks (local file rotation, generic webhook, OTLP/HTTP-JSON to an OTel collector, AWS CloudWatch Logs via SigV4), OTLP/HTTP-JSON exporter for traces + metrics (`b.audit`, `b.metrics`, `b.tracing`, `b.redact`, `b.logStream`, `b.otelExport`); operator-callable boot-time security policy assertions (`b.security.assertProduction`) and tamper-evident config-baseline drift detection signed with the audit-signing key (`b.configDrift`).
|
|
51
51
|
- **i18n** — CLDR plural rules, Accept-Language negotiation, Intl formatters, RTL (`b.i18n`).
|
|
52
52
|
- **Format helpers** — RFC 4180 CSV with Excel formula-injection prevention (`b.csv`), RFC 9562 UUID v4 + v7 (`b.uuid`), URL-safe slugs (`b.slug`), TZ-aware datetime (`b.time`), ZIP creation (`b.archive`), HMAC-signed cursor pagination (`b.pagination`), HTML form rendering + validation + CSRF (`b.forms`).
|
|
53
53
|
- **Production** — cluster leader election with fenced leases over Postgres/SQLite (`b.cluster`); cron + interval scheduler that runs exactly-once globally (`b.scheduler`); retry with full-jitter backoff + circuit breaker (`b.retry`); graceful shutdown (`b.appShutdown`); NTP boot check (`b.ntpCheck`); end-to-end-encrypted backup bundles with pre-flush fail-closed mode (`b.backup`); restore with pulled-bundle footprint preflight (`b.restore`); GDPR / PCI / HIPAA-shaped retention rules with multi-stage warn → archive → erase, legal-hold exemptions, dry-run preview, cross-table cascade (`b.retention`).
|
|
@@ -0,0 +1,309 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
/**
|
|
3
|
+
* AWS CloudWatch Logs sink — PutLogEvents over HTTPS with SigV4.
|
|
4
|
+
*
|
|
5
|
+
* Operator config:
|
|
6
|
+
*
|
|
7
|
+
* {
|
|
8
|
+
* region: "us-east-1"
|
|
9
|
+
* accessKeyId: env("AWS_ACCESS_KEY_ID")
|
|
10
|
+
* secretAccessKey: env("AWS_SECRET_ACCESS_KEY")
|
|
11
|
+
* sessionToken: env("AWS_SESSION_TOKEN") // optional, STS creds
|
|
12
|
+
* logGroupName: "my-app-logs" // operator pre-creates
|
|
13
|
+
* logStreamName: "instance-1" // operator pre-creates
|
|
14
|
+
* endpoint: "https://logs.us-east-1.amazonaws.com" // optional
|
|
15
|
+
* batchSize: 100 // CW caps at 10K events / 1 MiB per call
|
|
16
|
+
* maxBatchAgeMs: C.TIME.seconds(5)
|
|
17
|
+
* timeoutMs: C.TIME.seconds(30)
|
|
18
|
+
* retry: { maxAttempts, baseDelayMs, ... }
|
|
19
|
+
* bufferLimit: 10000
|
|
20
|
+
* onDrop: function ({ reason, batch, error }) { ... }
|
|
21
|
+
* }
|
|
22
|
+
*
|
|
23
|
+
* Wire format (Logs_20140328 PutLogEvents — JSON-1.1 over HTTPS):
|
|
24
|
+
*
|
|
25
|
+
* POST /
|
|
26
|
+
* X-Amz-Target: Logs_20140328.PutLogEvents
|
|
27
|
+
* Content-Type: application/x-amz-json-1.1
|
|
28
|
+
* Authorization: AWS4-HMAC-SHA256 Credential=... SignedHeaders=... Signature=...
|
|
29
|
+
* Body: { logGroupName, logStreamName, logEvents: [{ timestamp, message }, ...] }
|
|
30
|
+
*
|
|
31
|
+
* AWS quirks the framework handles:
|
|
32
|
+
* - Events MUST be sorted by timestamp ascending — sink sorts before send.
|
|
33
|
+
* - Per-batch caps: 10,000 events AND <= 1 MiB total payload. Operator
|
|
34
|
+
* batchSize is enforced; the framework also splits batches when the
|
|
35
|
+
* 1 MiB ceiling is reached mid-build.
|
|
36
|
+
* - Per-event 256 KiB hard cap. Oversized events are dropped at emit-time
|
|
37
|
+
* with onDrop fired.
|
|
38
|
+
* - sequenceToken is optional in modern CloudWatch (post-2023). If a
|
|
39
|
+
* legacy account requires it, CloudWatch returns
|
|
40
|
+
* InvalidSequenceTokenException with the expected token; the
|
|
41
|
+
* framework retries with that token transparently.
|
|
42
|
+
* - ResourceNotFoundException -> permanent error (operator forgot to
|
|
43
|
+
* create the log group or stream); surfaced via onDrop with a
|
|
44
|
+
* clear error.
|
|
45
|
+
*
|
|
46
|
+
* SigV4 signing reuses lib/object-store/sigv4.js with service: "logs".
|
|
47
|
+
*/
|
|
48
|
+
var C = require("./constants");
|
|
49
|
+
var nodeCrypto = require("node:crypto");
|
|
50
|
+
var sigv4 = require("./object-store/sigv4");
|
|
51
|
+
var retryHelper = require("./retry");
|
|
52
|
+
var { LogStreamError } = require("./framework-error");
|
|
53
|
+
var httpClient = require("./http-client");
|
|
54
|
+
|
|
55
|
+
var MAX_RESPONSE_BYTES = C.BYTES.mib(1);
|
|
56
|
+
var CW_MAX_EVENTS_PER_BATCH = 10000;
|
|
57
|
+
var CW_MAX_BATCH_BYTES = C.BYTES.mib(1);
|
|
58
|
+
var CW_MAX_EVENT_BYTES = 256 * 1024;
|
|
59
|
+
var CW_EVENT_OVERHEAD_BYTES = 26;
|
|
60
|
+
|
|
61
|
+
var DEFAULTS = {
|
|
62
|
+
batchSize: 100,
|
|
63
|
+
maxBatchAgeMs: C.TIME.seconds(5),
|
|
64
|
+
timeoutMs: C.TIME.seconds(30),
|
|
65
|
+
bufferLimit: 10000,
|
|
66
|
+
};
|
|
67
|
+
|
|
68
|
+
var _err = LogStreamError.factory;
|
|
69
|
+
|
|
70
|
+
function _resolveEndpoint(cfg) {
|
|
71
|
+
if (cfg.endpoint) return cfg.endpoint.replace(/\/+$/, "") + "/";
|
|
72
|
+
return "https://logs." + cfg.region + ".amazonaws.com/";
|
|
73
|
+
}
|
|
74
|
+
|
|
75
|
+
function _eventByteSize(message) {
|
|
76
|
+
return Buffer.byteLength(message, "utf8") + CW_EVENT_OVERHEAD_BYTES;
|
|
77
|
+
}
|
|
78
|
+
|
|
79
|
+
function _serializeBatch(events, cfg, sequenceToken) {
|
|
80
|
+
events.sort(function (a, b) { return a.timestamp - b.timestamp; });
|
|
81
|
+
var body = {
|
|
82
|
+
logGroupName: cfg.logGroupName,
|
|
83
|
+
logStreamName: cfg.logStreamName,
|
|
84
|
+
logEvents: events,
|
|
85
|
+
};
|
|
86
|
+
if (sequenceToken) body.sequenceToken = sequenceToken;
|
|
87
|
+
return Buffer.from(JSON.stringify(body), "utf8");
|
|
88
|
+
}
|
|
89
|
+
|
|
90
|
+
function _signedHeaders(cfg, body) {
|
|
91
|
+
var url = _resolveEndpoint(cfg);
|
|
92
|
+
var payloadHash = nodeCrypto.createHash("sha256").update(body).digest("hex");
|
|
93
|
+
var unsigned = {
|
|
94
|
+
"Content-Type": "application/x-amz-json-1.1",
|
|
95
|
+
"X-Amz-Target": "Logs_20140328.PutLogEvents",
|
|
96
|
+
};
|
|
97
|
+
var signed = sigv4.signRequest({
|
|
98
|
+
method: "POST",
|
|
99
|
+
url: url,
|
|
100
|
+
headers: unsigned,
|
|
101
|
+
payloadHash: payloadHash,
|
|
102
|
+
region: cfg.region,
|
|
103
|
+
service: "logs",
|
|
104
|
+
accessKeyId: cfg.accessKeyId,
|
|
105
|
+
secretAccessKey: cfg.secretAccessKey,
|
|
106
|
+
sessionToken: cfg.sessionToken || null,
|
|
107
|
+
});
|
|
108
|
+
return signed.headers;
|
|
109
|
+
}
|
|
110
|
+
|
|
111
|
+
function _post(cfg, body, headers) {
|
|
112
|
+
return httpClient.request({
|
|
113
|
+
method: "POST",
|
|
114
|
+
url: _resolveEndpoint(cfg),
|
|
115
|
+
headers: headers,
|
|
116
|
+
body: body,
|
|
117
|
+
idleTimeoutMs: cfg.timeoutMs,
|
|
118
|
+
maxResponseBytes: MAX_RESPONSE_BYTES,
|
|
119
|
+
errorClass: LogStreamError,
|
|
120
|
+
allowedProtocols: cfg.allowedProtocols,
|
|
121
|
+
allowInternal: cfg.allowInternal,
|
|
122
|
+
});
|
|
123
|
+
}
|
|
124
|
+
|
|
125
|
+
function _isPermanentAwsError(err) {
|
|
126
|
+
if (!err) return false;
|
|
127
|
+
var msg = err.message || "";
|
|
128
|
+
if (/ResourceNotFoundException/.test(msg)) return true;
|
|
129
|
+
if (/InvalidParameterException/.test(msg)) return true;
|
|
130
|
+
if (/UnrecognizedClientException/.test(msg)) return true;
|
|
131
|
+
if (/AccessDeniedException/.test(msg)) return true;
|
|
132
|
+
if (/SerializationException/.test(msg)) return true;
|
|
133
|
+
return false;
|
|
134
|
+
}
|
|
135
|
+
|
|
136
|
+
function create(config) {
|
|
137
|
+
if (!config || !config.region) {
|
|
138
|
+
throw _err("BAD_OPT", "log-stream cloudwatch requires { region }");
|
|
139
|
+
}
|
|
140
|
+
if (!config.accessKeyId || !config.secretAccessKey) {
|
|
141
|
+
throw _err("BAD_OPT",
|
|
142
|
+
"log-stream cloudwatch requires { accessKeyId, secretAccessKey } " +
|
|
143
|
+
"(IAM role or env-supplied STS credentials)");
|
|
144
|
+
}
|
|
145
|
+
if (!config.logGroupName || !config.logStreamName) {
|
|
146
|
+
throw _err("BAD_OPT",
|
|
147
|
+
"log-stream cloudwatch requires { logGroupName, logStreamName } " +
|
|
148
|
+
"(operator pre-creates both via aws logs create-log-group / create-log-stream " +
|
|
149
|
+
"or CDK / Terraform; the framework does NOT auto-create)");
|
|
150
|
+
}
|
|
151
|
+
var cfg = Object.assign({}, DEFAULTS, config);
|
|
152
|
+
var onDrop = typeof cfg.onDrop === "function" ? cfg.onDrop : null;
|
|
153
|
+
function _emitDrop(reason, batch, err) {
|
|
154
|
+
if (!onDrop) return;
|
|
155
|
+
try { onDrop({ reason: reason, batch: batch, error: err || null }); }
|
|
156
|
+
catch (_e) { /* best-effort */ }
|
|
157
|
+
}
|
|
158
|
+
var buffer = [];
|
|
159
|
+
var dropCount = 0;
|
|
160
|
+
var flushTimer = null;
|
|
161
|
+
var inFlight = false;
|
|
162
|
+
var closed = false;
|
|
163
|
+
var sequenceToken = null;
|
|
164
|
+
|
|
165
|
+
function _scheduleFlush() {
|
|
166
|
+
if (flushTimer) return;
|
|
167
|
+
flushTimer = setTimeout(function () { flushTimer = null; _flush(); }, cfg.maxBatchAgeMs);
|
|
168
|
+
flushTimer.unref();
|
|
169
|
+
}
|
|
170
|
+
|
|
171
|
+
function _takeBatch() {
|
|
172
|
+
var batch = [];
|
|
173
|
+
var totalBytes = 0;
|
|
174
|
+
while (buffer.length > 0) {
|
|
175
|
+
var nextEvent = buffer[0];
|
|
176
|
+
var size = _eventByteSize(nextEvent.message);
|
|
177
|
+
if (batch.length > 0 &&
|
|
178
|
+
(batch.length >= cfg.batchSize ||
|
|
179
|
+
batch.length >= CW_MAX_EVENTS_PER_BATCH ||
|
|
180
|
+
totalBytes + size > CW_MAX_BATCH_BYTES)) {
|
|
181
|
+
break;
|
|
182
|
+
}
|
|
183
|
+
batch.push(buffer.shift());
|
|
184
|
+
totalBytes += size;
|
|
185
|
+
}
|
|
186
|
+
return batch;
|
|
187
|
+
}
|
|
188
|
+
|
|
189
|
+
async function _flush() {
|
|
190
|
+
if (inFlight) return;
|
|
191
|
+
if (buffer.length === 0) return;
|
|
192
|
+
inFlight = true;
|
|
193
|
+
try {
|
|
194
|
+
while (buffer.length > 0 && !closed) {
|
|
195
|
+
var batch = _takeBatch();
|
|
196
|
+
if (batch.length === 0) break;
|
|
197
|
+
try {
|
|
198
|
+
await retryHelper.withRetry(function () {
|
|
199
|
+
return _send(batch);
|
|
200
|
+
}, Object.assign({
|
|
201
|
+
isPermanent: _isPermanentAwsError,
|
|
202
|
+
}, cfg.retry || {}));
|
|
203
|
+
} catch (e) {
|
|
204
|
+
dropCount += batch.length;
|
|
205
|
+
_emitDrop("retry-exhausted", batch, e);
|
|
206
|
+
break;
|
|
207
|
+
}
|
|
208
|
+
}
|
|
209
|
+
} finally {
|
|
210
|
+
inFlight = false;
|
|
211
|
+
if (buffer.length > 0) _scheduleFlush();
|
|
212
|
+
}
|
|
213
|
+
}
|
|
214
|
+
|
|
215
|
+
async function _send(batch) {
|
|
216
|
+
var body = _serializeBatch(batch, cfg, sequenceToken);
|
|
217
|
+
var headers = _signedHeaders(cfg, body);
|
|
218
|
+
var res;
|
|
219
|
+
try {
|
|
220
|
+
res = await _post(cfg, body, headers);
|
|
221
|
+
} catch (e) {
|
|
222
|
+
var match = /expected sequenceToken is:\s*(\S+)/.exec(e.message || "");
|
|
223
|
+
if (match) {
|
|
224
|
+
sequenceToken = match[1];
|
|
225
|
+
var retryBody = _serializeBatch(batch, cfg, sequenceToken);
|
|
226
|
+
var retryHeaders = _signedHeaders(cfg, retryBody);
|
|
227
|
+
res = await _post(cfg, retryBody, retryHeaders);
|
|
228
|
+
} else {
|
|
229
|
+
throw e;
|
|
230
|
+
}
|
|
231
|
+
}
|
|
232
|
+
if (res && res.body) {
|
|
233
|
+
try {
|
|
234
|
+
var parsed = JSON.parse(res.body.toString("utf8"));
|
|
235
|
+
if (parsed && parsed.nextSequenceToken) sequenceToken = parsed.nextSequenceToken;
|
|
236
|
+
} catch (_e) { /* response body not JSON; modern CW returns empty */ }
|
|
237
|
+
}
|
|
238
|
+
return res;
|
|
239
|
+
}
|
|
240
|
+
|
|
241
|
+
function emit(record) {
|
|
242
|
+
if (closed) return Promise.resolve({ accepted: false, reason: "sink closed" });
|
|
243
|
+
var message;
|
|
244
|
+
if (typeof record.message === "string") {
|
|
245
|
+
message = record.message;
|
|
246
|
+
} else {
|
|
247
|
+
message = JSON.stringify(record);
|
|
248
|
+
}
|
|
249
|
+
var size = _eventByteSize(message);
|
|
250
|
+
if (size > CW_MAX_EVENT_BYTES) {
|
|
251
|
+
_emitDrop("event-too-large", [{
|
|
252
|
+
timestamp: record.ts || Date.now(),
|
|
253
|
+
message: message.slice(0, 200) + "...[truncated for drop event]",
|
|
254
|
+
}], new Error("event exceeds 256 KiB CloudWatch hard cap (was " + size + " bytes)"));
|
|
255
|
+
dropCount += 1;
|
|
256
|
+
return Promise.resolve({ accepted: false, reason: "event too large" });
|
|
257
|
+
}
|
|
258
|
+
if (buffer.length >= cfg.bufferLimit) {
|
|
259
|
+
var dropped = buffer.shift();
|
|
260
|
+
dropCount += 1;
|
|
261
|
+
_emitDrop("overflow", [dropped], null);
|
|
262
|
+
}
|
|
263
|
+
buffer.push({
|
|
264
|
+
timestamp: record.ts || Date.now(),
|
|
265
|
+
message: message,
|
|
266
|
+
});
|
|
267
|
+
if (buffer.length >= cfg.batchSize) {
|
|
268
|
+
_flush().catch(function () {});
|
|
269
|
+
} else {
|
|
270
|
+
_scheduleFlush();
|
|
271
|
+
}
|
|
272
|
+
return Promise.resolve({ accepted: true, queued: buffer.length });
|
|
273
|
+
}
|
|
274
|
+
|
|
275
|
+
async function close() {
|
|
276
|
+
closed = true;
|
|
277
|
+
if (flushTimer) { clearTimeout(flushTimer); flushTimer = null; }
|
|
278
|
+
await _flush();
|
|
279
|
+
}
|
|
280
|
+
|
|
281
|
+
function stats() {
|
|
282
|
+
return {
|
|
283
|
+
queued: buffer.length,
|
|
284
|
+
dropped: dropCount,
|
|
285
|
+
inFlight: inFlight,
|
|
286
|
+
sequenceToken: sequenceToken,
|
|
287
|
+
endpoint: _resolveEndpoint(cfg),
|
|
288
|
+
};
|
|
289
|
+
}
|
|
290
|
+
|
|
291
|
+
return {
|
|
292
|
+
protocol: "cloudwatch",
|
|
293
|
+
emit: emit,
|
|
294
|
+
close: close,
|
|
295
|
+
stats: stats,
|
|
296
|
+
flush: _flush,
|
|
297
|
+
};
|
|
298
|
+
}
|
|
299
|
+
|
|
300
|
+
module.exports = {
|
|
301
|
+
create: create,
|
|
302
|
+
_resolveEndpoint: _resolveEndpoint,
|
|
303
|
+
_eventByteSize: _eventByteSize,
|
|
304
|
+
_serializeBatch: _serializeBatch,
|
|
305
|
+
_isPermanentAwsError: _isPermanentAwsError,
|
|
306
|
+
CW_MAX_EVENTS_PER_BATCH: CW_MAX_EVENTS_PER_BATCH,
|
|
307
|
+
CW_MAX_BATCH_BYTES: CW_MAX_BATCH_BYTES,
|
|
308
|
+
CW_MAX_EVENT_BYTES: CW_MAX_EVENT_BYTES,
|
|
309
|
+
};
|
|
@@ -0,0 +1,292 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
/**
|
|
3
|
+
* OTLP/HTTP-JSON log sink — OpenTelemetry Protocol logs over HTTP/JSON.
|
|
4
|
+
*
|
|
5
|
+
* Per the OTLP spec (v1.3+, Logs Data Model), log records are wrapped in
|
|
6
|
+
* a ResourceLogs envelope and POSTed to the operator's collector
|
|
7
|
+
* endpoint. Marshalling is OTel-native: timeUnixNano (string), severity
|
|
8
|
+
* mapping per OTel spec, attribute typing.
|
|
9
|
+
*
|
|
10
|
+
* Operator config:
|
|
11
|
+
*
|
|
12
|
+
* {
|
|
13
|
+
* url: "https://otel-collector.example.com:4318/v1/logs"
|
|
14
|
+
* // OR the collector root — "/v1/logs" auto-appended
|
|
15
|
+
* headers: { ... } // additional headers (api keys, tenant ids)
|
|
16
|
+
* auth: 'none' | 'bearer' | 'basic' | 'header'
|
|
17
|
+
* token / username+password
|
|
18
|
+
* serviceName: "my-service" // emitted as resource service.name
|
|
19
|
+
* serviceVersion: "1.2.3" // resource service.version
|
|
20
|
+
* resourceAttributes: { env: "prod", region: "us-east-1" }
|
|
21
|
+
* // additional resource attributes
|
|
22
|
+
* scopeName: "blamejs" // instrumentation scope name
|
|
23
|
+
* batchSize: 100
|
|
24
|
+
* maxBatchAgeMs: C.TIME.seconds(5)
|
|
25
|
+
* timeoutMs: C.TIME.seconds(30)
|
|
26
|
+
* retry: { maxAttempts, baseDelayMs, ... }
|
|
27
|
+
* bufferLimit: 10000 // ring-buffer cap; drops oldest on overflow
|
|
28
|
+
* onDrop: function ({ reason, batch, error }) { ... }
|
|
29
|
+
* }
|
|
30
|
+
*
|
|
31
|
+
* Severity mapping (per OTel Logs Data Model):
|
|
32
|
+
* debug → 5 (DEBUG)
|
|
33
|
+
* info → 9 (INFO)
|
|
34
|
+
* warn → 13 (WARN)
|
|
35
|
+
* error → 17 (ERROR)
|
|
36
|
+
*
|
|
37
|
+
* Endpoint convention: most OTel collectors expose:
|
|
38
|
+
* <base>:4318/v1/logs — HTTP/JSON path (per OTLP HTTP §4.5)
|
|
39
|
+
* <base>:4317 — gRPC path (NOT supported here)
|
|
40
|
+
*
|
|
41
|
+
* If `url` ends in `/v1/logs` it's used as-is. If not, `/v1/logs` is
|
|
42
|
+
* appended (collector root → logs endpoint). Operators with a custom
|
|
43
|
+
* routing prefix pass the full URL.
|
|
44
|
+
*
|
|
45
|
+
* Why JSON not protobuf: protobuf needs a schema parser (vendored dep
|
|
46
|
+
* or hand-rolled). The OTLP/HTTP spec accepts both Content-Type:
|
|
47
|
+
* application/json and application/x-protobuf; the JSON variant is the
|
|
48
|
+
* "no extra deps" path consistent with the framework's vendoring stance.
|
|
49
|
+
* Operators benchmarking >100K logs/s ship the OTel Collector locally
|
|
50
|
+
* (collector → upstream gRPC) — this sink hands JSON to the local
|
|
51
|
+
* collector which forwards.
|
|
52
|
+
*/
|
|
53
|
+
var C = require("./constants");
|
|
54
|
+
var retryHelper = require("./retry");
|
|
55
|
+
var { LogStreamError } = require("./framework-error");
|
|
56
|
+
var httpClient = require("./http-client");
|
|
57
|
+
var safeUrl = require("./safe-url");
|
|
58
|
+
var authHeader = require("./auth-header");
|
|
59
|
+
|
|
60
|
+
var MAX_RESPONSE_BYTES = C.BYTES.mib(1);
|
|
61
|
+
|
|
62
|
+
// OTel Logs Data Model severity numbers.
|
|
63
|
+
// https://opentelemetry.io/docs/specs/otel/logs/data-model/#field-severitynumber
|
|
64
|
+
var SEVERITY = {
|
|
65
|
+
debug: { number: 5, text: "DEBUG" },
|
|
66
|
+
info: { number: 9, text: "INFO" },
|
|
67
|
+
warn: { number: 13, text: "WARN" },
|
|
68
|
+
error: { number: 17, text: "ERROR" },
|
|
69
|
+
};
|
|
70
|
+
|
|
71
|
+
var DEFAULTS = {
|
|
72
|
+
scopeName: "blamejs",
|
|
73
|
+
batchSize: 100,
|
|
74
|
+
maxBatchAgeMs: C.TIME.seconds(5),
|
|
75
|
+
timeoutMs: C.TIME.seconds(30),
|
|
76
|
+
bufferLimit: 10000,
|
|
77
|
+
};
|
|
78
|
+
|
|
79
|
+
var _err = LogStreamError.factory;
|
|
80
|
+
|
|
81
|
+
function _resolveUrl(url) {
|
|
82
|
+
// If the URL already ends in "/v1/logs" (with or without trailing slash),
|
|
83
|
+
// use as-is. Otherwise treat as the collector root and append.
|
|
84
|
+
var trimmed = url.replace(/\/+$/, "");
|
|
85
|
+
if (/\/v1\/logs$/.test(trimmed)) return trimmed;
|
|
86
|
+
return trimmed + "/v1/logs";
|
|
87
|
+
}
|
|
88
|
+
|
|
89
|
+
function _authHeaders(config) {
|
|
90
|
+
if (config.auth === "header") return Object.assign({}, config.headers || {});
|
|
91
|
+
return authHeader.fromConfig(config);
|
|
92
|
+
}
|
|
93
|
+
|
|
94
|
+
// OTel attribute encoding — JSON form per OTLP spec. Each attribute is
|
|
95
|
+
// { key: <name>, value: { <type>Value: <v> } } where the type field
|
|
96
|
+
// matches the JS type at the boundary.
|
|
97
|
+
function _encodeAttrValue(v) {
|
|
98
|
+
if (v === null || v === undefined) return null;
|
|
99
|
+
var t = typeof v;
|
|
100
|
+
if (t === "string") return { stringValue: v };
|
|
101
|
+
if (t === "boolean") return { boolValue: v };
|
|
102
|
+
if (t === "number") {
|
|
103
|
+
if (Number.isInteger(v)) return { intValue: String(v) }; // OTLP int is string-encoded
|
|
104
|
+
return { doubleValue: v };
|
|
105
|
+
}
|
|
106
|
+
if (Array.isArray(v)) {
|
|
107
|
+
return { arrayValue: { values: v.map(function (e) { return _encodeAttrValue(e); }).filter(Boolean) } };
|
|
108
|
+
}
|
|
109
|
+
if (t === "object") {
|
|
110
|
+
return { kvlistValue: { values: _encodeAttrs(v) } };
|
|
111
|
+
}
|
|
112
|
+
return { stringValue: String(v) };
|
|
113
|
+
}
|
|
114
|
+
|
|
115
|
+
function _encodeAttrs(obj) {
|
|
116
|
+
if (!obj || typeof obj !== "object") return [];
|
|
117
|
+
return Object.keys(obj).map(function (k) {
|
|
118
|
+
var encoded = _encodeAttrValue(obj[k]);
|
|
119
|
+
if (!encoded) return null;
|
|
120
|
+
return { key: k, value: encoded };
|
|
121
|
+
}).filter(Boolean);
|
|
122
|
+
}
|
|
123
|
+
|
|
124
|
+
function _resourceAttrs(cfg) {
|
|
125
|
+
var attrs = {};
|
|
126
|
+
if (cfg.serviceName) attrs["service.name"] = cfg.serviceName;
|
|
127
|
+
if (cfg.serviceVersion) attrs["service.version"] = cfg.serviceVersion;
|
|
128
|
+
if (cfg.resourceAttributes && typeof cfg.resourceAttributes === "object") {
|
|
129
|
+
Object.assign(attrs, cfg.resourceAttributes);
|
|
130
|
+
}
|
|
131
|
+
return attrs;
|
|
132
|
+
}
|
|
133
|
+
|
|
134
|
+
function _toLogRecord(record) {
|
|
135
|
+
var sev = SEVERITY[record.level] || SEVERITY.info;
|
|
136
|
+
// OTel timeUnixNano is a string (JSON can't safely represent 64-bit ints).
|
|
137
|
+
var nanos = String(BigInt(record.ts) * 1000000n);
|
|
138
|
+
var attrs = record.meta ? _encodeAttrs(record.meta) : [];
|
|
139
|
+
return {
|
|
140
|
+
timeUnixNano: nanos,
|
|
141
|
+
observedTimeUnixNano: nanos,
|
|
142
|
+
severityNumber: sev.number,
|
|
143
|
+
severityText: sev.text,
|
|
144
|
+
body: { stringValue: record.message == null ? "" : String(record.message) },
|
|
145
|
+
attributes: attrs,
|
|
146
|
+
};
|
|
147
|
+
}
|
|
148
|
+
|
|
149
|
+
function _serializeBatch(records, cfg, scopeVersion) {
|
|
150
|
+
var resourceAttrs = _resourceAttrs(cfg);
|
|
151
|
+
return Buffer.from(JSON.stringify({
|
|
152
|
+
resourceLogs: [
|
|
153
|
+
{
|
|
154
|
+
resource: {
|
|
155
|
+
attributes: _encodeAttrs(resourceAttrs),
|
|
156
|
+
},
|
|
157
|
+
scopeLogs: [
|
|
158
|
+
{
|
|
159
|
+
scope: {
|
|
160
|
+
name: cfg.scopeName,
|
|
161
|
+
version: scopeVersion,
|
|
162
|
+
},
|
|
163
|
+
logRecords: records.map(_toLogRecord),
|
|
164
|
+
},
|
|
165
|
+
],
|
|
166
|
+
},
|
|
167
|
+
],
|
|
168
|
+
}), "utf8");
|
|
169
|
+
}
|
|
170
|
+
|
|
171
|
+
function _post(url, body, headers, timeoutMs, allowedProtocols, allowInternal) {
|
|
172
|
+
return httpClient.request({
|
|
173
|
+
method: "POST",
|
|
174
|
+
url: url,
|
|
175
|
+
headers: headers,
|
|
176
|
+
body: body,
|
|
177
|
+
idleTimeoutMs: timeoutMs,
|
|
178
|
+
maxResponseBytes: MAX_RESPONSE_BYTES,
|
|
179
|
+
errorClass: LogStreamError,
|
|
180
|
+
allowedProtocols: allowedProtocols,
|
|
181
|
+
allowInternal: allowInternal,
|
|
182
|
+
});
|
|
183
|
+
}
|
|
184
|
+
|
|
185
|
+
function create(config) {
|
|
186
|
+
if (!config || !config.url) throw _err("BAD_OPT", "log-stream otlp requires { url }");
|
|
187
|
+
var cfg = Object.assign({}, DEFAULTS, config);
|
|
188
|
+
var resolvedUrl = _resolveUrl(cfg.url);
|
|
189
|
+
// Validate URL shape + scheme at create time. Default HTTPS-only;
|
|
190
|
+
// operators with an internal cleartext OTel collector pass
|
|
191
|
+
// cfg.allowedProtocols (safeUrl.ALLOW_HTTP_ALL).
|
|
192
|
+
safeUrl.parse(resolvedUrl, {
|
|
193
|
+
allowedProtocols: cfg.allowedProtocols || safeUrl.ALLOW_HTTP_TLS,
|
|
194
|
+
errorClass: LogStreamError,
|
|
195
|
+
});
|
|
196
|
+
var scopeVersion = cfg.scopeVersion || (function () {
|
|
197
|
+
try { return require("../package.json").version; } catch (_e) { return "unknown"; }
|
|
198
|
+
})();
|
|
199
|
+
var headers = Object.assign({
|
|
200
|
+
"Content-Type": "application/json",
|
|
201
|
+
"Accept": "application/json",
|
|
202
|
+
}, _authHeaders(cfg));
|
|
203
|
+
var onDrop = typeof cfg.onDrop === "function" ? cfg.onDrop : null;
|
|
204
|
+
function _emitDrop(reason, batch, err) {
|
|
205
|
+
if (!onDrop) return;
|
|
206
|
+
try { onDrop({ reason: reason, batch: batch, error: err || null }); }
|
|
207
|
+
catch (_e) { /* drop callback is best-effort */ }
|
|
208
|
+
}
|
|
209
|
+
var buffer = [];
|
|
210
|
+
var dropCount = 0;
|
|
211
|
+
var flushTimer = null;
|
|
212
|
+
var inFlight = false;
|
|
213
|
+
var closed = false;
|
|
214
|
+
|
|
215
|
+
function _scheduleFlush() {
|
|
216
|
+
if (flushTimer) return;
|
|
217
|
+
flushTimer = setTimeout(function () { flushTimer = null; _flush(); }, cfg.maxBatchAgeMs);
|
|
218
|
+
flushTimer.unref();
|
|
219
|
+
}
|
|
220
|
+
|
|
221
|
+
async function _flush() {
|
|
222
|
+
if (inFlight) return;
|
|
223
|
+
if (buffer.length === 0) return;
|
|
224
|
+
inFlight = true;
|
|
225
|
+
try {
|
|
226
|
+
while (buffer.length > 0 && !closed) {
|
|
227
|
+
var batch = buffer.splice(0, cfg.batchSize);
|
|
228
|
+
var body = _serializeBatch(batch, cfg, scopeVersion);
|
|
229
|
+
try {
|
|
230
|
+
await retryHelper.withRetry(function () {
|
|
231
|
+
return _post(resolvedUrl, body, headers, cfg.timeoutMs, cfg.allowedProtocols, cfg.allowInternal);
|
|
232
|
+
}, cfg.retry);
|
|
233
|
+
} catch (e) {
|
|
234
|
+
dropCount += batch.length;
|
|
235
|
+
_emitDrop("retry-exhausted", batch, e);
|
|
236
|
+
break;
|
|
237
|
+
}
|
|
238
|
+
}
|
|
239
|
+
} finally {
|
|
240
|
+
inFlight = false;
|
|
241
|
+
if (buffer.length > 0) _scheduleFlush();
|
|
242
|
+
}
|
|
243
|
+
}
|
|
244
|
+
|
|
245
|
+
function emit(record) {
|
|
246
|
+
if (closed) return Promise.resolve({ accepted: false, reason: "sink closed" });
|
|
247
|
+
if (buffer.length >= cfg.bufferLimit) {
|
|
248
|
+
var dropped = buffer.shift();
|
|
249
|
+
dropCount += 1;
|
|
250
|
+
_emitDrop("overflow", [dropped], null);
|
|
251
|
+
}
|
|
252
|
+
buffer.push(record);
|
|
253
|
+
if (buffer.length >= cfg.batchSize) {
|
|
254
|
+
_flush().catch(function () {});
|
|
255
|
+
} else {
|
|
256
|
+
_scheduleFlush();
|
|
257
|
+
}
|
|
258
|
+
return Promise.resolve({ accepted: true, queued: buffer.length });
|
|
259
|
+
}
|
|
260
|
+
|
|
261
|
+
async function close() {
|
|
262
|
+
closed = true;
|
|
263
|
+
if (flushTimer) { clearTimeout(flushTimer); flushTimer = null; }
|
|
264
|
+
await _flush();
|
|
265
|
+
}
|
|
266
|
+
|
|
267
|
+
function stats() {
|
|
268
|
+
return {
|
|
269
|
+
queued: buffer.length,
|
|
270
|
+
dropped: dropCount,
|
|
271
|
+
inFlight: inFlight,
|
|
272
|
+
url: resolvedUrl,
|
|
273
|
+
};
|
|
274
|
+
}
|
|
275
|
+
|
|
276
|
+
return {
|
|
277
|
+
protocol: "otlp",
|
|
278
|
+
emit: emit,
|
|
279
|
+
close: close,
|
|
280
|
+
stats: stats,
|
|
281
|
+
flush: _flush,
|
|
282
|
+
};
|
|
283
|
+
}
|
|
284
|
+
|
|
285
|
+
module.exports = {
|
|
286
|
+
create: create,
|
|
287
|
+
// Exposed for tests + advanced operator wiring.
|
|
288
|
+
_resolveUrl: _resolveUrl,
|
|
289
|
+
_encodeAttrs: _encodeAttrs,
|
|
290
|
+
_toLogRecord: _toLogRecord,
|
|
291
|
+
SEVERITY: SEVERITY,
|
|
292
|
+
};
|
package/lib/log-stream.js
CHANGED
|
@@ -4,12 +4,26 @@
|
|
|
4
4
|
* redaction and bidirectional command channel.
|
|
5
5
|
*
|
|
6
6
|
* Same dispatcher pattern: backends configured per-name with a protocol
|
|
7
|
-
* + protocol-specific options. Built-in protocols
|
|
8
|
-
*
|
|
9
|
-
*
|
|
10
|
-
*
|
|
11
|
-
*
|
|
12
|
-
*
|
|
7
|
+
* + protocol-specific options. Built-in protocols:
|
|
8
|
+
*
|
|
9
|
+
* local — append-only file with rotation
|
|
10
|
+
* webhook — generic HTTP POST; covers Splunk HEC, Datadog, Sumo
|
|
11
|
+
* Logic, Loki, custom collectors that ingest JSON
|
|
12
|
+
* otlp — OpenTelemetry Protocol over HTTP/JSON; ResourceLogs
|
|
13
|
+
* envelope with severity mapping per OTel Logs Data
|
|
14
|
+
* Model. Operators with an OTel collector running
|
|
15
|
+
* (k8s, cloud) get standard log forwarding without a
|
|
16
|
+
* vendor-specific adapter.
|
|
17
|
+
* cloudwatch — AWS CloudWatch Logs (PutLogEvents) over HTTPS with
|
|
18
|
+
* SigV4. Operator pre-creates the log group + log stream;
|
|
19
|
+
* the framework signs and POSTs batches respecting the
|
|
20
|
+
* 10K-event / 1 MiB / 256 KiB-per-event AWS caps. Honors
|
|
21
|
+
* IAM role + STS session tokens.
|
|
22
|
+
*
|
|
23
|
+
* Adapters listed as deferred and surfacing a clear error when
|
|
24
|
+
* selected:
|
|
25
|
+
*
|
|
26
|
+
* syslog — RFC 5424 syslog over TLS
|
|
13
27
|
*
|
|
14
28
|
* Every emit goes through lib/redact.js BEFORE any sink sees it. PHI/PCI
|
|
15
29
|
* never reaches operational logs even on a misconfigured field name —
|
|
@@ -32,21 +46,26 @@
|
|
|
32
46
|
* logStream.shutdown()
|
|
33
47
|
* logStream.listSinks() → [{ name, protocol, stats }]
|
|
34
48
|
*/
|
|
35
|
-
var localProto
|
|
36
|
-
var webhookProto
|
|
37
|
-
var
|
|
38
|
-
var
|
|
49
|
+
var localProto = require("./log-stream-local");
|
|
50
|
+
var webhookProto = require("./log-stream-webhook");
|
|
51
|
+
var otlpProto = require("./log-stream-otlp");
|
|
52
|
+
var cloudwatchProto = require("./log-stream-cloudwatch");
|
|
53
|
+
var redactor = require("./redact");
|
|
54
|
+
var lazyRequire = require("./lazy-require");
|
|
39
55
|
var protocolDispatcher = require("./protocol-dispatcher");
|
|
40
56
|
var { LogStreamError } = require("./framework-error");
|
|
41
57
|
|
|
42
58
|
var dispatcher = protocolDispatcher.create({
|
|
43
59
|
name: "log-stream",
|
|
44
60
|
errorClass: LogStreamError,
|
|
45
|
-
protocols: {
|
|
61
|
+
protocols: {
|
|
62
|
+
"local": localProto,
|
|
63
|
+
"webhook": webhookProto,
|
|
64
|
+
"otlp": otlpProto,
|
|
65
|
+
"cloudwatch": cloudwatchProto,
|
|
66
|
+
},
|
|
46
67
|
deferred: {
|
|
47
68
|
"syslog": { description: "RFC 5424 syslog over TLS" },
|
|
48
|
-
"otlp": { description: "OpenTelemetry Logs OTLP/HTTP" },
|
|
49
|
-
"cloudwatch": { description: "AWS CloudWatch Logs (PutLogEvents)" },
|
|
50
69
|
},
|
|
51
70
|
fallbackProtocol: "local",
|
|
52
71
|
});
|
|
@@ -183,6 +202,71 @@ function listSinks() {
|
|
|
183
202
|
});
|
|
184
203
|
}
|
|
185
204
|
|
|
205
|
+
// ---- bootFromEnv ----
|
|
206
|
+
//
|
|
207
|
+
// Operator-friendly env-driven init that mirrors b.network.bootFromEnv.
|
|
208
|
+
// Reads BLAMEJS_LOG_STREAM_* env vars and constructs a single-sink
|
|
209
|
+
// configuration matching the operator's choice. Skipped silently when
|
|
210
|
+
// BLAMEJS_LOG_STREAM_PROTOCOL isn't set (operators using the in-code
|
|
211
|
+
// init() path keep their existing wiring).
|
|
212
|
+
//
|
|
213
|
+
// Recognised env vars:
|
|
214
|
+
// BLAMEJS_LOG_STREAM_PROTOCOL "local" | "webhook" | "otlp" | "cloudwatch"
|
|
215
|
+
// BLAMEJS_LOG_STREAM_MIN_LEVEL "debug" | "info" | "warn" | "error"
|
|
216
|
+
//
|
|
217
|
+
// webhook + otlp shared:
|
|
218
|
+
// BLAMEJS_LOG_STREAM_URL
|
|
219
|
+
// BLAMEJS_LOG_STREAM_TOKEN (auth: bearer)
|
|
220
|
+
// otlp-only:
|
|
221
|
+
// BLAMEJS_LOG_STREAM_SERVICE_NAME
|
|
222
|
+
// BLAMEJS_LOG_STREAM_SERVICE_VERSION
|
|
223
|
+
// cloudwatch-only (AWS_* are standard):
|
|
224
|
+
// AWS_REGION, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN
|
|
225
|
+
// BLAMEJS_LOG_STREAM_CLOUDWATCH_LOG_GROUP
|
|
226
|
+
// BLAMEJS_LOG_STREAM_CLOUDWATCH_LOG_STREAM
|
|
227
|
+
// local-only:
|
|
228
|
+
// BLAMEJS_LOG_STREAM_PATH
|
|
229
|
+
function bootFromEnv(opts) {
|
|
230
|
+
opts = opts || {};
|
|
231
|
+
var env = opts.env || process.env;
|
|
232
|
+
var proto = env.BLAMEJS_LOG_STREAM_PROTOCOL;
|
|
233
|
+
if (!proto) return false;
|
|
234
|
+
var sink = { protocol: proto };
|
|
235
|
+
if (proto === "webhook") {
|
|
236
|
+
sink.url = env.BLAMEJS_LOG_STREAM_URL;
|
|
237
|
+
if (env.BLAMEJS_LOG_STREAM_TOKEN) {
|
|
238
|
+
sink.auth = "bearer";
|
|
239
|
+
sink.token = env.BLAMEJS_LOG_STREAM_TOKEN;
|
|
240
|
+
}
|
|
241
|
+
} else if (proto === "otlp") {
|
|
242
|
+
sink.url = env.BLAMEJS_LOG_STREAM_URL;
|
|
243
|
+
sink.serviceName = env.BLAMEJS_LOG_STREAM_SERVICE_NAME || "blamejs";
|
|
244
|
+
sink.serviceVersion = env.BLAMEJS_LOG_STREAM_SERVICE_VERSION || null;
|
|
245
|
+
if (env.BLAMEJS_LOG_STREAM_TOKEN) {
|
|
246
|
+
sink.auth = "bearer";
|
|
247
|
+
sink.token = env.BLAMEJS_LOG_STREAM_TOKEN;
|
|
248
|
+
}
|
|
249
|
+
} else if (proto === "cloudwatch") {
|
|
250
|
+
sink.region = env.AWS_REGION;
|
|
251
|
+
sink.accessKeyId = env.AWS_ACCESS_KEY_ID;
|
|
252
|
+
sink.secretAccessKey = env.AWS_SECRET_ACCESS_KEY;
|
|
253
|
+
sink.sessionToken = env.AWS_SESSION_TOKEN || null;
|
|
254
|
+
sink.logGroupName = env.BLAMEJS_LOG_STREAM_CLOUDWATCH_LOG_GROUP;
|
|
255
|
+
sink.logStreamName = env.BLAMEJS_LOG_STREAM_CLOUDWATCH_LOG_STREAM;
|
|
256
|
+
} else if (proto === "local") {
|
|
257
|
+
sink.path = env.BLAMEJS_LOG_STREAM_PATH;
|
|
258
|
+
} else {
|
|
259
|
+
throw _err("BAD_OPT",
|
|
260
|
+
"BLAMEJS_LOG_STREAM_PROTOCOL='" + proto + "' is not one of " +
|
|
261
|
+
"local | webhook | otlp | cloudwatch (or a custom backend wired via init())");
|
|
262
|
+
}
|
|
263
|
+
init({
|
|
264
|
+
sinks: { primary: sink },
|
|
265
|
+
minLevel: env.BLAMEJS_LOG_STREAM_MIN_LEVEL || undefined,
|
|
266
|
+
});
|
|
267
|
+
return true;
|
|
268
|
+
}
|
|
269
|
+
|
|
186
270
|
function _resetForTest() {
|
|
187
271
|
Object.keys(sinks).forEach(function (n) {
|
|
188
272
|
try { if (sinks[n].raw.close) sinks[n].raw.close(); } catch (_e) {}
|
|
@@ -195,6 +279,7 @@ function _resetForTest() {
|
|
|
195
279
|
|
|
196
280
|
module.exports = {
|
|
197
281
|
init: init,
|
|
282
|
+
bootFromEnv: bootFromEnv,
|
|
198
283
|
emit: emit,
|
|
199
284
|
debug: debug,
|
|
200
285
|
info: info,
|
|
@@ -165,6 +165,12 @@ function signRequest(opts) {
|
|
|
165
165
|
var amzDate = _formatAmzDate(date);
|
|
166
166
|
var dateStamp = _formatDateStamp(date);
|
|
167
167
|
var url = opts.url instanceof URL ? opts.url : new URL(opts.url);
|
|
168
|
+
// service defaults to s3 for back-compat — every call site predating
|
|
169
|
+
// v0.6.25 was object-store / S3. Other AWS services (logs, sqs, sns,
|
|
170
|
+
// kinesis, etc.) pass opts.service explicitly. The credentialScope
|
|
171
|
+
// and signing-key derivation both incorporate the service name, so
|
|
172
|
+
// this MUST match what the target service expects.
|
|
173
|
+
var service = opts.service || SERVICE;
|
|
168
174
|
|
|
169
175
|
var headers = Object.assign({}, opts.headers || {});
|
|
170
176
|
headers["host"] = url.host;
|
|
@@ -177,9 +183,9 @@ function signRequest(opts) {
|
|
|
177
183
|
}
|
|
178
184
|
|
|
179
185
|
var canon = canonicalRequest(opts.method, url, headers, opts.payloadHash);
|
|
180
|
-
var credentialScope = dateStamp + "/" + opts.region + "/" +
|
|
186
|
+
var credentialScope = dateStamp + "/" + opts.region + "/" + service + "/aws4_request";
|
|
181
187
|
var sts = stringToSign(amzDate, credentialScope, canon);
|
|
182
|
-
var signingKey = deriveSigningKey(opts.secretAccessKey, dateStamp, opts.region,
|
|
188
|
+
var signingKey = deriveSigningKey(opts.secretAccessKey, dateStamp, opts.region, service);
|
|
183
189
|
var signature = nodeCrypto.createHmac("sha256", signingKey).update(sts).digest("hex");
|
|
184
190
|
|
|
185
191
|
var canonHeaders = canonicalHeaders(headers);
|
package/package.json
CHANGED
package/sbom.cyclonedx.json
CHANGED
|
@@ -2,10 +2,10 @@
|
|
|
2
2
|
"$schema": "http://cyclonedx.org/schema/bom-1.5.schema.json",
|
|
3
3
|
"bomFormat": "CycloneDX",
|
|
4
4
|
"specVersion": "1.5",
|
|
5
|
-
"serialNumber": "urn:uuid:
|
|
5
|
+
"serialNumber": "urn:uuid:c8d4ecf3-0360-4291-bedc-af54ef2eca2a",
|
|
6
6
|
"version": 1,
|
|
7
7
|
"metadata": {
|
|
8
|
-
"timestamp": "2026-05-02T14:
|
|
8
|
+
"timestamp": "2026-05-02T14:44:53.846Z",
|
|
9
9
|
"lifecycles": [
|
|
10
10
|
{
|
|
11
11
|
"phase": "build"
|
|
@@ -19,14 +19,14 @@
|
|
|
19
19
|
}
|
|
20
20
|
],
|
|
21
21
|
"component": {
|
|
22
|
-
"bom-ref": "@blamejs/core@0.6.
|
|
22
|
+
"bom-ref": "@blamejs/core@0.6.25",
|
|
23
23
|
"type": "library",
|
|
24
24
|
"name": "blamejs",
|
|
25
|
-
"version": "0.6.
|
|
25
|
+
"version": "0.6.25",
|
|
26
26
|
"scope": "required",
|
|
27
27
|
"author": "blamejs contributors",
|
|
28
28
|
"description": "The Node framework that owns its stack.",
|
|
29
|
-
"purl": "pkg:npm/%40blamejs/core@0.6.
|
|
29
|
+
"purl": "pkg:npm/%40blamejs/core@0.6.25",
|
|
30
30
|
"properties": [],
|
|
31
31
|
"externalReferences": [
|
|
32
32
|
{
|
|
@@ -54,7 +54,7 @@
|
|
|
54
54
|
"components": [],
|
|
55
55
|
"dependencies": [
|
|
56
56
|
{
|
|
57
|
-
"ref": "@blamejs/core@0.6.
|
|
57
|
+
"ref": "@blamejs/core@0.6.25",
|
|
58
58
|
"dependsOn": []
|
|
59
59
|
}
|
|
60
60
|
]
|