@fabbahiense/pulsar-pino-transport 0.2.1 → 0.4.0

package/CHANGELOG.md

@@ -0,0 +1,43 @@
+# Changelog
+
+All notable changes to `@fabbahiense/pulsar-pino-transport`. Versions before 0.4.0 were not documented here.
+
+## 0.4.0 (not yet published)
+
+Every 0.3.0 export and option keeps working. Since this is a 0.x minor, a `^0.3.0` range does not pick it up: update the range to `^0.4.0`.
+
+### Fixed
+
+- **`createPulsarTransport()` was unusable.** It pointed pino at `@pulsar/pino-transport`, a name that was never published, so pino could not resolve the target. It now uses `@fabbahiense/pulsar-pino-transport`.
+- **Logs buffered at shutdown were lost.** The final flush ran after the stream had already emitted `close`, and pino's worker exits on `close`. The flush now happens in the transport's `close` hook, which pino-abstract-transport awaits (bounded by `shutdownTimeout`).
+- **A slow or down Pulsar stalled pino.** The stream loop awaited each HTTP request (with retries) when a batch filled, so the worker stopped reading. Records are now buffered as they arrive and sent in the background.
+- **A 429 dropped logs.** Every 4xx was treated as permanent. `429`, `408`, `5xx`, network errors and timeouts are now retried with exponential backoff and jitter, honouring `Retry-After`, from a bounded buffer (`maxBufferSize`, drop oldest, counted). Requests time out, one is in flight at a time, `batchSize` is clamped to the server's 1000, a `413` splits the batch, and entries are capped at 256 KiB.
+- A non-string `msg` or `req.method` threw and ended the transport, losing every later log. Values are coerced; a custom `message` key is used as the message.
+- Level labels (`formatters.level`) were all sent as `info`; custom numeric levels too. Both are mapped now.
+- Epoch-seconds times (`stdTimeFunctions.unixTime`) became dates in 1970; they are converted.
+- `pulsarRequestLogger` kept every response chunk in memory (it captures every response); it now keeps at most `maxBodySize + 1` bytes. `Uint8Array` chunks, string encodings and circular request bodies are handled.
+
+### Added
+
+- Transport options: `maxBufferSize`, `maxRetries`, `requestTimeout`, `shutdownTimeout`, `warnOnDrop`. `createPulsarTransport()` passes them on.
+- Dropped logs (by reason: `overflow`, `rejected`, `retriesExhausted`, `shutdown`) produce one stderr line at most once a minute, unless `warnOnDrop: false`.
+
+### Security
+
+- `pulsarRequestLogger` captures request bodies on every request but only redacted header names, so a login body's `password` was logged in plain text; JSON response bodies were not redacted at all. The default list now also covers `password`, `passwd`, `password_hash`, `secret`, `client_secret`, `token`, `access_token`, `refresh_token`, `id_token`, `session_id`, `api_key` (any spelling) and `proxy-authorization`, and JSON response bodies are redacted too.
+- Redaction keys are now matched ignoring case, `-` and `_`: `access_token`, `access-token` and `accessToken` are the same key, for the defaults and for your `redactKeys`. Before, only the exact snake/kebab spelling matched, so camelCase fields such as a login response's `accessToken` were logged. Only whole keys match: `tokensUsed` or `passwordHint` are kept unless listed.
+
+### Packaging
+
+- **CommonJS TypeScript consumers** (`module: node16`/`nodenext`) got TS1479 because both conditions pointed at the ESM declarations. Each condition now has its own types (`.d.ts` / `.d.cts`). `/middleware` resolves under the legacy `node10` resolution too (`typesVersions`). `./package.json` is exported, `sideEffects` is `false`, and `prepublishOnly` builds and tests.
+- The middleware's JSDoc is now in English.
+
+### Distributed tracing
+
+Additive and opt-in: existing trace id fields keep working as before, nothing is patched, and no runtime dependency was added.
+
+- `pulsarMixin()`: a pino `mixin` that adds `traceId`, `spanId` and `parentSpanId` from the current trace context to every record, in the app's thread, so they reach the transport worker. One line: `pino({ mixin: pulsarMixin(), transport })`.
+- **The transport** sends `spanId`, `parentSpanId` and `spanKind` as top-level fields when they are non-empty strings (the server stores them as `metadata.spanId`, `metadata.parentSpanId` and `metadata.spanKind`, where they ended up before too). Other values stay in metadata.
+- **`pulsarRequestLogger()` runs each request in a span** and logs the request entry as its span-end entry (`spanId`, `parentSpanId`, `spanKind: "server"`). The trace is `req.id` when already set (as before), else the `x-trace-id` header, else a valid W3C `traceparent` (whose span becomes the parent), else a **new trace id**: requests without a trace id used to get none. A `req.id` set later (pino-http registered after it) still wins on the request entry. Skipped URLs are still not logged, but run in a span.
+- New exports from the root entry: `startSpan(name, fn, { logger })`, `instrumentFetch(fetch?, { logger? })`, `instrumentAxios(instance, { logger? })`, `traceHeaders`, `injectTrace`, `extractTrace`, `runWithTrace`, `getTraceContext`, `parseTraceparent`, `formatTraceparent`. The trace context store is shared with `@fabbahiense/pulsar-node` and `@fabbahiense/pulsar-next` in the same process.
+- `pino` 10.3.1 is a dev dependency, for the tests only.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Fabrício Bahiense Santos
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,168 +1,263 @@
 # @fabbahiense/pulsar-pino-transport
 
-Pino transport for [Pulsar](https://github.com/your-org/pulsar) — send your application logs to Pulsar in real-time.
+Pino transport + drop-in Express middleware for Pulsar observability. Send rich logs (method, URL, status, duration, request/response body, headers, query, params, custom context) with one line of middleware.
 
 ## Install
 
 ```bash
-npm install @fabbahiense/pulsar-pino-transport
+npm install @fabbahiense/pulsar-pino-transport pino-abstract-transport
 ```
 
-## Usage
+Requires Node.js 18 or later. `pino-abstract-transport` (v2 or later) is a peer dependency. Ships ESM and CommonJS builds with TypeScript types for both.
 
-```typescript
+## Quick Start — NestJS
+
+This is the recommended setup. Works with `nestjs-pino`.
+
+### 1. Logger config
+
+```ts
+// server/src/infra/logger.config.ts
 import pino from 'pino'
+import type { Params } from 'nestjs-pino'
 
-const logger = pino({
+export const pinoLogger = pino({
+  level: process.env.LOG_LEVEL || 'info',
   transport: {
-    target: '@fabbahiense/pulsar-pino-transport',
-    options: {
-      url: 'https://your-pulsar-instance.com',
-      apiKey: 'psk_your_api_key',
-      source: 'my-api',        // optional, default: 'default'
-      batchSize: 50,            // optional, default: 50
-      flushInterval: 2000,      // optional, default: 2000ms
-    },
+    targets: [
+      {
+        target: '@fabbahiense/pulsar-pino-transport',
+        level: 'info',
+        options: {
+          url: process.env.PULSAR_URL,
+          apiKey: process.env.PULSAR_API_KEY,
+          source: process.env.PULSAR_SOURCE,
+        },
+      },
+      // keep your other transports (pino-pretty in dev, pino/file in prod) here
+    ],
   },
 })
 
-logger.info('Server started')
-logger.error({ traceId: 'abc-123' }, 'Something went wrong')
+export const loggerConfig: Params = {
+  pinoHttp: {
+    logger: pinoLogger,
+    autoLogging: false,   // disabled — pulsarRequestLogger handles request auto-logs
+  },
+}
 ```
 
-## Features
+### 2. Bootstrap with the middleware
 
-- Batched log delivery (configurable batch size)
-- Automatic retry with exponential backoff (3 attempts)
-- In-memory queue — no logs lost on temporary network failures
-- Zero dependencies (uses native `fetch`)
-- TypeScript types included
-- Dual CJS/ESM package
+```ts
+// server/src/main.ts
+import { NestFactory } from '@nestjs/core'
+import { NestExpressApplication } from '@nestjs/platform-express'
+import { Logger } from 'nestjs-pino'
+import { pulsarRequestLogger } from '@fabbahiense/pulsar-pino-transport/middleware'
+import { AppModule } from './app.module'
+import { pinoLogger } from './infra/logger.config'
 
-## Options
+async function bootstrap() {
+  const app = await NestFactory.create<NestExpressApplication>(AppModule, { bufferLogs: true })
+  app.useLogger(app.get(Logger))
 
-| Option | Type | Default | Description |
-|--------|------|---------|-------------|
-| `url` | `string` | — | Pulsar server URL (required) |
-| `apiKey` | `string` | — | API key for authentication (required) |
-| `source` | `string` | `'default'` | Service name in Pulsar |
-| `batchSize` | `number` | `50` | Logs buffered before flush |
-| `flushInterval` | `number` | `2000` | Flush interval in ms |
+  app.use(pulsarRequestLogger({ logger: pinoLogger }))
 
-## Trace ID
-
-The transport automatically picks up trace IDs from these fields:
+  await app.listen(process.env.PORT ?? 3000)
+}
+bootstrap()
+```
 
-- `traceId`
-- `trace_id`
-- `requestId`
-- `req.id`
+### 3. Use the logger normally
 
-```typescript
-logger.info({ traceId: 'my-trace-id' }, 'Processing request')
-```
+```ts
+import { Logger } from '@nestjs/common'
 
-## Capturing request/response bodies (with `pino-http`)
+@Injectable()
+export class CampaignService {
+  private readonly logger = new Logger(CampaignService.name)
 
-The transport itself doesn't intercept HTTP — pino is a logger, not a middleware. To get rich request/response logs in Pulsar, pair it with [`pino-http`](https://github.com/pinojs/pino-http):
+  async tokensUsed(campaignId: string, tokensUsed: number) {
+    // Plain log — appears as a normal line
+    this.logger.log('Campaign processed')
 
-```bash
-npm install pino-http
+    // Rich log — appears with Context section in Pulsar's side panel
+    this.logger.log({ metadata: { campaignId, tokensUsed } }, 'Campaign analytics tokens used')
+  }
+}
 ```
 
-```typescript
+**Rule of thumb:** anything you want to see in Pulsar's side panel goes inside `metadata: {...}` as the first argument.
+
+## Quick Start — Express (no NestJS)
+
+```ts
 import express from 'express'
 import pino from 'pino'
-import pinoHttp from 'pino-http'
+import { pulsarRequestLogger } from '@fabbahiense/pulsar-pino-transport/middleware'
 
 const logger = pino({
   transport: {
     target: '@fabbahiense/pulsar-pino-transport',
-    options: { url: process.env.PULSAR_URL, apiKey: process.env.PULSAR_API_KEY },
+    options: {
+      url: process.env.PULSAR_URL,
+      apiKey: process.env.PULSAR_API_KEY,
+      source: 'my-api',
+    },
   },
 })
 
 const app = express()
 app.use(express.json())
+app.use(pulsarRequestLogger({ logger }))
 
-app.use(pinoHttp({
-  logger,
-  // Serialize request body
-  serializers: {
-    req(req) {
-      return {
-        method: req.method,
-        url: req.url,
-        headers: redact(req.headers),
-        requestBody: req.raw?.body,       // Pulsar UI recognizes this key
-        query: req.query,
-        params: req.params,
-      }
-    },
-    res(res) {
-      return {
-        statusCode: res.statusCode,
-        headers: redact(res.getHeaders?.() ?? {}),
-        // responseBody requires intercepting res.write/res.end — see below
-      }
-    },
-  },
-  customLogLevel(_req, res, err) {
-    if (err || res.statusCode >= 500) return 'error'
-    if (res.statusCode >= 400) return 'warn'
-    return 'info'
-  },
-}))
+app.post('/api/users', (req, res) => {
+  res.status(201).json({ ok: true })
+})
 
-function redact(headers: Record<string, unknown>): Record<string, unknown> {
-  const out: Record<string, unknown> = {}
-  const drop = new Set(['authorization', 'cookie', 'set-cookie', 'x-api-key'])
-  for (const [k, v] of Object.entries(headers)) {
-    out[k] = drop.has(k.toLowerCase()) ? '[REDACTED]' : v
-  }
-  return out
-}
+app.listen(3000)
 ```
 
-**No painel lateral do Pulsar** isso aparece em seções dedicadas: Request body, Headers, Query, Path params, status, duration.
-
-### Capturando response body (avançado)
-
-Pino-http não captura response body nativamente. Para capturar, intercepte `res.write`/`res.end`:
-
-```typescript
-app.use((req, res, next) => {
-  const chunks: Buffer[] = []
-  const originalWrite = res.write.bind(res)
-  const originalEnd = res.end.bind(res)
-  res.write = function (chunk: any) { if (chunk) chunks.push(Buffer.from(chunk)); return originalWrite(chunk) }
-  res.end = function (chunk: any) { if (chunk) chunks.push(Buffer.from(chunk)); return originalEnd(chunk) }
-  res.on('finish', () => {
-    try {
-      const body = Buffer.concat(chunks).toString('utf8').slice(0, 10240)
-      logger.info({ responseBody: tryParseJson(body), statusCode: res.statusCode, traceId: req.id }, `${req.method} ${req.url} ${res.statusCode}`)
-    } catch {}
-  })
-  next()
+## What appears in Pulsar
+
+For each request, **one** log entry shows up with:
+
+- **Chips**: status code (color-coded by 2xx/4xx/5xx), response time, trace id, user id
+- **Context section** (top): any custom fields you logged
+- **Endpoint**: `METHOD url`
+- **Tabs**:
+  - **Request**: query params, path params, request headers, request body
+  - **Response**: response headers, response body
+  - **Stack trace**: if there was an error
+
+All sections have syntax-highlighted JSON, byte-size badge and copy button.
+
+## Middleware options
+
+```ts
+pulsarRequestLogger({
+  logger,                                 // required: your pino instance
+  skipUrls: ['/health', '/metrics'],      // default — URLs to skip
+  redactKeys: ['cpf', 'cardNumber'],      // extra keys to redact (case-insensitive)
+  maxBodySize: 50_000,                    // bytes — bigger bodies are truncated
+  getUserId: req => req.user?.id,         // how to extract userId from req
+  getLevel:  (req, res) =>                // how to choose log level
+    res.statusCode >= 500 ? 'error' : res.statusCode >= 400 ? 'warn' : 'info',
 })
+```
 
-function tryParseJson(s: string) { try { return JSON.parse(s) } catch { return s } }
+**Default redacted keys** (always applied, in headers and in request and JSON response bodies):
+`password`, `passwd`, `password_hash`, `secret`, `client_secret`, `token`, `access_token`, `refresh_token`, `id_token`, `session_id`, `api_key`, `x-api-key`, `x-auth-token`, `authorization`, `proxy-authorization`, `cookie`, `set-cookie`.
+
+Keys are compared ignoring case, `-` and `_`, so `access_token`, `access-token` and `accessToken` are all redacted (the same applies to your `redactKeys`). Only whole keys match: `tokensUsed` or `passwordHint` are kept unless you list them.
+
+Redaction is recursive — it walks nested objects in the bodies and replaces matching keys with `[REDACTED]`. At most `maxBodySize` bytes of a response are kept in memory, however large the response is.
+
+
+## Transport options
+
+```ts
+{
+  target: '@fabbahiense/pulsar-pino-transport',
+  options: {
+    url: 'https://your-pulsar.com',       // required
+    apiKey: 'psk_xxx',                    // required
+    source: 'my-service',                 // optional — appears as serviceName (alias: serviceName)
+    batchSize: 50,                        // optional — buffered logs that trigger an immediate send (max 1000)
+    flushInterval: 2000,                  // optional — longest a log waits before it is sent, in ms
+    maxBufferSize: 5000,                  // optional — most logs held in memory while Pulsar is slow or down
+    maxRetries: 10,                       // optional — retries before a batch is dropped (at most maxRetries + 1 sends)
+    requestTimeout: 10000,                // optional — timeout per HTTP request, in ms
+    shutdownTimeout: 3000,                // optional — longest the final flush waits when pino closes, in ms
+    warnOnDrop: true,                     // optional — one stderr line (at most once a minute) when logs are dropped
+  },
+}
 ```
 
-> **Mais simples?** Se você puder, considere `@fabbahiense/pulsar-node` que já vem com `Pulsar.middleware({ captureResponseBody: true })` configurado.
+Or let the helper build that object:
 
-## Recognized metadata keys
+```ts
+import pino from 'pino'
+import { createPulsarTransport } from '@fabbahiense/pulsar-pino-transport'
+
+const logger = pino(createPulsarTransport({ url: process.env.PULSAR_URL!, apiKey: process.env.PULSAR_API_KEY!, source: 'my-api' }))
+```
 
-Pulsar UI auto-detects these key names and renders dedicated sections:
+## Delivery guarantees
 
-| Section | Recognized keys |
+- **Never blocks pino.** Records are buffered as they arrive and sent in the background, so a slow or unreachable Pulsar never stalls pino's worker (or, with `sync: true`, your app's thread).
+- **Batched.** Logs are sent in one request at a time to `/api/ingest/batch`, when `batchSize` logs are buffered or `flushInterval` passes. A backlog drains in requests of up to 1,000 logs / 512 KiB. Entries are capped at 256 KiB.
+- **Retried.** `429`, `408`, `5xx`, network errors and timeouts are retried with exponential backoff (1 s, doubling, capped at 30 s, with jitter), honouring `Retry-After`. A retry resends the same logs with the same ids, so the server deduplicates them. When the last of `maxRetries` retries also fails, that is after `maxRetries + 1` failed requests in a row, the batch is dropped. Other `4xx` responses are permanent (a `413` first splits the batch).
+- **Bounded memory.** At most `maxBufferSize` logs (and 16 MiB) are buffered. When full, the **oldest** logs are dropped and counted.
+- **Delivered on shutdown.** When pino closes the transport (it does as the process exits), what is left is sent before the stream closes, within `shutdownTimeout`. After an explicit `process.exit()` pino cannot wait for network I/O, so logs still buffered then may be lost.
+- **Reported.** Dropped logs (by reason: `overflow`, `rejected`, `retriesExhausted`, `shutdown`) are summarized on stderr at most once a minute, unless `warnOnDrop: false`.
+
+## How it works
+
+1. The transport runs in pino's worker thread and forwards pino records to Pulsar's `/api/ingest/batch`.
+2. `pulsarRequestLogger` intercepts `res.write`/`res.end` to capture the response body (up to `maxBodySize`), then on `res.on('finish')` emits one structured log via your pino logger.
+3. The log is shaped so the Pulsar UI recognizes the key names and renders dedicated sections:
+
+| Pulsar UI section | Recognized metadata keys |
 |---|---|
 | Request body | `body`, `requestBody`, `reqBody`, `payload`, `input` |
 | Response body | `response`, `responseBody`, `output`, `result`, `data` |
 | Request headers | `headers`, `requestHeaders`, `reqHeaders` |
 | Response headers | `responseHeaders`, `resHeaders` |
-| Query | `query`, `queryParams`, `qs` |
-| Path params | `params`, `pathParams` |
+| Query params | `query`, `queryParams`, `qs` |
+| Path params | `params`, `pathParams`, `routeParams` |
+| Entity chip | `userId`, `accountId`, `tenantId`, `email`, `workspaceId` |
+| Context tree | everything else |
+
+Levels: pino's numeric levels, level labels (`formatters.level`) and custom numeric levels (mapped to the nearest standard level below them) are all understood. Times can be epoch milliseconds, epoch seconds (`stdTimeFunctions.unixTime`) or ISO strings.
+
+## Features
+
+- One-line drop-in Express middleware
+- Captures request body, response body, headers, query, params
+- Recursive redaction of sensitive keys in bodies and headers
+- Body size limits with truncation (not drop)
+- Skip URL list for noisy endpoints
+- Automatic level inference: 5xx=error, 4xx=warn, else=info
+- Custom message synthesis when pino-http emits generic strings
+- Batched delivery with retries, backoff and `Retry-After`
+- Bounded in-memory buffer with an explicit, counted overflow policy
+- TypeScript types included
+- Dual CJS/ESM package
+- End-to-end tested
+
+## Trace ID
+
+The transport picks up trace IDs from these fields (first match wins):
+`traceId`, `trace_id`, `requestId`, `req.id`
+
+```ts
+logger.info({ traceId: 'my-trace-id' }, 'Processing request')
+```
+
+With the middleware, the request entry gets it automatically: `req.id` (if a middleware such as pino-http set it), else the `x-trace-id` header, else a W3C `traceparent`, else a new trace id.
+
+## Distributed tracing
+
+pino runs this transport in a worker thread, where the app's `AsyncLocalStorage` is out of reach, so a mixin writes the trace ids into each record in the app's thread. One line:
+
+```ts
+import pino from 'pino'
+import { pulsarMixin, instrumentFetch, startSpan } from '@fabbahiense/pulsar-pino-transport'
+
+const logger = pino({
+  mixin: pulsarMixin(),   // adds traceId, spanId, parentSpanId to every record logged inside a trace
+  transport: { target: '@fabbahiense/pulsar-pino-transport', options: { url, apiKey, source: 'checkout' } },
+})
+
+app.use(pulsarRequestLogger({ logger }))          // each request runs in its own span
+
+const tracedFetch = instrumentFetch({ logger })   // sends traceparent + x-trace-id, logs a client span
+await startSpan('db.orders.insert', () => db.insert(order), { logger })
+```
+
+Also exported from the root entry: `getTraceContext`, `runWithTrace`, `instrumentAxios`, `traceHeaders`, `injectTrace` / `extractTrace` (queues), `parseTraceparent`, `formatTraceparent`. With pino-http, register `pulsarRequestLogger()` first and set pino-http's `genReqId: () => getTraceContext()?.traceId ?? randomUUID()`, so both agree on the trace. Details: [docs/sdks.md, Distributed tracing](https://github.com/FabbSantos/pulsar/blob/main/docs/sdks.md#distributed-tracing).
 
 ## License