logroller 2.0.2 → 2.1.0

package/CHANGELOG.md

@@ -5,6 +5,24 @@ All notable changes to this project are documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.1.0] — 2026-08-28
+
+### Added
+
+- `symlink: true | string` — stable "current" pointer (e.g. `app-current.log`)
+  for `tail -f`/Filebeat/promtail; atomically flipped on every segment open;
+  degrades gracefully on symlink-denying filesystems; excluded from retention.
+- `maxTotalSize: "5g"` — byte-budget retention: deletes OLDEST archives until
+  under budget; composes with `maxFiles`.
+- `onSeal` hook (+ `sealHookTimeoutMs`) — awaited after a segment is archived
+  and BEFORE retention can delete it: the S3/GCS upload extension point.
+  Errors/timeouts degrade to `warn`; rotation never stalls.
+- `installShutdown(streams)` / `flushAll()` — graceful SIGINT/SIGTERM
+  handling that flushes buffered writes before exit (fixes the classic
+  "last log lines missing" support case).
+- `stats()` and `listSegments()` — health snapshot and per-file listing.
+- `compressionLevel` (0–9, default 6) — passes through to gzip.
+
 ## [2.0.2] — 2026-08-28
 
 ### Fixed

package/README.md

@@ -1,5 +1,3 @@
-## README.md
-
 <p align="center"><strong>logroller</strong></p>
 <h3 align="center">Zero-dependency rotating file streams for Node.js</h3>
 
@@ -14,8 +12,8 @@
 
 ---
 
-`logroller` turns a plain append destination into a resilient, observable
-log sink: **it is a Writable**, so `readable.pipe(log)` works, pino multistreams
+`logroller` turns a plain append destination into a resilient, observable log
+sink: **it is a Writable**, so `readable.pipe(log)` works, pino multistreams
 work, anything that consumes streams works.
 
 ```bash
@@ -27,16 +25,24 @@ npm install logroller
 - **Daily & interval rotation** — exact local midnight in any IANA zone,
   or epoch-aligned intervals (`30s`, `5m`, `2h`)
 - **Size caps** — `"20m"`-style limits roll numbered segments
-- **Gzip archives** — sealed segments compressed atomically
-- **Retention** — by age or count; unset means files live forever
+- **Gzip archives** — sealed segments compressed atomically, level 0–9
+- **Stable "current" path** — a symlink log collectors can follow
+  (`tail -F app-current.log`, Filebeat, promtail) while files rotate beneath
+- **Retention three ways** — by age, by count, **or by total byte budget**
+  ("never exceed 5 GB of archives")
+- **Upload pipeline** — an awaited `onSeal` hook that fires after a segment
+  is archived and _before_ retention can delete it: ship to S3/GCS safely
+- **Graceful shutdown** — one helper flushes buffered writes on
+  SIGINT/SIGTERM (no more missing last lines)
 - **Timezone-correct** — `Asia/Kolkata` means IST filenames, on a UTC box
-- **Crash-safe** — restarts resume mid-series byte-exactly;
-  gzips interrupted by a kill are completed on boot
-- **Observability** — `<stem>_audit.json` manifest with a uuid-tagged
-  event journal for every file's lifecycle
+- **Crash-safe** — restarts resume mid-series byte-exactly; gzips
+  interrupted by a kill are completed on boot
+- **Observability** — `<stem>_audit.json` manifest with a uuid-tagged event
+  journal, plus `stats()` / `listSegments()` introspection
 - **Testable time** — virtual clocks (`addHours`, `addHoursEveryMin`,
   `advanceClock()`) let a day pass per minute, or instantly
 - **Types first** — tuple-typed events, strict mode, dual ESM/CJS build
+- **Zero dependencies**
 
 ## Quick start
 
@@ -62,6 +68,25 @@ log.on("rotated", ({ reason, archive, newFile }) =>
 process.on("exit", () => log.end());
 ```
 
+### Production shape — collectors, budget, uploads, shutdown
+
+```ts
+import { createStream, installShutdown } from "logroller";
+
+const log = createStream({
+  filename: "logs/app-%DATE%.log",
+  zippedArchive: true,
+  maxSize: "20m",
+  maxTotalSize: "5g", // never hold more than 5 GB of archives
+  maxFiles: "90d",
+  symlink: true, // logs/app-current.log always points at the live file
+  onSeal: async ({ gz }) => uploadToS3(gz), // awaited BEFORE retention deletes
+});
+
+// SIGINT/SIGTERM → flush buffered writes, then exit
+installShutdown(log);
+```
+
 ### CommonJS
 
 ```js
@@ -81,15 +106,37 @@ const logger = pino(
 
 ## What lands on disk
 
+Example snapshot for `filename: "logs/app-%DATE%.log"` on **Aug 28**, with
+`zippedArchive: true` and `symlink: true`:
+
 ```
-app-2026-08-27.log          ← active
-app-2026-08-27.log.gz       ← sealed (hit 20 MB)
-app-2026-08-27.1.log        ← next segment mid-day
-app-2026-08-27.1.log.gz     ← sealed too
-app-2026-08-28.log          ← fresh day (in your tz), clean series
+app-current.log             ← symlink → app-2026-08-28.log   (only with symlink: true)
+app-2026-08-27.log.gz       ← yesterday, sealed (hit 20 MB)
+app-2026-08-27.1.log.gz     ← yesterday, sealed too
+app-2026-08-28.log          ← ACTIVE — the only raw file on disk
 app_audit.json              ← library-owned manifest
 ```
 
+Three invariants that hold at every moment:
+
+1. **Exactly one raw file** — the active segment. Everything else is a
+   `.gz` archive (or a raw tail kept only because gzip failed — with a
+   `warn` event, never silently).
+2. **A segment's `.log` and `.log.gz` never coexist.** The raw file is
+   deleted only _after_ its archive is verified.
+3. **`app-current.log` always points at the active segment**, flipped
+   atomically on every open. It exists only when `symlink` is enabled, and
+   retention never touches it.
+
+### How a day evolves (all states, in order)
+
+| Moment (tz)               | Directory contains                        | `app-current.log` →                                              |
+| ------------------------- | ----------------------------------------- | ---------------------------------------------------------------- |
+| morning, first write      | `…27.log`                                 | `…27.log`                                                        |
+| hit 20 MB → roll          | `…27.log.gz` + `…27.1.log`                | `…27.1.log`                                                      |
+| `.1` fills → roll         | `…27.log.gz` + `…27.1.log.gz`             | _(stale until next write — segments open lazily on first write)_ |
+| midnight → Aug 28 + write | `…27.log.gz` + `…27.1.log.gz` + `…28.log` | `…28.log`                                                        |
+
 A killed process resumes without gaps or duplicates:
 
 | Disk at crash                   | Restart continues into                |
@@ -98,6 +145,106 @@ A killed process resumes without gaps or duplicates:
 | `app-D.log.gz` + `app-D.1.log`  | `app-D.1.log`                         |
 | `app-D.1.log.gz` (chain sealed) | `app-D.2.log`                         |
 | died during a gzip              | boot finishes the gzip, then proceeds |
+| crash left yesterday's tail raw | boot seals (gzips) it automatically   |
+
+## Log collectors & the stable current path
+
+Rotating filenames are great for humans, hostile to consumers: `tail -f`,
+Filebeat, Fluent Bit and promtail want **one path that never changes**.
+
+```ts
+const log = createStream({
+  filename: "logs/app-%DATE%.log",
+  symlink: true, // → logs/app-current.log
+  // symlink: "CURRENT",         // → logs/CURRENT (custom name)
+});
+```
+
+```bash
+tail -F logs/app-current.log     # survives every rotation (capital -F: re-resolve the link)
+```
+
+- The pointer is flipped **atomically** (tmp + rename) on every segment open,
+  so followers never observe a gap or a half-written name.
+- The target is stored **relative**, so the directory stays portable.
+- Best-effort by design: on filesystems that deny symlinks (Windows without
+  Developer Mode) the stream warns once and continues without it.
+- The pointer is never touched by retention and never reported as a segment.
+
+## Upload pipeline — archive, ship, _then_ delete
+
+`onSeal` runs inside the rotation queue **after** the segment is compressed
+and **before** retention can remove it — the safe window for shipping:
+
+```ts
+const log = createStream({
+  filename: "logs/cdr-%DATE%.log",
+  zippedArchive: true,
+  maxFiles: "7d", // local copies are disposable
+  onSeal: async ({ raw, gz }) => {
+    await uploadToS3(gz ?? raw); // awaited — rotation waits
+  },
+  sealHookTimeoutMs: 60_000, // a wedged upload warns instead of stalling
+});
+```
+
+Guarantees:
+
+- invoked exactly once per sealed segment, with `{ raw, gz }` (`gz: null`
+  when compression is off or gzip failed)
+- **ordering**: hook → then retention sweep, so an in-flight upload can never
+  race a deletion
+- a throwing or timed-out hook degrades to a `warn` event — logging and
+  rotation continue
+
+## Graceful shutdown
+
+The classic support case: SIGTERM arrives, buffered lines are lost. One call
+ends it:
+
+```ts
+import { createStream, installShutdown } from "logroller";
+
+const log = createStream({ filename: "logs/app-%DATE%.log" });
+installShutdown(log); // SIGINT + SIGTERM → end() → exit 0
+```
+
+- `installShutdown(stream | streams, { signals?, timeoutMs?, code? })` —
+  installs process listeners once; streams accumulate in a registry.
+- On signal: every registered stream is ended (buffered writes flushed,
+  audit manifest finalized), with a hard `timeoutMs` deadline (default 5 s).
+- `flushAll()` is the same graceful end **without exiting** — for tests and
+  custom signal handling.
+
+## Introspection
+
+```ts
+const st = log.stats();
+// { dir, currentStamp, activeFile, activeIndex, segments, archived,
+//   totalBytes, clockOffsetMs, symlink }
+
+for (const seg of log.listSegments()) {
+  // { file, stamp, index, gzipped, bytes, state, openedAt?, archivedAt? }
+  console.log(seg.stamp, seg.index, seg.gzipped ? "gz" : "raw", seg.bytes);
+}
+```
+
+`listSegments()` reads the **directory** (disk is truth), so it also reports
+files the manifest doesn't know (state `"untracked"`). Perfect for
+`/healthz` endpoints, dashboards, and "where did my logs go" support.
+
+## Retention — three policies, composable
+
+```ts
+maxFiles: "90d"      // age: delete archives older than 90 days
+maxFiles: 500        // count: keep the newest 500 archives
+maxTotalSize: "5g"   // budget: delete OLDEST archives until under 5 GB
+maxFiles: "30d",
+maxTotalSize: "2g",  // combined: either policy may delete
+```
+
+Unset means files live forever until you delete them. The active segment is
+never a retention candidate, and the audit/symlink files are always reserved.
 
 ## API
 
@@ -110,8 +257,13 @@ A killed process resumes without gaps or duplicates:
 | `filename`            | **required**, must contain `%DATE%`   | —                  |
 | `datePattern`         | `YYYY YY MM DD HH mm ss`              | `YYYY-MM-DD`       |
 | `zippedArchive`       | boolean                               | `false`            |
+| `compressionLevel`    | gzip level `0`–`9` (0 = stored)       | `6`                |
 | `maxSize`             | `20m`, `512k`, `2g`, bytes, `0`=off   | off                |
 | `maxFiles`            | `"90d"` age, `500` count, unset=never | —                  |
+| `maxTotalSize`        | `"5g"`, `"500m"`, bytes, `0`=off      | off                |
+| `symlink`             | `true` (auto name) or custom string   | off                |
+| `onSeal`              | `async ({ raw, gz }) => void`         | —                  |
+| `sealHookTimeoutMs`   | hard deadline for one `onSeal` call   | `30000`            |
 | `frequency`           | `daily`, `30s`, `5m`, `2h`, ms        | `daily`            |
 | `tz`                  | any IANA zone                         | `UTC`              |
 | `audit` / `auditFile` | manifest control                      | `true` / auto-name |
@@ -122,24 +274,33 @@ A killed process resumes without gaps or duplicates:
 
 ### Methods
 
-| Method                  | Returns          | Purpose                    |
-| ----------------------- | ---------------- | -------------------------- |
-| `write / end / destroy` | stream semantics | normal `Writable`          |
-| `rotateNow()`           | `Promise<void>`  | force a rotation           |
-| `advanceClock()`        | `Promise<void>`  | apply one virtual tick now |
+| Method                  | Returns          | Purpose                         |
+| ----------------------- | ---------------- | ------------------------------- |
+| `write / end / destroy` | stream semantics | normal `Writable`               |
+| `rotateNow()`           | `Promise<void>`  | force a rotation                |
+| `advanceClock()`        | `Promise<void>`  | apply one virtual tick now      |
+| `stats()`               | `StreamStats`    | health snapshot                 |
+| `listSegments()`        | `SegmentInfo[]`  | every family file, oldest first |
+
+### Module functions
+
+| Function                          | Purpose                                                      |
+| --------------------------------- | ------------------------------------------------------------ |
+| `installShutdown(streams, opts?)` | register streams for SIGINT/SIGTERM flush; returns `flush()` |
+| `flushAll(timeoutMs?)`            | gracefully end all registered streams, no exit               |
 
 ### Events (typed)
 
-| Event     | Payload                               | Meaning                      |
-| --------- | ------------------------------------- | ---------------------------- |
-| `open`    | `file`                                | segment opened for appending |
-| `rotated` | `{reason, oldFile, archive, newFile}` | segment closed               |
-| `archive` | `gzFile`                              | gzip finished                |
-| `deleted` | `file`                                | retention removed it         |
-| `period`  | `stamp`                               | boundary crossed while idle  |
-| `clock`   | `{offsetMs, …}`                       | virtual clock engaged        |
-| `warn`    | `Error`                               | non-fatal problems           |
-| `error`   | `Error`                               | fatal write problems         |
+| Event     | Payload                               | Meaning                                                      |
+| --------- | ------------------------------------- | ------------------------------------------------------------ |
+| `open`    | `file`                                | segment opened for appending                                 |
+| `rotated` | `{reason, oldFile, archive, newFile}` | segment closed                                               |
+| `archive` | `gzFile`                              | gzip finished                                                |
+| `deleted` | `file`                                | retention removed it                                         |
+| `period`  | `stamp`                               | boundary crossed while idle                                  |
+| `clock`   | `{offsetMs, …}`                       | virtual clock engaged                                        |
+| `warn`    | `Error`                               | non-fatal (hook failure, gzip retry, symlink unavailable, …) |
+| `error`   | `Error`                               | fatal write problems                                         |
 
 Errors must be handled — attach a listener on every instance.
 
@@ -149,46 +310,71 @@ Errors must be handled — attach a listener on every instance.
 const s = createStream({
   filename: "logs/demo-%DATE%.log",
   zippedArchive: true,
-  addHoursEveryMin: "24h", // one day per real minute
+  addHours: "-48h", // start two days back
+  addHoursEveryMin: "24h", // one virtual day per real minute
   clockStepIntervalMs: 1000, // or faster
 });
 await s.advanceClock(); // jump instantly when useful
 ```
 
+Negative `addHoursEveryMin` deliberately triggers the backward-clock guard
+(warn, no rotation) — it exists so you can test that safety net too.
+
 ## FAQ
 
 **Why is `%DATE%` mandatory?**
 It is the anchor that decouples _rotation scheduling_ from _filenames_.
 Coarser patterns than your frequency collapse distinct segments onto one
-name — match tokens to cadence (`HH` for hourly, etc.).
+name — match tokens to cadence (`HH` for hourly, etc.). The library warns at
+construction when it detects a mismatch.
+
+**How do I ship logs to S3 without losing any?**
+`zippedArchive: true` + `onSeal` (see the upload pipeline section). The hook
+is awaited and runs before retention, so an in-flight upload can never race
+a deletion.
+
+**How do I keep `tail -f` working across rotations?**
+`symlink: true` — follow `app-current.log`. Use `tail -F` (capital) so the
+follower re-resolves the link after it flips.
 
 **Can two processes share one directory?**
-Different prefixes/families: yes. The _same_ family: no — the audit
-manifest detects a live second writer (`writer.heartbeat`) and warns loudly.
+Different prefixes/families: yes. The _same_ family: no — the audit manifest
+detects a live second writer (`writer.heartbeat`) and warns loudly.
 
 **Is the audit JSON required?**
 No — `audit: false` reduces behaviour to pure disk-scan recovery. Recommended
-to leave on: it repairs interrupted gzips and demotes zombies automatically.
+to leave on: it repairs interrupted gzips, demotes zombie records, and
+powers `listSegments()` states.
+
+**What if the system clock jumps backward?**
+The stream keeps the current segment, warns once, and resumes normal
+rotation when the clock catches up — the series is never forked into a past
+family. (Requires zero-padded `datePattern` tokens, which all presets are.)
 
 **Windows / macOS / Linux?**
 Pure Node APIs; path handling via `node:path`. All three supported
-(Node `>=16.14`).
+(Node `>=16.14`). Symlinks are best-effort on Windows — see above.
 
 ## Compare
 
-|                           | logroller         | file-stream-rotator | rotating-file-stream | winston-daily-rotate-file |
-| ------------------------- | ----------------- | ------------------- | -------------------- | ------------------------- |
-| Dependencies              | **0**             | 4+                  | 2+                   | winston-bound             |
-| Crash-safe resume         | ✅ audit-assisted | partial             | partial              | ✖                         |
-| Interrupted-gzip repair   | ✅                | ✖                   | ✖                    | ✖                         |
-| Event journal             | ✅                | ✖                   | ✖                    | ✖                         |
-| TZ-correct daily rollover | ✅                | partial             | ✅                   | ✅                        |
-| Virtual test clock        | ✅                | ✖                   | ✖                    | ✖                         |
-| Winston required          | no                | no                  | no                   | yes                       |
+|                                   | logroller         | file-stream-rotator | rotating-file-stream | winston-daily-rotate-file |
+| --------------------------------- | ----------------- | ------------------- | -------------------- | ------------------------- |
+| Dependencies                      | **0**             | 4+                  | 2+                   | winston-bound             |
+| Crash-safe resume                 | ✅ audit-assisted | partial             | partial              | ✖                         |
+| Interrupted-gzip repair           | ✅                | ✖                   | ✖                    | ✖                         |
+| Event journal                     | ✅                | partial             | ✖                    | ✖                         |
+| TZ-correct daily rollover         | ✅                | partial             | ✅                   | ✅                        |
+| Stable current path               | ✅ atomic         | ✅                  | ✅                   | ✅                        |
+| Byte-budget retention             | ✅                | ✅ (bytes only)     | ✖                    | ✖                         |
+| Awaited upload-before-delete hook | ✅                | ✖                   | ✖                    | ✖                         |
+| Graceful shutdown helper          | ✅                | ✖                   | ✖                    | via winston               |
+| Introspection API                 | ✅                | ✖                   | ✖                    | partial                   |
+| Virtual test clock                | ✅                | ✖                   | ✖                    | ✖                         |
+| Winston required                  | no                | no                  | no                   | yes                       |
 
 ## Contributing
 
-PRs welcome. `npm run verify` gates every change (typecheck + tests).
+PRs welcome. `npm run verify` gates every change (lint + typecheck + tests).
 Please file issues before large refactors.
 
 ## License

package/dist/fsio.d.ts

@@ -3,9 +3,9 @@ import fs from "node:fs";
 export declare const statSizeSafe: (p: string | null | undefined) => number;
 export declare const existsSafe: (p: string) => boolean;
 /**
- * Compress `src` → `dst`, deleting `src` ONLY on verified success.
- * On failure removes the partial `dst` and reports via callback —
+ * Compress `src` → `dst` at gzip `level`, deleting `src` ONLY on verified
+ * success. On failure removes the partial `dst` and reports via callback —
  * logs degrade to uncompressed rather than losing data.
  */
-export declare function gzipMove(src: string, dst: string, onWarn?: (err: Error) => void): Promise<boolean>;
+export declare function gzipMove(src: string, dst: string, onWarn?: (err: Error) => void, level?: number): Promise<boolean>;
 export declare const endStream: (ws: fs.WriteStream | null) => Promise<void>;