@wtfalch/reporting 0.1.0

package/README.md

@@ -0,0 +1,149 @@
+# @wtfalch/reporting
+
+The estate's operational event log and the housekeeping tick behind it: a
+writer that never fails a request, a claim protocol that needs no scheduler,
+and the readers an operator opens. Per-app Postgres, a host-supplied logger,
+a framework-neutral core and a Next entry. The design and its decisions are
+`docs/plans/reporting.md` and `docs/plans/reporting/core.md` in
+`wtfalch/app-template`.
+
+Three records make up reporting on the estate. This package holds the
+**events** (what the system did: jobs, drains, alerts, an integration saying
+no) and, from 0.2, **analytics** (how people use the product). The **audit**
+trail (who did what to what) is `@wtfalch/audit` and is not here: an audit
+row is transactional and its actor is unforgeable; an event row is best
+effort. The two share column names so a reader can walk between them, and
+nothing else.
+
+## Install
+
+```sh
+pnpm add @wtfalch/reporting
+pnpm exec reporting-migrations   # copies migrations/*.sql into drizzle/ as the next numbers
+```
+
+The copied migration creates `reporting_events`, `reporting_tasks` and
+`reporting_settings`, a `SECURITY DEFINER` prune function, and, when a role
+named `<database>_rt` exists, revokes UPDATE, DELETE and TRUNCATE on events
+from it. That is the estate's runtime-role convention; a host without the
+role gets the tables and the function and no revokes take effect. Your
+migrate script applies the file; this package never connects to apply
+anything.
+
+## Bind it once
+
+```ts
+// src/lib/reporting/core.ts, the one file that imports your raw handle
+import { createReporting } from '@wtfalch/reporting';
+import { createHousekeeping, pruneEvents, retentionLag } from '@wtfalch/reporting/housekeeping';
+import { deferWithAfter } from '@wtfalch/reporting/next';
+
+export const reporting = createReporting({
+  db,                       // your drizzle handle
+  log,                      // pino-shaped: info / warn / error
+  site: 'my-app',           // stamped on every row
+  defer: deferWithAfter,    // after() under Next; a timer elsewhere
+  onAlert: (f) => Sentry.captureMessage(f.message, { tags: { alert: f.check } }),  // optional
+});
+
+export const housekeeping = createHousekeeping({ reporting });
+housekeeping.register(pruneEvents);
+housekeeping.register(retentionLag);
+```
+
+Re-export the tables from your schema module so your drizzle instance knows
+them:
+
+```ts
+export { reportingEvents, reportingTasks, reportingSettings } from '@wtfalch/reporting';
+```
+
+## Write
+
+```ts
+reporting.event({
+  kind: 'mail.write_refused',          // namespace.name
+  level: 'warn',                       // info | warn | error | alert
+  message: 'the instance said no',     // ≤ 512
+  tenantId, requestId,                 // optional
+  actor: { class: 'human', id },       // optional; never a bare id
+  target: { type: 'mailbox', id },     // optional
+  data: { type: 'forbidden', attempt: 2 },  // flat scalars only, ≤ 16 KB, no banned keys
+});
+
+reporting.alert({ check: 'orphan_memberships', message: '2 rows', detail: rows });
+```
+
+`event()` writes a logger line first, queues the row, and flushes after the
+current unit of work and on a timer. It never throws in production: a row it
+refuses (a nested `data`, a key named `email`, `password`, `token`, ...)
+becomes one `reporting.invalid` row and a logger error. In development and
+test it throws, so the mistake is found. `alert()` projects a finding's
+`detail` into flat fields (an array becomes its count) and then calls your
+`onAlert`.
+
+## Read
+
+```ts
+const page = await reporting.events.page({ level: 'error', kindNs: 'mail', limit: 50 });
+// { items, next }  newest first; pass `after: page.next` for the next page
+```
+
+No permission is applied here. Gate in your page.
+
+## Housekeeping
+
+Hang the tick off something that already happens often. Under Next, the
+health route:
+
+```ts
+// src/app/api/health/route.ts
+import { healthHandler } from '@wtfalch/reporting/next';
+export const GET = healthHandler(housekeeping);
+```
+
+Each registered task has a cadence, a lease and a retry. The tick claims a
+due task with `SELECT ... FOR UPDATE SKIP LOCKED` and a token in one short
+transaction, runs it outside, and completes it fenced on the token, so any
+number of containers share the work without an advisory lock. Execution is
+at least once; write idempotent tasks. Register your own:
+
+```ts
+housekeeping.register({
+  name: 'authz.alerts', every: 24 * 3600_000, lease: 60_000, retry: 3600_000,
+  run: async (ctx) => { for (const f of await runAlerts(ctx.db)) ctx.reporting.alert(f); },
+});
+```
+
+## Settings
+
+`reporting.settings.get()` / `set(patch, by)`. `events.retention_days`
+(default 30, bounds 7 to 400) is what `pruneEvents` reads. The prune
+function clamps whatever it is handed to the same bounds, so an operator's
+edit can shorten history to a week and no further. Pass `audit` to
+`createReporting` to have every change handed to your audit ledger after it
+commits.
+
+## Shutdown
+
+```ts
+import { registerShutdownFlush } from '@wtfalch/reporting/next';
+registerShutdownFlush(reporting);   // one bounded flush on SIGTERM
+```
+
+Best effort. The queue is per process and bounded at 1,000; the logger has
+every row regardless.
+
+## Tests
+
+```ts
+import { createFakeReporting } from '@wtfalch/reporting/fake';
+```
+
+Same validation, no database; `fake.rows` is the table.
+
+## What this package never does
+
+Read a request or a session, decide a permission, connect to a database of
+its own, import `next` from the core, or store an ip, a user agent, an
+email address or a display name. See `PRIVACY.md`.

package/dist/alerts.d.ts

@@ -0,0 +1,17 @@
+import type { AlertFinding, FlatData } from './types.js';
+/**
+ * A finding becomes one `alert.<check>` row. Its `detail` is arbitrary and
+ * often nested (a list of orphan rows, an object of counts), and the table
+ * refuses nesting, so it is projected: an array becomes its count, an
+ * object keeps its top-level scalars with banned keys dropped and nested
+ * values named as such, anything else is described. Never the detail
+ * wholesale: that is how an address ends up in an operational row.
+ */
+export declare function alertKind(check: string): string;
+export declare function projectDetail(detail: unknown): FlatData;
+export declare function alertRow(finding: AlertFinding): {
+    kind: string;
+    level: 'alert';
+    message: string;
+    data: FlatData;
+};

package/dist/alerts.js

@@ -0,0 +1,53 @@
+import { BANNED_KEYS, KIND_PATTERN } from './schema.js';
+/**
+ * A finding becomes one `alert.<check>` row. Its `detail` is arbitrary and
+ * often nested (a list of orphan rows, an object of counts), and the table
+ * refuses nesting, so it is projected: an array becomes its count, an
+ * object keeps its top-level scalars with banned keys dropped and nested
+ * values named as such, anything else is described. Never the detail
+ * wholesale: that is how an address ends up in an operational row.
+ */
+export function alertKind(check) {
+    const name = check
+        .toLowerCase()
+        .replace(/[^a-z0-9_]+/g, '_')
+        .replace(/^_+|_+$/g, '')
+        .replace(/^[^a-z]+/, '');
+    const kind = `alert.${name || 'unnamed'}`;
+    return KIND_PATTERN.test(kind) ? kind : 'alert.unnamed';
+}
+export function projectDetail(detail) {
+    if (detail === undefined || detail === null)
+        return {};
+    if (Array.isArray(detail))
+        return { count: detail.length };
+    if (typeof detail === 'object') {
+        const out = {};
+        for (const [key, value] of Object.entries(detail)) {
+            if (BANNED_KEYS.includes(key))
+                continue;
+            if (value === null || typeof value === 'boolean')
+                out[key] = value;
+            else if (typeof value === 'number')
+                out[key] = Number.isFinite(value) ? value : String(value);
+            else if (typeof value === 'string')
+                out[key] = value.slice(0, 500);
+            else if (Array.isArray(value))
+                out[key] = `[${value.length} items]`;
+            else
+                out[key] = '[nested]';
+            if (Object.keys(out).length >= 40)
+                break;
+        }
+        return out;
+    }
+    return { detail: String(detail).slice(0, 500) };
+}
+export function alertRow(finding) {
+    return {
+        kind: alertKind(finding.check),
+        level: 'alert',
+        message: (finding.message || finding.check).slice(0, 512),
+        data: { check: finding.check.slice(0, 200), ...projectDetail(finding.detail) },
+    };
+}

package/dist/bin/copy.d.ts

@@ -0,0 +1,25 @@
+/**
+ * Migrations in this estate are hand-written SQL, numbered per app, applied
+ * by psql on boot, additive. A package cannot own a number in an app's
+ * sequence, so this copies any migration the app has not yet copied into the
+ * app's `drizzle/` as the next numbers and records which in a manifest. The
+ * same mechanism `@wtfalch/threads` uses; the manifest name differs.
+ */
+export declare const MANIFEST = ".reporting-migrations.json";
+export interface Manifest {
+    /** package file name -> the name it was copied to in the app */
+    copied: Record<string, string>;
+}
+export interface CopyResult {
+    copied: Array<{
+        from: string;
+        to: string;
+    }>;
+    manifest: Manifest;
+}
+export declare function copyMigrations(opts: {
+    from: string;
+    to: string;
+    version?: string;
+}): CopyResult;
+export declare function describeCopy(result: CopyResult): string;

package/dist/bin/copy.js

@@ -0,0 +1,57 @@
+import { existsSync, mkdirSync, readFileSync, readdirSync, writeFileSync } from 'node:fs';
+import { basename, join } from 'node:path';
+/**
+ * Migrations in this estate are hand-written SQL, numbered per app, applied
+ * by psql on boot, additive. A package cannot own a number in an app's
+ * sequence, so this copies any migration the app has not yet copied into the
+ * app's `drizzle/` as the next numbers and records which in a manifest. The
+ * same mechanism `@wtfalch/threads` uses; the manifest name differs.
+ */
+export const MANIFEST = '.reporting-migrations.json';
+const NUMBERED = /^(\d{4})_(.+\.sql)$/;
+function readManifest(dir) {
+    const file = join(dir, MANIFEST);
+    if (!existsSync(file))
+        return { copied: {} };
+    const parsed = JSON.parse(readFileSync(file, 'utf8'));
+    return { copied: parsed.copied ?? {} };
+}
+function nextNumber(dir) {
+    let max = -1;
+    for (const name of readdirSync(dir)) {
+        const m = NUMBERED.exec(name);
+        if (m?.[1])
+            max = Math.max(max, Number(m[1]));
+    }
+    return max + 1;
+}
+export function copyMigrations(opts) {
+    mkdirSync(opts.to, { recursive: true });
+    const manifest = readManifest(opts.to);
+    const copied = [];
+    const sources = readdirSync(opts.from)
+        .filter((n) => NUMBERED.test(n))
+        .sort();
+    let next = nextNumber(opts.to);
+    for (const name of sources) {
+        if (manifest.copied[name])
+            continue;
+        const rest = NUMBERED.exec(name)?.[2] ?? name;
+        const target = `${String(next).padStart(4, '0')}_${rest}`;
+        const body = readFileSync(join(opts.from, name), 'utf8');
+        const header = `-- Copied from @wtfalch/reporting${opts.version ? ` ${opts.version}` : ''} (migrations/${name}) by reporting-migrations.\n-- Do not edit here; the next package version ships the next file.\n\n`;
+        writeFileSync(join(opts.to, target), header + body);
+        manifest.copied[name] = target;
+        copied.push({ from: name, to: target });
+        next += 1;
+    }
+    writeFileSync(join(opts.to, MANIFEST), `${JSON.stringify(manifest, null, 2)}\n`);
+    return { copied, manifest };
+}
+export function describeCopy(result) {
+    if (result.copied.length === 0)
+        return 'reporting-migrations: nothing to copy';
+    return result.copied
+        .map((c) => `reporting-migrations: ${c.from} -> ${basename(c.to)}`)
+        .join('\n');
+}

package/dist/bin/migrations.d.ts

@@ -0,0 +1,2 @@
+#!/usr/bin/env node
+export {};

package/dist/bin/migrations.js

@@ -0,0 +1,22 @@
+#!/usr/bin/env node
+import { readFileSync } from 'node:fs';
+import { dirname, join, resolve } from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { copyMigrations, describeCopy } from './copy.js';
+/**
+ * `reporting-migrations [dir]`: copy this package's migrations the app has
+ * not yet copied into `dir` (default `drizzle`) as the next numbers.
+ * Idempotent; run it after every upgrade of @wtfalch/reporting, then commit
+ * what it wrote.
+ */
+const here = dirname(fileURLToPath(import.meta.url));
+const from = join(here, '..', 'migrations');
+const to = resolve(process.cwd(), process.argv[2] ?? 'drizzle');
+let version;
+try {
+    version = JSON.parse(readFileSync(join(here, '..', '..', 'package.json'), 'utf8')).version;
+}
+catch {
+    version = undefined;
+}
+console.log(describeCopy(copyMigrations({ from, to, version })));

package/dist/fake/index.d.ts

@@ -0,0 +1,16 @@
+import type { ReportingEventRow } from '../tables.js';
+import type { AlertFinding, Logger, Reporting } from '../types.js';
+/**
+ * An in-memory `Reporting` for a host's unit tests: same validation, same
+ * shape, no database. `rows` is the table; `alerts` is what `onAlert` would
+ * have received. Validation failures throw, as in development.
+ */
+export interface FakeReporting extends Reporting {
+    readonly rows: ReportingEventRow[];
+    readonly alerts: AlertFinding[];
+    reset(): void;
+}
+export declare function createFakeReporting(opts?: {
+    site?: string;
+    log?: Logger;
+}): FakeReporting;

package/dist/fake/index.js

@@ -0,0 +1,105 @@
+import { alertRow } from '../alerts.js';
+import { eventInputSchema } from '../schema.js';
+import { DEFAULT_SETTINGS, RETENTION_BOUNDS } from '../settings.js';
+import { tables } from '../tables.js';
+const silent = { info() { }, warn() { }, error() { } };
+export function createFakeReporting(opts = {}) {
+    const site = opts.site ?? 'test';
+    const log = opts.log ?? silent;
+    const rows = [];
+    const alerts = [];
+    let settings = { ...DEFAULT_SETTINGS };
+    let id = 0;
+    const fake = {
+        site,
+        log,
+        tables,
+        rows,
+        alerts,
+        // Not a database; a host's test that reaches `db` through the fake is
+        // testing the wrong thing, and this makes that loud.
+        get db() {
+            throw new Error('createFakeReporting has no database');
+        },
+        event(input) {
+            const v = eventInputSchema.parse(input);
+            id += 1;
+            rows.push({
+                id,
+                occurredAt: v.occurredAt ?? new Date(),
+                level: v.level,
+                kind: v.kind,
+                kindNs: v.kind.split('.')[0] ?? '',
+                site,
+                tenantId: v.tenantId ?? null,
+                actorClass: v.actor?.class ?? null,
+                actorId: v.actor?.id ?? null,
+                requestId: v.requestId ?? null,
+                targetType: v.target?.type ?? null,
+                targetId: v.target?.id ?? null,
+                message: v.message,
+                data: v.data,
+                schemaVersion: 1,
+            });
+        },
+        alert(finding) {
+            fake.event(alertRow(finding));
+            alerts.push(finding);
+        },
+        async flush() { },
+        events: {
+            async page(o = {}) {
+                const limit = Math.max(1, Math.min(o.limit ?? 50, 200));
+                let list = [...rows].sort((a, b) => b.occurredAt.getTime() - a.occurredAt.getTime() || b.id - a.id);
+                if (o.level)
+                    list = list.filter((r) => r.level === o.level);
+                if (o.kindNs)
+                    list = list.filter((r) => r.kindNs === o.kindNs);
+                if (o.tenantId)
+                    list = list.filter((r) => r.tenantId === o.tenantId);
+                if (o.site)
+                    list = list.filter((r) => r.site === o.site);
+                if (o.requestId)
+                    list = list.filter((r) => r.requestId === o.requestId);
+                if (o.after) {
+                    const { occurredAt, id: afterId } = o.after;
+                    list = list.filter((r) => r.occurredAt.getTime() < occurredAt.getTime() ||
+                        (r.occurredAt.getTime() === occurredAt.getTime() && r.id < afterId));
+                }
+                const items = list.slice(0, limit);
+                const last = items.at(-1);
+                return {
+                    items,
+                    next: list.length > limit && last ? { occurredAt: last.occurredAt, id: last.id } : null,
+                };
+            },
+        },
+        settings: {
+            async get() {
+                return settings;
+            },
+            async set(patch, by) {
+                const before = settings;
+                for (const [key, value] of Object.entries(patch)) {
+                    if (typeof value !== 'number' ||
+                        !Number.isInteger(value) ||
+                        value < RETENTION_BOUNDS.min ||
+                        value > RETENTION_BOUNDS.max) {
+                        throw new Error(`reporting.settings: ${key} out of bounds`);
+                    }
+                }
+                settings = { ...settings, ...patch };
+                const changed = Object.keys(patch).filter((k) => before[k] !== settings[k]);
+                return { before, after: settings, by, changed };
+            },
+        },
+        stats: () => ({ queued: 0, dropped: 0, invalid: 0, flushed: rows.length, failedFlushes: 0 }),
+        reset() {
+            rows.length = 0;
+            alerts.length = 0;
+            settings = { ...DEFAULT_SETTINGS };
+            id = 0;
+        },
+    };
+    return fake;
+}

package/dist/housekeeping/index.d.ts

@@ -0,0 +1,63 @@
+import type { Db, Reporting, Settings } from '../types.js';
+/**
+ * The tick, and the claim protocol that lets any number of containers share
+ * it without a scheduler.
+ *
+ * A host calls `tick()` from something that already happens often (the
+ * `./next` entry hangs it off the health check the platform sends every ten
+ * seconds). The tick debounces itself per process, then for every registered
+ * task opens one short transaction: SELECT the task's row FOR UPDATE SKIP
+ * LOCKED (no row: another container is deciding about it right now), then
+ * one conditional UPDATE that claims it only if it is due and no live lease
+ * holds it, stamping a fresh token and a lease. The task runs outside that
+ * transaction; renewing the lease and recording completion are UPDATEs fenced
+ * on the token, so a superseded worker touches zero rows.
+ *
+ * All time comparisons use the database's `now()`: two containers do not
+ * share a clock. Execution is at least once: a lease that expires under a
+ * paused worker lets its replacement start, so every task is idempotent.
+ *
+ * Not an advisory lock, which is session state on whichever pooled
+ * connection served the statement, and would have been released on another.
+ */
+export interface Task {
+    /** `namespace.name`, the same shape as an event kind. */
+    readonly name: string;
+    /** Cadence, ms. */
+    readonly every: number;
+    /** How long a claim is honoured before another container may take over, ms. Renew inside a long task. */
+    readonly lease: number;
+    /** Delay before the next attempt after a failure, ms. */
+    readonly retry: number;
+    run(ctx: TaskContext): Promise<void>;
+}
+export interface TaskContext {
+    readonly reporting: Reporting;
+    readonly db: Db;
+    readonly settings: Settings;
+    now(): Date;
+    /** Extends the lease if this claim still holds. `false` means superseded: stop scheduling work. */
+    renew(): Promise<boolean>;
+    /** The run must finish, or stop scheduling work, by this instant. */
+    readonly deadline: Date;
+}
+export interface HousekeepingOptions {
+    reporting: Reporting;
+    db?: Db;
+    now?: () => Date;
+    /** Per process. Default 60,000 ms. */
+    debounceMs?: number;
+    /** Tasks run at once. Default 2. */
+    concurrency?: number;
+}
+export interface Housekeeping {
+    /** Idempotent by name. The row is created on the next tick. */
+    register(task: Task): void;
+    /** Never throws. */
+    tick(): Promise<void>;
+    /** For tests and scripts: run the protocol for one task now, ignoring the debounce. */
+    runNow(name: string): Promise<'ran' | 'skipped' | 'failed'>;
+    tasks(): readonly Task[];
+}
+export declare function createHousekeeping(options: HousekeepingOptions): Housekeeping;
+export { pruneEvents, retentionLag } from './tasks.js';