@alexify/migronaut 2.2.0 → 2.4.0

package/CHANGELOG.md

@@ -3,6 +3,196 @@
 All notable changes to this project will be documented in this file.
 Release headings carry the publish date (`## vX.Y.Z — YYYY-MM-DD`).
 
+## v2.4.0 — 2026-10-07
+
+Migration logs for the application's users. Additive: a migration that never touches the new
+context fields, and a kit with no `migration:log` listener, behave as before. Everything new is
+experimental — its shape may still change in a minor release (named here).
+
+### Added
+
+- **`ctx.logger` in every migration** — the kit's logger with the run's correlation bound into the
+  fields of every line: `runId`, `migration`, `direction`, `batch`, `attempt`, and `jobId` /
+  `groupId` when a queue job runs it. Pino's `(fields, msg)` order is accepted too.
+- **`ctx.run`** — that correlation as a frozen object: `{ id, direction, migration?, batch?,
+  attempt?, jobId?, groupId?, requestedBy?, reason? }`. `attempt` is 2 or more when the driver
+  retried the transaction and the body runs again (each attempt gets a context of its own).
+- **The `migration:log` event** — a `ctx.logger` call whose fields hold `userland: true` is also
+  emitted, for the application to store and show its users; calls without the marker emit
+  nothing. The payload is `{ kind, runId, …correlation, level, msg, data, at, seq, truncated? }`:
+  `data` a bounded, redacted copy of the fields (8 levels, 1000 entries, 4096-character strings;
+  an `Error` as `{ name, message, code?, codeName? }` with the values a server error quotes masked,
+  as in `msg`; BSON values and binary data up to 4096 bytes kept as they are),
+  `seq` increasing within a run, `at` a `Date` (TTL-ready). It fires whatever the logger's level,
+  and with `logger: null`. Migronaut stores none of it — the
+  [Migration Logs](https://migronaut.vercel.app/guide/migration-logs) guide has the recipe.
+- **Hooks get it too** — `beforeAll`/`afterAll` the run's logger and `ctx.run` (no migration, no
+  attempt), `beforeEach`/`afterEach`/`onError` the migration's.
+- **`job: { id, groupId? }` on `up`, `down` and `redo`** — the queue job a run works for, bound into
+  `ctx.run`, its lines (the kit's own included) and its events; and `job: { id }` on
+  `runBackgroundSlice`.
+- **Queue adapter** — the processor passes each job's id and group to its run, and writes the
+  migration's `userland: true` lines into the job's log as one-line `✎ …` rows (matched by job and
+  run id, so a timed-out body never writes into the next job's log — nor into a later run of the
+  same job). A background lane does the same for its slice, into the lane job's log. The new
+  `userlandLogRows` option (`createMigrationQueue`, both processors; default 1000) caps the rows one
+  job's log takes — the rest are counted in one closing row.
+- **Background migrations** — `ctx.logger` is bound to `ctx.background`, which gains the lane's
+  `runId`, its `jobId` (and, for a lane an `up` drives with `backgroundInline`, the run's job and
+  `groupId` — its lines land in that migration job's log) and the transaction `attempt`; a
+  `userland: true` call emits `migration:log` with `kind: 'background'`. A dry run marks its lines
+  `dryRun: true` and emits nothing.
+- **Retried transactions, visible** — when the driver retried a migration's transaction and its
+  body ran again, `migration:success` / `migration:error` and the kit's `✔ Applied` / `✖ Error`
+  lines carry `attempts`, a failure's context too, and the `migronaut.migration` span always has
+  `migronaut.migration.attempts`. A run that names a `job` puts `migronaut.job.id` /
+  `migronaut.job.group_id` on its `migronaut.run` span.
+- **A `ctx.logger` call that is dropped** (a field whose getter throws) **or cut to the event's
+  bounds** leaves one debug line per run — never one per call.
+- **Types** — `MigrationRunInfo`, `JobRef`, `MigrationLogEvent` (`OrdinaryMigrationLogEvent |
+  BackgroundMigrationLogEvent`), `MigrationLogEventBase`, `MigrationLogLevel`,
+  `BackgroundRunInfo`, `MigrationLogger` (`ctx.logger`: `(msg, fields?)`, pino's `(fields, msg?)`
+  and an `Error` as the message — any `MigronautLogger` or pino instance is one); `logger?` and
+  `run?` on `MigrationContext` (optional, so a context built by hand still type-checks);
+  `attempts?` on `MigrationEvent`.
+
+### Changed
+
+- `ctx.background` of a background migration is now frozen, and typed as `BackgroundRunInfo`.
+  **A type change on an experimental API:** its `generation` is now `number | undefined` (as it
+  always was at runtime for the live drift watcher), so strict TypeScript that reads it as a
+  `number` needs a check; `ctx.logger` is typed `MigrationLogger`, which every `MigronautLogger`
+  still fits.
+- The kit's log lines of a run that names a `job` carry `jobId` / `groupId`.
+
+### Fixed
+
+- **An `async` event listener that rejects no longer crashes the process.** The kit is an
+  `EventEmitter` with `captureRejections`: a listener's rejected promise is logged at debug level,
+  like a listener that throws, instead of surfacing as an `unhandledRejection`.
+
+### Documentation
+
+- **How It Works** — a new guide page with architecture diagrams: the engine and its entry points,
+  the life of a run, the lock, where the state lives, which tool fits which change, background
+  migrations and what a run reports.
+- **Diagrams across the guides** — Core Concepts (a migration's states, batches, a run), Transactions,
+  Declared Collections (a converge run), Document Versioning (the contract, optimistic concurrency,
+  the revision invariant, expand → background → contract), Background Migrations (statuses, one
+  lane's batches, passes, drift), the queue adapter (the line, a failure, background jobs) and
+  Migration Logs. They are Mermaid, drawn in the browser by a small theme component; `mermaid` is a
+  docs-only devDependency, loaded only by a page that has a diagram.
+- **The home page** shows what 2.1 – 2.4 added — declared collections, Atlas Search, document
+  versioning, background migrations, OpenTelemetry, migration logs — with a diagram of how the parts
+  fit together.
+
+## v2.3.0 — 2026-10-06
+
+Document versioning and background migrations. Additive: nothing changes for a project that
+declares no `versioning` and writes no `background` file. Everything new is experimental — its
+shape may still change in a minor release (named here).
+
+### Added
+
+- **Document versioning** — a collection definition takes `versioning: { current, min?, field?,
+  revision?, revisionField?, index? }`: the shape version (`__v`) and the optimistic-concurrency
+  revision (`__rev`) as one contract. Converge folds it into the validator (the version an `int`
+  with a `minimum` of `min` and no `maximum`; the revision `int|long`; `moderate` when the
+  validator exists for versioning alone) and an index `{ __v: 1, _id: 1 }` — on a sharded
+  collection `{ __v: 1, …shard key, _id: 1 }`, the ordinary one kept with a row that says why.
+  Raising `min` over documents still below it is refused before any write (a `conflict` row, no
+  id named).
+- **`@alexify/migronaut/versioning`** — a third entry point, with no engine behind it, for the
+  repository layer: `defineShapes` (`stamp`, `onInsert`, `stampUpsert`, `upcaster`, `plugin`),
+  `updateWithRevision` / `replaceWithRevision` / `findOneAndUpdateWithRevision`
+  (`RevisionConflictError` with `reason: 'conflict' | 'not-found' | 'unknown'`),
+  `retryOnConflict`, `bumpRevision`, the in-memory `upcaster` (`ShapeVersionError`),
+  `isVersion`, and `versioningPlugin` for Mongoose (its version key moved to `__rev`, optimistic
+  concurrency on). Per-version shape types without codegen: `AnyShape`, `CurrentShape`,
+  `ShapeAt`, `Stamped`, `Body`, `BackgroundMigrationFor`, a typed `defineShapes<Shapes>()(…)`
+  (`versioning.d.ts` needs TypeScript ≥ 5.0).
+- **Background migrations** — a migration file with `export const background = { collection,
+  from, to, migrate | migrateBatch, revert?, … }` (or a free-form `step(ctx)`), which `up`
+  registers without running; the rewrite runs beside the migration line, never holding the
+  migration lock:
+  - **partitions and lanes**: the collection split into `_id` ranges from a sampled quantile per
+    BSON type — or, on a sharded collection, into runs of chunks per shard, with targeted reads
+    and writes, a shard-key guard (`shard-key-changed`) and `shardConcurrency`; leases are slots,
+    capped by a unique index at `maxParallel` across every process, every checkpoint fenced;
+  - **a coordinator** that plans passes, finalizes them and completes when nothing matches any
+    more (`maxPasses` against an old release that keeps writing); writes are `stampedDiff`s under
+    the optimistic filter, so an application write in between is never lost and untouched fields
+    keep their BSON types;
+  - **transactional batches** (`transaction: true`) with side writes through `ctx.session`;
+  - **pacing**: `pauseMs`, a `throttle` hook, replication lag and an adaptive (AIMD) controller;
+  - **`requires`** between background migrations (a DAG, unblocked as they complete) and from
+    ordinary migrations, which wait (`BackgroundPendingError`, or `onBackgroundPending: 'stop'`);
+    `backgroundInline` runs them inside `up`;
+  - **controls**: pause, resume, cancel, retry (`fromStart`), repin, unlock; **dry runs** on a
+    sample (`--validate` through the real write path in an always-aborted transaction) and of
+    `step` migrations in a sandbox (refusals exit 34);
+  - **drift after completion**: the drift watch (`verifyBackground`, every 10 minutes in each
+    runtime, `backgroundOnDrift: 'reopen' | 'report'`) and the **live drift watcher**
+    (`watchBackground`, change streams, one leader per collection, `backgroundDrift: 'poll' |
+    'stream' | 'both'`);
+  - **three runtimes**: `migronaut background run`, `startBackgroundRunner()` in the application,
+    and the queue (below); `status()`, `background status --partitions`, an audit check, events
+    `background:*`, and OpenTelemetry spans `migronaut.background.slice` /
+    `migronaut.background.coordinate` with their metrics.
+- **`migronaut background <action>`** — status, run, pause, resume, cancel, retry, repin,
+  dry-run, unlock, verify and watch; `create --background`.
+- **Background migrations on the queue** — `createMigrationQueue({ background: true | {…} })`: a
+  queue of their own, a coordinator job per background migration with its lanes as children,
+  heals from MongoDB (worker start, every sync tick, every drift-watch tick) and a stall
+  takeover; `startBackgroundWorker()`, `enqueueBackground()`, the `background-verify` schedule,
+  `createBackgroundProcessor()`. An `up` plan stops before a migration that waits for a
+  background one.
+- **Config**: `backgroundCollection`, `backgroundInline`, `backgroundOnDrift`, `backgroundDrift`,
+  `backgroundShardAware`, each with its `MIGRONAUT_*` variable.
+- **Errors and exit codes**: `REVISION_CONFLICT` (29), `SHAPE_VERSION_UNSUPPORTED` (30),
+  `BACKGROUND_PENDING` (31), `BACKGROUND_FAILED` (32), `BACKGROUND_CONFLICT` (33),
+  `SANDBOX_REFUSED` (34).
+- **`shapes.occ(name)`** — the revision guards and `bumpRevision` bound to a collection's field
+  names; `RevisionConflictContext` types an error's `context`.
+- **`startBackgroundRunner().stop({ timeoutMs })`**, `RunnableBackground` (what
+  `runnableBackground()` now returns: live leases, last progress, coordinator), `previous` on a
+  background status (the registration a re-registration replaced), `plan.atLeast`.
+
+### Changed
+
+For code written against 2.2 — the queue adapter's types and results, and two type-level details:
+
+- **`SyncJobResult.held` means a failure again, only.** A tick whose next migration waits for a
+  background migration reports `waiting: { migration, waitsFor }` instead; while it still waits
+  for the same thing, later ticks say so without planning. A group that stops at such a migration
+  has `upToDate: false` (`MigrationGroup.waiting` says why).
+- **`JOB_NAMES` has three more values** (`background`, `background-lane`, `background-verify`) and
+  `MigrationJobView.returnvalue` three more result types: an exhaustive `Record` over either needs
+  them.
+- **`MigronautErrorCode` has six more members** (above): an exhaustive `switch` with a `never`
+  default needs a case for each — as with the codes 2.1 added.
+- **`CollectionDefinition.indexes` and `searchIndexes` are `readonly` arrays**, so a definition
+  declared `as const` type-checks; code that pushes into a typed definition's array must copy it.
+- **Background metric names** (experimental): `migronaut.background.throttled`,
+  `migronaut.background.drift.detected`, `migronaut.background.transaction.retried`.
+
+### Notes
+
+- A typed `current` past the highest declared shape is a compile error. Written inline it reads
+  "Type 'number' is not assignable to type 'never'"; from an imported `as const` definition, it
+  names the two versions ("Type '3' is not assignable to type '2'").
+- The shard-aware mode needs `clusterMonitor` (it reads `config.collections` and
+  `config.chunks`); without it, a sharded collection is partitioned by `_id`.
+- Fail-closed limits: `maxDocumentErrors` is at most 1000 (the ids a state keeps); a revision
+  guard refuses an `_id` given as an operator; a background `filter` may not run server-side
+  JavaScript; `create --background`'s placeholder collection (`'TODO'`) is refused by `up`;
+  `watchBackground`, `enqueueBackground` and `createBackgroundProcessor` refuse options they do
+  not know.
+- A lane stopped by its process (a shutdown, a deploy) is not a failed slice, and
+  `maxSliceFailures` counts failed slices in a row. `down` of a one-way background migration pauses
+  it and waits for its lanes before it decides; a `step` migration counts as having rewritten
+  documents once one step was checkpointed.
+
 ## v2.2.0 — 2026-10-05
 
 Atlas Search and Vector Search indexes in declared collections. Additive: a definition without

package/README.md

@@ -66,6 +66,10 @@ change before it touches your database.
 - **Zero config files required** — drive everything from env vars if you prefer.
 - **Pino-friendly logging** — the `logger` option is pino-compatible; pass a pino instance directly
   and migronaut logs through it (with a `component: 'migronaut'` child binding).
+- **Migration logs for your users (experimental)** — every migration gets `ctx.logger`, bound to its
+  run (run id, migration, attempt, queue job), and `ctx.run`; a line marked `userland: true` is also
+  emitted as `migration:log` for your service to store and show — migronaut stores none of it
+  ([guide](https://migronaut.vercel.app/guide/migration-logs)).
 - **Your id format** — run ids and queue group ids are random UUIDs by default; pass
   `generateId: ulid` (or cuid2, nanoid, UUIDv7 — any `() => string`) and every id migronaut mints
   comes from your generator.
@@ -75,6 +79,15 @@ change before it touches your database.
 - **Migrations as a queue (optional)** — `@alexify/migronaut/bullmq` runs each migration as its own
   BullMQ job, in order, so migronaut can be a migration service: trigger it over HTTP, on a
   schedule, or from a deploy hook that waits for the result.
+- **Document versioning (experimental)** — declare a collection's shape version (`__v`) and
+  optimistic-concurrency revision (`__rev`) once; converge enforces them, and
+  `@alexify/migronaut/versioning` gives your repository layer `updateWithRevision`, typed
+  per-version shapes, an upcaster and a Mongoose plugin ([guide](https://migronaut.vercel.app/guide/versioning)).
+- **Background migrations (experimental)** — long data rewrites (`export const background = {…}`)
+  that `up` only registers and that run beside the migration line: partitions and parallel lanes
+  across pods, checkpoints, pause/resume, transactional batches, shard-aware on sharded clusters,
+  dry runs, and a drift watcher that upgrades old-shape writes after completion — from the CLI,
+  inside your app, or on the queue ([guide](https://migronaut.vercel.app/guide/background-migrations)).
 
 ### How it compares to `migrate-mongo`
 
@@ -180,7 +193,7 @@ Full docs, guides, and the API reference live at
 - [Core Concepts](https://migronaut.vercel.app/guide/concepts) — migrations, batches, the changelog, locking
 - [Getting Started](https://migronaut.vercel.app/guide/getting-started) & [Tutorial](https://migronaut.vercel.app/guide/tutorial)
 - [Configuration](https://migronaut.vercel.app/guide/configuration) · [Writing Migrations](https://migronaut.vercel.app/guide/writing-migrations) · [Transactions](https://migronaut.vercel.app/guide/transactions) · [Hooks](https://migronaut.vercel.app/guide/hooks)
-- [Programmatic API](https://migronaut.vercel.app/guide/api) · [Migrations as a Queue (BullMQ)](https://migronaut.vercel.app/guide/bullmq) · [OpenTelemetry](https://migronaut.vercel.app/guide/opentelemetry) · [CI/CD](https://migronaut.vercel.app/guide/ci-cd) · [Troubleshooting](https://migronaut.vercel.app/guide/troubleshooting)
+- [Programmatic API](https://migronaut.vercel.app/guide/api) · [Migrations as a Queue (BullMQ)](https://migronaut.vercel.app/guide/bullmq) · [Migration Logs](https://migronaut.vercel.app/guide/migration-logs) · [OpenTelemetry](https://migronaut.vercel.app/guide/opentelemetry) · [CI/CD](https://migronaut.vercel.app/guide/ci-cd) · [Troubleshooting](https://migronaut.vercel.app/guide/troubleshooting)
 - Reference: [CLI Cheatsheet](https://migronaut.vercel.app/reference/cli) · [Error Codes](https://migronaut.vercel.app/reference/error-codes)
 
 ---
@@ -200,6 +213,7 @@ Every command accepts the global flags `--uri`, `--db`, `--dir`, `--config`, `--
 | `migronaut down [file]` | Roll back the last batch, a chosen batch, the last N steps, one file, or to `--to <file>` |
 | `migronaut redo [file]` | Roll back then re-apply (the last migration, or one file) |
 | `migronaut converge` | Bring declared collections — indexes, search indexes and validators — to their declared state |
+| `migronaut background <action> [name]` | Background migrations: `status`, `run`, `pause`, `resume`, `cancel`, `retry`, `repin`, `dry-run`, `unlock`, `verify`, `watch` |
 | `migronaut status` | Print the full migration status table (`--check` to fail CI on pending) |
 | `migronaut list` | List migrations, filtered by status |
 | `migronaut dry-run <up\|down> [file]` | Preview a run without touching the database |
@@ -207,8 +221,8 @@ Every command accepts the global flags `--uri`, `--db`, `--dir`, `--config`, `--
 | `migronaut lock` | Show who currently holds the migration lock |
 | `migronaut unlock` | Force-release a stuck lock left behind by a crashed run |
 
-Most data commands (`up`, `down`, `redo`, `converge`, `status`, `list`, `dry-run`, `import`,
-`baseline`, `create`, `audit`, `lock`, `unlock`) accept **`--json`** for machine-readable output — see
+Most data commands (`up`, `down`, `redo`, `converge`, `background`, `status`, `list`, `dry-run`,
+`import`, `baseline`, `create`, `audit`, `lock`, `unlock`) accept **`--json`** for machine-readable output — see
 [CI & automation](#ci--automation).
 
 <details>
@@ -285,6 +299,15 @@ migronaut converge --yes           # no confirmation (required for drops/rebuild
 migronaut converge --no-lock       # skip the concurrency lock (local dev only)
 migronaut converge --json          # machine-readable output (the converge result)
 
+# background — background migrations (registered by up, rewritten in partitions)
+migronaut background status                 # every background migration
+migronaut background status --check         # exit 32 if one failed, 31 if one is not completed
+migronaut background run <name>             # drive it from here (--concurrency N, --once, --all)
+migronaut background pause <name> --wait    # stop its lanes at the next batch
+migronaut background dry-run <name> --validate   # on a sample, in an always-aborted transaction
+migronaut background verify                 # look for old-shape documents after completion
+migronaut background watch                  # upgrade old-shape writes as they land (Ctrl-C stops)
+
 # status — full status table
 migronaut status                   # the full status table
 migronaut status --check           # exit 2 if any migration is pending (CI gate)
@@ -734,6 +757,9 @@ await mq.enqueueConverge();              // declared indexes and validators, as
   failed one. Duplicate enqueues are deduplicated; an already-applied migration completes as
   `skipped`.
 - **Bring your own Worker** with `createMigrationProcessor()` (NestJS, BullMQ Pro).
+- **What a migration logs for your users** (`logger.info(…, { userland: true })`) lands in the
+  job's log and reaches `mq.kit.on('migration:log', …)` with the job's id — keep it in a
+  collection of your own ([Migration Logs](https://migronaut.vercel.app/guide/migration-logs)).
 
 → **[Migrations as a Queue](https://migronaut.vercel.app/guide/bullmq)** ·
 [runnable example service](examples/migration-service)
@@ -786,6 +812,13 @@ export default {
   // onSearchUnavailable: 'fail',     // 'skip' → converge without search indexes where Search is absent
   // waitForSearchIndexes: false,     // true → converge waits until search indexes are queryable
 
+  // ── Background migrations (experimental) — see `migronaut background` ───
+  // backgroundCollection: '_migronaut_background', // state (+ _partitions, _watch)
+  // backgroundInline: false,         // true → run it to the end inside the registering `up`
+  // backgroundOnDrift: 'reopen',     // 'report' → only report old-shape documents after completion
+  // backgroundDrift: 'poll',         // 'stream' | 'both' → watch drift with change streams
+  // backgroundShardAware: 'auto',    // 'off' → no shard-key partitions on sharded collections
+
   // ── Code-only options (omit in migronaut.config.json) ─────────────────────────
   // hooks: { beforeAll, afterAll, beforeEach, afterEach, onError },
   // mongoose: myMongooseInstance, // pass if your migrations use Mongoose models
@@ -923,6 +956,11 @@ optional rather than merely discouraged:
 | `MIGRONAUT_ON_SEARCH_UNAVAILABLE` | `onSearchUnavailable` | `fail` |
 | `MIGRONAUT_WAIT_FOR_SEARCH_INDEXES` | `waitForSearchIndexes` | `false` |
 | `MIGRONAUT_SEARCH_INDEX_WAIT_TIMEOUT_MS` | `searchIndexWaitTimeoutMs` | `600000` |
+| `MIGRONAUT_BACKGROUND_COLLECTION` | `backgroundCollection` | `_migronaut_background` |
+| `MIGRONAUT_BACKGROUND_INLINE` | `backgroundInline` | `false` |
+| `MIGRONAUT_BACKGROUND_ON_DRIFT` | `backgroundOnDrift` | `reopen` |
+| `MIGRONAUT_BACKGROUND_DRIFT` | `backgroundDrift` | `poll` |
+| `MIGRONAUT_BACKGROUND_SHARD_AWARE` | `backgroundShardAware` | `auto` |
 | `MIGRONAUT_ENV_FILE` | `envFile` | `.env` |
 
 `fileExtensions`, `clientOptions`, `collections`, `client`, `mongoose`, `hooks`, `logger`,