@alexify/migronaut 2.2.0 → 2.3.0

package/CHANGELOG.md

@@ -3,6 +3,113 @@
 All notable changes to this project will be documented in this file.
 Release headings carry the publish date (`## vX.Y.Z — YYYY-MM-DD`).
 
+## v2.3.0 — 2026-10-06
+
+Document versioning and background migrations. Additive: nothing changes for a project that
+declares no `versioning` and writes no `background` file. Everything new is experimental — its
+shape may still change in a minor release (named here).
+
+### Added
+
+- **Document versioning** — a collection definition takes `versioning: { current, min?, field?,
+  revision?, revisionField?, index? }`: the shape version (`__v`) and the optimistic-concurrency
+  revision (`__rev`) as one contract. Converge folds it into the validator (the version an `int`
+  with a `minimum` of `min` and no `maximum`; the revision `int|long`; `moderate` when the
+  validator exists for versioning alone) and an index `{ __v: 1, _id: 1 }` — on a sharded
+  collection `{ __v: 1, …shard key, _id: 1 }`, the ordinary one kept with a row that says why.
+  Raising `min` over documents still below it is refused before any write (a `conflict` row, no
+  id named).
+- **`@alexify/migronaut/versioning`** — a third entry point, with no engine behind it, for the
+  repository layer: `defineShapes` (`stamp`, `onInsert`, `stampUpsert`, `upcaster`, `plugin`),
+  `updateWithRevision` / `replaceWithRevision` / `findOneAndUpdateWithRevision`
+  (`RevisionConflictError` with `reason: 'conflict' | 'not-found' | 'unknown'`),
+  `retryOnConflict`, `bumpRevision`, the in-memory `upcaster` (`ShapeVersionError`),
+  `isVersion`, and `versioningPlugin` for Mongoose (its version key moved to `__rev`, optimistic
+  concurrency on). Per-version shape types without codegen: `AnyShape`, `CurrentShape`,
+  `ShapeAt`, `Stamped`, `Body`, `BackgroundMigrationFor`, a typed `defineShapes<Shapes>()(…)`
+  (`versioning.d.ts` needs TypeScript ≥ 5.0).
+- **Background migrations** — a migration file with `export const background = { collection,
+  from, to, migrate | migrateBatch, revert?, … }` (or a free-form `step(ctx)`), which `up`
+  registers without running; the rewrite runs beside the migration line, never holding the
+  migration lock:
+  - **partitions and lanes**: the collection split into `_id` ranges from a sampled quantile per
+    BSON type — or, on a sharded collection, into runs of chunks per shard, with targeted reads
+    and writes, a shard-key guard (`shard-key-changed`) and `shardConcurrency`; leases are slots,
+    capped by a unique index at `maxParallel` across every process, every checkpoint fenced;
+  - **a coordinator** that plans passes, finalizes them and completes when nothing matches any
+    more (`maxPasses` against an old release that keeps writing); writes are `stampedDiff`s under
+    the optimistic filter, so an application write in between is never lost and untouched fields
+    keep their BSON types;
+  - **transactional batches** (`transaction: true`) with side writes through `ctx.session`;
+  - **pacing**: `pauseMs`, a `throttle` hook, replication lag and an adaptive (AIMD) controller;
+  - **`requires`** between background migrations (a DAG, unblocked as they complete) and from
+    ordinary migrations, which wait (`BackgroundPendingError`, or `onBackgroundPending: 'stop'`);
+    `backgroundInline` runs them inside `up`;
+  - **controls**: pause, resume, cancel, retry (`fromStart`), repin, unlock; **dry runs** on a
+    sample (`--validate` through the real write path in an always-aborted transaction) and of
+    `step` migrations in a sandbox (refusals exit 34);
+  - **drift after completion**: the drift watch (`verifyBackground`, every 10 minutes in each
+    runtime, `backgroundOnDrift: 'reopen' | 'report'`) and the **live drift watcher**
+    (`watchBackground`, change streams, one leader per collection, `backgroundDrift: 'poll' |
+    'stream' | 'both'`);
+  - **three runtimes**: `migronaut background run`, `startBackgroundRunner()` in the application,
+    and the queue (below); `status()`, `background status --partitions`, an audit check, events
+    `background:*`, and OpenTelemetry spans `migronaut.background.slice` /
+    `migronaut.background.coordinate` with their metrics.
+- **`migronaut background <action>`** — status, run, pause, resume, cancel, retry, repin,
+  dry-run, unlock, verify and watch; `create --background`.
+- **Background migrations on the queue** — `createMigrationQueue({ background: true | {…} })`: a
+  queue of their own, a coordinator job per background migration with its lanes as children,
+  heals from MongoDB (worker start, every sync tick, every drift-watch tick) and a stall
+  takeover; `startBackgroundWorker()`, `enqueueBackground()`, the `background-verify` schedule,
+  `createBackgroundProcessor()`. An `up` plan stops before a migration that waits for a
+  background one.
+- **Config**: `backgroundCollection`, `backgroundInline`, `backgroundOnDrift`, `backgroundDrift`,
+  `backgroundShardAware`, each with its `MIGRONAUT_*` variable.
+- **Errors and exit codes**: `REVISION_CONFLICT` (29), `SHAPE_VERSION_UNSUPPORTED` (30),
+  `BACKGROUND_PENDING` (31), `BACKGROUND_FAILED` (32), `BACKGROUND_CONFLICT` (33),
+  `SANDBOX_REFUSED` (34).
+- **`shapes.occ(name)`** — the revision guards and `bumpRevision` bound to a collection's field
+  names; `RevisionConflictContext` types an error's `context`.
+- **`startBackgroundRunner().stop({ timeoutMs })`**, `RunnableBackground` (what
+  `runnableBackground()` now returns: live leases, last progress, coordinator), `previous` on a
+  background status (the registration a re-registration replaced), `plan.atLeast`.
+
+### Changed
+
+For code written against 2.2 — the queue adapter's types and results, and two type-level details:
+
+- **`SyncJobResult.held` means a failure again, only.** A tick whose next migration waits for a
+  background migration reports `waiting: { migration, waitsFor }` instead; while it still waits
+  for the same thing, later ticks say so without planning. A group that stops at such a migration
+  has `upToDate: false` (`MigrationGroup.waiting` says why).
+- **`JOB_NAMES` has three more values** (`background`, `background-lane`, `background-verify`) and
+  `MigrationJobView.returnvalue` three more result types: an exhaustive `Record` over either needs
+  them.
+- **`MigronautErrorCode` has six more members** (above): an exhaustive `switch` with a `never`
+  default needs a case for each — as with the codes 2.1 added.
+- **`CollectionDefinition.indexes` and `searchIndexes` are `readonly` arrays**, so a definition
+  declared `as const` type-checks; code that pushes into a typed definition's array must copy it.
+- **Background metric names** (experimental): `migronaut.background.throttled`,
+  `migronaut.background.drift.detected`, `migronaut.background.transaction.retried`.
+
+### Notes
+
+- A typed `current` past the highest declared shape is a compile error. Written inline it reads
+  "Type 'number' is not assignable to type 'never'"; from an imported `as const` definition, it
+  names the two versions ("Type '3' is not assignable to type '2'").
+- The shard-aware mode needs `clusterMonitor` (it reads `config.collections` and
+  `config.chunks`); without it, a sharded collection is partitioned by `_id`.
+- Fail-closed limits: `maxDocumentErrors` is at most 1000 (the ids a state keeps); a revision
+  guard refuses an `_id` given as an operator; a background `filter` may not run server-side
+  JavaScript; `create --background`'s placeholder collection (`'TODO'`) is refused by `up`;
+  `watchBackground`, `enqueueBackground` and `createBackgroundProcessor` refuse options they do
+  not know.
+- A lane stopped by its process (a shutdown, a deploy) is not a failed slice, and
+  `maxSliceFailures` counts failed slices in a row. `down` of a one-way background migration pauses
+  it and waits for its lanes before it decides; a `step` migration counts as having rewritten
+  documents once one step was checkpointed.
+
 ## v2.2.0 — 2026-10-05
 
 Atlas Search and Vector Search indexes in declared collections. Additive: a definition without

package/README.md

@@ -75,6 +75,15 @@ change before it touches your database.
 - **Migrations as a queue (optional)** — `@alexify/migronaut/bullmq` runs each migration as its own
   BullMQ job, in order, so migronaut can be a migration service: trigger it over HTTP, on a
   schedule, or from a deploy hook that waits for the result.
+- **Document versioning (experimental)** — declare a collection's shape version (`__v`) and
+  optimistic-concurrency revision (`__rev`) once; converge enforces them, and
+  `@alexify/migronaut/versioning` gives your repository layer `updateWithRevision`, typed
+  per-version shapes, an upcaster and a Mongoose plugin ([guide](https://migronaut.vercel.app/guide/versioning)).
+- **Background migrations (experimental)** — long data rewrites (`export const background = {…}`)
+  that `up` only registers and that run beside the migration line: partitions and parallel lanes
+  across pods, checkpoints, pause/resume, transactional batches, shard-aware on sharded clusters,
+  dry runs, and a drift watcher that upgrades old-shape writes after completion — from the CLI,
+  inside your app, or on the queue ([guide](https://migronaut.vercel.app/guide/background-migrations)).
 
 ### How it compares to `migrate-mongo`
 
@@ -200,6 +209,7 @@ Every command accepts the global flags `--uri`, `--db`, `--dir`, `--config`, `--
 | `migronaut down [file]` | Roll back the last batch, a chosen batch, the last N steps, one file, or to `--to <file>` |
 | `migronaut redo [file]` | Roll back then re-apply (the last migration, or one file) |
 | `migronaut converge` | Bring declared collections — indexes, search indexes and validators — to their declared state |
+| `migronaut background <action> [name]` | Background migrations: `status`, `run`, `pause`, `resume`, `cancel`, `retry`, `repin`, `dry-run`, `unlock`, `verify`, `watch` |
 | `migronaut status` | Print the full migration status table (`--check` to fail CI on pending) |
 | `migronaut list` | List migrations, filtered by status |
 | `migronaut dry-run <up\|down> [file]` | Preview a run without touching the database |
@@ -207,8 +217,8 @@ Every command accepts the global flags `--uri`, `--db`, `--dir`, `--config`, `--
 | `migronaut lock` | Show who currently holds the migration lock |
 | `migronaut unlock` | Force-release a stuck lock left behind by a crashed run |
 
-Most data commands (`up`, `down`, `redo`, `converge`, `status`, `list`, `dry-run`, `import`,
-`baseline`, `create`, `audit`, `lock`, `unlock`) accept **`--json`** for machine-readable output — see
+Most data commands (`up`, `down`, `redo`, `converge`, `background`, `status`, `list`, `dry-run`,
+`import`, `baseline`, `create`, `audit`, `lock`, `unlock`) accept **`--json`** for machine-readable output — see
 [CI & automation](#ci--automation).
 
 <details>
@@ -285,6 +295,15 @@ migronaut converge --yes           # no confirmation (required for drops/rebuild
 migronaut converge --no-lock       # skip the concurrency lock (local dev only)
 migronaut converge --json          # machine-readable output (the converge result)
 
+# background — background migrations (registered by up, rewritten in partitions)
+migronaut background status                 # every background migration
+migronaut background status --check         # exit 32 if one failed, 31 if one is not completed
+migronaut background run <name>             # drive it from here (--concurrency N, --once, --all)
+migronaut background pause <name> --wait    # stop its lanes at the next batch
+migronaut background dry-run <name> --validate   # on a sample, in an always-aborted transaction
+migronaut background verify                 # look for old-shape documents after completion
+migronaut background watch                  # upgrade old-shape writes as they land (Ctrl-C stops)
+
 # status — full status table
 migronaut status                   # the full status table
 migronaut status --check           # exit 2 if any migration is pending (CI gate)
@@ -786,6 +805,13 @@ export default {
   // onSearchUnavailable: 'fail',     // 'skip' → converge without search indexes where Search is absent
   // waitForSearchIndexes: false,     // true → converge waits until search indexes are queryable
 
+  // ── Background migrations (experimental) — see `migronaut background` ───
+  // backgroundCollection: '_migronaut_background', // state (+ _partitions, _watch)
+  // backgroundInline: false,         // true → run it to the end inside the registering `up`
+  // backgroundOnDrift: 'reopen',     // 'report' → only report old-shape documents after completion
+  // backgroundDrift: 'poll',         // 'stream' | 'both' → watch drift with change streams
+  // backgroundShardAware: 'auto',    // 'off' → no shard-key partitions on sharded collections
+
   // ── Code-only options (omit in migronaut.config.json) ─────────────────────────
   // hooks: { beforeAll, afterAll, beforeEach, afterEach, onError },
   // mongoose: myMongooseInstance, // pass if your migrations use Mongoose models
@@ -923,6 +949,11 @@ optional rather than merely discouraged:
 | `MIGRONAUT_ON_SEARCH_UNAVAILABLE` | `onSearchUnavailable` | `fail` |
 | `MIGRONAUT_WAIT_FOR_SEARCH_INDEXES` | `waitForSearchIndexes` | `false` |
 | `MIGRONAUT_SEARCH_INDEX_WAIT_TIMEOUT_MS` | `searchIndexWaitTimeoutMs` | `600000` |
+| `MIGRONAUT_BACKGROUND_COLLECTION` | `backgroundCollection` | `_migronaut_background` |
+| `MIGRONAUT_BACKGROUND_INLINE` | `backgroundInline` | `false` |
+| `MIGRONAUT_BACKGROUND_ON_DRIFT` | `backgroundOnDrift` | `reopen` |
+| `MIGRONAUT_BACKGROUND_DRIFT` | `backgroundDrift` | `poll` |
+| `MIGRONAUT_BACKGROUND_SHARD_AWARE` | `backgroundShardAware` | `auto` |
 | `MIGRONAUT_ENV_FILE` | `envFile` | `.env` |
 
 `fileExtensions`, `clientOptions`, `collections`, `client`, `mongoose`, `hooks`, `logger`,