@alexify/migronaut 2.0.0 → 2.1.0

package/CHANGELOG.md

@@ -3,6 +3,326 @@
 All notable changes to this project will be documented in this file.
 Release headings carry the publish date (`## vX.Y.Z — YYYY-MM-DD`).
 
+## v2.1.0 — 2026-10-04
+
+Migrations as a queue, ids in your own format, OpenTelemetry, and declared collections. Additive:
+nothing changes for anyone who uses none of them, with the narrow exceptions listed under
+**Changed**.
+
+### Added
+
+- **`@alexify/migronaut/bullmq`** — a new entry point that runs migrations as
+  [BullMQ](https://docs.bullmq.io/) jobs, **one migration per job**, so migronaut can be a
+  migration service: enqueue from an HTTP handler, a schedule or a deploy hook, and let a worker
+  apply them in order.
+  - `createMigrationQueue(options)` — the facade: `enqueueUp` / `enqueueDown` (returning a group
+    handle with `wait()`), `startWorker`, `status` / `pending` / `audit` / `lockInfo`, `getJob`,
+    `pause` / `resume`, `schedule` / `unschedule`, `close`.
+  - `createMigrationProcessor(options)` — the processor on its own, for a Worker you construct
+    (NestJS, BullMQ Pro), plus `enqueueUp` / `enqueueDown` / `planUpJobs` / `planDownJobs` /
+    `waitForGroup` for a Queue you own.
+  - **BullMQ is injected, never depended on** — `bullmq: { Queue, Worker, QueueEvents }` from
+    your own install. The package still has no `dependencies` and gains no peer; `src/` never
+    imports `bullmq` (a test enforces it).
+  - **Order comes from MongoDB, not from Redis.** Every job is a single-file run under the usual
+    lock; it refuses while an earlier migration is still pending, so a failed migration stops the
+    line (`MIGRATION_BLOCKED` for the jobs behind it). Jobs get one attempt on purpose — a BullMQ
+    retry re-queues behind the waiting jobs — and a held lock is waited out inside the job.
+  - **One batch per enqueue**, so `down` still rolls back a whole deploy; duplicate enqueues are
+    deduplicated, and a job whose migration is already applied completes as `skipped`.
+  - Job payloads are validated as untrusted input; messages, stacks and job logs are redacted. A
+    queue job's target must be a file of the migration sequence — a payload can never make the
+    worker import a dotfile, a declaration file or a helper module next to the migrations.
+  - **Shutdown puts unstarted work back.** A job that a closing worker stops before its migration
+    starts (waiting for the lock, or fetched during shutdown) is moved back to the head of the
+    queue instead of failing — so a rolling deploy no longer fails the rest of the enqueue it
+    interrupts as `MIGRATION_BLOCKED`. A migration already running always finishes.
+  - **A versioned, strict job contract.** A worker accepts every job data version from
+    `MIN_JOB_DATA_VERSION` (exported) up to its own and refuses newer ones and unknown fields
+    rather than ignoring what they mean — roll workers out before producers. Jobs always state
+    `ordered`, and may carry the plan-time `checksum` of their file.
+  - **Scheduler ticks are bounded**: they carry the queue's `jobOptions`, and keep the last 100
+    completed / 500 failed jobs when those set no retention.
+  - **Correlation**: a job's `runId` is on its progress (`completed` and `failed`), its failure log
+    row and its error's `context` (with `jobId` and `groupId`) — a failed job has no return value;
+    the worker's failure log line names the group, migration and run.
+  - An injected `Queue` (or `QueueEvents`) on another name or prefix than the facade's is rejected,
+    and the facade takes an injected queue's prefix by default; `startWorker()` can be retried
+    after a failed start; a closed queue refuses every further call.
+  - **A worker decides what a job may ask for** — the `allow` option (`{ down: true, force:
+    false, unordered: false }` by default) on `createMigrationProcessor` and
+    `createMigrationQueue`: a payload asking to re-run an applied migration or to skip the order
+    guard is refused (`QUEUE_JOB_INVALID`, `context.permission`) unless allowed. The facade's
+    `enqueue*` calls follow the same policy.
+  - **A job runs only the file it was planned with**: an `up` job carries the plan-time checksum,
+    and a worker with another version of the file fails it (`CHECKSUM_MISMATCH`,
+    `context.planned`) instead of applying it.
+  - **Several workers without global concurrency**: a job blocked only by earlier migrations that
+    have not failed (one may be in flight on another worker) waits for them within its lock-wait
+    budget instead of failing its group; `MigrationBlockedError` carries `context.failed`.
+  - `wait()` holds every await to its one `timeoutMs` budget, decides a timeout by the clock, and
+    its `QueueJobFailedError` carries the job's own typed `code`; an empty group can be waited for
+    without QueueEvents. A long lock wait reports progress every few seconds, not every poll.
+  - What the queue stores about a failure — `failedReason`, stack, job logs — masks the values a
+    duplicate-key error quotes, on top of credentials. `schedule({ every })` needs at least 1000 ms.
+  - Dedup ids encode file names reversibly (two names can no longer share one and absorb each
+    other's job), and a forced re-run has a dedup id of its own.
+  - **A schedule holds a failed migration**: a `sync` tick whose next migration failed, with its
+    file unchanged since, enqueues nothing (`returnvalue.held`, and a warning) instead of re-running
+    it every tick; a changed file or an explicit `enqueueUp(name)` resumes it.
+  - Jobs carry `requestedBy` / `reason` (`enqueueUp` / `enqueueDown` / `enqueueConverge` options).
+- **`bullmq.d.ts`** — hand-written types for the entry point, with structural `BullMQ*Like`
+  interfaces instead of an import of `bullmq`, generic over the classes you inject.
+- **`up(file, { batch })`** — stamp an explicit batch number instead of the next free one, and
+  **`MigratorKit.nextBatch()`** to peek at it: together they let several single-file runs form one
+  batch.
+- **`up(file, { ordered: true })` / `down(file, { ordered: true })`** — refuse a single-file run
+  that would go out of sequence (`MigrationBlockedError`); an ordered `up` also applies the
+  `strict` drift check and the `onOutOfOrder` policy a bulk run would, and only targets a file of
+  the migration sequence.
+- **`up(file, { checksum })`** — refuse (`CHECKSUM_MISMATCH`, `context.planned`) to apply any other
+  version of the file than the one with this SHA-256; `dryRun('up')` rows carry each file's
+  `checksum` for it.
+- **`list(filter, { checksums: false })`** / **`status({ checksums: false })`** — skip hashing the
+  applied files, for a caller that needs names and dates only.
+- **Who asked, and why** — `up`, `down`, `redo` and `converge` take `requestedBy` (≤ 128
+  characters) and `reason` (≤ 512), and `migronaut up` / `down` / `redo` / `converge` take
+  `--reason`. They are stamped on the changelog (`requestedBy` / `reason` on an apply,
+  `revertRequestedBy` / `revertReason` on a revert; a later apply that says nothing clears the old
+  ones) and on the converge history; `status()` rows show them. `executedBy` stays the OS user —
+  on a queue worker, the container's — which is why the requester has fields of its own. A failed
+  attempt's trace now also records the checksum of the file version that failed
+  (`StatusRow.failedChecksum`).
+- **`runMigrations(config, { signal })`** — an `AbortSignal` (wired to SIGTERM) stops a wait for
+  the lock between polls, and a run that holds it between migrations.
+- **`LockInfo.runId` and `LockInfo.ttlMs`** — which run holds the lock (also on `migronaut lock`),
+  and the holder's TTL, which the lock document now records.
+- **Three error codes**: `MIGRATION_BLOCKED` (exit 24), `QUEUE_JOB_INVALID` (25),
+  `QUEUE_JOB_FAILED` (26), with `MigrationBlockedError`, `QueueJobInvalidError` and
+  `QueueJobFailedError` exported from the package root.
+- **Runnable example** — `examples/migration-service`: a queue, a worker and a plain `node:http`
+  API (not published to npm).
+- **`generateId` config option** — your own identifier format (ULID, CUID, nanoid, UUIDv7, …)
+  everywhere migronaut used to call `crypto.randomUUID()`: the `runId` of every run — on changelog
+  records, events, log lines and the lock's owner token — and the `groupId` of every queue
+  enqueue. One option covers the kit, the CLI (through `migronaut.config.js`/`.ts`),
+  `runMigrations` and the queue adapter.
+  - **Injected, like the logger** — migronaut ships no generator but the default. It is called
+    with no arguments, so third-party functions pass straight through: `generateId: ulid`.
+  - **Checked on every call** — it must synchronously return a non-empty string of at most 128
+    characters. A throw, a promise or anything else fails the run with `CONFIG_INVALID` before a
+    migration starts (and an enqueue before a job is added).
+  - Code-only: no environment variable and no place in a JSON config, like `logger` and `hooks`.
+- **`MigratorKit.generateId()`** — a new id in the kit's configured format, for code that wants its
+  own ids to match (it is how the queue adapter mints group ids). Resolves the config; does not
+  connect.
+- **`IdGenerator` type** — `() => string`, exported from the package root.
+- **`telemetry` config option** — OpenTelemetry traces and metrics through a tracer and/or a meter
+  from your own `@opentelemetry/api`: `telemetry: { tracer, meter }`. One option covers the kit,
+  the CLI (through `migronaut.config.js`/`.ts`), `runMigrations` and the queue adapter.
+  - **Spans**: `migronaut.run` for every run that acquired the lock, and a child
+    `migronaut.migration` for every migration executed. The migration's span is the *active* one
+    while its hooks, its `up`/`down` and its changelog write run — so an instrumented MongoDB
+    driver nests its command spans under the migration that issued them. That is what lifecycle
+    events cannot do, and why this lives in the kit: at application startup there is no ambient
+    span, and the driver instrumentation records nothing without a parent.
+  - **Metrics**: `migronaut.run.duration`, `migronaut.migration.duration`,
+    `migronaut.lock.acquire.duration` and `migronaut.lock.wait.duration` (one point per wait for a
+    held lock, by `migronaut.lock.wait.outcome`: `acquired`, `timeout`, `aborted`) — histograms,
+    in seconds, with boundaries from 10ms to an hour — and the counters `migronaut.lock.refused`
+    and `migronaut.lock.lost`.
+  - **Dimensions**: every span and metric point carries `db.namespace` (the database name), plus
+    the caller's own static attributes from `telemetry.attributes` (at most 20 scalars).
+  - **Failures** set the span's status to `ERROR` with a redacted message (credentials and the
+    values a duplicate-key error quotes masked, at most 1 KB), and `error.type` — on
+    the span and on the metric point — to the typed error code. No exception event is recorded: it
+    would carry the unredacted message and stack.
+  - **A run that never got the lock emits no span.** A caller polling for a busy lock retries the
+    whole run every few hundred milliseconds; the refusals are counted
+    (`migronaut.lock.refused`) instead.
+  - **Injected, like the logger** — `@opentelemetry/api` is neither a dependency nor a peer, and
+    `src/` never imports it (a test enforces it). The types are structural: `MigronautTracer`,
+    `MigronautSpan`, `MigronautMeter`, `MigronautHistogram`, `MigronautCounter`,
+    `MigronautMetricOptions`, `MigronautAttributes` and `MigronautTelemetry`, exported from the
+    package root.
+  - **It can never fail a run.** Every call into the tracer, a span, the meter and an instrument
+    is guarded — a promise one of them returns included, so a rejecting SDK cannot surface as an
+    unhandled rejection — and a tracer that throws before or after running the work, or runs it
+    twice, still gets each migration executed exactly once.
+  - Code-only, like `logger` and `generateId`. The span, attribute and metric names are new and
+    should be treated as experimental.
+- **`bullmq.telemetry`** — `createMigrationQueue({ bullmq: { Queue, Worker, telemetry } })` hands
+  BullMQ's own telemetry object (`new BullMQOtel({ tracerName })` from `bullmq-otel`) to the Queue
+  and the Worker it constructs. Together with the kit's `telemetry` it gives one trace from the
+  request that enqueued, across Redis, to the MongoDB commands in the worker. Until now the facade
+  built its Queue without it, so the enqueuing side of that trace could not be joined.
+  `workerOptions.telemetry` and `startWorker({ telemetry })` override it for the worker alone.
+- **OpenTelemetry in the example** — `examples/migration-service` gains `tracing.js` and a Jaeger
+  in its `docker-compose.yml`: set `OTEL_EXPORTER_OTLP_ENDPOINT` and every enqueue is one trace.
+- **Declared collections** — indexes and validators declared as an end state, and applied by
+  `migronaut converge`, with no migration file per change. For what only ever has a current value
+  (which indexes a collection has, which validator guards it); migrations stay the tool for
+  changes with an order and a history.
+  - **`collections` config option** — an array of definitions, `{ name, indexes?, validator?,
+    validationLevel?, validationAction?, prune? }`, each index in the driver's own flat
+    `createIndexes` shape. Works in `migronaut.config.{ts,js,json}` (and the JSON Schema) and in
+    `new MigratorKit({ collections })`.
+  - **`collectionsDir` config option** (`MIGRONAUT_COLLECTIONS_DIR`) — one definition file per
+    collection (`.ts`/`.js` default export, or `.json`; the name defaults to the file name).
+    Opt-in, combined with `collections`, and loaded only when a converge runs — a broken file never
+    blocks `status` or an emergency `down`. A collection declared twice is `CONFIG_INVALID`.
+  - **Validated strictly**: an unknown definition key or index option is an error with its path
+    (`collections[2].indexes[0].uniqe`) — the driver silently drops an option it does not know, so
+    a typo would otherwise build the wrong index and then look in sync forever.
+  - **`MigratorKit.converge({ dryRun?, prune?, noLock?, ordered? })`** and **`migronaut converge`**
+    (`--dry-run`, `--check`, `--prune`, `--yes`). Stateless: every run reads `listCollections` and
+    `listIndexes`, plans, and carries the plan out one step at a time under the migration lock —
+    create the collection, set the validator, create indexes, `collMod` a TTL or `hidden` in place,
+    rebuild what changed otherwise, drop last. Nothing is recorded.
+  - **Safe by default.** An index you did not declare is kept and reported (`keep`), and dropped
+    only with `prune` (per definition, or for the definitions that do not decide). An identical
+    index under another name is accepted as is rather than rebuilt; a different one covering the
+    same key is a conflict that refuses the run before the first write. A rebuild whose new index
+    fails to build puts the old one back — and says so, with the reason, when it cannot.
+  - **A unique index is never rebuilt unasked.** Dropping it opens a window with no constraint,
+    and a duplicate written in that window leaves neither index buildable. Such a rebuild is a
+    `conflict` unless `converge({ rebuildUnique: true })` / `--rebuild-unique`; a converge after
+    `up` and a queue job never pass it.
+  - **Comparisons follow what the server stores** — text indexes in their `_fts` form, collations
+    field by field against the expanded spec (`strength`, `caseLevel` and `numericOrdering` at
+    their universal defaults when left out), the collection's default collation,
+    `{ locale: 'simple' }`, a flag stored as `1`. A compound `Map` key may hold an integer-like
+    field only first — the driver reads keys back as plain objects.
+    Anything applied that still compares as changed is reported under `unstable` instead of being
+    rebuilt on every run.
+  - **The CLI plans first** and asks before any drop or rebuild, and before changing the validator
+    of a collection that holds data; `--json` refuses such a plan without `--yes` (and applies a
+    purely additive one); a plan with a conflict is refused without asking. `--check` exits `28` on
+    drift — a CI gate; `--ordered` refuses while a migration is pending.
+  - **Results say what changed**: every row that changes, drops or keeps something carries
+    `from` and `to` — the live and the declared index or validator, as plain JSON.
+  - **History** — every converge that changes something or fails appends an entry to
+    `_migronaut_converge` (`convergeLogCollection`, `MIGRONAUT_CONVERGE_LOG_COLLECTION`): when,
+    the trigger, the run id, who ran it and where, who asked and why, and every row it touched
+    with its `from` / `to`. Read it with `MigratorKit.convergeHistory({ limit })` or
+    `migronaut converge --history [--limit n] [--json]`. Best-effort, and a database that never
+    converges never gets the collection.
+  - **In place where the server can**: making an index unique (MongoDB 7.0+, `collMod`
+    `prepareUnique` then `unique` — duplicates leave the index as it was) and adding a TTL to a
+    single-field index (5.1+) are `modify`, not a rebuild. (6.0 accepts the unique conversion but
+    did not enforce it in our tests, so it rebuilds.)
+  - **Sharded clusters**: behind a `mongos`, prune never drops the index backing a shard key (read
+    from `config.collections`, or kept with a warning when the server refuses the drop).
+  - **Re-planned before each collection**: every collection after the first is read and planned
+    again right before its turn; one that changed meanwhile into a conflict or a new drop/rebuild
+    stops the run (`ConvergeFailedError`, `phase: 'replan'`) before it is touched.
+  - **Regular expressions** in a validator or partial filter must use flags the driver stores as
+    written (`i`, `m`; a `BSONRegExp` for server options) — `g` would become dotAll and `s`, `u`,
+    `y` vanish — and compare in their stored form. A `__proto__` key in a JSON definition stays a
+    key.
+  - **At scale**: all declared collections are read with one `listCollections` and a bounded
+    fan-out of `listIndexes`; the new indexes of a collection are built by one `createIndexes`
+    (one pass over the data); a connection that fails mid-rebuild is reported, never "repaired"
+    by restoring the old index next to a build the server may still be running.
+  - **`convergeAfterUp` config option** (`MIGRONAUT_CONVERGE_AFTER_UP`) — a bulk `up` (no file, no
+    `to`; also `runMigrations`) ends by converging under the same lock, even when nothing was
+    pending, so a failed converge is retried by the next deploy. `up(undefined, { converge })` and
+    `migronaut up --converge` / `--no-converge` decide per run. `up` still returns its migration
+    rows; `runMigrations` adds `summary.converge`.
+  - **`MigratorKit.convergesAfterUp()`** — whether a bulk `up` on the kit ends by converging.
+  - **Events**: `converge:start`, `converge:action` (per step: `'started'` before it runs — an index
+    build can take hours — then `'applied'` or `'failed'`) and `converge:end` (with the full
+    result), for real runs. An index build also logs which index it is starting on. The run itself is an ordinary `run:start`/`run:end` with
+    `command: 'converge'`, and an ordinary `migronaut.run` span.
+  - **Types**: `CollectionDefinition`, `CollectionDefinitionFile`, `IndexDefinition`,
+    `IndexKeyDirection`, `IndexCollation`, `ValidationLevel`, `ValidationAction`, `ConvergeOptions`,
+    `ConvergeResult`, `CollectionConvergeResult`, `ConvergeAction`, `ConvergeActionKind`,
+    `ConvergeActionStatus`, `ConvergeTarget`, `ConvergeUnstable`, `ConvergeTrigger` and the three
+    event payloads, exported from the package root.
+  - The definition shape, the result shape and the queue contract below are new and should be
+    treated as experimental.
+- **Converge jobs in the queue adapter** — `JOB_NAMES.CONVERGE` (`'converge'`), `enqueueConverge()`
+  and `MigrationQueue.enqueueConverge()`, and `schedule({ job: 'converge' })` with its own default
+  id, `DEFAULT_CONVERGE_SCHEDULER_ID` (`'migronaut-converge'`). With `convergeAfterUp`,
+  `enqueueUp` ends a group that reaches the newest migration with a converge job (or adds a
+  converge-only job when nothing is pending but a dry run finds drift), and an idle `sync` tick does
+  the same. A converge job refuses as `MIGRATION_BLOCKED` while a migration is pending, is keyed for
+  deduplication on the migration it follows, and carries no `prune` — what may be dropped comes
+  from the worker's own definitions. New types: `ConvergeJobData`, `ConvergeJobResult`,
+  `ConvergeJobSpec`, `ConvergeHandle`, `EnqueueConvergeOptions`.
+- **Two exit codes**: `CONVERGE_FAILED` (27, with `ConvergeFailedError` exported from the package
+  root) and the CLI-only `COLLECTIONS_DRIFT` (28, from `converge --check`).
+
+### Changed
+
+- **`MigronautErrorCode` gained four members** (above). TypeScript consumers with an exhaustive
+  `switch` over the code union need a `default` branch or the new cases.
+- **`collections`, `collectionsDir` and `convergeAfterUp` config keys are now validated**
+  (`CONFIG_INVALID`). They were previously unknown and ignored, like any stray key.
+- **The CLI arg parser makes a `--x` / `--no-x` pair tri-state**, as commander does: when a command
+  declares both, neither given leaves the option unset instead of defaulting to `true`. Only
+  `up --converge` / `--no-converge` uses this.
+- **`dryRun('up')` now applies the out-of-order policy** of the run it previews: under
+  `onOutOfOrder: 'error'` a bulk preview refuses with `MIGRATION_OUT_OF_ORDER` instead of listing
+  rows the run would reject; under `'warn'` it logs the warning. A single-file preview is exempt,
+  as the single-file run is.
+- **`connect()` is safe to call concurrently** — overlapping first calls on one `MigratorKit`
+  share a single connection instead of each opening (and all but one leaking) a client. Matters
+  for a long-lived kit serving several callers.
+- The lock-wait loop of `runMigrations` moved to `src/core/lock-wait.js`, shared with the queue
+  processor. Two changes to how it waits: **polls back off**, doubling from `lockPollIntervalMs`
+  up to 5 s (and at most a quarter of the budget), so a fleet waiting out a long deploy no longer
+  hammers the lock document; and **the default `lockWaitTimeoutMs` follows the holder's TTL** —
+  `max(90 s, 1.5 × lockTTLSeconds)` — so a holder with a long TTL, whose heartbeat moves the lock
+  only every TTL/2, is no longer mistaken for a stalled one. An explicit `lockWaitTimeoutMs` is
+  used as given (with a warning when it is shorter than the holder's heartbeat). `waitedMs` is
+  now measured by the clock from the first refusal, and a wait that times out rethrows the
+  refusal with `context.timedOut`, `attempts` and `waitedMs`.
+- **`runMigrations` validates `onLockHeld`** (`CONFIG_INVALID`): a value other than `'throw'` or
+  `'wait'` — `'Wait'`, say — used to behave as `'throw'` without a word. A poll interval above the
+  largest timer (2³¹−1 ms, which Node fires after 1 ms) is refused too.
+- **The lock document gained a `nonce` field**, minted by migronaut on every acquire and matched
+  alongside `owner` when the lock is confirmed, renewed and released. The owner token is the run
+  id, whose format `generateId` now decides; the nonce keeps mutual exclusion independent of it,
+  so a generator that repeats an id can blur correlation but never let two runs hold the lock.
+  Older releases ignore the field and can share a database with this one. The nonce is never
+  exposed; the owner — the holder's run id — now is, as `LockInfo.runId` (`lock`, `lockInfo()`,
+  `LockAlreadyHeldError` context), next to the holder's `ttlMs`, also written since this release:
+  "which run holds the lock?" is the first question about a stuck one, and the nonce, not the
+  owner, is what proves ownership.
+- **A `generateId` config key that is not a function is now rejected** (`CONFIG_INVALID`). The key
+  was previously unknown and ignored, like any stray key.
+- **A `telemetry` config key that is not usable is now rejected** (`CONFIG_INVALID`): it must be an
+  object whose `tracer` has `startActiveSpan` and whose `meter` has `createHistogram` and
+  `createCounter`. Absent, `null` and `{}` all mean "off". The key was previously unknown and
+  ignored.
+- **A scheduled `sync` job no longer joins the trace that registered its schedule.** Its template
+  now carries `telemetry: { omitContext: true }`: BullMQ builds each scheduler iteration from the
+  previous job's options, so with telemetry on, every tick would otherwise have been appended to
+  one ever-growing trace. Each tick is a trace of its own; no effect on a queue without telemetry.
+
+### Fixed
+
+- `docs/reference/cli.md` lists exit code `23` (`MIGRATION_OUT_OF_ORDER`), missing since v2.0.0.
+- The events table in `docs/guide/hooks.md` had fallen behind the payloads: it now lists
+  `migration:skipped`, the `command` / `durationMs` / count fields of `run:start` and `run:end`,
+  and `ttlMs` / `acquireMs` on `lock:acquired`.
+
+### Tooling
+
+- A unit test pins `src/utils/id.js` as the only module that mints an identifier, so no id can
+  bypass `generateId`.
+- A unit test pins that nothing under `src/` or `bin/`, and neither declaration file, imports an
+  `@opentelemetry/*` package or `bullmq-otel`; they are devDependencies only. The structural types
+  are checked part by part against the real `@opentelemetry/api`, and one integration test runs the
+  real `@opentelemetry/instrumentation-mongodb` to prove the driver's spans nest under a migration.
+- Declared collections are pinned by a table-driven planner test, a fixed-point integration matrix
+  (every kind of index converges, then plans as unchanged on a real server), and the shared queue
+  scenarios on both the fake and the real BullMQ.
+- An in-tree fake BullMQ carries the adapter's unit and integration tests; the same scenarios run
+  against the real `bullmq` package when `MIGRONAUT_TEST_REDIS_URL` is set, which CI now does
+  (a Redis service on the `test` job). `bullmq` and `ioredis` are devDependencies for that only.
+
 ## v2.0.0 — 2026-08-30
 
 A major bump for three narrow contract changes (below); everything else is

package/README.md

@@ -43,7 +43,9 @@ change before it touches your database.
 
 ## Reasons to choose it
 
-- **Zero dependencies** — no runtime dependencies at all; only the `mongodb` driver as a peer.
+- **Zero dependencies** — no runtime dependencies at all; only the `mongodb` driver as a peer
+  (Mongoose, BullMQ and OpenTelemetry are optional integrations you inject — never installed for
+  you).
   Instant installs, nothing extra in your lockfile, no supply-chain surface.
 - **Run a single migration** — `migronaut up <file>`, not just "all pending".
 - **Roll back anything** — a batch (`--batch 3`), the last N (`--steps 2`), one file, or `redo`.
@@ -57,10 +59,21 @@ change before it touches your database.
   (warn by default, `onOutOfOrder: 'error'` to refuse) instead of silently applying.
 - **Lifecycle hooks** — `beforeAll`, `afterAll`, `beforeEach`, `afterEach`, `onError`.
 - **Opt-in transactions** — wrap a migration so it fully commits or fully aborts.
+- **Indexes and validators as an end state** — declare them, and `migronaut converge` brings the
+  database to match; no migration file per index change ([details](#declared-collections)).
 - **TypeScript, ESM & CommonJS** — all run with no `ts-node` plumbing.
 - **Zero config files required** — drive everything from env vars if you prefer.
 - **Pino-friendly logging** — the `logger` option is pino-compatible; pass a pino instance directly
   and migronaut logs through it (with a `component: 'migronaut'` child binding).
+- **Your id format** — run ids and queue group ids are random UUIDs by default; pass
+  `generateId: ulid` (or cuid2, nanoid, UUIDv7 — any `() => string`) and every id migronaut mints
+  comes from your generator.
+- **OpenTelemetry (optional)** — pass a tracer and a meter from your own `@opentelemetry/api`: a
+  span per run and per migration, active while the migration runs, so an instrumented MongoDB
+  driver nests its command spans under the migration that issued them — plus duration metrics.
+- **Migrations as a queue (optional)** — `@alexify/migronaut/bullmq` runs each migration as its own
+  BullMQ job, in order, so migronaut can be a migration service: trigger it over HTTP, on a
+  schedule, or from a deploy hook that waits for the result.
 
 ### How it compares to `migrate-mongo`
 
@@ -76,6 +89,7 @@ change before it touches your database.
 | Lifecycle hooks                                 |        ❌        |          ✅          |
 | First-class TypeScript (built-in)               |        ❌        |          ✅          |
 | History preserved on rollback (never deleted)   |        ❌        |          ✅          |
+| Declared indexes & validators (`converge`)      |        ❌        |          ✅          |
 | Adopt an existing `migrate-mongo` changelog     |        —        | ✅ `migronaut import` |
 
 <sub>Reflects `migrate-mongo`'s documented CLI as of mid-2026 (v14: optional
@@ -101,6 +115,8 @@ via a `client` argument; `migronaut` exposes the same plus a declarative per-fil
 | Changelog written inside the migration's transaction |   ❌        |          ✅          |
 | Credentials masked in errors, logs and `--json` |        ❌        |          ✅          |
 | Pino-compatible logger                          |        ❌        |          ✅          |
+| BullMQ queue adapter (`/bullmq` entry point)    |        ❌        |          ✅          |
+| Declared indexes & validators (`converge`)      |        ❌        |          ✅          |
 | Node floor                                      |      ≥ 18       |      ≥ 22.18        |
 
 <sub>Compared against `mongo-migrate-kit` 1.2.2 — the version this project forked from. The Node
@@ -161,7 +177,7 @@ Full docs, guides, and the API reference live at
 - [Core Concepts](https://migronaut.vercel.app/guide/concepts) — migrations, batches, the changelog, locking
 - [Getting Started](https://migronaut.vercel.app/guide/getting-started) & [Tutorial](https://migronaut.vercel.app/guide/tutorial)
 - [Configuration](https://migronaut.vercel.app/guide/configuration) · [Writing Migrations](https://migronaut.vercel.app/guide/writing-migrations) · [Transactions](https://migronaut.vercel.app/guide/transactions) · [Hooks](https://migronaut.vercel.app/guide/hooks)
-- [Programmatic API](https://migronaut.vercel.app/guide/api) · [CI/CD](https://migronaut.vercel.app/guide/ci-cd) · [Troubleshooting](https://migronaut.vercel.app/guide/troubleshooting)
+- [Programmatic API](https://migronaut.vercel.app/guide/api) · [Migrations as a Queue (BullMQ)](https://migronaut.vercel.app/guide/bullmq) · [OpenTelemetry](https://migronaut.vercel.app/guide/opentelemetry) · [CI/CD](https://migronaut.vercel.app/guide/ci-cd) · [Troubleshooting](https://migronaut.vercel.app/guide/troubleshooting)
 - Reference: [CLI Cheatsheet](https://migronaut.vercel.app/reference/cli) · [Error Codes](https://migronaut.vercel.app/reference/error-codes)
 
 ---
@@ -180,6 +196,7 @@ Every command accepts the global flags `--uri`, `--db`, `--dir`, `--config`, `--
 | `migronaut up [file]` | Run all pending migrations, one named file, or up to `--to <file>` |
 | `migronaut down [file]` | Roll back the last batch, a chosen batch, the last N steps, one file, or to `--to <file>` |
 | `migronaut redo [file]` | Roll back then re-apply (the last migration, or one file) |
+| `migronaut converge` | Bring declared collections — indexes and validators — to their declared state |
 | `migronaut status` | Print the full migration status table (`--check` to fail CI on pending) |
 | `migronaut list` | List migrations, filtered by status |
 | `migronaut dry-run <up\|down> [file]` | Preview a run without touching the database |
@@ -187,8 +204,8 @@ Every command accepts the global flags `--uri`, `--db`, `--dir`, `--config`, `--
 | `migronaut lock` | Show who currently holds the migration lock |
 | `migronaut unlock` | Force-release a stuck lock left behind by a crashed run |
 
-Most data commands (`up`, `down`, `redo`, `status`, `list`, `dry-run`, `import`, `baseline`,
-`create`, `audit`, `lock`, `unlock`) accept **`--json`** for machine-readable output — see
+Most data commands (`up`, `down`, `redo`, `converge`, `status`, `list`, `dry-run`, `import`,
+`baseline`, `create`, `audit`, `lock`, `unlock`) accept **`--json`** for machine-readable output — see
 [CI & automation](#ci--automation).
 
 <details>
@@ -236,6 +253,8 @@ migronaut up <file> --force        # re-run an ALREADY-applied file (asks for co
 migronaut up <file> --force --yes  # confirm a re-run non-interactively (required with --json)
 migronaut up --strict              # abort on any checksum mismatch
 migronaut up --no-lock             # skip the concurrency lock (local dev only)
+migronaut up --converge            # converge the declared collections afterwards (bulk runs only)
+migronaut up --no-converge         # don't, even with convergeAfterUp on
 migronaut up --json                # machine-readable output (array of run results)
 
 # down — roll back
@@ -253,6 +272,15 @@ migronaut redo <file>              # a specific file
 migronaut redo --no-lock           # skip the lock (dev only)
 migronaut redo --json              # machine-readable output (array of run results)
 
+# converge — declared indexes and validators → the database
+migronaut converge                 # plan, ask before any drop/rebuild, then apply
+migronaut converge --dry-run       # show the plan, change nothing
+migronaut converge --check         # exit 28 if anything would change (CI gate)
+migronaut converge --prune         # also drop indexes a definition does not declare
+migronaut converge --yes           # no confirmation (required for drops/rebuilds with --json)
+migronaut converge --no-lock       # skip the concurrency lock (local dev only)
+migronaut converge --json          # machine-readable output (the converge result)
+
 # status — full status table
 migronaut status                   # the full status table
 migronaut status --check           # exit 2 if any migration is pending (CI gate)
@@ -601,6 +629,95 @@ All errors extend `MigronautError` and carry a typed `code` (`LOCK_ALREADY_HELD`
 
 </details>
 
+<details id="declared-collections">
+<summary><b>Declared collections</b> — indexes and validators as an end state, with no migration file per change</summary>
+
+<br>
+
+When what matters is *the final shape* of a collection's indexes and validator — not the history
+of how it got there — declare it and let `migronaut converge` make the difference:
+
+```js
+// migronaut.config.js
+export default {
+  uri: process.env.MIGRONAUT_URI,
+  dbName: 'my_app',
+  collections: [
+    {
+      name: 'users',
+      indexes: [
+        { key: { email: 1 }, unique: true },
+        { key: { createdAt: 1 }, expireAfterSeconds: 60 * 60 * 24 * 30 },
+      ],
+      validator: { $jsonSchema: { bsonType: 'object', required: ['email'] } },
+    },
+  ],
+  collectionsDir: './collections', // …and/or one file per collection
+};
+```
+
+```bash
+migronaut converge --dry-run   # what would change
+migronaut converge             # apply — asks before dropping or rebuilding an index
+migronaut converge --check     # exit 28 on drift: a CI gate
+```
+
+- **Stateless.** Every run reads `listIndexes` / `listCollections` and compares; nothing is
+  recorded. Edit the declaration, converge again.
+- **Safe by default.** Missing indexes are created, a TTL or `hidden` change is applied in place,
+  a changed index is rebuilt (and put back if the rebuild fails). An index you did not declare is
+  **kept and reported** — dropped only with `prune`. An identical index under another name is
+  accepted as is, never silently rebuilt.
+- **Locked like a migration**, and refused before the first write when the plan has a conflict.
+- **After every deploy** with `convergeAfterUp: true` — a bulk `up` then ends by converging,
+  under the same lock — or as a [queue job](https://migronaut.vercel.app/guide/bullmq#converge-jobs).
+
+Experimental in 2.1. → **[Declared Collections](https://migronaut.vercel.app/guide/collections)**
+
+</details>
+
+<details>
+<summary><b>Migrations as a queue</b> — a migration service on BullMQ, one migration per job</summary>
+
+<br>
+
+`@alexify/migronaut/bullmq` enqueues each pending migration as its own BullMQ job and applies them
+in order with a single-concurrency worker. BullMQ is **injected** — it is your dependency, never
+migronaut's:
+
+```js
+const { Queue, Worker, QueueEvents } = require('bullmq');
+const { createMigrationQueue } = require('@alexify/migronaut/bullmq');
+
+const mq = createMigrationQueue({
+  config: { uri: process.env.MIGRONAUT_URI, dbName: 'my_app' },
+  bullmq: { Queue, Worker, QueueEvents },
+  connection: { host: 'redis', port: 6379 },
+});
+
+await mq.startWorker();                  // the process that applies migrations
+
+const group = await mq.enqueueUp();      // one job per pending migration, one shared batch
+const { results } = await group.wait();  // optional: block until they all finished
+
+await mq.enqueueDown();                  // roll the last batch back, newest first
+await mq.schedule({ every: 300_000 });   // or keep the database migrated on a schedule
+await mq.enqueueConverge();              // declared indexes and validators, as a job
+```
+
+- **Order comes from MongoDB, not from Redis.** Every job is a normal single-file run under the
+  usual lock, and refuses while an earlier migration is still pending — so a failed migration
+  stops the line (`MIGRATION_BLOCKED`), and a CLI `migronaut up` at the same moment is safe.
+- **One attempt per job, on purpose** — a queue retry would let later migrations overtake the
+  failed one. Duplicate enqueues are deduplicated; an already-applied migration completes as
+  `skipped`.
+- **Bring your own Worker** with `createMigrationProcessor()` (NestJS, BullMQ Pro).
+
+→ **[Migrations as a Queue](https://migronaut.vercel.app/guide/bullmq)** ·
+[runnable example service](examples/migration-service)
+
+</details>
+
 ---
 
 ## Configuration
@@ -633,19 +750,100 @@ export default {
   // ── Bookkeeping collections ─────────────────────────────────────────────
   migrationsCollection: '_migronaut_migrations', // the append-only audit trail
   lockCollection: '_migronaut_locks',            // the concurrency lock
+  convergeLogCollection: '_migronaut_converge',  // what each converge changed
   lockTTLSeconds: 60,                       // a lock older than this is reclaimable
 
   // ── Safety ──────────────────────────────────────────────────────────────
   strict: false,        // true → abort on a checksum mismatch (instead of warn + skip)
   useTransaction: false, // true → wrap every migration in a transaction (override per file)
 
+  // ── Declared collections (experimental) — see `migronaut converge` ──────
+  // collections: [{ name: 'users', indexes: [{ key: { email: 1 }, unique: true }] }],
+  // collectionsDir: './collections', // one definition file per collection
+  // convergeAfterUp: false,          // true → every bulk `up` ends by converging
+
   // ── Code-only options (omit in migronaut.config.json) ─────────────────────────
   // hooks: { beforeAll, afterAll, beforeEach, afterEach, onError },
   // mongoose: myMongooseInstance, // pass if your migrations use Mongoose models
   // logger: null,                 // null silences all output; a pino instance works directly
+  // generateId: ulid,             // your id format for run ids — any sync `() => string`
+  // telemetry: { tracer, meter }, // OpenTelemetry, from your own @opentelemetry/api
 };
 ```
 
+<details>
+<summary><b>Custom id format</b> — ULID, CUID, UUIDv7 or anything else instead of UUIDs</summary>
+
+<br>
+
+Every run gets a `runId` — stamped on its changelog records, events and log lines, and stored as
+the lock's owner token — and every queue enqueue gets a `groupId`. Both are `crypto.randomUUID()`
+by default. Pass `generateId` to mint them with the generator the rest of your system uses:
+
+```js
+const { ulid } = require('ulid');
+const { runMigrations } = require('@alexify/migronaut');
+
+await runMigrations({
+  uri: process.env.MIGRONAUT_URI,
+  dbName: 'my_app',
+  generateId: ulid, // createId (cuid2), nanoid, uuidv7 … pass straight through
+});
+```
+
+It is called with no arguments and must synchronously return a non-empty string of at most 128
+characters; anything else fails the run with `CONFIG_INVALID` before a migration starts. Ids are
+for correlation only — the lock carries a token of its own, so a generator that repeats a value
+can never let two runs hold the lock at once.
+
+</details>
+
+<details>
+<summary><b>OpenTelemetry</b> — a span per run and per migration, and their durations as metrics</summary>
+
+<br>
+
+Pass a tracer and/or a meter from your own `@opentelemetry/api` — migronaut never imports it:
+
+```js
+const { metrics, trace } = require('@opentelemetry/api');
+const { runMigrations } = require('@alexify/migronaut');
+
+await runMigrations({
+  uri: process.env.MIGRONAUT_URI,
+  dbName: 'my_app',
+  telemetry: {
+    tracer: trace.getTracer('@alexify/migronaut'),
+    meter: metrics.getMeter('@alexify/migronaut'),
+  },
+});
+```
+
+```
+migronaut.run                          one per run, once it holds the lock
+└─ migronaut.migration                 one per migration — the ACTIVE span while it runs
+   ├─ insert users                     ← your instrumented MongoDB driver nests here
+   └─ update _migronaut_migrations
+```
+
+That nesting is the point. An instrumented driver only records a command that has a parent span,
+and a migration run at application startup has none — so without this, the driver's spans for your
+migrations are simply missing. A [lifecycle event](#advanced-features) can tell you a migration
+started; only a span opened inside the kit can be the parent of what it does next.
+
+The meter gets `migronaut.run.duration`, `migronaut.migration.duration` and
+`migronaut.lock.acquire.duration` (histograms, seconds), plus the counters `migronaut.lock.refused`
+and `migronaut.lock.lost`. A failure sets the span's status to `ERROR` — with the message redacted
+like every log line — and `error.type` to the typed error code. A tracer or meter that throws never
+fails a run.
+
+Through the [BullMQ adapter](https://migronaut.vercel.app/guide/bullmq), add
+`bullmq: { Queue, Worker, telemetry: new BullMQOtel({ tracerName }) }` and one trace runs from the
+request that enqueued to the MongoDB commands in the worker. Full guide:
+[OpenTelemetry](https://migronaut.vercel.app/guide/opentelemetry).
+
+</details>
+
 <details>
 <summary><b>Structured logging with pino</b> — the logger option is pino-compatible</summary>
 
@@ -683,6 +881,7 @@ optional rather than merely discouraged:
 | `MIGRONAUT_MIGRATIONS_DIR` | `migrationsDir` | `./migrations` |
 | `MIGRONAUT_COLLECTION` | `migrationsCollection` | `_migronaut_migrations` |
 | `MIGRONAUT_LOCK_COLLECTION` | `lockCollection` | `_migronaut_locks` |
+| `MIGRONAUT_CONVERGE_LOG_COLLECTION` | `convergeLogCollection` | `_migronaut_converge` |
 | `MIGRONAUT_LOCK_TTL` | `lockTTLSeconds` | `60` |
 | `MIGRONAUT_STRICT` | `strict` | `false` |
 | `MIGRONAUT_USE_TRANSACTION` | `useTransaction` | `false` |
@@ -695,10 +894,13 @@ optional rather than merely discouraged:
 | `MIGRONAUT_ON_OUT_OF_ORDER` | `onOutOfOrder` | `warn` |
 | `MIGRONAUT_ENSURE_INDEXES` | `ensureIndexes` | `true` |
 | `MIGRONAUT_RELOAD_MIGRATIONS` | `reloadMigrations` | `false` |
+| `MIGRONAUT_COLLECTIONS_DIR` | `collectionsDir` | — *(none read)* |
+| `MIGRONAUT_CONVERGE_AFTER_UP` | `convergeAfterUp` | `false` |
 | `MIGRONAUT_ENV_FILE` | `envFile` | `.env` |
 
-`fileExtensions`, `clientOptions`, `client`, `mongoose`, `hooks` and `logger` are config-file/API
-only — they aren't scalars, so no environment variable can express them.
+`fileExtensions`, `clientOptions`, `collections`, `client`, `mongoose`, `hooks`, `logger`,
+`generateId` and `telemetry` are config-file/API only — they aren't scalars, so no environment
+variable can express them.
 
 A value that doesn't parse is **rejected, never coerced**: `MIGRONAUT_STRICT=on` or
 `MIGRONAUT_LOCK_TTL=abc` fails with an error naming the variable, rather than quietly turning a