@alexify/migronaut 2.1.0 → 2.2.0

package/CHANGELOG.md

@@ -3,6 +3,122 @@
 All notable changes to this project will be documented in this file.
 Release headings carry the publish date (`## vX.Y.Z — YYYY-MM-DD`).
 
+## v2.2.0 — 2026-10-05
+
+Atlas Search and Vector Search indexes in declared collections. Additive: a definition without
+`searchIndexes` behaves exactly as in 2.1 — converge never even asks the server about Search for
+it.
+
+### Added
+
+- **Declared search indexes** — a collection definition takes `searchIndexes: [{ name?, type?,
+  definition }]`: Atlas Search (`type: 'search'`, the default) and Atlas Vector Search
+  (`'vectorSearch'`) indexes, with the definition exactly as Atlas documents it — automated
+  embedding (`autoEmbed`) fields included. `converge` keeps them in step with the same lock,
+  history, events, re-plan and fixed-point check as regular indexes. Experimental.
+  - **Rows**: a new target, `searchIndex`. A missing index is created (one `createSearchIndexes`
+    per collection), a changed definition is updated **in place** (`modify` — the old definition
+    serves until the new one is built), an undeclared one is kept, or dropped last under
+    `prune`. A definition that declares only `searchIndexes` is valid, and its collection is
+    created when missing.
+  - **Never a rebuild**: a `$search` against a missing index returns nothing rather than fail, so
+    converge never drops a search index to build it again. A change no update can make — the
+    type, or an `autoEmbed` field's path, model, `numDimensions`, quantization or modality — is a
+    `conflict` naming the new-name recipe, and so is a declared index the server is still
+    deleting.
+  - **Comparison**: definitions compare whole, key order ignored, with the defaults the server
+    writes into what it reports filled in on both sides — top-level (`analyzer`,
+    `searchAnalyzer`, `dynamic`, `storedSource`, `numPartitions`), per field mapping (`string`,
+    `number`, `autocomplete`, `token`, `geo`, `document`, nested fields and `multi` included) and
+    per vector field (`quantization`, `indexingMethod`, `hnswOptions`; `autoEmbed`
+    `numDimensions` and `quantization`). Vector `fields`, and a field indexed as several types,
+    compare as sets. A server that reports no `type` (a self-managed `mongot`) has it inferred
+    from the definition, and its `latestVersion` is read as the definition version.
+  - **Server-only options**: an option the server reports that the declaration does not set,
+    and whose default migronaut does not know (a newer `mongot`'s), is left out of the
+    comparison — named on the row (`ignored`) and in one warning — instead of making every
+    converge update, and the server rebuild, the index. Only option objects are trimmed: a
+    field, a mapping type or a vector field only the server has is still a difference. The
+    cost: removing such an option from a declaration goes unnoticed; declare the value wanted.
+  - **What differs**: a `modify` row names it down to the option
+    (`mappings.fields.title.norms`) — the first five paths and how many more. A search index
+    that still differs after an update is `unstable`, with a warning that every update builds it
+    again.
+  - **Raw commands** (`createSearchIndexes`, `updateSearchIndex`, `dropSearchIndex`,
+    `$listSearchIndexes`), so every driver in the peer range works — the driver's helpers start
+    at 5.6. A vector index update is retried once with its type when a self-managed `mongot`
+    asks for it; where the server refuses that too (an Atlas CLI local deployment on 8.0 and
+    8.3), the step fails with the new-name recipe as its hint.
+  - **Build state**: rows carry `build` (`{ status, queryable, message?, updating? }`), and the
+    result `search` (`{ available, evidence, notReady, wait? }`) — how Search availability was
+    told, the declared indexes still building, updating, stale or failed, and how a wait ended.
+    A build under way does not count against `inSync`; a FAILED one fails `converge --check`
+    (exit 28). A STALE index (queryable, no longer replicating) is reported as such — table,
+    closing warning, `audit` — not as still building. A build message is kept to 500 characters.
+- **`waitForSearchIndexes`** (config, `MIGRONAUT_WAIT_FOR_SEARCH_INDEXES`, `converge({
+  waitForSearchIndexes })`, `converge --wait-search` / `--no-wait-search`) — hold the converge
+  until every declared search index serves its declaration; a FAILED build of an index the run
+  created or changed, or `searchIndexWaitTimeoutMs` (`MIGRONAUT_SEARCH_INDEX_WAIT_TIMEOUT_MS`,
+  default 10 minutes), fails it with `ConvergeFailedError` `phase: 'wait'`. Off by default.
+  - **Without the lock**: the wait only reads, so the migration lock is released when it starts
+    (`lock:released` with `early: true`) — the next deploy or a queue's next job need not wait
+    out a build. The run itself (its id, span, history entry, `converge:end`) ends with the
+    wait.
+  - **Only what the run started holds it**: an index that failed or went STALE before the run,
+    its definition unchanged, is named and warned about instead — converge cannot fix it, and a
+    deploy is not held up by it. `--check` still fails on a FAILED build.
+  - **Resilient**: a list that fails with a network or failover blip is read again at the next
+    poll, up to three in a row (then `reason: 'unreadable'`); a stop cuts the pause between polls
+    short.
+  - **Observable**: `converge:wait` events (`started`, `progress` every 30 s, then `ready`,
+    `failed`, `timeout`, `unreadable` or `aborted`), `result.search.wait`, the history entry, a
+    queue job's log lines and `search-wait` progress phase, and — with `telemetry` — the
+    histogram `migronaut.converge.search.wait.duration` by `migronaut.converge.search.wait.outcome`.
+- **`onSearchUnavailable`** (`MIGRONAUT_ON_SEARCH_UNAVAILABLE`) — declared search indexes on a
+  server without Atlas Search: `'fail'` (default) refuses the run before any write, with a hint;
+  `'skip'` converges everything else and reports `skip` rows. Detected once per run from the
+  server's own answer (checked against MongoDB 5.0, 6.0, 7.0, 8.0 and 8.2).
+- **`migronaut audit`** — a `search` check, when the definitions declare search indexes: whether
+  the server has Atlas Search, and whether a declared index failed to build.
+- **Types** — `SearchIndexDefinition`, `SearchDefinition`, `VectorSearchDefinition`,
+  `VectorSearchField`, `SearchIndexType`, `SearchIndexStatus`, `SearchIndexBuild`,
+  `SearchIndexNotReady`, `ConvergeSearchSummary`, `ConvergeWaitOutcome`, `ConvergeWaitEvent`;
+  `ConvergeTarget` gains `'searchIndex'`, `ConvergeActionKind` `'skip'`, `ConvergeAction`
+  `ignored`, `ConvergeHistoryEntry` `search`, `LockEvent` `early`, the event map
+  `'converge:wait'`; in `bullmq.d.ts` `ConvergeJobResult.search`, and `MigrationJobProgress`
+  gains the `'search-wait'` phase and `searchIndexes`. An exhaustive `switch` over one of these
+  unions needs a branch for the new member.
+- **Queue** — a converge job's result carries `search`, and its log lines name search indexes.
+  Whether a job waits for builds is the worker kit's `waitForSearchIndexes`; the job payload is
+  unchanged.
+
+### Changed
+
+- A definition with none of `indexes`, `searchIndexes` and `validator` is refused with "declares
+  no indexes, searchIndexes or validator — nothing to manage" (was "declares neither indexes nor
+  a validator").
+- The converge table's drop/rebuild count includes dropped search indexes, and `converge --yes`
+  is required for them in `--json` mode, like an index drop.
+- `ConvergeFailedError`'s documentation names every phase: `plan`, `replan` (already thrown by
+  2.1, undocumented), `apply` and the new `wait`. A search index list that cannot be read is
+  reported in the phase that read it — `plan` only before the first write — and a run that stops
+  for any reason settles every row and carries the result so far in `context.converge`.
+- `runWithLock` (internal) hands the work a `control` whose `release()` gives the lock up early;
+  `lock:released` may now come before `run:end`, with `early: true`.
+
+### Tooling
+
+- `tests/integration/search-atlas.test.js` — an opt-in, manual suite against
+  `mongodb/mongodb-atlas-local` (`MIGRONAUT_TEST_ATLAS_URI`): every scenario ends at a fixed
+  point. Passes against `mongodb/mongodb-atlas-local` 8.0 (8.0.32) and `latest` (8.3.11). CI does
+  not run it; the coverage gate comes from the unit tier's fake, which answers the search
+  commands with lag, normalization and build progress on demand. The suite stays strict about
+  server-only options (production tolerates them): one there is a default the tables in
+  `search-index-spec.js` lack. 9/9 on 8.0.32 and 8.3.11, the lock released during the wait
+  included.
+- `src/core/converge.js` is split: `converge-search-run.js` (the search half of a run) and
+  `server-info.js` (read options, read pace, server version, not-found codes).
+
 ## v2.1.0 — 2026-10-04
 
 Migrations as a queue, ids in your own format, OpenTelemetry, and declared collections. Additive:

package/README.md

@@ -59,8 +59,9 @@ change before it touches your database.
   (warn by default, `onOutOfOrder: 'error'` to refuse) instead of silently applying.
 - **Lifecycle hooks** — `beforeAll`, `afterAll`, `beforeEach`, `afterEach`, `onError`.
 - **Opt-in transactions** — wrap a migration so it fully commits or fully aborts.
-- **Indexes and validators as an end state** — declare them, and `migronaut converge` brings the
-  database to match; no migration file per index change ([details](#declared-collections)).
+- **Indexes, search indexes and validators as an end state** — declare them (Atlas Search and
+  Vector Search indexes included), and `migronaut converge` brings the database to match; no
+  migration file per index change ([details](#declared-collections)).
 - **TypeScript, ESM & CommonJS** — all run with no `ts-node` plumbing.
 - **Zero config files required** — drive everything from env vars if you prefer.
 - **Pino-friendly logging** — the `logger` option is pino-compatible; pass a pino instance directly
@@ -90,6 +91,7 @@ change before it touches your database.
 | First-class TypeScript (built-in)               |        ❌        |          ✅          |
 | History preserved on rollback (never deleted)   |        ❌        |          ✅          |
 | Declared indexes & validators (`converge`)      |        ❌        |          ✅          |
+| Declared Atlas Search / Vector Search indexes   |        ❌        |          ✅          |
 | Adopt an existing `migrate-mongo` changelog     |        —        | ✅ `migronaut import` |
 
 <sub>Reflects `migrate-mongo`'s documented CLI as of mid-2026 (v14: optional
@@ -117,6 +119,7 @@ via a `client` argument; `migronaut` exposes the same plus a declarative per-fil
 | Pino-compatible logger                          |        ❌        |          ✅          |
 | BullMQ queue adapter (`/bullmq` entry point)    |        ❌        |          ✅          |
 | Declared indexes & validators (`converge`)      |        ❌        |          ✅          |
+| Declared Atlas Search / Vector Search indexes   |        ❌        |          ✅          |
 | Node floor                                      |      ≥ 18       |      ≥ 22.18        |
 
 <sub>Compared against `mongo-migrate-kit` 1.2.2 — the version this project forked from. The Node
@@ -196,7 +199,7 @@ Every command accepts the global flags `--uri`, `--db`, `--dir`, `--config`, `--
 | `migronaut up [file]` | Run all pending migrations, one named file, or up to `--to <file>` |
 | `migronaut down [file]` | Roll back the last batch, a chosen batch, the last N steps, one file, or to `--to <file>` |
 | `migronaut redo [file]` | Roll back then re-apply (the last migration, or one file) |
-| `migronaut converge` | Bring declared collections — indexes and validators — to their declared state |
+| `migronaut converge` | Bring declared collections — indexes, search indexes and validators — to their declared state |
 | `migronaut status` | Print the full migration status table (`--check` to fail CI on pending) |
 | `migronaut list` | List migrations, filtered by status |
 | `migronaut dry-run <up\|down> [file]` | Preview a run without touching the database |
@@ -272,11 +275,12 @@ migronaut redo <file>              # a specific file
 migronaut redo --no-lock           # skip the lock (dev only)
 migronaut redo --json              # machine-readable output (array of run results)
 
-# converge — declared indexes and validators → the database
+# converge — declared indexes, search indexes and validators → the database
 migronaut converge                 # plan, ask before any drop/rebuild, then apply
 migronaut converge --dry-run       # show the plan, change nothing
 migronaut converge --check         # exit 28 if anything would change (CI gate)
 migronaut converge --prune         # also drop indexes a definition does not declare
+migronaut converge --wait-search   # wait until every declared search index is queryable
 migronaut converge --yes           # no confirmation (required for drops/rebuilds with --json)
 migronaut converge --no-lock       # skip the concurrency lock (local dev only)
 migronaut converge --json          # machine-readable output (the converge result)
@@ -630,12 +634,13 @@ All errors extend `MigronautError` and carry a typed `code` (`LOCK_ALREADY_HELD`
 </details>
 
 <details id="declared-collections">
-<summary><b>Declared collections</b> — indexes and validators as an end state, with no migration file per change</summary>
+<summary><b>Declared collections</b> — indexes, search indexes and validators as an end state, with no migration file per change</summary>
 
 <br>
 
-When what matters is *the final shape* of a collection's indexes and validator — not the history
-of how it got there — declare it and let `migronaut converge` make the difference:
+When what matters is *the final shape* of a collection's indexes, Atlas Search indexes and
+validator — not the history of how it got there — declare it and let `migronaut converge` make the
+difference:
 
 ```js
 // migronaut.config.js
@@ -651,6 +656,19 @@ export default {
       ],
       validator: { $jsonSchema: { bsonType: 'object', required: ['email'] } },
     },
+    {
+      name: 'movies',
+      searchIndexes: [
+        { definition: { mappings: { dynamic: true } } }, // Atlas Search, named "default"
+        {
+          name: 'plot_vectors',
+          type: 'vectorSearch',
+          definition: {
+            fields: [{ type: 'vector', path: 'embedding', numDimensions: 1536, similarity: 'cosine' }],
+          },
+        },
+      ],
+    },
   ],
   collectionsDir: './collections', // …and/or one file per collection
 };
@@ -668,11 +686,15 @@ migronaut converge --check     # exit 28 on drift: a CI gate
   a changed index is rebuilt (and put back if the rebuild fails). An index you did not declare is
   **kept and reported** — dropped only with `prune`. An identical index under another name is
   accepted as is, never silently rebuilt.
+- **Search indexes, too** — on Atlas, an Atlas CLI local deployment or MongoDB 8.3+ with `mongot`.
+  A changed definition is updated in place (on Atlas the old one serves until the new one is
+  built), never dropped and rebuilt; `--wait-search` waits until the builds are queryable. On a server without
+  Search, converge refuses — or skips them with `onSearchUnavailable: 'skip'`.
 - **Locked like a migration**, and refused before the first write when the plan has a conflict.
 - **After every deploy** with `convergeAfterUp: true` — a bulk `up` then ends by converging,
   under the same lock — or as a [queue job](https://migronaut.vercel.app/guide/bullmq#converge-jobs).
 
-Experimental in 2.1. → **[Declared Collections](https://migronaut.vercel.app/guide/collections)**
+Experimental since 2.1 (search indexes: 2.2). → **[Declared Collections](https://migronaut.vercel.app/guide/collections)**
 
 </details>
 
@@ -761,6 +783,8 @@ export default {
   // collections: [{ name: 'users', indexes: [{ key: { email: 1 }, unique: true }] }],
   // collectionsDir: './collections', // one definition file per collection
   // convergeAfterUp: false,          // true → every bulk `up` ends by converging
+  // onSearchUnavailable: 'fail',     // 'skip' → converge without search indexes where Search is absent
+  // waitForSearchIndexes: false,     // true → converge waits until search indexes are queryable
 
   // ── Code-only options (omit in migronaut.config.json) ─────────────────────────
   // hooks: { beforeAll, afterAll, beforeEach, afterEach, onError },
@@ -896,6 +920,9 @@ optional rather than merely discouraged:
 | `MIGRONAUT_RELOAD_MIGRATIONS` | `reloadMigrations` | `false` |
 | `MIGRONAUT_COLLECTIONS_DIR` | `collectionsDir` | — *(none read)* |
 | `MIGRONAUT_CONVERGE_AFTER_UP` | `convergeAfterUp` | `false` |
+| `MIGRONAUT_ON_SEARCH_UNAVAILABLE` | `onSearchUnavailable` | `fail` |
+| `MIGRONAUT_WAIT_FOR_SEARCH_INDEXES` | `waitForSearchIndexes` | `false` |
+| `MIGRONAUT_SEARCH_INDEX_WAIT_TIMEOUT_MS` | `searchIndexWaitTimeoutMs` | `600000` |
 | `MIGRONAUT_ENV_FILE` | `envFile` | `.env` |
 
 `fileExtensions`, `clientOptions`, `collections`, `client`, `mongoose`, `hooks`, `logger`,

package/bullmq.d.ts

@@ -1,6 +1,7 @@
 import type {
   AuditReport,
   CollectionConvergeResult,
+  ConvergeSearchSummary,
   ConvergeUnstable,
   LockInfo,
   MigratorKit,
@@ -237,6 +238,13 @@ export interface ConvergeJobResult {
   inSync: boolean;
   collections: CollectionConvergeResult[];
   unstable?: ConvergeUnstable[];
+  /**
+   * Atlas Search availability and the declared search indexes still building
+   * — when the worker's definitions declare search indexes. Whether the job
+   * waits for them is the worker kit's `waitForSearchIndexes`.
+   * @experimental New in 2.2
+   */
+  search?: ConvergeSearchSummary;
   runId?: string;
   /** Time (ms) spent waiting for the MongoDB migration lock */
   lockWaitMs: number;
@@ -247,7 +255,8 @@ export interface ConvergeJobResult {
  * @experimental New in 2.1 — the shape may still change in a minor release (named in the CHANGELOG).
  */
 export interface MigrationJobProgress {
-  phase: 'lock-wait' | 'running' | 'completed' | 'failed';
+  /** `search-wait` (New in 2.2): a converge job waiting for search index builds */
+  phase: 'lock-wait' | 'running' | 'search-wait' | 'completed' | 'failed';
   migration?: string;
   direction?: 'up' | 'down';
   groupId?: string;
@@ -256,7 +265,13 @@ export interface MigrationJobProgress {
   kind?: 'sync' | 'converge';
   /** `lock-wait` only */
   attempts?: number;
+  /** `lock-wait` and `search-wait` */
   waitedMs?: number;
+  /**
+   * `search-wait` only — how many search indexes the wait is for
+   * @experimental New in 2.2
+   */
+  searchIndexes?: number;
   /**
    * `failed` only — the typed error code, so nobody has to parse
    * `failedReason`; `'UNKNOWN'` for an error that is not migronaut's.