@nxgt/mongo-meilisearch 0.4.4 → 0.5.0

package/docs/guide/search-syncs/lifecycle.md

@@ -0,0 +1,301 @@
+# The search syncs' lifecycle
+
+One `syncIndexes`, one `reindexAll`, one `start`, one `close` for every sync the
+[config](wiring.md) named — and one promise to watch while they run.
+
+```ts
+import { bindIndex } from '@nxgt/meilisearch';
+import { openMongo, defineMongo } from '@nxgt/mongo';
+import { createSearchSyncs } from '@nxgt/mongo-meilisearch';
+import * as collections from './models';                 // the defineCollections
+import { meili } from './meili';                         // a Meilisearch client
+import { articleIndex, authorIndex } from './search';    // the defineIndexes
+
+const mongo = await openMongo(
+	defineMongo({ uri: process.env.MONGO_URI!, collections }),
+);
+
+const search = createSearchSyncs(mongo, {
+	articles: {
+		index: bindIndex(meili, articleIndex),
+		transform: (article) => ({ id: String(article._id), title: article.title }),
+	},
+	authors: {
+		index: bindIndex(meili, authorIndex),
+		transform: (author) => ({ id: String(author._id), name: author.name }),
+	},
+});
+
+await search.syncIndexes();             // create every index, apply its settings
+await search.reindexAll();              // fill every index
+const running = await search.start();   // follow every collection
+void running.failed.catch(() => process.exit(1)); // await it or catch it
+
+// …
+
+await running.close();                  // flush and stop every sync
+await mongo.close();                      // the Mongo is still yours
+```
+
+## `syncs`
+
+Each sync as `@nxgt/mongo-meilisearch` built it, under the key the config
+used — so anything `createSearchSyncs` does not wrap is still reachable:
+
+```ts
+search.syncs.articles.name;             // 'articles:articles'
+await search.syncs.articles.reindex();  // just this one
+const one = await search.syncs.authors.start();
+```
+
+## `state()`
+
+Where each sync stands, under its key; `undefined` for one that never
+reindexed:
+
+```ts
+await search.state();
+// { articles: undefined, authors: undefined }
+
+await search.reindexAll();
+(await search.state()).articles?.reindexedAt; // a Date
+```
+
+## `syncIndexes(options?)`
+
+Every index the config names brought in line with its definition — created
+with its primary key when it is missing, and only the settings that differ
+updated — one after another, each report under its key. It is
+`@nxgt/meilisearch`'s `syncIndex` per entry, so the options and the report
+are that package's:
+
+```ts
+const reports = await search.syncIndexes();
+// { articles: { uid: 'articles', created: true, changed: [ 'searchableAttributes' ], … },
+//   authors:  { uid: 'authors', created: false, changed: [], … } }
+
+await search.syncIndexes({ dryRun: true });        // what it would send, sends nothing
+await search.syncIndexes({ wait: { timeout: 120_000 } }); // the SDK's WaitOptions, per task
+```
+
+Run it twice and the second run sends nothing. The first index that throws
+stops the rest, and the indexes after it are not looked at: a
+`SearchIndexError` from `@nxgt/meilisearch` — `PRIMARY_KEY_MISMATCH` or
+`TASK_FAILED` — or the SDK's own error for a Meilisearch that refused or is
+not there. `dryRun` shows every **settings** difference at once, but not past
+a primary-key mismatch: that one throws in a dry run too, since no setting
+could make the index right.
+
+It is a **deployment step**, like the Mongo's `sync()`, and it comes
+first: `reindexAll` and `start` write documents, and an index a document write
+creates gets the primary key but none of the definition's settings until a
+sync runs — its filters and sorts are refused until then.
+
+## `reindexAll()`
+
+Every collection, one after another, each report under its key:
+
+```ts
+const reports = await search.reindexAll();
+// { articles: { indexed: 1, skipped: 1, removed: 0 }, authors: { … } }
+```
+
+**The first that throws stops the rest, and the reports already collected go
+with the rejection.** A reindex removes what a collection no longer gives, so
+a half-finished run is not a state to keep going from: read the error, fix
+it, run it again — a reindex that succeeded is idempotent.
+
+It cannot run beside a follower: `@nxgt/mongo-meilisearch` throws
+`SearchSyncError` with the code `RUNNING` for a sync of the same object that
+is already following, or whose name another process holds the lease on, and
+`createSearchSyncs` passes it through.
+
+Each reindex holds its sync's lease while it runs. One that finds the lease
+taken over — the process stalled past `leaseMs`, or the lease was removed —
+rejects with `LEASE_LOST`, having recorded nothing, and removed nothing unless
+the lease went while the removal itself ran; the pages it already sent stay in
+the index. `reindexAll()` passes that through, and stops there like any other
+failure.
+
+## `start()`
+
+Starts every sync and resolves once they are **all** hearing changes:
+
+```ts
+const running = await search.start();
+running.running.articles.ready;   // each RunningSearchSync is reachable
+```
+
+A sync that will not start closes the ones already started before the error
+comes back, so a caller that catches it owns nothing:
+
+```ts
+try {
+	await search.start();
+} catch (error) {
+	// Nothing is running: no sync to close, no leak.
+}
+```
+
+A sync with nothing recorded reindexes as it starts, which on a real
+collection is minutes — the syncs started before it are following changes
+throughout, which is why a failure closes them rather than leaving them
+behind.
+
+Each sync takes the lease on its own name as it starts, so a second process
+starting the same syncs is refused at the first name the other holds — a
+`SearchSyncError` with the code `RUNNING`, naming the holder and when its
+lease ends — and lets go of the names it had already taken:
+
+```ts
+import { SearchSyncError } from '@nxgt/mongo-meilisearch';
+
+try {
+	await search.start();
+} catch (error) {
+	const held =
+		error instanceof SearchSyncError &&
+		(error.code === 'RUNNING' || error.code === 'LEASE_LOST');
+	if (held) {
+		// Another process follows these collections, or took one over while
+		// this start reindexed: wait, and try again. On RUNNING,
+		// `error.expiresAt` says until when that name's lease runs.
+	} else throw error;
+}
+```
+
+A `RUNNING` the lease refused carries its `holder` and its `expiresAt`, a
+`Date` on MongoDB's clock, so a standby can wait exactly until that lease
+lapses; the loop is in the [troubleshooting entry](../../troubleshooting.md#search-sync-articlesarticles-is-held-by--until--wait-for-it-to-close-or-for-its-lease-to-lapse-before-you-start-it).
+Both are `undefined` on a `LEASE_LOST`, and on the refusal of syncs that are
+already running in this process.
+
+A lease lasts each entry's `leaseMs` (30 s) unrenewed, so a process that
+died holds its names that long at most. `start()` itself rejects with
+`LEASE_LOST` when a sync's first reindex finds its lease taken over, and
+closes the syncs already started, as for any failure. A running sync whose
+lease another process took over stops, and `failed` rejects with
+`LEASE_LOST`. The lease
+itself is the single sync's: see [Following changes](../following-changes.md).
+
+## `failed`
+
+| Member | Type | Effect |
+| --- | --- | --- |
+| `running` | `{ [K in keyof I]: RunningSearchSync }` | Each running sync, with its own `ready`, `closed` and `flush` |
+| `failed` | `Promise<never>` | Rejects with the **first** sync that stops on an error. Never resolves |
+| `flush()` | `Promise<void>` | Sends what every sync holds and records where each one is. Stops at the first that fails |
+| `close()` | `Promise<void>` | Flushes, then stops every sync. Idempotent |
+
+```ts
+running.failed.catch((error: SearchSyncError) => {
+	console.error({ sync: error.sync, code: error.code, cause: error.cause });
+	process.exit(1); // let the supervisor restart the process
+});
+```
+
+- **It must be taken.** A rejection nobody handles ends the process. `createSearchSyncs`
+  already takes each sync's own `closed`, so `failed` is the only one left to
+  you — handle it.
+- **It never resolves.** A clean stop is not an event to wait for, so
+  `await running.failed` after `close()` waits forever. It is for `catch`, or
+  for racing against your own shutdown.
+- **A dropped collection leaves it silent.** `@nxgt/mongo-meilisearch` treats
+  an invalidated stream as a clean stop — that sync's `closed` *resolves*
+  with `'invalidated'` — and `createSearchSyncs` forwards failures only. Watch
+  `running.running.articles.closed` when a drop has to be noticed.
+
+## `flush()` and `close()`
+
+```ts
+await running.flush(); // every sync sends what it holds, and records it
+await running.close(); // flushes, then stops; or `await using running = …`
+```
+
+`flush()` stops at the first sync that fails, like `reindexAll()`: the syncs
+after it in the config are neither sent nor recorded, and re-read those
+changes on the next start — safe, but not free.
+
+`close()` does not share that: it closes each sync in turn and keeps going
+past one that fails. It swallows the failure `failed` already carried — a
+caller should not have to wrap `close` to hear the same thing twice — and
+throws anything else, including a **second** sync that fell over after
+`failed` had settled, and anything that goes wrong while closing. If more
+than one throws, it reports the first.
+
+`close()` is idempotent, and `RunningSearchSyncs` is `AsyncDisposable`.
+
+## A worker process
+
+The Mongo and the search syncs, started in order and closed in reverse:
+
+```ts
+import { openMongo } from '@nxgt/mongo';
+import { createSearchSyncs } from '@nxgt/mongo-meilisearch';
+import type { SearchSyncError } from '@nxgt/mongo-meilisearch';
+import { config } from './db';
+import { searchConfig } from './search';
+
+const mongo = await openMongo(config);
+const search = createSearchSyncs(mongo, searchConfig(meili));
+
+const running = await search.start();
+
+const stop = async () => {
+	await running.close(); // flushes and records where each sync is
+	await mongo.close();     // the search syncs never close it
+	process.exit(0);
+};
+process.on('SIGTERM', stop);
+process.on('SIGINT', stop);
+
+// The only promise left to handle: the first sync that falls over.
+await running.failed.catch(async (error: SearchSyncError) => {
+	console.error({ sync: error.sync, code: error.code, cause: error.cause });
+	await running.close().catch(() => undefined);
+	await mongo.close();
+	process.exit(1);
+});
+```
+
+A deployment step is the same two objects, without `start`:
+
+```ts
+await using mongo = await openMongo(config);
+await mongo.sync();                                   // the collections
+const search = createSearchSyncs(mongo, searchConfig(meili));
+await search.syncIndexes();                         // the indexes
+const reports = await search.reindexAll();
+for (const [key, report] of Object.entries(reports)) {
+	console.log(`${key}: ${report.indexed} indexed, ${report.removed} removed`);
+}
+```
+
+## Signatures
+
+```ts
+interface SearchSyncs<S> {
+	readonly syncs: ByKey<S, SearchSync>;
+	state(): Promise<ByKey<S, SearchSyncState | undefined>>;
+	syncIndexes(options?: SyncOptions): Promise<ByKey<S, SyncReport>>; // @nxgt/meilisearch's
+	reindexAll(): Promise<ByKey<S, ReindexReport>>;
+	start(): Promise<RunningSearchSyncs<S>>;
+}
+
+interface RunningSearchSyncs<S> extends AsyncDisposable {
+	readonly running: ByKey<S, RunningSearchSync>;
+	readonly failed: Promise<never>;
+	flush(): Promise<void>;
+	close(): Promise<void>;
+}
+
+type ByKey<S, T> = { readonly [K in keyof S]: T };
+```
+
+`SearchSync`, `RunningSearchSync`, `SearchSyncState`, `ReindexReport` and
+`SearchSyncError` are `@nxgt/mongo-meilisearch`'s, unchanged.
+
+## Next
+
+- [Wiring it over a Mongo](wiring.md) — the config behind all of this.
+- [Troubleshooting](../../troubleshooting.md) — the messages, with their fixes.

package/docs/guide/search-syncs/wiring.md

@@ -0,0 +1,188 @@
+# Wiring it over a Mongo
+
+`createSearchSyncs(mongo, config)` takes what
+[`@nxgt/mongo`](https://www.npmjs.com/package/@nxgt/mongo)'s `openMongo`
+returned and one entry per collection to follow — an index and a transform — and gives back
+one object holding every sync.
+
+```ts
+import { bindIndex } from '@nxgt/meilisearch';
+import { createSearchSyncs } from '@nxgt/mongo-meilisearch';
+import { mongo } from './db';                            // the app's openMongo
+import { meili } from './meili';                       // a Meilisearch client
+import { articleIndex, authorIndex } from './search';  // its defineIndexes
+
+const search = createSearchSyncs(mongo, {
+	articles: {
+		index: bindIndex(meili, articleIndex),
+		transform: (article) =>
+			article.draft ? null : { id: String(article._id), title: article.title },
+	},
+	authors: {
+		index: bindIndex(meili, authorIndex),
+		transform: (author) => ({ id: String(author._id), name: author.name }),
+	},
+});
+```
+
+`createSearchSyncs` sends nothing: it builds the syncs.
+[`reindexAll` and `start`](lifecycle.md) are what write.
+
+## The keys are the Mongo's own
+
+A key is the name a collection is **exported** under — the same one
+`mongo.db.articles` answers to, not the collection's name on the server. That
+is what lets the transform be written inline: its argument is typed by the
+collection the key names, and its result by the index that entry carries.
+
+```ts
+createSearchSyncs(mongo, {
+	nowhere: { index: bindIndex(meili, articleIndex), transform: () => null },
+});
+// Type error: index: 'createSearchSyncs: this Mongo wires no collection called "nowhere"'
+// TypeError at run time: createSearchSyncs: this Mongo wires no collection called "nowhere"
+```
+
+A key that is a member of the driver's `Db` — `command`, `watch` — is refused
+the same way: the Mongo never wires a collection under one.
+
+Only the collections you name are followed. A Mongo wiring twenty collections
+and a config naming two builds two syncs.
+
+## One database
+
+`createSearchSyncs` follows the collections of **one** database, as `mongo.db` itself
+does:
+
+```ts
+const mongo = await openMongo(
+	defineMongo({
+		databases: {
+			main: { uri, collections },
+			analytics: { uri, database: 'analytics', collections: events },
+		},
+	}),
+);
+
+createSearchSyncs(mongo, { articles: { index, transform } });
+// TypeError: createSearchSyncs: this Mongo holds 2 databases (main, analytics),
+// and createSearchSyncs follows the collections of one. Build one
+// `createSearchSyncs` per database, from a Mongo that wires that database alone
+```
+
+A Mongo of several has no sole collections, so its keys are `never` and the
+config does not compile either — every entry is refused, with the same
+`wires no collection called "…"` message. Build one Mongo per database,
+and one `createSearchSyncs` over each.
+
+## What an entry takes
+
+Everything [`createSearchSync`](../sync-lifecycle.md)
+takes **except `collection`**, which the Mongo already holds:
+
+| Option | Type | Default | Effect |
+| --- | --- | --- | --- |
+| `index` | `TypedIndex<I>` | — | Where the documents go: from `bindIndex` |
+| `transform` | `Transform<Col, I>` | — | The document as the index holds it; `null` keeps it out, and takes it out |
+| `toIndexId` | `ToIndexId<Col, I>` | `String` | The index id of a `_id`. Optional while the index's ids are strings, required otherwise |
+| `name` | `string` | `'<collection>:<index uid>'` | What this sync's resume point is recorded under |
+| `stateCollection` | `string` | `'nxgt_search_sync'` | Where resume points are kept, in the Mongo's database |
+| `batchSize` | `number` | `500` | Changes, or documents, sent at once |
+| `flushIntervalMs` | `number` | `1000` | How long a change waits for others |
+| `positionIntervalMs` | `number` | `60000` | How often a quiet sync records where its stream is |
+| `leaseMs` | `number` | `30000` | How long the lease on this sync's name lasts unrenewed; a running sync renews it every third of that |
+| `pageSize` | `number` | `100` | Documents a reindex reads per page |
+| `onHistoryLost` | `'reindex' \| 'fail'` | `'reindex'` | What `start` does when a resume point is older than the server's history |
+
+Each entry sets its own: a large collection can take a bigger `pageSize`
+while a small one keeps the default.
+
+```ts
+const search = createSearchSyncs(mongo, {
+	articles: {
+		index: bindIndex(meili, articleIndex),
+		transform: toArticleHit,
+		pageSize: 500,
+		onHistoryLost: 'fail',
+	},
+	authors: {
+		index: bindIndex(meili, authorIndex),
+		transform: toAuthorHit,
+		name: 'authors:public',
+	},
+});
+```
+
+**Two `createSearchSyncs` built from one config share their resume points**, since
+the default `name` is `'<collection>:<index uid>'`. Give `name` when two are
+meant to be different.
+
+## The whole wiring
+
+The clients, the Mongo, the index settings, the syncs:
+
+```ts
+// src/search/index.ts
+import { bindIndex } from '@nxgt/meilisearch';
+import { openMongo } from '@nxgt/mongo';
+import { createSearchSyncs } from '@nxgt/mongo-meilisearch';
+import { Meilisearch } from 'meilisearch';
+import { config } from '../db';
+import { articleIndex, authorIndex, toArticleHit, toAuthorHit } from './indexes';
+
+export const meili = new Meilisearch({
+	host: process.env.MEILI_HOST!,
+	apiKey: process.env.MEILI_KEY!,
+});
+
+export async function buildSearch() {
+	const mongo = await openMongo(config);
+
+	const search = createSearchSyncs(mongo, {
+		articles: { index: bindIndex(meili, articleIndex), transform: toArticleHit },
+		authors: { index: bindIndex(meili, authorIndex), transform: toAuthorHit },
+	});
+
+	// Every index created and its settings applied, before anything writes
+	// documents: a deployment step, beside `mongo.sync()`.
+	await search.syncIndexes();
+
+	return { mongo, search };
+}
+```
+
+`createSearchSyncs` **does not own the Mongo**: closing its syncs stops them
+and nothing else, and `mongo.close()` stays the caller's to make.
+
+## Signatures
+
+```ts
+function createSearchSyncs<C, const I extends IndexMap<I>>(
+	mongo: Mongo<C>,
+	config: SearchSyncsConfig<C, I>,
+): SearchSyncs<I>;
+
+type SearchSyncEntry<Col, I> = Omit<SearchSyncOptions<Col, I>, 'collection'>;
+
+type SearchSyncsConfig<C, I extends IndexMap<I>> = {
+	[K in keyof I]: K extends keyof SoleCollections<C>
+		? SoleCollections<C>[K] extends infer Col extends AnyCollectionDefinition
+			? SearchSyncEntry<Col, I[K]>
+			: never
+		: {
+				index: `createSearchSyncs: this Mongo wires no collection called "${K & string}"`;
+			};
+};
+
+type SoleCollections<C>; // what a Mongo's sole database holds, by export name
+type IndexMap<I> = { [K in keyof I]: AnyIndexDefinition };
+type ByKey<S, T> = { readonly [K in keyof S]: T };
+```
+
+`I` is inferred from each entry's `index` alone, which is what lets a
+`transform` be written inline.
+
+## Next
+
+- [The search syncs' lifecycle](lifecycle.md) — `reindexAll`, `start`, `failed`,
+  `flush` and `close`.

package/docs/roadmap.md

@@ -26,12 +26,30 @@ _Nothing queued._
   but a change to one of them does not reach the index: the sync follows the
   collection it was given. Run a sync per collection that has to move the
   index.
+- **`createSearchSyncs` over several databases** — a Mongo holding more than
+  one gives `never` for its keys, and `createSearchSyncs` throws naming them.
+  Build one `createSearchSyncs` per database, from a Mongo that wires that
+  database alone.
+- **`createSearchSyncs` owning the Mongo** — closing the search syncs stops
+  them and nothing else: the clients, the databases and the collections
+  belong to the Mongo, and `mongo.close()` stays the caller's.
+- **A lock of its own for `createSearchSyncs`** — the lease already provides
+  it: each sync takes a lease on its name, so a second process starting the
+  same syncs is refused with `RUNNING`.
 - **One index fed by two collections, or by another writer** — a reindex
   removes every document the collection does not give it, whoever wrote it.
   The sync owns its index.
 
 ## Shipped
 
+- **Several collections from one config** — `createSearchSyncs(mongo, config)`
+  over what `@nxgt/mongo`'s `openMongo` returns: one entry per collection,
+  under the key the Mongo wires it as, and one `syncIndexes`, `reindexAll`,
+  `start` and `close` for all of them, with `failed` for the first sync that
+  stops. It is `@nxgt/mongo-search-kit`, folded in and renamed
+  (`createSearchKit` is `createSearchSyncs`), whose own roadmap is carried
+  here: `syncIndexes()` and one follower per sync name across processes came
+  with its 0.2.0, and the package is now a deprecated re-export — 0.5.0.
 - **A standby waits exactly for the holder** — a `RUNNING` the lease refused
   carries `holder` and `expiresAt`, read from the lease document, so a process
   waiting for the name sleeps until the holder's lease lapses instead of a