@nxgt/mongo-meilisearch 0.1.6 → 0.1.7

package/README.md

@@ -225,10 +225,11 @@ out of range, an empty `name`, or a `transform` that is not a function.
 
 ## Not included
 
-- **Several processes sharing one sync.** There is no lock: run one
-  follower per sync name. Two do the same writes twice, and a `reindex` in
-  one while the other follows removes documents the follower has already
-  indexed and will not send again.
+- **Several processes sharing one sync — not yet.** There is no lock today,
+  so run **one** follower per sync name. Two do the same writes twice, and a
+  `reindex` in one while the other follows removes documents the follower has
+  already indexed and will not send again. A lease on a sync name is being
+  worked on: [the roadmap](docs/roadmap.md) says where it stands.
 - **Partial updates.** A change sends the whole document the transform
   gives, never a patch.
 - **Keeping the index's settings.** That is `@nxgt/meilisearch`'s `sync`.
@@ -289,8 +290,9 @@ that is `{ code, sync, cause? }`; both types are exported.
 Each is a `@ts-expect-error` case in this package's type tests.
 
 - A transform that reads a field the collection's schema does not have.
-- A transform that gives a field the index's document does not have, or an
-  id of the wrong type (an `ObjectId` for a string id).
+- A transform that leaves out a field the index's document has, or gives an
+  id of the wrong type (an `ObjectId` for a string id). A field the document
+  does not have, beside all the ones it does, is passed on to Meilisearch.
 - A transform that gives something other than a document or `null`.
 - No `toIndexId` when the index's ids are not strings; one that gives
   another type than the index's ids, or takes another than the collection's.
@@ -318,9 +320,10 @@ Each is a `@ts-expect-error` case in this package's type tests.
 - **Without post-images, a change carries the document as it is now**, not
   as the change left it (`@nxgt/mongo`'s change streams). For an index, where
   only the latest state counts, that is what you want.
-- **Only one process per sync name.** There is no lock; see *Not included*.
-  Inside one process this package refuses it: `reindex()` and a second
-  `start()` throw `RUNNING` while a sync of the same object is following.
+- **Only one process per sync name**, while there is no lock; see *Not
+  included*. Inside one process this package refuses it: `reindex()` and a
+  second `start()` throw `RUNNING` while a sync of the same object is
+  following.
 - **A dropped collection stops the sync** (`'invalidated'`) and leaves the
   index as it was. What the collection had is removed by the reindex the
   next `start` runs.
@@ -335,6 +338,19 @@ Each is a `@ts-expect-error` case in this package's type tests.
   `documents.delete` and `tasks.get`, plus `indexes.create` unless the index
   already exists — the first write to an index that does not creates it.
 
+## Documentation
+
+- [Guide index](docs/README.md) — every page, and when to read it.
+- [The sync's lifecycle](docs/guide/sync-lifecycle.md) — the two definitions,
+  the transform, and every option with its default.
+- [Reindexing](docs/guide/reindex.md) — the full fill, and what it removes.
+- [Following changes](docs/guide/following-changes.md) — batches, the resume
+  point, and how a sync stops.
+- [What it leaves out](docs/guide/boundaries.md) — the settings, the joins,
+  and the one follower per sync name.
+- [Troubleshooting](docs/troubleshooting.md) — the errors, by their message.
+- [Roadmap](docs/roadmap.md) — what is next, and what is not planned.
+
 ## License
 
 MIT

package/docs/README.md

@@ -0,0 +1,13 @@
+# `@nxgt/mongo-meilisearch` documentation
+
+The [README](../README.md) is the short version: what the package is, and one
+example per area. These pages are the long one.
+
+| Page | Read it when |
+| --- | --- |
+| [The sync's lifecycle](guide/sync-lifecycle.md) | you are wiring a collection to an index for the first time, or looking for an option and its default |
+| [Reindexing](guide/reindex.md) | the index has to be filled, or refilled, from the collection |
+| [Following changes](guide/following-changes.md) | a process has to keep the index in step, and stop cleanly — or it stopped and you need to know why |
+| [What it leaves out](guide/boundaries.md) | you are about to run two followers, join another collection in, or wonder who applies the index settings |
+| [Troubleshooting](troubleshooting.md) | something threw, and you have the message |
+| [Roadmap](roadmap.md) | you want to know what is coming, and what will not |

package/docs/guide/boundaries.md

@@ -0,0 +1,131 @@
+# What it leaves out
+
+The sync writes documents into one index from one collection. Everything
+below is deliberately somebody else's job — either another package's, or the
+caller's — so that the sync stays a function of the collection it was given.
+
+```ts
+import { bindIndex } from '@nxgt/meilisearch';
+import { getCollection } from '@nxgt/mongo';
+import { createSearchSync } from '@nxgt/mongo-meilisearch';
+import { db, meili } from './clients';            // a driver Db, an SDK client
+import { articles, articleIndex } from './search';
+
+const articleSearch = createSearchSync({
+	collection: getCollection(db, articles),
+	index: bindIndex(meili, articleIndex),
+	transform: (article) => ({ id: String(article._id), title: article.title }),
+});
+
+// One follower, in one process: everything below is what that costs.
+const running = await articleSearch.start();
+```
+
+## Several processes sharing one sync — not yet
+
+There is **no lock today**: run one follower per sync name. Two processes
+under one name do the same writes twice, and both are wrong the moment one of
+them reindexes: a `reindex()` in one while the other follows removes
+documents the follower has already indexed and will never send again.
+
+Inside **one** process the package does refuse it: a second `start()`, or a
+`reindex()`, throws `SearchSyncError` with the code `RUNNING`.
+
+Give two syncs different `name`s when they are meant to be independent — the
+default, `'<collection>:<index uid>'`, is shared by any two syncs over the
+same pair.
+
+A lease on a sync name, renewed while a follower runs, is being worked on;
+[the roadmap](../roadmap.md) says where it stands. Until it ships, one writer
+per name is the rule.
+
+## Partial updates
+
+A change sends the **whole** document the transform gives, never a patch:
+
+```ts
+transform: (article) => ({
+	id: String(article._id),
+	title: article.title,
+	body: article.body,
+});
+```
+
+That is what makes a change safe to apply twice, which it may be — the resume
+point moves after a batch is applied, so a restart re-sends what was in
+flight.
+
+## The index's settings
+
+`searchableAttributes`, `filterableAttributes`, the ranking rules: they are
+`@nxgt/meilisearch`'s, applied as a deployment step.
+
+```ts
+import { bindIndex, syncIndexes } from '@nxgt/meilisearch';
+
+const index = bindIndex(meili, articleIndex);
+await index.sync();                       // this one index
+await syncIndexes(meili, [articleIndex]); // or every index of the app at once
+```
+
+This package writes documents, and creates nothing but them — the first write
+to an index that does not exist is what creates it, with no settings of its
+own.
+
+## Joining other collections in
+
+The transform may read another collection, but a change to **that**
+collection does not reach the index: the sync follows the one it was given.
+
+```ts
+export const articleIndex = defineIndex<{
+	id: string;
+	title: string;
+	author: string;
+}>()({ uid: 'articles', primaryKey: 'id' });
+
+import { authors } from './search'; // the authors collection's definition
+
+createSearchSync({
+	collection: getCollection(db, articles),
+	index: bindIndex(meili, articleIndex),
+	// The author's name is read here, so it is right when the article is
+	// written — and stale from the moment the author is renamed: nothing is
+	// following the authors.
+	transform: async (article) => ({
+		id: String(article._id),
+		title: article.title,
+		author:
+			(await getCollection(db, authors).findById(article.authorId))?.name ?? '',
+	}),
+});
+```
+
+Run a sync per collection that has to move the index, or denormalise the
+field into the collection the sync follows and let its own writes carry it.
+
+## One index, one collection
+
+A reindex removes every document the collection does not give it, whoever
+wrote it. Two collections pointed at one index, or an index another process
+also writes to, lose documents at the first reindex. The sync owns its index.
+
+## What the caller still owns
+
+- **Restarting a sync that stopped.** `closed` rejects with a
+  `SearchSyncError`; the supervisor is what brings the process back. See
+  [Following changes](following-changes.md).
+- **Choosing when a full reindex runs**, with `onHistoryLost: 'fail'`.
+- **The two clients.** This package creates neither the `MongoClient` nor the
+  Meilisearch client, and closes neither.
+- **The permissions.** The MongoDB user needs `find` and `changeStream` on
+  the collection and `find`, `insert`, `update` and `delete` on
+  `stateCollection`; the Meilisearch key needs `documents.add`,
+  `documents.get`, `documents.delete` and `tasks.get`, plus `indexes.create`
+  unless the index already exists.
+
+## Next
+
+- [The sync's lifecycle](sync-lifecycle.md) — the options behind each of
+  these.
+- [Roadmap](../roadmap.md) — what is being worked on, and what is not planned.

package/docs/guide/following-changes.md

@@ -0,0 +1,195 @@
+# Following changes
+
+`start()` follows the collection's change stream into the index, from where
+the last run stopped, and gives back the running sync.
+
+```ts
+import { bindIndex } from '@nxgt/meilisearch';
+import { getCollection, type ReadDocumentOf } from '@nxgt/mongo';
+import { createSearchSync } from '@nxgt/mongo-meilisearch';
+import { db, meili } from './clients';            // a driver Db, an SDK client
+import { articles, articleIndex } from './search';
+
+const collection = getCollection(db, articles);
+const index = bindIndex(meili, articleIndex);
+const transform = (article: ReadDocumentOf<typeof articles>) =>
+	article.draft ? null : { id: String(article._id), title: article.title };
+
+const articleSearch = createSearchSync({ collection, index, transform });
+
+const running = await articleSearch.start();
+await running.ready;   // already resolved: start waits for the stream to open
+await running.flush(); // sends what is waiting now, and records how far it got
+await running.close(); // flushes, then stops
+```
+
+`RunningSearchSync` is `AsyncDisposable`, so
+`await using running = await articleSearch.start()` closes it at the end of
+the block.
+
+| Member | Type | Effect |
+| --- | --- | --- |
+| `ready` | `Promise<void>` | Resolves once changes are being heard. `start` has already awaited it |
+| `closed` | `Promise<CloseReason>` | `'closed'` after `close()`, `'invalidated'` when the collection was dropped or renamed. **Rejects** with a `SearchSyncError` when an error stopped it |
+| `flush()` | `Promise<void>` | Sends what is waiting, and records the point it covers |
+| `close()` | `Promise<void>` | Flushes, then stops |
+
+## Where it starts from
+
+- **Nothing recorded** — the first run under this `name` — it
+  [reindexes](reindex.md), then follows from the position the reindex saved.
+- **Something recorded**: it resumes there, so a change made while no process
+  was following is applied now.
+- **A position the server's history no longer reaches**: see
+  [When the history is gone](#when-the-history-is-gone).
+
+## What reaches the index
+
+Every create, update, soft delete, restore and hard delete of the collection
+runs through the transform:
+
+```ts
+await collection.create({ title: 'written' });   // added to the index
+await collection.update(id, { draft: true });    // transform returns null → removed
+await collection.update(id, { draft: false });   // back in
+await collection.delete(id);                     // soft delete → removed
+await collection.hardDelete(id);                 // removed
+```
+
+Changes are sent in **batches**: one waits `flushIntervalMs` (1000 ms) for
+others, or goes at once as soon as `batchSize` (500) are waiting. Several
+changes to one document within a batch send only the last — the batch holds
+one entry per id, so adds and deletes never race.
+
+The resume point is recorded **after** the batch is applied. What was not
+sent when a process stops is sent again by the next `start`, so a change may
+reach the index twice; the transform is a function of the document, so
+applying it twice is applying it once.
+
+## A quiet collection keeps its position fresh
+
+Every `positionIntervalMs` (60 s), a sync with nothing to send records where
+the stream is anyway — one small write per interval, and none at all while
+the stream itself does not move.
+
+Without it, a collection nothing writes to for longer than the server's
+history covers would need a full reindex at the next start.
+
+## When the history is gone
+
+MongoDB keeps a bounded history of changes, the oplog. A sync stopped for
+longer than it covers cannot resume from its recorded point. By default
+`start` reindexes and follows from there. `onHistoryLost: 'fail'` hands the
+decision back:
+
+```ts
+import { SearchSyncError } from '@nxgt/mongo-meilisearch';
+
+const articleSearch = createSearchSync({
+	collection,
+	index,
+	transform,
+	onHistoryLost: 'fail',
+});
+
+let running: RunningSearchSync;
+try {
+	running = await articleSearch.start();
+} catch (error) {
+	if (error instanceof SearchSyncError && error.code === 'HISTORY_LOST') {
+		await articleSearch.reindex(); // when it suits you
+		running = await articleSearch.start();
+	} else throw error;
+}
+```
+
+`onHistoryLost` applies to `start` alone. A history that runs out **under** a
+running sync stops it with `FAILED`; the next `start` is where the choice is
+made again.
+
+## When it stops
+
+```ts
+running.closed.then(
+	(reason) => console.info(`sync stopped: ${reason}`), // 'closed' | 'invalidated'
+	(error: SearchSyncError) => {
+		console.error({ sync: error.sync, code: error.code, cause: error.cause });
+		process.exit(1); // let the supervisor restart it
+	},
+);
+```
+
+- **`close()`** — `closed` resolves `'closed'`. What was waiting is flushed
+  and recorded first.
+- **The collection is dropped or renamed** — `closed` resolves
+  `'invalidated'`, and the recorded point is forgotten, since nothing could
+  resume from inside a collection that no longer exists. The next `start`
+  reindexes what the recreated collection holds.
+- **An error** — `closed` rejects with a `SearchSyncError`, and so do `flush`
+  and `close` from then on. Nothing past the last applied batch was recorded,
+  so the next `start` sends it again.
+
+**Await `closed`, or catch it.** A rejection nobody handles ends the process.
+
+| `code` | When |
+| --- | --- |
+| `HISTORY_LOST` | `start` with `onHistoryLost: 'fail'`, and the recorded point is older than the server's history. `cause` is `@nxgt/mongo`'s `DataError`, `serverCode` 286 or 280 |
+| `ID_MISMATCH` | the transform gave a document whose primary key is not its index id |
+| `RUNNING` | a second `start()`, or a `reindex()`, while this sync is already following in this process |
+| `FAILED` | anything else: the transform threw, MongoDB or Meilisearch refused. The message says what the sync was doing, and `cause` carries the original error |
+
+[Troubleshooting](../troubleshooting.md) has each message with its fix.
+
+## In a worker
+
+```ts
+const running = await articleSearch.start();
+
+const stop = async () => {
+	await running.close(); // flushes and records before exiting
+	await mongo.close();
+	process.exit(0);
+};
+process.on('SIGTERM', stop);
+process.on('SIGINT', stop);
+
+await running.closed; // rejects if the sync falls over; the supervisor restarts it
+```
+
+One process per sync name: there is no lock, and two followers do the same
+writes twice. See [What it leaves out](boundaries.md).
+
+## Signatures
+
+```ts
+interface SearchSync {
+	start(): Promise<RunningSearchSync>;
+}
+
+interface RunningSearchSync extends AsyncDisposable {
+	readonly ready: Promise<void>;
+	readonly closed: Promise<CloseReason>; // 'closed' | 'invalidated'
+	flush(): Promise<void>;
+	close(): Promise<void>;
+}
+
+class SearchSyncError extends Error {
+	readonly code: SearchSyncErrorCode; // 'HISTORY_LOST' | 'ID_MISMATCH' | 'RUNNING' | 'FAILED'
+	readonly sync: string;
+	constructor(message: string, options: SearchSyncErrorOptions);
+}
+
+interface SearchSyncErrorOptions {
+	code: SearchSyncErrorCode;
+	sync: string;
+	cause?: unknown;
+}
+```
+
+`CloseReason` is `@nxgt/mongo`'s; its third value, `'failed'`, never occurs
+here — a failure rejects instead.
+
+## Next
+
+- [Reindexing](reindex.md) — the fill `start` runs the first time.
+- [What it leaves out](boundaries.md).

package/docs/guide/reindex.md

@@ -0,0 +1,132 @@
+# Reindexing
+
+`reindex()` fills the index from the collection: every live document through
+the transform, and then out of the index everything the collection no longer
+gives it.
+
+```ts
+import { bindIndex } from '@nxgt/meilisearch';
+import { getCollection } from '@nxgt/mongo';
+import { createSearchSync } from '@nxgt/mongo-meilisearch';
+import { db, meili } from './clients';            // a driver Db, an SDK client
+import { articles, articleIndex } from './search';
+
+const articleSearch = createSearchSync({
+	collection: getCollection(db, articles),
+	index: bindIndex(meili, articleIndex),
+	transform: (article) =>
+		article.draft ? null : { id: String(article._id), title: article.title },
+});
+
+const report = await articleSearch.reindex();
+// { indexed: 1204, skipped: 17, removed: 3 }
+```
+
+| Field | Type | Effect |
+| --- | --- | --- |
+| `indexed` | `number` | Documents sent to the index |
+| `skipped` | `number` | Live documents the transform returned `null` for |
+| `removed` | `number` | Documents the index held and the collection no longer gives it |
+
+## What it does, in order
+
+1. **Takes the collection's current position** — a change stream's first read
+   answers with it, without waiting for a change.
+2. **Reads every live document**, `pageSize` at a time (100 by default,
+   lowered to the collection's `maxPageSize` when it asks for more), runs the
+   transform, and sends what it gives in batches of `batchSize` (500).
+   Soft-deleted documents are not live, so they are never sent.
+3. **Pages the whole index** and deletes every document whose id the
+   collection did not just give it: deleted, turned away by the transform, or
+   never from this collection at all.
+4. **Records the position from step 1** as the resume point, and stamps
+   `reindexedAt`.
+
+Step 1 before step 2 is what makes it safe to reindex a live collection: a
+change made while the documents are read is followed again from that
+position, so nothing falls between the reindex and the stream. A reindex that
+throws records nothing, and the next one starts over.
+
+```ts
+const before = await articleSearch.state(); // undefined, the first time
+await articleSearch.reindex();
+const after = await articleSearch.state();
+after?.reindexedAt; // a Date
+```
+
+## When to run it
+
+- **The first `start`** runs it for you, when nothing is recorded under the
+  sync's name.
+- **After the transform changes** — a new field, a different rule for `null`
+  — since the index still holds what the old one produced.
+- **After a stop longer than the server's change history**, which `start`
+  does itself unless `onHistoryLost: 'fail'` asks to decide.
+  See [Following changes](following-changes.md).
+
+It is a **deployment step, not a request-time one**: it holds one id per live
+document in memory and scans the whole index, which on a collection of
+millions is hundreds of megabytes and a full pass over the index.
+
+## Not beside its own follower
+
+```ts
+const running = await articleSearch.start();
+await articleSearch.reindex();
+// SearchSyncError: Search sync "articles:articles" is already following
+// changes in this process: close it before you reindex.
+```
+
+The code is `RUNNING`. A reindex removes what the follower has just indexed,
+and the follower, having already handled that change, would never send it
+again. Close the follower, reindex, start again:
+
+```ts
+await running.close();
+await articleSearch.reindex();
+const fresh = await articleSearch.start();
+```
+
+The check is per object and per process: the package has no lock across
+processes, and [does not pretend to](boundaries.md).
+
+## Errors
+
+Anything that goes wrong while reindexing comes back as a `SearchSyncError`
+with the code `FAILED` — MongoDB refused a read, Meilisearch refused a batch,
+the transform threw — and the original error as its `cause`:
+
+```ts
+try {
+	await articleSearch.reindex();
+} catch (error) {
+	if (error instanceof SearchSyncError) {
+		console.error(error.code, error.sync, error.cause);
+	}
+	throw error;
+}
+```
+
+`ID_MISMATCH` is the one exception that means something more precise: the
+transform gave a document whose primary key is not that document's index id.
+[Troubleshooting](../troubleshooting.md) has both messages.
+
+## Signatures
+
+```ts
+interface SearchSync {
+	reindex(): Promise<ReindexReport>;
+}
+
+interface ReindexReport {
+	indexed: number;
+	skipped: number;
+	removed: number;
+}
+```
+
+## Next
+
+- [Following changes](following-changes.md) — what runs after the fill.
+- [The sync's lifecycle](sync-lifecycle.md) — `pageSize`, `batchSize` and the
+  rest.

package/docs/guide/sync-lifecycle.md

@@ -0,0 +1,234 @@
+# The sync's lifecycle
+
+A sync is the pairing of one MongoDB collection with one Meilisearch index:
+`createSearchSync` describes it, and then `reindex` fills the index and
+`start` keeps it in step.
+
+```ts
+import { bindIndex } from '@nxgt/meilisearch';
+import { getCollection } from '@nxgt/mongo';
+import { createSearchSync } from '@nxgt/mongo-meilisearch';
+import { db, meili } from './clients';            // a driver Db, an SDK client
+import { articles, articleIndex } from './search';
+
+const articleSearch = createSearchSync({
+	collection: getCollection(db, articles),
+	index: bindIndex(meili, articleIndex),
+	transform: (article) =>
+		article.draft
+			? null
+			: { id: String(article._id), title: article.title, body: article.body },
+});
+
+const running = await articleSearch.start(); // reindexes the first time
+// …
+await running.close();
+```
+
+`createSearchSync` **sends nothing**: it validates its options, resolves the
+defaults, and gives back an object. Every write comes from
+[`reindex`](reindex.md) or [`start`](following-changes.md).
+
+## The two definitions
+
+The collection is `@nxgt/mongo`'s and the index is `@nxgt/meilisearch`'s;
+this package creates neither, and neither knows about it.
+
+```ts
+import { defineIndex } from '@nxgt/meilisearch';
+import { defineCollection, id } from '@nxgt/mongo';
+import { z } from 'zod';
+
+export const articles = defineCollection({
+	name: 'articles',
+	schema: z.object({
+		_id: id(),
+		title: z.string(),
+		body: z.string(),
+		draft: z.boolean().default(false),
+	}),
+	timestamps: true,
+	softDelete: true,
+});
+
+export interface ArticleHit {
+	id: string;
+	title: string;
+	body: string;
+}
+
+export const articleIndex = defineIndex<ArticleHit>()({
+	uid: 'articles',
+	primaryKey: 'id',
+	settings: { searchableAttributes: ['title', 'body'] },
+});
+```
+
+The index's **settings** are `@nxgt/meilisearch`'s to apply, as a deployment
+step beside the collection's own sync:
+
+```ts
+const index = bindIndex(meili, articleIndex);
+await index.sync(); // creates the index and applies its settings
+```
+
+## The transform
+
+It takes the collection's document, typed by its schema, and gives the
+index's document, typed by its definition. `null` keeps a document **out** of
+the index — and takes it out if it was in.
+
+```ts
+const transform = (article: ReadDocumentOf<typeof articles>) =>
+	article.draft
+		? null
+		: { id: String(article._id), title: article.title, body: article.body };
+```
+
+It runs for every document of a reindex and for every change that is
+followed, so a document that stops qualifying is removed the moment it does.
+It may be `async` — the type allows a promise — and it should be a function
+of its document alone: a change may be applied twice, and a transform that
+throws stops the sync.
+
+The document it returns must carry its **index id** under the index's primary
+key. One under another id could never be taken out again, so it is refused
+with `ID_MISMATCH`.
+
+## Options
+
+| Option | Type | Default | Effect |
+| --- | --- | --- | --- |
+| `collection` | `TypedCollection<C>` | — | Where the documents are: from `getCollection` |
+| `index` | `TypedIndex<I>` | — | Where they go: from `bindIndex` |
+| `transform` | `Transform<C, I>` | — | The document as the index holds it, or `null` to keep it out |
+| `toIndexId` | `ToIndexId<C, I>` | `String` | The index id of a `_id`. Optional while the index's ids are strings, required otherwise |
+| `name` | `string` | `'<collection>:<index uid>'` | What the resume point is recorded under. Two syncs under one name share it |
+| `stateCollection` | `string` | `'nxgt_search_sync'` | Where the resume point is kept, in the collection's database |
+| `batchSize` | `number` | `500` | Changes, or documents, sent to Meilisearch at once |
+| `flushIntervalMs` | `number` | `1000` | How long a change waits for others; `0` sends at the next tick |
+| `positionIntervalMs` | `number` | `60000` | How often a sync with nothing to send records where the stream is |
+| `pageSize` | `number` | `100` | Documents a reindex reads per page; above the collection's `maxPageSize`, lowered to it |
+| `onHistoryLost` | `'reindex' \| 'fail'` | `'reindex'` | What `start` does when the resume point is older than the server's change history |
+
+### `toIndexId`
+
+`String(_id)` by default, which suits an `ObjectId`. When the index's primary
+key is not a string, the option stops being optional:
+
+```ts
+createSearchSync({
+	collection: getCollection(db, counters), // _id: z.int()
+	index: bindIndex(meili, counterIndex),   // primaryKey: 'n', a number
+	toIndexId: (id) => id,
+	transform: (counter) => ({ n: counter._id, label: counter.label }),
+});
+```
+
+### `name` and `stateCollection`
+
+The resume point is a document in the collection's own database, so a restart
+picks up where the last run stopped:
+
+```ts
+const sync = createSearchSync({
+	collection,
+	index,
+	transform,
+	name: 'articles:public',
+});
+
+await sync.state();
+// { _id: 'articles:public', resumeToken: …, updatedAt: …, reindexedAt: … }
+```
+
+`state()` is `undefined` until the first reindex records something.
+
+### `batchSize`, `flushIntervalMs`, `pageSize`
+
+`batchSize`, `positionIntervalMs` and `pageSize` must be whole numbers above
+zero and `flushIntervalMs` a whole number of milliseconds, or
+`createSearchSync` throws a `TypeError` before anything runs — as it does for
+an empty `name` or a `transform` that is not a function.
+
+```ts
+createSearchSync({ collection, index, transform, batchSize: 0 });
+// TypeError: createSearchSync: batchSize must be a whole number above 0, not 0
+```
+
+## A worker process
+
+Wiring, settings, first fill and shutdown, in the order they happen:
+
+```ts
+import { bindIndex } from '@nxgt/meilisearch';
+import { connectMongo, getCollection } from '@nxgt/mongo';
+import { createSearchSync, SearchSyncError } from '@nxgt/mongo-meilisearch';
+import { Meilisearch } from 'meilisearch';
+import { articles, articleIndex } from './search';
+
+const mongo = await connectMongo(process.env.MONGO_URI!);
+const meili = new Meilisearch({
+	host: process.env.MEILI_HOST!,
+	apiKey: process.env.MEILI_KEY!,
+});
+
+const index = bindIndex(meili, articleIndex);
+await index.sync();
+
+const articleSearch = createSearchSync({
+	collection: getCollection(mongo.db, articles),
+	index,
+	transform: (article) =>
+		article.draft
+			? null
+			: { id: String(article._id), title: article.title, body: article.body },
+});
+
+const running = await articleSearch.start();
+process.on('SIGTERM', () => {
+	void running.close().then(() => mongo.close());
+});
+
+// Rejects with a SearchSyncError if the sync stops on an error.
+await running.closed.catch((error: SearchSyncError) => {
+	console.error({ sync: error.sync, code: error.code, cause: error.cause });
+	process.exit(1); // let the supervisor restart it
+});
+```
+
+## Signatures
+
+```ts
+function createSearchSync<
+	C extends AnyCollectionDefinition,
+	I extends AnyIndexDefinition,
+>(options: SearchSyncOptions<C, I>): SearchSync;
+
+type Transform<C, I> = (
+	document: ReadDocumentOf<C>,
+) => DocumentOf<I> | null | Promise<DocumentOf<I> | null>;
+
+type ToIndexId<C, I> = (id: IdOf<C>) => IdOf<I>;
+
+interface SearchSync {
+	readonly name: string;
+	reindex(): Promise<ReindexReport>;
+	start(): Promise<RunningSearchSync>;
+	state(): Promise<SearchSyncState | undefined>;
+}
+
+interface SearchSyncState {
+	_id: string;
+	resumeToken: ResumeToken;
+	updatedAt: Date;
+	reindexedAt: Date | undefined;
+}
+```
+
+## Next
+
+- [Reindexing](reindex.md) — the full fill, and what it removes.
+- [Following changes](following-changes.md) — the change stream, its batches
+  and its errors.
+- [What it leaves out](boundaries.md).

package/docs/roadmap.md

@@ -0,0 +1,62 @@
+# Roadmap
+
+Where `@nxgt/mongo-meilisearch` is going. A direction, not a commitment: the
+version an item shipped in is the only number on this page.
+
+## Now
+
+- **A lease on a sync name, renewed while it runs** — a follower holds the
+  sync's name for a bounded time and renews it as it works, so a process that
+  dies is taken over once its lease expires rather than leaving the index
+  behind, and a name nothing is following no longer blocks the next start.
+
+## Next
+
+_Nothing queued._
+
+## Later
+
+_Nothing queued._
+
+## Not planned
+
+- **Partial updates** — a change sends the whole document the transform gives,
+  never a patch. The transform is a function of the document alone, which is
+  also what makes a change safe to apply twice.
+- **Keeping the index's settings** — that is `@nxgt/meilisearch`'s `syncIndex`
+  / `syncIndexes`, run as a deployment step. This package writes documents.
+- **Joining other collections into a document** — the transform may read them,
+  but a change to one of them does not reach the index: the sync follows the
+  collection it was given. Run a sync per collection that has to move the
+  index.
+- **One index fed by two collections, or by another writer** — a reindex
+  removes every document the collection does not give it, whoever wrote it.
+  The sync owns its index.
+
+## Shipped
+
+- **Documentation that travels with the package** — a guide page for the
+  sync's lifecycle, following a collection's changes, `reindex` and where the
+  package's boundaries are, a troubleshooting page whose headings are the
+  exact error text, and this roadmap, installed in `docs/` rather than left on
+  GitHub — 0.1.7.
+- **`@nxgt/mongo` 0.15.0** — a deduplicated file write two callers cannot
+  both win — 0.1.6.
+- **`@nxgt/mongo` 0.14.0** — files under its `./gridfs` subpath — 0.1.5.
+- **`@nxgt/mongo` 0.13.0** — `upsert` in one round trip — 0.1.4.
+- **A reindex's cost written down** — it holds one id per live document in
+  memory and pages the whole index, so it is a deployment step — 0.1.3.
+- **`@nxgt/mongo` 0.12.0** — strings from outside read from the schema —
+  0.1.2.
+- **A peer range that matches what it is built against** — `@nxgt/mongo`
+  `^0.11.0`, which has the `position` this package records while a collection
+  is quiet — 0.1.1.
+- **First release** — `createSearchSync` with a transform typed by both
+  definitions (`null` keeps a document out), `reindex`, and `start`, which
+  follows the collection's changes in batches and resumes from a point
+  recorded in MongoDB, reindexing when the server's history no longer reaches
+  it; `SearchSyncError` with `HISTORY_LOST`, `ID_MISMATCH`, `RUNNING` and
+  `FAILED` — 0.1.0.
+
+Everything released is in [`CHANGELOG.md`](https://github.com/softistx/nxgt-data/blob/develop/packages/mongo-meilisearch/CHANGELOG.md) — it is not in
+the published package, only in the repository.

package/docs/troubleshooting.md

@@ -0,0 +1,380 @@
+# Troubleshooting
+
+Every heading is the text the error prints, so the page can be searched with
+what you have in front of you. Stacks, ids and paths are cut, and a sync's
+name is written as it comes out by default — `<collection>:<index uid>`, here
+`articles:articles`.
+
+Everything this package throws is a `SearchSyncError` carrying a `code`
+(`HISTORY_LOST`, `ID_MISMATCH`, `RUNNING`, `FAILED`), the sync's `name`, and
+the original error as `cause` — except the options, which are refused with a
+`TypeError` before anything is opened.
+
+| Area | Entries |
+| --- | --- |
+| [Install](#install) | [ERESOLVE](#npm-error-eresolve-unable-to-resolve-dependency-tree) · [incorrect peer dependency](#warn-incorrect-peer-dependency-nxgtmongo0140) · [TS2307](#error-ts2307-cannot-find-module-nxgtmongo-or-its-corresponding-type-declarations) |
+| [Options](#options) | [transform](#createsearchsync-transform-must-be-a-function) · [batchSize](#createsearchsync-batchsize-must-be-a-whole-number-above-0-not-0) · [flushIntervalMs](#createsearchsync-flushintervalms-must-be-a-whole-number-of-milliseconds-not--1) · [name](#createsearchsync-name-must-not-be-empty) |
+| [Starting](#starting) | [no replica set](#search-sync-articlesarticles-failed-starting-the-changestream-stage-is-only-supported-on-replica-sets) · [privileges](#search-sync-articlesarticles-failed-reindexing-not-authorized-on-app-to-execute-command--aggregate-articles-pipeline---changestream-----) · [history lost](#search-sync-articlesarticles-was-last-at-a-point-the-servers-change-history-no-longer-reaches-reindex-it-or-start-it-with-onhistorylost-reindex) · [already running](#search-sync-articlesarticles-is-already-following-changes-in-this-process-close-it-before-you-start-it-twice) |
+| [The transform](#the-transform) | [an id that is not the index's](#search-sync-articlesarticles-transform-gave-id-other-for-the-document--whose-index-id-is-) · [not a document](#transform-must-return-a-document-or-null-not-nope) · [it threw](#search-sync-articlesarticles-failed-following-changes-boom) |
+| [Sending](#sending) | [an id Meilisearch refuses](#search-sync-articlesarticles-failed-sending-changes-task-3-documentadditionorupdate-on-index-articles-failed-document-identifier--is-invalid) |
+| [Stopping](#stopping) | [a dropped collection](#the-sync-stops-and-closed-resolves-with-invalidated) |
+
+## Install
+
+### `npm error ERESOLVE unable to resolve dependency tree`
+
+```
+npm error Found: @nxgt/mongo@0.14.0
+npm error Could not resolve dependency:
+npm error peer @nxgt/mongo@"^0.15.0" from @nxgt/mongo-meilisearch@0.1.6
+```
+
+**When:** `npm install`, before anything is downloaded.
+
+**Why:** this package is a bridge: `@nxgt/mongo` and `@nxgt/meilisearch` are
+**required peers**, and the ranges are carets on 0.x versions, so each accepts
+one minor only. The collection and the index are built by those two packages
+and handed here, so they have to be the copies this version was built against.
+`mongodb`, `meilisearch` and `typescript` are peers on the same terms.
+
+**Fix:**
+
+```sh
+npm install @nxgt/mongo@^0.15.0 @nxgt/meilisearch@^0.1.0 @nxgt/mongo-meilisearch mongodb meilisearch zod
+```
+
+Raise the siblings rather than install past the conflict: `--force` and
+`--legacy-peer-deps` leave two copies of a sibling in the tree, and the
+definitions one of them builds are not the ones this package reads.
+
+### `warn: incorrect peer dependency "@nxgt/mongo@0.14.0"`
+
+**When:** `bun install`, which prints it and carries on.
+
+**Why:** bun does not fail on an unmet peer — it warns and installs what the
+manifest asked for. The bridge then runs against a sibling it was not built
+against, and what is missing shows up at the first call that needs it rather
+than at install.
+
+**Fix:**
+
+```sh
+bun add @nxgt/mongo@^0.15.0 @nxgt/meilisearch@^0.1.0
+```
+
+Treat that warning as an error. `bun pm ls` shows which versions were resolved.
+
+### `error TS2307: Cannot find module '@nxgt/mongo' or its corresponding type declarations.`
+
+At run time the same tree gives
+`Error [ERR_MODULE_NOT_FOUND]: Cannot find package '@nxgt/mongo'` under Node,
+and `error: Cannot find module '@nxgt/mongo'` under Bun.
+
+**When:** the first build or the first import of your own code, typically
+under pnpm or npm with a strict `node_modules` layout.
+
+**Why:** a peer installed *for* this package is not a dependency of **yours**.
+It is resolved under the bridge alone, so `@nxgt/mongo-meilisearch` finds it
+and your own `import { getCollection } from '@nxgt/mongo'` does not.
+
+**Fix:**
+
+```sh
+bun add @nxgt/mongo @nxgt/meilisearch @nxgt/mongo-meilisearch mongodb meilisearch zod
+```
+
+You import both siblings yourself — `getCollection` and `bindIndex` are what
+build the two arguments this package takes.
+
+## Options
+
+`createSearchSync` sends nothing and opens nothing: everything below throws
+where the sync is described.
+
+### `createSearchSync: transform must be a function`
+
+**When:** calling `createSearchSync`.
+
+**Why:** `transform` is the one required function of the options, and it is
+missing or is not callable — usually an object destructured from a config, or
+an `await import` whose default was not unwrapped.
+
+**Fix:**
+
+```ts
+createSearchSync({
+	collection,
+	index,
+	transform: (article) => ({ id: String(article._id), title: article.title }),
+});
+```
+
+### `createSearchSync: batchSize must be a whole number above 0, not 0`
+
+The same refusal covers `positionIntervalMs` and `pageSize`.
+
+**When:** calling `createSearchSync`.
+
+**Why:** those three count documents or milliseconds, and `0`, a fraction,
+`NaN` and a negative number have no meaning for any of them. A value read from
+the environment is a string until it is parsed, and `Number('')` is `0`.
+
+**Fix:**
+
+```ts
+createSearchSync({ collection, index, transform, batchSize: 500 }); // the default
+```
+
+### `createSearchSync: flushIntervalMs must be a whole number of milliseconds, not -1`
+
+**When:** calling `createSearchSync`.
+
+**Why:** `flushIntervalMs` is the one option that accepts `0` — send every
+change as it comes — so it is checked apart from the three above. A negative
+number or a fraction is still refused.
+
+**Fix:**
+
+```ts
+createSearchSync({ collection, index, transform, flushIntervalMs: 1000 }); // the default
+```
+
+### `createSearchSync: name must not be empty`
+
+**When:** calling `createSearchSync` with `name: ''`.
+
+**Why:** the name is the `_id` of the document the resume point is recorded
+under, so it cannot be empty. Left out, it is `<collection>:<index uid>`.
+
+**Fix:**
+
+```ts
+createSearchSync({ collection, index, transform, name: 'articles-search' });
+```
+
+Give a name when two syncs would otherwise share the default one — and keep it
+stable, because changing it loses the recorded point and the next `start`
+reindexes.
+
+## Starting
+
+### `Search sync "articles:articles" failed starting: The $changeStream stage is only supported on replica sets`
+
+Code `FAILED`; the `cause` is the driver's `MongoServerError`, code 40573. The
+same server message comes out of a first `reindex()` as
+`… failed reindexing: …`, since a reindex takes the stream's position first.
+
+**When:** `start()` or `reindex()`, against a standalone `mongod`.
+
+**Why:** the sync follows the collection with a change stream, and MongoDB
+serves change streams on a replica set or a sharded cluster only.
+
+**Fix:** run a replica set — a single node is enough, and is what this
+package's own specs use:
+
+```sh
+mongod --replSet rs0 --dbpath ./data   # then, once: rs.initiate()
+```
+
+### `Search sync "articles:articles" failed reindexing: not authorized on app to execute command { aggregate: "articles", pipeline: [ { $changeStream: {} } ], … }`
+
+The same shape appears for the state collection:
+`… not authorized on app to execute command { update: "nxgt_search_sync", … }`.
+
+**When:** `reindex()` or `start()`, against a server with authentication.
+
+**Why:** a sync needs more than `find`. The MongoDB user needs `find` **and**
+`changeStream` on the collection, and `find`, `insert`, `update` and `delete`
+on the state collection (`nxgt_search_sync` by default).
+
+**Fix:** grant them, and name the state collection if it lives elsewhere:
+
+```ts
+createSearchSync({ collection, index, transform, stateCollection: 'search_state' });
+```
+
+The Meilisearch key needs `documents.add`, `documents.get`, `documents.delete`
+and `tasks.get`, plus `indexes.create` unless the index already exists.
+
+### `Search sync "articles:articles" was last at a point the server's change history no longer reaches. Reindex it, or start it with onHistoryLost: 'reindex'.`
+
+Code `HISTORY_LOST`; the `cause` carries the server's code, 286 or 280.
+
+**When:** `start()`, on a sync that was stopped for longer than the server's
+oplog covers.
+
+**Why:** the resume point is a token into the oplog. Once the oplog has rolled
+past it, MongoDB cannot say what happened in between, so following from there
+would silently miss changes.
+
+**Fix:** let it reindex, which is the default:
+
+```ts
+createSearchSync({ collection, index, transform, onHistoryLost: 'reindex' });
+```
+
+Keep `'fail'` where a reindex is too expensive to run unattended — then
+`reindex()` it yourself when the alert comes in.
+
+### `Search sync "articles:articles" is already following changes in this process: close it before you start it twice.`
+
+Code `RUNNING`. `reindex()` on a sync that is following ends
+`… before you reindex.`
+
+**When:** a second `start()`, or a `reindex()`, on a sync object that is
+already following.
+
+**Why:** a reindex removes what the index holds and the collection no longer
+gives it — including the documents the running follower has just indexed, which
+it will never send again. The refusal covers one process only: there is **no
+lock**, so two processes following one name is yours to prevent.
+
+**Fix:**
+
+```ts
+const running = await articleSearch.start();
+// …
+await running.close();              // then reindex, or start again
+await articleSearch.reindex();
+```
+
+## The transform
+
+### `Search sync "articles:articles": transform gave "id" "other" for the document …, whose index id is …`
+
+The message ends: *A document under another id could never be taken out of the
+index again.* Code `ID_MISMATCH`.
+
+**When:** `reindex()`, or a change the follower handles.
+
+**Why:** the index document's primary key has to be the id this sync derives
+from the Mongo `_id` — `String(_id)` by default, or whatever `toIndexId`
+returns. Under any other id, a later delete would look for a document that is
+not there and leave the wrong one in the index for good.
+
+**Fix:** build the key from the document's own `_id`:
+
+```ts
+transform: (article) => ({ id: String(article._id), title: article.title });
+```
+
+With a non-string primary key, give `toIndexId` and use the same value:
+
+```ts
+createSearchSync({
+	collection,
+	index,                                  // primaryKey: 'n', a number
+	toIndexId: (id) => Number(String(id).slice(0, 8)),
+	transform: (article) => ({ n: Number(String(article._id).slice(0, 8)), title: article.title }),
+});
+```
+
+### `transform must return a document or null, not nope`
+
+Reaches the caller wrapped: `Search sync "articles:articles" failed
+reindexing: transform must return a document or null, not nope`.
+
+**When:** `reindex()`, or a change the follower handles.
+
+**Why:** the transform gave something that is not a plain object and is not
+`null` — a string, a number, an array, or an implicit `undefined` from a branch
+that returns nothing.
+
+**Fix:** return `null` for a document that should stay out of the index, and
+make every branch return:
+
+```ts
+transform: (article) => (article.draft ? null : { id: String(article._id), title: article.title });
+```
+
+`null` also **removes** a document that was in the index, which is how a soft
+delete or an unpublish reaches search.
+
+### `Search sync "articles:articles" failed following changes: boom`
+
+Code `FAILED`; the `cause` is the error your transform threw, and `boom` is its
+message.
+
+**When:** while following, as soon as a change reaches a transform that throws.
+`closed` rejects with it, `flush()` and `close()` reject with the same error,
+and nothing past that change is recorded.
+
+**Why:** the sync cannot skip a document it could not build: the next `start`
+resumes from the last recorded point, so the same change arrives again and
+stops it again.
+
+**Fix:** keep the transform total — catch what you can, and return `null` for a
+document you cannot index:
+
+```ts
+transform: (article) => {
+	try {
+		return { id: String(article._id), title: render(article.body) };
+	} catch {
+		return null;            // out of the index rather than stopping the sync
+	}
+};
+```
+
+And take the rejection, whatever you do with it: a rejected `closed` that
+nobody handles ends the process.
+
+```ts
+running.closed.catch((error) => log.error(error));
+```
+
+## Sending
+
+### `Search sync "articles:articles" failed sending changes: Task 3 (documentAdditionOrUpdate) on index "articles" failed: Document identifier … is invalid`
+
+Meilisearch's own text follows: *A document identifier can be of type integer
+or string, only composed of alphanumeric characters (a-z A-Z 0-9), hyphens (-)
+and underscores (\_), and can not be more than 511 bytes.* Code `FAILED`; the
+`cause` is `@nxgt/meilisearch`'s `SearchIndexError`, carrying the failed task.
+
+**When:** a batch is sent — on `reindex()`, on a `flush()`, or from the
+follower's own timer, in which case it surfaces on `closed`.
+
+**Why:** Meilisearch has its own rules for ids, and the whole batch fails if
+one document breaks them. An `ObjectId`'s 24-character hex string is fine; a
+slug, an email or anything built with a space is not.
+
+**Fix:** derive the index id from the `_id` and nothing else:
+
+```ts
+createSearchSync({
+	collection,
+	index,
+	toIndexId: (id) => String(id),      // the default
+	transform: (article) => ({ id: String(article._id), title: article.title }),
+});
+```
+
+A failed batch is not skipped: fix the transform and reindex, or the same
+change stops the sync again.
+
+## Stopping
+
+### The sync stops and `closed` resolves with `'invalidated'`
+
+Not an error: `closed` **resolves**, and the index is left as it was.
+
+**When:** the collection is dropped or renamed under a running sync.
+
+**Why:** MongoDB invalidates the change stream, and the point it stopped at
+belongs to a collection that no longer exists. The recorded point is forgotten,
+so the next `start()` reindexes from scratch rather than resuming into nothing.
+
+**Fix:** watch `closed` for the reason, not only for a rejection — a dropped
+collection is silent if you only handle failures:
+
+```ts
+const running = await articleSearch.start();
+running.closed.then(
+	(reason) => { if (reason === 'invalidated') log.warn('collection dropped'); },
+	(error) => log.error(error),
+);
+```
+
+Whatever the collection holds after it is recreated is indexed by the reindex
+the next `start()` runs.

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@nxgt/mongo-meilisearch",
-	"version": "0.1.6",
+	"version": "0.1.7",
 	"description": "Keeps a Meilisearch index in step with a MongoDB collection: a typed transform, a full reindex, and a change stream that resumes where it stopped",
 	"license": "MIT",
 	"type": "module",
@@ -8,6 +8,7 @@
 	"types": "./dist/index.d.ts",
 	"files": [
 		"dist",
+		"docs",
 		"README.md",
 		"package.json",
 		"LICENSE"
@@ -48,8 +49,8 @@
 		]
 	},
 	"devDependencies": {
-		"@nxgt/meilisearch": "^0.1.0",
-		"@nxgt/mongo": "^0.15.0",
+		"@nxgt/meilisearch": "^0.1.1",
+		"@nxgt/mongo": "^0.15.1",
 		"@types/bun": "^1.4.0",
 		"meilisearch": "0.62.0",
 		"mongodb": "7.6.0",
@@ -57,8 +58,8 @@
 		"zod": "4.6.5"
 	},
 	"peerDependencies": {
-		"@nxgt/meilisearch": "^0.1.0",
-		"@nxgt/mongo": "^0.15.0",
+		"@nxgt/meilisearch": "^0.1.1",
+		"@nxgt/mongo": "^0.15.1",
 		"meilisearch": ">=0.62.0 <1",
 		"mongodb": ">=7.0.0 <8",
 		"typescript": "^6.0.3"