@nxgt/mongo-meilisearch 0.2.0 → 0.3.0

package/docs/guide/reindex.md

@@ -30,22 +30,30 @@ const report = await articleSearch.reindex();
 
 ## What it does, in order
 
+`reindex()` holds the sync name's
+[lease](following-changes.md#one-process-per-name-the-lease) throughout; see
+[Not beside a follower in another process](#not-beside-a-follower-in-another-process).
+
 1. **Takes the collection's current position** — a change stream's first read
    answers with it, without waiting for a change.
 2. **Reads every live document**, `pageSize` at a time (100 by default,
    lowered to the collection's `maxPageSize` when it asks for more), runs the
    transform, and sends what it gives in batches of `batchSize` (500).
    Soft-deleted documents are not live, so they are never sent.
-3. **Pages the whole index** and deletes every document whose id the
-   collection did not just give it: deleted, turned away by the transform, or
-   never from this collection at all.
-4. **Records the position from step 1** as the resume point, and stamps
-   `reindexedAt`.
+   After each page, it stops if a renewal found the lease lost.
+3. **Asks the server whether the lease is still its own**, then **pages the
+   whole index** and deletes every document whose id the collection did not
+   just give it: deleted, turned away by the transform, or never from this
+   collection at all.
+4. **Asks the server again**, then **records the position from step 1** as
+   the resume point, and stamps `reindexedAt`.
 
 Step 1 before step 2 is what makes it safe to reindex a live collection: a
 change made while the documents are read is followed again from that
 position, so nothing falls between the reindex and the stream. A reindex that
-throws records nothing, and the next one starts over.
+throws records nothing, and the next one starts over. One that throws because
+its lease was lost has removed nothing either, unless the lease went while the
+removal itself ran.
 
 ```ts
 const before = await articleSearch.state(); // undefined, the first time
@@ -87,14 +95,46 @@ await articleSearch.reindex();
 const fresh = await articleSearch.start();
 ```
 
-The check is per object and per process: the package has no lock across
-processes, and [does not pretend to](boundaries.md).
+## Not beside a follower in another process
+
+`reindex()` takes the sync name's [lease](following-changes.md#one-process-per-name-the-lease)
+for as long as it runs, renews it every third of `leaseMs`, and lets go when
+it is done. While another process follows the name, or reindexes it, it is
+refused before it reads anything:
+
+```ts
+await articleSearch.reindex();
+// SearchSyncError: Search sync "articles:articles" is held by
+// worker-1:4127:66f0c2e5a1b2c3d4e5f60718 until 2026-09-22T09:14:07.512Z: wait
+// for it to close, or for its lease to lapse, before you reindex.
+```
+
+The code is `RUNNING` again. Close the follower wherever it runs — its
+`close()` lets go of the name before it resolves — then reindex. While the
+reindex runs, a `start()` elsewhere is refused the same way.
+
+A reindex whose lease another process took over while it ran — it was not
+renewed within `leaseMs`, or it was removed — rejects with `LEASE_LOST`: after
+the page it just sent, when a renewal found it lost, or when the server says so
+before documents are removed or before the resume point is recorded. It has
+then **recorded nothing**, and removed nothing unless the lease went while the
+removal itself ran; the pages already sent stay in the index. The same holds
+for the reindex `start()` runs, which also asks before it opens the follower.
+Run it again once the name is free.
+
+```ts
+await articleSearch.reindex();
+// SearchSyncError: Search sync "articles:articles" lost its lease: another
+// process holds the name now, or the lease was removed (it lapses when not
+// renewed within 30000 ms). It stopped rather than run beside it.
+```
 
 ## Errors
 
 Anything that goes wrong while reindexing comes back as a `SearchSyncError`
 with the code `FAILED` — MongoDB refused a read, Meilisearch refused a batch,
-the transform threw — and the original error as its `cause`:
+the transform threw — and the original error as its `cause`. `RUNNING` and
+`LEASE_LOST` are the lease's, above:
 
 ```ts
 try {
@@ -107,7 +147,7 @@ try {
 }
 ```
 
-Two codes mean something more precise than `FAILED`, and both are the
+Two more codes mean something more precise than `FAILED`, and both are the
 transform's doing. `ID_MISMATCH`: it gave a document whose primary key is not
 that document's index id. `NOT_A_DOCUMENT`: it gave back something that is
 neither a document nor `null`.

package/docs/guide/sync-lifecycle.md

@@ -105,11 +105,12 @@ message reports its shape rather than its value.
 | `index` | `TypedIndex<I>` | — | Where they go: from `bindIndex` |
 | `transform` | `Transform<C, I>` | — | The document as the index holds it, or `null` to keep it out |
 | `toIndexId` | `ToIndexId<C, I>` | `String` | The index id of a `_id`. Optional while the index's ids are strings, required otherwise |
-| `name` | `string` | `'<collection>:<index uid>'` | What the resume point is recorded under. Two syncs under one name share it |
-| `stateCollection` | `string` | `'nxgt_search_sync'` | Where the resume point is kept, in the collection's database |
+| `name` | `string` | `'<collection>:<index uid>'` | What the resume point and the lease are recorded under. Two syncs under one name share both |
+| `stateCollection` | `string` | `'nxgt_search_sync'` | Where the resume point and the lease are kept, in the collection's database |
 | `batchSize` | `number` | `500` | Changes, or documents, sent to Meilisearch at once |
 | `flushIntervalMs` | `number` | `1000` | How long a change waits for others; `0` sends at the next tick |
 | `positionIntervalMs` | `number` | `60000` | How often a sync with nothing to send records where the stream is |
+| `leaseMs` | `number` | `30000` | How long the lease on the name lasts unrenewed; a running sync renews it every third of that. See [the lease](following-changes.md#one-process-per-name-the-lease) |
 | `pageSize` | `number` | `100` | Documents a reindex reads per page; above the collection's `maxPageSize`, lowered to it |
 | `onHistoryLost` | `'reindex' \| 'fail'` | `'reindex'` | What `start` does when the resume point is older than the server's change history |
 
@@ -144,12 +145,14 @@ await sync.state();
 // { _id: 'articles:public', resumeToken: …, updatedAt: …, reindexedAt: … }
 ```
 
-`state()` is `undefined` until the first reindex records something.
+`state()` is `undefined` until the first reindex records something. The same
+collection holds the name's lease, under `_id: { lease: 'articles:public' }`,
+while a process follows or reindexes it.
 
 ### `batchSize`, `flushIntervalMs`, `pageSize`
 
-`batchSize`, `positionIntervalMs` and `pageSize` must be whole numbers above
-zero and `flushIntervalMs` a whole number of milliseconds, or
+`batchSize`, `positionIntervalMs`, `leaseMs` and `pageSize` must be whole
+numbers above zero and `flushIntervalMs` a whole number of milliseconds, or
 `createSearchSync` throws a `TypeError` before anything runs — as it does for
 an empty `name` or a `transform` that is not a function.
 

package/docs/roadmap.md

@@ -5,10 +5,7 @@ version an item shipped in is the only number on this page.
 
 ## Now
 
-- **A lease on a sync name, renewed while it runs** — a follower holds the
-  sync's name for a bounded time and renews it as it works, so a process that
-  dies is taken over once its lease expires rather than leaving the index
-  behind, and a name nothing is following no longer blocks the next start.
+_Nothing in progress._
 
 ## Next
 
@@ -35,6 +32,14 @@ _Nothing queued._
 
 ## Shipped
 
+- **A lease on a sync name, renewed while it runs** — `start()` and
+  `reindex()` take a lease on the sync's name, kept in the state collection
+  and timed by the server, renewed every third of the new `leaseMs` option
+  (default 30 s); a second process gets `RUNNING` naming the holder, a process
+  that dies is taken over once its lease lapses, and a sync whose lease was
+  taken stops with the new `LEASE_LOST` code; a reindex asks the server that
+  the lease is still its own before it removes documents or records where
+  following resumes — 0.3.0.
 - **A transform that gives back something that is not a document has its own
   code** — `SearchSyncErrorCode` gains `NOT_A_DOCUMENT`, naming the sync and
   the document and reporting the shape of what came back, never its value, so

package/docs/troubleshooting.md

@@ -6,7 +6,8 @@ name is written as it comes out by default — `<collection>:<index uid>`, here
 `articles:articles`.
 
 Everything this package throws is a `SearchSyncError` carrying a `code`
-(`HISTORY_LOST`, `ID_MISMATCH`, `NOT_A_DOCUMENT`, `RUNNING`, `FAILED`), the
+(`HISTORY_LOST`, `ID_MISMATCH`, `NOT_A_DOCUMENT`, `RUNNING`, `LEASE_LOST`,
+`FAILED`), the
 sync's `name`, and the original error as `cause` — except the options, which
 are refused with a `TypeError` before anything is opened.
 
@@ -14,10 +15,10 @@ are refused with a `TypeError` before anything is opened.
 | --- | --- |
 | [Install](#install) | [ERESOLVE](#npm-error-eresolve-unable-to-resolve-dependency-tree) · [incorrect peer dependency](#warn-incorrect-peer-dependency-nxgtmongo0140) · [TS2307](#error-ts2307-cannot-find-module-nxgtmongo-or-its-corresponding-type-declarations) |
 | [Options](#options) | [transform](#createsearchsync-transform-must-be-a-function) · [batchSize](#createsearchsync-batchsize-must-be-a-whole-number-above-0-not-0) · [flushIntervalMs](#createsearchsync-flushintervalms-must-be-a-whole-number-of-milliseconds-not--1) · [name](#createsearchsync-name-must-not-be-empty) |
-| [Starting](#starting) | [no replica set](#search-sync-articlesarticles-failed-starting-the-changestream-stage-is-only-supported-on-replica-sets) · [privileges](#search-sync-articlesarticles-failed-reindexing-not-authorized-on-app-to-execute-command--aggregate-articles-pipeline---changestream-----) · [history lost](#search-sync-articlesarticles-was-last-at-a-point-the-servers-change-history-no-longer-reaches-reindex-it-or-start-it-with-onhistorylost-reindex) · [already running](#search-sync-articlesarticles-is-already-following-changes-in-this-process-close-it-before-you-start-it-twice) |
+| [Starting](#starting) | [no replica set](#search-sync-articlesarticles-failed-starting-the-changestream-stage-is-only-supported-on-replica-sets) · [privileges](#search-sync-articlesarticles-failed-reindexing-not-authorized-on-app-to-execute-command--aggregate-articles-pipeline---changestream-----) · [history lost](#search-sync-articlesarticles-was-last-at-a-point-the-servers-change-history-no-longer-reaches-reindex-it-or-start-it-with-onhistorylost-reindex) · [already running](#search-sync-articlesarticles-is-already-following-changes-in-this-process-close-it-before-you-start-it-twice) · [held by another process](#search-sync-articlesarticles-is-held-by--until--wait-for-it-to-close-or-for-its-lease-to-lapse-before-you-start-it) · [the lease cannot be taken](#search-sync-articlesarticles-failed-taking-its-lease-) · [the lease cannot be checked](#search-sync-articlesarticles-failed-checking-its-lease-) · [the lease lost while reindexing](#search-sync-articlesarticles-lost-its-lease-another-process-holds-the-name-now-or-the-lease-was-removed-it-lapses-when-not-renewed-within-30000-ms-it-stopped-rather-than-run-beside-it) |
 | [The transform](#the-transform) | [an id that is not the index's](#search-sync-articlesarticles-transform-gave-id-other-for-the-document--whose-index-id-is-) · [not a document](#search-sync-articlesarticles-transform-gave-a-string-for-the-document-) · [it threw](#search-sync-articlesarticles-failed-following-changes-boom) |
 | [Sending](#sending) | [an id Meilisearch refuses](#search-sync-articlesarticles-failed-sending-changes-task-3-documentadditionorupdate-on-index-articles-failed-document-identifier--is-invalid) |
-| [Stopping](#stopping) | [a dropped collection](#the-sync-stops-and-closed-resolves-with-invalidated) |
+| [Stopping](#stopping) | [lease lost](#search-sync-articlesarticles-lost-its-lease-another-process-holds-the-name-now-or-the-lease-was-removed-it-lapses-when-not-renewed-within-30000-ms-it-stopped-rather-than-run-beside-it) · [a dropped collection](#the-sync-stops-and-closed-resolves-with-invalidated) |
 
 ## Install
 
@@ -111,11 +112,11 @@ createSearchSync({
 
 ### `createSearchSync: batchSize must be a whole number above 0, not 0`
 
-The same refusal covers `positionIntervalMs` and `pageSize`.
+The same refusal covers `positionIntervalMs`, `pageSize` and `leaseMs`.
 
 **When:** calling `createSearchSync`.
 
-**Why:** those three count documents or milliseconds, and `0`, a fraction,
+**Why:** those four count documents or milliseconds, and `0`, a fraction,
 `NaN` and a negative number have no meaning for any of them. A value read from
 the environment is a string until it is parsed, and `Number('')` is `0`.
 
@@ -130,7 +131,7 @@ createSearchSync({ collection, index, transform, batchSize: 500 }); // the defau
 **When:** calling `createSearchSync`.
 
 **Why:** `flushIntervalMs` is the one option that accepts `0` — send every
-change as it comes — so it is checked apart from the three above. A negative
+change as it comes — so it is checked apart from the four above. A negative
 number or a fraction is still refused.
 
 **Fix:**
@@ -178,8 +179,8 @@ mongod --replSet rs0 --dbpath ./data   # then, once: rs.initiate()
 
 ### `Search sync "articles:articles" failed reindexing: not authorized on app to execute command { aggregate: "articles", pipeline: [ { $changeStream: {} } ], … }`
 
-The same shape appears for the state collection:
-`… not authorized on app to execute command { update: "nxgt_search_sync", … }`.
+For the state collection, the refusal comes first, from the lease:
+[`… failed taking its lease: …`](#search-sync-articlesarticles-failed-taking-its-lease-).
 
 **When:** `reindex()` or `start()`, against a server with authentication.
 
@@ -226,8 +227,10 @@ already following.
 
 **Why:** a reindex removes what the index holds and the collection no longer
 gives it — including the documents the running follower has just indexed, which
-it will never send again. The refusal covers one process only: there is **no
-lock**, so two processes following one name is yours to prevent.
+it will never send again. This check is the sync object's own and comes first;
+another process, or a second `createSearchSync` with the same name, is refused
+by the lease on the name instead, with
+[`… is held by …`](#search-sync-articlesarticles-is-held-by--until--wait-for-it-to-close-or-for-its-lease-to-lapse-before-you-start-it).
 
 **Fix:**
 
@@ -238,6 +241,90 @@ await running.close();              // then reindex, or start again
 await articleSearch.reindex();
 ```
 
+### `Search sync "articles:articles" is held by … until …: wait for it to close, or for its lease to lapse, before you start it.`
+
+Code `RUNNING`. The holder is written `<host>:<pid>:<24 hex digits>`, and the
+date is ISO, in UTC. A `reindex()` ends `… before you reindex.`
+
+**When:** `start()` or `reindex()`, while another process — or another sync
+object in this one — follows or reindexes the same name. Also after a process
+that held it died without closing: its lease still runs until `leaseMs`
+(default `30000`) after its last renewal.
+
+**Why:** a running sync, and a reindex, hold a lease on the sync's name — one
+document in the state collection, renewed every third of `leaseMs`. Two
+followers on one name would each send and record beside the other, and a
+reindex would remove what the follower had just sent. Expiry is decided on the
+MongoDB server's clock, so hosts whose clocks disagree still agree on it.
+
+**Fix:** run one follower per name. A second replica can stand by, retrying
+`start()` until the name is free — on `RUNNING`, and on `LEASE_LOST`, which a
+`start()` whose first reindex lost the name to another process rejects with:
+
+```ts
+import { type RunningSearchSync, SearchSyncError } from '@nxgt/mongo-meilisearch';
+
+// The name is held elsewhere, or was taken over while this start reindexed.
+const heldElsewhere = (error: unknown) =>
+	error instanceof SearchSyncError &&
+	(error.code === 'RUNNING' || error.code === 'LEASE_LOST');
+
+async function follow(): Promise<RunningSearchSync> {
+	for (;;) {
+		try {
+			return await articleSearch.start();
+		} catch (error) {
+			if (!heldElsewhere(error)) throw error;
+			await new Promise((resolve) => setTimeout(resolve, 10_000));
+		}
+	}
+}
+```
+
+Close the sync on shutdown (`await running.close()` on `SIGTERM`): that lets go
+of the name at once, where a killed process leaves it held until its lease
+lapses. To see who holds it, read `{ _id: { lease: 'articles:articles' } }` in
+the state collection (`nxgt_search_sync` by default). Do not delete a live
+holder's lease by hand: its next renewal finds it gone and it stops with
+[`LEASE_LOST`](#search-sync-articlesarticles-lost-its-lease-another-process-holds-the-name-now-or-the-lease-was-removed-it-lapses-when-not-renewed-within-30000-ms-it-stopped-rather-than-run-beside-it).
+
+### `Search sync "articles:articles" failed taking its lease: …`
+
+Code `FAILED`; the `cause` is the driver's error, and its message follows the
+colon — for example
+`not authorized on app to execute command { findAndModify: "nxgt_search_sync", … }`.
+
+**When:** `start()` or `reindex()`, before anything else is read or sent.
+
+**Why:** the lease lives in the state collection, beside the resume point, and
+taking it is one atomic upsert. The MongoDB user needs `find`, `insert` and
+`update` on that collection to take it, and `delete` to let go of it — or the
+server is not reachable at all.
+
+**Fix:** grant those four on the state collection, and name it if it lives
+elsewhere:
+
+```ts
+createSearchSync({ collection, index, transform, stateCollection: 'search_state' });
+```
+
+### `Search sync "articles:articles" failed checking its lease: …`
+
+Code `FAILED`; the `cause` is the driver's error.
+
+**When:** a reindex — `reindex()`, or the one `start()` runs first — just
+before it removes what the index should no longer hold, just before it records
+its resume point, or `start()` just before it opens the follower.
+
+**Why:** those steps would undo what another holder did, so each asks the
+server first that the lease is still this sync's. A check that does not reach
+MongoDB is not tried again later, as a timed renewal is: the reindex stops
+there. The pages it already sent stay in the index; nothing was removed or
+recorded after the failed check.
+
+**Fix:** it is the server or the network, not the lease. Run the reindex again
+once MongoDB answers; the next one starts over.
+
 ## The transform
 
 ### `Search sync "articles:articles": transform gave "id" "other" for the document …, whose index id is …`
@@ -362,6 +449,55 @@ change stops the sync again.
 
 ## Stopping
 
+### `Search sync "articles:articles" lost its lease: another process holds the name now, or the lease was removed (it lapses when not renewed within 30000 ms). It stopped rather than run beside it.`
+
+Code `LEASE_LOST`; the number is the sync's `leaseMs`. `closed` rejects with
+it, and so do `reindex()` and `start()`.
+
+**When:** the lease on the name is no longer this process's — the process
+could not renew it for a whole `leaseMs` (an event loop blocked by synchronous
+work, a long GC pause, MongoDB unreachable for longer than that) and another
+process took the name in between, or the lease document was deleted by hand.
+It is found:
+
+- **while following** — at a renewal. `closed` rejects, and nothing more is
+  sent or recorded; a flush already in flight still finishes.
+- **while reindexing** — a standalone `reindex()`, or `start()` during its
+  first reindex or the one after a lost history. A renewal that found it lost
+  stops the reindex after the page it just sent; the server is also asked
+  before documents are removed, again before the resume point is recorded,
+  and, for `start()`, before the follower opens. The call rejects **having
+  recorded nothing**, and having removed nothing unless the lease went while
+  the removal itself ran. The pages already sent stay in the index.
+
+**Why:** another follower may be running on the same name. Carrying on would
+send, remove and record beside it, so the sync stops, and leaves the new
+holder's lease alone.
+
+**Fix:** give `leaseMs` room above the longest pause you expect, and start
+again when it happens — `start()` waits its turn with `RUNNING` while the new
+holder runs. A reindex that stopped this way is run again once the name is
+free:
+
+```ts
+import { createSearchSync, SearchSyncError } from '@nxgt/mongo-meilisearch';
+
+declare function follow(): Promise<void>; // the standby loop, below
+
+const search = createSearchSync({ collection, index, transform, leaseMs: 120_000 });
+const running = await search.start();
+
+running.closed.catch((error) => {
+	if (error instanceof SearchSyncError && error.code === 'LEASE_LOST') return follow();
+	log.error(error);
+});
+```
+
+`follow()` is the standby loop from
+[`… is held by …`](#search-sync-articlesarticles-is-held-by--until--wait-for-it-to-close-or-for-its-lease-to-lapse-before-you-start-it).
+A larger `leaseMs` also means a process that dies holds the name that much
+longer.
+
 ### The sync stops and `closed` resolves with `'invalidated'`
 
 Not an error: `closed` **resolves**, and the index is left as it was.

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@nxgt/mongo-meilisearch",
-	"version": "0.2.0",
+	"version": "0.3.0",
 	"description": "Keeps a Meilisearch index in step with a MongoDB collection: a typed transform, a full reindex, and a change stream that resumes where it stopped",
 	"license": "MIT",
 	"type": "module",
@@ -49,7 +49,7 @@
 		]
 	},
 	"devDependencies": {
-		"@nxgt/meilisearch": "^0.2.0",
+		"@nxgt/meilisearch": "^0.3.0",
 		"@nxgt/mongo": "^0.17.0",
 		"@types/bun": "^1.4.0",
 		"meilisearch": "0.62.0",
@@ -58,7 +58,7 @@
 		"zod": "4.6.5"
 	},
 	"peerDependencies": {
-		"@nxgt/meilisearch": "^0.2.0",
+		"@nxgt/meilisearch": "^0.3.0",
 		"@nxgt/mongo": "^0.17.0",
 		"meilisearch": ">=0.62.0 <1",
 		"mongodb": ">=7.0.0 <8",