@nxgt/mongo-meilisearch 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Steve Tsala
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md ADDED
@@ -0,0 +1,336 @@
1
+ # @nxgt/mongo-meilisearch
2
+
3
+ Keeps a [Meilisearch](https://www.meilisearch.com) index in step with a
4
+ MongoDB collection: a typed transform from one to the other, a full reindex,
5
+ and a change stream that picks up where it stopped, its position kept in
6
+ MongoDB.
7
+
8
+ ```ts
9
+ import { createSearchSync } from '@nxgt/mongo-meilisearch';
10
+
11
+ const articleSearch = createSearchSync({
12
+ collection: getCollection(db, articles), // @nxgt/mongo
13
+ index: bindIndex(meili, articleIndex), // @nxgt/meilisearch
14
+ transform: (article) =>
15
+ article.draft
16
+ ? null
17
+ : { id: String(article._id), title: article.title, body: article.body },
18
+ });
19
+
20
+ const running = await articleSearch.start(); // reindexes the first time, then follows
21
+ // …
22
+ await running.close();
23
+ ```
24
+
25
+ `transform` takes the collection's document, typed by its schema, and gives
26
+ the index's document, typed by its definition; `null` keeps a document out
27
+ of the index, and takes it out if it was in. Every create, update, soft
28
+ delete, restore and hard delete reaches the index, in batches.
29
+
30
+ > **0.x, on `@nxgt/mongo` and `@nxgt/meilisearch`.** The API is still settling.
31
+
32
+ ## Install
33
+
34
+ ```sh
35
+ bun add @nxgt/mongo-meilisearch @nxgt/mongo @nxgt/meilisearch mongodb meilisearch zod
36
+ ```
37
+
38
+ - `@nxgt/mongo` and `@nxgt/meilisearch`: required peers. The collection and
39
+ the index are theirs, and so are their options; this package creates
40
+ neither client.
41
+ - `mongodb` `>=7.0.0 <8` and `meilisearch` `>=0.62.0 <1`: required peers,
42
+ as the two packages above need them. `zod` is `@nxgt/mongo`'s.
43
+ - `typescript` 6: required peer, the version every `@nxgt` package pins.
44
+ - A MongoDB **replica set** or sharded cluster, since change streams need
45
+ one, and a Meilisearch server, 1.x. Tested against MongoDB 8.2 and
46
+ Meilisearch 1.53.
47
+
48
+ ## Setup
49
+
50
+ The collection and the index are defined as their packages define them:
51
+
52
+ ```ts
53
+ import { defineCollection, id } from '@nxgt/mongo';
54
+ import { defineIndex } from '@nxgt/meilisearch';
55
+ import { z } from 'zod';
56
+
57
+ export const articles = defineCollection({
58
+ name: 'articles',
59
+ schema: z.object({
60
+ _id: id(),
61
+ title: z.string(),
62
+ body: z.string(),
63
+ draft: z.boolean().default(false),
64
+ }),
65
+ timestamps: true,
66
+ softDelete: true,
67
+ });
68
+
69
+ export interface ArticleHit {
70
+ id: string;
71
+ title: string;
72
+ body: string;
73
+ }
74
+
75
+ export const articleIndex = defineIndex<ArticleHit>()({
76
+ uid: 'articles',
77
+ primaryKey: 'id',
78
+ settings: { searchableAttributes: ['title', 'body'] },
79
+ });
80
+ ```
81
+
82
+ Then, in the process that keeps the index up to date:
83
+
84
+ ```ts
85
+ import { bindIndex } from '@nxgt/meilisearch';
86
+ import { getCollection } from '@nxgt/mongo';
87
+ import { createSearchSync } from '@nxgt/mongo-meilisearch';
88
+
89
+ const index = bindIndex(meili, articleIndex);
90
+ await index.sync(); // the index's settings are @nxgt/meilisearch's to apply
91
+
92
+ const articleSearch = createSearchSync({
93
+ collection: getCollection(db, articles),
94
+ index,
95
+ transform: (article) =>
96
+ article.draft
97
+ ? null
98
+ : { id: String(article._id), title: article.title, body: article.body },
99
+ });
100
+
101
+ const running = await articleSearch.start();
102
+ process.on('SIGTERM', () => running.close());
103
+ await running.closed; // rejects with a SearchSyncError if the sync stops on an error
104
+ ```
105
+
106
+ `createSearchSync` sends nothing: `reindex` and `start` do.
107
+
108
+ ## Reindex
109
+
110
+ ```ts
111
+ const report = await articleSearch.reindex();
112
+ // { indexed: 1204, skipped: 17, removed: 3 }
113
+ ```
114
+
115
+ It reads every live document, a page at a time (`pageSize`, 100 by
116
+ default), sends what the transform gives in batches (`batchSize`, 500), and
117
+ then removes from the index every document the collection no longer gives
118
+ it: deleted, turned away by the transform, or never from this collection.
119
+
120
+ It first records where the collection's changes are, and saves that point
121
+ when it is done: a change made while it reads is followed again from there,
122
+ so none falls between the reindex and the stream. A reindex that fails
123
+ records nothing.
124
+
125
+ ## Following changes
126
+
127
+ ```ts
128
+ const running = await articleSearch.start();
129
+ await running.ready; // already resolved: start waits for the stream to open
130
+ await running.flush(); // sends what is waiting now, and records how far it got
131
+ await running.close(); // flushes, then stops; or `await using running = …`
132
+ ```
133
+
134
+ - **The first `start` reindexes.** With nothing recorded under the sync's
135
+ name, it reindexes, then follows from the point the reindex saved.
136
+ - **Later ones pick up where the last stopped.** A change made while no
137
+ process was following is applied on the next `start`, from the recorded
138
+ point.
139
+ - **Changes go in batches.** A change waits `flushIntervalMs` (1000 ms) for
140
+ others, or goes at once when `batchSize` of them are waiting. Several
141
+ changes to one document send only the last.
142
+ - **The point is recorded after the batch is applied**, in
143
+ `stateCollection` (`nxgt_search_sync`, in the collection's database),
144
+ under the sync's `name` (`'<collection>:<index uid>'`). What was not sent
145
+ when a process stops is sent again by the next `start`.
146
+ - **Every change runs the transform.** A document that stops qualifying is
147
+ taken out; a soft-deleted one is taken out, and a restored one comes back.
148
+ - **A quiet collection keeps its point fresh.** Every `positionIntervalMs`
149
+ (60 s), a sync with nothing to send records where the stream is anyway.
150
+ Without it, a collection nothing writes to for longer than the server's
151
+ history covers would need a full reindex at the next start. Nothing is
152
+ written while the stream itself does not move on.
153
+ - **A dropped collection ends the sync** — `closed` resolves
154
+ `'invalidated'` — and its recorded point goes with it, since nothing could
155
+ resume from inside a collection that no longer exists. The next `start`
156
+ reindexes what the recreated collection holds.
157
+
158
+ ### When the history is gone
159
+
160
+ MongoDB keeps a bounded history of changes, the oplog. A sync stopped for
161
+ longer than it covers cannot resume. `start` then reindexes and follows from
162
+ there, or, with `onHistoryLost: 'fail'`, throws a `SearchSyncError` with the
163
+ code `HISTORY_LOST`, for an app that would rather decide when a full
164
+ reindex runs.
165
+
166
+ ```ts
167
+ const articleSearch = createSearchSync({ …, onHistoryLost: 'fail' });
168
+
169
+ try {
170
+ running = await articleSearch.start();
171
+ } catch (error) {
172
+ if (error instanceof SearchSyncError && error.code === 'HISTORY_LOST') {
173
+ await articleSearch.reindex(); // when it suits you
174
+ running = await articleSearch.start();
175
+ } else throw error;
176
+ }
177
+ ```
178
+
179
+ ### Ids that are not strings
180
+
181
+ The index id of a document is `String(_id)` by default, which suits an
182
+ `ObjectId`. When the index's primary key is a number, say how to get it:
183
+
184
+ ```ts
185
+ createSearchSync({
186
+ collection: getCollection(db, counters), // _id: z.int()
187
+ index: bindIndex(meili, counterIndex), // primaryKey: 'n', a number
188
+ toIndexId: (id) => id,
189
+ transform: (counter) => ({ n: counter._id, label: counter.label }),
190
+ });
191
+ ```
192
+
193
+ The transform must give each document its index id as primary key: a
194
+ document under another id could never be taken out again, and throws
195
+ `ID_MISMATCH`.
196
+
197
+ ## Errors
198
+
199
+ This package throws `SearchSyncError`; what caused it is its `cause`.
200
+
201
+ | `code` | When |
202
+ | --- | --- |
203
+ | `HISTORY_LOST` | `start` with `onHistoryLost: 'fail'`, and the recorded point is older than the server's history. `cause` is `@nxgt/mongo`'s `DataError`, `serverCode` 286 or 280 |
204
+ | `ID_MISMATCH` | the transform gave a document whose primary key is not its index id |
205
+ | `RUNNING` | `reindex()` or a second `start()` while this sync is already following changes in this process. A reindex beside its own follower would remove what the follower has just indexed |
206
+ | `FAILED` | anything else: the transform threw, MongoDB or Meilisearch refused. The message says what the sync was doing |
207
+
208
+ A running sync that meets one stops: `closed` rejects with it, and `flush`
209
+ and `close` reject with it too. Nothing past the last applied batch is
210
+ recorded, so the next `start` sends it again.
211
+
212
+ ```ts
213
+ running.closed.catch((error: SearchSyncError) => {
214
+ log.error({ sync: error.sync, code: error.code, cause: error.cause });
215
+ process.exit(1); // let the supervisor restart it
216
+ });
217
+ ```
218
+
219
+ `onHistoryLost` applies to `start` alone. A history that runs out under a
220
+ running sync stops it with `FAILED`; the next `start` is where the choice is
221
+ made again.
222
+
223
+ `createSearchSync` throws a `TypeError` before anything runs: for an option
224
+ out of range, an empty `name`, or a `transform` that is not a function.
225
+
226
+ ## Not included
227
+
228
+ - **Several processes sharing one sync.** There is no lock: run one
229
+ follower per sync name. Two do the same writes twice, and a `reindex` in
230
+ one while the other follows removes documents the follower has already
231
+ indexed and will not send again.
232
+ - **Partial updates.** A change sends the whole document the transform
233
+ gives, never a patch.
234
+ - **Keeping the index's settings.** That is `@nxgt/meilisearch`'s `sync`.
235
+ - **Joining other collections into a document.** The transform may read
236
+ them, but a change to them does not reach the index.
237
+
238
+ ## API
239
+
240
+ ### `createSearchSync(options)`
241
+
242
+ ```ts
243
+ function createSearchSync<C extends AnyCollectionDefinition, I extends AnyIndexDefinition>(
244
+ options: SearchSyncOptions<C, I>,
245
+ ): SearchSync;
246
+ ```
247
+
248
+ `SearchSyncOptions<C, I>`:
249
+
250
+ | Option | Default | |
251
+ | --- | --- | --- |
252
+ | `collection: TypedCollection<C>` | | from `getCollection` |
253
+ | `index: TypedIndex<I>` | | from `bindIndex` |
254
+ | `transform: Transform<C, I>` | | `(document: ReadDocumentOf<C>) => DocumentOf<I> \| null`, or a promise of it. `ReadDocumentOf` is `@nxgt/mongo`'s, `DocumentOf` `@nxgt/meilisearch`'s |
255
+ | `toIndexId: ToIndexId<C, I>` | `String` | `(id: IdOf<C>) => IdOf<I>`, the first from `@nxgt/mongo` and the second from `@nxgt/meilisearch`; required when the index's ids are not strings |
256
+ | `name: string` | `'<collection>:<index uid>'` | what the state is recorded under |
257
+ | `stateCollection: string` | `'nxgt_search_sync'` | in the collection's database |
258
+ | `batchSize: number` | `500` | changes sent at once |
259
+ | `flushIntervalMs: number` | `1000` | how long a change waits for others; `0` sends at the next tick |
260
+ | `positionIntervalMs: number` | `60000` | how often a sync with nothing to send records where the stream is |
261
+ | `pageSize: number` | `100` | documents a reindex reads per page; above the collection's `maxPageSize`, lowered to it |
262
+ | `onHistoryLost: 'reindex' \| 'fail'` | `'reindex'` | |
263
+
264
+ `SearchSync`:
265
+
266
+ | Member | |
267
+ | --- | --- |
268
+ | `name: string` | |
269
+ | `reindex(): Promise<ReindexReport>` | `{ indexed, skipped, removed }` |
270
+ | `start(): Promise<RunningSearchSync>` | resolves once changes are heard |
271
+ | `state(): Promise<SearchSyncState \| undefined>` | `{ _id, resumeToken, updatedAt, reindexedAt }`; `undefined` before the first reindex |
272
+
273
+ `RunningSearchSync`, also `AsyncDisposable`:
274
+
275
+ | Member | |
276
+ | --- | --- |
277
+ | `ready: Promise<void>` | resolved |
278
+ | `closed: Promise<CloseReason>` | `'closed'` after `close`, `'invalidated'` when the collection was dropped or renamed; rejects with a `SearchSyncError`. `CloseReason` is `@nxgt/mongo`'s, and its third value, `'failed'`, never occurs here |
279
+ | `flush(): Promise<void>` | sends what waits, records the point |
280
+ | `close(): Promise<void>` | flushes, then stops |
281
+
282
+ `class SearchSyncError extends Error`: `code: SearchSyncErrorCode`
283
+ (`'HISTORY_LOST' | 'ID_MISMATCH' | 'RUNNING' | 'FAILED'`), `sync: string`,
284
+ `cause`. Its constructor takes `(message, options: SearchSyncErrorOptions)`,
285
+ that is `{ code, sync, cause? }`; both types are exported.
286
+
287
+ ## What does not compile
288
+
289
+ Each is a `@ts-expect-error` case in this package's type tests.
290
+
291
+ - A transform that reads a field the collection's schema does not have.
292
+ - A transform that gives a field the index's document does not have, or an
293
+ id of the wrong type (an `ObjectId` for a string id).
294
+ - A transform that gives something other than a document or `null`.
295
+ - No `toIndexId` when the index's ids are not strings; one that gives
296
+ another type than the index's ids, or takes another than the collection's.
297
+ - A driver `Collection` for `collection`, or the SDK's `Index` for `index`.
298
+ - No `transform`, an option this package does not have, or an
299
+ `onHistoryLost` other than `'reindex'` or `'fail'`.
300
+
301
+ ## Traps
302
+
303
+ - **The sync owns its index.** A reindex removes every document the
304
+ collection does not give it, whoever wrote it. Do not point two
305
+ collections, or another writer, at one index.
306
+ - **A change may be applied twice.** Changes after the last recorded point
307
+ are sent again after a restart or a failure. Keep the transform a function
308
+ of the document alone.
309
+ - **A transform that throws stops the sync**, and the same document stops
310
+ it again on the next `start`. Catch what you can in the transform, and
311
+ return `null` for a document you cannot index.
312
+ - **Await `closed`, or catch it.** A sync that stops on an error rejects
313
+ it, and a rejection nobody handles ends the process.
314
+ - **Without post-images, a change carries the document as it is now**, not
315
+ as the change left it (`@nxgt/mongo`'s change streams). For an index, where
316
+ only the latest state counts, that is what you want.
317
+ - **Only one process per sync name.** There is no lock; see *Not included*.
318
+ Inside one process this package refuses it: `reindex()` and a second
319
+ `start()` throw `RUNNING` while a sync of the same object is following.
320
+ - **A dropped collection stops the sync** (`'invalidated'`) and leaves the
321
+ index as it was. What the collection had is removed by the reindex the
322
+ next `start` runs.
323
+ - **Meilisearch has its own rules for ids**: letters, digits, `-` and `_`.
324
+ An id it refuses fails the batch, and the sync. An `ObjectId`'s hex string
325
+ is fine.
326
+ - **A write waits up to two minutes** for Meilisearch to apply it; a batch
327
+ on a large index can be slow, and the sync waits.
328
+ - **The user needs** `find` and `changeStream` on the collection, and
329
+ `find`, `insert`, `update` and `delete` on `stateCollection`; the
330
+ Meilisearch key needs `documents.add`, `documents.get`,
331
+ `documents.delete` and `tasks.get`, plus `indexes.create` unless the index
332
+ already exists — the first write to an index that does not creates it.
333
+
334
+ ## License
335
+
336
+ MIT
@@ -0,0 +1,12 @@
1
+ import type { Doc, SyncContext } from './context';
2
+ import type { Entry } from './documents';
3
+ /** Adds documents, `batchSize` at a time, each batch applied before the next. */
4
+ export declare function sendDocuments(ctx: SyncContext, documents: readonly Doc[]): Promise<void>;
5
+ /** Deletes documents by id, `batchSize` at a time. */
6
+ export declare function deleteIds(ctx: SyncContext, ids: readonly unknown[]): Promise<void>;
7
+ /**
8
+ * Applies entries: one per id, so adds and deletes touch different
9
+ * documents and their order does not matter.
10
+ */
11
+ export declare function send(ctx: SyncContext, entries: readonly Entry[]): Promise<void>;
12
+ //# sourceMappingURL=batch.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"batch.d.ts","sourceRoot":"","sources":["../src/batch.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,GAAG,EAAE,WAAW,EAAE,MAAM,WAAW,CAAC;AAClD,OAAO,KAAK,EAAE,KAAK,EAAE,MAAM,aAAa,CAAC;AAgBzC,iFAAiF;AACjF,wBAAsB,aAAa,CAClC,GAAG,EAAE,WAAW,EAChB,SAAS,EAAE,SAAS,GAAG,EAAE,GACvB,OAAO,CAAC,IAAI,CAAC,CAIf;AAED,sDAAsD;AACtD,wBAAsB,SAAS,CAC9B,GAAG,EAAE,WAAW,EAChB,GAAG,EAAE,SAAS,OAAO,EAAE,GACrB,OAAO,CAAC,IAAI,CAAC,CAIf;AAED;;;GAGG;AACH,wBAAsB,IAAI,CACzB,GAAG,EAAE,WAAW,EAChB,OAAO,EAAE,SAAS,KAAK,EAAE,GACvB,OAAO,CAAC,IAAI,CAAC,CASf"}
@@ -0,0 +1,34 @@
1
+ import type { AnyIndexDefinition, TypedIndex } from '@nxgt/meilisearch';
2
+ import type { AnyCollectionDefinition, TypedCollection } from '@nxgt/mongo';
3
+ import type { Collection } from 'mongodb';
4
+ import type { SearchSyncOptions, SearchSyncState } from './types';
5
+ /** Loosely typed: this layer works on any documents; the public types are what callers see. */
6
+ export type Doc = Record<string, unknown>;
7
+ /**
8
+ * What a sync works from, resolved once. Data, and the caller's own two
9
+ * functions as they were given.
10
+ */
11
+ export interface SyncContext {
12
+ readonly name: string;
13
+ readonly collection: TypedCollection<AnyCollectionDefinition>;
14
+ readonly index: TypedIndex<AnyIndexDefinition>;
15
+ readonly primaryKey: string;
16
+ readonly state: Collection<SearchSyncState>;
17
+ readonly batchSize: number;
18
+ readonly flushIntervalMs: number;
19
+ readonly positionIntervalMs: number;
20
+ readonly pageSize: number;
21
+ readonly onHistoryLost: 'reindex' | 'fail';
22
+ readonly transform: (document: Doc) => unknown;
23
+ readonly toIndexId: (id: unknown) => unknown;
24
+ /**
25
+ * Whether this sync is following changes in this process. A reindex
26
+ * beside its own follower would remove what the follower has just
27
+ * indexed, and the follower would never send it again.
28
+ */
29
+ readonly follower: {
30
+ running: boolean;
31
+ };
32
+ }
33
+ export declare function createContext(options: SearchSyncOptions<AnyCollectionDefinition, AnyIndexDefinition>): SyncContext;
34
+ //# sourceMappingURL=context.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"context.d.ts","sourceRoot":"","sources":["../src/context.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,kBAAkB,EAAE,UAAU,EAAE,MAAM,mBAAmB,CAAC;AACxE,OAAO,KAAK,EAAE,uBAAuB,EAAE,eAAe,EAAE,MAAM,aAAa,CAAC;AAC5E,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,SAAS,CAAC;AAC1C,OAAO,KAAK,EAAE,iBAAiB,EAAE,eAAe,EAAE,MAAM,SAAS,CAAC;AAElE,+FAA+F;AAC/F,MAAM,MAAM,GAAG,GAAG,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;AAE1C;;;GAGG;AACH,MAAM,WAAW,WAAW;IAC3B,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,UAAU,EAAE,eAAe,CAAC,uBAAuB,CAAC,CAAC;IAC9D,QAAQ,CAAC,KAAK,EAAE,UAAU,CAAC,kBAAkB,CAAC,CAAC;IAC/C,QAAQ,CAAC,UAAU,EAAE,MAAM,CAAC;IAC5B,QAAQ,CAAC,KAAK,EAAE,UAAU,CAAC,eAAe,CAAC,CAAC;IAC5C,QAAQ,CAAC,SAAS,EAAE,MAAM,CAAC;IAC3B,QAAQ,CAAC,eAAe,EAAE,MAAM,CAAC;IACjC,QAAQ,CAAC,kBAAkB,EAAE,MAAM,CAAC;IACpC,QAAQ,CAAC,QAAQ,EAAE,MAAM,CAAC;IAC1B,QAAQ,CAAC,aAAa,EAAE,SAAS,GAAG,MAAM,CAAC;IAC3C,QAAQ,CAAC,SAAS,EAAE,CAAC,QAAQ,EAAE,GAAG,KAAK,OAAO,CAAC;IAC/C,QAAQ,CAAC,SAAS,EAAE,CAAC,EAAE,EAAE,OAAO,KAAK,OAAO,CAAC;IAC7C;;;;OAIG;IACH,QAAQ,CAAC,QAAQ,EAAE;QAAE,OAAO,EAAE,OAAO,CAAA;KAAE,CAAC;CACxC;AAcD,wBAAgB,aAAa,CAC5B,OAAO,EAAE,iBAAiB,CAAC,uBAAuB,EAAE,kBAAkB,CAAC,GACrE,WAAW,CAoCb"}
@@ -0,0 +1,21 @@
1
+ import type { AnyIndexDefinition } from '@nxgt/meilisearch';
2
+ import type { AnyCollectionDefinition } from '@nxgt/mongo';
3
+ import type { SearchSync, SearchSyncOptions } from './types';
4
+ /**
5
+ * Keeps an index in step with a collection.
6
+ *
7
+ * ```ts
8
+ * const usersSearch = createSearchSync({
9
+ * collection: getCollection(db, users),
10
+ * index: bindIndex(meili, userIndex),
11
+ * transform: (user) => (user.active ? { id: user.id, name: user.name } : null),
12
+ * });
13
+ * const running = await usersSearch.start(); // reindexes the first time
14
+ * // …
15
+ * await running.close();
16
+ * ```
17
+ *
18
+ * Nothing is sent before `reindex` or `start`.
19
+ */
20
+ export declare function createSearchSync<C extends AnyCollectionDefinition, I extends AnyIndexDefinition>(options: SearchSyncOptions<C, I>): SearchSync;
21
+ //# sourceMappingURL=create-search-sync.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"create-search-sync.d.ts","sourceRoot":"","sources":["../src/create-search-sync.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,kBAAkB,EAAE,MAAM,mBAAmB,CAAC;AAC5D,OAAO,KAAK,EAAE,uBAAuB,EAAE,MAAM,aAAa,CAAC;AAK3D,OAAO,KAAK,EAAE,UAAU,EAAE,iBAAiB,EAAE,MAAM,SAAS,CAAC;AAE7D;;;;;;;;;;;;;;;GAeG;AACH,wBAAgB,gBAAgB,CAC/B,CAAC,SAAS,uBAAuB,EACjC,CAAC,SAAS,kBAAkB,EAC3B,OAAO,EAAE,iBAAiB,CAAC,CAAC,EAAE,CAAC,CAAC,GAAG,UAAU,CAgB9C"}
@@ -0,0 +1,19 @@
1
+ import type { Doc, SyncContext } from './context';
2
+ /** One document's fate in the index: sent, or taken out. */
3
+ export type Entry = {
4
+ kind: 'upsert';
5
+ key: string;
6
+ document: Doc;
7
+ } | {
8
+ kind: 'delete';
9
+ key: string;
10
+ id: unknown;
11
+ };
12
+ /** An id as a key: `1` and `'1'` are different ids to Meilisearch. */
13
+ export declare const keyOf: (id: unknown) => string;
14
+ /**
15
+ * What the index should hold for a document of the collection: what the
16
+ * transform makes of it, or nothing when it is gone or kept out.
17
+ */
18
+ export declare function entryOf(ctx: SyncContext, mongoId: unknown, document: Doc | undefined): Promise<Entry>;
19
+ //# sourceMappingURL=documents.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"documents.d.ts","sourceRoot":"","sources":["../src/documents.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,GAAG,EAAE,WAAW,EAAE,MAAM,WAAW,CAAC;AAGlD,4DAA4D;AAC5D,MAAM,MAAM,KAAK,GACd;IAAE,IAAI,EAAE,QAAQ,CAAC;IAAC,GAAG,EAAE,MAAM,CAAC;IAAC,QAAQ,EAAE,GAAG,CAAA;CAAE,GAC9C;IAAE,IAAI,EAAE,QAAQ,CAAC;IAAC,GAAG,EAAE,MAAM,CAAC;IAAC,EAAE,EAAE,OAAO,CAAA;CAAE,CAAC;AAEhD,sEAAsE;AACtE,eAAO,MAAM,KAAK,GAAI,IAAI,OAAO,WAAiC,CAAC;AAKnE;;;GAGG;AACH,wBAAsB,OAAO,CAC5B,GAAG,EAAE,WAAW,EAChB,OAAO,EAAE,OAAO,EAChB,QAAQ,EAAE,GAAG,GAAG,SAAS,GACvB,OAAO,CAAC,KAAK,CAAC,CAsBhB"}
@@ -0,0 +1,26 @@
1
+ /** What stopped a sync, as a string to switch on. */
2
+ export type SearchSyncErrorCode =
3
+ /** The resume point is older than the server's change history. */
4
+ 'HISTORY_LOST'
5
+ /** The transform gave a document whose primary key is not its index id. */
6
+ | 'ID_MISMATCH'
7
+ /** This sync is already following changes in this process. */
8
+ | 'RUNNING'
9
+ /** Anything else: the cause says what. */
10
+ | 'FAILED';
11
+ export interface SearchSyncErrorOptions {
12
+ code: SearchSyncErrorCode;
13
+ sync: string;
14
+ cause?: unknown;
15
+ }
16
+ /** An error of a sync, with the sync's name and what caused it. */
17
+ export declare class SearchSyncError extends Error {
18
+ name: string;
19
+ readonly code: SearchSyncErrorCode;
20
+ /** The sync's name. */
21
+ readonly sync: string;
22
+ constructor(message: string, options: SearchSyncErrorOptions);
23
+ }
24
+ /** `error` as a `SearchSyncError`, kept as it is when it already is one. */
25
+ export declare function failed(sync: string, doing: string, error: unknown): SearchSyncError;
26
+ //# sourceMappingURL=errors.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"errors.d.ts","sourceRoot":"","sources":["../src/errors.ts"],"names":[],"mappings":"AAAA,qDAAqD;AACrD,MAAM,MAAM,mBAAmB;AAC9B,kEAAkE;AAChE,cAAc;AAChB,2EAA2E;GACzE,aAAa;AACf,8DAA8D;GAC5D,SAAS;AACX,0CAA0C;GACxC,QAAQ,CAAC;AAEZ,MAAM,WAAW,sBAAsB;IACtC,IAAI,EAAE,mBAAmB,CAAC;IAC1B,IAAI,EAAE,MAAM,CAAC;IACb,KAAK,CAAC,EAAE,OAAO,CAAC;CAChB;AAED,mEAAmE;AACnE,qBAAa,eAAgB,SAAQ,KAAK;IAChC,IAAI,SAAqB;IAClC,QAAQ,CAAC,IAAI,EAAE,mBAAmB,CAAC;IACnC,uBAAuB;IACvB,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;gBAEV,OAAO,EAAE,MAAM,EAAE,OAAO,EAAE,sBAAsB;CAQ5D;AAED,4EAA4E;AAC5E,wBAAgB,MAAM,CAAC,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,KAAK,EAAE,OAAO,mBAWjE"}
@@ -0,0 +1,9 @@
1
+ import type { SyncContext } from './context';
2
+ import type { RunningSearchSync } from './types';
3
+ /**
4
+ * Follows the collection from its recorded point — after a reindex when
5
+ * nothing is recorded, or when the point is older than the server's history
6
+ * and `onHistoryLost` allows it.
7
+ */
8
+ export declare function start(ctx: SyncContext): Promise<RunningSearchSync>;
9
+ //# sourceMappingURL=follow.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"follow.d.ts","sourceRoot":"","sources":["../src/follow.ts"],"names":[],"mappings":"AAOA,OAAO,KAAK,EAAO,WAAW,EAAE,MAAM,WAAW,CAAC;AAMlD,OAAO,KAAK,EAAE,iBAAiB,EAAE,MAAM,SAAS,CAAC;AAyJjD;;;;GAIG;AACH,wBAAsB,KAAK,CAAC,GAAG,EAAE,WAAW,GAAG,OAAO,CAAC,iBAAiB,CAAC,CA2BxE"}
@@ -0,0 +1,4 @@
1
+ export { createSearchSync } from './create-search-sync';
2
+ export { SearchSyncError, type SearchSyncErrorCode, type SearchSyncErrorOptions, } from './errors';
3
+ export type { ReindexReport, RunningSearchSync, SearchSync, SearchSyncOptions, SearchSyncState, ToIndexId, Transform, } from './types';
4
+ //# sourceMappingURL=index.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,gBAAgB,EAAE,MAAM,sBAAsB,CAAC;AACxD,OAAO,EACN,eAAe,EACf,KAAK,mBAAmB,EACxB,KAAK,sBAAsB,GAC3B,MAAM,UAAU,CAAC;AAClB,YAAY,EACX,aAAa,EACb,iBAAiB,EACjB,UAAU,EACV,iBAAiB,EACjB,eAAe,EACf,SAAS,EACT,SAAS,GACT,MAAM,SAAS,CAAC"}
package/dist/index.js ADDED
@@ -0,0 +1,383 @@
1
+ // src/context.ts
2
+ var DEFAULT_STATE_COLLECTION = "nxgt_search_sync";
3
+ function positive(option, value, fallback) {
4
+ const resolved = value ?? fallback;
5
+ if (!Number.isInteger(resolved) || resolved < 1) {
6
+ throw new TypeError(`createSearchSync: ${option} must be a whole number above 0, not ${String(value)}`);
7
+ }
8
+ return resolved;
9
+ }
10
+ function createContext(options) {
11
+ const { collection, index } = options;
12
+ if (typeof options.transform !== "function") {
13
+ throw new TypeError("createSearchSync: transform must be a function");
14
+ }
15
+ const name = options.name ?? `${collection.collectionName}:${index.uid}`;
16
+ if (name === "") {
17
+ throw new TypeError("createSearchSync: name must not be empty");
18
+ }
19
+ const flushIntervalMs = options.flushIntervalMs ?? 1000;
20
+ if (!Number.isInteger(flushIntervalMs) || flushIntervalMs < 0) {
21
+ throw new TypeError(`createSearchSync: flushIntervalMs must be a whole number of milliseconds, not ${String(options.flushIntervalMs)}`);
22
+ }
23
+ return {
24
+ name,
25
+ collection,
26
+ index,
27
+ primaryKey: index.definition.primaryKey,
28
+ state: collection.db.collection(options.stateCollection ?? DEFAULT_STATE_COLLECTION),
29
+ batchSize: positive("batchSize", options.batchSize, 500),
30
+ positionIntervalMs: positive("positionIntervalMs", options.positionIntervalMs, 60000),
31
+ flushIntervalMs,
32
+ pageSize: positive("pageSize", options.pageSize, 100),
33
+ onHistoryLost: options.onHistoryLost ?? "reindex",
34
+ transform: options.transform,
35
+ toIndexId: options.toIndexId ?? String,
36
+ follower: { running: false }
37
+ };
38
+ }
39
+
40
+ // src/batch.ts
41
+ var WAIT = { wait: { timeout: 120000 } };
42
+ function chunks(items, size) {
43
+ const out = [];
44
+ for (let i = 0;i < items.length; i += size) {
45
+ out.push(items.slice(i, i + size));
46
+ }
47
+ return out;
48
+ }
49
+ async function sendDocuments(ctx, documents) {
50
+ for (const batch of chunks(documents, ctx.batchSize)) {
51
+ await ctx.index.add(batch, WAIT);
52
+ }
53
+ }
54
+ async function deleteIds(ctx, ids) {
55
+ for (const batch of chunks(ids, ctx.batchSize)) {
56
+ await ctx.index.delete(batch, WAIT);
57
+ }
58
+ }
59
+ async function send(ctx, entries) {
60
+ const documents = [];
61
+ const ids = [];
62
+ for (const entry of entries) {
63
+ if (entry.kind === "upsert")
64
+ documents.push(entry.document);
65
+ else
66
+ ids.push(entry.id);
67
+ }
68
+ await sendDocuments(ctx, documents);
69
+ await deleteIds(ctx, ids);
70
+ }
71
+
72
+ // src/errors.ts
73
+ class SearchSyncError extends Error {
74
+ constructor(message, options) {
75
+ super(message, options.cause === undefined ? undefined : { cause: options.cause });
76
+ this.name = "SearchSyncError";
77
+ this.code = options.code;
78
+ this.sync = options.sync;
79
+ }
80
+ }
81
+ function failed(sync, doing, error) {
82
+ if (error instanceof SearchSyncError)
83
+ return error;
84
+ const reason = error instanceof Error ? error.message : String(error);
85
+ return new SearchSyncError(`Search sync "${sync}" failed ${doing}: ${reason}`, {
86
+ code: "FAILED",
87
+ sync,
88
+ cause: error
89
+ });
90
+ }
91
+
92
+ // src/documents.ts
93
+ var keyOf = (id) => `${typeof id}:${String(id)}`;
94
+ var isRecord = (value) => typeof value === "object" && value !== null && !Array.isArray(value);
95
+ async function entryOf(ctx, mongoId, document) {
96
+ const id = ctx.toIndexId(mongoId);
97
+ const key = keyOf(id);
98
+ if (!document)
99
+ return { kind: "delete", key, id };
100
+ const indexed = await ctx.transform(document);
101
+ if (indexed === null)
102
+ return { kind: "delete", key, id };
103
+ if (!isRecord(indexed)) {
104
+ throw new TypeError(`transform must return a document or null, not ${String(indexed)}`);
105
+ }
106
+ const given = indexed[ctx.primaryKey];
107
+ if (keyOf(given) !== key) {
108
+ throw new SearchSyncError(`Search sync "${ctx.name}": transform gave "${ctx.primaryKey}" ` + `${JSON.stringify(given)} for the document ${String(mongoId)}, whose ` + `index id is ${JSON.stringify(id)}. A document under another id ` + "could never be taken out of the index again.", { code: "ID_MISMATCH", sync: ctx.name });
109
+ }
110
+ return { kind: "upsert", key, document: indexed };
111
+ }
112
+
113
+ // src/reindex.ts
114
+ import { MeilisearchApiError } from "meilisearch";
115
+
116
+ // src/running.ts
117
+ function checkIdle(ctx, doing) {
118
+ if (!ctx.follower.running)
119
+ return;
120
+ throw new SearchSyncError(`Search sync "${ctx.name}" is already following changes in this ` + `process: close it before you ${doing}.`, { code: "RUNNING", sync: ctx.name });
121
+ }
122
+
123
+ // src/state.ts
124
+ async function readState(ctx) {
125
+ try {
126
+ return await ctx.state.findOne({ _id: ctx.name }) ?? undefined;
127
+ } catch (error) {
128
+ throw failed(ctx.name, "reading its resume point", error);
129
+ }
130
+ }
131
+ async function clearState(ctx) {
132
+ try {
133
+ await ctx.state.deleteOne({ _id: ctx.name });
134
+ } catch (error) {
135
+ throw failed(ctx.name, "forgetting its resume point", error);
136
+ }
137
+ }
138
+ async function saveState(ctx, resumeToken, reindexed) {
139
+ const now = new Date;
140
+ try {
141
+ await ctx.state.updateOne({ _id: ctx.name }, {
142
+ $set: {
143
+ resumeToken,
144
+ updatedAt: now,
145
+ ...reindexed ? { reindexedAt: now } : {}
146
+ }
147
+ }, { upsert: true });
148
+ } catch (error) {
149
+ throw failed(ctx.name, "recording its resume point", error);
150
+ }
151
+ }
152
+
153
+ // src/reindex.ts
154
+ async function currentToken(ctx) {
155
+ const stream = ctx.collection.raw.watch([], { maxAwaitTimeMS: 1 });
156
+ try {
157
+ await stream.tryNext();
158
+ return stream.resumeToken;
159
+ } finally {
160
+ await stream.close();
161
+ }
162
+ }
163
+ async function sendAll(ctx, wanted) {
164
+ let indexed = 0;
165
+ let skipped = 0;
166
+ let after;
167
+ do {
168
+ const page = await ctx.collection.paginateByCursor({
169
+ after,
170
+ limit: ctx.pageSize
171
+ });
172
+ const documents = [];
173
+ for (const document of page.items) {
174
+ const entry = await entryOf(ctx, document._id, document);
175
+ if (entry.kind === "delete") {
176
+ skipped += 1;
177
+ continue;
178
+ }
179
+ wanted.add(entry.key);
180
+ documents.push(entry.document);
181
+ }
182
+ await sendDocuments(ctx, documents);
183
+ indexed += documents.length;
184
+ after = page.nextCursor;
185
+ } while (after);
186
+ return { indexed, skipped };
187
+ }
188
+ var LIST_LIMIT = 1000;
189
+ var missingIndex = (error) => error instanceof MeilisearchApiError && error.cause?.code === "index_not_found";
190
+ async function removeUnwanted(ctx, wanted) {
191
+ const unwanted = [];
192
+ for (let offset = 0;; offset += LIST_LIMIT) {
193
+ const page = await ctx.index.raw.getDocuments({ fields: [ctx.primaryKey], limit: LIST_LIMIT, offset }).catch((error) => {
194
+ if (missingIndex(error))
195
+ return;
196
+ throw error;
197
+ });
198
+ if (!page)
199
+ return 0;
200
+ for (const document of page.results) {
201
+ const id = document[ctx.primaryKey];
202
+ if (!wanted.has(keyOf(id)))
203
+ unwanted.push(id);
204
+ }
205
+ if (offset + LIST_LIMIT >= page.total)
206
+ break;
207
+ }
208
+ await deleteIds(ctx, unwanted);
209
+ return unwanted.length;
210
+ }
211
+ async function reindex(ctx) {
212
+ checkIdle(ctx, "reindex");
213
+ try {
214
+ const token = await currentToken(ctx);
215
+ const wanted = new Set;
216
+ const { indexed, skipped } = await sendAll(ctx, wanted);
217
+ const removed = await removeUnwanted(ctx, wanted);
218
+ await saveState(ctx, token, true);
219
+ return { indexed, skipped, removed };
220
+ } catch (error) {
221
+ throw failed(ctx.name, "reindexing", error);
222
+ }
223
+ }
224
+
225
+ // src/follow.ts
226
+ function historyLost(error) {
227
+ const code = error?.serverCode;
228
+ return code === 286 || code === 280;
229
+ }
230
+ function flush(ctx, f, subscription) {
231
+ f.chain = f.chain.then(async () => {
232
+ if (f.failure)
233
+ throw f.failure;
234
+ if (f.done)
235
+ return;
236
+ clearTimeout(f.timer);
237
+ f.timer = undefined;
238
+ const entries = [...f.pending.values()];
239
+ const upTo = f.last;
240
+ f.pending.clear();
241
+ try {
242
+ if (entries.length > 0)
243
+ await send(ctx, entries);
244
+ const idle = entries.length === 0 && f.inFlight === 0;
245
+ const at = upTo ?? (idle ? subscription.position : undefined);
246
+ if (at && at !== f.saved) {
247
+ await saveState(ctx, at, false);
248
+ f.saved = at;
249
+ }
250
+ } catch (error) {
251
+ f.failure ??= failed(ctx.name, "sending changes", error);
252
+ throw f.failure;
253
+ }
254
+ });
255
+ return f.chain;
256
+ }
257
+ async function open(ctx, token) {
258
+ const f = {
259
+ pending: new Map,
260
+ last: undefined,
261
+ saved: token,
262
+ inFlight: 0,
263
+ timer: undefined,
264
+ beat: undefined,
265
+ chain: Promise.resolve(),
266
+ done: false,
267
+ failure: undefined
268
+ };
269
+ const handle = async (change) => {
270
+ if (f.failure)
271
+ throw f.failure;
272
+ f.inFlight += 1;
273
+ let entry;
274
+ try {
275
+ const document = change.type === "delete" ? undefined : change.document;
276
+ entry = await entryOf(ctx, change.id, document);
277
+ } catch (error) {
278
+ f.failure = failed(ctx.name, "following changes", error);
279
+ throw f.failure;
280
+ } finally {
281
+ f.inFlight -= 1;
282
+ }
283
+ f.pending.set(entry.key, entry);
284
+ f.last = change.resumeToken;
285
+ if (f.pending.size >= ctx.batchSize) {
286
+ await flush(ctx, f, subscription);
287
+ } else if (!f.timer) {
288
+ f.timer = setTimeout(() => {
289
+ flush(ctx, f, subscription).catch(() => subscription.close());
290
+ }, ctx.flushIntervalMs);
291
+ }
292
+ };
293
+ const subscription = ctx.collection.onChange(handle, { startAfter: token });
294
+ f.beat = setInterval(() => {
295
+ flush(ctx, f, subscription).catch(() => subscription.close());
296
+ }, ctx.positionIntervalMs);
297
+ const stop = () => {
298
+ f.done = true;
299
+ clearTimeout(f.timer);
300
+ clearInterval(f.beat);
301
+ ctx.follower.running = false;
302
+ };
303
+ const closed = subscription.closed.then(async (reason) => {
304
+ stop();
305
+ await f.chain.catch(() => {
306
+ return;
307
+ });
308
+ if (f.failure)
309
+ throw f.failure;
310
+ if (reason === "invalidated")
311
+ await clearState(ctx);
312
+ return reason;
313
+ }, (error) => {
314
+ stop();
315
+ throw f.failure ?? failed(ctx.name, "following changes", error);
316
+ });
317
+ try {
318
+ await subscription.ready;
319
+ } catch (error) {
320
+ stop();
321
+ closed.catch(() => {
322
+ return;
323
+ });
324
+ throw error;
325
+ }
326
+ ctx.follower.running = true;
327
+ const close = async () => {
328
+ try {
329
+ await flush(ctx, f, subscription);
330
+ } finally {
331
+ await subscription.close();
332
+ }
333
+ };
334
+ return {
335
+ ready: subscription.ready,
336
+ closed,
337
+ flush: () => flush(ctx, f, subscription),
338
+ close,
339
+ [Symbol.asyncDispose]: close
340
+ };
341
+ }
342
+ async function start(ctx) {
343
+ checkIdle(ctx, "start it twice");
344
+ let state = await readState(ctx);
345
+ if (!state) {
346
+ await reindex(ctx);
347
+ state = await readState(ctx);
348
+ }
349
+ try {
350
+ return await open(ctx, state.resumeToken);
351
+ } catch (error) {
352
+ if (!historyLost(error))
353
+ throw failed(ctx.name, "starting", error);
354
+ if (ctx.onHistoryLost === "fail") {
355
+ throw new SearchSyncError(`Search sync "${ctx.name}" was last at a point the server's change ` + "history no longer reaches. Reindex it, or start it with " + "onHistoryLost: 'reindex'.", { code: "HISTORY_LOST", sync: ctx.name, cause: error });
356
+ }
357
+ await reindex(ctx);
358
+ const fresh = await readState(ctx);
359
+ try {
360
+ return await open(ctx, fresh.resumeToken);
361
+ } catch (again) {
362
+ throw failed(ctx.name, "starting", again);
363
+ }
364
+ }
365
+ }
366
+
367
+ // src/create-search-sync.ts
368
+ function createSearchSync(options) {
369
+ const ctx = createContext(options);
370
+ return {
371
+ name: ctx.name,
372
+ reindex: () => reindex(ctx),
373
+ start: () => start(ctx),
374
+ state: () => readState(ctx)
375
+ };
376
+ }
377
+ export {
378
+ SearchSyncError,
379
+ createSearchSync
380
+ };
381
+
382
+ //# debugId=3F6CF388DE058E4464756E2164756E21
383
+ //# sourceMappingURL=index.js.map
@@ -0,0 +1,18 @@
1
+ {
2
+ "version": 3,
3
+ "sources": ["../src/context.ts", "../src/batch.ts", "../src/errors.ts", "../src/documents.ts", "../src/reindex.ts", "../src/running.ts", "../src/state.ts", "../src/follow.ts", "../src/create-search-sync.ts"],
4
+ "sourcesContent": [
5
+ "import type { AnyIndexDefinition, TypedIndex } from '@nxgt/meilisearch';\nimport type { AnyCollectionDefinition, TypedCollection } from '@nxgt/mongo';\nimport type { Collection } from 'mongodb';\nimport type { SearchSyncOptions, SearchSyncState } from './types';\n\n/** Loosely typed: this layer works on any documents; the public types are what callers see. */\nexport type Doc = Record<string, unknown>;\n\n/**\n * What a sync works from, resolved once. Data, and the caller's own two\n * functions as they were given.\n */\nexport interface SyncContext {\n\treadonly name: string;\n\treadonly collection: TypedCollection<AnyCollectionDefinition>;\n\treadonly index: TypedIndex<AnyIndexDefinition>;\n\treadonly primaryKey: string;\n\treadonly state: Collection<SearchSyncState>;\n\treadonly batchSize: number;\n\treadonly flushIntervalMs: number;\n\treadonly positionIntervalMs: number;\n\treadonly pageSize: number;\n\treadonly onHistoryLost: 'reindex' | 'fail';\n\treadonly transform: (document: Doc) => unknown;\n\treadonly toIndexId: (id: unknown) => unknown;\n\t/**\n\t * Whether this sync is following changes in this process. A reindex\n\t * beside its own follower would remove what the follower has just\n\t * indexed, and the follower would never send it again.\n\t */\n\treadonly follower: { running: boolean };\n}\n\nconst DEFAULT_STATE_COLLECTION = 'nxgt_search_sync';\n\nfunction positive(option: string, value: number | undefined, fallback: number) {\n\tconst resolved = value ?? fallback;\n\tif (!Number.isInteger(resolved) || resolved < 1) {\n\t\tthrow new TypeError(\n\t\t\t`createSearchSync: ${option} must be a whole number above 0, not ${String(value)}`,\n\t\t);\n\t}\n\treturn resolved;\n}\n\nexport function createContext(\n\toptions: SearchSyncOptions<AnyCollectionDefinition, AnyIndexDefinition>,\n): SyncContext {\n\tconst { collection, index } = options;\n\tif (typeof options.transform !== 'function') {\n\t\tthrow new TypeError('createSearchSync: transform must be a function');\n\t}\n\tconst name = options.name ?? `${collection.collectionName}:${index.uid}`;\n\tif (name === '') {\n\t\tthrow new TypeError('createSearchSync: name must not be empty');\n\t}\n\tconst flushIntervalMs = options.flushIntervalMs ?? 1000;\n\tif (!Number.isInteger(flushIntervalMs) || flushIntervalMs < 0) {\n\t\tthrow new TypeError(\n\t\t\t`createSearchSync: flushIntervalMs must be a whole number of milliseconds, not ${String(options.flushIntervalMs)}`,\n\t\t);\n\t}\n\treturn {\n\t\tname,\n\t\tcollection,\n\t\tindex,\n\t\tprimaryKey: index.definition.primaryKey as string,\n\t\tstate: collection.db.collection<SearchSyncState>(\n\t\t\toptions.stateCollection ?? DEFAULT_STATE_COLLECTION,\n\t\t),\n\t\tbatchSize: positive('batchSize', options.batchSize, 500),\n\t\tpositionIntervalMs: positive(\n\t\t\t'positionIntervalMs',\n\t\t\toptions.positionIntervalMs,\n\t\t\t60_000,\n\t\t),\n\t\tflushIntervalMs,\n\t\tpageSize: positive('pageSize', options.pageSize, 100),\n\t\tonHistoryLost: options.onHistoryLost ?? 'reindex',\n\t\ttransform: options.transform as (document: Doc) => unknown,\n\t\ttoIndexId: (options.toIndexId ?? String) as (id: unknown) => unknown,\n\t\tfollower: { running: false },\n\t};\n}\n",
6
+ "import type { Doc, SyncContext } from './context';\nimport type { Entry } from './documents';\n\n/**\n * How long a write may take to be applied. Meilisearch's own default is\n * five seconds, which a batch on a large index outlasts.\n */\nconst WAIT = { wait: { timeout: 120_000 } } as const;\n\nfunction chunks<T>(items: readonly T[], size: number): T[][] {\n\tconst out: T[][] = [];\n\tfor (let i = 0; i < items.length; i += size) {\n\t\tout.push(items.slice(i, i + size));\n\t}\n\treturn out;\n}\n\n/** Adds documents, `batchSize` at a time, each batch applied before the next. */\nexport async function sendDocuments(\n\tctx: SyncContext,\n\tdocuments: readonly Doc[],\n): Promise<void> {\n\tfor (const batch of chunks(documents, ctx.batchSize)) {\n\t\tawait ctx.index.add(batch, WAIT);\n\t}\n}\n\n/** Deletes documents by id, `batchSize` at a time. */\nexport async function deleteIds(\n\tctx: SyncContext,\n\tids: readonly unknown[],\n): Promise<void> {\n\tfor (const batch of chunks(ids, ctx.batchSize)) {\n\t\tawait ctx.index.delete(batch, WAIT);\n\t}\n}\n\n/**\n * Applies entries: one per id, so adds and deletes touch different\n * documents and their order does not matter.\n */\nexport async function send(\n\tctx: SyncContext,\n\tentries: readonly Entry[],\n): Promise<void> {\n\tconst documents: Doc[] = [];\n\tconst ids: unknown[] = [];\n\tfor (const entry of entries) {\n\t\tif (entry.kind === 'upsert') documents.push(entry.document);\n\t\telse ids.push(entry.id);\n\t}\n\tawait sendDocuments(ctx, documents);\n\tawait deleteIds(ctx, ids);\n}\n",
7
+ "/** What stopped a sync, as a string to switch on. */\nexport type SearchSyncErrorCode =\n\t/** The resume point is older than the server's change history. */\n\t| 'HISTORY_LOST'\n\t/** The transform gave a document whose primary key is not its index id. */\n\t| 'ID_MISMATCH'\n\t/** This sync is already following changes in this process. */\n\t| 'RUNNING'\n\t/** Anything else: the cause says what. */\n\t| 'FAILED';\n\nexport interface SearchSyncErrorOptions {\n\tcode: SearchSyncErrorCode;\n\tsync: string;\n\tcause?: unknown;\n}\n\n/** An error of a sync, with the sync's name and what caused it. */\nexport class SearchSyncError extends Error {\n\toverride name = 'SearchSyncError';\n\treadonly code: SearchSyncErrorCode;\n\t/** The sync's name. */\n\treadonly sync: string;\n\n\tconstructor(message: string, options: SearchSyncErrorOptions) {\n\t\tsuper(\n\t\t\tmessage,\n\t\t\toptions.cause === undefined ? undefined : { cause: options.cause },\n\t\t);\n\t\tthis.code = options.code;\n\t\tthis.sync = options.sync;\n\t}\n}\n\n/** `error` as a `SearchSyncError`, kept as it is when it already is one. */\nexport function failed(sync: string, doing: string, error: unknown) {\n\tif (error instanceof SearchSyncError) return error;\n\tconst reason = error instanceof Error ? error.message : String(error);\n\treturn new SearchSyncError(\n\t\t`Search sync \"${sync}\" failed ${doing}: ${reason}`,\n\t\t{\n\t\t\tcode: 'FAILED',\n\t\t\tsync,\n\t\t\tcause: error,\n\t\t},\n\t);\n}\n",
8
+ "import type { Doc, SyncContext } from './context';\nimport { SearchSyncError } from './errors';\n\n/** One document's fate in the index: sent, or taken out. */\nexport type Entry =\n\t| { kind: 'upsert'; key: string; document: Doc }\n\t| { kind: 'delete'; key: string; id: unknown };\n\n/** An id as a key: `1` and `'1'` are different ids to Meilisearch. */\nexport const keyOf = (id: unknown) => `${typeof id}:${String(id)}`;\n\nconst isRecord = (value: unknown): value is Doc =>\n\ttypeof value === 'object' && value !== null && !Array.isArray(value);\n\n/**\n * What the index should hold for a document of the collection: what the\n * transform makes of it, or nothing when it is gone or kept out.\n */\nexport async function entryOf(\n\tctx: SyncContext,\n\tmongoId: unknown,\n\tdocument: Doc | undefined,\n): Promise<Entry> {\n\tconst id = ctx.toIndexId(mongoId);\n\tconst key = keyOf(id);\n\tif (!document) return { kind: 'delete', key, id };\n\tconst indexed = await ctx.transform(document);\n\tif (indexed === null) return { kind: 'delete', key, id };\n\tif (!isRecord(indexed)) {\n\t\tthrow new TypeError(\n\t\t\t`transform must return a document or null, not ${String(indexed)}`,\n\t\t);\n\t}\n\tconst given = indexed[ctx.primaryKey];\n\tif (keyOf(given) !== key) {\n\t\tthrow new SearchSyncError(\n\t\t\t`Search sync \"${ctx.name}\": transform gave \"${ctx.primaryKey}\" ` +\n\t\t\t\t`${JSON.stringify(given)} for the document ${String(mongoId)}, whose ` +\n\t\t\t\t`index id is ${JSON.stringify(id)}. A document under another id ` +\n\t\t\t\t'could never be taken out of the index again.',\n\t\t\t{ code: 'ID_MISMATCH', sync: ctx.name },\n\t\t);\n\t}\n\treturn { kind: 'upsert', key, document: indexed };\n}\n",
9
+ "import type { ResumeToken } from '@nxgt/mongo';\nimport { MeilisearchApiError } from 'meilisearch';\nimport { deleteIds, sendDocuments } from './batch';\nimport type { Doc, SyncContext } from './context';\nimport { entryOf, keyOf } from './documents';\nimport { failed } from './errors';\nimport { checkIdle } from './running';\nimport { saveState } from './state';\nimport type { ReindexReport } from './types';\n\n/**\n * Where the collection's changes are now, without waiting for one: a new\n * stream's first read answers with its position. The server holds that read\n * for up to `maxAwaitTimeMS` — a second by default, measured.\n */\nasync function currentToken(ctx: SyncContext): Promise<ResumeToken> {\n\tconst stream = ctx.collection.raw.watch([], { maxAwaitTimeMS: 1 });\n\ttry {\n\t\tawait stream.tryNext();\n\t\treturn stream.resumeToken as unknown as ResumeToken;\n\t} finally {\n\t\tawait stream.close();\n\t}\n}\n\n/** Every live document through the transform, a page at a time. */\nasync function sendAll(ctx: SyncContext, wanted: Set<string>) {\n\tlet indexed = 0;\n\tlet skipped = 0;\n\tlet after: string | null | undefined;\n\tdo {\n\t\tconst page = await ctx.collection.paginateByCursor({\n\t\t\tafter,\n\t\t\tlimit: ctx.pageSize,\n\t\t});\n\t\tconst documents: Doc[] = [];\n\t\tfor (const document of page.items as Doc[]) {\n\t\t\tconst entry = await entryOf(ctx, document._id, document);\n\t\t\tif (entry.kind === 'delete') {\n\t\t\t\tskipped += 1;\n\t\t\t\tcontinue;\n\t\t\t}\n\t\t\twanted.add(entry.key);\n\t\t\tdocuments.push(entry.document);\n\t\t}\n\t\tawait sendDocuments(ctx, documents);\n\t\tindexed += documents.length;\n\t\tafter = page.nextCursor;\n\t} while (after);\n\treturn { indexed, skipped };\n}\n\nconst LIST_LIMIT = 1000;\n\n/** Nothing was ever written to it: an index that does not exist holds nothing. */\nconst missingIndex = (error: unknown) =>\n\terror instanceof MeilisearchApiError &&\n\terror.cause?.code === 'index_not_found';\n\n/** Takes out what the index holds and the collection no longer gives it. */\nasync function removeUnwanted(\n\tctx: SyncContext,\n\twanted: Set<string>,\n): Promise<number> {\n\tconst unwanted: unknown[] = [];\n\tfor (let offset = 0; ; offset += LIST_LIMIT) {\n\t\tconst page = await ctx.index.raw\n\t\t\t.getDocuments({ fields: [ctx.primaryKey], limit: LIST_LIMIT, offset })\n\t\t\t.catch((error: unknown) => {\n\t\t\t\tif (missingIndex(error)) return undefined;\n\t\t\t\tthrow error;\n\t\t\t});\n\t\tif (!page) return 0;\n\t\tfor (const document of page.results as Doc[]) {\n\t\t\tconst id = document[ctx.primaryKey];\n\t\t\tif (!wanted.has(keyOf(id))) unwanted.push(id);\n\t\t}\n\t\tif (offset + LIST_LIMIT >= page.total) break;\n\t}\n\tawait deleteIds(ctx, unwanted);\n\treturn unwanted.length;\n}\n\nexport async function reindex(ctx: SyncContext): Promise<ReindexReport> {\n\tcheckIdle(ctx, 'reindex');\n\ttry {\n\t\t// Taken first: a change made while the documents are read is followed\n\t\t// again from here, so it cannot fall between the two.\n\t\tconst token = await currentToken(ctx);\n\t\tconst wanted = new Set<string>();\n\t\tconst { indexed, skipped } = await sendAll(ctx, wanted);\n\t\tconst removed = await removeUnwanted(ctx, wanted);\n\t\tawait saveState(ctx, token, true);\n\t\treturn { indexed, skipped, removed };\n\t} catch (error) {\n\t\tthrow failed(ctx.name, 'reindexing', error);\n\t}\n}\n",
10
+ "import type { SyncContext } from './context';\nimport { SearchSyncError } from './errors';\n\n/**\n * Refuses to do beside a running follower what would undo its work: a\n * reindex removes what the follower has just indexed, and the follower,\n * having handled that change, never sends it again.\n */\nexport function checkIdle(ctx: SyncContext, doing: string): void {\n\tif (!ctx.follower.running) return;\n\tthrow new SearchSyncError(\n\t\t`Search sync \"${ctx.name}\" is already following changes in this ` +\n\t\t\t`process: close it before you ${doing}.`,\n\t\t{ code: 'RUNNING', sync: ctx.name },\n\t);\n}\n",
11
+ "import type { ResumeToken } from '@nxgt/mongo';\nimport type { SyncContext } from './context';\nimport { failed } from './errors';\nimport type { SearchSyncState } from './types';\n\n/** What the sync has recorded, or `undefined` before its first reindex. */\nexport async function readState(\n\tctx: SyncContext,\n): Promise<SearchSyncState | undefined> {\n\ttry {\n\t\treturn (await ctx.state.findOne({ _id: ctx.name })) ?? undefined;\n\t} catch (error) {\n\t\tthrow failed(ctx.name, 'reading its resume point', error);\n\t}\n}\n\n/** Forgets everything recorded: the next `start` reindexes. */\nexport async function clearState(ctx: SyncContext): Promise<void> {\n\ttry {\n\t\tawait ctx.state.deleteOne({ _id: ctx.name });\n\t} catch (error) {\n\t\tthrow failed(ctx.name, 'forgetting its resume point', error);\n\t}\n}\n\n/** Moves the resume point, and says a reindex finished when one did. */\nexport async function saveState(\n\tctx: SyncContext,\n\tresumeToken: ResumeToken,\n\treindexed: boolean,\n): Promise<void> {\n\tconst now = new Date();\n\ttry {\n\t\tawait ctx.state.updateOne(\n\t\t\t{ _id: ctx.name },\n\t\t\t{\n\t\t\t\t$set: {\n\t\t\t\t\tresumeToken,\n\t\t\t\t\tupdatedAt: now,\n\t\t\t\t\t...(reindexed ? { reindexedAt: now } : {}),\n\t\t\t\t},\n\t\t\t},\n\t\t\t{ upsert: true },\n\t\t);\n\t} catch (error) {\n\t\tthrow failed(ctx.name, 'recording its resume point', error);\n\t}\n}\n",
12
+ "import type {\n\tAnyCollectionDefinition,\n\tChangeOf,\n\tChangeSubscription,\n\tResumeToken,\n} from '@nxgt/mongo';\nimport { send } from './batch';\nimport type { Doc, SyncContext } from './context';\nimport { type Entry, entryOf } from './documents';\nimport { failed, SearchSyncError } from './errors';\nimport { reindex } from './reindex';\nimport { checkIdle } from './running';\nimport { clearState, readState, saveState } from './state';\nimport type { RunningSearchSync } from './types';\n\n/** The server no longer has the history a token points into. */\nfunction historyLost(error: unknown): boolean {\n\tconst code = (error as { serverCode?: number } | undefined)?.serverCode;\n\treturn code === 286 || code === 280;\n}\n\n/** What a running sync holds between changes. */\ninterface Following {\n\treadonly pending: Map<string, Entry>;\n\t/** The token of the last change in `pending`. */\n\tlast: ResumeToken | undefined;\n\tsaved: ResumeToken;\n\t/** Changes whose handler has not queued them yet. */\n\tinFlight: number;\n\ttimer: ReturnType<typeof setTimeout> | undefined;\n\t/** Records where the stream is while nothing is being sent. */\n\tbeat: ReturnType<typeof setInterval> | undefined;\n\t/** Flushes run one after the other. */\n\tchain: Promise<void>;\n\t/** The stream has stopped: what is queued no longer records anything. */\n\tdone: boolean;\n\tfailure: SearchSyncError | undefined;\n}\n\n/**\n * Sends what is pending, then records the token that covers it — or, with\n * nothing to send and no change being handled, where the stream is: a quiet\n * collection would otherwise keep a resume point until the server's history\n * no longer reaches it.\n */\nfunction flush(\n\tctx: SyncContext,\n\tf: Following,\n\tsubscription: ChangeSubscription,\n): Promise<void> {\n\tf.chain = f.chain.then(async () => {\n\t\tif (f.failure) throw f.failure;\n\t\tif (f.done) return;\n\t\tclearTimeout(f.timer);\n\t\tf.timer = undefined;\n\t\tconst entries = [...f.pending.values()];\n\t\tconst upTo = f.last;\n\t\tf.pending.clear();\n\t\ttry {\n\t\t\tif (entries.length > 0) await send(ctx, entries);\n\t\t\tconst idle = entries.length === 0 && f.inFlight === 0;\n\t\t\tconst at = upTo ?? (idle ? subscription.position : undefined);\n\t\t\tif (at && at !== f.saved) {\n\t\t\t\tawait saveState(ctx, at, false);\n\t\t\t\tf.saved = at;\n\t\t\t}\n\t\t} catch (error) {\n\t\t\tf.failure ??= failed(ctx.name, 'sending changes', error);\n\t\t\tthrow f.failure;\n\t\t}\n\t});\n\treturn f.chain;\n}\n\nasync function open(\n\tctx: SyncContext,\n\ttoken: ResumeToken,\n): Promise<RunningSearchSync> {\n\tconst f: Following = {\n\t\tpending: new Map(),\n\t\tlast: undefined,\n\t\tsaved: token,\n\t\tinFlight: 0,\n\t\ttimer: undefined,\n\t\tbeat: undefined,\n\t\tchain: Promise.resolve(),\n\t\tdone: false,\n\t\tfailure: undefined,\n\t};\n\tconst handle = async (change: ChangeOf<AnyCollectionDefinition>) => {\n\t\tif (f.failure) throw f.failure;\n\t\tf.inFlight += 1;\n\t\tlet entry: Entry;\n\t\ttry {\n\t\t\tconst document =\n\t\t\t\tchange.type === 'delete' ? undefined : (change.document as Doc);\n\t\t\tentry = await entryOf(ctx, change.id, document);\n\t\t} catch (error) {\n\t\t\t// Kept, so `flush` and `close` report it as `closed` does.\n\t\t\tf.failure = failed(ctx.name, 'following changes', error);\n\t\t\tthrow f.failure;\n\t\t} finally {\n\t\t\tf.inFlight -= 1;\n\t\t}\n\t\tf.pending.set(entry.key, entry);\n\t\tf.last = change.resumeToken;\n\t\tif (f.pending.size >= ctx.batchSize) {\n\t\t\tawait flush(ctx, f, subscription);\n\t\t} else if (!f.timer) {\n\t\t\tf.timer = setTimeout(() => {\n\t\t\t\tflush(ctx, f, subscription).catch(() => subscription.close());\n\t\t\t}, ctx.flushIntervalMs);\n\t\t}\n\t};\n\tconst subscription = ctx.collection.onChange(handle, { startAfter: token });\n\tf.beat = setInterval(() => {\n\t\tflush(ctx, f, subscription).catch(() => subscription.close());\n\t}, ctx.positionIntervalMs);\n\tconst stop = () => {\n\t\tf.done = true;\n\t\tclearTimeout(f.timer);\n\t\tclearInterval(f.beat);\n\t\tctx.follower.running = false;\n\t};\n\tconst closed = subscription.closed.then(\n\t\tasync (reason) => {\n\t\t\tstop();\n\t\t\t// A flush the timer or the beat had queued would otherwise record a\n\t\t\t// point after this.\n\t\t\tawait f.chain.catch(() => undefined);\n\t\t\tif (f.failure) throw f.failure;\n\t\t\t// The collection is gone, and so is the point the stream stopped at:\n\t\t\t// what is recorded is older, and every later start would stop here\n\t\t\t// again. Forgetting it makes the next one reindex.\n\t\t\tif (reason === 'invalidated') await clearState(ctx);\n\t\t\treturn reason;\n\t\t},\n\t\t(error: unknown) => {\n\t\t\tstop();\n\t\t\tthrow f.failure ?? failed(ctx.name, 'following changes', error);\n\t\t},\n\t);\n\ttry {\n\t\tawait subscription.ready;\n\t} catch (error) {\n\t\tstop();\n\t\tclosed.catch(() => undefined);\n\t\tthrow error;\n\t}\n\tctx.follower.running = true;\n\tconst close = async () => {\n\t\ttry {\n\t\t\tawait flush(ctx, f, subscription);\n\t\t} finally {\n\t\t\tawait subscription.close();\n\t\t}\n\t};\n\treturn {\n\t\tready: subscription.ready,\n\t\tclosed,\n\t\tflush: () => flush(ctx, f, subscription),\n\t\tclose,\n\t\t[Symbol.asyncDispose]: close,\n\t};\n}\n\n/**\n * Follows the collection from its recorded point — after a reindex when\n * nothing is recorded, or when the point is older than the server's history\n * and `onHistoryLost` allows it.\n */\nexport async function start(ctx: SyncContext): Promise<RunningSearchSync> {\n\tcheckIdle(ctx, 'start it twice');\n\tlet state = await readState(ctx);\n\tif (!state) {\n\t\tawait reindex(ctx);\n\t\tstate = await readState(ctx);\n\t}\n\ttry {\n\t\treturn await open(ctx, (state as { resumeToken: ResumeToken }).resumeToken);\n\t} catch (error) {\n\t\tif (!historyLost(error)) throw failed(ctx.name, 'starting', error);\n\t\tif (ctx.onHistoryLost === 'fail') {\n\t\t\tthrow new SearchSyncError(\n\t\t\t\t`Search sync \"${ctx.name}\" was last at a point the server's change ` +\n\t\t\t\t\t'history no longer reaches. Reindex it, or start it with ' +\n\t\t\t\t\t\"onHistoryLost: 'reindex'.\",\n\t\t\t\t{ code: 'HISTORY_LOST', sync: ctx.name, cause: error },\n\t\t\t);\n\t\t}\n\t\tawait reindex(ctx);\n\t\tconst fresh = (await readState(ctx)) as { resumeToken: ResumeToken };\n\t\ttry {\n\t\t\treturn await open(ctx, fresh.resumeToken);\n\t\t} catch (again) {\n\t\t\tthrow failed(ctx.name, 'starting', again);\n\t\t}\n\t}\n}\n",
13
+ "import type { AnyIndexDefinition } from '@nxgt/meilisearch';\nimport type { AnyCollectionDefinition } from '@nxgt/mongo';\nimport { createContext } from './context';\nimport { start } from './follow';\nimport { reindex } from './reindex';\nimport { readState } from './state';\nimport type { SearchSync, SearchSyncOptions } from './types';\n\n/**\n * Keeps an index in step with a collection.\n *\n * ```ts\n * const usersSearch = createSearchSync({\n * \tcollection: getCollection(db, users),\n * \tindex: bindIndex(meili, userIndex),\n * \ttransform: (user) => (user.active ? { id: user.id, name: user.name } : null),\n * });\n * const running = await usersSearch.start(); // reindexes the first time\n * // …\n * await running.close();\n * ```\n *\n * Nothing is sent before `reindex` or `start`.\n */\nexport function createSearchSync<\n\tC extends AnyCollectionDefinition,\n\tI extends AnyIndexDefinition,\n>(options: SearchSyncOptions<C, I>): SearchSync {\n\t// The one place the caller's types are let go: a method taking `IdOf<C>`\n\t// is not one taking any id, so the widening cannot be implicit. Below\n\t// here, documents are records and ids are values.\n\tconst ctx = createContext(\n\t\toptions as unknown as SearchSyncOptions<\n\t\t\tAnyCollectionDefinition,\n\t\t\tAnyIndexDefinition\n\t\t>,\n\t);\n\treturn {\n\t\tname: ctx.name,\n\t\treindex: () => reindex(ctx),\n\t\tstart: () => start(ctx),\n\t\tstate: () => readState(ctx),\n\t};\n}\n"
14
+ ],
15
+ "mappings": ";AAiCA,IAAM,2BAA2B;AAEjC,SAAS,QAAQ,CAAC,QAAgB,OAA2B,UAAkB;AAAA,EAC9E,MAAM,WAAW,SAAS;AAAA,EAC1B,IAAI,CAAC,OAAO,UAAU,QAAQ,KAAK,WAAW,GAAG;AAAA,IAChD,MAAM,IAAI,UACT,qBAAqB,8CAA8C,OAAO,KAAK,GAChF;AAAA,EACD;AAAA,EACA,OAAO;AAAA;AAGD,SAAS,aAAa,CAC5B,SACc;AAAA,EACd,QAAQ,YAAY,UAAU;AAAA,EAC9B,IAAI,OAAO,QAAQ,cAAc,YAAY;AAAA,IAC5C,MAAM,IAAI,UAAU,gDAAgD;AAAA,EACrE;AAAA,EACA,MAAM,OAAO,QAAQ,QAAQ,GAAG,WAAW,kBAAkB,MAAM;AAAA,EACnE,IAAI,SAAS,IAAI;AAAA,IAChB,MAAM,IAAI,UAAU,0CAA0C;AAAA,EAC/D;AAAA,EACA,MAAM,kBAAkB,QAAQ,mBAAmB;AAAA,EACnD,IAAI,CAAC,OAAO,UAAU,eAAe,KAAK,kBAAkB,GAAG;AAAA,IAC9D,MAAM,IAAI,UACT,iFAAiF,OAAO,QAAQ,eAAe,GAChH;AAAA,EACD;AAAA,EACA,OAAO;AAAA,IACN;AAAA,IACA;AAAA,IACA;AAAA,IACA,YAAY,MAAM,WAAW;AAAA,IAC7B,OAAO,WAAW,GAAG,WACpB,QAAQ,mBAAmB,wBAC5B;AAAA,IACA,WAAW,SAAS,aAAa,QAAQ,WAAW,GAAG;AAAA,IACvD,oBAAoB,SACnB,sBACA,QAAQ,oBACR,KACD;AAAA,IACA;AAAA,IACA,UAAU,SAAS,YAAY,QAAQ,UAAU,GAAG;AAAA,IACpD,eAAe,QAAQ,iBAAiB;AAAA,IACxC,WAAW,QAAQ;AAAA,IACnB,WAAY,QAAQ,aAAa;AAAA,IACjC,UAAU,EAAE,SAAS,MAAM;AAAA,EAC5B;AAAA;;;AC3ED,IAAM,OAAO,EAAE,MAAM,EAAE,SAAS,OAAQ,EAAE;AAE1C,SAAS,MAAS,CAAC,OAAqB,MAAqB;AAAA,EAC5D,MAAM,MAAa,CAAC;AAAA,EACpB,SAAS,IAAI,EAAG,IAAI,MAAM,QAAQ,KAAK,MAAM;AAAA,IAC5C,IAAI,KAAK,MAAM,MAAM,GAAG,IAAI,IAAI,CAAC;AAAA,EAClC;AAAA,EACA,OAAO;AAAA;AAIR,eAAsB,aAAa,CAClC,KACA,WACgB;AAAA,EAChB,WAAW,SAAS,OAAO,WAAW,IAAI,SAAS,GAAG;AAAA,IACrD,MAAM,IAAI,MAAM,IAAI,OAAO,IAAI;AAAA,EAChC;AAAA;AAID,eAAsB,SAAS,CAC9B,KACA,KACgB;AAAA,EAChB,WAAW,SAAS,OAAO,KAAK,IAAI,SAAS,GAAG;AAAA,IAC/C,MAAM,IAAI,MAAM,OAAO,OAAO,IAAI;AAAA,EACnC;AAAA;AAOD,eAAsB,IAAI,CACzB,KACA,SACgB;AAAA,EAChB,MAAM,YAAmB,CAAC;AAAA,EAC1B,MAAM,MAAiB,CAAC;AAAA,EACxB,WAAW,SAAS,SAAS;AAAA,IAC5B,IAAI,MAAM,SAAS;AAAA,MAAU,UAAU,KAAK,MAAM,QAAQ;AAAA,IACrD;AAAA,UAAI,KAAK,MAAM,EAAE;AAAA,EACvB;AAAA,EACA,MAAM,cAAc,KAAK,SAAS;AAAA,EAClC,MAAM,UAAU,KAAK,GAAG;AAAA;;;AClClB,MAAM,wBAAwB,MAAM;AAAA,EAM1C,WAAW,CAAC,SAAiB,SAAiC;AAAA,IAC7D,MACC,SACA,QAAQ,UAAU,YAAY,YAAY,EAAE,OAAO,QAAQ,MAAM,CAClE;AAAA,IATQ,YAAO;AAAA,IAUf,KAAK,OAAO,QAAQ;AAAA,IACpB,KAAK,OAAO,QAAQ;AAAA;AAEtB;AAGO,SAAS,MAAM,CAAC,MAAc,OAAe,OAAgB;AAAA,EACnE,IAAI,iBAAiB;AAAA,IAAiB,OAAO;AAAA,EAC7C,MAAM,SAAS,iBAAiB,QAAQ,MAAM,UAAU,OAAO,KAAK;AAAA,EACpE,OAAO,IAAI,gBACV,gBAAgB,gBAAgB,UAAU,UAC1C;AAAA,IACC,MAAM;AAAA,IACN;AAAA,IACA,OAAO;AAAA,EACR,CACD;AAAA;;;ACpCM,IAAM,QAAQ,CAAC,OAAgB,GAAG,OAAO,MAAM,OAAO,EAAE;AAE/D,IAAM,WAAW,CAAC,UACjB,OAAO,UAAU,YAAY,UAAU,QAAQ,CAAC,MAAM,QAAQ,KAAK;AAMpE,eAAsB,OAAO,CAC5B,KACA,SACA,UACiB;AAAA,EACjB,MAAM,KAAK,IAAI,UAAU,OAAO;AAAA,EAChC,MAAM,MAAM,MAAM,EAAE;AAAA,EACpB,IAAI,CAAC;AAAA,IAAU,OAAO,EAAE,MAAM,UAAU,KAAK,GAAG;AAAA,EAChD,MAAM,UAAU,MAAM,IAAI,UAAU,QAAQ;AAAA,EAC5C,IAAI,YAAY;AAAA,IAAM,OAAO,EAAE,MAAM,UAAU,KAAK,GAAG;AAAA,EACvD,IAAI,CAAC,SAAS,OAAO,GAAG;AAAA,IACvB,MAAM,IAAI,UACT,iDAAiD,OAAO,OAAO,GAChE;AAAA,EACD;AAAA,EACA,MAAM,QAAQ,QAAQ,IAAI;AAAA,EAC1B,IAAI,MAAM,KAAK,MAAM,KAAK;AAAA,IACzB,MAAM,IAAI,gBACT,gBAAgB,IAAI,0BAA0B,IAAI,iBACjD,GAAG,KAAK,UAAU,KAAK,sBAAsB,OAAO,OAAO,cAC3D,eAAe,KAAK,UAAU,EAAE,oCAChC,gDACD,EAAE,MAAM,eAAe,MAAM,IAAI,KAAK,CACvC;AAAA,EACD;AAAA,EACA,OAAO,EAAE,MAAM,UAAU,KAAK,UAAU,QAAQ;AAAA;;;AC1CjD;;;ACOO,SAAS,SAAS,CAAC,KAAkB,OAAqB;AAAA,EAChE,IAAI,CAAC,IAAI,SAAS;AAAA,IAAS;AAAA,EAC3B,MAAM,IAAI,gBACT,gBAAgB,IAAI,gDACnB,gCAAgC,UACjC,EAAE,MAAM,WAAW,MAAM,IAAI,KAAK,CACnC;AAAA;;;ACRD,eAAsB,SAAS,CAC9B,KACuC;AAAA,EACvC,IAAI;AAAA,IACH,OAAQ,MAAM,IAAI,MAAM,QAAQ,EAAE,KAAK,IAAI,KAAK,CAAC,KAAM;AAAA,IACtD,OAAO,OAAO;AAAA,IACf,MAAM,OAAO,IAAI,MAAM,4BAA4B,KAAK;AAAA;AAAA;AAK1D,eAAsB,UAAU,CAAC,KAAiC;AAAA,EACjE,IAAI;AAAA,IACH,MAAM,IAAI,MAAM,UAAU,EAAE,KAAK,IAAI,KAAK,CAAC;AAAA,IAC1C,OAAO,OAAO;AAAA,IACf,MAAM,OAAO,IAAI,MAAM,+BAA+B,KAAK;AAAA;AAAA;AAK7D,eAAsB,SAAS,CAC9B,KACA,aACA,WACgB;AAAA,EAChB,MAAM,MAAM,IAAI;AAAA,EAChB,IAAI;AAAA,IACH,MAAM,IAAI,MAAM,UACf,EAAE,KAAK,IAAI,KAAK,GAChB;AAAA,MACC,MAAM;AAAA,QACL;AAAA,QACA,WAAW;AAAA,WACP,YAAY,EAAE,aAAa,IAAI,IAAI,CAAC;AAAA,MACzC;AAAA,IACD,GACA,EAAE,QAAQ,KAAK,CAChB;AAAA,IACC,OAAO,OAAO;AAAA,IACf,MAAM,OAAO,IAAI,MAAM,8BAA8B,KAAK;AAAA;AAAA;;;AF9B5D,eAAe,YAAY,CAAC,KAAwC;AAAA,EACnE,MAAM,SAAS,IAAI,WAAW,IAAI,MAAM,CAAC,GAAG,EAAE,gBAAgB,EAAE,CAAC;AAAA,EACjE,IAAI;AAAA,IACH,MAAM,OAAO,QAAQ;AAAA,IACrB,OAAO,OAAO;AAAA,YACb;AAAA,IACD,MAAM,OAAO,MAAM;AAAA;AAAA;AAKrB,eAAe,OAAO,CAAC,KAAkB,QAAqB;AAAA,EAC7D,IAAI,UAAU;AAAA,EACd,IAAI,UAAU;AAAA,EACd,IAAI;AAAA,EACJ,GAAG;AAAA,IACF,MAAM,OAAO,MAAM,IAAI,WAAW,iBAAiB;AAAA,MAClD;AAAA,MACA,OAAO,IAAI;AAAA,IACZ,CAAC;AAAA,IACD,MAAM,YAAmB,CAAC;AAAA,IAC1B,WAAW,YAAY,KAAK,OAAgB;AAAA,MAC3C,MAAM,QAAQ,MAAM,QAAQ,KAAK,SAAS,KAAK,QAAQ;AAAA,MACvD,IAAI,MAAM,SAAS,UAAU;AAAA,QAC5B,WAAW;AAAA,QACX;AAAA,MACD;AAAA,MACA,OAAO,IAAI,MAAM,GAAG;AAAA,MACpB,UAAU,KAAK,MAAM,QAAQ;AAAA,IAC9B;AAAA,IACA,MAAM,cAAc,KAAK,SAAS;AAAA,IAClC,WAAW,UAAU;AAAA,IACrB,QAAQ,KAAK;AAAA,EACd,SAAS;AAAA,EACT,OAAO,EAAE,SAAS,QAAQ;AAAA;AAG3B,IAAM,aAAa;AAGnB,IAAM,eAAe,CAAC,UACrB,iBAAiB,uBACjB,MAAM,OAAO,SAAS;AAGvB,eAAe,cAAc,CAC5B,KACA,QACkB;AAAA,EAClB,MAAM,WAAsB,CAAC;AAAA,EAC7B,SAAS,SAAS,IAAK,UAAU,YAAY;AAAA,IAC5C,MAAM,OAAO,MAAM,IAAI,MAAM,IAC3B,aAAa,EAAE,QAAQ,CAAC,IAAI,UAAU,GAAG,OAAO,YAAY,OAAO,CAAC,EACpE,MAAM,CAAC,UAAmB;AAAA,MAC1B,IAAI,aAAa,KAAK;AAAA,QAAG;AAAA,MACzB,MAAM;AAAA,KACN;AAAA,IACF,IAAI,CAAC;AAAA,MAAM,OAAO;AAAA,IAClB,WAAW,YAAY,KAAK,SAAkB;AAAA,MAC7C,MAAM,KAAK,SAAS,IAAI;AAAA,MACxB,IAAI,CAAC,OAAO,IAAI,MAAM,EAAE,CAAC;AAAA,QAAG,SAAS,KAAK,EAAE;AAAA,IAC7C;AAAA,IACA,IAAI,SAAS,cAAc,KAAK;AAAA,MAAO;AAAA,EACxC;AAAA,EACA,MAAM,UAAU,KAAK,QAAQ;AAAA,EAC7B,OAAO,SAAS;AAAA;AAGjB,eAAsB,OAAO,CAAC,KAA0C;AAAA,EACvE,UAAU,KAAK,SAAS;AAAA,EACxB,IAAI;AAAA,IAGH,MAAM,QAAQ,MAAM,aAAa,GAAG;AAAA,IACpC,MAAM,SAAS,IAAI;AAAA,IACnB,QAAQ,SAAS,YAAY,MAAM,QAAQ,KAAK,MAAM;AAAA,IACtD,MAAM,UAAU,MAAM,eAAe,KAAK,MAAM;AAAA,IAChD,MAAM,UAAU,KAAK,OAAO,IAAI;AAAA,IAChC,OAAO,EAAE,SAAS,SAAS,QAAQ;AAAA,IAClC,OAAO,OAAO;AAAA,IACf,MAAM,OAAO,IAAI,MAAM,cAAc,KAAK;AAAA;AAAA;;;AG/E5C,SAAS,WAAW,CAAC,OAAyB;AAAA,EAC7C,MAAM,OAAQ,OAA+C;AAAA,EAC7D,OAAO,SAAS,OAAO,SAAS;AAAA;AA2BjC,SAAS,KAAK,CACb,KACA,GACA,cACgB;AAAA,EAChB,EAAE,QAAQ,EAAE,MAAM,KAAK,YAAY;AAAA,IAClC,IAAI,EAAE;AAAA,MAAS,MAAM,EAAE;AAAA,IACvB,IAAI,EAAE;AAAA,MAAM;AAAA,IACZ,aAAa,EAAE,KAAK;AAAA,IACpB,EAAE,QAAQ;AAAA,IACV,MAAM,UAAU,CAAC,GAAG,EAAE,QAAQ,OAAO,CAAC;AAAA,IACtC,MAAM,OAAO,EAAE;AAAA,IACf,EAAE,QAAQ,MAAM;AAAA,IAChB,IAAI;AAAA,MACH,IAAI,QAAQ,SAAS;AAAA,QAAG,MAAM,KAAK,KAAK,OAAO;AAAA,MAC/C,MAAM,OAAO,QAAQ,WAAW,KAAK,EAAE,aAAa;AAAA,MACpD,MAAM,KAAK,SAAS,OAAO,aAAa,WAAW;AAAA,MACnD,IAAI,MAAM,OAAO,EAAE,OAAO;AAAA,QACzB,MAAM,UAAU,KAAK,IAAI,KAAK;AAAA,QAC9B,EAAE,QAAQ;AAAA,MACX;AAAA,MACC,OAAO,OAAO;AAAA,MACf,EAAE,YAAY,OAAO,IAAI,MAAM,mBAAmB,KAAK;AAAA,MACvD,MAAM,EAAE;AAAA;AAAA,GAET;AAAA,EACD,OAAO,EAAE;AAAA;AAGV,eAAe,IAAI,CAClB,KACA,OAC6B;AAAA,EAC7B,MAAM,IAAe;AAAA,IACpB,SAAS,IAAI;AAAA,IACb,MAAM;AAAA,IACN,OAAO;AAAA,IACP,UAAU;AAAA,IACV,OAAO;AAAA,IACP,MAAM;AAAA,IACN,OAAO,QAAQ,QAAQ;AAAA,IACvB,MAAM;AAAA,IACN,SAAS;AAAA,EACV;AAAA,EACA,MAAM,SAAS,OAAO,WAA8C;AAAA,IACnE,IAAI,EAAE;AAAA,MAAS,MAAM,EAAE;AAAA,IACvB,EAAE,YAAY;AAAA,IACd,IAAI;AAAA,IACJ,IAAI;AAAA,MACH,MAAM,WACL,OAAO,SAAS,WAAW,YAAa,OAAO;AAAA,MAChD,QAAQ,MAAM,QAAQ,KAAK,OAAO,IAAI,QAAQ;AAAA,MAC7C,OAAO,OAAO;AAAA,MAEf,EAAE,UAAU,OAAO,IAAI,MAAM,qBAAqB,KAAK;AAAA,MACvD,MAAM,EAAE;AAAA,cACP;AAAA,MACD,EAAE,YAAY;AAAA;AAAA,IAEf,EAAE,QAAQ,IAAI,MAAM,KAAK,KAAK;AAAA,IAC9B,EAAE,OAAO,OAAO;AAAA,IAChB,IAAI,EAAE,QAAQ,QAAQ,IAAI,WAAW;AAAA,MACpC,MAAM,MAAM,KAAK,GAAG,YAAY;AAAA,IACjC,EAAO,SAAI,CAAC,EAAE,OAAO;AAAA,MACpB,EAAE,QAAQ,WAAW,MAAM;AAAA,QAC1B,MAAM,KAAK,GAAG,YAAY,EAAE,MAAM,MAAM,aAAa,MAAM,CAAC;AAAA,SAC1D,IAAI,eAAe;AAAA,IACvB;AAAA;AAAA,EAED,MAAM,eAAe,IAAI,WAAW,SAAS,QAAQ,EAAE,YAAY,MAAM,CAAC;AAAA,EAC1E,EAAE,OAAO,YAAY,MAAM;AAAA,IAC1B,MAAM,KAAK,GAAG,YAAY,EAAE,MAAM,MAAM,aAAa,MAAM,CAAC;AAAA,KAC1D,IAAI,kBAAkB;AAAA,EACzB,MAAM,OAAO,MAAM;AAAA,IAClB,EAAE,OAAO;AAAA,IACT,aAAa,EAAE,KAAK;AAAA,IACpB,cAAc,EAAE,IAAI;AAAA,IACpB,IAAI,SAAS,UAAU;AAAA;AAAA,EAExB,MAAM,SAAS,aAAa,OAAO,KAClC,OAAO,WAAW;AAAA,IACjB,KAAK;AAAA,IAGL,MAAM,EAAE,MAAM,MAAM,MAAG;AAAA,MAAG;AAAA,KAAS;AAAA,IACnC,IAAI,EAAE;AAAA,MAAS,MAAM,EAAE;AAAA,IAIvB,IAAI,WAAW;AAAA,MAAe,MAAM,WAAW,GAAG;AAAA,IAClD,OAAO;AAAA,KAER,CAAC,UAAmB;AAAA,IACnB,KAAK;AAAA,IACL,MAAM,EAAE,WAAW,OAAO,IAAI,MAAM,qBAAqB,KAAK;AAAA,GAEhE;AAAA,EACA,IAAI;AAAA,IACH,MAAM,aAAa;AAAA,IAClB,OAAO,OAAO;AAAA,IACf,KAAK;AAAA,IACL,OAAO,MAAM,MAAG;AAAA,MAAG;AAAA,KAAS;AAAA,IAC5B,MAAM;AAAA;AAAA,EAEP,IAAI,SAAS,UAAU;AAAA,EACvB,MAAM,QAAQ,YAAY;AAAA,IACzB,IAAI;AAAA,MACH,MAAM,MAAM,KAAK,GAAG,YAAY;AAAA,cAC/B;AAAA,MACD,MAAM,aAAa,MAAM;AAAA;AAAA;AAAA,EAG3B,OAAO;AAAA,IACN,OAAO,aAAa;AAAA,IACpB;AAAA,IACA,OAAO,MAAM,MAAM,KAAK,GAAG,YAAY;AAAA,IACvC;AAAA,KACC,OAAO,eAAe;AAAA,EACxB;AAAA;AAQD,eAAsB,KAAK,CAAC,KAA8C;AAAA,EACzE,UAAU,KAAK,gBAAgB;AAAA,EAC/B,IAAI,QAAQ,MAAM,UAAU,GAAG;AAAA,EAC/B,IAAI,CAAC,OAAO;AAAA,IACX,MAAM,QAAQ,GAAG;AAAA,IACjB,QAAQ,MAAM,UAAU,GAAG;AAAA,EAC5B;AAAA,EACA,IAAI;AAAA,IACH,OAAO,MAAM,KAAK,KAAM,MAAuC,WAAW;AAAA,IACzE,OAAO,OAAO;AAAA,IACf,IAAI,CAAC,YAAY,KAAK;AAAA,MAAG,MAAM,OAAO,IAAI,MAAM,YAAY,KAAK;AAAA,IACjE,IAAI,IAAI,kBAAkB,QAAQ;AAAA,MACjC,MAAM,IAAI,gBACT,gBAAgB,IAAI,mDACnB,6DACA,6BACD,EAAE,MAAM,gBAAgB,MAAM,IAAI,MAAM,OAAO,MAAM,CACtD;AAAA,IACD;AAAA,IACA,MAAM,QAAQ,GAAG;AAAA,IACjB,MAAM,QAAS,MAAM,UAAU,GAAG;AAAA,IAClC,IAAI;AAAA,MACH,OAAO,MAAM,KAAK,KAAK,MAAM,WAAW;AAAA,MACvC,OAAO,OAAO;AAAA,MACf,MAAM,OAAO,IAAI,MAAM,YAAY,KAAK;AAAA;AAAA;AAAA;;;AC3KpC,SAAS,gBAGf,CAAC,SAA8C;AAAA,EAI/C,MAAM,MAAM,cACX,OAID;AAAA,EACA,OAAO;AAAA,IACN,MAAM,IAAI;AAAA,IACV,SAAS,MAAM,QAAQ,GAAG;AAAA,IAC1B,OAAO,MAAM,MAAM,GAAG;AAAA,IACtB,OAAO,MAAM,UAAU,GAAG;AAAA,EAC3B;AAAA;",
16
+ "debugId": "3F6CF388DE058E4464756E2164756E21",
17
+ "names": []
18
+ }
@@ -0,0 +1,4 @@
1
+ import type { SyncContext } from './context';
2
+ import type { ReindexReport } from './types';
3
+ export declare function reindex(ctx: SyncContext): Promise<ReindexReport>;
4
+ //# sourceMappingURL=reindex.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"reindex.d.ts","sourceRoot":"","sources":["../src/reindex.ts"],"names":[],"mappings":"AAGA,OAAO,KAAK,EAAO,WAAW,EAAE,MAAM,WAAW,CAAC;AAKlD,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,SAAS,CAAC;AA2E7C,wBAAsB,OAAO,CAAC,GAAG,EAAE,WAAW,GAAG,OAAO,CAAC,aAAa,CAAC,CActE"}
@@ -0,0 +1,8 @@
1
+ import type { SyncContext } from './context';
2
+ /**
3
+ * Refuses to do beside a running follower what would undo its work: a
4
+ * reindex removes what the follower has just indexed, and the follower,
5
+ * having handled that change, never sends it again.
6
+ */
7
+ export declare function checkIdle(ctx: SyncContext, doing: string): void;
8
+ //# sourceMappingURL=running.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"running.d.ts","sourceRoot":"","sources":["../src/running.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,WAAW,CAAC;AAG7C;;;;GAIG;AACH,wBAAgB,SAAS,CAAC,GAAG,EAAE,WAAW,EAAE,KAAK,EAAE,MAAM,GAAG,IAAI,CAO/D"}
@@ -0,0 +1,10 @@
1
+ import type { ResumeToken } from '@nxgt/mongo';
2
+ import type { SyncContext } from './context';
3
+ import type { SearchSyncState } from './types';
4
+ /** What the sync has recorded, or `undefined` before its first reindex. */
5
+ export declare function readState(ctx: SyncContext): Promise<SearchSyncState | undefined>;
6
+ /** Forgets everything recorded: the next `start` reindexes. */
7
+ export declare function clearState(ctx: SyncContext): Promise<void>;
8
+ /** Moves the resume point, and says a reindex finished when one did. */
9
+ export declare function saveState(ctx: SyncContext, resumeToken: ResumeToken, reindexed: boolean): Promise<void>;
10
+ //# sourceMappingURL=state.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"state.d.ts","sourceRoot":"","sources":["../src/state.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,aAAa,CAAC;AAC/C,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,WAAW,CAAC;AAE7C,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,SAAS,CAAC;AAE/C,2EAA2E;AAC3E,wBAAsB,SAAS,CAC9B,GAAG,EAAE,WAAW,GACd,OAAO,CAAC,eAAe,GAAG,SAAS,CAAC,CAMtC;AAED,+DAA+D;AAC/D,wBAAsB,UAAU,CAAC,GAAG,EAAE,WAAW,GAAG,OAAO,CAAC,IAAI,CAAC,CAMhE;AAED,wEAAwE;AACxE,wBAAsB,SAAS,CAC9B,GAAG,EAAE,WAAW,EAChB,WAAW,EAAE,WAAW,EACxB,SAAS,EAAE,OAAO,GAChB,OAAO,CAAC,IAAI,CAAC,CAiBf"}
@@ -0,0 +1,114 @@
1
+ import type { AnyIndexDefinition, DocumentOf as IndexDocumentOf, IdOf as IndexIdOf, TypedIndex } from '@nxgt/meilisearch';
2
+ import type { AnyCollectionDefinition, CloseReason, IdOf as CollectionIdOf, ReadDocumentOf, ResumeToken, TypedCollection } from '@nxgt/mongo';
3
+ /**
4
+ * What a document of the collection becomes in the index: a document, or
5
+ * `null` to keep it out — and take it out, if it was in.
6
+ */
7
+ export type Transform<C extends AnyCollectionDefinition, I extends AnyIndexDefinition> = (document: ReadDocumentOf<C>) => IndexDocumentOf<I> | null | Promise<IndexDocumentOf<I> | null>;
8
+ /** The index id of a collection's `_id`. */
9
+ export type ToIndexId<C extends AnyCollectionDefinition, I extends AnyIndexDefinition> = (id: CollectionIdOf<C>) => IndexIdOf<I>;
10
+ /**
11
+ * `toIndexId` is optional when the index's ids are strings — `String(_id)`
12
+ * is then the default — and required otherwise.
13
+ */
14
+ type IdOption<C extends AnyCollectionDefinition, I extends AnyIndexDefinition> = string extends IndexIdOf<I> ? {
15
+ /** The index id of a document. Default `String(_id)`. */
16
+ toIndexId?: ToIndexId<C, I>;
17
+ } : {
18
+ /** The index id of a document: the index's ids are not strings. */
19
+ toIndexId: ToIndexId<C, I>;
20
+ };
21
+ export type SearchSyncOptions<C extends AnyCollectionDefinition, I extends AnyIndexDefinition> = {
22
+ /** Where the documents are: a collection from `getCollection`. */
23
+ collection: TypedCollection<C>;
24
+ /** Where they go: an index from `bindIndex`. */
25
+ index: TypedIndex<I>;
26
+ /**
27
+ * A document as the index holds it. Its primary key must be the document's
28
+ * index id. `null` keeps a document out of the index. It runs for every
29
+ * change, so a document that stops qualifying is taken out.
30
+ */
31
+ transform: Transform<C, I>;
32
+ /**
33
+ * What this sync's state is recorded under, in `stateCollection`.
34
+ * Default `'<collection>:<index uid>'`. Two syncs with one name share a
35
+ * resume point, which is never what you want.
36
+ */
37
+ name?: string;
38
+ /** Where the resume point is kept, in the collection's database. Default `'nxgt_search_sync'`. */
39
+ stateCollection?: string;
40
+ /** How many changes are sent to Meilisearch at once. Default `500`. */
41
+ batchSize?: number;
42
+ /** How long a change waits for others before it is sent, in ms. Default `1000`. */
43
+ flushIntervalMs?: number;
44
+ /**
45
+ * How often, in ms, a sync with nothing to send records where the stream
46
+ * is anyway. Default `60000`. It is what keeps a quiet collection's
47
+ * resume point inside the server's history; one write per interval, and
48
+ * none while nothing moves at all.
49
+ */
50
+ positionIntervalMs?: number;
51
+ /**
52
+ * How many documents a reindex reads per page; above the collection's
53
+ * `maxPageSize`, lowered to it. Default `100`.
54
+ */
55
+ pageSize?: number;
56
+ /**
57
+ * What `start` does when the resume point is older than the server's
58
+ * history: `'reindex'` (default) reindexes and follows from there;
59
+ * `'fail'` throws a `SearchSyncError` with the code `HISTORY_LOST`.
60
+ */
61
+ onHistoryLost?: 'reindex' | 'fail';
62
+ } & IdOption<C, I>;
63
+ /** What a sync has recorded. */
64
+ export interface SearchSyncState {
65
+ /** The sync's name. */
66
+ _id: string;
67
+ /** Where following resumes. */
68
+ resumeToken: ResumeToken;
69
+ /** When the resume point last moved. */
70
+ updatedAt: Date;
71
+ /** When the last full reindex finished. */
72
+ reindexedAt: Date | undefined;
73
+ }
74
+ export interface ReindexReport {
75
+ /** Documents sent to the index. */
76
+ indexed: number;
77
+ /** Documents the transform kept out. */
78
+ skipped: number;
79
+ /** Documents that were in the index and are no longer wanted there. */
80
+ removed: number;
81
+ }
82
+ /** A sync that is following the collection. */
83
+ export interface RunningSearchSync extends AsyncDisposable {
84
+ /** Resolves once changes are being heard. */
85
+ readonly ready: Promise<void>;
86
+ /**
87
+ * Settles when it stops: resolves on `close()` or when the collection is
88
+ * dropped; rejects with a `SearchSyncError` when an error stopped it. What
89
+ * was not sent is sent again by the next `start`.
90
+ */
91
+ readonly closed: Promise<CloseReason>;
92
+ /** Sends what is waiting, and records how far it got. */
93
+ flush(): Promise<void>;
94
+ /** Sends what is waiting, then stops. */
95
+ close(): Promise<void>;
96
+ }
97
+ export interface SearchSync {
98
+ readonly name: string;
99
+ /**
100
+ * Sends every live document through the transform to the index, removes
101
+ * what the index holds and should not, and records where following
102
+ * resumes: from before the reindex began, so no change is missed.
103
+ */
104
+ reindex(): Promise<ReindexReport>;
105
+ /**
106
+ * Follows the collection's changes into the index, from where the last
107
+ * run stopped. A sync with nothing recorded reindexes first.
108
+ */
109
+ start(): Promise<RunningSearchSync>;
110
+ /** What is recorded, or `undefined` before the first reindex. */
111
+ state(): Promise<SearchSyncState | undefined>;
112
+ }
113
+ export {};
114
+ //# sourceMappingURL=types.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"types.d.ts","sourceRoot":"","sources":["../src/types.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EACX,kBAAkB,EAClB,UAAU,IAAI,eAAe,EAC7B,IAAI,IAAI,SAAS,EACjB,UAAU,EACV,MAAM,mBAAmB,CAAC;AAC3B,OAAO,KAAK,EACX,uBAAuB,EACvB,WAAW,EACX,IAAI,IAAI,cAAc,EACtB,cAAc,EACd,WAAW,EACX,eAAe,EACf,MAAM,aAAa,CAAC;AAErB;;;GAGG;AACH,MAAM,MAAM,SAAS,CACpB,CAAC,SAAS,uBAAuB,EACjC,CAAC,SAAS,kBAAkB,IACzB,CACH,QAAQ,EAAE,cAAc,CAAC,CAAC,CAAC,KACvB,eAAe,CAAC,CAAC,CAAC,GAAG,IAAI,GAAG,OAAO,CAAC,eAAe,CAAC,CAAC,CAAC,GAAG,IAAI,CAAC,CAAC;AAEpE,4CAA4C;AAC5C,MAAM,MAAM,SAAS,CACpB,CAAC,SAAS,uBAAuB,EACjC,CAAC,SAAS,kBAAkB,IACzB,CAAC,EAAE,EAAE,cAAc,CAAC,CAAC,CAAC,KAAK,SAAS,CAAC,CAAC,CAAC,CAAC;AAE5C;;;GAGG;AACH,KAAK,QAAQ,CAAC,CAAC,SAAS,uBAAuB,EAAE,CAAC,SAAS,kBAAkB,IAC5E,MAAM,SAAS,SAAS,CAAC,CAAC,CAAC,GACxB;IACA,yDAAyD;IACzD,SAAS,CAAC,EAAE,SAAS,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;CAC5B,GACA;IACA,mEAAmE;IACnE,SAAS,EAAE,SAAS,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;CAC3B,CAAC;AAEL,MAAM,MAAM,iBAAiB,CAC5B,CAAC,SAAS,uBAAuB,EACjC,CAAC,SAAS,kBAAkB,IACzB;IACH,kEAAkE;IAClE,UAAU,EAAE,eAAe,CAAC,CAAC,CAAC,CAAC;IAC/B,gDAAgD;IAChD,KAAK,EAAE,UAAU,CAAC,CAAC,CAAC,CAAC;IACrB;;;;OAIG;IACH,SAAS,EAAE,SAAS,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;IAC3B;;;;OAIG;IACH,IAAI,CAAC,EAAE,MAAM,CAAC;IACd,kGAAkG;IAClG,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB,uEAAuE;IACvE,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,mFAAmF;IACnF,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB;;;;;OAKG;IACH,kBAAkB,CAAC,EAAE,MAAM,CAAC;IAC5B;;;OAGG;IACH,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB;;;;OAIG;IACH,aAAa,CAAC,EAAE,SAAS,GAAG,MAAM,CAAC;CACnC,GAAG,QAAQ,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;AAEnB,gCAAgC;AAChC,MAAM,WAAW,eAAe;IAC/B,uBAAuB;IACvB,GAAG,EAAE,MAAM,CAAC;IACZ,+BAA+B;IAC/B,WAAW,EAAE,WAAW,CAAC;IACzB,wCAAwC;IACxC,SAAS,EAAE,IAAI,CAAC;IAChB,2CAA2C;IAC3C,WAAW,EAAE,IAAI,GAAG,SAAS,CAAC;CAC9B;AAED,MAAM,WAAW,aAAa;IAC7B,mCAAmC;IACnC,OAAO,EAAE,MAAM,CAAC;IAChB,wCAAwC;IACxC,OAAO,EAAE,MAAM,CAAC;IAChB,uEAAuE;IACvE,OAAO,EAAE,MAAM,CAAC;CAChB;AAED,+CAA+C;AAC/C,MAAM,WAAW,iBAAkB,SAAQ,eAAe;IACzD,6CAA6C;IAC7C,QAAQ,CAAC,KAAK,EAAE,OAAO,CAAC,IAAI,CAAC,CAAC;IAC9B;;;;OAIG;IACH,QAAQ,CAAC,MAAM,EAAE,OAAO,CAAC,WAAW,CAAC,CAAC;IACtC,yDAAyD;IACzD,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC,CAAC;IACvB,yCAAyC;IACzC,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC,CAAC;CACvB;AAED,MAAM,WAAW,UAAU;IAC1B,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB;;;;OAIG;IACH,OAAO,IAAI,OAAO,CAAC,aAAa,CAAC,CAAC;IAClC;;;OAGG;IACH,KAAK,IAAI,OAAO,CAAC,iBAAiB,CAAC,CAAC;IACpC,iEAAiE;IACjE,KAAK,IAAI,OAAO,CAAC,eAAe,GAAG,SAAS,CAAC,CAAC;CAC9C"}
package/package.json ADDED
@@ -0,0 +1,66 @@
1
+ {
2
+ "name": "@nxgt/mongo-meilisearch",
3
+ "version": "0.1.0",
4
+ "description": "Keeps a Meilisearch index in step with a MongoDB collection: a typed transform, a full reindex, and a change stream that resumes where it stopped",
5
+ "license": "MIT",
6
+ "type": "module",
7
+ "main": "./dist/index.js",
8
+ "types": "./dist/index.d.ts",
9
+ "files": [
10
+ "dist",
11
+ "README.md",
12
+ "package.json",
13
+ "LICENSE"
14
+ ],
15
+ "exports": {
16
+ ".": {
17
+ "types": "./dist/index.d.ts",
18
+ "import": "./dist/index.js",
19
+ "default": "./dist/index.js"
20
+ },
21
+ "./package.json": "./package.json"
22
+ },
23
+ "keywords": [
24
+ "mongodb",
25
+ "meilisearch",
26
+ "search",
27
+ "sync",
28
+ "change-streams",
29
+ "typescript"
30
+ ],
31
+ "repository": {
32
+ "type": "git",
33
+ "url": "git+https://github.com/softistx/nxgt-data.git",
34
+ "directory": "packages/mongo-meilisearch"
35
+ },
36
+ "publishConfig": {
37
+ "registry": "https://registry.npmjs.org",
38
+ "access": "public"
39
+ },
40
+ "scripts": {
41
+ "build": "bun run ../../build.ts",
42
+ "test": "bun run ../../scripts/meilisearch.ts && bun test --timeout 30000 src",
43
+ "typecheck": "tsc --noEmit"
44
+ },
45
+ "nxgt": {
46
+ "entrypoints": [
47
+ "src/index.ts"
48
+ ]
49
+ },
50
+ "devDependencies": {
51
+ "@nxgt/meilisearch": "^0.1.0",
52
+ "@nxgt/mongo": "^0.10.0",
53
+ "@types/bun": "^1.4.0",
54
+ "meilisearch": "0.62.0",
55
+ "mongodb": "7.6.0",
56
+ "mongodb-memory-server-core": "11.2.0",
57
+ "zod": "4.6.5"
58
+ },
59
+ "peerDependencies": {
60
+ "@nxgt/meilisearch": "^0.1.0",
61
+ "@nxgt/mongo": "^0.10.0",
62
+ "meilisearch": ">=0.62.0 <1",
63
+ "mongodb": ">=7.0.0 <8",
64
+ "typescript": "^6.0.3"
65
+ }
66
+ }