@nxgt/mongo-meilisearch 0.4.3 → 0.5.0

package/README.md

@@ -1,9 +1,49 @@
 # @nxgt/mongo-meilisearch
 
-Keeps a [Meilisearch](https://www.meilisearch.com) index in step with a
-MongoDB collection: a typed transform from one to the other, a full reindex,
-and a change stream that picks up where it stopped, its position kept in
-MongoDB.
+Keeps [Meilisearch](https://www.meilisearch.com) indexes in step with MongoDB
+collections: a typed transform from one to the other, a full reindex, and a
+change stream that picks up where it stopped, its position kept in MongoDB.
+`createSearchSyncs` does it for every collection an `openMongo` wires, from one
+config and one `start()`; `createSearchSync` does it for one collection.
+
+```ts
+import { bindIndex } from '@nxgt/meilisearch';
+import { defineMongo, openMongo } from '@nxgt/mongo';
+import { createSearchSyncs } from '@nxgt/mongo-meilisearch';
+
+const mongo = await openMongo(defineMongo({ uri, collections })); // @nxgt/mongo
+
+const search = createSearchSyncs(mongo, {
+	articles: {
+		index: bindIndex(meili, articleIndex),        // @nxgt/meilisearch
+		// `article` is typed by the collection the key names, and the result
+		// by the index this entry carries. `null` keeps a document out.
+		transform: (article) =>
+			article.draft ? null : { id: String(article._id), title: article.title },
+	},
+	authors: {
+		index: bindIndex(meili, authorIndex),
+		transform: (author) => ({ id: String(author._id), name: author.name }),
+	},
+});
+
+await search.syncIndexes();             // every index, with its settings
+const running = await search.start();   // reindexes the first time, then follows
+running.failed.catch(exit);             // await it or catch it — see Traps
+// …
+await running.close();                  // flushes and stops every sync
+```
+
+Each collection's sync is the `createSearchSync` below, unchanged. What
+`createSearchSyncs` adds is the wiring: the collections come from the Mongo,
+under the key they are exported under (the same one `mongo.db.articles`
+answers to), so nothing is named twice, and the syncs are started and stopped
+together. A key the Mongo wires no collection for does not compile, and is
+refused by name at run time too. `createSearchSyncs` sends nothing:
+`syncIndexes`, `reindexAll` and `start` do. See [Several
+collections](#several-collections-createsearchsyncs) for its API.
+
+## One collection at a time
 
 ```ts
 import { createSearchSync } from '@nxgt/mongo-meilisearch';
@@ -359,10 +399,76 @@ before it could be read),
 `cause`. Its constructor takes `(message, options: SearchSyncErrorOptions)`,
 that is `{ code, sync, cause?, holder?, expiresAt? }`; both types are exported.
 
+## Several collections: `createSearchSyncs`
+
+```ts
+function createSearchSyncs<C, const I extends IndexMap<I>>(
+	mongo: Mongo<C>,
+	config: SearchSyncsConfig<C, I>,
+): SearchSyncs<I>;
+```
+
+`mongo` is what `@nxgt/mongo`'s `openMongo` returned, and `config` is one entry
+per collection, under the key the Mongo wires it under. An entry is everything
+`createSearchSync` takes **except `collection`**, which the Mongo already
+holds: `index`, `transform`, `toIndexId` (optional while the index's ids are
+strings, required otherwise), and optionally `name`, `stateCollection`,
+`batchSize`, `flushIntervalMs`, `positionIntervalMs`, `leaseMs`, `pageSize` and
+`onHistoryLost`.
+
+`SearchSyncs<I>`:
+
+| Member | |
+| --- | --- |
+| `syncs: { [K in keyof I]: SearchSync }` | each sync as `createSearchSync` built it, so anything this does not wrap is still reachable |
+| `state()` | where each sync stands, under its key; `undefined` for one that never reindexed |
+| `syncIndexes(options?)` | every index created and its settings applied where they differ: `@nxgt/meilisearch`'s `SyncReport` per key, `dryRun` and `wait` passed through. One after another; the first that throws stops the rest |
+| `reindexAll()` | a `ReindexReport` per key. One after another, and the first that throws stops the rest |
+| `start()` | every sync, resolving once they are all hearing changes; one that fails to start closes the ones already started |
+
+`RunningSearchSyncs<I>`, also `AsyncDisposable`:
+
+| Member | |
+| --- | --- |
+| `running: { [K in keyof I]: RunningSearchSync }` | each running sync, so its `ready`, its `closed` and its own `flush` are still reachable |
+| `failed: Promise<never>` | rejects with the **first** sync that stops on an error, and never settles otherwise. A later failure is `close()`'s to report |
+| `flush()` | sends what every sync holds, and records where each one is. Stops at the first that fails |
+| `close()` | flushes, then stops every sync. Idempotent |
+
+The types: `SearchSyncsConfig<C, I>` (a `SearchSyncEntry` per key of
+`SoleCollections<C>`, and for a key the Mongo wires no collection for, a
+refusal string that names it in place of the entry), `SearchSyncEntry<Col, I>`
+(`Omit<SearchSyncOptions<Col, I>, 'collection'>`), `SoleCollections<C>` (what a
+Mongo's sole database holds, by the name each definition is exported under;
+`never` when it holds several), `IndexMap<I>` (the index definitions a config
+names, from which `I` is inferred, which is what lets a `transform` be written
+inline) and `ByKey<S, T>` (`{ readonly [K in keyof S]: T }`, the shape of every
+per-key result).
+
+What it does not do:
+
+- **Follow several databases.** A Mongo that holds several gives `never` for
+  its keys, as `mongo.db` itself does, and `createSearchSyncs` throws naming
+  them. Build one `createSearchSyncs` per database, from a Mongo that wires
+  that database alone.
+- **Coordinate processes on its own.** Each entry's sync takes the lease on its
+  name when it starts: a second process starting the same syncs is refused
+  with `RUNNING` for the first sync whose name is held, and starts nothing.
+- **Own the Mongo.** Closing the search syncs stops them and nothing else:
+  the clients, the databases and the collections are the Mongo's, and
+  `mongo.close()` is still the caller's to make.
+
+The [guides](docs/guide/search-syncs/wiring.md) cover the config and the
+[lifecycle](docs/guide/search-syncs/lifecycle.md) in full.
+
 ## What does not compile
 
 Each is a `@ts-expect-error` case in this package's type tests.
 
+- A key the Mongo wires no collection for (the message names it), a key that
+  is a member of the driver's `Db`, such as `command`, and a Mongo that holds
+  more than one database: all `createSearchSyncs`'s. Its entries are
+  checked as below, the `transform` typed by the collection the key names.
 - A transform that reads a field the collection's schema does not have.
 - A transform that leaves out a field the index's document has, or gives an
   id of the wrong type (an `ObjectId` for a string id). A field the document
@@ -420,9 +526,48 @@ Each is a `@ts-expect-error` case in this package's type tests.
   `documents.delete` and `tasks.get`, plus `indexes.create` unless the index
   already exists — the first write to an index that does not creates it.
 
+Of `createSearchSyncs`:
+
+- **A failure nobody handles is silent, not loud.** A sync that stops on an
+  error rejects `failed`, and `createSearchSyncs` takes that rejection itself,
+  as it takes each sync's own `closed`, so nothing ends the process and nothing
+  is printed. That index simply stops updating. `failed` is the only place the
+  failure surfaces, so handle it; unhandled, the first sign is stale search
+  results.
+- **`failed` never resolves.** A clean stop is not an event to wait for, so
+  `await running.failed` after `close()` waits forever. It is for `catch`, or
+  for racing against your own shutdown.
+- **A dropped collection leaves `failed` silent.** An invalidated stream is a
+  clean stop (`closed` *resolves* with `'invalidated'`), and `failed` forwards
+  failures only. That sync is dead and nothing settles. Watch
+  `running.running.<key>.closed` if a drop has to be noticed.
+- **`close()` does not report the failure `failed` already carried.** That one
+  sync rejects its own `close` with what it stopped on, and it is swallowed, so
+  a caller does not hear it twice. Every *other* failure is thrown, including a
+  **second** sync that fell over after `failed` settled, and anything that goes
+  wrong *while* closing. If more than one throws, `close` reports the first.
+- **A failed `start()` leaves nothing running.** The syncs already started are
+  closed before the error comes back.
+- **`reindexAll()` and `flush()` stop at the first failure.** The keys already
+  done are lost with the rejection of `reindexAll`, because a reindex removes
+  what a collection no longer gives and a half-finished run is not a state to
+  keep going from: read the error, fix, run it again, since the reindexes that
+  succeeded are idempotent. A sync that has already fallen over rejects
+  `flush` at once and the ones after it are neither sent nor recorded, which is
+  safe on restart but not free. `close()` keeps going past a sync that fails.
+- **Two `createSearchSyncs` over one collection are two syncs over one
+  index.** The default `name` is `<collection>:<index uid>`, so two built from
+  the same config share their resume point and their lease: the second is
+  refused with `RUNNING` while the first runs. Give `name` if you mean them to
+  differ.
+
 ## Documentation
 
 - [Guide index](docs/README.md) — every page, and when to read it.
+- [Wiring it over a Mongo](docs/guide/search-syncs/wiring.md) — the
+  `createSearchSyncs` config, its keys, and every option an entry takes.
+- [The search syncs' lifecycle](docs/guide/search-syncs/lifecycle.md) —
+  `syncIndexes`, `reindexAll`, `start`, `failed`, `flush` and `close`.
 - [The sync's lifecycle](docs/guide/sync-lifecycle.md) — the two definitions,
   the transform, and every option with its default.
 - [Reindexing](docs/guide/reindex.md) — the full fill, and what it removes.

package/dist/index.d.ts

@@ -1,4 +1,6 @@
-export { createSearchSync } from './create-search-sync';
-export { SearchSyncError, type SearchSyncErrorCode, type SearchSyncErrorOptions, } from './errors';
-export type { ReindexReport, RunningSearchSync, SearchSync, SearchSyncOptions, SearchSyncState, ToIndexId, Transform, } from './types';
+export { createSearchSync } from './sync/create-search-sync';
+export { SearchSyncError, type SearchSyncErrorCode, type SearchSyncErrorOptions, } from './sync/errors';
+export type { ReindexReport, RunningSearchSync, SearchSync, SearchSyncOptions, SearchSyncState, ToIndexId, Transform, } from './sync/types';
+export { createSearchSyncs } from './syncs/create-search-syncs';
+export type { ByKey, IndexMap, RunningSearchSyncs, SearchSyncEntry, SearchSyncs, SearchSyncsConfig, SoleCollections, } from './syncs/types';
 //# sourceMappingURL=index.d.ts.map

package/dist/index.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,gBAAgB,EAAE,MAAM,sBAAsB,CAAC;AACxD,OAAO,EACN,eAAe,EACf,KAAK,mBAAmB,EACxB,KAAK,sBAAsB,GAC3B,MAAM,UAAU,CAAC;AAClB,YAAY,EACX,aAAa,EACb,iBAAiB,EACjB,UAAU,EACV,iBAAiB,EACjB,eAAe,EACf,SAAS,EACT,SAAS,GACT,MAAM,SAAS,CAAC"}
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,gBAAgB,EAAE,MAAM,2BAA2B,CAAC;AAC7D,OAAO,EACN,eAAe,EACf,KAAK,mBAAmB,EACxB,KAAK,sBAAsB,GAC3B,MAAM,eAAe,CAAC;AACvB,YAAY,EACX,aAAa,EACb,iBAAiB,EACjB,UAAU,EACV,iBAAiB,EACjB,eAAe,EACf,SAAS,EACT,SAAS,GACT,MAAM,cAAc,CAAC;AACtB,OAAO,EAAE,iBAAiB,EAAE,MAAM,6BAA6B,CAAC;AAChE,YAAY,EACX,KAAK,EACL,QAAQ,EACR,kBAAkB,EAClB,eAAe,EACf,WAAW,EACX,iBAAiB,EACjB,eAAe,GACf,MAAM,eAAe,CAAC"}

package/dist/index.js

@@ -1,49 +1,7 @@
-// src/context.ts
-var DEFAULT_STATE_COLLECTION = "nxgt_search_sync";
-function positive(option, value, fallback) {
-  const resolved = value ?? fallback;
-  if (!Number.isInteger(resolved) || resolved < 1) {
-    throw new TypeError(`createSearchSync: ${option} must be a whole number above 0, not ${String(value)}`);
-  }
-  return resolved;
-}
-function createContext(options) {
-  const { collection, index } = options;
-  if (typeof options.transform !== "function") {
-    throw new TypeError("createSearchSync: transform must be a function");
-  }
-  const name = options.name ?? `${collection.collectionName}:${index.uid}`;
-  if (name === "") {
-    throw new TypeError("createSearchSync: name must not be empty");
-  }
-  const flushIntervalMs = options.flushIntervalMs ?? 1000;
-  if (!Number.isInteger(flushIntervalMs) || flushIntervalMs < 0) {
-    throw new TypeError(`createSearchSync: flushIntervalMs must be a whole number of milliseconds, not ${String(options.flushIntervalMs)}`);
-  }
-  const stateCollection = options.stateCollection ?? DEFAULT_STATE_COLLECTION;
-  return {
-    name,
-    collection,
-    index,
-    primaryKey: index.definition.primaryKey,
-    state: collection.db.collection(stateCollection),
-    leases: collection.db.collection(stateCollection),
-    leaseMs: positive("leaseMs", options.leaseMs, 30000),
-    batchSize: positive("batchSize", options.batchSize, 500),
-    positionIntervalMs: positive("positionIntervalMs", options.positionIntervalMs, 60000),
-    flushIntervalMs,
-    pageSize: positive("pageSize", options.pageSize, 100),
-    onHistoryLost: options.onHistoryLost ?? "reindex",
-    transform: options.transform,
-    toIndexId: options.toIndexId ?? String,
-    follower: { running: false }
-  };
-}
-
-// src/reindex.ts
+// src/run/reindex.ts
 import { MeilisearchApiError } from "meilisearch";
 
-// src/batch.ts
+// src/sync/batch.ts
 var WAIT = { wait: { timeout: 120000 } };
 function chunks(items, size) {
   const out = [];
@@ -75,7 +33,7 @@ async function send(ctx, entries) {
   await deleteIds(ctx, ids);
 }
 
-// src/errors.ts
+// src/sync/errors.ts
 class SearchSyncError extends Error {
   name = "SearchSyncError";
   code;
@@ -101,7 +59,7 @@ function failed(sync, doing, error) {
   });
 }
 
-// src/documents.ts
+// src/sync/documents.ts
 var keyOf = (id) => `${typeof id}:${String(id)}`;
 var isRecord = (value) => typeof value === "object" && value !== null && !Array.isArray(value);
 function describe(value) {
@@ -129,7 +87,44 @@ async function entryOf(ctx, mongoId, document) {
   return { kind: "upsert", key, document: indexed };
 }
 
-// src/lease.ts
+// src/sync/running.ts
+function checkIdle(ctx, doing) {
+  if (!ctx.follower.running)
+    return;
+  throw new SearchSyncError(`Search sync "${ctx.name}" is already following changes in this ` + `process: close it before you ${doing}.`, { code: "RUNNING", sync: ctx.name });
+}
+
+// src/sync/state.ts
+async function readState(ctx) {
+  try {
+    return await ctx.state.findOne({ _id: ctx.name }) ?? undefined;
+  } catch (error) {
+    throw failed(ctx.name, "reading its resume point", error);
+  }
+}
+async function clearState(ctx) {
+  try {
+    await ctx.state.deleteOne({ _id: ctx.name });
+  } catch (error) {
+    throw failed(ctx.name, "forgetting its resume point", error);
+  }
+}
+async function saveState(ctx, resumeToken, reindexed) {
+  const now = new Date;
+  try {
+    await ctx.state.updateOne({ _id: ctx.name }, {
+      $set: {
+        resumeToken,
+        updatedAt: now,
+        ...reindexed ? { reindexedAt: now } : {}
+      }
+    }, { upsert: true });
+  } catch (error) {
+    throw failed(ctx.name, "recording its resume point", error);
+  }
+}
+
+// src/run/lease.ts
 import { hostname } from "node:os";
 import { ObjectId } from "mongodb";
 var idOf = (ctx) => ({ lease: ctx.name });
@@ -239,44 +234,7 @@ async function withLease(ctx, doing, fn) {
   }
 }
 
-// src/running.ts
-function checkIdle(ctx, doing) {
-  if (!ctx.follower.running)
-    return;
-  throw new SearchSyncError(`Search sync "${ctx.name}" is already following changes in this ` + `process: close it before you ${doing}.`, { code: "RUNNING", sync: ctx.name });
-}
-
-// src/state.ts
-async function readState(ctx) {
-  try {
-    return await ctx.state.findOne({ _id: ctx.name }) ?? undefined;
-  } catch (error) {
-    throw failed(ctx.name, "reading its resume point", error);
-  }
-}
-async function clearState(ctx) {
-  try {
-    await ctx.state.deleteOne({ _id: ctx.name });
-  } catch (error) {
-    throw failed(ctx.name, "forgetting its resume point", error);
-  }
-}
-async function saveState(ctx, resumeToken, reindexed) {
-  const now = new Date;
-  try {
-    await ctx.state.updateOne({ _id: ctx.name }, {
-      $set: {
-        resumeToken,
-        updatedAt: now,
-        ...reindexed ? { reindexedAt: now } : {}
-      }
-    }, { upsert: true });
-  } catch (error) {
-    throw failed(ctx.name, "recording its resume point", error);
-  }
-}
-
-// src/reindex.ts
+// src/run/reindex.ts
 async function currentToken(ctx) {
   const stream = ctx.collection.raw.watch([], { maxAwaitTimeMS: 1 });
   try {
@@ -355,7 +313,49 @@ async function reindexHeld(ctx, lease) {
   }
 }
 
-// src/follow.ts
+// src/sync/context.ts
+var DEFAULT_STATE_COLLECTION = "nxgt_search_sync";
+function positive(option, value, fallback) {
+  const resolved = value ?? fallback;
+  if (!Number.isInteger(resolved) || resolved < 1) {
+    throw new TypeError(`createSearchSync: ${option} must be a whole number above 0, not ${String(value)}`);
+  }
+  return resolved;
+}
+function createContext(options) {
+  const { collection, index } = options;
+  if (typeof options.transform !== "function") {
+    throw new TypeError("createSearchSync: transform must be a function");
+  }
+  const name = options.name ?? `${collection.collectionName}:${index.uid}`;
+  if (name === "") {
+    throw new TypeError("createSearchSync: name must not be empty");
+  }
+  const flushIntervalMs = options.flushIntervalMs ?? 1000;
+  if (!Number.isInteger(flushIntervalMs) || flushIntervalMs < 0) {
+    throw new TypeError(`createSearchSync: flushIntervalMs must be a whole number of milliseconds, not ${String(options.flushIntervalMs)}`);
+  }
+  const stateCollection = options.stateCollection ?? DEFAULT_STATE_COLLECTION;
+  return {
+    name,
+    collection,
+    index,
+    primaryKey: index.definition.primaryKey,
+    state: collection.db.collection(stateCollection),
+    leases: collection.db.collection(stateCollection),
+    leaseMs: positive("leaseMs", options.leaseMs, 30000),
+    batchSize: positive("batchSize", options.batchSize, 500),
+    positionIntervalMs: positive("positionIntervalMs", options.positionIntervalMs, 60000),
+    flushIntervalMs,
+    pageSize: positive("pageSize", options.pageSize, 100),
+    onHistoryLost: options.onHistoryLost ?? "reindex",
+    transform: options.transform,
+    toIndexId: options.toIndexId ?? String,
+    follower: { running: false }
+  };
+}
+
+// src/run/follow.ts
 var noop = () => {
   return;
 };
@@ -492,7 +492,7 @@ async function open(ctx, token, lease) {
   };
 }
 
-// src/start.ts
+// src/sync/start.ts
 function historyLost(error) {
   const code = error?.serverCode;
   return code === 286 || code === 280;
@@ -541,7 +541,7 @@ async function startHeld(ctx, lease) {
   }
 }
 
-// src/create-search-sync.ts
+// src/sync/create-search-sync.ts
 function createSearchSync(options) {
   const ctx = createContext(options);
   return {
@@ -551,10 +551,121 @@ function createSearchSync(options) {
     state: () => readState(ctx)
   };
 }
+// src/syncs/create-search-syncs.ts
+function soleScope(mongo) {
+  const names = Object.keys(mongo.databases);
+  if (names.length !== 1) {
+    throw new TypeError("createSearchSyncs: this Mongo holds " + `${names.length} databases (${names.join(", ")}), and ` + "createSearchSyncs follows the collections of one. Build one " + "`createSearchSyncs` per database, from a Mongo that wires that " + "database alone");
+  }
+  return mongo.db;
+}
+function collectionAt(scope, key) {
+  const collection = scope[key];
+  const definition = typeof collection === "object" && collection !== null ? collection.definition : undefined;
+  if (typeof definition !== "object" || definition === null || !("schema" in definition)) {
+    throw new TypeError(`createSearchSyncs: this Mongo wires no collection called "${key}"`);
+  }
+  return collection;
+}
+function createSearchSyncs(mongo, config) {
+  const scope = soleScope(mongo);
+  const keys = Object.keys(config);
+  const syncs = {};
+  const indexes = {};
+  for (const key of keys) {
+    const entry = config[key];
+    indexes[key] = entry["index"];
+    syncs[key] = createSearchSync({
+      ...entry,
+      collection: collectionAt(scope, key)
+    });
+  }
+  return {
+    syncs,
+    async state() {
+      const state = {};
+      for (const key of keys)
+        state[key] = await syncs[key].state();
+      return state;
+    },
+    async syncIndexes(options) {
+      const reports = {};
+      for (const key of keys)
+        reports[key] = await indexes[key].sync(options);
+      return reports;
+    },
+    async reindexAll() {
+      const reports = {};
+      for (const key of keys)
+        reports[key] = await syncs[key].reindex();
+      return reports;
+    },
+    async start() {
+      const running = {};
+      const started = [];
+      for (const key of keys) {
+        let one;
+        try {
+          one = await syncs[key].start();
+        } catch (error) {
+          await Promise.allSettled(started.map((it) => it.close()));
+          throw error;
+        }
+        one.closed.catch(() => {
+          return;
+        });
+        running[key] = one;
+        started.push(one);
+      }
+      return runningSyncs(running, started);
+    }
+  };
+}
+function runningSyncs(running, started) {
+  let closing;
+  let reported;
+  const failed = Promise.race(started.map((one) => one.closed.then(() => new Promise(() => {}), (reason) => {
+    reported ??= one;
+    throw reason;
+  })));
+  failed.catch(() => {
+    return;
+  });
+  const close = () => {
+    closing ??= (async () => {
+      let first;
+      let caught = false;
+      for (const one of started) {
+        try {
+          await one.close();
+        } catch (error) {
+          if (one === reported || caught)
+            continue;
+          first = error;
+          caught = true;
+        }
+      }
+      if (caught)
+        throw first;
+    })();
+    return closing;
+  };
+  return {
+    running,
+    failed,
+    async flush() {
+      for (const one of started)
+        await one.flush();
+    },
+    close,
+    [Symbol.asyncDispose]: close
+  };
+}
 export {
   SearchSyncError,
-  createSearchSync
+  createSearchSync,
+  createSearchSyncs
 };
 
-//# debugId=CB9FE6862D67E10C64756E2164756E21
+//# debugId=C960A5123B6682B664756E2164756E21
 //# sourceMappingURL=index.js.map