@ahrowe/mongo 0.0.1

package/README.md

@@ -0,0 +1,184 @@
+# @ahrowe/mongo
+
+A typed MongoDB CRUD service factory: `connect()` once, then `createService<T>(...)`
+gives you a collection wrapper with built-in transactions, an event bus
+(`created`/`updated`/`removed`), and optional document validation on writes.
+
+## Install
+
+```bash
+pnpm add @ahrowe/mongo
+```
+
+## Usage
+
+```ts
+import db from '@ahrowe/mongo';
+
+const { createService } = db.connect(process.env.MONGO_DB_URI!, 'myDb');
+
+type User = { _id: string; email: string; createdOn?: Date; updatedOn?: Date };
+const users = createService<User>('users');
+
+const user = await users.insert({ email: 'a@b.com' });
+const found = await users.findOne({ _id: user._id });
+await users.updateOne({ _id: user._id }, { $set: { email: 'c@d.com' } });
+```
+
+### Validation on writes
+
+Pass a validator function as the second argument to `createService`. It's called on
+every insert and on the document resulting from an update; a non-passing result
+throws. The validator contract is `(entity: T) => { error?: unknown }` — plug in
+whatever validation library you like, or none at all. On a non-empty `error`, that
+value is thrown as-is.
+
+With Zod:
+
+```ts
+import { z } from 'zod';
+
+const userSchema = z.object({ _id: z.string(), email: z.string().email() });
+const users = createService<User>('users', (entity) => userSchema.safeParse(entity));
+```
+
+Or a plain TS guard with no dependency at all:
+
+```ts
+const users = createService<User>('users', (entity) => (
+  entity.email.includes('@') ? {} : { error: new Error('email must contain @') }
+));
+```
+
+Pass `{ skipValidation: true }` on an individual call to bypass it.
+
+### Events (transactional outbox)
+
+```ts
+users.on('created', ({ doc, meta }) => { /* ... */ });
+users.on('updated', ({ prevDoc, doc, meta }) => { /* ... */ });
+users.on('removed', ({ doc, meta }) => { /* ... */ });
+```
+
+Writes don't dispatch events directly. Every `insert`/`updateOne`/`updateMany`/
+`removeOne`/`removeMany` that actually changes a document writes one row per
+affected document to an internal `outbox` collection, **in the same transaction**
+as the data write. A separate relay process claims and dispatches those rows to
+your `.on(...)` listeners. This makes delivery durable: events survive a crash
+between the write and the listener running, and are never lost or duplicated
+across multiple pods racing the same row (claims are leased atomically).
+`updateOne`/`updateMany` only enqueue `'updated'` for documents that actually
+changed.
+
+`updateMany`/`removeMany` always process every matched document individually —
+one atomic operation per document, all inside one transaction, so the whole
+batch still rolls back together on failure — so each outbox row's
+`{ prevDoc, doc }` pair is accurate and no document is missed even if the
+matched set changes mid-operation. Run the relay to drain the outbox:
+
+```ts
+import { startOutboxRelay } from '@ahrowe/mongo';
+
+const relay = startOutboxRelay({ instanceId: process.env.HOSTNAME ?? 'local' });
+// relay.stop() on shutdown
+```
+
+The reliable per-document path has **no size cap by default** — but a single
+transaction processing a very large matched set risks hitting MongoDB's
+transaction lifetime limit. Pass `{ maxBatchSize }` (per-call, or as a
+`createService` option for a service-wide default) to enforce one and fail fast
+instead:
+
+```ts
+await users.updateMany({}, { $set: { plan: 'pro' } }, { maxBatchSize: 500 });
+```
+
+Past that limit it throws, pointing you at `bulkWrite` for larger jobs (no
+transaction/validation/event guarantees, but no cap either).
+
+Pass `{ meta: {...} }` on a call to thread arbitrary data through to listeners,
+so a specific handler can decide to skip its own side effect without anyone else
+missing the event:
+
+```ts
+await users.updateOne({ _id }, { $set: { email: 'c@d.com' } }, { meta: { dontRegenPdf: true } });
+
+users.on('updated', ({ doc, meta }) => {
+  if (meta?.dontRegenPdf) return;
+  regenerateInvoicePdf(doc);
+});
+```
+
+Infra collections with no listeners (sequences, the outbox itself, etc.) can
+skip outbox writes entirely by passing `{ emitOutboxEvents: false }` to
+`createService` — writes take a leaner fast path with no pre-reads or
+transaction-wrapped outbox insert.
+
+If a listener needs request/actor context (e.g. an audit log reading from
+`AsyncLocalStorage`), register a provider once at startup. It's called
+**synchronously at write time** and the captured value is stored on the outbox
+row, so it's still available to the listener even after a process restart:
+
+```ts
+import { setOutboxContextProvider } from '@ahrowe/mongo';
+
+setOutboxContextProvider(() => myAsyncLocalStorage.getStore());
+```
+
+### Transactions
+
+```ts
+import db from '@ahrowe/mongo';
+
+const session = await db.startSession();
+await session.withTransaction(async () => {
+  await users.insert({ email: 'a@b.com' }, { session });
+  await otherService.updateOne({ _id }, { $set: { count: 1 } }, { session });
+});
+```
+
+`insert`/`updateOne`/`updateMany` automatically wrap themselves in a transaction when
+no `session` is passed.
+
+### Connection health
+
+```ts
+const { createService, on } = db.connect(process.env.MONGO_DB_URI!, 'myDb');
+
+on('error', ({ source, error }) => { /* 'primary' | 'read' */ });
+on('close', ({ source, error }) => { /* ... */ });
+```
+
+`connect()` wires up `error`/`close` handlers on both the primary and
+secondary-preferred read clients internally (so a connection issue can't crash the
+process), and re-emits them on a connection-level bus you can subscribe to for your
+own alerting.
+
+### Shutdown
+
+```ts
+import db from '@ahrowe/mongo';
+
+await db.disconnect();
+```
+
+`disconnect()` closes both MongoDB clients. Outbox rows already committed are
+durable in MongoDB regardless — call this on graceful shutdown (e.g. on
+`SIGTERM`); if you also run `startOutboxRelay` in-process, call `relay.stop()`
+first so it releases any leases it's holding.
+
+## API
+
+- `connect(connectionString, databaseName?) → { createService, on }`
+- `createService<T>(collectionName, validate?, { addCreatedOnField?, addUpdatedOnField?, maxLimit?, maxBatchSize?, emitOutboxEvents? }) → DbService<T>`
+- `withSession(cb)`, `startSession(options?)`, `disconnect()`
+- `setOutboxContextProvider(fn)` — capture request/actor context at write time for outbox rows
+- `startOutboxRelay({ instanceId, pollIntervalMs?, batchSize?, leaseMs?, maxAttempts?, autoStart? }) → { stop, tick, drain }`
+- `getServiceBus(collectionName)`, `getRegisteredCollections()`, `getOutboxCollection()` — relay/advanced internals
+- `OutboxOp`, `OutboxStatus`, `OutboxRow`, `OUTBOX_COLLECTION` — outbox row shape, for tooling/inspection
+- `DbService<T>` methods: `find`, `findOne`, `findCursor`, `insert`, `updateOne`,
+  `updateMany`, `removeOne`, `removeMany`, `count`, `exists`, `aggregate`, `distinct`,
+  `createIndex`, `bulkWrite`, `on`, `onPropertiesUpdated`, `eventBus`, `generateId`, `name`
+
+See [docs/CLAUDE.md](docs/CLAUDE.md) for gotchas (upserts are rejected, `findOne`
+strictness, the opt-in `maxLimit`/`maxBatchSize` caps, `bulkWrite` safety gate).

package/dist/batchify.d.ts

@@ -0,0 +1,36 @@
+interface BatchifyOptions<T> {
+    /**
+     * Returns the next batch of data, or an array containing all data (sliced into
+     * batches automatically). When a function: receives the requested batch size
+     * and the number of items already processed.
+     */
+    dataDelivery?: T[] | ((requestedBatchSize: number, alreadyProcessed: number) => T[] | Promise<T[]>);
+    /** Number of items fetched per `dataDelivery` call. Default 1000. */
+    batchSize?: number;
+    /**
+     * Number of items from the current batch processed concurrently. Defaults to
+     * `batchSize` (the whole batch at once). Set to `1` for strictly sequential
+     * processing — required when items share a MongoDB session/transaction, which
+     * doesn't allow concurrent operations.
+     */
+    concurrency?: number;
+    /** Function to run for each item. Receives the item and its global index. */
+    functionToRun?: (data: T, index: number) => void | Promise<void>;
+    /** Invoked after each batch completes, with the total number of items processed so far. */
+    onProgress?: (processedAmount: number) => void | Promise<void>;
+    /** Called when processing an item fails. Receives the failed item and the error. */
+    onError?: (entry: T, err: unknown) => void | Promise<void>;
+    /** If true, stop fetching further batches once any error has occurred. */
+    returnOnError?: boolean;
+    /**
+     * If true, rethrow the first error out of `batchify()` itself once the batch
+     * it occurred in has settled, instead of only invoking `onError`. Use this when
+     * the caller needs the failure to actually propagate (e.g. to abort a transaction).
+     */
+    throwOnError?: boolean;
+    /** Milliseconds to wait after completing each batch before starting the next. */
+    pauseAfterBatch?: number;
+}
+export declare function batchify<T>({ dataDelivery, batchSize, concurrency, functionToRun, onProgress, onError, returnOnError, throwOnError, pauseAfterBatch, }?: BatchifyOptions<T>): Promise<void>;
+export {};
+//# sourceMappingURL=batchify.d.ts.map

package/dist/batchify.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"batchify.d.ts","sourceRoot":"","sources":["../src/batchify.ts"],"names":[],"mappings":"AAAA,UAAU,eAAe,CAAC,CAAC;IACzB;;;;OAIG;IACH,YAAY,CAAC,EAAE,CAAC,EAAE,GAAG,CAAC,CAAC,kBAAkB,EAAE,MAAM,EAAE,gBAAgB,EAAE,MAAM,KAAK,CAAC,EAAE,GAAG,OAAO,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;IACpG,qEAAqE;IACrE,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB;;;;;OAKG;IACH,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,6EAA6E;IAC7E,aAAa,CAAC,EAAE,CAAC,IAAI,EAAE,CAAC,EAAE,KAAK,EAAE,MAAM,KAAK,IAAI,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC;IACjE,2FAA2F;IAC3F,UAAU,CAAC,EAAE,CAAC,eAAe,EAAE,MAAM,KAAK,IAAI,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC;IAC/D,oFAAoF;IACpF,OAAO,CAAC,EAAE,CAAC,KAAK,EAAE,CAAC,EAAE,GAAG,EAAE,OAAO,KAAK,IAAI,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC;IAC3D,0EAA0E;IAC1E,aAAa,CAAC,EAAE,OAAO,CAAC;IACxB;;;;OAIG;IACH,YAAY,CAAC,EAAE,OAAO,CAAC;IACvB,iFAAiF;IACjF,eAAe,CAAC,EAAE,MAAM,CAAC;CAC1B;AAED,wBAAsB,QAAQ,CAAC,CAAC,EAAE,EAChC,YAAuB,EACvB,SAAgB,EAChB,WAAW,EACX,aAA+B,EAC/B,UAA4B,EAC5B,OAAyB,EACzB,aAAqB,EACrB,YAAoB,EACpB,eAAmB,GACpB,GAAE,eAAe,CAAC,CAAC,CAAM,GAAG,OAAO,CAAC,IAAI,CAAC,CAuDzC"}

package/dist/batchify.js

@@ -0,0 +1,57 @@
+export async function batchify({ dataDelivery = () => [], batchSize = 1000, concurrency, functionToRun = () => undefined, onProgress = () => undefined, onError = () => undefined, returnOnError = false, throwOnError = false, pauseAfterBatch = 0, } = {}) {
+    const effectiveConcurrency = concurrency ?? batchSize;
+    let iteration = 0;
+    let hasError = false;
+    let firstError;
+    let currentBatch = [];
+    const getNextBatch = async () => {
+        if (typeof dataDelivery === 'function') {
+            currentBatch = await dataDelivery(batchSize, iteration * batchSize);
+        }
+        else {
+            currentBatch = dataDelivery.slice(batchSize * iteration, batchSize + batchSize * iteration);
+        }
+    };
+    const runItem = async (entry, globalIndex) => {
+        try {
+            await functionToRun(entry, globalIndex);
+        }
+        catch (error) {
+            hasError = true;
+            if (throwOnError && !firstError)
+                firstError = { error };
+            await onError(entry, error);
+        }
+    };
+    const processCurrentBatch = async () => {
+        for (let start = 0; start < currentBatch.length; start += effectiveConcurrency) {
+            if (firstError)
+                return;
+            const chunk = currentBatch.slice(start, start + effectiveConcurrency);
+            await Promise.allSettled(chunk.map((entry, i) => runItem(entry, batchSize * iteration + start + i)));
+        }
+    };
+    await getNextBatch();
+    await onProgress(0);
+    while (currentBatch.length) {
+        await processCurrentBatch();
+        if (firstError)
+            throw firstError.error;
+        if (currentBatch.length < batchSize) {
+            await onProgress(batchSize * iteration + currentBatch.length);
+            return;
+        }
+        iteration += 1;
+        await onProgress(batchSize * iteration);
+        if (hasError && returnOnError) {
+            return;
+        }
+        await getNextBatch();
+        if (currentBatch.length && pauseAfterBatch) {
+            await new Promise((resolve) => {
+                setTimeout(resolve, pauseAfterBatch);
+            });
+        }
+    }
+}
+//# sourceMappingURL=batchify.js.map

package/dist/batchify.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"batchify.js","sourceRoot":"","sources":["../src/batchify.ts"],"names":[],"mappings":"AAkCA,MAAM,CAAC,KAAK,UAAU,QAAQ,CAAI,EAChC,YAAY,GAAG,GAAG,EAAE,CAAC,EAAE,EACvB,SAAS,GAAG,IAAI,EAChB,WAAW,EACX,aAAa,GAAG,GAAG,EAAE,CAAC,SAAS,EAC/B,UAAU,GAAG,GAAG,EAAE,CAAC,SAAS,EAC5B,OAAO,GAAG,GAAG,EAAE,CAAC,SAAS,EACzB,aAAa,GAAG,KAAK,EACrB,YAAY,GAAG,KAAK,EACpB,eAAe,GAAG,CAAC,MACG,EAAE;IACxB,MAAM,oBAAoB,GAAG,WAAW,IAAI,SAAS,CAAC;IACtD,IAAI,SAAS,GAAG,CAAC,CAAC;IAClB,IAAI,QAAQ,GAAG,KAAK,CAAC;IACrB,IAAI,UAA0C,CAAC;IAE/C,IAAI,YAAY,GAAQ,EAAE,CAAC;IAE3B,MAAM,YAAY,GAAG,KAAK,IAAI,EAAE;QAC9B,IAAI,OAAO,YAAY,KAAK,UAAU,EAAE,CAAC;YACvC,YAAY,GAAG,MAAM,YAAY,CAAC,SAAS,EAAE,SAAS,GAAG,SAAS,CAAC,CAAC;QACtE,CAAC;aAAM,CAAC;YACN,YAAY,GAAG,YAAY,CAAC,KAAK,CAAC,SAAS,GAAG,SAAS,EAAE,SAAS,GAAG,SAAS,GAAG,SAAS,CAAC,CAAC;QAC9F,CAAC;IACH,CAAC,CAAC;IAEF,MAAM,OAAO,GAAG,KAAK,EAAE,KAAQ,EAAE,WAAmB,EAAE,EAAE;QACtD,IAAI,CAAC;YACH,MAAM,aAAa,CAAC,KAAK,EAAE,WAAW,CAAC,CAAC;QAC1C,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,QAAQ,GAAG,IAAI,CAAC;YAChB,IAAI,YAAY,IAAI,CAAC,UAAU;gBAAE,UAAU,GAAG,EAAE,KAAK,EAAE,CAAC;YACxD,MAAM,OAAO,CAAC,KAAK,EAAE,KAAK,CAAC,CAAC;QAC9B,CAAC;IACH,CAAC,CAAC;IAEF,MAAM,mBAAmB,GAAG,KAAK,IAAI,EAAE;QACrC,KAAK,IAAI,KAAK,GAAG,CAAC,EAAE,KAAK,GAAG,YAAY,CAAC,MAAM,EAAE,KAAK,IAAI,oBAAoB,EAAE,CAAC;YAC/E,IAAI,UAAU;gBAAE,OAAO;YACvB,MAAM,KAAK,GAAG,YAAY,CAAC,KAAK,CAAC,KAAK,EAAE,KAAK,GAAG,oBAAoB,CAAC,CAAC;YACtE,MAAM,OAAO,CAAC,UAAU,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,KAAK,EAAE,CAAC,EAAE,EAAE,CAAC,OAAO,CAAC,KAAK,EAAE,SAAS,GAAG,SAAS,GAAG,KAAK,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC;QACvG,CAAC;IACH,CAAC,CAAC;IAEF,MAAM,YAAY,EAAE,CAAC;IACrB,MAAM,UAAU,CAAC,CAAC,CAAC,CAAC;IACpB,OAAO,YAAY,CAAC,MAAM,EAAE,CAAC;QAC3B,MAAM,mBAAmB,EAAE,CAAC;QAC5B,IAAI,UAAU;YAAE,MAAM,UAAU,CAAC,KAAK,CAAC;QACvC,IAAI,YAAY,CAAC,MAAM,GAAG,SAAS,EAAE,CAAC;YACpC,MAAM,UAAU,CAAC,SAAS,GAAG,SAAS,GAAG,YAAY,CAAC,MAAM,CAAC,CAAC;YAC9D,OAAO;QACT,CAAC;QACD,SAAS,IAAI,CAAC,CAAC;QACf,MAAM,UAAU,CAAC,SAAS,GAAG,SAAS,CAAC,CAAC;QACxC,IAAI,QAAQ,IAAI,aAAa,EAAE,CAAC;YAC9B,OAAO;QACT,CAAC;QACD,MAAM,YAAY,EAAE,CAAC;QACrB,IAAI,YAAY,CAAC,MAAM,IAAI,eAAe,EAAE,CAAC;YAC3C,MAAM,IAAI,OAAO,CAAC,CAAC,OAAO,EAAE,EAAE;gBAC5B,UAAU,CAAC,OAAO,EAAE,eAAe,CAAC,CAAC;YACvC,CAAC,CAAC,CAAC;QACL,CAAC;IACH,CAAC;AACH,CAAC"}