@frogfish/k2db 2.0.7 → 3.0.2

package/README.md

@@ -23,9 +23,10 @@ Where it fits in the stack
 
 Deployment tips (Nomad, Lambda, etc.)
 - Environments: Targets Node.js runtimes (Node 18/20). Not suitable for non‑TCP “edge JS” (e.g., Cloudflare Workers) that cannot open Mongo sockets.
-- Connection reuse: Create one `K2DB` instance per process and reuse it.
-  - Lambda: instantiate outside the handler and `await init()` lazily once.
-  - Nomad: init on service start; reuse for the lifetime of the task.
+- Connection reuse: Create and reuse `K2DB` instances.
+  - The underlying MongoDB connection pool is shared across `K2DB` instances created with the same cluster/auth settings (hosts/user/password/authSource/replicaset).
+  - This means you can safely keep one `K2DB` instance per logical database name (`name`) without creating a new TCP pool per database.
+  - `release()` is ref-counted: it only closes the shared pool when the last instance releases it.
 - Example (AWS Lambda):
   ```ts
   import { K2DB } from "@frogfish/k2db";
@@ -34,11 +35,28 @@ Deployment tips (Nomad, Lambda, etc.)
   export const handler = async (event) => {
     ready = ready || db.init();
     await ready; // reused across warm invocations
-    const res = await db.find("hello", { _owner: event.userId }, {}, 0, 10);
+    const res = await db.find("hello", {}, {}, 0, 10, event.userId);
     return { statusCode: 200, body: JSON.stringify(res) };
   };
   ```
-- Pooling and timeouts: The driver manages a small pool by default.
+  If you serve multiple logical databases (multi-project / multi-tenant), cache `K2DB` by database name. Instances will still share a single underlying connection pool:
+
+  ```ts
+  import { K2DB } from "@frogfish/k2db";
+
+  const base = { hosts: [{ host: "cluster0.example.mongodb.net" }], user: process.env.DB_USER, password: process.env.DB_PASS };
+  const byName = new Map<string, K2DB>();
+
+  function dbFor(name: string) {
+    let db = byName.get(name);
+    if (!db) {
+      db = new K2DB({ ...base, name });
+      byName.set(name, db);
+    }
+    return db;
+  }
+  ```
+- Pooling and timeouts: The MongoDB driver manages a small pool by default, and k2db reuses that pool across `K2DB` instances that share cluster/auth config.
   - Serverless: keep `minPoolSize=0` (default), consider `maxIdleTimeMS` to drop idle sockets faster.
   - Long‑lived services (Nomad): you can tune pool sizing if needed.
   - You can adjust `connectTimeoutMS/serverSelectionTimeoutMS` in the code if your environment needs higher values.
@@ -82,7 +100,8 @@ Ownership (`_owner`)
 - Purpose: `_owner` is not a tenant ID nor a technical auth scope. It’s a required, opinionated piece of metadata that records who a document belongs to (the data subject or system principal that created/owns it).
 - Why it matters: Enables clear data lineage and supports privacy/jurisdiction workflows (GDPR/DSAR: “export all my data”, “delete my data”), audits, and stewardship.
 - Typical values: a user’s UUID when a signed-in human creates the record; for automated/system operations use a stable identifier like `"system"`, `"service:mailer"`, or `"migration:2024-09-01"`.
-- Not authorization: The library does not enforce access control based on `_owner`. Apply your authorization rules in the API/service layer using `_owner` as a helpful filter or join key.
+- Not authentication: k2db does not authenticate callers. Your API/service still decides *who* the caller is and whether they are allowed to act.
+- Optional enforcement: k2db can enforce owner scoping when you provide a per-call scope (see “Scope”), and can require scope on all calls when `ownershipMode: "strict"` is enabled.
 - Multi-tenant setups: If you have tenants, keep a distinct `tenantId` (or similar) alongside `_owner`. `_owner` continues to model “who owns this record” rather than “which tenant it belongs to”.
 
 Config
@@ -90,10 +109,11 @@ Config
 import { K2DB } from "@frogfish/k2db";
 
 const db = new K2DB({
-  name: "mydb",
+  name: "mydb", // logical database name; instances with the same hosts/auth share one connection pool
   hosts: [{ host: "cluster0.example.mongodb.net" }], // SRV if single host without port
   user: process.env.DB_USER,
   password: process.env.DB_PASS,
+  authSource: process.env.DB_AUTH_SOURCE, // optional (defaults to "admin" when user+password provided)
   slowQueryMs: 300,
   hooks: {
     beforeQuery: (op, d) => {},
@@ -105,9 +125,358 @@ await db.init();
 await db.ensureIndexes("myCollection");
 ```
 
+### Scope config
+
+- `ownershipMode?: "lax" | "strict"` (default: `"lax"`)
+  - `"lax"` (default): Passing a scope is optional. If you provide a scope, k2db enforces it as an owner filter; if omitted, behavior is unchanged from historical (no owner enforcement, but still enforces soft-delete).
+  - `"strict"`: Scope is *required* on all scopified methods (see “Scope” below). If you omit scope, k2db throws an error. This helps prevent accidental missing owner filters.
+
+Example config with strict mode:
+```ts
+const db = new K2DB({
+  name: "mydb",
+  hosts: [{ host: "cluster0.example.mongodb.net" }],
+  user: process.env.DB_USER,
+  password: process.env.DB_PASS,
+  ownershipMode: "strict",
+});
+```
+
+### Aggregation config
+
+- `aggregationMode?: "loose" | "guarded" | "strict"` (default: `"loose"`)
+  - `"loose"`: No aggregation pipeline validation (all MongoDB stages permitted).
+  - `"guarded"`: Denies `$out`, `$merge`, `$function`, `$accumulator`; requires a positive limit, caps max limit, and adds `maxTimeMS`.
+  - `"strict"`: Allows only `$match`, `$project`, `$sort`, `$skip`, `$limit` stages; also requires and caps limit.
+
+Example:
+```ts
+const db = new K2DB({
+  name: "mydb",
+  hosts: [{ host: "cluster0.example.mongodb.net" }],
+  aggregationMode: "guarded", // disables dangerous stages, enforces limit
+});
+```
+
+## Scope
+
+Scope is an optional per-call “owner filter” that k2db can apply to most data access and mutation methods. It provides a safety guardrail to help prevent missing `_owner` filters, especially in multi-tenant or user-data contexts.
+
+**Scope is not authentication or authorization:** It does not decide *who* may act. Your API/service is still responsible for authenticating callers and deciding what scopes/owners they are allowed to access.
+
+### Type
+
+```ts
+type Scope = string | "*";
+```
+- If a `string` is provided, only documents with `_owner === scope` are included or affected.
+- If `"*"` is provided, *all* owners are included (no owner filter). This is intended for admin/service-to-service operations—never pass untrusted `"*"` from user input.
+
+### Modes
+
+- In `ownershipMode: "lax"` (default):
+  - If `scope` is omitted: No owner filtering is applied (historical behavior; still enforces soft-delete).
+  - If `scope` is provided: Only docs with `_owner === scope` (or all docs if `"*"`).
+- In `ownershipMode: "strict"`:
+  - `scope` is *required* for all scopified methods (see below). If not provided, k2db throws an error.
+  - Use `"*"` for admin/system/service calls (do not invent a magic owner like `_system`).
+
+### Methods that support scope
+
+The following methods accept an optional `scope?: Scope | string` as their final argument:
+- `get`
+- `findOne`
+- `find`
+- `count`
+- `update`
+- `updateAll`
+- `delete`
+- `deleteAll`
+- `restore`
+- `purge`
+- `purgeDeletedOlderThan`
+- `drop` and `dropDatabase`: For destructive/admin operations, use `"*"` as scope to indicate you intend to operate without owner restriction.
+
+### Examples
+
+#### a) User reads own docs
+```ts
+// Only docs owned by this user
+await db.find("posts", {}, {}, 0, 20, userId);
+```
+
+#### b) Admin/service reads or mutates all docs
+```ts
+// Read all posts (no owner restriction)
+await db.find("posts", {}, {}, 0, 20, "*");
+
+// Delete a user (admin only)
+await db.delete("users", id, "*");
+```
+
+#### c) Strict mode with HTTP header mapping
+Suppose your API receives:
+```
+X-Scope: <userId> | *
+```
+You should map this header to the `scope` argument:
+```ts
+const scope = req.headers["x-scope"];
+// Only allow "*" for trusted admin/service credentials!
+await db.find("posts", {}, {}, 0, 20, scope);
+```
+
+### Rules of thumb
+- Never pass untrusted `"*"` as a scope—restrict this to trusted admin/service credentials only.
+- Prefer passing a scope everywhere; enabling strict mode helps you catch missing owner filters.
+- Normalize `_owner` values consistently—prefer lowercase and a single convention (e.g., always userId as lowercase UUID).
+
+
+## Aggregation
+
+Aggregation in k2db lets you run MongoDB pipelines with guardrails for soft-delete, secure fields, and pipeline safety.
+
+### `aggregate(collection, pipeline, skip?, limit?)`
+
+Runs an aggregation pipeline on the given collection, returning an array of documents. k2db automatically injects filters and enforces restrictions for safety and consistency.
+
+#### What k2db enforces automatically
+
+- **Soft-delete enforcement:** Automatically inserts a `$match: { _deleted: { $ne: true } }` stage near the start of your pipeline so only non-deleted documents are returned. For pipelines beginning with `$search`, `$geoNear`, or `$vectorSearch`, the filter is injected after the first stage to avoid breaking those operators.
+- **Nested enforcement:** For `$lookup`, `$unionWith`, `$graphLookup`, and `$facet`, any sub-pipeline is rewritten to ensure the non-deleted filter applies to foreign collections as well. Simple `$lookup` with `localField`/`foreignField` is rewritten to pipeline form so the non-deleted filter can be injected.
+- **Pagination:** If you pass `skip`/`limit`, those are appended to the pipeline. In `"guarded"`/`"strict"` aggregationMode, a positive limit is required and capped to a safe maximum, and `maxTimeMS` is set to prevent long-running queries.
+- **Secure fields:** If `secureFieldPrefixes` is configured (e.g. `["#"]`), any pipeline referencing a secure-prefixed field (such as `"#passport_number"`) is rejected, even in expressions. Returned documents are also stripped of any keys beginning with a secure prefix.
+
+#### What you cannot do (by default)
+
+- **Return soft-deleted documents:** The injected `$match: { _deleted: { $ne: true } }` filter means aggregate will never return soft-deleted docs, regardless of your pipeline.
+- **Use secure fields in aggregate:** Any pipeline referencing a field like `"#passport_number"` (including in `$project`, `$addFields`, or expressions) is rejected with an error. Even if a document contains a secure-prefixed field, it is stripped from the output.
+- **Use dangerous or non-allowlisted stages:** In `"guarded"` mode, you cannot use `$out`, `$merge`, `$function`, or `$accumulator`. In `"strict"` mode, only `$match`, `$project`, `$sort`, `$skip`, and `$limit` are allowed.
+
+#### Filtering level: root vs nested pipelines
+
+- The root pipeline always gets the non-deleted filter injected (unless the first stage is `$search`, `$geoNear`, or `$vectorSearch`, in which case it's injected after).
+- For nested pipelines in `$lookup`, `$unionWith`, `$graphLookup`, and `$facet`, k2db rewrites or injects the non-deleted filter into each sub-pipeline, so deleted foreign documents are excluded.
+- If a `$lookup` uses the simple `localField`/`foreignField` form, k2db rewrites it to a pipeline `$lookup` so filtering can be enforced.
+
+#### Examples
+
+**1) Basic aggregation: injected soft-delete filter**
+
+Suppose you call:
+```js
+const pipeline = [
+  { $match: { status: "active" } },
+  { $project: { name: 1, status: 1 } }
+];
+await db.aggregate("users", pipeline);
+```
+**Effective pipeline after k2db injection:**
+```js
+[
+  { $match: { _deleted: { $ne: true } } },      // injected
+  { $match: { status: "active" } },
+  { $project: { name: 1, status: 1 } }
+]
+```
+
+**2) `$lookup` rewritten for non-deleted foreign docs**
+
+Original pipeline:
+```js
+[
+  {
+    $lookup: {
+      from: "orders",
+      localField: "_uuid",
+      foreignField: "user_id",
+      as: "orders"
+    }
+  }
+]
+```
+k2db rewrites this to:
+```js
+[
+  {
+    $lookup: {
+      from: "orders",
+      let: { local_id: "$_uuid" },
+      pipeline: [
+        { $match: { _deleted: { $ne: true } } }, // injected for foreign docs
+        { $match: { $expr: { $eq: ["$user_id", "$$local_id"] } } }
+      ],
+      as: "orders"
+    }
+  }
+]
+```
+
+**3) Secure field reference is rejected**
+
+Pipeline:
+```js
+[
+  { $project: { name: 1, passport: "$#passport_number" } }
+]
+```
+Result: **Throws an error** — referencing a secure-prefixed field (`"#passport_number"`) is not allowed in aggregate pipelines.
+
+**4) Attempting to aggregate deleted docs returns nothing**
+
+Pipeline:
+```js
+[
+  { $match: { _deleted: true } }
+]
+```
+Result: Returns **no documents** — k2db injects `{ _deleted: { $ne: true } }` before your match, so the result set is always empty.
+
+---
+
+> **Note:** k2db's aggregation guardrails ensure you cannot accidentally leak deleted or secure data, or run dangerous stages in stricter modes.
+
+## Secure fields and encryption at rest
+
+k2db supports **secure fields**: fields whose keys start with a configurable prefix (recommended: `#`). Secure fields are designed to be hard to accidentally leak in bulk queries and hard to casually access in code.
+
+This feature has two layers:
+
+1) **Guardrails (always):** Secure fields are stripped from multi-record reads (`find`, `aggregate`) and cannot be explicitly projected. Aggregation pipelines are also rejected if they reference secure fields.
+2) **Encryption at rest (optional):** When enabled, secure-field values are encrypted before being written to MongoDB and decrypted on single-record reads.
+
+### Why `#` (friction) instead of `__somethingNice`
+
+The goal is to make secure fields “visually wrong” and **ergonomically annoying** to access on purpose:
+
+- `doc["#passport_number"]` is explicit and reviewable.
+- `doc.#passport_number` is impossible (forces bracket access).
+- It discourages lazy DTO copying and casual destructuring that can accidentally leak secrets.
+
+Using `__private` looks “normal”, which increases the chance developers will treat it like any other field and accidentally return it in list endpoints.
+
+Also, `_...` is reserved for k2db’s metadata (`_uuid`, `_owner`, `_created`, …), so `#...` cleanly avoids that namespace.
+
+### Configuration
+
+Enable secure fields by setting `secureFieldPrefixes`. To also encrypt them at rest, provide a base64 AES-256 key and a key id.
+
+```ts
+const db = new K2DB({
+  name: "mydb",
+  hosts: [{ host: "cluster0.example.mongodb.net" }],
+
+  // Treat "#..." fields as secure
+  secureFieldPrefixes: ["#"],
+
+  // Optional encryption-at-rest for secure fields:
+  // - must decode to 32 bytes (AES-256)
+  // - values are stored as "<keyid>:<payload>"
+  secureFieldEncryptionKeyId: "k1",
+  secureFieldEncryptionKey: process.env.K2DB_SECURE_KEY_B64,
+});
+```
+
+Key requirements:
+- `secureFieldEncryptionKey` must be **base64** and decode to **32 bytes**.
+- Encryption is enabled only when **both** `secureFieldEncryptionKey` and `secureFieldEncryptionKeyId` are provided.
+
+> Note: Today k2db decrypts only ciphertexts whose `keyid` matches the configured `secureFieldEncryptionKeyId`. If the stored `keyid` differs, the encrypted string is returned as-is (useful during key rotation rollout, but plan your rotation strategy accordingly).
+
+### Storage format
+
+When encryption is enabled, each secure field value is stored as a single string:
+
+```
+<keyid>:<ivB64>.<tagB64>.<ctB64>
+```
+
+- Algorithm: AES-256-GCM
+- Plaintext: `JSON.stringify(value)` (so secure fields may be strings, numbers, objects, arrays, etc.)
+
+### Behavior by method
+
+With `secureFieldPrefixes: ["#"]`:
+
+- **create / update / updateAll**:
+  - If encryption is enabled, `#...` values are encrypted before write.
+  - If encryption is disabled, values are stored as provided.
+- **get / findOne** (single-record reads):
+  - If encryption is enabled, `#...` values are decrypted and returned in the object.
+  - If encryption is disabled, values are returned as stored.
+- **find** (multi-record reads):
+  - Secure fields are **stripped** from every returned document (even if encryption is disabled).
+- **aggregate**:
+  - Secure fields are **not allowed to be referenced** in the pipeline (k2db throws).
+  - Secure fields are also **stripped** from returned documents as a second safety net.
+- **projections**:
+  - `findOne(fields=[...])` and `find(params.filter=[...])` cannot include secure fields (throws).
+
+### Examples
+
+#### 1) Store a secure field
+
+```ts
+const owner = userId.toLowerCase();
+
+const { id } = await db.create("profiles", owner, {
+  name: "Ada",
+  "#passport_number": "123456789",
+  "#home_address": { line1: "1 Example St", city: "Perth" },
+});
+```
+
+- If encryption is enabled, MongoDB stores `"#passport_number"` and `"#home_address"` as encrypted strings.
+- If encryption is disabled, they are stored as plaintext (still guarded on reads).
+
+#### 2) Single-record read returns secure fields
+
+```ts
+const profile = await db.get("profiles", id, owner);
+
+// Explicit access (friction by design)
+console.log(profile["#passport_number"]);
+```
+
+#### 3) Multi-record read strips secure fields
+
+```ts
+const list = await db.find("profiles", {}, {}, 0, 50, owner);
+
+// "#..." fields are removed from each document in list
+console.log(list[0]["#passport_number"]); // undefined
+```
+
+#### 4) Aggregation cannot reference secure fields
+
+```ts
+await db.aggregate("profiles", [
+  { $project: { name: 1, passport: "$#passport_number" } },
+]);
+// throws: secure-prefixed field referenced in pipeline
+```
+
+### What this is (and isn’t)
+
+- This is a **data safety guardrail** and optional encryption-at-rest mechanism.
+- It is **not authentication/authorization**: you still must decide who the caller is and what they’re allowed to do.
+- For admin/service operations, combine this with **Scope** rules: do not accept `"*"` from untrusted callers.
+
 Environment loader
 ```ts
-const conf = K2DB.fromEnv(); // K2DB_NAME, K2DB_HOSTS, K2DB_USER, K2DB_PASSWORD, K2DB_REPLICASET, K2DB_SLOW_MS
+const conf = K2DB.fromEnv(); // K2DB_NAME (logical db), K2DB_HOSTS, K2DB_USER, K2DB_PASSWORD, K2DB_AUTH_SOURCE, K2DB_REPLICASET, K2DB_SLOW_MS
+```
+
+Testing
+
+If you run many test suites in a single Node process and want to fully tear down shared MongoDB pools between suites, you can use the test helper:
+
+```ts
+import { resetSharedMongoClientsForTests } from "@frogfish/k2db";
+
+afterAll(async () => {
+  await resetSharedMongoClientsForTests();
+});
 ```
 
 Tips
@@ -168,6 +537,7 @@ export K2DB_NAME=mydb
 export K2DB_HOSTS=cluster0.xxxxxx.mongodb.net
 export K2DB_USER=your_user
 export K2DB_PASSWORD=your_pass
+export K2DB_AUTH_SOURCE=admin
 node hello.mjs
 ```
 ```ts
@@ -194,7 +564,7 @@ const db = new K2DB({
   password: process.env.DB_PASS,
 });
 
-await db.init();
+await db.init(); // safe to call multiple times; concurrent calls are deduped
 await db.ensureIndexes("hello"); // unique _uuid among non-deleted, plus helpful indexes
 
 // Create a document (owner is required)
@@ -273,19 +643,19 @@ db.clearSchema("hello");
 - `VersionInfo`: `{ _uuid: string; _v: number; _at: number }`
 
 Returns by method
-- `get(collection, id)`: `Promise<BaseDocument>`
-- `find(collection, filter, params?, skip?, limit?)`: `Promise<BaseDocument[]>`
-- `findOne(collection, criteria, fields?)`: `Promise<BaseDocument|null>`
+- `get(collection, id, scope?)`: `Promise<BaseDocument>`
+- `find(collection, filter, params?, skip?, limit?, scope?)`: `Promise<BaseDocument[]>`
+- `findOne(collection, criteria, fields?, scope?)`: `Promise<BaseDocument|null>`
 - `aggregate(collection, pipeline, skip?, limit?)`: `Promise<BaseDocument[]>`
 - `create(collection, owner, data)`: `Promise<CreateResult>`
-- `update(collection, id, data, replace?)`: `Promise<UpdateResult>`
-- `updateAll(collection, criteria, values)`: `Promise<UpdateResult>`
-- `delete(collection, id)`: `Promise<DeleteResult>`
-- `deleteAll(collection, criteria)`: `Promise<DeleteResult>`
-- `purge(collection, id)`: `Promise<PurgeResult>`
-- `restore(collection, criteria)`: `Promise<RestoreResult>`
-- `count(collection, criteria)`: `Promise<CountResult>`
-- `drop(collection)`: `Promise<DropResult>`
+- `update(collection, id, data, replace?, scope?)`: `Promise<UpdateResult>`
+- `updateAll(collection, criteria, values, scope?)`: `Promise<UpdateResult>`
+- `delete(collection, id, scope?)`: `Promise<DeleteResult>`
+- `deleteAll(collection, criteria, scope?)`: `Promise<DeleteResult>`
+- `purge(collection, id, scope?)`: `Promise<PurgeResult>`
+- `restore(collection, criteria, scope?)`: `Promise<RestoreResult>`
+- `count(collection, criteria, scope?)`: `Promise<CountResult>`
+- `drop(collection, scope?)`: `Promise<DropResult>`
 - `ensureIndexes(collection, opts?)`: `Promise<void>`
 - `ensureHistoryIndexes(collection)`: `Promise<void>`
 - `updateVersioned(collection, id, data, replace?, maxVersions?)`: `Promise<VersionedUpdateResult[]>`
@@ -302,7 +672,7 @@ _uuid = Crockford Base32 encoded UUID V7, Uppercase, with hyphens
 
 0J4F2-H6M8Q-7RX4V-9D3TN-8K2WZ
 
-// Canonical uppercase form with hyphens
+// Canonical uppercase form with hyphens Crockford 32
 const CROCKFORD_ID_REGEX = /^[0-9A-HJKMNP-TV-Z]{5}-[0-9A-HJKMNP-TV-Z]{5}-[0-9A-HJKMNP-TV-Z]{5}-[0-9A-HJKMNP-TV-Z]{5}-[0-9A-HJKMNP-TV-Z]{6}$/;
 
 // Example usage: