@klum-db/lobby 0.2.0-pre.29 → 0.2.0-pre.31

package/README.md

@@ -1,43 +1,65 @@
 # @klum-db/lobby
 
-> **The Lobby** orchestrates a *group* of sovereign [noy-db](https://github.com/vLannaAi/noy-db) vaults — federation, interchange, custody, and scoped sync — without ever dissolving their independence.
->
-> **noy-db is the vault (inward). klum-db is the Lobby (outward).**
-> A container runs perfectly alone; the engine orchestrates many. Docker is to a container what the Lobby is to a vault.
+> **Small enough to own, coordinated enough to scale.**
+> `@klum-db/lobby` is the **control plane** for a fleet of sovereign [noy-db](https://github.com/vLannaAi/noy-db) vaults. Each vault is small, **100%-encrypted**, and complete on its own — its own embedded schema, its own query engine, usable **offline**. The Lobby orchestrates many into one coordinated whole, **online or offline**: **coordination without custody** — it drives the fleet but never owns the data. *One-way — klum drives noy; noy never knows klum exists.*
 
 `@klum-db/lobby` · status: **preview** · depends on `@noy-db/hub`
 
 ---
 
-## The idea in 20 seconds
+## What it is, in 20 seconds
 
-Banking, accounting, health, insurance — the data that matters is **individual, single-entity, and owned by the subject**. noy-db makes each of those a small, sovereign, in-memory vault that is a *complete system on its own*. Working "small, in memory" isn't a limitation — it's the strong core. The limitation only bites when one actor must work across **many** datasets at once.
+A single vault is complete — its own keys, schema, and truth — but completeness is also a ceiling: alone, a vault can only answer for *itself*. The usual fix is to pour everything into one central store, trading away the sovereignty that made each vault worth trusting. **The Lobby takes the other path — it coordinates the vaults where they stand.**
 
-The **Lobby** is the framework for exactly that outward dimension: *the efficiency of a small independent dataset at the core, joined with an actor operating across many at the same time* — a counterbalance to tech giants holding user data hostage. Vaults stay independent and relocatable; the Lobby coordinates them without absorbing them. That's why it's a **group** (Thai *klum* กลุ่ม), never a cluster — federation here is non-aggregative.
-
-## Reads in a sentence
-
-> A firm is a **Custodian** in the **Lobby**: it holds operating grants to client **Vaults** that all reference one shared **Pool**. The client holds the **Deed**. To onboard, the firm **Relocates** a client's vault as a **Bundle**, **Migrates** it to the current schema, and **Merges** the Pool slice by field **Authority** using **Provenance**. The client can **Withdraw** anytime; an abandoned vault can be **Liberated**.
+```mermaid
+flowchart TD
+    L["<b>CONTROL PLANE</b> · @klum-db/lobby<br/>federate · interchange · custody · surface"]:::cp
+    L ==> V1["🔒 vault"]:::dp
+    L ==> V2["🔒 vault"]:::dp
+    L ==> V3["🔒 vault"]:::dp
+    L ==> V4["🔒 vault"]:::dp
+    classDef cp fill:#0f766e,stroke:#0f766e,color:#ffffff
+    classDef dp fill:#ecfdf5,stroke:#10b981,color:#065f46
+```
 
-Every bold word is a real, shipped capability.
+*DATA PLANE · `@noy-db/hub` — each vault sovereign & 100% encrypted, complete on its own. **klum drives noy one-way; noy never depends on klum.***
 
-## Architecture — one-way dependency, two complementary axes
+<details><summary>Text version (npm / non-Mermaid viewers)</summary>
 
 ```
-            outward / orchestration                 inward / the vault
-   ┌─────────────────────────────────┐     ┌──────────────────────────────────┐
-   │          @klum-db/lobby         │     │            @noy-db/hub            │
-   │                                 │     │                                   │
-   │  Lobby  ⊃  many Vaults          │ ──▶ │  Vault ⊃ Collection ⊃ Record ⊃ Field │
-   │   • Federation (fleets)         │     │   • keyring · per-record CEK      │
-   │   • Interchange (move data)     │     │   • computed/derivation · money   │
-   │   • Custody (Deed/Custodian)    │     │   • i18n · history · snapshots    │
-   │   • Surface (scoped sync)       │     │   • a vault + a store = complete  │
-   └─────────────────────────────────┘     └──────────────────────────────────┘
-          binds to the stable  ── @noy-db/hub/kernel ──  surface only
+  ┌────────────────────────────────────────────────────┐
+  │  CONTROL PLANE · @klum-db/lobby                      │
+  │  federate · interchange · custody · surface         │
+  └────────────────────────┬───────────────────────────┘
+                           │ drives — one-way (klum → noy)
+        ┌──────────┬───────┴───────┬──────────┐
+        ▼          ▼               ▼          ▼
+     ┌──────┐   ┌──────┐        ┌──────┐   ┌──────┐
+     │vault │   │vault │        │vault │   │vault │
+     └──────┘   └──────┘        └──────┘   └──────┘
+  DATA PLANE · @noy-db/hub — each vault sovereign & non-fungible,
+  complete on its own.  noy never depends on klum.
 ```
+</details>
+
+- **Coordination without custody.** klum drives the fleet — federate, move data, custody, sync — but **never owns the data**: keys, crypto, and records stay sovereign in each vault, and klum binds to one stable contract (`@noy-db/hub/kernel`), never hub internals. *(One-way, and enforced — a build-time guard means no `@noy-db` package can ever import `@klum-db`.)*
+- **Bring the work to the data, not the data to a lake.** Cross-vault queries resolve *across* the group — each vault answers for its own slice under its own keys, nothing pools. No central honeypot to breach.
+- **Small core, coordinated reach.** Each vault stays small, portable, and individually revocable; orchestration recovers the cross-cutting reach you'd otherwise need a monolith for.
+- **A group, not a cluster.** Vaults are sovereign and non-fungible — one subject = one vault, the subject holds the deed. *Joined, not merged; allied, not absorbed.* (Thai *klum* กลุ่ม = a group.)
+
+## Why it exists
+
+Banking, accounting, health, insurance — the data that matters is **individual, single-subject, and owned by the person it's about**. noy-db makes each of those a small, sovereign, **100%-encrypted** vault — a complete system on its own. *You own your data, in spite of the cloud.* The limit only bites when one actor must work across **many** subjects at once.
+
+That's the Lobby's dimension: the efficiency and privacy of a small sovereign dataset at the core, joined with an actor operating across many at once — **governance by default, not by checklist**. Access is **scoped, purpose-limited, and revocable** (the subject can withdraw anytime); the orchestrator coordinates without absorbing. A counterweight to the central data lake — without the lock-in, the lock-out, or the single honeypot.
 
-The dependency runs **one way**: `@klum-db/lobby` → `@noy-db/hub`. No `@noy-db` package ever imports `@klum-db` (enforced by a build-time architecture guard). The Lobby binds to a stable internal surface, **`@noy-db/hub/kernel`**, not hub internals. A vault is a complete, shippable system *without* the Lobby; the Lobby is what you reach for when one actor must work across many vaults at once.
+**Lineage & market context:** klum-db sits in the local-first / data-sovereignty tradition — [Local-first software](https://www.inkandswitch.com/essay/local-first/), [GDPR data portability](https://gdpr-info.eu/art-20-gdpr/), and the full picture in [**docs/positioning.md**](docs/positioning.md).
+
+## Reads in a sentence
+
+> A firm is a **Custodian** in the **Lobby**: it holds operating grants to client **Vaults** that all reference one shared **Pool**. The client holds the **Deed**. To onboard, the firm **Relocates** a client's vault as a **Bundle**, **Migrates** it to the current schema, and **Merges** the Pool slice by field **Authority** using **Provenance**. The client can **Withdraw** anytime; an abandoned vault can be **Liberated**.
+
+Every bold word is a real, shipped capability.
 
 ## Install
 
@@ -131,17 +153,22 @@ const { bundleBytes, transferKey } = await lobby.exportSurface('payroll-vault',
 await lobby.applySurface('tax-vault', surface, bundleBytes, transferKey)
 ```
 
+### On-ramp · Dock → `graduate()`
+
+A foreign, non-noy-db unit (a legacy DB, a raw export) can't be federated directly — the sovereign tier needs a vault's keyring and per-record keys. **Dock** carries it read-only at a lower tier; **`graduate()`** imports it into a fresh sovereign vault, unlocking the full tier (custody, provenance, field-Authority merge). It's the one move that *adds a node to the data plane* rather than coordinating existing ones.
+
 ---
 
 ## Relationship with noy-db
 
-- **Depends on `@noy-db/hub`**, binds to the stable **`@noy-db/hub/kernel`** subpath — never reaches into hub internals.
+The one-way law and the kernel seam are covered up top — here are the specifics that matter when you build:
+
 - **Custody is a vault-level concern** and lives *in* hub (keyring/CEK/consent primitives); the Lobby **re-exports** it (`createDeedOwner`, `liberateVault`, `CustodyApi`) so consumers have one import surface.
 - **Federation** lives in the Lobby, not in hub — open fleets with `lobby.openVaultGroup` (`@noy-db/hub` no longer ships the `openVaultGroup` / `openStateManagementVault` / `withVaultTemplate` fleet methods).
-- The dependency is enforced one-way at build time; an `@noy-db` package importing `@klum-db` fails the architecture check.
+- **Enforced, not conventional:** an `@noy-db` package importing `@klum-db` fails noy-db's build-time architecture check.
 
 ## Status
 
 Preview. `@klum-db/lobby` is its own repository and the sole publisher of `@klum-db/*` to npm. It depends on the **published** `@noy-db/*` packages through the stable `@noy-db/hub/kernel` boundary and versions **independently** (`0.2.0-pre.N`, decoupled from noy-db). Pilot-1 (FR-1…FR-9), the dock tier, and `Lobby.graduate()` are complete.
 
-See [`PROVENANCE.md`](./PROVENANCE.md) for origin and build history.
+See [`docs/architecture.md`](docs/architecture.md) for the detailed noy-db ↔ klum-db boundary, [`docs/roadmap.md`](docs/roadmap.md) for what's next, and [`PROVENANCE.md`](./PROVENANCE.md) for origin and build history.

package/dist/bin/klum.d.cts

@@ -1,4 +1,4 @@
-import { V as VaultGroup } from '../vault-group-BXjO5kHB.cjs';
+import { V as VaultGroup } from '../vault-group-DqEyXbN1.cjs';
 import '@noy-db/hub/kernel';
 import '@noy-db/hub';
 import '@noy-db/hub/bundle';

package/dist/bin/klum.d.ts

@@ -1,4 +1,4 @@
-import { V as VaultGroup } from '../vault-group-BXjO5kHB.js';
+import { V as VaultGroup } from '../vault-group-DqEyXbN1.js';
 import '@noy-db/hub/kernel';
 import '@noy-db/hub';
 import '@noy-db/hub/bundle';

package/dist/index.cjs

@@ -1268,6 +1268,43 @@ var init_insight_auto_push = __esm({
   }
 });
 
+// src/federation/partial-reduce.ts
+function canPartialReduce(spec) {
+  return Object.values(spec).every((r) => typeof r.merge === "function");
+}
+function reduceToPartial(records, spec) {
+  const out = {};
+  for (const [key, reducer] of Object.entries(spec)) {
+    const r = reducer;
+    let state = r.init();
+    for (const rec of records) state = r.step(state, rec);
+    out[key] = state;
+  }
+  return out;
+}
+function mergePartials(spec, partials) {
+  const out = {};
+  for (const [key, reducer] of Object.entries(spec)) {
+    const r = reducer;
+    let acc = r.init();
+    for (const p of partials) acc = r.merge(acc, p[key]);
+    out[key] = acc;
+  }
+  return out;
+}
+function finalizePartial(spec, merged) {
+  const out = {};
+  for (const [key, reducer] of Object.entries(spec)) {
+    out[key] = reducer.finalize(merged[key]);
+  }
+  return out;
+}
+var init_partial_reduce = __esm({
+  "src/federation/partial-reduce.ts"() {
+    "use strict";
+  }
+});
+
 // src/federation/aggregate-across.ts
 var import_kernel8, import_kernel9, CrossVaultAggregation, CrossVaultGroupedAggregation;
 var init_aggregate_across = __esm({
@@ -1276,6 +1313,7 @@ var init_aggregate_across = __esm({
     import_kernel8 = require("@noy-db/hub/kernel");
     import_kernel9 = require("@noy-db/hub/kernel");
     init_cross_vault_live();
+    init_partial_reduce();
     CrossVaultAggregation = class {
       constructor(src, spec, bind) {
         this.src = src;
@@ -1286,6 +1324,10 @@ var init_aggregate_across = __esm({
       spec;
       bind;
       async run(options = {}) {
+        if (this.src.fanoutReduce && canPartialReduce(this.spec)) {
+          const { partials, skippedVaults: skippedVaults2 } = await this.src.fanoutReduce(this.spec, options);
+          return { result: finalizePartial(this.spec, mergePartials(this.spec, partials)), skippedVaults: skippedVaults2 };
+        }
         const { records, skippedVaults } = await this.src.fanoutRecords(options);
         return { result: (0, import_kernel8.reduceRecords)(records, this.spec), skippedVaults };
       }
@@ -1412,6 +1454,7 @@ var init_vault_group = __esm({
     init_cross_vault_live();
     init_insight_auto_push();
     init_aggregate_across();
+    init_partial_reduce();
     SHARD_SEPARATOR = "--";
     SAFE_PARTITION_KEY = /^[A-Za-z0-9._-]+$/;
     VaultGroup = class {
@@ -1550,22 +1593,33 @@ var init_vault_group = __esm({
       collection(collectionName) {
         return new ShardedCollection(this, collectionName);
       }
-      /** @internal — eligible (openable-candidate) rows + drift/divergence skips. */
+      /** @internal — eligible (openable-candidate) rows + drift/divergence/unreachable skips. */
       async resolveEligible(options = {}) {
         const rows = await this.allRows();
+        const candidates = options.only ? rows.filter((r) => options.only.includes(r.partitionKey)) : rows;
         const skipped = [];
         const versionOk = [];
-        for (const row of rows) {
+        for (const row of candidates) {
           if (options.minVersion !== void 0 && row.schemaVersion < options.minVersion) {
             skipped.push({ vaultId: row.vaultId, reason: "schema-drift" });
           } else versionOk.push(row);
         }
-        const provisioned = await Promise.all(versionOk.map((r) => this.db._shardVaultProvisioned(r.vaultId)));
+        const probes = await Promise.all(
+          versionOk.map(async (row) => {
+            try {
+              return { row, provisioned: await this.db._shardVaultProvisioned(row.vaultId) };
+            } catch (err) {
+              if (options.failFast) throw err;
+              return { row, error: err };
+            }
+          })
+        );
         const eligible = [];
-        versionOk.forEach((row, i) => {
-          if (provisioned[i]) eligible.push(row);
-          else skipped.push({ vaultId: row.vaultId, reason: "error", error: new import_kernel10.ShardProvisioningError(row.vaultId, row.partitionKey) });
-        });
+        for (const p of probes) {
+          if ("error" in p) skipped.push({ vaultId: p.row.vaultId, reason: "error", error: p.error });
+          else if (p.provisioned) eligible.push(p.row);
+          else skipped.push({ vaultId: p.row.vaultId, reason: "error", error: new import_kernel10.ShardProvisioningError(p.row.vaultId, p.row.partitionKey) });
+        }
         return { eligible, skipped };
       }
       /** @internal — registered push-model cross-vault derivations (#271 Insight Vault). */
@@ -1625,12 +1679,19 @@ var init_vault_group = __esm({
        * Insight Vault keyed by partition key. Shards behind `minVersion`,
        * unprovisioned, or whose read errors are reported in `skippedVaults` and
        * are not written (a stale summary is never left behind for a failed shard).
+       * A shard whose backend is unreachable (provisioning probe or read throws) is
+       * **skipped** (`reason: 'error'`) and the pass continues for the others — its
+       * prior summary is left intact. Pass `failFast: true` for the legacy
+       * all-or-nothing throw. Reconcile a lagging shard later with
+       * `refreshInsights({ only: [pk] })` or `refreshDerivation(pk)`.
        */
       async refreshInsights(options = {}) {
         if (this.crossVaultDerivations.length === 0) return { written: 0, skippedVaults: [] };
-        const { eligible, skipped } = await this.resolveEligible(
-          options.minVersion !== void 0 ? { minVersion: options.minVersion } : {}
-        );
+        const { eligible, skipped } = await this.resolveEligible({
+          ...options.minVersion !== void 0 ? { minVersion: options.minVersion } : {},
+          ...options.only !== void 0 ? { only: options.only } : {},
+          ...options.failFast !== void 0 ? { failFast: options.failFast } : {}
+        });
         let written = 0;
         for (const spec of this.crossVaultDerivations) {
           const results = await this.db.queryAcross(
@@ -1647,6 +1708,7 @@ var init_vault_group = __esm({
             const row = eligible[i];
             const res = results[i];
             if (!res || res.result === void 0) {
+              if (options.failFast && res?.error) throw res.error;
               skipped.push({ vaultId: row.vaultId, reason: "error", ...res?.error ? { error: res.error } : {} });
               continue;
             }
@@ -1662,6 +1724,14 @@ var init_vault_group = __esm({
         }
         return { written, skippedVaults: skipped };
       }
+      /**
+       * Reconcile one shard's Insight summaries after its backend was unreachable.
+       * Equivalent to `refreshInsights({ only: [partitionKey] })` — runs every
+       * registered derivation (autoPush or not) for just this shard.
+       */
+      async refreshDerivation(partitionKey) {
+        return this.refreshInsights({ only: [partitionKey] });
+      }
       /** @internal — re-derive + push every autoPush derivation's summary for one shard. */
       async _recomputeShardInsights(partitionKey) {
         const row = await this.registry.get(this.registryId(partitionKey));
@@ -1897,6 +1967,34 @@ var init_vault_group = __esm({
         }
         return { records: results, skippedVaults: skipped };
       }
+      /**
+       * @internal — distributed partial-reduce (#8). Fan out across eligible shards,
+       * but fold each shard's where-filtered records to a partial reducer STATE
+       * inside the per-shard callback (never returning the rows). Only the scalar
+       * `.aggregate()` path calls this, and that path rejects join legs upstream.
+       */
+      async fanoutReduce(spec, options = {}) {
+        const { eligible, skipped } = await this.group.resolveEligible(options);
+        const across = await this.group.db.queryAcross(
+          eligible.map((r) => r.vaultId),
+          async (vault) => {
+            this.group.template.configure(vault);
+            const coll = vault.collection(this.collectionName);
+            await coll.list();
+            let q = coll.query();
+            for (const c of this.clauses) q = q.where(c.field, c.op, c.value);
+            const rows = q.toArray();
+            return reduceToPartial(rows, spec);
+          },
+          { concurrency: options.concurrency ?? 1, create: false }
+        );
+        const partials = [];
+        for (const r of across) {
+          if (r.error) skipped.push({ vaultId: r.vault, reason: classifyShardSkip(r.error), error: r.error });
+          else partials.push(r.result);
+        }
+        return { partials, skippedVaults: skipped };
+      }
       /** Fan out across eligible shards, merge, then apply any broadcast dimension legs. */
       async toArray(options = {}) {
         const { records, skippedVaults } = await this.fanoutRecords(options);