@x12i/catalox 1.1.1 → 2.0.0

package/README.md

@@ -1,222 +1,424 @@
-# `@x12i/catalox`
-
-Catalox is a **data-tier** package for managing **app-scoped catalogs** in Firebase **Firestore**, including:
-
-- **Catalog discovery** for an `appId` (what catalogs are available + access)
-- **First-class catalog descriptors** (capabilities, query metadata, identity metadata, field metadata)
-- **Native catalogs** (items stored in Firestore)
-- **Mapped catalogs** (items normalized from MongoDB or APIs)
-- **References + validation contracts** (standardized cross-catalog shapes)
-- **Seed/import/export + batch upsert** workflows for native catalogs
-
-Catalox **does not** own UI, workflow orchestration, remote execution, artifact blobs, or secret storage.
-
-## Install
-
-This repo is currently set up as a workspace package.
-
-- **Node**: `>=20`
-- **TypeScript**: builds to `dist/`
-
-`@x12i/helpers` is a **required dependency** (installed as `@x12i/helpers@^1.2.0`).
-
-## Configuration (real connections)
-
-Catalox is a library: you provide initialized clients + runtime env.
-
-### Firestore (Firebase Admin SDK)
-
-This implementation of Catalox expects a **Firebase Admin SDK** Firestore instance (`firebase-admin`).
-
-Important clarification:
-
-- **“Admin” here means privileged access to your Firebase/GCP project’s Firestore**, using a service account.
-- It is **not** “admin of the host machine / server OS”.
-- Admin SDK access typically **bypasses Firestore Security Rules** (rules are for client SDKs).
-
-Scoping:
-
-- You can scope the service account to **Firestore-related IAM roles** and to a specific **project**.
-- You generally cannot scope an Admin SDK credential to only certain collections/documents via IAM the way client rules work.
-
-To connect to a real Firebase project, initialize `firebase-admin` using `GOOGLE_APPLICATION_CREDENTIALS` (service account JSON) or your preferred Admin initialization strategy.
-
-Minimal example:
-
-```ts
-import { initializeApp, applicationDefault } from "firebase-admin/app";
-import { getFirestore } from "firebase-admin/firestore";
-
-initializeApp({ credential: applicationDefault() });
-const firestore = getFirestore();
-```
-
-### Mongo (mapped catalogs)
-
-For Mongo-mapped catalogs, provide:
-
-- `MONGO_URI` (or whatever you store in adapter config as `mongoUriEnvVar`)
-
-Example `.env` (do not commit secrets):
-
-```bash
-MONGO_URI=mongodb://127.0.0.1:27017
-GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
-FIRESTORE_LIVE_TESTS=1
-```
-
-## Firestore data model (logical collections)
-
-Metadata:
-
-- `apps/{appId}`
-- `catalogs/{catalogId}`
-- `catalogBindings/{bindingId}` (many-to-many app↔catalog)
-- `catalogDefinitions/{catalogId}` (native vs mapped specifics)
-- `catalogAdapters/{adapterId}` (mongo/api adapter definitions)
-- `catalogMappings/{mappingId}` (field mapping specs)
-- `catalogDescriptors/{catalogId}` (**descriptor metadata** for generic consumption)
-- `catalogReferences/{referenceId}` (standardized reference records)
-
-Data:
-
-- `catalogData/{catalogId}/items/{itemId}` (native items)
-- `catalogSnapshots/{catalogId}/items/{itemId}` (mapped snapshot mode)
-
-## Core usage (generic from `appId`)
-
-### Create stores and `Catalox`
-
-```ts
-import { FirestoreStore } from "@x12i/catalox";
-import { AppStore, CatalogStore, BindingStore, DefinitionStore, MappingStore, DescriptorStore, ReferenceStore, NativeItemStore, SnapshotStore, AdapterStore } from "@x12i/catalox";
-import { AuthorizationService, Catalox } from "@x12i/catalox";
-
-// firebase-admin initialization is up to the consumer
-import { getFirestore } from "firebase-admin/firestore";
-
-const firestore = getFirestore();
-const store = new FirestoreStore({ firestore });
-
-const deps = {
-  apps: new AppStore(store),
-  catalogs: new CatalogStore(store),
-  bindings: new BindingStore(store),
-  definitions: new DefinitionStore(store),
-  mappings: new MappingStore(store),
-  descriptors: new DescriptorStore(store),
-  references: new ReferenceStore(store),
-  nativeItems: new NativeItemStore(store),
-  snapshots: new SnapshotStore(store),
-  adapters: new AdapterStore(store),
-  authz: new AuthorizationService(new BindingStore(store)),
-};
-
-const catalox = new Catalox(deps);
-```
-
-### What you need persisted for “generic consumption”
-
-To let consumers operate *purely* from `appId` with no hardcoded catalog registrations, Catalox relies on these being present in Firestore:
-
-- **Bindings**: `catalogBindings` determine which catalogs an app can see/use.
-- **Descriptors**: `catalogDescriptors/{catalogId}` provide capabilities, query metadata, identity metadata, and field metadata.
-
-### Discover catalogs for an app
-
-```ts
-const context = { appId: "myAppId" };
-const catalogs = await catalox.listAppCatalogs(context, { appId: "myAppId" });
-```
-
-### Get a catalog descriptor
-
-```ts
-const descriptor = await catalox.getCatalogDescriptor(context, "signals");
-```
-
-### Bootstrap (all descriptors accessible to an app)
-
-```ts
-const bootstrap = await catalox.getAppCatalogBootstrap(context, "myAppId");
-```
-
-### Common “generic client” flow (no hardcoded catalog logic)
-
-1) `listAppCatalogs(appId)` → get catalog list + access
-2) `getAppCatalogBootstrap(appId)` → get descriptors (capabilities/query/identity/fields)
-3) `listCatalogItems(catalogId, filter/sort)` → fetch normalized items
-4) `getCatalogItem(catalogId, itemId)` → fetch one item
-5) Optional: `getCatalogItemReferences(...)`, `validateCatalogItem(...)`
-
-## Native catalogs (seed/import/export + batch upsert)
-
-### Import/Export JSON (helpers)
-
-```ts
-const items = catalox.importCatalogItemsFromJson<Array<Record<string, unknown>>>(jsonString);
-const jsonOut = catalox.exportCatalogItemsToJson(items);
-```
-
-### Upsert one item (descriptor-driven identity)
-
-```ts
-await catalox.upsertNativeCatalogItem({ appId: "myAppId" }, "signals", {
-  categoryId: "core",
-  code: "S1",
-  title: "Example",
-  indexed: { categoryId: "core", code: "S1" }
-});
-```
-
-### Batch upsert
-
-```ts
-await catalox.batchUpsertNativeCatalogItems({ appId: "myAppId" }, "signals", items);
-```
-
-### List native items with equality filtering
-
-```ts
-const res = await catalox.listCatalogItems({ appId: "myAppId" }, "signals", {
-  filter: { categoryId: "core" }
-});
-```
-
-Filtering is performed on `indexed.<field>` in stored native records (payload remains in `data`).
-
-## References
-
-```ts
-const refs = await catalox.getCatalogItemReferences({ appId: "myAppId" }, "signals", "core:S1");
-```
-
-## Validation
-
-Catalog-level and item-level validation APIs exist with standardized contracts. (Item validation logic can be implemented by upstream domain rules while keeping the contract stable.)
-
-## Tests
-
-Unit tests (default):
-
-```bash
-npm test
-```
-
-Integration tests (live Firestore, no mocks/emulators):
-
-```bash
-npm run test:integration
-```
-
-Integration tests are **live-only** (no mocks, no emulators). They require:
-
-- `FIRESTORE_LIVE_TESTS=1`
-- `FIREBASE_SERVICE_ACCOUNT_PATH=...`
-- `FIREBASE_PROJECT_ID=...`
-
-## Boundaries (important)
-
-- **Secrets**: do not store secret material (API keys, cloud creds) in Catalox. Store only non-secret refs like `credentialsRef`.
-- **Artifacts**: store descriptor metadata and remote keys; artifact blobs live in external object storage.
-
+# `@x12i/catalox`
+
+Catalox is a **data-tier** package for managing **app-scoped catalogs** in Firebase **Firestore**, including:
+
+- **Catalog discovery** for an `appId` (what catalogs are available + access)
+- **First-class catalog descriptors** (capabilities, query metadata, identity metadata, field metadata)
+- **Native catalogs** (items stored in Firestore)
+- **Mapped catalogs** (items normalized from MongoDB or APIs)
+- **References + validation contracts** (standardized cross-catalog shapes)
+- **Seed/import/export + batch upsert** workflows for native catalogs
+
+Catalox **does not** own UI, workflow orchestration, remote execution, artifact blobs, or secret storage.
+
+## Install
+
+This repo is currently set up as a workspace package.
+
+- **Node**: `>=20`
+- **TypeScript**: builds to `dist/`
+
+`@x12i/helpers` is a **required dependency** (installed as `@x12i/helpers@^1.2.0`).
+
+## Configuration (real connections)
+
+Catalox is a library: you provide initialized clients + runtime env.
+
+### Firestore (Firebase Admin SDK)
+
+This implementation of Catalox expects a **Firebase Admin SDK** Firestore instance (`firebase-admin`).
+
+Important clarification:
+
+- **“Admin” here means privileged access to your Firebase/GCP project’s Firestore**, using a service account.
+- It is **not** “admin of the host machine / server OS”.
+- Admin SDK access typically **bypasses Firestore Security Rules** (rules are for client SDKs).
+
+Scoping:
+
+- You can scope the service account to **Firestore-related IAM roles** and to a specific **project**.
+- You generally cannot scope an Admin SDK credential to only certain collections/documents via IAM the way client rules work.
+
+To connect to a real Firebase project, initialize `firebase-admin` using `GOOGLE_APPLICATION_CREDENTIALS` (service account JSON) or your preferred Admin initialization strategy.
+
+Minimal example:
+
+```ts
+import { initializeApp, applicationDefault } from "firebase-admin/app";
+import { getFirestore } from "firebase-admin/firestore";
+
+initializeApp({ credential: applicationDefault() });
+const firestore = getFirestore();
+```
+
+### Mongo (mapped catalogs)
+
+For Mongo-mapped catalogs, provide:
+
+- `MONGO_URI` (or whatever you store in adapter config as `mongoUriEnvVar`)
+
+Example `.env` (do not commit secrets):
+
+```bash
+MONGO_URI=mongodb://127.0.0.1:27017
+GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
+FIRESTORE_LIVE_TESTS=1
+```
+
+## Firestore data model (logical collections)
+
+Metadata:
+
+- `apps/{appId}`
+- `catalogs/{catalogId}`
+- `catalogBindings/{bindingId}` (many-to-many app↔catalog)
+- `catalogDefinitions/{catalogId}` (native vs mapped specifics)
+- `catalogAdapters/{adapterId}` (mongo/api adapter definitions)
+- `catalogMappings/{mappingId}` (field mapping specs)
+- `catalogDescriptors/{catalogId}` (**descriptor metadata** for generic consumption)
+- `catalogReferences/{referenceId}` (standardized reference records)
+
+Data:
+
+- `catalogData/{catalogId}/items/{itemId}` (native items)
+- `catalogSnapshots/{catalogId}/items/{itemId}` (mapped snapshot mode)
+
+## Core usage (generic from `appId`)
+
+### Create stores and `Catalox`
+
+```ts
+import { FirestoreStore } from "@x12i/catalox";
+import { AppStore, CatalogStore, BindingStore, DefinitionStore, MappingStore, DescriptorStore, ReferenceStore, NativeItemStore, SnapshotStore, AdapterStore } from "@x12i/catalox";
+import { AuthorizationService, Catalox } from "@x12i/catalox";
+
+// firebase-admin initialization is up to the consumer
+import { getFirestore } from "firebase-admin/firestore";
+
+const firestore = getFirestore();
+const store = new FirestoreStore({ firestore });
+
+const deps = {
+  apps: new AppStore(store),
+  catalogs: new CatalogStore(store),
+  bindings: new BindingStore(store),
+  definitions: new DefinitionStore(store),
+  mappings: new MappingStore(store),
+  descriptors: new DescriptorStore(store),
+  references: new ReferenceStore(store),
+  nativeItems: new NativeItemStore(store),
+  snapshots: new SnapshotStore(store),
+  adapters: new AdapterStore(store),
+  authz: new AuthorizationService(new BindingStore(store)),
+};
+
+const catalox = new Catalox(deps);
+```
+
+## Descriptor contract (planning-critical)
+
+Catalox is designed so upstream packages can be “generic” (no hardcoded catalog registrations). The stable contract is the persisted **descriptor**:
+
+- Stored at `catalogDescriptors/{catalogId}`
+- Retrieved via `getCatalogDescriptor(...)` or `getAppCatalogBootstrap(...)`
+- Used for identity, query, and client rendering metadata
+
+Below is the **actual current descriptor shape** (TypeScript), with the most planning-relevant subtypes.
+
+```ts
+export type CatalogCapabilitiesDescriptor = {
+  canList: boolean;
+  canGet: boolean;
+  canCreate: boolean;
+  canEdit: boolean;
+  canDelete: boolean;
+  canImport?: boolean;
+  canExport?: boolean;
+  canSync?: boolean;
+  canValidate?: boolean;
+  canViewReferences?: boolean;
+};
+
+export type CatalogFieldDescriptor = {
+  key: string;
+  label: string;
+  type:
+    | "string"
+    | "number"
+    | "boolean"
+    | "date"
+    | "datetime"
+    | "enum"
+    | "object"
+    | "array"
+    | "reference";
+  // Optional path into the stored item payload. If omitted, `key` is assumed.
+  path?: string;
+
+  // Query + indexing metadata.
+  filterable?: boolean;
+  sortable?: boolean;
+  indexed?: boolean;
+  multiValue?: boolean;
+
+  // Presentation + contract metadata.
+  listVisible?: boolean;
+  detailVisible?: boolean;
+  required?: boolean;
+  enumValues?: Array<string | number>;
+  reference?: { targetCatalogId?: string; targetField?: string };
+  metadata?: Record<string, unknown>;
+};
+
+export type CatalogIdentityDescriptor = {
+  itemIdStrategy: "natural" | "composite" | "generated";
+  itemIdField?: string;
+  compositeFields?: string[];
+
+  // Optional display decoration fields.
+  titleField?: string;
+  subtitleField?: string;
+  statusField?: string;
+  updatedAtField?: string;
+};
+
+export type CatalogDescriptor = {
+  catalogId: string;
+  label: string;
+  description?: string;
+  itemLabel?: string;
+  sourceMode: "native" | "mapped";
+  mappedSourceType?: "mongo" | "api";
+  status: "active" | "disabled" | "draft";
+  visibility?: "visible" | "hidden";
+  defaultSort?: { field: string; direction: "asc" | "desc" };
+  defaultFilters?: Record<string, unknown>;
+  capabilities: CatalogCapabilitiesDescriptor;
+  queryableFields: CatalogFieldDescriptor[];
+  queryCapabilities?: Record<string, unknown>;
+  filterSpec?: Record<string, unknown>;
+  presentationSpec?: Record<string, unknown>;
+  customRenderer?: Record<string, unknown>;
+  identity: CatalogIdentityDescriptor;
+  metadata?: Record<string, unknown>;
+};
+```
+
+### UI metadata & custom renderers (optional)
+
+Catalog descriptors can include optional UI metadata for generic presentation layers:
+
+- `filterSpec`: declarative filter configuration (built on `FieldSource`)
+- `presentationSpec`: declarative layout + view/edit semantics (grid/list/cards/form)
+- `customRenderer`: escape hatch that assigns a host-resolved JSX component (by registry key) and passes a stable render-map contract
+
+Custom renderer contracts:
+
+- [`docs/catalog-list-render-map.md`](docs/catalog-list-render-map.md)
+- [`docs/catalog-item-render-map.md`](docs/catalog-item-render-map.md)
+
+Important: host presentation layers are responsible for resolving `FieldSource` lookups (cross-catalog enums/refs) and populating `resolvedSources` for renderers.
+
+### What you need persisted for “generic consumption”
+
+To let consumers operate *purely* from `appId` with no hardcoded catalog registrations, Catalox relies on these being present in Firestore:
+
+- **Bindings**: `catalogBindings` determine which catalogs an app can see/use.
+- **Descriptors**: `catalogDescriptors/{catalogId}` provide capabilities, query metadata, identity metadata, and field metadata.
+
+Minimum viable setup for generic consumption is **catalog + descriptor + binding**.
+
+For **mapped** catalogs, the minimum viable setup is also **definition + mapping + adapter** (created automatically if you use `createCatalog` with `sourceMode: "mapped"`).
+
+### Discover catalogs for an app
+
+```ts
+const context = { appId: "myAppId" };
+const catalogs = await catalox.listAppCatalogs(context, { appId: "myAppId" });
+```
+
+### Get a catalog descriptor
+
+```ts
+const descriptor = await catalox.getCatalogDescriptor(context, "signals");
+```
+
+### Bootstrap (all descriptors accessible to an app)
+
+```ts
+const bootstrap = await catalox.getAppCatalogBootstrap(context, "myAppId");
+```
+
+### Common “generic client” flow (no hardcoded catalog logic)
+
+1) `listAppCatalogs(appId)` → get catalog list + access
+2) `getAppCatalogBootstrap(appId)` → get descriptors (capabilities/query/identity/fields)
+3) `listCatalogItems(catalogId, filter/sort)` → fetch normalized items
+4) `getCatalogItem(catalogId, itemId)` → fetch one item
+5) Optional: `getCatalogItemReferences(...)`, `validateCatalogItem(...)`
+
+## Provisioning (create catalog + bind + descriptor)
+
+Catalox includes provisioning helpers to reduce “manual Firestore wiring”.
+
+### Create a native catalog (also seeds a minimal descriptor)
+
+```ts
+const ctx = { appId: "myAppId", isGodMode: true }; // or bind admin access appropriately
+
+await catalox.createCatalog(ctx, {
+  catalogId: "signals",
+  name: "Signals",
+  sourceMode: "native",
+  native: { type: "native", firestoreCollectionPath: "catalogData/signals/items" },
+});
+```
+
+### Bind a catalog to an app (enables discovery + access)
+
+```ts
+await catalox.bindCatalogToApp(ctx, {
+  appId: "myAppId",
+  catalogId: "signals",
+  access: { canRead: true, canWrite: true, canAdmin: true },
+});
+```
+
+### Patch/upsert a descriptor (recommended for identity + query metadata)
+
+There is currently no `Catalox.setCatalogDescriptor(...)` method; consumers can upsert the persisted record via `DescriptorStore`.
+
+```ts
+await descriptors.upsert({
+  catalogId: "signals",
+  descriptorVersion: "2",
+  descriptor: {
+    catalogId: "signals",
+    label: "Signals",
+    sourceMode: "native",
+    status: "active",
+    capabilities: { canList: true, canGet: true, canCreate: true, canEdit: true, canDelete: true },
+    queryableFields: [
+      { key: "categoryId", label: "Category", type: "string", indexed: true, filterable: true },
+      { key: "code", label: "Code", type: "string", indexed: true, filterable: true },
+      { key: "title", label: "Title", type: "string" },
+    ],
+    identity: {
+      itemIdStrategy: "composite",
+      compositeFields: ["categoryId", "code"],
+      titleField: "title",
+    },
+  },
+  createdAt: new Date().toISOString(),
+  updatedAt: new Date().toISOString(),
+});
+```
+
+### Ensure helpers (idempotent)
+
+- `ensureCatalog(...)`: creates a minimal catalog record if missing (requires admin binding access).
+- `ensureBinding(...)`: creates a binding if missing (enforces god-mode for cross-app provisioning).
+
+## Native catalogs (seed/import/export + batch upsert)
+
+### Import/Export JSON (helpers)
+
+```ts
+const items = catalox.importCatalogItemsFromJson<Array<Record<string, unknown>>>(jsonString);
+const jsonOut = catalox.exportCatalogItemsToJson(items);
+```
+
+### Upsert one item (descriptor-driven identity)
+
+```ts
+await catalox.upsertNativeCatalogItem({ appId: "myAppId" }, "signals", {
+  categoryId: "core",
+  code: "S1",
+  title: "Example",
+  indexed: { categoryId: "core", code: "S1" }
+});
+```
+
+### Native write contract (exact)
+
+- **Input shape**: the write API accepts a plain object payload. `indexed` is a reserved top-level field used only for query performance and is **not** part of the domain payload.
+- **Stored shape** (native): items are stored as `{ itemId, catalogId, indexed, data }`.
+- **Identity**: `itemId` is resolved from `descriptor.identity`.
+  - `natural`: uses `identity.itemIdField` from the input payload.
+  - `composite`: joins `identity.compositeFields` with `:`.
+  - `generated`: **currently requires caller-supplied id** (Catalox will throw if descriptor uses `generated` and no id is provided).
+
+### Batch upsert
+
+```ts
+await catalox.batchUpsertNativeCatalogItems({ appId: "myAppId" }, "signals", items);
+```
+
+### List native items with equality filtering
+
+```ts
+const res = await catalox.listCatalogItems({ appId: "myAppId" }, "signals", {
+  filter: { categoryId: "core" }
+});
+```
+
+Filtering is performed on `indexed.<field>` in stored native records (payload remains in `data`).
+
+### Canonical `indexed` rule (native catalogs)
+
+- **Caller may provide `indexed`**: `indexed: { ... }` is accepted on writes.
+- **Catalox may derive `indexed`**: if omitted, Catalox derives `indexed` from the descriptor’s `queryableFields` (fields marked `indexed: true`, using `field.path ?? field.key`).
+- **Data cleanliness**: `indexed` is treated as reserved metadata and is **not duplicated inside `data`**.
+- **If `indexed` is missing/wrong**: equality filtering and indexed sorting may return incomplete results or fail due to missing Firestore indexes; the canonical fix is to correct the catalog descriptor and/or the write input and re-upsert.
+
+## References
+
+```ts
+const refs = await catalox.getCatalogItemReferences({ appId: "myAppId" }, "signals", "core:S1");
+```
+
+### Reference write/provision model (current)
+
+- References are persisted in `catalogReferences/{referenceId}`.
+- Catalox currently **reads** references via `getCatalogItemReferences(...)`; reference creation/maintenance is **consumer-owned** for now.
+- `referenceId` is the Firestore document id; consumers should adopt a deterministic scheme, e.g.:
+  - `${fromCatalogId}:${fromItemId}:${relationType}:${toCatalogId}:${toItemId}`
+
+## Validation
+
+Validation APIs exist with standardized contracts, but **current behavior is minimal**:
+
+- `validateCatalog(...)` and `validateCatalogItem(...)` currently return `{ isValid: true, issues: [] }`.
+- Upstream domain validation is expected to be layered on in later work while keeping the response contract stable.
+
+## Tests
+
+Unit tests (default):
+
+```bash
+npm test
+```
+
+Integration tests (live Firestore, no mocks/emulators):
+
+```bash
+npm run test:integration
+```
+
+Integration tests are **live-only** (no mocks, no emulators). They require:
+
+- `FIRESTORE_LIVE_TESTS=1`
+- `FIREBASE_SERVICE_ACCOUNT_PATH=...`
+- `FIREBASE_PROJECT_ID=...`
+
+### Live test safety (read before running)
+
+- **Never run against production credentials/projects.** Use a dedicated Firebase project for tests.
+- **Touched collections**: `apps`, `catalogs`, `catalogBindings`, `catalogDefinitions`, `catalogDescriptors`, and `catalogData/{catalogId}/items/...`.
+- **Cleanup**: tests do best-effort deletes of the docs they created, but do not guarantee full teardown.
+
+## Boundaries (important)
+
+- **Secrets**: do not store secret material (API keys, cloud creds) in Catalox. Store only non-secret refs like `credentialsRef`.
+- **Artifacts**: store descriptor metadata and remote keys; artifact blobs live in external object storage.
+