@x12i/memorix-retrieval 1.2.0 → 1.4.0

package/docs/MEMORIX-CATALOX-CONTRACTS.md

@@ -0,0 +1,540 @@
+# Memorix ↔ Catalox contracts
+
+**Canonical sync document** for all Memorix data packages and Memorix-consuming UIs (Explorer, completion pipelines, ingestion, graph tools).
+
+
+| Document | Scope |
+|----------|--------|
+| **This file** | Catalox catalogs, descriptor JSON formats, expected Mongo shape, cross-component behavior |
+| [MEMORIX-DATABASE-CONVENTIONS.md](./MEMORIX-DATABASE-CONVENTIONS.md) | Database names, collection naming, env vars, record envelope |
+| [DATA-TIER-CONTRACT.md](./DATA-TIER-CONTRACT.md) | What host apps may call (retrieval APIs only) |
+
+**Reference implementation:** `@x12i/memorix-retrieval` — descriptor types in `src/descriptors/descriptor-types.ts`, seeds in `catalox-seeds/inputs/`, validation in `src/descriptors/validate-descriptor.ts`.
+
+If this document and a package disagree, **update both together**.
+
+---
+
+## 1. Split of responsibilities
+
+```text
+┌─────────────────────────────────────────────────────────────────┐
+│ Catalox (metadata)                                               │
+│  • What entities exist                                           │
+│  • How to list / detail / relate them                            │
+│  • Property paths, filters, sorts, relations                     │
+└───────────────────────────────┬─────────────────────────────────┘
+                                │ listCatalogItems / getCatalogItem
+                                ▼
+┌─────────────────────────────────────────────────────────────────┐
+│ @x12i/memorix-retrieval (data tier)                              │
+│  • Load descriptors from Catalox                                 │
+│  • Resolve Mongo DB + collection from descriptor                 │
+│  • Read, compose, paginate, multi-match, relations, content      │
+└───────────────────────────────┬─────────────────────────────────┘
+                                │ fetchMemorixList / fetchMemorixItem / graph
+                                ▼
+┌─────────────────────────────────────────────────────────────────┐
+│ Host apps (Explorer, APIs, tools)                                │
+│  • Never bypass descriptors for entity I/O                       │
+│  • Never invent entity lists from env or collection names        │
+└─────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────┐
+│ MongoDB (payload)                                                │
+│  • memorix-entities / memorix-events                             │
+│  • Domain documents under `data`                                 │
+└─────────────────────────────────────────────────────────────────┘
+```
+
+| Layer | Owns | Must not own |
+|-------|------|--------------|
+| **Catalox** | Descriptor catalogs, item ids, scope tags, graph semantics | Raw Mongo queries, business payload writes |
+| **memorix-retrieval** | Descriptor-driven reads, composition, validation | Ad hoc catalog types outside the three Memorix catalogs |
+| **Completion / ingestion** | Writing `data` on Memorix records per [MEMORIX-DATABASE-CONVENTIONS](./MEMORIX-DATABASE-CONVENTIONS.md) | Defining list columns or UI field layouts (that's descriptors) |
+| **Explorer / UIs** | Presentation, routing, design model | Direct Mongo, xmemory-store, env-based entity discovery |
+
+---
+
+## 2. Catalox app and catalogs
+
+### 2.1 App id
+
+| Constant | Value |
+|----------|-------|
+| Default `appId` | `memorix` |
+| Env override | `CATALOX_APP_ID` (fallback: `MEMORIX_APP_ID`) |
+
+All Memorix descriptor catalogs live under this app. Peers must use the same `appId` unless a deployment explicitly namespaces apps. `createMemorixRetrieval`, `createMemorixRetrievalFromEnv`, and `createMemorixRetrievalStackFromEnv` resolve the app id via `resolveMemorixAppId()`.
+
+### 2.2 Catalog ids (only these three)
+
+| Catalog id | Item id field | Title field | Purpose |
+|------------|---------------|-------------|---------|
+| `memorix-entity-descriptors` | `id` | `entityName` | Entity schema: properties, content types, relations, default list/item |
+| `memorix-list-descriptors` | `id` | `title` | Tabular list views |
+| `memorix-item-descriptors` | `id` | `title` | Record detail layout |
+
+Catalogs use **`sourceMode: "native"`** and **`catalogType: "memorix"`** in seed manifests. Do not introduce parallel catalog names (e.g. `memorix_entity_content_types`) for new work.
+
+### 2.3 Item shape in Catalox
+
+Each catalog item is stored as:
+
+```json
+{
+  "id": "<descriptor-id>",
+  "...": "<descriptor fields — see §3>",
+  "scope": {
+    "domains": ["network", "vulnerabilities"],
+    "agents": ["neo"]
+  }
+}
+```
+
+- **`id`** must match the descriptor's own `id` field.
+- **`scope`** is applied at seed time from `catalox-seeds/inputs/scope.json`. All items in a seed bundle share the same scope unless a future manifest explicitly overrides per item.
+- Descriptor payload lives in the item **`data`** body when read via Catalox APIs.
+- **Seed apply note:** the Catalox seed CLI upserts `row.data` only, so `scripts/apply-seed-manifest.mjs` merges `scope` into the upsert `data` object. Hosts may see `scope` inside `data` until Catalox stores top-level scope natively; retrieval ignores unknown fields when validating descriptors.
+
+### 2.4 Expected Catalox client behavior
+
+Peers that integrate with Catalox must support:
+
+| Operation | Used for |
+|-----------|----------|
+| `listCatalogItems(context, catalogId, { limit, offset })` | Entity discovery, paginated catalog reads |
+| `getCatalogItem(context, catalogId, itemId)` | Load list/item/entity descriptor by id |
+
+**Discovery rule:** entity existence comes **only** from listing `memorix-entity-descriptors`. There is no env fallback (`MEMORIX_ENTITY_NAMES` and similar bypasses are forbidden in retrieval and hosts).
+
+If the catalog is empty or Catalox is unreachable, discovery returns `source: "none"` with a hint to run the seed apply script.
+
+---
+
+## 3. Descriptor formats
+
+Types below mirror `MemorixEntityDescriptor`, `MemorixListDescriptor`, and `MemorixItemDescriptor` in `@x12i/memorix-retrieval`.
+
+### 3.1 Entity descriptor (`memorix-entity-descriptors`)
+
+**Required top-level fields:**
+
+| Field | Type | Rules |
+|-------|------|-------|
+| `id` | string | Stable catalog item id; usually equals `entityName` |
+| `entityName` | string | Kebab-case domain id (`vulnerabilities`, `variabilities-groups`, `assets`) |
+| `defaultListDescriptorId` | string | **Option A** — id in `memorix-list-descriptors` |
+| `defaultItemDescriptorId` | string | **Option A** — id in `memorix-item-descriptors` |
+| `collectionPrefix` | string | Used with content-type `postfix` when `collection` is omitted |
+| `identity` | object | See below |
+| `defaults` | object | Canonical content type, `dataRoot`, effective-date paths |
+| `contentTypes` | object | Named content slices (see §3.4) |
+| `properties` | object | Field dictionary for list/item composition |
+
+**Optional:**
+
+| Field | Type | Default |
+|-------|------|---------|
+| `target` | `"entity"` \| `"event"` | `"entity"` — selects `memorix-entities` vs `memorix-events` |
+| `contentDefaults` | object | Default content-object storage/format |
+| `relations` | object | Cross-entity joins declared on this entity |
+
+**Identity block:**
+
+```json
+{
+  "allowedIdFields": ["entityId", "eventId"],
+  "requiredExactlyOne": true,
+  "defaultIdField": "entityId"
+}
+```
+
+**Defaults block:**
+
+```json
+{
+  "canonicalContentType": "snapshots",
+  "dataRoot": "data",
+  "effectiveDatePath": "capturedAt",
+  "fallbackEffectiveDatePaths": ["snapshot.capturedAt", "data.enrichment.enrichedAt"]
+}
+```
+
+**Property descriptor** (each key under `properties`):
+
+```json
+{
+  "label": "Asset IP",
+  "source": { "contentType": "snapshots", "path": "data.assetIp" },
+  "humanReadable": true,
+  "sortable": true,
+  "filterable": true,
+  "list": true,
+  "item": true,
+  "valueType": "string"
+}
+```
+
+| `valueType` | Notes |
+|-------------|-------|
+| `string`, `number`, `boolean`, `date`, `datetime`, `enum`, `object`, `array` | Scalar / structural |
+| `content` | Value is a `memorix-content-object` (see §3.7) |
+
+List views may only expose properties with **`humanReadable: true`**.
+
+**Relation descriptor** (under `relations`):
+
+```json
+{
+  "targetEntity": "assets",
+  "type": "manyToOne",
+  "source": { "contentType": "snapshots", "path": "data.assetIp" },
+  "target": { "contentType": "snapshots", "path": "data.ip_address" },
+  "defaultMode": "extendFields",
+  "defaultArrayProperty": "vulnerabilities",
+  "targetFields": ["ipAddress", "hostName"]
+}
+```
+
+| `type` | Meaning |
+|--------|---------|
+| `oneToOne`, `oneToMany`, `manyToOne`, `manyToMany` | Graph semantics for Explorer and `includeRelations` |
+
+Join paths are **Mongo document paths** on the respective content types, not Catalox ids.
+
+### 3.2 List descriptor (`memorix-list-descriptors`)
+
+**Required:**
+
+| Field | Rules |
+|-------|-------|
+| `id` | Unique list id (e.g. `vulnerabilities-main-list`) |
+| `entity` | `entityName` referencing an entity descriptor |
+| `leadingContentType` | Content type key used to drive pagination/filter/sort |
+| `pagination` | `{ "enabled": true, "defaultLimit": 50, "maxLimit": 200 }` |
+| `fields` | Non-empty array of **property keys** on the entity descriptor |
+
+**Common optional fields:**
+
+| Field | Purpose |
+|-------|---------|
+| `title` | Human label |
+| `filters` | Default filters (always applied) plus allowlist for request filters; request overrides same property |
+| `allowedSorts`, `defaultSort` | Sort whitelist and default |
+| `extensions` | Pull extra content types by identity (see §3.5) |
+| `includeRelations` | Embed relation data in list rows |
+| `allowSortDrivenLeadingOverride` | Allow sort to switch pagination driver content type |
+
+**Example filter:**
+
+```json
+{ "property": "severityLevel", "operator": "gte", "value": 4 }
+```
+
+**Filter operators:** `eq`, `ne`, `in`, `nin`, `gt`, `gte`, `lt`, `lte`, `exists`, `regex`.
+
+Each entity descriptor should define at least one list whose `id` equals `defaultListDescriptorId`. Additional lists (e.g. `critical-vulnerabilities-list`) are optional alternate views.
+
+### 3.3 Item descriptor (`memorix-item-descriptors`)
+
+**Required:**
+
+| Field | Rules |
+|-------|-------|
+| `id` | Unique item id (e.g. `vulnerability-detail-item`) |
+| `entity` | `entityName` |
+| `identity.idField` | `"entityId"` or `"eventId"` — must match request |
+| `contentTypes` | Array of `{ contentType, required, multiMatch? }` |
+| `sections` | Array of `{ id, title, fields[] }` — `fields` are property keys |
+
+**Optional:**
+
+| Field | Purpose |
+|-------|---------|
+| `includeRelations` | `{ relation, mode?, fields? }` — relation key from entity descriptor |
+| `content` | `{ allowed, defaultIncludeFullContent?, allowedFields?, maxBytes? }` |
+
+Each entity descriptor should define at least one item whose `id` equals `defaultItemDescriptorId`.
+
+### 3.4 Content type descriptor
+
+Under `entity.contentTypes.<name>`:
+
+```json
+{
+  "postfix": "snapshots",
+  "collection": "vulnerabilities-snapshots",
+  "dataRoot": "data",
+  "isCanonical": true,
+  "effectiveDatePath": "capturedAt",
+  "fallbackEffectiveDatePaths": ["snapshot.capturedAt"]
+}
+```
+
+**Collection resolution order** (retrieval):
+
+1. Explicit `collection` on the content type
+2. Env `MEMORIX_ENTITIES_COLLECTION_<ENTITY>` or `MEMORIX_EVENTS_COLLECTION_<ENTITY>`
+3. Heuristic: `{collectionPrefix}-{postfix}` (e.g. `assets-snapshots`)
+
+**Database resolution:** entity descriptor `target` → `memorix-entities` or `memorix-events` (see [MEMORIX-DATABASE-CONVENTIONS](./MEMORIX-DATABASE-CONVENTIONS.md)).
+
+Exactly one content type per entity should set **`isCanonical: true`**; it aligns with `defaults.canonicalContentType`.
+
+### 3.5 Content extensions (list)
+
+```json
+{
+  "contentType": "enrichment",
+  "mode": "extendFields",
+  "join": { "by": "entityId" },
+  "multiMatch": { "strategy": "last", "effectiveDatePath": "capturedAt" },
+  "fields": { "enrichedAt": "data.enrichment.enrichedAt" }
+}
+```
+
+| `mode` | Behavior |
+|--------|----------|
+| `extendFields` | Merge selected paths onto the row |
+| `array` | Attach array under `arrayProperty` |
+
+### 3.6 Multi-match
+
+When multiple Mongo documents share the same identity, retrieval picks records using:
+
+```json
+{
+  "strategy": "last" | "first" | "all" | "error" | "ignore",
+  "effectiveDatePath": "capturedAt",
+  "fallbackEffectiveDatePaths": ["snapshot.capturedAt"]
+}
+```
+
+Default for item fetches: **`last`** by effective date. Strategies are declared on item `contentTypes`, content-type defaults, or extension blocks.
+
+### 3.7 Content objects
+
+Properties with `valueType: "content"` store a **Memorix content object** in Mongo:
+
+```json
+{
+  "contentKey": "stories/asset-10-150-68-31.md",
+  "metadata": { "title": "Impact story" },
+  "preview": "First 200 chars…"
+}
+```
+
+Storage is configured on the entity (`contentDefaults`) or property (`content.storage`), never with inline credentials:
+
+```json
+{
+  "provider": "gcs" | "s3",
+  "bucket": "memorix-content",
+  "prefix": "stories/",
+  "maxBytes": 65536,
+  "encoding": "utf8",
+  "credentialsRef": "env:MEMORIX_GCS_CREDENTIALS"
+}
+```
+
+Hosts provide **`contentReaders`** at retrieval bootstrap; list views return preview/metadata only (`listBehavior.fetchContent: false`).
+
+---
+
+## 4. What we expect to find in Mongo
+
+### 4.1 Record envelope
+
+Documents in Memorix target databases follow [MEMORIX-DATABASE-CONVENTIONS](./MEMORIX-DATABASE-CONVENTIONS.md):
+
+```json
+{
+  "_id": "<ObjectId>",
+  "recordId": "<optional>",
+  "entityId": "<for entity-targeted records>",
+  "eventId": "<for event-targeted records>",
+  "capturedAt": "<ISO-8601>",
+  "data": { "... domain fields referenced by descriptor paths ..." }
+}
+```
+
+**Peer rules:**
+
+- Domain payload lives under **`data`** (or content-type-specific `dataRoot`, almost always `data`).
+- Descriptor `source.path` values are **dot paths from the document root** (e.g. `data.assetIp`, not `assetIp` alone).
+- Identity for list pagination and item fetch: **`entityId`** or **`eventId`** as declared on the item descriptor.
+- Legacy `snapshotId` may be read as `recordId` fallback; do not write it in new code.
+
+### 4.2 Shipped entity ↔ database mapping
+
+| Entity (`entityName`) | `target` | Typical collection(s) |
+|-----------------------|----------|------------------------|
+| `assets` | `entity` | `assets-snapshots` in `memorix-entities` |
+| `variabilities-groups` | `entity` | `variabilities-groups-snapshots` |
+| `vulnerabilities` | `event` | `vulnerabilities-snapshots` in `memorix-events` |
+
+Ingestion and completion must populate paths referenced in entity descriptors. If a property path is missing, retrieval returns null/omits the field and may emit **`EXTENSION_MISSING`** issues for required content types.
+
+### 4.3 Relations in data
+
+Relations are **not** stored as Catalox edges. They are resolved at read time by matching:
+
+- `relation.source.path` on the source record
+- `relation.target.path` on target entity records
+
+Both sides must use the same **`contentType`** keys declared on the relation. Target field projection uses `targetFields` / `includeRelations.fields`.
+
+---
+
+## 5. Runtime behavior (retrieval)
+
+All Memorix-consuming components should rely on these behaviors rather than reimplementing them. Public exports are listed in [DATA-TIER-CONTRACT.md](./DATA-TIER-CONTRACT.md).
+
+**Host bootstrap:** prefer `createMemorixRetrievalStackFromEnv()` for Catalox + Mongo wiring; see [EXPLORER-HOST-APIS.md](./EXPLORER-HOST-APIS.md) for graph/raw-read helpers.
+
+### 5.1 Discovery
+
+```
+discoverMemorixEntities(client)
+  → list all items in memorix-entity-descriptors
+  → validate each as MemorixEntityDescriptor
+  → return summaries: id, entityName, label, defaultListDescriptorId, defaultItemDescriptorId, target
+```
+
+### 5.2 Entity-default list and item
+
+```
+fetchMemorixListForEntity(client, { entityName, ... })
+  → load entity descriptor
+  → listId = entity.defaultListDescriptorId
+  → fetchMemorixList(client, { listId, ... })
+
+fetchMemorixItemForEntity(client, { entityName, entityId | eventId, ... })
+  → load entity descriptor
+  → itemDescriptorId = entity.defaultItemDescriptorId
+  → fetchMemorixItem(client, { itemDescriptorId, ... })
+```
+
+### 5.3 List composition
+
+1. Load list + entity descriptors from Catalox.
+2. Validate every `fields[]` entry exists and is `humanReadable`.
+3. Resolve pagination driver = `leadingContentType` (or sort override if allowed).
+4. Query driver collection with list default filters merged with request filters, plus optional `searchText` across human-readable fields.
+5. Optionally batch-fetch extensions and relations.
+6. Return rows as flat property maps + pagination + issues.
+
+### 5.4 Item composition
+
+1. Load item + entity descriptors.
+2. Fetch all documents per `contentTypes[]` matching identity.
+3. Apply `multiMatch` per content type.
+4. Build `sections[]` from property definitions.
+5. Resolve `includeRelations` and optional full content fetches.
+6. Return `{ identity, sections, relations?, issues? }`.
+
+### 5.5 Entity graph (Explorer)
+
+`buildMemorixEntityGraph` uses discovery + entity descriptors only:
+
+- Per entity: label, counts, content-type labels, **`relations` → connections**, default list/item ids.
+- `discovery.source` is `"catalox"` or `"none"` — never `"env"`.
+
+---
+
+## 6. Seeding and keeping peers in sync
+
+### 6.1 Source of truth
+
+| Artifact | Location |
+|----------|----------|
+| Editable seed inputs | `@x12i/memorix-retrieval` → `catalox-seeds/inputs/` |
+| Built manifest | `catalox-seeds/memorix-retrieval-descriptors.manifest.json` |
+| TypeScript types | `src/descriptors/descriptor-types.ts` |
+
+Workflow for descriptor changes:
+
+```bash
+# In memorix-retrieval
+npm run catalox:seed:build
+npm run catalox:seed:memorix-retrieval:validate
+npm run catalox:seed:memorix-retrieval:apply   # force apply
+npm run catalox:seed:ensure                    # apply only if items missing
+```
+
+Seed helpers (`buildMemorixRetrievalSeedManifest`, `checkMemorixRetrievalSeedNeeds`, …) are exported from the package; `scripts/` wrap them for CLI use. See [DATA-TIER-CONTRACT.md](./DATA-TIER-CONTRACT.md#catalox-seed-helpers-library).
+
+Then bump `@x12i/memorix-retrieval` and reinstall in consuming apps.
+
+### 6.2 Adding a new entity (checklist)
+
+1. Add `catalox-seeds/inputs/entity-descriptors/<name>.json` with **both default list and item ids**.
+2. Add at least one list under `list-descriptors/` and one item under `item-descriptors/`.
+3. Register ids in `catalox-seeds/inputs/manifest.json`.
+4. Rebuild and apply seed manifest.
+5. Ensure Mongo collections exist and documents match declared paths.
+6. Extend `src/tests/fixtures.ts` and validation tests in retrieval.
+7. Publish retrieval; update Explorer (or other hosts) dependency.
+
+### 6.3 Versioning
+
+- **Descriptor schema changes** → minor/major bump of `@x12i/memorix-retrieval` + coordinated seed apply.
+- **Seed-only changes** (fields, filters, new list views) → seed apply + retrieval patch as needed; hosts pick up via graph/list/item APIs automatically.
+- Do not fork catalog ids per app; use **`scope`** for domain/agent filtering instead.
+
+---
+
+## 7. Component matrix
+
+| Component | Reads Catalox | Writes Catalox | Reads Mongo | Writes Mongo |
+|-----------|---------------|----------------|-------------|--------------|
+| `@x12i/memorix-retrieval` | Yes (3 catalogs) | No (seeds via CLI) | Yes (descriptor-driven) | No |
+| `@x12i/memorix-completion` | No* | No | Yes (source + targets) | Yes (`data` on targets) |
+| `@x12i/memorix-explorer` | Via retrieval | No | No (via retrieval only) | No |
+| Ingestion pipelines | No* | No | Yes | Yes |
+
+\*Completion and ingestion should **align** with descriptor paths and collection names documented here; they do not need Catalox at runtime unless they also ship seed tooling.
+
+---
+
+## 8. Forbidden patterns
+
+These break cross-component sync and must not appear in new code:
+
+| Pattern | Why |
+|---------|-----|
+| Host Mongo list/get/count by collection name | Bypasses list/item descriptors |
+| `MEMORIX_ENTITY_NAMES` or env entity lists | Bypasses Catalox discovery |
+| `@x12i/xmemory-store` / scoped tier in Explorer | Legacy; replaced by retrieval + descriptors |
+| Parallel catalog types for the same semantics | Fragments truth |
+| Credentials inside descriptor JSON | Use `credentialsRef` + host-injected readers |
+| List fields with `humanReadable: false` | Validation error at fetch time |
+| Entity descriptor missing `defaultListDescriptorId` / `defaultItemDescriptorId` | Breaks Option A wiring |
+
+---
+
+## 9. Quick reference — shipped ids
+
+From `catalox-seeds/inputs/manifest.json`:
+
+**Entities:** `assets`, `vulnerabilities`, `variabilities-groups`
+
+**Lists:** `assets-main-list`, `vulnerabilities-main-list`, `critical-vulnerabilities-list`, `variabilities-groups-main-list`
+
+**Items:** `asset-detail-item`, `vulnerability-detail-item`, `variabilities-group-detail-item`
+
+**Scope (all seeded items):** `domains: [network, vulnerabilities]`, `agents: [neo]`
+
+---
+
+## 10. Related packages
+
+| Package | Role |
+|---------|------|
+| `@x12i/catalox` | Catalog storage and seed CLI |
+| `@x12i/memorix-retrieval` | Descriptor-driven data tier |
+| `@x12i/memorix-completion` | Source → Memorix enrichment (payload paths must match descriptors) |
+| `@x12i/memorix-explorer` | UI host — consumes retrieval APIs only |
+
+When adding a capability all hosts need (e.g. cross-entity workspace list), **extend retrieval and descriptors first**, then update hosts — never shim in a single app.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@x12i/memorix-retrieval",
-  "version": "1.2.0",
+  "version": "1.4.0",
   "description": "Descriptor-driven Memorix retrieval layer for lists, items, and content objects",
   "type": "module",
   "main": "./dist/index.js",
@@ -19,9 +19,10 @@
   "scripts": {
     "build": "tsc",
     "test": "vitest run",
-    "catalox:seed:build": "node scripts/build-seed-manifest.mjs",
+    "catalox:seed:build": "npm run build && node scripts/build-seed-manifest.mjs",
     "catalox:seed:memorix-retrieval:validate": "npm run catalox:seed:build && catalox seed validate --file catalox-seeds/memorix-retrieval-descriptors.manifest.json",
-    "catalox:seed:memorix-retrieval:apply": "npm run catalox:seed:build && node scripts/apply-seed-manifest.mjs",
+    "catalox:seed:memorix-retrieval:apply": "npm run build && node scripts/apply-seed-manifest.mjs",
+    "catalox:seed:ensure": "npm run build && node scripts/ensure-catalox-seed.mjs",
     "mongo:check-collections": "node scripts/check-mongo-collections.mjs",
     "smoke:retrieval": "npm run build && node scripts/smoke-retrieval.mjs"
   },