@x12i/memorix-retrieval 1.1.0

package/docs/MEMORIX-DATABASE-CONVENTIONS.md

@@ -0,0 +1,325 @@
+# Memorix Database Conventions
+
+This document defines the shared MongoDB naming and layout conventions for Memorix. All services, pipelines, and tools in the x12i ecosystem should follow these rules so entity data, event data, and source enrichment stay consistent across peers.
+
+## Overview
+
+Memorix uses **three logical database roles**:
+
+| Role | Purpose | Default database name |
+|------|---------|------------------------|
+| **Entities** | Canonical entity records (vulnerabilities, assets, groups, …) | `memorix-entities` |
+| **Events** | Event records derived from or linked to entities | `memorix-events` |
+| **Source** | Upstream operational / enrichment data used to extend Memorix records | *(environment-specific)* |
+
+All three may live on the **same MongoDB cluster** (single `MONGO_URI`). They are separated by **database name**, not by connection string.
+
+### Simple mode (default)
+
+Most peers only need:
+
+```bash
+MONGO_URI=mongodb://localhost:27017
+```
+
+Packages resolve `memorix-entities`, `memorix-events`, collection names, and entity vs event routing internally. Override with `MEMORIX_ENTITIES_DB`, `MEMORIX_EVENTS_DB`, or per-type collection env vars only when required.
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                     MongoDB cluster                          │
+│  ┌──────────────────┐  ┌──────────────────┐               │
+│  │ memorix-entities │  │  memorix-events  │  ← Memorix    │
+│  │  vulnerabilities │  │ vulnerability-   │    targets    │
+│  │  assets          │  │   events         │               │
+│  └────────▲─────────┘  └────────▲─────────┘               │
+│           │                       │                          │
+│           └───────────┬───────────┘                          │
+│                       │ join / enrich                        │
+│              ┌────────▼─────────┐                            │
+│              │  source-db       │  ← source databases       │
+│              │  *-information   │                            │
+│              └──────────────────┘                            │
+└─────────────────────────────────────────────────────────────┘
+```
+
+## Target types
+
+Every Memorix record belongs to exactly one **target**:
+
+| Target | Database | Typical use |
+|--------|----------|-------------|
+| `entity` | `memorix-entities` | Stable domain objects (vulnerability, asset, group, …) |
+| `event` | `memorix-events` | Occurrences, detections, or timeline entries |
+
+Tools must declare which target they read or write. Do not mix entity and event documents in the same collection.
+
+## Environment variables
+
+### Connection
+
+| Variable | Aliases | Required | Description |
+|----------|---------|----------|-------------|
+| `MONGO_URI` | `MONGO_CONNECTION_STRING` | Yes | MongoDB connection string for the cluster |
+
+### Memorix target databases
+
+Resolution order matters: **first non-empty value wins**.
+
+#### Entity database (`target: entity`)
+
+| Priority | Variable |
+|----------|----------|
+| 1 | `MEMORIX_ENTITIES_DB` |
+| 2 | `MEMORIX_DB_ENTITIES` |
+| 3 | `memorix_entities_db` |
+| 4 | `MEMORIX_DB` *(legacy fallback)* |
+| 5 | **Default:** `memorix-entities` |
+
+#### Event database (`target: event`)
+
+| Priority | Variable |
+|----------|----------|
+| 1 | `MEMORIX_EVENTS_DB` |
+| 2 | `MEMORIX_DB_EVENTS` |
+| 3 | `memorix_events_db` |
+| 4 | `MEMORIX_DB` *(legacy fallback)* |
+| 5 | **Default:** `memorix-events` |
+
+> **Peer rule:** Prefer `MEMORIX_ENTITIES_DB` and `MEMORIX_EVENTS_DB`. Treat `MEMORIX_DB` as a legacy single-database override only.
+
+### Source databases
+
+Source databases hold upstream data that Memorix tools use to **fill in or extend** entity/event `data` fields. They are never the write target for completion pipelines.
+
+| Priority | Variable | Description |
+|----------|----------|-------------|
+| 1 | `MEMORIX_SOURCE_DB` | Preferred default source database name |
+| 2 | `SOURCE_DATABASE_NAME` | Alternative default |
+| 3 | `SOURCEDATABASENAME` / `sourceDatabaseName` / `source_database_name` | Legacy aliases |
+
+Per-job overrides are allowed via mapping config or application code. When using JSON mappings with `@x12i/env-tokens`, reference env vars as `"databaseName": "{{ENV.sourceDatabaseName}}"`.
+
+### Collection overrides (per entity type)
+
+Normalize `entityType` by replacing `-` with `_` and uppercasing.
+
+| Target | Env key pattern | Example (`entityType: vulnerability`) |
+|--------|-----------------|----------------------------------------|
+| `entity` | `MEMORIX_ENTITIES_COLLECTION_<ENTITY_TYPE>` | `MEMORIX_ENTITIES_COLLECTION_VULNERABILITY` |
+| `event` | `MEMORIX_EVENTS_COLLECTION_<ENTITY_TYPE>` | `MEMORIX_EVENTS_COLLECTION_VULNERABILITY` |
+
+## Collection naming
+
+### Memorix entity collections (`memorix-entities`)
+
+Use **plural, kebab-case** collection names that describe the entity domain.
+
+| `entityType` | Recommended collection | Accepted fallbacks |
+|--------------|------------------------|--------------------|
+| `vulnerability` | `vulnerabilities` | `vulnerability` |
+| `variabilities-group` | `vulnerability-groups` | `vulnerabilities-groups`, `variabilities-groups` |
+| `assets` | `assets` | — |
+
+**Default heuristic** when nothing else is configured:
+
+```
+<entityType>            → e.g. vulnerability
+```
+
+Prefer explicit names in config over heuristics.
+
+### Memorix event collections (`memorix-events`)
+
+Use `<domain>-events` or `<domain>s-events`.
+
+| `entityType` | Recommended collection | Accepted fallbacks |
+|--------------|------------------------|--------------------|
+| `vulnerability` | `vulnerability-events` | `vulnerabilities-events` |
+| `assets` | `asset-events` | `assets-events` |
+
+**Default heuristic:**
+
+```
+<entityType>-events     → e.g. vulnerability-events
+```
+
+### Source collections
+
+Source collections typically use an **`-information`** suffix to distinguish them from Memorix canonical stores:
+
+| Source collection | Feeds entity type | Notes |
+|-------------------|-------------------|-------|
+| `vulnerabilities-information` | `vulnerability` | Raw / enriched vulnerability facts |
+| `vulnerability-groups-information` | `variabilities-group` | Group-level enrichment |
+| `assets-information` | `assets` | Asset metadata |
+
+**Peer rule:** Source collection names are referenced in `targetSelector.sourceCollection` on Memorix records and in completion mappings. Keep these strings stable — they are the join key for mapping selection.
+
+## Collection name resolution order
+
+When a tool needs to pick a Memorix collection, resolve in this order:
+
+1. Explicit override (CLI flag, API option, or code parameter)
+2. Mapping / config field `targetCollection`
+3. Environment variable (`MEMORIX_ENTITIES_COLLECTION_*` or `MEMORIX_EVENTS_COLLECTION_*`)
+4. `targetCollectionCandidates` from mapping config
+5. Built-in defaults for known `entityType` values (see tables above)
+6. Heuristic default (`<entityType>` or `<entityType>-events`)
+7. Verify the collection exists in the target database; fail clearly if not
+
+All peers should implement the same precedence so local, CI, and production behave identically given the same env.
+
+## Document shape
+
+### Memorix entity / event document
+
+Documents in `memorix-entities` and `memorix-events` share this layout:
+
+```json
+{
+  "_id": "<ObjectId>",
+  "recordId": "<optional stable id>",
+  "entityId": "<for entity records>",
+  "eventId": "<for event records>",
+  "capturedAt": "<ISO-8601 timestamp>",
+  "data": {
+    "... domain fields ..."
+  }
+}
+```
+
+| Field | Entity | Event | Notes |
+|-------|--------|-------|-------|
+| `_id` | Required | Required | MongoDB primary key |
+| `recordId` | Recommended | Recommended | Logical id; falls back to `_id` string |
+| `entityId` | Required* | Optional | Join key for entity completion |
+| `eventId` | — | Required* | Join key for event completion |
+| `data` | Required | Required | Payload; completion writes here |
+| `modifiedAt` | Optional | Optional | Set by writers on update (ISO-8601) |
+
+\*Required for completion pipelines that match on these paths.
+
+Legacy field `snapshotId` is still read as a fallback for `recordId` but **must not** be written by new code.
+
+### Provenance (logical, not always stored)
+
+Completion and ingestion tools track provenance in memory using:
+
+| Field | Description |
+|-------|-------------|
+| `sourceCollection` | Which source collection this record was enriched from |
+| `sourceDatabase` | Source database name |
+| `sourceId` | Id in the source system |
+| `idField` | Which id field was used when the record was created |
+
+These align with `targetSelector` in completion mappings.
+
+### Source document
+
+Source documents are ordinary MongoDB documents in the source database. There is no required Memorix envelope — tools match on configured paths (e.g. `vulnerabilityId`, `ip_address`).
+
+Do **not** write completion results back to source collections unless a separate, explicit pipeline owns that source.
+
+## Entity types
+
+`entityType` is a **kebab-case** string identifying the domain object:
+
+```
+vulnerability
+variabilities-group
+assets
+```
+
+Rules:
+
+- Use lowercase letters and hyphens only.
+- One `entityType` maps to one primary entity collection and optionally one event collection.
+- Env vars and config keys derive from `entityType` by replacing `-` with `_` and uppercasing.
+
+## Completion mapping conventions
+
+Tools that enrich Memorix from source data (e.g. `@x12i/memorix-completion`) use JSON mappings with these conventions:
+
+```json
+{
+  "name": "complete-vulnerability-entity-missing-data",
+  "entityType": "vulnerability",
+  "target": "entity",
+  "targetCollection": "vulnerabilities",
+  "targetSelector": {
+    "sourceCollection": "vulnerabilities-information",
+    "idField": "vulnerabilityId"
+  },
+  "source": {
+    "databaseName": "{{ENV.sourceDatabaseName}}",
+    "collection": "vulnerabilities-information"
+  },
+  "match": {
+    "targetPath": "entityId",
+    "sourcePath": "vulnerabilityId"
+  },
+  "writeRoot": "data",
+  "fields": [
+    { "sourcePath": "enrichment", "targetPath": "enrichment" }
+  ]
+}
+```
+
+| Field | Convention |
+|-------|------------|
+| `target` | `"entity"` or `"event"` — selects Memorix database |
+| `targetCollection` | Explicit Memorix collection (recommended) |
+| `targetSelector.sourceCollection` | Must match record's source lineage |
+| `source.databaseName` | Source DB; use `{{ENV.*}}` for env-driven deploys |
+| `source.collection` | Source collection to read from |
+| `match.targetPath` | Path on Memorix record (usually `entityId` or `eventId`) |
+| `match.sourcePath` | Path on source document |
+| `writeRoot` | Always `"data"` — never overwrite envelope fields via completion |
+
+Default completion mode: **fill missing fields only** (`onlyFillMissing: true`, `overwriteExisting: false`).
+
+## `.env` example
+
+```bash
+# Cluster
+MONGO_URI=mongodb://localhost:27017
+
+# Memorix targets (defaults shown — omit to use defaults)
+MEMORIX_ENTITIES_DB=memorix-entities
+MEMORIX_EVENTS_DB=memorix-events
+
+# Source database for enrichment pipelines
+MEMORIX_SOURCE_DB=poc-data-mapping
+# or: SOURCE_DATABASE_NAME=poc-data-mapping
+
+# Optional per-type collection overrides
+MEMORIX_ENTITIES_COLLECTION_VULNERABILITY=vulnerabilities
+MEMORIX_EVENTS_COLLECTION_VULNERABILITY=vulnerability-events
+```
+
+## Checklist for new peers
+
+When building a new service or tool that touches Memorix:
+
+- [ ] Connect with `MONGO_URI` / `MONGO_CONNECTION_STRING`
+- [ ] Select database via `target`: `entity` → `memorix-entities`, `event` → `memorix-events`
+- [ ] Respect env precedence documented above
+- [ ] Use recommended collection names or set explicit env overrides
+- [ ] Store domain payload under `data`
+- [ ] Read enrichment from source databases; write only to Memorix target databases
+- [ ] Use stable `sourceCollection` strings for mapping / provenance
+- [ ] Use kebab-case `entityType` values consistently
+- [ ] Fail with a clear error when a collection cannot be resolved
+
+## Reference implementation
+
+Shared env resolution is mirrored across peers:
+
+| Concern | Module |
+|---------|--------|
+| Database name from env | `@x12i/memorix-completion` / `@x12i/memorix-retrieval` → `src/mongo/env.ts` → `resolveMemorixDbName`, `resolveDefaultSourceDbName` |
+| Completion collection resolution | `@x12i/memorix-completion` → `src/mongo/resolve-collection.ts` → `resolveTargetCollectionName` |
+| Retrieval collection resolution | `@x12i/memorix-retrieval` → `src/data/collection-name.ts` → `resolveMemorixCollectionName` (descriptor + env override + prefix/postfix heuristic) |
+| Record document shape | `@x12i/memorix-completion` → `src/types.ts` → `MemorixRecord`, `CompletionMapping` |
+
+If this document and a package disagree, **update this document and the package together** — they are intended to stay in sync.

package/package.json

@@ -0,0 +1,52 @@
+{
+  "name": "@x12i/memorix-retrieval",
+  "version": "1.1.0",
+  "description": "Descriptor-driven Memorix retrieval layer for lists, items, and content objects",
+  "type": "module",
+  "main": "./dist/index.js",
+  "types": "./dist/index.d.ts",
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "import": "./dist/index.js"
+    }
+  },
+  "files": [
+    "dist",
+    "catalox-seeds",
+    "docs"
+  ],
+  "scripts": {
+    "build": "tsc",
+    "test": "vitest run",
+    "catalox:seed:build": "node scripts/build-seed-manifest.mjs",
+    "catalox:seed:memorix-retrieval:validate": "npm run catalox:seed:build && catalox seed validate --file catalox-seeds/memorix-retrieval-descriptors.manifest.json",
+    "catalox:seed:memorix-retrieval:apply": "npm run catalox:seed:build && node scripts/apply-seed-manifest.mjs",
+    "mongo:check-collections": "node scripts/check-mongo-collections.mjs",
+    "smoke:retrieval": "npm run build && node scripts/smoke-retrieval.mjs"
+  },
+  "engines": {
+    "node": ">=18.0.0"
+  },
+  "dependencies": {
+    "@x12i/catalox": "^4.1.2",
+    "@x12i/helpers": "^1.7.0",
+    "mongodb": "^6.21.0"
+  },
+  "peerDependencies": {
+    "@x12i/xronox": ">=3.7.0"
+  },
+  "peerDependenciesMeta": {
+    "@x12i/xronox": {
+      "optional": true
+    }
+  },
+  "devDependencies": {
+    "@types/node": "^20.14.0",
+    "typescript": "^5.6.0",
+    "vitest": "^2.1.0"
+  },
+  "publishConfig": {
+    "access": "public"
+  }
+}