@x12i/memorix-descriptors 1.4.2 → 1.5.0

package/docs/MEMORIX-DATABASE-CONVENTIONS.md

@@ -42,16 +42,19 @@ Packages resolve `memorix-entities`, `memorix-events`, collection names, and ent
 └─────────────────────────────────────────────────────────────┘
 ```
 
-## Target types
+## Storage targets (Mongo tier)
 
-Every Memorix record belongs to exactly one **target**:
+Every Memorix **record** belongs to exactly one **storage target**. This is independent of the **catalog object type** name (`assets`, `vulnerabilities`, …) declared in Catalox — see [MEMORIX-OBJECT-TYPES-AND-TARGETS.md](./MEMORIX-OBJECT-TYPES-AND-TARGETS.md).
 
-| Target | Database | Typical use |
-|--------|----------|-------------|
-| `entity` | `memorix-entities` | Stable domain objects (vulnerability, asset, group, …) |
-| `event` | `memorix-events` | Occurrences, detections, or timeline entries |
+| Target | Database | Typical catalog object types |
+|--------|----------|------------------------------|
+| `entity` | `memorix-entities` | `assets`, `variabilities-groups`, subnets, … |
+| `event` | `memorix-events` | `vulnerabilities` (findings), timeline entries |
+| `knowledge` | `memorix-knowledge` | Curated knowledge payloads (when wired) |
 
-Tools must declare which target they read or write. Do not mix entity and event documents in the same collection.
+Tools must declare which target they read or write. Do not mix entity- and event-target documents in the same collection.
+
+> **Naming trap:** Database `memorix-entities` means **target = entity**, not “every Memorix object type”. Object types with `target: "event"` live in `memorix-events` even if UI calls them “entities”.
 
 ## Environment variables
 

package/docs/MEMORIX-OBJECT-TYPES-AND-TARGETS.md

@@ -0,0 +1,207 @@
+# Memorix object types, storage targets, and naming
+
+**Canonical terminology** for the Memorix stack. Use this document when reading or writing code, Catalox catalog ids, Mongo env vars, and UI copy.
+
+If a name contains **`entity`**, check which row of [§2](#2-glossary) it belongs to — **five different meanings** are in play today.
+
+**Related:** [MEMORIX-CATALOX-CATALOG-MAP.md](./MEMORIX-CATALOX-CATALOG-MAP.md) (Firestore catalogs), [MEMORIX-CATALOX-CONTRACTS.md](./MEMORIX-CATALOX-CONTRACTS.md) (descriptor JSON), [MEMORIX-DATABASE-CONVENTIONS.md](./MEMORIX-DATABASE-CONVENTIONS.md) (Mongo layout).
+
+---
+
+## 1. Mental model (two layers)
+
+```text
+┌─────────────────────────────────────────────────────────────────────────┐
+│ CATALOX (metadata) — appId: memorix                                      │
+│                                                                          │
+│  Catalog object type  =  one registered domain in memorix-entity-      │
+│                          descriptors (field: entityName today)         │
+│                          e.g. assets, vulnerabilities, variabilities-  │
+│                          groups                                          │
+│                                                                          │
+│  Each object type has:                                                   │
+│    • storage target     →  entity | event | knowledge                    │
+│    • list / item views  →  memorix-list- / memorix-item-descriptors      │
+│    • content types      →  snapshots, scoped, … → Mongo collections      │
+└───────────────────────────────┬─────────────────────────────────────────┘
+                                │ descriptor-driven reads/writes
+                                ▼
+┌─────────────────────────────────────────────────────────────────────────┐
+│ MONGODB (payload) — separate databases per storage target                │
+│                                                                          │
+│  target "entity"     →  DB memorix-entities   (env: MEMORIX_ENTITIES_DB) │
+│  target "event"      →  DB memorix-events     (env: MEMORIX_EVENTS_DB)   │
+│  target "knowledge"  →  DB memorix-knowledge  (env: MEMORIX_KNOWLEDGE_DB)│
+│                                                                          │
+│  Collection naming:  {objectName}-{contentType}  e.g. assets-snapshots   │
+│  Record identity:    entityId | eventId | knowledgeId on envelope        │
+└─────────────────────────────────────────────────────────────────────────┘
+```
+
+**Rule:** A **catalog object type** is declared in Catalox. A **Mongo record** is stored under the **target database** chosen by that type’s `target` field. The word “entity” must not be used loosely for both.
+
+---
+
+## 2. Glossary
+
+| Term (use in new docs) | Code / config name today | Meaning | Example |
+|------------------------|--------------------------|---------|---------|
+| **Storage target** | `MemorixTarget`, `target` | Which Memorix **Mongo database role** holds payload for this object type | `vulnerabilities` → `target: "event"` → `memorix-events` |
+| **Catalog object type** | `entityName` on descriptor; `objectName` in inventory | Kebab-case **domain id** registered in `memorix-entity-descriptors` | `assets`, `variabilities-groups` |
+| **Object type descriptor** | `MemorixEntityDescriptor` (type name is legacy) | Catalox item in `memorix-entity-descriptors` describing one catalog object type | item id `assets` |
+| **Content type** | key under `contentTypes` | Named slice of data for an object type; drives collection suffix | `snapshots` → `assets-snapshots` |
+| **List view** | list descriptor `entity` field | Tabular descriptor bound to one object type | `assets-main-list` → object type `assets` |
+| **Item view** | item descriptor `entity` field | Detail descriptor bound to one object type | `asset-detail-item` |
+| **Record** | Mongo document | Payload row with `data` + identity field(s) | one row in `vulnerabilities-snapshots` |
+| **Knowledge artifact** | `knowledge` catalog item | Separate track (`knowledgeId`), not an object type descriptor | `skill-asset-context-analysis` |
+
+### 2.1 Storage targets (the three tiers)
+
+| Target | Default Mongo DB | Primary identity field | What belongs here |
+|--------|------------------|------------------------|-------------------|
+| `entity` | `memorix-entities` | `entityId` | Stable inventory-style objects (assets, groups, subnets, …) |
+| `event` | `memorix-events` | `eventId` (descriptor may still allow `entityId` in identity block) | Findings, detections, timeline-oriented vulns |
+| `knowledge` | `memorix-knowledge` | `knowledgeId` | Curated facts, skills, playbooks (ecosystem rollout coordinated across packages) |
+
+**Why three targets:** Same **object type name** could theoretically exist on different tiers; in practice each catalog object type picks **one** `target` so completion, inventory, and Xronox resolve the correct database.
+
+**Shipped examples:**
+
+| Catalog object type (`entityName`) | Storage target | Mongo DB |
+|-----------------------------------|----------------|----------|
+| `assets` | `entity` | `memorix-entities` |
+| `variabilities-groups` | `entity` | `memorix-entities` |
+| `vulnerabilities` | `event` | `memorix-events` |
+
+---
+
+## 3. What is overloaded today (“entity” mess)
+
+The same word **entity** refers to different concepts. This table is the root cause of confusion.
+
+| # | Name | Layer | What it actually is | Common mistake |
+|---|------|-------|---------------------|----------------|
+| A | **`target: "entity"`** | Descriptor + Mongo | Storage tier / DB role | Calling every catalog object type an “entity” |
+| B | **`entityName`** | Descriptor JSON | **Catalog object type** id | Thinking it only applies when `target === "entity"` |
+| C | **`memorix-entity-descriptors`** | Catalox catalog id | Registry of **all** object type descriptors (entity + event + knowledge targets) | Assuming the catalog only describes `target: "entity"` types |
+| D | **`memorix-entities`** (Mongo DB) | Mongo | Database for **target entity** payload | Confusing with catalog id (E) or object types (B) |
+| E | **`memorix-entities`** (Catalox catalog) | Firestore legacy | Inferred schema samples (`dataDescriptors`), **not** object type descriptors | Treating as source of truth for retrieval |
+| F | **`entityId`** | Mongo envelope | Record identity field (used across targets in places) | Assuming “entityId ⇒ target entity” (vuln items use `entityId` while `target` is `event`) |
+| G | **`cataloxObjects`** | Inventory API | Count of distinct **catalog object types** (`objectName` / `entityName`) | Sounds like Catalox “entities” catalog (E) |
+
+**Inventory / reconcile** partially uses clearer language: **`objectName`** = parsed from `assets-snapshots` → object type `assets`, aligned with descriptor `entityName`. Prefer **object type** in prose; treat `entityName` as a legacy field name.
+
+---
+
+## 4. Correct sentences (copy-paste patterns)
+
+| Instead of… | Write… |
+|-------------|--------|
+| “The vulnerabilities **entity**” (ambiguous) | “The **catalog object type** `vulnerabilities`” or “vulnerabilities **object type**” |
+| “Stored in the entities database” | “Stored with **target** `event` in **`memorix-events`**” |
+| “Entity descriptor catalog” | “**Object type descriptor** catalog (`memorix-entity-descriptors`)” |
+| “Catalox entities catalog” | “Legacy **`memorix-entities`** inference catalog (deprecated)” |
+| “Entity collection `assets-snapshots`” | “Mongo collection for object type **assets**, content type **snapshots**” |
+| “9 entities in Catalox” (legacy catalog) | “9 **inferred object type rows** in legacy catalog `memorix-entities`” |
+| “Discovery lists entities” | “Discovery lists **catalog object types** from `memorix-entity-descriptors`” |
+
+---
+
+## 5. Catalog object types vs Catalox catalogs
+
+Do not confuse **catalog object types** (domain ids) with **Catalox catalog ids** (Firestore buckets).
+
+| Catalox `catalogId` | Holds | Relates to object types? |
+|-------------------|-------|---------------------------|
+| `memorix-entity-descriptors` | One item per **catalog object type** | **Yes** — source of truth |
+| `memorix-list-descriptors` | List/workspace/slice views | References object type via `entity` field |
+| `memorix-item-descriptors` | Detail views | References object type via `entity` field |
+| `memorix-completion-mappings` | Enrichment jobs | References `entityType` (= object type name) |
+| `memorix-inventory-policies` | Ignored Mongo collections | Per **target** + collection |
+| `memorix-entities` | **Legacy** inference only | Parallel naming to Mongo DB — **misleading** |
+| `memorix_entity_content_types` | **Legacy** global content-type registry | Superseded by per–object-type `contentTypes` |
+| `knowledge` | Knowledge **artifacts** | Not the same as object types (separate product surface) |
+
+There is **no** separate Catalox catalog per storage target. Target is a **field on each object type descriptor**.
+
+---
+
+## 6. Collection naming ties object type + content type
+
+```text
+{objectName}-{contentType}
+     │              │
+     │              └── content type key (snapshots, scoped, …)
+     └── catalog object type (same as entityName / inventory objectName)
+```
+
+Parser: `parseMemorixCollectionName` — last hyphen splits object type from content type.
+
+Inventory row: `(target, databaseName, objectName, contentType, collectionName)`.
+
+---
+
+## 7. Rename roadmap (recommended — not yet applied in code)
+
+Goal: **one public noun** for domain types (**catalog object type** / **object type**), **target** only for Mongo tier, and **no catalog id** that sounds like Mongo `memorix-entities`.
+
+| Current | Proposed | Scope | Breaking? | Why |
+|---------|----------|-------|-----------|-----|
+| `entityName` (descriptor) | `objectType` or `objectName` | JSON + TS | **Yes** — major | Name matches inventory and collection parser |
+| `MemorixEntityDescriptor` | `MemorixObjectTypeDescriptor` | TS export | **Yes** | Type describes any target, not only `entity` |
+| `memorix-entity-descriptors` | `memorix-object-type-descriptors` | Catalox catalog id + seeds | **Yes** — migration | Removes false “entity-only” implication |
+| `entity` (list/item descriptor field) | `objectType` | JSON | **Yes** | Align with object-type vocabulary |
+| `entityType` (completion mapping) | `objectType` | JSON | **Yes** | Same |
+| `cataloxObjects` (inventory summary) | `declaredObjectTypes` | API | **Yes** | Accurate count label |
+| `registerEntityDescriptor` | `registerObjectTypeDescriptor` | API | **Yes** | Admin method names |
+| Catalox `memorix-entities` | Deprecate → `memorix-mongo-schema-inference` (or delete) | Firestore | Medium | Stops collision with Mongo DB name |
+| Mongo DB `memorix-entities` | Keep (or `memorix-object-store-entity` long-term) | Mongo | **Very high** | Industry-wide env vars; document only unless cluster migration |
+| `entityId` on records | Keep; document per-target | Mongo | High | Widespread; vuln stack already uses `entityId` on event target |
+
+**Phased approach:**
+
+1. **Docs + UI copy** — adopt §4 patterns immediately (this repo: done in linked docs).
+2. **Additive aliases** — accept `objectType` alongside `entityName` in validators (non-breaking).
+3. **Catalox catalog migration** — dual-write / seed both catalog ids, switch retrieval, retire old id.
+4. **Remove legacy** `memorix-entities` and `memorix_entity_content_types` Firestore catalogs when no consumer lists them.
+
+---
+
+## 8. API and code map (today)
+
+| Concept | Where in `@x12i/memorix-descriptors` |
+|---------|--------------------------------------|
+| Storage target | `MemorixTarget` in `src/types/index.ts` |
+| Object type id | `entityName` on descriptors; `objectName` in `src/reconcile/inventory-types.ts` |
+| Target → Mongo DB | `resolveMemorixDbName` in `src/collections/mongo-env.ts` |
+| Collection parse | `parseMemorixCollectionName` in `src/collections/parse.ts` |
+| Declared slots | `buildDeclaredSlots` in `src/reconcile/unified-inventory.ts` (`objectName: entity.entityName`) |
+| Catalog ids | `src/catalog/ids.ts` |
+
+Retrieval discovery: lists **`memorix-entity-descriptors`** → returns summaries with `entityName` (treat as **object type** in UI).
+
+---
+
+## 9. FAQ
+
+**Q: Is `vulnerabilities` an entity or an event?**  
+A: It is a **catalog object type** named `vulnerabilities` with **storage target** `event`. Colloquially “vulnerability entity” is wrong in strict terms; say “vulnerability object type” or “vulnerabilities (event target)”.
+
+**Q: Why does the descriptor catalog id say `entity`?**  
+A: Historical naming from when only `memorix-entities` Mongo tier existed. The catalog now holds object types for **all** targets. Rename planned (§7).
+
+**Q: What should Explorer show in a type picker?**  
+A: **Object types** from discovery (`entityName`), with badge for **target** (`entity` / `event` / `knowledge`) and DB name — not “Entities” as a single bucket.
+
+**Q: Does `knowledge` catalog replace object types?**  
+A: No. `knowledge` catalog holds **artifacts** (`knowledgeId`). Object types with `target: "knowledge"` would still be declared in `memorix-entity-descriptors` when that tier is fully wired.
+
+---
+
+## 10. Checklist for authors
+
+- [ ] Used **storage target** for `entity` | `event` | `knowledge` and Mongo DB choice  
+- [ ] Used **catalog object type** / **object type** for `assets`, `vulnerabilities`, etc.  
+- [ ] Did not refer to legacy Catalox `memorix-entities` as descriptor source of truth  
+- [ ] Distinguished Mongo **`memorix-entities`** (database) from catalog **`memorix-entity-descriptors`**  
+- [ ] Did not imply `memorix-entity-descriptors` only applies to `target: "entity"`  

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@x12i/memorix-descriptors",
-  "version": "1.4.2",
+  "version": "1.5.0",
   "description": "Manage Memorix entity, list, and item descriptors in Catalox",
   "type": "module",
   "main": "./dist/index.js",
@@ -45,6 +45,13 @@
     "test:live": "vitest run src/tests/live",
     "test:all": "vitest run",
     "validate:seeds": "node scripts/validate-descriptor-seeds.mjs",
+    "memorix:metadata:inspect": "tsx scripts/memorix-metadata/inspect.ts",
+    "memorix:metadata:plan": "tsx scripts/memorix-metadata/plan.ts",
+    "memorix:metadata:validate-plan": "tsx scripts/memorix-metadata/validate-plan.ts",
+    "memorix:metadata:apply": "tsx scripts/memorix-metadata/apply.ts",
+    "memorix:metadata:cleanup-legacy": "tsx scripts/memorix-metadata/cleanup-legacy.ts",
+    "memorix:metadata:verify": "tsx scripts/memorix-metadata/verify.ts",
+    "memorix:metadata:reset": "tsx scripts/memorix-metadata/reset.ts",
     "prepublishOnly": "npm run build"
   },
   "engines": {
@@ -60,6 +67,7 @@
     "@x12i/catalox": "^5.0.0",
     "@x12i/xronox": "^3.9.0",
     "dotenv": "^16.4.5",
+    "tsx": "^4.19.0",
     "typescript": "^5.6.0",
     "vitest": "^2.1.0"
   },