@x12i/memorix-format 2.2.2 → 2.2.3

package/docs/memorix-format-additions.md

@@ -1,956 +0,0 @@
-# Appendix: Extended Memorix Record Parts
-
-> Shipped with `@x12i/memorix-format`. Examples may show the older `identity` envelope for illustration; the **canonical** clean envelope is [`memorix-format.md`](./memorix-format.md).
-
-## A. Why this appendix exists
-
-The main Memorix 2.0 format defines the clean structure of a record:
-
-```txt id="v3yvcs"
-identity
-lifecycle
-concept
-data
-index
-sourceFacts
-_system
-```
-
-This appendix adds the missing higher-level distinction between four different kinds of information that are often mixed together:
-
-```txt id="dw987j"
-native facts
-associated context
-numeric analytics
-interpretive insights
-```
-
-The distinction matters because Memorix records need to be easy to consume, but also semantically clean.
-
-The existing Memorix contract already says that domain payload belongs in `data`, while system and provenance fields should stay outside `data`. The same idea should now be extended to associated context, analytics, and insights.
-
----
-
-## B. Full snapshot record shape
-
-The full-scale Memorix record shape is intended mainly for the `snapshot` content type of an object type.
-
-```ts id="km4sqn"
-type MemorixSnapshotRecord = {
-  _id?: string;
-
-  identity: MemorixIdentity;
-  lifecycle: MemorixLifecycle;
-  concept: MemorixConcept;
-
-  data: JsonObject;
-
-  associatedData?: JsonObject;
-  analytics?: MemorixAnalytics;
-  insights?: MemorixInsights;
-
-  index?: MemorixIndexBlock;
-  sourceFacts?: MemorixSourceFacts;
-
-  _system?: MemorixSystemBlock;
-};
-```
-
-This is the rich, readable, consumption-friendly version of the object.
-
-Other content types under the same object type do not need to carry all of these blocks. Memorix already stores records by object type and content type, for example `items-snapshots` or `findings-snapshots`.
-
-So the rule is:
-
-```txt id="s9zbmc"
-The full record shape belongs primarily to snapshots.
-Other content types should contain only what makes sense for that content type.
-```
-
----
-
-## C. Minimal shape for non-snapshot content types
-
-For the product **workflow** content type (`{objectType}-workflow`, format
-`memorix-workflow/1`), see [memorix-workflow-format.md](./memorix-workflow-format.md).
-Workflow is 1:1 product state (owner, status, audit, tasks); catalog-driven
-fields under `data` are defined in Catalox `memorix-workflow-field-schemas`.
-
-Not every collection under an object type should look like a full snapshot.
-
-For example:
-
-```txt id="evq5gl"
-items-snapshots
-items-events
-items-facts
-items-chunks
-items-history
-items-profiles
-```
-
-Only `items-snapshots` needs the full record surface.
-
-Other content types should have the required Memorix structure, but not unnecessary blocks.
-
-Recommended minimal shape:
-
-```ts id="uak8ve"
-type MemorixContentRecord = {
-  _id?: string;
-
-  identity: MemorixIdentity;
-  lifecycle?: MemorixLifecycle;
-  concept?: MemorixConcept;
-
-  data: JsonObject;
-
-  index?: MemorixIndexBlock;
-  sourceFacts?: MemorixSourceFacts;
-
-  _system?: MemorixSystemBlock;
-};
-```
-
-Required minimum:
-
-```txt id="rq22ev"
-identity
-data
-whatever metadata/keys Memorix needs for that content type
-```
-
-Memorix should decide the exact required keys by content type, descriptor, and object behavior.
-
-The important point:
-
-```txt id="6pdkqx"
-Do not force analytics, insights, or associatedData into every content type.
-```
-
----
-
-## D. `data`: native readable facts
-
-`data` is the record’s own readable truth.
-
-It should contain facts that are directly about the record itself.
-
-For a CVE record:
-
-```json id="lqeccc"
-{
-  "data": {
-    "docId": "DOC-9",
-    "severity": "critical",
-    "description": "Unrestricted file upload vulnerability in WSO2 products.",
-    "publishedAt": "2022-04-18"
-  }
-}
-```
-
-These are facts about the CVE itself.
-
-`data` should not contain unrelated records, source-system noise, enum codes, opaque ids, or product judgments.
-
----
-
-## E. `associatedData`: readable related context
-
-`associatedData` contains useful readable context attached to the record because of related records.
-
-It is not the record’s native truth, but it is very useful for consumption.
-
-For example, a CVE record may need to show the items affected by that CVE.
-
-Those items are not part of the CVE itself, so they should not live in `data`.
-
-But consumers still need them.
-
-So they belong in `associatedData`.
-
-```json id="m7pzc6"
-{
-  "associatedData": {
-    "affectedAssets": [
-      {
-        "assetName": "prod-api-01",
-        "sku": "10.100.18.64",
-        "recordId": "entity:asset:10.100.18.64"
-      },
-      {
-        "assetName": "prod-db-02",
-        "sku": "10.100.18.65",
-        "recordId": "entity:asset:10.100.18.65"
-      }
-    ]
-  }
-}
-```
-
-This gives consumers the useful joined view without pretending that affected items are native CVE fields.
-
-### Rule
-
-```txt id="cqzqcl"
-associatedData = readable materialized context from related records
-```
-
----
-
-## F. `analytics`: numeric rollups about linked lists
-
-`analytics` is not the associated data itself.
-
-`analytics` contains numbers about associated lists.
-
-It is used when the record has links to lists of other records, and Memorix wants to store useful numeric summaries about those lists.
-
-For example:
-
-```txt id="omk77f"
-CVE -> affected items
-asset -> related findings
-document -> exposed users
-user -> accessible sensitive files
-group -> member users
-policy -> affected resources
-```
-
-If a CVE is linked to many affected items, then `analytics.affectedAssets` contains numbers about that list.
-
-```json id="uerbc6"
-{
-  "analytics": {
-    "affectedAssets": {
-      "total": 12,
-      "internetExposed": 3,
-      "production": 5,
-      "withOpenFindings": 9
-    }
-  }
-}
-```
-
-The actual affected asset list belongs in `associatedData`.
-
-The internal record links belong in `index.recordLinks`.
-
-The numeric summaries belong in `analytics`.
-
-### What belongs in `analytics`
-
-```txt id="fdwu3l"
-counts
-totals
-distributions
-buckets
-aggregations
-list summaries
-numeric rollups
-```
-
-Examples:
-
-```json id="1dwp79"
-{
-  "analytics": {
-    "affectedAssets": {
-      "total": 12,
-      "byEnvironment": {
-        "production": 5,
-        "staging": 4,
-        "development": 3
-      },
-      "internetExposed": 3
-    },
-    "relatedFindings": {
-      "total": 18,
-      "open": 14,
-      "closed": 4
-    }
-  }
-}
-```
-
-### What does not belong in `analytics`
-
-Do not put the actual related records in `analytics`.
-
-Bad:
-
-```json id="8e7c4g"
-{
-  "analytics": {
-    "affectedAssets": [
-      {
-        "assetName": "prod-api-01",
-        "sku": "10.100.18.64"
-      }
-    ]
-  }
-}
-```
-
-Good:
-
-```json id="3t9cdo"
-{
-  "associatedData": {
-    "affectedAssets": [
-      {
-        "assetName": "prod-api-01",
-        "sku": "10.100.18.64"
-      }
-    ]
-  },
-  "analytics": {
-    "affectedAssets": {
-      "total": 1
-    }
-  }
-}
-```
-
-### Rule
-
-```txt id="7wh3kh"
-analytics answers: how many, how much, how distributed, how often.
-```
-
----
-
-## G. `insights`: interpretations, judgments, and decisions
-
-`insights` contains interpreted meaning about the record.
-
-It is the place for judgments, assessments, analysis, scores, conclusions, recommendations, classifications, confidence, and priority.
-
-```json id="2dxvna"
-{
-  "insights": {
-    "risk": {
-      "level": "critical",
-      "score": 92,
-      "confidence": 0.84,
-      "reason": "Critical CVE affecting production and internet-exposed items."
-    },
-    "priority": {
-      "level": "urgent",
-      "reason": "Affected items include production systems."
-    },
-    "recommendations": [
-      {
-        "action": "Patch affected items",
-        "urgency": "high"
-      }
-    ]
-  }
-}
-```
-
-### Why this must be separate from `data`
-
-A score is not the same as a fact.
-
-This is a fact:
-
-```json id="vfqhn8"
-{
-  "severity": "critical"
-}
-```
-
-This is an insight:
-
-```json id="tdrpzw"
-{
-  "riskScore": 92,
-  "priority": "urgent",
-  "recommendedAction": "Patch affected items"
-}
-```
-
-If insights are placed inside `data`, they can pollute the record’s native meaning.
-
-### Rule
-
-```txt id="41g5nl"
-insights answers: so what, how bad, what should we do, how confident are we.
-```
-
----
-
-## H. Difference between `associatedData`, `analytics`, and `insights`
-
-These three are easy to confuse, so the distinction must be strict.
-
-```txt id="d2vm9q"
-associatedData = the related things
-analytics = numbers about the related things
-insights = conclusions from the facts, context, and numbers
-```
-
-Example:
-
-```json id="3n6qlf"
-{
-  "data": {
-    "docId": "DOC-9",
-    "severity": "critical"
-  },
-
-  "associatedData": {
-    "affectedAssets": [
-      {
-        "assetName": "prod-api-01",
-        "sku": "10.100.18.64"
-      },
-      {
-        "assetName": "prod-db-02",
-        "sku": "10.100.18.65"
-      }
-    ]
-  },
-
-  "analytics": {
-    "affectedAssets": {
-      "total": 2,
-      "production": 2,
-      "internetExposed": 1
-    }
-  },
-
-  "insights": {
-    "risk": {
-      "level": "critical",
-      "reason": "The CVE affects production items, including one internet-exposed asset."
-    }
-  }
-}
-```
-
-This keeps the model clean:
-
-```txt id="sjw9xv"
-data says what the record is.
-associatedData says what is connected to it.
-analytics says how much or how many.
-insights says what Memorix concludes.
-```
-
----
-
-## I. Relationship with `index.recordLinks`
-
-`index.recordLinks` is the internal link/index surface.
-
-It is not meant to replace `associatedData`.
-
-It is used by Memorix for search, expansion, linking, retrieval, ranking, and internal navigation.
-
-The existing Memorix model already treats relations as read-time joins rather than stored graph edges.
-
-The Memorix 2.0 model keeps that philosophy but gives links an indexed representation when needed.
-
-Example:
-
-```json id="6akxmw"
-{
-  "index": {
-    "recordLinks": [
-      {
-        "key": "affectedAssets",
-        "relation": "affects",
-        "direction": "out",
-        "target": {
-          "recordId": "entity:asset:10.100.18.64",
-          "target": "entity",
-          "objectType": "items"
-        },
-        "display": {
-          "name": "prod-api-01"
-        },
-        "indexed": true
-      }
-    ]
-  }
-}
-```
-
-The same relationship may appear in different ways:
-
-```txt id="f0c6wq"
-index.recordLinks = internal link
-associatedData = readable related record view
-analytics = numeric summary of the linked list
-insights = interpretation based on the relationship
-```
-
----
-
-## J. Relationship with `sourceFacts`
-
-`sourceFacts` preserves raw values used for resolution and traceability.
-
-Example:
-
-```json id="x0e556"
-{
-  "sourceFacts": {
-    "schemaVersion": "sourceFacts.v1",
-    "values": {
-      "assetId": {
-        "rawValue": "a_918271",
-        "kind": "foreign-key",
-        "resolvedToPath": "associatedData.affectedAssets[0].assetName",
-        "resolvedValue": "prod-api-01"
-      }
-    }
-  }
-}
-```
-
-The raw value belongs in `sourceFacts`.
-
-The readable related object belongs in `associatedData`.
-
-The internal link belongs in `index.recordLinks`.
-
-The count belongs in `analytics`.
-
-The conclusion belongs in `insights`.
-
----
-
-## K. Relationship with Memorix-Hippox
-
-Memorix-Hippox is the metadata database of Memorix.
-
-It remembers how things relate, how values are resolved, and how source-system mess becomes Memorix meaning.
-
-For this appendix, Hippox should remember product-level metadata such as:
-
-```txt id="y7qppy"
-which object types can be linked
-which links are one-to-one
-which links are one-to-many
-which linked lists should produce analytics
-which linked lists should produce associatedData
-which enum/code values resolve to readable values
-which fields should become insights
-which fields are native data
-which fields are raw source facts
-which sourcePath/targetPath pairs join two object types
-```
-
-The record does not need to store the full Hippox metadata format.
-
-The record stores the result:
-
-```txt id="s0v4wk"
-sourceFacts keeps raw resolver material.
-index.recordLinks keeps internal links.
-associatedData keeps readable related context.
-analytics keeps numeric rollups.
-insights keeps interpreted conclusions.
-```
-
-### K.1 Three Hippox surfaces on a snapshot record
-
-| Layer | Who writes it | Hippox involvement |
-|-------|---------------|-------------------|
-| `index.recordLinks` | ingest / indexer | schema relations; optional Hippox-approved join metadata |
-| `associatedData` | snapshot materializer | readable output from approved context links (`retrieval.type: record` or `list`) |
-| `analytics` / `insights` | downstream jobs | Hippox may define which linked lists feed rollups or conclusions |
-
-Context link rules themselves live in Mongo database **`memorix-hippox`** (collection **`context-links`**). Records only carry materialized results.
-
-### K.2 Context links vs schema relations
-
-Schema relations (entity descriptors) declare durable product semantics and populate `index.recordLinks`.
-
-**Context links** (`@x12i/memorix-hippox`) let operators define property joins without changing schema first:
-
-```txt
-discover  → join-discovery ranks fields from sampled data
-simulate  → validates proposed join on samples
-approve   → persists reusable link definition
-resolve   → bounded runtime context (jobMemory.context)
-```
-
-Join paths use the same dot-path convention as descriptors: from document root, typically under `data.*`.
-
-Example operator flow for `items → locations`:
-
-```http
-GET /api/hippox/join-discovery?sourceObjectType=items&targetObjectType=locations
-POST /api/hippox/links/simulate
-POST /api/hippox/links/:id/approve
-POST /api/hippox/context/resolve
-```
-
-Discovery seeds from schema relations and previously approved Hippox links so declared joins stay rank 1 even on rediscovery.
-
-When a snapshot is fully materialized, readable joined targets belong in **`associatedData`**, not mixed into native `data` fields. Internal pointers stay in **`index.recordLinks`**.
-
-### K.3 Snapshot-only materialization rule
-
-`associatedData`, `analytics`, and `insights` are snapshot-only by default (see §L). Hippox context resolution at job time produces transient `jobMemory.context`; persisting that output onto a snapshot is an explicit materialization step, not automatic on every resolve call.
-
-### K.4 Automated fuzzy mapping (`@x12i/helpers` ≥ 2.0)
-
-For scopes **without** entity descriptors or schema relations — migration dumps, external API clusters, audit JSON — Hippox can use the **`@x12i/helpers/record-key-analysis`** pipeline to infer joins before an operator touches the UI.
-
-**Prerequisite:** `@x12i/helpers@^2.0.0` with `analyzeSingleRecordKeys`, `analyzeCollectionKeys`, and `mapCrossCollectionRelationships`.
-
-**Phase 1 — Ingestion and sampling**
-
-Loop all collections in scope. Sample up to **10,000** records per collection (same order of magnitude as Hippox discovery/simulation limits). Hippox reuses its Mongo collection resolver for Memorix object types; raw JSON arrays use the same sample cap in memory.
-
-**Phase 2 — Entity profiling (per collection)**
-
-1. `analyzeSingleRecordKeys(firstRecord)` — candidate property names with confidence scores.
-2. Keep properties where **`confidence >= 40`** (reduces CPU vs profiling every column).
-3. `analyzeCollectionKeys(fullSample, candidateProperties)` — classify each property:
-   - **`PRIMARY`** → entity primary identifiers (candidate join targets)
-   - **`FOREIGN`** → unresolved outbound keys (candidate Hippox `sourcePath`)
-   - **`ENUM`** → data dictionary entries (future `sourceFacts` resolver seeds)
-
-Store results in a central schema registry (in-memory map or JSON artifact).
-
-**Phase 3 — Cross-collection relationship mapping**
-
-After all collections are profiled:
-
-1. Build `targetCollectionsMap`: each collection name → `{ primaryKeys, records }` using Phase 2 `PRIMARY` paths.
-2. For every collection with **`FOREIGN`** keys, call `mapCrossCollectionRelationships(sourceSample, foreignKeyProperty, targetCollectionsMap)`.
-3. Accept edges where **`inclusionRatio > 0.95`** (helpers default threshold).
-4. Map winning edges to Hippox join candidates: `{ sourceObjectType, targetObjectType, join: [{ sourcePath, targetPath, operator: 'eq' }] }`.
-
-**Output handling**
-
-The orchestration produces a structured link graph of the previously unknown scope. Feed it into:
-
-- Hippox **simulate → approve** for durable context links
-- Schema **relation detect** for descriptor promotion
-- Migration validators and agentic schema tools
-
-This is **complementary** to `GET /api/hippox/join-discovery`, which assumes named Memorix object types and blends live overlap with schema/approved-link seeds. Fuzzy mapping is the cold-start path; join-discovery is the operator refinement path once types exist.
-
----
-
-## L. Snapshot-only full-scale rule
-
-The full-scale shape should be expected mainly in snapshot collections.
-
-Example:
-
-```txt id="ot7c3i"
-findings-snapshots
-items-snapshots
-users-snapshots
-documents-snapshots
-```
-
-A snapshot is the best place for:
-
-```txt id="kr0jcs"
-concept
-data
-associatedData
-analytics
-insights
-index
-sourceFacts
-_system
-```
-
-Other content types should stay smaller and purpose-specific.
-
-Example:
-
-```txt id="i0znxj"
-findings-events
-findings-facts
-findings-chunks
-findings-history
-```
-
-These may only need:
-
-```txt id="nqsr18"
-identity
-lifecycle
-data
-sourceFacts
-_system
-```
-
-or whatever Memorix decides is required by descriptor and use case.
-
-### Rule
-
-```txt id="l97dza"
-Snapshots are the full readable object view.
-Other content types are specialized records.
-```
-
----
-
-## M. Final full-scale snapshot example
-
-```json id="eey88y"
-{
-  "_id": "667f00000000000000000001",
-
-  "identity": {
-    "recordId": "entity:cve:DOC-9",
-    "target": "entity",
-    "targetId": "entity:cve:DOC-9",
-    "objectType": "findings",
-    "contentType": "snapshots",
-    "version": "memorix-record.2.0"
-  },
-
-  "lifecycle": {
-    "createdAt": "2026-06-30T10:00:00.000Z",
-    "modifiedAt": "2026-06-30T10:00:00.000Z",
-    "capturedAt": "2026-06-30T09:55:00.000Z",
-    "status": "active"
-  },
-
-  "concept": {
-    "kind": "cve",
-    "name": "DOC-9",
-    "title": "DOC-9",
-    "summary": "Critical unrestricted file upload vulnerability affecting WSO2 products.",
-    "keywords": ["DOC-9", "WSO2", "critical"]
-  },
-
-  "data": {
-    "docId": "DOC-9",
-    "severity": "critical",
-    "description": "Unrestricted file upload vulnerability affecting WSO2 products.",
-    "publishedAt": "2022-04-18"
-  },
-
-  "associatedData": {
-    "affectedAssets": [
-      {
-        "assetName": "prod-api-01",
-        "sku": "10.100.18.64",
-        "recordId": "entity:asset:10.100.18.64"
-      },
-      {
-        "assetName": "prod-db-02",
-        "sku": "10.100.18.65",
-        "recordId": "entity:asset:10.100.18.65"
-      }
-    ]
-  },
-
-  "analytics": {
-    "affectedAssets": {
-      "total": 2,
-      "production": 2,
-      "internetExposed": 1
-    },
-    "relatedFindings": {
-      "total": 4,
-      "open": 3,
-      "closed": 1
-    }
-  },
-
-  "insights": {
-    "risk": {
-      "level": "critical",
-      "score": 92,
-      "confidence": 0.84,
-      "reason": "The CVE affects production items, including one internet-exposed asset."
-    },
-    "priority": {
-      "level": "urgent",
-      "reason": "Production exposure and critical severity require fast remediation."
-    },
-    "recommendations": [
-      {
-        "action": "Patch affected items",
-        "urgency": "high"
-      }
-    ]
-  },
-
-  "index": {
-    "text": {
-      "title": "DOC-9",
-      "summary": "Critical unrestricted file upload vulnerability affecting WSO2 products.",
-      "keywords": ["DOC-9", "critical", "prod-api-01", "prod-db-02"]
-    },
-    "facets": {
-      "severity": "critical",
-      "affectedAssetsTotal": 2,
-      "internetExposedAssets": 1
-    },
-    "recordLinks": [
-      {
-        "key": "affectedAssets",
-        "relation": "affects",
-        "direction": "out",
-        "target": {
-          "recordId": "entity:asset:10.100.18.64",
-          "target": "entity",
-          "objectType": "items"
-        },
-        "display": {
-          "name": "prod-api-01"
-        },
-        "indexed": true
-      },
-      {
-        "key": "affectedAssets",
-        "relation": "affects",
-        "direction": "out",
-        "target": {
-          "recordId": "entity:asset:10.100.18.65",
-          "target": "entity",
-          "objectType": "items"
-        },
-        "display": {
-          "name": "prod-db-02"
-        },
-        "indexed": true
-      }
-    ]
-  },
-
-  "sourceFacts": {
-    "schemaVersion": "sourceFacts.v1",
-    "values": {
-      "severityCode": {
-        "rawValue": 4,
-        "kind": "enum-code",
-        "readable": false,
-        "resolvedToPath": "data.severity",
-        "resolvedValue": "critical"
-      }
-    }
-  },
-
-  "_system": {
-    "memorix": {
-      "entity": "findings",
-      "contentType": "snapshots",
-      "operation": "upsert"
-    },
-    "processing": {
-      "graphRuns": {},
-      "jobTypeRuns": {}
-    }
-  }
-}
-```
-
----
-
-## N. Final mental model
-
-Use this as the lighthouse:
-
-```txt id="tbgvry"
-identity = how Memorix identifies the record
-
-lifecycle = when the record exists and changed
-
-concept = what the record is
-
-data = native readable facts about the record
-
-associatedData = readable related records/context attached for consumption
-
-analytics = numeric rollups about linked lists
-
-insights = interpretations, judgments, scores, and recommendations
-
-index = internal search/link/index surface
-
-sourceFacts = raw source values and resolver inputs
-
-_system = Memorix-managed metadata, processing, provenance, and audit
-```
-
-The clean separation is:
-
-```txt id="blxjhb"
-Facts stay in data.
-Related context goes in associatedData.
-Numbers about related context go in analytics.
-Conclusions go in insights.
-Links go in index.
-Raw source mess goes in sourceFacts.
-System state goes in _system.
-```
-
-
----------------------------
-
-```txt id="m7zj8c"
-The full rich Memorix record shape exists only on snapshot content types / snapshot collections.
-```
-
-Meaning these extra consumption/summary layers should be **snapshot-only by default**:
-
-```txt id="g8756x"
-associatedData
-analytics
-insights
-full record-level index projections
-```
-
-So for example:
-
-```txt id="cpyqdi"
-findings-snapshots  ✅ full record shape
-findings-events     ❌ not full shape
-findings-facts      ❌ not full shape
-findings-chunks     ❌ not full shape
-findings-history    ❌ not full shape
-```
-
-The clean rule:
-
-```txt id="0fdcos"
-Snapshots are the full readable object view.
-Other content types are specialized records.
-```
-
-For non-snapshot collections, the record should usually have only what the content type actually needs:
-
-```txt id="nqyy5v"
-identity
-lifecycle, if relevant
-concept, if useful
-data
-sourceFacts, if raw/resolution material exists
-_system
-minimal index, only if that content type needs indexing
-```
-
-So the final model should say:
-
-```txt id="8ba054"
-data exists on every content record.
-
-associatedData, analytics, and insights belong to snapshots only by default.
-
-index may exist outside snapshots only when the content type itself requires indexing, like chunks or searchable facts.
-
-_system and sourceFacts can exist anywhere when needed.
-```
-
-I would add this sentence to the appendix:
-
-```txt id="65cb87"
-Unless a descriptor explicitly opts in, associatedData, analytics, and insights are allowed only on snapshot content types, because they represent the full materialized view of the object, not the minimal record format for every collection.
-```