@x12i/memorix-format 1.0.0

package/docs/association-format.md

@@ -0,0 +1,466 @@
+# Association format
+
+An **association** connects one snapshot record to one or more related records. Associated content is stored as root-level properties on snapshot records using the reserved `associated` prefix.
+
+Associations are materialized by `@x12i/memorix-associator` presets, orchestrated by `@x12i/memorix-pipeline` enrichment. See [Running associations](./running-associations.md).
+
+---
+
+## Core rule
+
+Record-to-record association is always resolved from **snapshot record to snapshot record first**:
+
+1. The lookup starts from the current object's snapshot record.
+2. The matching target is another snapshot record in a `*-snapshots` collection.
+3. Use the matched target snapshot's `recordId` to fetch non-snapshot content (`discovery`, `analysis`, `decisions`, etc.).
+4. Copy flat `content.data` into `associated<ObjectType><ContentType>` on the source snapshot.
+
+The snapshot resolves the link; embedded content is always fetched **after** snapshot match by `recordId` (or equivalent link field such as `concept.identifier.recordId` or `data.sourceRecordId`).
+
+**Never join a source snapshot directly to a non-snapshot content collection by business keys** (for example `vulnerabilities-snapshots.data.assetIp` → `vulnerabilities-discovery.data.assetIp`). Same-object associations still match snapshots first, then fetch content by matched snapshot `recordId`.
+
+---
+
+## Naming conventions
+
+### Legacy (live data today)
+
+Short names without object type segment: `associatedData`, `associatedDiscovery`, `associatedAnalysis`, `associatedAssets`, `associatedSubnets`, etc.
+
+- Remain on existing records; not bulk-migrated.
+- Registry entries carry `"namingConvention": "legacy"`.
+- Legacy items may include inline `recordId` on copied payloads.
+
+### v2: `associated<ObjectType><ContentType>`
+
+| Segment | Rule | Example |
+|---------|------|---------|
+| Prefix | `associated` (reserved) | — |
+| ObjectType | PascalCase canonical object type | `security-groups` → `SecurityGroups` |
+| ContentType | PascalCase canonical content type | `discovery` → `Discovery` |
+
+Examples:
+
+```ts
+associatedSecurityGroupsAnalysis
+associatedVulnerabilitiesGroupsDiscovery
+associatedVulnerabilitiesGroupsSnapshots
+associatedUsersDecision
+```
+
+Do **not** pluralize the content type (`Discovery`, not `Discoveries`). The property is already an array.
+
+Registry entries for new associations carry `"namingConvention": "v2"`.
+
+### Ranked top associations: `associatedTop<ObjectType><ContentType>`
+
+Same copy forms as above, but candidates are **sorted by importance** then **capped**.
+
+| Segment | Rule | Example |
+|---------|------|---------|
+| Prefix | `associatedTop` | — |
+| ObjectType | PascalCase | `Vulnerabilities` |
+| ContentType | PascalCase | `Analysis` |
+
+Example: `associatedTopVulnerabilitiesAnalysis`
+
+**Importance** comes from the linked object type’s descriptor `importanceIndicator` (e.g. vulnerabilities → `data.severity`), or from the association rule’s `rankBy` override.
+
+**Default cap:** `5` (`DEFAULT_TOP_ASSOCIATED_ITEMS`). Override per rule with `maxItems`.
+
+**Empty results:** omit the property entirely (never write `[]`).
+
+```json
+{
+  "associatedProperty": "associatedTopVulnerabilitiesAnalysis",
+  "maxItems": 5,
+  "rankBy": {
+    "path": "data.severity",
+    "valueKind": "severity",
+    "order": "desc"
+  },
+  "match": {
+    "collection": "vulnerabilities-snapshots",
+    "joinPaths": ["data.assetIp"]
+  },
+  "content": {
+    "collection": "vulnerabilities-analysis",
+    "linkPaths": ["data.sourceRecordId", "recordId"]
+  },
+  "payload": { "source": "target.data", "format": "v2" }
+}
+```
+
+For snapshot-mediated rules, `rankBy.path` is evaluated on the **matched snapshot** (not the copied content). Severity order: `critical > high > medium > low > none`.
+
+### Derived (not copy-form associations)
+
+Resolver outputs such as `associatedTopology` are **derived** fields computed during pipeline resolution — not associator copy forms. Document separately; do not rename to v2 copy naming.
+
+---
+
+## Association forms
+
+The associator supports four copy forms via `rule.payload.source` (see request schema in `@x12i/memorix-associator`).
+
+### 1. Normal data association
+
+**Property:** `associated<ObjectType><ContentType>`
+
+**Copy:** linked record's `data` object (flat, not wrapped).
+
+```json
+{
+  "associatedSecurityGroupsAnalysis": [
+    {
+      "_system": {
+        "association": {
+          "fromProperty": "data.securityGroups",
+          "toProperty": "identifier",
+          "associatedAt": "2026-07-08T09:00:00.000Z",
+          "recordId": "security-group-record-id-1"
+        }
+      },
+      "summary": "The group appears to provide privileged access.",
+      "impact": "High",
+      "classification": "Privileged Security Group"
+    }
+  ]
+}
+```
+
+Request payload:
+
+```json
+{
+  "payload": {
+    "format": "v2",
+    "source": "target.data",
+    "system": { "include": true, "fromProperty": "data.securityGroups", "toProperty": "identifier" }
+  }
+}
+```
+
+Rules:
+
+- Business data stays flat beside `_system`.
+- Do **not** wrap copied data inside another `data` object.
+- `_system` is optional. Omit when consumed as business or LLM context.
+
+### 2. Snapshot root property association
+
+**Property:** `associated<ObjectType>Snapshots<RootProperty>`
+
+Copies a root property from the associated **snapshot** record (not a separate content collection).
+
+Example: `associatedVulnerabilitiesGroupsSnapshotsAnalytics` — copies `analytics` from `vulnerabilities-groups-snapshots`.
+
+```json
+{
+  "payload": {
+    "format": "v2",
+    "source": { "snapshotRootProperty": "analytics" },
+    "system": { "include": true }
+  }
+}
+```
+
+When `system.include` is true, the item is wrapped as `{ _system, value }`. Prefer the dedicated enrichment form below when copying the `enrichment` bucket (flat copy, no `value` wrapper).
+
+### Analytics rollups (sibling — not association)
+
+**Do not** use association rules to *compute* counts into the source record’s own `analytics` bucket. Associations **copy** linked payloads into `associated*`. Numeric rollups are a separate JSON rule type:
+
+- Spec: `.operational/analytics-rollups/FORMAT.md` (`memorix.analyticsRollup.v1`)
+- Presets: `.operational/analytics-rollups/presets/*.json`
+- Runner: `scripts/memorix-2.0-migrate/patches/run-analytics-rollups.mjs`
+
+Use the **same join keys** as association presets (`assetIp`, `name`↔`vulnerabilityName`, CIDR). Writing `analytics` on the target is a rollup; copying another record’s `analytics` into an `associated*SnapshotsAnalytics` array is still an association (form 2 above).
+
+### 2b. Enrichment association
+
+**Property:** `associated<ObjectType>Enrichment`
+
+Copies the linked **snapshot** root `enrichment` object **flat** (same shape as content associations: fields beside optional `_system`, not wrapped in `value`).
+
+`enrichment` is an envelope bucket on snapshots (CVSS, EPSS, KEV, compositeScore, …) — **not** a separate `*-enrichment` content-type collection.
+
+Example: `associatedVulnerabilitiesGroupsEnrichment` on `vulnerabilities-snapshots`.
+
+```json
+{
+  "associatedVulnerabilitiesGroupsEnrichment": [
+    {
+      "cvss": { "baseScore": 9.8 },
+      "epss": { "epss": 0.94 },
+      "kev": { "isKnownExploited": true },
+      "compositeScore": 0.97,
+      "riskLevel": "CRITICAL"
+    }
+  ]
+}
+```
+
+```json
+{
+  "associatedProperty": "associatedVulnerabilitiesGroupsEnrichment",
+  "payload": {
+    "format": "v2",
+    "source": "target.enrichment",
+    "includeRecordId": false
+  },
+  "target": {
+    "collection": "vulnerabilities-groups-snapshots",
+    "joinPaths": ["data.name"]
+  }
+}
+```
+
+Rules:
+
+- Omit the property when no linked snapshot has a non-empty `enrichment` object.
+- Do not invent a `vulnerabilities-groups-enrichment` collection for this form.
+- Naming helper: `buildAssociatedPropertyName({ form: "enrichment", objectType: "vulnerabilities-groups" })`.
+
+### 3. Concepts association
+
+**Property:** `associated<ObjectType>Concepts`
+
+Copies `concept` from associated snapshot records (not full `data`).
+
+```json
+{
+  "associatedSecurityGroupsConcepts": [
+    {
+      "identifier": { "kind": "group-name", "groupName": "Domain Admins" },
+      "title": "Domain Admins"
+    }
+  ]
+}
+```
+
+```json
+{
+  "payload": {
+    "format": "v2",
+    "source": "target.concept",
+    "system": { "include": true }
+  }
+}
+```
+
+### 4. Record IDs association
+
+**Property:** `associated<ObjectType>RecordIds` (use `RecordIds`, not `RecordsIds`)
+
+**Value:** array of strings. Must **not** include `_system`.
+
+```json
+{
+  "associatedSecurityGroupsRecordIds": [
+    "security-group-record-id-1",
+    "security-group-record-id-2"
+  ]
+}
+```
+
+```json
+{
+  "payload": {
+    "format": "v2",
+    "source": "target.recordId"
+  }
+}
+```
+
+---
+
+## `_system.association` metadata
+
+Reserved for technical traceability — not business data.
+
+| Field | Meaning |
+|-------|---------|
+| `fromProperty` | Path on the source snapshot that supplied the join value |
+| `toProperty` | Path on the linked snapshot used for matching |
+| `associatedAt` | ISO date-time when the association was created or refreshed |
+| `recordId` | Linked snapshot `recordId` (v2, when enabled) |
+
+Included when `payload.format === "v2"` and `payload.system.include === true`. All fields are optional even when `_system` is present.
+
+Strip `_system` when presenting associated content to humans, LLMs, or downstream analysis.
+
+---
+
+## Identifier and identity matching
+
+Prefer the linked snapshot's `identity` or `concept.identifier`:
+
+```json
+{ "identity": "group-name:Domain Admins" }
+```
+
+```json
+{
+  "identifier": { "kind": "group-name", "groupName": "Domain Admins" }
+}
+```
+
+Canonical location is `concept.identifier`. Root `identifier` / `identity` are accepted when present.
+
+Generate identity strings deterministically: join identifier values in order with `:` (e.g. `assetIp:10.100.18.64`).
+
+Association rules declare join paths in `sourceJoinPaths` and snapshot match paths in `match.joinPaths` (or legacy `target.joinPaths` for snapshot-only rules). Content fetch rules add `content.linkPaths` that reference the matched snapshot `recordId`. v2 rules should target `concept.identifier` or generated `identity` where possible.
+
+Associator request shape for content associations:
+
+```json
+{
+  "sourceJoinPaths": ["data.assetIp"],
+  "match": {
+    "collection": "vulnerabilities-snapshots",
+    "joinPaths": ["data.assetIp", "concept.identifier.assetIp", "recordId"]
+  },
+  "content": {
+    "collection": "vulnerabilities-discovery",
+    "contentType": "discovery",
+    "linkPaths": ["concept.identifier.recordId", "data.sourceRecordId", "recordId"]
+  }
+}
+```
+
+---
+
+## Worked example: security group analysis into users
+
+Collections:
+
+```text
+users-snapshots
+security-groups-snapshots
+security-groups-analysis
+```
+
+User snapshot source:
+
+```json
+{
+  "data": { "securityGroups": ["Domain Admins", "VPN Users"] }
+}
+```
+
+Security group snapshot identifier:
+
+```json
+{
+  "recordId": "security-group-record-id-1",
+  "concept": {
+    "identifier": { "kind": "group-name", "groupName": "Domain Admins" },
+    "title": "Domain Admins"
+  }
+}
+```
+
+Process:
+
+1. Start from `users-snapshots` record.
+2. Read join values from `data.securityGroups`.
+3. Match against `security-groups-snapshots` via `concept.identifier` / identity.
+4. Use matched snapshot `recordId` to fetch `security-groups-analysis` records.
+5. Copy each analysis record's `data` flat into `associatedSecurityGroupsAnalysis`.
+
+Result:
+
+```json
+{
+  "recordId": "user-record-id-1",
+  "data": {
+    "username": "ami",
+    "securityGroups": ["Domain Admins"]
+  },
+  "associatedSecurityGroupsAnalysis": [
+    {
+      "_system": {
+        "association": {
+          "fromProperty": "data.securityGroups",
+          "toProperty": "concept.identifier",
+          "associatedAt": "2026-07-08T09:00:00.000Z"
+        }
+      },
+      "summary": "The group appears to provide privileged access.",
+      "impact": "High",
+      "classification": "Privileged Security Group"
+    }
+  ]
+}
+```
+
+---
+
+## Implementation status
+
+| Capability | Status |
+|------------|--------|
+| Legacy `associatedData` / `associatedDiscovery` / … presets | **Live** — registry `namingConvention: "legacy"` |
+| v2 property naming | **Target** — new registry entries use `associated<ObjectType><ContentType>` |
+| v2 payload forms (`target.data`, `target.concept`, `target.recordId`, snapshot root) | **Supported** in `@x12i/memorix-associator` request schema |
+| v2 `_system.association` metadata | **Supported** when `format: "v2"` and `system.include: true` |
+| Registry-driven preset generation | **Live** — `generate-presets-from-registry.mjs` |
+| Full enrichment orchestration | **Live** — `run-enrichment-pipeline.mjs` |
+
+New associations should be added as registry entries + presets using v2 naming and payload format. Existing legacy properties continue to refresh through their current presets without rename.
+
+### Pilot: vulnerability discovery and decisions (v2)
+
+Snapshot-mediated flow:
+
+1. Read source `vulnerabilities-snapshots.data.assetIp`.
+2. Match target `vulnerabilities-snapshots` by snapshot paths (`data.assetIp`, `concept.identifier.assetIp`, `identity`, `recordId`).
+3. Take matched target snapshot `recordId`.
+4. Fetch `vulnerabilities-discovery` / `vulnerabilities-decisions` by record-id link (`concept.identifier.recordId`, `data.sourceRecordId`, or root `recordId`).
+5. Copy flat `content.data` into `associatedVulnerabilitiesDiscovery` / `associatedVulnerabilitiesDecisions`.
+
+| Property | Match collection | Content collection | Join key |
+|----------|------------------|--------------------|----------|
+| `associatedVulnerabilitiesDiscovery` | `vulnerabilities-snapshots` | `vulnerabilities-discovery` | `data.assetIp` → snapshot → `recordId` |
+| `associatedVulnerabilitiesDecisions` | `vulnerabilities-snapshots` | `vulnerabilities-decisions` | `data.assetIp` → snapshot → `recordId` |
+
+Target source collection: `vulnerabilities-snapshots` (`MEMORIX_EVENTS_DB`).
+
+Refresh rule keys:
+
+- `vulnerabilities.vulnerabilities.discovery.by-assetIp`
+- `vulnerabilities.vulnerabilities.decisions.by-assetIp`
+
+### Pilot: asset-side vulnerability analysis (v2)
+
+Snapshot-mediated flow (source is **assets**, not vulnerabilities):
+
+1. Read source `assets-snapshots.data.assetIps` (also `data.assetIp`, `data.ip_address`).
+2. Match `vulnerabilities-snapshots` by `data.assetIp` / `concept.identifier.assetIp`.
+3. Take matched snapshot `recordId`.
+4. Fetch `vulnerabilities-analysis` by record-id link.
+5. Copy flat `content.data` into `associatedVulnerabilitiesAnalysis` on the asset snapshot.
+
+| Property | Match collection | Content collection | Join key |
+|----------|------------------|--------------------|----------|
+| `associatedVulnerabilitiesAnalysis` | `vulnerabilities-snapshots` | `vulnerabilities-analysis` | `data.assetIps` → snapshot `data.assetIp` → `recordId` |
+
+Target source collection: `assets-snapshots` (`MEMORIX_ENTITIES_DB`).
+
+Refresh rule key: `assets.vulnerabilities.analysis.by-assetIp`
+
+---
+
+## Legacy properties reference
+
+| Property | Typical link |
+|----------|--------------|
+| `associatedData` | Linked snapshots / generic data |
+| `associatedDiscovery` | Linked discovery |
+| `associatedAnalysis` | Linked analysis |
+| `associatedAssets` | Asset snapshots by IP |
+| `associatedSubnets` | Subnet snapshots (explicit) |
+| `associatedTopology` | Derived (resolver, not associator copy) |
+| `associatedVulnerabilityGroup` | Vuln-group snapshots (explicit) |
+| `associatedZoneAnalysis` | Topology-zone analysis |