@x12i/memorix-format 2.2.2 → 2.3.2

package/docs/association-format.md

@@ -1,466 +1,47 @@
 # Association format
 
-An **association** connects one snapshot record to one or more related records. Associated content is stored as root-level properties on snapshot records using the reserved `associated` prefix.
+Associated content is stored as **root-level open buckets** on snapshot records using the reserved `associated` prefix.
 
-Associations are materialized by `@x12i/memorix-associator` presets, orchestrated by `@x12i/memorix-pipeline` enrichment. See [Running associations](./running-associations.md).
+**Normative naming / forms:** packaged [`meta/associated-conventions.json`](../meta/associated-conventions.json).  
+**Operational detail** (presets, apply, enrichment orchestration): [`memorix-pipeline/docs/association-format.md`](../../memorix-pipeline/docs/association-format.md) and [running-associations.md](../../memorix-pipeline/docs/running-associations.md).
 
----
-
-## Core rule
-
-Record-to-record association is always resolved from **snapshot record to snapshot record first**:
-
-1. The lookup starts from the current object's snapshot record.
-2. The matching target is another snapshot record in a `*-snapshots` collection.
-3. Use the matched target snapshot's `recordId` to fetch non-snapshot content (`discovery`, `analysis`, `decisions`, etc.).
-4. Copy flat `content.data` into `associated<ObjectType><ContentType>` on the source snapshot.
-
-The snapshot resolves the link; embedded content is always fetched **after** snapshot match by `recordId` (or equivalent link field such as `concept._extIdentifier.recordId` or `data.sourceRecordId`).
-
-**Never join a source snapshot directly to a non-snapshot content collection by business keys** (for example `findings-snapshots.data.sku` → `findings-discovery.data.sku`). Same-object associations still match snapshots first, then fetch content by matched snapshot `recordId`.
-
----
-
-## Naming conventions
-
-### Legacy (live data today)
-
-Short names without object type segment: `associatedData`, `associatedDiscovery`, `associatedAnalysis`, `associatedAssets`, `associatedSubnets`, etc.
-
-- Remain on existing records; not bulk-migrated.
-- Registry entries carry `"namingConvention": "legacy"`.
-- Legacy items may include inline `recordId` on copied payloads.
-
-### v2: `associated<ObjectType><ContentType>`
-
-| Segment | Rule | Example |
-|---------|------|---------|
-| Prefix | `associated` (reserved) | — |
-| ObjectType | PascalCase canonical object type | `security-groups` → `SecurityGroups` |
-| ContentType | PascalCase canonical content type | `discovery` → `Discovery` |
-
-Examples:
-
-```ts
-associatedSecurityGroupsAnalysis
-associatedVulnerabilitiesGroupsDiscovery
-associatedVulnerabilitiesGroupsSnapshots
-associatedUsersDecision
-```
-
-Do **not** pluralize the content type (`Discovery`, not `Discoveries`). The property is already an array.
-
-Registry entries for new associations carry `"namingConvention": "v2"`.
-
-### Ranked top associations: `associatedTop<ObjectType><ContentType>`
-
-Same copy forms as above, but candidates are **sorted by importance** then **capped**.
-
-| Segment | Rule | Example |
-|---------|------|---------|
-| Prefix | `associatedTop` | — |
-| ObjectType | PascalCase | `Vulnerabilities` |
-| ContentType | PascalCase | `Analysis` |
-
-Example: `associatedTopVulnerabilitiesAnalysis`
-
-**Importance** comes from the linked object type’s descriptor `importanceIndicator` (e.g. findings → `data.severity`), or from the association rule’s `rankBy` override.
-
-**Default cap:** `5` (`DEFAULT_TOP_ASSOCIATED_ITEMS`). Override per rule with `maxItems`.
-
-**Empty results:** omit the property entirely (never write `[]`).
-
-```json
-{
-  "associatedProperty": "associatedTopVulnerabilitiesAnalysis",
-  "maxItems": 5,
-  "rankBy": {
-    "path": "data.severity",
-    "valueKind": "severity",
-    "order": "desc"
-  },
-  "match": {
-    "collection": "findings-snapshots",
-    "joinPaths": ["data.sku"]
-  },
-  "content": {
-    "collection": "findings-analysis",
-    "linkPaths": ["data.sourceRecordId", "recordId"]
-  },
-  "payload": { "source": "target.data", "format": "v2" }
-}
-```
-
-For snapshot-mediated rules, `rankBy.path` is evaluated on the **matched snapshot** (not the copied content). Severity order: `critical > high > medium > low > none`.
-
-### Derived (not copy-form associations)
-
-Resolver outputs such as `associatedTopology` are **derived** fields computed during pipeline resolution — not associator copy forms. Document separately; do not rename to v2 copy naming.
-
----
-
-## Association forms
-
-The associator supports four copy forms via `rule.payload.source` (see request schema in `@x12i/memorix-associator`).
-
-### 1. Normal data association
-
-**Property:** `associated<ObjectType><ContentType>`
-
-**Copy:** linked record's `data` object (flat, not wrapped).
-
-```json
-{
-  "associatedSecurityGroupsAnalysis": [
-    {
-      "_system": {
-        "association": {
-          "fromProperty": "data.securityGroups",
-          "toProperty": "identifier",
-          "associatedAt": "2026-07-08T09:00:00.000Z",
-          "recordId": "security-group-record-id-1"
-        }
-      },
-      "summary": "The group appears to provide privileged access.",
-      "impact": "High",
-      "classification": "Privileged Security Group"
-    }
-  ]
-}
-```
-
-Request payload:
-
-```json
-{
-  "payload": {
-    "format": "v2",
-    "source": "target.data",
-    "system": { "include": true, "fromProperty": "data.securityGroups", "toProperty": "identifier" }
-  }
-}
-```
-
-Rules:
-
-- Business data stays flat beside `_system`.
-- Do **not** wrap copied data inside another `data` object.
-- `_system` is optional. Omit when consumed as business or LLM context.
-
-### 2. Snapshot root property association
-
-**Property:** `associated<ObjectType>Snapshots<RootProperty>`
-
-Copies a root property from the associated **snapshot** record (not a separate content collection).
-
-Example: `associatedVulnerabilitiesGroupsSnapshotsAnalytics` — copies `analytics` from `item-groups-snapshots`.
-
-```json
-{
-  "payload": {
-    "format": "v2",
-    "source": { "snapshotRootProperty": "analytics" },
-    "system": { "include": true }
-  }
-}
-```
-
-When `system.include` is true, the item is wrapped as `{ _system, value }`. Prefer the dedicated enrichment form below when copying the `enrichment` bucket (flat copy, no `value` wrapper).
-
-### Analytics rollups (sibling — not association)
-
-**Do not** use association rules to *compute* counts into the source record’s own `analytics` bucket. Associations **copy** linked payloads into `associated*`. Numeric rollups are a separate JSON rule type:
-
-- Spec: `.operational/analytics-rollups/FORMAT.md` (`memorix.analyticsRollup.v1`)
-- Presets: `.operational/analytics-rollups/presets/*.json`
-- Runner: `scripts/memorix-2.0-migrate/patches/run-analytics-rollups.mjs`
-
-Use the **same join keys** as association presets (`sku`, `name`↔`vulnerabilityName`, CIDR). Writing `analytics` on the target is a rollup; copying another record’s `analytics` into an `associated*SnapshotsAnalytics` array is still an association (form 2 above).
-
-### 2b. Enrichment association
-
-**Property:** `associated<ObjectType>Enrichment`
-
-Copies the linked **snapshot** root `enrichment` object **flat** (same shape as content associations: fields beside optional `_system`, not wrapped in `value`).
-
-`enrichment` is an envelope bucket on snapshots (external feed values as received) — **not** a separate `*-enrichment` content-type collection.
-
-Example: `associatedVulnerabilitiesGroupsEnrichment` on `findings-snapshots`.
-
-```json
-{
-  "associatedVulnerabilitiesGroupsEnrichment": [
-    {
-      "cvss": { "baseScore": 9.8 },
-      "epss": { "epss": 0.94 },
-      "kev": { "isKnownExploited": true },
-      "compositeScore": 0.97,
-      "riskLevel": "CRITICAL"
-    }
-  ]
-}
-```
-
-```json
-{
-  "associatedProperty": "associatedVulnerabilitiesGroupsEnrichment",
-  "payload": {
-    "format": "v2",
-    "source": "target.enrichment",
-    "includeRecordId": false
-  },
-  "target": {
-    "collection": "item-groups-snapshots",
-    "joinPaths": ["data.name"]
-  }
-}
-```
-
-Rules:
-
-- Omit the property when no linked snapshot has a non-empty `enrichment` object.
-- Do not invent a `item-groups-enrichment` collection for this form.
-- Naming helper: `buildAssociatedPropertyName({ form: "enrichment", objectType: "item-groups" })`.
-
-### 3. Concepts association
-
-**Property:** `associated<ObjectType>Concepts`
-
-Copies `concept` from associated snapshot records (not full `data`).
-
-```json
-{
-  "associatedSecurityGroupsConcepts": [
-    {
-      "identifier": { "kind": "group-name", "groupName": "Domain Admins" },
-      "title": "Domain Admins"
-    }
-  ]
-}
-```
-
-```json
-{
-  "payload": {
-    "format": "v2",
-    "source": "target.concept",
-    "system": { "include": true }
-  }
-}
-```
-
-### 4. Record IDs association
-
-**Property:** `associated<ObjectType>RecordIds` (use `RecordIds`, not `RecordsIds`)
-
-**Value:** array of strings. Must **not** include `_system`.
-
-```json
-{
-  "associatedSecurityGroupsRecordIds": [
-    "security-group-record-id-1",
-    "security-group-record-id-2"
-  ]
-}
-```
-
-```json
-{
-  "payload": {
-    "format": "v2",
-    "source": "target.recordId"
-  }
-}
-```
+This package validates association **shape** (ASSOC rules in [memorix-format-requirements.md](./memorix-format-requirements.md)); it does not invent object-type names.
 
 ---
 
-## `_system.association` metadata
-
-Reserved for technical traceability — not business data.
-
-| Field | Meaning |
-|-------|---------|
-| `fromProperty` | Path on the source snapshot that supplied the join value |
-| `toProperty` | Path on the linked snapshot used for matching |
-| `associatedAt` | ISO date-time when the association was created or refreshed |
-| `recordId` | Linked snapshot `recordId` (v2, when enabled) |
-
-Included when `payload.format === "v2"` and `payload.system.include === true`. All fields are optional even when `_system` is present.
-
-Strip `_system` when presenting associated content to humans, LLMs, or downstream analysis.
-
----
-
-## Identifier and identity matching
-
-Prefer the linked snapshot's `identity` or `concept._extIdentifier`:
-
-```json
-{ "identity": "group-name:Domain Admins" }
-```
-
-```json
-{
-  "identifier": { "kind": "group-name", "groupName": "Domain Admins" }
-}
-```
-
-Canonical location is `concept._extIdentifier`. Root `identifier` / `identity` are accepted when present.
-
-Generate identity strings deterministically: join identifier values in order with `:` (e.g. `sku:10.100.18.64`).
+## Core rule
 
-Association rules declare join paths in `sourceJoinPaths` and snapshot match paths in `match.joinPaths` (or legacy `target.joinPaths` for snapshot-only rules). Content fetch rules add `content.linkPaths` that reference the matched snapshot `recordId`. v2 rules should target `concept._extIdentifier` or generated `identity` where possible.
+Association is always **snapshot-first**:
 
-Associator request shape for content associations:
+1. Start from the source object’s snapshot.
+2. Match a target **snapshot** (`*-snapshots`).
+3. Use the matched snapshot’s `recordId` to fetch non-snapshot content (`discovery`, `analysis`, `decisions`, …).
+4. Copy flat content into an `associated…` array on the source snapshot.
 
-```json
-{
-  "sourceJoinPaths": ["data.sku"],
-  "match": {
-    "collection": "findings-snapshots",
-    "joinPaths": ["data.sku", "concept._extIdentifier.sku", "recordId"]
-  },
-  "content": {
-    "collection": "findings-discovery",
-    "contentType": "discovery",
-    "linkPaths": ["concept._extIdentifier.recordId", "data.sourceRecordId", "recordId"]
-  }
-}
-```
+Never join a source snapshot directly to a non-snapshot collection by business keys.
 
 ---
 
-## Worked example: security group analysis into users
-
-Collections:
-
-```text
-users-snapshots
-security-groups-snapshots
-security-groups-analysis
-```
-
-User snapshot source:
-
-```json
-{
-  "data": { "securityGroups": ["Domain Admins", "VPN Users"] }
-}
-```
+## Naming
 
-Security group snapshot identifier:
+| Form | Pattern | Example (abstract) |
+|---|---|---|
+| Legacy short names | `associatedData`, `associatedDiscovery`, … | still present on older records |
+| v2 content | `associated<ObjectType><ContentType>` | `associatedItemGroupsAnalysis` |
+| Ranked top | `associatedTop<ObjectType><ContentType>` | `associatedTopItemsAnalysis` |
+| Enrichment copy | `associated<ObjectType>Enrichment` | `associatedItemGroupsEnrichment` |
 
-```json
-{
-  "recordId": "security-group-record-id-1",
-  "concept": {
-    "identifier": { "kind": "group-name", "groupName": "Domain Admins" },
-    "title": "Domain Admins"
-  }
-}
-```
-
-Process:
-
-1. Start from `users-snapshots` record.
-2. Read join values from `data.securityGroups`.
-3. Match against `security-groups-snapshots` via `concept._extIdentifier` / identity.
-4. Use matched snapshot `recordId` to fetch `security-groups-analysis` records.
-5. Copy each analysis record's `data` flat into `associatedSecurityGroupsAnalysis`.
-
-Result:
-
-```json
-{
-  "recordId": "user-record-id-1",
-  "data": {
-    "username": "ami",
-    "securityGroups": ["Domain Admins"]
-  },
-  "associatedSecurityGroupsAnalysis": [
-    {
-      "_system": {
-        "association": {
-          "fromProperty": "data.securityGroups",
-          "toProperty": "concept._extIdentifier",
-          "associatedAt": "2026-07-08T09:00:00.000Z"
-        }
-      },
-      "summary": "The group appears to provide privileged access.",
-      "impact": "High",
-      "classification": "Privileged Security Group"
-    }
-  ]
-}
-```
+Object-type segments come from Catalox descriptors — not from this format package.
 
 ---
 
-## Implementation status
-
-| Capability | Status |
-|------------|--------|
-| Legacy `associatedData` / `associatedDiscovery` / … presets | **Live** — registry `namingConvention: "legacy"` |
-| v2 property naming | **Target** — new registry entries use `associated<ObjectType><ContentType>` |
-| v2 payload forms (`target.data`, `target.concept`, `target.recordId`, snapshot root) | **Supported** in `@x12i/memorix-associator` request schema |
-| v2 `_system.association` metadata | **Supported** when `format: "v2"` and `system.include: true` |
-| Registry-driven preset generation | **Live** — `generate-presets-from-registry.mjs` |
-| Full enrichment orchestration | **Live** — `run-enrichment-pipeline.mjs` |
-
-New associations should be added as registry entries + presets using v2 naming and payload format. Existing legacy properties continue to refresh through their current presets without rename.
-
-### Pilot: vulnerability discovery and decisions (v2)
+## Payload rules
 
-Snapshot-mediated flow:
-
-1. Read source `findings-snapshots.data.sku`.
-2. Match target `findings-snapshots` by snapshot paths (`data.sku`, `concept._extIdentifier.sku`, `identity`, `recordId`).
-3. Take matched target snapshot `recordId`.
-4. Fetch `findings-discovery` / `findings-decisions` by record-id link (`concept._extIdentifier.recordId`, `data.sourceRecordId`, or root `recordId`).
-5. Copy flat `content.data` into `associatedVulnerabilitiesDiscovery` / `associatedVulnerabilitiesDecisions`.
-
-| Property | Match collection | Content collection | Join key |
-|----------|------------------|--------------------|----------|
-| `associatedVulnerabilitiesDiscovery` | `findings-snapshots` | `findings-discovery` | `data.sku` → snapshot → `recordId` |
-| `associatedVulnerabilitiesDecisions` | `findings-snapshots` | `findings-decisions` | `data.sku` → snapshot → `recordId` |
-
-Target source collection: `findings-snapshots` (`MEMORIX_EVENTS_DB`).
-
-Refresh rule keys:
-
-- `findings.findings.discovery.by-sku`
-- `findings.findings.decisions.by-sku`
-
-### Pilot: asset-side vulnerability analysis (v2)
-
-Snapshot-mediated flow (source is **items**, not findings):
-
-1. Read source `items-snapshots.data.skus` (also `data.sku`, `data.ip_address`).
-2. Match `findings-snapshots` by `data.sku` / `concept._extIdentifier.sku`.
-3. Take matched snapshot `recordId`.
-4. Fetch `findings-analysis` by record-id link.
-5. Copy flat `content.data` into `associatedVulnerabilitiesAnalysis` on the asset snapshot.
-
-| Property | Match collection | Content collection | Join key |
-|----------|------------------|--------------------|----------|
-| `associatedVulnerabilitiesAnalysis` | `findings-snapshots` | `findings-analysis` | `data.skus` → snapshot `data.sku` → `recordId` |
-
-Target source collection: `items-snapshots` (`MEMORIX_ENTITIES_DB`).
-
-Refresh rule key: `items.findings.analysis.by-sku`
-
----
+- Associated properties are **arrays of plain objects** (linked payloads), never full envelopes.
+- On **memorix-record/2**, root `associated*` may hold **lightweight refs only** (Option A — format-valid): `relationshipDefinitionId` + `recordId`, optional `relationshipId`. Copied related business payload is rejected.
+- **Authoritative relationship storage (Gate G4)** is centralized per-category collections (Candidate C), not Option A roots or sibling `relationships` content. See `docs/re-factoring-and-ops-agent/RELATIONSHIP-STORAGE-DECISION.md`.
 
-## Legacy properties reference
+- **Omit** the property when empty — never write `[]`.
+- Prefer traceability in optional item-level `_system.association` (v2) rather than wrapping envelopes.
 
-| Property | Typical link |
-|----------|--------------|
-| `associatedData` | Linked snapshots / generic data |
-| `associatedDiscovery` | Linked discovery |
-| `associatedAnalysis` | Linked analysis |
-| `associatedAssets` | Asset snapshots by IP |
-| `associatedSubnets` | Subnet snapshots (explicit) |
-| `associatedTopology` | Derived (resolver, not associator copy) |
-| `associatedVulnerabilityGroup` | Vuln-group snapshots (explicit) |
-| `associatedZoneAnalysis` | Topology-zone analysis |
+See requirements **ASSOC-*** and the pipeline association doc for matching / payload forms.