@x12i/memorix-associator 1.0.0

package/docs/running-associations.md

@@ -0,0 +1,340 @@
+# Running associations between object types
+
+This guide explains how to **plan**, **apply**, and **re-run** cross-record associations using `@x12i/memorix-associator`.
+
+An association copies linked records' `data` payloads onto source snapshot records under root-level `associated*` fields (for example `associatedData`, `associatedInferred`, or custom names like `associatedAssets`). The associator never writes back into linked collections.
+
+---
+
+## What you need before you start
+
+| Requirement | Notes |
+|-------------|-------|
+| MongoDB access | Set `MONGO_URI` (or the env var named in the request's `connection.mongoUriEnv`). |
+| Source collection | Snapshot records that will receive associated payloads. |
+| Linked collections | Records to join against; each must have a `data` object. |
+| Association request | A JSON file describing source, rules, and safety settings (see below). |
+| Built package | From this directory: `npm run build`, then use `npx memorix-associator` or the API. |
+
+---
+
+## Two ways to define an association
+
+### 1. Metadata-managed associations (preferred)
+
+When an entity descriptor already declares `associatedProperties`, the association is **managed**: metadata names the property, linked object type, linked content type, and usually a `refreshRuleKey`.
+
+Example from entity descriptors:
+
+```json
+{
+  "name": "associatedData",
+  "linkedObjectType": "subnets",
+  "linkedContentType": "snapshots",
+  "refreshRuleKey": "assets.subnets.snapshots.by-cidr",
+  "managed": true
+}
+```
+
+**How to run today**
+
+1. Find the managed property on the **source** entity descriptor (Schema UI → entity → Associated properties, or `associatedProperties` in the descriptor JSON).
+2. Use a request file whose `rules[].ruleKey` matches `refreshRuleKey` and whose join paths match the operational definition.
+3. Run the standard plan → review → apply workflow (below).
+
+The repo maintains a coverage registry at `.operational/metadata/associated-properties-registry.json` listing known managed associations, join paths, and last apply timestamps.
+
+**Known gap:** metadata describes *what* is associated and *which rule refreshes it*, but the associator still needs a full **request JSON** with join paths, collections, and resolver steps. There is not yet a `memorix-associator run --refresh-rule-key …` command that builds the request from metadata alone. Until that exists, keep request files alongside operational presets (or generate them from the registry).
+
+**Do not auto-refresh discovered-only properties.** Fields that exist only because records happen to have an `associated*` key (no managed descriptor entry) can be read by consumers, but you should not run a managed refresh for them until a rule and descriptor entry exist.
+
+### 2. Ad-hoc associations
+
+For experiments, one-off joins, or associations not yet promoted to metadata, write a request file directly. The same plan → apply → verify workflow applies.
+
+Use ad-hoc when:
+
+- You are prototyping a new link between two object types.
+- Metadata has not been updated yet.
+- You need a narrow `source.filter` (subset of records only).
+
+After a successful ad-hoc run, promote the association by adding `associatedProperties` to the entity descriptor and recording the `refreshRuleKey` in operational metadata.
+
+---
+
+## The workflow: plan → review → apply → verify
+
+Always run in this order. **Apply is gated:** when `safety.requireReviewedReportForApply` is `true` (recommended), apply refuses to run unless the plan fingerprint matches a reviewed dry-run report.
+
+```
+┌─────────┐     ┌─────────┐     ┌─────────┐     ┌─────────┐
+│  plan   │ ──► │ review  │ ──► │  apply  │ ──► │ verify  │
+│dry-run  │     │ report  │     │ writes  │     │ shapes  │
+└─────────┘     └─────────┘     └─────────┘     └─────────┘
+```
+
+### 1. Plan (dry-run)
+
+Scans source and linked collections, computes joins, and produces a report **without writing** to source snapshots (except failure collections are only written on apply).
+
+```bash
+export MONGO_URI="mongodb://…"
+
+memorix-associator plan \
+  --request examples/minimal-association-request.json \
+  --output temp/my-association-plan.json
+```
+
+**What to check in the report**
+
+| Field | Meaning |
+|-------|---------|
+| `planFingerprint` | Stable hash of pending updates; required for apply. |
+| `rules[]` | Per-rule scan counts, skipped rules, missing join stats. |
+| `sourceCollections[].sourceRecordsWouldUpdate` | How many source records would change. |
+| `sourceCollections[].associatedItemsWouldAppend` | New `data` payloads that would be appended. |
+| `failureCountsByReason` | Join/shape failures grouped by reason. |
+| `failureSamples` | Example failure records (when configured). |
+
+If counts look wrong, fix join paths, resolvers, or source data before applying.
+
+### 2. Review
+
+Open the plan report and confirm:
+
+- Rule keys and linked collections are correct.
+- Update counts are expected for the data volume.
+- Failure reasons are acceptable (optional rules may legitimately have gaps).
+- `planFingerprint` is noted (apply will require it).
+
+### 3. Apply
+
+Writes associated arrays on source records using **append-unique** merge. Optionally backs up the source collection when `safety.backupBeforeApply` is `true`.
+
+```bash
+memorix-associator apply \
+  --request examples/minimal-association-request.json \
+  --reviewed-report temp/my-association-plan.json
+```
+
+Apply fails if the current plan fingerprint does not match the reviewed report. That means underlying data or the request changed since the dry-run — **re-plan and review again**.
+
+### 4. Verify (optional but recommended)
+
+Checks that associated fields on source records are well-formed arrays of objects.
+
+```bash
+memorix-associator verify --request examples/minimal-association-request.json
+```
+
+You can also set `safety.verifyAfterPlan` or `safety.verifyAfterApply` in the request to run verification automatically.
+
+---
+
+## Re-running associations (important)
+
+Re-running is **safe and expected**. Association apply is **idempotent**: running the same request again after a successful apply should produce **zero** pending updates.
+
+### What re-run does
+
+1. **Normalizes** existing associated values to arrays and deduplicates by canonical deep equality.
+2. **Appends** linked `data` payloads that are not already present.
+3. **Omits** associated properties that would be empty.
+4. **Does not remove** previously appended items when a link disappears or linked `data` changes — merge is append-only. A changed linked payload may append as a *new* item if its canonical form differs; stale copies are not automatically pruned.
+
+### When to re-run
+
+| Trigger | Action |
+|---------|--------|
+| New source records ingested | Plan → apply same request. |
+| Linked records added or updated | Plan → apply; only new/changed payloads append. |
+| Join fields normalized (e.g. resolver wrote `data.subnetCidrs`) | Re-run after normalization/resolver steps. |
+| Previous apply failed partway | Fix data or request; re-plan from scratch. |
+| Routine refresh | Re-run managed requests on a schedule or after linked collection updates. |
+
+### How to confirm convergence
+
+After apply, run **plan** again with the same request:
+
+```bash
+memorix-associator plan \
+  --request examples/minimal-association-request.json \
+  --output temp/my-association-recheck.json
+```
+
+A converged run shows:
+
+- `associatedItemsWouldAppend`: **0**
+- `sourceRecordsWouldUpdate`: **0**
+- Empty or no pending updates in the plan
+
+That zero-append dry-run is the main signal that the process is deterministic and caught up.
+
+### Re-run checklist
+
+1. Use the **same** request file (or an updated version you have reviewed).
+2. Always **plan first** — never assume a previous fingerprint still applies.
+3. Compare the new fingerprint to the last reviewed report; if it changed, review the new plan before apply.
+4. For large collections, use `safety.limit` during investigation, then remove the limit for the full run.
+
+---
+
+## Request file structure (generic)
+
+Every run is driven by one JSON request. Minimal shape:
+
+```json
+{
+  "processId": "source-object.linked-object.by-join",
+  "mode": "dry-run",
+  "connection": {
+    "mongoUriEnv": "MONGO_URI"
+  },
+  "safety": {
+    "requireReviewedReportForApply": true,
+    "backupBeforeApply": true,
+    "verifyAfterApply": true
+  },
+  "source": {
+    "db": "entities",
+    "collection": "source-object-snapshots",
+    "objectType": "source-object",
+    "contentType": "snapshots"
+  },
+  "rules": [
+    {
+      "type": "association",
+      "ruleKey": "source-object.linked-object.snapshots.by-field",
+      "mandatory": false,
+      "sourceJoinPaths": ["data.joinField"],
+      "target": {
+        "db": "entities",
+        "collection": "linked-object-snapshots",
+        "objectType": "linked-object",
+        "contentType": "snapshots",
+        "joinPaths": ["data.matchField"]
+      },
+      "associatedProperty": "associatedData",
+      "payload": { "source": "target.data" }
+    }
+  ],
+  "failurePolicy": {
+    "writeFailures": true,
+    "includeSamplesInReport": 10
+  },
+  "writePolicy": {
+    "associatedArrayMode": "append-unique",
+    "omitEmptyAssociatedArrays": true,
+    "canonicalDeduplication": true
+  }
+}
+```
+
+### Rule fields
+
+| Field | Purpose |
+|-------|---------|
+| `ruleKey` | Stable id; should match `refreshRuleKey` in metadata when managed. |
+| `mandatory` | If `true`, missing links write a failure record to `<objectType>-failed`. |
+| `sourceJoinPaths` | Paths on the source record used as join keys (arrays fan out). |
+| `target.joinPaths` | Paths on linked records to match against. |
+| `target.bridge` | Optional bridge collection when join keys differ (e.g. name → record id). |
+| `associatedProperty` | Root field on the source snapshot; must start with `associated`. |
+| `payload.source` | Always `"target.data"` — only the linked record's `data` is copied. |
+
+### Optional: normalization and resolvers
+
+Run **before** association rules when join keys need preparation:
+
+- `normalization` — split comma-separated values into array join fields.
+- `resolvers` — domain logic (e.g. CIDR containment) via `@x12i/memorix-resolvers`.
+
+See [memorix-resolvers README](../../memorix-resolvers/README.md) for resolver types.
+
+### Optional: scope a run
+
+```json
+"source": {
+  "filter": { "recordId": "specific-record-id" },
+  ...
+},
+"safety": {
+  "limit": 100
+}
+```
+
+---
+
+## What gets written
+
+| Destination | Content |
+|-------------|---------|
+| Source snapshot `associated*` fields | Array of linked `data` objects (append-unique). |
+| `<source.objectType>-failed` | Technical failure records when `failurePolicy.writeFailures` is `true`. |
+| Backup collection | Created when `safety.backupBeforeApply` is `true` (apply only). |
+
+Associated arrays never contain join metadata, pointers, or wrapper objects — only the linked record's `data` payload.
+
+---
+
+## Programmatic API
+
+Same workflow as the CLI:
+
+```typescript
+import { MongoClient } from "mongodb";
+import {
+  planAssociations,
+  applyAssociations,
+  verifyAssociations,
+  createMongoRecordStore,
+} from "@x12i/memorix-associator";
+
+const client = new MongoClient(process.env.MONGO_URI!);
+await client.connect();
+const store = createMongoRecordStore(client);
+
+const plan = await planAssociations(request, { store });
+// Review plan.report.planFingerprint
+
+await applyAssociations(
+  {
+    ...request,
+    mode: "apply",
+    safety: {
+      ...request.safety,
+      expectedPlanFingerprint: plan.planFingerprint,
+    },
+  },
+  { store }
+);
+
+await verifyAssociations(request, { store });
+await client.close();
+```
+
+Explorer API also exposes `POST /associations/plan`, `/apply`, and `/verify` when running `memorix-explorer-api`.
+
+---
+
+## Failure reasons (quick reference)
+
+| Reason | Typical cause |
+|--------|----------------|
+| `JOIN_FIELD_MISSING_ON_SOURCE` | Source record lacks a join path value. |
+| `JOIN_FIELD_MISSING_ON_TARGET` | Linked record lacks a join path value. |
+| `MANDATORY_LINKED_CONTEXT_MISSING` | Mandatory rule found no match. |
+| `INVALID_DATA_SHAPE` | Linked record has no usable `data` object. |
+| `BRIDGE_COLLECTION_MISSING` | Bridge collection configured but not found. |
+| `TARGET_COLLECTION_MISSING` | Rule skipped — linked collection does not exist. |
+
+Inspect `failureSamples` in the plan report and the `<objectType>-failed` collection for details.
+
+---
+
+## Related documentation
+
+- [Package README](../README.md) — API surface and associated-property metadata.
+- [Memorix format — associated buckets](../../docs/memorix-format.md)
+- [Enrichment processes design note](../../memorix-schema-api/src/operational/memorix-enrichment-processes.md)
+- [Associated properties registry](../../.operational/metadata/associated-properties-registry.json) — live managed association catalog.

package/examples/minimal-association-request.json

@@ -0,0 +1,46 @@
+{
+  "processId": "source-object.linked-object.context",
+  "description": "Generic template — replace db, collection, object types, join paths, and associatedProperty for your association.",
+  "mode": "dry-run",
+  "connection": {
+    "mongoUriEnv": "MONGO_URI"
+  },
+  "safety": {
+    "requireReviewedReportForApply": true,
+    "backupBeforeApply": true,
+    "verifyAfterPlan": false,
+    "verifyAfterApply": true
+  },
+  "source": {
+    "db": "entities",
+    "collection": "source-object-snapshots",
+    "objectType": "source-object",
+    "contentType": "snapshots"
+  },
+  "rules": [
+    {
+      "type": "association",
+      "ruleKey": "source-object.linked-object.snapshots.by-field",
+      "mandatory": false,
+      "sourceJoinPaths": ["data.joinField"],
+      "target": {
+        "db": "entities",
+        "collection": "linked-object-snapshots",
+        "objectType": "linked-object",
+        "contentType": "snapshots",
+        "joinPaths": ["data.matchField"]
+      },
+      "associatedProperty": "associatedData",
+      "payload": { "source": "target.data" }
+    }
+  ],
+  "failurePolicy": {
+    "writeFailures": true,
+    "includeSamplesInReport": 10
+  },
+  "writePolicy": {
+    "associatedArrayMode": "append-unique",
+    "omitEmptyAssociatedArrays": true,
+    "canonicalDeduplication": true
+  }
+}

package/package.json

@@ -0,0 +1,57 @@
+{
+  "name": "@x12i/memorix-associator",
+  "version": "1.0.0",
+  "description": "Memorix cross-record association planning, apply, verification, and associated-property metadata",
+  "type": "module",
+  "main": "./dist/index.js",
+  "types": "./dist/index.d.ts",
+  "bin": {
+    "memorix-associator": "./dist/cli.js"
+  },
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "import": "./dist/index.js"
+    }
+  },
+  "files": [
+    "dist",
+    "README.md",
+    "docs",
+    "examples"
+  ],
+  "scripts": {
+    "prebuild": "rm -rf dist",
+    "build": "tsc -p tsconfig.build.json",
+    "test": "vitest run",
+    "test:watch": "vitest",
+    "prepack": "npm run build"
+  },
+  "engines": {
+    "node": ">=18.0.0"
+  },
+  "dependencies": {
+    "@x12i/memorix-resolvers": "^1.1.0",
+    "mongodb": "^6.21.0"
+  },
+  "devDependencies": {
+    "@types/node": "^20.14.0",
+    "typescript": "^5.6.0",
+    "vitest": "^2.1.0"
+  },
+  "publishConfig": {
+    "access": "public"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+ssh://git@github.com/x12i/memorix-mono-repo.git",
+    "directory": "memorix-associator"
+  },
+  "keywords": [
+    "memorix",
+    "associator",
+    "associated",
+    "enrichment"
+  ],
+  "license": "exellix-license"
+}