@x12i/memorix-format 2.2.1 → 2.2.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +4 -0
- package/README.md +67 -30
- package/docs/association-format.md +24 -446
- package/docs/memorix-format.md +53 -103
- package/docs/memorix-workflow-format.md +1 -1
- package/examples/valid-snapshot.json +10 -10
- package/package.json +1 -1
- package/docs/memorix-format-additions.md +0 -956
- package/docs/memorix-format-for-dummies-association-corrections.md +0 -113
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,9 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 2.2.3
|
|
4
|
+
|
|
5
|
+
- Docs hygiene: drop identity-era appendix and ebook correction notes; association doc is a thin present-tense stub pointing at `meta/associated-conventions.json` + pipeline ops docs.
|
|
6
|
+
|
|
3
7
|
## 2.2.0
|
|
4
8
|
|
|
5
9
|
- Depends on `@x12i/memorix-base-format@^1.2.0` (`concept._extIdentifier`, `_system.state`).
|
package/README.md
CHANGED
|
@@ -1,38 +1,25 @@
|
|
|
1
1
|
# @x12i/memorix-format
|
|
2
2
|
|
|
3
|
-
Memorix **product** record format —
|
|
3
|
+
Memorix **product** record format — open-bucket conventions and migrate tooling layered on
|
|
4
4
|
[`@x12i/memorix-base-format`](https://www.npmjs.com/package/@x12i/memorix-base-format).
|
|
5
5
|
|
|
6
6
|
| Package | Owns |
|
|
7
7
|
|---|---|
|
|
8
8
|
| `@x12i/memorix-base-format` | Fixed skeleton: `recordId`, `concept`, `data`, `_…`; unnamed open buckets; ContentTypeMeta |
|
|
9
|
-
| `@x12i/memorix-format` |
|
|
9
|
+
| `@x12i/memorix-format` | Known open-bucket meanings, `associated*` rules, snapshot SCOPE, built-in profiles, **migrate** CLI |
|
|
10
10
|
|
|
11
|
-
|
|
11
|
+
**Object types belong in Catalox descriptors, not here.** Format docs and examples use abstract names (`items`, `groups`, `locations`, `documents`) so domain packs are never confused with format law.
|
|
12
12
|
|
|
13
|
-
|
|
13
|
+
Product meta lives under [`meta/`](./meta/). Managed-descriptor inventory in `meta/inventory/` is not format law.
|
|
14
14
|
|
|
15
|
-
|
|
15
|
+
Normative: [docs/memorix-format-requirements.md](./docs/memorix-format-requirements.md). Narrative: [docs/memorix-format.md](./docs/memorix-format.md).
|
|
16
16
|
|
|
17
|
-
|
|
18
|
-
# Dry-run validate after transform (stdout summary)
|
|
19
|
-
npx memorix-format migrate ./export-dir/ --content-type=snapshots
|
|
20
|
-
|
|
21
|
-
# Write cleaned files
|
|
22
|
-
npx memorix-format migrate ./export-dir/ --out=./clean-dir/
|
|
23
|
-
|
|
24
|
-
# Or overwrite in place after validation passes
|
|
25
|
-
npx memorix-format migrate ./records.json --in-place
|
|
26
|
-
```
|
|
27
|
-
|
|
28
|
-
Defaults applied by migrate:
|
|
17
|
+
## Layering
|
|
29
18
|
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
Writers set `_system.state: "active"` on new records. Pipeline format enforce defaults to **hard** (`MEMORIX_FORMAT_ENFORCE=warn` to soften during cutover).
|
|
19
|
+
1. Base envelope (identity / core / underscore / open).
|
|
20
|
+
2. Product open buckets when present: `analytics`, `enrichment`, `insights`, `associated*`.
|
|
21
|
+
3. **SCOPE:** full open-bucket surface primarily on **snapshots**; other content types keep substance in `.data` unless ContentTypeMeta opts in.
|
|
22
|
+
4. Readers / human projection **omit** root `_…` and `concept._extIdentifier` by default; keep `recordId` for joins.
|
|
36
23
|
|
|
37
24
|
## Install
|
|
38
25
|
|
|
@@ -41,6 +28,27 @@ npm install @x12i/memorix-format
|
|
|
41
28
|
# pulls @x12i/memorix-base-format
|
|
42
29
|
```
|
|
43
30
|
|
|
31
|
+
## Abstract snapshot example
|
|
32
|
+
|
|
33
|
+
```json
|
|
34
|
+
{
|
|
35
|
+
"recordId": "rec-item-1",
|
|
36
|
+
"concept": {
|
|
37
|
+
"_extIdentifier": {
|
|
38
|
+
"kind": "sku",
|
|
39
|
+
"sku": "SKU-100",
|
|
40
|
+
"key": "sku:SKU-100"
|
|
41
|
+
},
|
|
42
|
+
"title": "Blue widget"
|
|
43
|
+
},
|
|
44
|
+
"data": { "sku": "SKU-100", "color": "blue" },
|
|
45
|
+
"associatedData": [{ "locationCode": "WH-A", "qty": 12 }],
|
|
46
|
+
"analytics": { "linkedLocations": { "total": 1 } },
|
|
47
|
+
"insights": { "priority": { "level": "high", "confidence": 0.9 } },
|
|
48
|
+
"_system": { "state": "active" }
|
|
49
|
+
}
|
|
50
|
+
```
|
|
51
|
+
|
|
44
52
|
## SDK
|
|
45
53
|
|
|
46
54
|
```ts
|
|
@@ -51,14 +59,12 @@ import {
|
|
|
51
59
|
snapshotsContentTypeMeta,
|
|
52
60
|
} from "@x12i/memorix-format";
|
|
53
61
|
|
|
54
|
-
// Product validation (base + ASSOC/SCOPE + optional profile)
|
|
55
62
|
validateMemorixRecord(record, {
|
|
56
63
|
profile: "snapshots",
|
|
57
|
-
objectType: "
|
|
64
|
+
objectType: "items",
|
|
58
65
|
contentType: "snapshots",
|
|
59
66
|
});
|
|
60
67
|
|
|
61
|
-
// Base only (no product vocabulary)
|
|
62
68
|
validateMemorixBaseRecord(record);
|
|
63
69
|
classifyRootKey("associatedData"); // "open"
|
|
64
70
|
```
|
|
@@ -66,13 +72,44 @@ classifyRootKey("associatedData"); // "open"
|
|
|
66
72
|
## CLI
|
|
67
73
|
|
|
68
74
|
```bash
|
|
69
|
-
npx memorix-format validate ./record.json --profile=snapshots --object-type=
|
|
75
|
+
npx memorix-format validate ./record.json --profile=snapshots --object-type=items
|
|
70
76
|
npx memorix-format validate ./dir/ --content-type=events
|
|
71
77
|
npx memorix-format profiles
|
|
72
78
|
npx memorix-format docs memorix-format-requirements.md
|
|
73
79
|
```
|
|
74
80
|
|
|
75
|
-
##
|
|
81
|
+
## Migrate (recommended transform)
|
|
82
|
+
|
|
83
|
+
Present-tense format only. Transform exports / sample JSON; **live Mongo apply is a separate ops checklist**.
|
|
84
|
+
|
|
85
|
+
```bash
|
|
86
|
+
# Dry-run (stdout summary; validates after transform)
|
|
87
|
+
npx memorix-format migrate ./export-dir/ --content-type=snapshots
|
|
88
|
+
|
|
89
|
+
# Write cleaned files
|
|
90
|
+
npx memorix-format migrate ./export-dir/ --out=./clean-dir/
|
|
91
|
+
|
|
92
|
+
# Overwrite in place after validation passes
|
|
93
|
+
npx memorix-format migrate ./records.json --in-place
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
Defaults applied by migrate:
|
|
97
|
+
|
|
98
|
+
- `concept.identifier` → `concept._extIdentifier` (`identifierKey` → `key`)
|
|
99
|
+
- root `identity` / `lifecycle` / `sourceFacts` / `index` removed (lifecycle events folded into `_system.lifecycle`)
|
|
100
|
+
- root `_graphRuns` / `_jobTypeRuns` → under `_system`
|
|
101
|
+
- `_system.state` set to `active` when missing (maps old `lifecycle.status` when present)
|
|
102
|
+
|
|
103
|
+
Writers set `_system.state: "active"` on new records. Pipeline format enforce defaults to **hard** (`MEMORIX_FORMAT_ENFORCE=warn` to soften during cutover).
|
|
104
|
+
|
|
105
|
+
### Runbook
|
|
106
|
+
|
|
107
|
+
1. Dry-run against an export or package examples.
|
|
108
|
+
2. Write `--out` and re-validate.
|
|
109
|
+
3. Only then apply to Mongo under ops change control (not part of this package’s default workflow).
|
|
110
|
+
|
|
111
|
+
## Related
|
|
76
112
|
|
|
77
|
-
|
|
78
|
-
|
|
113
|
+
- Base envelope SOT: [`@x12i/memorix-base-format`](https://www.npmjs.com/package/@x12i/memorix-base-format)
|
|
114
|
+
- Associations: [docs/association-format.md](./docs/association-format.md)
|
|
115
|
+
- Changelog: [CHANGELOG.md](./CHANGELOG.md)
|
|
@@ -1,466 +1,44 @@
|
|
|
1
1
|
# Association format
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Associated content is stored as **root-level open buckets** on snapshot records using the reserved `associated` prefix.
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
**Normative naming / forms:** packaged [`meta/associated-conventions.json`](../meta/associated-conventions.json).
|
|
6
|
+
**Operational detail** (presets, apply, enrichment orchestration): [`memorix-pipeline/docs/association-format.md`](../../memorix-pipeline/docs/association-format.md) and [running-associations.md](../../memorix-pipeline/docs/running-associations.md).
|
|
6
7
|
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
## Core rule
|
|
10
|
-
|
|
11
|
-
Record-to-record association is always resolved from **snapshot record to snapshot record first**:
|
|
12
|
-
|
|
13
|
-
1. The lookup starts from the current object's snapshot record.
|
|
14
|
-
2. The matching target is another snapshot record in a `*-snapshots` collection.
|
|
15
|
-
3. Use the matched target snapshot's `recordId` to fetch non-snapshot content (`discovery`, `analysis`, `decisions`, etc.).
|
|
16
|
-
4. Copy flat `content.data` into `associated<ObjectType><ContentType>` on the source snapshot.
|
|
17
|
-
|
|
18
|
-
The snapshot resolves the link; embedded content is always fetched **after** snapshot match by `recordId` (or equivalent link field such as `concept.identifier.recordId` or `data.sourceRecordId`).
|
|
19
|
-
|
|
20
|
-
**Never join a source snapshot directly to a non-snapshot content collection by business keys** (for example `vulnerabilities-snapshots.data.assetIp` → `vulnerabilities-discovery.data.assetIp`). Same-object associations still match snapshots first, then fetch content by matched snapshot `recordId`.
|
|
8
|
+
This package validates association **shape** (ASSOC rules in [memorix-format-requirements.md](./memorix-format-requirements.md)); it does not invent object-type names.
|
|
21
9
|
|
|
22
10
|
---
|
|
23
11
|
|
|
24
|
-
##
|
|
25
|
-
|
|
26
|
-
### Legacy (live data today)
|
|
27
|
-
|
|
28
|
-
Short names without object type segment: `associatedData`, `associatedDiscovery`, `associatedAnalysis`, `associatedAssets`, `associatedSubnets`, etc.
|
|
29
|
-
|
|
30
|
-
- Remain on existing records; not bulk-migrated.
|
|
31
|
-
- Registry entries carry `"namingConvention": "legacy"`.
|
|
32
|
-
- Legacy items may include inline `recordId` on copied payloads.
|
|
33
|
-
|
|
34
|
-
### v2: `associated<ObjectType><ContentType>`
|
|
35
|
-
|
|
36
|
-
| Segment | Rule | Example |
|
|
37
|
-
|---------|------|---------|
|
|
38
|
-
| Prefix | `associated` (reserved) | — |
|
|
39
|
-
| ObjectType | PascalCase canonical object type | `security-groups` → `SecurityGroups` |
|
|
40
|
-
| ContentType | PascalCase canonical content type | `discovery` → `Discovery` |
|
|
41
|
-
|
|
42
|
-
Examples:
|
|
43
|
-
|
|
44
|
-
```ts
|
|
45
|
-
associatedSecurityGroupsAnalysis
|
|
46
|
-
associatedVulnerabilitiesGroupsDiscovery
|
|
47
|
-
associatedVulnerabilitiesGroupsSnapshots
|
|
48
|
-
associatedUsersDecision
|
|
49
|
-
```
|
|
50
|
-
|
|
51
|
-
Do **not** pluralize the content type (`Discovery`, not `Discoveries`). The property is already an array.
|
|
52
|
-
|
|
53
|
-
Registry entries for new associations carry `"namingConvention": "v2"`.
|
|
54
|
-
|
|
55
|
-
### Ranked top associations: `associatedTop<ObjectType><ContentType>`
|
|
56
|
-
|
|
57
|
-
Same copy forms as above, but candidates are **sorted by importance** then **capped**.
|
|
58
|
-
|
|
59
|
-
| Segment | Rule | Example |
|
|
60
|
-
|---------|------|---------|
|
|
61
|
-
| Prefix | `associatedTop` | — |
|
|
62
|
-
| ObjectType | PascalCase | `Vulnerabilities` |
|
|
63
|
-
| ContentType | PascalCase | `Analysis` |
|
|
64
|
-
|
|
65
|
-
Example: `associatedTopVulnerabilitiesAnalysis`
|
|
66
|
-
|
|
67
|
-
**Importance** comes from the linked object type’s descriptor `importanceIndicator` (e.g. vulnerabilities → `data.severity`), or from the association rule’s `rankBy` override.
|
|
68
|
-
|
|
69
|
-
**Default cap:** `5` (`DEFAULT_TOP_ASSOCIATED_ITEMS`). Override per rule with `maxItems`.
|
|
70
|
-
|
|
71
|
-
**Empty results:** omit the property entirely (never write `[]`).
|
|
72
|
-
|
|
73
|
-
```json
|
|
74
|
-
{
|
|
75
|
-
"associatedProperty": "associatedTopVulnerabilitiesAnalysis",
|
|
76
|
-
"maxItems": 5,
|
|
77
|
-
"rankBy": {
|
|
78
|
-
"path": "data.severity",
|
|
79
|
-
"valueKind": "severity",
|
|
80
|
-
"order": "desc"
|
|
81
|
-
},
|
|
82
|
-
"match": {
|
|
83
|
-
"collection": "vulnerabilities-snapshots",
|
|
84
|
-
"joinPaths": ["data.assetIp"]
|
|
85
|
-
},
|
|
86
|
-
"content": {
|
|
87
|
-
"collection": "vulnerabilities-analysis",
|
|
88
|
-
"linkPaths": ["data.sourceRecordId", "recordId"]
|
|
89
|
-
},
|
|
90
|
-
"payload": { "source": "target.data", "format": "v2" }
|
|
91
|
-
}
|
|
92
|
-
```
|
|
93
|
-
|
|
94
|
-
For snapshot-mediated rules, `rankBy.path` is evaluated on the **matched snapshot** (not the copied content). Severity order: `critical > high > medium > low > none`.
|
|
95
|
-
|
|
96
|
-
### Derived (not copy-form associations)
|
|
97
|
-
|
|
98
|
-
Resolver outputs such as `associatedTopology` are **derived** fields computed during pipeline resolution — not associator copy forms. Document separately; do not rename to v2 copy naming.
|
|
99
|
-
|
|
100
|
-
---
|
|
101
|
-
|
|
102
|
-
## Association forms
|
|
103
|
-
|
|
104
|
-
The associator supports four copy forms via `rule.payload.source` (see request schema in `@x12i/memorix-associator`).
|
|
105
|
-
|
|
106
|
-
### 1. Normal data association
|
|
107
|
-
|
|
108
|
-
**Property:** `associated<ObjectType><ContentType>`
|
|
109
|
-
|
|
110
|
-
**Copy:** linked record's `data` object (flat, not wrapped).
|
|
111
|
-
|
|
112
|
-
```json
|
|
113
|
-
{
|
|
114
|
-
"associatedSecurityGroupsAnalysis": [
|
|
115
|
-
{
|
|
116
|
-
"_system": {
|
|
117
|
-
"association": {
|
|
118
|
-
"fromProperty": "data.securityGroups",
|
|
119
|
-
"toProperty": "identifier",
|
|
120
|
-
"associatedAt": "2026-07-08T09:00:00.000Z",
|
|
121
|
-
"recordId": "security-group-record-id-1"
|
|
122
|
-
}
|
|
123
|
-
},
|
|
124
|
-
"summary": "The group appears to provide privileged access.",
|
|
125
|
-
"impact": "High",
|
|
126
|
-
"classification": "Privileged Security Group"
|
|
127
|
-
}
|
|
128
|
-
]
|
|
129
|
-
}
|
|
130
|
-
```
|
|
131
|
-
|
|
132
|
-
Request payload:
|
|
133
|
-
|
|
134
|
-
```json
|
|
135
|
-
{
|
|
136
|
-
"payload": {
|
|
137
|
-
"format": "v2",
|
|
138
|
-
"source": "target.data",
|
|
139
|
-
"system": { "include": true, "fromProperty": "data.securityGroups", "toProperty": "identifier" }
|
|
140
|
-
}
|
|
141
|
-
}
|
|
142
|
-
```
|
|
143
|
-
|
|
144
|
-
Rules:
|
|
145
|
-
|
|
146
|
-
- Business data stays flat beside `_system`.
|
|
147
|
-
- Do **not** wrap copied data inside another `data` object.
|
|
148
|
-
- `_system` is optional. Omit when consumed as business or LLM context.
|
|
149
|
-
|
|
150
|
-
### 2. Snapshot root property association
|
|
151
|
-
|
|
152
|
-
**Property:** `associated<ObjectType>Snapshots<RootProperty>`
|
|
153
|
-
|
|
154
|
-
Copies a root property from the associated **snapshot** record (not a separate content collection).
|
|
155
|
-
|
|
156
|
-
Example: `associatedVulnerabilitiesGroupsSnapshotsAnalytics` — copies `analytics` from `vulnerabilities-groups-snapshots`.
|
|
157
|
-
|
|
158
|
-
```json
|
|
159
|
-
{
|
|
160
|
-
"payload": {
|
|
161
|
-
"format": "v2",
|
|
162
|
-
"source": { "snapshotRootProperty": "analytics" },
|
|
163
|
-
"system": { "include": true }
|
|
164
|
-
}
|
|
165
|
-
}
|
|
166
|
-
```
|
|
167
|
-
|
|
168
|
-
When `system.include` is true, the item is wrapped as `{ _system, value }`. Prefer the dedicated enrichment form below when copying the `enrichment` bucket (flat copy, no `value` wrapper).
|
|
169
|
-
|
|
170
|
-
### Analytics rollups (sibling — not association)
|
|
171
|
-
|
|
172
|
-
**Do not** use association rules to *compute* counts into the source record’s own `analytics` bucket. Associations **copy** linked payloads into `associated*`. Numeric rollups are a separate JSON rule type:
|
|
173
|
-
|
|
174
|
-
- Spec: `.operational/analytics-rollups/FORMAT.md` (`memorix.analyticsRollup.v1`)
|
|
175
|
-
- Presets: `.operational/analytics-rollups/presets/*.json`
|
|
176
|
-
- Runner: `scripts/memorix-2.0-migrate/patches/run-analytics-rollups.mjs`
|
|
177
|
-
|
|
178
|
-
Use the **same join keys** as association presets (`assetIp`, `name`↔`vulnerabilityName`, CIDR). Writing `analytics` on the target is a rollup; copying another record’s `analytics` into an `associated*SnapshotsAnalytics` array is still an association (form 2 above).
|
|
179
|
-
|
|
180
|
-
### 2b. Enrichment association
|
|
181
|
-
|
|
182
|
-
**Property:** `associated<ObjectType>Enrichment`
|
|
183
|
-
|
|
184
|
-
Copies the linked **snapshot** root `enrichment` object **flat** (same shape as content associations: fields beside optional `_system`, not wrapped in `value`).
|
|
185
|
-
|
|
186
|
-
`enrichment` is an envelope bucket on snapshots (CVSS, EPSS, KEV, compositeScore, …) — **not** a separate `*-enrichment` content-type collection.
|
|
187
|
-
|
|
188
|
-
Example: `associatedVulnerabilitiesGroupsEnrichment` on `vulnerabilities-snapshots`.
|
|
189
|
-
|
|
190
|
-
```json
|
|
191
|
-
{
|
|
192
|
-
"associatedVulnerabilitiesGroupsEnrichment": [
|
|
193
|
-
{
|
|
194
|
-
"cvss": { "baseScore": 9.8 },
|
|
195
|
-
"epss": { "epss": 0.94 },
|
|
196
|
-
"kev": { "isKnownExploited": true },
|
|
197
|
-
"compositeScore": 0.97,
|
|
198
|
-
"riskLevel": "CRITICAL"
|
|
199
|
-
}
|
|
200
|
-
]
|
|
201
|
-
}
|
|
202
|
-
```
|
|
203
|
-
|
|
204
|
-
```json
|
|
205
|
-
{
|
|
206
|
-
"associatedProperty": "associatedVulnerabilitiesGroupsEnrichment",
|
|
207
|
-
"payload": {
|
|
208
|
-
"format": "v2",
|
|
209
|
-
"source": "target.enrichment",
|
|
210
|
-
"includeRecordId": false
|
|
211
|
-
},
|
|
212
|
-
"target": {
|
|
213
|
-
"collection": "vulnerabilities-groups-snapshots",
|
|
214
|
-
"joinPaths": ["data.name"]
|
|
215
|
-
}
|
|
216
|
-
}
|
|
217
|
-
```
|
|
218
|
-
|
|
219
|
-
Rules:
|
|
220
|
-
|
|
221
|
-
- Omit the property when no linked snapshot has a non-empty `enrichment` object.
|
|
222
|
-
- Do not invent a `vulnerabilities-groups-enrichment` collection for this form.
|
|
223
|
-
- Naming helper: `buildAssociatedPropertyName({ form: "enrichment", objectType: "vulnerabilities-groups" })`.
|
|
224
|
-
|
|
225
|
-
### 3. Concepts association
|
|
226
|
-
|
|
227
|
-
**Property:** `associated<ObjectType>Concepts`
|
|
228
|
-
|
|
229
|
-
Copies `concept` from associated snapshot records (not full `data`).
|
|
230
|
-
|
|
231
|
-
```json
|
|
232
|
-
{
|
|
233
|
-
"associatedSecurityGroupsConcepts": [
|
|
234
|
-
{
|
|
235
|
-
"identifier": { "kind": "group-name", "groupName": "Domain Admins" },
|
|
236
|
-
"title": "Domain Admins"
|
|
237
|
-
}
|
|
238
|
-
]
|
|
239
|
-
}
|
|
240
|
-
```
|
|
241
|
-
|
|
242
|
-
```json
|
|
243
|
-
{
|
|
244
|
-
"payload": {
|
|
245
|
-
"format": "v2",
|
|
246
|
-
"source": "target.concept",
|
|
247
|
-
"system": { "include": true }
|
|
248
|
-
}
|
|
249
|
-
}
|
|
250
|
-
```
|
|
251
|
-
|
|
252
|
-
### 4. Record IDs association
|
|
253
|
-
|
|
254
|
-
**Property:** `associated<ObjectType>RecordIds` (use `RecordIds`, not `RecordsIds`)
|
|
255
|
-
|
|
256
|
-
**Value:** array of strings. Must **not** include `_system`.
|
|
257
|
-
|
|
258
|
-
```json
|
|
259
|
-
{
|
|
260
|
-
"associatedSecurityGroupsRecordIds": [
|
|
261
|
-
"security-group-record-id-1",
|
|
262
|
-
"security-group-record-id-2"
|
|
263
|
-
]
|
|
264
|
-
}
|
|
265
|
-
```
|
|
266
|
-
|
|
267
|
-
```json
|
|
268
|
-
{
|
|
269
|
-
"payload": {
|
|
270
|
-
"format": "v2",
|
|
271
|
-
"source": "target.recordId"
|
|
272
|
-
}
|
|
273
|
-
}
|
|
274
|
-
```
|
|
275
|
-
|
|
276
|
-
---
|
|
277
|
-
|
|
278
|
-
## `_system.association` metadata
|
|
279
|
-
|
|
280
|
-
Reserved for technical traceability — not business data.
|
|
281
|
-
|
|
282
|
-
| Field | Meaning |
|
|
283
|
-
|-------|---------|
|
|
284
|
-
| `fromProperty` | Path on the source snapshot that supplied the join value |
|
|
285
|
-
| `toProperty` | Path on the linked snapshot used for matching |
|
|
286
|
-
| `associatedAt` | ISO date-time when the association was created or refreshed |
|
|
287
|
-
| `recordId` | Linked snapshot `recordId` (v2, when enabled) |
|
|
288
|
-
|
|
289
|
-
Included when `payload.format === "v2"` and `payload.system.include === true`. All fields are optional even when `_system` is present.
|
|
290
|
-
|
|
291
|
-
Strip `_system` when presenting associated content to humans, LLMs, or downstream analysis.
|
|
292
|
-
|
|
293
|
-
---
|
|
294
|
-
|
|
295
|
-
## Identifier and identity matching
|
|
296
|
-
|
|
297
|
-
Prefer the linked snapshot's `identity` or `concept.identifier`:
|
|
298
|
-
|
|
299
|
-
```json
|
|
300
|
-
{ "identity": "group-name:Domain Admins" }
|
|
301
|
-
```
|
|
302
|
-
|
|
303
|
-
```json
|
|
304
|
-
{
|
|
305
|
-
"identifier": { "kind": "group-name", "groupName": "Domain Admins" }
|
|
306
|
-
}
|
|
307
|
-
```
|
|
308
|
-
|
|
309
|
-
Canonical location is `concept.identifier`. Root `identifier` / `identity` are accepted when present.
|
|
310
|
-
|
|
311
|
-
Generate identity strings deterministically: join identifier values in order with `:` (e.g. `assetIp:10.100.18.64`).
|
|
12
|
+
## Core rule
|
|
312
13
|
|
|
313
|
-
Association
|
|
14
|
+
Association is always **snapshot-first**:
|
|
314
15
|
|
|
315
|
-
|
|
16
|
+
1. Start from the source object’s snapshot.
|
|
17
|
+
2. Match a target **snapshot** (`*-snapshots`).
|
|
18
|
+
3. Use the matched snapshot’s `recordId` to fetch non-snapshot content (`discovery`, `analysis`, `decisions`, …).
|
|
19
|
+
4. Copy flat content into an `associated…` array on the source snapshot.
|
|
316
20
|
|
|
317
|
-
|
|
318
|
-
{
|
|
319
|
-
"sourceJoinPaths": ["data.assetIp"],
|
|
320
|
-
"match": {
|
|
321
|
-
"collection": "vulnerabilities-snapshots",
|
|
322
|
-
"joinPaths": ["data.assetIp", "concept.identifier.assetIp", "recordId"]
|
|
323
|
-
},
|
|
324
|
-
"content": {
|
|
325
|
-
"collection": "vulnerabilities-discovery",
|
|
326
|
-
"contentType": "discovery",
|
|
327
|
-
"linkPaths": ["concept.identifier.recordId", "data.sourceRecordId", "recordId"]
|
|
328
|
-
}
|
|
329
|
-
}
|
|
330
|
-
```
|
|
21
|
+
Never join a source snapshot directly to a non-snapshot collection by business keys.
|
|
331
22
|
|
|
332
23
|
---
|
|
333
24
|
|
|
334
|
-
##
|
|
335
|
-
|
|
336
|
-
Collections:
|
|
337
|
-
|
|
338
|
-
```text
|
|
339
|
-
users-snapshots
|
|
340
|
-
security-groups-snapshots
|
|
341
|
-
security-groups-analysis
|
|
342
|
-
```
|
|
343
|
-
|
|
344
|
-
User snapshot source:
|
|
345
|
-
|
|
346
|
-
```json
|
|
347
|
-
{
|
|
348
|
-
"data": { "securityGroups": ["Domain Admins", "VPN Users"] }
|
|
349
|
-
}
|
|
350
|
-
```
|
|
351
|
-
|
|
352
|
-
Security group snapshot identifier:
|
|
25
|
+
## Naming
|
|
353
26
|
|
|
354
|
-
|
|
355
|
-
|
|
356
|
-
|
|
357
|
-
|
|
358
|
-
|
|
359
|
-
|
|
360
|
-
}
|
|
361
|
-
}
|
|
362
|
-
```
|
|
27
|
+
| Form | Pattern | Example (abstract) |
|
|
28
|
+
|---|---|---|
|
|
29
|
+
| Legacy short names | `associatedData`, `associatedDiscovery`, … | still present on older records |
|
|
30
|
+
| v2 content | `associated<ObjectType><ContentType>` | `associatedItemGroupsAnalysis` |
|
|
31
|
+
| Ranked top | `associatedTop<ObjectType><ContentType>` | `associatedTopItemsAnalysis` |
|
|
32
|
+
| Enrichment copy | `associated<ObjectType>Enrichment` | `associatedItemGroupsEnrichment` |
|
|
363
33
|
|
|
364
|
-
|
|
365
|
-
|
|
366
|
-
1. Start from `users-snapshots` record.
|
|
367
|
-
2. Read join values from `data.securityGroups`.
|
|
368
|
-
3. Match against `security-groups-snapshots` via `concept.identifier` / identity.
|
|
369
|
-
4. Use matched snapshot `recordId` to fetch `security-groups-analysis` records.
|
|
370
|
-
5. Copy each analysis record's `data` flat into `associatedSecurityGroupsAnalysis`.
|
|
371
|
-
|
|
372
|
-
Result:
|
|
373
|
-
|
|
374
|
-
```json
|
|
375
|
-
{
|
|
376
|
-
"recordId": "user-record-id-1",
|
|
377
|
-
"data": {
|
|
378
|
-
"username": "ami",
|
|
379
|
-
"securityGroups": ["Domain Admins"]
|
|
380
|
-
},
|
|
381
|
-
"associatedSecurityGroupsAnalysis": [
|
|
382
|
-
{
|
|
383
|
-
"_system": {
|
|
384
|
-
"association": {
|
|
385
|
-
"fromProperty": "data.securityGroups",
|
|
386
|
-
"toProperty": "concept.identifier",
|
|
387
|
-
"associatedAt": "2026-07-08T09:00:00.000Z"
|
|
388
|
-
}
|
|
389
|
-
},
|
|
390
|
-
"summary": "The group appears to provide privileged access.",
|
|
391
|
-
"impact": "High",
|
|
392
|
-
"classification": "Privileged Security Group"
|
|
393
|
-
}
|
|
394
|
-
]
|
|
395
|
-
}
|
|
396
|
-
```
|
|
34
|
+
Object-type segments come from Catalox descriptors — not from this format package.
|
|
397
35
|
|
|
398
36
|
---
|
|
399
37
|
|
|
400
|
-
##
|
|
401
|
-
|
|
402
|
-
| Capability | Status |
|
|
403
|
-
|------------|--------|
|
|
404
|
-
| Legacy `associatedData` / `associatedDiscovery` / … presets | **Live** — registry `namingConvention: "legacy"` |
|
|
405
|
-
| v2 property naming | **Target** — new registry entries use `associated<ObjectType><ContentType>` |
|
|
406
|
-
| v2 payload forms (`target.data`, `target.concept`, `target.recordId`, snapshot root) | **Supported** in `@x12i/memorix-associator` request schema |
|
|
407
|
-
| v2 `_system.association` metadata | **Supported** when `format: "v2"` and `system.include: true` |
|
|
408
|
-
| Registry-driven preset generation | **Live** — `generate-presets-from-registry.mjs` |
|
|
409
|
-
| Full enrichment orchestration | **Live** — `run-enrichment-pipeline.mjs` |
|
|
410
|
-
|
|
411
|
-
New associations should be added as registry entries + presets using v2 naming and payload format. Existing legacy properties continue to refresh through their current presets without rename.
|
|
412
|
-
|
|
413
|
-
### Pilot: vulnerability discovery and decisions (v2)
|
|
414
|
-
|
|
415
|
-
Snapshot-mediated flow:
|
|
416
|
-
|
|
417
|
-
1. Read source `vulnerabilities-snapshots.data.assetIp`.
|
|
418
|
-
2. Match target `vulnerabilities-snapshots` by snapshot paths (`data.assetIp`, `concept.identifier.assetIp`, `identity`, `recordId`).
|
|
419
|
-
3. Take matched target snapshot `recordId`.
|
|
420
|
-
4. Fetch `vulnerabilities-discovery` / `vulnerabilities-decisions` by record-id link (`concept.identifier.recordId`, `data.sourceRecordId`, or root `recordId`).
|
|
421
|
-
5. Copy flat `content.data` into `associatedVulnerabilitiesDiscovery` / `associatedVulnerabilitiesDecisions`.
|
|
422
|
-
|
|
423
|
-
| Property | Match collection | Content collection | Join key |
|
|
424
|
-
|----------|------------------|--------------------|----------|
|
|
425
|
-
| `associatedVulnerabilitiesDiscovery` | `vulnerabilities-snapshots` | `vulnerabilities-discovery` | `data.assetIp` → snapshot → `recordId` |
|
|
426
|
-
| `associatedVulnerabilitiesDecisions` | `vulnerabilities-snapshots` | `vulnerabilities-decisions` | `data.assetIp` → snapshot → `recordId` |
|
|
427
|
-
|
|
428
|
-
Target source collection: `vulnerabilities-snapshots` (`MEMORIX_EVENTS_DB`).
|
|
429
|
-
|
|
430
|
-
Refresh rule keys:
|
|
431
|
-
|
|
432
|
-
- `vulnerabilities.vulnerabilities.discovery.by-assetIp`
|
|
433
|
-
- `vulnerabilities.vulnerabilities.decisions.by-assetIp`
|
|
434
|
-
|
|
435
|
-
### Pilot: asset-side vulnerability analysis (v2)
|
|
436
|
-
|
|
437
|
-
Snapshot-mediated flow (source is **assets**, not vulnerabilities):
|
|
438
|
-
|
|
439
|
-
1. Read source `assets-snapshots.data.assetIps` (also `data.assetIp`, `data.ip_address`).
|
|
440
|
-
2. Match `vulnerabilities-snapshots` by `data.assetIp` / `concept.identifier.assetIp`.
|
|
441
|
-
3. Take matched snapshot `recordId`.
|
|
442
|
-
4. Fetch `vulnerabilities-analysis` by record-id link.
|
|
443
|
-
5. Copy flat `content.data` into `associatedVulnerabilitiesAnalysis` on the asset snapshot.
|
|
444
|
-
|
|
445
|
-
| Property | Match collection | Content collection | Join key |
|
|
446
|
-
|----------|------------------|--------------------|----------|
|
|
447
|
-
| `associatedVulnerabilitiesAnalysis` | `vulnerabilities-snapshots` | `vulnerabilities-analysis` | `data.assetIps` → snapshot `data.assetIp` → `recordId` |
|
|
448
|
-
|
|
449
|
-
Target source collection: `assets-snapshots` (`MEMORIX_ENTITIES_DB`).
|
|
450
|
-
|
|
451
|
-
Refresh rule key: `assets.vulnerabilities.analysis.by-assetIp`
|
|
452
|
-
|
|
453
|
-
---
|
|
38
|
+
## Payload rules
|
|
454
39
|
|
|
455
|
-
|
|
40
|
+
- Associated properties are **arrays of plain objects** (linked payloads), never full envelopes.
|
|
41
|
+
- **Omit** the property when empty — never write `[]`.
|
|
42
|
+
- Prefer traceability in optional item-level `_system.association` (v2) rather than wrapping envelopes.
|
|
456
43
|
|
|
457
|
-
|
|
458
|
-
|----------|--------------|
|
|
459
|
-
| `associatedData` | Linked snapshots / generic data |
|
|
460
|
-
| `associatedDiscovery` | Linked discovery |
|
|
461
|
-
| `associatedAnalysis` | Linked analysis |
|
|
462
|
-
| `associatedAssets` | Asset snapshots by IP |
|
|
463
|
-
| `associatedSubnets` | Subnet snapshots (explicit) |
|
|
464
|
-
| `associatedTopology` | Derived (resolver, not associator copy) |
|
|
465
|
-
| `associatedVulnerabilityGroup` | Vuln-group snapshots (explicit) |
|
|
466
|
-
| `associatedZoneAnalysis` | Topology-zone analysis |
|
|
44
|
+
See requirements **ASSOC-*** and the pipeline association doc for matching / payload forms.
|