@x12i/memorix-format 2.2.1 → 2.2.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +4 -0
- package/README.md +67 -30
- package/docs/association-format.md +24 -446
- package/docs/memorix-format.md +53 -103
- package/docs/memorix-workflow-format.md +1 -1
- package/examples/valid-snapshot.json +10 -10
- package/package.json +1 -1
- package/docs/memorix-format-additions.md +0 -956
- package/docs/memorix-format-for-dummies-association-corrections.md +0 -113
package/docs/memorix-format.md
CHANGED
|
@@ -55,33 +55,29 @@ Every piece of business data lives in **exactly one** of these buckets (plus the
|
|
|
55
55
|
|
|
56
56
|
| Bucket | Contents |
|
|
57
57
|
|---|---|
|
|
58
|
-
| **`data`** |
|
|
59
|
-
| **`analytics`** |
|
|
60
|
-
| **`enrichment`** | External
|
|
61
|
-
| **`insights`** |
|
|
58
|
+
| **`data`** | Native facts about the subject itself |
|
|
59
|
+
| **`analytics`** | Numeric rollups computed from other records |
|
|
60
|
+
| **`enrichment`** | External / prior-system values as received |
|
|
61
|
+
| **`insights`** | Judgments, scores, recommendations |
|
|
62
62
|
| **`associatedData`** | Linked snapshot/data content copied from associated records' `data` payloads |
|
|
63
63
|
| **`associatedDiscovery`** | Linked discovery content copied from associated records' `data` payloads |
|
|
64
64
|
| **`associatedAnalysis`** | Linked analysis content copied from associated records' `data` payloads |
|
|
65
|
-
| **`associatedSiblings`** | Self-association: other snapshot `data` payloads sharing the same join key
|
|
66
|
-
| **`associatedChildren`** | Parent-to-child association
|
|
65
|
+
| **`associatedSiblings`** | Self-association: other snapshot `data` payloads sharing the same join key; each item includes linked `recordId`. Default cap: 99 items. |
|
|
66
|
+
| **`associatedChildren`** | Parent-to-child association: child `data` payloads whose join key matches the parent; each item includes linked `recordId`. Default cap: 99 items. |
|
|
67
67
|
|
|
68
68
|
Do not duplicate a value across buckets. When reshaping legacy records, promote `sourceFacts` values into the correct bucket rather than wrapping them.
|
|
69
69
|
|
|
70
|
-
Associated properties are always arrays and are omitted when no linked data was resolved. Do not write empty arrays, and do not switch between object and array shape. Custom associated content uses the same root-level pattern: `associatedCompliance`, `associatedBusinessContext`,
|
|
70
|
+
Associated properties are always arrays and are omitted when no linked data was resolved. Do not write empty arrays, and do not switch between object and array shape. Custom associated content uses the same root-level pattern: `associatedCompliance`, `associatedBusinessContext`, and so on.
|
|
71
71
|
|
|
72
|
-
**Ranked top associations** use `associatedTop<ObjectType><ContentType>` (e.g. `
|
|
72
|
+
**Ranked top associations** use `associatedTop<ObjectType><ContentType>` (e.g. `associatedTopItemsAnalysis`). Candidates are sorted by the linked object type’s `importanceIndicator` (or rule `rankBy`) and capped (default 5). See [association-format.md](./association-format.md).
|
|
73
73
|
|
|
74
|
-
**Enrichment associations** use `associated<ObjectType>Enrichment` (e.g. `
|
|
74
|
+
**Enrichment associations** use `associated<ObjectType>Enrichment` (e.g. `associatedItemGroupsEnrichment`). They copy the linked snapshot’s root `enrichment` bucket as a flat object array — not a separate `*-enrichment` content collection. See [association-format.md](./association-format.md).
|
|
75
75
|
|
|
76
76
|
Associated items contain the linked record's `data` payload (or `enrichment` / `concept` / root property for the specialized forms). Legacy associations may also include inline `recordId` and `contentType`; v2 associations store traceability in optional item-level `_system.association` metadata. Do not wrap associated items with redundant envelope keys; failure and diagnostic details belong in `<objectType>-failed` collections instead.
|
|
77
77
|
|
|
78
78
|
**Associated property metadata:** Consumers should prefer Memorix associated-property metadata when available (`@x12i/memorix-associator`). Associated snapshot properties are exposed through managed descriptors plus discovery. Until metadata is available, or when reading older/custom records, fall back to the prefix rule: root-level fields on snapshots whose names start with `associated`. The well-known names `associatedData`, `associatedDiscovery`, and `associatedAnalysis` are conventions only — the complete set comes from metadata plus the `associated*` fallback rule.
|
|
79
79
|
|
|
80
|
-
When a field can naturally have more than one value, processing contracts use array shape even if the current record has only one value.
|
|
81
|
-
|
|
82
|
-
For subnets, `data.cidr` is the canonical join key. Subnet records also carry structured `data.subnet` plus flattened `data.networkAddress`, `data.prefixLength`, and `data.mask`, all derived deterministically from `data.cidr`.
|
|
83
|
-
|
|
84
|
-
Assets also carry `data.assetIpClassifications[]` when topology resolution runs. Each item records `{ ip, scope, confidence, reason, reasoning, matchedSubnetCidr?, matchedSubnetIp? }`, where `scope` is `inside`, `outside`, or `unknown`. Convenience arrays `data.insideAssetIps`, `data.outsideAssetIps`, and `data.unknownAssetIps` may be populated from those classifications. `inside` means the IP either matched canonical topology with high confidence, or is RFC1918 private space without a matching topology CIDR with medium confidence. `outside` means public/non-RFC1918 with no topology match. `unknown` covers invalid, loopback, and link-local addresses.
|
|
80
|
+
When a field can naturally have more than one value, processing contracts use array shape even if the current record has only one value. Join-key recipes (which paths and shapes) are defined on **entity descriptors** for each Catalox object type — not in this format package.
|
|
85
81
|
|
|
86
82
|
---
|
|
87
83
|
|
|
@@ -89,49 +85,45 @@ Assets also carry `data.assetIpClassifications[]` when topology resolution runs.
|
|
|
89
85
|
|
|
90
86
|
```ts
|
|
91
87
|
type MemorixConcept = {
|
|
92
|
-
|
|
88
|
+
_extIdentifier: {
|
|
93
89
|
kind: string;
|
|
94
90
|
/** Primary value — property name matches `kind`. */
|
|
95
91
|
[kind: string]: string;
|
|
96
92
|
/**
|
|
97
93
|
* Deterministic join key: `kind` first, then the kind-named value,
|
|
98
94
|
* then any other scalar fields in sorted key order, joined by `:`.
|
|
99
|
-
* See `.operational/fr/2026-07-09-memorix-format-identifier-key.md`.
|
|
100
95
|
*/
|
|
101
|
-
|
|
96
|
+
key: string;
|
|
102
97
|
};
|
|
103
98
|
title: string;
|
|
104
99
|
};
|
|
105
100
|
```
|
|
106
101
|
|
|
107
|
-
`
|
|
102
|
+
`_extIdentifier` is a discriminated object: `kind` names the id type, and a same-named property holds the value. `key` is required. How `kind` / title are derived for an object type is defined on the **entity descriptor**, not here.
|
|
108
103
|
|
|
109
104
|
```json
|
|
110
105
|
"concept": {
|
|
111
|
-
"
|
|
112
|
-
"kind": "
|
|
113
|
-
"
|
|
114
|
-
"
|
|
106
|
+
"_extIdentifier": {
|
|
107
|
+
"kind": "sku",
|
|
108
|
+
"sku": "SKU-100",
|
|
109
|
+
"key": "sku:SKU-100"
|
|
115
110
|
},
|
|
116
|
-
"title": "
|
|
111
|
+
"title": "Blue widget"
|
|
117
112
|
}
|
|
118
113
|
```
|
|
119
114
|
|
|
120
|
-
`concept.name` is not used. `title` is the human-readable display name; it may differ from the identifier value
|
|
115
|
+
`concept.name` is not used. `concept.identifier` / `identifierKey` are legacy — migrate to `_extIdentifier` / `key`. `title` is the human-readable display name; it may differ from the identifier value.
|
|
121
116
|
|
|
122
|
-
### Identifier `kind`
|
|
117
|
+
### Identifier `kind` (illustrative — not format law)
|
|
123
118
|
|
|
124
|
-
| Object type | `kind` | Example
|
|
119
|
+
| Object type (abstract) | `kind` | Example |
|
|
125
120
|
|---|---|---|
|
|
126
|
-
|
|
|
127
|
-
|
|
|
128
|
-
|
|
|
129
|
-
|
|
|
130
|
-
| Topology zones | `zoneName` | `{"kind":"zoneName","zoneName":"LAN","identifierKey":"zoneName:LAN"}` |
|
|
131
|
-
| Topology VRs | `vrName` | `{"kind":"vrName","vrName":"VR-UPB-MED","identifierKey":"vrName:VR-UPB-MED"}` |
|
|
132
|
-
| Users | `sessionKey` | `{"kind":"sessionKey","sessionKey":"med\\cap\|10.39.44.205\|UIA","identifierKey":"sessionKey:med\\cap\|10.39.44.205\|UIA"}` |
|
|
121
|
+
| Items | `sku` | `{"kind":"sku","sku":"SKU-100","key":"sku:SKU-100"}` |
|
|
122
|
+
| Groups | `groupCode` | `{"kind":"groupCode","groupCode":"G-42","key":"groupCode:G-42"}` |
|
|
123
|
+
| Locations | `locationCode` | `{"kind":"locationCode","locationCode":"WH-A","key":"locationCode:WH-A"}` |
|
|
124
|
+
| Documents | `docId` | `{"kind":"docId","docId":"DOC-9","key":"docId:DOC-9"}` |
|
|
133
125
|
|
|
134
|
-
|
|
126
|
+
Domain packs (Catalox) may use different `kind` values; those recipes live on descriptors.
|
|
135
127
|
|
|
136
128
|
---
|
|
137
129
|
|
|
@@ -143,6 +135,7 @@ System metadata — provenance, processing history, graph-run stamps — never b
|
|
|
143
135
|
type MemorixLifecycleEvent = { process: string; processedAt: string };
|
|
144
136
|
|
|
145
137
|
type MemorixSystemBlock = {
|
|
138
|
+
state?: "active" | "archived" | "superseded";
|
|
146
139
|
lifecycle?: MemorixLifecycleEvent[];
|
|
147
140
|
_graphRuns?: Record<string, GraphRunEntry>;
|
|
148
141
|
_jobTypeRuns?: Record<string, GraphRunEntry>;
|
|
@@ -164,8 +157,8 @@ Root-level `lifecycle` is gone. Processing events live at `_system.lifecycle`:
|
|
|
164
157
|
- Each entry is `{ process, processedAt }`.
|
|
165
158
|
- When `createdAt`, `modifiedAt`, and `capturedAt` are identical (the common snapshot case), emit a single `"enriched"` event rather than three duplicate timestamps.
|
|
166
159
|
- Distinct events (e.g. a future re-enrichment) append as additional array entries.
|
|
167
|
-
- `
|
|
168
|
-
-
|
|
160
|
+
- Omit `lifecycle` entirely when no timestamp is known — do not emit an empty array.
|
|
161
|
+
- `_system.state` is the present-tense lifecycle state enum (`active` \| `archived` \| `superseded`).
|
|
169
162
|
|
|
170
163
|
`_system.provenance.enrichedAt` (when present) is separate provenance metadata about a prior enrichment step, not a substitute for `lifecycle`.
|
|
171
164
|
|
|
@@ -179,102 +172,59 @@ Dot-path references: `_system._graphRuns.<graphId>`, `_system._jobTypeRuns.<jobT
|
|
|
179
172
|
|
|
180
173
|
## Complete example
|
|
181
174
|
|
|
182
|
-
|
|
175
|
+
Abstract item snapshot showing envelope fields and open buckets:
|
|
183
176
|
|
|
184
177
|
```json
|
|
185
178
|
{
|
|
186
|
-
"
|
|
187
|
-
"recordId": "69d216ed358778e93215787e",
|
|
179
|
+
"recordId": "rec-item-1",
|
|
188
180
|
"concept": {
|
|
189
|
-
"
|
|
190
|
-
"kind": "
|
|
191
|
-
"
|
|
192
|
-
"
|
|
181
|
+
"_extIdentifier": {
|
|
182
|
+
"kind": "sku",
|
|
183
|
+
"sku": "SKU-100",
|
|
184
|
+
"key": "sku:SKU-100"
|
|
193
185
|
},
|
|
194
|
-
"title": "
|
|
186
|
+
"title": "Blue widget"
|
|
195
187
|
},
|
|
196
188
|
"data": {
|
|
197
|
-
"
|
|
198
|
-
"
|
|
199
|
-
"
|
|
200
|
-
"cveId": "CVE-2020-14882",
|
|
201
|
-
"severity": "critical",
|
|
202
|
-
"pluginId": 142594
|
|
189
|
+
"sku": "SKU-100",
|
|
190
|
+
"color": "blue",
|
|
191
|
+
"family": "widgets"
|
|
203
192
|
},
|
|
204
193
|
"associatedData": [
|
|
205
|
-
{ "
|
|
206
|
-
{ "
|
|
194
|
+
{ "locationCode": "WH-A", "qty": 12 },
|
|
195
|
+
{ "locationCode": "WH-B", "qty": 3 }
|
|
207
196
|
],
|
|
208
197
|
"analytics": {
|
|
209
|
-
"
|
|
198
|
+
"linkedLocations": { "total": 2 }
|
|
210
199
|
},
|
|
211
200
|
"enrichment": {
|
|
212
|
-
"
|
|
213
|
-
|
|
214
|
-
"cvssVector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H"
|
|
215
|
-
},
|
|
216
|
-
"epss": { "epss": 0.94454, "percentile": 0.99992 },
|
|
217
|
-
"kev": { "isKnownExploited": true },
|
|
218
|
-
"compositeScore": 0.974,
|
|
219
|
-
"riskLevel": "CRITICAL",
|
|
220
|
-
"vprScore": "9.2"
|
|
201
|
+
"catalogScore": 0.91,
|
|
202
|
+
"sourceSystem": "erp-feed"
|
|
221
203
|
},
|
|
222
204
|
"insights": {
|
|
223
|
-
"
|
|
224
|
-
"potentialImpact": { "level": "CRITICAL", "confidence": 0.88 }
|
|
205
|
+
"priority": { "level": "high", "confidence": 0.9 }
|
|
225
206
|
},
|
|
226
207
|
"_system": {
|
|
208
|
+
"state": "active",
|
|
227
209
|
"provenance": {
|
|
228
|
-
"sourceId": "6914a132880706529ece3da1",
|
|
229
|
-
"snapshotRecordId": "69d216ed358778e93215787e",
|
|
230
210
|
"source": {
|
|
231
|
-
"database": "
|
|
232
|
-
"collection": "
|
|
233
|
-
"server": "default"
|
|
211
|
+
"database": "example-db",
|
|
212
|
+
"collection": "items-rawdata"
|
|
234
213
|
}
|
|
235
214
|
},
|
|
236
215
|
"lifecycle": [
|
|
237
216
|
{ "process": "enriched", "processedAt": "2026-04-05T08:01:15.511Z" }
|
|
238
|
-
]
|
|
239
|
-
"_graphRuns": {
|
|
240
|
-
"graph-t92669z": {
|
|
241
|
-
"status": "done",
|
|
242
|
-
"startedAt": "2026-07-01T23:41:23.233Z",
|
|
243
|
-
"completedAt": "2026-07-01T23:42:05.604Z",
|
|
244
|
-
"lastJobRunId": "f7af7f7a-0c5c-4825-9282-d39bea836682",
|
|
245
|
-
"attempts": 1,
|
|
246
|
-
"result": {
|
|
247
|
-
"target": "entity",
|
|
248
|
-
"entityName": "vulnerabilities-groups",
|
|
249
|
-
"contentType": "inferences",
|
|
250
|
-
"recordId": "75de9244-8d44-43b8-8698-7e22621b1087"
|
|
251
|
-
}
|
|
252
|
-
}
|
|
253
|
-
}
|
|
217
|
+
]
|
|
254
218
|
}
|
|
255
219
|
}
|
|
256
220
|
```
|
|
257
221
|
|
|
258
|
-
Not every record populates every bucket.
|
|
222
|
+
Not every record populates every bucket. Non-snapshot content types usually keep substance in `.data` only (SCOPE).
|
|
259
223
|
|
|
260
224
|
---
|
|
261
225
|
|
|
262
226
|
## Migration notes
|
|
263
227
|
|
|
264
|
-
-
|
|
265
|
-
-
|
|
266
|
-
-
|
|
267
|
-
|
|
268
|
-
### Subnets / topology subnets merge (2026-07-05)
|
|
269
|
-
|
|
270
|
-
`topology-subnets` was merged into `subnets`. Canonical `subnets` records are topology
|
|
271
|
-
CIDR segments (`concept.identifier.kind: "cidr"`), spread across content types
|
|
272
|
-
`snapshots` / `core` / `inferences` / `scoped`. A subset of the old, pre-merge `subnets`
|
|
273
|
-
records (subnetIp-only, no CIDR) had zero overlap with any canonical CIDR and were
|
|
274
|
-
quarantined rather than dropped — they live in the `legacy` content type
|
|
275
|
-
(`subnets-legacy`), keyed by `concept.identifier.kind: "subnetIp"`, with an explicit
|
|
276
|
-
`_system.quarantine` block (`legacy: true`, `status`, `reason`) marking them as not
|
|
277
|
-
eligible for `assets -> subnets` joins. `assets.data.subnetCidr` (precomputed via CIDR
|
|
278
|
-
containment against canonical `subnets-snapshots`) is the join key used for that
|
|
279
|
-
relation — see `scripts/memorix-subnets-merge/` for the migration and backfill scripts,
|
|
280
|
-
and `temp/part-b-implementation-design-2026-07-05T06-20-00Z.md` for the full design.
|
|
228
|
+
- Prefer **`npx memorix-format migrate`** for transforms (dry-run → `--out` → validate). Live Mongo apply is a separate ops step.
|
|
229
|
+
- Historical reshape scripts under `scripts/` are not the recommended path.
|
|
230
|
+
- Thin envelope helpers may still be re-exported from `@x12i/memorix-descriptors`; **format packages are the SOT**.
|
|
@@ -73,7 +73,7 @@ type WorkflowTask = {
|
|
|
73
73
|
|
|
74
74
|
Catalog id: `memorix-workflow-field-schemas`
|
|
75
75
|
|
|
76
|
-
One item per object type (natural id e.g. `
|
|
76
|
+
One item per object type (natural id e.g. `items-workflow-fields`):
|
|
77
77
|
|
|
78
78
|
```ts
|
|
79
79
|
type WorkflowFieldSchemaItem = {
|
|
@@ -1,25 +1,25 @@
|
|
|
1
1
|
{
|
|
2
|
-
"recordId": "
|
|
2
|
+
"recordId": "rec-item-1",
|
|
3
3
|
"concept": {
|
|
4
4
|
"_extIdentifier": {
|
|
5
|
-
"kind": "
|
|
6
|
-
"
|
|
7
|
-
"key": "
|
|
5
|
+
"kind": "sku",
|
|
6
|
+
"sku": "SKU-100",
|
|
7
|
+
"key": "sku:SKU-100"
|
|
8
8
|
},
|
|
9
|
-
"title": "
|
|
9
|
+
"title": "Blue widget"
|
|
10
10
|
},
|
|
11
11
|
"data": {
|
|
12
|
-
"
|
|
13
|
-
"
|
|
12
|
+
"sku": "SKU-100",
|
|
13
|
+
"color": "blue"
|
|
14
14
|
},
|
|
15
15
|
"associatedData": [
|
|
16
|
-
{ "
|
|
16
|
+
{ "locationCode": "WH-A", "qty": 12 }
|
|
17
17
|
],
|
|
18
18
|
"analytics": {
|
|
19
|
-
"
|
|
19
|
+
"linkedLocations": { "total": 1 }
|
|
20
20
|
},
|
|
21
21
|
"insights": {
|
|
22
|
-
"
|
|
22
|
+
"priority": { "level": "high", "confidence": 0.9 }
|
|
23
23
|
},
|
|
24
24
|
"_system": {
|
|
25
25
|
"state": "active",
|
package/package.json
CHANGED