@x12i/memorix-writer 1.8.1 → 1.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +32 -8
- package/dist/writer/build-record.d.ts +8 -1
- package/dist/writer/build-record.d.ts.map +1 -1
- package/dist/writer/build-record.js +8 -1
- package/dist/writer/build-record.js.map +1 -1
- package/docs/GRAPH-RUNS.md +10 -4
- package/docs/JOB-TYPE-RUNS.md +3 -1
- package/docs/MEMORIX-DATABASE-CONVENTIONS.md +100 -46
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -75,15 +75,39 @@ Override defaults only when you need to:
|
|
|
75
75
|
|
|
76
76
|
## Collection resolution
|
|
77
77
|
|
|
78
|
-
Writes target **`memorix-entities`**, **`memorix-events`**, or **`memorix-knowledge`** based on the write descriptor (`target.kind`, defaulting from `identity.idField`). Database names follow [Memorix Database Conventions](./docs/MEMORIX-DATABASE-CONVENTIONS.md) (`MEMORIX_ENTITIES_DB`, `MEMORIX_EVENTS_DB`, `MEMORIX_KNOWLEDGE_DB`, …).
|
|
78
|
+
Writes target **`memorix-entities`**, **`memorix-events`**, or **`memorix-knowledge`** based on the write descriptor (`target.kind`, defaulting from the descriptor's `identity.idField` config). Database names follow [Memorix Database Conventions](./docs/MEMORIX-DATABASE-CONVENTIONS.md) (`MEMORIX_ENTITIES_DB`, `MEMORIX_EVENTS_DB`, `MEMORIX_KNOWLEDGE_DB`, …).
|
|
79
79
|
|
|
80
|
-
|
|
80
|
+
## Identity model
|
|
81
81
|
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
82
|
+
Every record this package writes uses the `memorix-record.2.0` envelope — a nested `identity` object, not flat top-level id fields:
|
|
83
|
+
|
|
84
|
+
```json
|
|
85
|
+
{
|
|
86
|
+
"identity": {
|
|
87
|
+
"recordId": "rec_...",
|
|
88
|
+
"target": "entity",
|
|
89
|
+
"targetId": "10.150.68.31",
|
|
90
|
+
"objectType": "assets",
|
|
91
|
+
"contentType": "analysis",
|
|
92
|
+
"version": "memorix-record.2.0"
|
|
93
|
+
},
|
|
94
|
+
"lifecycle": { "createdAt": "...", "modifiedAt": "...", "status": "active" },
|
|
95
|
+
"concept": { "kind": "assets", "name": "10.150.68.31" },
|
|
96
|
+
"data": { "...": "..." }
|
|
97
|
+
}
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
`entityId` / `eventId` / `knowledgeId` are **request-time parameter names** on `writeMemorixRecord` — pass exactly one to say which target and which id value you're writing:
|
|
101
|
+
|
|
102
|
+
| Target | Request field | Written to |
|
|
103
|
+
|--------|----------------|------------|
|
|
104
|
+
| `entity` | `entityId` | `identity.target: "entity"`, `identity.targetId` |
|
|
105
|
+
| `event` | `eventId` | `identity.target: "event"`, `identity.targetId` |
|
|
106
|
+
| `knowledge` | `knowledgeId` | `identity.target: "knowledge"`, `identity.targetId` |
|
|
107
|
+
|
|
108
|
+
They are never stored as flat root fields, and there is no client-facing `idField` concept — clients read/write identity via the `identity` object (and, for narrative/search context, the `concept` object). This package will not add a flat-id or `idField` public API even on request, since it would break the `memorix-record.2.0` contract for every downstream consumer. See [MEMORIX-DATABASE-CONVENTIONS.md](./docs/MEMORIX-DATABASE-CONVENTIONS.md#document-shape) for the full shape and rationale.
|
|
109
|
+
|
|
110
|
+
Internally, `@x12i/memorix-writer` tolerates **reading** older, pre-2.0 records that only have a flat `entityId`/`eventId` (see [docs/GRAPH-RUNS.md § Identity resolution](./docs/GRAPH-RUNS.md#identity-resolution-source-records)) so graph-run tracking keeps working during a tenant's migration window. That fallback is read-only, internal, and not exported — it does not change what gets written, and it is not something applications should rely on or target.
|
|
87
111
|
|
|
88
112
|
Collection name resolution order:
|
|
89
113
|
|
|
@@ -150,7 +174,7 @@ Shipped write descriptors include `asset-analysis-write` (entity) and corpus hel
|
|
|
150
174
|
|----|--------|
|
|
151
175
|
| AC-1 | Generic descriptor-driven write |
|
|
152
176
|
| AC-2 | Prefix/postfix collection resolution |
|
|
153
|
-
| AC-3 | Identity: exactly one of `entityId` / `eventId` |
|
|
177
|
+
| AC-3 | Identity: exactly one of `entityId` / `eventId` / `knowledgeId`, written to `identity.targetId` |
|
|
154
178
|
| AC-4 | Default operation `add` |
|
|
155
179
|
| AC-5 | Upsert only when allowed |
|
|
156
180
|
| AC-6 | Patch only when allowed |
|
|
@@ -2,7 +2,14 @@ import type { MemorixEntityDescriptor } from "../descriptors/entity-descriptor-t
|
|
|
2
2
|
import type { MemorixWriteDescriptor } from "../descriptors/write-descriptor-types.js";
|
|
3
3
|
import type { ResolvedIdentity } from "../data/identity.js";
|
|
4
4
|
import type { MemorixWriterDefaults } from "../client/types.js";
|
|
5
|
-
|
|
5
|
+
/**
|
|
6
|
+
* Canonical Memorix record envelope version. Must match the shared spec in
|
|
7
|
+
* `docs/memorix-format.md` and `@x12i/memorix-retrieval`'s
|
|
8
|
+
* `MEMORIX-DATABASE-CONVENTIONS.md`. Every record this package writes
|
|
9
|
+
* (domain writes and graph-run results) is stamped with this value —
|
|
10
|
+
* never a legacy/interim tag.
|
|
11
|
+
*/
|
|
12
|
+
export declare const MEMORIX_RECORD_VERSION = "memorix-record.2.0";
|
|
6
13
|
export type MemorixWrittenRecordBase = {
|
|
7
14
|
identity: {
|
|
8
15
|
recordId: string;
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"build-record.d.ts","sourceRoot":"","sources":["../../src/writer/build-record.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,uBAAuB,EAAE,MAAM,2CAA2C,CAAC;AACzF,OAAO,KAAK,EAAE,sBAAsB,EAAE,MAAM,0CAA0C,CAAC;AACvF,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,qBAAqB,CAAC;AAC5D,OAAO,KAAK,EAAE,qBAAqB,EAAE,MAAM,oBAAoB,CAAC;AAGhE,eAAO,MAAM,sBAAsB,
|
|
1
|
+
{"version":3,"file":"build-record.d.ts","sourceRoot":"","sources":["../../src/writer/build-record.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,uBAAuB,EAAE,MAAM,2CAA2C,CAAC;AACzF,OAAO,KAAK,EAAE,sBAAsB,EAAE,MAAM,0CAA0C,CAAC;AACvF,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,qBAAqB,CAAC;AAC5D,OAAO,KAAK,EAAE,qBAAqB,EAAE,MAAM,oBAAoB,CAAC;AAGhE;;;;;;GAMG;AACH,eAAO,MAAM,sBAAsB,uBAAuB,CAAC;AAE3D,MAAM,MAAM,wBAAwB,GAAG;IACrC,QAAQ,EAAE;QACR,QAAQ,EAAE,MAAM,CAAC;QACjB,MAAM,EAAE,QAAQ,GAAG,OAAO,GAAG,WAAW,CAAC;QACzC,QAAQ,EAAE,MAAM,CAAC;QACjB,UAAU,EAAE,MAAM,CAAC;QACnB,WAAW,EAAE,MAAM,CAAC;QACpB,OAAO,EAAE,MAAM,CAAC;KACjB,CAAC;IACF,SAAS,EAAE;QACT,SAAS,EAAE,MAAM,CAAC;QAClB,UAAU,EAAE,MAAM,CAAC;QACnB,MAAM,EAAE,QAAQ,CAAC;KAClB,CAAC;IACF,OAAO,EAAE;QACP,IAAI,EAAE,MAAM,CAAC;QACb,IAAI,EAAE,MAAM,CAAC;KACd,CAAC;IACF,IAAI,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;IAC9B,OAAO,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;CACnC,CAAC;AAUF,wBAAgB,mBAAmB,CAAC,MAAM,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,GAAG,MAAM,CAM3E;AAED,wBAAgB,oBAAoB,CAAC,KAAK,EAAE,MAAM,GAAG,MAAM,CAM1D;AAED,wBAAgB,mBAAmB,CAAC,QAAQ,EAAE,gBAAgB,GAAG,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAEvF;AAED,wBAAgB,eAAe,CAAC,IAAI,EAAE;IACpC,QAAQ,EAAE,gBAAgB,CAAC;IAC3B,eAAe,EAAE,sBAAsB,CAAC;IACxC,MAAM,EAAE,uBAAuB,CAAC;IAChC,QAAQ,EAAE,qBAAqB,CAAC;IAChC,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,GAAG,EAAE,MAAM,CAAC;IACZ,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,aAAa,CAAC,EAAE,MAAM,CAAC;IACvB,MAAM,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;IACjC,KAAK,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;IAChC,SAAS,EAAE,MAAM,CAAC;CACnB,GAAG,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAgD1B;AAED,wBAAgB,cAAc,CAC5B,MAAM,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,EAC/B,IAAI,EAAE,MAAM,EACZ,IAAI,SAAgC,GACnC,IAAI,CAEN"}
|
|
@@ -1,6 +1,13 @@
|
|
|
1
1
|
import { randomUUID } from "node:crypto";
|
|
2
2
|
import { deepSet } from "../utils/path.js";
|
|
3
|
-
|
|
3
|
+
/**
|
|
4
|
+
* Canonical Memorix record envelope version. Must match the shared spec in
|
|
5
|
+
* `docs/memorix-format.md` and `@x12i/memorix-retrieval`'s
|
|
6
|
+
* `MEMORIX-DATABASE-CONVENTIONS.md`. Every record this package writes
|
|
7
|
+
* (domain writes and graph-run results) is stamped with this value —
|
|
8
|
+
* never a legacy/interim tag.
|
|
9
|
+
*/
|
|
10
|
+
export const MEMORIX_RECORD_VERSION = "memorix-record.2.0";
|
|
4
11
|
function targetFromIdField(idField) {
|
|
5
12
|
if (idField === "eventId")
|
|
6
13
|
return "event";
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"build-record.js","sourceRoot":"","sources":["../../src/writer/build-record.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AAKzC,OAAO,EAAE,OAAO,EAAE,MAAM,kBAAkB,CAAC;AAE3C,MAAM,CAAC,MAAM,sBAAsB,GAAG,
|
|
1
|
+
{"version":3,"file":"build-record.js","sourceRoot":"","sources":["../../src/writer/build-record.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AAKzC,OAAO,EAAE,OAAO,EAAE,MAAM,kBAAkB,CAAC;AAE3C;;;;;;GAMG;AACH,MAAM,CAAC,MAAM,sBAAsB,GAAG,oBAAoB,CAAC;AAwB3D,SAAS,iBAAiB,CACxB,OAAoC;IAEpC,IAAI,OAAO,KAAK,SAAS;QAAE,OAAO,OAAO,CAAC;IAC1C,IAAI,OAAO,KAAK,aAAa;QAAE,OAAO,WAAW,CAAC;IAClD,OAAO,QAAQ,CAAC;AAClB,CAAC;AAED,MAAM,UAAU,mBAAmB,CAAC,MAA+B;IACjE,MAAM,QAAQ,GAAG,MAAM,CAAC,QAA+C,CAAC;IACxE,IAAI,OAAO,QAAQ,EAAE,QAAQ,KAAK,QAAQ,IAAI,QAAQ,CAAC,QAAQ,CAAC,IAAI,EAAE,EAAE,CAAC;QACvE,OAAO,QAAQ,CAAC,QAAQ,CAAC,IAAI,EAAE,CAAC;IAClC,CAAC;IACD,OAAO,EAAE,CAAC;AACZ,CAAC;AAED,MAAM,UAAU,oBAAoB,CAAC,KAAa;IAChD,IAAI,KAAK,KAAK,UAAU;QAAE,OAAO,mBAAmB,CAAC;IACrD,IAAI,KAAK,KAAK,UAAU,IAAI,KAAK,KAAK,SAAS,IAAI,KAAK,KAAK,aAAa,EAAE,CAAC;QAC3E,OAAO,mBAAmB,CAAC;IAC7B,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED,MAAM,UAAU,mBAAmB,CAAC,QAA0B;IAC5D,OAAO,EAAE,mBAAmB,EAAE,QAAQ,CAAC,OAAO,EAAE,CAAC;AACnD,CAAC;AAED,MAAM,UAAU,eAAe,CAAC,IAY/B;IACC,MAAM,QAAQ,GAAG,IAAI,CAAC,QAAQ,IAAI,OAAO,UAAU,EAAE,CAAC,OAAO,CAAC,IAAI,EAAE,EAAE,CAAC,EAAE,CAAC;IAC1E,MAAM,MAAM,GAAG,iBAAiB,CAAC,IAAI,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC;IACxD,MAAM,QAAQ,GAAG,IAAI,CAAC,QAAQ,CAAC,mBAAmB,CAAC;IAEnD,MAAM,MAAM,GAA4B;QACtC,QAAQ,EAAE;YACR,QAAQ;YACR,MAAM;YACN,QAAQ,EAAE,IAAI,CAAC,QAAQ,CAAC,OAAO;YAC/B,UAAU,EAAE,IAAI,CAAC,MAAM,CAAC,UAAU;YAClC,WAAW,EAAE,IAAI,CAAC,eAAe,CAAC,MAAM,CAAC,WAAW;YACpD,OAAO,EAAE,sBAAsB;SAChC;QACD,SAAS,EAAE;YACT,SAAS,EAAE,IAAI,CAAC,GAAG;YACnB,UAAU,EAAE,IAAI,CAAC,GAAG;YACpB,MAAM,EAAE,QAAQ;SACjB;QACD,OAAO,EAAE;YACP,IAAI,EAAE,IAAI,CAAC,MAAM,CAAC,UAAU;YAC5B,IAAI,EAAE,IAAI,CAAC,QAAQ,CAAC,OAAO;SAC5B;QACD,IAAI,EAAE,EAAE;KACT,CAAC;IAEF,MAAM,IAAI,GAA4B;QACpC,MAAM,EAAE,IAAI,CAAC,MAAM,CAAC,UAAU;QAC9B,WAAW,EAAE,IAAI,CAAC,eAAe,CAAC,MAAM,CAAC,WAAW;QACpD,iBAAiB,EAAE,IAAI,CAAC,eAAe,CAAC,EAAE;QAC1C,SAAS,EAAE,IAAI,CAAC,SAAS;KAC1B,CAAC;IACF,IAAI,IAAI,CAAC,cAAc;QAAE,IAAI,CAAC,cAAc,GAAG,IAAI,CAAC,cAAc,CAAC;IACnE,IAAI,IAAI,CAAC,aAAa;QAAE,IAAI,CAAC,aAAa,GAAG,IAAI,CAAC,aAAa,CAAC;IAChE,IAAI,IAAI,CAAC,MAAM;QAAE,IAAI,CAAC,MAAM,GAAG,IAAI,CAAC,MAAM,CAAC;IAC3C,IAAI,IAAI,CAAC,KAAK;QAAE,IAAI,CAAC,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC;IAExC,IAAI,QAAQ,KAAK,iBAAiB,IAAI,QAAQ,CAAC,UAAU,CAAC,UAAU,CAAC,EAAE,CAAC;QACtE,OAAO,CAAC,MAAM,EAAE,QAAQ,EAAE,IAAI,CAAC,CAAC;IAClC,CAAC;SAAM,IAAI,QAAQ,KAAK,UAAU,EAAE,CAAC;QACnC,MAAM,MAAM,GAAI,MAAM,CAAC,OAA+C,IAAI,EAAE,CAAC;QAC7E,MAAM,CAAC,OAAO,GAAG,IAAI,CAAC;QACtB,MAAM,CAAC,OAAO,GAAG,MAAM,CAAC;IAC1B,CAAC;SAAM,CAAC;QACN,MAAM,CAAC,QAAQ,CAAC,GAAG,IAAI,CAAC;IAC1B,CAAC;IAED,OAAO,MAAM,CAAC;AAChB,CAAC;AAED,MAAM,UAAU,cAAc,CAC5B,MAA+B,EAC/B,IAAY,EACZ,IAAI,GAAG,6BAA6B;IAEpC,OAAO,CAAC,MAAM,EAAE,IAAI,EAAE,IAAI,CAAC,CAAC;AAC9B,CAAC"}
|
package/docs/GRAPH-RUNS.md
CHANGED
|
@@ -54,10 +54,16 @@ Sets terminal `failed` status with `error` and `completedAt`. Call only when `jo
|
|
|
54
54
|
|
|
55
55
|
### `writeGraphRunResult(ref, graphId, opts)` — combined success path
|
|
56
56
|
|
|
57
|
-
1. Writes the graph result document to the persistency target collection (`<entityName>-<contentType>`).
|
|
58
|
-
- **`recordIdSource: { generate: true }` (MRX-FRS-002):** always **inserts** a new record with a new `recordId`. Same-entity runs keep the source identity
|
|
57
|
+
1. Writes the graph result document to the persistency target collection (`<entityName>-<contentType>`) as a full `memorix-record.2.0` envelope (`identity` / `lifecycle` / `concept` / `data`) — never a legacy flat shape.
|
|
58
|
+
- **`recordIdSource: { generate: true }` (MRX-FRS-002):** always **inserts** a new record with a new `identity.recordId`. Same-entity runs keep the source's `identity.targetId`; cross-entity runs allocate a new `identity.targetId` as well. Optional `link` applies an FK on the result document and verifies the relation descriptor on the result entity.
|
|
59
59
|
- **`recordIdSource: "same-as-input"` (deprecated):** upserts/replaces by source id (legacy 1.2.0 behavior).
|
|
60
|
-
2. Stamps the source record `_graphRuns.<graphId>` = `done` + result pointer (using the **new** `recordId` when `generate: true`) + `completedAt`.
|
|
60
|
+
2. Stamps the source record `_graphRuns.<graphId>` = `done` + result pointer (using the **new** `identity.recordId` when `generate: true`) + `completedAt`.
|
|
61
|
+
|
|
62
|
+
### Identity resolution (source records)
|
|
63
|
+
|
|
64
|
+
Source lookups are **tolerant**, in probe order: `identity.recordId` → `identity.targetId` → legacy flat `entityId`/`eventId` (pre-`memorix-record.2.0` tenants only) → `_id`. This lets `writeGraphRunResult`, `markGraphRunStarted`, and `markGraphRunFailed` find a record whether it was written under the current `memorix-record.2.0` envelope or an older flat schema, without the caller needing to know which.
|
|
65
|
+
|
|
66
|
+
This is a **read-time compatibility shim only**. It exists so old data keeps working during migration; it is not part of the public API (no `idField` type or helper is exported from `@x12i/memorix-writer`) and it never changes what gets **written** — new/updated documents always use the nested `identity` object, per [MEMORIX-DATABASE-CONVENTIONS.md](./MEMORIX-DATABASE-CONVENTIONS.md).
|
|
61
67
|
|
|
62
68
|
**Atomicity (Mongo transaction):** Result upsert and source stamp run inside a single Mongo transaction via `client.startSession().withTransaction()`. If either step fails, both are rolled back. **Requires a transaction-capable Mongo deployment** (replica set or sharded cluster). Standalone Mongo instances will fail at runtime when calling this API.
|
|
63
69
|
|
|
@@ -69,7 +75,7 @@ Sets terminal `failed` status with `error` and `completedAt`. Call only when `jo
|
|
|
69
75
|
|
|
70
76
|
Opt-in helper creating recommended compound indexes on the source collection:
|
|
71
77
|
|
|
72
|
-
- `{ "_graphRuns.<graphId>.status": 1,
|
|
78
|
+
- `{ "_graphRuns.<graphId>.status": 1, "identity.recordId": 1 }`
|
|
73
79
|
- `{ "_graphRuns.<graphId>.completedAt": 1 }`
|
|
74
80
|
|
|
75
81
|
## Retrieval filters (`@x12i/memorix-retrieval`)
|
package/docs/JOB-TYPE-RUNS.md
CHANGED
|
@@ -8,6 +8,8 @@ The reserved system field `_jobTypeRuns` is stored **top-level on the Mongo payl
|
|
|
8
8
|
|
|
9
9
|
Domain `writeMemorixRecord` operations **must not** accept `_jobTypeRuns` in user input.
|
|
10
10
|
|
|
11
|
+
Source-record lookups use the same tolerant identity resolution as `_graphRuns` — see [GRAPH-RUNS.md § Identity resolution](./GRAPH-RUNS.md#identity-resolution-source-records).
|
|
12
|
+
|
|
11
13
|
Shared types and helpers live in `@x12i/memorix-descriptors/types`:
|
|
12
14
|
|
|
13
15
|
- `JobTypeRunsField`, `GraphRunEntry`
|
|
@@ -45,7 +47,7 @@ When `jobTypeId` is provided, stamps both `_graphRuns.<graphId>` and `_jobTypeRu
|
|
|
45
47
|
|
|
46
48
|
Opt-in compound indexes for reprocessing filters:
|
|
47
49
|
|
|
48
|
-
- `{ "_jobTypeRuns.<key>.status": 1,
|
|
50
|
+
- `{ "_jobTypeRuns.<key>.status": 1, "identity.recordId": 1 }`
|
|
49
51
|
- `{ "_jobTypeRuns.<key>.completedAt": -1 }`
|
|
50
52
|
- `{ "_jobTypeRuns.<key>.status": 1, "_jobTypeRuns.<key>.completedAt": -1 }` (reprocessing)
|
|
51
53
|
|
|
@@ -2,6 +2,8 @@
|
|
|
2
2
|
|
|
3
3
|
This document defines the shared MongoDB naming and layout conventions for Memorix. All services, pipelines, and tools in the x12i ecosystem should follow these rules so entity data, event data, and source enrichment stay consistent across peers.
|
|
4
4
|
|
|
5
|
+
> This file is kept in sync with [`@x12i/memorix-retrieval`'s `MEMORIX-DATABASE-CONVENTIONS.md`](https://github.com/x12i/memorix-retrieval/blob/main/docs/MEMORIX-DATABASE-CONVENTIONS.md), which is the canonical copy. If the two disagree, treat the retrieval copy as authoritative and file a fix here.
|
|
6
|
+
|
|
5
7
|
## Overview
|
|
6
8
|
|
|
7
9
|
Memorix uses **three logical database roles**:
|
|
@@ -10,7 +12,7 @@ Memorix uses **three logical database roles**:
|
|
|
10
12
|
|------|---------|------------------------|
|
|
11
13
|
| **Entities** | Canonical entity records (vulnerabilities, assets, groups, …) | `memorix-entities` |
|
|
12
14
|
| **Events** | Event records derived from or linked to entities | `memorix-events` |
|
|
13
|
-
| **
|
|
15
|
+
| **Knowledge** | Knowledge-tier object payloads when configured | `memorix-knowledge` |
|
|
14
16
|
|
|
15
17
|
All three may live on the **same MongoDB cluster** (single `MONGO_URI`). They are separated by **database name**, not by connection string.
|
|
16
18
|
|
|
@@ -46,10 +48,17 @@ Packages resolve `memorix-entities`, `memorix-events`, collection names, and ent
|
|
|
46
48
|
|
|
47
49
|
Every Memorix record belongs to exactly one **target**:
|
|
48
50
|
|
|
49
|
-
| Target | Database | Typical use |
|
|
50
|
-
|
|
51
|
-
| `entity` | `memorix-entities` | Stable domain objects (vulnerability, asset, group, …) |
|
|
52
|
-
| `event` | `memorix-events` | Occurrences, detections, or timeline entries |
|
|
51
|
+
| Target | Database | Typical use | Identity target discriminator |
|
|
52
|
+
|--------|----------|-------------|--------------------------------|
|
|
53
|
+
| `entity` | `memorix-entities` | Stable domain objects (vulnerability, asset, group, …) | `entityId` |
|
|
54
|
+
| `event` | `memorix-events` | Occurrences, detections, or timeline entries | `eventId` |
|
|
55
|
+
| `knowledge` | `memorix-knowledge` | Knowledge-tier objects when configured | `knowledgeId` |
|
|
56
|
+
|
|
57
|
+
`entityId` / `eventId` / `knowledgeId` are **request-time parameter names** on write/lookup APIs (they tell the writer which target and identity value a call refers to). They are **not** stored as flat top-level Mongo fields — on disk the value lands in `identity.targetId` alongside `identity.target`. See [Document shape](#document-shape) below.
|
|
58
|
+
|
|
59
|
+
Multiple logical targets may share one physical database (e.g. `MEMORIX_ENTITIES_DB=memorix`, `MEMORIX_EVENTS_DB=memorix`, `MEMORIX_KNOWLEDGE_DB=memorix`). Retrieval returns separate logical bindings and marks `sharedPhysicalDatabaseWith`.
|
|
60
|
+
|
|
61
|
+
An empty or unconfigured knowledge target is **non-fatal** — inventory and health return `status: "not-configured"` or `"empty"` with informational warnings.
|
|
53
62
|
|
|
54
63
|
Tools must declare which target they read or write. Do not mix entity and event documents in the same collection.
|
|
55
64
|
|
|
@@ -85,7 +94,27 @@ Resolution order matters: **first non-empty value wins**.
|
|
|
85
94
|
| 4 | `MEMORIX_DB` *(legacy fallback)* |
|
|
86
95
|
| 5 | **Default:** `memorix-events` |
|
|
87
96
|
|
|
88
|
-
|
|
97
|
+
#### Knowledge database (`memorix-knowledge`)
|
|
98
|
+
|
|
99
|
+
| Priority | Variable |
|
|
100
|
+
|----------|----------|
|
|
101
|
+
| 1 | `MEMORIX_KNOWLEDGE_DB` |
|
|
102
|
+
| 2 | `MEMORIX_DB_KNOWLEDGE` |
|
|
103
|
+
| 3 | `memorix_knowledge_db` |
|
|
104
|
+
| 4 | **Default:** `memorix-knowledge` |
|
|
105
|
+
|
|
106
|
+
Identity target discriminator: `knowledgeId`.
|
|
107
|
+
|
|
108
|
+
> **Peer rule:** Prefer `MEMORIX_ENTITIES_DB`, `MEMORIX_EVENTS_DB`, and `MEMORIX_KNOWLEDGE_DB`. Treat `MEMORIX_DB` as a legacy single-database override only.
|
|
109
|
+
|
|
110
|
+
#### Neo / non-default deployments
|
|
111
|
+
|
|
112
|
+
Some clusters store Memorix data under non-default database names (for example `memorix-neo-entities` / `memorix-neo-events` instead of `memorix-entities` / `memorix-events`). Set:
|
|
113
|
+
|
|
114
|
+
```bash
|
|
115
|
+
MEMORIX_ENTITIES_DB=memorix-neo-entities
|
|
116
|
+
MEMORIX_EVENTS_DB=memorix-neo-events
|
|
117
|
+
```
|
|
89
118
|
|
|
90
119
|
### Source databases
|
|
91
120
|
|
|
@@ -107,6 +136,7 @@ Normalize `entityType` by replacing `-` with `_` and uppercasing.
|
|
|
107
136
|
|--------|-----------------|----------------------------------------|
|
|
108
137
|
| `entity` | `MEMORIX_ENTITIES_COLLECTION_<ENTITY_TYPE>` | `MEMORIX_ENTITIES_COLLECTION_VULNERABILITY` |
|
|
109
138
|
| `event` | `MEMORIX_EVENTS_COLLECTION_<ENTITY_TYPE>` | `MEMORIX_EVENTS_COLLECTION_VULNERABILITY` |
|
|
139
|
+
| `knowledge` | `MEMORIX_KNOWLEDGE_COLLECTION_<ENTITY_TYPE>` | `MEMORIX_KNOWLEDGE_COLLECTION_PLAYBOOK` |
|
|
110
140
|
|
|
111
141
|
## Collection naming
|
|
112
142
|
|
|
@@ -117,7 +147,6 @@ Use **plural, kebab-case** collection names that describe the entity domain.
|
|
|
117
147
|
| `entityType` | Recommended collection | Accepted fallbacks |
|
|
118
148
|
|--------------|------------------------|--------------------|
|
|
119
149
|
| `vulnerability` | `vulnerabilities` | `vulnerability` |
|
|
120
|
-
| `variabilities-group` | `vulnerability-groups` | `vulnerabilities-groups`, `variabilities-groups` |
|
|
121
150
|
| `assets` | `assets` | — |
|
|
122
151
|
|
|
123
152
|
**Default heuristic** when nothing else is configured:
|
|
@@ -143,6 +172,8 @@ Use `<domain>-events` or `<domain>s-events`.
|
|
|
143
172
|
<entityType>-events → e.g. vulnerability-events
|
|
144
173
|
```
|
|
145
174
|
|
|
175
|
+
For content-type slices (used by `@x12i/memorix-writer`), collections resolve as `{collectionPrefix}-{postfix}` (e.g. `assets` + `analysis` → `assets-analysis`). See the [memorix-writer README](../README.md#collection-resolution) for the full resolution order.
|
|
176
|
+
|
|
146
177
|
### Source collections
|
|
147
178
|
|
|
148
179
|
Source collections typically use an **`-information`** suffix to distinguish them from Memorix canonical stores:
|
|
@@ -150,10 +181,9 @@ Source collections typically use an **`-information`** suffix to distinguish the
|
|
|
150
181
|
| Source collection | Feeds entity type | Notes |
|
|
151
182
|
|-------------------|-------------------|-------|
|
|
152
183
|
| `vulnerabilities-information` | `vulnerability` | Raw / enriched vulnerability facts |
|
|
153
|
-
| `vulnerability-groups-information` | `variabilities-group` | Group-level enrichment |
|
|
154
184
|
| `assets-information` | `assets` | Asset metadata |
|
|
155
185
|
|
|
156
|
-
**Peer rule:** Source collection names are referenced in `targetSelector.sourceCollection`
|
|
186
|
+
**Peer rule:** Source collection names are referenced in `targetSelector.sourceCollection` in completion mappings and in `_system.provenance.source.collection` on Memorix records. Keep these strings stable — they are the join key for mapping selection.
|
|
157
187
|
|
|
158
188
|
## Collection name resolution order
|
|
159
189
|
|
|
@@ -171,48 +201,69 @@ All peers should implement the same precedence so local, CI, and production beha
|
|
|
171
201
|
|
|
172
202
|
## Document shape
|
|
173
203
|
|
|
174
|
-
### Memorix entity / event
|
|
204
|
+
### Memorix 2.0 record (entity / event / knowledge)
|
|
175
205
|
|
|
176
|
-
|
|
206
|
+
All Memorix records use the grouped envelope introduced in Memorix 2.0. Version string: `memorix-record.2.0`.
|
|
177
207
|
|
|
178
208
|
```json
|
|
179
209
|
{
|
|
180
210
|
"_id": "<ObjectId>",
|
|
181
|
-
"
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
"
|
|
211
|
+
"identity": {
|
|
212
|
+
"recordId": "<stable logical id>",
|
|
213
|
+
"target": "entity | event | knowledge",
|
|
214
|
+
"targetId": "<target-specific id>",
|
|
215
|
+
"objectType": "<object type>",
|
|
216
|
+
"contentType": "<snapshots | core | scoped | inferences | ...>",
|
|
217
|
+
"version": "memorix-record.2.0"
|
|
218
|
+
},
|
|
219
|
+
"lifecycle": {
|
|
220
|
+
"createdAt": "<ISO-8601>",
|
|
221
|
+
"modifiedAt": "<ISO-8601>",
|
|
222
|
+
"capturedAt": "<ISO-8601 optional>",
|
|
223
|
+
"status": "active"
|
|
224
|
+
},
|
|
225
|
+
"concept": {
|
|
226
|
+
"kind": "<readable kind>",
|
|
227
|
+
"name": "<primary readable label>",
|
|
228
|
+
"title": "<optional>"
|
|
229
|
+
},
|
|
230
|
+
"data": { "... domain fields ..." },
|
|
231
|
+
"associatedData": { "... snapshot-only related context ..." },
|
|
232
|
+
"analytics": { "... snapshot-only numeric rollups ..." },
|
|
233
|
+
"insights": { "... snapshot-only interpretations ..." },
|
|
234
|
+
"sourceFacts": { "schemaVersion": "sourceFacts.v1", "values": {} },
|
|
235
|
+
"_system": {
|
|
236
|
+
"memorix": {},
|
|
237
|
+
"provenance": {},
|
|
238
|
+
"processing": {},
|
|
239
|
+
"narratives": {}
|
|
187
240
|
}
|
|
188
241
|
}
|
|
189
242
|
```
|
|
190
243
|
|
|
191
|
-
|
|
|
192
|
-
|
|
193
|
-
| `
|
|
194
|
-
| `
|
|
195
|
-
| `
|
|
196
|
-
| `
|
|
197
|
-
| `
|
|
198
|
-
| `
|
|
199
|
-
|
|
200
|
-
\*Required for completion pipelines that match on these paths.
|
|
244
|
+
| Block | Required | Notes |
|
|
245
|
+
|-------|----------|-------|
|
|
246
|
+
| `identity` | Yes | Use `identity.recordId` and `identity.targetId` — not root `entityId` / `eventId` / `recordId` |
|
|
247
|
+
| `lifecycle` | Recommended | Timestamps and record status |
|
|
248
|
+
| `concept` | Snapshots | Readable core label |
|
|
249
|
+
| `data` | Yes | Domain payload; all content types |
|
|
250
|
+
| `associatedData` / `analytics` / `insights` | Snapshots only | Full materialized view blocks |
|
|
251
|
+
| `sourceFacts` | Optional | Raw/opaque source values |
|
|
252
|
+
| `_system` | Recommended | Memorix-managed metadata, provenance, processing |
|
|
201
253
|
|
|
202
|
-
Legacy
|
|
254
|
+
**Legacy root fields (`entityId`, `eventId`, `recordId`, `displayName`, `snapshotId`, `_memorix`, root `idField`, etc.) must not be written by any package in this monorepo, no matter what a client requests.** `@x12i/memorix-writer` only reads them as a tolerant, read-only fallback when locating pre-2.0 records that predate this envelope (see [GRAPH-RUNS.md § Identity resolution](./GRAPH-RUNS.md#identity-resolution-source-records)); it never writes them and never exposes an `idField` concept in its public API.
|
|
203
255
|
|
|
204
|
-
### Provenance (
|
|
205
|
-
|
|
206
|
-
Completion and ingestion tools track provenance in memory using:
|
|
256
|
+
### Provenance (under `_system.provenance`)
|
|
207
257
|
|
|
208
258
|
| Field | Description |
|
|
209
259
|
|-------|-------------|
|
|
210
|
-
| `
|
|
211
|
-
| `
|
|
260
|
+
| `source.collection` | Which source collection this record was enriched from |
|
|
261
|
+
| `source.database` | Source database name |
|
|
262
|
+
| `source.server` | Source server label (when applicable) |
|
|
212
263
|
| `sourceId` | Id in the source system |
|
|
213
|
-
| `
|
|
264
|
+
| `snapshotRecordId` | Snapshot lineage id (snapshots only) |
|
|
214
265
|
|
|
215
|
-
These align with `targetSelector` in completion mappings.
|
|
266
|
+
These align with `targetSelector` in completion mappings (`targetSelector.sourceCollection` matches `provenance.source.collection`). Source id-field names belong in source lookup paths such as `match.sourcePath`, not in record provenance.
|
|
216
267
|
|
|
217
268
|
### Source document
|
|
218
269
|
|
|
@@ -226,7 +277,6 @@ Do **not** write completion results back to source collections unless a separate
|
|
|
226
277
|
|
|
227
278
|
```
|
|
228
279
|
vulnerability
|
|
229
|
-
variabilities-group
|
|
230
280
|
assets
|
|
231
281
|
```
|
|
232
282
|
|
|
@@ -255,7 +305,7 @@ Tools that enrich Memorix from source data (e.g. `@x12i/memorix-completion`) use
|
|
|
255
305
|
"collection": "vulnerabilities-information"
|
|
256
306
|
},
|
|
257
307
|
"match": {
|
|
258
|
-
"targetPath": "
|
|
308
|
+
"targetPath": "identity.targetId",
|
|
259
309
|
"sourcePath": "vulnerabilityId"
|
|
260
310
|
},
|
|
261
311
|
"writeRoot": "data",
|
|
@@ -270,9 +320,10 @@ Tools that enrich Memorix from source data (e.g. `@x12i/memorix-completion`) use
|
|
|
270
320
|
| `target` | `"entity"` or `"event"` — selects Memorix database |
|
|
271
321
|
| `targetCollection` | Explicit Memorix collection (recommended) |
|
|
272
322
|
| `targetSelector.sourceCollection` | Must match record's source lineage |
|
|
323
|
+
| `targetSelector.idField` | Name of the **source** document's id field (source documents have no envelope) |
|
|
273
324
|
| `source.databaseName` | Source DB; use `{{ENV.*}}` for env-driven deploys |
|
|
274
325
|
| `source.collection` | Source collection to read from |
|
|
275
|
-
| `match.targetPath` | Path on Memorix record (
|
|
326
|
+
| `match.targetPath` | Path on the Memorix record — `identity.targetId` (never a root `entityId` / `eventId`) |
|
|
276
327
|
| `match.sourcePath` | Path on source document |
|
|
277
328
|
| `writeRoot` | Always `"data"` — never overwrite envelope fields via completion |
|
|
278
329
|
|
|
@@ -302,10 +353,10 @@ MEMORIX_EVENTS_COLLECTION_VULNERABILITY=vulnerability-events
|
|
|
302
353
|
When building a new service or tool that touches Memorix:
|
|
303
354
|
|
|
304
355
|
- [ ] Connect with `MONGO_URI` / `MONGO_CONNECTION_STRING`
|
|
305
|
-
- [ ] Select database via `target`: `entity` → `memorix-entities`, `event` → `memorix-events`
|
|
356
|
+
- [ ] Select database via `target`: `entity` → `memorix-entities`, `event` → `memorix-events`, `knowledge` → `memorix-knowledge`
|
|
306
357
|
- [ ] Respect env precedence documented above
|
|
307
358
|
- [ ] Use recommended collection names or set explicit env overrides
|
|
308
|
-
- [ ] Store domain payload under `data`
|
|
359
|
+
- [ ] Store domain payload under `data`, identity under `identity` — never write root `entityId` / `eventId` / `idField`
|
|
309
360
|
- [ ] Read enrichment from source databases; write only to Memorix target databases
|
|
310
361
|
- [ ] Use stable `sourceCollection` strings for mapping / provenance
|
|
311
362
|
- [ ] Use kebab-case `entityType` values consistently
|
|
@@ -313,12 +364,15 @@ When building a new service or tool that touches Memorix:
|
|
|
313
364
|
|
|
314
365
|
## Reference implementation
|
|
315
366
|
|
|
316
|
-
|
|
367
|
+
Shared env resolution is mirrored across peers:
|
|
317
368
|
|
|
318
369
|
| Concern | Module |
|
|
319
370
|
|---------|--------|
|
|
320
|
-
| Database name from env | `src/mongo/env.ts` → `resolveMemorixDbName`, `resolveDefaultSourceDbName` |
|
|
321
|
-
|
|
|
322
|
-
|
|
|
323
|
-
|
|
324
|
-
|
|
371
|
+
| Database name from env | `@x12i/memorix-completion` / `@x12i/memorix-retrieval` → `src/mongo/env.ts` → `resolveMemorixDbName`, `resolveDefaultSourceDbName` |
|
|
372
|
+
| Completion collection resolution | `@x12i/memorix-completion` → `src/mongo/resolve-collection.ts` → `resolveTargetCollectionName` |
|
|
373
|
+
| Retrieval collection resolution | `@x12i/memorix-retrieval` → `src/data/collection-name.ts` → `resolveMemorixCollectionName` (descriptor + env override + prefix/postfix heuristic) |
|
|
374
|
+
| Write collection resolution | `@x12i/memorix-writer` → `src/graph-runs/resolve-record-collection.ts` (descriptor-driven; see package README) |
|
|
375
|
+
| Tolerant identity lookup (read-only, backward-compat) | `@x12i/memorix-writer` → `src/data/record-lookup.ts` (internal — not exported) |
|
|
376
|
+
| Record document shape | `@x12i/memorix-completion` → `src/types.ts` → `MemorixRecord`, `CompletionMapping` |
|
|
377
|
+
|
|
378
|
+
If this document and a package disagree, **update this document and the package together** — they are intended to stay in sync.
|