@x12i/memorix-retrieval 1.5.1 → 1.6.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/client/create-client.d.ts.map +1 -1
- package/dist/client/create-client.js +4 -4
- package/dist/client/create-client.js.map +1 -1
- package/dist/client/xronox-adapter.d.ts.map +1 -1
- package/dist/client/xronox-adapter.js +7 -0
- package/dist/client/xronox-adapter.js.map +1 -1
- package/dist/client/xronox-like.d.ts +22 -0
- package/dist/client/xronox-like.d.ts.map +1 -1
- package/dist/data/memorix-count.d.ts +1 -0
- package/dist/data/memorix-count.d.ts.map +1 -1
- package/dist/data/memorix-count.js +1 -0
- package/dist/data/memorix-count.js.map +1 -1
- package/dist/data/memorix-read.d.ts.map +1 -1
- package/dist/data/memorix-read.js +5 -0
- package/dist/data/memorix-read.js.map +1 -1
- package/dist/explorer/collection-inventory.d.ts +77 -0
- package/dist/explorer/collection-inventory.d.ts.map +1 -0
- package/dist/explorer/collection-inventory.js +302 -0
- package/dist/explorer/collection-inventory.js.map +1 -0
- package/dist/explorer/entity-graph.d.ts +35 -12
- package/dist/explorer/entity-graph.d.ts.map +1 -1
- package/dist/explorer/entity-graph.js +117 -55
- package/dist/explorer/entity-graph.js.map +1 -1
- package/dist/index.d.ts +2 -1
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +1 -0
- package/dist/index.js.map +1 -1
- package/dist/tests/collection-inventory.test.d.ts +2 -0
- package/dist/tests/collection-inventory.test.d.ts.map +1 -0
- package/dist/tests/collection-inventory.test.js +207 -0
- package/dist/tests/collection-inventory.test.js.map +1 -0
- package/dist/tests/entity-graph.test.d.ts +2 -0
- package/dist/tests/entity-graph.test.d.ts.map +1 -0
- package/dist/tests/entity-graph.test.js +148 -0
- package/dist/tests/entity-graph.test.js.map +1 -0
- package/docs/DATA-TIER-CONTRACT.md +6 -3
- package/docs/EXPLORER-HOST-APIS.md +16 -2
- package/docs/MEMORIX-CATALOX-CONTRACTS.md +8 -4
- package/docs/XRONOX-DATA-TIER-REQUIREMENTS.md +46 -349
- package/docs/fr/README.md +15 -0
- package/docs/fr/xronox-fr-list-collections.md +196 -0
- package/docs/fr/xronox-fr-memorix-env-preset.md +69 -0
- package/docs/fr/xronox-fr-per-role-selftest.md +60 -0
- package/package.json +2 -2
|
@@ -2,13 +2,14 @@
|
|
|
2
2
|
|
|
3
3
|
**Audience:** `@x12i/xronox` maintainers
|
|
4
4
|
**Consumer:** `@x12i/memorix-retrieval` (v1.5+) and hosts (Memorix Explorer, APIs, tools)
|
|
5
|
-
**Status:**
|
|
5
|
+
**Status:** Active — open FRs in [`docs/fr/`](./fr/README.md). **No Mongo workarounds in production paths.**
|
|
6
6
|
|
|
7
7
|
Related docs:
|
|
8
8
|
|
|
9
9
|
- [DATA-TIER-CONTRACT.md](./DATA-TIER-CONTRACT.md) — public retrieval API
|
|
10
10
|
- [MEMORIX-DATABASE-CONVENTIONS.md](./MEMORIX-DATABASE-CONVENTIONS.md) — Mongo layout and env vars
|
|
11
11
|
- [MEMORIX-CATALOX-CONTRACTS.md](./MEMORIX-CATALOX-CONTRACTS.md) — descriptor metadata (Catalox, not Xronox)
|
|
12
|
+
- **[Feature requests for `@x12i/xronox`](./fr/README.md)** — implement here first; retrieval waits
|
|
12
13
|
|
|
13
14
|
---
|
|
14
15
|
|
|
@@ -22,7 +23,7 @@ Memorix separates **metadata** from **payload**:
|
|
|
22
23
|
| Payload | **Xronox** | Routed Mongo reads/counts by role + collection |
|
|
23
24
|
| Composition | **memorix-retrieval** | Join descriptors + Xronox I/O → list/item rows |
|
|
24
25
|
|
|
25
|
-
Hosts must **not** connect to Mongo directly. Retrieval’s default bootstrap is:
|
|
26
|
+
Hosts must **not** connect to Mongo directly in production. Retrieval’s default bootstrap is:
|
|
26
27
|
|
|
27
28
|
```ts
|
|
28
29
|
createMemorixRetrievalFromEnv({ catalox })
|
|
@@ -30,7 +31,7 @@ createMemorixRetrievalFromEnv({ catalox })
|
|
|
30
31
|
→ createMemorixXronoxAdapter(createXronox())
|
|
31
32
|
```
|
|
32
33
|
|
|
33
|
-
All list/item/workspace reads and counts
|
|
34
|
+
All list/item/workspace reads and counts flow through Xronox routing.
|
|
34
35
|
|
|
35
36
|
---
|
|
36
37
|
|
|
@@ -45,391 +46,87 @@ Every Memorix Mongo read uses this Xronox routing shape:
|
|
|
45
46
|
dataType: "metadata",
|
|
46
47
|
sourceType: "database",
|
|
47
48
|
subSourceType: "mongo",
|
|
48
|
-
role: "<xronox-role>",
|
|
49
|
-
source: "<collection-name>",
|
|
49
|
+
role: "<xronox-role>",
|
|
50
|
+
source: "<collection-name>",
|
|
50
51
|
query?: Record<string, unknown>,
|
|
51
52
|
project?: string[],
|
|
52
53
|
sort?: string | string[] | Record<string, 1 | -1>,
|
|
53
54
|
limit?: number,
|
|
54
|
-
skip?: number,
|
|
55
|
+
skip?: number,
|
|
55
56
|
}
|
|
56
57
|
```
|
|
57
58
|
|
|
58
|
-
Collection names and queries come from **Catalox entity descriptors** (not hardcoded in retrieval). Xronox resolves **which Mongo database** to use from `role`.
|
|
59
|
-
|
|
60
59
|
### 2.2 Role mapping (Memorix target → Xronox role)
|
|
61
60
|
|
|
62
|
-
Entity descriptors declare `target`:
|
|
63
|
-
|
|
64
61
|
| Entity descriptor `target` | Memorix database (logical) | Default Xronox `role` (3.8+) |
|
|
65
62
|
|--------------------------|----------------------------|------------------------------|
|
|
66
63
|
| `"entity"` (default) | `memorix-entities` | `memorix_entities` |
|
|
67
64
|
| `"event"` | `memorix-events` | `memorix_events` |
|
|
68
65
|
|
|
69
|
-
Legacy aliases still work in Xronox: `operational` / `logs`. Prefer `memorix_entities` / `memorix_events` — they map to `memory.entities` / `memory.events` with defaults `memorix-entities` / `memorix-events`.
|
|
70
|
-
|
|
71
|
-
Override per client: `createMemorixRetrieval({ xronoxRoles: { entity: "…", event: "…" } })`.
|
|
72
|
-
|
|
73
66
|
Constants in retrieval: `DEFAULT_MEMORIX_XRONOX_ROLES` (`src/data/memorix-read.ts`).
|
|
74
67
|
|
|
75
|
-
### 2.3
|
|
76
|
-
|
|
77
|
-
Retrieval initializes Xronox as:
|
|
78
|
-
|
|
79
|
-
```ts
|
|
80
|
-
await xronox.init({
|
|
81
|
-
engine: "nxMongo",
|
|
82
|
-
zeroConfig: true,
|
|
83
|
-
lazyInit: true,
|
|
84
|
-
envFile: false, // host already loaded .env
|
|
85
|
-
});
|
|
86
|
-
```
|
|
87
|
-
|
|
88
|
-
**Required env (minimum):**
|
|
89
|
-
|
|
90
|
-
| Variable | Purpose |
|
|
91
|
-
|----------|---------|
|
|
92
|
-
| `MONGO_URI` | Cluster connection (aliases: `MONGO_URL`, `XRONOX_MONGO_URI`) |
|
|
93
|
-
|
|
94
|
-
**Required for correct entity/event split (recommended explicit):**
|
|
95
|
-
|
|
96
|
-
| Xronox role | Env (Xronox role resolution) | Should resolve to (Memorix convention) |
|
|
97
|
-
|-------------|------------------------------|----------------------------------------|
|
|
98
|
-
| `operational` | `MONGO_OPERATIONAL_DB` (+ fallbacks) | `memorix-entities` or `MEMORIX_ENTITIES_DB` |
|
|
99
|
-
| `logs` | `MONGO_LOGS_DB` (+ fallbacks) | `memorix-events` or `MEMORIX_EVENTS_DB` |
|
|
100
|
-
|
|
101
|
-
> **Gap:** Memorix documents `MEMORIX_ENTITIES_DB` / `MEMORIX_EVENTS_DB`; Xronox zero-config defaults to `MONGO_DB` / role vars. Deployments must align both naming schemes until Xronox ships a Memorix preset (§4.6).
|
|
102
|
-
|
|
103
|
-
### 2.4 Operations retrieval performs via Xronox
|
|
104
|
-
|
|
105
|
-
| Operation | Retrieval API | Xronox usage | Frequency |
|
|
106
|
-
|-----------|---------------|--------------|-----------|
|
|
107
|
-
| Paginated list driver read | `fetchMemorixList` | `read` / `readArray` | High |
|
|
108
|
-
| Extension batch fetch | `fetchMemorixList` | `read` / `readArray` by id `$in` | High |
|
|
109
|
-
| Item detail fetch | `fetchMemorixItem` | `read` / `readArray` | Medium |
|
|
110
|
-
| List total (`includeTotal`) | `fetchMemorixList` | **count** (shimmed today) | Medium |
|
|
111
|
-
| Entity graph counts | `buildMemorixEntityGraph` | **count** per content type | Medium |
|
|
112
|
-
| Raw explorer reads | `listMemorixEntityContentTypeDocuments` | `read` + **count** | Medium |
|
|
113
|
-
| Scoped workspace (legacy) | `listScopedWorkspaceDocuments` | `read` + **count** | Low |
|
|
114
|
-
| Health check | `getMemorixRetrievalHealth` | `selfTest({ testConnectivity: true })` | Low |
|
|
115
|
-
|
|
116
|
-
Retrieval does **not** write to Mongo via Xronox (writes are `@x12i/memorix-writer` / completion pipelines).
|
|
117
|
-
|
|
118
|
-
---
|
|
119
|
-
|
|
120
|
-
## 3. What works in Xronox 3.7.3 (used today)
|
|
121
|
-
|
|
122
|
-
| Capability | Xronox API | Memorix usage |
|
|
123
|
-
|------------|------------|---------------|
|
|
124
|
-
| Routed Mongo read | `read` / `readArray` | All document fetches |
|
|
125
|
-
| Single document | `readOne` | Not used yet; candidate for `getMemorixEntityContentTypeDocument` |
|
|
126
|
-
| Connectivity check | `selfTest({ testConnectivity })` | Health endpoint |
|
|
127
|
-
| Zero-config init | `init({ engine: "nxMongo", zeroConfig: true })` | Default bootstrap |
|
|
128
|
-
| Role-based DB routing | `role` on `ReadParams` | Entity vs event databases |
|
|
129
|
-
| Collection as `source` | `source: string` | Descriptor-resolved collection name |
|
|
130
|
-
| Mongo filter query | `query` | List filters + searchText |
|
|
131
|
-
| Field projection | `project: string[]` | List field composition |
|
|
132
|
-
| Sort (string form) | `sort: string \| string[]` | List sort (adapter converts `{ path: 1 \| -1 }`) |
|
|
133
|
-
| Limit | `limit` | Pagination page size |
|
|
134
|
-
|
|
135
|
-
---
|
|
136
|
-
|
|
137
|
-
## 4. Gaps — what we need from Xronox
|
|
138
|
-
|
|
139
|
-
Priority: **P0** = blocks correct production behavior; **P1** = performance/ergonomics; **P2** = nice-to-have.
|
|
140
|
-
|
|
141
|
-
### 4.1 P0 — Native `countDocuments` (routed)
|
|
142
|
-
|
|
143
|
-
**Problem:** Xronox 3.7 has no public count API. Retrieval shims counts by loading all matching `_id` values through `readArray`:
|
|
144
|
-
|
|
145
|
-
```ts
|
|
146
|
-
// memorix-retrieval/src/client/xronox-adapter.ts (interim)
|
|
147
|
-
countDocuments(params) {
|
|
148
|
-
const rows = await readArray({ ...params, project: ["_id"], limit: undefined });
|
|
149
|
-
return rows.length;
|
|
150
|
-
}
|
|
151
|
-
```
|
|
152
|
-
|
|
153
|
-
This breaks down on large collections (assets, vulnerabilities) when hosts use `includeTotal: true` or graph counts.
|
|
154
|
-
|
|
155
|
-
**Request:** Add to `Xronox` interface:
|
|
156
|
-
|
|
157
|
-
```ts
|
|
158
|
-
countDocuments(params: CountParams): Promise<number>;
|
|
159
|
-
|
|
160
|
-
type CountParams = RoutingKey & {
|
|
161
|
-
subSourceType: "mongo";
|
|
162
|
-
source: string;
|
|
163
|
-
role?: string;
|
|
164
|
-
query?: Record<string, unknown>;
|
|
165
|
-
/** Optional — default false. When true, use estimatedDocumentCount (no filter). */
|
|
166
|
-
estimated?: boolean;
|
|
167
|
-
};
|
|
168
|
-
```
|
|
169
|
-
|
|
170
|
-
**Semantics:**
|
|
171
|
-
|
|
172
|
-
- Same binding resolution as `read` (role → connection → db → collection).
|
|
173
|
-
- Uses Mongo `countDocuments(query)` — **not** a full collection scan.
|
|
174
|
-
- Must respect `query` identically to `read`.
|
|
175
|
-
- Return type: `number` (throws on routing/connection errors, same as read).
|
|
176
|
-
|
|
177
|
-
**Acceptance:**
|
|
178
|
-
|
|
179
|
-
- `countDocuments` with filter matches Mongo shell `countDocuments` on the resolved collection.
|
|
180
|
-
- Works for both `role: operational` and `role: logs` in a two-DB Memorix deployment.
|
|
181
|
-
- Documented in Xronox README + `.env.example`.
|
|
182
|
-
|
|
183
|
-
---
|
|
184
|
-
|
|
185
|
-
### 4.2 P0 — `skip` (offset pagination)
|
|
186
|
-
|
|
187
|
-
**Problem:** Retrieval passes `skip` on every paginated list read (`page.offset`). Xronox `ReadParams` and `nxMongoEngine.readMongo` **do not forward `skip`** to the driver (verified in `@x12i/xronox@3.7.3`). Offset pagination silently returns wrong pages when using the Xronox data tier.
|
|
188
|
-
|
|
189
|
-
**Request:** Add to `ReadParams`:
|
|
190
|
-
|
|
191
|
-
```ts
|
|
192
|
-
skip?: number;
|
|
193
|
-
```
|
|
194
|
-
|
|
195
|
-
**Engine behavior:** Pass through to Mongo cursor `.skip(skip).limit(limit)`.
|
|
196
|
-
|
|
197
|
-
**Acceptance:**
|
|
198
|
-
|
|
199
|
-
- `readArray({ ..., query, sort, skip: 50, limit: 25 })` returns items 51–75.
|
|
200
|
-
- Matches retrieval `fetchMemorixList` pagination tests against direct Mongo.
|
|
201
|
-
|
|
202
|
-
---
|
|
203
|
-
|
|
204
|
-
### 4.3 P1 — `sort` as Mongo object
|
|
205
|
-
|
|
206
|
-
**Problem:** Retrieval builds sort as `Record<string, 1 | -1>` (Mongo native). Xronox `ReadParams.sort` is typed as `string | string[]` only. Retrieval converts in an adapter:
|
|
207
|
-
|
|
208
|
-
```ts
|
|
209
|
-
{ "data.ip_address": 1 } → ["data.ip_address"]
|
|
210
|
-
{ "data.priorityScore": -1 } → ["-data.priorityScore"]
|
|
211
|
-
```
|
|
212
|
-
|
|
213
|
-
**Request:** Extend `ReadParams.sort`:
|
|
214
|
-
|
|
215
|
-
```ts
|
|
216
|
-
sort?: string | string[] | Record<string, 1 | -1>;
|
|
217
|
-
```
|
|
218
|
-
|
|
219
|
-
**Acceptance:** Nested field paths (e.g. `data.xdr.host_name`) sort correctly.
|
|
220
|
-
|
|
221
|
-
---
|
|
222
|
-
|
|
223
|
-
### 4.4 P1 — Typed errors for routing failures
|
|
224
|
-
|
|
225
|
-
**Problem:** Hosts need to distinguish misconfiguration from empty results.
|
|
226
|
-
|
|
227
|
-
**Request:** Stable error codes (or `ErrorCodes` enum) for:
|
|
228
|
-
|
|
229
|
-
| Code | When |
|
|
230
|
-
|------|------|
|
|
231
|
-
| `ROUTING_NOT_FOUND` | No binding for role + subSourceType |
|
|
232
|
-
| `CONNECTION_NOT_FOUND` | Binding references missing connection alias |
|
|
233
|
-
| `DATABASE_NOT_RESOLVED` | Role/env missing DB name |
|
|
234
|
-
| `COLLECTION_NOT_FOUND` | Optional — collection absent (may be empty vs error) |
|
|
68
|
+
### 2.3 Operations retrieval performs via Xronox
|
|
235
69
|
|
|
236
|
-
Retrieval
|
|
70
|
+
| Operation | Retrieval API | Xronox usage | Blocked? |
|
|
71
|
+
|-----------|---------------|--------------|----------|
|
|
72
|
+
| Paginated list | `fetchMemorixList` | `read` / `readArray` | — |
|
|
73
|
+
| List total | `fetchMemorixList` | `countDocuments` | — |
|
|
74
|
+
| Entity graph counts | `buildMemorixEntityGraph` | `countDocuments` per content type | — |
|
|
75
|
+
| **Collection inventory scan** | `buildMemorixCollectionInventory` | **`listCollections` + `countDocuments`** | — (3.9.0+) |
|
|
76
|
+
| Raw explorer reads | `listMemorixEntityContentTypeDocuments` | `read` + `countDocuments` | — |
|
|
77
|
+
| Health | `getMemorixRetrievalHealth` | `selfTest` | Partial — [FR](./fr/xronox-fr-per-role-selftest.md) |
|
|
237
78
|
|
|
238
79
|
---
|
|
239
80
|
|
|
240
|
-
|
|
81
|
+
## 3. Shipped in Xronox (retrieval uses today)
|
|
241
82
|
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
|
|
250
|
-
|
|
251
|
-
|
|
252
|
-
**Problem:** Two parallel env naming schemes:
|
|
253
|
-
|
|
254
|
-
- Memorix: `MEMORIX_ENTITIES_DB`, `MEMORIX_EVENTS_DB`
|
|
255
|
-
- Xronox: `MONGO_OPERATIONAL_DB`, `MONGO_LOGS_DB`
|
|
256
|
-
|
|
257
|
-
**Request (choose one or both):**
|
|
258
|
-
|
|
259
|
-
**Option A — Env aliases in Xronox role resolution:**
|
|
260
|
-
|
|
261
|
-
```
|
|
262
|
-
MONGO_OPERATIONAL_DB ← fallback MEMORIX_ENTITIES_DB
|
|
263
|
-
MONGO_LOGS_DB ← fallback MEMORIX_EVENTS_DB
|
|
264
|
-
```
|
|
265
|
-
|
|
266
|
-
**Option B — Zero-config preset:**
|
|
267
|
-
|
|
268
|
-
```ts
|
|
269
|
-
await xronox.init({ engine: "nxMongo", preset: "memorix" });
|
|
270
|
-
```
|
|
271
|
-
|
|
272
|
-
Preset behavior:
|
|
273
|
-
|
|
274
|
-
- `operational` → `MEMORIX_ENTITIES_DB` ?? `memorix-entities`
|
|
275
|
-
- `logs` → `MEMORIX_EVENTS_DB` ?? `memorix-events`
|
|
276
|
-
- Single `MONGO_URI` cluster
|
|
277
|
-
|
|
278
|
-
**Acceptance:** Memorix smoke script works with only `MONGO_URI` + `MEMORIX_*_DB` set (no duplicate Xronox-specific DB vars).
|
|
83
|
+
| Capability | Xronox API | Since |
|
|
84
|
+
|------------|------------|-------|
|
|
85
|
+
| Routed read | `read` / `readArray` | 3.8+ |
|
|
86
|
+
| Routed count | `countDocuments` (incl. `estimated: true`) | 3.8+ |
|
|
87
|
+
| Offset pagination | `skip` + `limit` | 3.8+ |
|
|
88
|
+
| Object sort | `sort: Record<string, 1 \| -1>` | 3.8+ |
|
|
89
|
+
| Role-based DB routing | `role` on params | 3.8+ |
|
|
90
|
+
| Database collection listing | `listCollections({ role, source: "_db" })` | **3.9+** |
|
|
91
|
+
| Connectivity | `selfTest({ testConnectivity })` | 3.8+ |
|
|
279
92
|
|
|
280
93
|
---
|
|
281
94
|
|
|
282
|
-
|
|
95
|
+
## 4. Open — implement in Xronox (see FRs)
|
|
283
96
|
|
|
284
|
-
|
|
97
|
+
| Priority | FR | Blocks |
|
|
98
|
+
|----------|-----|--------|
|
|
99
|
+
| P1 | [Memorix env preset](./fr/xronox-fr-memorix-env-preset.md) | `MEMORIX_*_DB`-only zero-config |
|
|
100
|
+
| P2 | [Per-role selfTest](./fr/xronox-fr-per-role-selftest.md) | Health detail for both DBs |
|
|
285
101
|
|
|
286
|
-
|
|
287
|
-
const xronox = createXronox();
|
|
288
|
-
await xronox.init({ engine: "nxMongo", zeroConfig: true, lazyInit: true, envFile: false });
|
|
289
|
-
```
|
|
102
|
+
~~P0 listCollections~~ — **shipped 3.9.0**. See [FR](./fr/xronox-fr-list-collections.md).
|
|
290
103
|
|
|
291
|
-
**
|
|
292
|
-
|
|
293
|
-
```ts
|
|
294
|
-
createXronoxFromEnv(options?: {
|
|
295
|
-
preset?: "default" | "memorix";
|
|
296
|
-
envFile?: string | false;
|
|
297
|
-
engine?: EngineMode;
|
|
298
|
-
}): Promise<Xronox>;
|
|
299
|
-
```
|
|
300
|
-
|
|
301
|
-
Retrieval would replace `createMemorixXronoxFromEnv` with a thin wrapper or re-export.
|
|
104
|
+
**Do not** implement these in retrieval with `connectMemorixMongo` or other driver bypasses.
|
|
302
105
|
|
|
303
106
|
---
|
|
304
107
|
|
|
305
|
-
|
|
306
|
-
|
|
307
|
-
**Problem:** Health check runs one `selfTest`. Memorix needs both entity and event DBs reachable.
|
|
108
|
+
## 5. Retrieval policy (no workarounds)
|
|
308
109
|
|
|
309
|
-
|
|
310
|
-
|
|
311
|
-
|
|
312
|
-
|
|
313
|
-
|
|
314
|
-
|
|
315
|
-
role: string;
|
|
316
|
-
db: string;
|
|
317
|
-
ok: boolean;
|
|
318
|
-
latencyMs?: number;
|
|
319
|
-
error?: string;
|
|
320
|
-
}>;
|
|
321
|
-
}
|
|
322
|
-
```
|
|
323
|
-
|
|
324
|
-
Optional init flag: `selfTestRoles: ["operational", "logs"]`.
|
|
110
|
+
| Rule | Detail |
|
|
111
|
+
|------|--------|
|
|
112
|
+
| Production I/O | Xronox only |
|
|
113
|
+
| `buildMemorixCollectionInventory` | Uses Xronox `listCollections` + `countDocuments` (requires `@x12i/xronox` ≥ 3.9.0) |
|
|
114
|
+
| Tests | Mock `xronox.listCollections` / inject test `mongo` only in unit tests — not production bootstrap |
|
|
115
|
+
| Explorer host shims | Delete `memorix-explorer/server/collection-inventory.ts` after bumping to retrieval 1.6.1+ |
|
|
325
116
|
|
|
326
117
|
---
|
|
327
118
|
|
|
328
|
-
|
|
329
|
-
|
|
330
|
-
**Problem:** Cross-entity workspace merge (v1) may fetch many rows per entity. `read` returning `AsyncIterable` exists but `readArray` materializes everything.
|
|
331
|
-
|
|
332
|
-
**Request:** Document streaming pattern; ensure `countDocuments` never materializes rows. Optional `readStream` alias for clarity.
|
|
333
|
-
|
|
334
|
-
---
|
|
335
|
-
|
|
336
|
-
## 5. Proposed Xronox surface (summary)
|
|
337
|
-
|
|
338
|
-
```ts
|
|
339
|
-
interface Xronox {
|
|
340
|
-
// Existing
|
|
341
|
-
read(params: ReadParams): Promise<unknown[] | AsyncIterable<unknown>>;
|
|
342
|
-
readArray(params: ReadParams): Promise<unknown[]>;
|
|
343
|
-
readOne(params: ReadOneParams): Promise<unknown | null>;
|
|
344
|
-
selfTest(options?: { testConnectivity?: boolean }): Promise<SelfTestResult>;
|
|
345
|
-
init(opts: XronoxInitOptions): Promise<void>;
|
|
346
|
-
close(): Promise<void>;
|
|
347
|
-
|
|
348
|
-
// NEW — P0
|
|
349
|
-
countDocuments(params: CountParams): Promise<number>;
|
|
350
|
-
}
|
|
351
|
-
|
|
352
|
-
interface ReadParams extends RoutingKey {
|
|
353
|
-
query?: Record<string, unknown>;
|
|
354
|
-
project?: string[];
|
|
355
|
-
sort?: string | string[] | Record<string, 1 | -1>; // P1
|
|
356
|
-
limit?: number;
|
|
357
|
-
skip?: number; // P0
|
|
358
|
-
}
|
|
359
|
-
```
|
|
360
|
-
|
|
361
|
-
---
|
|
362
|
-
|
|
363
|
-
## 6. Consumer adapter (retrieval side)
|
|
364
|
-
|
|
365
|
-
Until Xronox ships §4.1–§4.3, retrieval keeps `createMemorixXronoxAdapter` (`src/client/xronox-adapter.ts`):
|
|
366
|
-
|
|
367
|
-
| Xronox gap | Retrieval shim |
|
|
368
|
-
|------------|----------------|
|
|
369
|
-
| No `countDocuments` | `_id` projection + `readArray` length |
|
|
370
|
-
| No `Record` sort | Convert to `string[]` before `read` |
|
|
371
|
-
| No `skip` | **None** — pagination may be wrong until Xronox fix |
|
|
372
|
-
|
|
373
|
-
When Xronox releases native APIs, retrieval will:
|
|
374
|
-
|
|
375
|
-
1. Delegate `countDocuments` directly (remove shim).
|
|
376
|
-
2. Pass `skip` / object `sort` through without conversion.
|
|
377
|
-
3. Drop direct Mongo escape hatch from default path (keep for tests only).
|
|
378
|
-
|
|
379
|
-
---
|
|
380
|
-
|
|
381
|
-
## 7. Test matrix (acceptance for Xronox PR)
|
|
382
|
-
|
|
383
|
-
Run against a fixture with:
|
|
384
|
-
|
|
385
|
-
- DB `memorix-entities`, collection `assets-snapshots`, role `operational`
|
|
386
|
-
- DB `memorix-events`, collection `vulnerabilities-snapshots`, role `logs`
|
|
387
|
-
|
|
388
|
-
| # | Test | Expected |
|
|
389
|
-
|---|------|----------|
|
|
390
|
-
| 1 | `readArray` with `limit: 10, skip: 0` vs `skip: 10` | Disjoint pages, stable sort |
|
|
391
|
-
| 2 | `countDocuments` with `{}` vs `readArray` length on small set | Equal |
|
|
392
|
-
| 3 | `countDocuments` with filter vs Mongo shell | Equal |
|
|
393
|
-
| 4 | `countDocuments` on 100k+ docs | Completes in O(1) network vs full scan |
|
|
394
|
-
| 5 | Wrong `role` | Typed routing error |
|
|
395
|
-
| 6 | `selfTest` with both roles configured | Both DBs reported |
|
|
396
|
-
|
|
397
|
-
Retrieval integration test (downstream): `npm run smoke:retrieval -- --workspace-list` with `includeTotal: true`.
|
|
398
|
-
|
|
399
|
-
---
|
|
400
|
-
|
|
401
|
-
## 8. Version alignment
|
|
119
|
+
## 6. Version alignment
|
|
402
120
|
|
|
403
121
|
| Package | Current | Notes |
|
|
404
122
|
|---------|---------|-------|
|
|
405
|
-
| `@x12i/memorix-retrieval` | 1.
|
|
406
|
-
| `@x12i/xronox` | ^3.
|
|
407
|
-
|
|
408
|
-
**Suggested Xronox release:** minor bump (e.g. 3.8.0) with `countDocuments` + `skip` — no breaking changes to existing `read` callers.
|
|
409
|
-
|
|
410
|
-
---
|
|
411
|
-
|
|
412
|
-
## 9. Out of scope for Xronox
|
|
413
|
-
|
|
414
|
-
These stay in **Catalox** / **memorix-retrieval**, not Xronox:
|
|
415
|
-
|
|
416
|
-
- Entity discovery, list columns, field paths, filters allowlist
|
|
417
|
-
- Row composition, multi-match, relations, content objects
|
|
418
|
-
- Cross-entity workspace merge logic
|
|
419
|
-
- Catalox seed / descriptor validation
|
|
420
|
-
|
|
421
|
-
Xronox should remain a **database routing + I/O layer**, not Memorix-aware.
|
|
123
|
+
| `@x12i/memorix-retrieval` | 1.6.1 | Inventory via Xronox `listCollections` |
|
|
124
|
+
| `@x12i/xronox` | ^3.9.0 | `listCollections` required for inventory |
|
|
422
125
|
|
|
423
126
|
---
|
|
424
127
|
|
|
425
|
-
##
|
|
128
|
+
## 7. Out of scope for Xronox
|
|
426
129
|
|
|
427
|
-
|
|
428
|
-
|------|---------|
|
|
429
|
-
| `src/data/memorix-read.ts` | Read routing + role resolution |
|
|
430
|
-
| `src/data/memorix-count.ts` | Count entry point |
|
|
431
|
-
| `src/client/xronox-adapter.ts` | Interim shims |
|
|
432
|
-
| `src/client/create-xronox-from-env.ts` | Bootstrap |
|
|
433
|
-
| `src/client/xronox-like.ts` | Minimal interface retrieval expects |
|
|
130
|
+
Entity discovery, list columns, row composition, Catalox seed, orphan name parsing — stay in Catalox / memorix-retrieval.
|
|
434
131
|
|
|
435
|
-
|
|
132
|
+
Xronox remains a **database routing + I/O layer**, not Memorix-aware.
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
# Feature requests — `@x12i/xronox` (Memorix data tier)
|
|
2
|
+
|
|
3
|
+
**Audience:** `@x12i/xronox` maintainers
|
|
4
|
+
**Consumer:** `@x12i/memorix-retrieval`, Memorix Explorer, APIs, tools
|
|
5
|
+
**Policy:** Retrieval and hosts **must not** bypass Xronox with direct Mongo for production paths. Open FRs block retrieval features until Xronox ships the generic API.
|
|
6
|
+
|
|
7
|
+
| FR | Status | Blocks |
|
|
8
|
+
|----|--------|--------|
|
|
9
|
+
| [xronox-fr-list-collections.md](./xronox-fr-list-collections.md) | **Shipped** (`@x12i/xronox` 3.9.0) | — |
|
|
10
|
+
| [xronox-fr-memorix-env-preset.md](./xronox-fr-memorix-env-preset.md) | **Open** | Zero-config with `MEMORIX_*_DB` only |
|
|
11
|
+
| [xronox-fr-per-role-selftest.md](./xronox-fr-per-role-selftest.md) | **Open** | Health reporting both entity + event DBs |
|
|
12
|
+
|
|
13
|
+
**Already in Xronox 3.8+ (no FR):** `countDocuments`, `skip`, `sort` as Mongo object, `countDocuments({ estimated: true })`.
|
|
14
|
+
|
|
15
|
+
See also: [XRONOX-DATA-TIER-REQUIREMENTS.md](../XRONOX-DATA-TIER-REQUIREMENTS.md) (integration index).
|
|
@@ -0,0 +1,196 @@
|
|
|
1
|
+
# Feature request — routed `listCollections` (`@x12i/xronox`)
|
|
2
|
+
|
|
3
|
+
**Date:** 2026-05-22
|
|
4
|
+
**Status:** **Shipped** — `@x12i/xronox@3.9.0`
|
|
5
|
+
**Consumer:** `@x12i/memorix-retrieval@1.6.1+`
|
|
6
|
+
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
## Summary
|
|
10
|
+
|
|
11
|
+
Memorix **collection inventory** lists all Mongo collections in **both** Memorix databases (`memorix-entities`, `memorix-events`) using the same **role → database** routing as `read` and `countDocuments`. Implemented in Xronox 3.9.0; retrieval uses it with **no direct Mongo bypass**.
|
|
12
|
+
|
|
13
|
+
---
|
|
14
|
+
|
|
15
|
+
## Why this belongs in Xronox (not retrieval)
|
|
16
|
+
|
|
17
|
+
| Concern | Host/retrieval bypass problem |
|
|
18
|
+
|---------|-------------------------------|
|
|
19
|
+
| Role routing | Must use same `role` → DB resolution as reads (`memorix_entities`, `memorix_events`) |
|
|
20
|
+
| Multi-DB | Inventory scans **both** databases; routing must stay in Xronox |
|
|
21
|
+
| Contract | [DATA-TIER-CONTRACT.md](../DATA-TIER-CONTRACT.md): production I/O through Xronox only |
|
|
22
|
+
| Reuse | Any x12i app with role-based Mongo needs collection discovery, not only Memorix |
|
|
23
|
+
|
|
24
|
+
---
|
|
25
|
+
|
|
26
|
+
## Product requirement
|
|
27
|
+
|
|
28
|
+
Expose a routed API that returns collection names (and optional types) for the **database resolved from `role`**, without targeting a single collection in `source`.
|
|
29
|
+
|
|
30
|
+
Orphan scan flow (retrieval consumer):
|
|
31
|
+
|
|
32
|
+
1. `listCollections({ role: "memorix_entities" })` → all non-system collections in entity DB
|
|
33
|
+
2. `listCollections({ role: "memorix_events" })` → all non-system collections in event DB
|
|
34
|
+
3. For each collection not in Catalox descriptor map: `countDocuments({ role, source: collectionName, estimated: true })` (already in Xronox 3.8+)
|
|
35
|
+
|
|
36
|
+
Declared slot counts continue to use descriptor-routed `countDocuments` via `countMemorixEntityContentTypeDocuments` (retrieval).
|
|
37
|
+
|
|
38
|
+
---
|
|
39
|
+
|
|
40
|
+
## Proposed API
|
|
41
|
+
|
|
42
|
+
```ts
|
|
43
|
+
listCollections(params: ListCollectionsParams): Promise<ListCollectionsResult>;
|
|
44
|
+
|
|
45
|
+
export interface ListedCollection {
|
|
46
|
+
name: string;
|
|
47
|
+
type?: string; // "collection" | "view" | ...
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
export interface ListCollectionsResult {
|
|
51
|
+
/** Resolved database name after role + tenancy (same as countDocuments/read) */
|
|
52
|
+
database: string;
|
|
53
|
+
collections: ListedCollection[];
|
|
54
|
+
}
|
|
55
|
+
|
|
56
|
+
export interface ListCollectionsParams extends RoutingKey {
|
|
57
|
+
subSourceType: "mongo";
|
|
58
|
+
/**
|
|
59
|
+
* Binding match token only — not a collection name.
|
|
60
|
+
* Zero-config bindings with no `match` accept any source (e.g. `"_db"`).
|
|
61
|
+
*/
|
|
62
|
+
source: string;
|
|
63
|
+
role?: string;
|
|
64
|
+
/** Forwarded to Mongo db.listCollections({ filter }) */
|
|
65
|
+
filter?: Record<string, unknown>;
|
|
66
|
+
/** Default true — omit names starting with `system.` */
|
|
67
|
+
excludeSystem?: boolean;
|
|
68
|
+
/** Default true — use driver nameOnly for lighter responses */
|
|
69
|
+
nameOnly?: boolean;
|
|
70
|
+
}
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
Add to `Xronox` interface and `Engine` (nxMongo engine only; xronoxCore may omit like `countDocuments`).
|
|
74
|
+
|
|
75
|
+
---
|
|
76
|
+
|
|
77
|
+
## Algorithm (normative)
|
|
78
|
+
|
|
79
|
+
1. `resolveBinding(config, params)` — **same as `countDocuments` and `read`**.
|
|
80
|
+
2. Resolve `connection` → `SimpleMongoHelper`.
|
|
81
|
+
3. Resolve `db` from binding + `params.role` (role overrides binding default db).
|
|
82
|
+
4. `sanitizeDatabaseName(db)` — same as existing count path.
|
|
83
|
+
5. `mongoDb.listCollections(filter, { nameOnly })`.
|
|
84
|
+
6. If `excludeSystem !== false`, filter out `system.*` names.
|
|
85
|
+
7. Return `{ database: db, collections }`.
|
|
86
|
+
|
|
87
|
+
**Do not** use `params.source` as a collection name. `source` exists only for binding selection.
|
|
88
|
+
|
|
89
|
+
---
|
|
90
|
+
|
|
91
|
+
## Example calls (Memorix)
|
|
92
|
+
|
|
93
|
+
```ts
|
|
94
|
+
// Entity store
|
|
95
|
+
const entityDb = await xronox.listCollections({
|
|
96
|
+
dataType: "metadata",
|
|
97
|
+
sourceType: "database",
|
|
98
|
+
subSourceType: "mongo",
|
|
99
|
+
source: "_db",
|
|
100
|
+
role: "memorix_entities",
|
|
101
|
+
});
|
|
102
|
+
|
|
103
|
+
// Event store
|
|
104
|
+
const eventDb = await xronox.listCollections({
|
|
105
|
+
dataType: "metadata",
|
|
106
|
+
sourceType: "database",
|
|
107
|
+
subSourceType: "mongo",
|
|
108
|
+
source: "_db",
|
|
109
|
+
role: "memorix_events",
|
|
110
|
+
});
|
|
111
|
+
|
|
112
|
+
// Orphan count (existing API)
|
|
113
|
+
const count = await xronox.countDocuments({
|
|
114
|
+
dataType: "metadata",
|
|
115
|
+
sourceType: "database",
|
|
116
|
+
subSourceType: "mongo",
|
|
117
|
+
role: "memorix_entities",
|
|
118
|
+
source: "orphan-entity-records",
|
|
119
|
+
estimated: true,
|
|
120
|
+
});
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
---
|
|
124
|
+
|
|
125
|
+
## Implementation guidance (`@x12i/xronox`)
|
|
126
|
+
|
|
127
|
+
| Layer | Change |
|
|
128
|
+
|-------|--------|
|
|
129
|
+
| `src/types.ts` | `ListCollectionsParams`, `ListCollectionsResult`, `ListedCollection`; extend `Xronox` |
|
|
130
|
+
| `src/engines/interface.ts` | `listCollections?(params, resolved)` |
|
|
131
|
+
| `src/engines/nxMongoEngine.ts` | Implement via `getDatabase` / `listCollections` (same helper lookup as `countDocuments`) |
|
|
132
|
+
| `src/index.ts` | `XronoxImpl.listCollections` → `resolveBinding` → engine |
|
|
133
|
+
| `README.md` | Document next to `countDocuments` example |
|
|
134
|
+
| Tests | Routing unit test (no Mongo); optional integration test with fixture DB |
|
|
135
|
+
|
|
136
|
+
Mirror error codes from count path: `CONNECTION_NOT_FOUND`, `DATABASE_NOT_RESOLVED`, `ROUTING_NOT_FOUND`.
|
|
137
|
+
|
|
138
|
+
---
|
|
139
|
+
|
|
140
|
+
## Retrieval integration (after Xronox ships)
|
|
141
|
+
|
|
142
|
+
| Step | Action |
|
|
143
|
+
|------|--------|
|
|
144
|
+
| 1 | Bump `@x12i/xronox` minimum (e.g. `^3.9.0`) |
|
|
145
|
+
| 2 | Extend `XronoxLike` + adapter to delegate `listCollections` |
|
|
146
|
+
| 3 | `buildMemorixCollectionInventory` calls Xronox only — **throws** if `listCollections` missing |
|
|
147
|
+
| 4 | Delete Explorer host shim `server/collection-inventory.ts` |
|
|
148
|
+
| 5 | `npm run smoke:retrieval -- --inventory` |
|
|
149
|
+
|
|
150
|
+
**No Mongo fallback in retrieval.** Tests may inject a mock `xronox` with `listCollections`; production requires real Xronox.
|
|
151
|
+
|
|
152
|
+
---
|
|
153
|
+
|
|
154
|
+
## Acceptance criteria
|
|
155
|
+
|
|
156
|
+
### Xronox
|
|
157
|
+
|
|
158
|
+
- [ ] `listCollections({ role: "memorix_entities" })` returns collections from `MEMORIX_ENTITIES_DB` / role map default
|
|
159
|
+
- [ ] `listCollections({ role: "memorix_events" })` returns collections from events DB
|
|
160
|
+
- [ ] `excludeSystem: true` (default) omits `system.*`
|
|
161
|
+
- [ ] `database` in result matches DB used for `countDocuments` on same role + collection
|
|
162
|
+
- [ ] Wrong/missing role → typed routing error (not empty list)
|
|
163
|
+
- [ ] nxMongo engine only; clear error on xronoxCore
|
|
164
|
+
|
|
165
|
+
### Retrieval (downstream)
|
|
166
|
+
|
|
167
|
+
- [ ] `buildMemorixCollectionInventory` completes without `connectMemorixMongo`
|
|
168
|
+
- [ ] Inventory `summary.byTarget` matches entries filtered by `target`
|
|
169
|
+
- [ ] Vitest mocks `xronox.listCollections` only (no Mongo client for scan)
|
|
170
|
+
|
|
171
|
+
---
|
|
172
|
+
|
|
173
|
+
## Non-goals
|
|
174
|
+
|
|
175
|
+
| Item | Reason |
|
|
176
|
+
|------|--------|
|
|
177
|
+
| Catalox orphan → descriptor seeding | Ops / Catalox tooling |
|
|
178
|
+
| Memorix-specific parsing in Xronox | Stays in retrieval (`parseOrphanCollection`) |
|
|
179
|
+
| List documents across all collections | Use inventory + per-collection `read` |
|
|
180
|
+
|
|
181
|
+
---
|
|
182
|
+
|
|
183
|
+
## Version
|
|
184
|
+
|
|
185
|
+
| Package | Minimum after FR |
|
|
186
|
+
|---------|------------------|
|
|
187
|
+
| `@x12i/xronox` | `3.9.0` (proposed minor) |
|
|
188
|
+
| `@x12i/memorix-retrieval` | Requires published Xronox with this FR |
|
|
189
|
+
|
|
190
|
+
---
|
|
191
|
+
|
|
192
|
+
## References
|
|
193
|
+
|
|
194
|
+
- Retrieval inventory: `src/explorer/collection-inventory.ts`
|
|
195
|
+
- Role constants: `src/data/memorix-read.ts` (`DEFAULT_MEMORIX_XRONOX_ROLES`)
|
|
196
|
+
- Explorer shim (delete after ship): `memorix-explorer/server/collection-inventory.ts`
|