@x12i/memorix-retrieval 1.5.1 → 1.6.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (44) hide show
  1. package/dist/client/create-client.d.ts.map +1 -1
  2. package/dist/client/create-client.js +4 -4
  3. package/dist/client/create-client.js.map +1 -1
  4. package/dist/client/xronox-adapter.d.ts.map +1 -1
  5. package/dist/client/xronox-adapter.js +7 -0
  6. package/dist/client/xronox-adapter.js.map +1 -1
  7. package/dist/client/xronox-like.d.ts +22 -0
  8. package/dist/client/xronox-like.d.ts.map +1 -1
  9. package/dist/data/memorix-count.d.ts +1 -0
  10. package/dist/data/memorix-count.d.ts.map +1 -1
  11. package/dist/data/memorix-count.js +1 -0
  12. package/dist/data/memorix-count.js.map +1 -1
  13. package/dist/data/memorix-read.d.ts.map +1 -1
  14. package/dist/data/memorix-read.js +5 -0
  15. package/dist/data/memorix-read.js.map +1 -1
  16. package/dist/explorer/collection-inventory.d.ts +77 -0
  17. package/dist/explorer/collection-inventory.d.ts.map +1 -0
  18. package/dist/explorer/collection-inventory.js +302 -0
  19. package/dist/explorer/collection-inventory.js.map +1 -0
  20. package/dist/explorer/entity-graph.d.ts +35 -12
  21. package/dist/explorer/entity-graph.d.ts.map +1 -1
  22. package/dist/explorer/entity-graph.js +117 -55
  23. package/dist/explorer/entity-graph.js.map +1 -1
  24. package/dist/index.d.ts +2 -1
  25. package/dist/index.d.ts.map +1 -1
  26. package/dist/index.js +1 -0
  27. package/dist/index.js.map +1 -1
  28. package/dist/tests/collection-inventory.test.d.ts +2 -0
  29. package/dist/tests/collection-inventory.test.d.ts.map +1 -0
  30. package/dist/tests/collection-inventory.test.js +207 -0
  31. package/dist/tests/collection-inventory.test.js.map +1 -0
  32. package/dist/tests/entity-graph.test.d.ts +2 -0
  33. package/dist/tests/entity-graph.test.d.ts.map +1 -0
  34. package/dist/tests/entity-graph.test.js +148 -0
  35. package/dist/tests/entity-graph.test.js.map +1 -0
  36. package/docs/DATA-TIER-CONTRACT.md +6 -3
  37. package/docs/EXPLORER-HOST-APIS.md +16 -2
  38. package/docs/MEMORIX-CATALOX-CONTRACTS.md +8 -4
  39. package/docs/XRONOX-DATA-TIER-REQUIREMENTS.md +46 -349
  40. package/docs/fr/README.md +15 -0
  41. package/docs/fr/xronox-fr-list-collections.md +196 -0
  42. package/docs/fr/xronox-fr-memorix-env-preset.md +69 -0
  43. package/docs/fr/xronox-fr-per-role-selftest.md +60 -0
  44. package/package.json +2 -2
@@ -2,13 +2,14 @@
2
2
 
3
3
  **Audience:** `@x12i/xronox` maintainers
4
4
  **Consumer:** `@x12i/memorix-retrieval` (v1.5+) and hosts (Memorix Explorer, APIs, tools)
5
- **Status:** Draftretrieval has migrated to Xronox as the default data tier; several gaps are shimmed in `memorix-retrieval` until Xronox ships native support.
5
+ **Status:** Activeopen FRs in [`docs/fr/`](./fr/README.md). **No Mongo workarounds in production paths.**
6
6
 
7
7
  Related docs:
8
8
 
9
9
  - [DATA-TIER-CONTRACT.md](./DATA-TIER-CONTRACT.md) — public retrieval API
10
10
  - [MEMORIX-DATABASE-CONVENTIONS.md](./MEMORIX-DATABASE-CONVENTIONS.md) — Mongo layout and env vars
11
11
  - [MEMORIX-CATALOX-CONTRACTS.md](./MEMORIX-CATALOX-CONTRACTS.md) — descriptor metadata (Catalox, not Xronox)
12
+ - **[Feature requests for `@x12i/xronox`](./fr/README.md)** — implement here first; retrieval waits
12
13
 
13
14
  ---
14
15
 
@@ -22,7 +23,7 @@ Memorix separates **metadata** from **payload**:
22
23
  | Payload | **Xronox** | Routed Mongo reads/counts by role + collection |
23
24
  | Composition | **memorix-retrieval** | Join descriptors + Xronox I/O → list/item rows |
24
25
 
25
- Hosts must **not** connect to Mongo directly. Retrieval’s default bootstrap is:
26
+ Hosts must **not** connect to Mongo directly in production. Retrieval’s default bootstrap is:
26
27
 
27
28
  ```ts
28
29
  createMemorixRetrievalFromEnv({ catalox })
@@ -30,7 +31,7 @@ createMemorixRetrievalFromEnv({ catalox })
30
31
  → createMemorixXronoxAdapter(createXronox())
31
32
  ```
32
33
 
33
- All list/item/workspace reads and counts should flow through Xronox routing — same pattern as `@x12i/memorix-writer` and other x12i data-tier peers.
34
+ All list/item/workspace reads and counts flow through Xronox routing.
34
35
 
35
36
  ---
36
37
 
@@ -45,391 +46,87 @@ Every Memorix Mongo read uses this Xronox routing shape:
45
46
  dataType: "metadata",
46
47
  sourceType: "database",
47
48
  subSourceType: "mongo",
48
- role: "<xronox-role>", // see §3
49
- source: "<collection-name>", // e.g. assets-snapshots, vulnerabilities-snapshots
49
+ role: "<xronox-role>",
50
+ source: "<collection-name>",
50
51
  query?: Record<string, unknown>,
51
52
  project?: string[],
52
53
  sort?: string | string[] | Record<string, 1 | -1>,
53
54
  limit?: number,
54
- skip?: number, // ⚠️ passed by retrieval; not implemented in Xronox 3.7 — see §4.2
55
+ skip?: number,
55
56
  }
56
57
  ```
57
58
 
58
- Collection names and queries come from **Catalox entity descriptors** (not hardcoded in retrieval). Xronox resolves **which Mongo database** to use from `role`.
59
-
60
59
  ### 2.2 Role mapping (Memorix target → Xronox role)
61
60
 
62
- Entity descriptors declare `target`:
63
-
64
61
  | Entity descriptor `target` | Memorix database (logical) | Default Xronox `role` (3.8+) |
65
62
  |--------------------------|----------------------------|------------------------------|
66
63
  | `"entity"` (default) | `memorix-entities` | `memorix_entities` |
67
64
  | `"event"` | `memorix-events` | `memorix_events` |
68
65
 
69
- Legacy aliases still work in Xronox: `operational` / `logs`. Prefer `memorix_entities` / `memorix_events` — they map to `memory.entities` / `memory.events` with defaults `memorix-entities` / `memorix-events`.
70
-
71
- Override per client: `createMemorixRetrieval({ xronoxRoles: { entity: "…", event: "…" } })`.
72
-
73
66
  Constants in retrieval: `DEFAULT_MEMORIX_XRONOX_ROLES` (`src/data/memorix-read.ts`).
74
67
 
75
- ### 2.3 Bootstrap (zero-config)
76
-
77
- Retrieval initializes Xronox as:
78
-
79
- ```ts
80
- await xronox.init({
81
- engine: "nxMongo",
82
- zeroConfig: true,
83
- lazyInit: true,
84
- envFile: false, // host already loaded .env
85
- });
86
- ```
87
-
88
- **Required env (minimum):**
89
-
90
- | Variable | Purpose |
91
- |----------|---------|
92
- | `MONGO_URI` | Cluster connection (aliases: `MONGO_URL`, `XRONOX_MONGO_URI`) |
93
-
94
- **Required for correct entity/event split (recommended explicit):**
95
-
96
- | Xronox role | Env (Xronox role resolution) | Should resolve to (Memorix convention) |
97
- |-------------|------------------------------|----------------------------------------|
98
- | `operational` | `MONGO_OPERATIONAL_DB` (+ fallbacks) | `memorix-entities` or `MEMORIX_ENTITIES_DB` |
99
- | `logs` | `MONGO_LOGS_DB` (+ fallbacks) | `memorix-events` or `MEMORIX_EVENTS_DB` |
100
-
101
- > **Gap:** Memorix documents `MEMORIX_ENTITIES_DB` / `MEMORIX_EVENTS_DB`; Xronox zero-config defaults to `MONGO_DB` / role vars. Deployments must align both naming schemes until Xronox ships a Memorix preset (§4.6).
102
-
103
- ### 2.4 Operations retrieval performs via Xronox
104
-
105
- | Operation | Retrieval API | Xronox usage | Frequency |
106
- |-----------|---------------|--------------|-----------|
107
- | Paginated list driver read | `fetchMemorixList` | `read` / `readArray` | High |
108
- | Extension batch fetch | `fetchMemorixList` | `read` / `readArray` by id `$in` | High |
109
- | Item detail fetch | `fetchMemorixItem` | `read` / `readArray` | Medium |
110
- | List total (`includeTotal`) | `fetchMemorixList` | **count** (shimmed today) | Medium |
111
- | Entity graph counts | `buildMemorixEntityGraph` | **count** per content type | Medium |
112
- | Raw explorer reads | `listMemorixEntityContentTypeDocuments` | `read` + **count** | Medium |
113
- | Scoped workspace (legacy) | `listScopedWorkspaceDocuments` | `read` + **count** | Low |
114
- | Health check | `getMemorixRetrievalHealth` | `selfTest({ testConnectivity: true })` | Low |
115
-
116
- Retrieval does **not** write to Mongo via Xronox (writes are `@x12i/memorix-writer` / completion pipelines).
117
-
118
- ---
119
-
120
- ## 3. What works in Xronox 3.7.3 (used today)
121
-
122
- | Capability | Xronox API | Memorix usage |
123
- |------------|------------|---------------|
124
- | Routed Mongo read | `read` / `readArray` | All document fetches |
125
- | Single document | `readOne` | Not used yet; candidate for `getMemorixEntityContentTypeDocument` |
126
- | Connectivity check | `selfTest({ testConnectivity })` | Health endpoint |
127
- | Zero-config init | `init({ engine: "nxMongo", zeroConfig: true })` | Default bootstrap |
128
- | Role-based DB routing | `role` on `ReadParams` | Entity vs event databases |
129
- | Collection as `source` | `source: string` | Descriptor-resolved collection name |
130
- | Mongo filter query | `query` | List filters + searchText |
131
- | Field projection | `project: string[]` | List field composition |
132
- | Sort (string form) | `sort: string \| string[]` | List sort (adapter converts `{ path: 1 \| -1 }`) |
133
- | Limit | `limit` | Pagination page size |
134
-
135
- ---
136
-
137
- ## 4. Gaps — what we need from Xronox
138
-
139
- Priority: **P0** = blocks correct production behavior; **P1** = performance/ergonomics; **P2** = nice-to-have.
140
-
141
- ### 4.1 P0 — Native `countDocuments` (routed)
142
-
143
- **Problem:** Xronox 3.7 has no public count API. Retrieval shims counts by loading all matching `_id` values through `readArray`:
144
-
145
- ```ts
146
- // memorix-retrieval/src/client/xronox-adapter.ts (interim)
147
- countDocuments(params) {
148
- const rows = await readArray({ ...params, project: ["_id"], limit: undefined });
149
- return rows.length;
150
- }
151
- ```
152
-
153
- This breaks down on large collections (assets, vulnerabilities) when hosts use `includeTotal: true` or graph counts.
154
-
155
- **Request:** Add to `Xronox` interface:
156
-
157
- ```ts
158
- countDocuments(params: CountParams): Promise<number>;
159
-
160
- type CountParams = RoutingKey & {
161
- subSourceType: "mongo";
162
- source: string;
163
- role?: string;
164
- query?: Record<string, unknown>;
165
- /** Optional — default false. When true, use estimatedDocumentCount (no filter). */
166
- estimated?: boolean;
167
- };
168
- ```
169
-
170
- **Semantics:**
171
-
172
- - Same binding resolution as `read` (role → connection → db → collection).
173
- - Uses Mongo `countDocuments(query)` — **not** a full collection scan.
174
- - Must respect `query` identically to `read`.
175
- - Return type: `number` (throws on routing/connection errors, same as read).
176
-
177
- **Acceptance:**
178
-
179
- - `countDocuments` with filter matches Mongo shell `countDocuments` on the resolved collection.
180
- - Works for both `role: operational` and `role: logs` in a two-DB Memorix deployment.
181
- - Documented in Xronox README + `.env.example`.
182
-
183
- ---
184
-
185
- ### 4.2 P0 — `skip` (offset pagination)
186
-
187
- **Problem:** Retrieval passes `skip` on every paginated list read (`page.offset`). Xronox `ReadParams` and `nxMongoEngine.readMongo` **do not forward `skip`** to the driver (verified in `@x12i/xronox@3.7.3`). Offset pagination silently returns wrong pages when using the Xronox data tier.
188
-
189
- **Request:** Add to `ReadParams`:
190
-
191
- ```ts
192
- skip?: number;
193
- ```
194
-
195
- **Engine behavior:** Pass through to Mongo cursor `.skip(skip).limit(limit)`.
196
-
197
- **Acceptance:**
198
-
199
- - `readArray({ ..., query, sort, skip: 50, limit: 25 })` returns items 51–75.
200
- - Matches retrieval `fetchMemorixList` pagination tests against direct Mongo.
201
-
202
- ---
203
-
204
- ### 4.3 P1 — `sort` as Mongo object
205
-
206
- **Problem:** Retrieval builds sort as `Record<string, 1 | -1>` (Mongo native). Xronox `ReadParams.sort` is typed as `string | string[]` only. Retrieval converts in an adapter:
207
-
208
- ```ts
209
- { "data.ip_address": 1 } → ["data.ip_address"]
210
- { "data.priorityScore": -1 } → ["-data.priorityScore"]
211
- ```
212
-
213
- **Request:** Extend `ReadParams.sort`:
214
-
215
- ```ts
216
- sort?: string | string[] | Record<string, 1 | -1>;
217
- ```
218
-
219
- **Acceptance:** Nested field paths (e.g. `data.xdr.host_name`) sort correctly.
220
-
221
- ---
222
-
223
- ### 4.4 P1 — Typed errors for routing failures
224
-
225
- **Problem:** Hosts need to distinguish misconfiguration from empty results.
226
-
227
- **Request:** Stable error codes (or `ErrorCodes` enum) for:
228
-
229
- | Code | When |
230
- |------|------|
231
- | `ROUTING_NOT_FOUND` | No binding for role + subSourceType |
232
- | `CONNECTION_NOT_FOUND` | Binding references missing connection alias |
233
- | `DATABASE_NOT_RESOLVED` | Role/env missing DB name |
234
- | `COLLECTION_NOT_FOUND` | Optional — collection absent (may be empty vs error) |
68
+ ### 2.3 Operations retrieval performs via Xronox
235
69
 
236
- Retrieval maps these to `MemorixRetrievalError` where appropriate.
70
+ | Operation | Retrieval API | Xronox usage | Blocked? |
71
+ |-----------|---------------|--------------|----------|
72
+ | Paginated list | `fetchMemorixList` | `read` / `readArray` | — |
73
+ | List total | `fetchMemorixList` | `countDocuments` | — |
74
+ | Entity graph counts | `buildMemorixEntityGraph` | `countDocuments` per content type | — |
75
+ | **Collection inventory scan** | `buildMemorixCollectionInventory` | **`listCollections` + `countDocuments`** | — (3.9.0+) |
76
+ | Raw explorer reads | `listMemorixEntityContentTypeDocuments` | `read` + `countDocuments` | — |
77
+ | Health | `getMemorixRetrievalHealth` | `selfTest` | Partial — [FR](./fr/xronox-fr-per-role-selftest.md) |
237
78
 
238
79
  ---
239
80
 
240
- ### 4.5 P1 `readOne` alignment for identity fetch
81
+ ## 3. Shipped in Xronox (retrieval uses today)
241
82
 
242
- **Problem:** `getMemorixEntityContentTypeDocument` uses `read` + `limit: 1`. Xronox already exposes `readOne`.
243
-
244
- **Request:** Document `readOne` for `{ filter, role, source, project }` as the preferred single-doc path. Ensure `readOne` supports the same routing key as `read`.
245
-
246
- **Optional retrieval change after Xronox confirms:** switch item/single-doc paths to `readOne`.
247
-
248
- ---
249
-
250
- ### 4.6 P1 Memorix deployment preset (env + zero-config)
251
-
252
- **Problem:** Two parallel env naming schemes:
253
-
254
- - Memorix: `MEMORIX_ENTITIES_DB`, `MEMORIX_EVENTS_DB`
255
- - Xronox: `MONGO_OPERATIONAL_DB`, `MONGO_LOGS_DB`
256
-
257
- **Request (choose one or both):**
258
-
259
- **Option A — Env aliases in Xronox role resolution:**
260
-
261
- ```
262
- MONGO_OPERATIONAL_DB ← fallback MEMORIX_ENTITIES_DB
263
- MONGO_LOGS_DB ← fallback MEMORIX_EVENTS_DB
264
- ```
265
-
266
- **Option B — Zero-config preset:**
267
-
268
- ```ts
269
- await xronox.init({ engine: "nxMongo", preset: "memorix" });
270
- ```
271
-
272
- Preset behavior:
273
-
274
- - `operational` → `MEMORIX_ENTITIES_DB` ?? `memorix-entities`
275
- - `logs` → `MEMORIX_EVENTS_DB` ?? `memorix-events`
276
- - Single `MONGO_URI` cluster
277
-
278
- **Acceptance:** Memorix smoke script works with only `MONGO_URI` + `MEMORIX_*_DB` set (no duplicate Xronox-specific DB vars).
83
+ | Capability | Xronox API | Since |
84
+ |------------|------------|-------|
85
+ | Routed read | `read` / `readArray` | 3.8+ |
86
+ | Routed count | `countDocuments` (incl. `estimated: true`) | 3.8+ |
87
+ | Offset pagination | `skip` + `limit` | 3.8+ |
88
+ | Object sort | `sort: Record<string, 1 \| -1>` | 3.8+ |
89
+ | Role-based DB routing | `role` on params | 3.8+ |
90
+ | Database collection listing | `listCollections({ role, source: "_db" })` | **3.9+** |
91
+ | Connectivity | `selfTest({ testConnectivity })` | 3.8+ |
279
92
 
280
93
  ---
281
94
 
282
- ### 4.7 P2`createXronoxFromEnv` helper
95
+ ## 4. Openimplement in Xronox (see FRs)
283
96
 
284
- **Problem:** Every host duplicates:
97
+ | Priority | FR | Blocks |
98
+ |----------|-----|--------|
99
+ | P1 | [Memorix env preset](./fr/xronox-fr-memorix-env-preset.md) | `MEMORIX_*_DB`-only zero-config |
100
+ | P2 | [Per-role selfTest](./fr/xronox-fr-per-role-selftest.md) | Health detail for both DBs |
285
101
 
286
- ```ts
287
- const xronox = createXronox();
288
- await xronox.init({ engine: "nxMongo", zeroConfig: true, lazyInit: true, envFile: false });
289
- ```
102
+ ~~P0 listCollections~~ — **shipped 3.9.0**. See [FR](./fr/xronox-fr-list-collections.md).
290
103
 
291
- **Request:** Export from `@x12i/xronox`:
292
-
293
- ```ts
294
- createXronoxFromEnv(options?: {
295
- preset?: "default" | "memorix";
296
- envFile?: string | false;
297
- engine?: EngineMode;
298
- }): Promise<Xronox>;
299
- ```
300
-
301
- Retrieval would replace `createMemorixXronoxFromEnv` with a thin wrapper or re-export.
104
+ **Do not** implement these in retrieval with `connectMemorixMongo` or other driver bypasses.
302
105
 
303
106
  ---
304
107
 
305
- ### 4.8 P2 Per-role `selfTest` detail
306
-
307
- **Problem:** Health check runs one `selfTest`. Memorix needs both entity and event DBs reachable.
108
+ ## 5. Retrieval policy (no workarounds)
308
109
 
309
- **Request:** Extend `SelfTestResult`:
310
-
311
- ```ts
312
- {
313
- passed: boolean;
314
- roles?: Array<{
315
- role: string;
316
- db: string;
317
- ok: boolean;
318
- latencyMs?: number;
319
- error?: string;
320
- }>;
321
- }
322
- ```
323
-
324
- Optional init flag: `selfTestRoles: ["operational", "logs"]`.
110
+ | Rule | Detail |
111
+ |------|--------|
112
+ | Production I/O | Xronox only |
113
+ | `buildMemorixCollectionInventory` | Uses Xronox `listCollections` + `countDocuments` (requires `@x12i/xronox` ≥ 3.9.0) |
114
+ | Tests | Mock `xronox.listCollections` / inject test `mongo` only in unit tests — not production bootstrap |
115
+ | Explorer host shims | Delete `memorix-explorer/server/collection-inventory.ts` after bumping to retrieval 1.6.1+ |
325
116
 
326
117
  ---
327
118
 
328
- ### 4.9 P2 — Streaming reads for large scans
329
-
330
- **Problem:** Cross-entity workspace merge (v1) may fetch many rows per entity. `read` returning `AsyncIterable` exists but `readArray` materializes everything.
331
-
332
- **Request:** Document streaming pattern; ensure `countDocuments` never materializes rows. Optional `readStream` alias for clarity.
333
-
334
- ---
335
-
336
- ## 5. Proposed Xronox surface (summary)
337
-
338
- ```ts
339
- interface Xronox {
340
- // Existing
341
- read(params: ReadParams): Promise<unknown[] | AsyncIterable<unknown>>;
342
- readArray(params: ReadParams): Promise<unknown[]>;
343
- readOne(params: ReadOneParams): Promise<unknown | null>;
344
- selfTest(options?: { testConnectivity?: boolean }): Promise<SelfTestResult>;
345
- init(opts: XronoxInitOptions): Promise<void>;
346
- close(): Promise<void>;
347
-
348
- // NEW — P0
349
- countDocuments(params: CountParams): Promise<number>;
350
- }
351
-
352
- interface ReadParams extends RoutingKey {
353
- query?: Record<string, unknown>;
354
- project?: string[];
355
- sort?: string | string[] | Record<string, 1 | -1>; // P1
356
- limit?: number;
357
- skip?: number; // P0
358
- }
359
- ```
360
-
361
- ---
362
-
363
- ## 6. Consumer adapter (retrieval side)
364
-
365
- Until Xronox ships §4.1–§4.3, retrieval keeps `createMemorixXronoxAdapter` (`src/client/xronox-adapter.ts`):
366
-
367
- | Xronox gap | Retrieval shim |
368
- |------------|----------------|
369
- | No `countDocuments` | `_id` projection + `readArray` length |
370
- | No `Record` sort | Convert to `string[]` before `read` |
371
- | No `skip` | **None** — pagination may be wrong until Xronox fix |
372
-
373
- When Xronox releases native APIs, retrieval will:
374
-
375
- 1. Delegate `countDocuments` directly (remove shim).
376
- 2. Pass `skip` / object `sort` through without conversion.
377
- 3. Drop direct Mongo escape hatch from default path (keep for tests only).
378
-
379
- ---
380
-
381
- ## 7. Test matrix (acceptance for Xronox PR)
382
-
383
- Run against a fixture with:
384
-
385
- - DB `memorix-entities`, collection `assets-snapshots`, role `operational`
386
- - DB `memorix-events`, collection `vulnerabilities-snapshots`, role `logs`
387
-
388
- | # | Test | Expected |
389
- |---|------|----------|
390
- | 1 | `readArray` with `limit: 10, skip: 0` vs `skip: 10` | Disjoint pages, stable sort |
391
- | 2 | `countDocuments` with `{}` vs `readArray` length on small set | Equal |
392
- | 3 | `countDocuments` with filter vs Mongo shell | Equal |
393
- | 4 | `countDocuments` on 100k+ docs | Completes in O(1) network vs full scan |
394
- | 5 | Wrong `role` | Typed routing error |
395
- | 6 | `selfTest` with both roles configured | Both DBs reported |
396
-
397
- Retrieval integration test (downstream): `npm run smoke:retrieval -- --workspace-list` with `includeTotal: true`.
398
-
399
- ---
400
-
401
- ## 8. Version alignment
119
+ ## 6. Version alignment
402
120
 
403
121
  | Package | Current | Notes |
404
122
  |---------|---------|-------|
405
- | `@x12i/memorix-retrieval` | 1.5.x | Default data tier = Xronox |
406
- | `@x12i/xronox` | ^3.8.0 | Peer dependency; pin after count/skip land |
407
-
408
- **Suggested Xronox release:** minor bump (e.g. 3.8.0) with `countDocuments` + `skip` — no breaking changes to existing `read` callers.
409
-
410
- ---
411
-
412
- ## 9. Out of scope for Xronox
413
-
414
- These stay in **Catalox** / **memorix-retrieval**, not Xronox:
415
-
416
- - Entity discovery, list columns, field paths, filters allowlist
417
- - Row composition, multi-match, relations, content objects
418
- - Cross-entity workspace merge logic
419
- - Catalox seed / descriptor validation
420
-
421
- Xronox should remain a **database routing + I/O layer**, not Memorix-aware.
123
+ | `@x12i/memorix-retrieval` | 1.6.1 | Inventory via Xronox `listCollections` |
124
+ | `@x12i/xronox` | ^3.9.0 | `listCollections` required for inventory |
422
125
 
423
126
  ---
424
127
 
425
- ## 10. References in this repo
128
+ ## 7. Out of scope for Xronox
426
129
 
427
- | File | Purpose |
428
- |------|---------|
429
- | `src/data/memorix-read.ts` | Read routing + role resolution |
430
- | `src/data/memorix-count.ts` | Count entry point |
431
- | `src/client/xronox-adapter.ts` | Interim shims |
432
- | `src/client/create-xronox-from-env.ts` | Bootstrap |
433
- | `src/client/xronox-like.ts` | Minimal interface retrieval expects |
130
+ Entity discovery, list columns, row composition, Catalox seed, orphan name parsing — stay in Catalox / memorix-retrieval.
434
131
 
435
- **Tracking:** When implementing in `@x12i/xronox`, link PR to this doc and update §3 / §4 status tables.
132
+ Xronox remains a **database routing + I/O layer**, not Memorix-aware.
@@ -0,0 +1,15 @@
1
+ # Feature requests — `@x12i/xronox` (Memorix data tier)
2
+
3
+ **Audience:** `@x12i/xronox` maintainers
4
+ **Consumer:** `@x12i/memorix-retrieval`, Memorix Explorer, APIs, tools
5
+ **Policy:** Retrieval and hosts **must not** bypass Xronox with direct Mongo for production paths. Open FRs block retrieval features until Xronox ships the generic API.
6
+
7
+ | FR | Status | Blocks |
8
+ |----|--------|--------|
9
+ | [xronox-fr-list-collections.md](./xronox-fr-list-collections.md) | **Shipped** (`@x12i/xronox` 3.9.0) | — |
10
+ | [xronox-fr-memorix-env-preset.md](./xronox-fr-memorix-env-preset.md) | **Open** | Zero-config with `MEMORIX_*_DB` only |
11
+ | [xronox-fr-per-role-selftest.md](./xronox-fr-per-role-selftest.md) | **Open** | Health reporting both entity + event DBs |
12
+
13
+ **Already in Xronox 3.8+ (no FR):** `countDocuments`, `skip`, `sort` as Mongo object, `countDocuments({ estimated: true })`.
14
+
15
+ See also: [XRONOX-DATA-TIER-REQUIREMENTS.md](../XRONOX-DATA-TIER-REQUIREMENTS.md) (integration index).
@@ -0,0 +1,196 @@
1
+ # Feature request — routed `listCollections` (`@x12i/xronox`)
2
+
3
+ **Date:** 2026-05-22
4
+ **Status:** **Shipped** — `@x12i/xronox@3.9.0`
5
+ **Consumer:** `@x12i/memorix-retrieval@1.6.1+`
6
+
7
+ ---
8
+
9
+ ## Summary
10
+
11
+ Memorix **collection inventory** lists all Mongo collections in **both** Memorix databases (`memorix-entities`, `memorix-events`) using the same **role → database** routing as `read` and `countDocuments`. Implemented in Xronox 3.9.0; retrieval uses it with **no direct Mongo bypass**.
12
+
13
+ ---
14
+
15
+ ## Why this belongs in Xronox (not retrieval)
16
+
17
+ | Concern | Host/retrieval bypass problem |
18
+ |---------|-------------------------------|
19
+ | Role routing | Must use same `role` → DB resolution as reads (`memorix_entities`, `memorix_events`) |
20
+ | Multi-DB | Inventory scans **both** databases; routing must stay in Xronox |
21
+ | Contract | [DATA-TIER-CONTRACT.md](../DATA-TIER-CONTRACT.md): production I/O through Xronox only |
22
+ | Reuse | Any x12i app with role-based Mongo needs collection discovery, not only Memorix |
23
+
24
+ ---
25
+
26
+ ## Product requirement
27
+
28
+ Expose a routed API that returns collection names (and optional types) for the **database resolved from `role`**, without targeting a single collection in `source`.
29
+
30
+ Orphan scan flow (retrieval consumer):
31
+
32
+ 1. `listCollections({ role: "memorix_entities" })` → all non-system collections in entity DB
33
+ 2. `listCollections({ role: "memorix_events" })` → all non-system collections in event DB
34
+ 3. For each collection not in Catalox descriptor map: `countDocuments({ role, source: collectionName, estimated: true })` (already in Xronox 3.8+)
35
+
36
+ Declared slot counts continue to use descriptor-routed `countDocuments` via `countMemorixEntityContentTypeDocuments` (retrieval).
37
+
38
+ ---
39
+
40
+ ## Proposed API
41
+
42
+ ```ts
43
+ listCollections(params: ListCollectionsParams): Promise<ListCollectionsResult>;
44
+
45
+ export interface ListedCollection {
46
+ name: string;
47
+ type?: string; // "collection" | "view" | ...
48
+ }
49
+
50
+ export interface ListCollectionsResult {
51
+ /** Resolved database name after role + tenancy (same as countDocuments/read) */
52
+ database: string;
53
+ collections: ListedCollection[];
54
+ }
55
+
56
+ export interface ListCollectionsParams extends RoutingKey {
57
+ subSourceType: "mongo";
58
+ /**
59
+ * Binding match token only — not a collection name.
60
+ * Zero-config bindings with no `match` accept any source (e.g. `"_db"`).
61
+ */
62
+ source: string;
63
+ role?: string;
64
+ /** Forwarded to Mongo db.listCollections({ filter }) */
65
+ filter?: Record<string, unknown>;
66
+ /** Default true — omit names starting with `system.` */
67
+ excludeSystem?: boolean;
68
+ /** Default true — use driver nameOnly for lighter responses */
69
+ nameOnly?: boolean;
70
+ }
71
+ ```
72
+
73
+ Add to `Xronox` interface and `Engine` (nxMongo engine only; xronoxCore may omit like `countDocuments`).
74
+
75
+ ---
76
+
77
+ ## Algorithm (normative)
78
+
79
+ 1. `resolveBinding(config, params)` — **same as `countDocuments` and `read`**.
80
+ 2. Resolve `connection` → `SimpleMongoHelper`.
81
+ 3. Resolve `db` from binding + `params.role` (role overrides binding default db).
82
+ 4. `sanitizeDatabaseName(db)` — same as existing count path.
83
+ 5. `mongoDb.listCollections(filter, { nameOnly })`.
84
+ 6. If `excludeSystem !== false`, filter out `system.*` names.
85
+ 7. Return `{ database: db, collections }`.
86
+
87
+ **Do not** use `params.source` as a collection name. `source` exists only for binding selection.
88
+
89
+ ---
90
+
91
+ ## Example calls (Memorix)
92
+
93
+ ```ts
94
+ // Entity store
95
+ const entityDb = await xronox.listCollections({
96
+ dataType: "metadata",
97
+ sourceType: "database",
98
+ subSourceType: "mongo",
99
+ source: "_db",
100
+ role: "memorix_entities",
101
+ });
102
+
103
+ // Event store
104
+ const eventDb = await xronox.listCollections({
105
+ dataType: "metadata",
106
+ sourceType: "database",
107
+ subSourceType: "mongo",
108
+ source: "_db",
109
+ role: "memorix_events",
110
+ });
111
+
112
+ // Orphan count (existing API)
113
+ const count = await xronox.countDocuments({
114
+ dataType: "metadata",
115
+ sourceType: "database",
116
+ subSourceType: "mongo",
117
+ role: "memorix_entities",
118
+ source: "orphan-entity-records",
119
+ estimated: true,
120
+ });
121
+ ```
122
+
123
+ ---
124
+
125
+ ## Implementation guidance (`@x12i/xronox`)
126
+
127
+ | Layer | Change |
128
+ |-------|--------|
129
+ | `src/types.ts` | `ListCollectionsParams`, `ListCollectionsResult`, `ListedCollection`; extend `Xronox` |
130
+ | `src/engines/interface.ts` | `listCollections?(params, resolved)` |
131
+ | `src/engines/nxMongoEngine.ts` | Implement via `getDatabase` / `listCollections` (same helper lookup as `countDocuments`) |
132
+ | `src/index.ts` | `XronoxImpl.listCollections` → `resolveBinding` → engine |
133
+ | `README.md` | Document next to `countDocuments` example |
134
+ | Tests | Routing unit test (no Mongo); optional integration test with fixture DB |
135
+
136
+ Mirror error codes from count path: `CONNECTION_NOT_FOUND`, `DATABASE_NOT_RESOLVED`, `ROUTING_NOT_FOUND`.
137
+
138
+ ---
139
+
140
+ ## Retrieval integration (after Xronox ships)
141
+
142
+ | Step | Action |
143
+ |------|--------|
144
+ | 1 | Bump `@x12i/xronox` minimum (e.g. `^3.9.0`) |
145
+ | 2 | Extend `XronoxLike` + adapter to delegate `listCollections` |
146
+ | 3 | `buildMemorixCollectionInventory` calls Xronox only — **throws** if `listCollections` missing |
147
+ | 4 | Delete Explorer host shim `server/collection-inventory.ts` |
148
+ | 5 | `npm run smoke:retrieval -- --inventory` |
149
+
150
+ **No Mongo fallback in retrieval.** Tests may inject a mock `xronox` with `listCollections`; production requires real Xronox.
151
+
152
+ ---
153
+
154
+ ## Acceptance criteria
155
+
156
+ ### Xronox
157
+
158
+ - [ ] `listCollections({ role: "memorix_entities" })` returns collections from `MEMORIX_ENTITIES_DB` / role map default
159
+ - [ ] `listCollections({ role: "memorix_events" })` returns collections from events DB
160
+ - [ ] `excludeSystem: true` (default) omits `system.*`
161
+ - [ ] `database` in result matches DB used for `countDocuments` on same role + collection
162
+ - [ ] Wrong/missing role → typed routing error (not empty list)
163
+ - [ ] nxMongo engine only; clear error on xronoxCore
164
+
165
+ ### Retrieval (downstream)
166
+
167
+ - [ ] `buildMemorixCollectionInventory` completes without `connectMemorixMongo`
168
+ - [ ] Inventory `summary.byTarget` matches entries filtered by `target`
169
+ - [ ] Vitest mocks `xronox.listCollections` only (no Mongo client for scan)
170
+
171
+ ---
172
+
173
+ ## Non-goals
174
+
175
+ | Item | Reason |
176
+ |------|--------|
177
+ | Catalox orphan → descriptor seeding | Ops / Catalox tooling |
178
+ | Memorix-specific parsing in Xronox | Stays in retrieval (`parseOrphanCollection`) |
179
+ | List documents across all collections | Use inventory + per-collection `read` |
180
+
181
+ ---
182
+
183
+ ## Version
184
+
185
+ | Package | Minimum after FR |
186
+ |---------|------------------|
187
+ | `@x12i/xronox` | `3.9.0` (proposed minor) |
188
+ | `@x12i/memorix-retrieval` | Requires published Xronox with this FR |
189
+
190
+ ---
191
+
192
+ ## References
193
+
194
+ - Retrieval inventory: `src/explorer/collection-inventory.ts`
195
+ - Role constants: `src/data/memorix-read.ts` (`DEFAULT_MEMORIX_XRONOX_ROLES`)
196
+ - Explorer shim (delete after ship): `memorix-explorer/server/collection-inventory.ts`