@x12i/static-memorix-explorer-api 1.2.0 → 1.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +52 -2
- package/dist/config.d.ts +9 -0
- package/dist/config.js +36 -2
- package/dist/engine/inventory.js +4 -4
- package/dist/engine/lists.d.ts +1 -1
- package/dist/engine/lists.js +2 -6
- package/dist/engine/narratives.js +6 -16
- package/dist/engine/write.js +2 -6
- package/dist/routes/index.js +9 -2
- package/dist/server.d.ts +1 -0
- package/dist/server.js +12 -3
- package/dist/storage/InMemoryStore.d.ts +2 -0
- package/dist/storage/InMemoryStore.js +70 -16
- package/dist/storage/fs.js +19 -4
- package/guides/demo-app.md +6 -3
- package/guides/managing-json-files.md +74 -0
- package/guides/running-the-service.md +26 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -121,7 +121,9 @@ The mock server accepts the following identity keys across fetch/write engines:
|
|
|
121
121
|
- `memoryId` — memory
|
|
122
122
|
|
|
123
123
|
Endpoints like `/records/full` and `/records/content` require exactly one identity key
|
|
124
|
-
per request. Memory collections are keyed by `memoryId
|
|
124
|
+
per request. Memory collections are keyed by `memoryId`; the unprefixed
|
|
125
|
+
`./mocks/data/memory/memory.json` is the seed and routed writes use the active
|
|
126
|
+
Catalox-prefixed filename.
|
|
125
127
|
|
|
126
128
|
## Snapshot alias translation
|
|
127
129
|
|
|
@@ -184,7 +186,7 @@ Full implementation walkthrough: [guides/demo-app.md](./guides/demo-app.md).
|
|
|
184
186
|
|-----|---------|-------|
|
|
185
187
|
| `PORT` | `5030` | Listen port |
|
|
186
188
|
| `HOST` | `0.0.0.0` | Listen host |
|
|
187
|
-
| `MOCKS_DIR` | `<
|
|
189
|
+
| `MOCKS_DIR` | bundled `<package>/mocks` | Fixture root |
|
|
188
190
|
| `DISK_FLUSH_DEBOUNCE_MS` | `300` | Debounce before disk flush |
|
|
189
191
|
| `MEMORIX_ORG_ID` | `memorix` | Organization prefix for DB routing |
|
|
190
192
|
| `MEMORIX_AGENT_ID` | `default-agent` | Agent prefix for catalox DB routing |
|
|
@@ -193,6 +195,11 @@ Full implementation walkthrough: [guides/demo-app.md](./guides/demo-app.md).
|
|
|
193
195
|
| `MEMORIX_EXPLORER_ENABLE_PIPELINE_WRITES` | `false` | Allow pipeline writes |
|
|
194
196
|
| `MEMORIX_EXPLORER_ENABLE_REGISTRY_WRITES` | `false` | Allow registry writes |
|
|
195
197
|
|
|
198
|
+
For mutation-capable installed deployments, set `MOCKS_DIR` to a writable,
|
|
199
|
+
persistent directory. Routing configuration is process-scoped and resolved on
|
|
200
|
+
first import; use one process per organization/agent context and one writer per
|
|
201
|
+
prefix.
|
|
202
|
+
|
|
196
203
|
## DB Router Simulation
|
|
197
204
|
|
|
198
205
|
The mock server does not connect to MongoDB, but it **tricks the frontend** into
|
|
@@ -207,6 +214,28 @@ These strings are derived from `MEMORIX_ORG_ID`, `MEMORIX_AGENT_ID`, and the
|
|
|
207
214
|
`MEMORIX_DEPLOYMENT_PROFILE` (which, when set to `ebook`, forces both IDs to
|
|
208
215
|
`ebooks`).
|
|
209
216
|
|
|
217
|
+
### File-backed routing
|
|
218
|
+
|
|
219
|
+
Those routed database names also select JSON files. The separator is `--`:
|
|
220
|
+
|
|
221
|
+
```text
|
|
222
|
+
mocks/data/product/neo-memorix-entities--snapshots.json
|
|
223
|
+
mocks/data/product/neo-memorix-events--events.json
|
|
224
|
+
mocks/data/memory/neo-agent-memorix-catalox--memory.json
|
|
225
|
+
mocks/metadata/lists/neo-agent-memorix-catalox--my-plate.json
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
For `MEMORIX_ORG_ID=neo` and `MEMORIX_AGENT_ID=neo-agent`, entity-like
|
|
229
|
+
collections use the first prefix, events use the second, and memory plus
|
|
230
|
+
metadata use the Catalox prefix. If a routed file does not exist, the server
|
|
231
|
+
loads its unprefixed counterpart as seed data, but all mutations are written
|
|
232
|
+
to the routed filename. This provides database-style isolation without a
|
|
233
|
+
database and preserves existing fixtures.
|
|
234
|
+
|
|
235
|
+
The health response exposes the resolved values under `fileRouting`.
|
|
236
|
+
See [guides/managing-json-files.md](./guides/managing-json-files.md) for the
|
|
237
|
+
complete precedence and migration rules.
|
|
238
|
+
|
|
210
239
|
## Feature Flag Enforcement
|
|
211
240
|
|
|
212
241
|
The mock server enforces the following feature-flag gates with error responses:
|
|
@@ -231,6 +260,8 @@ The mock's querying and writing engines now support `target=memory` and
|
|
|
231
260
|
and `knowledgeId`.
|
|
232
261
|
|
|
233
262
|
- Folder structure: `./mocks/data/memory/memory.json` (seeded with example memories).
|
|
263
|
+
- Runtime memory writes persist to the active
|
|
264
|
+
`<agent>-memorix-catalox--memory.json` route.
|
|
234
265
|
- Inventory engine accepts `target=memory` and includes memory collections in
|
|
235
266
|
the simulated `inventorySummary`.
|
|
236
267
|
- Records engine accepts `target=memory` and supports `memoryId` for fetching.
|
|
@@ -247,6 +278,25 @@ and `knowledgeId`.
|
|
|
247
278
|
|
|
248
279
|
## Release history
|
|
249
280
|
|
|
281
|
+
### v1.3.1
|
|
282
|
+
- Rejected unsafe routing IDs, object types, and metadata keys before path use.
|
|
283
|
+
- Changed malformed and empty JSON handling from silent fallback to explicit
|
|
284
|
+
startup failure, preventing accidental data loss.
|
|
285
|
+
- Made JSON persistence atomic through same-directory temporary-file renames.
|
|
286
|
+
- Made list deletion restart-safe with routed tombstones.
|
|
287
|
+
- Fixed installed npm CLI execution through symlink-aware entry detection.
|
|
288
|
+
- Added packed-install, restart, traversal, corruption, and atomic-write checks.
|
|
289
|
+
- Documented writable `MOCKS_DIR`, process isolation, and single-writer limits.
|
|
290
|
+
|
|
291
|
+
### v1.3.0
|
|
292
|
+
- Added database-routing parity through deterministic JSON filename prefixes.
|
|
293
|
+
- Added separate entity, event, and Catalox/metadata prefix resolution.
|
|
294
|
+
- Added migration-safe fallback from missing routed files to unprefixed seeds;
|
|
295
|
+
writes always persist to the selected routed file.
|
|
296
|
+
- Added resolved `fileRouting` prefixes to the health response.
|
|
297
|
+
- Updated metadata discovery to ignore other routed tenants.
|
|
298
|
+
- Added routed persistence and prefix regression coverage.
|
|
299
|
+
|
|
250
300
|
### v1.2.0
|
|
251
301
|
- Added native `--port` and `--host` CLI options.
|
|
252
302
|
- Added the canonical `static-memorix-explorer-api` executable while retaining
|
package/dist/config.d.ts
CHANGED
|
@@ -5,6 +5,7 @@ export declare const DATA_DIR: string;
|
|
|
5
5
|
export declare const PORT: number;
|
|
6
6
|
export declare const HOST: string;
|
|
7
7
|
export declare const DISK_FLUSH_DEBOUNCE_MS: number;
|
|
8
|
+
export declare function validateRouteId(value: string, variableName?: string): string;
|
|
8
9
|
export declare const MEMORIX_ORG_ID: string;
|
|
9
10
|
export declare const MEMORIX_AGENT_ID: string;
|
|
10
11
|
export declare const MEMORIX_DEPLOYMENT_PROFILE: string;
|
|
@@ -18,6 +19,14 @@ export declare function buildMemorixDbString(): string;
|
|
|
18
19
|
* e.g. "ebooks-memorix-catalox"
|
|
19
20
|
*/
|
|
20
21
|
export declare function buildCataloxDbString(): string;
|
|
22
|
+
/**
|
|
23
|
+
* Files emulate routed databases by using the resolved database name as a
|
|
24
|
+
* prefix. The `--` separator is reserved for routing and must not be used in
|
|
25
|
+
* an unprefixed fixture basename.
|
|
26
|
+
*/
|
|
27
|
+
export declare function buildDataFilePrefix(contentType: string): string;
|
|
28
|
+
export declare function buildMetadataFilePrefix(): string;
|
|
29
|
+
export declare function prefixedJsonFilename(prefix: string, basename: string): string;
|
|
21
30
|
export declare const FLAGS: {
|
|
22
31
|
METADATA_WRITES: boolean;
|
|
23
32
|
PIPELINE_WRITES: boolean;
|
package/dist/config.js
CHANGED
|
@@ -20,13 +20,23 @@ function resolveOrgId() {
|
|
|
20
20
|
if ((process.env.MEMORIX_DEPLOYMENT_PROFILE || "").toLowerCase() === "ebook") {
|
|
21
21
|
return "ebooks";
|
|
22
22
|
}
|
|
23
|
-
return process.env.MEMORIX_ORG_ID || "memorix";
|
|
23
|
+
return validateRouteId(process.env.MEMORIX_ORG_ID || "memorix", "MEMORIX_ORG_ID");
|
|
24
24
|
}
|
|
25
25
|
function resolveAgentId() {
|
|
26
26
|
if ((process.env.MEMORIX_DEPLOYMENT_PROFILE || "").toLowerCase() === "ebook") {
|
|
27
27
|
return "ebooks";
|
|
28
28
|
}
|
|
29
|
-
return process.env.MEMORIX_AGENT_ID || "default-agent";
|
|
29
|
+
return validateRouteId(process.env.MEMORIX_AGENT_ID || "default-agent", "MEMORIX_AGENT_ID");
|
|
30
|
+
}
|
|
31
|
+
export function validateRouteId(value, variableName = "route ID") {
|
|
32
|
+
if (!/^[A-Za-z0-9][A-Za-z0-9._-]*$/.test(value) ||
|
|
33
|
+
value.includes("--") ||
|
|
34
|
+
value === "." ||
|
|
35
|
+
value === "..") {
|
|
36
|
+
throw Object.assign(new Error(`${variableName} must contain only letters, numbers, dot, underscore, or hyphen; ` +
|
|
37
|
+
`it must not contain path separators or the reserved "--" delimiter`), { statusCode: 400 });
|
|
38
|
+
}
|
|
39
|
+
return value;
|
|
30
40
|
}
|
|
31
41
|
export const MEMORIX_ORG_ID = resolveOrgId();
|
|
32
42
|
export const MEMORIX_AGENT_ID = resolveAgentId();
|
|
@@ -46,6 +56,30 @@ export function buildMemorixDbString() {
|
|
|
46
56
|
export function buildCataloxDbString() {
|
|
47
57
|
return `${MEMORIX_AGENT_ID}-memorix-catalox`;
|
|
48
58
|
}
|
|
59
|
+
/**
|
|
60
|
+
* Files emulate routed databases by using the resolved database name as a
|
|
61
|
+
* prefix. The `--` separator is reserved for routing and must not be used in
|
|
62
|
+
* an unprefixed fixture basename.
|
|
63
|
+
*/
|
|
64
|
+
export function buildDataFilePrefix(contentType) {
|
|
65
|
+
if (contentType === "events")
|
|
66
|
+
return `${MEMORIX_ORG_ID}-memorix-events`;
|
|
67
|
+
if (contentType === "memory")
|
|
68
|
+
return buildCataloxDbString();
|
|
69
|
+
return `${MEMORIX_ORG_ID}-memorix-entities`;
|
|
70
|
+
}
|
|
71
|
+
export function buildMetadataFilePrefix() {
|
|
72
|
+
return buildCataloxDbString();
|
|
73
|
+
}
|
|
74
|
+
export function prefixedJsonFilename(prefix, basename) {
|
|
75
|
+
validateRouteId(prefix, "file routing prefix");
|
|
76
|
+
if (!/^[A-Za-z0-9][A-Za-z0-9._-]*$/.test(basename) || basename.includes("--")) {
|
|
77
|
+
throw Object.assign(new Error(`Unsafe JSON fixture basename: ${basename}`), {
|
|
78
|
+
statusCode: 400,
|
|
79
|
+
});
|
|
80
|
+
}
|
|
81
|
+
return `${prefix}--${basename}.json`;
|
|
82
|
+
}
|
|
49
83
|
// ---------------------------------------------------------------------------
|
|
50
84
|
// Feature Flag Enforcement
|
|
51
85
|
// ---------------------------------------------------------------------------
|
package/dist/engine/inventory.js
CHANGED
|
@@ -1,13 +1,13 @@
|
|
|
1
1
|
import path from "node:path";
|
|
2
2
|
import { store } from "../storage/InMemoryStore.js";
|
|
3
3
|
import { CONTENT_TYPES } from "../types.js";
|
|
4
|
-
import {
|
|
5
|
-
import { METADATA_DIR, DATA_DIR } from "../config.js";
|
|
4
|
+
import { DATA_DIR } from "../config.js";
|
|
6
5
|
import fs from "node:fs";
|
|
7
6
|
import { isDir } from "../storage/fs.js";
|
|
8
7
|
export function listObjectTypes() {
|
|
9
|
-
|
|
10
|
-
|
|
8
|
+
return store.getMetaKeys()
|
|
9
|
+
.filter((key) => key.startsWith("object-types/"))
|
|
10
|
+
.map((key) => key.slice("object-types/".length));
|
|
11
11
|
}
|
|
12
12
|
export function listDataObjectTypes() {
|
|
13
13
|
if (!isDir(DATA_DIR))
|
package/dist/engine/lists.d.ts
CHANGED
|
@@ -11,7 +11,7 @@ export declare function executeListDescriptor(desc: ListDescriptor, params?: Exp
|
|
|
11
11
|
export declare function suggestListExtensions(entityName: string): {
|
|
12
12
|
entityName: string;
|
|
13
13
|
suggestions: {
|
|
14
|
-
contentType: "
|
|
14
|
+
contentType: "events" | "memory" | "analysis" | "decisions";
|
|
15
15
|
collection: string;
|
|
16
16
|
suggestion: string;
|
|
17
17
|
}[];
|
package/dist/engine/lists.js
CHANGED
|
@@ -1,14 +1,10 @@
|
|
|
1
|
-
import path from "node:path";
|
|
2
1
|
import { store } from "../storage/InMemoryStore.js";
|
|
3
2
|
import { CONTENT_TYPES } from "../types.js";
|
|
4
|
-
import { listJsonFiles, basenameNoExt } from "../storage/fs.js";
|
|
5
|
-
import { METADATA_DIR } from "../config.js";
|
|
6
3
|
import { queryMockCollection } from "../engine/query.js";
|
|
7
4
|
export function listListDescriptors(entityName) {
|
|
8
|
-
const dir = path.join(METADATA_DIR, "lists");
|
|
9
5
|
const out = [];
|
|
10
|
-
for (const
|
|
11
|
-
const desc = store.getMeta(
|
|
6
|
+
for (const key of store.getMetaKeys().filter((item) => item.startsWith("lists/"))) {
|
|
7
|
+
const desc = store.getMeta(key);
|
|
12
8
|
if (desc && (!entityName || desc.entity === entityName || desc.entityName === entityName)) {
|
|
13
9
|
out.push(desc);
|
|
14
10
|
}
|
|
@@ -1,10 +1,5 @@
|
|
|
1
|
-
import path from "node:path";
|
|
2
1
|
import { store } from "../storage/InMemoryStore.js";
|
|
3
2
|
import { CONTENT_TYPES } from "../types.js";
|
|
4
|
-
import { listJsonFiles, basenameNoExt, readJson } from "../storage/fs.js";
|
|
5
|
-
import { METADATA_DIR } from "../config.js";
|
|
6
|
-
import { isDir } from "../storage/fs.js";
|
|
7
|
-
import fs from "node:fs";
|
|
8
3
|
/**
|
|
9
4
|
* Discover narrative keys tagged on records via doc.narratives.*.
|
|
10
5
|
*/
|
|
@@ -40,11 +35,10 @@ export function getMergedNarratives(entityName) {
|
|
|
40
35
|
return mergeCatalogWithSignals(authored, recordTags);
|
|
41
36
|
}
|
|
42
37
|
// global across all entities
|
|
43
|
-
const dir = path.join(METADATA_DIR, "narratives");
|
|
44
38
|
const out = {};
|
|
45
|
-
for (const
|
|
46
|
-
const name =
|
|
47
|
-
const authored =
|
|
39
|
+
for (const metaKey of store.getMetaKeys().filter((key) => key.startsWith("narratives/"))) {
|
|
40
|
+
const name = metaKey.slice("narratives/".length);
|
|
41
|
+
const authored = store.getMeta(metaKey) || {};
|
|
48
42
|
const recordTags = discoverRecordNarrativeTags(name);
|
|
49
43
|
out[name] = mergeCatalogWithSignals(authored, recordTags);
|
|
50
44
|
}
|
|
@@ -67,11 +61,7 @@ export function queryNarrativeRecords(entityName, key) {
|
|
|
67
61
|
return results;
|
|
68
62
|
}
|
|
69
63
|
export function listNarrativeEntities() {
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
return fs
|
|
74
|
-
.readdirSync(dir)
|
|
75
|
-
.filter((f) => f.endsWith(".json"))
|
|
76
|
-
.map((f) => basenameNoExt(f));
|
|
64
|
+
return store.getMetaKeys()
|
|
65
|
+
.filter((key) => key.startsWith("narratives/"))
|
|
66
|
+
.map((key) => key.slice("narratives/".length));
|
|
77
67
|
}
|
package/dist/engine/write.js
CHANGED
|
@@ -1,7 +1,4 @@
|
|
|
1
|
-
import path from "node:path";
|
|
2
1
|
import { store } from "../storage/InMemoryStore.js";
|
|
3
|
-
import { basenameNoExt, listJsonFiles } from "../storage/fs.js";
|
|
4
|
-
import { METADATA_DIR } from "../config.js";
|
|
5
2
|
export function loadWriteDescriptor(writeDescriptorId) {
|
|
6
3
|
const desc = store.getMeta(`write-descriptors/${writeDescriptorId}`);
|
|
7
4
|
if (!desc) {
|
|
@@ -156,10 +153,9 @@ function getNextId(collection, idField = "recordId") {
|
|
|
156
153
|
return () => String(counter++);
|
|
157
154
|
}
|
|
158
155
|
export function listWriteDescriptors() {
|
|
159
|
-
const dir = path.join(METADATA_DIR, "write-descriptors");
|
|
160
156
|
const out = [];
|
|
161
|
-
for (const
|
|
162
|
-
const desc = store.getMeta(
|
|
157
|
+
for (const key of store.getMetaKeys().filter((item) => item.startsWith("write-descriptors/"))) {
|
|
158
|
+
const desc = store.getMeta(key);
|
|
163
159
|
if (desc)
|
|
164
160
|
out.push(desc);
|
|
165
161
|
}
|
package/dist/routes/index.js
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
import { store } from "../storage/InMemoryStore.js";
|
|
2
|
-
import { buildMemorixDbString, buildCataloxDbString, metadataWritesEnabled, } from "../config.js";
|
|
2
|
+
import { buildMemorixDbString, buildCataloxDbString, buildDataFilePrefix, buildMetadataFilePrefix, metadataWritesEnabled, } from "../config.js";
|
|
3
3
|
import { listObjectTypes } from "../engine/inventory.js";
|
|
4
4
|
import { computeInventory, computeInventorySummary, computeInventoryIssues, computeGraph, } from "../engine/inventory.js";
|
|
5
5
|
import { qp, str } from "./helpers.js";
|
|
@@ -25,6 +25,12 @@ export async function registerRoutes(app) {
|
|
|
25
25
|
discoverySample: objectTypes,
|
|
26
26
|
memorixDb: buildMemorixDbString(),
|
|
27
27
|
cataloxDb: buildCataloxDbString(),
|
|
28
|
+
fileRouting: {
|
|
29
|
+
entitiesPrefix: buildDataFilePrefix("snapshots"),
|
|
30
|
+
eventsPrefix: buildDataFilePrefix("events"),
|
|
31
|
+
memoryPrefix: buildDataFilePrefix("memory"),
|
|
32
|
+
metadataPrefix: buildMetadataFilePrefix(),
|
|
33
|
+
},
|
|
28
34
|
};
|
|
29
35
|
if (includeInventory) {
|
|
30
36
|
const rows = computeInventory({ sourceLens: "catalox-first" });
|
|
@@ -222,7 +228,8 @@ export async function registerRoutes(app) {
|
|
|
222
228
|
return { ok: false, error: "Metadata writes are disabled" };
|
|
223
229
|
}
|
|
224
230
|
const { listId } = request.params;
|
|
225
|
-
store.
|
|
231
|
+
store.deleteMeta(`lists/${listId}`);
|
|
232
|
+
await store.flushAll();
|
|
226
233
|
return { ok: true, deleted: listId };
|
|
227
234
|
});
|
|
228
235
|
// ---------- F. Narratives Engine ----------
|
package/dist/server.d.ts
CHANGED
|
@@ -6,6 +6,7 @@ export interface StartServerOptions {
|
|
|
6
6
|
installSignalHandlers?: boolean;
|
|
7
7
|
}
|
|
8
8
|
export declare function parseCliArgs(args: string[]): StartServerOptions;
|
|
9
|
+
export declare function isCliEntrypoint(argvPath: string | undefined, moduleUrl: string): boolean;
|
|
9
10
|
/** Build a loaded Fastify instance without opening a network port. */
|
|
10
11
|
export declare function buildServer(options?: Pick<StartServerOptions, "logger">): Promise<import("fastify").FastifyInstance<import("http").Server<typeof import("http").IncomingMessage, typeof import("http").ServerResponse>, import("http").IncomingMessage, import("http").ServerResponse<import("http").IncomingMessage>, import("fastify").FastifyBaseLogger, import("fastify").FastifyTypeProviderDefault>>;
|
|
11
12
|
/** Build and listen. Options override PORT/HOST environment configuration. */
|
package/dist/server.js
CHANGED
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
2
|
import Fastify from "fastify";
|
|
3
|
+
import fs from "node:fs";
|
|
3
4
|
import path from "node:path";
|
|
4
5
|
import { fileURLToPath } from "node:url";
|
|
5
6
|
import { store } from "./storage/InMemoryStore.js";
|
|
@@ -31,6 +32,16 @@ export function parseCliArgs(args) {
|
|
|
31
32
|
}
|
|
32
33
|
return options;
|
|
33
34
|
}
|
|
35
|
+
export function isCliEntrypoint(argvPath, moduleUrl) {
|
|
36
|
+
if (!argvPath)
|
|
37
|
+
return false;
|
|
38
|
+
try {
|
|
39
|
+
return fs.realpathSync(path.resolve(argvPath)) === fileURLToPath(moduleUrl);
|
|
40
|
+
}
|
|
41
|
+
catch {
|
|
42
|
+
return false;
|
|
43
|
+
}
|
|
44
|
+
}
|
|
34
45
|
/** Build a loaded Fastify instance without opening a network port. */
|
|
35
46
|
export async function buildServer(options = {}) {
|
|
36
47
|
const app = Fastify({ logger: options.logger ?? false });
|
|
@@ -74,9 +85,7 @@ export async function startServer(options = {}) {
|
|
|
74
85
|
}
|
|
75
86
|
return app;
|
|
76
87
|
}
|
|
77
|
-
const isCliEntry = process.argv[1]
|
|
78
|
-
? path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)
|
|
79
|
-
: false;
|
|
88
|
+
const isCliEntry = isCliEntrypoint(process.argv[1], import.meta.url);
|
|
80
89
|
if (isCliEntry) {
|
|
81
90
|
Promise.resolve().then(() => startServer(parseCliArgs(process.argv.slice(2)))).catch((err) => {
|
|
82
91
|
// eslint-disable-next-line no-console
|
|
@@ -12,6 +12,8 @@ export declare class InMemoryStore {
|
|
|
12
12
|
private loadDataCollections;
|
|
13
13
|
getMeta<T = any>(key: string): T | undefined;
|
|
14
14
|
setMeta(key: string, data: any, persist?: boolean): void;
|
|
15
|
+
/** Persist a routed tombstone so an unprefixed seed does not reappear. */
|
|
16
|
+
deleteMeta(key: string): void;
|
|
15
17
|
getMetaKeys(): string[];
|
|
16
18
|
getCollection(objectType: string, ct: ContentType): any[];
|
|
17
19
|
setCollection(objectType: string, ct: ContentType, data: any[]): void;
|
|
@@ -1,11 +1,28 @@
|
|
|
1
1
|
import fs from "node:fs";
|
|
2
2
|
import path from "node:path";
|
|
3
3
|
import { CONTENT_TYPES } from "../types.js";
|
|
4
|
-
import { MOCKS_DIR, METADATA_DIR, DATA_DIR } from "../config.js";
|
|
4
|
+
import { MOCKS_DIR, METADATA_DIR, DATA_DIR, buildDataFilePrefix, buildMetadataFilePrefix, prefixedJsonFilename, validateRouteId, } from "../config.js";
|
|
5
5
|
import { validateRecordArray } from "../engine/identity.js";
|
|
6
6
|
import { isDir, listJsonFiles, readJson, writeJson, basenameNoExt, } from "./fs.js";
|
|
7
7
|
import { DISK_FLUSH_DEBOUNCE_MS } from "../config.js";
|
|
8
8
|
const DIRTY = Symbol("dirty");
|
|
9
|
+
function routedMetadataPath(key) {
|
|
10
|
+
const parts = key.split("/");
|
|
11
|
+
if (parts.length === 1 && parts[0] === "agents") {
|
|
12
|
+
return path.join(METADATA_DIR, prefixedJsonFilename(buildMetadataFilePrefix(), "agents"));
|
|
13
|
+
}
|
|
14
|
+
if (parts.length !== 2) {
|
|
15
|
+
throw Object.assign(new Error(`Unsafe metadata key: ${key}`), { statusCode: 400 });
|
|
16
|
+
}
|
|
17
|
+
const [section, name] = parts;
|
|
18
|
+
if (!["object-types", "lists", "write-descriptors", "narratives"].includes(section)) {
|
|
19
|
+
throw Object.assign(new Error(`Unsupported metadata section: ${section}`), {
|
|
20
|
+
statusCode: 400,
|
|
21
|
+
});
|
|
22
|
+
}
|
|
23
|
+
validateRouteId(name, "metadata fixture name");
|
|
24
|
+
return path.join(METADATA_DIR, section, prefixedJsonFilename(buildMetadataFilePrefix(), name));
|
|
25
|
+
}
|
|
9
26
|
export class InMemoryStore {
|
|
10
27
|
collections = new Map(); // key: `${objectType}/${contentType}`
|
|
11
28
|
metadata = new Map();
|
|
@@ -28,18 +45,33 @@ export class InMemoryStore {
|
|
|
28
45
|
"write-descriptors": "write-descriptors",
|
|
29
46
|
narratives: "narratives",
|
|
30
47
|
};
|
|
48
|
+
const prefix = buildMetadataFilePrefix();
|
|
31
49
|
for (const [subdir] of Object.entries(sections)) {
|
|
32
50
|
const dir = path.join(METADATA_DIR, subdir);
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
51
|
+
const files = listJsonFiles(dir);
|
|
52
|
+
const logicalNames = new Set();
|
|
53
|
+
for (const file of files) {
|
|
54
|
+
const name = basenameNoExt(file);
|
|
55
|
+
if (!name.includes("--"))
|
|
56
|
+
logicalNames.add(name);
|
|
57
|
+
if (name.startsWith(`${prefix}--`))
|
|
58
|
+
logicalNames.add(name.slice(prefix.length + 2));
|
|
59
|
+
}
|
|
60
|
+
for (const name of logicalNames) {
|
|
61
|
+
const baseFile = path.join(dir, `${name}.json`);
|
|
62
|
+
const routedFile = path.join(dir, prefixedJsonFilename(prefix, name));
|
|
63
|
+
const source = fs.existsSync(routedFile) ? routedFile : baseFile;
|
|
64
|
+
const key = `${subdir}/${name}`;
|
|
65
|
+
this.metadata.set(key, { path: routedFile, data: readJson(source, {}) });
|
|
36
66
|
}
|
|
37
67
|
}
|
|
38
|
-
const
|
|
39
|
-
|
|
68
|
+
const baseAgentsFile = path.join(METADATA_DIR, "agents.json");
|
|
69
|
+
const routedAgentsFile = path.join(METADATA_DIR, prefixedJsonFilename(prefix, "agents"));
|
|
70
|
+
const agentsSource = fs.existsSync(routedAgentsFile) ? routedAgentsFile : baseAgentsFile;
|
|
71
|
+
if (fs.existsSync(agentsSource)) {
|
|
40
72
|
this.metadata.set("agents", {
|
|
41
|
-
path:
|
|
42
|
-
data: readJson(
|
|
73
|
+
path: routedAgentsFile,
|
|
74
|
+
data: readJson(agentsSource, { agents: [] }),
|
|
43
75
|
});
|
|
44
76
|
}
|
|
45
77
|
}
|
|
@@ -51,9 +83,11 @@ export class InMemoryStore {
|
|
|
51
83
|
if (!isDir(otDir))
|
|
52
84
|
continue;
|
|
53
85
|
for (const ct of CONTENT_TYPES) {
|
|
54
|
-
const
|
|
86
|
+
const baseFile = path.join(otDir, `${ct}.json`);
|
|
87
|
+
const file = path.join(otDir, prefixedJsonFilename(buildDataFilePrefix(ct), ct));
|
|
55
88
|
const key = `${objectType}/${ct}`;
|
|
56
|
-
const
|
|
89
|
+
const source = fs.existsSync(file) ? file : baseFile;
|
|
90
|
+
const data = readJson(source, []);
|
|
57
91
|
// Startup validation: ensure records carry exactly one identity key
|
|
58
92
|
// (MemorixRecordArraySchema mock). Memory collections use memoryId.
|
|
59
93
|
const result = validateRecordArray(objectType, ct, data);
|
|
@@ -68,11 +102,13 @@ export class InMemoryStore {
|
|
|
68
102
|
});
|
|
69
103
|
}
|
|
70
104
|
// system overrides
|
|
71
|
-
const
|
|
72
|
-
|
|
105
|
+
const baseInvFile = path.join(DATA_DIR, "system", "inventory.json");
|
|
106
|
+
const invFile = path.join(DATA_DIR, "system", prefixedJsonFilename(buildDataFilePrefix("snapshots"), "inventory"));
|
|
107
|
+
const invSource = fs.existsSync(invFile) ? invFile : baseInvFile;
|
|
108
|
+
if (objectType === "system" && fs.existsSync(invSource)) {
|
|
73
109
|
this.collections.set("system/inventory", {
|
|
74
110
|
path: invFile,
|
|
75
|
-
data: readJson(
|
|
111
|
+
data: readJson(invSource, []),
|
|
76
112
|
});
|
|
77
113
|
}
|
|
78
114
|
}
|
|
@@ -89,7 +125,7 @@ export class InMemoryStore {
|
|
|
89
125
|
}
|
|
90
126
|
else {
|
|
91
127
|
this.metadata.set(key, {
|
|
92
|
-
path:
|
|
128
|
+
path: routedMetadataPath(key),
|
|
93
129
|
data,
|
|
94
130
|
[DIRTY]: true,
|
|
95
131
|
});
|
|
@@ -97,16 +133,33 @@ export class InMemoryStore {
|
|
|
97
133
|
if (persist)
|
|
98
134
|
this.scheduleMetaFlush(key);
|
|
99
135
|
}
|
|
136
|
+
/** Persist a routed tombstone so an unprefixed seed does not reappear. */
|
|
137
|
+
deleteMeta(key) {
|
|
138
|
+
const existing = this.metadata.get(key);
|
|
139
|
+
if (existing) {
|
|
140
|
+
existing.data = null;
|
|
141
|
+
existing[DIRTY] = true;
|
|
142
|
+
}
|
|
143
|
+
else {
|
|
144
|
+
this.metadata.set(key, {
|
|
145
|
+
path: routedMetadataPath(key),
|
|
146
|
+
data: null,
|
|
147
|
+
[DIRTY]: true,
|
|
148
|
+
});
|
|
149
|
+
}
|
|
150
|
+
this.scheduleMetaFlush(key);
|
|
151
|
+
}
|
|
100
152
|
getMetaKeys() {
|
|
101
153
|
return [...this.metadata.keys()];
|
|
102
154
|
}
|
|
103
155
|
// ---- Collection access ----
|
|
104
156
|
getCollection(objectType, ct) {
|
|
157
|
+
validateRouteId(objectType, "object type");
|
|
105
158
|
const key = `${objectType}/${ct}`;
|
|
106
159
|
const c = this.collections.get(key);
|
|
107
160
|
if (!c) {
|
|
108
161
|
const newC = {
|
|
109
|
-
path: path.join(DATA_DIR, objectType,
|
|
162
|
+
path: path.join(DATA_DIR, objectType, prefixedJsonFilename(buildDataFilePrefix(ct), ct)),
|
|
110
163
|
data: [],
|
|
111
164
|
};
|
|
112
165
|
this.collections.set(key, newC);
|
|
@@ -115,6 +168,7 @@ export class InMemoryStore {
|
|
|
115
168
|
return c.data;
|
|
116
169
|
}
|
|
117
170
|
setCollection(objectType, ct, data) {
|
|
171
|
+
validateRouteId(objectType, "object type");
|
|
118
172
|
const key = `${objectType}/${ct}`;
|
|
119
173
|
const c = this.collections.get(key);
|
|
120
174
|
if (c) {
|
|
@@ -123,7 +177,7 @@ export class InMemoryStore {
|
|
|
123
177
|
}
|
|
124
178
|
else {
|
|
125
179
|
this.collections.set(key, {
|
|
126
|
-
path: path.join(DATA_DIR, objectType,
|
|
180
|
+
path: path.join(DATA_DIR, objectType, prefixedJsonFilename(buildDataFilePrefix(ct), ct)),
|
|
127
181
|
data,
|
|
128
182
|
[DIRTY]: true,
|
|
129
183
|
});
|
package/dist/storage/fs.js
CHANGED
|
@@ -15,16 +15,31 @@ export function readJson(p, fallback) {
|
|
|
15
15
|
try {
|
|
16
16
|
const raw = fs.readFileSync(p, "utf8");
|
|
17
17
|
if (!raw.trim())
|
|
18
|
-
|
|
18
|
+
throw new Error("file is empty");
|
|
19
19
|
return JSON.parse(raw);
|
|
20
20
|
}
|
|
21
|
-
catch {
|
|
22
|
-
|
|
21
|
+
catch (error) {
|
|
22
|
+
if (error.code === "ENOENT")
|
|
23
|
+
return fallback;
|
|
24
|
+
const message = error instanceof Error ? error.message : String(error);
|
|
25
|
+
throw new Error(`Failed to read JSON fixture ${p}: ${message}`);
|
|
23
26
|
}
|
|
24
27
|
}
|
|
25
28
|
export function writeJson(p, data) {
|
|
26
29
|
ensureDir(path.dirname(p));
|
|
27
|
-
|
|
30
|
+
const serialized = JSON.stringify(data, null, 2);
|
|
31
|
+
if (serialized === undefined) {
|
|
32
|
+
throw new Error(`Cannot serialize undefined JSON fixture: ${p}`);
|
|
33
|
+
}
|
|
34
|
+
const temp = path.join(path.dirname(p), `.${path.basename(p)}.${process.pid}.tmp`);
|
|
35
|
+
try {
|
|
36
|
+
fs.writeFileSync(temp, serialized + "\n", "utf8");
|
|
37
|
+
fs.renameSync(temp, p);
|
|
38
|
+
}
|
|
39
|
+
finally {
|
|
40
|
+
if (fs.existsSync(temp))
|
|
41
|
+
fs.unlinkSync(temp);
|
|
42
|
+
}
|
|
28
43
|
}
|
|
29
44
|
export function listJsonFiles(dir) {
|
|
30
45
|
if (!isDir(dir))
|
package/guides/demo-app.md
CHANGED
|
@@ -324,7 +324,9 @@ with fields for title, area, status, priority, and assignee. On submit:
|
|
|
324
324
|
descriptor for the chosen area (`product-task-write`, etc.).
|
|
325
325
|
3. The server validates the payload against the descriptor's JSON schema.
|
|
326
326
|
4. On success, the record is appended to the in-memory collection and the
|
|
327
|
-
|
|
327
|
+
flush writes it to
|
|
328
|
+
`mocks/data/<area>/<org>-memorix-entities--snapshots.json`. If that routed
|
|
329
|
+
file does not exist yet, the unprefixed `snapshots.json` is used as seed data.
|
|
328
330
|
5. The UI unshifts the returned record into `state.tasks`, selects it, and
|
|
329
331
|
re-renders.
|
|
330
332
|
|
|
@@ -385,8 +387,9 @@ scale. For higher volume you'd model comments as `events` records keyed by
|
|
|
385
387
|
## Why each area is a separate collection
|
|
386
388
|
|
|
387
389
|
We could have put all tasks in one `tasks` collection with an `area` field.
|
|
388
|
-
The demo deliberately splits
|
|
389
|
-
`marketing/snapshots.json`, and `admin/snapshots.json
|
|
390
|
+
The demo deliberately splits the seed fixtures across `product/snapshots.json`,
|
|
391
|
+
`marketing/snapshots.json`, and `admin/snapshots.json`; runtime mutations go to
|
|
392
|
+
their routed, prefixed counterparts. This demonstrates:
|
|
390
393
|
|
|
391
394
|
1. **Multi-entity applications** — typical real apps have more than one object
|
|
392
395
|
type. Showing 3 object types is more representative than 1.
|
|
@@ -1,5 +1,79 @@
|
|
|
1
1
|
# Managing JSON Files
|
|
2
2
|
|
|
3
|
+
## File-backed database routing
|
|
4
|
+
|
|
5
|
+
The static server has no database, so database routing is represented by a
|
|
6
|
+
reserved filename prefix followed by `--`.
|
|
7
|
+
|
|
8
|
+
With:
|
|
9
|
+
|
|
10
|
+
```bash
|
|
11
|
+
MEMORIX_ORG_ID=neo
|
|
12
|
+
MEMORIX_AGENT_ID=neo-agent
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
the routing matrix is:
|
|
16
|
+
|
|
17
|
+
| Data kind | Filename prefix | Example |
|
|
18
|
+
| --- | --- | --- |
|
|
19
|
+
| Snapshots, analysis, decisions | `neo-memorix-entities--` | `data/product/neo-memorix-entities--snapshots.json` |
|
|
20
|
+
| Events | `neo-memorix-events--` | `data/product/neo-memorix-events--events.json` |
|
|
21
|
+
| Memory | `neo-agent-memorix-catalox--` | `data/memory/neo-agent-memorix-catalox--memory.json` |
|
|
22
|
+
| Metadata | `neo-agent-memorix-catalox--` | `metadata/lists/neo-agent-memorix-catalox--my-plate.json` |
|
|
23
|
+
|
|
24
|
+
### Read and write precedence
|
|
25
|
+
|
|
26
|
+
1. The server first looks for the routed, prefixed file.
|
|
27
|
+
2. If it does not exist, it reads the matching unprefixed file as seed data.
|
|
28
|
+
3. Any mutation writes the complete resulting collection to the routed file.
|
|
29
|
+
4. Unprefixed seed files are never overwritten while routed mode is active.
|
|
30
|
+
5. Files belonging to a different prefix are ignored by the active process.
|
|
31
|
+
|
|
32
|
+
Deleting a metadata document writes JSON `null` as a routed tombstone. This is
|
|
33
|
+
intentional: removing only the routed file would cause the unprefixed seed to
|
|
34
|
+
reappear on the next restart.
|
|
35
|
+
|
|
36
|
+
This makes onboarding migration-safe: existing files such as
|
|
37
|
+
`product/snapshots.json` continue to seed a new tenant, while the first write
|
|
38
|
+
creates `product/neo-memorix-entities--snapshots.json`. Starting with another
|
|
39
|
+
organization ID selects a different file and therefore different state.
|
|
40
|
+
|
|
41
|
+
The delimiter `--` is reserved. Do not use it inside ordinary, unprefixed
|
|
42
|
+
fixture basenames.
|
|
43
|
+
|
|
44
|
+
Routing IDs are validated before any files are opened. Organization IDs,
|
|
45
|
+
agent IDs, object types, and metadata names may contain letters, numbers,
|
|
46
|
+
dot, underscore, and hyphen, but cannot contain path separators, `..`, or the
|
|
47
|
+
reserved `--` delimiter.
|
|
48
|
+
|
|
49
|
+
Malformed or empty JSON stops startup with the affected path in the error;
|
|
50
|
+
it is never silently converted to an empty collection. Writes use a temporary
|
|
51
|
+
file and same-directory rename so readers never observe a partially written
|
|
52
|
+
JSON document.
|
|
53
|
+
|
|
54
|
+
Routing is process-scoped. Run one server process per organization/agent route,
|
|
55
|
+
and use only one writer process for a given prefix. Two processes writing the
|
|
56
|
+
same routed file concurrently use last-writer-wins semantics.
|
|
57
|
+
|
|
58
|
+
### Inspect the active route
|
|
59
|
+
|
|
60
|
+
```bash
|
|
61
|
+
curl http://localhost:5030/api/explorer/health
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
The response includes:
|
|
65
|
+
|
|
66
|
+
```json
|
|
67
|
+
{
|
|
68
|
+
"fileRouting": {
|
|
69
|
+
"entitiesPrefix": "neo-memorix-entities",
|
|
70
|
+
"eventsPrefix": "neo-memorix-events",
|
|
71
|
+
"memoryPrefix": "neo-agent-memorix-catalox",
|
|
72
|
+
"metadataPrefix": "neo-agent-memorix-catalox"
|
|
73
|
+
}
|
|
74
|
+
}
|
|
75
|
+
```
|
|
76
|
+
|
|
3
77
|
The mock server stores all state in JSON files under the `mocks/` directory. At
|
|
4
78
|
startup, every file is loaded into an in-memory store. Mutations happen in
|
|
5
79
|
memory and are debounced-flushed back to disk automatically.
|
|
@@ -104,7 +104,19 @@ All configuration is via environment variables. Defaults work out of the box.
|
|
|
104
104
|
| --------- | ----------- | ------------------------------ |
|
|
105
105
|
| `PORT` | `5030` | HTTP listen port |
|
|
106
106
|
| `HOST` | `0.0.0.0` | HTTP listen host |
|
|
107
|
-
| `MOCKS_DIR` | `<
|
|
107
|
+
| `MOCKS_DIR` | bundled `<package>/mocks` | Root directory for JSON fixtures |
|
|
108
|
+
|
|
109
|
+
`MOCKS_DIR` must be writable if you use mutation endpoints. For installed or
|
|
110
|
+
containerized deployments, point it at a persistent application-owned volume
|
|
111
|
+
rather than relying on the package's bundled seed directory:
|
|
112
|
+
|
|
113
|
+
```bash
|
|
114
|
+
MOCKS_DIR=/var/lib/my-app/memorix-mocks static-memorix-explorer-api --port 5030
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
Configuration is resolved when the package is imported. In programmatic usage,
|
|
118
|
+
set `process.env.MOCKS_DIR`, `MEMORIX_ORG_ID`, and `MEMORIX_AGENT_ID` before the
|
|
119
|
+
first dynamic import, and run separate processes for separate routing contexts.
|
|
108
120
|
|
|
109
121
|
### DB Router Simulation
|
|
110
122
|
|
|
@@ -121,6 +133,19 @@ correct tenant.
|
|
|
121
133
|
The health endpoint returns:
|
|
122
134
|
- `memorixDb`: `"<orgId>-memorix-entities + <orgId>-memorix-events"`
|
|
123
135
|
- `cataloxDb`: `"<agentId>-memorix-catalox"`
|
|
136
|
+
- `fileRouting`: the concrete entity, event, memory, and metadata filename
|
|
137
|
+
prefixes selected for this process
|
|
138
|
+
|
|
139
|
+
These are functional routes, not display-only labels. For example:
|
|
140
|
+
|
|
141
|
+
```bash
|
|
142
|
+
MEMORIX_ORG_ID=neo MEMORIX_AGENT_ID=neo-agent npm start
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
selects filenames beginning with `neo-memorix-entities--`,
|
|
146
|
+
`neo-memorix-events--`, and `neo-agent-memorix-catalox--`. See
|
|
147
|
+
[managing-json-files.md](./managing-json-files.md#file-backed-database-routing)
|
|
148
|
+
for exact examples and fallback behavior.
|
|
124
149
|
|
|
125
150
|
### Feature flags
|
|
126
151
|
|