@x12i/static-memorix-explorer-api 1.0.0 → 1.1.1

package/guides/managing-json-files.md

@@ -0,0 +1,310 @@
+# Managing JSON Files
+
+The mock server stores all state in JSON files under the `mocks/` directory. At
+startup, every file is loaded into an in-memory store. Mutations happen in
+memory and are debounced-flushed back to disk automatically.
+
+## Directory structure
+
+```
+mocks/
+  metadata/                         # schema, config, descriptors
+    agents.json
+    object-types/
+      assets.json
+      findings.json
+      memory.json
+      product.json
+      marketing.json
+      admin.json
+      users.json
+    lists/
+      assets-default.json
+      findings-default.json
+      my-plate.json
+      this-week.json
+      backlog.json
+    narratives/
+      assets.json
+      findings.json
+      product.json
+      marketing.json
+      admin.json
+    write-descriptors/
+      assets-analysis-write.json
+      memory-write.json
+      product-task-write.json
+      marketing-task-write.json
+      admin-task-write.json
+  data/                             # record collections
+    assets/
+      snapshots.json
+      analysis.json
+      decisions.json
+      events.json
+    findings/
+      snapshots.json
+      analysis.json
+      decisions.json
+      events.json
+    product/
+      snapshots.json
+    marketing/
+      snapshots.json
+    admin/
+      snapshots.json
+    users/
+      snapshots.json
+    memory/
+      memory.json
+    system/
+      inventory.json
+```
+
+## Metadata files
+
+Metadata files define the shape of the system: what object types exist, how
+lists are queried, what narratives are authored, and how writes are validated.
+
+### agents.json
+
+Registry of agents that participate in the pipeline.
+
+```json
+{
+  "agents": [
+    { "id": "agent-1", "name": "Discovery Agent", "role": "ingestion" },
+    { "id": "agent-2", "name": "Analysis Agent", "role": "enrichment" },
+    { "id": "agent-3", "name": "Decision Agent", "role": "remediation" }
+  ]
+}
+```
+
+### object-types/\*.json
+
+Each file defines one object type. The `catalogRelations` array declares
+relationships between objects. The `rootPropertyCatalog` is computed at runtime
+by the `POST .../root-property-catalog/compute` endpoint.
+
+```json
+{
+  "name": "assets",
+  "summary": "Infrastructure assets discovered and analyzed by Memorix.",
+  "catalogRelations": [
+    {
+      "sourceProperty": "linkedAssetId",
+      "targetObject": "assets",
+      "targetProperty": "recordId",
+      "relationType": "self-link"
+    },
+    {
+      "sourceProperty": "ownerId",
+      "targetObject": "agents",
+      "targetProperty": "id",
+      "relationType": "ownership"
+    }
+  ],
+  "rootPropertyCatalog": []
+}
+```
+
+To add a new object type: create a new `.json` file under
+`mocks/metadata/object-types/` and add a matching directory under
+`mocks/data/` with the content type files (at minimum `snapshots.json`).
+
+### lists/\*.json
+
+List descriptors define saved queries: which entity they target, which fields
+to project, what extensions to join, and default sort order.
+
+```json
+{
+  "id": "assets-default",
+  "entity": "assets",
+  "entityName": "assets",
+  "baseContentType": "snapshots",
+  "fields": ["recordId", "entityId", "severity", "status"],
+  "extensions": ["analysis", "decisions"],
+  "sort": { "severity": -1, "entityId": 1 },
+  "query": {},
+  "listId": "assets-default"
+}
+```
+
+- `fields` controls which properties appear in the projected result.
+- `extensions` names additional content types to join (e.g. `analysis`,
+  `decisions`) via the snapshot alias pipeline.
+- `sort` is a mingo-compatible sort document (`1` = asc, `-1` = desc).
+
+### narratives/\*.json
+
+Authored narratives are key-value maps. Each key becomes a virtual tag that
+records can carry.
+
+```json
+{
+  "has-vulns": {
+    "key": "has-vulns",
+    "label": "Has Vulnerabilities",
+    "description": "Asset carries at least one known vulnerability."
+  },
+  "exposed-public": {
+    "key": "exposed-public",
+    "label": "Exposed Publicly",
+    "description": "Asset is reachable from the public internet."
+  }
+}
+```
+
+Records reference narratives by setting `narratives` on the document:
+
+```json
+{
+  "recordId": "asset-1",
+  "narratives": { "has-vulns": true, "exposed-public": true }
+}
+```
+
+### write-descriptors/\*.json
+
+Write descriptors define the JSON Schema for write operations and the target
+collection.
+
+```json
+{
+  "writeDescriptorId": "assets-analysis-write",
+  "targetCollection": "assets/analysis",
+  "schema": {
+    "type": "object",
+    "required": ["recordId"],
+    "properties": {
+      "recordId": { "type": "string" },
+      "analysisType": { "type": "string" },
+      "score": { "type": "number" },
+      "notes": { "type": "array" }
+    }
+  }
+}
+```
+
+- `targetCollection` is an `objectType/contentType` pair (e.g. `assets/analysis`).
+- `schema` follows a subset of JSON Schema: `required` array and `properties`
+  map with `type` constraints. The write engine validates input against this
+  before executing the operation.
+- **Required semantics:** `required` is enforced fully only for `add` /
+  `replace`. For `upsert` / `patch` / `delete`, only the identity field
+  (`recordId`, or `memoryId` on the memory tier) is mandatory. This lets the
+  UI send partial updates like `{recordId, status}` without re-supplying every
+  required field.
+
+The demo app's task descriptors (`product-task-write.json`,
+`marketing-task-write.json`, `admin-task-write.json`) all use this pattern.
+See [demo-app.md](./demo-app.md) for the full walkthrough.
+
+## Data files
+
+Data files hold the actual record collections. Each object type gets its own
+directory, and within it, one file per content type.
+
+### Content types
+
+| Content type  | File                | Identity key   | Purpose                        |
+| ------------- | ------------------- | -------------- | ------------------------------ |
+| `snapshots`   | `snapshots.json`    | `recordId`     | Entity snapshots (primary tier)|
+| `analysis`    | `analysis.json`     | `recordId`     | Knowledge/analysis records     |
+| `decisions`   | `decisions.json`    | `recordId`     | Decision records               |
+| `events`      | `events.json`       | `eventId`      | Event records                  |
+| `memory`      | `memory.json`       | `memoryId`     | Fourth tier (standalone)       |
+
+### Record structure
+
+Records are plain JSON objects. Every record must carry exactly one identity
+key. Records can include narrative tags and associated data buckets.
+
+```json
+[
+  {
+    "recordId": "asset-1",
+    "entityId": "web-prod-01",
+    "severity": "critical",
+    "status": "open",
+    "region": "us-east-1",
+    "ownerId": "agent-1",
+    "narratives": { "has-vulns": true, "exposed-public": true },
+    "associatedData": [
+      { "kind": "tag", "value": "prod" },
+      { "kind": "tag", "value": "web" }
+    ],
+    "associatedInferred": [
+      { "kind": "discovery", "value": "public-ip" }
+    ],
+    "associatedAnalysis": [
+      { "kind": "analysis", "value": "cve-score-9" }
+    ]
+  }
+]
+```
+
+### Associated data buckets
+
+Records carry associated data in prefixed fields. The API translates these
+between storage format and the normalized `associated` bucket returned to
+clients.
+
+| Storage field           | Client alias    | Description            |
+| ----------------------- | --------------- | ---------------------- |
+| `associatedData`        | `data`          | Explicit tags          |
+| `associatedDiscovery`   | `discovery`     | Discovery-phase data   |
+| `associatedInferred`    | `discovery`     | Inferred discovery     |
+| `associatedAnalysis`    | `analysis`      | Analysis-phase data    |
+| `associated{Custom}`    | `{custom}`      | Custom bucket          |
+
+### Memory tier
+
+Memory is the fourth data tier. It lives under `mocks/data/memory/memory.json`
+and uses `memoryId` as its identity key (instead of `recordId`).
+
+```json
+[
+  {
+    "memoryId": "mem-1",
+    "entityId": "web-prod-01",
+    "kind": "conversation-summary",
+    "summary": "Discussed upgrade path for web-prod-01 from v2 to v3.",
+    "createdAt": "2026-07-10T08:00:00Z"
+  }
+]
+```
+
+### System inventory
+
+`mocks/data/system/inventory.json` is a special file used by the inventory
+engine for policy configuration.
+
+## How the store works
+
+1. On startup, `InMemoryStore.load()` reads all metadata and data JSON files
+   into JavaScript `Map` objects.
+2. All API reads serve from the in-memory maps (no disk I/O per request).
+3. Writes mutate the in-memory maps and mark files as "dirty".
+4. Dirty files are flushed to disk after a debounce (default 300ms, configured
+   via `DISK_FLUSH_DEBOUNCE_MS`).
+5. On `SIGINT`/`SIGTERM`, all dirty files are flushed immediately before
+   shutdown.
+
+You can modify JSON files on disk while the server is running, but changes
+won't be picked up until the next restart (the store loads once at boot). If
+you need to persist changes made via the API, they are written back
+automatically.
+
+## Adding a new object type
+
+1. Create `mocks/metadata/object-types/<name>.json` following the schema above.
+2. Create `mocks/data/<name>/` directory.
+3. Add at least `mocks/data/<name>/snapshots.json` with an empty array `[]`.
+4. Optionally add `analysis.json`, `decisions.json`, `events.json`.
+5. Create `mocks/metadata/lists/<name>-default.json` for a default list view.
+6. Create `mocks/metadata/narratives/<name>.json` if the object type has
+   authored narratives.
+7. Restart the server. The new object type appears in
+   `GET /api/explorer/object-types` and the inventory.

package/guides/running-the-service.md

@@ -0,0 +1,171 @@
+# Running the Service
+
+The mock server runs as a CLI tool or as a development server. It listens for
+HTTP requests on a configurable port and serves the full Memorix Explorer API
+from in-memory state backed by JSON files on disk.
+
+## Installation
+
+### From npm (global)
+
+```bash
+npm install -g @x12i/static-memorix-explorer-api
+```
+
+This makes the `mock-memorix-explorer-api` command available globally.
+
+### From source (git clone)
+
+```bash
+git clone https://github.com/x12i/static-memorix-explorer-api.git
+cd static-memorix-explorer-api
+npm install
+```
+
+## Starting the server
+
+### Production mode (build then run)
+
+```bash
+npm run build
+npm start
+```
+
+### Development mode (no build step)
+
+```bash
+npm run dev
+```
+
+Uses Node's `--experimental-strip-types` to run TypeScript directly. Faster
+iteration, but not suitable for distribution.
+
+### As a global CLI
+
+```bash
+mock-memorix-explorer-api
+```
+
+After `npm install -g`, the CLI starts the server with default settings. The
+mocks directory defaults to a `mocks/` folder relative to the installed package.
+
+### With `npx` (no global install)
+
+```bash
+npx @x12i/static-memorix-explorer-api
+```
+
+### From application code
+
+Importing the package does not open a port. Call `startServer()` explicitly:
+
+```js
+import { startServer } from "@x12i/static-memorix-explorer-api";
+
+const app = await startServer({
+  port: 5030,
+  host: "127.0.0.1",
+  installSignalHandlers: false,
+});
+
+// Later, for example during test teardown:
+await app.close();
+```
+
+For tests that do not need a real TCP listener, build the Fastify app and use
+its injection API:
+
+```js
+import { buildServer } from "@x12i/static-memorix-explorer-api";
+
+const app = await buildServer();
+const response = await app.inject({ method: "GET", url: "/health" });
+console.log(response.json()); // { ok: true }
+await app.close();
+```
+
+## Configuration
+
+All configuration is via environment variables. Defaults work out of the box.
+
+### Core
+
+| Variable  | Default     | Description                    |
+| --------- | ----------- | ------------------------------ |
+| `PORT`    | `5030`      | HTTP listen port               |
+| `HOST`    | `0.0.0.0`   | HTTP listen host               |
+| `MOCKS_DIR` | `<project>/mocks` | Root directory for JSON fixtures |
+
+### DB Router Simulation
+
+These variables control the fake database routing strings returned by the
+health endpoint. The frontend uses these to verify it is connected to the
+correct tenant.
+
+| Variable                    | Default           | Description                              |
+| --------------------------- | ----------------- | ---------------------------------------- |
+| `MEMORIX_ORG_ID`            | `memorix`         | Organization prefix                      |
+| `MEMORIX_AGENT_ID`          | `default-agent`   | Agent prefix for Catalox DB routing      |
+| `MEMORIX_DEPLOYMENT_PROFILE`| `default`         | When `ebook`, forces org/agent to `ebooks`|
+
+The health endpoint returns:
+- `memorixDb`: `"<orgId>-memorix-entities + <orgId>-memorix-events"`
+- `cataloxDb`: `"<agentId>-memorix-catalox"`
+
+### Feature flags
+
+Mutations (POST/PATCH/PUT/DELETE) on lists, narratives, and records are gated
+behind feature flags. All default to `false`.
+
+| Variable                                    | Controls              |
+| ------------------------------------------- | --------------------- |
+| `MEMORIX_EXPLORER_ENABLE_METADATA_WRITES`   | List & narrative CRUD |
+| `MEMORIX_EXPLORER_ENABLE_PIPELINE_WRITES`   | Pipeline writes       |
+| `MEMORIX_EXPLORER_ENABLE_REGISTRY_WRITES`   | Registry writes       |
+
+Set to `true` or `1` to enable:
+
+```bash
+MEMORIX_EXPLORER_ENABLE_METADATA_WRITES=true mock-memorix-explorer-api
+```
+
+### Disk flush
+
+| Variable                   | Default | Description                             |
+| -------------------------- | ------- | --------------------------------------- |
+| `DISK_FLUSH_DEBOUNCE_MS`   | `300`   | Milliseconds to debounce before writing dirty files to disk |
+
+### Example: full custom configuration
+
+```bash
+PORT=8080 \
+HOST=127.0.0.1 \
+MOCKS_DIR=/path/to/my/fixtures \
+MEMORIX_ORG_ID=neo \
+MEMORIX_AGENT_ID=neo-agent \
+MEMORIX_EXPLORER_ENABLE_METADATA_WRITES=true \
+mock-memorix-explorer-api
+```
+
+## Health checks
+
+Once running, verify the server is up:
+
+```bash
+curl localhost:5030/health
+# {"ok":true}
+
+curl "localhost:5030/api/explorer/health?includeInventory=1"
+# {"ok":true,"mongoUriConfigured":true,"discoverySample":["assets","findings","memory"],...}
+```
+
+## Graceful shutdown
+
+The server handles `SIGINT` and `SIGTERM`. On shutdown it:
+1. Flushes all dirty in-memory state to disk.
+2. Closes the Fastify listener.
+3. Exits with code 0.
+
+This means mutations made via the API are always persisted as long as the
+server shuts down cleanly. If the process is killed with `SIGKILL`, unsaved
+state is lost.