@x12i/static-memorix 2.0.0

package/guides/managing-json-files.md

@@ -0,0 +1,384 @@
+# Managing JSON Files
+
+## File-backed database routing
+
+The static server has no database, so database routing is represented by a
+reserved filename prefix followed by `--`.
+
+With:
+
+```bash
+MEMORIX_ORG_ID=neo
+MEMORIX_AGENT_ID=neo-agent
+```
+
+the routing matrix is:
+
+| Data kind | Filename prefix | Example |
+| --- | --- | --- |
+| Snapshots, analysis, decisions | `neo-memorix-entities--` | `data/product/neo-memorix-entities--snapshots.json` |
+| Events | `neo-memorix-events--` | `data/product/neo-memorix-events--events.json` |
+| Memory | `neo-agent-memorix-catalox--` | `data/memory/neo-agent-memorix-catalox--memory.json` |
+| Metadata | `neo-agent-memorix-catalox--` | `metadata/lists/neo-agent-memorix-catalox--my-plate.json` |
+
+### Read and write precedence
+
+1. The server first looks for the routed, prefixed file.
+2. If it does not exist, it reads the matching unprefixed file as seed data.
+3. Any mutation writes the complete resulting collection to the routed file.
+4. Unprefixed seed files are never overwritten while routed mode is active.
+5. Files belonging to a different prefix are ignored by the active process.
+
+Deleting a metadata document writes JSON `null` as a routed tombstone. This is
+intentional: removing only the routed file would cause the unprefixed seed to
+reappear on the next restart.
+
+This makes onboarding migration-safe: existing files such as
+`product/snapshots.json` continue to seed a new tenant, while the first write
+creates `product/neo-memorix-entities--snapshots.json`. Starting with another
+organization ID selects a different file and therefore different state.
+
+The delimiter `--` is reserved. Do not use it inside ordinary, unprefixed
+fixture basenames.
+
+Routing IDs are validated before any files are opened. Organization IDs,
+agent IDs, object types, and metadata names may contain letters, numbers,
+dot, underscore, and hyphen, but cannot contain path separators, `..`, or the
+reserved `--` delimiter.
+
+Malformed or empty JSON stops startup with the affected path in the error;
+it is never silently converted to an empty collection. Writes use a temporary
+file and same-directory rename so readers never observe a partially written
+JSON document.
+
+Routing is process-scoped. Run one server process per organization/agent route,
+and use only one writer process for a given prefix. Two processes writing the
+same routed file concurrently use last-writer-wins semantics.
+
+### Inspect the active route
+
+```bash
+curl http://localhost:5030/api/explorer/health
+```
+
+The response includes:
+
+```json
+{
+  "fileRouting": {
+    "entitiesPrefix": "neo-memorix-entities",
+    "eventsPrefix": "neo-memorix-events",
+    "memoryPrefix": "neo-agent-memorix-catalox",
+    "metadataPrefix": "neo-agent-memorix-catalox"
+  }
+}
+```
+
+The mock server stores all state in JSON files under the `mocks/` directory. At
+startup, every file is loaded into an in-memory store. Mutations happen in
+memory and are debounced-flushed back to disk automatically.
+
+## Directory structure
+
+```
+mocks/
+  metadata/                         # schema, config, descriptors
+    agents.json
+    object-types/
+      assets.json
+      findings.json
+      memory.json
+      product.json
+      marketing.json
+      admin.json
+      users.json
+    lists/
+      assets-default.json
+      findings-default.json
+      my-plate.json
+      this-week.json
+      backlog.json
+    narratives/
+      assets.json
+      findings.json
+      product.json
+      marketing.json
+      admin.json
+    write-descriptors/
+      assets-analysis-write.json
+      memory-write.json
+      product-task-write.json
+      marketing-task-write.json
+      admin-task-write.json
+  data/                             # record collections
+    assets/
+      snapshots.json
+      analysis.json
+      decisions.json
+      events.json
+    findings/
+      snapshots.json
+      analysis.json
+      decisions.json
+      events.json
+    product/
+      snapshots.json
+    marketing/
+      snapshots.json
+    admin/
+      snapshots.json
+    users/
+      snapshots.json
+    memory/
+      memory.json
+    system/
+      inventory.json
+```
+
+## Metadata files
+
+Metadata files define the shape of the system: what object types exist, how
+lists are queried, what narratives are authored, and how writes are validated.
+
+### agents.json
+
+Registry of agents that participate in the pipeline.
+
+```json
+{
+  "agents": [
+    { "id": "agent-1", "name": "Discovery Agent", "role": "ingestion" },
+    { "id": "agent-2", "name": "Analysis Agent", "role": "enrichment" },
+    { "id": "agent-3", "name": "Decision Agent", "role": "remediation" }
+  ]
+}
+```
+
+### object-types/\*.json
+
+Each file defines one object type. The `catalogRelations` array declares
+relationships between objects. The `rootPropertyCatalog` is computed at runtime
+by the `POST .../root-property-catalog/compute` endpoint.
+
+```json
+{
+  "name": "assets",
+  "summary": "Infrastructure assets discovered and analyzed by Memorix.",
+  "catalogRelations": [
+    {
+      "sourceProperty": "linkedAssetId",
+      "targetObject": "assets",
+      "targetProperty": "recordId",
+      "relationType": "self-link"
+    },
+    {
+      "sourceProperty": "ownerId",
+      "targetObject": "agents",
+      "targetProperty": "id",
+      "relationType": "ownership"
+    }
+  ],
+  "rootPropertyCatalog": []
+}
+```
+
+To add a new object type: create a new `.json` file under
+`mocks/metadata/object-types/` and add a matching directory under
+`mocks/data/` with the content type files (at minimum `snapshots.json`).
+
+### lists/\*.json
+
+List descriptors define saved queries: which entity they target, which fields
+to project, what extensions to join, and default sort order.
+
+```json
+{
+  "id": "assets-default",
+  "entity": "assets",
+  "entityName": "assets",
+  "baseContentType": "snapshots",
+  "fields": ["recordId", "entityId", "severity", "status"],
+  "extensions": ["analysis", "decisions"],
+  "sort": { "severity": -1, "entityId": 1 },
+  "query": {},
+  "listId": "assets-default"
+}
+```
+
+- `fields` controls which properties appear in the projected result.
+- `extensions` names additional content types to join (e.g. `analysis`,
+  `decisions`) via the snapshot alias pipeline.
+- `sort` is a mingo-compatible sort document (`1` = asc, `-1` = desc).
+
+### narratives/\*.json
+
+Authored narratives are key-value maps. Each key becomes a virtual tag that
+records can carry.
+
+```json
+{
+  "has-vulns": {
+    "key": "has-vulns",
+    "label": "Has Vulnerabilities",
+    "description": "Asset carries at least one known vulnerability."
+  },
+  "exposed-public": {
+    "key": "exposed-public",
+    "label": "Exposed Publicly",
+    "description": "Asset is reachable from the public internet."
+  }
+}
+```
+
+Records reference narratives by setting `narratives` on the document:
+
+```json
+{
+  "recordId": "asset-1",
+  "narratives": { "has-vulns": true, "exposed-public": true }
+}
+```
+
+### write-descriptors/\*.json
+
+Write descriptors define the JSON Schema for write operations and the target
+collection.
+
+```json
+{
+  "writeDescriptorId": "assets-analysis-write",
+  "targetCollection": "assets/analysis",
+  "schema": {
+    "type": "object",
+    "required": ["recordId"],
+    "properties": {
+      "recordId": { "type": "string" },
+      "analysisType": { "type": "string" },
+      "score": { "type": "number" },
+      "notes": { "type": "array" }
+    }
+  }
+}
+```
+
+- `targetCollection` is an `objectType/contentType` pair (e.g. `assets/analysis`).
+- `schema` follows a subset of JSON Schema: `required` array and `properties`
+  map with `type` constraints. The write engine validates input against this
+  before executing the operation.
+- **Required semantics:** `required` is enforced fully only for `add` /
+  `replace`. For `upsert` / `patch` / `delete`, only the identity field
+  (`recordId`, or `memoryId` on the memory tier) is mandatory. This lets the
+  UI send partial updates like `{recordId, status}` without re-supplying every
+  required field.
+
+The demo app's task descriptors (`product-task-write.json`,
+`marketing-task-write.json`, `admin-task-write.json`) all use this pattern.
+See [demo-app.md](./demo-app.md) for the full walkthrough.
+
+## Data files
+
+Data files hold the actual record collections. Each object type gets its own
+directory, and within it, one file per content type.
+
+### Content types
+
+| Content type  | File                | Identity key   | Purpose                        |
+| ------------- | ------------------- | -------------- | ------------------------------ |
+| `snapshots`   | `snapshots.json`    | `recordId`     | Entity snapshots (primary tier)|
+| `analysis`    | `analysis.json`     | `recordId`     | Knowledge/analysis records     |
+| `decisions`   | `decisions.json`    | `recordId`     | Decision records               |
+| `events`      | `events.json`       | `eventId`      | Event records                  |
+| `memory`      | `memory.json`       | `memoryId`     | Fourth tier (standalone)       |
+
+### Record structure
+
+Records are plain JSON objects. Every record must carry exactly one identity
+key. Records can include narrative tags and associated data buckets.
+
+```json
+[
+  {
+    "recordId": "asset-1",
+    "entityId": "web-prod-01",
+    "severity": "critical",
+    "status": "open",
+    "region": "us-east-1",
+    "ownerId": "agent-1",
+    "narratives": { "has-vulns": true, "exposed-public": true },
+    "associatedData": [
+      { "kind": "tag", "value": "prod" },
+      { "kind": "tag", "value": "web" }
+    ],
+    "associatedInferred": [
+      { "kind": "discovery", "value": "public-ip" }
+    ],
+    "associatedAnalysis": [
+      { "kind": "analysis", "value": "cve-score-9" }
+    ]
+  }
+]
+```
+
+### Associated data buckets
+
+Records carry associated data in prefixed fields. The API translates these
+between storage format and the normalized `associated` bucket returned to
+clients.
+
+| Storage field           | Client alias    | Description            |
+| ----------------------- | --------------- | ---------------------- |
+| `associatedData`        | `data`          | Explicit tags          |
+| `associatedDiscovery`   | `discovery`     | Discovery-phase data   |
+| `associatedInferred`    | `discovery`     | Inferred discovery     |
+| `associatedAnalysis`    | `analysis`      | Analysis-phase data    |
+| `associated{Custom}`    | `{custom}`      | Custom bucket          |
+
+### Memory tier
+
+Memory is the fourth data tier. It lives under `mocks/data/memory/memory.json`
+and uses `memoryId` as its identity key (instead of `recordId`).
+
+```json
+[
+  {
+    "memoryId": "mem-1",
+    "entityId": "web-prod-01",
+    "kind": "conversation-summary",
+    "summary": "Discussed upgrade path for web-prod-01 from v2 to v3.",
+    "createdAt": "2026-07-10T08:00:00Z"
+  }
+]
+```
+
+### System inventory
+
+`mocks/data/system/inventory.json` is a special file used by the inventory
+engine for policy configuration.
+
+## How the store works
+
+1. On startup, `InMemoryStore.load()` reads all metadata and data JSON files
+   into JavaScript `Map` objects.
+2. All API reads serve from the in-memory maps (no disk I/O per request).
+3. Writes mutate the in-memory maps and mark files as "dirty".
+4. Dirty files are flushed to disk after a debounce (default 300ms, configured
+   via `DISK_FLUSH_DEBOUNCE_MS`).
+5. On `SIGINT`/`SIGTERM`, all dirty files are flushed immediately before
+   shutdown.
+
+You can modify JSON files on disk while the server is running, but changes
+won't be picked up until the next restart (the store loads once at boot). If
+you need to persist changes made via the API, they are written back
+automatically.
+
+## Adding a new object type
+
+1. Create `mocks/metadata/object-types/<name>.json` following the schema above.
+2. Create `mocks/data/<name>/` directory.
+3. Add at least `mocks/data/<name>/snapshots.json` with an empty array `[]`.
+4. Optionally add `analysis.json`, `decisions.json`, `events.json`.
+5. Create `mocks/metadata/lists/<name>-default.json` for a default list view.
+6. Create `mocks/metadata/narratives/<name>.json` if the object type has
+   authored narratives.
+7. Restart the server. The new object type appears in
+   `GET /api/explorer/object-types` and the inventory.

package/guides/running-the-service.md

@@ -0,0 +1,269 @@
+# Running the Service
+
+The mock server runs as a CLI tool or as a development server. It listens for
+HTTP requests on a configurable port and serves the full Memorix Explorer API
+from in-memory state backed by JSON files on disk.
+
+## Installation
+
+### From npm (global)
+
+```bash
+npm install -g @x12i/static-memorix
+```
+
+This makes the `static-memorix` command available globally.
+`memorix-explorer` remains available as a short, friendly alias.
+
+### From source (git clone)
+
+```bash
+git clone https://github.com/x12i/static-memorix.git
+cd static-memorix
+npm install
+```
+
+## Starting the server
+
+### Production mode (build then run)
+
+```bash
+npm run build
+npm start
+```
+
+### Development mode (no build step)
+
+```bash
+npm run dev
+```
+
+Uses Node's `--experimental-strip-types` to run TypeScript directly. Faster
+iteration, but not suitable for distribution.
+
+### As a global CLI
+
+```bash
+static-memorix
+```
+
+After `npm install -g`, the CLI starts the server with default settings. The
+mocks directory defaults to the bundled fixtures, and the CLI prints the URLs
+to open:
+
+```text
+Memorix Explorer is ready
+  Demo:   http://localhost:5030/demo
+  API:    http://localhost:5030/api/explorer
+  Health: http://localhost:5030/health
+```
+
+### Create a writable demo
+
+The bundled fixtures may live inside a global npm installation. Create a
+user-owned copy before experimenting with create/edit/delete:
+
+```bash
+static-memorix create-demo
+```
+
+The command creates `./memorix-demo` and prints the exact next step:
+
+```text
+Demo created:
+  ./memorix-demo
+
+Start the server with:
+  static-memorix --mocks-dir "./memorix-demo"
+
+Then open:
+  http://localhost:5030/demo
+```
+
+The generated directory contains `demo.html`, `data/`, `metadata/`, and a
+README. It is safe to edit and suitable for source control. The command refuses
+to overwrite an existing file or directory.
+
+Use a custom folder or port:
+
+```bash
+static-memorix create-demo --dir ./fixtures
+static-memorix --mocks-dir ./fixtures --port 8080
+```
+
+### Stop the server
+
+Press Ctrl+C in the terminal where the server is running, or stop a managed
+instance from another terminal:
+
+```bash
+static-memorix stop
+static-memorix stop --port 8080
+```
+
+The CLI tracks only server processes it started and refuses to terminate a PID
+that does not identify as Memorix Explorer. Stale tracking files are removed
+automatically. If startup finds an occupied port, the error prints the matching
+stop command and suggests the next port.
+
+### With `npx` (no global install)
+
+```bash
+npx @x12i/static-memorix
+```
+
+Choose a port with a CLI option or environment variable. CLI options take
+precedence:
+
+```bash
+static-memorix --port 5030 --host 127.0.0.1
+PORT=5030 HOST=127.0.0.1 static-memorix
+```
+
+Built-in help and version output do not start the server:
+
+```bash
+static-memorix --help
+static-memorix --version
+```
+
+### From application code
+
+Importing the package does not open a port. Call `startServer()` explicitly:
+
+```js
+import { startServer } from "@x12i/static-memorix";
+
+const app = await startServer({
+  port: 5030,
+  host: "127.0.0.1",
+  installSignalHandlers: false,
+});
+
+// Later, for example during test teardown:
+await app.close();
+```
+
+For tests that do not need a real TCP listener, build the Fastify app and use
+its injection API:
+
+```js
+import { buildServer } from "@x12i/static-memorix";
+
+const app = await buildServer();
+const response = await app.inject({ method: "GET", url: "/health" });
+console.log(response.json()); // { ok: true }
+await app.close();
+```
+
+## Configuration
+
+All configuration is via environment variables. Defaults work out of the box.
+
+### Core
+
+| Variable  | Default     | Description                    |
+| --------- | ----------- | ------------------------------ |
+| `PORT`    | `5030`      | HTTP listen port               |
+| `HOST`    | `0.0.0.0`   | HTTP listen host               |
+| `MOCKS_DIR` | bundled `<package>/mocks` | Root directory for JSON fixtures |
+
+`MOCKS_DIR` must be writable if you use mutation endpoints. For installed or
+containerized deployments, point it at a persistent application-owned volume
+rather than relying on the package's bundled seed directory:
+
+```bash
+MOCKS_DIR=/var/lib/my-app/memorix-mocks static-memorix --port 5030
+```
+
+Configuration is resolved when the package is imported. In programmatic usage,
+set `process.env.MOCKS_DIR`, `MEMORIX_ORG_ID`, and `MEMORIX_AGENT_ID` before the
+first dynamic import, and run separate processes for separate routing contexts.
+
+### DB Router Simulation
+
+These variables control the fake database routing strings returned by the
+health endpoint. The frontend uses these to verify it is connected to the
+correct tenant.
+
+| Variable                    | Default           | Description                              |
+| --------------------------- | ----------------- | ---------------------------------------- |
+| `MEMORIX_ORG_ID`            | `memorix`         | Organization prefix                      |
+| `MEMORIX_AGENT_ID`          | `default-agent`   | Agent prefix for Catalox DB routing      |
+| `MEMORIX_DEPLOYMENT_PROFILE`| `default`         | When `ebook`, forces org/agent to `ebooks`|
+
+The health endpoint returns:
+- `memorixDb`: `"<orgId>-memorix-entities + <orgId>-memorix-events"`
+- `cataloxDb`: `"<agentId>-memorix-catalox"`
+- `fileRouting`: the concrete entity, event, memory, and metadata filename
+  prefixes selected for this process
+
+These are functional routes, not display-only labels. For example:
+
+```bash
+MEMORIX_ORG_ID=neo MEMORIX_AGENT_ID=neo-agent npm start
+```
+
+selects filenames beginning with `neo-memorix-entities--`,
+`neo-memorix-events--`, and `neo-agent-memorix-catalox--`. See
+[managing-json-files.md](./managing-json-files.md#file-backed-database-routing)
+for exact examples and fallback behavior.
+
+### Feature flags
+
+Metadata mutations on lists and narratives are gated behind feature flags.
+Record mutations are governed by write descriptors and are available by
+default.
+
+| Variable                                    | Controls              |
+| ------------------------------------------- | --------------------- |
+| `MEMORIX_EXPLORER_ENABLE_METADATA_WRITES`   | List & narrative CRUD |
+| `MEMORIX_EXPLORER_ENABLE_PIPELINE_WRITES`   | Pipeline writes       |
+| `MEMORIX_EXPLORER_ENABLE_REGISTRY_WRITES`   | Registry writes       |
+
+Set to `true` or `1` to enable:
+
+```bash
+MEMORIX_EXPLORER_ENABLE_METADATA_WRITES=true static-memorix
+```
+
+### Disk flush
+
+| Variable                   | Default | Description                             |
+| -------------------------- | ------- | --------------------------------------- |
+| `DISK_FLUSH_DEBOUNCE_MS`   | `300`   | Milliseconds to debounce before writing dirty files to disk |
+
+### Example: full custom configuration
+
+```bash
+PORT=8080 \
+HOST=127.0.0.1 \
+MOCKS_DIR=/path/to/my/fixtures \
+MEMORIX_ORG_ID=neo \
+MEMORIX_AGENT_ID=neo-agent \
+MEMORIX_EXPLORER_ENABLE_METADATA_WRITES=true \
+static-memorix
+```
+
+## Health checks
+
+Once running, verify the server is up:
+
+```bash
+curl localhost:5030/health
+# {"ok":true}
+
+curl "localhost:5030/api/explorer/health?includeInventory=1"
+# {"ok":true,"mongoUriConfigured":true,"discoverySample":["assets","findings","memory"],...}
+```
+
+## Graceful shutdown
+
+The server handles `SIGINT` and `SIGTERM`. On shutdown it:
+1. Flushes all dirty in-memory state to disk.
+2. Closes the Fastify listener.
+3. Exits with code 0.
+
+This means mutations made via the API are always persisted as long as the
+server shuts down cleanly. If the process is killed with `SIGKILL`, unsaved
+state is lost.