@mnmal-ai/substrate 1.0.0-alpha.9 → 1.0.0-beta.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (52) hide show
  1. package/README.md +31 -13
  2. package/dist/boot-store-BMQUbHKv.js +4086 -0
  3. package/dist/{runtime-metrics-DwpDJ8vz.d.ts → bridge-DnToJedN.d.ts} +2 -87
  4. package/dist/cli-DModIbnN.js +3390 -0
  5. package/dist/client.browser.js +2507 -0
  6. package/dist/client.d.ts +222 -171
  7. package/dist/client.js +6 -7
  8. package/dist/cluster-worker.js +0 -1
  9. package/dist/{content-type-C7T28xNb.js → content-type-wLljot5M.js} +0 -1
  10. package/dist/{env-utils-CGNazUNt.js → env-utils-s25Z1YSA.js} +0 -1
  11. package/dist/{errors-D4vKuwz8.d.ts → errors-Cmkp1qxf.d.ts} +44 -4
  12. package/dist/{fake-DUVtKVSE.js → fake-DdXtb-m8.js} +0 -1
  13. package/dist/{index-D7mSOFlM.d.ts → index-BI4KqFiR.d.ts} +7 -11
  14. package/dist/msgpackr-Dq18g6Ds.js +16 -0
  15. package/dist/node-version-BtK_Nq6z.js +32 -0
  16. package/dist/pack.d.ts +75 -11
  17. package/dist/pack.js +4 -4
  18. package/dist/{plugin-DpncES1p.js → plugin-CEOhrEme.js} +78 -8
  19. package/dist/proposals-bootstrap.js +6 -5
  20. package/dist/proposals.config.schema.json +5 -0
  21. package/dist/proposals.d.ts +17 -5
  22. package/dist/proposals.default-config.json +2 -1
  23. package/dist/proposals.js +2 -3
  24. package/dist/recall-bootstrap.js +3 -4
  25. package/dist/recall.d.ts +11 -12
  26. package/dist/recall.js +2 -3
  27. package/dist/{repl-DeKzfvNV.js → repl-BvDZy-Ut.js} +52 -26
  28. package/dist/{row-SRhura12.d.ts → row-D0If4qif.d.ts} +91 -13
  29. package/dist/run-BT123RV1.js +2443 -0
  30. package/dist/{plugin-t2g2wX0b.d.ts → schema-aIkY8sR3.d.ts} +1293 -63
  31. package/dist/{script-C6DSa_Na.js → script-ByQ9pzhC.js} +54 -26
  32. package/dist/script-runner.js +78 -0
  33. package/dist/sdk.d.ts +236 -38
  34. package/dist/sdk.js +5 -110
  35. package/dist/semver-D_zdq0hi.js +17 -0
  36. package/dist/src-BMD2OE--.js +8289 -0
  37. package/dist/{src-BLgbLq2k.js → src-B_kHkbsr.js} +780 -330
  38. package/dist/{src-DjIuRjzF.js → src-CKSJ13p8.js} +10231 -4237
  39. package/dist/{src-C12EA0w9.js → src-CW7eRDss.js} +179 -96
  40. package/dist/{row-DZ0Fnpc6.js → src-DmiUWOLp.js} +2 -1
  41. package/dist/{src-DCy7Iy2l.js → src-DmjwFMu8.js} +900 -77
  42. package/dist/{src-DiNTSaFk.js → src-FXO5JPyk.js} +686 -856
  43. package/dist/substrate-admin.js +17 -782
  44. package/dist/substrate.js +4 -5302
  45. package/dist/testing.d.ts +473 -96
  46. package/dist/testing.js +19 -165
  47. package/dist/{ws-C72iVS5T.d.ts → ws-DW8ng02H.d.ts} +4 -5
  48. package/package.json +26 -26
  49. package/dist/dataset-CqrqTYPz.js +0 -1535
  50. package/dist/pg-edges-store-KH_FQCxx.js +0 -381
  51. package/dist/schema-g4FQLr0V.d.ts +0 -433
  52. package/dist/src-BD27zeGF.js +0 -2339
package/README.md CHANGED
@@ -2,7 +2,7 @@
2
2
 
3
3
  The `substrate` binary: the server, the stdio MCP server, and the runtime scaffolder, plus `substrate-admin` for key management. The same package carries the libraries for plugin authors and clients, and two built-in plugins: `substrate-recall` (kernel) and `substrate-proposals` (coordinator).
4
4
 
5
- It runs as a real install: the binary and its plugins resolve from one shared `node_modules`, so every plugin sees the same copy of every shared package. `substrate init` scaffolds such a runtime directory. Node `>=26.2.0` is required; a durable deployment also needs Postgres with the `pgvector` extension. Walkthrough: [`docs/getting-started.md`](https://github.com/mnmal-ai/substrate/blob/main/docs/getting-started.md).
5
+ It runs as a real install: the binary and its plugins resolve from one shared `node_modules`, so every plugin sees the same copy of every shared package. `substrate init` scaffolds such a runtime directory. Node `>=26.13.1` is required; a durable deployment also needs PostgreSQL 18 or newer with the `pgvector` extension. Walkthrough: [`docs/getting-started.md`](https://github.com/mnmal-ai/substrate/blob/main/docs/getting-started.md).
6
6
 
7
7
  ## Library subpaths
8
8
 
@@ -21,7 +21,7 @@ A plugin package declares `@mnmal-ai/substrate` as a peer dependency and as a de
21
21
 
22
22
  | Subcommand | Purpose |
23
23
  |---|---|
24
- | `substrate serve` | Boot the HTTP/WS server (default port 3100) |
24
+ | `substrate serve` | Boot the HTTP server (default port 3100) |
25
25
  | `substrate mcp` | stdio MCP server; reads `SUBSTRATE_URL`, and `SUBSTRATE_MCP_TOOLS` (csv) to register only some tools |
26
26
  | `substrate init [dir] [flags]` | Scaffold a runtime directory (interactive; `--yes` for scripted use) |
27
27
  | `substrate ask <question…>` | Ask a natural-language question (needs `substrate-nl`) |
@@ -35,7 +35,7 @@ A plugin package declares `@mnmal-ai/substrate` as a peer dependency and as a de
35
35
 
36
36
  ### MCP tools
37
37
 
38
- `substrate mcp` serves `schema`, `query`, `mutate`, `recall`, `ask`, `whoami`, `propose_type`, `propose_field`, and the admin-gated `drop_namespace` / `drop_type`. It talks to a running server at `SUBSTRATE_URL` and signs requests when `SUBSTRATE_AGENT_KEY_ID` and `SUBSTRATE_AGENT_KEY_PATH` are set. [`docs/surface-parity.md`](https://github.com/mnmal-ai/substrate/blob/main/docs/surface-parity.md) maps what each surface (HTTP, MCP, WS, SSE) offers.
38
+ `substrate mcp` serves `schema`, `query`, `mutate`, `run`, `recall`, `ask`, `whoami`, `propose_type`, `propose_field`, `propose_index`, and the admin-gated `drop_namespace` / `drop_type`. `run` executes a short script against the store in one call (the same context as `substrate run`), in a child process with no file, network, process or environment access. It talks to a running server at `SUBSTRATE_URL` and signs requests when `SUBSTRATE_AGENT_KEY_ID` and `SUBSTRATE_AGENT_KEY_PATH` are set. [`docs/surface-parity.md`](https://github.com/mnmal-ai/substrate/blob/main/docs/surface-parity.md) maps what each surface (HTTP, MCP, frame streams) offers.
39
39
 
40
40
  ### `substrate ask`
41
41
 
@@ -51,7 +51,7 @@ Posts the question to `POST /substrate/v1/nl` on `SUBSTRATE_URL` and prints the
51
51
  substrate run <script.mts> [--args <json>] [--dry-run | --write] [--max-writes <n>] [--out <file>]
52
52
  ```
53
53
 
54
- Imports the script and calls its default export with a `ScriptContext` (`args`, `query`, `queryAll`, `recall`, `mutate`, `log`), so one call can chain many reads and writes and return one result. The client is signed from `SUBSTRATE_URL`, `SUBSTRATE_AGENT_KEY_ID` and `SUBSTRATE_AGENT_KEY_PATH`; an unsigned run is refused.
54
+ Imports the script and calls its default export with a `ScriptContext` (`args`, `query`, `queryAll`, `byShortId`, `recall`, `mutate`, `log`), so one call can chain many reads and writes and return one result. The client is signed from `SUBSTRATE_URL`, `SUBSTRATE_AGENT_KEY_ID` and `SUBSTRATE_AGENT_KEY_PATH`; an unsigned run is refused.
55
55
 
56
56
  ```ts
57
57
  import type { ScriptContext } from '@mnmal-ai/substrate/client';
@@ -72,7 +72,9 @@ Guardrails:
72
72
  - **A rejected frame throws.** A query whose key is missing from `data`, or a mutate the server refused, fails the run with the server's error rather than reading as empty.
73
73
  - **Truncation is reported.** `query` adds a warning when a result has `hasMore`; `queryAll` pages to the end.
74
74
 
75
- `s.recall({ query, namespace, type, k?, threshold?, where?, row? })` returns hits best first, through the `recall/Hit` frame. Each hit's `row` is `id` plus the handle fields; pass `row: { body: true }` (or `true`) for more, on the hits you will read.
75
+ `s.byShortId('cortext/Todo', 1787)` returns that row, and `s.byShortId('cortext/Todo', [1787, 1785], { title: true })` returns the rows in input order, through `where: { shortId: { eq | in } }`. It throws naming every shortId with no row, rather than returning fewer rows than asked for.
76
+
77
+ `s.recall({ query, namespace, type, k?, threshold?, where?, row? })` returns hits best first, through the `recall/Hit` frame, typed as `RecallHit` (exported beside `ScriptContext`). Each hit's `row` is `id` plus the handle fields (`name`, `title`, `topic`, `synopsis`, `description`, each typed `string | undefined`, so `h.row.name` needs no cast); pass `row: { body: true }` (or `true`) for more, on the hits you will read.
76
78
 
77
79
  `--args` takes a JSON object, which the script reads as `s.args` (`{}` when omitted), so one script serves many calls.
78
80
 
@@ -82,7 +84,7 @@ The pieces are exported for use outside the CLI: `createClientFromEnv()` builds
82
84
 
83
85
  ### `substrate-admin`
84
86
 
85
- A separate binary for key management: `bootstrap` (seed the genesis admin key), `doctor` (check a deployment end to end), `onboard`, `list`, `register`, `update`, `rotate`, `revoke`. `substrate-admin --help` documents each.
87
+ A separate binary for key management and store operations: `bootstrap` (seed the genesis admin key), `doctor` (check a deployment end to end), `onboard`, `list`, `register`, `update`, `rotate`, `revoke`, `promote-fields` (run the promotion sweep now on a running server, giving fields added at runtime their columns), and `migrate-store` (copy a store into a fresh, empty PostgreSQL 18 store on per-type tables and verify it; see [Upgrading to per-type tables](https://github.com/mnmal-ai/substrate/blob/main/docs/upgrading-to-type-tables.md)). `substrate-admin --help` documents each.
86
88
 
87
89
  ## Install
88
90
 
@@ -99,7 +101,7 @@ set -a && source .env && set +a
99
101
  ./node_modules/.bin/substrate serve
100
102
  ```
101
103
 
102
- `substrate init` writes a `package.json` pinning `@mnmal-ai/substrate` and the chosen optional plugins, a `pnpm-workspace.yaml`, a `.env` with `SUBSTRATE_PLUGIN_DIR=.`, a `substrate-recall.config.json` for the embedder, and, unless `--store=memory`, a `docker-compose.yml` running `pgvector/pgvector:pg17`. Flags: `--store=memory|postgres`, `--port`, `--pg-port`, `--plugins=<csv>` (from `cortext`, `substrate-nl`, `substrate-sentinel`; none by default), `--embedder=bge-base|minilm|ollama|none` (default `bge-base`), `--tls --tls-cert-path --tls-key-path`, `--substrate-url`, `--yes`, `--force`.
104
+ `substrate init` writes a `package.json` pinning `@mnmal-ai/substrate` and the chosen optional plugins, a `pnpm-workspace.yaml`, a `.env` with `SUBSTRATE_PLUGIN_DIR=.`, a `substrate-recall.config.json` for the embedder, and, unless `--store=memory`, a `docker-compose.yml` running `pgvector/pgvector:pg18`. Flags: `--store=memory|postgres`, `--port`, `--pg-port`, `--plugins=<csv>` (from `cortext`, `substrate-nl`, `substrate-sentinel`; none by default), `--embedder=bge-base|minilm|ollama|none` (default `bge-base`), `--tls --tls-cert-path --tls-key-path`, `--substrate-url`, `--yes`, `--force`.
103
105
 
104
106
  `SUBSTRATE_PLUGIN_DIR` is the directory whose `node_modules/` holds the plugins; in a scaffolded runtime that is the runtime directory itself. Recall and proposals load from `@mnmal-ai/substrate` itself, so they are present whenever the binary is installed there. Kernel plugins (`substrate-recall`, `substrate-nl`) compose first, then coordinators (`substrate-proposals`, `substrate-sentinel`, …), alphabetical within each tier. Per-plugin overrides live at `<SUBSTRATE_PLUGIN_DIR>/<pkg-basename>.config.json`. With `SUBSTRATE_PLUGIN_DIR` unset the binary boots with only the `_system` admin namespace and `/health`. Set, it must hold a `package.json` listing the plugins under `dependencies`, installed into its `node_modules/`; a missing `package.json`, or listed plugins with no `node_modules/`, refuses boot with a message naming the fix.
105
107
 
@@ -126,21 +128,33 @@ The repository's `docker-compose.yml` runs this binary with `substrate-nl` and `
126
128
 
127
129
  ## Environment variables
128
130
 
129
- The main ones; `substrate --help` and the header of `apps/substrate/src/main.ts` list the rest.
131
+ The main ones; `substrate --help` and the header of `apps/substrate/src/main.ts` list the rest. [Operator decisions](https://github.com/mnmal-ai/substrate/blob/main/docs/operator-decisions.md) explains what each choice costs.
130
132
 
131
133
  | Var | Purpose |
132
134
  |---|---|
133
- | `SUBSTRATE_DATABASE_URL` | `memory` (default, ephemeral) or `postgres://...` |
135
+ | `SUBSTRATE_DATABASE_URL` | `memory` (default, ephemeral) or `postgres://...`. A Postgres store keeps each type in a table of its own; a store in an old layout (`legacy` or `partitioned`) is refused at boot, naming `substrate-admin migrate-store`, which copies it into a fresh one. `SUBSTRATE_STORAGE_LAYOUT` is no longer read: setting it logs one warning. |
134
136
  | `SUBSTRATE_PLUGIN_DIR` | Plugin install root. The loader scans `<dir>/node_modules/{*,@*/*}/package.json` for packages with a `substrate` manifest and merges each plugin's `default-config.json` with `<dir>/<basename>.config.json`. |
135
- | `PORT` / `HOST` | Listen address (default `3100` / `0.0.0.0`) |
137
+ | `PORT` / `HOST` | Listen address (default `3100`; `HOST` defaults to `0.0.0.0` with auth on and `127.0.0.1` with auth off) |
138
+ | `SUBSTRATE_ALLOW_UNAUTHENTICATED_HOST` | `1` or `true` permits a non-loopback `HOST` with auth off; without it that combination refuses boot. |
136
139
  | `SUBSTRATE_SYSTEM_ACTOR` | Who the deployment writes as while auth is off. Required in that case: the server refuses to boot rather than write unattributed rows. |
137
140
  | `SUBSTRATE_AUTH_MODE` | `required` or `attribute` (anonymous callers allowed on non-admin routes). Setting it, or `SUBSTRATE_AUTH_KEY_REGISTRY`, enables auth. |
138
141
  | `SUBSTRATE_AUTH_KEY_REGISTRY` | Path to a JSON key registry file. Without it, auth reads keys registered in Postgres with `substrate-admin`. |
139
142
  | `SUBSTRATE_PUBLIC_ORIGIN` | The origins callers reach this server by, comma-separated (`https://host:3100,https://alias`). A token whose `aud` names none of them is refused. **Required when auth is enabled and `HOST` is not loopback**; on loopback it defaults to the loopback origins. |
140
- | `SUBSTRATE_STREAM_TICKET_SECRET` | HMAC key signing SSE stream tickets. **Required when auth is enabled**; at least 16 characters. The ticket signature is the only authentication on `GET /substrate/v1/stream` and `GET /substrate/v1/subscribe`, because `EventSource` can send no header or body. Rotating it invalidates every outstanding ticket. |
141
- | `SUBSTRATE_TLS_KEY_PATH` / `SUBSTRATE_TLS_CERT_PATH` / `SUBSTRATE_TLS_CA_PATH` | Serve HTTPS. Key and cert are required together. |
142
- | `SUBSTRATE_NAMESPACE` | Default namespace for recall requests that omit one. |
143
+ | `SUBSTRATE_CURSOR_SECRET` | Seals pagination cursors under row-filter policies, so every process sharing it opens the same cursors. Unset, the key is per process and a sealed cursor stops opening after a restart or on another replica. When it is unset, the retired `SUBSTRATE_STREAM_TICKET_SECRET` is read instead. Rotating it ends every outstanding sealed cursor. |
144
+ | `SUBSTRATE_TLS_KEY_PATH` / `SUBSTRATE_TLS_CERT_PATH` / `SUBSTRATE_TLS_CA_PATH` | Serve HTTPS, over HTTP/2 (the server is HTTP/2 only; without TLS it is h2c). Key and cert are required together. |
145
+ | `SUBSTRATE_NAMESPACE` | Default namespace for recall requests that omit one; with neither, recall refuses with `missing_namespace`. |
146
+ | `SUBSTRATE_STATEMENT_TIMEOUT_MS` | Statement timeout for typed edge walk reads (default 30000). A read it cancels is refused `query_timeout`. It also bounds the wait for a free database connection; a request that waits longer is refused `store_busy`. |
147
+ | `SUBSTRATE_PAGE_LIMIT_DEFAULT` / `SUBSTRATE_PAGE_LIMIT_MAX` | Page size for a remote query that names no `limit` (default 100), and the largest `limit` accepted (default 1000). |
148
+ | `SUBSTRATE_FILTER_WRITE_MAX_MATCH` / `SUBSTRATE_FILTER_WRITE_IDS_CAP` | `updateMany<Type>` / `deleteMany<Type>`: a filter matching more rows than the max is refused `filter_too_broad` (default 10000); the result lists at most the cap of ids (default 1000). |
149
+ | `SUBSTRATE_TYPE_TABLE_BUDGET` | Type tables above which the storage report warns (default 500). The deprecated `SUBSTRATE_PARTITION_BUDGET` is read while it is unset, with one warning. |
143
150
  | `SUBSTRATE_SWEEP_INTERVAL_MS` | Edge-sweep cadence (default 300000) |
151
+ | `SUBSTRATE_INDEX_DISK_PATH` | The database volume's mount, when the database is on this host: free space there is checked before each background index build. `off` skips the check. |
152
+ | `SUBSTRATE_INDEX_DISK_MARGIN_MB` | Free space a build must leave (default 2048; `0` requires only the estimate). |
153
+ | `SUBSTRATE_INDEX_RETRY_INTERVAL_MS` | How often pending or failed index builds are retried (default 600000; `0` leaves them to the next boot). |
154
+ | `SUBSTRATE_ADVISOR_INTERVAL_MS` | How often the index advisor writes `_system/IndexRecommendation` rows (default 600000; `0` turns it off). |
155
+ | `SUBSTRATE_PROMOTE_INTERVAL_MS` | How often the promotion sweep gives fields added at runtime (`propose_field`) their own column (default 21600000, 6 hours; `0` stops the timer, and `substrate-admin promote-fields` still runs it). |
156
+ | `SUBSTRATE_STORE_PROFILE` | `personal`, `growth` or `scale`: seeds `_system/StoreProfile` once, so changing it later does not change an existing store. |
157
+ | `SUBSTRATE_STORE_WRITE_HEAVY` | `1` or `true`, read on every boot: halves the write rate at which the advisor recommends removing an unused index. |
144
158
  | `SUBSTRATE_CORS_ORIGINS` | Comma-separated allowed origins, or `*`. Unset registers no CORS handling. |
145
159
 
146
160
  Natural-language query and the recall embedder are not configured by environment: they read `substrate-nl.config.json` and `substrate-recall.config.json` in the plugin directory.
@@ -149,6 +163,10 @@ Natural-language query and the recall embedder are not configured by environment
149
163
 
150
164
  - [Getting started](https://github.com/mnmal-ai/substrate/blob/main/docs/getting-started.md)
151
165
  - [Design spec](https://github.com/mnmal-ai/substrate/blob/main/docs/substrate-from-first-principles.md)
166
+ - [Operator decisions](https://github.com/mnmal-ai/substrate/blob/main/docs/operator-decisions.md)
167
+ - [Upgrading to per-type tables](https://github.com/mnmal-ai/substrate/blob/main/docs/upgrading-to-type-tables.md)
168
+ - [Upgrading to the warehouse grammar](https://github.com/mnmal-ai/substrate/blob/main/docs/upgrading-to-warehouse-grammar.md)
169
+ - [Upgrading to 1.0.0-beta.1](https://github.com/mnmal-ai/substrate/blob/main/docs/upgrading-to-beta.md)
152
170
  - [Plugin author guide](https://github.com/mnmal-ai/substrate/blob/main/docs/plugin-author-guide.md)
153
171
 
154
172
  ## License