@mnmal-ai/substrate 1.0.0-alpha.8 → 1.0.0-beta.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +32 -14
- package/dist/boot-store-BMQUbHKv.js +4086 -0
- package/dist/{runtime-metrics-DwpDJ8vz.d.ts → bridge-DnToJedN.d.ts} +2 -87
- package/dist/cli-DModIbnN.js +3390 -0
- package/dist/client.browser.js +2507 -0
- package/dist/client.d.ts +222 -171
- package/dist/client.js +6 -7
- package/dist/cluster-worker.js +0 -1
- package/dist/{content-type-C7T28xNb.js → content-type-wLljot5M.js} +0 -1
- package/dist/{env-utils-CGNazUNt.js → env-utils-s25Z1YSA.js} +0 -1
- package/dist/{errors-D4vKuwz8.d.ts → errors-Cmkp1qxf.d.ts} +44 -4
- package/dist/{fake-DUVtKVSE.js → fake-DdXtb-m8.js} +0 -1
- package/dist/{index-D7mSOFlM.d.ts → index-BI4KqFiR.d.ts} +7 -11
- package/dist/msgpackr-Dq18g6Ds.js +16 -0
- package/dist/node-version-BtK_Nq6z.js +32 -0
- package/dist/pack.d.ts +75 -11
- package/dist/pack.js +4 -4
- package/dist/{plugin-B3NxUCVn.js → plugin-CEOhrEme.js} +149 -6
- package/dist/proposals-bootstrap.js +6 -5
- package/dist/proposals.config.schema.json +5 -0
- package/dist/proposals.d.ts +17 -5
- package/dist/proposals.default-config.json +2 -1
- package/dist/proposals.js +2 -3
- package/dist/recall-bootstrap.js +5 -4
- package/dist/recall.config.schema.json +6 -0
- package/dist/recall.d.ts +27 -6
- package/dist/recall.js +2 -3
- package/dist/{repl-DeKzfvNV.js → repl-BvDZy-Ut.js} +52 -26
- package/dist/{row-ByLFE2Wx.d.ts → row-D0If4qif.d.ts} +92 -12
- package/dist/run-BT123RV1.js +2443 -0
- package/dist/{plugin-t2g2wX0b.d.ts → schema-aIkY8sR3.d.ts} +1293 -63
- package/dist/{script-C6DSa_Na.js → script-ByQ9pzhC.js} +54 -26
- package/dist/script-runner.js +78 -0
- package/dist/sdk.d.ts +236 -38
- package/dist/sdk.js +5 -110
- package/dist/semver-D_zdq0hi.js +17 -0
- package/dist/src-BMD2OE--.js +8289 -0
- package/dist/{src-BLgbLq2k.js → src-B_kHkbsr.js} +780 -330
- package/dist/{src-DjIuRjzF.js → src-CKSJ13p8.js} +10231 -4237
- package/dist/{src-CkZw6xVD.js → src-CW7eRDss.js} +188 -87
- package/dist/{row-DZ0Fnpc6.js → src-DmiUWOLp.js} +2 -1
- package/dist/{src-DCy7Iy2l.js → src-DmjwFMu8.js} +900 -77
- package/dist/{src-DiNTSaFk.js → src-FXO5JPyk.js} +686 -856
- package/dist/substrate-admin.js +17 -782
- package/dist/substrate.js +4 -5206
- package/dist/testing.d.ts +473 -96
- package/dist/testing.js +19 -165
- package/dist/{ws-C72iVS5T.d.ts → ws-DW8ng02H.d.ts} +4 -5
- package/package.json +26 -26
- package/dist/dataset-jZS_4ci4.js +0 -1535
- package/dist/pg-edges-store-KH_FQCxx.js +0 -381
- package/dist/schema-g4FQLr0V.d.ts +0 -433
- package/dist/src-BD27zeGF.js +0 -2339
package/README.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
The `substrate` binary: the server, the stdio MCP server, and the runtime scaffolder, plus `substrate-admin` for key management. The same package carries the libraries for plugin authors and clients, and two built-in plugins: `substrate-recall` (kernel) and `substrate-proposals` (coordinator).
|
|
4
4
|
|
|
5
|
-
It runs as a real install: the binary and its plugins resolve from one shared `node_modules`, so every plugin sees the same copy of every shared package. `substrate init` scaffolds such a runtime directory. Node `>=26.
|
|
5
|
+
It runs as a real install: the binary and its plugins resolve from one shared `node_modules`, so every plugin sees the same copy of every shared package. `substrate init` scaffolds such a runtime directory. Node `>=26.13.1` is required; a durable deployment also needs PostgreSQL 18 or newer with the `pgvector` extension. Walkthrough: [`docs/getting-started.md`](https://github.com/mnmal-ai/substrate/blob/main/docs/getting-started.md).
|
|
6
6
|
|
|
7
7
|
## Library subpaths
|
|
8
8
|
|
|
@@ -21,7 +21,7 @@ A plugin package declares `@mnmal-ai/substrate` as a peer dependency and as a de
|
|
|
21
21
|
|
|
22
22
|
| Subcommand | Purpose |
|
|
23
23
|
|---|---|
|
|
24
|
-
| `substrate serve` | Boot the HTTP
|
|
24
|
+
| `substrate serve` | Boot the HTTP server (default port 3100) |
|
|
25
25
|
| `substrate mcp` | stdio MCP server; reads `SUBSTRATE_URL`, and `SUBSTRATE_MCP_TOOLS` (csv) to register only some tools |
|
|
26
26
|
| `substrate init [dir] [flags]` | Scaffold a runtime directory (interactive; `--yes` for scripted use) |
|
|
27
27
|
| `substrate ask <question…>` | Ask a natural-language question (needs `substrate-nl`) |
|
|
@@ -35,7 +35,7 @@ A plugin package declares `@mnmal-ai/substrate` as a peer dependency and as a de
|
|
|
35
35
|
|
|
36
36
|
### MCP tools
|
|
37
37
|
|
|
38
|
-
`substrate mcp` serves `schema`, `query`, `mutate`, `recall`, `ask`, `whoami`, `propose_type`, `propose_field`, and the admin-gated `drop_namespace` / `drop_type`. It talks to a running server at `SUBSTRATE_URL` and signs requests when `SUBSTRATE_AGENT_KEY_ID` and `SUBSTRATE_AGENT_KEY_PATH` are set. [`docs/surface-parity.md`](https://github.com/mnmal-ai/substrate/blob/main/docs/surface-parity.md) maps what each surface (HTTP, MCP,
|
|
38
|
+
`substrate mcp` serves `schema`, `query`, `mutate`, `run`, `recall`, `ask`, `whoami`, `propose_type`, `propose_field`, `propose_index`, and the admin-gated `drop_namespace` / `drop_type`. `run` executes a short script against the store in one call (the same context as `substrate run`), in a child process with no file, network, process or environment access. It talks to a running server at `SUBSTRATE_URL` and signs requests when `SUBSTRATE_AGENT_KEY_ID` and `SUBSTRATE_AGENT_KEY_PATH` are set. [`docs/surface-parity.md`](https://github.com/mnmal-ai/substrate/blob/main/docs/surface-parity.md) maps what each surface (HTTP, MCP, frame streams) offers.
|
|
39
39
|
|
|
40
40
|
### `substrate ask`
|
|
41
41
|
|
|
@@ -51,7 +51,7 @@ Posts the question to `POST /substrate/v1/nl` on `SUBSTRATE_URL` and prints the
|
|
|
51
51
|
substrate run <script.mts> [--args <json>] [--dry-run | --write] [--max-writes <n>] [--out <file>]
|
|
52
52
|
```
|
|
53
53
|
|
|
54
|
-
Imports the script and calls its default export with a `ScriptContext` (`args`, `query`, `queryAll`, `recall`, `mutate`, `log`), so one call can chain many reads and writes and return one result. The client is signed from `SUBSTRATE_URL`, `SUBSTRATE_AGENT_KEY_ID` and `SUBSTRATE_AGENT_KEY_PATH`; an unsigned run is refused.
|
|
54
|
+
Imports the script and calls its default export with a `ScriptContext` (`args`, `query`, `queryAll`, `byShortId`, `recall`, `mutate`, `log`), so one call can chain many reads and writes and return one result. The client is signed from `SUBSTRATE_URL`, `SUBSTRATE_AGENT_KEY_ID` and `SUBSTRATE_AGENT_KEY_PATH`; an unsigned run is refused.
|
|
55
55
|
|
|
56
56
|
```ts
|
|
57
57
|
import type { ScriptContext } from '@mnmal-ai/substrate/client';
|
|
@@ -72,7 +72,9 @@ Guardrails:
|
|
|
72
72
|
- **A rejected frame throws.** A query whose key is missing from `data`, or a mutate the server refused, fails the run with the server's error rather than reading as empty.
|
|
73
73
|
- **Truncation is reported.** `query` adds a warning when a result has `hasMore`; `queryAll` pages to the end.
|
|
74
74
|
|
|
75
|
-
`s.
|
|
75
|
+
`s.byShortId('cortext/Todo', 1787)` returns that row, and `s.byShortId('cortext/Todo', [1787, 1785], { title: true })` returns the rows in input order, through `where: { shortId: { eq | in } }`. It throws naming every shortId with no row, rather than returning fewer rows than asked for.
|
|
76
|
+
|
|
77
|
+
`s.recall({ query, namespace, type, k?, threshold?, where?, row? })` returns hits best first, through the `recall/Hit` frame, typed as `RecallHit` (exported beside `ScriptContext`). Each hit's `row` is `id` plus the handle fields (`name`, `title`, `topic`, `synopsis`, `description`, each typed `string | undefined`, so `h.row.name` needs no cast); pass `row: { body: true }` (or `true`) for more, on the hits you will read.
|
|
76
78
|
|
|
77
79
|
`--args` takes a JSON object, which the script reads as `s.args` (`{}` when omitted), so one script serves many calls.
|
|
78
80
|
|
|
@@ -82,7 +84,7 @@ The pieces are exported for use outside the CLI: `createClientFromEnv()` builds
|
|
|
82
84
|
|
|
83
85
|
### `substrate-admin`
|
|
84
86
|
|
|
85
|
-
A separate binary for key management: `bootstrap` (seed the genesis admin key), `doctor` (check a deployment end to end), `onboard`, `list`, `register`, `update`, `rotate`, `revoke
|
|
87
|
+
A separate binary for key management and store operations: `bootstrap` (seed the genesis admin key), `doctor` (check a deployment end to end), `onboard`, `list`, `register`, `update`, `rotate`, `revoke`, `promote-fields` (run the promotion sweep now on a running server, giving fields added at runtime their columns), and `migrate-store` (copy a store into a fresh, empty PostgreSQL 18 store on per-type tables and verify it; see [Upgrading to per-type tables](https://github.com/mnmal-ai/substrate/blob/main/docs/upgrading-to-type-tables.md)). `substrate-admin --help` documents each.
|
|
86
88
|
|
|
87
89
|
## Install
|
|
88
90
|
|
|
@@ -99,9 +101,9 @@ set -a && source .env && set +a
|
|
|
99
101
|
./node_modules/.bin/substrate serve
|
|
100
102
|
```
|
|
101
103
|
|
|
102
|
-
`substrate init` writes a `package.json` pinning `@mnmal-ai/substrate` and the chosen optional plugins, a `pnpm-workspace.yaml`, a `.env` with `SUBSTRATE_PLUGIN_DIR=.`, a `substrate-recall.config.json` for the embedder, and, unless `--store=memory`, a `docker-compose.yml` running `pgvector/pgvector:
|
|
104
|
+
`substrate init` writes a `package.json` pinning `@mnmal-ai/substrate` and the chosen optional plugins, a `pnpm-workspace.yaml`, a `.env` with `SUBSTRATE_PLUGIN_DIR=.`, a `substrate-recall.config.json` for the embedder, and, unless `--store=memory`, a `docker-compose.yml` running `pgvector/pgvector:pg18`. Flags: `--store=memory|postgres`, `--port`, `--pg-port`, `--plugins=<csv>` (from `cortext`, `substrate-nl`, `substrate-sentinel`; none by default), `--embedder=bge-base|minilm|ollama|none` (default `bge-base`), `--tls --tls-cert-path --tls-key-path`, `--substrate-url`, `--yes`, `--force`.
|
|
103
105
|
|
|
104
|
-
`SUBSTRATE_PLUGIN_DIR` is the directory whose `node_modules/` holds the plugins; in a scaffolded runtime that is the runtime directory itself. Recall and proposals load from `@mnmal-ai/substrate` itself, so they are present whenever the binary is installed there. Kernel plugins (`substrate-recall`, `substrate-nl`) compose first, then coordinators (`substrate-proposals`, `substrate-sentinel`, …), alphabetical within each tier. Per-plugin overrides live at `<SUBSTRATE_PLUGIN_DIR>/<pkg-basename>.config.json`. With `SUBSTRATE_PLUGIN_DIR` unset the binary boots with only the `_system` admin namespace and `/health`.
|
|
106
|
+
`SUBSTRATE_PLUGIN_DIR` is the directory whose `node_modules/` holds the plugins; in a scaffolded runtime that is the runtime directory itself. Recall and proposals load from `@mnmal-ai/substrate` itself, so they are present whenever the binary is installed there. Kernel plugins (`substrate-recall`, `substrate-nl`) compose first, then coordinators (`substrate-proposals`, `substrate-sentinel`, …), alphabetical within each tier. Per-plugin overrides live at `<SUBSTRATE_PLUGIN_DIR>/<pkg-basename>.config.json`. With `SUBSTRATE_PLUGIN_DIR` unset the binary boots with only the `_system` admin namespace and `/health`. Set, it must hold a `package.json` listing the plugins under `dependencies`, installed into its `node_modules/`; a missing `package.json`, or listed plugins with no `node_modules/`, refuses boot with a message naming the fix.
|
|
105
107
|
|
|
106
108
|
### Adding a plugin
|
|
107
109
|
|
|
@@ -126,21 +128,33 @@ The repository's `docker-compose.yml` runs this binary with `substrate-nl` and `
|
|
|
126
128
|
|
|
127
129
|
## Environment variables
|
|
128
130
|
|
|
129
|
-
The main ones; `substrate --help` and the header of `apps/substrate/src/main.ts` list the rest.
|
|
131
|
+
The main ones; `substrate --help` and the header of `apps/substrate/src/main.ts` list the rest. [Operator decisions](https://github.com/mnmal-ai/substrate/blob/main/docs/operator-decisions.md) explains what each choice costs.
|
|
130
132
|
|
|
131
133
|
| Var | Purpose |
|
|
132
134
|
|---|---|
|
|
133
|
-
| `SUBSTRATE_DATABASE_URL` | `memory` (default, ephemeral) or `postgres
|
|
135
|
+
| `SUBSTRATE_DATABASE_URL` | `memory` (default, ephemeral) or `postgres://...`. A Postgres store keeps each type in a table of its own; a store in an old layout (`legacy` or `partitioned`) is refused at boot, naming `substrate-admin migrate-store`, which copies it into a fresh one. `SUBSTRATE_STORAGE_LAYOUT` is no longer read: setting it logs one warning. |
|
|
134
136
|
| `SUBSTRATE_PLUGIN_DIR` | Plugin install root. The loader scans `<dir>/node_modules/{*,@*/*}/package.json` for packages with a `substrate` manifest and merges each plugin's `default-config.json` with `<dir>/<basename>.config.json`. |
|
|
135
|
-
| `PORT` / `HOST` | Listen address (default `3100`
|
|
137
|
+
| `PORT` / `HOST` | Listen address (default `3100`; `HOST` defaults to `0.0.0.0` with auth on and `127.0.0.1` with auth off) |
|
|
138
|
+
| `SUBSTRATE_ALLOW_UNAUTHENTICATED_HOST` | `1` or `true` permits a non-loopback `HOST` with auth off; without it that combination refuses boot. |
|
|
136
139
|
| `SUBSTRATE_SYSTEM_ACTOR` | Who the deployment writes as while auth is off. Required in that case: the server refuses to boot rather than write unattributed rows. |
|
|
137
140
|
| `SUBSTRATE_AUTH_MODE` | `required` or `attribute` (anonymous callers allowed on non-admin routes). Setting it, or `SUBSTRATE_AUTH_KEY_REGISTRY`, enables auth. |
|
|
138
141
|
| `SUBSTRATE_AUTH_KEY_REGISTRY` | Path to a JSON key registry file. Without it, auth reads keys registered in Postgres with `substrate-admin`. |
|
|
139
142
|
| `SUBSTRATE_PUBLIC_ORIGIN` | The origins callers reach this server by, comma-separated (`https://host:3100,https://alias`). A token whose `aud` names none of them is refused. **Required when auth is enabled and `HOST` is not loopback**; on loopback it defaults to the loopback origins. |
|
|
140
|
-
| `
|
|
141
|
-
| `SUBSTRATE_TLS_KEY_PATH` / `SUBSTRATE_TLS_CERT_PATH` / `SUBSTRATE_TLS_CA_PATH` | Serve HTTPS. Key and cert are required together. |
|
|
142
|
-
| `SUBSTRATE_NAMESPACE` | Default namespace for recall requests that omit one
|
|
143
|
+
| `SUBSTRATE_CURSOR_SECRET` | Seals pagination cursors under row-filter policies, so every process sharing it opens the same cursors. Unset, the key is per process and a sealed cursor stops opening after a restart or on another replica. When it is unset, the retired `SUBSTRATE_STREAM_TICKET_SECRET` is read instead. Rotating it ends every outstanding sealed cursor. |
|
|
144
|
+
| `SUBSTRATE_TLS_KEY_PATH` / `SUBSTRATE_TLS_CERT_PATH` / `SUBSTRATE_TLS_CA_PATH` | Serve HTTPS, over HTTP/2 (the server is HTTP/2 only; without TLS it is h2c). Key and cert are required together. |
|
|
145
|
+
| `SUBSTRATE_NAMESPACE` | Default namespace for recall requests that omit one; with neither, recall refuses with `missing_namespace`. |
|
|
146
|
+
| `SUBSTRATE_STATEMENT_TIMEOUT_MS` | Statement timeout for typed edge walk reads (default 30000). A read it cancels is refused `query_timeout`. It also bounds the wait for a free database connection; a request that waits longer is refused `store_busy`. |
|
|
147
|
+
| `SUBSTRATE_PAGE_LIMIT_DEFAULT` / `SUBSTRATE_PAGE_LIMIT_MAX` | Page size for a remote query that names no `limit` (default 100), and the largest `limit` accepted (default 1000). |
|
|
148
|
+
| `SUBSTRATE_FILTER_WRITE_MAX_MATCH` / `SUBSTRATE_FILTER_WRITE_IDS_CAP` | `updateMany<Type>` / `deleteMany<Type>`: a filter matching more rows than the max is refused `filter_too_broad` (default 10000); the result lists at most the cap of ids (default 1000). |
|
|
149
|
+
| `SUBSTRATE_TYPE_TABLE_BUDGET` | Type tables above which the storage report warns (default 500). The deprecated `SUBSTRATE_PARTITION_BUDGET` is read while it is unset, with one warning. |
|
|
143
150
|
| `SUBSTRATE_SWEEP_INTERVAL_MS` | Edge-sweep cadence (default 300000) |
|
|
151
|
+
| `SUBSTRATE_INDEX_DISK_PATH` | The database volume's mount, when the database is on this host: free space there is checked before each background index build. `off` skips the check. |
|
|
152
|
+
| `SUBSTRATE_INDEX_DISK_MARGIN_MB` | Free space a build must leave (default 2048; `0` requires only the estimate). |
|
|
153
|
+
| `SUBSTRATE_INDEX_RETRY_INTERVAL_MS` | How often pending or failed index builds are retried (default 600000; `0` leaves them to the next boot). |
|
|
154
|
+
| `SUBSTRATE_ADVISOR_INTERVAL_MS` | How often the index advisor writes `_system/IndexRecommendation` rows (default 600000; `0` turns it off). |
|
|
155
|
+
| `SUBSTRATE_PROMOTE_INTERVAL_MS` | How often the promotion sweep gives fields added at runtime (`propose_field`) their own column (default 21600000, 6 hours; `0` stops the timer, and `substrate-admin promote-fields` still runs it). |
|
|
156
|
+
| `SUBSTRATE_STORE_PROFILE` | `personal`, `growth` or `scale`: seeds `_system/StoreProfile` once, so changing it later does not change an existing store. |
|
|
157
|
+
| `SUBSTRATE_STORE_WRITE_HEAVY` | `1` or `true`, read on every boot: halves the write rate at which the advisor recommends removing an unused index. |
|
|
144
158
|
| `SUBSTRATE_CORS_ORIGINS` | Comma-separated allowed origins, or `*`. Unset registers no CORS handling. |
|
|
145
159
|
|
|
146
160
|
Natural-language query and the recall embedder are not configured by environment: they read `substrate-nl.config.json` and `substrate-recall.config.json` in the plugin directory.
|
|
@@ -149,6 +163,10 @@ Natural-language query and the recall embedder are not configured by environment
|
|
|
149
163
|
|
|
150
164
|
- [Getting started](https://github.com/mnmal-ai/substrate/blob/main/docs/getting-started.md)
|
|
151
165
|
- [Design spec](https://github.com/mnmal-ai/substrate/blob/main/docs/substrate-from-first-principles.md)
|
|
166
|
+
- [Operator decisions](https://github.com/mnmal-ai/substrate/blob/main/docs/operator-decisions.md)
|
|
167
|
+
- [Upgrading to per-type tables](https://github.com/mnmal-ai/substrate/blob/main/docs/upgrading-to-type-tables.md)
|
|
168
|
+
- [Upgrading to the warehouse grammar](https://github.com/mnmal-ai/substrate/blob/main/docs/upgrading-to-warehouse-grammar.md)
|
|
169
|
+
- [Upgrading to 1.0.0-beta.1](https://github.com/mnmal-ai/substrate/blob/main/docs/upgrading-to-beta.md)
|
|
152
170
|
- [Plugin author guide](https://github.com/mnmal-ai/substrate/blob/main/docs/plugin-author-guide.md)
|
|
153
171
|
|
|
154
172
|
## License
|