@mnmal-ai/substrate 1.0.0-alpha.1 → 1.0.0-alpha.11

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (47) hide show
  1. package/README.md +48 -6
  2. package/dist/boot-store-C5vNZJ3i.js +4065 -0
  3. package/dist/{runtime-metrics-DwpDJ8vz.d.ts → bridge-COu0Tg_h.d.ts} +2 -87
  4. package/dist/cli-BrwW4tjO.js +3144 -0
  5. package/dist/client.d.ts +129 -8
  6. package/dist/client.js +6 -5
  7. package/dist/{agent-token-BIboWOU0.js → env-utils-CGNazUNt.js} +31 -2
  8. package/dist/{errors-D4vKuwz8.d.ts → errors-BOllGhVx.d.ts} +15 -2
  9. package/dist/{index-D7mSOFlM.d.ts → index-Cp74OcmS.d.ts} +1 -1
  10. package/dist/node-version-C5FSmwJX.js +26 -0
  11. package/dist/pack.d.ts +2 -2
  12. package/dist/pack.js +1 -1
  13. package/dist/pg-edges-store-BLsbcXkk.js +381 -0
  14. package/dist/{plugin-BuQQEzL-.js → plugin-DIYm2cZE.js} +171 -18
  15. package/dist/proposals-bootstrap.js +6 -4
  16. package/dist/proposals.config.schema.json +5 -0
  17. package/dist/proposals.d.ts +14 -4
  18. package/dist/proposals.default-config.json +2 -1
  19. package/dist/proposals.js +2 -2
  20. package/dist/recall-bootstrap.js +13 -4
  21. package/dist/recall.config.schema.json +12 -0
  22. package/dist/recall.d.ts +53 -9
  23. package/dist/recall.js +2 -2
  24. package/dist/{repl-C52wyfMs.js → repl-B8gy0YM7.js} +63 -30
  25. package/dist/{row-ByLFE2Wx.d.ts → row-DwaDU4b-.d.ts} +58 -9
  26. package/dist/run-B18qtGKc.js +2012 -0
  27. package/dist/{plugin-t2g2wX0b.d.ts → schema-CijHRfuk.d.ts} +957 -16
  28. package/dist/script-C6DSa_Na.js +134 -0
  29. package/dist/sdk.d.ts +104 -16
  30. package/dist/sdk.js +5 -110
  31. package/dist/{src-DuopV8u7.js → src-Br6aDUti.js} +153 -20
  32. package/dist/{src-5hVbeiAu.js → src-D041j6GS.js} +697 -57
  33. package/dist/{src-YXlv_RyL.js → src-DK7VEQT6.js} +4099 -1040
  34. package/dist/{src-C4DzaIQZ.js → src-Dbvuel6w.js} +392 -282
  35. package/dist/{row-DK7lPZVe.js → src-DpPdw2oL.js} +11 -11
  36. package/dist/{src-zcyUofMJ.js → src-DptNGNYQ.js} +39 -15
  37. package/dist/src-gz4I1F29.js +5097 -0
  38. package/dist/substrate-admin.js +17 -781
  39. package/dist/substrate.js +4 -5476
  40. package/dist/testing.d.ts +582 -9
  41. package/dist/testing.js +39 -3
  42. package/dist/{ws-C72iVS5T.d.ts → ws-DC8OhMaV.d.ts} +2 -3
  43. package/package.json +22 -20
  44. package/dist/dataset-DOXGRz50.js +0 -1535
  45. package/dist/env-utils-B7SGmoeJ.js +0 -21
  46. package/dist/schema-g4FQLr0V.d.ts +0 -433
  47. package/dist/src-CjEwiKXw.js +0 -2327
package/README.md CHANGED
@@ -2,7 +2,7 @@
2
2
 
3
3
  The `substrate` binary: the server, the stdio MCP server, and the runtime scaffolder, plus `substrate-admin` for key management. The same package carries the libraries for plugin authors and clients, and two built-in plugins: `substrate-recall` (kernel) and `substrate-proposals` (coordinator).
4
4
 
5
- It runs as a real install: the binary and its plugins resolve from one shared `node_modules`, so every plugin sees the same copy of every shared package. `substrate init` scaffolds such a runtime directory. Node `>=26.2.0` is required; a durable deployment also needs Postgres with the `pgvector` extension. Walkthrough: [`docs/getting-started.md`](https://github.com/mnmal-ai/substrate/blob/main/docs/getting-started.md).
5
+ It runs as a real install: the binary and its plugins resolve from one shared `node_modules`, so every plugin sees the same copy of every shared package. `substrate init` scaffolds such a runtime directory. Node `>=26.12.1` is required; a durable deployment also needs PostgreSQL 18 or newer with the `pgvector` extension. Walkthrough: [`docs/getting-started.md`](https://github.com/mnmal-ai/substrate/blob/main/docs/getting-started.md).
6
6
 
7
7
  ## Library subpaths
8
8
 
@@ -11,7 +11,7 @@ It runs as a real install: the binary and its plugins resolve from one shared `n
11
11
  | `@mnmal-ai/substrate/sdk` | Plugin authoring: `defineNamespace`, the field DSL, autoCRUD, engine helpers, `defineBootstrap`, policy combinators, route helpers, and all shared types |
12
12
  | `@mnmal-ai/substrate/client` | `SubstrateClient`, `watchRows`, client errors, the admin API, and the token and grant helpers |
13
13
  | `@mnmal-ai/substrate/pack` | The MessagePack wire codec |
14
- | `@mnmal-ai/substrate/testing` | `compose`, `MemorySubstrateServer`, `MemoryDataset`, `MemoryEdgesStore`, `loadPlugins`, `FakeEmbedder`, and the Postgres test-database guards |
14
+ | `@mnmal-ai/substrate/testing` | `compose`, `MemorySubstrateServer`, `MemoryDataset`, `MemoryEdgesStore`, `loadPlugins`, `FakeEmbedder`, `PgDataset`, `PgEdgesStore`, the pg migration runners, and the Postgres test-database guards |
15
15
  | `@mnmal-ai/substrate/recall` | The recall plugin's primitives (`EmbeddingsStore`, `LexicalStore`, `embedAndStore`, …) |
16
16
  | `@mnmal-ai/substrate/proposals` | The proposals plugin's exports |
17
17
 
@@ -25,6 +25,7 @@ A plugin package declares `@mnmal-ai/substrate` as a peer dependency and as a de
25
25
  | `substrate mcp` | stdio MCP server; reads `SUBSTRATE_URL`, and `SUBSTRATE_MCP_TOOLS` (csv) to register only some tools |
26
26
  | `substrate init [dir] [flags]` | Scaffold a runtime directory (interactive; `--yes` for scripted use) |
27
27
  | `substrate ask <question…>` | Ask a natural-language question (needs `substrate-nl`) |
28
+ | `substrate run <script.mts>` | Run a script against a signed, guarded client and print one JSON report |
28
29
  | `substrate federate <verb>` | Federation onboarding: `enroll` / `join` / `check` |
29
30
  | `substrate check-enums [--json]` | Report stored values outside a field's declared enum; exits 1 on drift, so it works as a CI gate |
30
31
  | `substrate normalise-vocab <ns> <Type> <field> [--apply]` | Rewrite stored vocabulary aliases to canonical values (admin) |
@@ -44,9 +45,44 @@ substrate ask [--ns <namespace>] [--project <name>] [--json] <question…>
44
45
 
45
46
  Posts the question to `POST /substrate/v1/nl` on `SUBSTRATE_URL` and prints the answer. `--ns` defaults to `SUBSTRATE_NAMESPACE`, and one of the two is required; `--project` defaults to the current directory's name. Human output leads with the frame line (`▸ <ns>/<Type> · parser · 3 rows`); a frame the engine rejected prints `· rejected` and the errors. `--json` prints the raw `{ outcome, trail }`. Exit codes: `0` answered, `1` refused / engine errors / request failed, `2` usage or configuration error.
46
47
 
48
+ ### `substrate run`
49
+
50
+ ```bash
51
+ substrate run <script.mts> [--args <json>] [--dry-run | --write] [--max-writes <n>] [--out <file>]
52
+ ```
53
+
54
+ Imports the script and calls its default export with a `ScriptContext` (`args`, `query`, `queryAll`, `recall`, `mutate`, `log`), so one call can chain many reads and writes and return one result. The client is signed from `SUBSTRATE_URL`, `SUBSTRATE_AGENT_KEY_ID` and `SUBSTRATE_AGENT_KEY_PATH`; an unsigned run is refused.
55
+
56
+ ```ts
57
+ import type { ScriptContext } from '@mnmal-ai/substrate/client';
58
+
59
+ // substrate run titled.mts --args '{"title":"hello"}'
60
+ export default async function (s: ScriptContext) {
61
+ const notes = await s.queryAll('notes/Note', {
62
+ params: { where: { title: { eq: s.args.title } }, limit: 100 },
63
+ fields: { id: true }
64
+ });
65
+ return { count: notes.length };
66
+ }
67
+ ```
68
+
69
+ Guardrails:
70
+
71
+ - **Read-only by default.** `mutate` throws unless `--dry-run` (records each payload, sends nothing, returns `null`) or `--write` (sends up to `--max-writes` frames, default 50).
72
+ - **A rejected frame throws.** A query whose key is missing from `data`, or a mutate the server refused, fails the run with the server's error rather than reading as empty.
73
+ - **Truncation is reported.** `query` adds a warning when a result has `hasMore`; `queryAll` pages to the end.
74
+
75
+ `s.recall({ query, namespace, type, k?, threshold?, where?, row? })` returns hits best first, through the `recall/Hit` frame. Each hit's `row` is `id` plus the handle fields; pass `row: { body: true }` (or `true`) for more, on the hits you will read.
76
+
77
+ `--args` takes a JSON object, which the script reads as `s.args` (`{}` when omitted), so one script serves many calls.
78
+
79
+ Stdout carries one JSON report, `{ ok, kid, mode, args, result, error?, mutations, warnings, stats }`. `mutations` lists every payload with `applied`, including those sent before a failure. `--out` writes the report to a file and prints a one-line summary. `.mts` scripts run as-is: the Node this package requires strips types natively. Exit codes: `0` ok, `1` the script failed, `2` usage or configuration error.
80
+
81
+ The pieces are exported for use outside the CLI: `createClientFromEnv()` builds the same signed client, and `runScript()` runs a function against the guarded context.
82
+
47
83
  ### `substrate-admin`
48
84
 
49
- A separate binary for key management: `bootstrap` (seed the genesis admin key), `doctor` (check a deployment end to end), `onboard`, `list`, `register`, `update`, `rotate`, `revoke`. `substrate-admin --help` documents each.
85
+ A separate binary for key management: `bootstrap` (seed the genesis admin key), `doctor` (check a deployment end to end), `onboard`, `list`, `register`, `update`, `rotate`, `revoke`, and `migrate-store` (copy a store into a fresh, empty, partitioned PostgreSQL 18 store and verify it; see [Upgrading](https://github.com/mnmal-ai/substrate/blob/main/packages/substrate-dataset-pg/README.md#upgrading)). `substrate-admin --help` documents each.
50
86
 
51
87
  ## Install
52
88
 
@@ -63,9 +99,9 @@ set -a && source .env && set +a
63
99
  ./node_modules/.bin/substrate serve
64
100
  ```
65
101
 
66
- `substrate init` writes a `package.json` pinning `@mnmal-ai/substrate` and the chosen optional plugins, a `.npmrc` for the `@mnmal-ai` registry, a `.env` with `SUBSTRATE_PLUGIN_DIR=.`, a `substrate-recall.config.json` for the embedder, and, unless `--store=memory`, a `docker-compose.yml` running `pgvector/pgvector:pg17`. Flags: `--store=memory|postgres`, `--port`, `--pg-port`, `--plugins=<csv>` (from `cortext`, `substrate-nl`, `substrate-sentinel`; none by default), `--embedder=minilm|bge-base|ollama|none`, `--tls --tls-cert-path --tls-key-path`, `--substrate-url`, `--yes`, `--force`.
102
+ `substrate init` writes a `package.json` pinning `@mnmal-ai/substrate` and the chosen optional plugins, a `pnpm-workspace.yaml`, a `.env` with `SUBSTRATE_PLUGIN_DIR=.`, a `substrate-recall.config.json` for the embedder, and, unless `--store=memory`, a `docker-compose.yml` running `pgvector/pgvector:pg18`. Flags: `--store=memory|postgres`, `--port`, `--pg-port`, `--plugins=<csv>` (from `cortext`, `substrate-nl`, `substrate-sentinel`; none by default), `--embedder=bge-base|minilm|ollama|none` (default `bge-base`), `--tls --tls-cert-path --tls-key-path`, `--substrate-url`, `--yes`, `--force`.
67
103
 
68
- `SUBSTRATE_PLUGIN_DIR` is the directory whose `node_modules/` holds the plugins; in a scaffolded runtime that is the runtime directory itself. Recall and proposals load from `@mnmal-ai/substrate` itself, so they are present whenever the binary is installed there. Kernel plugins (`substrate-recall`, `substrate-nl`) compose first, then coordinators (`substrate-proposals`, `substrate-sentinel`, …), alphabetical within each tier. Per-plugin overrides live at `<SUBSTRATE_PLUGIN_DIR>/<pkg-basename>.config.json`. With `SUBSTRATE_PLUGIN_DIR` unset the binary boots with only the `_system` admin namespace and `/health`.
104
+ `SUBSTRATE_PLUGIN_DIR` is the directory whose `node_modules/` holds the plugins; in a scaffolded runtime that is the runtime directory itself. Recall and proposals load from `@mnmal-ai/substrate` itself, so they are present whenever the binary is installed there. Kernel plugins (`substrate-recall`, `substrate-nl`) compose first, then coordinators (`substrate-proposals`, `substrate-sentinel`, …), alphabetical within each tier. Per-plugin overrides live at `<SUBSTRATE_PLUGIN_DIR>/<pkg-basename>.config.json`. With `SUBSTRATE_PLUGIN_DIR` unset the binary boots with only the `_system` admin namespace and `/health`. Set, it must hold a `package.json` listing the plugins under `dependencies`, installed into its `node_modules/`; a missing `package.json`, or listed plugins with no `node_modules/`, refuses boot with a message naming the fix.
69
105
 
70
106
  ### Adding a plugin
71
107
 
@@ -82,7 +118,7 @@ cat > substrate-recall.config.json <<'EOF'
82
118
  EOF
83
119
  ```
84
120
 
85
- `kind: "onnx"` runs the model in-process and needs `fastembed` installed in the runtime (the embedder code itself ships in `@mnmal-ai/substrate`); `substrate init` adds it for `--embedder=minilm` (the default) and `--embedder=bge-base`. `kind: "ollama"` takes a `baseUrl` instead.
121
+ `kind: "onnx"` runs the model in-process and needs `fastembed` installed in the runtime (the embedder code itself ships in `@mnmal-ai/substrate`); `substrate init` adds it for `--embedder=bge-base` (the default) and `--embedder=minilm`. `kind: "ollama"` takes a `baseUrl` instead.
86
122
 
87
123
  ## Docker
88
124
 
@@ -105,6 +141,12 @@ The main ones; `substrate --help` and the header of `apps/substrate/src/main.ts`
105
141
  | `SUBSTRATE_TLS_KEY_PATH` / `SUBSTRATE_TLS_CERT_PATH` / `SUBSTRATE_TLS_CA_PATH` | Serve HTTPS. Key and cert are required together. |
106
142
  | `SUBSTRATE_NAMESPACE` | Default namespace for recall requests that omit one. |
107
143
  | `SUBSTRATE_SWEEP_INTERVAL_MS` | Edge-sweep cadence (default 300000) |
144
+ | `SUBSTRATE_INDEX_DISK_PATH` | The database volume's mount, when the database is on this host: free space there is checked before each background index build. `off` skips the check. |
145
+ | `SUBSTRATE_INDEX_DISK_MARGIN_MB` | Free space a build must leave (default 2048; `0` requires only the estimate). |
146
+ | `SUBSTRATE_INDEX_RETRY_INTERVAL_MS` | How often pending or failed index builds are retried (default 600000; `0` leaves them to the next boot). |
147
+ | `SUBSTRATE_ADVISOR_INTERVAL_MS` | How often the index advisor writes `_system/IndexRecommendation` rows (default 600000; `0` turns it off). |
148
+ | `SUBSTRATE_STORE_PROFILE` | `personal`, `growth` or `scale`: seeds `_system/StoreProfile` once, so changing it later does not change an existing store. |
149
+ | `SUBSTRATE_STORE_WRITE_HEAVY` | `1` or `true`, read on every boot: halves the write rate at which the advisor recommends removing an unused index. |
108
150
  | `SUBSTRATE_CORS_ORIGINS` | Comma-separated allowed origins, or `*`. Unset registers no CORS handling. |
109
151
 
110
152
  Natural-language query and the recall embedder are not configured by environment: they read `substrate-nl.config.json` and `substrate-recall.config.json` in the plugin directory.