@mnmal-ai/substrate 1.0.0-alpha.0 → 1.0.0-alpha.10
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +51 -9
- package/dist/boot-store-BVqfFxh1.js +4060 -0
- package/dist/{runtime-metrics-DwpDJ8vz.d.ts → bridge-Cp4PaIYO.d.ts} +2 -87
- package/dist/cli-hSCBywhE.js +3126 -0
- package/dist/client.d.ts +130 -8
- package/dist/client.js +6 -5
- package/dist/{plugin-t2g2wX0b.d.ts → dataset-CejF-Fr_.d.ts} +1640 -1320
- package/dist/{agent-token-BIboWOU0.js → env-utils-CGNazUNt.js} +31 -2
- package/dist/{errors-D4vKuwz8.d.ts → errors-7mycvNGq.d.ts} +1 -1
- package/dist/{index-D7mSOFlM.d.ts → index-DgSfwx7M.d.ts} +1 -1
- package/dist/node-version-C5FSmwJX.js +26 -0
- package/dist/pack.d.ts +2 -2
- package/dist/pack.js +1 -1
- package/dist/pg-edges-store-Lxsl5ZnK.js +381 -0
- package/dist/{plugin-j1jt1l3n.js → plugin-CtwOINCM.js} +171 -18
- package/dist/proposals-bootstrap.js +6 -4
- package/dist/proposals.config.schema.json +5 -0
- package/dist/proposals.d.ts +15 -4
- package/dist/proposals.default-config.json +2 -1
- package/dist/proposals.js +2 -2
- package/dist/recall-bootstrap.js +13 -4
- package/dist/recall.config.schema.json +12 -0
- package/dist/recall.d.ts +54 -9
- package/dist/recall.js +2 -2
- package/dist/{repl-C52wyfMs.js → repl-DqfHelK7.js} +62 -29
- package/dist/{row-ByLFE2Wx.d.ts → row-BKKCXIG6.d.ts} +5 -3
- package/dist/run-B5dTbZ3g.js +1997 -0
- package/dist/{schema-g4FQLr0V.d.ts → schema-DoSc-4ig.d.ts} +44 -7
- package/dist/script-C6DSa_Na.js +134 -0
- package/dist/sdk.d.ts +90 -12
- package/dist/sdk.js +5 -110
- package/dist/{src-D91yij6C.js → src-5SJ5GEIM.js} +1557 -327
- package/dist/{src-DuopV8u7.js → src-7CUSMK66.js} +148 -19
- package/dist/{src-CrEgiMJQ.js → src-Bi849yhn.js} +39 -15
- package/dist/{src-D88LvCX8.js → src-CqT-QWZ_.js} +697 -57
- package/dist/src-CyvV7dCK.js +4333 -0
- package/dist/{row-DK7lPZVe.js → src-Da0t6si5.js} +11 -11
- package/dist/{src-C4DzaIQZ.js → src-hZb8wfvg.js} +12 -1
- package/dist/substrate-admin.js +17 -781
- package/dist/substrate.js +4 -5471
- package/dist/testing.d.ts +557 -8
- package/dist/testing.js +39 -3
- package/dist/{ws-C72iVS5T.d.ts → ws-ByZyHk7m.d.ts} +3 -3
- package/package.json +21 -20
- package/dist/dataset-BXkus75W.js +0 -1527
- package/dist/env-utils-B7SGmoeJ.js +0 -21
- package/dist/src-CzxvmhFn.js +0 -2327
package/README.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
The `substrate` binary: the server, the stdio MCP server, and the runtime scaffolder, plus `substrate-admin` for key management. The same package carries the libraries for plugin authors and clients, and two built-in plugins: `substrate-recall` (kernel) and `substrate-proposals` (coordinator).
|
|
4
4
|
|
|
5
|
-
It runs as a real install: the binary and its plugins resolve from one shared `node_modules`, so every plugin sees the same copy of every shared package. `substrate init` scaffolds such a runtime directory. Node `>=26.
|
|
5
|
+
It runs as a real install: the binary and its plugins resolve from one shared `node_modules`, so every plugin sees the same copy of every shared package. `substrate init` scaffolds such a runtime directory. Node `>=26.12.1` is required; a durable deployment also needs PostgreSQL 18 or newer with the `pgvector` extension. Walkthrough: [`docs/getting-started.md`](https://github.com/mnmal-ai/substrate/blob/main/docs/getting-started.md).
|
|
6
6
|
|
|
7
7
|
## Library subpaths
|
|
8
8
|
|
|
@@ -11,11 +11,11 @@ It runs as a real install: the binary and its plugins resolve from one shared `n
|
|
|
11
11
|
| `@mnmal-ai/substrate/sdk` | Plugin authoring: `defineNamespace`, the field DSL, autoCRUD, engine helpers, `defineBootstrap`, policy combinators, route helpers, and all shared types |
|
|
12
12
|
| `@mnmal-ai/substrate/client` | `SubstrateClient`, `watchRows`, client errors, the admin API, and the token and grant helpers |
|
|
13
13
|
| `@mnmal-ai/substrate/pack` | The MessagePack wire codec |
|
|
14
|
-
| `@mnmal-ai/substrate/testing` | `compose`, `MemorySubstrateServer`, `MemoryDataset`, `MemoryEdgesStore`, `loadPlugins`, `FakeEmbedder`, and the Postgres test-database guards |
|
|
14
|
+
| `@mnmal-ai/substrate/testing` | `compose`, `MemorySubstrateServer`, `MemoryDataset`, `MemoryEdgesStore`, `loadPlugins`, `FakeEmbedder`, `PgDataset`, `PgEdgesStore`, the pg migration runners, and the Postgres test-database guards |
|
|
15
15
|
| `@mnmal-ai/substrate/recall` | The recall plugin's primitives (`EmbeddingsStore`, `LexicalStore`, `embedAndStore`, …) |
|
|
16
16
|
| `@mnmal-ai/substrate/proposals` | The proposals plugin's exports |
|
|
17
17
|
|
|
18
|
-
A plugin package declares `@mnmal-ai/substrate` as a peer dependency and as a dev dependency (`pnpm add -D @mnmal-ai/substrate`).
|
|
18
|
+
A plugin package declares `@mnmal-ai/substrate` as a peer dependency and as a dev dependency (`pnpm add -D @mnmal-ai/substrate@next`).
|
|
19
19
|
|
|
20
20
|
## Subcommands
|
|
21
21
|
|
|
@@ -25,6 +25,7 @@ A plugin package declares `@mnmal-ai/substrate` as a peer dependency and as a de
|
|
|
25
25
|
| `substrate mcp` | stdio MCP server; reads `SUBSTRATE_URL`, and `SUBSTRATE_MCP_TOOLS` (csv) to register only some tools |
|
|
26
26
|
| `substrate init [dir] [flags]` | Scaffold a runtime directory (interactive; `--yes` for scripted use) |
|
|
27
27
|
| `substrate ask <question…>` | Ask a natural-language question (needs `substrate-nl`) |
|
|
28
|
+
| `substrate run <script.mts>` | Run a script against a signed, guarded client and print one JSON report |
|
|
28
29
|
| `substrate federate <verb>` | Federation onboarding: `enroll` / `join` / `check` |
|
|
29
30
|
| `substrate check-enums [--json]` | Report stored values outside a field's declared enum; exits 1 on drift, so it works as a CI gate |
|
|
30
31
|
| `substrate normalise-vocab <ns> <Type> <field> [--apply]` | Rewrite stored vocabulary aliases to canonical values (admin) |
|
|
@@ -44,15 +45,50 @@ substrate ask [--ns <namespace>] [--project <name>] [--json] <question…>
|
|
|
44
45
|
|
|
45
46
|
Posts the question to `POST /substrate/v1/nl` on `SUBSTRATE_URL` and prints the answer. `--ns` defaults to `SUBSTRATE_NAMESPACE`, and one of the two is required; `--project` defaults to the current directory's name. Human output leads with the frame line (`▸ <ns>/<Type> · parser · 3 rows`); a frame the engine rejected prints `· rejected` and the errors. `--json` prints the raw `{ outcome, trail }`. Exit codes: `0` answered, `1` refused / engine errors / request failed, `2` usage or configuration error.
|
|
46
47
|
|
|
48
|
+
### `substrate run`
|
|
49
|
+
|
|
50
|
+
```bash
|
|
51
|
+
substrate run <script.mts> [--args <json>] [--dry-run | --write] [--max-writes <n>] [--out <file>]
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
Imports the script and calls its default export with a `ScriptContext` (`args`, `query`, `queryAll`, `recall`, `mutate`, `log`), so one call can chain many reads and writes and return one result. The client is signed from `SUBSTRATE_URL`, `SUBSTRATE_AGENT_KEY_ID` and `SUBSTRATE_AGENT_KEY_PATH`; an unsigned run is refused.
|
|
55
|
+
|
|
56
|
+
```ts
|
|
57
|
+
import type { ScriptContext } from '@mnmal-ai/substrate/client';
|
|
58
|
+
|
|
59
|
+
// substrate run titled.mts --args '{"title":"hello"}'
|
|
60
|
+
export default async function (s: ScriptContext) {
|
|
61
|
+
const notes = await s.queryAll('notes/Note', {
|
|
62
|
+
params: { where: { title: { eq: s.args.title } }, limit: 100 },
|
|
63
|
+
fields: { id: true }
|
|
64
|
+
});
|
|
65
|
+
return { count: notes.length };
|
|
66
|
+
}
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
Guardrails:
|
|
70
|
+
|
|
71
|
+
- **Read-only by default.** `mutate` throws unless `--dry-run` (records each payload, sends nothing, returns `null`) or `--write` (sends up to `--max-writes` frames, default 50).
|
|
72
|
+
- **A rejected frame throws.** A query whose key is missing from `data`, or a mutate the server refused, fails the run with the server's error rather than reading as empty.
|
|
73
|
+
- **Truncation is reported.** `query` adds a warning when a result has `hasMore`; `queryAll` pages to the end.
|
|
74
|
+
|
|
75
|
+
`s.recall({ query, namespace, type, k?, threshold?, where?, row? })` returns hits best first, through the `recall/Hit` frame. Each hit's `row` is `id` plus the handle fields; pass `row: { body: true }` (or `true`) for more, on the hits you will read.
|
|
76
|
+
|
|
77
|
+
`--args` takes a JSON object, which the script reads as `s.args` (`{}` when omitted), so one script serves many calls.
|
|
78
|
+
|
|
79
|
+
Stdout carries one JSON report, `{ ok, kid, mode, args, result, error?, mutations, warnings, stats }`. `mutations` lists every payload with `applied`, including those sent before a failure. `--out` writes the report to a file and prints a one-line summary. `.mts` scripts run as-is: the Node this package requires strips types natively. Exit codes: `0` ok, `1` the script failed, `2` usage or configuration error.
|
|
80
|
+
|
|
81
|
+
The pieces are exported for use outside the CLI: `createClientFromEnv()` builds the same signed client, and `runScript()` runs a function against the guarded context.
|
|
82
|
+
|
|
47
83
|
### `substrate-admin`
|
|
48
84
|
|
|
49
|
-
A separate binary for key management: `bootstrap` (seed the genesis admin key), `doctor` (check a deployment end to end), `onboard`, `list`, `register`, `update`, `rotate`, `revoke
|
|
85
|
+
A separate binary for key management: `bootstrap` (seed the genesis admin key), `doctor` (check a deployment end to end), `onboard`, `list`, `register`, `update`, `rotate`, `revoke`, and `migrate-store` (copy a store into a fresh, empty, partitioned PostgreSQL 18 store and verify it; see [Upgrading](https://github.com/mnmal-ai/substrate/blob/main/packages/substrate-dataset-pg/README.md#upgrading)). `substrate-admin --help` documents each.
|
|
50
86
|
|
|
51
87
|
## Install
|
|
52
88
|
|
|
53
89
|
```bash
|
|
54
90
|
# 1. Scaffold a runtime directory (package.json, .env, and for postgres a compose file).
|
|
55
|
-
npx @mnmal-ai/substrate init ./my-runtime
|
|
91
|
+
npx @mnmal-ai/substrate@next init ./my-runtime
|
|
56
92
|
|
|
57
93
|
# 2. Install one shared node_modules: the binary and every chosen plugin.
|
|
58
94
|
cd my-runtime
|
|
@@ -63,14 +99,14 @@ set -a && source .env && set +a
|
|
|
63
99
|
./node_modules/.bin/substrate serve
|
|
64
100
|
```
|
|
65
101
|
|
|
66
|
-
`substrate init` writes a `package.json` pinning `@mnmal-ai/substrate` and the chosen optional plugins, a
|
|
102
|
+
`substrate init` writes a `package.json` pinning `@mnmal-ai/substrate` and the chosen optional plugins, a `pnpm-workspace.yaml`, a `.env` with `SUBSTRATE_PLUGIN_DIR=.`, a `substrate-recall.config.json` for the embedder, and, unless `--store=memory`, a `docker-compose.yml` running `pgvector/pgvector:pg18`. Flags: `--store=memory|postgres`, `--port`, `--pg-port`, `--plugins=<csv>` (from `cortext`, `substrate-nl`, `substrate-sentinel`; none by default), `--embedder=bge-base|minilm|ollama|none` (default `bge-base`), `--tls --tls-cert-path --tls-key-path`, `--substrate-url`, `--yes`, `--force`.
|
|
67
103
|
|
|
68
|
-
`SUBSTRATE_PLUGIN_DIR` is the directory whose `node_modules/` holds the plugins; in a scaffolded runtime that is the runtime directory itself. Recall and proposals load from `@mnmal-ai/substrate` itself, so they are present whenever the binary is installed there. Kernel plugins (`substrate-recall`, `substrate-nl`) compose first, then coordinators (`substrate-proposals`, `substrate-sentinel`, …), alphabetical within each tier. Per-plugin overrides live at `<SUBSTRATE_PLUGIN_DIR>/<pkg-basename>.config.json`. With `SUBSTRATE_PLUGIN_DIR` unset the binary boots with only the `_system` admin namespace and `/health`.
|
|
104
|
+
`SUBSTRATE_PLUGIN_DIR` is the directory whose `node_modules/` holds the plugins; in a scaffolded runtime that is the runtime directory itself. Recall and proposals load from `@mnmal-ai/substrate` itself, so they are present whenever the binary is installed there. Kernel plugins (`substrate-recall`, `substrate-nl`) compose first, then coordinators (`substrate-proposals`, `substrate-sentinel`, …), alphabetical within each tier. Per-plugin overrides live at `<SUBSTRATE_PLUGIN_DIR>/<pkg-basename>.config.json`. With `SUBSTRATE_PLUGIN_DIR` unset the binary boots with only the `_system` admin namespace and `/health`. Set, it must hold a `package.json` listing the plugins under `dependencies`, installed into its `node_modules/`; a missing `package.json`, or listed plugins with no `node_modules/`, refuses boot with a message naming the fix.
|
|
69
105
|
|
|
70
106
|
### Adding a plugin
|
|
71
107
|
|
|
72
108
|
```bash
|
|
73
|
-
pnpm add @mnmal-ai/substrate-sentinel
|
|
109
|
+
pnpm add @mnmal-ai/substrate-sentinel@next
|
|
74
110
|
# then restart `substrate serve`
|
|
75
111
|
```
|
|
76
112
|
|
|
@@ -82,7 +118,7 @@ cat > substrate-recall.config.json <<'EOF'
|
|
|
82
118
|
EOF
|
|
83
119
|
```
|
|
84
120
|
|
|
85
|
-
`kind: "onnx"` runs the model in-process and needs `fastembed` installed in the runtime (the embedder code itself ships in `@mnmal-ai/substrate`); `substrate init` adds it for `--embedder=
|
|
121
|
+
`kind: "onnx"` runs the model in-process and needs `fastembed` installed in the runtime (the embedder code itself ships in `@mnmal-ai/substrate`); `substrate init` adds it for `--embedder=bge-base` (the default) and `--embedder=minilm`. `kind: "ollama"` takes a `baseUrl` instead.
|
|
86
122
|
|
|
87
123
|
## Docker
|
|
88
124
|
|
|
@@ -105,6 +141,12 @@ The main ones; `substrate --help` and the header of `apps/substrate/src/main.ts`
|
|
|
105
141
|
| `SUBSTRATE_TLS_KEY_PATH` / `SUBSTRATE_TLS_CERT_PATH` / `SUBSTRATE_TLS_CA_PATH` | Serve HTTPS. Key and cert are required together. |
|
|
106
142
|
| `SUBSTRATE_NAMESPACE` | Default namespace for recall requests that omit one. |
|
|
107
143
|
| `SUBSTRATE_SWEEP_INTERVAL_MS` | Edge-sweep cadence (default 300000) |
|
|
144
|
+
| `SUBSTRATE_INDEX_DISK_PATH` | The database volume's mount, when the database is on this host: free space there is checked before each background index build. `off` skips the check. |
|
|
145
|
+
| `SUBSTRATE_INDEX_DISK_MARGIN_MB` | Free space a build must leave (default 2048; `0` requires only the estimate). |
|
|
146
|
+
| `SUBSTRATE_INDEX_RETRY_INTERVAL_MS` | How often pending or failed index builds are retried (default 600000; `0` leaves them to the next boot). |
|
|
147
|
+
| `SUBSTRATE_ADVISOR_INTERVAL_MS` | How often the index advisor writes `_system/IndexRecommendation` rows (default 600000; `0` turns it off). |
|
|
148
|
+
| `SUBSTRATE_STORE_PROFILE` | `personal`, `growth` or `scale`: seeds `_system/StoreProfile` once, so changing it later does not change an existing store. |
|
|
149
|
+
| `SUBSTRATE_STORE_WRITE_HEAVY` | `1` or `true`, read on every boot: halves the write rate at which the advisor recommends removing an unused index. |
|
|
108
150
|
| `SUBSTRATE_CORS_ORIGINS` | Comma-separated allowed origins, or `*`. Unset registers no CORS handling. |
|
|
109
151
|
|
|
110
152
|
Natural-language query and the recall embedder are not configured by environment: they read `substrate-nl.config.json` and `substrate-recall.config.json` in the plugin directory.
|