@aionis/substrate 0.1.0 → 0.1.1

package/docs/ADAPTER_CONTRACT.md

@@ -33,6 +33,7 @@ Examples:
 - lifecycle transition for a missing memory node must fail without appending an event;
 - relation with a missing source or target must fail without appending an event;
 - feedback for a missing memory node must fail without appending an event.
+- decision trace for a missing memory node must fail without appending an event.
 
 The file adapter validates against a cloned state before appending to `events.jsonl`.
 
@@ -54,6 +55,8 @@ The current schema version is `1`.
 
 The SQLite adapter must persist schema metadata in `substrate_metadata`, set SQLite `user_version`, and refuse to open a database whose schema version is newer than the runtime supports. Silent best-effort reads of future schemas are not allowed.
 
+SQLite schema changes must go through the adapter migration registry. Applied migrations are recorded in `substrate_schema_migrations` with version, name, and timestamp. A migration ledger whose recorded name no longer matches the current registry is treated as corruption and the store is rejected.
+
 The file adapter must write the schema version into `snapshot.json` and report the same version through `getStoreInfo`.
 
 ### 4. Scope Isolation
@@ -107,7 +110,19 @@ Search is not a vector index and not the full Runtime admission policy. It is a
 
 This is intentional: exported context must leave a receipt. Callers that need a side-effect-free preview must use `previewContext` instead of weakening `compileContext`.
 
-### 9. Backup and Restore Integrity
+### 9. Audit Read API Parity
+
+Adapters must expose the same scoped audit reads:
+
+- `listRelations(scope, memoryId?)`
+- `listFeedback({ scope, memoryId? })`
+- `listDecisions(scope)`
+
+These APIs must be side-effect-free. They must not append decision events, lifecycle transitions, relation writes, feedback writes, or checkpoints.
+
+When `memoryId` is supplied to `listRelations`, adapters must return relations where that memory is either the source or target. When `memoryId` is supplied to `listFeedback`, adapters must return feedback attached to that memory only.
+
+### 10. Backup and Restore Integrity
 
 Backup export must operate over the append-only event log, not only over derived read-model tables or snapshots.
 
@@ -116,7 +131,7 @@ Restore must verify:
 - supported backup and schema version;
 - contiguous event sequence;
 - duplicate event ids;
-- event reference integrity;
+- event reference integrity, including decision trace memory ids;
 - header event counts;
 - SHA-256 checksum.
 
@@ -129,6 +144,7 @@ The test suite currently checks:
 - failed lifecycle transition does not corrupt the file event log;
 - failed relation writes do not persist corrupt events or partial rows;
 - failed feedback writes do not persist corrupt events or partial rows;
+- failed decision writes do not persist corrupt events or partial rows;
 - file and SQLite adapters compile identical buckets for the same evidence;
 - superseded memory is blocked from direct use;
 - archived evidence becomes a rehydrate hook;
@@ -136,10 +152,14 @@ The test suite currently checks:
 - file snapshots can be rebuilt from append-only events;
 - concurrent writes are serialized with contiguous event sequences;
 - store schema version is reported by both adapters;
-- SQLite schema metadata is persisted and future unsupported schemas are rejected;
+- SQLite schema metadata and migration ledger are persisted, backfilled for legacy v1 stores, and future unsupported schemas are rejected;
 - event-log backups verify checksums and restore to file and SQLite stores;
+- event-log backups reject checksum-valid decision traces that reference missing memory nodes;
 - checkpoint compaction preserves governed state and restarts future event sequences after the checkpoint;
+- compacted checkpoints reject corrupt decision references on reopen;
+- relation, feedback, and decision writes remain atomic after checkpoint compaction;
 - file and SQLite adapters return identical read-only search results for the same scoped query;
+- file and SQLite adapters return identical scoped audit reads without mutating event logs;
 - Runtime snapshot import opens source SQLite read-only.
 
 ## Non-Goals

package/docs/API_USAGE.md

@@ -177,6 +177,31 @@ The compiled context has four surfaces:
 
 `compileContext` records a `memory.decision.recorded` event. It is intentionally auditable, not a pure read.
 
+## Read Audit Records
+
+```ts
+const relations = await store.listRelations("repo-a", "route-current");
+const feedback = await store.listFeedback({
+  scope: "repo-a",
+  memoryId: "route-current",
+});
+const decisions = await store.listDecisions("repo-a");
+
+console.log(relations.map((relation) => relation.kind));
+console.log(feedback.map((item) => item.outcome));
+console.log(decisions.at(-1)?.decisions);
+```
+
+Audit reads are scoped and side-effect-free:
+
+- `listRelations(scope)` returns all relations in a scope.
+- `listRelations(scope, memoryId)` returns relations where the memory is either source or target.
+- `listFeedback({ scope })` returns feedback in a scope.
+- `listFeedback({ scope, memoryId })` returns feedback attached to one memory.
+- `listDecisions(scope)` returns recorded decision receipts for the scope.
+
+These APIs read evidence and receipts. They do not append events, compile prompt surfaces, or decide whether memory can influence the next Agent turn.
+
 ## Compact the Event Log
 
 ```ts

package/docs/CLI.md

@@ -0,0 +1,89 @@
+# CLI
+
+`@aionis/substrate` publishes a small CLI for validation and sidecar integration work.
+
+It is intentionally narrow:
+
+- it runs read-only checks over existing Runtime evidence;
+- it writes reports to local files;
+- it does not start Aionis Runtime unless you explicitly use the separate repository script `check:runtime-dual-write`;
+- it does not mutate Runtime source code or replace Runtime storage.
+
+## Install
+
+Run without installing permanently:
+
+```bash
+npx @aionis/substrate --help
+```
+
+Install into a project:
+
+```bash
+npm install @aionis/substrate
+npx aionis-substrate --help
+```
+
+The CLI requires Node 24+.
+
+## Sidecar Check
+
+Use `sidecar` when you already have Runtime Lite SQLite evidence and want to check whether Substrate can mirror the governed context surface from outside the Runtime boundary.
+
+Snapshot parity:
+
+```bash
+npx aionis-substrate sidecar \
+  --source /path/to/aionis-runtime-lite.sqlite \
+  --scope repo-a \
+  --reference /path/to/runtime-guide-or-measure.json \
+  --output reports/runtime-sidecar/summary.json
+```
+
+Reference corpus parity:
+
+```bash
+npx aionis-substrate sidecar \
+  --source-root /path/to/runtime-sqlite-root \
+  --reference-root /path/to/runtime-reference-root \
+  --max-source-files all \
+  --max-scopes all \
+  --max-scopes-per-file 100 \
+  --max-references all
+```
+
+Combined report:
+
+```bash
+npx aionis-substrate sidecar \
+  --source /path/to/aionis-runtime-lite.sqlite \
+  --scope repo-a \
+  --reference /path/to/runtime-guide-or-measure.json \
+  --source-root /path/to/runtime-sqlite-root \
+  --reference-root /path/to/runtime-reference-root \
+  --output reports/runtime-sidecar/summary.json
+```
+
+The report contract is `aionis_runtime_sidecar_check_report_v1`.
+
+## What Passing Means
+
+Passing snapshot parity means one Runtime SQLite scope can be imported into an isolated Substrate store and compiled into matching governed buckets.
+
+Passing reference corpus parity means exported Runtime guide/measure JSON is traceable to real Runtime memory ids in the same source corpus.
+
+Passing these checks does not mean Substrate has become the full Runtime policy engine. Runtime still owns richer product policy; Substrate owns durable evidence, lifecycle state, relations, feedback, and the minimum governed context contract.
+
+## Common Failures
+
+`no_matched_reference`
+
+The reference files were scanned, but none of their memory ids overlap the discovered Runtime SQLite scopes. Point `--reference-root` at guide/measure outputs produced from the same Runtime data as `--source-root`.
+
+`snapshot_parity failed`
+
+The imported Substrate buckets do not match the supplied Runtime guide/measure surface. Inspect the generated `summary.json` before changing code; the mismatch may be a scope, reference, or fixture problem.
+
+`ExperimentalWarning: SQLite is an experimental feature`
+
+Node 24 currently marks `node:sqlite` as experimental. This is expected for the current embedded SQLite adapter.

package/docs/RUNTIME_SIDECAR_STABILIZATION.md

@@ -0,0 +1,96 @@
+# Runtime Sidecar Stabilization
+
+This page defines the Substrate sidecar validation path for Aionis Runtime.
+
+The sidecar path is intentionally external:
+
+- it does not mutate `AionisRuntime-focused` source code;
+- it does not replace Runtime Lite SQLite, Zvec, AIFS, or product storage;
+- it does not install a production dual-write adapter;
+- it only proves that Substrate can mirror, import, and inspect Runtime evidence from outside the Runtime boundary.
+
+## Sidecar Gates
+
+### Gate 1: Read-Only Snapshot Parity
+
+Import one Runtime Lite SQLite snapshot into an isolated Substrate store and compile context for one scope.
+
+```bash
+npm run check:runtime-sidecar -- \
+  --source /path/to/aionis-runtime-lite.sqlite \
+  --scope repo-a \
+  --reference /path/to/runtime-guide-or-measure.json
+```
+
+Without `--reference`, this is an import smoke: Runtime SQLite is opened read-only and Substrate writes to an isolated target.
+
+With `--reference`, the command compares the four governed buckets:
+
+- `use_now`
+- `inspect_before_use`
+- `do_not_use`
+- `rehydrate`
+
+### Gate 2: Same-Source Reference Corpus
+
+Scan Runtime SQLite files and Runtime guide/measure JSON references, then count only references that share concrete memory ids with a discovered Runtime scope.
+
+```bash
+npm run check:runtime-sidecar -- \
+  --source-root /path/to/runtime-sqlite-root \
+  --reference-root /path/to/runtime-reference-root \
+  --max-source-files all \
+  --max-scopes all \
+  --max-scopes-per-file 100 \
+  --max-references all
+```
+
+This gate rejects demo-only evidence: a reference JSON must overlap real Runtime memory ids to count.
+
+### Gate 3: Real Runtime Dual-Write Sidecar
+
+This stage starts focused Runtime with isolated Lite SQLite files, calls the public Runtime SDK loop, mirrors the observed memory ids and outcomes into a separate Substrate SQLite store, and compares Runtime guide surfaces against Substrate compiled context.
+
+```bash
+npm run check:runtime-dual-write -- \
+  --runtime-root /Volumes/ziel/AionisRuntime-focused \
+  --generated-count 8 \
+  --chain-probe-count 4 \
+  --concurrency 4
+```
+
+This command remains separate from `check:runtime-sidecar` because it starts a Runtime process. It must be explicit.
+
+## Combined Read-Only Report
+
+`check:runtime-sidecar` can run Gate 1 and Gate 2 in one report:
+
+```bash
+npm run check:runtime-sidecar -- \
+  --source /path/to/aionis-runtime-lite.sqlite \
+  --scope repo-a \
+  --reference /path/to/runtime-guide-or-measure.json \
+  --source-root /path/to/runtime-sqlite-root \
+  --reference-root /path/to/runtime-reference-root \
+  --output reports/runtime-sidecar-manual/summary.json
+```
+
+The report contract is `aionis_runtime_sidecar_check_report_v1`.
+
+It records:
+
+- requested stages;
+- pass/fail summary per stage;
+- snapshot import coverage and parity;
+- reference corpus matched/unmatched counts;
+- notes confirming that Runtime source and Runtime storage were not replaced.
+
+## Interpretation
+
+Passing Gate 1 means Substrate can represent one Runtime SQLite scope as governed substrate state.
+
+Passing Gate 2 means exported Runtime references are traceable to real Runtime SQLite memory ids, not only documentation examples.
+
+Passing Gate 3 means an external host can mirror a small real Runtime execution loop into Substrate and preserve the same admission surface after reopen.
+
+None of these gates mean Substrate has become the full Aionis Runtime policy engine. Runtime owns richer product policy; Substrate owns the durable evidence and minimum governed context contract.

package/docs/STORE_CONTRACT.md

@@ -40,6 +40,8 @@ Adapters must expose schema metadata through `getStoreInfo`:
 
 The SQLite adapter persists schema metadata in `substrate_metadata` and mirrors the same version into SQLite `user_version`. Opening a store with a newer unsupported schema must fail before any mutation occurs.
 
+SQLite schema changes are applied through an explicit adapter migration registry. Applied migrations are recorded in `substrate_schema_migrations`. The registry must stay contiguous and end at the current substrate schema version, and a tampered migration name must cause open to fail.
+
 The file adapter writes the same schema version into `snapshot.json`. The append-only event log remains the durable evidence source; the snapshot schema is the derived read-model format.
 
 ### Backup Boundary
@@ -48,6 +50,8 @@ Backups export the append-only event log plus schema metadata and a SHA-256 chec
 
 Restore must verify the backup before writing a target store. Restored stores preserve original event ids and sequence numbers, then rebuild derived read models.
 
+Backup verification must replay event reference integrity, including lifecycle targets, relation endpoints, feedback memory ids, and decision-trace memory ids.
+
 Payload files referenced by `payloadRef` are not embedded in the Substrate backup. They remain external artifacts and need their own retention policy.
 
 ### Checkpoint Compaction
@@ -156,12 +160,22 @@ Every compiled context must include a decision trace:
 - which reason code caused the decision
 - which relation caused the decision when applicable
 
+Every decision entry must reference a memory node that exists in the same scope as the trace. Decision traces are receipts over known substrate evidence; they cannot introduce orphan memory ids.
+
 The trace is for audit/debug/measure. It must not mutate admission by itself.
 
 `previewContext` is the side-effect-free admission preview. It returns the same bucket and reason-code shape as `compileContext`, but it must not append events or insert decision rows.
 
 `compileContext` is intentionally not a pure read. It records `memory.decision.recorded` so every exported context has an auditable receipt. Tools that need a side-effect-free view must use `previewContext` instead of treating `compileContext` as read-only.
 
+Decision and evidence records must also be inspectable without creating new events:
+
+- `listRelations(scope, memoryId?)`
+- `listFeedback({ scope, memoryId? })`
+- `listDecisions(scope)`
+
+These read APIs are for audit, debug, and measure surfaces. They do not compile context and do not mutate lifecycle, relation, feedback, decision, or checkpoint state.
+
 ## Adapter Requirements
 
 Every adapter must satisfy the same observable contract:
@@ -172,8 +186,10 @@ Every adapter must satisfy the same observable contract:
 4. Compile the same admission buckets from the same node/relation state.
 5. Return the same scoped search results from the same node state.
 6. Record decision traces as events.
-7. Reopen cleanly and recover the same read model.
-8. If compaction is supported, compact only through a validated checkpoint event.
+7. Return the same scoped audit reads from the same relation, feedback, and decision state.
+8. Reopen cleanly and recover the same read model.
+9. Reject corrupt lifecycle, relation, feedback, decision, and checkpoint references before persisting invalid events.
+10. If compaction is supported, compact only through a validated checkpoint event.
 
 Current adapters:
 

package/docs/V0_2_ROADMAP.md

@@ -0,0 +1,108 @@
+# Aionis Substrate v0.2 Roadmap
+
+This roadmap freezes the intended scope for the next Substrate release.
+
+Substrate remains an independent storage-contract layer. It is not the full Aionis Runtime policy engine, not a vector database, and not an Agent framework.
+
+## v0.2 Principles
+
+- Keep append-only evidence as the source of truth.
+- Keep lifecycle, relation, feedback, and decision receipts inspectable.
+- Preserve file and SQLite adapter parity for every public API.
+- Add product hardening only where it improves durability, auditability, or Runtime sidecar integration.
+- Do not encode task-specific benchmark fixtures or one-off Runtime policies in Substrate.
+
+## Included
+
+### 1. Migration Scaffold
+
+Add a small schema-migration boundary for future SQLite changes:
+
+- explicit migration registry;
+- current schema guard remains strict;
+- migration tests for already-created stores;
+- no best-effort reads of unsupported future schemas.
+
+Initial implementation status: SQLite now has a registry-backed v1 migration ledger. Future schema changes should add migrations through that registry instead of editing open-time schema creation ad hoc.
+
+### 2. Read API Hardening
+
+Expose scoped audit reads that do not mutate the event log:
+
+- `listRelations(scope, memoryId?)`
+- `listFeedback({ scope, memoryId? })`
+- `listDecisions(scope)`
+
+These APIs are for audit/debug/measure surfaces and adapter parity checks. They do not replace `compileContext`, and they do not make admission decisions.
+
+### 3. Durability Negative Tests
+
+Extend failure-mode coverage:
+
+- corrupt event references;
+- interrupted writes;
+- relation and feedback edge cases after compaction;
+- checkpoint reopen after mixed read/write traffic.
+
+Initial implementation status: decision trace references are validated like relation and feedback references. File and SQLite adapters reject missing decision targets before persisting events or rows; backup verification rejects checksum-valid orphan decision receipts; checkpoint reopen rejects corrupt decision references; post-checkpoint invalid relation, feedback, and decision writes stay atomic.
+
+### 4. Runtime Sidecar Stabilization
+
+Keep Runtime experiments isolated:
+
+- read-only Runtime snapshot import remains separate from Runtime source;
+- dual-write sidecar continues to write into a separate Substrate store;
+- no replacement of Aionis Runtime storage in v0.2.
+
+Initial implementation status: `check:runtime-sidecar` now combines read-only Runtime snapshot parity and same-source reference corpus parity into a single report contract. Real Runtime dual-write remains an explicit separate gate through `check:runtime-dual-write` because it starts focused Runtime.
+
+### 5. Product CLI and Docs
+
+Make the substrate boundary easier to consume without widening policy scope:
+
+- publish a package CLI entrypoint for read-only sidecar checks;
+- document install, minimal API usage, and sidecar reports separately;
+- keep repository-only Runtime process experiments explicit and separate.
+
+Initial implementation status: the package exposes `aionis-substrate sidecar` for read-only snapshot/reference checks. It does not start Runtime, mutate Runtime storage, or implement Runtime admission policy.
+
+## Excluded
+
+- Vector search, ANN, embeddings, or semantic recall.
+- Full Aionis Runtime admission policy.
+- LLM-as-judge or model-generated lifecycle policy.
+- Agent orchestration or external Agent harnesses.
+- SaaS tenancy, auth, billing, or cloud service behavior.
+- Replacing `AionisRuntime-focused` storage.
+
+## Release Gates
+
+Before v0.2 can be tagged:
+
+- `npm run typecheck`
+- `npm test`
+- `npm run bench:contract`
+- `npm run check:release`
+- `npm run check:scale -- --nodes 10000 --scopes 10 --relations 2000 --feedback 1000`
+- adapter parity tests for every new public API;
+- package install smoke from tarball;
+- published registry smoke after release.
+
+## v0.1 Baseline
+
+v0.1 already provides:
+
+- file and SQLite adapters;
+- append-only event log;
+- governed context buckets;
+- decision receipts;
+- read-only context preview;
+- backup and restore;
+- checkpoint compaction;
+- deterministic scoped search;
+- Runtime snapshot import;
+- external admission parity;
+- isolated Runtime dual-write sidecar experiments;
+- npm package and registry smoke checks.
+
+v0.2 should harden this substrate boundary instead of broadening the product surface.

package/package.json

@@ -1,10 +1,13 @@
 {
   "name": "@aionis/substrate",
-  "version": "0.1.0",
-  "description": "Aionis-native memory and execution-state substrate experiment.",
+  "version": "0.1.1",
+  "description": "Durable governed memory substrate for Aionis execution state.",
   "type": "module",
   "main": "./dist/index.js",
   "types": "./dist/index.d.ts",
+  "bin": {
+    "aionis-substrate": "dist/cli.js"
+  },
   "exports": {
     ".": {
       "types": "./dist/index.d.ts",
@@ -30,11 +33,14 @@
     "check:runtime-snapshot": "node scripts/runtime-snapshot-parity.ts",
     "check:runtime-corpus": "node scripts/runtime-snapshot-corpus.ts",
     "check:runtime-reference-corpus": "node scripts/runtime-reference-corpus.ts",
+    "check:runtime-sidecar": "node scripts/runtime-sidecar-check.ts",
     "make:runtime-product-reference": "node scripts/runtime-product-reference-fixture.ts",
     "check:external-admission-parity": "node scripts/external-admission-parity.ts",
     "check:runtime-dual-write": "node scripts/runtime-dual-write-experiment.ts",
     "check:pack": "npm run build && node scripts/verify-package.ts",
     "check:install-smoke": "npm run build && node scripts/install-smoke.ts",
+    "check:registry-install": "node scripts/registry-install-smoke.ts",
+    "check:published-runtime-smoke": "node scripts/published-runtime-smoke.ts",
     "check:scale": "node scripts/scale-test.ts",
     "example:basic": "npm run build && node examples/basic/index.mjs",
     "check:release": "npm run typecheck && npm test && npm run bench:contract && npm run example:basic && npm run check:pack && npm run check:install-smoke"