@aionis/substrate 0.1.0

package/docs/API_USAGE.md

@@ -0,0 +1,225 @@
+# API Usage
+
+This document shows the first public contract for using Aionis Substrate directly.
+
+The API is intentionally small:
+
+- write memory nodes;
+- write relations between memory nodes;
+- record outcome feedback;
+- search memory nodes with scoped filters;
+- preview governed context without writing a receipt;
+- compile governed context;
+- inspect decision traces.
+
+## Open a Store
+
+SQLite adapter:
+
+```ts
+import { openSqliteAionisSubstrate } from "@aionis/substrate";
+
+const store = await openSqliteAionisSubstrate({
+  path: "/tmp/aionis-substrate.sqlite",
+});
+```
+
+File adapter:
+
+```ts
+import { openFileAionisSubstrate } from "@aionis/substrate";
+
+const store = await openFileAionisSubstrate({
+  dir: "/tmp/aionis-substrate-file-store",
+});
+```
+
+Both adapters expose the same contract.
+
+## Inspect Store Metadata
+
+```ts
+const info = await store.getStoreInfo();
+
+console.log(info.adapter);
+console.log(info.schemaVersion);
+console.log(info.lastSequence);
+console.log(info.eventCount);
+```
+
+The schema version is part of the substrate contract. A newer unsupported SQLite schema is rejected on open instead of being read best-effort.
+
+## Write Execution Memory
+
+```ts
+await store.putNode({
+  id: "route-current",
+  scope: "repo-a",
+  kind: "procedure",
+  title: "Current route",
+  summary: "Use src/runtime.ts as the validated current route.",
+  lifecycle: "active",
+  authority: "trusted",
+  confidence: 0.92,
+  targetFiles: ["src/runtime.ts", "tests/runtime.test.ts"],
+  metadata: {
+    source: "agent_observe",
+    verifier: "npm test passed",
+  },
+});
+```
+
+## Keep Old Evidence Without Trusting It
+
+```ts
+await store.putNode({
+  id: "route-old",
+  scope: "repo-a",
+  kind: "procedure",
+  title: "Old route",
+  summary: "Use src/legacy.ts as the previous route.",
+  lifecycle: "active",
+  authority: "trusted",
+  confidence: 0.8,
+  targetFiles: ["src/legacy.ts"],
+});
+
+await store.putRelation({
+  scope: "repo-a",
+  kind: "supersedes",
+  sourceId: "route-current",
+  targetId: "route-old",
+  confidence: 0.88,
+  metadata: {
+    reason: "newer verifier evidence replaced the old route",
+  },
+});
+```
+
+The old route remains in the evidence log. It is not silently deleted. The relation prevents it from becoming direct-use context.
+
+## Record Feedback
+
+```ts
+await store.recordFeedback({
+  scope: "repo-a",
+  memoryId: "route-current",
+  outcome: "positive",
+  strength: "strong",
+  evidenceRef: "trace://run-2026-06-25/verifier",
+  runId: "run-2026-06-25",
+});
+```
+
+Feedback is evidence. It does not bypass lifecycle, relation, or authority checks.
+
+## Search Memory Nodes
+
+```ts
+const results = await store.searchNodes({
+  scope: "repo-a",
+  query: "runtime verifier",
+  lifecycle: ["active"],
+  authority: ["trusted"],
+  targetFiles: ["src/runtime.ts"],
+  minConfidence: 0.8,
+  limit: 10,
+});
+
+for (const result of results) {
+  console.log(result.node.id, result.score, result.reasons);
+}
+```
+
+Search is a scoped, read-only substrate query.
+
+It supports structured filters for kind, lifecycle, authority, target files, owner identity, confidence, and update time. The optional `query` is deterministic lexical scoring over node id, title, summary, target files, payload pointer, owner ids, and primitive metadata.
+
+It is not vector search, semantic adjudication, or admission policy. Use `compileContext` when the Agent needs governed prompt surfaces.
+
+## Preview Context
+
+```ts
+const preview = await store.previewContext({
+  scope: "repo-a",
+  query: "continue the runtime implementation",
+});
+
+console.log(preview.use_now.map((node) => node.id));
+console.log(preview.inspect_before_use.map((node) => node.id));
+console.log(preview.do_not_use.map((node) => node.id));
+console.log(preview.rehydrate.map((node) => node.id));
+```
+
+`previewContext` uses the same admission buckets and reason codes as `compileContext`, but it does not append `memory.decision.recorded` or insert a decision row. Use it for UI previews, dry runs, and adapter parity checks where no exported context receipt should be created.
+
+## Compile Context
+
+```ts
+const context = await store.compileContext({
+  scope: "repo-a",
+  query: "continue the runtime implementation",
+});
+
+console.log(context.use_now.map((node) => node.id));
+console.log(context.inspect_before_use.map((node) => node.id));
+console.log(context.do_not_use.map((node) => node.id));
+console.log(context.rehydrate.map((node) => node.id));
+console.log(context.decision_trace);
+```
+
+The compiled context has four surfaces:
+
+- `use_now`: directly usable context;
+- `inspect_before_use`: relevant but not authoritative;
+- `do_not_use`: blocked, superseded, invalidated, or unsafe;
+- `rehydrate`: payload is available but should not flood the prompt unless requested.
+
+`compileContext` records a `memory.decision.recorded` event. It is intentionally auditable, not a pure read.
+
+## Compact the Event Log
+
+```ts
+const report = await store.compact();
+
+console.log(report.compacted);
+console.log(report.before.eventCount);
+console.log(report.after.eventCount);
+```
+
+Compaction is physical log maintenance. It writes a `substrate.checkpoint.created` event that preserves the current nodes, relations, feedback, decision traces, and checksum metadata for the covered history.
+
+It does not change lifecycle state, admission buckets, or Runtime policy.
+
+## Export, Verify, and Restore
+
+```ts
+import {
+  exportAionisSubstrateBackup,
+  restoreAionisSubstrateBackupToSqlite,
+  verifyAionisSubstrateBackup,
+  writeAionisSubstrateBackupFile,
+} from "@aionis/substrate";
+
+const backup = await exportAionisSubstrateBackup(store);
+const report = verifyAionisSubstrateBackup(backup);
+
+if (!report.ok) throw new Error(report.errors.join("; "));
+
+await writeAionisSubstrateBackupFile("/backups/aionis-substrate.json", backup);
+await restoreAionisSubstrateBackupToSqlite(backup, "/tmp/restored-aionis-substrate.sqlite");
+```
+
+Backup is event-log based. Restore preserves original event ids and sequence numbers, then rebuilds the read model.
+
+## Close the Store
+
+```ts
+await store.close();
+```
+
+## Boundary
+
+This API is a substrate contract, not the complete Aionis Runtime product facade.
+
+Runtime-level features such as product `observe`, `guide`, `forget`, `measure`, lifecycle candidate inference, memory decision trace presentation, and context compiler UX belong above this substrate layer.

package/docs/BACKUP_RESTORE.md

@@ -0,0 +1,108 @@
+# Backup and Restore
+
+Aionis Substrate backup is event-log based.
+
+It exports the append-only evidence events plus a checksum. Restore rebuilds the read model from those events. It does not copy only the derived snapshot.
+
+## Backup Format
+
+The backup object contains:
+
+- backup format and version;
+- substrate schema version;
+- source adapter metadata from `getStoreInfo`;
+- event count and last sequence;
+- SHA-256 checksum over the canonical event list;
+- every append-only event in sequence order.
+
+The checksum covers the event list. If an event payload is changed after export, verification fails and restore is rejected.
+
+A compacted store may export a `substrate.checkpoint.created` event instead of the original full event history. The checkpoint includes covered event count, covered last sequence, and the SHA-256 checksum of the covered event list.
+
+## Export
+
+```ts
+import {
+  exportAionisSubstrateBackup,
+  openSqliteAionisSubstrate,
+  writeAionisSubstrateBackupFile,
+} from "@aionis/substrate";
+
+const store = await openSqliteAionisSubstrate({
+  path: "/data/aionis-substrate.sqlite",
+});
+
+const backup = await exportAionisSubstrateBackup(store);
+await writeAionisSubstrateBackupFile("/backups/aionis-substrate-backup.json", backup);
+
+await store.close();
+```
+
+## Verify
+
+```ts
+import {
+  readAionisSubstrateBackupFile,
+  verifyAionisSubstrateBackup,
+} from "@aionis/substrate";
+
+const backup = await readAionisSubstrateBackupFile("/backups/aionis-substrate-backup.json");
+const report = verifyAionisSubstrateBackup(backup);
+
+if (!report.ok) {
+  throw new Error(report.errors.join("; "));
+}
+```
+
+Verification checks:
+
+- backup format and version;
+- supported schema version;
+- event sequence continuity;
+- duplicate event ids;
+- relation / feedback / lifecycle references;
+- event count and last sequence headers;
+- SHA-256 checksum.
+
+For checkpoint events, verification also replays the checkpoint payload and checks relation / feedback references inside the checkpoint state.
+
+## Restore to File Store
+
+```ts
+import {
+  readAionisSubstrateBackupFile,
+  restoreAionisSubstrateBackupToFile,
+} from "@aionis/substrate";
+
+const backup = await readAionisSubstrateBackupFile("/backups/aionis-substrate-backup.json");
+await restoreAionisSubstrateBackupToFile(backup, "/data/aionis-substrate-file-store");
+```
+
+Restore writes:
+
+- `events.jsonl`
+- `snapshot.json`
+
+The target must be empty unless `overwrite: true` is passed.
+
+## Restore to SQLite
+
+```ts
+import {
+  readAionisSubstrateBackupFile,
+  restoreAionisSubstrateBackupToSqlite,
+} from "@aionis/substrate";
+
+const backup = await readAionisSubstrateBackupFile("/backups/aionis-substrate-backup.json");
+await restoreAionisSubstrateBackupToSqlite(backup, "/data/aionis-substrate-restored.sqlite");
+```
+
+Restore writes the original event ids and sequence numbers, rebuilds structured read-model tables, and preserves the next event sequence for future writes.
+
+The SQLite target must not exist unless `overwrite: true` is passed.
+
+## Boundary
+
+Backup and restore operate on the Substrate store only.
+
+They do not back up an Aionis Runtime database, an external vector index, or payload files referenced by `payloadRef`. Payload files remain external artifacts and need their own storage policy.

package/docs/CHECKPOINT_COMPACTION.md

@@ -0,0 +1,75 @@
+# Checkpoint Compaction
+
+Aionis Substrate stores durable memory changes as events. Over time, a long-running store can compact its physical event log into a checkpoint event.
+
+Compaction is storage maintenance. It is not forgetting, admission policy, or evidence deletion.
+
+## Checkpoint Event
+
+`compact()` rewrites the event log to one event:
+
+- `substrate.checkpoint.created`
+
+The checkpoint payload contains:
+
+- substrate schema version;
+- number of covered events;
+- last covered event sequence;
+- SHA-256 checksum of the covered canonical event list;
+- current memory nodes;
+- current relations;
+- current feedback records;
+- current decision traces.
+
+After compaction, the event log starts from the checkpoint at sequence `1`. Future writes continue from sequence `2`.
+
+## Adapter Behavior
+
+The file adapter:
+
+- validates the checkpoint with the same replay state machine used by backup verification;
+- atomically replaces `events.jsonl`;
+- rewrites `snapshot.json`.
+
+The SQLite adapter:
+
+- builds the checkpoint from the structured read model;
+- validates the checkpoint with the same replay state machine;
+- replaces the event table inside a transaction;
+- resets the event AUTOINCREMENT sequence.
+
+Both adapters preserve the same observable governed state after reopen.
+
+## Backup Boundary
+
+Backups can export compacted stores. A compacted backup may contain a checkpoint event instead of the original full event history.
+
+The checkpoint keeps audit metadata for the covered history through:
+
+- `coveredEventCount`
+- `coveredLastSequence`
+- `coveredEventsSha256`
+
+Payload files referenced by `payloadRef` are still external artifacts. Compaction does not embed them.
+
+## Usage
+
+```ts
+const report = await store.compact();
+
+console.log(report.before.eventCount);
+console.log(report.after.eventCount);
+console.log(report.after.checkpointEventId);
+```
+
+If the store has no events, `compact()` returns `compacted: false`.
+
+## Non-Goals
+
+Checkpoint compaction does not:
+
+- remove stale memories;
+- suppress memories;
+- archive payload files;
+- change `use_now`, `inspect_before_use`, `do_not_use`, or `rehydrate`;
+- replace Aionis Runtime policy.

package/docs/EXTERNAL_ADMISSION_PARITY.md

@@ -0,0 +1,98 @@
+# External Admission Parity
+
+`scripts/external-admission-parity.ts` validates Aionis Substrate against the real focused Runtime external memory governance path.
+
+It starts `AionisRuntime-focused` in local Lite mode, calls the product SDK method that backs `/v1/memory/govern`, projects the same external candidate memories into Aionis Substrate, and compares the four admission buckets:
+
+- `use_now`
+- `inspect_before_use`
+- `do_not_use`
+- `rehydrate`
+
+## Run
+
+```bash
+npm run check:external-admission-parity -- \
+  --runtime-root /Volumes/ziel/AionisRuntime-focused
+```
+
+By default this runs 6 fixed contract scenarios plus 24 deterministic generated variants:
+
+```bash
+npm run check:external-admission-parity -- \
+  --runtime-root /Volumes/ziel/AionisRuntime-focused \
+  --generated-count 100 \
+  --seed external-admission-parity-v2
+```
+
+The runner writes a report under:
+
+```text
+reports/external-admission-parity-*/summary.json
+```
+
+## What This Proves
+
+This validates that Substrate's minimum admission contract can express the same four-bucket result as focused Runtime's external candidate governance route.
+
+The runner covers:
+
+- trusted current execution state admitted to `use_now`;
+- unverified candidate memory routed to `inspect_before_use`;
+- failed, stale, suppressed, or blocked memory kept out of direct use;
+- raw trace or payload-only memory routed to `rehydrate`.
+- trusted ordinary preference/fact memory outside code execution traces;
+- known-source memory that remains inspect-only under focused Runtime firewall behavior;
+- explicit authority requirements that override otherwise current lifecycle hints.
+
+Generated variants expand these surfaces across execution routes, procedures, handoff state, ordinary preference memory, failed/stale/contested/suppressed negative history, and payload-only evidence. They are deterministic contract variants, not a public benchmark leaderboard.
+
+The focused Runtime route may apply stricter product firewall behavior before bucket emission. For example, an inspect-like external candidate outside the active target cluster can be blocked as `do_not_use`. This runner intentionally keeps fixtures inside the shared external admission contract so the comparison validates Substrate bucket expressiveness instead of forcing full Runtime product policy into the substrate layer.
+
+## What This Does Not Prove
+
+This is not full Aionis Runtime policy parity.
+
+It does not replace focused Runtime's guide policy, lifecycle candidate inference, relation adjudication, feedback attribution, or product context compiler. It is deliberately narrower: external candidate admission into the four bucket contract.
+
+It also does not mutate Runtime source code or Runtime product databases. The focused Runtime service is started with isolated temporary SQLite paths under the report directory.
+
+## Current Focused Runtime Check
+
+The expanded focused Runtime parity run currently produces:
+
+```text
+scenario_count: 30
+base_scenario_count: 6
+generated_scenario_count: 24
+exact_scenario_count: 30
+failed_scenario_count: 0
+seed: external-admission-parity-v2
+```
+
+Report:
+
+```text
+reports/external-admission-parity-2026-06-25T14-01-55-065Z/summary.json
+```
+
+Reports are generated artifacts and are not committed by default.
+
+A larger smoke run also passed:
+
+```text
+scenario_count: 106
+base_scenario_count: 6
+generated_scenario_count: 100
+exact_scenario_count: 106
+failed_scenario_count: 0
+seed: external-admission-parity-v2-large
+```
+
+## Why This Exists
+
+Runtime snapshot parity validates read-only import from existing Runtime SQLite databases.
+
+Runtime product reference parity validates same-source execution continuity produced by real `observe` / `guide` / `measure` loops.
+
+External admission parity validates the missing product boundary: external memory candidates already carrying candidate governance signals can be routed into the same four bucket contract without relying on Runtime's internal storage.

package/docs/RUNTIME_DUAL_WRITE_EXPERIMENT.md

@@ -0,0 +1,132 @@
+# Runtime Dual-Write Experiment
+
+This experiment validates Aionis Substrate as an external sidecar for real focused Runtime execution state.
+
+It does not replace Runtime storage. It does not mutate `AionisRuntime-focused`. It starts a real focused Runtime process with isolated Lite SQLite paths, calls the public SDK loop, mirrors the observed memory ids and outcomes into a separate Substrate SQLite store, and compares the resulting admission buckets.
+
+## Why This Exists
+
+Snapshot import proves Substrate can read existing Runtime databases.
+
+External admission parity proves Substrate can project candidate memories into the same admission buckets as focused Runtime's external memory route.
+
+Dual-write is the next integration step. It checks whether an external host can write the same execution-state facts into Runtime and Substrate during one real run, then get equivalent governed context from both systems.
+
+## What It Runs
+
+The runner executes this loop for each scenario:
+
+1. Start focused Runtime with isolated Lite SQLite paths.
+2. Call `execution.guideForRole` before any memory exists.
+3. Call `execution.observeStep` for an accepted active route.
+4. Call `execution.observeStep` for a rejected failed branch.
+5. Write the same returned Runtime memory ids into Substrate.
+6. Add a Substrate relation from the active route to the failed branch.
+7. Call `execution.guideForRole` again and extract Runtime guide surfaces.
+8. Call `store.compileContext` for the same scope and compare Substrate surfaces.
+9. Call `execution.feedbackFromOutcome` and `execution.measureRun`.
+10. Write matching Substrate feedback.
+11. Probe invalid Substrate relation/feedback writes and verify no partial events are appended.
+12. Optionally run independent Substrate lifecycle/relation chain probes.
+13. Close and reopen Substrate, then compare persisted surfaces again.
+
+The compared surfaces are:
+
+- `use_now`
+- `inspect_before_use`
+- `do_not_use`
+- `rehydrate`
+
+## Command
+
+```bash
+npm run check:runtime-dual-write -- \
+  --runtime-root /Volumes/ziel/AionisRuntime-focused \
+  --generated-count 8 \
+  --chain-probe-count 4 \
+  --concurrency 4
+```
+
+Optional flags:
+
+```bash
+--scenario-count 4
+--generated-count 24
+--chain-probe-count 4
+--concurrency 4
+--seed runtime-dual-write-v2
+--max-per-bucket 8
+--output-dir reports/runtime-dual-write-manual
+```
+
+Sustained soak example:
+
+```bash
+npm run check:runtime-dual-write -- \
+  --runtime-root /Volumes/ziel/AionisRuntime-focused \
+  --generated-count 96 \
+  --chain-probe-count 16 \
+  --concurrency 8
+```
+
+## Report
+
+The runner writes:
+
+```text
+reports/runtime-dual-write-*/summary.json
+```
+
+The report includes:
+
+- focused Runtime base URL and isolated Lite SQLite paths
+- Substrate SQLite path
+- Runtime memory ids returned by `observeStep`
+- Runtime guide surfaces
+- Substrate compiled surfaces
+- per-bucket parity details
+- Substrate event counts
+- generated/fixed scenario counts
+- invalid write rollback probes
+- chain probe counts for lifecycle transitions, supersession, invalidation, support, and payload relations
+- persisted parity after close/reopen
+- scenario, chain-probe, and reopen latency summaries
+- append-only event sequence continuity
+- Substrate and focused Runtime SQLite file sizes
+
+## Chain Probes
+
+Chain probes run in independent Substrate scopes after the real Runtime sidecar scenarios.
+
+Each probe writes:
+
+- one verified active memory expected in `use_now`
+- one candidate procedure expected in `inspect_before_use`
+- one prior memory superseded by the active memory
+- one failed branch invalidated by the active memory and transitioned to `blocked`
+- one raw payload pointer that requires rehydration
+
+The probe then compiles context, compares the four admission buckets, closes and reopens the SQLite store, and compares again. This validates the substrate relation and lifecycle contract without adding benchmark-specific policy to focused Runtime.
+
+## Soak Metrics
+
+The report `soak` object is for sustained sidecar checks:
+
+- `scenario_latency`: p50/p95/max latency for real Runtime sidecar scenarios.
+- `chain_probe_latency`: p50/p95/max latency for Substrate-only lifecycle/relation probes.
+- `reopen_latency`: p50/p95/max latency for persisted scenario checks after closing and reopening SQLite.
+- `chain_probe_reopen_latency`: persisted chain probe reopen latency.
+- `event_sequence`: verifies append-only event sequence continuity, duplicate count, and gap count.
+- `db_sizes`: records Substrate SQLite/WAL/SHM file sizes and focused Runtime Lite SQLite file sizes.
+
+## Boundary
+
+This experiment is intentionally sidecar-only:
+
+- no Runtime source mutation
+- no Runtime storage replacement
+- no production dual-write API
+- no Agent framework logic
+- no benchmark-specific Runtime policy
+
+Passing this experiment means Substrate can mirror a small real Runtime execution loop and preserve the same admission surface after reopen. It does not mean Substrate is ready to become the focused Runtime storage engine.