skillscript-runtime 0.13.2 → 0.13.5

package/docs/sqlite-skill-store.md

@@ -0,0 +1,167 @@
+# SqliteSkillStore — example
+
+Worked example of `SkillStore` against a SQLite database. Copy this directory into your codebase, customize per your storage needs, register with skillscript-runtime's `Registry`. This README is written for the agent implementing your adopter's connector — including the human reviewing the PR.
+
+**What this demonstrates**: the locked SkillStore contract surface (8 methods + `staticCapabilities`) wired through real SQL with two-table versioning, transactional status transitions, and JSON-extract tag filters.
+
+---
+
+## The three-leg model
+
+SkillStore is pluggable. Three legs ship out of the box, but the third is the open one:
+
+```
+SkillStore (choose which connector)
+   ├── FilesystemSkillStore (bundled, FS-backed)
+   ├── SqliteSkillStore     (this example, DB-backed)
+   └── Adopter-custom        (you write your own)
+```
+
+If your substrate is AMP, Pinecone, S3, Postgres, or anything else, write a `class FooSkillStore implements SkillStore { ... }` and call `registry.registerSkillStore("primary", new FooSkillStore(...))`. The runtime is none the wiser.
+
+This SqliteSkillStore is one such impl — useful as a copy-paste starting point, or directly usable if your needs match.
+
+---
+
+## Quick start
+
+```typescript
+import { Registry } from "skillscript-runtime";
+import { SqliteSkillStore } from "./SqliteSkillStore.js";
+
+const registry = new Registry();
+registry.registerSkillStore("primary", new SqliteSkillStore({ dbPath: "skills.db" }));
+
+// Wire the rest of the runtime (scheduler, mcpServer) using the registry.
+```
+
+Author skills programmatically:
+
+```typescript
+const store = registry.getSkillStore("primary");
+await store.store("morning-status", `# Skill: morning-status
+# Status: Approved
+t:
+    ! report status
+default: t
+`);
+```
+
+Or via the dashboard, or via `skill_write` MCP tool — same backend, same result.
+
+---
+
+## When to use SqliteSkillStore (and when not)
+
+**Use it when**:
+
+- You're embedding skillscript-runtime as a library and want skills in a database rather than `.skill.md` files
+- Your deployment has no persistent filesystem (container-only) but does have SQLite
+- You need richer query semantics than filesystem listing (tag filters, transactional status transitions)
+
+**Don't use it when**:
+
+- You're using the bundled CLI (`skillfile compile`, `skillfile lint`, `skillfile list`). The CLI is filesystem-first by design — `vim foo.skill.md && skillfile lint foo` is the natural authoring loop. The CLI does NOT use SqliteSkillStore. Sqlite-backed skills are authored via dashboard or `skill_write` MCP tool.
+- Your skills are authored as files-on-disk and committed to git as part of the source tree. FilesystemSkillStore is the right choice there.
+
+This is "first-class" in the **programmatic-embedding** sense, not the CLI sense. Treat the example as a substrate-portability proof point + a copy-paste starting point.
+
+---
+
+## Schema
+
+Two tables:
+
+```sql
+CREATE TABLE skills (
+  name TEXT PRIMARY KEY,
+  current_version TEXT NOT NULL,
+  content_hash TEXT NOT NULL,
+  status TEXT NOT NULL,                  -- 'Draft' | 'Approved' | 'Disabled'
+  source TEXT NOT NULL,                  -- the .skill.md body bytes
+  description TEXT,                      -- extracted from `# Description:` header
+  metadata_json TEXT,                    -- optional metadata bag (tags, author, etc.)
+  created_at INTEGER NOT NULL,
+  updated_at INTEGER NOT NULL,
+  status_changed_at INTEGER
+);
+
+CREATE TABLE skill_versions (
+  name TEXT NOT NULL,
+  version TEXT NOT NULL,                 -- short hash (12 chars)
+  content_hash TEXT NOT NULL,
+  source TEXT NOT NULL,                  -- full body, preserved per version
+  status TEXT NOT NULL,
+  previous_status TEXT,                  -- populated by update_status
+  changed_at INTEGER NOT NULL,
+  changed_by TEXT,
+  PRIMARY KEY (name, version)
+);
+```
+
+`skills` is the fast-path read for `load()` / `metadata()` / `query()`. `skill_versions` is the append-only history for `versions()` — and unlike FilesystemSkillStore, it preserves full body bytes per version, so `load(name, version)` can return historical content.
+
+WAL is enabled at bootstrap so concurrent readers don't block writers.
+
+---
+
+## Footgun guard: `delete()` is hard-cascade
+
+**`delete()` removes both the `skills` row AND its `skill_versions` rows.** If you need recovery, back up adopter-side BEFORE calling delete. Skill names can be reused after delete (no orphan history left behind).
+
+This is the locked semantic. If your adopter substrate has compliance requirements (audit-grade retention) or you hit a "wait, I deleted that" moment, the upgrade path is soft-delete (tombstone `status='Deleted'` + filter from query results) — but that's an adopter-side choice; the bundled SqliteSkillStore stays hard-cascade.
+
+---
+
+## Feature flags
+
+`staticCapabilities()` declares:
+
+| Feature | Value | What it means |
+|---|---|---|
+| `supports_writes` | ✓ | `store()` / `update_status()` / `delete()` mutate state |
+| `supports_versioning` | ✓ | `versions()` returns history; `load(name, version)` returns historical bytes |
+| `supports_tag_filter` | ✓ | `query({ tag: "foo" })` works via `json_extract(metadata_json, '$.tags')` (O(n) scan) |
+| `supports_audit_trail` | ✓ | `update_status()` populates `previous_status` on every transition |
+| `supports_atomic_status_transitions` | ✓ | UPDATE skills + INSERT skill_versions wrapped in a transaction |
+
+The atomic transitions are the SQL advantage over FilesystemSkillStore (which declares `supports_atomic_status_transitions: false` because filesystem writes can tear between body rewrite + sidecar append).
+
+---
+
+## Authoring loop
+
+SqliteSkillStore is the storage layer. Authoring happens above:
+
+- **Dashboard**: visit `http://localhost:7878`, create/edit skills through the UI; dashboard writes via `skill_write` MCP tool → SqliteSkillStore
+- **MCP tool**: agents call `skill_write` directly; same path as the dashboard
+- **Programmatic**: your code calls `store.store(name, source, metadata?)` directly
+
+The dashboard does NOT default to SqliteSkillStore today — `skillfile dashboard` bootstraps with FilesystemSkillStore. If you want a SqliteSkillStore-backed dashboard, write a custom bootstrap (small surface; see the runtime `Registry` API).
+
+---
+
+## Approval-token stamping
+
+Skill bodies that declare `# Status: Approved` get a token stamped on `store()` automatically: `# Status: Approved v1:<hex>`. Same behavior as FilesystemSkillStore. Transitions to `Approved` via `update_status()` stamp the token; transitions away strip it.
+
+Adopters who want a stronger `f()` for the token (HMAC-SHA256 instead of the default) can register a custom approval fn before calling `store()`/`update_status()`:
+
+```typescript
+import { registerApprovalFn } from "skillscript-runtime";
+registerApprovalFn("v2", (body) => hmacSha256(SECRET, body));
+```
+
+---
+
+## Forking checklist
+
+When forking into your codebase:
+
+1. Rename the class (e.g., `PostgresSkillStore`, `AmpSkillStore`)
+2. Replace the SQL with your substrate's API (HTTP, MemoryStore, vector DB, etc.)
+3. Update `staticCapabilities()` to match what your substrate actually supports — drop `supports_versioning` if you can't track history, drop `supports_tag_filter` if querying tags isn't tractable
+4. Update `manifest()` to describe your substrate (`kind: "amp"` or whatever)
+5. Tests: copy `tests/SqliteSkillStore.test.ts` as a starting point + run the conformance suite (`SkillStoreConformance.buildTests()` from `skillscript-runtime/testing`)
+
+The conformance suite catches drift from the contract surface. If your fork passes, the runtime treats it interchangeably with FilesystemSkillStore / SqliteSkillStore / etc.

package/examples/skillscripts/hello.skill.provenance.json

@@ -2,7 +2,7 @@
   "provenance_version": "1.0",
   "language_version": "1.0",
   "compiler_version": "0.1.0-dev",
-  "compiled_at": "2026-05-29T18:50:40.018Z",
+  "compiled_at": "2026-05-29T20:29:47.591Z",
   "source_skill": {
     "name": "hello"
   },

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "skillscript-runtime",
-  "version": "0.13.2",
+  "version": "0.13.5",
   "description": "Runtime, compiler, lint, CLI, and dashboard for Skillscript — a small declarative language for authoring agent workflows.",
   "license": "MIT",
   "author": "Scott Shwarts <scotts@pobox.com>",
@@ -61,7 +61,7 @@
     "dist",
     "scaffold",
     "examples",
-    "ARCHITECTURE.md",
+    "docs/*.md",
     "README.md",
     "LICENSE"
   ],
@@ -69,7 +69,8 @@
     "node": ">=22.5"
   },
   "scripts": {
-    "build": "tsc -p tsconfig.build.json && node scripts/copy-dashboard-assets.mjs",
+    "build:dist": "tsc -p tsconfig.build.json && node scripts/copy-dashboard-assets.mjs",
+    "build": "pnpm run build:dist && node scripts/check-published-paths.mjs",
     "dev": "tsc -p tsconfig.build.json --watch",
     "test": "vitest run",
     "test:watch": "vitest",

package/ARCHITECTURE.md

@@ -1,102 +0,0 @@
-# Architecture
-
-One-page map of the `skillscript-runtime` codebase. Per ERD §1, the *narrow core* (parser + compile + runtime + lint + connectors/) stays under the nudged LOC ceiling — currently 5650, tracked by `scripts/loc-ceiling.mjs` with full history in that file's header comment. Auxiliary surface (CLI, dashboard, MCP server, scheduler, observability) is reported but doesn't gate the build. Tests count separately.
-
-## Top-level layout
-
-```
-src/
-  index.ts              — library entrypoint; named exports for embedders
-  cli.ts                — `skillfile` CLI entrypoint
-  parser.ts             — source text → AST (NARROW CORE)
-  compile.ts            — AST → resolved skill model → rendered artifact (NARROW CORE; owns toposort)
-  filters.ts            — pipe-filter implementations (NARROW CORE via connectors/; see notes)
-  lint.ts               — structural validation (NARROW CORE)
-  runtime.ts            — executor: walks compiled artifact, dispatches ops (NARROW CORE)
-  composition.ts        — `$ execute_skill` + `& invoke` runtime dispatch (v0.2.8)
-  version.ts            — single-source RUNTIME_VERSION from package.json (v0.2.12)
-  help-content.ts       — `help({topic})` MCP tool content
-  errors.ts             — OpError class hierarchy + structured runtime errors
-  provenance.ts         — ProvenanceBlock + content_hash recording (Phase 3)
-  scheduler.ts          — trigger registry + cron scan + EVENT.* ambient population
-  audit.ts              — `skillfile audit` recompile-staleness detector
-  trace.ts              — TraceBuilder + on-disk trace store
-  metrics.ts            — health-metrics aggregator
-  skill-manager.ts      — high-level skill lifecycle helpers
-  bootstrap.ts          — wires everything: SkillStore + Registry + MCP server + dashboard
-  mcp-server.ts         — JSON-RPC 2.0 MCP server (13 tools)
-  connectors/           — NARROW CORE
-    types.ts            — contracts: SkillStore, MemoryStore, LocalModel, McpConnector
-    agent.ts            — AgentConnector contract (Augmenting/Template delivery)
-    agent-noop.ts       — default AgentConnector (no-op delivery)
-    registry.ts         — per-kind instance registry + three-layer resolution
-    skill-store.ts      — bundled default: filesystem at $SKILLSCRIPT_HOME/skills/
-    memory-store.ts     — bundled default: SQLite + FTS at $SKILLSCRIPT_HOME/memory.db
-    local-model.ts      — bundled default: Ollama at localhost:11434
-    mcp.ts              — bundled default: stub; no servers wired by default
-    config.ts           — connectors.json loader (v0.4.0+): parse + validate + ${ENV}/env-block-as-scope substitution + closed-set class registry + gitignore-detection (v0.4.1)
-    mcp-remote.ts       — RemoteMcpConnector (v0.4.1): child-process spawn + JSON-RPC stdio framing (lsp + newline) + initialize handshake + tool dispatch + lifecycle
-    index.ts            — barrel re-exports
-  dashboard/            — Vite SPA + dashboard HTTP server (v0.2.7)
-  testing/              — test-only helpers shipped with the package
-```
-
-Narrow-core LOC history (nudges driven by language extensions):
-- 5000 (T7 baseline) → 5100 (v0.2.10 parser robustness) → 5200 (v0.2.12 lint coverage) → 5400 (v0.3.0 `$append` accumulator) → 5500 (v0.3.1 forward-reference deferred resolution) → 5650 (v0.3.2 and/or/not + filter chain + `|json_parse`) → 5700 (v0.3.3 `$ json_parse` op + lint advisory + Bug D parser-recovery; `|json_parse` filter removed) → 5750 (v0.3.4 conditional multi-filter chain + parse-error dedup + unified sink-scope recovery) → 6000 (v0.4.0 connectors.json loader + closed-set class registry + two new lint rules + credential discipline) → 6600 (v0.4.1 RemoteMcpConnector + allowed_tools allowlist + env-block-as-scope + framing config + gitignore-detect + lint auto-wire + foreach-over-parsed-JSON + kwarg coercion)
-
-## What each narrow-core file owns
-
-| File | Responsibility |
-| --- | --- |
-| `parser.ts` | Tokenize and parse skill source. Header lines, target blocks, op grammar (`!`/`$`/`$set`/`$append`/`?`/`@`/`>`/`~`/`&`/`??`), `if`/`elif`/`else`/`foreach`, compound conditions (`and`/`or`/`not` + parens since v0.3.2). Produces AST. Recursive structural decomposition for compound conditions in `validateCondition`. Syntax errors only — semantic checks downstream. |
-| `compile.ts` | Three subsystems: (1) variable resolution against `# Requires:` cascade + caller inputs; (2) data-skill compile-time inlining; (3) topo-sort + render. Output formats: `prompt` (canonical), `prose`. Forward-reference deferral for missing `&` targets (v0.3.1). Produces compiled artifact + provenance sidecar. |
-| `filters.ts` | Pipe-filter implementations dispatched by `$(NAME\|filter)` syntax. v1 set: `url`, `shell`, `json`, `trim`, `length`. (`json_parse` was a v0.3.2 addition removed in v0.3.3 — use the `$ json_parse $(VAR) -> OUT` op instead, which binds structured shape.) Adding a new filter = adding a case to `applyFilter` + registering in `KNOWN_FILTERS` + documenting in `help-content.ts`. |
-| `lint.ts` | Structured diagnostics across 3 tiers. ~30 rules covering parse errors, var resolution, condition grammar, composition refs (`unknown-skill-reference` demoted to tier-2 in v0.3.1 with tier-3 `deferred-skill-reference` advisory), shell safety (`unsafe-shell-op`, `unsafe-shell-disabled`, `unsafe-shell-ambiguous-subst`), mutation safety (`unconfirmed-mutation`), accumulator safety (v0.3.0 `uninitialized-append`, `foreach-local-accumulator-target`, `append-to-non-list`), retrieval-arg validation, credential leak detection. |
-| `runtime.ts` | Executor that walks the compiled artifact and dispatches ops through connector instances. Owns `evalCondition` (compound + leaf shapes), `substituteRuntime` (filter-chain aware since v0.3.2), `resolveRef` (dotted + indexed field access). Handles error propagation, per-op timeout chain, `foreach` iteration with loop-local scope, target-level `else:` error handler, `# OnError:` skill fallback, mechanical-mode placeholders. |
-| `connectors/*` | The integration boundary — every external system (skill storage, memory, local model, MCP, agent delivery) plugs in through one of the typed contracts. Registry handles multi-instance + three-layer resolution: per-call override > skill-declared > primary default. |
-
-## Auxiliary surface (outside narrow core)
-
-| File | Responsibility |
-| --- | --- |
-| `cli.ts` | `skillfile` CLI entrypoint. Commands: `init`, `execute` (since v0.2.11; `run` alias dropped v0.2.12), `compile`, `audit`, `lint`, `list`, `fires`, `diagram`, `sign`, `verify`, `replay`, `health`, `serve`, `dashboard`. Per-command `--help`. Version from `src/version.ts`. |
-| `mcp-server.ts` | JSON-RPC 2.0 MCP server exposing 13 tools: `skill_list/metadata/status/write`, `list/register/unregister_trigger`, `health_metrics`, `runtime_capabilities`, `lint_skill`, `compile_skill`, `execute_skill`, `help`. Rolled-by-hand JSON-RPC handler (no `@modelcontextprotocol/sdk` dependency). |
-| `composition.ts` | In-skill composition primitive runtime. `$ execute_skill` intercept + recursion-depth guard (default 10). Distinct from data-skill `&` inlining which is compile-time. |
-| `scheduler.ts` | Trigger registry + cron firing + EVENT.* ambient auto-population (`fired_at`, `fired_at_unix`, `fired_at_plus_{1h,1d,7d}_unix`). Status-aware: skips Draft/Disabled at fire time. Persistent trigger registry on disk (v0.2.7). |
-| `bootstrap.ts` | Top-level wiring: takes `{skillsDir, traceDir, enableUnsafeShell?}` → returns `{registry, skillStore, scheduler, mcpServer, ...}`. The integration test entry point. |
-| `dashboard/` | Vite SPA + Express HTTP server. Skill list + status + trace viewer. Mounted under `/` when `skillfile dashboard` runs; serve-headless mode (`skillfile serve`) omits the SPA. |
-| `audit.ts` | `skillfile audit <provenance.json>` — detects stale compiled artifacts when source data-skills have been re-stored since compile. |
-| `trace.ts` | TraceBuilder + FilesystemTraceStore. Records per-op timing, dispatch, error chain. |
-| `metrics.ts` | Aggregates trace data into `health_metrics` MCP response (request counts, op latencies, error rates). |
-| `help-content.ts` | Static markdown content for the `help({topic})` MCP tool. Topics: `ops`, `frontmatter`, `examples`, `composition`, `connectors`, `lint-codes`. |
-| `version.ts` | Reads `package.json` at module load to single-source the runtime version (added v0.2.12 after a missed bump exposed the triple-source duplication). |
-
-## Non-source
-
-```
-docs/                   — spec docs (ERD, Language Reference, README)
-examples/               — bundled example skills (5 worked examples, see help({topic:"examples"}))
-scripts/loc-ceiling.mjs — CI check; fails if narrow core exceeds budget. Header has full nudge history.
-tests/                  — vitest specs (39 files, 829 passing)
-tests/fixtures/harness/ — 66 cold-author skills + classified manifest (regression corpus)
-.github/workflows/      — CI: release.yml fires on tag push → test → GHCR multi-arch + GitHub Release + npm publish
-Dockerfile              — multi-arch (linux/amd64 + linux/arm64) image base
-```
-
-## CI pipeline (release.yml)
-
-Tag push (`vX.Y.Z`) → typecheck → loc-check → build → full vitest → version verify (tag matches package.json) → Docker Buildx multi-arch GHCR push → GitHub Release with CHANGELOG section as body → npm publish.
-
-Required secret: `NPM_TOKEN` (granular access token with **Bypass two-factor authentication when publishing** enabled). Token requirement documented in memory `feedback_npm_publish_2fa`. Resolved end-to-end at v0.3.1 after the chronic v0.2.5–v0.2.12 failure pattern (root cause: secret missing, then set without bypass-2FA flag).
-
-## Build + dev
-
-- `pnpm install --frozen-lockfile` — install deps
-- `pnpm run build` — `tsc -p tsconfig.build.json` + copy dashboard SPA assets to `dist/`
-- `pnpm exec vitest run` — full suite
-- `pnpm run loc-check` — narrow-core ceiling check (CI gate)
-- `node dist/cli.js dashboard --host 0.0.0.0 --port 7878` — local dashboard
-- `node dist/cli.js execute <skill>` — run a skill end-to-end
-- `node dist/cli.js compile <skill>` — render compiled artifact without executing
-
-ESM-only. Node 22+ required (`node:sqlite`). pnpm 11.