theorum 0.1.13 → 0.1.15

package/README.md

@@ -10,6 +10,8 @@
 
 # THEORUM: The Flat Agent Kernel
 
+**Current release: `0.1.15`** (`jsr:@theorum/core` / npm `theorum`).
+
 > **"Profiles describe the contract. Providers move bytes. The runner enforces the turn."**
 
 THEORUM is a compact TypeScript agent kernel for apps that need deterministic agent execution without embedding product logic inside the runtime. It gives a host application one runner, typed profiles, multimodal input normalization, dynamic tool dispatch, provider adapters, trace sinks, and guardrail hooks.
@@ -339,15 +341,55 @@ as `createLocalProvider` from the main / providers entrypoints. Importing
 
 Internal files remain present in source for maintainability, but package consumers should use the public entrypoints above.
 
+### Exported API (`mod.ts`)
+
+Named exports from the root barrel (same symbols hosts get from `theorum` /
+`jsr:@theorum/core`):
+
+| Group | Symbols |
+| --- | --- |
+| Guardrails errors | `describeError`, `isAbortError`, `publicError`, `TheorumError`, `throwIfAborted`, `toErrorEvent` |
+| Quota | `QuotaSlotStatus`, `clientIp`, `quotaMessage`, `releaseSlot`, `resetSlots`, `skipQuota`, `takeSlot` |
+| Sanitize | `PROJECT_ID_MAX`, `sanitizeProjectId`, `sanitizeText`, `sanitizeTurnRequest` |
+| Compaction | `CompactionSplit`, `CompactionTokens`, `compactionMeter`, `compactionNeeded`, `estimateHistoryTokens`, `HISTORY_MEDIA_TOKENS`, `HISTORY_TEXT_ENCODING`, `resolveCompactionTokens`, `resolveHistoryTokens`, `shouldCompact`, `splitForCompaction` |
+| Runner | `runTurn` |
+| Catalog | `CATALOG`, `clampThinkingLevel`, `clampThinkingLevelForApiId`, `mediaKindForMime`, `getTool`, `listBuiltinIds`, `mimeAllowed`, `mimeEssence`, `modelEntryByApiId`, `registerTools`, `requireModelSpec`, `resetTools` |
+| Profiles | `ProfileDefinition`, `clearProfiles`, `defineProfile`, `getProfile`, `hasProfile`, `listProfiles`, `registerProfile`, `registerProfiles`, `projectProfile`, `resolveTurn` |
+| Structured | `getStructured`, `registerStructured`, `executeTool` |
+| Stop / resume | `ProfileResumeSpec`, `TurnContinueFrom`, `TurnStop`, `TurnStopKind`, `AUTO_CONTINUE_DELAY_MS`, `CONTINUE_INSTRUCTION`, `DEFAULT_AUTO_CONTINUE`, `GenerationStopError`, `isGenerationStopError`, `isResumeableStop`, `isUserCancelledStop`, `shouldAutoContinue`, `turnStopFromClientStreamEnd`, `turnStopFromInteractionStatus`, `turnStopFromOpenRouter` |
+| Observability | `jsonlSink`, `memorySink`, `noopSink`, `resolveTraceDir`, `sinkFromDir`, `writeTrace`, `TraceRecord` |
+| Providers | `CreateProviderOptions`, `GeminiTransport`, `GeminiVault`, `LocalProviderConfig`, `createLocalProvider`, `createProvider`, `DEFAULT_LOCAL_BASE_URL` |
+
+Kernel types re-exported through this barrel follow `export type *` from
+`src/kernel/types.ts` (see `src/kernel/CONTRACT.md`).
+
 ---
 
 ## Documentation
 
-| Doc | Topic |
+Package docs are co-located with each public export (plus this README for `.`):
+
+| Doc | Export |
 | :--- | :--- |
-| [`docs/SECRETS.md`](docs/SECRETS.md) | Host-owned credentials; `createProvider` args; no env reads |
-| [`docs/COMPACTION.md`](docs/COMPACTION.md) | History / input meters, lazy BPE, `trigger`, signals |
-| [`docs/STOP.md`](docs/STOP.md) | Normalized `done.stop`, resume policy, `continueFrom` |
+| [`src/kernel/CONTRACT.md`](src/kernel/CONTRACT.md) | `theorum/kernel` — profiles, runner, compaction, stop/resume |
+| [`src/providers/CONTRACT.md`](src/providers/CONTRACT.md) | `theorum/providers` — `createProvider`, secrets boundary |
+| [`src/providers/OPENROUTER.md`](src/providers/OPENROUTER.md) | `theorum/openrouter` |
+| [`src/guardrails/CONTRACT.md`](src/guardrails/CONTRACT.md) | `theorum/guardrails` |
+| [`src/observability/CONTRACT.md`](src/observability/CONTRACT.md) | `theorum/observability` |
+| [`src/host/CONTRACT.md`](src/host/CONTRACT.md) | `theorum/host` |
+| [`src/cli/CONTRACT.md`](src/cli/CONTRACT.md) | `theorum/cli` |
+| [`src/presets/CONTRACT.md`](src/presets/CONTRACT.md) | `theorum/presets` |
+| [`src/presets/GOOGLE.md`](src/presets/GOOGLE.md) | `theorum/presets/google` |
+| [`src/streaming/CONTRACT.md`](src/streaming/CONTRACT.md) | `theorum/streaming` |
+
+Document health is enforced by `npm run lint:docs` — the **first** step of
+`npm run lint` / `deno task lint` (`docs/_map.mjs`):
+
+- Full production-file ownership (`mod.ts`, `src/**/*.ts`, `package.json`, docs-truth scripts)
+- Export parity with `package.json` and export-drift vs entry `mod.ts` files
+- Doc + **section** freshness on every code change (no Export-only gaming)
+- Behavioral sections require `contract_test` evidence (≥2 supports each)
+- Pre-commit runs `lint:docs` automatically (`prepare` installs the hook on `npm install`)
 
 ---
 
@@ -415,3 +457,40 @@ If an app needs domain rules, platform delivery policy, product copy, database a
 ## License
 
 MIT License. Copyright (c) ORCHID AI LLC.
+
+```theorum-evidence
+{
+  "sections": {
+    "Core Principles": {
+      "supports": [
+        { "kind": "source", "path": "mod.ts" },
+        { "kind": "contract_test", "path": "tests/kernel/theorum.test.ts" }
+      ]
+    },
+    "Architecture": {
+      "supports": [
+        { "kind": "source", "path": "src/kernel/engine/runner.ts" },
+        { "kind": "contract_test", "path": "tests/kernel/theorum.test.ts" }
+      ]
+    },
+    "Public Entrypoints": {
+      "supports": [
+        { "kind": "config", "path": "package.json" },
+        { "kind": "contract_test", "path": "scripts/docs-truth/graph.test.mjs" }
+      ]
+    },
+    "Documentation": {
+      "supports": [
+        { "kind": "graph", "path": "docs/_map.mjs" },
+        { "kind": "contract_test", "path": "scripts/docs-truth/graph.test.mjs" }
+      ]
+    },
+    "Package Boundary": {
+      "supports": [
+        { "kind": "source", "path": "src/providers/create-provider.ts" },
+        { "kind": "contract_test", "path": "tests/providers/create-provider.test.ts" }
+      ]
+    }
+  }
+}
+```

package/docs/cli.md

@@ -0,0 +1,97 @@
+# CLI (`theorum/cli`)
+
+Profile inspection and stress-test CLI. On npm this entry is also the
+`theorum` binary. Hosts must register profiles (and providers) in-process
+before commands that execute turns — the CLI does not embed app profiles.
+
+## Export
+
+| Field | Value |
+| --- | --- |
+| Import | `theorum/cli` / `jsr:@theorum/core/cli` |
+| Module | `src/cli/index.ts` |
+| Binary | `theorum` (npm `bin`) |
+
+## Ownership
+
+| Path | Role |
+| --- | --- |
+| `src/cli/index.ts` | Argument parser + command dispatch |
+| `src/cli/commands/*` | `bench`, `fuzz`, `test`, `run`, `profile` |
+| `src/cli/matrix/*` | Permutation synthesizer + fixtures |
+
+## Commands
+
+```text
+theorum <command> [options]
+```
+
+| Command | Purpose |
+| --- | --- |
+| `bench` | Synthetic kernel performance benchmark (`--chunks`, `--iterations`, `--warmup`) |
+| `fuzz` | Adversarial guardrail fuzzer |
+| `test` | Stress matrix or custom profile tests (`--profile`, `--all`, `--lite`, `--matrix`, `--mode`, `--search`, `--map`) |
+| `run` | Execute a turn with streaming output (`--profile`, `--prompt`, `--mode`, …) |
+| `profile list` / `profile show <id>` | Inspect registered profile blueprints |
+| `help` | Usage |
+
+Exit code `1` on failed `test` runs. `run` requires `--profile` (or `-p`).
+
+```bash
+theorum test --profile my.agent --matrix
+theorum run --profile my.agent --prompt "ping"
+```
+
+## Matrix and fixtures
+
+| Module | Role |
+| --- | --- |
+| `matrix/synthesizer.ts` | Builds valid permutation cases (modes, optional tools) |
+| `matrix/fixtures.ts` | Shared harness fixtures (not product personas) |
+
+The matrix respects profile allowlists — e.g. `--search` only applies when
+`googleSearch` is allowlisted.
+
+## Exported API
+
+The entry module is the CLI program itself (side-effect main when run as a
+bin). Prefer `deno task theorum` / `npx theorum` over importing commands in
+application code.
+
+```theorum-evidence
+{
+  "sections": {
+    "Export": {
+      "supports": [
+        { "kind": "source", "path": "src/cli/index.ts" },
+        { "kind": "config", "path": "package.json" }
+      ]
+    },
+    "Ownership": {
+      "supports": [
+        { "kind": "source", "path": "src/cli/index.ts" },
+        { "kind": "graph", "path": "docs/_map.mjs" }
+      ]
+    },
+    "Commands": {
+      "supports": [
+        { "kind": "source", "path": "src/cli/index.ts" },
+        { "kind": "source", "path": "src/cli/commands/run.ts" },
+        { "kind": "contract_test", "path": "tests/cli/cli.test.ts" }
+      ]
+    },
+    "Matrix and fixtures": {
+      "supports": [
+        { "kind": "source", "path": "src/cli/matrix/synthesizer.ts" },
+        { "kind": "contract_test", "path": "tests/cli/cli.test.ts" }
+      ]
+    },
+    "Exported API": {
+      "supports": [
+        { "kind": "source", "path": "src/cli/index.ts" },
+        { "kind": "contract_test", "path": "tests/cli/cli.test.ts" }
+      ]
+    }
+  }
+}
+```

package/docs/guardrails.md

@@ -0,0 +1,178 @@
+# Guardrails (`theorum/guardrails`)
+
+Generic inbound and outbound guardrail primitives. App-specific policy,
+product copy, and channel UX remain host-owned — this entry ships reusable
+detectors, sanitizers, public error mapping, and optional per-day quota slots.
+
+## Export
+
+| Field | Value |
+| --- | --- |
+| Import | `theorum/guardrails` / `jsr:@theorum/core/guardrails` |
+| Module | `src/guardrails/mod.ts` |
+| Also on | Root `theorum` re-exports common error/sanitize/quota helpers |
+
+## Ownership
+
+Owns every module under `src/guardrails/`.
+
+| Module | Role |
+| --- | --- |
+| `error.ts` | `TheorumError`, `publicError`, abort helpers |
+| `sanitize.ts` | Turn + text sanitization |
+| `injection.ts` | Prompt-injection span patterns |
+| `sensitive.ts` | Credential / PII span patterns |
+| `normalize.ts` | Detection normalization |
+| `quota.ts` | In-memory daily slots for HTTP hosts |
+
+## Public errors
+
+`TheorumError` marks expected contract failures. Never show raw internal
+messages to end users — map through `publicError(err)` (or `toErrorEvent` for
+streams).
+
+| Internal marker | Public copy |
+| --- | --- |
+| `UPSTREAM_FAILED` | `PUBLIC_UNAVAILABLE` |
+| `canary leaked` / egress violations | `PUBLIC_CANARY` |
+| Abort | `PUBLIC_CANCELLED` |
+| Tool / MIME / size denials | `PUBLIC_ACTION` / `PUBLIC_FILE_*` |
+
+`describeError` returns structured detail for logs. `throwIfAborted(signal)`
+rethrows `AbortError` when a turn should stop early.
+
+Exact-message and regex rules live in `error.ts` (`EXACT`, `RULES` arrays) —
+extend there when adding new stable public mappings.
+
+## Sanitization
+
+Driven by profile `guardrails.sanitizeInput` and `guardrails.redactSensitive`
+(default both on in `sanitizeText` unless overridden).
+
+| API | Role |
+| --- | --- |
+| `sanitizeText` | Strip injection + sensitive spans from one string |
+| `sanitizeTurnRequest` | Full turn: text, slots, dynamic tool args, blobs |
+| `sanitizeProjectId` | Bound project id strings (`PROJECT_ID_MAX`) |
+
+`injectionSpans` and `sensitiveSpans` return `RedactSpan[]`; `applySpans`
+(from observability) performs replacement. Detection runs on normalized text
+(`normalizeForDetection`).
+
+### Injection categories (non-exhaustive)
+
+Patterns target untrusted user text before provider submission:
+
+- Instruction override (`ignore previous instructions`, `disregard rules`, …)
+- Mode hijack (`developer mode`, `jailbreak`, `DAN`, `do anything now`)
+- Safety bypass (`disable safety filters`, …)
+- Role / delimiter forgery (`<system>`, `[System Message]`, ChatML tokens)
+- Prompt exfiltration (`reveal your system prompt`, …)
+- Multilingual override fragments
+
+False-positive tuning belongs in `injection.ts` tests
+(`tests/guardrails/false-positives.test.ts`).
+
+## Sensitive data
+
+| API | Role |
+| --- | --- |
+| `sensitiveSpans` | Credential / PII span detection |
+| `redactSensitiveOnly` | Model output path without injection patterns |
+
+`sensitiveSpans` redacts credential-like and PII patterns from inbound text and,
+when enabled, outbound paths. Use `redactSensitiveOnly` on model output when
+injection patterns should not run.
+
+## Quota
+
+**Not** enforced inside `runTurn`. HTTP hosts call:
+
+```ts
+const ip = clientIp(peer, req);
+if (skipQuota(peer, req)) { /* local dev */ }
+const status = takeSlot(profile, ip, Date.now());
+// 'ok' | 'busy' | 'quota' | 'not_configured'
+try {
+  await runTurn(...);
+} finally {
+  releaseSlot(profile, ip);
+}
+```
+
+| Status | Meaning |
+| --- | --- |
+| `ok` | Slot taken; increment daily count |
+| `busy` | Same ip/profile already in flight |
+| `quota` | `perDay` exhausted |
+| `not_configured` | Profile has no `guardrails.quota` |
+
+`quotaMessage(profile)` uses `identity.handle` for user-facing limit copy.
+`resetSlots()` clears in-memory state (tests).
+
+## Exported API
+
+From `src/guardrails/mod.ts`:
+
+| Group | Symbols |
+| --- | --- |
+| Public errors | `describeError`, `isAbortError`, `publicError`, `TheorumError`, `throwIfAborted`, `toErrorEvent`, `PUBLIC_ACTION`, `PUBLIC_CANARY`, `PUBLIC_CANCELLED`, `PUBLIC_FILE_COUNT`, `PUBLIC_FILE_SIZE`, `PUBLIC_FILE_TYPE`, `PUBLIC_GENERIC`, `PUBLIC_IMAGE_SIZE`, `PUBLIC_UNAVAILABLE`, `UPSTREAM_FAILED` |
+| Injection / sensitive | `injectionSpans`, `sensitiveSpans` |
+| Quota | `QuotaSlotStatus`, `clientIp`, `quotaMessage`, `releaseSlot`, `resetSlots`, `skipQuota`, `takeSlot` |
+| Sanitize | `PROJECT_ID_MAX`, `sanitizeProjectId`, `sanitizeText`, `sanitizeTurnRequest` |
+
+```theorum-evidence
+{
+  "sections": {
+    "Export": {
+      "supports": [
+        { "kind": "source", "path": "src/guardrails/mod.ts" },
+        { "kind": "config", "path": "package.json" }
+      ]
+    },
+    "Ownership": {
+      "supports": [
+        { "kind": "source", "path": "src/guardrails/mod.ts" },
+        { "kind": "graph", "path": "docs/_map.mjs" }
+      ]
+    },
+    "Public errors": {
+      "supports": [
+        { "kind": "source", "path": "src/guardrails/error.ts" },
+        { "kind": "contract_test", "path": "tests/guardrails/error.test.ts" }
+      ]
+    },
+    "Sanitization": {
+      "supports": [
+        { "kind": "source", "path": "src/guardrails/sanitize.ts" },
+        { "kind": "source", "path": "src/guardrails/injection.ts" },
+        { "kind": "contract_test", "path": "tests/guardrails/sanitize.test.ts" }
+      ]
+    },
+    "Injection categories (non-exhaustive)": {
+      "supports": [
+        { "kind": "source", "path": "src/guardrails/injection.ts" },
+        { "kind": "contract_test", "path": "tests/guardrails/false-positives.test.ts" }
+      ]
+    },
+    "Sensitive data": {
+      "supports": [
+        { "kind": "source", "path": "src/guardrails/sensitive.ts" },
+        { "kind": "contract_test", "path": "tests/guardrails/sanitize.test.ts" }
+      ]
+    },
+    "Quota": {
+      "supports": [
+        { "kind": "source", "path": "src/guardrails/quota.ts" },
+        { "kind": "contract_test", "path": "tests/guardrails/quota.test.ts" }
+      ]
+    },
+    "Exported API": {
+      "supports": [
+        { "kind": "source", "path": "src/guardrails/mod.ts" },
+        { "kind": "contract_test", "path": "tests/guardrails/error.test.ts" }
+      ]
+    }
+  }
+}
+```

package/docs/host.md

@@ -0,0 +1,97 @@
+# Host (`theorum/host`)
+
+Optional Deno HTTP helpers for host applications. **Not** part of the turn
+kernel — import only when you want shared reply/status glue and cutout-trace
+flushing without reimplementing it per route.
+
+## Export
+
+| Field | Value |
+| --- | --- |
+| Import | `theorum/host` / `jsr:@theorum/core/host` |
+| Module | `src/host/mod.ts` |
+
+## Ownership
+
+| Path | Role |
+| --- | --- |
+| `src/host/reply.ts` | JSON responses + HTTP status constants |
+| `src/host/mint-trace.ts` | Cutout mint trace flush helpers |
+| `src/host/mod.ts` | Public barrel |
+
+## HTTP replies
+
+| Export | Role |
+| --- | --- |
+| `json(status, body, cors)` | JSON `Response` with merged CORS headers |
+| `caughtStatus(err)` | `400` for `TheorumError`, else `500` |
+| `HTTP_OK` | `200` |
+| `HTTP_BUSY` | `429` |
+| `HTTP_NOT_FOUND` | `404` |
+| `HTTP_METHOD` | `405` |
+
+Example:
+
+```ts
+import { caughtStatus, HTTP_BUSY, json } from "theorum/host";
+
+try {
+  return json(200, { ok: true }, cors);
+} catch (err) {
+  return json(caughtStatus(err), { error: publicError(err) }, cors);
+}
+```
+
+Quota busy responses typically use `HTTP_BUSY` after `takeSlot` returns `busy`.
+
+## Cutout mint trace
+
+| Export | Role |
+| --- | --- |
+| `flushMintTrace` | Flush pending cutout mint records after a turn |
+| `CutoutTape` | Tape type for mint/cutout correlation |
+
+Use when your Deno HTTP host records mint/cutout telemetry alongside THEORUM
+turns. Skip entirely for non-HTTP or non-Deno hosts.
+
+## Exported API
+
+Live list: `src/host/mod.ts` (`json`, status constants, `caughtStatus`,
+`flushMintTrace`, `CutoutTape`).
+
+```theorum-evidence
+{
+  "sections": {
+    "Export": {
+      "supports": [
+        { "kind": "source", "path": "src/host/mod.ts" },
+        { "kind": "config", "path": "package.json" }
+      ]
+    },
+    "Ownership": {
+      "supports": [
+        { "kind": "source", "path": "src/host/mod.ts" },
+        { "kind": "graph", "path": "docs/_map.mjs" }
+      ]
+    },
+    "HTTP replies": {
+      "supports": [
+        { "kind": "source", "path": "src/host/reply.ts" },
+        { "kind": "contract_test", "path": "tests/host/host.test.ts" }
+      ]
+    },
+    "Cutout mint trace": {
+      "supports": [
+        { "kind": "source", "path": "src/host/mint-trace.ts" },
+        { "kind": "contract_test", "path": "tests/host/host.test.ts" }
+      ]
+    },
+    "Exported API": {
+      "supports": [
+        { "kind": "source", "path": "src/host/mod.ts" },
+        { "kind": "contract_test", "path": "tests/host/host.test.ts" }
+      ]
+    }
+  }
+}
+```