audrey 0.23.1 → 1.0.0

package/CHANGELOG.md

@@ -1,24 +1,86 @@
 # Changelog
 
-## 0.23.1 - 2026-05-08
-
-### Added - Audrey Guard chassis
-
-- Added `MemoryController` as the first orchestration layer for memory-before-action workflows. `beforeAction()` returns `allow` / `warn` / `block` with evidence, reflexes, recommendations, and an optional capsule; `afterAction()` records redacted tool outcomes and turns failures into tool-result memories.
-- Added `audrey guard --tool <Tool> "<action>"` with `--json`, `--explain`, `--override`, and `--fail-on-warn`.
-- Added `audrey demo --scenario repeated-failure`, a deterministic no-network demo where Audrey records a failed deploy, blocks the repeat attempt, validates the lesson, and prints impact.
-- `Audrey.encodeBatch()` now uses provider-level `embedBatch()` and validates the batch before embedding, avoiding N sequential cloud embedding calls for valid batches.
-- Recall now surfaces partial vector/FTS failures on the returned result array. Capsules preserve those diagnostics, strict Guard preflights block when recall is degraded, and `/v1/status` / `memory_status` expose the latest recall degradation signal.
-- Added `docs/AUDREY_PAPER_OUTLINE.md`, framing Audrey Guard as local-first pre-action memory control for tool-using agents and outlining the GuardBench evaluation plan.
-
-### Fixed
-
-- Docker Compose now requires `AUDREY_API_KEY` instead of starting a non-loopback unauthenticated REST sidecar that the server correctly refuses.
-- Guard exact-failure matching now redacts before trimming, matches tool names case-insensitively, and includes file scope in the action hash.
-- Redaction-aware truncation keeps complete `[REDACTED:*]` markers in long tool errors and output summaries.
-- `npm test` and `npm run test:watch` now set a repo-local Vitest temp directory before Vitest starts, avoiding locked-down Windows user-temp failures.
-- `npm audit --omit=dev --audit-level=moderate` is clean after refreshing Hono, Zod, and transitive rate-limit packages.
-- README benchmark sample values now match `benchmarks/snapshots/perf-0.22.2.json`; the paper evidence ledger was re-checked for the repeated-failure demo line range and live bibliography URLs before release prep.
+## 1.0.0 - 2026-05-13
+
+### Audrey Guard
+
+- Ships Audrey Guard as the release-defining loop: receipt-backed `go`,
+  `caution`, and `block` decisions before tool use, followed by auditable
+  outcome capture through CLI, REST, MCP, and SDK surfaces.
+- Adds Claude Code hook generation and an idempotent hook-apply path so
+  `guard --hook --fail-on-warn` can run at `PreToolUse` and post-tool events
+  can feed Audrey's redacted trace memory.
+- Binds validation feedback to preflight event ids, evidence ids, and action
+  fingerprints so remembered guidance can be audited after use.
+
+### GuardBench And Paper Artifacts
+
+- Ships GuardBench, a local comparative benchmark for pre-action memory control
+  across Audrey Guard, no-memory, recent-window, vector-only, and FTS-only
+  baselines.
+- Adds portable GuardBench bundles, conformance cards, JSON schemas, adapter
+  self-tests, leaderboard generation, external adapter dry-runs, and pending
+  external evidence reports for Mem0 Platform and Zep Cloud.
+- Ships the Audrey Guard paper source, claim register, publication-pack
+  verifier, browser launch plan/results ledger, deterministic arXiv source
+  package, local arXiv compile proof, and paper submission bundle.
+
+### Release Controls
+
+- Adds pending-aware `release:readiness` and strict `release:readiness:strict`
+  gates so code, paper, source control, npm, PyPI, browser publication, and
+  external-evidence blockers stay separate.
+- Adds `release:cut:plan` and `release:cut:apply` so npm, lockfile, MCP,
+  Python, and changelog version surfaces are cut consistently.
+- Adds production dependency audit coverage to release gates and keeps
+  `npm audit --omit=dev --audit-level=moderate` clean.
+
+### Runtime And Client Hardening
+
+- `Audrey.encodeBatch()` now calls provider-level `embedBatch()` once per batch
+  and writes each episode through the existing `encodeEpisode()` path with the
+  precomputed vector.
+- OpenAI embedding batches are chunked by `batchSize` so large batch encodes do
+  not turn into one oversized API request.
+- Improves recall degradation reporting across capsules, strict preflights,
+  status surfaces, and Guard decisions.
+
+## 0.23.0 - 2026-05-05
+
+### Audrey Guard — memory before action becomes the product loop
+
+- Added Audrey Guard as a first-class controller loop: `beforeAction()` checks memory before an agent touches tools, returns a receipt-backed `go` / `caution` / `block` decision, and `afterAction()` records what happened afterward.
+- Added JavaScript SDK exports and `Audrey.beforeAction()` / `Audrey.afterAction()` methods so agent runtimes can use the same loop without going through CLI or REST.
+- Added `POST /v1/guard/before` and `POST /v1/guard/after` REST routes for sidecar agents.
+- Added `memory_guard_before` and `memory_guard_after` MCP tools for hosts that want memory decisions at the tool boundary.
+- Added `npx audrey guard` and `npx audrey guard-after` CLI commands, including JSON output for hooks and automation.
+
+### Release-defining behavior
+
+- Guard decisions reuse the existing preflight and reflex machinery without doing two independent recall passes.
+- Guard receipts are stored as `memory_events` rows with guard metadata, evidence ids, reflex ids, preflight decision, warning counts, and redacted tool-trace linkage.
+- `guard-after` now validates evidence feedback before mutating memory, rejects non-guard receipts, and prevents replaying the same receipt to apply duplicate feedback.
+- A failed guarded tool run becomes future memory: the next guard check for the same tool can produce a recent-failure warning and reflex before the agent repeats the mistake.
+- Strict guard mode can block high-severity must-follow memories before risky actions, which is the release's headline "memory firewall" behavior.
+
+### Benchmarks
+
+- Added an Agent Guard Loop benchmark suite covering prior tool-failure caution, strict must-follow blocking, receipt replay rejection, and non-guard receipt rejection.
+- Added `npm run bench:memory:guard` for focused guard-loop regression testing.
+- Kept guard-loop cases out of the comparable retrieval/lifecycle aggregate when all suites are run, so the local baseline chart remains honest rather than inflated by no-controller placeholders.
+- Committed a fresh `benchmarks/snapshots/perf-0.23.0.json` performance snapshot and fixed direct snapshot runs so they resolve Audrey's package version without depending on npm-injected environment.
+- Added a CLI smoke script to the release gate and Node CI jobs so `--version`, `doctor --json`, and `demo` are proven before pack dry-run.
+- Included benchmark harness files and snapshots in the npm package so advertised benchmark scripts work from the published tarball.
+- Added a package-lock consistency test so release versions cannot drift between `package.json` and `package-lock.json` again.
+
+### Docs and release posture
+
+- Updated README quick-start, surface tables, and benchmark notes around Audrey Guard.
+- Added `docs/MEMORY_BENCHMARKING.md` to state the release's benchmark policy and map Audrey against LongMemEval, LoCoMo, MemoryAgentBench, StructMemEval, and MemGUI-Bench.
+- Added release design and implementation docs under `docs/superpowers/`.
+- Updated the production backlog to mark the v0.23 controller slice as shipped and to focus the next work on hook installation, external benchmark evidence, batching, and partial recall diagnostics.
+- Bumped JavaScript, MCP CLI, and Python client version surfaces to `0.23.0`.
+- Added the Python 3.9 `eval-type-backport` dependency marker required by Pydantic for Audrey's modern type annotations, and moved Python package metadata to the current setuptools license form.
 
 ## 0.22.2 - 2026-05-01
 

package/LICENSE

@@ -1,21 +1,21 @@
-MIT License
-
-Copyright (c) 2026 evilander
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+MIT License
+
+Copyright (c) 2026 evilander
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -27,7 +27,7 @@ Audrey turns those hard-won lessons into a local memory runtime:
 - `memory_recall` finds durable context by semantic similarity.
 - `memory_preflight` checks prior failures, risks, rules, and relevant procedures before an action.
 - `memory_reflexes` converts remembered evidence into trigger-response guidance agents can follow.
-- `memory_validate` closes the loop after the action — `helpful`, `used`, or `wrong` outcomes feed salience and decay.
+- `memory_validate` closes the loop after the action: `helpful`, `used`, or `wrong` outcomes feed salience and can bind back to the exact preflight event, evidence ids, and Guard action fingerprint.
 - `memory_dream` consolidates episodes into principles and applies decay.
 - `audrey impact` and `audrey doctor` tell a human or CI system whether the runtime is doing real work and is actually ready.
 
@@ -52,7 +52,7 @@ npx audrey guard --tool Bash "npm run deploy"
 Expected first-run shape:
 
 ```text
-Audrey Doctor v0.23.1
+Audrey Doctor v1.0.0
 Store health: not initialized
 Verdict: ready
 ```
@@ -75,6 +75,7 @@ Generate raw config blocks:
 npx audrey mcp-config codex
 npx audrey mcp-config generic
 npx audrey mcp-config vscode
+npx audrey hook-config claude-code
 ```
 
 Claude Code can be registered directly:
@@ -84,6 +85,15 @@ npx audrey install
 claude mcp list
 ```
 
+For memory-before-action hooks, preview with `npx audrey hook-config
+claude-code`, then apply with `npx audrey hook-config claude-code --apply
+--scope project` for `.claude/settings.local.json` or `--scope user` for
+`~/.claude/settings.json`. Audrey merges the hook block into existing settings
+and writes a timestamped backup before changing a non-empty file. The generated
+`PreToolUse` hook runs `audrey guard --hook --fail-on-warn`; the `PostToolUse`
+and `PostToolUseFailure` hooks record redacted tool traces. Verify the active
+hook set inside Claude Code with `/hooks`.
+
 All local MCP paths default to local embeddings and one shared SQLite-backed memory directory. Use `AUDREY_DATA_DIR` to isolate projects, tenants, or host identities.
 
 Installer-generated host config does not include provider API keys by default. Prefer setting `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GOOGLE_API_KEY`, or `GEMINI_API_KEY` in the host runtime environment; use `npx audrey install --include-secrets` only if you explicitly accept argv/config exposure.
@@ -121,7 +131,7 @@ Core sidecar tools:
 | Surface | Status |
 |---|---|
 | MCP stdio server | 20 tools plus status/recent/principles resources and briefing/recall/reflection prompts |
-| CLI | `doctor`, `demo`, `guard`, `install`, `mcp-config`, `status`, `dream`, `reembed`, `observe-tool`, `promote`, `impact` |
+| CLI | `doctor`, `demo`, `guard`, `install`, `mcp-config`, `hook-config`, `status`, `dream`, `reembed`, `observe-tool`, `promote`, `impact` |
 | REST API | Hono server with `/health` and `/v1/*` routes |
 | JavaScript SDK | Direct TypeScript/Node import from `audrey` |
 | Python client | `pip install audrey-memory`, calls the REST sidecar |
@@ -191,6 +201,9 @@ Release gates used for this package:
 
 ```bash
 npm run release:gate
+npm run python:release:check
+npm run bench:guard:card
+npm run bench:guard:validate
 npx audrey doctor
 npx audrey demo
 ```
@@ -237,7 +250,7 @@ Production controls you still own:
 
 ## Benchmarks
 
-Audrey ships two benchmark commands.
+Audrey ships three benchmark families.
 
 ### Performance snapshot
 
@@ -268,6 +281,196 @@ npm run bench:memory          # full regression suite (writes JSON + report)
 npm run bench:memory:check    # release gate, exits non-zero on regression
 ```
 
+### GuardBench comparative suite
+
+`npm run bench:guard:check` runs Audrey's local GuardBench comparative suite:
+ten pre-action scenarios across Audrey Guard, no-memory, recent-window,
+vector-only, and FTS-only adapters. The scenarios cover exact repeated
+failures, required procedures, changed file scopes, changed commands,
+recovered failures, recall degradation, redaction safety, conflicting
+instructions, and noisy stores. It writes
+`benchmarks/output/guardbench-summary.json`,
+`benchmarks/output/guardbench-manifest.json`, and
+`benchmarks/output/guardbench-raw.json`. The emitted manifest, summary, and raw
+output shapes are validated by JSON schemas under `benchmarks/schemas/`.
+
+Latest local result in this checkout: 10/10 scenarios passed, 100% prevention
+rate, 0% false-block rate, 0 raw secret leaks, 0 published artifact leaks in
+the raw-secret sweep, and 3.214ms / 21.395ms
+p50/p95 guard latency under the mock-provider methodology. Local baseline
+decision accuracy was: no-memory 10%, recent-window 60%, vector-only 40%, and
+FTS-only 10%; none passed the full GuardBench decision-plus-evidence contract.
+
+```bash
+npm run bench:guard
+npm run bench:guard:check
+npm run bench:guard:manifest
+npm run bench:guard:validate
+npm run bench:guard:card
+npm run bench:guard:bundle
+npm run bench:guard:bundle:verify
+npm run bench:guard:leaderboard
+npm run bench:guard:adapter-registry:validate
+npm run bench:guard:adapter-module:validate
+npm run bench:guard:adapter-self-test
+npm run bench:guard:adapter-self-test:validate
+npm run bench:guard:publication:verify
+npm run bench:guard:adapter-smoke
+npm run bench:guard:adapter-conformance
+npm run bench:guard:external:dry-run
+npm run bench:guard:mem0 -- --dry-run
+npm run bench:guard:zep -- --dry-run
+node benchmarks/adapter-self-test.mjs --adapter ./path/to/adapter.mjs
+node benchmarks/guardbench.js --adapter ./path/to/adapter.mjs --check
+```
+
+External GuardBench adapters are ESM modules that export either `default`,
+`adapter`, or `createGuardBenchAdapter()`. The adapter receives scenario seed
+data and the proposed action, but the harness withholds `expectedDecision` and
+`requiredEvidence` until scoring. Start from
+`benchmarks/adapters/example-allow.mjs` when wiring a new system. Adapter
+authors can import `defineGuardBenchAdapter()` and `defineGuardBenchResult()`
+from `benchmarks/adapter-kit.mjs` to validate module shape and decision output
+while developing.
+
+The published adapter registry lives at `benchmarks/adapters/registry.json`.
+Run `npm run bench:guard:adapter-registry:validate` to verify registry shape,
+adapter paths, and credential-free module loading.
+
+Before running the full self-test, validate the ESM module shape quickly:
+
+```bash
+npm run bench:guard:adapter-module:validate -- --adapter ./path/to/adapter.mjs
+```
+
+Before publishing a new adapter, run `npm run bench:guard:adapter-self-test --
+--adapter ./path/to/adapter.mjs`. The self-test validates the external adapter
+contract and row conformance while explicitly allowing low benchmark scores, so
+authors can separate "valid submission shape" from "competitive GuardBench
+performance." The generated self-test report is validated against
+`benchmarks/schemas/guardbench-adapter-self-test.schema.json`. Reviewers can
+validate a submitted report without rerunning an adapter through `npm run
+bench:guard:adapter-self-test:validate -- --report ./guardbench-adapter-self-test.json`.
+
+Audrey ships external adapters for Mem0 Platform and Zep Cloud. Run them only
+with runtime API keys:
+
+```bash
+set MEM0_API_KEY=...
+npm run bench:guard:mem0
+
+set ZEP_API_KEY=...
+npm run bench:guard:zep
+```
+
+The Zep adapter uses the current REST surface for users, sessions, `memory.add`,
+`graph.search`, and benchmark-user cleanup. If Zep graph ingestion needs more
+time in a live account, set `ZEP_GUARDBENCH_INGEST_DELAY_MS` before the run.
+
+Run `npm run bench:guard:external:dry-run` before coordinating credentialed
+runs. It walks the runtime-env adapter registry, writes non-secret
+`external-run-metadata.json` files for each adapter, and reports which runtime
+environment variables are still missing. The external dry-run matrix report is schema-bound by
+`benchmarks/schemas/guardbench-external-dry-run.schema.json` and written to
+`benchmarks/output/external/guardbench-external-dry-run.json`.
+
+Run `npm run bench:guard:external:evidence` after dry-runs or live runs to
+write `benchmarks/output/external/guardbench-external-evidence.json`. This
+external evidence verification report is schema-bound by
+`benchmarks/schemas/guardbench-external-evidence.schema.json`, treats dry-run
+or missing-key rows as pending in normal release gates, and checks that saved
+metadata does not contain runtime credential values. Use
+`npm run bench:guard:external:evidence:strict` when Mem0/Zep keys have been
+provided; strict mode fails until every runtime-env adapter has a passed live
+bundle.
+
+External runs write `external-run-metadata.json` alongside the GuardBench
+summary, manifest, and raw output bundle under
+`benchmarks/output/external/<adapter>/`. The external runner validates the
+emitted bundle with `benchmarks/validate-guardbench-artifacts.mjs` before
+marking the run passed, and separately records adapter conformance so a valid
+low-scoring adapter is distinguished from a malformed adapter. When
+`external-run-metadata.json` is present, the validator also checks it against
+`benchmarks/schemas/guardbench-external-run.schema.json` and verifies any
+recorded SHA-256 artifact hashes against the bundle on disk.
+
+For a shareable submission artifact, run `npm run bench:guard:card -- --dir
+<output-dir>`. This writes `guardbench-conformance-card.json` with the subject
+name, run status, score, conformance result, artifact hashes, optional
+external-run metadata hash, and machine provenance. The standalone validator
+checks the card when it is present.
+
+For a portable submission directory, run `npm run bench:guard:bundle -- --dir
+<output-dir>`. This creates `submission-bundle/` with the raw GuardBench
+artifacts, conformance card, JSON schemas, validation report, and
+`submission-manifest.json` with SHA-256 hashes for every bundled file.
+Reviewers can run `npm run bench:guard:bundle:verify -- --dir
+<submission-bundle>` to check manifest hashes, bundled schemas, and artifact
+validation from the bundle alone.
+
+For benchmark aggregation, run `npm run bench:guard:leaderboard -- --bundle
+<submission-bundle>`. The leaderboard builder verifies each bundle before
+ranking and writes JSON plus Markdown reports under `benchmarks/output/leaderboard/`.
+
+Before publishing benchmark artifacts, run `npm run
+bench:guard:publication:verify`. This single benchmark-focused verifier checks
+the adapter registry, default adapter module, adapter self-test report,
+GuardBench manifest/summary/raw artifacts, submission bundle, external dry-run
+matrix, external evidence verification report, leaderboard, and a local
+absolute-path sweep over the public artifact set.
+The verifier validates its own machine-readable report against
+`benchmarks/schemas/guardbench-publication-verification.schema.json` before it
+exits.
+
+Before turning the paper into public posts or submissions, run `npm run
+paper:claims`. It validates `docs/paper/claim-register.json` against the
+current paper, README, GuardBench artifacts, publication verifier, and external
+evidence status so pending Mem0/Zep live-score claims cannot slip into public
+copy.
+Run `npm run paper:publication-pack` to verify the ready-to-use arXiv, Hacker
+News, Reddit, X, and LinkedIn drafts in `docs/paper/publication-pack.json`
+before browser-based submission. The X URL reserve is explicit: the first X
+post carries `reservedUrlChars: 24`, and submitted artifact-url targets in
+`browser-launch-results.json` must record the final `artifactUrl`.
+Run `npm run paper:arxiv` to generate a deterministic TeX source package under
+`docs/paper/output/arxiv/`, and `npm run paper:arxiv:verify` to check hashes,
+citation conversion, bibliography coverage, seeded-secret redaction, and local
+absolute-path leakage before arXiv upload.
+Run `npm run paper:arxiv:compile` to record a schema-bound compile report at
+`docs/paper/output/arxiv-compile-report.json`. It attempts `tectonic`,
+`latexmk`, `pdflatex`/`bibtex`, or `uvx tecto` with a local bundle proxy when
+available; `npm run paper:arxiv:compile:strict` stays blocked on hosts without
+supported TeX tooling.
+Run `npm run paper:launch-plan` to verify
+`docs/paper/browser-launch-plan.json`, which maps those drafts to manual
+browser targets, login/captcha expectations, platform-rule checks, source
+URLs, and post-submit URL capture.
+Run `npm run paper:launch-results` to validate
+`docs/paper/browser-launch-results.json`, the post-submit ledger for arXiv,
+Hacker News, Reddit, X, and LinkedIn targets. The normal verifier allows
+pending rows with explicit blockers; `npm run paper:launch-results:strict`
+fails until every target has a submitted, operator-verified public URL.
+Run `npm run paper:bundle` to generate
+`docs/paper/output/submission-bundle/`, a hash-manifested package containing
+paper sources, claim and publication registers, GuardBench outputs, schemas,
+and package metadata. `npm run paper:bundle:verify` checks the manifest and
+file hashes before browser upload.
+Run `npm run release:readiness` for the pending-aware Audrey 1.0 checklist.
+It keeps code/paper readiness separate from publish blockers; `npm run
+release:readiness:strict` fails until the 1.0 version surfaces,
+source-control state, live remote-head verification, Python artifacts, npm
+registry/auth readiness, PyPI publish readiness, arXiv compile proof, browser
+publication URLs, and live Mem0/Zep evidence are complete.
+Run `npm run release:cut:plan` to preview the exact 1.0 version/changelog
+edits across npm, lockfile, MCP, and Python surfaces. `npm run
+release:cut:apply -- --target-version 1.0.0` writes those edits only when the
+final cut is intentional. The generated changelog section is release-note copy,
+not a TODO scaffold; `release:readiness:strict` rejects placeholder changelog
+markers before publication.
+Run `npm run security:audit` before packaging or publishing; the release gates
+call it after artifact verification so production dependency advisories cannot
+slip past the final package check.
+
 ## Command Reference
 
 ```bash
@@ -279,6 +482,7 @@ npx audrey demo
 npx audrey install --host codex --dry-run
 npx audrey mcp-config codex
 npx audrey mcp-config generic
+npx audrey hook-config claude-code
 npx audrey install
 npx audrey uninstall
 
@@ -317,7 +521,7 @@ The Node sidecar defaults to `127.0.0.1:7437`. The Docker image intentionally bi
 npm ci
 npm run release:gate
 python -m unittest discover -s python/tests -v
-python -m build --no-isolation python
+npm run python:release:check
 ```
 
 `npm test` uses a repo-local Vitest launcher so locked-down Windows temp

package/SECURITY.md

@@ -6,7 +6,8 @@ Security fixes are best-effort for the current published release line and the cu
 
 | Version | Supported |
 |---|---|
-| `0.22.x` | Yes |
+| `0.23.x` | Yes |
+| `0.22.x` | Best effort |
 | `< 0.22.0` | No |
 
 ## Reporting a Vulnerability

package/benchmarks/adapter-kit.mjs

@@ -0,0 +1,20 @@
+import { validateGuardBenchAdapter, validateAdapterResult } from './guardbench.js';
+
+export const GUARDBENCH_ADAPTER_CONTRACT_VERSION = '1.0.0';
+export const GUARDBENCH_DECISIONS = Object.freeze(['allow', 'warn', 'block']);
+export const GUARDBENCH_RESULT_FIELDS = Object.freeze([
+  'decision',
+  'riskScore',
+  'evidenceIds',
+  'recommendedActions',
+  'summary',
+  'recallErrors',
+]);
+
+export function defineGuardBenchAdapter(adapter) {
+  return validateGuardBenchAdapter(adapter, adapter?.name ?? 'inline adapter');
+}
+
+export function defineGuardBenchResult(result, adapterName = 'adapter', scenarioId = 'scenario') {
+  return validateAdapterResult(result, adapterName, scenarioId);
+}

package/benchmarks/adapter-self-test.mjs

@@ -0,0 +1,166 @@
+import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs';
+import { basename, dirname, resolve } from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { loadExternalAdapters, runGuardBench } from './guardbench.js';
+import { evaluateAdapterConformance } from './run-external-guardbench.mjs';
+import { validateSchema } from './validate-guardbench-artifacts.mjs';
+import { publicPath } from './public-paths.mjs';
+
+const ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..');
+const DEFAULT_ADAPTER = 'benchmarks/adapters/example-allow.mjs';
+const DEFAULT_OUT = 'benchmarks/output/adapter-self-test/guardbench-adapter-self-test.json';
+const DEFAULT_SCHEMA = 'benchmarks/schemas/guardbench-adapter-self-test.schema.json';
+const RESULT_FIELDS = [
+  'decision',
+  'riskScore',
+  'evidenceIds',
+  'recommendedActions',
+  'summary',
+  'recallErrors',
+];
+
+export function parseAdapterSelfTestArgs(argv = process.argv.slice(2)) {
+  const args = {
+    adapter: DEFAULT_ADAPTER,
+    out: DEFAULT_OUT,
+    json: false,
+    noWrite: false,
+  };
+
+  for (let i = 0; i < argv.length; i++) {
+    const token = argv[i];
+    if (token === '--adapter' && argv[i + 1]) args.adapter = argv[++i];
+    else if (token === '--out' && argv[i + 1]) args.out = argv[++i];
+    else if (token === '--json') args.json = true;
+    else if (token === '--no-write') args.noWrite = true;
+    else if (token === '--help' || token === '-h') args.help = true;
+    else throw new Error(`Unknown argument: ${token}`);
+  }
+
+  return args;
+}
+
+function usage() {
+  return `Usage: node benchmarks/adapter-self-test.mjs [options]
+
+Options:
+  --adapter <path>   ESM GuardBench adapter path. Default: ${DEFAULT_ADAPTER}.
+  --out <path>       JSON report path. Default: ${DEFAULT_OUT}.
+  --json             Print the full JSON report.
+  --no-write         Do not write the JSON report.
+`;
+}
+
+function systemSummary(report, adapterName) {
+  return report.systemSummaries.find(row => row.system === adapterName) ?? null;
+}
+
+function scoreFromReport(report, adapterName) {
+  const summary = systemSummary(report, adapterName);
+  return {
+    scenarios: summary?.scenarios ?? 0,
+    fullContractPassRate: summary?.passRate ?? null,
+    decisionAccuracy: summary?.decisionAccuracy ?? null,
+    evidenceRecall: summary?.evidenceRecall ?? null,
+    redactionLeaks: summary?.redactionLeaks ?? null,
+    latency: summary?.latency ?? null,
+  };
+}
+
+function readJson(path) {
+  return JSON.parse(readFileSync(path, 'utf-8'));
+}
+
+export function validateAdapterSelfTestReport(report, options = {}) {
+  const schemaPath = resolve(ROOT, options.schema ?? DEFAULT_SCHEMA);
+  const schema = options.schemaObject ?? readJson(schemaPath);
+  return validateSchema(report, schema, 'guardbench-adapter-self-test');
+}
+
+export async function runGuardBenchAdapterSelfTest(options = {}) {
+  const adapterPath = resolve(ROOT, options.adapterPath ?? options.adapter ?? DEFAULT_ADAPTER);
+  if (!existsSync(adapterPath)) {
+    throw new Error(`GuardBench adapter not found: ${adapterPath}`);
+  }
+
+  const adapters = await loadExternalAdapters([adapterPath]);
+  if (adapters.length !== 1) {
+    throw new Error(`GuardBench adapter self-test expected 1 adapter, got ${adapters.length}`);
+  }
+
+  const [adapter] = adapters;
+  const report = await runGuardBench({ externalAdapters: adapters });
+  const conformance = evaluateAdapterConformance(report, adapter.name);
+  const score = scoreFromReport(report, conformance.adapter);
+  const selfTest = {
+    schemaVersion: '1.0.0',
+    suite: 'GuardBench adapter self-test',
+    generatedAt: new Date().toISOString(),
+    ok: conformance.ok,
+    adapter: {
+      name: adapter.name,
+      path: publicPath(adapterPath),
+      moduleFile: basename(adapterPath),
+      description: adapter.description ?? null,
+    },
+    conformance,
+    score,
+    contract: {
+      expectedAnswersWithheld: true,
+      lowScoreAllowed: true,
+      requiredScenarioRows: report.scenarios,
+      requiredResultFields: RESULT_FIELDS,
+      redactionLeakTolerance: 0,
+    },
+    failures: conformance.failures,
+  };
+  const schemaErrors = validateAdapterSelfTestReport(selfTest);
+  if (schemaErrors.length > 0) {
+    throw new Error(`GuardBench adapter self-test schema validation failed: ${schemaErrors.join('; ')}`);
+  }
+
+  if (options.out && options.write !== false) {
+    const outPath = resolve(ROOT, options.out);
+    mkdirSync(dirname(outPath), { recursive: true });
+    writeFileSync(outPath, `${JSON.stringify(selfTest, null, 2)}\n`, 'utf-8');
+    selfTest.outPath = publicPath(outPath);
+  }
+
+  return selfTest;
+}
+
+async function main() {
+  const args = parseAdapterSelfTestArgs();
+  if (args.help) {
+    console.log(usage());
+    return;
+  }
+
+  const result = await runGuardBenchAdapterSelfTest({
+    adapter: args.adapter,
+    out: args.noWrite ? null : args.out,
+    write: !args.noWrite,
+  });
+
+  if (args.json) {
+    console.log(JSON.stringify(result, null, 2));
+  } else if (result.ok) {
+    console.log(`GuardBench adapter self-test passed: ${result.adapter.name}`);
+    console.log(`Contract rows: ${result.conformance.scenarios}/${result.conformance.expectedScenarios}`);
+    console.log(`Full-contract score: ${(result.score.fullContractPassRate * 100).toFixed(1)}%`);
+    console.log(`Decision accuracy: ${(result.score.decisionAccuracy * 100).toFixed(1)}%`);
+    if (result.outPath) console.log(`Self-test report: ${result.outPath}`);
+  } else {
+    console.error(`GuardBench adapter self-test failed: ${result.adapter.name}`);
+    for (const failure of result.failures) console.error(`- ${failure}`);
+  }
+
+  process.exitCode = result.ok ? 0 : 1;
+}
+
+if (process.argv[1] && resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
+  main().catch(error => {
+    console.error(error.message);
+    process.exit(1);
+  });
+}

package/benchmarks/adapters/example-allow.mjs

@@ -0,0 +1,28 @@
+import { defineGuardBenchAdapter } from '../adapter-kit.mjs';
+
+export default defineGuardBenchAdapter({
+  name: 'Example Allow Adapter',
+  description: 'Credential-free GuardBench adapter example. It always allows and is useful for adapter-loading smoke tests.',
+  async setup({ scenario }) {
+    return {
+      memoryCount: (scenario.seed.seededMemories ?? []).length,
+      toolEventCount: (scenario.seed.seededToolEvents ?? []).length,
+      hasFaultInjection: Boolean(scenario.seed.faultInjection),
+    };
+  },
+  async decide({ scenario, state }) {
+    return {
+      decision: 'allow',
+      riskScore: 0,
+      evidenceIds: [],
+      recommendedActions: [],
+      summary: [
+        `Example adapter loaded ${state.memoryCount} seeded memories`,
+        `${state.toolEventCount} seeded tool events`,
+        scenario.seed.seededNoise ? `${scenario.seed.seededNoise.count} noise memories` : 'no noise block',
+        state.hasFaultInjection ? 'fault injection present but unsupported' : 'no fault injection',
+      ].join('; '),
+    };
+  },
+  async cleanup() {},
+});