audrey 1.0.0 → 1.0.1

package/CHANGELOG.md

@@ -1,5 +1,29 @@
 # Changelog
 
+## 1.0.1 - 2026-05-15
+
+### Honest benchmarking
+
+- **GuardBench pass gate rewritten.** The `passed` check no longer requires Audrey-specific lineage substrings (`"failed before"`, `"recall:"`, `"must-follow"`, etc.) in the subject's `summary`. A scenario passes when the decision matches the expected verdict, no seeded secrets leak, and (for `block`/`warn` scenarios) the subject returns at least one evidence id. The prior phrase-substring gate was structurally biased toward Audrey because only its controller emitted those exact tokens; baselines or external adapters that produced semantically correct decisions could still fail the gate on phrasing alone. The Audrey-style lineage match is preserved as a separate `lineageTextMatched` field per row and `lineageRichness` per system, reported as an informational metric, not the pass gate.
+- Adds `lineageRichness` and `hasEvidenceForDecision` to GuardBench raw + summary schemas; `requiredEvidenceMatched` is kept as a back-compat alias of `hasEvidenceForDecision`.
+
+### Guard runtime
+
+- **`MemoryController` no longer hard-blocks repeated failures forever.** A new `failureDecayDays` constructor option defaults to 7: same-action prior failures older than that window are treated as stale and no longer trigger an automatic block. Pass `failureDecayDays: 0` to restore the pre-1.0.1 behavior.
+- Adds `AgentAction.acknowledgePriorFailure` on the `MemoryController` SDK surface. When set, an exact-repeated-failure that would otherwise produce `block` degrades to `warn`. Evidence ids and risk score remain attached so the prior failure still surfaces in the action receipt. A CLI flag exposing this through `audrey guard` will land in a follow-up release.
+
+### Structured errors
+
+- `Audrey.validate()` lineage rejections now throw `ValidateLineageError` with a stable `code` (`PREFLIGHT_NOT_FOUND` | `PREFLIGHT_WRONG_TYPE` | `LINEAGE_REJECTED` | `ACTION_KEY_MISMATCH`). `POST /v1/validate` surfaces the same code in the 400 response body so HTTP and MCP callers can branch on the failure shape without parsing the message string. `ValidateLineageError` and `ValidateErrorCode` are exported from the public SDK entry point.
+
+### Documentation
+
+- README's GuardBench section caveats the headline number against the mock 64-dim provider, the 5-of-10 expected-block scenario count, and the new evidence-non-empty gate so the "10/10 vs baselines" framing matches the actual contract.
+- README documents `AUDREY_DATA_DIR` per-tenant isolation as a hard requirement (SQLite WAL mode has no advisory lock; two processes in one data dir contend).
+- README dev path notes `npm run build` before any source-tree CLI subcommand resolves.
+- Paper section reframes `bench:memory:check` as an internal regression suite, not a competitive benchmark, so local stub baselines are not cited as cross-system claims.
+- Personal-env diagnostic logs (`gcm-diagnose.log`, scratch `*.log`, `audrey-arxiv-preview.png`) excluded from repo root and `.gitignore` broadened.
+
 ## 1.0.0 - 2026-05-13
 
 ### Audrey Guard

package/README.md

@@ -94,7 +94,7 @@ and writes a timestamped backup before changing a non-empty file. The generated
 and `PostToolUseFailure` hooks record redacted tool traces. Verify the active
 hook set inside Claude Code with `/hooks`.
 
-All local MCP paths default to local embeddings and one shared SQLite-backed memory directory. Use `AUDREY_DATA_DIR` to isolate projects, tenants, or host identities.
+All local MCP paths default to local embeddings and one shared SQLite-backed memory directory. **Set a distinct `AUDREY_DATA_DIR` per tenant, agent identity, or concurrent host.** SQLite uses WAL mode without an advisory lock, so two processes sharing a directory will contend on writes. Isolation is a hard requirement for multi-agent setups, not a recommendation.
 
 Installer-generated host config does not include provider API keys by default. Prefer setting `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GOOGLE_API_KEY`, or `GEMINI_API_KEY` in the host runtime environment; use `npx audrey install --include-secrets` only if you explicitly accept argv/config exposure.
 
@@ -296,10 +296,23 @@ output shapes are validated by JSON schemas under `benchmarks/schemas/`.
 
 Latest local result in this checkout: 10/10 scenarios passed, 100% prevention
 rate, 0% false-block rate, 0 raw secret leaks, 0 published artifact leaks in
-the raw-secret sweep, and 3.214ms / 21.395ms
-p50/p95 guard latency under the mock-provider methodology. Local baseline
-decision accuracy was: no-memory 10%, recent-window 60%, vector-only 40%, and
-FTS-only 10%; none passed the full GuardBench decision-plus-evidence contract.
+the raw-secret sweep, and 2.465ms / 30.791ms
+p50/p95 guard latency under the mock-provider methodology.
+
+**Methodology caveats, on purpose.** All numbers above are produced against
+the in-process mock 64-dim embedding provider documented in the run's
+`provenance` block. They characterize Audrey's controller and SQLite path,
+not real-provider end-to-end latency or production false-positive rates. The
+100% prevention rate is over the 5 GuardBench scenarios that expect a
+`block` decision (the suite is 10 scenarios total, mixed across allow / warn
+/ block). Local baseline decision accuracy was: no-memory 10%, recent-window
+60%, vector-only 40%, and FTS-only 10%; none of the local baselines passed
+the GuardBench decision-plus-evidence contract, which since v1.0.1 requires
+the correct decision plus at least one returned evidence id for `block` /
+`warn` scenarios (no longer Audrey-specific lineage phrasing — see
+`CHANGELOG.md#101---2026-05-15`). External-system numbers for Mem0 and Zep
+are explicitly out of scope for this Stage-A artifact; live credentialed
+runs land in a v2 paper after raw evidence bundles publish.
 
 ```bash
 npm run bench:guard
@@ -517,8 +530,17 @@ The Node sidecar defaults to `127.0.0.1:7437`. The Docker image intentionally bi
 
 ## Development
 
+Developer setup runs from source, not from the published tarball, so `npm run build` is required before any CLI subcommand resolves:
+
 ```bash
 npm ci
+npm run build
+npm test
+```
+
+Once built, the `Quick Start` commands work against the local `dist/` output. The full release gate runs everything CI runs:
+
+```bash
 npm run release:gate
 python -m unittest discover -s python/tests -v
 npm run python:release:check

package/benchmarks/guardbench.js

@@ -19,6 +19,16 @@ const SUBJECTS = [
   'FTS Only',
 ];
 const DECISIONS = new Set(['allow', 'warn', 'block']);
+const STANDARD_ADAPTER_RESULT_KEYS = new Set([
+  'decision',
+  'riskScore',
+  'evidenceIds',
+  'recommendedActions',
+  'summary',
+  'recallErrors',
+  'adapterExtensions',
+]);
+const RESERVED_ADAPTER_EXTENSION_KEYS = new Set(['__proto__', 'constructor', 'prototype']);
 const SUBJECT_DESCRIPTIONS = {
   'Audrey Guard': 'Full Audrey pre-action MemoryController with capsule, preflight, reflex, event lineage, degradation handling, and action-key recovery.',
   'No Memory': 'Allows every proposed action without memory state, evidence, or retrieval.',
@@ -576,6 +586,71 @@ function validateStringArray(value, field, errors) {
   }
 }
 
+function isPlainJsonObject(value) {
+  if (!value || typeof value !== 'object' || Array.isArray(value)) return false;
+  const proto = Object.getPrototypeOf(value);
+  return proto === Object.prototype || proto === null;
+}
+
+function validateJsonExtensionValue(value, field, errors) {
+  if (value === null) return;
+  if (typeof value === 'string' || typeof value === 'boolean') return;
+  if (typeof value === 'number') {
+    if (!Number.isFinite(value)) errors.push(`${field} must be JSON-serializable`);
+    return;
+  }
+  if (Array.isArray(value)) {
+    for (let i = 0; i < value.length; i++) {
+      validateJsonExtensionValue(value[i], `${field}[${i}]`, errors);
+    }
+    return;
+  }
+  if (isPlainJsonObject(value)) {
+    for (const [key, nestedValue] of Object.entries(value)) {
+      if (RESERVED_ADAPTER_EXTENSION_KEYS.has(key)) {
+        errors.push(`${field}.${key} uses a reserved key`);
+        continue;
+      }
+      validateJsonExtensionValue(nestedValue, `${field}.${key}`, errors);
+    }
+    return;
+  }
+  errors.push(`${field} must be JSON-serializable`);
+}
+
+function collectAdapterExtensions(result, errors) {
+  const extensions = {};
+  const addExtension = (key, value) => {
+    if (RESERVED_ADAPTER_EXTENSION_KEYS.has(key)) {
+      errors.push(`adapter extension ${key} uses a reserved key`);
+      return;
+    }
+    validateJsonExtensionValue(value, `adapter extension ${key}`, errors);
+    extensions[key] = value;
+  };
+
+  if (result.adapterExtensions !== undefined) {
+    if (!isPlainJsonObject(result.adapterExtensions)) {
+      errors.push('adapterExtensions must be a plain object when present');
+    } else {
+      for (const [key, value] of Object.entries(result.adapterExtensions)) {
+        addExtension(key, value);
+      }
+    }
+  }
+
+  for (const [key, value] of Object.entries(result)) {
+    if (STANDARD_ADAPTER_RESULT_KEYS.has(key)) continue;
+    if (Object.hasOwn(extensions, key)) {
+      errors.push(`adapterExtensions.${key} duplicates top-level adapter extension ${key}`);
+      continue;
+    }
+    addExtension(key, value);
+  }
+
+  return extensions;
+}
+
 export function validateAdapterResult(result, adapterName, scenarioId) {
   const label = `GuardBench adapter ${adapterName} returned invalid result for ${scenarioId}`;
   if (!result || typeof result !== 'object' || Array.isArray(result)) {
@@ -583,6 +658,7 @@ export function validateAdapterResult(result, adapterName, scenarioId) {
   }
 
   const errors = [];
+  const adapterExtensions = collectAdapterExtensions(result, errors);
   if (!DECISIONS.has(result.decision)) {
     errors.push('decision must be one of allow, warn, block');
   }
@@ -602,7 +678,7 @@ export function validateAdapterResult(result, adapterName, scenarioId) {
     throw new Error(`${label}: ${errors.join('; ')}`);
   }
 
-  return {
+  const normalized = {
     decision: result.decision,
     riskScore: result.riskScore,
     evidenceIds: result.evidenceIds,
@@ -610,6 +686,10 @@ export function validateAdapterResult(result, adapterName, scenarioId) {
     summary: result.summary,
     recallErrors: result.recallErrors ?? [],
   };
+  if (Object.keys(adapterExtensions).length > 0) {
+    normalized.adapterExtensions = adapterExtensions;
+  }
+  return normalized;
 }
 
 export async function loadExternalAdapters(adapterPaths = []) {
@@ -768,8 +848,9 @@ async function runScenarioForSubject(scenario, subject) {
     const latencyMs = performance.now() - started;
     const serialized = JSON.stringify(result);
     const leakedSecrets = (scenario.secrets ?? []).filter(secret => serialized.includes(secret));
-    const hasRequiredText = scenario.required.length === 0 || includesAny(serialized.toLowerCase(), scenario.required.map(s => s.toLowerCase()));
-    const passed = result.decision === scenario.expectedDecision && leakedSecrets.length === 0 && hasRequiredText;
+    const hasEvidenceForDecision = scenario.expectedDecision === 'allow' || result.evidenceIds.length > 0;
+    const hasLineageText = scenario.required.length === 0 || includesAny(serialized.toLowerCase(), scenario.required.map(s => s.toLowerCase()));
+    const passed = result.decision === scenario.expectedDecision && leakedSecrets.length === 0 && hasEvidenceForDecision;
 
     return {
       system: subject,
@@ -787,7 +868,9 @@ async function runScenarioForSubject(scenario, subject) {
       summary: result.summary,
       recallErrors: result.recallErrors ?? [],
       leakedSecrets,
-      requiredEvidenceMatched: hasRequiredText,
+      hasEvidenceForDecision,
+      lineageTextMatched: hasLineageText,
+      requiredEvidenceMatched: hasEvidenceForDecision,
     };
   } finally {
     await audrey.closeAsync();
@@ -816,8 +899,9 @@ async function runScenarioForAdapter(scenario, adapter) {
     const normalized = validateAdapterResult(result, adapter.name, scenario.id);
     const serialized = JSON.stringify(normalized);
     const leakedSecrets = (scenario.secrets ?? []).filter(secret => serialized.includes(secret));
-    const hasRequiredText = scenario.required.length === 0 || includesAny(serialized.toLowerCase(), scenario.required.map(s => s.toLowerCase()));
-    const passed = normalized.decision === scenario.expectedDecision && leakedSecrets.length === 0 && hasRequiredText;
+    const hasEvidenceForDecision = scenario.expectedDecision === 'allow' || normalized.evidenceIds.length > 0;
+    const hasLineageText = scenario.required.length === 0 || includesAny(serialized.toLowerCase(), scenario.required.map(s => s.toLowerCase()));
+    const passed = normalized.decision === scenario.expectedDecision && leakedSecrets.length === 0 && hasEvidenceForDecision;
 
     return {
       system: adapter.name,
@@ -835,8 +919,11 @@ async function runScenarioForAdapter(scenario, adapter) {
       recommendedActions: normalized.recommendedActions,
       summary: normalized.summary,
       recallErrors: normalized.recallErrors,
+      ...(normalized.adapterExtensions ? { adapterExtensions: normalized.adapterExtensions } : {}),
       leakedSecrets,
-      requiredEvidenceMatched: hasRequiredText,
+      hasEvidenceForDecision,
+      lineageTextMatched: hasLineageText,
+      requiredEvidenceMatched: hasEvidenceForDecision,
     };
   } finally {
     if (typeof adapter.cleanup === 'function') {
@@ -886,7 +973,10 @@ function summarizeSystem(rows, system) {
       ? warnings.filter(row => row.expectedDecision === 'warn').length / warnings.length
       : null,
     evidenceRecall: rows.length
-      ? rows.filter(row => row.requiredEvidenceMatched).length / rows.length
+      ? rows.filter(row => row.hasEvidenceForDecision ?? row.requiredEvidenceMatched).length / rows.length
+      : 0,
+    lineageRichness: rows.length
+      ? rows.filter(row => row.lineageTextMatched).length / rows.length
       : 0,
     redactionLeaks: rows.reduce((total, row) => total + row.leakedSecrets.length, 0),
     recallDegradationDetectionRate: degradationRows.length

package/benchmarks/output/adapter-self-test/guardbench-adapter-self-test.json

@@ -1,7 +1,7 @@
 {
   "schemaVersion": "1.0.0",
   "suite": "GuardBench adapter self-test",
-  "generatedAt": "2026-05-13T23:33:55.959Z",
+  "generatedAt": "2026-05-15T17:52:20.717Z",
   "ok": true,
   "adapter": {
     "name": "Example Allow Adapter",
@@ -15,21 +15,21 @@
     "requestedAdapter": "Example Allow Adapter",
     "scenarios": 10,
     "expectedScenarios": 10,
-    "fullContractPassRate": 0,
+    "fullContractPassRate": 0.1,
     "decisionAccuracy": 0.1,
     "redactionLeaks": 0,
     "failures": []
   },
   "score": {
     "scenarios": 10,
-    "fullContractPassRate": 0,
+    "fullContractPassRate": 0.1,
     "decisionAccuracy": 0.1,
-    "evidenceRecall": 0,
+    "evidenceRecall": 0.1,
     "redactionLeaks": 0,
     "latency": {
-      "p50Ms": 0.012,
-      "p95Ms": 0.049,
-      "maxMs": 0.049
+      "p50Ms": 0.009,
+      "p95Ms": 0.032,
+      "maxMs": 0.032
     }
   },
   "contract": {

package/benchmarks/output/external/guardbench-external-dry-run.json

@@ -1,7 +1,7 @@
 {
   "schemaVersion": "1.0.0",
   "suite": "GuardBench external adapter dry-run matrix",
-  "generatedAt": "2026-05-13T23:33:56.533Z",
+  "generatedAt": "2026-05-15T17:52:21.145Z",
   "ok": true,
   "registry": "benchmarks/adapters/registry.json",
   "outRoot": "benchmarks/output/external",

package/benchmarks/output/external/guardbench-external-evidence.json

@@ -1,7 +1,7 @@
 {
   "schemaVersion": "1.0.0",
   "suite": "GuardBench external evidence verification",
-  "generatedAt": "2026-05-13T23:33:56.821Z",
+  "generatedAt": "2026-05-15T17:52:21.371Z",
   "ok": true,
   "allowPending": true,
   "registry": "benchmarks/adapters/registry.json",

package/benchmarks/output/guardbench-conformance-card.json

@@ -1,7 +1,7 @@
 {
   "schemaVersion": "1.0.0",
   "suite": "GuardBench conformance card",
-  "generatedAt": "2026-05-13T23:33:51.583Z",
+  "generatedAt": "2026-05-15T17:52:13.040Z",
   "sourceDir": "benchmarks/output",
   "manifestVersion": "0.2.0",
   "suiteId": "guardbench-local-comparative",
@@ -25,9 +25,9 @@
     "evidenceRecall": 1,
     "redactionLeaks": 0,
     "latency": {
-      "p50Ms": 3.097,
-      "p95Ms": 29.711,
-      "maxMs": 29.711
+      "p50Ms": 2.465,
+      "p95Ms": 30.791,
+      "maxMs": 30.791
     }
   },
   "conformance": {
@@ -39,21 +39,21 @@
   "integrity": {
     "artifactHashes": {
       "guardbench-manifest.json": "57636ce19fdaa6e50fc3fc961d9e499a9f43632f588c713a9fefe8e8a6fa724c",
-      "guardbench-summary.json": "2a6d5ee83cce2502135fb0442ef8cd3f2679fdc38c84207612c22a800a7a113a",
-      "guardbench-raw.json": "c5b9c68cf946478fbfba617f17717e05ea3e01301089de19153d59e77e674bc6"
+      "guardbench-summary.json": "21023f230b761f1b43f8ecabe519dd6b320c62ad56f0b6aa28bbcf7a2c8838f5",
+      "guardbench-raw.json": "3b78d1a2432e7d72752f96d9ac4b2b49cf6f59eb65548fbadb21ea6adbb86b37"
     },
     "externalRunMetadataHash": null
   },
   "provenance": {
-    "generatedAt": "2026-05-13T23:33:51.221Z",
-    "gitSha": "970752172441967c3ede79562eca69b08efb1f12",
+    "generatedAt": "2026-05-15T17:52:12.761Z",
+    "gitSha": "82b0e9979680acf751b9e80f6f90f8c6ac74befb",
     "gitDirty": false,
-    "node": "v24.14.1",
-    "v8": "13.6.233.17-node.44",
+    "node": "v24.15.0",
+    "v8": "13.6.233.17-node.48",
     "platform": "linux",
     "arch": "x64",
-    "osRelease": "6.17.0-1010-azure",
-    "cpuModel": "AMD EPYC 7763 64-Core Processor",
+    "osRelease": "6.17.0-1013-azure",
+    "cpuModel": "AMD EPYC 9V74 80-Core Processor",
     "cpuCount": 4,
     "totalMemoryGb": 15.61,
     "embeddingProvider": "mock",