audrey 1.1.3 → 1.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +93 -0
- package/README.md +39 -12
- package/dist/mcp-server/codex-hooks-probe.d.ts +22 -0
- package/dist/mcp-server/codex-hooks-probe.d.ts.map +1 -0
- package/dist/mcp-server/codex-hooks-probe.js +160 -0
- package/dist/mcp-server/codex-hooks-probe.js.map +1 -0
- package/dist/mcp-server/config.d.ts +26 -2
- package/dist/mcp-server/config.d.ts.map +1 -1
- package/dist/mcp-server/config.js +52 -20
- package/dist/mcp-server/config.js.map +1 -1
- package/dist/mcp-server/hooks.d.ts +28 -0
- package/dist/mcp-server/hooks.d.ts.map +1 -1
- package/dist/mcp-server/hooks.js +91 -2
- package/dist/mcp-server/hooks.js.map +1 -1
- package/dist/mcp-server/index.d.ts +242 -2
- package/dist/mcp-server/index.d.ts.map +1 -1
- package/dist/mcp-server/index.js +806 -240
- package/dist/mcp-server/index.js.map +1 -1
- package/dist/mcp-server/tool-schemas.d.ts +32 -2
- package/dist/mcp-server/tool-schemas.d.ts.map +1 -1
- package/dist/mcp-server/tool-schemas.js +55 -0
- package/dist/mcp-server/tool-schemas.js.map +1 -1
- package/dist/src/audrey.d.ts +14 -2
- package/dist/src/audrey.d.ts.map +1 -1
- package/dist/src/audrey.js +152 -28
- package/dist/src/audrey.js.map +1 -1
- package/dist/src/autopilot.d.ts +22 -0
- package/dist/src/autopilot.d.ts.map +1 -1
- package/dist/src/autopilot.js +271 -50
- package/dist/src/autopilot.js.map +1 -1
- package/dist/src/capsule.d.ts +23 -0
- package/dist/src/capsule.d.ts.map +1 -1
- package/dist/src/capsule.js +56 -11
- package/dist/src/capsule.js.map +1 -1
- package/dist/src/confidence.d.ts +9 -1
- package/dist/src/confidence.d.ts.map +1 -1
- package/dist/src/confidence.js +16 -2
- package/dist/src/confidence.js.map +1 -1
- package/dist/src/consolidate.d.ts +4 -0
- package/dist/src/consolidate.d.ts.map +1 -1
- package/dist/src/consolidate.js +206 -34
- package/dist/src/consolidate.js.map +1 -1
- package/dist/src/controller.d.ts +17 -1
- package/dist/src/controller.d.ts.map +1 -1
- package/dist/src/controller.js +244 -128
- package/dist/src/controller.js.map +1 -1
- package/dist/src/db.d.ts.map +1 -1
- package/dist/src/db.js +233 -56
- package/dist/src/db.js.map +1 -1
- package/dist/src/encode.d.ts +2 -1
- package/dist/src/encode.d.ts.map +1 -1
- package/dist/src/encode.js +41 -4
- package/dist/src/encode.js.map +1 -1
- package/dist/src/events.d.ts +48 -1
- package/dist/src/events.d.ts.map +1 -1
- package/dist/src/events.js +160 -62
- package/dist/src/events.js.map +1 -1
- package/dist/src/export.d.ts.map +1 -1
- package/dist/src/export.js +4 -4
- package/dist/src/export.js.map +1 -1
- package/dist/src/grounding.d.ts +111 -0
- package/dist/src/grounding.d.ts.map +1 -0
- package/dist/src/grounding.js +259 -0
- package/dist/src/grounding.js.map +1 -0
- package/dist/src/hybrid-recall.d.ts +16 -7
- package/dist/src/hybrid-recall.d.ts.map +1 -1
- package/dist/src/hybrid-recall.js +22 -11
- package/dist/src/hybrid-recall.js.map +1 -1
- package/dist/src/import.d.ts +10 -0
- package/dist/src/import.d.ts.map +1 -1
- package/dist/src/import.js +122 -22
- package/dist/src/import.js.map +1 -1
- package/dist/src/llm.d.ts.map +1 -1
- package/dist/src/llm.js +16 -0
- package/dist/src/llm.js.map +1 -1
- package/dist/src/preflight.d.ts.map +1 -1
- package/dist/src/preflight.js +89 -47
- package/dist/src/preflight.js.map +1 -1
- package/dist/src/project.d.ts +4 -1
- package/dist/src/project.d.ts.map +1 -1
- package/dist/src/project.js +65 -5
- package/dist/src/project.js.map +1 -1
- package/dist/src/recall.d.ts.map +1 -1
- package/dist/src/recall.js +138 -13
- package/dist/src/recall.js.map +1 -1
- package/dist/src/redact.d.ts +1 -1
- package/dist/src/redact.d.ts.map +1 -1
- package/dist/src/redact.js +185 -5
- package/dist/src/redact.js.map +1 -1
- package/dist/src/routes.d.ts.map +1 -1
- package/dist/src/routes.js +91 -18
- package/dist/src/routes.js.map +1 -1
- package/dist/src/trust.d.ts +35 -0
- package/dist/src/trust.d.ts.map +1 -0
- package/dist/src/trust.js +77 -0
- package/dist/src/trust.js.map +1 -0
- package/dist/src/types.d.ts +24 -0
- package/dist/src/types.d.ts.map +1 -1
- package/package.json +3 -37
- package/benchmarks/adapter-kit.mjs +0 -20
- package/benchmarks/adapter-self-test.mjs +0 -170
- package/benchmarks/adapters/example-allow.mjs +0 -31
- package/benchmarks/adapters/mem0-platform.mjs +0 -274
- package/benchmarks/adapters/registry.json +0 -51
- package/benchmarks/adapters/zep-cloud.mjs +0 -304
- package/benchmarks/baselines.js +0 -174
- package/benchmarks/build-leaderboard.mjs +0 -183
- package/benchmarks/cases.js +0 -549
- package/benchmarks/create-conformance-card.mjs +0 -148
- package/benchmarks/create-submission-bundle.mjs +0 -190
- package/benchmarks/dry-run-external-adapters.mjs +0 -177
- package/benchmarks/guardbench.js +0 -1265
- package/benchmarks/output/adapter-self-test/guardbench-adapter-self-test.json +0 -50
- package/benchmarks/output/external/guardbench-external-dry-run.json +0 -69
- package/benchmarks/output/external/guardbench-external-evidence.json +0 -56
- package/benchmarks/output/guardbench-conformance-card.json +0 -63
- package/benchmarks/output/guardbench-manifest.json +0 -414
- package/benchmarks/output/guardbench-raw.json +0 -1279
- package/benchmarks/output/guardbench-summary.json +0 -2123
- package/benchmarks/output/leaderboard/guardbench-leaderboard.json +0 -93
- package/benchmarks/output/leaderboard/guardbench-leaderboard.md +0 -7
- package/benchmarks/output/submission-bundle/guardbench-conformance-card.json +0 -63
- package/benchmarks/output/submission-bundle/guardbench-manifest.json +0 -414
- package/benchmarks/output/submission-bundle/guardbench-raw.json +0 -1279
- package/benchmarks/output/submission-bundle/guardbench-summary.json +0 -2123
- package/benchmarks/output/submission-bundle/schemas/guardbench-adapter-registry.schema.json +0 -69
- package/benchmarks/output/submission-bundle/schemas/guardbench-adapter-self-test.schema.json +0 -156
- package/benchmarks/output/submission-bundle/schemas/guardbench-conformance-card.schema.json +0 -184
- package/benchmarks/output/submission-bundle/schemas/guardbench-external-dry-run.schema.json +0 -74
- package/benchmarks/output/submission-bundle/schemas/guardbench-external-evidence.schema.json +0 -108
- package/benchmarks/output/submission-bundle/schemas/guardbench-external-run.schema.json +0 -160
- package/benchmarks/output/submission-bundle/schemas/guardbench-leaderboard.schema.json +0 -179
- package/benchmarks/output/submission-bundle/schemas/guardbench-manifest.schema.json +0 -213
- package/benchmarks/output/submission-bundle/schemas/guardbench-publication-verification.schema.json +0 -47
- package/benchmarks/output/submission-bundle/schemas/guardbench-raw.schema.json +0 -184
- package/benchmarks/output/submission-bundle/schemas/guardbench-submission-manifest.schema.json +0 -151
- package/benchmarks/output/submission-bundle/schemas/guardbench-summary.schema.json +0 -249
- package/benchmarks/output/submission-bundle/submission-manifest.json +0 -131
- package/benchmarks/output/submission-bundle/validation-report.json +0 -31
- package/benchmarks/output/summary.json +0 -2354
- package/benchmarks/perf-snapshot.js +0 -307
- package/benchmarks/perf.bench.js +0 -169
- package/benchmarks/public-paths.mjs +0 -97
- package/benchmarks/reference-results.js +0 -75
- package/benchmarks/report.js +0 -280
- package/benchmarks/run-external-guardbench.mjs +0 -303
- package/benchmarks/run.js +0 -735
- package/benchmarks/schemas/guardbench-adapter-registry.schema.json +0 -69
- package/benchmarks/schemas/guardbench-adapter-self-test.schema.json +0 -156
- package/benchmarks/schemas/guardbench-conformance-card.schema.json +0 -184
- package/benchmarks/schemas/guardbench-external-dry-run.schema.json +0 -74
- package/benchmarks/schemas/guardbench-external-evidence.schema.json +0 -108
- package/benchmarks/schemas/guardbench-external-run.schema.json +0 -160
- package/benchmarks/schemas/guardbench-leaderboard.schema.json +0 -179
- package/benchmarks/schemas/guardbench-manifest.schema.json +0 -213
- package/benchmarks/schemas/guardbench-publication-verification.schema.json +0 -47
- package/benchmarks/schemas/guardbench-raw.schema.json +0 -184
- package/benchmarks/schemas/guardbench-submission-manifest.schema.json +0 -151
- package/benchmarks/schemas/guardbench-summary.schema.json +0 -249
- package/benchmarks/snapshots/perf-0.22.2.json +0 -123
- package/benchmarks/snapshots/perf-0.23.0.json +0 -123
- package/benchmarks/validate-adapter-module.mjs +0 -107
- package/benchmarks/validate-adapter-registry.mjs +0 -145
- package/benchmarks/validate-adapter-self-test.mjs +0 -96
- package/benchmarks/validate-guardbench-artifacts.mjs +0 -400
- package/benchmarks/verify-external-evidence.mjs +0 -351
- package/benchmarks/verify-publication-artifacts.mjs +0 -309
- package/benchmarks/verify-submission-bundle.mjs +0 -172
- package/docs/AUDREY_PAPER_OUTLINE.md +0 -178
- package/docs/MEMORY_BENCHMARKING.md +0 -59
- package/docs/PRODUCTION_BACKLOG.md +0 -321
- package/docs/assets/audrey-feature-grid.jpg +0 -0
- package/docs/assets/audrey-logo.svg +0 -45
- package/docs/assets/audrey-wordmark.png +0 -0
- package/docs/paper/00-master.md +0 -48
- package/docs/paper/01-introduction.md +0 -27
- package/docs/paper/02-related-work.md +0 -47
- package/docs/paper/03-problem-definition.md +0 -108
- package/docs/paper/04-design.md +0 -164
- package/docs/paper/05-guardbench-spec.md +0 -412
- package/docs/paper/06-implementation.md +0 -115
- package/docs/paper/07-evaluation.md +0 -168
- package/docs/paper/08-discussion-limitations.md +0 -61
- package/docs/paper/09-conclusion.md +0 -11
- package/docs/paper/SUBMISSION_README.md +0 -162
- package/docs/paper/appendix-a-demo-transcript.md +0 -114
- package/docs/paper/arxiv-compile-report.schema.json +0 -116
- package/docs/paper/arxiv-source.schema.json +0 -61
- package/docs/paper/audrey-paper-v1.md +0 -1108
- package/docs/paper/browser-launch-plan.json +0 -209
- package/docs/paper/browser-launch-plan.schema.json +0 -100
- package/docs/paper/browser-launch-results.json +0 -86
- package/docs/paper/browser-launch-results.schema.json +0 -66
- package/docs/paper/claim-register.json +0 -138
- package/docs/paper/claim-register.schema.json +0 -81
- package/docs/paper/evidence-ledger.md +0 -103
- package/docs/paper/output/arxiv/README-arxiv.txt +0 -8
- package/docs/paper/output/arxiv/arxiv-manifest.json +0 -41
- package/docs/paper/output/arxiv/main.tex +0 -951
- package/docs/paper/output/arxiv/references.bib +0 -222
- package/docs/paper/output/arxiv-compile-report.json +0 -24
- package/docs/paper/output/submission-bundle/LICENSE +0 -21
- package/docs/paper/output/submission-bundle/README.md +0 -440
- package/docs/paper/output/submission-bundle/benchmarks/output/adapter-self-test/guardbench-adapter-self-test.json +0 -50
- package/docs/paper/output/submission-bundle/benchmarks/output/external/guardbench-external-dry-run.json +0 -69
- package/docs/paper/output/submission-bundle/benchmarks/output/external/guardbench-external-evidence.json +0 -56
- package/docs/paper/output/submission-bundle/benchmarks/output/guardbench-conformance-card.json +0 -63
- package/docs/paper/output/submission-bundle/benchmarks/output/guardbench-manifest.json +0 -414
- package/docs/paper/output/submission-bundle/benchmarks/output/guardbench-raw.json +0 -1279
- package/docs/paper/output/submission-bundle/benchmarks/output/guardbench-summary.json +0 -2123
- package/docs/paper/output/submission-bundle/benchmarks/output/leaderboard/guardbench-leaderboard.json +0 -93
- package/docs/paper/output/submission-bundle/benchmarks/output/leaderboard/guardbench-leaderboard.md +0 -7
- package/docs/paper/output/submission-bundle/benchmarks/output/submission-bundle/submission-manifest.json +0 -131
- package/docs/paper/output/submission-bundle/benchmarks/output/submission-bundle/validation-report.json +0 -31
- package/docs/paper/output/submission-bundle/benchmarks/output/summary.json +0 -2354
- package/docs/paper/output/submission-bundle/benchmarks/schemas/guardbench-adapter-registry.schema.json +0 -69
- package/docs/paper/output/submission-bundle/benchmarks/schemas/guardbench-adapter-self-test.schema.json +0 -156
- package/docs/paper/output/submission-bundle/benchmarks/schemas/guardbench-conformance-card.schema.json +0 -184
- package/docs/paper/output/submission-bundle/benchmarks/schemas/guardbench-external-dry-run.schema.json +0 -74
- package/docs/paper/output/submission-bundle/benchmarks/schemas/guardbench-external-evidence.schema.json +0 -108
- package/docs/paper/output/submission-bundle/benchmarks/schemas/guardbench-external-run.schema.json +0 -160
- package/docs/paper/output/submission-bundle/benchmarks/schemas/guardbench-leaderboard.schema.json +0 -179
- package/docs/paper/output/submission-bundle/benchmarks/schemas/guardbench-manifest.schema.json +0 -213
- package/docs/paper/output/submission-bundle/benchmarks/schemas/guardbench-publication-verification.schema.json +0 -47
- package/docs/paper/output/submission-bundle/benchmarks/schemas/guardbench-raw.schema.json +0 -184
- package/docs/paper/output/submission-bundle/benchmarks/schemas/guardbench-submission-manifest.schema.json +0 -151
- package/docs/paper/output/submission-bundle/benchmarks/schemas/guardbench-summary.schema.json +0 -249
- package/docs/paper/output/submission-bundle/docs/AUDREY_PAPER_OUTLINE.md +0 -178
- package/docs/paper/output/submission-bundle/docs/paper/00-master.md +0 -48
- package/docs/paper/output/submission-bundle/docs/paper/01-introduction.md +0 -27
- package/docs/paper/output/submission-bundle/docs/paper/02-related-work.md +0 -47
- package/docs/paper/output/submission-bundle/docs/paper/03-problem-definition.md +0 -108
- package/docs/paper/output/submission-bundle/docs/paper/04-design.md +0 -164
- package/docs/paper/output/submission-bundle/docs/paper/05-guardbench-spec.md +0 -412
- package/docs/paper/output/submission-bundle/docs/paper/06-implementation.md +0 -115
- package/docs/paper/output/submission-bundle/docs/paper/07-evaluation.md +0 -168
- package/docs/paper/output/submission-bundle/docs/paper/08-discussion-limitations.md +0 -61
- package/docs/paper/output/submission-bundle/docs/paper/09-conclusion.md +0 -11
- package/docs/paper/output/submission-bundle/docs/paper/SUBMISSION_README.md +0 -162
- package/docs/paper/output/submission-bundle/docs/paper/appendix-a-demo-transcript.md +0 -114
- package/docs/paper/output/submission-bundle/docs/paper/arxiv-compile-report.schema.json +0 -116
- package/docs/paper/output/submission-bundle/docs/paper/arxiv-source.schema.json +0 -61
- package/docs/paper/output/submission-bundle/docs/paper/audrey-paper-v1.md +0 -1108
- package/docs/paper/output/submission-bundle/docs/paper/browser-launch-plan.json +0 -209
- package/docs/paper/output/submission-bundle/docs/paper/browser-launch-plan.schema.json +0 -100
- package/docs/paper/output/submission-bundle/docs/paper/browser-launch-results.json +0 -86
- package/docs/paper/output/submission-bundle/docs/paper/browser-launch-results.schema.json +0 -66
- package/docs/paper/output/submission-bundle/docs/paper/claim-register.json +0 -138
- package/docs/paper/output/submission-bundle/docs/paper/claim-register.schema.json +0 -81
- package/docs/paper/output/submission-bundle/docs/paper/evidence-ledger.md +0 -103
- package/docs/paper/output/submission-bundle/docs/paper/output/arxiv/README-arxiv.txt +0 -8
- package/docs/paper/output/submission-bundle/docs/paper/output/arxiv/arxiv-manifest.json +0 -41
- package/docs/paper/output/submission-bundle/docs/paper/output/arxiv/main.tex +0 -951
- package/docs/paper/output/submission-bundle/docs/paper/output/arxiv/references.bib +0 -222
- package/docs/paper/output/submission-bundle/docs/paper/output/arxiv-compile-report.json +0 -24
- package/docs/paper/output/submission-bundle/docs/paper/paper-submission-bundle.schema.json +0 -70
- package/docs/paper/output/submission-bundle/docs/paper/publication-pack.json +0 -81
- package/docs/paper/output/submission-bundle/docs/paper/publication-pack.schema.json +0 -60
- package/docs/paper/output/submission-bundle/docs/paper/references.bib +0 -222
- package/docs/paper/output/submission-bundle/package.json +0 -235
- package/docs/paper/output/submission-bundle/paper-submission-manifest.json +0 -379
- package/docs/paper/paper-submission-bundle.schema.json +0 -70
- package/docs/paper/publication-pack.json +0 -81
- package/docs/paper/publication-pack.schema.json +0 -60
- package/docs/paper/references.bib +0 -222
- package/scripts/audit-release-completion.mjs +0 -440
- package/scripts/create-arxiv-source.mjs +0 -368
- package/scripts/create-paper-submission-bundle.mjs +0 -214
- package/scripts/finalize-release.mjs +0 -601
- package/scripts/prepare-release-cut.mjs +0 -277
- package/scripts/publish-release-bundle.mjs +0 -252
- package/scripts/publish-release-github-api.mjs +0 -494
- package/scripts/run-vitest.mjs +0 -34
- package/scripts/smoke-cli.js +0 -92
- package/scripts/sync-paper-artifacts.mjs +0 -112
- package/scripts/verify-arxiv-compile.mjs +0 -476
- package/scripts/verify-arxiv-source.mjs +0 -224
- package/scripts/verify-browser-launch-plan.mjs +0 -254
- package/scripts/verify-browser-launch-results.mjs +0 -303
- package/scripts/verify-paper-artifacts.mjs +0 -798
- package/scripts/verify-paper-claims.mjs +0 -254
- package/scripts/verify-paper-submission-bundle.mjs +0 -218
- package/scripts/verify-publication-pack.mjs +0 -210
- package/scripts/verify-python-package.py +0 -201
- package/scripts/verify-release-readiness.mjs +0 -920
|
@@ -1,115 +0,0 @@
|
|
|
1
|
-
# 6. Implementation
|
|
2
|
-
|
|
3
|
-
Audrey is implemented as a local-first Node and TypeScript runtime with SQLite storage, vector and full-text indexes, MCP stdio tools, a REST sidecar, a Python client, a CLI, and release gates. The implementation is small enough to inspect directly, which is why this paper treats source-linked evidence as part of the artifact.
|
|
4
|
-
|
|
5
|
-
## Runtime Stack
|
|
6
|
-
|
|
7
|
-
The package requires Node 20 or newer and is implemented in TypeScript. The runtime uses `better-sqlite3` for local SQLite storage, `sqlite-vec` for vector tables, Hono for the REST sidecar, the Model Context Protocol SDK for the stdio MCP server, and `@huggingface/transformers` for the local embedding provider (Ledger: E31). Audrey also ships a Python client that calls the REST sidecar through synchronous and asynchronous methods (Ledger: E36).
|
|
8
|
-
|
|
9
|
-
Embedding providers are explicit. The default resolver selects the local provider with 384 dimensions and device `gpu` unless configured otherwise. The local provider uses `Xenova/all-MiniLM-L6-v2`. The mock provider uses 64 dimensions for deterministic tests and benchmarks. OpenAI and Gemini providers are supported but are not auto-selected from ambient API keys; operators must set `AUDREY_EMBEDDING_PROVIDER=openai` or `AUDREY_EMBEDDING_PROVIDER=gemini`. Their default dimensions are 1536 and 3072 respectively (Ledger: E31).
|
|
10
|
-
|
|
11
|
-
## Storage Schema
|
|
12
|
-
|
|
13
|
-
Audrey stores memory in SQLite. The schema is created and migrated in code, not through standalone SQL migration files. The database has typed memory tables, event tables, contradiction tables, vector tables, and FTS5 tables (Ledger: E29-E30).
|
|
14
|
-
|
|
15
|
-
| Storage Component | Role | Implementation Evidence |
|
|
16
|
-
|---|---|---|
|
|
17
|
-
| `episodes` | Episodic memories, including source, confidence, tags, context, affect, privacy, agent scope, usage, and consolidation fields. | E29 |
|
|
18
|
-
| `semantics` | Durable factual or preference memories with state, confidence, provenance, agent scope, salience, and usage fields. | E29 |
|
|
19
|
-
| `procedures` | Reusable operating procedures with trigger, steps, state, confidence, salience, and usage fields. | E29 |
|
|
20
|
-
| `contradictions` | Records unresolved or resolved conflicts between claims. | E29 |
|
|
21
|
-
| `memory_events` | Tool and memory event log, including session, tool, outcome, hashes, redacted summaries, and metadata. | E29 |
|
|
22
|
-
| `causal_links`, `consolidation_runs`, `consolidation_metrics`, `audrey_config` | Supporting tables for consolidation, lineage, metrics, and schema versioning. | E29 |
|
|
23
|
-
| `vec_episodes`, `vec_semantics`, `vec_procedures` | sqlite-vec indexes for typed vector search. | E30 |
|
|
24
|
-
| `fts_episodes`, `fts_semantics`, `fts_procedures` | FTS5 indexes for typed lexical search. | E30 |
|
|
25
|
-
|
|
26
|
-
The implementation keeps vector and FTS storage type-specific instead of flattening all memory into one undifferentiated index. That matters for control behavior because preflight can ask for risks, procedures, contradictions, and recent tool outcomes separately before producing a decision (Ledger: E5, E9, E29-E30).
|
|
27
|
-
|
|
28
|
-
## Recall and FTS
|
|
29
|
-
|
|
30
|
-
Audrey implements vector, keyword, and hybrid recall modes. Hybrid recall fuses vector KNN and FTS5 BM25 with reciprocal rank fusion. The current constants are `RRF_K = 60`, `VECTOR_WEIGHT = 0.3`, and `FTS_WEIGHT = 0.7` (Ledger: E14). FTS search is backed by separate `fts_episodes`, `fts_semantics`, and `fts_procedures` tables and returns BM25-ranked matches joined back to the live typed memory tables (Ledger: E30).
|
|
31
|
-
|
|
32
|
-
Recall is failure-aware. Before vector search, Audrey checks whether each expected vector table exists. During recall, it catches per-type KNN failures and FTS lookup failures, records them as `RecallError[]`, marks the result as a partial failure, and still returns available partial results (Ledger: E15, E40). Preflight then turns recall errors into high-severity memory-health warnings, so degraded retrieval changes control output instead of being silently swallowed (Ledger: E9-E10).
|
|
33
|
-
|
|
34
|
-
This distinction is important for GuardBench. A retrieval system that fails open under missing indexes can look accurate on happy-path queries while making unsafe tool decisions under degraded memory. Audrey exposes degradation through the same evidence and warning channels used for ordinary risks.
|
|
35
|
-
|
|
36
|
-
## Capsule Implementation
|
|
37
|
-
|
|
38
|
-
A memory capsule is the bounded context object that bridges recall and control. It contains budget metadata, truncation status, must-follow rules, project facts, preferences, procedures, risks, recent changes, contradictions, uncertain or disputed memories, evidence IDs, and recall errors (Ledger: E5). Capsule construction forces `scope: 'agent'`, preserving agent-local memory boundaries during recall (Ledger: E7).
|
|
39
|
-
|
|
40
|
-
Capsules also enforce a control-source gate. A `must-follow` style memory becomes a control signal only when the source is trusted as `direct-observation` or `told-by-user`; otherwise it is routed to uncertain or disputed context rather than treated as an instruction (Ledger: E6). This prevents untrusted memory content from escalating itself into an operational rule.
|
|
41
|
-
|
|
42
|
-
Budget enforcement is performed before the capsule is handed to preflight. The implementation tracks whether the capsule was truncated and keeps evidence IDs available even when natural-language sections are shortened (Ledger: E5). The pre-action controller therefore receives both a bounded textual packet and an auditable evidence list.
|
|
43
|
-
|
|
44
|
-
## Preflight and Reflexes
|
|
45
|
-
|
|
46
|
-
Preflight is the risk-scoring layer. Its output contract includes `go`, `caution`, and `block` decisions, a risk score, warnings, recent failures, status, recommended actions, evidence IDs, an optional event ID, and an optional capsule (Ledger: E8). Internally, preflight builds a capsule with risks and contradictions enabled, checks memory health, converts recall errors into high-severity warnings, and adds warning sources for recent failures, must-follow rules, risks, procedures, contradictions, and uncertain or disputed memories (Ledger: E9).
|
|
47
|
-
|
|
48
|
-
Warnings are sorted by severity. Risk score is derived from the highest warning severity, and strict mode blocks high-severity warnings (Ledger: E10). Reflex generation maps preflight warnings into `guide`, `warn`, or `block` reflexes with evidence IDs, reasons, recommendations, and optional embedded preflight data (Ledger: E11).
|
|
49
|
-
|
|
50
|
-
Evidence propagation is preserved across these layers. A warning generated from a procedure, contradiction, recall error, or prior failure carries evidence IDs into the preflight response; reflexes then carry those IDs into the agent-facing guard report (Ledger: E8-E11).
|
|
51
|
-
|
|
52
|
-
## Controller Implementation
|
|
53
|
-
|
|
54
|
-
`MemoryController.beforeAction()` is the pre-action entry point. It runs `audrey.reflexes()` in strict mode, requests the preflight and capsule, records a preflight event, and scopes recall to the current agent (Ledger: E2). The external guard result is `allow`, `warn`, or `block`, with a risk score, summary, evidence IDs, recommendations, optional capsule, reflexes, and optional preflight event ID (Ledger: E1).
|
|
55
|
-
|
|
56
|
-
Exact repeated-failure control is deterministic. Audrey computes an action identity from the tool name, redacted command or action text, normalized working directory, and sorted normalized file scope. If a prior failed tool event carries the same identity, the controller blocks the action before tool use (Ledger: E3). This is stricter than semantic similarity: the repeat block is keyed to the normalized action, not to whether an embedding happens to retrieve the old failure.
|
|
57
|
-
|
|
58
|
-
`MemoryController.afterAction()` closes the loop after tool execution. It records tool outcomes through `observeTool()`, stores the `audrey_guard_action_key`, redacts action, command, and error text, and encodes failed outcomes as tool-result memories (Ledger: E4). The same event stream supports future preflight checks and impact reporting.
|
|
59
|
-
|
|
60
|
-
## Redaction Implementation
|
|
61
|
-
|
|
62
|
-
Audrey redacts before tool traces enter durable memory. The tool-tracing module states that raw tool input, output, and error text do not leave the module without redaction, and it stores hashes, redacted summaries or details, file fingerprints, redaction state, and a memory event (Ledger: E12).
|
|
63
|
-
|
|
64
|
-
The redaction catalog covers named credentials, generic credentials, payment and PII patterns, and entropy-based fallbacks (Ledger: E13). Examples of concrete coverage:
|
|
65
|
-
|
|
66
|
-
| Pattern Class | Example Input Shape | Output Shape |
|
|
67
|
-
|---|---|---|
|
|
68
|
-
| AWS access key | `AKIA` followed by 16 uppercase alphanumeric characters. | `[REDACTED:aws_access_key:tail]` |
|
|
69
|
-
| Anthropic API key | `sk-ant-` followed by a long token. | `[REDACTED:anthropic_api_key:tail]` |
|
|
70
|
-
| OpenAI API key | `sk-...` or `sk-proj-...` long token. | `[REDACTED:openai_api_key:tail]` |
|
|
71
|
-
| GitHub token | `ghp_`, `gho_`, `ghu_`, `ghs_`, or `ghr_` token. | `[REDACTED:github_token:tail]` |
|
|
72
|
-
| Stripe key | `sk_live_`, `rk_live_`, `pk_live_`, or test equivalents. | `[REDACTED:stripe_live_key:tail]` or `[REDACTED:stripe_test_key:tail]` |
|
|
73
|
-
| Google API key | `AIza` plus the expected key body. | `[REDACTED:google_api_key:tail]` |
|
|
74
|
-
| Slack token | `xoxb-`, `xoxa-`, `xoxp-`, `xoxr-`, or `xoxs-` style token. | `[REDACTED:slack_token:tail]` |
|
|
75
|
-
| Bearer or Basic auth | `Bearer <token>` or `Basic <base64>`. | `Bearer [REDACTED:generic_bearer]` or `Basic [REDACTED:basic_auth]` |
|
|
76
|
-
| Private key block | PEM private-key block. | `[REDACTED:private_key_block]` |
|
|
77
|
-
| URL credentials | `scheme://user:password@host`. | `scheme://user:[REDACTED:url_credentials]@host` |
|
|
78
|
-
| Password assignment | `password=...`, `api_key: ...`, `auth_token=...`, or similar keys. | Original key plus `[REDACTED:password_assignment]` |
|
|
79
|
-
| Payment and PII | Luhn-valid card numbers, CVV labels, and US SSNs. | Payment or PII class marker. |
|
|
80
|
-
| Signed URLs and sessions | Signature, token, and session-cookie query or cookie fields. | Preserved field name plus redaction marker. |
|
|
81
|
-
| High-entropy fallback | Long mixed-character token with sufficient entropy. | `[REDACTED:high_entropy_secret:tail]` |
|
|
82
|
-
|
|
83
|
-
The JSON walker redacts sensitive keys and values recursively. If a value sits under a sensitive key and does not match a named pattern, it is still replaced with a password-assignment marker (Ledger: E13). Truncation is applied after redaction and preserves redaction markers rather than cutting them in half or dropping the only proof that a secret was removed (Ledger: E13).
|
|
84
|
-
|
|
85
|
-
## MCP, CLI, and REST Surfaces
|
|
86
|
-
|
|
87
|
-
The MCP server registers 22 tools: `memory_dream`, `memory_encode`, `memory_recall`, `memory_consolidate`, `memory_introspect`, `memory_resolve_truth`, `memory_export`, `memory_import`, `memory_forget`, `memory_validate`, `memory_decay`, `memory_status`, `memory_reflect`, `memory_greeting`, `memory_observe_tool`, `memory_recent_failures`, `memory_capsule`, `memory_preflight`, `memory_guard_before`, `memory_guard_after`, `memory_reflexes`, and `memory_promote` (Ledger: E32). The Guard-relevant MCP surface is `memory_observe_tool`, `memory_recent_failures`, `memory_capsule`, `memory_preflight`, `memory_guard_before`, `memory_guard_after`, and `memory_reflexes`, with `memory_validate` supporting closed-loop validation and REST or CLI impact reporting supporting aggregate impact inspection (Ledger: E16-E19, E32-E34).
|
|
88
|
-
|
|
89
|
-
The CLI recognizes `install`, `uninstall`, `mcp-config`, `hook-config`, `demo`, `guard`, `guard-after`, `hook`, `reembed`, `dream`, `greeting`, `reflect`, `serve`, `status`, `doctor`, `observe-tool`, `promote`, and `impact` (Ledger: E34). The `guard` subcommand remains available for explicit checks, while the host-facing `hook` subcommand normalizes current Claude Code and Codex lifecycle payloads. `audrey install --host auto` discovers installed supported CLIs, registers the MCP server, and applies host-specific lifecycle hooks from a stable Node and Audrey entrypoint. Generated settings preserve unrelated hooks, replace older Audrey-owned handlers, write backups before changing non-empty files, and are idempotent. Claude Code supports local, project, and user hook scopes; Codex supports project and user scopes and requires one-time review of non-managed hooks through `/hooks` (Ledger: E26, E43).
|
|
90
|
-
|
|
91
|
-
The generated lifecycle covers session start, user prompt submission, guarded pre-tool use, correlated post-tool outcomes, post-compaction, and turn stop. Session and prompt hooks inject bounded, agent-scoped memory context. Retrieval queries are bounded before embedding: large prompts and edits contribute redacted previews, lengths, and hashes rather than unbounded bodies, while exact Guard identity digests the full normalized, redacted action. Pre-tool hooks create Guard receipts, and post-tool hooks close the matching receipt by `session_id + tool_use_id`; sanitized failures may become durable memories. Claude Code exposes an explicit failure event. Current Codex Bash hooks can omit exit status, so Audrey records an opaque result as `unknown` unless structured fields or response text establish success or failure. Stop and post-compaction hooks run consolidation only when due. Audrey fails open on hook-runtime errors by default, with explicit fail-closed operation available through `AUDREY_HOOK_FAIL_CLOSED=1`. This is lifecycle integration, not a complete enforcement boundary: Audrey can inspect only the events and tool paths emitted by each host, and the short-lived hook process still pays provider startup cost instead of using a persistent daemon (Ledger: E43, E44).
|
|
92
|
-
|
|
93
|
-
The REST sidecar exposes routes for health, encode, recall, validate, mark-used, capsule, preflight, Guard before/after receipts, reflexes, consolidate, dream, introspect, impact, resolve-truth, export, import, forget, decay, status, reflect, and greeting (Ledger: E33). The sidecar defaults to loopback binding, refuses non-loopback binds without `AUDREY_API_KEY` unless `AUDREY_ALLOW_NO_AUTH=1`, and emits an explicit warning when that no-auth override is used (Ledger: E35). Export, import, and forget are disabled unless `AUDREY_ENABLE_ADMIN_TOOLS=1` (Ledger: E33).
|
|
94
|
-
|
|
95
|
-
## Configuration
|
|
96
|
-
|
|
97
|
-
The README documents the runtime environment variables that affect storage, provider selection, server exposure, admin tools, and performance behavior (Ledger: E37). The security-critical defaults are:
|
|
98
|
-
|
|
99
|
-
| Variable | Default | Security Role |
|
|
100
|
-
|---|---|---|
|
|
101
|
-
| `AUDREY_HOST` | `127.0.0.1` | Keeps the REST sidecar on loopback by default. Non-loopback exposure requires auth or an explicit unsafe override (Ledger: E35, E37). |
|
|
102
|
-
| `AUDREY_API_KEY` | unset | Required bearer token for non-loopback REST traffic (Ledger: E35, E37). |
|
|
103
|
-
| `AUDREY_ALLOW_NO_AUTH` | `0` | Escape hatch for non-loopback without auth. The docs explicitly mark this as unsafe (Ledger: E35, E37). |
|
|
104
|
-
| `AUDREY_ENABLE_ADMIN_TOOLS` | `0` | Keeps export, import, and forget routes/tools disabled by default (Ledger: E33, E37). |
|
|
105
|
-
| `AUDREY_PROMOTE_ROOTS` | unset | Restricts `audrey promote --yes` writes to `process.cwd()` unless additional roots are explicitly listed (Ledger: E37). |
|
|
106
|
-
|
|
107
|
-
`AUDREY_MODEL` is not part of the documented Audrey environment matrix and is not used as an Audrey runtime variable in the inspected source. Model selection is currently represented by provider defaults and provider-specific environment variables, not by a single `AUDREY_MODEL` switch (Ledger: E38).
|
|
108
|
-
|
|
109
|
-
## Testing and Release Gates
|
|
110
|
-
|
|
111
|
-
The package scripts wire build, typecheck, Vitest, perf benchmark, performance snapshot, memory regression check, npm pack dry-run, release gate, and sandbox release gate commands (Ledger: E39). The documented release path also includes Python unittest discovery and Python package build commands (Ledger: E39).
|
|
112
|
-
|
|
113
|
-
`bench:memory:check` is a regression gate. It runs retrieval and lifecycle benchmark suites, compares Audrey against vector-only, keyword-plus-recency, and recent-window local baselines, and enforces score, pass-rate, and margin guardrails (Ledger: E23). Section 7 reports the current output as a regression-gate result, not as a cross-system leaderboard.
|
|
114
|
-
|
|
115
|
-
For the repeated-failure evaluation transcript in this paper, the project was rebuilt with `npm run build` before running `node dist/mcp-server/index.js demo --scenario repeated-failure`; the build completed successfully (Ledger: E41).
|
|
@@ -1,168 +0,0 @@
|
|
|
1
|
-
# 7. Evaluation
|
|
2
|
-
|
|
3
|
-
This Stage-A evaluation reports implemented Audrey artifacts and local GuardBench adapters only. Section 5 specifies GuardBench as a reproducibility contract for pre-action memory control. The empirical claims below come from the repository's existing performance snapshot, the current behavioral regression output, the local comparative GuardBench runner, source-linked implementation inspection, and a freshly captured repeated-failure demo transcript.
|
|
4
|
-
|
|
5
|
-
## Methodology Disclosure
|
|
6
|
-
|
|
7
|
-
The evaluation separates specification from implementation evidence. GuardBench defines the scenarios, baselines, metrics, and reproducibility requirements for future external-system evaluation. This paper reports what the current repository already supports: encode and hybrid-recall latency from the canonical 0.22.2 snapshot, the `bench:memory:check` regression gate output, a local comparative GuardBench run, and a deterministic demo showing Audrey Guard blocking a repeated failed action (Ledger: E20-E25, E41-E42, E46).
|
|
8
|
-
|
|
9
|
-
This is narrower than a cross-system benchmark. It is also the honest Stage-A claim: the paper introduces the control problem, specifies the benchmark, reports local comparative results, and avoids unrun external-system comparisons.
|
|
10
|
-
|
|
11
|
-
## Performance Snapshot
|
|
12
|
-
|
|
13
|
-
The canonical performance snapshot is `benchmarks/snapshots/perf-0.22.2.json`. It was generated on 2026-05-01T02:15:29.400Z from git SHA `e2e821b`, using mock in-process 64-dimensional embeddings, a mock in-process LLM provider, hybrid vector-plus-lexical retrieval with limit 5, corpus sizes 100, 1000, and 5000, and 50 recall runs per corpus size. The machine provenance is Node 25.5.0, V8 14.1.146.11-node.18, Windows x64, 24-core AMD Ryzen 9 7900X3D, and 62.9 GB RAM (Ledger: E20).
|
|
14
|
-
|
|
15
|
-
The snapshot reports the following encode and hybrid recall latencies in milliseconds:
|
|
16
|
-
|
|
17
|
-
| Corpus Size | Encode p50 | Encode p95 | Encode p99 | Hybrid Recall p50 | Hybrid Recall p95 | Hybrid Recall p99 |
|
|
18
|
-
|---:|---:|---:|---:|---:|---:|---:|
|
|
19
|
-
| 100 | 0.331 | 0.589 | 7.65 | 0.539 | 1.82 | 2.712 |
|
|
20
|
-
| 1000 | 0.307 | 2.147 | 9.672 | 1.566 | 2.364 | 21.177 |
|
|
21
|
-
| 5000 | 0.308 | 1.838 | 10.45 | 2.091 | 3.417 | 16.58 |
|
|
22
|
-
|
|
23
|
-
These numbers measure Audrey's local call path under an in-process mock embedding provider. They do not measure real embedding-provider latency, local transformer warmup cost, GPU/CPU variance for 384-dimensional local embeddings, OpenAI or Gemini network latency, or production-load concurrency. The snapshot itself notes that cloud and local 384-dimensional providers will report higher recall latency dominated by embedding cost and network (Ledger: E20-E22, E31).
|
|
24
|
-
|
|
25
|
-
## Behavioral Regression Result
|
|
26
|
-
|
|
27
|
-
The current `benchmarks/output/summary.json` was generated on 2026-07-13T15:07:26.566Z with command `node benchmarks/run.js --provider mock --dimensions 64` (Ledger: E24). It reports:
|
|
28
|
-
|
|
29
|
-
| System | Score Percent | Pass Rate | Average Duration Ms |
|
|
30
|
-
|---|---:|---:|---:|
|
|
31
|
-
| Audrey | 100 | 100 | 19.333333333333332 |
|
|
32
|
-
| Vector Only | 41.66666666666667 | 25 | 0.25 |
|
|
33
|
-
| Keyword + Recency | 41.66666666666667 | 25 | 0.5833333333333334 |
|
|
34
|
-
| Recent Window | 37.5 | 25 | 0.08333333333333333 |
|
|
35
|
-
|
|
36
|
-
This output is a regression-gate result. The baselines are toy local baselines used to catch retrieval and lifecycle regressions in the Audrey codebase. They are not external systems, not tuned competitor implementations, and not GuardBench baselines (Ledger: E23-E24). The current suite covers retrieval and operation families such as information extraction, knowledge updates, multi-session reasoning, conflict resolution, procedural learning, privacy boundary, overwrite, delete-and-abstain, semantic merge, and procedural merge (Ledger: E23-E24).
|
|
37
|
-
|
|
38
|
-
## GuardBench Local Comparative Result
|
|
39
|
-
|
|
40
|
-
The current `benchmarks/output/guardbench-summary.json`,
|
|
41
|
-
`benchmarks/output/guardbench-manifest.json`, and
|
|
42
|
-
`benchmarks/output/guardbench-raw.json` were generated on 2026-05-12 with:
|
|
43
|
-
|
|
44
|
-
```bash
|
|
45
|
-
npm run bench:guard:check
|
|
46
|
-
```
|
|
47
|
-
|
|
48
|
-
It reports local adapters only, not external-system comparisons (Ledger: E46):
|
|
49
|
-
|
|
50
|
-
| Metric | Result |
|
|
51
|
-
|---|---:|
|
|
52
|
-
| Scenarios passed | 10 / 10 |
|
|
53
|
-
| Prevention rate | 100% |
|
|
54
|
-
| False-block rate | 0% |
|
|
55
|
-
| Evidence recall | 100% |
|
|
56
|
-
| Redaction leaks | 0 |
|
|
57
|
-
| Recall-degradation detection | 100% |
|
|
58
|
-
| Guard latency p50 / p95 | 4.607 ms / 21.585 ms |
|
|
59
|
-
| Published artifact raw-secret leaks | 0 |
|
|
60
|
-
| Audrey Guard decision accuracy | 100% |
|
|
61
|
-
| No-memory decision accuracy | 10% |
|
|
62
|
-
| Recent-window decision accuracy | 60% |
|
|
63
|
-
| Vector-only decision accuracy | 40% |
|
|
64
|
-
| FTS-only decision accuracy | 10% |
|
|
65
|
-
|
|
66
|
-
The ten scenarios cover exact repeated failures, required procedures, changed
|
|
67
|
-
file scopes, changed commands, recovered failures, vector recall degradation,
|
|
68
|
-
FTS recall degradation, truncation-boundary secret redaction, conflicting
|
|
69
|
-
instructions, and noisy-store control-memory recall. These results are the
|
|
70
|
-
first public local comparative Audrey GuardBench numbers. The emitted manifest
|
|
71
|
-
records the ten scenario actions, seeded memories, seeded tool events, fault
|
|
72
|
-
injections, expected evidence classes, and non-secret references for seeded
|
|
73
|
-
redaction probes; the raw output file records every local adapter result for
|
|
74
|
-
each case. The harness also sweeps the summary, manifest, and raw output for
|
|
75
|
-
seeded raw secrets and fails the run on artifact leaks. External ESM adapters
|
|
76
|
-
receive private seed values at runtime while expected decisions and evidence
|
|
77
|
-
remain withheld during execution. The first concrete external adapter targets
|
|
78
|
-
Mem0 Platform via its REST APIs, but it has not yet been run with a live key, so
|
|
79
|
-
this section does not report Mem0 scores.
|
|
80
|
-
|
|
81
|
-
## Repeated-Failure Demo Transcript
|
|
82
|
-
|
|
83
|
-
The qualitative control figure is the deterministic repeated-failure demo. The project was rebuilt with `npm run build`, then the demo was run with:
|
|
84
|
-
|
|
85
|
-
```bash
|
|
86
|
-
node dist/mcp-server/index.js demo --scenario repeated-failure
|
|
87
|
-
```
|
|
88
|
-
|
|
89
|
-
The run produced the following transcript. The temporary path, memory IDs, and timestamp-bearing failure ID are run-specific; the decision structure is the evaluated behavior (Ledger: E25, E41-E42). Appendix A provides the same transcript as a standalone reproduction artifact with line annotations.
|
|
90
|
-
|
|
91
|
-
```text
|
|
92
|
-
Audrey Guard repeated-failure demo
|
|
93
|
-
|
|
94
|
-
Memory store: [LOCAL-TEMP]/audrey-demo-AkCROa
|
|
95
|
-
Step 1: the agent tries a deploy and hits a real setup failure.
|
|
96
|
-
Step 2: Audrey stores the failure and the operational rule it implies.
|
|
97
|
-
Lesson memory: 01KR491DG2YZHVEM79QVW5BHZA
|
|
98
|
-
|
|
99
|
-
Step 3: a new preflight checks the same action before tool use.
|
|
100
|
-
|
|
101
|
-
Audrey Guard: BLOCKED
|
|
102
|
-
|
|
103
|
-
Reason: Blocked: this exact Bash action failed before. Stop: 3 memory reflexes, 2 blocking, 1 warning matched.
|
|
104
|
-
Risk score: 0.90
|
|
105
|
-
|
|
106
|
-
Evidence:
|
|
107
|
-
- 01KR491DFZYZ20TFK71KJHC88F
|
|
108
|
-
- 01KR491DG2YZHVEM79QVW5BHZA
|
|
109
|
-
- failure:Bash:2026-05-08T17:09:22.047Z
|
|
110
|
-
|
|
111
|
-
Recommended action:
|
|
112
|
-
- Do not repeat the exact failed action until the prior error is understood or the command is changed.
|
|
113
|
-
- Do not proceed until the high-severity memory warning is addressed.
|
|
114
|
-
- Apply this must-follow rule before acting.
|
|
115
|
-
- Mitigate this remembered risk before proceeding.
|
|
116
|
-
- Before re-running Bash, check what changed since the last failure.
|
|
117
|
-
|
|
118
|
-
Memory reflexes:
|
|
119
|
-
- block: Apply this must-follow rule before acting. Before running npm run deploy, run npm run db:generate because Prisma client must be generated first.
|
|
120
|
-
- block: Mitigate this remembered risk before proceeding. Before running npm run deploy, run npm run db:generate because Prisma client must be generated first.
|
|
121
|
-
- warn: Before re-running Bash, check what changed since the last failure.
|
|
122
|
-
|
|
123
|
-
Next: fix the warning and retry, or pass --override to allow this guard check.
|
|
124
|
-
|
|
125
|
-
Impact:
|
|
126
|
-
- 1 repeated failure prevented
|
|
127
|
-
- 1 helpful memory validation recorded
|
|
128
|
-
- 3 evidence ids attached
|
|
129
|
-
|
|
130
|
-
Audrey saw the agent fail once.
|
|
131
|
-
Audrey stopped it from failing twice.
|
|
132
|
-
```
|
|
133
|
-
|
|
134
|
-
The transcript demonstrates the core pre-action memory-control loop for one scenario: a failed tool action is observed, a procedural lesson is encoded, a later identical action is intercepted before tool use, the guard returns a block decision, and the decision carries evidence IDs, recommendations, reflexes, and impact accounting (Ledger: E3-E4, E8-E11, E16-E17, E25, E42).
|
|
135
|
-
|
|
136
|
-
## Implemented-Evidence Summary
|
|
137
|
-
|
|
138
|
-
| Design Claim | Ledger IDs | Evidence Type |
|
|
139
|
-
|---|---|---|
|
|
140
|
-
| Audrey exposes a pre-action `allow`/`warn`/`block` controller result. | E1-E2 | Source inspection |
|
|
141
|
-
| Exact repeated failures are blocked by action identity. | E3, E25, E42 | Source inspection and demo |
|
|
142
|
-
| Post-action observation stores redacted tool outcomes and action identity. | E4, E12-E13 | Source inspection |
|
|
143
|
-
| Capsules preserve structured memory sections, evidence IDs, budget state, and recall errors. | E5-E7 | Source inspection |
|
|
144
|
-
| Preflight converts memory health, failures, risks, procedures, contradictions, and disputed memories into severity-sorted warnings. | E8-E10 | Source inspection |
|
|
145
|
-
| Reflexes carry response type, recommendations, and evidence IDs. | E11 | Source inspection |
|
|
146
|
-
| Recall degradation is represented and propagated as `RecallError[]`. | E15, E40 | Source inspection |
|
|
147
|
-
| Hybrid recall uses vector plus FTS with RRF constants `60`, `0.3`, and `0.7`. | E14 | Source inspection |
|
|
148
|
-
| Redaction covers named credentials, generic auth, private keys, payment/PII patterns, sessions, signed URLs, entropy fallback, JSON sensitive keys, and truncation marker preservation. | E12-E13 | Source inspection |
|
|
149
|
-
| The runtime includes SQLite, sqlite-vec, FTS5, MCP stdio, Hono REST, CLI, and Python client surfaces. | E29-E36 | Source inspection |
|
|
150
|
-
| Security-relevant defaults keep REST local, require API keys for non-loopback, disable no-auth exposure, disable admin tools, and restrict promote roots. | E33, E35, E37-E38 | Source inspection |
|
|
151
|
-
| Encode and hybrid-recall latency are reported from the canonical 0.22.2 perf snapshot. | E20-E22 | Snapshot |
|
|
152
|
-
| Behavioral regression status is reported from the current `bench:memory:check` output. | E23-E24 | Regression output |
|
|
153
|
-
| Local comparative GuardBench status is reported from `bench:guard:check`. | E46 | GuardBench comparative output |
|
|
154
|
-
| The repeated-failure control loop is demonstrated by the `demo --scenario repeated-failure` transcript. | E25, E41-E42 | Demo |
|
|
155
|
-
|
|
156
|
-
## What This Section Does Not Claim
|
|
157
|
-
|
|
158
|
-
This section does not claim cross-system superiority over Mem0, Letta/MemGPT, Zep, Graphiti, MemOS, LangMem, Supermemory, Cognee, or any production memory service.
|
|
159
|
-
|
|
160
|
-
This section claims local comparative GuardBench results, the adapter contract, and the existence of the Mem0 adapter only. External-system GuardBench outputs are deferred until live runs are captured.
|
|
161
|
-
|
|
162
|
-
This section does not claim production-load measurements. The performance snapshot is a local mock-provider benchmark, not a concurrency, soak, storage-pressure, or real-provider benchmark.
|
|
163
|
-
|
|
164
|
-
This section does not claim real-hardware variance. It reports one machine's provenance and raw numbers from the repository snapshot.
|
|
165
|
-
|
|
166
|
-
This section does not claim perfect redaction coverage. It reports implemented pattern coverage and the current GuardBench artifact sweep for seeded raw-secret leakage.
|
|
167
|
-
|
|
168
|
-
This section does not claim that local toy baselines represent external systems. The current `bench:memory:check` baselines exist to detect Audrey regressions.
|
|
@@ -1,61 +0,0 @@
|
|
|
1
|
-
# 8. Discussion and Limitations
|
|
2
|
-
|
|
3
|
-
## What the Implementation Changes
|
|
4
|
-
|
|
5
|
-
Repeated-failure prevention is the clearest implemented behavior. The demo records one failed `npm run deploy`, stores the operational lesson that Prisma generation must run first, and blocks the identical future action before tool use. The guard output includes a `BLOCKED` decision, risk score `0.90`, two blocking reflexes, one warning reflex, evidence IDs, concrete recommendations, and impact accounting (Ledger: E25, E42). The change is not better recall in a chat answer. The change is that the second tool call does not happen.
|
|
6
|
-
|
|
7
|
-
The redacted evidence trail changes what can be safely remembered. Audrey's tool-tracing path states that raw tool input, output, and error text do not leave the module without redaction, and it stores redacted summaries, hashes, file fingerprints, redaction state, and a memory event (Ledger: E12). The redaction catalog covers named API keys, auth headers, private keys, URL credentials, password assignments, payment and PII patterns, signed URLs, session cookies, high-entropy tokens, JSON sensitive keys, and truncation marker preservation (Ledger: E13). This lets guard decisions point back to prior tool evidence without turning the memory store into an unfiltered secret archive.
|
|
8
|
-
|
|
9
|
-
Action-key determinism gives the controller a hard repeated-failure path. Audrey hashes tool name, redacted command or action text, normalized working directory, and sorted normalized file scope, then matches the hash against prior failed tool events for the same agent (Ledger: E3). This is separate from semantic retrieval. A prior failure does not need to be semantically similar enough to rank first; the exact repeated action is a structured event match.
|
|
10
|
-
|
|
11
|
-
Recall-degradation handling changes failure posture. Audrey records missing vector tables, per-type KNN failures, and FTS lookup failures as `RecallError[]`, preserves partial results, exposes degradation in memory status, and carries errors into capsules (Ledger: E15, E40). Preflight converts recall errors into high-severity memory-health warnings, and strict guard mode blocks high-severity warnings (Ledger: E9-E10). A degraded memory substrate therefore becomes a visible control condition instead of an invisible recall-quality drop.
|
|
12
|
-
|
|
13
|
-
## Conservative Control Choices
|
|
14
|
-
|
|
15
|
-
Audrey is conservative by design. Strict mode blocks high-severity warnings rather than asking the model to decide whether a remembered warning matters (Ledger: E10). This increases false-positive risk when a high-severity warning is stale or overbroad. The trade-off is appropriate for pre-action control because the guarded operations include file mutation, shell execution, credential exposure, network calls, and destructive tools. The safer default is to force inspection when memory reports high risk.
|
|
16
|
-
|
|
17
|
-
Agent-scoped recall is another conservative choice. Capsule construction forces `scope: 'agent'`, so a caller cannot accidentally widen a capsule to shared memory by passing broad recall options (Ledger: E7). This reduces cross-agent leakage at the cost of hiding useful memory learned by another agent. Cross-agent sharing belongs behind an explicit federation policy, not an accidental default.
|
|
18
|
-
|
|
19
|
-
The trusted-control-source gate is also conservative. A memory tagged as must-follow becomes a control rule only when its source is `direct-observation` or `told-by-user`; untrusted must-follow tags are routed to uncertain or disputed context (Ledger: E6). This blocks an obvious memory-injection path. It also creates false positives when a useful operating rule comes from a source that has not been promoted into trust.
|
|
20
|
-
|
|
21
|
-
## What This Paper Does Not Claim
|
|
22
|
-
|
|
23
|
-
This paper reports one local comparative GuardBench run. It does not report GuardBench results across external memory systems. Section 5 specifies GuardBench; v2 reports the full external-system run.
|
|
24
|
-
|
|
25
|
-
It does not report cross-system comparisons against Mem0, Letta/MemGPT, Zep, Graphiti, MemOS, LangMem, Supermemory, Cognee, or hosted memory services.
|
|
26
|
-
|
|
27
|
-
It does not report production-load measurements, concurrent-agent soak tests, storage-pressure behavior, or long-running operational telemetry.
|
|
28
|
-
|
|
29
|
-
It does not report real-provider embedding latency. The canonical performance snapshot uses a mock in-process 64-dimensional embedding provider (Ledger: E20-E22).
|
|
30
|
-
|
|
31
|
-
It does not prove redaction completeness. The implementation has a broad rule catalog, but unknown credential formats and adversarial encodings remain out of scope (Ledger: E13).
|
|
32
|
-
|
|
33
|
-
It does not prevent first-time errors. Pre-action memory control works from prior evidence, remembered rules, contradictions, and recall health. A novel error with no remembered signal still reaches the underlying tool policy.
|
|
34
|
-
|
|
35
|
-
It does not replace sandboxing, OS permissions, MCP permission systems, human approval, or network isolation. Audrey is a memory-derived control layer that fits inside a host's broader tool-use safety stack.
|
|
36
|
-
|
|
37
|
-
## Threats to Current Claims
|
|
38
|
-
|
|
39
|
-
Action-key fidelity is a central threat. Repeated-failure prevention depends on stable normalization of command text, working directory, and file scope. If a host supplies incomplete file lists, unstable cwd values, or action text that hides the meaningful operation inside nested arguments, exact-repeat detection loses coverage (Ledger: E3). The current hash is deterministic, not omniscient.
|
|
40
|
-
|
|
41
|
-
Redaction is rule-based. It covers many common credential and PII formats, sensitive JSON keys, high-entropy strings, and truncation boundaries (Ledger: E13). It remains incomplete for novel secret formats, multi-part secrets split across fields, encoded payloads, and tool outputs crafted to evade regex and entropy checks.
|
|
42
|
-
|
|
43
|
-
Capsule pruning is priority-based, not learned. Capsules enforce a character budget and preserve structured sections and evidence IDs (Ledger: E5). The current implementation uses explicit sectioning and truncation logic rather than a learned policy that optimizes downstream guard accuracy. Budget pressure can hide useful non-control context while keeping control evidence.
|
|
44
|
-
|
|
45
|
-
Reflex generation is deterministic, not adaptive. Reflexes are mapped from preflight warnings into `guide`, `warn`, or `block` responses with evidence and recommendations (Ledger: E11). The mapping does not learn from later validation events. Validation updates memory salience and bookkeeping, but risk scoring remains fixed by severity rules (Ledger: E16, E45).
|
|
46
|
-
|
|
47
|
-
## Open Problems
|
|
48
|
-
|
|
49
|
-
Host hook coverage and process model. Audrey 1.1 generates, applies, and removes lifecycle hooks for current Claude Code and Codex CLIs; the shared adapter injects bounded context, runs Guard before selected side-effectful tools, correlates post-tool outcomes, and performs due-only maintenance (Ledger: E43). This closes the configuration gap, not the enforcement gap. Hook execution still depends on host trust and on the host emitting a supported event for the relevant tool path. Alternate or incompletely intercepted host tool paths remain outside Audrey's view. Each hook also starts a short-lived Audrey process, so the default local embedding provider can pay cold-start cost repeatedly. Production follow-up is a persistent local runtime, measured compatibility gates against supported host versions, and organization-managed deployment that binds tenant identity and enforcement policy outside a caller-controlled routing header. Claude Code's documented hook surface remains an example of the underlying host boundary [@anthropic2026claudecodehooks].
|
|
50
|
-
|
|
51
|
-
Validation lineage is implemented but not yet policy-adaptive. Audrey can bind validation events to the exact preflight event, evidence IDs, and action key that produced a decision, and rejects mismatched evidence claims (Ledger: E44). The next step is using that closed-loop signal to tune warning priority and recommendation wording without giving the model direct control over policy.
|
|
52
|
-
|
|
53
|
-
Cross-agent memory federation. Audrey currently protects capsules through agent-scoped recall (Ledger: E7). Multi-agent runtimes need explicit federation rules: which memories transfer across agents, which remain private, which require user confirmation, and how contradictions propagate across agent identities.
|
|
54
|
-
|
|
55
|
-
Adaptive risk scoring. Preflight uses a fixed severity map and strict mode blocks high-severity warnings (Ledger: E45). Validation feedback should eventually tune risk scoring, warning ordering, and recommendation wording without giving the model direct control over the safety policy.
|
|
56
|
-
|
|
57
|
-
Adversarial-memory robustness. The trusted-control-source gate blocks untrusted must-follow tags, but poisoned tool outputs that enter through trusted observation paths remain a hard problem (Ledger: E6, E12). Future work needs adversarial memory tests where attacker-controlled output resembles an operational rule, a project instruction, or a false recovery signal.
|
|
58
|
-
|
|
59
|
-
## Local-First as a Feature
|
|
60
|
-
|
|
61
|
-
Pre-action memory control operates on sensitive operational history: shell commands, file paths, build failures, project rules, API error summaries, user instructions, and redacted tool outputs. Sending that control surface to a hosted memory service creates an avoidable privacy, availability, and latency dependency. Audrey's local-first SQLite, sqlite-vec, FTS5, loopback REST default, and no-auth network refusal keep the controller deployable inside local agents and air-gapped environments (Ledger: E29-E30, E35, E37). The local design is not just an implementation convenience; it is aligned with the data that a pre-action controller must inspect.
|
|
@@ -1,11 +0,0 @@
|
|
|
1
|
-
# 9. Conclusion
|
|
2
|
-
|
|
3
|
-
Agent memory should be judged by whether it changes future tool actions. Audrey implements that claim for local agents through a host-side memory controller that runs before tool use and returns an auditable `allow`, `warn`, or `block` decision with evidence.
|
|
4
|
-
|
|
5
|
-
The implemented contribution is Audrey Guard: a local-first loop that observes tool outcomes, redacts traces, retrieves hybrid memory, builds bounded capsules, scores preflight risk, generates reflexes, blocks exact repeated failures, records post-action outcomes, and reports validation-linked impact (Ledger: E1-E17, E25-E26, E29-E42). The specified contribution is GuardBench: a scenario manifest, baseline set, metric suite, and reproducibility contract for evaluating memory by action effect rather than retrieved-text relevance.
|
|
6
|
-
|
|
7
|
-
The Stage-A evidence is intentionally bounded. This paper reports source-linked implementation evidence, the canonical 0.22.2 performance snapshot, the current behavioral regression gate output, a local comparative GuardBench run, and a deterministic repeated-failure demo transcript (Ledger: E20-E25, E41-E42, E46). It also includes the external adapter contract and Mem0 evidence-bundle path, but it does not report full external-system GuardBench results.
|
|
8
|
-
|
|
9
|
-
The v2 paper should run live external adapters, publish raw per-scenario output bundles, run expanded redaction sweeps, and report guard-overhead p50/p95 under machine-provenance controls.
|
|
10
|
-
|
|
11
|
-
The core result is simple: Audrey saw the agent fail once. Audrey stopped it from failing twice.
|
|
@@ -1,162 +0,0 @@
|
|
|
1
|
-
# Submission README
|
|
2
|
-
|
|
3
|
-
## What's Complete
|
|
4
|
-
|
|
5
|
-
- Nine drafted body sections in `01-introduction.md` through `09-conclusion.md`.
|
|
6
|
-
- Consolidated Markdown master: `audrey-paper-v1.md`.
|
|
7
|
-
- Evidence ledger with 97 rows: `evidence-ledger.md`.
|
|
8
|
-
- Machine-readable claim register: `claim-register.json`.
|
|
9
|
-
- Machine-readable publication pack: `publication-pack.json`.
|
|
10
|
-
- Machine-readable browser launch plan: `browser-launch-plan.json`.
|
|
11
|
-
- Machine-readable browser launch results ledger: `browser-launch-results.json`.
|
|
12
|
-
- Machine-readable arXiv source package: `docs/paper/output/arxiv/`.
|
|
13
|
-
- Machine-readable arXiv compile report: `docs/paper/output/arxiv-compile-report.json`.
|
|
14
|
-
- Compiled arXiv PDF and compile log: `docs/paper/output/arxiv-compile/main.pdf`, `docs/paper/output/arxiv-compile/arxiv-compile.log`.
|
|
15
|
-
- Machine-readable paper submission bundle: `docs/paper/output/submission-bundle/`.
|
|
16
|
-
- Primary-source bibliography with 21 entries: `references.bib`.
|
|
17
|
-
- Verbatim repeated-failure demo transcript: `appendix-a-demo-transcript.md`.
|
|
18
|
-
- GuardBench Stage-A specification with scenario manifest, baselines, metrics, reproducibility contract, JSON schemas, and Stage-A/Stage-B boundary.
|
|
19
|
-
- Machine-readable GuardBench manifest schema: `benchmarks/schemas/guardbench-manifest.schema.json`.
|
|
20
|
-
- Machine-readable GuardBench summary schema: `benchmarks/schemas/guardbench-summary.schema.json`.
|
|
21
|
-
- Machine-readable GuardBench raw-output schema: `benchmarks/schemas/guardbench-raw.schema.json`.
|
|
22
|
-
- Machine-readable GuardBench external-run metadata schema with SHA-256 artifact hashes: `benchmarks/schemas/guardbench-external-run.schema.json`.
|
|
23
|
-
- Machine-readable GuardBench external dry-run matrix schema: `benchmarks/schemas/guardbench-external-dry-run.schema.json`.
|
|
24
|
-
- Machine-readable GuardBench external evidence verification schema: `benchmarks/schemas/guardbench-external-evidence.schema.json`.
|
|
25
|
-
- Machine-readable GuardBench conformance-card schema: `benchmarks/schemas/guardbench-conformance-card.schema.json`.
|
|
26
|
-
- Machine-readable GuardBench submission-manifest schema: `benchmarks/schemas/guardbench-submission-manifest.schema.json`.
|
|
27
|
-
- Machine-readable GuardBench leaderboard schema: `benchmarks/schemas/guardbench-leaderboard.schema.json`.
|
|
28
|
-
- Machine-readable GuardBench adapter self-test schema: `benchmarks/schemas/guardbench-adapter-self-test.schema.json`.
|
|
29
|
-
- Machine-readable GuardBench adapter registry schema: `benchmarks/schemas/guardbench-adapter-registry.schema.json`.
|
|
30
|
-
- Machine-readable GuardBench publication verifier schema: `benchmarks/schemas/guardbench-publication-verification.schema.json`.
|
|
31
|
-
- Machine-readable browser launch-plan schema: `docs/paper/browser-launch-plan.schema.json`.
|
|
32
|
-
- Machine-readable browser launch-results schema: `docs/paper/browser-launch-results.schema.json`.
|
|
33
|
-
- Machine-readable arXiv source-manifest schema: `docs/paper/arxiv-source.schema.json`.
|
|
34
|
-
- Machine-readable arXiv compile-report schema: `docs/paper/arxiv-compile-report.schema.json`.
|
|
35
|
-
- Machine-readable paper submission-bundle schema: `docs/paper/paper-submission-bundle.schema.json`.
|
|
36
|
-
- Standalone GuardBench artifact validator: `npm run bench:guard:validate`.
|
|
37
|
-
- Public artifact path normalizer and local absolute-path sweep: `benchmarks/public-paths.mjs`.
|
|
38
|
-
- Shareable GuardBench conformance-card generator: `npm run bench:guard:card`.
|
|
39
|
-
- Portable GuardBench submission-bundle generator: `npm run bench:guard:bundle`.
|
|
40
|
-
- Offline GuardBench submission-bundle verifier: `npm run bench:guard:bundle:verify`.
|
|
41
|
-
- Verified GuardBench leaderboard builder: `npm run bench:guard:leaderboard`.
|
|
42
|
-
- Adapter registry: `benchmarks/adapters/registry.json`.
|
|
43
|
-
- Adapter registry validator: `npm run bench:guard:adapter-registry:validate`.
|
|
44
|
-
- Adapter author kit: `benchmarks/adapter-kit.mjs`.
|
|
45
|
-
- Fast adapter module validator: `npm run bench:guard:adapter-module:validate`.
|
|
46
|
-
- External adapter self-test: `npm run bench:guard:adapter-self-test`.
|
|
47
|
-
- Saved adapter self-test validator: `npm run bench:guard:adapter-self-test:validate`.
|
|
48
|
-
- GuardBench publication artifact verifier: `npm run bench:guard:publication:verify`.
|
|
49
|
-
- External adapter dry-run matrix: `npm run bench:guard:external:dry-run`.
|
|
50
|
-
- External evidence verifier: `npm run bench:guard:external:evidence`.
|
|
51
|
-
- Strict external evidence gate for credentialed runs: `npm run bench:guard:external:evidence:strict`.
|
|
52
|
-
- Local comparative GuardBench runner with a passing 10-scenario Audrey Guard check via `npm run bench:guard:check`.
|
|
53
|
-
- Strict external GuardBench adapter contract via `node benchmarks/guardbench.js --adapter ./adapter.mjs --check`.
|
|
54
|
-
- External adapters: `benchmarks/adapters/mem0-platform.mjs` (requires runtime `MEM0_API_KEY`) and `benchmarks/adapters/zep-cloud.mjs` (requires runtime `ZEP_API_KEY`).
|
|
55
|
-
- External evidence-bundle runners: `npm run bench:guard:mem0` and `npm run bench:guard:zep` with dry-run metadata capture.
|
|
56
|
-
- Paper metric sync command: `npm run paper:sync`.
|
|
57
|
-
- Paper claim verifier: `npm run paper:claims`.
|
|
58
|
-
- Publication pack verifier: `npm run paper:publication-pack`.
|
|
59
|
-
- arXiv source-package generator: `npm run paper:arxiv`.
|
|
60
|
-
- arXiv source-package verifier: `npm run paper:arxiv:verify`.
|
|
61
|
-
- arXiv compile-report gate: `npm run paper:arxiv:compile`.
|
|
62
|
-
- Strict arXiv compile gate: `npm run paper:arxiv:compile:strict`.
|
|
63
|
-
- Browser launch-plan verifier: `npm run paper:launch-plan`.
|
|
64
|
-
- Browser launch-results verifier: `npm run paper:launch-results`.
|
|
65
|
-
- Strict browser launch-results gate: `npm run paper:launch-results:strict`.
|
|
66
|
-
- Paper submission-bundle generator: `npm run paper:bundle`.
|
|
67
|
-
- Paper submission-bundle verifier: `npm run paper:bundle:verify`.
|
|
68
|
-
- Paper artifact verifier: `npm run paper:verify`.
|
|
69
|
-
- Pending-aware 1.0 release readiness verifier: `npm run release:readiness`.
|
|
70
|
-
- Strict 1.0 release readiness verifier: `npm run release:readiness:strict`.
|
|
71
|
-
- Source-control release-state check inside `npm run release:readiness`.
|
|
72
|
-
- Live remote-head verification inside `npm run release:readiness`.
|
|
73
|
-
- npm registry/auth readiness check inside `npm run release:readiness`.
|
|
74
|
-
- Dry-run release cut planner: `npm run release:cut:plan`.
|
|
75
|
-
- Final release cut writer: `npm run release:cut:apply`.
|
|
76
|
-
- Python package release verifier: `npm run python:release:check`.
|
|
77
|
-
- Paper-aware release gate: `npm run release:gate:paper`.
|
|
78
|
-
- Stage-A scope is explicit: implemented Audrey evidence, existing performance snapshot, current regression gate output, local comparative GuardBench output, and deterministic demo transcript.
|
|
79
|
-
|
|
80
|
-
## arXiv Preprint v1 Checklist
|
|
81
|
-
|
|
82
|
-
1. Generate the deterministic TeX source package with `npm run paper:arxiv`.
|
|
83
|
-
2. Verify the generated source package with `npm run paper:arxiv:verify`.
|
|
84
|
-
3. Run `npm run paper:arxiv:compile`; it compiles with `tectonic`, `latexmk`, `pdflatex`/`bibtex`, or `uvx tecto` plus a local bundle proxy when present and otherwise records a pending toolchain blocker in `docs/paper/output/arxiv-compile-report.json`.
|
|
85
|
-
4. Preview the arXiv-compiled PDF in the browser before pressing final submit.
|
|
86
|
-
5. Confirm the workshop variant stays under the target page limit. arXiv can run longer; workshops often need approximately 9 body pages.
|
|
87
|
-
6. Upload to arXiv with `cs.AI` as the primary category and `cs.CR` as secondary.
|
|
88
|
-
|
|
89
|
-
## Pre-Submission Status - 2026-05-08
|
|
90
|
-
|
|
91
|
-
- Done: README benchmark table values now match `benchmarks/snapshots/perf-0.22.2.json` and Ledger `E28` is resolved.
|
|
92
|
-
- Done: Ledger `E25` was re-checked against the current repeated-failure demo at `mcp-server/index.ts:825-879`.
|
|
93
|
-
- Done: primary-source URLs in `references.bib` were re-checked live on 2026-05-08; all 21 entries were reachable. The MCP schema reference now points at the current `2025-11-25` schema page.
|
|
94
|
-
- Update 2026-05-12: `npm run bench:guard:check` now runs a local comparative GuardBench suite across Audrey Guard, no-memory, recent-window, vector-only, and FTS-only adapters. Audrey Guard passes 10/10 scenarios. The harness supports external ESM adapters, the first concrete Mem0 Platform adapter is implemented, the output bundle includes an artifact redaction sweep, and `npm run bench:guard:mem0 -- --dry-run` captures the live-run command and metadata without storing credentials. A live Mem0 run is still pending a runtime key (Ledger: E46-E51).
|
|
95
|
-
- Update 2026-05-13: the adapter registry now includes Zep Cloud. `benchmarks/adapters/zep-cloud.mjs` creates a benchmark user/session, writes scenario memory with `memory.add`, searches user graph memory with `graph.search`, deletes the benchmark user during cleanup, and is covered by module validation plus mocked REST-flow tests. A live Zep run is still pending a runtime key (Ledger: E77).
|
|
96
|
-
- Update 2026-05-13: `npm run bench:guard:external:dry-run` writes non-secret dry-run metadata for every runtime-env adapter in the registry and is included in the release gates so live-run readiness cannot silently drift (Ledger: E78).
|
|
97
|
-
- Update 2026-05-13: the external dry-run matrix is now schema-bound by `benchmarks/schemas/guardbench-external-dry-run.schema.json`, written to `benchmarks/output/external/guardbench-external-dry-run.json`, included in package dry-run contents, and validated by `paper:verify` (Ledger: E79).
|
|
98
|
-
- Update 2026-05-13: `npm run bench:guard:publication:verify` now checks the schema-bound external dry-run matrix alongside registry, module, self-test, artifacts, submission bundle, and leaderboard, so the one-command public benchmark verifier covers every current GuardBench readiness artifact (Ledger: E80).
|
|
99
|
-
- Update 2026-05-13: `npm run bench:guard:external:evidence` writes a schema-bound external evidence verification report at `benchmarks/output/external/guardbench-external-evidence.json`, reports Mem0/Zep as pending when only dry-run metadata exists, checks saved metadata for runtime credential leaks, and ships a strict mode that fails until credentialed live bundles pass (Ledger: E81).
|
|
100
|
-
- Update 2026-05-13: `docs/paper/claim-register.json` now records supported and pending public claims, and `npm run paper:claims` verifies claim text, forbidden overclaims, evidence files, GuardBench artifacts, and the external-score Stage-B boundary before publication (Ledger: E82).
|
|
101
|
-
- Update 2026-05-13: `docs/paper/publication-pack.json` now carries arXiv, Hacker News, Reddit, X, and LinkedIn launch copy, and `npm run paper:publication-pack` verifies character limits, required entries, claim IDs, forbidden overclaims, pending Mem0/Zep boundary language, and secret leakage before browser-based posting (Ledger: E83).
|
|
102
|
-
- Update 2026-05-13: `npm run paper:bundle` now writes `docs/paper/output/submission-bundle/` with paper files, claim and publication registers, GuardBench outputs, schemas, README/package metadata, and `paper-submission-manifest.json` hashes; `npm run paper:bundle:verify` checks the manifest, file hashes, required files, GuardBench snapshot, claim verifier, and publication-pack verifier before upload (Ledger: E84).
|
|
103
|
-
- Update 2026-05-13: `docs/paper/browser-launch-plan.json` now maps the verified publication copy to arXiv, Hacker News, Reddit, X, and LinkedIn browser targets, records current source URLs/rules checked on 2026-05-13, requires human login/captcha/manual rule checks where needed, and is verified by `npm run paper:launch-plan` before browser posting (Ledger: E85).
|
|
104
|
-
- Update 2026-05-13: `npm run paper:arxiv` now writes a deterministic arXiv TeX source package under `docs/paper/output/arxiv/`; `npm run paper:arxiv:verify` checks the manifest, hashes, bibliography count, converted citations, missing bib IDs, seeded-secret redaction, and absence of local absolute paths before browser upload (Ledger: E86).
|
|
105
|
-
- Update 2026-05-13: `docs/paper/browser-launch-results.json` now records the post-submit state for arXiv, Hacker News, Reddit, X, and LinkedIn targets. `npm run paper:launch-results` validates target alignment, post-submit URL hosts, completed checklist fields, blocker text, and leakage boundaries while allowing pending targets; `npm run paper:launch-results:strict` fails until every target has a submitted, operator-verified public URL (Ledger: E87).
|
|
106
|
-
- Update 2026-05-13: public GuardBench and paper bundle artifacts now normalize repo-local paths before writing saved JSON/Markdown and run a local absolute-path sweep in `bench:guard:publication:verify`, `bench:guard:bundle:verify`, `paper:bundle:verify`, and `paper:verify` (Ledger: E88).
|
|
107
|
-
- Update 2026-05-13: the X launch copy now reserves `reservedUrlChars: 24` for the final public artifact URL using the `x-counting-characters` source, and `paper:launch-results` rejects submitted artifact-url targets unless `artifactUrl` records the public paper or repo URL (Ledger: E89).
|
|
108
|
-
- Update 2026-05-13: `npm run release:readiness` now emits the pending-aware Audrey 1.0 prompt-to-artifact checklist, while `npm run release:readiness:strict` fails until the target version, Python artifacts, npm registry/auth readiness, PyPI publish readiness, browser publication URLs, live Mem0/Zep evidence, and package publish readiness are complete (Ledger: E90).
|
|
109
|
-
- Update 2026-05-13: `npm audit --omit=dev --audit-level=moderate` is clean after refreshing the transitive `protobufjs`/`@protobufjs/*` lockfile chain used through `onnxruntime-web` (Ledger: E91).
|
|
110
|
-
- Update 2026-05-13: `npm run release:cut:plan` now previews the exact 1.0 version/changelog edits with publishable release notes instead of TODO scaffolding, and `npm run release:cut:apply` writes them only when the final cut is intentional (Ledger: E92).
|
|
111
|
-
- Update 2026-05-13: `npm run python:release:check` now builds the Python wheel/sdist, verifies package metadata and typed package contents, checks for local path leakage, and runs `twine check`; release gates run it before package publishing checks (Ledger: E93).
|
|
112
|
-
- Update 2026-05-13: `npm run release:readiness` now also checks source-control release state: branch, upstream, origin push remote, ahead/behind count, clean working tree, and `v1.0.0` tag placement (Ledger: E94).
|
|
113
|
-
- Update 2026-05-13: `npm run release:readiness` now checks `audrey@1.0.0` on the npm registry and requires `npm whoami` when the version is still unpublished, so package publish readiness cannot pass from the version bump alone (Ledger: E95).
|
|
114
|
-
- Update 2026-05-13: `npm run release:readiness` now verifies the live `origin/<branch>` head with `git ls-remote`, retries through Git's OpenSSL backend when Windows Schannel fails with `SEC_E_NO_CREDENTIALS`, and blocks when the local tracking ref is stale (Ledger: E96).
|
|
115
|
-
- Update 2026-05-13: `npm run paper:arxiv:compile` now attempts a local TeX compile with `tectonic`, `latexmk`, `pdflatex`/`bibtex`, or `uvx tecto` through a local bundle proxy, writes the schema-bound `docs/paper/output/arxiv-compile-report.json`, and lets the release-readiness gate keep missing TeX tooling as an explicit pending blocker instead of an undocumented host gap (Ledger: E97).
|
|
116
|
-
|
|
117
|
-
## Final Upload Checks
|
|
118
|
-
|
|
119
|
-
- Re-run URL verification if the arXiv upload moves to a later day.
|
|
120
|
-
- Run `npm run paper:sync` after benchmark outputs change.
|
|
121
|
-
- Run `npm run paper:claims` before public posts or submissions.
|
|
122
|
-
- Run `npm run paper:publication-pack` before using the launch copy.
|
|
123
|
-
- Run `npm run paper:arxiv` before arXiv browser upload.
|
|
124
|
-
- Run `npm run paper:arxiv:verify` before arXiv browser upload.
|
|
125
|
-
- Run `npm run paper:arxiv:compile` before arXiv browser upload; run `npm run paper:arxiv:compile:strict` only when a supported TeX toolchain is installed and compile proof must be complete.
|
|
126
|
-
- Run `npm run paper:launch-plan` before opening browser submission targets.
|
|
127
|
-
- Run `npm run paper:launch-results` after any browser submission or skipped/failed target update.
|
|
128
|
-
- Run `npm run paper:launch-results:strict` only when you intend to prove every launch target has a recorded public result.
|
|
129
|
-
- Run `npm run paper:bundle` after `paper:sync` and before upload.
|
|
130
|
-
- Run `npm run paper:bundle:verify` before uploading `docs/paper/output/submission-bundle/`.
|
|
131
|
-
- Run `npm run paper:verify` after any benchmark or paper edit.
|
|
132
|
-
- Run `npm run release:readiness` to capture the current 1.0 checklist without hiding pending publish blockers.
|
|
133
|
-
- Run `npm run release:readiness:strict` only after version bump, committed/tagged source-control state, live remote-head verification, Python artifacts, npm registry/auth readiness, PyPI publish readiness, browser public URLs, and live Mem0/Zep evidence are complete.
|
|
134
|
-
- Run `npm run release:cut:plan -- --target-version 1.0.0 --json` before applying the final version/changelog bump.
|
|
135
|
-
- Run `npm run release:cut:apply -- --target-version 1.0.0` only when the final 1.0 cut is intentional, then confirm strict readiness sees no placeholder changelog markers.
|
|
136
|
-
- Run `npm run python:release:check` after the final version cut and before PyPI upload.
|
|
137
|
-
- Run `npm audit --omit=dev --audit-level=moderate` before publishing package artifacts.
|
|
138
|
-
- Run `npm run release:gate:paper` before publishing or submitting public claims.
|
|
139
|
-
- Compile `docs/paper/output/arxiv/main.tex` with a local TeX toolchain before final arXiv upload; on hosts without `tectonic`, `latexmk`, `pdflatex`/`bibtex`, or `uvx tecto`, `paper:arxiv:compile` records the blocker and strict readiness remains pending.
|
|
140
|
-
- Upload to arXiv with `cs.AI` as the primary category and `cs.CR` as secondary.
|
|
141
|
-
- Use `publication-pack.json` for the first HN, Reddit, X, and LinkedIn drafts after `paper:publication-pack` passes.
|
|
142
|
-
- Keep the first X post's `reservedUrlChars` budget intact when adding the final public artifact URL.
|
|
143
|
-
- Use `browser-launch-plan.json` for target order, login/captcha handling, manual platform-rule checks, and post-submit URL capture.
|
|
144
|
-
- Update `browser-launch-results.json` with returned public URLs or blockers after each browser action.
|
|
145
|
-
- Use `docs/paper/output/submission-bundle/` as the machine-readable browser/upload package after `paper:bundle:verify` passes.
|
|
146
|
-
|
|
147
|
-
## Stage-B Work for v2
|
|
148
|
-
|
|
149
|
-
- Run `npm run bench:guard:mem0` with a runtime `MEM0_API_KEY` and publish the output bundle.
|
|
150
|
-
- Run `npm run bench:guard:zep` with a runtime `ZEP_API_KEY` and publish the output bundle.
|
|
151
|
-
- Run `npm run bench:guard:external:evidence:strict` after both live bundles exist.
|
|
152
|
-
- Add adapters for additional external memory systems using the current GuardBench ESM adapter contract and verify each one with `npm run bench:guard:adapter-self-test -- --adapter <adapter.mjs>`.
|
|
153
|
-
- Add external per-scenario confusion matrices, expanded multi-secret redaction sweeps, machine-provenance latency tables, and raw output artifacts.
|
|
154
|
-
- Strip evidence-ledger references from prose after the GuardBench claim set is stable.
|
|
155
|
-
- Compress the body to approximately 7,000 words for workshop submission.
|
|
156
|
-
|
|
157
|
-
## Suggested Venue Order
|
|
158
|
-
|
|
159
|
-
1. arXiv preprint: immediate, no review gate, stakes priority date.
|
|
160
|
-
2. NeurIPS workshop on foundation models for decision making, memory and agents, or LLM systems, depending on accepted workshop calls.
|
|
161
|
-
3. EMNLP Industry Track if GuardBench results land before the deadline.
|
|
162
|
-
4. SOSP/OSDI as a stretch target after full GuardBench results across external systems are ready.
|