gauntlet-guard 0.1.0__tar.gz

gauntlet_guard-0.1.0/PKG-INFO

@@ -0,0 +1,162 @@
+Metadata-Version: 2.3
+Name: gauntlet-guard
+Version: 0.1.0
+Summary: The anti-cheating layer for evaluating self-improving agents: sealed holdouts, persistence-leak guards, and keep/revert verdicts.
+Keywords: agents,evaluation,holdout,contamination,regression,llm
+Author: hamdani
+Author-email: hamdani <mhamdani35@gmail.com>
+License: MIT
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Topic :: Software Development :: Testing
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.11
+Project-URL: Homepage, https://github.com/dhanizael/gauntlet
+Project-URL: Repository, https://github.com/dhanizael/gauntlet
+Project-URL: Issues, https://github.com/dhanizael/gauntlet/issues
+Project-URL: Releases, https://github.com/dhanizael/gauntlet/releases
+Project-URL: Changelog, https://github.com/dhanizael/gauntlet/blob/main/CHANGELOG.md
+Description-Content-Type: text/markdown
+
+# gauntlet
+
+**The anti-cheating layer for evaluating self-improving agents.**
+
+> Your agent has memory now. That makes your agent evaluations quietly lie.
+
+Everyone evaluates their agents, skills, prompts, and AGENTS.md files. Almost nobody
+audits **the evaluator itself**. `gauntlet` closes that gap: sealed holdouts generated
+from private seeds, a persistence-leak guard that treats your agent's own memory files
+as the cheating channel they are, and blind paired-arm verdicts (`keep / revert /
+provisional`) with repeated-run statistics — because agents are stochastic and a
+single run proves nothing.
+
+## The problem, concretely
+
+You improved a skill. You A/B tested it on your eval tasks. It won. You shipped it.
+Two weeks later you find out:
+
+- the eval task text was paraphrased into `LESSONS.md` / `now.md` / a vector store by
+  the agent **during** an earlier run — the next runs studied for the test;
+- the winning arm won by noise (n=1, no spread reported);
+- your `provisional → verified` promotion was decided by vibes, not a gate.
+
+Benchmark contamination literature worries about *pretraining corpora* (n-grams,
+MinHash, memorized MMLU). None of it addresses the agent-native leak path:
+**persistent memory inside the evaluated system itself.** gauntlet's threat model
+starts there.
+
+## Quickstart
+
+```bash
+pipx install gauntlet-guard        # or: uv tool install gauntlet-guard
+# (PyPI publish is the next step after launch; day-one install from source:)
+git clone https://github.com/dhanizael/gauntlet && cd gauntlet \
+  && uv run gauntlet guard selftest   # prove the tool before you trust it
+
+# 1. Register a holdout instance (run this from the PRIVATE side, never in agent context)
+gauntlet manifest add ~/.private/eval/manifest.jsonl \
+    --instance sched-frostgate-0001 --seed s-77 \
+    task-input.txt answer-key.md
+
+# 2. After every eval session: scan the agent's persistent stores for leakage
+gauntlet guard scan ~/.private/eval/manifest.jsonl \
+    --store ~/agent-workspace/.agent-state/ \
+    --store ~/agent-workspace/LESSONS.md \
+    --store ~/agent-transcripts/
+# findings are reported by hash reference, never by content (see Design principles)
+# exit code 1 = leak found → retire the instance, regenerate from an unused seed
+
+# 3. Prove the tool works (synthetic planted leak + skip accounting):
+gauntlet guard selftest
+```
+
+## We ran our own gauntlet first
+
+The first real scan target was the author's actual agent infrastructure: 12 sealed
+task instances (the development pack of a private eval suite), scanned against the
+agent's persistent stores — long-term memory file, journal log, working-state
+directories, and evaluation run results. 271 files, **3.1 seconds**, zero runtime
+dependencies.
+
+```
+$ gauntlet guard scan manifest.jsonl --store LESSONS.md --store journal.log ...
+[NEAR ] results/…/L5-lens-r1/config.py          :: instance L5 (20/217 shingles, 9.22%)
+[NEAR ] results/…/L5-lens-r2/config.py          :: instance L5 (14/217 shingles, 6.45%)
+[TRACE] results/…/L5-raw-r1/config.py           :: instance L5 (9/217 shingles, 4.15%)
+[TRACE] results/…/L5-*/test_config_loader.py    :: instance L5 (6/217 shingles)  ×3
+[TRACE] results/…/L1-raw-r1/count_patterns.py   :: instance L1 (3/133 shingles, 2.26%)
+
+scanned 239 file(s), skipped 32 (17 binary, 15 too-large)
+7 finding(s).
+```
+
+The result is a perfect demonstration of why this tool exists: the agent's own memory
+files were **clean**, but task fixtures persisted verbatim into run-result directories —
+real, previously uncounted contamination vectors, exactly the failure mode a sealed
+holdout is designed to survive. Severity tracking is calibrated: shared boilerplate
+surfaced as `TRACE` (human look), fixture content surfaced as `NEAR` (retire it). A
+control scan of unrelated notes returned clean with no false positives, and the
+machine-readable report contains **zero task text** — verified by assertion in CI.
+
+## Design principles
+
+1. **The scanner must not leak.** Guard reports file, word-position, match counts and
+   *hash references only* — never the matched text. The report is safe to paste into
+   an issue even when the task is secret.
+2. **Manifests are private keys.** Task content lives only in the manifest file;
+   guard consumes it without echoing it. Keep manifests out of any agent-readable
+   surface (`chmod 700` the directory).
+3. **Deterministic first, judge second.** Hash/shingle evidence outranks any LLM
+   opinion; `guard` is fully executable in CI.
+4. **Leaks are lifecycle events, not warnings.** Detection → `retire` → regenerate
+   from an unused seed. A benchmark instance that leaked once is dead; say so in
+   the record.
+5. **Verdicts over scores.** `keep / revert / provisional` with ≥N repeated runs and
+   reported spread — an improvement that cannot survive the gauntlet is not an
+   improvement.
+
+## How detection works
+
+Task content is fingerprinted into 8-word normalized shingles (unicode-folded,
+punctuation-free, case-folded → SHA-256 prefixes). A store is scanned in a single
+pass; findings are classified:
+
+| severity | trigger | action |
+|---|---|---|
+| `exact` | normalized task text appears verbatim | retire + regenerate |
+| `near`  | ≥5% of task shingles matched (paraphrase, copy-edit) | retire + regenerate |
+| `trace` | ≥2 matched shingles, below the near threshold | human review |
+
+Oversize (>8 MiB default, configurable) and binary files are skipped **and reported** —
+a guard that silently hides what it didn't scan is the bug we found on day one of
+real-world use and refused to keep.
+
+## Status
+
+- `guard` + `manifest`: **shipped (v0.1)** — 12 tests, CI on pytest/ruff/ty, selftest
+  in the build pipeline
+- `run` (blind paired arms, isolated workspaces, environment drift fingerprinting): next
+- `grade` (verdict engine, ≥N repeats + spread, provisional→verified lifecycle): next
+- `holdout` (seed generators + retirement ledger, contract spec): next
+
+Built from a battle-tested private protocol: doctrine changes tagged `PROVISIONAL`
+until they beat the previous version on repeated, holdout-guarded evals — gauntlet
+is that gauntlet, made a tool.
+
+## Development
+
+```bash
+uv sync
+uv run pytest
+uv run ruff check && uv run ruff format --check
+uv run ty check src/
+uv run gauntlet guard selftest
+```
+
+## License
+
+MIT

gauntlet_guard-0.1.0/README.md

@@ -0,0 +1,139 @@
+# gauntlet
+
+**The anti-cheating layer for evaluating self-improving agents.**
+
+> Your agent has memory now. That makes your agent evaluations quietly lie.
+
+Everyone evaluates their agents, skills, prompts, and AGENTS.md files. Almost nobody
+audits **the evaluator itself**. `gauntlet` closes that gap: sealed holdouts generated
+from private seeds, a persistence-leak guard that treats your agent's own memory files
+as the cheating channel they are, and blind paired-arm verdicts (`keep / revert /
+provisional`) with repeated-run statistics — because agents are stochastic and a
+single run proves nothing.
+
+## The problem, concretely
+
+You improved a skill. You A/B tested it on your eval tasks. It won. You shipped it.
+Two weeks later you find out:
+
+- the eval task text was paraphrased into `LESSONS.md` / `now.md` / a vector store by
+  the agent **during** an earlier run — the next runs studied for the test;
+- the winning arm won by noise (n=1, no spread reported);
+- your `provisional → verified` promotion was decided by vibes, not a gate.
+
+Benchmark contamination literature worries about *pretraining corpora* (n-grams,
+MinHash, memorized MMLU). None of it addresses the agent-native leak path:
+**persistent memory inside the evaluated system itself.** gauntlet's threat model
+starts there.
+
+## Quickstart
+
+```bash
+pipx install gauntlet-guard        # or: uv tool install gauntlet-guard
+# (PyPI publish is the next step after launch; day-one install from source:)
+git clone https://github.com/dhanizael/gauntlet && cd gauntlet \
+  && uv run gauntlet guard selftest   # prove the tool before you trust it
+
+# 1. Register a holdout instance (run this from the PRIVATE side, never in agent context)
+gauntlet manifest add ~/.private/eval/manifest.jsonl \
+    --instance sched-frostgate-0001 --seed s-77 \
+    task-input.txt answer-key.md
+
+# 2. After every eval session: scan the agent's persistent stores for leakage
+gauntlet guard scan ~/.private/eval/manifest.jsonl \
+    --store ~/agent-workspace/.agent-state/ \
+    --store ~/agent-workspace/LESSONS.md \
+    --store ~/agent-transcripts/
+# findings are reported by hash reference, never by content (see Design principles)
+# exit code 1 = leak found → retire the instance, regenerate from an unused seed
+
+# 3. Prove the tool works (synthetic planted leak + skip accounting):
+gauntlet guard selftest
+```
+
+## We ran our own gauntlet first
+
+The first real scan target was the author's actual agent infrastructure: 12 sealed
+task instances (the development pack of a private eval suite), scanned against the
+agent's persistent stores — long-term memory file, journal log, working-state
+directories, and evaluation run results. 271 files, **3.1 seconds**, zero runtime
+dependencies.
+
+```
+$ gauntlet guard scan manifest.jsonl --store LESSONS.md --store journal.log ...
+[NEAR ] results/…/L5-lens-r1/config.py          :: instance L5 (20/217 shingles, 9.22%)
+[NEAR ] results/…/L5-lens-r2/config.py          :: instance L5 (14/217 shingles, 6.45%)
+[TRACE] results/…/L5-raw-r1/config.py           :: instance L5 (9/217 shingles, 4.15%)
+[TRACE] results/…/L5-*/test_config_loader.py    :: instance L5 (6/217 shingles)  ×3
+[TRACE] results/…/L1-raw-r1/count_patterns.py   :: instance L1 (3/133 shingles, 2.26%)
+
+scanned 239 file(s), skipped 32 (17 binary, 15 too-large)
+7 finding(s).
+```
+
+The result is a perfect demonstration of why this tool exists: the agent's own memory
+files were **clean**, but task fixtures persisted verbatim into run-result directories —
+real, previously uncounted contamination vectors, exactly the failure mode a sealed
+holdout is designed to survive. Severity tracking is calibrated: shared boilerplate
+surfaced as `TRACE` (human look), fixture content surfaced as `NEAR` (retire it). A
+control scan of unrelated notes returned clean with no false positives, and the
+machine-readable report contains **zero task text** — verified by assertion in CI.
+
+## Design principles
+
+1. **The scanner must not leak.** Guard reports file, word-position, match counts and
+   *hash references only* — never the matched text. The report is safe to paste into
+   an issue even when the task is secret.
+2. **Manifests are private keys.** Task content lives only in the manifest file;
+   guard consumes it without echoing it. Keep manifests out of any agent-readable
+   surface (`chmod 700` the directory).
+3. **Deterministic first, judge second.** Hash/shingle evidence outranks any LLM
+   opinion; `guard` is fully executable in CI.
+4. **Leaks are lifecycle events, not warnings.** Detection → `retire` → regenerate
+   from an unused seed. A benchmark instance that leaked once is dead; say so in
+   the record.
+5. **Verdicts over scores.** `keep / revert / provisional` with ≥N repeated runs and
+   reported spread — an improvement that cannot survive the gauntlet is not an
+   improvement.
+
+## How detection works
+
+Task content is fingerprinted into 8-word normalized shingles (unicode-folded,
+punctuation-free, case-folded → SHA-256 prefixes). A store is scanned in a single
+pass; findings are classified:
+
+| severity | trigger | action |
+|---|---|---|
+| `exact` | normalized task text appears verbatim | retire + regenerate |
+| `near`  | ≥5% of task shingles matched (paraphrase, copy-edit) | retire + regenerate |
+| `trace` | ≥2 matched shingles, below the near threshold | human review |
+
+Oversize (>8 MiB default, configurable) and binary files are skipped **and reported** —
+a guard that silently hides what it didn't scan is the bug we found on day one of
+real-world use and refused to keep.
+
+## Status
+
+- `guard` + `manifest`: **shipped (v0.1)** — 12 tests, CI on pytest/ruff/ty, selftest
+  in the build pipeline
+- `run` (blind paired arms, isolated workspaces, environment drift fingerprinting): next
+- `grade` (verdict engine, ≥N repeats + spread, provisional→verified lifecycle): next
+- `holdout` (seed generators + retirement ledger, contract spec): next
+
+Built from a battle-tested private protocol: doctrine changes tagged `PROVISIONAL`
+until they beat the previous version on repeated, holdout-guarded evals — gauntlet
+is that gauntlet, made a tool.
+
+## Development
+
+```bash
+uv sync
+uv run pytest
+uv run ruff check && uv run ruff format --check
+uv run ty check src/
+uv run gauntlet guard selftest
+```
+
+## License
+
+MIT

gauntlet_guard-0.1.0/pyproject.toml

@@ -0,0 +1,88 @@
+[project]
+name = "gauntlet-guard"
+version = "0.1.0"
+description = "The anti-cheating layer for evaluating self-improving agents: sealed holdouts, persistence-leak guards, and keep/revert verdicts."
+readme = "README.md"
+requires-python = ">=3.11"
+keywords = [
+    "agents",
+    "evaluation",
+    "holdout",
+    "contamination",
+    "regression",
+    "llm",
+]
+dependencies = []
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Environment :: Console",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "Programming Language :: Python :: 3 :: Only",
+    "Topic :: Software Development :: Testing",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+
+[[project.authors]]
+name = "hamdani"
+email = "mhamdani35@gmail.com"
+
+[project.license]
+text = "MIT"
+
+[project.urls]
+Homepage = "https://github.com/dhanizael/gauntlet"
+Repository = "https://github.com/dhanizael/gauntlet"
+Issues = "https://github.com/dhanizael/gauntlet/issues"
+Releases = "https://github.com/dhanizael/gauntlet/releases"
+Changelog = "https://github.com/dhanizael/gauntlet/blob/main/CHANGELOG.md"
+
+[project.scripts]
+gauntlet = "gauntlet.cli:main"
+
+[build-system]
+requires = ["uv_build>=0.12.13,<0.13.0"]
+build-backend = "uv_build"
+
+[tool.uv.build-backend]
+module-root = "src"
+module-name = "gauntlet"
+
+[tool.ruff]
+line-length = 100
+target-version = "py311"
+
+[tool.ruff.lint]
+select = ["ALL"]
+ignore = [
+    "D",
+    "COM812",
+    "ISC001",
+    "TRY003",
+    "EM101",
+    "EM102",
+    "S",
+    "FBT001",
+    "FBT002",
+    "PLR2004",
+    "C901",
+    "T201",
+    "CPY001",
+    "TC001",
+    "TC003",
+    "PLC0415",
+]
+
+[tool.ruff.lint.per-file-ignores]
+"tests/*" = ["ALL"]
+
+[tool.pytest.ini_options]
+addopts = "-q"
+testpaths = ["tests"]
+
+[dependency-groups]
+dev = [
+    "pytest>=9.1.1",
+    "ruff>=0.16.8",
+    "ty>=0.0.81",
+]

gauntlet_guard-0.1.0/pyproject.toml.orig

@@ -0,0 +1,62 @@
+[project]
+name = "gauntlet-guard"
+version = "0.1.0"
+description = "The anti-cheating layer for evaluating self-improving agents: sealed holdouts, persistence-leak guards, and keep/revert verdicts."
+readme = "README.md"
+authors = [
+    { name = "hamdani", email = "mhamdani35@gmail.com" }
+]
+requires-python = ">=3.11"
+license = { text = "MIT" }
+keywords = ["agents", "evaluation", "holdout", "contamination", "regression", "llm"]
+dependencies = []
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Environment :: Console",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "Programming Language :: Python :: 3 :: Only",
+    "Topic :: Software Development :: Testing",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+
+[project.urls]
+Homepage = "https://github.com/dhanizael/gauntlet"
+Repository = "https://github.com/dhanizael/gauntlet"
+Issues = "https://github.com/dhanizael/gauntlet/issues"
+Releases = "https://github.com/dhanizael/gauntlet/releases"
+Changelog = "https://github.com/dhanizael/gauntlet/blob/main/CHANGELOG.md"
+
+[project.scripts]
+gauntlet = "gauntlet.cli:main"
+
+[build-system]
+requires = ["uv_build>=0.12.13,<0.13.0"]
+build-backend = "uv_build"
+
+[tool.uv.build-backend]
+module-root = "src"
+module-name = "gauntlet"
+
+[dependency-groups]
+dev = [
+    "pytest>=9.1.1",
+    "ruff>=0.16.8",
+    "ty>=0.0.81",
+]
+
+[tool.ruff]
+line-length = 100
+target-version = "py311"
+
+[tool.ruff.lint]
+select = ["ALL"]
+ignore = ["D", "COM812", "ISC001", "TRY003", "EM101", "EM102", "S", "FBT001", "FBT002",
+           "PLR2004", "C901", "T201", "CPY001", "TC001", "TC003", "PLC0415"]
+
+[tool.ruff.lint.per-file-ignores]
+"tests/*" = ["ALL"]
+
+[tool.pytest.ini_options]
+addopts = "-q"
+testpaths = ["tests"]

gauntlet_guard-0.1.0/src/gauntlet/__init__.py

@@ -0,0 +1,3 @@
+"""gauntlet — the anti-cheating layer for evaluating self-improving agents."""
+
+__version__ = "0.1.0"

gauntlet_guard-0.1.0/src/gauntlet/cli.py

@@ -0,0 +1,159 @@
+"""gauntlet CLI — sealed holdouts, persistence-leak guards, keep/revert verdicts.
+
+v0 commands:
+  gauntlet manifest add  MANIFEST --instance ID --seed S FILES...
+  gauntlet manifest list MANIFEST
+  gauntlet manifest retire MANIFEST --instance ID
+  gauntlet guard scan    MANIFEST --store PATH [--store PATH] [--json OUT]
+  gauntlet guard selftest
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+import tempfile
+from pathlib import Path
+
+from .guard import finding_to_dict, scan_store
+from .manifest import add_instance, load_manifest, retire_instance
+
+BANNER = "gauntlet 0.1.0 — the anti-cheating layer for evaluating self-improving agents"
+
+
+def main(argv: list[str] | None = None) -> int:
+    ap = argparse.ArgumentParser(prog="gauntlet", description=BANNER)
+    sub = ap.add_subparsers(dest="cmd", required=True)
+
+    m = sub.add_parser("manifest", help="manage instance manifests (SECRET — private side only)")
+    msub = m.add_subparsers(dest="mcmd", required=True)
+    ma = msub.add_parser("add", help="register a task instance from its files")
+    ma.add_argument("manifest", type=Path)
+    ma.add_argument("--instance", required=True)
+    ma.add_argument("--seed", default="")
+    ma.add_argument("files", type=Path, nargs="+")
+    ml = msub.add_parser("list", help="list instances (metadata only, no content)")
+    ml.add_argument("manifest", type=Path)
+    mr = msub.add_parser("retire", help="mark an instance retired (its content leaked)")
+    mr.add_argument("manifest", type=Path)
+    mr.add_argument("--instance", required=True)
+
+    g = sub.add_parser("guard", help="scan agent persistent stores for task leakage")
+    gsub = g.add_subparsers(dest="gcmd", required=True)
+    gs = gsub.add_parser("scan", help="scan stores against a manifest")
+    gs.add_argument("manifest", type=Path)
+    gs.add_argument("--store", type=Path, action="append", required=True)
+    gs.add_argument("--min-overlap", type=int, default=2)
+    gs.add_argument("--max-bytes", type=int, default=8 * 1024 * 1024)
+    gs.add_argument("--json", type=Path, default=None)
+    gsub.add_parser("selftest", help="plant a synthetic leak and prove it is caught")
+
+    args = ap.parse_args(argv)
+
+    if args.cmd == "manifest" and args.mcmd == "add":
+        rec = add_instance(args.manifest, args.instance, args.seed, args.files)
+        print(f"registered {rec.instance} ({len(rec.shingles)} shingles, seed={rec.seed!r})")
+        return 0
+    if args.cmd == "manifest" and args.mcmd == "list":
+        for r in load_manifest(args.manifest):
+            print(
+                f"{r.instance:24s} seed={r.seed:12s} retired={r.retired} shingles={len(r.shingles)}"
+            )
+        return 0
+    if args.cmd == "manifest" and args.mcmd == "retire":
+        ok = retire_instance(args.manifest, args.instance)
+        print("retired" if ok else "instance not found (or already retired)")
+        return 0 if ok else 1
+    if args.cmd == "guard" and args.gcmd == "scan":
+        return _guard_scan(args)
+    if args.cmd == "guard" and args.gcmd == "selftest":
+        return _guard_selftest()
+    return 2
+
+
+def _guard_scan(args: argparse.Namespace) -> int:
+    records = load_manifest(args.manifest)
+    result = scan_store(records, args.store, args.min_overlap, args.max_bytes)
+    for f in result.findings:
+        print(
+            f"[{f.severity.upper():5s}] {f.store_file} :: instance {f.instance} "
+            f"({f.matched_shingles}/{f.total_shingles} shingles, {f.overlap_pct}% overlap, "
+            f"word@{f.first_word_pos}, ref {f.content_ref})"
+        )
+    if args.json:
+        args.json.write_text(json.dumps([finding_to_dict(f) for f in result.findings], indent=1))
+    reasons: dict[str, int] = {}
+    for _, why in result.skipped:
+        reasons[why] = reasons.get(why, 0) + 1
+    skip_note = ""
+    if result.skipped:
+        breakdown = ", ".join(f"{n} {k}" for k, n in sorted(reasons.items()))
+        skip_note = f", skipped {len(result.skipped)} ({breakdown})"
+    print(f"\nscanned {result.scanned} file(s){skip_note}")
+    if not result.findings:
+        print("clean: no task content detected in stores")
+        return 0
+    print(
+        f"{len(result.findings)} finding(s). "
+        "Retire leaked instances and regenerate from unused seeds."
+    )
+    return 1
+
+
+def _guard_selftest() -> int:
+    """Synthetic end-to-end proof: plant a paraphrased leak next to one clean note,
+    plus one oversized and one binary file that also contain the leak, and assert
+    detection, redaction, no false positives, and honest skip accounting."""
+    with tempfile.TemporaryDirectory() as td:
+        tdp = Path(td)
+        task = tdp / "task.txt"
+        task.write_text(
+            "A logistics firm must schedule exactly nine trucks across four docks under the "
+            "frostgate constraint: no dock may receive two consecutive refrigerated trucks, "
+            "and the vermilion manifest must depart before the azure convoy clears weighbridge "
+            "seven. Produce the minimal feasible ordering and prove minimality."
+        )
+        store = tdp / "agent_memory"
+        store.mkdir()
+        leak_text = (
+            "# Lessons\n- the frostgate constraint (no dock may receive two consecutive "
+            "refrigerated trucks, and the vermilion manifest must depart before the azure "
+            "convoy clears weighbridge seven) keeps biting us on scheduling tasks\n"
+            "- unrelated: prefer uv over pip\n"
+        )
+        (store / "LESSONS.md").write_text(leak_text)
+        (store / "clean_note.md").write_text("# Notes\n- always pin tool versions in CI\n")
+        (store / "huge_memory.md").write_text(leak_text + "x" * (4 * 1024 * 1024))
+        (store / "blob.bin").write_bytes(b"\x00\x01\x02" + leak_text.encode() + b"\x00")
+
+        mf = tdp / "manifest.jsonl"
+        add_instance(mf, "synthetic-1", "seed-42", [task])
+        result = scan_store(load_manifest(mf), [store], max_bytes=2 * 1024 * 1024)
+        findings = result.findings
+        by_file = {Path(f.store_file).name: f for f in findings}
+        assert "LESSONS.md" in by_file, "selftest FAILED: planted leak not detected"
+        assert "clean_note.md" not in by_file, "selftest FAILED: false positive"
+        assert "huge_memory.md" not in by_file, "selftest FAILED: oversize file was scanned"
+        assert "blob.bin" not in by_file, "selftest FAILED: binary file was scanned"
+        skip_reasons = {Path(p).name: why for p, why in result.skipped}
+        assert skip_reasons.get("huge_memory.md") == "too-large", (
+            "selftest FAILED: size skip unreported"
+        )
+        assert skip_reasons.get("blob.bin") == "binary", "selftest FAILED: binary skip unreported"
+        leak = by_file["LESSONS.md"]
+        print(
+            f"selftest PASS: caught planted leak in LESSONS.md [{leak.severity}] "
+            f"{leak.matched_shingles} shingles matched"
+        )
+        print(f"  skipped reported: {sorted(skip_reasons.items())}")
+        # verify redaction: no printable content in the serialized findings
+        blob = json.dumps([finding_to_dict(f) for f in findings]).lower()
+        for probe in ("frostgate", "vermilion manifest must depart", "logistics firm"):
+            assert probe not in blob, f"selftest FAILED: report contained task text {probe!r}"
+        print("selftest PASS: report is content-free (redaction holds)")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

gauntlet_guard-0.1.0/src/gauntlet/fingerprint.py

@@ -0,0 +1,39 @@
+"""Text fingerprinting: normalize, shingle, hash.
+
+Everything the scanner needs to compare stores against instances without
+ever treating task content as printable output.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import re
+import unicodedata
+
+SHINGLE_WORDS = 8
+
+
+def normalize(text: str) -> str:
+    """Aggressive but deterministic: unicode-fold, lowercase, strip punctuation."""
+    text = unicodedata.normalize("NFKC", text)
+    text = text.lower()
+    text = re.sub(r"[^a-z0-9\s]", " ", text)
+    return re.sub(r"\s+", " ", text).strip()
+
+
+def word_shingles(norm_text: str, k: int = SHINGLE_WORDS) -> list[tuple[int, str]]:
+    """Return (word_position, shingle_hash) for every k-word window of normalized text."""
+    words = norm_text.split()
+    out: list[tuple[int, str]] = []
+    for i in range(max(0, len(words) - k + 1)):
+        window = " ".join(words[i : i + k])
+        out.append((i, hashlib.sha256(window.encode()).hexdigest()[:16]))
+    return out
+
+
+def shingle_set(norm_text: str, k: int = SHINGLE_WORDS) -> set[str]:
+    return {h for _, h in word_shingles(norm_text, k)}
+
+
+def sha256_hex(data: bytes) -> str:
+    return hashlib.sha256(data).hexdigest()

gauntlet_guard-0.1.0/src/gauntlet/guard.py

@@ -0,0 +1,156 @@
+"""Leak guard: detect holdout task content inside agent persistent stores.
+
+The threat model is agent-native: agents now carry memory across sessions
+(LESSONS.md, now.md, journal logs, vector stores, transcripts). Holdout
+secrecy fails silently through these channels: a paraphrase saved today
+becomes an unfair advantage next week.
+
+This module scans persistent stores against an instance manifest and reports
+LEAKAGE WITHOUT EMITTING CONTENT: every finding is file, word-position,
+match counts, and shingle hash references — never the matched text. The
+scanner itself must not become a leak channel.
+
+Operational hardening (learned dogfooding on real agent stores): size caps,
+binary sniffing, and per-file shingle computation so scanning a workspace of
+multi-gigabyte transcripts terminates in bounded time.
+"""
+
+from __future__ import annotations
+
+import hashlib
+from collections import defaultdict
+from collections.abc import Iterator
+from dataclasses import asdict, dataclass, field
+from pathlib import Path
+
+from .fingerprint import normalize, word_shingles
+from .manifest import InstanceRecord
+
+SEVERITY_ORDER = {"trace": 0, "near": 1, "exact": 2}
+
+DEFAULT_SKIP_NAMES = {".git", ".venv", "node_modules", "__pycache__", ".uv-cache"}
+DEFAULT_MAX_BYTES = 8 * 1024 * 1024
+NEAR_OVERLAP_PCT = 5.0
+
+
+@dataclass(frozen=True)
+class Finding:
+    store_file: str
+    instance: str
+    seed: str
+    severity: str  # exact | near | trace
+    matched_shingles: int
+    total_shingles: int
+    overlap_pct: float
+    first_word_pos: int
+    evidence_shingle: str  # hash reference only — never printable content
+    content_ref: str  # double indirection: sha256 of the instance's full-content hash
+
+
+@dataclass
+class ScanResult:
+    findings: list[Finding]
+    scanned: int = 0
+    skipped: list[tuple[str, str]] = field(default_factory=list)
+
+
+def iter_store_files(
+    store_paths: list[Path], skip: set[str] = DEFAULT_SKIP_NAMES
+) -> Iterator[Path]:
+    for p in store_paths:
+        if p.is_file():
+            yield p
+        elif p.is_dir():
+            for f in sorted(p.rglob("*")):
+                if f.is_file() and not any(part in skip for part in f.parts):
+                    yield f
+
+
+def scan_store(
+    records: list[InstanceRecord],
+    store_paths: list[Path],
+    min_overlap_shingles: int = 2,
+    max_bytes: int = DEFAULT_MAX_BYTES,
+) -> ScanResult:
+    result = ScanResult(findings=[])
+    active = [r for r in records if not r.retired]
+    by_k: dict[int, list[InstanceRecord]] = defaultdict(list)
+    for r in active:
+        by_k[r.shingle_words].append(r)
+
+    for path in iter_store_files(store_paths):
+        try:
+            if path.stat().st_size > max_bytes:
+                result.skipped.append((str(path), "too-large"))
+                continue
+            with path.open("rb") as fh:
+                if b"\x00" in fh.read(1024):
+                    result.skipped.append((str(path), "binary"))
+                    continue
+            text = path.read_text(encoding="utf-8", errors="replace")
+        except OSError:
+            continue
+        result.scanned += 1
+        norm = normalize(text)
+        shingles_by_k = {k: word_shingles(norm, k) for k in by_k}
+        for rec in active:
+            f = _match_one(path, norm, shingles_by_k[rec.shingle_words], rec, min_overlap_shingles)
+            if f is not None:
+                result.findings.append(f)
+
+    result.findings.sort(key=lambda x: (-SEVERITY_ORDER[x.severity], -x.overlap_pct))
+    return result
+
+
+def _match_one(
+    path: Path,
+    norm: str,
+    shingles: list[tuple[int, str]],
+    rec: InstanceRecord,
+    min_hits: int,
+) -> Finding | None:
+    severity = "near"
+    first_pos = -1
+    matched: set[str] = set()
+
+    exact = (
+        rec.normalized is not None
+        and len(rec.normalized.split()) >= rec.shingle_words
+        and rec.normalized in norm
+    )
+    if exact:
+        severity = "exact"
+        first_pos = norm.find(rec.normalized)
+        matched.update(h for _, h in word_shingles(rec.normalized, rec.shingle_words))
+    else:
+        for pos, h in shingles:
+            if h in rec.shingles:
+                matched.add(h)
+                if first_pos < 0:
+                    first_pos = pos
+        if len(matched) < min_hits:
+            return None
+        severity = (
+            "near"
+            if len(matched) / max(1, len(rec.shingles)) * 100 >= NEAR_OVERLAP_PCT
+            else "trace"
+        )
+
+    total = max(1, len(rec.shingles))
+    overlap = len(matched) / total * 100.0
+    return Finding(
+        store_file=str(path),
+        instance=rec.instance,
+        seed=rec.seed,
+        severity=severity,
+        matched_shingles=len(matched),
+        total_shingles=total,
+        overlap_pct=round(overlap, 2),
+        first_word_pos=first_pos,
+        evidence_shingle=min(matched),
+        content_ref=hashlib.sha256(rec.full_sha256.encode()).hexdigest()[:16],
+    )
+
+
+def finding_to_dict(f: Finding) -> dict:
+    return asdict(f)

gauntlet_guard-0.1.0/src/gauntlet/manifest.py

@@ -0,0 +1,109 @@
+"""Instance manifests: the secret side of a gauntlet run.
+
+A manifest is JSONL, one record per holdout task instance. It is generated by
+the private side (never the evaluated agent) from a seed, and it holds enough
+fingerprint detail for `guard` to detect leaks WITHOUT being given the raw
+task files at scan time.
+
+Treat manifest files as private keys: readable by the grader side only.
+"""
+
+from __future__ import annotations
+
+import json
+from dataclasses import dataclass, replace
+from datetime import UTC, datetime
+from pathlib import Path
+
+from .fingerprint import SHINGLE_WORDS, normalize, sha256_hex, shingle_set
+
+
+@dataclass(frozen=True)
+class InstanceRecord:
+    instance: str
+    seed: str
+    created: str
+    retired: bool
+    full_sha256: str
+    normalized: str  # kept so guard can find substrings; NEVER print this
+    shingles: frozenset[str]
+    shingle_words: int
+
+
+def add_instance(
+    manifest_path: Path,
+    instance_id: str,
+    seed: str,
+    files: list[Path],
+) -> InstanceRecord:
+    blob = "\n".join(f.read_text(encoding="utf-8", errors="replace") for f in files)
+    norm = normalize(blob)
+    rec = InstanceRecord(
+        instance=instance_id,
+        seed=seed,
+        created=datetime.now(UTC).isoformat(timespec="seconds"),
+        retired=False,
+        full_sha256=sha256_hex(blob.encode()),
+        normalized=norm,
+        shingles=frozenset(shingle_set(norm)),
+        shingle_words=SHINGLE_WORDS,
+    )
+    _append(manifest_path, rec)
+    return rec
+
+
+def load_manifest(manifest_path: Path) -> list[InstanceRecord]:
+    records = []
+    for line in manifest_path.read_text(encoding="utf-8").splitlines():
+        if line.strip():
+            d = json.loads(line)
+            records.append(
+                InstanceRecord(
+                    instance=d["instance"],
+                    seed=d["seed"],
+                    created=d["created"],
+                    retired=bool(d.get("retired", False)),
+                    full_sha256=d["full_sha256"],
+                    normalized=d["normalized"],
+                    shingles=frozenset(d["shingles"]),
+                    shingle_words=d.get("shingle_words", SHINGLE_WORDS),
+                )
+            )
+    return records
+
+
+def retire_instance(manifest_path: Path, instance_id: str) -> bool:
+    """Mark an instance retired (its content leaked); rewrite manifest atomically."""
+    records = load_manifest(manifest_path)
+    found = False
+    for i, r in enumerate(records):
+        if r.instance == instance_id and not r.retired:
+            records[i] = replace(r, retired=True)
+            found = True
+    if found:
+        tmp = manifest_path.with_suffix(".tmp")
+        with tmp.open("w", encoding="utf-8") as fh:
+            for r in records:
+                fh.write(_line(r))
+        tmp.replace(manifest_path)
+    return found
+
+
+def _append(manifest_path: Path, rec: InstanceRecord) -> None:
+    manifest_path.parent.mkdir(parents=True, exist_ok=True)
+    with manifest_path.open("a", encoding="utf-8") as fh:
+        fh.write(_line(rec))
+
+
+def _line(rec: InstanceRecord) -> str:
+    d = {
+        "instance": rec.instance,
+        "seed": rec.seed,
+        "created": rec.created,
+        "retired": rec.retired,
+        "full_sha256": rec.full_sha256,
+        "normalized": rec.normalized,
+        "shingles": sorted(rec.shingles),
+        "shingle_words": rec.shingle_words,
+    }
+    return json.dumps(d, sort_keys=True) + "\n"