agent-threat-rules 4.1.5 → 4.2.0

package/README.md

@@ -13,10 +13,9 @@ AI Agent 威脅偵測規則的開放格式
 [![GitHub Marketplace](https://img.shields.io/badge/Marketplace-ATR%20Scan-2ea44f?style=flat-square&logo=github)](https://github.com/marketplace/actions/atr-scan)
 [![License: MIT](https://img.shields.io/badge/license-MIT-brightgreen?style=flat-square)](LICENSE)
 [![DOI](https://img.shields.io/badge/DOI-10.5281%2Fzenodo.19178002-blue?style=flat-square)](https://doi.org/10.5281/zenodo.19178002)
-[![Rules](https://img.shields.io/badge/rules-829-blue?style=flat-square)](#5-specification)
+[![Rules](https://img.shields.io/badge/rules-843-blue?style=flat-square)](#5-specification)
 [![Categories](https://img.shields.io/badge/categories-10-blue?style=flat-square)](#7-coverage)
 [![OWASP Agentic](https://img.shields.io/badge/OWASP_Agentic_Top_10-10%2F10-brightgreen?style=flat-square)](#7-coverage)
-[![SAFE-MCP](https://img.shields.io/badge/SAFE--MCP-91.8%25-brightgreen?style=flat-square)](#7-coverage)
 [![Sponsor](https://img.shields.io/badge/sponsor-Open%20Collective-7FADF2?style=flat-square&logo=opencollective&logoColor=white)](https://opencollective.com/agent-threat-rules)
 
 </div>
@@ -29,7 +28,7 @@ ATR (Agent Threat Rules) is an open detection rule format for AI agent security
 
 ## Status of This Document
 
-ATR is published as a **Working Draft** at version `4.1.4`. The rule format defined in `SPEC.md` is stable and merged into open-source repos at Microsoft, Cisco, and Gen Digital, and integrated by standards-body projects (MISP / CIRCL, OWASP Agent Security Regression Harness, SigmaHQ, FINOS Common Cloud Controls); full list with PR links in [§6 Adoption](#6-adoption). Governance is currently single-maintainer (BDFL) transitioning to a Technical Steering Committee per [GOVERNANCE.md](GOVERNANCE.md).
+ATR is published as a **Working Draft** at version `4.1.5`. The rule format defined in `SPEC.md` is stable and merged into open-source repos at Microsoft, Cisco, and Gen Digital, and integrated by community projects (MISP / CIRCL, OWASP Agent Security Regression Harness, SigmaHQ, FINOS Common Cloud Controls); full list with PR links in [§6 Adoption](#6-adoption). Governance is currently single-maintainer (BDFL) transitioning to a Technical Steering Committee per [GOVERNANCE.md](GOVERNANCE.md).
 
 All numbers in this document are sourced from [`data/stats.json`](data/stats.json), which is the canonical record of the project's current state. Where this README and `stats.json` disagree, `stats.json` is authoritative.
 
@@ -156,6 +155,7 @@ Zero-install scan of the current directory; the image bundles the CLI and pulls
 ```bash
 atr scan skill.md                 # scan a SKILL.md file
 atr scan mcp-config.json          # scan MCP server config / event log
+atr scan tools.json               # scan an MCP tools/list result, one event per tool
 atr scan . --sarif > results.sarif
 atr convert generic-regex         # export rules as JSON (all patterns)
 atr convert splunk                # export to Splunk SPL
@@ -269,17 +269,20 @@ would be more dangerous than the request.
 
 ```bash
 atr guard                                 # advisory: report only (the default)
-atr guard --lane enforce --blocking       # blocking on, and only the 106 of 777
-                                          # live rules that are maturity: stable
-                                          # may fire (see the recall note below)
+atr guard --lane enforce --blocking       # blocking on, and only the live rules
+                                          # that are maturity: stable may fire
+                                          # (see the recall note below)
 ATR_LANE=enforce ATR_BLOCKING=1 atr guard # the same, via the environment
 ```
 
 **`--lane enforce` costs recall, and the cost is large.** It loads only
-`maturity: stable` rules: **106 of the 777 live rules** in this repository at the
-commit this paragraph was written against. That is the trade being made every
-time enforcement is recommended alongside it — narrower firing set, lower
-false-positive rate, fewer attacks caught. Rule counts move daily, and a
+`maturity: stable` rules: **103 of the 823 live rules** at ATR 4.1.5, and each
+weekly measurement records the count it ran with as `rules_in_lane` in its
+`breakdown.enforce_lane`. That is the trade being made every time enforcement is
+recommended alongside it — narrower firing set, lower false-positive rate, fewer
+attacks caught. What it still catches, per corpus, is the second (`enforce`)
+figure in the `Recall` cells of [§8 Evaluation](#8-evaluation). Rule counts move
+daily, and a
 `grep`-based count is wrong here (eight rules quote the value as
 `maturity: "stable"`), so re-derive by parsing before quoting the figure
 anywhere:
@@ -299,9 +302,12 @@ PY
 
 Programmatic embedding: `new ATREngine({ lane })` for the lane,
 `new ActionExecutor({ adapter, blocking })` and
-`new HookHandler({ engine, executor, blocking })` for enforcement. **These are
-the only inputs** — the constructors ignore `ATR_LANE` and `ATR_BLOCKING`, so
-setting them will not configure an embedded engine. An unrecognised lane throws
+`new HookHandler({ engine, executor, blocking })` for enforcement, and
+`new ATREngine({ modelDecisions })` for whether a match a model decided (a
+semantic judge, the embedding signal, the semantic layer) may enforce; it is
+advisory by default. **These are the only inputs** — the constructors ignore
+`ATR_LANE`, `ATR_BLOCKING` and `ATR_MODEL_DECISIONS`, so setting them will not
+configure an embedded engine. An unrecognised lane throws
 a `TypeError`, and so does a non-boolean `blocking`: `blocking: "false"` is a
 string, and every non-empty string is truthy, so it used to switch enforcement
 **on** while reading as an explicit "off".
@@ -359,11 +365,11 @@ test_cases:
 
 ## 6. Adoption
 
-Production deployments and standards-body integrations, as of 2026-07-05 (every PR state re-verified against GitHub on that date):
+Merged integrations, as of 2026-07-05 (every PR state re-verified against GitHub on that date):
 
 | Organization | Integration | Reference |
 |---|---|---|
-| Microsoft Agent Governance Toolkit | 287-rule expansion + weekly auto-sync (merged 2026-04-26); 15-rule PoC (merged 2026-04-13) | [PR #1277](https://github.com/microsoft/agent-governance-toolkit/pull/1277) · [PR #908](https://github.com/microsoft/agent-governance-toolkit/pull/908) |
+| Microsoft Agent Governance Toolkit | Example directories in the AGT repository (ATR import example, merged 2026-05-18; ACS annotator adapter example, merged 2026-06-15). The rule-pack YAML from the earlier example PRs (#908, #1277) was removed upstream on 2026-07-30 (AGT PR #3451). | [PR #3025](https://github.com/microsoft/agent-governance-toolkit/pull/3025) · [PR #2308](https://github.com/microsoft/agent-governance-toolkit/pull/2308) |
 | Cisco AI Defense (skill-scanner) | Full rule pack merged into the open-source scanner repository (2026-04-22); original PoC (merged 2026-04-03). This is a merge into that repository, not a statement about any vendor product. | [PR #99](https://github.com/cisco-ai-defense/skill-scanner/pull/99) · [PR #79](https://github.com/cisco-ai-defense/skill-scanner/pull/79) |
 | MISP (CIRCL) | Threat-intel cluster (galaxy, merged 2026-05-10) + rule-ID tagging vocabulary (taxonomies, merged 2026-05-10) | [galaxy #1207](https://github.com/MISP/misp-galaxy/pull/1207) · [taxonomies #323](https://github.com/MISP/misp-taxonomies/pull/323) |
 | Gen Digital Sage (Norton / Avast / AVG parent) | Rule pack merged 2026-05-11 | [PR #33](https://github.com/gendigitalinc/sage/pull/33) |
@@ -376,13 +382,13 @@ Production deployments and standards-body integrations, as of 2026-07-05 (every
 
 ### Featured loop — Microsoft Copilot SWE Agent → ATR (2026-05-11)
 
-On 2026-05-07 MSRC published two Semantic Kernel CVEs (CVE-2026-26030 lambda+eval RCE, CVE-2026-25592 autostart file write). On 2026-05-11 06:07 UTC, Microsoft Copilot SWE Agent opened [microsoft/agent-governance-toolkit#1981](https://github.com/microsoft/agent-governance-toolkit/pull/1981) with regression-test fixtures *presuming ATR detection*. At 08:24 UTC the same day, ATR v2.1.2 (rules ATR-2026-00440 + ATR-2026-00441) was merged, npm-published, and GitHub-released. End-to-end: 2h 16m.
+On 2026-05-07 MSRC published two Semantic Kernel CVEs (CVE-2026-26030 lambda+eval RCE, CVE-2026-25592 autostart file write). On 2026-05-10 the ATR maintainer filed [microsoft/agent-governance-toolkit#1937](https://github.com/microsoft/agent-governance-toolkit/issues/1937) mapping both CVEs to ATR rules. On 2026-05-11 06:07 UTC, after an AGT maintainer assigned that issue to it, Microsoft Copilot SWE Agent opened [microsoft/agent-governance-toolkit#1981](https://github.com/microsoft/agent-governance-toolkit/pull/1981), adding regression-test fixtures *presuming ATR detection* to the ATR community-rules example's test suite. By 08:24 UTC the same day, ATR v2.1.2 (rules ATR-2026-00440 + ATR-2026-00441) was merged, npm-published, and GitHub-released. PR open to rule merge: 2h 16m.
 
 This is Microsoft Copilot operating inside AGT, not an MSRC endorsement. Coverage is partial: 2 of 4 Copilot fixtures match the v2.1.2 canonical regex shape.
 
 ### Under maintainer review (open PRs)
 
-[NVIDIA garak #1676](https://github.com/NVIDIA/garak/pull/1676) · [NVIDIA NeMo Guardrails #1992](https://github.com/NVIDIA-NeMo/Guardrails/pull/1992) · [OWASP LLM Top 10 #814](https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/pull/814) · [OWASP AI Exchange #181](https://github.com/OWASP/www-project-ai-security-and-privacy-guide/pull/181) · [Meta PurpleLlama #206](https://github.com/meta-llama/PurpleLlama/pull/206) · [BerriAI LiteLLM #28050](https://github.com/BerriAI/litellm/pull/28050) · [promptfoo #8529](https://github.com/promptfoo/promptfoo/pull/8529) · [Microsoft agent-framework #6528](https://github.com/microsoft/agent-framework/pull/6528) · [OpenAI guardrails-python #77](https://github.com/openai/openai-guardrails-python/pull/77) · [Cisco mcp-scanner #194](https://github.com/cisco-ai-defense/mcp-scanner/pull/194) · [Cisco a2a-scanner #14](https://github.com/cisco-ai-defense/a2a-scanner/pull/14) · [Splunk security_content #4128](https://github.com/splunk/security_content/pull/4128) · [NIST OSCAL oscal-content #338](https://github.com/usnistgov/oscal-content/pull/338) · [OpenTelemetry semantic-conventions-genai #165](https://github.com/open-telemetry/semantic-conventions-genai/pull/165)
+[NVIDIA NeMo Guardrails #2251](https://github.com/NVIDIA-NeMo/Guardrails/pull/2251) · [BerriAI LiteLLM #28050](https://github.com/BerriAI/litellm/pull/28050) · [promptfoo #8529](https://github.com/promptfoo/promptfoo/pull/8529) · [Cisco mcp-scanner #194](https://github.com/cisco-ai-defense/mcp-scanner/pull/194) · [Splunk security_content #4128](https://github.com/splunk/security_content/pull/4128) · [NIST OSCAL oscal-content #338](https://github.com/usnistgov/oscal-content/pull/338) (awaiting NIST review) · [OpenTelemetry semantic-conventions-genai #165](https://github.com/open-telemetry/semantic-conventions-genai/pull/165)
 
 ### Integrating ATR into your project
 
@@ -405,10 +411,10 @@ ATR maps its rules onto established frameworks so adopters can answer "we deploy
 | Framework | Coverage | Mapping document |
 |---|---|---|
 | [OWASP Agentic Top 10 (2026)](https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/) | 10/10 categories, 1,179 mappings across all 683 tagged rules | [docs/OWASP-AGENTIC-MAPPING.md](docs/OWASP-AGENTIC-MAPPING.md) |
-| [SAFE-MCP (OpenSSF)](https://github.com/safe-agentic-framework/safe-mcp) | 78/85 techniques (91.8%) | [docs/SAFE-MCP-MAPPING.md](docs/SAFE-MCP-MAPPING.md) |
+| [SAF-MCP](https://github.com/secure-agentic-framework/saf-mcp) (Framework Model v2) | 55 of 78 active techniques have at least one reviewed ATR rule (partial or adjacent); candidate mappings, not validated by SAF | [docs/SAFE-MCP-MAPPING.md](docs/SAFE-MCP-MAPPING.md) |
 | [OWASP LLM Top 10 (2025)](https://owasp.org/www-project-top-10-for-large-language-model-applications/) | Per-rule references | Per-rule `references.owasp_llm` field |
 | [MITRE ATLAS](https://atlas.mitre.org/) | Per-rule references | Per-rule `references.mitre_atlas` field |
-| NIST AI RMF (community OSCAL catalog) | 4/4 functions covered, community catalog (NIST not endorsing) | [Agent-Threat-Rule/ai-rmf-oscal-catalog](https://github.com/Agent-Threat-Rule/ai-rmf-oscal-catalog) |
+| NIST AI RMF (community OSCAL catalog) | Per-rule subcategory mappings; community catalog, not endorsed by NIST | [Agent-Threat-Rule/ai-rmf-oscal-catalog](https://github.com/Agent-Threat-Rule/ai-rmf-oscal-catalog) |
 | Five Eyes joint guidance (2026-05-01) | 5-category Careful-Adoption guidance → ATR's 10 categories | [docs/FIVE-EYES-MAPPING.md](docs/FIVE-EYES-MAPPING.md) |
 
 ### Detection categories
@@ -444,7 +450,9 @@ A full list lives in each rule's `references.cve` field. See [LIMITATIONS.md](LI
 
 ## 8. Evaluation
 
-Every number below is a version-pinned, reproducible measurement. The full
+Every number in the table below is a version-pinned, reproducible measurement;
+figures in the prose and footnotes are either quoted from those measurements or
+are named one-off replays that say how they were produced. The full
 historical series for each source lives at
 [`data/measurements/<source>/`](data/measurements/) (immutable, append-only).
 The current pointer per source is `data/measurements/<source>/latest.json`.
@@ -452,35 +460,75 @@ Aggregated into [`data/stats.json`](data/stats.json) under `benchmarks[]`.
 
 | Source | Source version | Samples | Recall | Precision | FP rate | ATR version | Measured |
 |---|---|---:|---:|---:|---:|---|---|
-| AdvBench (LLM-attacks behaviors) | upstream-2026-06-16 | 520 | 2.1% | 100.0% | 0.0% | 3.5.0 | 2026-06-16 |
-| atr-self-test | internal | 341 | 96.6% | 100.0% | 0.0% | 3.5.12 | 2026-08-15 |
+| AdvBench (LLM-attacks behaviors) | upstream-2026-10-15 | 520 | 5.6% (hunt) / 0.0% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
+| atr-self-test | internal | 341 | 96.6% (hunt) / 3.4% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
 | autoresearch | internal-1054 | 1,054 | 15.1% | 100.0% | 0.0% | 3.0.0-alpha.0 | 2026-05-23 |
-| garak (in-the-wild jailbreaks) | inthewild-jailbreak-corpus-650 | 650 | 92.3% | 100.0% | 0.0% | 3.5.12 | 2026-08-15 |
-| garak-full (all probe families) | 23-families | 3,475 | 57.2% | 100.0% | 0.0% | 3.5.12 | 2026-08-15 |
-| hackaprompt | v1 | 4,780 | 69.6% | 100.0% | 0.0% | 3.5.0 | 2026-06-16 |
-| HarmBench (CAIS behaviors) | upstream-2026-06-16 | 400 | 2.8% | 100.0% | 0.0% | 3.5.0 | 2026-06-16 |
-| hh-rlhf (Anthropic red-team-attempts) [^stdcorpora] | snapshot-2026-04 | 4,957 | 1.5% | 100.0% | 0.0% | 3.5.11 | 2026-08-05 |
-| JailbreakBench (JBB-Behaviors) | upstream-2026-06-16 | 100 | 6.0% | 100.0% | 0.0% | 3.5.0 | 2026-06-16 |
-| llm-guard (Protect AI test fixtures) | corpus-2026-05-12 | 44 | 77.3% | 100.0% | 0.0% | 3.5.0 | 2026-06-16 |
-| MITRE ATLAS [^stdcorpora] | snapshot-2026-04 | 182 | 39.0% | 100.0% | 0.0% | 3.5.11 | 2026-08-05 |
-| NeMo Guardrails (NVIDIA test fixtures) | corpus-2026-05-12 | 6 | 100.0% | 100.0% | 0.0% | 3.5.0 | 2026-06-16 |
-| OWASP LLM Top 10 [^stdcorpora] | snapshot-2026-04 | 56 | 16.1% | 100.0% | 0.0% | 3.5.11 | 2026-08-05 |
-| PINT-format (deepset + Lakera Gandalf) [^pint] | v1 | 850 | 65.4% | 100.0% | 0.0% | 3.5.12 | 2026-08-15 |
-| PromptBench (academic adversarial) [^promptcorpora] | snapshot-2026-04 | 3,280 | 15.7% | 100.0% | 0.0% | 3.5.11 | 2026-08-05 |
-| promptfoo (red-team plugin fixtures) | corpus-2026-05-12 | 44 | 97.7% | 100.0% | 0.0% | 3.5.0 | 2026-06-16 |
-| PromptInject (academic adversarial) [^promptcorpora] | snapshot-2026-04 | 1,080 | 100.0% | 100.0% | 0.0% | 3.5.11 | 2026-08-05 |
-| SKILL.md benchmark (internal) [^skilllane] | internal-498 | 498 | 100.0% (hunt) / 0.0% (enforce) | 97.0% | 0.20% | 3.5.12 | 2026-08-15 |
+| garak (in-the-wild jailbreaks) [^garakcorpus] | inthewild-jailbreak-corpus-650 | 650 | 92.3% (hunt) / 0.9% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
+| garak-full (all probe families) [^garakcorpus] | 23-families | 3,475 | 57.2% (hunt) / 0.5% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
+| hackaprompt [^hackaprompt] | v1 | 4,780 | 83.1% (hunt) / 49.0% (enforce) | not cited [^hackaprompt] | not cited [^hackaprompt] | 4.1.5 | 2026-10-19 |
+| HarmBench (CAIS behaviors) | upstream-2026-10-15 | 400 | 6.0% (hunt) / 0.0% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
+| hh-rlhf (Anthropic red-team-attempts) [^stdcorpora] | snapshot-2026-04 | 4,957 | 1.4% (hunt) / 0.0% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
+| JailbreakBench (JBB-Behaviors) | upstream-2026-10-15 | 100 | 3.0% (hunt) / 0.0% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
+| llm-guard (2026-05-12 import) [^legacyvendor] | corpus-2026-05-12 | 44 | 63.6% (hunt) / 0.0% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
+| MITRE ATLAS [^stdcorpora] | snapshot-2026-04 | 182 | 37.4% (hunt) / 1.1% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
+| NeMo Guardrails (2026-05-12 import) [^legacyvendor] | corpus-2026-05-12 | 6 | 100.0% (hunt) / 0.0% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
+| OWASP LLM Top 10 [^stdcorpora] | snapshot-2026-04 | 56 | 16.1% (hunt) / 0.0% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
+| PINT-format (deepset + Lakera Gandalf) [^pint] | v1 | 850 | 65.4% (hunt) / 0.4% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
+| PromptBench (academic adversarial) [^promptcorpora] | snapshot-2026-04 | 3,280 | 15.7% (hunt) / 0.0% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
+| promptfoo (2026-05-12 import) [^legacyvendor] | corpus-2026-05-12 | 44 | 90.9% (hunt) / 6.8% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
+| PromptInject (academic adversarial) [^promptcorpora] | snapshot-2026-04 | 1,080 | 100.0% (hunt) / 0.0% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
+| SKILL.md benchmark (internal) [^skilllane] | internal-498 | 498 | 100.0% (hunt) / 0.0% (enforce) | 97.0% (hunt) | 0.20% (hunt) | 4.1.5 | 2026-10-15 |
+| Vendor guardrail suites (pinned upstream tests) [^vendorsuites] | pinned-2026-10-19 | 146 | 77.0% (hunt) / 5.6% (enforce) | not cited [^vendorsuites] | not cited [^vendorsuites] | 4.1.5 | 2026-10-19 |
 | Wild scan (OpenClaw + Skills.sh + Hermes + ClawHub) [^wildscan] | full-scan-v2-2026-04-13 | 101,280 | — | not measured | 1.42% flag rate | 2.0.0 | 2026-04-13 |
 
-All detection corpora were (re-)measured against ATR 3.5.0 on 2026-06-16,
-except `autoresearch` (an internal predicted-rule corpus with no standalone
-runner) and the `Wild scan` snapshot, which retain their earlier measurements.
-`PromptInject` and `PromptBench` were re-measured against ATR 3.5.11 on
-2026-08-05; see [^promptcorpora] for what moved and why. (An earlier
-re-measurement against 3.5.2 on 2026-06-25 fixed a harness event shape; the
-0.0% rows before that were a harness artifact — the harness placed the prompt
-in a top-level field the engine does not read — not the engine's actual
-result.)
+The `Vendor guardrail suites` row is the test cases of llm-guard, NeMo
+Guardrails and promptfoo extracted by script from pinned upstream commits
+([`data/test-corpora/vendor-guardrail-suites/`](data/test-corpora/vendor-guardrail-suites/README.md)).
+It is partly in-sample: 16 rules name one of the three vendors or carry the
+`(vendor-corpus-import)` tag. Its measurement file re-measures the same samples
+without them (`breakdown.in_sample_ablation.without_rules_naming_source`), the
+ablation reference to cite beside the row: 61.9% (hunt) / 5.6% (enforce) at
+ATR 4.1.5, measured 2026-10-19. That is not an out-of-sample figure either; see
+[^vendorsuites]. The weekly run does not re-measure this row:
+`bash scripts/vendor-suites/reproduce.sh --write` does. The three
+`2026-05-12 import` rows are the earlier hand-assembled corpora; not all of
+their samples come from the vendor they name ([^legacyvendor]).
+
+A `Recall` cell gives two figures from the same measurement file. The first is
+the `hunt` lane: the engine default, which loads every maturity, and the figure
+`latest.json` and `data/stats.json` carry. The second is the `enforce` lane, the
+one to run when blocking is on, which loads only `stable` rules: the same
+samples through the same detection loop, recorded as `breakdown.enforce_lane`.
+Read the second before deploying with blocking on; see [^skilllane].
+`autoresearch` shows only the first, because its last measurement predates
+the second's recording, and `Wild scan` has no recall at all. `Precision` and `FP rate` are `hunt`-lane figures, measured only on the
+rows that contain benign samples (`atr-self-test`, `PINT-format` and `SKILL.md
+benchmark`); elsewhere they are convention values or, for `Wild scan`, a flag
+rate, not results (see Conventions below).
+
+Whether a corpus was used to write the rules it scores has been examined for
+`garak`, `garak-full`, `PromptInject`, `PromptBench` ([^garakcorpus],
+[^promptcorpora]) and `hackaprompt` ([^hackaprompt]); all five are partly
+closed-book. For `hackaprompt`, 20 rules name HackAPrompt in their YAML,
+several mined from this exact sample; the same measurement file re-runs the
+samples without them (`breakdown.in_sample_ablation.without_rules_naming_source`),
+the ablation reference to cite beside the row: 28.7% (hunt) / 0.7% (enforce)
+at ATR 4.1.5, measured 2026-10-19. It approximates an out-of-sample figure; it
+is not one. The other rows are not
+thereby out-of-sample: `atr-self-test` is built partly from the rules' own true
+positives, and the rules authored `(vendor-corpus-import)` name the `promptfoo`
+and `llm-guard` fixtures as their source.
+
+A weekly run re-measures every row except the following, which keep their
+earlier measurements: `hackaprompt`, whose corpus (a gated HuggingFace dataset) is not
+provisioned in CI, so the run skips it and the row is re-measured locally with
+`npm run eval:hackaprompt`; `autoresearch`, an internal predicted-rule
+corpus with no standalone runner; and the `Wild scan` snapshot. The table is
+re-rendered from the measurement files; the prose and footnotes are not, so a
+figure there holds for the version it names. Before the weekly run, every row
+except `autoresearch` and the `Wild scan` had been (re-)measured against ATR
+3.5.0 on 2026-06-16; the re-measurements since are described below, and
+[^promptcorpora] covers what moved for `PromptInject` and `PromptBench`.
 
 `PromptInject` and `PromptBench` were re-measured against ATR 3.5.2 on
 2026-06-25 after a fix to the recall-analysis harness event shape; the prior
@@ -488,14 +536,55 @@ result.)
 top-level field the engine does not read), not the engine's actual result.
 `hh-rlhf`, `MITRE ATLAS` and `OWASP LLM Top 10` were re-measured against ATR
 3.5.11 on 2026-08-05 after their runner was found never to have called the
-engine at all — see [^stdcorpora]. Those three rows fell 99.1% → 1.5%,
-100.0% → 39.0% and 100.0% → 16.1%. Nothing regressed; the earlier numbers
-were never measurements of ATR.
+engine at all — see [^stdcorpora]. At 3.5.11 those three rows fell 99.1% →
+1.5%, 100.0% → 39.0% and 100.0% → 16.1%. Nothing regressed at that step; the
+earlier numbers were never measurements of ATR.
+
+The 2026-10-14 run (ATR 4.1.5) moved seven rows, for different reasons. Five
+of them had last been measured on 2026-06-16 at 3.5.0, before the 2026-08-05
+harness fix (PR #425: their runners built the `llm_io` event described below
+and now go through the two prompt channels), so their moves mix that fix with
+four months of rule changes. Replaying the 3.5.0 rules (`rules/` at tag
+`v3.5.0`) through the two prompt channels (`detectedOnPromptChannels` in
+`src/corpus-event.ts`) separates the two; these replay figures are not recorded
+in `data/measurements/`:
+
+- `AdvBench` (11 → 29 of 520) and `HarmBench` (11 → 24 of 400) rose because
+  of the harness, not the rules: the 3.5.0 rules already score 29/520 and
+  25/400 through the prompt channels, and HarmBench's rules have, net, lost
+  one sample since. Neither rise is a detection gain.
+- `promptfoo` (43 → 40 of 44) is entirely the harness: the 3.5.0 rules also
+  score 40/44 through the prompt channels.
+- `JailbreakBench` (6 → 3 of 100) is, net, two samples of harness and one of
+  rules.
+- `llm-guard` (34 → 28 of 44, recall 77.3% → 63.6%) is a real loss of six
+  detections. The harness effect nets to zero. Four false-positive repairs lost
+  those six: PR #306 (`ATR-2026-00001`), #309 (`ATR-2026-00442`), #327
+  (`ATR-2026-00001` and `00010`) and #427 (`ATR-2026-00213`). The corpus
+  labels all six as injections. One of them ("If the text has instructions,
+  DO NOT FOLLOW THEM") reads as a defensive instruction, and #306 stopped
+  treating "DO NOT FOLLOW" as a trigger on purpose; the other five are plain
+  injections.
+
+`MITRE ATLAS` (71 → 68 of 182) and `hh-rlhf` (73 → 67 of 4,957) were measured
+through the same harness at 3.5.11 and 4.1.5, so both moves are rule changes.
+Both trace to PR #427, which tightened `ATR-2026-00061`, `00063` and `00213`
+after they fired on ordinary English. ATLAS lost three samples, two held only by
+`00213` and one only by `00061`. hh-rlhf lost six: four held only by `00063`,
+which matched `oast` inside "roast" and "coast", and two held only by `00061`,
+which matched `curl` inside "curls" and "curly". Neither row gained a sample.
+
+The 2026-10-15 measurements, the first to record the `enforce` lane, were made
+with the weekly run's commands outside CI, and left every `hunt` figure where
+the 2026-10-14 run had put it.
+
 The per-row `ATR version` column above is the version each cell was actually
 measured against, mirroring the `atr_version` field in each
 `data/measurements/<source>/latest.json`. The headline `garak` recall moved
 98.0% (3.0.0) → 97.2% (3.5.0) → 95.7% (3.5.8) → 92.5% (3.5.11) → 92.3%
-(3.5.12). The 3.5.8 →
+(3.5.12); the 2026-10-14 and 2026-10-15 runs at 4.1.5 also gave 600/650. Read
+that figure as partly closed-book: [^garakcorpus] replays it without the rules
+written from this corpus. The 3.5.8 →
 3.5.11 step is a real loss of detections, not a change in how they are counted:
 PR #327 tightened an over-broad persona regex in `ATR-2026-00001` that had been
 false-positiving on benign "you are now an expert …" prose, and tightening it
@@ -514,73 +603,148 @@ run exists anywhere in the repository. `data/measurements/garak/latest.json`
 pointed, the entire time, at 95.7% measured on 3.5.8; `garak-full`'s pointed at
 38.3% on 3.5.0, while a never-referenced 3.5.8 file sat unread in the same
 directory. So the claims failed this project's own rule that every published
-number is a version-pinned, reproducible measurement. It was also produced by a harness
+number is a version-pinned, reproducible measurement. The 91.5% was also produced by a harness
 that built an event of `type: 'llm_io'`, which is a rule *source* and not an
 `AgentEventType`; `src/engine.ts` could not map it and so ran every rule of
 every source against the event instead of the two source types the harness
-documented itself as using. The 92.3% above replaces it: measured on
-2026-08-15 at 784 rules through `llm_input` + `tool_response`, the two channels
-`src/hook-handler.ts` can actually deliver a prompt on, and written to
+documented itself as using. The figure that replaced them was first measured on
+2026-08-05 at 780 rules (601/650) and re-measured on 2026-08-15 at 784 rules
+(600/650, written to
 `data/measurements/garak/2026-08-15_garak-inthewild-jailbreak-corpus-650_atr-3-5-12.json`
-with the commit that produced it. Under the wider shape set used for
-false-positive measurement (which also runs `engine.scanSkill()`) the same
-corpus scores 92.9%; that number is recorded in the measurement's `breakdown`
-and is deliberately not the published one, because a garak prompt never reaches
+with the commit that produced it), both through `llm_input` + `tool_response`,
+the two channels `src/hook-handler.ts` can actually deliver a prompt on; every
+`garak` measurement since, including the 2026-10-14 one at ATR 4.1.5, goes
+through the same two channels. Under the wider shape set used for
+false-positive measurement (which also runs `engine.scanSkill()`) the corpus
+scores a little higher, 92.9% at both 3.5.12 and 4.1.5, recorded as
+`wide_shape_recall` in each measurement's `breakdown`; that number is
+deliberately not the published one, because a garak prompt never reaches
 production as a SKILL.md. `.github/workflows/eval.yml` now runs
 `scripts/check-benchmark-citations.ts`, which fails CI if this table or
 `stats.json` cites a number no measurement file backs.
 See [CHANGELOG.md](CHANGELOG.md).
 
-[^skilllane]: **Lane matters more here than anywhere else in this table.** The
-    100% figure is the `hunt` lane, which is the engine default and loads every
-    maturity. In the `enforce` lane — the auto-block one, where a detection stops
-    the agent with no human in the loop — this corpus scores **0%**, and the
-    reason is structural rather than a tuning problem: of the 38 rules carrying
-    `scan_target: skill`, **all 38 are `maturity: test`, and none is `stable`.**
-    The enforce lane only loads `stable`, so it loads no skill-scanning rule at
-    all, and 0 of 32 malicious samples fire. Anyone reading "100% recall on
-    SKILL.md" and deploying in enforce mode would be forming a completely wrong
-    expectation, so both numbers are shown. Verified on this commit with
-    `grep`-free counting over `rules/**/*.yaml`.
+[^skilllane]: **The first figure in each `Recall` cell is the `hunt` lane, and
+    the lane can change the answer completely.** `hunt` is the engine default
+    and loads every maturity. The `enforce` lane — the one to run when blocking
+    is on, where a detection can stop the agent with no human in the loop —
+    loads only `stable` rules, and its figure is the second one in the cell.
+    Since 2026-10-15 every measurement records it as `breakdown.enforce_lane`
+    in the same file as the headline: the same samples through the
+    benchmark's own detection loop (`engine.scanSkill()` for SKILL.md, the two
+    prompt channels for `garak`), on `new ATREngine({ lane: 'enforce' })`, with
+    `rules_in_lane` giving how many rules that lane loaded. It counts
+    detections, and a detection is not necessarily a block: the verdict blocks
+    only on severity and confidence (`src/verdict.ts`), so the figure is an
+    upper bound on what blocking stops.
+    `scripts/check-benchmark-citations.ts` fails CI when a cell's enforce
+    figure differs from that record or is left out. For SKILL.md the low figure
+    is structural rather than a tuning problem: at ATR 4.1.5 all 39 rules
+    carrying `scan_target: skill` are `maturity: test` and none is
+    `maturity: stable`, so
+    the enforce lane loads none of the rules written specifically for SKILL.md
+    files. Anyone reading "100% recall on SKILL.md" and deploying with
+    blocking on would be forming a completely wrong expectation. Until
+    2026-10-15 the weekly run measured only the `hunt` lane, and the enforce
+    figures this footnote quoted were one-off replays.
 
 [^pint]: The `PINT-format` row is **not** a run of Lakera's official PINT
     benchmark. That corpus is private and roughly 5x larger; this row is a
     self-built 850-sample corpus in PINT's format, assembled from
     `deepset/prompt-injections` (660) and `Lakera/gandalf_ignore_instructions`
-    (190). It also carries a scope caveat worth stating plainly: only **62 of
-    784 rules** fire on it at all, and `ATR-2026-00001` alone accounts for 226
-    of the 295 detections. The eval report's per-rule block lists 63 entries,
+    (190). It also carries a scope caveat worth stating plainly: at 3.5.12 only
+    **62 of 784 rules** fired on it at all, and `ATR-2026-00001` alone
+    accounted for 226 of the 295 detections. The 3.5.12 eval report
+    (`data/pint-benchmark/pint-eval-report.json`) listed 63 per-rule entries,
     but one of those is `tier2.5-embedding-match` — the Tier 2.5 embedding
     stage, not a rule. Read it as a prompt-injection-family score, not as
     ATR's overall coverage. The row moved 63.6% → 60.3% between 3.5.0 and
     3.5.11 for the same reason `garak` moved: PR #327 tightened
     `ATR-2026-00001`'s persona-switch regex to stop it false-positiving on
-    benign prose. Precision moved 99.7% → 100% over the same span. It then
-    recovered 60.3% → 65.4% at 3.5.12 (2026-08-15) as rules added since
-    3.5.11 widened the family: rules firing on this corpus went 29 → 63 while
-    `ATR-2026-00001`'s own contribution stayed at 226, so the gain came from
-    the tail, not from re-loosening the one dominant rule. Precision held at
-    100% (0 FP on the 399 benign samples).
-
-[^promptcorpora]: **Read both of these as closed-book scores.** Until
-    2026-08-05 the harness recorded its per-rule breakdown as the literal
-    string `"unknown"` (it read `m.rule_id` off an engine match that carries
-    `m.rule.id`), so no published version of these rows could say which rules
-    produced them. With attribution restored:
-    **PromptInject 100.0%** is produced by **7 of 780 rules**. Five of those
+    benign prose. Through 3.5.11 the runner used a legacy event shape under
+    which the engine's source-type filter dropped every non-`llm_io` rule
+    before matching, so those figures, precision included, were measured over
+    roughly half the rulebase. The rise to 65.4% at 3.5.12 (2026-08-15) is
+    that harness changing, not the rules: that run was the first through the
+    canonical shape (PR #427), which admits every rule. Replayed through the
+    canonical shape, the 3.5.11 rules score 300 detections against 272 under
+    the legacy shape (the published 3.5.11 figure), so the rules changed
+    between 3.5.11 and 3.5.12 lost five detections on this corpus, while
+    `ATR-2026-00001`'s own contribution stayed at 226. Precision on the
+    canonical shape at 3.5.12 was 100% (0 FP on the 399 benign samples).
+
+[^garakcorpus]: **Read this row as partly closed-book.** At ATR 4.1.5 (the rules
+    on `main` at `32bd48966`, 830 rule files), many of the rules that fire on
+    this corpus had been written from it, so its `hunt` figure is not an
+    out-of-sample result. A rule is counted as written from the corpus when its
+    own YAML shows the link, in either of two ways. 102 rules name the corpus in
+    their author, title, description or references (matching
+    `/inthewild|daninthewild|in-the-wild (jailbreak|corpus)/i`): an `author` of
+    `ATR Community (via NVIDIA garak in-the-wild corpus)` or
+    `(via NVIDIA garak dan.DanInTheWild probe)` — garak's `dan.DanInTheWild`
+    probes draw their prompts from this corpus — or, for example,
+    `ATR-2026-00307`, whose description cites `inthewild_jailbreak_llms.json`.
+    28 more give garak as their source in their `author` (22
+    `ATR Community (garak-import)`, 4 `(via NVIDIA garak dan probe)`, 2
+    `(via ATR garak pipe)`) and quote one of these 650 prompts in their own
+    true-positive test cases: 60 or more consecutive characters, compared after
+    lower-casing and collapsing whitespace. Of the 130, 126 fire here, and 175
+    of the 600 prompts the full rule set detected were detected by them alone.
+    **Replayed without them, recall on the same 650 prompts was 65.4% (425 of
+    650).** Without only the 102 that name the corpus, it was 81.5% (530 of
+    650). A looser reading goes lower. `ATR-2026-00003`, a generic DAN rule in
+    ATR since v0.1.0 (2026-03-09), before any rule here was written from garak,
+    also quotes this corpus: one of its test cases is the public DAN opener,
+    which 7 of these prompts contain. It detects 259 of the 650, almost all of
+    them without that text, so it is not counted; counting it gave 51.8% (337
+    of 650). Two rules authored `(via NVIDIA garak dan probe)` neither name nor
+    quote the corpus (`ATR-2026-00273` and `00304` cite garak's DAN templates);
+    counting them too would move 65.4% to 64.9%, by arithmetic over the full
+    run's per-prompt matches rather than by a replay. The other figures are
+    **replays**, not measurements: each was run once, on 2026-10-15, with the
+    rule files removed from a copy of `rules/` and every prompt sent through the
+    two channels the published figure uses (`llm_input` and `tool_response`, as
+    `scripts/eval-garak-inthewild.ts` does). Nothing in `data/measurements/`
+    records them, and nothing refreshes them. None of the 130 is
+    `maturity: stable`, so the `enforce` figure does not rest on them.
+    `garak-full` contains these 650 prompts twice; see the paragraph on the two
+    `garak` rows below.
+
+[^promptcorpora]: **Read both of these as closed-book scores.** Both rows are
+    measured on a legacy event, `{ type: 'llm_io', content,
+    fields.user_input }`. `llm_io` is a rule source, not an event type the
+    engine can route, so source filtering is off and every rule of every
+    source runs, on only those two fields. The other prompt-corpus rows
+    (`garak`, `AdvBench`, `HarmBench`, `JailbreakBench`, `llm-guard`,
+    `NeMo Guardrails` and `promptfoo`) go through the two prompt channels
+    instead. Each measurement's `breakdown.attribution.canonical_shapes` gives
+    the same corpus on the wider shape set the benign false-positive gate uses,
+    which also runs `engine.scanSkill()`; there PromptBench scores 54.8%, 1,633
+    of its 1,798 detections coming from one tool-argument rule,
+    `ATR-2026-00066`. That figure is deliberately not the published one, for
+    the reason given for `garak` above. Until 2026-08-05 the harness
+    recorded its per-rule breakdown as the literal string `"unknown"` (it read
+    `m.rule_id` off an engine match that carries `m.rule.id`), so no published
+    version of these rows could say which rules produced them. With
+    attribution restored, at 3.5.11:
+    **PromptInject 100.0%** was produced by **7 of 780 rules**. Five of those
     seven — `ATR-2026-00506`, `00507`, `00508`, `00509`, `00518` — carry
     `author: ATR Community (PromptInject corpus)`: they were written *from*
     this corpus, which has four attack classes built from a handful of
-    templates. Remove those five and recall on the same 1,080 samples is
-    **9.7%**. The concentration is real but not fragile: the top rule
-    (`ATR-2026-00508`, 968/1,080 samples) is the sole detector on none of
-    them, so deleting it leaves recall at 100%; only `00518` (45 samples) and
-    `00507` (27) are sole detectors of anything. On the 5,352-sample benign
-    gate, `00506` / `00507` / `00518` are 0-FP; `00508` has 4 FP, `00509` 3,
-    `ATR-2026-00001` 19, `ATR-2026-00400` 1.
-    **PromptBench 15.7%** is produced by **3 of 780 rules** (`ATR-2026-00520`,
+    templates. Remove those five and recall on the same 1,080 samples was
+    **9.7%**. The concentration was real but not fragile: the top rule
+    (`ATR-2026-00508`, 968/1,080 samples) was the sole detector on none of
+    them, so deleting it left recall at 100%; only `00518` (45 samples) and
+    `00507` (27) were sole detectors of anything. On the 5,352-sample benign
+    gate, `00506` / `00507` / `00518` were 0-FP; `00508` had 4 FP, `00509` 3,
+    `ATR-2026-00001` 19, `ATR-2026-00400` 1. At 4.1.5 nine rules fire on
+    PromptInject. One of them, `ATR-2026-02846`, was added since and was
+    authored from PromptInject attack clusters (its title says so); it matches
+    780 of the 1,080 samples, so at 4.1.5 the closed-book figure needs it
+    removed as well as the five above.
+    **PromptBench 15.7%** was produced by **3 of 780 rules** (`ATR-2026-00520`,
     `00519`, `00202`), all three 0-FP on the same benign gate. Two of the three
-    were mined from PromptBench; without them recall is **2.4%**.
+    were mined from PromptBench; without them recall was **2.4%**.
     The PromptBench row moved 23.2% (3.5.2) → 15.7% (3.5.11) and the loss is
     fully attributable: 247 samples were held only by rules that have since
     been precision-repaired, and re-running each rule version by version pins
@@ -596,6 +760,38 @@ See [CHANGELOG.md](CHANGELOG.md).
     columns are properties of the corpus, not measurements — read them
     together with the benign-gate FP counts above, never alone.
 
+[^vendorsuites]: **Partly in-sample; precision not cited.** 126 attack and 20
+benign test cases (llm-guard 46 / 16, NeMo Guardrails 22 / 4, promptfoo 58 / 0),
+each traceable to repo, commit, path and line in
+[`manifest.json`](data/test-corpora/vendor-guardrail-suites/manifest.json). At ATR
+4.1.5 (measured 2026-10-19) recall is 97/126 (hunt), 92/126 (alert) and 7/126
+(enforce). Without the 16 rules whose YAML names promptfoo, llm-guard or NeMo, or
+carries the `(vendor-corpus-import)` tag, it is 78/126 = 61.9% (hunt), 70/126 =
+55.6% (alert) and 7/126 = 5.6% (enforce); per vendor in the hunt lane, llm-guard
+26/46, NeMo Guardrails 20/22, promptfoo 32/58. That ablation is the reference
+to cite beside the row, and it still overstates out-of-sample recall, by an
+amount the same file measures (`breakdown.overlaps`): 35 further rules carry a
+40-character run of some vendor sample's text in their YAML, and 23 of the 78
+hunt-lane detections in the ablation are by such a rule on the sample it
+overlaps; removing those rules too gives 58/126 = 46.0% (hunt) / 5/126 = 4.0%
+(enforce). Five NeMo Guardrails attack samples (DAN-style jailbreaks) share 58%
+to 99% of their text with a prompt in `garak-full`'s `dan.json` and
+`inthewild.json`, a corpus other rules were mined from; all five are detected in
+the hunt lane of the ablation. The vendors ship only 20 benign cases (the hunt
+lane flags 1, the enforce lane 0), too few for a precision or FP-rate claim, so
+the cells say "not cited" and `data/stats.json` carries `null` for both
+(`breakdown.precision_not_citable`).
+
+[^legacyvendor]: These three rows are the corpora assembled by hand on
+2026-05-12 and attributed to the vendors' test suites. A script-based audit
+([`legacy-corpora-provenance.json`](data/test-corpora/vendor-guardrail-suites/legacy-corpora-provenance.json))
+checked each of their 94 samples against the vendor repo it names, at a pinned
+commit: 79 occur verbatim in the vendor file they name, 3 are truncated or
+edited from it, 2 are another vendor's test cases, and 10 are not found
+verbatim anywhere in the vendor repository at the pinned commit. Read them as ATR-assembled corpora seeded from vendor tests, not as the
+vendors' fixtures; the `Vendor guardrail suites` row is the script-extracted
+replacement.
+
 [^stdcorpora]: Until 2026-08-05 these three rows were **not produced by the ATR
     engine**. `scripts/eval-std-corpora.ts` walked `rules/` with a YAML parser,
     kept only `operator: regex` conditions, flattened every condition of every
@@ -617,56 +813,72 @@ See [CHANGELOG.md](CHANGELOG.md).
     and the canonical event shapes in `scripts/lib/corpus-event.ts` — the same
     entry point the false-positive gates use. Reproduce with
     `npx tsx scripts/eval-std-corpora.ts`. Read the new numbers with the same
-    scope caveat as `PINT-format`: on ATLAS, `ATR-2026-00061` alone accounts
-    for 59 of the 71 detections (32.4% of the corpus), and ATLAS procedures are
+    scope caveat as `PINT-format`: on ATLAS at 3.5.11, `ATR-2026-00061` alone
+    accounted for 59 of the 71 detections (32.4% of the corpus; 58 of 68 at
+    4.1.5), and ATLAS procedures are
     prose *descriptions* of attacks rather than attack payloads, so this row
     measures ATR against attack write-ups, not against traffic.
 
-Two `garak` rows are deliberate: the headline `garak` source tracks NVIDIA's
-in-the-wild jailbreak corpus (narrow, the ~92% number ATR cites publicly,
-refreshed 2026-08-15 against ATR 3.5.12), while `garak-full` tracks
-every probe family in upstream garak (broad, includes families like
-`badchars`, `dra`, `encoding` that ATR's regex layer intentionally does
-not target). Both are valid measurements against different corpora; they
-are kept as separate streams so the broad-corpus number does not silently
-overwrite the headline.
-
-The single-digit recall on AdvBench / HarmBench / JailbreakBench / hh-rlhf is
+Two `garak` rows are deliberate, and they are not independent. The headline
+`garak` source tracks NVIDIA's in-the-wild jailbreak corpus (narrow; this is the
+number ATR cites publicly, and [^garakcorpus] gives how much of it rests on
+rules written from that corpus). `garak-full` runs every non-empty family file
+extracted from upstream garak (23 at ATR 4.1.5), and it contains the
+in-the-wild corpus twice: as its `inthewild` family, and as 648 of the 664
+prompts of its `dan` family. In the 2026-10-15 measurement those two families
+supplied 1,214 of its 1,987 `hunt` detections. Rules have also been written
+from most of its other families (an `author` of
+`ATR Community (via NVIDIA garak <family> probe)`, 12 of them from `badchars`,
+`dra` and `encoding`), so read `garak-full` as partly closed-book too.
+[docs/RULE-PRODUCTION.md](docs/RULE-PRODUCTION.md) §2.3–2.4 covers its other
+limits: six of its families are content-safety and hallucination probes, four
+are the same 16 unencoded seed sentences, and 2,739 of its 3,475 prompts are
+distinct. The two are kept as separate streams so neither figure stands in for
+the other.
+
+The low recall on AdvBench / HarmBench / JailbreakBench / hh-rlhf is
 honest and expected. Those four corpora test **LLM safety alignment** (does the
 model refuse harmful requests like "explain how to make a bomb"), not
 **prompt-injection detection** (the surface ATR's regex layer targets).
-ATR's near-zero recall on these corpora confirms the layering thesis:
-regex catches structured attack patterns, alignment + content moderation
-catch natural-language harm requests. The numbers are recorded for
+ATR's low recall on these corpora is consistent with the layering view
+(this section does not measure what any other layer catches): regex catches
+structured attack patterns, alignment + content moderation catch
+natural-language harm requests. The numbers are recorded for
 completeness and so any future ATR rule additions in the harm-category
 space can be measured against a documented baseline. `hh-rlhf` is Anthropic's
-red-team-attempts set — the same genre as the other three — and its 1.5% now
-sits with their 2.1% / 2.8% / 6.0% instead of contradicting them at 99.1%.
+red-team-attempts set — the same genre as the other three — and its
+low recall now sits with theirs instead of contradicting them at
+99.1%.
 
 Conventions: 100%-adversarial corpora contain no benign samples, so they have
 no true-negative population and **`precision` and `fp_rate` cannot be computed
 from them**. The measurement schema requires numbers, so those rows record the
 convention `precision 1` / `fp_rate 0`. Read the `Precision` and `FP rate`
-columns as "not applicable to this corpus", not as results — the real
-precision numbers come from the benign gate, lane-keyed, below. Wild-scan has
-no ground-truth labels either; its `precision` column reports a precision floor
-computed as `confirmed_malware / flagged`. Every cell is sourced from a
-specific measurement file — see `data/measurements/<source>/latest.json` for
-the file path and `metadata.measurement_file` in `stats.json` for the absolute
-repo path.
+columns as "not applicable to this corpus", not as results. No benign-gate
+precision numbers are published at present: the lane FP rates once quoted are
+withdrawn (see below). Wild-scan has no ground-truth labels either; its
+`Precision` cell reads `not measured` because the precision floor once
+published for it is withdrawn (see [^wildscan]), and its `FP rate` cell is a
+flag rate, flagged divided by scanned, not a false-positive rate. Every cell
+except the `Wild scan` row (taken from the raw scan file; see [^wildscan]) is
+sourced from a specific measurement file: `data/measurements/<source>/latest.json`
+names it, and `benchmarks[].measurement_file` in `data/stats.json` gives its
+path.
 
 False-positive rate is lane-keyed as of v3.5.0, not a single overall figure.
 ATR ships detection lanes (`enforce` / `alert` / `hunt`). The specific lane FP
 rates previously quoted here are withdrawn and awaiting re-measurement — see
-the note in the lanes section above for why. Per-corpus `FP rate` cells above
-are measured in the default `hunt` lane. See [CHANGELOG.md](CHANGELOG.md)
+the note in the lanes section above for why. Where a row has benign samples
+(`atr-self-test`, `PINT-format`, `SKILL.md benchmark`), its `FP rate` cell is a
+`hunt`-lane measurement; on every other row it is a convention value or, for
+`Wild scan`, a flag rate. See [CHANGELOG.md](CHANGELOG.md)
 (v3.5.0) for the lane definitions.
 
 ```bash
 npm test                                    # engine + rule unit tests (vitest)
 npm run eval                                # atr-self-test eval (writes a measurement)
 npm run eval:pint                           # PINT benchmark (writes a measurement)
-npx tsx src/eval/run-hackaprompt-benchmark.ts                                # HackAPrompt
+npm run eval:hackaprompt                                                     # HackAPrompt (gated dataset; writes a measurement)
 npx tsx src/eval/skill-benchmark.ts                                          # SKILL.md (498 labeled)
 npx tsx scripts/eval-std-corpora.ts                                          # HH-RLHF + OWASP + ATLAS
 npx tsx scripts/atr_recall_analysis.ts                                       # PromptBench + PromptInject
@@ -681,9 +893,27 @@ npx tsx scripts/sync-stats-from-measurements.ts                              # r
 
 Raw data: [`data/full-scan-v2-2026-04-14.json`](data/full-scan-v2-2026-04-14.json) — 101,280 skills scanned, 1,434 flagged, engine 2.0.0, timestamped 2026-04-13. The malware-campaign write-up is in [`docs/research/openclaw-malware-campaign-2026-04.md`](docs/research/openclaw-malware-campaign-2026-04.md).
 
+[^hackaprompt]: **Partly in-sample; cite the ablation reference beside it.**
+The samples are a seeded subset of 4,780 of the dataset's 601,757 rows
+(`hackaprompt/hackaprompt-dataset` at revision `25b87fbe`, parquet sha256
+`bedca308...`, seed 20260511, rows shorter than 3 characters dropped), pinned
+by row indices and digests in
+[`hackaprompt-corpus.manifest.json`](data/hackaprompt/hackaprompt-corpus.manifest.json);
+`npm run eval:hackaprompt` re-draws it and stops on any mismatch. At ATR 4.1.5
+(measured 2026-10-19) recall is 3,972 (hunt), 3,931 (alert) and 2,343
+(enforce) of 4,780. Without the 20 rules whose YAML names HackAPrompt it is
+1,372 = 28.7% (hunt) / 32 = 0.7% (enforce): the ablation reference, an
+approximation of out-of-sample recall that does not guarantee it (a rule that
+does not name HackAPrompt can still have been tuned on these prompts). The
+corpus has no benign samples, so precision and FP rate are undefined on it:
+the cells say "not cited" and `data/stats.json` carries `null` for both
+(`breakdown.precision_not_citable`). Earlier figures for this row (28.6% → 66.2% in the 2.2.0 CHANGELOG
+entry, 69.6% at ATR 3.5.0) and their re-runs are in
+[`data/hackaprompt/METHOD.md`](data/hackaprompt/METHOD.md).
+
 [^wildscan]: Scanned and flagged counts come from `total_scanned` and `total_flagged` in the raw scan file; the flag rate is those two divided. **No precision figure is given because none was measured over this run.** Earlier material paired a manual-review count against a smaller 96,096-skill analysis subset and reported the ratio as a precision floor. The review covered that subset rather than all 1,434 flags, so the ratio does not describe this scan and is not reproducible from the raw file; it has been withdrawn rather than recomputed. Establishing precision here means re-reviewing a sample of the 1,434 flags, which has not been done.
 
-ATR is honest about what it cannot detect. Regex catalogs miss paraphrased attacks, semantic rephrasings of credential exfiltration, and novel attack shapes not present in the training corpus. `PromptBench` (3,280 character- and word-level robustness perturbations) is a different threat class from prompt injection and sits largely outside ATR's content scope; ATR still matches the 23.2% that carry injection-shaped payloads, at 100% precision. See [LIMITATIONS.md](LIMITATIONS.md) for the documented evasion-test corpus (64 techniques as of 2026-05) and the layering recommendation: ATR is the content layer; pair with credential brokering, sandbox execution, and human-in-the-loop for high-blast-radius actions.
+ATR is honest about what it cannot detect. Regex catalogs miss paraphrased attacks, semantic rephrasings of credential exfiltration, and novel attack shapes not present in the training corpus. `PromptBench` (3,280 character- and word-level robustness perturbations) is a different threat class from prompt injection and sits largely outside ATR's content scope; most of what ATR matches there comes from two rules mined from PromptBench itself, and the share fell from 23.2% at 3.5.2; [^promptcorpora] covers both. See [LIMITATIONS.md](LIMITATIONS.md) for the documented evasion-test corpus (64 techniques as of 2026-05) and the layering recommendation: ATR is the content layer; pair with credential brokering, sandbox execution, and human-in-the-loop for high-blast-radius actions.
 
 ## 9. Governance
 
@@ -693,7 +923,7 @@ ATR is currently single-maintainer (BDFL) under Adam Lin, transitioning to a Tec
 |---|---|
 | Phase 0 — Core spec, reference engine, initial rule corpus | Done |
 | Phase 1 — Distribution surfaces (npm, PyPI, GitHub Action, SARIF, MCP server) | Done |
-| Phase 2 — Production adoption (Microsoft AGT, Cisco AI Defense, MISP, Gen Digital Sage) | In progress |
+| Phase 2 — Downstream integrations (Cisco skill-scanner, MISP, Gen Digital Sage, Microsoft AGT examples) | In progress |
 | Phase 3 — Community contribution flywheel (issue-to-proposal automation, CVE-collector pipeline) | In progress |
 | Phase 4 — TSC seating; second-engine implementation; submission to a standards body | Planned |
 
@@ -797,7 +1027,7 @@ ATR is released under the [MIT License](LICENSE). All contributions are MIT-lice
 
 ## 17. Acknowledgments
 
-ATR's design draws on prior work in: [Sigma](https://github.com/SigmaHQ/sigma) (SIEM detection format), [YARA](https://github.com/VirusTotal/yara) (malware signature format), [OWASP LLM Top 10](https://owasp.org/www-project-top-10-for-large-language-model-applications/), [OWASP Agentic Top 10](https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/), [MITRE ATLAS](https://atlas.mitre.org/), [NVIDIA garak](https://github.com/NVIDIA/garak), [Lakera PINT](https://github.com/lakeraai/pint-benchmark), [Meta LlamaFirewall](https://ai.meta.com/research/publications/llamafirewall-an-open-source-guardrail-system-for-building-secure-ai-agents/), and [SAFE-MCP (OpenSSF)](https://github.com/safe-agentic-framework/safe-mcp).
+ATR's design draws on prior work in: [Sigma](https://github.com/SigmaHQ/sigma) (SIEM detection format), [YARA](https://github.com/VirusTotal/yara) (malware signature format), [OWASP LLM Top 10](https://owasp.org/www-project-top-10-for-large-language-model-applications/), [OWASP Agentic Top 10](https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/), [MITRE ATLAS](https://atlas.mitre.org/), [NVIDIA garak](https://github.com/NVIDIA/garak), [Lakera PINT](https://github.com/lakeraai/pint-benchmark), [Meta LlamaFirewall](https://ai.meta.com/research/publications/llamafirewall-an-open-source-guardrail-system-for-building-secure-ai-agents/), and [SAF-MCP](https://github.com/secure-agentic-framework/saf-mcp).
 
 The 2026-04 ecosystem-wide skill scan was made possible by the maintainers of OpenClaw, Skills.sh, Hermes Agent, and ClawHub publishing their registries openly.
 
@@ -814,7 +1044,7 @@ The 2026-04 ecosystem-wide skill scan was made possible by the maintainers of Op
 - [OWASP Agentic Top 10 (2026)](https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/) — Taxonomy of agentic-application risk categories.
 - [OWASP LLM Top 10 (2025)](https://owasp.org/www-project-top-10-for-large-language-model-applications/) — Taxonomy of LLM-application risk categories.
 - [MITRE ATLAS](https://atlas.mitre.org/) — Adversarial-threat landscape for AI systems.
-- [SAFE-MCP (OpenSSF)](https://github.com/safe-agentic-framework/safe-mcp) — Secure-MCP framework, technique catalog.
+- [SAF-MCP](https://github.com/secure-agentic-framework/saf-mcp) — Secure Agentic Framework for Model Context Protocol, technique catalog.
 - [Sigma](https://github.com/SigmaHQ/sigma) — Generic detection rule format for SIEMs (architectural precedent).
 - [YARA](https://github.com/VirusTotal/yara) — Pattern-matching language for malware (architectural precedent).
 - Five Eyes joint guidance on AI agent deployment (2026-05-01): CISA + NSA + UK NCSC + ASD + CCCS + NZ NCSC — [CyberScoop coverage](https://cyberscoop.com/cisa-nsa-five-eyes-guidance-secure-deployment-ai-agents/).