agent-threat-rules 4.0.0 → 4.2.0

package/README.md

@@ -13,10 +13,9 @@ AI Agent 威脅偵測規則的開放格式
 [![GitHub Marketplace](https://img.shields.io/badge/Marketplace-ATR%20Scan-2ea44f?style=flat-square&logo=github)](https://github.com/marketplace/actions/atr-scan)
 [![License: MIT](https://img.shields.io/badge/license-MIT-brightgreen?style=flat-square)](LICENSE)
 [![DOI](https://img.shields.io/badge/DOI-10.5281%2Fzenodo.19178002-blue?style=flat-square)](https://doi.org/10.5281/zenodo.19178002)
-[![Rules](https://img.shields.io/badge/rules-785-blue?style=flat-square)](#5-specification)
+[![Rules](https://img.shields.io/badge/rules-843-blue?style=flat-square)](#5-specification)
 [![Categories](https://img.shields.io/badge/categories-10-blue?style=flat-square)](#7-coverage)
 [![OWASP Agentic](https://img.shields.io/badge/OWASP_Agentic_Top_10-10%2F10-brightgreen?style=flat-square)](#7-coverage)
-[![SAFE-MCP](https://img.shields.io/badge/SAFE--MCP-91.8%25-brightgreen?style=flat-square)](#7-coverage)
 [![Sponsor](https://img.shields.io/badge/sponsor-Open%20Collective-7FADF2?style=flat-square&logo=opencollective&logoColor=white)](https://opencollective.com/agent-threat-rules)
 
 </div>
@@ -29,7 +28,7 @@ ATR (Agent Threat Rules) is an open detection rule format for AI agent security
 
 ## Status of This Document
 
-ATR is published as a **Working Draft** at version `3.0.0-alpha.1`. The rule format defined in `SPEC.md` is stable and merged into open-source repos at Microsoft, Cisco, and Gen Digital, and integrated by standards-body projects (MISP / CIRCL, OWASP Agent Security Regression Harness, SigmaHQ, FINOS Common Cloud Controls); full list with PR links in [§6 Adoption](#6-adoption). Governance is currently single-maintainer (BDFL) transitioning to a Technical Steering Committee per [GOVERNANCE.md](GOVERNANCE.md).
+ATR is published as a **Working Draft** at version `4.1.5`. The rule format defined in `SPEC.md` is stable and merged into open-source repos at Microsoft, Cisco, and Gen Digital, and integrated by community projects (MISP / CIRCL, OWASP Agent Security Regression Harness, SigmaHQ, FINOS Common Cloud Controls); full list with PR links in [§6 Adoption](#6-adoption). Governance is currently single-maintainer (BDFL) transitioning to a Technical Steering Committee per [GOVERNANCE.md](GOVERNANCE.md).
 
 All numbers in this document are sourced from [`data/stats.json`](data/stats.json), which is the canonical record of the project's current state. Where this README and `stats.json` disagree, `stats.json` is authoritative.
 
@@ -70,13 +69,14 @@ ATD is ATR's technique catalog: an enumeration of agent-runtime attack *techniqu
 - [8. Evaluation](#8-evaluation)
 - [9. Governance](#9-governance)
 - [10. Security](#10-security)
-- [11. Contributing](#11-contributing)
-- [12. Citation](#12-citation)
-- [13. Maintainers](#13-maintainers)
-- [14. Sponsorship](#14-sponsorship)
-- [15. License](#15-license)
-- [16. Acknowledgments](#16-acknowledgments)
-- [17. References](#17-references)
+- [11. Telemetry](#11-telemetry)
+- [12. Contributing](#12-contributing)
+- [13. Citation](#13-citation)
+- [14. Maintainers](#14-maintainers)
+- [15. Sponsorship](#15-sponsorship)
+- [16. License](#16-license)
+- [17. Acknowledgments](#17-acknowledgments)
+- [18. References](#18-references)
 
 ---
 
@@ -155,6 +155,7 @@ Zero-install scan of the current directory; the image bundles the CLI and pulls
 ```bash
 atr scan skill.md                 # scan a SKILL.md file
 atr scan mcp-config.json          # scan MCP server config / event log
+atr scan tools.json               # scan an MCP tools/list result, one event per tool
 atr scan . --sarif > results.sarif
 atr convert generic-regex         # export rules as JSON (all patterns)
 atr convert splunk                # export to Splunk SPL
@@ -212,13 +213,15 @@ the next section covers. Neither implies the other.
 
 Each rule carries a maturity-driven **lane**, so a consumer can trade recall for precision instead of running every rule at one fixed threshold:
 
-| Lane | Fires | Intended use | FP on a 65K-sample benign gate |
+| Lane | Fires | Intended use | FP on a benign gate |
 |---|---|---|---:|
-| `enforce` | `stable` only | Narrowest, highest-precision set — the one to run when blocking is on | ~0.24% |
+| `enforce` | `stable` only | Narrowest, highest-precision set — the one to run when blocking is on | withdrawn |
 | `alert` | `stable` + `test` | Analyst / correlation | — |
-| `hunt` | all rules except `deprecated` | Broadest visibility (**default**) | ~9% |
+| `hunt` | all rules except `deprecated` | Broadest visibility (**default**) | withdrawn |
 
-Lanes are opt-in and backward-compatible for detection: the default is `hunt`, so every integration sees exactly the rules it saw before. Selecting `enforce` raises precision by firing only the most mature rules — and therefore catches fewer attacks. Report false-positive rates lane-keyed (`enforce` ~0.24% / `hunt` ~9% on the 65K-sample benign gate), not as a single overall figure. That gate is a separate corpus from the per-source measurements in [§8 Evaluation](#8-evaluation).
+Lanes are opt-in and backward-compatible for detection: the default is `hunt`, so every integration sees exactly the rules it saw before. Selecting `enforce` raises precision by firing only the most mature rules — and therefore catches fewer attacks. False-positive rates should always be reported lane-keyed rather than as a single overall figure, and that gate is a separate corpus from the per-source measurements in [§8 Evaluation](#8-evaluation).
+
+**The two lane FP rates previously published here (`enforce` ~0.24% / `hunt` ~9% over a "65K-sample benign gate") are withdrawn, and should not be cited.** Three things were wrong with them. The gate was labelled 65,000 samples; both rates were actually measured on a 10,863-sample subsample of it. The benign corpus contained real jailbreak samples — material the rules are meant to catch, sitting in the set that defines a false positive — until the exclusion filter merged on 2026-08-04 (#373), so the corpus changed underneath both figures. And the ladder that produced the `hunt` figure depended on a file that is no longer in the repository, so that one cannot be reproduced at all. Re-measurement against the current benign corpus is pending; until it lands there is a gap here rather than a number.
 
 ### Detection and enforcement are separate switches
 
@@ -266,17 +269,20 @@ would be more dangerous than the request.
 
 ```bash
 atr guard                                 # advisory: report only (the default)
-atr guard --lane enforce --blocking       # blocking on, and only the 106 of 777
-                                          # live rules that are maturity: stable
-                                          # may fire (see the recall note below)
+atr guard --lane enforce --blocking       # blocking on, and only the live rules
+                                          # that are maturity: stable may fire
+                                          # (see the recall note below)
 ATR_LANE=enforce ATR_BLOCKING=1 atr guard # the same, via the environment
 ```
 
 **`--lane enforce` costs recall, and the cost is large.** It loads only
-`maturity: stable` rules: **106 of the 777 live rules** in this repository at the
-commit this paragraph was written against. That is the trade being made every
-time enforcement is recommended alongside it — narrower firing set, lower
-false-positive rate, fewer attacks caught. Rule counts move daily, and a
+`maturity: stable` rules: **103 of the 823 live rules** at ATR 4.1.5, and each
+weekly measurement records the count it ran with as `rules_in_lane` in its
+`breakdown.enforce_lane`. That is the trade being made every time enforcement is
+recommended alongside it — narrower firing set, lower false-positive rate, fewer
+attacks caught. What it still catches, per corpus, is the second (`enforce`)
+figure in the `Recall` cells of [§8 Evaluation](#8-evaluation). Rule counts move
+daily, and a
 `grep`-based count is wrong here (eight rules quote the value as
 `maturity: "stable"`), so re-derive by parsing before quoting the figure
 anywhere:
@@ -296,9 +302,12 @@ PY
 
 Programmatic embedding: `new ATREngine({ lane })` for the lane,
 `new ActionExecutor({ adapter, blocking })` and
-`new HookHandler({ engine, executor, blocking })` for enforcement. **These are
-the only inputs** — the constructors ignore `ATR_LANE` and `ATR_BLOCKING`, so
-setting them will not configure an embedded engine. An unrecognised lane throws
+`new HookHandler({ engine, executor, blocking })` for enforcement, and
+`new ATREngine({ modelDecisions })` for whether a match a model decided (a
+semantic judge, the embedding signal, the semantic layer) may enforce; it is
+advisory by default. **These are the only inputs** — the constructors ignore
+`ATR_LANE`, `ATR_BLOCKING` and `ATR_MODEL_DECISIONS`, so setting them will not
+configure an embedded engine. An unrecognised lane throws
 a `TypeError`, and so does a non-boolean `blocking`: `blocking: "false"` is a
 string, and every non-empty string is truthy, so it used to switch enforcement
 **on** while reading as an explicit "off".
@@ -309,9 +318,9 @@ string, and every non-empty string is truthy, so it used to switch enforcement
 |---|---|---|
 | Specification (canonical pointer) | [SPEC.md](SPEC.md) | Resolves to the authoritative documents below |
 | Rule format spec (normative) | [SPEC.md](SPEC.md) | Rule format, identifier scheme, evaluation semantics |
-| Framework spec | [ATR-FRAMEWORK-SPEC.md](ATR-FRAMEWORK-SPEC.md) | Multi-layer detection framework design |
+| Framework spec | [ATR-FRAMEWORK-SPEC.md](ATR-FRAMEWORK-SPEC.md) | **Superseded 2026-05-16 by [SPEC.md](SPEC.md).** Kept as a historical artifact; multi-layer detection framework design |
 | Machine-readable schema | [spec/atr-schema.yaml](spec/atr-schema.yaml) | Authoritative validation source |
-| Schema field reference | [docs/schema-spec.md](docs/schema-spec.md) | Human-readable schema docs |
+| Schema field reference | [docs/schema-spec.md](docs/schema-spec.md) | **Superseded 2026-05-16 by [SPEC.md](SPEC.md).** Kept as a historical artifact; human-readable schema docs |
 | Quality standard | [docs/QUALITY-STANDARD.md](docs/QUALITY-STANDARD.md) | Rule promotion criteria (experimental → stable) |
 | Quality gate | [docs/QUALITY-GATE.md](docs/QUALITY-GATE.md) | Safety-gate semantics for community PRs |
 | Enforcement model | [docs/ENFORCEMENT-MODEL.md](docs/ENFORCEMENT-MODEL.md) | Lane and blocking switches, decision channels, migration (reference implementation, not normative) |
@@ -356,12 +365,12 @@ test_cases:
 
 ## 6. Adoption
 
-Production deployments and standards-body integrations, as of 2026-07-05 (every PR state re-verified against GitHub on that date):
+Merged integrations, as of 2026-07-05 (every PR state re-verified against GitHub on that date):
 
 | Organization | Integration | Reference |
 |---|---|---|
-| Microsoft Agent Governance Toolkit | 287-rule expansion + weekly auto-sync (merged 2026-04-26); 15-rule PoC (merged 2026-04-13) | [PR #1277](https://github.com/microsoft/agent-governance-toolkit/pull/1277) · [PR #908](https://github.com/microsoft/agent-governance-toolkit/pull/908) |
-| Cisco AI Defense (skill-scanner) | Full rule pack in production (merged 2026-04-22); original PoC (merged 2026-04-03) | [PR #99](https://github.com/cisco-ai-defense/skill-scanner/pull/99) · [PR #79](https://github.com/cisco-ai-defense/skill-scanner/pull/79) |
+| Microsoft Agent Governance Toolkit | Example directories in the AGT repository (ATR import example, merged 2026-05-18; ACS annotator adapter example, merged 2026-06-15). The rule-pack YAML from the earlier example PRs (#908, #1277) was removed upstream on 2026-07-30 (AGT PR #3451). | [PR #3025](https://github.com/microsoft/agent-governance-toolkit/pull/3025) · [PR #2308](https://github.com/microsoft/agent-governance-toolkit/pull/2308) |
+| Cisco AI Defense (skill-scanner) | Full rule pack merged into the open-source scanner repository (2026-04-22); original PoC (merged 2026-04-03). This is a merge into that repository, not a statement about any vendor product. | [PR #99](https://github.com/cisco-ai-defense/skill-scanner/pull/99) · [PR #79](https://github.com/cisco-ai-defense/skill-scanner/pull/79) |
 | MISP (CIRCL) | Threat-intel cluster (galaxy, merged 2026-05-10) + rule-ID tagging vocabulary (taxonomies, merged 2026-05-10) | [galaxy #1207](https://github.com/MISP/misp-galaxy/pull/1207) · [taxonomies #323](https://github.com/MISP/misp-taxonomies/pull/323) |
 | Gen Digital Sage (Norton / Avast / AVG parent) | Rule pack merged 2026-05-11 | [PR #33](https://github.com/gendigitalinc/sage/pull/33) |
 | OWASP Agent Security Regression Harness | ATR referenced as the canonical agent-threat detection ruleset in the threat catalogue (merged 2026-05-11) | [PR #74](https://github.com/OWASP/agent-security-regression-harness/pull/74) |
@@ -373,13 +382,13 @@ Production deployments and standards-body integrations, as of 2026-07-05 (every
 
 ### Featured loop — Microsoft Copilot SWE Agent → ATR (2026-05-11)
 
-On 2026-05-07 MSRC published two Semantic Kernel CVEs (CVE-2026-26030 lambda+eval RCE, CVE-2026-25592 autostart file write). On 2026-05-11 06:07 UTC, Microsoft Copilot SWE Agent opened [microsoft/agent-governance-toolkit#1981](https://github.com/microsoft/agent-governance-toolkit/pull/1981) with regression-test fixtures *presuming ATR detection*. At 08:24 UTC the same day, ATR v2.1.2 (rules ATR-2026-00440 + ATR-2026-00441) was merged, npm-published, and GitHub-released. End-to-end: 2h 16m.
+On 2026-05-07 MSRC published two Semantic Kernel CVEs (CVE-2026-26030 lambda+eval RCE, CVE-2026-25592 autostart file write). On 2026-05-10 the ATR maintainer filed [microsoft/agent-governance-toolkit#1937](https://github.com/microsoft/agent-governance-toolkit/issues/1937) mapping both CVEs to ATR rules. On 2026-05-11 06:07 UTC, after an AGT maintainer assigned that issue to it, Microsoft Copilot SWE Agent opened [microsoft/agent-governance-toolkit#1981](https://github.com/microsoft/agent-governance-toolkit/pull/1981), adding regression-test fixtures *presuming ATR detection* to the ATR community-rules example's test suite. By 08:24 UTC the same day, ATR v2.1.2 (rules ATR-2026-00440 + ATR-2026-00441) was merged, npm-published, and GitHub-released. PR open to rule merge: 2h 16m.
 
 This is Microsoft Copilot operating inside AGT, not an MSRC endorsement. Coverage is partial: 2 of 4 Copilot fixtures match the v2.1.2 canonical regex shape.
 
 ### Under maintainer review (open PRs)
 
-[NVIDIA garak #1676](https://github.com/NVIDIA/garak/pull/1676) · [NVIDIA NeMo Guardrails #1992](https://github.com/NVIDIA-NeMo/Guardrails/pull/1992) · [OWASP LLM Top 10 #814](https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/pull/814) · [OWASP AI Exchange #181](https://github.com/OWASP/www-project-ai-security-and-privacy-guide/pull/181) · [Meta PurpleLlama #206](https://github.com/meta-llama/PurpleLlama/pull/206) · [BerriAI LiteLLM #28050](https://github.com/BerriAI/litellm/pull/28050) · [promptfoo #8529](https://github.com/promptfoo/promptfoo/pull/8529) · [Microsoft agent-framework #6528](https://github.com/microsoft/agent-framework/pull/6528) · [OpenAI guardrails-python #77](https://github.com/openai/openai-guardrails-python/pull/77) · [Cisco mcp-scanner #194](https://github.com/cisco-ai-defense/mcp-scanner/pull/194) · [Cisco a2a-scanner #14](https://github.com/cisco-ai-defense/a2a-scanner/pull/14) · [Splunk security_content #4128](https://github.com/splunk/security_content/pull/4128) · [NIST OSCAL oscal-content #338](https://github.com/usnistgov/oscal-content/pull/338) · [OpenTelemetry semantic-conventions-genai #165](https://github.com/open-telemetry/semantic-conventions-genai/pull/165)
+[NVIDIA NeMo Guardrails #2251](https://github.com/NVIDIA-NeMo/Guardrails/pull/2251) · [BerriAI LiteLLM #28050](https://github.com/BerriAI/litellm/pull/28050) · [promptfoo #8529](https://github.com/promptfoo/promptfoo/pull/8529) · [Cisco mcp-scanner #194](https://github.com/cisco-ai-defense/mcp-scanner/pull/194) · [Splunk security_content #4128](https://github.com/splunk/security_content/pull/4128) · [NIST OSCAL oscal-content #338](https://github.com/usnistgov/oscal-content/pull/338) (awaiting NIST review) · [OpenTelemetry semantic-conventions-genai #165](https://github.com/open-telemetry/semantic-conventions-genai/pull/165)
 
 ### Integrating ATR into your project
 
@@ -402,10 +411,10 @@ ATR maps its rules onto established frameworks so adopters can answer "we deploy
 | Framework | Coverage | Mapping document |
 |---|---|---|
 | [OWASP Agentic Top 10 (2026)](https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/) | 10/10 categories, 1,179 mappings across all 683 tagged rules | [docs/OWASP-AGENTIC-MAPPING.md](docs/OWASP-AGENTIC-MAPPING.md) |
-| [SAFE-MCP (OpenSSF)](https://github.com/safe-agentic-framework/safe-mcp) | 78/85 techniques (91.8%) | [docs/SAFE-MCP-MAPPING.md](docs/SAFE-MCP-MAPPING.md) |
+| [SAF-MCP](https://github.com/secure-agentic-framework/saf-mcp) (Framework Model v2) | 55 of 78 active techniques have at least one reviewed ATR rule (partial or adjacent); candidate mappings, not validated by SAF | [docs/SAFE-MCP-MAPPING.md](docs/SAFE-MCP-MAPPING.md) |
 | [OWASP LLM Top 10 (2025)](https://owasp.org/www-project-top-10-for-large-language-model-applications/) | Per-rule references | Per-rule `references.owasp_llm` field |
 | [MITRE ATLAS](https://atlas.mitre.org/) | Per-rule references | Per-rule `references.mitre_atlas` field |
-| NIST AI RMF (community OSCAL catalog) | 4/4 functions covered, community catalog (NIST not endorsing) | [Agent-Threat-Rule/ai-rmf-oscal-catalog](https://github.com/Agent-Threat-Rule/ai-rmf-oscal-catalog) |
+| NIST AI RMF (community OSCAL catalog) | Per-rule subcategory mappings; community catalog, not endorsed by NIST | [Agent-Threat-Rule/ai-rmf-oscal-catalog](https://github.com/Agent-Threat-Rule/ai-rmf-oscal-catalog) |
 | Five Eyes joint guidance (2026-05-01) | 5-category Careful-Adoption guidance → ATR's 10 categories | [docs/FIVE-EYES-MAPPING.md](docs/FIVE-EYES-MAPPING.md) |
 
 ### Detection categories
@@ -441,7 +450,9 @@ A full list lives in each rule's `references.cve` field. See [LIMITATIONS.md](LI
 
 ## 8. Evaluation
 
-Every number below is a version-pinned, reproducible measurement. The full
+Every number in the table below is a version-pinned, reproducible measurement;
+figures in the prose and footnotes are either quoted from those measurements or
+are named one-off replays that say how they were produced. The full
 historical series for each source lives at
 [`data/measurements/<source>/`](data/measurements/) (immutable, append-only).
 The current pointer per source is `data/measurements/<source>/latest.json`.
@@ -449,35 +460,75 @@ Aggregated into [`data/stats.json`](data/stats.json) under `benchmarks[]`.
 
 | Source | Source version | Samples | Recall | Precision | FP rate | ATR version | Measured |
 |---|---|---:|---:|---:|---:|---|---|
-| AdvBench (LLM-attacks behaviors) | upstream-2026-06-16 | 520 | 2.1% | 100.0% | 0.0% | 3.5.0 | 2026-06-16 |
-| atr-self-test | internal | 341 | 96.6% | 100.0% | 0.0% | 3.5.12 | 2026-08-15 |
+| AdvBench (LLM-attacks behaviors) | upstream-2026-10-15 | 520 | 5.6% (hunt) / 0.0% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
+| atr-self-test | internal | 341 | 96.6% (hunt) / 3.4% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
 | autoresearch | internal-1054 | 1,054 | 15.1% | 100.0% | 0.0% | 3.0.0-alpha.0 | 2026-05-23 |
-| garak (in-the-wild jailbreaks) | inthewild-jailbreak-corpus-650 | 650 | 92.3% | 100.0% | 0.0% | 3.5.12 | 2026-08-15 |
-| garak-full (all probe families) | 23-families | 3,475 | 57.2% | 100.0% | 0.0% | 3.5.12 | 2026-08-15 |
-| hackaprompt | v1 | 4,780 | 69.6% | 100.0% | 0.0% | 3.5.0 | 2026-06-16 |
-| HarmBench (CAIS behaviors) | upstream-2026-06-16 | 400 | 2.8% | 100.0% | 0.0% | 3.5.0 | 2026-06-16 |
-| hh-rlhf (Anthropic red-team-attempts) [^stdcorpora] | snapshot-2026-04 | 4,957 | 1.5% | 100.0% | 0.0% | 3.5.11 | 2026-08-05 |
-| JailbreakBench (JBB-Behaviors) | upstream-2026-06-16 | 100 | 6.0% | 100.0% | 0.0% | 3.5.0 | 2026-06-16 |
-| llm-guard (Protect AI test fixtures) | corpus-2026-05-12 | 44 | 77.3% | 100.0% | 0.0% | 3.5.0 | 2026-06-16 |
-| MITRE ATLAS [^stdcorpora] | snapshot-2026-04 | 182 | 39.0% | 100.0% | 0.0% | 3.5.11 | 2026-08-05 |
-| NeMo Guardrails (NVIDIA test fixtures) | corpus-2026-05-12 | 6 | 100.0% | 100.0% | 0.0% | 3.5.0 | 2026-06-16 |
-| OWASP LLM Top 10 [^stdcorpora] | snapshot-2026-04 | 56 | 16.1% | 100.0% | 0.0% | 3.5.11 | 2026-08-05 |
-| PINT-format (deepset + Lakera Gandalf) [^pint] | v1 | 850 | 65.4% | 100.0% | 0.0% | 3.5.12 | 2026-08-15 |
-| PromptBench (academic adversarial) [^promptcorpora] | snapshot-2026-04 | 3,280 | 15.7% | 100.0% | 0.0% | 3.5.11 | 2026-08-05 |
-| promptfoo (red-team plugin fixtures) | corpus-2026-05-12 | 44 | 97.7% | 100.0% | 0.0% | 3.5.0 | 2026-06-16 |
-| PromptInject (academic adversarial) [^promptcorpora] | snapshot-2026-04 | 1,080 | 100.0% | 100.0% | 0.0% | 3.5.11 | 2026-08-05 |
-| SKILL.md benchmark (internal) [^skilllane] | internal-498 | 498 | 100.0% (hunt) / 0.0% (enforce) | 97.0% | 0.20% | 3.5.12 | 2026-08-15 |
-| Wild scan (OpenClaw + Skills.sh + Hermes + ClawHub) | corpus-2026-04-14 | 96,096 | — | 57.7% (floor) | 1.35% flag rate | 2.0.0 | 2026-04-14 |
-
-All detection corpora were (re-)measured against ATR 3.5.0 on 2026-06-16,
-except `autoresearch` (an internal predicted-rule corpus with no standalone
-runner) and the `Wild scan` snapshot, which retain their earlier measurements.
-`PromptInject` and `PromptBench` were re-measured against ATR 3.5.11 on
-2026-08-05; see [^promptcorpora] for what moved and why. (An earlier
-re-measurement against 3.5.2 on 2026-06-25 fixed a harness event shape; the
-0.0% rows before that were a harness artifact — the harness placed the prompt
-in a top-level field the engine does not read — not the engine's actual
-result.)
+| garak (in-the-wild jailbreaks) [^garakcorpus] | inthewild-jailbreak-corpus-650 | 650 | 92.3% (hunt) / 0.9% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
+| garak-full (all probe families) [^garakcorpus] | 23-families | 3,475 | 57.2% (hunt) / 0.5% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
+| hackaprompt [^hackaprompt] | v1 | 4,780 | 83.1% (hunt) / 49.0% (enforce) | not cited [^hackaprompt] | not cited [^hackaprompt] | 4.1.5 | 2026-10-19 |
+| HarmBench (CAIS behaviors) | upstream-2026-10-15 | 400 | 6.0% (hunt) / 0.0% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
+| hh-rlhf (Anthropic red-team-attempts) [^stdcorpora] | snapshot-2026-04 | 4,957 | 1.4% (hunt) / 0.0% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
+| JailbreakBench (JBB-Behaviors) | upstream-2026-10-15 | 100 | 3.0% (hunt) / 0.0% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
+| llm-guard (2026-05-12 import) [^legacyvendor] | corpus-2026-05-12 | 44 | 63.6% (hunt) / 0.0% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
+| MITRE ATLAS [^stdcorpora] | snapshot-2026-04 | 182 | 37.4% (hunt) / 1.1% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
+| NeMo Guardrails (2026-05-12 import) [^legacyvendor] | corpus-2026-05-12 | 6 | 100.0% (hunt) / 0.0% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
+| OWASP LLM Top 10 [^stdcorpora] | snapshot-2026-04 | 56 | 16.1% (hunt) / 0.0% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
+| PINT-format (deepset + Lakera Gandalf) [^pint] | v1 | 850 | 65.4% (hunt) / 0.4% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
+| PromptBench (academic adversarial) [^promptcorpora] | snapshot-2026-04 | 3,280 | 15.7% (hunt) / 0.0% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
+| promptfoo (2026-05-12 import) [^legacyvendor] | corpus-2026-05-12 | 44 | 90.9% (hunt) / 6.8% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
+| PromptInject (academic adversarial) [^promptcorpora] | snapshot-2026-04 | 1,080 | 100.0% (hunt) / 0.0% (enforce) | 100.0% | 0.0% | 4.1.5 | 2026-10-15 |
+| SKILL.md benchmark (internal) [^skilllane] | internal-498 | 498 | 100.0% (hunt) / 0.0% (enforce) | 97.0% (hunt) | 0.20% (hunt) | 4.1.5 | 2026-10-15 |
+| Vendor guardrail suites (pinned upstream tests) [^vendorsuites] | pinned-2026-10-19 | 146 | 77.0% (hunt) / 5.6% (enforce) | not cited [^vendorsuites] | not cited [^vendorsuites] | 4.1.5 | 2026-10-19 |
+| Wild scan (OpenClaw + Skills.sh + Hermes + ClawHub) [^wildscan] | full-scan-v2-2026-04-13 | 101,280 | — | not measured | 1.42% flag rate | 2.0.0 | 2026-04-13 |
+
+The `Vendor guardrail suites` row is the test cases of llm-guard, NeMo
+Guardrails and promptfoo extracted by script from pinned upstream commits
+([`data/test-corpora/vendor-guardrail-suites/`](data/test-corpora/vendor-guardrail-suites/README.md)).
+It is partly in-sample: 16 rules name one of the three vendors or carry the
+`(vendor-corpus-import)` tag. Its measurement file re-measures the same samples
+without them (`breakdown.in_sample_ablation.without_rules_naming_source`), the
+ablation reference to cite beside the row: 61.9% (hunt) / 5.6% (enforce) at
+ATR 4.1.5, measured 2026-10-19. That is not an out-of-sample figure either; see
+[^vendorsuites]. The weekly run does not re-measure this row:
+`bash scripts/vendor-suites/reproduce.sh --write` does. The three
+`2026-05-12 import` rows are the earlier hand-assembled corpora; not all of
+their samples come from the vendor they name ([^legacyvendor]).
+
+A `Recall` cell gives two figures from the same measurement file. The first is
+the `hunt` lane: the engine default, which loads every maturity, and the figure
+`latest.json` and `data/stats.json` carry. The second is the `enforce` lane, the
+one to run when blocking is on, which loads only `stable` rules: the same
+samples through the same detection loop, recorded as `breakdown.enforce_lane`.
+Read the second before deploying with blocking on; see [^skilllane].
+`autoresearch` shows only the first, because its last measurement predates
+the second's recording, and `Wild scan` has no recall at all. `Precision` and `FP rate` are `hunt`-lane figures, measured only on the
+rows that contain benign samples (`atr-self-test`, `PINT-format` and `SKILL.md
+benchmark`); elsewhere they are convention values or, for `Wild scan`, a flag
+rate, not results (see Conventions below).
+
+Whether a corpus was used to write the rules it scores has been examined for
+`garak`, `garak-full`, `PromptInject`, `PromptBench` ([^garakcorpus],
+[^promptcorpora]) and `hackaprompt` ([^hackaprompt]); all five are partly
+closed-book. For `hackaprompt`, 20 rules name HackAPrompt in their YAML,
+several mined from this exact sample; the same measurement file re-runs the
+samples without them (`breakdown.in_sample_ablation.without_rules_naming_source`),
+the ablation reference to cite beside the row: 28.7% (hunt) / 0.7% (enforce)
+at ATR 4.1.5, measured 2026-10-19. It approximates an out-of-sample figure; it
+is not one. The other rows are not
+thereby out-of-sample: `atr-self-test` is built partly from the rules' own true
+positives, and the rules authored `(vendor-corpus-import)` name the `promptfoo`
+and `llm-guard` fixtures as their source.
+
+A weekly run re-measures every row except the following, which keep their
+earlier measurements: `hackaprompt`, whose corpus (a gated HuggingFace dataset) is not
+provisioned in CI, so the run skips it and the row is re-measured locally with
+`npm run eval:hackaprompt`; `autoresearch`, an internal predicted-rule
+corpus with no standalone runner; and the `Wild scan` snapshot. The table is
+re-rendered from the measurement files; the prose and footnotes are not, so a
+figure there holds for the version it names. Before the weekly run, every row
+except `autoresearch` and the `Wild scan` had been (re-)measured against ATR
+3.5.0 on 2026-06-16; the re-measurements since are described below, and
+[^promptcorpora] covers what moved for `PromptInject` and `PromptBench`.
 
 `PromptInject` and `PromptBench` were re-measured against ATR 3.5.2 on
 2026-06-25 after a fix to the recall-analysis harness event shape; the prior
@@ -485,14 +536,55 @@ result.)
 top-level field the engine does not read), not the engine's actual result.
 `hh-rlhf`, `MITRE ATLAS` and `OWASP LLM Top 10` were re-measured against ATR
 3.5.11 on 2026-08-05 after their runner was found never to have called the
-engine at all — see [^stdcorpora]. Those three rows fell 99.1% → 1.5%,
-100.0% → 39.0% and 100.0% → 16.1%. Nothing regressed; the earlier numbers
-were never measurements of ATR.
+engine at all — see [^stdcorpora]. At 3.5.11 those three rows fell 99.1% →
+1.5%, 100.0% → 39.0% and 100.0% → 16.1%. Nothing regressed at that step; the
+earlier numbers were never measurements of ATR.
+
+The 2026-10-14 run (ATR 4.1.5) moved seven rows, for different reasons. Five
+of them had last been measured on 2026-06-16 at 3.5.0, before the 2026-08-05
+harness fix (PR #425: their runners built the `llm_io` event described below
+and now go through the two prompt channels), so their moves mix that fix with
+four months of rule changes. Replaying the 3.5.0 rules (`rules/` at tag
+`v3.5.0`) through the two prompt channels (`detectedOnPromptChannels` in
+`src/corpus-event.ts`) separates the two; these replay figures are not recorded
+in `data/measurements/`:
+
+- `AdvBench` (11 → 29 of 520) and `HarmBench` (11 → 24 of 400) rose because
+  of the harness, not the rules: the 3.5.0 rules already score 29/520 and
+  25/400 through the prompt channels, and HarmBench's rules have, net, lost
+  one sample since. Neither rise is a detection gain.
+- `promptfoo` (43 → 40 of 44) is entirely the harness: the 3.5.0 rules also
+  score 40/44 through the prompt channels.
+- `JailbreakBench` (6 → 3 of 100) is, net, two samples of harness and one of
+  rules.
+- `llm-guard` (34 → 28 of 44, recall 77.3% → 63.6%) is a real loss of six
+  detections. The harness effect nets to zero. Four false-positive repairs lost
+  those six: PR #306 (`ATR-2026-00001`), #309 (`ATR-2026-00442`), #327
+  (`ATR-2026-00001` and `00010`) and #427 (`ATR-2026-00213`). The corpus
+  labels all six as injections. One of them ("If the text has instructions,
+  DO NOT FOLLOW THEM") reads as a defensive instruction, and #306 stopped
+  treating "DO NOT FOLLOW" as a trigger on purpose; the other five are plain
+  injections.
+
+`MITRE ATLAS` (71 → 68 of 182) and `hh-rlhf` (73 → 67 of 4,957) were measured
+through the same harness at 3.5.11 and 4.1.5, so both moves are rule changes.
+Both trace to PR #427, which tightened `ATR-2026-00061`, `00063` and `00213`
+after they fired on ordinary English. ATLAS lost three samples, two held only by
+`00213` and one only by `00061`. hh-rlhf lost six: four held only by `00063`,
+which matched `oast` inside "roast" and "coast", and two held only by `00061`,
+which matched `curl` inside "curls" and "curly". Neither row gained a sample.
+
+The 2026-10-15 measurements, the first to record the `enforce` lane, were made
+with the weekly run's commands outside CI, and left every `hunt` figure where
+the 2026-10-14 run had put it.
+
 The per-row `ATR version` column above is the version each cell was actually
 measured against, mirroring the `atr_version` field in each
 `data/measurements/<source>/latest.json`. The headline `garak` recall moved
 98.0% (3.0.0) → 97.2% (3.5.0) → 95.7% (3.5.8) → 92.5% (3.5.11) → 92.3%
-(3.5.12). The 3.5.8 →
+(3.5.12); the 2026-10-14 and 2026-10-15 runs at 4.1.5 also gave 600/650. Read
+that figure as partly closed-book: [^garakcorpus] replays it without the rules
+written from this corpus. The 3.5.8 →
 3.5.11 step is a real loss of detections, not a change in how they are counted:
 PR #327 tightened an over-broad persona regex in `ATR-2026-00001` that had been
 false-positiving on benign "you are now an expert …" prose, and tightening it
@@ -511,71 +603,148 @@ run exists anywhere in the repository. `data/measurements/garak/latest.json`
 pointed, the entire time, at 95.7% measured on 3.5.8; `garak-full`'s pointed at
 38.3% on 3.5.0, while a never-referenced 3.5.8 file sat unread in the same
 directory. So the claims failed this project's own rule that every published
-number is a version-pinned, reproducible measurement. It was also produced by a harness
+number is a version-pinned, reproducible measurement. The 91.5% was also produced by a harness
 that built an event of `type: 'llm_io'`, which is a rule *source* and not an
 `AgentEventType`; `src/engine.ts` could not map it and so ran every rule of
 every source against the event instead of the two source types the harness
-documented itself as using. The 92.3% above replaces it: measured on
-2026-08-15 at 784 rules through `llm_input` + `tool_response`, the two channels
-`src/hook-handler.ts` can actually deliver a prompt on, and written to
+documented itself as using. The figure that replaced them was first measured on
+2026-08-05 at 780 rules (601/650) and re-measured on 2026-08-15 at 784 rules
+(600/650, written to
 `data/measurements/garak/2026-08-15_garak-inthewild-jailbreak-corpus-650_atr-3-5-12.json`
-with the commit that produced it. Under the wider shape set used for
-false-positive measurement (which also runs `engine.scanSkill()`) the same
-corpus scores 92.9%; that number is recorded in the measurement's `breakdown`
-and is deliberately not the published one, because a garak prompt never reaches
+with the commit that produced it), both through `llm_input` + `tool_response`,
+the two channels `src/hook-handler.ts` can actually deliver a prompt on; every
+`garak` measurement since, including the 2026-10-14 one at ATR 4.1.5, goes
+through the same two channels. Under the wider shape set used for
+false-positive measurement (which also runs `engine.scanSkill()`) the corpus
+scores a little higher, 92.9% at both 3.5.12 and 4.1.5, recorded as
+`wide_shape_recall` in each measurement's `breakdown`; that number is
+deliberately not the published one, because a garak prompt never reaches
 production as a SKILL.md. `.github/workflows/eval.yml` now runs
 `scripts/check-benchmark-citations.ts`, which fails CI if this table or
 `stats.json` cites a number no measurement file backs.
 See [CHANGELOG.md](CHANGELOG.md).
 
-[^skilllane]: **Lane matters more here than anywhere else in this table.** The
-    100% figure is the `hunt` lane, which is the engine default and loads every
-    maturity. In the `enforce` lane — the auto-block one, where a detection stops
-    the agent with no human in the loop — this corpus scores **0%**, and the
-    reason is structural rather than a tuning problem: of the 38 rules carrying
-    `scan_target: skill`, **all 38 are `maturity: test`, and none is `stable`.**
-    The enforce lane only loads `stable`, so it loads no skill-scanning rule at
-    all, and 0 of 32 malicious samples fire. Anyone reading "100% recall on
-    SKILL.md" and deploying in enforce mode would be forming a completely wrong
-    expectation, so both numbers are shown. Verified on this commit with
-    `grep`-free counting over `rules/**/*.yaml`.
+[^skilllane]: **The first figure in each `Recall` cell is the `hunt` lane, and
+    the lane can change the answer completely.** `hunt` is the engine default
+    and loads every maturity. The `enforce` lane — the one to run when blocking
+    is on, where a detection can stop the agent with no human in the loop —
+    loads only `stable` rules, and its figure is the second one in the cell.
+    Since 2026-10-15 every measurement records it as `breakdown.enforce_lane`
+    in the same file as the headline: the same samples through the
+    benchmark's own detection loop (`engine.scanSkill()` for SKILL.md, the two
+    prompt channels for `garak`), on `new ATREngine({ lane: 'enforce' })`, with
+    `rules_in_lane` giving how many rules that lane loaded. It counts
+    detections, and a detection is not necessarily a block: the verdict blocks
+    only on severity and confidence (`src/verdict.ts`), so the figure is an
+    upper bound on what blocking stops.
+    `scripts/check-benchmark-citations.ts` fails CI when a cell's enforce
+    figure differs from that record or is left out. For SKILL.md the low figure
+    is structural rather than a tuning problem: at ATR 4.1.5 all 39 rules
+    carrying `scan_target: skill` are `maturity: test` and none is
+    `maturity: stable`, so
+    the enforce lane loads none of the rules written specifically for SKILL.md
+    files. Anyone reading "100% recall on SKILL.md" and deploying with
+    blocking on would be forming a completely wrong expectation. Until
+    2026-10-15 the weekly run measured only the `hunt` lane, and the enforce
+    figures this footnote quoted were one-off replays.
 
 [^pint]: The `PINT-format` row is **not** a run of Lakera's official PINT
     benchmark. That corpus is private and roughly 5x larger; this row is a
     self-built 850-sample corpus in PINT's format, assembled from
     `deepset/prompt-injections` (660) and `Lakera/gandalf_ignore_instructions`
-    (190). It also carries a scope caveat worth stating plainly: only **63 of
-    784 rules** fire on it at all, and `ATR-2026-00001` alone accounts for 226
-    of the 295 detections. Read it as a prompt-injection-family score, not as
+    (190). It also carries a scope caveat worth stating plainly: at 3.5.12 only
+    **62 of 784 rules** fired on it at all, and `ATR-2026-00001` alone
+    accounted for 226 of the 295 detections. The 3.5.12 eval report
+    (`data/pint-benchmark/pint-eval-report.json`) listed 63 per-rule entries,
+    but one of those is `tier2.5-embedding-match` — the Tier 2.5 embedding
+    stage, not a rule. Read it as a prompt-injection-family score, not as
     ATR's overall coverage. The row moved 63.6% → 60.3% between 3.5.0 and
     3.5.11 for the same reason `garak` moved: PR #327 tightened
     `ATR-2026-00001`'s persona-switch regex to stop it false-positiving on
-    benign prose. Precision moved 99.7% → 100% over the same span. It then
-    recovered 60.3% → 65.4% at 3.5.12 (2026-08-15) as rules added since
-    3.5.11 widened the family: rules firing on this corpus went 29 → 63 while
-    `ATR-2026-00001`'s own contribution stayed at 226, so the gain came from
-    the tail, not from re-loosening the one dominant rule. Precision held at
-    100% (0 FP on the 399 benign samples).
-
-[^promptcorpora]: **Read both of these as closed-book scores.** Until
-    2026-08-05 the harness recorded its per-rule breakdown as the literal
-    string `"unknown"` (it read `m.rule_id` off an engine match that carries
-    `m.rule.id`), so no published version of these rows could say which rules
-    produced them. With attribution restored:
-    **PromptInject 100.0%** is produced by **7 of 780 rules**. Five of those
+    benign prose. Through 3.5.11 the runner used a legacy event shape under
+    which the engine's source-type filter dropped every non-`llm_io` rule
+    before matching, so those figures, precision included, were measured over
+    roughly half the rulebase. The rise to 65.4% at 3.5.12 (2026-08-15) is
+    that harness changing, not the rules: that run was the first through the
+    canonical shape (PR #427), which admits every rule. Replayed through the
+    canonical shape, the 3.5.11 rules score 300 detections against 272 under
+    the legacy shape (the published 3.5.11 figure), so the rules changed
+    between 3.5.11 and 3.5.12 lost five detections on this corpus, while
+    `ATR-2026-00001`'s own contribution stayed at 226. Precision on the
+    canonical shape at 3.5.12 was 100% (0 FP on the 399 benign samples).
+
+[^garakcorpus]: **Read this row as partly closed-book.** At ATR 4.1.5 (the rules
+    on `main` at `32bd48966`, 830 rule files), many of the rules that fire on
+    this corpus had been written from it, so its `hunt` figure is not an
+    out-of-sample result. A rule is counted as written from the corpus when its
+    own YAML shows the link, in either of two ways. 102 rules name the corpus in
+    their author, title, description or references (matching
+    `/inthewild|daninthewild|in-the-wild (jailbreak|corpus)/i`): an `author` of
+    `ATR Community (via NVIDIA garak in-the-wild corpus)` or
+    `(via NVIDIA garak dan.DanInTheWild probe)` — garak's `dan.DanInTheWild`
+    probes draw their prompts from this corpus — or, for example,
+    `ATR-2026-00307`, whose description cites `inthewild_jailbreak_llms.json`.
+    28 more give garak as their source in their `author` (22
+    `ATR Community (garak-import)`, 4 `(via NVIDIA garak dan probe)`, 2
+    `(via ATR garak pipe)`) and quote one of these 650 prompts in their own
+    true-positive test cases: 60 or more consecutive characters, compared after
+    lower-casing and collapsing whitespace. Of the 130, 126 fire here, and 175
+    of the 600 prompts the full rule set detected were detected by them alone.
+    **Replayed without them, recall on the same 650 prompts was 65.4% (425 of
+    650).** Without only the 102 that name the corpus, it was 81.5% (530 of
+    650). A looser reading goes lower. `ATR-2026-00003`, a generic DAN rule in
+    ATR since v0.1.0 (2026-03-09), before any rule here was written from garak,
+    also quotes this corpus: one of its test cases is the public DAN opener,
+    which 7 of these prompts contain. It detects 259 of the 650, almost all of
+    them without that text, so it is not counted; counting it gave 51.8% (337
+    of 650). Two rules authored `(via NVIDIA garak dan probe)` neither name nor
+    quote the corpus (`ATR-2026-00273` and `00304` cite garak's DAN templates);
+    counting them too would move 65.4% to 64.9%, by arithmetic over the full
+    run's per-prompt matches rather than by a replay. The other figures are
+    **replays**, not measurements: each was run once, on 2026-10-15, with the
+    rule files removed from a copy of `rules/` and every prompt sent through the
+    two channels the published figure uses (`llm_input` and `tool_response`, as
+    `scripts/eval-garak-inthewild.ts` does). Nothing in `data/measurements/`
+    records them, and nothing refreshes them. None of the 130 is
+    `maturity: stable`, so the `enforce` figure does not rest on them.
+    `garak-full` contains these 650 prompts twice; see the paragraph on the two
+    `garak` rows below.
+
+[^promptcorpora]: **Read both of these as closed-book scores.** Both rows are
+    measured on a legacy event, `{ type: 'llm_io', content,
+    fields.user_input }`. `llm_io` is a rule source, not an event type the
+    engine can route, so source filtering is off and every rule of every
+    source runs, on only those two fields. The other prompt-corpus rows
+    (`garak`, `AdvBench`, `HarmBench`, `JailbreakBench`, `llm-guard`,
+    `NeMo Guardrails` and `promptfoo`) go through the two prompt channels
+    instead. Each measurement's `breakdown.attribution.canonical_shapes` gives
+    the same corpus on the wider shape set the benign false-positive gate uses,
+    which also runs `engine.scanSkill()`; there PromptBench scores 54.8%, 1,633
+    of its 1,798 detections coming from one tool-argument rule,
+    `ATR-2026-00066`. That figure is deliberately not the published one, for
+    the reason given for `garak` above. Until 2026-08-05 the harness
+    recorded its per-rule breakdown as the literal string `"unknown"` (it read
+    `m.rule_id` off an engine match that carries `m.rule.id`), so no published
+    version of these rows could say which rules produced them. With
+    attribution restored, at 3.5.11:
+    **PromptInject 100.0%** was produced by **7 of 780 rules**. Five of those
     seven — `ATR-2026-00506`, `00507`, `00508`, `00509`, `00518` — carry
     `author: ATR Community (PromptInject corpus)`: they were written *from*
     this corpus, which has four attack classes built from a handful of
-    templates. Remove those five and recall on the same 1,080 samples is
-    **9.7%**. The concentration is real but not fragile: the top rule
-    (`ATR-2026-00508`, 968/1,080 samples) is the sole detector on none of
-    them, so deleting it leaves recall at 100%; only `00518` (45 samples) and
-    `00507` (27) are sole detectors of anything. On the 5,352-sample benign
-    gate, `00506` / `00507` / `00518` are 0-FP; `00508` has 4 FP, `00509` 3,
-    `ATR-2026-00001` 19, `ATR-2026-00400` 1.
-    **PromptBench 15.7%** is produced by **3 of 780 rules** (`ATR-2026-00520`,
+    templates. Remove those five and recall on the same 1,080 samples was
+    **9.7%**. The concentration was real but not fragile: the top rule
+    (`ATR-2026-00508`, 968/1,080 samples) was the sole detector on none of
+    them, so deleting it left recall at 100%; only `00518` (45 samples) and
+    `00507` (27) were sole detectors of anything. On the 5,352-sample benign
+    gate, `00506` / `00507` / `00518` were 0-FP; `00508` had 4 FP, `00509` 3,
+    `ATR-2026-00001` 19, `ATR-2026-00400` 1. At 4.1.5 nine rules fire on
+    PromptInject. One of them, `ATR-2026-02846`, was added since and was
+    authored from PromptInject attack clusters (its title says so); it matches
+    780 of the 1,080 samples, so at 4.1.5 the closed-book figure needs it
+    removed as well as the five above.
+    **PromptBench 15.7%** was produced by **3 of 780 rules** (`ATR-2026-00520`,
     `00519`, `00202`), all three 0-FP on the same benign gate. Two of the three
-    were mined from PromptBench; without them recall is **2.4%**.
+    were mined from PromptBench; without them recall was **2.4%**.
     The PromptBench row moved 23.2% (3.5.2) → 15.7% (3.5.11) and the loss is
     fully attributable: 247 samples were held only by rules that have since
     been precision-repaired, and re-running each rule version by version pins
@@ -591,6 +760,38 @@ See [CHANGELOG.md](CHANGELOG.md).
     columns are properties of the corpus, not measurements — read them
     together with the benign-gate FP counts above, never alone.
 
+[^vendorsuites]: **Partly in-sample; precision not cited.** 126 attack and 20
+benign test cases (llm-guard 46 / 16, NeMo Guardrails 22 / 4, promptfoo 58 / 0),
+each traceable to repo, commit, path and line in
+[`manifest.json`](data/test-corpora/vendor-guardrail-suites/manifest.json). At ATR
+4.1.5 (measured 2026-10-19) recall is 97/126 (hunt), 92/126 (alert) and 7/126
+(enforce). Without the 16 rules whose YAML names promptfoo, llm-guard or NeMo, or
+carries the `(vendor-corpus-import)` tag, it is 78/126 = 61.9% (hunt), 70/126 =
+55.6% (alert) and 7/126 = 5.6% (enforce); per vendor in the hunt lane, llm-guard
+26/46, NeMo Guardrails 20/22, promptfoo 32/58. That ablation is the reference
+to cite beside the row, and it still overstates out-of-sample recall, by an
+amount the same file measures (`breakdown.overlaps`): 35 further rules carry a
+40-character run of some vendor sample's text in their YAML, and 23 of the 78
+hunt-lane detections in the ablation are by such a rule on the sample it
+overlaps; removing those rules too gives 58/126 = 46.0% (hunt) / 5/126 = 4.0%
+(enforce). Five NeMo Guardrails attack samples (DAN-style jailbreaks) share 58%
+to 99% of their text with a prompt in `garak-full`'s `dan.json` and
+`inthewild.json`, a corpus other rules were mined from; all five are detected in
+the hunt lane of the ablation. The vendors ship only 20 benign cases (the hunt
+lane flags 1, the enforce lane 0), too few for a precision or FP-rate claim, so
+the cells say "not cited" and `data/stats.json` carries `null` for both
+(`breakdown.precision_not_citable`).
+
+[^legacyvendor]: These three rows are the corpora assembled by hand on
+2026-05-12 and attributed to the vendors' test suites. A script-based audit
+([`legacy-corpora-provenance.json`](data/test-corpora/vendor-guardrail-suites/legacy-corpora-provenance.json))
+checked each of their 94 samples against the vendor repo it names, at a pinned
+commit: 79 occur verbatim in the vendor file they name, 3 are truncated or
+edited from it, 2 are another vendor's test cases, and 10 are not found
+verbatim anywhere in the vendor repository at the pinned commit. Read them as ATR-assembled corpora seeded from vendor tests, not as the
+vendors' fixtures; the `Vendor guardrail suites` row is the script-extracted
+replacement.
+
 [^stdcorpora]: Until 2026-08-05 these three rows were **not produced by the ATR
     engine**. `scripts/eval-std-corpora.ts` walked `rules/` with a YAML parser,
     kept only `operator: regex` conditions, flattened every condition of every
@@ -612,56 +813,72 @@ See [CHANGELOG.md](CHANGELOG.md).
     and the canonical event shapes in `scripts/lib/corpus-event.ts` — the same
     entry point the false-positive gates use. Reproduce with
     `npx tsx scripts/eval-std-corpora.ts`. Read the new numbers with the same
-    scope caveat as `PINT-format`: on ATLAS, `ATR-2026-00061` alone accounts
-    for 59 of the 71 detections (32.4% of the corpus), and ATLAS procedures are
+    scope caveat as `PINT-format`: on ATLAS at 3.5.11, `ATR-2026-00061` alone
+    accounted for 59 of the 71 detections (32.4% of the corpus; 58 of 68 at
+    4.1.5), and ATLAS procedures are
     prose *descriptions* of attacks rather than attack payloads, so this row
     measures ATR against attack write-ups, not against traffic.
 
-Two `garak` rows are deliberate: the headline `garak` source tracks NVIDIA's
-in-the-wild jailbreak corpus (narrow, the ~92% number ATR cites publicly,
-refreshed 2026-08-15 against ATR 3.5.12), while `garak-full` tracks
-every probe family in upstream garak (broad, includes families like
-`badchars`, `dra`, `encoding` that ATR's regex layer intentionally does
-not target). Both are valid measurements against different corpora; they
-are kept as separate streams so the broad-corpus number does not silently
-overwrite the headline.
-
-The single-digit recall on AdvBench / HarmBench / JailbreakBench / hh-rlhf is
+Two `garak` rows are deliberate, and they are not independent. The headline
+`garak` source tracks NVIDIA's in-the-wild jailbreak corpus (narrow; this is the
+number ATR cites publicly, and [^garakcorpus] gives how much of it rests on
+rules written from that corpus). `garak-full` runs every non-empty family file
+extracted from upstream garak (23 at ATR 4.1.5), and it contains the
+in-the-wild corpus twice: as its `inthewild` family, and as 648 of the 664
+prompts of its `dan` family. In the 2026-10-15 measurement those two families
+supplied 1,214 of its 1,987 `hunt` detections. Rules have also been written
+from most of its other families (an `author` of
+`ATR Community (via NVIDIA garak <family> probe)`, 12 of them from `badchars`,
+`dra` and `encoding`), so read `garak-full` as partly closed-book too.
+[docs/RULE-PRODUCTION.md](docs/RULE-PRODUCTION.md) §2.3–2.4 covers its other
+limits: six of its families are content-safety and hallucination probes, four
+are the same 16 unencoded seed sentences, and 2,739 of its 3,475 prompts are
+distinct. The two are kept as separate streams so neither figure stands in for
+the other.
+
+The low recall on AdvBench / HarmBench / JailbreakBench / hh-rlhf is
 honest and expected. Those four corpora test **LLM safety alignment** (does the
 model refuse harmful requests like "explain how to make a bomb"), not
 **prompt-injection detection** (the surface ATR's regex layer targets).
-ATR's near-zero recall on these corpora confirms the layering thesis:
-regex catches structured attack patterns, alignment + content moderation
-catch natural-language harm requests. The numbers are recorded for
+ATR's low recall on these corpora is consistent with the layering view
+(this section does not measure what any other layer catches): regex catches
+structured attack patterns, alignment + content moderation catch
+natural-language harm requests. The numbers are recorded for
 completeness and so any future ATR rule additions in the harm-category
 space can be measured against a documented baseline. `hh-rlhf` is Anthropic's
-red-team-attempts set — the same genre as the other three — and its 1.5% now
-sits with their 2.1% / 2.8% / 6.0% instead of contradicting them at 99.1%.
+red-team-attempts set — the same genre as the other three — and its
+low recall now sits with theirs instead of contradicting them at
+99.1%.
 
 Conventions: 100%-adversarial corpora contain no benign samples, so they have
 no true-negative population and **`precision` and `fp_rate` cannot be computed
 from them**. The measurement schema requires numbers, so those rows record the
 convention `precision 1` / `fp_rate 0`. Read the `Precision` and `FP rate`
-columns as "not applicable to this corpus", not as results — the real
-precision numbers come from the benign gate, lane-keyed, below. Wild-scan has
-no ground-truth labels either; its `precision` column reports a precision floor
-computed as `confirmed_malware / flagged`. Every cell is sourced from a
-specific measurement file — see `data/measurements/<source>/latest.json` for
-the file path and `metadata.measurement_file` in `stats.json` for the absolute
-repo path.
+columns as "not applicable to this corpus", not as results. No benign-gate
+precision numbers are published at present: the lane FP rates once quoted are
+withdrawn (see below). Wild-scan has no ground-truth labels either; its
+`Precision` cell reads `not measured` because the precision floor once
+published for it is withdrawn (see [^wildscan]), and its `FP rate` cell is a
+flag rate, flagged divided by scanned, not a false-positive rate. Every cell
+except the `Wild scan` row (taken from the raw scan file; see [^wildscan]) is
+sourced from a specific measurement file: `data/measurements/<source>/latest.json`
+names it, and `benchmarks[].measurement_file` in `data/stats.json` gives its
+path.
 
 False-positive rate is lane-keyed as of v3.5.0, not a single overall figure.
-ATR ships detection lanes (`enforce` / `alert` / `hunt`); on a 65K-sample
-benign gate the `enforce` lane (stable + `confirm`-gated rules) holds ~0.24%
-FP, while the default `hunt` lane (all rules) runs ~9% FP. Per-corpus `FP rate`
-cells above are measured in the default `hunt` lane. See [CHANGELOG.md](CHANGELOG.md)
+ATR ships detection lanes (`enforce` / `alert` / `hunt`). The specific lane FP
+rates previously quoted here are withdrawn and awaiting re-measurement — see
+the note in the lanes section above for why. Where a row has benign samples
+(`atr-self-test`, `PINT-format`, `SKILL.md benchmark`), its `FP rate` cell is a
+`hunt`-lane measurement; on every other row it is a convention value or, for
+`Wild scan`, a flag rate. See [CHANGELOG.md](CHANGELOG.md)
 (v3.5.0) for the lane definitions.
 
 ```bash
 npm test                                    # engine + rule unit tests (vitest)
 npm run eval                                # atr-self-test eval (writes a measurement)
 npm run eval:pint                           # PINT benchmark (writes a measurement)
-npx tsx src/eval/run-hackaprompt-benchmark.ts                                # HackAPrompt
+npm run eval:hackaprompt                                                     # HackAPrompt (gated dataset; writes a measurement)
 npx tsx src/eval/skill-benchmark.ts                                          # SKILL.md (498 labeled)
 npx tsx scripts/eval-std-corpora.ts                                          # HH-RLHF + OWASP + ATLAS
 npx tsx scripts/atr_recall_analysis.ts                                       # PromptBench + PromptInject
@@ -674,9 +891,29 @@ npx tsx scripts/measurement/verify.ts       # validate every measurement file
 npx tsx scripts/sync-stats-from-measurements.ts                              # refresh stats.json benchmarks[]
 ```
 
-Raw data: [`data/full-scan-v2-2026-04-14.json`](data/full-scan-v2-2026-04-14.json) (96,096-skill scan; 1,302 flagged, 552 confirmed malicious after manual review); full malware-campaign report in [`docs/research/openclaw-malware-campaign-2026-04.md`](docs/research/openclaw-malware-campaign-2026-04.md).
-
-ATR is honest about what it cannot detect. Regex catalogs miss paraphrased attacks, semantic rephrasings of credential exfiltration, and novel attack shapes not present in the training corpus. `PromptBench` (3,280 character- and word-level robustness perturbations) is a different threat class from prompt injection and sits largely outside ATR's content scope; ATR still matches the 23.2% that carry injection-shaped payloads, at 100% precision. See [LIMITATIONS.md](LIMITATIONS.md) for the documented evasion-test corpus (64 techniques as of 2026-05) and the layering recommendation: ATR is the content layer; pair with credential brokering, sandbox execution, and human-in-the-loop for high-blast-radius actions.
+Raw data: [`data/full-scan-v2-2026-04-14.json`](data/full-scan-v2-2026-04-14.json) — 101,280 skills scanned, 1,434 flagged, engine 2.0.0, timestamped 2026-04-13. The malware-campaign write-up is in [`docs/research/openclaw-malware-campaign-2026-04.md`](docs/research/openclaw-malware-campaign-2026-04.md).
+
+[^hackaprompt]: **Partly in-sample; cite the ablation reference beside it.**
+The samples are a seeded subset of 4,780 of the dataset's 601,757 rows
+(`hackaprompt/hackaprompt-dataset` at revision `25b87fbe`, parquet sha256
+`bedca308...`, seed 20260511, rows shorter than 3 characters dropped), pinned
+by row indices and digests in
+[`hackaprompt-corpus.manifest.json`](data/hackaprompt/hackaprompt-corpus.manifest.json);
+`npm run eval:hackaprompt` re-draws it and stops on any mismatch. At ATR 4.1.5
+(measured 2026-10-19) recall is 3,972 (hunt), 3,931 (alert) and 2,343
+(enforce) of 4,780. Without the 20 rules whose YAML names HackAPrompt it is
+1,372 = 28.7% (hunt) / 32 = 0.7% (enforce): the ablation reference, an
+approximation of out-of-sample recall that does not guarantee it (a rule that
+does not name HackAPrompt can still have been tuned on these prompts). The
+corpus has no benign samples, so precision and FP rate are undefined on it:
+the cells say "not cited" and `data/stats.json` carries `null` for both
+(`breakdown.precision_not_citable`). Earlier figures for this row (28.6% → 66.2% in the 2.2.0 CHANGELOG
+entry, 69.6% at ATR 3.5.0) and their re-runs are in
+[`data/hackaprompt/METHOD.md`](data/hackaprompt/METHOD.md).
+
+[^wildscan]: Scanned and flagged counts come from `total_scanned` and `total_flagged` in the raw scan file; the flag rate is those two divided. **No precision figure is given because none was measured over this run.** Earlier material paired a manual-review count against a smaller 96,096-skill analysis subset and reported the ratio as a precision floor. The review covered that subset rather than all 1,434 flags, so the ratio does not describe this scan and is not reproducible from the raw file; it has been withdrawn rather than recomputed. Establishing precision here means re-reviewing a sample of the 1,434 flags, which has not been done.
+
+ATR is honest about what it cannot detect. Regex catalogs miss paraphrased attacks, semantic rephrasings of credential exfiltration, and novel attack shapes not present in the training corpus. `PromptBench` (3,280 character- and word-level robustness perturbations) is a different threat class from prompt injection and sits largely outside ATR's content scope; most of what ATR matches there comes from two rules mined from PromptBench itself, and the share fell from 23.2% at 3.5.2; [^promptcorpora] covers both. See [LIMITATIONS.md](LIMITATIONS.md) for the documented evasion-test corpus (64 techniques as of 2026-05) and the layering recommendation: ATR is the content layer; pair with credential brokering, sandbox execution, and human-in-the-loop for high-blast-radius actions.
 
 ## 9. Governance
 
@@ -686,15 +923,50 @@ ATR is currently single-maintainer (BDFL) under Adam Lin, transitioning to a Tec
 |---|---|
 | Phase 0 — Core spec, reference engine, initial rule corpus | Done |
 | Phase 1 — Distribution surfaces (npm, PyPI, GitHub Action, SARIF, MCP server) | Done |
-| Phase 2 — Production adoption (Microsoft AGT, Cisco AI Defense, MISP, Gen Digital Sage) | In progress |
+| Phase 2 — Downstream integrations (Cisco skill-scanner, MISP, Gen Digital Sage, Microsoft AGT examples) | In progress |
 | Phase 3 — Community contribution flywheel (issue-to-proposal automation, CVE-collector pipeline) | In progress |
 | Phase 4 — TSC seating; second-engine implementation; submission to a standards body | Planned |
 
 ## 10. Security
 
-Vulnerability reports are coordinated under [SECURITY.md](SECURITY.md). Please use the private security advisory channel on the GitHub repository, not public issues, for any report concerning a vulnerability in the engine or the rule corpus.
+Vulnerability reports are coordinated under [SECURITY.md](SECURITY.md). Report privately — not in a public issue — for anything concerning a vulnerability in the engine or the rule corpus. Two channels are open: a [GitHub Security Advisory](https://github.com/Agent-Threat-Rule/agent-threat-rules/security/advisories/new) (private vulnerability reporting is enabled on this repository, verified 2026-09-22), or email to <adam@agentthreatrule.org>. Response-time commitments are in [SECURITY.md](SECURITY.md).
+
+## 11. Telemetry
+
+**Nothing in this project sends anything anywhere unless you ask it to.** Scanning is a local operation: `atr scan`, `atr scan-skill`, `atr guard`, `atr validate`, and the library (`new ATREngine(...)`) are all silent by default.
+
+Anonymous detection reporting is available as an opt-in, for deployments that want to contribute detections to a shared sensor network.
+
+- **Turn it on:** `--report-to-cloud`. Without this flag nothing leaves your machine.
+- **Endpoint:** `--tc-url <url>`, or the `ATR_TC_URL` environment variable. **One of the two is required** — there is no built-in default, so opting in without naming a recipient is an error, not a silent choice of one. ATR is an open standard rather than any single collector's client, and a package that carried a destination for detection data would be making that choice on your behalf. Pass the base URL only (`https://host`); `/api/threats` is appended for you. HTTPS is required for anything that is not localhost.
+- `--no-report` is still accepted and is now a no-op, since off is the default.
+
+When reporting is on, `atr scan` prints a notice saying so before it starts and names the endpoint when it finishes. The rest of this section describes what is sent **when you have opted in**.
+
+**Every scanned item produces at least one event, including items where no rule matched** — those are reported as a single `__clean__` event rather than being skipped. An item that matches more than one rule produces one event per match, so a noisy file can emit several. Events are batched (50 per request, or every 60 seconds, whichever comes first) and flushed once more before the process exits. An unreachable endpoint is not fatal: the batch is re-queued and your scan results are unaffected.
+
+A detection event on the wire looks like this, verbatim apart from the UUID and hash:
+
+```json
+{"events":[{"attackSourceIP":"<random UUID>","attackType":"skill-compromise","mitreTechnique":"ATR-2026-00120","sigmaRuleMatched":"ATR-2026-00120","timestamp":"2026-09-22T08:09:40.946Z","region":"unknown","severity":"critical","confidence":0.92,"contentHash":"<sha256 hex>","scanTarget":"skill"}]}
+```
+
+| Field | What it actually carries |
+|---|---|
+| `attackSourceIP` | A random UUID generated fresh for each process. Despite the field name, no IP address is read or sent — though, as with any HTTP request, the receiving server sees the network source address. The same UUID is repeated in a client-id request header, so events from one run are linkable to each other, but not across runs. |
+| `attackType` | The matched rule's category, or `clean`. |
+| `mitreTechnique` / `sigmaRuleMatched` | Both carry the ATR rule ID, or `__clean__`. The names are legacy on the receiving side; neither holds a MITRE or Sigma identifier. |
+| `severity`, `confidence` | Taken from the matched rule. |
+| `contentHash` | A full, unsalted SHA-256 of the scanned content. It is not the content and cannot be reversed into it, but it is an exact fingerprint: anyone who already holds a candidate file can hash it and learn whether you scanned that file. |
+| `scanTarget` | The literal string `skill`, or the event type for MCP event scans. Never a file path. |
+| `timestamp` | ISO 8601, when the rule was evaluated. |
+| `region` | Always the literal string `unknown`. |
+
+Not sent: the scanned content, file names, file paths, your username, hostname, environment variables, or the ATR version. Requests are unauthenticated unless `TC_API_KEY` is set in the environment, in which case it is attached as a bearer token.
+
+If you are scanning material whose existence you cannot disclose, simply do not pass `--report-to-cloud`.
 
-## 11. Contributing
+## 12. Contributing
 
 The fastest contribution path requires no local setup:
 
@@ -706,7 +978,7 @@ Other contribution paths (evasion reports, false-positive reports, full rule aut
 
 All contributions are MIT-licensed by submission. There is no CLA.
 
-## 12. Citation
+## 13. Citation
 
 If you use ATR in academic work or security research, please cite the dataset via DOI:
 
@@ -725,15 +997,15 @@ The companion research paper is published on Zenodo: [PDF](docs/paper/ATR-Paper-
 
 Machine-readable citation metadata is available in [CITATION.cff](CITATION.cff) (CFF v1.2.0).
 
-## 13. Maintainers
+## 14. Maintainers
 
 - **Adam Lin (林冠辛)** — BDFL, [@eeee2345](https://github.com/eeee2345), adam@agentthreatrule.org, Taiwan.
 
 The TSC seating process is open per [GOVERNANCE.md](GOVERNANCE.md).
 
-## 14. Sponsorship
+## 15. Sponsorship
 
-ATR's rules, engine, and pipeline are MIT licensed in perpetuity. Maintenance — CVE-class response, weekly cross-ecosystem sync, the auto-review pipeline — runs on community sponsorship through [Open Source Collective, Inc.](https://opencollective.com/opensource) (501(c)(6), EIN 81-1567737).
+ATR's rules, engine, and pipeline are MIT licensed in perpetuity. Maintenance — CVE-class response, weekly cross-ecosystem sync, the auto-review pipeline — runs on community sponsorship through [Open Source Collective, Inc.](https://opencollective.com/opensource) (501(c)(6), EIN 82-2037583).
 
 **Sponsor page: [opencollective.com/agent-threat-rules](https://opencollective.com/agent-threat-rules)**
 
@@ -749,17 +1021,17 @@ Three funding milestones make the trajectory concrete:
 
 Organizations that want a deeper engagement — a named maintainer contact, faster turnaround on CVE-class updates, or co-authored rules attributed to your organization — can arrange a custom sponsorship tier through Open Source Collective. Email <adam@agentthreatrule.org>.
 
-## 15. License
+## 16. License
 
 ATR is released under the [MIT License](LICENSE). All contributions are MIT-licensed by submission.
 
-## 16. Acknowledgments
+## 17. Acknowledgments
 
-ATR's design draws on prior work in: [Sigma](https://github.com/SigmaHQ/sigma) (SIEM detection format), [YARA](https://github.com/VirusTotal/yara) (malware signature format), [OWASP LLM Top 10](https://owasp.org/www-project-top-10-for-large-language-model-applications/), [OWASP Agentic Top 10](https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/), [MITRE ATLAS](https://atlas.mitre.org/), [NVIDIA garak](https://github.com/NVIDIA/garak), [Lakera PINT](https://github.com/lakeraai/pint-benchmark), [Meta LlamaFirewall](https://ai.meta.com/research/publications/llamafirewall-an-open-source-guardrail-system-for-building-secure-ai-agents/), and [SAFE-MCP (OpenSSF)](https://github.com/safe-agentic-framework/safe-mcp).
+ATR's design draws on prior work in: [Sigma](https://github.com/SigmaHQ/sigma) (SIEM detection format), [YARA](https://github.com/VirusTotal/yara) (malware signature format), [OWASP LLM Top 10](https://owasp.org/www-project-top-10-for-large-language-model-applications/), [OWASP Agentic Top 10](https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/), [MITRE ATLAS](https://atlas.mitre.org/), [NVIDIA garak](https://github.com/NVIDIA/garak), [Lakera PINT](https://github.com/lakeraai/pint-benchmark), [Meta LlamaFirewall](https://ai.meta.com/research/publications/llamafirewall-an-open-source-guardrail-system-for-building-secure-ai-agents/), and [SAF-MCP](https://github.com/secure-agentic-framework/saf-mcp).
 
-The 96,096-skill ecosystem scan was made possible by the maintainers of OpenClaw, Skills.sh, Hermes Agent, and ClawHub publishing their registries openly.
+The 2026-04 ecosystem-wide skill scan was made possible by the maintainers of OpenClaw, Skills.sh, Hermes Agent, and ClawHub publishing their registries openly.
 
-## 17. References
+## 18. References
 
 ### Normative
 
@@ -772,7 +1044,7 @@ The 96,096-skill ecosystem scan was made possible by the maintainers of OpenClaw
 - [OWASP Agentic Top 10 (2026)](https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/) — Taxonomy of agentic-application risk categories.
 - [OWASP LLM Top 10 (2025)](https://owasp.org/www-project-top-10-for-large-language-model-applications/) — Taxonomy of LLM-application risk categories.
 - [MITRE ATLAS](https://atlas.mitre.org/) — Adversarial-threat landscape for AI systems.
-- [SAFE-MCP (OpenSSF)](https://github.com/safe-agentic-framework/safe-mcp) — Secure-MCP framework, technique catalog.
+- [SAF-MCP](https://github.com/secure-agentic-framework/saf-mcp) — Secure Agentic Framework for Model Context Protocol, technique catalog.
 - [Sigma](https://github.com/SigmaHQ/sigma) — Generic detection rule format for SIEMs (architectural precedent).
 - [YARA](https://github.com/VirusTotal/yara) — Pattern-matching language for malware (architectural precedent).
 - Five Eyes joint guidance on AI agent deployment (2026-05-01): CISA + NSA + UK NCSC + ASD + CCCS + NZ NCSC — [CyberScoop coverage](https://cyberscoop.com/cisa-nsa-five-eyes-guidance-secure-deployment-ai-agents/).