agent-threat-rules 2.1.5 → 3.0.5

package/spec/conformance/baseline/fixtures/README.md

@@ -0,0 +1,120 @@
+# ATR L1 Baseline Conformance Fixtures
+
+> **STATUS: PROPOSED v1.0 — populating begun 2026-05-25.** This is the
+> ground-truth input/output corpus that any conformant ATR engine MUST
+> reproduce. See `../README.md` for the level definitions and
+> `../../README.md` for spec index.
+
+---
+
+## Directory layout
+
+```
+fixtures/
+└── <fixture_id>/
+    ├── input.json     ← the AgentEvent fed into the engine
+    └── expected.json  ← the ATREvent shape the engine MUST produce
+```
+
+## Fixture ID convention
+
+`<rule-id>-<kind>-<seq>` where:
+
+- `<rule-id>` is the canonical or sovereign rule identifier (e.g.,
+  `ATR-2026-00001`, `ATR-COR-2026-00001`, `ATR-TW-2026-00001`).
+- `<kind>` is one of:
+  - `tp` — true positive (rule should fire)
+  - `tn` — true negative (rule must NOT fire)
+  - `evasion` — known evasion attempt (rule should still fire)
+  - `multi` — input that exercises multiple rules
+- `<seq>` is a zero-padded sequence number within the rule + kind.
+
+Examples:
+- `ATR-2026-00001-tp-001` — rule 00001, first true-positive fixture
+- `ATR-2026-00001-tn-003` — rule 00001, third true-negative fixture
+- `ATR-2026-00001-evasion-001` — rule 00001, first evasion fixture
+
+## input.json schema
+
+```json
+{
+  "fixture_id": "string (matches directory name)",
+  "fixture_kind": "true_positive | true_negative | evasion | multi",
+  "description": "string (human-readable, sources cited)",
+  "target_rule": "string (the primary rule this fixture exercises)",
+  "input_event": {
+    "type": "AgentEventType (per atr-event-v1.0.md)",
+    "timestamp": "RFC 3339",
+    "content": "string (optional)",
+    "fields": { "...": "..." },
+    "metadata": { "...": "..." },
+    "sessionId": "string",
+    "scanContext": "runtime | skill | agent_message | ..."
+  }
+}
+```
+
+## expected.json schema
+
+```json
+{
+  "fixture_id": "string (matches directory name)",
+  "expected_match": "bool (true if any rule should fire)",
+  "expected_rules_fired": ["array of rule_id strings"],
+  "expected_event_shape": {
+    "atr.rule_id": "must-match",
+    "atr.severity": "must-match",
+    "atr.category": "must-match",
+    "...": "engine-supplied fields use <engine-supplied> sentinel"
+  },
+  "match_tolerance": {
+    "min_confidence": "float 0..1",
+    "max_confidence": "float 0..1",
+    "allow_additional_rule_matches": "bool",
+    "additional_match_allowlist": ["array of rule_ids permitted to also fire"]
+  },
+  "notes": ["array of human-readable conformance reasoning"]
+}
+```
+
+## Conformance verdict
+
+For each fixture, the verdict is one of:
+
+- **PASS** — engine fired exactly the rules in `expected_rules_fired` (or
+  fired those plus rules in `additional_match_allowlist`), with event
+  fields matching `expected_event_shape` (modulo `<engine-supplied>`
+  sentinels), and `confidence` within `match_tolerance`.
+- **FAIL** — any required rule did not fire, OR any forbidden rule fired,
+  OR event shape mismatch on a MUST-match field.
+- **PARTIAL** — required rules fired but field shape was off (e.g., wrong
+  severity). Treated as FAIL for L1 conformance claim, but reported
+  separately for diagnostic.
+
+L1-baseline pass threshold is in `../manifest.json` (precision 1.00,
+recall 0.95).
+
+## Current fixture count
+
+| Status | Count | Note |
+|---|---|---|
+| Populated | 1 | ATR-2026-00001-tp-001 |
+| Targeted for v1.0 ratification | ~100 | one TP + one TN per stable canonical rule |
+| Targeted for L2 (profile) | TBD | covers `atr-baseline-runtime` and `atr-nist-rmf-measure` |
+| Targeted for L3 (correlation) | TBD | exercises each correlation type at least once |
+
+Contributors: open a PR adding a fixture directory. CI will validate
+schema. Maintainer review confirms the test case is canonical.
+
+## Provenance
+
+Fixtures are sourced from:
+
+1. Rule `test_cases` blocks (already-validated true positives and
+   negatives that ship with each rule).
+2. Public CVE reproductions where ATR rules exist (e.g., CVE-2024-5184
+   for ATR-2026-00001).
+3. Published academic adversarial datasets (PINT MCP, Garak, METR,
+   SpAIware) where licensing permits.
+4. Community-contributed adversarial inputs (under DCO sign-off per
+   `legal/CLA.md`).

package/spec/conformance/baseline/manifest.json

@@ -0,0 +1,56 @@
+{
+  "schema_version": "1.0",
+  "corpus_version": "1.0.0",
+  "spec_version": "1.0",
+  "level": "L1-baseline",
+  "description": "ATR Baseline Conformance Test Corpus v1.0. Every conformant engine must pass this corpus to claim L1 baseline conformance.",
+  "license": "CC0-1.0",
+  "build_timestamp": "2026-05-25T00:00:00Z",
+  "rule_corpus_version_required": "agent-threat-rules>=3.1.0",
+  "fixture_sources": {
+    "attack-fixtures": {
+      "path": "attack-fixtures/",
+      "format": "JSON files, one event per file conforming to spec/schema/event.schema.json input form",
+      "expected_count_at_v1_0_0": "≥50 (initial corpus seeded from existing tests/attack-corpus/)",
+      "expected_fires": "non-empty set of rule IDs per fixture"
+    },
+    "benign-fixtures": {
+      "path": "benign-fixtures/",
+      "format": "SKILL.md text files, one fixture per file",
+      "expected_count_at_v1_0_0": "432 (sourced from data/skill-benchmark/benign/)",
+      "expected_fires": "EMPTY SET — any rule fire is a false positive"
+    },
+    "language-detection-fixtures": {
+      "path": "language-detection-fixtures/v1.0.json",
+      "format": "JSON array of {text, expected_language_code}",
+      "expected_count_at_v1_0_0": "≥200",
+      "expected_match": "exact ISO 639-1 / 639-3 code per spec/atr-language-detection-v1.0.md"
+    },
+    "research-mentions": {
+      "path": "research-mentions/corpus.jsonl",
+      "format": "JSONL, one record per line",
+      "expected_count_at_v1_0_0": "sourced from data/research-mentions/corpus.jsonl",
+      "expected_fires": "EMPTY SET — text MENTIONS attacks (papers, READMEs, blogs) but is not an attack"
+    }
+  },
+  "conformance_thresholds": {
+    "precision_min": 1.00,
+    "recall_min": 0.95,
+    "language_detection_accuracy_min": 1.00,
+    "fp_count_max": 0
+  },
+  "linked_references": [
+    "../README.md",
+    "../SIGNING.md",
+    "../expected-results.schema.json",
+    "../../atr-event-v1.0.md",
+    "../../atr-language-detection-v1.0.md",
+    "../../atr-schema.yaml"
+  ],
+  "build_provenance": {
+    "builder": "Adam Lin <adam@agentthreatrule.org>",
+    "bootstrap_phase": true,
+    "tsc_ratified": false,
+    "notes": "v1.0.0 manifest is the bootstrap fixture before TSC seating per governance/CHARTER.md § 11. Source-of-truth fixtures reference the existing data/skill-benchmark/ and data/research-mentions/ paths to avoid duplication during bootstrap. On TSC ratification + Phase 2 completion, fixtures will be COPIED (not referenced) into spec/conformance/baseline/ so the corpus is self-contained."
+  }
+}

package/spec/conformance/expected-results.schema.json

@@ -0,0 +1,121 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://spec.agentthreatrule.org/conformance/v1.0/expected-results.schema.json",
+  "title": "ATR Conformance Expected Results v1.0",
+  "description": "Schema for the canonical expected-results.json file shipped with each conformance level (baseline, profiles, correlation), AND for engine-results.json reports produced by implementations under test.",
+  "type": "object",
+  "required": [
+    "schema_version",
+    "corpus_version",
+    "spec_version",
+    "level",
+    "fixtures"
+  ],
+  "additionalProperties": false,
+  "properties": {
+    "schema_version": {
+      "const": "1.0"
+    },
+    "corpus_version": {
+      "type": "string",
+      "pattern": "^\\d+\\.\\d+\\.\\d+$"
+    },
+    "spec_version": {
+      "type": "string",
+      "pattern": "^\\d+\\.\\d+$"
+    },
+    "level": {
+      "type": "string",
+      "enum": ["L1-baseline", "L2-profile", "L3-correlation"]
+    },
+    "engine_id": {
+      "type": "string",
+      "description": "Only present in engine-results.json reports. Format: <vendor>/<product>/<version>."
+    },
+    "engine_run_timestamp": {
+      "type": "string",
+      "format": "date-time",
+      "description": "Only present in engine-results.json reports."
+    },
+    "rule_corpus_version_used": {
+      "type": "string",
+      "description": "ATR rule corpus version the engine loaded (e.g., agent-threat-rules@3.1.0)."
+    },
+    "fixtures": {
+      "type": "array",
+      "minItems": 1,
+      "items": {
+        "type": "object",
+        "required": ["fixture_id", "fixture_path"],
+        "properties": {
+          "fixture_id": {
+            "type": "string",
+            "description": "Unique identifier within this corpus, typically slug of fixture filename."
+          },
+          "fixture_path": {
+            "type": "string",
+            "description": "Path relative to spec/conformance/<level>/ root."
+          },
+          "input_summary": {
+            "type": "string",
+            "description": "1-line description of what this fixture exercises."
+          },
+          "expected_rules": {
+            "type": "array",
+            "items": {
+              "type": "string",
+              "pattern": "^ATR-(?:COR-)?(?:[A-Z]{2}-)?[0-9]{4}-[0-9]{5}$"
+            },
+            "description": "Rule IDs that MUST fire on this fixture. Empty array = MUST NOT fire any rule (TN fixture)."
+          },
+          "expected_event_partial": {
+            "type": "object",
+            "description": "Optional partial event match — engine output must contain these key-value pairs. Useful for asserting matched_field, category, severity uplift."
+          },
+          "expected_language_code": {
+            "type": "string",
+            "pattern": "^([a-z]{2}(?:-[A-Z][a-z]{3}|-[A-Z]{2})?|und)$",
+            "description": "For language-detection fixtures only: the expected ISO 639-1 / 639-3 code."
+          },
+          "engine_observed": {
+            "type": "object",
+            "description": "Only present in engine-results.json reports. The actual observation from the engine under test.",
+            "properties": {
+              "rules_fired": {
+                "type": "array",
+                "items": {
+                  "type": "string",
+                  "pattern": "^ATR-(?:COR-)?(?:[A-Z]{2}-)?[0-9]{4}-[0-9]{5}$"
+                }
+              },
+              "language_detected": {
+                "type": "string"
+              },
+              "pass": {"type": "boolean"},
+              "miss_reason": {
+                "type": "string",
+                "description": "Human-readable reason if pass=false."
+              }
+            }
+          }
+        }
+      }
+    },
+    "summary": {
+      "type": "object",
+      "description": "Only present in engine-results.json reports.",
+      "properties": {
+        "total_fixtures": {"type": "integer", "minimum": 0},
+        "passed": {"type": "integer", "minimum": 0},
+        "failed": {"type": "integer", "minimum": 0},
+        "precision": {"type": "number", "minimum": 0.0, "maximum": 1.0},
+        "recall": {"type": "number", "minimum": 0.0, "maximum": 1.0},
+        "language_detection_accuracy": {"type": "number", "minimum": 0.0, "maximum": 1.0},
+        "conformance_claim": {
+          "type": "string",
+          "enum": ["pass", "fail", "partial"]
+        }
+      }
+    }
+  }
+}

package/spec/external-registries/cccs-yara.md

@@ -0,0 +1,142 @@
+# ATR ↔ CCCS-Yara Cross-Reference Convention
+
+Version: 1.0.0
+Status: Draft
+Date: 2026-05-29
+Editor: Adam Lin (林冠辛) <adam@agentthreatrule.org>
+Trigger: CybercentreCanada/CCCS-Yara#100 closing comment (2026-05-26)
+  by cccs-rs: "better to handle the cross-reference on the ATR side at
+  this time... we can revisit later if we want to standardize across
+  the board"
+
+---
+
+## 1. Purpose
+
+CCCS-Yara is the Canadian Centre for Cyber Security's public YARA rule
+collection. Some ATR Rules cover threats that overlap with CCCS-Yara
+rules — for example, an ATR rule detecting a malicious agent skill
+package may share a SHA-256 indicator with a CCCS-Yara rule detecting
+the dropper binary that delivered it.
+
+When such overlap exists, ATR uses `references.external_references.cccs_yara`
+to cite the corresponding CCCS-Yara rule name. ATR does NOT execute or
+validate the CCCS-Yara rule; the reference is evidence only.
+
+## 2. Format
+
+ATR rule YAML carries the cross-reference under the existing
+`references.external_references.cccs_yara` field (per
+`spec/atr-schema.yaml`):
+
+```yaml
+references:
+  external_references:
+    cccs_yara:
+      - "APT_CN_BEACON_2024"
+      - "Malware_RAT_AsyncRAT"
+```
+
+Values are opaque strings matching the `rule` keyword in the upstream
+`.yar` file at https://github.com/CybercentreCanada/CCCS-Yara. ATR
+authors SHOULD verify the rule name exists in the upstream repository
+at authoring time and SHOULD pin the CCCS-Yara commit hash in
+`references.research` if long-term stability matters:
+
+```yaml
+references:
+  external_references:
+    cccs_yara: ["APT_CN_BEACON_2024"]
+  research:
+    - "CCCS-Yara@5d2f8a (https://github.com/CybercentreCanada/CCCS-Yara/blob/5d2f8a/...)"
+```
+
+## 3. Semantics
+
+The cross-reference is non-normative in either direction:
+
+- ATR engines MUST NOT load, parse, or execute CCCS-Yara rules.
+- CCCS-Yara engines MUST NOT load, parse, or execute ATR rules.
+- The cross-reference is data flowing through SIEM / SOAR / OSCAL
+  pipelines so analysts can pivot between ecosystems.
+
+When an ATR Rule fires and emits a Match (SPEC.md §7), engines MAY
+include the cited `external_references.cccs_yara` entries in the
+Match output to help downstream correlation. The reference does NOT
+guarantee that running CCCS-Yara on the same Input would also fire.
+
+## 4. Versioning
+
+CCCS-Yara rule names are NOT versioned in the upstream repository.
+A rule's content may change while keeping the same name. ATR Rule
+authors SHOULD:
+
+- Pin a commit hash in `references.research` when first authoring the
+  cross-reference.
+- Re-verify the cross-reference annually as part of rule maintenance.
+- Drop the cross-reference (do NOT silently update) if the upstream
+  CCCS-Yara rule changes scope.
+
+## 5. Reverse-direction convention
+
+If CCCS-Yara contributors later choose to cite ATR Rule IDs from their
+side, the recommended field is `metadata.atr_rule_ids` on the upstream
+`.yar` rule. This convention is documented here as a courtesy; the
+authoritative source is CCCS-Yara's own metadata conventions if and
+when they choose to adopt it.
+
+## 6. Example worked cross-reference
+
+A future ATR Rule covering skill-package supply-chain compromise via
+known-malicious SHA-256 indicators:
+
+```yaml
+id: ATR-2026-DRAFT-cccs-cross-ref-example
+title: "Skill package matching CCCS-Yara dropper signature"
+status: draft
+severity: critical
+description: >
+  Detects skill packages whose content hash matches a CCCS-Yara
+  rule for a known dropper. Cross-references the CCCS-Yara rule
+  name as evidence that the indicator is also recognised by the
+  Canadian Cybercentre's public corpus.
+tags:
+  category: skill-compromise
+  scan_target: skill
+detection:
+  method: signature
+  signature:
+    indicators:
+      - type: sha256
+        value: "<hash>"
+        target_field: skill.content
+references:
+  external_references:
+    cccs_yara: ["Malware_Dropper_GenericLoader_2024"]
+  research:
+    - "CCCS-Yara@<commit-hash>"
+response:
+  actions: [block_request, log_alert]
+```
+
+When a Match fires, the Match output (SPEC.md §7) can carry
+`external_references.cccs_yara` so a SOC analyst pivoting from the
+ATR-side detection can immediately query the same SHA-256 against
+the CCCS-Yara corpus.
+
+## 7. Open Items
+
+- No reciprocity yet from CCCS-Yara side. Per cccs-rs's 2026-05-26
+  comment, they may revisit standardization "across the board" once
+  ATR has shipped worked examples. Aim: accumulate ≥10 cross-ref
+  examples over 3-6 months, then re-engage cccs-rs with adoption
+  evidence.
+- Schema slot is intentionally generic. If CCCS-Yara later publishes
+  formal rule IDs (UUIDs / hashes), this convention extends naturally;
+  authors simply use the new identifier format.
+
+## 8. References
+
+- Schema field: `spec/atr-schema.yaml` → `references.external_references.cccs_yara`
+- Closing comment: https://github.com/CybercentreCanada/CCCS-Yara/pull/100
+- ATR ↔ external registry convention: this document