sigmaforge 0.1.0__tar.gz

sigmaforge-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Christian Huhn
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

sigmaforge-0.1.0/PKG-INFO

@@ -0,0 +1,121 @@
+Metadata-Version: 2.4
+Name: sigmaforge
+Version: 0.1.0
+Summary: Honest Sigma-rule backtest harness
+Author-email: Christian Huhn <duathron@gmail.com>
+License-Expression: MIT
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: typer>=0.12.0
+Requires-Dist: rich>=13.7.0
+Requires-Dist: pydantic>=2.7.0
+Requires-Dist: shipwright-kit>=0.8.0
+Dynamic: license-file
+
+# sigmaforge
+
+**Honest Sigma-rule backtest harness.** Measures detection rules against real log
+corpora and reports two numbers per rule — **recall** (does it catch attacks of its
+ATT&CK sub-technique?) and **precision / false-positives** (does it fire on benign
+activity?) — with honesty gates that return `unmeasured` instead of a fake `0` or a
+tautological `1.0` when the data can't support a number.
+
+[![CI](https://github.com/duathron/sigmaforge/actions/workflows/ci.yml/badge.svg)](https://github.com/duathron/sigmaforge/actions/workflows/ci.yml)
+![python](https://img.shields.io/badge/python-3.11%2B-blue)
+![license](https://img.shields.io/badge/license-MIT-green)
+
+> [!NOTE]
+> **Learning / portfolio project, built by directing AI coding agents.** Christian
+> Huhn (photography → SOC career change) designed, reviewed, and gated the work; the
+> implementation was AI-pair-programmed. It is an honest measurement harness, not a
+> polished product — see *Status* below for exactly what works and what doesn't yet.
+
+## What problem it solves
+
+Every SOC ships dozens to hundreds of detection rules and rarely measures them.
+sigmaforge answers two questions with reproducible evidence:
+
+- **Which rules are noise generators?** (high false-positives on legitimate activity)
+- **Which rules catch nothing?** (zero recall against real attacks of their technique)
+
+Example finding from a real run: *Suspicious Windows Service Tampering* produced 66
+false-positives on a benign corpus — every one a Ninite / TeamViewer installer, not
+an attack.
+
+## How it actually works
+
+```mermaid
+flowchart LR
+    R[SigmaHQ rules] -->|partition high/critical| C[compile to one Zircolite ruleset]
+    C --> E[Zircolite engine]
+    A[(attack corpus<br/>sub-technique-labeled)] --> E
+    B[(benign corpus<br/>Nextron + OpTC)] --> E
+    E --> S[score: recall per technique<br/>+ precision/FP label-aware]
+    S --> G[honesty gates<br/>floor · positive-control · no-self-review]
+    G --> O[report.md + manifest]
+```
+
+The real pipeline is **script-driven** (`scripts/run6_backtest.py` is the current
+end-to-end path):
+
+```bash
+uv run python scripts/compile_loaded_ruleset.py   # rules -> one Zircolite ruleset
+uv run python scripts/run6_backtest.py            # backtest -> reports/run6.md
+```
+
+> [!WARNING]
+> The shipped CLI (`sigmaforge backtest`) is a **weaker, work-in-progress path** and
+> is not the way the real reports were produced. Use the scripts above. The CLI is
+> kept for the future one-command experience, not parity.
+
+## Status
+
+| Area | State |
+|------|-------|
+| Recall (per sub-technique, no sibling dilution) | **Working** — 338/609 rules measurable, 70 fire (run5) |
+| Precision / false-positives (label-aware, gated) | **Working** — 7/609 measurable on current benign corpus (run6) |
+| Honesty gates (floor, positive-control, no-self-review) | **Working** |
+| Reproducible manifest (run_hash, corpus SHAs, provenance) | **Working** |
+| One-command CLI (`sigmaforge backtest`) | **WIP** — weaker than the scripts |
+| Self-generated benign corpus | **Kit ready** (`scripts/selfgen/`), needs a Windows VM run |
+
+> [!IMPORTANT]
+> **The log corpora are not shipped.** They are large, separately licensed, and
+> gitignored. `pip install sigmaforge` installs the harness code, not the data — a
+> full end-to-end backtest needs the corpora and a local Zircolite checkout (also not
+> bundled). The package is useful as a library / reference; the runnable pipeline
+> needs the local setup documented in `scripts/`.
+
+## Install
+
+```bash
+pip install sigmaforge
+```
+
+Installs the harness package and the `sigmaforge` CLI. The detection engine
+([Zircolite](https://github.com/wagga40/Zircolite)) and the log corpora are obtained
+separately (see above).
+
+## Corpora used (all verified, portfolio-safe licenses)
+
+| Corpus | Role | License |
+|--------|------|---------|
+| [splunk/attack_data](https://github.com/splunk/attack_data) | recall (sub-technique-labeled attacks) | Apache-2.0 |
+| [DARPA OpTC](https://github.com/FiveDirections/OpTC-data) | precision (real enterprise benign week) | Public domain |
+| [NextronSystems/evtx-baseline](https://github.com/NextronSystems/evtx-baseline) | precision (goodware baseline) | Apache-2.0 |
+| Self-generated (`scripts/selfgen/`) | precision (targeted admin/LOLBin noise) | your own lab |
+
+## Development
+
+Built with the [Shipwright](https://github.com/duathron/shipwright) dev framework.
+
+```bash
+uv sync --dev
+uv run pytest        # 108 tests
+uv run ruff check .
+```
+
+## License
+
+MIT © Christian Huhn. Corpus data retains its upstream license (see table above).

sigmaforge-0.1.0/README.md

@@ -0,0 +1,106 @@
+# sigmaforge
+
+**Honest Sigma-rule backtest harness.** Measures detection rules against real log
+corpora and reports two numbers per rule — **recall** (does it catch attacks of its
+ATT&CK sub-technique?) and **precision / false-positives** (does it fire on benign
+activity?) — with honesty gates that return `unmeasured` instead of a fake `0` or a
+tautological `1.0` when the data can't support a number.
+
+[![CI](https://github.com/duathron/sigmaforge/actions/workflows/ci.yml/badge.svg)](https://github.com/duathron/sigmaforge/actions/workflows/ci.yml)
+![python](https://img.shields.io/badge/python-3.11%2B-blue)
+![license](https://img.shields.io/badge/license-MIT-green)
+
+> [!NOTE]
+> **Learning / portfolio project, built by directing AI coding agents.** Christian
+> Huhn (photography → SOC career change) designed, reviewed, and gated the work; the
+> implementation was AI-pair-programmed. It is an honest measurement harness, not a
+> polished product — see *Status* below for exactly what works and what doesn't yet.
+
+## What problem it solves
+
+Every SOC ships dozens to hundreds of detection rules and rarely measures them.
+sigmaforge answers two questions with reproducible evidence:
+
+- **Which rules are noise generators?** (high false-positives on legitimate activity)
+- **Which rules catch nothing?** (zero recall against real attacks of their technique)
+
+Example finding from a real run: *Suspicious Windows Service Tampering* produced 66
+false-positives on a benign corpus — every one a Ninite / TeamViewer installer, not
+an attack.
+
+## How it actually works
+
+```mermaid
+flowchart LR
+    R[SigmaHQ rules] -->|partition high/critical| C[compile to one Zircolite ruleset]
+    C --> E[Zircolite engine]
+    A[(attack corpus<br/>sub-technique-labeled)] --> E
+    B[(benign corpus<br/>Nextron + OpTC)] --> E
+    E --> S[score: recall per technique<br/>+ precision/FP label-aware]
+    S --> G[honesty gates<br/>floor · positive-control · no-self-review]
+    G --> O[report.md + manifest]
+```
+
+The real pipeline is **script-driven** (`scripts/run6_backtest.py` is the current
+end-to-end path):
+
+```bash
+uv run python scripts/compile_loaded_ruleset.py   # rules -> one Zircolite ruleset
+uv run python scripts/run6_backtest.py            # backtest -> reports/run6.md
+```
+
+> [!WARNING]
+> The shipped CLI (`sigmaforge backtest`) is a **weaker, work-in-progress path** and
+> is not the way the real reports were produced. Use the scripts above. The CLI is
+> kept for the future one-command experience, not parity.
+
+## Status
+
+| Area | State |
+|------|-------|
+| Recall (per sub-technique, no sibling dilution) | **Working** — 338/609 rules measurable, 70 fire (run5) |
+| Precision / false-positives (label-aware, gated) | **Working** — 7/609 measurable on current benign corpus (run6) |
+| Honesty gates (floor, positive-control, no-self-review) | **Working** |
+| Reproducible manifest (run_hash, corpus SHAs, provenance) | **Working** |
+| One-command CLI (`sigmaforge backtest`) | **WIP** — weaker than the scripts |
+| Self-generated benign corpus | **Kit ready** (`scripts/selfgen/`), needs a Windows VM run |
+
+> [!IMPORTANT]
+> **The log corpora are not shipped.** They are large, separately licensed, and
+> gitignored. `pip install sigmaforge` installs the harness code, not the data — a
+> full end-to-end backtest needs the corpora and a local Zircolite checkout (also not
+> bundled). The package is useful as a library / reference; the runnable pipeline
+> needs the local setup documented in `scripts/`.
+
+## Install
+
+```bash
+pip install sigmaforge
+```
+
+Installs the harness package and the `sigmaforge` CLI. The detection engine
+([Zircolite](https://github.com/wagga40/Zircolite)) and the log corpora are obtained
+separately (see above).
+
+## Corpora used (all verified, portfolio-safe licenses)
+
+| Corpus | Role | License |
+|--------|------|---------|
+| [splunk/attack_data](https://github.com/splunk/attack_data) | recall (sub-technique-labeled attacks) | Apache-2.0 |
+| [DARPA OpTC](https://github.com/FiveDirections/OpTC-data) | precision (real enterprise benign week) | Public domain |
+| [NextronSystems/evtx-baseline](https://github.com/NextronSystems/evtx-baseline) | precision (goodware baseline) | Apache-2.0 |
+| Self-generated (`scripts/selfgen/`) | precision (targeted admin/LOLBin noise) | your own lab |
+
+## Development
+
+Built with the [Shipwright](https://github.com/duathron/shipwright) dev framework.
+
+```bash
+uv sync --dev
+uv run pytest        # 108 tests
+uv run ruff check .
+```
+
+## License
+
+MIT © Christian Huhn. Corpus data retains its upstream license (see table above).

sigmaforge-0.1.0/pyproject.toml

@@ -0,0 +1,36 @@
+[build-system]
+requires = ["setuptools>=77.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "sigmaforge"
+version = "0.1.0"
+description = "Honest Sigma-rule backtest harness"
+readme = "README.md"
+license = "MIT"
+requires-python = ">=3.11"
+authors = [{ name = "Christian Huhn", email = "duathron@gmail.com" }]
+dependencies = [
+    "typer>=0.12.0",
+    "rich>=13.7.0",
+    "pydantic>=2.7.0",
+    "shipwright-kit>=0.8.0",  # PyPI release (PyPI rejects git+ direct-URL deps)
+]
+
+[tool.shipwright]
+preset = "security"
+
+[project.scripts]
+sigmaforge = "sigmaforge.main:main"
+
+[dependency-groups]
+dev = ["pytest>=8.0", "ruff>=0.8", "pre-commit>=4.0", "hypothesis>=6.0"]
+
+[tool.setuptools.packages.find]
+include = ["sigmaforge*"]
+
+[tool.ruff]
+extend = "tooling/ruff-base.toml"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

sigmaforge-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

sigmaforge-0.1.0/sigmaforge/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.1.0"

sigmaforge-0.1.0/sigmaforge/backtest/runner.py

@@ -0,0 +1,21 @@
+from concurrent.futures import ProcessPoolExecutor
+
+from sigmaforge.ingest.chunker import chunk_lines
+from sigmaforge.records import MatchRecord
+
+
+def aggregate(shard_results: list[list[MatchRecord]]) -> set[MatchRecord]:
+    merged: set[MatchRecord] = set()
+    for s in shard_results:
+        merged.update(s)  # set union = order-independent, dedup across shard boundaries
+    return merged
+
+
+def backtest(items, shard_size, workers, shard_fn) -> set[MatchRecord]:
+    shards = list(chunk_lines(items, shard_size))
+    if workers == 1:
+        results = [shard_fn(s) for s in shards]
+    else:
+        with ProcessPoolExecutor(max_workers=workers) as ex:
+            results = list(ex.map(shard_fn, shards))
+    return aggregate(results)

sigmaforge-0.1.0/sigmaforge/banner.py

@@ -0,0 +1,15 @@
+"""sigmaforge banner — uses the Shipwright design system."""
+
+from __future__ import annotations
+
+import sys
+
+from shipwright_kit.design.banner import make_banner
+
+from sigmaforge import __version__
+
+
+def show_banner(*, quiet: bool = False) -> None:
+    if quiet or not sys.stderr.isatty():
+        return
+    print(make_banner("sigmaforge", __version__, "Honest Sigma-rule backtest harness"), file=sys.stderr)

sigmaforge-0.1.0/sigmaforge/config.py

@@ -0,0 +1,34 @@
+"""sigmaforge configuration: ~/.sigmaforge/config.yaml + env > defaults."""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Optional
+
+import yaml
+from pydantic import BaseModel
+from shipwright_kit.config import app_dir, load_config
+
+_APP_DIR = app_dir("sigmaforge")
+
+
+class OutputConfig(BaseModel):
+    default_format: str = "rich"
+
+
+class AppConfig(BaseModel):
+    output: OutputConfig = OutputConfig()
+
+
+def _load_yaml(path: Path) -> dict:
+    with open(path) as f:
+        return yaml.safe_load(f) or {}
+
+
+def load(config_path: Optional[Path] = None) -> AppConfig:
+    """Resolve config: explicit > ~/.sigmaforge/config.yaml > ./config.yaml > defaults."""
+    return load_config(
+        [config_path, _APP_DIR / "config.yaml", Path("config.yaml")],
+        loader=_load_yaml,
+        validator=AppConfig.model_validate,
+    )

sigmaforge-0.1.0/sigmaforge/crosscheck/chainsaw.py

@@ -0,0 +1,12 @@
+def compare_loaded_intersection(z_hits, c_hits, z_loaded, c_loaded) -> dict:
+    """A6 cross-engine integrity: compare Zircolite vs Chainsaw ONLY over rules BOTH engines
+    loaded. Rules only one engine loaded are a load artifact, reported separately — never as
+    detection disagreement. z_hits/c_hits: dict[rule -> set[event_id]]."""
+    both = z_loaded & c_loaded
+    agree = {r for r in both if z_hits.get(r, set()) == c_hits.get(r, set())}
+    return {
+        "compared_rules": both,
+        "agree": agree,
+        "disagree": both - agree,
+        "load_artifact_only": z_loaded ^ c_loaded,  # symmetric difference = load artifact
+    }

sigmaforge-0.1.0/sigmaforge/detect.py

@@ -0,0 +1,42 @@
+"""Example parse boundary — classify an untrusted input string.
+
+Framework input-contract rule: external/garbage input never raises;
+unrecognized input returns "unknown". Replace with your real logic.
+"""
+
+from __future__ import annotations
+
+import re
+
+_IPV4 = re.compile(r"^(?:\d{1,3}\.){3}\d{1,3}$")
+_HASH = re.compile(r"^[A-Fa-f0-9]{32,64}$")
+_DOMAIN = re.compile(r"^(?=.{1,253}$)(?!-)[A-Za-z0-9-]{1,63}(?:\.[A-Za-z]{2,})+$")
+# File extensions that look like TLDs but are not valid domains for our purposes.
+_FILE_EXTS = re.compile(
+    r"\.(dll|exe|so|dylib|sys|bin|bat|cmd|sh|ps1"
+    r"|py|js|ts|rb|go|rs|c|cpp|h|java|class|jar|war"
+    r"|zip|tar|gz|bz2|xz|7z|rar"
+    r"|pdf|doc|docx|xls|xlsx|ppt|pptx"
+    r"|png|jpg|jpeg|gif|svg|ico|mp3|mp4|avi|mov|mkv"
+    r"|log|txt|csv|json|xml|yaml|yml|toml|ini|cfg|conf|env)$",
+    re.IGNORECASE,
+)
+
+
+def classify(value: object) -> str:
+    """Return a coarse type for value. Never raises; unknown -> "unknown"."""
+    if not isinstance(value, str):
+        return "unknown"
+    v = value.strip()
+    if not v:
+        return "unknown"
+    if _IPV4.match(v):
+        parts = v.split(".")
+        if all(p.isdigit() and 0 <= int(p) <= 255 for p in parts):
+            return "ipv4"
+        return "unknown"
+    if _HASH.match(v):
+        return "hash"
+    if _DOMAIN.match(v) and not _FILE_EXTS.search(v):
+        return "domain"
+    return "unknown"

sigmaforge-0.1.0/sigmaforge/ingest/chunker.py

@@ -0,0 +1,22 @@
+from typing import Iterator, Sequence, TypeVar
+
+T = TypeVar("T")
+
+
+def chunk_lines(items: Sequence[T], shard_size: int) -> Iterator[list[T]]:
+    """Partition items into chunks of shard_size.
+
+    Args:
+        items: Sequence to partition
+        shard_size: Size of each chunk (must be >= 1)
+
+    Yields:
+        Lists of items, each of size shard_size (except possibly the last chunk)
+
+    Raises:
+        ValueError: If shard_size < 1
+    """
+    if shard_size < 1:
+        raise ValueError("shard_size must be >= 1")
+    for i in range(0, len(items), shard_size):
+        yield list(items[i : i + shard_size])

sigmaforge-0.1.0/sigmaforge/ingest/ruleload.py

@@ -0,0 +1,16 @@
+def _is_stateful(rule: dict) -> bool:
+    if "correlation" in rule:
+        return True
+    cond = str(rule.get("detection", {}).get("condition", ""))
+    return any(tok in cond for tok in ("count(", "sum(", "avg(", "| near", "temporal"))
+
+
+def select_by_level(rules: list[dict], levels: tuple[str, ...]) -> list[dict]:
+    return [r for r in rules if str(r.get("level", "")).lower() in levels]
+
+
+def partition_rules(rules: list[dict], levels: tuple[str, ...] = ("high", "critical")) -> tuple[list[dict], list[dict]]:
+    in_scope = select_by_level(rules, levels)
+    loaded = [r for r in in_scope if not _is_stateful(r)]
+    excluded = [r for r in in_scope if _is_stateful(r)]
+    return loaded, excluded

sigmaforge-0.1.0/sigmaforge/ingest/zircolite_runner.py

@@ -0,0 +1,92 @@
+import hashlib
+import json
+import subprocess
+import tempfile
+
+from sigmaforge.records import MatchRecord
+
+ZIRCOLITE = ["uv", "run", "python", "Zircolite/zircolite.py"]  # vendored 3.7.6
+
+
+def _stable_event_id(row: dict) -> str:
+    """Globally-unique event key (fix C). EventRecordID is a PER-EVTX-FILE counter, so using it
+    alone collapses record 42 of fileA with record 42 of fileB across a multi-file attack run and
+    silently deflates recall. Hash the whole flattened row instead: it carries Computer/UtcTime/
+    Image/CommandLine/... which differ across files even when EventRecordID repeats. Two genuinely
+    identical events still hash-collapse — that is correct dedup, not a collision bug.
+    (On real data each row also carries Zircolite's autoincrement `row_id`, globally unique
+    across a single multi-file run, so real events never over-split. NB: if `--parallel`
+    ingestion is ever enabled, `row_id` resets per chunk — uniqueness then rests on the
+    content fields, UtcTime/ProcessGuid/etc., which the whole-row hash already includes.)"""
+    canonical = json.dumps(row, sort_keys=True, default=str)
+    return hashlib.sha1(canonical.encode()).hexdigest()
+
+
+def parse_detections(
+    detections: list[dict],
+    corpus_label: str | None = None,
+    file_technique_map: dict[str, str] | None = None,
+    event_technique_out: dict[str, str] | None = None,
+) -> list[MatchRecord]:
+    """Parse Zircolite detections into MatchRecords.
+
+    FIX B: when ``file_technique_map`` (source-EVTX basename -> ATT&CK technique) and
+    ``event_technique_out`` are supplied, also populate ``event_technique_out`` mapping
+    each fired event's ``event_id`` -> its ground-truth technique. The technique is keyed
+    on the SAME identity the engine emits (``_stable_event_id``), so a fire and its
+    technique join correctly downstream. The source file is read from each match row's
+    ``OriginalLogfile`` (the EVTX basename, set by Zircolite's streaming flattener)."""
+    out: list[MatchRecord] = []
+    for d in detections:
+        for m in d.get("matches", []):
+            # benign COMISET rows carry the injected hash; native-EVTX rows do NOT -> derive a
+            # globally-unique key from the row (NOT bare EventRecordID, which collides across files).
+            eid = m.get("sigmaforge_eid") or _stable_event_id(m)
+            label = m.get("sigmaforge_label") or corpus_label or "benign"
+            out.append(MatchRecord(rule_id=d["title"], event_id=str(eid), event_label=label))
+            if file_technique_map is not None and event_technique_out is not None:
+                src = m.get("OriginalLogfile")
+                tech = file_technique_map.get(src) if src else None
+                if tech:
+                    event_technique_out[str(eid)] = tech
+    return out
+
+
+def run_shard(
+    events_path: str,
+    ruleset_glob: str,
+    mapping_path: str | None = None,
+    json_input: bool = True,
+    xml_input: bool = False,
+    corpus_label: str | None = None,
+    file_technique_map: dict[str, str] | None = None,
+    event_technique_out: dict[str, str] | None = None,
+) -> list[MatchRecord]:
+    """Run Zircolite over a shard and parse detections.
+
+    FIX B: pass ``file_technique_map`` + ``event_technique_out`` to also harvest the
+    per-event ground-truth technique (see ``parse_detections``).
+
+    FIX B3: ``xml_input=True`` ingests EVTX-converted-to-XML files (one wrapped
+    ``<Events>...</Events>`` document per file, Zircolite ``--xml-input``). Like the
+    native-EVTX path, each event's ``OriginalLogfile`` is set to the .xml basename,
+    so the same ``file_technique_map`` (basename -> (sub-)technique) recall join
+    applies. ``json_input`` and ``xml_input`` are mutually exclusive."""
+    if json_input and xml_input:
+        raise ValueError("json_input and xml_input are mutually exclusive")
+    out = tempfile.NamedTemporaryFile(suffix=".json", delete=False).name
+    cmd = [*ZIRCOLITE, "--events", events_path, "--ruleset", ruleset_glob, "--outfile", out]
+    if json_input:
+        cmd += ["--json-input"]
+    if xml_input:
+        cmd += ["--xml-input"]
+    if mapping_path:
+        cmd += ["--config", mapping_path]
+    subprocess.run(cmd, check=True, cwd="/Users/christianhuhn/PycharmProjects/ai_project1/sigmaforge")
+    with open(out) as fh:
+        return parse_detections(
+            json.load(fh),
+            corpus_label=corpus_label,
+            file_technique_map=file_technique_map,
+            event_technique_out=event_technique_out,
+        )

sigmaforge-0.1.0/sigmaforge/main.py

@@ -0,0 +1,73 @@
+"""sigmaforge — CLI entry point."""
+
+from __future__ import annotations
+
+from rich.console import Console
+from shipwright_kit.cli import build_typer
+
+from sigmaforge import __version__
+from sigmaforge.detect import classify as classify_input
+
+app = build_typer("sigmaforge", "Honest Sigma-rule backtest harness", version=__version__)
+console = Console()
+
+
+@app.command()
+def classify(value: str) -> None:
+    """Classify an input string (example parse boundary)."""
+    console.print(classify_input(value))
+
+
+@app.command()
+def backtest(
+    rules: str,
+    attack: str,
+    benign: str,
+    out: str,
+    mapping: str = "data/mappings/comiset.yaml",
+    workers: int = 4,
+    min_events: int = 1000,
+    attack_events: int = 0,
+) -> None:
+    """Backtest Sigma rules: recall on the native-EVTX attack corpus, precision@COMISET on
+    the benign corpus. Writes the FP-tuning report to OUT. (Live end-to-end run; meaningful
+    precision numbers require the COMISET benign sample.)"""
+    import json
+    from pathlib import Path
+
+    import yaml
+
+    from sigmaforge.ingest.ruleload import partition_rules
+    from sigmaforge.ingest.zircolite_runner import run_shard
+    from sigmaforge.orchestrate import run_backtest
+
+    rule_docs = [
+        doc
+        for p in Path(rules).rglob("*.yml")
+        for doc in [yaml.safe_load(p.read_text())]
+        if isinstance(doc, dict) and doc.get("title")
+    ]
+    loaded, _excluded = partition_rules(rule_docs)
+    benign_events = [json.loads(line) for line in Path(benign).read_text().splitlines()]
+    attack_fires = set(run_shard(attack, rules, json_input=False, corpus_label="malicious"))
+    benign_fires = set(run_shard(benign, rules, mapping_path=mapping, json_input=True))
+    pc_fired = any(f.event_label == "malicious" for f in benign_fires)
+    _rows, _funnel, md = run_backtest(
+        loaded,
+        attack_fires,
+        benign_fires,
+        benign_events,
+        n_attack_events=attack_events,  # attack-corpus event count = recall denominator (provide via --attack-events)
+        positive_control_fired=pc_fired,
+        min_events=min_events,
+    )
+    Path(out).write_text(md)
+    console.print(f"report written: {out}")
+
+
+def main() -> None:
+    app()
+
+
+if __name__ == "__main__":
+    main()