ciphergate-cli 0.13.0__tar.gz → 0.14.0__tar.gz

{ciphergate_cli-0.13.0 → ciphergate_cli-0.14.0}/CHANGELOG.md

@@ -6,6 +6,22 @@ versioning: [PEP 440](https://peps.python.org/pep-0440/) with SemVer meaning (pr
 
 ## [Unreleased]
 
+## [0.14.0] - 2026-11-02
+
+Adds `--extension-format` for mixed directories and tests the optional NER detector against a
+real Presidio install. No existing behaviour changes.
+
+### Added
+
+- `--extension-format EXT=FORMAT` on `inspect`, `plan`, `sanitize` and `rescan` (repeatable) reads
+  files with that extension as FORMAT, for directories and single files. It adds unknown
+  extensions or replaces built-in ones, and cannot be combined with `--format`. The Python API
+  takes `RuntimeSettings(extension_formats=...)`.
+- Tests that run the optional NER detector against a real Presidio install with the
+  `en_core_web_sm` model, including a `sanitize` run with network access disabled. They skip
+  when Presidio is not installed. A new non-required CI job, `ner`, installs the extra and a
+  hash-pinned model and runs them.
+
 ## [0.13.0] - 2026-11-02
 
 Fewer random IDs are reported as payment cards. Cards with an unrecognised prefix and no card
@@ -363,7 +379,8 @@ Pre-release (alpha). Licensed under Apache-2.0. Not a final release: the securit
 See `docs/limitations.md`. Deferred: optional NER/Presidio detectors, stdin/stdout streaming,
 directory processing, structured log parsers, key files, reversible token vault, lossy decoding.
 
-[Unreleased]: https://github.com/aashishkumar-tech/ciphergate/compare/v0.13.0...HEAD
+[Unreleased]: https://github.com/aashishkumar-tech/ciphergate/compare/v0.14.0...HEAD
+[0.14.0]: https://github.com/aashishkumar-tech/ciphergate/compare/v0.13.0...v0.14.0
 [0.13.0]: https://github.com/aashishkumar-tech/ciphergate/compare/v0.12.0...v0.13.0
 [0.12.0]: https://github.com/aashishkumar-tech/ciphergate/compare/v0.11.0...v0.12.0
 [0.11.0]: https://github.com/aashishkumar-tech/ciphergate/compare/v0.10.0...v0.11.0

{ciphergate_cli-0.13.0 → ciphergate_cli-0.14.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: ciphergate-cli
-Version: 0.13.0
+Version: 0.14.0
 Summary: Local-first, environment- and destination-aware inspection and sanitization of logs and test datasets.
 Project-URL: Homepage, https://github.com/aashishkumar-tech/ciphergate
 Project-URL: Source, https://github.com/aashishkumar-tech/ciphergate
@@ -50,7 +50,7 @@ vendor ticket. CipherGate replaces that with a local, explainable, scriptable ga
 inspect  ->  plan  ->  sanitize --apply  ->  rescan
 ```
 
-> **Status: 0.13.0 (alpha).** No detector finds every sensitive value. A passing run means the
+> **Status: 0.14.0 (alpha).** No detector finds every sensitive value. A passing run means the
 > configured processing completed under the selected policy; it is **not** proof that the output
 > contains no sensitive information, and it is not a compliance (GDPR, HIPAA, PCI DSS, ...) or
 > anonymization guarantee. Policy templates must be reviewed before use.
@@ -295,6 +295,10 @@ ciphergate sanitize logs/ --policy ... --output build/logs.sanitized --apply
   `rescan` fails (exit 7) when it skipped any, and `sanitize` leaves them out of the output.
   `--format` on a directory applies to every file, and the run warns when that contradicts an
   extension (for example a CSV read as text loses its column names).
+- `--extension-format EXT=FORMAT` (repeatable) reads files with that extension as FORMAT, for
+  example `--extension-format .dat=csv --extension-format .log=jsonlog`. It adds unknown extensions
+  and replaces built-in ones, works for single files too, and cannot be combined with `--format`.
+  From Python, pass `RuntimeSettings(extension_formats={".dat": InputFormat.CSV})`.
   At most 1000 files by default (`--max-files N`); more fails the run (exit 4) before anything
   is processed. An empty selection fails with `batch_empty` (exit 4).
 - Each file is processed exactly like a single-file run, in sorted path order; a failing file

{ciphergate_cli-0.13.0 → ciphergate_cli-0.14.0}/README.md

@@ -11,7 +11,7 @@ vendor ticket. CipherGate replaces that with a local, explainable, scriptable ga
 inspect  ->  plan  ->  sanitize --apply  ->  rescan
 ```
 
-> **Status: 0.13.0 (alpha).** No detector finds every sensitive value. A passing run means the
+> **Status: 0.14.0 (alpha).** No detector finds every sensitive value. A passing run means the
 > configured processing completed under the selected policy; it is **not** proof that the output
 > contains no sensitive information, and it is not a compliance (GDPR, HIPAA, PCI DSS, ...) or
 > anonymization guarantee. Policy templates must be reviewed before use.
@@ -256,6 +256,10 @@ ciphergate sanitize logs/ --policy ... --output build/logs.sanitized --apply
   `rescan` fails (exit 7) when it skipped any, and `sanitize` leaves them out of the output.
   `--format` on a directory applies to every file, and the run warns when that contradicts an
   extension (for example a CSV read as text loses its column names).
+- `--extension-format EXT=FORMAT` (repeatable) reads files with that extension as FORMAT, for
+  example `--extension-format .dat=csv --extension-format .log=jsonlog`. It adds unknown extensions
+  and replaces built-in ones, works for single files too, and cannot be combined with `--format`.
+  From Python, pass `RuntimeSettings(extension_formats={".dat": InputFormat.CSV})`.
   At most 1000 files by default (`--max-files N`); more fails the run (exit 4) before anything
   is processed. An empty selection fails with `batch_empty` (exit 4).
 - Each file is processed exactly like a single-file run, in sorted path order; a failing file

{ciphergate_cli-0.13.0 → ciphergate_cli-0.14.0}/docs/detector-coverage.md

@@ -28,8 +28,12 @@ Needs `pip install 'ciphergate-cli[ner]'` and a spaCy model installed by the use
 `python -m spacy download en_core_web_lg`). CipherGate checks that the model package is importable
 before importing Presidio and fails with `ner_model_missing` instead of letting Presidio download
 it. Strings with no letters or under two characters are not sent to the model. Tests
-(`tests/unit/test_presidio_ner.py`) use a fake engine. Presidio itself is **not** exercised in
-CI, so accuracy on real text has not been measured by this project.
+in `tests/unit/test_presidio_ner.py` use a fake engine. `tests/integration/test_presidio_real.py`
+runs the real Presidio with `en_core_web_sm`: it finds a common synthetic name and place, ignores
+text with no entities, removes them in a `sanitize` run with network access disabled, and fails
+closed when the model is missing. CI runs it in the `ner` job (not a required check). These are
+smoke tests on a few sentences: accuracy on real text has not been measured by this project,
+and the small model misses names that the larger `en_core_web_lg` may find.
 
 ## Text/log specifics
 

{ciphergate_cli-0.13.0 → ciphergate_cli-0.14.0}/src/ciphergate/__init__.py

@@ -24,7 +24,7 @@ Supported public API (everything else is internal and may change without notice)
     rescan_directory(root, policy, context=None, *, recursive=False, strict=None) -> BatchReport
 """
 
-__version__ = "0.13.0"
+__version__ = "0.14.0"
 
 from ciphergate.config import RuntimeSettings
 from ciphergate.constants import (

{ciphergate_cli-0.13.0 → ciphergate_cli-0.14.0}/src/ciphergate/cli.py

@@ -117,6 +117,15 @@ PolicyOpt = Annotated[Path, typer.Option("--policy", "-p", help="Policy YAML fil
 FormatOpt = Annotated[
     InputFormat | None, typer.Option("--format", help="Input format (default: from extension).")
 ]
+ExtFormatOpt = Annotated[
+    list[str] | None,
+    typer.Option(
+        "--extension-format",
+        metavar="EXT=FORMAT",
+        help="Read files with this extension as FORMAT, e.g. .dat=csv or .log=jsonlog. "
+        "Repeatable. Cannot be combined with --format.",
+    ),
+]
 ReportOpt = Annotated[
     Path | None, typer.Option("--report", help="Write the privacy-safe report to this file.")
 ]
@@ -192,18 +201,52 @@ def _runtime(
     encoding: str | None,
     path_display: PathDisplayChoice | None,
     max_files: int | None = None,
+    extension_formats: list[str] | None = None,
+    fmt: InputFormat | None = None,
 ) -> RuntimeSettings:
+    mapping = _parse_extension_formats(extension_formats or [])
+    if mapping and fmt is not None:
+        raise UsageError(
+            "use either --format (one format for every file) or --extension-format, not both",
+            code="format_conflict",
+        )
     try:
         return RuntimeSettings().with_overrides(
             max_input_bytes=max_bytes,
             encoding=encoding,
             path_display=str(path_display) if path_display else None,
             max_batch_files=max_files,
+            extension_formats=mapping or None,
         )
     except ValueError as exc:
         raise UsageError(str(exc), code="invalid_runtime_setting") from None
 
 
+def _parse_extension_formats(values: list[str]) -> dict[str, InputFormat]:
+    mapping: dict[str, InputFormat] = {}
+    for value in values:
+        ext, sep, name = value.partition("=")
+        ext = ext.strip().lower()
+        if not ext.startswith("."):
+            ext = "." + ext
+        try:
+            fmt = InputFormat(name.strip().lower())
+        except ValueError:
+            fmt = None
+        if not sep or fmt is None:
+            raise UsageError(
+                "--extension-format needs EXT=FORMAT with FORMAT one of " + "|".join(InputFormat),
+                code="invalid_extension_format",
+            )
+        if ext in mapping and mapping[ext] != fmt:
+            raise UsageError(
+                "--extension-format maps the same extension to two formats",
+                code="invalid_extension_format",
+            )
+        mapping[ext] = fmt
+    return mapping
+
+
 def _err(message: str) -> None:
     sys.stderr.write(escape_control(message)[:2000] + "\n")
 
@@ -501,6 +544,7 @@ def inspect_command(
     ],
     policy: PolicyOpt,
     fmt: FormatOpt = None,
+    extension_format: ExtFormatOpt = None,
     report: ReportOpt = None,
     report_format: ReportFormatOpt = None,
     environment: EnvOpt = None,
@@ -531,7 +575,7 @@ def inspect_command(
             report,
             report_format,
             _context(environment, target_environment, destination, use_case, profile),
-            _runtime(max_bytes, encoding, path_display, max_files),
+            _runtime(max_bytes, encoding, path_display, max_files, extension_format, fmt),
             quiet,
             verbose,
             force,
@@ -547,6 +591,7 @@ def plan_command(
     ],
     policy: PolicyOpt,
     fmt: FormatOpt = None,
+    extension_format: ExtFormatOpt = None,
     report: ReportOpt = None,
     report_format: ReportFormatOpt = None,
     environment: EnvOpt = None,
@@ -577,7 +622,7 @@ def plan_command(
             report,
             report_format,
             _context(environment, target_environment, destination, use_case, profile),
-            _runtime(max_bytes, encoding, path_display, max_files),
+            _runtime(max_bytes, encoding, path_display, max_files, extension_format, fmt),
             quiet,
             verbose,
             force,
@@ -604,6 +649,7 @@ def sanitize_command(
         bool, typer.Option("--apply", help="Actually write output (else plan).")
     ] = False,
     fmt: FormatOpt = None,
+    extension_format: ExtFormatOpt = None,
     report: ReportOpt = None,
     report_format: ReportFormatOpt = None,
     environment: EnvOpt = None,
@@ -636,7 +682,9 @@ def sanitize_command(
                 "--output - uses stdout for data; write the JSON report with --report PATH",
                 code="stdout_conflict",
             )
-        runtime = _runtime(max_bytes, encoding, path_display)
+        runtime = _runtime(
+            max_bytes, encoding, path_display, extension_formats=extension_format, fmt=fmt
+        )
         _check_report_path(report, [p for p in (input_path, output) if not is_stdio(p)], force)
         loaded = load_policy(policy)
         options = SanitizeOptions(apply=apply, force=force, format=selected_fmt, runtime=runtime)
@@ -689,7 +737,7 @@ def sanitize_command(
             raise UsageError(
                 "a directory INPUT needs --output DIR, not stdout", code="stdout_unsupported"
             )
-        runtime = _runtime(max_bytes, encoding, path_display, max_files)
+        runtime = _runtime(max_bytes, encoding, path_display, max_files, extension_format, fmt)
         _check_batch_report_path(report, [input_path, output], force)
         batch = sanitize_directory(
             input_path,
@@ -731,6 +779,7 @@ def rescan_command(
         ),
     ] = None,
     fmt: FormatOpt = None,
+    extension_format: ExtFormatOpt = None,
     report: ReportOpt = None,
     report_format: ReportFormatOpt = None,
     environment: EnvOpt = None,
@@ -764,7 +813,7 @@ def rescan_command(
             report,
             report_format,
             _context(environment, target_environment, destination, use_case, profile),
-            _runtime(max_bytes, encoding, path_display, max_files),
+            _runtime(max_bytes, encoding, path_display, max_files, extension_format, fmt),
             quiet,
             verbose,
             force,

{ciphergate_cli-0.13.0 → ciphergate_cli-0.14.0}/src/ciphergate/config.py

@@ -2,10 +2,14 @@
 
 from __future__ import annotations
 
+import re
 from dataclasses import dataclass, field, replace
 from typing import Literal
 
+from ciphergate.constants import InputFormat
+
 PathDisplay = Literal["basename", "redacted", "full"]
+_EXTENSION_RE = re.compile(r"^\.[a-z0-9_-]{1,16}$")
 
 MIB = 1024 * 1024
 
@@ -27,6 +31,9 @@ class RuntimeSettings:
     encoding: str = "utf-8"
     csv_delimiter: str = ","
     path_display: PathDisplay = "basename"
+    # Extra or replacement extension -> format mappings, e.g. {".dat": csv}. Keys are
+    # lower-case single suffixes. An explicit --format still wins for every file.
+    extension_formats: dict[str, InputFormat] = field(default_factory=dict)
     # Stored for completeness of the run report; never changes decisions.
     extra: dict[str, str] = field(default_factory=dict)
 
@@ -44,6 +51,12 @@ class RuntimeSettings:
                 raise ValueError(f"{name} must be a positive integer")
         if len(self.csv_delimiter) != 1 or self.csv_delimiter in '\r\n"':
             raise ValueError("csv_delimiter must be a single character other than quote or newline")
+        for ext in self.extension_formats:
+            if not _EXTENSION_RE.match(ext):
+                raise ValueError(
+                    "extension_formats keys must look like '.dat' (a dot, then 1-16 "
+                    "lower-case letters, digits, '_' or '-')"
+                )
 
     def with_overrides(self, **changes: object) -> RuntimeSettings:
         clean = {k: v for k, v in changes.items() if v is not None}

{ciphergate_cli-0.13.0 → ciphergate_cli-0.14.0}/src/ciphergate/pipeline/batch.py

@@ -24,7 +24,7 @@ import shutil
 import stat
 import sys
 import tempfile
-from collections.abc import Callable
+from collections.abc import Callable, Mapping
 from dataclasses import dataclass, field
 from pathlib import Path
 from typing import Any
@@ -171,7 +171,14 @@ def _is_link(entry: os.DirEntry[str]) -> bool:
     return False
 
 
-def discover(root: Path, *, recursive: bool, fmt: InputFormat | None, max_files: int) -> Discovery:
+def discover(
+    root: Path,
+    *,
+    recursive: bool,
+    fmt: InputFormat | None,
+    max_files: int,
+    extensions: Mapping[str, InputFormat] | None = None,
+) -> Discovery:
     """List the files a batch run will process. Raises on limits; never follows links."""
     try:
         st = os.lstat(root)
@@ -210,7 +217,7 @@ def discover(root: Path, *, recursive: bool, fmt: InputFormat | None, max_files:
                         walk(rel, depth + 1)
                 elif not entry.is_file(follow_symlinks=False):
                     skipped.append((rel, "not_regular_file"))
-                elif fmt is None and rel.suffix.lower() not in EXTENSION_FORMATS:
+                elif fmt is None and not _known_extension(rel, extensions):
                     skipped.append((rel, "unsupported_extension"))
                 else:
                     files.append(rel)
@@ -226,6 +233,11 @@ def discover(root: Path, *, recursive: bool, fmt: InputFormat | None, max_files:
     return Discovery(root, sorted(files, key=lambda p: p.as_posix()), skipped)
 
 
+def _known_extension(rel: Path, extensions: Mapping[str, InputFormat] | None) -> bool:
+    suffix = rel.suffix.lower()
+    return suffix in EXTENSION_FORMATS or suffix in (extensions or {})
+
+
 def display_relative(rel: Path, index: int, root: Path, mode: str) -> str:
     if mode == "redacted":
         return f"<redacted:{index:04d}>{rel.suffix}"
@@ -291,7 +303,8 @@ def coverage_warnings(
             f"{unknown} file(s) were skipped and not scanned because their extension is not "
             "recognised"
             + ("; they are not in the output" if sanitize else "")
-            + "; rename them, or pass --format to read every file as one format"
+            + "; map them with --extension-format EXT=FORMAT, or pass --format to read every "
+            "file as one format"
         )
     if fmt is not None:
         mismatched = sum(
@@ -332,7 +345,13 @@ def analyze_directory(
 ) -> BatchReport:
     """Batch ``inspect`` / ``plan`` / ``rescan``. Read-only, like the single-file commands."""
     root = Path(root)
-    discovery = discover(root, recursive=recursive, fmt=fmt, max_files=runtime.max_batch_files)
+    discovery = discover(
+        root,
+        recursive=recursive,
+        fmt=fmt,
+        max_files=runtime.max_batch_files,
+        extensions=runtime.extension_formats,
+    )
     _require_files(discovery)
 
     def run(path: Path, rel: Path) -> RunReport:
@@ -471,7 +490,11 @@ def sanitize_directory(
     runtime = options.runtime or RuntimeSettings()
     validate_output_directory(root, output)
     discovery = discover(
-        root, recursive=recursive, fmt=options.format, max_files=runtime.max_batch_files
+        root,
+        recursive=recursive,
+        fmt=options.format,
+        max_files=runtime.max_batch_files,
+        extensions=runtime.extension_formats,
     )
     _require_files(discovery)
     mode = runtime.path_display

{ciphergate_cli-0.13.0 → ciphergate_cli-0.14.0}/src/ciphergate/pipeline/inspector.py

@@ -85,7 +85,7 @@ def prepare_run(
     ctx = resolve_context(policy, context)
     path = Path(input_path)
     validate_input_path(path, runtime.max_input_bytes)
-    detected = detect_format(path, fmt)
+    detected = detect_format(path, fmt, runtime.extension_formats)
     options = ReaderOptions(
         runtime=runtime,
         policy_settings=policy.document.settings,

{ciphergate_cli-0.13.0 → ciphergate_cli-0.14.0}/src/ciphergate/readers/__init__.py

@@ -2,6 +2,7 @@
 
 from __future__ import annotations
 
+from collections.abc import Mapping
 from pathlib import Path
 from typing import IO
 
@@ -49,11 +50,19 @@ _JSON_PREFIXES = (
 )
 
 
-def detect_format(path: Path, explicit: InputFormat | None = None) -> InputFormat:
-    """Explicit ``--format`` wins; else use the extension, refusing contradicting content."""
+def detect_format(
+    path: Path,
+    explicit: InputFormat | None = None,
+    extensions: Mapping[str, InputFormat] | None = None,
+) -> InputFormat:
+    """Explicit ``--format`` wins; then ``extensions``, then the built-in extension table.
+
+    Content that contradicts a JSON format is refused either way.
+    """
     if explicit is not None:
         return explicit
-    fmt = EXTENSION_FORMATS.get(path.suffix.lower())
+    suffix = path.suffix.lower()
+    fmt = (extensions or {}).get(suffix) or EXTENSION_FORMATS.get(suffix)
     if fmt is None:
         raise UsageError(
             "cannot infer the input format from this file extension; pass --format "

{ciphergate_cli-0.13.0 → ciphergate_cli-0.14.0}/tests/integration/test_batch.py

@@ -405,3 +405,94 @@ def test_directory_summary_carries_next_steps(tmp_path: Path) -> None:
     (root / "app.log").write_text(SECRET_LINE, encoding="utf-8")
     r = invoke("inspect", root, "--policy", POLICIES / "developer_logs.yml")
     assert r.exit_code == 1 and "next: pass --environment" in r.stdout
+
+
+# --------------------------------------------------------------------------- --extension-format
+
+
+def _mixed(tmp_path: Path) -> Path:
+    root = tmp_path / "in"
+    root.mkdir()
+    (root / "users.dat").write_text("email\nalice@example.com\n", encoding="utf-8")
+    (root / "app.log").write_text(
+        '{"msg": "started", "user": "bob@example.com"}\n', encoding="utf-8"
+    )
+    return root
+
+
+def test_extension_format_adds_an_extension_to_discovery(tmp_path: Path) -> None:
+    root = _mixed(tmp_path)
+    assert [
+        p.as_posix() for p in discover(root, recursive=False, fmt=None, max_files=10).files
+    ] == ["app.log"]
+    mapped = discover(
+        root, recursive=False, fmt=None, max_files=10, extensions={".dat": InputFormat.CSV}
+    )
+    assert [p.as_posix() for p in mapped.files] == ["app.log", "users.dat"] and not mapped.skipped
+
+
+def test_extension_format_sets_each_files_format(tmp_path: Path, env) -> None:
+    root = _mixed(tmp_path)
+    report = tmp_path / "report.json"
+    r = invoke(
+        "inspect", root, *PRD_EXTERNAL, "--report", report,
+        "--extension-format", ".dat=csv", "--extension-format", "LOG=jsonlog",
+        env=env,
+    )  # fmt: skip
+    assert r.exit_code in (0, 1), r.stdout
+    formats = {f["path"]: f["report"]["input"]["format"] for f in _report(report)["files"]}
+    assert formats == {"app.log": "jsonlog", "users.dat": "csv"}
+    assert not _report(report)["skipped"]
+
+
+def test_extension_format_works_for_a_single_file(tmp_path: Path, env) -> None:
+    src = _mixed(tmp_path) / "users.dat"
+    plain = invoke("inspect", src, *PRD_EXTERNAL, env=env)
+    assert plain.exit_code == 2 and "format_required" in plain.stdout + plain.stderr
+    report = tmp_path / "report.json"
+    mapped = invoke(
+        "inspect", src, *PRD_EXTERNAL, "--extension-format", ".dat=csv", "--report", report, env=env
+    )
+    assert mapped.exit_code in (0, 1) and _report(report)["input"]["format"] == "csv"
+
+
+@pytest.mark.parametrize(
+    ("args", "code"),
+    [
+        (["--format", "text", "--extension-format", ".dat=csv"], "format_conflict"),
+        (["--extension-format", ".dat=xml"], "invalid_extension_format"),
+        (["--extension-format", ".dat"], "invalid_extension_format"),
+        (
+            ["--extension-format", ".dat=csv", "--extension-format", ".dat=text"],
+            "invalid_extension_format",
+        ),
+        (["--extension-format", "../x=csv"], "invalid_runtime_setting"),
+    ],
+)
+def test_bad_extension_format_is_a_usage_error(
+    args: list[str], code: str, tmp_path: Path, env
+) -> None:
+    r = invoke("inspect", _mixed(tmp_path), *PRD_EXTERNAL, *args, env=env)
+    assert r.exit_code == 2 and code in r.stdout + r.stderr
+
+
+@pytest.mark.parametrize("command", ["plan", "rescan"])
+def test_extension_format_on_other_commands(command: str, tmp_path: Path, env) -> None:
+    root = _mixed(tmp_path)
+    report = tmp_path / "report.json"
+    invoke(
+        command, root, *PRD_EXTERNAL, "--extension-format", ".dat=csv", "--report", report, env=env
+    )
+    assert [f["path"] for f in _report(report)["files"]] == ["app.log", "users.dat"]
+
+
+def test_sanitize_directory_with_extension_format(tmp_path: Path, env) -> None:
+    root = _mixed(tmp_path)
+    out = tmp_path / "out"
+    r = invoke(
+        "sanitize", root, "--policy", POLICIES / "dev_local.yml", "-o", out, "--apply",
+        "--extension-format", ".dat=csv", env=env,
+    )  # fmt: skip
+    assert r.exit_code == 0, r.stdout
+    assert sorted(p.name for p in out.iterdir()) == ["app.log", "users.dat"]
+    assert "alice@example.com" not in (out / "users.dat").read_text(encoding="utf-8")

ciphergate_cli-0.14.0/tests/integration/test_presidio_real.py

@@ -0,0 +1,138 @@
+"""The optional NER detector against a real Presidio install (skipped unless one is present).
+
+The unit tests in ``tests/unit/test_presidio_ner.py`` use a fake engine. These run the real
+``presidio-analyzer`` with the small ``en_core_web_sm`` spaCy model. Install both yourself to run
+them (CI does this in the ``ner`` job); CipherGate never downloads a model. The NER model is
+statistical, so the assertions use common, clearly person-like synthetic names and check that
+something was found, not exact scores.
+"""
+
+from __future__ import annotations
+
+import importlib.util
+import json
+import os
+import socket
+from collections.abc import Iterator
+from pathlib import Path
+
+import pytest
+from typer.testing import CliRunner
+
+from ciphergate.cli import app
+from ciphergate.detectors import presidio_ner
+from ciphergate.detectors.presidio_ner import PresidioDetector, load_analyzer
+from ciphergate.models.finding import ScanItem
+from ciphergate.models.policy import PresidioSettings
+from tests.conftest import MINIMAL_POLICY
+
+MODEL = "en_core_web_sm"
+# CI sets this so a broken install fails the job instead of silently skipping every test.
+REQUIRED = os.environ.get("CIPHERGATE_REQUIRE_NER") == "1"
+
+pytestmark = [
+    pytest.mark.skipif(
+        not REQUIRED and importlib.util.find_spec("presidio_analyzer") is None,
+        reason="optional 'ner' extra (presidio-analyzer) not installed",
+    ),
+    pytest.mark.skipif(
+        not REQUIRED and importlib.util.find_spec(MODEL) is None,
+        reason=f"spaCy model {MODEL} not installed",
+    ),
+]
+
+NAME = "John Smith"  # synthetic; a common name the small model recognises reliably
+TEXT = f"ticket opened by {NAME} in London after the export failed\n"
+
+NER_POLICY = MINIMAL_POLICY.replace(
+    "settings: {strict: false}\n",
+    "settings: {strict: false}\n"
+    "detectors:\n"
+    f"  presidio: {{enabled: true, model: {MODEL}, min_score: 0.5}}\n",
+).replace(
+    "defaults:\n",
+    "  - id: redact-coarse-location\n"
+    "    priority: 100\n"
+    "    match: {categories: [location.coarse]}\n"
+    "    action: redact\n"
+    "defaults:\n",
+)
+
+runner = CliRunner()
+
+
+@pytest.fixture
+def fresh_analyzer() -> Iterator[None]:
+    # Build the engine inside each test, so the no-network test covers construction too.
+    load_analyzer.cache_clear()
+    yield
+    load_analyzer.cache_clear()
+
+
+def _item(text: str) -> ScanItem:
+    return ScanItem(record_index=0, line=1, field_path="", field_name=None, value=text)
+
+
+def test_real_engine_finds_a_person_and_a_place(fresh_analyzer: None) -> None:
+    detector = PresidioDetector(PresidioSettings(enabled=True, model=MODEL, min_score=0.5))
+    findings = detector.detect(_item(TEXT))
+    spans = {
+        TEXT[f.location.span[0] : f.location.span[1]]: f.category
+        for f in findings
+        if f.location.span
+    }
+    assert spans.get(NAME) == "pii.person_name"
+    assert spans.get("London") == "location.coarse"
+    assert all(f.detector == "presidio_ner" and 0 < f.confidence <= 1 for f in findings)
+
+
+def test_real_engine_ignores_text_without_entities(fresh_analyzer: None) -> None:
+    detector = PresidioDetector(PresidioSettings(enabled=True, model=MODEL, min_score=0.5))
+    assert detector.detect(_item("export failed with status 500 after 3 retries")) == []
+
+
+def test_sanitize_with_real_ner_removes_the_name_offline(
+    fresh_analyzer: None, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
+) -> None:
+    def deny(*_: object, **__: object) -> None:
+        raise AssertionError("network access attempted")
+
+    for name in ("socket", "create_connection", "getaddrinfo", "gethostbyname"):
+        monkeypatch.setattr(socket, name, deny)
+
+    policy = tmp_path / "ner.yml"
+    policy.write_text(NER_POLICY, encoding="utf-8")
+    src = tmp_path / "app.log"
+    src.write_text(TEXT, encoding="utf-8")
+    out, report = tmp_path / "out.log", tmp_path / "report.json"
+    r = runner.invoke(
+        app,
+        [
+            *("sanitize", str(src), "--policy", str(policy), "-o", str(out)),
+            *("--report", str(report), "--apply"),
+        ],
+    )
+    assert r.exit_code == 0, r.stdout
+    sanitized = out.read_text(encoding="utf-8")
+    assert NAME not in sanitized and "London" not in sanitized
+    assert "[REDACTED:pii.person_name]" in sanitized
+    assert "export failed" in sanitized  # the debugging context survives
+    data = json.loads(report.read_text(encoding="utf-8"))
+    assert data["summary"]["by_category"].get("pii.person_name", 0) >= 1
+    for text in (r.stdout, report.read_text(encoding="utf-8")):
+        assert NAME not in text and "London" not in text
+
+
+def test_missing_model_fails_closed_with_the_real_package(
+    fresh_analyzer: None, tmp_path: Path
+) -> None:
+    policy = tmp_path / "ner.yml"
+    policy.write_text(NER_POLICY.replace(MODEL, "en_core_web_not_installed"), encoding="utf-8")
+    src = tmp_path / "app.log"
+    src.write_text(TEXT, encoding="utf-8")
+    out = tmp_path / "out.log"
+    args = ["sanitize", str(src), "--policy", str(policy), "-o", str(out), "--apply"]
+    r = runner.invoke(app, args)
+    assert r.exit_code == 6 and "ner_model_missing" in r.stdout + r.stderr
+    assert not out.exists()
+    assert presidio_ner._installed("presidio_analyzer")