ciphergate-cli 0.8.0__tar.gz → 0.10.0__tar.gz

{ciphergate_cli-0.8.0 → ciphergate_cli-0.10.0}/CHANGELOG.md

@@ -6,6 +6,42 @@ versioning: [PEP 440](https://peps.python.org/pep-0440/) with SemVer meaning (pr
 
 ## [Unreleased]
 
+## [0.10.0] - 2026-10-30
+
+Fixes a password leak in nested values and detects more credentials and identifiers. A run that
+passed before can now redact, review or block, so re-check your policies before upgrading a gate.
+
+### Added
+
+- `ssn_pattern` detector for dashed US Social Security numbers (`pii.government_id`). They were
+  labelled `pii.phone`, so government-ID rules did not apply and a strict template can now block.
+
+### Fixed
+
+- `secret_pattern` no longer stops at `{` for nested values such as Actuator dumps, which left
+  the real value in the output.
+- `secret_pattern` now finds camelCase password names, `Cookie` and `Set-Cookie` headers, and
+  session and CSRF cookie values. `luhn_card` finds labelled CVV and CVC values.
+- Fewer false alarms: shell `PWD` paths, code such as `settings.X` or `request.Y` on the right of
+  an assignment, and an HTTP status followed by a byte count.
+
+## [0.9.0] - 2026-10-30
+
+Detects more values. A run that passed before can now redact, review or block, so re-check your
+policies before upgrading a gate.
+
+### Added
+
+- `mac_address` detector (`pii.device_identifier`) and `iban` detector (`pii.bank_account`, mod-97
+  check). Both are on by default and can be turned off with `detectors.disabled`.
+
+### Changed
+
+- `email_regex` also finds percent-encoded addresses (`%40`) in text, and `secret_pattern` finds
+  values assigned to `PGPASSWORD`, `MYSQL_PWD` and similar names (`DBPASSWORD`, `ROOTPASSWORD`).
+  These values now follow your policy rules, so a run can redact, review or block content it
+  used to pass.
+
 ## [0.8.0] - 2026-10-30
 
 Adds the packaged policy templates to the Python API. Existing behaviour is unchanged.
@@ -279,7 +315,9 @@ Pre-release (alpha). Licensed under Apache-2.0. Not a final release: the securit
 See `docs/limitations.md`. Deferred: optional NER/Presidio detectors, stdin/stdout streaming,
 directory processing, structured log parsers, key files, reversible token vault, lossy decoding.
 
-[Unreleased]: https://github.com/aashishkumar-tech/ciphergate/compare/v0.8.0...HEAD
+[Unreleased]: https://github.com/aashishkumar-tech/ciphergate/compare/v0.10.0...HEAD
+[0.10.0]: https://github.com/aashishkumar-tech/ciphergate/compare/v0.9.0...v0.10.0
+[0.9.0]: https://github.com/aashishkumar-tech/ciphergate/compare/v0.8.0...v0.9.0
 [0.8.0]: https://github.com/aashishkumar-tech/ciphergate/compare/v0.7.1...v0.8.0
 [0.7.1]: https://github.com/aashishkumar-tech/ciphergate/compare/v0.7.0...v0.7.1
 [0.7.0]: https://github.com/aashishkumar-tech/ciphergate/compare/v0.6.1...v0.7.0

{ciphergate_cli-0.8.0 → ciphergate_cli-0.10.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: ciphergate-cli
-Version: 0.8.0
+Version: 0.10.0
 Summary: Local-first, environment- and destination-aware inspection and sanitization of logs and test datasets.
 Project-URL: Homepage, https://github.com/aashishkumar-tech/ciphergate
 Project-URL: Source, https://github.com/aashishkumar-tech/ciphergate
@@ -50,7 +50,7 @@ vendor ticket. CipherGate replaces that with a local, explainable, scriptable ga
 inspect  ->  plan  ->  sanitize --apply  ->  rescan
 ```
 
-> **Status: 0.8.0 (alpha).** No detector finds every sensitive value. A passing run means the
+> **Status: 0.10.0 (alpha).** No detector finds every sensitive value. A passing run means the
 > configured processing completed under the selected policy; it is **not** proof that the output
 > contains no sensitive information, and it is not a compliance (GDPR, HIPAA, PCI DSS, ...) or
 > anonymization guarantee. Policy templates must be reviewed before use.

{ciphergate_cli-0.8.0 → ciphergate_cli-0.10.0}/README.md

@@ -11,7 +11,7 @@ vendor ticket. CipherGate replaces that with a local, explainable, scriptable ga
 inspect  ->  plan  ->  sanitize --apply  ->  rescan
 ```
 
-> **Status: 0.8.0 (alpha).** No detector finds every sensitive value. A passing run means the
+> **Status: 0.10.0 (alpha).** No detector finds every sensitive value. A passing run means the
 > configured processing completed under the selected policy; it is **not** proof that the output
 > contains no sensitive information, and it is not a compliance (GDPR, HIPAA, PCI DSS, ...) or
 > anonymization guarantee. Policy templates must be reviewed before use.

{ciphergate_cli-0.8.0 → ciphergate_cli-0.10.0}/docs/detector-coverage.md

@@ -9,12 +9,15 @@ are engineering estimates mapped to bands (`low` < 0.5 <= `medium` < 0.8 <= `hig
 |---|---|---|---|---|---|
 | `field_name` | field names -> many categories | `email`, `contactEmail`, `api_key`, `date-of-birth`, `userId`, `trace_id` | `token_count`, `key_name`, `monkey`, `product_name`, `email_verified` | 0.45–0.85 per term | heuristic dictionary; benign values in sensitive-looking fields are flagged; sensitive data in generic fields (`notes`, `value`) is missed unless content detectors match |
 | `policy_classification` | policy-declared fields | `admission_date` (healthcare template) | | configured | only as good as the policy |
-| `email_regex` | `pii.email` | `alice.test@example.com` | `a..b@example.com`, `user@localhost` | 0.9 | ASCII only; no IDN; obfuscated forms (`alice at example dot com`) missed |
+| `email_regex` | `pii.email` | `alice.test@example.com`, `alice.test%40example.com` | `a..b@example.com`, `user@localhost` | 0.9 | ASCII only; no IDN; obfuscated forms (`alice at example dot com`) missed; `%40` is the only percent-encoding recognised outside `accesslog` |
 | `phone_pattern` | `pii.phone` | `+1 202 555 0143` (high), `(202) 555-0143` (medium) | dates, IPs, card numbers, repeated digits, < 7 digits | 0.3 bare digits, 0.55–0.8 prefixed/formatted, +0.2 keyword | no numbering-plan validation; IDs can look like phones (low band) |
 | `ip_address` | `pii.ip_address` (global), `network.telemetry` (private/loopback/link-local) | `8.8.8.8`, `10.0.0.1`, `2606:4700::1111`, `fe80::1%eth0` | `256.1.1.1`, `std::vector`, `12:30:45` | 0.85 / 0.8 | version strings like `1.2.3.4` match; documentation ranges flagged with evidence |
-| `luhn_card` | `pii.payment_card` | `4111 1111 1111 1111` | Luhn-invalid, mixed separators, repeated digits | 0.4 Luhn-only, 0.8 issuer prefix, +0.15 keyword | random IDs pass Luhn ~10% (low band); cards split across fields missed |
+| `luhn_card` | `pii.payment_card` | `4111 1111 1111 1111` | Luhn-invalid, mixed separators, repeated digits | 0.4 Luhn-only, 0.8 issuer prefix, +0.15 keyword; labelled CVV/CVC values 0.85 | random IDs pass Luhn ~10% (low band); cards split across fields missed |
+| `mac_address` | `pii.device_identifier` | `00:1A:2B:3C:4D:5E`, `00-1a-2b-3c-4d-5e`, `001a.2b3c.4d5e` | all-zero, broadcast, mixed separators, `12:30:45` | 0.85, 0.6 when all digits | bare 12-hex values are missed; digit-only IDs in this shape are flagged |
+| `iban` | `pii.bank_account` | a mod-97-valid IBAN, compact or space-grouped | wrong check digits, lowercase | 0.9 | country length and national format are not checked; about 1 in 97 random strings of this shape pass; lowercase forms are missed |
+| `ssn_pattern` | `pii.government_id` | `219-09-9999` | never-issued areas (`000`, `666`, `9xx`), group `00`, serial `0000`, undashed | 0.8, 0.95 with a keyword | US only; undashed numbers are missed; other dashed IDs in this shape are flagged |
 | `date_pattern` | `pii.date_of_birth` with field/keyword evidence; `date.generic` if enabled | `dob: 1990-04-12`, field `birth_date` | `created 2026-01-05` (no DOB evidence), invalid dates | 0.85 / 0.6 | `DD/MM` vs `MM/DD` ambiguity flagged; free-text DOBs without keywords missed |
-| `secret_pattern` | `credential.secret` | PEM blocks, `Bearer ...`, `AKIA...`, `ghp_...`, `xox?-`, `sk_test_...`, `AIza...`, JWTs, `scheme://u:pw@`, `Password=...;`, `password=`, `api_key:`, `token=` | `password=`, `password: null`, markers, `******` | 0.8–0.99 | base64/hex-wrapped, split, custom-format or high-entropy secrets without keywords are **missed**; prose like `password: required` is flagged |
+| `secret_pattern` | `credential.secret` | PEM blocks, `Bearer ...`, `AKIA...`, `ghp_...`, `xox?-`, `sk_test_...`, `AIza...`, JWTs, `scheme://u:pw@`, `Password=...;`, `password=`, assignments to `PGPASSWORD`, `MYSQL_PWD` or camelCase `...Password` names, `Cookie` and `Set-Cookie` headers, session and CSRF cookie values, `api_key:`, `token=` | `password=`, `password: null`, markers, `******` | 0.8–0.99 | base64/hex-wrapped, split, custom-format or high-entropy secrets without keywords are **missed**; prose like `password: required` is flagged |
 | `internal_host` | `network.telemetry` | `api-1.prod.internal`, `db.corp`, `localhost` | `example.com`, `a.internalx` | 0.75 | internal names without listed suffixes missed; configurable |
 | `custom:<id>` | configured | | | configured | regex-safety checks are heuristics |
 | `presidio_ner` (optional, off by default) | Presidio entities -> mapped categories (default `PERSON` -> `pii.person_name`, `LOCATION` -> `location.coarse`) | `ticket opened by <a person's name>` | numbers, single characters, CipherGate markers/pseudonyms | Presidio score, filtered by `min_score` | statistical model: misses and false positives, language/model dependent, slow; requires `ciphergate-cli[ner]` plus a locally installed spaCy model |

{ciphergate_cli-0.8.0 → ciphergate_cli-0.10.0}/docs/policy-schema.md

@@ -52,7 +52,8 @@ A pinned value combined with a different caller value fails with
 ## Detectors
 
 `disabled` (built-in IDs: `field_name`, `secret_pattern`, `luhn_card`, `ip_address`,
-`date_pattern`, `email_regex`, `internal_host`, `phone_pattern`), `dates_emit_generic` (report
+`date_pattern`, `email_regex`, `internal_host`, `phone_pattern`, `mac_address`, `iban`, `ssn_pattern`),
+`dates_emit_generic` (report
 non-DOB dates as `date.generic`), `internal_host_suffixes` (default `.internal .local .corp .lan
 .intranet .localdomain`), `custom` (list of `{id, category, pattern, confidence, ignore_case}`;
 patterns are checked for compile errors, empty matches, nested quantifiers, backreferences and

{ciphergate_cli-0.8.0 → ciphergate_cli-0.10.0}/pyproject.toml

@@ -110,6 +110,8 @@ ignore = [
 [tool.ruff.lint.per-file-ignores]
 "tests/**" = ["S101", "S105", "S106", "S108", "S311", "PLC0415", "S607"]
 "scripts/**" = ["S607", "T201"]
+# Reproducible synthetic data, and optional tools that may not be installed.
+"scripts/benchmark/**" = ["S311", "PLC0415", "S607", "T201"]
 
 [tool.mypy]
 python_version = "3.11"

ciphergate_cli-0.10.0/scripts/benchmark/README.md

@@ -0,0 +1,39 @@
+# Detection benchmark
+
+Measures what CipherGate catches and flags wrongly on generated, labelled log lines, next to other
+local tools. It is an engineering aid for finding gaps. It is not part of the package, CI or the
+test suite, and it never contacts a service.
+
+## Do not use the results as a public claim
+
+- The corpus is written by the people who wrote the detectors, so it leans toward CipherGate.
+- The data is synthetic. Real logs behave differently.
+- Other tools run on default settings, and their results depend on versions and models.
+- A high score here does not show that sensitive data is gone from any file.
+
+## Run it
+
+Use a separate virtual environment, because these tools are not project dependencies:
+
+```bash
+python -m venv /path/outside/repo/venv
+# Activate it, then:
+python -m pip install . detect-secrets scrubadub presidio-analyzer
+python -m pip install en_core_web_sm  # see the spaCy docs for the model install command
+python scripts/benchmark/compare_tools.py --out /path/outside/repo/results
+```
+
+Only CipherGate is required. Any other tool that is not installed is skipped and listed under
+`not run`. To include gitleaks, install it yourself and pass `--gitleaks PATH`.
+
+`--out` must be outside the repository, because `corpus.jsonl` holds format-shaped fake
+credentials that must not be committed. The tool refuses a path inside the repository. Results
+are repeatable for the same `--seed` and tool versions.
+
+## Reading the output
+
+Each row is a data class, and each cell is caught/total. A positive counts only if the finding
+overlaps the planted value; for CipherGate, the value must be gone from the output. `NEGATIVE`
+counts false alarms on clean lines. `presidio@0.5` is Presidio with a 0.5 score threshold.
+
+Random 16-digit IDs pass the card checksum about 10% of the time, so `NEGATIVE` varies by seed.

{ciphergate_cli-0.8.0 → ciphergate_cli-0.10.0}/src/ciphergate/__init__.py

@@ -24,7 +24,7 @@ Supported public API (everything else is internal and may change without notice)
     rescan_directory(root, policy, context=None, *, recursive=False, strict=None) -> BatchReport
 """
 
-__version__ = "0.8.0"
+__version__ = "0.10.0"
 
 from ciphergate.config import RuntimeSettings
 from ciphergate.constants import (

{ciphergate_cli-0.8.0 → ciphergate_cli-0.10.0}/src/ciphergate/detectors/email.py

@@ -12,6 +12,12 @@ _EMAIL_RE = re.compile(
     r"([A-Za-z0-9._%+-]{1,64})@((?:[A-Za-z0-9-]{1,63}\.){1,8}[A-Za-z]{2,63})"
     r"(?![A-Za-z0-9-])"
 )
+# Percent-encoded form in URLs and query strings: the "@" is written as %40.
+_ENCODED_EMAIL_RE = re.compile(
+    r"(?<![A-Za-z0-9._%+-])"
+    r"([A-Za-z0-9._%+-]{1,64})%40((?:[A-Za-z0-9-]{1,63}\.){1,8}[A-Za-z]{2,63})"
+    r"(?![A-Za-z0-9-])"
+)
 
 RESERVED_EMAIL_DOMAINS = ("example.com", "example.org", "example.net")
 RESERVED_EMAIL_TLDS = (".invalid", ".example", ".test", ".localhost")
@@ -31,10 +37,10 @@ class EmailDetector:
 
     def detect(self, item: ScanItem) -> list[Finding]:
         text = item.text
-        if not text or "@" not in text:
+        if not text or ("@" not in text and "%40" not in text):
             return []
         out: list[Finding] = []
-        for m in _EMAIL_RE.finditer(text):
+        for m in (*_EMAIL_RE.finditer(text), *_ENCODED_EMAIL_RE.finditer(text)):
             local, domain = m.group(1), m.group(2)
             if local.startswith(".") or local.endswith(".") or ".." in local or ".." in domain:
                 continue

ciphergate_cli-0.10.0/src/ciphergate/detectors/iban.py

@@ -0,0 +1,53 @@
+"""IBAN detector: compact or space-grouped, uppercase, validated with the ISO 7064 mod-97 check.
+
+Country-specific lengths and national formats are not checked. Lowercase forms are not matched.
+"""
+
+from __future__ import annotations
+
+import re
+
+from ciphergate.detectors.base import make_finding
+from ciphergate.models.finding import Finding, ScanItem
+
+_IBAN_RE = re.compile(
+    r"(?<![0-9A-Za-z])"
+    r"([A-Z]{2}\d{2}(?:[A-Z0-9]{11,30}|(?: [A-Z0-9]{4}){2,7}(?: [A-Z0-9]{1,3})?))"
+    r"(?![0-9A-Za-z])"
+)
+
+
+def iban_valid(compact: str) -> bool:
+    if not 15 <= len(compact) <= 34:
+        return False
+    rearranged = compact[4:] + compact[:4]
+    return int("".join(str(int(c, 36)) for c in rearranged)) % 97 == 1
+
+
+class IbanDetector:
+    id = "iban"
+
+    def detect(self, item: ScanItem) -> list[Finding]:
+        text = item.text
+        if not text or len(text) < 15:
+            return []
+        out: list[Finding] = []
+        for m in _IBAN_RE.finditer(text):
+            # A following uppercase word can join a spaced match, so retry without trailing groups.
+            groups = m.group(1).split(" ")
+            while groups:
+                if iban_valid("".join(groups)):
+                    end = m.start(1) + len(" ".join(groups))
+                    out.append(
+                        make_finding(
+                            item,
+                            category="pii.bank_account",
+                            detector=self.id,
+                            confidence=0.9,
+                            evidence=("pattern_match", "checksum_valid"),
+                            span=(m.start(1), end),
+                        )
+                    )
+                    break
+                groups.pop()
+        return out

ciphergate_cli-0.10.0/src/ciphergate/detectors/mac_address.py

@@ -0,0 +1,47 @@
+"""MAC address detector: six hex pairs with one separator, or Cisco dotted form.
+
+All-digit forms are medium confidence because they can be other identifiers. The all-zero and
+broadcast addresses identify no device and are skipped.
+"""
+
+from __future__ import annotations
+
+import re
+
+from ciphergate.detectors.base import make_finding
+from ciphergate.models.finding import Finding, ScanItem
+
+_MAC_RE = re.compile(
+    r"(?<![0-9A-Fa-f:.-])"
+    r"([0-9A-Fa-f]{2}(?:[:-][0-9A-Fa-f]{2}){5}|[0-9A-Fa-f]{4}(?:\.[0-9A-Fa-f]{4}){2})"
+    r"(?![0-9A-Za-z:.-])"
+)
+_NON_DEVICE = frozenset({"0" * 12, "f" * 12})
+
+
+class MacAddressDetector:
+    id = "mac_address"
+
+    def detect(self, item: ScanItem) -> list[Finding]:
+        text = item.text
+        if not text or len(text) < 14:
+            return []
+        out: list[Finding] = []
+        for m in _MAC_RE.finditer(text):
+            raw = m.group(1)
+            if len({c for c in raw if c in ":-."}) > 1:
+                continue
+            compact = re.sub(r"[:.-]", "", raw).lower()
+            if compact in _NON_DEVICE:
+                continue
+            out.append(
+                make_finding(
+                    item,
+                    category="pii.device_identifier",
+                    detector=self.id,
+                    confidence=0.85 if not compact.isdigit() else 0.6,
+                    evidence=("pattern_match",),
+                    span=(m.start(1), m.end(1)),
+                )
+            )
+        return out

{ciphergate_cli-0.8.0 → ciphergate_cli-0.10.0}/src/ciphergate/detectors/payment_card.py

@@ -12,6 +12,9 @@ from ciphergate.detectors.base import make_finding, preceding_context
 from ciphergate.models.finding import Finding, ScanItem
 
 _CARD_RE = re.compile(r"(?<![0-9A-Za-z_])(\d(?:[ -]?\d){12,18})(?![0-9A-Za-z_])")
+_CVV_RE = re.compile(
+    r"(?i)(?<![a-z0-9])(?:cvv2?|cvc2?)(?:\\?[\"'])?\s{0,3}[:=]\s{0,3}(?:\\?[\"'])?(\d{3,4})(?!\d)"
+)
 _KEYWORDS = ("card", "visa", "mastercard", "amex", "pan", "cc", "credit", "debit", "payment")
 
 
@@ -45,9 +48,21 @@ class PaymentCardDetector:
 
     def detect(self, item: ScanItem) -> list[Finding]:
         text = item.text
-        if not text or sum(c.isdigit() for c in text) < 13:
+        if not text:
             return []
-        out: list[Finding] = []
+        out = [
+            make_finding(
+                item,
+                category="pii.payment_card",
+                detector=self.id,
+                confidence=0.85,
+                evidence=("pattern_match", "context_keyword"),
+                span=(m.start(1), m.end(1)),
+            )
+            for m in _CVV_RE.finditer(text)
+        ]
+        if sum(c.isdigit() for c in text) < 13:
+            return out
         for m in _CARD_RE.finditer(text):
             raw = m.group(1)
             digits = raw.replace(" ", "").replace("-", "")

{ciphergate_cli-0.8.0 → ciphergate_cli-0.10.0}/src/ciphergate/detectors/phone.py

@@ -21,6 +21,7 @@ _PHONE_RE = re.compile(
 )
 _DATE_LIKE = re.compile(r"^\d{4}[-./]\d{1,2}[-./]\d{1,2}$|^\d{1,2}[-./]\d{1,2}[-./]\d{2,4}$")
 _SEPARATED = re.compile(r"^\(?\d{2,4}\)?[ .-]\d{2,5}[ .-]\d{2,5}(?:[ .-]\d{2,5})?$")
+_HTTP_STATUS_BYTES = re.compile(r"^[1-5]\d{2} \d{1,12}$")
 _KEYWORDS = ("phone", "tel", "mobile", "cell", "call", "fax", "sms", "whatsapp", "contact")
 
 
@@ -40,6 +41,9 @@ class PhoneDetector:
                 continue
             if _DATE_LIKE.match(core.strip()):
                 continue
+            context = preceding_context(text, m.start(), _KEYWORDS)
+            if _HTTP_STATUS_BYTES.match(core.strip()) and not context:
+                continue
             evidence = ["pattern_match"]
             if core.startswith("+"):
                 confidence = 0.8 if len(digits) >= 8 else 0.55
@@ -49,7 +53,7 @@ class PhoneDetector:
                 evidence.append("separator_layout")
             else:
                 confidence = 0.3
-            if preceding_context(text, m.start(), _KEYWORDS):
+            if context:
                 confidence += 0.2
                 evidence.append("context_keyword")
             if "x" in candidate.lower() or "ext" in candidate.lower():

{ciphergate_cli-0.8.0 → ciphergate_cli-0.10.0}/src/ciphergate/detectors/registry.py

@@ -9,11 +9,14 @@ from ciphergate.detectors.custom_regex import CustomRegexDetector
 from ciphergate.detectors.dates import DateDetector
 from ciphergate.detectors.email import EmailDetector, is_reserved_email_domain
 from ciphergate.detectors.field_names import FieldNameDetector, PolicyClassificationDetector
+from ciphergate.detectors.iban import IbanDetector
 from ciphergate.detectors.ip_address import IPAddressDetector
+from ciphergate.detectors.mac_address import MacAddressDetector
 from ciphergate.detectors.network import InternalHostDetector
 from ciphergate.detectors.payment_card import PaymentCardDetector
 from ciphergate.detectors.phone import PhoneDetector
 from ciphergate.detectors.secrets import PEM_BEGIN_RE, PEM_END_RE, SecretDetector
+from ciphergate.detectors.ssn import SsnDetector
 from ciphergate.models.finding import Finding, ScanItem
 from ciphergate.models.policy import DetectorSettings, PolicyDocument
 
@@ -26,10 +29,23 @@ BUILTIN_DETECTOR_IDS: tuple[str, ...] = (
     "email_regex",
     "internal_host",
     "phone_pattern",
+    "mac_address",
+    "iban",
+    "ssn_pattern",
 )
 
 # Phone matching is the least specific pattern; drop phone spans that overlap these detectors.
-_PHONE_SUPPRESSORS = frozenset({"luhn_card", "ip_address", "date_pattern", "secret_pattern"})
+_PHONE_SUPPRESSORS = frozenset(
+    {
+        "luhn_card",
+        "ip_address",
+        "date_pattern",
+        "secret_pattern",
+        "mac_address",
+        "iban",
+        "ssn_pattern",
+    }
+)
 
 
 FIELD_CONTEXT_DETECTORS = frozenset({"field_name", "policy_classification"})
@@ -59,6 +75,9 @@ class DetectorSet:
             "email_regex": EmailDetector(),
             "internal_host": InternalHostDetector(settings.internal_host_suffixes),
             "phone_pattern": PhoneDetector(),
+            "mac_address": MacAddressDetector(),
+            "iban": IbanDetector(),
+            "ssn_pattern": SsnDetector(),
         }
         for detector_id in BUILTIN_DETECTOR_IDS:
             if detector_id not in disabled:
@@ -173,6 +192,12 @@ def unsafe_replacement_reason(value: str) -> str | None:
         return f"looks like a credential ({secrets[0].evidence[-1]})"
     if PaymentCardDetector().detect(probe):
         return "looks like a payment card number"
+    if IbanDetector().detect(probe):
+        return "looks like a real IBAN"
+    if MacAddressDetector().detect(probe):
+        return "looks like a real MAC address"
+    if SsnDetector().detect(probe):
+        return "looks like a real government identifier"
     for f in EmailDetector().detect(probe):
         if "reserved_domain" not in f.evidence:
             return "email replacements must use a reserved domain such as example.invalid"

{ciphergate_cli-0.8.0 → ciphergate_cli-0.10.0}/src/ciphergate/detectors/secrets.py

@@ -65,20 +65,72 @@ _PATTERNS: tuple[_Pattern, ...] = (
         "assignment",
         re.compile(
             r"(?i)(?<![a-z0-9])"
-            r"(?:password|passwd|passphrase|secret|api[_-]?key|apikey|access[_-]?token|"
+            r"(?:(?:pg|db|mysql|admin|root|user)?password|mysql_pwd|passwd|passphrase|secret|"
+            r"api[_-]?key|apikey|access[_-]?token|"
             r"refresh[_-]?token|auth[_-]?token|client[_-]?secret|secret[_-]?key|private[_-]?key|"
             r"access[_-]?key|token|credentials?)"
+            r"(?:\\?[\"'])?\s{0,3}[:=]\s{0,3}"
+            # Spring Actuator and similar dumps nest the value in an object.
+            r"(?:\{\s{0,3}\\?[\"']value\\?[\"']\s{0,3}:\s{0,3})?(?:\\?[\"'])?"
+            r"([^\s\"'\\,;&}]{1,512})"
+        ),
+        1,
+        0.8,
+    ),
+    _Pattern(
+        "camel_case_password",
+        re.compile(
+            r"(?<![A-Za-z0-9])[a-z][A-Za-z0-9]{0,40}Password"
             r"(?:\\?[\"'])?\s{0,3}[:=]\s{0,3}(?:\\?[\"'])?"
             r"([^\s\"'\\,;&}]{1,512})"
         ),
         1,
         0.8,
     ),
+    _Pattern(
+        "cookie_header",
+        re.compile(
+            r"(?i)(?<![a-z0-9_-])(?:set-)?cookie(?:\\?[\"'])?\s{0,3}:\s{0,3}(?:\\?[\"'])?"
+            r"([^\s=;\"']{1,128}=[^\r\n\"']{1,2048})"
+        ),
+        1,
+        0.85,
+    ),
+    _Pattern(
+        "session_cookie",
+        re.compile(
+            r"(?i)(?<![a-z0-9_-])(?:(?:sessionid|jsessionid|phpsessid|connect\.sid|csrftoken|"
+            r"csrfmiddlewaretoken|xsrf-token)(?:\\?[\"'])?\s{0,3}[:=]\s{0,3}|session=)"
+            r"(?:\\?[\"'])?([^\s\"'\\,;&}]{8,512})"
+        ),
+        1,
+        0.8,
+    ),
 )
 
 _PLACEHOLDER_VALUES = frozenset(
-    {"", "null", "none", "nil", "undefined", "true", "false", "redacted", "***", "******", "xxx"}
+    {
+        "",
+        "null",
+        "none",
+        "nil",
+        "undefined",
+        "true",
+        "false",
+        "redacted",
+        "***",
+        "******",
+        "xxx",
+        "{",
+        "[",
+    }
 )
+# Code that reads a secret, not the secret. Only well-known names, because a dotted value can
+# itself be a password.
+_CODE_REFERENCE_RE = re.compile(
+    r"(?:settings|config|request|self|this|args|params|env|os\.environ|process\.env)[.\[]"
+)
+_PATH_VALUE_RE = re.compile(r"(?:/|~/|[A-Za-z]:\\)")
 
 
 def _is_placeholder(value: str) -> bool:
@@ -109,7 +161,7 @@ class SecretDetector:
             for m in pattern.regex.finditer(text):
                 start, end = m.start(pattern.group), m.end(pattern.group)
                 value = m.group(pattern.group)
-                if _is_placeholder(value):
+                if _is_placeholder(value) or self._is_reference(pattern.name, m, value):
                     continue
                 if any(s <= start and end <= e for s, e in covered):
                     continue
@@ -117,6 +169,15 @@ class SecretDetector:
                 out.append(self._finding(item, (start, end), pattern.name, pattern.confidence))
         return out
 
+    @staticmethod
+    def _is_reference(family: str, match: re.Match[str], value: str) -> bool:
+        if family not in ("assignment", "connection_string_password"):
+            return False
+        if _CODE_REFERENCE_RE.match(value):
+            return True
+        # PWD is also the shell's working directory.
+        return match.group(0)[:3].lower() == "pwd" and bool(_PATH_VALUE_RE.match(value))
+
     def _finding(
         self, item: ScanItem, span: tuple[int, int], family: str, confidence: float
     ) -> Finding:

ciphergate_cli-0.10.0/src/ciphergate/detectors/ssn.py

@@ -0,0 +1,45 @@
+"""US Social Security number detector: dashed form, with the never-issued ranges excluded.
+
+Bare nine-digit runs are not matched because they are indistinguishable from other identifiers.
+Other countries' identifiers are not covered.
+"""
+
+from __future__ import annotations
+
+import re
+
+from ciphergate.detectors.base import make_finding, preceding_context
+from ciphergate.models.finding import Finding, ScanItem
+
+_SSN_RE = re.compile(r"(?<![\d-])(\d{3})-(\d{2})-(\d{4})(?![\d-])")
+_KEYWORDS = ("ssn", "social", "tax", "tin")
+
+
+class SsnDetector:
+    id = "ssn_pattern"
+
+    def detect(self, item: ScanItem) -> list[Finding]:
+        text = item.text
+        if not text or len(text) < 11:
+            return []
+        out: list[Finding] = []
+        for m in _SSN_RE.finditer(text):
+            area, group, serial = m.groups()
+            if area in ("000", "666") or area[0] == "9" or group == "00" or serial == "0000":
+                continue
+            evidence = ["pattern_match"]
+            confidence = 0.8
+            if preceding_context(text, m.start(), _KEYWORDS):
+                confidence = 0.95
+                evidence.append("context_keyword")
+            out.append(
+                make_finding(
+                    item,
+                    category="pii.government_id",
+                    detector=self.id,
+                    confidence=confidence,
+                    evidence=tuple(evidence),
+                    span=(m.start(), m.end()),
+                )
+            )
+        return out

ciphergate_cli-0.10.0/tests/tooling/test_benchmark.py

@@ -0,0 +1,76 @@
+"""The detection benchmark: repeatable corpus, valid labels, results kept out of the repository."""
+
+from __future__ import annotations
+
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+from ciphergate import load_policy
+from ciphergate.detectors.iban import iban_valid
+from ciphergate.detectors.payment_card import luhn_valid
+from tests.conftest import ROOT
+
+sys.path.insert(0, str(ROOT / "scripts" / "benchmark"))
+
+import compare_tools
+
+
+def test_corpus_is_repeatable_for_a_seed() -> None:
+    assert compare_tools.Corpus(1).build(3) == compare_tools.Corpus(1).build(3)
+    assert compare_tools.Corpus(1).build(3) != compare_tools.Corpus(2).build(3)
+
+
+def test_every_class_has_templates_and_labels_are_present_in_the_text() -> None:
+    corpus = compare_tools.Corpus(1)
+    assert set(corpus.generators) == set(compare_tools.TEMPLATES)
+    for case in corpus.build(4):
+        assert case["value"] is None or case["value"] in case["text"]
+
+
+def test_negatives_carry_no_label() -> None:
+    negatives = [c for c in compare_tools.Corpus(1).build(2) if c["cls"] == "NEGATIVE"]
+    assert len(negatives) == 2 * compare_tools.NEGATIVE_FACTOR
+    assert all(c["value"] is None for c in negatives)
+
+
+def test_generated_values_pass_their_own_checks() -> None:
+    corpus = compare_tools.Corpus(1)
+    for _ in range(50):
+        assert iban_valid(corpus.iban())
+        assert luhn_valid(corpus.luhn("4"))
+
+
+def test_benchmark_policy_is_valid() -> None:
+    assert load_policy(compare_tools.POLICY).name == "benchmark-redact-everything"
+
+
+def test_ciphergate_scoring_on_a_small_corpus(tmp_path: Path) -> None:
+    cases = compare_tools.Corpus(1).build(1)
+    hits, _ = compare_tools.run_ciphergate(cases, tmp_path)
+    scored = {c["cls"]: hit for c, hit in zip(cases, hits, strict=True) if c["value"]}
+    assert scored["email"] and scored["iban"] and scored["private_key_pem"]
+    assert not scored["person_name"]  # a known gap without NER; the score must be able to say so
+
+
+@pytest.mark.parametrize("inside", [ROOT, ROOT / "build" / "bench"])
+def test_refuses_to_write_inside_the_repository(inside: Path) -> None:
+    assert compare_tools.is_inside_repo(inside)
+    assert compare_tools.main(["--out", str(inside), "--per-class", "1"]) == 2
+
+
+def test_output_outside_the_repository_is_allowed(tmp_path: Path) -> None:
+    assert not compare_tools.is_inside_repo(tmp_path)
+
+
+def test_results_are_not_tracked() -> None:
+    tracked = subprocess.run(
+        ["git", "ls-files", "scripts/benchmark"],
+        cwd=ROOT,
+        capture_output=True,
+        text=True,
+        check=True,
+    ).stdout.split()
+    assert not [f for f in tracked if f.endswith((".jsonl", ".json"))]