datafog-core 0.3.1__tar.gz → 0.4.1__tar.gz

{datafog_core-0.3.1 → datafog_core-0.4.1}/Cargo.lock

@@ -80,7 +80,7 @@ checksum = "914a755b7c2d4af2bdcff7ce1739e2db9a1b81a9b07123d8015786ae03c0980d"
 
 [[package]]
 name = "datafog-core"
-version = "0.3.1"
+version = "0.4.1"
 dependencies = [
  "base64",
  "futures",
@@ -94,7 +94,7 @@ dependencies = [
 
 [[package]]
 name = "datafog-core-python"
-version = "0.3.1"
+version = "0.4.1"
 dependencies = [
  "datafog-core",
  "pyo3",
@@ -104,7 +104,7 @@ dependencies = [
 
 [[package]]
 name = "datafog-node"
-version = "0.3.1"
+version = "0.4.1"
 dependencies = [
  "datafog-core",
  "napi",
@@ -115,7 +115,7 @@ dependencies = [
 
 [[package]]
 name = "datafog-wasm"
-version = "0.3.1"
+version = "0.4.1"
 dependencies = [
  "datafog-core",
  "serde",

{datafog_core-0.3.1 → datafog_core-0.4.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: datafog-core
-Version: 0.3.1
+Version: 0.4.1
 Classifier: Development Status :: 3 - Alpha
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Programming Language :: Python :: 3
@@ -25,7 +25,7 @@ Project-URL: Repository, https://github.com/DataFog/datafog-core
 
 Fast structured PII detection, implemented in Rust and exposed for Rust, Python, Node.js, and browsers.
 
-It detects `EMAIL`, `PHONE`, `SSN`, `CREDIT_CARD`, `IP_ADDRESS`, `DATE`, and `ZIP_CODE`. Every binding returns the same finding information:
+It detects `EMAIL`, `PHONE`, `SSN`, `CREDIT_CARD`, `IP_ADDRESS`, `DATE`, and `ZIP_CODE`. Version 0.4.0 also detects `JWT` tokens by default; see the [JWT reference](docs/reference/jwt.mdx). With an explicit German locale, it also detects `DE_IBAN`, `DE_VAT_ID`, `DE_TAX_ID`, `DE_SOCIAL_SECURITY_NUMBER`, `DE_POSTAL_CODE`, `DE_PASSPORT_NUMBER`, and `DE_RESIDENCE_PERMIT_NUMBER`. Complete PEM private-key blocks are also detected as `PRIVATE_KEY` by default; see the [private-key reference](docs/reference/private-keys.mdx). Context-labeled `US_ROUTING_NUMBER` detection is also available; see [the detector rules](docs/reference/us-routing-number.mdx). Context-labeled `NPI` detection is also available; see [the detector rules](docs/reference/npi.mdx). Every binding returns the same finding information:
 
 ```text
 entity type, matched text, byte range, code-point range,
@@ -79,7 +79,7 @@ full-match regex values:
 }
 ```
 
-`scan_and_transform` uses `{ scan?: { locale?: string }, transform: ... }` so
+`scan_and_transform` uses `{ scan?: { locale?: string, detect_uuid?: boolean }, transform: ... }` so
 detection settings remain separate from transformation policy.
 
 ## Packages
@@ -297,3 +297,42 @@ fixtures/           Shared conformance fixtures
 
 [MIT](LICENSE)
 
+## German structured identifiers
+
+Pass `{"locale":"de"}` to text or structured scans. Trimmed, ASCII
+case-insensitive `de`, `de-DE`, and `de_DE` activate all seven German detectors;
+omitted locale and recognized `en-US`/`fr` aliases keep base detection only.
+Version 0.4.0 rejects unsupported explicit locales.
+
+```python
+from datafog_core import scan_and_transform
+
+result = scan_and_transform("IBAN DE44 5001 0517 5407 3249 31", {
+    "scan": {"locale": "de"},
+    "transform": {"default": {"strategy": "redact"}, "entities": ["DE_IBAN"]},
+})
+assert result.text == "IBAN [DE_IBAN]"
+```
+
+These are format/context detectors, not official identifier validators. IBAN
+checksums and account existence are not checked. Digits are ASCII; permitted
+internal separators are space, tab, NBSP and narrow NBSP at specified group
+boundaries, never newlines. Returned text and offsets preserve the source.
+Passport and residence-permit patterns are legacy heuristics with limited
+coverage. See the [German entity reference](docs/reference/german-entities.mdx)
+and [migration differences](docs/guides/migrating-from-datafog-python.mdx).
+Core 0.4.0 is published. The higher-level Python adapter and its dependency
+update remain separate and unreleased in [draft PR #179](https://github.com/DataFog/datafog-python/pull/179).
+
+## UUID identifiers
+
+Canonical UUID detection is opt-in: pass `{"detect_uuid":true}` to text or
+structured scans, independently of locale. It emits `UUID` findings for versions
+1–8 with the IETF variant and original casing/ranges. UUID syntax does not imply
+sensitivity. See the [UUID reference](docs/reference/uuid.mdx) for boundaries,
+excluded sentinel forms and transformation examples.
+
+**0.4.0 is published** for Rust, Python, Node.js, and WASM. The higher-level Python adapter remains a separate unreleased integration in [draft PR #179](https://github.com/DataFog/datafog-python/pull/179). See the [release notes](docs/releases/0-4-0.mdx), [runtime capabilities](docs/reference/capabilities.mdx), and [0.4.x compatibility policy](docs/reference/compatibility.mdx).
+
+The unreleased **0.4.1 candidate** adds three default detectors: provider-prefixed [`API_KEY`](docs/reference/api-keys.mdx), context-required [`BEARER_TOKEN`](docs/reference/bearer-token.mdx), and whole PostgreSQL [`CREDENTIAL_URI`](docs/reference/credential-uri.mdx). The registry reports 23 supported entities and 14 defaults with capability contract 1. See the [candidate release notes](docs/releases/0-4-1.mdx) for exact scopes, preserved legacy Python overlap behavior, and the required exact-wheel Python 4.9 integration gate. Python's default backend is unchanged.
+

{datafog_core-0.3.1/crates/core → datafog_core-0.4.1}/README.md

@@ -2,7 +2,7 @@
 
 Fast structured PII detection, implemented in Rust and exposed for Rust, Python, Node.js, and browsers.
 
-It detects `EMAIL`, `PHONE`, `SSN`, `CREDIT_CARD`, `IP_ADDRESS`, `DATE`, and `ZIP_CODE`. Every binding returns the same finding information:
+It detects `EMAIL`, `PHONE`, `SSN`, `CREDIT_CARD`, `IP_ADDRESS`, `DATE`, and `ZIP_CODE`. Version 0.4.0 also detects `JWT` tokens by default; see the [JWT reference](docs/reference/jwt.mdx). With an explicit German locale, it also detects `DE_IBAN`, `DE_VAT_ID`, `DE_TAX_ID`, `DE_SOCIAL_SECURITY_NUMBER`, `DE_POSTAL_CODE`, `DE_PASSPORT_NUMBER`, and `DE_RESIDENCE_PERMIT_NUMBER`. Complete PEM private-key blocks are also detected as `PRIVATE_KEY` by default; see the [private-key reference](docs/reference/private-keys.mdx). Context-labeled `US_ROUTING_NUMBER` detection is also available; see [the detector rules](docs/reference/us-routing-number.mdx). Context-labeled `NPI` detection is also available; see [the detector rules](docs/reference/npi.mdx). Every binding returns the same finding information:
 
 ```text
 entity type, matched text, byte range, code-point range,
@@ -56,7 +56,7 @@ full-match regex values:
 }
 ```
 
-`scan_and_transform` uses `{ scan?: { locale?: string }, transform: ... }` so
+`scan_and_transform` uses `{ scan?: { locale?: string, detect_uuid?: boolean }, transform: ... }` so
 detection settings remain separate from transformation policy.
 
 ## Packages
@@ -273,3 +273,42 @@ fixtures/           Shared conformance fixtures
 ## License
 
 [MIT](LICENSE)
+
+## German structured identifiers
+
+Pass `{"locale":"de"}` to text or structured scans. Trimmed, ASCII
+case-insensitive `de`, `de-DE`, and `de_DE` activate all seven German detectors;
+omitted locale and recognized `en-US`/`fr` aliases keep base detection only.
+Version 0.4.0 rejects unsupported explicit locales.
+
+```python
+from datafog_core import scan_and_transform
+
+result = scan_and_transform("IBAN DE44 5001 0517 5407 3249 31", {
+    "scan": {"locale": "de"},
+    "transform": {"default": {"strategy": "redact"}, "entities": ["DE_IBAN"]},
+})
+assert result.text == "IBAN [DE_IBAN]"
+```
+
+These are format/context detectors, not official identifier validators. IBAN
+checksums and account existence are not checked. Digits are ASCII; permitted
+internal separators are space, tab, NBSP and narrow NBSP at specified group
+boundaries, never newlines. Returned text and offsets preserve the source.
+Passport and residence-permit patterns are legacy heuristics with limited
+coverage. See the [German entity reference](docs/reference/german-entities.mdx)
+and [migration differences](docs/guides/migrating-from-datafog-python.mdx).
+Core 0.4.0 is published. The higher-level Python adapter and its dependency
+update remain separate and unreleased in [draft PR #179](https://github.com/DataFog/datafog-python/pull/179).
+
+## UUID identifiers
+
+Canonical UUID detection is opt-in: pass `{"detect_uuid":true}` to text or
+structured scans, independently of locale. It emits `UUID` findings for versions
+1–8 with the IETF variant and original casing/ranges. UUID syntax does not imply
+sensitivity. See the [UUID reference](docs/reference/uuid.mdx) for boundaries,
+excluded sentinel forms and transformation examples.
+
+**0.4.0 is published** for Rust, Python, Node.js, and WASM. The higher-level Python adapter remains a separate unreleased integration in [draft PR #179](https://github.com/DataFog/datafog-python/pull/179). See the [release notes](docs/releases/0-4-0.mdx), [runtime capabilities](docs/reference/capabilities.mdx), and [0.4.x compatibility policy](docs/reference/compatibility.mdx).
+
+The unreleased **0.4.1 candidate** adds three default detectors: provider-prefixed [`API_KEY`](docs/reference/api-keys.mdx), context-required [`BEARER_TOKEN`](docs/reference/bearer-token.mdx), and whole PostgreSQL [`CREDENTIAL_URI`](docs/reference/credential-uri.mdx). The registry reports 23 supported entities and 14 defaults with capability contract 1. See the [candidate release notes](docs/releases/0-4-1.mdx) for exact scopes, preserved legacy Python overlap behavior, and the required exact-wheel Python 4.9 integration gate. Python's default backend is unchanged.

{datafog_core-0.3.1 → datafog_core-0.4.1}/bindings/python/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "datafog-core-python"
-version = "0.3.1"
+version = "0.4.1"
 edition = "2024"
 rust-version = "1.88"
 description = "Python bindings for datafog-core"

{datafog_core-0.3.1 → datafog_core-0.4.1}/bindings/python/src/lib.rs

@@ -956,6 +956,14 @@ fn privacy_error(py: Python<'_>, error: core::PrivacyError) -> PyErr {
     exception
 }
 
+/// Return an independent, JSON-compatible snapshot of the Core capability contract.
+#[pyfunction]
+fn capabilities(py: Python<'_>) -> PyResult<Py<PyAny>> {
+    let json = serde_json::to_string(&core::capabilities())
+        .map_err(|_| internal_error(py, "failed to serialize Core capabilities"))?;
+    Ok(py.import("json")?.call_method1("loads", (json,))?.unbind())
+}
+
 /// Scan text for supported PII findings.
 #[pyfunction]
 #[pyo3(signature = (text, config=None))]
@@ -1300,6 +1308,7 @@ fn datafog_core(module: &Bound<'_, PyModule>) -> PyResult<()> {
     module.add_class::<Restoration>()?;
     module.add_class::<RestoreResult>()?;
     module.add_class::<PrivacyManager>()?;
+    module.add_function(wrap_pyfunction!(capabilities, module)?)?;
     module.add_function(wrap_pyfunction!(scan, module)?)?;
     module.add_function(wrap_pyfunction!(transform, module)?)?;
     module.add_function(wrap_pyfunction!(scan_and_transform, module)?)?;

datafog_core-0.4.1/bindings/python/tests/api_key_conformance.py

@@ -0,0 +1,122 @@
+"""Shared API-key fixtures against the installed wheel, including provider round trips."""
+
+import json
+from pathlib import Path
+import datafog_core as api
+from synthetic_fixtures import expand_fixture
+
+RECORDS = [
+    expand_fixture(json.loads(line))
+    for line in (Path(__file__).resolve().parents[3] / "fixtures/api-key.jsonl")
+    .read_text()
+    .splitlines()
+]
+
+
+def project(text, findings):
+    result = []
+    for f in findings:
+        if f.entity_type != "API_KEY":
+            continue
+        assert (
+            text.encode()[f.byte_range.start : f.byte_range.end].decode()
+            == f.matched_text
+        )
+        assert text[f.codepoint_range.start : f.codepoint_range.end] == f.matched_text
+        assert f.confidence is None
+        assert f.detector_name == "datafog-core/" + f.entity_type.lower().replace(
+            "_", "-"
+        )
+        assert f.detector_version
+        result.append(
+            dict(
+                label=f.entity_type,
+                text=f.matched_text,
+                start=f.codepoint_range.start,
+                end=f.codepoint_range.end,
+            )
+        )
+    return result
+
+
+def verify():
+    for row in RECORDS:
+        findings = api.scan(row["text"], row["config"])
+        assert project(row["text"], findings) == row["entities"], row["id"]
+        data = {"a/b": row["text"], "array": [row["text"]], "z": row["text"]}
+        located = api.scan_structured(data, row["config"]).findings
+        for path in ["/a~1b", "/array/0", "/z"]:
+            assert (
+                project(row["text"], [f.finding for f in located if f.path == path])
+                == row["entities"]
+            ), row["id"]
+        for case in row.get("transforms", []):
+            result = api.scan_and_transform(
+                row["text"], {"scan": row["config"], "transform": case["config"]}
+            )
+            assert result.text == case["text"], row["id"]
+            assert api.transform(row["text"], findings, case["config"]) == result
+            for t in result.transformations:
+                assert (
+                    result.text.encode()[
+                        t.output_byte_range.start : t.output_byte_range.end
+                    ].decode()
+                    == t.replacement
+                )
+                assert (
+                    result.text[
+                        t.output_codepoint_range.start : t.output_codepoint_range.end
+                    ]
+                    == t.replacement
+                )
+                assert any(
+                    f.byte_range == t.source_byte_range
+                    and f.entity_type == t.entity_type
+                    for f in findings
+                )
+                assert not hasattr(t, "matched_text") and not hasattr(t, "finding")
+            structured = api.scan_and_transform_structured(
+                data, {"scan": row["config"], "transform": case["config"]}
+            )
+            assert structured.data == {
+                "a/b": case["text"],
+                "array": [case["text"]],
+                "z": case["text"],
+            }
+            explicit = api.transform_structured(data, located, case["config"])
+            assert explicit.data == structured.data
+            assert [(t.path, t.transformation) for t in explicit.transformations] == [
+                (t.path, t.transformation) for t in structured.transformations
+            ]
+async def verify_providers(manager, token_manager, context):
+    for row in RECORDS:
+        if not row.get("sample"):
+            continue
+        entities = list(dict.fromkeys(e["label"] for e in row["entities"]))
+        pseudonyms = await manager.scan_and_transform(
+            row["text"],
+            {
+                "scan": row["config"],
+                "transform": {
+                    "default": {"strategy": "pseudonymize", "key_ref": "german"},
+                    "entities": entities,
+                },
+            },
+        )
+        assert (
+            len(pseudonyms.transformations) == len(row["entities"])
+            and pseudonyms.text != row["text"]
+        )
+        tokens = await token_manager.scan_and_transform(
+            row["text"],
+            {
+                "scan": row["config"],
+                "transform": {
+                    "default": {"strategy": "tokenize", "token_ref": "german"},
+                    "entities": entities,
+                },
+            },
+            context,
+        )
+        assert len(tokens.transformations) == len(row["entities"])
+        assert (await token_manager.restore(tokens.text, context)).text == row["text"]

datafog_core-0.4.1/bindings/python/tests/bearer_token_conformance.py

@@ -0,0 +1,134 @@
+"""Shared BEARER_TOKEN fixtures against the installed wheel, including provider round trips."""
+
+import json
+from pathlib import Path
+import datafog_core as api
+
+RECORDS = [
+    json.loads(line)
+    for line in (Path(__file__).resolve().parents[3] / "fixtures/bearer-token.jsonl")
+    .read_text()
+    .splitlines()
+]
+
+
+def project(text, findings):
+    result = []
+    for f in findings:
+        if not f.entity_type == "BEARER_TOKEN":
+            continue
+        assert (
+            text.encode()[f.byte_range.start : f.byte_range.end].decode()
+            == f.matched_text
+        )
+        assert text[f.codepoint_range.start : f.codepoint_range.end] == f.matched_text
+        assert f.confidence is None
+        assert f.detector_name == "datafog-core/" + f.entity_type.lower().replace(
+            "_", "-"
+        )
+        assert f.detector_version
+        result.append(
+            dict(
+                label=f.entity_type,
+                text=f.matched_text,
+                start=f.codepoint_range.start,
+                end=f.codepoint_range.end,
+            )
+        )
+    return result
+
+
+def verify():
+    for row in RECORDS:
+        findings = api.scan(row["text"], row["config"])
+        if "overlap" in row:
+            assert any(f.entity_type == row["overlap"] for f in findings)
+        assert project(row["text"], findings) == row["entities"], row["id"]
+        data = {"a/b": row["text"], "array": [row["text"]], "z": row["text"]}
+        located = api.scan_structured(data, row["config"]).findings
+        for path in ["/a~1b", "/array/0", "/z"]:
+            assert (
+                project(row["text"], [f.finding for f in located if f.path == path])
+                == row["entities"]
+            ), row["id"]
+        for case in row.get("transforms", []):
+            result = api.scan_and_transform(
+                row["text"], {"scan": row["config"], "transform": case["config"]}
+            )
+            assert result.text == case["text"], row["id"]
+            assert api.transform(row["text"], findings, case["config"]) == result
+            for t in result.transformations:
+                assert (
+                    result.text.encode()[
+                        t.output_byte_range.start : t.output_byte_range.end
+                    ].decode()
+                    == t.replacement
+                )
+                assert (
+                    result.text[
+                        t.output_codepoint_range.start : t.output_codepoint_range.end
+                    ]
+                    == t.replacement
+                )
+                assert any(
+                    f.byte_range == t.source_byte_range
+                    and f.entity_type == t.entity_type
+                    for f in findings
+                )
+                assert not hasattr(t, "matched_text") and not hasattr(t, "finding")
+            structured = api.scan_and_transform_structured(
+                data, {"scan": row["config"], "transform": case["config"]}
+            )
+            assert structured.data == {
+                "a/b": case["text"],
+                "array": [case["text"]],
+                "z": case["text"],
+            }
+            explicit = api.transform_structured(data, located, case["config"])
+            assert explicit.data == structured.data
+            assert [(t.path, t.transformation) for t in explicit.transformations] == [
+                (t.path, t.transformation) for t in structured.transformations
+            ]
+
+    data = {"👋/~":{"aUtHoRiZaTiOn":"  bEaReR abc._~+/==\t"}, "Authorization":["Bearer hidden"], "X-Authorization":"Bearer hidden", "a":"Authorization:", "b":"Bearer hidden", "invalid":{"Authorization":"Bearer bad!value"}}
+    found = [f for f in api.scan_structured(data).findings if f.finding.entity_type == "BEARER_TOKEN"]
+    assert len(found) == 1
+    assert found[0].path == "/👋~1~0/aUtHoRiZaTiOn"
+    assert project(data["👋/~"]["aUtHoRiZaTiOn"], [found[0].finding]) == [dict(label="BEARER_TOKEN",text="abc._~+/==",start=9,end=19)]
+    expected = {**data,"👋/~":{"aUtHoRiZaTiOn":"  bEaReR [BEARER_TOKEN]\t"}}
+    assert api.scan_and_transform_structured(data,{"transform":{"default":{"strategy":"redact"},"entities":["BEARER_TOKEN"]}}).data == expected
+    assert not any(f.entity_type == "BEARER_TOKEN" for f in api.scan("Bearer abc123"))
+
+
+async def verify_providers(manager, token_manager, context):
+    for row in RECORDS:
+        if not row.get("sample"):
+            continue
+        entities = list(dict.fromkeys(e["label"] for e in row["entities"]))
+        pseudonyms = await manager.scan_and_transform(
+            row["text"],
+            {
+                "scan": row["config"],
+                "transform": {
+                    "default": {"strategy": "pseudonymize", "key_ref": "jwt"},
+                    "entities": entities,
+                },
+            },
+        )
+        assert (
+            len(pseudonyms.transformations) == len(row["entities"])
+            and pseudonyms.text != row["text"]
+        )
+        tokens = await token_manager.scan_and_transform(
+            row["text"],
+            {
+                "scan": row["config"],
+                "transform": {
+                    "default": {"strategy": "tokenize", "token_ref": "jwt"},
+                    "entities": entities,
+                },
+            },
+            context,
+        )
+        assert len(tokens.transformations) == len(row["entities"])
+        assert (await token_manager.restore(tokens.text, context)).text == row["text"]

datafog_core-0.4.1/bindings/python/tests/capabilities_conformance.py

@@ -0,0 +1,86 @@
+"""Capability discovery contract against the installed, compiled Core wheel."""
+import json
+from pathlib import Path
+
+import datafog_core as api
+
+SUPPORTED = sorted([
+    "BEARER_TOKEN", "CREDIT_CARD", "DATE", "DE_IBAN", "DE_PASSPORT_NUMBER", "DE_POSTAL_CODE", "API_KEY", "CREDENTIAL_URI",
+    "DE_RESIDENCE_PERMIT_NUMBER", "DE_SOCIAL_SECURITY_NUMBER", "DE_TAX_ID",
+    "DE_VAT_ID", "EMAIL", "IP_ADDRESS", "JWT", "NPI", "PERSON", "PHONE",
+    "PRIVATE_KEY", "SSN", "US_ROUTING_NUMBER", "UUID", "ZIP_CODE",
+])
+GERMAN = [entity for entity in SUPPORTED if entity.startswith("DE_")]
+DEFAULT = [entity for entity in SUPPORTED if entity not in GERMAN + ["PERSON", "UUID"]]
+
+
+def verify():
+    assert callable(api.capabilities)
+    try:
+        api.capabilities({})
+    except TypeError:
+        pass
+    else:
+        raise AssertionError("capabilities accepted arguments")
+    snapshot = api.capabilities()
+    assert type(snapshot) is dict
+    assert snapshot["contract_version"] == 1
+    assert json.loads(json.dumps(snapshot)) == snapshot
+    assert snapshot["supported_entities"] == SUPPORTED
+    assert snapshot["default_entities"] == DEFAULT
+    assert sorted(snapshot["entities"]) == SUPPORTED
+    for key in ("supported_entities", "default_entities"):
+        assert snapshot[key] == sorted(set(snapshot[key]))
+    assert set(snapshot["locales"]) == {"de", "de-DE", "de_DE", "en-US", "fr"}
+    for locale, metadata in snapshot["locales"].items():
+        assert metadata["enabled_entities"] == (GERMAN if locale.startswith("de") else [])
+    for label, entity in snapshot["entities"].items():
+        assert entity["scopes"] == (["structured"] if label == "PERSON" else ["structured", "text"])
+        kind = "locale" if label in GERMAN else "config" if label == "UUID" else "structured" if label == "PERSON" else "default"
+        assert entity["activation"]["kind"] == kind
+    uuid_config = snapshot["entities"]["UUID"]["activation"]["scan_config"]
+    assert uuid_config == {"detect_uuid": True}
+    text = "👋 café é 550e8400-e29b-41d4-a716-446655440000"
+    assert not any(f.entity_type == "UUID" for f in api.scan(text))
+    uuid_findings = [f for f in api.scan(text, uuid_config) if f.entity_type == "UUID"]
+    assert len(uuid_findings) == 1
+    finding = uuid_findings[0]
+    assert text[finding.codepoint_range.start:finding.codepoint_range.end] == finding.matched_text
+    assert text.encode()[finding.byte_range.start:finding.byte_range.end].decode() == finding.matched_text
+    samples = {}
+    for line in (Path(__file__).resolve().parents[3] / "fixtures/german.jsonl").read_text().splitlines():
+        row = json.loads(line)
+        for expected in row["entities"]:
+            samples.setdefault(expected["label"], row["text"])
+    for locale in ("de", "de-DE", "de_DE", "DE-de", " De "):
+        for label, text in samples.items():
+            assert any(f.entity_type == label for f in api.scan(text, {"locale": locale})), (locale, label)
+    for label, text in samples.items():
+        activation = snapshot["entities"][label]["activation"]["scan_config"]
+        assert any(f.entity_type == label for f in api.scan(text, activation))
+    for locale in ("en-US", "fr"):
+        assert api.scan("jane@example.com", {"locale": locale}) == api.scan("jane@example.com")
+    for call, path in [
+        (lambda: api.scan("", {"locale": "zz-ZZ"}), "/locale"),
+        (lambda: api.scan_structured({}, {"locale": "zz-ZZ"}), "/locale"),
+        (lambda: api.scan_and_transform("", {"scan": {"locale": "zz-ZZ"}, "transform": {"default": {"strategy": "redact"}}}), "/scan/locale"),
+        (lambda: api.scan_and_transform_structured({}, {"scan": {"locale": "zz-ZZ"}, "transform": {"default": {"strategy": "redact"}}}), "/scan/locale"),
+    ]:
+        try:
+            call()
+        except api.DataFogConfigurationError as error:
+            assert error.code == "invalid_configuration"
+            assert error.reason == "invalid_value"
+            assert error.path == path
+        else:
+            raise AssertionError("unsupported locale accepted")
+    assert not any(f.entity_type == "PERSON" for f in api.scan("Jane Doe"))
+    assert any(f.finding.entity_type == "PERSON" for f in api.scan_structured({"full_name": "Jane Doe"}).findings)
+    mutated = api.capabilities()
+    assert mutated is not snapshot
+    mutated["supported_entities"].clear()
+    mutated["default_entities"].append("FAKE")
+    mutated["locales"]["de"]["enabled_entities"].clear()
+    mutated["entities"]["EMAIL"]["scopes"].clear()
+    mutated["entities"]["UUID"]["activation"]["scan_config"]["detect_uuid"] = False
+    assert api.capabilities() == snapshot

datafog_core-0.4.1/bindings/python/tests/credential_overlap_conformance.py

@@ -0,0 +1,103 @@
+"""Cross-detector overlap selection against the installed wheel."""
+
+import json
+from pathlib import Path
+import datafog_core as api
+from synthetic_fixtures import expand_fixture
+
+RECORDS = [
+    expand_fixture(json.loads(line))
+    for line in (Path(__file__).resolve().parents[3] / "fixtures/credential-overlaps.jsonl")
+    .read_text()
+    .splitlines()
+]
+
+
+def project(text, findings):
+    result = []
+    for f in findings:
+        if f.entity_type not in {"API_KEY", "BEARER_TOKEN", "CREDENTIAL_URI", "JWT"}:
+            continue
+        assert (
+            text.encode()[f.byte_range.start : f.byte_range.end].decode()
+            == f.matched_text
+        )
+        assert text[f.codepoint_range.start : f.codepoint_range.end] == f.matched_text
+        assert f.confidence is None
+        assert f.detector_name == "datafog-core/" + f.entity_type.lower().replace(
+            "_", "-"
+        )
+        assert f.detector_version
+        result.append(
+            dict(
+                label=f.entity_type,
+                text=f.matched_text,
+                start=f.codepoint_range.start,
+                end=f.codepoint_range.end,
+            )
+        )
+    return result
+
+
+def verify():
+    for row in RECORDS:
+        findings = api.scan(row["text"], row["config"])
+        if "authorization_value" in row:
+            value = row["authorization_value"]
+            data = {"👋/headers": {"aUtHoRiZaTiOn": value}}
+            located = api.scan_structured(data).findings
+            assert all(item.path == "/👋~1headers/aUtHoRiZaTiOn" for item in located)
+            expected = [dict(entity, start=7, end=len(value)) for entity in row["entities"]]
+            assert project(value, [item.finding for item in located]) == expected
+            for entity in row["entities"]:
+                label = entity["label"]
+                result = api.scan_and_transform_structured(data, {"transform": {"default": {"strategy": "redact"}, "entities": [label]}})
+                assert result.data == {"👋/headers": {"aUtHoRiZaTiOn": f"Bearer [{label}]"}}
+        if "overlap" in row:
+            assert any(f.entity_type == row["overlap"] for f in findings)
+        assert project(row["text"], findings) == row["entities"], row["id"]
+        data = {"a/b": row["text"], "array": [row["text"]], "z": row["text"]}
+        located = api.scan_structured(data, row["config"]).findings
+        for path in ["/a~1b", "/array/0", "/z"]:
+            assert (
+                project(row["text"], [f.finding for f in located if f.path == path])
+                == row["entities"]
+            ), row["id"]
+        for case in row.get("transforms", []):
+            result = api.scan_and_transform(
+                row["text"], {"scan": row["config"], "transform": case["config"]}
+            )
+            assert result.text == case["text"], row["id"]
+            assert api.transform(row["text"], findings, case["config"]) == result
+            for t in result.transformations:
+                assert (
+                    result.text.encode()[
+                        t.output_byte_range.start : t.output_byte_range.end
+                    ].decode()
+                    == t.replacement
+                )
+                assert (
+                    result.text[
+                        t.output_codepoint_range.start : t.output_codepoint_range.end
+                    ]
+                    == t.replacement
+                )
+                assert any(
+                    f.byte_range == t.source_byte_range
+                    and f.entity_type == t.entity_type
+                    for f in findings
+                )
+                assert not hasattr(t, "matched_text") and not hasattr(t, "finding")
+            structured = api.scan_and_transform_structured(
+                data, {"scan": row["config"], "transform": case["config"]}
+            )
+            assert structured.data == {
+                "a/b": case["text"],
+                "array": [case["text"]],
+                "z": case["text"],
+            }
+            explicit = api.transform_structured(data, located, case["config"])
+            assert explicit.data == structured.data
+            assert [(t.path, t.transformation) for t in explicit.transformations] == [
+                (t.path, t.transformation) for t in structured.transformations
+            ]