datafog-core 0.4.0__tar.gz → 0.4.1__tar.gz

{datafog_core-0.4.0 → datafog_core-0.4.1}/Cargo.lock

@@ -80,7 +80,7 @@ checksum = "914a755b7c2d4af2bdcff7ce1739e2db9a1b81a9b07123d8015786ae03c0980d"
 
 [[package]]
 name = "datafog-core"
-version = "0.4.0"
+version = "0.4.1"
 dependencies = [
  "base64",
  "futures",
@@ -94,7 +94,7 @@ dependencies = [
 
 [[package]]
 name = "datafog-core-python"
-version = "0.4.0"
+version = "0.4.1"
 dependencies = [
  "datafog-core",
  "pyo3",
@@ -104,7 +104,7 @@ dependencies = [
 
 [[package]]
 name = "datafog-node"
-version = "0.4.0"
+version = "0.4.1"
 dependencies = [
  "datafog-core",
  "napi",
@@ -115,7 +115,7 @@ dependencies = [
 
 [[package]]
 name = "datafog-wasm"
-version = "0.4.0"
+version = "0.4.1"
 dependencies = [
  "datafog-core",
  "serde",

{datafog_core-0.4.0 → datafog_core-0.4.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: datafog-core
-Version: 0.4.0
+Version: 0.4.1
 Classifier: Development Status :: 3 - Alpha
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Programming Language :: Python :: 3
@@ -25,7 +25,7 @@ Project-URL: Repository, https://github.com/DataFog/datafog-core
 
 Fast structured PII detection, implemented in Rust and exposed for Rust, Python, Node.js, and browsers.
 
-It detects `EMAIL`, `PHONE`, `SSN`, `CREDIT_CARD`, `IP_ADDRESS`, `DATE`, and `ZIP_CODE`. The next release also detects `JWT` tokens by default; see the [JWT reference](docs/reference/jwt.mdx). With an explicit German locale, it also detects `DE_IBAN`, `DE_VAT_ID`, `DE_TAX_ID`, `DE_SOCIAL_SECURITY_NUMBER`, `DE_POSTAL_CODE`, `DE_PASSPORT_NUMBER`, and `DE_RESIDENCE_PERMIT_NUMBER` (unreleased). Complete PEM private-key blocks are also detected as `PRIVATE_KEY` by default (unreleased); see the [private-key reference](docs/reference/private-keys.mdx). Context-labeled `US_ROUTING_NUMBER` detection is also available (unreleased); see [the detector rules](docs/reference/us-routing-number.mdx). Context-labeled `NPI` detection is also available (unreleased); see [the detector rules](docs/reference/npi.mdx). Every binding returns the same finding information:
+It detects `EMAIL`, `PHONE`, `SSN`, `CREDIT_CARD`, `IP_ADDRESS`, `DATE`, and `ZIP_CODE`. Version 0.4.0 also detects `JWT` tokens by default; see the [JWT reference](docs/reference/jwt.mdx). With an explicit German locale, it also detects `DE_IBAN`, `DE_VAT_ID`, `DE_TAX_ID`, `DE_SOCIAL_SECURITY_NUMBER`, `DE_POSTAL_CODE`, `DE_PASSPORT_NUMBER`, and `DE_RESIDENCE_PERMIT_NUMBER`. Complete PEM private-key blocks are also detected as `PRIVATE_KEY` by default; see the [private-key reference](docs/reference/private-keys.mdx). Context-labeled `US_ROUTING_NUMBER` detection is also available; see [the detector rules](docs/reference/us-routing-number.mdx). Context-labeled `NPI` detection is also available; see [the detector rules](docs/reference/npi.mdx). Every binding returns the same finding information:
 
 ```text
 entity type, matched text, byte range, code-point range,
@@ -297,12 +297,12 @@ fixtures/           Shared conformance fixtures
 
 [MIT](LICENSE)
 
-## German structured identifiers (unreleased)
+## German structured identifiers
 
 Pass `{"locale":"de"}` to text or structured scans. Trimmed, ASCII
 case-insensitive `de`, `de-DE`, and `de_DE` activate all seven German detectors;
 omitted locale and recognized `en-US`/`fr` aliases keep base detection only.
-The 0.4.0 candidate rejects unsupported explicit locales.
+Version 0.4.0 rejects unsupported explicit locales.
 
 ```python
 from datafog_core import scan_and_transform
@@ -321,10 +321,10 @@ boundaries, never newlines. Returned text and offsets preserve the source.
 Passport and residence-permit patterns are legacy heuristics with limited
 coverage. See the [German entity reference](docs/reference/german-entities.mdx)
 and [migration differences](docs/guides/migrating-from-datafog-python.mdx).
-The Python 4.9 adapter requires a subsequently published compatible Core wheel;
-this source change does not update its extra pin or publish a release.
+Core 0.4.0 is published. The higher-level Python adapter and its dependency
+update remain separate and unreleased in [draft PR #179](https://github.com/DataFog/datafog-python/pull/179).
 
-## UUID identifiers (unreleased)
+## UUID identifiers
 
 Canonical UUID detection is opt-in: pass `{"detect_uuid":true}` to text or
 structured scans, independently of locale. It emits `UUID` findings for versions
@@ -332,5 +332,7 @@ structured scans, independently of locale. It emits `UUID` findings for versions
 sensitivity. See the [UUID reference](docs/reference/uuid.mdx) for boundaries,
 excluded sentinel forms and transformation examples.
 
-The source candidate targets **0.4.0**; publication and downstream Python integration are separate release gates. See the [candidate release checklist](docs/releases/0-4-0.mdx), [runtime capabilities](docs/reference/capabilities.mdx), and [0.4.x compatibility policy](docs/reference/compatibility.mdx).
+**0.4.0 is published** for Rust, Python, Node.js, and WASM. The higher-level Python adapter remains a separate unreleased integration in [draft PR #179](https://github.com/DataFog/datafog-python/pull/179). See the [release notes](docs/releases/0-4-0.mdx), [runtime capabilities](docs/reference/capabilities.mdx), and [0.4.x compatibility policy](docs/reference/compatibility.mdx).
+
+The unreleased **0.4.1 candidate** adds three default detectors: provider-prefixed [`API_KEY`](docs/reference/api-keys.mdx), context-required [`BEARER_TOKEN`](docs/reference/bearer-token.mdx), and whole PostgreSQL [`CREDENTIAL_URI`](docs/reference/credential-uri.mdx). The registry reports 23 supported entities and 14 defaults with capability contract 1. See the [candidate release notes](docs/releases/0-4-1.mdx) for exact scopes, preserved legacy Python overlap behavior, and the required exact-wheel Python 4.9 integration gate. Python's default backend is unchanged.
 

{datafog_core-0.4.0/crates/core → datafog_core-0.4.1}/README.md

@@ -2,7 +2,7 @@
 
 Fast structured PII detection, implemented in Rust and exposed for Rust, Python, Node.js, and browsers.
 
-It detects `EMAIL`, `PHONE`, `SSN`, `CREDIT_CARD`, `IP_ADDRESS`, `DATE`, and `ZIP_CODE`. The next release also detects `JWT` tokens by default; see the [JWT reference](docs/reference/jwt.mdx). With an explicit German locale, it also detects `DE_IBAN`, `DE_VAT_ID`, `DE_TAX_ID`, `DE_SOCIAL_SECURITY_NUMBER`, `DE_POSTAL_CODE`, `DE_PASSPORT_NUMBER`, and `DE_RESIDENCE_PERMIT_NUMBER` (unreleased). Complete PEM private-key blocks are also detected as `PRIVATE_KEY` by default (unreleased); see the [private-key reference](docs/reference/private-keys.mdx). Context-labeled `US_ROUTING_NUMBER` detection is also available (unreleased); see [the detector rules](docs/reference/us-routing-number.mdx). Context-labeled `NPI` detection is also available (unreleased); see [the detector rules](docs/reference/npi.mdx). Every binding returns the same finding information:
+It detects `EMAIL`, `PHONE`, `SSN`, `CREDIT_CARD`, `IP_ADDRESS`, `DATE`, and `ZIP_CODE`. Version 0.4.0 also detects `JWT` tokens by default; see the [JWT reference](docs/reference/jwt.mdx). With an explicit German locale, it also detects `DE_IBAN`, `DE_VAT_ID`, `DE_TAX_ID`, `DE_SOCIAL_SECURITY_NUMBER`, `DE_POSTAL_CODE`, `DE_PASSPORT_NUMBER`, and `DE_RESIDENCE_PERMIT_NUMBER`. Complete PEM private-key blocks are also detected as `PRIVATE_KEY` by default; see the [private-key reference](docs/reference/private-keys.mdx). Context-labeled `US_ROUTING_NUMBER` detection is also available; see [the detector rules](docs/reference/us-routing-number.mdx). Context-labeled `NPI` detection is also available; see [the detector rules](docs/reference/npi.mdx). Every binding returns the same finding information:
 
 ```text
 entity type, matched text, byte range, code-point range,
@@ -274,12 +274,12 @@ fixtures/           Shared conformance fixtures
 
 [MIT](LICENSE)
 
-## German structured identifiers (unreleased)
+## German structured identifiers
 
 Pass `{"locale":"de"}` to text or structured scans. Trimmed, ASCII
 case-insensitive `de`, `de-DE`, and `de_DE` activate all seven German detectors;
 omitted locale and recognized `en-US`/`fr` aliases keep base detection only.
-The 0.4.0 candidate rejects unsupported explicit locales.
+Version 0.4.0 rejects unsupported explicit locales.
 
 ```python
 from datafog_core import scan_and_transform
@@ -298,10 +298,10 @@ boundaries, never newlines. Returned text and offsets preserve the source.
 Passport and residence-permit patterns are legacy heuristics with limited
 coverage. See the [German entity reference](docs/reference/german-entities.mdx)
 and [migration differences](docs/guides/migrating-from-datafog-python.mdx).
-The Python 4.9 adapter requires a subsequently published compatible Core wheel;
-this source change does not update its extra pin or publish a release.
+Core 0.4.0 is published. The higher-level Python adapter and its dependency
+update remain separate and unreleased in [draft PR #179](https://github.com/DataFog/datafog-python/pull/179).
 
-## UUID identifiers (unreleased)
+## UUID identifiers
 
 Canonical UUID detection is opt-in: pass `{"detect_uuid":true}` to text or
 structured scans, independently of locale. It emits `UUID` findings for versions
@@ -309,4 +309,6 @@ structured scans, independently of locale. It emits `UUID` findings for versions
 sensitivity. See the [UUID reference](docs/reference/uuid.mdx) for boundaries,
 excluded sentinel forms and transformation examples.
 
-The source candidate targets **0.4.0**; publication and downstream Python integration are separate release gates. See the [candidate release checklist](docs/releases/0-4-0.mdx), [runtime capabilities](docs/reference/capabilities.mdx), and [0.4.x compatibility policy](docs/reference/compatibility.mdx).
+**0.4.0 is published** for Rust, Python, Node.js, and WASM. The higher-level Python adapter remains a separate unreleased integration in [draft PR #179](https://github.com/DataFog/datafog-python/pull/179). See the [release notes](docs/releases/0-4-0.mdx), [runtime capabilities](docs/reference/capabilities.mdx), and [0.4.x compatibility policy](docs/reference/compatibility.mdx).
+
+The unreleased **0.4.1 candidate** adds three default detectors: provider-prefixed [`API_KEY`](docs/reference/api-keys.mdx), context-required [`BEARER_TOKEN`](docs/reference/bearer-token.mdx), and whole PostgreSQL [`CREDENTIAL_URI`](docs/reference/credential-uri.mdx). The registry reports 23 supported entities and 14 defaults with capability contract 1. See the [candidate release notes](docs/releases/0-4-1.mdx) for exact scopes, preserved legacy Python overlap behavior, and the required exact-wheel Python 4.9 integration gate. Python's default backend is unchanged.

{datafog_core-0.4.0 → datafog_core-0.4.1}/bindings/python/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "datafog-core-python"
-version = "0.4.0"
+version = "0.4.1"
 edition = "2024"
 rust-version = "1.88"
 description = "Python bindings for datafog-core"

datafog_core-0.4.1/bindings/python/tests/api_key_conformance.py

@@ -0,0 +1,122 @@
+"""Shared API-key fixtures against the installed wheel, including provider round trips."""
+
+import json
+from pathlib import Path
+import datafog_core as api
+from synthetic_fixtures import expand_fixture
+
+RECORDS = [
+    expand_fixture(json.loads(line))
+    for line in (Path(__file__).resolve().parents[3] / "fixtures/api-key.jsonl")
+    .read_text()
+    .splitlines()
+]
+
+
+def project(text, findings):
+    result = []
+    for f in findings:
+        if f.entity_type != "API_KEY":
+            continue
+        assert (
+            text.encode()[f.byte_range.start : f.byte_range.end].decode()
+            == f.matched_text
+        )
+        assert text[f.codepoint_range.start : f.codepoint_range.end] == f.matched_text
+        assert f.confidence is None
+        assert f.detector_name == "datafog-core/" + f.entity_type.lower().replace(
+            "_", "-"
+        )
+        assert f.detector_version
+        result.append(
+            dict(
+                label=f.entity_type,
+                text=f.matched_text,
+                start=f.codepoint_range.start,
+                end=f.codepoint_range.end,
+            )
+        )
+    return result
+
+
+def verify():
+    for row in RECORDS:
+        findings = api.scan(row["text"], row["config"])
+        assert project(row["text"], findings) == row["entities"], row["id"]
+        data = {"a/b": row["text"], "array": [row["text"]], "z": row["text"]}
+        located = api.scan_structured(data, row["config"]).findings
+        for path in ["/a~1b", "/array/0", "/z"]:
+            assert (
+                project(row["text"], [f.finding for f in located if f.path == path])
+                == row["entities"]
+            ), row["id"]
+        for case in row.get("transforms", []):
+            result = api.scan_and_transform(
+                row["text"], {"scan": row["config"], "transform": case["config"]}
+            )
+            assert result.text == case["text"], row["id"]
+            assert api.transform(row["text"], findings, case["config"]) == result
+            for t in result.transformations:
+                assert (
+                    result.text.encode()[
+                        t.output_byte_range.start : t.output_byte_range.end
+                    ].decode()
+                    == t.replacement
+                )
+                assert (
+                    result.text[
+                        t.output_codepoint_range.start : t.output_codepoint_range.end
+                    ]
+                    == t.replacement
+                )
+                assert any(
+                    f.byte_range == t.source_byte_range
+                    and f.entity_type == t.entity_type
+                    for f in findings
+                )
+                assert not hasattr(t, "matched_text") and not hasattr(t, "finding")
+            structured = api.scan_and_transform_structured(
+                data, {"scan": row["config"], "transform": case["config"]}
+            )
+            assert structured.data == {
+                "a/b": case["text"],
+                "array": [case["text"]],
+                "z": case["text"],
+            }
+            explicit = api.transform_structured(data, located, case["config"])
+            assert explicit.data == structured.data
+            assert [(t.path, t.transformation) for t in explicit.transformations] == [
+                (t.path, t.transformation) for t in structured.transformations
+            ]
+async def verify_providers(manager, token_manager, context):
+    for row in RECORDS:
+        if not row.get("sample"):
+            continue
+        entities = list(dict.fromkeys(e["label"] for e in row["entities"]))
+        pseudonyms = await manager.scan_and_transform(
+            row["text"],
+            {
+                "scan": row["config"],
+                "transform": {
+                    "default": {"strategy": "pseudonymize", "key_ref": "german"},
+                    "entities": entities,
+                },
+            },
+        )
+        assert (
+            len(pseudonyms.transformations) == len(row["entities"])
+            and pseudonyms.text != row["text"]
+        )
+        tokens = await token_manager.scan_and_transform(
+            row["text"],
+            {
+                "scan": row["config"],
+                "transform": {
+                    "default": {"strategy": "tokenize", "token_ref": "german"},
+                    "entities": entities,
+                },
+            },
+            context,
+        )
+        assert len(tokens.transformations) == len(row["entities"])
+        assert (await token_manager.restore(tokens.text, context)).text == row["text"]

datafog_core-0.4.1/bindings/python/tests/bearer_token_conformance.py

@@ -0,0 +1,134 @@
+"""Shared BEARER_TOKEN fixtures against the installed wheel, including provider round trips."""
+
+import json
+from pathlib import Path
+import datafog_core as api
+
+RECORDS = [
+    json.loads(line)
+    for line in (Path(__file__).resolve().parents[3] / "fixtures/bearer-token.jsonl")
+    .read_text()
+    .splitlines()
+]
+
+
+def project(text, findings):
+    result = []
+    for f in findings:
+        if not f.entity_type == "BEARER_TOKEN":
+            continue
+        assert (
+            text.encode()[f.byte_range.start : f.byte_range.end].decode()
+            == f.matched_text
+        )
+        assert text[f.codepoint_range.start : f.codepoint_range.end] == f.matched_text
+        assert f.confidence is None
+        assert f.detector_name == "datafog-core/" + f.entity_type.lower().replace(
+            "_", "-"
+        )
+        assert f.detector_version
+        result.append(
+            dict(
+                label=f.entity_type,
+                text=f.matched_text,
+                start=f.codepoint_range.start,
+                end=f.codepoint_range.end,
+            )
+        )
+    return result
+
+
+def verify():
+    for row in RECORDS:
+        findings = api.scan(row["text"], row["config"])
+        if "overlap" in row:
+            assert any(f.entity_type == row["overlap"] for f in findings)
+        assert project(row["text"], findings) == row["entities"], row["id"]
+        data = {"a/b": row["text"], "array": [row["text"]], "z": row["text"]}
+        located = api.scan_structured(data, row["config"]).findings
+        for path in ["/a~1b", "/array/0", "/z"]:
+            assert (
+                project(row["text"], [f.finding for f in located if f.path == path])
+                == row["entities"]
+            ), row["id"]
+        for case in row.get("transforms", []):
+            result = api.scan_and_transform(
+                row["text"], {"scan": row["config"], "transform": case["config"]}
+            )
+            assert result.text == case["text"], row["id"]
+            assert api.transform(row["text"], findings, case["config"]) == result
+            for t in result.transformations:
+                assert (
+                    result.text.encode()[
+                        t.output_byte_range.start : t.output_byte_range.end
+                    ].decode()
+                    == t.replacement
+                )
+                assert (
+                    result.text[
+                        t.output_codepoint_range.start : t.output_codepoint_range.end
+                    ]
+                    == t.replacement
+                )
+                assert any(
+                    f.byte_range == t.source_byte_range
+                    and f.entity_type == t.entity_type
+                    for f in findings
+                )
+                assert not hasattr(t, "matched_text") and not hasattr(t, "finding")
+            structured = api.scan_and_transform_structured(
+                data, {"scan": row["config"], "transform": case["config"]}
+            )
+            assert structured.data == {
+                "a/b": case["text"],
+                "array": [case["text"]],
+                "z": case["text"],
+            }
+            explicit = api.transform_structured(data, located, case["config"])
+            assert explicit.data == structured.data
+            assert [(t.path, t.transformation) for t in explicit.transformations] == [
+                (t.path, t.transformation) for t in structured.transformations
+            ]
+
+    data = {"👋/~":{"aUtHoRiZaTiOn":"  bEaReR abc._~+/==\t"}, "Authorization":["Bearer hidden"], "X-Authorization":"Bearer hidden", "a":"Authorization:", "b":"Bearer hidden", "invalid":{"Authorization":"Bearer bad!value"}}
+    found = [f for f in api.scan_structured(data).findings if f.finding.entity_type == "BEARER_TOKEN"]
+    assert len(found) == 1
+    assert found[0].path == "/👋~1~0/aUtHoRiZaTiOn"
+    assert project(data["👋/~"]["aUtHoRiZaTiOn"], [found[0].finding]) == [dict(label="BEARER_TOKEN",text="abc._~+/==",start=9,end=19)]
+    expected = {**data,"👋/~":{"aUtHoRiZaTiOn":"  bEaReR [BEARER_TOKEN]\t"}}
+    assert api.scan_and_transform_structured(data,{"transform":{"default":{"strategy":"redact"},"entities":["BEARER_TOKEN"]}}).data == expected
+    assert not any(f.entity_type == "BEARER_TOKEN" for f in api.scan("Bearer abc123"))
+
+
+async def verify_providers(manager, token_manager, context):
+    for row in RECORDS:
+        if not row.get("sample"):
+            continue
+        entities = list(dict.fromkeys(e["label"] for e in row["entities"]))
+        pseudonyms = await manager.scan_and_transform(
+            row["text"],
+            {
+                "scan": row["config"],
+                "transform": {
+                    "default": {"strategy": "pseudonymize", "key_ref": "jwt"},
+                    "entities": entities,
+                },
+            },
+        )
+        assert (
+            len(pseudonyms.transformations) == len(row["entities"])
+            and pseudonyms.text != row["text"]
+        )
+        tokens = await token_manager.scan_and_transform(
+            row["text"],
+            {
+                "scan": row["config"],
+                "transform": {
+                    "default": {"strategy": "tokenize", "token_ref": "jwt"},
+                    "entities": entities,
+                },
+            },
+            context,
+        )
+        assert len(tokens.transformations) == len(row["entities"])
+        assert (await token_manager.restore(tokens.text, context)).text == row["text"]

{datafog_core-0.4.0 → datafog_core-0.4.1}/bindings/python/tests/capabilities_conformance.py

@@ -5,7 +5,7 @@ from pathlib import Path
 import datafog_core as api
 
 SUPPORTED = sorted([
-    "CREDIT_CARD", "DATE", "DE_IBAN", "DE_PASSPORT_NUMBER", "DE_POSTAL_CODE",
+    "BEARER_TOKEN", "CREDIT_CARD", "DATE", "DE_IBAN", "DE_PASSPORT_NUMBER", "DE_POSTAL_CODE", "API_KEY", "CREDENTIAL_URI",
     "DE_RESIDENCE_PERMIT_NUMBER", "DE_SOCIAL_SECURITY_NUMBER", "DE_TAX_ID",
     "DE_VAT_ID", "EMAIL", "IP_ADDRESS", "JWT", "NPI", "PERSON", "PHONE",
     "PRIVATE_KEY", "SSN", "US_ROUTING_NUMBER", "UUID", "ZIP_CODE",

datafog_core-0.4.1/bindings/python/tests/credential_overlap_conformance.py

@@ -0,0 +1,103 @@
+"""Cross-detector overlap selection against the installed wheel."""
+
+import json
+from pathlib import Path
+import datafog_core as api
+from synthetic_fixtures import expand_fixture
+
+RECORDS = [
+    expand_fixture(json.loads(line))
+    for line in (Path(__file__).resolve().parents[3] / "fixtures/credential-overlaps.jsonl")
+    .read_text()
+    .splitlines()
+]
+
+
+def project(text, findings):
+    result = []
+    for f in findings:
+        if f.entity_type not in {"API_KEY", "BEARER_TOKEN", "CREDENTIAL_URI", "JWT"}:
+            continue
+        assert (
+            text.encode()[f.byte_range.start : f.byte_range.end].decode()
+            == f.matched_text
+        )
+        assert text[f.codepoint_range.start : f.codepoint_range.end] == f.matched_text
+        assert f.confidence is None
+        assert f.detector_name == "datafog-core/" + f.entity_type.lower().replace(
+            "_", "-"
+        )
+        assert f.detector_version
+        result.append(
+            dict(
+                label=f.entity_type,
+                text=f.matched_text,
+                start=f.codepoint_range.start,
+                end=f.codepoint_range.end,
+            )
+        )
+    return result
+
+
+def verify():
+    for row in RECORDS:
+        findings = api.scan(row["text"], row["config"])
+        if "authorization_value" in row:
+            value = row["authorization_value"]
+            data = {"👋/headers": {"aUtHoRiZaTiOn": value}}
+            located = api.scan_structured(data).findings
+            assert all(item.path == "/👋~1headers/aUtHoRiZaTiOn" for item in located)
+            expected = [dict(entity, start=7, end=len(value)) for entity in row["entities"]]
+            assert project(value, [item.finding for item in located]) == expected
+            for entity in row["entities"]:
+                label = entity["label"]
+                result = api.scan_and_transform_structured(data, {"transform": {"default": {"strategy": "redact"}, "entities": [label]}})
+                assert result.data == {"👋/headers": {"aUtHoRiZaTiOn": f"Bearer [{label}]"}}
+        if "overlap" in row:
+            assert any(f.entity_type == row["overlap"] for f in findings)
+        assert project(row["text"], findings) == row["entities"], row["id"]
+        data = {"a/b": row["text"], "array": [row["text"]], "z": row["text"]}
+        located = api.scan_structured(data, row["config"]).findings
+        for path in ["/a~1b", "/array/0", "/z"]:
+            assert (
+                project(row["text"], [f.finding for f in located if f.path == path])
+                == row["entities"]
+            ), row["id"]
+        for case in row.get("transforms", []):
+            result = api.scan_and_transform(
+                row["text"], {"scan": row["config"], "transform": case["config"]}
+            )
+            assert result.text == case["text"], row["id"]
+            assert api.transform(row["text"], findings, case["config"]) == result
+            for t in result.transformations:
+                assert (
+                    result.text.encode()[
+                        t.output_byte_range.start : t.output_byte_range.end
+                    ].decode()
+                    == t.replacement
+                )
+                assert (
+                    result.text[
+                        t.output_codepoint_range.start : t.output_codepoint_range.end
+                    ]
+                    == t.replacement
+                )
+                assert any(
+                    f.byte_range == t.source_byte_range
+                    and f.entity_type == t.entity_type
+                    for f in findings
+                )
+                assert not hasattr(t, "matched_text") and not hasattr(t, "finding")
+            structured = api.scan_and_transform_structured(
+                data, {"scan": row["config"], "transform": case["config"]}
+            )
+            assert structured.data == {
+                "a/b": case["text"],
+                "array": [case["text"]],
+                "z": case["text"],
+            }
+            explicit = api.transform_structured(data, located, case["config"])
+            assert explicit.data == structured.data
+            assert [(t.path, t.transformation) for t in explicit.transformations] == [
+                (t.path, t.transformation) for t in structured.transformations
+            ]

datafog_core-0.4.1/bindings/python/tests/credential_uri_conformance.py

@@ -0,0 +1,127 @@
+"""Shared numeric identifier fixtures against the installed wheel, including provider round trips."""
+
+import json
+from pathlib import Path
+import datafog_core as api
+
+RECORDS = [
+    json.loads(line)
+    for line in (Path(__file__).resolve().parents[3] / "fixtures/credential-uri.jsonl")
+    .read_text()
+    .splitlines()
+]
+
+
+def project(text, findings):
+    result = []
+    for f in findings:
+        if f.entity_type != "CREDENTIAL_URI":
+            continue
+        assert (
+            text.encode()[f.byte_range.start : f.byte_range.end].decode()
+            == f.matched_text
+        )
+        assert text[f.codepoint_range.start : f.codepoint_range.end] == f.matched_text
+        assert f.confidence is None
+        assert f.detector_name == "datafog-core/" + f.entity_type.lower().replace(
+            "_", "-"
+        )
+        assert f.detector_version
+        result.append(
+            dict(
+                label=f.entity_type,
+                text=f.matched_text,
+                start=f.codepoint_range.start,
+                end=f.codepoint_range.end,
+            )
+        )
+    return result
+
+
+def verify():
+    for row in RECORDS:
+        findings = api.scan(row["text"], row["config"])
+        if "overlaps" in row:
+            assert all(any(f.entity_type == label for f in findings) for label in row["overlaps"])
+            assert api.transform(row["text"], findings, {"default": {"strategy": "redact"}}).text == row["default_text"]
+
+        assert project(row["text"], findings) == row["entities"], row["id"]
+        data = {"a/b": row["text"], "array": [row["text"]], "z": row["text"]}
+        located = api.scan_structured(data, row["config"]).findings
+        for path in ["/a~1b", "/array/0", "/z"]:
+            assert (
+                project(row["text"], [f.finding for f in located if f.path == path])
+                == row["entities"]
+            ), row["id"]
+        for case in row.get("transforms", []):
+            result = api.scan_and_transform(
+                row["text"], {"scan": row["config"], "transform": case["config"]}
+            )
+            assert result.text == case["text"], row["id"]
+            assert api.transform(row["text"], findings, case["config"]) == result
+            for t in result.transformations:
+                assert (
+                    result.text.encode()[
+                        t.output_byte_range.start : t.output_byte_range.end
+                    ].decode()
+                    == t.replacement
+                )
+                assert (
+                    result.text[
+                        t.output_codepoint_range.start : t.output_codepoint_range.end
+                    ]
+                    == t.replacement
+                )
+                assert any(
+                    f.byte_range == t.source_byte_range
+                    and f.entity_type == t.entity_type
+                    for f in findings
+                )
+                assert not hasattr(t, "matched_text") and not hasattr(t, "finding")
+            structured = api.scan_and_transform_structured(
+                data, {"scan": row["config"], "transform": case["config"]}
+            )
+            assert structured.data == {
+                "a/b": case["text"],
+                "array": [case["text"]],
+                "z": case["text"],
+            }
+            explicit = api.transform_structured(data, located, case["config"])
+            assert explicit.data == structured.data
+            assert [(t.path, t.transformation) for t in explicit.transformations] == [
+                (t.path, t.transformation) for t in structured.transformations
+            ]
+
+
+async def verify_providers(manager, token_manager, context):
+    for row in RECORDS:
+        if not row.get("sample"):
+            continue
+        entities = list(dict.fromkeys(e["label"] for e in row["entities"]))
+        pseudonyms = await manager.scan_and_transform(
+            row["text"],
+            {
+                "scan": row["config"],
+                "transform": {
+                    "default": {"strategy": "pseudonymize", "key_ref": "german"},
+                    "entities": entities,
+                },
+            },
+        )
+        assert (
+            len(pseudonyms.transformations) == len(row["entities"])
+            and pseudonyms.text != row["text"]
+        )
+        tokens = await token_manager.scan_and_transform(
+            row["text"],
+            {
+                "scan": row["config"],
+                "transform": {
+                    "default": {"strategy": "tokenize", "token_ref": "german"},
+                    "entities": entities,
+                },
+            },
+            context,
+        )
+        assert len(tokens.transformations) == len(row["entities"])
+        assert (await token_manager.restore(tokens.text, context)).text == row["text"]