datafog 4.9.0a2__tar.gz → 4.10.0__tar.gz

{datafog-4.9.0a2 → datafog-4.10.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: datafog
-Version: 4.9.0a2
+Version: 4.10.0
 Summary: Lightning-fast PII detection and anonymization library, 100x+ faster than NER-based detection (see benchmarks/)
 Author: Sid Mohan
 Author-email: sid@datafog.ai
@@ -10,7 +10,7 @@ Project-URL: Discord, https://discord.gg/bzDth394R4
 Project-URL: Twitter, https://twitter.com/datafoginc
 Project-URL: GitHub, https://github.com/datafog/datafog-python
 Keywords: pii detection anonymization privacy regex performance
-Classifier: Development Status :: 4 - Beta
+Classifier: Development Status :: 5 - Production/Stable
 Classifier: Intended Audience :: Developers
 Classifier: License :: OSI Approved :: Apache Software License
 Classifier: Programming Language :: Python :: 3
@@ -25,11 +25,12 @@ Classifier: Topic :: Security
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
+Requires-Dist: datafog-core<2,>=1.3.0
 Requires-Dist: pydantic<3.0,>=2.0
 Requires-Dist: pydantic-settings>=2.0.0
 Requires-Dist: typing-extensions>=4.0
 Provides-Extra: rust
-Requires-Dist: datafog-core<0.5,>=0.4.0; extra == "rust"
+Requires-Dist: datafog-core<2,>=1.3.0; extra == "rust"
 Provides-Extra: nlp
 Requires-Dist: click<9.0,>=8.0; extra == "nlp"
 Requires-Dist: spacy<4.0,>=3.7.0; extra == "nlp"
@@ -133,8 +134,9 @@ values never echoed into logs or transcripts:
 
 - **Claude Code hook** (`datafog-hook`): gates agent tool calls (shell
   commands, web requests, file writes, MCP tools) and warns the model when
-  prompts or tool results carry PII. ~70–90ms per invocation including
-  process startup. Easiest install is the
+  prompts or tool results carry PII. ~105–110ms per invocation including
+  process startup and the native engine load (`DATAFOG_BACKEND=python` trims
+  ~20ms at the cost of detection coverage). Easiest install is the
   [Claude Code plugin](https://github.com/DataFog/datafog-claude-plugin):
 
   ```
@@ -146,8 +148,8 @@ values never echoed into logs or transcripts:
 
 - **LiteLLM guardrail** (`DataFogGuardrail`): redacts or blocks PII in
   requests and responses at the gateway, for any LiteLLM-proxied provider.
-  In-process (~40µs per message scanned; a request clears the guardrail in
-  well under a millisecond), no sidecar service. Setup:
+  In-process (tens of microseconds per message scanned on the native engine),
+  no sidecar service. Setup:
   [examples/litellm_guardrail/](examples/litellm_guardrail/).
 
 Both default to the high-precision entity set (`EMAIL`, `PHONE`,
@@ -165,7 +167,7 @@ NER live in [benchmarks/](benchmarks/).
 ## Installation
 
 ```bash
-# Core install (regex engine)
+# Core install: native engine, no extra packages needed
 pip install datafog
 
 # Add spaCy support
@@ -184,9 +186,33 @@ pip install datafog[distributed]
 pip install datafog[all]
 ```
 
+`pip install datafog` pulls `datafog-core`, which is the native engine. Wheels
+exist for Linux (glibc and musl, x86_64/aarch64/s390x/ppc64le), macOS and
+Windows (amd64/arm64); anywhere else the source build needs a Rust toolchain.
+If Core cannot be imported, detection falls back to the pure-Python engine and
+warns once — `DATAFOG_BACKEND=python` selects that path deliberately.
+
 The `nlp-advanced` extra includes SentencePiece and protobuf for GLiNER's
 multilingual tokenizer; installing the OCR extra is not required for GLiNER.
 
+## Backends
+
+`backend` defaults to `auto`: the native engine when `datafog-core` is
+importable, the pure-Python regex engine otherwise. `DATAFOG_BACKEND` sets the
+default and `backend="python"` / `backend="rust"` override it per call.
+`auto` never selects the native engine for `engine="spacy"`, `"gliner"` or
+`"smart"`, which stay on the Python ML path.
+
+The native engine adds detectors the Python engine never had — `API_KEY`,
+`BEARER_TOKEN`, `JWT`, `PRIVATE_KEY`, `CREDENTIAL_URI`, `ACCOUNT_NUMBER`,
+`US_ROUTING_NUMBER`, `NPI`, `UUID` and a German set — and serves prose `PERSON`
+and `ADDRESS` from an embedded model, so name and address detection needs no
+spaCy and no model download. Findings keep `engine="regex"` provenance and
+`confidence=1.0` as compatibility values, not native confidence estimates.
+Reviewed differences from the Python engine are listed in
+[tests/contracts/rust-1.3.0.json](tests/contracts/rust-1.3.0.json).
+`datafog.native` exposes the native schema directly.
+
 Python 3.13 support is certified for the core SDK, CLI, `nlp`,
 `nlp-advanced`, and `ocr` install profiles. Donut OCR still requires a model
 that is available locally before runtime use. `distributed` and `all` remain
@@ -302,10 +328,12 @@ scan/redact helpers, or guardrail helpers.
     model.
   - A Java runtime is required by PySpark.
 
-DataFog 4.9.0 deprecates OCR and Spark with visible use-time
-warnings; their APIs and extras will be removed in 5.0. They remain functional
-in 4.9. Users who need these features can stay on the final 4.x release. See
-the [4.9 migration guide](docs/migration-4.9.md) for the transition plan.
+DataFog deprecates OCR and Spark with visible use-time
+warnings; their APIs and extras will be removed in 5.0. The last supported
+version is 4.10.0 — pin it with `pip install "datafog==4.10.0"` — and every
+retired surface is tracked in the [retirement schedule](docs/retirement.rst).
+See the [4.9 migration guide](docs/migration-4.9.md) for the transition plan,
+and [VERSIONING.md](VERSIONING.md) for what a version number promises.
 
 ## Backward-Compatible APIs
 
@@ -375,11 +403,9 @@ The [4.9 migration guide](docs/migration-4.9.md) explains opt-in Rust detection,
 the native `datafog.v5` preview, and the revised 5.0 retirement schedule for
 `detect`/`process`, OCR, and Spark. The Python detector remains the default.
 
-The experimental Rust adapter in 4.9.0 requires Core `>=0.4.0,<0.5` and capability
-contract 1. Upgrade with `python -m pip install --upgrade "datafog[rust]==4.9.0"`
-to evaluate it. Base-only users can install `datafog==4.9.0` without Core;
-installing the Rust extra does not change the default backend. See the
-[4.9.0 release notes](RELEASE_NOTES_4.9.0.md).
+4.10.0 requires Core `>=1.3.0,<2` and capability contract 1, and installs it by
+default. See the [4.10.0 release notes](RELEASE_NOTES_4.10.0.md) and the
+[4.10 migration guide](docs/migration-4.10.md).
 Entity labels, locales, and activation settings come from the installed
 Core, allowing compatible releases to add detectors without a Python update.
 German detection is opt-in through locale or entity selection; UUID is opt-in

{datafog-4.9.0a2 → datafog-4.10.0}/README.md

@@ -19,8 +19,9 @@ values never echoed into logs or transcripts:
 
 - **Claude Code hook** (`datafog-hook`): gates agent tool calls (shell
   commands, web requests, file writes, MCP tools) and warns the model when
-  prompts or tool results carry PII. ~70–90ms per invocation including
-  process startup. Easiest install is the
+  prompts or tool results carry PII. ~105–110ms per invocation including
+  process startup and the native engine load (`DATAFOG_BACKEND=python` trims
+  ~20ms at the cost of detection coverage). Easiest install is the
   [Claude Code plugin](https://github.com/DataFog/datafog-claude-plugin):
 
   ```
@@ -32,8 +33,8 @@ values never echoed into logs or transcripts:
 
 - **LiteLLM guardrail** (`DataFogGuardrail`): redacts or blocks PII in
   requests and responses at the gateway, for any LiteLLM-proxied provider.
-  In-process (~40µs per message scanned; a request clears the guardrail in
-  well under a millisecond), no sidecar service. Setup:
+  In-process (tens of microseconds per message scanned on the native engine),
+  no sidecar service. Setup:
   [examples/litellm_guardrail/](examples/litellm_guardrail/).
 
 Both default to the high-precision entity set (`EMAIL`, `PHONE`,
@@ -51,7 +52,7 @@ NER live in [benchmarks/](benchmarks/).
 ## Installation
 
 ```bash
-# Core install (regex engine)
+# Core install: native engine, no extra packages needed
 pip install datafog
 
 # Add spaCy support
@@ -70,9 +71,33 @@ pip install datafog[distributed]
 pip install datafog[all]
 ```
 
+`pip install datafog` pulls `datafog-core`, which is the native engine. Wheels
+exist for Linux (glibc and musl, x86_64/aarch64/s390x/ppc64le), macOS and
+Windows (amd64/arm64); anywhere else the source build needs a Rust toolchain.
+If Core cannot be imported, detection falls back to the pure-Python engine and
+warns once — `DATAFOG_BACKEND=python` selects that path deliberately.
+
 The `nlp-advanced` extra includes SentencePiece and protobuf for GLiNER's
 multilingual tokenizer; installing the OCR extra is not required for GLiNER.
 
+## Backends
+
+`backend` defaults to `auto`: the native engine when `datafog-core` is
+importable, the pure-Python regex engine otherwise. `DATAFOG_BACKEND` sets the
+default and `backend="python"` / `backend="rust"` override it per call.
+`auto` never selects the native engine for `engine="spacy"`, `"gliner"` or
+`"smart"`, which stay on the Python ML path.
+
+The native engine adds detectors the Python engine never had — `API_KEY`,
+`BEARER_TOKEN`, `JWT`, `PRIVATE_KEY`, `CREDENTIAL_URI`, `ACCOUNT_NUMBER`,
+`US_ROUTING_NUMBER`, `NPI`, `UUID` and a German set — and serves prose `PERSON`
+and `ADDRESS` from an embedded model, so name and address detection needs no
+spaCy and no model download. Findings keep `engine="regex"` provenance and
+`confidence=1.0` as compatibility values, not native confidence estimates.
+Reviewed differences from the Python engine are listed in
+[tests/contracts/rust-1.3.0.json](tests/contracts/rust-1.3.0.json).
+`datafog.native` exposes the native schema directly.
+
 Python 3.13 support is certified for the core SDK, CLI, `nlp`,
 `nlp-advanced`, and `ocr` install profiles. Donut OCR still requires a model
 that is available locally before runtime use. `distributed` and `all` remain
@@ -188,10 +213,12 @@ scan/redact helpers, or guardrail helpers.
     model.
   - A Java runtime is required by PySpark.
 
-DataFog 4.9.0 deprecates OCR and Spark with visible use-time
-warnings; their APIs and extras will be removed in 5.0. They remain functional
-in 4.9. Users who need these features can stay on the final 4.x release. See
-the [4.9 migration guide](docs/migration-4.9.md) for the transition plan.
+DataFog deprecates OCR and Spark with visible use-time
+warnings; their APIs and extras will be removed in 5.0. The last supported
+version is 4.10.0 — pin it with `pip install "datafog==4.10.0"` — and every
+retired surface is tracked in the [retirement schedule](docs/retirement.rst).
+See the [4.9 migration guide](docs/migration-4.9.md) for the transition plan,
+and [VERSIONING.md](VERSIONING.md) for what a version number promises.
 
 ## Backward-Compatible APIs
 
@@ -261,11 +288,9 @@ The [4.9 migration guide](docs/migration-4.9.md) explains opt-in Rust detection,
 the native `datafog.v5` preview, and the revised 5.0 retirement schedule for
 `detect`/`process`, OCR, and Spark. The Python detector remains the default.
 
-The experimental Rust adapter in 4.9.0 requires Core `>=0.4.0,<0.5` and capability
-contract 1. Upgrade with `python -m pip install --upgrade "datafog[rust]==4.9.0"`
-to evaluate it. Base-only users can install `datafog==4.9.0` without Core;
-installing the Rust extra does not change the default backend. See the
-[4.9.0 release notes](RELEASE_NOTES_4.9.0.md).
+4.10.0 requires Core `>=1.3.0,<2` and capability contract 1, and installs it by
+default. See the [4.10.0 release notes](RELEASE_NOTES_4.10.0.md) and the
+[4.10 migration guide](docs/migration-4.10.md).
 Entity labels, locales, and activation settings come from the installed
 Core, allowing compatible releases to add detectors without a Python update.
 German detection is opt-in through locale or entity selection; UUID is opt-in

datafog-4.10.0/datafog/__about__.py

@@ -0,0 +1 @@
+__version__ = "4.10.0"

{datafog-4.9.0a2 → datafog-4.10.0}/datafog/__init__.py

@@ -12,6 +12,7 @@ Optional extras available for advanced features:
 import warnings
 
 from .__about__ import __version__
+from ._legacy_retirement import LAST_SUPPORTED as _LAST_SUPPORTED
 from .agent import create_guardrail, filter_output, sanitize, scan_prompt
 
 # Core API functions - always available (lightweight)
@@ -81,6 +82,13 @@ def __getattr__(name: str):
 
     elif name in _LAZY_EXPORTS:
         module_path, attr_name, extra_name = _LAZY_EXPORTS[name]
+        if attr_name is None:
+            # Submodule export (``datafog.native``): return the module itself.
+            import importlib
+
+            value = importlib.import_module(module_path)
+            globals()[name] = value
+            return value
         try:
             module = __import__(module_path, fromlist=[attr_name])
             value = getattr(module, attr_name)
@@ -104,6 +112,7 @@ def __getattr__(name: str):
 
 
 _LAZY_EXPORTS = {
+    "native": ("datafog.native", None, None),
     "app": ("datafog.client", "app", None),
     "DataFog": ("datafog.main", "DataFog", None),
     "TextPIIAnnotator": ("datafog.main", "TextPIIAnnotator", None),
@@ -131,7 +140,8 @@ _LAZY_EXPORTS = {
 def _warn_v5_replacement(old_api: str, replacement: str) -> None:
     warnings.warn(
         f"datafog.{old_api}() is deprecated and will be removed in 5.0. Use {replacement} instead. "
-        "The earlier promise to retain this shim through 5.x has been revised.",
+        f'Last supported version is {_LAST_SUPPORTED}. Pin it with: pip install "datafog=={_LAST_SUPPORTED}". '
+        "The earlier promise to retain this shim through 5.x has been revised. See docs/retirement.rst.",
         FutureWarning,
         stacklevel=3,
     )
@@ -145,7 +155,7 @@ def scan(
     allowlist: list[str] | None = None,
     allowlist_patterns: list[str] | None = None,
     *,
-    backend: str = "python",
+    backend: str | None = None,
 ) -> ScanResult:
     """Scan using the 4.x compatibility API; Rust detection is opt-in."""
     return _compat_v4.scan(
@@ -170,7 +180,7 @@ def redact(
     allowlist: list[str] | None = None,
     allowlist_patterns: list[str] | None = None,
     *,
-    backend: str = "python",
+    backend: str | None = None,
 ) -> RedactResult:
     """Redact using the 4.x compatibility API and legacy strategies."""
     return _compat_v4.redact(

{datafog-4.9.0a2 → datafog-4.10.0}/datafog/_core_capabilities.py

@@ -85,7 +85,7 @@ def _requested_locales(locales: Any, lookup: dict[str, str]) -> set[str]:
     return {_resolve_locale(locale, lookup) for locale in locales or []}
 
 
-def _activation_config(label: str, metadata: dict) -> dict:
+def _activation_config(label: str, metadata: dict) -> dict | None:
     details = metadata.get(label)
     if not isinstance(details, dict):
         raise _invalid(f"missing entity metadata for {label!r}")
@@ -95,10 +95,9 @@ def _activation_config(label: str, metadata: dict) -> dict:
     ):
         raise _invalid(f"invalid scopes for {label!r}")
     if "text" not in scopes:
-        raise ValueError(
-            f"Entity {label!r} is not available for Rust text scanning "
-            "(structured-only entity)"
-        )
+        # Structured-only labels are served by the caller, usually from the
+        # embedded model, so a mixed selection is not an error.
+        return None
     activation = details.get("activation")
     if not isinstance(activation, dict):
         raise _invalid(f"missing activation metadata for {label!r}")
@@ -120,13 +119,15 @@ def scan_configs(core: Any, requested: set[str], locales: Any) -> list[dict]:
     capabilities = getattr(core, "capabilities", None)
     if not callable(capabilities):
         raise _invalid(
-            "contract version 1 is required; install a compatible Core 0.4.x"
+            "contract version 1 is required; install a compatible datafog-core"
         )
     supported, lookup, metadata = _read_capabilities(capabilities)
     selected_locales = _requested_locales(locales, lookup)
     common: dict[str, Any] = {}
     for label in sorted(requested & supported):
         config = _activation_config(label, metadata)
+        if config is None:
+            continue
         for key, value in config.items():
             if key == "locale":
                 selected_locales.add(_resolve_locale(value, lookup))

{datafog-4.9.0a2 → datafog-4.10.0}/datafog/_legacy_retirement.py

@@ -2,6 +2,9 @@
 
 import warnings
 
+#: Final release on which retired optional surfaces still work.
+LAST_SUPPORTED = "4.10.0"
+
 
 class LegacySurfaceWarning(FutureWarning):
     """An optional DataFog surface scheduled for removal in 5.0."""
@@ -11,7 +14,8 @@ def warn_legacy_surface(surface: str) -> None:
     """Warn at the caller's public API boundary, before optional imports."""
     warnings.warn(
         f"DataFog {surface} support is deprecated in 4.9 and will be removed in 5.0. "
-        "Remain on the final 4.x release if you need continued support.",
+        f'Last supported version is {LAST_SUPPORTED}. Pin it with: pip install "datafog=={LAST_SUPPORTED}". '
+        "See docs/retirement.rst.",
         LegacySurfaceWarning,
         stacklevel=3,
     )

{datafog-4.9.0a2 → datafog-4.10.0}/datafog/compat/v4.py

@@ -6,6 +6,7 @@ not include the deprecated detect/process shims.
 
 from ..engine import Entity, RedactResult, ScanResult
 from ..engine import redact as _redact_entities
+from ..engine import resolve_backend
 from ..engine import scan as _scan
 from ..engine import scan_and_redact as _scan_and_redact
 
@@ -29,13 +30,13 @@ def scan(
     allowlist: list[str] | None = None,
     allowlist_patterns: list[str] | None = None,
     *,
-    backend: str = "python",
+    backend: str | None = None,
 ) -> ScanResult:
     """
-    Scan with the legacy result schema; Rust detection is opt-in.
+    Scan with the legacy result schema.
 
-    Defaults to the lightweight regex engine so the core install works without
-    optional dependency fallback warnings.
+    Backend selection follows ``datafog.engine.resolve_backend``: native when
+    ``datafog-core`` is importable, otherwise Python with a one-time warning.
 
     ``allowlist`` exempts exact entity texts (your own support address, doc
     placeholders); ``allowlist_patterns`` exempts entities whose full text
@@ -64,7 +65,7 @@ def redact(
     allowlist: list[str] | None = None,
     allowlist_patterns: list[str] | None = None,
     *,
-    backend: str = "python",
+    backend: str | None = None,
 ) -> RedactResult:
     """
     Redact with legacy strategies and result schema.
@@ -75,8 +76,8 @@ def redact(
     full-text regex match respectively); they apply to the scan path and are
     rejected when explicit ``entities`` are supplied.
     """
-    if backend not in ("python", "rust"):
-        raise ValueError("backend must be one of: python, rust")
+    # Validate the backend even when explicit entities skip the scan path.
+    resolve_backend(backend, engine)
 
     if preset is not None:
         try: