deidkit 0.1.4__tar.gz → 0.1.6__tar.gz

{deidkit-0.1.4/src/deidkit.egg-info → deidkit-0.1.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: deidkit
-Version: 0.1.4
+Version: 0.1.6
 Summary: Schema-driven pseudonymization for clinical/tabular datasets: synthetic names, interval-preserving date shifting, multi-stage free-text PII detection, and before/after audit exports.
 Author: Meridian Data
 License: MIT
@@ -26,13 +26,14 @@ Requires-Dist: openpyxl>=3.0
 Requires-Dist: PyYAML>=5.4
 Requires-Dist: SQLAlchemy>=1.4
 Requires-Dist: pyodbc>=4.0
-Provides-Extra: ner
-Requires-Dist: spacy>=3.4; extra == "ner"
+Requires-Dist: spacy>=3.4
 Provides-Extra: parquet
 Requires-Dist: pyarrow>=7.0; extra == "parquet"
 Provides-Extra: sql
 Requires-Dist: SQLAlchemy>=1.4; extra == "sql"
 Requires-Dist: pyodbc>=4.0; extra == "sql"
+Provides-Extra: ner
+Requires-Dist: spacy>=3.4; extra == "ner"
 Provides-Extra: all
 Requires-Dist: spacy>=3.4; extra == "all"
 Requires-Dist: pyarrow>=7.0; extra == "all"
@@ -283,8 +284,7 @@ REPLACE   PHONE        0.99  regex                  '300 123 4567'
 ```
 `REPLACE` = will be scrubbed. `review` = queued for a human instead.
 
-*In code:* `dk.Detector("es", "conservative").detect(text)` returns the same
-detections.
+*In code:* `dk.Detector(lang="es").detect(text)` returns the same detections.
 
 ### The audit workbook — evidence of every change
 
@@ -408,29 +408,43 @@ extra_stoplist_files=["learned/learned_stoplist.txt"])`.
 
 ## 8. Tuning recall & precision
 
+**By default deidkit auto-redacts** (`balanced` mode) **with the statistical NER
+stage on**, so it catches names automatically — including novel names with no
+surrounding cue — **with no setup and no flags**. spaCy ships with deidkit, and
+the ~500 MB model (too big for the pip package) is **downloaded automatically on
+the first run** that needs it:
+
+```bash
+deidkit run <input> deid_out
+# first run only: "[deidkit] NER model not found — downloading once (~500 MB)…"
+# then it's cached and every later run uses it instantly.
+```
+
+Nothing to remember, nothing to install separately. Fallbacks, if the auto-
+download can't happen (no network, read-only box):
+
+- deidkit **never fails** — it warns once and runs on the deterministic rules
+  (recall on bare/novel names is lower until the model is present).
+- Pre-warm it deliberately with `deidkit download-model`, or install the model
+  wheel offline (see the offline note below).
+- Turn auto-download off with `DEIDKIT_AUTO_DOWNLOAD=0` (then it's rules-only
+  unless the model is already installed), or drop NER entirely with `--no-ner`.
+
+Knobs, only if you want to change the defaults:
+
 | Lever | What it does |
 | --- | --- |
-| `--mode conservative` (default) | replace only clear names; queue the rest to `review.csv` |
-| `--mode balanced` / `aggressive` | replace more eagerly (higher recall, more to review) |
-| `--spacy-model es_core_news_lg` | add a statistical NER second opinion (needs `deidkit[ner]`) |
+| `--mode balanced` (default) | auto-redact confident detections, incl. single NER hits |
+| `--mode conservative` | stricter — only clear names; send the ambiguous tail to `review.csv` |
+| `--mode aggressive` | highest recall (also lone dictionary tokens); more to review |
+| `--no-ner` | turn the NER stage off (deterministic rules only) |
+| `--spacy-model NAME` | use a different spaCy model (default `es_core_news_lg`) |
 | `--id-checksums DNI,NIE` | catch *bare* national IDs (no cue word) with near-zero false positives |
 | `--name-style tagged\|placeholder\|token\|realistic` | how fakes look — `tagged` = "Juan Gómez [SINTÉTICO]" (obvious); `realistic` = unmarked |
 | `--no-medical-vocab` | (advanced) stop treating ~21k drug/disease terms as safe non-names |
 
-Start conservative, work the review queue down, and turn on the NER stage only if
-you need more recall — the audit is how you prove the result.
-
-**Enabling NER** is a one-time model install (the model is ~500 MB and ships
-separately from the package, not bundled):
-
-```bash
-pip install "deidkit[ner]"                 # installs spaCy
-deidkit download-model es_core_news_lg      # fetches the model (once)
-deidkit run <input> deid_out --spacy-model es_core_news_lg
-```
-
-If the model isn't installed, deidkit still runs on the deterministic rules and
-prints a clear warning — it never fails, but free-text name recall is lower.
+Work the `review.csv` queue for whatever the automatic pass leaves uncertain — the
+audit is how you prove the result.
 
 *In code*, the same knobs are `Policy` fields:
 
@@ -439,6 +453,17 @@ dk.Policy(mode="balanced", spacy_model="es_core_news_lg",
           id_checksums=["DNI", "NIE"], name_style="tagged")
 ```
 
+**Offline / locked-down machine** (no internet on the box that runs deidkit): the
+model is a normal pip wheel, so fetch it once on a connected machine and copy it
+over — no runtime download needed.
+
+```bash
+# on a machine with internet:
+pip download es_core_news_lg -d ./models
+# on the offline runtime box:
+pip install ./models/es_core_news_lg-*.whl
+```
+
 ---
 
 ## 9. Reading from a database (Fabric / Azure SQL)
@@ -704,7 +729,8 @@ Run `deidkit <command> --help` for any command's flags.
 ### Python `Policy` fields
 
 `entity_key` (default `"patient_id"`), `date_max_days` (365), `lang` (`"es"`),
-`mode` (`"conservative"`), `name_style` (`"tagged"`), `shift_dates_in_text`
+`mode` (`"balanced"`), `name_style` (`"tagged"`), `spacy_model`
+(`"es_core_news_lg"`, NER on by default), `shift_dates_in_text`
 (True), `only`, `ignore`, `rules` (`list[FieldRule]`), `spacy_model`,
 `known_name_columns`, `known_id_columns`, `extra_known_names`,
 `known_token_min_len` (5), `id_checksums`, `use_medical_vocab` (True),
@@ -744,7 +770,7 @@ per-patient date offsets, and write the audit yourself:
 ```python
 import deidkit as dk
 
-policy = dk.Policy(lang="es", mode="conservative", entity_key="patient_id",
+policy = dk.Policy(lang="es", entity_key="patient_id",   # mode defaults to balanced + NER
                    known_id_columns=["patient_id"], ignore={"blood_type"})
 
 deid = dk.Deidentifier(

{deidkit-0.1.4 → deidkit-0.1.6}/README.md

@@ -239,8 +239,7 @@ REPLACE   PHONE        0.99  regex                  '300 123 4567'
 ```
 `REPLACE` = will be scrubbed. `review` = queued for a human instead.
 
-*In code:* `dk.Detector("es", "conservative").detect(text)` returns the same
-detections.
+*In code:* `dk.Detector(lang="es").detect(text)` returns the same detections.
 
 ### The audit workbook — evidence of every change
 
@@ -364,29 +363,43 @@ extra_stoplist_files=["learned/learned_stoplist.txt"])`.
 
 ## 8. Tuning recall & precision
 
+**By default deidkit auto-redacts** (`balanced` mode) **with the statistical NER
+stage on**, so it catches names automatically — including novel names with no
+surrounding cue — **with no setup and no flags**. spaCy ships with deidkit, and
+the ~500 MB model (too big for the pip package) is **downloaded automatically on
+the first run** that needs it:
+
+```bash
+deidkit run <input> deid_out
+# first run only: "[deidkit] NER model not found — downloading once (~500 MB)…"
+# then it's cached and every later run uses it instantly.
+```
+
+Nothing to remember, nothing to install separately. Fallbacks, if the auto-
+download can't happen (no network, read-only box):
+
+- deidkit **never fails** — it warns once and runs on the deterministic rules
+  (recall on bare/novel names is lower until the model is present).
+- Pre-warm it deliberately with `deidkit download-model`, or install the model
+  wheel offline (see the offline note below).
+- Turn auto-download off with `DEIDKIT_AUTO_DOWNLOAD=0` (then it's rules-only
+  unless the model is already installed), or drop NER entirely with `--no-ner`.
+
+Knobs, only if you want to change the defaults:
+
 | Lever | What it does |
 | --- | --- |
-| `--mode conservative` (default) | replace only clear names; queue the rest to `review.csv` |
-| `--mode balanced` / `aggressive` | replace more eagerly (higher recall, more to review) |
-| `--spacy-model es_core_news_lg` | add a statistical NER second opinion (needs `deidkit[ner]`) |
+| `--mode balanced` (default) | auto-redact confident detections, incl. single NER hits |
+| `--mode conservative` | stricter — only clear names; send the ambiguous tail to `review.csv` |
+| `--mode aggressive` | highest recall (also lone dictionary tokens); more to review |
+| `--no-ner` | turn the NER stage off (deterministic rules only) |
+| `--spacy-model NAME` | use a different spaCy model (default `es_core_news_lg`) |
 | `--id-checksums DNI,NIE` | catch *bare* national IDs (no cue word) with near-zero false positives |
 | `--name-style tagged\|placeholder\|token\|realistic` | how fakes look — `tagged` = "Juan Gómez [SINTÉTICO]" (obvious); `realistic` = unmarked |
 | `--no-medical-vocab` | (advanced) stop treating ~21k drug/disease terms as safe non-names |
 
-Start conservative, work the review queue down, and turn on the NER stage only if
-you need more recall — the audit is how you prove the result.
-
-**Enabling NER** is a one-time model install (the model is ~500 MB and ships
-separately from the package, not bundled):
-
-```bash
-pip install "deidkit[ner]"                 # installs spaCy
-deidkit download-model es_core_news_lg      # fetches the model (once)
-deidkit run <input> deid_out --spacy-model es_core_news_lg
-```
-
-If the model isn't installed, deidkit still runs on the deterministic rules and
-prints a clear warning — it never fails, but free-text name recall is lower.
+Work the `review.csv` queue for whatever the automatic pass leaves uncertain — the
+audit is how you prove the result.
 
 *In code*, the same knobs are `Policy` fields:
 
@@ -395,6 +408,17 @@ dk.Policy(mode="balanced", spacy_model="es_core_news_lg",
           id_checksums=["DNI", "NIE"], name_style="tagged")
 ```
 
+**Offline / locked-down machine** (no internet on the box that runs deidkit): the
+model is a normal pip wheel, so fetch it once on a connected machine and copy it
+over — no runtime download needed.
+
+```bash
+# on a machine with internet:
+pip download es_core_news_lg -d ./models
+# on the offline runtime box:
+pip install ./models/es_core_news_lg-*.whl
+```
+
 ---
 
 ## 9. Reading from a database (Fabric / Azure SQL)
@@ -660,7 +684,8 @@ Run `deidkit <command> --help` for any command's flags.
 ### Python `Policy` fields
 
 `entity_key` (default `"patient_id"`), `date_max_days` (365), `lang` (`"es"`),
-`mode` (`"conservative"`), `name_style` (`"tagged"`), `shift_dates_in_text`
+`mode` (`"balanced"`), `name_style` (`"tagged"`), `spacy_model`
+(`"es_core_news_lg"`, NER on by default), `shift_dates_in_text`
 (True), `only`, `ignore`, `rules` (`list[FieldRule]`), `spacy_model`,
 `known_name_columns`, `known_id_columns`, `extra_known_names`,
 `known_token_min_len` (5), `id_checksums`, `use_medical_vocab` (True),
@@ -700,7 +725,7 @@ per-patient date offsets, and write the audit yourself:
 ```python
 import deidkit as dk
 
-policy = dk.Policy(lang="es", mode="conservative", entity_key="patient_id",
+policy = dk.Policy(lang="es", entity_key="patient_id",   # mode defaults to balanced + NER
                    known_id_columns=["patient_id"], ignore={"blood_type"})
 
 deid = dk.Deidentifier(

{deidkit-0.1.4 → deidkit-0.1.6}/examples/quickstart.md

@@ -4,8 +4,9 @@
 
 ```bash
 pip install -e .
-# optional: statistical NER second opinion
-pip install -e ".[ner]" && python -m spacy download es_core_news_lg
+# NER runs by default; its ~500 MB model auto-downloads on the first run that
+# needs it (spaCy already ships with deidkit). Pre-fetch it with:
+#   deidkit download-model
 ```
 
 If the `deidkit` command isn't on your PATH, use `python -m deidkit ...`
@@ -66,6 +67,6 @@ python examples/benchmark.py --n 400 --spacy es_core_news_lg
 | tokens instead of fake names           | `--names-token`                       |
 | don't shift dates inside free text     | `--no-date-text`                      |
 | wider/narrower date window             | `--date-max-days 30`                  |
-| enable statistical NER                 | `--spacy-model es_core_news_lg`       |
+| turn OFF the default NER stage         | `--no-ner`                            |
 | Parquet output                         | `--format parquet`                    |
 | load a YAML policy                     | `--policy examples/policy.example.yaml` |

{deidkit-0.1.4 → deidkit-0.1.6}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "deidkit"
-version = "0.1.4"
+version = "0.1.6"
 description = "Schema-driven pseudonymization for clinical/tabular datasets: synthetic names, interval-preserving date shifting, multi-stage free-text PII detection, and before/after audit exports."
 readme = "README.md"
 requires-python = ">=3.9"
@@ -40,16 +40,17 @@ dependencies = [
   # from the Microsoft Fabric SQL analytics endpoint (the primary data source).
   "SQLAlchemy>=1.4",
   "pyodbc>=4.0",
+  # spaCy is core so NER is on by default with no extra install step or bracket.
+  # The model itself is separate (too big for PyPI): `deidkit download-model`.
+  "spacy>=3.4",
 ]
 
 [project.optional-dependencies]
-# Statistical NER stage (second-opinion recall). After install run:
-#   python -m spacy download es_core_news_lg
-ner = ["spacy>=3.4"]
 # Read/write Parquet.
 parquet = ["pyarrow>=7.0"]
-# Back-compat alias: the DB drivers are now core, so `deidkit[sql]` == `deidkit`.
+# Back-compat aliases: these are now core, so `deidkit[sql]` / `deidkit[ner]` == `deidkit`.
 sql = ["SQLAlchemy>=1.4", "pyodbc>=4.0"]
+ner = ["spacy>=3.4"]
 # Everything except the heavy spaCy model download.
 all = ["spacy>=3.4", "pyarrow>=7.0", "SQLAlchemy>=1.4", "pyodbc>=4.0"]
 dev = ["pytest>=7.0"]
@@ -70,3 +71,12 @@ where = ["src"]
 
 [tool.setuptools.package-data]
 deidkit = ["data/*.txt", "data/*.md"]
+
+[tool.pytest.ini_options]
+filterwarnings = [
+  # The NER model / spaCy aren't present in CI; NER's graceful-fallback warnings
+  # are expected there and are exercised directly in test_ner_model_helpers.py.
+  "ignore:spaCy is missing.*:UserWarning",
+  "ignore:NER model.*auto-download is off.*:UserWarning",
+  "ignore:could not auto-download.*:UserWarning",
+]

{deidkit-0.1.4 → deidkit-0.1.6}/src/deidkit/__init__.py

@@ -4,7 +4,7 @@ Quick start (library)::
 
     import deidkit as dk
 
-    policy = dk.Policy(lang="es", mode="conservative", ignore=["blood_type"])
+    policy = dk.Policy(lang="es", ignore=["blood_type"])   # balanced + NER by default
     deid = dk.Deidentifier(
         policy,
         secret="keep-this-secret",

{deidkit-0.1.4 → deidkit-0.1.6}/src/deidkit/cli.py

@@ -50,6 +50,8 @@ def build_policy(args) -> Policy:
         policy.date_max_days = args.date_max_days
     if getattr(args, "spacy_model", None):
         policy.spacy_model = args.spacy_model
+    if getattr(args, "no_ner", False):
+        policy.spacy_model = None
     if getattr(args, "names_token", False):
         policy.names_synthetic = False
     if getattr(args, "no_date_text", False):
@@ -222,14 +224,14 @@ def cmd_run_db(args) -> int:
 
 
 def cmd_download_model(args) -> int:
-    """Download the spaCy model the optional NER stage uses (one-time, ~500 MB)."""
+    """Download the spaCy NER model (used by default). One-time, ~500 MB."""
     model = args.model
     try:
         import spacy  # noqa: F401
         from spacy.cli import download
     except ImportError:
-        print('error: spaCy is not installed. First run:  pip install "deidkit[ner]"',
-              file=sys.stderr)
+        print("error: spaCy is missing (it ships with deidkit) — reinstall with "
+              "`pip install --force-reinstall deidkit`.", file=sys.stderr)
         return 2
     print(f"Downloading spaCy model '{model}' (~500 MB, one-time)…")
     try:
@@ -239,8 +241,8 @@ def cmd_download_model(args) -> int:
             print(f"error: could not download '{model}'. Check the model name and "
                   f"your network.", file=sys.stderr)
             return 2
-    print(f"Done. Enable NER with:  --spacy-model {model}  "
-          f"(or spacy_model='{model}' in code).")
+    print("Done. NER now runs by default; no flag needed. "
+          "(Turn it off with --no-ner if you ever want rules-only.)")
     return 0
 
 
@@ -282,7 +284,8 @@ def cmd_run(args) -> int:
         print(f"      secret  : deidkit-secret.key (KEEP — needed to re-run consistently)")
     print(f"\n  cells changed      : {summ['cells_changed']:,}")
     print(f"  to review          : {summ['review_detections']:,}")
-    print(f"  detection          : {summ['ner_status']} (deterministic rules; no AI by default)")
+    print(f"  NER stage          : {summ['ner_status']}  (deterministic rules always run; "
+          f"{'add --no-ner for rules-only' if summ['ner_status'].startswith('active') else 'download-model to enable'})")
     for etype, cnt in sorted(summ["entity_counts"].items(), key=lambda x: -x[1]):
         print(f"      {etype:<14} {cnt:,}")
     return 0
@@ -294,7 +297,7 @@ def _add_policy_args(p):
     p.add_argument("--lang", choices=["es", "en", "multi"],
                    help="Language(s) for names/detection ('multi' = ES+EN).")
     p.add_argument("--mode", choices=sorted(ALL_MODES),
-                   help="Free-text detection aggressiveness (default conservative).")
+                   help="Free-text detection aggressiveness (default balanced = auto-redact).")
     p.add_argument("--name-style", choices=["tagged", "placeholder", "token", "realistic"],
                    help="How fake names look (default tagged = obviously synthetic).")
     p.add_argument("--only", help="Comma list: transform ONLY these columns.")
@@ -308,7 +311,9 @@ def _add_policy_args(p):
     p.add_argument("--no-date-text", action="store_true",
                    help="Do not shift date strings found inside free text.")
     p.add_argument("--spacy-model",
-                   help="Enable spaCy NER stage with this model (e.g. es_core_news_lg).")
+                   help="Override the NER model (default es_core_news_lg; on by default).")
+    p.add_argument("--no-ner", action="store_true",
+                   help="Turn off the statistical NER stage (deterministic rules only).")
     p.add_argument("--known-name-column", action="append",
                    help="Column whose value is a known person name to force-scrub "
                         "from that row's free text (repeatable).")
@@ -404,7 +409,7 @@ def build_parser() -> argparse.ArgumentParser:
 
     pdm = sub.add_parser(
         "download-model",
-        help="Download the spaCy NER model for the optional NER stage (needs deidkit[ner]).")
+        help="Download the spaCy NER model (used by default; one-time ~500 MB).")
     pdm.add_argument("model", nargs="?", default="es_core_news_lg",
                      help="spaCy model name (default es_core_news_lg).")
     pdm.set_defaults(func=cmd_download_model)

{deidkit-0.1.4 → deidkit-0.1.6}/src/deidkit/config.py

@@ -49,9 +49,9 @@ ALL_STRATEGIES = frozenset(
 )
 
 # Detection aggressiveness for the free-text stage.
-MODE_CONSERVATIVE = "conservative"  # only replace when clearly a name (default)
-MODE_BALANCED = "balanced"          # also accept strong single-signal name hits
-MODE_AGGRESSIVE = "aggressive"      # accept any plausible person span (higher recall)
+MODE_CONSERVATIVE = "conservative"  # only replace when clearly a name (review the rest)
+MODE_BALANCED = "balanced"          # also auto-redact single-signal / NER hits (default)
+MODE_AGGRESSIVE = "aggressive"      # accept any plausible person span (highest recall)
 ALL_MODES = frozenset({MODE_CONSERVATIVE, MODE_BALANCED, MODE_AGGRESSIVE})
 
 # How replacement person names look. All are OBVIOUSLY synthetic except
@@ -129,21 +129,24 @@ class Policy:
     rules:
         Explicit per-column overrides evaluated before the schema auto-plan.
     spacy_model:
-        If set (e.g. ``"es_core_news_lg"``) the optional statistical NER stage
-        is enabled as a corroborating detector.
+        The statistical NER model, ON by default (``"es_core_news_lg"``) so novel
+        names are caught with no extra flags. It loads lazily and only if present
+        — if the model isn't downloaded yet, deidkit warns once and runs on the
+        deterministic stages. Download it with ``deidkit download-model``; set to
+        ``None`` to turn NER off.
     """
 
     entity_key: str = "patient_id"
     date_max_days: int = 365
     lang: str = "es"
-    mode: str = MODE_CONSERVATIVE
+    mode: str = MODE_BALANCED
     name_style: str = NAME_TAGGED
     names_synthetic: bool = True
     shift_dates_in_text: bool = True
     only: Optional[set] = None
     ignore: set = field(default_factory=set)
     rules: List[FieldRule] = field(default_factory=list)
-    spacy_model: Optional[str] = None
+    spacy_model: Optional[str] = "es_core_news_lg"
     # Advanced: extra gazetteer/stoplist files to merge on top of the bundled ones.
     extra_name_files: List[str] = field(default_factory=list)
     extra_stoplist_files: List[str] = field(default_factory=list)

{deidkit-0.1.4 → deidkit-0.1.6}/src/deidkit/detect/pipeline.py

@@ -52,7 +52,7 @@ class Detector:
     def __init__(
         self,
         lang: str = "es",
-        mode: str = "conservative",
+        mode: str = "balanced",
         spacy_model: Optional[str] = None,
         extra_name_files=(),
         extra_stoplist_files=(),
@@ -70,9 +70,7 @@ class Detector:
 
     @property
     def ner_status(self) -> str:
-        if self.ner is None:
-            return "disabled"
-        return "active" if self.ner.available else f"unavailable ({self.ner.error})"
+        return "disabled" if self.ner is None else self.ner.ner_status
 
     # ------------------------------------------------------------------ #
     def detect(self, text: str) -> List[Detection]:
@@ -94,7 +92,7 @@ class Detector:
             raw += checksums.find_valid_ids(text, self.id_checksums)
         raw += context.find(text, tokens, self.gaz)
         raw += gazetteer.find(text, tokens, self.gaz)
-        if self.ner is not None and self.ner.available:
+        if self.ner is not None:  # find() lazy-loads the model and returns [] if absent
             raw += self.ner.find(text)
 
         idents = [d for d in raw if d.entity_type in _IDENTIFIER_TYPES]

deidkit-0.1.6/src/deidkit/detect/spacy_ner.py

@@ -0,0 +1,131 @@
+"""Statistical NER stage (spaCy).
+
+On by default (``Policy.spacy_model`` = ``"es_core_news_lg"``). The model loads
+**lazily** on first use, and if it isn't installed yet deidkit **auto-downloads
+it once** — so there is no separate install step for anyone to forget. Everything
+degrades gracefully: if the download can't happen (no network, read-only env),
+``available`` is ``False``, :meth:`find` returns nothing, a one-time warning names
+the fix, and the tool still runs on the deterministic stages.
+
+Auto-download is on by default; set ``DEIDKIT_AUTO_DOWNLOAD=0`` to disable it (for
+offline/locked-down machines, pre-install the model wheel instead).
+"""
+
+from __future__ import annotations
+
+import importlib
+import os
+import sys
+import warnings
+from typing import List
+
+from .types import Detection, PERSON
+
+_PERSON_LABELS = {"PER", "PERSON"}
+
+
+def _auto_download_enabled() -> bool:
+    return os.environ.get("DEIDKIT_AUTO_DOWNLOAD", "1").strip().lower() not in (
+        "0", "false", "no", "off",
+    )
+
+
+class SpacyNER:
+    def __init__(self, model: str):
+        self.model = model
+        self._nlp = None
+        self._loaded = False
+        self.available = False
+        self.error = None
+
+    def _try_load(self) -> bool:
+        try:
+            import spacy
+
+            self._nlp = spacy.load(self.model, disable=["lemmatizer", "textcat"])
+            self.available = True
+            return True
+        except Exception as exc:  # OSError (model absent) / ImportError (spaCy absent)
+            self.error = str(exc)
+            return False
+
+    def _ensure_loaded(self) -> None:
+        """Load on first use; auto-download the model once if it isn't installed."""
+        if self._loaded:
+            return
+        self._loaded = True
+
+        try:
+            import spacy  # noqa: F401
+        except Exception as exc:
+            self.error = str(exc)
+            warnings.warn(
+                f"spaCy is missing (it ships with deidkit); NER is off. Reinstall "
+                f"with `pip install --force-reinstall deidkit`. ({exc})",
+                stacklevel=3,
+            )
+            return
+
+        if self._try_load():
+            return  # already installed
+
+        if not _auto_download_enabled():
+            warnings.warn(
+                f"NER model {self.model!r} is not installed and auto-download is off "
+                f"(DEIDKIT_AUTO_DOWNLOAD=0); running rules-only. Pre-install it with "
+                f"`deidkit download-model {self.model}`.",
+                stacklevel=3,
+            )
+            return
+
+        # Auto-download once (this run), then load in-process.
+        try:
+            print(f"[deidkit] NER model {self.model!r} not found — downloading once "
+                  f"(~500 MB, first run only)…", file=sys.stderr, flush=True)
+            from spacy.cli import download
+
+            download(self.model)
+            importlib.invalidate_caches()  # make the just-installed package importable now
+            if self._try_load():
+                print("[deidkit] NER model ready.", file=sys.stderr, flush=True)
+                return
+        except SystemExit as exc:      # spacy.cli.download can sys.exit on failure
+            self.error = f"download exited ({exc.code})"
+        except Exception as exc:
+            self.error = str(exc)
+
+        warnings.warn(
+            f"could not auto-download NER model {self.model!r} ({self.error}); running "
+            f"on the deterministic rules only. Fixes: ensure network access and rerun, "
+            f"run `deidkit download-model {self.model}`, or set DEIDKIT_AUTO_DOWNLOAD=0 "
+            f"to stay rules-only quietly.",
+            stacklevel=3,
+        )
+
+    @property
+    def ner_status(self) -> str:
+        if not self._loaded:
+            return "pending (loads on first use)"
+        return "active" if self.available else f"unavailable ({self.error})"
+
+    def find(self, text: str) -> List[Detection]:
+        if not text:
+            return []
+        self._ensure_loaded()
+        if not self.available:
+            return []
+        doc = self._nlp(text)
+        out: List[Detection] = []
+        for ent in doc.ents:
+            if ent.label_ in _PERSON_LABELS:
+                out.append(
+                    Detection(
+                        start=ent.start_char,
+                        end=ent.end_char,
+                        text=ent.text,
+                        entity_type=PERSON,
+                        methods={"ner"},
+                        meta={"gender": "u"},
+                    )
+                )
+        return out

{deidkit-0.1.4 → deidkit-0.1.6}/src/deidkit/version.py

@@ -1,3 +1,3 @@
 """Single source of truth for the package version."""
 
-__version__ = "0.1.4"
+__version__ = "0.1.6"

{deidkit-0.1.4 → deidkit-0.1.6/src/deidkit.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: deidkit
-Version: 0.1.4
+Version: 0.1.6
 Summary: Schema-driven pseudonymization for clinical/tabular datasets: synthetic names, interval-preserving date shifting, multi-stage free-text PII detection, and before/after audit exports.
 Author: Meridian Data
 License: MIT
@@ -26,13 +26,14 @@ Requires-Dist: openpyxl>=3.0
 Requires-Dist: PyYAML>=5.4
 Requires-Dist: SQLAlchemy>=1.4
 Requires-Dist: pyodbc>=4.0
-Provides-Extra: ner
-Requires-Dist: spacy>=3.4; extra == "ner"
+Requires-Dist: spacy>=3.4
 Provides-Extra: parquet
 Requires-Dist: pyarrow>=7.0; extra == "parquet"
 Provides-Extra: sql
 Requires-Dist: SQLAlchemy>=1.4; extra == "sql"
 Requires-Dist: pyodbc>=4.0; extra == "sql"
+Provides-Extra: ner
+Requires-Dist: spacy>=3.4; extra == "ner"
 Provides-Extra: all
 Requires-Dist: spacy>=3.4; extra == "all"
 Requires-Dist: pyarrow>=7.0; extra == "all"
@@ -283,8 +284,7 @@ REPLACE   PHONE        0.99  regex                  '300 123 4567'
 ```
 `REPLACE` = will be scrubbed. `review` = queued for a human instead.
 
-*In code:* `dk.Detector("es", "conservative").detect(text)` returns the same
-detections.
+*In code:* `dk.Detector(lang="es").detect(text)` returns the same detections.
 
 ### The audit workbook — evidence of every change
 
@@ -408,29 +408,43 @@ extra_stoplist_files=["learned/learned_stoplist.txt"])`.
 
 ## 8. Tuning recall & precision
 
+**By default deidkit auto-redacts** (`balanced` mode) **with the statistical NER
+stage on**, so it catches names automatically — including novel names with no
+surrounding cue — **with no setup and no flags**. spaCy ships with deidkit, and
+the ~500 MB model (too big for the pip package) is **downloaded automatically on
+the first run** that needs it:
+
+```bash
+deidkit run <input> deid_out
+# first run only: "[deidkit] NER model not found — downloading once (~500 MB)…"
+# then it's cached and every later run uses it instantly.
+```
+
+Nothing to remember, nothing to install separately. Fallbacks, if the auto-
+download can't happen (no network, read-only box):
+
+- deidkit **never fails** — it warns once and runs on the deterministic rules
+  (recall on bare/novel names is lower until the model is present).
+- Pre-warm it deliberately with `deidkit download-model`, or install the model
+  wheel offline (see the offline note below).
+- Turn auto-download off with `DEIDKIT_AUTO_DOWNLOAD=0` (then it's rules-only
+  unless the model is already installed), or drop NER entirely with `--no-ner`.
+
+Knobs, only if you want to change the defaults:
+
 | Lever | What it does |
 | --- | --- |
-| `--mode conservative` (default) | replace only clear names; queue the rest to `review.csv` |
-| `--mode balanced` / `aggressive` | replace more eagerly (higher recall, more to review) |
-| `--spacy-model es_core_news_lg` | add a statistical NER second opinion (needs `deidkit[ner]`) |
+| `--mode balanced` (default) | auto-redact confident detections, incl. single NER hits |
+| `--mode conservative` | stricter — only clear names; send the ambiguous tail to `review.csv` |
+| `--mode aggressive` | highest recall (also lone dictionary tokens); more to review |
+| `--no-ner` | turn the NER stage off (deterministic rules only) |
+| `--spacy-model NAME` | use a different spaCy model (default `es_core_news_lg`) |
 | `--id-checksums DNI,NIE` | catch *bare* national IDs (no cue word) with near-zero false positives |
 | `--name-style tagged\|placeholder\|token\|realistic` | how fakes look — `tagged` = "Juan Gómez [SINTÉTICO]" (obvious); `realistic` = unmarked |
 | `--no-medical-vocab` | (advanced) stop treating ~21k drug/disease terms as safe non-names |
 
-Start conservative, work the review queue down, and turn on the NER stage only if
-you need more recall — the audit is how you prove the result.
-
-**Enabling NER** is a one-time model install (the model is ~500 MB and ships
-separately from the package, not bundled):
-
-```bash
-pip install "deidkit[ner]"                 # installs spaCy
-deidkit download-model es_core_news_lg      # fetches the model (once)
-deidkit run <input> deid_out --spacy-model es_core_news_lg
-```
-
-If the model isn't installed, deidkit still runs on the deterministic rules and
-prints a clear warning — it never fails, but free-text name recall is lower.
+Work the `review.csv` queue for whatever the automatic pass leaves uncertain — the
+audit is how you prove the result.
 
 *In code*, the same knobs are `Policy` fields:
 
@@ -439,6 +453,17 @@ dk.Policy(mode="balanced", spacy_model="es_core_news_lg",
           id_checksums=["DNI", "NIE"], name_style="tagged")
 ```
 
+**Offline / locked-down machine** (no internet on the box that runs deidkit): the
+model is a normal pip wheel, so fetch it once on a connected machine and copy it
+over — no runtime download needed.
+
+```bash
+# on a machine with internet:
+pip download es_core_news_lg -d ./models
+# on the offline runtime box:
+pip install ./models/es_core_news_lg-*.whl
+```
+
 ---
 
 ## 9. Reading from a database (Fabric / Azure SQL)
@@ -704,7 +729,8 @@ Run `deidkit <command> --help` for any command's flags.
 ### Python `Policy` fields
 
 `entity_key` (default `"patient_id"`), `date_max_days` (365), `lang` (`"es"`),
-`mode` (`"conservative"`), `name_style` (`"tagged"`), `shift_dates_in_text`
+`mode` (`"balanced"`), `name_style` (`"tagged"`), `spacy_model`
+(`"es_core_news_lg"`, NER on by default), `shift_dates_in_text`
 (True), `only`, `ignore`, `rules` (`list[FieldRule]`), `spacy_model`,
 `known_name_columns`, `known_id_columns`, `extra_known_names`,
 `known_token_min_len` (5), `id_checksums`, `use_medical_vocab` (True),
@@ -744,7 +770,7 @@ per-patient date offsets, and write the audit yourself:
 ```python
 import deidkit as dk
 
-policy = dk.Policy(lang="es", mode="conservative", entity_key="patient_id",
+policy = dk.Policy(lang="es", entity_key="patient_id",   # mode defaults to balanced + NER
                    known_id_columns=["patient_id"], ignore={"blood_type"})
 
 deid = dk.Deidentifier(

{deidkit-0.1.4 → deidkit-0.1.6}/src/deidkit.egg-info/requires.txt

@@ -3,6 +3,7 @@ openpyxl>=3.0
 PyYAML>=5.4
 SQLAlchemy>=1.4
 pyodbc>=4.0
+spacy>=3.4
 
 [all]
 spacy>=3.4

deidkit-0.1.6/tests/test_ner_model_helpers.py

@@ -0,0 +1,43 @@
+"""The NER model is separate from the package (too big for PyPI); it must be a
+one-command install, and a missing model must warn loudly (not silently reduce
+recall). These tests cover the helper command and the warning — without ever
+downloading a model."""
+
+import warnings
+
+from deidkit.cli import build_parser, cmd_download_model
+from deidkit.detect.spacy_ner import SpacyNER
+
+
+def test_missing_model_degrades_gracefully(monkeypatch):
+    monkeypatch.setenv("DEIDKIT_AUTO_DOWNLOAD", "0")            # never hit the network here
+    ner = SpacyNER("definitely_not_a_real_model_zzz")
+    assert ner.available is False                               # lazy: not loaded yet
+    with warnings.catch_warnings(record=True) as caught:
+        warnings.simplefilter("always")
+        result = ner.find("Paciente Juan Perez consulta.")      # loads lazily -> warns
+    assert result == []                                         # degrades, no crash
+    assert ner.available is False                               # load failed
+    msgs = " ".join(str(w.message) for w in caught)
+    assert "NER" in msgs and "deidkit" in msgs                  # loud + actionable
+
+
+def test_auto_download_enabled_env_toggle(monkeypatch):
+    from deidkit.detect.spacy_ner import _auto_download_enabled
+
+    monkeypatch.delenv("DEIDKIT_AUTO_DOWNLOAD", raising=False)
+    assert _auto_download_enabled() is True                     # on by default
+    for off in ("0", "false", "no", "off"):
+        monkeypatch.setenv("DEIDKIT_AUTO_DOWNLOAD", off)
+        assert _auto_download_enabled() is False
+
+
+def test_cli_wires_download_model_with_default():
+    args = build_parser().parse_args(["download-model"])
+    assert args.func is cmd_download_model
+    assert args.model == "es_core_news_lg"
+
+
+def test_cli_download_model_accepts_explicit_name():
+    args = build_parser().parse_args(["download-model", "en_core_web_sm"])
+    assert args.model == "en_core_web_sm"

deidkit-0.1.4/src/deidkit/detect/spacy_ner.py

@@ -1,63 +0,0 @@
-"""Optional statistical NER stage (spaCy).
-
-Enabled only when ``Policy.spacy_model`` is set and spaCy + the model are
-installed. It is used as an independent PERSON detector and to corroborate
-gazetteer hits. Everything degrades gracefully: if the model is missing,
-``available`` is ``False`` and :meth:`find` returns nothing, so the tool still
-runs on the deterministic stages alone.
-
-    pip install "deidkit[ner]"
-    python -m spacy download es_core_news_lg
-"""
-
-from __future__ import annotations
-
-from typing import List
-
-from .types import Detection, PERSON
-
-_PERSON_LABELS = {"PER", "PERSON"}
-
-
-class SpacyNER:
-    def __init__(self, model: str):
-        self.model = model
-        self._nlp = None
-        self.available = False
-        self.error = None
-        try:
-            import spacy
-
-            self._nlp = spacy.load(model, disable=["lemmatizer", "textcat"])
-            self.available = True
-        except Exception as exc:  # ImportError or OSError (model not found)
-            self.error = str(exc)
-            import warnings
-
-            warnings.warn(
-                f"NER requested but spaCy model {model!r} is not available "
-                f"({type(exc).__name__}). The NER stage is OFF, so free-text name "
-                f"recall is reduced — names not in the gazetteer/anchored may be "
-                f"missed. Install it once with:  deidkit download-model {model}  "
-                f'(after: pip install "deidkit[ner]").',
-                stacklevel=3,
-            )
-
-    def find(self, text: str) -> List[Detection]:
-        if not self.available or not text:
-            return []
-        doc = self._nlp(text)
-        out: List[Detection] = []
-        for ent in doc.ents:
-            if ent.label_ in _PERSON_LABELS:
-                out.append(
-                    Detection(
-                        start=ent.start_char,
-                        end=ent.end_char,
-                        text=ent.text,
-                        entity_type=PERSON,
-                        methods={"ner"},
-                        meta={"gender": "u"},
-                    )
-                )
-        return out

deidkit-0.1.4/tests/test_ner_model_helpers.py

@@ -1,30 +0,0 @@
-"""The NER model is separate from the package (too big for PyPI); it must be a
-one-command install, and a missing model must warn loudly (not silently reduce
-recall). These tests cover the helper command and the warning — without ever
-downloading a model."""
-
-import warnings
-
-from deidkit.cli import build_parser, cmd_download_model
-from deidkit.detect.spacy_ner import SpacyNER
-
-
-def test_missing_model_warns_and_disables():
-    with warnings.catch_warnings(record=True) as caught:
-        warnings.simplefilter("always")
-        ner = SpacyNER("definitely_not_a_real_model_zzz")
-    assert ner.available is False
-    assert ner.find("Paciente Juan Perez consulta.") == []      # degrades, no crash
-    msgs = " ".join(str(w.message) for w in caught)
-    assert "download-model" in msgs and "NER" in msgs           # loud + actionable
-
-
-def test_cli_wires_download_model_with_default():
-    args = build_parser().parse_args(["download-model"])
-    assert args.func is cmd_download_model
-    assert args.model == "es_core_news_lg"
-
-
-def test_cli_download_model_accepts_explicit_name():
-    args = build_parser().parse_args(["download-model", "en_core_web_sm"])
-    assert args.model == "en_core_web_sm"