deidkit 0.1.0__tar.gz → 0.1.2__tar.gz

{deidkit-0.1.0/src/deidkit.egg-info → deidkit-0.1.2}/PKG-INFO

@@ -1,12 +1,12 @@
 Metadata-Version: 2.4
 Name: deidkit
-Version: 0.1.0
+Version: 0.1.2
 Summary: Schema-driven pseudonymization for clinical/tabular datasets: synthetic names, interval-preserving date shifting, multi-stage free-text PII detection, and before/after audit exports.
 Author: Meridian Data
 License: MIT
-Project-URL: Homepage, https://github.com/your-org/deidkit
-Project-URL: Documentation, https://github.com/your-org/deidkit#readme
-Project-URL: Issues, https://github.com/your-org/deidkit/issues
+Project-URL: Homepage, https://github.com/Meridian-Int/meridian-deid-engine
+Project-URL: Documentation, https://github.com/Meridian-Int/meridian-deid-engine#readme
+Project-URL: Issues, https://github.com/Meridian-Int/meridian-deid-engine/issues
 Keywords: de-identification,pseudonymization,hipaa,phi,ner,clinical,healthcare,privacy
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.9
@@ -79,18 +79,6 @@ re-runnable.
 pip install deidkit
 ```
 
-One command, no square brackets, nothing to pick. The database drivers needed to
-read your Azure SQL / Fabric source (SQLAlchemy + pyodbc) are bundled in, so a
-plain `pip install deidkit` is ready to connect.
-
-> **Heads-up — not on PyPI yet.** Until the package is published (see
-> [Publishing](#publishing-to-pypi)), install it straight from the repository:
-> ```bash
-> pip install git+https://github.com/<your-org>/deidkit.git
-> # or, from a local clone:  pip install .
-> ```
-> Both give you the identical `deidkit` command and library.
-
 One system-level thing is required **once**, on the machine that will connect:
 the Microsoft **ODBC Driver 18 for SQL Server** — the actual database driver that
 `pyodbc` talks to:
@@ -113,38 +101,46 @@ deidkit --version
 
 ## 2. Quick start
 
-**CLI — one line:**
+Your data is the **Azure SQL Database mirrored into Microsoft Fabric**, so the
+run connects to that endpoint, de-identifies the six tables together, and writes
+a clean copy out. That's the whole thing:
 
 ```bash
-deidkit run data/
+deidkit init-secret --out secret.key       # do this once; keep the file private
 ```
 
-Reads every table in `data/`, de-identifies it, and writes two folders:
-`data_deid/` (ship this) and `data_deid-PRIVATE/` (keep this — see §5).
-
-**Python — one call:**
-
 ```python
 import deidkit as dk
+from deidkit.io import read_sql_table
 
-result = dk.deidentify("data/", out="deid_out", secret_file="secret.key")
-print(result.summary)                      # counts of what changed
-clean = result.tables["tbl_notes"]         # or write to files via out=
-```
-
-**The recommended run** (pins a secret, uses the schema-tuned policy):
+# connection string built from the values in section 10 (nothing writes back)
+CONN = (
+    "mssql+pyodbc://@<sql-endpoint>.datawarehouse.fabric.microsoft.com/<database>"
+    "?driver=ODBC+Driver+18+for+SQL+Server&authentication=ActiveDirectoryInteractive"
+)
+TABLES = ["tbl_patient", "tbl_encounter", "tbl_notes",
+          "tbl_lab", "tbl_imaging", "tbl_medication"]
 
-```bash
-deidkit init-secret --out secret.key       # do this once, keep the file private
-deidkit run data/ deid_out \
-    --policy examples/policy.meridian.yaml \
-    --secret-file secret.key
+tables = {t: read_sql_table(CONN, t) for t in TABLES}     # read the DB (read-only)
+result = dk.deidentify(
+    tables, out="deid_out",                               # de-identify all six together
+    secret_file="secret.key",
+    lang="es",
+    entity_key="patient_id",
+    known_id_columns=["patient_id"],
+)
+print(result.summary)
+# -> deid_out/ (ship this)  +  deid_out-PRIVATE/ (keep private — see section 5)
 ```
 
-> The `data/` above is a folder of table files, used here just to show the
-> shape of a run. **Your data is in the Fabric / Azure SQL database, not a
-> folder** — for that path (what you connect to and how), jump to **§9** and
-> **§10**; it's a few lines that read straight from the SQL endpoint.
+That reads the tables from Fabric, never writes back, and produces the two output
+folders. Section 9 walks through the connection (plus a database-to-database
+variant), and section 10 lists the exact connection values to put in `CONN`.
+
+> **Handed a file export instead of database access?** If you receive the tables
+> as files (CSV etc.) rather than a live endpoint, the same run is one CLI line —
+> `deidkit run data/` — writing `data_deid/` and `data_deid-PRIVATE/`. See
+> section 3.
 
 > **CLI or code — same engine.** Everything below is shown for **both**
 > interfaces. The rule of thumb: every CLI `--flag` is a field on the Python
@@ -163,10 +159,10 @@ mirrored into Microsoft Fabric. deidkit connects to the Fabric **SQL analytics
 endpoint** (read-only), reads the six tables directly, de-identifies them, and
 writes the clean copy out. It **never writes back** to your database.
 
-To connect, we need a few **connection details from you** (the data owner) — the
-SQL endpoint, the database name, and how to authenticate. Exactly what to send,
-and how the connection works, is spelled out in **§9 (how it runs)** and **§10
-(the connection details you provide)**. Start there.
+deidkit connects on its own — it only needs the connection details for that
+endpoint (the SQL analytics endpoint host, the database name, and how to
+authenticate). Those go into one connection string; section 9 shows how a run
+works and section 10 lists the exact values. Start there.
 
 ### Or: a data export as files
 
@@ -189,7 +185,7 @@ deidkit run data/            # a folder of table files
 - **Column names drive the plan.** deidkit recognises clinical column names in
   **English and Spanish** (e.g. `note_full`, `result_interpretation`,
   `report_text`, `clinical_justification_pbs`) and classifies each one
-  automatically. You can override any decision (§12). An optional
+  automatically. You can override any decision (section 12). An optional
   **data-dictionary JSON** (`--schema dict.json`) makes the classification
   authoritative instead of heuristic.
 
@@ -212,7 +208,7 @@ force any of them with a rule.
 | `redact` | blank the value | |
 | `drop` | remove the column entirely | |
 
-Preview the full plan before running anything with `deidkit plan` (§6).
+Preview the full plan before running anything with `deidkit plan` (section 6).
 
 ---
 
@@ -233,9 +229,9 @@ deid_out/                         ← SHIP THIS — de-identified data only
 deid_out-PRIVATE/                 ← NEVER SHARE — the re-identification key
     deidkit-secret.key            ← the secret (only if it was auto-generated)
     mapping.private.json          ← value → surrogate map (reverses/extends the run)
-    deid_audit.PRIVATE.xlsx       ← before/after audit workbook (see §6)
+    deid_audit.PRIVATE.xlsx       ← before/after audit workbook (see section 6)
     deid_audit.PRIVATE_freetext_changes.csv
-    review.csv                    ← uncertain detections for a human (see §7)
+    review.csv                    ← uncertain detections for a human (see section 7)
 ```
 
 - **`deid_out/`** — the pseudonymized tables, same structure as the input. This
@@ -245,7 +241,7 @@ deid_out-PRIVATE/                 ← NEVER SHARE — the re-identification key
 
 **Output format** defaults to CSV; use `--format parquet|tsv|json` (CLI) or
 `fmt=` (Python) to change it. Reading from a DB and want the result written back
-to a DB? See `run-db` in §9.
+to a DB? See `run-db` in section 9.
 
 ---
 
@@ -316,32 +312,70 @@ print(los.describe())   # identical to the source — only absolute dates moved
 
 ## 7. Training it — the review → learn loop
 
-deidkit deliberately **does not guess** on ambiguous names. Instead it puts them
-in `review.csv`, and you teach it. This gets better every pass, is fully
-auditable, and uses no black-box model.
+deidkit **never guesses** on an ambiguous name. Anything it detects but isn't sure
+about (for example a lone surname that is also a common word) is left untouched
+and listed in **`review.csv`** for a person to rule on. Confirm or reject those,
+and the next run does better — deterministically, with no model.
+
+`review.csv` holds only that **uncertain tail**; the confident detections are
+already scrubbed in the output and never appear here. It is written to the
+**private** folder on every run (`deid_out-PRIVATE/review.csv`) and contains raw
+candidate text, so treat it as sensitive — never ship it.
+
+### What the reviewer does
+
+Open `deid_out-PRIVATE/review.csv` (Excel, Google Sheets, any editor). Each row is
+one uncertain candidate, with a `context` snippet so you can judge it. Fill in the
+**first column only** — `decision(y/n)` — and save:
+
+| decision(y/n) | entity_type | candidate | context | table | column | row_id | confidence | detectors |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| `y` | PERSON | `Bello` | `…paciente Bello refiere dolor…` | tbl_notes | note_full | 87 | 0.6 | context |
+| `n` | PERSON | `Cruz` | `…la Cruz Roja atendió…` | tbl_notes | note_full | 42 | 0.55 | gazetteer |
+| *(blank)* | PERSON | `Sur` | `…zona Sur del hospital…` | tbl_notes | note_full | 91 | 0.5 | gazetteer |
+
+- **`y`** — yes, this **is** a real person's name (it should be redacted).
+- **`n`** — no, it's a **false positive** — not a name (e.g. "Cruz" in "Cruz Roja").
+- **blank** — undecided; that row is skipped and simply stays for a later pass.
+
+That is the whole reviewer job: read the `context`, type `y` or `n`, save the
+file. (Lenient spellings work too: `yes` / `1` / `true` = yes; `no` / `0` /
+`false` / `fp` = no.)
+
+### The four steps
 
 ```bash
-# 1. Run with a learn directory. Uncertain detections go to review.csv.
-deidkit run data/ deid_out --learn-dir learned/
+# 1. Run. Uncertain detections are written to deid_out-PRIVATE/review.csv.
+deidkit run <input> deid_out
 
-# 2. A human opens deid_out-PRIVATE/review.csv and sets the first column:
-#       y  = real PII      (learn it as a name  → higher recall next time)
-#       n  = false positive (learn it as a stopword → higher precision next time)
+# 2. A reviewer opens deid_out-PRIVATE/review.csv and marks each row y / n (above).
 
-# 3. Fold those decisions into the learned dictionaries.
+# 3. Fold those decisions into a "learn" directory.
 deidkit learn --review deid_out-PRIVATE/review.csv --learn-dir learned/
 
-# 4. Re-run. Confirmed names are now caught; rejected ones are suppressed.
-deidkit run data/ deid_out --learn-dir learned/
+# 4. Re-run pointing at what it learned: confirmed names are now caught
+#    automatically, and rejected candidates are suppressed.
+deidkit run <input> deid_out --learn-dir learned/
 ```
 
-`learned/` accumulates `learned_names.txt` and `learned_stoplist.txt`. Point every
-future run at the same `--learn-dir` and it auto-loads them.
+Step 3 appends to two plain-text files that `learned/` **accumulates** across
+cycles (so it keeps improving):
+
+- `learned_names.txt` — words from every `y` → added to the name list (**higher recall**),
+- `learned_stoplist.txt` — words from every `n` → added to the stoplist (**higher precision**).
 
-*In code:* pass the learned files onto the policy —
+Both are one token per line and human-editable. *In code*, load them with
 `Policy(extra_name_files=["learned/learned_names.txt"],
 extra_stoplist_files=["learned/learned_stoplist.txt"])`.
 
+**Good to know:**
+- `review.csv` is produced on **every** run — `--learn-dir` is only needed to
+  *load* what was learned (steps 3–4), not to create the queue.
+- Learning is **token-level**: marking "María Gómez" as `y` teaches both `maría`
+  and `gómez` (lowercased, accent-folded), so they are caught elsewhere too.
+- The loop tunes the **borderline** cases. A name the detector missed entirely
+  will not appear in `review.csv` — for those, use the known-PHI columns below.
+
 ### Two stronger levers (deterministic, near-100% recall)
 
 - **Known-PHI columns** — if you already know a row's real name or ID (from a
@@ -397,27 +431,27 @@ dk.Policy(mode="balanced", spacy_model="es_core_news_lg",
 
 ## 9. Reading from a database (Fabric / Azure SQL)
 
-### How we get access to your system
+### How the connection works
 
-We do **not** log into your Azure portal or touch your production database. The
-flow is the standard Fabric mirror share:
+deidkit talks only to the Fabric **SQL analytics endpoint** — it never logs into
+the Azure portal or the production database. The setup is the standard Fabric
+mirror:
 
-1. **You (the data owner)** mirror the Azure SQL Database into a Microsoft Fabric
-   workspace and **share the mirrored database with us as read-only** — grant
-   *Read* + *Read all SQL analytics endpoint data*, nothing else. (This is the
-   setup in the onboarding document; no write or admin access is ever needed.)
-2. **You send us four connection details** (listed in §10) — the SQL analytics
-   endpoint, the database name, the authentication method, and the workspace URL.
-3. **We connect to that read-only SQL endpoint** from our side using those
-   details, read the six tables, de-identify them, and write out the clean copy.
-   deidkit **never writes back** to your database — it only reads.
+1. The Azure SQL Database is mirrored into a Microsoft Fabric workspace, and the
+   mirrored database is shared **read-only** — *Read* + *Read all SQL analytics
+   endpoint data*, nothing else. No write or admin access is ever required.
+2. deidkit is pointed at that endpoint with a connection string built from four
+   values (see section 10): the SQL analytics endpoint, the database name, the
+   authentication method, and the workspace URL.
+3. It connects to the read-only endpoint, reads the six tables, de-identifies
+   them, and writes out the clean copy. It **never writes back** — it only reads.
 
-So the only thing you provide is read-only access plus those four details; the
-de-identification runs entirely on our side against the shared endpoint.
+All that's needed is read-only access to the shared endpoint plus those values.
 
 ### Then it runs — two patterns
 
-The database drivers ship with deidkit (§1). Pick based on where you want the output.
+The database drivers ship with deidkit (section 1). Pick based on where you want
+the output.
 
 ### Pattern 1 — Database → files (recommended, simplest)
 
@@ -490,9 +524,9 @@ dk.deidentify_database(
 
 ## 10. Database connection parameters — exactly what to fill in
 
-### What you (the data owner) send us
+### The four connection values
 
-After sharing the mirrored database read-only, send us these **four items** (they
+Once the mirrored database is shared read-only, gather these **four values** (they
 are exactly what the Fabric "share" step produces):
 
 | # | Item | Example | Used for |
@@ -502,10 +536,11 @@ are exactly what the Fabric "share" step produces):
 | 3 | **Authentication method** — Microsoft Entra ID *or* SQL Login | `Microsoft Entra ID` | connecting |
 | 4 | **Fabric Workspace URL** | `https://app.fabric.microsoft.com/groups/...` | locating & confirming the shared database |
 
-Items 1–3 go into the connection string; item 4 is how we find and verify the
-shared item in Fabric (it is not part of the connection string).
+Items 1–3 go into the connection string; item 4 (the workspace URL) only locates
+and confirms the shared database in Fabric — it is not part of the connection
+string.
 
-### How we assemble the connection string
+### The connection string
 
 Connections are **SQLAlchemy URLs**. For the Fabric source, the shape is:
 
@@ -515,10 +550,10 @@ mssql+pyodbc://@<HOST>/<DATABASE>?driver=<DRIVER>&authentication=<AUTH>
 
 | Placeholder | Filled from | Example |
 | --- | --- | --- |
-| `<HOST>` | item 1 — the SQL analytics endpoint you send us | `abc123....datawarehouse.fabric.microsoft.com` |
-| `<DATABASE>` | item 2 — the database name you send us | `ProductionMirror` |
-| `<AUTH>` | item 3 — your authentication method | `ActiveDirectoryInteractive` |
-| `<DRIVER>` | **installed on our machine** (§1), not sent by you | `ODBC+Driver+18+for+SQL+Server` |
+| `<HOST>` | item 1 — the SQL analytics endpoint | `abc123....datawarehouse.fabric.microsoft.com` |
+| `<DATABASE>` | item 2 — the database name | `ProductionMirror` |
+| `<AUTH>` | item 3 — the authentication method | `ActiveDirectoryInteractive` |
+| `<DRIVER>` | installed locally (section 1), not a Fabric value | `ODBC+Driver+18+for+SQL+Server` |
 
 > Note the `@` before `<HOST>`: `...pyodbc://@abc123...`. With Entra ID auth there
 > is no username/password in the URL — the `@` with an empty user is intentional.
@@ -724,42 +759,13 @@ print(deid.summary())
 | Symptom | Fix |
 | --- | --- |
 | `ModuleNotFoundError: pyodbc` / SQLAlchemy | Reinstall: `pip install --force-reinstall deidkit` (the DB drivers ship with it) |
-| `Data source name not found` / driver error | Install **ODBC Driver 18 for SQL Server** and make the `driver=` value match exactly (§10) |
+| `Data source name not found` / driver error | Install **ODBC Driver 18 for SQL Server** and make the `driver=` value match exactly (section 10) |
 | Browser never opens for `ActiveDirectoryInteractive` | Run from an interactive session; for servers use `ActiveDirectoryServicePrincipal` |
 | `cannot reach the output database` (run-db) | Create the empty output DB first and pass a reachable `--out-db` URL |
 | A column was changed that shouldn’t be | Add it to `--ignore`, or add a `FieldRule(..., "passthrough")`; confirm with `deidkit plan` |
-| A free-text name slipped through | Try `--mode balanced`, enable `--spacy-model`, or add the name via the review→learn loop (§7) |
+| A free-text name slipped through | Try `--mode balanced`, enable `--spacy-model`, or add the name via the review→learn loop (section 7) |
 | Non-UTF-8 CSV export | Handled automatically (deidkit falls back to latin-1/cp1252) |
-| Different output each run | You’re not pinning the secret — pass the same `--secret-file` every time (§2) |
-
----
-
-## Publishing to PyPI
-
-deidkit is **not on PyPI yet**, so `pip install deidkit` will only work once you
-publish it under your own PyPI account. The package already builds cleanly into a
-wheel (verified). To publish:
-
-```bash
-# 0. one-time: make a PyPI account at https://pypi.org and create an API token.
-#    Also edit the [project.urls] in pyproject.toml (they still say "your-org").
-
-pip install build twine          # the standard build + upload tools
-
-python -m build                  # produces dist/deidkit-0.1.0-py3-none-any.whl (+ .tar.gz)
-python -m twine check dist/*     # validates the package metadata
-
-# optional dry run on the test index first:
-python -m twine upload --repository testpypi dist/*
-
-# the real publish (asks for your PyPI token):
-python -m twine upload dist/*
-```
-
-After that first upload, `pip install deidkit` works for everyone, and the name
-`deidkit` is yours (it is currently unclaimed on PyPI). Bump `version` in
-`pyproject.toml` for each subsequent release — PyPI refuses to overwrite an
-existing version.
+| Different output each run | You’re not pinning the secret — pass the same `--secret-file` every time (section 2) |
 
 ---