pii-detector-ca 0.1.0__tar.gz

pii_detector_ca-0.1.0/.gitignore

@@ -0,0 +1,33 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+dist/
+build/
+.venv/
+.pytest_cache/
+.ruff_cache/
+
+# Benchmark artifacts (generated; gold split is kept in-repo)
+benchmark/data/generated/
+
+# Real clinical text must never be committed (licensing/PHI)
+benchmark/data/real/
+
+# Raw external datasets (e.g. StatCan ODA CSVs): too large to commit; a small
+# curated sample is committed instead (see benchmark/curate_oda.py).
+benchmark/data/external/
+
+# OS / editors
+.DS_Store
+.idea/
+.vscode/
+
+# Secrets
+.env
+*.key
+*credentials*.json
+
+# Tool caches
+.hypothesis/
+.mypy_cache/

pii_detector_ca-0.1.0/CHANGELOG.md

@@ -0,0 +1,35 @@
+# Changelog
+
+All notable changes to this project are documented here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project uses [Semantic Versioning](https://semver.org/).
+
+## [Unreleased]
+
+## [0.1.0] - unreleased
+
+First public release, Canada only. (Pre-release history: the package was developed as `presidio-recognizers-ca-in` with experimental Indian recognizers; those were removed before 0.1.0 and will return as a separate `pii-detector-in` package once they have their own benchmark.)
+
+### Added
+
+- Canadian recognizers: `CA_SIN` (Luhn), `CA_OHIP` (Luhn + version code), `CA_RAMQ` (structural), `CA_BC_PHN` (mod-11), `CA_AB_ULI` (context gated), `CA_HEALTH_CARD` (generic, context gated), `CA_POSTAL_CODE`, `CA_STREET_ADDRESS`.
+- `presidio-analyzer` is an optional extra (`pip install pii-detector-ca[presidio]`); the base install has no dependencies.
+- Framework-neutral core (`pii_detector_ca.core`): stdlib-only validators, entity specs, and a `scan()` API that runs without Presidio or spaCy. Presidio recognizers are thin adapters over the same specs.
+- `add_recognizers(registry, exclude=[...])` / `get_recognizers(exclude=[...])` to skip entities that Presidio's own (default-disabled) `CA_SIN` / `CA_POSTAL_CODE` recognizers already cover.
+- Synthetic clinical-note benchmark: a seeded, deterministic generator with a determinism manifest, a false-positive negatives corpus, an adversarial hard split, and a street-address holdout derived from the StatCan Open Database of Addresses.
+- Evaluation harness: leak-oriented and strict recall, bootstrap 95% CIs, three baselines (stock Presidio, Presidio with its CA built-ins enabled, Presidio + pack), and a CI threshold gate in a locked environment.
+- Provenance docs for every identifier (`docs/identifiers/`), a core contract doc, and a non-technical user manual.
+
+### Changed (pre-release)
+
+- `CA_AB_ULI` now requires Alberta-specific context ("alberta", "uli", "ahcip", "netcare"). A 9-digit number under generic "health card" wording is reported as `CA_HEALTH_CARD` only; the generic recognizer gained separated-form patterns (5-4, 3-3-3, 4-3-3) so those renderings are still covered. ULI typed-exact recall on the benchmark is 0 by design (values are labelled under generic headers); leak recall stays 1.0.
+- "card" is not a context token: it boosted credit-card chunks ("payment card on file") by +200 FPs/300 notes on the negatives corpus.
+- `evaluation/run_eval.py` reports a third system, "Presidio + CA builtins", and `results_en_core_web_lg.*` are regenerated for the Canada-only corpus.
+
+### Fixed (pre-release)
+
+- `CA_OHIP`: the separated pattern was compiled case-insensitively, so a lowercase English word following the number ("… 897 on file") was absorbed into the span as a bogus version code at score 1.0. Version codes are now matched case-sensitively, and the validator enforces the MOH HCV schema's non-zero first digit.
+- `CA_HEALTH_CARD` and the French context terms never fired through Presidio: its lemma context enhancer matches single tokens, and the context lists were multi-word phrases. Single-token context words ("health", "card", "maladie", "msp", "hsn", …) added.
+- Core-scanner evaluation counted correct `CA_STREET_ADDRESS` / `CA_POSTAL_CODE` predictions as false positives because `ADDRESS` gold was filtered out; predictions are now type-mapped and `ADDRESS` kept.
+- Benchmark realism: dates of birth were drawn from the encounter-date range (including future dates); RAMQ letter blocks did not match the patient named in the note; RAMQ birth years stopped at 1999.
+
+[Unreleased]: https://github.com/thisiskarthikeyan/pii-detector-ca/compare/v0.1.0...HEAD
+[0.1.0]: https://github.com/thisiskarthikeyan/pii-detector-ca/releases/tag/v0.1.0

pii_detector_ca-0.1.0/CITATION.cff

@@ -0,0 +1,28 @@
+cff-version: 1.2.0
+message: "If you use this software, please cite it using the metadata below."
+title: "pii-detector-ca: Checksum-validated recognizers and a synthetic benchmark for Canadian clinical de-identification"
+type: software
+authors:
+  - family-names: Murugaiah
+    given-names: Karthikeyan
+    # TODO: orcid before minting the Zenodo DOI.
+    email: blue1agave@gmail.com
+    alias: thisiskarthikeyan
+version: 0.1.0
+date-released: 2026-10-27
+license: MIT
+repository-code: "https://github.com/thisiskarthikeyan/pii-detector-ca"
+url: "https://github.com/thisiskarthikeyan/pii-detector-ca"
+abstract: >-
+  Checksum-validated detectors for Canadian health and government identifiers
+  (SIN, Ontario OHIP, Quebec RAMQ, BC PHN, Alberta ULI, postal codes, street
+  addresses), usable standalone or as Microsoft Presidio recognizers, with a
+  reproducible synthetic clinical-text benchmark, adversarial and
+  false-positive splits, and an evaluation harness.
+keywords:
+  - de-identification
+  - PII
+  - PHI
+  - clinical NLP
+  - Presidio
+  - Canada

pii_detector_ca-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,53 @@
+# Contributing
+
+Thanks for helping close the locale gap in clinical de-identification. This project's value is that every detection decision is **auditable and reproducible**, so contributions are held to that bar: sourced rules, deterministic tests, and no regressions on the evaluation gate.
+
+## Ground rules
+
+- **Never commit real patient, customer, or personal data.** That includes in tests, issues, PRs, and screenshots. Use synthetic values that pass the relevant checksum (the generators in `benchmark/pii_generators.py` produce them). `benchmark/data/real/` is gitignored for this reason.
+- **Canada only.** Indian identifiers are planned as a separate package (`pii-detector-in`); PRs adding other jurisdictions to this package will be redirected.
+- Missed-identifier (leak) findings with real-world impact go through [SECURITY.md](SECURITY.md), not public issues.
+
+## Development setup
+
+Requires Python ≥ 3.10 and [uv](https://docs.astral.sh/uv/).
+
+```bash
+uv sync --extra dev   # includes the presidio extra
+uv pip install -r evaluation/requirements-eval.txt   # pinned spaCy model, for evaluation
+```
+
+Fast checks (run all three before opening a PR; CI runs them on 3.10 and 3.12):
+
+```bash
+uv run ruff check .
+uv run mypy
+uv run pytest -q
+```
+
+The full evaluation gate (CI's `eval` job) is the command sequence in the README's *Benchmark* section, ending with `python evaluation/check_thresholds.py`.
+
+## Adding or changing an identifier
+
+Work through these in order. A PR that skips a step will be asked to add it.
+
+1. **Source the rule.** Find the authoritative format and checksum definition (government or issuing-authority publication). If no official source exists, say so explicitly.
+2. **Validator.** Add a stdlib-only function to `src/pii_detector_ca/core/validators.py`. Checksumless identifiers must never *boost* a score on plausibility alone: return `False` to reject or `None` to defer to context, never `True` (see `tests/test_recognizers.py` for the regression tests that enforce this).
+3. **Entity spec.** Add an `EntitySpec` in `core/specs.py` (patterns, base scores, context words, validator binding, provenance doc path). The core `scan()` API picks it up automatically. Context lists must include **single tokens**: Presidio's lemma context enhancer never matches a multi-word phrase.
+4. **Presidio adapter.** Add a thin `SpecRecognizer` subclass in `canada/` and register it in `get_canada_recognizers()`.
+5. **Tests.** Positive cases for every surface variant (compact / spaced / dashed), negatives for near-misses and checksum failures, and a property-based test where a checksum exists (`tests/test_properties.py`).
+6. **Generator + template slot.** Add a `gen_*` function in `benchmark/pii_generators.py`, wire it into the slot filler in `benchmark/generate_corpus.py`, and place a `("SLOT", "<ENTITY>")` segment in the relevant templates in `benchmark/templates.py`. Generators must take the passed `random.Random` instance, never global `random`, or determinism breaks.
+7. **Regenerate and re-pin.** Regenerate the corpora, then run `python benchmark/check_determinism.py --update` and commit the updated `benchmark/data/checksums.json` together with the generator change.
+8. **Threshold.** Add a floor for the new entity in `evaluation/thresholds.json`. Checksum-backed identifiers are expected to hold 1.0 recall.
+9. **Provenance doc.** Add `docs/identifiers/<entity>.md` (source, implemented rule, known gaps) and a row in `docs/identifiers/README.md`.
+10. **Docs.** Update the entity table in `README.md` and `docs/user_manual.md`, and add a `CHANGELOG.md` entry.
+
+## Changing evaluation results
+
+Committed files under `evaluation/results/` back the paper's numbers (manuscript on the `paper` branch). If your change moves them, regenerate them in the locked environment (`uv sync --locked`), commit them in the same PR, and explain the delta in the PR description. Lowering a threshold needs a stated reason. "It failed" is not one.
+
+## Pull requests
+
+- Keep PRs focused; one identifier or one fix per PR.
+- Describe *why*, and include before/after numbers when detection behavior changes.
+- By contributing you agree your work is released under the project's MIT license.

pii_detector_ca-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Karthik
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pii_detector_ca-0.1.0/PKG-INFO

@@ -0,0 +1,172 @@
+Metadata-Version: 2.5
+Name: pii-detector-ca
+Version: 0.1.0
+Summary: Checksum-validated detectors for Canadian health and government identifiers (SIN, OHIP, RAMQ, BC PHN, Alberta ULI, postal codes, street addresses): standalone or as Microsoft Presidio recognizers, with a synthetic clinical-text benchmark
+Project-URL: Homepage, https://github.com/thisiskarthikeyan/pii-detector-ca
+Project-URL: Repository, https://github.com/thisiskarthikeyan/pii-detector-ca
+Project-URL: Issues, https://github.com/thisiskarthikeyan/pii-detector-ca/issues
+Project-URL: Changelog, https://github.com/thisiskarthikeyan/pii-detector-ca/blob/main/CHANGELOG.md
+Author-email: Karthikeyan Murugaiah <blue1agave@gmail.com>
+Maintainer-email: Karthikeyan Murugaiah <blue1agave@gmail.com>
+License: MIT License
+        
+        Copyright (c) 2026 Karthik
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+License-File: LICENSE
+Keywords: canada,clinical-nlp,de-identification,health-card,phi,pii,presidio,sin
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Healthcare Industry
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: >=3.10
+Provides-Extra: benchmark
+Requires-Dist: pandas>=2.0; extra == 'benchmark'
+Provides-Extra: dev
+Requires-Dist: hypothesis>=6.100; extra == 'dev'
+Requires-Dist: mypy>=1.10; extra == 'dev'
+Requires-Dist: pandas>=2.0; extra == 'dev'
+Requires-Dist: presidio-analyzer>=2.2; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff>=0.4; extra == 'dev'
+Provides-Extra: presidio
+Requires-Dist: presidio-analyzer>=2.2; extra == 'presidio'
+Description-Content-Type: text/markdown
+
+# pii-detector-ca
+
+Checksum-validated detectors for **Canadian** health and government identifiers, usable standalone or as [Microsoft Presidio](https://microsoft.github.io/presidio/) recognizers, plus a synthetic clinical-text benchmark and evaluation harness.
+
+Most clinical de-identification tooling is built and tested on US identifier formats. Presidio's default registry has no provincial health-card recognizers: an Ontario health number, a Quebec RAMQ number or a BC PHN passes through a stock de-identification pipeline silently. This package closes that gap with deterministic, checksum-backed recognizers that plug into any Presidio `AnalyzerEngine`, or run standalone with no Presidio/spaCy dependency at runtime.
+
+> **Scope.** Canada only. An Indian identifier pack is planned as a separate package (`pii-detector-in`); it is not part of this one.
+
+## Supported entities
+
+| Entity | Identifier | Validation | Spec |
+|---|---|---|---|
+| `CA_SIN` | Social Insurance Number | Luhn checksum; leading digit not 0/8 | [doc](docs/identifiers/ca_sin.md) |
+| `CA_OHIP` | Ontario health number (10 digits + version code) | Luhn (mod-10); leading digit 1–9 | [doc](docs/identifiers/ca_ohip.md) |
+| `CA_RAMQ` | Quebec health insurance number (NAM) | Structural (name/DOB encoding) | [doc](docs/identifiers/ca_ramq.md) |
+| `CA_BC_PHN` | British Columbia Personal Health Number | Mod-11 weighted checksum | [doc](docs/identifiers/ca_bc_phn.md) |
+| `CA_AB_ULI` | Alberta Unique Lifetime Identifier | Pattern + context gated | [doc](docs/identifiers/ca_ab_uli.md) |
+| `CA_HEALTH_CARD` | Other provinces/territories (generic) | Context gated only | [doc](docs/identifiers/ca_health_card_generic.md) |
+| `CA_POSTAL_CODE` | Canadian postal code | Structural (letter-class rules) | [doc](docs/identifiers/ca_postal_code.md) |
+| `CA_STREET_ADDRESS` | Full civic/mailing address | Structural (postal + province anchor) | [doc](docs/identifiers/ca_street_address.md) |
+
+**Relationship to Presidio's own Canadian recognizers.** Presidio ≥ 2.2.363 ships a `CA_SIN` recognizer and ≥ 2.2.365 a `CA_POSTAL_CODE` recognizer, both **disabled by default** in `default_recognizers.yaml`. The six health-card and address entities above have no upstream equivalent. If you enable Presidio's built-ins, skip ours to avoid duplicate detections: `add_recognizers(registry, exclude=["CA_SIN", "CA_POSTAL_CODE"])`. The evaluation reports a "Presidio + CA builtins" baseline so the health-card contribution is measured against the strongest no-pack configuration.
+
+## Install
+
+```bash
+pip install pii-detector-ca              # standalone scan(): no dependencies
+pip install "pii-detector-ca[presidio]"  # + Presidio recognizers
+```
+
+The Presidio integration also needs a spaCy model: `python -m spacy download en_core_web_sm` (see the snippet below for why `sm` is configured explicitly). The standalone `scan()` API needs neither.
+
+From source: `git clone https://github.com/thisiskarthikeyan/pii-detector-ca.git && cd pii-detector-ca && pip install -e ".[presidio]"`.
+
+## Usage
+
+### Standalone detection (no Presidio required at runtime)
+
+```python
+from pii_detector_ca import scan
+
+matches = scan("Patient health card 7577 752 616 AB, SIN 123-456-782.", threshold=0.35)
+for hit in matches:
+    print(hit.entity, hit.start, hit.end, hit.text, hit.score)
+```
+
+`scan()` is pattern + validator only — it applies **no context-word boosting**. Checksum/structure-backed entities (`CA_SIN`, `CA_OHIP`, `CA_RAMQ`, `CA_BC_PHN`, `CA_POSTAL_CODE`, `CA_STREET_ADDRESS`) work fully; the context-gated `CA_AB_ULI` and `CA_HEALTH_CARD` never reach the default threshold standalone and need the Presidio integration. Entity rules live in `pii_detector_ca.core` (validators, specs, scanner); the Presidio recognizers are thin adapters over the same specs. See [docs/architecture/core_contract.md](docs/architecture/core_contract.md).
+
+### Presidio integration
+
+```python
+from presidio_analyzer import AnalyzerEngine
+from presidio_analyzer.nlp_engine import NlpEngineProvider
+from pii_detector_ca import add_recognizers
+
+# Same NLP engine as the published evaluation. A bare AnalyzerEngine()
+# defaults to en_core_web_lg (~400 MB) and downloads it on first use.
+nlp_engine = NlpEngineProvider(nlp_configuration={
+    "nlp_engine_name": "spacy",
+    "models": [{"lang_code": "en", "model_name": "en_core_web_sm"}],
+}).create_engine()
+
+analyzer = AnalyzerEngine(nlp_engine=nlp_engine, supported_languages=["en"])
+add_recognizers(analyzer.registry)
+
+results = analyzer.analyze(
+    text="Patient health card 7577 752 616 AB, SIN 123-456-782.",
+    language="en",
+)
+for r in results:
+    print(r.entity_type, r.start, r.end, round(r.score, 2))
+```
+
+A non-technical walkthrough is in [docs/user_manual.md](docs/user_manual.md).
+
+## What this is not
+
+- **Not a compliance guarantee.** Using this package does not by itself satisfy PIPEDA, PHIPA, provincial health-information acts, or any other regulation. It is one detection layer; keep human review and your organization's privacy process in the loop.
+- **Not a full de-identifier.** It detects identifiers. Person names, dates, and free-text clues come from Presidio's underlying NLP engine (spaCy), which this package does not improve. Most remaining misses on the benchmark are non-Anglophone person names.
+- **Not validated on real clinical text.** All reported numbers come from a synthetic benchmark (below). Real notes carry OCR noise, dictation artifacts, and formats a template generator cannot anticipate.
+- **Checksum-valid ≠ real, and the collision floor is not small.** About 8% of random 9-digit strings pass the SIN check and about 10% of random 10-digit strings pass the OHIP check, with no context required. Compact phone numbers, MRNs and accession numbers will be flagged at that rate. Checksumless types (Alberta ULI, generic health cards) rely on nearby context words instead.
+- **French prose is out of scope.** French context words are implemented (and exercised in tests) but the benchmark is English-language clinical prose only.
+
+## Benchmark
+
+`benchmark/` contains a seeded, deterministic generator for synthetic clinical notes (six note types, from radiology reports to ED triage records) with span-labeled PII injection. It also has a false-positive negatives corpus, an adversarial hard split, and a street-address holdout built from the StatCan Open Database of Addresses. `evaluation/` compares stock Presidio, Presidio with its own Canadian built-ins enabled, and Presidio + this pack; results are committed under `evaluation/results/`.
+
+```bash
+pip install -e ".[dev]" && pip install -r evaluation/requirements-eval.txt
+python benchmark/generate_corpus.py --n 1000
+python benchmark/generate_negatives.py --n 300
+python benchmark/generate_hard.py --per-cell 10
+python benchmark/generate_address_eval.py --n 300
+python benchmark/check_determinism.py
+python evaluation/run_eval.py --all
+python evaluation/run_negatives.py
+python evaluation/run_hard.py
+python evaluation/run_address_eval.py
+python evaluation/check_thresholds.py
+```
+
+CI runs this exact sequence in a locked environment (`uv.lock` + pinned spaCy model) and fails on any regression below `evaluation/thresholds.json`.
+
+**Benchmark caveat:** the corpus is generated by the same project that wrote the recognizers, so in-distribution recall is partly circular. The robust findings are the stock-Presidio gap and the results on the negatives, hard, and address-holdout splits. The "gold" split is the first 100 notes of the same generator stream, not an independently annotated set. Synthetic data only; no real patient data is used or included.
+
+## Contributing, security, citation
+
+- [CONTRIBUTING.md](CONTRIBUTING.md): dev setup and the recipe for adding an identifier.
+- [SECURITY.md](SECURITY.md): report missed identifiers (leaks) privately, never with real data.
+- [CITATION.cff](CITATION.cff): how to cite this software.
+- [CHANGELOG.md](CHANGELOG.md).
+
+## License
+
+MIT. Benchmark data derived from Statistics Canada sources is used under the Open Government Licence – Canada; see the READMEs in `benchmark/data/oda/` and `benchmark/data/statcan_names/`.

pii_detector_ca-0.1.0/README.md

@@ -0,0 +1,115 @@
+# pii-detector-ca
+
+Checksum-validated detectors for **Canadian** health and government identifiers, usable standalone or as [Microsoft Presidio](https://microsoft.github.io/presidio/) recognizers, plus a synthetic clinical-text benchmark and evaluation harness.
+
+Most clinical de-identification tooling is built and tested on US identifier formats. Presidio's default registry has no provincial health-card recognizers: an Ontario health number, a Quebec RAMQ number or a BC PHN passes through a stock de-identification pipeline silently. This package closes that gap with deterministic, checksum-backed recognizers that plug into any Presidio `AnalyzerEngine`, or run standalone with no Presidio/spaCy dependency at runtime.
+
+> **Scope.** Canada only. An Indian identifier pack is planned as a separate package (`pii-detector-in`); it is not part of this one.
+
+## Supported entities
+
+| Entity | Identifier | Validation | Spec |
+|---|---|---|---|
+| `CA_SIN` | Social Insurance Number | Luhn checksum; leading digit not 0/8 | [doc](docs/identifiers/ca_sin.md) |
+| `CA_OHIP` | Ontario health number (10 digits + version code) | Luhn (mod-10); leading digit 1–9 | [doc](docs/identifiers/ca_ohip.md) |
+| `CA_RAMQ` | Quebec health insurance number (NAM) | Structural (name/DOB encoding) | [doc](docs/identifiers/ca_ramq.md) |
+| `CA_BC_PHN` | British Columbia Personal Health Number | Mod-11 weighted checksum | [doc](docs/identifiers/ca_bc_phn.md) |
+| `CA_AB_ULI` | Alberta Unique Lifetime Identifier | Pattern + context gated | [doc](docs/identifiers/ca_ab_uli.md) |
+| `CA_HEALTH_CARD` | Other provinces/territories (generic) | Context gated only | [doc](docs/identifiers/ca_health_card_generic.md) |
+| `CA_POSTAL_CODE` | Canadian postal code | Structural (letter-class rules) | [doc](docs/identifiers/ca_postal_code.md) |
+| `CA_STREET_ADDRESS` | Full civic/mailing address | Structural (postal + province anchor) | [doc](docs/identifiers/ca_street_address.md) |
+
+**Relationship to Presidio's own Canadian recognizers.** Presidio ≥ 2.2.363 ships a `CA_SIN` recognizer and ≥ 2.2.365 a `CA_POSTAL_CODE` recognizer, both **disabled by default** in `default_recognizers.yaml`. The six health-card and address entities above have no upstream equivalent. If you enable Presidio's built-ins, skip ours to avoid duplicate detections: `add_recognizers(registry, exclude=["CA_SIN", "CA_POSTAL_CODE"])`. The evaluation reports a "Presidio + CA builtins" baseline so the health-card contribution is measured against the strongest no-pack configuration.
+
+## Install
+
+```bash
+pip install pii-detector-ca              # standalone scan(): no dependencies
+pip install "pii-detector-ca[presidio]"  # + Presidio recognizers
+```
+
+The Presidio integration also needs a spaCy model: `python -m spacy download en_core_web_sm` (see the snippet below for why `sm` is configured explicitly). The standalone `scan()` API needs neither.
+
+From source: `git clone https://github.com/thisiskarthikeyan/pii-detector-ca.git && cd pii-detector-ca && pip install -e ".[presidio]"`.
+
+## Usage
+
+### Standalone detection (no Presidio required at runtime)
+
+```python
+from pii_detector_ca import scan
+
+matches = scan("Patient health card 7577 752 616 AB, SIN 123-456-782.", threshold=0.35)
+for hit in matches:
+    print(hit.entity, hit.start, hit.end, hit.text, hit.score)
+```
+
+`scan()` is pattern + validator only — it applies **no context-word boosting**. Checksum/structure-backed entities (`CA_SIN`, `CA_OHIP`, `CA_RAMQ`, `CA_BC_PHN`, `CA_POSTAL_CODE`, `CA_STREET_ADDRESS`) work fully; the context-gated `CA_AB_ULI` and `CA_HEALTH_CARD` never reach the default threshold standalone and need the Presidio integration. Entity rules live in `pii_detector_ca.core` (validators, specs, scanner); the Presidio recognizers are thin adapters over the same specs. See [docs/architecture/core_contract.md](docs/architecture/core_contract.md).
+
+### Presidio integration
+
+```python
+from presidio_analyzer import AnalyzerEngine
+from presidio_analyzer.nlp_engine import NlpEngineProvider
+from pii_detector_ca import add_recognizers
+
+# Same NLP engine as the published evaluation. A bare AnalyzerEngine()
+# defaults to en_core_web_lg (~400 MB) and downloads it on first use.
+nlp_engine = NlpEngineProvider(nlp_configuration={
+    "nlp_engine_name": "spacy",
+    "models": [{"lang_code": "en", "model_name": "en_core_web_sm"}],
+}).create_engine()
+
+analyzer = AnalyzerEngine(nlp_engine=nlp_engine, supported_languages=["en"])
+add_recognizers(analyzer.registry)
+
+results = analyzer.analyze(
+    text="Patient health card 7577 752 616 AB, SIN 123-456-782.",
+    language="en",
+)
+for r in results:
+    print(r.entity_type, r.start, r.end, round(r.score, 2))
+```
+
+A non-technical walkthrough is in [docs/user_manual.md](docs/user_manual.md).
+
+## What this is not
+
+- **Not a compliance guarantee.** Using this package does not by itself satisfy PIPEDA, PHIPA, provincial health-information acts, or any other regulation. It is one detection layer; keep human review and your organization's privacy process in the loop.
+- **Not a full de-identifier.** It detects identifiers. Person names, dates, and free-text clues come from Presidio's underlying NLP engine (spaCy), which this package does not improve. Most remaining misses on the benchmark are non-Anglophone person names.
+- **Not validated on real clinical text.** All reported numbers come from a synthetic benchmark (below). Real notes carry OCR noise, dictation artifacts, and formats a template generator cannot anticipate.
+- **Checksum-valid ≠ real, and the collision floor is not small.** About 8% of random 9-digit strings pass the SIN check and about 10% of random 10-digit strings pass the OHIP check, with no context required. Compact phone numbers, MRNs and accession numbers will be flagged at that rate. Checksumless types (Alberta ULI, generic health cards) rely on nearby context words instead.
+- **French prose is out of scope.** French context words are implemented (and exercised in tests) but the benchmark is English-language clinical prose only.
+
+## Benchmark
+
+`benchmark/` contains a seeded, deterministic generator for synthetic clinical notes (six note types, from radiology reports to ED triage records) with span-labeled PII injection. It also has a false-positive negatives corpus, an adversarial hard split, and a street-address holdout built from the StatCan Open Database of Addresses. `evaluation/` compares stock Presidio, Presidio with its own Canadian built-ins enabled, and Presidio + this pack; results are committed under `evaluation/results/`.
+
+```bash
+pip install -e ".[dev]" && pip install -r evaluation/requirements-eval.txt
+python benchmark/generate_corpus.py --n 1000
+python benchmark/generate_negatives.py --n 300
+python benchmark/generate_hard.py --per-cell 10
+python benchmark/generate_address_eval.py --n 300
+python benchmark/check_determinism.py
+python evaluation/run_eval.py --all
+python evaluation/run_negatives.py
+python evaluation/run_hard.py
+python evaluation/run_address_eval.py
+python evaluation/check_thresholds.py
+```
+
+CI runs this exact sequence in a locked environment (`uv.lock` + pinned spaCy model) and fails on any regression below `evaluation/thresholds.json`.
+
+**Benchmark caveat:** the corpus is generated by the same project that wrote the recognizers, so in-distribution recall is partly circular. The robust findings are the stock-Presidio gap and the results on the negatives, hard, and address-holdout splits. The "gold" split is the first 100 notes of the same generator stream, not an independently annotated set. Synthetic data only; no real patient data is used or included.
+
+## Contributing, security, citation
+
+- [CONTRIBUTING.md](CONTRIBUTING.md): dev setup and the recipe for adding an identifier.
+- [SECURITY.md](SECURITY.md): report missed identifiers (leaks) privately, never with real data.
+- [CITATION.cff](CITATION.cff): how to cite this software.
+- [CHANGELOG.md](CHANGELOG.md).
+
+## License
+
+MIT. Benchmark data derived from Statistics Canada sources is used under the Open Government Licence – Canada; see the READMEs in `benchmark/data/oda/` and `benchmark/data/statcan_names/`.

pii_detector_ca-0.1.0/SECURITY.md

@@ -0,0 +1,29 @@
+# Security policy
+
+This package sits on a privacy boundary: a missed identifier is a data leak. Please report such findings privately so they can be fixed before they are public.
+
+## What to report privately
+
+- **Leaks (false negatives):** a real-world identifier format that the recognizers fail to detect, especially a *class* of misses (a formatting variant, a province's card layout, a separator style).
+- **Validation bypasses:** inputs that defeat a checksum or context gate in a way that hides identifiers.
+- **Unsafe handling:** any code path that logs, persists, or transmits input text.
+
+Ordinary false positives, documentation errors, and feature requests are fine as public GitHub issues.
+
+## How to report
+
+Use GitHub's private vulnerability reporting: open the repository's **Security** tab, then click **Report a vulnerability**.
+
+**Never include real personal or patient data in a report.** Reproduce the issue with a synthetic value of the same shape (`benchmark/pii_generators.py` generates checksum-valid ones), or describe the format with digits replaced by `X`.
+
+You can expect an acknowledgement within 7 days. Confirmed leaks get a fix, a regression test, and a CHANGELOG entry crediting you (unless you prefer otherwise).
+
+## Supported versions
+
+| Version | Supported |
+|---|---|
+| 0.1.x | Yes |
+
+## Scope reminder
+
+This project is a detection aid, not a compliance guarantee. See *What this is not* in the [README](README.md).

pii_detector_ca-0.1.0/benchmark/check_determinism.py

@@ -0,0 +1,91 @@
+"""Verify the benchmark corpus regenerates bit-identically from the fixed seed.
+
+The reproducibility claim of this project depends on the generator being
+deterministic across environments. This script hashes the corpus JSONL files
+and compares them against the committed manifest, so any drift (stdlib RNG
+behavior change, accidental generator edit without regeneration, template
+change) fails CI immediately.
+
+Usage:
+    python benchmark/check_determinism.py            # verify against manifest
+    python benchmark/check_determinism.py --update   # rewrite the manifest
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import sys
+from pathlib import Path
+
+DATA_DIR = Path(__file__).parent / "data"
+MANIFEST = DATA_DIR / "checksums.json"
+
+CORPUS_FILES = [
+    "gold/ca_gold.jsonl",
+    "generated/ca.jsonl",
+    "negatives/ca_negatives.jsonl",
+    "hard/ca_hard.jsonl",
+    "addresses/ca_address_eval.jsonl",
+]
+
+
+def sha256_of(path: Path) -> str:
+    h = hashlib.sha256()
+    with open(path, "rb") as f:
+        for chunk in iter(lambda: f.read(1 << 20), b""):
+            h.update(chunk)
+    return h.hexdigest()
+
+
+def compute_checksums() -> dict[str, str]:
+    checksums = {}
+    for rel in CORPUS_FILES:
+        path = DATA_DIR / rel
+        if not path.exists():
+            sys.exit(f"ERROR: {path} missing — run the benchmark generators first.")
+        checksums[rel] = sha256_of(path)
+    return checksums
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--update", action="store_true", help="rewrite the manifest from current files"
+    )
+    args = parser.parse_args()
+
+    current = compute_checksums()
+
+    if args.update:
+        MANIFEST.write_text(json.dumps(current, indent=2) + "\n", encoding="utf-8")
+        print(f"Wrote {MANIFEST}")
+        for rel, digest in current.items():
+            print(f"  {digest}  {rel}")
+        return
+
+    if not MANIFEST.exists():
+        sys.exit(f"ERROR: manifest {MANIFEST} missing — run with --update to create it.")
+
+    expected = json.loads(MANIFEST.read_text(encoding="utf-8"))
+    failures = []
+    for rel in CORPUS_FILES:
+        exp = expected.get(rel)
+        got = current[rel]
+        status = "OK" if exp == got else "MISMATCH"
+        if exp != got:
+            failures.append(rel)
+        print(f"{status:9s} {rel}\n          expected {exp}\n          got      {got}")
+
+    if failures:
+        sys.exit(
+            f"\nDeterminism check FAILED for: {', '.join(failures)}.\n"
+            "If the generator changed intentionally, regenerate the corpus, re-run the "
+            "evaluation, and update the manifest with --update."
+        )
+    print("\nDeterminism check passed: corpus regenerates bit-identically.")
+
+
+if __name__ == "__main__":
+    main()