phiproxy 0.1.0a2__tar.gz → 0.1.0a3__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (49) hide show
  1. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/.github/workflows/ci.yml +2 -0
  2. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/.gitignore +1 -0
  3. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/PKG-INFO +51 -25
  4. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/README.md +46 -24
  5. phiproxy-0.1.0a3/benchmark/README.md +62 -0
  6. phiproxy-0.1.0a3/benchmark/blind/blind_v1.source.jsonl +420 -0
  7. phiproxy-0.1.0a3/benchmark/blind/convert.py +56 -0
  8. phiproxy-0.1.0a3/benchmark/data/patient_comms_blind_v1.jsonl +420 -0
  9. phiproxy-0.1.0a3/benchmark/results_blind_v1.json +182 -0
  10. phiproxy-0.1.0a3/benchmark/results_blind_v1_frozen.json +182 -0
  11. phiproxy-0.1.0a3/benchmark/results_v0.json +130 -0
  12. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/benchmark/score.py +38 -9
  13. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/pyproject.toml +1 -0
  14. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/src/phiproxy/__init__.py +1 -1
  15. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/src/phiproxy/detectors/base.py +1 -1
  16. phiproxy-0.1.0a3/src/phiproxy/detectors/gliner.py +156 -0
  17. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/src/phiproxy/detectors/regex.py +97 -10
  18. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/src/phiproxy/entities.py +1 -0
  19. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/src/phiproxy/masker.py +15 -1
  20. phiproxy-0.1.0a3/tests/test_gliner.py +75 -0
  21. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/tests/test_regex.py +57 -0
  22. phiproxy-0.1.0a2/benchmark/README.md +0 -38
  23. phiproxy-0.1.0a2/benchmark/results_v0.json +0 -56
  24. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/.github/workflows/release.yml +0 -0
  25. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/CONTRIBUTING.md +0 -0
  26. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/LICENSE +0 -0
  27. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/NOTICE +0 -0
  28. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/SECURITY.md +0 -0
  29. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/benchmark/data/patient_comms_v0.jsonl +0 -0
  30. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/benchmark/generate.py +0 -0
  31. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/deploy/.env.example +0 -0
  32. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/deploy/Dockerfile +0 -0
  33. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/deploy/docker-compose.yml +0 -0
  34. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/deploy/litellm.config.yaml +0 -0
  35. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/examples/e2e/config.yaml +0 -0
  36. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/examples/e2e/fake_llm.py +0 -0
  37. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/examples/e2e/run_e2e.py +0 -0
  38. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/src/phiproxy/dates.py +0 -0
  39. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/src/phiproxy/detectors/__init__.py +0 -0
  40. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/src/phiproxy/detectors/known.py +0 -0
  41. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/src/phiproxy/detectors/presidio.py +0 -0
  42. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/src/phiproxy/integrations/__init__.py +0 -0
  43. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/src/phiproxy/integrations/litellm.py +0 -0
  44. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/src/phiproxy/vault.py +0 -0
  45. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/tests/test_dates.py +0 -0
  46. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/tests/test_known.py +0 -0
  47. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/tests/test_litellm.py +0 -0
  48. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/tests/test_masker.py +0 -0
  49. {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/tests/test_vault.py +0 -0
@@ -23,6 +23,8 @@ jobs:
23
23
  - run: pytest -q
24
24
  - name: Benchmark (regex+known must stay at or above 85% recall)
25
25
  run: python benchmark/score.py --config regex+known --min-recall 0.85
26
+ - name: Blind benchmark (regex+known must stay at or above 65% recall)
27
+ run: python benchmark/score.py --data benchmark/data/patient_comms_blind_v1.jsonl --config regex+known --min-recall 0.65
26
28
 
27
29
  e2e:
28
30
  runs-on: ubuntu-latest
@@ -8,3 +8,4 @@ build/
8
8
  deploy/.env
9
9
  received.jsonl
10
10
  .DS_Store
11
+ .venv-ml/
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.5
2
2
  Name: phiproxy
3
- Version: 0.1.0a2
3
+ Version: 0.1.0a3
4
4
  Summary: Your patients' stand-in on the way to any LLM, and back: reversible PHI masking proxy (LiteLLM guardrail + Python library).
5
5
  Project-URL: Homepage, https://decisiontree.tech/open-source/phiproxy
6
6
  Project-URL: Source, https://github.com/DecisionTreeTechnology/phiproxy
@@ -22,6 +22,10 @@ Requires-Python: >=3.10
22
22
  Provides-Extra: dev
23
23
  Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
24
24
  Requires-Dist: pytest>=8; extra == 'dev'
25
+ Provides-Extra: gliner
26
+ Requires-Dist: gliner>=0.2.20; extra == 'gliner'
27
+ Requires-Dist: protobuf; extra == 'gliner'
28
+ Requires-Dist: sentencepiece; extra == 'gliner'
25
29
  Provides-Extra: litellm
26
30
  Requires-Dist: litellm>=1.70; extra == 'litellm'
27
31
  Provides-Extra: presidio
@@ -51,7 +55,7 @@ When the reply comes back, phiproxy swaps the stand-ins for the real values, so
51
55
  "Thanks Sarah Chen, we'll call 416-555-0142 today." ◄──
52
56
  ```
53
57
 
54
- > **Status: pre-alpha (0.1.0a2).** APIs will change. Do not rely on it as your only control.
58
+ > **Status: pre-alpha (0.1.0a3).** APIs will change. Do not rely on it as your only control.
55
59
 
56
60
  ## Read this first
57
61
 
@@ -61,7 +65,8 @@ When the reply comes back, phiproxy swaps the stand-ins for the real values, so
61
65
 
62
66
  ## What makes it different
63
67
 
64
- - **Your own records as a detector.** Pass the identifiers you already hold for the patient (name, DOB, phone, health card). Exact and one-typo matching against them catches lowercase names, nicknames in context and misspellings that pattern matching misses. On the benchmark below this lifts recall from 61% to 86%.
68
+ - **Your own records as a detector.** Pass the identifiers you already hold for the patient (name, DOB, phone, health card). Exact and one-typo matching against them catches lowercase names and misspellings that pattern matching misses. On the blind benchmark below this lifts recall from 51% to 68%.
69
+ - **Optional local model.** `phiproxy[gliner]` adds a small named-entity model that runs on your own machine and finds names, cities and workplaces that no rule or record covers ("my husband ravi", "I live in Sudbury"). It lifts blind-benchmark recall to 92%, at about 50 ms per message on CPU.
65
70
  - **Stable tokens per conversation.** The same value gets the same token on every turn of a session, so the model can follow who is who and provider prompt caching keeps working. Tokens are sequential, not derived from the value (see 45 CFR 164.514(c)).
66
71
  - **Fail closed.** If masking errors, or a request holds media the policy does not allow, the request is rejected and nothing is sent. Clients cannot switch the guardrail off through request metadata.
67
72
  - **Clinic-specific identifiers.** Canadian health card numbers (OHIP, RAMQ), SIN, donor and embryo IDs, spoken numbers and dates from call transcripts.
@@ -149,40 +154,61 @@ docker compose -f deploy/docker-compose.yml up --build
149
154
 
150
155
  | Type | Examples | Layer |
151
156
  |---|---|---|
152
- | `NAME` | "this is Maria Lopez", "my husband Dave", any name from your records (any case, one typo) | regex context + known |
157
+ | `NAME` | "this is Maria Lopez", "my husband Dave", any name from your records (any case, one typo), "my husband ravi" | regex context + known + local model |
153
158
  | `PHONE` | 416-555-0142, (416) 555 0142, +44 20 7946 0958, "four one six five five five…" | regex + known |
154
159
  | `EMAIL`, `URL`, `IP_ADDRESS` | | regex |
155
160
  | `DATE` | 2026-11-03, 11/3, March 5, 1990, Nov 8th, "march fifteenth 1988" (year alone is not masked) | regex + known DOB |
156
161
  | `AGE` | ages over 89 only (Safe Harbor) | regex |
157
- | `ADDRESS`, `POSTAL_CODE`, `ZIP` | 22 Elm Street apt 4, PO Box 12, M5V 2T6, TX 78701 | regex + known |
162
+ | `ADDRESS`, `POSTAL_CODE`, `ZIP` | 22 Elm Street apt 4, PO Box 12, M5V 2T6, TX 78701, "I live in Sudbury" | regex + known + local model |
158
163
  | `HEALTH_CARD`, `SIN`, `SSN` | OHIP 1234-567-890-AB, RAMQ, keyword + number, Luhn-checked SIN | regex + known |
159
164
  | `MRN`, `ACCOUNT`, `DONOR_ID`, `ID` | "MRN: 00482913", "policy number: XK-99812", "donor D-4521" | regex + known |
160
- | `PROVIDER` | "Dr. Patel" (passes by default, as clinician names are not patient identifiers; enable with `Policy(disabled_types=set())`) | regex |
165
+ | `PROVIDER` | "Dr. Patel" (passes by default, as clinician names are not patient identifiers; enable with `Policy(disabled_types=set())`) | regex + local model |
166
+ | `EMPLOYER` | "I work at the Tim Hortons on Barton" | local model |
161
167
 
162
- Optional NER layer: `pip install phiproxy[presidio]` and add `PresidioDetector()` to the detector list. It runs in-process; nothing is sent anywhere to detect PHI.
168
+ Dates are also matched in French, Spanish, Polish and Hebrew ("le 12 mai 1990", "jueves 6 de noviembre", "5 בנובמבר"), and spoken digits in those languages. Code-shaped identifiers without a keyword ("SL-4471920-03", "W123456789", any bare number of 7+ digits) are masked as `ID`.
169
+
170
+ ### Optional local model
171
+
172
+ ```bash
173
+ pip install "phiproxy[gliner]"
174
+ ```
175
+
176
+ ```python
177
+ from phiproxy import Masker, RegexDetector
178
+ from phiproxy.detectors.gliner import GlinerDetector
179
+
180
+ masker = Masker(detectors=[RegexDetector(), GlinerDetector()])
181
+ ```
182
+
183
+ `GlinerDetector` uses [GLiNER](https://github.com/urchade/GLiNER) with the Apache-2.0 model `urchade/gliner_multi_pii-v1` (about 1.2 GB). It runs in-process: nothing is sent anywhere to detect PHI. The model is downloaded from Hugging Face on first use, so in production download it once and pass its local path as `model=`, or set `HF_HUB_OFFLINE=1`. Model guesses never override the policy: a guess on "Dr. Haddad" does not mask a clinician name that the policy lets through.
184
+
185
+ Presidio is also supported: `pip install phiproxy[presidio]` and add `PresidioDetector()`.
163
186
 
164
187
  ## Benchmark
165
188
 
166
- `benchmark/` holds a synthetic **patient communications** set: SMS, web chat, call transcripts and referral notes, the kind of text clinics actually send to LLMs. Most existing de-identification work targets clinical notes instead. All people and numbers are invented. Run it with `python benchmark/score.py`.
189
+ `benchmark/` holds two synthetic **patient communications** sets: SMS, web chat, call transcripts, emails and referral notes, the kind of text clinics actually send to LLMs. Most existing de-identification work targets clinical notes instead. All people and numbers are invented.
167
190
 
168
- v0 results (288 messages, 560 labelled spans):
191
+ - **v0** (288 messages, English): template-based, written by the detector authors.
192
+ - **blind v1** (420 messages; English, French, Spanish, Polish, Hebrew and mixed; 129 messages with no identifiers but full of lookalikes such as drug doses and embryo grades): written by a separate AI agent that never saw the detector code.
169
193
 
170
- | config | recall | missed |
171
- |---|---|---|
172
- | regex | 61.4% | 216 |
173
- | regex + known entities | **85.7%** | 80 |
194
+ | recall | v0 | blind v1, first score | blind v1, now |
195
+ |---|---|---|---|
196
+ | regex | 61.4% | 36.6% | 50.5% |
197
+ | regex + known entities | 85.7% | 54.4% | 67.8% |
198
+ | regex + known + local model | 99.1% | 80.1% | **92.2%** |
174
199
 
175
- | recall by channel | regex | regex + known |
176
- |---|---|---|
177
- | SMS | 55% | 95% |
178
- | web chat | 74% | 84% |
179
- | call transcripts | 50% | 80% |
180
- | referral notes | 63% | 79% |
200
+ | blind v1, regex + known + local model | |
201
+ |---|---|
202
+ | by channel | SMS 94%, chat 91%, email 96%, notes 94%, call transcripts 86% |
203
+ | by language | en 91%, fr 94%, es 93%, pl 95%, he 92%, mixed 97% |
204
+ | precision | 86.9% |
205
+ | messages without identifiers where something was masked | 19% |
181
206
 
182
207
  **How to read this honestly:**
183
- - v0 was written by the same team that wrote the detectors. Expect real-world recall to be **lower**. An independent, larger set is the next milestone, and contributions are welcome.
184
- - Precision is reported as 100% on v0, but v0 has almost no non-PHI distractor text, so that number means little yet.
185
- - Known misses today: **cities and workplaces** ("I live in Sudbury", "I work at the Tim Hortons on King") and **names not in your records written in lowercase** ("my husband ravi"). These need the NER layer or a local model, which is on the roadmap.
208
+ - The first blind score is the fair one: the detectors were frozen before v1 was scored. "Now" includes fixes made after looking at v1's misses, so it is optimistic. A new blind set will re-test it.
209
+ - v0 overstated recall by about 30 points. Expect real traffic to score lower than v1 too.
210
+ - Known misses: dates spoken in words outside English, some cities, workplaces described in a phrase, and lowercase names that are also words ("joy", "will"). Without the local model, cities and names not in your records are mostly missed.
211
+ - Details, label rules and the full tables are in [`benchmark/README.md`](https://github.com/DecisionTreeTechnology/phiproxy/blob/main/benchmark/README.md).
186
212
 
187
213
  ## Deployment shapes
188
214
 
@@ -194,9 +220,9 @@ The vault holds real values. Protect Redis like the PHI store it is: private net
194
220
 
195
221
  ## Roadmap
196
222
 
197
- - Larger independent benchmark with distractor text, plus multilingual sets (French, Polish, Hebrew, Spanish)
198
- - Local small-model detector for names, places and workplaces
199
- - Gazetteer for cities and regions
223
+ - A benchmark set written by clinic staff, and a second blind AI-written set to re-test the current numbers
224
+ - Gazetteer for cities the model misses; spoken dates in French, Spanish, Polish and Hebrew
225
+ - Smaller and faster local models (distilled or ONNX)
200
226
  - HTTP sidecar; Anthropic `/v1/messages` streaming
201
227
  - Masked logging hook for LiteLLM
202
228
 
@@ -19,7 +19,7 @@ When the reply comes back, phiproxy swaps the stand-ins for the real values, so
19
19
  "Thanks Sarah Chen, we'll call 416-555-0142 today." ◄──
20
20
  ```
21
21
 
22
- > **Status: pre-alpha (0.1.0a2).** APIs will change. Do not rely on it as your only control.
22
+ > **Status: pre-alpha (0.1.0a3).** APIs will change. Do not rely on it as your only control.
23
23
 
24
24
  ## Read this first
25
25
 
@@ -29,7 +29,8 @@ When the reply comes back, phiproxy swaps the stand-ins for the real values, so
29
29
 
30
30
  ## What makes it different
31
31
 
32
- - **Your own records as a detector.** Pass the identifiers you already hold for the patient (name, DOB, phone, health card). Exact and one-typo matching against them catches lowercase names, nicknames in context and misspellings that pattern matching misses. On the benchmark below this lifts recall from 61% to 86%.
32
+ - **Your own records as a detector.** Pass the identifiers you already hold for the patient (name, DOB, phone, health card). Exact and one-typo matching against them catches lowercase names and misspellings that pattern matching misses. On the blind benchmark below this lifts recall from 51% to 68%.
33
+ - **Optional local model.** `phiproxy[gliner]` adds a small named-entity model that runs on your own machine and finds names, cities and workplaces that no rule or record covers ("my husband ravi", "I live in Sudbury"). It lifts blind-benchmark recall to 92%, at about 50 ms per message on CPU.
33
34
  - **Stable tokens per conversation.** The same value gets the same token on every turn of a session, so the model can follow who is who and provider prompt caching keeps working. Tokens are sequential, not derived from the value (see 45 CFR 164.514(c)).
34
35
  - **Fail closed.** If masking errors, or a request holds media the policy does not allow, the request is rejected and nothing is sent. Clients cannot switch the guardrail off through request metadata.
35
36
  - **Clinic-specific identifiers.** Canadian health card numbers (OHIP, RAMQ), SIN, donor and embryo IDs, spoken numbers and dates from call transcripts.
@@ -117,40 +118,61 @@ docker compose -f deploy/docker-compose.yml up --build
117
118
 
118
119
  | Type | Examples | Layer |
119
120
  |---|---|---|
120
- | `NAME` | "this is Maria Lopez", "my husband Dave", any name from your records (any case, one typo) | regex context + known |
121
+ | `NAME` | "this is Maria Lopez", "my husband Dave", any name from your records (any case, one typo), "my husband ravi" | regex context + known + local model |
121
122
  | `PHONE` | 416-555-0142, (416) 555 0142, +44 20 7946 0958, "four one six five five five…" | regex + known |
122
123
  | `EMAIL`, `URL`, `IP_ADDRESS` | | regex |
123
124
  | `DATE` | 2026-11-03, 11/3, March 5, 1990, Nov 8th, "march fifteenth 1988" (year alone is not masked) | regex + known DOB |
124
125
  | `AGE` | ages over 89 only (Safe Harbor) | regex |
125
- | `ADDRESS`, `POSTAL_CODE`, `ZIP` | 22 Elm Street apt 4, PO Box 12, M5V 2T6, TX 78701 | regex + known |
126
+ | `ADDRESS`, `POSTAL_CODE`, `ZIP` | 22 Elm Street apt 4, PO Box 12, M5V 2T6, TX 78701, "I live in Sudbury" | regex + known + local model |
126
127
  | `HEALTH_CARD`, `SIN`, `SSN` | OHIP 1234-567-890-AB, RAMQ, keyword + number, Luhn-checked SIN | regex + known |
127
128
  | `MRN`, `ACCOUNT`, `DONOR_ID`, `ID` | "MRN: 00482913", "policy number: XK-99812", "donor D-4521" | regex + known |
128
- | `PROVIDER` | "Dr. Patel" (passes by default, as clinician names are not patient identifiers; enable with `Policy(disabled_types=set())`) | regex |
129
+ | `PROVIDER` | "Dr. Patel" (passes by default, as clinician names are not patient identifiers; enable with `Policy(disabled_types=set())`) | regex + local model |
130
+ | `EMPLOYER` | "I work at the Tim Hortons on Barton" | local model |
129
131
 
130
- Optional NER layer: `pip install phiproxy[presidio]` and add `PresidioDetector()` to the detector list. It runs in-process; nothing is sent anywhere to detect PHI.
132
+ Dates are also matched in French, Spanish, Polish and Hebrew ("le 12 mai 1990", "jueves 6 de noviembre", "5 בנובמבר"), and spoken digits in those languages. Code-shaped identifiers without a keyword ("SL-4471920-03", "W123456789", any bare number of 7+ digits) are masked as `ID`.
133
+
134
+ ### Optional local model
135
+
136
+ ```bash
137
+ pip install "phiproxy[gliner]"
138
+ ```
139
+
140
+ ```python
141
+ from phiproxy import Masker, RegexDetector
142
+ from phiproxy.detectors.gliner import GlinerDetector
143
+
144
+ masker = Masker(detectors=[RegexDetector(), GlinerDetector()])
145
+ ```
146
+
147
+ `GlinerDetector` uses [GLiNER](https://github.com/urchade/GLiNER) with the Apache-2.0 model `urchade/gliner_multi_pii-v1` (about 1.2 GB). It runs in-process: nothing is sent anywhere to detect PHI. The model is downloaded from Hugging Face on first use, so in production download it once and pass its local path as `model=`, or set `HF_HUB_OFFLINE=1`. Model guesses never override the policy: a guess on "Dr. Haddad" does not mask a clinician name that the policy lets through.
148
+
149
+ Presidio is also supported: `pip install phiproxy[presidio]` and add `PresidioDetector()`.
131
150
 
132
151
  ## Benchmark
133
152
 
134
- `benchmark/` holds a synthetic **patient communications** set: SMS, web chat, call transcripts and referral notes, the kind of text clinics actually send to LLMs. Most existing de-identification work targets clinical notes instead. All people and numbers are invented. Run it with `python benchmark/score.py`.
153
+ `benchmark/` holds two synthetic **patient communications** sets: SMS, web chat, call transcripts, emails and referral notes, the kind of text clinics actually send to LLMs. Most existing de-identification work targets clinical notes instead. All people and numbers are invented.
135
154
 
136
- v0 results (288 messages, 560 labelled spans):
155
+ - **v0** (288 messages, English): template-based, written by the detector authors.
156
+ - **blind v1** (420 messages; English, French, Spanish, Polish, Hebrew and mixed; 129 messages with no identifiers but full of lookalikes such as drug doses and embryo grades): written by a separate AI agent that never saw the detector code.
137
157
 
138
- | config | recall | missed |
139
- |---|---|---|
140
- | regex | 61.4% | 216 |
141
- | regex + known entities | **85.7%** | 80 |
158
+ | recall | v0 | blind v1, first score | blind v1, now |
159
+ |---|---|---|---|
160
+ | regex | 61.4% | 36.6% | 50.5% |
161
+ | regex + known entities | 85.7% | 54.4% | 67.8% |
162
+ | regex + known + local model | 99.1% | 80.1% | **92.2%** |
142
163
 
143
- | recall by channel | regex | regex + known |
144
- |---|---|---|
145
- | SMS | 55% | 95% |
146
- | web chat | 74% | 84% |
147
- | call transcripts | 50% | 80% |
148
- | referral notes | 63% | 79% |
164
+ | blind v1, regex + known + local model | |
165
+ |---|---|
166
+ | by channel | SMS 94%, chat 91%, email 96%, notes 94%, call transcripts 86% |
167
+ | by language | en 91%, fr 94%, es 93%, pl 95%, he 92%, mixed 97% |
168
+ | precision | 86.9% |
169
+ | messages without identifiers where something was masked | 19% |
149
170
 
150
171
  **How to read this honestly:**
151
- - v0 was written by the same team that wrote the detectors. Expect real-world recall to be **lower**. An independent, larger set is the next milestone, and contributions are welcome.
152
- - Precision is reported as 100% on v0, but v0 has almost no non-PHI distractor text, so that number means little yet.
153
- - Known misses today: **cities and workplaces** ("I live in Sudbury", "I work at the Tim Hortons on King") and **names not in your records written in lowercase** ("my husband ravi"). These need the NER layer or a local model, which is on the roadmap.
172
+ - The first blind score is the fair one: the detectors were frozen before v1 was scored. "Now" includes fixes made after looking at v1's misses, so it is optimistic. A new blind set will re-test it.
173
+ - v0 overstated recall by about 30 points. Expect real traffic to score lower than v1 too.
174
+ - Known misses: dates spoken in words outside English, some cities, workplaces described in a phrase, and lowercase names that are also words ("joy", "will"). Without the local model, cities and names not in your records are mostly missed.
175
+ - Details, label rules and the full tables are in [`benchmark/README.md`](https://github.com/DecisionTreeTechnology/phiproxy/blob/main/benchmark/README.md).
154
176
 
155
177
  ## Deployment shapes
156
178
 
@@ -162,9 +184,9 @@ The vault holds real values. Protect Redis like the PHI store it is: private net
162
184
 
163
185
  ## Roadmap
164
186
 
165
- - Larger independent benchmark with distractor text, plus multilingual sets (French, Polish, Hebrew, Spanish)
166
- - Local small-model detector for names, places and workplaces
167
- - Gazetteer for cities and regions
187
+ - A benchmark set written by clinic staff, and a second blind AI-written set to re-test the current numbers
188
+ - Gazetteer for cities the model misses; spoken dates in French, Spanish, Polish and Hebrew
189
+ - Smaller and faster local models (distilled or ONNX)
168
190
  - HTTP sidecar; Anthropic `/v1/messages` streaming
169
191
  - Masked logging hook for LiteLLM
170
192
 
@@ -0,0 +1,62 @@
1
+ # Patient-communications PHI benchmark
2
+
3
+ Labelled sets of the kind of text clinics send to LLMs: patient SMS, web chat, phone-call transcripts, emails and referral or staff notes. Most open de-identification datasets cover clinical notes; these cover patient communications. Every person and number in them is invented.
4
+
5
+ ## The two sets
6
+
7
+ | set | messages | labelled identifiers | languages | written by |
8
+ |---|---|---|---|---|
9
+ | `patient_comms_v0.jsonl` | 288 | 560 | English (one French template) | generated from templates by the detector authors |
10
+ | `patient_comms_blind_v1.jsonl` | 420 (129 with no identifiers) | 664 + 57 clinician names | en 260, fr 50, es 40, pl 30, he 20, mixed 20 | written blind, see below |
11
+
12
+ **How v1 was written.** v1 was written by a separate AI agent (Claude) that was given only a brief: the label rules, the channels, the languages, realistic mess (typos, lowercase names, spoken numbers, spelled-out names) and a list of things that look like identifiers but are not (drug doses, embryo grades, cycle days, billing codes). It was told not to read the detector code, the tests or v0, and it wrote every message from scratch. So v1 is independent of the detector code, but it is not independent of AI-written text. A set written by clinic staff is still the goal.
13
+
14
+ **How v1 was scored.** The local-model detector was tuned on v0 only. Its settings were committed (082c2e1) and v1 was then scored once. Those first scores are kept in `results_blind_v1_frozen.json`. After that, v1 was used to find and fix gaps, so the current v1 numbers are optimistic in the same way v0's always were. The next blind set will re-test them.
15
+
16
+ | recall | v0 (current) | v1 first blind score | v1 after fixes from v1 |
17
+ |---|---|---|---|
18
+ | regex | 61.4% | 36.6% | 50.5% |
19
+ | regex + known entities | 85.7% | 54.4% | 67.8% |
20
+ | regex + local model | 98.2% | 78.2% | 89.9% |
21
+ | regex + known + local model | 99.1% | 80.1% | 92.2% |
22
+
23
+ The 30-point gap between v0 and v1 is the honest lesson: a set written by the people who wrote the detectors flatters them.
24
+
25
+ ## Files
26
+
27
+ - `generate.py`: deterministic generator (fixed seed). It writes `data/patient_comms_v0.jsonl`.
28
+ - `blind/blind_v1.source.jsonl`: v1 as written, with inline labels (`[[NAME|jess]]`). `blind/convert.py` turns it into `data/patient_comms_blind_v1.jsonl`.
29
+ - `score.py`: scores every available detector configuration and prints recall by type, channel and language. The local-model configurations appear when `phiproxy[gliner]` is installed.
30
+ - `results_v0.json`, `results_blind_v1.json`, `results_blind_v1_frozen.json`: scores for the built-in configurations.
31
+
32
+ Run `python benchmark/score.py` for v0 and `python benchmark/score.py --data benchmark/data/patient_comms_blind_v1.jsonl` for v1.
33
+
34
+ ## Record format
35
+
36
+ ```json
37
+ {
38
+ "id": "sms-00-3",
39
+ "channel": "sms",
40
+ "text": "Hi it's Priya, can I move my bloodwork to Oct 3? My cell is 647-555-0112",
41
+ "spans": [{"start": 8, "end": 13, "type": "NAME"}, ...],
42
+ "known": {"names": ["Priya Natarajan"], "phones": ["647-555-0112"], ...}
43
+ }
44
+ ```
45
+
46
+ `known` is what the clinic system would already hold about the patient. It lets the benchmark measure the known-entity layer.
47
+
48
+ ## Scoring rules
49
+
50
+ - A labelled span is **caught** when detected spans cover at least 80% of its characters. The detected type does not have to match, because catching a phone number as an `ID` still protects it.
51
+ - Scoring uses the default policy, under which clinician names pass through. Clinician names (`PROVIDER`) are therefore not counted in recall, and masking one counts as a false positive.
52
+ - A detected span is **correct** when it overlaps a labelled identifier. **Clean messages masked** is the share of messages without identifiers in which something was masked anyway.
53
+ - What counts as an identifier follows HIPAA Safe Harbor (45 CFR 164.514(b)(2)): names of patients and relatives, all date elements except year, ages over 89, geographic units smaller than a state or province (street, city, postal code), contact details, and record, account, plan and other identifying numbers. Workplaces are labelled too because they can identify a person.
54
+
55
+ ## Known limits
56
+
57
+ - v0 is template-based and was written by the detector authors, so its scores are optimistic.
58
+ - v1 is AI-written. Real messages are messier still, and the labels have a few judgment calls: the author labelled some weekdays ("Sunday") and "yesterday" as dates, left some cities unlabelled, mostly where they are the clinic's location ("are you a clinic in Hamilton or only Toronto?"), left clinic holiday dates unlabelled, and did not label a few spelled-out names. These are left as written.
59
+ - Still missed on v1: dates spoken in words in French, Spanish, Polish and Hebrew, some cities the model does not know ("Hamilton", "kitchener"), workplaces described in a phrase ("the canada post depot on lakeshore"), and lowercase names that are also words ("joy", "will").
60
+ - With the local model, something is masked in 19% of messages that contain no identifiers (drug names such as Gonal-f, staff first names, holidays, phone extensions, and role words such as "Paciente"). That costs answer quality, not privacy.
61
+
62
+ Contributions are welcome, especially messages written by people who work in clinics (see ../CONTRIBUTING.md). All data must be synthetic.