phiproxy 0.1.0a2__tar.gz → 0.1.0a3__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/.github/workflows/ci.yml +2 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/.gitignore +1 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/PKG-INFO +51 -25
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/README.md +46 -24
- phiproxy-0.1.0a3/benchmark/README.md +62 -0
- phiproxy-0.1.0a3/benchmark/blind/blind_v1.source.jsonl +420 -0
- phiproxy-0.1.0a3/benchmark/blind/convert.py +56 -0
- phiproxy-0.1.0a3/benchmark/data/patient_comms_blind_v1.jsonl +420 -0
- phiproxy-0.1.0a3/benchmark/results_blind_v1.json +182 -0
- phiproxy-0.1.0a3/benchmark/results_blind_v1_frozen.json +182 -0
- phiproxy-0.1.0a3/benchmark/results_v0.json +130 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/benchmark/score.py +38 -9
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/pyproject.toml +1 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/src/phiproxy/__init__.py +1 -1
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/src/phiproxy/detectors/base.py +1 -1
- phiproxy-0.1.0a3/src/phiproxy/detectors/gliner.py +156 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/src/phiproxy/detectors/regex.py +97 -10
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/src/phiproxy/entities.py +1 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/src/phiproxy/masker.py +15 -1
- phiproxy-0.1.0a3/tests/test_gliner.py +75 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/tests/test_regex.py +57 -0
- phiproxy-0.1.0a2/benchmark/README.md +0 -38
- phiproxy-0.1.0a2/benchmark/results_v0.json +0 -56
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/.github/workflows/release.yml +0 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/CONTRIBUTING.md +0 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/LICENSE +0 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/NOTICE +0 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/SECURITY.md +0 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/benchmark/data/patient_comms_v0.jsonl +0 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/benchmark/generate.py +0 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/deploy/.env.example +0 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/deploy/Dockerfile +0 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/deploy/docker-compose.yml +0 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/deploy/litellm.config.yaml +0 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/examples/e2e/config.yaml +0 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/examples/e2e/fake_llm.py +0 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/examples/e2e/run_e2e.py +0 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/src/phiproxy/dates.py +0 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/src/phiproxy/detectors/__init__.py +0 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/src/phiproxy/detectors/known.py +0 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/src/phiproxy/detectors/presidio.py +0 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/src/phiproxy/integrations/__init__.py +0 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/src/phiproxy/integrations/litellm.py +0 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/src/phiproxy/vault.py +0 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/tests/test_dates.py +0 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/tests/test_known.py +0 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/tests/test_litellm.py +0 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/tests/test_masker.py +0 -0
- {phiproxy-0.1.0a2 → phiproxy-0.1.0a3}/tests/test_vault.py +0 -0
|
@@ -23,6 +23,8 @@ jobs:
|
|
|
23
23
|
- run: pytest -q
|
|
24
24
|
- name: Benchmark (regex+known must stay at or above 85% recall)
|
|
25
25
|
run: python benchmark/score.py --config regex+known --min-recall 0.85
|
|
26
|
+
- name: Blind benchmark (regex+known must stay at or above 65% recall)
|
|
27
|
+
run: python benchmark/score.py --data benchmark/data/patient_comms_blind_v1.jsonl --config regex+known --min-recall 0.65
|
|
26
28
|
|
|
27
29
|
e2e:
|
|
28
30
|
runs-on: ubuntu-latest
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: phiproxy
|
|
3
|
-
Version: 0.1.
|
|
3
|
+
Version: 0.1.0a3
|
|
4
4
|
Summary: Your patients' stand-in on the way to any LLM, and back: reversible PHI masking proxy (LiteLLM guardrail + Python library).
|
|
5
5
|
Project-URL: Homepage, https://decisiontree.tech/open-source/phiproxy
|
|
6
6
|
Project-URL: Source, https://github.com/DecisionTreeTechnology/phiproxy
|
|
@@ -22,6 +22,10 @@ Requires-Python: >=3.10
|
|
|
22
22
|
Provides-Extra: dev
|
|
23
23
|
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
|
|
24
24
|
Requires-Dist: pytest>=8; extra == 'dev'
|
|
25
|
+
Provides-Extra: gliner
|
|
26
|
+
Requires-Dist: gliner>=0.2.20; extra == 'gliner'
|
|
27
|
+
Requires-Dist: protobuf; extra == 'gliner'
|
|
28
|
+
Requires-Dist: sentencepiece; extra == 'gliner'
|
|
25
29
|
Provides-Extra: litellm
|
|
26
30
|
Requires-Dist: litellm>=1.70; extra == 'litellm'
|
|
27
31
|
Provides-Extra: presidio
|
|
@@ -51,7 +55,7 @@ When the reply comes back, phiproxy swaps the stand-ins for the real values, so
|
|
|
51
55
|
"Thanks Sarah Chen, we'll call 416-555-0142 today." ◄──
|
|
52
56
|
```
|
|
53
57
|
|
|
54
|
-
> **Status: pre-alpha (0.1.
|
|
58
|
+
> **Status: pre-alpha (0.1.0a3).** APIs will change. Do not rely on it as your only control.
|
|
55
59
|
|
|
56
60
|
## Read this first
|
|
57
61
|
|
|
@@ -61,7 +65,8 @@ When the reply comes back, phiproxy swaps the stand-ins for the real values, so
|
|
|
61
65
|
|
|
62
66
|
## What makes it different
|
|
63
67
|
|
|
64
|
-
- **Your own records as a detector.** Pass the identifiers you already hold for the patient (name, DOB, phone, health card). Exact and one-typo matching against them catches lowercase names
|
|
68
|
+
- **Your own records as a detector.** Pass the identifiers you already hold for the patient (name, DOB, phone, health card). Exact and one-typo matching against them catches lowercase names and misspellings that pattern matching misses. On the blind benchmark below this lifts recall from 51% to 68%.
|
|
69
|
+
- **Optional local model.** `phiproxy[gliner]` adds a small named-entity model that runs on your own machine and finds names, cities and workplaces that no rule or record covers ("my husband ravi", "I live in Sudbury"). It lifts blind-benchmark recall to 92%, at about 50 ms per message on CPU.
|
|
65
70
|
- **Stable tokens per conversation.** The same value gets the same token on every turn of a session, so the model can follow who is who and provider prompt caching keeps working. Tokens are sequential, not derived from the value (see 45 CFR 164.514(c)).
|
|
66
71
|
- **Fail closed.** If masking errors, or a request holds media the policy does not allow, the request is rejected and nothing is sent. Clients cannot switch the guardrail off through request metadata.
|
|
67
72
|
- **Clinic-specific identifiers.** Canadian health card numbers (OHIP, RAMQ), SIN, donor and embryo IDs, spoken numbers and dates from call transcripts.
|
|
@@ -149,40 +154,61 @@ docker compose -f deploy/docker-compose.yml up --build
|
|
|
149
154
|
|
|
150
155
|
| Type | Examples | Layer |
|
|
151
156
|
|---|---|---|
|
|
152
|
-
| `NAME` | "this is Maria Lopez", "my husband Dave", any name from your records (any case, one typo) | regex context + known |
|
|
157
|
+
| `NAME` | "this is Maria Lopez", "my husband Dave", any name from your records (any case, one typo), "my husband ravi" | regex context + known + local model |
|
|
153
158
|
| `PHONE` | 416-555-0142, (416) 555 0142, +44 20 7946 0958, "four one six five five five…" | regex + known |
|
|
154
159
|
| `EMAIL`, `URL`, `IP_ADDRESS` | | regex |
|
|
155
160
|
| `DATE` | 2026-11-03, 11/3, March 5, 1990, Nov 8th, "march fifteenth 1988" (year alone is not masked) | regex + known DOB |
|
|
156
161
|
| `AGE` | ages over 89 only (Safe Harbor) | regex |
|
|
157
|
-
| `ADDRESS`, `POSTAL_CODE`, `ZIP` | 22 Elm Street apt 4, PO Box 12, M5V 2T6, TX 78701 | regex + known |
|
|
162
|
+
| `ADDRESS`, `POSTAL_CODE`, `ZIP` | 22 Elm Street apt 4, PO Box 12, M5V 2T6, TX 78701, "I live in Sudbury" | regex + known + local model |
|
|
158
163
|
| `HEALTH_CARD`, `SIN`, `SSN` | OHIP 1234-567-890-AB, RAMQ, keyword + number, Luhn-checked SIN | regex + known |
|
|
159
164
|
| `MRN`, `ACCOUNT`, `DONOR_ID`, `ID` | "MRN: 00482913", "policy number: XK-99812", "donor D-4521" | regex + known |
|
|
160
|
-
| `PROVIDER` | "Dr. Patel" (passes by default, as clinician names are not patient identifiers; enable with `Policy(disabled_types=set())`) | regex |
|
|
165
|
+
| `PROVIDER` | "Dr. Patel" (passes by default, as clinician names are not patient identifiers; enable with `Policy(disabled_types=set())`) | regex + local model |
|
|
166
|
+
| `EMPLOYER` | "I work at the Tim Hortons on Barton" | local model |
|
|
161
167
|
|
|
162
|
-
|
|
168
|
+
Dates are also matched in French, Spanish, Polish and Hebrew ("le 12 mai 1990", "jueves 6 de noviembre", "5 בנובמבר"), and spoken digits in those languages. Code-shaped identifiers without a keyword ("SL-4471920-03", "W123456789", any bare number of 7+ digits) are masked as `ID`.
|
|
169
|
+
|
|
170
|
+
### Optional local model
|
|
171
|
+
|
|
172
|
+
```bash
|
|
173
|
+
pip install "phiproxy[gliner]"
|
|
174
|
+
```
|
|
175
|
+
|
|
176
|
+
```python
|
|
177
|
+
from phiproxy import Masker, RegexDetector
|
|
178
|
+
from phiproxy.detectors.gliner import GlinerDetector
|
|
179
|
+
|
|
180
|
+
masker = Masker(detectors=[RegexDetector(), GlinerDetector()])
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
`GlinerDetector` uses [GLiNER](https://github.com/urchade/GLiNER) with the Apache-2.0 model `urchade/gliner_multi_pii-v1` (about 1.2 GB). It runs in-process: nothing is sent anywhere to detect PHI. The model is downloaded from Hugging Face on first use, so in production download it once and pass its local path as `model=`, or set `HF_HUB_OFFLINE=1`. Model guesses never override the policy: a guess on "Dr. Haddad" does not mask a clinician name that the policy lets through.
|
|
184
|
+
|
|
185
|
+
Presidio is also supported: `pip install phiproxy[presidio]` and add `PresidioDetector()`.
|
|
163
186
|
|
|
164
187
|
## Benchmark
|
|
165
188
|
|
|
166
|
-
`benchmark/` holds
|
|
189
|
+
`benchmark/` holds two synthetic **patient communications** sets: SMS, web chat, call transcripts, emails and referral notes, the kind of text clinics actually send to LLMs. Most existing de-identification work targets clinical notes instead. All people and numbers are invented.
|
|
167
190
|
|
|
168
|
-
v0
|
|
191
|
+
- **v0** (288 messages, English): template-based, written by the detector authors.
|
|
192
|
+
- **blind v1** (420 messages; English, French, Spanish, Polish, Hebrew and mixed; 129 messages with no identifiers but full of lookalikes such as drug doses and embryo grades): written by a separate AI agent that never saw the detector code.
|
|
169
193
|
|
|
170
|
-
|
|
|
171
|
-
|
|
172
|
-
| regex | 61.4% |
|
|
173
|
-
| regex + known entities |
|
|
194
|
+
| recall | v0 | blind v1, first score | blind v1, now |
|
|
195
|
+
|---|---|---|---|
|
|
196
|
+
| regex | 61.4% | 36.6% | 50.5% |
|
|
197
|
+
| regex + known entities | 85.7% | 54.4% | 67.8% |
|
|
198
|
+
| regex + known + local model | 99.1% | 80.1% | **92.2%** |
|
|
174
199
|
|
|
175
|
-
|
|
|
176
|
-
|
|
177
|
-
|
|
|
178
|
-
|
|
|
179
|
-
|
|
|
180
|
-
|
|
|
200
|
+
| blind v1, regex + known + local model | |
|
|
201
|
+
|---|---|
|
|
202
|
+
| by channel | SMS 94%, chat 91%, email 96%, notes 94%, call transcripts 86% |
|
|
203
|
+
| by language | en 91%, fr 94%, es 93%, pl 95%, he 92%, mixed 97% |
|
|
204
|
+
| precision | 86.9% |
|
|
205
|
+
| messages without identifiers where something was masked | 19% |
|
|
181
206
|
|
|
182
207
|
**How to read this honestly:**
|
|
183
|
-
-
|
|
184
|
-
-
|
|
185
|
-
- Known misses
|
|
208
|
+
- The first blind score is the fair one: the detectors were frozen before v1 was scored. "Now" includes fixes made after looking at v1's misses, so it is optimistic. A new blind set will re-test it.
|
|
209
|
+
- v0 overstated recall by about 30 points. Expect real traffic to score lower than v1 too.
|
|
210
|
+
- Known misses: dates spoken in words outside English, some cities, workplaces described in a phrase, and lowercase names that are also words ("joy", "will"). Without the local model, cities and names not in your records are mostly missed.
|
|
211
|
+
- Details, label rules and the full tables are in [`benchmark/README.md`](https://github.com/DecisionTreeTechnology/phiproxy/blob/main/benchmark/README.md).
|
|
186
212
|
|
|
187
213
|
## Deployment shapes
|
|
188
214
|
|
|
@@ -194,9 +220,9 @@ The vault holds real values. Protect Redis like the PHI store it is: private net
|
|
|
194
220
|
|
|
195
221
|
## Roadmap
|
|
196
222
|
|
|
197
|
-
-
|
|
198
|
-
-
|
|
199
|
-
-
|
|
223
|
+
- A benchmark set written by clinic staff, and a second blind AI-written set to re-test the current numbers
|
|
224
|
+
- Gazetteer for cities the model misses; spoken dates in French, Spanish, Polish and Hebrew
|
|
225
|
+
- Smaller and faster local models (distilled or ONNX)
|
|
200
226
|
- HTTP sidecar; Anthropic `/v1/messages` streaming
|
|
201
227
|
- Masked logging hook for LiteLLM
|
|
202
228
|
|
|
@@ -19,7 +19,7 @@ When the reply comes back, phiproxy swaps the stand-ins for the real values, so
|
|
|
19
19
|
"Thanks Sarah Chen, we'll call 416-555-0142 today." ◄──
|
|
20
20
|
```
|
|
21
21
|
|
|
22
|
-
> **Status: pre-alpha (0.1.
|
|
22
|
+
> **Status: pre-alpha (0.1.0a3).** APIs will change. Do not rely on it as your only control.
|
|
23
23
|
|
|
24
24
|
## Read this first
|
|
25
25
|
|
|
@@ -29,7 +29,8 @@ When the reply comes back, phiproxy swaps the stand-ins for the real values, so
|
|
|
29
29
|
|
|
30
30
|
## What makes it different
|
|
31
31
|
|
|
32
|
-
- **Your own records as a detector.** Pass the identifiers you already hold for the patient (name, DOB, phone, health card). Exact and one-typo matching against them catches lowercase names
|
|
32
|
+
- **Your own records as a detector.** Pass the identifiers you already hold for the patient (name, DOB, phone, health card). Exact and one-typo matching against them catches lowercase names and misspellings that pattern matching misses. On the blind benchmark below this lifts recall from 51% to 68%.
|
|
33
|
+
- **Optional local model.** `phiproxy[gliner]` adds a small named-entity model that runs on your own machine and finds names, cities and workplaces that no rule or record covers ("my husband ravi", "I live in Sudbury"). It lifts blind-benchmark recall to 92%, at about 50 ms per message on CPU.
|
|
33
34
|
- **Stable tokens per conversation.** The same value gets the same token on every turn of a session, so the model can follow who is who and provider prompt caching keeps working. Tokens are sequential, not derived from the value (see 45 CFR 164.514(c)).
|
|
34
35
|
- **Fail closed.** If masking errors, or a request holds media the policy does not allow, the request is rejected and nothing is sent. Clients cannot switch the guardrail off through request metadata.
|
|
35
36
|
- **Clinic-specific identifiers.** Canadian health card numbers (OHIP, RAMQ), SIN, donor and embryo IDs, spoken numbers and dates from call transcripts.
|
|
@@ -117,40 +118,61 @@ docker compose -f deploy/docker-compose.yml up --build
|
|
|
117
118
|
|
|
118
119
|
| Type | Examples | Layer |
|
|
119
120
|
|---|---|---|
|
|
120
|
-
| `NAME` | "this is Maria Lopez", "my husband Dave", any name from your records (any case, one typo) | regex context + known |
|
|
121
|
+
| `NAME` | "this is Maria Lopez", "my husband Dave", any name from your records (any case, one typo), "my husband ravi" | regex context + known + local model |
|
|
121
122
|
| `PHONE` | 416-555-0142, (416) 555 0142, +44 20 7946 0958, "four one six five five five…" | regex + known |
|
|
122
123
|
| `EMAIL`, `URL`, `IP_ADDRESS` | | regex |
|
|
123
124
|
| `DATE` | 2026-11-03, 11/3, March 5, 1990, Nov 8th, "march fifteenth 1988" (year alone is not masked) | regex + known DOB |
|
|
124
125
|
| `AGE` | ages over 89 only (Safe Harbor) | regex |
|
|
125
|
-
| `ADDRESS`, `POSTAL_CODE`, `ZIP` | 22 Elm Street apt 4, PO Box 12, M5V 2T6, TX 78701 | regex + known |
|
|
126
|
+
| `ADDRESS`, `POSTAL_CODE`, `ZIP` | 22 Elm Street apt 4, PO Box 12, M5V 2T6, TX 78701, "I live in Sudbury" | regex + known + local model |
|
|
126
127
|
| `HEALTH_CARD`, `SIN`, `SSN` | OHIP 1234-567-890-AB, RAMQ, keyword + number, Luhn-checked SIN | regex + known |
|
|
127
128
|
| `MRN`, `ACCOUNT`, `DONOR_ID`, `ID` | "MRN: 00482913", "policy number: XK-99812", "donor D-4521" | regex + known |
|
|
128
|
-
| `PROVIDER` | "Dr. Patel" (passes by default, as clinician names are not patient identifiers; enable with `Policy(disabled_types=set())`) | regex |
|
|
129
|
+
| `PROVIDER` | "Dr. Patel" (passes by default, as clinician names are not patient identifiers; enable with `Policy(disabled_types=set())`) | regex + local model |
|
|
130
|
+
| `EMPLOYER` | "I work at the Tim Hortons on Barton" | local model |
|
|
129
131
|
|
|
130
|
-
|
|
132
|
+
Dates are also matched in French, Spanish, Polish and Hebrew ("le 12 mai 1990", "jueves 6 de noviembre", "5 בנובמבר"), and spoken digits in those languages. Code-shaped identifiers without a keyword ("SL-4471920-03", "W123456789", any bare number of 7+ digits) are masked as `ID`.
|
|
133
|
+
|
|
134
|
+
### Optional local model
|
|
135
|
+
|
|
136
|
+
```bash
|
|
137
|
+
pip install "phiproxy[gliner]"
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
```python
|
|
141
|
+
from phiproxy import Masker, RegexDetector
|
|
142
|
+
from phiproxy.detectors.gliner import GlinerDetector
|
|
143
|
+
|
|
144
|
+
masker = Masker(detectors=[RegexDetector(), GlinerDetector()])
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
`GlinerDetector` uses [GLiNER](https://github.com/urchade/GLiNER) with the Apache-2.0 model `urchade/gliner_multi_pii-v1` (about 1.2 GB). It runs in-process: nothing is sent anywhere to detect PHI. The model is downloaded from Hugging Face on first use, so in production download it once and pass its local path as `model=`, or set `HF_HUB_OFFLINE=1`. Model guesses never override the policy: a guess on "Dr. Haddad" does not mask a clinician name that the policy lets through.
|
|
148
|
+
|
|
149
|
+
Presidio is also supported: `pip install phiproxy[presidio]` and add `PresidioDetector()`.
|
|
131
150
|
|
|
132
151
|
## Benchmark
|
|
133
152
|
|
|
134
|
-
`benchmark/` holds
|
|
153
|
+
`benchmark/` holds two synthetic **patient communications** sets: SMS, web chat, call transcripts, emails and referral notes, the kind of text clinics actually send to LLMs. Most existing de-identification work targets clinical notes instead. All people and numbers are invented.
|
|
135
154
|
|
|
136
|
-
v0
|
|
155
|
+
- **v0** (288 messages, English): template-based, written by the detector authors.
|
|
156
|
+
- **blind v1** (420 messages; English, French, Spanish, Polish, Hebrew and mixed; 129 messages with no identifiers but full of lookalikes such as drug doses and embryo grades): written by a separate AI agent that never saw the detector code.
|
|
137
157
|
|
|
138
|
-
|
|
|
139
|
-
|
|
140
|
-
| regex | 61.4% |
|
|
141
|
-
| regex + known entities |
|
|
158
|
+
| recall | v0 | blind v1, first score | blind v1, now |
|
|
159
|
+
|---|---|---|---|
|
|
160
|
+
| regex | 61.4% | 36.6% | 50.5% |
|
|
161
|
+
| regex + known entities | 85.7% | 54.4% | 67.8% |
|
|
162
|
+
| regex + known + local model | 99.1% | 80.1% | **92.2%** |
|
|
142
163
|
|
|
143
|
-
|
|
|
144
|
-
|
|
145
|
-
|
|
|
146
|
-
|
|
|
147
|
-
|
|
|
148
|
-
|
|
|
164
|
+
| blind v1, regex + known + local model | |
|
|
165
|
+
|---|---|
|
|
166
|
+
| by channel | SMS 94%, chat 91%, email 96%, notes 94%, call transcripts 86% |
|
|
167
|
+
| by language | en 91%, fr 94%, es 93%, pl 95%, he 92%, mixed 97% |
|
|
168
|
+
| precision | 86.9% |
|
|
169
|
+
| messages without identifiers where something was masked | 19% |
|
|
149
170
|
|
|
150
171
|
**How to read this honestly:**
|
|
151
|
-
-
|
|
152
|
-
-
|
|
153
|
-
- Known misses
|
|
172
|
+
- The first blind score is the fair one: the detectors were frozen before v1 was scored. "Now" includes fixes made after looking at v1's misses, so it is optimistic. A new blind set will re-test it.
|
|
173
|
+
- v0 overstated recall by about 30 points. Expect real traffic to score lower than v1 too.
|
|
174
|
+
- Known misses: dates spoken in words outside English, some cities, workplaces described in a phrase, and lowercase names that are also words ("joy", "will"). Without the local model, cities and names not in your records are mostly missed.
|
|
175
|
+
- Details, label rules and the full tables are in [`benchmark/README.md`](https://github.com/DecisionTreeTechnology/phiproxy/blob/main/benchmark/README.md).
|
|
154
176
|
|
|
155
177
|
## Deployment shapes
|
|
156
178
|
|
|
@@ -162,9 +184,9 @@ The vault holds real values. Protect Redis like the PHI store it is: private net
|
|
|
162
184
|
|
|
163
185
|
## Roadmap
|
|
164
186
|
|
|
165
|
-
-
|
|
166
|
-
-
|
|
167
|
-
-
|
|
187
|
+
- A benchmark set written by clinic staff, and a second blind AI-written set to re-test the current numbers
|
|
188
|
+
- Gazetteer for cities the model misses; spoken dates in French, Spanish, Polish and Hebrew
|
|
189
|
+
- Smaller and faster local models (distilled or ONNX)
|
|
168
190
|
- HTTP sidecar; Anthropic `/v1/messages` streaming
|
|
169
191
|
- Masked logging hook for LiteLLM
|
|
170
192
|
|
|
@@ -0,0 +1,62 @@
|
|
|
1
|
+
# Patient-communications PHI benchmark
|
|
2
|
+
|
|
3
|
+
Labelled sets of the kind of text clinics send to LLMs: patient SMS, web chat, phone-call transcripts, emails and referral or staff notes. Most open de-identification datasets cover clinical notes; these cover patient communications. Every person and number in them is invented.
|
|
4
|
+
|
|
5
|
+
## The two sets
|
|
6
|
+
|
|
7
|
+
| set | messages | labelled identifiers | languages | written by |
|
|
8
|
+
|---|---|---|---|---|
|
|
9
|
+
| `patient_comms_v0.jsonl` | 288 | 560 | English (one French template) | generated from templates by the detector authors |
|
|
10
|
+
| `patient_comms_blind_v1.jsonl` | 420 (129 with no identifiers) | 664 + 57 clinician names | en 260, fr 50, es 40, pl 30, he 20, mixed 20 | written blind, see below |
|
|
11
|
+
|
|
12
|
+
**How v1 was written.** v1 was written by a separate AI agent (Claude) that was given only a brief: the label rules, the channels, the languages, realistic mess (typos, lowercase names, spoken numbers, spelled-out names) and a list of things that look like identifiers but are not (drug doses, embryo grades, cycle days, billing codes). It was told not to read the detector code, the tests or v0, and it wrote every message from scratch. So v1 is independent of the detector code, but it is not independent of AI-written text. A set written by clinic staff is still the goal.
|
|
13
|
+
|
|
14
|
+
**How v1 was scored.** The local-model detector was tuned on v0 only. Its settings were committed (082c2e1) and v1 was then scored once. Those first scores are kept in `results_blind_v1_frozen.json`. After that, v1 was used to find and fix gaps, so the current v1 numbers are optimistic in the same way v0's always were. The next blind set will re-test them.
|
|
15
|
+
|
|
16
|
+
| recall | v0 (current) | v1 first blind score | v1 after fixes from v1 |
|
|
17
|
+
|---|---|---|---|
|
|
18
|
+
| regex | 61.4% | 36.6% | 50.5% |
|
|
19
|
+
| regex + known entities | 85.7% | 54.4% | 67.8% |
|
|
20
|
+
| regex + local model | 98.2% | 78.2% | 89.9% |
|
|
21
|
+
| regex + known + local model | 99.1% | 80.1% | 92.2% |
|
|
22
|
+
|
|
23
|
+
The 30-point gap between v0 and v1 is the honest lesson: a set written by the people who wrote the detectors flatters them.
|
|
24
|
+
|
|
25
|
+
## Files
|
|
26
|
+
|
|
27
|
+
- `generate.py`: deterministic generator (fixed seed). It writes `data/patient_comms_v0.jsonl`.
|
|
28
|
+
- `blind/blind_v1.source.jsonl`: v1 as written, with inline labels (`[[NAME|jess]]`). `blind/convert.py` turns it into `data/patient_comms_blind_v1.jsonl`.
|
|
29
|
+
- `score.py`: scores every available detector configuration and prints recall by type, channel and language. The local-model configurations appear when `phiproxy[gliner]` is installed.
|
|
30
|
+
- `results_v0.json`, `results_blind_v1.json`, `results_blind_v1_frozen.json`: scores for the built-in configurations.
|
|
31
|
+
|
|
32
|
+
Run `python benchmark/score.py` for v0 and `python benchmark/score.py --data benchmark/data/patient_comms_blind_v1.jsonl` for v1.
|
|
33
|
+
|
|
34
|
+
## Record format
|
|
35
|
+
|
|
36
|
+
```json
|
|
37
|
+
{
|
|
38
|
+
"id": "sms-00-3",
|
|
39
|
+
"channel": "sms",
|
|
40
|
+
"text": "Hi it's Priya, can I move my bloodwork to Oct 3? My cell is 647-555-0112",
|
|
41
|
+
"spans": [{"start": 8, "end": 13, "type": "NAME"}, ...],
|
|
42
|
+
"known": {"names": ["Priya Natarajan"], "phones": ["647-555-0112"], ...}
|
|
43
|
+
}
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
`known` is what the clinic system would already hold about the patient. It lets the benchmark measure the known-entity layer.
|
|
47
|
+
|
|
48
|
+
## Scoring rules
|
|
49
|
+
|
|
50
|
+
- A labelled span is **caught** when detected spans cover at least 80% of its characters. The detected type does not have to match, because catching a phone number as an `ID` still protects it.
|
|
51
|
+
- Scoring uses the default policy, under which clinician names pass through. Clinician names (`PROVIDER`) are therefore not counted in recall, and masking one counts as a false positive.
|
|
52
|
+
- A detected span is **correct** when it overlaps a labelled identifier. **Clean messages masked** is the share of messages without identifiers in which something was masked anyway.
|
|
53
|
+
- What counts as an identifier follows HIPAA Safe Harbor (45 CFR 164.514(b)(2)): names of patients and relatives, all date elements except year, ages over 89, geographic units smaller than a state or province (street, city, postal code), contact details, and record, account, plan and other identifying numbers. Workplaces are labelled too because they can identify a person.
|
|
54
|
+
|
|
55
|
+
## Known limits
|
|
56
|
+
|
|
57
|
+
- v0 is template-based and was written by the detector authors, so its scores are optimistic.
|
|
58
|
+
- v1 is AI-written. Real messages are messier still, and the labels have a few judgment calls: the author labelled some weekdays ("Sunday") and "yesterday" as dates, left some cities unlabelled, mostly where they are the clinic's location ("are you a clinic in Hamilton or only Toronto?"), left clinic holiday dates unlabelled, and did not label a few spelled-out names. These are left as written.
|
|
59
|
+
- Still missed on v1: dates spoken in words in French, Spanish, Polish and Hebrew, some cities the model does not know ("Hamilton", "kitchener"), workplaces described in a phrase ("the canada post depot on lakeshore"), and lowercase names that are also words ("joy", "will").
|
|
60
|
+
- With the local model, something is masked in 19% of messages that contain no identifiers (drug names such as Gonal-f, staff first names, holidays, phone extensions, and role words such as "Paciente"). That costs answer quality, not privacy.
|
|
61
|
+
|
|
62
|
+
Contributions are welcome, especially messages written by people who work in clinics (see ../CONTRIBUTING.md). All data must be synthetic.
|