ALOTT-MEDS 0.0.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- alott_meds-0.0.1/LICENSE +21 -0
- alott_meds-0.0.1/PKG-INFO +296 -0
- alott_meds-0.0.1/README.md +280 -0
- alott_meds-0.0.1/pyproject.toml +74 -0
- alott_meds-0.0.1/setup.cfg +4 -0
- alott_meds-0.0.1/src/ALOTT_MEDS/__init__.py +12 -0
- alott_meds-0.0.1/src/ALOTT_MEDS/event_configs.yaml +243 -0
- alott_meds-0.0.1/src/ALOTT_MEDS.egg-info/PKG-INFO +296 -0
- alott_meds-0.0.1/src/ALOTT_MEDS.egg-info/SOURCES.txt +11 -0
- alott_meds-0.0.1/src/ALOTT_MEDS.egg-info/dependency_links.txt +1 -0
- alott_meds-0.0.1/src/ALOTT_MEDS.egg-info/entry_points.txt +2 -0
- alott_meds-0.0.1/src/ALOTT_MEDS.egg-info/requires.txt +1 -0
- alott_meds-0.0.1/src/ALOTT_MEDS.egg-info/top_level.txt +1 -0
alott_meds-0.0.1/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2025 Robin van de Water
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,296 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: ALOTT-MEDS
|
|
3
|
+
Version: 0.0.1
|
|
4
|
+
Summary: An ETL pipeline to extract the ALOTT (Carescape) EMR data into the MEDS format.
|
|
5
|
+
Author-email: Robin van de Water <rpv2114@cumc.columbia.edu>
|
|
6
|
+
Project-URL: Homepage, https://github.com/rvandewater/ALOTT_MEDS
|
|
7
|
+
Project-URL: Issues, https://github.com/rvandewater/ALOTT_MEDS/issues
|
|
8
|
+
Classifier: Programming Language :: Python :: 3
|
|
9
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
10
|
+
Classifier: Operating System :: OS Independent
|
|
11
|
+
Requires-Python: <3.14,>=3.11.4
|
|
12
|
+
Description-Content-Type: text/markdown
|
|
13
|
+
License-File: LICENSE
|
|
14
|
+
Requires-Dist: meds-extract[download]<0.8,>=0.7.1
|
|
15
|
+
Dynamic: license-file
|
|
16
|
+
|
|
17
|
+
# ALOTT-MEDS
|
|
18
|
+
|
|
19
|
+
<!-- Placeholders until available: the PyPI badges resolve once ALOTT-MEDS is published; replace
|
|
20
|
+
CODECOV_TOKEN_PLACEHOLDER with the repo's Codecov badge token and zenodo.XXXXXXX with its Zenodo DOI. -->
|
|
21
|
+
|
|
22
|
+
[](https://pypi.org/project/ALOTT-MEDS/)
|
|
23
|
+
[](https://codecov.io/gh/rvandewater/ALOTT_MEDS)
|
|
24
|
+
[](https://github.com/rvandewater/ALOTT_MEDS/actions/workflows/synthetic-etl.yaml)
|
|
25
|
+
[](https://github.com/rvandewater/ALOTT_MEDS/actions/workflows/code-quality-main.yaml)
|
|
26
|
+

|
|
27
|
+

|
|
28
|
+
[](https://github.com/rvandewater/ALOTT_MEDS#license)
|
|
29
|
+
[](https://github.com/rvandewater/ALOTT_MEDS/pulls)
|
|
30
|
+
[](https://github.com/rvandewater/ALOTT_MEDS/graphs/contributors)
|
|
31
|
+
[](https://doi.org/10.5281/zenodo.XXXXXXX)
|
|
32
|
+
|
|
33
|
+
ALOTT (ALarms, Outcomes Telemetry with Timing) is a pilot bedside-EMR database from two
|
|
34
|
+
hospitals, collected to support algorithms that predict emergency response team events. It
|
|
35
|
+
pairs bedside telemetry from 270 beds (alarms, monitor measurements and waveforms) with
|
|
36
|
+
electronic medical record data: admissions, diagnoses, ICU stays and bed moves, labs,
|
|
37
|
+
orders, medications and nursing flowsheets.
|
|
38
|
+
This pipeline extracts the EMR tables of ALOTT (from PhysioNet,
|
|
39
|
+
https://physionet.org/content/alott/) into the
|
|
40
|
+
[MEDS](https://github.com/Medical-Event-Data-Standard/meds) format. It is built the same way
|
|
41
|
+
as [MIMIC_IV_MEDS](https://github.com/Medical-Event-Data-Standard/MIMIC_IV_MEDS): one
|
|
42
|
+
declarative [MEDS-Extract](https://github.com/mmcdermott/MEDS_extract) spec, no Python.
|
|
43
|
+
|
|
44
|
+
## Usage:
|
|
45
|
+
|
|
46
|
+
```bash
|
|
47
|
+
pip install git+https://github.com/rvandewater/ALOTT_MEDS
|
|
48
|
+
export DATASET_DOWNLOAD_USERNAME=$PHYSIONET_USERNAME
|
|
49
|
+
export DATASET_DOWNLOAD_PASSWORD=$PHYSIONET_PASSWORD
|
|
50
|
+
|
|
51
|
+
meds-extract-run spec=ALOTT output_dir=$MEDS_OUTPUT_DIR
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
That downloads only the `emr-datasets/` folder of the ALOTT 1.0.0 release (the telemetry is
|
|
55
|
+
skipped) into `$MEDS_OUTPUT_DIR/.meds_extract_run/raw_input` and runs the full MEDS-Extract
|
|
56
|
+
pipeline, writing `data/` and `metadata/` to `$MEDS_OUTPUT_DIR`. Files that already exist and
|
|
57
|
+
verify against PhysioNet's checksums are skipped, so an interrupted download resumes. Pass
|
|
58
|
+
`download_dest_dir=$RAW_INPUT_DIR` to keep the raw data somewhere durable.
|
|
59
|
+
|
|
60
|
+
ALOTT is a credentialed release: the PhysioNet account must have signed the ALOTT data use
|
|
61
|
+
agreement.
|
|
62
|
+
|
|
63
|
+
To reuse a download you already have, point `input_dir` at the directory that *contains*
|
|
64
|
+
`emr-datasets/`:
|
|
65
|
+
|
|
66
|
+
```bash
|
|
67
|
+
meds-extract-run spec=ALOTT output_dir=$MEDS_OUTPUT_DIR do_download=false input_dir=$RAW_INPUT_DIR
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
> [!NOTE]
|
|
71
|
+
> The folder the EMR CSVs sit in on PhysioNet, assumed to be `emr-datasets/`, has not been
|
|
72
|
+
> checked against a download. If your copy is a flat folder of the ten CSVs
|
|
73
|
+
> (`admission.csv`, `labs.csv`, ...), point `input_dir` at a directory holding an
|
|
74
|
+
> `emr-datasets` link to it.
|
|
75
|
+
|
|
76
|
+
Run `meds-extract-run --help` for the full set of arguments.
|
|
77
|
+
|
|
78
|
+
## Configuration
|
|
79
|
+
|
|
80
|
+
The whole pipeline is one file,
|
|
81
|
+
[`src/ALOTT_MEDS/event_configs.yaml`](https://github.com/rvandewater/ALOTT_MEDS/blob/main/src/ALOTT_MEDS/event_configs.yaml), written in MESSY
|
|
82
|
+
(MEDS-Extract Specification Syntax YAML) and registered under the `MEDS_extract.pipelines`
|
|
83
|
+
entry-point group as `ALOTT`. That registration is what makes `spec=ALOTT` work from anywhere
|
|
84
|
+
and what stamps the output's `metadata/dataset.json` with
|
|
85
|
+
`<ALOTT release>:<this package's version>`. The file holds:
|
|
86
|
+
|
|
87
|
+
- a `sources:` block pointing at the PhysioNet release, restricted to `emr-datasets/*`;
|
|
88
|
+
- an `etl:` block with the run's few knobs;
|
|
89
|
+
- one block per raw EMR table describing the events extracted from it.
|
|
90
|
+
|
|
91
|
+
### Timestamps
|
|
92
|
+
|
|
93
|
+
ALOTT shifts each patient's dates by a random -180..180 days but keeps the time of day. Most
|
|
94
|
+
tables store each timestamp as one shifted `%Y-%m-%d %H:%M:%S` column. Labs and orders carry
|
|
95
|
+
only a shifted **date**, and medication start and end times are always `00:00:00`, i.e. dates
|
|
96
|
+
too. Those events are placed at the **end of that day (23:59:59)**: midnight would make a
|
|
97
|
+
result or a new medication visible before it existed, which leaks the future into any
|
|
98
|
+
prediction made later that day. Same-day lab orders and results therefore share a time.
|
|
99
|
+
Medication administrations keep their recorded time of day. A row with a null timestamp
|
|
100
|
+
produces no event.
|
|
101
|
+
|
|
102
|
+
Absolute dates are shifted per patient, so only differences *within* a subject are
|
|
103
|
+
meaningful.
|
|
104
|
+
|
|
105
|
+
### IDs
|
|
106
|
+
|
|
107
|
+
`aMRN` is the MEDS `subject_id`; the encounter key `aCSN` is carried as `hadm_id`. Lab and
|
|
108
|
+
order results use the resulting encounter, and orders the ordering one. The release
|
|
109
|
+
writes the string `nan` for an unresulted encounter; it is cast leniently, so it becomes a
|
|
110
|
+
null and `hadm_id` stays `Int64`.
|
|
111
|
+
|
|
112
|
+
### Codes
|
|
113
|
+
|
|
114
|
+
Every code is `FAMILY//...`, from general to specific, so a prefix selects a group (all
|
|
115
|
+
labs, one lab panel, one drug). Where a table records both an order and its outcome, the
|
|
116
|
+
second component says which: `ORDERED` or `RESULT`, `START` or `STOP`, or `MAR`
|
|
117
|
+
(Medication Administration Record: the bedside log of each dose given, held or refused).
|
|
118
|
+
`UNK` stands for a null component that must not drop the row (see
|
|
119
|
+
[Null handling](#null-handling)).
|
|
120
|
+
|
|
121
|
+
| Code | Table | Time | Value and extra columns |
|
|
122
|
+
| ------------------------------------------------ | ------------ | --------------------- | -------------------------------------------------------------------------------------- |
|
|
123
|
+
| `HOSPITAL_ADMISSION//<admission type>//<source>` | admission | admission | reason for visit in `text_value` |
|
|
124
|
+
| `HOSPITAL_DISCHARGE//<destination>` | admission | discharge | |
|
|
125
|
+
| `DIAGNOSIS//ICD//10//<ICD-10-CM code>` | admission | discharge | diagnosis text; `diagnosis_rank` `PRIMARY` or `SECONDARY` |
|
|
126
|
+
| `<FIELD>//<value>` | demographics | static | `FIELD` is `GENDER`, `MARITAL_STATUS`, `RACE`, `ETHNICITY`, `LANGUAGE` or `BLOOD_TYPE` |
|
|
127
|
+
| `ALLERGY//<allergen>` | allergy | static | reaction in `text_value` |
|
|
128
|
+
| `FLOWSHEET//<FLO_MEAS_ID>` | flowsheet | recorded | reading or score; `flow_type` |
|
|
129
|
+
| `ICU_ADMISSION`, `ICU_DISCHARGE` | icu | start, end | |
|
|
130
|
+
| `TRANSFER` | location | start | |
|
|
131
|
+
| `MEDICATION//START//<drug>` | medications | start date, 23:59:59 | ordered dose (raw text if a range); `dose_unit`, `frequency` |
|
|
132
|
+
| `MEDICATION//STOP//<drug>` | medications | end date, 23:59:59 | |
|
|
133
|
+
| `MEDICATION//MAR//<drug>//<action>` | medications | administration | given dose; `dose_unit`, `route` |
|
|
134
|
+
| `LAB//ORDERED//<PROC_ID>` | labs | order date, 23:59:59 | |
|
|
135
|
+
| `LAB//RESULT//<PROC_ID>//<component>` | labs | result date, 23:59:59 | result; `order_status` |
|
|
136
|
+
| `ORDER//ORDERED//<order type>//<PROC_ID>` | orders | order date, 23:59:59 | `order_status` |
|
|
137
|
+
| `ORDER//RESULT//<order type>//<PROC_ID>` | orders | result date, 23:59:59 | result; `order_status` |
|
|
138
|
+
|
|
139
|
+
`codes.parquet` carries a description for flowsheet, lab and order codes, taken from the
|
|
140
|
+
release's display names.
|
|
141
|
+
|
|
142
|
+
### Null handling
|
|
143
|
+
|
|
144
|
+
A null component of a composite code drops the row under MEDS-Extract 0.7. Components are
|
|
145
|
+
therefore coalesced with `?? 'UNK'` wherever a null must not drop the row (admission type and
|
|
146
|
+
source, discharge destination, lab component, order type, medication administration action).
|
|
147
|
+
|
|
148
|
+
The rest are deliberately left uncoalesced. In the release, the code columns of
|
|
149
|
+
flowsheets, labs, orders and allergies are never null, and `MEDICATION` is null in 10 of
|
|
150
|
+
12,698,867 rows, so a coalesce would add nothing. A missing ICD-10 code or demographic
|
|
151
|
+
produces no event rather than an `UNK` category, because a missing value is not an observed
|
|
152
|
+
one.
|
|
153
|
+
|
|
154
|
+
### Diagnoses
|
|
155
|
+
|
|
156
|
+
The primary and secondary ICD-10 codes on each visit become `DIAGNOSIS//ICD//10//<code>`
|
|
157
|
+
events at discharge time, with `diagnosis_rank` set to `PRIMARY` or `SECONDARY`. About 4% of
|
|
158
|
+
cells list up to four codes separated by a comma and a space; each code becomes its own event, carrying the
|
|
159
|
+
cell's full diagnosis text. No
|
|
160
|
+
`parent_codes` metadata is attached: the same code occurs as both primary and secondary, and
|
|
161
|
+
MEDS-Extract rejects one code produced by two code templates.
|
|
162
|
+
`DIAGNOSIS//ICD//10//<code>` maps to `ICD10CM/<code>` directly.
|
|
163
|
+
|
|
164
|
+
### Other tables
|
|
165
|
+
|
|
166
|
+
- **ICU stays and bed moves** carry only their start and end times in the release (no unit,
|
|
167
|
+
room or bed), so they become bare `ICU_ADMISSION`, `ICU_DISCHARGE` and `TRANSFER`
|
|
168
|
+
events.
|
|
169
|
+
- **Demographics and allergies** carry no time in the release and are static events.
|
|
170
|
+
- **Flowsheet, lab and order-result values** are free text. The numeric reading is kept where
|
|
171
|
+
it parses; scored flowsheet items charted as `<score>--><label>` (e.g. `-1-->Drowsy`) keep
|
|
172
|
+
their score. The raw string goes to `text_value` unless it is just the number. Epic's
|
|
173
|
+
nonnumeric sentinel `9999999` never becomes a numeric value.
|
|
174
|
+
- **Medications:** each row of the release is one administration and repeats its order's
|
|
175
|
+
start and end, so an order yields one `START`, one `STOP` and one `MAR` event per
|
|
176
|
+
administration.
|
|
177
|
+
- **Orders** hold every order type except labs; roughly half carry a single result.
|
|
178
|
+
- **Birth and death:** the release has no birth date, age or death time, so there are no
|
|
179
|
+
`MEDS_BIRTH` or `MEDS_DEATH` events.
|
|
180
|
+
- **Not extracted:** the telemetry (alarms, 2-second measurements, waveforms; per-admission
|
|
181
|
+
files keyed by file name, far larger than the EMR), `social_hx` (no event times and
|
|
182
|
+
unlabeled Epic category codes), and the medication order date.
|
|
183
|
+
|
|
184
|
+
### Validation against the release
|
|
185
|
+
|
|
186
|
+
The spec was first written from the authors' release code
|
|
187
|
+
([Carescape](https://github.com/johnclawrence/Carescape)) and did not run on the published
|
|
188
|
+
files. Profiling the 1.0.0 EMR CSVs (value *shapes* and null counts only, never values)
|
|
189
|
+
showed what had to change:
|
|
190
|
+
|
|
191
|
+
- Tables are lowercase (`admission`, `labs`, ...) rather than `Patient`, `Labs`, ...; the
|
|
192
|
+
keys are `aMRN` and `aCSN`, not `csID` and `CS_PAT_ENC_CSN_ID`.
|
|
193
|
+
- Each timestamp is one already-shifted column rather than a date plus an `HH:MM:SS`
|
|
194
|
+
column. Labs and orders keep only the date (see [Timestamps](#timestamps)).
|
|
195
|
+
- `icu` and `location` have no department, event type, room or bed columns.
|
|
196
|
+
- `resulting_aCSN` holds the string `nan` for unresulted rows.
|
|
197
|
+
- 1,536 ICD-10 codes occur as both primary and secondary diagnoses, and some cells list
|
|
198
|
+
several codes (see [Diagnoses](#diagnoses)). The synthetic test covers both.
|
|
199
|
+
- Medication start and end times are always midnight, and `orders` carries results of its
|
|
200
|
+
own (it holds no lab orders).
|
|
201
|
+
|
|
202
|
+
All ten CSVs are valid UTF-8. On the full 1.0.0 EMR tables (6.3 GB of CSVs read),
|
|
203
|
+
`meds-extract-run` with MEDS-Extract 0.7.1 finishes in about 90 seconds on a laptop with about
|
|
204
|
+
6.5 GB peak memory, producing 52.6M events for 18,472 subjects. Fewer lab, order and
|
|
205
|
+
medication start/stop events come out than raw rows because identical events are merged: lab
|
|
206
|
+
and order times have day resolution, and the medication table repeats an order's start and
|
|
207
|
+
end on every administration row.
|
|
208
|
+
|
|
209
|
+
## MEDS-transforms settings
|
|
210
|
+
|
|
211
|
+
If you want to convert a large dataset, you can parallelize the MEDS-transforms stages
|
|
212
|
+
(the MEDS-transformation step that takes the longest).
|
|
213
|
+
|
|
214
|
+
Using local parallelization with the `hydra-joblib-launcher` package:
|
|
215
|
+
|
|
216
|
+
```bash
|
|
217
|
+
pip install hydra-joblib-launcher --upgrade
|
|
218
|
+
```
|
|
219
|
+
|
|
220
|
+
Then write a stage-runner file and pass it to the run:
|
|
221
|
+
|
|
222
|
+
```yaml
|
|
223
|
+
# runner.yaml
|
|
224
|
+
parallelize:
|
|
225
|
+
n_workers: 8
|
|
226
|
+
launcher: joblib
|
|
227
|
+
```
|
|
228
|
+
|
|
229
|
+
```bash
|
|
230
|
+
meds-extract-run spec=ALOTT output_dir=$MEDS_OUTPUT_DIR stage_runner_fp=runner.yaml
|
|
231
|
+
```
|
|
232
|
+
|
|
233
|
+
The number of subjects per shard is `n_subjects_per_shard` in the spec's `etl:` block
|
|
234
|
+
(1000 by default); raise it to cut the parallelization overhead.
|
|
235
|
+
|
|
236
|
+
## Development
|
|
237
|
+
|
|
238
|
+
```bash
|
|
239
|
+
uv sync --group dev
|
|
240
|
+
uv run pytest tests/
|
|
241
|
+
uv run pre-commit run --all-files
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
A release bumps the version in `pyproject.toml` with `uv version --bump`, commits it and pushes
|
|
245
|
+
a tag of that exact version, which publishes to PyPI and creates the GitHub release. See [`RELEASING.md`](https://github.com/rvandewater/ALOTT_MEDS/blob/main/RELEASING.md) for choosing the
|
|
246
|
+
version, the one-time PyPI setup and recovering from a failed release.
|
|
247
|
+
|
|
248
|
+
ALOTT has no open demo, so CI runs the spec end to end over a small **synthetic** input
|
|
249
|
+
generated in [`tests/e2e_synthetic_test.py`](https://github.com/rvandewater/ALOTT_MEDS/blob/main/tests/e2e_synthetic_test.py). That checks the
|
|
250
|
+
spec's mechanics, not its agreement with the real release. Never commit raw ALOTT data or
|
|
251
|
+
MEDS output derived from it.
|
|
252
|
+
|
|
253
|
+
## Citation
|
|
254
|
+
|
|
255
|
+
Please cite the following original dataset when using ALOTT:
|
|
256
|
+
|
|
257
|
+
```bibtex
|
|
258
|
+
@article{PhysioNet-alott-1.0.0,
|
|
259
|
+
author = {Lawrence, John and Rayo, Mike and Huerta, Timothy},
|
|
260
|
+
title = {{ALarms, Outcomes Telemetry with Timing (ALOTT): a Bedside-EMR Database}},
|
|
261
|
+
journal = {{PhysioNet}},
|
|
262
|
+
year = {2025},
|
|
263
|
+
month = mar,
|
|
264
|
+
note = {Version 1.0.0},
|
|
265
|
+
doi = {10.13026/sbq5-dy17},
|
|
266
|
+
url = {https://doi.org/10.13026/sbq5-dy17}
|
|
267
|
+
}
|
|
268
|
+
```
|
|
269
|
+
|
|
270
|
+
Also cite the ETL framework used to extract the dataset:
|
|
271
|
+
|
|
272
|
+
```bibtex
|
|
273
|
+
@software{van_de_Water_ALOTT_MEDS_ETL_2025,
|
|
274
|
+
author = {van de Water, Robin Philippus},
|
|
275
|
+
license = {MIT},
|
|
276
|
+
month = apr,
|
|
277
|
+
title = {{ALOTT\_MEDS ETL}},
|
|
278
|
+
url = {https://github.com/rvandewater/ALOTT_MEDS},
|
|
279
|
+
year = {2025}
|
|
280
|
+
}
|
|
281
|
+
```
|
|
282
|
+
|
|
283
|
+
For citing MEDS in general:
|
|
284
|
+
|
|
285
|
+
```bibtex
|
|
286
|
+
@article{mcdermott2026meds,
|
|
287
|
+
title={MEDS—An Emerging Data Standard and Ecosystem for Health AI Research},
|
|
288
|
+
author={McDermott, Matthew BA and Steinberg, Ethan and Fries, Jason A and van de Water, Robin P and Pang, Chao and Rockenschaub, Patrick and Renc, Pawel and Oh, Jungwoo and Stankevi{\v{c}}i{\=u}t{\.e}, Kamil{\.e} and Xu, Justin and others},
|
|
289
|
+
journal={NEJM AI},
|
|
290
|
+
volume={3},
|
|
291
|
+
number={6},
|
|
292
|
+
pages={AIra2501253},
|
|
293
|
+
year={2026},
|
|
294
|
+
publisher={Massachusetts Medical Society}
|
|
295
|
+
}
|
|
296
|
+
```
|
|
@@ -0,0 +1,280 @@
|
|
|
1
|
+
# ALOTT-MEDS
|
|
2
|
+
|
|
3
|
+
<!-- Placeholders until available: the PyPI badges resolve once ALOTT-MEDS is published; replace
|
|
4
|
+
CODECOV_TOKEN_PLACEHOLDER with the repo's Codecov badge token and zenodo.XXXXXXX with its Zenodo DOI. -->
|
|
5
|
+
|
|
6
|
+
[](https://pypi.org/project/ALOTT-MEDS/)
|
|
7
|
+
[](https://codecov.io/gh/rvandewater/ALOTT_MEDS)
|
|
8
|
+
[](https://github.com/rvandewater/ALOTT_MEDS/actions/workflows/synthetic-etl.yaml)
|
|
9
|
+
[](https://github.com/rvandewater/ALOTT_MEDS/actions/workflows/code-quality-main.yaml)
|
|
10
|
+

|
|
11
|
+

|
|
12
|
+
[](https://github.com/rvandewater/ALOTT_MEDS#license)
|
|
13
|
+
[](https://github.com/rvandewater/ALOTT_MEDS/pulls)
|
|
14
|
+
[](https://github.com/rvandewater/ALOTT_MEDS/graphs/contributors)
|
|
15
|
+
[](https://doi.org/10.5281/zenodo.XXXXXXX)
|
|
16
|
+
|
|
17
|
+
ALOTT (ALarms, Outcomes Telemetry with Timing) is a pilot bedside-EMR database from two
|
|
18
|
+
hospitals, collected to support algorithms that predict emergency response team events. It
|
|
19
|
+
pairs bedside telemetry from 270 beds (alarms, monitor measurements and waveforms) with
|
|
20
|
+
electronic medical record data: admissions, diagnoses, ICU stays and bed moves, labs,
|
|
21
|
+
orders, medications and nursing flowsheets.
|
|
22
|
+
This pipeline extracts the EMR tables of ALOTT (from PhysioNet,
|
|
23
|
+
https://physionet.org/content/alott/) into the
|
|
24
|
+
[MEDS](https://github.com/Medical-Event-Data-Standard/meds) format. It is built the same way
|
|
25
|
+
as [MIMIC_IV_MEDS](https://github.com/Medical-Event-Data-Standard/MIMIC_IV_MEDS): one
|
|
26
|
+
declarative [MEDS-Extract](https://github.com/mmcdermott/MEDS_extract) spec, no Python.
|
|
27
|
+
|
|
28
|
+
## Usage:
|
|
29
|
+
|
|
30
|
+
```bash
|
|
31
|
+
pip install git+https://github.com/rvandewater/ALOTT_MEDS
|
|
32
|
+
export DATASET_DOWNLOAD_USERNAME=$PHYSIONET_USERNAME
|
|
33
|
+
export DATASET_DOWNLOAD_PASSWORD=$PHYSIONET_PASSWORD
|
|
34
|
+
|
|
35
|
+
meds-extract-run spec=ALOTT output_dir=$MEDS_OUTPUT_DIR
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
That downloads only the `emr-datasets/` folder of the ALOTT 1.0.0 release (the telemetry is
|
|
39
|
+
skipped) into `$MEDS_OUTPUT_DIR/.meds_extract_run/raw_input` and runs the full MEDS-Extract
|
|
40
|
+
pipeline, writing `data/` and `metadata/` to `$MEDS_OUTPUT_DIR`. Files that already exist and
|
|
41
|
+
verify against PhysioNet's checksums are skipped, so an interrupted download resumes. Pass
|
|
42
|
+
`download_dest_dir=$RAW_INPUT_DIR` to keep the raw data somewhere durable.
|
|
43
|
+
|
|
44
|
+
ALOTT is a credentialed release: the PhysioNet account must have signed the ALOTT data use
|
|
45
|
+
agreement.
|
|
46
|
+
|
|
47
|
+
To reuse a download you already have, point `input_dir` at the directory that *contains*
|
|
48
|
+
`emr-datasets/`:
|
|
49
|
+
|
|
50
|
+
```bash
|
|
51
|
+
meds-extract-run spec=ALOTT output_dir=$MEDS_OUTPUT_DIR do_download=false input_dir=$RAW_INPUT_DIR
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
> [!NOTE]
|
|
55
|
+
> The folder the EMR CSVs sit in on PhysioNet, assumed to be `emr-datasets/`, has not been
|
|
56
|
+
> checked against a download. If your copy is a flat folder of the ten CSVs
|
|
57
|
+
> (`admission.csv`, `labs.csv`, ...), point `input_dir` at a directory holding an
|
|
58
|
+
> `emr-datasets` link to it.
|
|
59
|
+
|
|
60
|
+
Run `meds-extract-run --help` for the full set of arguments.
|
|
61
|
+
|
|
62
|
+
## Configuration
|
|
63
|
+
|
|
64
|
+
The whole pipeline is one file,
|
|
65
|
+
[`src/ALOTT_MEDS/event_configs.yaml`](https://github.com/rvandewater/ALOTT_MEDS/blob/main/src/ALOTT_MEDS/event_configs.yaml), written in MESSY
|
|
66
|
+
(MEDS-Extract Specification Syntax YAML) and registered under the `MEDS_extract.pipelines`
|
|
67
|
+
entry-point group as `ALOTT`. That registration is what makes `spec=ALOTT` work from anywhere
|
|
68
|
+
and what stamps the output's `metadata/dataset.json` with
|
|
69
|
+
`<ALOTT release>:<this package's version>`. The file holds:
|
|
70
|
+
|
|
71
|
+
- a `sources:` block pointing at the PhysioNet release, restricted to `emr-datasets/*`;
|
|
72
|
+
- an `etl:` block with the run's few knobs;
|
|
73
|
+
- one block per raw EMR table describing the events extracted from it.
|
|
74
|
+
|
|
75
|
+
### Timestamps
|
|
76
|
+
|
|
77
|
+
ALOTT shifts each patient's dates by a random -180..180 days but keeps the time of day. Most
|
|
78
|
+
tables store each timestamp as one shifted `%Y-%m-%d %H:%M:%S` column. Labs and orders carry
|
|
79
|
+
only a shifted **date**, and medication start and end times are always `00:00:00`, i.e. dates
|
|
80
|
+
too. Those events are placed at the **end of that day (23:59:59)**: midnight would make a
|
|
81
|
+
result or a new medication visible before it existed, which leaks the future into any
|
|
82
|
+
prediction made later that day. Same-day lab orders and results therefore share a time.
|
|
83
|
+
Medication administrations keep their recorded time of day. A row with a null timestamp
|
|
84
|
+
produces no event.
|
|
85
|
+
|
|
86
|
+
Absolute dates are shifted per patient, so only differences *within* a subject are
|
|
87
|
+
meaningful.
|
|
88
|
+
|
|
89
|
+
### IDs
|
|
90
|
+
|
|
91
|
+
`aMRN` is the MEDS `subject_id`; the encounter key `aCSN` is carried as `hadm_id`. Lab and
|
|
92
|
+
order results use the resulting encounter, and orders the ordering one. The release
|
|
93
|
+
writes the string `nan` for an unresulted encounter; it is cast leniently, so it becomes a
|
|
94
|
+
null and `hadm_id` stays `Int64`.
|
|
95
|
+
|
|
96
|
+
### Codes
|
|
97
|
+
|
|
98
|
+
Every code is `FAMILY//...`, from general to specific, so a prefix selects a group (all
|
|
99
|
+
labs, one lab panel, one drug). Where a table records both an order and its outcome, the
|
|
100
|
+
second component says which: `ORDERED` or `RESULT`, `START` or `STOP`, or `MAR`
|
|
101
|
+
(Medication Administration Record: the bedside log of each dose given, held or refused).
|
|
102
|
+
`UNK` stands for a null component that must not drop the row (see
|
|
103
|
+
[Null handling](#null-handling)).
|
|
104
|
+
|
|
105
|
+
| Code | Table | Time | Value and extra columns |
|
|
106
|
+
| ------------------------------------------------ | ------------ | --------------------- | -------------------------------------------------------------------------------------- |
|
|
107
|
+
| `HOSPITAL_ADMISSION//<admission type>//<source>` | admission | admission | reason for visit in `text_value` |
|
|
108
|
+
| `HOSPITAL_DISCHARGE//<destination>` | admission | discharge | |
|
|
109
|
+
| `DIAGNOSIS//ICD//10//<ICD-10-CM code>` | admission | discharge | diagnosis text; `diagnosis_rank` `PRIMARY` or `SECONDARY` |
|
|
110
|
+
| `<FIELD>//<value>` | demographics | static | `FIELD` is `GENDER`, `MARITAL_STATUS`, `RACE`, `ETHNICITY`, `LANGUAGE` or `BLOOD_TYPE` |
|
|
111
|
+
| `ALLERGY//<allergen>` | allergy | static | reaction in `text_value` |
|
|
112
|
+
| `FLOWSHEET//<FLO_MEAS_ID>` | flowsheet | recorded | reading or score; `flow_type` |
|
|
113
|
+
| `ICU_ADMISSION`, `ICU_DISCHARGE` | icu | start, end | |
|
|
114
|
+
| `TRANSFER` | location | start | |
|
|
115
|
+
| `MEDICATION//START//<drug>` | medications | start date, 23:59:59 | ordered dose (raw text if a range); `dose_unit`, `frequency` |
|
|
116
|
+
| `MEDICATION//STOP//<drug>` | medications | end date, 23:59:59 | |
|
|
117
|
+
| `MEDICATION//MAR//<drug>//<action>` | medications | administration | given dose; `dose_unit`, `route` |
|
|
118
|
+
| `LAB//ORDERED//<PROC_ID>` | labs | order date, 23:59:59 | |
|
|
119
|
+
| `LAB//RESULT//<PROC_ID>//<component>` | labs | result date, 23:59:59 | result; `order_status` |
|
|
120
|
+
| `ORDER//ORDERED//<order type>//<PROC_ID>` | orders | order date, 23:59:59 | `order_status` |
|
|
121
|
+
| `ORDER//RESULT//<order type>//<PROC_ID>` | orders | result date, 23:59:59 | result; `order_status` |
|
|
122
|
+
|
|
123
|
+
`codes.parquet` carries a description for flowsheet, lab and order codes, taken from the
|
|
124
|
+
release's display names.
|
|
125
|
+
|
|
126
|
+
### Null handling
|
|
127
|
+
|
|
128
|
+
A null component of a composite code drops the row under MEDS-Extract 0.7. Components are
|
|
129
|
+
therefore coalesced with `?? 'UNK'` wherever a null must not drop the row (admission type and
|
|
130
|
+
source, discharge destination, lab component, order type, medication administration action).
|
|
131
|
+
|
|
132
|
+
The rest are deliberately left uncoalesced. In the release, the code columns of
|
|
133
|
+
flowsheets, labs, orders and allergies are never null, and `MEDICATION` is null in 10 of
|
|
134
|
+
12,698,867 rows, so a coalesce would add nothing. A missing ICD-10 code or demographic
|
|
135
|
+
produces no event rather than an `UNK` category, because a missing value is not an observed
|
|
136
|
+
one.
|
|
137
|
+
|
|
138
|
+
### Diagnoses
|
|
139
|
+
|
|
140
|
+
The primary and secondary ICD-10 codes on each visit become `DIAGNOSIS//ICD//10//<code>`
|
|
141
|
+
events at discharge time, with `diagnosis_rank` set to `PRIMARY` or `SECONDARY`. About 4% of
|
|
142
|
+
cells list up to four codes separated by a comma and a space; each code becomes its own event, carrying the
|
|
143
|
+
cell's full diagnosis text. No
|
|
144
|
+
`parent_codes` metadata is attached: the same code occurs as both primary and secondary, and
|
|
145
|
+
MEDS-Extract rejects one code produced by two code templates.
|
|
146
|
+
`DIAGNOSIS//ICD//10//<code>` maps to `ICD10CM/<code>` directly.
|
|
147
|
+
|
|
148
|
+
### Other tables
|
|
149
|
+
|
|
150
|
+
- **ICU stays and bed moves** carry only their start and end times in the release (no unit,
|
|
151
|
+
room or bed), so they become bare `ICU_ADMISSION`, `ICU_DISCHARGE` and `TRANSFER`
|
|
152
|
+
events.
|
|
153
|
+
- **Demographics and allergies** carry no time in the release and are static events.
|
|
154
|
+
- **Flowsheet, lab and order-result values** are free text. The numeric reading is kept where
|
|
155
|
+
it parses; scored flowsheet items charted as `<score>--><label>` (e.g. `-1-->Drowsy`) keep
|
|
156
|
+
their score. The raw string goes to `text_value` unless it is just the number. Epic's
|
|
157
|
+
nonnumeric sentinel `9999999` never becomes a numeric value.
|
|
158
|
+
- **Medications:** each row of the release is one administration and repeats its order's
|
|
159
|
+
start and end, so an order yields one `START`, one `STOP` and one `MAR` event per
|
|
160
|
+
administration.
|
|
161
|
+
- **Orders** hold every order type except labs; roughly half carry a single result.
|
|
162
|
+
- **Birth and death:** the release has no birth date, age or death time, so there are no
|
|
163
|
+
`MEDS_BIRTH` or `MEDS_DEATH` events.
|
|
164
|
+
- **Not extracted:** the telemetry (alarms, 2-second measurements, waveforms; per-admission
|
|
165
|
+
files keyed by file name, far larger than the EMR), `social_hx` (no event times and
|
|
166
|
+
unlabeled Epic category codes), and the medication order date.
|
|
167
|
+
|
|
168
|
+
### Validation against the release
|
|
169
|
+
|
|
170
|
+
The spec was first written from the authors' release code
|
|
171
|
+
([Carescape](https://github.com/johnclawrence/Carescape)) and did not run on the published
|
|
172
|
+
files. Profiling the 1.0.0 EMR CSVs (value *shapes* and null counts only, never values)
|
|
173
|
+
showed what had to change:
|
|
174
|
+
|
|
175
|
+
- Tables are lowercase (`admission`, `labs`, ...) rather than `Patient`, `Labs`, ...; the
|
|
176
|
+
keys are `aMRN` and `aCSN`, not `csID` and `CS_PAT_ENC_CSN_ID`.
|
|
177
|
+
- Each timestamp is one already-shifted column rather than a date plus an `HH:MM:SS`
|
|
178
|
+
column. Labs and orders keep only the date (see [Timestamps](#timestamps)).
|
|
179
|
+
- `icu` and `location` have no department, event type, room or bed columns.
|
|
180
|
+
- `resulting_aCSN` holds the string `nan` for unresulted rows.
|
|
181
|
+
- 1,536 ICD-10 codes occur as both primary and secondary diagnoses, and some cells list
|
|
182
|
+
several codes (see [Diagnoses](#diagnoses)). The synthetic test covers both.
|
|
183
|
+
- Medication start and end times are always midnight, and `orders` carries results of its
|
|
184
|
+
own (it holds no lab orders).
|
|
185
|
+
|
|
186
|
+
All ten CSVs are valid UTF-8. On the full 1.0.0 EMR tables (6.3 GB of CSVs read),
|
|
187
|
+
`meds-extract-run` with MEDS-Extract 0.7.1 finishes in about 90 seconds on a laptop with about
|
|
188
|
+
6.5 GB peak memory, producing 52.6M events for 18,472 subjects. Fewer lab, order and
|
|
189
|
+
medication start/stop events come out than raw rows because identical events are merged: lab
|
|
190
|
+
and order times have day resolution, and the medication table repeats an order's start and
|
|
191
|
+
end on every administration row.
|
|
192
|
+
|
|
193
|
+
## MEDS-transforms settings
|
|
194
|
+
|
|
195
|
+
If you want to convert a large dataset, you can parallelize the MEDS-transforms stages
|
|
196
|
+
(the MEDS-transformation step that takes the longest).
|
|
197
|
+
|
|
198
|
+
Using local parallelization with the `hydra-joblib-launcher` package:
|
|
199
|
+
|
|
200
|
+
```bash
|
|
201
|
+
pip install hydra-joblib-launcher --upgrade
|
|
202
|
+
```
|
|
203
|
+
|
|
204
|
+
Then write a stage-runner file and pass it to the run:
|
|
205
|
+
|
|
206
|
+
```yaml
|
|
207
|
+
# runner.yaml
|
|
208
|
+
parallelize:
|
|
209
|
+
n_workers: 8
|
|
210
|
+
launcher: joblib
|
|
211
|
+
```
|
|
212
|
+
|
|
213
|
+
```bash
|
|
214
|
+
meds-extract-run spec=ALOTT output_dir=$MEDS_OUTPUT_DIR stage_runner_fp=runner.yaml
|
|
215
|
+
```
|
|
216
|
+
|
|
217
|
+
The number of subjects per shard is `n_subjects_per_shard` in the spec's `etl:` block
|
|
218
|
+
(1000 by default); raise it to cut the parallelization overhead.
|
|
219
|
+
|
|
220
|
+
## Development
|
|
221
|
+
|
|
222
|
+
```bash
|
|
223
|
+
uv sync --group dev
|
|
224
|
+
uv run pytest tests/
|
|
225
|
+
uv run pre-commit run --all-files
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
A release bumps the version in `pyproject.toml` with `uv version --bump`, commits it and pushes
|
|
229
|
+
a tag of that exact version, which publishes to PyPI and creates the GitHub release. See [`RELEASING.md`](https://github.com/rvandewater/ALOTT_MEDS/blob/main/RELEASING.md) for choosing the
|
|
230
|
+
version, the one-time PyPI setup and recovering from a failed release.
|
|
231
|
+
|
|
232
|
+
ALOTT has no open demo, so CI runs the spec end to end over a small **synthetic** input
|
|
233
|
+
generated in [`tests/e2e_synthetic_test.py`](https://github.com/rvandewater/ALOTT_MEDS/blob/main/tests/e2e_synthetic_test.py). That checks the
|
|
234
|
+
spec's mechanics, not its agreement with the real release. Never commit raw ALOTT data or
|
|
235
|
+
MEDS output derived from it.
|
|
236
|
+
|
|
237
|
+
## Citation
|
|
238
|
+
|
|
239
|
+
Please cite the following original dataset when using ALOTT:
|
|
240
|
+
|
|
241
|
+
```bibtex
|
|
242
|
+
@article{PhysioNet-alott-1.0.0,
|
|
243
|
+
author = {Lawrence, John and Rayo, Mike and Huerta, Timothy},
|
|
244
|
+
title = {{ALarms, Outcomes Telemetry with Timing (ALOTT): a Bedside-EMR Database}},
|
|
245
|
+
journal = {{PhysioNet}},
|
|
246
|
+
year = {2025},
|
|
247
|
+
month = mar,
|
|
248
|
+
note = {Version 1.0.0},
|
|
249
|
+
doi = {10.13026/sbq5-dy17},
|
|
250
|
+
url = {https://doi.org/10.13026/sbq5-dy17}
|
|
251
|
+
}
|
|
252
|
+
```
|
|
253
|
+
|
|
254
|
+
Also cite the ETL framework used to extract the dataset:
|
|
255
|
+
|
|
256
|
+
```bibtex
|
|
257
|
+
@software{van_de_Water_ALOTT_MEDS_ETL_2025,
|
|
258
|
+
author = {van de Water, Robin Philippus},
|
|
259
|
+
license = {MIT},
|
|
260
|
+
month = apr,
|
|
261
|
+
title = {{ALOTT\_MEDS ETL}},
|
|
262
|
+
url = {https://github.com/rvandewater/ALOTT_MEDS},
|
|
263
|
+
year = {2025}
|
|
264
|
+
}
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
For citing MEDS in general:
|
|
268
|
+
|
|
269
|
+
```bibtex
|
|
270
|
+
@article{mcdermott2026meds,
|
|
271
|
+
title={MEDS—An Emerging Data Standard and Ecosystem for Health AI Research},
|
|
272
|
+
author={McDermott, Matthew BA and Steinberg, Ethan and Fries, Jason A and van de Water, Robin P and Pang, Chao and Rockenschaub, Patrick and Renc, Pawel and Oh, Jungwoo and Stankevi{\v{c}}i{\=u}t{\.e}, Kamil{\.e} and Xu, Justin and others},
|
|
273
|
+
journal={NEJM AI},
|
|
274
|
+
volume={3},
|
|
275
|
+
number={6},
|
|
276
|
+
pages={AIra2501253},
|
|
277
|
+
year={2026},
|
|
278
|
+
publisher={Massachusetts Medical Society}
|
|
279
|
+
}
|
|
280
|
+
```
|
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
[build-system]
|
|
2
|
+
requires = ["setuptools>=64", "wheel"]
|
|
3
|
+
build-backend = "setuptools.build_meta"
|
|
4
|
+
|
|
5
|
+
[project]
|
|
6
|
+
name = "ALOTT-MEDS"
|
|
7
|
+
# The last released version. A release bumps it with `uv version --bump` and tags the commit with
|
|
8
|
+
# the result (see RELEASING.md); CI refuses a tag that differs from it. 0.0.0 = nothing released yet.
|
|
9
|
+
version = "0.0.1"
|
|
10
|
+
authors = [
|
|
11
|
+
{ name="Robin van de Water", email="rpv2114@cumc.columbia.edu" },
|
|
12
|
+
]
|
|
13
|
+
description = "An ETL pipeline to extract the ALOTT (Carescape) EMR data into the MEDS format."
|
|
14
|
+
readme = "README.md"
|
|
15
|
+
# Mirrors MEDS-Extract's own floor and ceiling; a looser range admits interpreters it
|
|
16
|
+
# cannot be installed on, and uv refuses to resolve at all.
|
|
17
|
+
requires-python = ">=3.11.4,<3.14"
|
|
18
|
+
classifiers = [
|
|
19
|
+
"Programming Language :: Python :: 3",
|
|
20
|
+
"License :: OSI Approved :: MIT License",
|
|
21
|
+
"Operating System :: OS Independent",
|
|
22
|
+
]
|
|
23
|
+
# This package ships no Python: the whole ETL is `event_configs.yaml`, run by
|
|
24
|
+
# MEDS-Extract's own `meds-extract-run`. So there is exactly one dependency — the
|
|
25
|
+
# `[download]` extra adds `meds-extract-download`, which the runner spawns to stage the
|
|
26
|
+
# raw data from PhysioNet. MEDS-transforms, polars and hydra all arrive through it at the
|
|
27
|
+
# versions it fences; duplicating those pins here would only create drift.
|
|
28
|
+
dependencies = [
|
|
29
|
+
"meds-extract[download]>=0.7.1,<0.8",
|
|
30
|
+
]
|
|
31
|
+
|
|
32
|
+
[dependency-groups]
|
|
33
|
+
# `shfmt-py` bundles the `shfmt` binary that the `mdformat-shfmt` pre-commit hook shells
|
|
34
|
+
# out to, keeping local + CI behavior in lock step.
|
|
35
|
+
dev = [
|
|
36
|
+
"pre-commit~=3.8",
|
|
37
|
+
"ruff>=0.11,<1.0",
|
|
38
|
+
"pytest~=9.0",
|
|
39
|
+
"shfmt-py==3.12.0.2",
|
|
40
|
+
]
|
|
41
|
+
|
|
42
|
+
[project.entry-points."MEDS_extract.pipelines"]
|
|
43
|
+
ALOTT = "ALOTT_MEDS:event_configs.yaml"
|
|
44
|
+
|
|
45
|
+
[project.urls]
|
|
46
|
+
Homepage = "https://github.com/rvandewater/ALOTT_MEDS"
|
|
47
|
+
Issues = "https://github.com/rvandewater/ALOTT_MEDS/issues"
|
|
48
|
+
|
|
49
|
+
[tool.setuptools.package-data]
|
|
50
|
+
# The MESSY spec is the whole ETL. It is not a Python file, so it has to be listed, or installed
|
|
51
|
+
# wheels lack it.
|
|
52
|
+
ALOTT_MEDS = ["event_configs.yaml"]
|
|
53
|
+
|
|
54
|
+
[tool.ruff]
|
|
55
|
+
target-version = "py311"
|
|
56
|
+
line-length = 110
|
|
57
|
+
|
|
58
|
+
[tool.ruff.lint]
|
|
59
|
+
select = [
|
|
60
|
+
"E", # pycodestyle
|
|
61
|
+
"F", # pyflakes
|
|
62
|
+
"I", # isort
|
|
63
|
+
"UP", # pyupgrade
|
|
64
|
+
"B", # flake8-bugbear
|
|
65
|
+
"C4", # flake8-comprehensions
|
|
66
|
+
"RUF", # Ruff-specific rules
|
|
67
|
+
"SIM", # flake8-simplify
|
|
68
|
+
]
|
|
69
|
+
ignore = [
|
|
70
|
+
"SIM108", # ternary operator suggestions
|
|
71
|
+
]
|
|
72
|
+
|
|
73
|
+
[tool.ruff.lint.per-file-ignores]
|
|
74
|
+
"__init__.py" = ["F401"]
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
"""The ALOTT MEDS ETL — a configuration package, not a library.
|
|
2
|
+
|
|
3
|
+
This distribution ships no code. Its entire content is ``ALOTT_MEDS/event_configs.yaml``, a MESSY
|
|
4
|
+
(MEDS-Extract Specification Syntax YAML) document declaring where the raw ALOTT EMR tables come from
|
|
5
|
+
and how every event is extracted from them. ``pyproject.toml`` registers that file with MEDS-Extract's
|
|
6
|
+
``MEDS_extract.pipelines`` entry-point group under the name ``ALOTT``, so the ETL runs as::
|
|
7
|
+
|
|
8
|
+
meds-extract-run spec=ALOTT output_dir=$MEDS_OUTPUT_DIR
|
|
9
|
+
|
|
10
|
+
This package exists so the YAML is addressable as package data — ``importlib.resources.files("ALOTT_MEDS")``
|
|
11
|
+
— from an installed wheel. Nothing here is importable API; deliberately, there is nothing to import.
|
|
12
|
+
"""
|
|
@@ -0,0 +1,243 @@
|
|
|
1
|
+
# The complete ALOTT MEDS ETL. This one file *is* the package: `sources:` says where the raw
|
|
2
|
+
# data comes from, `etl:` supplies the handful of run knobs, and every other top-level key is
|
|
3
|
+
# a raw table whose events are extracted from it. It is registered with `meds-extract-run`
|
|
4
|
+
# under the name `ALOTT` (see this repo's `pyproject.toml`), so the whole ETL is one command:
|
|
5
|
+
#
|
|
6
|
+
# meds-extract-run spec=ALOTT output_dir=$MEDS_OUTPUT_DIR
|
|
7
|
+
#
|
|
8
|
+
# Only the EMR tables (`emr-datasets/`) are extracted. The telemetry folders (alarms,
|
|
9
|
+
# 2-second measurements, WFDB waveforms) are per-admission files whose subject and encounter
|
|
10
|
+
# IDs live in the *file name*, not in a column, and they are several orders of magnitude
|
|
11
|
+
# larger; they are out of scope for this spec.
|
|
12
|
+
#
|
|
13
|
+
# SCHEMA PROVENANCE. The table and column names below were checked against the ten EMR CSVs
|
|
14
|
+
# of the 1.0.0 release (`admission.csv`, `labs.csv`, ...). They differ from the dataset
|
|
15
|
+
# authors' release code (github.com/johnclawrence/Carescape): the published files use
|
|
16
|
+
# `aMRN`/`aCSN` as keys and carry each timestamp in a single already-shifted column. The
|
|
17
|
+
# name of the folder the EMR CSVs sit in on PhysioNet (`emr-datasets/` here) has not been
|
|
18
|
+
# checked against a download.
|
|
19
|
+
#
|
|
20
|
+
# Date shifting: every patient's dates are shifted by a random -180..180 days, time of day is
|
|
21
|
+
# not. Most timestamps are `%Y-%m-%d %H:%M:%S`; labs, orders and medication start/end carry
|
|
22
|
+
# only a shifted *date*, so their events are placed at the end of that day (23:59:59).
|
|
23
|
+
# Midnight would make a result visible before it existed (future leakage for any prediction
|
|
24
|
+
# made later that day).
|
|
25
|
+
|
|
26
|
+
etl:
|
|
27
|
+
dataset_name: ALOTT
|
|
28
|
+
n_subjects_per_shard: 10000
|
|
29
|
+
|
|
30
|
+
# `aMRN` is the de-identified patient key; `aCSN` is the de-identified encounter key, carried
|
|
31
|
+
# as `hadm_id` wherever a table has one.
|
|
32
|
+
_defaults:
|
|
33
|
+
subject_id: $aMRN
|
|
34
|
+
|
|
35
|
+
emr-datasets/admission:
|
|
36
|
+
# One row per hospital visit.
|
|
37
|
+
admission:
|
|
38
|
+
code: f"HOSPITAL_ADMISSION//{$Admission_Type ?? 'UNK'}//{$Admit_Source ?? 'UNK'}"
|
|
39
|
+
time: '$admsnOffset::"%Y-%m-%d %H:%M:%S"'
|
|
40
|
+
hadm_id: $aCSN
|
|
41
|
+
text_value: $REASON_VISIT_NAME
|
|
42
|
+
discharge:
|
|
43
|
+
code: f"HOSPITAL_DISCHARGE//{$Discharge_Destination ?? 'UNK'}"
|
|
44
|
+
time: '$dischOffset::"%Y-%m-%d %H:%M:%S"'
|
|
45
|
+
hadm_id: $aCSN
|
|
46
|
+
# Diagnoses are only known at the end of the visit, so they are timed at discharge (as in
|
|
47
|
+
# MIMIC-IV MEDS). The free-text name rides in `text_value` rather than `description`: it is
|
|
48
|
+
# per-visit text, not guaranteed to be a function of the code. No `parent_codes` metadata:
|
|
49
|
+
# the same ICD code occurs in both columns, and MEDS-Extract refuses a code produced by two
|
|
50
|
+
# code templates. The codes are already dotted ICD-10-CM, so `DIAGNOSIS//ICD//10//<code>`
|
|
51
|
+
# maps to `ICD10CM/<code>` directly.
|
|
52
|
+
#
|
|
53
|
+
# About 4% of cells hold a `, `-separated list of up to four codes. MESSY cannot explode a
|
|
54
|
+
# list into rows, so each position is its own block: `_k` extracts the k-th code and yields
|
|
55
|
+
# no event when the list is shorter. The full diagnosis text rides on every one. The blocks
|
|
56
|
+
# run from the 4th position down because MESSY-synth builds its synthetic value from the
|
|
57
|
+
# first pattern it meets, and only a 4-code list satisfies all four.
|
|
58
|
+
primary_diagnosis_4: &primary_diagnosis
|
|
59
|
+
code: f"DIAGNOSIS//ICD//10//{extract group 1 of /^[^,]+, [^,]+, [^,]+, ([^,]+)/ from $Primary_ICD10}"
|
|
60
|
+
time: '$dischOffset::"%Y-%m-%d %H:%M:%S"'
|
|
61
|
+
hadm_id: $aCSN
|
|
62
|
+
text_value: $Primary_Diagnosis
|
|
63
|
+
diagnosis_rank: "'PRIMARY'"
|
|
64
|
+
primary_diagnosis_3:
|
|
65
|
+
<<: *primary_diagnosis
|
|
66
|
+
code: f"DIAGNOSIS//ICD//10//{extract group 1 of /^[^,]+, [^,]+, ([^,]+)/ from $Primary_ICD10}"
|
|
67
|
+
primary_diagnosis_2:
|
|
68
|
+
<<: *primary_diagnosis
|
|
69
|
+
code: f"DIAGNOSIS//ICD//10//{extract group 1 of /^[^,]+, ([^,]+)/ from $Primary_ICD10}"
|
|
70
|
+
primary_diagnosis:
|
|
71
|
+
<<: *primary_diagnosis
|
|
72
|
+
code: f"DIAGNOSIS//ICD//10//{extract group 1 of /^([^,]+)/ from $Primary_ICD10}"
|
|
73
|
+
secondary_diagnosis_4: &secondary_diagnosis
|
|
74
|
+
code: f"DIAGNOSIS//ICD//10//{extract group 1 of /^[^,]+, [^,]+, [^,]+, ([^,]+)/ from $Secondary_ICD10}"
|
|
75
|
+
time: '$dischOffset::"%Y-%m-%d %H:%M:%S"'
|
|
76
|
+
hadm_id: $aCSN
|
|
77
|
+
text_value: $Secondary_Diagnosis
|
|
78
|
+
diagnosis_rank: "'SECONDARY'"
|
|
79
|
+
secondary_diagnosis_3:
|
|
80
|
+
<<: *secondary_diagnosis
|
|
81
|
+
code: f"DIAGNOSIS//ICD//10//{extract group 1 of /^[^,]+, [^,]+, ([^,]+)/ from $Secondary_ICD10}"
|
|
82
|
+
secondary_diagnosis_2:
|
|
83
|
+
<<: *secondary_diagnosis
|
|
84
|
+
code: f"DIAGNOSIS//ICD//10//{extract group 1 of /^[^,]+, ([^,]+)/ from $Secondary_ICD10}"
|
|
85
|
+
secondary_diagnosis:
|
|
86
|
+
<<: *secondary_diagnosis
|
|
87
|
+
code: f"DIAGNOSIS//ICD//10//{extract group 1 of /^([^,]+)/ from $Secondary_ICD10}"
|
|
88
|
+
|
|
89
|
+
emr-datasets/demographics:
|
|
90
|
+
# Subject-level properties, so static (null time). The table is keyed per encounter, so a
|
|
91
|
+
# subject contributes one row per visit; identical static events are deduplicated
|
|
92
|
+
# downstream. Nulls are deliberately not coalesced: a missing demographic produces no
|
|
93
|
+
# event rather than a `RACE//UNK` code that would read as an observed category.
|
|
94
|
+
gender:
|
|
95
|
+
code: f"GENDER//{$Gender}"
|
|
96
|
+
time: null
|
|
97
|
+
marital_status:
|
|
98
|
+
code: f"MARITAL_STATUS//{$Marital_Status}"
|
|
99
|
+
time: null
|
|
100
|
+
race:
|
|
101
|
+
code: f"RACE//{$Patient_Race}"
|
|
102
|
+
time: null
|
|
103
|
+
ethnicity:
|
|
104
|
+
code: f"ETHNICITY//{$Ethnicity}"
|
|
105
|
+
time: null
|
|
106
|
+
language:
|
|
107
|
+
code: f"LANGUAGE//{$Preferred_Language}"
|
|
108
|
+
time: null
|
|
109
|
+
blood_type:
|
|
110
|
+
code: f"BLOOD_TYPE//{$ABO_RH_Blood_Type}"
|
|
111
|
+
time: null
|
|
112
|
+
|
|
113
|
+
emr-datasets/allergy:
|
|
114
|
+
# The release carries no time for allergies, so they are static.
|
|
115
|
+
allergy:
|
|
116
|
+
code: f"ALLERGY//{$Allergy}"
|
|
117
|
+
time: null
|
|
118
|
+
text_value: $Allergy_Reaction
|
|
119
|
+
|
|
120
|
+
emr-datasets/flowsheet:
|
|
121
|
+
# Provider-charted vitals, risk scores and rapid-response events. `MEAS_VALUE` is free
|
|
122
|
+
# text; `::?float64` keeps the numeric reading where there is one and null otherwise.
|
|
123
|
+
# Scored scale items are charted as `<score>--><label>` (e.g. `-1-->Drowsy`), so their
|
|
124
|
+
# leading score is the numeric reading. The raw string is kept in `text_value` unless it is
|
|
125
|
+
# just the number (MEDS-Extract drops a `text_value` equal to `numeric_value`).
|
|
126
|
+
flowsheet:
|
|
127
|
+
code: f"FLOWSHEET//{$FLO_MEAS_ID}"
|
|
128
|
+
time: '$recordedOffsetDate::"%Y-%m-%d %H:%M:%S"'
|
|
129
|
+
hadm_id: $aCSN
|
|
130
|
+
numeric_value: coalesce($MEAS_VALUE::?float64, (extract group 1 of /^(-?[0-9]+)-->/ from $MEAS_VALUE::str)::?float64)
|
|
131
|
+
text_value: $MEAS_VALUE
|
|
132
|
+
flow_type: $Flow_Type
|
|
133
|
+
_metadata:
|
|
134
|
+
_self:
|
|
135
|
+
description: coalesce($disp_name, $flo_meas_name)
|
|
136
|
+
|
|
137
|
+
emr-datasets/icu:
|
|
138
|
+
# The release keeps only the ICU stay's times, not which unit it was.
|
|
139
|
+
icu_admission:
|
|
140
|
+
code: "'ICU_ADMISSION'"
|
|
141
|
+
time: '$startOffsetDate::"%Y-%m-%d %H:%M:%S"'
|
|
142
|
+
hadm_id: $aCSN
|
|
143
|
+
icu_discharge:
|
|
144
|
+
code: "'ICU_DISCHARGE'"
|
|
145
|
+
time: '$endOffsetDate::"%Y-%m-%d %H:%M:%S"'
|
|
146
|
+
hadm_id: $aCSN
|
|
147
|
+
|
|
148
|
+
emr-datasets/location:
|
|
149
|
+
# Bed-level movement. The release keeps only each stay's start and end, not the department,
|
|
150
|
+
# room or bed, so a transfer is just the moment the patient moved.
|
|
151
|
+
transfer:
|
|
152
|
+
code: "'TRANSFER'"
|
|
153
|
+
time: '$startOffsetDate::"%Y-%m-%d %H:%M:%S"'
|
|
154
|
+
hadm_id: $aCSN
|
|
155
|
+
|
|
156
|
+
emr-datasets/medications:
|
|
157
|
+
# One row per administration, each repeating its order's start and end. Those two are
|
|
158
|
+
# datetimes in form but always 00:00:00 in the release, i.e. dates, so they are placed at
|
|
159
|
+
# the end of the day like labs and orders. The ordered dose is free text (`1-2` for a
|
|
160
|
+
# range); its numeric reading is kept where there is one and the raw string otherwise.
|
|
161
|
+
medication_start:
|
|
162
|
+
code: f"MEDICATION//START//{$MEDICATION}"
|
|
163
|
+
time: '$startOffsetDate::"%Y-%m-%d %H:%M:%S" @ 23:59:59'
|
|
164
|
+
hadm_id: $aCSN
|
|
165
|
+
numeric_value: $HV_DISCRETE_DOSE::?float64
|
|
166
|
+
text_value: $HV_DISCRETE_DOSE
|
|
167
|
+
dose_unit: $HV_DISCRET_DOSE_UNIT
|
|
168
|
+
frequency: $FREQUENCY
|
|
169
|
+
medication_stop:
|
|
170
|
+
code: f"MEDICATION//STOP//{$MEDICATION}"
|
|
171
|
+
time: '$endOffsetDate::"%Y-%m-%d %H:%M:%S" @ 23:59:59'
|
|
172
|
+
hadm_id: $aCSN
|
|
173
|
+
# The Medication Administration Record (MAR), the bedside log of each dose, is the only
|
|
174
|
+
# medication time with a time of day. The recorded action is part of the code, since a held
|
|
175
|
+
# or refused dose is not a given one; a row without an action is kept as `UNK`.
|
|
176
|
+
medication_administration:
|
|
177
|
+
code: f"MEDICATION//MAR//{$MEDICATION}//{$MAR_ACTION ?? 'UNK'}"
|
|
178
|
+
time: '$OffsetMAR_TAKEN_TIME::"%Y-%m-%d %H:%M:%S"'
|
|
179
|
+
hadm_id: $aCSN
|
|
180
|
+
numeric_value: $MAR_DOSE::?float64
|
|
181
|
+
dose_unit: $MAR_DOSE_UNIT
|
|
182
|
+
route: $MAR_ROUTE
|
|
183
|
+
|
|
184
|
+
emr-datasets/labs:
|
|
185
|
+
# Ordering and resulting encounters can differ; the resulting one is the visit the value
|
|
186
|
+
# belongs to. Unresulted rows carry the string `nan` as their resulting encounter, which the
|
|
187
|
+
# lenient cast turns into a null (and keeps `hadm_id` an integer across tables). The end-of-day
|
|
188
|
+
# time is attached with `date @ time` rather than an f-string so MESSY-synth can type the column.
|
|
189
|
+
lab_ordered:
|
|
190
|
+
code: f"LAB//ORDERED//{$PROC_ID}"
|
|
191
|
+
time: '$orderingOffsetDate::"%Y-%m-%d" @ 23:59:59'
|
|
192
|
+
hadm_id: $ordering_aCSN
|
|
193
|
+
_metadata:
|
|
194
|
+
_self:
|
|
195
|
+
description: $DESCRIPTION
|
|
196
|
+
lab_result:
|
|
197
|
+
code: f"LAB//RESULT//{$PROC_ID}//{$COMPONENT_NAME ?? 'UNK'}"
|
|
198
|
+
time: '$resultsOffsetDate::"%Y-%m-%d" @ 23:59:59'
|
|
199
|
+
hadm_id: $resulting_aCSN::?int64
|
|
200
|
+
numeric_value: "$ORD_NUM_VALUE::?float64 if $ORD_NUM_VALUE::?float64 != 9999999"
|
|
201
|
+
text_value: $ORD_VALUE
|
|
202
|
+
order_status: $order_status
|
|
203
|
+
_metadata:
|
|
204
|
+
_self:
|
|
205
|
+
description: f"{$DESCRIPTION} - {$COMPONENT_NAME}"
|
|
206
|
+
|
|
207
|
+
emr-datasets/orders:
|
|
208
|
+
# Every order type except labs. About half the rows carry a single result, with the same
|
|
209
|
+
# `nan` resulting-encounter and 9999999 nonnumeric sentinels as labs.
|
|
210
|
+
order:
|
|
211
|
+
code: f"ORDER//ORDERED//{$Order_type ?? 'UNK'}//{$PROC_ID}"
|
|
212
|
+
time: '$orderingOffsetDate::"%Y-%m-%d" @ 23:59:59'
|
|
213
|
+
hadm_id: $ordering_aCSN
|
|
214
|
+
order_status: $order_status
|
|
215
|
+
_metadata:
|
|
216
|
+
_self:
|
|
217
|
+
description: $DESCRIPTION
|
|
218
|
+
order_result:
|
|
219
|
+
code: f"ORDER//RESULT//{$Order_type ?? 'UNK'}//{$PROC_ID}"
|
|
220
|
+
time: '$resultsOffsetDate::"%Y-%m-%d" @ 23:59:59'
|
|
221
|
+
hadm_id: $resulting_aCSN::?int64
|
|
222
|
+
numeric_value: "$ORD_NUM_VALUE::?float64 if $ORD_NUM_VALUE::?float64 != 9999999"
|
|
223
|
+
text_value: $ORD_VALUE
|
|
224
|
+
order_status: $order_status
|
|
225
|
+
_metadata:
|
|
226
|
+
_self:
|
|
227
|
+
description: $DESCRIPTION
|
|
228
|
+
|
|
229
|
+
# `emr-datasets/social_hx` (tobacco / alcohol / drug use) is not extracted: it has no event
|
|
230
|
+
# time other than quit dates, and its `*_C` columns are Epic category codes whose labels are
|
|
231
|
+
# not in the release.
|
|
232
|
+
|
|
233
|
+
sources:
|
|
234
|
+
# Stamped by `meds-extract-run` into the output's `etl_metadata.dataset_version` as
|
|
235
|
+
# `<release>:<this package's version>`.
|
|
236
|
+
dataset_version: "1.0.0"
|
|
237
|
+
dataset:
|
|
238
|
+
- type: physionet
|
|
239
|
+
base_url: https://physionet.org/files/alott/${sources.dataset_version}
|
|
240
|
+
username: ${oc.env:DATASET_DOWNLOAD_USERNAME}
|
|
241
|
+
password: ${oc.env:DATASET_DOWNLOAD_PASSWORD}
|
|
242
|
+
# Only the EMR tables; the telemetry is out of scope (see the header).
|
|
243
|
+
include: ["emr-datasets/*"]
|
|
@@ -0,0 +1,296 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: ALOTT-MEDS
|
|
3
|
+
Version: 0.0.1
|
|
4
|
+
Summary: An ETL pipeline to extract the ALOTT (Carescape) EMR data into the MEDS format.
|
|
5
|
+
Author-email: Robin van de Water <rpv2114@cumc.columbia.edu>
|
|
6
|
+
Project-URL: Homepage, https://github.com/rvandewater/ALOTT_MEDS
|
|
7
|
+
Project-URL: Issues, https://github.com/rvandewater/ALOTT_MEDS/issues
|
|
8
|
+
Classifier: Programming Language :: Python :: 3
|
|
9
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
10
|
+
Classifier: Operating System :: OS Independent
|
|
11
|
+
Requires-Python: <3.14,>=3.11.4
|
|
12
|
+
Description-Content-Type: text/markdown
|
|
13
|
+
License-File: LICENSE
|
|
14
|
+
Requires-Dist: meds-extract[download]<0.8,>=0.7.1
|
|
15
|
+
Dynamic: license-file
|
|
16
|
+
|
|
17
|
+
# ALOTT-MEDS
|
|
18
|
+
|
|
19
|
+
<!-- Placeholders until available: the PyPI badges resolve once ALOTT-MEDS is published; replace
|
|
20
|
+
CODECOV_TOKEN_PLACEHOLDER with the repo's Codecov badge token and zenodo.XXXXXXX with its Zenodo DOI. -->
|
|
21
|
+
|
|
22
|
+
[](https://pypi.org/project/ALOTT-MEDS/)
|
|
23
|
+
[](https://codecov.io/gh/rvandewater/ALOTT_MEDS)
|
|
24
|
+
[](https://github.com/rvandewater/ALOTT_MEDS/actions/workflows/synthetic-etl.yaml)
|
|
25
|
+
[](https://github.com/rvandewater/ALOTT_MEDS/actions/workflows/code-quality-main.yaml)
|
|
26
|
+

|
|
27
|
+

|
|
28
|
+
[](https://github.com/rvandewater/ALOTT_MEDS#license)
|
|
29
|
+
[](https://github.com/rvandewater/ALOTT_MEDS/pulls)
|
|
30
|
+
[](https://github.com/rvandewater/ALOTT_MEDS/graphs/contributors)
|
|
31
|
+
[](https://doi.org/10.5281/zenodo.XXXXXXX)
|
|
32
|
+
|
|
33
|
+
ALOTT (ALarms, Outcomes Telemetry with Timing) is a pilot bedside-EMR database from two
|
|
34
|
+
hospitals, collected to support algorithms that predict emergency response team events. It
|
|
35
|
+
pairs bedside telemetry from 270 beds (alarms, monitor measurements and waveforms) with
|
|
36
|
+
electronic medical record data: admissions, diagnoses, ICU stays and bed moves, labs,
|
|
37
|
+
orders, medications and nursing flowsheets.
|
|
38
|
+
This pipeline extracts the EMR tables of ALOTT (from PhysioNet,
|
|
39
|
+
https://physionet.org/content/alott/) into the
|
|
40
|
+
[MEDS](https://github.com/Medical-Event-Data-Standard/meds) format. It is built the same way
|
|
41
|
+
as [MIMIC_IV_MEDS](https://github.com/Medical-Event-Data-Standard/MIMIC_IV_MEDS): one
|
|
42
|
+
declarative [MEDS-Extract](https://github.com/mmcdermott/MEDS_extract) spec, no Python.
|
|
43
|
+
|
|
44
|
+
## Usage:
|
|
45
|
+
|
|
46
|
+
```bash
|
|
47
|
+
pip install git+https://github.com/rvandewater/ALOTT_MEDS
|
|
48
|
+
export DATASET_DOWNLOAD_USERNAME=$PHYSIONET_USERNAME
|
|
49
|
+
export DATASET_DOWNLOAD_PASSWORD=$PHYSIONET_PASSWORD
|
|
50
|
+
|
|
51
|
+
meds-extract-run spec=ALOTT output_dir=$MEDS_OUTPUT_DIR
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
That downloads only the `emr-datasets/` folder of the ALOTT 1.0.0 release (the telemetry is
|
|
55
|
+
skipped) into `$MEDS_OUTPUT_DIR/.meds_extract_run/raw_input` and runs the full MEDS-Extract
|
|
56
|
+
pipeline, writing `data/` and `metadata/` to `$MEDS_OUTPUT_DIR`. Files that already exist and
|
|
57
|
+
verify against PhysioNet's checksums are skipped, so an interrupted download resumes. Pass
|
|
58
|
+
`download_dest_dir=$RAW_INPUT_DIR` to keep the raw data somewhere durable.
|
|
59
|
+
|
|
60
|
+
ALOTT is a credentialed release: the PhysioNet account must have signed the ALOTT data use
|
|
61
|
+
agreement.
|
|
62
|
+
|
|
63
|
+
To reuse a download you already have, point `input_dir` at the directory that *contains*
|
|
64
|
+
`emr-datasets/`:
|
|
65
|
+
|
|
66
|
+
```bash
|
|
67
|
+
meds-extract-run spec=ALOTT output_dir=$MEDS_OUTPUT_DIR do_download=false input_dir=$RAW_INPUT_DIR
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
> [!NOTE]
|
|
71
|
+
> The folder the EMR CSVs sit in on PhysioNet, assumed to be `emr-datasets/`, has not been
|
|
72
|
+
> checked against a download. If your copy is a flat folder of the ten CSVs
|
|
73
|
+
> (`admission.csv`, `labs.csv`, ...), point `input_dir` at a directory holding an
|
|
74
|
+
> `emr-datasets` link to it.
|
|
75
|
+
|
|
76
|
+
Run `meds-extract-run --help` for the full set of arguments.
|
|
77
|
+
|
|
78
|
+
## Configuration
|
|
79
|
+
|
|
80
|
+
The whole pipeline is one file,
|
|
81
|
+
[`src/ALOTT_MEDS/event_configs.yaml`](https://github.com/rvandewater/ALOTT_MEDS/blob/main/src/ALOTT_MEDS/event_configs.yaml), written in MESSY
|
|
82
|
+
(MEDS-Extract Specification Syntax YAML) and registered under the `MEDS_extract.pipelines`
|
|
83
|
+
entry-point group as `ALOTT`. That registration is what makes `spec=ALOTT` work from anywhere
|
|
84
|
+
and what stamps the output's `metadata/dataset.json` with
|
|
85
|
+
`<ALOTT release>:<this package's version>`. The file holds:
|
|
86
|
+
|
|
87
|
+
- a `sources:` block pointing at the PhysioNet release, restricted to `emr-datasets/*`;
|
|
88
|
+
- an `etl:` block with the run's few knobs;
|
|
89
|
+
- one block per raw EMR table describing the events extracted from it.
|
|
90
|
+
|
|
91
|
+
### Timestamps
|
|
92
|
+
|
|
93
|
+
ALOTT shifts each patient's dates by a random -180..180 days but keeps the time of day. Most
|
|
94
|
+
tables store each timestamp as one shifted `%Y-%m-%d %H:%M:%S` column. Labs and orders carry
|
|
95
|
+
only a shifted **date**, and medication start and end times are always `00:00:00`, i.e. dates
|
|
96
|
+
too. Those events are placed at the **end of that day (23:59:59)**: midnight would make a
|
|
97
|
+
result or a new medication visible before it existed, which leaks the future into any
|
|
98
|
+
prediction made later that day. Same-day lab orders and results therefore share a time.
|
|
99
|
+
Medication administrations keep their recorded time of day. A row with a null timestamp
|
|
100
|
+
produces no event.
|
|
101
|
+
|
|
102
|
+
Absolute dates are shifted per patient, so only differences *within* a subject are
|
|
103
|
+
meaningful.
|
|
104
|
+
|
|
105
|
+
### IDs
|
|
106
|
+
|
|
107
|
+
`aMRN` is the MEDS `subject_id`; the encounter key `aCSN` is carried as `hadm_id`. Lab and
|
|
108
|
+
order results use the resulting encounter, and orders the ordering one. The release
|
|
109
|
+
writes the string `nan` for an unresulted encounter; it is cast leniently, so it becomes a
|
|
110
|
+
null and `hadm_id` stays `Int64`.
|
|
111
|
+
|
|
112
|
+
### Codes
|
|
113
|
+
|
|
114
|
+
Every code is `FAMILY//...`, from general to specific, so a prefix selects a group (all
|
|
115
|
+
labs, one lab panel, one drug). Where a table records both an order and its outcome, the
|
|
116
|
+
second component says which: `ORDERED` or `RESULT`, `START` or `STOP`, or `MAR`
|
|
117
|
+
(Medication Administration Record: the bedside log of each dose given, held or refused).
|
|
118
|
+
`UNK` stands for a null component that must not drop the row (see
|
|
119
|
+
[Null handling](#null-handling)).
|
|
120
|
+
|
|
121
|
+
| Code | Table | Time | Value and extra columns |
|
|
122
|
+
| ------------------------------------------------ | ------------ | --------------------- | -------------------------------------------------------------------------------------- |
|
|
123
|
+
| `HOSPITAL_ADMISSION//<admission type>//<source>` | admission | admission | reason for visit in `text_value` |
|
|
124
|
+
| `HOSPITAL_DISCHARGE//<destination>` | admission | discharge | |
|
|
125
|
+
| `DIAGNOSIS//ICD//10//<ICD-10-CM code>` | admission | discharge | diagnosis text; `diagnosis_rank` `PRIMARY` or `SECONDARY` |
|
|
126
|
+
| `<FIELD>//<value>` | demographics | static | `FIELD` is `GENDER`, `MARITAL_STATUS`, `RACE`, `ETHNICITY`, `LANGUAGE` or `BLOOD_TYPE` |
|
|
127
|
+
| `ALLERGY//<allergen>` | allergy | static | reaction in `text_value` |
|
|
128
|
+
| `FLOWSHEET//<FLO_MEAS_ID>` | flowsheet | recorded | reading or score; `flow_type` |
|
|
129
|
+
| `ICU_ADMISSION`, `ICU_DISCHARGE` | icu | start, end | |
|
|
130
|
+
| `TRANSFER` | location | start | |
|
|
131
|
+
| `MEDICATION//START//<drug>` | medications | start date, 23:59:59 | ordered dose (raw text if a range); `dose_unit`, `frequency` |
|
|
132
|
+
| `MEDICATION//STOP//<drug>` | medications | end date, 23:59:59 | |
|
|
133
|
+
| `MEDICATION//MAR//<drug>//<action>` | medications | administration | given dose; `dose_unit`, `route` |
|
|
134
|
+
| `LAB//ORDERED//<PROC_ID>` | labs | order date, 23:59:59 | |
|
|
135
|
+
| `LAB//RESULT//<PROC_ID>//<component>` | labs | result date, 23:59:59 | result; `order_status` |
|
|
136
|
+
| `ORDER//ORDERED//<order type>//<PROC_ID>` | orders | order date, 23:59:59 | `order_status` |
|
|
137
|
+
| `ORDER//RESULT//<order type>//<PROC_ID>` | orders | result date, 23:59:59 | result; `order_status` |
|
|
138
|
+
|
|
139
|
+
`codes.parquet` carries a description for flowsheet, lab and order codes, taken from the
|
|
140
|
+
release's display names.
|
|
141
|
+
|
|
142
|
+
### Null handling
|
|
143
|
+
|
|
144
|
+
A null component of a composite code drops the row under MEDS-Extract 0.7. Components are
|
|
145
|
+
therefore coalesced with `?? 'UNK'` wherever a null must not drop the row (admission type and
|
|
146
|
+
source, discharge destination, lab component, order type, medication administration action).
|
|
147
|
+
|
|
148
|
+
The rest are deliberately left uncoalesced. In the release, the code columns of
|
|
149
|
+
flowsheets, labs, orders and allergies are never null, and `MEDICATION` is null in 10 of
|
|
150
|
+
12,698,867 rows, so a coalesce would add nothing. A missing ICD-10 code or demographic
|
|
151
|
+
produces no event rather than an `UNK` category, because a missing value is not an observed
|
|
152
|
+
one.
|
|
153
|
+
|
|
154
|
+
### Diagnoses
|
|
155
|
+
|
|
156
|
+
The primary and secondary ICD-10 codes on each visit become `DIAGNOSIS//ICD//10//<code>`
|
|
157
|
+
events at discharge time, with `diagnosis_rank` set to `PRIMARY` or `SECONDARY`. About 4% of
|
|
158
|
+
cells list up to four codes separated by a comma and a space; each code becomes its own event, carrying the
|
|
159
|
+
cell's full diagnosis text. No
|
|
160
|
+
`parent_codes` metadata is attached: the same code occurs as both primary and secondary, and
|
|
161
|
+
MEDS-Extract rejects one code produced by two code templates.
|
|
162
|
+
`DIAGNOSIS//ICD//10//<code>` maps to `ICD10CM/<code>` directly.
|
|
163
|
+
|
|
164
|
+
### Other tables
|
|
165
|
+
|
|
166
|
+
- **ICU stays and bed moves** carry only their start and end times in the release (no unit,
|
|
167
|
+
room or bed), so they become bare `ICU_ADMISSION`, `ICU_DISCHARGE` and `TRANSFER`
|
|
168
|
+
events.
|
|
169
|
+
- **Demographics and allergies** carry no time in the release and are static events.
|
|
170
|
+
- **Flowsheet, lab and order-result values** are free text. The numeric reading is kept where
|
|
171
|
+
it parses; scored flowsheet items charted as `<score>--><label>` (e.g. `-1-->Drowsy`) keep
|
|
172
|
+
their score. The raw string goes to `text_value` unless it is just the number. Epic's
|
|
173
|
+
nonnumeric sentinel `9999999` never becomes a numeric value.
|
|
174
|
+
- **Medications:** each row of the release is one administration and repeats its order's
|
|
175
|
+
start and end, so an order yields one `START`, one `STOP` and one `MAR` event per
|
|
176
|
+
administration.
|
|
177
|
+
- **Orders** hold every order type except labs; roughly half carry a single result.
|
|
178
|
+
- **Birth and death:** the release has no birth date, age or death time, so there are no
|
|
179
|
+
`MEDS_BIRTH` or `MEDS_DEATH` events.
|
|
180
|
+
- **Not extracted:** the telemetry (alarms, 2-second measurements, waveforms; per-admission
|
|
181
|
+
files keyed by file name, far larger than the EMR), `social_hx` (no event times and
|
|
182
|
+
unlabeled Epic category codes), and the medication order date.
|
|
183
|
+
|
|
184
|
+
### Validation against the release
|
|
185
|
+
|
|
186
|
+
The spec was first written from the authors' release code
|
|
187
|
+
([Carescape](https://github.com/johnclawrence/Carescape)) and did not run on the published
|
|
188
|
+
files. Profiling the 1.0.0 EMR CSVs (value *shapes* and null counts only, never values)
|
|
189
|
+
showed what had to change:
|
|
190
|
+
|
|
191
|
+
- Tables are lowercase (`admission`, `labs`, ...) rather than `Patient`, `Labs`, ...; the
|
|
192
|
+
keys are `aMRN` and `aCSN`, not `csID` and `CS_PAT_ENC_CSN_ID`.
|
|
193
|
+
- Each timestamp is one already-shifted column rather than a date plus an `HH:MM:SS`
|
|
194
|
+
column. Labs and orders keep only the date (see [Timestamps](#timestamps)).
|
|
195
|
+
- `icu` and `location` have no department, event type, room or bed columns.
|
|
196
|
+
- `resulting_aCSN` holds the string `nan` for unresulted rows.
|
|
197
|
+
- 1,536 ICD-10 codes occur as both primary and secondary diagnoses, and some cells list
|
|
198
|
+
several codes (see [Diagnoses](#diagnoses)). The synthetic test covers both.
|
|
199
|
+
- Medication start and end times are always midnight, and `orders` carries results of its
|
|
200
|
+
own (it holds no lab orders).
|
|
201
|
+
|
|
202
|
+
All ten CSVs are valid UTF-8. On the full 1.0.0 EMR tables (6.3 GB of CSVs read),
|
|
203
|
+
`meds-extract-run` with MEDS-Extract 0.7.1 finishes in about 90 seconds on a laptop with about
|
|
204
|
+
6.5 GB peak memory, producing 52.6M events for 18,472 subjects. Fewer lab, order and
|
|
205
|
+
medication start/stop events come out than raw rows because identical events are merged: lab
|
|
206
|
+
and order times have day resolution, and the medication table repeats an order's start and
|
|
207
|
+
end on every administration row.
|
|
208
|
+
|
|
209
|
+
## MEDS-transforms settings
|
|
210
|
+
|
|
211
|
+
If you want to convert a large dataset, you can parallelize the MEDS-transforms stages
|
|
212
|
+
(the MEDS-transformation step that takes the longest).
|
|
213
|
+
|
|
214
|
+
Using local parallelization with the `hydra-joblib-launcher` package:
|
|
215
|
+
|
|
216
|
+
```bash
|
|
217
|
+
pip install hydra-joblib-launcher --upgrade
|
|
218
|
+
```
|
|
219
|
+
|
|
220
|
+
Then write a stage-runner file and pass it to the run:
|
|
221
|
+
|
|
222
|
+
```yaml
|
|
223
|
+
# runner.yaml
|
|
224
|
+
parallelize:
|
|
225
|
+
n_workers: 8
|
|
226
|
+
launcher: joblib
|
|
227
|
+
```
|
|
228
|
+
|
|
229
|
+
```bash
|
|
230
|
+
meds-extract-run spec=ALOTT output_dir=$MEDS_OUTPUT_DIR stage_runner_fp=runner.yaml
|
|
231
|
+
```
|
|
232
|
+
|
|
233
|
+
The number of subjects per shard is `n_subjects_per_shard` in the spec's `etl:` block
|
|
234
|
+
(1000 by default); raise it to cut the parallelization overhead.
|
|
235
|
+
|
|
236
|
+
## Development
|
|
237
|
+
|
|
238
|
+
```bash
|
|
239
|
+
uv sync --group dev
|
|
240
|
+
uv run pytest tests/
|
|
241
|
+
uv run pre-commit run --all-files
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
A release bumps the version in `pyproject.toml` with `uv version --bump`, commits it and pushes
|
|
245
|
+
a tag of that exact version, which publishes to PyPI and creates the GitHub release. See [`RELEASING.md`](https://github.com/rvandewater/ALOTT_MEDS/blob/main/RELEASING.md) for choosing the
|
|
246
|
+
version, the one-time PyPI setup and recovering from a failed release.
|
|
247
|
+
|
|
248
|
+
ALOTT has no open demo, so CI runs the spec end to end over a small **synthetic** input
|
|
249
|
+
generated in [`tests/e2e_synthetic_test.py`](https://github.com/rvandewater/ALOTT_MEDS/blob/main/tests/e2e_synthetic_test.py). That checks the
|
|
250
|
+
spec's mechanics, not its agreement with the real release. Never commit raw ALOTT data or
|
|
251
|
+
MEDS output derived from it.
|
|
252
|
+
|
|
253
|
+
## Citation
|
|
254
|
+
|
|
255
|
+
Please cite the following original dataset when using ALOTT:
|
|
256
|
+
|
|
257
|
+
```bibtex
|
|
258
|
+
@article{PhysioNet-alott-1.0.0,
|
|
259
|
+
author = {Lawrence, John and Rayo, Mike and Huerta, Timothy},
|
|
260
|
+
title = {{ALarms, Outcomes Telemetry with Timing (ALOTT): a Bedside-EMR Database}},
|
|
261
|
+
journal = {{PhysioNet}},
|
|
262
|
+
year = {2025},
|
|
263
|
+
month = mar,
|
|
264
|
+
note = {Version 1.0.0},
|
|
265
|
+
doi = {10.13026/sbq5-dy17},
|
|
266
|
+
url = {https://doi.org/10.13026/sbq5-dy17}
|
|
267
|
+
}
|
|
268
|
+
```
|
|
269
|
+
|
|
270
|
+
Also cite the ETL framework used to extract the dataset:
|
|
271
|
+
|
|
272
|
+
```bibtex
|
|
273
|
+
@software{van_de_Water_ALOTT_MEDS_ETL_2025,
|
|
274
|
+
author = {van de Water, Robin Philippus},
|
|
275
|
+
license = {MIT},
|
|
276
|
+
month = apr,
|
|
277
|
+
title = {{ALOTT\_MEDS ETL}},
|
|
278
|
+
url = {https://github.com/rvandewater/ALOTT_MEDS},
|
|
279
|
+
year = {2025}
|
|
280
|
+
}
|
|
281
|
+
```
|
|
282
|
+
|
|
283
|
+
For citing MEDS in general:
|
|
284
|
+
|
|
285
|
+
```bibtex
|
|
286
|
+
@article{mcdermott2026meds,
|
|
287
|
+
title={MEDS—An Emerging Data Standard and Ecosystem for Health AI Research},
|
|
288
|
+
author={McDermott, Matthew BA and Steinberg, Ethan and Fries, Jason A and van de Water, Robin P and Pang, Chao and Rockenschaub, Patrick and Renc, Pawel and Oh, Jungwoo and Stankevi{\v{c}}i{\=u}t{\.e}, Kamil{\.e} and Xu, Justin and others},
|
|
289
|
+
journal={NEJM AI},
|
|
290
|
+
volume={3},
|
|
291
|
+
number={6},
|
|
292
|
+
pages={AIra2501253},
|
|
293
|
+
year={2026},
|
|
294
|
+
publisher={Massachusetts Medical Society}
|
|
295
|
+
}
|
|
296
|
+
```
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
LICENSE
|
|
2
|
+
README.md
|
|
3
|
+
pyproject.toml
|
|
4
|
+
src/ALOTT_MEDS/__init__.py
|
|
5
|
+
src/ALOTT_MEDS/event_configs.yaml
|
|
6
|
+
src/ALOTT_MEDS.egg-info/PKG-INFO
|
|
7
|
+
src/ALOTT_MEDS.egg-info/SOURCES.txt
|
|
8
|
+
src/ALOTT_MEDS.egg-info/dependency_links.txt
|
|
9
|
+
src/ALOTT_MEDS.egg-info/entry_points.txt
|
|
10
|
+
src/ALOTT_MEDS.egg-info/requires.txt
|
|
11
|
+
src/ALOTT_MEDS.egg-info/top_level.txt
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
meds-extract[download]<0.8,>=0.7.1
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
ALOTT_MEDS
|