Zigong-MEDS 0.3.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2024 Matthew McDermott
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,177 @@
1
+ Metadata-Version: 2.4
2
+ Name: Zigong-MEDS
3
+ Version: 0.3.0
4
+ Summary: An ETL pipeline to extract the Zigong Fourth People's Hospital ICU infection database into the MEDS format.
5
+ Author-email: Robin van de Water <rpv2114@cumc.columbia.edu>
6
+ Project-URL: Homepage, https://github.com/rvandewater/ZIGONG_MEDS
7
+ Project-URL: Issues, https://github.com/rvandewater/ZIGONG_MEDS/issues
8
+ Project-URL: Dataset, https://physionet.org/content/icu-infection-zigong-fourth/1.1/
9
+ Classifier: Programming Language :: Python :: 3
10
+ Classifier: License :: OSI Approved :: MIT License
11
+ Classifier: Operating System :: OS Independent
12
+ Requires-Python: <3.14,>=3.11.4
13
+ Description-Content-Type: text/markdown
14
+ License-File: LICENSE
15
+ Requires-Dist: meds-extract[download]<0.8,>=0.7.0
16
+ Dynamic: license-file
17
+
18
+ # Zigong MEDS Extraction ETL
19
+
20
+ [![tests](https://github.com/rvandewater/ZIGONG_MEDS/actions/workflows/tests.yaml/badge.svg)](https://github.com/rvandewater/ZIGONG_MEDS/actions/workflows/tests.yaml)
21
+ [![synthetic ETL](https://github.com/rvandewater/ZIGONG_MEDS/actions/workflows/synthetic-etl.yaml/badge.svg)](https://github.com/rvandewater/ZIGONG_MEDS/actions/workflows/synthetic-etl.yaml)
22
+ [![code-quality](https://github.com/rvandewater/ZIGONG_MEDS/actions/workflows/code-quality-main.yaml/badge.svg)](https://github.com/rvandewater/ZIGONG_MEDS/actions/workflows/code-quality-main.yaml)
23
+ ![python](https://img.shields.io/badge/-Python_3.11--3.13-blue?logo=python&logoColor=white)
24
+ [![MEDS-Extract 0.7](https://img.shields.io/badge/MEDS--Extract-0.7-blue)](https://github.com/mmcdermott/MEDS_extract)
25
+ [![license](https://img.shields.io/badge/License-MIT-green.svg?labelColor=gray)](LICENSE)
26
+
27
+ This pipeline extracts the
28
+ [Zigong Fourth People's Hospital ICU infection database v1.1](https://physionet.org/content/icu-infection-zigong-fourth/1.1/)
29
+ from PhysioNet into the [MEDS format](https://medical-event-data-standard.github.io/). The release holds
30
+ 2,790 ICU patients with infection, one hospital admission each, from Zigong Fourth People's Hospital in
31
+ Sichuan, China. It was built from the
32
+ [ETL_MEDS_Template](https://github.com/Medical-Event-Data-Standard/ETL_MEDS_Template) and uses the
33
+ config-only layout introduced by [MEDS-Extract](https://github.com/mmcdermott/MEDS_extract) 0.7.
34
+
35
+ ## Usage
36
+
37
+ Zigong is a credentialed dataset. You need a PhysioNet account that has signed the project's data use
38
+ agreement.
39
+
40
+ ```bash
41
+ pip install git+https://github.com/rvandewater/ZIGONG_MEDS.git
42
+ export DATASET_DOWNLOAD_USERNAME=$PHYSIONET_USERNAME
43
+ export DATASET_DOWNLOAD_PASSWORD=$PHYSIONET_PASSWORD
44
+ meds-extract-run spec=Zigong output_dir=$MEDS_OUTPUT_DIR
45
+ ```
46
+
47
+ When you run this, the program will:
48
+
49
+ 1. Download the release (seven `dt*.csv` tables, about 840 MB) from PhysioNet into
50
+ `$MEDS_OUTPUT_DIR/.meds_extract_run/raw_input`. Pass `download_dest_dir=$RAW_INPUT_DIR` to keep the raw
51
+ data somewhere durable and reuse it across runs.
52
+ 2. Build the MEDS cohort from those files and write it to `$MEDS_OUTPUT_DIR` (`data/` and `metadata/`).
53
+ All transformations and joins are declared in
54
+ [`src/Zigong_MEDS/event_configs.yaml`](src/Zigong_MEDS/event_configs.yaml).
55
+
56
+ If you already have the PhysioNet release on disk, skip the download and point `input_dir` at the
57
+ directory that contains `dtBaseline.csv`, `dtDrugs.csv`, `dtICD.csv`, `dtLab.csv`, `dtNursingChart.csv`,
58
+ `dtOutCome.csv` and `dtTransfer.csv`:
59
+
60
+ ```bash
61
+ meds-extract-run spec=Zigong output_dir=$MEDS_OUTPUT_DIR \
62
+ do_download=false input_dir=/path/to/icu-infection-zigong-fourth-1.1
63
+ ```
64
+
65
+ To parallelize the stages, pass a runner file:
66
+
67
+ ```bash
68
+ printf 'parallelize:\n n_workers: 8\n launcher: joblib\n' >runner.yaml
69
+ meds-extract-run spec=Zigong output_dir=$MEDS_OUTPUT_DIR stage_runner_fp=runner.yaml
70
+ ```
71
+
72
+ Run `meds-extract-run --help` for every option.
73
+
74
+ ## How this ETL is defined
75
+
76
+ The package contains no Python. The whole ETL is one MESSY (MEDS-Extract Specification Syntax YAML)
77
+ file, [`src/Zigong_MEDS/event_configs.yaml`](src/Zigong_MEDS/event_configs.yaml). It declares:
78
+
79
+ - **`sources:`**: where the raw data lives (the PhysioNet release).
80
+ - **`etl:`**: the dataset name and sharding options.
81
+ - **One block per raw table**: how each row becomes one or more MEDS events.
82
+
83
+ `pyproject.toml` registers the file with MEDS-Extract's `MEDS_extract.pipelines` entry-point group under
84
+ the name `Zigong`. That registration is what makes `spec=Zigong` work. No pre-MEDS step is needed:
85
+ tables without `PATIENT_ID` (`dtLab`, `dtNursingChart`) are joined to `dtBaseline` on `INP_NO` inside the
86
+ config.
87
+
88
+ **Times.** Every source time is an offset in hours from hospital admission. The ETL places them on a
89
+ shared pseudotime anchor, 1970-01-01 00:00 = admission, which preserves order and intervals without
90
+ claiming a real calendar date. Each patient has exactly one admission, so offsets never collide.
91
+
92
+ ## Events
93
+
94
+ | Table | Event codes | Notes |
95
+ | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
96
+ | `dtBaseline` | `HOSPITAL_ADMISSION`, `AGE_AT_ADMISSION`, `SEX//…`, `ADMISSION_DEPARTMENT//…`, `INFECTION_SITE//…`, `ICU_DISCHARGE`, `HOSPITAL_DISCHARGE` | Age is the `numeric_value`. |
97
+ | `dtTransfer` | `TRANSFER_TO//{dept}`, `TRANSFER_FROM//{dept}` | The reason is kept in `transfer_reason`. See [#3](https://github.com/rvandewater/ZIGONG_MEDS/issues/3). |
98
+ | `dtICD` | `DIAGNOSIS//ICD//{code}` | Placed at hospital discharge. The clinician's description is the `text_value`; `diagnosis_sequence` and `discharge_diagnosis_status` are kept. |
99
+ | `dtLab` | `LAB//{category}//{item}//{unit}` | A missing unit becomes `UNK`. `reference_range` is kept. |
100
+ | `dtDrugs` | `MEDICATION//{drug}` | Prescription time. `Formula` is the `text_value`. |
101
+ | `dtNursingChart` | `NURSING//{MEASUREMENT}` (vitals, ventilation, consciousness, GCS components, lines and tubes, risk and pain scores, intake and output, notes, …) | One code per charted column. Vital signs carry a `unit` column. GCS components carry the score as the `numeric_value` and the charted label as text. |
102
+ | `dtOutCome` | `MEDS_DEATH`, `FOLLOW_UP_VITAL_STATUS//…`, `SF36//{ITEM}//{answer}`, `DISCHARGE_DEPARTMENT//…` | Follow-up events are placed at `Follow_Date`; the discharge department at hospital discharge. |
103
+
104
+ Every event linked to a stay carries `inpatient_stay_id` (`INP_NO`). The R-written CSVs encode missing
105
+ values as the string `NA` or a quoted empty string; both are treated as missing, as are text entries with
106
+ no letters or digits.
107
+
108
+ ## Data notes
109
+
110
+ - **Coarse nursing chart timestamps.** `dtNursingChart` has about 1 million rows with a usable
111
+ `ChartTime` but only about 23,000 distinct (patient, `ChartTime`) pairs. A typical timestamp covers 26
112
+ source rows, and the largest covers 266. This comes from the source release, not the ETL. In the MEDS
113
+ output, one timestamp therefore holds many readings of the same measurement, on average about 15
114
+ `NURSING//HEART_RATE` values, and similarly for blood pressure, SpO2 and other vitals. The order of events
115
+ within a timestamp carries no clinical meaning. When defining tasks or features:
116
+ - Do not rely on order within a timestamp, such as "last value before X", minute-level features, or
117
+ sequence models that treat event order as real time order.
118
+ - Choose prediction times that do not fall inside a timestamp's group of events, and aggregate values
119
+ within a timestamp (for example mean, min, or max) rather than picking one.
120
+ - **Dropped nursing rows.** 313,974 nursing rows (23%) belong to 723 stays that are not in the cohort
121
+ (`dtBaseline`), and all of them have `ChartTime = NA`; they are dropped. Rows charted more than 24 h before
122
+ admission (9,011 rows, including some corrupt offsets) or after discharge (141) are dropped too.
123
+ - **Dropped labs.** Results timed after hospital discharge (4,415 rows, 0.3%) and placeholder values made
124
+ only of dots or asterisks are dropped.
125
+ - **Coarse death dates.** `Death_Date` has about day precision. In-hospital deaths typically fall about 12 h
126
+ before the recorded discharge, so a stay's last events can follow `MEDS_DEATH`. See
127
+ [#4](https://github.com/rvandewater/ZIGONG_MEDS/issues/4).
128
+ - **Line and tube columns** (`CVC`, `PICC`, `Gastric_tube`, …) hold the length in cm, used as the
129
+ `numeric_value`, or a status such as "removed" kept as text. The numeric values of `Endotracheal_intubation`
130
+ duplicate `Endotracheal_intubation_Depth`, so only its statuses are emitted.
131
+ - **Bounded scores.** RASS, NRS and CPOT values outside their scales (-5..+4, 0..10, 0..8) are dropped.
132
+ Other vitals are not range-filtered; see [#7](https://github.com/rvandewater/ZIGONG_MEDS/issues/7).
133
+ - **Unmapped columns.** Five nursing columns are NA in every row. `MAP`, `PSIPAP`, `R` and `FIO2` hold only
134
+ unit strings. `Invasive_SBP._1` is a documented technical duplicate, and `F` is documented as having no
135
+ clinical relevance. See [#6](https://github.com/rvandewater/ZIGONG_MEDS/issues/6).
136
+ - **Local vocabularies.** Diagnoses use the release's Chinese national extension of ICD-10, and labs and
137
+ drugs use local names. No standard-terminology parents are attached yet; see
138
+ [#1](https://github.com/rvandewater/ZIGONG_MEDS/issues/1) and
139
+ [#8](https://github.com/rvandewater/ZIGONG_MEDS/issues/8). SF-36 answers are inconsistently coded in the
140
+ source; see [#5](https://github.com/rvandewater/ZIGONG_MEDS/issues/5).
141
+ - **No ICU admission event.** The release records ICU discharge but not ICU entry; see
142
+ [#2](https://github.com/rvandewater/ZIGONG_MEDS/issues/2).
143
+
144
+ ## Testing
145
+
146
+ Zigong has no open demo release, so there are two synthetic checks:
147
+
148
+ - `tests/e2e_synthetic_test.py` writes a few fully synthetic rows with the release's exact headers and R
149
+ `write.csv` encoding, runs `meds-extract-run spec=Zigong` over them, and checks the events, times, values
150
+ and missing-value handling it produces.
151
+ - The [Synthetic ETL workflow](.github/workflows/synthetic-etl.yaml) generates raw files straight from the
152
+ config with [MESSY-synth](https://pypi.org/project/MESSY-synth/) and runs the ETL over them. It fails if
153
+ any event block produces no events.
154
+
155
+ ```bash
156
+ uv sync --group dev
157
+ uv run pytest tests/ -v
158
+ uv run pre-commit run --all-files
159
+
160
+ # MESSY-synth needs Python >= 3.12.
161
+ uv run --with MESSY-synth==0.0.1 MESSY-synth ./src/Zigong_MEDS/event_configs.yaml -o raw_synthetic/
162
+ uv run meds-extract-run spec=Zigong output_dir=zigong_meds_synth/ \
163
+ do_download=false input_dir=raw_synthetic/
164
+ ```
165
+
166
+ For a reusable checklist on mapping a new dataset, see [docs/MEDS_ETL_MAPPING.md](docs/MEDS_ETL_MAPPING.md).
167
+
168
+ ## Citation
169
+
170
+ If you use this ETL, please cite the Zigong database and its descriptor paper:
171
+
172
+ - Xu, P., Chen, L., & Zhang, Z. (2022). *Critical care database comprising patients with infection at
173
+ Zigong Fourth People's Hospital* (version 1.1). PhysioNet. https://doi.org/10.13026/xpt9-z726
174
+ - Xu, P., Chen, L., Zhu, Y. et al. Critical Care Database Comprising Patients With Infection. *Front.
175
+ Public Health* 10 (2022). https://doi.org/10.3389/fpubh.2022.852410
176
+
177
+ See also [`CITATION.cff`](CITATION.cff).
@@ -0,0 +1,160 @@
1
+ # Zigong MEDS Extraction ETL
2
+
3
+ [![tests](https://github.com/rvandewater/ZIGONG_MEDS/actions/workflows/tests.yaml/badge.svg)](https://github.com/rvandewater/ZIGONG_MEDS/actions/workflows/tests.yaml)
4
+ [![synthetic ETL](https://github.com/rvandewater/ZIGONG_MEDS/actions/workflows/synthetic-etl.yaml/badge.svg)](https://github.com/rvandewater/ZIGONG_MEDS/actions/workflows/synthetic-etl.yaml)
5
+ [![code-quality](https://github.com/rvandewater/ZIGONG_MEDS/actions/workflows/code-quality-main.yaml/badge.svg)](https://github.com/rvandewater/ZIGONG_MEDS/actions/workflows/code-quality-main.yaml)
6
+ ![python](https://img.shields.io/badge/-Python_3.11--3.13-blue?logo=python&logoColor=white)
7
+ [![MEDS-Extract 0.7](https://img.shields.io/badge/MEDS--Extract-0.7-blue)](https://github.com/mmcdermott/MEDS_extract)
8
+ [![license](https://img.shields.io/badge/License-MIT-green.svg?labelColor=gray)](LICENSE)
9
+
10
+ This pipeline extracts the
11
+ [Zigong Fourth People's Hospital ICU infection database v1.1](https://physionet.org/content/icu-infection-zigong-fourth/1.1/)
12
+ from PhysioNet into the [MEDS format](https://medical-event-data-standard.github.io/). The release holds
13
+ 2,790 ICU patients with infection, one hospital admission each, from Zigong Fourth People's Hospital in
14
+ Sichuan, China. It was built from the
15
+ [ETL_MEDS_Template](https://github.com/Medical-Event-Data-Standard/ETL_MEDS_Template) and uses the
16
+ config-only layout introduced by [MEDS-Extract](https://github.com/mmcdermott/MEDS_extract) 0.7.
17
+
18
+ ## Usage
19
+
20
+ Zigong is a credentialed dataset. You need a PhysioNet account that has signed the project's data use
21
+ agreement.
22
+
23
+ ```bash
24
+ pip install git+https://github.com/rvandewater/ZIGONG_MEDS.git
25
+ export DATASET_DOWNLOAD_USERNAME=$PHYSIONET_USERNAME
26
+ export DATASET_DOWNLOAD_PASSWORD=$PHYSIONET_PASSWORD
27
+ meds-extract-run spec=Zigong output_dir=$MEDS_OUTPUT_DIR
28
+ ```
29
+
30
+ When you run this, the program will:
31
+
32
+ 1. Download the release (seven `dt*.csv` tables, about 840 MB) from PhysioNet into
33
+ `$MEDS_OUTPUT_DIR/.meds_extract_run/raw_input`. Pass `download_dest_dir=$RAW_INPUT_DIR` to keep the raw
34
+ data somewhere durable and reuse it across runs.
35
+ 2. Build the MEDS cohort from those files and write it to `$MEDS_OUTPUT_DIR` (`data/` and `metadata/`).
36
+ All transformations and joins are declared in
37
+ [`src/Zigong_MEDS/event_configs.yaml`](src/Zigong_MEDS/event_configs.yaml).
38
+
39
+ If you already have the PhysioNet release on disk, skip the download and point `input_dir` at the
40
+ directory that contains `dtBaseline.csv`, `dtDrugs.csv`, `dtICD.csv`, `dtLab.csv`, `dtNursingChart.csv`,
41
+ `dtOutCome.csv` and `dtTransfer.csv`:
42
+
43
+ ```bash
44
+ meds-extract-run spec=Zigong output_dir=$MEDS_OUTPUT_DIR \
45
+ do_download=false input_dir=/path/to/icu-infection-zigong-fourth-1.1
46
+ ```
47
+
48
+ To parallelize the stages, pass a runner file:
49
+
50
+ ```bash
51
+ printf 'parallelize:\n n_workers: 8\n launcher: joblib\n' >runner.yaml
52
+ meds-extract-run spec=Zigong output_dir=$MEDS_OUTPUT_DIR stage_runner_fp=runner.yaml
53
+ ```
54
+
55
+ Run `meds-extract-run --help` for every option.
56
+
57
+ ## How this ETL is defined
58
+
59
+ The package contains no Python. The whole ETL is one MESSY (MEDS-Extract Specification Syntax YAML)
60
+ file, [`src/Zigong_MEDS/event_configs.yaml`](src/Zigong_MEDS/event_configs.yaml). It declares:
61
+
62
+ - **`sources:`**: where the raw data lives (the PhysioNet release).
63
+ - **`etl:`**: the dataset name and sharding options.
64
+ - **One block per raw table**: how each row becomes one or more MEDS events.
65
+
66
+ `pyproject.toml` registers the file with MEDS-Extract's `MEDS_extract.pipelines` entry-point group under
67
+ the name `Zigong`. That registration is what makes `spec=Zigong` work. No pre-MEDS step is needed:
68
+ tables without `PATIENT_ID` (`dtLab`, `dtNursingChart`) are joined to `dtBaseline` on `INP_NO` inside the
69
+ config.
70
+
71
+ **Times.** Every source time is an offset in hours from hospital admission. The ETL places them on a
72
+ shared pseudotime anchor, 1970-01-01 00:00 = admission, which preserves order and intervals without
73
+ claiming a real calendar date. Each patient has exactly one admission, so offsets never collide.
74
+
75
+ ## Events
76
+
77
+ | Table | Event codes | Notes |
78
+ | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
79
+ | `dtBaseline` | `HOSPITAL_ADMISSION`, `AGE_AT_ADMISSION`, `SEX//…`, `ADMISSION_DEPARTMENT//…`, `INFECTION_SITE//…`, `ICU_DISCHARGE`, `HOSPITAL_DISCHARGE` | Age is the `numeric_value`. |
80
+ | `dtTransfer` | `TRANSFER_TO//{dept}`, `TRANSFER_FROM//{dept}` | The reason is kept in `transfer_reason`. See [#3](https://github.com/rvandewater/ZIGONG_MEDS/issues/3). |
81
+ | `dtICD` | `DIAGNOSIS//ICD//{code}` | Placed at hospital discharge. The clinician's description is the `text_value`; `diagnosis_sequence` and `discharge_diagnosis_status` are kept. |
82
+ | `dtLab` | `LAB//{category}//{item}//{unit}` | A missing unit becomes `UNK`. `reference_range` is kept. |
83
+ | `dtDrugs` | `MEDICATION//{drug}` | Prescription time. `Formula` is the `text_value`. |
84
+ | `dtNursingChart` | `NURSING//{MEASUREMENT}` (vitals, ventilation, consciousness, GCS components, lines and tubes, risk and pain scores, intake and output, notes, …) | One code per charted column. Vital signs carry a `unit` column. GCS components carry the score as the `numeric_value` and the charted label as text. |
85
+ | `dtOutCome` | `MEDS_DEATH`, `FOLLOW_UP_VITAL_STATUS//…`, `SF36//{ITEM}//{answer}`, `DISCHARGE_DEPARTMENT//…` | Follow-up events are placed at `Follow_Date`; the discharge department at hospital discharge. |
86
+
87
+ Every event linked to a stay carries `inpatient_stay_id` (`INP_NO`). The R-written CSVs encode missing
88
+ values as the string `NA` or a quoted empty string; both are treated as missing, as are text entries with
89
+ no letters or digits.
90
+
91
+ ## Data notes
92
+
93
+ - **Coarse nursing chart timestamps.** `dtNursingChart` has about 1 million rows with a usable
94
+ `ChartTime` but only about 23,000 distinct (patient, `ChartTime`) pairs. A typical timestamp covers 26
95
+ source rows, and the largest covers 266. This comes from the source release, not the ETL. In the MEDS
96
+ output, one timestamp therefore holds many readings of the same measurement, on average about 15
97
+ `NURSING//HEART_RATE` values, and similarly for blood pressure, SpO2 and other vitals. The order of events
98
+ within a timestamp carries no clinical meaning. When defining tasks or features:
99
+ - Do not rely on order within a timestamp, such as "last value before X", minute-level features, or
100
+ sequence models that treat event order as real time order.
101
+ - Choose prediction times that do not fall inside a timestamp's group of events, and aggregate values
102
+ within a timestamp (for example mean, min, or max) rather than picking one.
103
+ - **Dropped nursing rows.** 313,974 nursing rows (23%) belong to 723 stays that are not in the cohort
104
+ (`dtBaseline`), and all of them have `ChartTime = NA`; they are dropped. Rows charted more than 24 h before
105
+ admission (9,011 rows, including some corrupt offsets) or after discharge (141) are dropped too.
106
+ - **Dropped labs.** Results timed after hospital discharge (4,415 rows, 0.3%) and placeholder values made
107
+ only of dots or asterisks are dropped.
108
+ - **Coarse death dates.** `Death_Date` has about day precision. In-hospital deaths typically fall about 12 h
109
+ before the recorded discharge, so a stay's last events can follow `MEDS_DEATH`. See
110
+ [#4](https://github.com/rvandewater/ZIGONG_MEDS/issues/4).
111
+ - **Line and tube columns** (`CVC`, `PICC`, `Gastric_tube`, …) hold the length in cm, used as the
112
+ `numeric_value`, or a status such as "removed" kept as text. The numeric values of `Endotracheal_intubation`
113
+ duplicate `Endotracheal_intubation_Depth`, so only its statuses are emitted.
114
+ - **Bounded scores.** RASS, NRS and CPOT values outside their scales (-5..+4, 0..10, 0..8) are dropped.
115
+ Other vitals are not range-filtered; see [#7](https://github.com/rvandewater/ZIGONG_MEDS/issues/7).
116
+ - **Unmapped columns.** Five nursing columns are NA in every row. `MAP`, `PSIPAP`, `R` and `FIO2` hold only
117
+ unit strings. `Invasive_SBP._1` is a documented technical duplicate, and `F` is documented as having no
118
+ clinical relevance. See [#6](https://github.com/rvandewater/ZIGONG_MEDS/issues/6).
119
+ - **Local vocabularies.** Diagnoses use the release's Chinese national extension of ICD-10, and labs and
120
+ drugs use local names. No standard-terminology parents are attached yet; see
121
+ [#1](https://github.com/rvandewater/ZIGONG_MEDS/issues/1) and
122
+ [#8](https://github.com/rvandewater/ZIGONG_MEDS/issues/8). SF-36 answers are inconsistently coded in the
123
+ source; see [#5](https://github.com/rvandewater/ZIGONG_MEDS/issues/5).
124
+ - **No ICU admission event.** The release records ICU discharge but not ICU entry; see
125
+ [#2](https://github.com/rvandewater/ZIGONG_MEDS/issues/2).
126
+
127
+ ## Testing
128
+
129
+ Zigong has no open demo release, so there are two synthetic checks:
130
+
131
+ - `tests/e2e_synthetic_test.py` writes a few fully synthetic rows with the release's exact headers and R
132
+ `write.csv` encoding, runs `meds-extract-run spec=Zigong` over them, and checks the events, times, values
133
+ and missing-value handling it produces.
134
+ - The [Synthetic ETL workflow](.github/workflows/synthetic-etl.yaml) generates raw files straight from the
135
+ config with [MESSY-synth](https://pypi.org/project/MESSY-synth/) and runs the ETL over them. It fails if
136
+ any event block produces no events.
137
+
138
+ ```bash
139
+ uv sync --group dev
140
+ uv run pytest tests/ -v
141
+ uv run pre-commit run --all-files
142
+
143
+ # MESSY-synth needs Python >= 3.12.
144
+ uv run --with MESSY-synth==0.0.1 MESSY-synth ./src/Zigong_MEDS/event_configs.yaml -o raw_synthetic/
145
+ uv run meds-extract-run spec=Zigong output_dir=zigong_meds_synth/ \
146
+ do_download=false input_dir=raw_synthetic/
147
+ ```
148
+
149
+ For a reusable checklist on mapping a new dataset, see [docs/MEDS_ETL_MAPPING.md](docs/MEDS_ETL_MAPPING.md).
150
+
151
+ ## Citation
152
+
153
+ If you use this ETL, please cite the Zigong database and its descriptor paper:
154
+
155
+ - Xu, P., Chen, L., & Zhang, Z. (2022). *Critical care database comprising patients with infection at
156
+ Zigong Fourth People's Hospital* (version 1.1). PhysioNet. https://doi.org/10.13026/xpt9-z726
157
+ - Xu, P., Chen, L., Zhu, Y. et al. Critical Care Database Comprising Patients With Infection. *Front.
158
+ Public Health* 10 (2022). https://doi.org/10.3389/fpubh.2022.852410
159
+
160
+ See also [`CITATION.cff`](CITATION.cff).
@@ -0,0 +1,82 @@
1
+ [build-system]
2
+ requires = ["setuptools>=64", "wheel"]
3
+ build-backend = "setuptools.build_meta"
4
+
5
+ [project]
6
+ name = "Zigong-MEDS"
7
+ # Bump with `uv version --bump patch|minor|major`, then tag the commit with the same version.
8
+ version = "0.3.0"
9
+ authors = [
10
+ { name="Robin van de Water", email="rpv2114@cumc.columbia.edu" },
11
+ ]
12
+ description = "An ETL pipeline to extract the Zigong Fourth People's Hospital ICU infection database into the MEDS format."
13
+ readme = "README.md"
14
+ # Mirrors MEDS-Extract 0.7's own interpreter range; a looser range here admits interpreters it
15
+ # cannot be installed on.
16
+ requires-python = ">=3.11.4,<3.14"
17
+ classifiers = [
18
+ "Programming Language :: Python :: 3",
19
+ "License :: OSI Approved :: MIT License",
20
+ "Operating System :: OS Independent",
21
+ ]
22
+ # This package ships no Python: the whole ETL is `event_configs.yaml`, run by
23
+ # MEDS-Extract's own `meds-extract-run`. So there is exactly one dependency — the
24
+ # `[download]` extra adds `meds-extract-download` (httpx + tenacity), which the runner
25
+ # spawns to stage the raw data. MEDS-transforms, polars and hydra all arrive through it
26
+ # at the versions it fences; duplicating those pins here would only create drift.
27
+ #
28
+ # The floor is 0.7.0: this config is written in MEDS-Extract 0.7's single-file MESSY format, which
29
+ # earlier releases cannot read.
30
+ dependencies = [
31
+ "meds-extract[download]>=0.7.0,<0.8",
32
+ ]
33
+
34
+ [dependency-groups]
35
+ # `shfmt-py` bundles the `shfmt` binary that the `mdformat-shfmt` pre-commit hook shells out to;
36
+ # declaring it here keeps local and CI behavior in lock step. `pyyaml` is read by the e2e test to
37
+ # enumerate the config's event blocks. MESSY-synth (Python >= 3.12) is installed by its own CI
38
+ # workflow rather than here, so the dev group still resolves on Python 3.11.
39
+ dev = [
40
+ "pre-commit~=3.8",
41
+ "ruff>=0.11,<1.0",
42
+ "pytest~=9.0",
43
+ "pyyaml>=6.0",
44
+ "shfmt-py==3.12.0.2",
45
+ ]
46
+
47
+ [project.entry-points."MEDS_extract.pipelines"]
48
+ Zigong = "Zigong_MEDS:event_configs.yaml"
49
+
50
+ [project.urls]
51
+ Homepage = "https://github.com/rvandewater/ZIGONG_MEDS"
52
+ Issues = "https://github.com/rvandewater/ZIGONG_MEDS/issues"
53
+ Dataset = "https://physionet.org/content/icu-infection-zigong-fourth/1.1/"
54
+
55
+ [tool.setuptools.packages.find]
56
+ where = ["src"]
57
+ include = ["Zigong_MEDS*"]
58
+
59
+ [tool.setuptools.package-data]
60
+ Zigong_MEDS = ["*.yaml"]
61
+
62
+ [tool.ruff]
63
+ target-version = "py311"
64
+ line-length = 110
65
+
66
+ [tool.ruff.lint]
67
+ select = [
68
+ "E", # pycodestyle
69
+ "F", # pyflakes
70
+ "I", # isort
71
+ "UP", # pyupgrade
72
+ "B", # flake8-bugbear
73
+ "C4", # flake8-comprehensions
74
+ "RUF", # Ruff-specific rules
75
+ "SIM", # flake8-simplify
76
+ ]
77
+ ignore = [
78
+ "SIM108", # ternary operator suggestions
79
+ ]
80
+
81
+ [tool.ruff.lint.per-file-ignores]
82
+ "__init__.py" = ["F401"]
@@ -0,0 +1,4 @@
1
+ [egg_info]
2
+ tag_build =
3
+ tag_date = 0
4
+
@@ -0,0 +1,12 @@
1
+ """The Zigong ICU infection database MEDS ETL — a configuration package, not a library.
2
+
3
+ This distribution ships no code. Its entire content is ``Zigong_MEDS/event_configs.yaml``, a MESSY
4
+ (MEDS-Extract Specification Syntax YAML) document declaring where the raw Zigong data comes from and
5
+ how every event is extracted from it. ``pyproject.toml`` registers that file with MEDS-Extract's
6
+ ``MEDS_extract.pipelines`` entry-point group under the name ``Zigong``, so the ETL runs as::
7
+
8
+ meds-extract-run spec=Zigong output_dir=$MEDS_OUTPUT_DIR
9
+
10
+ This package exists so the YAML is addressable as package data —
11
+ ``importlib.resources.files("Zigong_MEDS")`` — from an installed wheel. Nothing here is importable API.
12
+ """
@@ -0,0 +1,570 @@
1
+ # The complete Zigong Fourth People's Hospital ICU infection database (PhysioNet v1.1) MEDS ETL.
2
+ # This one file *is* the package: `sources:` says where the raw data comes from, `etl:` supplies
3
+ # the run knobs, and every other top-level key is a raw table whose events are extracted from it.
4
+ # It is registered with `meds-extract-run` under the name `Zigong` (see this repo's
5
+ # `pyproject.toml`), so the whole ETL is one command:
6
+ #
7
+ # meds-extract-run spec=Zigong output_dir=$MEDS_OUTPUT_DIR
8
+ #
9
+ # Conventions used throughout:
10
+ #
11
+ # - All source times are deidentified hours from hospital admission. 1970-01-01 is a pseudotime
12
+ # anchor: it preserves ordering and intervals without asserting a real calendar date. Every
13
+ # patient has exactly one admission in v1.1, so per-admission offsets never collide.
14
+ # - The release was written by R's write.csv: a missing value is the bare string NA, and missing
15
+ # text in several dtNursingChart and dtLab columns is a quoted empty string. Neither is read as
16
+ # null, so every code and text value below guards against both. Free-text nursing fields also
17
+ # require a word character, which drops whitespace-, DEL- and punctuation-only entries.
18
+ # - Codes keep the source's own vocabulary. No standard-terminology parent is claimed where the
19
+ # release does not document one.
20
+
21
+ etl:
22
+ # `meds-extract-run` defaults the dataset name to the registered entry-point name; this fallback
23
+ # makes the spec self-describing when it is resolved by `pkg://` reference or filesystem path.
24
+ dataset_name: Zigong
25
+ n_subjects_per_shard: 1000
26
+
27
+ _defaults:
28
+ subject_id: $PATIENT_ID
29
+
30
+ dtBaseline:
31
+ hospital_admission:
32
+ code: HOSPITAL_ADMISSION
33
+ time: 1970-01-01 @ 00:00
34
+ inpatient_stay_id: $INP_NO
35
+ age_at_admission:
36
+ code: '"AGE_AT_ADMISSION" if $Age::?float64 >= 0'
37
+ time: 1970-01-01 @ 00:00
38
+ numeric_value: $Age::?float64
39
+ inpatient_stay_id: $INP_NO
40
+ sex:
41
+ code: f"SEX//{$SEX}"
42
+ time: 1970-01-01 @ 00:00
43
+ inpatient_stay_id: $INP_NO
44
+ admission_department:
45
+ code: f"ADMISSION_DEPARTMENT//{$AdmitDept}"
46
+ time: 1970-01-01 @ 00:00
47
+ inpatient_stay_id: $INP_NO
48
+ infection_site:
49
+ code: f"INFECTION_SITE//{$InfectionSite}"
50
+ time: 1970-01-01 @ 00:00
51
+ inpatient_stay_id: $INP_NO
52
+ icu_discharge:
53
+ code: ICU_DISCHARGE
54
+ time: "(1970-01-01 @ 00:00) + $ICU_discharge_time::hours"
55
+ inpatient_stay_id: $INP_NO
56
+ hospital_discharge:
57
+ code: HOSPITAL_DISCHARGE
58
+ time: "(1970-01-01 @ 00:00) + $DISCHARGE_DATE_TIME::hours"
59
+ inpatient_stay_id: $INP_NO
60
+
61
+ dtDrugs:
62
+ # dtDrugs has no INP_NO; with one admission per patient the stay is recovered from dtBaseline.
63
+ _table:
64
+ join:
65
+ dtBaseline:
66
+ key: PATIENT_ID
67
+ cols: [INP_NO]
68
+ prescription:
69
+ code: f"MEDICATION//{$DrugName}"
70
+ time: "(1970-01-01 @ 00:00) + $Drug_time::hours"
71
+ text_value: $Formula
72
+ inpatient_stay_id: $INP_NO
73
+
74
+ dtICD:
75
+ _table:
76
+ join:
77
+ dtBaseline:
78
+ key: INP_NO
79
+ cols: [DISCHARGE_DATE_TIME]
80
+ diagnosis:
81
+ code: 'f"DIAGNOSIS//ICD//{$ICD_Code}" if $ICD_Code != "NA"'
82
+ time: "(1970-01-01 @ 00:00) + $DISCHARGE_DATE_TIME::hours"
83
+ # ICD_desc is the clinician's free-text diagnosis, not the code's title: most codes carry
84
+ # several distinct descriptions, so it stays per-event and is not used as code metadata.
85
+ text_value: $ICD_desc
86
+ diagnosis_sequence: $Diagnosis_seq
87
+ discharge_diagnosis_status: '$Status_Discharge if $Status_Discharge != "NA"'
88
+ inpatient_stay_id: $INP_NO
89
+
90
+ dtLab:
91
+ _table:
92
+ join:
93
+ dtBaseline:
94
+ key: INP_NO
95
+ cols: [PATIENT_ID, DISCHARGE_DATE_TIME]
96
+ # Results timed after hospital discharge (~0.3% of rows) are dropped; pre-admission results
97
+ # (down to about -24 h) are kept. Placeholder values made only of dots or asterisks (".....",
98
+ # "****") carry no result and are dropped.
99
+ lab_result:
100
+ code: >-
101
+ f"LAB//{$Category}//{$Item}//{$Unit if $Unit != '' else 'UNK'}"
102
+ if /[^.*\s]/ in $LabValue::str
103
+ time: "(1970-01-01 @ 00:00) + $LabTime::hours if $LabTime <= $DISCHARGE_DATE_TIME"
104
+ numeric_value: $LabValue::?float64
105
+ text_value: $LabValue
106
+ reference_range: '$Ref if $Ref != ""'
107
+ inpatient_stay_id: $INP_NO
108
+
109
+ dtTransfer:
110
+ transfer_start:
111
+ code: f"TRANSFER_TO//{$TransferDept}"
112
+ time: "(1970-01-01 @ 00:00) + $StartTime::hours"
113
+ transfer_reason: $StartReason
114
+ inpatient_stay_id: $INP_NO
115
+ transfer_end:
116
+ code: f"TRANSFER_FROM//{$TransferDept}"
117
+ time: "(1970-01-01 @ 00:00) + $StopTime::hours"
118
+ transfer_reason: $StopReason
119
+ inpatient_stay_id: $INP_NO
120
+
121
+ # Every row with ChartTime = NA belongs to a stay that is not in dtBaseline, so the inner join
122
+ # drops them. Chart times more than 24 h before admission (including some corrupt offsets of
123
+ # millions of hours) or after discharge are dropped too. Several columns are not mapped: Rhythm,
124
+ # Lower_extremity_blood_pressure_monitoring, Left/Right_pupil_reflex and ETCO2 are NA in every
125
+ # row; MAP, PSIPAP, R and FIO2 hold only unit strings; Invasive_SBP._1 is a documented technical
126
+ # duplicate; F is documented as having no clinical relevance.
127
+ dtNursingChart:
128
+ _table:
129
+ join:
130
+ dtBaseline:
131
+ key: INP_NO
132
+ cols: [PATIENT_ID, DISCHARGE_DATE_TIME]
133
+ cols:
134
+ chart_time: >-
135
+ (1970-01-01 @ 00:00) + $ChartTime::?float64::hours
136
+ if $ChartTime::?float64 >= -24 and $ChartTime::?float64 <= $DISCHARGE_DATE_TIME
137
+ temperature:
138
+ code: '"NURSING//TEMPERATURE" if $temperature::?float64 >= 0'
139
+ time: $chart_time
140
+ numeric_value: $temperature::?float64
141
+ unit: degC
142
+ inpatient_stay_id: $INP_NO
143
+ heart_rate:
144
+ code: '"NURSING//HEART_RATE" if $heart_rate::?float64 >= 0'
145
+ time: $chart_time
146
+ numeric_value: $heart_rate::?float64
147
+ unit: "'/min'"
148
+ inpatient_stay_id: $INP_NO
149
+ respiratory_rate:
150
+ code: '"NURSING//RESPIRATORY_RATE" if $breathing::?float64 >= 0'
151
+ time: $chart_time
152
+ numeric_value: $breathing::?float64
153
+ unit: "'/min'"
154
+ inpatient_stay_id: $INP_NO
155
+ oxygen_saturation:
156
+ code: '"NURSING//OXYGEN_SATURATION" if $Blood_oxygen_saturation::?float64 >= 0'
157
+ time: $chart_time
158
+ numeric_value: $Blood_oxygen_saturation::?float64
159
+ unit: '"%"'
160
+ inpatient_stay_id: $INP_NO
161
+ systolic_blood_pressure:
162
+ code: '"NURSING//SYSTOLIC_BLOOD_PRESSURE" if $Blood_pressure_high::?float64 >= 0'
163
+ time: $chart_time
164
+ numeric_value: $Blood_pressure_high::?float64
165
+ unit: mmHg
166
+ inpatient_stay_id: $INP_NO
167
+ diastolic_blood_pressure:
168
+ code: '"NURSING//DIASTOLIC_BLOOD_PRESSURE" if $Blood_pressure_low::?float64 >= 0'
169
+ time: $chart_time
170
+ numeric_value: $Blood_pressure_low::?float64
171
+ unit: mmHg
172
+ inpatient_stay_id: $INP_NO
173
+ invasive_systolic_blood_pressure:
174
+ code: '"NURSING//INVASIVE_SYSTOLIC_BLOOD_PRESSURE" if $Invasive_SBP::?float64 >= 0'
175
+ time: $chart_time
176
+ numeric_value: $Invasive_SBP::?float64
177
+ inpatient_stay_id: $INP_NO
178
+ invasive_diastolic_blood_pressure:
179
+ code: '"NURSING//INVASIVE_DIASTOLIC_BLOOD_PRESSURE" if $Invasive_diastolic_blood_pressure::?float64 >= 0'
180
+ time: $chart_time
181
+ numeric_value: $Invasive_diastolic_blood_pressure::?float64
182
+ inpatient_stay_id: $INP_NO
183
+ central_venous_pressure:
184
+ code: '"NURSING//CENTRAL_VENOUS_PRESSURE" if $Central_venous_pressure::?float64 >= 0'
185
+ time: $chart_time
186
+ numeric_value: $Central_venous_pressure::?float64
187
+ inpatient_stay_id: $INP_NO
188
+ blood_sugar:
189
+ code: '"NURSING//BLOOD_SUGAR" if $blood_sugar::?float64 >= 0'
190
+ time: $chart_time
191
+ numeric_value: $blood_sugar::?float64
192
+ inpatient_stay_id: $INP_NO
193
+ oxygen_inhalation_mode:
194
+ code: '"NURSING//OXYGEN_INHALATION_MODE" if $Oxygen_inhalation_mode != "NA" and /\w/ in $Oxygen_inhalation_mode'
195
+ time: $chart_time
196
+ text_value: $Oxygen_inhalation_mode
197
+ inpatient_stay_id: $INP_NO
198
+ oxygen_flow:
199
+ code: '"NURSING//OXYGEN_FLOW" if $Oxygen_flow::?float64 >= 0'
200
+ time: $chart_time
201
+ numeric_value: $Oxygen_flow::?float64
202
+ inpatient_stay_id: $INP_NO
203
+ ventilation_mode:
204
+ code: '"NURSING//VENTILATION_MODE" if $Breathing_pattern != "NA" and /\w/ in $Breathing_pattern'
205
+ time: $chart_time
206
+ text_value: $Breathing_pattern
207
+ inpatient_stay_id: $INP_NO
208
+ tidal_volume_setting:
209
+ code: '"NURSING//TIDAL_VOLUME_SETTING" if $Vt_setting::?float64 >= 0'
210
+ time: $chart_time
211
+ numeric_value: $Vt_setting::?float64
212
+ inpatient_stay_id: $INP_NO
213
+ monitored_tidal_volume:
214
+ code: '"NURSING//MONITORED_TIDAL_VOLUME" if $Vt_Supervisor::?float64 >= 0'
215
+ time: $chart_time
216
+ numeric_value: $Vt_Supervisor::?float64
217
+ inpatient_stay_id: $INP_NO
218
+ consciousness:
219
+ code: '"NURSING//CONSCIOUSNESS" if $consciousness != "NA" and /\w/ in $consciousness'
220
+ time: $chart_time
221
+ text_value: $consciousness
222
+ inpatient_stay_id: $INP_NO
223
+ left_pupil_size:
224
+ code: '"NURSING//LEFT_PUPIL_SIZE" if $Left_pupil_size::?float64 >= 0'
225
+ time: $chart_time
226
+ numeric_value: $Left_pupil_size::?float64
227
+ inpatient_stay_id: $INP_NO
228
+ right_pupil_size:
229
+ code: '"NURSING//RIGHT_PUPIL_SIZE" if $Right_pupil_size::?float64 >= 0'
230
+ time: $chart_time
231
+ numeric_value: $Right_pupil_size::?float64
232
+ inpatient_stay_id: $INP_NO
233
+ # GCS components are charted as "<score>→<Chinese label>", e.g. "3→呼唤睁眼". The leading score
234
+ # is the numeric_value; non-scorable entries ("T" intubated, "S" motor disorder) keep only the
235
+ # text.
236
+ gcs_eye_response:
237
+ code: '"NURSING//GCS_EYE_RESPONSE" if $`open_one''s_eyes` != "NA" and /\w/ in $`open_one''s_eyes`'
238
+ time: $chart_time
239
+ numeric_value: '(extract group 1 of /^\s*([1-4])\s*[^\d\s]/ from $`open_one''s_eyes`)::?float64'
240
+ text_value: $`open_one's_eyes`
241
+ inpatient_stay_id: $INP_NO
242
+ gcs_motor_response:
243
+ code: '"NURSING//GCS_MOTOR_RESPONSE" if $motion != "NA" and /\w/ in $motion'
244
+ time: $chart_time
245
+ numeric_value: '(extract group 1 of /^\s*([1-6])\s*[^\d\s]/ from $motion)::?float64'
246
+ text_value: $motion
247
+ inpatient_stay_id: $INP_NO
248
+ gcs_verbal_response:
249
+ code: '"NURSING//GCS_VERBAL_RESPONSE" if $language != "NA" and /\w/ in $language'
250
+ time: $chart_time
251
+ numeric_value: '(extract group 1 of /^\s*([1-5])\s*[^\d\s]/ from $language)::?float64'
252
+ text_value: $language
253
+ inpatient_stay_id: $INP_NO
254
+ # Line and tube columns hold the charted length in cm, or a status such as "拔除" (removed) or
255
+ # "不详" (unknown); the length is the numeric_value and the raw entry stays in text_value. The
256
+ # `::str` casts keep the text guards valid whichever dtype the column is inferred as.
257
+ cvc:
258
+ code: '"NURSING//CVC" if $CVC::str != "NA" and /\w/ in $CVC::str'
259
+ time: $chart_time
260
+ numeric_value: $CVC::?float64
261
+ text_value: $CVC::str
262
+ inpatient_stay_id: $INP_NO
263
+ picc:
264
+ code: '"NURSING//PICC" if $PICC::str != "NA" and /\w/ in $PICC::str'
265
+ time: $chart_time
266
+ numeric_value: $PICC::?float64
267
+ text_value: $PICC::str
268
+ inpatient_stay_id: $INP_NO
269
+ # Every numeric Endotracheal_intubation value equals Endotracheal_intubation_Depth on the same
270
+ # row, so only its non-numeric statuses (e.g. "拔除", removed) are emitted here; the depth is
271
+ # emitted by endotracheal_intubation_depth below.
272
+ endotracheal_intubation:
273
+ code: >-
274
+ "NURSING//ENDOTRACHEAL_INTUBATION"
275
+ if $Endotracheal_intubation::str != "NA" and /\w/ in $Endotracheal_intubation::str
276
+ and /[^0-9.\s]/ in $Endotracheal_intubation::str
277
+ time: $chart_time
278
+ text_value: $Endotracheal_intubation::str
279
+ inpatient_stay_id: $INP_NO
280
+ gastric_tube:
281
+ code: '"NURSING//GASTRIC_TUBE" if $Gastric_tube::str != "NA" and /\w/ in $Gastric_tube::str'
282
+ time: $chart_time
283
+ numeric_value: $Gastric_tube::?float64
284
+ text_value: $Gastric_tube::str
285
+ inpatient_stay_id: $INP_NO
286
+ medium_length_catheter:
287
+ code: '"NURSING//MEDIUM_LENGTH_CATHETER" if $Medium_length_catheter::str != "NA" and /\w/ in $Medium_length_catheter::str'
288
+ time: $chart_time
289
+ numeric_value: $Medium_length_catheter::?float64
290
+ text_value: $Medium_length_catheter::str
291
+ inpatient_stay_id: $INP_NO
292
+ hemodialysis_tube:
293
+ code: '"NURSING//HEMODIALYSIS_TUBE" if $Hemodialysis_tube::str != "NA" and /\w/ in $Hemodialysis_tube::str'
294
+ time: $chart_time
295
+ numeric_value: $Hemodialysis_tube::?float64
296
+ text_value: $Hemodialysis_tube::str
297
+ inpatient_stay_id: $INP_NO
298
+ basic_nursing:
299
+ code: '"NURSING//BASIC_NURSING" if $basic_nursing != "NA" and /\w/ in $basic_nursing'
300
+ time: $chart_time
301
+ text_value: $basic_nursing
302
+ inpatient_stay_id: $INP_NO
303
+ skin_care:
304
+ code: '"NURSING//SKIN_CARE" if $Skin_care != "NA" and /\w/ in $Skin_care'
305
+ time: $chart_time
306
+ text_value: $Skin_care
307
+ inpatient_stay_id: $INP_NO
308
+ acral_circulation:
309
+ code: '"NURSING//ACRAL_CIRCULATION" if $Acral_circulation != "NA" and /\w/ in $Acral_circulation'
310
+ time: $chart_time
311
+ text_value: $Acral_circulation
312
+ inpatient_stay_id: $INP_NO
313
+ pressure_ulcer_risk:
314
+ code: '"NURSING//PRESSURE_ULCER_RISK" if $Pressure_ulcer_risk_assessment::?float64 >= 0'
315
+ time: $chart_time
316
+ numeric_value: $Pressure_ulcer_risk_assessment::?float64
317
+ inpatient_stay_id: $INP_NO
318
+ catheter_drop_risk:
319
+ code: '"NURSING//CATHETER_DROP_RISK" if $Catheter_drop_assessment::?float64 >= 0'
320
+ time: $chart_time
321
+ numeric_value: $Catheter_drop_assessment::?float64
322
+ inpatient_stay_id: $INP_NO
323
+ fall_risk:
324
+ code: '"NURSING//FALL_RISK" if $Fall_bed_assessment::?float64 >= 0'
325
+ time: $chart_time
326
+ numeric_value: $Fall_bed_assessment::?float64
327
+ inpatient_stay_id: $INP_NO
328
+ # Scores outside the scale's defined range (RASS -5..+4, NRS 0..10, CPOT 0..8) are charting
329
+ # errors, e.g. negative RASS-like values in the NRS column, and are dropped.
330
+ rass_sedation_score:
331
+ code: >-
332
+ "NURSING//RASS_SEDATION_SCORE"
333
+ if $RASS_sedation_score::?float64 >= -5 and $RASS_sedation_score::?float64 <= 4
334
+ time: $chart_time
335
+ numeric_value: $RASS_sedation_score::?float64
336
+ inpatient_stay_id: $INP_NO
337
+ nrs_pain_score:
338
+ code: '"NURSING//NRS_PAIN_SCORE" if $NRS_pain_score::?float64 >= 0 and $NRS_pain_score::?float64 <= 10'
339
+ time: $chart_time
340
+ numeric_value: $NRS_pain_score::?float64
341
+ inpatient_stay_id: $INP_NO
342
+ cpot_pain_score:
343
+ code: '"NURSING//CPOT_PAIN_SCORE" if $Cpot_pain_score::?float64 >= 0 and $Cpot_pain_score::?float64 <= 8'
344
+ time: $chart_time
345
+ numeric_value: $Cpot_pain_score::?float64
346
+ inpatient_stay_id: $INP_NO
347
+ airbag_pressure:
348
+ code: '"NURSING//AIRBAG_PRESSURE" if $Airbag_pressure::?float64 >= 0'
349
+ time: $chart_time
350
+ numeric_value: $Airbag_pressure::?float64
351
+ inpatient_stay_id: $INP_NO
352
+ milk_intake:
353
+ code: '"NURSING//MILK_INTAKE" if $milk::?float64 >= 0'
354
+ time: $chart_time
355
+ numeric_value: $milk::?float64
356
+ inpatient_stay_id: $INP_NO
357
+ rice_intake:
358
+ code: '"NURSING//RICE_INTAKE" if $rice::?float64 >= 0'
359
+ time: $chart_time
360
+ numeric_value: $rice::?float64
361
+ inpatient_stay_id: $INP_NO
362
+ rice_noodles_intake:
363
+ code: '"NURSING//RICE_NOODLES_INTAKE" if $Rice_noodles::?float64 >= 0'
364
+ time: $chart_time
365
+ numeric_value: $Rice_noodles::?float64
366
+ inpatient_stay_id: $INP_NO
367
+ soup_intake:
368
+ code: '"NURSING//SOUP_INTAKE" if $soup::?float64 >= 0'
369
+ time: $chart_time
370
+ numeric_value: $soup::?float64
371
+ inpatient_stay_id: $INP_NO
372
+ traditional_chinese_medicine_intake:
373
+ code: '"NURSING//TRADITIONAL_CHINESE_MEDICINE_INTAKE" if $traditional_Chinese_medicine::?float64 >= 0'
374
+ time: $chart_time
375
+ numeric_value: $traditional_Chinese_medicine::?float64
376
+ inpatient_stay_id: $INP_NO
377
+ soft_food_intake:
378
+ code: '"NURSING//SOFT_FOOD_INTAKE" if $Soft_food::?float64 >= 0'
379
+ time: $chart_time
380
+ numeric_value: $Soft_food::?float64
381
+ inpatient_stay_id: $INP_NO
382
+ rice_soup_intake:
383
+ code: '"NURSING//RICE_SOUP_INTAKE" if $the_rice_soup::?float64 >= 0'
384
+ time: $chart_time
385
+ numeric_value: $the_rice_soup::?float64
386
+ inpatient_stay_id: $INP_NO
387
+ warm_water_intake:
388
+ code: '"NURSING//WARM_WATER_INTAKE" if $warm_water::?float64 >= 0'
389
+ time: $chart_time
390
+ numeric_value: $warm_water::?float64
391
+ inpatient_stay_id: $INP_NO
392
+ fruit_juice_intake:
393
+ code: '"NURSING//FRUIT_JUICE_INTAKE" if $fruit_juice::?float64 >= 0'
394
+ time: $chart_time
395
+ numeric_value: $fruit_juice::?float64
396
+ inpatient_stay_id: $INP_NO
397
+ stool_output:
398
+ code: '"NURSING//STOOL_OUTPUT" if $shit::?float64 >= 0'
399
+ time: $chart_time
400
+ numeric_value: $shit::?float64
401
+ inpatient_stay_id: $INP_NO
402
+ urine_output:
403
+ code: '"NURSING//URINE_OUTPUT" if $Urine_volume::?float64 >= 0'
404
+ time: $chart_time
405
+ numeric_value: $Urine_volume::?float64
406
+ inpatient_stay_id: $INP_NO
407
+ gastric_juice_output:
408
+ code: '"NURSING//GASTRIC_JUICE_OUTPUT" if $gastric_juice::?float64 >= 0'
409
+ time: $chart_time
410
+ numeric_value: $gastric_juice::?float64
411
+ inpatient_stay_id: $INP_NO
412
+ sputum_output:
413
+ code: '"NURSING//SPUTUM_OUTPUT" if $Sputum_volume::?float64 >= 0'
414
+ time: $chart_time
415
+ numeric_value: $Sputum_volume::?float64
416
+ inpatient_stay_id: $INP_NO
417
+ nursing_note:
418
+ code: '"NURSING//NOTE" if $NURSING_DESC != "NA" and /\w/ in $NURSING_DESC'
419
+ time: $chart_time
420
+ text_value: $NURSING_DESC
421
+ inpatient_stay_id: $INP_NO
422
+ endotracheal_intubation_depth:
423
+ code: '"NURSING//ENDOTRACHEAL_INTUBATION_DEPTH" if $Endotracheal_intubation_Depth::?float64 >= 0'
424
+ time: $chart_time
425
+ numeric_value: $Endotracheal_intubation_Depth::?float64
426
+ inpatient_stay_id: $INP_NO
427
+ extubation:
428
+ code: '"NURSING//EXTUBATION" if $Extubation == true'
429
+ time: $chart_time
430
+ inpatient_stay_id: $INP_NO
431
+
432
+ # One row per patient. Follow-up answers are timed at Follow_Date; rows without one (247) produce
433
+ # no follow-up events.
434
+ dtOutCome:
435
+ _table:
436
+ join:
437
+ dtBaseline:
438
+ key: PATIENT_ID
439
+ cols: [INP_NO, DISCHARGE_DATE_TIME]
440
+ follow_up_vital_status:
441
+ code: 'f"FOLLOW_UP_VITAL_STATUS//{$Follow_Vital}" if $Follow_Vital != "NA" and $Follow_Vital != ""'
442
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
443
+ # Death_Date has about day precision: in-hospital deaths typically fall ~12 h before the recorded
444
+ # discharge, so a stay's last events can follow MEDS_DEATH.
445
+ death:
446
+ code: MEDS_DEATH
447
+ time: "(1970-01-01 @ 00:00) + $Death_Date::?float64::hours"
448
+ sf36_general_health:
449
+ code: 'f"SF36//GENERAL_HEALTH//{$SF36_GeneralHeath}" if $SF36_GeneralHeath != "NA" and $SF36_GeneralHeath != ""'
450
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
451
+ sf36_sicker_than_others:
452
+ code: 'f"SF36//SICKER_THAN_OTHERS//{$SF36_GH_SickerEasierThanOthers}" if $SF36_GH_SickerEasierThanOthers != "NA" and $SF36_GH_SickerEasierThanOthers != ""'
453
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
454
+ sf36_healthier_than_others:
455
+ code: 'f"SF36//HEALTHIER_THAN_OTHERS//{$SF36_GH_HeathyasOthers}" if $SF36_GH_HeathyasOthers != "NA" and $SF36_GH_HeathyasOthers != ""'
456
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
457
+ sf36_health_get_worse:
458
+ code: 'f"SF36//HEALTH_GET_WORSE//{$SF36_GH_GetWorse}" if $SF36_GH_GetWorse != "NA" and $SF36_GH_GetWorse != ""'
459
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
460
+ sf36_excellent_health:
461
+ code: 'f"SF36//EXCELLENT_HEALTH//{$SF36_GH_ExcellentHealth}" if $SF36_GH_ExcellentHealth != "NA" and $SF36_GH_ExcellentHealth != ""'
462
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
463
+ sf36_vigorous_activity:
464
+ code: 'f"SF36//ACTIVITY_LIMIT_VIGOROUS//{$SF36_ActivityLimit_VigorousActivity}" if $SF36_ActivityLimit_VigorousActivity != "NA" and $SF36_ActivityLimit_VigorousActivity != ""'
465
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
466
+ sf36_moderate_activity:
467
+ code: 'f"SF36//ACTIVITY_LIMIT_MODERATE//{$SF36_ActivityLimit_ModerateActivity}" if $SF36_ActivityLimit_ModerateActivity != "NA" and $SF36_ActivityLimit_ModerateActivity != ""'
468
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
469
+ sf36_lifting_groceries:
470
+ code: 'f"SF36//ACTIVITY_LIMIT_LIFTING_GROCERIES//{$SF36_ActivityLimit_LiftingGroceries}" if $SF36_ActivityLimit_LiftingGroceries != "NA" and $SF36_ActivityLimit_LiftingGroceries != ""'
471
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
472
+ sf36_climbing_several_flights:
473
+ code: 'f"SF36//ACTIVITY_LIMIT_CLIMBING_SEVERAL_FLIGHTS//{$SF36_ActivityLimit_ClimingSeveralFlights}" if $SF36_ActivityLimit_ClimingSeveralFlights != "NA" and $SF36_ActivityLimit_ClimingSeveralFlights != ""'
474
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
475
+ sf36_climbing_one_flight:
476
+ code: 'f"SF36//ACTIVITY_LIMIT_CLIMBING_ONE_FLIGHT//{$SF36_ActivityLimit_ClimingOneFlight}" if $SF36_ActivityLimit_ClimingOneFlight != "NA" and $SF36_ActivityLimit_ClimingOneFlight != ""'
477
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
478
+ sf36_bending_kneeling_stooping:
479
+ code: 'f"SF36//ACTIVITY_LIMIT_BENDING_KNEELING_STOOPING//{$SF36_ActivityLimit_BendingKneelStoop}" if $SF36_ActivityLimit_BendingKneelStoop != "NA" and $SF36_ActivityLimit_BendingKneelStoop != ""'
480
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
481
+ sf36_walking_more_than_1500m:
482
+ code: 'f"SF36//ACTIVITY_LIMIT_WALKING_MORE_THAN_1500M//{$SF36_ActivityLimit_WalkingMore1500M}" if $SF36_ActivityLimit_WalkingMore1500M != "NA" and $SF36_ActivityLimit_WalkingMore1500M != ""'
483
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
484
+ sf36_walking_1000m:
485
+ code: 'f"SF36//ACTIVITY_LIMIT_WALKING_1000M//{$SF36_ActivityLimit_Walking1000M}" if $SF36_ActivityLimit_Walking1000M != "NA" and $SF36_ActivityLimit_Walking1000M != ""'
486
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
487
+ sf36_walking_100m:
488
+ code: 'f"SF36//ACTIVITY_LIMIT_WALKING_100M//{$SF36_ActivityLimit_Walking100M}" if $SF36_ActivityLimit_Walking100M != "NA" and $SF36_ActivityLimit_Walking100M != ""'
489
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
490
+ sf36_bathing_dressing:
491
+ code: 'f"SF36//ACTIVITY_LIMIT_BATHING_DRESSING//{$SF36_ActivityLimit_BathingDressing}" if $SF36_ActivityLimit_BathingDressing != "NA" and $SF36_ActivityLimit_BathingDressing != ""'
492
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
493
+ sf36_body_pain:
494
+ code: 'f"SF36//BODY_PAIN//{$SF36_Pain_BodyPainPast4wk}" if $SF36_Pain_BodyPainPast4wk != "NA" and $SF36_Pain_BodyPainPast4wk != ""'
495
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
496
+ sf36_pain_interferes_housework:
497
+ code: 'f"SF36//PAIN_INTERFERES_HOUSEWORK//{$SF36_Pain_BodyPainPast4wkInterHousework}" if $SF36_Pain_BodyPainPast4wkInterHousework != "NA" and $SF36_Pain_BodyPainPast4wkInterHousework != ""'
498
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
499
+ sf36_physical_health_cut_work:
500
+ code: 'f"SF36//PHYSICAL_HEALTH_CUT_WORK//{$SF36_PhysicalHeath_CutWork}" if $SF36_PhysicalHeath_CutWork != "NA" and $SF36_PhysicalHeath_CutWork != ""'
501
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
502
+ sf36_physical_health_accomplish_less:
503
+ code: 'f"SF36//PHYSICAL_HEALTH_ACCOMPLISH_LESS//{$SF36_PhysicalHeath_AccomplishLess}" if $SF36_PhysicalHeath_AccomplishLess != "NA" and $SF36_PhysicalHeath_AccomplishLess != ""'
504
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
505
+ sf36_physical_health_limited_work:
506
+ code: 'f"SF36//PHYSICAL_HEALTH_LIMITED_WORK//{$SF36_PhysicalHeath_LimitedWork}" if $SF36_PhysicalHeath_LimitedWork != "NA" and $SF36_PhysicalHeath_LimitedWork != ""'
507
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
508
+ sf36_physical_health_difficult_work:
509
+ code: 'f"SF36//PHYSICAL_HEALTH_DIFFICULT_WORK//{$SF36_PhysicalHeath_DiffiPerformingWork}" if $SF36_PhysicalHeath_DiffiPerformingWork != "NA" and $SF36_PhysicalHeath_DiffiPerformingWork != ""'
510
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
511
+ sf36_emotional_nervous:
512
+ code: 'f"SF36//EMOTIONAL_NERVOUS//{$SF36_Emotional_Nervous}" if $SF36_Emotional_Nervous != "NA" and $SF36_Emotional_Nervous != ""'
513
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
514
+ sf36_emotional_down_nothing_cheer:
515
+ code: 'f"SF36//EMOTIONAL_DOWN_NOTHING_CHEER//{$SF36_Emotional_DownNothingCheer}" if $SF36_Emotional_DownNothingCheer != "NA" and $SF36_Emotional_DownNothingCheer != ""'
516
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
517
+ sf36_emotional_calm:
518
+ code: 'f"SF36//EMOTIONAL_CALM//{$SF36_Emotional_Calm}" if $SF36_Emotional_Calm != "NA" and $SF36_Emotional_Calm != ""'
519
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
520
+ sf36_emotional_downhearted:
521
+ code: 'f"SF36//EMOTIONAL_DOWNHEARTED//{$SF36_Emotional_DownHearted}" if $SF36_Emotional_DownHearted != "NA" and $SF36_Emotional_DownHearted != ""'
522
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
523
+ sf36_emotional_happy:
524
+ code: 'f"SF36//EMOTIONAL_HAPPY//{$SF36_Emotional_HappyPerson}" if $SF36_Emotional_HappyPerson != "NA" and $SF36_Emotional_HappyPerson != ""'
525
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
526
+ sf36_emotional_energy:
527
+ code: 'f"SF36//EMOTIONAL_ENERGY//{$SF36_Emotional_aLotEnergy}" if $SF36_Emotional_aLotEnergy != "NA" and $SF36_Emotional_aLotEnergy != ""'
528
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
529
+ sf36_emotional_pep:
530
+ code: 'f"SF36//EMOTIONAL_PEP//{$SF36_Emotional_FullPep}" if $SF36_Emotional_FullPep != "NA" and $SF36_Emotional_FullPep != ""'
531
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
532
+ sf36_emotional_worn_out:
533
+ code: 'f"SF36//EMOTIONAL_WORN_OUT//{$SF36_Emotional_WornOut}" if $SF36_Emotional_WornOut != "NA" and $SF36_Emotional_WornOut != ""'
534
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
535
+ sf36_emotional_tired:
536
+ code: 'f"SF36//EMOTIONAL_TIRED//{$SF36_Emotional_Tired}" if $SF36_Emotional_Tired != "NA" and $SF36_Emotional_Tired != ""'
537
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
538
+ sf36_social_activity_emotional_or_physical:
539
+ code: 'f"SF36//SOCIAL_ACTIVITY_EMOTIONAL_OR_PHYSICAL//{$SF36_4wksSocialActivityEmotionOrPhysical}" if $SF36_4wksSocialActivityEmotionOrPhysical != "NA" and $SF36_4wksSocialActivityEmotionOrPhysical != ""'
540
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
541
+ sf36_no_interference_social_activity:
542
+ code: 'f"SF36//NO_INTERFERENCE_SOCIAL_ACTIVITY//{$SF36_NoInfSocialActivity}" if $SF36_NoInfSocialActivity != "NA" and $SF36_NoInfSocialActivity != ""'
543
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
544
+ sf36_reduced_time_emotional_or_physical:
545
+ code: 'f"SF36//REDUCED_TIME_EMOTIONAL_OR_PHYSICAL//{$SF36_ReducedTime4wksSocialActivityEmotion}" if $SF36_ReducedTime4wksSocialActivityEmotion != "NA" and $SF36_ReducedTime4wksSocialActivityEmotion != ""'
546
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
547
+ sf36_reduced_amount_emotional_or_physical:
548
+ code: 'f"SF36//REDUCED_AMOUNT_EMOTIONAL_OR_PHYSICAL//{$SF36_ReducedAmount4wksSocialActivityEmotion}" if $SF36_ReducedAmount4wksSocialActivityEmotion != "NA" and $SF36_ReducedAmount4wksSocialActivityEmotion != ""'
549
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
550
+ sf36_reduced_quality_emotional_or_physical:
551
+ code: 'f"SF36//REDUCED_QUALITY_EMOTIONAL_OR_PHYSICAL//{$SF36_ReducedQuality4wksSocialActivityEmotion}" if $SF36_ReducedQuality4wksSocialActivityEmotion != "NA" and $SF36_ReducedQuality4wksSocialActivityEmotion != ""'
552
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
553
+ sf36_one_year_health_comparison:
554
+ code: 'f"SF36//ONE_YEAR_HEALTH_COMPARISON//{$SF36_OneYearCompareHealthCondition}" if $SF36_OneYearCompareHealthCondition != "NA" and $SF36_OneYearCompareHealthCondition != ""'
555
+ time: "(1970-01-01 @ 00:00) + $Follow_Date::?float64::hours"
556
+ discharge_department:
557
+ code: f"DISCHARGE_DEPARTMENT//{$Discharge_Dept}"
558
+ time: "(1970-01-01 @ 00:00) + $DISCHARGE_DATE_TIME::hours"
559
+ inpatient_stay_id: $INP_NO
560
+
561
+ sources:
562
+ # The raw release version. Reserved (never treated as a bucket), interpolatable into the entries
563
+ # below, and stamped by `meds-extract-run` into the output's `etl_metadata.dataset_version`.
564
+ dataset_version: "1.1"
565
+ dataset:
566
+ # Zigong is credentialed: a PhysioNet account with a signed DUA for this project is required.
567
+ - type: physionet
568
+ base_url: https://physionet.org/files/icu-infection-zigong-fourth/${sources.dataset_version}
569
+ username: ${oc.env:DATASET_DOWNLOAD_USERNAME}
570
+ password: ${oc.env:DATASET_DOWNLOAD_PASSWORD}
@@ -0,0 +1,177 @@
1
+ Metadata-Version: 2.4
2
+ Name: Zigong-MEDS
3
+ Version: 0.3.0
4
+ Summary: An ETL pipeline to extract the Zigong Fourth People's Hospital ICU infection database into the MEDS format.
5
+ Author-email: Robin van de Water <rpv2114@cumc.columbia.edu>
6
+ Project-URL: Homepage, https://github.com/rvandewater/ZIGONG_MEDS
7
+ Project-URL: Issues, https://github.com/rvandewater/ZIGONG_MEDS/issues
8
+ Project-URL: Dataset, https://physionet.org/content/icu-infection-zigong-fourth/1.1/
9
+ Classifier: Programming Language :: Python :: 3
10
+ Classifier: License :: OSI Approved :: MIT License
11
+ Classifier: Operating System :: OS Independent
12
+ Requires-Python: <3.14,>=3.11.4
13
+ Description-Content-Type: text/markdown
14
+ License-File: LICENSE
15
+ Requires-Dist: meds-extract[download]<0.8,>=0.7.0
16
+ Dynamic: license-file
17
+
18
+ # Zigong MEDS Extraction ETL
19
+
20
+ [![tests](https://github.com/rvandewater/ZIGONG_MEDS/actions/workflows/tests.yaml/badge.svg)](https://github.com/rvandewater/ZIGONG_MEDS/actions/workflows/tests.yaml)
21
+ [![synthetic ETL](https://github.com/rvandewater/ZIGONG_MEDS/actions/workflows/synthetic-etl.yaml/badge.svg)](https://github.com/rvandewater/ZIGONG_MEDS/actions/workflows/synthetic-etl.yaml)
22
+ [![code-quality](https://github.com/rvandewater/ZIGONG_MEDS/actions/workflows/code-quality-main.yaml/badge.svg)](https://github.com/rvandewater/ZIGONG_MEDS/actions/workflows/code-quality-main.yaml)
23
+ ![python](https://img.shields.io/badge/-Python_3.11--3.13-blue?logo=python&logoColor=white)
24
+ [![MEDS-Extract 0.7](https://img.shields.io/badge/MEDS--Extract-0.7-blue)](https://github.com/mmcdermott/MEDS_extract)
25
+ [![license](https://img.shields.io/badge/License-MIT-green.svg?labelColor=gray)](LICENSE)
26
+
27
+ This pipeline extracts the
28
+ [Zigong Fourth People's Hospital ICU infection database v1.1](https://physionet.org/content/icu-infection-zigong-fourth/1.1/)
29
+ from PhysioNet into the [MEDS format](https://medical-event-data-standard.github.io/). The release holds
30
+ 2,790 ICU patients with infection, one hospital admission each, from Zigong Fourth People's Hospital in
31
+ Sichuan, China. It was built from the
32
+ [ETL_MEDS_Template](https://github.com/Medical-Event-Data-Standard/ETL_MEDS_Template) and uses the
33
+ config-only layout introduced by [MEDS-Extract](https://github.com/mmcdermott/MEDS_extract) 0.7.
34
+
35
+ ## Usage
36
+
37
+ Zigong is a credentialed dataset. You need a PhysioNet account that has signed the project's data use
38
+ agreement.
39
+
40
+ ```bash
41
+ pip install git+https://github.com/rvandewater/ZIGONG_MEDS.git
42
+ export DATASET_DOWNLOAD_USERNAME=$PHYSIONET_USERNAME
43
+ export DATASET_DOWNLOAD_PASSWORD=$PHYSIONET_PASSWORD
44
+ meds-extract-run spec=Zigong output_dir=$MEDS_OUTPUT_DIR
45
+ ```
46
+
47
+ When you run this, the program will:
48
+
49
+ 1. Download the release (seven `dt*.csv` tables, about 840 MB) from PhysioNet into
50
+ `$MEDS_OUTPUT_DIR/.meds_extract_run/raw_input`. Pass `download_dest_dir=$RAW_INPUT_DIR` to keep the raw
51
+ data somewhere durable and reuse it across runs.
52
+ 2. Build the MEDS cohort from those files and write it to `$MEDS_OUTPUT_DIR` (`data/` and `metadata/`).
53
+ All transformations and joins are declared in
54
+ [`src/Zigong_MEDS/event_configs.yaml`](src/Zigong_MEDS/event_configs.yaml).
55
+
56
+ If you already have the PhysioNet release on disk, skip the download and point `input_dir` at the
57
+ directory that contains `dtBaseline.csv`, `dtDrugs.csv`, `dtICD.csv`, `dtLab.csv`, `dtNursingChart.csv`,
58
+ `dtOutCome.csv` and `dtTransfer.csv`:
59
+
60
+ ```bash
61
+ meds-extract-run spec=Zigong output_dir=$MEDS_OUTPUT_DIR \
62
+ do_download=false input_dir=/path/to/icu-infection-zigong-fourth-1.1
63
+ ```
64
+
65
+ To parallelize the stages, pass a runner file:
66
+
67
+ ```bash
68
+ printf 'parallelize:\n n_workers: 8\n launcher: joblib\n' >runner.yaml
69
+ meds-extract-run spec=Zigong output_dir=$MEDS_OUTPUT_DIR stage_runner_fp=runner.yaml
70
+ ```
71
+
72
+ Run `meds-extract-run --help` for every option.
73
+
74
+ ## How this ETL is defined
75
+
76
+ The package contains no Python. The whole ETL is one MESSY (MEDS-Extract Specification Syntax YAML)
77
+ file, [`src/Zigong_MEDS/event_configs.yaml`](src/Zigong_MEDS/event_configs.yaml). It declares:
78
+
79
+ - **`sources:`**: where the raw data lives (the PhysioNet release).
80
+ - **`etl:`**: the dataset name and sharding options.
81
+ - **One block per raw table**: how each row becomes one or more MEDS events.
82
+
83
+ `pyproject.toml` registers the file with MEDS-Extract's `MEDS_extract.pipelines` entry-point group under
84
+ the name `Zigong`. That registration is what makes `spec=Zigong` work. No pre-MEDS step is needed:
85
+ tables without `PATIENT_ID` (`dtLab`, `dtNursingChart`) are joined to `dtBaseline` on `INP_NO` inside the
86
+ config.
87
+
88
+ **Times.** Every source time is an offset in hours from hospital admission. The ETL places them on a
89
+ shared pseudotime anchor, 1970-01-01 00:00 = admission, which preserves order and intervals without
90
+ claiming a real calendar date. Each patient has exactly one admission, so offsets never collide.
91
+
92
+ ## Events
93
+
94
+ | Table | Event codes | Notes |
95
+ | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
96
+ | `dtBaseline` | `HOSPITAL_ADMISSION`, `AGE_AT_ADMISSION`, `SEX//…`, `ADMISSION_DEPARTMENT//…`, `INFECTION_SITE//…`, `ICU_DISCHARGE`, `HOSPITAL_DISCHARGE` | Age is the `numeric_value`. |
97
+ | `dtTransfer` | `TRANSFER_TO//{dept}`, `TRANSFER_FROM//{dept}` | The reason is kept in `transfer_reason`. See [#3](https://github.com/rvandewater/ZIGONG_MEDS/issues/3). |
98
+ | `dtICD` | `DIAGNOSIS//ICD//{code}` | Placed at hospital discharge. The clinician's description is the `text_value`; `diagnosis_sequence` and `discharge_diagnosis_status` are kept. |
99
+ | `dtLab` | `LAB//{category}//{item}//{unit}` | A missing unit becomes `UNK`. `reference_range` is kept. |
100
+ | `dtDrugs` | `MEDICATION//{drug}` | Prescription time. `Formula` is the `text_value`. |
101
+ | `dtNursingChart` | `NURSING//{MEASUREMENT}` (vitals, ventilation, consciousness, GCS components, lines and tubes, risk and pain scores, intake and output, notes, …) | One code per charted column. Vital signs carry a `unit` column. GCS components carry the score as the `numeric_value` and the charted label as text. |
102
+ | `dtOutCome` | `MEDS_DEATH`, `FOLLOW_UP_VITAL_STATUS//…`, `SF36//{ITEM}//{answer}`, `DISCHARGE_DEPARTMENT//…` | Follow-up events are placed at `Follow_Date`; the discharge department at hospital discharge. |
103
+
104
+ Every event linked to a stay carries `inpatient_stay_id` (`INP_NO`). The R-written CSVs encode missing
105
+ values as the string `NA` or a quoted empty string; both are treated as missing, as are text entries with
106
+ no letters or digits.
107
+
108
+ ## Data notes
109
+
110
+ - **Coarse nursing chart timestamps.** `dtNursingChart` has about 1 million rows with a usable
111
+ `ChartTime` but only about 23,000 distinct (patient, `ChartTime`) pairs. A typical timestamp covers 26
112
+ source rows, and the largest covers 266. This comes from the source release, not the ETL. In the MEDS
113
+ output, one timestamp therefore holds many readings of the same measurement, on average about 15
114
+ `NURSING//HEART_RATE` values, and similarly for blood pressure, SpO2 and other vitals. The order of events
115
+ within a timestamp carries no clinical meaning. When defining tasks or features:
116
+ - Do not rely on order within a timestamp, such as "last value before X", minute-level features, or
117
+ sequence models that treat event order as real time order.
118
+ - Choose prediction times that do not fall inside a timestamp's group of events, and aggregate values
119
+ within a timestamp (for example mean, min, or max) rather than picking one.
120
+ - **Dropped nursing rows.** 313,974 nursing rows (23%) belong to 723 stays that are not in the cohort
121
+ (`dtBaseline`), and all of them have `ChartTime = NA`; they are dropped. Rows charted more than 24 h before
122
+ admission (9,011 rows, including some corrupt offsets) or after discharge (141) are dropped too.
123
+ - **Dropped labs.** Results timed after hospital discharge (4,415 rows, 0.3%) and placeholder values made
124
+ only of dots or asterisks are dropped.
125
+ - **Coarse death dates.** `Death_Date` has about day precision. In-hospital deaths typically fall about 12 h
126
+ before the recorded discharge, so a stay's last events can follow `MEDS_DEATH`. See
127
+ [#4](https://github.com/rvandewater/ZIGONG_MEDS/issues/4).
128
+ - **Line and tube columns** (`CVC`, `PICC`, `Gastric_tube`, …) hold the length in cm, used as the
129
+ `numeric_value`, or a status such as "removed" kept as text. The numeric values of `Endotracheal_intubation`
130
+ duplicate `Endotracheal_intubation_Depth`, so only its statuses are emitted.
131
+ - **Bounded scores.** RASS, NRS and CPOT values outside their scales (-5..+4, 0..10, 0..8) are dropped.
132
+ Other vitals are not range-filtered; see [#7](https://github.com/rvandewater/ZIGONG_MEDS/issues/7).
133
+ - **Unmapped columns.** Five nursing columns are NA in every row. `MAP`, `PSIPAP`, `R` and `FIO2` hold only
134
+ unit strings. `Invasive_SBP._1` is a documented technical duplicate, and `F` is documented as having no
135
+ clinical relevance. See [#6](https://github.com/rvandewater/ZIGONG_MEDS/issues/6).
136
+ - **Local vocabularies.** Diagnoses use the release's Chinese national extension of ICD-10, and labs and
137
+ drugs use local names. No standard-terminology parents are attached yet; see
138
+ [#1](https://github.com/rvandewater/ZIGONG_MEDS/issues/1) and
139
+ [#8](https://github.com/rvandewater/ZIGONG_MEDS/issues/8). SF-36 answers are inconsistently coded in the
140
+ source; see [#5](https://github.com/rvandewater/ZIGONG_MEDS/issues/5).
141
+ - **No ICU admission event.** The release records ICU discharge but not ICU entry; see
142
+ [#2](https://github.com/rvandewater/ZIGONG_MEDS/issues/2).
143
+
144
+ ## Testing
145
+
146
+ Zigong has no open demo release, so there are two synthetic checks:
147
+
148
+ - `tests/e2e_synthetic_test.py` writes a few fully synthetic rows with the release's exact headers and R
149
+ `write.csv` encoding, runs `meds-extract-run spec=Zigong` over them, and checks the events, times, values
150
+ and missing-value handling it produces.
151
+ - The [Synthetic ETL workflow](.github/workflows/synthetic-etl.yaml) generates raw files straight from the
152
+ config with [MESSY-synth](https://pypi.org/project/MESSY-synth/) and runs the ETL over them. It fails if
153
+ any event block produces no events.
154
+
155
+ ```bash
156
+ uv sync --group dev
157
+ uv run pytest tests/ -v
158
+ uv run pre-commit run --all-files
159
+
160
+ # MESSY-synth needs Python >= 3.12.
161
+ uv run --with MESSY-synth==0.0.1 MESSY-synth ./src/Zigong_MEDS/event_configs.yaml -o raw_synthetic/
162
+ uv run meds-extract-run spec=Zigong output_dir=zigong_meds_synth/ \
163
+ do_download=false input_dir=raw_synthetic/
164
+ ```
165
+
166
+ For a reusable checklist on mapping a new dataset, see [docs/MEDS_ETL_MAPPING.md](docs/MEDS_ETL_MAPPING.md).
167
+
168
+ ## Citation
169
+
170
+ If you use this ETL, please cite the Zigong database and its descriptor paper:
171
+
172
+ - Xu, P., Chen, L., & Zhang, Z. (2022). *Critical care database comprising patients with infection at
173
+ Zigong Fourth People's Hospital* (version 1.1). PhysioNet. https://doi.org/10.13026/xpt9-z726
174
+ - Xu, P., Chen, L., Zhu, Y. et al. Critical Care Database Comprising Patients With Infection. *Front.
175
+ Public Health* 10 (2022). https://doi.org/10.3389/fpubh.2022.852410
176
+
177
+ See also [`CITATION.cff`](CITATION.cff).
@@ -0,0 +1,12 @@
1
+ LICENSE
2
+ README.md
3
+ pyproject.toml
4
+ src/Zigong_MEDS/__init__.py
5
+ src/Zigong_MEDS/event_configs.yaml
6
+ src/Zigong_MEDS.egg-info/PKG-INFO
7
+ src/Zigong_MEDS.egg-info/SOURCES.txt
8
+ src/Zigong_MEDS.egg-info/dependency_links.txt
9
+ src/Zigong_MEDS.egg-info/entry_points.txt
10
+ src/Zigong_MEDS.egg-info/requires.txt
11
+ src/Zigong_MEDS.egg-info/top_level.txt
12
+ tests/test_config.py
@@ -0,0 +1,2 @@
1
+ [MEDS_extract.pipelines]
2
+ Zigong = Zigong_MEDS:event_configs.yaml
@@ -0,0 +1 @@
1
+ meds-extract[download]<0.8,>=0.7.0
@@ -0,0 +1 @@
1
+ Zigong_MEDS
@@ -0,0 +1,17 @@
1
+ from MEDS_extract.config import MessyConfig
2
+
3
+
4
+ def test_zigong_config_loads_and_covers_release_tables():
5
+ config = MessyConfig.load("Zigong")
6
+
7
+ assert config.dataset_name == "Zigong"
8
+ assert set(config.table_prefixes) == {
9
+ "dtBaseline",
10
+ "dtDrugs",
11
+ "dtICD",
12
+ "dtLab",
13
+ "dtNursingChart",
14
+ "dtOutCome",
15
+ "dtTransfer",
16
+ }
17
+ assert config.raw_version_for() == "1.1"