geniekit 0.0.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- geniekit-0.0.1/CHANGELOG.md +32 -0
- geniekit-0.0.1/LICENSE +21 -0
- geniekit-0.0.1/MANIFEST.in +7 -0
- geniekit-0.0.1/NOTICE.md +50 -0
- geniekit-0.0.1/PKG-INFO +230 -0
- geniekit-0.0.1/README.md +187 -0
- geniekit-0.0.1/data-raw/README.md +21 -0
- geniekit-0.0.1/data-raw/build_dictionary.py +168 -0
- geniekit-0.0.1/data-raw/build_releases.py +326 -0
- geniekit-0.0.1/examples/synthetic_cohorts.py +29 -0
- geniekit-0.0.1/pyproject.toml +86 -0
- geniekit-0.0.1/scripts/check_vocabulary.py +107 -0
- geniekit-0.0.1/setup.cfg +4 -0
- geniekit-0.0.1/src/geniekit/__init__.py +30 -0
- geniekit-0.0.1/src/geniekit/_version.py +8 -0
- geniekit-0.0.1/src/geniekit/cli.py +200 -0
- geniekit-0.0.1/src/geniekit/dictionary.py +317 -0
- geniekit-0.0.1/src/geniekit/io/__init__.py +36 -0
- geniekit-0.0.1/src/geniekit/io/cohort.py +161 -0
- geniekit-0.0.1/src/geniekit/io/local.py +275 -0
- geniekit-0.0.1/src/geniekit/io/synapse.py +334 -0
- geniekit-0.0.1/src/geniekit/releases.py +318 -0
- geniekit-0.0.1/src/geniekit/resources/__init__.py +10 -0
- geniekit-0.0.1/src/geniekit/resources/dictionary.csv.gz +0 -0
- geniekit-0.0.1/src/geniekit/resources/dictionary.meta.json +13 -0
- geniekit-0.0.1/src/geniekit/resources/releases.json +1880 -0
- geniekit-0.0.1/src/geniekit/synthetic/__init__.py +15 -0
- geniekit-0.0.1/src/geniekit/synthetic/generate.py +1272 -0
- geniekit-0.0.1/src/geniekit/synthetic/spec.py +331 -0
- geniekit-0.0.1/src/geniekit.egg-info/PKG-INFO +230 -0
- geniekit-0.0.1/src/geniekit.egg-info/SOURCES.txt +53 -0
- geniekit-0.0.1/src/geniekit.egg-info/dependency_links.txt +1 -0
- geniekit-0.0.1/src/geniekit.egg-info/entry_points.txt +2 -0
- geniekit-0.0.1/src/geniekit.egg-info/requires.txt +22 -0
- geniekit-0.0.1/src/geniekit.egg-info/top_level.txt +1 -0
- geniekit-0.0.1/tests/__init__.py +0 -0
- geniekit-0.0.1/tests/conftest.py +21 -0
- geniekit-0.0.1/tests/fixtures/README.md +24 -0
- geniekit-0.0.1/tests/fixtures/nsclc_sim/cBioPortal_files/data_CNA.txt.gz +0 -0
- geniekit-0.0.1/tests/fixtures/nsclc_sim/cBioPortal_files/data_fusions.txt.gz +0 -0
- geniekit-0.0.1/tests/fixtures/nsclc_sim/cBioPortal_files/data_mutations_extended.txt.gz +0 -0
- geniekit-0.0.1/tests/fixtures/nsclc_sim/clinical_data/cancer_level_dataset_index.csv.gz +0 -0
- geniekit-0.0.1/tests/fixtures/nsclc_sim/clinical_data/cancer_level_dataset_non_index.csv.gz +0 -0
- geniekit-0.0.1/tests/fixtures/nsclc_sim/clinical_data/cancer_panel_test_level_dataset.csv.gz +0 -0
- geniekit-0.0.1/tests/fixtures/nsclc_sim/clinical_data/imaging_level_dataset.csv.gz +0 -0
- geniekit-0.0.1/tests/fixtures/nsclc_sim/clinical_data/med_onc_note_level_dataset.csv.gz +0 -0
- geniekit-0.0.1/tests/fixtures/nsclc_sim/clinical_data/pathology_report_level_dataset.csv.gz +0 -0
- geniekit-0.0.1/tests/fixtures/nsclc_sim/clinical_data/patient_level_dataset.csv.gz +0 -0
- geniekit-0.0.1/tests/fixtures/nsclc_sim/clinical_data/regimen_cancer_level_dataset.csv.gz +0 -0
- geniekit-0.0.1/tests/test_cli.py +94 -0
- geniekit-0.0.1/tests/test_dictionary.py +133 -0
- geniekit-0.0.1/tests/test_io_local.py +171 -0
- geniekit-0.0.1/tests/test_io_synapse.py +205 -0
- geniekit-0.0.1/tests/test_releases.py +155 -0
- geniekit-0.0.1/tests/test_synthetic.py +209 -0
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to geniekit are recorded here. The format follows
|
|
4
|
+
Keep a Changelog, and the project follows semantic versioning.
|
|
5
|
+
|
|
6
|
+
## [Unreleased]
|
|
7
|
+
|
|
8
|
+
## [0.0.1] - 2026-10-17
|
|
9
|
+
|
|
10
|
+
First public release: the foundation modules.
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
- `geniekit.releases`: registry of the six public GENIE BPC releases (NSCLC
|
|
14
|
+
v2.0, CRC v2.0, BrCa v1.0, PANC v1.0, Prostate v1.0, BLADDER v1.0) with the
|
|
15
|
+
Synapse id of every clinical, genomic, cBioPortal and documentation file,
|
|
16
|
+
generated from the public Synapse entity tree.
|
|
17
|
+
- `geniekit.dictionary`: the six Variable Synopsis workbooks compiled into one
|
|
18
|
+
data dictionary, with presence matrices and cross-cohort comparison of
|
|
19
|
+
types, labels and permitted values.
|
|
20
|
+
- `geniekit` command line interface with `releases` and `dictionary`
|
|
21
|
+
subcommands.
|
|
22
|
+
- Build scripts in `data-raw/` that regenerate the packaged metadata.
|
|
23
|
+
- `geniekit.io`: `CohortData` container, `load_release` for release folders on
|
|
24
|
+
disk (Synapse layout, geniekit download layout, flat folders, gzip), and a
|
|
25
|
+
Synapse client (`SynapseSession`, `download_release`) with token
|
|
26
|
+
resolution, access-status reporting and MD5-verified downloads.
|
|
27
|
+
- `geniekit.synthetic`: dictionary-driven synthetic cohorts for all six
|
|
28
|
+
public releases, with coherent patient histories, regimens, imaging and
|
|
29
|
+
oncologist assessments, sequencing samples and genomic tables.
|
|
30
|
+
- CLI commands `access`, `download` and `inspect`.
|
|
31
|
+
- Test fixture `tests/fixtures/nsclc_sim` derived from the simulated data
|
|
32
|
+
shipped with the R package genieBPC (MIT).
|
geniekit-0.0.1/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Muntaser Syed
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
include README.md CHANGELOG.md NOTICE.md LICENSE MANIFEST.in
|
|
2
|
+
recursive-include src/geniekit/resources *.json *.csv.gz
|
|
3
|
+
recursive-include tests *.py *.md *.gz
|
|
4
|
+
recursive-include data-raw *.py *.md
|
|
5
|
+
recursive-include scripts *.py
|
|
6
|
+
recursive-include examples *.py
|
|
7
|
+
prune data-raw/cache
|
geniekit-0.0.1/NOTICE.md
ADDED
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
# Notice
|
|
2
|
+
|
|
3
|
+
geniekit is an independent, community-developed tool. It is not produced,
|
|
4
|
+
endorsed or maintained by the American Association for Cancer Research (AACR),
|
|
5
|
+
AACR Project GENIE, Sage Bionetworks or any GENIE-participating institution.
|
|
6
|
+
|
|
7
|
+
## Data
|
|
8
|
+
|
|
9
|
+
geniekit ships no patient-level data. All GENIE and GENIE BPC data are
|
|
10
|
+
obtained by the user from Synapse (https://www.synapse.org) under the GENIE
|
|
11
|
+
access requirement and the terms of use that accompany each release. Users are
|
|
12
|
+
responsible for complying with those terms, including restrictions on
|
|
13
|
+
redistribution and on attempts to re-identify individuals.
|
|
14
|
+
|
|
15
|
+
## Packaged metadata and its sources
|
|
16
|
+
|
|
17
|
+
The files in `src/geniekit/resources/` are derived from public materials:
|
|
18
|
+
|
|
19
|
+
- `releases.json` is generated from the public entity tree of the GENIE BPC
|
|
20
|
+
"Data Releases" folder on Synapse (syn27056700) through the Synapse REST API,
|
|
21
|
+
plus cohort facts published on the AACR Project GENIE website
|
|
22
|
+
(https://aacrprojectgenie.org/data/).
|
|
23
|
+
- `dictionary.csv.gz` is compiled from the Variable Synopsis workbooks that
|
|
24
|
+
AACR publishes with each public BPC release. The workbooks are linked from
|
|
25
|
+
the cohort pages on https://aacrprojectgenie.org/data/ and are the
|
|
26
|
+
authoritative description of each release; geniekit reproduces variable
|
|
27
|
+
names, field labels, data types and permitted values so that software can
|
|
28
|
+
reason about them. Consult the Analytic Data Guide of each release for the
|
|
29
|
+
full definitions.
|
|
30
|
+
|
|
31
|
+
## Test fixture
|
|
32
|
+
|
|
33
|
+
`tests/fixtures/nsclc_sim/` is derived from the `nsclc_test_data` object of
|
|
34
|
+
the R package genieBPC (https://github.com/GENIE-BPC/genieBPC, MIT licence,
|
|
35
|
+
copyright 2022 genieBPC authors), which its authors describe as simulated fake
|
|
36
|
+
data for examples and tests. It contains no real patient records. See
|
|
37
|
+
`tests/fixtures/README.md`.
|
|
38
|
+
|
|
39
|
+
AACR Project GENIE and GENIE BPC are trademarks of their respective owners.
|
|
40
|
+
|
|
41
|
+
## Citing the data
|
|
42
|
+
|
|
43
|
+
Users of GENIE data are asked by AACR to cite:
|
|
44
|
+
|
|
45
|
+
AACR Project GENIE Consortium. AACR Project GENIE: Powering Precision Medicine
|
|
46
|
+
through an International Consortium. Cancer Discovery 2017;7(8):818-831.
|
|
47
|
+
https://doi.org/10.1158/2159-8290.CD-17-0151
|
|
48
|
+
|
|
49
|
+
and, for BPC cohorts, the cohort-specific publication listed on the cohort's
|
|
50
|
+
page at https://aacrprojectgenie.org/data/.
|
geniekit-0.0.1/PKG-INFO
ADDED
|
@@ -0,0 +1,230 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: geniekit
|
|
3
|
+
Version: 0.0.1
|
|
4
|
+
Summary: Harmonise the AACR Project GENIE BPC cohorts, compute lines of therapy, and run aggregate-only analyses.
|
|
5
|
+
Author-email: Muntaser Syed <jemsbhai@gmail.com>
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/jemsbhai/geniekit
|
|
8
|
+
Project-URL: Issues, https://github.com/jemsbhai/geniekit/issues
|
|
9
|
+
Project-URL: AACR Project GENIE, https://aacrprojectgenie.org/
|
|
10
|
+
Keywords: AACR,GENIE,BPC,oncology,clinicogenomic,line of therapy,harmonization,Synapse
|
|
11
|
+
Classifier: Development Status :: 3 - Alpha
|
|
12
|
+
Classifier: Intended Audience :: Science/Research
|
|
13
|
+
Classifier: Programming Language :: Python :: 3
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
18
|
+
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
|
|
19
|
+
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
|
|
20
|
+
Requires-Python: >=3.10
|
|
21
|
+
Description-Content-Type: text/markdown
|
|
22
|
+
License-File: LICENSE
|
|
23
|
+
License-File: NOTICE.md
|
|
24
|
+
Requires-Dist: pandas<4,>=2.2
|
|
25
|
+
Requires-Dist: numpy<3,>=1.26
|
|
26
|
+
Requires-Dist: requests<3,>=2.32
|
|
27
|
+
Requires-Dist: pyyaml<7,>=6.0
|
|
28
|
+
Provides-Extra: synapse
|
|
29
|
+
Requires-Dist: synapseclient<5,>=4.4; extra == "synapse"
|
|
30
|
+
Provides-Extra: survival
|
|
31
|
+
Requires-Dist: lifelines<1,>=0.29; extra == "survival"
|
|
32
|
+
Provides-Extra: notebook
|
|
33
|
+
Requires-Dist: jupyter<2,>=1.1; extra == "notebook"
|
|
34
|
+
Requires-Dist: matplotlib<4,>=3.9; extra == "notebook"
|
|
35
|
+
Provides-Extra: dev
|
|
36
|
+
Requires-Dist: pytest<9,>=8.3; extra == "dev"
|
|
37
|
+
Requires-Dist: pytest-cov<8,>=6.0; extra == "dev"
|
|
38
|
+
Requires-Dist: ruff<1,>=0.9; extra == "dev"
|
|
39
|
+
Requires-Dist: openpyxl<4,>=3.1; extra == "dev"
|
|
40
|
+
Requires-Dist: build<2,>=1.2; extra == "dev"
|
|
41
|
+
Requires-Dist: twine<7,>=6.0; extra == "dev"
|
|
42
|
+
Dynamic: license-file
|
|
43
|
+
|
|
44
|
+
# geniekit
|
|
45
|
+
|
|
46
|
+
Python tools for the AACR Project GENIE Biopharma Collaborative (BPC)
|
|
47
|
+
clinicogenomic cohorts.
|
|
48
|
+
|
|
49
|
+
- **Harmonise** the six public BPC cohorts (NSCLC, CRC, BrCa, PANC, Prostate,
|
|
50
|
+
BLADDER) to one schema so that cross-cohort analyses do not have to deal
|
|
51
|
+
with per-cohort variable names, types and code lists.
|
|
52
|
+
- **Lines of therapy**: derive first and subsequent lines from the regimen
|
|
53
|
+
data relative to an index event that the user chooses (diagnosis,
|
|
54
|
+
metastatic diagnosis, first progression, first systemic therapy, or any
|
|
55
|
+
custom event), with explicit, configurable rules.
|
|
56
|
+
- **Aggregate-only analytic layer**: produce cohort-level summaries with
|
|
57
|
+
disclosure control (minimum cell sizes, suppression, no row-level output)
|
|
58
|
+
for users who may not access patient-level records.
|
|
59
|
+
|
|
60
|
+
geniekit ships metadata only. The data themselves come from Synapse under the
|
|
61
|
+
GENIE access requirement (see [Data access](#data-access)).
|
|
62
|
+
|
|
63
|
+
## Status
|
|
64
|
+
|
|
65
|
+
Version 0.0.1 is the first public release and contains the foundation: the
|
|
66
|
+
release registry, the data dictionary, data loading and download, and the
|
|
67
|
+
synthetic cohorts. The harmonisation, line-of-therapy and aggregate layers
|
|
68
|
+
follow in the next minor releases.
|
|
69
|
+
|
|
70
|
+
| Module | Purpose | State |
|
|
71
|
+
|---|---|---|
|
|
72
|
+
| `geniekit.releases` | Registry of the six public BPC releases with the Synapse id of every clinical, genomic, cBioPortal and documentation file | done |
|
|
73
|
+
| `geniekit.dictionary` | The six Variable Synopsis workbooks compiled into one data dictionary, with cross-cohort comparison helpers | done |
|
|
74
|
+
| `geniekit.io` | Download from Synapse (personal access token, access-status check, MD5 verification) and load release folders from disk into one `CohortData` object | done |
|
|
75
|
+
| `geniekit.synthetic` | Dictionary-driven synthetic cohorts: every declared column, cohort-specific naming, coherent simulated histories | done |
|
|
76
|
+
| `geniekit.harmonize` | Crosswalk-driven normalisation of the six cohorts | planned |
|
|
77
|
+
| `geniekit.lot` | Line-of-therapy derivation with user-defined index events | planned |
|
|
78
|
+
| `geniekit.aggregate` | Aggregate-only analyses with disclosure control | planned |
|
|
79
|
+
|
|
80
|
+
## Installation
|
|
81
|
+
|
|
82
|
+
```powershell
|
|
83
|
+
pip install geniekit
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
Optional extras: `geniekit[synapse]` (official Synapse client), `geniekit[survival]`
|
|
87
|
+
(Kaplan-Meier summaries), `geniekit[notebook]` (Jupyter and plotting),
|
|
88
|
+
`geniekit[dev]` (tests and linting).
|
|
89
|
+
|
|
90
|
+
From a clone:
|
|
91
|
+
|
|
92
|
+
```powershell
|
|
93
|
+
pip install -e ".[dev]"
|
|
94
|
+
pytest tests/ -v --cov=geniekit --cov-report=term
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
## Quick start
|
|
98
|
+
|
|
99
|
+
```python
|
|
100
|
+
from geniekit import releases, dictionary
|
|
101
|
+
|
|
102
|
+
# What is available, and where on Synapse
|
|
103
|
+
for r in releases.list_releases():
|
|
104
|
+
print(r.label, r.n_patients, r.release_folder, sorted(r.clinical))
|
|
105
|
+
|
|
106
|
+
# How the cohorts differ before any harmonisation
|
|
107
|
+
dd = dictionary.load()
|
|
108
|
+
print(dd.compare("ca_cigarette", table="cancer").summary())
|
|
109
|
+
print(dd.differences(table="patient").head())
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
Loading data (after approval, see [Data access](#data-access)):
|
|
113
|
+
|
|
114
|
+
```python
|
|
115
|
+
from geniekit import io
|
|
116
|
+
|
|
117
|
+
# download one release with a personal access token in SYNAPSE_AUTH_TOKEN
|
|
118
|
+
folder = io.download_release("CRC", dest="data/genie_bpc") # data/genie_bpc/CRC_v2.0-public
|
|
119
|
+
crc = io.load_release(folder) # CohortData with 12 tables
|
|
120
|
+
crc["regimen"].head()
|
|
121
|
+
|
|
122
|
+
# or point at a folder you downloaded through the Synapse website
|
|
123
|
+
nsclc = io.load_release("downloads/NSCLC_2.0-public")
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
Working without the data:
|
|
127
|
+
|
|
128
|
+
```python
|
|
129
|
+
from geniekit import synthetic
|
|
130
|
+
|
|
131
|
+
fake = synthetic.generate("BLADDER", n_patients=300, seed=0) # same tables and columns as the real release
|
|
132
|
+
fake["cancer_index"][["record_id", "stage_dx", "dist_mets_liver", "tt_os_dx_days"]].head()
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
Synthetic cohorts carry the exact column set of the corresponding release
|
|
136
|
+
(checked against the dictionary in the test suite) and mutually consistent
|
|
137
|
+
identifiers, intervals and outcomes, but their values are invented. They
|
|
138
|
+
exist so that code can be written and tested before access is granted; never
|
|
139
|
+
report numbers computed from them.
|
|
140
|
+
|
|
141
|
+
Command line:
|
|
142
|
+
|
|
143
|
+
```powershell
|
|
144
|
+
geniekit releases --files
|
|
145
|
+
geniekit dictionary lookup naaccr_seer_sum_stage --table cancer --values
|
|
146
|
+
geniekit dictionary diff --table cancer --kind dtype --kind value_set
|
|
147
|
+
geniekit dictionary presence --table patient --only-partial
|
|
148
|
+
geniekit access # token and GENIE approval status
|
|
149
|
+
geniekit download CRC --dest data/genie_bpc # after approval
|
|
150
|
+
geniekit inspect data/genie_bpc/CRC_v2.0-public # table summary and column check
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
## The six public releases
|
|
154
|
+
|
|
155
|
+
| Cohort | Version | Patients | Institutions | Synapse | Schema generation |
|
|
156
|
+
|---|---|---|---|---|---|
|
|
157
|
+
| NSCLC | v2.0-public | 1846 | DFCI, MSK, UHN, VICC | syn27056697 | 1 |
|
|
158
|
+
| CRC | v2.0-public | 1485 | DFCI, MSK, VICC | syn30991637 | 1 |
|
|
159
|
+
| BrCa | v1.0-public | 1130 | DFCI, MSK, VICC | syn71825332 | 2 |
|
|
160
|
+
| PANC | v1.0-public | 1109 | DFCI, MSK, UHN, VICC | syn72666404 | 2 |
|
|
161
|
+
| Prostate | v1.0-public | 1116 | DFCI, MSK, UHN, VICC | syn73719283 | 2 |
|
|
162
|
+
| BLADDER | v1.0-public | 716 | DFCI, MSK, UHN, VICC | syn75201334 | 2 |
|
|
163
|
+
|
|
164
|
+
"Schema generation" is a geniekit label, not a GENIE term. The two 2022
|
|
165
|
+
releases (NSCLC, CRC) and the four 2025 to 2026 releases share most of their
|
|
166
|
+
variables, but the later releases renamed or recoded a measurable subset. Of
|
|
167
|
+
the 1105 distinct (table, variable) pairs declared across the six Variable
|
|
168
|
+
Synopses, 505 are declared in all six cohorts. Examples of what differs:
|
|
169
|
+
|
|
170
|
+
- smoking history is `ca_lung_cigarette` in generation 1 and `ca_cigarette` in generation 2;
|
|
171
|
+
- distant metastasis sites are `dmets_*` / `dx_to_dmets_*` (9 sites) in generation 1 and `dist_mets_*` / `dx_to_dist_mets_*` (17 sites) in generation 2;
|
|
172
|
+
- `naaccr_seer_sum_stage` is free text in generation 1 and a coded numeric field in generation 2;
|
|
173
|
+
- `cpt_seq_date` is character in generation 1 and numeric in generation 2;
|
|
174
|
+
- `ca_grade`, `pfs_cohort`, `ca_heme_type`, `ca_clin_t_stage` and others have different permitted value sets;
|
|
175
|
+
- tumor-marker tables exist only for CRC, BrCa, PANC and Prostate; radiation-therapy tables only for PANC, Prostate and BLADDER;
|
|
176
|
+
- structural variants are `data_fusions.txt` in generation 1 and `data_sv.txt` in generation 2;
|
|
177
|
+
- UHN contributes to NSCLC, PANC, Prostate and BLADDER but not to CRC or BrCa.
|
|
178
|
+
|
|
179
|
+
`geniekit dictionary diff` lists every such difference.
|
|
180
|
+
|
|
181
|
+
## Data access
|
|
182
|
+
|
|
183
|
+
1. Create a Synapse account and request access to the GENIE BPC data: open
|
|
184
|
+
https://www.synapse.org/Synapse:syn27056700 and click "Request Access".
|
|
185
|
+
One access requirement (id 9606645) covers both the BPC cohorts and the
|
|
186
|
+
main GENIE registry.
|
|
187
|
+
2. Complete the Sage Bionetworks Access and Compliance Team (ACT) form linked
|
|
188
|
+
from that dialog, using your Synapse username. Both steps are required.
|
|
189
|
+
3. After approval, create a personal access token (Synapse profile, Settings,
|
|
190
|
+
Personal Access Tokens) with "view" and "download" scope and expose it as
|
|
191
|
+
`SYNAPSE_AUTH_TOKEN` (see `.env.example`). Never commit the token.
|
|
192
|
+
`geniekit access` reports whether the token works and whether the access
|
|
193
|
+
request is approved, pending or missing.
|
|
194
|
+
|
|
195
|
+
The per-cohort Analytic Data Guides and Variable Synopses are public and are
|
|
196
|
+
linked from https://aacrprojectgenie.org/data/.
|
|
197
|
+
|
|
198
|
+
## Relationship to the R package genieBPC
|
|
199
|
+
|
|
200
|
+
The R package [genieBPC](https://github.com/GENIE-BPC/genieBPC) by the MSK
|
|
201
|
+
Biostatistics team downloads BPC releases and builds analytic cohorts one
|
|
202
|
+
cohort at a time. geniekit uses the same logical table names where they exist
|
|
203
|
+
(`releases.GENIEBPC_TABLE_NAMES` maps between the two) so that workflows can
|
|
204
|
+
move between the languages, and adds the cross-cohort harmonisation, line-of-
|
|
205
|
+
therapy and aggregate-only layers.
|
|
206
|
+
|
|
207
|
+
## Development
|
|
208
|
+
|
|
209
|
+
```powershell
|
|
210
|
+
git clone https://github.com/jemsbhai/geniekit.git
|
|
211
|
+
cd geniekit
|
|
212
|
+
python -m venv .venv
|
|
213
|
+
.\.venv\Scripts\Activate.ps1
|
|
214
|
+
pip install -e ".[dev]"
|
|
215
|
+
pytest tests/ -v --cov=geniekit --cov-report=term
|
|
216
|
+
ruff check src tests data-raw
|
|
217
|
+
python scripts/check_vocabulary.py
|
|
218
|
+
```
|
|
219
|
+
|
|
220
|
+
Regenerating the packaged metadata (requires network access, no Synapse login):
|
|
221
|
+
|
|
222
|
+
```powershell
|
|
223
|
+
python data-raw/build_releases.py
|
|
224
|
+
python data-raw/build_dictionary.py
|
|
225
|
+
```
|
|
226
|
+
|
|
227
|
+
## License
|
|
228
|
+
|
|
229
|
+
MIT. See `LICENSE`. See `NOTICE.md` for the provenance of the packaged
|
|
230
|
+
metadata and for how to cite the data.
|
geniekit-0.0.1/README.md
ADDED
|
@@ -0,0 +1,187 @@
|
|
|
1
|
+
# geniekit
|
|
2
|
+
|
|
3
|
+
Python tools for the AACR Project GENIE Biopharma Collaborative (BPC)
|
|
4
|
+
clinicogenomic cohorts.
|
|
5
|
+
|
|
6
|
+
- **Harmonise** the six public BPC cohorts (NSCLC, CRC, BrCa, PANC, Prostate,
|
|
7
|
+
BLADDER) to one schema so that cross-cohort analyses do not have to deal
|
|
8
|
+
with per-cohort variable names, types and code lists.
|
|
9
|
+
- **Lines of therapy**: derive first and subsequent lines from the regimen
|
|
10
|
+
data relative to an index event that the user chooses (diagnosis,
|
|
11
|
+
metastatic diagnosis, first progression, first systemic therapy, or any
|
|
12
|
+
custom event), with explicit, configurable rules.
|
|
13
|
+
- **Aggregate-only analytic layer**: produce cohort-level summaries with
|
|
14
|
+
disclosure control (minimum cell sizes, suppression, no row-level output)
|
|
15
|
+
for users who may not access patient-level records.
|
|
16
|
+
|
|
17
|
+
geniekit ships metadata only. The data themselves come from Synapse under the
|
|
18
|
+
GENIE access requirement (see [Data access](#data-access)).
|
|
19
|
+
|
|
20
|
+
## Status
|
|
21
|
+
|
|
22
|
+
Version 0.0.1 is the first public release and contains the foundation: the
|
|
23
|
+
release registry, the data dictionary, data loading and download, and the
|
|
24
|
+
synthetic cohorts. The harmonisation, line-of-therapy and aggregate layers
|
|
25
|
+
follow in the next minor releases.
|
|
26
|
+
|
|
27
|
+
| Module | Purpose | State |
|
|
28
|
+
|---|---|---|
|
|
29
|
+
| `geniekit.releases` | Registry of the six public BPC releases with the Synapse id of every clinical, genomic, cBioPortal and documentation file | done |
|
|
30
|
+
| `geniekit.dictionary` | The six Variable Synopsis workbooks compiled into one data dictionary, with cross-cohort comparison helpers | done |
|
|
31
|
+
| `geniekit.io` | Download from Synapse (personal access token, access-status check, MD5 verification) and load release folders from disk into one `CohortData` object | done |
|
|
32
|
+
| `geniekit.synthetic` | Dictionary-driven synthetic cohorts: every declared column, cohort-specific naming, coherent simulated histories | done |
|
|
33
|
+
| `geniekit.harmonize` | Crosswalk-driven normalisation of the six cohorts | planned |
|
|
34
|
+
| `geniekit.lot` | Line-of-therapy derivation with user-defined index events | planned |
|
|
35
|
+
| `geniekit.aggregate` | Aggregate-only analyses with disclosure control | planned |
|
|
36
|
+
|
|
37
|
+
## Installation
|
|
38
|
+
|
|
39
|
+
```powershell
|
|
40
|
+
pip install geniekit
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
Optional extras: `geniekit[synapse]` (official Synapse client), `geniekit[survival]`
|
|
44
|
+
(Kaplan-Meier summaries), `geniekit[notebook]` (Jupyter and plotting),
|
|
45
|
+
`geniekit[dev]` (tests and linting).
|
|
46
|
+
|
|
47
|
+
From a clone:
|
|
48
|
+
|
|
49
|
+
```powershell
|
|
50
|
+
pip install -e ".[dev]"
|
|
51
|
+
pytest tests/ -v --cov=geniekit --cov-report=term
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
## Quick start
|
|
55
|
+
|
|
56
|
+
```python
|
|
57
|
+
from geniekit import releases, dictionary
|
|
58
|
+
|
|
59
|
+
# What is available, and where on Synapse
|
|
60
|
+
for r in releases.list_releases():
|
|
61
|
+
print(r.label, r.n_patients, r.release_folder, sorted(r.clinical))
|
|
62
|
+
|
|
63
|
+
# How the cohorts differ before any harmonisation
|
|
64
|
+
dd = dictionary.load()
|
|
65
|
+
print(dd.compare("ca_cigarette", table="cancer").summary())
|
|
66
|
+
print(dd.differences(table="patient").head())
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
Loading data (after approval, see [Data access](#data-access)):
|
|
70
|
+
|
|
71
|
+
```python
|
|
72
|
+
from geniekit import io
|
|
73
|
+
|
|
74
|
+
# download one release with a personal access token in SYNAPSE_AUTH_TOKEN
|
|
75
|
+
folder = io.download_release("CRC", dest="data/genie_bpc") # data/genie_bpc/CRC_v2.0-public
|
|
76
|
+
crc = io.load_release(folder) # CohortData with 12 tables
|
|
77
|
+
crc["regimen"].head()
|
|
78
|
+
|
|
79
|
+
# or point at a folder you downloaded through the Synapse website
|
|
80
|
+
nsclc = io.load_release("downloads/NSCLC_2.0-public")
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
Working without the data:
|
|
84
|
+
|
|
85
|
+
```python
|
|
86
|
+
from geniekit import synthetic
|
|
87
|
+
|
|
88
|
+
fake = synthetic.generate("BLADDER", n_patients=300, seed=0) # same tables and columns as the real release
|
|
89
|
+
fake["cancer_index"][["record_id", "stage_dx", "dist_mets_liver", "tt_os_dx_days"]].head()
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
Synthetic cohorts carry the exact column set of the corresponding release
|
|
93
|
+
(checked against the dictionary in the test suite) and mutually consistent
|
|
94
|
+
identifiers, intervals and outcomes, but their values are invented. They
|
|
95
|
+
exist so that code can be written and tested before access is granted; never
|
|
96
|
+
report numbers computed from them.
|
|
97
|
+
|
|
98
|
+
Command line:
|
|
99
|
+
|
|
100
|
+
```powershell
|
|
101
|
+
geniekit releases --files
|
|
102
|
+
geniekit dictionary lookup naaccr_seer_sum_stage --table cancer --values
|
|
103
|
+
geniekit dictionary diff --table cancer --kind dtype --kind value_set
|
|
104
|
+
geniekit dictionary presence --table patient --only-partial
|
|
105
|
+
geniekit access # token and GENIE approval status
|
|
106
|
+
geniekit download CRC --dest data/genie_bpc # after approval
|
|
107
|
+
geniekit inspect data/genie_bpc/CRC_v2.0-public # table summary and column check
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
## The six public releases
|
|
111
|
+
|
|
112
|
+
| Cohort | Version | Patients | Institutions | Synapse | Schema generation |
|
|
113
|
+
|---|---|---|---|---|---|
|
|
114
|
+
| NSCLC | v2.0-public | 1846 | DFCI, MSK, UHN, VICC | syn27056697 | 1 |
|
|
115
|
+
| CRC | v2.0-public | 1485 | DFCI, MSK, VICC | syn30991637 | 1 |
|
|
116
|
+
| BrCa | v1.0-public | 1130 | DFCI, MSK, VICC | syn71825332 | 2 |
|
|
117
|
+
| PANC | v1.0-public | 1109 | DFCI, MSK, UHN, VICC | syn72666404 | 2 |
|
|
118
|
+
| Prostate | v1.0-public | 1116 | DFCI, MSK, UHN, VICC | syn73719283 | 2 |
|
|
119
|
+
| BLADDER | v1.0-public | 716 | DFCI, MSK, UHN, VICC | syn75201334 | 2 |
|
|
120
|
+
|
|
121
|
+
"Schema generation" is a geniekit label, not a GENIE term. The two 2022
|
|
122
|
+
releases (NSCLC, CRC) and the four 2025 to 2026 releases share most of their
|
|
123
|
+
variables, but the later releases renamed or recoded a measurable subset. Of
|
|
124
|
+
the 1105 distinct (table, variable) pairs declared across the six Variable
|
|
125
|
+
Synopses, 505 are declared in all six cohorts. Examples of what differs:
|
|
126
|
+
|
|
127
|
+
- smoking history is `ca_lung_cigarette` in generation 1 and `ca_cigarette` in generation 2;
|
|
128
|
+
- distant metastasis sites are `dmets_*` / `dx_to_dmets_*` (9 sites) in generation 1 and `dist_mets_*` / `dx_to_dist_mets_*` (17 sites) in generation 2;
|
|
129
|
+
- `naaccr_seer_sum_stage` is free text in generation 1 and a coded numeric field in generation 2;
|
|
130
|
+
- `cpt_seq_date` is character in generation 1 and numeric in generation 2;
|
|
131
|
+
- `ca_grade`, `pfs_cohort`, `ca_heme_type`, `ca_clin_t_stage` and others have different permitted value sets;
|
|
132
|
+
- tumor-marker tables exist only for CRC, BrCa, PANC and Prostate; radiation-therapy tables only for PANC, Prostate and BLADDER;
|
|
133
|
+
- structural variants are `data_fusions.txt` in generation 1 and `data_sv.txt` in generation 2;
|
|
134
|
+
- UHN contributes to NSCLC, PANC, Prostate and BLADDER but not to CRC or BrCa.
|
|
135
|
+
|
|
136
|
+
`geniekit dictionary diff` lists every such difference.
|
|
137
|
+
|
|
138
|
+
## Data access
|
|
139
|
+
|
|
140
|
+
1. Create a Synapse account and request access to the GENIE BPC data: open
|
|
141
|
+
https://www.synapse.org/Synapse:syn27056700 and click "Request Access".
|
|
142
|
+
One access requirement (id 9606645) covers both the BPC cohorts and the
|
|
143
|
+
main GENIE registry.
|
|
144
|
+
2. Complete the Sage Bionetworks Access and Compliance Team (ACT) form linked
|
|
145
|
+
from that dialog, using your Synapse username. Both steps are required.
|
|
146
|
+
3. After approval, create a personal access token (Synapse profile, Settings,
|
|
147
|
+
Personal Access Tokens) with "view" and "download" scope and expose it as
|
|
148
|
+
`SYNAPSE_AUTH_TOKEN` (see `.env.example`). Never commit the token.
|
|
149
|
+
`geniekit access` reports whether the token works and whether the access
|
|
150
|
+
request is approved, pending or missing.
|
|
151
|
+
|
|
152
|
+
The per-cohort Analytic Data Guides and Variable Synopses are public and are
|
|
153
|
+
linked from https://aacrprojectgenie.org/data/.
|
|
154
|
+
|
|
155
|
+
## Relationship to the R package genieBPC
|
|
156
|
+
|
|
157
|
+
The R package [genieBPC](https://github.com/GENIE-BPC/genieBPC) by the MSK
|
|
158
|
+
Biostatistics team downloads BPC releases and builds analytic cohorts one
|
|
159
|
+
cohort at a time. geniekit uses the same logical table names where they exist
|
|
160
|
+
(`releases.GENIEBPC_TABLE_NAMES` maps between the two) so that workflows can
|
|
161
|
+
move between the languages, and adds the cross-cohort harmonisation, line-of-
|
|
162
|
+
therapy and aggregate-only layers.
|
|
163
|
+
|
|
164
|
+
## Development
|
|
165
|
+
|
|
166
|
+
```powershell
|
|
167
|
+
git clone https://github.com/jemsbhai/geniekit.git
|
|
168
|
+
cd geniekit
|
|
169
|
+
python -m venv .venv
|
|
170
|
+
.\.venv\Scripts\Activate.ps1
|
|
171
|
+
pip install -e ".[dev]"
|
|
172
|
+
pytest tests/ -v --cov=geniekit --cov-report=term
|
|
173
|
+
ruff check src tests data-raw
|
|
174
|
+
python scripts/check_vocabulary.py
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
Regenerating the packaged metadata (requires network access, no Synapse login):
|
|
178
|
+
|
|
179
|
+
```powershell
|
|
180
|
+
python data-raw/build_releases.py
|
|
181
|
+
python data-raw/build_dictionary.py
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
## License
|
|
185
|
+
|
|
186
|
+
MIT. See `LICENSE`. See `NOTICE.md` for the provenance of the packaged
|
|
187
|
+
metadata and for how to cite the data.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
# data-raw
|
|
2
|
+
|
|
3
|
+
Scripts that regenerate the packaged metadata in `src/geniekit/resources/`.
|
|
4
|
+
They need network access but no Synapse login; everything they read is public.
|
|
5
|
+
|
|
6
|
+
| Script | Output | Source |
|
|
7
|
+
|---|---|---|
|
|
8
|
+
| `build_releases.py` | `releases.json` | Synapse REST API: entity tree of the BPC "Data Releases" folder (syn27056700); cohort facts from aacrprojectgenie.org |
|
|
9
|
+
| `build_dictionary.py` | `dictionary.csv.gz`, `dictionary.meta.json` | The six Variable Synopsis workbooks linked from aacrprojectgenie.org (downloaded into `cache/`, which is gitignored) |
|
|
10
|
+
|
|
11
|
+
Run from the repository root, in this order:
|
|
12
|
+
|
|
13
|
+
```powershell
|
|
14
|
+
python data-raw/build_releases.py
|
|
15
|
+
python data-raw/build_dictionary.py
|
|
16
|
+
pytest tests/ -q
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
Commit the regenerated resources together with the script change that
|
|
20
|
+
produced them; the diff of `releases.json` is the review artefact when a new
|
|
21
|
+
public release appears.
|