geniekit 0.0.1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (55) hide show
  1. geniekit-0.0.1/CHANGELOG.md +32 -0
  2. geniekit-0.0.1/LICENSE +21 -0
  3. geniekit-0.0.1/MANIFEST.in +7 -0
  4. geniekit-0.0.1/NOTICE.md +50 -0
  5. geniekit-0.0.1/PKG-INFO +230 -0
  6. geniekit-0.0.1/README.md +187 -0
  7. geniekit-0.0.1/data-raw/README.md +21 -0
  8. geniekit-0.0.1/data-raw/build_dictionary.py +168 -0
  9. geniekit-0.0.1/data-raw/build_releases.py +326 -0
  10. geniekit-0.0.1/examples/synthetic_cohorts.py +29 -0
  11. geniekit-0.0.1/pyproject.toml +86 -0
  12. geniekit-0.0.1/scripts/check_vocabulary.py +107 -0
  13. geniekit-0.0.1/setup.cfg +4 -0
  14. geniekit-0.0.1/src/geniekit/__init__.py +30 -0
  15. geniekit-0.0.1/src/geniekit/_version.py +8 -0
  16. geniekit-0.0.1/src/geniekit/cli.py +200 -0
  17. geniekit-0.0.1/src/geniekit/dictionary.py +317 -0
  18. geniekit-0.0.1/src/geniekit/io/__init__.py +36 -0
  19. geniekit-0.0.1/src/geniekit/io/cohort.py +161 -0
  20. geniekit-0.0.1/src/geniekit/io/local.py +275 -0
  21. geniekit-0.0.1/src/geniekit/io/synapse.py +334 -0
  22. geniekit-0.0.1/src/geniekit/releases.py +318 -0
  23. geniekit-0.0.1/src/geniekit/resources/__init__.py +10 -0
  24. geniekit-0.0.1/src/geniekit/resources/dictionary.csv.gz +0 -0
  25. geniekit-0.0.1/src/geniekit/resources/dictionary.meta.json +13 -0
  26. geniekit-0.0.1/src/geniekit/resources/releases.json +1880 -0
  27. geniekit-0.0.1/src/geniekit/synthetic/__init__.py +15 -0
  28. geniekit-0.0.1/src/geniekit/synthetic/generate.py +1272 -0
  29. geniekit-0.0.1/src/geniekit/synthetic/spec.py +331 -0
  30. geniekit-0.0.1/src/geniekit.egg-info/PKG-INFO +230 -0
  31. geniekit-0.0.1/src/geniekit.egg-info/SOURCES.txt +53 -0
  32. geniekit-0.0.1/src/geniekit.egg-info/dependency_links.txt +1 -0
  33. geniekit-0.0.1/src/geniekit.egg-info/entry_points.txt +2 -0
  34. geniekit-0.0.1/src/geniekit.egg-info/requires.txt +22 -0
  35. geniekit-0.0.1/src/geniekit.egg-info/top_level.txt +1 -0
  36. geniekit-0.0.1/tests/__init__.py +0 -0
  37. geniekit-0.0.1/tests/conftest.py +21 -0
  38. geniekit-0.0.1/tests/fixtures/README.md +24 -0
  39. geniekit-0.0.1/tests/fixtures/nsclc_sim/cBioPortal_files/data_CNA.txt.gz +0 -0
  40. geniekit-0.0.1/tests/fixtures/nsclc_sim/cBioPortal_files/data_fusions.txt.gz +0 -0
  41. geniekit-0.0.1/tests/fixtures/nsclc_sim/cBioPortal_files/data_mutations_extended.txt.gz +0 -0
  42. geniekit-0.0.1/tests/fixtures/nsclc_sim/clinical_data/cancer_level_dataset_index.csv.gz +0 -0
  43. geniekit-0.0.1/tests/fixtures/nsclc_sim/clinical_data/cancer_level_dataset_non_index.csv.gz +0 -0
  44. geniekit-0.0.1/tests/fixtures/nsclc_sim/clinical_data/cancer_panel_test_level_dataset.csv.gz +0 -0
  45. geniekit-0.0.1/tests/fixtures/nsclc_sim/clinical_data/imaging_level_dataset.csv.gz +0 -0
  46. geniekit-0.0.1/tests/fixtures/nsclc_sim/clinical_data/med_onc_note_level_dataset.csv.gz +0 -0
  47. geniekit-0.0.1/tests/fixtures/nsclc_sim/clinical_data/pathology_report_level_dataset.csv.gz +0 -0
  48. geniekit-0.0.1/tests/fixtures/nsclc_sim/clinical_data/patient_level_dataset.csv.gz +0 -0
  49. geniekit-0.0.1/tests/fixtures/nsclc_sim/clinical_data/regimen_cancer_level_dataset.csv.gz +0 -0
  50. geniekit-0.0.1/tests/test_cli.py +94 -0
  51. geniekit-0.0.1/tests/test_dictionary.py +133 -0
  52. geniekit-0.0.1/tests/test_io_local.py +171 -0
  53. geniekit-0.0.1/tests/test_io_synapse.py +205 -0
  54. geniekit-0.0.1/tests/test_releases.py +155 -0
  55. geniekit-0.0.1/tests/test_synthetic.py +209 -0
@@ -0,0 +1,32 @@
1
+ # Changelog
2
+
3
+ All notable changes to geniekit are recorded here. The format follows
4
+ Keep a Changelog, and the project follows semantic versioning.
5
+
6
+ ## [Unreleased]
7
+
8
+ ## [0.0.1] - 2026-10-17
9
+
10
+ First public release: the foundation modules.
11
+
12
+ ### Added
13
+ - `geniekit.releases`: registry of the six public GENIE BPC releases (NSCLC
14
+ v2.0, CRC v2.0, BrCa v1.0, PANC v1.0, Prostate v1.0, BLADDER v1.0) with the
15
+ Synapse id of every clinical, genomic, cBioPortal and documentation file,
16
+ generated from the public Synapse entity tree.
17
+ - `geniekit.dictionary`: the six Variable Synopsis workbooks compiled into one
18
+ data dictionary, with presence matrices and cross-cohort comparison of
19
+ types, labels and permitted values.
20
+ - `geniekit` command line interface with `releases` and `dictionary`
21
+ subcommands.
22
+ - Build scripts in `data-raw/` that regenerate the packaged metadata.
23
+ - `geniekit.io`: `CohortData` container, `load_release` for release folders on
24
+ disk (Synapse layout, geniekit download layout, flat folders, gzip), and a
25
+ Synapse client (`SynapseSession`, `download_release`) with token
26
+ resolution, access-status reporting and MD5-verified downloads.
27
+ - `geniekit.synthetic`: dictionary-driven synthetic cohorts for all six
28
+ public releases, with coherent patient histories, regimens, imaging and
29
+ oncologist assessments, sequencing samples and genomic tables.
30
+ - CLI commands `access`, `download` and `inspect`.
31
+ - Test fixture `tests/fixtures/nsclc_sim` derived from the simulated data
32
+ shipped with the R package genieBPC (MIT).
geniekit-0.0.1/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Muntaser Syed
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,7 @@
1
+ include README.md CHANGELOG.md NOTICE.md LICENSE MANIFEST.in
2
+ recursive-include src/geniekit/resources *.json *.csv.gz
3
+ recursive-include tests *.py *.md *.gz
4
+ recursive-include data-raw *.py *.md
5
+ recursive-include scripts *.py
6
+ recursive-include examples *.py
7
+ prune data-raw/cache
@@ -0,0 +1,50 @@
1
+ # Notice
2
+
3
+ geniekit is an independent, community-developed tool. It is not produced,
4
+ endorsed or maintained by the American Association for Cancer Research (AACR),
5
+ AACR Project GENIE, Sage Bionetworks or any GENIE-participating institution.
6
+
7
+ ## Data
8
+
9
+ geniekit ships no patient-level data. All GENIE and GENIE BPC data are
10
+ obtained by the user from Synapse (https://www.synapse.org) under the GENIE
11
+ access requirement and the terms of use that accompany each release. Users are
12
+ responsible for complying with those terms, including restrictions on
13
+ redistribution and on attempts to re-identify individuals.
14
+
15
+ ## Packaged metadata and its sources
16
+
17
+ The files in `src/geniekit/resources/` are derived from public materials:
18
+
19
+ - `releases.json` is generated from the public entity tree of the GENIE BPC
20
+ "Data Releases" folder on Synapse (syn27056700) through the Synapse REST API,
21
+ plus cohort facts published on the AACR Project GENIE website
22
+ (https://aacrprojectgenie.org/data/).
23
+ - `dictionary.csv.gz` is compiled from the Variable Synopsis workbooks that
24
+ AACR publishes with each public BPC release. The workbooks are linked from
25
+ the cohort pages on https://aacrprojectgenie.org/data/ and are the
26
+ authoritative description of each release; geniekit reproduces variable
27
+ names, field labels, data types and permitted values so that software can
28
+ reason about them. Consult the Analytic Data Guide of each release for the
29
+ full definitions.
30
+
31
+ ## Test fixture
32
+
33
+ `tests/fixtures/nsclc_sim/` is derived from the `nsclc_test_data` object of
34
+ the R package genieBPC (https://github.com/GENIE-BPC/genieBPC, MIT licence,
35
+ copyright 2022 genieBPC authors), which its authors describe as simulated fake
36
+ data for examples and tests. It contains no real patient records. See
37
+ `tests/fixtures/README.md`.
38
+
39
+ AACR Project GENIE and GENIE BPC are trademarks of their respective owners.
40
+
41
+ ## Citing the data
42
+
43
+ Users of GENIE data are asked by AACR to cite:
44
+
45
+ AACR Project GENIE Consortium. AACR Project GENIE: Powering Precision Medicine
46
+ through an International Consortium. Cancer Discovery 2017;7(8):818-831.
47
+ https://doi.org/10.1158/2159-8290.CD-17-0151
48
+
49
+ and, for BPC cohorts, the cohort-specific publication listed on the cohort's
50
+ page at https://aacrprojectgenie.org/data/.
@@ -0,0 +1,230 @@
1
+ Metadata-Version: 2.4
2
+ Name: geniekit
3
+ Version: 0.0.1
4
+ Summary: Harmonise the AACR Project GENIE BPC cohorts, compute lines of therapy, and run aggregate-only analyses.
5
+ Author-email: Muntaser Syed <jemsbhai@gmail.com>
6
+ License-Expression: MIT
7
+ Project-URL: Homepage, https://github.com/jemsbhai/geniekit
8
+ Project-URL: Issues, https://github.com/jemsbhai/geniekit/issues
9
+ Project-URL: AACR Project GENIE, https://aacrprojectgenie.org/
10
+ Keywords: AACR,GENIE,BPC,oncology,clinicogenomic,line of therapy,harmonization,Synapse
11
+ Classifier: Development Status :: 3 - Alpha
12
+ Classifier: Intended Audience :: Science/Research
13
+ Classifier: Programming Language :: Python :: 3
14
+ Classifier: Programming Language :: Python :: 3.10
15
+ Classifier: Programming Language :: Python :: 3.11
16
+ Classifier: Programming Language :: Python :: 3.12
17
+ Classifier: Programming Language :: Python :: 3.13
18
+ Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
19
+ Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
20
+ Requires-Python: >=3.10
21
+ Description-Content-Type: text/markdown
22
+ License-File: LICENSE
23
+ License-File: NOTICE.md
24
+ Requires-Dist: pandas<4,>=2.2
25
+ Requires-Dist: numpy<3,>=1.26
26
+ Requires-Dist: requests<3,>=2.32
27
+ Requires-Dist: pyyaml<7,>=6.0
28
+ Provides-Extra: synapse
29
+ Requires-Dist: synapseclient<5,>=4.4; extra == "synapse"
30
+ Provides-Extra: survival
31
+ Requires-Dist: lifelines<1,>=0.29; extra == "survival"
32
+ Provides-Extra: notebook
33
+ Requires-Dist: jupyter<2,>=1.1; extra == "notebook"
34
+ Requires-Dist: matplotlib<4,>=3.9; extra == "notebook"
35
+ Provides-Extra: dev
36
+ Requires-Dist: pytest<9,>=8.3; extra == "dev"
37
+ Requires-Dist: pytest-cov<8,>=6.0; extra == "dev"
38
+ Requires-Dist: ruff<1,>=0.9; extra == "dev"
39
+ Requires-Dist: openpyxl<4,>=3.1; extra == "dev"
40
+ Requires-Dist: build<2,>=1.2; extra == "dev"
41
+ Requires-Dist: twine<7,>=6.0; extra == "dev"
42
+ Dynamic: license-file
43
+
44
+ # geniekit
45
+
46
+ Python tools for the AACR Project GENIE Biopharma Collaborative (BPC)
47
+ clinicogenomic cohorts.
48
+
49
+ - **Harmonise** the six public BPC cohorts (NSCLC, CRC, BrCa, PANC, Prostate,
50
+ BLADDER) to one schema so that cross-cohort analyses do not have to deal
51
+ with per-cohort variable names, types and code lists.
52
+ - **Lines of therapy**: derive first and subsequent lines from the regimen
53
+ data relative to an index event that the user chooses (diagnosis,
54
+ metastatic diagnosis, first progression, first systemic therapy, or any
55
+ custom event), with explicit, configurable rules.
56
+ - **Aggregate-only analytic layer**: produce cohort-level summaries with
57
+ disclosure control (minimum cell sizes, suppression, no row-level output)
58
+ for users who may not access patient-level records.
59
+
60
+ geniekit ships metadata only. The data themselves come from Synapse under the
61
+ GENIE access requirement (see [Data access](#data-access)).
62
+
63
+ ## Status
64
+
65
+ Version 0.0.1 is the first public release and contains the foundation: the
66
+ release registry, the data dictionary, data loading and download, and the
67
+ synthetic cohorts. The harmonisation, line-of-therapy and aggregate layers
68
+ follow in the next minor releases.
69
+
70
+ | Module | Purpose | State |
71
+ |---|---|---|
72
+ | `geniekit.releases` | Registry of the six public BPC releases with the Synapse id of every clinical, genomic, cBioPortal and documentation file | done |
73
+ | `geniekit.dictionary` | The six Variable Synopsis workbooks compiled into one data dictionary, with cross-cohort comparison helpers | done |
74
+ | `geniekit.io` | Download from Synapse (personal access token, access-status check, MD5 verification) and load release folders from disk into one `CohortData` object | done |
75
+ | `geniekit.synthetic` | Dictionary-driven synthetic cohorts: every declared column, cohort-specific naming, coherent simulated histories | done |
76
+ | `geniekit.harmonize` | Crosswalk-driven normalisation of the six cohorts | planned |
77
+ | `geniekit.lot` | Line-of-therapy derivation with user-defined index events | planned |
78
+ | `geniekit.aggregate` | Aggregate-only analyses with disclosure control | planned |
79
+
80
+ ## Installation
81
+
82
+ ```powershell
83
+ pip install geniekit
84
+ ```
85
+
86
+ Optional extras: `geniekit[synapse]` (official Synapse client), `geniekit[survival]`
87
+ (Kaplan-Meier summaries), `geniekit[notebook]` (Jupyter and plotting),
88
+ `geniekit[dev]` (tests and linting).
89
+
90
+ From a clone:
91
+
92
+ ```powershell
93
+ pip install -e ".[dev]"
94
+ pytest tests/ -v --cov=geniekit --cov-report=term
95
+ ```
96
+
97
+ ## Quick start
98
+
99
+ ```python
100
+ from geniekit import releases, dictionary
101
+
102
+ # What is available, and where on Synapse
103
+ for r in releases.list_releases():
104
+ print(r.label, r.n_patients, r.release_folder, sorted(r.clinical))
105
+
106
+ # How the cohorts differ before any harmonisation
107
+ dd = dictionary.load()
108
+ print(dd.compare("ca_cigarette", table="cancer").summary())
109
+ print(dd.differences(table="patient").head())
110
+ ```
111
+
112
+ Loading data (after approval, see [Data access](#data-access)):
113
+
114
+ ```python
115
+ from geniekit import io
116
+
117
+ # download one release with a personal access token in SYNAPSE_AUTH_TOKEN
118
+ folder = io.download_release("CRC", dest="data/genie_bpc") # data/genie_bpc/CRC_v2.0-public
119
+ crc = io.load_release(folder) # CohortData with 12 tables
120
+ crc["regimen"].head()
121
+
122
+ # or point at a folder you downloaded through the Synapse website
123
+ nsclc = io.load_release("downloads/NSCLC_2.0-public")
124
+ ```
125
+
126
+ Working without the data:
127
+
128
+ ```python
129
+ from geniekit import synthetic
130
+
131
+ fake = synthetic.generate("BLADDER", n_patients=300, seed=0) # same tables and columns as the real release
132
+ fake["cancer_index"][["record_id", "stage_dx", "dist_mets_liver", "tt_os_dx_days"]].head()
133
+ ```
134
+
135
+ Synthetic cohorts carry the exact column set of the corresponding release
136
+ (checked against the dictionary in the test suite) and mutually consistent
137
+ identifiers, intervals and outcomes, but their values are invented. They
138
+ exist so that code can be written and tested before access is granted; never
139
+ report numbers computed from them.
140
+
141
+ Command line:
142
+
143
+ ```powershell
144
+ geniekit releases --files
145
+ geniekit dictionary lookup naaccr_seer_sum_stage --table cancer --values
146
+ geniekit dictionary diff --table cancer --kind dtype --kind value_set
147
+ geniekit dictionary presence --table patient --only-partial
148
+ geniekit access # token and GENIE approval status
149
+ geniekit download CRC --dest data/genie_bpc # after approval
150
+ geniekit inspect data/genie_bpc/CRC_v2.0-public # table summary and column check
151
+ ```
152
+
153
+ ## The six public releases
154
+
155
+ | Cohort | Version | Patients | Institutions | Synapse | Schema generation |
156
+ |---|---|---|---|---|---|
157
+ | NSCLC | v2.0-public | 1846 | DFCI, MSK, UHN, VICC | syn27056697 | 1 |
158
+ | CRC | v2.0-public | 1485 | DFCI, MSK, VICC | syn30991637 | 1 |
159
+ | BrCa | v1.0-public | 1130 | DFCI, MSK, VICC | syn71825332 | 2 |
160
+ | PANC | v1.0-public | 1109 | DFCI, MSK, UHN, VICC | syn72666404 | 2 |
161
+ | Prostate | v1.0-public | 1116 | DFCI, MSK, UHN, VICC | syn73719283 | 2 |
162
+ | BLADDER | v1.0-public | 716 | DFCI, MSK, UHN, VICC | syn75201334 | 2 |
163
+
164
+ "Schema generation" is a geniekit label, not a GENIE term. The two 2022
165
+ releases (NSCLC, CRC) and the four 2025 to 2026 releases share most of their
166
+ variables, but the later releases renamed or recoded a measurable subset. Of
167
+ the 1105 distinct (table, variable) pairs declared across the six Variable
168
+ Synopses, 505 are declared in all six cohorts. Examples of what differs:
169
+
170
+ - smoking history is `ca_lung_cigarette` in generation 1 and `ca_cigarette` in generation 2;
171
+ - distant metastasis sites are `dmets_*` / `dx_to_dmets_*` (9 sites) in generation 1 and `dist_mets_*` / `dx_to_dist_mets_*` (17 sites) in generation 2;
172
+ - `naaccr_seer_sum_stage` is free text in generation 1 and a coded numeric field in generation 2;
173
+ - `cpt_seq_date` is character in generation 1 and numeric in generation 2;
174
+ - `ca_grade`, `pfs_cohort`, `ca_heme_type`, `ca_clin_t_stage` and others have different permitted value sets;
175
+ - tumor-marker tables exist only for CRC, BrCa, PANC and Prostate; radiation-therapy tables only for PANC, Prostate and BLADDER;
176
+ - structural variants are `data_fusions.txt` in generation 1 and `data_sv.txt` in generation 2;
177
+ - UHN contributes to NSCLC, PANC, Prostate and BLADDER but not to CRC or BrCa.
178
+
179
+ `geniekit dictionary diff` lists every such difference.
180
+
181
+ ## Data access
182
+
183
+ 1. Create a Synapse account and request access to the GENIE BPC data: open
184
+ https://www.synapse.org/Synapse:syn27056700 and click "Request Access".
185
+ One access requirement (id 9606645) covers both the BPC cohorts and the
186
+ main GENIE registry.
187
+ 2. Complete the Sage Bionetworks Access and Compliance Team (ACT) form linked
188
+ from that dialog, using your Synapse username. Both steps are required.
189
+ 3. After approval, create a personal access token (Synapse profile, Settings,
190
+ Personal Access Tokens) with "view" and "download" scope and expose it as
191
+ `SYNAPSE_AUTH_TOKEN` (see `.env.example`). Never commit the token.
192
+ `geniekit access` reports whether the token works and whether the access
193
+ request is approved, pending or missing.
194
+
195
+ The per-cohort Analytic Data Guides and Variable Synopses are public and are
196
+ linked from https://aacrprojectgenie.org/data/.
197
+
198
+ ## Relationship to the R package genieBPC
199
+
200
+ The R package [genieBPC](https://github.com/GENIE-BPC/genieBPC) by the MSK
201
+ Biostatistics team downloads BPC releases and builds analytic cohorts one
202
+ cohort at a time. geniekit uses the same logical table names where they exist
203
+ (`releases.GENIEBPC_TABLE_NAMES` maps between the two) so that workflows can
204
+ move between the languages, and adds the cross-cohort harmonisation, line-of-
205
+ therapy and aggregate-only layers.
206
+
207
+ ## Development
208
+
209
+ ```powershell
210
+ git clone https://github.com/jemsbhai/geniekit.git
211
+ cd geniekit
212
+ python -m venv .venv
213
+ .\.venv\Scripts\Activate.ps1
214
+ pip install -e ".[dev]"
215
+ pytest tests/ -v --cov=geniekit --cov-report=term
216
+ ruff check src tests data-raw
217
+ python scripts/check_vocabulary.py
218
+ ```
219
+
220
+ Regenerating the packaged metadata (requires network access, no Synapse login):
221
+
222
+ ```powershell
223
+ python data-raw/build_releases.py
224
+ python data-raw/build_dictionary.py
225
+ ```
226
+
227
+ ## License
228
+
229
+ MIT. See `LICENSE`. See `NOTICE.md` for the provenance of the packaged
230
+ metadata and for how to cite the data.
@@ -0,0 +1,187 @@
1
+ # geniekit
2
+
3
+ Python tools for the AACR Project GENIE Biopharma Collaborative (BPC)
4
+ clinicogenomic cohorts.
5
+
6
+ - **Harmonise** the six public BPC cohorts (NSCLC, CRC, BrCa, PANC, Prostate,
7
+ BLADDER) to one schema so that cross-cohort analyses do not have to deal
8
+ with per-cohort variable names, types and code lists.
9
+ - **Lines of therapy**: derive first and subsequent lines from the regimen
10
+ data relative to an index event that the user chooses (diagnosis,
11
+ metastatic diagnosis, first progression, first systemic therapy, or any
12
+ custom event), with explicit, configurable rules.
13
+ - **Aggregate-only analytic layer**: produce cohort-level summaries with
14
+ disclosure control (minimum cell sizes, suppression, no row-level output)
15
+ for users who may not access patient-level records.
16
+
17
+ geniekit ships metadata only. The data themselves come from Synapse under the
18
+ GENIE access requirement (see [Data access](#data-access)).
19
+
20
+ ## Status
21
+
22
+ Version 0.0.1 is the first public release and contains the foundation: the
23
+ release registry, the data dictionary, data loading and download, and the
24
+ synthetic cohorts. The harmonisation, line-of-therapy and aggregate layers
25
+ follow in the next minor releases.
26
+
27
+ | Module | Purpose | State |
28
+ |---|---|---|
29
+ | `geniekit.releases` | Registry of the six public BPC releases with the Synapse id of every clinical, genomic, cBioPortal and documentation file | done |
30
+ | `geniekit.dictionary` | The six Variable Synopsis workbooks compiled into one data dictionary, with cross-cohort comparison helpers | done |
31
+ | `geniekit.io` | Download from Synapse (personal access token, access-status check, MD5 verification) and load release folders from disk into one `CohortData` object | done |
32
+ | `geniekit.synthetic` | Dictionary-driven synthetic cohorts: every declared column, cohort-specific naming, coherent simulated histories | done |
33
+ | `geniekit.harmonize` | Crosswalk-driven normalisation of the six cohorts | planned |
34
+ | `geniekit.lot` | Line-of-therapy derivation with user-defined index events | planned |
35
+ | `geniekit.aggregate` | Aggregate-only analyses with disclosure control | planned |
36
+
37
+ ## Installation
38
+
39
+ ```powershell
40
+ pip install geniekit
41
+ ```
42
+
43
+ Optional extras: `geniekit[synapse]` (official Synapse client), `geniekit[survival]`
44
+ (Kaplan-Meier summaries), `geniekit[notebook]` (Jupyter and plotting),
45
+ `geniekit[dev]` (tests and linting).
46
+
47
+ From a clone:
48
+
49
+ ```powershell
50
+ pip install -e ".[dev]"
51
+ pytest tests/ -v --cov=geniekit --cov-report=term
52
+ ```
53
+
54
+ ## Quick start
55
+
56
+ ```python
57
+ from geniekit import releases, dictionary
58
+
59
+ # What is available, and where on Synapse
60
+ for r in releases.list_releases():
61
+ print(r.label, r.n_patients, r.release_folder, sorted(r.clinical))
62
+
63
+ # How the cohorts differ before any harmonisation
64
+ dd = dictionary.load()
65
+ print(dd.compare("ca_cigarette", table="cancer").summary())
66
+ print(dd.differences(table="patient").head())
67
+ ```
68
+
69
+ Loading data (after approval, see [Data access](#data-access)):
70
+
71
+ ```python
72
+ from geniekit import io
73
+
74
+ # download one release with a personal access token in SYNAPSE_AUTH_TOKEN
75
+ folder = io.download_release("CRC", dest="data/genie_bpc") # data/genie_bpc/CRC_v2.0-public
76
+ crc = io.load_release(folder) # CohortData with 12 tables
77
+ crc["regimen"].head()
78
+
79
+ # or point at a folder you downloaded through the Synapse website
80
+ nsclc = io.load_release("downloads/NSCLC_2.0-public")
81
+ ```
82
+
83
+ Working without the data:
84
+
85
+ ```python
86
+ from geniekit import synthetic
87
+
88
+ fake = synthetic.generate("BLADDER", n_patients=300, seed=0) # same tables and columns as the real release
89
+ fake["cancer_index"][["record_id", "stage_dx", "dist_mets_liver", "tt_os_dx_days"]].head()
90
+ ```
91
+
92
+ Synthetic cohorts carry the exact column set of the corresponding release
93
+ (checked against the dictionary in the test suite) and mutually consistent
94
+ identifiers, intervals and outcomes, but their values are invented. They
95
+ exist so that code can be written and tested before access is granted; never
96
+ report numbers computed from them.
97
+
98
+ Command line:
99
+
100
+ ```powershell
101
+ geniekit releases --files
102
+ geniekit dictionary lookup naaccr_seer_sum_stage --table cancer --values
103
+ geniekit dictionary diff --table cancer --kind dtype --kind value_set
104
+ geniekit dictionary presence --table patient --only-partial
105
+ geniekit access # token and GENIE approval status
106
+ geniekit download CRC --dest data/genie_bpc # after approval
107
+ geniekit inspect data/genie_bpc/CRC_v2.0-public # table summary and column check
108
+ ```
109
+
110
+ ## The six public releases
111
+
112
+ | Cohort | Version | Patients | Institutions | Synapse | Schema generation |
113
+ |---|---|---|---|---|---|
114
+ | NSCLC | v2.0-public | 1846 | DFCI, MSK, UHN, VICC | syn27056697 | 1 |
115
+ | CRC | v2.0-public | 1485 | DFCI, MSK, VICC | syn30991637 | 1 |
116
+ | BrCa | v1.0-public | 1130 | DFCI, MSK, VICC | syn71825332 | 2 |
117
+ | PANC | v1.0-public | 1109 | DFCI, MSK, UHN, VICC | syn72666404 | 2 |
118
+ | Prostate | v1.0-public | 1116 | DFCI, MSK, UHN, VICC | syn73719283 | 2 |
119
+ | BLADDER | v1.0-public | 716 | DFCI, MSK, UHN, VICC | syn75201334 | 2 |
120
+
121
+ "Schema generation" is a geniekit label, not a GENIE term. The two 2022
122
+ releases (NSCLC, CRC) and the four 2025 to 2026 releases share most of their
123
+ variables, but the later releases renamed or recoded a measurable subset. Of
124
+ the 1105 distinct (table, variable) pairs declared across the six Variable
125
+ Synopses, 505 are declared in all six cohorts. Examples of what differs:
126
+
127
+ - smoking history is `ca_lung_cigarette` in generation 1 and `ca_cigarette` in generation 2;
128
+ - distant metastasis sites are `dmets_*` / `dx_to_dmets_*` (9 sites) in generation 1 and `dist_mets_*` / `dx_to_dist_mets_*` (17 sites) in generation 2;
129
+ - `naaccr_seer_sum_stage` is free text in generation 1 and a coded numeric field in generation 2;
130
+ - `cpt_seq_date` is character in generation 1 and numeric in generation 2;
131
+ - `ca_grade`, `pfs_cohort`, `ca_heme_type`, `ca_clin_t_stage` and others have different permitted value sets;
132
+ - tumor-marker tables exist only for CRC, BrCa, PANC and Prostate; radiation-therapy tables only for PANC, Prostate and BLADDER;
133
+ - structural variants are `data_fusions.txt` in generation 1 and `data_sv.txt` in generation 2;
134
+ - UHN contributes to NSCLC, PANC, Prostate and BLADDER but not to CRC or BrCa.
135
+
136
+ `geniekit dictionary diff` lists every such difference.
137
+
138
+ ## Data access
139
+
140
+ 1. Create a Synapse account and request access to the GENIE BPC data: open
141
+ https://www.synapse.org/Synapse:syn27056700 and click "Request Access".
142
+ One access requirement (id 9606645) covers both the BPC cohorts and the
143
+ main GENIE registry.
144
+ 2. Complete the Sage Bionetworks Access and Compliance Team (ACT) form linked
145
+ from that dialog, using your Synapse username. Both steps are required.
146
+ 3. After approval, create a personal access token (Synapse profile, Settings,
147
+ Personal Access Tokens) with "view" and "download" scope and expose it as
148
+ `SYNAPSE_AUTH_TOKEN` (see `.env.example`). Never commit the token.
149
+ `geniekit access` reports whether the token works and whether the access
150
+ request is approved, pending or missing.
151
+
152
+ The per-cohort Analytic Data Guides and Variable Synopses are public and are
153
+ linked from https://aacrprojectgenie.org/data/.
154
+
155
+ ## Relationship to the R package genieBPC
156
+
157
+ The R package [genieBPC](https://github.com/GENIE-BPC/genieBPC) by the MSK
158
+ Biostatistics team downloads BPC releases and builds analytic cohorts one
159
+ cohort at a time. geniekit uses the same logical table names where they exist
160
+ (`releases.GENIEBPC_TABLE_NAMES` maps between the two) so that workflows can
161
+ move between the languages, and adds the cross-cohort harmonisation, line-of-
162
+ therapy and aggregate-only layers.
163
+
164
+ ## Development
165
+
166
+ ```powershell
167
+ git clone https://github.com/jemsbhai/geniekit.git
168
+ cd geniekit
169
+ python -m venv .venv
170
+ .\.venv\Scripts\Activate.ps1
171
+ pip install -e ".[dev]"
172
+ pytest tests/ -v --cov=geniekit --cov-report=term
173
+ ruff check src tests data-raw
174
+ python scripts/check_vocabulary.py
175
+ ```
176
+
177
+ Regenerating the packaged metadata (requires network access, no Synapse login):
178
+
179
+ ```powershell
180
+ python data-raw/build_releases.py
181
+ python data-raw/build_dictionary.py
182
+ ```
183
+
184
+ ## License
185
+
186
+ MIT. See `LICENSE`. See `NOTICE.md` for the provenance of the packaged
187
+ metadata and for how to cite the data.
@@ -0,0 +1,21 @@
1
+ # data-raw
2
+
3
+ Scripts that regenerate the packaged metadata in `src/geniekit/resources/`.
4
+ They need network access but no Synapse login; everything they read is public.
5
+
6
+ | Script | Output | Source |
7
+ |---|---|---|
8
+ | `build_releases.py` | `releases.json` | Synapse REST API: entity tree of the BPC "Data Releases" folder (syn27056700); cohort facts from aacrprojectgenie.org |
9
+ | `build_dictionary.py` | `dictionary.csv.gz`, `dictionary.meta.json` | The six Variable Synopsis workbooks linked from aacrprojectgenie.org (downloaded into `cache/`, which is gitignored) |
10
+
11
+ Run from the repository root, in this order:
12
+
13
+ ```powershell
14
+ python data-raw/build_releases.py
15
+ python data-raw/build_dictionary.py
16
+ pytest tests/ -q
17
+ ```
18
+
19
+ Commit the regenerated resources together with the script change that
20
+ produced them; the diff of `releases.json` is the review artefact when a new
21
+ public release appears.