appfl-bio-suite 0.3.0__tar.gz

appfl_bio_suite-0.3.0/CITATION.cff

@@ -0,0 +1,59 @@
+cff-version: 1.2.0
+message: "If you use this software, please cite it as below."
+title: "appfl-bio-suite: cross-silo federated learning experiments in computational biology"
+abstract: >-
+  A downstream extension package for APPFL that packages federated learning
+  experiments in computational biology -- a FLamby Fed-Heart-Disease benchmark, a
+  federated GWAS with polygenic-score evaluation, federated cross-ancestry
+  fine-mapping governed by the GA4GH DUO, DRS, TRS and TES specifications, and a
+  work-in-progress federated DNA-methylation subtype classifier simulated on one
+  machine -- together with the operational tooling, partner-onboarding workflow,
+  federation network map, and reproducible data-simulation pipeline needed to run
+  them across independently administered HPC centers over Globus Compute.
+type: software
+license: MIT
+version: 0.3.0
+
+# TODO: add ORCID for each author, and any co-authors from the partner sites who should
+# be credited.
+authors:
+  - family-names: "Joel"
+    given-names: "Matthew"
+    affiliation: "Madduri Lab, Argonne National Laboratory"
+
+keywords:
+  - federated learning
+  - genome-wide association study
+  - statistical fine-mapping
+  - DNA methylation
+  - GA4GH
+  - polygenic risk score
+  - differential privacy
+  - Globus Compute
+  - APPFL
+  - reproducible research
+
+references:
+  - type: article
+    title: >-
+      Simulation Benchmarking of Differential Privacy and Federated Learning for
+      Secure Genetic Analysis and Polygenic Risk Scoring
+    authors:
+      - family-names: "Joel"
+        given-names: "Matthew"
+    notes: >-
+      The GWAS experiment reproduces this paper's simulation design. See
+      docs/experiments/gwas/ABOUT.md, and read docs/experiments/gwas/DATA.md for the
+      documented limits on reproducing the published figures from this repository alone.
+
+  - type: software
+    title: "APPFL: An open-source package for privacy-preserving federated learning"
+    repository-code: "https://github.com/APPFL/APPFL"
+    license: MIT
+
+  - type: article
+    title: >-
+      FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic
+      Healthcare Settings
+    year: 2022
+    notes: "NeurIPS 2022 Datasets and Benchmarks track. https://arxiv.org/abs/2210.04620"

appfl_bio_suite-0.3.0/LICENSE

@@ -0,0 +1,33 @@
+MIT License
+
+Copyright (c) 2026 appfl-bio-suite contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+---
+
+This project depends on, but does not include, third-party software and data
+with their own licenses:
+
+* APPFL -- MIT License. https://github.com/APPFL/APPFL
+* FLamby -- MIT License. https://github.com/owkin/FLamby
+  FLamby ships no data. Each of its datasets requires accepting the license on
+  the original host. The Fed-Heart-Disease data derives from the UCI Heart
+  Disease dataset (CC BY 4.0); attribute the original authors.
+* PGS Catalog scoring files -- see https://www.pgscatalog.org/about/ for terms.

appfl_bio_suite-0.3.0/MANIFEST.in

@@ -0,0 +1,5 @@
+recursive-include docs/partner *.md
+recursive-include partners/templates *.j2 *.py
+include constraints.txt
+include requirements-multisusie.txt
+include CITATION.cff

appfl_bio_suite-0.3.0/PKG-INFO

@@ -0,0 +1,349 @@
+Metadata-Version: 2.4
+Name: appfl-bio-suite
+Version: 0.3.0
+Summary: Cross-silo federated learning experiments in computational biology, built on APPFL and Globus Compute
+License: MIT License
+        
+        Copyright (c) 2026 appfl-bio-suite contributors
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+        ---
+        
+        This project depends on, but does not include, third-party software and data
+        with their own licenses:
+        
+        * APPFL -- MIT License. https://github.com/APPFL/APPFL
+        * FLamby -- MIT License. https://github.com/owkin/FLamby
+          FLamby ships no data. Each of its datasets requires accepting the license on
+          the original host. The Fed-Heart-Disease data derives from the UCI Heart
+          Disease dataset (CC BY 4.0); attribute the original authors.
+        * PGS Catalog scoring files -- see https://www.pgscatalog.org/about/ for terms.
+        
+Project-URL: Homepage, https://github.com/mfjoel01/appfl-bio-suite
+Project-URL: Repository, https://github.com/mfjoel01/appfl-bio-suite
+Project-URL: Documentation, https://github.com/mfjoel01/appfl-bio-suite/tree/main/docs
+Keywords: federated-learning,globus-compute,appfl,gwas,polygenic-score,fine-mapping,susiex,flamby,dna-methylation
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
+Requires-Python: <3.13,>=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: appfl==1.10.0
+Requires-Dist: globus-compute-sdk==4.9.0
+Requires-Dist: globus-compute-endpoint==4.9.0
+Requires-Dist: globus-sdk==4.4.1
+Requires-Dist: globus-identity-mapping==0.5.0
+Requires-Dist: click==8.1.8
+Requires-Dist: PyYAML>=6.0
+Requires-Dist: pydantic<3,>=2.7
+Requires-Dist: Jinja2<4,>=3.1
+Requires-Dist: numpy==1.26.4
+Requires-Dist: torch==2.8.0
+Provides-Extra: flamby
+Requires-Dist: pandas==2.2.2; extra == "flamby"
+Requires-Dist: scikit-learn==1.7.2; extra == "flamby"
+Requires-Dist: tqdm==4.67.1; extra == "flamby"
+Requires-Dist: wget; extra == "flamby"
+Provides-Extra: gwas
+Requires-Dist: pandas==2.2.2; extra == "gwas"
+Requires-Dist: scipy==1.16.2; extra == "gwas"
+Requires-Dist: scikit-learn==1.7.2; extra == "gwas"
+Requires-Dist: pandas-plink==2.3.2; extra == "gwas"
+Provides-Extra: gwas-sim
+Requires-Dist: appfl-bio-suite[gwas]; extra == "gwas-sim"
+Requires-Dist: xarray==2025.9.1; extra == "gwas-sim"
+Requires-Dist: dask==2024.9.0; extra == "gwas-sim"
+Requires-Dist: matplotlib==3.10.6; extra == "gwas-sim"
+Provides-Extra: finemapping
+Requires-Dist: pandas==2.2.2; extra == "finemapping"
+Requires-Dist: scipy==1.16.2; extra == "finemapping"
+Provides-Extra: finemapping-sim
+Requires-Dist: appfl-bio-suite[finemapping]; extra == "finemapping-sim"
+Requires-Dist: joblib==1.4.2; extra == "finemapping-sim"
+Requires-Dist: matplotlib==3.10.6; extra == "finemapping-sim"
+Requires-Dist: Jinja2<4,>=3.1; extra == "finemapping-sim"
+Provides-Extra: multisusie
+Requires-Dist: appfl-bio-suite[finemapping-sim]; extra == "multisusie"
+Provides-Extra: methylation
+Requires-Dist: scikit-learn==1.7.2; extra == "methylation"
+Requires-Dist: matplotlib==3.10.6; extra == "methylation"
+Provides-Extra: watch
+Requires-Dist: hivewatch==0.2.1; extra == "watch"
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: ruff>=0.6; extra == "dev"
+Provides-Extra: all
+Requires-Dist: appfl-bio-suite[flamby]; extra == "all"
+Requires-Dist: appfl-bio-suite[gwas]; extra == "all"
+Requires-Dist: appfl-bio-suite[gwas-sim]; extra == "all"
+Requires-Dist: appfl-bio-suite[finemapping]; extra == "all"
+Requires-Dist: appfl-bio-suite[finemapping-sim]; extra == "all"
+Requires-Dist: appfl-bio-suite[watch]; extra == "all"
+Requires-Dist: appfl-bio-suite[methylation]; extra == "all"
+Dynamic: license-file
+
+# appfl-bio-suite
+
+Cross-silo **federated learning experiments in computational biology**, built on
+[APPFL](https://github.com/APPFL/APPFL),
+[hivewatch](https://github.com/APPFL/hivewatch) and
+[Globus Compute](https://www.globus.org/compute).
+
+Institutions train on their own data, on their own hardware, and return only results.
+The standard multi-user deployment uses individual identities, with no shared credential,
+VPN, or inbound network access.
+
+> **Experimental: single-user service-account onboarding.** An opt-in prototype uses one service
+> identity per site, with credentials held by that site and the coordinator. It is not
+> yet the default or a production-ready replacement for multi-user endpoints.
+> See the [trial guide](docs/coordinator/service-account-trial.md) and
+> [design evaluation](docs/coordinator/endpoint-sharing-evaluation.md).
+
+**→ [mfjoel01.github.io/appfl-bio-suite](https://mfjoel01.github.io/appfl-bio-suite/)** —
+the project site, with a live preview of the [federation network
+map](https://mfjoel01.github.io/appfl-bio-suite/network.html).
+
+```
+    YOU (coordinator)                          PARTNER SITES
+    ┌──────────────┐
+    │   driver     │ ──── task ────►    their cluster, their data
+    │  aggregator  │ ◄─── result ───    (weights, or summary statistics)
+    └──────────────┘
+```
+
+---
+
+## The experiments
+
+Fine-mapping also has a completed optional
+[MultiSuSiE coordinator pipeline](docs/experiments/fine-mapping/MULTISUSIE.md) and
+[matched-filter comparison against SuSiEx](docs/experiments/fine-mapping/MATCHED_RESULTS.md).
+Use `run fine-mapping --backend susiex` or `--backend multisusie` with either the
+local or remote driver; `--results-dir` keeps their outputs separate.
+The [comparison report and figures](docs/experiments/fine-mapping/COMPARISON_FIGURES.md)
+cover both methods across cohorts and filter policies.
+
+
+Four experiments, one toolchain. The first three dispatch to partner sites; the fourth is
+in development.
+
+| | What it is | Data |
+| --- | --- | --- |
+| **[FLamby Fed-Heart-Disease](docs/experiments/flamby-heart-disease/ABOUT.md)** | Multi-round FedAvg on a public tabular benchmark with natural hospital splits | Partners download it; ~40 KB, CC BY 4.0 |
+| **[Federated GWAS](docs/experiments/gwas/ABOUT.md)** | Single-round summary-statistic meta-analysis with polygenic-score evaluation | Simulated and distributed by the coordinator |
+| **[Federated fine-mapping](docs/experiments/fine-mapping/ABOUT.md)** | Single-round cross-ancestry credible-set construction from per-site second moments — provably equal to a pooled fit, not an approximation of one. Governed by [GA4GH DUO, DRS, TRS and TES](docs/coordinator/ga4gh.md) | Simulated and distributed by the coordinator |
+| **[Federated methylation](docs/experiments/methylation/ABOUT.md)** · *work in progress* | Multi-round FedAvg for methylation-based subtype classification on sparse, nanopore-like input, against pooled and local-only training. Currently a single-machine simulation | Synthetic, or open TCGA-LAML methylation data from the NCI GDC |
+
+**Federated methylation is a work in progress.** All of its sites are simulated in one
+process today, so there is no partner bundle or endpoint for it yet. Its results so far,
+including one real-data run, are recorded as measured in its
+[record of runs](docs/experiments/methylation/ABOUT.md#record-of-runs).
+
+## Quickstart
+
+Python 3.12 is required. Install the released package from PyPI:
+
+```bash
+python -m pip install "appfl-bio-suite[all]==0.3.0"
+```
+
+For coordinator scripts, example federation files, and the pinned constraints, use
+the matching source checkout:
+
+```bash
+git clone --branch v0.3.0 https://github.com/mfjoel01/appfl-bio-suite.git
+cd appfl-bio-suite
+python -m venv .venv && source .venv/bin/activate
+pip install -e ".[all]" -c constraints.txt
+```
+
+Prove the install works — a complete federation on one machine, no partners, no network,
+and nothing to fill in first:
+
+```bash
+appfl-bio-suite simulate gwas --scenario ci-tiny --out /tmp/gwas-ci
+appfl-bio-suite run gwas --config loopback --driver serial --data-root /tmp/gwas-ci
+```
+
+Results land in `local/output/gwas-loopback/`. This is the same pair of commands CI runs.
+
+Then stand up a real federation:
+
+```bash
+cp federation.yaml.example local/federation.yaml
+$EDITOR local/federation.yaml
+appfl-bio-suite preflight
+```
+
+**→ [docs/coordinator/new-federation.md](docs/coordinator/new-federation.md)** takes you
+from here to a running experiment across institutions you recruit yourself.
+
+## The CLI
+
+```
+appfl-bio-suite preflight [--experiment X]          environment, pins, configs, endpoints
+appfl-bio-suite endpoint status [SITE]              is it online
+appfl-bio-suite endpoint smoke [SITE]               round-trip: does it actually work
+appfl-bio-suite identity validate <mapping.json>    catch mapping bugs offline, in a second
+appfl-bio-suite simulate EXPERIMENT --scenario NAME  generate and split synthetic data
+appfl-bio-suite run EXPERIMENT                      run it
+appfl-bio-suite partner-bundle EXPERIMENT --site X  a partner's complete setup package
+appfl-bio-suite watch build | serve | export        the federation, on a map
+appfl-bio-suite ga4gh duo | drs | trs | tes         data use, data objects, tools, tasks
+```
+
+---
+
+## How it works
+
+**One config file.** `federation.yaml` declares who you are, who your partners are, and
+what each is doing. Every generator, config, document template and check reads from it.
+Nothing in `src/` contains a coordinator identity, an endpoint UUID, or a site name, and
+CI verifies that.
+
+**Generated partner bundles, not templates.** `partner-bundle` emits a directory with a
+partner's configuration fully resolved — their site ID, their data assignment, *your*
+identity, their scheduler's provider block — plus their two setup documents with no
+placeholders left to interpret.
+
+**A closed set for partner-facing content.** Bundles are built from `docs/partner/` and
+nowhere else, enforced in code, so nothing outside that tree can reach a partner.
+
+**Exact version pinning.** Every site runs byte-identical versions of the Globus Compute
+stack. Skew does not fail at install time; it fails as a deserialization error several
+rounds into a run on somebody else's cluster. `constraints.txt` pins it and
+`preflight --check pins` verifies it.
+
+**Failures that name the fix.** The
+[troubleshooting registry](docs/coordinator/troubleshooting.md) is indexed by symptom and
+gives the mechanism alongside the fix.
+
+## GA4GH standards
+
+The fine-mapping experiment is assembled out of four GA4GH specifications, each closing a
+hole the design otherwise had.
+
+| | | answers |
+| --- | --- | --- |
+| **DUO** | Data Use Ontology | May this site's data be used for this study? |
+| **DRS** | Data Repository Service | Exactly which bytes did a site compute over? |
+| **TRS** | Tool Registry Service | Exactly which tool version computed them? |
+| **TES** | Task Execution Service | Run that tool, there, on those bytes. |
+
+Together: *a named tool version, pinned by checksum, runs over content-addressed data
+whose use terms permit the study that dispatched it.* Every clause of that sentence used
+to be an assumption.
+
+**A site enforces its own consent code.** Each bundle carries a `DATA_USE.json`, and the
+site's worker matches it against the study the coordinator declared — refusing before it
+opens a genotype file, on its own hardware, in the account it controls. The coordinator
+runs the same check offline before dispatch, but that one only saves a queue wait.
+
+**Bundles are content-addressed.** A sha-256 per file and a Merkle id over the set, so
+"did this site run on the bundle I cut for it" is a string comparison. Without it, a site
+running last month's bundle produces well-formed aggregates over the wrong individuals,
+pools without complaint, and is wrong in no visible way.
+
+**Results name the code that made them.** The tool pin covers the source of the two
+modules APPFL ships to workers, is verified at launch, and is written into every run's
+`ga4gh_provenance.json` alongside what each site attests to having actually done.
+
+```bash
+appfl-bio-suite ga4gh duo check --experiment fine-mapping   # will every site permit this?
+appfl-bio-suite ga4gh drs register --data-root local/data/… # address the bundles
+appfl-bio-suite ga4gh trs publish --out local/trs           # pin the tool
+appfl-bio-suite ga4gh tes task --site X                     # what would run, as a document
+```
+
+All of it is opt-in per experiment: a `federation.yaml` with no `ga4gh` blocks produces
+byte-identical configs to the ones it produced before any of this existed.
+
+**→ [docs/coordinator/ga4gh.md](docs/coordinator/ga4gh.md)**
+
+## The network map
+
+Everyone in the federation — coordinator and partners alike — can open the same page:
+every institution on a world map, which experiments each runs, what kind of data each
+holds, how many samples they declared, and whether their endpoint is answering.
+
+```bash
+pip install 'appfl-bio-suite[watch]' -c constraints.txt
+
+appfl-bio-suite watch build              # draw it from federation.yaml
+appfl-bio-suite watch serve              # look at it locally
+appfl-bio-suite watch export --out site/ # three static files anyone can host
+appfl-bio-suite run gwas --watch         # a live run, on the same map
+```
+
+Built on [hivewatch](https://github.com/APPFL/hivewatch), APPFL's monitoring toolkit,
+which supplies the event schema, the map format and the viewer.
+
+**The map exists before any run does.** It is drawn from `federation.yaml` alone, so the
+day the config names a partner, that partner is on the map — which is what makes it a
+picture of the network rather than a dashboard for one job.
+
+Site coordinates are declared in `federation.yaml`, never resolved from an IP address: a
+partner's compute node has no outbound web access, and you already know where your
+partners are. A site with no coordinates is reported as unplaced rather than drawn at a
+guessed position. Service accounts, cluster paths, your Globus identity and (by default)
+full endpoint UUIDs never reach the published page.
+
+**→ [docs/coordinator/network-map.md](docs/coordinator/network-map.md)**
+
+## Reproducibility
+
+- **Every simulation run writes a manifest** recording the scenario, every seed, the suite
+  commit, input and output checksums, and package versions.
+  `simulate gwas --verify <manifest>` confirms a rerun mechanically.
+- **The GWAS simulation is deterministic.** For a fixed seed, all 35 outputs are
+  byte-identical, verified against recorded checksums.
+- **Known limits are documented.**
+  [docs/experiments/gwas/DATA.md](docs/experiments/gwas/DATA.md) states which input cohort
+  is not regenerable, and what the shipped substitute does and does not reproduce.
+
+## Documentation
+
+Organized by audience.
+
+| | |
+| --- | --- |
+| [docs/README.md](docs/README.md) | Index: which document, for whom, in what order |
+| [docs/coordinator/](docs/coordinator/) | Running a federation |
+| [docs/partner/](docs/partner/) | What a partner receives — a closed set |
+| [docs/experiments/](docs/experiments/) | Per-experiment design, data, and runbooks |
+
+## Requirements
+
+Python 3.12. A Globus account. Partners willing to run a privileged process on their own
+clusters.
+
+Not required: a shared filesystem, a VPN, matching operating systems or schedulers, or
+inbound network access.
+
+## Licence
+
+MIT. See [LICENSE](LICENSE) — and note the third-party terms recorded there for APPFL,
+FLamby, and PGS Catalog data.
+
+For automated partner setup, see [bundle-based onboarding](docs/coordinator/onboarding.md).