gset 0.1.0__tar.gz

gset-0.1.0/PKG-INFO

@@ -0,0 +1,255 @@
+Metadata-Version: 2.4
+Name: gset
+Version: 0.1.0
+Summary: GSET: Gene Signals from Embedding Trajectories select compact contexts for mechanistic GRN inference
+Author: Zeev Weizmann, Elias Ventre
+Project-URL: Repository, https://github.com/ZeevWeizmann/gset
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Requires-Dist: numpy
+Requires-Dist: scipy
+Requires-Dist: pandas
+Requires-Dist: scikit-learn
+Requires-Dist: torch
+Requires-Dist: torch_geometric
+Requires-Dist: pot
+
+# GSET: Gene Signals from Embedding Trajectories select compact contexts for mechanistic GRN inference
+
+Zeev Weizmann and Elias Ventre
+
+Code for the paper *GSET: Gene Signals from Embedding Trajectories select compact contexts for mechanistic GRN inference*. It contains the method (also an installable Python package, see "Use on your own data"),
+the simulated data and the code that makes the figures and tables of the paper (see the list of what is
+not included below).
+
+## What is in the folders
+
+```
+gset/                 the method (the installable package)
+  data.py               reads the simulations
+  graph.py              co-expression graph of the genes
+  embeddings.py         GCN (co-expression) and DeepSets (expression) encoders of the simulations
+  signals.py            KL signal and OT residual
+  lags.py               lagged similarity matrices and the scores of the genes
+  programs.py           finds the programs, their order and their core genes
+  selection.py          builds the gene set (bridges + cores) and the baselines HVG, TF list, random
+  metrics.py            coverage of relay edges and other scores
+  clustering.py         scores for the recovery of programs
+  cardamom_eval.py      scores a network inferred by CardamomOT
+  ot_similarity.py      gene-gene OT distance (Huizing et al.)
+  pipeline.py           trains the encoders and gives the vectors of all transitions
+  reproducible.py       makes the neural networks give the same numbers in every run
+rmsexplorer/          the co-expression encoder of the Gene Program Explorer, shared by the RMS and reprogramming experiments
+baselines/            wrappers of methods from other authors (OTVelo, DELVE, lagged OT similarity)
+data/simulations/     the 41 simulated networks (HARISSA) used in the paper
+data/rms/             the rhabdomyosarcoma counts with the six pseudotime bins and the UMAP coordinates
+data/schiebinger_days0-3.h5ad   the prepared reprogramming data (days 0-3), unpacked by experiments/schiebinger/unpack_data.py
+simulation_code/      the code that made the simulated networks (as it was used, see its README)
+experiments/<block>/  computes the numbers of one block of the paper, saves them in results/<block>/
+results/<block>/      saved numbers
+figures/<block>/      make_figures.py and the pictures in output/
+tables/<block>/       make_tables.py, prints the LaTeX of the tables
+run_pipeline.py       runs the blocks
+```
+
+## Where is what in the paper
+
+Sections, appendices, tables and figures are numbered as in the submitted PDF of the paper
+(`run_pipeline.py --list` prints the same references). `--all` runs the blocks in this order.
+
+| Block (`run_pipeline.py --block ...`) | Paper |
+|---|---|
+| `nb_signals` | Appendix B: negative binomial benchmark (Figs. 4 and 5) |
+| `endpoint_signals` | Appendix C: endpoint maps (Figs. 6 and 7), Table 1, dropout figure (Fig. 8) |
+| `program_recovery` | Section 3.3 and Appendix C: Table 3 (program recovery for all representations) |
+| `lagged_association` | Section 3.4 and Appendix D: discrimination, direction, sampling resolution (Tables 4 to 6), lag figure (Fig. 9) |
+| `rms` | Section 3.5 and Appendix L: query genes on the rhabdomyosarcoma data (Figs. 11 to 15, Tables 20 and 21) |
+| `schiebinger` | Section 3.5 and Appendix K, the reprogramming time course (Schiebinger et al., days 0-3): Fig. 2, Table 19, the bridge genes between the programs of Lox and Birc5 (the query genes with `query_genes.py`; `bridges.py`, with the overlap of the bridges between training seeds against chance and the lagged-correlation baseline; the terms in which the bridges are enriched, which are not significant after correction, with `bridges_terms.py`; OTVelo as a baseline with `otvelo_baseline.py`) (needs the data, see below) |
+| `retrieval_curves` | Section 3.6 and Appendix G: recall curves (Figs. 3 and 10) |
+| `retrieval_selections` | Appendix F: ablation of the retrieval channels (Tables 9 and 10); the gene sets used by CardamomOT |
+| `experiments/raw_embeddings/run.py` | Appendix F: Tables 11 and 12, GSET without the learned encoders (a script of its own, not a block of `run_pipeline.py`; the table is printed by `tables/raw_embeddings/make_tables.py`) |
+| `experiments/spike_in/run.py`, `run_raw.py` | Appendix H: Table 15, 1000 real genes added to the 150-gene cascades, with and without the learned encoders (scripts of their own, results in `results/spike_in` and `results/spike_in_raw`; need `data/schiebinger`) |
+| `baseline_selections` | Appendix E: the gene sets of DELVE, OTVelo and OT similarity (needs `DELVE_REPO` and `OTVELO_REPO`) |
+| `extra_baselines` | Appendix E: Table 8 (temporal DE, GRNBoost2, GENIE3, WGCNA hubs, Hotspot; needs arboreto and hotspotsc, see `requirements.txt`) |
+| `cardamom` | Section 3.6, Appendices E and G: CardamomOT on the retrieved contexts (Tables 2, 7, 13 and 14) (needs `CARDAMOM_REPO` to infer the networks again) |
+| `scalability` | Appendix J: run times and memory (Tables 16 to 18) |
+
+## What is not in the repository
+
+* The whole reprogramming time course (165,892 cells, Schiebinger et al.; GEO: GSE122662) and the raw single-cell
+  RNA-seq counts of the rhabdomyosarcoma samples (GEO: GSE248183). Instead, the repository has the prepared days 0-3
+  of the reprogramming data (`data/schiebinger_days0-3.h5ad`) and the rhabdomyosarcoma counts with the pseudotime bins
+  (`data/rms/rms_counts.h5ad`). The code that makes these files from the full data is included:
+  `experiments/schiebinger/prepare_data.py` and `experiments/rms/make_pseudotime.py`.
+* **Code that made the simulations** is in `simulation_code/` as it was used and was not run again for
+  this repository. The simulated networks themselves are in `data/simulations/`.
+* **Baselines of other authors.** DELVE and OTVelo are not included (see Notes); the pipeline runs them
+  through the wrappers in `baselines/` when `DELVE_REPO` and `OTVELO_REPO` are set, and otherwise uses
+  the gene sets saved in `results/retrieval_selections/`. The networks inferred by CardamomOT are saved
+  in `results/cardamom_networks/`, and the tables are made from them.
+* The overview figure of the paper is a drawing and is not made by code.
+
+## Installation
+
+Tested with Python 3.13.
+
+```bash
+pip install -r requirements.txt
+```
+
+This is enough to rebuild every figure and table from the saved results
+(`python run_pipeline.py --all --from-results`) and to run GSET itself.
+
+To use the method in your own code, install the package `gset` too:
+
+```bash
+pip install -e .
+```
+
+### Methods of other authors (only to run them again)
+
+The results of these methods are saved in `results/`. To compute them again, install the method
+and set the variable; without the variable, `run_pipeline.py` skips the step and uses the saved
+results. The versions below are the ones checked to give the results of the paper.
+
+**CardamomOT** (block `cardamom`):
+
+```bash
+git clone https://github.com/eliasventre/CardamomOT
+cd CardamomOT && git checkout 36c5ca2
+pip install -e . && cd ..
+export CARDAMOM_REPO=$PWD/CardamomOT
+```
+
+Only on macOS with Apple silicon: if the process aborts when torch is loaded, apply
+`experiments/cardamom/cardamomot_serial_numba.patch` before `pip install -e .` (`git apply /path/to/this/repository/experiments/cardamom/cardamomot_serial_numba.patch`).
+It makes two numba loops serial instead of parallel. The networks of the paper were inferred with it.
+
+**OTVelo** (block `baseline_selections`; GPL-3.0, so it is not copied here):
+
+```bash
+git clone https://github.com/Sandstede-Lab/OT-Velocity
+cd OT-Velocity && git checkout 8b8df25 && cd ..
+export OTVELO_REPO=$PWD/OT-Velocity/Utils
+```
+
+**DELVE** (block `baseline_selections`). DELVE is used from its folder and not installed with pip,
+because its package asks for numpy < 2:
+
+```bash
+git clone https://github.com/jranek/delve
+cd delve && git checkout ff08435 && cd ..
+export DELVE_REPO=$PWD/delve
+pip install --no-deps sketchKH==0.1.4 tqdm_joblib==0.0.5
+```
+
+**GRNBoost2, GENIE3 and Hotspot** (block `extra_baselines`):
+
+```bash
+pip install --no-deps arboreto==0.1.6 hotspotsc==1.1.3
+```
+
+## Run
+
+A quick check that everything works (about ten seconds): it makes Table 1 and Figs. 6 to 8 from the saved results.
+
+```bash
+python run_pipeline.py --block endpoint_signals --from-results
+```
+
+The other blocks:
+
+```bash
+python run_pipeline.py --list                          # the blocks
+python run_pipeline.py --block endpoint_signals        # one block: experiment, figures, tables
+python run_pipeline.py --all --from-results            # only the figures and tables, from the saved results
+python run_pipeline.py --all                           # everything again (many hours)
+```
+
+Every experiment can also be run alone, for example
+`python experiments/program_recovery/run.py twophase60_s1 1`.
+
+The pseudotime bins of the rhabdomyosarcoma data can be made again from the raw single-cell RNA-seq counts
+(10x Genomics format). Set the folders of the two samples in `experiments/rms/make_pseudotime.py` and run
+(about five minutes):
+
+```bash
+pip install palantir leidenalg igraph
+python experiments/rms/make_pseudotime.py
+```
+
+## Use on your own data
+
+Install the package from the folder of this repository:
+
+```bash
+pip install -e .
+```
+
+Then:
+
+```python
+import anndata as ad
+from gset import select_genes
+
+adata = ad.read_h5ad("my_data.h5ad")    # raw counts in X, the time of every cell in adata.obs["time"]
+genes = adata.var_names[select_genes(adata, budget=30, n_programs=3)]
+```
+
+`select_genes` trains the encoders on your data (see Notes for the two co-expression encoders) and returns the
+indices of the selected genes: bridges between consecutive gene programs and the core
+genes of the programs. By default a third of the budget goes to the cores and two thirds to the bridges, and the
+programs are found by clustering, with their number given. Other settings:
+
+```python
+# the numbers of cores and bridges (the budget is their sum), and of the two channels of the bridges
+genes = select_genes(adata, n_cores=10, n_bridges=20, n_kl=14, n_ot=6, n_programs=3)
+
+# programs built around one query gene each, in the order in which the programs become active.
+# The program of a query gene is the gene and its program_size - 1 nearest genes, here 10 genes in all.
+queries = [adata.var_names.get_loc("Lox"), adata.var_names.get_loc("Birc5")]
+genes = select_genes(adata, budget=30, query_genes=queries, program_size=10)
+
+# programs of your own: the program of every gene (0, 1, 2, ...) and the order of the programs
+genes = select_genes(adata, budget=30, labels=labels, order=[0, 1, 2])
+
+# the co-expression encoder of the experiments on the real data, for thousands of genes
+genes = select_genes(adata, budget=30, n_programs=3, coexpr_encoder="real_data")
+
+# the data as a list of count matrices (cells x genes), one per timepoint, instead of an AnnData object
+genes = select_genes([X_t0, X_t1, X_t2], budget=30, n_programs=3)
+```
+
+With several replicates, pass a list of such lists. The genes can then be given to CardamomOT or another network
+inference method. The genes related to a query gene can also be explored interactively in the Gene Program Explorer
+(https://rmsexplorer.org).
+
+## Notes
+
+* **Same numbers on every run.** Every experiment calls `make_reproducible()`: torch uses one
+  thread and deterministic algorithms. Without it, the sums of torch change in the last digits
+  when it uses several threads, and this can change which genes are ranked first.
+* **Encoders.** The co-expression embedding, which gives the KL signal, has two encoders. The one in
+  `gset/embeddings.py` is used on the simulations, which have tens to a few hundred genes, and is the default of
+  `select_genes` (`coexpr_encoder="simulation"`). The one in `rmsexplorer/encoder.py` is used on the real data
+  (rhabdomyosarcoma, reprogramming), with thousands of genes (`coexpr_encoder="real_data"`), see `experiments/rms` and
+  `experiments/schiebinger`.
+* **Methods of other authors** (CardamomOT, OTVelo, DELVE, arboreto, Hotspot) are not part of this
+  repository; see Installation.
+* **Data.** The four data sets of the paper:
+  * *Negative binomial mixtures.* They are not stored: `experiments/nb_signals/nb_data.py` simulates them (40 genes,
+    5,000 cells per timepoint, two scenarios, rewiring and positional deviation, at several dropout rates). The scores are
+    in `results/nb_signals/`.
+  * *HARISSA simulations* (https://github.com/ulysseherbach/harissa). Cascades, switches and the families with
+    TF-to-TF relays, 41 networks in `data/simulations/`, made by `simulation_code/`.
+  * *Reprogramming time course* of Schiebinger et al. (Cell, 2019; GEO: GSE122662, serum condition), days 0-3,
+    34,101 cells and 4,007 genes, in `data/schiebinger_days0-3.h5ad`. Run
+    `python experiments/schiebinger/unpack_data.py` once to unpack it into `data/schiebinger/`, which the block
+    `schiebinger` and `experiments/spike_in/` use. The same files can be made from the original serum data, saved as
+    `data/schiebinger_raw/reprogramming_schiebinger_only_serum.h5ad`, with `experiments/schiebinger/prepare_data.py`.
+    `query_genes.py` also needs Table S2 of the paper (`mmc2.xlsx`), saved as
+    `data/schiebinger_raw/schiebinger2019_mmc2.xlsx`.
+  * *Rhabdomyosarcoma (RMS)* of Savary et al. (Cell Reports Medicine, 2023; GEO: GSE248183), 13,968 cells and 8,442
+    genes, with the six pseudotime bins (the column `time`), the cell types (`cell_type`) and the UMAP coordinates, in
+    `data/rms/`. All the analyses use this file. `experiments/rms/make_pseudotime.py` makes a similar file from the raw
+    single-cell RNA-seq counts (see Run). It needs `palantir`, `leidenalg` and `igraph`.

gset-0.1.0/README.md

@@ -0,0 +1,239 @@
+# GSET: Gene Signals from Embedding Trajectories select compact contexts for mechanistic GRN inference
+
+Zeev Weizmann and Elias Ventre
+
+Code for the paper *GSET: Gene Signals from Embedding Trajectories select compact contexts for mechanistic GRN inference*. It contains the method (also an installable Python package, see "Use on your own data"),
+the simulated data and the code that makes the figures and tables of the paper (see the list of what is
+not included below).
+
+## What is in the folders
+
+```
+gset/                 the method (the installable package)
+  data.py               reads the simulations
+  graph.py              co-expression graph of the genes
+  embeddings.py         GCN (co-expression) and DeepSets (expression) encoders of the simulations
+  signals.py            KL signal and OT residual
+  lags.py               lagged similarity matrices and the scores of the genes
+  programs.py           finds the programs, their order and their core genes
+  selection.py          builds the gene set (bridges + cores) and the baselines HVG, TF list, random
+  metrics.py            coverage of relay edges and other scores
+  clustering.py         scores for the recovery of programs
+  cardamom_eval.py      scores a network inferred by CardamomOT
+  ot_similarity.py      gene-gene OT distance (Huizing et al.)
+  pipeline.py           trains the encoders and gives the vectors of all transitions
+  reproducible.py       makes the neural networks give the same numbers in every run
+rmsexplorer/          the co-expression encoder of the Gene Program Explorer, shared by the RMS and reprogramming experiments
+baselines/            wrappers of methods from other authors (OTVelo, DELVE, lagged OT similarity)
+data/simulations/     the 41 simulated networks (HARISSA) used in the paper
+data/rms/             the rhabdomyosarcoma counts with the six pseudotime bins and the UMAP coordinates
+data/schiebinger_days0-3.h5ad   the prepared reprogramming data (days 0-3), unpacked by experiments/schiebinger/unpack_data.py
+simulation_code/      the code that made the simulated networks (as it was used, see its README)
+experiments/<block>/  computes the numbers of one block of the paper, saves them in results/<block>/
+results/<block>/      saved numbers
+figures/<block>/      make_figures.py and the pictures in output/
+tables/<block>/       make_tables.py, prints the LaTeX of the tables
+run_pipeline.py       runs the blocks
+```
+
+## Where is what in the paper
+
+Sections, appendices, tables and figures are numbered as in the submitted PDF of the paper
+(`run_pipeline.py --list` prints the same references). `--all` runs the blocks in this order.
+
+| Block (`run_pipeline.py --block ...`) | Paper |
+|---|---|
+| `nb_signals` | Appendix B: negative binomial benchmark (Figs. 4 and 5) |
+| `endpoint_signals` | Appendix C: endpoint maps (Figs. 6 and 7), Table 1, dropout figure (Fig. 8) |
+| `program_recovery` | Section 3.3 and Appendix C: Table 3 (program recovery for all representations) |
+| `lagged_association` | Section 3.4 and Appendix D: discrimination, direction, sampling resolution (Tables 4 to 6), lag figure (Fig. 9) |
+| `rms` | Section 3.5 and Appendix L: query genes on the rhabdomyosarcoma data (Figs. 11 to 15, Tables 20 and 21) |
+| `schiebinger` | Section 3.5 and Appendix K, the reprogramming time course (Schiebinger et al., days 0-3): Fig. 2, Table 19, the bridge genes between the programs of Lox and Birc5 (the query genes with `query_genes.py`; `bridges.py`, with the overlap of the bridges between training seeds against chance and the lagged-correlation baseline; the terms in which the bridges are enriched, which are not significant after correction, with `bridges_terms.py`; OTVelo as a baseline with `otvelo_baseline.py`) (needs the data, see below) |
+| `retrieval_curves` | Section 3.6 and Appendix G: recall curves (Figs. 3 and 10) |
+| `retrieval_selections` | Appendix F: ablation of the retrieval channels (Tables 9 and 10); the gene sets used by CardamomOT |
+| `experiments/raw_embeddings/run.py` | Appendix F: Tables 11 and 12, GSET without the learned encoders (a script of its own, not a block of `run_pipeline.py`; the table is printed by `tables/raw_embeddings/make_tables.py`) |
+| `experiments/spike_in/run.py`, `run_raw.py` | Appendix H: Table 15, 1000 real genes added to the 150-gene cascades, with and without the learned encoders (scripts of their own, results in `results/spike_in` and `results/spike_in_raw`; need `data/schiebinger`) |
+| `baseline_selections` | Appendix E: the gene sets of DELVE, OTVelo and OT similarity (needs `DELVE_REPO` and `OTVELO_REPO`) |
+| `extra_baselines` | Appendix E: Table 8 (temporal DE, GRNBoost2, GENIE3, WGCNA hubs, Hotspot; needs arboreto and hotspotsc, see `requirements.txt`) |
+| `cardamom` | Section 3.6, Appendices E and G: CardamomOT on the retrieved contexts (Tables 2, 7, 13 and 14) (needs `CARDAMOM_REPO` to infer the networks again) |
+| `scalability` | Appendix J: run times and memory (Tables 16 to 18) |
+
+## What is not in the repository
+
+* The whole reprogramming time course (165,892 cells, Schiebinger et al.; GEO: GSE122662) and the raw single-cell
+  RNA-seq counts of the rhabdomyosarcoma samples (GEO: GSE248183). Instead, the repository has the prepared days 0-3
+  of the reprogramming data (`data/schiebinger_days0-3.h5ad`) and the rhabdomyosarcoma counts with the pseudotime bins
+  (`data/rms/rms_counts.h5ad`). The code that makes these files from the full data is included:
+  `experiments/schiebinger/prepare_data.py` and `experiments/rms/make_pseudotime.py`.
+* **Code that made the simulations** is in `simulation_code/` as it was used and was not run again for
+  this repository. The simulated networks themselves are in `data/simulations/`.
+* **Baselines of other authors.** DELVE and OTVelo are not included (see Notes); the pipeline runs them
+  through the wrappers in `baselines/` when `DELVE_REPO` and `OTVELO_REPO` are set, and otherwise uses
+  the gene sets saved in `results/retrieval_selections/`. The networks inferred by CardamomOT are saved
+  in `results/cardamom_networks/`, and the tables are made from them.
+* The overview figure of the paper is a drawing and is not made by code.
+
+## Installation
+
+Tested with Python 3.13.
+
+```bash
+pip install -r requirements.txt
+```
+
+This is enough to rebuild every figure and table from the saved results
+(`python run_pipeline.py --all --from-results`) and to run GSET itself.
+
+To use the method in your own code, install the package `gset` too:
+
+```bash
+pip install -e .
+```
+
+### Methods of other authors (only to run them again)
+
+The results of these methods are saved in `results/`. To compute them again, install the method
+and set the variable; without the variable, `run_pipeline.py` skips the step and uses the saved
+results. The versions below are the ones checked to give the results of the paper.
+
+**CardamomOT** (block `cardamom`):
+
+```bash
+git clone https://github.com/eliasventre/CardamomOT
+cd CardamomOT && git checkout 36c5ca2
+pip install -e . && cd ..
+export CARDAMOM_REPO=$PWD/CardamomOT
+```
+
+Only on macOS with Apple silicon: if the process aborts when torch is loaded, apply
+`experiments/cardamom/cardamomot_serial_numba.patch` before `pip install -e .` (`git apply /path/to/this/repository/experiments/cardamom/cardamomot_serial_numba.patch`).
+It makes two numba loops serial instead of parallel. The networks of the paper were inferred with it.
+
+**OTVelo** (block `baseline_selections`; GPL-3.0, so it is not copied here):
+
+```bash
+git clone https://github.com/Sandstede-Lab/OT-Velocity
+cd OT-Velocity && git checkout 8b8df25 && cd ..
+export OTVELO_REPO=$PWD/OT-Velocity/Utils
+```
+
+**DELVE** (block `baseline_selections`). DELVE is used from its folder and not installed with pip,
+because its package asks for numpy < 2:
+
+```bash
+git clone https://github.com/jranek/delve
+cd delve && git checkout ff08435 && cd ..
+export DELVE_REPO=$PWD/delve
+pip install --no-deps sketchKH==0.1.4 tqdm_joblib==0.0.5
+```
+
+**GRNBoost2, GENIE3 and Hotspot** (block `extra_baselines`):
+
+```bash
+pip install --no-deps arboreto==0.1.6 hotspotsc==1.1.3
+```
+
+## Run
+
+A quick check that everything works (about ten seconds): it makes Table 1 and Figs. 6 to 8 from the saved results.
+
+```bash
+python run_pipeline.py --block endpoint_signals --from-results
+```
+
+The other blocks:
+
+```bash
+python run_pipeline.py --list                          # the blocks
+python run_pipeline.py --block endpoint_signals        # one block: experiment, figures, tables
+python run_pipeline.py --all --from-results            # only the figures and tables, from the saved results
+python run_pipeline.py --all                           # everything again (many hours)
+```
+
+Every experiment can also be run alone, for example
+`python experiments/program_recovery/run.py twophase60_s1 1`.
+
+The pseudotime bins of the rhabdomyosarcoma data can be made again from the raw single-cell RNA-seq counts
+(10x Genomics format). Set the folders of the two samples in `experiments/rms/make_pseudotime.py` and run
+(about five minutes):
+
+```bash
+pip install palantir leidenalg igraph
+python experiments/rms/make_pseudotime.py
+```
+
+## Use on your own data
+
+Install the package from the folder of this repository:
+
+```bash
+pip install -e .
+```
+
+Then:
+
+```python
+import anndata as ad
+from gset import select_genes
+
+adata = ad.read_h5ad("my_data.h5ad")    # raw counts in X, the time of every cell in adata.obs["time"]
+genes = adata.var_names[select_genes(adata, budget=30, n_programs=3)]
+```
+
+`select_genes` trains the encoders on your data (see Notes for the two co-expression encoders) and returns the
+indices of the selected genes: bridges between consecutive gene programs and the core
+genes of the programs. By default a third of the budget goes to the cores and two thirds to the bridges, and the
+programs are found by clustering, with their number given. Other settings:
+
+```python
+# the numbers of cores and bridges (the budget is their sum), and of the two channels of the bridges
+genes = select_genes(adata, n_cores=10, n_bridges=20, n_kl=14, n_ot=6, n_programs=3)
+
+# programs built around one query gene each, in the order in which the programs become active.
+# The program of a query gene is the gene and its program_size - 1 nearest genes, here 10 genes in all.
+queries = [adata.var_names.get_loc("Lox"), adata.var_names.get_loc("Birc5")]
+genes = select_genes(adata, budget=30, query_genes=queries, program_size=10)
+
+# programs of your own: the program of every gene (0, 1, 2, ...) and the order of the programs
+genes = select_genes(adata, budget=30, labels=labels, order=[0, 1, 2])
+
+# the co-expression encoder of the experiments on the real data, for thousands of genes
+genes = select_genes(adata, budget=30, n_programs=3, coexpr_encoder="real_data")
+
+# the data as a list of count matrices (cells x genes), one per timepoint, instead of an AnnData object
+genes = select_genes([X_t0, X_t1, X_t2], budget=30, n_programs=3)
+```
+
+With several replicates, pass a list of such lists. The genes can then be given to CardamomOT or another network
+inference method. The genes related to a query gene can also be explored interactively in the Gene Program Explorer
+(https://rmsexplorer.org).
+
+## Notes
+
+* **Same numbers on every run.** Every experiment calls `make_reproducible()`: torch uses one
+  thread and deterministic algorithms. Without it, the sums of torch change in the last digits
+  when it uses several threads, and this can change which genes are ranked first.
+* **Encoders.** The co-expression embedding, which gives the KL signal, has two encoders. The one in
+  `gset/embeddings.py` is used on the simulations, which have tens to a few hundred genes, and is the default of
+  `select_genes` (`coexpr_encoder="simulation"`). The one in `rmsexplorer/encoder.py` is used on the real data
+  (rhabdomyosarcoma, reprogramming), with thousands of genes (`coexpr_encoder="real_data"`), see `experiments/rms` and
+  `experiments/schiebinger`.
+* **Methods of other authors** (CardamomOT, OTVelo, DELVE, arboreto, Hotspot) are not part of this
+  repository; see Installation.
+* **Data.** The four data sets of the paper:
+  * *Negative binomial mixtures.* They are not stored: `experiments/nb_signals/nb_data.py` simulates them (40 genes,
+    5,000 cells per timepoint, two scenarios, rewiring and positional deviation, at several dropout rates). The scores are
+    in `results/nb_signals/`.
+  * *HARISSA simulations* (https://github.com/ulysseherbach/harissa). Cascades, switches and the families with
+    TF-to-TF relays, 41 networks in `data/simulations/`, made by `simulation_code/`.
+  * *Reprogramming time course* of Schiebinger et al. (Cell, 2019; GEO: GSE122662, serum condition), days 0-3,
+    34,101 cells and 4,007 genes, in `data/schiebinger_days0-3.h5ad`. Run
+    `python experiments/schiebinger/unpack_data.py` once to unpack it into `data/schiebinger/`, which the block
+    `schiebinger` and `experiments/spike_in/` use. The same files can be made from the original serum data, saved as
+    `data/schiebinger_raw/reprogramming_schiebinger_only_serum.h5ad`, with `experiments/schiebinger/prepare_data.py`.
+    `query_genes.py` also needs Table S2 of the paper (`mmc2.xlsx`), saved as
+    `data/schiebinger_raw/schiebinger2019_mmc2.xlsx`.
+  * *Rhabdomyosarcoma (RMS)* of Savary et al. (Cell Reports Medicine, 2023; GEO: GSE248183), 13,968 cells and 8,442
+    genes, with the six pseudotime bins (the column `time`), the cell types (`cell_type`) and the UMAP coordinates, in
+    `data/rms/`. All the analyses use this file. `experiments/rms/make_pseudotime.py` makes a similar file from the raw
+    single-cell RNA-seq counts (see Run). It needs `palantir`, `leidenalg` and `igraph`.

gset-0.1.0/gset/__init__.py

@@ -0,0 +1,11 @@
+from .graph import build_graph
+from .embeddings import (train_gcn, coexpr_embedding,
+                         train_deepsets, expr_embedding, train_both)
+from .signals import kl_vector, kl_score, ot_residual, ot_score
+from .lags import lag_matrix, symmetric_score
+from .programs import find_programs, program_order, core_genes, programs_from_queries
+from .selection import gset_context, bridge_genes, hvg_ranking, tf_selection, random_selection
+from .metrics import relay_coverage, within_between_gap
+from .pipeline import transition_vectors
+from .reproducible import make_reproducible
+from .api import select_genes, matrices_from_anndata

gset-0.1.0/gset/api.py

@@ -0,0 +1,109 @@
+"""select_genes: GSET in one function.
+
+    from gset import select_genes
+    genes = select_genes(adata, budget=30, n_programs=3)
+
+It follows the steps of the paper:
+  1. train the encoders and compute the KL and OT signals of every transition
+  2. find the gene programs, and their order
+  3. choose the bridge genes between consecutive programs and the core genes of the programs
+"""
+import numpy as np
+
+from .lags import lag_matrix
+from .pipeline import transition_vectors
+from .programs import concat_embedding, find_programs, mean_profiles, program_order, programs_from_queries
+from .selection import gset_context
+
+
+def matrices_from_anndata(adata, time_key="time"):
+    """The count matrices (cells x genes), one per timepoint, in the order of the time.
+    The counts are in adata.X and the time of every cell is in adata.obs[time_key]."""
+    X = adata.X.toarray() if hasattr(adata.X, "toarray") else np.asarray(adata.X)
+    times = np.asarray(adata.obs[time_key])
+    return [X[times == t].astype(float) for t in np.unique(times)]
+
+
+def _runs_from(data, time_key):
+    """Every run is a dict {timepoint: matrix}. Here: one run, or one for every replicate."""
+    if hasattr(data, "obs"):                      # an AnnData object
+        data = matrices_from_anndata(data, time_key)
+    if isinstance(data[0], np.ndarray):           # one run: a list of matrices
+        data = [data]
+    runs = [dict(enumerate(run)) for run in data]
+    return runs, list(range(len(runs[0])))
+
+
+def _split_budget(budget, n_cores, n_bridges, n_kl, n_ot):
+    """How many genes are cores, bridges from the KL signal and bridges from the OT signal."""
+    if n_cores is None and n_bridges is None:
+        n_cores = budget // 3                     # a third of the budget for the cores
+        n_bridges = budget - n_cores              # two thirds for the bridges
+    elif n_cores is None:
+        n_cores = budget - n_bridges
+    elif n_bridges is None:
+        n_bridges = budget - n_cores
+    if n_kl is None and n_ot is None:
+        n_ot = int(round(0.3 * n_bridges))        # 70% of the bridges from KL, 30% from OT
+    if n_kl is None:
+        n_kl = n_bridges - n_ot
+    if n_ot is None:
+        n_ot = n_bridges - n_kl
+    return n_cores + n_bridges, n_kl, n_ot
+
+
+def select_genes(data, budget=30, n_programs=3, n_cores=None, n_bridges=None, n_kl=None, n_ot=None,
+                 query_genes=None, program_size=30, labels=None, order=None,
+                 coexpr_encoder="simulation", time_key="time", seed=42, n_epochs=None):
+    """Select a compact set of genes from time-resolved single-cell data.
+
+    Returns the sorted list of the indices of the selected genes: the bridge genes between consecutive
+    programs and the core genes of the programs.
+
+    Data
+      data           an AnnData object (raw counts in X, the time of every cell in obs[time_key]) or a list of
+                     count matrices (cells x genes), one per timepoint in the order of the time. With several
+                     replicates, a list of such lists. The cells of different timepoints do not have to be matched.
+      time_key       the column of obs with the time of the cells ("time")
+
+    Number of genes
+      budget         how many genes to select (30)
+      n_cores        how many of them are core genes (default: a third of the budget)
+      n_bridges      how many are bridges (default: the rest). If you give both n_cores and n_bridges, the budget
+                     is their sum.
+      n_kl, n_ot     how many of the bridges come from the KL signal without lag and from the OT signal with lag 1
+                     (default: 70% and 30%)
+
+    Programs (the groups of genes that work together). Choose one of the three ways:
+      n_programs     the number of programs, found by clustering the genes (3)
+      query_genes    the indices of one gene for every program, in the order in which the programs become active.
+                     The program of a query gene is the gene and its program_size - 1 nearest genes (program_size
+                     is 30), and the other genes go to the program with the nearest centroid.
+      labels, order  the program of every gene (0, 1, 2, ...) and the order of the programs (by default 0, 1, 2, ...)
+
+    Encoders
+      coexpr_encoder  the encoder of the co-expression embedding (the KL signal). "simulation" (default): the
+                     encoder of the simulations, for panels of tens to a few hundred genes. "real_data": the encoder
+                     of the experiments on the real data, for thousands of genes (rmsexplorer/encoder.py).
+      seed, n_epochs the training. n_epochs: 300 for "simulation", 200 and 150 for "real_data" (default).
+    """
+    runs, times = _runs_from(data, time_key)
+
+    # 1. the signals of every transition
+    vectors = transition_vectors(runs, times, seed=seed, n_epochs=n_epochs, coexpr_encoder=coexpr_encoder)
+    kl_lag0 = lag_matrix(vectors["kl"], 0)
+    ot_lag1 = lag_matrix(vectors["ot"], 1)
+    z_concat = concat_embedding(vectors["gcn"], runs, vectors["active"], embed=vectors["embed"])
+
+    # 2. the programs and their order
+    if query_genes is not None:
+        labels, order = programs_from_queries(z_concat, query_genes, program_size)
+    elif labels is None:
+        labels = find_programs(z_concat, n_programs)
+        order, _ = program_order(labels, mean_profiles(runs, vectors["active"]))
+    elif order is None:
+        order = list(range(int(np.max(labels)) + 1))
+
+    # 3. the bridges and the cores
+    budget, n_kl, n_ot = _split_budget(budget, n_cores, n_bridges, n_kl, n_ot)
+    return gset_context(budget, n_kl, n_ot, np.asarray(labels), order, kl_lag0, ot_lag1, z_concat)