discotoolkit 1.1.4__tar.gz → 1.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- discotoolkit-1.2.0/MANIFEST.in +2 -0
- discotoolkit-1.2.0/PKG-INFO +140 -0
- discotoolkit-1.2.0/README.md +110 -0
- {discotoolkit-1.1.4 → discotoolkit-1.2.0}/discotoolkit/CELLiD.py +41 -22
- {discotoolkit-1.1.4 → discotoolkit-1.2.0}/discotoolkit/DownloadDiscoData.py +3 -3
- {discotoolkit-1.1.4 → discotoolkit-1.2.0}/discotoolkit/GeneSearch.py +4 -2
- {discotoolkit-1.1.4 → discotoolkit-1.2.0}/discotoolkit/GetMetadata.py +4 -6
- discotoolkit-1.2.0/discotoolkit/GlobalVariable.py +81 -0
- {discotoolkit-1.1.4 → discotoolkit-1.2.0}/discotoolkit/__init__.py +2 -1
- discotoolkit-1.2.0/discotoolkit.egg-info/PKG-INFO +140 -0
- {discotoolkit-1.1.4 → discotoolkit-1.2.0}/discotoolkit.egg-info/SOURCES.txt +6 -1
- {discotoolkit-1.1.4 → discotoolkit-1.2.0}/discotoolkit.egg-info/requires.txt +4 -0
- discotoolkit-1.2.0/pyproject.toml +3 -0
- discotoolkit-1.2.0/requirements.txt +12 -0
- {discotoolkit-1.1.4 → discotoolkit-1.2.0}/setup.py +5 -6
- discotoolkit-1.2.0/tests/test_server_contract.py +160 -0
- discotoolkit-1.2.0/tests/test_settings.py +90 -0
- discotoolkit-1.2.0/tests/test_version.py +41 -0
- discotoolkit-1.1.4/MANIFEST.in +0 -1
- discotoolkit-1.1.4/PKG-INFO +0 -13
- discotoolkit-1.1.4/README.md +0 -72
- discotoolkit-1.1.4/discotoolkit/GlobalVariable.py +0 -33
- discotoolkit-1.1.4/discotoolkit.egg-info/PKG-INFO +0 -13
- {discotoolkit-1.1.4 → discotoolkit-1.2.0}/discotoolkit/CellMapper.py +0 -0
- {discotoolkit-1.1.4 → discotoolkit-1.2.0}/discotoolkit/DiscoClass.py +0 -0
- {discotoolkit-1.1.4 → discotoolkit-1.2.0}/discotoolkit/Utilities.py +0 -0
- {discotoolkit-1.1.4 → discotoolkit-1.2.0}/discotoolkit.egg-info/dependency_links.txt +0 -0
- {discotoolkit-1.1.4 → discotoolkit-1.2.0}/discotoolkit.egg-info/top_level.txt +0 -0
- {discotoolkit-1.1.4 → discotoolkit-1.2.0}/setup.cfg +0 -0
|
@@ -0,0 +1,140 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: discotoolkit
|
|
3
|
+
Version: 1.2.0
|
|
4
|
+
Summary: DISCOtoolkit is an python package that allows users to access data and use the tools provided by the DISCO database.
|
|
5
|
+
Home-page: https://disco.bii.a-star.edu.sg/
|
|
6
|
+
Author: Li Mengwei, Rom Uddamvathanak
|
|
7
|
+
Author-email: uddamvathanak_rom@immunol.a-star.edu.sg
|
|
8
|
+
Requires-Python: >=3.9
|
|
9
|
+
Description-Content-Type: text/markdown
|
|
10
|
+
Requires-Dist: pandas>=1.4.2
|
|
11
|
+
Requires-Dist: numpy>=1.21.6
|
|
12
|
+
Requires-Dist: scanpy>=1.9.3
|
|
13
|
+
Requires-Dist: scipy>=1.10.1
|
|
14
|
+
Requires-Dist: joblib>=1.1.0
|
|
15
|
+
Requires-Dist: pandarallel>=1.6.5
|
|
16
|
+
Requires-Dist: requests>=2.28.1
|
|
17
|
+
Requires-Dist: colorcet>=3.0.1
|
|
18
|
+
Requires-Dist: leidenalg
|
|
19
|
+
Requires-Dist: h5py>=3.8
|
|
20
|
+
Requires-Dist: matplotlib>=3.5
|
|
21
|
+
Requires-Dist: seaborn>=0.12
|
|
22
|
+
Dynamic: author
|
|
23
|
+
Dynamic: author-email
|
|
24
|
+
Dynamic: description
|
|
25
|
+
Dynamic: description-content-type
|
|
26
|
+
Dynamic: home-page
|
|
27
|
+
Dynamic: requires-dist
|
|
28
|
+
Dynamic: requires-python
|
|
29
|
+
Dynamic: summary
|
|
30
|
+
|
|
31
|
+
<!--
|
|
32
|
+
* @Descripttion:
|
|
33
|
+
* @version:
|
|
34
|
+
* @Author: Mengwei Li
|
|
35
|
+
* @Date: 2023-04-16 21:20:42
|
|
36
|
+
* @LastEditors: Mengwei Li
|
|
37
|
+
* @LastEditTime: 2023-04-16 21:22:03
|
|
38
|
+
-->
|
|
39
|
+
[](https://discotoolkit-py.readthedocs.io/en/latest/?badge=latest) [](https://pepy.tech/project/discotoolkit) [](https://pypi.org/project/discotoolkit)
|
|
40
|
+
|
|
41
|
+
# DISCOtoolkit 1.2.0
|
|
42
|
+
|
|
43
|
+
DISCOtoolkit is a Python package for accessing the data and tools of the [DISCO database](https://disco.bii.a-star.edu.sg/) (DISCO v1). Read the documentation at [discotoolkit-py.readthedocs.io](https://discotoolkit-py.readthedocs.io/en/latest/).
|
|
44
|
+
|
|
45
|
+
- Filter and download DISCO data based on sample metadata and cell type information
|
|
46
|
+
- Gene search: a gene's expression across cell types and tissues
|
|
47
|
+
- CELLiD: cell type annotation
|
|
48
|
+
- scEnrichment: gene set enrichment using DISCO DEGs
|
|
49
|
+
|
|
50
|
+
> **DISCO v1 and v2.** The toolkit talks to **DISCO v1** by default. DISCO v2 has its own R package ([DISCOtoolkit](https://github.com/JinmiaoChenLab/DISCOtoolkit)); the Python package can still be pointed at a v2 server, see [Choosing a server](#choosing-a-server).
|
|
51
|
+
|
|
52
|
+
## Quickstart (Google Colab)
|
|
53
|
+
|
|
54
|
+
[](https://colab.research.google.com/github/JinmiaoChenLab/DISCOtoolkit_py/blob/main/docs/quickstart_disco_v1.ipynb)
|
|
55
|
+
|
|
56
|
+
Installs, checks the server, filters, downloads and plots in about ten cells, with nothing to set up locally.
|
|
57
|
+
|
|
58
|
+
## Installation
|
|
59
|
+
|
|
60
|
+
Python 3.9 or newer. In your current environment:
|
|
61
|
+
|
|
62
|
+
```
|
|
63
|
+
pip install discotoolkit
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
To try the latest development version straight from GitHub:
|
|
67
|
+
|
|
68
|
+
```
|
|
69
|
+
pip install "git+https://github.com/JinmiaoChenLab/DISCOtoolkit_py.git"
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
To work on the code, or to test changes before they are released, install from a local clone. `-e` links the install to the folder, so edits take effect without reinstalling:
|
|
73
|
+
|
|
74
|
+
```
|
|
75
|
+
git clone https://github.com/JinmiaoChenLab/DISCOtoolkit_py.git
|
|
76
|
+
cd DISCOtoolkit_py
|
|
77
|
+
pip install -e .
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
`pip install discotoolkit` and a local install give the same code when the versions match. The difference is the source: PyPI delivers a released, packaged copy, while a local install uses the files in your folder.
|
|
81
|
+
|
|
82
|
+
Dependencies (installed automatically): numpy, pandas, scanpy, scipy, joblib, pandarallel, requests, colorcet, leidenalg, h5py, matplotlib, seaborn.
|
|
83
|
+
|
|
84
|
+
We recommend a virtual environment, for example with miniconda:
|
|
85
|
+
|
|
86
|
+
```
|
|
87
|
+
conda create --name disco python=3.10
|
|
88
|
+
conda activate disco
|
|
89
|
+
conda install ipykernel
|
|
90
|
+
python -m ipykernel install --user --name disco --display-name "disco"
|
|
91
|
+
python -m pip install -U discotoolkit
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
## Choosing a server
|
|
95
|
+
|
|
96
|
+
```python
|
|
97
|
+
import discotoolkit as dt
|
|
98
|
+
|
|
99
|
+
dt.get_server() # 'https://disco.bii.a-star.edu.sg/disco_v3_api/' (DISCO v1, the default)
|
|
100
|
+
dt.set_server("v2") # DISCO v2
|
|
101
|
+
dt.set_server("https://my.server/disco_v3_api/") # any server with the same API
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
or set the `DISCO_API_URL` environment variable before importing the package.
|
|
105
|
+
|
|
106
|
+
## Testing
|
|
107
|
+
|
|
108
|
+
Two small test files need nothing beyond `requests`, so they run anywhere, including Colab:
|
|
109
|
+
|
|
110
|
+
```
|
|
111
|
+
python tests/test_settings.py # offline: server selection logic
|
|
112
|
+
python tests/test_version.py # offline: setup.py and __init__.py agree on the version
|
|
113
|
+
python tests/test_server_contract.py # online: does the server answer everything the toolkit needs?
|
|
114
|
+
python tests/test_server_contract.py https://immunesinglecell.org/disco_v3_api/ # check another server
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
GitHub Actions runs these, and installs the package on Python 3.9-3.12, on every push. Maintainers: see [RELEASING.md](RELEASING.md) for how a release is published.
|
|
118
|
+
|
|
119
|
+
`test_server_contract.py` checks status codes and the shape of each response (the columns and file types the toolkit reads). It only samples the large reference files, it does not download them.
|
|
120
|
+
|
|
121
|
+
## Basic Usage
|
|
122
|
+
Example in Jupyter notebook.
|
|
123
|
+
|
|
124
|
+
<em>please select disco as the kernel for running the jupyter notebook</em>
|
|
125
|
+
|
|
126
|
+
### [Quickstart for DISCO v1](https://github.com/JinmiaoChenLab/DISCOtoolkit_py/blob/main/docs/quickstart_disco_v1.ipynb)
|
|
127
|
+
|
|
128
|
+
### [Filter and download DISCO data](https://github.com/JinmiaoChenLab/DISCOtoolkit_py/blob/main/docs/download_data.ipynb)
|
|
129
|
+
|
|
130
|
+
### [Cell Type Annotation using CELLiD](https://github.com/JinmiaoChenLab/DISCOtoolkit_py/blob/main/docs/CELLiD_celltype_annotation.ipynb)
|
|
131
|
+
|
|
132
|
+
### [scEnrichment](https://github.com/JinmiaoChenLab/DISCOtoolkit_py/blob/main/docs/scEnrichment.ipynb)
|
|
133
|
+
|
|
134
|
+
## Citation
|
|
135
|
+
1. [Li, Mengwei, et al. "DISCO: a database of Deeply Integrated human Single-Cell Omics data." Nucleic acids research 50.D1 (2022): D596-D602.](https://academic.oup.com/nar/article/50/D1/D596/6430491)
|
|
136
|
+
2. [Mengwei Li, Kok Siong Ang, Brian Teo, Uddamvathanak Rom, Minh N Nguyen, Sebastian Maurer-Stroh, Jinmiao Chen. "Rediscovering publicly available single-cell data with the DISCO platform." Nucleic Acids Research (2024): gkae1108](https://academic.oup.com/nar/advance-article/doi/10.1093/nar/gkae1108/7899529)
|
|
137
|
+
|
|
138
|
+
## Follow us on our social media!
|
|
139
|
+
- [HSCRM2](https://twitter.com/HSCRM2)
|
|
140
|
+
- [JinmiaoChenLab Github repo](https://github.com/JinmiaoChenLab)
|
|
@@ -0,0 +1,110 @@
|
|
|
1
|
+
<!--
|
|
2
|
+
* @Descripttion:
|
|
3
|
+
* @version:
|
|
4
|
+
* @Author: Mengwei Li
|
|
5
|
+
* @Date: 2023-04-16 21:20:42
|
|
6
|
+
* @LastEditors: Mengwei Li
|
|
7
|
+
* @LastEditTime: 2023-04-16 21:22:03
|
|
8
|
+
-->
|
|
9
|
+
[](https://discotoolkit-py.readthedocs.io/en/latest/?badge=latest) [](https://pepy.tech/project/discotoolkit) [](https://pypi.org/project/discotoolkit)
|
|
10
|
+
|
|
11
|
+
# DISCOtoolkit 1.2.0
|
|
12
|
+
|
|
13
|
+
DISCOtoolkit is a Python package for accessing the data and tools of the [DISCO database](https://disco.bii.a-star.edu.sg/) (DISCO v1). Read the documentation at [discotoolkit-py.readthedocs.io](https://discotoolkit-py.readthedocs.io/en/latest/).
|
|
14
|
+
|
|
15
|
+
- Filter and download DISCO data based on sample metadata and cell type information
|
|
16
|
+
- Gene search: a gene's expression across cell types and tissues
|
|
17
|
+
- CELLiD: cell type annotation
|
|
18
|
+
- scEnrichment: gene set enrichment using DISCO DEGs
|
|
19
|
+
|
|
20
|
+
> **DISCO v1 and v2.** The toolkit talks to **DISCO v1** by default. DISCO v2 has its own R package ([DISCOtoolkit](https://github.com/JinmiaoChenLab/DISCOtoolkit)); the Python package can still be pointed at a v2 server, see [Choosing a server](#choosing-a-server).
|
|
21
|
+
|
|
22
|
+
## Quickstart (Google Colab)
|
|
23
|
+
|
|
24
|
+
[](https://colab.research.google.com/github/JinmiaoChenLab/DISCOtoolkit_py/blob/main/docs/quickstart_disco_v1.ipynb)
|
|
25
|
+
|
|
26
|
+
Installs, checks the server, filters, downloads and plots in about ten cells, with nothing to set up locally.
|
|
27
|
+
|
|
28
|
+
## Installation
|
|
29
|
+
|
|
30
|
+
Python 3.9 or newer. In your current environment:
|
|
31
|
+
|
|
32
|
+
```
|
|
33
|
+
pip install discotoolkit
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
To try the latest development version straight from GitHub:
|
|
37
|
+
|
|
38
|
+
```
|
|
39
|
+
pip install "git+https://github.com/JinmiaoChenLab/DISCOtoolkit_py.git"
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
To work on the code, or to test changes before they are released, install from a local clone. `-e` links the install to the folder, so edits take effect without reinstalling:
|
|
43
|
+
|
|
44
|
+
```
|
|
45
|
+
git clone https://github.com/JinmiaoChenLab/DISCOtoolkit_py.git
|
|
46
|
+
cd DISCOtoolkit_py
|
|
47
|
+
pip install -e .
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
`pip install discotoolkit` and a local install give the same code when the versions match. The difference is the source: PyPI delivers a released, packaged copy, while a local install uses the files in your folder.
|
|
51
|
+
|
|
52
|
+
Dependencies (installed automatically): numpy, pandas, scanpy, scipy, joblib, pandarallel, requests, colorcet, leidenalg, h5py, matplotlib, seaborn.
|
|
53
|
+
|
|
54
|
+
We recommend a virtual environment, for example with miniconda:
|
|
55
|
+
|
|
56
|
+
```
|
|
57
|
+
conda create --name disco python=3.10
|
|
58
|
+
conda activate disco
|
|
59
|
+
conda install ipykernel
|
|
60
|
+
python -m ipykernel install --user --name disco --display-name "disco"
|
|
61
|
+
python -m pip install -U discotoolkit
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
## Choosing a server
|
|
65
|
+
|
|
66
|
+
```python
|
|
67
|
+
import discotoolkit as dt
|
|
68
|
+
|
|
69
|
+
dt.get_server() # 'https://disco.bii.a-star.edu.sg/disco_v3_api/' (DISCO v1, the default)
|
|
70
|
+
dt.set_server("v2") # DISCO v2
|
|
71
|
+
dt.set_server("https://my.server/disco_v3_api/") # any server with the same API
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
or set the `DISCO_API_URL` environment variable before importing the package.
|
|
75
|
+
|
|
76
|
+
## Testing
|
|
77
|
+
|
|
78
|
+
Two small test files need nothing beyond `requests`, so they run anywhere, including Colab:
|
|
79
|
+
|
|
80
|
+
```
|
|
81
|
+
python tests/test_settings.py # offline: server selection logic
|
|
82
|
+
python tests/test_version.py # offline: setup.py and __init__.py agree on the version
|
|
83
|
+
python tests/test_server_contract.py # online: does the server answer everything the toolkit needs?
|
|
84
|
+
python tests/test_server_contract.py https://immunesinglecell.org/disco_v3_api/ # check another server
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
GitHub Actions runs these, and installs the package on Python 3.9-3.12, on every push. Maintainers: see [RELEASING.md](RELEASING.md) for how a release is published.
|
|
88
|
+
|
|
89
|
+
`test_server_contract.py` checks status codes and the shape of each response (the columns and file types the toolkit reads). It only samples the large reference files, it does not download them.
|
|
90
|
+
|
|
91
|
+
## Basic Usage
|
|
92
|
+
Example in Jupyter notebook.
|
|
93
|
+
|
|
94
|
+
<em>please select disco as the kernel for running the jupyter notebook</em>
|
|
95
|
+
|
|
96
|
+
### [Quickstart for DISCO v1](https://github.com/JinmiaoChenLab/DISCOtoolkit_py/blob/main/docs/quickstart_disco_v1.ipynb)
|
|
97
|
+
|
|
98
|
+
### [Filter and download DISCO data](https://github.com/JinmiaoChenLab/DISCOtoolkit_py/blob/main/docs/download_data.ipynb)
|
|
99
|
+
|
|
100
|
+
### [Cell Type Annotation using CELLiD](https://github.com/JinmiaoChenLab/DISCOtoolkit_py/blob/main/docs/CELLiD_celltype_annotation.ipynb)
|
|
101
|
+
|
|
102
|
+
### [scEnrichment](https://github.com/JinmiaoChenLab/DISCOtoolkit_py/blob/main/docs/scEnrichment.ipynb)
|
|
103
|
+
|
|
104
|
+
## Citation
|
|
105
|
+
1. [Li, Mengwei, et al. "DISCO: a database of Deeply Integrated human Single-Cell Omics data." Nucleic acids research 50.D1 (2022): D596-D602.](https://academic.oup.com/nar/article/50/D1/D596/6430491)
|
|
106
|
+
2. [Mengwei Li, Kok Siong Ang, Brian Teo, Uddamvathanak Rom, Minh N Nguyen, Sebastian Maurer-Stroh, Jinmiao Chen. "Rediscovering publicly available single-cell data with the DISCO platform." Nucleic Acids Research (2024): gkae1108](https://academic.oup.com/nar/advance-article/doi/10.1093/nar/gkae1108/7899529)
|
|
107
|
+
|
|
108
|
+
## Follow us on our social media!
|
|
109
|
+
- [HSCRM2](https://twitter.com/HSCRM2)
|
|
110
|
+
- [JinmiaoChenLab Github repo](https://github.com/JinmiaoChenLab)
|
|
@@ -18,7 +18,7 @@ from pandarallel import pandarallel # multiprocessing library
|
|
|
18
18
|
pd.options.mode.chained_assignment = None
|
|
19
19
|
|
|
20
20
|
# import variable and class from other script
|
|
21
|
-
from .GlobalVariable import logging,
|
|
21
|
+
from .GlobalVariable import logging, api_url, timeout
|
|
22
22
|
from .DiscoClass import FilterData, Filter
|
|
23
23
|
from .GetMetadata import check_in_list
|
|
24
24
|
|
|
@@ -77,7 +77,7 @@ def get_atlas(ref_data=None, ref_path=None):
|
|
|
77
77
|
if not (os.path.exists(ref_path + "/ref_data.pkl")):
|
|
78
78
|
logging.info("Downloading reference dataset...")
|
|
79
79
|
response = requests.get(
|
|
80
|
-
url=
|
|
80
|
+
url=api_url("toolkit/getRef?type=pkl"), timeout=timeout
|
|
81
81
|
)
|
|
82
82
|
open(ref_path + "/ref_data.pkl", "wb").write(response.content)
|
|
83
83
|
ref_data = pd.read_pickle(
|
|
@@ -89,6 +89,42 @@ def get_atlas(ref_data=None, ref_path=None):
|
|
|
89
89
|
return list(set(all_atlas))
|
|
90
90
|
|
|
91
91
|
|
|
92
|
+
def _load_geneset(ref_path: str) -> pd.DataFrame:
|
|
93
|
+
"""Get the gene-set reference (columns logfc, gene, atlas, name), downloading it once.
|
|
94
|
+
|
|
95
|
+
DISCO v1 serves it as plain CSV (`toolkit/getGeneSet?format=csv`), which is read with
|
|
96
|
+
pandas.read_csv. Older servers only offer a pickle (`getGeneSetPkl`); that is used as a
|
|
97
|
+
fallback, with a warning, because unpickling data from a network is only safe if the
|
|
98
|
+
server is trusted.
|
|
99
|
+
"""
|
|
100
|
+
csv_file = ref_path + "/ref_geneset.csv"
|
|
101
|
+
pkl_file = ref_path + "/ref_geneset.pkl"
|
|
102
|
+
|
|
103
|
+
if os.path.exists(csv_file):
|
|
104
|
+
return pd.read_csv(csv_file)
|
|
105
|
+
if os.path.exists(pkl_file):
|
|
106
|
+
return pd.read_pickle(pkl_file, compression={"method": "gzip", "compresslevel": 6})
|
|
107
|
+
|
|
108
|
+
logging.info("Downloading gene set reference...")
|
|
109
|
+
response = requests.get(
|
|
110
|
+
url=api_url("toolkit/getGeneSet"), params={"format": "csv"}, timeout=timeout
|
|
111
|
+
)
|
|
112
|
+
if response.status_code == 200:
|
|
113
|
+
with open(csv_file, "wb") as handle:
|
|
114
|
+
handle.write(response.content)
|
|
115
|
+
return pd.read_csv(csv_file)
|
|
116
|
+
|
|
117
|
+
logging.warning(
|
|
118
|
+
"This server has no CSV gene set (HTTP %s); falling back to the pickle download.",
|
|
119
|
+
response.status_code,
|
|
120
|
+
)
|
|
121
|
+
response = requests.get(url=api_url("getGeneSetPkl"), timeout=timeout)
|
|
122
|
+
response.raise_for_status()
|
|
123
|
+
with open(pkl_file, "wb") as handle:
|
|
124
|
+
handle.write(response.content)
|
|
125
|
+
return pd.read_pickle(pkl_file, compression={"method": "gzip", "compresslevel": 6})
|
|
126
|
+
|
|
127
|
+
|
|
92
128
|
def CELLiD_cluster(
|
|
93
129
|
rna,
|
|
94
130
|
ref_data: pd.DataFrame = None,
|
|
@@ -144,7 +180,7 @@ def CELLiD_cluster(
|
|
|
144
180
|
if not (os.path.exists(ref_path + "/ref_data.pkl")):
|
|
145
181
|
logging.info("Downloading reference dataset...")
|
|
146
182
|
response = requests.get(
|
|
147
|
-
url=
|
|
183
|
+
url=api_url("toolkit/getRef?type=pkl"), timeout=timeout
|
|
148
184
|
)
|
|
149
185
|
open(ref_path + "/ref_data.pkl", "wb").write(response.content)
|
|
150
186
|
ref_data = pd.read_pickle(
|
|
@@ -157,7 +193,7 @@ def CELLiD_cluster(
|
|
|
157
193
|
if not (os.path.exists(ref_path + "/ref_deg.pkl")):
|
|
158
194
|
logging.info("Downloading deg dataset...")
|
|
159
195
|
response = requests.get(
|
|
160
|
-
url=
|
|
196
|
+
url=api_url("toolkit/getRefDeg?type=pkl"), timeout=timeout
|
|
161
197
|
)
|
|
162
198
|
open(ref_path + "/ref_deg.pkl", "wb").write(response.content)
|
|
163
199
|
ref_deg = pd.read_pickle(
|
|
@@ -347,24 +383,7 @@ def CELLiD_enrichment(
|
|
|
347
383
|
if not os.path.exists(ref_path):
|
|
348
384
|
os.mkdir(ref_path)
|
|
349
385
|
|
|
350
|
-
|
|
351
|
-
if not (os.path.exists(ref_path + "/ref_geneset.pkl")):
|
|
352
|
-
# downloading the geneset data from disco database
|
|
353
|
-
response = requests.get(
|
|
354
|
-
url=prefix_disco_url + "/getGeneSetPkl", timeout=timeout
|
|
355
|
-
)
|
|
356
|
-
open(ref_path + "/ref_geneset.pkl", "wb").write(response.content)
|
|
357
|
-
reference = pd.read_pickle(
|
|
358
|
-
ref_path + "/ref_geneset.pkl",
|
|
359
|
-
compression={"method": "gzip", "compresslevel": 6},
|
|
360
|
-
)
|
|
361
|
-
|
|
362
|
-
else:
|
|
363
|
-
# read the data into pandas dataframe for subsequent analysis
|
|
364
|
-
reference = pd.read_pickle(
|
|
365
|
-
ref_path + "/ref_geneset.pkl",
|
|
366
|
-
compression={"method": "gzip", "compresslevel": 6},
|
|
367
|
-
)
|
|
386
|
+
reference = _load_geneset(ref_path)
|
|
368
387
|
|
|
369
388
|
# rename the name data to include the reference atlas
|
|
370
389
|
reference["name"] = reference["name"] + " in " + reference["atlas"]
|
|
@@ -17,7 +17,7 @@ import hashlib
|
|
|
17
17
|
import scanpy as sc
|
|
18
18
|
|
|
19
19
|
# import variable and class from other script
|
|
20
|
-
from .GlobalVariable import logging,
|
|
20
|
+
from .GlobalVariable import logging, api_url
|
|
21
21
|
from .DiscoClass import FilterData, Filter
|
|
22
22
|
from .GetMetadata import check_in_list
|
|
23
23
|
|
|
@@ -52,11 +52,11 @@ def download_disco_data(metadata, output_dir: str = "DISCOtmp"):
|
|
|
52
52
|
if os.path.exists(output_file):
|
|
53
53
|
os.remove(output_file)
|
|
54
54
|
|
|
55
|
-
url =
|
|
55
|
+
url = api_url("download/getRawH5/" + p + "/" + s)
|
|
56
56
|
rna = sc.read_10x_h5(filename=output_file, backup_url=url)
|
|
57
57
|
|
|
58
58
|
cell = pd.read_csv(
|
|
59
|
-
|
|
59
|
+
api_url("toolkit/getCellTypeSample?sampleId=" + s),
|
|
60
60
|
sep="\t",
|
|
61
61
|
header=0,
|
|
62
62
|
)
|
|
@@ -7,6 +7,8 @@ import numpy as np
|
|
|
7
7
|
from typing import Union
|
|
8
8
|
import colorcet as cc
|
|
9
9
|
|
|
10
|
+
from .GlobalVariable import api_url
|
|
11
|
+
|
|
10
12
|
|
|
11
13
|
def gene_search(gene : str, atlas : Union[str, list] = None, figsize : tuple = None, dpi : int = 300):
|
|
12
14
|
|
|
@@ -27,8 +29,8 @@ def gene_search(gene : str, atlas : Union[str, list] = None, figsize : tuple = N
|
|
|
27
29
|
atlas = [atlas]
|
|
28
30
|
|
|
29
31
|
# url to get the API data
|
|
30
|
-
|
|
31
|
-
response
|
|
32
|
+
response = requests.get(api_url("geneExp/getRefExp"), params={"gene": gene})
|
|
33
|
+
response.raise_for_status()
|
|
32
34
|
data = response.json()
|
|
33
35
|
|
|
34
36
|
# Extract relevant information from JSON data
|
|
@@ -11,7 +11,7 @@ import numpy as np
|
|
|
11
11
|
from typing import Union
|
|
12
12
|
|
|
13
13
|
# import variable and class from other script
|
|
14
|
-
from .GlobalVariable import logging,
|
|
14
|
+
from .GlobalVariable import logging, api_url
|
|
15
15
|
from .DiscoClass import FilterData, Filter
|
|
16
16
|
|
|
17
17
|
|
|
@@ -35,7 +35,7 @@ def get_json(url: str, info_msg: str, error_msg: str, prefix: bool = True):
|
|
|
35
35
|
if not prefix:
|
|
36
36
|
response = requests.get(url)
|
|
37
37
|
else:
|
|
38
|
-
response = requests.get(
|
|
38
|
+
response = requests.get(api_url(url))
|
|
39
39
|
|
|
40
40
|
# if the loading of JSON is successful, we get the text
|
|
41
41
|
if response.status_code == 200:
|
|
@@ -51,9 +51,7 @@ def get_sample_ct_info():
|
|
|
51
51
|
Returns:
|
|
52
52
|
Pandas DataFrame: return pandas dataframe to the user
|
|
53
53
|
"""
|
|
54
|
-
temp = pd.read_csv(
|
|
55
|
-
prefix_disco_url + "toolkit/" + "getCellTypeSummary", sep="\t", header=0
|
|
56
|
-
)
|
|
54
|
+
temp = pd.read_csv(api_url("toolkit/getCellTypeSummary"), sep="\t", header=0)
|
|
57
55
|
return temp # return Dataframe of the JSON data
|
|
58
56
|
|
|
59
57
|
|
|
@@ -101,7 +99,7 @@ def get_disco_metadata():
|
|
|
101
99
|
"""
|
|
102
100
|
|
|
103
101
|
metadata = pd.read_csv(
|
|
104
|
-
"
|
|
102
|
+
api_url("toolkit/getSampleMetadata"),
|
|
105
103
|
sep="\t",
|
|
106
104
|
header=0,
|
|
107
105
|
)
|
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
"""
|
|
2
|
+
Settings shared by the rest of the package: logging, the request timeout and --
|
|
3
|
+
most importantly -- which DISCO server the toolkit talks to.
|
|
4
|
+
|
|
5
|
+
By default the toolkit talks to DISCO v1 (https://disco.bii.a-star.edu.sg). To use
|
|
6
|
+
another server:
|
|
7
|
+
|
|
8
|
+
import discotoolkit as dt
|
|
9
|
+
dt.set_server("v2") # a named preset
|
|
10
|
+
dt.set_server("https://my.server/disco_v3_api/") # any server with the same API
|
|
11
|
+
|
|
12
|
+
or set the DISCO_API_URL environment variable before importing the package.
|
|
13
|
+
"""
|
|
14
|
+
|
|
15
|
+
import logging
|
|
16
|
+
import os
|
|
17
|
+
|
|
18
|
+
logging.basicConfig(level=logging.INFO)
|
|
19
|
+
|
|
20
|
+
# seconds to wait for the server; the reference downloads are tens of MB
|
|
21
|
+
timeout = 600
|
|
22
|
+
|
|
23
|
+
# The API roots the toolkit knows by name. Both serve the same `toolkit/...` endpoints.
|
|
24
|
+
SERVERS = {
|
|
25
|
+
"v1": "https://disco.bii.a-star.edu.sg/disco_v3_api/",
|
|
26
|
+
"v2": "https://immunesinglecell.org/disco_v3_api/",
|
|
27
|
+
}
|
|
28
|
+
DEFAULT_SERVER = "v1"
|
|
29
|
+
|
|
30
|
+
|
|
31
|
+
def _normalise(server: str) -> str:
|
|
32
|
+
"""Turn a preset name or a URL into an API root that ends with exactly one '/'."""
|
|
33
|
+
if not isinstance(server, str) or not server.strip():
|
|
34
|
+
raise ValueError("server must be a preset name (%s) or a URL" % ", ".join(SERVERS))
|
|
35
|
+
server = server.strip()
|
|
36
|
+
if server.lower() in SERVERS:
|
|
37
|
+
return SERVERS[server.lower()]
|
|
38
|
+
if not server.lower().startswith(("http://", "https://")):
|
|
39
|
+
raise ValueError(
|
|
40
|
+
"unknown server %r: use one of %s or a full http(s):// URL" % (server, ", ".join(SERVERS))
|
|
41
|
+
)
|
|
42
|
+
return server.rstrip("/") + "/"
|
|
43
|
+
|
|
44
|
+
|
|
45
|
+
_server = _normalise(os.environ.get("DISCO_API_URL") or DEFAULT_SERVER)
|
|
46
|
+
|
|
47
|
+
|
|
48
|
+
def set_server(server: str) -> str:
|
|
49
|
+
"""Choose which DISCO server the toolkit uses for the rest of the session.
|
|
50
|
+
|
|
51
|
+
Args:
|
|
52
|
+
server (str): "v1", "v2", or the full URL of an API root such as
|
|
53
|
+
"https://disco.bii.a-star.edu.sg/disco_v3_api/".
|
|
54
|
+
|
|
55
|
+
Returns:
|
|
56
|
+
str: the API root now in use.
|
|
57
|
+
"""
|
|
58
|
+
global _server
|
|
59
|
+
_server = _normalise(server)
|
|
60
|
+
logging.info("DISCOtoolkit is using %s", _server)
|
|
61
|
+
return _server
|
|
62
|
+
|
|
63
|
+
|
|
64
|
+
def get_server() -> str:
|
|
65
|
+
"""The API root the toolkit is currently using."""
|
|
66
|
+
return _server
|
|
67
|
+
|
|
68
|
+
|
|
69
|
+
def api_url(path: str) -> str:
|
|
70
|
+
"""Full URL for an API path, e.g. api_url("toolkit/getSampleMetadata").
|
|
71
|
+
|
|
72
|
+
Looked up on every call, so set_server() takes effect immediately everywhere.
|
|
73
|
+
"""
|
|
74
|
+
return _server + path.lstrip("/")
|
|
75
|
+
|
|
76
|
+
|
|
77
|
+
def __getattr__(name):
|
|
78
|
+
# `prefix_disco_url` was a plain variable in earlier releases; keep it readable.
|
|
79
|
+
if name == "prefix_disco_url":
|
|
80
|
+
return _server
|
|
81
|
+
raise AttributeError("module %r has no attribute %r" % (__name__, name))
|
|
@@ -0,0 +1,140 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: discotoolkit
|
|
3
|
+
Version: 1.2.0
|
|
4
|
+
Summary: DISCOtoolkit is an python package that allows users to access data and use the tools provided by the DISCO database.
|
|
5
|
+
Home-page: https://disco.bii.a-star.edu.sg/
|
|
6
|
+
Author: Li Mengwei, Rom Uddamvathanak
|
|
7
|
+
Author-email: uddamvathanak_rom@immunol.a-star.edu.sg
|
|
8
|
+
Requires-Python: >=3.9
|
|
9
|
+
Description-Content-Type: text/markdown
|
|
10
|
+
Requires-Dist: pandas>=1.4.2
|
|
11
|
+
Requires-Dist: numpy>=1.21.6
|
|
12
|
+
Requires-Dist: scanpy>=1.9.3
|
|
13
|
+
Requires-Dist: scipy>=1.10.1
|
|
14
|
+
Requires-Dist: joblib>=1.1.0
|
|
15
|
+
Requires-Dist: pandarallel>=1.6.5
|
|
16
|
+
Requires-Dist: requests>=2.28.1
|
|
17
|
+
Requires-Dist: colorcet>=3.0.1
|
|
18
|
+
Requires-Dist: leidenalg
|
|
19
|
+
Requires-Dist: h5py>=3.8
|
|
20
|
+
Requires-Dist: matplotlib>=3.5
|
|
21
|
+
Requires-Dist: seaborn>=0.12
|
|
22
|
+
Dynamic: author
|
|
23
|
+
Dynamic: author-email
|
|
24
|
+
Dynamic: description
|
|
25
|
+
Dynamic: description-content-type
|
|
26
|
+
Dynamic: home-page
|
|
27
|
+
Dynamic: requires-dist
|
|
28
|
+
Dynamic: requires-python
|
|
29
|
+
Dynamic: summary
|
|
30
|
+
|
|
31
|
+
<!--
|
|
32
|
+
* @Descripttion:
|
|
33
|
+
* @version:
|
|
34
|
+
* @Author: Mengwei Li
|
|
35
|
+
* @Date: 2023-04-16 21:20:42
|
|
36
|
+
* @LastEditors: Mengwei Li
|
|
37
|
+
* @LastEditTime: 2023-04-16 21:22:03
|
|
38
|
+
-->
|
|
39
|
+
[](https://discotoolkit-py.readthedocs.io/en/latest/?badge=latest) [](https://pepy.tech/project/discotoolkit) [](https://pypi.org/project/discotoolkit)
|
|
40
|
+
|
|
41
|
+
# DISCOtoolkit 1.2.0
|
|
42
|
+
|
|
43
|
+
DISCOtoolkit is a Python package for accessing the data and tools of the [DISCO database](https://disco.bii.a-star.edu.sg/) (DISCO v1). Read the documentation at [discotoolkit-py.readthedocs.io](https://discotoolkit-py.readthedocs.io/en/latest/).
|
|
44
|
+
|
|
45
|
+
- Filter and download DISCO data based on sample metadata and cell type information
|
|
46
|
+
- Gene search: a gene's expression across cell types and tissues
|
|
47
|
+
- CELLiD: cell type annotation
|
|
48
|
+
- scEnrichment: gene set enrichment using DISCO DEGs
|
|
49
|
+
|
|
50
|
+
> **DISCO v1 and v2.** The toolkit talks to **DISCO v1** by default. DISCO v2 has its own R package ([DISCOtoolkit](https://github.com/JinmiaoChenLab/DISCOtoolkit)); the Python package can still be pointed at a v2 server, see [Choosing a server](#choosing-a-server).
|
|
51
|
+
|
|
52
|
+
## Quickstart (Google Colab)
|
|
53
|
+
|
|
54
|
+
[](https://colab.research.google.com/github/JinmiaoChenLab/DISCOtoolkit_py/blob/main/docs/quickstart_disco_v1.ipynb)
|
|
55
|
+
|
|
56
|
+
Installs, checks the server, filters, downloads and plots in about ten cells, with nothing to set up locally.
|
|
57
|
+
|
|
58
|
+
## Installation
|
|
59
|
+
|
|
60
|
+
Python 3.9 or newer. In your current environment:
|
|
61
|
+
|
|
62
|
+
```
|
|
63
|
+
pip install discotoolkit
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
To try the latest development version straight from GitHub:
|
|
67
|
+
|
|
68
|
+
```
|
|
69
|
+
pip install "git+https://github.com/JinmiaoChenLab/DISCOtoolkit_py.git"
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
To work on the code, or to test changes before they are released, install from a local clone. `-e` links the install to the folder, so edits take effect without reinstalling:
|
|
73
|
+
|
|
74
|
+
```
|
|
75
|
+
git clone https://github.com/JinmiaoChenLab/DISCOtoolkit_py.git
|
|
76
|
+
cd DISCOtoolkit_py
|
|
77
|
+
pip install -e .
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
`pip install discotoolkit` and a local install give the same code when the versions match. The difference is the source: PyPI delivers a released, packaged copy, while a local install uses the files in your folder.
|
|
81
|
+
|
|
82
|
+
Dependencies (installed automatically): numpy, pandas, scanpy, scipy, joblib, pandarallel, requests, colorcet, leidenalg, h5py, matplotlib, seaborn.
|
|
83
|
+
|
|
84
|
+
We recommend a virtual environment, for example with miniconda:
|
|
85
|
+
|
|
86
|
+
```
|
|
87
|
+
conda create --name disco python=3.10
|
|
88
|
+
conda activate disco
|
|
89
|
+
conda install ipykernel
|
|
90
|
+
python -m ipykernel install --user --name disco --display-name "disco"
|
|
91
|
+
python -m pip install -U discotoolkit
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
## Choosing a server
|
|
95
|
+
|
|
96
|
+
```python
|
|
97
|
+
import discotoolkit as dt
|
|
98
|
+
|
|
99
|
+
dt.get_server() # 'https://disco.bii.a-star.edu.sg/disco_v3_api/' (DISCO v1, the default)
|
|
100
|
+
dt.set_server("v2") # DISCO v2
|
|
101
|
+
dt.set_server("https://my.server/disco_v3_api/") # any server with the same API
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
or set the `DISCO_API_URL` environment variable before importing the package.
|
|
105
|
+
|
|
106
|
+
## Testing
|
|
107
|
+
|
|
108
|
+
Two small test files need nothing beyond `requests`, so they run anywhere, including Colab:
|
|
109
|
+
|
|
110
|
+
```
|
|
111
|
+
python tests/test_settings.py # offline: server selection logic
|
|
112
|
+
python tests/test_version.py # offline: setup.py and __init__.py agree on the version
|
|
113
|
+
python tests/test_server_contract.py # online: does the server answer everything the toolkit needs?
|
|
114
|
+
python tests/test_server_contract.py https://immunesinglecell.org/disco_v3_api/ # check another server
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
GitHub Actions runs these, and installs the package on Python 3.9-3.12, on every push. Maintainers: see [RELEASING.md](RELEASING.md) for how a release is published.
|
|
118
|
+
|
|
119
|
+
`test_server_contract.py` checks status codes and the shape of each response (the columns and file types the toolkit reads). It only samples the large reference files, it does not download them.
|
|
120
|
+
|
|
121
|
+
## Basic Usage
|
|
122
|
+
Example in Jupyter notebook.
|
|
123
|
+
|
|
124
|
+
<em>please select disco as the kernel for running the jupyter notebook</em>
|
|
125
|
+
|
|
126
|
+
### [Quickstart for DISCO v1](https://github.com/JinmiaoChenLab/DISCOtoolkit_py/blob/main/docs/quickstart_disco_v1.ipynb)
|
|
127
|
+
|
|
128
|
+
### [Filter and download DISCO data](https://github.com/JinmiaoChenLab/DISCOtoolkit_py/blob/main/docs/download_data.ipynb)
|
|
129
|
+
|
|
130
|
+
### [Cell Type Annotation using CELLiD](https://github.com/JinmiaoChenLab/DISCOtoolkit_py/blob/main/docs/CELLiD_celltype_annotation.ipynb)
|
|
131
|
+
|
|
132
|
+
### [scEnrichment](https://github.com/JinmiaoChenLab/DISCOtoolkit_py/blob/main/docs/scEnrichment.ipynb)
|
|
133
|
+
|
|
134
|
+
## Citation
|
|
135
|
+
1. [Li, Mengwei, et al. "DISCO: a database of Deeply Integrated human Single-Cell Omics data." Nucleic acids research 50.D1 (2022): D596-D602.](https://academic.oup.com/nar/article/50/D1/D596/6430491)
|
|
136
|
+
2. [Mengwei Li, Kok Siong Ang, Brian Teo, Uddamvathanak Rom, Minh N Nguyen, Sebastian Maurer-Stroh, Jinmiao Chen. "Rediscovering publicly available single-cell data with the DISCO platform." Nucleic Acids Research (2024): gkae1108](https://academic.oup.com/nar/advance-article/doi/10.1093/nar/gkae1108/7899529)
|
|
137
|
+
|
|
138
|
+
## Follow us on our social media!
|
|
139
|
+
- [HSCRM2](https://twitter.com/HSCRM2)
|
|
140
|
+
- [JinmiaoChenLab Github repo](https://github.com/JinmiaoChenLab)
|
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
MANIFEST.in
|
|
2
2
|
README.md
|
|
3
|
+
pyproject.toml
|
|
4
|
+
requirements.txt
|
|
3
5
|
setup.py
|
|
4
6
|
discotoolkit/CELLiD.py
|
|
5
7
|
discotoolkit/CellMapper.py
|
|
@@ -14,4 +16,7 @@ discotoolkit.egg-info/PKG-INFO
|
|
|
14
16
|
discotoolkit.egg-info/SOURCES.txt
|
|
15
17
|
discotoolkit.egg-info/dependency_links.txt
|
|
16
18
|
discotoolkit.egg-info/requires.txt
|
|
17
|
-
discotoolkit.egg-info/top_level.txt
|
|
19
|
+
discotoolkit.egg-info/top_level.txt
|
|
20
|
+
tests/test_server_contract.py
|
|
21
|
+
tests/test_settings.py
|
|
22
|
+
tests/test_version.py
|
|
@@ -6,16 +6,15 @@ with open("requirements.txt", "r") as f:
|
|
|
6
6
|
|
|
7
7
|
setup(
|
|
8
8
|
name='discotoolkit',
|
|
9
|
-
version="1.
|
|
10
|
-
url='
|
|
9
|
+
version="1.2.0",
|
|
10
|
+
url='https://disco.bii.a-star.edu.sg/',
|
|
11
11
|
author='Li Mengwei, Rom Uddamvathanak',
|
|
12
12
|
author_email='uddamvathanak_rom@immunol.a-star.edu.sg',
|
|
13
13
|
description='DISCOtoolkit is an python package that allows users to access data and use the tools provided by the DISCO database.',
|
|
14
14
|
packages=find_packages(include=["discotoolkit", "discotoolkit.*"]),
|
|
15
15
|
install_requires=requirements,
|
|
16
|
-
python_requires = '>=3.
|
|
16
|
+
python_requires = '>=3.9',
|
|
17
17
|
include_package_data=True,
|
|
18
|
-
long_description="
|
|
19
|
-
|
|
20
|
-
cell type annotation\nscEnrichment: geneset enrichment using DISCO DEGs",
|
|
18
|
+
long_description=open("README.md", encoding="utf-8").read(),
|
|
19
|
+
long_description_content_type="text/markdown",
|
|
21
20
|
)
|
|
@@ -0,0 +1,160 @@
|
|
|
1
|
+
"""
|
|
2
|
+
Does a DISCO server answer everything discotoolkit needs?
|
|
3
|
+
|
|
4
|
+
Needs only `requests`, so it runs anywhere -- including a fresh Colab notebook:
|
|
5
|
+
|
|
6
|
+
!pip install requests
|
|
7
|
+
!python tests/test_server_contract.py # DISCO v1 (default)
|
|
8
|
+
!python tests/test_server_contract.py https://immunesinglecell.org/disco_v3_api/ # another server
|
|
9
|
+
|
|
10
|
+
or with pytest: DISCO_API_URL=<api root> pytest tests/test_server_contract.py
|
|
11
|
+
|
|
12
|
+
It checks status codes and the *shape* of each response (the columns and file types the
|
|
13
|
+
toolkit reads), not the science. Large reference files are only sampled (first bytes), never
|
|
14
|
+
downloaded in full.
|
|
15
|
+
"""
|
|
16
|
+
import os
|
|
17
|
+
import sys
|
|
18
|
+
|
|
19
|
+
import requests
|
|
20
|
+
|
|
21
|
+
DEFAULT = "https://disco.bii.a-star.edu.sg/disco_v3_api/"
|
|
22
|
+
BASE = (sys.argv[1] if len(sys.argv) > 1 and sys.argv[1].startswith("http") else os.environ.get("DISCO_API_URL", DEFAULT)).rstrip("/") + "/"
|
|
23
|
+
TIMEOUT = 120
|
|
24
|
+
|
|
25
|
+
|
|
26
|
+
def url(path):
|
|
27
|
+
return BASE + path.lstrip("/")
|
|
28
|
+
|
|
29
|
+
|
|
30
|
+
def first_lines(path, n=3, **params):
|
|
31
|
+
"""First n text lines of a (possibly huge) response, without downloading the rest."""
|
|
32
|
+
with requests.get(url(path), params=params, stream=True, timeout=TIMEOUT) as r:
|
|
33
|
+
assert r.status_code == 200, "HTTP %s for %s" % (r.status_code, path)
|
|
34
|
+
# Downloads are served as application/octet-stream, which has no charset, so without
|
|
35
|
+
# this requests would hand back bytes instead of text.
|
|
36
|
+
r.encoding = "utf-8"
|
|
37
|
+
lines = []
|
|
38
|
+
for line in r.iter_lines(decode_unicode=True):
|
|
39
|
+
lines.append(line)
|
|
40
|
+
if len(lines) >= n:
|
|
41
|
+
break
|
|
42
|
+
return lines
|
|
43
|
+
|
|
44
|
+
|
|
45
|
+
def first_bytes(path, n=8, **params):
|
|
46
|
+
with requests.get(url(path), params=params, stream=True, timeout=TIMEOUT) as r:
|
|
47
|
+
assert r.status_code == 200, "HTTP %s for %s" % (r.status_code, path)
|
|
48
|
+
size = r.headers.get("Content-Length")
|
|
49
|
+
return next(r.iter_content(chunk_size=n)), (int(size) if size else None)
|
|
50
|
+
|
|
51
|
+
|
|
52
|
+
def columns(line):
|
|
53
|
+
return line.split("\t")
|
|
54
|
+
|
|
55
|
+
|
|
56
|
+
# ---- metadata the Filter / FilterData classes work on ------------------------------------
|
|
57
|
+
|
|
58
|
+
def test_sample_metadata_has_the_filter_columns():
|
|
59
|
+
header, row = first_lines("toolkit/getSampleMetadata", 2)
|
|
60
|
+
need = {"sample_id", "project_id", "sample_type", "tissue", "disease", "platform", "cell_number"}
|
|
61
|
+
missing = need - set(columns(header))
|
|
62
|
+
assert not missing, "getSampleMetadata lacks columns: %s" % sorted(missing)
|
|
63
|
+
assert len(columns(row)) == len(columns(header)), "rows are not tab-aligned with the header"
|
|
64
|
+
|
|
65
|
+
|
|
66
|
+
def test_cell_type_summary_columns():
|
|
67
|
+
header, row = first_lines("toolkit/getCellTypeSummary", 2)
|
|
68
|
+
assert columns(header) == ["sample_id", "cell_type", "cell_type_score", "cell_number"], columns(header)
|
|
69
|
+
float(columns(row)[2]); int(columns(row)[3])
|
|
70
|
+
|
|
71
|
+
|
|
72
|
+
def test_cell_ontology_is_a_list_of_cell_name_and_parent():
|
|
73
|
+
r = requests.get(url("toolkit/getCellOntology"), timeout=TIMEOUT)
|
|
74
|
+
assert r.status_code == 200
|
|
75
|
+
data = r.json()
|
|
76
|
+
assert isinstance(data, list) and len(data) > 100
|
|
77
|
+
assert {"cell_name", "parent"} <= set(data[0]), data[0]
|
|
78
|
+
|
|
79
|
+
|
|
80
|
+
# ---- per-sample download (download_disco_data) ---------------------------------------------
|
|
81
|
+
|
|
82
|
+
def _a_sample():
|
|
83
|
+
header, row = first_lines("toolkit/getSampleMetadata", 2)
|
|
84
|
+
cols = columns(header)
|
|
85
|
+
values = columns(row)
|
|
86
|
+
return values[cols.index("project_id")], values[cols.index("sample_id")]
|
|
87
|
+
|
|
88
|
+
|
|
89
|
+
def test_cell_type_labels_for_one_sample():
|
|
90
|
+
project, sample = _a_sample()
|
|
91
|
+
header, row = first_lines("toolkit/getCellTypeSample", 2, sampleId=sample)
|
|
92
|
+
cols = columns(header)
|
|
93
|
+
# the toolkit reads columns 0, 2 and 5 by position, so their order matters
|
|
94
|
+
assert cols[0] == "cell_id" and cols[2] == "cell_type" and cols[5] == "cell_type_score", cols
|
|
95
|
+
|
|
96
|
+
|
|
97
|
+
def test_raw_10x_h5_is_downloadable_and_is_hdf5():
|
|
98
|
+
project, sample = _a_sample()
|
|
99
|
+
head, size = first_bytes("download/getRawH5/%s/%s" % (project, sample))
|
|
100
|
+
assert head.startswith(b"\x89HDF"), "not an HDF5 file: %r" % head
|
|
101
|
+
assert size is None or size > 1000
|
|
102
|
+
|
|
103
|
+
|
|
104
|
+
# ---- CELLiD / scEnrichment reference data ---------------------------------------------------
|
|
105
|
+
|
|
106
|
+
def test_cellid_reference_is_a_gzip_pickle():
|
|
107
|
+
head, size = first_bytes("toolkit/getRef", type="pkl")
|
|
108
|
+
assert head[:2] == b"\x1f\x8b", "getRef is not gzip data: %r" % head
|
|
109
|
+
assert size is None or size > 1_000_000
|
|
110
|
+
|
|
111
|
+
|
|
112
|
+
def test_cellid_deg_reference_is_a_gzip_pickle():
|
|
113
|
+
head, size = first_bytes("toolkit/getRefDeg", type="pkl")
|
|
114
|
+
assert head[:2] == b"\x1f\x8b", "getRefDeg is not gzip data: %r" % head
|
|
115
|
+
|
|
116
|
+
|
|
117
|
+
def test_gene_set_reference_is_available_as_plain_csv():
|
|
118
|
+
header, row = first_lines("toolkit/getGeneSet", 2, format="csv")
|
|
119
|
+
cols = [c.strip('"') for c in header.split(",")]
|
|
120
|
+
assert cols == ["logfc", "gene", "atlas", "name"], cols
|
|
121
|
+
|
|
122
|
+
|
|
123
|
+
# ---- gene_search ------------------------------------------------------------------------------
|
|
124
|
+
|
|
125
|
+
def test_gene_reference_expression_rows_have_the_shape_gene_search_reads():
|
|
126
|
+
r = requests.get(url("geneExp/getRefExp"), params={"gene": "CD4"}, timeout=TIMEOUT)
|
|
127
|
+
assert r.status_code == 200
|
|
128
|
+
rows = r.json()
|
|
129
|
+
assert rows, "no reference expression for CD4"
|
|
130
|
+
# entry[0] ';'-separated values, entry[1] cell type, entry[3] atlas / tissue
|
|
131
|
+
assert isinstance(rows[0][0], str) and ";" in rows[0][0]
|
|
132
|
+
assert isinstance(rows[0][1], str) and isinstance(rows[0][3], str)
|
|
133
|
+
|
|
134
|
+
|
|
135
|
+
# ---- housekeeping ---------------------------------------------------------------------------------
|
|
136
|
+
|
|
137
|
+
def test_server_reports_a_toolkit_version_and_api_url():
|
|
138
|
+
v = requests.get(url("getToolkitVersion"), timeout=TIMEOUT).json()
|
|
139
|
+
assert v.get("version"), v
|
|
140
|
+
u = requests.get(url("getToolkitUrl"), timeout=TIMEOUT).json()
|
|
141
|
+
assert u.get("url", "").startswith("http"), u
|
|
142
|
+
|
|
143
|
+
|
|
144
|
+
def _run_all():
|
|
145
|
+
tests = [(n, f) for n, f in sorted(globals().items()) if n.startswith("test_") and callable(f)]
|
|
146
|
+
print("DISCO toolkit contract test against %s\n" % BASE)
|
|
147
|
+
failed = 0
|
|
148
|
+
for name, fn in tests:
|
|
149
|
+
try:
|
|
150
|
+
fn()
|
|
151
|
+
print(" PASS %s" % name)
|
|
152
|
+
except Exception as error: # report every failure, do not stop at the first
|
|
153
|
+
failed += 1
|
|
154
|
+
print(" FAIL %s\n %s: %s" % (name, type(error).__name__, error))
|
|
155
|
+
print("\n%d/%d passed" % (len(tests) - failed, len(tests)))
|
|
156
|
+
return 1 if failed else 0
|
|
157
|
+
|
|
158
|
+
|
|
159
|
+
if __name__ == "__main__":
|
|
160
|
+
sys.exit(_run_all())
|
|
@@ -0,0 +1,90 @@
|
|
|
1
|
+
"""Offline tests for choosing a DISCO server. No network, and no pandas/scanpy needed:
|
|
2
|
+
GlobalVariable is loaded straight from its file, so this runs on any Python >= 3.9.
|
|
3
|
+
|
|
4
|
+
python tests/test_settings.py or pytest tests/test_settings.py
|
|
5
|
+
"""
|
|
6
|
+
import importlib.util
|
|
7
|
+
import os
|
|
8
|
+
import sys
|
|
9
|
+
|
|
10
|
+
HERE = os.path.dirname(os.path.abspath(__file__))
|
|
11
|
+
PATH = os.path.join(HERE, "..", "discotoolkit", "GlobalVariable.py")
|
|
12
|
+
|
|
13
|
+
|
|
14
|
+
def load(env=None):
|
|
15
|
+
"""A fresh copy of GlobalVariable, as if the package had just been imported."""
|
|
16
|
+
saved = os.environ.get("DISCO_API_URL")
|
|
17
|
+
if env is None:
|
|
18
|
+
os.environ.pop("DISCO_API_URL", None)
|
|
19
|
+
else:
|
|
20
|
+
os.environ["DISCO_API_URL"] = env
|
|
21
|
+
try:
|
|
22
|
+
spec = importlib.util.spec_from_file_location("GlobalVariable_under_test", PATH)
|
|
23
|
+
module = importlib.util.module_from_spec(spec)
|
|
24
|
+
spec.loader.exec_module(module)
|
|
25
|
+
return module
|
|
26
|
+
finally:
|
|
27
|
+
if saved is None:
|
|
28
|
+
os.environ.pop("DISCO_API_URL", None)
|
|
29
|
+
else:
|
|
30
|
+
os.environ["DISCO_API_URL"] = saved
|
|
31
|
+
|
|
32
|
+
|
|
33
|
+
def test_default_server_is_disco_v1():
|
|
34
|
+
g = load()
|
|
35
|
+
assert g.get_server() == "https://disco.bii.a-star.edu.sg/disco_v3_api/"
|
|
36
|
+
|
|
37
|
+
|
|
38
|
+
def test_api_url_joins_paths_with_exactly_one_slash():
|
|
39
|
+
g = load()
|
|
40
|
+
assert g.api_url("toolkit/getSampleMetadata") == g.get_server() + "toolkit/getSampleMetadata"
|
|
41
|
+
assert g.api_url("/toolkit/getSampleMetadata") == g.get_server() + "toolkit/getSampleMetadata"
|
|
42
|
+
|
|
43
|
+
|
|
44
|
+
def test_named_presets_and_urls():
|
|
45
|
+
g = load()
|
|
46
|
+
assert g.set_server("v2") == "https://immunesinglecell.org/disco_v3_api/"
|
|
47
|
+
assert g.api_url("x") == "https://immunesinglecell.org/disco_v3_api/x" # takes effect at once
|
|
48
|
+
assert g.set_server("V1") == g.SERVERS["v1"] # preset names are case-insensitive
|
|
49
|
+
assert g.set_server("https://example.org/api") == "https://example.org/api/" # trailing slash added
|
|
50
|
+
assert g.set_server("http://localhost:8889/disco_v3_api///") == "http://localhost:8889/disco_v3_api/"
|
|
51
|
+
|
|
52
|
+
|
|
53
|
+
def test_bad_servers_are_rejected_without_changing_the_current_one():
|
|
54
|
+
g = load()
|
|
55
|
+
before = g.get_server()
|
|
56
|
+
for bad in ["", " ", "v3", "ftp://x", "disco.bii.a-star.edu.sg", None, 5]:
|
|
57
|
+
try:
|
|
58
|
+
g.set_server(bad)
|
|
59
|
+
except ValueError:
|
|
60
|
+
pass
|
|
61
|
+
else:
|
|
62
|
+
raise AssertionError("accepted %r" % (bad,))
|
|
63
|
+
assert g.get_server() == before
|
|
64
|
+
|
|
65
|
+
|
|
66
|
+
def test_environment_variable_sets_the_starting_server():
|
|
67
|
+
assert load("v2").get_server() == "https://immunesinglecell.org/disco_v3_api/"
|
|
68
|
+
assert load("https://example.org/disco_v3_api").get_server() == "https://example.org/disco_v3_api/"
|
|
69
|
+
|
|
70
|
+
|
|
71
|
+
def test_old_prefix_variable_still_readable():
|
|
72
|
+
g = load()
|
|
73
|
+
g.set_server("v2")
|
|
74
|
+
assert g.prefix_disco_url == "https://immunesinglecell.org/disco_v3_api/"
|
|
75
|
+
|
|
76
|
+
|
|
77
|
+
def _run_all():
|
|
78
|
+
tests = [(n, f) for n, f in sorted(globals().items()) if n.startswith("test_") and callable(f)]
|
|
79
|
+
failed = 0
|
|
80
|
+
for name, fn in tests:
|
|
81
|
+
try:
|
|
82
|
+
fn(); print(" PASS %s" % name)
|
|
83
|
+
except Exception as error:
|
|
84
|
+
failed += 1; print(" FAIL %s\n %s: %s" % (name, type(error).__name__, error))
|
|
85
|
+
print("\n%d/%d passed" % (len(tests) - failed, len(tests)))
|
|
86
|
+
return 1 if failed else 0
|
|
87
|
+
|
|
88
|
+
|
|
89
|
+
if __name__ == "__main__":
|
|
90
|
+
sys.exit(_run_all())
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
"""The version must be the same in setup.py and discotoolkit/__init__.py (and in the release tag).
|
|
2
|
+
|
|
3
|
+
Pure text checks, so they need no dependencies:
|
|
4
|
+
|
|
5
|
+
python tests/test_version.py # setup.py == __init__.py
|
|
6
|
+
python tests/test_version.py v1.2.0 # ... and equals the tag
|
|
7
|
+
"""
|
|
8
|
+
import os
|
|
9
|
+
import re
|
|
10
|
+
import sys
|
|
11
|
+
|
|
12
|
+
ROOT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..")
|
|
13
|
+
|
|
14
|
+
|
|
15
|
+
def find(path, pattern):
|
|
16
|
+
with open(os.path.join(ROOT, path)) as handle:
|
|
17
|
+
match = re.search(pattern, handle.read())
|
|
18
|
+
assert match, "no version found in %s" % path
|
|
19
|
+
return match.group(1)
|
|
20
|
+
|
|
21
|
+
|
|
22
|
+
def test_setup_and_init_agree():
|
|
23
|
+
setup_version = find("setup.py", r'version\s*=\s*"([^"]+)"')
|
|
24
|
+
init_version = find("discotoolkit/__init__.py", r'__version__\s*=\s*"([^"]+)"')
|
|
25
|
+
assert setup_version == init_version, "setup.py says %s, __init__.py says %s" % (setup_version, init_version)
|
|
26
|
+
|
|
27
|
+
|
|
28
|
+
def main():
|
|
29
|
+
test_setup_and_init_agree()
|
|
30
|
+
version = find("setup.py", r'version\s*=\s*"([^"]+)"')
|
|
31
|
+
if len(sys.argv) > 1:
|
|
32
|
+
tag = sys.argv[1]
|
|
33
|
+
assert tag == "v" + version, "tag %s does not match package version %s (expected v%s)" % (tag, version, version)
|
|
34
|
+
print("version %s OK" % version)
|
|
35
|
+
|
|
36
|
+
|
|
37
|
+
if __name__ == "__main__":
|
|
38
|
+
try:
|
|
39
|
+
main()
|
|
40
|
+
except AssertionError as error:
|
|
41
|
+
sys.exit("FAIL: %s" % error)
|
discotoolkit-1.1.4/MANIFEST.in
DELETED
|
@@ -1 +0,0 @@
|
|
|
1
|
-
include README.md
|
discotoolkit-1.1.4/PKG-INFO
DELETED
|
@@ -1,13 +0,0 @@
|
|
|
1
|
-
Metadata-Version: 2.1
|
|
2
|
-
Name: discotoolkit
|
|
3
|
-
Version: 1.1.4
|
|
4
|
-
Summary: DISCOtoolkit is an python package that allows users to access data and use the tools provided by the DISCO database.
|
|
5
|
-
Home-page: http://www.immunesinglecell.org/
|
|
6
|
-
Author: Li Mengwei, Rom Uddamvathanak
|
|
7
|
-
Author-email: uddamvathanak_rom@immunol.a-star.edu.sg
|
|
8
|
-
Requires-Python: >=3.8.16
|
|
9
|
-
|
|
10
|
-
DISCOtoolkit is an python package that allows users to access data and use the tools provided by the DISCO database. It provides the following functions
|
|
11
|
-
Filter and download DISCO data based on sample metadata and cell type information
|
|
12
|
-
CELLiD: cell type annotation
|
|
13
|
-
scEnrichment: geneset enrichment using DISCO DEGs
|
discotoolkit-1.1.4/README.md
DELETED
|
@@ -1,72 +0,0 @@
|
|
|
1
|
-
<!--
|
|
2
|
-
* @Descripttion:
|
|
3
|
-
* @version:
|
|
4
|
-
* @Author: Mengwei Li
|
|
5
|
-
* @Date: 2023-04-16 21:20:42
|
|
6
|
-
* @LastEditors: Mengwei Li
|
|
7
|
-
* @LastEditTime: 2023-04-16 21:22:03
|
|
8
|
-
-->
|
|
9
|
-
[](https://discotoolkit-py.readthedocs.io/en/latest/?badge=latest) [](https://pepy.tech/project/discotoolkit) [](https://pypi.org/project/discotoolkit)
|
|
10
|
-
|
|
11
|
-
# DISCOtoolkit 1.1.4
|
|
12
|
-
|
|
13
|
-
DISCOtoolkit is an python package that allows users to access data and use the tools provided by the [DISCO database](https://www.immunesinglecell.org/). Read the documentation [DISCOtoolkit](https://discotoolkit-py.readthedocs.io/en/latest/). It provides the following functions:
|
|
14
|
-
|
|
15
|
-
- Filter and download DISCO data based on sample metadata and cell type information
|
|
16
|
-
- CELLiD: cell type annotation
|
|
17
|
-
- scEnrichment: geneset enrichment using DISCO DEGs
|
|
18
|
-
|
|
19
|
-
Dependency Requirements:
|
|
20
|
-
|
|
21
|
-
- Numpy >= 1.21.6
|
|
22
|
-
- Pandas >= 1.4.2
|
|
23
|
-
- Scanpy >= 1.9.3
|
|
24
|
-
- Scipy >= 1.8.0
|
|
25
|
-
- joblib >= 1.1.0
|
|
26
|
-
- pandarallel >= 1.6.5
|
|
27
|
-
|
|
28
|
-
## Minimal installation:
|
|
29
|
-
|
|
30
|
-
The DISCOtoolkit can be easily installed in the current Python environment using `pip`:
|
|
31
|
-
|
|
32
|
-
```
|
|
33
|
-
pip install discotoolkit
|
|
34
|
-
```
|
|
35
|
-
|
|
36
|
-
## Installation guide:
|
|
37
|
-
|
|
38
|
-
we recommend to install miniconda first and install discotoolkit in virtual env
|
|
39
|
-
|
|
40
|
-
```
|
|
41
|
-
conda create --name disco python=3.8
|
|
42
|
-
```
|
|
43
|
-
```
|
|
44
|
-
conda activate disco
|
|
45
|
-
```
|
|
46
|
-
```
|
|
47
|
-
conda install ipykernel
|
|
48
|
-
```
|
|
49
|
-
```
|
|
50
|
-
python -m ipykernel install --user --name disco --display-name "disco"
|
|
51
|
-
```
|
|
52
|
-
```
|
|
53
|
-
python -m pip install discotoolkit # adding -U for installing the latest version
|
|
54
|
-
```
|
|
55
|
-
|
|
56
|
-
## Basic Usage
|
|
57
|
-
Example in Jupyter notebook.
|
|
58
|
-
|
|
59
|
-
<em>please select disco as the kernel for running the jupyter notebook</em>
|
|
60
|
-
|
|
61
|
-
### [Filter and download DISCO data](https://github.com/JinmiaoChenLab/DISCOtoolkit_py/blob/main/docs/download_data.ipynb)
|
|
62
|
-
|
|
63
|
-
### [Cell Type Annotation using CELLiD](https://github.com/JinmiaoChenLab/DISCOtoolkit_py/blob/main/docs/CELLiD_celltype_annotation.ipynb)
|
|
64
|
-
|
|
65
|
-
### [scEnrichment](https://github.com/JinmiaoChenLab/DISCOtoolkit_py/blob/main/docs/scEnrichment.ipynb)
|
|
66
|
-
|
|
67
|
-
## Citation
|
|
68
|
-
1. [Li, Mengwei, et al. "DISCO: a database of Deeply Integrated human Single-Cell Omics data." Nucleic acids research 50.D1 (2022): D596-D602.](https://academic.oup.com/nar/article/50/D1/D596/6430491)
|
|
69
|
-
|
|
70
|
-
## Follow us on our social media!
|
|
71
|
-
- [HSCRM2](https://twitter.com/HSCRM2)
|
|
72
|
-
- [JinmiaoChenLab Github repo](https://github.com/JinmiaoChenLab)
|
|
@@ -1,33 +0,0 @@
|
|
|
1
|
-
'''
|
|
2
|
-
Descripttion:
|
|
3
|
-
version:
|
|
4
|
-
Author: Mengwei Li
|
|
5
|
-
Date: 2023-07-06 16:59:59
|
|
6
|
-
LastEditors: Mengwei Li
|
|
7
|
-
LastEditTime: 2024-10-23 15:52:29
|
|
8
|
-
'''
|
|
9
|
-
"""
|
|
10
|
-
Global variable file to import for the subsequent script.
|
|
11
|
-
|
|
12
|
-
Main purpose is for the logging to be imported to other script and getting the url
|
|
13
|
-
"""
|
|
14
|
-
|
|
15
|
-
import requests
|
|
16
|
-
import json
|
|
17
|
-
import logging
|
|
18
|
-
# import random
|
|
19
|
-
|
|
20
|
-
# setting the logging level
|
|
21
|
-
# logger = logging.StreamHandler()
|
|
22
|
-
# logger.setLevel(0)
|
|
23
|
-
logging.basicConfig(level = logging.INFO)
|
|
24
|
-
|
|
25
|
-
timeout = 600
|
|
26
|
-
|
|
27
|
-
# Define package-level variable
|
|
28
|
-
# response = requests.get("http://www.immunesinglecell.org/api/vishuo/getToolkitUrl")
|
|
29
|
-
|
|
30
|
-
# if response.status_code == 200:
|
|
31
|
-
# prefix_disco_url = json.loads(response.text)["url"]
|
|
32
|
-
# else:
|
|
33
|
-
prefix_disco_url = "https://immunesinglecell.org/disco_v3_api/"
|
|
@@ -1,13 +0,0 @@
|
|
|
1
|
-
Metadata-Version: 2.1
|
|
2
|
-
Name: discotoolkit
|
|
3
|
-
Version: 1.1.4
|
|
4
|
-
Summary: DISCOtoolkit is an python package that allows users to access data and use the tools provided by the DISCO database.
|
|
5
|
-
Home-page: http://www.immunesinglecell.org/
|
|
6
|
-
Author: Li Mengwei, Rom Uddamvathanak
|
|
7
|
-
Author-email: uddamvathanak_rom@immunol.a-star.edu.sg
|
|
8
|
-
Requires-Python: >=3.8.16
|
|
9
|
-
|
|
10
|
-
DISCOtoolkit is an python package that allows users to access data and use the tools provided by the DISCO database. It provides the following functions
|
|
11
|
-
Filter and download DISCO data based on sample metadata and cell type information
|
|
12
|
-
CELLiD: cell type annotation
|
|
13
|
-
scEnrichment: geneset enrichment using DISCO DEGs
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|