PyPI - discotoolkit - Versions diffs - 1.2.0__tar.gz → 1.2.2__tar.gz - Mend

discotoolkit 1.2.0tar.gz → 1.2.2tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (27) hide show

{discotoolkit-1.2.0 → discotoolkit-1.2.2}/PKG-INFO RENAMED Viewed

@@ -1,10 +1,14 @@
 Metadata-Version: 2.4
 Name: discotoolkit
-Version: 1.2.0
+Version: 1.2.2
 Summary: DISCOtoolkit is an python package that allows users to access data and use the tools provided by the DISCO database.
 Home-page: https://disco.bii.a-star.edu.sg/
 Author: Li Mengwei, Rom Uddamvathanak
 Author-email: uddamvathanak_rom@immunol.a-star.edu.sg
+Project-URL: Documentation, https://disco.bii.a-star.edu.sg/v1/docs/toolkit/guide/
+Project-URL: Source, https://github.com/JinmiaoChenLab/DISCOtoolkit_py
+Project-URL: Issues, https://github.com/JinmiaoChenLab/DISCOtoolkit_py/issues
+Project-URL: DISCO, https://disco.bii.a-star.edu.sg/v1/
 Requires-Python: >=3.9
 Description-Content-Type: text/markdown
 Requires-Dist: pandas>=1.4.2
@@ -24,6 +28,7 @@ Dynamic: author-email
 Dynamic: description
 Dynamic: description-content-type
 Dynamic: home-page
+Dynamic: project-url
 Dynamic: requires-dist
 Dynamic: requires-python
 Dynamic: summary
@@ -36,18 +41,18 @@ Dynamic: summary
  * @LastEditors: Mengwei Li
  * @LastEditTime: 2023-04-16 21:22:03
 -->
-[![Documentation Status](https://readthedocs.org/projects/discotoolkit-py/badge/?version=latest)](https://discotoolkit-py.readthedocs.io/en/latest/?badge=latest) [![Downloads](https://static.pepy.tech/personalized-badge/discotoolkit?period=total&units=international_system&left_color=black&right_color=orange&left_text=Downloads)](https://pepy.tech/project/discotoolkit) [![PyPI version](https://img.shields.io/pypi/v/discotoolkit)](https://pypi.org/project/discotoolkit)
+[![Documentation](https://img.shields.io/badge/docs-disco.bii.a--star.edu.sg-blue)](https://disco.bii.a-star.edu.sg/v1/docs/toolkit/guide/) [![Downloads](https://static.pepy.tech/personalized-badge/discotoolkit?period=total&units=international_system&left_color=black&right_color=orange&left_text=Downloads)](https://pepy.tech/project/discotoolkit) [![PyPI version](https://img.shields.io/pypi/v/discotoolkit)](https://pypi.org/project/discotoolkit)
-# DISCOtoolkit 1.2.0
+# DISCOtoolkit 1.2.2
-DISCOtoolkit is a Python package for accessing the data and tools of the [DISCO database](https://disco.bii.a-star.edu.sg/) (DISCO v1). Read the documentation at [discotoolkit-py.readthedocs.io](https://discotoolkit-py.readthedocs.io/en/latest/).
+DISCOtoolkit is a Python package for accessing the data and tools of the [DISCO database](https://disco.bii.a-star.edu.sg/) (DISCO v1). Read the documentation, with tutorials and the API reference, at [disco.bii.a-star.edu.sg/v1/docs/toolkit/guide](https://disco.bii.a-star.edu.sg/v1/docs/toolkit/guide/).
 - Filter and download DISCO data based on sample metadata and cell type information
 - Gene search: a gene's expression across cell types and tissues
 - CELLiD: cell type annotation
 - scEnrichment: gene set enrichment using DISCO DEGs
-> **DISCO v1 and v2.** The toolkit talks to **DISCO v1** by default. DISCO v2 has its own R package ([DISCOtoolkit](https://github.com/JinmiaoChenLab/DISCOtoolkit)); the Python package can still be pointed at a v2 server, see [Choosing a server](#choosing-a-server).
+> **Use DISCO v1.** The toolkit is developed and tested against **DISCO v1**, which is the default, and that is the server we recommend. DISCO v2 has its own R package ([DISCOtoolkit](https://github.com/JinmiaoChenLab/DISCOtoolkit)). The Python package can be pointed at a v2 server, but that is not tested; see [Choosing a server](#choosing-a-server).
 ## Quickstart (Google Colab)
@@ -97,7 +102,7 @@ python -m pip install -U discotoolkit
 import discotoolkit as dt
 dt.get_server()                       # 'https://disco.bii.a-star.edu.sg/disco_v3_api/'  (DISCO v1, the default)
-dt.set_server("v2")                   # DISCO v2
+dt.set_server("v2")                   # DISCO v2 (not tested; DISCO v1 is the supported server)
 dt.set_server("https://my.server/disco_v3_api/")   # any server with the same API
 ```
@@ -110,11 +115,12 @@ Two small test files need nothing beyond `requests`, so they run anywhere, inclu
 ```
 python tests/test_settings.py           # offline: server selection logic
 python tests/test_version.py            # offline: setup.py and __init__.py agree on the version
+python tests/test_docs.py               # offline: notebooks are clean, every public function is in the API reference
 python tests/test_server_contract.py    # online: does the server answer everything the toolkit needs?
-python tests/test_server_contract.py https://immunesinglecell.org/disco_v3_api/   # check another server
+python tests/test_server_contract.py <api root>   # check another server
 ```
-GitHub Actions runs these, and installs the package on Python 3.9-3.12, on every push. Maintainers: see [RELEASING.md](RELEASING.md) for how a release is published.
+GitHub Actions runs these, and installs the package on Python 3.9-3.12, on every push. Maintainers: see [RELEASING.md](RELEASING.md) for how a release, and its documentation, is published.
 `test_server_contract.py` checks status codes and the shape of each response (the columns and file types the toolkit reads). It only samples the large reference files, it does not download them.

{discotoolkit-1.2.0 → discotoolkit-1.2.2}/README.md RENAMED Viewed

@@ -6,18 +6,18 @@
  * @LastEditors: Mengwei Li
  * @LastEditTime: 2023-04-16 21:22:03
 -->
-[![Documentation Status](https://readthedocs.org/projects/discotoolkit-py/badge/?version=latest)](https://discotoolkit-py.readthedocs.io/en/latest/?badge=latest) [![Downloads](https://static.pepy.tech/personalized-badge/discotoolkit?period=total&units=international_system&left_color=black&right_color=orange&left_text=Downloads)](https://pepy.tech/project/discotoolkit) [![PyPI version](https://img.shields.io/pypi/v/discotoolkit)](https://pypi.org/project/discotoolkit)
+[![Documentation](https://img.shields.io/badge/docs-disco.bii.a--star.edu.sg-blue)](https://disco.bii.a-star.edu.sg/v1/docs/toolkit/guide/) [![Downloads](https://static.pepy.tech/personalized-badge/discotoolkit?period=total&units=international_system&left_color=black&right_color=orange&left_text=Downloads)](https://pepy.tech/project/discotoolkit) [![PyPI version](https://img.shields.io/pypi/v/discotoolkit)](https://pypi.org/project/discotoolkit)
-# DISCOtoolkit 1.2.0
+# DISCOtoolkit 1.2.2
-DISCOtoolkit is a Python package for accessing the data and tools of the [DISCO database](https://disco.bii.a-star.edu.sg/) (DISCO v1). Read the documentation at [discotoolkit-py.readthedocs.io](https://discotoolkit-py.readthedocs.io/en/latest/).
+DISCOtoolkit is a Python package for accessing the data and tools of the [DISCO database](https://disco.bii.a-star.edu.sg/) (DISCO v1). Read the documentation, with tutorials and the API reference, at [disco.bii.a-star.edu.sg/v1/docs/toolkit/guide](https://disco.bii.a-star.edu.sg/v1/docs/toolkit/guide/).
 - Filter and download DISCO data based on sample metadata and cell type information
 - Gene search: a gene's expression across cell types and tissues
 - CELLiD: cell type annotation
 - scEnrichment: gene set enrichment using DISCO DEGs
-> **DISCO v1 and v2.** The toolkit talks to **DISCO v1** by default. DISCO v2 has its own R package ([DISCOtoolkit](https://github.com/JinmiaoChenLab/DISCOtoolkit)); the Python package can still be pointed at a v2 server, see [Choosing a server](#choosing-a-server).
+> **Use DISCO v1.** The toolkit is developed and tested against **DISCO v1**, which is the default, and that is the server we recommend. DISCO v2 has its own R package ([DISCOtoolkit](https://github.com/JinmiaoChenLab/DISCOtoolkit)). The Python package can be pointed at a v2 server, but that is not tested; see [Choosing a server](#choosing-a-server).
 ## Quickstart (Google Colab)
@@ -67,7 +67,7 @@ python -m pip install -U discotoolkit
 import discotoolkit as dt
 dt.get_server()                       # 'https://disco.bii.a-star.edu.sg/disco_v3_api/'  (DISCO v1, the default)
-dt.set_server("v2")                   # DISCO v2
+dt.set_server("v2")                   # DISCO v2 (not tested; DISCO v1 is the supported server)
 dt.set_server("https://my.server/disco_v3_api/")   # any server with the same API
 ```
@@ -80,11 +80,12 @@ Two small test files need nothing beyond `requests`, so they run anywhere, inclu
 ```
 python tests/test_settings.py           # offline: server selection logic
 python tests/test_version.py            # offline: setup.py and __init__.py agree on the version
+python tests/test_docs.py               # offline: notebooks are clean, every public function is in the API reference
 python tests/test_server_contract.py    # online: does the server answer everything the toolkit needs?
-python tests/test_server_contract.py https://immunesinglecell.org/disco_v3_api/   # check another server
+python tests/test_server_contract.py <api root>   # check another server
 ```
-GitHub Actions runs these, and installs the package on Python 3.9-3.12, on every push. Maintainers: see [RELEASING.md](RELEASING.md) for how a release is published.
+GitHub Actions runs these, and installs the package on Python 3.9-3.12, on every push. Maintainers: see [RELEASING.md](RELEASING.md) for how a release, and its documentation, is published.
 `test_server_contract.py` checks status codes and the shape of each response (the columns and file types the toolkit reads). It only samples the large reference files, it does not download them.

{discotoolkit-1.2.0 → discotoolkit-1.2.2}/discotoolkit/CELLiD.py RENAMED Viewed

@@ -56,11 +56,11 @@ def generate_correlation_map(x, y):
 # get the total atlas from DISCO website for the user to select which atlas to use
-def get_atlas(ref_data=None, ref_path=None):
+def get_atlas(ref_data=None, ref_path=None) -> list:
     """get the all the atlas string from the DISCO website and return to the user
     Returns:
-        List: return list of string
+        list: the atlas names, as a list of strings
     """
     # Download reference data and ref_deg if missing
     if ref_data is None:
@@ -72,23 +72,56 @@ def get_atlas(ref_data=None, ref_path=None):
         if not os.path.exists(ref_path):
             os.mkdir(ref_path)
-        # download reference data from the server in pickle extension and read as pandas dataframe
+        # download the reference data from the server (once) and read it as a pandas dataframe
         if ref_data is None:
-            if not (os.path.exists(ref_path + "/ref_data.pkl")):
-                logging.info("Downloading reference dataset...")
-                response = requests.get(
-                    url=api_url("toolkit/getRef?type=pkl"), timeout=timeout
-                )
-                open(ref_path + "/ref_data.pkl", "wb").write(response.content)
-            ref_data = pd.read_pickle(
-                ref_path + "/ref_data.pkl",
-                compression={"method": "gzip", "compresslevel": 6},
-            )
+            ref_data = _load_reference(ref_path, "ref_data", "getRef", index_col=0)
     all_atlas = [each.split("--")[1] for each in ref_data.columns]
     return list(set(all_atlas))
+def _load_reference(ref_path: str, name: str, endpoint: str, **csv_options) -> pd.DataFrame:
+    """Get a CELLiD reference table (`name` is "ref_data" or "ref_deg"), downloading it once.
+    DISCO v1 serves it as plain CSV (`toolkit/<endpoint>?type=csv`, gzip-compressed on the wire),
+    read with pandas.read_csv. A copy already in `ref_path` is used as it is (a CSV, or the
+    pickle that earlier versions saved). Older servers only offer a pickle; that is the fallback,
+    with a warning, because unpickling data from a network is only safe if the server is trusted.
+    """
+    csv_file = "%s/%s.csv" % (ref_path, name)
+    pkl_file = "%s/%s.pkl" % (ref_path, name)
+    if os.path.exists(csv_file):
+        return pd.read_csv(csv_file, **csv_options)
+    if os.path.exists(pkl_file):
+        return pd.read_pickle(pkl_file, compression={"method": "gzip", "compresslevel": 6})
+    logging.info("Downloading reference dataset (%s)...", name)
+    with requests.get(
+        url=api_url("toolkit/" + endpoint), params={"type": "csv"}, stream=True, timeout=timeout
+    ) as response:
+        if response.status_code == 200:
+            # write to a temporary name so an interrupted download is not mistaken for the file
+            with open(csv_file + ".part", "wb") as handle:
+                for chunk in response.iter_content(chunk_size=1 << 20):
+                    handle.write(chunk)
+            os.replace(csv_file + ".part", csv_file)
+            return pd.read_csv(csv_file, **csv_options)
+    logging.warning(
+        "This server has no CSV for %s (HTTP %s); falling back to the pickle download.",
+        name,
+        response.status_code,
+    )
+    response = requests.get(
+        url=api_url("toolkit/" + endpoint), params={"type": "pkl"}, timeout=timeout
+    )
+    response.raise_for_status()
+    with open(pkl_file, "wb") as handle:
+        handle.write(response.content)
+    return pd.read_pickle(pkl_file, compression={"method": "gzip", "compresslevel": 6})
 def _load_geneset(ref_path: str) -> pd.DataFrame:
     """Get the gene-set reference (columns logfc, gene, atlas, name), downloading it once.
@@ -133,7 +166,7 @@ def CELLiD_cluster(
     n_predict: int = 1,
     ref_path: str = None,
     ncores: int = 10,
-):
+) -> pd.DataFrame:
     """Cell type annotation using reference data and compute the correlation between the user cell gene expression as compare
         to the reference data. The celltype with highest correlation will be concluded as the celltype
@@ -175,31 +208,13 @@ def CELLiD_cluster(
         if not os.path.exists(ref_path):
             os.mkdir(ref_path)
-        # download reference data from the server in pickle extension and read as pandas dataframe
+        # download the reference data from the server (once) and read it as a pandas dataframe
         if ref_data is None:
-            if not (os.path.exists(ref_path + "/ref_data.pkl")):
-                logging.info("Downloading reference dataset...")
-                response = requests.get(
-                    url=api_url("toolkit/getRef?type=pkl"), timeout=timeout
-                )
-                open(ref_path + "/ref_data.pkl", "wb").write(response.content)
-            ref_data = pd.read_pickle(
-                ref_path + "/ref_data.pkl",
-                compression={"method": "gzip", "compresslevel": 6},
-            )
+            ref_data = _load_reference(ref_path, "ref_data", "getRef", index_col=0)
         # similarly do the same for the DEG reference
         if ref_deg is None:
-            if not (os.path.exists(ref_path + "/ref_deg.pkl")):
-                logging.info("Downloading deg dataset...")
-                response = requests.get(
-                    url=api_url("toolkit/getRefDeg?type=pkl"), timeout=timeout
-                )
-                open(ref_path + "/ref_deg.pkl", "wb").write(response.content)
-            ref_deg = pd.read_pickle(
-                ref_path + "/ref_deg.pkl",
-                compression={"method": "gzip", "compresslevel": 6},
-            )
+            ref_deg = _load_reference(ref_path, "ref_deg", "getRefDeg")
     ####
     # write list of atlas function
@@ -290,7 +305,7 @@ def CELLiD_cluster(
         predict.columns = ["cor"]
         predict = predict.sort_values(["cor"], ascending=False)
         res = [
-            [each.split("--")[0], each.split("--")[1], predict["cor"][index], i]
+            [each.split("--")[0], each.split("--")[1], predict["cor"].iloc[index], i]
             for index, each in enumerate(predict.index)
             if index < n_predict
         ]
@@ -341,7 +356,7 @@ def CELLiD_enrichment(
     reference: pd.DataFrame = None,
     ref_path: str = None,
     ncores: int = 1,
-):
+) -> pd.DataFrame:
     """Function to generate enrichment analysis based on the reference gene sets and following the DISCO pipeline.
     Args:

{discotoolkit-1.2.0 → discotoolkit-1.2.2}/discotoolkit/DiscoClass.py RENAMED Viewed

@@ -3,21 +3,25 @@ Class for filtering dataset
 """
 class Filter:
-    """
-    Filter class object to save the attributes for filtering the dataset from DISCO
-    sample_id                      String    e.g. ERX2757110;
-    project_id                     String;
-    tissue                      String    e.g. lung, bladder;
-    disease                     String    e.g. COVID-19;
-    platform                    String    e.g. 10x3';
-    sample_type                 String;
-    cell_type                   String;
-    cell_type_confidence        String    e.g. high;
-    include_cell_type_children  Bool      e.g. True;
-    min_cell_per_sample         Int       e.g. 300;
-    return Class object
+    """Which DISCO samples and cells to look for.
+    Every argument is optional; a filter with none of them matches everything. Give a string or a
+    list of strings. The list of values each field accepts is `list_metadata_item(field)`.
+    Args:
+        sample_id (str or list, optional): sample identifier, e.g. "ERX2757110".
+        project_id (str or list, optional): project (a study or dataset), e.g. "GSE147520".
+        tissue (str or list, optional): e.g. "lung", "bladder".
+        disease (str or list, optional): e.g. "COVID-19".
+        platform (str or list, optional): sequencing platform, e.g. "10x3'".
+        sample_type (str or list, optional): e.g. "control".
+        cell_type (str or list, optional): keep samples that contain these cell types, and download
+            only those cells.
+        cell_type_confidence (str): how sure DISCO's annotation must be: "high", "medium" or "all".
+            Defaults to "medium".
+        include_cell_type_children (bool): also match the more specific cell types under
+            `cell_type`. Defaults to True.
+        min_cell_per_sample (int): drop samples with fewer matching cells than this. Defaults to 100.
     """
     def __init__(self, sample_id = None, project_id = None, tissue = None, disease = None, platform = None, sample_type = None,
@@ -43,8 +47,16 @@ class Filter:
 class FilterData:
-    """
-    Wrapper class on top of the Filter object to get dataset's summary such as cell count and sample count and the metadata
+    """The result of a filter: the matching samples, with their counts.
+    Returned by `filter_disco_metadata` and passed to `download_disco_data`.
+    Attributes:
+        sample_metadata (pandas.DataFrame): one row per matching sample.
+        cell_type_metadata (pandas.DataFrame): the cell types found in each sample.
+        sample_count (int): number of matching samples.
+        cell_count (int): number of matching cells.
+        filter (Filter): the filter that produced this result.
     """
     def __init__(self, sample_metadata = None, cell_type_metadata = None, sample_count = None, cell_count = None, filter = Filter()):
         self.sample_metadata = sample_metadata

{discotoolkit-1.2.0 → discotoolkit-1.2.2}/discotoolkit/DownloadDiscoData.py RENAMED Viewed

@@ -22,14 +22,18 @@ from .DiscoClass import FilterData, Filter
 from .GetMetadata import check_in_list
-def download_disco_data(metadata, output_dir: str = "DISCOtmp"):
-    """function to download the data based on the given filter
+def download_disco_data(metadata, output_dir: str = "DISCOtmp") -> None:
+    """Download the samples of a filter result, one AnnData `.h5ad` file per sample.
+    Each file holds only the cells that matched the filter, with DISCO's annotation in
+    `obs["cell_type"]`.
     Args:
-        metadata (FilterData) : FilterData class to filters data from DISCO database
-        output_dir (Sting) : directory for storing the downloaded data. Default DISCOtmp
+        metadata (FilterData): the result of `filter_disco_metadata`.
+        output_dir (str): directory to save the files in; created if missing. Defaults to "DISCOtmp".
     Returns:
-        None: This function does not return any object and instead download the data for the user
+        None: the files are written to `output_dir`.
     """
     # define a list to store the error sample that can not be download

{discotoolkit-1.2.0 → discotoolkit-1.2.2}/discotoolkit/GeneSearch.py RENAMED Viewed

@@ -10,7 +10,7 @@ import colorcet as cc
 from .GlobalVariable import api_url
-def gene_search(gene : str, atlas : Union[str, list] = None, figsize : tuple = None, dpi : int = 300):
+def gene_search(gene : str, atlas : Union[str, list] = None, figsize : tuple = None, dpi : int = 300) -> None:
     """Function to search for the gene expression level the same as the input gene search bar in DISCO website.
@@ -58,8 +58,8 @@ def gene_search(gene : str, atlas : Union[str, list] = None, figsize : tuple = N
     if atlas is not None:
         sample_df = sample_df[sample_df["atlases"].isin(atlas)].copy()
-    # Define the colorcet paletteasdasd
-    custom_palette = cc.glasbey_hv
+    # Define the colorcet palette, one colour per atlas shown
+    custom_palette = cc.glasbey_hv[: max(sample_df["atlases"].nunique(), 1)]
     # saturation for the plot color
     saturation=0.8
@@ -80,7 +80,7 @@ def gene_search(gene : str, atlas : Union[str, list] = None, figsize : tuple = N
     fig = plt.figure(figsize=figsize)
     # Create the violin plot with different colors based on the 'Category' variable and adding boxplot to assist user with median value
-    ax = sns.violinplot(data=sample_df, x='cell types', y='value', hue="atlases", width=0.8, dodge=False, cut=1, saturation = saturation,  plot_kws={'alpha':0.1},
+    ax = sns.violinplot(data=sample_df, x='cell types', y='value', hue="atlases", width=0.8, dodge=False, cut=1, saturation = saturation,
                         inner=None, linewidth=0.0, palette=custom_palette)
     ax_2 = sns.boxplot(x='cell types', y='value', hue="atlases", data=sample_df, palette=custom_palette, dodge=False, width=0.5, saturation = saturation,
@@ -113,7 +113,7 @@ def gene_search(gene : str, atlas : Union[str, list] = None, figsize : tuple = N
     # Set labels and title
     plt.xlabel('Cell Types')
     plt.ylabel('Expression Value')
-    plt.title(gene, fontweight='bold')
+    plt.title(gene, fontweight='bold', loc='left')  # left: the legend sits above the centre
     # Show the plot
     plt.show()

{discotoolkit-1.2.0 → discotoolkit-1.2.2}/discotoolkit/GetMetadata.py RENAMED Viewed

@@ -55,15 +55,15 @@ def get_sample_ct_info():
     return temp  # return Dataframe of the JSON data
-def find_celltype(term: str = "", cell_ontology: dict = None):
+def find_celltype(term: str = "", cell_ontology: dict = None) -> list:
     """find the celltype within the disco dataset
     Args:
         term (String): term refer to string of the cell type
-        cell_ontology (Dict) = cell_ontology can be provided by the user in the format of dictionary datatype in python. Defaults to None.
+        cell_ontology (dict, optional): cell_ontology can be provided by the user in the format of dictionary datatype in python. Defaults to None.
     Returns:
-        Celltype List: List of matched celltype
+        list: the cell types that match the term
     """
     # when the user do not have cell ontology, we get the default one from disco database
@@ -113,7 +113,7 @@ def get_disco_metadata():
     return metadata
-def get_celltype_children(cell_type: Union[str, list], cell_ontology: dict = None):
+def get_celltype_children(cell_type: Union[str, list], cell_ontology: dict = None) -> list:
     """get the children of the input celltype from the user
     Args:
@@ -121,7 +121,7 @@ def get_celltype_children(cell_type: Union[str, list], cell_ontology: dict = Non
         cell_ontology (dict, optional): cell_ontology can be provided by the user in the format of dictionary datatype in python. Defaults to None.
     Returns:
-        List of String: return the children of the defined celltype in list of String
+        list: the children of the defined cell type, as a list of strings
     """
     if cell_ontology is None:
         cell_ontology = pd.DataFrame(
@@ -159,14 +159,14 @@ def get_celltype_children(cell_type: Union[str, list], cell_ontology: dict = Non
     return children
-def list_metadata_item(field: str):
+def list_metadata_item(field: str) -> list:
     """List element inside the metadata columns
     Args:
         field (str): metadata columns or field from the disco database
     Returns:
-        List: return the list of unique element in the metadata columns to the users as reference
+        list: the unique elements of the metadata column, as a reference for filtering
     """
     # first get the disco metadata
@@ -182,11 +182,11 @@ def list_metadata_item(field: str):
         return None
-def list_all_columns():
+def list_all_columns() -> list:
     """list all the columns found in the metadata of the disco database
     Returns:
-        List: return the name of the metadata in the form of list of string
+        list: the names of the metadata columns, as a list of strings
     """
     # get the disco metadata
@@ -194,14 +194,14 @@ def list_all_columns():
     return list(metadata.columns)
-def filter_disco_metadata(filter: Filter = Filter()):
-    """filter function option for the disco data
+def filter_disco_metadata(filter: Filter = Filter()) -> FilterData:
+    """Find the DISCO samples and cells that match a filter.
     Args:
-        Filter (Class): predefined Filter class with default attribute to filter data for the user
-        FilterData.filter.cell_type_confidence (String): requires string to be in ["high", "medium", "all"]
+        filter (Filter): what to look for. With no argument, everything matches.
     Returns:
-        FilterData (Class): return the FilterData object which will then be used to filter and download data
+        FilterData: the matching samples and counts, to pass to `download_disco_data`.
     """
     # starting with defining variable

{discotoolkit-1.2.0 → discotoolkit-1.2.2}/discotoolkit/GlobalVariable.py RENAMED Viewed

@@ -2,8 +2,9 @@
 Settings shared by the rest of the package: logging, the request timeout and --
 most importantly -- which DISCO server the toolkit talks to.
-By default the toolkit talks to DISCO v1 (https://disco.bii.a-star.edu.sg). To use
-another server:
+By default the toolkit talks to DISCO v1 (https://disco.bii.a-star.edu.sg). DISCO v1 is the
+server the toolkit is developed and tested against, and the one we recommend. Other servers
+are possible but untested (the "v2" preset in particular):
     import discotoolkit as dt
     dt.set_server("v2")                          # a named preset

{discotoolkit-1.2.0 → discotoolkit-1.2.2}/discotoolkit/__init__.py RENAMED Viewed

@@ -5,4 +5,4 @@ from .CELLiD import *
 from .GeneSearch import *
 from .GlobalVariable import set_server, get_server
-__version__ = "1.2.0"
+__version__ = "1.2.2"

{discotoolkit-1.2.0 → discotoolkit-1.2.2}/discotoolkit.egg-info/PKG-INFO RENAMED Viewed

@@ -1,10 +1,14 @@
 Metadata-Version: 2.4
 Name: discotoolkit
-Version: 1.2.0
+Version: 1.2.2
 Summary: DISCOtoolkit is an python package that allows users to access data and use the tools provided by the DISCO database.
 Home-page: https://disco.bii.a-star.edu.sg/
 Author: Li Mengwei, Rom Uddamvathanak
 Author-email: uddamvathanak_rom@immunol.a-star.edu.sg
+Project-URL: Documentation, https://disco.bii.a-star.edu.sg/v1/docs/toolkit/guide/
+Project-URL: Source, https://github.com/JinmiaoChenLab/DISCOtoolkit_py
+Project-URL: Issues, https://github.com/JinmiaoChenLab/DISCOtoolkit_py/issues
+Project-URL: DISCO, https://disco.bii.a-star.edu.sg/v1/
 Requires-Python: >=3.9
 Description-Content-Type: text/markdown
 Requires-Dist: pandas>=1.4.2
@@ -24,6 +28,7 @@ Dynamic: author-email
 Dynamic: description
 Dynamic: description-content-type
 Dynamic: home-page
+Dynamic: project-url
 Dynamic: requires-dist
 Dynamic: requires-python
 Dynamic: summary
@@ -36,18 +41,18 @@ Dynamic: summary
  * @LastEditors: Mengwei Li
  * @LastEditTime: 2023-04-16 21:22:03
 -->
-[![Documentation Status](https://readthedocs.org/projects/discotoolkit-py/badge/?version=latest)](https://discotoolkit-py.readthedocs.io/en/latest/?badge=latest) [![Downloads](https://static.pepy.tech/personalized-badge/discotoolkit?period=total&units=international_system&left_color=black&right_color=orange&left_text=Downloads)](https://pepy.tech/project/discotoolkit) [![PyPI version](https://img.shields.io/pypi/v/discotoolkit)](https://pypi.org/project/discotoolkit)
+[![Documentation](https://img.shields.io/badge/docs-disco.bii.a--star.edu.sg-blue)](https://disco.bii.a-star.edu.sg/v1/docs/toolkit/guide/) [![Downloads](https://static.pepy.tech/personalized-badge/discotoolkit?period=total&units=international_system&left_color=black&right_color=orange&left_text=Downloads)](https://pepy.tech/project/discotoolkit) [![PyPI version](https://img.shields.io/pypi/v/discotoolkit)](https://pypi.org/project/discotoolkit)
-# DISCOtoolkit 1.2.0
+# DISCOtoolkit 1.2.2
-DISCOtoolkit is a Python package for accessing the data and tools of the [DISCO database](https://disco.bii.a-star.edu.sg/) (DISCO v1). Read the documentation at [discotoolkit-py.readthedocs.io](https://discotoolkit-py.readthedocs.io/en/latest/).
+DISCOtoolkit is a Python package for accessing the data and tools of the [DISCO database](https://disco.bii.a-star.edu.sg/) (DISCO v1). Read the documentation, with tutorials and the API reference, at [disco.bii.a-star.edu.sg/v1/docs/toolkit/guide](https://disco.bii.a-star.edu.sg/v1/docs/toolkit/guide/).
 - Filter and download DISCO data based on sample metadata and cell type information
 - Gene search: a gene's expression across cell types and tissues
 - CELLiD: cell type annotation
 - scEnrichment: gene set enrichment using DISCO DEGs
-> **DISCO v1 and v2.** The toolkit talks to **DISCO v1** by default. DISCO v2 has its own R package ([DISCOtoolkit](https://github.com/JinmiaoChenLab/DISCOtoolkit)); the Python package can still be pointed at a v2 server, see [Choosing a server](#choosing-a-server).
+> **Use DISCO v1.** The toolkit is developed and tested against **DISCO v1**, which is the default, and that is the server we recommend. DISCO v2 has its own R package ([DISCOtoolkit](https://github.com/JinmiaoChenLab/DISCOtoolkit)). The Python package can be pointed at a v2 server, but that is not tested; see [Choosing a server](#choosing-a-server).
 ## Quickstart (Google Colab)
@@ -97,7 +102,7 @@ python -m pip install -U discotoolkit
 import discotoolkit as dt
 dt.get_server()                       # 'https://disco.bii.a-star.edu.sg/disco_v3_api/'  (DISCO v1, the default)
-dt.set_server("v2")                   # DISCO v2
+dt.set_server("v2")                   # DISCO v2 (not tested; DISCO v1 is the supported server)
 dt.set_server("https://my.server/disco_v3_api/")   # any server with the same API
 ```
@@ -110,11 +115,12 @@ Two small test files need nothing beyond `requests`, so they run anywhere, inclu
 ```
 python tests/test_settings.py           # offline: server selection logic
 python tests/test_version.py            # offline: setup.py and __init__.py agree on the version
+python tests/test_docs.py               # offline: notebooks are clean, every public function is in the API reference
 python tests/test_server_contract.py    # online: does the server answer everything the toolkit needs?
-python tests/test_server_contract.py https://immunesinglecell.org/disco_v3_api/   # check another server
+python tests/test_server_contract.py <api root>   # check another server
 ```
-GitHub Actions runs these, and installs the package on Python 3.9-3.12, on every push. Maintainers: see [RELEASING.md](RELEASING.md) for how a release is published.
+GitHub Actions runs these, and installs the package on Python 3.9-3.12, on every push. Maintainers: see [RELEASING.md](RELEASING.md) for how a release, and its documentation, is published.
 `test_server_contract.py` checks status codes and the shape of each response (the columns and file types the toolkit reads). It only samples the large reference files, it does not download them.

{discotoolkit-1.2.0 → discotoolkit-1.2.2}/discotoolkit.egg-info/SOURCES.txt RENAMED Viewed

@@ -17,6 +17,9 @@ discotoolkit.egg-info/SOURCES.txt
 discotoolkit.egg-info/dependency_links.txt
 discotoolkit.egg-info/requires.txt
 discotoolkit.egg-info/top_level.txt
+tests/test_docs.py
+tests/test_gene_search.py
+tests/test_reference_loading.py
 tests/test_server_contract.py
 tests/test_settings.py
 tests/test_version.py

{discotoolkit-1.2.0 → discotoolkit-1.2.2}/setup.py RENAMED Viewed

@@ -6,7 +6,7 @@ with open("requirements.txt", "r") as f:
 setup(
     name='discotoolkit',
-    version="1.2.0",
+    version="1.2.2",
     url='https://disco.bii.a-star.edu.sg/',
     author='Li Mengwei, Rom Uddamvathanak',
     author_email='uddamvathanak_rom@immunol.a-star.edu.sg',
@@ -17,4 +17,11 @@ setup(
     include_package_data=True,
     long_description=open("README.md", encoding="utf-8").read(),
     long_description_content_type="text/markdown",
+    # shown on the PyPI page
+    project_urls={
+        "Documentation": "https://disco.bii.a-star.edu.sg/v1/docs/toolkit/guide/",
+        "Source": "https://github.com/JinmiaoChenLab/DISCOtoolkit_py",
+        "Issues": "https://github.com/JinmiaoChenLab/DISCOtoolkit_py/issues",
+        "DISCO": "https://disco.bii.a-star.edu.sg/v1/",
+    },
 )

discotoolkit-1.2.2/tests/test_docs.py ADDED Viewed

@@ -0,0 +1,112 @@
+"""
+The documentation is built from this repository on every change. These checks keep it honest and
+need no network and no third-party packages:
+  * the committed notebooks carry no saved output (the build re-runs them against the live
+    database, so a saved output could only ever be stale);
+  * every page in the navigation exists;
+  * every public function of the package appears in the API reference, so a new function cannot
+    ship undocumented.
+    python tests/test_docs.py
+"""
+import ast
+import glob
+import json
+import os
+import re
+import sys
+ROOT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..")
+# helpers that are not part of the public interface
+INTERNAL = {
+    "api_url", "check_in_list", "generate_correlation_map", "get_disco_metadata", "get_json",
+    "get_sample_ct_info", "write_10X_h5",
+}
+def read(path):
+    with open(os.path.join(ROOT, path), encoding="utf-8") as handle:
+        return handle.read()
+def test_committed_notebooks_have_no_saved_output():
+    dirty = []
+    for path in sorted(glob.glob(os.path.join(ROOT, "docs", "*.ipynb"))):
+        for cell in json.load(open(path, encoding="utf-8"))["cells"]:
+            if cell["cell_type"] == "code" and (cell.get("outputs") or cell.get("execution_count")):
+                dirty.append(os.path.basename(path))
+                break
+    assert not dirty, "run `python tools/execute_notebooks.py --clear` on: %s" % ", ".join(dirty)
+def test_every_page_in_the_navigation_exists():
+    nav = read("mkdocs.yml").split("nav:")[1].split("\nplugins:")[0]
+    pages = re.findall(r":\s+([\w./-]+\.(?:md|ipynb))\s*$", nav, re.M)
+    assert pages, "found no pages in mkdocs.yml"
+    missing = [p for p in pages if not os.path.exists(os.path.join(ROOT, "docs", p))]
+    assert not missing, "in mkdocs.yml but not in docs/: %s" % missing
+def test_every_public_function_is_in_the_api_reference():
+    api = read("docs/API.md")
+    documented = set(re.findall(r"^:::\s+discotoolkit\.\w+\.(\w+)", api, re.M))
+    public = set()
+    for path in glob.glob(os.path.join(ROOT, "discotoolkit", "*.py")):
+        tree = ast.parse(open(path, encoding="utf-8").read())
+        for node in tree.body:
+            if isinstance(node, (ast.FunctionDef, ast.ClassDef)) and not node.name.startswith("_"):
+                public.add(node.name)
+    undocumented = sorted(public - documented - INTERNAL)
+    assert not undocumented, (
+        "not in docs/API.md (add `::: discotoolkit.<Module>.<name>`, or list it as internal in "
+        "tests/test_docs.py): %s" % undocumented
+    )
+def test_api_reference_names_only_things_that_exist():
+    api = read("docs/API.md")
+    for module, name in re.findall(r"^:::\s+discotoolkit\.(\w+)\.(\w+)", api, re.M):
+        source = read("discotoolkit/%s.py" % module)
+        assert re.search(r"^(def|class)\s+%s\b" % name, source, re.M), "%s.%s is gone" % (module, name)
+def test_tutorials_link_to_pages_not_to_notebook_files_and_not_to_the_v2_site():
+    problems = []
+    for path in sorted(glob.glob(os.path.join(ROOT, "docs", "*.ipynb"))):
+        text = "\n".join("".join(c["source"]) for c in json.load(open(path, encoding="utf-8"))["cells"])
+        name = os.path.basename(path)
+        # a relative link to another .ipynb works on GitHub but is a dead link on the site
+        for target in re.findall(r"\]\(([^)\s]+)\)", text):
+            if target.endswith(".ipynb") and not target.startswith("http"):
+                problems.append("%s links to the file %s; link to the page on the docs site" % (name, target))
+        if "immunesinglecell.org" in text:
+            problems.append("%s sends readers to the DISCO V2 site; V1 is the supported server" % name)
+    assert not problems, "; ".join(problems)
+def test_the_changelog_has_notes_for_the_current_version():
+    # the release workflow builds the GitHub Release text from it, and refuses a version without notes
+    version = re.search(r'__version__\s*=\s*"([^"]+)"', read("discotoolkit/__init__.py")).group(1)
+    assert re.search(r"^## %s\b" % re.escape(version), read("CHANGELOG.md"), re.M), (
+        "add a '## %s' section to CHANGELOG.md before releasing" % version
+    )
+def _run_all():
+    tests = [(n, f) for n, f in sorted(globals().items()) if n.startswith("test_") and callable(f)]
+    failed = 0
+    for name, fn in tests:
+        try:
+            fn()
+            print("  PASS  %s" % name)
+        except Exception as error:
+            failed += 1
+            print("  FAIL  %s\n          %s: %s" % (name, type(error).__name__, error))
+    print("\n%d/%d passed" % (len(tests) - failed, len(tests)))
+    return 1 if failed else 0
+if __name__ == "__main__":
+    sys.exit(_run_all())

discotoolkit-1.2.2/tests/test_gene_search.py ADDED Viewed

@@ -0,0 +1,61 @@
+"""
+gene_search draws with seaborn and matplotlib, which change between releases (a plot argument
+that was ignored in one release is an error in the next). This runs the real plotting code on a
+fake server answer, so a library upgrade that breaks it fails here instead of in a user's notebook.
+    python tests/test_gene_search.py
+"""
+import os
+import sys
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt  # noqa: E402
+from unittest import mock  # noqa: E402
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
+from discotoolkit import GeneSearch  # noqa: E402
+# [semicolon-separated values, cell type, (unused), atlas]
+ROWS = [
+    ["10;12;11;13;9;14", "T cell", "x", "thymus"],
+    ["30;28;31;29;33;27", "Cardiomyocyte", "x", "heart"],
+    ["3;4;2;5;3;4", "Fibroblast", "x", "heart"],
+]
+class FakeResponse:
+    def raise_for_status(self):
+        pass
+    def json(self):
+        return ROWS
+def test_gene_search_draws_a_figure_for_all_atlases_and_for_a_subset():
+    with mock.patch.object(GeneSearch.requests, "get", return_value=FakeResponse()):
+        for atlas in (None, ["heart"], "thymus"):
+            plt.close("all")
+            drawn = []
+            with mock.patch.object(plt, "show", lambda *a, **k: drawn.append(plt.gcf())):
+                assert GeneSearch.gene_search("CD4", atlas=atlas, figsize=(6, 4), dpi=60) is None
+            assert len(drawn) == 1 and drawn[0].axes, "no figure for atlas=%r" % (atlas,)
+def _run_all():
+    tests = [(n, f) for n, f in sorted(globals().items()) if n.startswith("test_") and callable(f)]
+    failed = 0
+    for name, fn in tests:
+        try:
+            fn()
+            print("  PASS  %s" % name)
+        except Exception as error:
+            failed += 1
+            print("  FAIL  %s\n          %s: %s" % (name, type(error).__name__, error))
+    print("\n%d/%d passed" % (len(tests) - failed, len(tests)))
+    return 1 if failed else 0
+if __name__ == "__main__":
+    sys.exit(_run_all())

discotoolkit-1.2.2/tests/test_reference_loading.py ADDED Viewed

@@ -0,0 +1,98 @@
+"""
+_load_reference: the CELLiD reference tables are downloaded once, as CSV when the server has it,
+and fall back to the old pickle when it does not. No network: the server is faked.
+    python tests/test_reference_loading.py
+"""
+import os
+import sys
+import tempfile
+from unittest import mock
+import pandas as pd
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
+from discotoolkit import CELLiD  # noqa: E402
+CSV = b'"","T cell--atlas_a","B cell--atlas_a"\n"CD4",1.5,0.0\n"MS4A1",0.0,2.5\n'
+class FakeResponse:
+    def __init__(self, status, body=b""):
+        self.status_code = status
+        self.content = body
+    def __enter__(self):
+        return self
+    def __exit__(self, *args):
+        return False
+    def iter_content(self, chunk_size=None):
+        yield self.content
+    def raise_for_status(self):
+        if self.status_code >= 400:
+            raise RuntimeError("HTTP %s" % self.status_code)
+def server(csv_status, calls):
+    def get(url, params=None, **kwargs):
+        calls.append((url.rsplit("/", 1)[-1], (params or {}).get("type")))
+        if (params or {}).get("type") == "csv":
+            return FakeResponse(csv_status, CSV if csv_status == 200 else b"")
+        # the pickle fallback: a gzip pickle of a small table
+        path = os.path.join(tempfile.gettempdir(), "disco_fake.pkl")
+        pd.DataFrame({"gene": ["CD4"], "group": ["T cell--a"]}).to_pickle(
+            path, compression={"method": "gzip", "compresslevel": 6}
+        )
+        return FakeResponse(200, open(path, "rb").read())
+    return get
+def test_csv_is_downloaded_once_and_read_with_genes_as_index():
+    calls = []
+    with tempfile.TemporaryDirectory() as d, mock.patch.object(CELLiD.requests, "get", server(200, calls)):
+        first = CELLiD._load_reference(d, "ref_data", "getRef", index_col=0)
+        again = CELLiD._load_reference(d, "ref_data", "getRef", index_col=0)
+        assert list(first.index) == ["CD4", "MS4A1"] and list(first.columns)[0] == "T cell--atlas_a"
+        assert first.equals(again)
+        assert calls == [("getRef", "csv")], "second call must use the saved file: %s" % calls
+        assert not os.path.exists(os.path.join(d, "ref_data.csv.part"))
+def test_old_server_without_csv_falls_back_to_the_pickle():
+    calls = []
+    with tempfile.TemporaryDirectory() as d, mock.patch.object(CELLiD.requests, "get", server(400, calls)):
+        deg = CELLiD._load_reference(d, "ref_deg", "getRefDeg")
+        assert list(deg.columns) == ["gene", "group"]
+        assert calls == [("getRefDeg", "csv"), ("getRefDeg", "pkl")], calls
+def test_a_pickle_saved_by_an_earlier_version_is_still_used():
+    calls = []
+    with tempfile.TemporaryDirectory() as d, mock.patch.object(CELLiD.requests, "get", server(200, calls)):
+        pd.DataFrame({"gene": ["X"], "group": ["g"]}).to_pickle(
+            os.path.join(d, "ref_deg.pkl"), compression={"method": "gzip", "compresslevel": 6}
+        )
+        deg = CELLiD._load_reference(d, "ref_deg", "getRefDeg")
+        assert list(deg["gene"]) == ["X"] and calls == [], "must not touch the network"
+def _run_all():
+    tests = [(n, f) for n, f in sorted(globals().items()) if n.startswith("test_") and callable(f)]
+    failed = 0
+    for name, fn in tests:
+        try:
+            fn()
+            print("  PASS  %s" % name)
+        except Exception as error:
+            failed += 1
+            print("  FAIL  %s\n          %s: %s" % (name, type(error).__name__, error))
+    print("\n%d/%d passed" % (len(tests) - failed, len(tests)))
+    return 1 if failed else 0
+if __name__ == "__main__":
+    sys.exit(_run_all())

{discotoolkit-1.2.0 → discotoolkit-1.2.2}/tests/test_server_contract.py RENAMED Viewed

@@ -5,7 +5,7 @@ Needs only `requests`, so it runs anywhere -- including a fresh Colab notebook:
     !pip install requests
     !python tests/test_server_contract.py                                # DISCO v1 (default)
-    !python tests/test_server_contract.py https://immunesinglecell.org/disco_v3_api/   # another server
+    !python tests/test_server_contract.py <api root>                                   # another server
 or with pytest:  DISCO_API_URL=<api root> pytest tests/test_server_contract.py
@@ -114,6 +114,19 @@ def test_cellid_deg_reference_is_a_gzip_pickle():
     assert head[:2] == b"\x1f\x8b", "getRefDeg is not gzip data: %r" % head
+def test_cellid_reference_is_available_as_plain_csv():
+    header, row = first_lines("toolkit/getRef", 2, type="csv")
+    cols = [c.strip('"') for c in header.split(",")]
+    assert cols[0] == "" and len(cols) > 1000, "expected genes x (cell type--atlas) columns"
+    assert all("--" in c for c in cols[1:5]), cols[1:5]
+    assert row.split(",")[0].strip('"'), "rows should start with a gene name"
+def test_cellid_deg_reference_is_available_as_plain_csv():
+    header, row = first_lines("toolkit/getRefDeg", 2, type="csv")
+    assert [c.strip('"') for c in header.split(",")] == ["gene", "group"], header
 def test_gene_set_reference_is_available_as_plain_csv():
     header, row = first_lines("toolkit/getGeneSet", 2, format="csv")
     cols = [c.strip('"') for c in header.split(",")]