impc-api 1.0.3__tar.gz → 1.0.4__tar.gz

impc_api-1.0.4/MANIFEST.in

@@ -0,0 +1 @@
+include impc_api/utils/core_fields.json

{impc_api-1.0.3 → impc_api-1.0.4}/PKG-INFO

@@ -1,9 +1,10 @@
 Metadata-Version: 2.1
 Name: impc_api
-Version: 1.0.3
-Summary: A package to facilitate making API request to the IMPC Solr API
+Version: 1.0.4
+Summary: A package to facilitate making API requests to the IMPC Solr API
 Home-page: https://github.com/mpi2/impc-data-api-workshop
 Author: MPI2, Marina Kan, Diego Pava
+Project-URL: Homepage, https://github.com/mpi2/impc-data-api-workshop
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE

impc_api-1.0.4/README.md

@@ -0,0 +1,194 @@
+# IMPC_API
+
+`impc_api` is a Python package.
+  
+The functions in this package are intended for use in a Jupyter Notebook.
+
+## Installation Instructions
+
+1. **Create a virtual environment (optional but recommended)**:
+On Mac:
+ ```bash
+ python3 -m venv .venv
+ source .venv/bin/activate
+ ```
+
+2. **Install the package**: `pip install impc_api`
+3. **Install Jupyter**: `pip install jupyter`
+4. **Run the Jupyter Notebook**: `jupyter notebook`
+
+After executing the command, the Jupyter interface should open in your browser. If it does not, follow the instructions provided in the terminal.
+
+5. **Try it out**:
+
+Create a [Jupyter Notebook](https://jupyter-notebook.readthedocs.io/en/latest/) and try some of the examples below:
+
+## Available functions
+
+The available functions can be imported as:
+
+```python
+from impc_api import solr_request, batch_solr_request
+```
+
+# 1. Solr request
+
+The most basic request to the IMPC solr API
+
+```python
+num_found, df = solr_request(
+    core='genotype-phenotype', 
+    params={
+        'q': '*:*',
+        'rows': 10, 
+        'fl': 'marker_symbol,allele_symbol,parameter_stable_id'
+    }
+)
+```
+
+## a. Facet request
+
+`solr_request` allows facet requests
+
+```python
+num_found, df = solr_request(
+    core="genotype-phenotype",
+    params={
+         "q": "*:*",
+         "rows": 0,
+         "facet": "on",
+         "facet.field": "zygosity",
+         "facet.limit": 15,
+         "facet.mincount": 1,
+    }
+)
+```
+
+## b. Solr request validation
+
+A common pitfall when writing a query is the misspelling of `core` and `fields` arguments. For this, we have included a `validate` argument that raises a warning when these values are not as expected. Note this does not prevent you from executing a query; it just alerts you to a potential issue.
+
+### Core validation
+
+```python
+num_found, df = solr_request(
+    core='invalid_core',
+    params={
+        'q': '*:*',
+        'rows': 10
+    },
+    validate=True
+)
+
+> InvalidCoreWarning: Invalid core: "genotype-phenotyp", select from the available cores:
+> dict_keys(['experiment', 'genotype-phenotype', 'impc_images', 'phenodigm', 'statistical-result'])
+```
+
+### Field list validation
+
+```python
+num_found, df = solr_request(
+    core='genotype-phenotype',
+    params={
+        'q': '*:*',
+        'rows': 10,
+        'fl': 'invalid_field,marker_symbol,allele_symbol'
+    },
+    validate=True
+)
+> InvalidFieldWarning: Unexpected field name: "invalid_field". Check the spelling of fields.
+> To see expected fields check the documentation at: https://www.ebi.ac.uk/mi/impc/solrdoc/
+```
+
+# 2. Batch Solr Request
+
+`batch_solr_request` is available for large queries. This solves issues where a request is too large to fit into memory or where it puts a lot of strain on the API.
+
+Use `batch_solr_request` for:
+
+- Large queries (>1,000,000)
+- Querying multiple items in a list
+- Downloading data in `json` or `csv` format.
+
+## Large queries
+
+For large queries you can choose between seeing them in a DataFrame or downloading them in `json` or `csv` format.
+
+## a. Large query - see in DataFrame
+
+This will fetch your data using the API responsibly and return a Pandas DataFrame
+
+When your request is larger than recommended and you have not opted for downloading the data, a warning will be presented and you should follow the instructions to proceed.
+
+```python
+df = batch_solr_request(
+    core='genotype-phenotype',
+    params={
+        'q':'*:*'
+    },
+    download=False,
+    batch_size=30000
+)
+print(df.head())
+```
+
+## b. Large query - Download
+
+When using the `download=True` option, a file with the requested information will be saved as `filename`. The format is selected based on the `wt` parameter.
+A DataFrame may be returned, provided it does not exceed the memory available on your laptop. If the DataFrame is too large, an error will be raised. For these cases, we recommend you read the downloaded file in batches/chunks.  
+
+```python
+df = batch_solr_request(
+    core='genotype-phenotype',
+    params={
+        'q':'*:*',
+        'wt':'csv'
+    },
+    download=True,
+    filename='geno_pheno_query',
+    batch_size=100000
+)
+print(df.head())
+```
+
+## c. Query by multiple values
+
+`batch_solr_request` also allows to search multiple items in a list provided they belong to them same field.
+Pass the list to the `field_list` param and specify the type of `fl` in `field_type`.
+
+```python
+# List of gene symbols
+genes = ["Zfp580", "Firrm", "Gpld1", "Mbip"]
+
+df = batch_solr_request(
+    core='genotype-phenotype',
+    params={
+        'q':'*:*',
+        'fl': 'marker_symbol,mp_term_name,p_value',
+        'field_list': genes,
+        'field_type': 'marker_symbol'
+    },
+    download = False
+)
+print(df.head())
+```
+
+This can be downloaded too:
+
+```python
+# List of gene symbols
+genes = ["Zfp580", "Firrm", "Gpld1", "Mbip"]
+
+df = batch_solr_request(
+    core='genotype-phenotype',
+    params={
+        'q':'*:*',
+        'fl': 'marker_symbol,mp_term_name,p_value',
+        'field_list': genes,
+        'field_type': 'marker_symbol'
+    },
+    download = True,
+    filename='gene_list_query'
+)
+print(df.head())
+```

impc_api-1.0.4/impc_api/__init__.py

@@ -0,0 +1,6 @@
+from .solr_request import solr_request
+from .batch_solr_request import batch_solr_request
+from .utils import validators, warnings
+
+# Control what gets imported by client
+__all__ = ["solr_request", "batch_solr_request"]

impc_api-1.0.4/impc_api/batch_solr_request.py

@@ -0,0 +1,266 @@
+import json
+import warnings
+from pathlib import Path
+
+import pandas as pd
+import requests
+from tqdm import tqdm
+from IPython.display import display
+
+from impc_api.utils.validators import DownloadFormatValidator
+from impc_api.utils.warnings import (
+    warning_config,
+    RowsParamIgnored,
+    UnsupportedDownloadFormatError,
+)
+from .solr_request import solr_request
+
+# Initialise warning config
+warning_config()
+
+def batch_solr_request(
+    core, params, download=False, batch_size=5000, filename="batch_request"
+):
+    """Function for large API requests (>1,000,000 results). Fetches the data in batches and
+    produces a Pandas DataFrame or downloads a file in json or csv formats.
+
+    Additionally, allows to search multiple items in a list provided they belong to them same field.
+
+    Args:
+        core (str): name of IMPC solr core.
+        params (dict): dictionary containing the API call parameters.
+        download (bool, optional): True for download a local file, False to display results as a DataFrame. Defaults to False.
+        batch_size (int, optional): Size of batches to fetch the data. Defaults to 5000.
+        filename (str, optional): When download=True, select the name of the file. Defaults to 'batch_request'.
+
+
+    Returns:
+        pd.DataFrame: if download=False, displays a DataFrame with the results.
+        None: if download=True, displays a statement on the console and returns None.
+    """
+
+    # If params["rows"] is passed, the user is warned about no effect
+    if params.get("rows") is not None:
+        warnings.warn(
+            message='The "rows" param will be ignored in batch_solr_request. To set a batch size, specify a "batch_size" argument.',
+            category=RowsParamIgnored,
+        )
+
+    # Set params for batch request
+    params["start"] = 0  # Start at the first result
+    # Override param rows in case there was input. Defines batch size to 5000 by default
+    params["rows"] = batch_size
+
+    # If user did not specify format, defaults to json.
+    if params.get("wt") is None:
+        params["wt"] = "json"
+
+    # Check if it's multiple request
+    if params.get("field_list") is not None:
+        # Extract entities_list and entity_type from params
+        field_list = params.pop("field_list")
+        field_type = params.pop("field_type")
+
+        # Construct the filter query with grouped model IDs
+        fq = "{}:({})".format(
+            field_type, " OR ".join(["{}".format(id) for id in field_list])
+        )
+        # Show users the field and field values they passed to the function
+        print("Queried field:", fq)
+        # Set internal params the users should not change
+        params["fq"] = fq
+
+    # Determine the total number of rows. Note that we do not request any data (rows = 0).
+    num_results, _ = solr_request(
+        core=core, params={**params, "start": 0, "rows": 0, "wt": "json"}, silent=True
+    )
+    print(f"Number of found documents: {num_results}")
+
+    # # Download only logic
+    # If user decides to download, a generator is used to fetch data in batches without storing results in memory.
+    if download:
+        try:
+            # Check if the format is supported
+            DownloadFormatValidator(wt=params["wt"])
+
+            # Implement loop behaviour
+            print("Downloading file...")
+            filename_path = Path(f"{filename}.{params['wt']}")
+            gen = _batch_solr_generator(core, params, num_results)
+            _solr_downloader(params, filename_path, gen)
+            print(f"File saved as: {filename_path}")
+        except UnsupportedDownloadFormatError as e:
+            raise e
+        except Exception as e:
+            raise (f"An error ocurred while downloading the data:{e}")
+
+            # Try to read the downloaded file
+        try:
+            print("Reading downloaded file...")
+            return _read_downloaded_file(filename_path, params["wt"])
+        except Exception as e:
+            raise Exception(f"An unexpected error occured:{e}")
+
+    # If the number of results is small enough and download is off, it's okay to show as df
+    if num_results < 1000000 and not download:
+        return _batch_to_df(core, params, num_results)
+
+    # If it's too big, warn the user and ask if they want to proceed.
+    else:
+        print(
+            "Your request might exceed the available memory. We suggest setting 'download=True' and reading the file in batches"
+        )
+        prompt = input(
+            "Do you wish to proceed anyway? press ('y' or enter to proceed) / type('n' or 'exit' to cancel)"
+        )
+        match prompt:
+            case "n" | "exit":
+                print("Exiting gracefully")
+                exit()
+            case "y" | "":
+                print("Fetching data...")
+                return _batch_to_df(core, params, num_results)
+
+
+# Helper batch_to_df
+def _batch_to_df(core, params, num_results):
+    """Helper function to fetch data in batches and display them in a DataFrame
+
+    Args:
+        core (str): name of IMPC solr core.
+        params (dict): dictionary containing the API call parameters.
+        num_results (int): Number of docs available
+
+    Returns:
+        pd.DataFrame: DataFrame with the results.
+    """
+    start = params["start"]
+    batch_size = params["rows"]
+    chunks = []
+    # If the 'wt' param was changed by error, we set it to 'json'
+    params["wt"] = "json"
+
+    # Request chunks until we have complete data.
+    with tqdm(total=num_results) as pbar:
+        while start < num_results:
+            # Request chunk. We don't need num_results anymore because it does not change.
+            _, df_chunk = solr_request(
+                core=core,
+                params={**params, "start": start, "rows": batch_size},
+                silent=True,
+            )
+
+            # Update progress bar with the number of rows requested.
+            pbar.update(batch_size)
+            pbar.refresh()
+            # Record chunk.
+            chunks.append(df_chunk)
+            # Increment start.
+            start += batch_size
+        # Prepare final dataframe.
+        return pd.concat(chunks, ignore_index=True)
+
+
+def _batch_solr_generator(core, params, num_results):
+    """Generator function to fetch results from the SOLR API in batches using pagination.
+
+    Args:
+        core (str): name of IMPC solr core.
+        params (dict): dictionary containing the API call parameters.
+        num_results (int): Number of docs available
+
+    Raises:
+        Exception: If a problem occurs during the download, an exception is raised.
+
+    Yields:
+        ([dict, str]): A JSON object or plain text with the results.
+    """
+    base_url = "https://www.ebi.ac.uk/mi/impc/solr/"
+    solr_url = base_url + core + "/select"
+    start = params["start"]
+    batch_size = params["rows"]
+
+    with tqdm(total=num_results) as pbar:
+        while start <= num_results:
+            params["start"] = start
+            response = requests.get(solr_url, params=params, timeout=10)
+
+            if response.status_code == 200:
+                if params.get("wt") == "json":
+                    data = response.json()["response"]["docs"]
+                else:
+                    data = response.text
+
+                # Update and refresh the progress bar after getting the data
+                pbar.update(batch_size)
+                pbar.refresh()
+                yield data
+
+            else:
+                raise Exception(f"Request failed. Status code: {response.status_code}")
+
+            # pbar.update(batch_size)
+            start += batch_size
+        print(f"Your request URL after the last call:{response.url}")
+
+
+# File writer
+def _solr_downloader(params, filename, solr_generator):
+    """Function to write the data from the generator into the specified format.
+    Supports json and csv only.
+
+    Args:
+        params (dict): dictionary containing the API call parameters.
+        filename (Path): name for the file to be downloaded. Defaults to "core.format" as passed by parent function.
+        solr_generator ([dict, str]): Generator object with the results.
+    """
+    with open(filename, "w", encoding="UTF-8") as f:
+        if params.get("wt") == "json":
+            f.write("[\n")
+            first_chunk = True
+
+            for chunk in solr_generator:
+                for item in chunk:
+                    if not first_chunk:
+                        f.write(",\n")
+                    json.dump(item, f, ensure_ascii=False)
+                    first_chunk = False
+            f.write("\n]\n")
+
+        elif params.get("wt") == "csv":
+            first_chunk = True
+            for chunk in solr_generator:
+                lines = chunk.splitlines()
+                if first_chunk:
+                    # Write all lines in the first chunk
+                    f.write(chunk)
+                    first_chunk = False
+                else:
+                    # Skip the first line (header) in subsequent chunks
+                    f.write("\n" + "\n".join(lines[1:]) + "\n")
+
+
+# File reader
+def _read_downloaded_file(filename: Path, request_format):
+    """Wrapper for reading files into Pandas DataFrames
+
+    Args:
+        filename (Path): Name of the file to read
+        request_format (str): Format of the file to read. Only 'json' and 'csv' are supported.
+
+    Raises:
+        MemoryError: When there is not enough memory to read the file.
+
+    Returns:
+        pd.DataFrame: Returns a pd.DataFrame with the data from the file.
+    """
+    try:
+        match request_format:
+            case "json":
+                return pd.read_json(filename)
+            case "csv":
+                return pd.read_csv(filename)
+    except MemoryError as exc:
+        raise MemoryError(
+            "MemoryError: Insuficient memory to read the file. Consider reading file in batches using Pandas or Polars."
+        ) from exc

impc_api-1.0.4/impc_api/solr_request.py

@@ -0,0 +1,191 @@
+import pandas as pd
+import requests
+from IPython.display import display
+from tqdm import tqdm
+
+from impc_api.utils.validators import CoreParamsValidator
+
+# Display the whole dataframe <15
+pd.set_option("display.max_rows", 15)
+pd.set_option("display.max_columns", None)
+
+# Create helper function
+def solr_request(core, params, silent=False, validate=False):
+    """Performs a single Solr request to the IMPC Solr API.
+
+    Args:
+        core (str): name of IMPC solr core.
+        params (dict): dictionary containing the API call parameters.
+        silent (bool, optional): default False
+            If True, displays: URL of API call, the number of found docs
+            and a portion of the DataFrame.
+        validate (bool, optional): default False
+            If True, validates the parameters against the core schema and raises warnings
+            if any parameter seems invalid.
+
+
+    Returns:
+        num_found: Number of docs found.
+        pandas.DataFrame: Pandas.DataFrame object with the information requested.
+
+    Example query:
+        num_found, df = solr_request(
+            core='genotype-phenotype',
+            params={
+                'q': '*:*',  # Your query, '*' retrieves all documents.
+                'rows': 10,  # Number of rows to retrieve.
+                'fl': 'marker_symbol,allele_symbol,parameter_stable_id',  # Fields to retrieve.
+            }
+        )
+
+    Faceting query provides a summary of data distribution across the specified fields.
+    Example faceting query:
+        num_found, df = solr_request(
+            core='genotype-phenotype',
+            params={
+                'q': '*:*',  # Your query, '*' retrieves all documents.
+                'rows': 0,  # Number of rows to retrieve.
+                'facet': 'on',  # Enable facet counts in query response.
+                'facet.field': 'zygosity',  # Identifies a field to be treated as a facet.
+                'facet.limit': 15,  # Controls how many constraints should be returned for each facet.
+                'facet.mincount': 1  # Specifies the minimum counts required be included in response.
+            }
+        )
+
+    When querying the phenodigm core, pass 'q': 'type:...'
+    Example phenodigm query:
+        num_found, df = solr_request(
+            core='phenodigm',
+            params={
+                'q': 'type:disease_model_summary',  # Pass the type within the core and filters.
+                'rows': 5
+            }
+        )
+    """
+
+    if validate:
+        CoreParamsValidator(core=core, params=params)
+
+    base_url = "https://www.ebi.ac.uk/mi/impc/solr/"
+    solr_url = base_url + core + "/select"
+
+    response = requests.get(solr_url, params=params)
+    if not silent:
+        print(f"\nYour request:\n{response.request.url}\n")
+
+    # Check if the request was successful (status code 200).
+    if response.status_code == 200:
+        # Parse the JSON response.
+        data = response.json()
+        num_found = data["response"]["numFound"]
+        if not silent:
+            print(f"Number of found documents: {num_found}\n")
+
+        # For faceting query.
+        if params.get("facet") == "on":
+            df = _process_faceting(data, params)
+
+        # For regular query.
+        else:
+            # Extract and add search results to the list.
+            search_results = []
+            for doc in data["response"]["docs"]:
+                search_results.append(doc)
+
+            # Convert the list of dictionaries into a DataFrame and print the DataFrame.
+            df = pd.DataFrame(search_results)
+
+        if not silent:
+            display(df)
+        return num_found, df
+
+    else:
+        print("Error:", response.status_code, response.text)
+
+
+def _process_faceting(data, params):
+    """Processes the faceting data from an API response.
+    Note: This function should not be used alone but only as a helper function for solr_request().
+
+    Args:
+        data (dict): The JSON response from the API containing faceting information.
+        params (dict): Dictionary containing the API call parameters, including the facet field.
+
+    Returns:
+        pandas.DataFrame: A DataFrame with the facet field values and their corresponding counts.
+    """
+
+    # Extract and add faceting query results to the list.
+    facet_counts = data["facet_counts"]["facet_fields"][params["facet.field"]]
+    # Initialize an empty dictionary.
+    faceting_dict = {}
+    # Iterate over the list, taking pairs of elements.
+    for i in range(0, len(facet_counts), 2):
+        # Assign label as key and count as value.
+        label = facet_counts[i]
+        count = facet_counts[i + 1]
+        faceting_dict[label] = [count]
+
+    # Convert the list of dictionaries into a DataFrame and print the DataFrame.
+    df = pd.DataFrame.from_dict(
+        faceting_dict, orient="index", columns=["counts"]
+    ).reset_index()
+    # Rename the columns.
+    df.columns = [params["facet.field"], "count_per_category"]
+    return df
+
+
+# Batch request based on solr_request.
+def batch_request(core, params, batch_size):
+    """Calls `solr_request` multiple times with `params`
+     to retrieve results in chunk `batch_size` rows at a time.
+
+     Passing parameter `rows` is ignored and replaced with `batch_size`
+
+    Args:
+         core (str): name of IMPC solr core.
+         params (dict): dictionary containing the API call parameters.
+         batch_size (int): Size of batches (number of docs) per request.
+
+     Returns:
+         pandas.DataFrame: Pandas.DataFrame object with the information requested.
+
+     Example query:
+         df = batch_request(
+             core="genotype-phenotype",
+             params={
+                 'q': 'top_level_mp_term_name:"cardiovascular system phenotype" AND effect_size:[* TO *] AND life_stage_name:"Late adult"',
+                 'fl': 'allele_accession_id,life_stage_name,marker_symbol,mp_term_name,p_value,parameter_name,parameter_stable_id,phenotyping_center,statistical_method,top_level_mp_term_name,effect_size'
+             },
+             batch_size=100
+         )
+    """
+
+    if "rows" in "params":
+        print(
+            "WARN: You have specified the `params` -> `rows` value. It will be ignored, because the data is retrieved `batch_size` rows at a time."
+        )
+    # Determine the total number of rows. Note that we do not request any data (rows = 0).
+    num_results, _ = solr_request(
+        core=core, params={**params, "start": 0, "rows": 0}, silent=True
+    )
+    # Initialise everything for data retrieval.
+    start = 0
+    chunks = []
+    # Request chunks until we have complete data.
+    with tqdm(total=num_results) as pbar:  # Initialize tqdm progress bar.
+        while start < num_results:
+            # Update progress bar with the number of rows requested.
+            pbar.update(batch_size)
+            # Request chunk. We don't need num_results anymore because it does not change.
+            _, df_chunk = solr_request(
+                core=core,
+                params={**params, "start": start, "rows": batch_size},
+                silent=True,
+            )
+            # Record chunk.
+            chunks.append(df_chunk)
+            # Increment start.
+            start += batch_size
+    # Prepare final dataframe.
+    return pd.concat(chunks, ignore_index=True)