impc-api 1.0.6__tar.gz → 2.0.0__tar.gz

{impc_api-1.0.6 → impc_api-2.0.0}/PKG-INFO

@@ -1,41 +1,41 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.4
 Name: impc_api
-Version: 1.0.6
+Version: 2.0.0
 Summary: A package to facilitate making API requests to the IMPC Solr API
 Author: MPI2, Marina Kan, Diego Pava
-Project-URL: Homepage, https://github.com/mpi2/impc-data-api-workshop
-Requires-Python: >=3.10
-Description-Content-Type: text/markdown
+License-Expression: Apache-2.0
 License-File: LICENSE
+Requires-Dist: ipykernel>=6.29.5
+Requires-Dist: notebook>=7.2.2
 Requires-Dist: pandas>=2.2.0
-Requires-Dist: requests>=2.31.0
+Requires-Dist: requests>=2.33.0
 Requires-Dist: tqdm>=4.66.4
 Requires-Dist: pydantic>=2.9
-Provides-Extra: dev
-Requires-Dist: pytest>=8.2.2; extra == "dev"
+Requires-Python: >=3.10
+Project-URL: Homepage, https://github.com/mpi2/impc-api
+Description-Content-Type: text/markdown
 
 # IMPC_API
 
-`impc_api` is a Python package which provides several helper functions that wrap around the IMPC SOLR API.
-  
-The functions in this package are intended for use in a Jupyter Notebook.
+`impc_api` is a Python package which provides several helper functions that wrap around the IMPC SOLR API. The functions in this package are intended for use in a Jupyter Notebook.
 
 ## Installation Instructions
 
+0. Ensure that Python is installed on your system. The minimum required version is 3.10.
+
 1. **Create a virtual environment (optional but recommended)**:
-On Mac:
+On Mac or Linux:
  ```bash
  python3 -m venv .venv
  source .venv/bin/activate
  ```
 
 2. **Install the package**: `pip install impc_api`
-3. **Install Jupyter**: `pip install jupyter`
-4. **Run the Jupyter Notebook**: `jupyter notebook`
+3. **Run the Jupyter Notebook**: `jupyter notebook`
 
 After executing the command, the Jupyter interface should open in your browser. If it does not, follow the instructions provided in the terminal.
 
-5. **Try it out**:
+4. **Try it out**:
 
 Create a [Jupyter Notebook](https://jupyter-notebook.readthedocs.io/en/latest/) and try some of the examples below:
 
@@ -44,7 +44,7 @@ Create a [Jupyter Notebook](https://jupyter-notebook.readthedocs.io/en/latest/)
 The available functions can be imported as:
 
 ```python
-from impc_api import solr_request, batch_solr_request
+from impc_api import batch_solr_request, get_core_fields, solr_request
 ```
 
 # 1. Solr request
@@ -62,6 +62,20 @@ num_found, df = solr_request(
 )
 ```
 
+By default, `solr_request` uses a 10 second request timeout. To use a different timeout, pass the optional `timeout` argument:
+
+```python
+num_found, df = solr_request(
+    core='genotype-phenotype',
+    params={
+        'q': '*:*',
+        'rows': 10,
+        'fl': 'marker_symbol,allele_symbol,parameter_stable_id'
+    },
+    timeout=30
+)
+```
+
 ## a. Facet request
 
 `solr_request` allows facet requests
@@ -96,7 +110,7 @@ num_found, df = solr_request(
     validate=True
 )
 
-> InvalidCoreWarning: Invalid core: "genotype-phenotyp", select from the available cores:
+> InvalidCoreWarning: Invalid core: "invalid_core", select from the available cores:
 > dict_keys(['experiment', 'genotype-phenotype', 'impc_images', 'phenodigm', 'statistical-result'])
 ```
 
@@ -116,13 +130,33 @@ num_found, df = solr_request(
 > To see expected fields check the documentation at: https://www.ebi.ac.uk/mi/impc/solrdoc/
 ```
 
+## c. URL only
+
+Users might want help producing the URL to fetch the data without the need of a DataFrame. Use the flag `url_only=True` to print or return the URL for your query.
+
+```python
+url, _ = solr_request(
+    core='genotype-phenotype',
+    params={
+        'q': '*:*',
+        'rows': 10,
+        'fl': 'marker_symbol,allele_symbol'
+    },
+    url_only=True
+)
+> "https://www.ebi.ac.uk/mi/impc/solr/genotype-phenotype/select?q=%2A%3A%2A&rows=10&fl=marker_symbol%2Callele_symbol"
+
+print(url)
+> "https://www.ebi.ac.uk/mi/impc/solr/genotype-phenotype/select?q=%2A%3A%2A&rows=10&fl=marker_symbol%2Callele_symbol"
+```
+
 # 2. Batch Solr Request
 
 `batch_solr_request` is available for large queries. This solves issues where a request is too large to fit into memory or where it puts a lot of strain on the API.
 
 Use `batch_solr_request` for:
 
-- Large queries (>1,000,000)
+- Large queries (>1,000,000 rows)
 - Querying multiple items in a list
 - Downloading data in `json` or `csv` format.
 
@@ -134,7 +168,7 @@ For large queries you can choose between seeing them in a DataFrame or downloadi
 
 This will fetch your data using the API responsibly and return a Pandas DataFrame
 
-When your request is larger than recommended and you have not opted for downloading the data, a warning will be presented and you should follow the instructions to proceed.
+When your request is larger than recommended and you have not opted to download the data, a warning will be presented suggesting that you retry your request and download the data.
 
 ```python
 df = batch_solr_request(
@@ -208,3 +242,20 @@ df = batch_solr_request(
 )
 print(df.head())
 ```
+# 3.Get Core Fields
+
+## Retrieve valid fields for a core
+
+Use `get_core_fields` before constructing a request to discover the documented
+fields available for a public IMPC Solr core. The function uses the same source
+of truth as request validation and does not make a network request.
+
+```python
+from impc_api import get_core_fields
+
+get_core_fields("genotype-phenotype")
+```
+
+The complete list is returned. An unknown core returns an empty list. Skills and
+other integrations should call this function instead of maintaining their own
+copies of core field names.

impc_api-1.0.6/impc_api.egg-info/PKG-INFO → impc_api-2.0.0/README.md

@@ -1,41 +1,24 @@
-Metadata-Version: 2.1
-Name: impc_api
-Version: 1.0.6
-Summary: A package to facilitate making API requests to the IMPC Solr API
-Author: MPI2, Marina Kan, Diego Pava
-Project-URL: Homepage, https://github.com/mpi2/impc-data-api-workshop
-Requires-Python: >=3.10
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: pandas>=2.2.0
-Requires-Dist: requests>=2.31.0
-Requires-Dist: tqdm>=4.66.4
-Requires-Dist: pydantic>=2.9
-Provides-Extra: dev
-Requires-Dist: pytest>=8.2.2; extra == "dev"
-
 # IMPC_API
 
-`impc_api` is a Python package which provides several helper functions that wrap around the IMPC SOLR API.
-  
-The functions in this package are intended for use in a Jupyter Notebook.
+`impc_api` is a Python package which provides several helper functions that wrap around the IMPC SOLR API. The functions in this package are intended for use in a Jupyter Notebook.
 
 ## Installation Instructions
 
+0. Ensure that Python is installed on your system. The minimum required version is 3.10.
+
 1. **Create a virtual environment (optional but recommended)**:
-On Mac:
+On Mac or Linux:
  ```bash
  python3 -m venv .venv
  source .venv/bin/activate
  ```
 
 2. **Install the package**: `pip install impc_api`
-3. **Install Jupyter**: `pip install jupyter`
-4. **Run the Jupyter Notebook**: `jupyter notebook`
+3. **Run the Jupyter Notebook**: `jupyter notebook`
 
 After executing the command, the Jupyter interface should open in your browser. If it does not, follow the instructions provided in the terminal.
 
-5. **Try it out**:
+4. **Try it out**:
 
 Create a [Jupyter Notebook](https://jupyter-notebook.readthedocs.io/en/latest/) and try some of the examples below:
 
@@ -44,7 +27,7 @@ Create a [Jupyter Notebook](https://jupyter-notebook.readthedocs.io/en/latest/)
 The available functions can be imported as:
 
 ```python
-from impc_api import solr_request, batch_solr_request
+from impc_api import batch_solr_request, get_core_fields, solr_request
 ```
 
 # 1. Solr request
@@ -62,6 +45,20 @@ num_found, df = solr_request(
 )
 ```
 
+By default, `solr_request` uses a 10 second request timeout. To use a different timeout, pass the optional `timeout` argument:
+
+```python
+num_found, df = solr_request(
+    core='genotype-phenotype',
+    params={
+        'q': '*:*',
+        'rows': 10,
+        'fl': 'marker_symbol,allele_symbol,parameter_stable_id'
+    },
+    timeout=30
+)
+```
+
 ## a. Facet request
 
 `solr_request` allows facet requests
@@ -96,7 +93,7 @@ num_found, df = solr_request(
     validate=True
 )
 
-> InvalidCoreWarning: Invalid core: "genotype-phenotyp", select from the available cores:
+> InvalidCoreWarning: Invalid core: "invalid_core", select from the available cores:
 > dict_keys(['experiment', 'genotype-phenotype', 'impc_images', 'phenodigm', 'statistical-result'])
 ```
 
@@ -116,13 +113,33 @@ num_found, df = solr_request(
 > To see expected fields check the documentation at: https://www.ebi.ac.uk/mi/impc/solrdoc/
 ```
 
+## c. URL only
+
+Users might want help producing the URL to fetch the data without the need of a DataFrame. Use the flag `url_only=True` to print or return the URL for your query.
+
+```python
+url, _ = solr_request(
+    core='genotype-phenotype',
+    params={
+        'q': '*:*',
+        'rows': 10,
+        'fl': 'marker_symbol,allele_symbol'
+    },
+    url_only=True
+)
+> "https://www.ebi.ac.uk/mi/impc/solr/genotype-phenotype/select?q=%2A%3A%2A&rows=10&fl=marker_symbol%2Callele_symbol"
+
+print(url)
+> "https://www.ebi.ac.uk/mi/impc/solr/genotype-phenotype/select?q=%2A%3A%2A&rows=10&fl=marker_symbol%2Callele_symbol"
+```
+
 # 2. Batch Solr Request
 
 `batch_solr_request` is available for large queries. This solves issues where a request is too large to fit into memory or where it puts a lot of strain on the API.
 
 Use `batch_solr_request` for:
 
-- Large queries (>1,000,000)
+- Large queries (>1,000,000 rows)
 - Querying multiple items in a list
 - Downloading data in `json` or `csv` format.
 
@@ -134,7 +151,7 @@ For large queries you can choose between seeing them in a DataFrame or downloadi
 
 This will fetch your data using the API responsibly and return a Pandas DataFrame
 
-When your request is larger than recommended and you have not opted for downloading the data, a warning will be presented and you should follow the instructions to proceed.
+When your request is larger than recommended and you have not opted to download the data, a warning will be presented suggesting that you retry your request and download the data.
 
 ```python
 df = batch_solr_request(
@@ -208,3 +225,20 @@ df = batch_solr_request(
 )
 print(df.head())
 ```
+# 3.Get Core Fields
+
+## Retrieve valid fields for a core
+
+Use `get_core_fields` before constructing a request to discover the documented
+fields available for a public IMPC Solr core. The function uses the same source
+of truth as request validation and does not make a network request.
+
+```python
+from impc_api import get_core_fields
+
+get_core_fields("genotype-phenotype")
+```
+
+The complete list is returned. An unknown core returns an empty list. Skills and
+other integrations should call this function instead of maintaining their own
+copies of core field names.

{impc_api-1.0.6 → impc_api-2.0.0}/impc_api/__init__.py

@@ -1,6 +1,7 @@
 from .solr_request import solr_request
 from .batch_solr_request import batch_solr_request
+from .get_core_fields import get_core_fields
 from .utils import validators, warnings
 
 # Control what gets imported by client
-__all__ = ["solr_request", "batch_solr_request"]
+__all__ = ["solr_request", "batch_solr_request", "get_core_fields"]

{impc_api-1.0.6 → impc_api-2.0.0}/impc_api/batch_solr_request.py

@@ -1,6 +1,8 @@
 import json
 import warnings
+from collections.abc import Iterator
 from pathlib import Path
+from typing import Any, Dict, Iterable, List, Union
 
 import pandas as pd
 import requests
@@ -12,15 +14,23 @@ from impc_api.utils.warnings import (
     warning_config,
     RowsParamIgnored,
     UnsupportedDownloadFormatError,
+    LargeRequestMemoryWarning
 )
 from .solr_request import solr_request
 
 # Initialise warning config
 warning_config()
 
+BatchSolrChunk = Union[List[Dict[str, Any]], str]
+
+
 def batch_solr_request(
-    core, params, download=False, batch_size=5000, filename="batch_request"
-):
+    core: str,
+    params: Dict[str, Any],
+    download: bool = False,
+    batch_size: int = 5000,
+    filename: str = "batch_request",
+) -> pd.DataFrame:
     """Function for large API requests (>1,000,000 results). Fetches the data in batches and
     produces a Pandas DataFrame or downloads a file in json or csv formats.
 
@@ -36,7 +46,6 @@ def batch_solr_request(
 
     Returns:
         pd.DataFrame: if download=False, displays a DataFrame with the results.
-        None: if download=True, displays a statement on the console and returns None.
     """
 
     # If params["rows"] is passed, the user is warned about no effect
@@ -92,38 +101,28 @@ def batch_solr_request(
         except UnsupportedDownloadFormatError as e:
             raise e
         except Exception as e:
-            raise (f"An error ocurred while downloading the data:{e}")
+            raise Exception(f"An error occurred while downloading the data:{e}")
 
             # Try to read the downloaded file
         try:
             print("Reading downloaded file...")
             return _read_downloaded_file(filename_path, params["wt"])
         except Exception as e:
-            raise Exception(f"An unexpected error occured:{e}")
-
-    # If the number of results is small enough and download is off, it's okay to show as df
-    if num_results < 1000000 and not download:
-        return _batch_to_df(core, params, num_results)
+            raise Exception(f"An unexpected error occurred:{e}")
 
-    # If it's too big, warn the user and ask if they want to proceed.
-    else:
-        print(
-            "Your request might exceed the available memory. We suggest setting 'download=True' and reading the file in batches"
-        )
-        prompt = input(
-            "Do you wish to proceed anyway? press ('y' or enter to proceed) / type('n' or 'exit' to cancel)"
+    
+    # If a request is too large, warn the user to use download=true.
+    if num_results > 1000000 and not download:
+        warnings.warn(
+            message="This request may exceed available memory. If the download fails, set 'download=True' and try again.",
+            category=LargeRequestMemoryWarning,
         )
-        match prompt:
-            case "n" | "exit":
-                print("Exiting gracefully")
-                exit()
-            case "y" | "":
-                print("Fetching data...")
-                return _batch_to_df(core, params, num_results)
-
+    return _batch_to_df(core, params, num_results)
 
 # Helper batch_to_df
-def _batch_to_df(core, params, num_results):
+def _batch_to_df(
+    core: str, params: Dict[str, Any], num_results: int
+) -> pd.DataFrame:
     """Helper function to fetch data in batches and display them in a DataFrame
 
     Args:
@@ -161,7 +160,9 @@ def _batch_to_df(core, params, num_results):
         return pd.concat(chunks, ignore_index=True)
 
 
-def _batch_solr_generator(core, params, num_results):
+def _batch_solr_generator(
+    core: str, params: Dict[str, Any], num_results: int
+) -> Iterator[BatchSolrChunk]:
     """Generator function to fetch results from the SOLR API in batches using pagination.
 
     Args:
@@ -205,7 +206,11 @@ def _batch_solr_generator(core, params, num_results):
 
 
 # File writer
-def _solr_downloader(params, filename, solr_generator):
+def _solr_downloader(
+    params: Dict[str, Any],
+    filename: Path,
+    solr_generator: Iterable[BatchSolrChunk],
+) -> None:
     """Function to write the data from the generator into the specified format.
     Supports json and csv only.
 
@@ -230,6 +235,7 @@ def _solr_downloader(params, filename, solr_generator):
         elif params.get("wt") == "csv":
             first_chunk = True
             for chunk in solr_generator:
+                assert isinstance(chunk, str)
                 lines = chunk.splitlines()
                 if first_chunk:
                     # Write all lines in the first chunk
@@ -241,7 +247,7 @@ def _solr_downloader(params, filename, solr_generator):
 
 
 # File reader
-def _read_downloaded_file(filename: Path, request_format):
+def _read_downloaded_file(filename: Path, request_format: str) -> pd.DataFrame:
     """Wrapper for reading files into Pandas DataFrames
 
     Args:

impc_api-2.0.0/impc_api/get_core_fields.py

@@ -0,0 +1,22 @@
+from .utils.validators import ValidationJson
+
+
+def get_core_fields(core: str) -> list[str]:
+    """Return the documented fields available for an IMPC Solr core.
+
+    Fields are read from the same source of truth used to validate requests.
+    This function does not make a request to the IMPC Solr API.
+
+    Args:
+        core: Name of a public IMPC Solr core.
+
+    Returns:
+        The documented field names, or an empty list when ``core`` is unknown.
+
+    Example:
+        >>> from impc_api import get_core_fields
+        >>> fields = get_core_fields("genotype-phenotype")
+        >>> fields[:3]
+        ['doc_id', 'ontology_db_id', 'assertion_type']
+    """
+    return ValidationJson().valid_fields(core)

{impc_api-1.0.6 → impc_api-2.0.0}/impc_api/solr_request.py

@@ -1,3 +1,5 @@
+from typing import Any, Dict, Optional, Tuple, Union
+
 import pandas as pd
 import requests
 from IPython.display import display
@@ -5,12 +7,26 @@ from tqdm import tqdm
 
 from impc_api.utils.validators import CoreParamsValidator
 
+RequestTimeout = Union[float, Tuple[float, float]]
+SolrRequestResult = Optional[
+    Union[Tuple[int, pd.DataFrame], Tuple[Optional[str], None]]
+]
+
+DEFAULT_REQUEST_TIMEOUT: RequestTimeout = 10.0
+
 # Display the whole dataframe <15
 pd.set_option("display.max_rows", 15)
 pd.set_option("display.max_columns", None)
 
 # Create helper function
-def solr_request(core, params, silent=False, validate=False):
+def solr_request(
+    core: str,
+    params: Dict[str, Any],
+    silent: bool = False,
+    validate: bool = False,
+    url_only: bool = False,
+    timeout: RequestTimeout = DEFAULT_REQUEST_TIMEOUT,
+) -> SolrRequestResult:
     """Performs a single Solr request to the IMPC Solr API.
 
     Args:
@@ -22,11 +38,24 @@ def solr_request(core, params, silent=False, validate=False):
         validate (bool, optional): default False
             If True, validates the parameters against the core schema and raises warnings
             if any parameter seems invalid.
+        url_only (bool, optional): default False
+            If true, returns the request URL but no data. 
+        timeout (int or float, optional): default 10
+            Number of seconds to wait for the API request before timing out.
 
 
     Returns:
-        num_found: Number of docs found.
-        pandas.DataFrame: Pandas.DataFrame object with the information requested.
+        Union[Tuple[int, pandas.DataFrame], Tuple[str, None]]:
+            - If url_only is False:
+                A tuple containing:
+                    num_found (int): Number of documents found.
+                    df (pandas.DataFrame): DataFrame object with the information requested.
+            
+            - If url_only is True:
+                A tuple containing:
+                    url (str): The URL of the request if status_code is 200, otherwise None.
+                    None: Placeholder for consistency.
+
 
     Example query:
         num_found, df = solr_request(
@@ -69,7 +98,17 @@ def solr_request(core, params, silent=False, validate=False):
     base_url = "https://www.ebi.ac.uk/mi/impc/solr/"
     solr_url = base_url + core + "/select"
 
-    response = requests.get(solr_url, params=params)
+    response = requests.get(solr_url, params=params, timeout=timeout)
+
+    if url_only:
+        if response.status_code == 200:
+            print(response.request.url)
+            return response.request.url, None
+        else:
+            print(f"Error: {response.status_code}\n{response.text}")
+        return None, None
+
+
     if not silent:
         print(f"\nYour request:\n{response.request.url}\n")
 
@@ -103,7 +142,7 @@ def solr_request(core, params, silent=False, validate=False):
         print("Error:", response.status_code, response.text)
 
 
-def _process_faceting(data, params):
+def _process_faceting(data: Dict[str, Any], params: Dict[str, Any]) -> pd.DataFrame:
     """Processes the faceting data from an API response.
     Note: This function should not be used alone but only as a helper function for solr_request().
 
@@ -136,7 +175,9 @@ def _process_faceting(data, params):
 
 
 # Batch request based on solr_request.
-def batch_request(core, params, batch_size):
+def batch_request(
+    core: str, params: Dict[str, Any], batch_size: int
+) -> pd.DataFrame:
     """Calls `solr_request` multiple times with `params`
      to retrieve results in chunk `batch_size` rows at a time.