cmsdials 1.0.1__tar.gz → 1.2.0__tar.gz

{cmsdials-1.0.1 → cmsdials-1.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: cmsdials
-Version: 1.0.1
+Version: 1.2.0
 Summary: The Python api client interface to DIALS service
 Home-page: https://github.com/cms-DQM/dials-py
 Author: Gabriel Moreira
@@ -15,8 +15,12 @@ Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Provides-Extra: pandas
+Provides-Extra: tqdm
+Requires-Dist: pandas (>=1.5.3) ; extra == "pandas"
 Requires-Dist: pydantic (>=1,<2)
 Requires-Dist: requests (>=2.31.0,<3.0.0)
+Requires-Dist: tqdm (>=4.66.1) ; extra == "tqdm"
 Requires-Dist: typing-extensions (>=3.6.6,<4.6.0)
 Project-URL: Repository, https://github.com/cms-DQM/dials-py
 Description-Content-Type: text/markdown
@@ -33,6 +37,12 @@ To install dials-py, simply
 $ pip install cmsdials
 ```
 
+It is also possible to specify the following extras, to enable optional features:
+
+```
+pandas, tqdm
+```
+
 ## Usage
 
 Before interfacing with any route you need to generate valid credentials, it is possible to authenticate trough the `device authorization flow` or using the `client secret key` of any application registered in DIALS. Note that, the device flow is an interactively authentication procedure that is possible to distinguish users in DIALS backend and the client secret flow is not interactive and is not possible to distinguish users so it should only be used for automation scripts.
@@ -97,11 +107,11 @@ This package interacts with DIALS api endpoints using underlying classes in `Dia
 ### Retrieving a specific object using `get`
 
 ```python
-dials.file_index.get(id=3386119397)
-dials.h1d.get(id=1)
-dials.h2d.get(id=1)
-dials.lumi.get(dataset_id=14677060, run_number=367094, ls_number=1)
-dials.run.get(dataset_id=14677060, run_number=367094)
+dials.file_index.get(dataset_id=14677060, file_id=3393809397)
+dials.h1d.get(dataset_id=14677060, run_number=367112, ls_number=10, me_id=1)
+dials.h1d.get(dataset_id=14677060, run_number=367112, ls_number=10, me_id=96)
+dials.lumi.get(dataset_id=14677060, run_number=367112, ls_number=10)
+dials.run.get(dataset_id=14677060, run_number=367112)
 ```
 
 ### Retrieving a list of objects per page using `list`
@@ -160,12 +170,36 @@ dials.run.list_all(RunFilters(run_number__gte=360392, run_number__lte=365000), m
 
 ### Dials MEs
 
-It is possible to inspect the list of selected MEs considered in DIALS during ETL requesting the endpoint `configured-mes` trough the method:
+It is possible to inspect the list of ingested MEs in DIALS listing the endpoint `mes` trough the method:
+
+```python
+dials.mes.list()
+```
+
+### Automatically convert paginated results to pandas DataFrame
+
+You can enable this optional feature by installing the package with the `pandas` extra.
+
+All `Paginated` metaclasses contain the method `to_pandas` that will automatically transform the `results` attribute of the metaclass into a pandas dataframe, for example:
+
+```python
+data = dials.h1d.list_all(LumisectionHistogram1DFilters(me="PixelPhase1/Tracks/PXBarrel/charge_PXLayer_2"), max_pages=5)
+data.to_pandas()
+```
+
+### Indefinite progress bar when fetch multi-page result
+
+You can enable this optional feature by installing the package with the `tqdm` extra.
+
+Whenever you call a `list_all` method that fetches multiple pages a dynamic progress will be rendered to indicate duration and number of pages, for example:
 
 ```python
-dials.lumi.configured_mes()
+>>> dials.h2d.list_all(LumisectionHistogram2DFilters(me__regex="PXBarrel", ls_number=78, entries__gte=100), max_pages=5)
+Progress: 100%|█████████████████████████████████████| 5/5 [00:02<00:00,  1.70it/s]
 ```
 
+The total attribute of the bar is dynamically updated while fetching the pages.
+
 ## Usage with local DIALS
 
 All classes that interface the DIALS service inherits the class `BaseAPIClient` which propagate the `base_url`, `route` and `version` attributes with production values. In order to use dials-py with a local version of DIALS it is possible to overwrite those attributes when instantiating the `AuthClient` and the `Dials` client, for example:
@@ -186,3 +220,19 @@ dials = Dials(creds, base_url=DEV_URL)
 dials.h2d.list_all(LumisectionHistogram2DFilters(me__regex="EEOT digi occupancy EE +", entries__gte=100, run_number__gte=360392, run_number__lte=365000), max_pages=5)
 ```
 
+## Running tests
+
+The repository has some tests written to make sure DIALS responses are compatible with pydantic metaclasses, you can use pytest to run all tests but you need to specify a secret key to authenticate non-interactively against DIALS api:
+
+```bash
+SECRET_KEY=... pytest tests
+```
+
+The secret key is an api client enabled secret key and can be obtained from the applications portal, any api client secret key whitelisted in DIALS can be used. The interactive authentication flow should be tested manually, an example for this can be found in [this line](/tests/integration/test_auth_client.py#L33).
+
+If testing against a local version of DIALS you need to specify the BASE_URL:
+
+```bash
+SECRET_KEY=... BASE_URl=http://localhost:8000 pytest tests
+```
+

{cmsdials-1.0.1 → cmsdials-1.2.0}/README.md

@@ -10,6 +10,12 @@ To install dials-py, simply
 $ pip install cmsdials
 ```
 
+It is also possible to specify the following extras, to enable optional features:
+
+```
+pandas, tqdm
+```
+
 ## Usage
 
 Before interfacing with any route you need to generate valid credentials, it is possible to authenticate trough the `device authorization flow` or using the `client secret key` of any application registered in DIALS. Note that, the device flow is an interactively authentication procedure that is possible to distinguish users in DIALS backend and the client secret flow is not interactive and is not possible to distinguish users so it should only be used for automation scripts.
@@ -74,11 +80,11 @@ This package interacts with DIALS api endpoints using underlying classes in `Dia
 ### Retrieving a specific object using `get`
 
 ```python
-dials.file_index.get(id=3386119397)
-dials.h1d.get(id=1)
-dials.h2d.get(id=1)
-dials.lumi.get(dataset_id=14677060, run_number=367094, ls_number=1)
-dials.run.get(dataset_id=14677060, run_number=367094)
+dials.file_index.get(dataset_id=14677060, file_id=3393809397)
+dials.h1d.get(dataset_id=14677060, run_number=367112, ls_number=10, me_id=1)
+dials.h1d.get(dataset_id=14677060, run_number=367112, ls_number=10, me_id=96)
+dials.lumi.get(dataset_id=14677060, run_number=367112, ls_number=10)
+dials.run.get(dataset_id=14677060, run_number=367112)
 ```
 
 ### Retrieving a list of objects per page using `list`
@@ -137,12 +143,36 @@ dials.run.list_all(RunFilters(run_number__gte=360392, run_number__lte=365000), m
 
 ### Dials MEs
 
-It is possible to inspect the list of selected MEs considered in DIALS during ETL requesting the endpoint `configured-mes` trough the method:
+It is possible to inspect the list of ingested MEs in DIALS listing the endpoint `mes` trough the method:
+
+```python
+dials.mes.list()
+```
+
+### Automatically convert paginated results to pandas DataFrame
+
+You can enable this optional feature by installing the package with the `pandas` extra.
+
+All `Paginated` metaclasses contain the method `to_pandas` that will automatically transform the `results` attribute of the metaclass into a pandas dataframe, for example:
+
+```python
+data = dials.h1d.list_all(LumisectionHistogram1DFilters(me="PixelPhase1/Tracks/PXBarrel/charge_PXLayer_2"), max_pages=5)
+data.to_pandas()
+```
+
+### Indefinite progress bar when fetch multi-page result
+
+You can enable this optional feature by installing the package with the `tqdm` extra.
+
+Whenever you call a `list_all` method that fetches multiple pages a dynamic progress will be rendered to indicate duration and number of pages, for example:
 
 ```python
-dials.lumi.configured_mes()
+>>> dials.h2d.list_all(LumisectionHistogram2DFilters(me__regex="PXBarrel", ls_number=78, entries__gte=100), max_pages=5)
+Progress: 100%|█████████████████████████████████████| 5/5 [00:02<00:00,  1.70it/s]
 ```
 
+The total attribute of the bar is dynamically updated while fetching the pages.
+
 ## Usage with local DIALS
 
 All classes that interface the DIALS service inherits the class `BaseAPIClient` which propagate the `base_url`, `route` and `version` attributes with production values. In order to use dials-py with a local version of DIALS it is possible to overwrite those attributes when instantiating the `AuthClient` and the `Dials` client, for example:
@@ -162,3 +192,19 @@ dials = Dials(creds, base_url=DEV_URL)
 
 dials.h2d.list_all(LumisectionHistogram2DFilters(me__regex="EEOT digi occupancy EE +", entries__gte=100, run_number__gte=360392, run_number__lte=365000), max_pages=5)
 ```
+
+## Running tests
+
+The repository has some tests written to make sure DIALS responses are compatible with pydantic metaclasses, you can use pytest to run all tests but you need to specify a secret key to authenticate non-interactively against DIALS api:
+
+```bash
+SECRET_KEY=... pytest tests
+```
+
+The secret key is an api client enabled secret key and can be obtained from the applications portal, any api client secret key whitelisted in DIALS can be used. The interactive authentication flow should be tested manually, an example for this can be found in [this line](/tests/integration/test_auth_client.py#L33).
+
+If testing against a local version of DIALS you need to specify the BASE_URL:
+
+```bash
+SECRET_KEY=... BASE_URl=http://localhost:8000 pytest tests
+```

{cmsdials-1.0.1 → cmsdials-1.2.0}/cmsdials/__init__.py

@@ -1,3 +1,4 @@
 from .cmsdials import Dials
 
+
 __all__ = ["Dials"]

{cmsdials-1.0.1 → cmsdials-1.2.0}/cmsdials/auth/_base.py

@@ -1,10 +1,10 @@
 import threading
 from abc import ABC, abstractmethod
+from collections.abc import Mapping
 from datetime import datetime, timedelta
 from enum import Enum
-from typing import Mapping
 
-from .exceptions import ImpossibleToRefreshToken
+from .exceptions import ImpossibleToRefreshTokenError
 
 
 class TokenState(Enum):
@@ -116,7 +116,7 @@ class BaseCredentials(ABC):
         # If token is invalid (both token and refresh token expired): Throw error
         # User need to interactively authenticate again
         if self.token_state == TokenState.INVALID:
-            raise ImpossibleToRefreshToken
+            raise ImpossibleToRefreshTokenError
 
         # If token is stale (token is close to expire)
         # Re-use the token but launch a new thread to refresh the token

{cmsdials-1.0.1 → cmsdials-1.2.0}/cmsdials/auth/bearer.py

@@ -1,8 +1,9 @@
 import json
 import os
 import os.path
+from collections.abc import Mapping
 from datetime import datetime
-from typing import Mapping, Optional
+from typing import Optional
 
 from ..utils._json import TokenDecoder, TokenEncoder
 from ..utils.logger import logger
@@ -10,6 +11,7 @@ from ._base import BaseCredentials, TokenState
 from .client import AuthClient
 from .models import Token
 
+
 DEFAULT_CACHE_DIR = ".cache"
 DEFAULT_CACHE_FILENAME = "creds"
 
@@ -103,7 +105,7 @@ class Credentials(BaseCredentials):
         client = client or AuthClient()
         fpath = Credentials._handle_creds_filepath(cache_dir)
         try:
-            with open(fpath, "r") as filebuffer:
+            with open(fpath) as filebuffer:
                 fcontents = json.load(filebuffer, cls=TokenDecoder)
             token = Token(**fcontents)
         except FileNotFoundError:

{cmsdials-1.0.1 → cmsdials-1.2.0}/cmsdials/auth/client.py

@@ -14,6 +14,8 @@ class AuthClient(BaseAPIClient):
     Client for DIALS auth endpoint
     """
 
+    default_timeout = 30  # seconds
+
     def __init__(self, *args, **kwargs) -> None:
         super().__init__(*args, **kwargs)
 
@@ -24,6 +26,7 @@ class AuthClient(BaseAPIClient):
             headers={
                 "accept": "application/json",
             },
+            timeout=self.default_timeout,
         )
         response.raise_for_status()
         response = response.json()
@@ -38,6 +41,7 @@ class AuthClient(BaseAPIClient):
                 "Content-Type": "application/json",
             },
             json={"device_code": device_code},
+            timeout=self.default_timeout,
         )
 
         if response.status_code == 400 and "authorization_pending" in response.text:
@@ -63,6 +67,7 @@ class AuthClient(BaseAPIClient):
                 "Authorization": f"{token_type} {access_token}",
             },
             json={"refresh_token": refresh_token},
+            timeout=self.default_timeout,
         )
         response.raise_for_status()
         response = response.json()

{cmsdials-1.0.1 → cmsdials-1.2.0}/cmsdials/auth/exceptions.py

@@ -2,5 +2,5 @@ class PendingAuthorizationError(Exception):
     pass
 
 
-class ImpossibleToRefreshToken(Exception):
+class ImpossibleToRefreshTokenError(Exception):
     pass

{cmsdials-1.0.1 → cmsdials-1.2.0}/cmsdials/auth/secret_key.py

@@ -1,4 +1,4 @@
-from typing import Mapping
+from collections.abc import Mapping
 
 from ._base import BaseCredentials
 

cmsdials-1.2.0/cmsdials/clients/file_index/client.py

@@ -0,0 +1,27 @@
+import requests
+from requests.exceptions import HTTPError
+
+from ...utils.api_client import BaseAuthorizedAPIClient
+from ...utils.logger import logger
+from .models import FileIndex, FileIndexFilters, PaginatedFileIndexList
+
+
+class FileIndexClient(BaseAuthorizedAPIClient):
+    data_model = FileIndex
+    pagination_model = PaginatedFileIndexList
+    filter_class = FileIndexFilters
+    lookup_url = "file-index/"
+
+    def get(self, dataset_id: int, file_id: int):
+        endpoint_url = f"{self.api_url}{self.lookup_url}{dataset_id}/{file_id}/"
+        headers = self._build_headers()
+        response = requests.get(endpoint_url, headers=headers, timeout=self.default_timeout)
+
+        try:
+            response.raise_for_status()
+        except HTTPError as err:
+            logger.info(f"Api raw response: {response.text}")
+            raise err
+
+        response = response.json()
+        return self.data_model(**response)

{cmsdials-1.0.1 → cmsdials-1.2.0}/cmsdials/clients/file_index/models.py

@@ -1,9 +1,9 @@
 from datetime import datetime
-from typing import List, Optional
+from typing import Optional
 
 from pydantic import AnyUrl, BaseModel, Field
 
-from ...utils.base_model import OBaseModel
+from ...utils.base_model import OBaseModel, PaginatedBaseModel
 
 
 class FileIndex(BaseModel):
@@ -18,10 +18,10 @@ class FileIndex(BaseModel):
     err_trace: Optional[str] = Field(..., max_length=5000)
 
 
-class PaginatedFileIndexList(BaseModel):
+class PaginatedFileIndexList(PaginatedBaseModel):
     next: Optional[AnyUrl]
     previous: Optional[AnyUrl]
-    results: List[FileIndex]
+    results: list[FileIndex]
 
 
 class FileIndexFilters(OBaseModel):

cmsdials-1.2.0/cmsdials/clients/h1d/client.py

@@ -0,0 +1,27 @@
+import requests
+from requests.exceptions import HTTPError
+
+from ...utils.api_client import BaseAuthorizedAPIClient
+from ...utils.logger import logger
+from .models import LumisectionHistogram1D, LumisectionHistogram1DFilters, PaginatedLumisectionHistogram1DList
+
+
+class LumisectionHistogram1DClient(BaseAuthorizedAPIClient):
+    data_model = LumisectionHistogram1D
+    pagination_model = PaginatedLumisectionHistogram1DList
+    filter_class = LumisectionHistogram1DFilters
+    lookup_url = "th1/"
+
+    def get(self, dataset_id: int, run_number: int, ls_number: int, me_id: int):
+        endpoint_url = f"{self.api_url}{self.lookup_url}{dataset_id}/{run_number}/{ls_number}/{me_id}/"
+        headers = self._build_headers()
+        response = requests.get(endpoint_url, headers=headers, timeout=self.default_timeout)
+
+        try:
+            response.raise_for_status()
+        except HTTPError as err:
+            logger.info(f"Api raw response: {response.text}")
+            raise err
+
+        response = response.json()
+        return self.data_model(**response)

{cmsdials-1.0.1 → cmsdials-1.2.0}/cmsdials/clients/h1d/models.py

@@ -1,12 +1,11 @@
-from typing import List, Optional
+from typing import Optional
 
 from pydantic import AnyUrl, BaseModel, Field
 
-from ...utils.base_model import OBaseModel
+from ...utils.base_model import OBaseModel, PaginatedBaseModel
 
 
 class LumisectionHistogram1D(BaseModel):
-    hist_id: int
     dataset: str = Field(..., max_length=255)
     me: str = Field(..., max_length=255)
     dataset_id: int
@@ -21,10 +20,10 @@ class LumisectionHistogram1D(BaseModel):
     data: list[float]
 
 
-class PaginatedLumisectionHistogram1DList(BaseModel):
+class PaginatedLumisectionHistogram1DList(PaginatedBaseModel):
     next: Optional[AnyUrl]
     previous: Optional[AnyUrl]
-    results: List[LumisectionHistogram1D]
+    results: list[LumisectionHistogram1D]
 
 
 class LumisectionHistogram1DFilters(OBaseModel):

cmsdials-1.2.0/cmsdials/clients/h2d/client.py

@@ -0,0 +1,27 @@
+import requests
+from requests.exceptions import HTTPError
+
+from ...utils.api_client import BaseAuthorizedAPIClient
+from ...utils.logger import logger
+from .models import LumisectionHistogram2D, LumisectionHistogram2DFilters, PaginatedLumisectionHistogram2DList
+
+
+class LumisectionHistogram2DClient(BaseAuthorizedAPIClient):
+    data_model = LumisectionHistogram2D
+    pagination_model = PaginatedLumisectionHistogram2DList
+    filter_class = LumisectionHistogram2DFilters
+    lookup_url = "th2/"
+
+    def get(self, dataset_id: int, run_number: int, ls_number: int, me_id: int):
+        endpoint_url = f"{self.api_url}{self.lookup_url}{dataset_id}/{run_number}/{ls_number}/{me_id}/"
+        headers = self._build_headers()
+        response = requests.get(endpoint_url, headers=headers, timeout=self.default_timeout)
+
+        try:
+            response.raise_for_status()
+        except HTTPError as err:
+            logger.info(f"Api raw response: {response.text}")
+            raise err
+
+        response = response.json()
+        return self.data_model(**response)

{cmsdials-1.0.1 → cmsdials-1.2.0}/cmsdials/clients/h2d/models.py

@@ -2,11 +2,10 @@ from typing import Optional
 
 from pydantic import AnyUrl, BaseModel, Field
 
-from ...utils.base_model import OBaseModel
+from ...utils.base_model import OBaseModel, PaginatedBaseModel
 
 
 class LumisectionHistogram2D(BaseModel):
-    hist_id: int
     dataset: str = Field(..., max_length=255)
     me: str = Field(..., max_length=255)
     dataset_id: int
@@ -24,7 +23,7 @@ class LumisectionHistogram2D(BaseModel):
     data: list[list[float]]
 
 
-class PaginatedLumisectionHistogram2DList(BaseModel):
+class PaginatedLumisectionHistogram2DList(PaginatedBaseModel):
     next: Optional[AnyUrl]
     previous: Optional[AnyUrl]
     results: list[LumisectionHistogram2D]

{cmsdials-1.0.1 → cmsdials-1.2.0}/cmsdials/clients/lumisection/client.py

@@ -15,7 +15,7 @@ class LumisectionClient(BaseAuthorizedAPIClient):
     def get(self, dataset_id: int, run_number: int, ls_number: int):
         endpoint_url = f"{self.api_url}{self.lookup_url}{dataset_id}/{run_number}/{ls_number}/"
         headers = self._build_headers()
-        response = requests.get(endpoint_url, headers=headers)
+        response = requests.get(endpoint_url, headers=headers, timeout=self.default_timeout)
 
         try:
             response.raise_for_status()

{cmsdials-1.0.1 → cmsdials-1.2.0}/cmsdials/clients/lumisection/models.py

@@ -2,7 +2,7 @@ from typing import Optional
 
 from pydantic import AnyUrl, BaseModel, Field
 
-from ...utils.base_model import OBaseModel
+from ...utils.base_model import OBaseModel, PaginatedBaseModel
 
 
 class Lumisection(BaseModel):
@@ -14,7 +14,7 @@ class Lumisection(BaseModel):
     th2_count: int
 
 
-class PaginatedLumisectionList(BaseModel):
+class PaginatedLumisectionList(PaginatedBaseModel):
     next: Optional[AnyUrl]
     previous: Optional[AnyUrl]
     results: list[Lumisection]

cmsdials-1.2.0/cmsdials/clients/mes/client.py

@@ -0,0 +1,8 @@
+from ...utils.api_client import BaseAuthorizedAPIClient
+from .models import MEFilters, MonitoringElement
+
+
+class MonitoringElementClient(BaseAuthorizedAPIClient):
+    data_model = MonitoringElement
+    filter_class = MEFilters
+    lookup_url = "mes/"

cmsdials-1.2.0/cmsdials/clients/mes/models.py

@@ -0,0 +1,18 @@
+from typing import Optional
+
+from pydantic import BaseModel, Field
+
+from ...utils.base_model import OBaseModel
+
+
+class MonitoringElement(BaseModel):
+    me_id: int
+    me: str = Field(..., max_length=255)
+    count: int
+    dim: int
+
+
+class MEFilters(OBaseModel):
+    me: Optional[str] = None
+    me__regex: Optional[str] = None
+    dim: Optional[int] = None

{cmsdials-1.0.1 → cmsdials-1.2.0}/cmsdials/clients/run/client.py

@@ -15,7 +15,7 @@ class RunClient(BaseAuthorizedAPIClient):
     def get(self, dataset_id: int, run_number: int):
         endpoint_url = f"{self.api_url}{self.lookup_url}{dataset_id}/{run_number}/"
         headers = self._build_headers()
-        response = requests.get(endpoint_url, headers=headers)
+        response = requests.get(endpoint_url, headers=headers, timeout=self.default_timeout)
 
         try:
             response.raise_for_status()

{cmsdials-1.0.1 → cmsdials-1.2.0}/cmsdials/clients/run/models.py

@@ -2,7 +2,7 @@ from typing import Optional
 
 from pydantic import AnyUrl, BaseModel, Field
 
-from ...utils.base_model import OBaseModel
+from ...utils.base_model import OBaseModel, PaginatedBaseModel
 
 
 class Run(BaseModel):
@@ -12,7 +12,7 @@ class Run(BaseModel):
     ls_count: int
 
 
-class PaginatedRunList(BaseModel):
+class PaginatedRunList(PaginatedBaseModel):
     next: Optional[AnyUrl]
     previous: Optional[AnyUrl]
     results: list[Run]

{cmsdials-1.0.1 → cmsdials-1.2.0}/cmsdials/cmsdials.py

@@ -5,6 +5,7 @@ from .clients.file_index.client import FileIndexClient
 from .clients.h1d.client import LumisectionHistogram1DClient
 from .clients.h2d.client import LumisectionHistogram2DClient
 from .clients.lumisection.client import LumisectionClient
+from .clients.mes.client import MonitoringElementClient
 from .clients.run.client import RunClient
 
 
@@ -15,3 +16,4 @@ class Dials:
         self.h2d = LumisectionHistogram2DClient(creds, workspace, *args, **kwargs)
         self.lumi = LumisectionClient(creds, workspace, *args, **kwargs)
         self.run = RunClient(creds, workspace, *args, **kwargs)
+        self.mes = MonitoringElementClient(creds, workspace, *args, **kwargs)

{cmsdials-1.0.1 → cmsdials-1.2.0}/cmsdials/filters.py

@@ -2,12 +2,15 @@ from .clients.file_index.models import FileIndexFilters
 from .clients.h1d.models import LumisectionHistogram1DFilters
 from .clients.h2d.models import LumisectionHistogram2DFilters
 from .clients.lumisection.models import LumisectionFilters
+from .clients.mes.models import MEFilters
 from .clients.run.models import RunFilters
 
+
 __all__ = [
     "FileIndexFilters",
     "LumisectionHistogram1DFilters",
     "LumisectionHistogram2DFilters",
     "LumisectionFilters",
     "RunFilters",
+    "MEFilters",
 ]

{cmsdials-1.0.1 → cmsdials-1.2.0}/cmsdials/utils/api_client.py

@@ -1,3 +1,4 @@
+from importlib import util as importlib_util
 from typing import Optional
 from urllib.parse import parse_qs, urlparse
 
@@ -8,6 +9,14 @@ from ..auth._base import BaseCredentials
 from ..utils.logger import logger
 
 
+if importlib_util.find_spec("tqdm"):
+    from tqdm.auto import tqdm
+
+    TQDM_INSTALLED = True
+else:
+    TQDM_INSTALLED = False
+
+
 class BaseAPIClient:
     PRODUCTION_BASE_URL = "https://cmsdials-api.web.cern.ch/"
     PRODUCTION_API_ROUTE = "api/"
@@ -36,6 +45,7 @@ class BaseAuthorizedAPIClient(BaseAPIClient):
     pagination_model = None
     filter_class = None
     lookup_url = None
+    default_timeout = 30  # seconds
 
     def __init__(
         self,
@@ -55,10 +65,10 @@ class BaseAuthorizedAPIClient(BaseAPIClient):
         self.creds.before_request(base)
         return base
 
-    def get(self, id: int):
+    def get(self, id: int):  # noqa: A002
         endpoint_url = self.api_url + self.lookup_url + str(id) + "/"
         headers = self._build_headers()
-        response = requests.get(endpoint_url, headers=headers)
+        response = requests.get(endpoint_url, headers=headers, timeout=self.default_timeout)
 
         try:
             response.raise_for_status()
@@ -73,7 +83,7 @@ class BaseAuthorizedAPIClient(BaseAPIClient):
         filters = filters or self.filter_class()
         endpoint_url = self.api_url + self.lookup_url
         headers = self._build_headers()
-        response = requests.get(endpoint_url, headers=headers, params=filters.cleandict())
+        response = requests.get(endpoint_url, headers=headers, params=filters.cleandict(), timeout=self.default_timeout)
 
         try:
             response.raise_for_status()
@@ -82,13 +92,22 @@ class BaseAuthorizedAPIClient(BaseAPIClient):
             raise err
 
         response = response.json()
-        return self.pagination_model(**response)
+        if self.pagination_model:
+            return self.pagination_model(**response)
+        elif isinstance(response, list):
+            return [self.data_model(**res) for res in response]
 
-    def __list_sync(self, filters, max_pages: Optional[int] = None):
+        raise ValueError("pagination model is None and response is not a list.")
+
+    def __list_sync(self, filters, max_pages: Optional[int] = None, enable_progress: bool = False):
         next_token = None
         results = []
         is_last_page = False
         total_pages = 0
+        use_tqdm = TQDM_INSTALLED and enable_progress
+
+        if use_tqdm:
+            progress = tqdm(desc="Progress", total=1)
 
         while is_last_page is False:
             curr_filters = self.filter_class(**filters.dict())
@@ -98,12 +117,26 @@ class BaseAuthorizedAPIClient(BaseAPIClient):
             is_last_page = response.next is None
             next_token = parse_qs(urlparse(response.next).query).get("next_token") if response.next else None
             total_pages += 1
-            if max_pages and total_pages > max_pages:
+            max_pages_reached = max_pages and total_pages >= max_pages
+            if use_tqdm:
+                if is_last_page or max_pages_reached:
+                    progress.update()
+                else:
+                    progress.total = total_pages + 1
+                    progress.update(1)
+            if max_pages_reached:
                 break
 
+        if use_tqdm:
+            progress.close()
+
         return self.pagination_model(
-            next=None, previous=None, results=results  # No problem re-using last response count
+            next=None,
+            previous=None,
+            results=results,  # No problem re-using last response count
         )
 
-    def list_all(self, filters, max_pages: Optional[int] = None):
-        return self.__list_sync(filters, max_pages)
+    def list_all(self, filters, max_pages: Optional[int] = None, **kwargs):
+        if self.pagination_model is None:
+            return self.list(filters)
+        return self.__list_sync(filters, max_pages, **kwargs)

cmsdials-1.2.0/cmsdials/utils/base_model.py

@@ -0,0 +1,25 @@
+from importlib import util as importlib_util
+
+from pydantic import BaseModel
+
+
+if importlib_util.find_spec("pandas"):
+    import pandas as pd
+
+    PANDAS_NOT_INSTALLED = False
+else:
+    PANDAS_NOT_INSTALLED = True
+
+
+class OBaseModel(BaseModel):
+    def cleandict(self):
+        return {key: value for key, value in self.dict().items() if value is not None}
+
+
+class PaginatedBaseModel(BaseModel):
+    def to_pandas(self):
+        if PANDAS_NOT_INSTALLED:
+            raise RuntimeError(
+                "The 'pandas' package is not installed, you can re-install cmsdials specifying the pandas extra: pip install cmsdials[pandas]"
+            )
+        return pd.DataFrame([res.__dict__ for res in self.results])

{cmsdials-1.0.1 → cmsdials-1.2.0}/cmsdials/utils/logger.py

@@ -1,5 +1,6 @@
 import logging
 
+
 logging.basicConfig(format="[%(asctime)s] %(levelname)s: %(message)s", level=logging.INFO)
 
 logger = logging.getLogger(__name__)

{cmsdials-1.0.1 → cmsdials-1.2.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "cmsdials"
-version = "1.0.1"
+version = "1.2.0"
 description = "The Python api client interface to DIALS service"
 authors = ["Gabriel Moreira <gabrielmscampos@gmail.com>"]
 readme = "README.md"
@@ -21,10 +21,18 @@ python = "^3.9"
 requests = "^2.31.0"
 pydantic = "<2, >=1"
 typing-extensions = "<4.6.0, >=3.6.6"
+pandas = {version = ">=1.5.3", optional = true}
+tqdm = {version = ">=4.66.1", optional = true}
 
 [tool.poetry.group.dev.dependencies]
 pre-commit = "^3.6.2"
 ruff = "^0.4.2"
+notebook = "^7.1.3"
+pytest = "^8.2.0"
+
+[tool.poetry.extras]
+pandas = ["pandas"]
+tqdm = ["tqdm"]
 
 [tool.ruff]
 exclude = [
@@ -52,7 +60,7 @@ exclude = [
 ]
 line-length = 120
 indent-width = 4
-target-version = "py310"
+target-version = "py39"
 
 [tool.ruff.lint]
 select = [

cmsdials-1.0.1/cmsdials/clients/file_index/client.py

@@ -1,9 +0,0 @@
-from ...utils.api_client import BaseAuthorizedAPIClient
-from .models import FileIndex, FileIndexFilters, PaginatedFileIndexList
-
-
-class FileIndexClient(BaseAuthorizedAPIClient):
-    data_model = FileIndex
-    pagination_model = PaginatedFileIndexList
-    filter_class = FileIndexFilters
-    lookup_url = "file-index/"

cmsdials-1.0.1/cmsdials/clients/h1d/client.py

@@ -1,9 +0,0 @@
-from ...utils.api_client import BaseAuthorizedAPIClient
-from .models import LumisectionHistogram1D, LumisectionHistogram1DFilters, PaginatedLumisectionHistogram1DList
-
-
-class LumisectionHistogram1DClient(BaseAuthorizedAPIClient):
-    data_model = LumisectionHistogram1D
-    pagination_model = PaginatedLumisectionHistogram1DList
-    filter_class = LumisectionHistogram1DFilters
-    lookup_url = "th1/"

cmsdials-1.0.1/cmsdials/clients/h2d/client.py

@@ -1,9 +0,0 @@
-from ...utils.api_client import BaseAuthorizedAPIClient
-from .models import LumisectionHistogram2D, LumisectionHistogram2DFilters, PaginatedLumisectionHistogram2DList
-
-
-class LumisectionHistogram2DClient(BaseAuthorizedAPIClient):
-    data_model = LumisectionHistogram2D
-    pagination_model = PaginatedLumisectionHistogram2DList
-    filter_class = LumisectionHistogram2DFilters
-    lookup_url = "th2/"

cmsdials-1.0.1/cmsdials/utils/base_model.py

@@ -1,6 +0,0 @@
-from pydantic import BaseModel
-
-
-class OBaseModel(BaseModel):
-    def cleandict(self):
-        return {key: value for key, value in self.dict().items() if value is not None}