cloudnet-api-client 0.1.3__tar.gz → 0.2.0__tar.gz

{cloudnet_api_client-0.1.3 → cloudnet_api_client-0.2.0}/CHANGELOG.md

@@ -5,6 +5,14 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## 0.2.0 – 2025-04-01
+
+- Add download progress bar
+- Add `adownload` function for asynchronous context
+- Only retry on aiohttp errors
+- Extend date parameters
+- Improve type hints
+
 ## 0.1.3 – 2025-03-31
 
 - Add py.typed

cloudnet_api_client-0.2.0/PKG-INFO

@@ -0,0 +1,160 @@
+Metadata-Version: 2.4
+Name: cloudnet-api-client
+Version: 0.2.0
+Summary: Cloudnet API client
+Author-email: Simo Tukiainen <simo.tukiainen@fmi.fi>
+License-File: LICENSE
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Atmospheric Science
+Requires-Python: >=3.10
+Requires-Dist: aiohttp
+Requires-Dist: numpy
+Requires-Dist: requests
+Requires-Dist: tqdm
+Provides-Extra: dev
+Requires-Dist: pre-commit; extra == 'dev'
+Requires-Dist: release-version; extra == 'dev'
+Requires-Dist: types-requests; extra == 'dev'
+Requires-Dist: types-tqdm; extra == 'dev'
+Provides-Extra: test
+Requires-Dist: mypy; extra == 'test'
+Requires-Dist: pytest; extra == 'test'
+Description-Content-Type: text/markdown
+
+[![CI](https://github.com/actris-cloudnet/cloudnet-api-client/actions/workflows/test.yml/badge.svg)](https://github.com/actris-cloudnet/cloudnet-api-client/actions/workflows/test.yml)
+
+# cloudnet-api-client
+
+Official Python client for the [Cloudnet data portal API](https://docs.cloudnet.fmi.fi/api/data-portal.html).
+
+## Installation
+
+```bash
+python3 -m pip install cloudnet-api-client
+```
+
+## Quickstart
+
+```python
+import cloudnet_api_client as cac
+
+client = cac.APIClient()
+
+sites = client.sites(type="cloudnet")
+products = client.products()
+
+metadata = client.metadata("hyytiala", "2021-01-01", product=["mwr", "radar"])
+cac.download(metadata, "data/")
+
+raw_metadata = client.raw_metadata("granada", date="2024-01", instrument_id="parsivel")
+cac.download(raw_metadata, "data_raw/")
+```
+
+## Documentation
+
+### `APIClient().metadata()` and `raw_metadata()` &rarr; `list[Metadata]`
+
+Fetch product and raw file metadata from the Cloudnet data portal.
+
+Parameters:
+
+| name            | type                        | default | example                                              |
+| --------------- | --------------------------- | ------- | ---------------------------------------------------- |
+| site_id         | `str`                       |         | "hyytiala"                                           |
+| date            | `str` or `date`             | `None`  | "2024-01-01"                                         |
+| date_from       | `str` or `date`             | `None`  | "2025-01-01"                                         |
+| date_to         | `str` or `date`             | `None`  | "2025-01-01"                                         |
+| updated_at      | `str`, `date` or `datetime` | `None`  | "2025-01-01T12:00:00"                                |
+| updated_at_from | `str`, `date` or `datetime` | `None`  | "2025-01-01T12:00:00"                                |
+| updated_at_to   | `str`, `date` or `datetime` | `None`  | "2025-01-01T12:00:00"                                |
+| instrument_id   | `str` or `list[str]`        | `None`  | "rpg-fmcw-94"                                        |
+| instrument_pid  | `str` or `list[str]`        | `None`  | "https://hdl.handle.net/21.12132/3.191564170f8a4686" |
+| product\*       | `str` or `list[str]`        | `None`  | "classification"                                     |
+| show_legacy\*   | `bool`                      | `False` |                                                      |
+
+\* = only in `metadata()`
+
+**Date Handling**
+
+The `date`, `date_from` and `date_to` parameters support:
+
+- "YYYY-MM-DD" — a specific date
+- "YYYY-MM" — the entire month
+- "YYYY" — the entire year
+- Or directly as `datetime.date` object
+
+In addition to these, the `updated_at`, `updated_at_from` and `updated_at_to` parameters support:
+
+- "YYYY-MM-DDTHH" — a specific hour
+- "YYYY-MM-DDTHH:MM" — a specific minute
+- "YYYY-MM-DDTHH:MM:SS" — a specific second
+- "YYYY-MM-DDTHH:MM:SS.FFFFFF" — a specific microsecond
+- Or directly as `datetime.datetime` object
+
+**Return value**
+
+Both methods return a list of `dataclass` instances, `ProductMetadata` and `RawMetadata`, respectively.
+
+### `APIClient().filter(list[Metadata])` &rarr; `list[Metadata]`
+
+Additional filtering of fetched metadata.
+
+Parameters:
+
+| name               | type                                           | default |
+| ------------------ | ---------------------------------------------- | ------- |
+| metadata           | `list[RawMetadata]` or `list[ProductMetadata]` |         |
+| include_pattern    | `str`                                          | `None`  |
+| exclude_pattern    | `str`                                          | `None`  |
+| filename_prefix    | `str`                                          | `None`  |
+| filename_suffix    | `str`                                          | `None`  |
+| include_tag_subset | `set[str]`                                     | `None`  |
+| exclude_tag_subset | `set[str]`                                     | `None`  |
+
+### `APIClient().sites()` &rarr; `list[Site]`
+
+Fetch cloudnet sites.
+
+Parameters:
+
+| name | type                 | Choices                                   | default |
+| ---- | -------------------- | ----------------------------------------- | ------- |
+| type | `str` or `list[str]` | "cloudnet", "campaign", "model", "hidden" | `None`  |
+
+### `APIClient().products()` &rarr; `list[Product]`
+
+Fetch cloudnet products.
+
+Parameters:
+
+| name | type                 | Choices                                   | default |
+| ---- | -------------------- | ----------------------------------------- | ------- |
+| type | `str` or `list[str]` | "instrument", "geophysical", "evaluation" | `None`  |
+
+### `APIClient().instruments()` &rarr; `list[Instrument]`
+
+Fetch cloudnet instruments.
+
+### `cloudnet_api_client.download(list[Metadata])`
+
+Download files from the fetched metadata.
+
+Parameters:
+
+| name              | type                                           | default |
+| ----------------- | ---------------------------------------------- | ------- |
+| metadata          | `list[RawMetadata]` or `list[ProductMetadata]` |         |
+| output_directory  | `PathLike` or `str`                            |         |
+| concurrency_limit | `int`                                          | 5       |
+| progress          | `bool` or `None`                               | `None`  |
+
+There's also an asynchronous version of this function:
+`cloudnet_api_client.adownload`. It's useful for usage inside Jupyter notebook.
+
+## License
+
+MIT

cloudnet_api_client-0.2.0/README.md

@@ -0,0 +1,133 @@
+[![CI](https://github.com/actris-cloudnet/cloudnet-api-client/actions/workflows/test.yml/badge.svg)](https://github.com/actris-cloudnet/cloudnet-api-client/actions/workflows/test.yml)
+
+# cloudnet-api-client
+
+Official Python client for the [Cloudnet data portal API](https://docs.cloudnet.fmi.fi/api/data-portal.html).
+
+## Installation
+
+```bash
+python3 -m pip install cloudnet-api-client
+```
+
+## Quickstart
+
+```python
+import cloudnet_api_client as cac
+
+client = cac.APIClient()
+
+sites = client.sites(type="cloudnet")
+products = client.products()
+
+metadata = client.metadata("hyytiala", "2021-01-01", product=["mwr", "radar"])
+cac.download(metadata, "data/")
+
+raw_metadata = client.raw_metadata("granada", date="2024-01", instrument_id="parsivel")
+cac.download(raw_metadata, "data_raw/")
+```
+
+## Documentation
+
+### `APIClient().metadata()` and `raw_metadata()` → `list[Metadata]`
+
+Fetch product and raw file metadata from the Cloudnet data portal.
+
+Parameters:
+
+| name            | type                        | default | example                                              |
+| --------------- | --------------------------- | ------- | ---------------------------------------------------- |
+| site_id         | `str`                       |         | "hyytiala"                                           |
+| date            | `str` or `date`             | `None`  | "2024-01-01"                                         |
+| date_from       | `str` or `date`             | `None`  | "2025-01-01"                                         |
+| date_to         | `str` or `date`             | `None`  | "2025-01-01"                                         |
+| updated_at      | `str`, `date` or `datetime` | `None`  | "2025-01-01T12:00:00"                                |
+| updated_at_from | `str`, `date` or `datetime` | `None`  | "2025-01-01T12:00:00"                                |
+| updated_at_to   | `str`, `date` or `datetime` | `None`  | "2025-01-01T12:00:00"                                |
+| instrument_id   | `str` or `list[str]`        | `None`  | "rpg-fmcw-94"                                        |
+| instrument_pid  | `str` or `list[str]`        | `None`  | "https://hdl.handle.net/21.12132/3.191564170f8a4686" |
+| product\*       | `str` or `list[str]`        | `None`  | "classification"                                     |
+| show_legacy\*   | `bool`                      | `False` |                                                      |
+
+\* = only in `metadata()`
+
+**Date Handling**
+
+The `date`, `date_from` and `date_to` parameters support:
+
+- "YYYY-MM-DD" — a specific date
+- "YYYY-MM" — the entire month
+- "YYYY" — the entire year
+- Or directly as `datetime.date` object
+
+In addition to these, the `updated_at`, `updated_at_from` and `updated_at_to` parameters support:
+
+- "YYYY-MM-DDTHH" — a specific hour
+- "YYYY-MM-DDTHH:MM" — a specific minute
+- "YYYY-MM-DDTHH:MM:SS" — a specific second
+- "YYYY-MM-DDTHH:MM:SS.FFFFFF" — a specific microsecond
+- Or directly as `datetime.datetime` object
+
+**Return value**
+
+Both methods return a list of `dataclass` instances, `ProductMetadata` and `RawMetadata`, respectively.
+
+### `APIClient().filter(list[Metadata])` → `list[Metadata]`
+
+Additional filtering of fetched metadata.
+
+Parameters:
+
+| name               | type                                           | default |
+| ------------------ | ---------------------------------------------- | ------- |
+| metadata           | `list[RawMetadata]` or `list[ProductMetadata]` |         |
+| include_pattern    | `str`                                          | `None`  |
+| exclude_pattern    | `str`                                          | `None`  |
+| filename_prefix    | `str`                                          | `None`  |
+| filename_suffix    | `str`                                          | `None`  |
+| include_tag_subset | `set[str]`                                     | `None`  |
+| exclude_tag_subset | `set[str]`                                     | `None`  |
+
+### `APIClient().sites()` → `list[Site]`
+
+Fetch cloudnet sites.
+
+Parameters:
+
+| name | type                 | Choices                                   | default |
+| ---- | -------------------- | ----------------------------------------- | ------- |
+| type | `str` or `list[str]` | "cloudnet", "campaign", "model", "hidden" | `None`  |
+
+### `APIClient().products()` → `list[Product]`
+
+Fetch cloudnet products.
+
+Parameters:
+
+| name | type                 | Choices                                   | default |
+| ---- | -------------------- | ----------------------------------------- | ------- |
+| type | `str` or `list[str]` | "instrument", "geophysical", "evaluation" | `None`  |
+
+### `APIClient().instruments()` → `list[Instrument]`
+
+Fetch cloudnet instruments.
+
+### `cloudnet_api_client.download(list[Metadata])`
+
+Download files from the fetched metadata.
+
+Parameters:
+
+| name              | type                                           | default |
+| ----------------- | ---------------------------------------------- | ------- |
+| metadata          | `list[RawMetadata]` or `list[ProductMetadata]` |         |
+| output_directory  | `PathLike` or `str`                            |         |
+| concurrency_limit | `int`                                          | 5       |
+| progress          | `bool` or `None`                               | `None`  |
+
+There's also an asynchronous version of this function:
+`cloudnet_api_client.adownload`. It's useful for usage inside Jupyter notebook.
+
+## License
+
+MIT

{cloudnet_api_client-0.1.3 → cloudnet_api_client-0.2.0}/cloudnet_api_client/__init__.py

@@ -1,2 +1,3 @@
 from .client import APIClient as APIClient
+from .dl import adownload as adownload
 from .dl import download as download

{cloudnet_api_client-0.1.3 → cloudnet_api_client-0.2.0}/cloudnet_api_client/client.py

@@ -22,6 +22,7 @@ from cloudnet_api_client.containers import (
 
 T = TypeVar("T")
 DateParam = str | datetime.date | None
+DateTimeParam = str | datetime.datetime | datetime.date | None
 QueryParam = str | list[str] | None
 
 
@@ -72,8 +73,9 @@ class APIClient:
         date: DateParam = None,
         date_from: DateParam = None,
         date_to: DateParam = None,
-        updated_at_from: DateParam = None,
-        updated_at_to: DateParam = None,
+        updated_at: DateTimeParam = None,
+        updated_at_from: DateTimeParam = None,
+        updated_at_to: DateTimeParam = None,
         instrument_id: QueryParam = None,
         instrument_pid: QueryParam = None,
         model_id: QueryParam = None,
@@ -87,10 +89,10 @@ class APIClient:
             "product": product,
             "showLegacy": show_legacy,
         }
-        date_params = _mangle_dates(
-            date, date_from, date_to, updated_at_from, updated_at_to
+        _add_date_params(
+            params, date, date_from, date_to, updated_at, updated_at_from, updated_at_to
         )
-        params.update(date_params)
+
         files_res = self._get_response("files", params)
 
         # Add model files if requested
@@ -100,7 +102,7 @@ class APIClient:
             params["model"] = model_id
             files_res += self._get_response("model-files", params)
 
-        return _build_objects(files_res, ProductMetadata)
+        return _build_meta_objects(files_res)
 
     def raw_metadata(
         self,
@@ -108,8 +110,9 @@ class APIClient:
         date: DateParam = None,
         date_from: DateParam = None,
         date_to: DateParam = None,
-        updated_at_from: DateParam = None,
-        updated_at_to: DateParam = None,
+        updated_at: DateTimeParam = None,
+        updated_at_from: DateTimeParam = None,
+        updated_at_to: DateTimeParam = None,
         instrument_id: QueryParam = None,
         instrument_pid: QueryParam = None,
     ) -> list[RawMetadata]:
@@ -118,10 +121,9 @@ class APIClient:
             "instrument": instrument_id,
             "instrumentPid": instrument_pid,
         }
-        date_params = _mangle_dates(
-            date, date_from, date_to, updated_at_from, updated_at_to
+        _add_date_params(
+            params, date, date_from, date_to, updated_at, updated_at_from, updated_at_to
         )
-        params.update(date_params)
         res = self._get_response("raw-files", params)
         return _build_raw_meta_objects(res)
 
@@ -172,48 +174,107 @@ class APIClient:
         return res.json()
 
 
-def _mangle_dates(
+def _add_date_params(
+    params: dict,
     date: DateParam,
     date_from: DateParam,
     date_to: DateParam,
-    updated_at_from: DateParam,
-    updated_at_to: DateParam,
-) -> dict:
-    params = {}
-    if isinstance(date, datetime.date):
-        params["date"] = date
-    elif isinstance(date, str):
-        if re.fullmatch(r"\d{4}-\d{2}-\d{2}", date):
-            params["date"] = _parse_date(date)
-        elif re.fullmatch(r"\d{4}-\d{2}", date):
-            date = datetime.datetime.strptime(date, "%Y-%m")
-            last_day_number = calendar.monthrange(date.year, date.month)[1]
-            params["dateFrom"] = datetime.date(date.year, date.month, 1)
-            params["dateTo"] = datetime.date(date.year, date.month, last_day_number)
-        elif re.fullmatch(r"\d{4}", date):
-            params["dateFrom"] = datetime.date(int(date), 1, 1)
-            params["dateTo"] = datetime.date(int(date), 12, 31)
-        else:
-            raise ValueError("Invalid date format")
-    else:
-        if date_from:
-            params["dateFrom"] = _parse_date(date_from)
-        if date_to:
-            params["dateTo"] = _parse_date(date_to)
-    if updated_at_from:
-        params["updatedAtFrom"] = _parse_date(updated_at_from)
-    if updated_at_to:
-        params["updatedAtTo"] = _parse_date(updated_at_to)
-    return params
-
-
-def _parse_date(date: str | datetime.date) -> datetime.date:
+    updated_at: DateTimeParam,
+    updated_at_from: DateTimeParam,
+    updated_at_to: DateTimeParam,
+):
+    if date is not None and (date_from is not None or date_to is not None):
+        msg = "Cannot use 'date' with 'date_from' and 'date_to'"
+        raise ValueError(msg)
+    if date is not None:
+        start, stop = _parse_date(date)
+        params["dateFrom"] = start.isoformat()
+        params["dateTo"] = stop.isoformat()
+    if date_from is not None:
+        params["dateFrom"] = _parse_date(date_from)[0].isoformat()
+    if date_to is not None:
+        params["dateTo"] = _parse_date(date_to)[1].isoformat()
+
+    if updated_at is not None and (
+        updated_at_from is not None or updated_at_to is not None
+    ):
+        msg = "Cannot use 'updated_at' with 'updated_at_from' and 'updated_at_to'"
+        raise ValueError(msg)
+    if updated_at is not None:
+        start, stop = _parse_datetime(updated_at)
+        params["updatedAtFrom"] = start.isoformat()
+        params["updatedAtTo"] = stop.isoformat()
+    if updated_at_from is not None:
+        params["updatedAtFrom"] = _parse_datetime(updated_at_from)[0].isoformat()
+    if updated_at_to is not None:
+        params["updatedAtTo"] = _parse_datetime(updated_at_to)[1].isoformat()
+
+
+def _parse_date(date: DateParam) -> tuple[datetime.date, datetime.date]:
     if isinstance(date, datetime.date):
-        return date
-    try:
-        return datetime.datetime.strptime(date, "%Y-%m-%d").date()
-    except ValueError as e:
-        raise ValueError(f"Invalid date format: {date}") from e
+        return date, date
+    error = ValueError(f"Invalid date format: {date}")
+    if isinstance(date, str):
+        try:
+            parts = [int(part) for part in date.split("-")]
+        except ValueError:
+            raise error from None
+        match parts:
+            case [year, month, day]:
+                date = datetime.date(year, month, day)
+                return date, date
+            case [year, month]:
+                last_day_number = calendar.monthrange(year, month)[1]
+                return datetime.date(year, month, 1), datetime.date(
+                    year, month, last_day_number
+                )
+            case [year]:
+                return datetime.date(year, 1, 1), datetime.date(year, 12, 31)
+    raise error
+
+
+def _parse_datetime(dt: DateTimeParam) -> tuple[datetime.datetime, datetime.datetime]:
+    if isinstance(dt, datetime.datetime):
+        return dt, dt
+    if isinstance(dt, datetime.date):
+        return datetime.datetime.combine(
+            dt, datetime.time(0, 0, 0, 0)
+        ), datetime.datetime.combine(dt, datetime.time(23, 59, 59, 999999))
+    if isinstance(dt, str):
+        patterns = {
+            ("%Y", "years"),
+            ("%Y-%m", "months"),
+            ("%Y-%m-%d", "days"),
+            ("%Y-%m-%dT%H", "hours"),
+            ("%Y-%m-%dT%H:%M", "minutes"),
+            ("%Y-%m-%dT%H:%M:%S", "seconds"),
+            ("%Y-%m-%dT%H:%M:%S.%f", "microseconds"),
+        }
+        for fmt, unit in patterns:
+            try:
+                start_date = datetime.datetime.strptime(dt, fmt)
+            except ValueError:
+                continue
+            if unit == "years":
+                end_date = start_date.replace(year=start_date.year + 1)
+            elif unit == "months":
+                if start_date.month == 12:
+                    end_date = start_date.replace(year=start_date.year + 1, month=1)
+                else:
+                    end_date = start_date.replace(month=start_date.month + 1)
+            elif unit == "days":
+                end_date = start_date + datetime.timedelta(days=1)
+            elif unit == "hours":
+                end_date = start_date + datetime.timedelta(hours=1)
+            elif unit == "minutes":
+                end_date = start_date + datetime.timedelta(minutes=1)
+            elif unit == "seconds":
+                end_date = start_date + datetime.timedelta(seconds=1)
+            elif unit == "microseconds":
+                return start_date, start_date
+            return start_date, end_date - datetime.timedelta(microseconds=1)
+    msg = f"Invalid datetime format: {dt}"
+    raise ValueError(msg)
 
 
 def _build_objects(res: list[dict], object_type: type[T]) -> list[T]:
@@ -228,6 +289,22 @@ def _build_objects(res: list[dict], object_type: type[T]) -> list[T]:
     return cast(list[T], objects)
 
 
+def _build_meta_objects(res: list[dict]) -> list[ProductMetadata]:
+    field_names = {f.name for f in fields(ProductMetadata)} - {"product"}
+    return [
+        ProductMetadata(
+            **{_to_snake(k): v for k, v in obj.items() if _to_snake(k) in field_names},
+            product=Product(
+                id=obj["product"]["id"],
+                human_readable_name=obj["product"]["humanReadableName"],
+                type=[obj["product"]["type"][1:-1]],
+                experimental=obj["product"]["experimental"],
+            ),
+        )
+        for obj in res
+    ]
+
+
 def _build_raw_meta_objects(res: list[dict]) -> list[RawMetadata]:
     field_names = {f.name for f in fields(RawMetadata)} - {"instrument"}
     return [

{cloudnet_api_client-0.1.3 → cloudnet_api_client-0.2.0}/cloudnet_api_client/containers.py

@@ -19,7 +19,7 @@ class Site:
     country: str
     country_code: str
     country_subdivision_code: str | None
-    type: SITE_TYPE
+    type: list[SITE_TYPE]
     status: Literal["active", "inactive"]
     gaw: str | None
 
@@ -28,7 +28,7 @@ class Site:
 class Product:
     id: str
     human_readable_name: str
-    type: PRODUCT_TYPE
+    type: list[PRODUCT_TYPE]
     experimental: bool
 
 

{cloudnet_api_client-0.1.3 → cloudnet_api_client-0.2.0}/cloudnet_api_client/dl.py

@@ -5,6 +5,8 @@ from os import PathLike
 from pathlib import Path
 
 import aiohttp
+from tqdm import tqdm
+from tqdm.asyncio import tqdm_asyncio
 
 from cloudnet_api_client import utils
 from cloudnet_api_client.containers import ProductMetadata, RawMetadata
@@ -13,14 +15,32 @@ MetadataList = list[ProductMetadata] | list[RawMetadata]
 
 
 def download(
-    metadata: MetadataList, output_directory: str | PathLike, concurrency_limit: int = 5
+    metadata: MetadataList,
+    output_directory: str | PathLike,
+    concurrency_limit: int = 5,
+    progress: bool | None = None,
+):
+    asyncio.run(adownload(metadata, output_directory, concurrency_limit, progress))
+
+
+async def adownload(
+    metadata: MetadataList,
+    output_directory: str | PathLike,
+    concurrency_limit: int = 5,
+    progress: bool | None = None,
 ) -> None:
+    disable_progress = not progress if progress is not None else None
     os.makedirs(output_directory, exist_ok=True)
-    asyncio.run(_download_files(metadata, output_directory, concurrency_limit))
+    await _download_files(
+        metadata, output_directory, concurrency_limit, disable_progress
+    )
 
 
 async def _download_files(
-    metadata: MetadataList, output_path: str | PathLike, concurrency_limit: int
+    metadata: MetadataList,
+    output_path: str | PathLike,
+    concurrency_limit: int,
+    disable_progress: bool | None,
 ) -> None:
     semaphore = asyncio.Semaphore(concurrency_limit)
     async with aiohttp.ClientSession() as session:
@@ -32,11 +52,13 @@ async def _download_files(
                 continue
             task = asyncio.create_task(
                 _download_file_with_retries(
-                    session, meta.download_url, destination, semaphore
+                    session, meta.download_url, destination, semaphore, disable_progress
                 )
             )
             tasks.append(task)
-        await asyncio.gather(*tasks)
+        await tqdm_asyncio.gather(
+            *tasks, desc="Completed files", disable=disable_progress
+        )
 
 
 async def _download_file_with_retries(
@@ -44,14 +66,15 @@ async def _download_file_with_retries(
     url: str,
     destination: Path,
     semaphore: asyncio.Semaphore,
+    disable_progress: bool | None,
     max_retries: int = 3,
 ) -> None:
     """Attempt to download a file, retrying up to max_retries times if needed."""
     for attempt in range(1, max_retries + 1):
         try:
-            await _download_file(session, url, destination, semaphore)
+            await _download_file(session, url, destination, semaphore, disable_progress)
             return
-        except Exception as e:
+        except aiohttp.ClientError as e:
             logging.warning(f"Attempt {attempt} failed for {url}: {e}")
             if attempt == max_retries:
                 logging.error(f"Giving up on {url} after {max_retries} attempts.")
@@ -65,16 +88,28 @@ async def _download_file(
     url: str,
     destination: Path,
     semaphore: asyncio.Semaphore,
+    disable_progress: bool | None,
 ) -> None:
     async with semaphore:
         async with session.get(url) as response:
             response.raise_for_status()
-            with destination.open("wb") as file_out:
+            with (
+                destination.open("wb") as file_out,
+                tqdm(
+                    desc=destination.name,
+                    total=response.content_length,
+                    unit="iB",
+                    unit_scale=True,
+                    unit_divisor=1024,
+                    disable=disable_progress,
+                ) as bar,
+            ):
                 while True:
                     chunk = await response.content.read(8192)
                     if not chunk:
                         break
                     file_out.write(chunk)
+                    bar.update(len(chunk))
         logging.info(f"Downloaded: {destination}")
 
 

cloudnet_api_client-0.2.0/cloudnet_api_client/version.py

@@ -0,0 +1 @@
+__version__ = "0.2.0"

{cloudnet_api_client-0.1.3 → cloudnet_api_client-0.2.0}/pyproject.toml

@@ -18,12 +18,12 @@ classifiers = [
   "Programming Language :: Python :: 3",
   "Topic :: Scientific/Engineering :: Atmospheric Science",
 ]
-dependencies = ["aiohttp", "numpy", "requests"]
+dependencies = ["aiohttp", "numpy", "requests", "tqdm"]
 dynamic = ["version"]
 
 [project.optional-dependencies]
 test = ["mypy", "pytest"]
-dev = ["pre-commit", "release-version", "types-requests"]
+dev = ["pre-commit", "release-version", "types-requests", "types-tqdm"]
 
 [tool.hatch.version]
 path = "cloudnet_api_client/version.py"

cloudnet_api_client-0.1.3/PKG-INFO

@@ -1,148 +0,0 @@
-Metadata-Version: 2.4
-Name: cloudnet-api-client
-Version: 0.1.3
-Summary: Cloudnet API client
-Author-email: Simo Tukiainen <simo.tukiainen@fmi.fi>
-License-File: LICENSE
-Classifier: Development Status :: 3 - Alpha
-Classifier: Intended Audience :: Science/Research
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Operating System :: OS Independent
-Classifier: Programming Language :: Python :: 3
-Classifier: Topic :: Scientific/Engineering :: Atmospheric Science
-Requires-Python: >=3.10
-Requires-Dist: aiohttp
-Requires-Dist: numpy
-Requires-Dist: requests
-Provides-Extra: dev
-Requires-Dist: pre-commit; extra == 'dev'
-Requires-Dist: release-version; extra == 'dev'
-Requires-Dist: types-requests; extra == 'dev'
-Provides-Extra: test
-Requires-Dist: mypy; extra == 'test'
-Requires-Dist: pytest; extra == 'test'
-Description-Content-Type: text/markdown
-
-[![CI](https://github.com/actris-cloudnet/cloudnet-api-client/actions/workflows/test.yml/badge.svg)](https://github.com/actris-cloudnet/cloudnet-api-client/actions/workflows/test.yml)
-
-# cloudnet-api-client
-
-Official Python client for the [Cloudnet data portal API](https://docs.cloudnet.fmi.fi/api/data-portal.html).
-
-## Installation
-
-```bash
-python3 -m pip install cloudnet-api-client
-```
-
-## Quickstart
-
-```python
-import cloudnet_api_client as cac
-
-client = cac.APIClient()
-
-sites = client.sites(type="cloudnet")
-products = client.products()
-
-metadata = client.metadata("hyytiala", "2021-01-01", product=["mwr", "radar"])
-cac.download(metadata, "data/")
-
-raw_metadata = client.raw_metadata("granada", date="2024-01", instrument_id="parsivel")
-cac.download(raw_metadata, "data_raw/")
-```
-
-## Documentation
-
-### `APIClient().metadata()` and `raw_metadata()` &rarr; `[Metadata]`
-
-Fetch product and raw file metadata from the Cloudnet data portal.
-
-Parameters:
-
-| name            | type                     | default | example                                              |
-| --------------- | ------------------------ | ------- | ---------------------------------------------------- |
-| site_id         | `str`                    |         | "hyytiala"                                           |
-| date            | `str` or `datetime.date` | `None`  | "2024-01-01"                                         |
-| date_from       | `str` or `datetime.date` | `None`  | "2025-01-01"                                         |
-| date_to         | `str` or `datetime.date` | `None`  | "2025-01-01"                                         |
-| updated_at_from | `str` or `datetime.date` | `None`  | "2025-01-01"                                         |
-| updated_at_to   | `str` or `datetime.date` | `None`  | "2025-01-01"                                         |
-| instrument_id   | `str` or `[str]`         | `None`  | "rpg-fmcw-94"                                        |
-| instrument_pid  | `str` or `[str]`         | `None`  | "https://hdl.handle.net/21.12132/3.191564170f8a4686" |
-| product\*       | `str` or `[str]`         | `None`  | "classification"                                     |
-| show_legacy\*   | `bool`                   | `False` |                                                      |
-
-\* = only in `metadata()`
-
-**Date Handling**
-
-The `date` parameter supports:
-
-- "YYYY-MM-DD" — a specific date
-- "YYYY-MM" — the entire month
-- "YYYY" — the entire year
-- Or directly as `datetime.date` object
-
-The `date_from`, `date_to`, `updated_at_from` and `updated_at_to` parameters
-should be of form "YYYY-MM-DD" or `datetime.date`. Note that, if `date` is defined, `date_from` and `date_to` have no effect.
-
-**Return value**
-
-Both methods return a list of `dataclass` instances, `ProductMetadata` and `RawMetadata`, respectively.
-
-### `APIClient().filter([Metadata])` &rarr; `[Metadata]`
-
-Additional filtering of fetched metadata.
-
-Parameters:
-
-| name               | type                                   | default |
-| ------------------ | -------------------------------------- | ------- |
-| metadata           | `[RawMetadata]` or `[ProductMetadata]` |         |
-| include_pattern    | `str`                                  | `None`  |
-| exclude_pattern    | `str`                                  | `None`  |
-| filename_prefix    | `str`                                  | `None`  |
-| filename_suffix    | `str`                                  | `None`  |
-| include_tag_subset | `{str}`                                | `None`  |
-| exclude_tag_subset | `{str}`                                | `None`  |
-
-### `APIClient().sites()` &rarr; `[Site]`
-
-Fetch cloudnet sites.
-
-Parameters:
-
-| name | type             | Choices                                   | default |
-| ---- | ---------------- | ----------------------------------------- | ------- |
-| type | `str` or `[str]` | "cloudnet", "campaign", "model", "hidden" | `None`  |
-
-### `APIClient().products()` &rarr; `[Product]`
-
-Fetch cloudnet products.
-
-Parameters:
-
-| name | type             | Choices                                   | default |
-| ---- | ---------------- | ----------------------------------------- | ------- |
-| type | `str` or `[str]` | "instrument", "geophysical", "evaluation" | `None`  |
-
-### `APIClient().instruments()` &rarr; `[Instrument]`
-
-Fetch cloudnet instruments.
-
-### `cloudnet_api_client.download([Metadata])`
-
-Download files from the fetched metadata.
-
-Parameters:
-
-| name              | type                                   | default |
-| ----------------- | -------------------------------------- | ------- |
-| metadata          | `[RawMetadata]` or `[ProductMetadata]` |         |
-| output_directory  | `PathLike` or `str`                    |         |
-| concurrency_limit | `int`                                  | 5       |
-
-## License
-
-MIT

cloudnet_api_client-0.1.3/README.md

@@ -1,123 +0,0 @@
-[![CI](https://github.com/actris-cloudnet/cloudnet-api-client/actions/workflows/test.yml/badge.svg)](https://github.com/actris-cloudnet/cloudnet-api-client/actions/workflows/test.yml)
-
-# cloudnet-api-client
-
-Official Python client for the [Cloudnet data portal API](https://docs.cloudnet.fmi.fi/api/data-portal.html).
-
-## Installation
-
-```bash
-python3 -m pip install cloudnet-api-client
-```
-
-## Quickstart
-
-```python
-import cloudnet_api_client as cac
-
-client = cac.APIClient()
-
-sites = client.sites(type="cloudnet")
-products = client.products()
-
-metadata = client.metadata("hyytiala", "2021-01-01", product=["mwr", "radar"])
-cac.download(metadata, "data/")
-
-raw_metadata = client.raw_metadata("granada", date="2024-01", instrument_id="parsivel")
-cac.download(raw_metadata, "data_raw/")
-```
-
-## Documentation
-
-### `APIClient().metadata()` and `raw_metadata()` → `[Metadata]`
-
-Fetch product and raw file metadata from the Cloudnet data portal.
-
-Parameters:
-
-| name            | type                     | default | example                                              |
-| --------------- | ------------------------ | ------- | ---------------------------------------------------- |
-| site_id         | `str`                    |         | "hyytiala"                                           |
-| date            | `str` or `datetime.date` | `None`  | "2024-01-01"                                         |
-| date_from       | `str` or `datetime.date` | `None`  | "2025-01-01"                                         |
-| date_to         | `str` or `datetime.date` | `None`  | "2025-01-01"                                         |
-| updated_at_from | `str` or `datetime.date` | `None`  | "2025-01-01"                                         |
-| updated_at_to   | `str` or `datetime.date` | `None`  | "2025-01-01"                                         |
-| instrument_id   | `str` or `[str]`         | `None`  | "rpg-fmcw-94"                                        |
-| instrument_pid  | `str` or `[str]`         | `None`  | "https://hdl.handle.net/21.12132/3.191564170f8a4686" |
-| product\*       | `str` or `[str]`         | `None`  | "classification"                                     |
-| show_legacy\*   | `bool`                   | `False` |                                                      |
-
-\* = only in `metadata()`
-
-**Date Handling**
-
-The `date` parameter supports:
-
-- "YYYY-MM-DD" — a specific date
-- "YYYY-MM" — the entire month
-- "YYYY" — the entire year
-- Or directly as `datetime.date` object
-
-The `date_from`, `date_to`, `updated_at_from` and `updated_at_to` parameters
-should be of form "YYYY-MM-DD" or `datetime.date`. Note that, if `date` is defined, `date_from` and `date_to` have no effect.
-
-**Return value**
-
-Both methods return a list of `dataclass` instances, `ProductMetadata` and `RawMetadata`, respectively.
-
-### `APIClient().filter([Metadata])` → `[Metadata]`
-
-Additional filtering of fetched metadata.
-
-Parameters:
-
-| name               | type                                   | default |
-| ------------------ | -------------------------------------- | ------- |
-| metadata           | `[RawMetadata]` or `[ProductMetadata]` |         |
-| include_pattern    | `str`                                  | `None`  |
-| exclude_pattern    | `str`                                  | `None`  |
-| filename_prefix    | `str`                                  | `None`  |
-| filename_suffix    | `str`                                  | `None`  |
-| include_tag_subset | `{str}`                                | `None`  |
-| exclude_tag_subset | `{str}`                                | `None`  |
-
-### `APIClient().sites()` → `[Site]`
-
-Fetch cloudnet sites.
-
-Parameters:
-
-| name | type             | Choices                                   | default |
-| ---- | ---------------- | ----------------------------------------- | ------- |
-| type | `str` or `[str]` | "cloudnet", "campaign", "model", "hidden" | `None`  |
-
-### `APIClient().products()` → `[Product]`
-
-Fetch cloudnet products.
-
-Parameters:
-
-| name | type             | Choices                                   | default |
-| ---- | ---------------- | ----------------------------------------- | ------- |
-| type | `str` or `[str]` | "instrument", "geophysical", "evaluation" | `None`  |
-
-### `APIClient().instruments()` → `[Instrument]`
-
-Fetch cloudnet instruments.
-
-### `cloudnet_api_client.download([Metadata])`
-
-Download files from the fetched metadata.
-
-Parameters:
-
-| name              | type                                   | default |
-| ----------------- | -------------------------------------- | ------- |
-| metadata          | `[RawMetadata]` or `[ProductMetadata]` |         |
-| output_directory  | `PathLike` or `str`                    |         |
-| concurrency_limit | `int`                                  | 5       |
-
-## License
-
-MIT

cloudnet_api_client-0.1.3/cloudnet_api_client/version.py

@@ -1 +0,0 @@
-__version__ = "0.1.3"