esgpull 0.8.0__tar.gz → 0.9.1__tar.gz

{esgpull-0.8.0 → esgpull-0.9.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: esgpull
-Version: 0.8.0
+Version: 0.9.1
 Summary: ESGF data discovery, download, replication tool
 Project-URL: Repository, https://github.com/ESGF/esgf-download
 Project-URL: Documentation, https://esgf.github.io/esgf-download/
@@ -23,6 +23,7 @@ Requires-Dist: click>=8.1.3
 Requires-Dist: httpx>=0.23.0
 Requires-Dist: myproxyclient>=2.1.0
 Requires-Dist: nest-asyncio>=1.5.6
+Requires-Dist: packaging>=25.0
 Requires-Dist: platformdirs>=2.6.2
 Requires-Dist: pyopenssl>=22.1.0
 Requires-Dist: pyparsing>=3.0.9
@@ -62,14 +63,29 @@ for dataset in datasets:
 
 ## Installation
 
-Install `esgpull` using pip or conda:
+`esgpull` is distributed via PyPI:
 
 ```shell
 pip install esgpull
+esgpull --help
+```
+
+For isolated installation, [`uv`](https://github.com/astral-sh/uv) or
+[`pipx`](https://github.com/pypa/pipx) are recommended:
+
+```shell
+# with uv
+uv tool install esgpull
+esgpull --help
+
+# alternatively, uvx enables running without explicit installation (comes with uv)
+uvx esgpull --help
 ```
 
 ```shell
-conda install -c conda-forge ipsl::esgpull
+# with pipx
+pipx install esgpull
+esgpull --help
 ```
 
 ## Usage

{esgpull-0.8.0 → esgpull-0.9.1}/README.md

@@ -27,14 +27,29 @@ for dataset in datasets:
 
 ## Installation
 
-Install `esgpull` using pip or conda:
+`esgpull` is distributed via PyPI:
 
 ```shell
 pip install esgpull
+esgpull --help
+```
+
+For isolated installation, [`uv`](https://github.com/astral-sh/uv) or
+[`pipx`](https://github.com/pypa/pipx) are recommended:
+
+```shell
+# with uv
+uv tool install esgpull
+esgpull --help
+
+# alternatively, uvx enables running without explicit installation (comes with uv)
+uvx esgpull --help
 ```
 
 ```shell
-conda install -c conda-forge ipsl::esgpull
+# with pipx
+pipx install esgpull
+esgpull --help
 ```
 
 ## Usage

{esgpull-0.8.0 → esgpull-0.9.1}/docs/docs/configuration.md

@@ -41,6 +41,9 @@ distrib = "false"
 latest = "true"
 replica = "none"
 retracted = "false"
+
+[plugins]
+enabled = "false"
 ```
 
 To modify a config item from the command line, the dot-separated path to that item must
@@ -124,3 +127,13 @@ Certificates are missing.
 
 The credentials will then be saved under the ``~/.esgpull/auth`` directory, within
 ``credentials.toml``, which can then be used for future sessions.
+
+## Plugins
+
+Plugin functionality is disabled by default and can be enabled with:
+
+```shell
+$ esgpull config plugins.enabled true
+```
+
+Plugin-specific configurations are managed separately through the `esgpull plugins config` command and stored in a dedicated `plugins.toml` file. See the [Plugins page](plugins) for more information.

{esgpull-0.8.0 → esgpull-0.9.1}/docs/docs/glossary.md

@@ -3,3 +3,6 @@ __Dataset__
 
 __Facet__
 : basic element of a dataset's _metadata_. Pair of strings in the form `name:value`, equivalent to a python dictionary's item.
+
+__Plugin__
+: custom code that extends `esgpull` functionality by responding to specific events.

{esgpull-0.8.0 → esgpull-0.9.1}/docs/docs/index.md

@@ -10,6 +10,7 @@ It handles scanning, downloading and updating **datasets**, **files** and *queri
 - Simple syntax for fast data exploration
 - Asynchronous download
 - Highly configurable
+- Plugin system for extensibility
 
 !!! tip "Search datasets"
 
@@ -75,6 +76,10 @@ Have a look at the [Installation page](installation) for more ways to install.
 
 Jump directly to the [Quickstart guide](quickstart) to get to know how to use `esgpull`.
 
+## How to Cite
+
+If you use esgpull in your research, please cite it using the information provided in our [CITATION.cff](https://github.com/ESGF/esgf-download/blob/main/CITATION.cff) file.
+
 
 <!-- [ESGF portal]: https://esgf-node.ipsl.upmc.fr/search/cmip6-ipsl -->
 [ESGF Search API]: https://esgf.github.io/esg-search/ESGF_Search_RESTful_API.html

{esgpull-0.8.0 → esgpull-0.9.1}/docs/docs/installation.md

@@ -7,50 +7,31 @@ This document covers a few ways to install `esgpull`, a necessary first step int
 !!! note "Supporting lower python versions could be done with future releases, do not hesitate to ask for it."
 
 
-## Installing with conda / mamba
+## Installation
 
-To install `esgpull` in a new `conda` environment, run:
+`esgpull` is distributed via PyPI:
 
-```sh
-conda create --name my_env_name esgpull --channel ipsl --channel conda-forge
-```
-
-For `mamba` users:
-
-```sh
-mamba create --name my_env_name esgpull --channel ipsl --channel conda-forge
-```
-
-You can start using `esgpull` after activating the environment:
-
-```sh
-conda activate my_env_name
-esgpull --version
+```shell
+pip install esgpull
+esgpull --help
 ```
 
+For isolated installation, [`uv`](https://github.com/astral-sh/uv) or
+[`pipx`](https://github.com/pypa/pipx) are recommended:
 
-## Install with pip
-
-Make sure your python version meets the requirements (>=3.10), then you can run:
+```shell
+# with uv
+uv tool install esgpull
+esgpull --help
 
-```shell title="Install esgpull from pip"
-pip install git+https://github.com/ESGF/esgf-download
+# alternatively, uvx enables running without explicit installation (comes with uv)
+uvx esgpull --help
 ```
 
-
-## Install from source
-
-Esgpull is developed and maintained on GitHub, you can clone the public repository with:
-
 ```shell
-git clone https://github.com/ESGF/esgf-download
-```
-
-And then install with `pip`:
-
-```
-cd esg-pull
-python -m pip install .
+# with pipx
+pipx install esgpull
+esgpull --help
 ```
 
 
@@ -93,4 +74,4 @@ Installing `esgpull` is the first step to using it, but not the only one.
 
 ## Configuration
 
-`esgpull` is highly configurable, and it is recommended to take a look at the [configuration](configuration.md) page to learn more about it.
+`esgpull` is highly configurable, and it is recommended to take a look at the [configuration](configuration.md) page to learn more about it.

esgpull-0.9.1/docs/docs/plugins.md

@@ -0,0 +1,283 @@
+# Plugins
+
+`esgpull` includes a plugin system that allows you to extend functionality by running custom code when specific events occur during data operations.
+
+## Available events
+
+- **file_complete**: A file download completes successfully
+- **file_error**: A file download fails
+- **dataset_complete**: All files from a dataset are downloaded
+
+Each event handler receives only the parameters relevant to that specific event.
+
+## Enable plugins
+
+Plugins are disabled by default. Enable them using the main config command:
+
+```shell
+$ esgpull config plugins.enabled true
+```
+
+```shell
+[plugins]
+enabled = true
+
+Previous value: False
+```
+
+
+## Plugin management
+
+### List available plugins
+
+```shell
+$ esgpull plugins ls
+```
+
+```shell
+        plugin        │      event       │        function         
+══════════════════════╪══════════════════╪═════════════════════════
+ 🟢 notification     │  file_complete │       notify_download   
+                      │ file_error │        notify_error   
+ 🔴 checksum_verify  │  file_complete │      verify_checksum    
+ 🔴 archive_backup   │  file_complete │      backup_file  
+                    │  dataset_complete │        backup_dataset
+```
+
+This detailed information can also be shown with JSON format with `--json`.
+
+### Create a new plugin
+
+Create a plugin template with all available events:
+
+```shell
+$ esgpull plugins create -n notification
+```
+
+```shell
+Plugin template created at: /path/to/esgpull/plugins/notification.py
+Edit the file to implement your custom plugin logic.
+```
+
+Create a plugin for specific events only:
+
+```shell
+$ esgpull plugins create -n notification file_complete file_error
+```
+
+```shell
+Plugin template created at: /path/to/esgpull/plugins/notification.py
+Edit the file to implement your custom plugin logic.
+```
+
+### Enable and disable plugins
+
+```shell
+$ esgpull plugins enable notification
+```
+
+```shell
+Plugin 'notification' enabled.
+```
+
+```shell
+$ esgpull plugins disable notification
+```
+
+```shell
+Plugin 'notification' disabled.
+```
+
+
+### Test plugins
+
+Test plugins by triggering one event with sample data:
+
+```shell
+$ esgpull plugins test file_complete
+```
+
+```shell
+[2025-06-19 17:33:09]  INFO      esgpull.plugins.notification
+✅ Downloaded: tas_Amon_CESM2_historical_r1i1p1f1_gn_185001-201412.nc
+   Size: 524288000 bytes
+
+[2025-06-19 17:33:09]  INFO      esgpull.plugins.checksum_verify
+✓ Checksum verified for tas_Amon_CESM2_historical_r1i1p1f1_gn_185001-201412.nc
+```
+
+This is the primary debugging tool for plugin development. Use it to verify handlers work correctly before running actual downloads or updates.
+
+## Plugin example
+
+Here's a simple notification plugin that sends a message when files are downloaded:
+
+```python title="plugins/notification.py"
+import pathlib
+import logging
+from esgpull.plugin import Event, on
+import esgpull.models
+
+@on(Event.file_complete, priority="normal")
+def notify_download(file: esgpull.models.File, destination: pathlib.Path, logger: logging.Logger):
+    """Send notification when a file is downloaded."""
+    print(f"✅ Downloaded: {file.filename}")
+    print(f"   Size: {file.size} bytes")
+    logger.info(f"Notified download: {file.filename}")
+
+@on(Event.file_error, priority="normal") 
+def notify_error(file: esgpull.models.File, exception: Exception, logger: logging.Logger):
+    """Send notification when a download fails."""
+    print(f"❌ Failed: {file.filename}")
+    print(f"   Error: {exception}")
+    logger.error(f"Notified error: {file.filename}")
+```
+
+## Plugin configuration
+
+Plugins are configured via an optional `Config` class in the plugin code. The `Config` class attributes define parameters and must include default values.
+
+Plugin configuration is stored separately from esgpull's main config file. This file contains a manifest of enabled/disabled plugins and their custom configurations (which override the `Config` class defaults). Use the `esgpull plugins config` subcommand to manage these settings.
+
+Let's extend our notification plugin with configuration:
+
+```python title="plugins/notification.py"
+import pathlib
+import logging
+from esgpull.plugin import Event, on
+import esgpull.models
+
+class Config:
+    enabled = True
+    email_address = "user@example.com"
+    include_size = True
+    error_alerts = True
+
+@on(Event.file_complete, priority="normal")
+def notify_download(file: esgpull.models.File, destination: pathlib.Path, logger: logging.Logger):
+    """Send notification when a file is downloaded."""
+    if Config.enabled:
+        print(f"✅ Downloaded: {file.filename}")
+        if Config.include_size:
+            print(f"   Size: {file.size} bytes")
+        logger.info(f"Notified download to {Config.email_address}: {file.filename}")
+
+@on(Event.file_error, priority="normal") 
+def notify_error(file: esgpull.models.File, exception: Exception, logger: logging.Logger):
+    """Send notification when a download fails."""
+    if Config.enabled and Config.error_alerts:
+        print(f"❌ Failed: {file.filename}")
+        print(f"   Error: {exception}")
+        logger.error(f"Notified error to {Config.email_address}: {file.filename}")
+```
+
+View all plugin configurations:
+```shell
+$ esgpull plugins config
+```
+
+```shell
+─ /path/to/esgpull/plugins.toml ─
+[notification]
+enabled = true
+email_address = "admin@myproject.org"
+include_size = true
+error_alerts = true
+
+[checksum_verify]
+enabled = false
+algorithm = "sha256"
+
+[archive_backup]
+enabled = false
+archive_path = "/backup/esgf"
+```
+
+View specific plugin configuration:
+```shell
+$ esgpull plugins config notification
+```
+
+```shell
+[notification]
+enabled = true
+email_address = "admin@myproject.org"
+include_size = true
+error_alerts = true
+```
+
+View a specific configuration value:
+```shell
+$ esgpull plugins config notification.email_address
+```
+
+```shell
+[notification]
+email_address = "admin@myproject.org"
+```
+
+Set a configuration value:
+```shell
+$ esgpull plugins config notification.email_address alerts@newdomain.com
+```
+
+```shell
+[notification]
+email_address = "alerts@newdomain.com"
+
+Previous value: admin@myproject.org
+```
+
+Reset to default value:
+```shell
+$ esgpull plugins config notification.email_address --default
+```
+
+```shell
+👍 Config reset to default for notification.email_address
+```
+
+### Config class details
+
+The `Config` class supports these data types:
+
+- **Strings** (`str`): Text values like `"INFO"` or `"user@example.com"`
+- **Integers** (`int`): Whole numbers like `3` or `100`  
+- **Floats** (`float`): Decimal numbers like `0.5` or `10.75`
+- **Booleans** (`bool`): `True` or `False` values
+
+```python
+class Config:
+    """Example showing all supported data types"""
+    # String configuration
+    log_level = "INFO"
+    email_address = "user@example.com"
+    
+    # Integer configuration  
+    max_retries = 3
+    timeout_seconds = 30
+    
+    # Float configuration
+    threshold = 0.75
+    delay_factor = 1.5
+    
+    # Boolean configuration
+    notifications_enabled = True
+    debug_mode = False
+```
+
+**Important limitations:**
+- Lists and dictionaries can be used in the `Config` class, but cannot be configured through the CLI `esgpull plugins config` command
+- For list/dictionary attributes, you must edit the `plugins.toml` file directly
+- Only simple scalar values (strings, integers, floats, booleans) can be managed via CLI commands
+- Configuration values are automatically type-checked when modified through CLI
+
+### Configuration management
+
+Plugin configurations are stored separately from the main `esgpull` configuration in a `plugins.toml` file. Only plugins with a `Config` class can be configured.
+
+The configuration system follows these principles:
+
+- **Defaults in code**: Class attributes define default values
+- **Overrides in file**: Only explicitly changed values are saved to `plugins.toml`
+- **Type safety**: Values are automatically converted and validated based on the default type

{esgpull-0.8.0 → esgpull-0.9.1}/docs/docs/quickstart.md

@@ -87,5 +87,17 @@ $ esgpull status
 
 Loop at the [download page](../download) for more information.
 
+## Extending with plugins
+
+`esgpull` can be extended with custom plugins that respond to download events:
+
+```shell
+$ esgpull config plugins.enabled true
+$ esgpull plugins create -n notification file_complete
+$ esgpull plugins enable notification
+```
+
+See the [Plugins page](../plugins) for more details on creating and managing plugins.
+
 
 [ESGF Search API]: https://esgf.github.io/esg-search/ESGF_Search_RESTful_API.html

{esgpull-0.8.0 → esgpull-0.9.1}/docs/mkdocs.yml

@@ -7,6 +7,7 @@ nav:
   # - Queries: queries.md
   - Data discovery: search.md
   - Download: download.md
+  - Plugins: plugins.md
   - Glossary: glossary.md
   # - Usage guide: usage.md
   # - API Reference: reference.md

{esgpull-0.8.0 → esgpull-0.9.1}/esgpull/cli/__init__.py

@@ -7,9 +7,9 @@ from esgpull import __version__
 from esgpull.cli.add import add
 from esgpull.cli.config import config
 from esgpull.cli.convert import convert
-from esgpull.cli.datasets import datasets
 from esgpull.cli.download import download
 from esgpull.cli.login import login
+from esgpull.cli.plugins import plugins
 from esgpull.cli.remove import remove
 from esgpull.cli.retry import retry
 from esgpull.cli.search import search
@@ -35,13 +35,13 @@ SUBCOMMANDS: list[click.Command] = [
     # autoremove,
     config,
     convert,
-    datasets,
     download,
     # facet,
     # get,
     self,
     # install,
     login,
+    plugins,
     remove,
     retry,
     search,

{esgpull-0.8.0 → esgpull-0.9.1}/esgpull/cli/add.py

@@ -89,6 +89,7 @@ def add(
         esg.ui.print(subgraph)
         empty = Query()
         empty.compute_sha()
+        next_steps_queries = []
         for query in queries:
             query.compute_sha()
             esg.graph.resolve_require(query)
@@ -99,7 +100,8 @@ def add(
                 esg.ui.print(f"Skipping existing query: {query.rich_name}")
             else:
                 esg.graph.add(query)
-                esg.ui.print(f"New query added: {query.rich_name}")
+                if query.tracked:
+                    next_steps_queries.append(query)
         new_queries = esg.graph.merge()
         nb = len(new_queries)
         ies = "ies" if nb > 1 else "y"
@@ -107,4 +109,8 @@ def add(
             esg.ui.print(f":+1: {nb} new quer{ies} added.")
         else:
             esg.ui.print(":stop_sign: No new query was added.")
+        if next_steps_queries:
+            esg.ui.print("\nNext steps:\n")
+            for query in next_steps_queries:
+                esg.ui.print(f"\tesgpull update {query.sha[:6]}")
         esg.ui.raise_maybe_record(Exit(0))

{esgpull-0.8.0 → esgpull-0.9.1}/esgpull/cli/config.py

@@ -4,26 +4,11 @@ import click
 from click.exceptions import Abort, BadOptionUsage, Exit
 
 from esgpull.cli.decorators import args, opts
-from esgpull.cli.utils import init_esgpull
+from esgpull.cli.utils import extract_subdict, init_esgpull
 from esgpull.config import ConfigKind
 from esgpull.tui import Verbosity
 
 
-def extract_command(doc: dict, key: str | None) -> dict:
-    if key is None:
-        return doc
-    for part in key.split("."):
-        if not part:
-            raise KeyError(key)
-        elif part in doc:
-            doc = doc[part]
-        else:
-            raise KeyError(part)
-    for part in key.split(".")[::-1]:
-        doc = {part: doc}
-    return doc
-
-
 @click.command()
 @args.key
 @args.value
@@ -73,7 +58,7 @@ def config(
                 )
             kind = esg.config.kind
             old_value = esg.config.update_item(key, value, empty_ok=True)
-            info = extract_command(esg.config.dump(), key)
+            info = extract_subdict(esg.config.dump(), key)
             esg.config.write()
             esg.ui.print(info, toml=True)
             if kind == ConfigKind.NoFile:
@@ -86,12 +71,12 @@ def config(
         elif key is not None:
             if default:
                 old_value = esg.config.set_default(key)
-                info = extract_command(esg.config.dump(), key)
+                info = extract_subdict(esg.config.dump(), key)
                 esg.config.write()
                 esg.ui.print(info, toml=True)
                 esg.ui.print(f"Previous value: {old_value}")
             else:
-                info = extract_command(esg.config.dump(), key)
+                info = extract_subdict(esg.config.dump(), key)
                 esg.ui.print(info, toml=True)
         elif generate:
             overwrite = False
@@ -102,8 +87,7 @@ def config(
                 )
                 esg.ui.raise_maybe_record(Exit(0))
             elif esg.config.kind == ConfigKind.Partial and esg.ui.ask(
-                "A config file already exists,"
-                " fill it with missing defaults?",
+                "A config file already exists, fill it with missing defaults?",
                 default=False,
             ):
                 overwrite = True