esgpull 0.7.3__tar.gz → 0.9.0__tar.gz

esgpull-0.9.0/CITATION.cff

@@ -0,0 +1,33 @@
+# This CITATION.cff file was generated with cffinit.
+
+cff-version: 1.2.0
+title: esgpull
+message: ESGF command line download tool esgpull
+type: software
+authors:
+  - given-names: Sven
+    family-names: Rodriguez
+    email: sven.rodriguez@ipsl.fr
+    affiliation: Sorbonne University
+  - given-names: Atef
+    family-names: Ben Nasser
+    email: abennasser@ipsl.fr
+    affiliation: Centre National de Recherche Scientifique
+    orcid: 'https://orcid.org/0000-0001-6948-8735'
+  - given-names: Guillaume
+    family-names: Levavasseur
+    email: glipsl@ipsl.fr
+    affiliation: Sorbonne University
+    orcid: 'https://orcid.org/0000-0002-0801-0890'
+repository-code: 'https://github.com/ESGF/esgf-download.git'
+url: 'https://esgf.github.io/esgf-download/'
+abstract: >-
+  esgpull is a modern ESGF data management tool, bundled
+  with a custom asynchronous interface with the ESGF Search
+  API. It handles scanning, downloading and updating
+  datasets, files and queries from ESGF.
+license: BSD-3-Clause
+commit: 762494e
+version: 0.7.4
+doi: 10.5281/zenodo.14228984
+date-released: '2024-11-27'

{esgpull-0.7.3 → esgpull-0.9.0}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.3
+Metadata-Version: 2.4
 Name: esgpull
-Version: 0.7.3
+Version: 0.9.0
 Summary: ESGF data discovery, download, replication tool
 Project-URL: Repository, https://github.com/ESGF/esgf-download
 Project-URL: Documentation, https://esgf.github.io/esgf-download/
@@ -23,6 +23,7 @@ Requires-Dist: click>=8.1.3
 Requires-Dist: httpx>=0.23.0
 Requires-Dist: myproxyclient>=2.1.0
 Requires-Dist: nest-asyncio>=1.5.6
+Requires-Dist: packaging>=25.0
 Requires-Dist: platformdirs>=2.6.2
 Requires-Dist: pyopenssl>=22.1.0
 Requires-Dist: pyparsing>=3.0.9
@@ -99,3 +100,11 @@ Commands:
   untrack   Untrack queries
   update    Fetch files, link files <-> queries, send files to download...
 ```
+
+## Useful links
+* [ESGF Webinar: An Introduction to esgpull, A Replacement for Synda](https://www.youtube.com/watch?v=xv2RVMd1iCA)
+
+
+## Contributions
+
+You can use the common github workflow (through pull requests and issues) to contribute.

{esgpull-0.7.3 → esgpull-0.9.0}/README.md

@@ -64,3 +64,11 @@ Commands:
   untrack   Untrack queries
   update    Fetch files, link files <-> queries, send files to download...
 ```
+
+## Useful links
+* [ESGF Webinar: An Introduction to esgpull, A Replacement for Synda](https://www.youtube.com/watch?v=xv2RVMd1iCA)
+
+
+## Contributions
+
+You can use the common github workflow (through pull requests and issues) to contribute.

{esgpull-0.7.3 → esgpull-0.9.0}/alembic.ini

@@ -3,12 +3,6 @@ script_location = esgpull/migrations
 prepend_sys_path = .
 version_path_separator = os  # Use os.pathsep. Default configuration used for new projects.
 
-[post_write_hooks]
-hooks = black
-black.type = console_scripts
-black.entrypoint = black
-black.options = -l 79 REVISION_SCRIPT_FILENAME
-
 # Logging configuration
 [loggers]
 keys = root,sqlalchemy,alembic

{esgpull-0.7.3 → esgpull-0.9.0}/docs/docs/configuration.md

@@ -41,6 +41,9 @@ distrib = "false"
 latest = "true"
 replica = "none"
 retracted = "false"
+
+[plugins]
+enabled = "false"
 ```
 
 To modify a config item from the command line, the dot-separated path to that item must
@@ -124,3 +127,13 @@ Certificates are missing.
 
 The credentials will then be saved under the ``~/.esgpull/auth`` directory, within
 ``credentials.toml``, which can then be used for future sessions.
+
+## Plugins
+
+Plugin functionality is disabled by default and can be enabled with:
+
+```shell
+$ esgpull config plugins.enabled true
+```
+
+Plugin-specific configurations are managed separately through the `esgpull plugins config` command and stored in a dedicated `plugins.toml` file. See the [Plugins page](plugins) for more information.

{esgpull-0.7.3 → esgpull-0.9.0}/docs/docs/glossary.md

@@ -3,3 +3,6 @@ __Dataset__
 
 __Facet__
 : basic element of a dataset's _metadata_. Pair of strings in the form `name:value`, equivalent to a python dictionary's item.
+
+__Plugin__
+: custom code that extends `esgpull` functionality by responding to specific events.

{esgpull-0.7.3 → esgpull-0.9.0}/docs/docs/index.md

@@ -10,6 +10,7 @@ It handles scanning, downloading and updating **datasets**, **files** and *queri
 - Simple syntax for fast data exploration
 - Asynchronous download
 - Highly configurable
+- Plugin system for extensibility
 
 !!! tip "Search datasets"
 

esgpull-0.9.0/docs/docs/plugins.md

@@ -0,0 +1,283 @@
+# Plugins
+
+`esgpull` includes a plugin system that allows you to extend functionality by running custom code when specific events occur during data operations.
+
+## Available events
+
+- **file_complete**: A file download completes successfully
+- **file_error**: A file download fails
+- **dataset_complete**: All files from a dataset are downloaded
+
+Each event handler receives only the parameters relevant to that specific event.
+
+## Enable plugins
+
+Plugins are disabled by default. Enable them using the main config command:
+
+```shell
+$ esgpull config plugins.enabled true
+```
+
+```shell
+[plugins]
+enabled = true
+
+Previous value: False
+```
+
+
+## Plugin management
+
+### List available plugins
+
+```shell
+$ esgpull plugins ls
+```
+
+```shell
+        plugin        │      event       │        function         
+══════════════════════╪══════════════════╪═════════════════════════
+ 🟢 notification     │  file_complete │       notify_download   
+                      │ file_error │        notify_error   
+ 🔴 checksum_verify  │  file_complete │      verify_checksum    
+ 🔴 archive_backup   │  file_complete │      backup_file  
+                    │  dataset_complete │        backup_dataset
+```
+
+This detailed information can also be shown with JSON format with `--json`.
+
+### Create a new plugin
+
+Create a plugin template with all available events:
+
+```shell
+$ esgpull plugins create -n notification
+```
+
+```shell
+Plugin template created at: /path/to/esgpull/plugins/notification.py
+Edit the file to implement your custom plugin logic.
+```
+
+Create a plugin for specific events only:
+
+```shell
+$ esgpull plugins create -n notification file_complete file_error
+```
+
+```shell
+Plugin template created at: /path/to/esgpull/plugins/notification.py
+Edit the file to implement your custom plugin logic.
+```
+
+### Enable and disable plugins
+
+```shell
+$ esgpull plugins enable notification
+```
+
+```shell
+Plugin 'notification' enabled.
+```
+
+```shell
+$ esgpull plugins disable notification
+```
+
+```shell
+Plugin 'notification' disabled.
+```
+
+
+### Test plugins
+
+Test plugins by triggering one event with sample data:
+
+```shell
+$ esgpull plugins test file_complete
+```
+
+```shell
+[2025-06-19 17:33:09]  INFO      esgpull.plugins.notification
+✅ Downloaded: tas_Amon_CESM2_historical_r1i1p1f1_gn_185001-201412.nc
+   Size: 524288000 bytes
+
+[2025-06-19 17:33:09]  INFO      esgpull.plugins.checksum_verify
+✓ Checksum verified for tas_Amon_CESM2_historical_r1i1p1f1_gn_185001-201412.nc
+```
+
+This is the primary debugging tool for plugin development. Use it to verify handlers work correctly before running actual downloads or updates.
+
+## Plugin example
+
+Here's a simple notification plugin that sends a message when files are downloaded:
+
+```python title="plugins/notification.py"
+import pathlib
+import logging
+from esgpull.plugin import Event, on
+import esgpull.models
+
+@on(Event.file_complete, priority="normal")
+def notify_download(file: esgpull.models.File, destination: pathlib.Path, logger: logging.Logger):
+    """Send notification when a file is downloaded."""
+    print(f"✅ Downloaded: {file.filename}")
+    print(f"   Size: {file.size} bytes")
+    logger.info(f"Notified download: {file.filename}")
+
+@on(Event.file_error, priority="normal") 
+def notify_error(file: esgpull.models.File, exception: Exception, logger: logging.Logger):
+    """Send notification when a download fails."""
+    print(f"❌ Failed: {file.filename}")
+    print(f"   Error: {exception}")
+    logger.error(f"Notified error: {file.filename}")
+```
+
+## Plugin configuration
+
+Plugins are configured via an optional `Config` class in the plugin code. The `Config` class attributes define parameters and must include default values.
+
+Plugin configuration is stored separately from esgpull's main config file. This file contains a manifest of enabled/disabled plugins and their custom configurations (which override the `Config` class defaults). Use the `esgpull plugins config` subcommand to manage these settings.
+
+Let's extend our notification plugin with configuration:
+
+```python title="plugins/notification.py"
+import pathlib
+import logging
+from esgpull.plugin import Event, on
+import esgpull.models
+
+class Config:
+    enabled = True
+    email_address = "user@example.com"
+    include_size = True
+    error_alerts = True
+
+@on(Event.file_complete, priority="normal")
+def notify_download(file: esgpull.models.File, destination: pathlib.Path, logger: logging.Logger):
+    """Send notification when a file is downloaded."""
+    if Config.enabled:
+        print(f"✅ Downloaded: {file.filename}")
+        if Config.include_size:
+            print(f"   Size: {file.size} bytes")
+        logger.info(f"Notified download to {Config.email_address}: {file.filename}")
+
+@on(Event.file_error, priority="normal") 
+def notify_error(file: esgpull.models.File, exception: Exception, logger: logging.Logger):
+    """Send notification when a download fails."""
+    if Config.enabled and Config.error_alerts:
+        print(f"❌ Failed: {file.filename}")
+        print(f"   Error: {exception}")
+        logger.error(f"Notified error to {Config.email_address}: {file.filename}")
+```
+
+View all plugin configurations:
+```shell
+$ esgpull plugins config
+```
+
+```shell
+─ /path/to/esgpull/plugins.toml ─
+[notification]
+enabled = true
+email_address = "admin@myproject.org"
+include_size = true
+error_alerts = true
+
+[checksum_verify]
+enabled = false
+algorithm = "sha256"
+
+[archive_backup]
+enabled = false
+archive_path = "/backup/esgf"
+```
+
+View specific plugin configuration:
+```shell
+$ esgpull plugins config notification
+```
+
+```shell
+[notification]
+enabled = true
+email_address = "admin@myproject.org"
+include_size = true
+error_alerts = true
+```
+
+View a specific configuration value:
+```shell
+$ esgpull plugins config notification.email_address
+```
+
+```shell
+[notification]
+email_address = "admin@myproject.org"
+```
+
+Set a configuration value:
+```shell
+$ esgpull plugins config notification.email_address alerts@newdomain.com
+```
+
+```shell
+[notification]
+email_address = "alerts@newdomain.com"
+
+Previous value: admin@myproject.org
+```
+
+Reset to default value:
+```shell
+$ esgpull plugins config notification.email_address --default
+```
+
+```shell
+👍 Config reset to default for notification.email_address
+```
+
+### Config class details
+
+The `Config` class supports these data types:
+
+- **Strings** (`str`): Text values like `"INFO"` or `"user@example.com"`
+- **Integers** (`int`): Whole numbers like `3` or `100`  
+- **Floats** (`float`): Decimal numbers like `0.5` or `10.75`
+- **Booleans** (`bool`): `True` or `False` values
+
+```python
+class Config:
+    """Example showing all supported data types"""
+    # String configuration
+    log_level = "INFO"
+    email_address = "user@example.com"
+    
+    # Integer configuration  
+    max_retries = 3
+    timeout_seconds = 30
+    
+    # Float configuration
+    threshold = 0.75
+    delay_factor = 1.5
+    
+    # Boolean configuration
+    notifications_enabled = True
+    debug_mode = False
+```
+
+**Important limitations:**
+- Lists and dictionaries can be used in the `Config` class, but cannot be configured through the CLI `esgpull plugins config` command
+- For list/dictionary attributes, you must edit the `plugins.toml` file directly
+- Only simple scalar values (strings, integers, floats, booleans) can be managed via CLI commands
+- Configuration values are automatically type-checked when modified through CLI
+
+### Configuration management
+
+Plugin configurations are stored separately from the main `esgpull` configuration in a `plugins.toml` file. Only plugins with a `Config` class can be configured.
+
+The configuration system follows these principles:
+
+- **Defaults in code**: Class attributes define default values
+- **Overrides in file**: Only explicitly changed values are saved to `plugins.toml`
+- **Type safety**: Values are automatically converted and validated based on the default type

{esgpull-0.7.3 → esgpull-0.9.0}/docs/docs/quickstart.md

@@ -87,5 +87,17 @@ $ esgpull status
 
 Loop at the [download page](../download) for more information.
 
+## Extending with plugins
+
+`esgpull` can be extended with custom plugins that respond to download events:
+
+```shell
+$ esgpull config plugins.enabled true
+$ esgpull plugins create -n notification file_complete
+$ esgpull plugins enable notification
+```
+
+See the [Plugins page](../plugins) for more details on creating and managing plugins.
+
 
 [ESGF Search API]: https://esgf.github.io/esg-search/ESGF_Search_RESTful_API.html

{esgpull-0.7.3 → esgpull-0.9.0}/docs/mkdocs.yml

@@ -7,6 +7,7 @@ nav:
   # - Queries: queries.md
   - Data discovery: search.md
   - Download: download.md
+  - Plugins: plugins.md
   - Glossary: glossary.md
   # - Usage guide: usage.md
   # - API Reference: reference.md

{esgpull-0.7.3 → esgpull-0.9.0}/esgpull/cli/__init__.py

@@ -7,9 +7,9 @@ from esgpull import __version__
 from esgpull.cli.add import add
 from esgpull.cli.config import config
 from esgpull.cli.convert import convert
-from esgpull.cli.datasets import datasets
 from esgpull.cli.download import download
 from esgpull.cli.login import login
+from esgpull.cli.plugins import plugins
 from esgpull.cli.remove import remove
 from esgpull.cli.retry import retry
 from esgpull.cli.search import search
@@ -35,13 +35,13 @@ SUBCOMMANDS: list[click.Command] = [
     # autoremove,
     config,
     convert,
-    datasets,
     download,
     # facet,
     # get,
     self,
     # install,
     login,
+    plugins,
     remove,
     retry,
     search,

{esgpull-0.7.3 → esgpull-0.9.0}/esgpull/cli/add.py

@@ -89,6 +89,7 @@ def add(
         esg.ui.print(subgraph)
         empty = Query()
         empty.compute_sha()
+        next_steps_queries = []
         for query in queries:
             query.compute_sha()
             esg.graph.resolve_require(query)
@@ -99,7 +100,8 @@ def add(
                 esg.ui.print(f"Skipping existing query: {query.rich_name}")
             else:
                 esg.graph.add(query)
-                esg.ui.print(f"New query added: {query.rich_name}")
+                if query.tracked:
+                    next_steps_queries.append(query)
         new_queries = esg.graph.merge()
         nb = len(new_queries)
         ies = "ies" if nb > 1 else "y"
@@ -107,4 +109,8 @@ def add(
             esg.ui.print(f":+1: {nb} new quer{ies} added.")
         else:
             esg.ui.print(":stop_sign: No new query was added.")
+        if next_steps_queries:
+            esg.ui.print("\nNext steps:\n")
+            for query in next_steps_queries:
+                esg.ui.print(f"\tesgpull update {query.sha[:6]}")
         esg.ui.raise_maybe_record(Exit(0))

{esgpull-0.7.3 → esgpull-0.9.0}/esgpull/cli/config.py

@@ -4,26 +4,11 @@ import click
 from click.exceptions import Abort, BadOptionUsage, Exit
 
 from esgpull.cli.decorators import args, opts
-from esgpull.cli.utils import init_esgpull
+from esgpull.cli.utils import extract_subdict, init_esgpull
 from esgpull.config import ConfigKind
 from esgpull.tui import Verbosity
 
 
-def extract_command(doc: dict, key: str | None) -> dict:
-    if key is None:
-        return doc
-    for part in key.split("."):
-        if not part:
-            raise KeyError(key)
-        elif part in doc:
-            doc = doc[part]
-        else:
-            raise KeyError(part)
-    for part in key.split(".")[::-1]:
-        doc = {part: doc}
-    return doc
-
-
 @click.command()
 @args.key
 @args.value
@@ -73,7 +58,7 @@ def config(
                 )
             kind = esg.config.kind
             old_value = esg.config.update_item(key, value, empty_ok=True)
-            info = extract_command(esg.config.dump(), key)
+            info = extract_subdict(esg.config.dump(), key)
             esg.config.write()
             esg.ui.print(info, toml=True)
             if kind == ConfigKind.NoFile:
@@ -86,12 +71,12 @@ def config(
         elif key is not None:
             if default:
                 old_value = esg.config.set_default(key)
-                info = extract_command(esg.config.dump(), key)
+                info = extract_subdict(esg.config.dump(), key)
                 esg.config.write()
                 esg.ui.print(info, toml=True)
                 esg.ui.print(f"Previous value: {old_value}")
             else:
-                info = extract_command(esg.config.dump(), key)
+                info = extract_subdict(esg.config.dump(), key)
                 esg.ui.print(info, toml=True)
         elif generate:
             overwrite = False
@@ -102,8 +87,7 @@ def config(
                 )
                 esg.ui.raise_maybe_record(Exit(0))
             elif esg.config.kind == ConfigKind.Partial and esg.ui.ask(
-                "A config file already exists,"
-                " fill it with missing defaults?",
+                "A config file already exists, fill it with missing defaults?",
                 default=False,
             ):
                 overwrite = True