rda-python-globus 1.0.0__tar.gz

rda_python_globus-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 NSF National Center for Atmospheric Research
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

rda_python_globus-1.0.0/PKG-INFO

@@ -0,0 +1,119 @@
+Metadata-Version: 2.4
+Name: rda_python_globus
+Version: 1.0.0
+Summary: Tools for managing Globus transfers supporting the NSF NCAR Research Data Archive
+Author-email: Thomas Cram <tcram@ucar.edu>
+Project-URL: Homepage, https://github.com/NCAR/rda-python-globus
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Development Status :: 5 - Production/Stable
+Requires-Python: >=3.7
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: globus_sdk
+Requires-Dist: click
+Requires-Dist: pyyaml
+Requires-Dist: six
+Dynamic: license-file
+
+# dsglobus
+
+This application is a command-line tool for Globus data transfer and management of files 
+archived in the NSF NCAR Research Data Archive.
+
+## Installation
+
+Use the package manager [pip](https://pip.pypa.io/en/stable/) to 
+install `rda_python_globus`.
+
+From within your Python virtual environment:
+- run `pip install git+https://github.com/NCAR/rda-python-globus`
+
+After installation, the cli command `dsglobus` will be available in
+the /bin directory of your virtual environment.
+
+## Command-line usage
+
+The `dsglobus` app is run with the following subcommands.  Each supports a
+`--help/-h` option for details and examples on its usage:
+```
+dsglobus transfer --help
+dsglobus get-task --help
+dsglobus task-list --help
+dsglobus cancel-task --help
+dsglobus ls --help
+dsglobus mkdir --help
+dsglobus rename --help
+dsglobus delete --help
+```
+
+### Example usage
+1. Transfer a single file from the `NCAR RDA GLADE` endpoint to the `NCAR RDA Quasar`
+endpoint:
+```
+$ dsglobus transfer \
+    --source-endpoint rda-glade \
+    --destination-endpoint rda-quasar \
+    --source-file /data/d999009/file.txt \
+    --destination-file /d999009/file.txt
+```
+2. Multiple files can be transferred with a single `dsglobus transfer` call by 
+passing a JSON formatted list of files.  To transfer a batch of files from a JSON file:
+```
+$ dsglobus transfer \
+    --source-endpoint SOURCE_ENDPOINT \
+    --destination-endpoint DESTINATION_ENDPOINT \
+    --batch /path/to/batch.json
+```
+where the contents of `batch.json` is formatted with `source_file/destination_file`
+pairs as:
+```
+{
+    "files": [
+        {"source_file": "/data/d999009/file1.tar", "destination_file": "/d999009/file1.tar"},
+        {"source_file": "/data/d999009/file2.tar", "destination_file": "/d999009/file2.tar"},
+        {"source_file": "/data/d999009/file3.tar", "destination_file": "/d999009/file3.tar"}
+    ]
+}
+```
+
+### Listing contents of a directory on a Globus endpoint
+
+A listing of files on a Globus endpoint can be retrieved via the `dsglobus ls` command.  This
+command supports filtering the results subject to the following rules:
+
+- Filter patterns must start with `--, ~, !`, or `!~`.  If none of these are given, `=` will 
+be used
+- `=` does exact matching
+- `~` does regex matching, supporting globs (`*`)
+- `!` does inverse `=` matching
+- `!~` does inverse `~` matching
+- `~*.txt` matches all `.txt` files, for example
+
+Examples:
+```
+$ dsglobus ls -ep <endpoint> -p <path> --filter '~*.txt'       # all txt files
+$ dsglobus ls -ep <endpoint> -p <path> --filter '!~file1.*'    # not starting in "file1."
+$ dsglobus ls -ep <endpoint> -p <path> --filter '~*ile3.tx*'   # anything with "ile3.tx"
+$ dsglobus ls -ep <endpoint> -p <path> --filter '=file2.txt'   # only "file2.txt"
+$ dsglobus ls -ep <endpoint> -p <path> --filter 'file2.txt'    # same as '=file2.txt'
+$ dsglobus ls -ep <endpoint> -p <path> --filter '!=file2.txt'  # anything but "file2.txt"
+```
+
+## Customizing and extending dsglobus
+
+This app can be modified and adapted to be used on other Globus clients and endpoints with
+minimal effort.  Simply update the client ID, token storage, endpoint IDs, endpoint aliases, and 
+other configuration parameters in `rda_globus_python/lib/config.py` to adapt the app to your use 
+case and specific needs.
+
+## Resources
+
+This app is adapted from the fully featured [Globus Command Line Interface (CLI)](https://docs.globus.org/cli/) 
+and uses the 
+[`TransferClient` class from the Globus SDK](https://globus-sdk-python.readthedocs.io/en/stable/services/transfer.html).
+
+The full [Globus Transfer documentation](https://docs.globus.org/api/transfer/) offers full
+details about the service and reference documentation for all of
+its supported methods and features.

rda_python_globus-1.0.0/README.md

@@ -0,0 +1,100 @@
+# dsglobus
+
+This application is a command-line tool for Globus data transfer and management of files 
+archived in the NSF NCAR Research Data Archive.
+
+## Installation
+
+Use the package manager [pip](https://pip.pypa.io/en/stable/) to 
+install `rda_python_globus`.
+
+From within your Python virtual environment:
+- run `pip install git+https://github.com/NCAR/rda-python-globus`
+
+After installation, the cli command `dsglobus` will be available in
+the /bin directory of your virtual environment.
+
+## Command-line usage
+
+The `dsglobus` app is run with the following subcommands.  Each supports a
+`--help/-h` option for details and examples on its usage:
+```
+dsglobus transfer --help
+dsglobus get-task --help
+dsglobus task-list --help
+dsglobus cancel-task --help
+dsglobus ls --help
+dsglobus mkdir --help
+dsglobus rename --help
+dsglobus delete --help
+```
+
+### Example usage
+1. Transfer a single file from the `NCAR RDA GLADE` endpoint to the `NCAR RDA Quasar`
+endpoint:
+```
+$ dsglobus transfer \
+    --source-endpoint rda-glade \
+    --destination-endpoint rda-quasar \
+    --source-file /data/d999009/file.txt \
+    --destination-file /d999009/file.txt
+```
+2. Multiple files can be transferred with a single `dsglobus transfer` call by 
+passing a JSON formatted list of files.  To transfer a batch of files from a JSON file:
+```
+$ dsglobus transfer \
+    --source-endpoint SOURCE_ENDPOINT \
+    --destination-endpoint DESTINATION_ENDPOINT \
+    --batch /path/to/batch.json
+```
+where the contents of `batch.json` is formatted with `source_file/destination_file`
+pairs as:
+```
+{
+    "files": [
+        {"source_file": "/data/d999009/file1.tar", "destination_file": "/d999009/file1.tar"},
+        {"source_file": "/data/d999009/file2.tar", "destination_file": "/d999009/file2.tar"},
+        {"source_file": "/data/d999009/file3.tar", "destination_file": "/d999009/file3.tar"}
+    ]
+}
+```
+
+### Listing contents of a directory on a Globus endpoint
+
+A listing of files on a Globus endpoint can be retrieved via the `dsglobus ls` command.  This
+command supports filtering the results subject to the following rules:
+
+- Filter patterns must start with `--, ~, !`, or `!~`.  If none of these are given, `=` will 
+be used
+- `=` does exact matching
+- `~` does regex matching, supporting globs (`*`)
+- `!` does inverse `=` matching
+- `!~` does inverse `~` matching
+- `~*.txt` matches all `.txt` files, for example
+
+Examples:
+```
+$ dsglobus ls -ep <endpoint> -p <path> --filter '~*.txt'       # all txt files
+$ dsglobus ls -ep <endpoint> -p <path> --filter '!~file1.*'    # not starting in "file1."
+$ dsglobus ls -ep <endpoint> -p <path> --filter '~*ile3.tx*'   # anything with "ile3.tx"
+$ dsglobus ls -ep <endpoint> -p <path> --filter '=file2.txt'   # only "file2.txt"
+$ dsglobus ls -ep <endpoint> -p <path> --filter 'file2.txt'    # same as '=file2.txt'
+$ dsglobus ls -ep <endpoint> -p <path> --filter '!=file2.txt'  # anything but "file2.txt"
+```
+
+## Customizing and extending dsglobus
+
+This app can be modified and adapted to be used on other Globus clients and endpoints with
+minimal effort.  Simply update the client ID, token storage, endpoint IDs, endpoint aliases, and 
+other configuration parameters in `rda_globus_python/lib/config.py` to adapt the app to your use 
+case and specific needs.
+
+## Resources
+
+This app is adapted from the fully featured [Globus Command Line Interface (CLI)](https://docs.globus.org/cli/) 
+and uses the 
+[`TransferClient` class from the Globus SDK](https://globus-sdk-python.readthedocs.io/en/stable/services/transfer.html).
+
+The full [Globus Transfer documentation](https://docs.globus.org/api/transfer/) offers full
+details about the service and reference documentation for all of
+its supported methods and features.

rda_python_globus-1.0.0/pyproject.toml

@@ -0,0 +1,33 @@
+[build-system]
+requires = [
+  "setuptools>=61.0",
+]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "rda_python_globus"
+version = "1.0.0"
+authors = [
+  { name="Thomas Cram", email="tcram@ucar.edu" },
+]
+description = "Tools for managing Globus transfers supporting the NSF NCAR Research Data Archive"
+readme = "README.md"
+requires-python = ">=3.7"
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+    "Development Status :: 5 - Production/Stable",
+]
+dependencies = [
+    "globus_sdk",
+    "click",
+    "pyyaml",
+    "six",
+]
+
+[project.urls]
+"Homepage" = "https://github.com/NCAR/rda-python-globus"
+
+[project.scripts]
+dsglobus = "rda_python_globus:cli"

rda_python_globus-1.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

rda_python_globus-1.0.0/src/rda_python_globus/__init__.py

@@ -0,0 +1,3 @@
+from .main import cli
+
+__all__ = ("cli",)

rda_python_globus-1.0.0/src/rda_python_globus/file_management.py

@@ -0,0 +1,327 @@
+import sys
+import click
+import textwrap
+import typing as t
+from globus_sdk import DeleteData, GlobusAPIError, NetworkError
+
+from .lib import (
+    common_options,
+    task_submission_options,
+    path_options,
+    endpoint_options,
+    transfer_client,
+    process_json_stream,
+)
+
+import logging
+logger = logging.getLogger(__name__)
+
+def add_batch_to_delete_data(batch, delete_data):
+    """ Add batch of files to delete data object. """
+    delete_files = process_json_stream(batch)
+    for file in delete_files:
+        delete_data.add_item(file)
+
+    return delete_data
+
+@click.command(
+    "mkdir",
+    short_help="Create a directory on a Globus endpoint.",
+    epilog='''
+\b
+=== Examples ===
+\b
+1. Create a directory on the RDA Quasar endpoint:
+\b
+   $ dsglobus mkdir \\
+       --endpoint rda-quasar \\
+       --path /d999009/new_directory
+'''
+)
+@endpoint_options
+@path_options
+@common_options
+def mkdir_command(
+    endpoint: str,
+    path: str,
+) -> None:
+    """
+    Create a directory on a Globus endpoint. Directory path is relative to the endpoint host path.
+    """
+    tc = transfer_client()
+    try:
+        res = tc.operation_mkdir(endpoint, path=path)
+        click.echo(f"{res['message']}")
+    except (GlobusAPIError, NetworkError) as e:
+        logger.error(f"Error creating directory: {e}")
+        raise click.Abort()
+
+@click.command(
+    "rename",
+    short_help="Rename a file or directory on a Globus endpoint.",
+    epilog='''
+\b
+=== Examples ===
+\b
+1. Rename a single file on the RDA Quasar endpoint:
+\b
+   $ dsglobus rename \\
+       --endpoint rda-quasar \\
+       --old-path /d999009/old_file.txt \\
+       --new-path /d999009/new_file.txt
+\b
+2. Rename a directory on the RDA Quasar endpoint:
+\b
+   $ dsglobus rename \\
+       --endpoint rda-quasar \\
+       --old-path /d999009/old_directory \\
+       --new-path /d999009/new_directory
+\b
+3. Rename a batch of files/directories on the RDA Quasar endpoint:
+\b
+   $ dsglobus rename \\
+       --endpoint rda-quasar \\
+       --batch /path/to/batch.json
+\b
+   The batch file should contain a JSON array of file paths to rename.
+\b
+   Example batch file contents:
+\b
+   [
+       {
+           "old_path": "/d999009/file_old.txt",
+           "new_path": "/d999009/file_new.txt"
+       },
+       {
+           "old_path": "/d999009/file2_old.txt",
+           "new_path": "/d999009/file2_new.txt"
+       },
+       {
+           "old_path": "/d999009/old_directory/",
+           "new_path": "/d999009/new_directory/"
+       }
+   ]
+\b
+4. The batch files can also be read from stdin using '-':
+\b
+   $ dsglobus rename \\
+       --endpoint rda-quasar \\
+       --batch -
+   [
+       {
+           "old_path": "/d999009/file_old.txt",
+           "new_path": "/d999009/file_new.txt"
+       },
+       {
+           "old_path": "/d999009/file2_old.txt",
+           "new_path": "/d999009/file2_new.txt"
+       },
+       {
+           "old_path": "/d999009/old_directory/",
+           "new_path": "/d999009/new_directory/"
+       }
+   ]
+   <Ctrl+D>
+'''
+)
+@click.option(
+    "--old-path",
+    "-op",
+    type=str,
+    help="Old file or directory path on the endpoint. Ignored if --batch is used.",
+)
+@click.option(
+    "--new-path",
+    "-np",
+    type=str,
+    help="New file or directory path on the endpoint. Ignored if --batch is used.",
+)
+@click.option(
+    "--batch",
+	type=click.File('r'),
+    help=textwrap.dedent("""\
+        Accept a batch of multiple file/directory name pairs from a file. 
+        Use '-' to read from stdin, and close the stream with 'Ctrl+D'.  
+        See examples below.
+    """),
+)
+@endpoint_options
+@common_options
+def rename_command(
+    endpoint: str,
+    old_path: str,
+    new_path: str,
+    batch: t.TextIO
+) -> None:
+    """
+    Rename a file or directory on a Globus endpoint. Path is relative to the endpoint host path.
+    """
+    if old_path is None and new_path is None and batch is None:
+        raise click.UsageError('--old-path and --new-path, or --batch is required.')
+
+    if batch:
+        files = process_json_stream(batch)
+    else:
+        if old_path is None or new_path is None:
+            raise click.UsageError('--old-path and --new-path are required if --batch is not used.')
+        files = [
+            {
+                "old_path": old_path,
+                "new_path": new_path
+            }
+        ]
+    
+    tc = transfer_client()
+    for file in files:
+        old_path = file["old_path"]
+        new_path = file["new_path"]
+        try:
+            res = tc.operation_rename(endpoint, oldpath=old_path, newpath=new_path)
+            click.echo(f"old path: {old_path}\nnew path: {new_path}\n{res['message']}")
+        except (GlobusAPIError, NetworkError) as e:
+            logger.error(f"Error renaming file/directory: {e}")
+            raise click.Abort()
+
+@click.command(
+    "delete",
+    short_help="Delete files and/or directories on a Globus endpoint.",
+    epilog='''
+\b
+=== Examples ===
+\b
+1. Delete a single file on the RDA Quasar endpoint:
+\b
+   $ dsglobus delete \\
+       --endpoint rda-quasar \\
+       --target-file /d999009/file.txt
+
+\b
+2. Delete a directory on the RDA Quasar endpoint.  --recursive is required
+   to delete a directory and its contents:
+\b
+   $ dsglobus delete \\
+       --endpoint rda-quasar \\
+       --target-file /d999009/dir \\
+       --recursive
+\b
+3. Delete a batch of files/directories on the RDA Quasar endpoint:
+\b
+   $ dsglobus delete \\
+       --endpoint rda-quasar \\
+       --batch /path/to/batch.json \\
+       --recursive
+\b
+   The batch file should contain a JSON array of file paths to delete.
+\b
+   Example batch file contents:
+\b
+   [
+       "/d999009/file1.txt",
+       "/d999009/file2.txt",
+       "/d999009/dir1",
+       "/d999009/dir2"
+   ]
+\b
+4. The batch files can also be read from stdin using '-':
+\b
+   $ dsglobus delete \\
+       --endpoint rda-quasar \\
+       --recursive \\
+       --batch -
+   [
+       "/d999009/file1.txt",
+       "/d999009/file2.txt",
+       "/d999009/dir1",
+       "/d999009/dir2"
+   ]
+   <Ctrl+D>
+'''
+)
+@click.option(
+    "--target-file",
+    "-tf",
+    type=str,
+    help="File or directory to delete on the endpoint. Ignored if --batch is used.",
+)
+@click.option(
+	"--batch",
+	type=click.File('r'),
+    help=textwrap.dedent("""\
+        Accept a batch of files/directories from a file. 
+        Use '-' to read from stdin, and close the stream with 'Ctrl+D'.  
+        See examples below.
+    """),
+)
+@click.option(
+    "--recursive",
+    "-r",
+    is_flag=True,
+    default=False,
+    show_default=True,
+    help="Recursively delete directories and their contents.  Required if deleting a directory.",
+)
+@endpoint_options
+@task_submission_options
+@common_options
+def delete_command(
+    endpoint: str,
+    target_file: str,
+    label: str,
+    batch: t.TextIO,
+    dry_run: bool,
+    recursive: bool,
+) -> None:
+    """
+    Delete files and/or directories on a Globus endpoint. Directory
+    path is relative to the endpoint host path.
+    """
+    tc = transfer_client()
+    delete_data = DeleteData(tc, endpoint, label=label, recursive=recursive)
+
+    # If a batch file is provided, read the file and add to delete data
+    if batch:
+        try:
+            delete_data = add_batch_to_delete_data(batch, delete_data)
+        except ValueError as e:
+            logger.error(f"Error processing batch file: {e}")
+            raise click.Abort()
+    else:
+        if target_file is None:
+            raise click.UsageError('--target-file is required if --batch is not used.')
+        # Add the target file to delete data
+        try:
+            delete_data.add_item(target_file)
+        except ValueError as e:
+            logger.error(f"Error adding target file: {e}")
+            raise click.Abort()
+
+    # If dry run is specified, print the delete data and exit
+    if dry_run:
+        click.echo("Dry run: delete data to be submitted:")
+        data = delete_data.data
+        click.echo(f"Endpoint: {data['endpoint']}")
+        try:
+            click.echo(f"Label: {data['label']}")
+        except KeyError:
+            click.echo("Label: None")
+        click.echo("Files to delete:")
+        for item in data["DATA"]:
+            click.echo(f"  {item}")
+        click.echo("\n")
+
+        # exit safely
+        sys.exit(1)
+
+    # Submit the task
+    try:
+        delete_response = tc.submit_delete(delete_data)
+        task_id = delete_response["task_id"]
+    except (GlobusAPIError, NetworkError) as e:
+        logger.error(f"Error submitting task: {e}")
+        raise click.Abort()
+    click.echo(f'Task ID: {task_id}\n{delete_response["message"]}')
+
+def add_commands(group):
+    group.add_command(mkdir_command)
+    group.add_command(rename_command)
+    group.add_command(delete_command)