data-transfer-cli 0.3.2__py3-none-any.whl

data_transfer_cli-0.3.2.dist-info/METADATA

@@ -0,0 +1,235 @@
+Metadata-Version: 2.3
+Name: data-transfer-cli
+Version: 0.3.2
+Summary: HiDALGO Data Transfer CLI provides commands to transfer data between different data providers and consumers using NIFI pipelines
+License: APL-2.0
+Author: Jesús Gorroñogoitia
+Author-email: jesus.gorronogoitia@eviden.com
+Requires-Python: >=3.11, <4.0
+Classifier: License :: Other/Proprietary License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Dist: hid_data_transfer_lib (>=0.3.2)
+Requires-Dist: paramiko (>=3.3.1)
+Requires-Dist: pyyaml (>=6.0.2,<7.0.0)
+Requires-Dist: requests (>=2.31.0)
+Description-Content-Type: text/markdown
+
+# Hidalgo2 Data Transfer Tool
+This repository contains the implementation of the Hidalgo2 data transfer tool. It uses [Apache NIFI](https://nifi.apache.org/) to transfer data from different data sources to specified targets
+
+## Features
+This tool is planning to support the following features:
+- transfer datasets from Cloud Providers to HDFS
+- transfer datasets from Cloud Providers to CKAN
+- transfer datasets from/to Hadoop HDFS to/from HPC
+- transfer datasets from/to Hadoop HDFS to/from CKAN
+- transfer datasets from/to a CKAN to/from HPC
+- transfer datasets from/to local filesystem to/from CKAN
+
+## Prototype
+Current prototype supports the following features:
+- transfer datasets from/to Hadoop HDFS to/from HPC
+- transfer datasets from/to Hadoop HDFS to/from CKAN
+- transfer datasets from/to a CKAN to/from HPC
+- transfer datasets from/to local filesystem to/from CKAN
+
+
+## Implementation
+Current implementation is based on Python. It is implemented as a CLI that executes a transfer command, by creating a NIFI process group out of the worflow definition reqistered in NIFI registry. It uses the parameters given within the CLI command invocation to populate a NIFI parameter context that is asociated to the created process group. Then, the process group processors are executed once (or until the incoming flowfile queues is empty), one after another, following the group sequence flow, until the flow is completed. To check the status of the transfer command, the CLI offers a check-status command. The Data Transfer CLI tool sends requests to NIFI through its REST API. 
+
+## Requirements
+To use the Data Transfer CLI tool, it is required the following requirements:
+ - **Python3** execution environment
+ - **Poetry** python package management tool
+ - **NIFI** instance, with either a NIFI service or a KEYCLOAK account, plus a NIFI server account (for keys transfer)
+ - **HDFS** instance, with a user Kerberos token (i.e. authenticated Kerberos principal) if required
+ - **CKAN** instance, with an user APIKey
+
+ Python3 and Poetry should be installed in the computer where Data Transfer CLI tool will be used.
+ To install Poetry, follows [this instructions](https://python-poetry.org/docs/#installing-with-the-official-installer)
+
+ For a quick download, setup, configuration and execution of the DTCLI go to section [Quick Deployment, setup, configuration and execution](#quick-deployment-setup-configuration-and-execution)
+
+## CLI configuration
+### Configuration file
+Before using the Data Transfer CLI tool, you should configure it to point at the target NIFI. The configuration file is located at the *src/dtcli.cfg* file.
+
+```
+[Nifi]
+nifi_endpoint=http://localhost:8443
+nifi_upload_folder=/opt/nifi/data/upload
+nifi_download_folder=/opt/nifi/data/download
+nifi_secure_connection=True
+
+[Keycloak]
+keycloak_endpoint=https://idm.hidalgo2.eu
+keycloak_client_id=nifi
+keycloak_client_secret=Tkt8BQmTfUkSceknml6HDSbmGyNRik9V
+```
+
+Under the NIFI section, 
+- We define the url of the NIFI service (*nifi_endpoint*), 
+- We also specify a folder (*nifi_upload_folder*) in NIFI server where to upload files 
+- And another folder (*nifi_download_folder*) where from to download files. These folder must be accessible by the NIFI service (ask NIFI administrator for details). 
+- Additionally, you cat set if NIFI servers listens on a secure HTTPS connection (*nifi_secure_connection*=True) or on a non-secure HTTP (*nifi_secure_connection*=False)
+
+Under the Keycloak section, you can configure the Keycloak integrated with NIFI, specifying:
+- The Keycloak service endpoint (*keycloak_endpoint*)
+- The NIFI client in Keycloak (*keycloak_client*)
+- The NIFI secret in Keycloak (*keycloak_client_secret*)
+
+### NIFI and Keycloak credentials in environment variables
+We must also specify a user account (username, private_key) that grants to upload/download files to the NIFI server (as requested to upload temporary HPC keys or to support local file transfer). This user's account is provided by Hidalgo2 infrastructure provider and it is user's specific. This account is setup in the following environment variables
+- NIFI_SERVER_USERNAME: `export NIFI_SERVER_USERNAME=<nifi_server_username>`
+- NIFI_SERVER_PRIVATE_KEY: `export NIFI_SERVER_PRIVATE_KEY=<path_to_private_key>`
+
+Additionally, a user account granted with access to the NIFI service must be specified, either a
+
+#### A) NIFI User Account
+The NIFI account must be configured in the following environment variables:
+- NIFI_LOGIN: `export NIFI_LOGIN=<nifi_login>`
+- NIFI_PASSWORD: `export NIFI_PASSWORD=<nifi_password>`
+
+This NIFI account is provided by the NIFI administrator, or a
+
+#### B) Keycloak Account with access to NIFI
+The Keycloak account must be configured in the following environment variables:
+- KEYCLOAK_LOGIN: `export KEYCLOAK_LOGIN=<keycloak_login>`
+- KEYCLOAK_PASSWORD: `export KEYCLOAK_PASSWORD=<keycloak_password>`
+
+This Keycloak account is provided by the Keycloak administrator.
+
+## Quick Deployment, setup, configuration and execution
+### From GitLab repository
+1. Clone Data Transfer CLI repository. 
+2. Setup the hid-data_transfer_lib project. 
+  Go to folder *hid-data-management/data-transfer/nifi/hid_data_transfer_lib*. 
+  On the prompt, do: `poetry install & poetry build`
+  Note: this is only required meanwhile this tool is under development, as data-transfer-cli references the local hid_data_transfer_lib and not the one published on Pypi.
+2. Setup the data-transfer-cli project with poetry. 
+  Go to folder *hid-data-management/data-transfer/nifi/data-transfer-cli*. 
+  On the prompt, run `./setup.sh`
+3. Configure your NIFI and Keycloak services, by modifying the default DT CLI configuration (attached to HiDALGO2 NIFI and KEYCLOAK) located at *src/dtcli.cfg*
+4. Edit *setenv.sh*. Provide your accounts for KEYCLOAK and NIFI server. Contact the HiDALGO2 administrator to request them.
+```
+export NIFI_SERVER_USERNAME="<username>"
+export NIFI_SERVER_PRIVATE_KEY="<relative_path_ssh_private_key"
+export KEYCLOAK_LOGIN="<username>"
+export KEYCLOAK_PASSWORD="<password>"
+```
+5. Run Data Transfer CLI tool. In this example, we ask it for help: `dtcli -h`
+
+### From Pipy installation
+To be done
+
+## Usage
+The Data Transfer CLI tool can be executed by invoking the command `dtcli`. Add this command location to your path, either by adding the *data_transfer_cli* folder (when cloned from GitLab) or its location when installed with pip from Pypi:
+
+`./dtcli command <arguments>`
+
+To get help execute:
+
+`./dtcli -h`
+
+obtaining:
+
+```
+usage: ['-h'] [-h]
+              {check-status,hdfs2hpc,hpc2hdfs,ckan2hdfs,hdfs2ckan,ckan2hpc,hpc2ckan,local2ckan,ckan2local}
+              ...
+
+positional arguments:
+  {check-status,hdfs2hpc,hpc2hdfs,ckan2hdfs,hdfs2ckan,ckan2hpc,hpc2ckan,local2ckan,ckan2local}
+                        supported commands to transfer data
+    check-status        check the status of a command
+    hdfs2hpc            transfer data from HDFS to target HPC
+    hpc2hdfs            transfer data from HPC to target HDFS
+    ckan2hdfs           transfer data from CKAN to target HDFS
+    hdfs2ckan           transfer data from HDFS to a target CKAN
+    ckan2hpc            transfer data from CKAN to target HPC
+    hpc2ckan            transfer data from HPC to a target CKAN
+    local2ckan          transfer data from a local filesystem to a target CKAN
+    ckan2local          transfer data from CKAN to a local filesystem
+
+options:
+  -h, --help            show this help message and exit
+```
+
+To get help of a particular command:
+
+`./dtcli hdfs2hpc -h`
+
+obtaining:
+
+```
+usage: ['hdfs2hpc', '-h'] hdfs2hpc [-h] -s DATA_SOURCE [-t DATA_TARGET] [-kpr KERBEROS_PRINCIPAL] [-kp KERBEROS_PASSWORD] -H HPC_HOST [-z HPC_PORT] -u HPC_USERNAME [-p HPC_PASSWORD] [-k HPC_SECRET_KEY] [-P HPC_SECRET_KEY_PASSWORD]
+
+options:
+  -h, --help            show this help message and exit
+  -s DATA_SOURCE, --data-source DATA_SOURCE
+                        HDFS file path
+  -t DATA_TARGET, --data-target DATA_TARGET
+                        [Optional] HPC folder
+  -kpr KERBEROS_PRINCIPAL, --kerberos-principal KERBEROS_PRINCIPAL
+                        [Optional] Kerberos principal (mandatory for a Kerberized HDFS)
+  -kp KERBEROS_PASSWORD, --kerberos-password KERBEROS_PASSWORD
+                        [Optional] Kerberos principal password (mandatory for a Kerberized HDFS)
+  -H HPC_HOST, --hpc-host HPC_HOST
+                        Target HPC ssh host
+  -z HPC_PORT, --hpc-port HPC_PORT
+                        [Optional] Target HPC ssh port
+  -u HPC_USERNAME, --hpc-username HPC_USERNAME
+                        Username for HPC account
+  -p HPC_PASSWORD, --hpc-password HPC_PASSWORD
+                        [Optional] Password for HPC account. Either password or secret key is required
+  -k HPC_SECRET_KEY, --hpc-secret-key HPC_SECRET_KEY
+                        [Optional] Path to HPC secret key. Either password or secret key is required
+  -P HPC_SECRET_KEY_PASSWORD, --hpc-secret-key-password HPC_SECRET_KEY_PASSWORD
+                        [Optional] Password for HPC secret key
+  -2fa, --two-factor-authentication
+                        [Optional] HPC requires 2FA authentication
+```
+
+A common command flow (e.g. transfer data from hdfs to hpc) would be like this:
+
+- execute *hdfs2hcp* CLI command to transfer data from an hdfs location (e.g. /users/yosu/data/genome-tags.csv) to a remote HPC (e.g. LUMI, at $HOME/data folder)
+- check status of *hdfs2hcp* transfer (and possible warnings/errors) with *check-status* CLI command
+
+## Support for HPC clusters that require a 2FA token
+The Data Transfer CLI tool's commands support transferring data to/from HPC clusters that require a 2FA token. These commands offer an optional flag *_2fa*. If set by the user, the command prompts the user (in the standard input) for the token when required. 
+
+## Predefined profiles for data hosts
+To avoid feeding the Data Transfer CLI tool with many inputs decribing the hosts of the source and target data providers/consumers, the user can defined them in the `~/dtcli/config` YAML file, as shown in the following YAML code snippet:
+```
+# Meluxina
+login.lxp.lu:
+   username: u102309 
+   port: 8822
+   secret-key: ~/.ssh/<secret_key>
+   secret-key-password: <password>
+
+# CKAN
+ckan.hidalgo2.eu:
+   api-key: <api-key>
+   organization: atos
+   dataset: test-dataset
+```
+
+where details for Meluxina HPC and CKAN are given. For a HPC cluster, provide the HPC host as key, followed by colon, and below, with identation, any of the hpc parameters described by the Data Tranfer CLI tool help, without the *hpc_* prefix. For instance, if the Data Transfer CLI tool help mentions:
+```
+-u HPC_USERNAME, --hpc-username HPC_USERNAME
+                      Username for HPC account
+``` 
+that is, *--hpc-username* as parameter, use *username* as nested property for the HPC profile's description in the YAML config file, as shown in the example below. Similarly, proceed for other HPC parameters, such as *port*, *password*, *secret-key*, etc.
+The same procedure can be adopted to describe the CKAN host's parameters.
+
+Note: Hidalgo2 HPDA configuration is included in the Data Transfer CLI tool implementation and does not require to be included in this config file.
+
+Then, when you launch a Data Tranfer CLI tool command, any parameter not included in the command line will be retrieved from the config file if the corresponding host entry is included. After that, if the command line gets complete (i.e. all required parameters are provided), the command will be executed, otherwise the corresponding error will be triggered.
+
+
+
+

data_transfer_cli-0.3.2.dist-info/RECORD

@@ -0,0 +1,12 @@
+src/.env,sha256=vGRFD3D5PPwoMBhtPvqN_1alGOHLsNAOJB_3r34BoFA,16
+src/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+src/conf/cli.cfg,sha256=Djxan_HjTkqVJOJZWuXt3DsypAgGVPYMj40YWEtjGFs,206
+src/data_transfer_cli.py,sha256=abM6Dpik9-LHpOQT4_GUqZRiXImICJu0YpcYUrCL7sQ,5646
+src/data_transfer_proxy.py,sha256=4dN5RNT_zJyNUaKGxjvyUw-AMdNVnr3uBpzGXgupko4,13877
+src/dtcli.cfg,sha256=4eWBXiKB5WtG9I6jq6mh1u8ZDb73owmzqpfHAv-2K7s,367
+src/parser/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+src/parser/cli_parser.py,sha256=IxbtaQxs_zqkVBDmLMngJdNR-l8ER5DvO5KtFpowczU,14775
+data_transfer_cli-0.3.2.dist-info/METADATA,sha256=v6hcAsBxa9nk114VDo1MjTczZtozf-_RT1PmfEExwVk,12362
+data_transfer_cli-0.3.2.dist-info/WHEEL,sha256=XbeZDeTWKc1w7CSIyre5aMDU_-PohRwTQceYnisIYYY,88
+data_transfer_cli-0.3.2.dist-info/entry_points.txt,sha256=vQK-69hamV_JVD9AMg1e0HquJCZtjeYxj92ULtYSBso,46
+data_transfer_cli-0.3.2.dist-info/RECORD,,

data_transfer_cli-0.3.2.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: poetry-core 2.1.1
+Root-Is-Purelib: true
+Tag: py3-none-any

data_transfer_cli-0.3.2.dist-info/entry_points.txt

@@ -0,0 +1,3 @@
+[console_scripts]
+cli=data_transfer_cli:main
+

src/.env

@@ -0,0 +1 @@
+PYTHONPATH=./src

src/conf/cli.cfg

@@ -0,0 +1,8 @@
+[Nifi]
+nifi_endpoint=https://nifi.hidalgo2.eu:9443
+
+nifi_server_user_name=yosu
+nifi_server_private_key=~/.ssh/hidalgo2
+nifi_upload_folder=/opt/nifi/data/upload
+nifi_download_folder=/opt/nifi/data/download
+

src/data_transfer_cli.py

@@ -0,0 +1,171 @@
+'''
+Copyright 2024 Eviden
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an 'AS IS' BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+CLI tool for data transfer based on Apache NIFI
+Initial Poc features
+- hdfs2hpc: transfer data from hdfs to target hpc, using sftp processor:
+  - inputs:
+      - hpc_host: hpc frontend hostname
+      - hpc_username: user account name
+      - hpc_secret_key_path: user's secret key location
+      - data-source: HDFS file path
+      - data-target: HPC remote folder
+
+- hpc2hdfs: transfer data file from hpc folder to target hdfs folder,
+using sftp processor:
+  - inputs:
+      - hpc_host: hpc frontend hostname
+      - hpc_username: user account name
+      - hpc_secret_key_path: user's secret key location
+      - data-source: HPC file path
+      - data-target: HDFS remote folder
+
+- ckan2hpc: transfer data from ckan to target hpc,
+    using ckan and sftp processors:
+    - inputs:
+        - ckan_host: CKAN host endpoint
+        - ckan_api_key: CKAN API key
+        - ckan_organization: CKAN organization
+        - ckan_dataset: CKAN dataset
+        - ckan_resource: CKAN resource
+        - hpc_host: hpc frontend hostname
+        - hpc_username: user account name
+        - hpc_secret_key_path: user's secret key location
+        - data-target: HPC remote folder
+
+- hpc2ckan: transfer data from hpc to target ckan,
+    using ckan and sftp processors:
+    - inputs:
+        - ckan_host: CKAN host endpoint
+        - ckan_api_key: CKAN API key
+        - ckan_organization: CKAN organization
+        - ckan_dataset: CKAN dataset
+        - hpc_host: hpc frontend hostname
+        - hpc_username: user account name
+        - hpc_secret_key_path: user's secret key location
+        - data_source: HPC file path
+
+- local2ckan: transfer data from a local filesystem to a target ckan
+    - inputs:
+        - ckan_host: CKAN host endpoint
+        - ckan_api_key: CKAN API key
+        - ckan_organization: CKAN organization
+        - ckan_dataset: CKAN dataset
+        - ckan_resource: CKAN resource to receive the data
+        - data_source: local file path to the data to transfer
+
+- ckan2local: transfer data from ckan to a local filesystem
+    - inputs:
+        - ckan_host: CKAN host endpoint
+        - ckan_api_key: CKAN API key
+        - ckan_organization: CKAN organization
+        - ckan_dataset: CKAN dataset
+        - ckan_resource: CKAN resource to transfer
+        - data_target: local target directory where to transfer the resource
+
+- check-status: check the execution state of a command
+  - inputs:
+      - command_id: uuid of command executed
+        (uuid is reported after command execution)
+
+
+This CLI uses NIFI account to get an access token,
+It uses NIFI REST API to send requests.
+It uses a predefined and installed process group HDSF2HPC template
+with associated parameter context
+'''
+
+import sys
+import os
+import threading
+import traceback
+import warnings
+
+from hid_data_transfer_lib.exceptions.hid_dt_exceptions import HidDataTransferException
+from hid_data_transfer_lib.conf.hid_dt_configuration import HidDataTransferConfiguration
+
+from src.data_transfer_proxy import DataTransferProxy
+from src.parser.cli_parser import CLIParser
+
+
+warnings.filterwarnings("ignore")
+
+
+# Get CLI configuration
+os.environ["HID_DT_CONFIG_FILE"] = \
+    str(os.path.dirname(os.path.realpath(__file__))) + "/dtcli.cfg"
+config = HidDataTransferConfiguration()
+
+# Data Transfer proxy to the library
+dt_proxy = DataTransferProxy(config, True)
+
+
+class ThreadRaisingExceptions(threading.Thread):
+    """Thread class that raises exceptions in the main thread
+    when the thread finishes with an exception"""
+
+    def __init__(self, *args, **kwargs):
+        self._exception = None
+        self._process_group_id = None
+        super().__init__(*args, **kwargs)
+
+    def run(self):
+        try:
+            self._process_group_id = self._target(*self._args, **self._kwargs)
+        except HidDataTransferException as e:
+            self._exception = e
+            raise e
+
+    def join(self, *args, **kwargs):
+        super().join(*args, **kwargs)
+        if self._exception:
+            raise self._exception
+        return self._process_group_id
+
+
+def main(args=None):
+    """Main entry point for the Data Transfer CLI"""
+    if not args:
+        args = sys.argv[1:]
+    # Parse arguments
+    cli_parser = CLIParser(args)
+
+    try:
+        if len(args) == 0:
+            cli_parser.print_help()
+            sys.exit(1)
+
+        # Read user's config file to complete missing arguments with default ones
+        args = cli_parser.fill_missing_args_from_config(args)
+        args = cli_parser.parse_arguments(args, dt_proxy)
+
+        # executes associated command in data_transfer_cli module
+        thread = ThreadRaisingExceptions(target=args.func, args=(args,))
+        thread.start()
+        thread.join()
+    except HidDataTransferException as e:
+        if e.process_group_id():
+            sys.stderr.write(
+                (
+                    f"Got error{e} when executing process group "
+                    f"with id {e.process_group_id()}"
+                )
+            )
+        else:
+            traceback.print_exc(file=sys.stderr)
+            raise e
+
+
+if __name__ == "__main__":
+    main()

src/data_transfer_proxy.py

@@ -0,0 +1,331 @@
+"""
+Copyright 2024 Eviden
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an 'AS IS' BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+
+This module defines the NIFI API client class.
+It provides methods to interface the NIFI server to in instantiate templates,
+and run processors in a process group.
+"""
+
+from hid_data_transfer_lib.exceptions.hid_dt_exceptions import HidDataTransferException
+from hid_data_transfer_lib.conf.hid_dt_configuration import HidDataTransferConfiguration
+from hid_data_transfer_lib.hid_dt_lib import HIDDataTransfer
+
+
+class DataTransferProxy:
+    """interface to the hid_data_transfer_lib to run data transfer commands"""
+
+    def __init__(
+        self, conf: HidDataTransferConfiguration, secure: bool = False
+    ) -> None:
+        """constructs a data transfer client,"""
+        self.__conf = conf
+        self.dt_client = HIDDataTransfer(conf=conf, secure=secure)
+        self.__logger = self.__conf.logger("nifi.v2.client")
+
+    def format_args_to_string(self, args):
+        """Format dtcli command arguments to string for logging"""
+        return " ".join(
+            [
+                f"ckan_host={args.ckan_host}," if hasattr(args, "ckan_host") else "",
+                (
+                    f"ckan_organization={args.ckan_organization},"
+                    if hasattr(args, "ckan_organization")
+                    else ""
+                ),
+                (
+                    f"ckan_dataset={args.ckan_dataset},"
+                    if hasattr(args, "ckan_dataset")
+                    else ""
+                ),
+                (
+                    f"ckan_resource={args.ckan_resource},"
+                    if hasattr(args, "ckan_resource")
+                    else ""
+                ),
+                (
+                    f"data_source={args.data_source},"
+                    if hasattr(args, "data_source")
+                    else ""
+                ),
+                (
+                    f"data_target={args.data_target},"
+                    if hasattr(args, "data_target")
+                    else ""
+                ),
+                f"hpc_host={args.hpc_host}," if hasattr(args, "hpc_host") else "",
+                f"hpc_port={args.hpc_port}," if hasattr(args, "hpc_port") else "",
+                (
+                    f"hpc_username={args.hpc_username},"
+                    if hasattr(args, "hpc_username")
+                    else ""
+                ),
+                (
+                    f"hpc_secret_key={args.hpc_secret_key},"
+                    if hasattr(args, "hpc_secret_key")
+                    else ""
+                ),
+                f"command_id={args.command_id}," if hasattr(args, "command_id") else "",
+            ]
+        )
+
+    # MAIN CLI commands
+
+    def hdfs2hpc(self, args) -> str:
+        """transfer data from HDFS to hpc using SFTP"""
+        self.__logger.info(
+            "executing hdfs2hpc command with args: %s", self.format_args_to_string(args)
+        )
+        try:
+            # Check if 2FA is enabled
+            if args.two_factor_authentication:
+                return self.dt_client.hdfs2hpc_2fa(
+                    hpc_host=args.hpc_host,
+                    hpc_port=args.hpc_port,
+                    hpc_username=args.hpc_username,
+                    hpc_secret_key_path=args.hpc_secret_key,
+                    hpc_secret_key_password=args.hpc_secret_key_password,
+                    data_source=args.data_source,
+                    data_target=args.data_target,
+                    kerberos_principal=args.kerberos_principal,
+                    kerberos_password=args.kerberos_password,
+                )
+            return self.dt_client.hdfs2hpc(
+                hpc_host=args.hpc_host,
+                hpc_port=args.hpc_port,
+                hpc_username=args.hpc_username,
+                hpc_password=args.hpc_password,
+                hpc_secret_key_path=args.hpc_secret_key,
+                hpc_secret_key_password=args.hpc_secret_key_password,
+                data_source=args.data_source,
+                data_target=args.data_target,
+                kerberos_principal=args.kerberos_principal,
+                kerberos_password=args.kerberos_password,
+            )
+
+        except Exception as ex:
+            raise HidDataTransferException(ex) from ex
+
+    def hpc2hdfs(self, args) -> str:
+        """transfer data from HPC to hdfs using SFTP"""
+        self.__logger.info(
+            "executing hpc2hdfs command with args: %s", self.format_args_to_string(args)
+        )
+        try:
+            # Check if 2FA is enabled
+            if args.two_factor_authentication:
+                return self.dt_client.hpc2hdfs_2fa(
+                    hpc_host=args.hpc_host,
+                    hpc_port=args.hpc_port,
+                    hpc_username=args.hpc_username,
+                    hpc_secret_key_path=args.hpc_secret_key,
+                    hpc_secret_key_password=args.hpc_secret_key_password,
+                    data_source=args.data_source,
+                    data_target=args.data_target,
+                    kerberos_principal=args.kerberos_principal,
+                    kerberos_password=args.kerberos_password,
+                )
+            return self.dt_client.hpc2hdfs(
+                    hpc_host=args.hpc_host,
+                    hpc_port=args.hpc_port,
+                    hpc_username=args.hpc_username,
+                    hpc_password=args.hpc_password,
+                    hpc_secret_key_path=args.hpc_secret_key,
+                    hpc_secret_key_password=args.hpc_secret_key_password,
+                    data_source=args.data_source,
+                    data_target=args.data_target,
+                    kerberos_principal=args.kerberos_principal,
+                    kerberos_password=args.kerberos_password,
+                )
+
+        except Exception as ex:
+            raise HidDataTransferException(ex) from ex
+
+    def hdfs2ckan(self, args) -> str:
+        """transfer data from HDFS to CKAN using SFTP"""
+        self.__logger.info(
+            "executing hpc2ckan command with args: %s", self.format_args_to_string(args)
+        )
+        try:
+            return self.dt_client.hdfs2ckan(
+                ckan_host=args.ckan_host
+                if args.ckan_host.startswith("https") else f"https://{args.ckan_host}",
+                ckan_api_key=args.ckan_api_key,
+                ckan_organization=args.ckan_organization,
+                ckan_dataset=args.ckan_dataset,
+                data_source=args.data_source,
+                kerberos_principal=args.kerberos_principal,
+                kerberos_password=args.kerberos_password,
+            )
+
+        except Exception as ex:
+            raise HidDataTransferException(ex) from ex
+
+    def ckan2hdfs(self, args) -> str:
+        """transfer data from CKAN to HPC using SFTP"""
+        self.__logger.info(
+            "executing ckan2hpc command with args: %s", self.format_args_to_string(args)
+        )
+        try:
+            return self.dt_client.ckan2hdfs(
+                ckan_host=args.ckan_host
+                if args.ckan_host.startswith("https") else f"https://{args.ckan_host}",
+                ckan_api_key=args.ckan_api_key,
+                ckan_organization=args.ckan_organization,
+                ckan_dataset=args.ckan_dataset,
+                ckan_resource=args.ckan_resource,
+                data_target=args.data_target,
+                kerberos_principal=args.kerberos_principal,
+                kerberos_password=args.kerberos_password,
+            )
+
+        except Exception as ex:
+            raise HidDataTransferException(ex) from ex
+
+    def hpc2ckan(self, args) -> str:
+        """transfer data from hpc to CKAN using SFTP"""
+        self.__logger.info(
+            "executing hpc2ckan command with args: %s", self.format_args_to_string(args)
+        )
+        try:
+            # Check if 2FA is enabled
+            if args.two_factor_authentication:
+                return self.dt_client.hpc2ckan_2fa(
+                    ckan_host=args.ckan_host
+                    if args.ckan_host.startswith("https")
+                    else f"https://{args.ckan_host}",
+                    ckan_api_key=args.ckan_api_key,
+                    ckan_organization=args.ckan_organization,
+                    ckan_dataset=args.ckan_dataset,
+                    hpc_host=args.hpc_host,
+                    hpc_port=args.hpc_port,
+                    hpc_username=args.hpc_username,
+                    hpc_secret_key_path=args.hpc_secret_key,
+                    hpc_secret_key_password=args.hpc_secret_key_password,
+                    data_source=args.data_source,
+                )
+            return self.dt_client.hpc2ckan(
+                ckan_host=args.ckan_host
+                if args.ckan_host.startswith("https") else f"https://{args.ckan_host}",
+                ckan_api_key=args.ckan_api_key,
+                ckan_organization=args.ckan_organization,
+                ckan_dataset=args.ckan_dataset,
+                hpc_host=args.hpc_host,
+                hpc_port=args.hpc_port,
+                hpc_username=args.hpc_username,
+                hpc_password=args.hpc_password,
+                hpc_secret_key_path=args.hpc_secret_key,
+                hpc_secret_key_password=args.hpc_secret_key_password,
+                data_source=args.data_source,
+            )
+
+        except Exception as ex:
+            raise HidDataTransferException(ex) from ex
+
+    def ckan2hpc(self, args) -> str:
+        """transfer data from CKAN to hpc using SFTP"""
+        self.__logger.info(
+            "executing ckan2hpc command with args: %s", self.format_args_to_string(args)
+        )
+        try:
+            # Check if 2FA is enabled
+            if args.two_factor_authentication:
+                return self.dt_client.ckan2hpc_2fa(
+                    ckan_host=args.ckan_host
+                    if args.ckan_host.startswith("https")
+                    else f"https://{args.ckan_host}",
+                    ckan_api_key=args.ckan_api_key,
+                    ckan_organization=args.ckan_organization,
+                    ckan_dataset=args.ckan_dataset,
+                    ckan_resource=args.ckan_resource,
+                    hpc_host=args.hpc_host,
+                    hpc_port=args.hpc_port,
+                    hpc_username=args.hpc_username,
+                    hpc_secret_key_path=args.hpc_secret_key,
+                    hpc_secret_key_password=args.hpc_secret_key_password,
+                    data_target=args.data_target,
+                )
+            return self.dt_client.ckan2hpc(
+                ckan_host=args.ckan_host
+                if args.ckan_host.startswith("https") else f"https://{args.ckan_host}",
+                ckan_api_key=args.ckan_api_key,
+                ckan_organization=args.ckan_organization,
+                ckan_dataset=args.ckan_dataset,
+                ckan_resource=args.ckan_resource,
+                hpc_host=args.hpc_host,
+                hpc_port=args.hpc_port,
+                hpc_username=args.hpc_username,
+                hpc_password=args.hpc_password,
+                hpc_secret_key_path=args.hpc_secret_key,
+                hpc_secret_key_password=args.hpc_secret_key_password,
+                data_target=args.data_target,
+            )
+
+        except Exception as ex:
+            raise HidDataTransferException(ex) from ex
+
+    def local2ckan(self, args) -> str:
+        """transfer data from local filesystem to CKAN using SFTP"""
+        self.__logger.info(
+            "executing local2ckan command with args: %s",
+            self.format_args_to_string(args),
+        )
+
+        try:
+            return self.dt_client.local2ckan(
+                ckan_host=args.ckan_host
+                if args.ckan_host.startswith("https") else f"https://{args.ckan_host}",
+                ckan_api_key=args.ckan_api_key,
+                ckan_organization=args.ckan_organization,
+                ckan_dataset=args.ckan_dataset,
+                ckan_resource=args.ckan_resource,
+                data_source=args.data_source,
+            )
+
+        except Exception as ex:
+            raise HidDataTransferException(ex) from ex
+
+    def ckan2local(self, args) -> str:
+        """transfer data from CKAN to the local filesystem using SFTP"""
+        self.__logger.info(
+            "executing ckan2local command with args: %s",
+            self.format_args_to_string(args),
+        )
+        try:
+            return self.dt_client.ckan2local(
+                ckan_host=args.ckan_host
+                if args.ckan_host.startswith("https") else f"https://{args.ckan_host}",
+                ckan_api_key=args.ckan_api_key,
+                ckan_organization=args.ckan_organization,
+                ckan_dataset=args.ckan_dataset,
+                ckan_resource=args.ckan_resource,
+                data_target=args.data_target,
+            )
+
+        except Exception as ex:
+            raise HidDataTransferException(ex) from ex
+
+    def check_command_status(self, args):
+        """Checks the status of a CLI command by ID"""
+        # Check process group state by id
+        # This implies to check the execution state of the last processor in the group
+        self.__logger.info(
+            "executing check_command_status command with args: %s",
+            self.format_args_to_string(args),
+        )
+        try:
+            return self.dt_client.check_command_status(args.command_id)
+
+        except Exception as ex:
+            raise HidDataTransferException(ex) from ex

src/dtcli.cfg

@@ -0,0 +1,16 @@
+[Nifi]
+nifi_endpoint=https://nifi.hidalgo2.eu:9443
+nifi_upload_folder=/opt/nifi/data/upload
+nifi_download_folder=/opt/nifi/data/download
+nifi_secure_connection=True
+
+[Keycloak]
+keycloak_endpoint=https://idm.hidalgo2.eu
+keycloak_client_id=nifi
+keycloak_client_secret=Tkt8BQmTfUkSceknml6HDSbmGyNRik9V
+
+[Logging]
+logging_level=INFO
+
+[Network]
+check_status_sleep_lapse=1

src/parser/cli_parser.py

@@ -0,0 +1,370 @@
+"""
+Copyright 2024 Eviden
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an 'AS IS' BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+
+This module provides a parser for the command line arguments.
+It defines the command line arguments that the user can pass to the CLI
+"""
+
+import argparse
+import os
+
+import yaml
+
+
+class CLIParser(argparse.ArgumentParser):
+    """Parser of the command line arguments"""
+
+    def fill_missing_args_from_config(self, args):
+        """Fill missing arguments from the config file"""
+        # Read default dtcli YAML config file from ~/.dtcli/config if exists
+        dtcli_config_file = os.path.expanduser("~/.dtcli/config")
+        if not os.path.exists(dtcli_config_file):
+            return args
+        with open(dtcli_config_file, "r", encoding="utf8") as f:
+            config = yaml.safe_load(f)
+
+            # For each host in command, find it in the config file,
+            # For each host in command, complete the arguments collected
+            # from the config file that are not already set in the command line
+            for host in config.keys():
+                if host in args:
+                    host_index = args.index(host)
+                    if (args[host_index-1] == "-H" or
+                            args[host_index-1] == "--hpc-host"):
+                        self.process_hpc_args(args, config[host])
+                    elif (args[host_index-1] == "-c" or
+                            args[host_index-1] == "--ckan-host"):
+                        self.process_ckan_args(args, config[host])
+                # Kerberos
+                if host.upper() == "KERBEROS" and 'hdfs' in args[0]:
+                    kerberos_config = config[host]
+                    self.process_kerberos_args(args, kerberos_config)
+        return args
+
+    def process_kerberos_args(self, args, kerberos_config):
+        """Process the Kerberos arguments from the config file"""
+        if (("-kpr" not in args and "--kerberos-principal" not in args)
+                and "principal" in kerberos_config):
+            args.append("-kpr")
+            args.append(str(kerberos_config["principal"]))
+        if (("-kp" not in args and "--kerberos-password" not in args)
+                and "password" in kerberos_config):
+            args.append("-kp")
+            args.append(str(kerberos_config["password"]))
+
+    def process_hpc_args(self, args, hpc_config):
+        """Process the HPC arguments from the config file"""
+        if (("-z" not in args and "--hpc-port" not in args)
+                and "port" in hpc_config):
+            args.append("-z")
+            args.append(str(hpc_config["port"]))
+        if (("-u" not in args and "--hpc-username" not in args)
+                and "username" in hpc_config):
+            args.append("-u")
+            args.append(hpc_config["username"])
+        if (("-p" not in args and "--hpc-password" not in args)
+                and "password" in hpc_config):
+            args.append("-p")
+            args.append(hpc_config["password"])
+        if (("-k" not in args and "--hpc-secret-key" not in args)
+                and "secret-key" in hpc_config):
+            args.append("-k")
+            args.append(hpc_config["secret-key"])
+        if (("-P" not in args and "--hpc-secret-key-password" not in args)
+                and "secret-key-password" in hpc_config):
+            args.append("-P")
+            args.append(hpc_config["secret-key-password"])
+
+    def process_ckan_args(self, args, ckan_config):
+        """Process the CKAN arguments from the config file"""
+        if (("-a" not in args and "--ckan-api-key" not in args)
+                and "api-key" in ckan_config):
+            args.append("-a")
+            args.append(ckan_config["api-key"])
+        if (("-o" not in args and "--ckan-organization" not in args)
+                and "organization" in ckan_config):
+            args.append("-o")
+            args.append(ckan_config["organization"])
+        if (("-d" not in args and "--ckan-dataset" not in args)
+                and "dataset" in ckan_config):
+            args.append("-d")
+            args.append(ckan_config["dataset"])
+
+    def add_default_hpc_arguments(self, parser):
+        """Add default HPC arguments to the parser"""
+        parser.add_argument(
+            "-H", "--hpc-host", required=True, help="Target HPC ssh host"
+        )
+        parser.add_argument(
+            "-z", "--hpc-port", required=False, help="[Optional] Target HPC ssh port"
+        )
+        parser.add_argument(
+            "-u", "--hpc-username", required=True, help="Username for HPC account"
+        )
+        parser.add_argument(
+            "-p",
+            "--hpc-password",
+            required=False,
+            help="[Optional] Password for HPC account. "
+            "Either password or secret key is required",
+        )
+        parser.add_argument(
+            "-k",
+            "--hpc-secret-key",
+            required=False,
+            help="[Optional] Path to HPC secret key. "
+            "Either password or secret key is required",
+        )
+        parser.add_argument(
+            "-P",
+            "--hpc-secret-key-password",
+            required=False,
+            help="[Optional] Password for HPC secret key",
+        )
+        return parser
+
+    def add_default_ckan_arguments(self, parser):
+        """Add default CKAN arguments to the parser"""
+        parser.add_argument(
+            "-c", "--ckan-host", required=True, help="CKAN host endpoint"
+        )
+        parser.add_argument(
+            "-a",
+            "--ckan-api-key",
+            required=True,
+            help="CKAN API key",
+        )
+        parser.add_argument(
+            "-o",
+            "--ckan-organization",
+            required=True,
+            help="Identifier of the CKAN organization that hosts \
+            the dataset resource to transfer. \
+            Could be identified by the organization id, name or title",
+        )
+        parser.add_argument(
+            "-d",
+            "--ckan-dataset",
+            required=True,
+            help="Identifier of the CKAN dataset that hosts the resource to transfer. \
+            Could be identified by the dataset id, name or title",
+        )
+        return parser
+
+    def add_default_kerberos_arguments(self, parser):
+        """Add default Kerberos arguments to the parser"""
+        parser.add_argument(
+            "-kpr", "--kerberos-principal",
+            required=False,
+            help="[Optional] Kerberos principal (mandatory for a Kerberized HDFS)"
+        )
+        parser.add_argument(
+            "-kp", "--kerberos-password",
+            required=False,
+            help="[Optional] Kerberos principal password "
+            "(mandatory for a Kerberized HDFS)"
+        )
+        return parser
+
+    def parse_arguments(self, args, target):
+        """parse the command line arguments
+
+        Args:
+            args (list): list of command line arguments
+            target (object): target object to execute the command
+
+        Returns:
+            dict: a dictionary of parsed arguments
+        """
+
+        # commands
+        commands_parsers = self.add_subparsers(
+            help="supported commands to transfer data"
+        )
+        # check status
+        check_status_parser = commands_parsers.add_parser(
+            "check-status", help="check the status of a command"
+        )
+        check_status_parser.add_argument(
+            "-i",
+            "--command_id",
+            required=True,
+            help="id of command to check status",
+        )
+        check_status_parser.set_defaults(func=target.check_command_status)
+
+        # hdfs2hpc
+        hdfs2hpc_parser = commands_parsers.add_parser(
+            "hdfs2hpc", help="transfer data from HDFS to target HPC"
+        )
+        hdfs2hpc_parser.add_argument(
+            "-s", "--data-source",
+            required=True, help="HDFS file path"
+        )
+        hdfs2hpc_parser.add_argument(
+            "-t", "--data-target",
+            required=False, help="[Optional] HPC folder"
+        )
+        hdfs2hpc_parser = self.add_default_kerberos_arguments(hdfs2hpc_parser)
+        hdfs2hpc_parser = self.add_default_hpc_arguments(hdfs2hpc_parser)
+        hdfs2hpc_parser.add_argument(
+            "-2fa", "--two-factor-authentication",
+            required=False, action="store_true", default=False,
+            help="[Optional] HPC requires 2FA authentication"
+        )
+        hdfs2hpc_parser.set_defaults(func=target.hdfs2hpc)
+
+        # hpc2hdfs
+        hpc2hdfs_parser = commands_parsers.add_parser(
+            "hpc2hdfs", help="transfer data from HPC to target HDFS"
+        )
+
+        hpc2hdfs_parser.add_argument(
+            "-s", "--data-source", required=True, help="HPC file path"
+        )
+        hpc2hdfs_parser.add_argument(
+            "-t", "--data-target", required=True, help="HDFS folder"
+        )
+        hpc2hdfs_parser = self.add_default_kerberos_arguments(hpc2hdfs_parser)
+        hpc2hdfs_parser = self.add_default_hpc_arguments(hpc2hdfs_parser)
+        hpc2hdfs_parser.add_argument(
+            "-2fa", "--two-factor-authentication",
+            required=False, action="store_true", default=False,
+            help="[Optional] HPC requires 2FA authentication"
+        )
+        hpc2hdfs_parser.set_defaults(func=target.hpc2hdfs)
+
+        # ckan2hdfs
+        ckan2hdfs_parser = commands_parsers.add_parser(
+            "ckan2hdfs", help="transfer data from CKAN to target HDFS"
+        )
+        ckan2hdfs_parser = self.add_default_ckan_arguments(ckan2hdfs_parser)
+        ckan2hdfs_parser.add_argument(
+            "-r",
+            "--ckan-resource",
+            required=False,
+            help="[Optional] CKAN resource to transfer. \
+            Could be identified by the dataset id or name. \
+            If empty, all resources in the dataset will be transferred. \
+            A regex is also accepted to filter the resources to transfer",
+        )
+        ckan2hdfs_parser.add_argument(
+            "-t", "--data-target", required=False, help="[Optional] target HDFS folder"
+        )
+        ckan2hdfs_parser = self.add_default_kerberos_arguments(ckan2hdfs_parser)
+        ckan2hdfs_parser.set_defaults(func=target.ckan2hdfs)
+
+        # hdfs2ckan
+        hdfs2ckan_parser = commands_parsers.add_parser(
+            "hdfs2ckan", help="transfer data from HDFS to a target CKAN"
+        )
+        hdfs2ckan_parser = self.add_default_ckan_arguments(hdfs2ckan_parser)
+        hdfs2ckan_parser = self.add_default_kerberos_arguments(hdfs2ckan_parser)
+        hdfs2ckan_parser.add_argument(
+            "-s",
+            "--data-source",
+            required=True,
+            help="File path to HDFS file or directory to transfer",
+        )
+        hdfs2ckan_parser.set_defaults(func=target.hdfs2ckan)
+
+        # ckan2hpc
+        ckan2hpc_parser = commands_parsers.add_parser(
+            "ckan2hpc", help="transfer data from CKAN to target HPC"
+        )
+        ckan2hpc_parser = self.add_default_ckan_arguments(ckan2hpc_parser)
+        ckan2hpc_parser.add_argument(
+            "-r",
+            "--ckan-resource",
+            required=False,
+            help="[Optional] CKAN resource to transfer. \
+            Could be identified by the dataset id or name. \
+            If empty, all resources in the dataset will be transferred. \
+            A regex is also accepted to filter the resources to transfer",
+        )
+        ckan2hpc_parser.add_argument(
+            "-t", "--data-target", required=False, help="[Optional] target HPC folder"
+        )
+        ckan2hpc_parser = self.add_default_hpc_arguments(ckan2hpc_parser)
+        ckan2hpc_parser.add_argument(
+            "-2fa", "--two-factor-authentication",
+            required=False, action="store_true", default=False,
+            help="[Optional] HPC requires 2FA authentication"
+        )
+        ckan2hpc_parser.set_defaults(func=target.ckan2hpc)
+
+        # hpc2ckan
+        hpc2ckan_parser = commands_parsers.add_parser(
+            "hpc2ckan", help="transfer data from HPC to a target CKAN"
+        )
+        hpc2ckan_parser = self.add_default_ckan_arguments(hpc2ckan_parser)
+        hpc2ckan_parser = self.add_default_hpc_arguments(hpc2ckan_parser)
+        hpc2ckan_parser.add_argument(
+            "-2fa", "--two-factor-authentication",
+            required=False, action="store_true", default=False,
+            help="[Optional] HPC requires 2FA authentication"
+        )
+        hpc2ckan_parser.add_argument(
+            "-s",
+            "--data-source",
+            required=True,
+            help="File path to HPC file or directory to transfer",
+        )
+        hpc2ckan_parser.set_defaults(func=target.hpc2ckan)
+
+        # local2ckan
+        local2ckan_parser = commands_parsers.add_parser(
+            "local2ckan", help="transfer data from a local filesystem to a target CKAN"
+        )
+        local2ckan_parser = self.add_default_ckan_arguments(local2ckan_parser)
+        local2ckan_parser.add_argument(
+            "-r",
+            "--ckan-resource",
+            required=False,
+            help="[Optional] CKAN resource to create from transferred sources. \
+            If omitted, target resource name will adopt the source file or folder name",
+        )
+
+        local2ckan_parser.add_argument(
+            "-s",
+            "--data-source",
+            required=True,
+            help="File path to local file or directory to transfer",
+        )
+        local2ckan_parser.set_defaults(func=target.local2ckan)
+
+        # ckan2local
+        ckan2local_parser = commands_parsers.add_parser(
+            "ckan2local", help="transfer data from CKAN to a local filesystem"
+        )
+        ckan2local_parser = self.add_default_ckan_arguments(ckan2local_parser)
+        ckan2local_parser.add_argument(
+            "-r",
+            "--ckan-resource",
+            required=False,
+            help="[Optional] CKAN resource to transfer. \
+            If omitted, all resources in the dataset will be transferred",
+        )
+
+        ckan2local_parser.add_argument(
+            "-t",
+            "--data-target",
+            required=False,
+            help="Local directory where to transfer the data. \
+            If omitted, data will be transferred to the current directory",
+        )
+        ckan2local_parser.set_defaults(func=target.ckan2local)
+
+        return self.parse_args(args)