hid_data_transfer_lib 0.4.1__tar.gz → 0.5.1__tar.gz

{hid_data_transfer_lib-0.4.1 → hid_data_transfer_lib-0.5.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hid_data_transfer_lib
-Version: 0.4.1
+Version: 0.5.1
 Summary: HiDALGO Data Transfer library provides methods to transfer data between different data providers and consumers using NIFI pipelines
 License: APL-2.0
 Author: Jesús Gorroñogoitia
@@ -12,16 +12,20 @@ Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
 Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: Python :: 3.15
 Requires-Dist: paramiko (>=3.3.1)
 Requires-Dist: requests (>=2.31.0)
 Requires-Dist: setuptools (>=79.0.0,<80.0.0)
 Description-Content-Type: text/markdown
 
 # Hidalgo2 Data Transfer Lib
+
 This repository contains the implementation of the Hidalgo2 data transfer library. It uses [Apache NIFI](https://nifi.apache.org/) to transfer data from different data sources to specified targets
 
 ## Features
+
 This library is planning to support the following features:
+
 - transfer datasets from Cloud Providers to HDFS
 - transfer datasets from Cloud Providers to CKAN
 - transfer datasets from/to Hadoop HDFS to/from HPC
@@ -31,39 +35,47 @@ This library is planning to support the following features:
 - transfer datasets from/to local filesystem to/from CKAN
 
 ## Current version
+
 Current version of the library supports the following features:
+
+- transfer datasets from Cloud Providers (ECMWF and Meteofrance) to HDFS
+- transfer datasets from Cloud Providers (ECMWF and Meteofrance) to CKAN
 - transfer datasets from/to Hadoop HDFS to/from HPC
 - transfer datasets from/to Hadoop HDFS to/from CKAN
 - transfer datasets from/to a CKAN to/from HPC
 - transfer datasets from/to local filesystem to/from CKAN
 
-
 ## Implementation
+
 This is a Python library that offers specialized API methods to transfer data from data sources to targets. 
-Each API method launches a NIFI pipeline, by instantiating a NIFI process group out of its workflow definition registered in the NIFI registry. 
-It uses the parameters given within the library method invocation to populate a NIFI parameter context that is asociated to the process group. 
+Each API method launches a NIFI pipeline, by instantiating a NIFI process group out of its workflow definition registered in the NIFI registry.
+It uses the parameters given within the library method invocation to populate a NIFI parameter context that is asociated to the process group.
 Then, processors in the process group are executed once (or forever until the incomining processor's flowfile queue gets empty), one after another, following the group sequence flow, until the flow is completed. 
-A processor is executed after the previous one has terminated. To check the status of the transfer command, the library offers another check-status command. 
-Upon termination, the NIFI environment is cleaned up, by removing the created entities (i.e. the process group and its paramenter context). 
-The Data Transfer Library sends requests to NIFI through its REST API. 
+A processor is executed after the previous one has terminated. To check the status of the transfer command, the library offers another check-status command.
+Upon termination, the NIFI environment is cleaned up, by removing the created entities (i.e. the process group and its paramenter context).
+The Data Transfer Library sends requests to NIFI through its REST API.
 
 ## Requirements
+
 To use the Data Transfer library, it is required the following requirements:
- - **Python3** execution environment
- - **Poetry** python package management tool (optional)
- - **NIFI** instance, with a NIFI 2.X server SSH account (for keys transfer). Latest NIFI supported version: 2.8
- - **Keycloak** instance, with a KEYCLOAK user's account
- - **HDFS** instance, with an user's Kerberos principal account
- - **CKAN** instance, with an user APIKey
+
+- **Python3** execution environment
+- **Poetry** python package management tool (optional)
+- **NIFI** instance, with a NIFI 2.X server SSH account (for keys transfer). Latest NIFI supported version: 2.8
+- **Keycloak** instance, with a KEYCLOAK user's account
+- **HDFS** instance, with an user's Kerberos principal account
+- **CKAN** instance, with an user APIKey
 
  Python3 should be installed in the computer where Data Transfer CLI will be used.
  To install Poetry, follows [this instructions](https://python-poetry.org/docs/#installing-with-the-official-installer)
 
 ## Data Transfer lib configuration
+
 ### Configuration file
+
 Before using the Data Transfer library, you should configure it to point at the target NIFI and Keycloak services. The default configuration file is located, by default, at the *data_transfer_cli/conf/hid_dt.cfg* file. However, this default configuration should be complemented (and optionally overriden) with user's specific settings, placed in a configuration file whose location should be also specified (e.g., *~/.dtcli/dtcli.cfg*). Settings in this latter user's configuration will override those in the former default library configuration. The user should not modify the default library configuration, but the the user's specific one, including therein any additional required settings (see below) or modifications to the default ones. The location of this user's specific configuration file is passed as parameter to the library when setting it programmatically.
 
-```
+```script
 [Nifi]
 nifi_endpoint=https://nifi.hidalgo2.eu:9443
 nifi_upload_folder=/opt/nifi/data/upload
@@ -82,7 +94,8 @@ logging_level=INFO
 check_status_sleep_lapse=5
 ```
 
-Under the NIFI section, 
+Under the NIFI section,
+
 - We define the url of the NIFI service (*nifi_endpoint*), 
 - We also specify a folder (*nifi_upload_folder*) in NIFI server where to upload files 
 - And another folder (*nifi_download_folder*) where from to download files. These folder must be accessible by the NIFI service (ask NIFI administrator for details). 
@@ -91,6 +104,7 @@ Under the NIFI section,
 These default library settings works with the HiDALGO2 NIFI, so not additional modifications are required.
 
 Under the Keycloak section, you can configure the Keycloak integrated with NIFI, specifying:
+
 - The Keycloak service endpoint (*keycloak_endpoint*)
 - The NIFI client in Keycloak (*keycloak_client*)
 - The NIFI secret in Keycloak (*keycloak_client_secret*). This setting must be set in the user's  configuration (e.g., *~/.dtcli/dtcli.cfg*).
@@ -113,7 +127,7 @@ HiDALGO2 developers can contact the Keycloak administrator for the *keycloak_cli
 
 Additional user's accounts are specified in the user's specific configuration file (e.g., *~/.dtcli/dtcli.cfg*):
 
-```
+```script
 [Nifi]
 nifi_server_username=<user_name>
 nifi_server_private_key=<path/to/private/key>
@@ -129,16 +143,19 @@ Under the Keycloak section, you must specify your Keycloak account (username and
 
 For HiDALGO2 developers, NIFI (Service, Server) and Keycloak accounts are provided by the HiDALGO2 administrator.
 
-
 ## Usage
+
 The data transfer library can be invoked following two procedures:
 
 ### Using user's configuration (e.g., ~/.dtcli/dtcli.cfg)
+
 In this case, user's configuration will be read from a give file, such as *~/.dtcli/dtcli.cfg*, whose location is programmatically passed as a parameter upon the setup of the library (see procedure below).
 
 ### Providing configuration in a dictionary
+
 In this case, the user's configuration is provided in a dictionary, with this structure:
-```
+
+```script
 {
   'Nifi': 
     {
@@ -158,6 +175,7 @@ In this case, the user's configuration is provided in a dictionary, with this st
   }
 }
 ```
+
 In this settings' dictionary you should add the user's specific accounts for Nifi and Keycloak, and optionally, other settings, as shown for the logging level or the sleep lapse time for checking the processors status on the Nifi pipeline. This dictionary is programmatically passed as parameter to the library upon its setup (see procedure below).
 
 The remaining procedure goes as follows:
@@ -168,7 +186,7 @@ The remaining procedure goes as follows:
 
 Example with user's configuration file
 
-```
+```script
 from hid_data_transfer_lib.hid_dt_lib import HIDDataTransfer
 from hid_data_transfer_lib.conf.hid_dt_configuration import (
     HidDataTransferConfiguration
@@ -207,7 +225,8 @@ dt_client = HIDDataTransfer(
 ```
 
 Example with user's configuration dictionary:
-```
+
+```script
 user_config_dict = {...} # See example of dictionary given above
 config = HidDataTransferConfiguration().configure(
     user_config_dict=user_config_dict,
@@ -215,9 +234,9 @@ config = HidDataTransferConfiguration().configure(
 dt_client = HIDDataTransfer(conf=config, secure=True)
 ```
 
-
 - Invoke any data transfer library method using the created object to tranfer data
-```
+
+```script
 pipeline_id, accounting = dt_client.hpc2ckan(
   hpc_host=<hpc_endpoint>,
   hpc_username=<hpc_username>,
@@ -232,17 +251,22 @@ pipeline_id, accounting = dt_client.hpc2ckan(
   recursive=False,
 )
 ```
+
 This method returns the id of the data transfer pipeline that NIFI executed and an AccountingInfo object that contains statistics of data transfer, including the *pipeline_timespan* (or total transfer time) and the flowfiles_sizes, a dictionary whose keys are the names of the transferred files and the values are their lengths.
 
 ## Data Transfer optimization
+
 You can improve the data transfer rate by setting the optional parameter concurrent_tasks (*integer*) to the number of concurrent tasks that will be used in the NIFI pipeline (default is 1). The maximum number of tasks that improve the transfer throughput depends on the physical resources of the NIFI server (consult its administrator). The parallel transfer is currently supported to/from HPC and HDFS data servers, but not to/from CKAN (under development)
 
 ## Support for HPC clusters that require a 2FA token
+
 This library includes methods (suffixed as _2fa) to transfer data to/from HPC clusters that require a 2FA token. These methods offer an optional parameter *callback_2fa* that points to a method that should return (as str) the 2FA token when invoked by the library. If not set by the method caller, these methods call a default implementation that prompts the user (in the standard input) for the token. 
 
 ## Data transfer process with NIFI
+
 The following UML Sequence Diagram describes the data transfer process for each command, for instance *ckan2hpc*, leveraging the associated NIFI pipeline.
 The Data Transfer (DT) Consumer (a client of this library) invokes a *ckan2hpc* command by following these steps:
+
 - Creates an instance of HidDataTransferConfiguration, which reads the file and environment configuration (see [Installation Instructions](#data-transfer-lib-configuration)).
 - Creates an instance of HIDDataTransfer with that configuration object, with secure mode activated, and by passing a dictionary with Keycloak token details. A renewable Keycloak token is required to invoke the remote NIFI REST APIs.
 

{hid_data_transfer_lib-0.4.1 → hid_data_transfer_lib-0.5.1}/README.md

@@ -1,8 +1,11 @@
 # Hidalgo2 Data Transfer Lib
+
 This repository contains the implementation of the Hidalgo2 data transfer library. It uses [Apache NIFI](https://nifi.apache.org/) to transfer data from different data sources to specified targets
 
 ## Features
+
 This library is planning to support the following features:
+
 - transfer datasets from Cloud Providers to HDFS
 - transfer datasets from Cloud Providers to CKAN
 - transfer datasets from/to Hadoop HDFS to/from HPC
@@ -12,39 +15,47 @@ This library is planning to support the following features:
 - transfer datasets from/to local filesystem to/from CKAN
 
 ## Current version
+
 Current version of the library supports the following features:
+
+- transfer datasets from Cloud Providers (ECMWF and Meteofrance) to HDFS
+- transfer datasets from Cloud Providers (ECMWF and Meteofrance) to CKAN
 - transfer datasets from/to Hadoop HDFS to/from HPC
 - transfer datasets from/to Hadoop HDFS to/from CKAN
 - transfer datasets from/to a CKAN to/from HPC
 - transfer datasets from/to local filesystem to/from CKAN
 
-
 ## Implementation
+
 This is a Python library that offers specialized API methods to transfer data from data sources to targets. 
-Each API method launches a NIFI pipeline, by instantiating a NIFI process group out of its workflow definition registered in the NIFI registry. 
-It uses the parameters given within the library method invocation to populate a NIFI parameter context that is asociated to the process group. 
+Each API method launches a NIFI pipeline, by instantiating a NIFI process group out of its workflow definition registered in the NIFI registry.
+It uses the parameters given within the library method invocation to populate a NIFI parameter context that is asociated to the process group.
 Then, processors in the process group are executed once (or forever until the incomining processor's flowfile queue gets empty), one after another, following the group sequence flow, until the flow is completed. 
-A processor is executed after the previous one has terminated. To check the status of the transfer command, the library offers another check-status command. 
-Upon termination, the NIFI environment is cleaned up, by removing the created entities (i.e. the process group and its paramenter context). 
-The Data Transfer Library sends requests to NIFI through its REST API. 
+A processor is executed after the previous one has terminated. To check the status of the transfer command, the library offers another check-status command.
+Upon termination, the NIFI environment is cleaned up, by removing the created entities (i.e. the process group and its paramenter context).
+The Data Transfer Library sends requests to NIFI through its REST API.
 
 ## Requirements
+
 To use the Data Transfer library, it is required the following requirements:
- - **Python3** execution environment
- - **Poetry** python package management tool (optional)
- - **NIFI** instance, with a NIFI 2.X server SSH account (for keys transfer). Latest NIFI supported version: 2.8
- - **Keycloak** instance, with a KEYCLOAK user's account
- - **HDFS** instance, with an user's Kerberos principal account
- - **CKAN** instance, with an user APIKey
+
+- **Python3** execution environment
+- **Poetry** python package management tool (optional)
+- **NIFI** instance, with a NIFI 2.X server SSH account (for keys transfer). Latest NIFI supported version: 2.8
+- **Keycloak** instance, with a KEYCLOAK user's account
+- **HDFS** instance, with an user's Kerberos principal account
+- **CKAN** instance, with an user APIKey
 
  Python3 should be installed in the computer where Data Transfer CLI will be used.
  To install Poetry, follows [this instructions](https://python-poetry.org/docs/#installing-with-the-official-installer)
 
 ## Data Transfer lib configuration
+
 ### Configuration file
+
 Before using the Data Transfer library, you should configure it to point at the target NIFI and Keycloak services. The default configuration file is located, by default, at the *data_transfer_cli/conf/hid_dt.cfg* file. However, this default configuration should be complemented (and optionally overriden) with user's specific settings, placed in a configuration file whose location should be also specified (e.g., *~/.dtcli/dtcli.cfg*). Settings in this latter user's configuration will override those in the former default library configuration. The user should not modify the default library configuration, but the the user's specific one, including therein any additional required settings (see below) or modifications to the default ones. The location of this user's specific configuration file is passed as parameter to the library when setting it programmatically.
 
-```
+```script
 [Nifi]
 nifi_endpoint=https://nifi.hidalgo2.eu:9443
 nifi_upload_folder=/opt/nifi/data/upload
@@ -63,7 +74,8 @@ logging_level=INFO
 check_status_sleep_lapse=5
 ```
 
-Under the NIFI section, 
+Under the NIFI section,
+
 - We define the url of the NIFI service (*nifi_endpoint*), 
 - We also specify a folder (*nifi_upload_folder*) in NIFI server where to upload files 
 - And another folder (*nifi_download_folder*) where from to download files. These folder must be accessible by the NIFI service (ask NIFI administrator for details). 
@@ -72,6 +84,7 @@ Under the NIFI section,
 These default library settings works with the HiDALGO2 NIFI, so not additional modifications are required.
 
 Under the Keycloak section, you can configure the Keycloak integrated with NIFI, specifying:
+
 - The Keycloak service endpoint (*keycloak_endpoint*)
 - The NIFI client in Keycloak (*keycloak_client*)
 - The NIFI secret in Keycloak (*keycloak_client_secret*). This setting must be set in the user's  configuration (e.g., *~/.dtcli/dtcli.cfg*).
@@ -94,7 +107,7 @@ HiDALGO2 developers can contact the Keycloak administrator for the *keycloak_cli
 
 Additional user's accounts are specified in the user's specific configuration file (e.g., *~/.dtcli/dtcli.cfg*):
 
-```
+```script
 [Nifi]
 nifi_server_username=<user_name>
 nifi_server_private_key=<path/to/private/key>
@@ -110,16 +123,19 @@ Under the Keycloak section, you must specify your Keycloak account (username and
 
 For HiDALGO2 developers, NIFI (Service, Server) and Keycloak accounts are provided by the HiDALGO2 administrator.
 
-
 ## Usage
+
 The data transfer library can be invoked following two procedures:
 
 ### Using user's configuration (e.g., ~/.dtcli/dtcli.cfg)
+
 In this case, user's configuration will be read from a give file, such as *~/.dtcli/dtcli.cfg*, whose location is programmatically passed as a parameter upon the setup of the library (see procedure below).
 
 ### Providing configuration in a dictionary
+
 In this case, the user's configuration is provided in a dictionary, with this structure:
-```
+
+```script
 {
   'Nifi': 
     {
@@ -139,6 +155,7 @@ In this case, the user's configuration is provided in a dictionary, with this st
   }
 }
 ```
+
 In this settings' dictionary you should add the user's specific accounts for Nifi and Keycloak, and optionally, other settings, as shown for the logging level or the sleep lapse time for checking the processors status on the Nifi pipeline. This dictionary is programmatically passed as parameter to the library upon its setup (see procedure below).
 
 The remaining procedure goes as follows:
@@ -149,7 +166,7 @@ The remaining procedure goes as follows:
 
 Example with user's configuration file
 
-```
+```script
 from hid_data_transfer_lib.hid_dt_lib import HIDDataTransfer
 from hid_data_transfer_lib.conf.hid_dt_configuration import (
     HidDataTransferConfiguration
@@ -188,7 +205,8 @@ dt_client = HIDDataTransfer(
 ```
 
 Example with user's configuration dictionary:
-```
+
+```script
 user_config_dict = {...} # See example of dictionary given above
 config = HidDataTransferConfiguration().configure(
     user_config_dict=user_config_dict,
@@ -196,9 +214,9 @@ config = HidDataTransferConfiguration().configure(
 dt_client = HIDDataTransfer(conf=config, secure=True)
 ```
 
-
 - Invoke any data transfer library method using the created object to tranfer data
-```
+
+```script
 pipeline_id, accounting = dt_client.hpc2ckan(
   hpc_host=<hpc_endpoint>,
   hpc_username=<hpc_username>,
@@ -213,17 +231,22 @@ pipeline_id, accounting = dt_client.hpc2ckan(
   recursive=False,
 )
 ```
+
 This method returns the id of the data transfer pipeline that NIFI executed and an AccountingInfo object that contains statistics of data transfer, including the *pipeline_timespan* (or total transfer time) and the flowfiles_sizes, a dictionary whose keys are the names of the transferred files and the values are their lengths.
 
 ## Data Transfer optimization
+
 You can improve the data transfer rate by setting the optional parameter concurrent_tasks (*integer*) to the number of concurrent tasks that will be used in the NIFI pipeline (default is 1). The maximum number of tasks that improve the transfer throughput depends on the physical resources of the NIFI server (consult its administrator). The parallel transfer is currently supported to/from HPC and HDFS data servers, but not to/from CKAN (under development)
 
 ## Support for HPC clusters that require a 2FA token
+
 This library includes methods (suffixed as _2fa) to transfer data to/from HPC clusters that require a 2FA token. These methods offer an optional parameter *callback_2fa* that points to a method that should return (as str) the 2FA token when invoked by the library. If not set by the method caller, these methods call a default implementation that prompts the user (in the standard input) for the token. 
 
 ## Data transfer process with NIFI
+
 The following UML Sequence Diagram describes the data transfer process for each command, for instance *ckan2hpc*, leveraging the associated NIFI pipeline.
 The Data Transfer (DT) Consumer (a client of this library) invokes a *ckan2hpc* command by following these steps:
+
 - Creates an instance of HidDataTransferConfiguration, which reads the file and environment configuration (see [Installation Instructions](#data-transfer-lib-configuration)).
 - Creates an instance of HIDDataTransfer with that configuration object, with secure mode activated, and by passing a dictionary with Keycloak token details. A renewable Keycloak token is required to invoke the remote NIFI REST APIs.
 

{hid_data_transfer_lib-0.4.1 → hid_data_transfer_lib-0.5.1}/hid_data_transfer_lib/hid_dt_lib.py

@@ -22,7 +22,9 @@ import sys
 from typing import Any, Dict, Optional, Callable
 import uuid
 import os
+import json
 from types import SimpleNamespace
+from pathlib import Path
 
 from hid_data_transfer_lib.conf.hid_dt_configuration import (
     HidDataTransferConfiguration
@@ -46,6 +48,8 @@ HPC_2FA = "hpc.2fa"
 NIFI_DOWNLOAD_FOLDER = "nifi.download_folder"
 RECURSIVE = "recursive"
 
+PYTHON_PATH_ECMWF = "/opt/nifi/data/scripts/ecmwf/nenv/bin/python"
+
 HPC_ACCOUNT_MSG = "Either hpc_password or hpc_secret_key must be provided"
 
 
@@ -70,7 +74,7 @@ class HIDDataTransfer:
         """
         self.__conf = conf
         self.__secure = secure
-        self.use_keycloak_account = (keycloak_token is None)
+        self.use_keycloak_account = keycloak_token is None
         self.__nifi_client = NIFIClient().configure(
             conf, secure, use_keycloak_account=self.use_keycloak_account)
         self.__logger = self.__conf.logger("hid_data_transfer_lib")
@@ -93,21 +97,31 @@ class HIDDataTransfer:
             name = name.replace("?", ".")
         return name
 
-    def generate_ckan_resource_name(self, args):
+    def generate_ckan_resource_name(self, args, prefix_zip:bool = True):
         '''computes the ckan resource name'''
         ckan_resource = args.ckan_resource
         if ckan_resource:
-            ckan_resource = ckan_resource + ".zip" \
+            # Process wildcards
+            ckan_resource = ckan_resource.replace("*", ".*")
+            ckan_resource = ckan_resource.replace("?", ".")
+
+            if prefix_zip and not ckan_resource.endswith(".zip"):
+                ckan_resource = ckan_resource + ".zip" \
                     if not ckan_resource.endswith(".zip") \
                     else ckan_resource
         else:
-            ckan_resource = (
-                    args.data_source.replace("/", "_")[1:]
-                    if not args.data_source.endswith("/")
-                    else args.data_source[:-1].replace("/", "_")
+            if not hasattr(args, "data_source"):
+                ckan_resource = ".*"
+            else:
+                start_idx = 1 if args.data_source.startswith("/") else 0
+                ends_with_slash = args.data_source.endswith("/")
+                ckan_resource = (
+                    args.data_source[start_idx:].replace("/", "_")[0:]
+                        if not ends_with_slash 
+                    else args.data_source[start_idx:-1].replace("/", "_")
                 )
-            if not ckan_resource.endswith(".zip"):
-                ckan_resource += ".zip"
+                if prefix_zip and not ckan_resource.endswith(".zip"):
+                    ckan_resource += ".zip"
         return ckan_resource
 
     # MAIN API methods
@@ -561,6 +575,7 @@ class HIDDataTransfer:
         )
         try:
             hpc_filename = args.data_source[args.data_source.rfind("/") + 1 :]
+            hpc_filename = self.process_wildcards(hpc_filename)
             arguments: Dict[str, Any] = {}
             self.__nifi_client.add_default_hpc_parameters(arguments, args)
             arguments[HPC_FILENAME] = hpc_filename if hpc_filename else ".*"
@@ -1070,7 +1085,7 @@ class HIDDataTransfer:
             arguments: Dict[str, Any] = {}
             self.__nifi_client.add_default_ckan_parameters(arguments, args)
             arguments[self.__nifi_client.CKAN_RESOURCE] = \
-                self.process_wildcards(args.ckan_resource)
+                self.generate_ckan_resource_name(args, prefix_zip=False)
 
             # If multiple ckan resource files are selected, data_target represents a folder
             # Else If data_target ends with / represents a folder, otherwise a file
@@ -1206,7 +1221,7 @@ class HIDDataTransfer:
             self.__nifi_client.add_default_hpc_parameters(arguments, args)
             self.__nifi_client.add_default_ckan_parameters(arguments, args)
             arguments[self.__nifi_client.CKAN_RESOURCE] = \
-                self.process_wildcards(args.ckan_resource)
+                self.generate_ckan_resource_name(args, prefix_zip=False)
 
             # If multiple ckan resource files are selected, data_target represents a folder
             # Else If data_target ends with / represents a folder, otherwise a file
@@ -1405,7 +1420,7 @@ class HIDDataTransfer:
             arguments[HPC_TARGET_FOLDER] = args.data_target
             self.__nifi_client.add_default_ckan_parameters(arguments, args)
             arguments[self.__nifi_client.CKAN_RESOURCE] = \
-                self.process_wildcards(args.ckan_resource)
+                self.generate_ckan_resource_name(args, prefix_zip=False)
 
             keys_arguments = {
                 self.__nifi_client.HPC_SECRET_KEY_PATH: arguments[
@@ -2247,6 +2262,417 @@ class HIDDataTransfer:
             if download_folder is not None:
                 self.__nifi_client.remove_download_folder(download_folder)
 
+    def aggregate(self, accountings: list[AccountingInfo]) -> AccountingInfo:
+        """Aggregate the accounting information from multiple process groups"""
+        aggregated_accounting = AccountingInfo(
+            pipeline_timespan = 0,
+            flowfiles_sizes = {}
+        )
+        for accounting in accountings:
+            aggregated_accounting.pipeline_timespan += accounting.pipeline_timespan
+            aggregated_accounting.flowfiles_sizes.update(accounting.flowfiles_sizes)
+
+        return aggregated_accounting
+
+
+    def meteofrance2ckan(
+        self,
+        meteofrance_config,
+        meteofrance_apikey,
+        ckan_host,
+        ckan_api_key,
+        ckan_organization,
+        ckan_dataset
+    ) -> tuple[str, AccountingInfo]:
+        """
+        transfer data from MeteoFrance service to CKAN
+
+        Args:
+            ckan_host (str): The CKAN host URL.
+            ckan_organization (str): The CKAN organization name.
+            ckan_dataset (str): The CKAN dataset name.
+            # ckan_resource (str): The name of the CKAN resource
+            #     to be created with the data.
+            meteofrance_config (str): path to json file with request configuration
+            meteofrance_apikey (str): The apikey for MeteoFrance
+
+        Returns:
+            tuple[str, AccountingInfo]: The result of the process group execution.
+
+        Raises:
+            HidDataTransferException: If an error occurs during
+            the data transfer process.
+
+        """
+
+        args = SimpleNamespace(
+            ckan_host=ckan_host,
+            ckan_api_key=ckan_api_key,
+            ckan_organization=ckan_organization,
+            ckan_dataset=ckan_dataset,
+            meteofrance_config=meteofrance_config,
+            meteofrance_apikey=meteofrance_apikey,
+        )
+
+        self.__logger.info(
+            "executing meteofrance2ckan command with args: %s",
+            self.__nifi_client.format_args_to_string(args),
+        )
+        try:
+            arguments: Dict[str, Any] = {}
+
+            with open(meteofrance_config, encoding="utf-8") as config:
+                mtf_config = json.load(config)
+
+            arguments['meteofrance.url'] = mtf_config['url']
+            coverages = mtf_config['coverages']
+            process_ids = []
+            accountings = []
+            for entry in coverages:
+                arguments['meteofrance.coverageid'] = '&coverageid=' + entry['coverageid']
+                arguments['meteofrance.subsets'] = \
+                    ''.join(['&subset=' + i for i in entry['subsets']])
+                time = [i for i in entry['subsets'] if 'time' in i][0].strip('time(').strip(')')
+
+                args.ckan_resource = entry['coverageid'] + '_' + time + '.tif'
+                self.__nifi_client.add_default_ckan_parameters(arguments, args)
+                arguments[self.__nifi_client.CKAN_RESOURCE] = \
+                    self.generate_ckan_resource_name(args, False)
+
+                keys_arguments: Dict[str, Any] = {}
+
+                processors_with_meteofrance_sensitive_parameters = ["InvokeHTTP"]
+                sensitive_parameters = \
+                    self.__nifi_client.set_sensitive_meteofrance_parameters(
+                        processors_with_meteofrance_sensitive_parameters, args
+                    )
+
+                processors_with_ckan_sensitive_parameters = ["CKANFileUploader"]
+                sensitive_parameters.update(
+                    self.__nifi_client.set_sensitive_ckan_parameters(
+                        processors_with_ckan_sensitive_parameters, args
+                    )
+                )
+
+                execution_type = {
+                    "InvokeHTTP": ExecutionType.ONCE,
+                    "UpdateAttribute": ExecutionType.ONCE,
+                    "CKANFileUploader": ExecutionType.ONCE,
+                }
+
+                process_id, accounting = self.__nifi_client.run_process_group(
+                    "meteofrance2ckan",
+                    execution_type,
+                    arguments,
+                    keys_arguments,
+                    sensitive_parameters,
+                    keycloak_token=self.__keycloak_token,
+                    accounting_target="CKANFileUploader",
+                )
+                process_ids.append(process_id)
+                accountings.append(accounting)
+
+            return '|'.join(process_ids), self.aggregate(accountings)
+
+        except Exception as ex:
+            raise HidDataTransferException(ex) from ex
+
+    def meteofrance2hdfs(
+        self,
+        meteofrance_config,
+        meteofrance_apikey,
+        data_target,
+        kerberos_principal=None,
+        kerberos_password=None
+    ) -> tuple[str, AccountingInfo]:
+        """
+        transfer data from MeteoFrance service to CKAN
+
+        Args:
+            kerberos_principal (str): The kerberos principal.
+            kerberos_password (str): The kerberos password.
+            data_target (str): The target path in HDFS.
+            data_source (str): The filename in HDFS.
+            meteofrance_config (str): path to json file with request configuration
+            meteofrance_apikey (str): The apikey for MeteoFrance
+
+        Returns:
+            tuple[str, AccountingInfo]: The result of the process group execution.
+
+        Raises:
+            HidDataTransferException: If an error occurs during
+            the data transfer process.
+
+        """
+
+        args = SimpleNamespace(
+            meteofrance_config=meteofrance_config,
+            meteofrance_apikey=meteofrance_apikey,
+            data_target=data_target,
+            kerberos_principal=kerberos_principal,
+            kerberos_password=kerberos_password,
+        )
+
+        processors_with_kerberos = ["PutHDFS"]
+        kerberos_account = SimpleNamespace(
+            kerberos_principal=kerberos_principal,
+            kerberos_password=kerberos_password,
+        )
+
+        self.__logger.info(
+            "executing meteofrance2hdfs command with args: %s",
+            self.__nifi_client.format_args_to_string(args),
+        )
+        try:
+            arguments: Dict[str, Any] = {}
+            arguments[HDFS_DATA_FOLDER] = args.data_target
+
+            with open(meteofrance_config, encoding="utf-8") as config:
+                mtf_config = json.load(config)
+
+            arguments['meteofrance.url'] = mtf_config['url']
+            coverages = mtf_config['coverages']
+            process_ids = []
+            accountings = []
+            for entry in coverages:
+                arguments['meteofrance.coverageid'] = '&coverageid=' + entry['coverageid']
+                arguments['meteofrance.subsets'] = \
+                    ''.join(['&subset=' + i for i in entry['subsets']])
+                time = [i for i in entry['subsets'] if 'time' in i][0].strip('time(').strip(')')
+                filename = entry['coverageid'] + '_' + time + '.tif'
+                arguments[HDFS_FILENAME] = filename.replace(':', '_')
+
+                keys_arguments: Dict[str, Any] = {}
+
+                processors_with_meteofrance_sensitive_parameters = ["InvokeHTTP"]
+                sensitive_parameters = \
+                    self.__nifi_client.set_sensitive_meteofrance_parameters(
+                        processors_with_meteofrance_sensitive_parameters, args
+                    )
+
+
+                execution_type = {
+                    "InvokeHTTP": ExecutionType.ONCE,
+                    "UpdateAttribute": ExecutionType.ONCE,
+                    "PutHDFS": ExecutionType.FOREVER,
+                }
+
+                process_id, accounting = self.__nifi_client.run_process_group(
+                    "meteofrance2hdfs",
+                    execution_type,
+                    arguments,
+                    keys_arguments,
+                    sensitive_parameters,
+                    keycloak_token=self.__keycloak_token,
+                    processors_with_kerberos=processors_with_kerberos,
+                    kerberos_account=kerberos_account,
+                    accounting_target="PutHDFS",
+                )
+                process_ids.append(process_id)
+                accountings.append(accounting)
+
+            return '|'.join(process_ids), self.aggregate(accountings)
+
+        except Exception as ex:
+            raise HidDataTransferException(ex) from ex
+
+    def ecmwf2ckan(
+        self,
+        data_script,
+        ckan_host,
+        ckan_api_key,
+        ckan_organization,
+        ckan_dataset,
+        ckan_resource
+    ) -> tuple[str, AccountingInfo]:
+        """
+        transfer data from ECMWF open data service to CKAN
+
+        Args:
+            data_script (str): Path to the python script.
+            ckan_host (str): The CKAN host URL.
+            ckan_organization (str): The CKAN organization name.
+            ckan_dataset (str): The CKAN dataset name.
+            ckan_resource (str): The name of the CKAN resource
+                to be created with the data.
+
+        Returns:
+            tuple[str, AccountingInfo]: The result of the process group execution.
+
+        Raises:
+            HidDataTransferException: If an error occurs during
+            the data transfer process.
+
+        """
+
+        args = SimpleNamespace(
+            ckan_host=ckan_host,
+            ckan_api_key=ckan_api_key,
+            ckan_organization=ckan_organization,
+            ckan_dataset=ckan_dataset,
+            data_script=data_script,
+        )
+
+        self.__logger.info(
+            "executing ecmwf2ckan command with args: %s",
+            self.__nifi_client.format_args_to_string(args),
+        )
+
+        upload_folder = None
+        try:
+            arguments: Dict[str, Any] = {}
+
+            local_regexp = \
+                args.data_script[args.data_script.rfind("/") + 1 :]
+            local_folder = args.data_script[: args.data_script.rfind("/")]
+
+            # Transfer files from data source to NIFI upload subfolder
+            # (within user folder)
+            upload_folder, num_files = self.__nifi_client.upload_files(
+                local_folder, local_regexp
+            )
+            if num_files == 0:
+                raise HidDataTransferException(
+                    f"File not found in {local_folder}"
+                )
+            arguments["local.folder"] = upload_folder
+            # arguments["local.numfiles"] = num_files
+
+            arguments['nifi.download_folder'] = upload_folder
+            arguments['python.path'] = PYTHON_PATH_ECMWF
+            arguments['python.command'] = str(Path(upload_folder, local_regexp))
+
+            args.ckan_resource = ckan_resource
+            self.__nifi_client.add_default_ckan_parameters(arguments, args)
+            arguments[self.__nifi_client.CKAN_RESOURCE] = \
+                self.generate_ckan_resource_name(args, False)
+
+            keys_arguments: Dict[str, Any] = {}
+
+            processors_with_ckan_sensitive_parameters = ["CKANFileUploader"]
+            sensitive_parameters = \
+                self.__nifi_client.set_sensitive_ckan_parameters(
+                    processors_with_ckan_sensitive_parameters, args
+                )
+
+            execution_type = {
+                "GenerateFlowFile": ExecutionType.ONCE,
+                "ExecuteStreamCommand": ExecutionType.ONCE,
+                "FetchFile": ExecutionType.ONCE,
+                "UpdateAttribute": ExecutionType.ONCE,
+                "CKANFileUploader": ExecutionType.ONCE,
+            }
+
+            return self.__nifi_client.run_process_group(
+                    "ecmwf2ckan_v1.0",
+                    execution_type,
+                    arguments,
+                    keys_arguments,
+                    sensitive_parameters,
+                    keycloak_token=self.__keycloak_token,
+                    accounting_target="CKANFileUploader",
+                )
+
+        except Exception as ex:
+            raise HidDataTransferException(ex) from ex
+
+    def ecmwf2hdfs(
+        self,
+        data_script,
+        data_target,
+        data_source,
+        kerberos_principal=None,
+        kerberos_password=None
+    ) -> tuple[str, AccountingInfo]:
+        """
+        transfer data from ECMWF open data service to HDFS
+
+        Args:
+            data_target (str): The target path in HDFS.
+            data_source (str): The filename in script.
+            data_script (str): Path to the python script.
+            kerberos_principal (str): The kerberos principal.
+            kerberos_password (str): The kerberos password.
+
+        Returns:
+            tuple[str, AccountingInfo]: The result of the process group execution.
+
+        Raises:
+            HidDataTransferException: If an error occurs during
+            the data transfer process.
+
+        """
+
+        args = SimpleNamespace(
+            data_source=data_source,
+            data_target=data_target,
+            data_script=data_script,
+            kerberos_principal=kerberos_principal,
+            kerberos_password=kerberos_password,
+        )
+
+        processors_with_kerberos = ["PutHDFS"]
+        kerberos_account = SimpleNamespace(
+            kerberos_principal=kerberos_principal,
+            kerberos_password=kerberos_password,
+        )
+
+        self.__logger.info(
+            "executing ecmwf2hdfs command with args: %s",
+            self.__nifi_client.format_args_to_string(args),
+        )
+
+        upload_folder = None
+        try:
+            arguments: Dict[str, Any] = {}
+            arguments[HDFS_DATA_FOLDER] = args.data_target
+            arguments[HDFS_FILENAME] = args.data_source
+
+            local_regexp = \
+                args.data_script[args.data_script.rfind("/") + 1 :]
+            local_folder = args.data_script[: args.data_script.rfind("/")]
+
+            # Transfer script file to NIFI upload subfolder
+            # (within user folder)
+            upload_folder, num_files = self.__nifi_client.upload_files(
+                local_folder, local_regexp
+            )
+            if num_files == 0:
+                raise HidDataTransferException(
+                    f"File not found in {local_folder}"
+                )
+            arguments["local.folder"] = upload_folder
+            # arguments["local.numfiles"] = num_files
+
+            arguments['nifi.download_folder'] = upload_folder
+            arguments['python.path'] = PYTHON_PATH_ECMWF
+            arguments['python.command'] = str(Path(upload_folder, local_regexp))
+
+            keys_arguments: Dict[str, Any] = {}
+
+            execution_type = {
+                "GenerateFlowFile": ExecutionType.ONCE,
+                "ExecuteStreamCommand": ExecutionType.ONCE,
+                "FetchFile": ExecutionType.ONCE,
+                "UpdateAttribute": ExecutionType.ONCE,
+                "PutHDFS": ExecutionType.FOREVER,
+            }
+
+            return self.__nifi_client.run_process_group(
+                    "ecmwf2hdfs",
+                    execution_type,
+                    arguments,
+                    keys_arguments,
+                    keycloak_token=self.__keycloak_token,
+                    processors_with_kerberos=processors_with_kerberos,
+                    kerberos_account=kerberos_account,
+                    accounting_target="PutHDFS",
+                )
+
+        except Exception as ex:
+            raise HidDataTransferException(ex) from ex
+
+
     def check_command_status(self, command_id):
         """
         Checks the status of a NIFI pipeline (process group) by ID

{hid_data_transfer_lib-0.4.1 → hid_data_transfer_lib-0.5.1}/hid_data_transfer_lib/nifi/nifi_client.py

@@ -125,6 +125,7 @@ class NIFIClient:
     CKAN_RESOURCE = "ckan.resource"
     CKAN_CONCURRENT_TASKS = "ckan.concurrent_tasks"
     KERBEROS_PASSWORD = "Kerberos Password"
+    APIKEY = "apikey" #Meteofrance APIKEY
 
     def __init__(self):
         """Constructor method"""
@@ -1775,6 +1776,19 @@ class NIFIClient:
                 )
         return sensitive_parameters
 
+    def set_sensitive_meteofrance_parameters(
+        self, processors_with_meteofrance_sensitive_parameters, args):
+        """Set sensitive parameters for METEOFRANCE connection"""
+        sensitive_parameters = {}
+        for processor in processors_with_meteofrance_sensitive_parameters:
+            inner_sensitive_parameters = {}
+            sensitive_parameters[processor] = inner_sensitive_parameters
+            if args.meteofrance_apikey:
+                inner_sensitive_parameters[self.APIKEY] = (
+                    args.meteofrance_apikey
+                )
+        return sensitive_parameters
+
     def set_kerberos_configuration(self, process_group_id,
         kerberos_account, processors_with_kerberos):
         """Set Kerberos configuration for processors"""

{hid_data_transfer_lib-0.4.1 → hid_data_transfer_lib-0.5.1}/pyproject.toml

@@ -1,9 +1,10 @@
 [project]
 name = "hid_data_transfer_lib"
-version = "0.4.1"
+version = "0.5.1"
 description = "HiDALGO Data Transfer library provides methods to transfer data between different data providers and consumers using NIFI pipelines"
 authors = [
     { name = "Jesús Gorroñogoitia", email = "jesus.gorronogoitia@eviden.com" },
+    { name = "Angela Rivera", email = "angela@meteogrid.com" },
 ]
 license = "APL-2.0"
 readme = "README.md"
@@ -41,7 +42,7 @@ pytest = "^8.3.5"
 pytest-cov = "^6.1.0"
 pytest-env = "^1.1.5"
 numpy = "^2.3.4"
-python-dotenv = "^1.2.3"
+python-dotenv = "^1.2.4"
 
 [build-system]
 requires = ["poetry-core"]