dotstat_io 0.2.7__tar.gz

dotstat_io-0.2.7/PKG-INFO

@@ -0,0 +1,150 @@
+Metadata-Version: 2.1
+Name: dotstat_io
+Version: 0.2.7
+Summary: Utility to download or upload data from/to .Stat Suite using ADFS authentication to connect to it
+License: MIT
+Author: Gyorgy Gyomai
+Author-email: gyorgy.gyomai@oecd.org
+Requires-Python: >=3.10,<4.0
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Dist: chardet (>=5.1.0,<6.0.0)
+Requires-Dist: msal (>=1.22.0,<2.0.0)
+Requires-Dist: requests (>=2.29.0,<3.0.0)
+Requires-Dist: requests-kerberos (>=0.14.0,<0.15.0)
+Description-Content-Type: text/markdown
+
+### DotStat_IO: 
+A generic python package which could be integrated with other end-user applications and Gitlab runner to perform basic io with .Stat Suite. 
+Its role is to mask the complexities of authentication to connect to .Stat Suite.
+The user needs to provide a set of parameters and the package exposes a couple of methods which will download or upload data from/to .Stat Suite.
+
+### This package contains three modules:
+- ADFSAuthentication module
+- KeycloakAuthentication module
+- Download_upload module
+
+### In ADFSAuthentication module, four methods are available:
+#### 1. To initialise the module for interactive use:
+```python
+from dotstat_io.authentication import AdfsAuthentication
+with AdfsAuthentication.interactive(
+        client_id=client_id,
+        sdmx_resource_id=sdmx_resource_id,
+        scopes=scopes,
+        authority_url=authority_url,
+        redirect_port=redirect_port) as Interactive_obj:
+    access_token = Interactive_obj.get_token()
+```
+* `client_id:` The Application (client) ID that the ADFS assigned to your app
+* `sdmx_resource_id:` The ID of the application to be accessed such as .Stat Suite
+* `scopes:` Scopes requested to access a protected API (a resource defined by sdmx_resource_id)
+* `authority_url:` URL that identifies a token authority
+* `redirect_port:` The port of the address to return to upon receiving a response from the authority
+
+#### 2. To initialise the module for non-interactive use using a secret: 
+```python
+from dotstat_io.authentication import AdfsAuthentication
+with AdfsAuthentication.nointeractive_with_secret(
+        client_id=client_id,
+        sdmx_resource_id=sdmx_resource_id,
+        scopes=scopes,
+        authority_url=authority_url,
+        client_secret=client_secret) as Nointeractive_with_secret_obj:
+    access_token = Nointeractive_with_secret_obj.get_token()
+```
+* `client_id:` The Application (client) ID that the ADFS assigned to your app
+* `sdmx_resource_id:` The ID of the application to be accessed such as .Stat Suite
+* `scopes:` Scopes requested to access a protected API (a resource defined by sdmx_resource_id)
+* `authority_url:` URL that identifies a token authority
+* `client_secret:` The application secret that you created during app registration in ADFS
+
+#### 3. To initialise the module for non-interactive use using windows client authentication:
+```python
+from dotstat_io.authentication import AdfsAuthentication
+with AdfsAuthentication.nointeractive_with_adfs(
+        client_id=client_id,
+        sdmx_resource_id=sdmx_resource_id,
+        token_url=token_url) as Nointeractive_with_adfs_obj:
+    access_token = Nointeractive_with_adfs_obj.get_token()
+```
+* `client_id:` The Application (client) ID that the ADFS assigned to your app
+* `sdmx_resource_id:` The ID of the application to be accessed such as .Stat Suite
+* `token_url:` URL of the authentication service
+
+#### 4. To get a token after initialisation of "AdfsAuthentication" object as shown above: 
+```python
+access_token = [Authentication Object Name].get_token()
+```
+
+### In KeycloakAuthentication module, two methods are available:
+#### 1. To initialise the module for Keycloak authentication:
+```python
+from dotstat_io.authentication import KeycloakAuthentication
+with KeycloakAuthentication.nointeractive_with_secret(
+        user=user,
+        password=password, 
+        token_url=token_url, 
+        proxy=proxy) as Nointeractive_with_keycloak_obj:
+    access_token = Nointeractive_with_keycloak_obj.get_token()
+```
+* `user:` User name for .Stat Suite authentication
+* `password:` User name's password
+* `token_url:` URL of the authentication service
+* `proxy:` URL of the SSL certificates
+
+#### 2. To get a token after initialisation of "KeycloakAuthentication" object as shown above: 
+```python
+access_token = [Authentication Object Name].get_token()
+```
+
+### In Download_upload module, four methods are available:
+#### 1. To download a file from .Stat Suite:
+```python
+from dotstat_io.download_upload import Download_upload
+with Download_upload(adfsAuthentication_obj, access_token) as Download_upload_obj:
+    Returned_Message = Download_upload_obj.download_file(
+        dotstat_url, content_format, Path(file_path))
+```
+* `dotstat_url:` URL of data to be downloaded from .Stat Suite
+* `content_format:` Format of the downloaded content
+* `file_path:` The full path where the file will downloaded
+
+#### 2. To download streamed content from .Stat Suite:
+```python
+from dotstat_io.download_upload import Download_upload
+with Download_upload(adfsAuthentication_obj, access_token) as Download_upload_obj:
+    Returned_Message = Download_upload_obj.download_stream(
+        dotstat_url, content_format)
+```
+* `dotstat_url:` URL of data to be downloaded from .Stat Suite
+* `content_format:` Format of the downloaded content
+
+#### 3. To upload a data file to .Stat Suite:
+```python
+from dotstat_io.download_upload import Download_upload
+with Download_upload(adfsAuthentication_obj, access_token) as Download_upload_obj:
+    Returned_Message = Download_upload_obj.upload_file(
+        transfer_url, Path(file_path), space, validationType, use_filepath)
+```
+* `transfer_url:` URL of the transfer service
+* `file_path:` The full path of the SDMX-CSV file, which will be uploaded to .Stat Suite
+* `space:` Data space where the file will be uploaded
+* `validationType:` The type of validation to use during upload. Possible values: Basic Validation (0), Advanced Validation (1)
+* `use_filepath:` Use a file path of a shared folder accessible by the .Stat Suite data upload engine (for unlimited file sizes)
+
+#### 4. To upload a structure to .Stat Suite:
+```python
+from dotstat_io.download_upload import Download_upload
+with Download_upload(adfsAuthentication_obj, access_token) as Download_upload_obj:
+    Returned_Message = Download_upload_obj.upload_structure(
+        transfer_url, Path(file_path))
+```
+* `transfer_url:` URL of the transfer service
+* `file_path:` The full path of the SDMX-ML file, which will be uploaded to .Stat Suite
+
+### For more information about how to use this package, all test cases can be accessed from this [`link`](https://gitlab.algobank.oecd.org/sdd-legacy/dotstat_io/-/blob/main/tests/test_cases.py)
+

dotstat_io-0.2.7/README.md

@@ -0,0 +1,130 @@
+### DotStat_IO: 
+A generic python package which could be integrated with other end-user applications and Gitlab runner to perform basic io with .Stat Suite. 
+Its role is to mask the complexities of authentication to connect to .Stat Suite.
+The user needs to provide a set of parameters and the package exposes a couple of methods which will download or upload data from/to .Stat Suite.
+
+### This package contains three modules:
+- ADFSAuthentication module
+- KeycloakAuthentication module
+- Download_upload module
+
+### In ADFSAuthentication module, four methods are available:
+#### 1. To initialise the module for interactive use:
+```python
+from dotstat_io.authentication import AdfsAuthentication
+with AdfsAuthentication.interactive(
+        client_id=client_id,
+        sdmx_resource_id=sdmx_resource_id,
+        scopes=scopes,
+        authority_url=authority_url,
+        redirect_port=redirect_port) as Interactive_obj:
+    access_token = Interactive_obj.get_token()
+```
+* `client_id:` The Application (client) ID that the ADFS assigned to your app
+* `sdmx_resource_id:` The ID of the application to be accessed such as .Stat Suite
+* `scopes:` Scopes requested to access a protected API (a resource defined by sdmx_resource_id)
+* `authority_url:` URL that identifies a token authority
+* `redirect_port:` The port of the address to return to upon receiving a response from the authority
+
+#### 2. To initialise the module for non-interactive use using a secret: 
+```python
+from dotstat_io.authentication import AdfsAuthentication
+with AdfsAuthentication.nointeractive_with_secret(
+        client_id=client_id,
+        sdmx_resource_id=sdmx_resource_id,
+        scopes=scopes,
+        authority_url=authority_url,
+        client_secret=client_secret) as Nointeractive_with_secret_obj:
+    access_token = Nointeractive_with_secret_obj.get_token()
+```
+* `client_id:` The Application (client) ID that the ADFS assigned to your app
+* `sdmx_resource_id:` The ID of the application to be accessed such as .Stat Suite
+* `scopes:` Scopes requested to access a protected API (a resource defined by sdmx_resource_id)
+* `authority_url:` URL that identifies a token authority
+* `client_secret:` The application secret that you created during app registration in ADFS
+
+#### 3. To initialise the module for non-interactive use using windows client authentication:
+```python
+from dotstat_io.authentication import AdfsAuthentication
+with AdfsAuthentication.nointeractive_with_adfs(
+        client_id=client_id,
+        sdmx_resource_id=sdmx_resource_id,
+        token_url=token_url) as Nointeractive_with_adfs_obj:
+    access_token = Nointeractive_with_adfs_obj.get_token()
+```
+* `client_id:` The Application (client) ID that the ADFS assigned to your app
+* `sdmx_resource_id:` The ID of the application to be accessed such as .Stat Suite
+* `token_url:` URL of the authentication service
+
+#### 4. To get a token after initialisation of "AdfsAuthentication" object as shown above: 
+```python
+access_token = [Authentication Object Name].get_token()
+```
+
+### In KeycloakAuthentication module, two methods are available:
+#### 1. To initialise the module for Keycloak authentication:
+```python
+from dotstat_io.authentication import KeycloakAuthentication
+with KeycloakAuthentication.nointeractive_with_secret(
+        user=user,
+        password=password, 
+        token_url=token_url, 
+        proxy=proxy) as Nointeractive_with_keycloak_obj:
+    access_token = Nointeractive_with_keycloak_obj.get_token()
+```
+* `user:` User name for .Stat Suite authentication
+* `password:` User name's password
+* `token_url:` URL of the authentication service
+* `proxy:` URL of the SSL certificates
+
+#### 2. To get a token after initialisation of "KeycloakAuthentication" object as shown above: 
+```python
+access_token = [Authentication Object Name].get_token()
+```
+
+### In Download_upload module, four methods are available:
+#### 1. To download a file from .Stat Suite:
+```python
+from dotstat_io.download_upload import Download_upload
+with Download_upload(adfsAuthentication_obj, access_token) as Download_upload_obj:
+    Returned_Message = Download_upload_obj.download_file(
+        dotstat_url, content_format, Path(file_path))
+```
+* `dotstat_url:` URL of data to be downloaded from .Stat Suite
+* `content_format:` Format of the downloaded content
+* `file_path:` The full path where the file will downloaded
+
+#### 2. To download streamed content from .Stat Suite:
+```python
+from dotstat_io.download_upload import Download_upload
+with Download_upload(adfsAuthentication_obj, access_token) as Download_upload_obj:
+    Returned_Message = Download_upload_obj.download_stream(
+        dotstat_url, content_format)
+```
+* `dotstat_url:` URL of data to be downloaded from .Stat Suite
+* `content_format:` Format of the downloaded content
+
+#### 3. To upload a data file to .Stat Suite:
+```python
+from dotstat_io.download_upload import Download_upload
+with Download_upload(adfsAuthentication_obj, access_token) as Download_upload_obj:
+    Returned_Message = Download_upload_obj.upload_file(
+        transfer_url, Path(file_path), space, validationType, use_filepath)
+```
+* `transfer_url:` URL of the transfer service
+* `file_path:` The full path of the SDMX-CSV file, which will be uploaded to .Stat Suite
+* `space:` Data space where the file will be uploaded
+* `validationType:` The type of validation to use during upload. Possible values: Basic Validation (0), Advanced Validation (1)
+* `use_filepath:` Use a file path of a shared folder accessible by the .Stat Suite data upload engine (for unlimited file sizes)
+
+#### 4. To upload a structure to .Stat Suite:
+```python
+from dotstat_io.download_upload import Download_upload
+with Download_upload(adfsAuthentication_obj, access_token) as Download_upload_obj:
+    Returned_Message = Download_upload_obj.upload_structure(
+        transfer_url, Path(file_path))
+```
+* `transfer_url:` URL of the transfer service
+* `file_path:` The full path of the SDMX-ML file, which will be uploaded to .Stat Suite
+
+### For more information about how to use this package, all test cases can be accessed from this [`link`](https://gitlab.algobank.oecd.org/sdd-legacy/dotstat_io/-/blob/main/tests/test_cases.py)

dotstat_io-0.2.7/dotstat_io/authentication.py

@@ -0,0 +1,335 @@
+import requests
+import msal
+import requests_kerberos
+import json
+import time
+
+from enum import IntEnum
+from datetime import datetime
+
+
+class AuthenticationMode(IntEnum):
+    INTERACTIVE = 1
+    NONINTERACTIVE_WITH_SECRET = 2
+    NONINTERACTIVE_WITH_ADFS = 3
+
+# class to manage ADFS authentication using OIDC flows
+class AdfsAuthentication():
+
+    # Declare constants
+    __ERROR = "An error occurred: "
+    __SUCCESS = "Successful authentication"
+
+    # Initialise Adfs_authentication
+    def __init__(
+        self,
+        mode: AuthenticationMode,
+        client_id: str,
+        sdmx_resource_id: str,
+        scopes: list[str] = [],
+        authority_url: str = None,
+        token_url: str = None,
+        redirect_port: int = 3000,
+        client_secret: str = None
+    ):
+        self.__access_token = None
+
+        self.__refresh_token = None
+        self.__creation_time = None
+        self.__expiration_time = None
+
+        self.result = None
+        self.mode = mode
+        self.client_id = client_id
+        self.sdmx_resource_id = sdmx_resource_id
+        self.scopes = scopes
+        self.authority_url = authority_url
+        self.token_url = token_url
+        self.redirect_port = redirect_port
+        self.client_secret = client_secret
+
+        match self.mode:
+            case AuthenticationMode.INTERACTIVE:
+                self.app = msal.PublicClientApplication(
+                    self.client_id, authority=self.authority_url)
+            case AuthenticationMode.NONINTERACTIVE_WITH_SECRET:
+                self.app = msal.ConfidentialClientApplication(
+                    self.client_id, authority=self.authority_url, client_credential=self.client_secret)
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        self.__access_token = None
+        self.__refresh_token = None
+        self.__creation_time = None
+        self.__expiration_time = None
+
+    @classmethod
+    def interactive(
+        cls, 
+        client_id: str, 
+        sdmx_resource_id: str, 
+        scopes: list[str], 
+        authority_url: str, 
+        redirect_port: int = 3000,
+        mode: AuthenticationMode = AuthenticationMode.INTERACTIVE
+    ):
+        return cls(
+            client_id=client_id, 
+            sdmx_resource_id=sdmx_resource_id, 
+            scopes=scopes, 
+            authority_url=authority_url,
+            redirect_port=redirect_port,
+            mode=mode
+        )
+
+    @classmethod
+    def nointeractive_with_secret(
+        cls, 
+        client_id: str, 
+        sdmx_resource_id: str,
+        scopes: list[str],
+        authority_url: str,
+        client_secret: str,
+        mode: AuthenticationMode = AuthenticationMode.NONINTERACTIVE_WITH_SECRET
+    ):
+        return cls(
+            client_id=client_id,
+            sdmx_resource_id=sdmx_resource_id,
+            scopes=scopes,
+            authority_url=authority_url,   
+            client_secret=client_secret,  
+            mode=mode
+            )
+
+    @classmethod
+    def nointeractive_with_adfs(
+        cls, 
+        client_id: str, 
+        sdmx_resource_id: str,
+        token_url: str,
+        mode: AuthenticationMode = AuthenticationMode.NONINTERACTIVE_WITH_ADFS
+    ):
+        return cls(
+            client_id=client_id, 
+            sdmx_resource_id=sdmx_resource_id,
+            token_url=token_url,
+            mode=mode)
+
+
+    def get_token(self):
+        self.__access_token = None
+        self.__refresh_token = None
+        self.__creation_time = None
+        self.__expiration_time = None
+        self.result = None
+        match self.mode:
+            case AuthenticationMode.INTERACTIVE:
+                self._acquire_token_interactive()
+            case AuthenticationMode.NONINTERACTIVE_WITH_SECRET:
+                self._acquire_token_noninteractive_with_secret()
+            case AuthenticationMode.NONINTERACTIVE_WITH_ADFS:
+                self._acquire_token_noninteractive_with_adfs()
+
+        return self.__access_token
+    
+    # Check the token is not expired
+    def is_access_token_expired(self):
+        # convert the timestamp to a datetime object
+        if datetime.fromtimestamp(self.__expiration_time) is not None:
+            if datetime.now() <= datetime.fromtimestamp(self.__expiration_time):
+                return False
+            else:
+                return True
+        else:
+            return True
+
+    # Authentication interactively - aka Authorization Code flow
+    def _acquire_token_interactive(self):
+        try:
+
+            # We now check the cache to see
+            # whether we already have some accounts that the end user already used to sign in before.
+            accounts = self.app.get_accounts()
+            if accounts:
+                account = accounts[0]
+            else:
+                account = None
+
+            # Firstly, looks up a access_token from cache, or using a refresh token
+            response_silent = self.app.acquire_token_silent(
+                self.scopes, account=account)
+            if not response_silent:
+                # Prompt the user to sign in interactively
+                response_interactive = self.app.acquire_token_interactive(
+                    scopes=self.scopes, port=self.redirect_port)
+                if "access_token" in response_interactive:
+                    self.__access_token = response_interactive.get("access_token")
+                    self.__refresh_token = response_interactive.get("refresh_token")
+                    self.__creation_time = time.time()
+                    self.__expiration_time = time.time() + int(response_interactive.get("expires_in")) -  60 # one minute margin
+                    self.result = self.__SUCCESS
+                else:
+                    self.result = f'{self.__ERROR}{response_interactive.get("error")} Error description: {response_interactive.get("error_description")}'
+            else:
+                if "access_token" in response_silent:
+                    self.__access_token = response_silent.get("access_token")
+                    self.__refresh_token = response_silent.get("refresh_token")
+                    self.__creation_time = time.time()
+                    self.__expiration_time = time.time() + int(response_silent.get("expires_in")) -  60 # one minute margin
+                    self.result = self.__SUCCESS
+                else:
+                    self.result = f'{self.__ERROR}{response_silent.get("error")} Error description: {response_silent.get("error_description")}'
+        except Exception as err:
+            self.result = f'{self.__ERROR}{err}\n'
+
+
+    # Authentication non-interactively using any account - aka Client Credentials flow
+    def _acquire_token_noninteractive_with_secret(self):
+        try:
+            response = self.app.acquire_token_for_client(scopes=self.scopes)
+            if "access_token" in response:
+                self.__access_token = response.get("access_token")
+                self.__creation_time = time.time()
+                self.__expiration_time = time.time() + int(response.get("expires_in")) -  60 # one minute margin
+                self.result = self.__SUCCESS
+            else:
+                self.result = f'{self.__ERROR}{response.get("error")} Error description: {response.get("error_description")}'
+
+        except Exception as err:
+            self.result = f'{self.__ERROR}{err}\n'
+
+
+    # Authentication non-interactively using service account - aka Windows Client Authentication
+    def _acquire_token_noninteractive_with_adfs(self):
+        try:
+            headers = {
+                "Content-Type": "application/x-www-form-urlencoded"
+            }
+
+            payload = {
+                'client_id': self.client_id,
+                'use_windows_client_authentication': 'true',
+                'grant_type': 'client_credentials',
+                'scope': 'openid',
+                'resource': self.sdmx_resource_id
+            }
+
+            kerberos_auth = requests_kerberos.HTTPKerberosAuth(
+                mutual_authentication=requests_kerberos.OPTIONAL, force_preemptive=True)
+            response = requests.post(url=self.token_url, data=payload, auth=kerberos_auth)
+
+            if response.status_code != 200:
+                message = f'{self.__ERROR} Error code: {response.status_code} Reason: {response.reason}\n'
+                if len(response.text) > 0:
+                    # Use the json module to load response into a dictionary.
+                    response_dict = json.loads(response.text)
+                    message += f'{self.__ERROR}{response_dict.get("error")} Error description: {response_dict.get("error_description")}\n'
+                    
+                self.result = message
+            else:
+                self.__access_token = response.json()['access_token']
+                self.__creation_time = time.time()
+                self.__expiration_time = time.time() + int(response.json()['expires_in']) -  60 # one minute margin
+                self.result = self.__SUCCESS
+
+        except Exception as err:
+            self.result = f'{self.__ERROR} {err}\n'
+
+
+# class to manage ADFS authentication using OIDC flows
+class KeycloakAuthentication():
+
+    # Declare constants
+    __ERROR = "An error occurred: "
+    __SUCCESS = "Successful authentication"
+
+    # Initialise Adfs_authentication
+    def __init__(
+        self,
+        mode: AuthenticationMode,
+        token_url: str,
+        user: str,
+        password: str,
+        client_id: str,
+        client_secret: str,
+        proxy: str | None,
+        scopes: list[str] = []
+    ):
+        self.__token = None
+        self.result = None
+        self.mode = mode
+        self.client_id = client_id
+        self.client_secret = client_secret     
+        self.token_url = token_url
+        self.user = user
+        self.password = password
+        self.scopes = scopes   #TODO check if needs explicit adjustment
+
+        if proxy:
+            self.proxies = {
+                "http": proxy,
+                "https": proxy
+            }
+        else:
+            self.proxies = None
+
+        
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        self.__token = None
+
+    @classmethod
+    def nointeractive_with_secret(
+        cls, 
+        token_url: str,
+        user: str,
+        password: str,
+        client_id: str = "app",
+        client_secret: str = "",
+        proxy: str | None = None,
+        scopes: list[str] = [],
+        mode: AuthenticationMode = AuthenticationMode.NONINTERACTIVE_WITH_SECRET
+    ):
+        return cls(           
+            token_url=token_url,
+            user=user,
+            password=password,
+            client_id=client_id,   
+            client_secret=client_secret,
+            scopes=scopes,
+            proxy=proxy,  
+            mode=mode
+            )
+
+    def get_token(self):
+        self.__token = None
+        self.result = None
+        match self.mode:
+           case AuthenticationMode.NONINTERACTIVE_WITH_SECRET:
+                self._acquire_token_noninteractive_with_secret()
+
+        return self.__token
+
+    # Authentication non-interactively using any account - aka Client Credentials flow
+    def _acquire_token_noninteractive_with_secret(self):
+        payload = {
+           'grant_type': 'password',
+           'client_id': self.client_id,
+           'client_secret': self.client_secret,
+           'username': self.user,
+           'password': self.password
+        }
+
+        headers = {"Content-Type": "application/x-www-form-urlencoded"}
+
+        response = requests.post(self.token_url, proxies=self.proxies, headers=headers, data=payload)
+        if response.status_code not in {200, 201}:
+                self.result = f'{self.__ERROR}{response}'
+        else:
+                self.__token = response.json()['access_token']
+                self.result = self.__SUCCESS
+        

dotstat_io-0.2.7/dotstat_io/download_upload.py

@@ -0,0 +1,376 @@
+import os
+import requests
+import chardet
+import logging
+import xml.etree.ElementTree as ET
+
+from pathlib import Path
+from time import sleep
+
+
+FORMAT = '%(message)s'
+logging.basicConfig(format=FORMAT, level=logging.INFO)
+log = logging.getLogger(__name__)
+
+
+# class to download or upload data from/to .Stat Suite
+class Download_upload():
+
+    # Declare constants
+    __ERROR  = "An error occurred: "
+    __NO_ACCESS_TOKEN = "No access token"
+    __EXECUTION_IN_QUEUED = "Queued"
+    __EXECUTION_IN_PROGRESS = "InProgress"
+    __CONNECTION_ABORTED = "An existing connection was forcibly closed by the remote host"
+    __DOWNLOAD_SUCCESS = "Successful download"
+    __UPLOAD_SUCCESS = "The request was successfully processed "
+    __UPLOAD_FAILED = "The request failed with status code "
+
+    __NAMESPACE_MESSAGE = "{http://www.sdmx.org/resources/sdmxml/schemas/v2_1/message}"
+    __NAMESPACE_COMMON = "{http://www.sdmx.org/resources/sdmxml/schemas/v2_1/common}"
+  
+    # Initialise Download_upload
+    def __init__(self, adfsAuthentication_obj, access_token):
+        self.adfsAuthentication_obj = adfsAuthentication_obj
+        self.access_token = access_token
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        self.adfsAuthentication_obj = None
+        self.access_token = None
+
+
+    # Download a file from .STAT
+    def download_file(self, dotstat_url: str, content_format: str, file_path: Path):
+        try:
+            Returned_Message = ""
+
+            if (self.access_token == None):
+                Returned_Message = self.__ERROR  + self.__NO_ACCESS_TOKEN + os.linesep
+
+                # Write the result to the log
+                for line in Returned_Message.split(os.linesep):
+                    if len(line) > 0:
+                        log.info('   ' + line)
+            else:
+                if self.adfsAuthentication_obj.is_access_token_expired():
+                    self.access_token = self.adfsAuthentication_obj.get_token()
+
+                headers = {
+                   'accept': content_format,
+                   'authorization': 'Bearer '+self.access_token
+                }
+
+                #
+                response = requests.get(dotstat_url, verify=True, headers=headers)
+        except Exception as err:
+            Returned_Message = self.__ERROR  + str(err) + os.linesep
+
+            # Write the result to the log
+            for line in Returned_Message.split(os.linesep):
+                if len(line) > 0:
+                    log.info('   ' + line)
+        else:
+            if response.status_code != 200:
+                Returned_Message = self.__ERROR  + 'Error code: ' + \
+                    str(response.status_code) + ' Reason: ' + str(response.reason)
+                if len(response.text) > 0:
+                    Returned_Message += os.linesep + 'Text: ' + response.text
+            else:
+                if os.path.isfile(file_path):
+                    os.remove(file_path)
+                with open(file_path, "wb") as file:
+                    file.write(response.content)
+                    Returned_Message = self.__DOWNLOAD_SUCCESS
+            
+            # Write the result to the log
+            for line in Returned_Message.split(os.linesep):
+                if len(line) > 0:
+                    log.info('   ' + line)
+        finally:
+            return Returned_Message
+
+
+    # Download streamed content from .STAT
+    def download_stream(self, dotstat_url: str, content_format: str):
+        try:
+            Returned_Message = ""
+
+            if (self.access_token == None):
+                Returned_Message = self.__ERROR  + self.__NO_ACCESS_TOKEN + os.linesep
+            else:
+                if self.adfsAuthentication_obj.is_access_token_expired():
+                    self.access_token = self.adfsAuthentication_obj.get_token()
+
+                headers = {
+                    'accept': content_format,
+                    'Transfer-Encoding': 'chunked',
+                    'authorization': 'Bearer '+self.access_token
+                }
+
+                #
+                return requests.get(dotstat_url, verify=True, headers=headers, stream=True)
+        except Exception as err:
+            Returned_Message = self.__ERROR  + str(err) + os.linesep
+            return Returned_Message
+
+
+    # Upload a file to .STAT
+    def upload_file(self, 
+                    transfer_url: str, 
+                    file_path: Path, 
+                    space: str, 
+                    validationType: int, 
+                    use_filepath: bool = False):
+        try:
+            Returned_Message = ""
+
+            if (self.access_token == None):
+                Returned_Message = self.__ERROR  + self.__NO_ACCESS_TOKEN + os.linesep
+
+                # Write the result to the log
+                for line in Returned_Message.split(os.linesep):
+                    if len(line) > 0:
+                        log.info('   ' + line)
+            else:
+                if self.adfsAuthentication_obj.is_access_token_expired():
+                    self.access_token = self.adfsAuthentication_obj.get_token()
+
+                payload = {
+                    'dataspace': space,
+                    'validationType': validationType
+                }
+
+                headers = {
+                    'accept': 'application/json',
+                    'authorization': "Bearer "+self.access_token
+                }
+
+                if  use_filepath:
+                    files = {
+                        'dataspace': (None, payload['dataspace']),
+                        'validationType': (None, payload['validationType']),
+                        'filepath': (None, str(file_path))
+                    }
+                else:
+                    files = {
+                        'dataspace': (None, payload['dataspace']),
+                        'validationType': (None, payload['validationType']),
+                        'file': (os.path.realpath(file_path), open(os.path.realpath(file_path), 'rb'), 'text/csv', '')
+                    }
+
+                #
+                response = requests.post(transfer_url, verify=True, headers=headers, files=files)
+        except Exception as err:
+            Returned_Message = self.__ERROR  + str(err) + os.linesep
+
+            # Write the result to the log
+            for line in Returned_Message.split(os.linesep):
+                if len(line) > 0:
+                    log.info('   ' + line)
+        else:
+            if response.status_code != 200:
+                Returned_Message = self.__ERROR  + 'Error code: ' + \
+                    str(response.status_code) + ' Reason: ' + str(response.reason)
+                if len(response.text) > 0:
+                    Returned_Message = Returned_Message + os.linesep + 'Text: ' + response.text
+
+                Returned_Message = Returned_Message + os.linesep
+                # Write the result to the log
+                for line in Returned_Message.split(os.linesep):
+                    if len(line) > 0:
+                        log.info('   ' + line)
+            else:
+                try:
+                    Result = response.json()['message']
+
+                    # Write the result to the log
+                    for line in Result.split(os.linesep):
+                        if len(line) > 0:
+                            log.info('   ' + line)
+
+                    Returned_Message = Result + os.linesep
+
+                    # Check the request status
+                    if (Result != "" and Result.find(self.__ERROR ) == -1):
+                        # Extract the request ID the returned message
+                        start = 'with ID'
+                        end = 'was successfully'
+                        requestId = Result[Result.find(
+                            start)+len(start):Result.rfind(end)]
+
+                        # Sleep a little bit before checking the request status
+                        sleep(3)
+
+                        # To avoid this error: maximum recursion depth exceeded while calling a Python object
+                        # replace the recursive calls with while loops.
+                        Result = self.__check_request_status(transfer_url, requestId, space)
+
+                        # Write the result to the log
+                        for line in Result.split(os.linesep):
+                            if len(line) > 0:
+                                log.info('   ' + line)
+                        sleep(3)
+
+                        Previous_Result = Result
+                        while Result in [self.__EXECUTION_IN_PROGRESS, self.__EXECUTION_IN_QUEUED, self.__CONNECTION_ABORTED]:
+                            Result = self.__check_request_status(transfer_url, requestId, space)
+
+                            # Prevent loging again the same information such as "Queued" or "InProgress" 
+                            if Previous_Result != Result:
+                               Previous_Result = Result
+
+                            # Write the result to the log
+                            for line in Previous_Result.split(os.linesep):
+                              if (len(line) > 0 and line not in [self.__EXECUTION_IN_PROGRESS, self.__EXECUTION_IN_QUEUED, self.__CONNECTION_ABORTED]):
+                                 log.info('   ' + line)
+                            sleep(3)
+
+                        Returned_Message = Returned_Message + Result + os.linesep
+                except Exception as err:
+                    Returned_Message = self.__ERROR  + str(err) + os.linesep
+                    if len(response.text) > 0:
+                        Returned_Message = Returned_Message + 'Text: ' + response.text + os.linesep
+
+                    # Write the result to the log
+                    for line in Returned_Message.split(os.linesep):
+                        if len(line) > 0:
+                            log.info('   ' + line)
+        finally:
+            return Returned_Message
+
+
+    # Upload a structure to .STAT
+    def upload_structure(self, transfer_url: str, file_path: Path):
+        try:
+            Returned_Message = ""
+
+            if (self.access_token == None):
+                Returned_Message = self.__ERROR  + self.__NO_ACCESS_TOKEN + os.linesep
+
+                # Write the result to the log
+                for line in Returned_Message.split(os.linesep):
+                    if len(line) > 0:
+                        log.info('   ' + line)
+            else:
+                if self.adfsAuthentication_obj.is_access_token_expired():
+                    self.access_token = self.adfsAuthentication_obj.get_token()
+
+                # Detect the encoding used in file 
+                detected_encoding = self.__detect_encode(file_path)
+
+                # Read file as a string "r+" with the detected encoding
+                with open(file=file_path, mode="r+", encoding=detected_encoding.get("encoding")) as file:
+                    xml_data = file.read()
+
+                # Make sure the encoding is "utf-8"
+                tree = ET.fromstring(xml_data)
+                xml_data = ET.tostring(tree, encoding="utf-8", method='xml', xml_declaration=True)
+
+                headers = {
+                    'Content-Type': 'application/xml',
+                    'authorization': "Bearer "+self.access_token
+                }
+
+                #
+                response = requests.post(transfer_url, verify=True, headers=headers, data=xml_data)
+        except Exception as err:
+            Returned_Message = self.__ERROR  + str(err) + os.linesep
+
+            # Write the result to the log
+            for line in Returned_Message.split(os.linesep):
+                if len(line) > 0:
+                    log.info('   ' + line)
+        else:
+            try:
+                response.raise_for_status()
+            except requests.exceptions.HTTPError as e:
+                Returned_Message = f'{self.__UPLOAD_FAILED}{response.status_code}: {e}'
+
+                # Write the result to the log
+                for line in Returned_Message.split(os.linesep):
+                    if len(line) > 0:
+                        log.info('   ' + line)
+            else:
+                response_tree = ET.XML(response.content)
+                for error_message in response_tree.findall("./{0}ErrorMessage".format(self.__NAMESPACE_MESSAGE)):
+                    text_element = error_message.find("./{0}Text".format(self.__NAMESPACE_COMMON))
+                    if (text_element is not None):
+                        if Returned_Message == "":
+                            Returned_Message = f'{self.__UPLOAD_SUCCESS}with status code: {response.status_code}' + os.linesep
+                        Returned_Message = Returned_Message + text_element.text + os.linesep
+
+                # Write the result to the log
+                for line in Returned_Message.split(os.linesep):
+                    if len(line) > 0:
+                        log.info('   ' + line)
+        finally:
+            return Returned_Message
+        
+
+    # Detect the encoding used in file
+    def __detect_encode(self, file_path):
+        detector = chardet.UniversalDetector()
+        detector.reset()
+        with open(file=file_path, mode="rb") as file:
+            for row in file:
+                detector.feed(row)
+                if detector.done: 
+                    break
+
+        detector.close()
+
+        return detector.result
+
+
+    # Check request sent to .STAT status
+    # To avoid this error: maximum recursion depth exceeded while calling a Python object
+    # replace the recursive calls with while loops.
+    def __check_request_status(self, transfer_url, requestId, space):
+        try:
+            Returned_Message = ""
+
+            if (self.access_token == None):
+                Returned_Message = self.__ERROR  + self.__NO_ACCESS_TOKEN + os.linesep
+            else:
+                if self.adfsAuthentication_obj.is_access_token_expired():
+                    self.access_token = self.adfsAuthentication_obj.get_token()
+
+                headers = {
+                    'accept': 'application/json',
+                    'authorization': "Bearer "+self.access_token
+                }
+
+                payload = {
+                    'dataspace': space,
+                    'id': requestId
+                }
+
+                transfer_url = transfer_url.replace("import", "status")
+                transfer_url = transfer_url.replace("sdmxFile", "request")
+
+                #
+                response = requests.post(transfer_url, verify=True, headers=headers, data=payload)
+        except Exception as err:
+            Returned_Message = self.__ERROR  + str(err)
+        else:
+            if response.status_code != 200:
+                Returned_Message = self.__ERROR  + 'Error code: ' + \
+                    str(response.status_code) + ' Reason: ' + str(response.reason)
+                if len(response.text) > 0:
+                    Returned_Message = Returned_Message + os.linesep + 'Text: ' + response.text
+            else:
+                executionStatus = 'Execution status: ' + response.json()['executionStatus']
+                if response.json()['executionStatus'] in [self.__EXECUTION_IN_PROGRESS, self.__EXECUTION_IN_QUEUED, self.__CONNECTION_ABORTED]:
+                    Returned_Message = response.json()['executionStatus']
+                else:
+                    Returned_Message = executionStatus + os.linesep + 'Outcome: ' + response.json()['outcome'] + os.linesep
+                    index = 0
+                    while index < len(response.json()['logs']):
+                        Returned_Message = Returned_Message + 'Log' + str(index) + ': ' + response.json()['logs'][index]['message'] + os.linesep
+                        index += 1
+        finally:
+            return Returned_Message
+    

dotstat_io-0.2.7/pyproject.toml

@@ -0,0 +1,23 @@
+[tool.poetry]
+name = "dotstat_io"
+version = "0.2.7"
+description = "Utility to download or upload data from/to .Stat Suite using ADFS authentication to connect to it"
+license = "MIT"
+authors = ["Gyorgy Gyomai <gyorgy.gyomai@oecd.org>", "Abdel Aliaoui <abdel.aliaoui@oecd.org>"]
+readme = "README.md"
+packages = [{include = "dotstat_io"}]
+
+[tool.poetry.dependencies]
+python = "^3.10"
+requests = "^2.29.0"
+requests-kerberos = "^0.14.0"
+msal = "^1.22.0"
+chardet = "^5.1.0"
+
+
+[tool.poetry.group.dev.dependencies]
+pytest = "^7.3.1"
+
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"