oscar-python 1.2.1__tar.gz → 1.3.1__tar.gz

{oscar_python-1.2.1 → oscar_python-1.3.1}/PKG-INFO

@@ -1,11 +1,12 @@
 Metadata-Version: 2.1
 Name: oscar_python
-Version: 1.2.1
+Version: 1.3.1
 Summary: OSCAR API for python
 Home-page: https://github.com/grycap/oscar_python
 Author: GRyCAP - Universitat Politecnica de Valencia
 Author-email: calarcon@i3m.upv.es
 License: Apache 2.0
+Platform: UNKNOWN
 Classifier: Programming Language :: Python :: 3
 Classifier: License :: OSI Approved :: Apache Software License
 Description-Content-Type: text/markdown
@@ -13,7 +14,7 @@ License-File: LICENSE
 
 ## Python OSCAR client
 
-[![Build](https://github.com/grycap/oscar_python/actions/workflows/main.yaml/badge.svg)](https://github.com/grycap/oscar_python/actions/workflows/main.yaml)
+[![Build](https://github.com/grycap/oscar_python/actions/workflows/release.yaml/badge.svg)](https://github.com/grycap/oscar_python/actions/workflows/main.yaml)
 ![PyPI](https://img.shields.io/pypi/v/oscar_python)
 
 This package provides a client to interact with OSCAR (https://oscar.grycap.net) clusters and services. It is available on Pypi with the name [oscar-python](https://pypi.org/project/oscar-python/).
@@ -68,6 +69,33 @@ If you already have a valid token, you can use the parameter `oidc_token` instea
 ```
 An example of using a generated token is if you want to use EGI Notebooks. Since you can't use oidc-agent on the Notebook, you can make use of the generated token that EGI provides on path `/var/run/secrets/egi.eu/access_token`.
 
+If you have a valid refresh token (long live token), you can use the parameter `refresh_token` instead.
+
+``` python
+  options_oidc_auth = {'cluster_id':'cluster-id',
+                'endpoint':'https://cluster-endpoint',
+                'refresh_token':'token',
+                'ssl':'True'}
+                
+  client = Client(options = options_oidc_auth)
+```
+
+You can get a refresh token from EGI Check-In using the [Token Portal](https://aai.egi.eu/token).
+
+In case of using other OIDC provider you must provide two additional parameters `token_endpoint`
+and `scopes`:
+
+``` python
+  options_oidc_auth = {'cluster_id':'cluster-id',
+                'endpoint':'https://cluster-endpoint',
+                'refresh_token':'token',
+                'scopes': ["openid", "profile", "email"],
+                'token_endpoint': "http://issuer.com/token",
+                'ssl':'True'}
+                
+  client = Client(options = options_oidc_auth)
+```
+
 ### Sample usage
 
 - Sample code that creates a client and gets information about the cluster
@@ -170,6 +198,9 @@ response = client.remove_service("service_name") # returns an http response
 ``` python
 # make a synchronous execution 
 response = client.run_service("service_name", input="input", output="out.png", timeout=100) # returns an http response
+
+# make an asynchronous execution
+response = client.run_service("service_name", input="input", async_call=True) # returns an http response
 ```
 
 #### Logs methods
@@ -200,11 +231,19 @@ response = client.remove_all_jobs("service_name") # returns an http response
 
 #### Storage usage
 
-You can create a storage object to operate over the different storage providers defined on a service with the method `create_storage_client` as follows:
+You can create a storage object to operate over the different storage providers defined on a service with the method `create_storage_client`. This constructor returns a storage object with methos to interact with the storage providers.
+
+The default constructor, seen as follows, will create a provider to interact with the default MinIO instance through the user's credentials. 
+
+``` python
+storage_service = client.create_storage_client() # returns a storage object
+```
+Additionally, if you need to interact with specific storage providers defined on a service, the constructor accepts a `svc` parameter where you can state the service name from which to search for additional credentials.
 
 ``` python
 storage_service = client.create_storage_client("service_name") # returns a storage object
 ```
+
 > _Note_ : The `storage_provider` parameter on the storage methods follows the format: `["storage_provider_type"].["storage_provider_name"]` where `storage_provider_type` is one of the suported storage providers (minIO, S3, Onedata or webdav) and `storage_provider_name` is the identifier _(ex: minio.default)_
 
 **list_files_from_path**
@@ -227,3 +266,5 @@ response = storage_service.upload_file("storage_provider", "local_path", "remote
 # download a file from a remote path to a local path 
 response = storage_service.download_file("storage_provider", "local_path", "remote_path")
 ```
+
+

{oscar_python-1.2.1 → oscar_python-1.3.1}/README.md

@@ -1,6 +1,6 @@
 ## Python OSCAR client
 
-[![Build](https://github.com/grycap/oscar_python/actions/workflows/main.yaml/badge.svg)](https://github.com/grycap/oscar_python/actions/workflows/main.yaml)
+[![Build](https://github.com/grycap/oscar_python/actions/workflows/release.yaml/badge.svg)](https://github.com/grycap/oscar_python/actions/workflows/main.yaml)
 ![PyPI](https://img.shields.io/pypi/v/oscar_python)
 
 This package provides a client to interact with OSCAR (https://oscar.grycap.net) clusters and services. It is available on Pypi with the name [oscar-python](https://pypi.org/project/oscar-python/).
@@ -55,6 +55,33 @@ If you already have a valid token, you can use the parameter `oidc_token` instea
 ```
 An example of using a generated token is if you want to use EGI Notebooks. Since you can't use oidc-agent on the Notebook, you can make use of the generated token that EGI provides on path `/var/run/secrets/egi.eu/access_token`.
 
+If you have a valid refresh token (long live token), you can use the parameter `refresh_token` instead.
+
+``` python
+  options_oidc_auth = {'cluster_id':'cluster-id',
+                'endpoint':'https://cluster-endpoint',
+                'refresh_token':'token',
+                'ssl':'True'}
+                
+  client = Client(options = options_oidc_auth)
+```
+
+You can get a refresh token from EGI Check-In using the [Token Portal](https://aai.egi.eu/token).
+
+In case of using other OIDC provider you must provide two additional parameters `token_endpoint`
+and `scopes`:
+
+``` python
+  options_oidc_auth = {'cluster_id':'cluster-id',
+                'endpoint':'https://cluster-endpoint',
+                'refresh_token':'token',
+                'scopes': ["openid", "profile", "email"],
+                'token_endpoint': "http://issuer.com/token",
+                'ssl':'True'}
+                
+  client = Client(options = options_oidc_auth)
+```
+
 ### Sample usage
 
 - Sample code that creates a client and gets information about the cluster
@@ -157,6 +184,9 @@ response = client.remove_service("service_name") # returns an http response
 ``` python
 # make a synchronous execution 
 response = client.run_service("service_name", input="input", output="out.png", timeout=100) # returns an http response
+
+# make an asynchronous execution
+response = client.run_service("service_name", input="input", async_call=True) # returns an http response
 ```
 
 #### Logs methods
@@ -187,11 +217,19 @@ response = client.remove_all_jobs("service_name") # returns an http response
 
 #### Storage usage
 
-You can create a storage object to operate over the different storage providers defined on a service with the method `create_storage_client` as follows:
+You can create a storage object to operate over the different storage providers defined on a service with the method `create_storage_client`. This constructor returns a storage object with methos to interact with the storage providers.
+
+The default constructor, seen as follows, will create a provider to interact with the default MinIO instance through the user's credentials. 
+
+``` python
+storage_service = client.create_storage_client() # returns a storage object
+```
+Additionally, if you need to interact with specific storage providers defined on a service, the constructor accepts a `svc` parameter where you can state the service name from which to search for additional credentials.
 
 ``` python
 storage_service = client.create_storage_client("service_name") # returns a storage object
 ```
+
 > _Note_ : The `storage_provider` parameter on the storage methods follows the format: `["storage_provider_type"].["storage_provider_name"]` where `storage_provider_type` is one of the suported storage providers (minIO, S3, Onedata or webdav) and `storage_provider_name` is the identifier _(ex: minio.default)_
 
 **list_files_from_path**

oscar_python-1.3.1/oscar_python/_oidc.py

@@ -0,0 +1,96 @@
+"""
+Class to manage OIDC JWT tokens
+"""
+import json
+import base64
+import re
+import time
+import requests
+
+
+class OIDC(object):
+
+    @staticmethod
+    def _b64d(b):
+        """Decode some base64-encoded bytes.
+
+        Raises Exception if the string contains invalid characters or padding.
+
+        :param b: bytes
+        """
+
+        cb = b.rstrip(b"=")  # shouldn't but there you are
+
+        # Python's base64 functions ignore invalid characters, so we need to
+        # check for them explicitly.
+        b64_re = re.compile(b"^[A-Za-z0-9_-]*$")
+        if not b64_re.match(cb):
+            raise Exception(cb, "base64-encoded data contains illegal characters")
+
+        if cb == b:
+            b = OIDC._add_padding(b)
+
+        return base64.urlsafe_b64decode(b)
+
+    @staticmethod
+    def _add_padding(b):
+        # add padding chars
+        m = len(b) % 4
+        if m == 1:
+            # NOTE: for some reason b64decode raises *TypeError* if the
+            # padding is incorrect.
+            raise Exception(b, "incorrect padding")
+        elif m == 2:
+            b += b"=="
+        elif m == 3:
+            b += b"="
+        return b
+
+    @staticmethod
+    def get_info(token):
+        """
+        Unpacks a JWT into its parts and base64 decodes the parts
+        individually, returning the part 1 json decoded, where the
+        token info is stored.
+
+        :param token: The JWT token
+        """
+        part = tuple(token.encode("utf-8").split(b"."))
+        part = [OIDC._b64d(p) for p in part]
+        return json.loads(part[1].decode("utf-8"))
+
+    @staticmethod
+    def is_access_token_expired(token):
+        """
+        Check if the current access token is expired
+        """
+        try:
+            decoded_token = OIDC.get_info(token)
+            now = int(time.time())
+            expires = int(decoded_token['exp'])
+            validity = expires - now
+            if validity < 0:
+                return True
+            else:
+                return False
+        except Exception:
+            return True
+
+    @staticmethod
+    def refresh_access_token(refresh_token, scopes, token_endpoint):
+        """
+        Refresh the access token using the refresh token
+        """
+        data = {
+            'grant_type': 'refresh_token',
+            'refresh_token': refresh_token,
+            'client_id': 'token-portal',
+            'scope': ' '.join(scopes)
+        }
+
+        response = requests.post(token_endpoint, data=data)
+
+        if response.status_code == 200:
+            return response.json()['access_token']
+        else:
+            return None

{oscar_python-1.2.1 → oscar_python-1.3.1}/oscar_python/_providers/_minio.py

@@ -10,15 +10,16 @@
 # distributed under the License is distributed on an "AS IS" BASIS,
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
-# limitations under the License. 
+# limitations under the License.
 
 import boto3
 from oscar_python._providers._s3 import S3
 
 _DEFAULT_MINIO_ENDPOINT = 'http://minio-service.minio:9000'
 
+
 class Minio(S3):
-   
+
     def _get_client(self, c):
         """Return Minio client with user configuration."""
         if c["endpoint"] == '':
@@ -30,4 +31,4 @@ class Minio(S3):
                             region_name=c["region"],
                             verify=c["verify"],
                             aws_access_key_id=c["access_key"],
-                            aws_secret_access_key=c["secret_key"])
+                            aws_secret_access_key=c["secret_key"])

{oscar_python-1.2.1 → oscar_python-1.3.1}/oscar_python/_providers/_onedata.py

@@ -10,29 +10,30 @@
 # distributed under the License is distributed on an "AS IS" BASIS,
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
-# limitations under the License. 
+# limitations under the License.
 
 from oscar_python._providers._providers_base import StorageProvider
 import requests
 import json
 
-#TODO fix error returns
+
+# TODO fix error returns
 class Onedata(StorageProvider):
-    
+
     _CDMI_PATH = '/cdmi'
     _CDMI_VERSION_HEADER = {'X-CDMI-Specification-Version': '1.1.1'}
 
     def __init__(self, credentials) -> None:
         super().__init__()
         self._get_client(credentials)
-    
+
     def _get_client(self, c):
         self.oneprovider_space = c["space"]
         self.oneprovider_host = c["oneprovider_host"]
         self.headers = {'X-Auth-Token': c["token"]}
         self.url = (f'https://{self.oneprovider_host}{self._CDMI_PATH}/'
                     f'{self.oneprovider_space}/')
-    
+
     def upload_file(self, local_path, remote_path):
         file_name = local_path.split('/')[-1]
         if not self._folder_exists(remote_path):
@@ -57,12 +58,13 @@ class Onedata(StorageProvider):
         file_name = remote_path.split('/')[-1]
         url = self.url+remote_path
         res = requests.get(url=url, headers=self.headers)
-        if res.status_code == 200: print("Saving file to {0}/{1}".format(local_path, file_name))
+        if res.status_code == 200:
+            print("Saving file to {0}/{1}".format(local_path, file_name))
         self._save_file(local_path+"/"+file_name, res.content, mode='wb')
-    
+
     def list_files_from_path(self, path):
         headers = {**self._CDMI_VERSION_HEADER, **self.headers}
-        return requests.get(url = self.url+path+"/", headers=headers)
+        return requests.get(url=self.url+path+"/", headers=headers)
 
     def _save_file(self, path, content, mode='w'):
         with open(path, mode) as fwc:
@@ -72,12 +74,13 @@ class Onedata(StorageProvider):
 
     def _folder_exists(self, folder_name):
         headers = {**self._CDMI_VERSION_HEADER, **self.headers}
-        response = requests.get(url = self.url+folder_name+"/", headers=headers)
+        response = requests.get(url=self.url+folder_name+"/", headers=headers)
         if response.status_code == 200:
             return True
         return False
-    
+
     def _create_folder(self, folder_name):
-        response = requests.put(url = self.url+folder_name+"/", headers=self.headers)
-        if response.status_code != 201: 
-            return False
+        response = requests.put(url=self.url+folder_name+"/", headers=self.headers)
+        if response.status_code == 201:
+            return True
+        return False

{oscar_python-1.2.1 → oscar_python-1.3.1}/oscar_python/_providers/_providers_base.py

@@ -10,10 +10,11 @@
 # distributed under the License is distributed on an "AS IS" BASIS,
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
-# limitations under the License. 
+# limitations under the License.
 
 import abc
 
+
 class StorageProvider(abc.ABC):
     @abc.abstractmethod
     def download_file(self, local_path, remote_path):
@@ -22,11 +23,11 @@ class StorageProvider(abc.ABC):
     @abc.abstractmethod
     def upload_file(self, local_path, remote_path):
         """Generic method to be implemented by all the storage providers."""
-    
+
     @abc.abstractmethod
     def list_files_from_path(self, path):
         """Generic method to be implemented by all the storage providers."""
 
     @abc.abstractmethod
     def _get_client(self):
-        """Generic method to be implemented by all the storage providers."""
+        """Generic method to be implemented by all the storage providers."""

{oscar_python-1.2.1 → oscar_python-1.3.1}/oscar_python/_providers/_s3.py

@@ -10,19 +10,19 @@
 # distributed under the License is distributed on an "AS IS" BASIS,
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
-# limitations under the License. 
+# limitations under the License.
 
 from aiohttp import ClientError
-import logging
 import boto3
 from oscar_python._providers._providers_base import StorageProvider
 
+
 class S3(StorageProvider):
     def __init__(self, credentials) -> None:
         super().__init__()
         self.client = self._get_client(credentials)
         self.resource = self._get_resource(credentials)
-    
+
     def _get_client(self, c):
         """Returns S3 client with default configuration."""
         if c is None:
@@ -42,39 +42,41 @@ class S3(StorageProvider):
         if region == '':
             region = None
         return boto3.resource('s3',
-                            region_name=region,
-                            aws_access_key_id=c["access_key"],
-                            aws_secret_access_key=c["secret_key"])
+                              region_name=region,
+                              aws_access_key_id=c["access_key"],
+                              aws_secret_access_key=c["secret_key"])
 
     def list_files_from_path(self, path):
         prefix = ""
         path_split = path.split('/')
         bucket = path_split[0]
         if len(path_split) > 1:
-            prefix = path.split('/',2)[1]
-        print("Reading content from path:",path)
+            prefix = path.split('/', 2)[1]
+        print("Reading content from path:", path)
         return self.client.list_objects(Bucket=bucket, Prefix=prefix)
-    
+
     def upload_file(self, local_path, remote_path):
         bucket_name = remote_path.split('/')[0]
-        file_key = remote_path.split('/',1)[1]
+        file_key = remote_path.split('/', 1)[1]
         file_name = local_path.split('/')[-1]
         print("Uploading to bucket '{0}' with key '{1}'".format(bucket_name,file_key))
         with open(local_path, 'rb') as data:
             try:
-                self.client.upload_fileobj(data, bucket_name, file_key+"/"+file_name)
+                self.client.upload_fileobj(data, bucket_name, file_key + "/" + file_name)
+                return True
             except ClientError as err:
                 print("Error uploading file: ", err)
                 return False
 
     def download_file(self, local_path, remote_path):
         bucket_name = remote_path.split('/')[0]
-        file_key = remote_path.split('/',1)[1]
+        file_key = remote_path.split('/', 1)[1]
         file_path = local_path+"/"+remote_path.split('/')[-1]
         print("Downloading from bucket '{0}' to path '{1}' with key '{2}'".format(bucket_name, file_path, file_key))
         with open(file_path, 'wb') as data:
             try:
                 self.client.download_fileobj(bucket_name, file_key, data)
+                return True
             except ClientError as err:
                 print("Error downloading file: ", err)
-                return False
+                return False

{oscar_python-1.2.1 → oscar_python-1.3.1}/oscar_python/_providers/_webdav.py

@@ -10,12 +10,13 @@
 # distributed under the License is distributed on an "AS IS" BASIS,
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
-# limitations under the License. 
+# limitations under the License.
 
 from oscar_python._providers._providers_base import StorageProvider
 from webdav3.client import Client
 from webdav3.exceptions import WebDavException
 
+
 class WebDav(StorageProvider):
     def __init__(self, credentials) -> None:
         self.client = self._get_client(credentials)
@@ -26,9 +27,9 @@ class WebDav(StorageProvider):
         if 'https://' not in hostname:
             hostname = 'https://'+hostname
         options = {
-        'webdav_hostname': hostname,
-        'webdav_login':    c["login"],
-        'webdav_password': c["password"]
+            'webdav_hostname': hostname,
+            'webdav_login':    c["login"],
+            'webdav_password': c["password"]
         }
         return Client(options=options)
 
@@ -45,13 +46,12 @@ class WebDav(StorageProvider):
         except WebDavException as exception:
             print("error uploading file to path '{0}': {1}".format(remote_path+"/"+file_name, exception))
 
-    
     def download_file(self, local_path, remote_path):
         file_name = remote_path.split('/')[-1]
         try:
             self.client.download_sync(remote_path, local_path+"/"+file_name)
         except WebDavException as exception:
             print("error downloading file from path '{0}': {1}".format(remote_path, exception))
-    
+
     def list_files_from_path(self, path):
-        return self.client.list(path)
+        return self.client.list(path)

{oscar_python-1.2.1 → oscar_python-1.3.1}/oscar_python/_utils.py

@@ -10,7 +10,7 @@
 # distributed under the License is distributed on an "AS IS" BASIS,
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
-# limitations under the License. 
+# limitations under the License.
 
 import base64
 import os
@@ -18,18 +18,19 @@ import requests
 import liboidcagent as agent
 _DEFAULT_TIMEOUT = 60
 
-""" Generic http request """
-def make_request(c , path, method, **kwargs):
 
-    headers = get_headers(c)  
+def make_request(c, path, method, **kwargs):
+    """ Generic http request """
 
-    if "timeout" in kwargs.keys() and kwargs["timeout"]: 
+    headers = get_headers(c)
+
+    if "timeout" in kwargs.keys() and kwargs["timeout"]:
         timeout = kwargs["timeout"]
-    else: 
+    else:
         timeout = _DEFAULT_TIMEOUT
 
     url = c.endpoint+path
-    
+
     if method in ["post", "put"]:
         if "token" in kwargs.keys() and kwargs["token"]: 
             headers = get_headers_with_token(kwargs["token"])
@@ -40,37 +41,42 @@ def make_request(c , path, method, **kwargs):
 
     if "handle" in kwargs.keys() and kwargs["handle"] == False:
         return result
-    
+
     result.raise_for_status()
     return result
 
-""" Function to generate headers with basic authentication or OIDC """
+
 def get_headers(c):
+    """ Function to generate headers with basic authentication or OIDC """
     if c._AUTH_TYPE == "basicauth":
-        usr_pass_as_bytes = bytes(c.user+":"+c.password,"utf-8")
+        usr_pass_as_bytes = bytes(c.user + ":" + c.password, "utf-8")
         usr_pass_base_64 = base64.b64encode(usr_pass_as_bytes).decode("utf-8")
-        return {"Authorization": "Basic "+ usr_pass_base_64}
+        return {"Authorization": "Basic " + usr_pass_base_64}
     if c._AUTH_TYPE == "oidc-agent":
         token = agent.get_access_token(c.shortname)
         return get_headers_with_token(token)
     if c._AUTH_TYPE == "oidc":
-        return get_headers_with_token(c.oidc_token)
+        return get_headers_with_token(c.get_access_token())
+
 
-""" Function to generate headers with token auth """
 def get_headers_with_token(token):
-    return {"Authorization": "Bearer "+ str(token)}
+    """ Function to generate headers with token auth """
+    return {"Authorization": "Bearer " + str(token)}
+
 
 def write_text_file(content, file_path):
     with open(file_path, 'w') as f:
         f.write(content)
 
+
 def isBase64(st):
     try:
         base64.b64decode(st)
         return True
-    except:
+    except Exception:
         return False
 
+
 def decode_b64(b64_str, file_out):
     file_extension = os.path.splitext(file_out)[1]
     try:
@@ -80,7 +86,7 @@ def decode_b64(b64_str, file_out):
             decode = 'w'
             decoded_data = decoded_data.decode("utf-8")
         else:
-            decode = 'wb' 
+            decode = 'wb'
 
         with open(file_out, decode) as f:
             f.write(decoded_data)
@@ -90,6 +96,7 @@ def decode_b64(b64_str, file_out):
     except OSError:
         print('Error decoding output: Failed to write decoded data to file.')   
 
+
 def encode_input(data):
     if os.path.isfile(data):
         try:
@@ -103,12 +110,11 @@ def encode_input(data):
         message_bytes = data.encode('ascii')
         return base64.b64encode(message_bytes)
 
+
 def decode_output(output, file_path):
-    if(isBase64(output)):
+    if isBase64(output):
         decode_b64(output, file_path)
         return
-    if(isinstance(output,str)):
-        write_text_file(output,file_path)
+    if isinstance(output, str):
+        write_text_file(output, file_path)
         return
-
-