oscar-python 1.3.0__tar.gz → 1.3.2__tar.gz

{oscar_python-1.3.0 → oscar_python-1.3.2}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.2
+Metadata-Version: 2.4
 Name: oscar_python
-Version: 1.3.0
+Version: 1.3.2
 Summary: OSCAR API for python
 Home-page: https://github.com/grycap/oscar_python
 Author: GRyCAP - Universitat Politecnica de Valencia
@@ -24,12 +24,13 @@ Dynamic: description
 Dynamic: description-content-type
 Dynamic: home-page
 Dynamic: license
+Dynamic: license-file
 Dynamic: requires-dist
 Dynamic: summary
 
 ## Python OSCAR client
 
-[![Build](https://github.com/grycap/oscar_python/actions/workflows/main.yaml/badge.svg)](https://github.com/grycap/oscar_python/actions/workflows/main.yaml)
+[![Build](https://github.com/grycap/oscar_python/actions/workflows/release.yaml/badge.svg)](https://github.com/grycap/oscar_python/actions/workflows/main.yaml)
 ![PyPI](https://img.shields.io/pypi/v/oscar_python)
 
 This package provides a client to interact with OSCAR (https://oscar.grycap.net) clusters and services. It is available on Pypi with the name [oscar-python](https://pypi.org/project/oscar-python/).
@@ -84,6 +85,33 @@ If you already have a valid token, you can use the parameter `oidc_token` instea
 ```
 An example of using a generated token is if you want to use EGI Notebooks. Since you can't use oidc-agent on the Notebook, you can make use of the generated token that EGI provides on path `/var/run/secrets/egi.eu/access_token`.
 
+If you have a valid refresh token (long live token), you can use the parameter `refresh_token` instead.
+
+``` python
+  options_oidc_auth = {'cluster_id':'cluster-id',
+                'endpoint':'https://cluster-endpoint',
+                'refresh_token':'token',
+                'ssl':'True'}
+                
+  client = Client(options = options_oidc_auth)
+```
+
+You can get a refresh token from EGI Check-In using the [Token Portal](https://aai.egi.eu/token).
+
+In case of using other OIDC provider you must provide two additional parameters `token_endpoint`
+and `scopes`:
+
+``` python
+  options_oidc_auth = {'cluster_id':'cluster-id',
+                'endpoint':'https://cluster-endpoint',
+                'refresh_token':'token',
+                'scopes': ["openid", "profile", "email"],
+                'token_endpoint': "http://issuer.com/token",
+                'ssl':'True'}
+                
+  client = Client(options = options_oidc_auth)
+```
+
 ### Sample usage
 
 - Sample code that creates a client and gets information about the cluster
@@ -219,11 +247,19 @@ response = client.remove_all_jobs("service_name") # returns an http response
 
 #### Storage usage
 
-You can create a storage object to operate over the different storage providers defined on a service with the method `create_storage_client` as follows:
+You can create a storage object to operate over the different storage providers defined on a service with the method `create_storage_client`. This constructor returns a storage object with methos to interact with the storage providers.
+
+The default constructor, seen as follows, will create a provider to interact with the default MinIO instance through the user's credentials. 
+
+``` python
+storage_service = client.create_storage_client() # returns a storage object
+```
+Additionally, if you need to interact with specific storage providers defined on a service, the constructor accepts a `svc` parameter where you can state the service name from which to search for additional credentials.
 
 ``` python
 storage_service = client.create_storage_client("service_name") # returns a storage object
 ```
+
 > _Note_ : The `storage_provider` parameter on the storage methods follows the format: `["storage_provider_type"].["storage_provider_name"]` where `storage_provider_type` is one of the suported storage providers (minIO, S3, Onedata or webdav) and `storage_provider_name` is the identifier _(ex: minio.default)_
 
 **list_files_from_path**
@@ -232,7 +268,7 @@ This method returns a JSON with the info except for OneData, which returns an HT
 
 ``` python
 # get a list of the files of one of the service storage provider 
-files = storage_service.list_files_from_path("storage_provider") # returns json
+files = storage_service.list_files_from_path("storage_provider", "remote_path") # returns json
 ```
 
 **upload_file**

{oscar_python-1.3.0 → oscar_python-1.3.2}/README.md

@@ -1,6 +1,6 @@
 ## Python OSCAR client
 
-[![Build](https://github.com/grycap/oscar_python/actions/workflows/main.yaml/badge.svg)](https://github.com/grycap/oscar_python/actions/workflows/main.yaml)
+[![Build](https://github.com/grycap/oscar_python/actions/workflows/release.yaml/badge.svg)](https://github.com/grycap/oscar_python/actions/workflows/main.yaml)
 ![PyPI](https://img.shields.io/pypi/v/oscar_python)
 
 This package provides a client to interact with OSCAR (https://oscar.grycap.net) clusters and services. It is available on Pypi with the name [oscar-python](https://pypi.org/project/oscar-python/).
@@ -55,6 +55,33 @@ If you already have a valid token, you can use the parameter `oidc_token` instea
 ```
 An example of using a generated token is if you want to use EGI Notebooks. Since you can't use oidc-agent on the Notebook, you can make use of the generated token that EGI provides on path `/var/run/secrets/egi.eu/access_token`.
 
+If you have a valid refresh token (long live token), you can use the parameter `refresh_token` instead.
+
+``` python
+  options_oidc_auth = {'cluster_id':'cluster-id',
+                'endpoint':'https://cluster-endpoint',
+                'refresh_token':'token',
+                'ssl':'True'}
+                
+  client = Client(options = options_oidc_auth)
+```
+
+You can get a refresh token from EGI Check-In using the [Token Portal](https://aai.egi.eu/token).
+
+In case of using other OIDC provider you must provide two additional parameters `token_endpoint`
+and `scopes`:
+
+``` python
+  options_oidc_auth = {'cluster_id':'cluster-id',
+                'endpoint':'https://cluster-endpoint',
+                'refresh_token':'token',
+                'scopes': ["openid", "profile", "email"],
+                'token_endpoint': "http://issuer.com/token",
+                'ssl':'True'}
+                
+  client = Client(options = options_oidc_auth)
+```
+
 ### Sample usage
 
 - Sample code that creates a client and gets information about the cluster
@@ -190,11 +217,19 @@ response = client.remove_all_jobs("service_name") # returns an http response
 
 #### Storage usage
 
-You can create a storage object to operate over the different storage providers defined on a service with the method `create_storage_client` as follows:
+You can create a storage object to operate over the different storage providers defined on a service with the method `create_storage_client`. This constructor returns a storage object with methos to interact with the storage providers.
+
+The default constructor, seen as follows, will create a provider to interact with the default MinIO instance through the user's credentials. 
+
+``` python
+storage_service = client.create_storage_client() # returns a storage object
+```
+Additionally, if you need to interact with specific storage providers defined on a service, the constructor accepts a `svc` parameter where you can state the service name from which to search for additional credentials.
 
 ``` python
 storage_service = client.create_storage_client("service_name") # returns a storage object
 ```
+
 > _Note_ : The `storage_provider` parameter on the storage methods follows the format: `["storage_provider_type"].["storage_provider_name"]` where `storage_provider_type` is one of the suported storage providers (minIO, S3, Onedata or webdav) and `storage_provider_name` is the identifier _(ex: minio.default)_
 
 **list_files_from_path**
@@ -203,7 +238,7 @@ This method returns a JSON with the info except for OneData, which returns an HT
 
 ``` python
 # get a list of the files of one of the service storage provider 
-files = storage_service.list_files_from_path("storage_provider") # returns json
+files = storage_service.list_files_from_path("storage_provider", "remote_path") # returns json
 ```
 
 **upload_file**

oscar_python-1.3.2/oscar_python/_oidc.py

@@ -0,0 +1,96 @@
+"""
+Class to manage OIDC JWT tokens
+"""
+import json
+import base64
+import re
+import time
+import requests
+
+
+class OIDC(object):
+
+    @staticmethod
+    def _b64d(b):
+        """Decode some base64-encoded bytes.
+
+        Raises Exception if the string contains invalid characters or padding.
+
+        :param b: bytes
+        """
+
+        cb = b.rstrip(b"=")  # shouldn't but there you are
+
+        # Python's base64 functions ignore invalid characters, so we need to
+        # check for them explicitly.
+        b64_re = re.compile(b"^[A-Za-z0-9_-]*$")
+        if not b64_re.match(cb):
+            raise Exception(cb, "base64-encoded data contains illegal characters")
+
+        if cb == b:
+            b = OIDC._add_padding(b)
+
+        return base64.urlsafe_b64decode(b)
+
+    @staticmethod
+    def _add_padding(b):
+        # add padding chars
+        m = len(b) % 4
+        if m == 1:
+            # NOTE: for some reason b64decode raises *TypeError* if the
+            # padding is incorrect.
+            raise Exception(b, "incorrect padding")
+        elif m == 2:
+            b += b"=="
+        elif m == 3:
+            b += b"="
+        return b
+
+    @staticmethod
+    def get_info(token):
+        """
+        Unpacks a JWT into its parts and base64 decodes the parts
+        individually, returning the part 1 json decoded, where the
+        token info is stored.
+
+        :param token: The JWT token
+        """
+        part = tuple(token.encode("utf-8").split(b"."))
+        part = [OIDC._b64d(p) for p in part]
+        return json.loads(part[1].decode("utf-8"))
+
+    @staticmethod
+    def is_access_token_expired(token):
+        """
+        Check if the current access token is expired
+        """
+        try:
+            decoded_token = OIDC.get_info(token)
+            now = int(time.time())
+            expires = int(decoded_token['exp'])
+            validity = expires - now
+            if validity < 0:
+                return True
+            else:
+                return False
+        except Exception:
+            return True
+
+    @staticmethod
+    def refresh_access_token(refresh_token, scopes, token_endpoint):
+        """
+        Refresh the access token using the refresh token
+        """
+        data = {
+            'grant_type': 'refresh_token',
+            'refresh_token': refresh_token,
+            'client_id': 'token-portal',
+            'scope': ' '.join(scopes)
+        }
+
+        response = requests.post(token_endpoint, data=data)
+
+        if response.status_code == 200:
+            return response.json()['access_token']
+        else:
+            return None

{oscar_python-1.3.0 → oscar_python-1.3.2}/oscar_python/_utils.py

@@ -56,7 +56,7 @@ def get_headers(c):
         token = agent.get_access_token(c.shortname)
         return get_headers_with_token(token)
     if c._AUTH_TYPE == "oidc":
-        return get_headers_with_token(c.oidc_token)
+        return get_headers_with_token(c.get_access_token())
 
 
 def get_headers_with_token(token):

{oscar_python-1.3.0 → oscar_python-1.3.2}/oscar_python/client.py

@@ -18,6 +18,7 @@ import os
 import yaml
 import liboidcagent as agent
 import oscar_python._utils as utils
+from oscar_python._oidc import OIDC
 from oscar_python.default_client import DefaultClient
 from oscar_python.storage import Storage
 
@@ -37,10 +38,16 @@ _POST = "post"
 _PUT = "put"
 _DELETE = "delete"
 
+# Default values for OIDC refresh token using EGI CheckIn
+_DEFAULT_SCOPES = ['openid', 'email', 'profile', 'voperson_id', 'eduperson_entitlement']
+_DEFAULT_TOKEN_ENDPOINT = 'https://aai.egi.eu/auth/realms/egi/protocol/openid-connect/token'
+
 
 class Client(DefaultClient):
     # Cluster info
     def __init__(self, options) -> None:
+        self.id = options['cluster_id']
+        self.endpoint = options['endpoint']
         self.set_auth_type(options)
         if self._AUTH_TYPE == 'basicauth':
             self.basic_auth_client(options)
@@ -50,22 +57,20 @@ class Client(DefaultClient):
             self.oidc_client(options)
 
     def basic_auth_client(self, options):
-        self.id = options['cluster_id']
-        self.endpoint = options['endpoint']
         self.user = options['user']
         self.password = options['password']
         self.ssl = bool(options['ssl'])
 
     def oidc_agent_client(self, options):
-        self.id = options['cluster_id']
-        self.endpoint = options['endpoint']
         self.shortname = options['shortname']
         self.ssl = bool(options['ssl'])
 
     def oidc_client(self, options):
-        self.id = options['cluster_id']
-        self.endpoint = options['endpoint']
-        self.oidc_token = options['oidc_token']
+        self.oidc_token = options.get('oidc_token')
+        self.refresh_token = options.get('refresh_token')
+        self.scopes = options.get('scopes', _DEFAULT_SCOPES)
+        self.token_endpoint = options.get('token_endpoint',
+                                          _DEFAULT_TOKEN_ENDPOINT)
         self.ssl = bool(options['ssl'])
 
     def set_auth_type(self, options):
@@ -77,18 +82,28 @@ class Client(DefaultClient):
                 agent.get_access_token(options['shortname'])
             except agent.OidcAgentError as e:
                 print("ERROR oidc-agent: {}".format(e))
-        elif 'oidc_token' in options:
+        elif 'oidc_token' in options or 'refresh_token' in options:
             self._AUTH_TYPE = "oidc"
         else:
             raise ValueError("Unrecognized authentication credentials in options")
 
+    def get_access_token(self):
+        if self.refresh_token and OIDC.is_access_token_expired(self.oidc_token):
+            self.oidc_token = OIDC.refresh_access_token(self.refresh_token,
+                                                        self.scopes,
+                                                        self.token_endpoint)
+        return self.oidc_token
+
     """ Creates a generic storage client to interact with the storage providers
     defined on a specific service of the refered OSCAR cluster """
-    def create_storage_client(self, svc):
-        return Storage(
-                client_obj=self,
-                svc_name=svc)
-
+    def create_storage_client(self, svc=None):
+        if svc != None:
+            return Storage(
+                client_obj=self, svc_name=svc)
+        else:
+            return Storage(
+                    client_obj=self)
+        
     """ Function to get cluster info """
     def get_cluster_info(self):
         return utils.make_request(self, _INFO_PATH, _GET)

{oscar_python-1.3.0 → oscar_python-1.3.2}/oscar_python/storage.py

@@ -23,21 +23,37 @@ _MINIO = "minio"
 _S3 = "s3"
 _ONE_DATA = "onedata"
 _WEBDAV = "webdav"
+
+_GET = "get"
+_CONFIG_PATH = "/system/config"
 _SVC_PATH = "/system/services"
 
 
 # TODO check returns from functions
 class Storage:
-    def __init__(self, client_obj, svc_name) -> None:
+    def __init__(self, client_obj, svc_name = None) -> None:
         self.client_obj = client_obj
-        self.svc_name = svc_name
-        self._store_providers()
+        self.storage_providers = {}
+        if svc_name != None:
+            self.svc_name = svc_name
+            self._store_provider_from_service()
+        self._store_default_minio_provider()
 
     """ Function to store all the providers of the service """
-    def _store_providers(self):
-        svc = utils.make_request(self.client_obj, _SVC_PATH + "/" + self.svc_name, "get")
+    def _store_provider_from_service(self):
+        svc = utils.make_request(self.client_obj, _SVC_PATH + "/" + self.svc_name, _GET)
         self.storage_providers = json.loads(svc.text)["storage_providers"]
 
+    """ Function to store the user credentials for the default MinIO provider """
+    def _store_default_minio_provider(self):
+        config = utils.make_request(self.client_obj, _CONFIG_PATH + "/" , _GET)
+        if _MINIO in self.storage_providers:
+            self.storage_providers[_MINIO]["default"] = json.loads(config.text)["minio_provider"]
+        else:
+            default = {"default": json.loads(config.text)["minio_provider"]}
+            self.storage_providers[_MINIO] = default
+        
+
     """ Function to retreive credentials of a specific storage provider """
     def _get_provider_creds(self, provider, provider_name):
         return self.storage_providers[provider][provider_name]

{oscar_python-1.3.0 → oscar_python-1.3.2}/oscar_python.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.2
+Metadata-Version: 2.4
 Name: oscar_python
-Version: 1.3.0
+Version: 1.3.2
 Summary: OSCAR API for python
 Home-page: https://github.com/grycap/oscar_python
 Author: GRyCAP - Universitat Politecnica de Valencia
@@ -24,12 +24,13 @@ Dynamic: description
 Dynamic: description-content-type
 Dynamic: home-page
 Dynamic: license
+Dynamic: license-file
 Dynamic: requires-dist
 Dynamic: summary
 
 ## Python OSCAR client
 
-[![Build](https://github.com/grycap/oscar_python/actions/workflows/main.yaml/badge.svg)](https://github.com/grycap/oscar_python/actions/workflows/main.yaml)
+[![Build](https://github.com/grycap/oscar_python/actions/workflows/release.yaml/badge.svg)](https://github.com/grycap/oscar_python/actions/workflows/main.yaml)
 ![PyPI](https://img.shields.io/pypi/v/oscar_python)
 
 This package provides a client to interact with OSCAR (https://oscar.grycap.net) clusters and services. It is available on Pypi with the name [oscar-python](https://pypi.org/project/oscar-python/).
@@ -84,6 +85,33 @@ If you already have a valid token, you can use the parameter `oidc_token` instea
 ```
 An example of using a generated token is if you want to use EGI Notebooks. Since you can't use oidc-agent on the Notebook, you can make use of the generated token that EGI provides on path `/var/run/secrets/egi.eu/access_token`.
 
+If you have a valid refresh token (long live token), you can use the parameter `refresh_token` instead.
+
+``` python
+  options_oidc_auth = {'cluster_id':'cluster-id',
+                'endpoint':'https://cluster-endpoint',
+                'refresh_token':'token',
+                'ssl':'True'}
+                
+  client = Client(options = options_oidc_auth)
+```
+
+You can get a refresh token from EGI Check-In using the [Token Portal](https://aai.egi.eu/token).
+
+In case of using other OIDC provider you must provide two additional parameters `token_endpoint`
+and `scopes`:
+
+``` python
+  options_oidc_auth = {'cluster_id':'cluster-id',
+                'endpoint':'https://cluster-endpoint',
+                'refresh_token':'token',
+                'scopes': ["openid", "profile", "email"],
+                'token_endpoint': "http://issuer.com/token",
+                'ssl':'True'}
+                
+  client = Client(options = options_oidc_auth)
+```
+
 ### Sample usage
 
 - Sample code that creates a client and gets information about the cluster
@@ -219,11 +247,19 @@ response = client.remove_all_jobs("service_name") # returns an http response
 
 #### Storage usage
 
-You can create a storage object to operate over the different storage providers defined on a service with the method `create_storage_client` as follows:
+You can create a storage object to operate over the different storage providers defined on a service with the method `create_storage_client`. This constructor returns a storage object with methos to interact with the storage providers.
+
+The default constructor, seen as follows, will create a provider to interact with the default MinIO instance through the user's credentials. 
+
+``` python
+storage_service = client.create_storage_client() # returns a storage object
+```
+Additionally, if you need to interact with specific storage providers defined on a service, the constructor accepts a `svc` parameter where you can state the service name from which to search for additional credentials.
 
 ``` python
 storage_service = client.create_storage_client("service_name") # returns a storage object
 ```
+
 > _Note_ : The `storage_provider` parameter on the storage methods follows the format: `["storage_provider_type"].["storage_provider_name"]` where `storage_provider_type` is one of the suported storage providers (minIO, S3, Onedata or webdav) and `storage_provider_name` is the identifier _(ex: minio.default)_
 
 **list_files_from_path**
@@ -232,7 +268,7 @@ This method returns a JSON with the info except for OneData, which returns an HT
 
 ``` python
 # get a list of the files of one of the service storage provider 
-files = storage_service.list_files_from_path("storage_provider") # returns json
+files = storage_service.list_files_from_path("storage_provider", "remote_path") # returns json
 ```
 
 **upload_file**

{oscar_python-1.3.0 → oscar_python-1.3.2}/oscar_python.egg-info/SOURCES.txt

@@ -2,6 +2,7 @@ LICENSE
 README.md
 setup.py
 oscar_python/__init__.py
+oscar_python/_oidc.py
 oscar_python/_utils.py
 oscar_python/client.py
 oscar_python/client_anon.py
@@ -21,6 +22,7 @@ oscar_python/_providers/_s3.py
 oscar_python/_providers/_webdav.py
 tests/test_client.py
 tests/test_default_client.py
+tests/test_oidc.py
 tests/test_onedata.py
 tests/test_s3.py
 tests/test_storage.py

{oscar_python-1.3.0 → oscar_python-1.3.2}/tests/test_client.py

@@ -44,6 +44,15 @@ def test_oidc_client(options):
     assert client.oidc_token == options['oidc_token']
     assert client.ssl == options['ssl']
 
+    del options['oidc_token']
+    options['refresh_token'] = 'test_refresh_token'
+    options['scopes'] = ['openid', 'profile', 'email']
+    options['token_endpoint'] = 'test_token_endpoint'
+    client = Client(options)
+    assert client.refresh_token == options['refresh_token']
+    assert client.scopes == ['openid', 'profile', 'email']
+    assert client.token_endpoint == options['token_endpoint']
+
 
 def test_set_auth_type(options):
     client = Client(options)

oscar_python-1.3.2/tests/test_oidc.py

@@ -0,0 +1,38 @@
+from unittest.mock import patch, MagicMock
+from oscar_python._oidc import OIDC
+
+
+def test_is_access_token_expired():
+    token = ("eyJraWQiOiJyc2ExIiwiYWxnIjoiUlMyNTYifQ.eyJzdWIiOiJkYzVkNWFiNy02ZGI5LTQwNzktOTg1Yy04MGFjMDUwMTcw"
+             "NjYiLCJpc3MiOiJodHRwczpcL1wvaWFtLXRlc3QuaW5kaWdvLWRhdGFjbG91ZC5ldVwvIiwiZXhwIjoxNDY2MDkzOTE3LCJ"
+             "pYXQiOjE0NjYwOTAzMTcsImp0aSI6IjE1OTU2N2U2LTdiYzItNDUzOC1hYzNhLWJjNGU5MmE1NjlhMCJ9.eINKxJa2J--xd"
+             "GAZWIOKtx9Wi0Vz3xHzaSJWWY-UHWy044TQ5xYtt0VTvmY5Af-ngwAMGfyaqAAvNn1VEP-_fMYQZdwMqcXLsND4KkDi1ygiC"
+             "IwQ3JBz9azBT1o_oAHE5BsPsE2BjfDoVRasZxxW5UoXCmBslonYd8HK2tUVjz0")
+    assert OIDC.is_access_token_expired(token)
+
+
+def test_refresh_access_token():
+    mock_response = MagicMock()
+    mock_response.status_code = 200
+    mock_response.json.return_value = {
+        "access_token": "new_access_token",
+        "expires_in": 3600,
+        "refresh_token": "new_refresh_token"
+    }
+
+    with patch("requests.post") as mock_post:
+        mock_post.return_value = mock_response
+        access_token = OIDC.refresh_access_token("old_refresh_token",
+                                                 ["openid", "profile", "email"],
+                                                 "http://test.com/token")
+
+        assert access_token == "new_access_token"
+        mock_post.assert_called_once_with(
+            "http://test.com/token",
+            data={
+                "grant_type": "refresh_token",
+                "refresh_token": "old_refresh_token",
+                "client_id": "token-portal",
+                "scope": "openid profile email"
+            }
+        )

{oscar_python-1.3.0 → oscar_python-1.3.2}/tests/test_storage.py

@@ -11,9 +11,9 @@ def mock_client_obj():
 @pytest.fixture
 def storage(mock_client_obj):
     mock_response = MagicMock()
-    mock_response.text = '{"storage_providers": {"minio": {"default": {"access_key": "key","secret_key": "secret", "endpoint": "http://test.endpoint", "region": "us-east-1", "verify": false}}}}'
+    mock_response.text = '{"minio_provider": {"access_key": "key","secret_key": "secret", "endpoint": "http://test.endpoint", "region": "us-east-1", "verify": false}}'
     with patch('oscar_python._utils.make_request', return_value=mock_response):
-        return Storage(mock_client_obj, "test_service")
+        return Storage(mock_client_obj)
 
 
 def test_get_provider_creds(storage):

{oscar_python-1.3.0 → oscar_python-1.3.2}/tests/test_utils.py

@@ -29,7 +29,9 @@ def test_get_headers_with_oidc_agent():
 def test_get_headers_with_oidc():
     class MockClient:
         _AUTH_TYPE = "oidc"
-        oidc_token = "test_oidc_token"
+
+        def get_access_token(self):
+            return "test_oidc_token"
 
     c = MockClient()
     headers = utils.get_headers(c)