datahub-api-connector 1.0__tar.gz

datahub_api_connector-1.0/LICENSE

@@ -0,0 +1,19 @@
+Copyright (c) 2025 Verity
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

datahub_api_connector-1.0/PKG-INFO

@@ -0,0 +1,78 @@
+Metadata-Version: 2.4
+Name: datahub-api-connector
+Version: 1.0
+Summary: Package to help connect to Verity's Data Hub API
+Author-email: Verity <support@opinum.com>
+Maintainer-email: Verity <support@opinum.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/opinum/datahub-api-connector
+Project-URL: Bug Tracker, https://github.com/opinum/datahub-api-connector/issues
+Keywords: datahub,Verity,Opinum,API,Keycloak
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Programming Language :: Python
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests
+Requires-Dist: requests_oauthlib
+Requires-Dist: oauthlib
+Dynamic: license-file
+
+This package simplifies the calls to Data Hub's API.
+
+No magic. You need to follow the [Data Hub Swagger Documentation](https://api.opinum.com) for correct formatting of your requests
+
+Be aware that this is a new package, certainly missing a lot of features. Feel free to contribute.
+
+You first need to create an instance of the ApiConnector class with following parameters:
+
+*ATTENTION:* this package can only be used from Data Hub version 7.0 (1 July 2025) onward, as it uses the authentication to the the new technology (Keycloak).
+For previous versions, the opinum-api-connector package can be used (https://github.com/opinum/opinum-api-connector) instead.
+
+> _environment_
+> > a dictionary of environment variables
+> >
+> > if `None`, ApiConnector uses your environment variables (_os.environ_)
+> >
+> > Mandatory environment variables are:
+> >
+> > * _DATAHUB_USERNAME_: the Datahub user. <br>
+> > TAKE CARE: if this user has access to multiple tenants and if you do not specify a tenant id,
+> > ApiConnector will use the last tenant used.
+> > * _DATAHUB_PASSWORD_: the password for the user
+> > * _DATAHUB_CLIENT_ID_: the client id for accessing the API
+> > * _DATAHUB_CLIENT_SECRET_ the corresponding secret
+> > 
+> > Optional environment variables are:
+> >
+> > * _DATAHUB_API_URL_: another API URL than the Europe SaaS one (https://api.opinum.com)
+> > * _DATAHUB_AUTH_URL_: another authentication URL than the Europe SaaS one (https://auth.opinum.com)
+> > * _DATAHUB_PUSH_URL_: another push URL than the Europe SaaS one (https://push.opinum.com)
+> > * _DATAHUB_SCOPE_: the scope of you session (default: "_datahub-api_")<br>
+> > if you want to push data, the scope should be "_datahub-api push-data_"
+
+> _account_id_
+> > one of the tenant ids available for the Datahub user (default: `None`)
+
+> _retries_when_connection_failure_
+> > number of extra attempts when no 200 or 204 return code (default: 0, maximum: 5)
+
+Once you have your ApiConnector instance, you may use the class methods
+
+* get
+* post
+* patch
+* put
+* delete
+* send_file_to_storage
+
+All keyword arguments will be converted to path parameters in the API call with one important exception,
+the _data_ keyword referring to the body of your call.
+
+There are two other class methods for data pushing because we have another API for this
+
+* push_data
+* push_dataframe_data
+
+There is a little bit of magic with the method multi_thread_request_on_path that splits a list of parameters
+Allowing to make parallel calls.

datahub_api_connector-1.0/README.md

@@ -0,0 +1,58 @@
+This package simplifies the calls to Data Hub's API.
+
+No magic. You need to follow the [Data Hub Swagger Documentation](https://api.opinum.com) for correct formatting of your requests
+
+Be aware that this is a new package, certainly missing a lot of features. Feel free to contribute.
+
+You first need to create an instance of the ApiConnector class with following parameters:
+
+*ATTENTION:* this package can only be used from Data Hub version 7.0 (1 July 2025) onward, as it uses the authentication to the the new technology (Keycloak).
+For previous versions, the opinum-api-connector package can be used (https://github.com/opinum/opinum-api-connector) instead.
+
+> _environment_
+> > a dictionary of environment variables
+> >
+> > if `None`, ApiConnector uses your environment variables (_os.environ_)
+> >
+> > Mandatory environment variables are:
+> >
+> > * _DATAHUB_USERNAME_: the Datahub user. <br>
+> > TAKE CARE: if this user has access to multiple tenants and if you do not specify a tenant id,
+> > ApiConnector will use the last tenant used.
+> > * _DATAHUB_PASSWORD_: the password for the user
+> > * _DATAHUB_CLIENT_ID_: the client id for accessing the API
+> > * _DATAHUB_CLIENT_SECRET_ the corresponding secret
+> > 
+> > Optional environment variables are:
+> >
+> > * _DATAHUB_API_URL_: another API URL than the Europe SaaS one (https://api.opinum.com)
+> > * _DATAHUB_AUTH_URL_: another authentication URL than the Europe SaaS one (https://auth.opinum.com)
+> > * _DATAHUB_PUSH_URL_: another push URL than the Europe SaaS one (https://push.opinum.com)
+> > * _DATAHUB_SCOPE_: the scope of you session (default: "_datahub-api_")<br>
+> > if you want to push data, the scope should be "_datahub-api push-data_"
+
+> _account_id_
+> > one of the tenant ids available for the Datahub user (default: `None`)
+
+> _retries_when_connection_failure_
+> > number of extra attempts when no 200 or 204 return code (default: 0, maximum: 5)
+
+Once you have your ApiConnector instance, you may use the class methods
+
+* get
+* post
+* patch
+* put
+* delete
+* send_file_to_storage
+
+All keyword arguments will be converted to path parameters in the API call with one important exception,
+the _data_ keyword referring to the body of your call.
+
+There are two other class methods for data pushing because we have another API for this
+
+* push_data
+* push_dataframe_data
+
+There is a little bit of magic with the method multi_thread_request_on_path that splits a list of parameters
+Allowing to make parallel calls.

datahub_api_connector-1.0/pyproject.toml

@@ -0,0 +1,32 @@
+[build-system]
+requires = ['setuptools>= 77.0.3']
+build-backend = 'setuptools.build_meta'
+
+[project]
+name = "datahub-api-connector"
+version = "1.0"
+dependencies = [
+    "requests",
+    "requests_oauthlib",
+    "oauthlib"
+]
+requires-python = ">=3.10"
+authors = [
+  {name = "Verity", email = "support@opinum.com"}
+]
+maintainers = [
+  {name = "Verity", email = "support@opinum.com"}
+]
+description = "Package to help connect to Verity's Data Hub API"
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+keywords = ["datahub", "Verity", "Opinum", "API", "Keycloak"]
+classifiers = [
+  "Development Status :: 5 - Production/Stable",
+  "Programming Language :: Python"
+]
+
+[project.urls]
+Homepage = "https://github.com/opinum/datahub-api-connector"
+"Bug Tracker" = "https://github.com/opinum/datahub-api-connector/issues"

datahub_api_connector-1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

datahub_api_connector-1.0/src/datahub_api_connector/__init__.py

@@ -0,0 +1,248 @@
+import os
+from oauthlib.oauth2 import LegacyApplicationClient
+from requests_oauthlib import OAuth2Session
+import json
+import requests
+import datetime as dt
+import logging
+from time import sleep
+import concurrent.futures
+
+DEFAULT_API_URL = 'https://api.opinum.com'
+DEFAULT_AUTH_URL = 'https://auth.opinum.com'
+DEFAULT_SCOPE = 'datahub-api'
+DEFAULT_PUSH_URL = 'https://push.opinum.com'
+
+
+class ApiConnector:
+    """
+    A class for connection to Data Hub API
+
+    :param environment: a dictionary with all environment variables
+    :param account_id: the account id to use (for users having access to multiple tenants)
+    :param retries_when_connection_failure: allows to make several attempts to have a successful query (connection issues can happen)
+    """
+
+    time_limit = 3 * 60  # Three minutes
+
+    MAX_RETRIES_WHEN_CONNECTION_FAILURE = 5
+
+    def __init__(self,
+                 environment=None,
+                 account_id=None,
+                 retries_when_connection_failure=0,
+                 seconds_between_retries=5):
+        self.environment = os.environ if environment is None else environment
+        self.api_url = self.environment.get('DATAHUB_API_URL', self.environment.get('OPINUM_API_URL', DEFAULT_API_URL))
+        self.auth_url = f"{self.environment.get('DATAHUB_AUTH_URL', self.environment.get('OPINUM_AUTH_URL', DEFAULT_AUTH_URL))}/realms/opinum/protocol/openid-connect/token"
+        self.push_url = f"{self.environment.get('DATAHUB_PUSH_URL', self.environment.get('OPINUM_PUSH_URL', DEFAULT_PUSH_URL))}/api/data/"
+        self.scope = self.environment.get('DATAHUB_SCOPE', self.environment.get('OPINUM_SCOPE', DEFAULT_SCOPE))
+        self.username = self.environment.get('DATAHUB_USERNAME', self.environment.get('OPINUM_USERNAME'))
+        self.password = self.environment.get('DATAHUB_PASSWORD', self.environment.get('OPINUM_PASSWORD'))
+        self.client_id = self.environment.get('DATAHUB_CLIENT_ID', self.environment.get('OPINUM_CLIENT_ID'))
+        self.client_secret = self.environment.get('DATAHUB_CLIENT_SECRET', self.environment.get('OPINUM_SECRET'))
+        self.account_id = account_id
+        self.creation_time = None
+        self.token = None
+        self._set_token()
+        self.max_call_attempts = 1 + min(retries_when_connection_failure, self.MAX_RETRIES_WHEN_CONNECTION_FAILURE)
+        self.seconds_between_retries = seconds_between_retries
+
+    def _set_token(self):
+        oauth = OAuth2Session(client=LegacyApplicationClient(client_id=self.client_id))
+        args = {
+            'token_url': f"{self.auth_url}",
+            'scope': self.scope,
+            'username': self.username,
+            'password': self.password,
+            'client_id': self.client_id,
+            'client_secret': self.client_secret,
+            'auth': None
+        }
+        if self.account_id is not None:
+            args['account'] = self.account_id
+        self.token = oauth.fetch_token(**args)
+        self.creation_time = dt.datetime.now()
+
+    @property
+    def _headers(self):
+        if (dt.datetime.now() - self.creation_time).total_seconds() > self.time_limit:
+            self._set_token()
+        return {"Content-Type": "application/json",
+                "Authorization": f"Bearer {self.token['access_token']}"}
+
+    def _process_request(self, method, url, data, **kwargs):
+        attempts = 0
+        error = Exception('Unknown exception')
+        while attempts < self.max_call_attempts:
+            try:
+                if data is not None:
+                    data = json.dumps(data)
+                params = dict()
+                for k, v in kwargs.items():
+                    if isinstance(v, dt.datetime):
+                        v = v.strftime('%Y-%m-%dT%H:%M:%S')
+                    if k == 'date_from':
+                        k = 'from'
+                    params[k] = v
+
+                response = method(url, data=data, params=params, headers=self._headers)
+                response.raise_for_status()
+                return response
+            except (requests.exceptions.ConnectionError, AssertionError) as e:
+                error = e
+                attempts += 1
+                logging.warning(f"Failure {attempts}")
+                sleep(self.seconds_between_retries)
+        if attempts == self.max_call_attempts:
+            logging.error(error)
+            raise error
+
+    def get(self, endpoint, data=None, **kwargs):
+        """
+        Method for data query in the API
+
+        :param endpoint: the Data Hub API endpoint
+        :param data: body of the request. Should always be None for a get.
+        :param kwargs: dictionary of API call parameters
+        :return: the http request response
+        """
+
+        return self._process_request(requests.get,
+                                     f"{self.api_url}/{endpoint}",
+                                     data=data,
+                                     **kwargs)
+
+    def post(self, endpoint, data=None, **kwargs):
+        """
+        Method for data creation in the API
+
+        :param endpoint: the Data Hub API endpoint
+        :param data: body of the request
+        :param kwargs: dictionary of API call parameters
+        :return: the http request response
+        """
+        return self._process_request(requests.post,
+                                     f"{self.api_url}/{endpoint}",
+                                     data=data,
+                                     **kwargs)
+
+    def patch(self, endpoint, data=None, **kwargs):
+        """
+        Method for data patching in the API
+
+        :param endpoint: the Data Hub API endpoint
+        :param data: body of the request
+        :param kwargs: dictionary of API call parameters; see https://jsonpatch.com/
+        :return: the http request response
+        """
+        return self._process_request(requests.patch,
+                                     f"{self.api_url}/{endpoint}",
+                                     data=data,
+                                     **kwargs)
+
+    def put(self, endpoint, data=None, **kwargs):
+        """
+        Method for data update in the API
+
+        :param endpoint: the Data Hub API endpoint
+        :param data: body of the request
+        :param kwargs: dictionary of API call parameters
+        :return: the http request response
+        """
+        return self._process_request(requests.put,
+                                     f"{self.api_url}/{endpoint}",
+                                     data=data,
+                                     **kwargs)
+
+    def delete(self, endpoint, data=None, **kwargs):
+        """
+        Method for data deletion in the API
+
+        :param endpoint: the Data Hub API endpoint
+        :param data: body of the request
+        :param kwargs: dictionary of API call parameters
+        :return: the http request response
+        """
+        return self._process_request(requests.delete,
+                                     f"{self.api_url}/{endpoint}",
+                                     data=data,
+                                     **kwargs)
+
+    def push_data(self, body, operation_id: str=None):
+        """
+        Method for data push in the API
+
+        :param body: see https://docs.opinum.com/articles/push-formats/standard-format.html
+        :param operation_id: a string representing the operationId of the push; see https://docs.opinum.com/articles/push-formats/standard-format.html#ask-for-a-webhook-notification
+        :return: the http request response
+        """
+        return self._process_request(requests.post,
+                                     self.push_url+("?"+str(operation_id) if operation_id is not None else ""),
+                                     body)
+
+    def push_dataframe_data(self, df, **kwargs):
+        """
+        Method for data push in the API using a pandas DataFrame
+
+        :param df: a pandas dataframe with dates in ISO format in 'date' column and values in 'value' column
+        :param kwargs: dictionary of API call parameters, allowing to identify the target variable (see https://docs.opinum.com/articles/push-formats/standard-format.html)
+        :return: the http request response
+        """
+        kwargs['data'] = df.to_dict('records')
+        return self.push_data([kwargs])
+
+    def send_file_to_storage(self, filename, file_io, mime_type):
+        """
+        Method for sending a file to the storage
+
+        :param filename: The file name you want to give in the storage
+        :param file_io: a Bytes IO or a file opened in binary
+        :param mime_type: The file MIME Type
+        :return: the http request response
+        """
+        return requests.post(f"{self.api_url}/storage?filename={filename}",
+                             files={'data': (filename, file_io, mime_type)},
+                             headers={"Authorization": self._headers['Authorization']})
+
+
+def default_response_callback(response):
+    return response
+
+
+def multi_thread_request_on_path(method, endpoint,
+                                 split_parameter, max_parameter_entities, max_futures, workers=16,
+                                 response_callback=default_response_callback,
+                                 **kwargs):
+    """
+
+    :param method: The method to use. Most used is api_connector.get where api_connector is an instance of ApiConnector
+    :param endpoint: The endpoint
+    :param split_parameter: The parameter having a list as input that we will split in smaller calls
+    :param max_parameter_entities: The maximum number of parameters in each separate call. Mostly driven by the limit in length of the url on a http get
+    :param max_futures: Preparing at once all threads is not optimal. We better loop on several groups of calls
+    :param workers: The number of parallel threads. default: 16
+    :param response_callback: a method with a requests response as input returning what you expect. default: a method returning the response as is
+    :param kwargs: the list of http parameters
+    :return: a generator returning the results of your response_callback
+    """
+    futures = list()
+    entities = kwargs[split_parameter]
+    future_run_entities = max_parameter_entities * max_futures
+    with concurrent.futures.ProcessPoolExecutor(max_workers=workers) as executor:
+        for block in [entities[i: i + future_run_entities] for i in range(0, len(entities), future_run_entities)]:
+            for sub_block in [block[i: i + max_futures] for i in range(0, len(block), max_futures)]:
+                run_args = kwargs.copy()
+                run_args[split_parameter] = sub_block
+                futures.append(executor.submit(method, endpoint, **run_args))
+            while True:
+                all_finished = True
+                for i, future in enumerate(futures):
+                    if future.done():
+                        yield response_callback(future.result())
+                        futures.pop(i)
+                    else:
+                        all_finished = False
+                if all_finished:
+                    break
+

datahub_api_connector-1.0/src/datahub_api_connector/utils/site_content_duplicator.py

@@ -0,0 +1,260 @@
+import logging
+from copy import deepcopy
+
+from datahub_api_connector import ApiConnector
+
+logging.basicConfig(level=logging.INFO)
+
+
+def default_source_callback(source, target_site):
+    return source
+
+
+def default_get_allowed_sources(site, site_content_duplicator):
+    return None
+
+
+def default_get_excluded_sources(site, site_content_duplicator):
+    return None
+
+
+def default_get_allowed_variables(site, source, site_content_duplicator):
+    return None
+
+
+def default_get_excluded_variables(site, source, site_content_duplicator):
+    return None
+
+
+def do_not_process_source(source_key, allowed_sources, excluded_sources):
+    not_allowed = allowed_sources is not None and source_key not in allowed_sources
+    excluded = excluded_sources is not None and source_key in excluded_sources
+    return not_allowed or excluded
+
+
+def do_not_process_variable(mapping_config, allowed_variables, excluded_variables):
+    not_allowed = allowed_variables is not None and mapping_config not in allowed_variables
+    excluded = excluded_variables is not None and mapping_config in excluded_variables
+    return not_allowed or excluded
+
+
+class SiteContentDuplicator:
+    """
+    SiteContentDuplicator allows make a full copy of a site
+
+    This implies a copy of all sources in the site with their variables (raw and calculated)
+    """
+    def __init__(self, target_site_ids, template_site_id, source_key_field, account_id=None,
+                 source_callback=default_source_callback,
+                 get_allowed_sources=default_get_allowed_sources,
+                 get_excluded_sources=default_get_excluded_sources,
+                 get_allowed_variables=default_get_allowed_variables,
+                 get_excluded_variables=default_get_excluded_variables,
+                 ignore_missing_calculated_inputs=True,
+                 update_sources=False,
+                 update_variables=False):
+        """
+
+        :param target_site_ids: a list of site ids that will host the new sources
+        :param template_site_id: the site id that we copy
+        :param source_key_field: allows to identify the sources: 'name', 'serialNumber', 'meternumber' or 'eanNumber'
+        :param account_id: the tenant id
+        :param source_callback: a method that transforms the source received input. Manipulating the source_key_field is allowed.
+        :param get_allowed_sources: a method that returns the list of allowed source (via their key) for a given site. None means all sources are allowed.
+        :param get_excluded_sources: a method that returns the list of excluded source (via their key) for a given site. None means no exclusion.
+        :param get_allowed_variables: a method that returns the list of allowed variables (via their mapping config) for a given site/source. None means all variables are allowed.
+        :param get_excluded_variables: a method that returns the list of allowed variables (via their mapping config) for a given site/source. None means all variables are allowed.
+        :param ignore_missing_calculated_inputs: tell what to do with calculated variables inputs without associated source or variable in the target site
+        :param update: to force updates
+        """
+        logging.info('Start Site Duplicator')
+        self.target_site_ids = target_site_ids
+        self.template_site_id = template_site_id
+        self.source_key_field = source_key_field
+        self.source_callback = source_callback
+        self.get_allowed_sources = get_allowed_sources
+        self.get_excluded_sources = get_excluded_sources
+        self.get_allowed_variables = get_allowed_variables
+        self.get_excluded_variables = get_excluded_variables
+        self.ignore_missing_calculated_inputs = ignore_missing_calculated_inputs
+        self.update_sources = update_sources
+        self.update_variables = update_variables
+        self.api_connector = ApiConnector(account_id=account_id)
+        self.template_sources = dict()
+        self.template_vars = dict()
+        self.template_var_id_infos = dict()
+        for source in self.api_connector.get('sources',
+                                             siteId=self.template_site_id,
+                                             displayLevel='Site').json():
+            logging.info(f"Getting info for source {source[self.source_key_field]}")
+            self.template_sources[source['id']] = source
+            self.template_vars[source[self.source_key_field]] = dict()
+            for variable in self.api_connector.get('variables',
+                                                   sourceId=source['id'],
+                                                   displayLevel='Verbose').json():
+                if 'mappingConfig' not in variable:
+                    logging.warning(f"Variable named {variable['name']} has no mapping config. Ignored")
+                    continue
+                logging.info(f"\tGetting info for variable {variable['mappingConfig']}")
+                self.template_vars[source[self.source_key_field]][variable['mappingConfig']] = variable
+                self.template_var_id_infos[variable['id']] = {
+                    'source': source,
+                    'variable': variable
+                }
+
+    def process_site(self, site):
+        logging.info(f"Processing site {site['name']}")
+        existing_sources = {s[self.source_key_field]: s for s in self.api_connector.get('sources',
+                                                                                        siteId=site['id'],
+                                                                                        displayLevel='Site').json()}
+        source_key_mappings = dict()
+        allowed_sources = self.get_allowed_sources(site, self)
+        excluded_sources = self.get_excluded_sources(site, self)
+        for source in self.template_sources.values():
+            # key for new source can be different with template
+            new_source = self.source_callback(deepcopy(source), site)
+            source_key = new_source[self.source_key_field]
+            if do_not_process_source(source_key, allowed_sources, excluded_sources):
+                logging.info(f"Source {source_key} is excluded from processing")
+                continue
+            logging.info(f"Checking source {source_key}")
+            new_source.pop('siteName')
+            new_source['siteId'] = site['id']
+            if source_key not in existing_sources:
+                logging.info("\tCreating source")
+                new_source.pop('id')
+                existing_sources[source_key] = self.api_connector.post('sources', data=new_source).json()
+            elif self.update_sources:
+                logging.info("\tUpdating source")
+                new_source['id'] = existing_sources[source_key]['id']
+                self.api_connector.put(f"sources/{new_source['id']}", data=new_source)
+
+            source_key_mappings[source[self.source_key_field]] = source_key
+
+        # Second run now that all sources exist
+        calculated_variables_templates = dict()
+        existing_variables = dict()
+        for source_key, source in existing_sources.items():
+            if source[self.source_key_field] not in self.template_vars:
+                logging.warning(f"Source {source_key} is not existing in template. No variables to handle")
+                continue
+            if do_not_process_source(source_key, allowed_sources, excluded_sources):
+                logging.info(f"Source {source_key} is excluded from processing. No variables to handle")
+                continue
+            logging.info(f"Checking variables for source {source_key}")
+            allowed_variables = self.get_allowed_variables(site, source, self)
+            excluded_variables = self.get_excluded_variables(site, source, self)
+            existing_variables[source_key] = {v['mappingConfig']: v for v in self.api_connector.get('variables',
+                                                                                                    sourceId=source['id'],
+                                                                                                    displayLevel='Verbose').json()}
+            for mapping, variable in self.template_vars[source[self.source_key_field]].items():
+                if do_not_process_variable(mapping, allowed_variables, excluded_variables):
+                    logging.info(f"Variable {mapping} is excluded from processing")
+                    continue
+                new_var = deepcopy(variable)
+                new_var['sourceId'] = source['id']
+                if mapping in existing_variables[source_key]:
+                    if self.update_variables:
+                        new_var_id = existing_variables[source_key][mapping]['id']
+                        new_var['id'] = new_var_id
+                        if 'calculated' in new_var:
+                            calculated_variables_templates.setdefault(source_key, list()).append(variable)
+                            new_var['calculated']['calculatedVariableFormulas'] = [{}]
+                        logging.info(f"\tUpdating variable {mapping}")
+                        self.api_connector.put(f"sources/{source['id']}/variables/{new_var_id}", data=new_var)
+                else:
+                    if 'calculated' in new_var:
+                        # Keep another copy for later
+                        calculated_variables_templates.setdefault(source_key, list()).append(variable)
+                        new_var['calculated']['calculatedVariableFormulas'] = [{}]
+                    new_var.pop('id')
+                    logging.info(f"\tCreating variable {mapping}")
+                    existing_variables[source_key][mapping] = self.api_connector.post(f"variables/source/{source['id']}",
+                                                                                      data=new_var).json()
+
+        # Third run now that all variables exist (without calculations possibly)
+        for source_key, variables in calculated_variables_templates.items():
+            source = existing_sources[source_key]
+            logging.info(f"Finalising calculated variables for source {source_key}")
+            for new_var in variables:
+                mapping = new_var['mappingConfig']
+                logging.info(f"\tUpdating variable {mapping}")
+                new_var_id = existing_variables[source_key][mapping]['id']
+                new_var['id'] = new_var_id
+                new_var['sourceId'] = source['id']
+                for formula in new_var['calculated']['calculatedVariableFormulas']:
+                    new_variables_list = list()
+                    for sub_var in formula['variables']:
+                        if sub_var['siteId'] != self.template_site_id:
+                            new_variables_list.append(deepcopy(sub_var))
+                            continue
+                        if sub_var['variableId'] not in self.template_var_id_infos:
+                            message = (f"Source {sub_source_info['source'][self.source_key_field]} has an input variable"
+                                       f" without mapping config.")
+                            if self.ignore_missing_calculated_inputs:
+                                logging.info(f"{message} Dropped from the list of input variables")
+                            else:
+                                logging.info(f"{message} We keep the template variable")
+                                new_variables_list.append(deepcopy(sub_var))
+                            continue
+                        sub_source_info = self.template_var_id_infos[sub_var['variableId']]
+                        if sub_source_info['source'][self.source_key_field] not in source_key_mappings:
+                            message = (f"Source {sub_source_info['source'][self.source_key_field]} has an input variable"
+                                       f" but is not covered by this run.")
+                            if self.ignore_missing_calculated_inputs:
+                                logging.info(f"{message} Dropped from the list of input variables")
+                            else:
+                                logging.info(f"{message} We keep the template variable")
+                                new_variables_list.append(deepcopy(sub_var))
+                            continue
+                        sub_source_key = source_key_mappings[sub_source_info['source'][self.source_key_field]]
+                        sub_mapping = sub_source_info['variable']['mappingConfig']
+                        if sub_mapping not in existing_variables[sub_source_key]:
+                            message = (f"Variable {sub_mapping} from source {sub_source_key} is an input variable"
+                                       f" but is not covered by this run.")
+                            if self.ignore_missing_calculated_inputs:
+                                logging.info(f"{message} Dropped from the list of input variables")
+                            else:
+                                logging.info(f"{message} We keep the template variable")
+                                new_variables_list.append(deepcopy(sub_var))
+                            continue
+                        sub_var['siteId'] = site['id']
+                        sub_var['sourceId'] = existing_sources[sub_source_key]['id']
+                        sub_var['variableId'] = existing_variables[sub_source_key][sub_mapping]['id']
+                        new_variables_list.append(sub_var)
+                    formula['variables'] = new_variables_list
+                    new_entities_list = list()
+                    for form_value in formula['entities']:
+                        if form_value['siteId'] != self.template_site_id:
+                            new_entities_list.append(deepcopy(form_value))
+                            continue
+                        new_form_value = deepcopy(form_value)
+                        if new_form_value['entityType'] == 1: # Site form
+                            new_form_value['siteId'] = site['id']
+                            new_form_value['entityId'] = site['id']
+                            new_entities_list.append(new_form_value)
+                            continue
+                        sub_source = self.template_sources[form_value['sourceId']]
+                        if sub_source[self.source_key_field] not in source_key_mappings:
+                            message = (f"Source {sub_source_info['source'][self.source_key_field]} has an input form field"
+                                       f" but is not covered by this run.")
+                            if self.ignore_missing_calculated_inputs:
+                                logging.info(f"{message} Dropped from the list of input form fields")
+                            else:
+                                logging.info(f"{message} We keep the template form field")
+                                new_entities_list.append(new_form_value)
+                            continue
+                        sub_source_key = source_key_mappings[sub_source_info['source'][self.source_key_field]]
+                        new_form_value['siteId'] = site['id']
+                        new_form_value['sourceId'] = existing_sources[sub_source_key]['id']
+                        new_form_value['entityId'] = existing_sources[sub_source_key]['id']
+                        new_entities_list.append(new_form_value)
+                    formula['entities'] = new_entities_list
+                self.api_connector.put(f"sources/{source['id']}/variables/{new_var_id}", data=new_var)
+                existing_variables[mapping] = new_var
+
+    def run(self):
+        for site_id in self.target_site_ids:
+            site = self.api_connector.get('sites', siteIds=[site_id], displayLevel='VerboseSite').json()[0]
+            self.process_site(site)
+

datahub_api_connector-1.0/src/datahub_api_connector.egg-info/PKG-INFO

@@ -0,0 +1,78 @@
+Metadata-Version: 2.4
+Name: datahub-api-connector
+Version: 1.0
+Summary: Package to help connect to Verity's Data Hub API
+Author-email: Verity <support@opinum.com>
+Maintainer-email: Verity <support@opinum.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/opinum/datahub-api-connector
+Project-URL: Bug Tracker, https://github.com/opinum/datahub-api-connector/issues
+Keywords: datahub,Verity,Opinum,API,Keycloak
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Programming Language :: Python
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests
+Requires-Dist: requests_oauthlib
+Requires-Dist: oauthlib
+Dynamic: license-file
+
+This package simplifies the calls to Data Hub's API.
+
+No magic. You need to follow the [Data Hub Swagger Documentation](https://api.opinum.com) for correct formatting of your requests
+
+Be aware that this is a new package, certainly missing a lot of features. Feel free to contribute.
+
+You first need to create an instance of the ApiConnector class with following parameters:
+
+*ATTENTION:* this package can only be used from Data Hub version 7.0 (1 July 2025) onward, as it uses the authentication to the the new technology (Keycloak).
+For previous versions, the opinum-api-connector package can be used (https://github.com/opinum/opinum-api-connector) instead.
+
+> _environment_
+> > a dictionary of environment variables
+> >
+> > if `None`, ApiConnector uses your environment variables (_os.environ_)
+> >
+> > Mandatory environment variables are:
+> >
+> > * _DATAHUB_USERNAME_: the Datahub user. <br>
+> > TAKE CARE: if this user has access to multiple tenants and if you do not specify a tenant id,
+> > ApiConnector will use the last tenant used.
+> > * _DATAHUB_PASSWORD_: the password for the user
+> > * _DATAHUB_CLIENT_ID_: the client id for accessing the API
+> > * _DATAHUB_CLIENT_SECRET_ the corresponding secret
+> > 
+> > Optional environment variables are:
+> >
+> > * _DATAHUB_API_URL_: another API URL than the Europe SaaS one (https://api.opinum.com)
+> > * _DATAHUB_AUTH_URL_: another authentication URL than the Europe SaaS one (https://auth.opinum.com)
+> > * _DATAHUB_PUSH_URL_: another push URL than the Europe SaaS one (https://push.opinum.com)
+> > * _DATAHUB_SCOPE_: the scope of you session (default: "_datahub-api_")<br>
+> > if you want to push data, the scope should be "_datahub-api push-data_"
+
+> _account_id_
+> > one of the tenant ids available for the Datahub user (default: `None`)
+
+> _retries_when_connection_failure_
+> > number of extra attempts when no 200 or 204 return code (default: 0, maximum: 5)
+
+Once you have your ApiConnector instance, you may use the class methods
+
+* get
+* post
+* patch
+* put
+* delete
+* send_file_to_storage
+
+All keyword arguments will be converted to path parameters in the API call with one important exception,
+the _data_ keyword referring to the body of your call.
+
+There are two other class methods for data pushing because we have another API for this
+
+* push_data
+* push_dataframe_data
+
+There is a little bit of magic with the method multi_thread_request_on_path that splits a list of parameters
+Allowing to make parallel calls.

datahub_api_connector-1.0/src/datahub_api_connector.egg-info/SOURCES.txt

@@ -0,0 +1,12 @@
+LICENSE
+README.md
+pyproject.toml
+src/__init__.py
+src/datahub_api_connector/__init__.py
+src/datahub_api_connector.egg-info/PKG-INFO
+src/datahub_api_connector.egg-info/SOURCES.txt
+src/datahub_api_connector.egg-info/dependency_links.txt
+src/datahub_api_connector.egg-info/requires.txt
+src/datahub_api_connector.egg-info/top_level.txt
+src/datahub_api_connector/utils/__init__.py
+src/datahub_api_connector/utils/site_content_duplicator.py

datahub_api_connector-1.0/src/datahub_api_connector.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

datahub_api_connector-1.0/src/datahub_api_connector.egg-info/requires.txt

@@ -0,0 +1,3 @@
+requests
+requests_oauthlib
+oauthlib

datahub_api_connector-1.0/src/datahub_api_connector.egg-info/top_level.txt

@@ -0,0 +1,2 @@
+__init__
+datahub_api_connector