vantage6-algorithm-store 4.3.0__tar.gz

vantage6-algorithm-store-4.3.0/PKG-INFO

@@ -0,0 +1,168 @@
+Metadata-Version: 2.1
+Name: vantage6-algorithm-store
+Version: 4.3.0
+Summary: Vantage6 algorithm store
+Home-page: https://github.com/vantage6/vantage6
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Provides-Extra: dev
+
+<h1 align="center">
+  <br>
+  <a href="https://vantage6.ai"><img src="https://github.com/IKNL/guidelines/blob/master/resources/logos/vantage6.png?raw=true" alt="vantage6" width="350"></a>
+</h1>
+
+<h3 align=center> A Privacy Enhancing Technology (PET) Operations platform</h3>
+<h3 align="center">
+
+<!-- Badges go here-->
+[![Release](https://github.com/vantage6/vantage6/actions/workflows/release.yml/badge.svg)](https://github.com/vantage6/vantage6/actions/workflows/release.yml)
+[![PyPI vantage6](https://badge.fury.io/py/vantage6.svg)](https://badge.fury.io/py/vantage6)
+[![Unittests](https://github.com/vantage6/vantage6/actions/workflows/unit_tests.yml/badge.svg)](https://github.com/vantage6/vantage6/actions/workflows/unit_tests.yml)
+[![Coverage Status](https://coveralls.io/repos/github/vantage6/vantage6/badge.svg?branch=main)](https://coveralls.io/github/vantage6/vantage6?branch=main)
+[![Codacy Badge](https://app.codacy.com/project/badge/Grade/2e60ac3b3f284620805f7399cba317be)](https://www.codacy.com/gh/vantage6/vantage6/dashboard?utm_source=github.com&amp;utm_medium=referral&amp;utm_content=vantage6/vantage6&amp;utm_campaign=Badge_Grade)
+[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.7382602.svg)](https://doi.org/10.5281/zenodo.7382602)
+</h3>
+
+<p align="center">
+  <a href="#books-quickstart">Quickstart</a> •
+  <a href="#project-structure">Project structure</a> •
+  <a href="#gift_heart-join-the-community">Join the community</a> •
+  <a href="#black_nib-references">References</a>
+</p>
+
+
+-----------------------------------------------------------------------------------------------------
+This repository is contains all the **vantage6** infrastructure source code. The **vantage6** technology enables to manage and deploy privacy enhancing technologies like Federated Learning (FL) and Multi-Party Computation (MPC). Please visit our [website (vantage6.ai)](https://vantage6.ai) to learn more!
+
+You can find more (user) documentation at [readthedocs (docs.vantage6.ai)](https://docs.vantage6.ai). If you have any questions, suggestions or just want to chat about federated learning: join our [Discord (https://discord.gg/yAyFf6Y)](https://discord.gg/yAyFf6Y) channel.
+
+## Infrastructure overview
+
+![Vantage6 architecture overview](docs/images/overview-infrastructure.png)
+
+*A High level overview of the vantage6 infrastructure. Vantage6 has both a client-server and peer-to-peer architecture. The client is used by the researcher to create (PET) computation requests. It is also used to manage users, organizations and collaborations. The server contains users, organizations, collaborations, tasks and their results. It provides a central access point for both the clients and nodes. The nodes have access to privacy sensitive data and handle computation requests retrieved from the server. Computation request are executed as separate containers on the node. These containers are connected to containers at other nodes by a VPN network.*
+
+## :books: Quickstart
+
+### Requirements
+The **vantage6** infrastructure is delivered in Docker images. To run these images, you need to have [Docker](https://docs.docker.com/get-docker/) installed. To install the latest version of the vantage6 CLI, you need to have [Python](https://www.python.org/downloads/), we recommend using an environment manager like [mini-conda](https://docs.conda.io/en/latest/miniconda.html).
+
+Install the latest version of the vantage6 CLI by using:
+```bash
+pip install vantage6
+```
+
+This install the `v6` commands, which allows you to manage your nodes and servers. To view all available options, run:
+
+```bash
+v6 --help
+```
+
+For example you can create a local test setup by using:
+
+```bash
+v6 dev create-demo-network
+```
+
+This creates a local network with a server and two nodes. You can start the network by running:
+
+```bash
+v6 dev start-demo-network
+```
+
+This will start the server and nodes in the background. You can view the logs by running:
+
+```bash
+# View node logs
+v6 node attach
+
+# View server logs
+v6 server attach
+```
+
+From here you can use the [vantage6-client](https://pypi.org/project/vantage6-client) to interact with the server. The demo network has a pre-configured organization with the following credentials:
+
+* Username: `org_1-admin`
+* Password: `password`
+
+For example, you can create a new organization by running:
+
+```python
+from vantage6.client import Client
+
+client = Client('http://127.0.0.1', 5000, '/api', log_level='debug')
+client.authenticate('org_1-admin', 'password')
+client.setup_encryption(None)
+
+client.organization.create(
+    name='My organization',
+    address1='My address',
+    address2='My address',
+    zipcode='1234AB',
+    country='The Netherlands',
+    domain='my-organization.com'
+)
+```
+
+You can find more (user) documentation at [readthedocs (docs.vantage6.ai)](https://docs.vantage6.ai)
+
+## Project structure
+
+### PYPI packages
+This repository is home to 6 PyPi packages:
+
+* [vantage6](https://pypi.org/project/vantage6) -> _CLI for managing node and server instances_
+* [vantage6-client](https://pypi.org/project/vantage6-client) -> _Python client for interacting with the vantage6-server_
+* [vantage6-algorithm-tools](https://pypi.org/project/vantage6-algorithm-tools) -> _Python tools to facilitate algorithm development_
+* [vantage6-node](https://pypi.org/project/vantage6-node) -> _Node application package_
+* [vantage6-server](https://pypi.org/project/vantage6-server) -> _Server application package_
+* [vantage6-algorithm-store](https://pypi.org/project/vantage6-algorithm-store) -> _Algorithm store application package_
+* [vantage6-common](https://pypi.org/project/vantage6-common) -> _Package with common vantage6 functions_
+* [vantage6-backend-common](https://pypi.org/project/vantage6-backend-common) -> _Package with functions common to central server and algorithm store_
+
+**Note that when using vantage6 you do not install the _server_ and _node_ packages. These are delivered to you in Docker images.**
+
+### Docker images
+The vantage6 infrastructure is delivered in Docker images. All Docker images are stored in our private [Harbor](https://goharbor.io/) registry. The most important images contain the server and node:
+
+* `harbor2.vantage6.ai/infrastructure/node:VERSION` -> _Node application Docker image_
+* `harbor2.vantage6.ai/infrastructure/server:VERSION` -> _Server application Docker image_
+
+with `VERSION` being the full semantic version of the vantage6 infrastructure, e.g. `4.0.0` or `4.1.0rc0`.
+
+Several other images are used to support the infrastructure:
+
+* `harbor2.vantage6.ai/infrastructure/infrastructure-base:VERSION` -> _Base image for the infrastructure_
+* `harbor2.vantage6.ai/infrastructure/squid:VERSION` -> _Squid proxy image used for the whitelisting service_
+* `harbor2.vantage6.ai/infrastructure/alpine` -> _Alpine image used for vpn traffic forwarding_
+* `harbor2.vantage6.ai/infrastructure/vpn-client` -> _VPN image used to connect to the VPN_
+* `harbor2.vantage6.ai/infrastructure/vpn-configurator` -> _VPN image used for initialization_
+* `harbor2.vantage6.ai/infrastructure/ssh-tunnel` -> _SSH tunnel image used for connecting algorithms to external services_
+
+And finally there are some images released for algorithm development:
+
+* `harbor2.vantage6.ai/infrastructure/algorithm-base:MAJOR.MINOR` -> _Base image for algorithm development_
+* `harbor2.vantage6.ai/infrastructure/algorithm-ohdsi-base:MAJOR.MINOR` -> _Extended algorithm base image for OHDSI algorithm development_
+
+## :gift_heart: Join the community!
+We hope to continue developing, improving, and supporting **vantage6** with the help of the federated learning community. If you are interested in contributing, first of all, thank you! Second, please take a look at our [contributing guidelines](https://docs.vantage6.ai/en/main/devops/contribute.html)
+
+<a href="https://github.com/vantage6/vantage6/graphs/contributors">
+  <img src="https://contrib.rocks/image?repo=vantage6/vantage6" />
+</a>
+
+## :black_nib: References
+If you are using **vantage6**, please cite this repository as well as the accompanying papers as follows:
+
+> * F. Martin, M. Sieswerda, H. Alradhi, et al. vantage6. Available at https://doi.org/10.5281/zenodo.7221216. Accessed on MONTH, 20XX.
+> * A. Moncada-Torres, F. Martin, M. Sieswerda, J. van Soest, G. Gelijnse. VANTAGE6: an open source priVAcy preserviNg federaTed leArninG infrastructurE for Secure Insight eXchange. AMIA Annual Symposium Proceedings, 2020, p. 870-877. [[BibTeX](https://arturomoncadatorres.com/bibtex/moncada-torres2020vantage6.txt), [PDF](https://vantage6.ai/vantage6/)]
+> * D. Smits\*, B. van Beusekom\*, F. Martin, L. Veen, G. Geleijnse, A. Moncada-Torres, An Improved Infrastructure for Privacy-Preserving Analysis of Patient Data, Proceedings of the International Conference of Informatics, Management, and Technology in Healthcare (ICIMTH), vol. 25, 2022, p. 144-147. [[BibTeX](https://arturomoncadatorres.com/bibtex/smits2022improved.txt), [PDF](https://ebooks.iospress.nl/volumearticle/60190)]
+
+-----------------------------------------------------------------------------------------------------
+<p align="center">
+  <a href="https://vantage6.ai">vantage6.ai</a> •
+  <a href="https://discord.gg/yAyFf6Y">Discord</a> •
+  <a href="https://vantage6.discourse.group/">Discourse</a> •
+  <a href="https://docs.vantage6.ai">User documentation</a> •
+</p>

vantage6-algorithm-store-4.3.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

vantage6-algorithm-store-4.3.0/setup.py

@@ -0,0 +1,55 @@
+import codecs
+import os
+
+from os import path
+from setuptools import setup, find_namespace_packages
+from pathlib import Path
+
+# get current directory
+here = Path(path.abspath(path.dirname(__file__)))
+parent_dir = here.parent.absolute()
+
+# get the long description from the README file
+with codecs.open(path.join(parent_dir, "README.md"), encoding="utf-8") as f:
+    long_description = f.read()
+
+# Read the API version from disk. This file should be located in the package
+# folder, since it's also used to set the pkg.__version__ variable.
+version_path = os.path.join(here, "vantage6", "algorithm", "store", "_version.py")
+version_ns = {"__file__": version_path}
+with codecs.open(version_path) as f:
+    exec(f.read(), {}, version_ns)
+
+# setup the package
+setup(
+    name="vantage6-algorithm-store",
+    version=version_ns["__version__"],
+    description="Vantage6 algorithm store",
+    long_description=long_description,
+    long_description_content_type="text/markdown",
+    url="https://github.com/vantage6/vantage6",
+    packages=find_namespace_packages(),
+    python_requires=">=3.10",
+    install_requires=[
+        "flasgger==0.9.5",
+        "flask==2.2.5",
+        "Flask-Cors==3.0.10",
+        "Flask-Principal==0.4.0",
+        "flask-marshmallow==0.15.0",
+        "Flask-RESTful==0.3.10",
+        "gevent==23.9.1",
+        "marshmallow==3.19.0",
+        "requests==2.31.0",
+        "schema==0.7.5",
+        "SQLAlchemy==1.4.46",
+        "werkzeug==3.0.1",
+        f'vantage6 == {version_ns["__version__"]}',
+        f'vantage6-common == {version_ns["__version__"]}',
+    ],
+    extras_require={"dev": ["coverage==6.4.4"]},
+    package_data={
+        "vantage6.algorithm.store": [
+            "__build__",
+        ],
+    },
+)

vantage6-algorithm-store-4.3.0/vantage6/algorithm/store/__build__

@@ -0,0 +1 @@
+0

vantage6-algorithm-store-4.3.0/vantage6/algorithm/store/__init__.py

@@ -0,0 +1,358 @@
+"""
+The algorithm store holds the algorithms that are available to the vantage6
+nodes. It is a repository of algorithms that can be coupled to a vantage6
+server. The algorithms are stored in a database and can be managed through
+the API. Note that is possible to couple multiple algorithm stores to a
+vantage6 server. This allows both coupling a community store and a private
+store to a vantage6 server.
+"""
+
+import os
+from gevent import monkey
+
+from vantage6.algorithm.store.default_roles import get_default_roles
+
+# This is a workaround for readthedocs
+if not os.environ.get("READTHEDOCS"):
+    # flake8: noqa: E402 (ignore import error)
+    monkey.patch_all()
+
+# pylint: disable=C0413, C0411
+import importlib
+import logging
+import json
+import traceback
+
+from http import HTTPStatus
+from werkzeug.exceptions import HTTPException
+from flask import Flask, make_response, request, send_from_directory, Request, Response
+from flask_cors import CORS
+from flask_marshmallow import Marshmallow
+from flask_restful import Api
+from flask_principal import Principal
+from flasgger import Swagger
+from pathlib import Path
+
+from vantage6.common import logger_name
+from vantage6.common.globals import APPNAME
+from vantage6.backend.common.resource.output_schema import BaseHATEOASModelSchema
+
+# TODO move this to common, then remove dependency on CLI in algorithm store
+from vantage6.cli.context.algorithm_store import AlgorithmStoreContext
+from vantage6.algorithm.store._version import __version__
+from vantage6.algorithm.store.globals import API_PATH, HOST_URI_ENV
+from vantage6.algorithm.store.globals import RESOURCES, SERVER_MODULE_NAME
+
+# TODO the following are simply copies of the same files in the server - refactor
+from vantage6.algorithm.store.model.base import Base, DatabaseSessionManager, Database
+from vantage6.algorithm.store import db
+
+# TODO move server imports to common / refactor
+from vantage6.algorithm.store.permission import PermissionManager
+
+module_name = logger_name(__name__)
+log = logging.getLogger(module_name)
+
+
+class AlgorithmStoreApp:
+    """
+    Vantage6 server instance.
+
+    Attributes
+    ----------
+    ctx : AlgorithmStoreContext
+        Context object that contains the configuration of the algorithm store.
+    """
+
+    def __init__(self, ctx: AlgorithmStoreContext) -> None:
+        """Create a vantage6-server application."""
+
+        self.ctx = ctx
+
+        # initialize, configure Flask
+        self.app = Flask(
+            SERVER_MODULE_NAME,
+            root_path=Path(__file__),
+            template_folder=Path(__file__).parent / "templates",
+            static_folder=Path(__file__).parent / "static",
+        )
+        self.debug: dict = self.ctx.config.get("debug", {})
+        self.configure_flask()
+
+        # Setup SQLAlchemy and Marshmallow for marshalling/serializing
+        self.ma = Marshmallow(self.app)
+
+        # Setup Principal, granular API access manegement
+        self.principal = Principal(self.app, use_sessions=False)
+
+        # Enable cross-origin resource sharing
+        self.cors = CORS(self.app)
+
+        # SWAGGER documentation
+        self.swagger = Swagger(self.app, template={})
+
+        # setup the permission manager for the API endpoints
+        self.permissions = PermissionManager()
+
+        # Api - REST JSON-rpc
+        self.api = Api(self.app)
+        self.configure_api()
+        self.load_resources()
+
+        # set the server version
+        self.__version__ = __version__
+
+        # set environment variable for dev environment
+        host_uri = self.ctx.config.get("dev", {}).get("host_uri")
+        if host_uri:
+            os.environ[HOST_URI_ENV] = host_uri
+
+        log.info("Initialization done")
+
+    def configure_flask(self) -> None:
+        """Configure the Flask settings of the vantage6 server."""
+
+        # let us handle exceptions
+        self.app.config["PROPAGATE_EXCEPTIONS"] = True
+
+        # Open Api Specification (f.k.a. swagger)
+        self.app.config["SWAGGER"] = {
+            "title": f"{APPNAME} algorithm store",
+            "uiversion": "3",
+            "openapi": "3.0.0",
+            "version": __version__,
+        }
+
+        debug_mode = self.debug.get("flask", False)
+        if debug_mode:
+            log.debug("Flask debug mode enabled")
+        self.app.debug = debug_mode
+
+        def _get_request_path(request: Request) -> str:
+            """
+            Return request extension of request URL, e.g.
+            http://localhost:5000/api/task/1 -> api/task/1
+
+            Parameters
+            ----------
+            request: Request
+                Flask request object
+
+            Returns
+            -------
+            string:
+                The endpoint path of the request
+            """
+            return request.url.replace(request.url_root, "")
+
+        # before request
+        @self.app.before_request
+        def do_before_request():
+            """Before every flask request method."""
+            # Add log message before each request
+            log.debug(
+                "Received request: %s %s", request.method, _get_request_path(request)
+            )
+
+            # This will obtain a (scoped) db session from the session factory
+            # that is linked to the flask request global `g`. In every endpoint
+            # we then can access the database by using this session. We ensure
+            # that the session is removed (and uncommited changes are rolled
+            # back) at the end of every request.
+            DatabaseSessionManager.new_session()
+
+        @self.app.after_request
+        def remove_db_session(response):
+            """After every flask request.
+
+            This will close the database session created by the
+            `before_request`.
+            """
+            DatabaseSessionManager.clear_session()
+            return response
+
+        @self.app.errorhandler(HTTPException)
+        def error_remove_db_session(error: HTTPException):
+            """In case an HTTP-exception occurs during the request.
+
+            It is important to close the db session to avoid having dangling
+            sessions.
+            """
+            if error.code == 404:
+                log.debug("404 error for route '%s'", _get_request_path(request))
+            else:
+                log.warning("HTTP Exception occured during request")
+                log.debug("%s", traceback.format_exc())
+            DatabaseSessionManager.clear_session()
+            return error.get_response()
+
+        @self.app.errorhandler(Exception)
+        def error2_remove_db_session(error):
+            """In case an exception occurs during the request.
+
+            It is important to close the db session to avoid having dangling
+            sessions.
+            """
+            log.exception("Exception occured during request")
+            DatabaseSessionManager.clear_session()
+            return {
+                "msg": "An unexpected error occurred on the server!"
+            }, HTTPStatus.INTERNAL_SERVER_ERROR
+
+        @self.app.route("/robots.txt")
+        def static_from_root():
+            return send_from_directory(self.app.static_folder, request.path[1:])
+
+    def configure_api(self) -> None:
+        """Define global API output and its structure."""
+
+        # helper to create HATEOAS schemas
+        BaseHATEOASModelSchema.api = self.api
+
+        # whatever you get try to json it
+        @self.api.representation("application/json")
+        # pylint: disable=unused-argument
+        def output_json(
+            data: Base | list[Base], code: HTTPStatus, headers: dict = None
+        ) -> Response:
+            """
+            Return jsonified data for request responses.
+
+            Parameters
+            ----------
+            data: Base | list[Base]
+                The data to be jsonified
+            code: HTTPStatus
+                The HTTP status code of the response
+            headers: dict
+                Additional headers to be added to the response
+            """
+
+            if isinstance(data, Base):
+                data = db.jsonable(data)
+            elif isinstance(data, list) and len(data) and isinstance(data[0], Base):
+                data = db.jsonable(data)
+
+            resp = make_response(json.dumps(data), code)
+            resp.headers.extend(headers or {})
+            return resp
+
+    def load_resources(self) -> None:
+        """Import the modules containing API resources."""
+
+        # make services available to the endpoints, this way each endpoint can
+        # make use of 'em.
+        services = {
+            "api": self.api,
+            "config": self.ctx.config,
+            "permissions": self.permissions,
+        }
+
+        for res in RESOURCES:
+            module = importlib.import_module("vantage6.algorithm.store.resource." + res)
+            module.setup(self.api, API_PATH, services)
+
+    @staticmethod
+    def _add_default_roles() -> None:
+        for role in get_default_roles():
+            if not db.Role.get_by_name(role["name"]):
+                log.warn("Creating new default role %s", role["name"])
+                new_role = db.Role(
+                    name=role["name"],
+                    description=role["description"],
+                    rules=role["rules"],
+                )
+                new_role.save()
+
+    def start(self) -> None:
+        """
+        Start the server.
+
+        Before server is really started, some database settings are checked and
+        (re)set where appropriate.
+        """
+        self._add_default_roles()
+
+        # add whitelisted server and root user from config file if they do not exist
+        if root_user := self.ctx.config.get("root_user", {}):
+            whitelisted_uri = root_user.get("v6_server_uri")
+            root_username = root_user.get("username")
+            if whitelisted_uri and root_username:
+                if not (v6_server := db.Vantage6Server.get_by_url(whitelisted_uri)):
+                    log.debug("This server will be whitelisted: %s", whitelisted_uri)
+                    v6_server = db.Vantage6Server(url=whitelisted_uri)
+                    v6_server.save()
+
+                # if the user does not exist already, add it
+                if not db.User.get_by_server(
+                    username=root_username, v6_server_id=v6_server.id
+                ):
+                    log.warning("Creating root user")
+
+                    root = db.Role.get_by_name("Root")
+
+                    user = db.User(
+                        v6_server_id=v6_server.id,
+                        username=root_username,
+                        roles=[root],
+                    )
+                    user.save()
+                else:
+                    log.info(
+                        "The root user given in the configuration already exists -"
+                        " no action taken."
+                    )
+
+            else:
+                log.warning(
+                    "No v6_server_uri and/or username found in the configuration file "
+                    "in the root_user section. This means no-one can alter resources on"
+                    " this server, unless one or more users were already authorized to "
+                    "make changes to the algorithm store previously."
+                )
+        else:
+            log.warning(
+                "No root user found in the configuration file. This means "
+                "no-one can alter resources on this server, unless one or "
+                "more users were already authorized to make changes to the "
+                "algorithm store prevoiusly."
+            )
+        return self
+
+
+def run_server(config: str, system_folders: bool = True) -> AlgorithmStoreApp:
+    """
+    Run a vantage6 server.
+
+    Parameters
+    ----------
+    config: str
+        Configuration file path
+    system_folders: bool
+        Whether to use system or user folders. Default is True.
+
+    Returns
+    -------
+    AlgorithmStoreApp
+        A running instance of the vantage6 server
+    """
+    ctx = AlgorithmStoreContext.from_external_config_file(config, system_folders)
+    allow_drop_all = ctx.config["allow_drop_all"]
+    Database().connect(uri=ctx.get_database_uri(), allow_drop_all=allow_drop_all)
+    return AlgorithmStoreApp(ctx).start()
+
+
+def run_dev_server(server_app: AlgorithmStoreApp, *args, **kwargs) -> None:
+    """
+    Run a vantage6 development server (outside of a Docker container).
+
+    Parameters
+    ----------
+    server_app: AlgorithmStoreApp
+        Instance of a vantage6 server
+    """
+    log.warning("*" * 80)
+    log.warning(" DEVELOPMENT SERVER ".center(80, "*"))
+    log.warning("*" * 80)
+    kwargs.setdefault("log_output", False)
+    server_app.socketio.run(server_app.app, *args, **kwargs)

vantage6-algorithm-store-4.3.0/vantage6/algorithm/store/_version.py

@@ -0,0 +1,23 @@
+import os
+import json
+
+here = os.path.abspath(os.path.dirname(__file__))
+
+with open(os.path.join(here, "__build__")) as fp:
+    __build__ = json.load(fp)
+
+# Module version
+version_info = (4, 3, 0, "final", __build__, 0)
+
+# Module version stage suffix map
+_specifier_ = {"alpha": "a", "beta": "b", "candidate": "rc", "final": ""}
+version = f"{version_info[0]}.{version_info[1]}.{version_info[2]}"
+pre_release = (
+    ""
+    if version_info[3] == "final"
+    else _specifier_[version_info[3]] + str(version_info[4])
+)
+post_release = "" if not version_info[5] else f".post{version_info[5]}"
+
+# Module version accessible using thomas.__version__
+__version__ = f"{version}{pre_release}{post_release}"