aiondemand 0.1.0b0__tar.gz

aiondemand-0.1.0b0/LICENSE

@@ -0,0 +1,22 @@
+MIT License
+
+Copyright (c) 2023, Jean Matias
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+

aiondemand-0.1.0b0/PKG-INFO

@@ -0,0 +1,82 @@
+Metadata-Version: 2.1
+Name: aiondemand
+Version: 0.1.0b0
+Summary: Python SDK for the AIoD metadata catalogue
+Author-email: Jean Matias <smatias.jean@gmail.com>, Pieter Gijsbers <p.gijsbers@tue.nl>
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: aiohttp
+Requires-Dist: pandas
+Requires-Dist: python-keycloak
+Requires-Dist: requests
+Provides-Extra: dev
+Requires-Dist: pre-commit; extra == "dev"
+Requires-Dist: responses; extra == "dev"
+Requires-Dist: aioresponses; extra == "dev"
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: pytest-asyncio; extra == "dev"
+Provides-Extra: doc
+Requires-Dist: mkdocs-material; extra == "doc"
+Requires-Dist: mkdocstrings-python; extra == "doc"
+Requires-Dist: mkdocs-jupyter; extra == "doc"
+Requires-Dist: griffe<1.0; extra == "doc"
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![python](https://img.shields.io/badge/Python-3.11-3776AB.svg?style=flat&logo=python&logoColor=white)](https://www.python.org)
+
+# AI-on-Demand 
+
+The [AI-on-Demand](https://aiod.eu) (AIOD) platform empowers AI research and innovation for industry and academia. 
+At its core if the [metadata catalogue](https://api.aiod.eu) which indexes countless AI resources, such as datasets, papers, and educational material, 
+from many different platforms such as [Zenodo](https://www.zenodo.org), [OpenML](https://www.openml.org), and [AIDA](https://https://www.i-aida.org/ai-educational-resources/).
+This package allows you to explore all resources in the metadata catalogue through Python.
+You can also browse the contents of the AI-on-Demand metadata catalogue through the [MyLibrary](https://mylibrary.aiod.eu) service.
+
+## Installation
+The `aiondemand` package is on [PyPI](https://pypi.org/project/aiondemand/):
+
+```bash
+$ pip install aiondemand
+```
+
+Tip: install your dependencies in a [virtual environment](https://realpython.com/python-virtual-environments-a-primer/).
+
+## Usage
+You can directly access endpoints through the Python API, for example to browse datasets:
+```python
+import aiod
+
+aiod.datasets.get_list()
+```
+And results will be returned as a [Pandas](https://pandas.pydata.org/docs/getting_started/overview.html) dataframe (though the `data_format` may be used to get JSON instead):
+```bash
+      platform platform_resource_identifier                    name       date_published                                            same_as  is_accessible_for_free  ...  relevant_link  relevant_resource relevant_to research_area scientific_domain identifier
+0  huggingface       acronym_identification  acronym_identification  2022-03-02T23:29:22  https://huggingface.co/datasets/acronym_identi...                    True  ...             []                 []          []            []                []          1
+...
+9  huggingface              allegro_reviews         allegro_reviews  2022-03-02T23:29:22    https://huggingface.co/datasets/allegro_reviews                    True  ...             []                 []          []            []                []         10
+
+[10 rows x 30 columns]
+```
+
+You can even query the elastic search endpoints:
+```python
+aiod.publications.search(search_query="Robotics")
+```
+```bash
+      platform platform_resource_identifier                                               name date_published                                            same_as is_accessible_for_free  ... relevant_resource relevant_to      research_area  scientific_domain  type  identifier
+0  robotics4eu                         1803  Responsible Robotics &amp; non-tech barriers t...           None  https://www.robotics4eu.eu/publications/respon...                   None  ...                []          []  [other materials]  [other materials]  None           4
+
+[1 rows x 36 columns]
+```
+## Contributing
+
+Interested in contributing? Check out the [contributing guidelines](contributing.md).
+By contributing to this project, you agree to abide by our [Code of Conduct](conduct.md).
+
+## Credits
+
+The `aiondemand` package is being developed with funding from EU’s Horizon Europe research and innovation program under grant agreement [No. 101070000 (AI4EUROPE)](https://cordis.europa.eu/project/id/101070000).
+Not all contributors need be affiliated with this funding.
+
+[`cookiecutter`](https://cookiecutter.readthedocs.io/en/latest/) and the `py-pkgs-cookiecutter` [template](https://github.com/py-pkgs/py-pkgs-cookiecutter) were used to create the repository structure.

aiondemand-0.1.0b0/docs/README.md

@@ -0,0 +1,58 @@
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![python](https://img.shields.io/badge/Python-3.11-3776AB.svg?style=flat&logo=python&logoColor=white)](https://www.python.org)
+
+# AI-on-Demand 
+
+The [AI-on-Demand](https://aiod.eu) (AIOD) platform empowers AI research and innovation for industry and academia. 
+At its core if the [metadata catalogue](https://api.aiod.eu) which indexes countless AI resources, such as datasets, papers, and educational material, 
+from many different platforms such as [Zenodo](https://www.zenodo.org), [OpenML](https://www.openml.org), and [AIDA](https://https://www.i-aida.org/ai-educational-resources/).
+This package allows you to explore all resources in the metadata catalogue through Python.
+You can also browse the contents of the AI-on-Demand metadata catalogue through the [MyLibrary](https://mylibrary.aiod.eu) service.
+
+## Installation
+The `aiondemand` package is on [PyPI](https://pypi.org/project/aiondemand/):
+
+```bash
+$ pip install aiondemand
+```
+
+Tip: install your dependencies in a [virtual environment](https://realpython.com/python-virtual-environments-a-primer/).
+
+## Usage
+You can directly access endpoints through the Python API, for example to browse datasets:
+```python
+import aiod
+
+aiod.datasets.get_list()
+```
+And results will be returned as a [Pandas](https://pandas.pydata.org/docs/getting_started/overview.html) dataframe (though the `data_format` may be used to get JSON instead):
+```bash
+      platform platform_resource_identifier                    name       date_published                                            same_as  is_accessible_for_free  ...  relevant_link  relevant_resource relevant_to research_area scientific_domain identifier
+0  huggingface       acronym_identification  acronym_identification  2022-03-02T23:29:22  https://huggingface.co/datasets/acronym_identi...                    True  ...             []                 []          []            []                []          1
+...
+9  huggingface              allegro_reviews         allegro_reviews  2022-03-02T23:29:22    https://huggingface.co/datasets/allegro_reviews                    True  ...             []                 []          []            []                []         10
+
+[10 rows x 30 columns]
+```
+
+You can even query the elastic search endpoints:
+```python
+aiod.publications.search(search_query="Robotics")
+```
+```bash
+      platform platform_resource_identifier                                               name date_published                                            same_as is_accessible_for_free  ... relevant_resource relevant_to      research_area  scientific_domain  type  identifier
+0  robotics4eu                         1803  Responsible Robotics & non-tech barriers t...           None  https://www.robotics4eu.eu/publications/respon...                   None  ...                []          []  [other materials]  [other materials]  None           4
+
+[1 rows x 36 columns]
+```
+## Contributing
+
+Interested in contributing? Check out the [contributing guidelines](contributing.md).
+By contributing to this project, you agree to abide by our [Code of Conduct](conduct.md).
+
+## Credits
+
+The `aiondemand` package is being developed with funding from EU’s Horizon Europe research and innovation program under grant agreement [No. 101070000 (AI4EUROPE)](https://cordis.europa.eu/project/id/101070000).
+Not all contributors need be affiliated with this funding.
+
+[`cookiecutter`](https://cookiecutter.readthedocs.io/en/latest/) and the `py-pkgs-cookiecutter` [template](https://github.com/py-pkgs/py-pkgs-cookiecutter) were used to create the repository structure.

aiondemand-0.1.0b0/pyproject.toml

@@ -0,0 +1,36 @@
+[project]
+name = "aiondemand"
+version = "0.1.0b0"
+description = "Python SDK for the AIoD metadata catalogue"
+authors = [
+  { name = "Jean Matias", email = "smatias.jean@gmail.com" },
+  { name = "Pieter Gijsbers", email = "p.gijsbers@tue.nl" },
+]
+readme = "docs/README.md"
+
+requires-python = ">=3.11"
+
+dependencies = ["aiohttp", "pandas", "python-keycloak", "requests"]
+
+[project.optional-dependencies]
+dev = [
+    "pre-commit",
+    "responses",
+    "aioresponses",
+    "pytest",
+    "pytest-asyncio",
+]
+doc = [
+    "mkdocs-material",
+    "mkdocstrings-python",
+    "mkdocs-jupyter",
+    "griffe<1.0",
+]
+
+[tool.commitizen]
+name = "cz_conventional_commits"
+version = "0.1.0b0"
+version_files = ["pyproject.toml:version", "src/aiod/__version__.py"]
+
+[project.entry-points."mkdocs.plugins"]
+asset-type-replacer = "docs.tools.asset_type_replacer:AssetTypeReplacer"

aiondemand-0.1.0b0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

aiondemand-0.1.0b0/src/aiod/__init__.py

@@ -0,0 +1,19 @@
+from aiod.authentication import login, logout, get_current_user
+from aiod.configuration import config
+from aiod.resources import case_studies
+from aiod.resources import computational_assets
+from aiod.resources import contacts
+from aiod.default.counts import asset_counts as counts
+from aiod.resources import datasets
+from aiod.resources import educational_resources
+from aiod.resources import events
+from aiod.resources import experiments
+from aiod.resources import ml_models
+from aiod.resources import news
+from aiod.resources import organisations
+from aiod.resources import persons
+from aiod.resources import platforms
+from aiod.resources import projects
+from aiod.resources import publications
+from aiod.resources import services
+from aiod.resources import teams

aiondemand-0.1.0b0/src/aiod/__version__.py

@@ -0,0 +1 @@
+version = '0.1.0b0'

aiondemand-0.1.0b0/src/aiod/authentication/__init__.py

@@ -0,0 +1 @@
+from .authentication import login, logout, get_current_user

aiondemand-0.1.0b0/src/aiod/authentication/authentication.py

@@ -0,0 +1,118 @@
+import http.client
+import requests
+from keycloak import KeycloakOpenID, KeycloakAuthenticationError, KeycloakPostError
+from typing import Sequence, NamedTuple
+
+from aiod.configuration import config
+
+
+def _connect_keycloak() -> KeycloakOpenID:
+    return KeycloakOpenID(
+        server_url=config.auth_server_url,
+        client_id=config.client_id,
+        realm_name=config.realm,
+    )
+
+
+def keycloak_openid() -> KeycloakOpenID:
+    global _keycloak_openid
+    if _keycloak_openid is None:
+        _keycloak_openid = _connect_keycloak()
+    return _keycloak_openid
+
+
+def on_keycloak_config_changed(_: str, __: str, ___: str) -> None:
+    global _keycloak_openid
+    logout(ignore_post_error=True)
+    _keycloak_openid = None
+
+
+config.subscribe("auth_server_url", on_change=on_keycloak_config_changed)
+config.subscribe("realm", on_change=on_keycloak_config_changed)
+config.subscribe("client_id", on_change=on_keycloak_config_changed)
+
+_keycloak_openid: KeycloakOpenID | None = None
+
+
+class User(NamedTuple):
+    name: str
+    roles: Sequence[str]
+
+
+def login(username: str, password: str) -> None:
+    """Logs in the user with the provided username and password.
+
+    Args:
+        username (str): The username of the user.
+        password (str): The password of the user.
+
+    Raises:
+        ValueError: When username or password is empty.
+        FailedAuthenticationError: When the username or password is wrong.
+    """
+    if not username:
+        raise ValueError(
+            f"{username!r} is not a valid username, must be a non-empty string."
+        )
+    if not password:
+        raise ValueError(
+            f"{password!r} is not a valid password, must be a non-empty string."
+        )
+
+    try:
+        token = keycloak_openid().token(username, password)
+    except KeycloakAuthenticationError as e:
+        raise FailedAuthenticationError(
+            "Authentication failed! Please verify your credentials."
+        ) from e
+    config.access_token = token["access_token"]
+    config.refresh_token = token["refresh_token"]
+
+
+def logout(ignore_post_error: bool = False) -> None:
+    """Logs out the current user.
+
+    Args:
+        ignore_post_error:
+            If true, do not raise an error if the logout attempt failed.
+
+    """
+    try:
+        keycloak_openid().logout(config.refresh_token)
+    except KeycloakPostError as e:
+        if not ignore_post_error:
+            raise e
+
+    config.access_token = None
+    config.refresh_token = None
+
+
+def get_current_user() -> User:
+    """Return name and roles of the user that is currently authenticated.
+
+    Returns:
+        User: The user information for the currently authenticated user.
+
+    Raises:
+        NotAuthenticatedError: When the user is not authenticated.
+    """
+    response = requests.get(
+        f"{config.api_base_url}authorization_test",
+        headers={"Authorization": f"Bearer {config.access_token}"},
+    )
+
+    content = response.json()
+    if response.status_code == http.client.UNAUTHORIZED:
+        raise NotAuthenticatedError(content)
+    return User(
+        name=content["name"],
+        roles=tuple(content["roles"]),
+    )
+
+
+class FailedAuthenticationError(Exception):
+    """Raised when an authentication error occurred."""
+
+
+class NotAuthenticatedError(Exception):
+    """Raised when an endpoint that requires authentication is called without authentication."""

aiondemand-0.1.0b0/src/aiod/calls/calls.py

@@ -0,0 +1,299 @@
+import aiohttp
+import asyncio
+import requests
+
+import pandas as pd
+from typing import Literal
+from functools import partial
+
+from aiod.calls.urls import (
+    url_to_get_asset,
+    url_to_get_content,
+    url_to_get_list,
+    url_to_get_list_from_platform,
+    url_to_get_asset_from_platform,
+    url_to_resource_counts,
+    url_to_search,
+)
+
+from aiod.calls.utils import format_response, wrap_calls
+
+
+def get_list(
+    *,
+    asset_type: str,
+    platform: str | None = None,
+    offset: int = 0,
+    limit: int = 10,
+    version: str | None = None,
+    data_format: Literal["pandas", "json"] = "pandas",
+) -> pd.DataFrame | list[dict]:
+    """
+    Retrieve a list of ASSET_TYPE from the catalogue.
+
+    Parameters:
+        platform: Return metadata of ASSET_TYPE assets of this platform (default is None).
+        offset: The offset for pagination (default is 0).
+        limit: The maximum number of items to retrieve (default is 10).
+        version: The version of the endpoint (default is None).
+        data_format: The desired format for the response (default is "pandas").
+            For "json" formats, the returned type is a json decoded type, i.e. in this case a list of dicts.
+
+    Returns:
+        The retrieved metadata in the specified format.
+    """
+    url = (
+        url_to_get_list_from_platform(asset_type, platform, offset, limit, version)
+        if platform is not None
+        else url_to_get_list(asset_type, offset, limit, version)
+    )
+    res = requests.get(url)
+    resources = format_response(res.json(), data_format)
+    return resources
+
+
+def counts(
+    *, asset_type: str, version: str | None = None, per_platform: bool = False
+) -> int | dict[str, int]:
+    """
+    Retrieve the number of ASSET_TYPE assets in the metadata catalogue.
+
+    Parameters:
+        version: The version of the endpoint (default is None).
+        per_platform: Whether to list counts per platform (default is False).
+
+    Returns:
+        The number ASSET_TYPE assets in the metadata catalogue. If the parameter per_platform is True, it returns a dictionary with platform names as keys and the number of ASSET_TYPE assets from that platform as values.
+    """
+
+    url = url_to_resource_counts(version, per_platform, asset_type)
+    res = requests.get(url)
+    return res.json()
+
+
+def get_asset(
+    *,
+    asset_type: str,
+    identifier: int,
+    version: str | None = None,
+    data_format: Literal["pandas", "json"] = "pandas",
+) -> pd.Series | dict:
+    """
+    Retrieve metadata for a specific ASSET_TYPE.
+
+    Parameters:
+        identifier: The identifier of the ASSET_TYPE to retrieve.
+        version: The version of the endpoint (default is None).
+        data_format: The desired format for the response (default is "pandas").
+            For "json" formats, the returned type is a json decoded type, in this case a dict.
+
+    Returns:
+        The retrieved metadata for the specified ASSET_TYPE.
+    """
+    url = url_to_get_asset(asset_type, identifier, version)
+    res = requests.get(url)
+    resources = format_response(res.json(), data_format)
+    return resources
+
+
+def get_asset_from_platform(
+    *,
+    asset_type: str,
+    platform: str,
+    platform_identifier: str,
+    version: str | None = None,
+    data_format: Literal["pandas", "json"] = "pandas",
+) -> pd.Series | dict:
+    """
+    Retrieve metadata for a specific ASSET_TYPE identified by the external platform identifier.
+
+    Parameters:
+        platform: The platform where the ASSET_TYPE asset is retrieved from.
+        platform_identifier: The identifier under which the ASSET_TYPE is known by the platform.
+        version: The version of the endpoint (default is None).
+        data_format: The desired format for the response (default is "pandas").
+            For "json" formats, the returned type is a json decoded type, in this case a dict.
+
+    Returns:
+        The retrieved metadata for the specified ASSET_TYPE.
+    """
+    url = url_to_get_asset_from_platform(
+        asset_type, platform, platform_identifier, version
+    )
+    res = requests.get(url)
+    resources = format_response(res.json(), data_format)
+    return resources
+
+
+def get_content(
+    *,
+    asset_type: str,
+    identifier: int,
+    distribution_idx: int = 0,
+    version: str | None = None,
+) -> bytes:
+    """
+    Retrieve the data content of a specific ASSET_TYPE.
+
+    Parameters:
+        identifier: The identifier of the ASSET_TYPE asset.
+        distribution_idx: The index of a specific distribution from the distribution list (default is 0).
+        version: The version of the endpoint (default is None).
+
+    Returns:
+        The data content for the specified ASSET_TYPE.
+    """
+    url = url_to_get_content(asset_type, identifier, distribution_idx, version)
+    res = requests.get(url)
+    distribution = res.content
+    return distribution
+
+
+def search(
+    *,
+    asset_type: str,
+    search_query: str,
+    platforms: list[str] | None = None,
+    offset: int = 0,
+    limit: int = 10,
+    search_field: (
+        None | Literal["name", "issn", "description_html", "description_plain"]
+    ) = None,
+    get_all: bool = True,
+    version: str | None = None,
+    data_format: Literal["pandas", "json"] = "pandas",
+) -> pd.DataFrame | list[dict]:
+    """
+    Search metadata for ASSET_TYPE type using the Elasticsearch endpoint of the AIoD metadata catalogue.
+
+    Parameters:
+        search_query: The string to be matched against the search fields.
+        platforms: The platforms to filter the search results.
+            If None, results from all platforms will be returned (default is None).
+        offset: The offset for pagination (default is 0).
+        limit: The maximum number of results to retrieve (default is 10).
+        search_field:
+            The specific fields to search within. If None, the query will be matched against all fields (default is None).
+        get_all: If true, a request to the database is made to retrieve all data.
+            If false, only the indexed information is returned. (default is True).
+        version: The version of the endpoint to use (default is None).
+        data_format: The desired format for the response (default is "pandas").
+            For "json" formats, the returned type is a json decoded type, in this case a list of dict's.
+
+    Returns:
+        The retrieved metadata in the specified format.
+    """
+    url = url_to_search(
+        asset_type,
+        search_query,
+        platforms,
+        offset,
+        limit,
+        search_field,
+        get_all,
+        version,
+    )
+    res = requests.get(url)
+    resources = format_response(res.json()["resources"], data_format)
+    return resources
+
+
+async def get_assets_async(
+    *,
+    asset_type: str,
+    identifiers: list[int],
+    version: str | None = None,
+    data_format: Literal["pandas", "json"] = "pandas",
+) -> pd.DataFrame | list[dict]:
+    """
+    Asynchronously retrieve metadata for a list of ASSET_TYPE identifiers.
+
+    Parameters:
+        identifiers: The list of identifiers of the ASSET_TYPE to retrieve.
+        version: The version of the endpoint (default is None).
+        data_format: The desired format for the response (default is "pandas").
+            For "json" formats, the returned type is a json decoded type, in this case a list of dicts.
+
+    Returns:
+        The retrieved metadata for the specified ASSET_TYPE.
+    """
+    urls = [
+        url_to_get_asset(asset_type, identifier, version) for identifier in identifiers
+    ]
+    response_data = await _fetch_resources(urls)
+    resources = format_response(response_data, data_format)
+    return resources
+
+
+async def get_list_async(
+    *,
+    asset_type: str,
+    offset: int = 0,
+    limit: int = 100,
+    batch_size: int = 10,
+    version: str | None = None,
+    data_format: Literal["pandas", "json"] = "pandas",
+) -> pd.DataFrame | list[dict]:
+    """
+    Asynchronously retrieve a list of ASSET_TYPE from the catalogue in batches.
+
+    Parameters:
+        offset: The offset for pagination (default is 0).
+        limit: The maximum number of items to retrieve (default is 10).
+        batch_size: The number of items in a a batch.
+        version: The version of the endpoint (default is None).
+        data_format: The desired format for the response (default is "pandas").
+            For "json" formats, the returned type is a json decoded type, in this case a list of dicts.
+
+    Returns:
+        The retrieved metadata in the specified format.
+
+    Raises:
+        ValueError: Batch size must be larger than 0.
+    """
+    if batch_size <= 0:
+        raise ValueError(
+            "batch_size must be larger than 0, otherwise you can use the synchronous get_list function!"
+        )
+
+    offsets = range(offset, offset + limit, batch_size)
+    last_batch_size = (limit % batch_size) or batch_size
+    batch_sizes = [batch_size] * (len(offsets) - 1) + [last_batch_size]
+
+    urls = [
+        url_to_get_list(asset_type, offset, limit, version)
+        for offset, limit in zip(offsets, batch_sizes)
+    ]
+
+    response_data = await _fetch_resources(urls)
+    flattened_response_data = [
+        response for batch in response_data for response in batch
+    ]
+    resources = format_response(flattened_response_data, data_format)
+    return resources
+
+
+async def _fetch_resources(urls) -> dict:
+    async def _fetch_data(session, url) -> dict:
+        async with session.get(url) as response:
+            return await response.json()
+
+    async with aiohttp.ClientSession() as session:
+        tasks = [_fetch_data(session, url) for url in urls]
+        response_data = await asyncio.gather(*tasks)
+    return response_data
+
+
+wrap_common_calls = partial(
+    wrap_calls,
+    calls=[
+        get_list,
+        counts,
+        get_asset,
+        get_asset_from_platform,
+        get_content,
+        get_assets_async,
+        get_list_async,
+    ],
+)
+wrap_search_call = partial(wrap_calls, calls=[search])