ixmp4 0.1.0__tar.gz

ixmp4-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 IIASA
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

ixmp4-0.1.0/PKG-INFO

@@ -0,0 +1,121 @@
+Metadata-Version: 2.1
+Name: ixmp4
+Version: 0.1.0
+Summary: a data warehouse for scenario analysis
+Home-page: https://software.ece.iiasa.ac.at
+License: MIT
+Author: Max Wolschlager
+Author-email: wolschlager@iiasa.ac.at
+Requires-Python: >=3.10,<4.0
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Requires-Dist: PyJWT (>=2.4.0,<3.0.0)
+Requires-Dist: SQLAlchemy-Utils (>=0.40.0,<0.41.0)
+Requires-Dist: SQLAlchemy[mypy] (>=2.0.5,<3.0.0)
+Requires-Dist: alembic (>=1.10.2,<2.0.0)
+Requires-Dist: dask (>=2023.4.0,<2024.0.0)
+Requires-Dist: fastapi (>=0.94.0,<0.95.0)
+Requires-Dist: httpx[http2] (>=0.23.3,<0.24.0)
+Requires-Dist: openpyxl (>=3.0.9,<4.0.0)
+Requires-Dist: oracledb (>=1.2.2,<2.0.0)
+Requires-Dist: pandas (>=2.0.0,<3.0.0)
+Requires-Dist: pandera (>=0.13.4,<0.14.0)
+Requires-Dist: psycopg2 (>=2.9.3,<3.0.0)
+Requires-Dist: pydantic (>=1.10.5,<2.0.0)
+Requires-Dist: python-dotenv (>=0.19.0,<0.20.0)
+Requires-Dist: requests (>=2.27.1,<3.0.0)
+Requires-Dist: rtoml (>=0.8.0,<0.9.0)
+Requires-Dist: typer (>=0.4.0,<0.5.0)
+Project-URL: Documentation, https://docs.ece.iiasa.ac.at/projects/ixmp4
+Project-URL: Repository, https://github.com/iiasa/ixmp4
+Description-Content-Type: text/markdown
+
+# The ixmp4 package for scenario data management
+
+Copyright (c) 2023 IIASA - Energy, Climate, and Environment Program (ECE)
+
+[![license: MIT](https://img.shields.io/badge/license-MIT-brightgreen.svg)](https://github.com/iiasa/ixmp4/blob/main/LICENSE)
+[![python](https://img.shields.io/badge/python-3.10-blue?logo=python&logoColor=white)](https://github.com/iiasa/ixmp4)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+
+## Overview
+
+The **ixmp4** package is a data warehouse for high-powered scenario analysis
+in the domain of integrated assessment of climate change and energy systems modeling.
+
+## License
+
+The **ixmp4** package is released under the [MIT license](https://github.com/iiasa/ixmp4/blob/main/LICENSE).
+
+## Requirements
+
+This project requires Python 3.10 and poetry (>= 1.2).
+
+## Setup
+
+```bash
+# Install Poetry, minimum version >=1.2 required
+curl -sSL https://install.python-poetry.org | python -
+
+# You may have to reinitialize your shell at this point.
+source ~/.bashrc
+
+# Activate in-project virtualenvs
+poetry config virtualenvs.in-project true
+
+# Install dependencies
+# (using "--with dev,docs,server" if dev and docs dependencies should be installed as well)
+poetry install --with dev,docs,server
+
+# Activate virtual environment
+poetry shell
+
+# Copy the template environment configuration
+cp template.env .env
+
+# Add a test platform
+ixmp4 platforms add test
+
+# Start the asgi server
+ixmp4 server start
+```
+
+## CLI
+
+```bash
+ixmp4 --help
+```
+
+## Docs
+
+Check [doc/README.md](doc/README.md) on how to build and serve dev documentation locally.
+
+## Docker Image
+
+Check [docker/README.md](docker/README.md) on how to build and publish docker images.
+
+## Developing
+
+See [DEVELOPING.md](DEVELOPING.md) for guidance. When contributing to this project via
+a Pull Request, add your name to the "authors" section in the `pyproject.toml` file.
+
+## Funding ackownledgement
+
+<img src="./doc/source/_static/ECEMF-logo.png" width="264" height="100"
+alt="ECEMF logo" />
+<img src="./doc/source/_static/openENTRANCE-logo.png" width="187" height="120"
+alt="openENTRANCE logo" />
+<img src="./doc/source/_static/ariadne-bmbf-logo.png" width="353" height="100"
+alt="Kopernikus project ARIADNE logo" />
+
+The development of the **ixmp4** package was funded from the EU Horizon 2020 projects
+[openENTRANCE](https://openentrance.eu) and [ECEMF](https://ecemf.eu)
+as well as the BMBF Kopernikus project [ARIADNE](https://ariadneprojekt.de)
+(FKZ 03SFK5A by the German Federal Ministry of Education and Research).
+
+<img src="./doc/source/_static/EU-logo-300x201.jpg" width="80" height="54" align="left"
+alt="EU logo" /> This project has received funding from the European Union’s Horizon
+2020 research and innovation programme under grant agreement No. 835896 and 101022622.
+

ixmp4-0.1.0/README.md

@@ -0,0 +1,86 @@
+# The ixmp4 package for scenario data management
+
+Copyright (c) 2023 IIASA - Energy, Climate, and Environment Program (ECE)
+
+[![license: MIT](https://img.shields.io/badge/license-MIT-brightgreen.svg)](https://github.com/iiasa/ixmp4/blob/main/LICENSE)
+[![python](https://img.shields.io/badge/python-3.10-blue?logo=python&logoColor=white)](https://github.com/iiasa/ixmp4)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+
+## Overview
+
+The **ixmp4** package is a data warehouse for high-powered scenario analysis
+in the domain of integrated assessment of climate change and energy systems modeling.
+
+## License
+
+The **ixmp4** package is released under the [MIT license](https://github.com/iiasa/ixmp4/blob/main/LICENSE).
+
+## Requirements
+
+This project requires Python 3.10 and poetry (>= 1.2).
+
+## Setup
+
+```bash
+# Install Poetry, minimum version >=1.2 required
+curl -sSL https://install.python-poetry.org | python -
+
+# You may have to reinitialize your shell at this point.
+source ~/.bashrc
+
+# Activate in-project virtualenvs
+poetry config virtualenvs.in-project true
+
+# Install dependencies
+# (using "--with dev,docs,server" if dev and docs dependencies should be installed as well)
+poetry install --with dev,docs,server
+
+# Activate virtual environment
+poetry shell
+
+# Copy the template environment configuration
+cp template.env .env
+
+# Add a test platform
+ixmp4 platforms add test
+
+# Start the asgi server
+ixmp4 server start
+```
+
+## CLI
+
+```bash
+ixmp4 --help
+```
+
+## Docs
+
+Check [doc/README.md](doc/README.md) on how to build and serve dev documentation locally.
+
+## Docker Image
+
+Check [docker/README.md](docker/README.md) on how to build and publish docker images.
+
+## Developing
+
+See [DEVELOPING.md](DEVELOPING.md) for guidance. When contributing to this project via
+a Pull Request, add your name to the "authors" section in the `pyproject.toml` file.
+
+## Funding ackownledgement
+
+<img src="./doc/source/_static/ECEMF-logo.png" width="264" height="100"
+alt="ECEMF logo" />
+<img src="./doc/source/_static/openENTRANCE-logo.png" width="187" height="120"
+alt="openENTRANCE logo" />
+<img src="./doc/source/_static/ariadne-bmbf-logo.png" width="353" height="100"
+alt="Kopernikus project ARIADNE logo" />
+
+The development of the **ixmp4** package was funded from the EU Horizon 2020 projects
+[openENTRANCE](https://openentrance.eu) and [ECEMF](https://ecemf.eu)
+as well as the BMBF Kopernikus project [ARIADNE](https://ariadneprojekt.de)
+(FKZ 03SFK5A by the German Federal Ministry of Education and Research).
+
+<img src="./doc/source/_static/EU-logo-300x201.jpg" width="80" height="54" align="left"
+alt="EU logo" /> This project has received funding from the European Union’s Horizon
+2020 research and innovation programme under grant agreement No. 835896 and 101022622.

ixmp4-0.1.0/ixmp4/__init__.py

@@ -0,0 +1,22 @@
+# flake8: noqa
+
+from ixmp4.core import (
+    Platform as Platform,
+    Region as Region,
+    Unit as Unit,
+    Run as Run,
+    Variable as Variable,
+    Model as Model,
+    Scenario as Scenario,
+)
+
+from ixmp4.core.exceptions import (
+    NotFound as NotFound,
+    NotUnique as NotUnique,
+    IxmpError as IxmpError,
+    InconsistentIamcType as InconsistentIamcType,
+)
+
+from ixmp4.data.abstract import DataPoint as DataPoint
+
+__version__ = "0.1.0"

ixmp4-0.1.0/ixmp4/__main__.py

@@ -0,0 +1,4 @@
+from ixmp4.cli import app
+
+if __name__ == "__main__":
+    app()

ixmp4-0.1.0/ixmp4/cli/__init__.py

@@ -0,0 +1,102 @@
+"""
+Check the CLI help command on how to use it:
+
+.. code:: bash
+
+    ixmp4 --help
+    ixmp4 platforms --help
+    ixmp4 test --help
+    ixmp4 server --help
+
+"""
+
+from typing import Optional
+import typer
+
+
+from ixmp4.core.exceptions import InvalidCredentials
+from ixmp4.conf.auth import ManagerAuth
+from ixmp4.conf import settings
+from ixmp4.cli import platforms
+from . import utils
+
+
+app = typer.Typer()
+app.add_typer(platforms.app, name="platforms")
+
+try:
+    from . import server
+
+    app.add_typer(server.app, name="server")
+except ImportError:
+    # No server installed
+    pass
+
+
+@app.command()
+def login(
+    username: str = typer.Argument(..., help="Your username."),
+    password: str = typer.Option(
+        ...,
+        help="Your password. Will be saved in plain-text.",
+        prompt=True,
+        hide_input=True,
+    ),
+):
+    try:
+        auth = ManagerAuth(username, password, settings.manager_url)
+        user = auth.get_user()
+        utils.good(f"Successfully authenticated as user '{user.username}'.")
+        if typer.confirm(
+            "Are you sure you want to save your credentials in plain-text for future use?"
+        ):
+            settings.credentials.set("default", username, password)
+
+    except InvalidCredentials:
+        raise typer.BadParameter(
+            "The credentials you provided are not valid. (Wrong username or password?)"
+        )
+
+
+try:
+    import pytest
+
+    @app.command(
+        context_settings={"allow_extra_args": True, "ignore_unknown_options": True}
+    )
+    def test(
+        ctx: typer.Context,
+        with_backend: Optional[bool] = False,
+        with_benchmarks: Optional[bool] = False,
+        dry: Optional[bool] = False,
+    ):
+        opts = [
+            "--cov-report",
+            "xml:.coverage.xml",
+            "--cov-report",
+            "term",
+            "--cov=ixmp4",
+            "-rsx",
+        ] + ctx.args
+
+        if not with_backend:
+            opts += ["--ignore=tests/data"]
+
+        if not with_benchmarks:
+            opts += ["--benchmark-skip"]
+        else:
+            opts += ["--benchmark-group-by=func", "--benchmark-columns=min"]
+
+        if dry:
+            utils.echo("pytest " + " ".join(opts))
+            raise typer.Exit(0)
+        exit_code = pytest.main(opts)
+        raise typer.Exit(
+            code=exit_code.value
+            if isinstance(exit_code, pytest.ExitCode)
+            else exit_code
+        )
+
+except ImportError:
+    # No pytest installed
+    pass

ixmp4-0.1.0/ixmp4/cli/platforms.py

@@ -0,0 +1,181 @@
+import re
+from typing import Optional
+from pathlib import Path
+
+import typer
+
+from ixmp4.conf import settings
+from ixmp4.conf.manager import ManagerPlatformInfo
+from ixmp4.conf.toml import TomlPlatformInfo
+from ixmp4.core.exceptions import PlatformNotFound
+from ixmp4.db.utils import alembic, sqlite
+
+from . import utils
+
+
+app = typer.Typer()
+
+
+def validate_name(name: str):
+    match = re.match(r"^[\w\-_]*$", name)
+    if match is None:
+        raise typer.BadParameter("Platform name must be slug-like.")
+    else:
+        return name
+
+
+def validate_dsn(dsn: str | None):
+    if dsn is None:
+        return None
+    match = re.match(r"^(sqlite|postgresql|oracle|https|http)(\:\/\/)", dsn)
+    if match is None:
+        raise typer.BadParameter(
+            "Platform dsn must be a valid URl or database connection string."
+        )
+    else:
+        return dsn
+
+
+def prompt_sqlite_dsn(name: str):
+    path = sqlite.get_database_path(name)
+    dsn = sqlite.get_dsn(path)
+    if path.exists():
+        if typer.confirm(
+            f"A file at the standard filesystem location for name '{name}' already exists. "
+            "Do you want to add the existing file to the platform registry?"
+        ):
+            return dsn
+        else:
+            raise typer.Exit()
+    else:
+        if typer.confirm(
+            f"No file at the standard filesystem location for name '{name}' exists. "
+            "Do you want to create a new database?"
+        ):
+            alembic.upgrade_database(dsn, "head")
+            return dsn
+        else:
+            raise typer.Exit()
+
+
+@app.command(help="Adds a new platform to ixmp4's toml registry.")
+def add(
+    name: str = typer.Argument(
+        ...,
+        help="The string identifier of the platform to add. Must be slug-like.",
+        callback=validate_name,
+    ),
+    dsn: Optional[str] = typer.Option(
+        None,
+        help="A data source name. Can be a http(s) URl or a database connection string.",
+        callback=validate_dsn,
+    ),
+):
+    try:
+        settings.toml.get_platform(name)
+        raise typer.BadParameter(
+            f"Platform with name '{name}' already exists. "
+            "Choose another name or remove the existing platform."
+        )
+    except PlatformNotFound:
+        pass
+
+    if dsn is None:
+        utils.echo(
+            "No DSN supplied, assuming you want to add a local sqlite database..."
+        )
+        dsn = prompt_sqlite_dsn(name)
+
+    settings.toml.add_platform(name, dsn)
+    utils.good("\nPlatform added successfully.")
+
+
+def prompt_sqlite_removal(dsn: str):
+    path = Path(dsn.replace("sqlite://", ""))
+    path_str = typer.style(path, fg=typer.colors.CYAN)
+    if typer.confirm(
+        "Do you want to remove the associated database file at "
+        f"{path_str} aswell?"  # type: ignore
+    ):
+        path.unlink()
+        utils.echo("\nDatabase file deleted.")
+    else:
+        utils.echo("\nDatabase file left intact.")
+
+
+@app.command(help="Removes a platform from ixmp4's toml registry.")
+def remove(
+    name: str = typer.Argument(
+        ..., help="The string identifier of the platform to remove."
+    )
+):
+    try:
+        platform = settings.toml.get_platform(name)
+    except PlatformNotFound:
+        raise typer.BadParameter(f"Platform '{name}' does not exist.")
+
+    if typer.confirm(
+        f"Are you sure you want to remove the platform '{platform.name}' with dsn '{platform.dsn}'?"
+    ):
+        if platform.dsn.startswith("sqlite://"):
+            prompt_sqlite_removal(platform.dsn)
+        settings.toml.remove_platform(name)
+
+
+def tabulate_platforms(
+    platforms: list[TomlPlatformInfo] | list[ManagerPlatformInfo],
+):
+    utils.echo("Platform".ljust(15) + "DSN\n".ljust(15))
+    total = 0
+    for p in platforms:
+        name = p.name
+        if len(name) > 12:
+            name = name[:9] + "..."
+        name = name.ljust(15)
+
+        utils.important(name, nl=False)
+        utils.echo(p.dsn.ljust(10))
+        total += 1
+
+    utils.info("\n" + str(total), nl=False)
+    utils.echo(" total \n")
+
+
+@app.command("list", help="Lists all registered platforms.")
+def list_():
+    toml_path_str = typer.style(settings.toml.path, fg=typer.colors.CYAN)
+    toml_platforms = settings.toml.list_platforms()
+    utils.echo(f"Platforms in '{toml_path_str}':")
+    tabulate_platforms(toml_platforms)
+
+    if settings.manager is not None:
+        manager_url_str = typer.style(settings.manager.url, fg=typer.colors.CYAN)
+        manager_platforms = settings.manager.list_platforms()
+        utils.echo(f"Platforms accessible via '{manager_url_str}':")
+        tabulate_platforms(manager_platforms)
+
+
+@app.command(
+    help="Migrates all database platforms from your local toml file to the newest revision."
+)
+def upgrade():
+    for c in settings.toml.list_platforms():
+        if c.dsn.startswith("http"):
+            utils.echo(f"Skipping '{c.name}' because it is a REST platform.")
+        else:
+            utils.echo(f"Upgrading platform '{c.name}' with dsn '{c.dsn}'...")
+            alembic.upgrade_database(c.dsn, "head")
+
+
+@app.command(
+    help="Stamps all database platforms from your local toml file with the given revision."
+)
+def stamp(revision: str) -> None:
+    for c in settings.toml.list_platforms():
+        if c.dsn.startswith("http"):
+            utils.echo(f"Skipping '{c.name}' because it is a REST platform.")
+        else:
+            utils.echo(
+                f"Stamping platform '{c.name}' with dsn '{c.dsn}' to '{revision}'..."
+            )
+            alembic.stamp_database(c.dsn, revision)

ixmp4-0.1.0/ixmp4/cli/server.py

@@ -0,0 +1,46 @@
+from typing import Optional
+import json
+
+import uvicorn  # type: ignore[import]
+import typer
+
+from fastapi.openapi.utils import get_openapi
+from ixmp4.server import v1
+
+from . import utils
+
+app = typer.Typer()
+
+
+@app.command()
+def start(
+    host: Optional[str] = typer.Option(
+        "127.0.0.1",
+        help="The hostname to bind to.",
+    ),
+    port: Optional[int] = typer.Option(
+        9000,
+        help="Requested server port.",
+    ),
+) -> None:
+    """Starts the ixmp4 web api."""
+
+    uvicorn.run("ixmp4.server:app", host=host, port=port, reload=True)
+
+
+@app.command()
+def dump_schema(output_file: Optional[typer.FileTextWrite] = typer.Option(None, "-o")):
+    schema = get_openapi(
+        title=v1.title,
+        version=v1.version,
+        openapi_version=v1.openapi_version,
+        description=v1.description,
+        routes=v1.routes,
+    )
+    if output_file is None:
+        utils.echo(json.dumps(schema))
+    else:
+        json.dump(
+            schema,
+            output_file,
+        )

ixmp4-0.1.0/ixmp4/cli/utils.py

@@ -0,0 +1,10 @@
+from functools import partial
+import typer
+
+echo = typer.echo
+good = partial(typer.secho, fg=typer.colors.GREEN)
+bad = partial(typer.secho, fg=typer.colors.RED)
+error = partial(typer.secho, fg=typer.colors.RED, err=True)
+info = partial(typer.secho, fg=typer.colors.CYAN)
+important = partial(typer.secho, fg=typer.colors.MAGENTA)
+secondary = partial(typer.secho, fg=typer.colors.BRIGHT_BLACK)

ixmp4-0.1.0/ixmp4/conf/__init__.py

@@ -0,0 +1,83 @@
+from dotenv import load_dotenv
+from pathlib import Path
+
+from pydantic import BaseSettings, Field, validator, HttpUrl, Extra
+
+from ixmp4.core.exceptions import InvalidCredentials
+from .credentials import Credentials
+from .toml import TomlConfig
+from .manager import ManagerConfig
+from .auth import ManagerAuth
+from .user import local_user
+from .base import PlatformInfo as PlatformInfo
+
+
+class Settings(BaseSettings):
+    mode: str = "production"
+    storage_directory: Path = Field("~/.local/share/ixmp4/", env="ixmp4_dir")
+    secret_hs256: str = "default_secret_hs256"
+    migration_db_uri: str = "sqlite:///./run/db.sqlite"
+    manager_url: HttpUrl = Field("https://api.manager.ece.iiasa.ac.at/v1")
+
+    class Config:
+        env_prefix = "ixmp4_"
+        extra = Extra.allow
+
+    def __init__(self, *args, **kwargs) -> None:
+        super().__init__(*args, **kwargs)
+
+        self.storage_directory.mkdir(parents=True, exist_ok=True)
+
+        database_dir = self.storage_directory / "databases"
+        database_dir.mkdir(exist_ok=True)
+        self.load_credentials()
+        self.load_manager_config()
+        self.load_toml_config()
+
+    def load_credentials(self):
+        credentials_config = self.storage_directory / "credentials.toml"
+        credentials_config.touch()
+        self.credentials = Credentials(credentials_config)
+
+        self.default_credentials = None
+        self.default_auth = None
+        try:
+            self.default_credentials = self.credentials.get("default")
+        except KeyError:
+            # TODO: WARNING: No default credentials provided.
+            pass
+
+        if self.default_credentials is not None:
+            try:
+                username, password = self.default_credentials
+                self.default_auth = ManagerAuth(username, password, self.manager_url)
+            except InvalidCredentials:
+                # TODO: WARNING: Default credentials invalid.
+                pass
+
+    def load_manager_config(self):
+        self.manager = None
+        if self.default_auth is not None:
+            self.manager = ManagerConfig(
+                self.manager_url, self.default_auth, remote=True
+            )
+
+    def load_toml_config(self):
+        if self.default_auth is not None:
+            toml_user = self.default_auth.get_user()
+        else:
+            toml_user = local_user
+        toml_config = self.storage_directory / "platforms.toml"
+        toml_config.touch()
+        self.toml = TomlConfig(toml_config, toml_user)
+
+    @validator("storage_directory")
+    def expand_user(cls, v):
+        # translate ~/asdf into /home/user/asdf
+        return Path.expanduser(v)
+
+
+load_dotenv()
+# strict typechecking fails due to a bug
+# https://docs.pydantic.dev/visual_studio_code/#adding-a-default-with-field
+settings = Settings()  # type: ignore