cofy-api 0.1.0__tar.gz

cofy_api-0.1.0/PKG-INFO

@@ -0,0 +1,163 @@
+Metadata-Version: 2.4
+Name: cofy-api
+Version: 0.1.0
+Summary: An open-source modular framework for ingesting, standardising, storing, and computing energy-related data.
+Author: Jorg
+Author-email: EnergyID <info@energieid.be>
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+Requires-Dist: alembic>=1.18.4
+Requires-Dist: entsoe-py>=0.7.10
+Requires-Dist: fastapi[standard]>=0.128.0
+Requires-Dist: narwhals>=2.15.0
+Requires-Dist: pandas>=3.0.0
+Requires-Dist: polars>=1.37.1
+Requires-Dist: saq[redis,web]>=0.26.1
+
+**Cofy Cloud** is an open-source modular framework for ingesting, standardising, storing, and computing energy-related data, designed to run anywhere from local deployments to cloud environments.
+
+Right now this is very much a work in progress.
+With the development of a first proof of concept. This is not ready for production use and the api is likely to change significantly in the near future.
+
+## Setup
+### Install
+*TODO*
+### Configure
+*TODO*
+### Development
+We use [astral](https://docs.astral.sh/) python tooling for our development environment.
+We use [poethepoet](https://poethepoet.natn.io) to define some essential tasks.
+The demo run task is also available as vscode execution task, making it easy to run and debug the demo application from within vscode.
+
+#### Install/update dependencies:
+First install [uv](https://docs.astral.sh/uv/) if you don't have it yet.
+
+Then install/update dependencies:
+```sh
+uv sync
+```
+
+Install [poethepoet](https://poethepoet.natn.io) and [pre-commit](https://pre-commit.com/)
+```sh
+uv tool install poethepoet
+uv tool install pre-commit
+```
+
+Activate [pre-commit](https://pre-commit.com/) hooks that enforce code style on every commit:
+```sh
+pre-commit install
+```
+
+#### Configure environment variables:
+Our demo application uses some API keys for external services. You can provide these `.env.local` file in the root of the repository, following the structure of `.env.example`.
+
+#### Set up the database:
+The demo application uses a SQLite database. A different database can be configured via the environment variables.
+To create the database and seed it with example data:
+
+```sh
+poe db-seed
+```
+
+This will run all pending migrations and load the example CSV data into the database.
+
+#### Run development demo application:
+
+```sh
+poe demo
+```
+
+#### Run background worker:
+We use Cofy Worker to run background jobs that perform data ingestion and processing tasks asynchronously.
+
+The background worker requires a Redis instance to run. You can configure the Redis URL via the environment variables.
+The simplest way to run Redis locally is via Docker:
+
+```sh
+docker run -p 6379:6379 redis
+```
+
+Then start the worker:
+
+```sh
+poe worker
+```
+
+#### Code style checks:
+```sh
+poe lint    # Check code style
+poe format  # Format code
+poe check   # Run type checks
+```
+
+#### Run tests:
+```sh
+poe test
+```
+#### Build & publish
+TODO
+
+## Database & Migrations
+
+Cofy uses [Alembic](https://alembic.sqlalchemy.org/) for database migrations. Each module that needs database storage owns its own migration branch, keeping schemas independent and composable.
+
+### Commands
+
+| Command | Description |
+|---|---|
+| `poe db-seed` | Run all migrations and seed the database with example data |
+| `poe db-migrate` | Run all pending migrations |
+| `poe db-reset` | Drop all tables and re-run migrations (⚠️ destroys all data) |
+| `poe db-generate` | Generate a new migration file for a specific module branch |
+
+### Generating a migration
+
+When you change a module's SQLAlchemy model, generate a migration for that module's branch:
+
+```sh
+poe db-generate "add phone number to member" --head members_core@head --rev-id members_core_0002
+```
+
+| Argument | Required | Description |
+|---|---|---|
+| `message` (positional) | ✅ | Short description of the change |
+| `--head` | ✅ | Branch to extend, e.g. `members_core@head` |
+| `--rev-id` | ❌ | Custom revision ID (Alembic generates one if omitted) |
+| `--no-autogenerate` | ❌ | Create an empty migration instead of diffing model changes |
+
+The generated file will be placed in the module's own `migrations/versions/` directory automatically.
+
+### How branches work
+
+Each module declares its own Alembic branch label in its initial migration. This allows multiple modules to coexist in the same database without interfering with each other.
+
+For example, the members module uses branch `members_core`:
+
+```
+src/modules/members/migrations/versions/
+├── members_core_0001_members_core_initial.py   # branch_labels = ("members_core",)
+└── members_core_0002_add_phone_number.py       # extends members_core@head
+```
+
+A separate module `foo` would have its own branch `foo_core` with revisions in its own directory:
+
+```
+src/modules/foo/migrations/versions/
+├── foo_core_0001_initial.py                    # branch_labels = ("foo_core",)
+└── foo_core_0002_add_index.py                  # extends foo_core@head
+```
+
+When `poe db-migrate` runs, Alembic upgrades all branches to their latest head — both `members_core` and `foo_core` are applied independently.
+
+### Extending a module's schema
+
+If you build a custom implementation that extends an existing module's schema, you can create a new branch that depends on a specific revision:
+
+```python
+revision = "members_custom_0001"
+down_revision = None
+branch_labels = ("members_custom",)
+depends_on = "members_core_0001"  # ensures the base schema exists first
+```
+
+This guarantees the base tables are created before your extension runs, while keeping both migration chains independent.

cofy_api-0.1.0/README.md

@@ -0,0 +1,147 @@
+**Cofy Cloud** is an open-source modular framework for ingesting, standardising, storing, and computing energy-related data, designed to run anywhere from local deployments to cloud environments.
+
+Right now this is very much a work in progress.
+With the development of a first proof of concept. This is not ready for production use and the api is likely to change significantly in the near future.
+
+## Setup
+### Install
+*TODO*
+### Configure
+*TODO*
+### Development
+We use [astral](https://docs.astral.sh/) python tooling for our development environment.
+We use [poethepoet](https://poethepoet.natn.io) to define some essential tasks.
+The demo run task is also available as vscode execution task, making it easy to run and debug the demo application from within vscode.
+
+#### Install/update dependencies:
+First install [uv](https://docs.astral.sh/uv/) if you don't have it yet.
+
+Then install/update dependencies:
+```sh
+uv sync
+```
+
+Install [poethepoet](https://poethepoet.natn.io) and [pre-commit](https://pre-commit.com/)
+```sh
+uv tool install poethepoet
+uv tool install pre-commit
+```
+
+Activate [pre-commit](https://pre-commit.com/) hooks that enforce code style on every commit:
+```sh
+pre-commit install
+```
+
+#### Configure environment variables:
+Our demo application uses some API keys for external services. You can provide these `.env.local` file in the root of the repository, following the structure of `.env.example`.
+
+#### Set up the database:
+The demo application uses a SQLite database. A different database can be configured via the environment variables.
+To create the database and seed it with example data:
+
+```sh
+poe db-seed
+```
+
+This will run all pending migrations and load the example CSV data into the database.
+
+#### Run development demo application:
+
+```sh
+poe demo
+```
+
+#### Run background worker:
+We use Cofy Worker to run background jobs that perform data ingestion and processing tasks asynchronously.
+
+The background worker requires a Redis instance to run. You can configure the Redis URL via the environment variables.
+The simplest way to run Redis locally is via Docker:
+
+```sh
+docker run -p 6379:6379 redis
+```
+
+Then start the worker:
+
+```sh
+poe worker
+```
+
+#### Code style checks:
+```sh
+poe lint    # Check code style
+poe format  # Format code
+poe check   # Run type checks
+```
+
+#### Run tests:
+```sh
+poe test
+```
+#### Build & publish
+TODO
+
+## Database & Migrations
+
+Cofy uses [Alembic](https://alembic.sqlalchemy.org/) for database migrations. Each module that needs database storage owns its own migration branch, keeping schemas independent and composable.
+
+### Commands
+
+| Command | Description |
+|---|---|
+| `poe db-seed` | Run all migrations and seed the database with example data |
+| `poe db-migrate` | Run all pending migrations |
+| `poe db-reset` | Drop all tables and re-run migrations (⚠️ destroys all data) |
+| `poe db-generate` | Generate a new migration file for a specific module branch |
+
+### Generating a migration
+
+When you change a module's SQLAlchemy model, generate a migration for that module's branch:
+
+```sh
+poe db-generate "add phone number to member" --head members_core@head --rev-id members_core_0002
+```
+
+| Argument | Required | Description |
+|---|---|---|
+| `message` (positional) | ✅ | Short description of the change |
+| `--head` | ✅ | Branch to extend, e.g. `members_core@head` |
+| `--rev-id` | ❌ | Custom revision ID (Alembic generates one if omitted) |
+| `--no-autogenerate` | ❌ | Create an empty migration instead of diffing model changes |
+
+The generated file will be placed in the module's own `migrations/versions/` directory automatically.
+
+### How branches work
+
+Each module declares its own Alembic branch label in its initial migration. This allows multiple modules to coexist in the same database without interfering with each other.
+
+For example, the members module uses branch `members_core`:
+
+```
+src/modules/members/migrations/versions/
+├── members_core_0001_members_core_initial.py   # branch_labels = ("members_core",)
+└── members_core_0002_add_phone_number.py       # extends members_core@head
+```
+
+A separate module `foo` would have its own branch `foo_core` with revisions in its own directory:
+
+```
+src/modules/foo/migrations/versions/
+├── foo_core_0001_initial.py                    # branch_labels = ("foo_core",)
+└── foo_core_0002_add_index.py                  # extends foo_core@head
+```
+
+When `poe db-migrate` runs, Alembic upgrades all branches to their latest head — both `members_core` and `foo_core` are applied independently.
+
+### Extending a module's schema
+
+If you build a custom implementation that extends an existing module's schema, you can create a new branch that depends on a specific revision:
+
+```python
+revision = "members_custom_0001"
+down_revision = None
+branch_labels = ("members_custom",)
+depends_on = "members_core_0001"  # ensures the base schema exists first
+```
+
+This guarantees the base tables are created before your extension runs, while keeping both migration chains independent.

cofy_api-0.1.0/pyproject.toml

@@ -0,0 +1,76 @@
+[project]
+name = "cofy-api"
+authors = [
+    { name="EnergyID", email = "info@energieid.be" },
+    { name="Jorg" }
+]
+version = "0.1.0"
+description = "An open-source modular framework for ingesting, standardising, storing, and computing energy-related data."
+readme = "README.md"
+requires-python = ">=3.12"
+dependencies = [
+    "alembic>=1.18.4",
+    "entsoe-py>=0.7.10",
+    "fastapi[standard]>=0.128.0",
+    "narwhals>=2.15.0",
+    "pandas>=3.0.0",
+    "polars>=1.37.1",
+    "saq[redis,web]>=0.26.1",
+]
+[dependency-groups]
+dev = [
+    "pytest-asyncio>=1.3.0",
+    "pytest-cov>=7.0.0",
+    "pytest>=9.0.2",
+    "ruff>=0.14.13",
+    "ty>=0.0.12",
+]
+[tool.poe.tasks]
+lint = "ruff check --fix"
+format = "ruff format"
+check = "ty check"
+[tool.poe.tasks.demo]
+cmd = "fastapi dev src/demo/main.py"
+env = { PYTHONPATH = "." }
+envfile = ".env.local"
+[tool.poe.tasks.db-seed]
+cmd = "python -m src.demo.db.seed"
+envfile = ".env.local"
+env = { PYTHONPATH = "." }
+[tool.poe.tasks.db-reset]
+cmd = "python -m src.demo.db.reset"
+envfile = ".env.local"
+[tool.poe.tasks.db-migrate]
+cmd = "python -m src.demo.db.migrate"
+envfile = ".env.local"
+env = { PYTHONPATH = "." }
+[tool.poe.tasks.db-generate]
+cmd = "python -m src.demo.db.generate"
+envfile = ".env.local"
+env = { PYTHONPATH = "." }
+[tool.poe.tasks.worker]
+cmd = "saq src.demo.worker.settings"
+env = { PYTHONPATH = "." }
+envfile = ".env.local"
+[tool.poe.tasks.test]
+cmd = "pytest --cov=src --cov-report=term-missing --import-mode=importlib"
+env = { PYTHONPATH = "." }
+[tool.coverage.report]
+skip_covered = true
+[tool.coverage.run]
+source = ["src"]
+omit = ["src/demo/*", "src/modules/*/migrations/*", "src/cofy/db/alembic/*"]
+[tool.ruff]
+line-length = 120
+[tool.ruff.lint]
+select = [
+    "E", # pycodestyle
+    "F", # Pyflakes
+    "UP", # pyupgrade
+    "B", # flake8-bugbear
+    "SIM", # flake8-simplify
+    "I", # isort
+]
+ignore = [
+    "E501", #Line too long
+]

cofy_api-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

cofy_api-0.1.0/src/cofy/cofy_api.py

@@ -0,0 +1,43 @@
+from typing import Any
+
+from fastapi import FastAPI
+from fastapi.openapi.utils import get_openapi
+
+from src.cofy.docs_router import DocsRouter
+from src.shared.module import Module
+
+DEFAULT_ARGS: dict[str, Any] = {
+    "title": "Cofy API",
+    "version": "0.1.0",
+    "description": "Modular cloud API for energy data",
+    "docs_url": None,
+    "redoc_url": None,
+    "openapi_url": None,
+}
+
+
+class CofyApi(FastAPI):
+    def __init__(self, **kwargs):
+        super().__init__(**(DEFAULT_ARGS | kwargs))
+        self._modules: list[Module] = []
+        self.include_router(DocsRouter(self.openapi))
+
+    def openapi(self):
+        return get_openapi(
+            title=self.title,
+            version=self.version,
+            routes=self.routes,
+            tags=self.tags_metadata,
+        )
+
+    def register_module(self, module: Module):
+        self._modules.append(module)
+        self.include_router(module)
+
+    @property
+    def tags_metadata(self) -> list[dict[str, Any]]:
+        return [module.tag for module in self._modules]
+
+    @property
+    def modules(self) -> tuple[Module, ...]:
+        return tuple(self._modules)

cofy_api-0.1.0/src/cofy/db/alembic/env.py

@@ -0,0 +1,52 @@
+from logging.config import fileConfig
+
+from alembic import context
+from sqlalchemy import engine_from_config, pool
+
+from src.cofy.db.base import Base
+
+config = context.config
+
+if config.config_file_name is not None:
+    fileConfig(config.config_file_name)
+
+configured_metadata = config.attributes.get("target_metadata")
+if isinstance(configured_metadata, list):
+    target_metadata = configured_metadata[0] if configured_metadata else Base.metadata
+elif configured_metadata is not None:
+    target_metadata = configured_metadata
+else:
+    target_metadata = Base.metadata
+
+
+def run_migrations_offline() -> None:
+    url = config.get_main_option("sqlalchemy.url")
+    context.configure(
+        url=url,
+        target_metadata=target_metadata,
+        literal_binds=True,
+        dialect_opts={"paramstyle": "named"},
+    )
+
+    with context.begin_transaction():
+        context.run_migrations()
+
+
+def run_migrations_online() -> None:
+    connectable = engine_from_config(
+        config.get_section(config.config_ini_section, {}),
+        prefix="sqlalchemy.",
+        poolclass=pool.NullPool,
+    )
+
+    with connectable.connect() as connection:
+        context.configure(connection=connection, target_metadata=target_metadata)
+
+        with context.begin_transaction():
+            context.run_migrations()
+
+
+if context.is_offline_mode():
+    run_migrations_offline()
+else:
+    run_migrations_online()

cofy_api-0.1.0/src/cofy/db/base.py

@@ -0,0 +1,5 @@
+from sqlalchemy.orm import DeclarativeBase
+
+
+class Base(DeclarativeBase):
+    pass

cofy_api-0.1.0/src/cofy/db/cofy_db.py

@@ -0,0 +1,104 @@
+from __future__ import annotations
+
+from collections.abc import Iterable
+from importlib import resources
+from pathlib import Path
+from typing import TYPE_CHECKING, Any
+
+from alembic import command
+from alembic.config import Config
+from sqlalchemy import MetaData, create_engine
+
+from src.cofy.db.database_backed_source import DatabaseBackedSource
+
+if TYPE_CHECKING:
+    from sqlalchemy.engine import Engine
+
+    from src.cofy.cofy_api import CofyApi
+    from src.shared.module import Module
+
+
+class CofyDB:
+    def __init__(self, url: str | None = None, **engine_kwargs):
+        if url is None:
+            raise ValueError("CofyDB requires a database URL to be configured.")
+        self._sources: list[DatabaseBackedSource] = []
+        self._url = url
+        self.engine: Engine = create_engine(self._url, **engine_kwargs)
+
+    def register_module(self, module: Module):
+        source = getattr(module, "source", None)
+        if isinstance(source, DatabaseBackedSource):
+            self._sources.append(source)
+
+    def register_modules(self, modules: Iterable[Module]):
+        for module in modules:
+            self.register_module(module)
+
+    def bind_api(self, api: CofyApi):
+        self.register_modules(api.modules)
+
+    @property
+    def migration_locations(self) -> list[str]:
+        locations: list[str] = []
+        for source in self._sources:
+            for location in source.migration_locations:
+                resolved_location = str(Path(location).resolve())
+                if resolved_location not in locations:
+                    locations.append(resolved_location)
+        return locations
+
+    @property
+    def target_metadata(self) -> list[Any]:
+        metadata: list[Any] = []
+        for source in self._sources:
+            if source.target_metadata is not None and source.target_metadata not in metadata:
+                metadata.append(source.target_metadata)
+        return metadata
+
+    def _build_config(self) -> Config:
+        config = Config()
+        config.set_main_option("script_location", str(resources.files("src.cofy.db").joinpath("alembic")))
+        config.set_main_option("sqlalchemy.url", self._url)
+        config.set_main_option("path_separator", "os")
+
+        if self.migration_locations:
+            config.set_main_option("version_locations", " ".join(self.migration_locations))
+
+        config.attributes["target_metadata"] = self.target_metadata
+        return config
+
+    def run_migrations(self, revision: str = "heads") -> None:
+        command.upgrade(self._build_config(), revision)
+
+    def generate_migration(
+        self,
+        message: str,
+        head: str,
+        rev_id: str | None = None,
+        autogenerate: bool = True,
+    ) -> None:
+        """Generate a new Alembic migration revision for a specific module branch.
+
+        Args:
+            message: Description of the migration.
+            head: The branch head to extend, e.g. "members_core@head".
+            rev_id: Optional custom revision ID. If None, Alembic generates one.
+            autogenerate: Whether to autogenerate the migration from model changes.
+        """
+        command.revision(
+            self._build_config(),
+            message=message,
+            autogenerate=autogenerate,
+            head=head,
+            rev_id=rev_id,
+        )
+
+    def reset(self) -> None:
+        """usefull in development to reset the database to a clean state
+        Warning: this will delete all data in the database, use with caution!
+        """
+        meta = MetaData()
+        meta.reflect(bind=self.engine)
+        meta.drop_all(bind=self.engine)
+        self.run_migrations()

cofy_api-0.1.0/src/cofy/db/database_backed_source.py

@@ -0,0 +1,13 @@
+from collections.abc import Sequence
+from typing import Any, Protocol, runtime_checkable
+
+
+@runtime_checkable
+class DatabaseBackedSource(Protocol):
+    @property
+    def migration_locations(self) -> Sequence[str]:
+        """Filesystem paths containing alembic version files."""
+
+    @property
+    def target_metadata(self) -> Any | None:
+        """SQLAlchemy metadata used for migration autogeneration."""

cofy_api-0.1.0/src/cofy/db/timestamp_mixin.py

@@ -0,0 +1,9 @@
+import datetime as dt
+
+from sqlalchemy import DateTime, func
+from sqlalchemy.orm import Mapped, mapped_column
+
+
+class TimestampMixin:
+    created_at: Mapped[dt.datetime] = mapped_column(DateTime, nullable=False, default=func.now())
+    updated_at: Mapped[dt.datetime] = mapped_column(DateTime, nullable=False, default=func.now(), onupdate=func.now())

cofy_api-0.1.0/src/cofy/docs_router.py

@@ -0,0 +1,37 @@
+from collections.abc import Callable
+
+from fastapi import APIRouter, Request
+from fastapi.openapi.docs import get_swagger_ui_html
+from fastapi.responses import HTMLResponse
+
+
+class DocsRouter(APIRouter):
+    def __init__(self, get_openapi: Callable[[], dict]):
+        super().__init__()
+        self.get_openapi = get_openapi
+        self.add_api_route(
+            "/docs",
+            self.get_swagger_ui_html,
+            include_in_schema=False,
+        )
+        self.add_api_route("/openapi.json", self.get_openapi, include_in_schema=False)
+
+    async def get_swagger_ui_html(self, request: Request):
+        response = get_swagger_ui_html(
+            openapi_url="/openapi.json",
+            title="Docs",
+            swagger_ui_parameters={
+                "spec": self.get_openapi(),
+                "onComplete": "AUTHORIZE_API",
+            },
+        )
+
+        if hasattr(request.state, "auth_info"):
+            assert isinstance(response.body, bytes)
+            content = response.body.decode()
+            content = content.replace(
+                '"AUTHORIZE_API"',
+                f'() => ui.preauthorizeApiKey("{request.state.auth_info["scheme"]}", "{request.state.auth_info["content"]}")',
+            )
+            response = HTMLResponse(content=content, status_code=response.status_code)
+        return response