scse-spatial 1.0.0__tar.gz

scse_spatial-1.0.0/PKG-INFO

@@ -0,0 +1,136 @@
+Metadata-Version: 2.4
+Name: scse-spatial
+Version: 1.0.0
+Summary: A registry enabling the indexing and querying of spatial data resources on other services.
+Author: Max Wolschlager
+Author-email: rxtx@ungut.at
+Requires-Python: >=3.11,<3.15
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Requires-Dist: alembic (>=1.12.0,<2.0.0)
+Requires-Dist: dask (>=2024.0.0)
+Requires-Dist: geoalchemy2 (>=0.20.0,<0.21.0)
+Requires-Dist: geopandas (>=1.0,<2.0)
+Requires-Dist: jsonschema (>=4.26.0,<5.0.0)
+Requires-Dist: litestar (>=2.0.0,<3.0.0)
+Requires-Dist: mapbox-vector-tile (>=2,<3)
+Requires-Dist: morecantile (>=7.0.3,<8.0)
+Requires-Dist: netcdf4 (>=1.7.3,<2.0.0)
+Requires-Dist: pmtiles (>=3.7,<4)
+Requires-Dist: psycopg[binary] (>=3.1.12,<4.0.0)
+Requires-Dist: pydantic (>=2.4.2,<3.0.0)
+Requires-Dist: pydantic-settings (>=2.0.3,<3.0.0)
+Requires-Dist: pyproj (>=3.1,<4.0)
+Requires-Dist: pystac[validation] (>=1.12,<1.13)
+Requires-Dist: rasterio (>=1.4.3,<2.0.0)
+Requires-Dist: rich (>=13.9.4,<14.0.0)
+Requires-Dist: rioxarray (>=0.19,<1.0)
+Requires-Dist: s3fs (>=2024.10.0,<2027.0.0)
+Requires-Dist: scse-toolkit (>=2,<3)
+Requires-Dist: sqlalchemy (>=2.0.21,<3)
+Requires-Dist: tippecanoe (>=2.45)
+Requires-Dist: titiler-core (>=2.0,<3.0)
+Requires-Dist: titiler-xarray[fs] (>=2.0,<3.0)
+Requires-Dist: typer (>=0.24.1,<0.25.0)
+Requires-Dist: uvicorn (>=0.52.1,<0.53.0)
+Requires-Dist: xarray (>=2025.1.0,<2026)
+Requires-Dist: xxhash (>=3.4.1,<4.0.0)
+Requires-Dist: zarr (>=3.0.0,<4.0)
+Description-Content-Type: text/markdown
+
+# scse-spatial
+
+A registry that indexes and queries spatial data resources on other services.
+
+## Overview
+
+scse-spatial is the spatial data extension to a scenario services web
+application stack. It turns geodata files into fast, web-optimized resources
+and serves them to any map app.
+
+A workflow loads data. A web app reads it. This repository provides both
+sides: the ingest path and the HTTP API.
+
+### Feeding it
+
+Workflows push data in with the CLI or a few lines of Python. The `Spatial`
+object connects to a PostGIS database and to object storage. It exposes
+services for rasters, features, and a STAC catalog.
+
+```python
+with Spatial("<platform>", settings=Settings()) as spatial:
+    spatial.rasters.sync_hierarchy(
+        "emissions",
+        [path],
+        parameters_class=EmissionParameters,
+        metadata_class=EmissionMetadata,
+        extract_meta=extract_em_meta,
+    )
+    spatial.stac.export_catalog()
+```
+
+The platform converts each source file into optimized formats, uploads them,
+and indexes them. A re-run skips files whose content did not change.
+
+Workflows live in their own repositories. See `WORKFLOWS.md` for more information.
+
+### Connecting to a platform
+
+In multi-tenant deployments the connection details come from the manager
+API. Staff log in with their user account once; the credentials are cached
+locally and used in place of the shared service secret.
+
+```bash
+spatial login          # prompts for username and password, caches them
+spatial whoami         # show the identity the CLI would use
+spatial platforms      # list the platforms you can access
+```
+
+The manager DSN's secret material stays in the local environment through
+`{env:...}` placeholders. Imports are attributed to the logged-in user.
+
+### HTTP API
+
+Web apps can read data over HTTP. A Litestar server lists hierarchies and
+layers, renders map tiles through TiTiler, and streams vector tiles and
+GeoJSON from object storage.
+
+```bash
+curl -X PATCH http://<scse-spatial>/v1/{platform}/rasters/ \
+     -H "Content-Type: application/json" \
+     -d '{"hierarchy_id": 3, "parameters__contains": {"sector": "power"}}'
+```
+
+Point an scse-app at the `/v1` or `/titiler` endpoints, or open `/docs` and
+`/titiler/docs` for interactive API docs. See `HTTP_API.md` for more information.
+
+## Development Setup
+
+```bash
+# Install Poetry, minimum version >=1.2 required
+pipx install poetry
+
+# You may have to reinitialize your shell at this point.
+source ~/.bashrc
+
+# Activate in-project virtualenvs
+poetry config virtualenvs.in-project true
+
+# Add dynamic versioning plugin
+poetry self add "poetry-dynamic-versioning[plugin]"
+
+# Install dependencies
+# (using "--with dev" if dev dependencies should be installed as well)
+poetry install --with dev
+
+# Activate virtual environment
+poetry shell
+
+# Copy the template environment configuration
+cp template.env .env
+
+```
+

scse_spatial-1.0.0/README.md

@@ -0,0 +1,92 @@
+# scse-spatial
+
+A registry that indexes and queries spatial data resources on other services.
+
+## Overview
+
+scse-spatial is the spatial data extension to a scenario services web
+application stack. It turns geodata files into fast, web-optimized resources
+and serves them to any map app.
+
+A workflow loads data. A web app reads it. This repository provides both
+sides: the ingest path and the HTTP API.
+
+### Feeding it
+
+Workflows push data in with the CLI or a few lines of Python. The `Spatial`
+object connects to a PostGIS database and to object storage. It exposes
+services for rasters, features, and a STAC catalog.
+
+```python
+with Spatial("<platform>", settings=Settings()) as spatial:
+    spatial.rasters.sync_hierarchy(
+        "emissions",
+        [path],
+        parameters_class=EmissionParameters,
+        metadata_class=EmissionMetadata,
+        extract_meta=extract_em_meta,
+    )
+    spatial.stac.export_catalog()
+```
+
+The platform converts each source file into optimized formats, uploads them,
+and indexes them. A re-run skips files whose content did not change.
+
+Workflows live in their own repositories. See `WORKFLOWS.md` for more information.
+
+### Connecting to a platform
+
+In multi-tenant deployments the connection details come from the manager
+API. Staff log in with their user account once; the credentials are cached
+locally and used in place of the shared service secret.
+
+```bash
+spatial login          # prompts for username and password, caches them
+spatial whoami         # show the identity the CLI would use
+spatial platforms      # list the platforms you can access
+```
+
+The manager DSN's secret material stays in the local environment through
+`{env:...}` placeholders. Imports are attributed to the logged-in user.
+
+### HTTP API
+
+Web apps can read data over HTTP. A Litestar server lists hierarchies and
+layers, renders map tiles through TiTiler, and streams vector tiles and
+GeoJSON from object storage.
+
+```bash
+curl -X PATCH http://<scse-spatial>/v1/{platform}/rasters/ \
+     -H "Content-Type: application/json" \
+     -d '{"hierarchy_id": 3, "parameters__contains": {"sector": "power"}}'
+```
+
+Point an scse-app at the `/v1` or `/titiler` endpoints, or open `/docs` and
+`/titiler/docs` for interactive API docs. See `HTTP_API.md` for more information.
+
+## Development Setup
+
+```bash
+# Install Poetry, minimum version >=1.2 required
+pipx install poetry
+
+# You may have to reinitialize your shell at this point.
+source ~/.bashrc
+
+# Activate in-project virtualenvs
+poetry config virtualenvs.in-project true
+
+# Add dynamic versioning plugin
+poetry self add "poetry-dynamic-versioning[plugin]"
+
+# Install dependencies
+# (using "--with dev" if dev dependencies should be installed as well)
+poetry install --with dev
+
+# Activate virtual environment
+poetry shell
+
+# Copy the template environment configuration
+cp template.env .env
+
+```

scse_spatial-1.0.0/pyproject.toml

@@ -0,0 +1,182 @@
+[tool.poetry]
+name = "scse-spatial"
+version = "1.0.0"
+description = "A registry enabling the indexing and querying of spatial data resources on other services."
+authors = ["Max Wolschlager <rxtx@ungut.at>"]
+readme = "README.md"
+packages = [{ include = "spatial" }]
+requires-poetry = ">=2.0,<3.0"
+
+[tool.poetry.dependencies]
+python = ">=3.11, <3.15"
+alembic = "^1.12.0"
+sqlalchemy = ">=2.0.21,<3"
+psycopg = { extras = ["binary"], version = "^3.1.12" }
+geoalchemy2 = "^0.20.0"
+
+pydantic = "^2.4.2"
+litestar = "^2.0.0"
+uvicorn = "^0.52.1"
+
+pydantic-settings = "^2.0.3"
+xxhash = "^3.4.1"
+
+typer = "^0.24.1"
+geopandas = "^1.0"
+rasterio = "^1.4.3"
+pyproj = ">=3.1,<4.0"
+morecantile = ">=7.0.3,<8.0"
+rioxarray = ">=0.19,<1.0"
+xarray = ">=2025.1.0,<2026"
+dask = ">=2024.0.0"
+netcdf4 = "^1.7.3"
+scse-toolkit = ">=2,<3"
+titiler-core = "^2.0"
+titiler-xarray = { version = "^2.0", extras = ["fs"] }
+zarr = ">=3.0.0,<4.0"
+s3fs = ">=2024.10.0,<2027.0.0"
+pystac = { version = ">=1.12,<1.13", extras = ["validation"] }
+pmtiles = ">=3.7,<4"
+mapbox-vector-tile = ">=2,<3"
+tippecanoe = ">=2.45"
+rich = "^13.9.4"
+jsonschema = "^4.26.0"
+
+
+[tool.poetry.group.dev]
+optional = true
+
+[tool.poetry.group.dev.dependencies]
+pytest = "^7.4.2"
+ruff = ">=0.5.3"
+mypy = "^1.6.0"
+pytest-cov = "^4.1.0"
+httpx = "^0.27.2"
+pandas-stubs = "^3.0.0.260204"
+
+[tool.poetry.scripts]
+spatial = "spatial.__main__:app"
+
+[build-system]
+requires = ["poetry-core>=1.0.0", "poetry-dynamic-versioning>=1.0.0,<2.0.0"]
+build-backend = "poetry_dynamic_versioning.backend"
+
+[tool.ruff]
+# Exclude a variety of commonly ignored directories.
+exclude = [
+    ".eggs",
+    ".git",
+    ".git-rewrite",
+    ".mypy_cache",
+    ".pytype",
+    ".ruff_cache",
+    ".venv",
+    "__pypackages__",
+    "build",
+]
+
+# Same as Black.
+line-length = 88
+
+[tool.ruff.lint]
+# Enable the pycodestyle (`E`) and Pyflakes (`F`) rules by default.
+# Unlike Flake8, Ruff doesn't enable pycodestyle warnings (`W`) or
+# McCabe complexity (`C901`) by default.
+select = ["E", "F"]
+ignore = []
+
+fixable = ["ALL"]
+unfixable = []
+
+[tool.ruff.lint.per-file-ignores]
+"__init__.py" = ["E402", "F401", "F403"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+markers = [
+    "unit: heavily mocked unit tests",
+    "server: server endpoint tests",
+    "data: integration tests on a live database",
+]
+
+
+[tool.mypy]
+exclude = []
+disable_error_code = ['override']
+show_error_codes = true
+plugins = ['pydantic.mypy']
+# The following are equivalent to --strict mypy as seen in
+# https://mypy.readthedocs.io/en/stable/existing_code.html#introduce-stricter-options
+warn_unused_configs = true
+warn_redundant_casts = true
+warn_unused_ignores = true
+strict_equality = true
+extra_checks = true
+check_untyped_defs = true
+disallow_subclassing_any = true
+disallow_untyped_decorators = true
+disallow_any_generics = true
+disallow_untyped_calls = true
+disallow_incomplete_defs = true
+disallow_untyped_defs = true
+no_implicit_reexport = true
+warn_return_any = true
+# These are bonus, it seems:
+disallow_any_unimported = true
+no_implicit_optional = true
+warn_unreachable = true
+
+[[tool.mypy.overrides]]
+module = "geopandas.*"
+ignore_missing_imports = true
+
+[[tool.mypy.overrides]]
+module = "rasterio.*"
+ignore_missing_imports = true
+
+[[tool.mypy.overrides]]
+module = "pystac.*"
+ignore_missing_imports = true
+
+[[tool.mypy.overrides]]
+module = "tippecanoe.*"
+ignore_missing_imports = true
+
+[[tool.mypy.overrides]]
+module = "pmtiles.*"
+follow_imports = "skip"
+ignore_missing_imports = true
+
+[[tool.mypy.overrides]]
+module = "mapbox_vector_tile.*"
+ignore_missing_imports = true
+
+[[tool.mypy.overrides]]
+module = "shapely.*"
+ignore_missing_imports = true
+
+
+[[tool.mypy.overrides]]
+module = "fsspec.*"
+ignore_missing_imports = true
+
+
+[[tool.mypy.overrides]]
+module = "botocore.*"
+ignore_missing_imports = true
+
+
+[[tool.mypy.overrides]]
+module = "boto3.*"
+ignore_missing_imports = true
+
+
+[[tool.mypy.overrides]]
+module = "jsonschema.*"
+ignore_missing_imports = true
+
+
+[tool.poetry-dynamic-versioning]
+enable = false
+style = "pep440"
+vcs = "git"

scse_spatial-1.0.0/spatial/__init__.py

@@ -0,0 +1,7 @@
+from ._version import __version__ as __version__
+from ._version import __version_tuple__ as __version_tuple__
+from .spatial import Spatial as Spatial
+from .data.base.exceptions import Skip as Skip
+from .data.features import Feature as Feature
+from .data.rasters import RasterLayer as RasterLayer
+from .data.rasters import TileParams as TileParams

scse_spatial-1.0.0/spatial/__main__.py

@@ -0,0 +1,4 @@
+from spatial.cli import app
+
+if __name__ == "__main__":
+    app()

scse_spatial-1.0.0/spatial/_version.py

@@ -0,0 +1,2 @@
+__version__ = "1.0.0"
+__version_tuple__ = (1, 0, 0)

scse_spatial-1.0.0/spatial/cli/__init__.py

@@ -0,0 +1,19 @@
+"""Command line interface of the spatial data platform.
+
+See the CLI help for usage information:
+
+.. code:: bash
+
+    spatial --help
+    spatial server --help
+"""
+
+import typer
+from . import server, alembic, workflow, manager
+
+app = typer.Typer()
+app.add_typer(server.app, name="server")
+app.add_typer(alembic.app, name="alembic")
+app.add_typer(workflow.app)
+app.add_typer(manager.app)
+app.add_typer(manager.platforms_app, name="platforms")

scse_spatial-1.0.0/spatial/cli/alembic.py

@@ -0,0 +1,85 @@
+import typer
+from toolkit.db.alembic import AlembicCli
+from typing_extensions import Annotated
+
+from spatial.conf import Settings
+from spatial.conf.platforms import (
+    resolve_dsn_env_tokens,
+    resolve_spatial_dsn,
+)
+from spatial.db import get_alembic_controller
+
+app = AlembicCli(
+    help="Manages alembic migrations on selected platforms via subcommands."
+)
+
+
+def collect_platforms(
+    settings: Settings,
+    *,
+    platform: list[str] | None,
+    manager: bool,
+) -> list[str]:
+    """Collect the DSNs of the selected command targets."""
+    candidates: list[str] = []
+    if settings.manager_url is not None and (platform or manager):
+        try:
+            platforms = settings.get_manager_platforms()
+        except ValueError as exc:
+            raise typer.BadParameter(str(exc)) from exc
+        if manager:
+            for p in platforms.list_spatial_platforms():
+                try:
+                    dsn = resolve_spatial_dsn(p)
+                except ValueError:
+                    continue  # skip platforms with unresolvable dsn
+                candidates.append(dsn)
+
+        if platform is not None:
+            for name in platform:
+                platform_obj = platforms.get_spatial_platform(name)
+                dsn = resolve_spatial_dsn(platform_obj)
+                candidates.append(dsn)
+
+    if not candidates and settings.postgres_dsn is not None:
+        candidates.append(resolve_dsn_env_tokens(settings.postgres_dsn))
+
+    return candidates
+
+
+@app.callback()
+def alembic(
+    ctx: typer.Context,
+    platform: Annotated[
+        list[str] | None,
+        typer.Option(
+            "--platform",
+            "-p",
+            help="Use (a) platform name(s) as an alembic command target.",
+        ),
+    ] = None,
+    manager: Annotated[
+        bool,
+        typer.Option(
+            help="Use platforms from the manager api as alembic targets.",
+        ),
+    ] = False,
+) -> None:
+    """Run alembic commands on platform databases.
+    Use '--platform/-p' or '--manager' to select the
+    target databases. Falls back to
+    `SPATIAL_POSTGRES_DSN` if no platform is specified."""
+    settings = Settings()
+    settings.configure_logging()
+
+    dsns = collect_platforms(settings, platform=platform, manager=manager)
+
+    if len(dsns) == 0:
+        raise typer.BadParameter(
+            "No platforms found. Use '--platform/-p', '--manager', or set "
+            "`SPATIAL_POSTGRES_DSN`."
+        )
+
+    controllers = [get_alembic_controller(dsn) for dsn in dsns]
+    ctx.ensure_object(dict)
+    ctx.obj["controllers"] = controllers

scse_spatial-1.0.0/spatial/cli/imports.py

@@ -0,0 +1,89 @@
+"""Resolve ``<module>:<name>`` references passed on the command line.
+
+Workflows live in their own repositories, so the objects they hand to the
+CLI (Pydantic models, extractor callables) cannot be imported by the CLI
+itself. They are passed as ``<module>:<name>`` strings, resolved against
+the working directory, and validated against an expected kind.
+"""
+
+from __future__ import annotations
+
+import sys
+from importlib import import_module
+from typing import Any, Callable, cast
+
+import pydantic as pyd
+import typer
+
+ModelClassParser = Callable[[str], type[pyd.BaseModel]]
+CallableParser = Callable[[str], Callable[..., Any]]
+
+
+def import_member(reference: str) -> tuple[Any, str, str]:
+    """Import the member named by ``<module>:<name>``.
+
+    The working directory is added to ``sys.path`` so a workflow repository
+    can point at its own modules without being installed.
+
+    Returns
+    -------
+    tuple[Any, str, str]
+        The imported object, its name, and its module path.
+
+    Raises
+    ------
+    typer.BadParameter
+        If *reference* is malformed or the module cannot be imported.
+    """
+    if ":" not in reference:
+        raise typer.BadParameter(
+            f"'{reference}' is not a '<module>:<name>' reference."
+        )
+    module_name, name = reference.rsplit(":", 1)
+    if not module_name or not name:
+        raise typer.BadParameter(
+            f"'{reference}' is not a '<module>:<name>' reference."
+        )
+
+    cwd = "."
+    if cwd not in sys.path:
+        sys.path.insert(0, cwd)
+
+    try:
+        module = import_module(module_name)
+    except ImportError as exc:
+        raise typer.BadParameter(
+            f"Cannot import module '{module_name}' for reference '{reference}': {exc}"
+        ) from exc
+
+    member = getattr(module, name, None)
+    if member is None:
+        raise typer.BadParameter(
+            f"'{name}' does not exist in module '{module_name}'."
+        )
+    return member, name, module_name
+
+
+def parse_model_class(reference: str) -> type[pyd.BaseModel]:
+    """Parse ``<module>:<name>`` into a Pydantic model class."""
+    member, name, module_name = import_member(reference)
+    if not isinstance(member, type) or not issubclass(member, pyd.BaseModel):
+        raise typer.BadParameter(
+            f"'{name}' in module '{module_name}' is not a Pydantic model."
+        )
+    return member
+
+
+def parse_callable(reference: str) -> Callable[..., Any]:
+    """Parse ``<module>:<name>`` into a callable."""
+    member, name, module_name = import_member(reference)
+    if not callable(member):
+        raise typer.BadParameter(
+            f"'{name}' in module '{module_name}' is not callable."
+        )
+    return cast(Callable[..., Any], member)
+
+
+def parse_csv_paths(value: str) -> list[str]:
+    """Parse a comma-separated list of paths, dropping empty entries."""
+    return [item.strip() for item in value.split(",") if item.strip()]