buildingdata 0.2.0__tar.gz

buildingdata-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Yassine Abdelouadoud
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

buildingdata-0.2.0/PKG-INFO

@@ -0,0 +1,110 @@
+Metadata-Version: 2.4
+Name: buildingdata
+Version: 0.2.0
+Summary: Data management layer for buildingmodel — reference data download, BDTOPO retrieval, ERA5 weather
+License-Expression: MIT
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: google-cloud-storage>=2.0
+Requires-Dist: polars>=0.20
+Requires-Dist: geopandas>=0.14
+Requires-Dist: pyarrow>=14.0
+Requires-Dist: tqdm>=4.0
+Requires-Dist: requests>=2.28
+Requires-Dist: platformdirs>=3.0
+Provides-Extra: era5
+Requires-Dist: cdsapi>=0.6; extra == "era5"
+Requires-Dist: xarray>=2023.0; extra == "era5"
+Requires-Dist: pvlib>=0.10; extra == "era5"
+Requires-Dist: netcdf4>=1.6; extra == "era5"
+Requires-Dist: zarr>=2.18; extra == "era5"
+Provides-Extra: pipeline
+Requires-Dist: snakemake>=8.0; extra == "pipeline"
+Requires-Dist: openpyxl>=3.0; extra == "pipeline"
+Requires-Dist: py7zr>=0.20; extra == "pipeline"
+Provides-Extra: docs
+Requires-Dist: sphinx>=7.0; extra == "docs"
+Requires-Dist: sphinx-autoapi>=3.0; extra == "docs"
+Requires-Dist: pydata-sphinx-theme>=0.15; extra == "docs"
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Dynamic: license-file
+
+# buildingdata
+
+[![PyPI](https://img.shields.io/pypi/v/buildingdata.svg)](https://pypi.org/project/buildingdata/)
+[![Python](https://img.shields.io/pypi/pyversions/buildingdata.svg)](https://pypi.org/project/buildingdata/)
+
+**Data-management layer for [`buildingmodel`](https://gitlab.com/energytransition).**
+`buildingdata` delivers clean, ready-to-use building, demographic and weather
+datasets for French building energy inference, and caches everything locally so
+repeated calls don't re-download.
+
+```bash
+pip install buildingdata
+```
+
+## Quick start
+
+```python
+import buildingdata as bd
+
+# One-time configuration (bucket, cache dir, credentials).
+# The public bucket works anonymously, so this is optional.
+bd.configure()
+
+# Reference datasets (downloaded once from Google Cloud Storage, then cached)
+census    = bd.get_census()        # INSEE census            -> polars DataFrame
+districts = bd.get_districts()     # IRIS district geometry  -> geopandas GeoDataFrame
+diagnosis = bd.get_diagnosis()     # ADEME energy diagnoses  -> polars DataFrame
+gas       = bd.get_gas_network()   # GRDF gas network routes -> geopandas GeoDataFrame
+
+# On-demand datasets (fetched live from public APIs)
+buildings = bd.get_bdtopo("751010101")           # per-IRIS building geometry (IGN WFS)
+epw       = bd.get_era5_climate(48.85, 2.35, 2020)  # ERA5 weather -> synthetic EPW
+```
+
+You can also configure from the command line:
+
+```bash
+buildingdata configure --bucket my-bucket --cache-dir ~/.cache/buildingdata
+```
+
+## What it provides
+
+| Function | Source | Returns |
+| --- | --- | --- |
+| `get_census()` | INSEE census (GCS) | polars `DataFrame` |
+| `get_districts()` | IRIS geometries (GCS) | geopandas `GeoDataFrame` |
+| `get_diagnosis()` | ADEME energy performance diagnoses (GCS) | polars `DataFrame` |
+| `get_gas_network()` | GRDF gas network routes (GCS) | geopandas `GeoDataFrame` |
+| `get_bdtopo(iris_code)` | IGN Géoplateforme WFS (live) | geopandas `GeoDataFrame` |
+| `get_era5_climate(lat, lon, year)` | Copernicus CDS (live) | path to EPW file |
+
+Reference datasets are pulled from a public Google Cloud Storage bucket and
+cached locally with generation-based freshness checks. French geospatial data
+uses CRS **EPSG:2154 (Lambert-93)**.
+
+## Configuration
+
+Settings are resolved from (in order) explicit arguments, environment variables,
+and `~/.config/buildingdata/config.ini`:
+
+- **bucket** — GCS bucket holding the reference datasets
+- **cache directory** — where downloaded data is stored locally
+- **credentials** — path to a GCS service-account JSON (omit for anonymous access
+  to the public bucket)
+
+## Installation extras
+
+```bash
+pip install "buildingdata[era5]"   # ERA5 weather (cdsapi, xarray, pvlib, ...)
+pip install "buildingdata[docs]"   # build the Sphinx documentation
+```
+
+Requires **Python ≥ 3.10**.
+
+## License
+
+Released under the [MIT License](LICENSE).

buildingdata-0.2.0/README.md

@@ -0,0 +1,77 @@
+# buildingdata
+
+[![PyPI](https://img.shields.io/pypi/v/buildingdata.svg)](https://pypi.org/project/buildingdata/)
+[![Python](https://img.shields.io/pypi/pyversions/buildingdata.svg)](https://pypi.org/project/buildingdata/)
+
+**Data-management layer for [`buildingmodel`](https://gitlab.com/energytransition).**
+`buildingdata` delivers clean, ready-to-use building, demographic and weather
+datasets for French building energy inference, and caches everything locally so
+repeated calls don't re-download.
+
+```bash
+pip install buildingdata
+```
+
+## Quick start
+
+```python
+import buildingdata as bd
+
+# One-time configuration (bucket, cache dir, credentials).
+# The public bucket works anonymously, so this is optional.
+bd.configure()
+
+# Reference datasets (downloaded once from Google Cloud Storage, then cached)
+census    = bd.get_census()        # INSEE census            -> polars DataFrame
+districts = bd.get_districts()     # IRIS district geometry  -> geopandas GeoDataFrame
+diagnosis = bd.get_diagnosis()     # ADEME energy diagnoses  -> polars DataFrame
+gas       = bd.get_gas_network()   # GRDF gas network routes -> geopandas GeoDataFrame
+
+# On-demand datasets (fetched live from public APIs)
+buildings = bd.get_bdtopo("751010101")           # per-IRIS building geometry (IGN WFS)
+epw       = bd.get_era5_climate(48.85, 2.35, 2020)  # ERA5 weather -> synthetic EPW
+```
+
+You can also configure from the command line:
+
+```bash
+buildingdata configure --bucket my-bucket --cache-dir ~/.cache/buildingdata
+```
+
+## What it provides
+
+| Function | Source | Returns |
+| --- | --- | --- |
+| `get_census()` | INSEE census (GCS) | polars `DataFrame` |
+| `get_districts()` | IRIS geometries (GCS) | geopandas `GeoDataFrame` |
+| `get_diagnosis()` | ADEME energy performance diagnoses (GCS) | polars `DataFrame` |
+| `get_gas_network()` | GRDF gas network routes (GCS) | geopandas `GeoDataFrame` |
+| `get_bdtopo(iris_code)` | IGN Géoplateforme WFS (live) | geopandas `GeoDataFrame` |
+| `get_era5_climate(lat, lon, year)` | Copernicus CDS (live) | path to EPW file |
+
+Reference datasets are pulled from a public Google Cloud Storage bucket and
+cached locally with generation-based freshness checks. French geospatial data
+uses CRS **EPSG:2154 (Lambert-93)**.
+
+## Configuration
+
+Settings are resolved from (in order) explicit arguments, environment variables,
+and `~/.config/buildingdata/config.ini`:
+
+- **bucket** — GCS bucket holding the reference datasets
+- **cache directory** — where downloaded data is stored locally
+- **credentials** — path to a GCS service-account JSON (omit for anonymous access
+  to the public bucket)
+
+## Installation extras
+
+```bash
+pip install "buildingdata[era5]"   # ERA5 weather (cdsapi, xarray, pvlib, ...)
+pip install "buildingdata[docs]"   # build the Sphinx documentation
+```
+
+Requires **Python ≥ 3.10**.
+
+## License
+
+Released under the [MIT License](LICENSE).

buildingdata-0.2.0/buildingdata/__init__.py

@@ -0,0 +1,26 @@
+# -*- coding: utf-8 -*-
+from .config import write_config
+from .reference import (
+    get_census,
+    get_diagnosis,
+    get_districts,
+    get_elmas,
+    get_enedis_national,
+    get_enedis_regional,
+    get_gas_network,
+    get_occupant_diaries,
+    get_ore,
+)
+from .simulation import get_bdtopo, get_era5_climate
+
+
+def configure(bucket=None, cache_dir=None, credentials=None):
+    """Write or update ~/.config/buildingdata/config.ini.
+
+    Args:
+        bucket (str, optional): GCS bucket name.
+        cache_dir (str or pathlib.Path, optional): local cache directory.
+        credentials (str or pathlib.Path, optional): path to GCS service
+            account JSON. Pass None to use Application Default Credentials.
+    """
+    write_config(bucket=bucket, cache_dir=cache_dir, credentials=credentials)

buildingdata-0.2.0/buildingdata/_cli.py

@@ -0,0 +1,29 @@
+# -*- coding: utf-8 -*-
+import argparse
+
+from .config import write_config
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        prog="buildingdata",
+        description="Configure the buildingdata package.",
+    )
+    sub = parser.add_subparsers(dest="command")
+
+    cfg_parser = sub.add_parser("configure", help="Write ~/.config/buildingdata/config.ini")
+    cfg_parser.add_argument("--bucket", default=None, help="GCS bucket name")
+    cfg_parser.add_argument("--cache-dir", default=None, dest="cache_dir", help="Local cache directory")
+    cfg_parser.add_argument("--credentials", default=None, help="Path to GCS service account JSON")
+
+    args = parser.parse_args()
+
+    if args.command == "configure":
+        write_config(
+            bucket=args.bucket,
+            cache_dir=args.cache_dir,
+            credentials=args.credentials,
+        )
+        print("Configuration saved.")
+    else:
+        parser.print_help()

buildingdata-0.2.0/buildingdata/cache.py

@@ -0,0 +1,92 @@
+# -*- coding: utf-8 -*-
+import json
+from datetime import datetime, timezone
+from pathlib import Path
+
+from .config import get_cache_dir
+from .exceptions import CacheError
+
+
+def cache_path(name):
+    """Return the local Path where a named artifact should be cached.
+
+    Args:
+        name (str): logical artifact name, e.g. "census_latest.parquet".
+
+    Returns:
+        pathlib.Path: absolute path under the configured cache dir.
+    """
+    return get_cache_dir() / name
+
+
+def _sidecar_path(name):
+    return get_cache_dir() / (name + ".meta.json")
+
+
+def is_cached(name):
+    """Return True if the artifact file and a valid sidecar both exist.
+
+    Does not contact GCS. Use needs_refresh() for a freshness check.
+
+    Args:
+        name (str): logical artifact name.
+
+    Returns:
+        bool: True if the local copy can be used without downloading.
+    """
+    return cache_path(name).exists() and _sidecar_path(name).exists()
+
+
+def needs_refresh(name, blob):
+    """Return True if the local copy is older than the GCS blob.
+
+    Args:
+        name (str): logical artifact name.
+        blob (google.cloud.storage.Blob): live blob object from GCS.
+
+    Returns:
+        bool: True if the blob generation has advanced past the cached one.
+    """
+    sidecar = _sidecar_path(name)
+    if not sidecar.exists():
+        return True
+    try:
+        with open(sidecar) as f:
+            meta = json.load(f)
+        return int(meta.get("generation", -1)) < blob.generation
+    except Exception:
+        return True
+
+
+def write_sidecar(name, blob):
+    """Write a .meta.json sidecar after a successful download.
+
+    Args:
+        name (str): logical artifact name.
+        blob (google.cloud.storage.Blob): the blob that was downloaded.
+    """
+    sidecar = _sidecar_path(name)
+    meta = {
+        "blob_name": blob.name,
+        "generation": blob.generation,
+        "downloaded_at": datetime.now(timezone.utc).isoformat(),
+    }
+    try:
+        with open(sidecar, "w") as f:
+            json.dump(meta, f, indent=2)
+    except OSError as e:
+        raise CacheError(f"Failed to write cache sidecar {sidecar}: {e}") from e
+
+
+def ensure_subdir(subdir):
+    """Return a subdirectory of the cache dir, creating it if absent.
+
+    Args:
+        subdir (str): subdirectory name, e.g. "bdtopo" or "era5".
+
+    Returns:
+        pathlib.Path: absolute path to the subdirectory.
+    """
+    path = get_cache_dir() / subdir
+    path.mkdir(parents=True, exist_ok=True)
+    return path

buildingdata-0.2.0/buildingdata/config.py

@@ -0,0 +1,119 @@
+# -*- coding: utf-8 -*-
+import configparser
+import hashlib
+import os
+from pathlib import Path
+
+from platformdirs import user_data_dir
+
+_DEFAULT_BUCKET = "building-inference-data"
+_CONFIG_FILE = Path.home() / ".config" / "buildingdata" / "config.ini"
+
+
+def _install_id():
+    """Return a short, stable id for this package installation.
+
+    Derived from the on-disk location of the installed package, so that
+    each install (e.g. a separate virtualenv) maps to a distinct id while
+    reinstalling in place keeps the same one.
+
+    Returns:
+        str: 8-char hex digest identifying this installation.
+    """
+    install_root = str(Path(__file__).resolve().parent)
+    return hashlib.sha256(install_root.encode("utf-8")).hexdigest()[:8]
+
+
+def _default_cache_dir():
+    """Return the per-installation default cache directory.
+
+    Uses the platform user-data dir as the base and namespaces it by
+    installation id so multiple installs do not overwrite each other's
+    cache.
+
+    Returns:
+        pathlib.Path: absolute path to the default cache directory.
+    """
+    return Path(user_data_dir("buildingdata")) / "cache" / _install_id()
+
+
+def _read_config():
+    cfg = configparser.ConfigParser()
+    if _CONFIG_FILE.exists():
+        cfg.read(_CONFIG_FILE)
+    return cfg
+
+
+def get_bucket():
+    """Return the GCS bucket name.
+
+    Precedence: BUILDINGDATA_BUCKET env var → config file → default.
+
+    Returns:
+        str: GCS bucket name without gs:// prefix.
+    """
+    if "BUILDINGDATA_BUCKET" in os.environ:
+        return os.environ["BUILDINGDATA_BUCKET"]
+    cfg = _read_config()
+    return cfg.get("gcs", "bucket", fallback=_DEFAULT_BUCKET)
+
+
+def get_cache_dir():
+    """Return the local cache directory, creating it if absent.
+
+    Precedence: BUILDINGDATA_CACHE_DIR env var → config file → default.
+
+    Returns:
+        pathlib.Path: absolute path to the cache directory.
+    """
+    if "BUILDINGDATA_CACHE_DIR" in os.environ:
+        path = Path(os.environ["BUILDINGDATA_CACHE_DIR"])
+    else:
+        cfg = _read_config()
+        raw = cfg.get("cache", "dir", fallback=str(_default_cache_dir()))
+        path = Path(raw)
+    path.mkdir(parents=True, exist_ok=True)
+    return path
+
+
+def get_credentials_file():
+    """Return path to a GCS service account JSON, or None for anonymous access.
+
+    When None, the storage client reads public buckets anonymously, with no
+    authentication required.
+
+    Precedence: GOOGLE_APPLICATION_CREDENTIALS env var → config file → None.
+
+    Returns:
+        pathlib.Path or None: path to credentials JSON, or None.
+    """
+    if "GOOGLE_APPLICATION_CREDENTIALS" in os.environ:
+        return Path(os.environ["GOOGLE_APPLICATION_CREDENTIALS"])
+    cfg = _read_config()
+    raw = cfg.get("gcs", "credentials", fallback=None)
+    return Path(raw) if raw else None
+
+
+def write_config(bucket=None, cache_dir=None, credentials=None):
+    """Write or update ~/.config/buildingdata/config.ini.
+
+    Args:
+        bucket (str, optional): GCS bucket name.
+        cache_dir (str or pathlib.Path, optional): local cache directory.
+        credentials (str or pathlib.Path, optional): path to GCS service
+            account JSON. Pass None to leave unchanged.
+    """
+    _CONFIG_FILE.parent.mkdir(parents=True, exist_ok=True)
+    cfg = _read_config()
+    if not cfg.has_section("gcs"):
+        cfg.add_section("gcs")
+    if not cfg.has_section("cache"):
+        cfg.add_section("cache")
+    if bucket is not None:
+        cfg.set("gcs", "bucket", bucket)
+    if credentials is not None:
+        cfg.set("gcs", "credentials", str(credentials))
+    if cache_dir is not None:
+        cfg.set("cache", "dir", str(cache_dir))
+    with open(_CONFIG_FILE, "w") as f:
+        cfg.write(f)

buildingdata-0.2.0/buildingdata/exceptions.py

@@ -0,0 +1,17 @@
+# -*- coding: utf-8 -*-
+
+
+class BuildingDataError(Exception):
+    """Base exception for buildingdata errors."""
+
+
+class CacheError(BuildingDataError):
+    """Raised when a cache read or write operation fails."""
+
+
+class RemoteNotAvailableError(BuildingDataError):
+    """Raised when a remote endpoint returns a non-success status."""
+
+
+class ConfigurationError(BuildingDataError):
+    """Raised when required configuration (credentials, API key) is missing."""

buildingdata-0.2.0/buildingdata/gcs.py

@@ -0,0 +1,82 @@
+# -*- coding: utf-8 -*-
+from pathlib import Path
+
+from tqdm import tqdm
+
+from .config import get_bucket, get_credentials_file
+
+
+def get_client():
+    """Return a google.cloud.storage.Client.
+
+    Uses the credentials file from config if one is set. Otherwise returns
+    an anonymous client, which can read public buckets without any
+    authentication or Google Cloud project.
+
+    Returns:
+        google.cloud.storage.Client: storage client.
+    """
+    from google.cloud import storage
+
+    creds_file = get_credentials_file()
+    if creds_file is not None:
+        return storage.Client.from_service_account_json(str(creds_file))
+    return storage.Client.create_anonymous_client()
+
+
+def get_blob(name):
+    """Return the Blob object for a given name, or None if not found.
+
+    Args:
+        name (str): blob name inside the configured bucket.
+
+    Returns:
+        google.cloud.storage.Blob or None.
+    """
+    client = get_client()
+    bucket = client.bucket(get_bucket())
+    blob = bucket.blob(name)
+    blob.reload()
+    return blob if blob.exists() else None
+
+
+def download_blob(name, dest_path, show_progress=True):
+    """Download a blob from the configured bucket to dest_path.
+
+    Args:
+        name (str): blob name inside the bucket, e.g. "census_latest.parquet".
+        dest_path (pathlib.Path): local destination file path.
+        show_progress (bool): display a tqdm progress bar. Defaults to True.
+    """
+    from google.cloud import storage
+
+    dest_path = Path(dest_path)
+    dest_path.parent.mkdir(parents=True, exist_ok=True)
+
+    client = get_client()
+    bucket = client.bucket(get_bucket())
+    blob = bucket.blob(name)
+    blob.reload()
+
+    total = blob.size or 0
+    tmp_path = dest_path.with_suffix(dest_path.suffix + ".tmp")
+
+    with open(tmp_path, "wb") as f:
+        if show_progress:
+            with tqdm(
+                total=total,
+                unit="B",
+                unit_scale=True,
+                desc=name,
+                leave=True,
+            ) as bar:
+                def _callback(chunk):
+                    bar.update(len(chunk))
+
+                blob.download_to_file(f, checksum="md5", raw_download=True)
+                bar.update(total - bar.n)
+        else:
+            blob.download_to_file(f)
+
+    tmp_path.replace(dest_path)
+    return blob

buildingdata-0.2.0/buildingdata/reference/__init__.py

@@ -0,0 +1,9 @@
+# -*- coding: utf-8 -*-
+from .census import get_census
+from .diagnosis import get_diagnosis
+from .districts import get_districts
+from .elmas import get_elmas
+from .enedis import get_enedis_national, get_enedis_regional
+from .gas_network import get_gas_network
+from .occupant_diaries import get_occupant_diaries
+from .ore import get_ore

buildingdata-0.2.0/buildingdata/reference/census.py

@@ -0,0 +1,48 @@
+# -*- coding: utf-8 -*-
+import polars as pl
+
+from ..cache import cache_path, is_cached, needs_refresh, write_sidecar
+from ..exceptions import RemoteNotAvailableError
+from ..gcs import download_blob, get_blob
+
+_BLOB_NAME = "census_latest.parquet"
+
+
+def get_census(city_group_list=None, refresh=False):
+    """Return the processed INSEE census DataFrame, optionally pre-filtered.
+
+    Downloads census_latest.parquet from GCS to the local cache on first call
+    (or when refresh=True). Filtering by city_group_list uses Polars lazy scan
+    so only the needed rows are loaded into memory.
+
+    Args:
+        city_group_list (list of str, optional): EPCI codes to filter on. If
+            None, the full national census is returned. Defaults to None.
+        refresh (bool): force re-download even if the cache is warm.
+            Defaults to False.
+
+    Returns:
+        polars.DataFrame: census records with columns occupancy_type,
+            occupant_status, living_area_class, occupant_count, heating_system,
+            residential_type, construction_year_class, district, city,
+            city_group, department, region, main_cooking_energy,
+            secondary_cooking_energy, ipondl.
+    """
+    dest = cache_path(_BLOB_NAME)
+
+    if refresh or not is_cached(_BLOB_NAME):
+        blob = get_blob(_BLOB_NAME)
+        if blob is None:
+            raise RemoteNotAvailableError(f"Blob {_BLOB_NAME!r} not found in GCS bucket.")
+        download_blob(_BLOB_NAME, dest)
+        write_sidecar(_BLOB_NAME, blob)
+    elif refresh:
+        blob = get_blob(_BLOB_NAME)
+        if blob is not None and needs_refresh(_BLOB_NAME, blob):
+            download_blob(_BLOB_NAME, dest)
+            write_sidecar(_BLOB_NAME, blob)
+
+    lf = pl.scan_parquet(dest)
+    if city_group_list is not None:
+        lf = lf.filter(pl.col("city_group").is_in(city_group_list))
+    return lf.collect()