buildingdata 0.7.0__tar.gz → 0.9.0__tar.gz

{buildingdata-0.7.0 → buildingdata-0.9.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: buildingdata
-Version: 0.7.0
+Version: 0.9.0
 Summary: Data management layer for buildingmodel — reference data download, BDTOPO retrieval, ERA5 weather
 License-Expression: MIT
 Requires-Python: >=3.10
@@ -22,9 +22,12 @@ Requires-Dist: zarr>=2.18; extra == "era5"
 Requires-Dist: dask>=2024.1; extra == "era5"
 Provides-Extra: bulk
 Requires-Dist: py7zr>=0.20; extra == "bulk"
+Provides-Extra: timezone
+Requires-Dist: timezonefinder>=6.0; extra == "timezone"
 Provides-Extra: pipeline
 Requires-Dist: snakemake>=8.0; extra == "pipeline"
 Requires-Dist: openpyxl>=3.0; extra == "pipeline"
+Requires-Dist: fastexcel>=0.11; extra == "pipeline"
 Requires-Dist: py7zr>=0.20; extra == "pipeline"
 Provides-Extra: docs
 Requires-Dist: sphinx>=7.0; extra == "docs"
@@ -66,6 +69,9 @@ gas       = bd.get_gas_network()   # GRDF gas network routes -> geopandas GeoDat
 # On-demand datasets (fetched live from public APIs)
 buildings = bd.get_bdtopo("751010101")           # per-IRIS building geometry (IGN WFS)
 epw       = bd.get_era5_climate(48.85, 2.35, 2020)  # ERA5 weather -> synthetic EPW
+                                                    # hour axis is UTC; pass
+                                                    # timezone="Europe/Paris"
+                                                    # for local standard time
 ```
 
 You can also configure from the command line:
@@ -76,19 +82,32 @@ buildingdata configure --bucket my-bucket --cache-dir ~/.cache/buildingdata
 
 ## What it provides
 
-| Function | Source | Returns | Consumed by |
-| --- | --- | --- | --- |
-| `get_census()` | INSEE census (GCS) | polars `DataFrame` | buildingmodel — dwelling inference |
-| `get_districts()` | IRIS geometries (GCS) | geopandas `GeoDataFrame` | buildingmodel — geocoding; building_eload — district lookup |
-| `get_diagnosis()` | ADEME energy performance diagnoses (GCS) | polars `DataFrame` | buildingmodel — envelope & system inference |
-| `get_gas_network()` | GRDF gas network routes (GCS) | geopandas `GeoDataFrame` | buildingmodel — gas-connection inference |
-| `get_ore()` | Agence ORE annual consumption per IRIS (GCS) | polars `DataFrame` | building_eload — calibration target |
-| `get_enedis_national()` / `get_enedis_regional()` | Enedis conso-inf36 measured consumption (GCS) | polars `DataFrame` | building_eload — validation reference |
-| `get_elmas(table)` | ELMAS non-residential load curves (GCS) | polars `DataFrame` | building_eload — non-residential model |
-| `get_occupant_diaries()` | synthetic occupant activity calendar (GCS) | polars `DataFrame` | building_eload — occupancy models |
-| `get_elecdom()` | Enedis panel Elecdom end-use load curves (GCS) | polars `DataFrame` | provenance only — source of building_eload constants |
-| `get_bdtopo(iris_code)` | IGN Géoplateforme WFS (live) or bulk cache | geopandas `GeoDataFrame` | building_eload → buildingmodel — input geometry |
-| `get_era5_climate(lat, lon, year)` | Copernicus CDS (live) or bulk cache | path to EPW file | buildingmodel — weather boundary condition |
+| Function | Source | Licence | Returns | Consumed by |
+| --- | --- | --- | --- | --- |
+| `get_census()` | INSEE census (GCS) | Licence Ouverte 2.0 | polars `DataFrame` | buildingmodel — dwelling inference |
+| `get_census_individuals()` | INSEE census individuals, activitymodel vocabulary (GCS) | Licence Ouverte 2.0 | polars `DataFrame` | buildingmodel — occupant inference (planned) |
+| `get_districts()` | IRIS geometries (GCS) | Licence Ouverte 2.0 | geopandas `GeoDataFrame` | buildingmodel — geocoding; building_eload — district lookup |
+| `get_diagnosis()` | ADEME energy performance diagnoses (GCS) | Licence Ouverte 2.0 | polars `DataFrame` | buildingmodel — envelope & system inference |
+| `get_gas_network()` | GRDF gas network routes (GCS) | Licence Ouverte (Etalab) | geopandas `GeoDataFrame` | buildingmodel — gas-connection inference |
+| `get_ore()` | Agence ORE annual consumption per IRIS (GCS) | Licence Ouverte 2.0 | polars `DataFrame` | building_eload — calibration target |
+| `get_enedis_national()` / `get_enedis_regional()` | Enedis conso-inf36 measured consumption (GCS) | Licence Ouverte 2.0 | polars `DataFrame` | building_eload — validation reference |
+| `get_elmas(table)` | ELMAS non-residential load curves (GCS) | **CC BY 4.0** | polars `DataFrame` | building_eload — non-residential model |
+| `get_occupant_diaries()` | synthetic occupant activity calendar (GCS) | maintainers' synthetic artifact; derived from a **restricted-access** survey | polars `DataFrame` | building_eload — occupancy models |
+| `get_activity_library(table)` | activitymodel diary library from the INSEE time-use survey (GCS) | maintainers' derived artifact, no survey identifier; derived from a **restricted-access** survey | polars `DataFrame` | building_eload — occupant calendars via activitymodel (planned) |
+| `get_elecdom()` | Enedis panel Elecdom end-use load curves (GCS) | Licence Ouverte 2.0 | polars `DataFrame` | provenance only — source of building_eload constants |
+| `get_bdtopo(iris_code)` | IGN Géoplateforme WFS (live) or bulk cache | Licence Ouverte 2.0 | geopandas `GeoDataFrame` | building_eload → buildingmodel — input geometry |
+| `get_era5_climate(lat, lon, year)` | Copernicus CDS (live) or bulk cache | **CC BY 4.0** (CDS account required) | path to EPW file (hour axis **UTC** by default, see `timezone=`) | buildingmodel — weather boundary condition |
+
+The MIT `LICENSE` covers the **code only**. Each dataset keeps its upstream
+licence, and every derived blob inherits the licence of the source it came
+from — there is no separate "buildingdata data licence". Two rows are not
+Licence Ouverte, and both matter for a publication: ELMAS is CC BY 4.0 (cite the
+paper), and ERA5 is CC BY 4.0 behind a personal Copernicus account, so an
+ERA5-dependent workflow is not anonymously reproducible. The occupant diaries
+and the activity library derive from the INSEE *Enquête Emploi du temps
+2009-2010*, which is obtained on request and is not redistributed here. The *Datasets* documentation
+page carries the per-dataset attribution string to reproduce, the exact upstream
+vintage, and the verification date.
 
 Reference datasets are pulled from a public Google Cloud Storage bucket and
 cached locally with generation-based freshness checks. French geospatial data
@@ -162,6 +181,7 @@ Because `~/.config/buildingdata/config.ini` is stored in your home directory, co
 ```bash
 pip install "buildingdata[era5]"   # ERA5 weather (cdsapi, xarray, pvlib, ...)
 pip install "buildingdata[bulk]"   # BDTOPO bulk prefetch (py7zr)
+pip install "buildingdata[timezone]"  # ERA5 timezone="local_standard" (timezonefinder)
 pip install "buildingdata[docs]"   # build the Sphinx documentation
 ```
 

{buildingdata-0.7.0 → buildingdata-0.9.0}/README.md

@@ -30,6 +30,9 @@ gas       = bd.get_gas_network()   # GRDF gas network routes -> geopandas GeoDat
 # On-demand datasets (fetched live from public APIs)
 buildings = bd.get_bdtopo("751010101")           # per-IRIS building geometry (IGN WFS)
 epw       = bd.get_era5_climate(48.85, 2.35, 2020)  # ERA5 weather -> synthetic EPW
+                                                    # hour axis is UTC; pass
+                                                    # timezone="Europe/Paris"
+                                                    # for local standard time
 ```
 
 You can also configure from the command line:
@@ -40,19 +43,32 @@ buildingdata configure --bucket my-bucket --cache-dir ~/.cache/buildingdata
 
 ## What it provides
 
-| Function | Source | Returns | Consumed by |
-| --- | --- | --- | --- |
-| `get_census()` | INSEE census (GCS) | polars `DataFrame` | buildingmodel — dwelling inference |
-| `get_districts()` | IRIS geometries (GCS) | geopandas `GeoDataFrame` | buildingmodel — geocoding; building_eload — district lookup |
-| `get_diagnosis()` | ADEME energy performance diagnoses (GCS) | polars `DataFrame` | buildingmodel — envelope & system inference |
-| `get_gas_network()` | GRDF gas network routes (GCS) | geopandas `GeoDataFrame` | buildingmodel — gas-connection inference |
-| `get_ore()` | Agence ORE annual consumption per IRIS (GCS) | polars `DataFrame` | building_eload — calibration target |
-| `get_enedis_national()` / `get_enedis_regional()` | Enedis conso-inf36 measured consumption (GCS) | polars `DataFrame` | building_eload — validation reference |
-| `get_elmas(table)` | ELMAS non-residential load curves (GCS) | polars `DataFrame` | building_eload — non-residential model |
-| `get_occupant_diaries()` | synthetic occupant activity calendar (GCS) | polars `DataFrame` | building_eload — occupancy models |
-| `get_elecdom()` | Enedis panel Elecdom end-use load curves (GCS) | polars `DataFrame` | provenance only — source of building_eload constants |
-| `get_bdtopo(iris_code)` | IGN Géoplateforme WFS (live) or bulk cache | geopandas `GeoDataFrame` | building_eload → buildingmodel — input geometry |
-| `get_era5_climate(lat, lon, year)` | Copernicus CDS (live) or bulk cache | path to EPW file | buildingmodel — weather boundary condition |
+| Function | Source | Licence | Returns | Consumed by |
+| --- | --- | --- | --- | --- |
+| `get_census()` | INSEE census (GCS) | Licence Ouverte 2.0 | polars `DataFrame` | buildingmodel — dwelling inference |
+| `get_census_individuals()` | INSEE census individuals, activitymodel vocabulary (GCS) | Licence Ouverte 2.0 | polars `DataFrame` | buildingmodel — occupant inference (planned) |
+| `get_districts()` | IRIS geometries (GCS) | Licence Ouverte 2.0 | geopandas `GeoDataFrame` | buildingmodel — geocoding; building_eload — district lookup |
+| `get_diagnosis()` | ADEME energy performance diagnoses (GCS) | Licence Ouverte 2.0 | polars `DataFrame` | buildingmodel — envelope & system inference |
+| `get_gas_network()` | GRDF gas network routes (GCS) | Licence Ouverte (Etalab) | geopandas `GeoDataFrame` | buildingmodel — gas-connection inference |
+| `get_ore()` | Agence ORE annual consumption per IRIS (GCS) | Licence Ouverte 2.0 | polars `DataFrame` | building_eload — calibration target |
+| `get_enedis_national()` / `get_enedis_regional()` | Enedis conso-inf36 measured consumption (GCS) | Licence Ouverte 2.0 | polars `DataFrame` | building_eload — validation reference |
+| `get_elmas(table)` | ELMAS non-residential load curves (GCS) | **CC BY 4.0** | polars `DataFrame` | building_eload — non-residential model |
+| `get_occupant_diaries()` | synthetic occupant activity calendar (GCS) | maintainers' synthetic artifact; derived from a **restricted-access** survey | polars `DataFrame` | building_eload — occupancy models |
+| `get_activity_library(table)` | activitymodel diary library from the INSEE time-use survey (GCS) | maintainers' derived artifact, no survey identifier; derived from a **restricted-access** survey | polars `DataFrame` | building_eload — occupant calendars via activitymodel (planned) |
+| `get_elecdom()` | Enedis panel Elecdom end-use load curves (GCS) | Licence Ouverte 2.0 | polars `DataFrame` | provenance only — source of building_eload constants |
+| `get_bdtopo(iris_code)` | IGN Géoplateforme WFS (live) or bulk cache | Licence Ouverte 2.0 | geopandas `GeoDataFrame` | building_eload → buildingmodel — input geometry |
+| `get_era5_climate(lat, lon, year)` | Copernicus CDS (live) or bulk cache | **CC BY 4.0** (CDS account required) | path to EPW file (hour axis **UTC** by default, see `timezone=`) | buildingmodel — weather boundary condition |
+
+The MIT `LICENSE` covers the **code only**. Each dataset keeps its upstream
+licence, and every derived blob inherits the licence of the source it came
+from — there is no separate "buildingdata data licence". Two rows are not
+Licence Ouverte, and both matter for a publication: ELMAS is CC BY 4.0 (cite the
+paper), and ERA5 is CC BY 4.0 behind a personal Copernicus account, so an
+ERA5-dependent workflow is not anonymously reproducible. The occupant diaries
+and the activity library derive from the INSEE *Enquête Emploi du temps
+2009-2010*, which is obtained on request and is not redistributed here. The *Datasets* documentation
+page carries the per-dataset attribution string to reproduce, the exact upstream
+vintage, and the verification date.
 
 Reference datasets are pulled from a public Google Cloud Storage bucket and
 cached locally with generation-based freshness checks. French geospatial data
@@ -126,6 +142,7 @@ Because `~/.config/buildingdata/config.ini` is stored in your home directory, co
 ```bash
 pip install "buildingdata[era5]"   # ERA5 weather (cdsapi, xarray, pvlib, ...)
 pip install "buildingdata[bulk]"   # BDTOPO bulk prefetch (py7zr)
+pip install "buildingdata[timezone]"  # ERA5 timezone="local_standard" (timezonefinder)
 pip install "buildingdata[docs]"   # build the Sphinx documentation
 ```
 

{buildingdata-0.7.0 → buildingdata-0.9.0}/buildingdata/__init__.py

@@ -1,7 +1,9 @@
 # -*- coding: utf-8 -*-
 from .config import write_config
 from .reference import (
+    get_activity_library,
     get_census,
+    get_census_individuals,
     get_diagnosis,
     get_districts,
     get_elecdom,

{buildingdata-0.7.0 → buildingdata-0.9.0}/buildingdata/reference/__init__.py

@@ -1,5 +1,7 @@
 # -*- coding: utf-8 -*-
+from .activity_library import get_activity_library
 from .census import get_census
+from .census_individuals import get_census_individuals
 from .diagnosis import get_diagnosis
 from .districts import get_districts
 from .elecdom import get_elecdom

buildingdata-0.9.0/buildingdata/reference/activity_library.py

@@ -0,0 +1,241 @@
+# -*- coding: utf-8 -*-
+import json
+
+import polars as pl
+
+from ..exceptions import SchemaValidationError
+from ..gcs import download_blob, ensure_blob_cached, get_blob
+from ..validation import require_columns, require_dtype
+
+# The activity diary library built by ``activitymodel`` from the INSEE Enquete
+# Emploi du temps 2009-2010 ships as four tables, one parquet blob each (the
+# ELMAS pattern). Consumers assemble them with
+# ``activitymodel.DiaryLibrary.from_frames``.
+_BLOB_NAMES = {
+    "days": "activity_days_latest.parquet",
+    "clusters": "activity_clusters_latest.parquet",
+    "patterns": "activity_patterns_latest.parquet",
+    "manifest": "activity_manifest_latest.parquet",
+}
+
+_DATASET = "activity diary library"
+
+# Per-table contract, transcribed from activitymodel 0.1's ``library`` module
+# (DAY_META_SCHEMA, CLUSTER_SCHEMA, PATTERN_VALUE_SCHEMA and the manifest
+# key/value frame). Transcribed rather than imported because activitymodel is
+# optional here. The library schema is activitymodel's API: a change bumps its
+# minor version and must be mirrored here in the same release cycle.
+_SCHEMAS = {
+    "days": {
+        "day_id": pl.UInt32,
+        "day_type": pl.String,
+        "cluster_id": pl.Int16,
+        "weight": pl.Float64,
+    },
+    "clusters": {
+        "cluster_id": pl.Int16,
+        "day_type": pl.String,
+        "cluster_index": pl.Int16,
+        "n_days": pl.UInt32,
+        "weight_sum": pl.Float64,
+    },
+    "patterns": {"weight": pl.Float64, "n_persons": pl.UInt32},
+    "manifest": {"key": pl.String, "value": pl.String},
+}
+
+# One detailed activity code per 10-minute slot from 00:00, Europe/Paris wall
+# clock. building_eload hardcodes 144 steps per day.
+_SLOT_COLUMNS = tuple(f"slot_{i:03d}" for i in range(144))
+
+# ``patterns`` links each socio-demographic cell to one cluster per day type,
+# in columns ``cluster_<day_type>``.
+_PATTERN_CLUSTER_PREFIX = "cluster_"
+
+
+def _table_dataset(table):
+    return f"{_DATASET} ({table!r} table)"
+
+
+def _check_days(df, dataset):
+    slots = {c for c in df.columns if c.startswith("slot_")}
+    missing = [c for c in _SLOT_COLUMNS if c not in slots]
+    unexpected = sorted(slots - set(_SLOT_COLUMNS))
+    if missing or unexpected:
+        raise SchemaValidationError(
+            f"{dataset}: expected the 144 slot columns slot_000..slot_143, found "
+            f"{len(slots)}; missing {missing[:5]}, unexpected {unexpected[:5]}. "
+            f"building_eload hardcodes 144 ten-minute steps per day, so a day of "
+            f"any other length would be silently misplaced on the timeline."
+        )
+    require_dtype(df, _SLOT_COLUMNS, pl.UInt8, dataset)
+
+    # The contract lists every column of ``days`` so that it provably holds no
+    # survey microdata: a survey identifier or a socio-demographic attribute
+    # next to the activity sequence would re-attach the day to a person.
+    extra = [c for c in df.columns if c not in _SCHEMAS["days"] and c not in slots]
+    if extra:
+        raise SchemaValidationError(
+            f"{dataset}: column(s) {extra[:10]} are not part of the contract. The "
+            f"published pool is activity sequences detached from any person or "
+            f"household; an identifier or socio-demographic column must not be "
+            f"published, and no consumer may rely on one."
+        )
+
+
+def _check_patterns(df, dataset):
+    cluster_columns = [c for c in df.columns if c.startswith(_PATTERN_CLUSTER_PREFIX)]
+    if not cluster_columns:
+        raise SchemaValidationError(
+            f"{dataset}: no '{_PATTERN_CLUSTER_PREFIX}<day_type>' column found in "
+            f"{df.columns}; without them the patterns link people to no cluster."
+        )
+    require_dtype(df, cluster_columns, pl.Int16, dataset)
+
+
+def _check_manifest(df, dataset):
+    duplicated = df.filter(pl.col("key").is_duplicated())["key"].unique().sort().to_list()
+    if duplicated:
+        raise SchemaValidationError(
+            f"{dataset}: duplicated key(s) {duplicated}. The manifest is read into a "
+            f"dict, where the last duplicate would silently win."
+        )
+    for key, value in df.select("key", "value").iter_rows():
+        try:
+            json.loads(value)
+        except (TypeError, ValueError):
+            raise SchemaValidationError(
+                f"{dataset}: the value of {key!r} is not JSON-encoded: {value!r:.80}."
+            ) from None
+
+
+def _check_table(table, df):
+    """Assert one table carries its documented columns and dtypes."""
+    dataset = _table_dataset(table)
+    schema = _SCHEMAS[table]
+    require_columns(df, schema, dataset)
+    for column, dtype in schema.items():
+        require_dtype(df, (column,), dtype, dataset)
+    if table == "days":
+        _check_days(df, dataset)
+    elif table == "patterns":
+        _check_patterns(df, dataset)
+    elif table == "manifest":
+        _check_manifest(df, dataset)
+
+
+def _load_activitymodel():
+    """Return ``(activitymodel, LibraryValidationError)``, or None when not installed."""
+    try:
+        import activitymodel
+        from activitymodel.exceptions import LibraryValidationError
+    except ModuleNotFoundError as exc:
+        # Only a missing activitymodel means "not installed": an installed one
+        # whose own import fails is broken and must say so.
+        if exc.name != "activitymodel":
+            raise
+        return None
+    return activitymodel, LibraryValidationError
+
+
+def _read(table, refresh):
+    dest = ensure_blob_cached(
+        _BLOB_NAMES[table], refresh=refresh, get_blob_fn=get_blob, download_blob_fn=download_blob
+    )
+    return pl.read_parquet(dest)
+
+
+def _validate(table, df, refresh):
+    """Check ``table``; with activitymodel installed, check the whole library too."""
+    _check_table(table, df)
+
+    loaded = _load_activitymodel()
+    if loaded is None:
+        return df
+    activitymodel, library_validation_error = loaded
+
+    # DiaryLibrary.validate() is cross-table (a day's cluster must exist and
+    # share its day type, the manifest taxonomy must match the installed
+    # activitymodel's), so the sibling tables are read too.
+    frames = {name: df if name == table else _read(name, refresh) for name in _BLOB_NAMES}
+    for name, frame in frames.items():
+        if name != table:
+            _check_table(name, frame)
+    try:
+        activitymodel.DiaryLibrary.from_frames(**frames, validate=True)
+    except library_validation_error as exc:
+        raise SchemaValidationError(
+            f"{_DATASET}: the four tables fail activitymodel "
+            f"{activitymodel.__version__}'s DiaryLibrary.validate(): {exc}. If the "
+            f"blobs were restaged, call again with refresh=True so every table is "
+            f"re-read at its live generation; otherwise the library and the "
+            f"installed activitymodel disagree on the contract."
+        ) from exc
+    return df
+
+
+def get_activity_library(table="days", refresh=False):
+    """Return one table of the activity diary library (polars).
+
+    The library is built by ``activitymodel`` from the INSEE *Enquête Emploi du
+    temps 2009-2010* (PROGEDO-ADISP) and holds no survey microdata: ``days``
+    are activity sequences detached from any person, ``patterns`` are weighted
+    counts over socio-demographic cells. Each table is its own GCS blob;
+    ``building_eload`` assembles all four with::
+
+        from activitymodel import DiaryLibrary
+        from activitymodel.library import TABLE_NAMES
+
+        library = DiaryLibrary.from_frames(
+            **{name: get_activity_library(name) for name in TABLE_NAMES}
+        )
+
+    * ``"days"`` (default) -> ``activity_days_latest.parquet``
+      The pool of survey days: ``day_id`` (``UInt32``), ``day_type``
+      (``weekday`` / ``saturday`` / ``sunday``), ``cluster_id`` (``Int16``),
+      ``weight`` (``Float64``, survey diary weight) and ``slot_000`` ..
+      ``slot_143`` (``UInt8``), the detailed activity code of each 10-minute
+      slot from 00:00, Europe/Paris wall clock. Codes index the manifest's
+      ``activities``. No other column.
+    * ``"clusters"`` -> ``activity_clusters_latest.parquet``
+      ``cluster_id``, ``day_type``, ``cluster_index`` (``Int16``, 0..k-1 by
+      decreasing size within the day type), ``n_days`` (``UInt32``),
+      ``weight_sum`` (``Float64``).
+    * ``"patterns"`` -> ``activity_patterns_latest.parquet``
+      One ``String`` column per matching key (``sex``, ``age_class``, ...),
+      one ``Int16`` ``cluster_<day_type>`` column per day type (null when the
+      cell kept no diary of that day type), ``weight`` (``Float64``, summed
+      person weight) and ``n_persons`` (``UInt32``).
+    * ``"manifest"`` -> ``activity_manifest_latest.parquet``
+      ``key`` / ``value`` strings, values JSON-encoded: taxonomy, day types,
+      matching keys, clustering parameters, provenance.
+
+    Every call checks the requested table's columns and dtypes, including the
+    exact 144 ``UInt8`` slot columns and no column beyond the contract in
+    ``days``. When ``activitymodel`` is installed, the other three tables are
+    read as well and the four are handed to ``DiaryLibrary.validate()``, which
+    adds the cross-table and value checks (cluster references, day types,
+    code range, weights, taxonomy of the installed version).
+
+    Args:
+        table (str): which table to return, one of ``"days"``, ``"clusters"``,
+            ``"patterns"``, ``"manifest"``. Defaults to ``"days"``.
+        refresh (bool): force re-download even if the cache is warm. Also
+            applies to the sibling tables read for the cross-table check, so a
+            refreshed table is never checked against a stale one. Defaults to
+            False.
+
+    Returns:
+        polars.DataFrame: the requested table.
+
+    Raises:
+        ValueError: if ``table`` is not a library table.
+        RemoteNotAvailableError: if the blob (or, with activitymodel
+            installed, a sibling blob) is not found in the GCS bucket.
+        SchemaValidationError: if the table, or the library as a whole,
+            breaks the contract.
+    """
+    if table not in _BLOB_NAMES:
+        raise ValueError(
+            f"Unknown activity library table {table!r}; expected one of {sorted(_BLOB_NAMES)}."
+        )
+    return _validate(table, _read(table, refresh), refresh)

buildingdata-0.9.0/buildingdata/reference/census_individuals.py

@@ -0,0 +1,118 @@
+# -*- coding: utf-8 -*-
+import polars as pl
+
+from ..gcs import download_blob, ensure_blob_cached, get_blob
+from ..validation import require_columns, require_dtype
+
+_BLOB_NAME = "census_individuals_latest.parquet"
+
+_DATASET = "INSEE census individuals"
+
+# The published schema, mirrored from the pipeline's
+# ``process_census_individuals.py::OUTPUT_SCHEMA``. The person columns are
+# activitymodel's harmonised vocabulary; the dwelling columns use
+# census_latest's labels, so buildingmodel can place a household on a dwelling
+# by joining on them. Every categorical column is a plain ``String``.
+_SCHEMA = {
+    # geography: district/city are null for people the census locates only to
+    # their canton-ou-ville; city_group is null when that canton spans several
+    "district": pl.String,
+    "city": pl.String,
+    "canton_ou_ville": pl.String,
+    "city_group": pl.String,
+    "city_group_candidates": pl.List(pl.String),
+    "department": pl.String,
+    # household
+    "household_key": pl.String,
+    "household_position": pl.String,
+    "ipondi": pl.Float64,
+    # person (activitymodel vocabulary)
+    "sex": pl.String,
+    "age": pl.Int64,
+    "age_class": pl.String,
+    "activity_status": pl.String,
+    "csp_group": pl.String,
+    "employment_status": pl.String,
+    "couple_status": pl.String,
+    "household_size": pl.Int64,
+    "household_size_class": pl.String,
+    # dwelling (census_latest vocabulary)
+    "residential_type": pl.String,
+    "construction_year_class_simplified": pl.String,
+    "heating_system": pl.String,
+    "occupancy_type": pl.String,
+    "occupant_count": pl.Int32,
+    "living_area_class": pl.String,
+}
+
+
+def get_census_individuals(city_group_list=None, refresh=False):
+    """Return the INSEE census individuals (polars), optionally pre-filtered.
+
+    One row per person of an ordinary household living in a house or an
+    apartment in metropolitan France, from the RP2022 *Individus localisés au
+    canton-ou-ville* file (~16.5 M rows nationally), harmonised to the
+    activitymodel vocabulary. It is the input of
+    buildingmodel's occupant inference, which draws one census household per
+    dwelling. Downloaded from ``census_individuals_latest.parquet`` on GCS on
+    first call. Filtering uses a polars lazy scan, so only the requested rows
+    are materialised.
+
+    Columns:
+
+    * geography: ``district`` (IRIS), ``city``, ``canton_ou_ville`` (INSEE
+      pseudo-canton), ``city_group`` (EPCI, in ``get_districts``'s coding),
+      ``city_group_candidates`` (list) and ``department``. The census locates a
+      person to the IRIS only in IRIS-split communes. Everyone else (41.6 % of
+      the weighted population) has a null ``district``/``city``, and
+      ``city_group_candidates`` lists the city groups of their canton's
+      unsplit communes; ``city_group`` is set only when there is exactly one
+      (null for 27.4 % of the population).
+    * household: ``household_key`` (canton-ou-ville + INSEE household number),
+      ``household_position`` (``reference_person``, ``partner``, ``child``,
+      ``other``), ``ipondi`` (the INSEE person weight).
+    * person, activitymodel vocabulary: ``sex``, ``age``, ``age_class``,
+      ``activity_status``, ``csp_group`` (null outside employment),
+      ``employment_status``, ``couple_status``, ``household_size``,
+      ``household_size_class``.
+    * dwelling, census_latest vocabulary: ``residential_type``,
+      ``construction_year_class_simplified`` (the census file only has the
+      regrouped construction period, whose 2006+ classes do not split at 2012
+      like ``construction_year_class``), ``heating_system``,
+      ``occupancy_type`` (always ``primary residence``), ``occupant_count``
+      (equal to ``household_size``) and ``living_area_class``.
+
+    Args:
+        city_group_list (list of str, optional): EPCI codes to filter on. A
+            person is kept when any of their ``city_group_candidates`` is in the
+            list: someone located only to a canton that straddles two EPCIs is
+            returned for both. Filter on ``district`` being non-null to keep
+            IRIS-located people only. If None, the full national table is
+            returned. Defaults to None.
+        refresh (bool): force re-download even if the cache is warm.
+            Defaults to False.
+
+    Returns:
+        polars.DataFrame: one row per person, columns as listed above.
+
+    Raises:
+        RemoteNotAvailableError: if the blob is not found in the GCS bucket.
+        SchemaValidationError: if the blob lacks a documented column or one
+            has another dtype.
+    """
+    dest = ensure_blob_cached(
+        _BLOB_NAME, refresh=refresh, get_blob_fn=get_blob, download_blob_fn=download_blob
+    )
+
+    lf = pl.scan_parquet(dest)
+    schema = lf.collect_schema()
+    require_columns(schema, _SCHEMA, _DATASET)
+    for column, dtype in _SCHEMA.items():
+        require_dtype(schema, (column,), dtype, _DATASET)
+
+    if city_group_list is not None:
+        codes = [str(code) for code in city_group_list]
+        lf = lf.filter(
+            pl.col("city_group_candidates").list.eval(pl.element().is_in(codes)).list.any()
+        )
+    return lf.collect()