gla-data 0.2.1__tar.gz → 0.3.0__tar.gz

gla_data-0.3.0/PKG-INFO

@@ -0,0 +1,165 @@
+Metadata-Version: 2.5
+Name: gla-data
+Version: 0.3.0
+Summary: Shared tools for loading, transforming, and visualising the GLA datasets used in City Data analysis.
+Author-email: Conor Dempsey <conor.dempsey@london.gov.uk>
+License: MIT
+Requires-Python: >=3.12
+Requires-Dist: beautifulsoup4>=4.12
+Requires-Dist: boto3>=1.35
+Requires-Dist: botocore>=1.35
+Requires-Dist: filelock>=3.12
+Requires-Dist: geopandas>=1.0
+Requires-Dist: numpy>=1.26
+Requires-Dist: pandas>=2.2
+Requires-Dist: pandera>=0.32.1
+Requires-Dist: pyarrow>=24.0.0
+Requires-Dist: pydantic-settings
+Requires-Dist: pydantic>=2.0
+Requires-Dist: python-dotenv
+Requires-Dist: requests>=2.32
+Requires-Dist: rich>=13.0
+Requires-Dist: shapely>=2.0
+Requires-Dist: sqlalchemy>=2.0
+Requires-Dist: typer>=0.26.8
+Description-Content-Type: text/markdown
+
+# gla-data
+
+Shared tools for loading, transforming, and visualising the GLA datasets used in City Data analysis.
+
+## Installation
+
+### CLI
+
+To install `gla-data` globally so the `gla-data` CLI is available anywhere in your terminal:
+
+```bash
+uv tool install gla-data
+```
+
+This requires [uv](https://docs.astral.sh/uv/) to be installed.
+
+### pip
+
+```bash
+pip install gla-data
+```
+
+### pixi
+
+```bash
+pixi add gla-data
+```
+
+Pixi pulls geospatial dependencies (geopandas, pyproj, etc.) from conda-forge, which provides reliable pre-built binaries across all platforms and Python versions.
+
+## Usage
+
+### Python API
+
+```python
+import gla_data
+
+# Load 2021 LSOA boundaries for London
+lsoa_gdf = gla_data.load_boundaries(geography='lsoa', year=2021)
+
+# Load census dwelling counts
+dwellings_2021 = gla_data.load_census_dwellings(year=2021)
+dwellings_2011 = gla_data.load_census_dwellings(year=2011)
+
+# Crosswalk 2011 dwelling counts onto 2021 LSOA boundaries
+dwellings_2011_xw = gla_data.crosswalk(
+    dwellings_2011,
+    from_year=2011,
+    to_year=2021,
+    value_cols=['dwellings_total'],
+)
+
+# Load an OA → LSOA geography lookup
+lookup = gla_data.load_geography_lookup(year=2021, smallest_geography='oa')
+
+# Aggregate OA-level data up to LSOA level
+lsoa_totals = gla_data.aggregate(
+    df,
+    from_geography='oa',
+    to_geography='lsoa',
+    value_cols=['count'],
+    year=2021,
+)
+
+# OS MasterMap buildings, with heights, in one 1km grid square plus a 100m
+# margin (GLA only; needs the database settings below)
+buildings = gla_data.load_mastermap_topography(
+    bbox=(530000, 180000, 531000, 181000),
+    buffer_m=100,
+    descriptive_group='Building',
+)
+
+# GLA tree canopy in the same square plus a 100m margin. The first call
+# downloads the data (about 270MB) and builds a local file from it, which
+# takes several minutes; later calls read that file.
+canopy = gla_data.load_tree_canopy(
+    bbox=(530000, 180000, 531000, 181000),
+    buffer_m=100,
+)
+```
+
+### CLI
+
+The package installs a `gla-data` command. Run `catalogue` to list all available datasets:
+
+```bash
+gla-data catalogue
+```
+
+This prints a table of every dataset, the function used to load it, a description, and its access tier (`public`, `registered`, `licensed` or `internal`; see Configuration for the last three).
+
+## Configuration
+
+Public datasets need no setup. Datasets held inside the GLA (the GIS team's Postgres database and S3 buckets) need settings that are not included in this package: connection details, table names and bucket names. GLA staff can get the values from the City Intelligence Data Science team.
+
+### Where to put settings
+
+Every setting is named `GLA_DATA_<NAME>` and can be set in any of these places. Higher in the list wins:
+
+1. An environment variable, e.g. `GLA_DATA_BLUESKY_IMAGERY_BUCKET`.
+2. A `.env` file in the folder you run your code from (a project's own settings).
+3. Your user-level settings file, `~/.config/gla-data/.env` (on Windows, `C:\Users\<you>\.config\gla-data\.env`). Use this for settings shared by all your projects, such as database connections and bucket names. Set `GLA_DATA_USER_ENV_FILE` to use a different path.
+
+A settings file holds one setting per line:
+
+```
+GLA_DATA_BLUESKY_IMAGERY_BUCKET=<bucket name>
+```
+
+A misspelt `GLA_DATA_` setting in a settings file raises an error rather than being ignored. Other keys in a project's `.env` are left alone.
+
+### Settings for GLA-internal datasets
+
+| Setting | Needed for |
+|---|---|
+| `GLA_DATA_GIS_DB_HOST`, `_PORT`, `_NAME`, `_USER`, `_PASSWORD` | Anything from the GIS team's Postgres database (`source="postgres"`, `load_gis_layer`, `load_front_gardens`, `load_mastermap_topography`, `gla-data db ...`). Use a read-only login. |
+| `GLA_DATA_GIS_DB_SCHEMA` | Optional: the AddressBase schema, if not the default search path. |
+| `GLA_DATA_ADDRESSBASE_BLPU_TABLE`, `_CLASSIFICATION_TABLE` | `load_addressbase(source="postgres")`, if the tables are not named `blpu` / `classification`. |
+| `GLA_DATA_GISAPDATA_<LAYER>_TABLE` | Each `gisapdata` layer used by `load_boundaries(source="postgres")`, `load_gis_layer` and `load_front_gardens`. |
+| `GLA_DATA_GISAPDATA_MASTERMAP_TOPO_2024_TABLE` | `load_mastermap_topography` (2024 snapshot). |
+| `GLA_DATA_BLUESKY_IMAGERY_BUCKET` | Bluesky aerial imagery (`build_imagery_tile_index`, `download_imagery_tiles`). |
+| `GLA_DATA_BLUESKY_IMAGERY_DIR` | Optional: folder for the imagery tile index and downloads (default `data/bluesky_imagery`). Tiles are 256MB each. |
+| `GLA_DATA_AWS_PROFILE`, `GLA_DATA_AWS_REGION` | Optional: pin the AWS profile or region used for S3. |
+| `GLA_DATA_TREE_CANOPY_DIR` | Optional: folder for the tree canopy download and the GeoPackage built from it (default `data/tree_canopy`). Point every project at the same folder so the data is downloaded once. |
+
+AWS credentials are never set here. gla-data uses the standard AWS sources: an AWS profile (`~/.aws`), `AWS_*` environment variables, or the machine's own role on GLA cloud machines. Keep database passwords in your user-level file, never in a project's repository.
+
+## Development
+
+```bash
+git clone <repo-url>
+cd gla-data
+pixi install
+
+pixi run test       # run tests
+pixi run lint       # check code style
+pixi run format     # auto-format
+pixi run typecheck  # type checking
+```

gla_data-0.3.0/README.md

@@ -0,0 +1,139 @@
+# gla-data
+
+Shared tools for loading, transforming, and visualising the GLA datasets used in City Data analysis.
+
+## Installation
+
+### CLI
+
+To install `gla-data` globally so the `gla-data` CLI is available anywhere in your terminal:
+
+```bash
+uv tool install gla-data
+```
+
+This requires [uv](https://docs.astral.sh/uv/) to be installed.
+
+### pip
+
+```bash
+pip install gla-data
+```
+
+### pixi
+
+```bash
+pixi add gla-data
+```
+
+Pixi pulls geospatial dependencies (geopandas, pyproj, etc.) from conda-forge, which provides reliable pre-built binaries across all platforms and Python versions.
+
+## Usage
+
+### Python API
+
+```python
+import gla_data
+
+# Load 2021 LSOA boundaries for London
+lsoa_gdf = gla_data.load_boundaries(geography='lsoa', year=2021)
+
+# Load census dwelling counts
+dwellings_2021 = gla_data.load_census_dwellings(year=2021)
+dwellings_2011 = gla_data.load_census_dwellings(year=2011)
+
+# Crosswalk 2011 dwelling counts onto 2021 LSOA boundaries
+dwellings_2011_xw = gla_data.crosswalk(
+    dwellings_2011,
+    from_year=2011,
+    to_year=2021,
+    value_cols=['dwellings_total'],
+)
+
+# Load an OA → LSOA geography lookup
+lookup = gla_data.load_geography_lookup(year=2021, smallest_geography='oa')
+
+# Aggregate OA-level data up to LSOA level
+lsoa_totals = gla_data.aggregate(
+    df,
+    from_geography='oa',
+    to_geography='lsoa',
+    value_cols=['count'],
+    year=2021,
+)
+
+# OS MasterMap buildings, with heights, in one 1km grid square plus a 100m
+# margin (GLA only; needs the database settings below)
+buildings = gla_data.load_mastermap_topography(
+    bbox=(530000, 180000, 531000, 181000),
+    buffer_m=100,
+    descriptive_group='Building',
+)
+
+# GLA tree canopy in the same square plus a 100m margin. The first call
+# downloads the data (about 270MB) and builds a local file from it, which
+# takes several minutes; later calls read that file.
+canopy = gla_data.load_tree_canopy(
+    bbox=(530000, 180000, 531000, 181000),
+    buffer_m=100,
+)
+```
+
+### CLI
+
+The package installs a `gla-data` command. Run `catalogue` to list all available datasets:
+
+```bash
+gla-data catalogue
+```
+
+This prints a table of every dataset, the function used to load it, a description, and its access tier (`public`, `registered`, `licensed` or `internal`; see Configuration for the last three).
+
+## Configuration
+
+Public datasets need no setup. Datasets held inside the GLA (the GIS team's Postgres database and S3 buckets) need settings that are not included in this package: connection details, table names and bucket names. GLA staff can get the values from the City Intelligence Data Science team.
+
+### Where to put settings
+
+Every setting is named `GLA_DATA_<NAME>` and can be set in any of these places. Higher in the list wins:
+
+1. An environment variable, e.g. `GLA_DATA_BLUESKY_IMAGERY_BUCKET`.
+2. A `.env` file in the folder you run your code from (a project's own settings).
+3. Your user-level settings file, `~/.config/gla-data/.env` (on Windows, `C:\Users\<you>\.config\gla-data\.env`). Use this for settings shared by all your projects, such as database connections and bucket names. Set `GLA_DATA_USER_ENV_FILE` to use a different path.
+
+A settings file holds one setting per line:
+
+```
+GLA_DATA_BLUESKY_IMAGERY_BUCKET=<bucket name>
+```
+
+A misspelt `GLA_DATA_` setting in a settings file raises an error rather than being ignored. Other keys in a project's `.env` are left alone.
+
+### Settings for GLA-internal datasets
+
+| Setting | Needed for |
+|---|---|
+| `GLA_DATA_GIS_DB_HOST`, `_PORT`, `_NAME`, `_USER`, `_PASSWORD` | Anything from the GIS team's Postgres database (`source="postgres"`, `load_gis_layer`, `load_front_gardens`, `load_mastermap_topography`, `gla-data db ...`). Use a read-only login. |
+| `GLA_DATA_GIS_DB_SCHEMA` | Optional: the AddressBase schema, if not the default search path. |
+| `GLA_DATA_ADDRESSBASE_BLPU_TABLE`, `_CLASSIFICATION_TABLE` | `load_addressbase(source="postgres")`, if the tables are not named `blpu` / `classification`. |
+| `GLA_DATA_GISAPDATA_<LAYER>_TABLE` | Each `gisapdata` layer used by `load_boundaries(source="postgres")`, `load_gis_layer` and `load_front_gardens`. |
+| `GLA_DATA_GISAPDATA_MASTERMAP_TOPO_2024_TABLE` | `load_mastermap_topography` (2024 snapshot). |
+| `GLA_DATA_BLUESKY_IMAGERY_BUCKET` | Bluesky aerial imagery (`build_imagery_tile_index`, `download_imagery_tiles`). |
+| `GLA_DATA_BLUESKY_IMAGERY_DIR` | Optional: folder for the imagery tile index and downloads (default `data/bluesky_imagery`). Tiles are 256MB each. |
+| `GLA_DATA_AWS_PROFILE`, `GLA_DATA_AWS_REGION` | Optional: pin the AWS profile or region used for S3. |
+| `GLA_DATA_TREE_CANOPY_DIR` | Optional: folder for the tree canopy download and the GeoPackage built from it (default `data/tree_canopy`). Point every project at the same folder so the data is downloaded once. |
+
+AWS credentials are never set here. gla-data uses the standard AWS sources: an AWS profile (`~/.aws`), `AWS_*` environment variables, or the machine's own role on GLA cloud machines. Keep database passwords in your user-level file, never in a project's repository.
+
+## Development
+
+```bash
+git clone <repo-url>
+cd gla-data
+pixi install
+
+pixi run test       # run tests
+pixi run lint       # check code style
+pixi run format     # auto-format
+pixi run typecheck  # type checking
+```

{gla_data-0.2.1 → gla_data-0.3.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "gla-data"
-version = "0.2.1"
+version = "0.3.0"
 description = "Shared tools for loading, transforming, and visualising the GLA datasets used in City Data analysis."
 readme = "README.md"
 license = { text = "MIT" }
@@ -10,12 +10,20 @@ dependencies = [
     "pandas>=2.2",
     "geopandas>=1.0",
     "requests>=2.32",
+    "pydantic>=2.0",
     "pydantic-settings",
+    "python-dotenv",
+    "shapely>=2.0",
     "pyarrow>=24.0.0",
     "typer>=0.26.8",
+    "rich>=13.0",
     "pandera>=0.32.1",
     "sqlalchemy>=2.0",
     "beautifulsoup4>=4.12",
+    "boto3>=1.35",
+    "botocore>=1.35",
+    "filelock>=3.12",
+    "numpy>=1.26",
     # Postgres driver (psycopg2) is a conda-only dep here (see pixi.toml) to
     # avoid pip/conda installing it twice in the pixi env, matching how
     # openpyxl (needed by census.py) is handled — a plain `pip install
@@ -30,6 +38,16 @@ gla-data = "gla_data.cli:main"
 requires = ["hatchling"]
 build-backend = "hatchling.build"
 
+# The package is published on public PyPI while the repo is private, so the
+# source distribution lists exactly what ships (pyproject.toml is always
+# included). Everything else stays out: internal docs (DESIGN.md, docs/),
+# notebooks, CI workflows, .env.example, Claude files, lock files.
+[tool.hatch.build.targets.sdist]
+only-include = ["src", "tests", "README.md"]
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/gla_data"]
+
 [tool.pytest.ini_options]
 testpaths = ["tests"]
 addopts   = "-v --tb=short"
@@ -43,7 +61,22 @@ source = ["src"]
 
 
 [tool.ruff.lint]
-select = ["E", "F", "I", "UP", "B"]
+# T20: no print() in library code; BLE: no blind `except Exception`;
+# C90: cyclomatic complexity limit (see mccabe below); D: numpy-style
+# docstrings (this package's standard, decided 2026-10-19).
+select = ["E", "F", "I", "UP", "B", "T20", "BLE", "C90", "D"]
+
+[tool.ruff.lint.pydocstyle]
+convention = "numpy"
+
+[tool.ruff.lint.mccabe]
+max-complexity = 10
+
+[tool.ruff.lint.per-file-ignores]
+"tests/**" = ["T20", "D"]
+
+[tool.deptry]
+known_first_party = ["gla_data"]
 
 [tool.mypy]
 strict                 = true
@@ -56,7 +89,7 @@ disallow_untyped_defs = false
 disallow_incomplete_defs = false
 
 [tool.bumpversion]
-current_version = "0.2.1"
+current_version = "0.3.0"
 commit = true
 tag    = true
 

gla_data-0.3.0/src/gla_data/__init__.py

@@ -0,0 +1,60 @@
+"""Shared tools for loading and transforming GLA datasets.
+
+Every public function is importable from the top level, e.g.
+``gla_data.load_boundaries``; ``gla_data.catalogue()`` lists the datasets.
+"""
+
+from importlib.metadata import version
+
+from gla_data.addressbase import (
+    assign_area_code,
+    build_uprn_net_change_from_ngd,
+    compute_uprn_net_change,
+    load_addressbase,
+    load_ngd_addressbase,
+)
+from gla_data.aerial_imagery import (
+    build_imagery_tile_index,
+    download_imagery_tiles,
+    get_imagery_tile_path,
+    load_imagery_tile_index,
+    remove_imagery_tiles,
+)
+from gla_data.catalogue import catalogue
+from gla_data.census import load_census_dwellings
+from gla_data.explore import describe_table, list_schemas, list_tables
+from gla_data.front_gardens import load_front_gardens
+from gla_data.geography import load_boundaries, load_geography_lookup, load_gis_layer
+from gla_data.lbsm import load_lbsm
+from gla_data.mastermap import load_mastermap_topography
+from gla_data.transforms import aggregate, crosswalk
+from gla_data.tree_canopy import download_tree_canopy, load_tree_canopy
+
+__version__ = version("gla-data")
+__all__ = [
+    "load_boundaries",
+    "load_geography_lookup",
+    "load_gis_layer",
+    "load_census_dwellings",
+    "crosswalk",
+    "aggregate",
+    "catalogue",
+    "load_addressbase",
+    "compute_uprn_net_change",
+    "load_ngd_addressbase",
+    "assign_area_code",
+    "build_uprn_net_change_from_ngd",
+    "load_lbsm",
+    "list_schemas",
+    "list_tables",
+    "describe_table",
+    "load_front_gardens",
+    "build_imagery_tile_index",
+    "load_imagery_tile_index",
+    "download_imagery_tiles",
+    "get_imagery_tile_path",
+    "remove_imagery_tiles",
+    "load_mastermap_topography",
+    "load_tree_canopy",
+    "download_tree_canopy",
+]

gla_data-0.2.1/src/gla_data/addressbase.py → gla_data-0.3.0/src/gla_data/_addressbase.py

@@ -1,44 +1,31 @@
-"""OS AddressBase Premium: residential UPRN net change by area and financial year.
-
-AddressBase Premium is a ``licensed`` dataset (see DESIGN.md's data access tiers) —
-``load_addressbase`` is the raw loader, not yet implemented pending GLA's S3
-location for the extract. ``compute_uprn_net_change`` is a pure transform that
-works on any already-loaded BLPU-shaped DataFrame, so it is fully testable today
-against synthetic fixtures without AddressBase access itself, and composes with
-``load_addressbase``'s output once that lands.
-
-Ported from an exploratory notebook (external reference, not part of this
-package) that produced this same net-change-by-OA-by-financial-year table for
-the housing_projections project. That notebook's first approach computed a
-*cumulative* running UPRN count per area and diffed it year-over-year — wrong
-wherever the underlying data was processed in row-chunks rather than as one
-geographically complete set, since a chunk boundary could split one area's
-history across chunks. The event-based approach here (sum +1/-1 events within
-each financial year directly, no cumulative step) sidesteps that failure mode
-structurally: call ``compute_uprn_net_change`` once per geographic scope (e.g.
-once per borough, not on arbitrary row partitions of a larger file) and there is
-no cross-call state for a chunk boundary to corrupt.
-"""
+"""Shared AddressBase logic used by both routes (Premium in Postgres, NGD).
 
-from typing import Literal
+The residential classification rules, the financial-year convention and the
+net-change transform. Public names are re-exported from ``addressbase``.
+"""
 
-import geopandas as gpd
 import pandas as pd
 
-from gla_data import _postgres
 from gla_data._schemas import addressbase_net_change_schema
-from gla_data.config import settings
 
-# AddressBase Premium classification scheme: residential codes start with 'R'.
-# RC01 (residential car park) and RG* (garage) are excluded — not living space.
+# AddressBase classification scheme (shared by AddressBase Premium and NGD):
+# residential codes start with 'R'. RC01 (residential car park) and RG*
+# (garage) are excluded — not living space.
 _RESIDENTIAL_PREFIX = "R"
 _EXCLUDED_PREFIXES = ("RC01", "RG")
 
-# AddressBase Premium coverage is unreliable before this financial year in past
-# uses of this data (the source notebook dropped 2007/2008 for the same reason).
+# AddressBase coverage is unreliable before this financial year in past uses
+# of this data (the source notebook dropped 2007/2008 for the same reason).
 _MIN_FINANCIAL_YEAR = 2009
 
 
+def _is_residential_classification(code: pd.Series) -> pd.Series:
+    """Boolean mask: residential classification code, excluding car parks/garages."""
+    is_residential = code.str.startswith(_RESIDENTIAL_PREFIX)
+    is_excluded = code.str.startswith(_EXCLUDED_PREFIXES)
+    return is_residential & ~is_excluded
+
+
 def _filter_residential_uprns(classification: pd.DataFrame) -> pd.Series:
     """Return the deduplicated set of residential UPRNs.
 
@@ -59,9 +46,7 @@ def _filter_residential_uprns(classification: pd.DataFrame) -> pd.Series:
         with multiple qualifying classification rows.
     """
     code = classification["classification_code"]
-    is_residential = code.str.startswith(_RESIDENTIAL_PREFIX)
-    is_excluded = code.str.startswith(_EXCLUDED_PREFIXES)
-    residential = classification.loc[is_residential & ~is_excluded, "uprn"]
+    residential = classification.loc[_is_residential_classification(code), "uprn"]
     return residential.drop_duplicates()
 
 
@@ -176,69 +161,3 @@ def compute_uprn_net_change(
 
     addressbase_net_change_schema(area_col).validate(result)
     return result
-
-
-def load_addressbase(
-    source: Literal["postgres", "s3", "local"] = "postgres",
-    table: str | None = None,
-) -> gpd.GeoDataFrame | pd.DataFrame:
-    """Load raw AddressBase Premium records for London.
-
-    GLA's AddressBase Premium extract lives in Postgres/PostGIS, so
-    ``source="postgres"`` is implemented; ``"s3"`` and ``"local"`` are not,
-    yet.
-
-    **This currently returns the raw configured table as-is** — no column
-    renaming, residential filtering, or schema validation. The real
-    table/column names are still being confirmed against the live database
-    (see ``notebooks/addressbase_postgres_setup.ipynb``, which reflects the
-    schema and exports it for that purpose). Once confirmed, this will be
-    tightened to return a validated, BLPU-shaped GeoDataFrame ready to pass
-    directly into ``compute_uprn_net_change``.
-
-    Parameters
-    ----------
-    source:
-        ``"postgres"`` (implemented, default), ``"s3"`` or ``"local"``
-        (not yet implemented).
-    table:
-        Postgres table to read. Defaults to the configured
-        ``GLA_DATA_ADDRESSBASE_BLPU_TABLE`` (see ``.env.example``).
-        Only used when ``source="postgres"``.
-
-    Returns
-    -------
-    gpd.GeoDataFrame or pd.DataFrame
-        Whatever is in the configured table, unmodified. GeoDataFrame if a
-        geometry column is present, otherwise a plain DataFrame.
-
-    Raises
-    ------
-    NotImplementedError
-        For ``source="s3"`` or ``source="local"``.
-    EnvironmentError
-        For ``source="postgres"``, if required
-        ``GLA_DATA_GIS_DB_*`` settings are missing — see
-        ``.env.example``.
-    ValueError
-        For an unrecognised ``source``.
-
-    Access tier: ``licensed`` (OS AddressBase Premium via GLA's own licence).
-    """
-    if source == "postgres":
-        engine = _postgres.get_engine()
-        table = table or settings.addressbase_blpu_table
-        return _postgres.read_table(engine, table, schema=settings.gis_db_schema)
-    if source == "s3":
-        raise NotImplementedError(
-            "load_addressbase(source='s3') is not yet implemented — GLA's "
-            "AddressBase Premium extract currently lives in Postgres, not "
-            "S3. Use source='postgres' instead."
-        )
-    if source == "local":
-        raise NotImplementedError(
-            "load_addressbase(source='local') is not yet implemented."
-        )
-    raise ValueError(
-        f"source={source!r} is not supported. Choose from: 'postgres', 's3', 'local'."
-    )