bcdata 0.16.0__tar.gz → 0.17.0__tar.gz

{bcdata-0.16.0 → bcdata-0.17.0}/PKG-INFO

@@ -1,27 +1,26 @@
 Metadata-Version: 2.4
 Name: bcdata
-Version: 0.16.0
+Version: 0.17.0
 Summary: Download open data, monitor and report on changes
 Author-email: Simon Norris <snorris@hillcrestgeo.ca>
 Project-URL: Homepage, https://github.com/smnorris/bcdata
 Project-URL: Issues, https://github.com/smnorris/bcdata
 Classifier: Development Status :: 4 - Beta
-Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
-Classifier: License :: OSI Approved :: MIT License
+Classifier: License :: OSI Approved :: Apache Software License
 Classifier: Topic :: Scientific/Engineering :: GIS
 Classifier: Operating System :: OS Independent
-Requires-Python: >=3.9
+Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE.txt
+Requires-Dist: click
 Requires-Dist: geoalchemy2
 Requires-Dist: geopandas
 Requires-Dist: owslib
 Requires-Dist: psycopg2-binary
-Requires-Dist: rasterio
 Requires-Dist: requests
 Requires-Dist: sqlalchemy
 Requires-Dist: stamina
@@ -42,7 +41,7 @@ There is a [wealth of British Columbia geographic information available as open
 data](https://catalogue.data.gov.bc.ca/dataset?download_audience=Public),
 but most sources are available only via WFS - and the syntax to download WFS data via `ogr2ogr` and/or `curl/wget` can be awkward.
 
-This tool attempts to simplify downloads of BC geographic data and smoothly integrate with PostGIS and Python GIS tools like `geopandas`, `fiona` and `rasterio`. The tool only accesses geographic data [avialable via WFS](https://bcgov.github.io/data-publication/pages/tips_tricks_webservices.html) - for other BC open data, download the files directly with `requests` / `curl` / `ogr2ogr` / etc (or the [bcdata R package](https://github.com/bcgov/bcdata)).
+This tool attempts to simplify downloads of BC geographic data and smoothly integrate with PostGIS and Python GIS tools like `geopandas` and `fiona`. The tool only accesses geographic data [avialable via WFS](https://bcgov.github.io/data-publication/pages/tips_tricks_webservices.html) - for other BC open data, download the files directly with `requests` / `curl` / `ogr2ogr` / etc (or the [bcdata R package](https://github.com/bcgov/bcdata)).
 
 If downloads are failing, consult the [BCGov Data Systems and Services uptime monitor](https://uptime.com/statuspage/bcgov-dss) - this tool depends on BC Open Geospatial Web Services API at [openmaps.gov.bc.ca](https://openmaps.gov.bc.ca/geo/pub/ows?service=WFS&request=Getcapabilities) and the [BC Data Catalogue API](https://catalogue.data.gov.bc.ca/dataset/bc-data-catalogue-api). 
 
@@ -359,19 +358,20 @@ Use some other tool to reproject the data as required.
 
 ## Development and testing
 
-`bc2pg` tests require database `postgresql://postgres@localhost:5432/test_bcdata` exists and has PostGIS installed:
-
-    psql -c "create database test_bcdata"
-    psql test_bcdata -c "create extension postgis"
-
-Create virtualenv and install `bcdata` in development mode:
+Clone the repo:
 
     $ git clone git@github.com:bcgov/bcdata_py.git
     $ cd bcdata_py
-    $ python -m venv .venv
-    $ source .venv/bin/activate
-    (.venv)$ pip install -e .[test]
-    (.venv)$ py.test
+
+`bc2pg` tests require a PostGIS database, referenced via the `DATABASE_URL` environment variable (matching the image used in CI, see `.github/workflows/tests.yml`). Start one locally with `docker compose`:
+
+    $ docker compose up -d
+    $ export DATABASE_URL=postgresql://postgres:postgres@localhost:5432/postgres
+
+install `bcdata` in development mode with [`uv`](https://docs.astral.sh/uv/):
+
+    $ uv sync --extra test
+    $ uv run pytest
 
 
 ## Other implementations

{bcdata-0.16.0 → bcdata-0.17.0}/README.md

@@ -9,7 +9,7 @@ There is a [wealth of British Columbia geographic information available as open
 data](https://catalogue.data.gov.bc.ca/dataset?download_audience=Public),
 but most sources are available only via WFS - and the syntax to download WFS data via `ogr2ogr` and/or `curl/wget` can be awkward.
 
-This tool attempts to simplify downloads of BC geographic data and smoothly integrate with PostGIS and Python GIS tools like `geopandas`, `fiona` and `rasterio`. The tool only accesses geographic data [avialable via WFS](https://bcgov.github.io/data-publication/pages/tips_tricks_webservices.html) - for other BC open data, download the files directly with `requests` / `curl` / `ogr2ogr` / etc (or the [bcdata R package](https://github.com/bcgov/bcdata)).
+This tool attempts to simplify downloads of BC geographic data and smoothly integrate with PostGIS and Python GIS tools like `geopandas` and `fiona`. The tool only accesses geographic data [avialable via WFS](https://bcgov.github.io/data-publication/pages/tips_tricks_webservices.html) - for other BC open data, download the files directly with `requests` / `curl` / `ogr2ogr` / etc (or the [bcdata R package](https://github.com/bcgov/bcdata)).
 
 If downloads are failing, consult the [BCGov Data Systems and Services uptime monitor](https://uptime.com/statuspage/bcgov-dss) - this tool depends on BC Open Geospatial Web Services API at [openmaps.gov.bc.ca](https://openmaps.gov.bc.ca/geo/pub/ows?service=WFS&request=Getcapabilities) and the [BC Data Catalogue API](https://catalogue.data.gov.bc.ca/dataset/bc-data-catalogue-api). 
 
@@ -326,19 +326,20 @@ Use some other tool to reproject the data as required.
 
 ## Development and testing
 
-`bc2pg` tests require database `postgresql://postgres@localhost:5432/test_bcdata` exists and has PostGIS installed:
-
-    psql -c "create database test_bcdata"
-    psql test_bcdata -c "create extension postgis"
-
-Create virtualenv and install `bcdata` in development mode:
+Clone the repo:
 
     $ git clone git@github.com:bcgov/bcdata_py.git
     $ cd bcdata_py
-    $ python -m venv .venv
-    $ source .venv/bin/activate
-    (.venv)$ pip install -e .[test]
-    (.venv)$ py.test
+
+`bc2pg` tests require a PostGIS database, referenced via the `DATABASE_URL` environment variable (matching the image used in CI, see `.github/workflows/tests.yml`). Start one locally with `docker compose`:
+
+    $ docker compose up -d
+    $ export DATABASE_URL=postgresql://postgres:postgres@localhost:5432/postgres
+
+install `bcdata` in development mode with [`uv`](https://docs.astral.sh/uv/):
+
+    $ uv sync --extra test
+    $ uv run pytest
 
 
 ## Other implementations

{bcdata-0.16.0 → bcdata-0.17.0}/pyproject.toml

@@ -11,24 +11,23 @@ authors = [
   {name="Simon Norris", email="snorris@hillcrestgeo.ca"},
 ]
 description = "Download open data, monitor and report on changes"
-requires-python = ">=3.9"
+requires-python = ">=3.10"
 classifiers = [
     "Development Status :: 4 - Beta",
-    "Programming Language :: Python :: 3.9",
     "Programming Language :: Python :: 3.10",
     "Programming Language :: Python :: 3.11",
     "Programming Language :: Python :: 3.12",
     "Programming Language :: Python :: 3.13",
-    "License :: OSI Approved :: MIT License",
+    "License :: OSI Approved :: Apache Software License",
     "Topic :: Scientific/Engineering :: GIS",
     "Operating System :: OS Independent"
 ]
 dependencies = [
+    "click",
     "geoalchemy2",
     "geopandas",
     "owslib",
     "psycopg2-binary",
-    "rasterio",
     "requests",
     "sqlalchemy",
     "stamina"

{bcdata-0.16.0 → bcdata-0.17.0}/src/bcdata/__init__.py

@@ -18,7 +18,7 @@ response = requests.get(PRIMARY_KEY_DB_URL)
 if response.status_code == 200:
     primary_keys = response.json()
 else:
-    raise Exception(f"Failed to download primary key database at {PRIMARY_KEY_DB_URL}")
+    raise ConnectionError(f"Failed to download primary key database at {PRIMARY_KEY_DB_URL}")
     primary_keys = {}
 
-__version__ = "0.16.0"
+__version__ = "0.17.0"

{bcdata-0.16.0 → bcdata-0.17.0}/src/bcdata/bc2pg.py

@@ -28,7 +28,7 @@ SUPPORTED_TYPES = [
 ]
 
 
-def bc2pg(  # noqa: C901
+def bc2pg(
     dataset,
     db_url,
     table=None,
@@ -102,21 +102,20 @@ def bc2pg(  # noqa: C901
 
         # if geometry type is still not populated try the last request
         # (in case all entrys with geom are near the bottom)
-        if not geometry_type:
-            if not urls[-1] == urls[0]:
-                df_temp = WFS.request_features(
-                    url=urls[-1],
-                    as_gdf=True,
-                    lowercase=True,
-                    silent=True,
-                )
-                geometry_type = df_temp.geom_type.unique()[0]  # keep only the first type
-                if numpy.any(
-                    df_temp.has_z.unique()[0]
-                ):  # geopandas does not include Z in geom_type string
-                    geometry_type = geometry_type + "Z"
-                # drop the last request dataframe to free up memory
-                del df_temp
+        if not geometry_type and urls[-1] != urls[0]:
+            df_temp = WFS.request_features(
+                url=urls[-1],
+                as_gdf=True,
+                lowercase=True,
+                silent=True,
+            )
+            geometry_type = df_temp.geom_type.unique()[0]  # keep only the first type
+            if numpy.any(
+                df_temp.has_z.unique()[0]
+            ):  # geopandas does not include Z in geom_type string
+                geometry_type = geometry_type + "Z"
+            # drop the last request dataframe to free up memory
+            del df_temp
 
         # ensure geom type is valid
         geometry_type = geometry_type.upper()

{bcdata-0.16.0 → bcdata-0.17.0}/src/bcdata/bcdc.py

@@ -100,25 +100,26 @@ def get_table_definition(table_name):
             # iterate through resources associated with each package
             for resource in result["resources"]:
                 # only examine geographic resources with object name key
-                if "object_name" in resource.keys() and resource["bcdc_type"] == "geographic":
-                    # confirm that object name matches table name and schema is present
-                    if (
-                        (
-                            table_name == resource["object_name"]
-                            # hack to handle object name / table name mismatch for NR Districts
-                            or (
-                                table_name == "WHSE_ADMIN_BOUNDARIES.ADM_NR_DISTRICTS_SPG"
-                                and resource["object_name"]
-                                == "WHSE_ADMIN_BOUNDARIES.ADM_NR_DISTRICTS_SP"
-                            )
+                # confirm that object name matches table name and schema is present
+                if (
+                    "object_name" in resource
+                    and resource["bcdc_type"] == "geographic"
+                    and (
+                        table_name == resource["object_name"]
+                        # hack to handle object name / table name mismatch for NR Districts
+                        or (
+                            table_name == "WHSE_ADMIN_BOUNDARIES.ADM_NR_DISTRICTS_SPG"
+                            and resource["object_name"]
+                            == "WHSE_ADMIN_BOUNDARIES.ADM_NR_DISTRICTS_SP"
                         )
-                        and "details" in resource.keys()
-                        and resource["details"] != []
-                    ):
-                        table_definition["schema"] = resource["details"]
-                        # look for comments only if details/schema was found
-                        if "object_table_comments" in resource.keys():
-                            table_definition["comments"] = resource["object_table_comments"]
+                    )
+                    and "details" in resource
+                    and resource["details"] != []
+                ):
+                    table_definition["schema"] = resource["details"]
+                    # look for comments only if details/schema was found
+                    if "object_table_comments" in resource:
+                        table_definition["comments"] = resource["object_table_comments"]
 
     if not table_definition["schema"]:
         log.warning(f"BC Data Catalouge API search provides no schema for: {table_name}")

{bcdata-0.16.0 → bcdata-0.17.0}/src/bcdata/cli.py

@@ -5,13 +5,26 @@ import re
 import sys
 
 import click
-from cligj import compact_opt, indent_opt, quiet_opt, verbose_opt
 
 import bcdata
 from bcdata.database import Database
 
 LOG_FORMAT = "%(asctime)s:%(levelname)s:%(name)s: %(message)s"
 
+# inlined from cligj (unmaintained), as per
+# https://github.com/rasterio/rasterio/pull/3364
+verbose_opt = click.option("--verbose", "-v", count=True, help="Increase verbosity.")
+
+quiet_opt = click.option("--quiet", "-q", count=True, help="Decrease verbosity.")
+
+indent_opt = click.option(
+    "--indent", type=int, default=None, help="Indentation level for JSON output"
+)
+
+compact_opt = click.option(
+    "--compact/--not-compact", default=False, help="Use compact separators (',', ':')."
+)
+
 
 def configure_logging(verbosity):
     log_level = max(10, 30 - 10 * verbosity)
@@ -45,10 +58,8 @@ def bounds_handler(ctx, param, value):
             retval = tuple(float(x) for x in re.split(r"[,\s]+", value))
             assert len(retval) == 4
             return retval
-        except Exception:
-            raise click.BadParameter(
-                "{0!r} is not a valid bounding box representation".format(value)
-            )
+        except Exception:  # noqa: BLE001 - any parse failure should become a clean CLI error
+            raise click.BadParameter(f"{value!r} is not a valid bounding box representation")
     else:  # pragma: no cover
         return retval
 
@@ -396,7 +407,7 @@ def bc2pg(
     # if refreshing, flush from temp bcdata schema to target schema
     if refresh:
         db = Database(db_url)
-        s, table = out_table.split(".")
+        _, table = out_table.split(".")
         db.refresh(schema_target, table)
         out_table = schema_target + "." + table
         if timestamp:
@@ -404,4 +415,4 @@ def bc2pg(
 
     # do not notify of data load completion when no data load has occured
     if not schema_only:
-        log.info("Load of {} to {} in {} complete".format(dataset, out_table, db_url))
+        log.info(f"Load of {dataset} to {out_table} in {db_url} complete")

{bcdata-0.16.0 → bcdata-0.17.0}/src/bcdata/database.py

@@ -9,12 +9,12 @@ from sqlalchemy.dialects.postgresql import DATE, NUMERIC, VARCHAR
 log = logging.getLogger(__name__)
 
 
-class Database(object):
+class Database:
     """Wrapper around sqlalchemy"""
 
-    def __init__(self, url=os.environ.get("DATABASE_URL")):
-        self.url = url
-        self.engine = create_engine(url)
+    def __init__(self, url=None):
+        self.url = url or os.environ.get("DATABASE_URL")
+        self.engine = create_engine(self.url)
         # make sure postgis is available
         try:
             self.query("SELECT postgis_full_version()")
@@ -132,7 +132,7 @@ class Database(object):
     ):
         """build sqlalchemy table definition from bcdc provided json definitions"""
         # remove columns of unsupported types, redundant columns
-        table_details = [c for c in table_details if c["data_type"] in self.supported_types.keys()]
+        table_details = [c for c in table_details if c["data_type"] in self.supported_types]
         table_details = [
             c
             for c in table_details
@@ -148,7 +148,7 @@ class Database(object):
             if table_details[i]["data_type"] == "VARCHAR2":
                 column_type = column_type(int(table_details[i]["data_precision"]))
             # check that comments are present
-            if "column_comments" in table_details[i].keys():
+            if "column_comments" in table_details[i]:
                 column_comments = table_details[i]["column_comments"]
             else:
                 column_comments = None

{bcdata-0.16.0 → bcdata-0.17.0}/src/bcdata/wfs.py

@@ -5,7 +5,7 @@ import os
 import sys
 import warnings
 import xml.etree.ElementTree as ET
-from datetime import datetime, timedelta
+from datetime import datetime, timedelta, timezone
 from pathlib import Path
 from urllib.parse import urlencode
 
@@ -51,7 +51,51 @@ class ServiceException(Exception):
     pass
 
 
-class BCWFS(object):
+# status codes worth retrying - 5xx are presumed transient service errors,
+# 429 is DataBC's WFS telling us we're rate limited
+RETRYABLE_STATUS_CODES = {429, 500, 502, 503, 504}
+
+
+def _rate_limit_backoff(response):
+    """Return the server-specified backoff (seconds) for a 429 response, if given.
+
+    Checks the standard Retry-After header as well as RateLimit-Reset, which
+    is what DataBC's Kong gateway sends. Returns None if neither is present
+    or parseable, in which case the caller should fall back to the default
+    exponential backoff.
+    """
+    for header in ("Retry-After", "RateLimit-Reset"):
+        value = response.headers.get(header)
+        if value is not None:
+            try:
+                return float(value)
+            except ValueError:
+                continue
+    return None
+
+
+def _is_retryable(exc):
+    """Decide whether a request exception is worth retrying.
+
+    Used as the ``on=`` hook for @stamina.retry - see
+    https://stamina.hynek.me/en/stable/tutorial.html
+
+    Returning a float instead of True overrides the default backoff, which
+    is used to honour a 429 response's server-specified retry delay.
+    """
+    if isinstance(exc, requests.HTTPError):
+        response = exc.response
+        if response is None or response.status_code not in RETRYABLE_STATUS_CODES:
+            return False
+        if response.status_code == 429:
+            backoff = _rate_limit_backoff(response)
+            if backoff is not None:
+                return backoff
+        return True
+    return isinstance(exc, (requests.ConnectionError, requests.Timeout))
+
+
+class BCWFS:
     """Wrapper around web feature service"""
 
     def __init__(self, refresh=False):
@@ -94,17 +138,16 @@ class BCWFS(object):
         if not os.path.exists(os.path.join(cache_file)):
             return True
         else:
-            mod_date = datetime.fromtimestamp(os.path.getmtime(cache_file))
+            mod_date = datetime.fromtimestamp(os.path.getmtime(cache_file), tz=timezone.utc)
             # if file older than specified days or empty, return true
-            if (
-                mod_date < (datetime.now() - timedelta(days=self.cache_refresh_days))
+            return (
+                mod_date < (datetime.now(tz=timezone.utc) - timedelta(days=self.cache_refresh_days))
                 or os.stat(cache_file).st_size == 0
-            ):
-                return True
-            else:
-                return False
+            )
 
-    @stamina.retry(on=requests.HTTPError, timeout=60)
+    @stamina.retry(
+        on=(requests.HTTPError, requests.ConnectionError, requests.Timeout), attempts=5, timeout=300
+    )
     def _request_schema(self, table):
         schema = wfs_schema.get_schema(
             "https://openmaps.gov.bc.ca/geo/pub/ows",
@@ -113,7 +156,9 @@ class BCWFS(object):
         )
         return schema
 
-    @stamina.retry(on=requests.HTTPError, timeout=60)
+    @stamina.retry(
+        on=(requests.HTTPError, requests.ConnectionError, requests.Timeout), attempts=5, timeout=300
+    )
     def _request_capabilities(self):
         capabilities = ET.tostring(
             wfs200.WebFeatureService_2_0_0(self.ows_url, "2.0.0", None, False)._capabilities,
@@ -121,7 +166,39 @@ class BCWFS(object):
         )
         return capabilities
 
-    @stamina.retry(on=requests.HTTPError, timeout=60)
+    @stamina.retry(on=_is_retryable, attempts=5, timeout=300)
+    def _request(self, url, params=None, silent=False):
+        """Submit a request to DataBC WFS and return the raw response.
+
+        The WFS gateway itself can take up to ~60s to fail with a 504, so
+        the retry timeout above is sized to fit several such attempts
+        rather than being exhausted by a single one.
+        """
+        r = requests.get(url, params=params, headers=self.request_headers, timeout=(10, 65))
+        if not silent:
+            log.info(r.url)
+        else:
+            log.debug(r.url)
+        if r.status_code in [400, 401, 404]:
+            log.error(f"HTTP error {r.status_code}")
+            log.error(f"Response headers: {r.headers}")
+            log.error(f"Response text: {r.text}")
+            raise ServiceException(r.text)  # presumed request error
+        if r.status_code == 429:
+            backoff = _rate_limit_backoff(r)
+            if backoff is not None:
+                log.warning(
+                    f"Rate limited by DataBC WFS, backing off for {backoff}s before retrying"
+                )
+            else:
+                log.warning("Rate limited by DataBC WFS, retrying")
+        elif r.status_code in RETRYABLE_STATUS_CODES:
+            log.warning(f"HTTP error: {r.status_code}, retrying")
+            log.warning(f"Response headers: {r.headers}")
+            log.warning(f"Response text: {r.text}")
+        r.raise_for_status()
+        return r
+
     def _request_count(self, table, query=None, bounds=None, bounds_crs=None, geom_column=None):
         payload = {
             "service": "WFS",
@@ -138,60 +215,16 @@ class BCWFS(object):
                 bounds_crs=bounds_crs,
                 geom_column=geom_column,
             )
-
-        r = requests.get(self.wfs_url, params=payload, headers=self.request_headers)
-        log.debug(r.url)
-        if r.status_code in [400, 401, 404]:
-            log.error(f"HTTP error {r.status_code}")
-            log.error(f"Response headers: {r.headers}")
-            log.error(f"Response text: {r.text}")
-            raise ServiceException(r.text)  # presumed request error
-        elif r.status_code in [500, 502, 503, 504]:  # presumed serivce error, retry
-            log.warning(f"HTTP error: {r.status_code}, retrying")
-            log.warning(f"Response headers: {r.headers}")
-            log.warning(f"Response text: {r.text}")
-            r.raise_for_status()
+        r = self._request(self.wfs_url, params=payload, silent=True)
         return int(ET.fromstring(r.text).attrib["numberMatched"])
 
-    @stamina.retry(on=requests.HTTPError, timeout=60)
     def _request_features(self, url, silent=False):
-        """Submit a getfeature request to DataBC WFS and return feature collection"""
-        r = requests.get(url, headers=self.request_headers)
-        if not silent:
-            log.info(r.url)
-        else:
-            log.debug(r.url)
-        if r.status_code in [400, 401, 404]:
-            log.error(f"HTTP error {r.status_code}")
-            log.error(f"Response headers: {r.headers}")
-            log.error(f"Response text: {r.text}")
-            raise ServiceException(r.text)  # presumed request error
-        elif r.status_code in [500, 502, 503, 504]:  # presumed serivce error, retry
-            log.warning(f"HTTP error: {r.status_code}")
-            log.warning(f"Response headers: {r.headers}")
-            log.warning(f"Response text: {r.text}")
-            r.raise_for_status()
-        return r.json()["features"]
+        """Submit a getfeature request to DataBC WFS and return features"""
+        return self._request(url, silent=silent).json()["features"]
 
-    @stamina.retry(on=requests.HTTPError, timeout=60)
     def _request_featurecollection(self, url, silent=False):
         """Submit a getfeature request to DataBC WFS and return feature collection"""
-        r = requests.get(url, headers=self.request_headers)
-        if not silent:
-            log.info(r.url)
-        else:
-            log.debug(r.url)
-        if r.status_code in [400, 401, 404]:
-            log.error(f"HTTP error {r.status_code}")
-            log.error(f"Response headers: {r.headers}")
-            log.error(f"Response text: {r.text}")
-            raise ServiceException(r.text)  # presumed request error
-        elif r.status_code in [500, 502, 503, 504]:  # presumed serivce error, retry
-            log.warning(f"HTTP error: {r.status_code}")
-            log.warning(f"Response headers: {r.headers}")
-            log.warning(f"Response text: {r.text}")
-            r.raise_for_status()
-        return r.json()
+        return self._request(url, silent=silent).json()
 
     def build_bounds_filter(self, query, bounds, bounds_crs, geom_column):
         """The bbox param shortcut is mutually exclusive with CQL_FILTER,
@@ -337,8 +370,7 @@ class BCWFS(object):
                 bounds_crs=bounds_crs,
                 geom_column=geom_column,
             )
-            if count > n:
-                count = n
+            count = min(count, n)
 
         log.info(f"Total features requested: {count}")
 
@@ -443,8 +475,10 @@ def get_data(
         gdf = pd.concat(results)
     elif len(results) == 1:
         gdf = results[0]
+    # generally all datasets should have records, but handle the case of zero results
+    # by creating an empty geodataframe
     else:
-        gdf = gpd.GeoDataFrame()
+        gdf = gpd.GeoDataFrame(geometry=[])
     if as_gdf:
         return gdf
     else:

{bcdata-0.16.0 → bcdata-0.17.0}/src/bcdata.egg-info/PKG-INFO

@@ -1,27 +1,26 @@
 Metadata-Version: 2.4
 Name: bcdata
-Version: 0.16.0
+Version: 0.17.0
 Summary: Download open data, monitor and report on changes
 Author-email: Simon Norris <snorris@hillcrestgeo.ca>
 Project-URL: Homepage, https://github.com/smnorris/bcdata
 Project-URL: Issues, https://github.com/smnorris/bcdata
 Classifier: Development Status :: 4 - Beta
-Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
-Classifier: License :: OSI Approved :: MIT License
+Classifier: License :: OSI Approved :: Apache Software License
 Classifier: Topic :: Scientific/Engineering :: GIS
 Classifier: Operating System :: OS Independent
-Requires-Python: >=3.9
+Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE.txt
+Requires-Dist: click
 Requires-Dist: geoalchemy2
 Requires-Dist: geopandas
 Requires-Dist: owslib
 Requires-Dist: psycopg2-binary
-Requires-Dist: rasterio
 Requires-Dist: requests
 Requires-Dist: sqlalchemy
 Requires-Dist: stamina
@@ -42,7 +41,7 @@ There is a [wealth of British Columbia geographic information available as open
 data](https://catalogue.data.gov.bc.ca/dataset?download_audience=Public),
 but most sources are available only via WFS - and the syntax to download WFS data via `ogr2ogr` and/or `curl/wget` can be awkward.
 
-This tool attempts to simplify downloads of BC geographic data and smoothly integrate with PostGIS and Python GIS tools like `geopandas`, `fiona` and `rasterio`. The tool only accesses geographic data [avialable via WFS](https://bcgov.github.io/data-publication/pages/tips_tricks_webservices.html) - for other BC open data, download the files directly with `requests` / `curl` / `ogr2ogr` / etc (or the [bcdata R package](https://github.com/bcgov/bcdata)).
+This tool attempts to simplify downloads of BC geographic data and smoothly integrate with PostGIS and Python GIS tools like `geopandas` and `fiona`. The tool only accesses geographic data [avialable via WFS](https://bcgov.github.io/data-publication/pages/tips_tricks_webservices.html) - for other BC open data, download the files directly with `requests` / `curl` / `ogr2ogr` / etc (or the [bcdata R package](https://github.com/bcgov/bcdata)).
 
 If downloads are failing, consult the [BCGov Data Systems and Services uptime monitor](https://uptime.com/statuspage/bcgov-dss) - this tool depends on BC Open Geospatial Web Services API at [openmaps.gov.bc.ca](https://openmaps.gov.bc.ca/geo/pub/ows?service=WFS&request=Getcapabilities) and the [BC Data Catalogue API](https://catalogue.data.gov.bc.ca/dataset/bc-data-catalogue-api). 
 
@@ -359,19 +358,20 @@ Use some other tool to reproject the data as required.
 
 ## Development and testing
 
-`bc2pg` tests require database `postgresql://postgres@localhost:5432/test_bcdata` exists and has PostGIS installed:
-
-    psql -c "create database test_bcdata"
-    psql test_bcdata -c "create extension postgis"
-
-Create virtualenv and install `bcdata` in development mode:
+Clone the repo:
 
     $ git clone git@github.com:bcgov/bcdata_py.git
     $ cd bcdata_py
-    $ python -m venv .venv
-    $ source .venv/bin/activate
-    (.venv)$ pip install -e .[test]
-    (.venv)$ py.test
+
+`bc2pg` tests require a PostGIS database, referenced via the `DATABASE_URL` environment variable (matching the image used in CI, see `.github/workflows/tests.yml`). Start one locally with `docker compose`:
+
+    $ docker compose up -d
+    $ export DATABASE_URL=postgresql://postgres:postgres@localhost:5432/postgres
+
+install `bcdata` in development mode with [`uv`](https://docs.astral.sh/uv/):
+
+    $ uv sync --extra test
+    $ uv run pytest
 
 
 ## Other implementations

{bcdata-0.16.0 → bcdata-0.17.0}/src/bcdata.egg-info/requires.txt

@@ -1,8 +1,8 @@
+click
 geoalchemy2
 geopandas
 owslib
 psycopg2-binary
-rasterio
 requests
 sqlalchemy
 stamina

{bcdata-0.16.0 → bcdata-0.17.0}/tests/test_bc2pg.py

@@ -1,6 +1,7 @@
 import os
 
 import pytest
+from psycopg2 import errors as pg_errors
 
 import bcdata
 from bcdata.database import Database
@@ -83,17 +84,17 @@ def test_bc2pg_geometry_type():
 
 
 def test_bc2pg_geometry_type_mismatch():
-    with pytest.raises(Exception):
+    with pytest.raises(pg_errors.InvalidParameterValue):
         bcdata.bc2pg(AIRPORTS_TABLE, DB_URL, count=10, geometry_type="LINESTRING")
 
 
 def test_bc2pg_geometry_type_invalid():
-    with pytest.raises(Exception):
+    with pytest.raises(ValueError):
         bcdata.bc2pg(AIRPORTS_TABLE, DB_URL, count=10, geometry_type="MULTIPOLYGONZ")
 
 
 def test_bc2pg_primary_key_invalid():
-    with pytest.raises(Exception):
+    with pytest.raises(ValueError):
         bcdata.bc2pg(AIRPORTS_TABLE, DB_URL, count=10, primary_key="airport_primary_key")
 
 

{bcdata-0.16.0 → bcdata-0.17.0}/tests/test_cli.py

@@ -3,6 +3,7 @@ import os
 
 from click.testing import CliRunner
 
+import bcdata
 from bcdata.cli import cli
 from bcdata.database import Database
 
@@ -16,24 +17,23 @@ BBOX_LL = "-123.396104,48.404465,-123.342588,48.425401"
 RIVERS_TABLE = "whse_basemapping.fwa_rivers_poly"
 ASSESSMENTS_TABLE = "whse_fish.pscis_assessment_svw"
 
-# Note that these tests depend on airport counts.
-# If airports are added to or removed from source layer, tests will fail
-
 
 def test_info_table():
     runner = CliRunner()
+    expected_count = bcdata.get_count(AIRPORTS_TABLE)
     result = runner.invoke(cli, ["info", AIRPORTS_TABLE])
     assert result.exit_code == 0
-    assert 'name": "{}"'.format(AIRPORTS_TABLE) in result.output
-    assert '"count": 455' in result.output
+    assert f'name": "{AIRPORTS_TABLE}"' in result.output
+    assert f'"count": {expected_count}' in result.output
 
 
 def test_info_package():
     runner = CliRunner()
+    expected_count = bcdata.get_count(AIRPORTS_TABLE)
     result = runner.invoke(cli, ["info", AIRPORTS_PACKAGE])
     assert result.exit_code == 0
-    assert 'name": "{}"'.format(AIRPORTS_TABLE) in result.output
-    assert '"count": 455' in result.output
+    assert f'name": "{AIRPORTS_TABLE}"' in result.output
+    assert f'"count": {expected_count}' in result.output
 
 
 def test_list():
@@ -45,9 +45,10 @@ def test_list():
 
 def test_cat():
     runner = CliRunner()
+    expected_count = bcdata.get_count(AIRPORTS_TABLE)
     result = runner.invoke(cli, ["cat", AIRPORTS_TABLE])
     assert result.exit_code == 0
-    assert len(result.output.split("\n")) == 456
+    assert len(result.output.split("\n")) == expected_count + 1
 
 
 def test_cat_query():

{bcdata-0.16.0 → bcdata-0.17.0}/tests/test_wfs.py

@@ -35,6 +35,61 @@ def test_http_error_404():
             bcdata.get_data(AIRPORTS_TABLE)
 
 
+def _http_error(status_code):
+    response = requests.Response()
+    response.status_code = status_code
+    return requests.HTTPError(response=response)
+
+
+def test_is_retryable_5xx_and_429():
+    """5xx (presumed transient service errors) and 429 (rate limited) are retried"""
+    for code in (429, 500, 502, 503, 504):
+        assert bcdata.wfs._is_retryable(_http_error(code)) is True
+
+
+def test_is_retryable_4xx_not_retried():
+    """4xx client errors (other than 429) are not retried"""
+    for code in (400, 401, 404):
+        assert bcdata.wfs._is_retryable(_http_error(code)) is False
+
+
+def test_is_retryable_connection_and_timeout_errors():
+    """Connection/timeout failures are also worth retrying, not just bad HTTP responses"""
+    assert bcdata.wfs._is_retryable(requests.ConnectionError()) is True
+    assert bcdata.wfs._is_retryable(requests.Timeout()) is True
+
+
+def test_is_retryable_429_without_headers_uses_default_backoff():
+    assert bcdata.wfs._is_retryable(_http_error(429)) is True
+
+
+def test_is_retryable_429_honours_retry_after_header():
+    response = requests.Response()
+    response.status_code = 429
+    response.headers["Retry-After"] = "12"
+    backoff = bcdata.wfs._is_retryable(requests.HTTPError(response=response))
+    assert backoff == 12.0
+
+
+def test_is_retryable_429_honours_ratelimit_reset_header():
+    """DataBC's Kong gateway sends RateLimit-Reset rather than Retry-After"""
+    response = requests.Response()
+    response.status_code = 429
+    response.headers["RateLimit-Reset"] = "5"
+    backoff = bcdata.wfs._is_retryable(requests.HTTPError(response=response))
+    assert backoff == 5.0
+
+
+def test_rate_limit_backoff_unparseable_header_falls_back_to_none():
+    response = requests.Response()
+    response.headers["Retry-After"] = "Wed, 21 Oct 2026 07:28:00 GMT"
+    assert bcdata.wfs._rate_limit_backoff(response) is None
+
+
+def test_is_retryable_other_exceptions_not_retried():
+    assert bcdata.wfs._is_retryable(ValueError("nope")) is False
+
+
 def test_validate_table_lowercase():
     table = bcdata.validate_name(AIRPORTS_TABLE.lower())
     assert table == AIRPORTS_TABLE
@@ -81,6 +136,11 @@ def test_get_null_gdf():
     assert type(gdf) is GeoDataFrame
 
 
+def test_get_null_geojson():
+    data = bcdata.get_data(UTMZONES_KEY, query="UTM_ZONE=9999")
+    assert data == {"type": "FeatureCollection", "features": []}
+
+
 def test_get_data_small():
     data = bcdata.get_data(AIRPORTS_TABLE)
     assert data["type"] == "FeatureCollection"
@@ -93,7 +153,7 @@ def test_get_data_count():
 
 def test_get_data_lowercase():
     data = bcdata.get_data(AIRPORTS_TABLE, lowercase=True, count=1)
-    assert "airport_name" in data["features"][0]["properties"].keys()
+    assert "airport_name" in data["features"][0]["properties"]
 
 
 def test_get_data_crs():