geodata-sync 0.1.0__tar.gz

geodata_sync-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Nathan Leeks
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

geodata_sync-0.1.0/PKG-INFO

@@ -0,0 +1,135 @@
+Metadata-Version: 2.4
+Name: geodata-sync
+Version: 0.1.0
+Summary: Open GIS data (Ordnance Survey, CKAN portals, WFS, OGC API Features, plain links) loaded into PostGIS and kept up to date, from YAML files
+License-Expression: MIT
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pyyaml>=6
+Requires-Dist: jsonschema>=4.18
+Requires-Dist: requests>=2.28
+Requires-Dist: psycopg2-binary>=2.9
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Dynamic: license-file
+
+# geodata-sync
+
+Open GIS data loaded into PostGIS, and kept up to date, from YAML files. Say which datasets you want, where they come from and how often to check them; `geodata sync` downloads each new version, loads it into staging tables, checks it, backs up the old tables and swaps the new ones in, in one transaction.
+
+- **Find data:** the Ordnance Survey's OpenData downloads, CKAN portals (data.gov.uk and many city, regional and national ones), WFS and OGC API Features services, or plain links. `geodata discover` lists what a source offers and the layers inside a download.
+- **Load it safely:** staging, invalid geometries left out, row counts checked, the old tables kept as backups; a load that fails changes nothing.
+- **Keep it up to date:** each dataset has a schedule (`daily`, `weekly`, `monthly` or a cron expression); `geodata sync` run nightly checks those that are due, and loads only a new version.
+- **Rasters too:** a raster download (OS Terrain 50's 2,800 grid tiles, say) becomes one Cloud-Optimised GeoTIFF.
+
+It only loads data. To publish the tables in GeoServer or GeoServer Cloud, use [GeoServer ConfigOps](https://github.com/Yibbzz/geoserver-configops): `configops refresh` brings GeoServer up to date after a load.
+
+> **Status: early.** Expect breaking changes.
+
+## Install
+
+```bash
+pip install geodata-sync                               # the geodata command; it needs GDAL's tools (apt install gdal-bin)
+docker run --rm ghcr.io/yibbzz/geodata-sync --help     # or its image, GDAL included
+```
+
+## Quick start
+
+With `docker` and `geodata`. This starts a PostGIS database ([`examples/compose.yaml`](examples/compose.yaml)) and loads [`examples/quickstart`](examples/quickstart/project.yaml): Surrey Heath's conservation areas, from data.gov.uk.
+
+```yaml
+apiVersion: geodata/v1alpha1
+kind: Project
+metadata: { name: quickstart }
+spec:
+  database:
+    connection: { secret: database }     # keys: host, port, database, username, password
+  crs: EPSG:27700
+---
+apiVersion: geodata/v1alpha1
+kind: Source
+metadata: { name: data-gov-uk }
+spec: { type: ckan, url: https://data.gov.uk }
+---
+apiVersion: geodata/v1alpha1
+kind: Dataset
+metadata: { name: conservation-areas }
+spec:
+  source: data-gov-uk
+  product: conservation-areas125         # `geodata discover data-gov-uk --search "conservation areas"` finds it
+  update: monthly
+  tables:
+    conservation_areas: {}               # the layer in the download; table conservation_areas_conservation_areas
+```
+
+```bash
+docker compose -f examples/compose.yaml up -d --wait
+
+cat > examples/quickstart/secrets.yaml <<'EOF'          # git-ignored
+database: { host: localhost, port: "5433", database: postgres, username: postgres, password: postgres }
+EOF
+
+geodata sync -C examples/quickstart
+#   conservation-areas: loaded 2026-11-11 in 0s: data_layers.conservation_areas_conservation_areas (10 rows)
+#     note: 1 invalid geometry left out
+geodata status -C examples/quickstart
+#   DATASET             UPDATE   LOADED      AT                ROWS  NEXT CHECK        LAST CHECK
+#   conservation-areas  monthly  2026-11-11  2026-11-13 08:51  10    2026-12-13 07:51  success
+geodata sync -C examples/quickstart
+#   conservation-areas: not due (monthly)
+```
+
+`docker compose -f examples/compose.yaml down -v` removes it all. [docs/spec.md](docs/spec.md) describes every kind, field and command.
+
+## Examples
+
+| Project | What it holds |
+| --- | --- |
+| [`examples/quickstart`](examples/quickstart/) | The quick start above: one dataset from data.gov.uk, 10 polygons. Loads in seconds |
+| [`examples/uk-basemap`](examples/uk-basemap/) | OS Open Roads (7.3M features), OS Boundary-Line, the Environment Agency's flood zones (415k polygons) and the conservation areas. Downloads about 3 GB; the first load takes 20 minutes or so. GeoServer ConfigOps's `examples/uk-basemap` publishes them |
+| [`examples/terrain`](examples/terrain/) | OS Terrain 50 (heights of Great Britain, 50 m) made into one Cloud-Optimised GeoTIFF |
+
+## How to
+
+**Find a dataset:**
+
+```bash
+geodata discover os-downloads                                   # OS products
+geodata discover data-gov-uk --search "flood zones" -C my-data  # a Source in the project
+geodata discover data-gov-uk <product> -C my-data               # its files and the layers inside
+geodata validate my-data                                        # problems come out as file:line
+```
+
+**Load now** rather than when due: `geodata sync -C my-data <dataset>` (`--force` loads it even when the version hasn't changed).
+
+**Refresh GeoServer after a load:** `geodata sync -C my-data --loaded loaded.txt` writes what it loaded (`schema.table`, or a raster's path), and `configops refresh -C my-geoserver --from loaded.txt` works out the bounds of the layers on them again and clears their cached tiles.
+
+**Keep the logins out of the files.** A project names Secrets (`connection: { secret: database }`); `geodata` reads each from a git-ignored `secrets.yaml` in the project folder, from `GEODATA_<SECRET>_<KEY>` environment variables, or from files `$GEODATA_SECRETS_DIR/<secret>/<key>` (how Kubernetes mounts Secrets).
+
+**Run it in Kubernetes:** [docs/kubernetes.md](docs/kubernetes.md) has a nightly CronJob that loads, then refreshes GeoServer.
+
+[docs/sources-notes.md](docs/sources-notes.md) has what we learned about the sources: big layers over WFS or OGC API Features don't load reliably, so use a file download when there is one.
+
+## Testing
+
+```bash
+python3 -m venv .venv && .venv/bin/pip install -e '.[dev]'
+.venv/bin/pytest                                         # the PostGIS tests skip without one
+
+docker run -d --rm --name test-postgis -p 55432:5432 \
+  -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=gis postgis/postgis:18-3.7
+TEST_DB_HOST=localhost TEST_DB_PORT=55432 .venv/bin/pytest -m integration
+```
+
+CI runs the tests with a real PostGIS (test), shellcheck and a schema check of the Kubernetes example (lint), and the image build (image).
+
+## Releasing
+
+Set the version in `pyproject.toml` and `geodata_sync/__init__.py`, merge, then push the tag `v<version>`: the Release workflow publishes the package to PyPI (`geodata-sync`), the image to `ghcr.io/yibbzz/geodata-sync`, and a GitHub release.
+
+## Licence and data attribution
+
+MIT (see `LICENSE`). This code started in [GeoServer ConfigOps](https://github.com/Yibbzz/geoserver-configops), where it was the `sync` and `discover` commands.
+
+Data you load keeps its own licence. OS OpenData is released under the [Open Government Licence v3.0](https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/) and needs the attribution "Contains OS data © Crown copyright and database right &lt;year&gt;" wherever it's shown.

geodata_sync-0.1.0/README.md

@@ -0,0 +1,119 @@
+# geodata-sync
+
+Open GIS data loaded into PostGIS, and kept up to date, from YAML files. Say which datasets you want, where they come from and how often to check them; `geodata sync` downloads each new version, loads it into staging tables, checks it, backs up the old tables and swaps the new ones in, in one transaction.
+
+- **Find data:** the Ordnance Survey's OpenData downloads, CKAN portals (data.gov.uk and many city, regional and national ones), WFS and OGC API Features services, or plain links. `geodata discover` lists what a source offers and the layers inside a download.
+- **Load it safely:** staging, invalid geometries left out, row counts checked, the old tables kept as backups; a load that fails changes nothing.
+- **Keep it up to date:** each dataset has a schedule (`daily`, `weekly`, `monthly` or a cron expression); `geodata sync` run nightly checks those that are due, and loads only a new version.
+- **Rasters too:** a raster download (OS Terrain 50's 2,800 grid tiles, say) becomes one Cloud-Optimised GeoTIFF.
+
+It only loads data. To publish the tables in GeoServer or GeoServer Cloud, use [GeoServer ConfigOps](https://github.com/Yibbzz/geoserver-configops): `configops refresh` brings GeoServer up to date after a load.
+
+> **Status: early.** Expect breaking changes.
+
+## Install
+
+```bash
+pip install geodata-sync                               # the geodata command; it needs GDAL's tools (apt install gdal-bin)
+docker run --rm ghcr.io/yibbzz/geodata-sync --help     # or its image, GDAL included
+```
+
+## Quick start
+
+With `docker` and `geodata`. This starts a PostGIS database ([`examples/compose.yaml`](examples/compose.yaml)) and loads [`examples/quickstart`](examples/quickstart/project.yaml): Surrey Heath's conservation areas, from data.gov.uk.
+
+```yaml
+apiVersion: geodata/v1alpha1
+kind: Project
+metadata: { name: quickstart }
+spec:
+  database:
+    connection: { secret: database }     # keys: host, port, database, username, password
+  crs: EPSG:27700
+---
+apiVersion: geodata/v1alpha1
+kind: Source
+metadata: { name: data-gov-uk }
+spec: { type: ckan, url: https://data.gov.uk }
+---
+apiVersion: geodata/v1alpha1
+kind: Dataset
+metadata: { name: conservation-areas }
+spec:
+  source: data-gov-uk
+  product: conservation-areas125         # `geodata discover data-gov-uk --search "conservation areas"` finds it
+  update: monthly
+  tables:
+    conservation_areas: {}               # the layer in the download; table conservation_areas_conservation_areas
+```
+
+```bash
+docker compose -f examples/compose.yaml up -d --wait
+
+cat > examples/quickstart/secrets.yaml <<'EOF'          # git-ignored
+database: { host: localhost, port: "5433", database: postgres, username: postgres, password: postgres }
+EOF
+
+geodata sync -C examples/quickstart
+#   conservation-areas: loaded 2026-11-11 in 0s: data_layers.conservation_areas_conservation_areas (10 rows)
+#     note: 1 invalid geometry left out
+geodata status -C examples/quickstart
+#   DATASET             UPDATE   LOADED      AT                ROWS  NEXT CHECK        LAST CHECK
+#   conservation-areas  monthly  2026-11-11  2026-11-13 08:51  10    2026-12-13 07:51  success
+geodata sync -C examples/quickstart
+#   conservation-areas: not due (monthly)
+```
+
+`docker compose -f examples/compose.yaml down -v` removes it all. [docs/spec.md](docs/spec.md) describes every kind, field and command.
+
+## Examples
+
+| Project | What it holds |
+| --- | --- |
+| [`examples/quickstart`](examples/quickstart/) | The quick start above: one dataset from data.gov.uk, 10 polygons. Loads in seconds |
+| [`examples/uk-basemap`](examples/uk-basemap/) | OS Open Roads (7.3M features), OS Boundary-Line, the Environment Agency's flood zones (415k polygons) and the conservation areas. Downloads about 3 GB; the first load takes 20 minutes or so. GeoServer ConfigOps's `examples/uk-basemap` publishes them |
+| [`examples/terrain`](examples/terrain/) | OS Terrain 50 (heights of Great Britain, 50 m) made into one Cloud-Optimised GeoTIFF |
+
+## How to
+
+**Find a dataset:**
+
+```bash
+geodata discover os-downloads                                   # OS products
+geodata discover data-gov-uk --search "flood zones" -C my-data  # a Source in the project
+geodata discover data-gov-uk <product> -C my-data               # its files and the layers inside
+geodata validate my-data                                        # problems come out as file:line
+```
+
+**Load now** rather than when due: `geodata sync -C my-data <dataset>` (`--force` loads it even when the version hasn't changed).
+
+**Refresh GeoServer after a load:** `geodata sync -C my-data --loaded loaded.txt` writes what it loaded (`schema.table`, or a raster's path), and `configops refresh -C my-geoserver --from loaded.txt` works out the bounds of the layers on them again and clears their cached tiles.
+
+**Keep the logins out of the files.** A project names Secrets (`connection: { secret: database }`); `geodata` reads each from a git-ignored `secrets.yaml` in the project folder, from `GEODATA_<SECRET>_<KEY>` environment variables, or from files `$GEODATA_SECRETS_DIR/<secret>/<key>` (how Kubernetes mounts Secrets).
+
+**Run it in Kubernetes:** [docs/kubernetes.md](docs/kubernetes.md) has a nightly CronJob that loads, then refreshes GeoServer.
+
+[docs/sources-notes.md](docs/sources-notes.md) has what we learned about the sources: big layers over WFS or OGC API Features don't load reliably, so use a file download when there is one.
+
+## Testing
+
+```bash
+python3 -m venv .venv && .venv/bin/pip install -e '.[dev]'
+.venv/bin/pytest                                         # the PostGIS tests skip without one
+
+docker run -d --rm --name test-postgis -p 55432:5432 \
+  -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=gis postgis/postgis:18-3.7
+TEST_DB_HOST=localhost TEST_DB_PORT=55432 .venv/bin/pytest -m integration
+```
+
+CI runs the tests with a real PostGIS (test), shellcheck and a schema check of the Kubernetes example (lint), and the image build (image).
+
+## Releasing
+
+Set the version in `pyproject.toml` and `geodata_sync/__init__.py`, merge, then push the tag `v<version>`: the Release workflow publishes the package to PyPI (`geodata-sync`), the image to `ghcr.io/yibbzz/geodata-sync`, and a GitHub release.
+
+## Licence and data attribution
+
+MIT (see `LICENSE`). This code started in [GeoServer ConfigOps](https://github.com/Yibbzz/geoserver-configops), where it was the `sync` and `discover` commands.
+
+Data you load keeps its own licence. OS OpenData is released under the [Open Government Licence v3.0](https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/) and needs the attribution "Contains OS data © Crown copyright and database right &lt;year&gt;" wherever it's shown.

geodata_sync-0.1.0/geodata_sync/__init__.py

@@ -0,0 +1,4 @@
+"""geodata-sync: open GIS data loaded into PostGIS, and kept up to date, from YAML files."""
+
+__version__ = "0.1.0"
+API_VERSION = "geodata/v1alpha1"

geodata_sync-0.1.0/geodata_sync/__main__.py

@@ -0,0 +1,3 @@
+from .cli import main
+
+raise SystemExit(main())

geodata_sync-0.1.0/geodata_sync/cli.py

@@ -0,0 +1,141 @@
+"""The geodata command."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from pathlib import Path
+
+from . import __version__
+from .discover import DiscoverError
+from .discover import run as discover
+from .status import run as status
+from .sync import SyncError
+from .sync import run as sync
+from .loader import load
+from .validate import validate
+
+def shown(path: Path) -> str:
+    """A path relative to the current directory when that's shorter."""
+    try:
+        rel = os.path.relpath(path)
+    except ValueError:
+        return str(path)
+    return rel if len(rel) < len(str(path)) else str(path)
+
+
+def cmd_validate(args) -> int:
+    root = Path(args.dir)
+    if not root.is_dir():
+        print(f"geodata: {root} is not a directory", file=sys.stderr)
+        return 2
+    project = load(root)
+    problems = validate(project)
+    errors = [p for p in problems if p.severity == "error"]
+
+    if args.format == "json":
+        print(json.dumps([
+            {"file": shown(p.file), "line": p.line, "severity": p.severity, "resource": p.resource, "message": p.message}
+            for p in problems
+        ], indent=2))
+    else:
+        for p in problems:
+            where = f"{shown(p.file)}:{p.line}" if p.line else shown(p.file)
+            resource = f"{p.resource}: " if p.resource else ""
+            print(f"{where}: {p.severity}: {resource}{p.message}")
+        warnings = len(problems) - len(errors)
+        summary = f"{len(project.resources)} resources in {len(project.files)} files"
+        if problems:
+            print(f"\n{len(errors)} error(s), {warnings} warning(s); {summary}")
+        else:
+            print(f"Valid: {summary}")
+    return 1 if errors else 0
+
+
+def cmd_discover(args) -> int:
+    try:
+        return discover(
+            args.source, args.product, Path(args.dir), fmt=args.format,
+            layers=not args.no_layers, use_cache=not args.refresh,
+            search=args.search, limit=args.limit, resource=args.resource, url=args.url,
+        )
+    except DiscoverError as e:
+        print(f"geodata discover: {e}", file=sys.stderr)
+        return e.code
+
+
+def cmd_status(args) -> int:
+    try:
+        return status(Path(args.dir), fmt=args.format)
+    except SyncError as e:
+        print(f"geodata status: {e}", file=sys.stderr)
+        return e.code
+
+
+def cmd_sync(args) -> int:
+    try:
+        return sync(Path(args.dir), args.datasets, force=args.force, loaded_file=args.loaded)
+    except SyncError as e:
+        print(f"geodata sync: {e}", file=sys.stderr)
+        return e.code
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(prog="geodata", description="Open GIS data loaded into PostGIS, and kept up to date, from YAML files.")
+    parser.add_argument("--version", action="version", version=f"geodata {__version__}")
+    commands = parser.add_subparsers(dest="command", metavar="command")
+
+    v = commands.add_parser("validate", help="check the files against the spec (no network)")
+    v.add_argument("dir", nargs="?", default=".", help="project directory (default: current)")
+    v.add_argument("--format", choices=["text", "json"], default="text")
+    v.set_defaults(func=cmd_validate)
+
+    d = commands.add_parser(
+        "discover", help="list what a source offers, or the layers inside one product",
+        description="List the products a source offers, or the downloads and layers of one product. "
+        "SOURCE is a Source in the project (e.g. os) or a source type: os-downloads, ckan, wfs or ogc-api-features (with --url), "
+        "or url (the product is then a file's address).",
+    )
+    d.add_argument("source", help="a Source name from the project, or a source type")
+    d.add_argument("product", nargs="?", help="a product ID (a CKAN package, a file's URL), to list its downloads and layers")
+    d.add_argument("--search", metavar="WORDS", help="list only products matching these words")
+    d.add_argument("--limit", type=int, metavar="N", help="list at most N products (CKAN default: 20)")
+    d.add_argument("--resource", metavar="ID", help="read this download of the product (a CKAN resource ID or name)")
+    d.add_argument("--url", help="the source's address, e.g. a CKAN portal (overrides the Source's spec.url)")
+    d.add_argument("-C", "--dir", default=".", help="project directory, for Sources and secrets (default: current)")
+    d.add_argument("--format", choices=["text", "json"], default="text")
+    d.add_argument("--no-layers", action="store_true", help="list downloads only; don't read layers from the file")
+    d.add_argument("--refresh", action="store_true", help="read layers again instead of using the cache")
+    d.set_defaults(func=cmd_discover)
+
+    st = commands.add_parser("status", help="datasets: loaded, when, next check",
+                             description="Show the project's datasets from the ingest log. Changes nothing.")
+    st.add_argument("-C", "--dir", default=".", help="project directory (default: current)")
+    st.add_argument("--format", choices=["text", "json"], default="text")
+    st.set_defaults(func=cmd_status)
+
+    s = commands.add_parser(
+        "sync", help="load new versions of datasets into PostGIS",
+        description="Check each due Dataset for a new version and load it: stage, check, back up, swap in. "
+        "Named datasets are checked whatever their schedule. The database comes from the Project's connection secret.",
+    )
+    s.add_argument("datasets", nargs="*", help="only these Datasets, due or not")
+    s.add_argument("-C", "--dir", default=".", help="project directory (default: current)")
+    s.add_argument("--force", action="store_true", help="load even when the version hasn't changed")
+    s.add_argument("--loaded", type=Path, metavar="FILE",
+                   help="write what was loaded to FILE, one per line (schema.table, or a raster's path), for `configops refresh --from FILE`")
+    s.set_defaults(func=cmd_sync)
+
+    argv = sys.argv[1:] if argv is None else argv
+    if argv[:1] == ["discover"]:
+        # options may come between the source and the product
+        args = d.parse_intermixed_args(argv[1:])
+        args.command = "discover"
+    else:
+        args = parser.parse_args(argv)
+    if not args.command:
+        parser.print_help()
+        return 2
+    return args.func(args)

geodata_sync-0.1.0/geodata_sync/crs.py

@@ -0,0 +1,21 @@
+"""Coordinate reference system names, in the short form the spec uses (EPSG:27700)."""
+
+from __future__ import annotations
+
+import re
+
+# urn:ogc:def:crs:EPSG::27700, urn:x-ogc:def:crs:EPSG:27700, EPSG:27700,
+# http://www.opengis.net/def/crs/EPSG/0/27700, http://www.opengis.net/gml/srs/epsg.xml#27700
+EPSG = re.compile(r"(?:EPSG(?::[\d.]*)?:|/EPSG/[\d.]+/|epsg\.xml#)(\d+)$", re.IGNORECASE)
+CRS84 = re.compile(r"(?:OGC/1\.3/|OGC:(?:1\.3:)?)CRS84$", re.IGNORECASE)
+
+
+def short(name: str | None) -> str | None:
+    """EPSG:<code> or CRS84 when recognised, else the name as given."""
+    if not name:
+        return None
+    name = name.strip()
+    if CRS84.search(name):
+        return "CRS84"
+    m = EPSG.search(name)
+    return f"EPSG:{m.group(1)}" if m else name