geoserver-configops 0.1.0__tar.gz

geoserver_configops-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Nathan Leeks
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

geoserver_configops-0.1.0/PKG-INFO

@@ -0,0 +1,341 @@
+Metadata-Version: 2.4
+Name: geoserver-configops
+Version: 0.1.0
+Summary: GeoServer ConfigOps: GeoServer and GeoServer Cloud configuration as code, with plan, apply, import and drift detection
+License-Expression: MIT
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pyyaml>=6
+Requires-Dist: jsonschema>=4.18
+Requires-Dist: requests>=2.28
+Requires-Dist: psycopg2-binary>=2.9
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Dynamic: license-file
+
+# GeoServer ConfigOps
+
+![GeoServer Cloud coverage](https://img.shields.io/badge/GeoServer_Cloud-93%25-brightgreen) ![GeoServer coverage](https://img.shields.io/badge/GeoServer-100%25-brightgreen) Of the 662 settings GeoServer can configure, the e2e tests prove configops sets 618 on GeoServer Cloud (44 more it can't set) and 662 on standard GeoServer ([field by field](docs/geoserver-fields.yaml)).
+
+GeoServer configuration as code, for [GeoServer Cloud](https://github.com/geoserver/geoserver-cloud) and standard GeoServer. `configops` keeps what GeoServer publishes and how it is set up in YAML files, with an OpenTofu-style `plan` and `apply`:
+
+- **publish** layers, styles and layer groups from PostGIS tables, SQL views and other stores (Cloud-Optimised GeoTIFFs, GeoPackages and other files, cascaded WMS, WMTS and WFS, other databases), with tile caching;
+- **set GeoServer up:** its settings and services (contact details, titles, limits), and its security (users, groups, roles, who may see and do what, the master password);
+- **catch drift and keep it:** `plan` on a schedule shows when GeoServer no longer matches the files, `apply` puts it back, and `import` brings a change made in the web UI back into the files instead. `import` is also how you start from a GeoServer you already have.
+
+Point it at a GeoServer you already run: `https://<host>/geoserver-cloud` for GeoServer Cloud, `https://<host>/geoserver` for standard GeoServer. Standard GeoServer can also take a few settings GeoServer Cloud can't (below); when the files set one, configops asks GeoServer which it is (by the version it reports), and on GeoServer Cloud `plan` shows it as `!` with the reason and leaves it out.
+
+Loading open data into PostGIS (Ordnance Survey, CKAN portals such as data.gov.uk, WFS, OGC API Features) and keeping it up to date is a separate tool, [geodata-sync](https://github.com/Yibbzz/geodata-sync); configops publishes the tables, whoever loads them.
+
+> **Status: early.** The same e2e tests run against GeoServer Cloud and against standard GeoServer 3.0.2 (the version GeoServer Cloud 3.0 is built on). Expect breaking changes. [docs/coverage.md](docs/coverage.md) goes through the web UI page by page; the figures above count every field of GeoServer's configuration, read from its source (`tests/geoserver_coverage.py`), and what one of them can't set counts against it.
+
+## Install
+
+```bash
+pip install geoserver-configops                          # the configops command: Python 3.10+, nothing else
+docker run --rm ghcr.io/yibbzz/geoserver-configops --help  # or its image
+```
+
+## Quick start
+
+A whole GeoServer set up from one file, [`examples/quickstart/project.yaml`](examples/quickstart/project.yaml): a workspace, the contact details and WMS title, a Sentinel-2 image read in place on AWS (a Cloud-Optimised GeoTIFF) with its own style, EOX's cloudless mosaic cascaded from their WMS, both as one cached map, and an editor with what only editors may do. Nothing to load.
+
+```yaml
+apiVersion: configops/v1alpha1
+kind: Project
+metadata: { name: quickstart }
+spec:
+  geoserver:
+    url: http://localhost:8080/geoserver           # or https://<host>/geoserver-cloud
+    credentials: { secret: geoserver-admin }       # keys username, password: secrets.yaml, env or a Kubernetes Secret
+  workspace: demo
+---
+# Who runs this GeoServer, as its capabilities documents say
+apiVersion: configops/v1alpha1
+kind: Settings
+metadata: { name: global }
+spec:
+  contact: { contactPerson: Map Team, contactOrganization: Example Council, contactEmail: maps@example.org }
+---
+apiVersion: configops/v1alpha1
+kind: Service
+metadata: { name: wms }
+spec: { title: Example Council maps, maxRenderingTime: 60 }
+---
+# A Sentinel-2 image, a Cloud-Optimised GeoTIFF read in place on AWS
+apiVersion: configops/v1alpha1
+kind: Store
+metadata: { name: sentinel-2 }
+spec:
+  type: cog
+  url: https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/30/U/XC/2025/8/S2C_30UXC_20250825_0_L2A/TCI.tif
+---
+apiVersion: configops/v1alpha1
+kind: Layer
+metadata: { name: imagery }
+spec: { store: sentinel-2, title: Sentinel-2 true colour, style: imagery }   # styles/imagery.sld
+---
+# EOX's cloudless mosaic, cascaded from their WMS
+apiVersion: configops/v1alpha1
+kind: Store
+metadata: { name: eox }
+spec: { type: wms, url: "https://tiles.maps.eox.at/wms?service=WMS&request=GetCapabilities" }
+---
+apiVersion: configops/v1alpha1
+kind: Layer
+metadata: { name: cloudless }
+spec: { store: eox, resource: s2cloudless-2024, title: Sentinel-2 cloudless (EOX) }
+---
+# Both as one map, its tiles cached
+apiVersion: configops/v1alpha1
+kind: LayerGroup
+metadata: { name: basemap }
+spec:
+  title: Basemap
+  layers: [cloudless, imagery]                   # drawn bottom first
+  cache: true
+---
+# An editor, and what only editors may do
+apiVersion: configops/v1alpha1
+kind: Role
+metadata: { name: editor }                       # GeoServer role EDITOR
+spec: {}
+---
+apiVersion: configops/v1alpha1
+kind: User
+metadata: { name: alice }
+spec: { password: { secret: geoserver-users, key: alice }, roles: [EDITOR] }
+---
+apiVersion: configops/v1alpha1
+kind: AccessRule
+metadata: { name: demo-write }
+spec: { layer: { workspace: demo, layer: "*", access: write }, roles: [EDITOR] }
+```
+
+Start a GeoServer to try it on ([`examples/compose.yaml`](examples/compose.yaml)), give configops the logins the file names, and see what it would do:
+
+```bash
+docker compose -f examples/compose.yaml up -d --wait    # standard GeoServer on :8080; a minute or two
+
+cat > examples/quickstart/secrets.yaml <<'EOF'          # git-ignored
+geoserver-admin: { username: admin, password: geoserver }
+geoserver-users: { alice: change-me }
+EOF
+
+configops plan -C examples/quickstart
+```
+
+```
+  + workspace demo
+  ~ settings global
+      contact.contactPerson: None -> "Map Team"
+      contact.contactOrganization: None -> "Example Council"
+      contact.contactEmail: None -> "maps@example.org"
+  ~ service wms
+      title: None -> "Example Council maps"
+      maxRenderingTime: 0 -> 60
+  + role EDITOR
+  + user alice   (groups: none; roles: EDITOR)
+  + store demo:sentinel_2   (cog: https://sentinel-cogs.s3.us-west-2.amazonaws.com/...)
+  + store demo:eox   (wms: https://tiles.maps.eox.at/wms?service=WMS&request=GetCapabilities)
+  + style demo:imagery   (styles/imagery.sld)
+  + layer demo:imagery   (cog store sentinel-2)
+  + layer demo:cloudless   (wms store eox: s2cloudless-2024)
+  + group demo:basemap   (2 members)
+  + rule layers demo.*.w   (roles: EDITOR)
+
+Plan: 10 to add, 2 to change, 0 to delete, 0 unmanaged.
+```
+
+Make it so, and look: <http://localhost:8080/geoserver/demo/wms?service=WMS&version=1.3.0&request=GetMap&layers=demo:basemap&styles=&crs=EPSG:4326&bbox=50.4,-3.0,51.4,-1.4&width=768&height=480&format=image/png>
+
+```bash
+configops apply -C examples/quickstart     # ... Applied 12 change(s).
+configops plan  -C examples/quickstart     # No changes: GeoServer matches the files.
+```
+
+Now change something in the web UI, say the cloudless layer's title (Layers → cloudless). `plan` notices, with exit code 2, which is what a scheduled drift check looks for:
+
+```
+$ configops plan -C examples/quickstart
+  ~ layer demo:cloudless
+      title: "Cloudless, renamed in the web UI" -> "Sentinel-2 cloudless (EOX)"
+```
+
+Either `apply` puts it back, or `import` keeps it: it updates the file in place, only what changed, and leaves the rest of the file, its order and its comments as they were.
+
+```
+$ configops import demo -C examples/quickstart
+Updated the files: 1 changed: Layer cloudless (title)
+Next: git diff, then configops plan -C examples/quickstart
+
+$ configops status -C examples/quickstart
+LAYER                 PUBLISHED  DRIFTED  TILES CACHED
+demo:imagery          yes        no       no
+demo:cloudless        yes        no       no
+demo:basemap (group)  yes        no       yes
+```
+
+`docker compose -f examples/compose.yaml down -v` removes it all. [docs/spec.md](docs/spec.md) describes every kind and field.
+
+## Examples
+
+| Project | What it holds |
+| --- | --- |
+| [`examples/quickstart`](examples/quickstart/) | The quick start above: one file, no data to load |
+| [`examples/uk-basemap`](examples/uk-basemap/) | The worked example: OS Open Roads, OS Boundary-Line, the Environment Agency's flood zones and Surrey's conservation areas, from tables geodata-sync loads (its own `examples/uk-basemap`); 6 layers with their own styles, a basemap group drawn as one layer, tile caching on British National Grid, WMS and WFS service settings, and a layer on a table you load yourself |
+| [`examples/terrain`](examples/terrain/) | A raster in GeoServer's `/data` folder (OS Terrain 50, which geodata-sync's `examples/terrain` makes) drawn with a colour ramp |
+
+## How to
+
+**Publish a table:** a `Layer` with `table: roads` (in the Project's `database.schema`) or `table: { schema: field_data, name: sites }`. The Project names the database, as GeoServer reaches it (`jndi`) and as configops does (`connection`, to see whether a table exists yet). Until the table exists, `plan` shows the layer as `!` (can't be done yet) and `apply` leaves it, and its place in groups, for later.
+
+**After loading new data,** refresh GeoServer: `configops refresh -C my-project data_layers.roads` works the layer's bounds out again and clears its cached tiles, and those of the groups that hold it. `geodata sync --loaded loaded.txt` writes what it loaded, for `configops refresh --from loaded.txt`.
+
+**Keep the logins out of the files.** A project names Secrets (`credentials: { secret: geoserver-admin }`); configops reads each from a git-ignored `secrets.yaml` in the project folder, from `CONFIGOPS_<SECRET>_<KEY>` environment variables, or from files `$CONFIGOPS_SECRETS_DIR/<secret>/<key>` (how Kubernetes mounts Secrets).
+
+**See what would change, or what's loaded:**
+
+```bash
+configops plan   -C my-project    # exit code 2 when there are changes
+configops status -C my-project    # layers: published, drifted, tiles cached
+configops plan   -C my-project --geoserver https://test.example.org/geoserver   # the same files against another GeoServer
+```
+
+**Catch drift.** Run `configops plan` on a schedule: it exits 2 when GeoServer no longer matches the files, say a layer changed in the web UI. `configops apply` puts GeoServer back; or keep the change by running `configops import` (below) and committing it.
+
+**Run it in Kubernetes**, beside GeoServer: [docs/kubernetes.md](docs/kubernetes.md) has a Job that applies the project and an hourly drift check, from an image with your project in it.
+
+**Publish data GeoServer reads where it is:** a `Store` for a Cloud-Optimised GeoTIFF (over `https://` or `s3://`), a remote WMS, WMTS or WFS to cascade, or another PostGIS database, and a `Layer` with `store:`:
+
+```yaml
+apiVersion: configops/v1alpha1
+kind: Store
+metadata: { name: sentinel-2-surrey }
+spec: { type: cog, url: "https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/30/U/XC/2025/8/S2C_30UXC_20250825_0_L2A/TCI.tif" }
+---
+apiVersion: configops/v1alpha1
+kind: Layer
+metadata: { name: sentinel-2 }
+spec: { store: sentinel-2-surrey, title: Sentinel-2 true colour }
+```
+
+Files in GeoServer's `/data` folder (GeoTIFFs, GeoPackages, shapefiles, FlatGeobuf, PMTiles, GeoParquet, CSV, image mosaics and pyramids, app-schema mappings) are `Store`s with `path:`; SQL Server and Oracle are `Store`s too.
+
+**Publish a SQL view:** a Layer with `sql: { schema, query, key, geometry }` instead of a table.
+
+**Cache tiles:** `cache: true` on a layer or group, or `cache: { gridsets, formats, metaTiles, blobStore, ... }`; a `Gridset` adds your own (British National Grid in uk-basemap) and a `BlobStore` keeps tiles in a folder on the data volume or in S3.
+
+**Set contact details, service titles and limits:** a `Settings` resource for GeoServer's global settings (or a workspace's own), and a `Service` per OGC service, in GeoServer's own field names. Only the fields you set are managed:
+
+```yaml
+apiVersion: configops/v1alpha1
+kind: Service
+metadata: { name: wfs }          # wms, wfs, wcs or wmts
+spec:
+  title: Example Council features
+  maxFeatures: 100000
+```
+
+**Control who sees and does what:** `Role`, `Group` and `User` resources, with passwords from a Secret, and an `AccessRule` per layer, service or REST rule:
+
+```yaml
+apiVersion: configops/v1alpha1
+kind: User
+metadata: { name: alice }
+spec: { password: { secret: geoserver-users, key: alice }, groups: [editors] }
+---
+apiVersion: configops/v1alpha1
+kind: AccessRule
+metadata: { name: flood-zones-for-editors }
+spec: { layer: { workspace: uk, layer: flood_zones, access: read }, roles: [EDITOR] }
+```
+
+`plan` spots a changed password by logging in with it, and a REST rule GeoServer would never reach. GeoServer's own `admin` user, roles and default rules are left alone unless a file names them.
+
+**Bring in an existing GeoServer:** `configops import <workspace> --settings --security -C my-project` writes the files that describe the workspace, GeoServer's settings and services, and its security, so `plan` on them has nothing to do. Passwords can't be read back: put each user's in Secret `geoserver-users`, as the import's note says.
+
+**Allow web pages on other sites to call GeoServer (CORS).** On standard GeoServer, `Settings` `global.cors`. GeoServer Cloud's gateway answers cross-origin requests itself, from any origin; to allow only yours, give the gateway the environment variable `SPRING_APPLICATION_JSON` = `{"spring": {"cloud": {"gateway": {"server": {"webmvc": {"globalcors": {"cors-configurations": {"[/**]": {"allowedOrigins": ["https://maps.example.org"]}}}}}}}}}` (`allowedMethods`, `allowedHeaders`, `allowCredentials` and `maxAge` go beside `allowedOrigins`).
+
+**Let GeoServer fetch remote styles and graphics.** GeoServer Cloud checks every external URL (a style given as `SLD=<url>` in a GetMap, a graphic a style points to) and refuses those no check allows, which at first is all of them. Allow yours with a `URLCheck`:
+
+```yaml
+apiVersion: configops/v1alpha1
+kind: URLCheck
+metadata: { name: council-styles }
+spec: { regex: '^https://styles\.example\.org/.*$', description: Our published styles }
+```
+
+`Settings` `global.urlChecks: false` turns the checks off. `global.requestLogging: { enabled: true, headers: true }` logs every request in the services' logs.
+
+**Keep a change made in the web UI:** run the same `import` in the project folder. It updates the files in place, only what changed, keeping your names, order and comments; review it with `git diff` and commit. `--prune` also takes out what GeoServer no longer has.
+
+## What can't be done as code
+
+All of GeoServer's configuration can be kept in the files on standard GeoServer: [docs/coverage.md](docs/coverage.md) goes page by page, and [docs/geoserver-fields.yaml](docs/geoserver-fields.yaml) field by field. GeoServer Cloud has no way to set these, so they work only on standard GeoServer:
+
+| What | In the files | Why GeoServer Cloud can't |
+| --- | --- | --- |
+| Group styles | `LayerGroup` `styles` | Its catalog database doesn't keep their layers: the group breaks after a restart |
+| Oracle stores | `Store` `type: oracle` | Its images don't ship Oracle's JDBC driver |
+| Tile cache disk quota | `Settings` `global.diskQuota` | It switches the disk quota off and has no REST endpoint for it |
+| Content Security Policy | `Settings` `global.contentSecurityPolicy` | Its services don't send GeoServer's policy header; the web UI sends its own |
+| CORS | `Settings` `global.cors` | Its gateway answers CORS: set the gateway's instead (above) |
+
+Some settings take effect only when GeoServer's services restart, and `apply` says so: the tile caching defaults, and on standard GeoServer CORS and the disk quota's store. A few belong to how GeoServer is deployed rather than its configuration: which GeoServer Cloud services run, logging, WPS and GeoServer ACL.
+
+## How it fits together
+
+```
+ project files (YAML, SLD) ──► configops plan / apply / import / status ──► GeoServer or GeoServer Cloud (REST API)
+                                                                                    │
+ geodata-sync, or your own loads ──► PostGIS tables ──── GeoServer reads ───────────┤
+                                 └─► configops refresh: bounds, cached tiles        ▼
+                                                                     WMS / WFS / WCS / tiles
+```
+
+## Repository layout
+
+| Path | What |
+| --- | --- |
+| `configops/`, `pyproject.toml`, `Dockerfile` | The `configops` command and its image |
+| `examples/` | Example projects, and `compose.yaml`: a GeoServer and a database to try them on |
+| `docs/spec.md` | The file format and every command |
+| `docs/kubernetes.md`, `docs/kubernetes/` | Running configops in Kubernetes |
+| `docs/coverage.md` | Every page of GeoServer's web UI, and what the tool covers |
+| `docs/geoserver-cloud-notes.md` | What we learned about GeoServer Cloud's REST API, GeoWebCache and services: quirks, shapes, timing |
+| `tests/` | Unit tests; `tests/e2e/` runs GeoServer Cloud, or standard GeoServer, in Docker for the e2e tests |
+
+## Testing
+
+CI runs on every pull request and push to `main`: shellcheck and a schema check of the Kubernetes example (lint), the unit tests (test), the image build (configops-image), and the e2e tests against GeoServer Cloud in Docker, with its gateway, REST, WMS, WFS, WCS and tile cache services (e2e-cloud), and against standard GeoServer with the same extensions (e2e-standard).
+
+Locally (with `docker`, `curl`, and GDAL's command-line tools, which some e2e tests use to make files):
+
+```bash
+python3 -m venv .venv && .venv/bin/pip install -e '.[dev]'
+.venv/bin/pytest                      # e2e tests skip without their services
+
+tests/e2e/up.sh                       # GeoServer Cloud; about 3 GB of RAM
+E2E_GEOSERVER_URL=http://localhost:59090/geoserver-cloud/rest E2E_SERVICES_URL=http://localhost:59090/geoserver-cloud \
+  E2E_DB_PORT=55433 E2E_DATA_DIR=tests/e2e/data E2E_LDAP_URL=ldap://ldap:389 \
+  E2E_REST_CONTAINER=gscloud-e2e-rest-1 E2E_WMS_CONTAINER=gscloud-e2e-wms-1 .venv/bin/pytest -m e2e
+tests/e2e/up.sh down
+
+tests/e2e/up.sh standard              # standard GeoServer instead (downloads its extensions once)
+E2E_GEOSERVER_URL=http://localhost:59091/geoserver/rest E2E_SERVICES_URL=http://localhost:59091/geoserver \
+  E2E_GWC_URL=http://localhost:59091/geoserver/gwc/rest E2E_DB_PORT=55433 E2E_DB_HOST_IN_GEOSERVER=db \
+  E2E_DATA_DIR=tests/e2e/data E2E_LDAP_URL=ldap://ldap:389 E2E_REST_CONTAINER=gscloud-e2e-geoserver-1 .venv/bin/pytest -m e2e
+tests/e2e/up.sh down
+```
+
+## Releasing
+
+Set the version in `pyproject.toml` and `configops/__init__.py`, merge, then push the tag `v<version>`: the Release workflow publishes the package to PyPI (`geoserver-configops`), the image to `ghcr.io/yibbzz/geoserver-configops`, and a GitHub release.
+
+## Licence and data attribution
+
+MIT (see `LICENSE`); see `NOTICE.md` for credits.
+
+Data you publish keeps its own licence: give a layer the attribution it asks for (`attribution:`). OS OpenData is released under the [Open Government Licence v3.0](https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/) and needs "Contains OS data © Crown copyright and database right &lt;year&gt;", as the examples' OS layers say.