osmsg 1.0.2__tar.gz → 1.1.0__tar.gz

{osmsg-1.0.2 → osmsg-1.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: osmsg
-Version: 1.0.2
+Version: 1.1.0
 Summary: OpenStreetMap Stats Generator: Commandline
 Keywords: osm,stats,commandline,openstreetmap
 Author: Kshitij Raj Sharma
@@ -41,11 +41,12 @@ Description-Content-Type: text/markdown
 [![uv](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json)](https://github.com/astral-sh/uv)
 [![Container](https://img.shields.io/badge/ghcr.io-osgeonepal%2Fosmsg-2496ED?logo=docker)](https://github.com/osgeonepal/osmsg/pkgs/container/osmsg)
 
-**OpenStreetMap Stats Generator.** A tiny CLI (and Python library) that turns OSM history into per-user counts of nodes, ways, and relations created, modified, or deleted, written to parquet, csv, json, markdown, or Postgres.
+**OpenStreetMap Stats Generator.** A tiny CLI (and Python library) that turns OSM history into per-user counts
+of nodes, ways, and relations created, modified, or deleted, written to parquet, csv, json, markdown, or Postgres.
 
 A Project of [OSGeo Nepal](https://osgeonepal.org).
 
-## What you get
+## Features
 
 - Per-user create/modify/delete counts over any time window.
 - Tag and hashtag breakdowns (e.g. `building`, `#hotosm`).
@@ -58,11 +59,15 @@ A Project of [OSGeo Nepal](https://osgeonepal.org).
 Pick the one that fits how you work.
 
 ```bash
+uvx --from osmsg osmsg --last hour       # zero-install, one-shot run
 pip install osmsg                        # into your project
 uv tool install osmsg                    # standalone CLI
 docker run --rm -v "$PWD:/work" -w /work ghcr.io/osgeonepal/osmsg:latest --last hour
 ```
 
+`uvx` can run osmsg in a throwaway environment , no install, no virtualenv to manage. Works
+with any flag combination, e.g. `uvx --from osmsg osmsg --last hour --tags building --summary -f parquet -f markdown`.
+
 ## Quick start
 
 ```bash
@@ -81,7 +86,8 @@ That's it. A `stats.duckdb` and a `stats.parquet` show up in your current folder
 osmsg --country nepal --last day
 ```
 
-`--country` resolves through Geofabrik and needs an OSM account. Set `OSM_USERNAME` and `OSM_PASSWORD` in your shell or a `.env` file:
+`--country` resolves through Geofabrik and needs an OSM account. Set `OSM_USERNAME` and `OSM_PASSWORD`
+in your shell or a `.env` file:
 
 ```bash
 export OSM_USERNAME=you
@@ -115,7 +121,24 @@ duckdb stats.duckdb -c "SELECT username, SUM(nodes_created) AS n
 
 Same schema in DuckDB and Postgres: `users`, `changesets`, `changeset_stats`, `state`.
 
-### 5. Use it as a library
+### 5. Run the API
+
+Push stats into Postgres, then start the Litestar API:
+
+```bash
+osmsg --last day --format psql --psql-dsn "postgresql://user:pass@localhost/osmsg"
+litestar --app api.app:app run --host 0.0.0.0 --port 8000
+```
+
+```text
+GET /health
+GET /api/v1/user-stats?start=2026-05-01T00:00:00Z&end=2026-05-02T00:00:00Z
+GET /docs
+```
+
+For self-hosting with Docker Compose and systemd, see [docs/infra.md](./docs/infra.md).
+
+### 6. Use it as a library
 
 ```python
 from datetime import datetime, UTC
@@ -132,7 +155,8 @@ print(result["files"]["parquet"])
 
 Same pipeline as the CLI.
 
-### 6. Long flag lists? Use a config
+### 7. Long flag lists? Use a config
+
 
 ```bash
 osmsg --config nepal.yaml
@@ -142,12 +166,37 @@ Any flag works as a YAML key. See [docs/Manual.md](./docs/Manual.md) for the ful
 
 ## Output formats
 
-Every run writes `stats.duckdb` (or `<--name>.duckdb`) plus the formats you ask for via `-f parquet|csv|json|markdown|psql`. Parquet is the default. Open it with duckdb, polars, pandas, anything.
+Every run writes `stats.duckdb` (or `<--name>.duckdb`) plus the formats you ask for via
+`-f parquet|csv|json|markdown|psql`. Parquet is the default. Open it with duckdb, polars, pandas, anything.
+
+## Configuration
+
+Every meaningful flag has a matching `OSMSG_*` env var so the CLI, a `.env` file, and a
+docker-compose `environment:` block all reach the same setting. CLI flag wins over env var.
+
+| CLI flag | Env var | Default | Notes |
+| --- | --- | --- | --- |
+| `--name` | `OSMSG_NAME` | `stats` | Output basename; sets `<name>.duckdb`. |
+| `--country` | `OSMSG_COUNTRY` | unset | Geofabrik region id(s). Comma-separated when set via env. |
+| `--boundary` | `OSMSG_BOUNDARY` | unset | GeoJSON path or inline GeoJSON. |
+| `--url` | `OSMSG_URL` | `minute` | `minute`/`hour`/`day` shortcut or full URL. Comma-separated when set via env. |
+| `--workers` | `OSMSG_WORKERS` | cpu count | Parallel workers. |
+| `--cache-dir` | `OSMSG_CACHE_DIR` | platform cache | Where downloaded OSM files are kept across runs. |
+| `--output-dir` | `OSMSG_OUTPUT_DIR` | `.` | Where `<name>.duckdb` and exports are written. |
+| `--format` / `-f` | `OSMSG_FORMAT` | `parquet` | Repeat for multiple. Comma-separated when set via env. |
+| `--psql-dsn` | `OSMSG_PSQL_DSN` | unset | libpq DSN for `-f psql`. |
+| `--changeset-pad-hours` | `OSMSG_CHANGESET_PAD_HOURS` | `1` | See below. |
+| (auto-bootstrap on `--update`) | `OSMSG_BOOTSTRAP` | `hour` | `hour`, `day`, or `week`. Used when `--update` runs against an empty DB. |
+| (auto-bootstrap on `--update`) | `OSMSG_BOOTSTRAP_DAYS` | unset | Integer N; overrides `OSMSG_BOOTSTRAP`. |
+| OSM credentials (Geofabrik) | `OSM_USERNAME`, `OSM_PASSWORD` | unset | Required only when a Geofabrik URL is in use. |
+
+A `.env` file at the working directory is loaded automatically.
 
 ## Documentation
 
 - [Installation](./docs/Installation.md)
 - [Manual](./docs/Manual.md) (every flag, with examples)
+- [Self-hosting / Docker Compose](./docs/infra.md)
 - [Version control / release notes](./docs/Version_control.md)
 
 ## Contributing
@@ -162,7 +211,8 @@ uv run pre-commit install
 uv run pytest -m "not network"
 ```
 
-Please read [CONTRIBUTING.md](./CONTRIBUTING.md) and the [Code of Conduct](./CODE_OF_CONDUCT.md) before opening a PR. Use [Conventional Commits](https://www.conventionalcommits.org/) (`cz commit`).
+Please read [CONTRIBUTING.md](./CONTRIBUTING.md) and the [Code of Conduct](./CODE_OF_CONDUCT.md) before opening a PR.
+Use [Conventional Commits](https://www.conventionalcommits.org/) (`cz commit`).
 
 ## License
 

{osmsg-1.0.2 → osmsg-1.1.0}/README.md

@@ -9,11 +9,12 @@
 [![uv](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json)](https://github.com/astral-sh/uv)
 [![Container](https://img.shields.io/badge/ghcr.io-osgeonepal%2Fosmsg-2496ED?logo=docker)](https://github.com/osgeonepal/osmsg/pkgs/container/osmsg)
 
-**OpenStreetMap Stats Generator.** A tiny CLI (and Python library) that turns OSM history into per-user counts of nodes, ways, and relations created, modified, or deleted, written to parquet, csv, json, markdown, or Postgres.
+**OpenStreetMap Stats Generator.** A tiny CLI (and Python library) that turns OSM history into per-user counts
+of nodes, ways, and relations created, modified, or deleted, written to parquet, csv, json, markdown, or Postgres.
 
 A Project of [OSGeo Nepal](https://osgeonepal.org).
 
-## What you get
+## Features
 
 - Per-user create/modify/delete counts over any time window.
 - Tag and hashtag breakdowns (e.g. `building`, `#hotosm`).
@@ -26,11 +27,15 @@ A Project of [OSGeo Nepal](https://osgeonepal.org).
 Pick the one that fits how you work.
 
 ```bash
+uvx --from osmsg osmsg --last hour       # zero-install, one-shot run
 pip install osmsg                        # into your project
 uv tool install osmsg                    # standalone CLI
 docker run --rm -v "$PWD:/work" -w /work ghcr.io/osgeonepal/osmsg:latest --last hour
 ```
 
+`uvx` can run osmsg in a throwaway environment , no install, no virtualenv to manage. Works
+with any flag combination, e.g. `uvx --from osmsg osmsg --last hour --tags building --summary -f parquet -f markdown`.
+
 ## Quick start
 
 ```bash
@@ -49,7 +54,8 @@ That's it. A `stats.duckdb` and a `stats.parquet` show up in your current folder
 osmsg --country nepal --last day
 ```
 
-`--country` resolves through Geofabrik and needs an OSM account. Set `OSM_USERNAME` and `OSM_PASSWORD` in your shell or a `.env` file:
+`--country` resolves through Geofabrik and needs an OSM account. Set `OSM_USERNAME` and `OSM_PASSWORD`
+in your shell or a `.env` file:
 
 ```bash
 export OSM_USERNAME=you
@@ -83,7 +89,24 @@ duckdb stats.duckdb -c "SELECT username, SUM(nodes_created) AS n
 
 Same schema in DuckDB and Postgres: `users`, `changesets`, `changeset_stats`, `state`.
 
-### 5. Use it as a library
+### 5. Run the API
+
+Push stats into Postgres, then start the Litestar API:
+
+```bash
+osmsg --last day --format psql --psql-dsn "postgresql://user:pass@localhost/osmsg"
+litestar --app api.app:app run --host 0.0.0.0 --port 8000
+```
+
+```text
+GET /health
+GET /api/v1/user-stats?start=2026-05-01T00:00:00Z&end=2026-05-02T00:00:00Z
+GET /docs
+```
+
+For self-hosting with Docker Compose and systemd, see [docs/infra.md](./docs/infra.md).
+
+### 6. Use it as a library
 
 ```python
 from datetime import datetime, UTC
@@ -100,7 +123,8 @@ print(result["files"]["parquet"])
 
 Same pipeline as the CLI.
 
-### 6. Long flag lists? Use a config
+### 7. Long flag lists? Use a config
+
 
 ```bash
 osmsg --config nepal.yaml
@@ -110,12 +134,37 @@ Any flag works as a YAML key. See [docs/Manual.md](./docs/Manual.md) for the ful
 
 ## Output formats
 
-Every run writes `stats.duckdb` (or `<--name>.duckdb`) plus the formats you ask for via `-f parquet|csv|json|markdown|psql`. Parquet is the default. Open it with duckdb, polars, pandas, anything.
+Every run writes `stats.duckdb` (or `<--name>.duckdb`) plus the formats you ask for via
+`-f parquet|csv|json|markdown|psql`. Parquet is the default. Open it with duckdb, polars, pandas, anything.
+
+## Configuration
+
+Every meaningful flag has a matching `OSMSG_*` env var so the CLI, a `.env` file, and a
+docker-compose `environment:` block all reach the same setting. CLI flag wins over env var.
+
+| CLI flag | Env var | Default | Notes |
+| --- | --- | --- | --- |
+| `--name` | `OSMSG_NAME` | `stats` | Output basename; sets `<name>.duckdb`. |
+| `--country` | `OSMSG_COUNTRY` | unset | Geofabrik region id(s). Comma-separated when set via env. |
+| `--boundary` | `OSMSG_BOUNDARY` | unset | GeoJSON path or inline GeoJSON. |
+| `--url` | `OSMSG_URL` | `minute` | `minute`/`hour`/`day` shortcut or full URL. Comma-separated when set via env. |
+| `--workers` | `OSMSG_WORKERS` | cpu count | Parallel workers. |
+| `--cache-dir` | `OSMSG_CACHE_DIR` | platform cache | Where downloaded OSM files are kept across runs. |
+| `--output-dir` | `OSMSG_OUTPUT_DIR` | `.` | Where `<name>.duckdb` and exports are written. |
+| `--format` / `-f` | `OSMSG_FORMAT` | `parquet` | Repeat for multiple. Comma-separated when set via env. |
+| `--psql-dsn` | `OSMSG_PSQL_DSN` | unset | libpq DSN for `-f psql`. |
+| `--changeset-pad-hours` | `OSMSG_CHANGESET_PAD_HOURS` | `1` | See below. |
+| (auto-bootstrap on `--update`) | `OSMSG_BOOTSTRAP` | `hour` | `hour`, `day`, or `week`. Used when `--update` runs against an empty DB. |
+| (auto-bootstrap on `--update`) | `OSMSG_BOOTSTRAP_DAYS` | unset | Integer N; overrides `OSMSG_BOOTSTRAP`. |
+| OSM credentials (Geofabrik) | `OSM_USERNAME`, `OSM_PASSWORD` | unset | Required only when a Geofabrik URL is in use. |
+
+A `.env` file at the working directory is loaded automatically.
 
 ## Documentation
 
 - [Installation](./docs/Installation.md)
 - [Manual](./docs/Manual.md) (every flag, with examples)
+- [Self-hosting / Docker Compose](./docs/infra.md)
 - [Version control / release notes](./docs/Version_control.md)
 
 ## Contributing
@@ -130,7 +179,8 @@ uv run pre-commit install
 uv run pytest -m "not network"
 ```
 
-Please read [CONTRIBUTING.md](./CONTRIBUTING.md) and the [Code of Conduct](./CODE_OF_CONDUCT.md) before opening a PR. Use [Conventional Commits](https://www.conventionalcommits.org/) (`cz commit`).
+Please read [CONTRIBUTING.md](./CONTRIBUTING.md) and the [Code of Conduct](./CODE_OF_CONDUCT.md) before opening a PR.
+Use [Conventional Commits](https://www.conventionalcommits.org/) (`cz commit`).
 
 ## License
 

osmsg-1.1.0/osmsg/__version__.py

@@ -0,0 +1 @@
+__version__ = "1.1.0"

osmsg-1.1.0/osmsg/_tick.py

@@ -0,0 +1,71 @@
+"""Worker tick: bootstrap on first run, --update thereafter."""
+
+import fcntl
+import os
+import shlex
+import subprocess
+import sys
+from pathlib import Path
+
+from .db import connect, create_tables, get_state
+from .geofabrik import country_update_url
+from .replication import resolve_url
+
+
+def _has_state(db_path: Path, source_url: str) -> bool:
+    if not db_path.exists():
+        return False
+    conn = connect(str(db_path))
+    create_tables(conn)
+    result = get_state(conn, source_url) is not None
+    conn.close()
+    return result
+
+
+def _parse_arg(args: list[str], flag: str) -> str | None:
+    for i, arg in enumerate(args):
+        if arg == flag and i + 1 < len(args):
+            return args[i + 1]
+    return None
+
+
+def main() -> int:
+    extra_args = shlex.split(os.environ.get("OSMSG_EXTRA_ARGS", ""))
+    bootstrap = os.environ.get("OSMSG_BOOTSTRAP", "hour")
+    bootstrap_days = os.environ.get("OSMSG_BOOTSTRAP_DAYS")
+    name = _parse_arg(extra_args, "--name") or "stats"
+    out = Path(_parse_arg(extra_args, "--output-dir") or "/var/lib/osmsg")
+    country = _parse_arg(extra_args, "--country")
+    url = _parse_arg(extra_args, "--url") or "minute"
+
+    out.mkdir(parents=True, exist_ok=True)
+
+    lock_path = out / f"{name}.lock"
+    lock_fd = os.open(str(lock_path), os.O_CREAT | os.O_RDWR, 0o644)
+    try:
+        fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
+    except BlockingIOError:
+        print("[osmsg-tick] previous tick still running, skipping", flush=True)
+        return 0
+
+    source_url = country_update_url(country) if country else resolve_url(url)
+    db_path = out / f"{name}.duckdb"
+
+    extra_set = set(extra_args)
+    cmd = ["osmsg"] + extra_args
+    if not (extra_set & {"--all", "--keys"}):
+        cmd.append("--all")
+
+    if _has_state(db_path, source_url):
+        cmd.append("--update")
+    elif bootstrap_days:
+        cmd.extend(["--days", bootstrap_days])
+    else:
+        cmd.extend(["--last", bootstrap])
+
+    print(f"[osmsg-tick] {' '.join(cmd)}", flush=True)
+    return subprocess.call(cmd)
+
+
+if __name__ == "__main__":
+    sys.exit(main())

osmsg-1.1.0/osmsg/boundary.py

@@ -0,0 +1,37 @@
+"""Boundary GeoJSON parsing."""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any
+
+from shapely.geometry import MultiPolygon, Polygon, shape
+from shapely.geometry.base import BaseGeometry
+
+from .exceptions import UnknownRegionError
+from .geofabrik import country_geometry
+
+
+def load_boundary(input_data: str) -> BaseGeometry:
+    try:
+        payload: Any = json.loads(input_data)
+    except json.JSONDecodeError:
+        path = Path(input_data)
+        if path.is_file():
+            payload = json.loads(path.read_text())
+        else:
+            try:
+                return country_geometry(input_data)
+            except UnknownRegionError:
+                raise ValueError(
+                    f"--boundary {input_data!r} is not valid JSON, a file path, or a known Geofabrik region name."
+                ) from None
+
+    geometry = payload.get("geometry") if "geometry" in payload else payload
+    if not geometry or geometry.get("type") not in ("Polygon", "MultiPolygon"):
+        raise ValueError("Boundary must be a Polygon or MultiPolygon GeoJSON.")
+    geom = shape(geometry)
+    if isinstance(geom, (Polygon, MultiPolygon)):
+        return geom
+    raise ValueError(f"Unexpected geometry type: {type(geom).__name__}")

{osmsg-1.0.2 → osmsg-1.1.0}/osmsg/cli.py

@@ -111,18 +111,29 @@ def main(
         bool | None,
         typer.Option("--version", callback=_version_callback, is_eager=True, help="Print version and exit."),
     ] = None,
-    name: Annotated[str, typer.Option(help="Output basename. Writes <name>.duckdb + selected formats.")] = "stats",
+    name: Annotated[
+        str,
+        typer.Option(envvar="OSMSG_NAME", help="Output basename. Writes <name>.duckdb + selected formats."),
+    ] = "stats",
     start: Annotated[str | None, typer.Option(help="ISO start (UTC). 'YYYY-MM-DD HH:MM:SS'.")] = None,
     end: Annotated[str | None, typer.Option(help="ISO end (UTC). Defaults to now.")] = None,
     last: Annotated[Period | None, typer.Option(help="Convenience: hour|day|week|month|year.")] = None,
     days: Annotated[int | None, typer.Option(help="Last N days (mutually exclusive with --last).")] = None,
     country: Annotated[
         list[str] | None,
-        typer.Option("--country", help="Geofabrik region id(s); resolved live. Requires OSM credentials."),
+        typer.Option(
+            "--country",
+            envvar="OSMSG_COUNTRY",
+            help="Geofabrik region id(s); resolved live. Requires OSM credentials. Comma-separated when set via env.",
+        ),
     ] = None,
     url: Annotated[
         list[str] | None,
-        typer.Option("--url", help="Replication URL(s). Shortcuts: minute, hour, day."),
+        typer.Option(
+            "--url",
+            envvar="OSMSG_URL",
+            help="Replication URL(s). Shortcuts: minute, hour, day. Comma-separated when set via env.",
+        ),
     ] = None,
     hashtags: Annotated[
         list[str] | None,
@@ -134,29 +145,54 @@ def main(
         list[str] | None,
         typer.Option("--users", help="Filter to OSM usernames (case-sensitive, exact match). Repeat for more."),
     ] = None,
-    workers: Annotated[int | None, typer.Option(help="Parallel workers (default: cpu count).")] = None,
+    workers: Annotated[
+        int | None,
+        typer.Option(envvar="OSMSG_WORKERS", help="Parallel workers (default: cpu count)."),
+    ] = None,
     rows: Annotated[
         int | None,
         typer.Option(help="Cap rows shown in the console table. Files always carry the full set."),
     ] = None,
-    boundary: Annotated[str | None, typer.Option(help="Path to GeoJSON or inline geojson string.")] = None,
-    formats: Annotated[list[Format] | None, typer.Option("--format", "-f", help="One or more output formats.")] = None,
+    boundary: Annotated[
+        str | None,
+        typer.Option(
+            envvar="OSMSG_BOUNDARY",
+            help="Boundary filter: Geofabrik region name (e.g. 'nepal'), GeoJSON file path, or inline GeoJSON.",
+        ),
+    ] = None,
+    formats: Annotated[
+        list[Format] | None,
+        typer.Option(
+            "--format",
+            "-f",
+            envvar="OSMSG_FORMAT",
+            help="One or more output formats. Comma-separated when set via env.",
+        ),
+    ] = None,
     summary: Annotated[bool, typer.Option(help="Also write <name>_summary.parquet + summary.md.")] = False,
     changeset: Annotated[bool, typer.Option(hidden=True)] = False,
-    all_tags: Annotated[bool, typer.Option("--all-tags", help="Track every tag key.")] = False,
-    key_value: Annotated[bool, typer.Option("--key-value", help="Store key=value combos. Implies --all-tags.")] = False,
+    all_stats: Annotated[
+        bool,
+        typer.Option(
+            "--all",
+            help="Collect all tag key=value stats and changeset metadata (hashtags, editors).",
+        ),
+    ] = False,
+    keys_only: Annotated[bool, typer.Option("--keys", help="Collect tag key stats only (no value breakdown).")] = False,
     exact_lookup: Annotated[
         bool, typer.Option("--exact-lookup", help="Hashtag whole-word match. Only meaningful with --hashtags.")
     ] = False,
     tm_stats: Annotated[bool, typer.Option("--tm-stats", help="Attach Tasking Manager totals.")] = False,
     update: Annotated[bool, typer.Option(help="Append to existing <name>.duckdb.")] = False,
     cache_dir: Annotated[
-        Path, typer.Option("--cache-dir", help="Cache dir for downloaded OSM files.")
+        Path,
+        typer.Option("--cache-dir", envvar="OSMSG_CACHE_DIR", help="Cache dir for downloaded OSM files."),
     ] = DEFAULT_CACHE_DIR,
     output_dir: Annotated[
         Path,
         typer.Option(
             "--output-dir",
+            envvar="OSMSG_OUTPUT_DIR",
             help="Where to write <name>.duckdb + selected formats. Defaults to current directory.",
         ),
     ] = Path("."),
@@ -175,7 +211,21 @@ def main(
             help="Read OSM password from stdin (one line). Else $OSM_PASSWORD, then prompt.",
         ),
     ] = False,
-    psql_dsn: Annotated[str | None, typer.Option("--psql-dsn", help="libpq DSN for --format psql.")] = None,
+    psql_dsn: Annotated[
+        str | None,
+        typer.Option("--psql-dsn", envvar="OSMSG_PSQL_DSN", help="libpq DSN for --format psql."),
+    ] = None,
+    changeset_pad_hours: Annotated[
+        int,
+        typer.Option(
+            "--changeset-pad-hours",
+            envvar="OSMSG_CHANGESET_PAD_HOURS",
+            help="Backward pad (hours) on first runs of changeset replication. "
+            "Set to 24 to capture long-running open changesets. --update runs skip the pad.",
+            min=0,
+            max=48,
+        ),
+    ] = 1,
 ) -> None:
     """Run osmsg."""
     if formats is None:
@@ -194,13 +244,13 @@ def main(
         end_date=_parse_dt(end),
         countries=country,
         urls=url or ["minute"],
+        url_explicit=url is not None,
         workers=workers,
         additional_tags=tags,
         hashtags=hashtags,
         length_tags=length,
         users_filter=users,
-        all_tags=all_tags or key_value,
-        key_value=key_value,
+        tag_mode="all" if all_stats else ("keys" if keys_only else "none"),
         exact_lookup=exact_lookup,
         changeset=changeset,
         summary=summary,
@@ -214,6 +264,7 @@ def main(
         osm_username=username,
         osm_password=_read_password_stdin() if password_stdin else None,
         psql_dsn=psql_dsn,
+        changeset_pad_hours=changeset_pad_hours,
     )
 
     if last is not None:

osmsg-1.1.0/osmsg/db/duckdb_schema.py

@@ -0,0 +1,41 @@
+# No FKs: DuckDB rejects UPDATE on FK-referenced LIST/GEOMETRY columns, which would block changeset upgrades.
+DUCKDB_SCHEMA = """
+CREATE TABLE IF NOT EXISTS users (
+    uid      BIGINT PRIMARY KEY,
+    username VARCHAR NOT NULL
+);
+CREATE TABLE IF NOT EXISTS changesets (
+    changeset_id BIGINT PRIMARY KEY,
+    uid          BIGINT NOT NULL,
+    created_at   TIMESTAMPTZ,
+    hashtags     VARCHAR[],
+    editor       VARCHAR,
+    geom         GEOMETRY
+);
+CREATE INDEX IF NOT EXISTS idx_changesets_created_at ON changesets(created_at);
+CREATE TABLE IF NOT EXISTS changeset_stats (
+    changeset_id   BIGINT NOT NULL,
+    seq_id         BIGINT NOT NULL,
+    uid            BIGINT NOT NULL,
+    nodes_created  INTEGER DEFAULT 0,
+    nodes_modified INTEGER DEFAULT 0,
+    nodes_deleted  INTEGER DEFAULT 0,
+    ways_created   INTEGER DEFAULT 0,
+    ways_modified  INTEGER DEFAULT 0,
+    ways_deleted   INTEGER DEFAULT 0,
+    rels_created   INTEGER DEFAULT 0,
+    rels_modified  INTEGER DEFAULT 0,
+    rels_deleted   INTEGER DEFAULT 0,
+    poi_created    INTEGER DEFAULT 0,
+    poi_modified   INTEGER DEFAULT 0,
+    tag_stats      JSON,
+    PRIMARY KEY (seq_id, changeset_id)
+);
+CREATE INDEX IF NOT EXISTS idx_changeset_stats_uid ON changeset_stats(uid);
+CREATE TABLE IF NOT EXISTS state (
+    source_url  VARCHAR PRIMARY KEY,
+    last_seq    BIGINT NOT NULL,
+    last_ts     TIMESTAMPTZ NOT NULL,
+    updated_at  TIMESTAMPTZ NOT NULL
+);
+"""

{osmsg-1.0.2 → osmsg-1.1.0}/osmsg/db/ingest.py

@@ -114,14 +114,44 @@ def merge_parquet_files(conn: duckdb.DuckDBPyConnection, parquet_dir: Path, *, c
         if any(parquet_dir.glob("temp_*_users_*.parquet")):
             conn.execute(f"INSERT OR IGNORE INTO users SELECT uid, username FROM read_parquet('{pattern('users')}')")
         if any(parquet_dir.glob("temp_*_changesets_*.parquet")):
+            conn.execute("INSTALL spatial")
+            conn.execute("LOAD spatial")
             conn.execute(
                 f"""
                 INSERT OR IGNORE INTO changesets
                 SELECT changeset_id, uid, created_at, hashtags, editor,
-                       min_lon, min_lat, max_lon, max_lat
+                       CASE WHEN min_lon IS NOT NULL
+                           THEN ST_MakeEnvelope(min_lon, min_lat, max_lon, max_lat)
+                       END
                 FROM read_parquet('{pattern("changesets")}')
                 """
             )
+            # Newer non-NULL wins; dedupe src so multiple emits per window don't trip the PK on UPDATE.
+            conn.execute(
+                f"""
+                UPDATE changesets c
+                SET created_at = COALESCE(src.created_at, c.created_at),
+                    hashtags   = COALESCE(src.hashtags,   c.hashtags),
+                    editor     = COALESCE(src.editor,     c.editor),
+                    geom       = COALESCE(src.geom,       c.geom)
+                FROM (
+                    SELECT DISTINCT ON (changeset_id)
+                           changeset_id, created_at, hashtags, editor,
+                           CASE WHEN min_lon IS NOT NULL
+                               THEN ST_MakeEnvelope(min_lon, min_lat, max_lon, max_lat)
+                           END AS geom
+                    FROM read_parquet('{pattern("changesets")}')
+                    ORDER BY changeset_id,
+                             (min_lon IS NOT NULL) DESC,
+                             (editor IS NOT NULL)  DESC,
+                             (hashtags IS NOT NULL) DESC,
+                             created_at DESC NULLS LAST
+                ) src
+                WHERE c.changeset_id = src.changeset_id
+                  AND (src.created_at IS NOT NULL OR src.hashtags IS NOT NULL
+                       OR src.editor IS NOT NULL OR src.geom IS NOT NULL)
+                """
+            )
         if any(parquet_dir.glob("temp_*_changeset_stats_*.parquet")):
             conn.execute(
                 f"""