geobatch 0.1.0__tar.gz

geobatch-0.1.0/CHANGELOG.md

@@ -0,0 +1,34 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+## [Unreleased]
+
+## [0.1.0] - 2026-11-09
+### Added
+- `Geocoder` batch orchestrator: de-duplication of normalised queries, thread pool with
+  per-provider token-bucket rate limits and concurrency limits, provider fallback chain
+  (on errors and, optionally, on "not found"), circuit breaker after consecutive failures,
+  `BatchInterrupted` with the partial result on Ctrl+C.
+- Persistent SQLite cache keyed by provider endpoint, query kind, normalised query and
+  options; caches found and not-found answers, never errors; resume by re-running;
+  per-provider TTL and "do not store" rules.
+- HTTP transport on `urllib` with timeouts, exponential backoff with jitter, `Retry-After`
+  (seconds or HTTP date) shared across threads, `max_retry_after` guard, and redaction of
+  API keys from every message.
+- Providers: Nominatim (OSMF usage policy enforced on the public instance: identifying
+  User-Agent required, 1 req/s, single thread, cache required, 5000 uncached lookups per
+  run, policy link printed), Photon (fair-use limits on the public server), Pelias,
+  generic URL-template provider, and API-key adapters for OpenCage, Geoapify, Mapbox
+  (temporary results not cached) and Google (30-day cache TTL).
+- Structured queries (street/city/county/state/postcode/country), country and bbox
+  biasing, language, address normalisation.
+- Structured results (lat, lon, display name, normalised components, confidence,
+  importance, bbox, precision, raw JSON) and quality flags: `low_confidence`,
+  `country_mismatch`, `outside_bbox`, `duplicate_coordinates`, `region_centroid`,
+  `fallback_provider`.
+- I/O: CSV and Excel CSV (BOM, delimiter sniffing) in; CSV, GeoJSON and a
+  standard-library GeoPackage writer out; stdlib progress bar; run summary (text/JSON).
+- CLI: `geobatch geocode`, `geobatch reverse`, `geobatch cache stats|clear`,
+  `geobatch providers`, `geobatch flags`.
+- Optional pandas helpers: `geocode_dataframe`, `reverse_dataframe`.

geobatch-0.1.0/CITATION.cff

@@ -0,0 +1,20 @@
+cff-version: 1.2.0
+message: "If you use this software, please cite it as below."
+authors:
+  - family-names: "Eminoğlu"
+    given-names: "Yusuf"
+    email: "yusufeminoglu@gmail.com"
+    orcid: "https://orcid.org/0009-0005-6000-2934"
+title: "geobatch"
+version: "0.1.0"
+date-released: "2026-11-09"
+license: "MIT"
+repository-code: "https://github.com/YusufEminoglu/geobatch_sdk"
+url: "https://pypi.org/project/geobatch/"
+abstract: "Polite, resumable batch geocoding and reverse geocoding of spreadsheets over Nominatim, Photon, Pelias and API-key providers, with a persistent SQLite cache, token-bucket rate limiting, Retry-After aware retries, provider fallback and quality flags."
+keywords:
+  - geocoding
+  - reverse geocoding
+  - nominatim
+  - openstreetmap
+  - gis

geobatch-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Yusuf Eminoğlu
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

geobatch-0.1.0/MANIFEST.in

@@ -0,0 +1,9 @@
+include LICENSE
+include README.md
+include CHANGELOG.md
+include CITATION.cff
+include pyproject.toml
+include src/geobatch/py.typed
+recursive-include src/geobatch *.py
+recursive-include tests *.py
+recursive-include examples *.py *.csv

geobatch-0.1.0/PKG-INFO

@@ -0,0 +1,338 @@
+Metadata-Version: 2.4
+Name: geobatch
+Version: 0.1.0
+Summary: Polite, resumable batch geocoding and reverse geocoding of spreadsheets over Nominatim, Photon, Pelias and API-key providers, with caching, rate limiting, fallback and quality flags.
+Author-email: Yusuf Eminoğlu <yusufeminoglu@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/YusufEminoglu/geobatch_sdk
+Project-URL: Repository, https://github.com/YusufEminoglu/geobatch_sdk
+Project-URL: Issues, https://github.com/YusufEminoglu/geobatch_sdk/issues
+Project-URL: Changelog, https://github.com/YusufEminoglu/geobatch_sdk/blob/main/CHANGELOG.md
+Keywords: geocoding,reverse-geocoding,batch-geocoding,nominatim,photon,pelias,openstreetmap,address,csv,geojson,geopackage,gis,rate-limiting,cache
+Classifier: Development Status :: 4 - Beta
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Operating System :: OS Independent
+Classifier: Environment :: Console
+Classifier: Topic :: Scientific/Engineering :: GIS
+Classifier: Topic :: Utilities
+Classifier: Intended Audience :: Science/Research
+Classifier: Intended Audience :: Developers
+Classifier: Typing :: Typed
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: pandas
+Requires-Dist: pandas>=1.3.0; extra == "pandas"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
+Requires-Dist: ruff>=0.1.0; extra == "dev"
+Requires-Dist: mypy>=1.0.0; extra == "dev"
+Dynamic: license-file
+
+# geobatch
+
+**Polite, resumable batch geocoding for spreadsheets.**
+
+`geobatch` turns a CSV of thousands of addresses into coordinates (or a CSV of
+coordinates into addresses) reliably and without upsetting the services you
+depend on. It runs on the Python standard library alone and puts several
+providers behind one interface:
+
+- **Nominatim** (public or self-hosted). The OpenStreetMap Foundation usage policy is enforced on the public instance.
+- **Photon** (komoot public demo or self-hosted).
+- **Pelias** (self-hosted or hosted).
+- **Custom HTTP template** for any JSON geocoding API.
+- Thin adapters for API-key services: **OpenCage**, **Geoapify**, **Mapbox** and **Google**. Keys are read only from environment variables and are never logged.
+
+Every lookup goes through a persistent SQLite cache, per-provider token-bucket
+rate limiting, retries that honour `Retry-After`, a fallback chain and
+quality checks. A batch that stops halfway can be resumed: run the same
+command again.
+
+```bash
+pip install geobatch            # stdlib only
+pip install "geobatch[pandas]"  # + DataFrame helpers
+```
+
+Requires Python 3.9 or newer.
+
+---
+
+## Quick start (command line)
+
+```bash
+geobatch geocode addresses.csv --address-col address \
+    --provider nominatim \
+    --user-agent "MyProject/1.0 (https://my-project.org; gis-team@my-project.org)" \
+    -o geocoded.csv
+```
+
+Illustrative output (abridged):
+
+```
+geobatch: Nominatim usage policy applies: https://operations.osmfoundation.org/policies/nominatim/ (1 req/s, results cached, at most 5000 new lookups per run)
+geocoding [##############----------------] 1412/3000  47.1% cached=388 failed=3 elapsed=17:04 eta=19:10
+...
+geobatch forward summary
+  rows            3000
+  ok              2941
+  not found       52
+  errors          3
+  skipped (empty) 4
+  unique lookups  2710
+  cache hits      388
+  HTTP requests   2325 (retries 3)
+  by provider     nominatim=2941
+  by precision    locality=61, rooftop=2402, street=478
+  quality flags   country_mismatch=4, duplicate_coordinates=18, region_centroid=61
+```
+
+The output keeps every input column and adds `geobatch_status`, `geobatch_lat`,
+`geobatch_lon`, `geobatch_display_name`, `geobatch_precision`,
+`geobatch_confidence`, `geobatch_importance`, `geobatch_country_code`,
+`geobatch_provider`, `geobatch_flags`, `geobatch_from_cache` and
+`geobatch_error`.
+
+More examples:
+
+```bash
+# Structured columns, Germany only, fallback to Photon, GeoJSON output
+geobatch geocode customers.csv --street-col street --postcode-col zip --city-col city \
+    --country-codes de --provider nominatim,photon \
+    --user-agent "MyProject/1.0 (https://my-project.org)" -o customers.geojson
+
+# Self-hosted Nominatim: no public-policy limits, so raise the rate and parallelism
+geobatch geocode big.csv --address-col addr --url nominatim=http://localhost:8080 \
+    --rate 50 --concurrency 8 -o big.gpkg
+
+# Reverse geocoding
+geobatch reverse points.csv --lat-col lat --lon-col lon \
+    --user-agent "MyProject/1.0 (https://my-project.org)" -o points_addresses.csv
+
+# A commercial provider (key from the environment, never from the command line)
+export OPENCAGE_API_KEY=...
+geobatch geocode addresses.csv --address-col address --provider opencage -o out.csv
+
+# Cache maintenance
+geobatch cache stats
+geobatch cache clear --provider photon
+geobatch cache clear --expired
+
+# Provider terms and key variables, quality flag meanings
+geobatch providers
+geobatch flags
+```
+
+Input files: UTF-8 CSV, including the BOM that Excel writes. The delimiter
+(`,` `;` tab `|`) is detected automatically, so semicolon "CSV" exports from
+European Excel work as they are. `--excel` writes the output with a BOM so
+Excel shows accented characters correctly. Output format follows the file
+extension: `.csv`, `.tsv`, `.geojson` or `.gpkg`. GeoPackage is written with
+the standard-library `sqlite3` as an OGC GeoPackage 1.3 point layer in EPSG:4326,
+so GDAL is not needed. QGIS and GDAL read it directly.
+
+Exit codes: `0` done, `2` usage or configuration error (including policy
+refusals), `3` aborted after too many consecutive failures, `130` interrupted
+(a partial output is still written).
+
+---
+
+## Responsible use
+
+Free geocoders are run by volunteers and small teams on donated hardware. A
+careless batch job can get your IP blocked, and it also costs everyone else
+capacity. geobatch builds the rules in rather than leaving them to the
+documentation.
+
+| Provider | Terms / policy | What geobatch enforces |
+|---|---|---|
+| Nominatim (public `nominatim.openstreetmap.org`) | [OSMF Nominatim usage policy](https://operations.osmfoundation.org/policies/nominatim/), [OSM copyright & ODbL](https://www.openstreetmap.org/copyright) | Refuses to run without an identifying `user_agent=` / `--user-agent`. Stock library agents, geobatch's own default agent and placeholders such as `example.org` are rejected, and geobatch never fills in an e-mail address for you. Max **1 request/second**, **single thread**, cache required, at most **5000 uncached lookups per run** ("smaller one-time bulk tasks" only). The policy link is printed on every run, and the policy cannot be switched off for this host. |
+| Photon (public `photon.komoot.io`) | [photon.komoot.io](https://photon.komoot.io/): fair use, heavy use is throttled; [source](https://github.com/komoot/photon) | 1 request/second, single thread, cache required (a conservative choice by geobatch). |
+| Pelias | [documentation](https://github.com/pelias/documentation); hosted services such as [geocode.earth](https://geocode.earth) have their own terms | Optional `PELIAS_API_KEY`, redacted from logs. |
+| OpenCage | [terms](https://opencagedata.com/terms) | Key from `OPENCAGE_API_KEY`; default 1 req/s (free-trial limit). |
+| Geoapify | [terms](https://www.geoapify.com/terms-and-conditions/) | Key from `GEOAPIFY_API_KEY`; default 5 req/s. Attribution is required by Geoapify. |
+| Mapbox | [terms of service](https://www.mapbox.com/legal/tos) | Token from `MAPBOX_ACCESS_TOKEN`. Temporary geocoding results are **not cached**; pass `permanent=True` / `--mapbox-permanent` only if your account includes permanent geocoding. |
+| Google Geocoding | [Maps Platform terms](https://cloud.google.com/maps-platform/terms), [geocoding policies](https://developers.google.com/maps/documentation/geocoding/policies) | Key from `GOOGLE_MAPS_API_KEY`. Cache entries **expire after 30 days**. Google requires results to be used with a Google map. |
+
+More rules that apply to every provider:
+
+- **Keys** are read from environment variables only and never accepted as
+  arguments. They are not part of cache keys, and they are replaced by `***` in
+  every error message.
+- **Retries** use exponential backoff with jitter, and a server's
+  `Retry-After` always wins. A 429 pauses the provider's token bucket for all
+  threads, not just the thread that received it. If a server asks you to wait
+  more than `max_retry_after` (300 s by default; usually a used-up daily
+  quota), geobatch gives up on that lookup instead of hammering or blocking
+  for hours.
+- **Circuit breaker:** after 25 consecutive failed lookups the batch stops,
+  because the service is down or blocking you.
+- **De-duplication:** identical addresses (after normalisation) are looked up
+  once.
+- **Attribution:** OpenStreetMap-based results (Nominatim, Photon, most Pelias
+  installations) are © OpenStreetMap contributors under the ODbL. Credit them
+  wherever you publish the data.
+
+For large or recurring jobs, run your own Nominatim, Photon or Pelias, or use a
+commercial provider. Self-hosted instances have no built-in limits in
+geobatch. Set `--rate` and `--concurrency` to what your server can handle.
+
+---
+
+## Python API
+
+```python
+from geobatch import Geocoder, Nominatim, Photon, Query, SearchOptions, QualityConfig
+
+ua = "MyProject/1.0 (https://my-project.org)"
+geocoder = Geocoder(
+    [Nominatim(user_agent=ua), Photon(user_agent=ua)],   # fallback chain
+    cache="geocache.sqlite",                             # default: per-user cache dir
+    options=SearchOptions(country_codes=("de", "at"), language="en"),
+    quality=QualityConfig(min_confidence=0.5, expected_countries=("de", "at")),
+)
+
+result = geocoder.geocode_many(
+    [
+        "Pariser Platz 1, 10117 Berlin",
+        Query(street="Stephansplatz 3", postcode="1010", city="Wien", country="Austria"),
+        {"city": "Hamburg"},
+    ],
+    progress=True,
+)
+for rec in result:
+    if rec.ok:
+        r = rec.result
+        print(r.lat, r.lon, r.display_name, r.precision, r.confidence, rec.flags)
+    else:
+        print(rec.status, rec.error)
+print(result.summary.format())
+
+# one-offs
+rec = geocoder.geocode("Brandenburger Tor, Berlin")
+rec = geocoder.reverse(52.5163, 13.3777)
+```
+
+`GeocodeResult` fields: `provider`, `lat`, `lon`, `display_name`,
+`components` (normalised keys such as `house_number`, `road`, `postcode`,
+`city`, `state`, `country`, `country_code`), `confidence` (0..1), `importance`,
+`bbox` `(min_lon, min_lat, max_lon, max_lat)`, `precision` (`rooftop`,
+`interpolated`, `street`, `postcode`, `locality`, `region`, `country`,
+`unknown`), `result_type` and `raw` (the provider's original JSON). If a
+provider gives no confidence score (Nominatim, Photon), `confidence` is
+derived from `precision`.
+
+Reading and writing files:
+
+```python
+from geobatch import read_table, build_queries, write_output
+
+columns, rows = read_table("addresses.csv")
+queries = build_queries(columns, rows, street_col="street", city_col="city", postcode_col="zip")
+result = geocoder.geocode_many(queries, rows=rows)
+write_output(result.records, "addresses.gpkg")
+```
+
+With pandas (`pip install "geobatch[pandas]"`):
+
+```python
+from geobatch.pandas_ext import geocode_dataframe, reverse_dataframe
+
+df = geocode_dataframe(df, geocoder, address_col="address", expected_country_col="iso2")
+df.attrs["geobatch_summary"]
+```
+
+### Custom HTTP template provider
+
+Any JSON API can be described by URL templates and dotted JSON paths:
+
+```python
+from geobatch import TemplateProvider
+
+inhouse = TemplateProvider(
+    "https://geo.internal/find?address={query}&token={env:GEO_TOKEN}",
+    reverse_url="https://geo.internal/reverse?lat={lat}&lon={lon}&token={env:GEO_TOKEN}",
+    results_path="data.matches", lat_field="y", lon_field="x",
+    display_field="label", confidence_field="score", confidence_scale=0.01,
+    name="inhouse", rate=10, concurrency=4,
+)
+```
+
+The available placeholders are `{query}`, `{street}`, `{city}`, `{county}`,
+`{state}`, `{postcode}`, `{country}`, `{country_codes}`, `{language}`,
+`{limit}`, `{lat}`, `{lon}` and `{env:NAME}`. A parameter that uses `{env:...}`
+is redacted from logs and kept out of the cache key. GeoJSON
+FeatureCollections are recognised automatically. On the command line, use
+`--provider template --template-url ...`.
+
+---
+
+## How it works
+
+**Normalisation.** Before lookup and before computing cache keys, addresses
+are cleaned: Unicode NFKC (full-width digits, non-breaking spaces), control
+characters, collapsed whitespace, surrounding quotes, doubled or trailing
+separators, and spreadsheet null tokens (`NaN`, `N/A`, `-`) become empty.
+Wording is never rewritten. Cache keys are case-insensitive.
+
+**Cache and resume.** The SQLite cache (WAL mode, thread-safe) is keyed by
+provider and endpoint, query kind, normalised query and search options. It
+stores found results and "not found" answers, and each one is committed as
+soon as it arrives. Network errors are never cached. If a batch is
+interrupted (Ctrl+C, crash, laptop lid), run the same command again: finished
+lookups come from the cache and only the rest goes to the network. In Python,
+`BatchInterrupted.result` holds the partial result. The default location is
+the per-user cache directory (`%LOCALAPPDATA%\geobatch`, `~/Library/Caches/geobatch`
+or `~/.cache/geobatch`). Override it with `--cache` or `GEOBATCH_CACHE`.
+
+**Rate limiting and concurrency.** Each provider has a token bucket
+(`rate` requests per second, no initial burst) and a concurrency limit, shared
+by all worker threads.
+
+**Fallback.** Providers are tried in order. The next one is asked when a
+provider fails, and also when it finds nothing (turn that off with
+`fallback_on_not_found=False` / `--no-fallback-on-not-found`).
+
+**Biasing.** `country_codes`, `bbox` and `bounded` are translated into each
+provider's own parameters, for example Nominatim
+`countrycodes`/`viewbox`/`bounded`, Pelias `boundary.*`/`focus.point`,
+Geoapify `filter`/`bias` and Google `components`/`bounds`. Structured queries
+use native structured endpoints where they exist (Nominatim, Pelias, Geoapify,
+Mapbox) and are joined into one line elsewhere.
+
+### Quality flags
+
+| Flag | Meaning |
+|---|---|
+| `low_confidence` | confidence below `min_confidence` (default 0.5) |
+| `country_mismatch` | result is outside the expected countries (`--country-codes` or a per-row `--expected-country-col`) |
+| `outside_bbox` | result is outside the expected bounding box (`--bbox --bounded`) |
+| `duplicate_coordinates` | the same point was returned for *different* addresses, a typical sign of a fallback to a street or town centroid |
+| `region_centroid` | a postcode, city or region centroid was returned for a query that had street-level detail, or a region or country centroid was returned for any query |
+| `fallback_provider` | answered by a provider other than the first in the chain |
+
+---
+
+## Development
+
+```bash
+pip install -e ".[dev,pandas]"
+pytest --cov=geobatch
+ruff check . && ruff format --check src tests && mypy src
+```
+
+The test suite never touches the real network. It runs a local stdlib HTTP
+server (`tests/mockserver.py`) that imitates every provider's JSON, including
+429/503 with `Retry-After`, malformed JSON, timeouts, rate-limit timing,
+cache hits, resume and fallback.
+
+## License
+
+MIT © 2026 Yusuf Eminoğlu. See [LICENSE](LICENSE). To cite geobatch, see
+[CITATION.cff](CITATION.cff).