lgd-query 0.1.0__tar.gz

lgd_query-0.1.0/MANIFEST.in

@@ -0,0 +1 @@
+include tools/check_dist.py

lgd_query-0.1.0/PKG-INFO

@@ -0,0 +1,227 @@
+Metadata-Version: 2.4
+Name: lgd-query
+Version: 0.1.0
+Summary: Resolve, download and query public Local Government Directory Parquet releases
+Project-URL: Repository, https://github.com/ramSeraph/opendata
+Project-URL: Documentation, https://github.com/ramSeraph/opendata/tree/master/lgd-query
+Project-URL: Issues, https://github.com/ramSeraph/opendata/issues
+Keywords: india,lgd,local-government-directory,parquet,duckdb
+Classifier: Development Status :: 3 - Alpha
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Database
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Provides-Extra: duckdb
+Requires-Dist: duckdb>=2.0.0; extra == "duckdb"
+Provides-Extra: cli
+Requires-Dist: rich>=14; extra == "cli"
+
+# lgd-query
+
+A small Python client for the public [LGD Parquet releases](https://github.com/ramSeraph/opendata/releases/tag/lgd-parquet-latest).
+It has three public APIs: metadata resolution, verified downloading and DuckDB
+access. The CLI uses those same APIs. No GitHub account or API server is needed.
+
+This directory is a standalone distribution. Its wheel contains only
+`lgd_query`; it has no dependency on the repository's scrapers, converters or
+history code. Python 3.11+ is required, matching DuckDB 2's minimum. Metadata and downloads use the standard
+library; SQL access additionally requires DuckDB 2 or newer.
+Rich is an optional dependency for CLI tables.
+
+## Installation
+
+Until the first PyPI release, install from this directory:
+
+```sh
+pip install .                 # Metadata API, download API and their CLI commands
+pip install '.[duckdb]'       # Also include DuckDB querying
+pip install '.[cli,duckdb]'    # All APIs and Rich table output
+```
+
+The distribution name is `lgd-query`; the Python import is `lgd_query`.
+After publication, the corresponding installs are `pip install lgd-query` and
+`pip install 'lgd-query[duckdb]'`.
+
+## 1. Metadata API
+
+```python
+from lgd_query import load_release, load_manifest
+
+release = load_release()
+villages = release.dataset("villages")
+
+print(release.generation, release.published_at, release.immutable_url)
+print(villages.observed_on, villages.rows, villages.status)
+print(villages.parquet.url, villages.parquet.size, villages.parquet.sha256)
+
+# Optional extra request for canonical schema and original-source provenance.
+manifest = load_manifest(villages)
+print(manifest["schema"])
+```
+
+`load_release()` fetches `metadata.json` once. Its `Release` object retains that
+generation, and datasets refer directly to immutable files even if the latest
+index changes. Accessing `release.datasets` or `release.dataset()` makes no
+requests. New datasets can be discovered without updating a hardcoded registry.
+`load_manifest()` verifies the manifest's size/hash and agreement with the index.
+
+`release.datasets` is a read-only mapping including unavailable datasets. Each
+`Dataset` exposes `name`, `available`, `status`, `reason`, `representation`,
+`observed_on`, `uploaded_at`, `rows`, `parquet`, `manifest`, `identity`,
+`coverage` and `metadata`. `parquet` and `manifest` are `Asset` references with
+`url`, `size` and `sha256`. An unavailable dataset has no file references;
+`release.dataset(name)` raises `DatasetUnavailableError` for it. An unknown name
+raises `KeyError`.
+
+“Latest” means the files advertised in one release index. Datasets can have
+different source dates. A failed update can retain a usable older file, with its
+actual date and failure status. Require an exact date when freshness matters:
+
+```python
+villages = release.dataset("villages", observed_on="2026-10-23")
+```
+
+A mismatch raises `ObservationDateError`; this is a freshness check, not a
+historical lookup. Cumulative `changes` and `transactions` reports have
+`observed_on=None`; their `coverage` describes event dates and collection coverage.
+Publication/upload timestamps do not establish an observation date. These files
+contain supplied report rows; omissions do not prove deletions.
+
+To reopen a generation, pass its immutable URL to `load_release(url)`. For an
+offline metadata lookup, save `release.metadata` as JSON and reopen it with
+`Release.from_dict(document)`. `metadata` and `coverage` return detached copies.
+Metadata requests accept `timeout=30` and `retries=2`. Transient HTTP responses,
+including a brief 404 during index replacement, get bounded retries. The reader
+does not silently fall back to `metadata.previous.json`; that URL can be supplied
+explicitly if the preceding generation is wanted.
+
+## 2. Download API
+
+```python
+from lgd_query import download
+
+path = download(villages, directory="./data")
+```
+
+The returned `pathlib.Path` names a complete Parquet file whose size and SHA-256
+match the resolved metadata. Files are named `<dataset>.<sha256>.parquet`, so
+different contents coexist in the cache. Every reuse verifies the cached file.
+A corrupt cached file is replaced only after its replacement verifies. Downloads
+stream to unique temporary files and rename after verification; failures clean
+up temporary files and never expose partial final files. No cache pruning occurs.
+
+`download()` accepts `timeout=30` and `retries=2` for HTTP connection/response
+retries. A failed transfer or checksum check raises an error; a subsequent call
+can retry the download. It requires neither DuckDB nor another metadata request.
+
+## 3. DuckDB API
+
+```python
+from lgd_query import connect
+
+states = release.dataset("states")
+districts = release.dataset("districts")
+
+with connect([states, districts]) as db:
+    rows = db.sql("""
+        SELECT s.state_name, count(*) AS district_count
+        FROM districts d JOIN states s USING (state_code)
+        GROUP BY s.state_name ORDER BY s.state_name
+    """).fetchall()
+```
+
+`connect()` returns a native `duckdb.DuckDBPyConnection`, with one view per
+selected dataset using its canonical name. Close it or use a context manager.
+Only the requested datasets are opened. Resolve them from one `Release` to pin
+one publication generation; their actual observation dates can still differ.
+
+By default DuckDB queries the resolved remote Parquet URLs with HTTP range
+requests. Its `httpfs` extension may be installed automatically on first use.
+Range queries do not verify the checksum of the entire file. For fully verified
+files and reusable downloads:
+
+```python
+with connect([states, districts], cache_dir="./data") as db:
+    rows = db.sql("SELECT * FROM districts WHERE state_code = ?", params=["33"]).fetchall()
+```
+
+With valid cached files and a saved index, this path works offline. DuckDB
+settings can be supplied with `config={"threads": 2, "memory_limit": "512MB"}`.
+`timeout` and `retries` affect cache downloads; remote range-request settings
+are controlled by DuckDB. No domain-specific query helpers are included yet.
+
+## CLI
+
+```sh
+lgd-query list
+lgd-query list --available-only
+lgd-query list --format json
+lgd-query resolve villages
+lgd-query resolve villages --manifest --observed-on 2026-10-23
+lgd-query download states districts --directory ./data
+lgd-query query 'SELECT state_code, count(*) FROM districts GROUP BY state_code' --datasets districts
+lgd-query query 'SELECT * FROM districts WHERE state_code = ?' --datasets districts --params '["33"]' --cache-dir ./data
+lgd-query query 'SELECT count(*) FROM districts' --datasets districts --format json
+```
+
+All commands default to Rich tables. Use `--format json` for scriptable output;
+JSON mode does not require Rich. Resolved metadata tables summarize the dataset
+and optional schema; JSON retains all manifest/source details. SQL JSON results
+have `columns` and `rows` arrays, preserving duplicate column names. Both formats
+use batches of at most 1,000 query rows; tables repeat their headers per batch.
+Dates and other non-JSON scalar types are represented as strings. Non-finite
+SQL floats use the JSON strings `"NaN"`, `"Infinity"` and `"-Infinity"`, including
+inside lists and structs; SQL NULL remains JSON null. Errors exit
+nonzero with a diagnostic.
+Maps with non-string keys use parallel `key` and `value` arrays, matching
+DuckDB's representation for complex map keys, so JSON object keys cannot
+silently change their meaning. Maps with string keys remain JSON objects.
+`python -m lgd_query` is equivalent to the installed `lgd-query` command.
+
+Pass `--metadata-url URL` before the command to select an immutable generation
+or mirror. `resolve`, `download` and `query` accept `--observed-on YYYY-MM-DD` to
+require the exact report date. Every command fetches discovery metadata only once.
+
+## Development and packaging
+
+```sh
+pip install -e '.[cli,duckdb]'
+python -m unittest discover -s tests -v
+uv build --out-dir dist
+```
+
+Build from this directory. Package discovery is restricted to `src/lgd_query`;
+the rest of the repository is outside the distribution. Tests use synthetic
+release metadata and small generated Parquet files. Tests that require DuckDB
+are skipped when the optional dependency is absent.
+
+## Version releases
+
+The repository workflow is `.github/workflows/lgd-query-publish.yml`, matching
+the PyPI trusted-publisher configuration. It uses the `pypi` GitHub environment
+and [OIDC trusted publishing](https://docs.pypi.org/trusted-publishers/using-a-publisher/).
+
+A pushed `lgd-query-v<version>` tag builds this directory, checks that the tag
+matches `pyproject.toml`, validates the distribution contents and PyPI metadata,
+and tests the built wheel on Python 3.11 and 3.14. Only after these checks pass
+does it upload the same wheel/source archive to PyPI and create a corresponding
+GitHub release with those files attached. No scraper or history files are bundled.
+
+For example, after committing version `0.1.0`:
+
+```sh
+git tag lgd-query-v0.1.0
+git push origin lgd-query-v0.1.0
+```
+
+Manual workflow runs on a branch only build/test. A manual run on a matching
+version tag also publishes. Failed publication jobs can be retried using the
+same built artifacts. GitHub release titles and tags use the library prefix to
+keep package versions separate from the data releases.
+
+Before a retry uploads files, the workflow compares any existing PyPI files with
+the checked local sizes and SHA-256 hashes. Different bytes under the same version
+stop publication. Before creating/updating the GitHub release, it verifies that
+both files match PyPI. If PyPI's metadata has not caught up yet, retry that job;
+it will not attach unconfirmed files. For a partial upload, rerun failed jobs
+with the original build artifacts instead of rebuilding the same version.

lgd_query-0.1.0/README.md

@@ -0,0 +1,209 @@
+# lgd-query
+
+A small Python client for the public [LGD Parquet releases](https://github.com/ramSeraph/opendata/releases/tag/lgd-parquet-latest).
+It has three public APIs: metadata resolution, verified downloading and DuckDB
+access. The CLI uses those same APIs. No GitHub account or API server is needed.
+
+This directory is a standalone distribution. Its wheel contains only
+`lgd_query`; it has no dependency on the repository's scrapers, converters or
+history code. Python 3.11+ is required, matching DuckDB 2's minimum. Metadata and downloads use the standard
+library; SQL access additionally requires DuckDB 2 or newer.
+Rich is an optional dependency for CLI tables.
+
+## Installation
+
+Until the first PyPI release, install from this directory:
+
+```sh
+pip install .                 # Metadata API, download API and their CLI commands
+pip install '.[duckdb]'       # Also include DuckDB querying
+pip install '.[cli,duckdb]'    # All APIs and Rich table output
+```
+
+The distribution name is `lgd-query`; the Python import is `lgd_query`.
+After publication, the corresponding installs are `pip install lgd-query` and
+`pip install 'lgd-query[duckdb]'`.
+
+## 1. Metadata API
+
+```python
+from lgd_query import load_release, load_manifest
+
+release = load_release()
+villages = release.dataset("villages")
+
+print(release.generation, release.published_at, release.immutable_url)
+print(villages.observed_on, villages.rows, villages.status)
+print(villages.parquet.url, villages.parquet.size, villages.parquet.sha256)
+
+# Optional extra request for canonical schema and original-source provenance.
+manifest = load_manifest(villages)
+print(manifest["schema"])
+```
+
+`load_release()` fetches `metadata.json` once. Its `Release` object retains that
+generation, and datasets refer directly to immutable files even if the latest
+index changes. Accessing `release.datasets` or `release.dataset()` makes no
+requests. New datasets can be discovered without updating a hardcoded registry.
+`load_manifest()` verifies the manifest's size/hash and agreement with the index.
+
+`release.datasets` is a read-only mapping including unavailable datasets. Each
+`Dataset` exposes `name`, `available`, `status`, `reason`, `representation`,
+`observed_on`, `uploaded_at`, `rows`, `parquet`, `manifest`, `identity`,
+`coverage` and `metadata`. `parquet` and `manifest` are `Asset` references with
+`url`, `size` and `sha256`. An unavailable dataset has no file references;
+`release.dataset(name)` raises `DatasetUnavailableError` for it. An unknown name
+raises `KeyError`.
+
+“Latest” means the files advertised in one release index. Datasets can have
+different source dates. A failed update can retain a usable older file, with its
+actual date and failure status. Require an exact date when freshness matters:
+
+```python
+villages = release.dataset("villages", observed_on="2026-10-23")
+```
+
+A mismatch raises `ObservationDateError`; this is a freshness check, not a
+historical lookup. Cumulative `changes` and `transactions` reports have
+`observed_on=None`; their `coverage` describes event dates and collection coverage.
+Publication/upload timestamps do not establish an observation date. These files
+contain supplied report rows; omissions do not prove deletions.
+
+To reopen a generation, pass its immutable URL to `load_release(url)`. For an
+offline metadata lookup, save `release.metadata` as JSON and reopen it with
+`Release.from_dict(document)`. `metadata` and `coverage` return detached copies.
+Metadata requests accept `timeout=30` and `retries=2`. Transient HTTP responses,
+including a brief 404 during index replacement, get bounded retries. The reader
+does not silently fall back to `metadata.previous.json`; that URL can be supplied
+explicitly if the preceding generation is wanted.
+
+## 2. Download API
+
+```python
+from lgd_query import download
+
+path = download(villages, directory="./data")
+```
+
+The returned `pathlib.Path` names a complete Parquet file whose size and SHA-256
+match the resolved metadata. Files are named `<dataset>.<sha256>.parquet`, so
+different contents coexist in the cache. Every reuse verifies the cached file.
+A corrupt cached file is replaced only after its replacement verifies. Downloads
+stream to unique temporary files and rename after verification; failures clean
+up temporary files and never expose partial final files. No cache pruning occurs.
+
+`download()` accepts `timeout=30` and `retries=2` for HTTP connection/response
+retries. A failed transfer or checksum check raises an error; a subsequent call
+can retry the download. It requires neither DuckDB nor another metadata request.
+
+## 3. DuckDB API
+
+```python
+from lgd_query import connect
+
+states = release.dataset("states")
+districts = release.dataset("districts")
+
+with connect([states, districts]) as db:
+    rows = db.sql("""
+        SELECT s.state_name, count(*) AS district_count
+        FROM districts d JOIN states s USING (state_code)
+        GROUP BY s.state_name ORDER BY s.state_name
+    """).fetchall()
+```
+
+`connect()` returns a native `duckdb.DuckDBPyConnection`, with one view per
+selected dataset using its canonical name. Close it or use a context manager.
+Only the requested datasets are opened. Resolve them from one `Release` to pin
+one publication generation; their actual observation dates can still differ.
+
+By default DuckDB queries the resolved remote Parquet URLs with HTTP range
+requests. Its `httpfs` extension may be installed automatically on first use.
+Range queries do not verify the checksum of the entire file. For fully verified
+files and reusable downloads:
+
+```python
+with connect([states, districts], cache_dir="./data") as db:
+    rows = db.sql("SELECT * FROM districts WHERE state_code = ?", params=["33"]).fetchall()
+```
+
+With valid cached files and a saved index, this path works offline. DuckDB
+settings can be supplied with `config={"threads": 2, "memory_limit": "512MB"}`.
+`timeout` and `retries` affect cache downloads; remote range-request settings
+are controlled by DuckDB. No domain-specific query helpers are included yet.
+
+## CLI
+
+```sh
+lgd-query list
+lgd-query list --available-only
+lgd-query list --format json
+lgd-query resolve villages
+lgd-query resolve villages --manifest --observed-on 2026-10-23
+lgd-query download states districts --directory ./data
+lgd-query query 'SELECT state_code, count(*) FROM districts GROUP BY state_code' --datasets districts
+lgd-query query 'SELECT * FROM districts WHERE state_code = ?' --datasets districts --params '["33"]' --cache-dir ./data
+lgd-query query 'SELECT count(*) FROM districts' --datasets districts --format json
+```
+
+All commands default to Rich tables. Use `--format json` for scriptable output;
+JSON mode does not require Rich. Resolved metadata tables summarize the dataset
+and optional schema; JSON retains all manifest/source details. SQL JSON results
+have `columns` and `rows` arrays, preserving duplicate column names. Both formats
+use batches of at most 1,000 query rows; tables repeat their headers per batch.
+Dates and other non-JSON scalar types are represented as strings. Non-finite
+SQL floats use the JSON strings `"NaN"`, `"Infinity"` and `"-Infinity"`, including
+inside lists and structs; SQL NULL remains JSON null. Errors exit
+nonzero with a diagnostic.
+Maps with non-string keys use parallel `key` and `value` arrays, matching
+DuckDB's representation for complex map keys, so JSON object keys cannot
+silently change their meaning. Maps with string keys remain JSON objects.
+`python -m lgd_query` is equivalent to the installed `lgd-query` command.
+
+Pass `--metadata-url URL` before the command to select an immutable generation
+or mirror. `resolve`, `download` and `query` accept `--observed-on YYYY-MM-DD` to
+require the exact report date. Every command fetches discovery metadata only once.
+
+## Development and packaging
+
+```sh
+pip install -e '.[cli,duckdb]'
+python -m unittest discover -s tests -v
+uv build --out-dir dist
+```
+
+Build from this directory. Package discovery is restricted to `src/lgd_query`;
+the rest of the repository is outside the distribution. Tests use synthetic
+release metadata and small generated Parquet files. Tests that require DuckDB
+are skipped when the optional dependency is absent.
+
+## Version releases
+
+The repository workflow is `.github/workflows/lgd-query-publish.yml`, matching
+the PyPI trusted-publisher configuration. It uses the `pypi` GitHub environment
+and [OIDC trusted publishing](https://docs.pypi.org/trusted-publishers/using-a-publisher/).
+
+A pushed `lgd-query-v<version>` tag builds this directory, checks that the tag
+matches `pyproject.toml`, validates the distribution contents and PyPI metadata,
+and tests the built wheel on Python 3.11 and 3.14. Only after these checks pass
+does it upload the same wheel/source archive to PyPI and create a corresponding
+GitHub release with those files attached. No scraper or history files are bundled.
+
+For example, after committing version `0.1.0`:
+
+```sh
+git tag lgd-query-v0.1.0
+git push origin lgd-query-v0.1.0
+```
+
+Manual workflow runs on a branch only build/test. A manual run on a matching
+version tag also publishes. Failed publication jobs can be retried using the
+same built artifacts. GitHub release titles and tags use the library prefix to
+keep package versions separate from the data releases.
+
+Before a retry uploads files, the workflow compares any existing PyPI files with
+the checked local sizes and SHA-256 hashes. Different bytes under the same version
+stop publication. Before creating/updating the GitHub release, it verifies that
+both files match PyPI. If PyPI's metadata has not caught up yet, retry that job;
+it will not attach unconfirmed files. For a partial upload, rerun failed jobs
+with the original build artifacts instead of rebuilding the same version.

lgd_query-0.1.0/pyproject.toml

@@ -0,0 +1,36 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "lgd-query"
+version = "0.1.0"
+description = "Resolve, download and query public Local Government Directory Parquet releases"
+readme = "README.md"
+requires-python = ">=3.11"
+dependencies = []
+keywords = ["india", "lgd", "local-government-directory", "parquet", "duckdb"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Programming Language :: Python :: 3",
+    "Topic :: Database",
+]
+
+[project.optional-dependencies]
+duckdb = ["duckdb>=2.0.0"]
+cli = ["rich>=14"]
+
+[project.scripts]
+lgd-query = "lgd_query.cli:main"
+
+[project.urls]
+Repository = "https://github.com/ramSeraph/opendata"
+Documentation = "https://github.com/ramSeraph/opendata/tree/master/lgd-query"
+Issues = "https://github.com/ramSeraph/opendata/issues"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+include = ["lgd_query", "lgd_query.*"]
+
+[tool.setuptools.package-data]
+lgd_query = ["py.typed"]

lgd_query-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

lgd_query-0.1.0/src/lgd_query/__init__.py

@@ -0,0 +1,12 @@
+"""Public APIs for LGD release metadata, downloads and optional DuckDB access."""
+
+from .metadata import Asset, Dataset, Release, DEFAULT_METADATA_URL, load_manifest, load_release
+from .files import download
+from .query import connect
+from .errors import DatasetUnavailableError, IntegrityError, MetadataError, ObservationDateError
+
+__all__ = [
+    'Asset', 'Dataset', 'Release', 'DEFAULT_METADATA_URL', 'load_release', 'load_manifest',
+    'download', 'connect', 'DatasetUnavailableError', 'IntegrityError', 'MetadataError',
+    'ObservationDateError',
+]

lgd_query-0.1.0/src/lgd_query/__main__.py

@@ -0,0 +1,3 @@
+from .cli import main
+
+raise SystemExit(main())

lgd_query-0.1.0/src/lgd_query/_http.py

@@ -0,0 +1,66 @@
+"""Anonymous HTTP transport shared by metadata and file downloads."""
+
+import json
+import time
+from urllib.error import HTTPError, URLError
+from urllib.parse import urlsplit
+from urllib.request import Request, urlopen
+
+from .errors import MetadataError
+
+
+MAX_METADATA_BYTES = 8 * 1024 * 1024
+
+
+def http_url(value):
+    if not isinstance(value, str):
+        raise MetadataError('Expected an HTTP(S) URL')
+    try:
+        parts = urlsplit(value)
+        # Accessing port performs validation; urlsplit alone accepts bad ports.
+        parts.port
+        valid = (parts.scheme in ('https', 'http') and parts.hostname and
+                 parts.username is None and parts.password is None and not parts.fragment and
+                 not any(char.isspace() or ord(char) < 32 or ord(char) == 127 for char in value))
+    except ValueError as error:
+        raise MetadataError(f'Invalid public HTTP(S) URL: {value!r}') from error
+    if not valid:
+        raise MetadataError(f'Invalid public HTTP(S) URL: {value!r}')
+    return value
+
+
+def open_url(url, *, timeout=30, retries=2):
+    http_url(url)
+    if timeout <= 0 or type(retries) is not int or retries < 0:
+        raise ValueError('timeout must be positive and retries a nonnegative integer')
+    request = Request(url, headers={'User-Agent': 'lgd-query', 'Accept-Encoding': 'identity'})
+    for attempt in range(retries + 1):
+        try:
+            return urlopen(request, timeout=timeout)
+        except HTTPError as error:
+            # metadata.json can briefly disappear during release replacement.
+            error.close()
+            if error.code not in (404, 408, 429, 500, 502, 503, 504) or attempt == retries:
+                raise
+        except (URLError, TimeoutError):
+            if attempt == retries:
+                raise
+        time.sleep(0.5 * 2**attempt)
+
+
+def read_json(url, *, asset=None, timeout=30, retries=2):
+    if asset is not None and asset.size > MAX_METADATA_BYTES:
+        raise MetadataError('Manifest exceeds the metadata size limit')
+    with open_url(url, timeout=timeout, retries=retries) as response:
+        content = response.read(MAX_METADATA_BYTES + 1)
+    if len(content) > MAX_METADATA_BYTES:
+        raise MetadataError('Response exceeds the metadata size limit')
+    if asset is not None:
+        asset.verify(content)
+    try:
+        result = json.loads(content)
+    except (ValueError, UnicodeError) as error:
+        raise MetadataError(f'Invalid JSON metadata: {url}') from error
+    if not isinstance(result, dict):
+        raise MetadataError('Metadata must be a JSON object')
+    return result