crc-sdk 0.1.0__tar.gz

crc_sdk-0.1.0/LICENSE

@@ -0,0 +1,18 @@
+CRC SDK
+Copyright (C) 2026 Riskthinking.AI
+
+This program is free software: you can redistribute it and/or modify it
+under the terms of the GNU Affero General Public License as published by
+the Free Software Foundation, either version 3 of the License, or (at your
+option) any later version.
+
+This program is distributed in the hope that it will be useful, but
+WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public
+License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program. If not, see <https://www.gnu.org/licenses/>.
+
+The complete license text is available at:
+<https://www.gnu.org/licenses/agpl-3.0.txt>

crc_sdk-0.1.0/PKG-INFO

@@ -0,0 +1,374 @@
+Metadata-Version: 2.4
+Name: crc-sdk
+Version: 0.1.0
+Summary: Python SDK for Climate Risk Commons data access and workflows
+Author: Riskthinking.AI
+License-Expression: AGPL-3.0-or-later
+Classifier: Development Status :: 3 - Alpha
+Classifier: Programming Language :: Python :: 3
+Classifier: Typing :: Typed
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: crc-framework<0.3,>=0.2
+Requires-Dist: pydantic<3,>=2
+Requires-Dist: duckdb<2,>=1
+Requires-Dist: pyarrow>=14
+Requires-Dist: psutil>=5
+Requires-Dist: fsspec>=2024
+Requires-Dist: s3fs>=2024
+Requires-Dist: gcsfs>=2024
+Provides-Extra: zarr
+Requires-Dist: zarr<3,>=2.18; extra == "zarr"
+Provides-Extra: raster
+Requires-Dist: rasterio>=1.4; extra == "raster"
+Provides-Extra: geometry
+Requires-Dist: h3>=4; extra == "geometry"
+Requires-Dist: h3ronpy>=0.19; extra == "geometry"
+Requires-Dist: shapely>=2; extra == "geometry"
+Provides-Extra: test
+Requires-Dist: mypy>=1.10; extra == "test"
+Requires-Dist: pytest>=8; extra == "test"
+Requires-Dist: ruff>=0.5; extra == "test"
+Dynamic: license-file
+
+# CRC SDK
+
+CRC SDK is the higher-level Python interface for Climate Risk Commons data
+access, storage providers, geometry utilities, and analytical workflows.
+Numerical distributions, curve fitting, impact transforms, and risk metrics are
+provided by the versioned
+[`crc-framework`](https://pypi.org/project/crc-framework/) dependency.
+
+## Development
+
+uv is the recommended tool for managing the development environment:
+
+```shell
+uv sync --all-extras
+
+uv run pytest
+uv run mypy
+uv run ruff check .
+```
+
+Or simply:
+
+```shell
+python -m venv .venv
+.venv/bin/python -m pip install -e ".[zarr,raster,geometry,test]"
+.venv/bin/python -m pytest
+.venv/bin/python -m mypy
+.venv/bin/python -m ruff check .
+```
+
+## Dependencies
+
+DuckDB, Arrow (`pyarrow`), `psutil` (resource detection), and remote-storage
+transport (`fsspec`, `s3fs`, `gcsfs`) are baseline dependencies — every
+connector and workflow in this SDK is built on that stack, so gating it
+behind an extra would just move the same install onto every real caller.
+
+Everything else is a specific data-format adapter or a pure-geometry
+dependency, opted into only by the callers that need it:
+
+| Extra | Adds | Used by |
+|---|---|---|
+| `zarr` | `zarr` | `OSClimateProvider`/`ZarrRaster` (OS-Climate Zarr raster ingest) |
+| `raster` | `rasterio` | `GeoTiffRaster` (GeoTIFF/COG ingest, streamed via GDAL VSI) |
+| `geometry` | `h3`, `h3ronpy`, `shapely` | `H3Indexer`, `intersecting_cells`, `cell_polygon`, other abstract H3/geometry math, vectorized batch H3 ops on Arrow data (`polyfill_wkb`, `expand_polygon_candidates`, raster-to-H3 sampling) |
+| `test` | `mypy`, `pytest`, `ruff` | Development only |
+
+Every function that needs an extra-gated dependency imports it lazily and
+raises a clear `ImportError` naming the extra to install if it's missing —
+importing `crc_sdk` (or any of its subpackages) itself never requires more
+than the baseline dependencies.
+
+**OS-level dependency (not a pip extra):** `tippecanoe` and `tile-join`
+(https://github.com/felt/tippecanoe) must be present on `PATH` for
+`crc_sdk.geometry.pmtiles` — they're assumed to already be installed on the
+runtime image, not `pip install`-able, so there's no extra for them. Verify
+availability with `require_tippecanoe()`/`require_tile_join()`, which raise a
+friendly, actionable error (with install instructions) if either is missing.
+
+## Package boundaries
+
+- `crc_sdk.core`, `crc_sdk.fitting`, and `crc_sdk.impacts` expose the stable
+  public API of `crc_framework`.
+- `crc_sdk.connectors` handles external formats and query engines: DuckDB
+  connection helpers (`DuckDBConnection`, `RuntimeResources`, streaming
+  Parquet writes), OS-Climate Zarr ingest (`zarr` extra), and GeoTIFF/COG
+  ingest (`GeoTiffRaster`, `raster` extra) — the latter streams directly
+  from local paths or `gs://`/`s3://`/`http(s)://` URIs via GDAL's own
+  range-request support, with no local download by default.
+- `crc_sdk.providers` describes storage and dataset discovery.
+- `crc_sdk.geometry` contains geometry conversion, DuckDB-native H3 polyfill
+  (`H3Indexer`), Arrow batch polyfill (`polyfill_wkb`, `geometry`
+  extra for h3ronpy), raster-to-H3 sampling primitives
+  (`pixel_grid_resolution`, `sample_grid_to_h3`), exploded coverage writers
+  (`write_exploded_coverage`), optional nested lookup derivation
+  (`LookupCatalog`, `write_lookup_contract`, `write_partitioned_lookup`), and
+  PMTiles generation (`crc_sdk.geometry.pmtiles` — also reachable flattened
+  as `crc_sdk.geometry.PMTilesBuild`, etc.): `PMTilesBuild` streams a
+  GeoParquet source (a single file, or a Hive-partitioned dataset glob) into
+  one `.pmtiles` archive in one tiling pass, building the GeoParquet ->
+  GeoJSON bridge itself in DuckDB `spatial`-extension SQL
+  (`ST_AsGeoJSON`/`ST_ReducePrecision`/`ST_Transform`) rather than shelling
+  out to an external converter, streamed via the same Arrow-batched-reader
+  pattern used elsewhere in this SDK. `tippecanoe_threads`/`duckdb_threads`
+  default to every detected core (no conservative per-thread cap, unlike
+  DuckDB's own GEOS-throttled default) since tippecanoe's tile-building has
+  no documented per-thread memory ceiling. A pre-flight budget check raises
+  a clear, actionable error if a source is estimated to exceed available
+  scratch disk, rather than silently degrading into a slower multi-batch
+  fallback — provisioning more disk or narrowing the run's scope is left to
+  the caller.
+- `crc_sdk.schema` defines columnar data contracts.
+- `crc_sdk.types` contains SDK-owned Pydantic configuration and metadata.
+- `crc_sdk.workflows` coordinates data access and computation.
+
+DuckDB resource limits are detected when requested
+(`RuntimeResources.detect` / `DuckDBConnection.for_analytics`) and relayed
+through the connection `config` mapping. Thread count is
+`min(cpus, usable_RAM / GiB_per_thread)` with usable RAM ≈ 60% of detected
+memory and a default of ~2.5 GiB/thread (GEOS spatial work often slows when
+over-threaded). `memory_limit` and `max_temp_directory_size` remain hard
+process caps. Override with `CRC_DUCKDB_THREADS`, `CRC_DUCKDB_MEMORY`, and/or
+`CRC_DUCKDB_BYTES_PER_THREAD_GIB`, or pass an explicit `config` dict. Set
+`CRC_DUCKDB_PROFILE=1` to enable detailed query profiling around
+enrich/coverage stages in h3geo.
+
+Constructors with no natural caller-supplied directory of their own
+(`OSClimateProvider`, `ZarrRaster`, `H3Indexer`) build a resource-tuned
+connection by default — via `DuckDBConnection.for_analytics` — instead of a
+bare, untuned one, so this scales out of the box with no configuration.
+Passing an explicit `connection`/`con` always wins and skips this entirely.
+Otherwise the spill/temp directory defaults to a stable location under the
+system temp directory (`default_work_dir()`, not a fresh one per call), and
+can be set per-call via each constructor's own `work_dir` parameter, or
+globally via `CRC_DUCKDB_WORK_DIR`.
+
+Private/authenticated remote sources (a non-public GCS/S3 bucket) are
+configured the same idiomatic-DuckDB way as everything else here: raw
+`CREATE OR REPLACE SECRET` SQL, passed as `setup_sql=(...)` to
+`DuckDBConnection`/`DuckDBConnection.for_analytics` to have it run
+automatically on `.connect()`, right after extensions load. There is
+deliberately no secret-builder type in the SDK — DuckDB's own secret DDL
+(https://duckdb.org/docs/configuration/secrets_manager) is already the
+documented interface, and a caller's own connection-setup module is a more
+natural home for its specific credentials than a generic wrapper trying to
+track every provider/type DuckDB supports. `sql_quote`/`sql_identifier` are
+exported for safely building that SQL; the one DuckDB quirk worth knowing is
+that `PROVIDER` is a bare keyword (`config`, `credential_chain`, ...), not a
+quoted string literal, unlike every other secret option:
+
+```python
+import os
+from crc_sdk.connectors.duckdb import DuckDBConnection, sql_quote
+
+setup_sql = []
+key_id, secret = os.getenv("GCS_ACCESS_KEY"), os.getenv("GCS_ACCESS_SECRET")
+if key_id and secret:
+    setup_sql.append(
+        f"CREATE OR REPLACE SECRET gcs (TYPE GCS, KEY_ID {sql_quote(key_id)}, "
+        f"SECRET {sql_quote(secret)})"
+    )
+con = DuckDBConnection.for_analytics(work_dir, setup_sql=setup_sql).connect()
+```
+
+## Canonical hazard datasets
+
+The SDK internalizes fitted hazards as one versioned Arrow/Parquet contract.
+Rows contain a canonical unsigned H3 `cell_index`, stable `source_id`, optional
+source WKB, scenario dimensions, and the parameters needed to reconstruct
+either a `crc_framework.FittedDistribution` or `HurdleDistribution`.
+`curve_shape` is nullable because Gumbel families do not use a shape parameter;
+atom probability and location are present only when `curve_kind` is `hurdle`.
+
+The logical row key is
+`(hazard_name, horizon, pathway, cell_index, source_id)`. `cell_index` is the
+spatial join key, not a globally unique identifier. Canonical files are sorted
+by that row key for predicate pruning and merge joins.
+
+Dataset-wide facts are stored once as a complete JSON payload under the
+`crc.hazard.metadata` Parquet key: schema version, one uncompacted H3
+resolution, non-exceedance probability convention, value unit and semantics,
+WKB CRS, producer, source provenance, and creation version.
+
+Each dataset is one self-describing Parquet file, expanded by H3 cell for
+spatial joins. The caller chooses its full destination path and filename.
+Writes use DuckDB, and an optional configured DuckDB connection allows the same
+API to use its local or cloud filesystems, extensions, secrets, and settings.
+Source knots and fit diagnostics are transient ingest inputs, not a second
+persisted data contract.
+
+External connectors remain source-format readers. Ingest adapters perform the
+explicit conversion:
+
+```text
+external raster/table -> source curves and geometry -> selected family fit
+  -> conservative intersecting H3 cells -> canonical Arrow -> Parquet
+```
+
+Boundary candidate generation uses H3 overlap coverage, not center polyfill.
+This makes the integer join a conservative superset before an exact
+`ST_Contains(source_geometry, asset_point)` refinement. Resolution estimates
+report measured coverage error and expanded row count, while ingest policy
+selects and records the dataset resolution.
+
+OS-Climate return-period rasters can be canonicalized with
+`OSClimateIngestPolicy` and `canonicalize_os_climate`. The caller must choose
+the distribution family and, for zero-heavy hazards, provide an explicit
+`HurdleFitPolicy`; the SDK does not infer an exact point mass from sparse
+knots. Plain curves use `fit_quantiles`, while hurdle curves use
+`fit_hurdle_quantiles`. `LocalProvider` queries persisted hazard rows through
+`HazardQuery`.
+
+### Evaluating asset portfolios at return periods
+
+Canonical curve parameters can be evaluated for a portfolio without returning
+to the external source format or refitting the data. The workflow joins every
+asset to its canonical curve and writes one row per asset, hazard, horizon, and
+pathway, with one value column per requested return period:
+
+```python
+import pyarrow as pa
+
+from crc_sdk.workflows import HazardDataset
+
+assets = pa.table(
+    {
+        "asset_id": ["warehouse-a", "warehouse-b"],
+        "longitude": [6.9603, 7.5010],
+        "latitude": [50.9375, 51.0030],
+        "sector": ["logistics", "manufacturing"],
+    }
+)
+
+result = (
+    HazardDataset.local("flood.parquet")
+    .for_assets(assets)
+    .select(horizons=[2050], pathways=["ssp585"])
+    .return_periods([25, 50, 100, 250, 500, 1000])
+    .write_parquet("portfolio-flood.parquet")
+)
+```
+
+The resulting value columns are `value_rp25`, `value_rp50`, `value_rp100`,
+`value_rp250`, `value_rp500`, and `value_rp1000`. For upper-tail hazards, each
+return period `RP` is evaluated at non-exceedance probability `1 - 1/RP`.
+Value unit, value semantics, and the complete return-period/probability/column
+mapping are stored under `crc.hazard.evaluation` in Parquet metadata.
+
+An impact function can replace the sampled hazard values with event-aligned
+impact values before writing:
+
+```python
+import numpy as np
+
+impact_result = (
+    HazardDataset.local("flood.parquet")
+    .for_assets(assets)
+    .return_periods([25, 100, 250])
+    .impact(
+        lambda depth: np.clip(depth / 2.0, 0.0, 1.0),
+        name="depth_damage_ratio",
+        value_unit="fraction",
+        value_semantics="damage ratio",
+    )
+    .write_parquet(
+        "portfolio-impact.parquet",
+        execution=ExecutionOptions(max_workers=1),
+    )
+)
+```
+
+The SDK first samples each hazard return period and then calls
+`impact.evaluate(...)` on that row's value vector. Therefore
+`value_rp100 = impact(hazard_rp100)`: the return period continues to identify
+the source hazard event. This differs intentionally from transforming a full
+distribution and then taking an impact quantile, which can reorder decreasing
+or non-monotonic impacts. Use the distribution interface in `crc-framework`
+for that risk-analysis interpretation.
+
+Built-in and registry-backed framework impacts use the same fluent method:
+
+```python
+from crc_sdk.impacts import PiecewiseLinearImpact, impacts
+from crc_sdk.workflows import ImpactContextColumns
+
+damage_curve = PiecewiseLinearImpact(
+    exposure=[0.0, 0.2, 1.0, 2.0],
+    impact=[0.0, 0.0, 0.25, 1.0],
+)
+
+request = request.impact(
+    damage_curve,
+    name="flood_damage_ratio",
+    value_unit="fraction",
+    value_semantics="damage ratio",
+)
+
+registry_request = request.impact(
+    impacts.for_factor("inundation"),
+    context=ImpactContextColumns(
+        country="country",
+        continent="continent",
+        building_type="building_type",
+        historic_mean="historic_mean",
+    ),
+    name="inundation_impact",
+    value_unit="fraction",
+    value_semantics="damage ratio",
+)
+```
+
+The generated H3 `cell_index` is always supplied to the framework impact
+context. Configured context columns are read from each asset, including when
+they are not retained as output passthrough columns. Stored registry context
+provides fallback values for fields without an asset value. Impact metadata
+records the event-aligned interpretation, source hazard units and semantics,
+output units and semantics, function name/type, and context-column mapping.
+
+Framework impact objects and top-level Python callables can run in the existing
+process pool. Lambdas and closures are not picklable, so they run serially when
+the worker count is implicit; explicitly requesting more than one worker for
+one raises an error.
+
+Point assets are converted to the H3 resolution recorded by the canonical
+dataset. The H3 join is refined with `ST_Covers(source_geometry, asset_point)`
+when source WKB is present; rows without WKB retain cell-level precision.
+`source_id` and `spatial_match` (`exact_geometry` or `h3_cell`) remain in the
+output. Multiple source curves for one asset/hazard/horizon/pathway raise
+instead of being silently selected or aggregated, and missing asset/scenario
+matches raise rather than being dropped from the output.
+
+When assets already contain canonical H3 indexes, use
+`cell_index_column="cell_index"` instead of longitude/latitude columns. This
+avoids point conversion and exact source-geometry refinement:
+
+```python
+(
+    HazardDataset.local("flood.parquet")
+    .for_assets(assets_with_cells)
+    .return_periods([25, 50, 100, 250, 500, 1000])
+    .write_parquet("portfolio-flood.parquet")
+)
+```
+
+Arrow tables can be registered directly as shown above. A `Path` reads an
+asset Parquet file, while a string is treated as caller-supplied DuckDB SQL.
+Output evaluation is streamed in bounded Arrow batches to compressed Parquet.
+For already selected canonical rows, `distribution_from_hazard_row` remains
+available as the low-level curve reconstruction utility.
+
+The common column names `asset_id`, `longitude`/`latitude`, and `cell_index`
+are inferred. Use `AssetPortfolio`, `PointColumns`, or `CellColumn` only for a
+nonstandard asset schema. Worker, batch, and connection controls are grouped
+under `ExecutionOptions` on `write_parquet`, keeping execution tuning out of
+the normal workflow.
+
+## License
+
+CRC SDK is licensed under the GNU Affero General Public License, version 3 or
+later.