core-lens 0.1.dev169__tar.gz → 0.1.dev172__tar.gz

{core_lens-0.1.dev169 → core_lens-0.1.dev172}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: core-lens
-Version: 0.1.dev169
+Version: 0.1.dev172
 Summary: Query, analyse, and visualise CoreStack's microwatershed and Earth science data through a clean, composable Python API.
 Project-URL: Homepage, https://github.com/ApoorvaKashyap/core-lens
 Project-URL: Issues, https://github.com/ApoorvaKashyap/core-lens/issues

core_lens-0.1.dev172/benchmarks/README.md

@@ -0,0 +1,77 @@
+# core-lens Benchmarks
+
+Focused Scalene profiling scripts — one per module/subsystem.
+
+## Structure
+
+```
+benchmarks/
+├── README.md
+├── run_all.sh                  # Run all benchmarks in sequence
+├── bench_aoi.py                # AoI init, register, named-boundary resolution, current_season/year
+├── bench_entity.py             # BaseEntity._resolve, where, spatial_filter, spatial_join, caches  [NEW]
+├── bench_spatial.py            # build_bbox_index, bbox_intersects_geometry, exact_spatial_filter, execute_spatial_join
+├── bench_paths.py              # is_cloud_uri, resolve_fs_and_path, path_exists, join_uri  [NEW]
+├── bench_view.py               # View.where, spatial_filter, between (all modes + errors), spatial_join, materialise
+├── bench_result.py             # Result.derive, aggregate (all groupings + errors), with_geometry, df/gdf, stats/plot
+├── bench_season.py             # resolve_time_filter, add_temporal_columns, SeasonConfig.season_for
+├── bench_schema.py             # schema detection (_infer_* helpers, _is_year_col, _find_lon_companion, from_file)
+├── bench_polars_utils.py       # scan_with_key_filter, collect_lf, cached_read_schema (CPU path)
+└── bench_export.py             # parquet, json, csv, geoparquet, geojson, GeoJSONSeq, error paths
+```
+
+## Running
+
+### Single benchmark
+
+```bash
+uv run scalene --html --outfile benchmarks/profiles/bench_spatial.html benchmarks/bench_spatial.py
+```
+
+### All benchmarks (sequential, JSON output via Scalene)
+
+```bash
+chmod +x benchmarks/run_all.sh
+./benchmarks/run_all.sh
+```
+
+Profiles land in `benchmarks/profiles/`.
+
+### Quick timing-only run (no Scalene overhead)
+
+```bash
+uv run python benchmarks/bench_spatial.py
+```
+
+## What to look for in Scalene output
+
+| Signal | Meaning |
+|---|---|
+| High **CPU %** on line | Compute bottleneck — vectorise or push into Polars expr |
+| High **memory** spike | Possible copy-on-write or large intermediate frame |
+| `[native]` vs `[Python]` | Native = C ext (fast). Python = pure Python (optimise) |
+| **GPU %** column | cudf path hit or not |
+
+## Coverage map
+
+| File | Functions / paths covered |
+|---|---|
+| `bench_aoi.py` | `AoI.register`, `AoI.__init__` (bbox/geometry/named/seasons), `__getattr__`, `current_season/year`, `registered_entities`, `deregister` |
+| `bench_entity.py` | `_resolve` (local/cloud), `where` (single/composite/list/error), `spatial_filter` (bbox/geometry/area/error), `spatial_join`, `schema_profile` cache, `_index` cache, `geometry_lazy`, `_so_key`, `_entity_name`, `_cached_detect`, `EntityValidationError` |
+| `bench_spatial.py` | `build_bbox_index`, `bbox_intersects_geometry`, `exact_spatial_filter` (centroid/area/all-India), `execute_spatial_join`, `_bbox_sidecar_path`, `_read_bbox_sidecar`, error path |
+| `bench_paths.py` | `is_cloud_uri` (positive/negative), `resolve_fs_and_path` (local/cloud), `path_exists` (exists/absent), `join_uri` (local/cloud) |
+| `bench_view.py` | `where` (single/composite), `spatial_filter` (bbox), `between` (dates/season/CURRENT/year-range/errors), `spatial_join` (spec/double-error), `static`/`annual`/`sub_annual` materialisation |
+| `bench_result.py` | `df`, `lazy`, `gdf` (WKB decode), `materialise`, `derive` (single/chain), `with_geometry` (noop/annual), `aggregate` (no-by/year/month/season/year_month/season_year/errors), `gdf` TypeError guard, `stats`/`plot` accessors |
+| `bench_season.py` | `SeasonConfig.__post_init__` (default/custom), `season_for`, `resolve_time_filter` (date-range/kharif/rabi/year-range), `add_temporal_columns` (first/second noop call), Polars filter execution |
+| `bench_schema.py` | `_read_schema`, `_infer_geometry_type`, `_infer_bbox_cols`, `_infer_time_col`, `_is_year_col_from_schema`, `_find_lon_companion`, `_require_cols` (ok/error), `detect` (static/full), `SchemaProfile.from_file`, `entity.schema_profile` (cold/warm) |
+| `bench_polars_utils.py` | `_gpu_available`, `scan_with_key_filter` (build/+time_expr), `collect_lf` (1-key/small/all), `cached_read_schema` (warm/cold) |
+| `bench_export.py` | `_get_duckdb_conn` (cold/cached), `parquet`/`json`/`csv`/`geoparquet`/`geojson` (small/all), `geojson(GeoJSONSeq)`, all error paths, format size comparison |
+
+## Prerequisites
+
+```bash
+uv sync --group dev
+```
+
+Each script needs `DATA_ROOT` pointing at a directory with the expected entity Parquet files.
+Edit the `DATA_ROOT` constant at the top of each file to match your local data path.

core_lens-0.1.dev172/benchmarks/bench_entity.py

@@ -0,0 +1,308 @@
+"""bench_entity.py — Benchmark for core_lens.base.entity.BaseEntity
+===================================================================
+Targets:
+  - BaseEntity._resolve()        absolute / relative / cloud URI resolution
+  - BaseEntity.where()           attribute filter (single col, composite, list val)
+  - BaseEntity.where()           entity-kwarg spatial lookup path
+  - BaseEntity.spatial_filter()  full pipeline (bbox + geometry mode)
+  - BaseEntity.spatial_join()    deferred join spec construction
+  - BaseEntity.schema_profile    instance-level cache (hasattr guard)
+  - BaseEntity._index            process-level cache (_cached_build_index)
+  - BaseEntity.geometry_lazy     lazy geometry scan construction
+  - _so_key()                    storage_options → hashable tuple
+  - _entity_name()               entity class name → string
+  - _cached_detect()             process-level schema cache (functools.cache)
+  - _cached_build_index()        process-level bbox index cache
+  - EntityValidationError        exception class (import overhead)
+
+Scalene focuses on:
+  - _resolve() pathlib.Path ops vs cloud URI fast-path
+  - cached_read_schema() in where() — hot path schema lookup
+  - bbox_intersects_geometry() → exact_spatial_filter() inside where()
+  - functools.cache lookup overhead in _cached_detect / _cached_build_index
+
+Usage:
+    uv run scalene run --html --outfile benchmarks/profiles/bench_entity.html benchmarks/bench_entity.py
+    uv run python benchmarks/bench_entity.py
+"""
+
+from __future__ import annotations
+
+import time
+
+import shapely.geometry as sgeom
+
+from core_lens import AoI
+from core_lens.entities.mws import MWSEntity
+from core_lens.base.entity import (
+    _so_key,
+    _entity_name,
+    _cached_detect,
+    _cached_build_index,
+    EntityValidationError,
+)
+
+# ── Config ────────────────────────────────────────────────────────────────────
+DATA_ROOT = "data/"
+INDIA_BBOX: tuple[float, float, float, float] = (68.0, 6.0, 98.0, 38.0)
+SMALL_BBOX: tuple[float, float, float, float] = (86.0, 21.0, 87.5, 22.0)
+MWS_ID = "12_322881"  # update to a valid mws_id in your dataset
+
+AoI.register(MWSEntity)
+entity = MWSEntity(data_root=DATA_ROOT)
+static_path = entity._resolve(entity.static_path)
+
+
+def _section(title: str) -> None:
+    print(f"\n{'─' * 60}")
+    print(f"  {title}")
+    print("─" * 60)
+
+
+# ── 1. _so_key() — storage_options → hashable tuple ──────────────────────────
+_section("1. _so_key()  [dict → sorted tuple — O(n log n)]")
+REPS = 1_000_000
+
+so_empty: dict[str, str] = {}
+so_aws: dict[str, str] = {"region": "ap-south-1", "access_key": "AKIAIOSFODNN7EXAMPLE"}
+
+t0 = time.perf_counter()
+for _ in range(REPS):
+    _so_key(so_empty)
+t1 = time.perf_counter()
+print(
+    f"_so_key (empty) ×{REPS}: {(t1 - t0) * 1000:.2f} ms total  "
+    f"({(t1 - t0) / REPS * 1e9:.1f} ns/call)"
+)
+
+t0 = time.perf_counter()
+for _ in range(REPS):
+    _so_key(so_aws)
+t1 = time.perf_counter()
+print(
+    f"_so_key (2 keys) ×{REPS}: {(t1 - t0) * 1000:.2f} ms total  "
+    f"({(t1 - t0) / REPS * 1e9:.1f} ns/call)"
+)
+print(f"  _so_key(aws): {_so_key(so_aws)}")
+
+
+# ── 2. _entity_name() — class → string ───────────────────────────────────────
+_section("2. _entity_name()  [class → name string, 'Entity' suffix strip]")
+REPS_EN = 1_000_000
+t0 = time.perf_counter()
+for _ in range(REPS_EN):
+    _entity_name(MWSEntity)
+t1 = time.perf_counter()
+print(
+    f"_entity_name ×{REPS_EN}: {(t1 - t0) * 1000:.2f} ms total  "
+    f"({(t1 - t0) / REPS_EN * 1e9:.1f} ns/call)"
+)
+print(f"  _entity_name(MWSEntity): {_entity_name(MWSEntity)!r}  (expect 'mws')")
+
+
+# ── 3. BaseEntity._resolve() — local relative path ───────────────────────────
+_section("3. BaseEntity._resolve()  [relative → absolute local path]")
+REPS_R = 100_000
+t0 = time.perf_counter()
+for _ in range(REPS_R):
+    entity._resolve(entity.static_path)
+t1 = time.perf_counter()
+print(
+    f"_resolve (relative) ×{REPS_R}: {(t1 - t0) * 1000:.2f} ms total  "
+    f"({(t1 - t0) / REPS_R * 1e6:.2f} µs/call)"
+)
+print(f"  resolved: {entity._resolve(entity.static_path)!r}")
+
+# Cloud URI fast-path (no pathlib.Path, no existence check).
+entity_cloud_sim = MWSEntity.__new__(MWSEntity)  # bypass __init__ for testing
+entity_cloud_sim._data_root = "s3://my-bucket/data"
+entity_cloud_sim._storage_options = {}
+REPS_CR = 100_000
+t0 = time.perf_counter()
+for _ in range(REPS_CR):
+    entity_cloud_sim._resolve("s3://my-bucket/data/mws/static/")
+t1 = time.perf_counter()
+print(
+    f"_resolve (cloud abs) ×{REPS_CR}: {(t1 - t0) * 1000:.2f} ms total  "
+    f"({(t1 - t0) / REPS_CR * 1e6:.2f} µs/call)"
+)
+
+
+# ── 4. BaseEntity.schema_profile — instance cache ────────────────────────────
+_section("4. BaseEntity.schema_profile  [hasattr guard — instance-level cache]")
+entity2 = MWSEntity(data_root=DATA_ROOT)
+t0 = time.perf_counter()
+entity2.schema_profile  # cold — triggers _cached_detect
+t1 = time.perf_counter()
+print(f"schema_profile (cold) : {(t1 - t0) * 1000:.2f} ms")
+
+REPS_SP = 100_000
+t0 = time.perf_counter()
+for _ in range(REPS_SP):
+    entity2.schema_profile  # warm — hasattr guard short-circuits
+t1 = time.perf_counter()
+print(
+    f"schema_profile (warm) ×{REPS_SP}: {(t1 - t0) * 1000:.2f} ms total  "
+    f"({(t1 - t0) / REPS_SP * 1e6:.3f} µs/call)"
+)
+
+
+# ── 5. _cached_detect() — process-level functools.cache ──────────────────────
+_section("5. _cached_detect()  [process-level functools.cache lookup]")
+profile = entity.schema_profile
+key = (
+    static_path,
+    tuple(entity.key_cols),
+    entity.geometry_col,
+    entity._resolve(entity.annual_path) if entity.annual_path else None,
+    entity._resolve(entity.sub_annual_path) if entity.sub_annual_path else None,
+    _so_key(entity._storage_options),
+)
+# Prime the cache.
+_cached_detect(*key)
+
+REPS_CD = 100_000
+t0 = time.perf_counter()
+for _ in range(REPS_CD):
+    _cached_detect(*key)
+t1 = time.perf_counter()
+print(
+    f"_cached_detect (warm) ×{REPS_CD}: {(t1 - t0) * 1000:.2f} ms total  "
+    f"({(t1 - t0) / REPS_CD * 1e6:.3f} µs/call)"
+)
+print(f"Cache info: {_cached_detect.cache_info()}")
+
+
+# ── 6. BaseEntity._index — process-level bbox cache ──────────────────────────
+_section("6. BaseEntity._index  [process-level bbox index cache]")
+t0 = time.perf_counter()
+idx = entity._index  # may be warm from earlier modules
+t1 = time.perf_counter()
+print(f"_index (first access) : {(t1 - t0) * 1000:.2f} ms")
+print(f"Index shape           : {idx.shape}")
+
+REPS_IX = 100_000
+t0 = time.perf_counter()
+for _ in range(REPS_IX):
+    entity._index  # warm — hasattr guard short-circuits
+t1 = time.perf_counter()
+print(
+    f"_index (cached) ×{REPS_IX}: {(t1 - t0) * 1000:.2f} ms total  "
+    f"({(t1 - t0) / REPS_IX * 1e6:.3f} µs/call)"
+)
+
+
+# ── 7. BaseEntity.geometry_lazy — lazy geometry LazyFrame ────────────────────
+_section("7. BaseEntity.geometry_lazy  [lazy scan construction + hasattr cache]")
+entity3 = MWSEntity(data_root=DATA_ROOT)
+t0 = time.perf_counter()
+lf = entity3.geometry_lazy  # cold — builds pl.scan_parquet plan
+t1 = time.perf_counter()
+print(f"geometry_lazy (cold)  : {(t1 - t0) * 1000:.2f} ms")
+
+REPS_GL = 100_000
+t0 = time.perf_counter()
+for _ in range(REPS_GL):
+    entity3.geometry_lazy  # warm — hasattr guard short-circuits
+t1 = time.perf_counter()
+print(
+    f"geometry_lazy (warm) ×{REPS_GL}: {(t1 - t0) * 1000:.2f} ms total  "
+    f"({(t1 - t0) / REPS_GL * 1e6:.3f} µs/call)"
+)
+
+
+# ── 8. BaseEntity.where() — single attribute filter ──────────────────────────
+_section("8. BaseEntity.where(state='Bihar')  [single attribute filter]")
+t0 = time.perf_counter()
+view_attr = entity.where(state="Bihar")
+t1 = time.perf_counter()
+print(f"where(state)         : {(t1 - t0) * 1000:.2f} ms")
+print(f"Bihar keys           : {view_attr.keys.collect().shape}")
+
+
+# ── 9. BaseEntity.where() — composite attribute filter ───────────────────────
+_section("9. BaseEntity.where(state=..., district=..., tehsil=...)  [composite AND]")
+t0 = time.perf_counter()
+view_composite = entity.where(state="Bihar", district="Banka", tehsil="Banka")
+t1 = time.perf_counter()
+print(f"where(3 cols)        : {(t1 - t0) * 1000:.2f} ms")
+print(f"Banka keys           : {view_composite.keys.collect().shape}")
+
+
+# ── 10. BaseEntity.where() — list value (is_in) ──────────────────────────────
+_section("10. BaseEntity.where(state=['Bihar', 'Jharkhand'])  [list value → is_in]")
+t0 = time.perf_counter()
+view_list = entity.where(state=["Bihar", "Jharkhand"])
+t1 = time.perf_counter()
+print(f"where(list)          : {(t1 - t0) * 1000:.2f} ms")
+print(f"Bihar+Jharkhand keys : {view_list.keys.collect().shape}")
+
+
+# ── 11. BaseEntity.where() — unknown kwarg error ─────────────────────────────
+_section("11. BaseEntity.where()  [unknown kwarg → ValueError guard]")
+err_raised = False
+try:
+    entity.where(totally_unknown_column="xyz")
+except ValueError:
+    err_raised = True
+print(f"ValueError guard     : {err_raised} (expect True)")
+
+
+# ── 12. BaseEntity.spatial_filter() — bbox mode ──────────────────────────────
+_section("12. BaseEntity.spatial_filter(bbox=SMALL_BBOX)  [full pipeline]")
+t0 = time.perf_counter()
+view_sf = entity.spatial_filter(bbox=SMALL_BBOX)
+t1 = time.perf_counter()
+print(f"spatial_filter (bbox): {(t1 - t0) * 1000:.2f} ms")
+print(f"Keys                 : {view_sf.keys.collect().shape}")
+
+# geometry mode
+small_poly = sgeom.box(*SMALL_BBOX)
+t0 = time.perf_counter()
+view_sf2 = entity.spatial_filter(geometry=small_poly)
+t1 = time.perf_counter()
+print(f"spatial_filter (geom): {(t1 - t0) * 1000:.2f} ms")
+
+# area relationship
+t0 = time.perf_counter()
+view_sf_area = entity.spatial_filter(
+    bbox=SMALL_BBOX, relationship="area", threshold=0.3
+)
+t1 = time.perf_counter()
+print(
+    f"spatial_filter (area): {(t1 - t0) * 1000:.2f} ms  keys={view_sf_area.keys.collect().shape}"
+)
+
+
+# ── 13. BaseEntity.spatial_join() — deferred spec ────────────────────────────
+_section("13. BaseEntity.spatial_join()  [deferred spec construction — no I/O]")
+REPS_SJ = 10_000
+t0 = time.perf_counter()
+for _ in range(REPS_SJ):
+    entity.spatial_join(other=entity, agg={"area_in_ha": "sum"})
+t1 = time.perf_counter()
+print(
+    f"spatial_join (spec) ×{REPS_SJ}: {(t1 - t0) * 1000:.2f} ms total  "
+    f"({(t1 - t0) / REPS_SJ * 1e6:.2f} µs/call)"
+)
+
+
+# ── 14. EntityValidationError — import / raise overhead ──────────────────────
+_section("14. EntityValidationError  [raise + catch overhead]")
+REPS_EV = 100_000
+t0 = time.perf_counter()
+for _ in range(REPS_EV):
+    try:
+        raise EntityValidationError("bench test")
+    except EntityValidationError:
+        pass
+t1 = time.perf_counter()
+print(
+    f"raise+catch ×{REPS_EV}: {(t1 - t0) * 1000:.2f} ms total  "
+    f"({(t1 - t0) / REPS_EV * 1e6:.2f} µs/call)"
+)
+
+
+# ── Cleanup ───────────────────────────────────────────────────────────────────
+AoI.deregister(MWSEntity)
+print("\n✓ bench_entity.py complete")

{core_lens-0.1.dev169 → core_lens-0.1.dev172}/benchmarks/bench_export.py

@@ -42,8 +42,8 @@ AoI.register(MWSEntity)
 aoi_all = AoI(DATA_ROOT, bbox=INDIA_BBOX)
 aoi_small = AoI(DATA_ROOT, bbox=SMALL_BBOX)
 
-result_all = aoi_all.mws.static  # full dataset — geometry present
-result_small = aoi_small.mws.static  # small subset
+result_all = aoi_all.mws.static.materialise()  # full dataset — geometry present
+result_small = aoi_small.mws.static.materialise()  # small subset
 
 print(f"All   result : {result_all.df().shape}")
 print(f"Small result : {result_small.df().shape}")
@@ -205,6 +205,69 @@ with tempfile.TemporaryDirectory() as tmpdir:
         f"Ratio      : geojson is {t_gj / t_gpq:.2f}× {'slower' if t_gj > t_gpq else 'faster'}"
     )
 
+    # ── 8. Export error paths — geometry guards ───────────────────────────────
+    _section("8. Export error paths  [TypeError guards — no write, O(1)]")
+
+    errs = 0
+
+    # parquet() with geometry raises TypeError.
+    try:
+        parquet(result_small, tmp / "err.parquet")
+    except TypeError:
+        errs += 1
+
+    # json() with geometry raises TypeError.
+    try:
+        json(result_small, tmp / "err.json")
+    except TypeError:
+        errs += 1
+
+    # csv() with geometry raises TypeError.
+    try:
+        csv(result_small, tmp / "err.csv")
+    except TypeError:
+        errs += 1
+
+    # geoparquet() without geometry raises TypeError.
+    try:
+        geoparquet(result_no_geom, tmp / "err.geoparquet")
+    except TypeError:
+        errs += 1
+
+    # geojson() without geometry raises TypeError.
+    try:
+        geojson(result_no_geom, tmp / "err.geojson")
+    except TypeError:
+        errs += 1
+
+    print(f"TypeError guards raised  : {errs}/5 (expect 5)")
+
+    # ── 9. GeoJSONSeq (newline-delimited GeoJSON) driver ─────────────────────
+    _section("9. geojson(driver='GeoJSONSeq')  [streaming newline-delimited GeoJSON]")
+    out_gjseq = tmp / "bench.geojsonl"
+    t0 = time.perf_counter()
+    for _ in range(REPS_IO):
+        geojson(result_small, out_gjseq, driver="GeoJSONSeq")
+    t1 = time.perf_counter()
+    sz = out_gjseq.stat().st_size
+    print(
+        f"GeoJSONSeq (small) ×{REPS_IO}: {(t1 - t0) * 1000:.2f} ms total  "
+        f"({(t1 - t0) / REPS_IO * 1000:.2f} ms/call)  → {sz // 1024} KB"
+    )
+
+    # ── 10. Format comparison summary (small dataset) ─────────────────────────
+    _section("10. Format size comparison  [small dataset]")
+    sizes = {
+        "parquet": (tmp / "bench.parquet").stat().st_size,
+        "json": (tmp / "bench.json").stat().st_size,
+        "csv": (tmp / "bench.csv").stat().st_size,
+        "geoparquet": (tmp / "bench.geoparquet").stat().st_size,
+        "geojson": (tmp / "bench.geojson").stat().st_size,
+        "geojsonseq": (tmp / "bench.geojsonl").stat().st_size,
+    }
+    for fmt, sz_bytes in sorted(sizes.items(), key=lambda x: x[1]):
+        print(f"  {fmt:12} : {sz_bytes // 1024:6} KB")
+
 
 # ── Cleanup ───────────────────────────────────────────────────────────────────
 AoI.deregister(MWSEntity)