core-lens 0.1.dev180__tar.gz → 0.1.dev183__tar.gz

{core_lens-0.1.dev180 → core_lens-0.1.dev183}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: core-lens
-Version: 0.1.dev180
+Version: 0.1.dev183
 Summary: Query, analyse, and visualise CoreStack's microwatershed and Earth science data through a clean, composable Python API.
 Project-URL: Homepage, https://github.com/ApoorvaKashyap/core-lens
 Project-URL: Issues, https://github.com/ApoorvaKashyap/core-lens/issues

core_lens-0.1.dev183/SKILLS.md

@@ -0,0 +1,242 @@
+---
+name: core-lens
+description: >
+  A Python library for querying, analysing, and visualising geospatial Earth science data.
+  Provides a unified interface over microwatersheds (MWS), administrative boundaries, and pluggable domain entities.
+  Use this skill to understand how to interact with AoI (Area of Interest), Views, Results, and Statistical analysis.
+---
+
+# Agent Instructions: Core-Lens Library
+
+You are an AI Agent tasked with writing or debugging code using the `core-lens` library. `core-lens` is a high-performance Python library designed for querying, filtering, and manipulating large-scale geospatial datasets using deferred execution with **Polars**.
+
+**Read these instructions carefully before writing any code.**
+
+## 1. Core Architectural Constraints
+
+- **Area of Interest (AoI) is the Primary Entry Point**: Never load parquet files directly. Always create an `AoI` object to define the geographical scope and directory context.
+- **Lazy Evaluation**: `core-lens` is lazy by default. Accessing an entity like `aoi.mws` creates a `View`. No data is loaded into memory until you call a materialisation method (`.static`, `.annual`, or `.sub_annual`).
+- **Polars Over Pandas**: `core-lens` uses Polars for zero-copy operations. Only convert to GeoPandas (`.gdf()`) when strictly necessary for heavy geometric processing.
+- **Separation of Concerns**:
+  - `AoI`: Filters boundaries, resolves geography, scopes entities, and handles time constraints.
+  - `Result`: Produced by materializing a view. Handles all computation, spatial joins, aggregations, stats, and plotting.
+  - `Export`: Separate namespace (`from core_lens.export import geoparquet`).
+
+## 1.5 Logging and Tracing
+
+`core_lens` uses Loguru and is disabled by default. If you need to debug spatial resolution issues or trace validation errors, enable the internal logger:
+
+```python
+import sys
+from loguru import logger
+
+logger.enable("core_lens")
+logger.add(sys.stdout, level="DEBUG")
+```
+
+## 2. Initialization and Setup
+
+Before querying, you MUST register the specific entities you plan to use.
+
+```python
+from core_lens import AoI, SeasonConfig
+from core_lens.entities import MWSEntity, TehsilEntity, DistrictEntity
+
+# 1. Register entities FIRST
+AoI.register(MWSEntity)
+AoI.register(DistrictEntity)
+# ... register others as needed
+
+# 2. Scoping the AoI (Choose ONE mode)
+DATA_ROOT = "data/"
+# Mode A: By Administrative Hierarchy
+aoi = AoI(DATA_ROOT, tehsil="Pangi", district="Chamba")
+# Mode B: By Bounding Box
+aoi = AoI(DATA_ROOT, bbox=(76.0, 31.0, 78.0, 33.0))
+# Mode C: By specific Entity ID
+aoi = AoI(DATA_ROOT, mws_id="13_551")
+
+# Optional: Override default seasons (kharif, rabi, zaid)
+custom_seasons = SeasonConfig(kharif=("06-01", "10-15"), rabi=("10-16", "02-28"), zaid=("03-01", "05-31"))
+aoi_custom = AoI(DATA_ROOT, tehsil="Pangi", seasons=custom_seasons)
+```
+
+## 3. Querying & Filtering (Views)
+
+Entities on the `AoI` object are accessed as **Views**. Views are lazy and can be chained.
+
+```python
+# Access the view
+mws_view = aoi.mws
+
+# Attribute Filtering
+filtered = mws_view.where(state="Himachal Pradesh")
+
+# Spatial Filtering (inside the AoI)
+spatial_filtered = mws_view.spatial_filter(bbox=(76.5, 31.5, 77.5, 32.5))
+
+# Temporal Filtering
+# Note: You can filter by exact years, a range of years, or predefined seasons.
+from core_lens.base.view import Season
+temporal_view = mws_view.between(year=(2020, 2022), season=Season.KHARIF)
+current_view = mws_view.between(season=Season.CURRENT)
+```
+
+## 4. Materialization (Loading Data)
+
+You must materialize the view into a `Result` to perform computations.
+Choose the resolution based on what you need:
+- `.static`: One row per entity, includes `geometry`.
+- `.annual`: Time-series, aggregated annually. (NO geometry by default).
+- `.sub_annual`: High-frequency time-series. (NO geometry by default).
+
+```python
+static_result = filtered.static.materialise()
+annual_result = temporal_view.annual.materialise()
+```
+*Note: Depending on library version, you may only need `.static` instead of `.static.materialise()`. If `.static` returns a Result directly, use that.*
+
+```python
+# To access underlying data structures:
+df = static_result.df()        # Polars DataFrame (Zero-copy)
+lazy_df = static_result.lazy() # Polars LazyFrame
+gdf = static_result.gdf()      # GeoPandas GeoDataFrame (Heavy!)
+```
+
+## 5. Computation and Data Manipulation (Result API)
+
+All data manipulation happens on the `Result` object.
+
+```python
+import polars as pl
+
+# Derive new columns
+res = annual_result.derive("water_shortage", pl.col("rainfall") - pl.col("storage"))
+
+# Aggregation (Supported 'by' keys: "year", "month", "season", "season_year")
+agg_res = res.aggregate(pl.mean("ndvi").alias("avg_ndvi"), by="year")
+
+# Reattaching Geometry
+# Time-series results drop geometry to save memory. Reattach it if you need to map them:
+geo_result = agg_res.with_geometry()
+```
+
+## 6. Spatial Joins
+
+Spatial joins are lazy and enrich a primary entity with aggregated data from a secondary entity.
+
+```python
+# Calculate the total forest tree_cover inside each MWS
+joined_view = aoi.mws.spatial_join(
+    aoi.forest, agg={"tree_cover": "sum", "canopy_density": "mean"}
+)
+# Resulting columns are prefixed, e.g., 'forest_tree_cover', 'forest_canopy_density'
+```
+
+## 7. Statistical Analysis (`Result.stats`)
+
+All stats methods are available under `result.stats` and return a NEW `Result`.
+- Computed data is in `result.data` (which is a Polars DataFrame)
+- Parameters and testing metrics are stored in `result.metadata`
+
+```python
+# Descriptive stats
+result.stats.describe(columns=["ndvi", "rainfall"])
+
+# Correlation (pearson, spearman, kendall)
+from core_lens.base.namespaces.stats import CorrelateMethod
+result.stats.correlate(columns=["ndvi", "rainfall"], method=CorrelateMethod.PEARSON, across="entity")
+
+# Hypothesis Testing (t-test, mann-whitney, wilcoxon, ks, chi-square)
+from core_lens.base.namespaces.stats import TestMethod
+result.stats.test(column="cropping_intensity", groups="temperature_zone", method=TestMethod.MANN_WHITNEY)
+
+# Change Detection (absolute, percentage, trend)
+from core_lens.base.namespaces.stats import ChangeMethod
+result.stats.change(column="tree_cover", from_period=2018, to_period=2023, method=ChangeMethod.PERCENTAGE)
+
+# Anomaly Detection
+from core_lens.base.namespaces.stats import AnomalyTsMethod, AnomalyCrossMethod
+# Mode 1: Cross-sectional (zscore, iqr, percentile, threshold)
+result.stats.anomaly(column="ndvi", mode="cross_sectional", method=AnomalyCrossMethod.ZSCORE, baseline=(2010, 2020))
+# Mode 2: Time-series (stl, cusum, mad)
+result.stats.anomaly(column="ndvi", mode="timeseries", method=AnomalyTsMethod.STL, baseline=(2010, 2018))
+
+# Similarity Matching (euclidean, cosine, mahalanobis, manhattan)
+from core_lens.base.namespaces.stats import SimilarityMethod
+result.stats.similarity(
+    target="13_551",
+    columns={"rainfall": ("annual", {"year": 2018})},
+    method=SimilarityMethod.EUCLIDEAN,
+    top_n=10
+)
+```
+
+## 8. Plotting and Visualization (`Result.plot`)
+
+Plots are explicitly accessed via the `result.plot` namespace. Maps use Lonboard (requires geometry); charts use Plotly. By default, charts render the top 10 entities.
+
+```python
+# Map plotting (Requires result.with_geometry() to have been called for time-series)
+result.plot.choropleth("ndvi", subplot_on="season_year")
+
+# Charts
+result.plot.timeseries(x="year", y="ndvi", top_n=10)
+result.plot.scatter(x="rainfall", y="ndvi", top_n=10)
+result.plot.distribution(x="ndvi", top_n=10)
+result.plot.correlation(columns=["ndvi", "rainfall"], top_n=10)
+result.plot.heatmap(x="year", y="month", value="ndvi", top_n=10)
+result.plot.matrix()
+
+# Plotting the AoI boundary itself
+aoi.plot()
+aoi.plot(overlay=result)
+```
+
+## 9. Exporting Results
+
+Exporting uses standalone functions. Do not export massive datasets (like country-wide `mws`) to GeoJSON as it will bloat the file size. Use GeoParquet instead.
+
+```python
+from core_lens.export import geoparquet, parquet, geojson
+
+# For results with geometry
+geoparquet(result, "output.geoparquet")
+
+# For tabular results without geometry
+parquet(result, "output.parquet")
+```
+
+## 10. Plugin System (Custom Entities)
+
+You can extend core-lens with custom entities by subclassing `BaseEntity`.
+
+```python
+from core_lens.base import BaseEntity
+
+class CustomEntity(BaseEntity):
+    @property
+    def key_cols(self) -> list[str]:
+        return ["custom_id"]
+
+    @property
+    def geometry_col(self) -> str:
+        return "geometry"
+
+    @property
+    def static_path(self) -> str:
+        return "custom/static.geoparquet"
+
+    @property
+    def annual_path(self) -> str | None:
+        return "custom/annual.parquet"
+
+AoI.register(CustomEntity)
+```
+
+## Agent Reminders & Pitfalls to Avoid
+- **DO NOT** attempt to read parquet files directly using `pandas.read_parquet` or `polars.read_parquet`. Always construct an `AoI` and use the library's materialized `Result` output.
+- **DO NOT** assume `annual` or `sub_annual` results contain geometry. You must call `.with_geometry()` to reattach it.
+- **DO NOT** forget to import and register entities before initializing the `AoI`.
+- **DO** use Polars expressions (`import polars as pl`) inside `.derive()` and `.aggregate()` methods for maximum performance.
+- **DO** always read parameters/testing metrics from `result.metadata` and underlying stats from `result.data` after running any methods in `result.stats`.

{core_lens-0.1.dev180 → core_lens-0.1.dev183}/docs/source/export.md

@@ -32,12 +32,7 @@ from core_lens.export import geoparquet, geojson
 spatial_res = aoi.mws.annual.with_geometry()
 
 # Export to GeoParquet with DuckDB COPY options (e.g., partitioning)
-geoparquet(
-    spatial_res,
-    "output.geoparquet",
-    compression="ZSTD",
-    partition_by="year"
-)
+geoparquet(spatial_res, "output.geoparquet", compression="ZSTD", partition_by="year")
 
 # Export to GeoJSON
 geojson(spatial_res, "output.json")

{core_lens-0.1.dev180 → core_lens-0.1.dev183}/docs/source/plugins.md

@@ -5,6 +5,7 @@ You can extend CoreLens by defining your own entities. Simply subclass `BaseEnti
 ```python
 from core_lens.base import BaseEntity
 
+
 class CustomEntity(BaseEntity):
     @property
     def key_cols(self) -> list[str]:
@@ -22,6 +23,7 @@ class CustomEntity(BaseEntity):
     def annual_path(self) -> str | None:
         return "custom/annual.parquet"
 
+
 # Register before use
 AoI.register(CustomEntity)
 ```

{core_lens-0.1.dev180 → core_lens-0.1.dev183}/docs/source/queries.md

@@ -27,8 +27,7 @@ import polars as pl
 # 1. Derive new columns (works on any Result)
 res_annual = aoi.mws.annual
 derived_res = res_annual.derive(
-    "drought_flag",
-    pl.when(pl.col("rainfall") < 500).then(1).otherwise(0)
+    "drought_flag", pl.when(pl.col("rainfall") < 500).then(1).otherwise(0)
 )
 
 # 2. Aggregate (temporal grouping like "month" requires sub-annual, but "year" works on annual too)
@@ -47,11 +46,7 @@ Spatially join data from two different entities (e.g. Microwatersheds and Forest
 
 # Join forest metrics onto MWS geometries
 joined_view = aoi.mws.spatial_join(
-    aoi.forest,
-    agg={
-        "tree_cover": "area",
-        "canopy_density": "mean"
-    }
+    aoi.forest, agg={"tree_cover": "area", "canopy_density": "mean"}
 )
 
 # Materialise the joined view

{core_lens-0.1.dev180 → core_lens-0.1.dev183}/docs/source/quickstart.md

@@ -29,15 +29,11 @@ Create an `AoI` (Area of Interest) by specifying the `data_root` (which can be a
 
 ```python
 # Create an AoI scoped to a specific Tehsil using a local path
-aoi = AoI(
-    data_root="/path/to/data",
-    tehsil="Pangi"
-)
+aoi = AoI(data_root="/path/to/data", tehsil="Pangi")
 
 # Or scope it directly to a specific list of entity IDs using an S3 bucket
 aoi_specific = AoI(
-    data_root="s3://my-cloud-bucket/core-lens-data",
-    mws_id=["13_001", "13_002"]
+    data_root="s3://my-cloud-bucket/core-lens-data", mws_id=["13_001", "13_002"]
 )
 
 # You can now access entities scoped to this AoI:

{core_lens-0.1.dev180 → core_lens-0.1.dev183}/docs/source/stats.md

@@ -21,8 +21,8 @@ from core_lens.base.namespaces.stats import CorrelateMethod
 
 corr = res.stats.correlate(
     columns=["ndvi", "rainfall", "temperature"],
-    method=CorrelateMethod.PEARSON, # or SPEARMAN, KENDALL
-    across="entity"      # correlate across entities or time
+    method=CorrelateMethod.PEARSON,  # or SPEARMAN, KENDALL
+    across="entity",  # correlate across entities or time
 )
 ```
 
@@ -37,14 +37,12 @@ from core_lens.base.namespaces.stats import TestMethod
 test_res = res.stats.test(
     column="cropping_intensity",
     groups="temperature_zone",
-    method=TestMethod.MANN_WHITNEY
+    method=TestMethod.MANN_WHITNEY,
 )
 
 # Period-based testing
 test_period = res.stats.test(
-    column="ndvi",
-    periods=[(2010, 2015), (2016, 2023)],
-    method=TestMethod.T_TEST
+    column="ndvi", periods=[(2010, 2015), (2016, 2023)], method=TestMethod.T_TEST
 )
 ```
 
@@ -57,10 +55,7 @@ from core_lens.base.namespaces.stats import ChangeMethod
 
 # Trend over time
 trend = res.stats.change(
-    column="ndvi",
-    from_period=2010,
-    to_period=2023,
-    method=ChangeMethod.TREND
+    column="ndvi", from_period=2010, to_period=2023, method=ChangeMethod.TREND
 )
 
 # Absolute or percentage change
@@ -68,7 +63,7 @@ pct_change = res.stats.change(
     column="tree_cover",
     from_period=2018,
     to_period=2023,
-    method=ChangeMethod.PERCENTAGE
+    method=ChangeMethod.PERCENTAGE,
 )
 ```
 
@@ -81,10 +76,7 @@ from core_lens.base.namespaces.stats import AnomalyTsMethod, AnomalyCrossMethod
 
 # Timeseries anomaly against its own history
 ts_anomalies = res.stats.anomaly(
-    column="ndvi",
-    mode="timeseries",
-    method=AnomalyTsMethod.STL,
-    baseline=(2010, 2018)
+    column="ndvi", mode="timeseries", method=AnomalyTsMethod.STL, baseline=(2010, 2018)
 )
 
 # Cross-sectional anomaly against other entities
@@ -92,7 +84,7 @@ cross_anomalies = res.stats.anomaly(
     column="ndvi",
     mode="cross_sectional",
     method=AnomalyCrossMethod.ZSCORE,
-    baseline=(2010, 2020)
+    baseline=(2010, 2020),
 )
 ```
 
@@ -107,9 +99,9 @@ similar = res.stats.similarity(
     target="13_551",
     columns={
         "rainfall": ("annual", {"year": 2018}),
-        "ndvi": ("sub_annual", {"season": "kharif", "year": 2020})
+        "ndvi": ("sub_annual", {"season": "kharif", "year": 2020}),
     },
     method=SimilarityMethod.EUCLIDEAN,
-    top_n=10
+    top_n=10,
 )
 ```

{core_lens-0.1.dev180 → core_lens-0.1.dev183}/src/core_lens/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.1.dev180'
-__version_tuple__ = version_tuple = (0, 1, 'dev180')
+__version__ = version = '0.1.dev183'
+__version_tuple__ = version_tuple = (0, 1, 'dev183')
 
 __commit_id__ = commit_id = None

{core_lens-0.1.dev180 → core_lens-0.1.dev183}/src/core_lens/schema/detection.py

@@ -102,10 +102,12 @@ def detect(
     if lon_col:
         reserved_static.add(lon_col)
     extra_static_cols = [c for c in static_schema if c not in reserved_static]
+    col_types_static = {col: str(dtype) for col, dtype in static_schema.items()}
 
     annual_time_col: str | None = None
     extra_annual_cols: list[str] = []
     annual_is_year_col: bool | None = None
+    col_types_annual: dict[str, str] = {}
     if annual_path:
         annual_schema = _read_schema(
             annual_path, label="annual", storage_options=_so or None
@@ -116,10 +118,12 @@ def detect(
         )
         extra_annual_cols = [c for c in annual_schema if c not in reserved_annual]
         annual_is_year_col = _is_year_col_from_schema(annual_schema, annual_time_col)
+        col_types_annual = {col: str(dtype) for col, dtype in annual_schema.items()}
 
     sub_annual_time_col: str | None = None
     extra_sub_annual_cols: list[str] = []
     sub_annual_is_year_col: bool | None = None
+    col_types_sub_annual: dict[str, str] = {}
     if sub_annual_path:
         sub_annual_schema = _read_schema(
             sub_annual_path, label="sub_annual", storage_options=_so or None
@@ -132,6 +136,9 @@ def detect(
         sub_annual_is_year_col = _is_year_col_from_schema(
             sub_annual_schema, sub_annual_time_col
         )
+        col_types_sub_annual = {
+            col: str(dtype) for col, dtype in sub_annual_schema.items()
+        }
 
     return SchemaProfile(
         key_cols=key_cols,
@@ -143,6 +150,9 @@ def detect(
         extra_static_cols=extra_static_cols,
         extra_annual_cols=extra_annual_cols,
         extra_sub_annual_cols=extra_sub_annual_cols,
+        col_types_static=col_types_static,
+        col_types_annual=col_types_annual,
+        col_types_sub_annual=col_types_sub_annual,
         annual_is_year_col=annual_is_year_col,
         sub_annual_is_year_col=sub_annual_is_year_col,
     )
@@ -153,10 +163,18 @@ def _read_schema(
     label: str,
     storage_options: dict[str, Any] | None = None,
 ) -> pl.Schema:
+    import os
+
     _so = storage_options or {}
+    # When path is a directory, use a glob to read only .parquet files.
+    # This prevents polars from raising InvalidOperationError when the directory
+    # contains mixed-extension files (e.g. data_dictionary.csv alongside .parquet).
+    scan_path = path
+    if os.path.isdir(path):
+        scan_path = os.path.join(path, "**", "*.parquet")
     try:
         return pl.scan_parquet(
-            path,
+            scan_path,
             hive_partitioning=True,
             storage_options=_so or None,
         ).collect_schema()

{core_lens-0.1.dev180 → core_lens-0.1.dev183}/src/core_lens/schema/profile.py

@@ -56,6 +56,12 @@ class SchemaProfile(BaseModel):
             is used for validation and documentation only.
         extra_annual_cols: Additional attribute columns in the annual file.
         extra_sub_annual_cols: Additional attribute columns in the sub_annual file.
+        col_types_static: Mapping of **all** column names in the static file to
+            their Polars dtype string (e.g. ``{"basin_id": "Int32", "geometry": "Binary"}``).
+            Populated by the auto-detection layer; ``{}`` when the profile is
+            constructed manually.
+        col_types_annual: Same as ``col_types_static`` but for the annual file.
+        col_types_sub_annual: Same as ``col_types_static`` but for the sub_annual file.
 
     """
 
@@ -70,6 +76,11 @@ class SchemaProfile(BaseModel):
     extra_static_cols: list[str] = []
     extra_annual_cols: list[str] = []
     extra_sub_annual_cols: list[str] = []
+    # Column name → Polars dtype string for every column in each resolution.
+    # Populated by the auto-detection layer; empty dict when built manually.
+    col_types_static: dict[str, str] = {}
+    col_types_annual: dict[str, str] = {}
+    col_types_sub_annual: dict[str, str] = {}
     # Whether the time column is an integer-year column (True), a Date/Datetime
     # column (False), or absent/unknown (None).  Determined at detect() time from
     # the Parquet schema — eliminates a redundant collect_schema() call in

{core_lens-0.1.dev180 → core_lens-0.1.dev183}/usage.md

@@ -89,8 +89,8 @@ spatial_view = mws_view.spatial_filter(bbox=(76.5, 31.5, 77.5, 32.5))
 from core_lens.base.view import Season
 
 temporal_view = mws_view.between(
-    year=(2020, 2022),       # Inclusive year range
-    season=Season.KHARIF     # Filter to specific season
+    year=(2020, 2022),  # Inclusive year range
+    season=Season.KHARIF,  # Filter to specific season
 )
 ```
 
@@ -101,8 +101,7 @@ You can define a spatial join to enrich one entity with aggregated data from ano
 ```python
 # Join MWS with Tehsil entity to calculate the sum of tehsil areas per MWS
 joined_view = mws_view.spatial_join(
-    other=TehsilEntity(data_root=DATA_ROOT),
-    agg={"Shape_Area": "sum"}
+    other=TehsilEntity(data_root=DATA_ROOT), agg={"Shape_Area": "sum"}
 )
 ```
 
@@ -127,22 +126,24 @@ A `Result` object provides access to the materialized data. `core-lens` uses zer
 
 ```python
 # 1. Zero-copy access (Instantaneous)
-pl_df = result_static.df()          # Polars DataFrame
-pl_lazy = result_static.lazy()      # Polars LazyFrame
+pl_df = result_static.df()  # Polars DataFrame
+pl_lazy = result_static.lazy()  # Polars LazyFrame
 
 # 2. Decode WKB geometries to Shapely (CPU intensive)
-gdf = result_static.gdf()           # GeoPandas GeoDataFrame
+gdf = result_static.gdf()  # GeoPandas GeoDataFrame
 
 # 3. Data Transformations (Returns a new Result)
 # Derive new columns using Polars expressions
 import polars as pl
-result_derived = result_annual.derive("yield_per_ha", pl.col("yield") / pl.col("area_in_ha"))
+
+result_derived = result_annual.derive(
+    "yield_per_ha", pl.col("yield") / pl.col("area_in_ha")
+)
 
 # 4. Fast aggregations (Returns a new Result)
 # Aggregate by a key column using Polars expressions
 result_agg = result_derived.aggregate(
-    pl.col("yield_per_ha").mean().alias("avg_yield_per_ha"),
-    by="year"
+    pl.col("yield_per_ha").mean().alias("avg_yield_per_ha"), by="year"
 )
 ```
 
@@ -173,9 +174,7 @@ By default, `core-lens` uses Indian agricultural seasons. You can inject a custo
 from core_lens.aoi import SeasonConfig
 
 custom_seasons = SeasonConfig(
-    kharif=("06-01", "10-15"),
-    rabi=("10-16", "02-28"),
-    zaid=("03-01", "05-31")
+    kharif=("06-01", "10-15"), rabi=("10-16", "02-28"), zaid=("03-01", "05-31")
 )
 
 aoi = AoI(DATA_ROOT, bbox=(...), seasons=custom_seasons)