core-lens 0.1.dev164__tar.gz → 0.1.dev166__tar.gz

{core_lens-0.1.dev164 → core_lens-0.1.dev166}/CONTRIBUTING.md

@@ -143,7 +143,7 @@ Examples:
 ```text
 feature/stats-similarity
 feature/plugin-validation
-fix/version-clash-fortnightly
+fix/version-clash-sub-annual
 ```
 
 Workflow:
@@ -204,7 +204,7 @@ Examples:
 
 ```text
 feat: add stats.similarity()
-fix: resolve VersionClashError on fortnightly
+fix: resolve VersionClashError on sub-annual
 docs: add plugin guide
 test: add aggregate validation fixtures
 chore: update ruff

{core_lens-0.1.dev164 → core_lens-0.1.dev166}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: core-lens
-Version: 0.1.dev164
+Version: 0.1.dev166
 Summary: Query, analyse, and visualise CoreStack's microwatershed and Earth science data through a clean, composable Python API.
 Project-URL: Homepage, https://github.com/ApoorvaKashyap/core-lens
 Project-URL: Issues, https://github.com/ApoorvaKashyap/core-lens/issues

{core_lens-0.1.dev164 → core_lens-0.1.dev166}/SKILLS.md

@@ -47,7 +47,7 @@ view.between(season="kharif", year=2020)
 # Materialise into a Result object
 static_result = view.static         # has geometry
 annual_result = view.annual         # no geometry
-fortnightly_result = view.fortnightly # no geometry
+sub_annual_result = view.sub_annual # no geometry
 ```
 
 ## 3. Operations on Results (Compute & Aggregation)
@@ -129,7 +129,7 @@ result.stats.similarity(
     target="13_551",
     columns={
         "rainfall": ("annual", {"year": 2018}),
-        "ndvi": ("fortnightly", {"season": "kharif", "year": 2020})
+        "ndvi": ("sub_annual", {"season": "kharif", "year": 2020})
     },
     method="euclidean",
     top_n=10
@@ -189,8 +189,8 @@ AoI.register(ForestEntity)
 
 ## 9. Execution Environment Context
 
-- **Loading Data**: There is no explicit `AoI.load_data()` method. The `AoI(data_root="...")` constructor acts as the main entry point and defines the directory scope. Data reading happens lazily using Polars `scan_parquet()` when you explicitly call materialization methods like `.static`, `.annual`, or `.fortnightly` on a `View`.
-- **Spatial and Temporal Relationship**: To conserve memory, temporal views (`.annual` and `.fortnightly`) drop geometry columns by default. To establish the spatial relationship between the static boundaries (`mws.parquet`) and the temporal time-series (`annual.parquet`), you use the `Result.with_geometry()` method. This triggers an internal join on the entity's primary key (e.g., `mws_id`) bringing the geometry back into the DataFrame for spatial operations and plotting.
+- **Loading Data**: There is no explicit `AoI.load_data()` method. The `AoI(data_root="...")` constructor acts as the main entry point and defines the directory scope. Data reading happens lazily using Polars `scan_parquet()` when you explicitly call materialization methods like `.static`, `.annual`, or `.sub_annual` on a `View`.
+- **Spatial and Temporal Relationship**: To conserve memory, temporal views (`.annual` and `.sub_annual`) drop geometry columns by default. To establish the spatial relationship between the static boundaries (`mws.parquet`) and the temporal time-series (`annual.parquet`), you use the `Result.with_geometry()` method. This triggers an internal join on the entity's primary key (e.g., `mws_id`) bringing the geometry back into the DataFrame for spatial operations and plotting.
 
 - **Expected Directory Structure**: When initializing `AoI(data_root="data/")`, the library expects a specific folder structure inside the `data_root` for each registered entity. For example, the `MWSEntity` expects:
   ```text
@@ -199,7 +199,7 @@ AoI.register(ForestEntity)
       ├── static/
       │   └── mws.parquet        ← GeoParquet with boundaries
       ├── annual/                ← Directory of Parquets (often partitioned by basin)
-      └── fortnightly/           ← Directory of Parquets (partitioned by year/basin)
+      └── sub-annual/           ← Directory of Parquets (partitioned by year/basin)
   ```
   *(Note: The exact paths for these files/directories are defined within the `BaseEntity` subclasses, e.g., `MWSEntity.static_path`)*
 
@@ -229,9 +229,9 @@ view = aoi.mws.where(ba_name="Barmer")
 temporal_view = view.between(season="kharif", year=2021)
 ```
 
-**5. Materialization:** Trigger the actual Parquet I/O to pull the data into memory by accessing `.static`, `.annual`, or `.fortnightly`.
+**5. Materialization:** Trigger the actual Parquet I/O to pull the data into memory by accessing `.static`, `.annual`, or `.sub_annual`.
 ```python
-result = temporal_view.fortnightly
+result = temporal_view.sub_annual
 ```
 
 **6. Compute & Analysis:** Chain computations, derive metrics, or run statistical tests on the returned `Result`.

core_lens-0.1.dev166/analyze_summaries.py

@@ -0,0 +1,90 @@
+import argparse
+import pandas as pd
+import os
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Analyze summary CSVs for missing data."
+    )
+    parser.add_argument(
+        "district", type=str, help="Name of the district (e.g. 'Sehore')"
+    )
+    parser.add_argument(
+        "--dir",
+        type=str,
+        default="/home/akashyap54/Projects/CoreStack/core-lens/district_summaries",
+        help="Directory with summaries",
+    )
+    args = parser.parse_args()
+
+    safe_dist = str(args.district).replace("/", "_").replace(" ", "_").replace(",", "")
+    dist_dir = os.path.join(args.dir, safe_dist)
+
+    if not os.path.exists(dist_dir):
+        print(f"Error: No summaries found for district '{args.district}' at {dist_dir}")
+        return
+
+    print(f"--- Data Completeness Report for {args.district} ---")
+
+    for cat in ["static", "annual", "sub-annual"]:
+        csv_path = os.path.join(dist_dir, f"{cat}_summary.csv")
+        print(f"\n[{cat.upper()}]")
+
+        if not os.path.exists(csv_path):
+            print("  Status: No Data (Missing File)")
+            continue
+
+        try:
+            df = pd.read_csv(csv_path)
+
+            # Check for completely empty table
+            if len(df) == 0:
+                print("  Status: Empty summary table")
+                continue
+
+            # Depending on DuckDB version, null_percentage could be string '100.0%' or float
+            if "null_percentage" in df.columns:
+                if df["null_percentage"].dtype == object:
+                    df["null_pct_float"] = (
+                        df["null_percentage"]
+                        .astype(str)
+                        .str.rstrip("%")
+                        .astype("float")
+                    )
+                else:
+                    df["null_pct_float"] = df["null_percentage"]
+            else:
+                print("  Status: Unknown schema (no null_percentage column)")
+                continue
+
+            total_cols = len(df)
+            fully_missing = df[df["null_pct_float"] >= 100.0]["column_name"].tolist()
+            partially_missing = df[
+                (df["null_pct_float"] > 0.0) & (df["null_pct_float"] < 100.0)
+            ]
+
+            print(f"  Total Columns: {total_cols}")
+
+            if fully_missing:
+                print(f"  Completely Null Columns ({len(fully_missing)}):")
+                print("    " + ", ".join(fully_missing))
+            else:
+                print("  Completely Null Columns: None")
+
+            if not partially_missing.empty:
+                print(f"  Partially Null Columns ({len(partially_missing)}):")
+                for _, row in partially_missing.iterrows():
+                    print(f"    - {row['column_name']}: {row['null_percentage']} null")
+            else:
+                print("  Partially Null Columns: None")
+
+            if not fully_missing and partially_missing.empty:
+                print("  Status: COMPLETE (No missing values)")
+
+        except Exception as e:
+            print(f"  Error reading {cat}: {e}")
+
+
+if __name__ == "__main__":
+    main()

{core_lens-0.1.dev164 → core_lens-0.1.dev166}/benchmarks/README.md

@@ -10,7 +10,7 @@ benchmarks/
 ├── run_all.sh                  # Run all benchmarks in sequence
 ├── bench_aoi.py                # AoI init, register, named-boundary resolution
 ├── bench_spatial.py            # build_bbox_index, bbox_intersects_geometry, exact_spatial_filter
-├── bench_view.py               # View.where, spatial_filter, between, materialise (static/annual/fortnightly)
+├── bench_view.py               # View.where, spatial_filter, between, materialise (static/annual/sub-annual)
 ├── bench_result.py             # Result.derive, aggregate, with_geometry, df/gdf
 ├── bench_season.py             # resolve_time_filter, add_temporal_columns, SeasonConfig.season_for
 ├── bench_schema.py             # schema detection (detect, _infer_* helpers)

{core_lens-0.1.dev164 → core_lens-0.1.dev166}/benchmarks/bench_result.py

@@ -7,7 +7,7 @@ Targets:
   - Result.with_geometry()      static-join to attach geometry to non-static result
   - Result.derive()             with_columns expression append
   - Result.aggregate()          group_by + agg (no temporal grouping)
-  - Result.aggregate(by=...)    temporal grouping (annual / fortnightly)
+  - Result.aggregate(by=...)    temporal grouping (annual / sub_annual)
 
 Scalene focuses on:
   - shapely.wkb.loads() in gdf() — Python loop over WKB bytes
@@ -130,7 +130,7 @@ print(
 )
 
 
-# ── Note: meaningful with_geometry() test requires annual/fortnightly result ──
+# ── Note: meaningful with_geometry() test requires annual/sub_annual result ──
 # Uncomment with an MWS entity that has annual_path set:
 
 result_annual = aoi.mws.annual  # no geometry

{core_lens-0.1.dev164 → core_lens-0.1.dev166}/benchmarks/bench_schema.py

@@ -36,7 +36,7 @@ DATA_ROOT = "data/"
 entity = MWSEntity(data_root=DATA_ROOT)
 static_path = entity._resolve(entity.static_path)
 annual_path = entity._resolve(entity.annual_path) if entity.annual_path else None
-fn_path = entity._resolve(entity.fortnightly_path) if entity.fortnightly_path else None
+fn_path = entity._resolve(entity.sub_annual_path) if entity.sub_annual_path else None
 
 
 def _section(title: str) -> None:
@@ -104,7 +104,7 @@ print(
 print(f"Detected time col: {_infer_time_col(schema, static_path)!r}")
 
 
-# ── 5. detect() — cold (no annual/fortnightly) ────────────────────────────────
+# ── 5. detect() — cold (no annual/sub_annual) ────────────────────────────────
 _section("5. detect(static only)  [full detection — cold]")
 REPS = 10
 profile = detect(  # ensure bound before loop
@@ -112,7 +112,7 @@ profile = detect(  # ensure bound before loop
     key_cols=entity.key_cols,
     geometry_col=entity.geometry_col,
     annual_path=None,
-    fortnightly_path=None,
+    sub_annual_path=None,
 )
 t0 = time.perf_counter()
 for _ in range(REPS):
@@ -121,7 +121,7 @@ for _ in range(REPS):
         key_cols=entity.key_cols,
         geometry_col=entity.geometry_col,
         annual_path=None,
-        fortnightly_path=None,
+        sub_annual_path=None,
     )
 t1 = time.perf_counter()
 print(
@@ -133,15 +133,15 @@ print(f"Profile bbox_cols     : {profile.bbox_cols!r}")
 print(f"Profile extra_static  : {profile.extra_static_cols}")
 
 
-# ── 6. detect() — with annual + fortnightly ───────────────────────────────────
+# ── 6. detect() — with annual + sub_annual ───────────────────────────────────
 if annual_path or fn_path:
-    _section("6. detect(static + annual + fortnightly)  [3× schema reads]")
+    _section("6. detect(static + annual + sub_annual)  [3× schema reads]")
     profile_full = detect(  # ensure bound before loop
         static_path=static_path,
         key_cols=entity.key_cols,
         geometry_col=entity.geometry_col,
         annual_path=annual_path,
-        fortnightly_path=fn_path,
+        sub_annual_path=fn_path,
     )
     t0 = time.perf_counter()
     for _ in range(REPS):
@@ -150,7 +150,7 @@ if annual_path or fn_path:
             key_cols=entity.key_cols,
             geometry_col=entity.geometry_col,
             annual_path=annual_path,
-            fortnightly_path=fn_path,
+            sub_annual_path=fn_path,
         )
     t1 = time.perf_counter()
     print(
@@ -158,10 +158,10 @@ if annual_path or fn_path:
         f"({(t1 - t0) / REPS * 1000:.2f} ms/call)"
     )
     print(f"annual_time_col      : {profile_full.annual_time_col!r}")
-    print(f"fortnightly_time_col : {profile_full.fortnightly_time_col!r}")
+    print(f"sub_annual_time_col : {profile_full.sub_annual_time_col!r}")
 else:
     print(
-        "\n[skip] No annual/fortnightly paths on TehsilEntity — add an entity with those paths"
+        "\n[skip] No annual/sub_annual paths on TehsilEntity — add an entity with those paths"
     )
 
 

{core_lens-0.1.dev164 → core_lens-0.1.dev166}/benchmarks/bench_season.py

@@ -6,7 +6,7 @@ Targets:
   - resolve_time_filter()           dict → Polars Expr build (date-range + season modes)
   - _date_range_expr()              expression build for date ranges
   - _season_expr()                  expression build for seasons (year-crossing vs not)
-  - add_temporal_columns()          fortnightly vectorised temporal derivation
+  - add_temporal_columns()          sub_annual vectorised temporal derivation
 
 Scalene focuses on:
   - Python datetime.strptime calls in __post_init__
@@ -35,22 +35,22 @@ from core_lens.utils.season import add_temporal_columns, resolve_time_filter
 START_DATE = date(2010, 1, 1)
 END_DATE = date(2023, 12, 31)
 
-# Number of fortnightly periods between START_DATE and END_DATE (roughly).
-N_FORTNIGHTLY = (END_DATE - START_DATE).days // 14
+# Number of sub_annual periods between START_DATE and END_DATE (roughly).
+N_SUB_ANNUAL = (END_DATE - START_DATE).days // 14
 
-# Representative synthetic fortnightly dataframe for add_temporal_columns.
+# Representative synthetic sub_annual dataframe for add_temporal_columns.
 N_MWS = 500  # number of entity instances (e.g. MWS)
-DATES = [START_DATE + timedelta(days=14 * i) for i in range(N_FORTNIGHTLY)]
-# Long format: N_MWS × N_FORTNIGHTLY rows
+DATES = [START_DATE + timedelta(days=14 * i) for i in range(N_SUB_ANNUAL)]
+# Long format: N_MWS × N_SUB_ANNUAL rows
 _all_dates = DATES * N_MWS
-FORTNIGHTLY_DF = pl.DataFrame(
+SUB_ANNUAL_DF = pl.DataFrame(
     {
         "mws_id": [str(mid) for mid in range(N_MWS) for _ in DATES],
         "date": _all_dates,
-        "ndvi": [0.5] * (N_MWS * N_FORTNIGHTLY),
+        "ndvi": [0.5] * (N_MWS * N_SUB_ANNUAL),
     }
 )
-print(f"Synthetic fortnightly frame: {FORTNIGHTLY_DF.shape}")
+print(f"Synthetic sub_annual frame: {SUB_ANNUAL_DF.shape}")
 
 
 def _section(title: str) -> None:
@@ -162,15 +162,15 @@ print(
 )
 
 
-# ── 8. add_temporal_columns — large fortnightly DataFrame ─────────────────────
-_section(f"8. add_temporal_columns()  [{FORTNIGHTLY_DF.shape[0]:,} rows]")
+# ── 8. add_temporal_columns — large sub_annual DataFrame ─────────────────────
+_section(f"8. add_temporal_columns()  [{SUB_ANNUAL_DF.shape[0]:,} rows]")
 t0 = time.perf_counter()
-result_df = add_temporal_columns(FORTNIGHTLY_DF, "date", cfg)
+result_df = add_temporal_columns(SUB_ANNUAL_DF, "date", cfg)
 t1 = time.perf_counter()
 print(f"add_temporal_columns: {(t1 - t0) * 1000:.2f} ms")
 print(f"Output shape        : {cast(pl.DataFrame, result_df).shape}")
 print(
-    f"New columns added   : {[c for c in cast(pl.DataFrame, result_df).columns if c not in FORTNIGHTLY_DF.columns]}"
+    f"New columns added   : {[c for c in cast(pl.DataFrame, result_df).columns if c not in SUB_ANNUAL_DF.columns]}"
 )
 
 # Second call — all columns already present, should be near-zero.

{core_lens-0.1.dev164 → core_lens-0.1.dev166}/benchmarks/bench_view.py

@@ -4,14 +4,14 @@ Targets:
   - View.where()                attribute filter (Polars semi-join on static file)
   - View.spatial_filter()       STRtree filter narrowing existing keys
   - View.between()              time-filter dict construction (no I/O)
-  - View._materialise()         static / annual / fortnightly Parquet reads
+  - View._materialise()         static / annual / sub_annual Parquet reads
   - View chaining               where → spatial_filter → between → static
 
 Scalene focuses on:
   - scan_with_key_filter()      predicate-pushdown LazyFrame
   - collect_lf()                streaming collect (CPU or GPU)
   - resolve_time_filter()       expression build for time predicates
-  - add_temporal_columns()      fortnightly temporal column derivation
+  - add_temporal_columns()      sub_annual temporal column derivation
 
 Usage:
     uv run scalene run --html --outfile benchmarks/profiles/bench_view.html benchmarks/bench_view.py
@@ -140,12 +140,12 @@ print(f"annual           : {(t1 - t0) * 1000:.2f} ms")
 print(f"Shape            : {result_annual.df().shape}")
 
 
-# ── 11. fortnightly materialisation ───────────────────────────────────────────
-_section("11. aoi_small.mws.fortnightly  [fortnightly materialisation]")
+# ── 11. sub_annual materialisation ───────────────────────────────────────────
+_section("11. aoi_small.mws.sub_annual  [sub_annual materialisation]")
 t0 = time.perf_counter()
-result_fn = aoi_small.mws.between(season=Season.KHARIF, year=2022).fortnightly
+result_fn = aoi_small.mws.between(season=Season.KHARIF, year=2022).sub_annual
 t1 = time.perf_counter()
-print(f"fortnightly      : {(t1 - t0) * 1000:.2f} ms")
+print(f"sub_annual      : {(t1 - t0) * 1000:.2f} ms")
 print(f"Shape            : {result_fn.df().shape}")
 
 

core_lens-0.1.dev166/district_report.py

@@ -0,0 +1,127 @@
+import argparse
+import pandas as pd
+import os
+import duckdb
+import json
+import glob
+from collections import defaultdict
+
+def analyze_district(district, base_dir, out_dir):
+    safe_dist = str(district).strip().replace("/", "_").replace(" ", "_").replace(",", "")
+    if not safe_dist:
+        return
+        
+    print(f"Processing district: {district}")
+    
+    con = duckdb.connect()
+    
+    # Check if district exists
+    found_any = False
+    for cat in ["static", "annual", "sub-annual"]:
+        if not glob.glob(f"{base_dir}/{cat}/**/*.parquet", recursive=True):
+            continue
+        try:
+            q = f"SELECT count(*) FROM read_parquet('{base_dir}/{cat}/**/*.parquet', union_by_name=true) WHERE list_contains(district, '{district.replace("'", "''")}')"
+            if con.execute(q).fetchone()[0] > 0:
+                found_any = True
+                break
+        except Exception:
+            pass
+            
+    if not found_any:
+        print(f"  -> Error: District '{district}' not found. Skipping.")
+        con.close()
+        return
+
+    report = {
+        "district": district.strip(),
+        "categories": defaultdict(lambda: {
+            "null_percentages": {},
+            "completely_null_columns": [],
+            "most_empty_tehsils": {}
+        })
+    }
+    
+    for cat in ["static", "annual", "sub-annual"]:
+        if not glob.glob(f"{base_dir}/{cat}/**/*.parquet", recursive=True):
+            continue
+            
+        query = f"""
+            SELECT * FROM read_parquet('{base_dir}/{cat}/**/*.parquet', union_by_name=true)
+            WHERE list_contains(district, '{district.replace("'", "''")}')
+        """
+        try:
+            df = con.execute(query).df()
+            if len(df) == 0:
+                continue
+                
+            df_exploded = df.explode('tehsil') if 'tehsil' in df.columns else df
+            
+            # Identify data columns
+            exclude = ['district', 'tehsil', 'state', 'geometry', 'id', 'key']
+            data_cols = [c for c in df.columns if c not in exclude]
+            
+            total_rows = len(df)
+            cat_dict = report["categories"][cat]
+            
+            # Column null percentages
+            for c in data_cols:
+                nulls = int(df[c].isna().sum())
+                pct = float((nulls / total_rows) * 100) if total_rows > 0 else 100.0
+                cat_dict["null_percentages"][c] = round(pct, 2)
+                if pct == 100.0:
+                    cat_dict["completely_null_columns"].append(c)
+            
+            # Tehsil missingness
+            if 'tehsil' in df_exploded.columns and len(data_cols) > 0:
+                missing_by_tehsil = df_exploded[data_cols].isna().sum(axis=1).groupby(df_exploded['tehsil']).sum()
+                total_by_tehsil = df_exploded.groupby('tehsil').size() * len(data_cols)
+                pct_series = (missing_by_tehsil / total_by_tehsil) * 100
+                
+                # Top 3 most empty
+                top_empty = pct_series.sort_values(ascending=False).head(3)
+                
+                empty_dict = {}
+                for t, v in top_empty.items():
+                    if pd.notna(v) and t is not None:
+                        empty_dict[str(t)] = round(float(v), 2)
+                        
+                cat_dict["most_empty_tehsils"] = empty_dict
+
+        except Exception as e:
+            print(f"  -> Error processing {cat} for {district}: {e}")
+            
+    con.close()
+    
+    # Convert defaultdict to standard dict for JSON serialization
+    report["categories"] = dict(report["categories"])
+    
+    out_file = os.path.join(out_dir, f"{safe_dist}_report.json")
+    with open(out_file, 'w') as f:
+        json.dump(report, f, indent=2)
+        
+    print(f"  -> Saved report to {out_file}")
+
+def main():
+    parser = argparse.ArgumentParser(description="Generate district data completeness report in JSON.")
+    parser.add_argument("districts_file", type=str, help="File containing one district name per line")
+    parser.add_argument("--base-dir", type=str, default="/home/akashyap54/workstation/CoREStack-Data/mws", help="Base directory of raw data")
+    parser.add_argument("--out-dir", type=str, default="/home/akashyap54/Projects/CoreStack/core-lens/district_reports", help="Output directory")
+    args = parser.parse_args()
+
+    os.makedirs(args.out_dir, exist_ok=True)
+    
+    if not os.path.exists(args.districts_file):
+        print(f"Error: Input file {args.districts_file} does not exist.")
+        return
+        
+    with open(args.districts_file, 'r') as f:
+        districts = [line.strip() for line in f if line.strip()]
+        
+    print(f"Found {len(districts)} districts to process.")
+    
+    for d in districts:
+        analyze_district(d, args.base_dir, args.out_dir)
+        
+if __name__ == "__main__":
+    main()