@saibolla/ada 0.1.2

package/skills/geomaster/references/troubleshooting.md

@@ -0,0 +1,439 @@
+# GeoMaster Troubleshooting Guide
+
+Solutions to common geospatial problems and debugging strategies.
+
+## Installation Issues
+
+### GDAL Installation Problems
+
+```bash
+# Problem: "gdal-config not found" or rasterio install fails
+
+# Solution 1: Use conda (recommended)
+conda install -c conda-forge gdal rasterio
+
+# Solution 2: System packages (Ubuntu/Debian)
+sudo apt-get install gdal-bin libgdal-dev
+export CPLUS_INCLUDE_PATH=/usr/include/gdal
+export C_INCLUDE_PATH=/usr/include/gdal
+pip install rasterio
+
+# Solution 3: Wheel files
+pip install rasterio --find-links=https://gis.wheelwrights.com/
+
+# Verify installation
+python -c "from osgeo import gdal; print(gdal.__version__)"
+python -c "import rasterio; print(rasterio.__version__)"
+```
+
+### Python Binding Issues
+
+```bash
+# Problem: "DLL load failed" on Windows
+# Solution: Reinstall with conda
+conda install -c conda-forge --force-reinstall gdal rasterio fiona
+
+# Problem: "Symbol not found" on macOS
+# Solution: Rebuild from source or use conda
+brew install gdal
+pip install rasterio --no-binary rasterio
+
+# Problem: GEOS errors
+brew install geos
+pip install shapely --no-binary shapely
+```
+
+## Runtime Errors
+
+### CRS Transformation Errors
+
+```python
+# Problem: "Invalid projection" or "CRS mismatch"
+import geopandas as gpd
+
+# Check CRS
+print(f"CRS: {gdf.crs}")
+
+# If None, set it
+if gdf.crs is None:
+    gdf.set_crs("EPSG:4326", inplace=True)
+
+# If unknown, try to detect
+gdf = gdf.to_crs(gdf.estimate_utm_crs())
+```
+
+### Memory Errors with Large Rasters
+
+```python
+# Problem: "MemoryError" when reading large files
+# Solution: Read in chunks or use windows
+
+import rasterio
+from rasterio.windows import Window
+
+# Windowed reading
+with rasterio.open('large.tif') as src:
+    window = Window(0, 0, 1000, 1000)  # (col_off, row_off, width, height)
+    subset = src.read(1, window=window)
+
+# Block-by-block processing
+with rasterio.open('large.tif') as src:
+    for i, window in src.block_windows(1):
+        block = src.read(1, window=window)
+        # Process block...
+
+# Use Dask for very large files
+import dask.array as da
+dask_array = da.from_rasterio('large.tif', chunks=(1, 1024, 1024))
+```
+
+### Geometry Validation Errors
+
+```python
+# Problem: "TopologyException" or "Self-intersection"
+import geopandas as gpd
+from shapely.validation import make_valid
+
+# Check invalid geometries
+invalid = gdf[~gdf.is_valid]
+print(f"Invalid geometries: {len(invalid)}")
+
+# Fix invalid geometries
+gdf['geometry'] = gdf.geometry.make_valid()
+
+# Buffer with 0 to fix (alternative method)
+gdf['geometry'] = gdf.geometry.buffer(0)
+```
+
+### Coordinate Order Confusion
+
+```python
+# Problem: Points in wrong location (lon/lat swapped)
+from pyproj import Transformer
+
+# Common mistake: lon, lat vs lat, lon
+# Always specify axis order
+transformer = Transformer.from_crs(
+    "EPSG:4326",
+    "EPSG:32610",
+    always_xy=True  # Input: x=lon, y=lat (not y=lat, x=lon)
+)
+
+# Correct usage
+x, y = transformer.transform(lon, lat)  # not lat, lon
+```
+
+## Performance Issues
+
+### Slow Spatial Joins
+
+```python
+# Problem: sjoin takes too long
+import geopandas as gpd
+
+# Solution: Use spatial index
+gdf1.sindex  # Auto-created
+gdf2.sindex
+
+# For nearest neighbor joins, use specialized function
+result = gpd.sjoin_nearest(gdf1, gdf2, max_distance=1000)
+
+# Use intersection predicate (faster than 'intersects' for points)
+result = gpd.sjoin(points, polygons, predicate='within')
+
+# Clip to bounding box first
+bbox = gdf1.total_bounds
+gdf2_clipped = gdf2.cx[bbox[0]:bbox[2], bbox[1]:bbox[3]]
+result = gpd.sjoin(gdf1, gdf2_clipped, predicate='intersects')
+```
+
+### Slow Raster I/O
+
+```python
+# Problem: Reading/writing rasters is slow
+import rasterio
+
+# Solution 1: Use compression when writing
+profile.update(
+    compress='DEFLATE',
+    predictor=2,
+    zlevel=4
+)
+
+# Solution 2: Use tiled output
+profile.update(
+    tiled=True,
+    blockxsize=256,
+    blockysize=256
+)
+
+# Solution 3: Enable caching
+from osgeo import gdal
+gdal.SetCacheMax(2**30)  # 1GB
+
+# Solution 4: Use COG format for cloud access
+from rio_cogeo.cogeo import cog_translate
+cog_translate('input.tif', 'output.tif', profile)
+```
+
+### Slow Reprojection
+
+```python
+# Problem: to_crs() is very slow
+import geopandas as gpd
+
+# Solution 1: Simplify geometry first
+gdf_simple = gdf.geometry.simplify(tolerance=0.0001)
+gdf_reproj = gdf_simple.to_crs(target_crs)
+
+# Solution 2: Use lower precision for display
+gdf_reproj = gdf.to_crs(target_crs, geometry_precision=2)
+
+# Solution 3: Reproject in parallel
+import multiprocessing as mp
+from functools import partial
+
+def reproj_chunk(chunk, target_crs):
+    return chunk.to_crs(target_crs)
+
+chunks = np.array_split(gdf, mp.cpu_count())
+with mp.Pool() as pool:
+    results = pool.map(partial(reproj_chunk, target_crs=target_crs), chunks)
+gdf_reproj = gpd.GeoDataFrame(pd.concat(results))
+```
+
+## Common Pitfalls
+
+### Area in Degrees
+
+```python
+# WRONG: Area in square degrees
+gdf = gpd.read_file('data.geojson')
+area = gdf.geometry.area  # Wrong!
+
+# CORRECT: Use projected CRS
+gdf_proj = gdf.to_crs(gdf.estimate_utm_crs())
+area_sqm = gdf_proj.geometry.area
+area_sqkm = area_sqm / 1_000_000
+```
+
+### Buffer in Geographic CRS
+
+```python
+# WRONG: Buffer of 1000 degrees
+gdf['buffer'] = gdf.geometry.buffer(1000)
+
+# CORRECT: Project first
+gdf_proj = gdf.to_crs("EPSG:32610")
+gdf_proj['buffer_km'] = gdf_proj.geometry.buffer(1000)  # 1000 meters
+```
+
+### Web Mercator Distortion
+
+```python
+# WRONG: Area calculation in Web Mercator
+gdf = gdf.to_crs("EPSG:3857")
+area = gdf.geometry.area  # Significant distortion!
+
+# CORRECT: Use appropriate projection
+gdf = gdf.to_crs(gdf.estimate_utm_crs())
+area = gdf.geometry.area  # Accurate
+```
+
+### Mixing CRS
+
+```python
+# WRONG: Spatial join without checking CRS
+result = gpd.sjoin(gdf1, gdf2, predicate='intersects')
+
+# CORRECT: Ensure same CRS
+if gdf1.crs != gdf2.crs:
+    gdf2 = gdf2.to_crs(gdf1.crs)
+result = gpd.sjoin(gdf1, gdf2, predicate='intersects')
+```
+
+## Data Issues
+
+### Missing/Missing CRS
+
+```python
+# Problem: CRS is None
+gdf = gpd.read_file('data.geojson')
+if gdf.crs is None:
+    # Try to detect from data extent
+    lon_min, lat_min, lon_max, lat_max = gdf.total_bounds
+
+    if -180 <= lon_min <= 180 and -90 <= lat_min <= 90:
+        gdf.set_crs("EPSG:4326", inplace=True)
+        print("Assumed WGS 84 (EPSG:4326)")
+    else:
+        gdf.set_crs(gdf.estimate_utm_crs(), inplace=True)
+        print("Estimated UTM zone")
+```
+
+### Invalid Coordinates
+
+```python
+# Problem: Coordinates out of valid range
+gdf = gpd.read_file('data.geojson')
+
+# Check for invalid coordinates
+invalid_lon = (gdf.geometry.x < -180) | (gdf.geometry.x > 180)
+invalid_lat = (gdf.geometry.y < -90) | (gdf.geometry.y > 90)
+
+if invalid_lon.any() or invalid_lat.any():
+    print("Warning: Invalid coordinates found")
+    gdf = gdf[~invalid_lon & ~invalid_lat]
+```
+
+### Empty Geometries
+
+```python
+# Problem: Processing fails with empty geometries
+# Remove empty geometries
+gdf = gdf[~gdf.geometry.is_empty]
+
+# Or fill with None
+gdf.loc[gdf.geometry.is_empty, 'geometry'] = None
+
+# Check before operations
+if gdf.geometry.is_empty.any():
+    print(f"Warning: {gdf.geometry.is_empty.sum()} empty geometries")
+```
+
+## Remote Sensing Issues
+
+### Sentinel-2 Band Ordering
+
+```python
+# Problem: Wrong band indices
+# Sentinel-2 L2A SAFE structure:
+# B01 (60m), B02 (10m), B03 (10m), B04 (10m), B05 (20m),
+# B06 (20m), B07 (20m), B08 (10m), B08A (20m), B09 (60m),
+# B11 (20m), B12 (20m)
+
+# Sentinel-2 (resampled to 10m):
+# B02=1, B03=2, B04=3, B05=4, B06=5, B07=6, B08=7, B8A=8, B11=9, B12=10
+
+# For 10m bands only:
+blue = src.read(1)   # B02
+green = src.read(2)  # B03
+red = src.read(3)    # B04
+nir = src.read(4)    # B08
+```
+
+### Cloud Shadow Masking
+
+```python
+# Problem: Clouds and shadows not properly masked
+def improved_cloud_mask(scl):
+    """
+    Improved cloud masking using SCL layer.
+    Classes: 0=No data, 1=Saturated, 2=Dark, 3=Cloud shadow,
+    4=Vegetation, 5=Bare soil, 6=Water, 7=Cloud low prob,
+    8=Cloud med prob, 9=Cloud high prob, 10=Thin cirrus
+    """
+    # Mask: clouds, cloud shadows, saturated
+    mask = scl.isin([0, 1, 3, 8, 9, 10])
+    return mask
+
+# Apply
+scl = s2_image.select('SCL')
+cloud_mask = improved_cloud_mask(scl)
+image_clean = s2_image.updateMask(cloud_mask.Not())
+```
+
+## Error Messages Reference
+
+| Error | Cause | Solution |
+|-------|-------|----------|
+| `CRS mismatch` | Different coordinate systems | `gdf2 = gdf2.to_crs(gdf1.crs)` |
+| `TopologyException` | Invalid/self-intersecting geometry | `gdf.geometry = gdf.geometry.make_valid()` |
+| `MemoryError` | Large dataset | Use Dask or chunked reading |
+| `Invalid projection` | Unknown CRS code | Check EPSG code, use `CRS.from_user_input()` |
+| `Empty geometry` | Null geometries | `gdf = gdf[~gdf.geometry.is_empty]` |
+| `Bounds error` | Coordinates out of range | Filter invalid coordinates |
+| `DLL load failed` | GDAL not installed | Use conda: `conda install gdal` |
+| `Symbol not found` | Library linking issue | Reinstall with binary wheels or conda |
+| `Self-intersection` | Invalid polygon | Buffer(0) or make_valid() |
+
+## Debugging Strategies
+
+### 1. Check Data Integrity
+
+```python
+def check_geodataframe(gdf):
+    """Comprehensive GeoDataFrame health check."""
+    print(f"Shape: {gdf.shape}")
+    print(f"CRS: {gdf.crs}")
+    print(f"Bounds: {gdf.total_bounds}")
+    print(f"Invalid geometries: {(~gdf.is_valid).sum()}")
+    print(f"Empty geometries: {gdf.geometry.is_empty.sum()}")
+    print(f"None geometries: {gdf.geometry.isna().sum()}")
+    print(f"Duplicate geometries: {gdf.geometry.duplicated().sum()}")
+    print("\nGeometry types:")
+    print(gdf.geometry.type.value_counts())
+    print("\nCoordinate range:")
+    print(f"  X: {gdf.geometry.x.min():.2f} to {gdf.geometry.x.max():.2f}")
+    print(f"  Y: {gdf.geometry.y.min():.2f} to {gdf.geometry.y.max():.2f}")
+
+check_geodataframe(gdf)
+```
+
+### 2. Test Transformations
+
+```python
+def test_reprojection(gdf, target_crs):
+    """Test if reprojection is reversible."""
+    original = gdf.copy()
+    gdf_proj = gdf.to_crs(target_crs)
+    gdf_back = gdf_proj.to_crs(gdf.crs)
+
+    diff = original.geometry.distance(gdf_back.geometry).max()
+    if diff > 1:  # More than 1 meter
+        print(f"Warning: Max error: {diff:.2f}m")
+        return False
+    return True
+```
+
+### 3. Profile Code
+
+```python
+import time
+
+def time_operation(func, *args, **kwargs):
+    """Time a geospatial operation."""
+    start = time.time()
+    result = func(*args, **kwargs)
+    elapsed = time.time() - start
+    print(f"{func.__name__}: {elapsed:.2f}s")
+    return result
+
+# Usage
+time_operation(gdf.to_crs, "EPSG:32610")
+```
+
+## Getting Help
+
+### Check Versions
+
+```python
+import sys
+import geopandas as gpd
+import rasterio
+from osgeo import gdal
+
+print(f"Python: {sys.version}")
+print(f"GeoPandas: {gpd.__version__}")
+print(f"Rasterio: {rasterio.__version__}")
+print(f"GDAL: {gdal.__version__}")
+print(f"PROJ: {gdal.VersionInfo('PROJ')}")
+```
+
+### Useful Resources
+
+- **GeoPandas docs**: https://geopandas.org/
+- **Rasterio docs**: https://rasterio.readthedocs.io/
+- **PROJ datab**: https://epsg.org/
+- **Stack Overflow**: Tag with `gis` and `python`
+- **GIS Stack Exchange**: https://gis.stackexchange.com/

package/skills/geopandas/SKILL.md

@@ -0,0 +1,249 @@
+---
+name: geopandas
+description: Python library for working with geospatial vector data including shapefiles, GeoJSON, and GeoPackage files. Use when working with geographic data for spatial analysis, geometric operations, coordinate transformations, spatial joins, overlay operations, choropleth mapping, or any task involving reading/writing/analyzing vector geographic data. Supports PostGIS databases, interactive maps, and integration with matplotlib/folium/cartopy. Use for tasks like buffer analysis, spatial joins between datasets, dissolving boundaries, clipping data, calculating areas/distances, reprojecting coordinate systems, creating maps, or converting between spatial file formats.
+license: BSD-3-Clause license
+metadata:
+    skill-author: K-Dense Inc.
+---
+
+# GeoPandas
+
+GeoPandas extends pandas to enable spatial operations on geometric types. It combines the capabilities of pandas and shapely for geospatial data analysis.
+
+## Installation
+
+```bash
+uv pip install geopandas
+```
+
+### Optional Dependencies
+
+```bash
+# For interactive maps
+uv pip install folium
+
+# For classification schemes in mapping
+uv pip install mapclassify
+
+# For faster I/O operations (2-4x speedup)
+uv pip install pyarrow
+
+# For PostGIS database support
+uv pip install psycopg2
+uv pip install geoalchemy2
+
+# For basemaps
+uv pip install contextily
+
+# For cartographic projections
+uv pip install cartopy
+```
+
+## Quick Start
+
+```python
+import geopandas as gpd
+
+# Read spatial data
+gdf = gpd.read_file("data.geojson")
+
+# Basic exploration
+print(gdf.head())
+print(gdf.crs)
+print(gdf.geometry.geom_type)
+
+# Simple plot
+gdf.plot()
+
+# Reproject to different CRS
+gdf_projected = gdf.to_crs("EPSG:3857")
+
+# Calculate area (use projected CRS for accuracy)
+gdf_projected['area'] = gdf_projected.geometry.area
+
+# Save to file
+gdf.to_file("output.gpkg")
+```
+
+## Core Concepts
+
+### Data Structures
+
+- **GeoSeries**: Vector of geometries with spatial operations
+- **GeoDataFrame**: Tabular data structure with geometry column
+
+See [data-structures.md](references/data-structures.md) for details.
+
+### Reading and Writing Data
+
+GeoPandas reads/writes multiple formats: Shapefile, GeoJSON, GeoPackage, PostGIS, Parquet.
+
+```python
+# Read with filtering
+gdf = gpd.read_file("data.gpkg", bbox=(xmin, ymin, xmax, ymax))
+
+# Write with Arrow acceleration
+gdf.to_file("output.gpkg", use_arrow=True)
+```
+
+See [data-io.md](references/data-io.md) for comprehensive I/O operations.
+
+### Coordinate Reference Systems
+
+Always check and manage CRS for accurate spatial operations:
+
+```python
+# Check CRS
+print(gdf.crs)
+
+# Reproject (transforms coordinates)
+gdf_projected = gdf.to_crs("EPSG:3857")
+
+# Set CRS (only when metadata missing)
+gdf = gdf.set_crs("EPSG:4326")
+```
+
+See [crs-management.md](references/crs-management.md) for CRS operations.
+
+## Common Operations
+
+### Geometric Operations
+
+Buffer, simplify, centroid, convex hull, affine transformations:
+
+```python
+# Buffer by 10 units
+buffered = gdf.geometry.buffer(10)
+
+# Simplify with tolerance
+simplified = gdf.geometry.simplify(tolerance=5, preserve_topology=True)
+
+# Get centroids
+centroids = gdf.geometry.centroid
+```
+
+See [geometric-operations.md](references/geometric-operations.md) for all operations.
+
+### Spatial Analysis
+
+Spatial joins, overlay operations, dissolve:
+
+```python
+# Spatial join (intersects)
+joined = gpd.sjoin(gdf1, gdf2, predicate='intersects')
+
+# Nearest neighbor join
+nearest = gpd.sjoin_nearest(gdf1, gdf2, max_distance=1000)
+
+# Overlay intersection
+intersection = gpd.overlay(gdf1, gdf2, how='intersection')
+
+# Dissolve by attribute
+dissolved = gdf.dissolve(by='region', aggfunc='sum')
+```
+
+See [spatial-analysis.md](references/spatial-analysis.md) for analysis operations.
+
+### Visualization
+
+Create static and interactive maps:
+
+```python
+# Choropleth map
+gdf.plot(column='population', cmap='YlOrRd', legend=True)
+
+# Interactive map
+gdf.explore(column='population', legend=True).save('map.html')
+
+# Multi-layer map
+import matplotlib.pyplot as plt
+fig, ax = plt.subplots()
+gdf1.plot(ax=ax, color='blue')
+gdf2.plot(ax=ax, color='red')
+```
+
+See [visualization.md](references/visualization.md) for mapping techniques.
+
+## Detailed Documentation
+
+- **[Data Structures](references/data-structures.md)** - GeoSeries and GeoDataFrame fundamentals
+- **[Data I/O](references/data-io.md)** - Reading/writing files, PostGIS, Parquet
+- **[Geometric Operations](references/geometric-operations.md)** - Buffer, simplify, affine transforms
+- **[Spatial Analysis](references/spatial-analysis.md)** - Joins, overlay, dissolve, clipping
+- **[Visualization](references/visualization.md)** - Plotting, choropleth maps, interactive maps
+- **[CRS Management](references/crs-management.md)** - Coordinate reference systems and projections
+
+## Common Workflows
+
+### Load, Transform, Analyze, Export
+
+```python
+# 1. Load data
+gdf = gpd.read_file("data.shp")
+
+# 2. Check and transform CRS
+print(gdf.crs)
+gdf = gdf.to_crs("EPSG:3857")
+
+# 3. Perform analysis
+gdf['area'] = gdf.geometry.area
+buffered = gdf.copy()
+buffered['geometry'] = gdf.geometry.buffer(100)
+
+# 4. Export results
+gdf.to_file("results.gpkg", layer='original')
+buffered.to_file("results.gpkg", layer='buffered')
+```
+
+### Spatial Join and Aggregate
+
+```python
+# Join points to polygons
+points_in_polygons = gpd.sjoin(points_gdf, polygons_gdf, predicate='within')
+
+# Aggregate by polygon
+aggregated = points_in_polygons.groupby('index_right').agg({
+    'value': 'sum',
+    'count': 'size'
+})
+
+# Merge back to polygons
+result = polygons_gdf.merge(aggregated, left_index=True, right_index=True)
+```
+
+### Multi-Source Data Integration
+
+```python
+# Read from different sources
+roads = gpd.read_file("roads.shp")
+buildings = gpd.read_file("buildings.geojson")
+parcels = gpd.read_postgis("SELECT * FROM parcels", con=engine, geom_col='geom')
+
+# Ensure matching CRS
+buildings = buildings.to_crs(roads.crs)
+parcels = parcels.to_crs(roads.crs)
+
+# Perform spatial operations
+buildings_near_roads = buildings[buildings.geometry.distance(roads.union_all()) < 50]
+```
+
+## Performance Tips
+
+1. **Use spatial indexing**: GeoPandas creates spatial indexes automatically for most operations
+2. **Filter during read**: Use `bbox`, `mask`, or `where` parameters to load only needed data
+3. **Use Arrow for I/O**: Add `use_arrow=True` for 2-4x faster reading/writing
+4. **Simplify geometries**: Use `.simplify()` to reduce complexity when precision isn't critical
+5. **Batch operations**: Vectorized operations are much faster than iterating rows
+6. **Use appropriate CRS**: Projected CRS for area/distance, geographic for visualization
+
+## Best Practices
+
+1. **Always check CRS** before spatial operations
+2. **Use projected CRS** for area and distance calculations
+3. **Match CRS** before spatial joins or overlays
+4. **Validate geometries** with `.is_valid` before operations
+5. **Use `.copy()`** when modifying geometry columns to avoid side effects
+6. **Preserve topology** when simplifying for analysis
+7. **Use GeoPackage** format for modern workflows (better than Shapefile)
+8. **Set max_distance** in sjoin_nearest for better performance
+