geo-data-audit 0.1.0__tar.gz

geo_data_audit-0.1.0/PKG-INFO

@@ -0,0 +1,105 @@
+Metadata-Version: 2.4
+Name: geo-data-audit
+Version: 0.1.0
+Summary: Audit heterogeneous geospatial datasets for common data-quality and spatial-consistency problems before they enter downstream geospatial ML pipelines.
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/souro26/geo-audit
+Project-URL: Repository, https://github.com/souro26/geo-audit
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: pandas>=2.0
+Requires-Dist: numpy>=1.24
+Requires-Dist: geopandas>=0.14
+Requires-Dist: shapely>=2.0
+Requires-Dist: pyproj>=3.5
+Requires-Dist: typer>=0.9
+Requires-Dist: rich>=13.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.4; extra == "dev"
+Requires-Dist: pytest-cov>=4.1; extra == "dev"
+Requires-Dist: ruff>=0.1.0; extra == "dev"
+Requires-Dist: mypy>=1.5; extra == "dev"
+
+# geo-audit
+
+`geo-audit` is a command-line tool for auditing raw geospatial datasets before ingestion into spatial analysis, GIS workflows, or machine learning pipelines. It inspects CSV and GeoJSON files in a directory to identify coordinate anomalies, missing spatial metadata, attribute missingness, exact duplicate locations, and cross-dataset bounding box overlaps without modifying source data.
+
+## Purpose
+
+Geospatial data collected from field surveys, legacy archives, and third-party GIS exports frequently contains quality defects—such as out-of-range latitude/longitude coordinates, unprojected planar coordinates lacking CRS definitions, silent duplicate coordinates, and missing values across attribute columns. `geo-audit` runs static health checks across all datasets in a target directory to surface critical defects and risk levels in a single pass.
+
+## Installation
+
+Requires Python 3.10 or higher.
+
+```bash
+git clone https://github.com/souro26/geo-audit
+cd geo-audit
+pip install -e .
+```
+
+## Quick Start
+
+Scan a directory of geospatial files for a human-readable terminal report:
+
+```bash
+geo-audit data/
+```
+
+Generate machine-readable JSON output for automated pipelines:
+
+```bash
+geo-audit data/ --format json
+```
+
+Save JSON directly to a file (suppresses terminal progress):
+
+```bash
+geo-audit data/ --format json --output report.json
+```
+
+## Audit Capabilities
+
+- **Geographic Coordinate Validation**: Auto-detects `latitude`/`longitude` columns (including variants like `lat`/`lon`/`lng`), coerces non-numeric entries, and flags values outside valid ranges ([-90, 90] and [-180, 180]).
+- **Planar Coordinate Inspection**: Identifies planar coordinate pairs (`easting`/`northing`, `x`/`y`) and flags missing values. Planar coordinates are reported separately because their Coordinate Reference System (CRS) is unknown.
+- **Duplicate Location Detection**: Identifies exact duplicate coordinate pairs across records within CSV datasets.
+- **Attribute Missingness Profiling**: Checks column-level completeness. Attribute columns with missingness exceeding 5% are flagged (`WARN` at 5-20%, `HIGH` above 20%). Coordinate columns are excluded from column missingness to prevent duplicate reporting.
+- **CRS and Bounding Box Analysis**: Extracts declared CRS from GeoJSON datasets and computes spatial bounding boxes for geographic datasets to evaluate spatial coverage and cross-dataset bounding box overlaps.
+- **Cross-Dataset Schema Comparison**: Identifies normalized column name matches and data type discrepancies across datasets.
+
+## Dataset Readiness Levels
+
+Every scanned dataset is assigned an overall readiness status based on the highest severity finding detected:
+
+- **READY**: No blocking or moderate issues detected by active checks.
+- **REVIEW**: Moderate-severity findings detected (e.g., 5-20% missingness, duplicate locations, missing planar coordinates).
+- **HIGH RISK**: High-severity findings detected (e.g., >20% missing or invalid geographic coordinates, high column missingness).
+- **ERROR**: Unparseable files or critical dataset structural failure.
+
+## Demo Data
+
+The included `data/` directory contains synthetic test cases demonstrating common data issues:
+
+| File | Description | Primary Findings |
+|---|---|---|
+| `exploration_samples.csv` | 12 sample points | Out-of-range coordinates, missing lat/lon, 41.7% missing notes, duplicate locations |
+| `historical_surveys.csv` | 10 survey points | Clean dataset using `lat`/`lon` column headers |
+| `geology.geojson` | 5 Point geometries | Clean GeoJSON dataset with EPSG:4326 |
+| `planar_samples.csv` | 8 sample points | Easting/northing planar coordinates with 12.5% missing values |
+
+Run the demo against the included data:
+
+```bash
+geo-audit data/
+```
+
+## Scope and Limitations
+
+- **Supported Formats**: CSV (`.csv`) and GeoJSON (`.geojson`, `.json`).
+- **Duplicate Detection**: Performs exact coordinate pair matching; near-duplicate spatial clustering is not performed.
+- **Spatial Overlap**: Evaluates bounding box intersection; polygon geometry intersections are out of scope.
+- **CRS Transformations**: Does not attempt automatic CRS inference or coordinate re-projection.
+
+## License
+
+MIT

geo_data_audit-0.1.0/README.md

@@ -0,0 +1,83 @@
+# geo-audit
+
+`geo-audit` is a command-line tool for auditing raw geospatial datasets before ingestion into spatial analysis, GIS workflows, or machine learning pipelines. It inspects CSV and GeoJSON files in a directory to identify coordinate anomalies, missing spatial metadata, attribute missingness, exact duplicate locations, and cross-dataset bounding box overlaps without modifying source data.
+
+## Purpose
+
+Geospatial data collected from field surveys, legacy archives, and third-party GIS exports frequently contains quality defects—such as out-of-range latitude/longitude coordinates, unprojected planar coordinates lacking CRS definitions, silent duplicate coordinates, and missing values across attribute columns. `geo-audit` runs static health checks across all datasets in a target directory to surface critical defects and risk levels in a single pass.
+
+## Installation
+
+Requires Python 3.10 or higher.
+
+```bash
+git clone https://github.com/souro26/geo-audit
+cd geo-audit
+pip install -e .
+```
+
+## Quick Start
+
+Scan a directory of geospatial files for a human-readable terminal report:
+
+```bash
+geo-audit data/
+```
+
+Generate machine-readable JSON output for automated pipelines:
+
+```bash
+geo-audit data/ --format json
+```
+
+Save JSON directly to a file (suppresses terminal progress):
+
+```bash
+geo-audit data/ --format json --output report.json
+```
+
+## Audit Capabilities
+
+- **Geographic Coordinate Validation**: Auto-detects `latitude`/`longitude` columns (including variants like `lat`/`lon`/`lng`), coerces non-numeric entries, and flags values outside valid ranges ([-90, 90] and [-180, 180]).
+- **Planar Coordinate Inspection**: Identifies planar coordinate pairs (`easting`/`northing`, `x`/`y`) and flags missing values. Planar coordinates are reported separately because their Coordinate Reference System (CRS) is unknown.
+- **Duplicate Location Detection**: Identifies exact duplicate coordinate pairs across records within CSV datasets.
+- **Attribute Missingness Profiling**: Checks column-level completeness. Attribute columns with missingness exceeding 5% are flagged (`WARN` at 5-20%, `HIGH` above 20%). Coordinate columns are excluded from column missingness to prevent duplicate reporting.
+- **CRS and Bounding Box Analysis**: Extracts declared CRS from GeoJSON datasets and computes spatial bounding boxes for geographic datasets to evaluate spatial coverage and cross-dataset bounding box overlaps.
+- **Cross-Dataset Schema Comparison**: Identifies normalized column name matches and data type discrepancies across datasets.
+
+## Dataset Readiness Levels
+
+Every scanned dataset is assigned an overall readiness status based on the highest severity finding detected:
+
+- **READY**: No blocking or moderate issues detected by active checks.
+- **REVIEW**: Moderate-severity findings detected (e.g., 5-20% missingness, duplicate locations, missing planar coordinates).
+- **HIGH RISK**: High-severity findings detected (e.g., >20% missing or invalid geographic coordinates, high column missingness).
+- **ERROR**: Unparseable files or critical dataset structural failure.
+
+## Demo Data
+
+The included `data/` directory contains synthetic test cases demonstrating common data issues:
+
+| File | Description | Primary Findings |
+|---|---|---|
+| `exploration_samples.csv` | 12 sample points | Out-of-range coordinates, missing lat/lon, 41.7% missing notes, duplicate locations |
+| `historical_surveys.csv` | 10 survey points | Clean dataset using `lat`/`lon` column headers |
+| `geology.geojson` | 5 Point geometries | Clean GeoJSON dataset with EPSG:4326 |
+| `planar_samples.csv` | 8 sample points | Easting/northing planar coordinates with 12.5% missing values |
+
+Run the demo against the included data:
+
+```bash
+geo-audit data/
+```
+
+## Scope and Limitations
+
+- **Supported Formats**: CSV (`.csv`) and GeoJSON (`.geojson`, `.json`).
+- **Duplicate Detection**: Performs exact coordinate pair matching; near-duplicate spatial clustering is not performed.
+- **Spatial Overlap**: Evaluates bounding box intersection; polygon geometry intersections are out of scope.
+- **CRS Transformations**: Does not attempt automatic CRS inference or coordinate re-projection.
+
+## License
+
+MIT

geo_data_audit-0.1.0/geo_audit/__init__.py

@@ -0,0 +1,3 @@
+"""geo-audit: Audit heterogeneous geospatial datasets."""
+
+__version__ = "0.1.0"

geo_data_audit-0.1.0/geo_audit/cli.py

@@ -0,0 +1,205 @@
+"""Main CLI entry point."""
+
+import sys
+from datetime import datetime
+from pathlib import Path
+
+import typer
+from rich.console import Console
+
+from geo_audit.coordinates import detect_coordinate_columns, detect_planar_coordinates
+from geo_audit.discovery import discover_files, get_file_size
+from geo_audit.findings import (
+    build_audit_summary,
+    compute_dataset_readiness,
+    generate_findings,
+)
+from geo_audit.loaders import (
+    calculate_bounding_box,
+    calculate_bounding_box_from_coords,
+    infer_geometry_types,
+    load_dataset,
+    profile_columns,
+)
+from geo_audit.models import (
+    AuditReport,
+    DatasetInfo,
+    FileFormat,
+)
+from geo_audit.quality import detect_exact_duplicates
+from geo_audit.reporting import generate_json_report, generate_terminal_report
+from geo_audit.spatial import compute_cross_dataset_overlaps
+
+APP_HELP = (
+    "Audit heterogeneous geospatial datasets for data-quality "
+    "and spatial-consistency problems."
+)
+
+app = typer.Typer(
+    name="geo-audit",
+    help=APP_HELP,
+    add_completion=False,
+)
+
+# Separate console for progress (stderr) and report (stdout)
+progress_console = Console(
+    stderr=True, force_terminal=True, color_system="standard", legacy_windows=False
+)
+
+
+@app.command()
+def audit(
+    input_dir: Path = typer.Argument(..., help="Directory containing geospatial datasets"),
+    output: Path | None = typer.Option(
+        None, "--output", "-o", help="Output file path (only with --format json)"
+    ),
+    format: str = typer.Option("terminal", "--format", "-f", help="Output format: terminal, json"),
+):
+    """Audit geospatial datasets in a directory."""
+    if not input_dir.exists() or not input_dir.is_dir():
+        progress_console.print(f"[red]Error: {input_dir} is not a valid directory[/red]")
+        raise typer.Exit(1)
+
+    if output and format != "json":
+        progress_console.print("[red]Error: --output only works with --format json[/red]")
+        raise typer.Exit(1)
+
+    datasets = []
+
+    for file_path, file_format in discover_files(input_dir):
+        if format == "terminal":
+            progress_console.print(f"[dim]Processing {file_path.name}...[/dim]")
+        ds_info = process_dataset(file_path, file_format)
+        datasets.append(ds_info)
+
+    if not datasets:
+        progress_console.print("[yellow]No supported datasets found[/yellow]")
+        raise typer.Exit(0)
+
+    overlaps = compute_cross_dataset_overlaps(datasets)
+
+    # Generate findings and readiness
+    findings = generate_findings(datasets)
+    dataset_readiness = compute_dataset_readiness(datasets, findings)
+    summary = build_audit_summary(datasets, findings)
+
+    report = AuditReport(
+        datasets=datasets,
+        cross_dataset_overlaps=overlaps,
+        summary=summary.to_dict(),
+        generated_at=datetime.now().isoformat(),
+        findings=findings,
+        dataset_readiness=dataset_readiness,
+    )
+
+    if format == "json":
+        json_output = generate_json_report(report)
+        if output:
+            output.write_text(json_output)
+            # Don't print success message to stdout in JSON mode
+        else:
+            # Print JSON to stdout (only JSON, no progress messages)
+            sys.stdout.write(json_output + "\n")
+    else:
+        generate_terminal_report(report)
+
+
+def process_dataset(file_path: Path, file_format: FileFormat) -> DatasetInfo:
+    """Process a single dataset and return DatasetInfo."""
+    file_size = get_file_size(file_path)
+    data, parse_errors = load_dataset(file_path, file_format)
+
+    if data is None or (hasattr(data, 'empty') and data.empty):
+        return DatasetInfo(
+            filename=file_path.name,
+            format=file_format,
+            row_count=0,
+            columns=[],
+            column_profiles=[],
+            file_size_bytes=file_size,
+            parse_errors=parse_errors,
+        )
+
+    if file_format == FileFormat.CSV:
+        columns = list(data.columns)
+        column_profiles = profile_columns(data)
+        row_count = len(data)
+
+        coord_info = detect_coordinate_columns(data)
+        planar_info = detect_planar_coordinates(data)
+
+        duplicate_count = 0
+        duplicate_pct = 0.0
+        if coord_info.lat_column and coord_info.lon_column:
+            duplicate_count, duplicate_pct = detect_exact_duplicates(
+                data, coord_info.lat_column, coord_info.lon_column
+            )
+
+        bounding_box = None
+        if coord_info.lat_column and coord_info.lon_column:
+            bounding_box = calculate_bounding_box_from_coords(
+                data, coord_info.lat_column, coord_info.lon_column
+            )
+
+        # Warnings kept for backward compatibility in DatasetInfo
+        warnings = []
+        if coord_info.lat_column and coord_info.lon_column:
+            if coord_info.missing_percentage > 5:
+                warnings.append(f"{coord_info.missing_percentage:.1f}% missing coordinates")
+            if coord_info.invalid_percentage > 0:
+                warnings.append(f"{coord_info.invalid_percentage:.1f}% invalid coordinates")
+            if duplicate_pct > 1:
+                warnings.append(f"{duplicate_pct:.1f}% exact duplicate locations")
+        if planar_info:
+            if planar_info.missing_percentage > 5:
+                msg = f"Planar coordinates ({planar_info.x_column}/{planar_info.y_column}): "
+                msg += f"{planar_info.missing_percentage:.1f}% missing"
+                warnings.append(msg)
+
+        return DatasetInfo(
+            filename=file_path.name,
+            format=file_format,
+            row_count=row_count,
+            columns=columns,
+            column_profiles=column_profiles,
+            file_size_bytes=file_size,
+            coordinate_info=coord_info,
+            planar_coordinate_info=planar_info,
+            bounding_box=bounding_box,
+            duplicate_count=duplicate_count,
+            duplicate_percentage=duplicate_pct,
+            parse_errors=parse_errors,
+            warnings=warnings,
+        )
+
+    else:
+        columns = [c for c in data.columns if c != 'geometry']
+        non_geom = data.drop(columns=['geometry']) if 'geometry' in data.columns else data
+        column_profiles = profile_columns(non_geom)
+        row_count = len(data)
+
+        geometry_types = infer_geometry_types(data)
+        crs = str(data.crs) if data.crs else None
+        bounding_box = calculate_bounding_box(data)
+
+        warnings = []
+        if crs is None:
+            warnings.append("No CRS declared in file (GeoJSON spec assumes WGS84)")
+
+        return DatasetInfo(
+            filename=file_path.name,
+            format=file_format,
+            row_count=row_count,
+            columns=columns,
+            column_profiles=column_profiles,
+            file_size_bytes=file_size,
+            geometry_types=geometry_types,
+            crs=crs,
+            bounding_box=bounding_box,
+            parse_errors=parse_errors,
+            warnings=warnings,
+        )
+
+
+if __name__ == "__main__":
+    app()

geo_data_audit-0.1.0/geo_audit/coordinates.py

@@ -0,0 +1,115 @@
+"""Coordinate detection and validation."""
+
+
+import pandas as pd
+
+from geo_audit.models import CoordinateInfo, PlanarCoordinateInfo
+
+LAT_PATTERNS = [
+    "latitude", "lat", "lat_dd", "lat_ddm", "lat_dms",
+    "y_lat", "ycoord_lat", "gps_lat", "gps_latitude",
+]
+
+LON_PATTERNS = [
+    "longitude", "lon", "lng", "long",
+    "lon_dd", "lon_ddm", "lon_dms",
+    "x_lon", "xcoord_lon", "gps_lon", "gps_longitude",
+]
+
+PLANAR_PATTERNS = [
+    ("x", "y"),
+    ("x_coord", "y_coord"),
+    ("xcoordinate", "ycoordinate"),
+    ("easting", "northing"),
+]
+
+
+def detect_coordinate_columns(df: pd.DataFrame) -> CoordinateInfo:
+    """Detect latitude/longitude columns in a DataFrame.
+
+    Only explicit latitude/longitude column names are treated as geographic coordinates.
+    Generic x/y or easting/northing columns are NOT assumed to be lat/lon.
+    """
+    cols_lower = {c.lower().strip(): c for c in df.columns}
+    info = CoordinateInfo()
+
+    lat_col = _find_column(cols_lower, LAT_PATTERNS)
+    lon_col = _find_column(cols_lower, LON_PATTERNS)
+
+    if lat_col and lon_col:
+        info.lat_column = lat_col
+        info.lon_column = lon_col
+        info.detection_method = "explicit_lat_lon"
+        info.confidence = 0.95
+        _validate_coordinates(df, info)
+
+    return info
+
+
+def detect_planar_coordinates(df: pd.DataFrame) -> PlanarCoordinateInfo | None:
+    """Detect planar coordinate columns (x/y, easting/northing).
+
+    These are NOT validated as lat/lon since their CRS is unknown.
+    Returns info about the columns if found.
+    """
+    cols_lower = {c.lower().strip(): c for c in df.columns}
+
+    for x_pat, y_pat in PLANAR_PATTERNS:
+        x_col = _find_column(cols_lower, [x_pat])
+        y_col = _find_column(cols_lower, [y_pat])
+        if x_col and y_col:
+            total = len(df)
+            missing = int(df[x_col].isna().sum() | df[y_col].isna().sum())
+            return PlanarCoordinateInfo(
+                x_column=x_col,
+                y_column=y_col,
+                detection_method="planar_heuristic",
+                count=total,
+                missing_count=missing,
+            )
+    return None
+
+
+def _find_column(cols_lower: dict[str, str], patterns: list[str]) -> str | None:
+    """Find a column matching any of the patterns."""
+    for pat in patterns:
+        if pat in cols_lower:
+            return cols_lower[pat]
+    return None
+
+
+def _validate_coordinates(df: pd.DataFrame, info: CoordinateInfo) -> None:
+    """Validate coordinate values as lat/lon."""
+    lat_col = info.lat_column
+    lon_col = info.lon_column
+
+    if not lat_col or not lon_col:
+        return
+
+    total = len(df)
+    info.total_count = total
+
+    lat_series = pd.to_numeric(df[lat_col], errors="coerce")
+    lon_series = pd.to_numeric(df[lon_col], errors="coerce")
+
+    missing_mask = lat_series.isna() | lon_series.isna()
+    info.missing_count = int(missing_mask.sum())
+
+    valid_rows_lat = lat_series[~missing_mask]
+    valid_rows_lon = lon_series[~missing_mask]
+    if valid_rows_lat.empty:
+        info.valid_count = 0
+        info.invalid_count = 0
+        return
+
+    lat_valid = (valid_rows_lat >= -90) & (valid_rows_lat <= 90)
+    lon_valid = (valid_rows_lon >= -180) & (valid_rows_lon <= 180)
+    both_valid = lat_valid & lon_valid
+
+    info.valid_count = int(both_valid.sum())
+    info.invalid_count = int((~both_valid).sum())
+
+
+def validate_lat_lon(lat: float, lon: float) -> bool:
+    """Check if lat/lon values are valid."""
+    return -90 <= lat <= 90 and -180 <= lon <= 180

geo_data_audit-0.1.0/geo_audit/discovery.py

@@ -0,0 +1,40 @@
+"""Dataset discovery module."""
+
+from collections.abc import Iterator
+from pathlib import Path
+
+from geo_audit.models import FileFormat
+
+SUPPORTED_EXTENSIONS = {
+    ".csv": FileFormat.CSV,
+    ".geojson": FileFormat.GEOJSON,
+    ".json": FileFormat.GEOJSON,
+}
+
+
+def discover_files(root: Path) -> Iterator[tuple[Path, FileFormat]]:
+    """Recursively discover supported files in a directory."""
+    if not root.exists():
+        raise FileNotFoundError(f"Path does not exist: {root}")
+    if not root.is_dir():
+        raise NotADirectoryError(f"Path is not a directory: {root}")
+
+    for path in root.rglob("*"):
+        if path.is_file():
+            ext = path.suffix.lower()
+            if ext in SUPPORTED_EXTENSIONS:
+                yield path, SUPPORTED_EXTENSIONS[ext]
+
+
+def get_file_format(path: Path) -> FileFormat:
+    """Determine file format from extension."""
+    ext = path.suffix.lower()
+    return SUPPORTED_EXTENSIONS.get(ext, FileFormat.UNKNOWN)
+
+
+def get_file_size(path: Path) -> int:
+    """Get file size in bytes."""
+    try:
+        return path.stat().st_size
+    except OSError:
+        return 0