als-finder 0.0.0__tar.gz

als_finder-0.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Jonathan Greenberg / University of Nevada, Reno
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

als_finder-0.0.0/MANIFEST.in

@@ -0,0 +1,4 @@
+include README.md
+include LICENSE
+recursive-include examples *
+recursive-include src/als_finder/data *

als_finder-0.0.0/PKG-INFO

@@ -0,0 +1,447 @@
+Metadata-Version: 2.4
+Name: als-finder
+Version: 0.0.0
+Summary: A high-performance, cloud-native CLI engine for discovering and parsing raw LiDAR point cloud metadata.
+Home-page: https://github.com/cms-2024-hudak/als-finder
+Author: Jonathan Greenberg
+Author-email: jgreenberg@unr.edu
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: click
+Requires-Dist: requests
+Requires-Dist: geopandas
+Requires-Dist: shapely
+Requires-Dist: pyproj
+Requires-Dist: python-dotenv
+Requires-Dist: tqdm
+Requires-Dist: pyogrio
+Requires-Dist: pystac
+Requires-Dist: stac-validator
+Requires-Dist: psutil
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: setuptools_scm; extra == "dev"
+Requires-Dist: build; extra == "dev"
+Provides-Extra: pdal
+Requires-Dist: pdal; extra == "pdal"
+Provides-Extra: all
+Requires-Dist: pdal; extra == "all"
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license-file
+Dynamic: provides-extra
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# als-finder
+
+**A high-performance, cloud-native CLI engine for discovering, streaming, and partitioning raw LiDAR point cloud data across the globe.**
+
+[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/cms-2024-hudak/als-finder)
+
+`als-finder` gathers complete acquisition footprints (project boundaries), true WGS84 point densities, and metadata from **USGS 3DEP (EPT/COPC)**, **NOAA Digital Coast**, **NASA G-LiHT**, **NEON**, and **OpenTopography** into clean `.json` manifests, `QGIS`-ready `.gpkg` tables, and on-demand metric spatial tiles with automated Out-Of-Memory (OOM) protection.
+
+---
+
+## 🚀 Quick Install
+
+Because `als-finder` uses advanced C++ geospatial libraries (GDAL, PDAL, GEOS), **Conda** or **Docker** is the recommended installation method:
+
+```bash
+# 1. Create environment with all C++ and Python dependencies
+conda create -n als-finder -c conda-forge python=3.11 geopandas pdal python-pdal pystac stac-validator psutil shapely pyproj tqdm pyogrio requests click python-dotenv laspy -y
+
+# 2. Activate environment & install als-finder
+conda activate als-finder
+pip install git+https://github.com/cms-2024-hudak/als-finder.git
+```
+
+> [!TIP]
+> 📖 **Full Installation Guide:** See [**docs/INSTALLATION.md**](docs/INSTALLATION.md) for Conda-Forge releases, Docker container runs, Singularity/Apptainer HPC setup, and Windows/WSL2 instructions.
+
+---
+
+## 🧭 The Big-Data Architecture: The 4-Stage ALS Lifecycle
+
+Traditional tools force researchers to download massive, multi-gigabyte survey archives before doing any work. `als-finder` is built from the ground up for **cloud-native lazy evaluation and HPC job arrays**:
+
+```
+┌─────────────────┐     ┌──────────────────┐     ┌──────────────────┐     ┌──────────────────┐
+│  1. DISCOVERY   │ ──> │   2. PLANNING    │ ──> │  3. ACQUISITION  │ ──> │   4. DERIVATIVE  │
+│    (search)     │     │     (plan)       │     │     (fetch)      │     │    & MOSAIC      │
+└─────────────────┘     └──────────────────┘     └──────────────────┘     └──────────────────┘
+ • Federated APIs        • Spatial tiling index   • On-demand streaming    • Crop buffers
+ • ROI & temporal        • Memory risk audit      • OOM auto-quadrants     • gdalbuildvrt
+ • Provenance catalog    • Slurm task lists       • Sidecar crop bounds    • Zero edge-effects
+```
+
+---
+
+## ⚡ Master Tutorial: End-to-End Walkthrough
+
+This single, comprehensive tutorial walks through the complete lifecycle using a single workspace: `scratch/test_workspace`.
+
+### Step 0: Extract the Bundled Example Area of Interest (AOI)
+`als-finder` comes with a bundled Lake Tahoe Basin Management Unit (LTBMU) boundary polygon so you can follow along immediately:
+
+```bash
+# Extract the example polygon (ltbmu_boundary.gpkg) to your current directory
+als-finder get-example-roi
+```
+
+---
+
+### Step 1: Federated Discovery (`als-finder search`)
+Search across all public LiDAR registries intersecting your AOI. Using `--cloud-native` filters exclusively for streaming-ready formats (USGS EPT / COPC), and `--dedup` eliminates duplicate survey records:
+
+```bash
+als-finder search \
+  --roi ./ltbmu_boundary.gpkg \
+  --workspace scratch/test_workspace \
+  --cloud-native \
+  --dedup
+```
+
+**Console Output:**
+```text
+=================================================================================================================
+ LiDAR Data Search Results 
+=================================================================================================================
+ | Provider        | Name                                   | Date         |   Est (GB) |   pts/m2 |   Area km2 |
+-----------------------------------------------------------------------------------------------------------------
+ | USGS_EPT        | CA_SierraNevada_5_2022                 | 2022-??-??   |    1380.20 |  29.1800 |    6347.47 |
+ | USGS_EPT        | CA_SierraNevada_6_2022                 | 2022-??-??   |    1136.46 |  26.2100 |    5819.74 |
+ | USGS_EPT        | CA_SierraNevada_8_2022                 | 2022-??-??   |    1171.62 |  25.1400 |    6255.39 |
+ | NASA_GLIHT      | NASA G-LiHT Sierra Nevada / Lake Tahoe | 2022-07-18   |      30.73 |  32.5000 |     126.90 |
+ | USGS_EPT        | NV_WestCentralEarthMRI_3_2020          | 2020-??-??   |     433.16 |   5.3400 |   10890.04 |
+ | USGS_EPT        | CA_UpperSouthAmerican_Eldorado_2019    | 2019-??-??   |    2075.29 |  43.1600 |    6454.20 |
+ | NOAA_STAC       | DigitalCoast_DAV:id_9452               | 2019-10-21   |    2075.29 |  46.7700 |    5954.91 |
+ | USGS_EPT        | USGS_LPC_CA_NoCAL_Wildfires_B1_2018    | 2018-??-??   |     643.56 |  10.8900 |    7928.51 |
+ | NOAA_STAC       | DigitalCoast_DAV:id_9036               | 2018-07-07   |     253.48 |   0.7800 |   43731.68 |
+ | USGS_EPT        | USGS_LPC_NV_Reno_Carson_QL1_2017_LAS_2 | 2017-??-??   |     151.15 |   9.5400 |    2126.64 |
+ | USGS_EPT        | CA_PlacerCo_2012                       | 2012-??-??   |      36.96 |   3.9500 |    1254.54 |
+=================================================================================================================
+ TOTAL DATASETS: 11 | ESTIMATED PAYLOAD: 9426.32 GB | QUERY TIME: 4.31s 
+-----------------------------------------------------------------------------------------------------------------
+ CATALOG TBL: scratch/test_workspace/catalog/catalog.gpkg
+ JSON METADATA: scratch/test_workspace/catalog/manifest.json
+=================================================================================================================
+```
+
+> [!NOTE]
+> **Zero Heavy Downloads:** Notice that 9.4 TB of data was discovered in 4 seconds. No heavy point cloud files were downloaded to your hard drive.
+
+---
+
+### Step 2: Spatial Planning & Memory Risk Audit (`als-finder plan`)
+Before launching jobs, use `als-finder plan` to inspect the study area's metric grid and audit memory safety against a target point budget (e.g. 4,000,000 points $\approx$ 4 GB RAM):
+
+```bash
+als-finder plan \
+  --workspace scratch/test_workspace \
+  --tile-size 500 \
+  --buffer-size 50 \
+  --max-points 4000000
+```
+
+**Console Output:**
+```text
+==================================================
+ ALS-FINDER SPATIAL PLANNING & GRID METRICS
+==================================================
+  Master Tiles:      5,657
+  Total Leaf Tasks:  22,628
+  Tile ID Range:     0 to 5656
+  Tile Size:         500m (core)
+  Buffer Size:       50m (overlap)
+  Grid CRS:          EPSG:32610
+  Target Tile ID:    0
+  Basename:          CA_SierraNevada_5_2022_tile_E0737500_N4329500
+  Hive Directory:    provider=USGS_EPT/dataset=CA_SierraNevada_5_2022/tilesize=500/buffer=50
+  Crop PDAL Bounds:  ([737500.0, 738000.0], [4329000.0, 4329500.0])
+  Crop GDAL -te:     737500.0 4329000.0 738000.0 4329500.0
+  Grid File:         scratch/test_workspace/catalog/grids/tilesize=500/buffer=50/grid.gpkg
+--------------------------------------------------
+ MEMORY RISK AUDIT (PRE-FLIGHT):
+  Max Points Budget: 4,000,000
+  Est Points/Tile:   10,504,800
+  Subdivision:       5657 tiles exceed budget (split into 4 quadrants each)
+  Slurm Array Size:  --array=1-22628
+==================================================
+```
+
+#### Extracting Fields Directly for Shell Scripts (`--field` and `--format env`):
+```bash
+# Query exact scalar fields:
+als-finder plan --workspace scratch/test_workspace --tile-size 500 --buffer-size 50 --field total_tiles
+# Output: 5657
+
+# Export full environment variables into your active shell:
+eval $(als-finder plan --workspace scratch/test_workspace --tile-id 15 --tile-size 500 --buffer-size 50 --format env)
+echo "Target Basename: $BASENAME"
+echo "Crop Bounds: $CROP_GDAL_TE"
+```
+
+> [!TIP]
+> **Standardized Grid Alignment (1200m Core / 30m Buffer):**
+> By default, `als-finder` generates $1200\,\text{m}$ core tiles with a $30\,\text{m}$ buffer. This geometry guarantees exact pixel integer divisibility across standard remote sensing raster resolutions—$30\,\text{m}$ ($40\,\text{px}$ core, $42\,\text{px}$ buffered), $10\,\text{m}$ ($120\,\text{px}$), and $1\,\text{m}$ ($1200\,\text{px}$)—completely preventing sub-pixel edge interpolation artifacts.
+
+---
+
+### Step 3: Generating HPC Slurm Task Lists & Manifests (`--tasks` / `--tasks-csv`)
+When scheduling Slurm job arrays, you can output either a plain leaf tile list (`--tasks`) or a complete CSV manifest (`--tasks-csv`) with spatial coordinates, Hive paths, point counts, and recommended Slurm memory allocations:
+
+```bash
+# 1. Plain leaf task list for simple Slurm array indexes:
+als-finder plan \
+  --workspace scratch/test_workspace \
+  --tasks > slurm_tasks.txt
+
+# 2. Comprehensive CSV manifest for Slurm arrays, HPC job schedulers, and tracking:
+als-finder plan \
+  --workspace scratch/test_workspace \
+  --tasks-csv slurm_manifest.csv
+```
+
+**Manifest Columns in `slurm_manifest.csv`:**
+- `task_id`: Unique leaf task identifier (`0`, `15_NW`, etc.)
+- `tile_id`: Parent master tile integer
+- `basename`: Standardized spatial basename (`<dataset>_tile_E<easting>_N<northing>[_quad]`)
+- `hive_dir` & `hive_path`: Deterministic Hive partition directory and destination file path
+- `core_minx`, `core_miny`, `core_maxx`, `core_maxy`: Core bounding coordinates
+- `point_density`, `est_points`: Acquisition point density ($pts/m^2$) and pre-flight point count
+- `recommended_mem_gb`: Pre-calculated Slurm job memory allocation ($\approx 1.5 \times$ point payload)
+
+---
+
+### Step 4: On-Demand Point Cloud Streaming (`als-finder fetch tile`)
+Acquire standard tile 15 directly over HTTP using clean positional syntax. Adding `--sidecar` writes a companion `.json` metadata file:
+
+```bash
+als-finder fetch tile 15 \
+  --workspace scratch/test_workspace \
+  --tile-size 500 \
+  --buffer-size 50 \
+  --sidecar \
+  --overwrite
+```
+
+**Console Output:**
+```text
+Successfully streamed tile 15 to scratch/test_workspace/data/tiles/provider=USGS_EPT/dataset=CA_SierraNevada_5_2022/tilesize=500/buffer=50/CA_SierraNevada_5_2022_tile_E0738000_N4331000.laz
+Wrote metadata sidecar to scratch/test_workspace/data/tiles/provider=USGS_EPT/dataset=CA_SierraNevada_5_2022/tilesize=500/buffer=50/CA_SierraNevada_5_2022_tile_E0738000_N4331000.json
+```
+
+---
+
+### Step 5: Automatic OOM Protection & Quadrant Streaming
+If a tile is hyper-dense and would exceed your RAM limit, passing `--max-points` causes `als-finder` to automatically bisect it into 4 safe quadrants:
+
+```bash
+als-finder fetch tile 15 \
+  --workspace scratch/test_workspace \
+  --tile-size 500 \
+  --buffer-size 50 \
+  --max-points 4000000 \
+  --sidecar \
+  --overwrite
+```
+
+**Console Output:**
+```text
+Tile 15 exceeded budget. Subdivided into 4 quadrants:
+  - Streamed 15_NW to .../CA_SierraNevada_5_2022_tile_E0738000_N4331000_NW.laz
+  - Streamed 15_NE to .../CA_SierraNevada_5_2022_tile_E0738000_N4331000_NE.laz
+  - Streamed 15_SW to .../CA_SierraNevada_5_2022_tile_E0738000_N4331000_SW.laz
+  - Streamed 15_SE to .../CA_SierraNevada_5_2022_tile_E0738000_N4331000_SE.laz
+```
+
+> [!IMPORTANT]
+> **Co-Located Subtiles & Constant Buffer:**
+> 1. All quadrants sit directly in the parent Hive directory (`tilesize=500/buffer=50`), eliminating nested directory crawling.
+> 2. The 50m buffer is **fully preserved** across all subdivisions, preventing edge-effect interpolation degradation.
+
+---
+
+### Step 6: Multi-Tile Batch Streaming
+Stream multiple tiles in a single concise command:
+
+```bash
+als-finder fetch tile 14 16 \
+  --workspace scratch/test_workspace \
+  --tile-size 500 \
+  --buffer-size 50 \
+  --sidecar \
+  --overwrite
+```
+
+**Console Output:**
+```text
+Successfully processed 2 target tile(s) (2 file(s) generated):
+  - Tile 14 -> .../CA_SierraNevada_5_2022_tile_E0738000_N4330500.laz
+  - Tile 16 -> .../CA_SierraNevada_5_2022_tile_E0738000_N4331500.laz
+```
+
+---
+
+### Step 7: Downstream Unbuffering & 1-Line Mosaicking
+
+Because every `.laz` file includes a companion `.json` sidecar storing exact unbuffering crop parameters (`crop_gdal_te` / `crop_pdal_bounds`), downstream processing and mosaicking are completely automated:
+
+```bash
+TILES_DIR="scratch/test_workspace/data/tiles/provider=USGS_EPT/dataset=CA_SierraNevada_5_2022/tilesize=500/buffer=50"
+
+# 1. Process point clouds and crop away the 50m buffer
+for LAZ in "$TILES_DIR"/*.laz; do
+    SIDECAR="${LAZ%.laz}.json"
+    CROP_BOX=$(jq -r '.crop_gdal_te' "$SIDECAR")
+
+    # Example: compute Canopy Height Model (CHM) with buffer, then crop to core:
+    # my_chm_tool "$LAZ" temp.tif
+    # gdal_translate -projwin $CROP_BOX temp.tif "${LAZ%.laz}_chm.tif"
+done
+
+# 2. Build a seamless mosaic across the entire study area (works for both 500m and 250m tiles):
+# gdalbuildvrt full_study_area_chm.vrt "$TILES_DIR"/*_chm.tif
+```
+
+#### Native lidR (R) Integration & Buffer Point Tagging
+Streamed `.laz` tiles embed a native LAS 1.4 `buffer` extra dimension (`buffer=0` for points inside the core tile, `buffer=1` for boundary buffer overlap points). In R using `lidR`, you can filter away buffer points after edge-artifact-free metric calculation with zero geometry math:
+
+```r
+library(lidR)
+
+# Read LAS tile with extra dimensions:
+las <- readLAS("tile.laz", select = "xyzc", filter = "")
+
+# Compute metrics (e.g., CHM, canopy cover) with buffer points preventing edge voids:
+# ...
+
+# Instant zero-copy filter of core points only:
+core_pts <- filter_poi(las, buffer == 0)
+```
+
+---
+
+### Step 8: High-Throughput Slurm Job Array Recipe
+
+Here is the production Slurm script for running on-demand streaming across hundreds of HPC cores with zero OOM risk and instant scratch cleanup:
+
+```bash
+#!/usr/bin/env bash
+#SBATCH --job-name=als_tiles
+#SBATCH --array=0-5656
+#SBATCH --cpus-per-task=1
+#SBATCH --mem=4G
+#SBATCH --time=00:30:00
+
+set -e
+
+WORKSPACE="/shared/project/workspace"
+FINAL_DIR="/shared/project/products/chm"
+LOCAL_SCRATCH="$TMPDIR/als_work_${SLURM_ARRAY_TASK_ID}"
+mkdir -p "$LOCAL_SCRATCH" "$FINAL_DIR"
+
+# 1. Fetch on demand to local scratch (auto-subdivides if dense)
+als-finder fetch tile "$SLURM_ARRAY_TASK_ID" \
+    --workspace "$WORKSPACE" \
+    --tile-size 500 \
+    --buffer-size 50 \
+    --max-points 4000000 \
+    --output "$LOCAL_SCRATCH" \
+    --sidecar \
+    --overwrite
+
+# 2. Process all files in scratch (whether 1 tile or 4 quadrants)
+for LAZ in "$LOCAL_SCRATCH"/*.laz; do
+    SIDECAR="${LAZ%.laz}.json"
+    BASENAME=$(basename "$LAZ" .laz)
+    CROP_BOX=$(jq -r '.crop_gdal_te' "$SIDECAR")
+
+    # Run point cloud metric tool
+    # compute_metric "$LAZ" "$LOCAL_SCRATCH/${BASENAME}_raw.tif"
+
+    # Crop off buffer to permanent storage
+    # gdal_translate -projwin $CROP_BOX "$LOCAL_SCRATCH/${BASENAME}_raw.tif" "$FINAL_DIR/${BASENAME}_chm.tif"
+
+    # Immediate cleanup of raw point cloud
+    rm -f "$LAZ" "$SIDECAR"
+done
+
+# Clean scratch
+rm -rf "$LOCAL_SCRATCH"
+```
+
+---
+
+### Step 9: Bulk Offline Survey Archiving (`als-finder fetch survey`)
+For researchers who prefer to pre-download the entire raw dataset locally for air-gapped or offline processing:
+
+```bash
+als-finder fetch survey \
+  --workspace scratch/test_workspace \
+  --name "CA_SierraNevada_5_2022" \
+  --execute
+```
+
+---
+
+### Step 10: Workspace Maintenance (`als-finder workspace`)
+Update catalog filters or clean interim scratch files:
+
+```bash
+# Update catalog criteria (e.g. restrict to QL1):
+als-finder workspace update --workspace scratch/test_workspace --density QL1
+
+# Clean scratch workspace:
+als-finder workspace clean --workspace scratch/test_workspace
+```
+
+---
+
+## 🐍 Programmatic Python API
+
+You can also import and use `als_finder` directly in Python scripts and Jupyter Notebooks:
+
+```python
+import als_finder
+
+# 1. Query tile specification and unbuffering bounds
+spec = als_finder.get_tile_spec(
+    workspace="scratch/test_workspace",
+    tile_id=15,
+    tile_size=500,
+    buffer_size=50
+)
+
+print(f"Basename: {spec['basename']}")
+print(f"Grid CRS: {spec['grid_crs']}")
+print(f"Crop GDAL -te: {spec['crop_gdal_te']}")
+
+# 2. Inspect Central Core Area vs Buffered Footprint
+core = spec['core_poly']
+buffered = spec['buffered_poly']
+
+print(f"Core Tile Dimensions: {round(core.bounds[2] - core.bounds[0])}m x {round(core.bounds[3] - core.bounds[1])}m")
+print(f"Buffer Extension: {round((buffered.bounds[2] - core.bounds[2]))}m on all sides")
+```
+
+---
+
+## 🏛️ Authorship & Citation
+
+This software is released under the open-source **MIT License**. Copyright **Jonathan Greenberg**.
+
+**Project Authors & Contributors:**
+* **Jonathan Greenberg** (University of Nevada, Reno): Lead Developer and Core Project Architect.
+* **Andrew Hudak** (US Forest Service): Advisory feedback and scientific computing grant alignment.
+* **Antigravity (Google DeepMind)**: AI Software Engineering Partner.