giga-spatial 0.6.0__tar.gz

giga_spatial-0.6.0/.env_sample

@@ -0,0 +1,20 @@
+export ADLS_CONNECTION_STRING=""
+export ADLS_CONTAINER_NAME=""
+export GOOGLE_SERVICE_ACCOUNT=""
+export API_PROFILE_FILE_PATH=""
+export API_SHARE_NAME=""
+export API_SCHEMA_NAME=""
+export MAPBOX_ACCESS_TOKEN=""
+export MAXAR_USERNAME=""
+export MAXAR_PASSWORD=""
+export MAXAR_CONNECTION_STRING=""
+export OPENCELLID_ACCESS_TOKEN=""
+export GEOREPO_API_KEY=""
+export GEOREPO_USER_EMAIL=""
+export GIGA_SCHOOL_LOCATION_API_KEY=""
+export BRONZE_DIR=""
+export SILVER_DIR=""
+export GOLD_DIR=""
+export VIEWS_DIR=""
+export CACHE_DIR=""
+export ADMIN_BOUNDARIES_DIR=""

giga_spatial-0.6.0/CHANGELOG.md

@@ -0,0 +1,269 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+## [v0.6.0] - 2025-06-09
+
+### Added
+
+#### POI View Generator
+- **`map_zonal_stats`**: New method for enriched spatial mapping with support for:
+  - Raster point sampling (value at POI location)
+  - Raster zonal statistics (with buffer zone)
+  - Polygon aggregation (with optional area-weighted averaging)
+- **Auto-generated POI IDs** in `_init_points_gdf` for consistent point tracking.
+- **Support for area-weighted aggregation** for polygon-based statistics.
+
+#### BaseHandler Orchestration Layer
+- New abstract `BaseHandler` class providing unified lifecycle orchestration for config, downloader, and reader.
+- High-level interface methods:
+  - `ensure_data_available()`
+  - `load_data()`
+  - `download_and_load()`
+  - `get_available_data_info()`
+- Integrated factory pattern for safe and standardized component creation.
+- Built-in context manager support for resource cleanup.
+- Fully backwards compatible with existing handler architecture.
+
+#### Handlers Updated to Use BaseHandler
+- `GoogleOpenBuildingsHandler`
+- `MicrosoftBuildingsHandler`
+- `GHSLDataHandler`
+  - All now inherit from `BaseHandler`, supporting standardized behavior and cleaner APIs.
+
+---
+
+### Changed
+
+#### POI View Generator
+- `map_built_s` and `map_smod` now internally use the new `map_zonal_stats` method.
+- `tif_processors` renamed to `data` to support both raster and polygon inputs.
+- Removed parameters:
+  - `id_column` (now handled internally)
+  - `area_column` (now automatically calculated)
+
+#### Internals and Usability
+- Improved error handling with clearer validation messages.
+- Enhanced logging for better visibility during enrichment.
+- More consistent use of coordinate column naming.
+- Refined type hints and parameter documentation across key methods.
+
+---
+
+### Notes
+
+- Removed legacy POI generator classes and redundant `poi.py` file.
+- Simplified imports and removed unused handler dependencies.
+- All POI generator methods now include updated docstrings, parameter explanations, and usage examples.
+- Added docs on the new `BaseHandler` interface and handler refactors.
+
+## [v0.5.0] - 2025-06-02
+
+### Changed
+
+- **Refactored data loading architecture**:
+  - Introduced **dedicated reader classes** for major datasets (Microsoft Global Buildings, Google Open Buildings, GHSL), each inheriting from a new `BaseHandlerReader`.
+  - Centralized **file existence checks** and **raster/tabular loading** methods in `BaseHandlerReader`.
+  - Improved maintainability by encapsulating dataset-specific logic inside each reader class.
+
+- **Modularized source resolution**:
+  - Each reader now supports resolving data **by country, geometry, or individual points**, improving code reuse and flexibility.
+
+- **Unified POI enrichment**:
+  - Merged all POI generators (Google Open Buildings, Microsoft Global Buildings, GHSL Built Surface, GHSL SMOD) into a single `PoiViewGenerator` class.
+  - Supports flexible inputs: list of `(lat, lon)` tuples, list of dicts, DataFrame, or GeoDataFrame.
+  - Maintains consistent internal state via `points_gdf`, updated after each mapping.
+  - Enables **chained enrichment** of POI data using multiple datasets.
+
+- **Modernized internal data access**:
+  - All data loading now uses dedicated **handler/reader classes**, improving consistency and long-term maintainability.
+
+### Fixed
+
+- **Full DataStore integration**:
+  - Fixed `OpenCellID` and `HDX` handlers to fully support the `DataStore` abstraction.
+  - All file reads, writes, and checks now use the configured `DataStore` (local or cloud).
+  - Temporary files are only used during downloads; final data is always stored and accessed via the DataStore interface.
+
+### Removed
+
+- Removed deprecated POI generator classes and the now-obsolete poi submodule. All enrichment is handled through the unified `PoiViewGenerator`.
+
+### Notes
+
+- This release finalizes the architectural refactors started in `v0.5.0`.
+- While marked stable, please report any issues or regressions from the new modular structure.
+
+## [v0.5.0b1] - 2025-05-27
+
+### Added
+- **New Handlers**:
+  - `hdx.py`: Handler for downloading and managing Humanitarian Data Exchange datasets.
+  - `rwi.py`: Handler for the Relative Wealth Index dataset.
+  - `opencellid.py`: Handler for OpenCellID tower locations.
+  - `unicef_georepo.py`: Integration with UNICEF’s GeoRepo asset repository.
+- **Zonal Generators**:
+  - Introduced the `generators/zonal/` module to support spatial aggregations of various data types (points, polygons, rasters)
+    to zonal geometries such as grid tiles or catchment areas.
+- **New Geo-Processing Methods**:
+  - Added methods to compute centroids of (Multi)Polygon geometries.
+  - Added methods to calculate area of (Multi)Polygon geometries in square meters.
+
+### Changed
+- **Refactored**:
+  - `config.py`: Added support for new environment variables (OpenCellID and UNICEF GeoRepo keys).
+  - `geo.py`: Enhanced spatial join functions for improved performance and clarity.
+  - `handlers/`: 
+    - Minor robustness improvements in `google_open_buildings` and `microsoft_global_buildings`.
+    - Added a new class method in `boundaries` for initializing admin boundaries from UNICEF GeoRepo.
+  - `core/io/`: 
+    - Added `list_directories` method to both ADLS and local storage backends.
+- **Documentation & Project Structure**:
+  - Updated `.env_sample` and `.gitignore` to align with new environment variables and data handling practices.
+
+### Dependencies
+- Updated `requirements.txt` and `setup.py` to reflect new dependencies and ensure compatibility.
+
+### Notes
+- This is a **pre-release** (`v0.5.0b1`) and is intended for testing and feedback.
+- Some new modules, especially in `handlers` and `generators`, are experimental and may be refined in upcoming releases.
+
+## [v0.4.1] - 2025-04-17
+### Added
+- **Documentation**:
+	- Added **API Reference** documentation for all modules, classes, and functions.
+	- Added a **Configuration Guide** to explain how to set up paths, API keys, and other.
+- **TifProcessor**: added new to_dataframe method.
+- **config**: added set_path method for dynamic path management.
+
+### Changed
+- **Documentation**:
+	- Restructured the `docs/` directory to improve organization and navigation.
+	- Updated the `index.md` for the **User Guide** to provide a clear overview of available documentation.
+	- Updated **Examples** for downloading, processing, and storing geospatial data - more to come.
+- **README**:
+	- Updated the README with a clear description of the package’s purpose and key features.
+	- Added a section on **View Generators** to explain spatial context enrichment and mapping to grid or POI locations.
+	- Included a **Supported Datasets** section with an image of dataset provider logos.
+
+### Fixed
+- Handled errors when processing nodes, relations, and ways in **OSMLocationFetcher**.
+- Made `admin1` and `admin1_id_giga` optional in GigaEntity instances for countries with no admin level 1 divisions.
+
+## [v0.4.0] - 2025-04-01
+### Added
+- **POI View Generators**: Introduced a new module, generators, containing a base class for POI view generation.
+- **Expanded POI Support**: Added new classes for generating POI views from:
+	- **Google Open Buildings**
+	- **Microsoft Global Buildings**
+	- **GHSL Settlement Model**
+	- **GHSL Built Surface**
+- **New Reader**: Added read_gzipped_json_or_csv to handle compressed JSON/CSV files.
+
+### Changed
+- **ADLSDataStore Enhancements**: Updated methods to match LocalDataStore for improved consistency.
+- **Geo Processing Updates**:
+	- Improved convert_to_dataframe for more efficient data conversion.
+	- Enhanced annotate_with_admin_regions to improve spatial joins.
+- **New TifProcessor Methods**:
+	- sample_by_polygons for polygon-based raster sampling.
+	- sample_multiple_tifs_by_coordinates & sample_multiple_tifs_by_polygons to manage multi-raster sampling.
+- **Fixed Global Config Handling**: Resolved issues with handling configurations inside classes.
+
+## [v0.3.2] - 2025-03-21
+### Added
+- Added a method to efficiently assign unique IDs to features.
+
+### Changed
+- Enhanced logging for better debugging and clarity.
+
+### Fixed
+- Minor bug fix in config.py
+
+## [0.3.1] - 2025-03-20
+
+### Added
+- Enhanced AdminBoundaries handler with improved error handling for cases where administrative level data is unavailable for a country.
+- Added pyproject.toml and setup.py, enabling pip install support for the package.
+- Introduced a new method annotate_with_admin_regions in geo.py to perform spatial joins between input points and administrative boundaries (levels 1 and 2), handling conflicts where points intersect multiple admin regions.
+
+### Removed
+- Removed the utils module containing logger.py and integrated LOG_FORMAT and get_logger into config.py for a more streamlined logging approach.
+
+## [0.3.0] - 2025-03-18  
+### Added  
+- Compression support in readers for improved efficiency  
+- New GHSL data handler to manage GHSL dataset downloads  
+
+### Fixed  
+- Small fixes/improvements in Microsoft Buildings, Maxar, and Overture handlers  
+
+## [v0.2.2] - 2025-03-12
+- **Refactored Handlers**: Improved structure and performance of maxar_image.py, osm.py and overture.py to enhance geospatial data handling.
+
+- **Documentation Improvements**:
+	- Updated index.md, advanced.md, and use-cases.md for better clarity.
+	- Added installation.md under docs/getting-started for setup guidance.
+	- Refined API documentation in docs/api/index.md.
+
+- **Configuration & Setup Enhancements**:
+	•	Improved .gitignore to exclude unnecessary files.
+	•	Updated mkdocs.yml for better documentation structuring.
+- **Bug Fixes & Minor Optimizations**: Small fixes and improvements across the codebase for stability and maintainability.
+
+## [v0.2.1] - 2025-02-28
+### Added
+- Introduced WorldPopDownloader feature to handlers
+- Refactored TifProcessor class for better performance
+
+### Fixed
+- Minor bug fixes and performance improvements
+
+## [v0.2.0] - MaxarImageDownloader & Bug Fixes - 2025-02-24
+- **New Handler**: MaxarImageDownloader for downloading Maxar images.
+- **Bug Fixes**: Various improvements and bug fixes.
+- **Enhancements**: Minor optimizations in handlers.
+
+## [v0.1.1] - 2025-02-24
+
+### Added
+- **Local Data Store**: Introduced a new local data store alongside ADLS to improve data storage and read/write functionality.
+- **Boundaries Handler**: Added `boundaries.py`, a new handler that allows to read administrative boundaries from GADM.
+
+### Changed
+- **Handler Refactoring**: Refactored existing handlers to improve modularity and data handling.
+- **Configuration Management**: Added `config.py` to manage paths, runtime settings, and environment variables.
+
+### Removed
+- **Administrative Schema**: Removed `administrative.py` since its functionality is now handled by the `boundaries` handler.
+- **Globals Module**: Removed `globals.py` and replaced it with `config.py` for better configuration management.
+
+### Updated Files
+- `config.py`
+- `boundaries.py`
+- `google_open_buildings.py`
+- `mapbox_image.py`
+- `microsoft_global_buildings.py`
+- `ookla_speedtest.py`
+- `mercator_tiles.py`
+- `adls_data_store.py`
+- `data_store.py`
+- `local_data_store.py`
+- `readers.py`
+- `writers.py`
+- `entity.py`
+
+## [v0.1.0] - 2025-02-07
+### Added
+- New data handlers: `google_open_buildings.py`, `microsoft_global_buildings.py`, `overture.py`, `mapbox_image.py`, `osm.py`
+- Processing functions in `tif_processor.py`, `geo.py` and `transform.py`
+- Grid generation modules: `h3_tiles.py`, `mercator_tiles.py`
+- View managers: `grid_view.py` and `national_view.py`
+- Schemas: `administrative.py`
+
+### Changed
+- Updated `requirements.txt` with new dependencies
+- Improved logging and data storage mechanisms
+
+### Removed
+- Deprecated views: `h3_view.py`, `mercator_view.py`

giga_spatial-0.6.0/CODE_OF_CONDUCT.md

@@ -0,0 +1,86 @@
+# Code of Conduct
+
+At Giga, we're committed to maintaining an environment that's respectful, inclusive, and harassment-free for everyone involved in our project and community. We welcome contributors and participants from diverse backgrounds and pledge to uphold these standards:
+
+## Our Pledge​
+
+We promise to ensure a welcoming environment, regardless of:
+
+- Age
+- Body size
+- Disability
+- Ethnicity
+- Gender identity and expression
+- Level of experience
+- Education
+- Socio-economic status
+- Nationality
+- Personal appearance
+- Race
+- Religion
+- Sexual identity and orientation
+
+## Our Standards​
+
+Examples of behavior that contributes to creating a positive environment include:
+
+- Using language that's welcoming and inclusive
+- Respecting differing viewpoints and experiences
+- Embracing constructive criticism gracefully
+- Prioritizing the well-being of the community
+- Showing empathy towards fellow community members
+
+Examples of unacceptable behavior by participants include:
+
+- Using sexualized language or imagery, or making unwelcome advances
+- Engaging in trolling, derogatory comments, or personal attacks
+- Harassment in public or private spaces
+- Publishing others' private information without explicit consent
+- Any conduct that could be deemed inappropriate in a professional setting
+
+## Our Responsibilities​
+
+Project maintainers are entrusted with defining and upholding acceptable behavior standards. They have the authority to:
+
+- Clarify the standards of acceptable behavior
+- Take fair and appropriate action in response to unacceptable behavior
+- Remove, edit, or reject contributions that violate the Code of Conduct
+- Temporarily or permanently ban contributors engaging in inappropriate, offensive, or harmful behavior
+
+## Scope​
+
+This Code of Conduct applies within project spaces and when representing the project or community publicly. This representation includes using official project communication channels, social media, or acting as a representative at online/offline events. Project maintainers will define and clarify what constitutes representation.
+
+## Enforcement and Reporting​
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at [giga@unicef.org](mailto:giga@unicef.org). All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the reporter of any incident.
+
+## Community Spaces​
+
+- Twitter
+- Gigas' Website
+- Discord
+- Mail
+
+## Enforcement Guidelines​
+
+Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:
+
+- **Correction**
+  - Community Impact: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community.
+  - Consequence: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.
+- **Warning**
+  - Community Impact: A violation through a single incident or series of actions.
+  - Consequence: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.
+- **Temporary Ban**
+  - Community Impact: A serious violation of community standards, including sustained inappropriate behavior.
+  - Consequence: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.
+- **Permanent Ban**
+  - Community Impact: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals.
+  - Consequence: A permanent ban from any sort of public interaction within the community.
+
+## Attribution​
+
+This Code of Conduct is adapted from  the [Contributor Covenant code of conduct](https://www.contributor-covenant.org/version/2/1/code_of_conduct/).

giga_spatial-0.6.0/CONTRIBUTING.md

@@ -0,0 +1,49 @@
+# Contribution Guidelines
+
+Thank you for considering contributing to Giga! We value your input and aim to make the contribution process as accessible and transparent as possible. Whether you're interested in reporting bugs, discussing code, submitting fixes, proposing features, becoming a maintainer, or engaging with the Giga community, we welcome your involvement.
+
+## How to Contribute to our Giga Project?
+
+1. **Familiarize Yourself:** Before contributing, familiarize yourself with the project by reviewing the README, code of conduct, and existing issues or pull requests.
+2. **Issues and Feature Requests:** Check the issue tracker for existing issues or create a new one to report bugs, suggest improvements, or propose new features.
+3. **Fork and Branch:** Fork the repository and create a branch for your contribution. Branch names should be descriptive (e.g., feature/add-new-functionality, bugfix/issue-description).
+4. **Code Changes:** Make changes or additions following our coding conventions and style guide. Ensure to write clear commit messages that explain the purpose of each commit.
+5. **Testing:** If applicable, include tests for the changes made to ensure code reliability. Ensure existing tests pass.
+6. **Documentation:** Update relevant documentation, including README files or any other necessary guides.
+7. **Pull Request:** Open a pull request (PR) against the main branch. Clearly describe the changes introduced, referencing any related issues.
+
+## Report a Bug or Suggestion
+
+- **Bug Reports:** Help us understand and address issues by submitting detailed bug reports via GitHub issues. Include as many relevant details as possible in the provided template to expedite resolutions.
+- **Suggestions:** Share your ideas, feedback, or stay updated on Giga by joining our [Discord channel]((https://discord.com/invite/NStBwE7kyv)).
+
+## Making Changes and Pull Requests
+
+To contribute code changes:
+
+1. Fork the repository and create a new branch for your contribution 
+>`git checkout -b 'my-contribution'`.
+2. Make your changes on the created branch.
+3. Commit with clear messages describing the updates.
+4. Submit a pull request in the main repository, ensuring the following:
+   - Clear use case or demonstration of bug fix/new feature.
+   - Inclusion of relevant tests (unit, functional, and fuzz tests where applicable).
+   - Adherence to code style guidelines.
+   - No breaking changes to the existing test suite.
+   - Bug fixes accompanied by tests to prevent regression.
+   - Update of relevant comments and documentation reflecting code behavior changes.
+
+## Contributing with an Issue
+
+If you encounter a bug but aren't sure how to fix it or submit a pull request, you can create an issue. Issues serve as avenues for bug reports, feedback, and general discussions within the GigaSpatial GitHub repository.
+
+## Other Ways to Contribute
+
+Beyond code contributions:
+
+- **Feedback and Insights:** Share your expertise and experiences related to cash transfer by contacting us at [giga@unicef.org](mailto:giga@unicef.org).
+- **Documentation:** Contribute to our journey by sharing reports, case studies, articles, blogs, or surveys. Contact us to contribute and learn more via [giga@unicef.org](mailto:giga@unicef.org).
+- **Designs:** If you're passionate about UI/UX, animations, graphics, tutorials, etc., contact us to create visuals for the Giga community via [giga@unicef.org](mailto:giga@unicef.org).
+
+## Connect with Giga Contributors
+Connect with fellow contributors via our Discord channel to engage with the Giga community: [Click](https://discord.com/invite/NStBwE7kyv)