giga-spatial 0.6.3__tar.gz → 0.6.4__tar.gz

{giga_spatial-0.6.3 → giga_spatial-0.6.4}/.env_sample

@@ -12,6 +12,8 @@ export OPENCELLID_ACCESS_TOKEN=""
 export GEOREPO_API_KEY=""
 export GEOREPO_USER_EMAIL=""
 export GIGA_SCHOOL_LOCATION_API_KEY=""
+export GIGA_SCHOOL_PROFILE_API_KEY=""
+export GIGA_SCHOOL_MEASUREMENTS_API_KEY=""
 export ROOT_DATA_DIR=""
 export BRONZE_DIR=""
 export SILVER_DIR=""

{giga_spatial-0.6.3 → giga_spatial-0.6.4}/CHANGELOG.md

@@ -2,6 +2,42 @@
 
 All notable changes to this project will be documented in this file.
 
+## [v0.6.4] - 2025-06-19
+
+### Added
+
+- **GigaSchoolProfileFetcher**  
+  - New class to fetch and process school profile data from the Giga School Profile API  
+  - Supports paginated fetching, filtering by country and school ID  
+  - Includes methods to generate connectivity summary statistics by region, connection type, and source
+
+- **GigaSchoolMeasurementsFetcher**  
+  - New class to fetch and process daily real-time connectivity measurements from the Giga API  
+  - Supports filtering by date range and school  
+  - Includes performance summary generation (download/upload speeds, latency, quality flags)
+
+- **AdminBoundaries.from_geoboundaries**  
+  - New class method to download and process geoBoundaries data by country and admin level  
+  - Automatically handles HDX dataset discovery, downloading, and fallback logic
+
+- **HDXConfig.search_datasets**  
+  - Static method to search HDX datasets without full handler initialization  
+  - Supports query string, sort order, result count, HDX site selection, and custom user agent
+
+### Fixed
+
+- Typo in `MaxarImageDownloader` causing runtime error
+
+### Documentation
+
+- **Improved Configuration Guide** (`docs/user-guide/configuration.md`)  
+  - Added comprehensive table of environment variables with defaults and descriptions  
+  - Synced `.env_sample` and `config.py` with docs  
+  - Example `.env` file and guidance on path overrides using `config.set_path`  
+  - New section on `config.ensure_directories_exist` and troubleshooting tips  
+  - Clearer handling of credentials and security notes  
+  - Improved formatting and structure for clarity
+
 ## [v0.6.3] - 2025-06-16
 
 ### Added

{giga_spatial-0.6.3 → giga_spatial-0.6.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: giga-spatial
-Version: 0.6.3
+Version: 0.6.4
 Summary: A package for spatial data download & processing
 Home-page: https://github.com/unicef/giga-spatial
 Author: Utku Can Ozturk

giga_spatial-0.6.4/docs/changelog.md

@@ -0,0 +1 @@
+--8<-- "CHANGELOG.md"

giga_spatial-0.6.4/docs/user-guide/configuration.md

@@ -0,0 +1,130 @@
+# Configuration
+
+The `gigaspatial` package uses a unified configuration system, managed by the `config.py` file, to handle paths, API keys, and other settings. This guide explains how to configure the package for your environment.
+
+---
+
+## Environment Variables Overview
+
+Configuration is primarily managed via environment variables, which can be set in a `.env` file at the project root. Below is a table of all supported environment variables, their defaults, and descriptions:
+
+| Variable                        | Default         | Description                                      |
+|----------------------------------|-----------------|--------------------------------------------------|
+| ADLS_CONNECTION_STRING           | ""              | Azure Data Lake connection string                 |
+| ADLS_CONTAINER_NAME              | ""              | Azure Data Lake container name                    |
+| GOOGLE_SERVICE_ACCOUNT           | ""              | Google service account credentials                |
+| API_PROFILE_FILE_PATH            | profile.share   | Path to API profile file                          |
+| API_SHARE_NAME                   | ""              | API share name                                    |
+| API_SCHEMA_NAME                  | ""              | API schema name                                   |
+| MAPBOX_ACCESS_TOKEN              | ""              | Mapbox API access token                           |
+| MAXAR_USERNAME                   | ""              | Maxar API username                                |
+| MAXAR_PASSWORD                   | ""              | Maxar API password                                |
+| MAXAR_CONNECTION_STRING          | ""              | Maxar API connection string/key                   |
+| OPENCELLID_ACCESS_TOKEN          | ""              | OpenCellID API access token                       |
+| GEOREPO_API_KEY                  | ""              | UNICEF GeoRepo API key                            |
+| GEOREPO_USER_EMAIL               | ""              | UNICEF GeoRepo user email                         |
+| GIGA_SCHOOL_LOCATION_API_KEY     | ""              | GIGA School Location API key                      |
+| GIGA_SCHOOL_PROFILE_API_KEY      | ""              | GIGA School Profile API key                       |
+| GIGA_SCHOOL_MEASUREMENTS_API_KEY | ""              | GIGA School Measurements API key                  |
+| ROOT_DATA_DIR                    | .               | Root directory for all data tiers                 |
+| BRONZE_DIR                       | bronze          | Directory for raw/bronze tier data                |
+| SILVER_DIR                       | silver          | Directory for processed/silver tier data          |
+| GOLD_DIR                         | gold            | Directory for final/gold tier data                |
+| VIEWS_DIR                        | views           | Directory for views data                          |
+| CACHE_DIR                        | cache           | Directory for cache/temp files                    |
+| ADMIN_BOUNDARIES_DIR             | admin_boundaries| Directory for admin boundary data                 |
+
+> **Tip:** You can copy `.env_sample` to `.env` and fill in your values.
+
+---
+
+## Example `.env` File
+
+```bash
+# Data directories
+BRONZE_DIR=/path/to/your/bronze_tier_data
+SILVER_DIR=/path/to/your/silver_tier_data
+GOLD_DIR=/path/to/your/gold_tier_data
+VIEWS_DIR=/path/to/your/views_data
+CACHE_DIR=/path/to/your/cache
+ADMIN_BOUNDARIES_DIR=/path/to/your/admin_boundaries
+
+# API keys and credentials
+MAPBOX_ACCESS_TOKEN=your_mapbox_token_here
+MAXAR_USERNAME=your_maxar_username_here
+MAXAR_PASSWORD=your_maxar_password_here
+MAXAR_CONNECTION_STRING=your_maxar_key_here
+# ... other keys ...
+```
+
+---
+
+## How Configuration is Loaded
+
+- The `config.py` file uses [pydantic-settings](https://docs.pydantic.dev/latest/concepts/pydantic_settings/) to load environment variables from `.env` (if present) or the system environment.
+- All directory paths are resolved as `Path` objects. If a path is relative, it is resolved relative to the current working directory.
+- Defaults are used if environment variables are not set.
+
+---
+
+## Setting Paths and Keys Programmatically
+
+You can override directory paths in your code using the `set_path` method:
+
+```python
+from gigaspatial.config import config
+
+# Set custom data storage paths
+config.set_path("bronze", "/path/to/your/bronze_tier_data")
+config.set_path("gold", "/path/to/your/gold_tier_data")
+config.set_path("views", "/path/to/your/views_data")
+```
+
+> **Note:** API keys and credentials should be set via environment variables for security.
+
+---
+
+## Ensuring Directories Exist
+
+To ensure all configured directories exist (and optionally create them if missing):
+
+```python
+from gigaspatial.config import config
+
+# Raise error if any directory does not exist
+config.ensure_directories_exist(create=False)
+
+# Or, create missing directories automatically
+config.ensure_directories_exist(create=True)
+```
+
+---
+
+## Verifying the Configuration
+
+You can print the current configuration for debugging:
+
+```python
+from gigaspatial.config import config
+print(config)
+```
+
+---
+
+## Troubleshooting
+
+- **.env File Location:** Ensure `.env` is in your project root.
+- **Absolute Paths:** Use absolute paths for directories to avoid confusion.
+- **Environment Variable Precedence:** Values in `.env` override defaults, but can be overridden by system environment variables.
+- **Missing Directories:** Use `config.ensure_directories_exist(create=True)` to create missing directories.
+- **API Keys:** Double-check that all required API keys are set for the services you use.
+
+---
+
+## Next Steps
+
+Once configuration is set up, proceed to the [Data Handling Guide](data-handling/downloading.md) to start using `gigaspatial`.
+
+---
+
+[Back to User Guide](../index.md)

{giga_spatial-0.6.3 → giga_spatial-0.6.4}/giga_spatial.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: giga-spatial
-Version: 0.6.3
+Version: 0.6.4
 Summary: A package for spatial data download & processing
 Home-page: https://github.com/unicef/giga-spatial
 Author: Utku Can Ozturk

giga_spatial-0.6.4/gigaspatial/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.6.4"

{giga_spatial-0.6.3 → giga_spatial-0.6.4}/gigaspatial/config.py

@@ -32,6 +32,12 @@ class Config(BaseSettings):
     GIGA_SCHOOL_LOCATION_API_KEY: str = Field(
         default="", alias="GIGA_SCHOOL_LOCATION_API_KEY"
     )
+    GIGA_SCHOOL_PROFILE_API_KEY: str = Field(
+        default="", alias="GIGA_SCHOOL_PROFILE_API_KEY"
+    )
+    GIGA_SCHOOL_MEASUREMENTS_API_KEY: str = Field(
+        default="", alias="GIGA_SCHOOL_MEASUREMENTS_API_KEY"
+    )
 
     ROOT_DATA_DIR: Path = Field(
         default=Path("."),

{giga_spatial-0.6.3 → giga_spatial-0.6.4}/gigaspatial/handlers/__init__.py

@@ -37,4 +37,8 @@ from gigaspatial.handlers.unicef_georepo import (
     GeoRepoClient,
     get_country_boundaries_by_iso3,
 )
-from gigaspatial.handlers.giga import GigaSchoolLocationFetcher
+from gigaspatial.handlers.giga import (
+    GigaSchoolLocationFetcher,
+    GigaSchoolProfileFetcher,
+    GigaSchoolMeasurementsFetcher,
+)

{giga_spatial-0.6.3 → giga_spatial-0.6.4}/gigaspatial/handlers/boundaries.py

@@ -4,10 +4,12 @@ import geopandas as gpd
 from pathlib import Path
 from urllib.error import HTTPError
 from shapely.geometry import Polygon, MultiPolygon, shape
+import tempfile
 import pycountry
 
 from gigaspatial.core.io.data_store import DataStore
 from gigaspatial.core.io.readers import read_dataset
+from gigaspatial.handlers.hdx import HDXConfig
 from gigaspatial.config import config
 
 
@@ -61,8 +63,31 @@ class AdminBoundaries(BaseModel):
             "name_en": "name_en",
             "country_code": "iso_3166_1_alpha_3",
         },
+        "geoBoundaries": {
+            "id": "shapeID",
+            "name": "shapeName",
+            "country_code": "shapeGroup",
+        },
     }
 
+    def to_geodataframe(self) -> gpd.GeoDataFrame:
+        """Convert the AdminBoundaries to a GeoDataFrame."""
+        if not self.boundaries:
+            if hasattr(self, "_empty_schema"):
+                columns = self._empty_schema
+            else:
+                columns = ["id", "name", "country_code", "geometry"]
+                if self.level > 0:
+                    columns.append("parent_id")
+
+            return gpd.GeoDataFrame(columns=columns, geometry="geometry", crs=4326)
+
+        return gpd.GeoDataFrame(
+            [boundary.model_dump() for boundary in self.boundaries],
+            geometry="geometry",
+            crs=4326,
+        )
+
     @classmethod
     def get_schema_config(cls) -> Dict[str, Dict[str, str]]:
         """Return field mappings for different data sources"""
@@ -100,6 +125,7 @@ class AdminBoundaries(BaseModel):
             cls.logger.warning(
                 f"Error loading GADM data for {country_code} at admin level {admin_level}: {str(e)}"
             )
+            cls.logger.info("Falling back to empty instance")
             return cls._create_empty_instance(country_code, admin_level, "gadm")
 
     @classmethod
@@ -138,6 +164,7 @@ class AdminBoundaries(BaseModel):
             cls.logger.warning(
                 f"No data found at {path} for admin level {admin_level}: {str(e)}"
             )
+            cls.logger.info("Falling back to empty instance")
             return cls._create_empty_instance(None, admin_level, "internal")
 
     @classmethod
@@ -202,6 +229,69 @@ class AdminBoundaries(BaseModel):
 
         return cls(boundaries=boundaries, level=admin_level)
 
+    @classmethod
+    def from_geoboundaries(cls, country_code, admin_level: int = 0):
+        cls.logger.info(
+            f"Searching for geoBoundaries data for country: {country_code}, admin level: {admin_level}"
+        )
+
+        country_datasets = HDXConfig.search_datasets(
+            query=f'dataseries_name:"geoBoundaries - Subnational Administrative Boundaries" AND groups:"{country_code.lower()}"',
+            rows=1,
+        )
+        if not country_datasets:
+            cls.logger.error(f"No datasets found for country: {country_code}")
+            raise ValueError(
+                "No resources found for the specified country. Please check your search parameters and try again."
+            )
+
+        cls.logger.info(f"Found dataset: {country_datasets[0].get('title', 'Unknown')}")
+
+        resources = [
+            resource
+            for resource in country_datasets[0].get_resources()
+            if (
+                resource.data["name"]
+                == f"geoBoundaries-{country_code.upper()}-ADM{admin_level}.geojson"
+            )
+        ]
+
+        if not resources:
+            cls.logger.error(
+                f"No resources found for {country_code} at admin level {admin_level}"
+            )
+            raise ValueError(
+                "No resources found for the specified criteria. Please check your search parameters and try again."
+            )
+
+        cls.logger.info(f"Found resource: {resources[0].data.get('name', 'Unknown')}")
+
+        try:
+            cls.logger.info("Downloading and processing boundary data...")
+            with tempfile.TemporaryDirectory() as tmpdir:
+                url, local_path = resources[0].download(folder=tmpdir)
+                cls.logger.debug(f"Downloaded file to temporary path: {local_path}")
+                with open(local_path, "rb") as f:
+                    gdf = gpd.read_file(f)
+
+            gdf = cls._map_fields(gdf, "geoBoundaries", admin_level)
+            boundaries = [
+                AdminBoundary(**row_dict) for row_dict in gdf.to_dict("records")
+            ]
+            cls.logger.info(
+                f"Successfully created {len(boundaries)} AdminBoundary objects"
+            )
+            return cls(boundaries=boundaries, level=admin_level)
+
+        except (ValueError, HTTPError, FileNotFoundError) as e:
+            cls.logger.warning(
+                f"Error loading geoBoundaries data for {country_code} at admin level {admin_level}: {str(e)}"
+            )
+            cls.logger.info("Falling back to empty instance")
+            return cls._create_empty_instance(
+                country_code, admin_level, "geoBoundaries"
+            )
+
     @classmethod
     def create(
         cls,
@@ -211,45 +301,126 @@ class AdminBoundaries(BaseModel):
         path: Optional[Union[str, "Path"]] = None,
         **kwargs,
     ) -> "AdminBoundaries":
-        """Factory method to create AdminBoundaries instance from either GADM or data store."""
+        """Factory method to create AdminBoundaries instance from either GADM or data store.
+
+        Args:
+            country_code: ISO country code (2 or 3 letter) or country name
+            admin_level: Administrative level (0=country, 1=state/province, etc.)
+            data_store: Optional data store instance for loading from existing data
+            path: Optional path to data file (used with data_store)
+            **kwargs: Additional arguments passed to the underlying creation methods
+
+        Returns:
+            AdminBoundaries: Configured instance
+
+        Raises:
+            ValueError: If neither country_code nor (data_store, path) are provided,
+                    or if country_code lookup fails
+
+        Example:
+            # From country code
+            boundaries = AdminBoundaries.create(country_code="USA", admin_level=1)
+
+            # From data store
+            boundaries = AdminBoundaries.create(data_store=store, path="data.shp")
+        """
         cls.logger.info(
-            f"Creating AdminBoundaries instance. Country: {country_code}, admin level: {admin_level}, data_store provided: {data_store is not None}, path provided: {path is not None}"
+            f"Creating AdminBoundaries instance. Country: {country_code}, "
+            f"admin level: {admin_level}, data_store provided: {data_store is not None}, "
+            f"path provided: {path is not None}"
         )
-        iso3_code = pycountry.countries.lookup(country_code).alpha_3
+
+        # Validate input parameters
+        if not country_code and not data_store:
+            raise ValueError("Either country_code or data_store must be provided.")
+
+        if data_store and not path and not country_code:
+            raise ValueError(
+                "If data_store is provided, either path or country_code must also be specified."
+            )
+
+        # Handle data store path first
         if data_store is not None:
-            if path is None:
-                if country_code is None:
-                    ValueError(
-                        "If data_store is provided, path or country_code must also be specified."
-                    )
+            iso3_code = None
+            if country_code:
+                try:
+                    iso3_code = pycountry.countries.lookup(country_code).alpha_3
+                except LookupError as e:
+                    raise ValueError(f"Invalid country code '{country_code}': {e}")
+
+            # Generate path if not provided
+            if path is None and iso3_code:
                 path = config.get_admin_path(
                     country_code=iso3_code,
                     admin_level=admin_level,
                 )
+
             return cls.from_data_store(data_store, path, admin_level, **kwargs)
-        elif country_code is not None:
-            from gigaspatial.handlers.unicef_georepo import GeoRepoClient
 
+        # Handle country code path
+        if country_code is not None:
             try:
-                client = GeoRepoClient()
-                if client.check_connection():
-                    cls.logger.info("GeoRepo connection successful.")
-                    return cls.from_georepo(
-                        iso3_code,
-                        admin_level=admin_level,
-                    )
-            except ValueError as e:
+                iso3_code = pycountry.countries.lookup(country_code).alpha_3
+            except LookupError as e:
+                raise ValueError(f"Invalid country code '{country_code}': {e}")
+
+            # Try GeoRepo first
+            if cls._try_georepo(iso3_code, admin_level):
+                return cls.from_georepo(iso3_code, admin_level=admin_level)
+
+            # Fallback to GADM
+            try:
+                cls.logger.info("Attempting to load from GADM.")
+                return cls.from_gadm(iso3_code, admin_level, **kwargs)
+            except Exception as e:
                 cls.logger.warning(
-                    f"GeoRepo initialization failed: {str(e)}. Falling back to GADM."
+                    f"GADM loading failed: {e}. Falling back to geoBoundaries."
                 )
+
+            # Final fallback to geoBoundaries
+            try:
+                return cls.from_geoboundaries(iso3_code, admin_level)
             except Exception as e:
-                cls.logger.warning(f"GeoRepo error: {str(e)}. Falling back to GADM.")
+                cls.logger.error(f"All data sources failed. geoBoundaries error: {e}")
+                raise RuntimeError(
+                    f"Failed to load administrative boundaries for {country_code} "
+                    f"from all available sources (GeoRepo, GADM, geoBoundaries)."
+                ) from e
 
-            return cls.from_gadm(iso3_code, admin_level, **kwargs)
-        else:
-            raise ValueError(
-                "Either country_code or (data_store, path) must be provided."
-            )
+        # This should never be reached due to validation above
+        raise ValueError("Unexpected error: no valid data source could be determined.")
+
+    @classmethod
+    def _try_georepo(cls, iso3_code: str, admin_level: int) -> bool:
+        """Helper method to test GeoRepo availability.
+
+        Args:
+            iso3_code: ISO3 country code
+            admin_level: Administrative level
+
+        Returns:
+            bool: True if GeoRepo is available and working, False otherwise
+        """
+        try:
+            from gigaspatial.handlers.unicef_georepo import GeoRepoClient
+
+            client = GeoRepoClient()
+            if client.check_connection():
+                cls.logger.info("GeoRepo connection successful.")
+                return True
+            else:
+                cls.logger.info("GeoRepo connection failed.")
+                return False
+
+        except ImportError:
+            cls.logger.info("GeoRepo client not available (import failed).")
+            return False
+        except ValueError as e:
+            cls.logger.warning(f"GeoRepo initialization failed: {e}")
+            return False
+        except Exception as e:
+            cls.logger.warning(f"GeoRepo error: {e}")
+            return False
 
     @classmethod
     def _create_empty_instance(
@@ -288,21 +459,3 @@ class AdminBoundaries(BaseModel):
                 field_mapping[v] = k
 
         return gdf.rename(columns=field_mapping)
-
-    def to_geodataframe(self) -> gpd.GeoDataFrame:
-        """Convert the AdminBoundaries to a GeoDataFrame."""
-        if not self.boundaries:
-            if hasattr(self, "_empty_schema"):
-                columns = self._empty_schema
-            else:
-                columns = ["id", "name", "country_code", "geometry"]
-                if self.level > 0:
-                    columns.append("parent_id")
-
-            return gpd.GeoDataFrame(columns=columns, geometry="geometry", crs=4326)
-
-        return gpd.GeoDataFrame(
-            [boundary.model_dump() for boundary in self.boundaries],
-            geometry="geometry",
-            crs=4326,
-        )