giga-spatial 0.6.2__py3-none-any.whl → 0.6.3__py3-none-any.whl

{giga_spatial-0.6.2.dist-info → giga_spatial-0.6.3.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: giga-spatial
-Version: 0.6.2
+Version: 0.6.3
 Summary: A package for spatial data download & processing
 Home-page: https://github.com/unicef/giga-spatial
 Author: Utku Can Ozturk
@@ -51,17 +51,27 @@ Dynamic: summary
 
 # GigaSpatial
 
-# ![GigaSpatial logo](https://raw.githubusercontent.com/unicef/giga-spatial/main/docs/assets/logo.png)
-Giga is a UNICEF-ITU initiative to connect every school to the Internet and every young person to information, opportunity and choice. 
-Giga maps schools' Internet access in real time, creates models for innovative financing, and supports governments contracting for connectivity. 
+## About Giga
 
----
+[Giga](https://giga.global/) is a UNICEF-ITU initiative to connect every school to the Internet and every young person to information, opportunity and choice. 
+Giga maps schools' Internet access in real time, creates models for innovative financing, and supports governments contracting for connectivity. 
 
 ## About GigaSpatial
 
 **GigaSpatial** is a Python package developed as part of the Giga Applied Science Team to handle geospatial data efficiently. It provides tools for downloading, processing, and analyzing geospatial data, enabling users to work with datasets such as OpenStreetMap (OSM), Global Human Settlement Layer (GHSL), Microsoft Global Buildings, Google Open Buildings, and more. The package is designed to support Giga's mission by providing robust geospatial capabilities for mapping and analyzing school connectivity.
 
-### Key Features
+## Installation
+
+See the [installation docs](https://unicef.github.io/giga-spatial/getting-started/installation/) for all details. GigaSpatial requires Python 3.10 or above and depends on the following key packages:
+
+- geopandas
+- pandas
+- shapely
+- rasterio
+
+We recommend using a virtual environment for installation. See the [installation docs](https://unicef.github.io/giga-spatial/getting-started/installation/) for more details.
+
+## Key Features
 - **Data Downloading**: Download geospatial data from various sources including GHSL, Microsoft Global Buildings, Google Open Buildings, OpenCellID, and HDX datasets.
 - **Data Processing**: Process and transform geospatial data, such as GeoTIFF files and vector data, with support for compression and efficient handling.
 - **View Generators**: 
@@ -74,7 +84,7 @@ Giga maps schools' Internet access in real time, creates models for innovative f
   - Centralized configuration via environment variables or `.env` file
   - Easy setup of API keys and paths
 
-### Supported Datasets
+## Supported Datasets
 
 The `gigaspatial` package supports data from the following providers:
 
@@ -84,7 +94,7 @@ The `gigaspatial` package supports data from the following providers:
 
 ---
 
-### View Generators
+## View Generators
 
 The **view generators** in GigaSpatial are designed to enrich the spatial context of school locations and map data into grid or POI locations. This enables users to analyze and visualize geospatial data in meaningful ways.
 

{giga_spatial-0.6.2.dist-info → giga_spatial-0.6.3.dist-info}/RECORD

@@ -1,5 +1,5 @@
-giga_spatial-0.6.2.dist-info/licenses/LICENSE,sha256=hIahDEOTzuHCU5J2nd07LWwkLW7Hko4UFO__ffsvB-8,34523
-gigaspatial/__init__.py,sha256=jFlbxEJFS0G44LE-yXXVSwXACA1J_NyYDk5E20_2zpc,22
+giga_spatial-0.6.3.dist-info/licenses/LICENSE,sha256=hIahDEOTzuHCU5J2nd07LWwkLW7Hko4UFO__ffsvB-8,34523
+gigaspatial/__init__.py,sha256=zYiFHqR7JwbvdK9dvKrh-RTNfUqjHUwC4CTcFAPVYLc,22
 gigaspatial/config.py,sha256=yMf1ofOU0_I6iKDqshiFSYmK6TDIVpPm1AZo4e2okHU,8166
 gigaspatial/core/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
 gigaspatial/core/io/__init__.py,sha256=y4QNWx6US1-adTuAO_NZwLmjzSQj25HNDL5hUGvEHZc,263
@@ -19,13 +19,13 @@ gigaspatial/generators/zonal/geometry.py,sha256=XPcX5lT7X7Z1vn72sN-VKLb2hDP9F_w3
 gigaspatial/generators/zonal/mercator.py,sha256=R_KlaqF4lnc0cRqVfcNVO8i0Re21_6w7pnclVKSohcY,3125
 gigaspatial/grid/__init__.py,sha256=H8SnNAMDafJXJ9bUp2zU0Z3t6s8niqY5rGP5nFhnbLA,45
 gigaspatial/grid/mercator_tiles.py,sha256=Z_3M4sy1tyxywAo2wmBb6niBP3x-IWgwMkmUp8LOSDg,10492
-gigaspatial/handlers/__init__.py,sha256=gh6LIzqr2fASX4fEzQ20v1j_IhnYpym-lcgOkAGHrdE,1430
+gigaspatial/handlers/__init__.py,sha256=pqK3rJtelOAkBaWNhpGy2t-p_zrwO-9BqABLQufTXF0,1449
 gigaspatial/handlers/base.py,sha256=rL94c3wDjsqzLp4na8FfYXW6tNjVGX6v4M-Ce4LrAro,26413
 gigaspatial/handlers/boundaries.py,sha256=hoO-b5MlFYwlCWogApcFyEx6OnxMJG29lqJurNGwOWg,11260
-gigaspatial/handlers/ghsl.py,sha256=YTXuhjSjNMl11nmgd6ZOtVMsJGg2PxDcZn4pu8aaypc,27749
+gigaspatial/handlers/ghsl.py,sha256=GHao8lkmj1C0-QFqNwH9jr0Lqzu6NTj_7ooQdj1h6ok,27760
 gigaspatial/handlers/giga.py,sha256=2aP1EenDAQXn-h-uCyuVxEVZvAFEvrL17_z0MiS8FDs,4867
 gigaspatial/handlers/google_open_buildings.py,sha256=Liqk7qJhDtB4Ia4uhBe44LFcf-XVKBjRfj-pWlE5erY,16594
-gigaspatial/handlers/hdx.py,sha256=8VAEHfO8UgvEx6_Xdx0_vNsPqO18z5Iiq6TjuEbg05k,8789
+gigaspatial/handlers/hdx.py,sha256=DNw-LhxuJU3eNGihQGyPJT0a1PaOCupNHr7BDGal4Zo,18088
 gigaspatial/handlers/mapbox_image.py,sha256=M_nkJ_b1PD8FG1ajVgSycCb0NRTAI_SLpHdzszNetKA,7786
 gigaspatial/handlers/maxar_image.py,sha256=g5YVGV-8JjeG9bGBOp7ZfKani22J4izXX4hnB9A99Jk,10272
 gigaspatial/handlers/microsoft_global_buildings.py,sha256=bQ5WHIv3v0wWrZZUbZkKPRjgdlqIxlK7CV_0zSvdrTw,20292
@@ -33,15 +33,15 @@ gigaspatial/handlers/ookla_speedtest.py,sha256=EcvSAxJZ9GPfzYnT_C85Qgy2ecc9ndf70
 gigaspatial/handlers/opencellid.py,sha256=KuJqd-5-RO5ZzyDaBSrTgCK2ib5N_m3RUcPlX5heWwI,10683
 gigaspatial/handlers/osm.py,sha256=sLNMkOVh1v50jrWw7Z0-HILY5QTQjgKCHCeAfXj5jA8,14084
 gigaspatial/handlers/overture.py,sha256=lKeNw00v5Qia7LdWORuYihnlKEqxE9m38tdeRrvag9k,4218
-gigaspatial/handlers/rwi.py,sha256=g37bUd4Y02r8m53ZkjASCscg0mejMH5820HAh8PQZqQ,6032
+gigaspatial/handlers/rwi.py,sha256=GDpQH9K96QZD3yezJOBiy5yZvYmrj4xbjUNSjYfNAh0,4875
 gigaspatial/handlers/unicef_georepo.py,sha256=ODYNvkU_UKgOHXT--0MqmJ4Uk6U1_mp9xgehbTzKpX8,31924
 gigaspatial/handlers/worldpop.py,sha256=oJ39NGajXi0rn829ZoFiaeG4_wavyPvljdActpxs12I,9850
 gigaspatial/processing/__init__.py,sha256=QDVL-QbLCrIb19lrajP7LrHNdGdnsLeGcvAs_jQpdRM,183
 gigaspatial/processing/geo.py,sha256=D-S3IlhQwLIxrCcxy6NhNmKLrOIjoRHfK_eZJGKpe2U,36947
 gigaspatial/processing/sat_images.py,sha256=YUbH5MFNzl6NX49Obk14WaFcr1s3SyGJIOk-kRpbBNg,1429
-gigaspatial/processing/tif_processor.py,sha256=4qCNx6ODtVsYm6lbSToIdUCrduIrPsWFBAABTQ2Vs24,17259
+gigaspatial/processing/tif_processor.py,sha256=zqcP_ioo9KHNJ6H0uba4UghW4MToTRwq1iE-nZbb8zA,21101
 gigaspatial/processing/utils.py,sha256=HC85vGKQakxlkoQAkZmeAXWHsenAwTIRn7jPKUA7x20,1500
-giga_spatial-0.6.2.dist-info/METADATA,sha256=PXfUtURxAxnej1IDhF3MGbOOdC8MYgsGtRHHTiPjQSQ,7104
-giga_spatial-0.6.2.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
-giga_spatial-0.6.2.dist-info/top_level.txt,sha256=LZsccgw6H4zXT7m6Y4XChm-Y5LjHAwZ2hkGN_B3ExmI,12
-giga_spatial-0.6.2.dist-info/RECORD,,
+giga_spatial-0.6.3.dist-info/METADATA,sha256=Aw5adPdTcA3AuJBmZgAG4rJQYW4dJqw2GT90mYE7cgU,7467
+giga_spatial-0.6.3.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
+giga_spatial-0.6.3.dist-info/top_level.txt,sha256=LZsccgw6H4zXT7m6Y4XChm-Y5LjHAwZ2hkGN_B3ExmI,12
+giga_spatial-0.6.3.dist-info/RECORD,,

gigaspatial/__init__.py

@@ -1 +1 @@
-__version__ = "0.6.2"
+__version__ = "0.6.3"

gigaspatial/handlers/__init__.py

@@ -31,8 +31,8 @@ from gigaspatial.handlers.opencellid import (
     OpenCellIDDownloader,
     OpenCellIDReader,
 )
-from gigaspatial.handlers.hdx import HDXConfig, HDXDownloader, HDXReader
-from gigaspatial.handlers.rwi import RWIConfig, RelativeWealthIndexDownloader
+from gigaspatial.handlers.hdx import HDXConfig, HDXDownloader, HDXReader, HDXHandler
+from gigaspatial.handlers.rwi import RWIConfig, RWIDownloader, RWIReader, RWIHandler
 from gigaspatial.handlers.unicef_georepo import (
     GeoRepoClient,
     get_country_boundaries_by_iso3,

gigaspatial/handlers/ghsl.py

@@ -74,8 +74,6 @@ class GHSLDataConfig(BaseHandlerConfig):
 
     def __post_init__(self):
         super().__post_init__()
-        self.TILES_URL = self.TILES_URL.format(self.coord_system.value)
-        self._load_tiles()
 
     def _load_tiles(self):
         """Load GHSL tiles from tiles shapefile."""
@@ -158,6 +156,9 @@ class GHSLDataConfig(BaseHandlerConfig):
                 )
                 self.coord_system = CoordSystem.Mollweide
 
+        self.TILES_URL = self.TILES_URL.format(self.coord_system.value)
+        self._load_tiles()
+
         return self
 
     @property
@@ -176,7 +177,7 @@ class GHSLDataConfig(BaseHandlerConfig):
         self, points: Iterable[Union[Point, tuple]], **kwargs
     ) -> List[dict]:
         """
-        Return intersecting tiles for a list of points.
+        Return intersecting tiles f or a list of points.
         """
         return self._get_relevant_tiles(points)
 
@@ -240,8 +241,8 @@ class GHSLDataConfig(BaseHandlerConfig):
             ValueError: If the input `source` is not one of the supported types.
         """
         if isinstance(source, gpd.GeoDataFrame):
-            if source.crs != "EPSG:4326":
-                source = source.to_crs("EPSG:4326")
+            # if source.crs != "EPSG:4326":
+            #    source = source.to_crs("EPSG:4326")
             search_geom = source.geometry.unary_union
         elif isinstance(
             source,
@@ -282,7 +283,7 @@ class GHSLDataConfig(BaseHandlerConfig):
             else ("3ss" if self.resolution == 100 else "30ss")
         )
         product_folder = f"{self.product}_GLOBE_{self.release}"
-        product_name = f"{self.product}_E{self.year}_GLOBE_{self.release}_{self.coord_system}_{resolution_str}"
+        product_name = f"{self.product}_E{self.year}_GLOBE_{self.release}_{self.coord_system.value}_{resolution_str}"
         product_version = 2 if self.product == "GHS_SMOD" else 1
 
         return {

gigaspatial/handlers/hdx.py

@@ -1,88 +1,77 @@
 import os
 import logging
+from tqdm import tqdm
 from pathlib import Path
-from typing import List, Optional, Union, Dict, Any
+from typing import List, Optional, Tuple, Union, Dict, Any, Iterable
 import tempfile
+import functools
+import multiprocessing
 
 import pandas as pd
 import geopandas as gpd
-from pydantic import BaseModel, Field
+from pydantic import Field, ConfigDict
+from pydantic.dataclasses import dataclass
+from shapely.geometry.base import BaseGeometry
+from shapely.geometry import Point
+import pycountry
 
 from hdx.api.configuration import Configuration
 from hdx.data.dataset import Dataset
 from hdx.data.resource import Resource
 
 from gigaspatial.core.io.data_store import DataStore
-from gigaspatial.core.io.local_data_store import LocalDataStore
 from gigaspatial.core.io.readers import read_dataset
 from gigaspatial.config import config as global_config
+from gigaspatial.handlers.base import (
+    BaseHandlerConfig,
+    BaseHandlerDownloader,
+    BaseHandlerReader,
+    BaseHandler,
+)
 
 
-class HDXConfig(BaseModel):
+@dataclass(config=ConfigDict(arbitrary_types_allowed=True))
+class HDXConfig(BaseHandlerConfig):
     """Configuration for HDX data access"""
 
     # User configuration
-    dataset_name: str = Field(..., description="Name of the HDX dataset to download")
+    dataset_name: str = Field(
+        default=..., description="Name of the HDX dataset to download"
+    )
+
+    # Optional configuration with defaults
     base_path: Path = Field(default=global_config.get_path("hdx", "bronze"))
     user_agent: str = Field(
         default="gigaspatial", description="User agent for HDX API requests"
     )
     hdx_site: str = Field(default="prod", description="HDX site to use (prod or test)")
-    resource_filter: Optional[Dict[str, Any]] = Field(
-        default=None, description="Filter to apply to resources"
-    )
-
-    @property
-    def output_dir_path(self) -> Path:
-        """Path to save the downloaded HDX dataset"""
-        return self.base_path / self.dataset_name
-
-    def __repr__(self) -> str:
-        return (
-            f"HDXConfig(\n"
-            f"  dataset_name='{self.dataset_name}'\n"
-            f"  base_path='{self.base_path}'\n"
-            f"  hdx_site='{self.hdx_site}'\n"
-            f"  user_agent='{self.user_agent}'\n"
-            f")"
-        )
-
 
-class HDXDownloader:
-    """Downloader for HDX datasets"""
+    # Internal state
+    _hdx_configured: bool = Field(default=False, init=False)
+    dataset: Optional[Dataset] = Field(default=None, init=False)
 
-    def __init__(
-        self,
-        config: Union[HDXConfig, dict],
-        data_store: Optional[DataStore] = None,
-        logger: Optional[logging.Logger] = None,
-    ):
-        if isinstance(config, dict):
-            self.config = HDXConfig(**config)
-        else:
-            self.config = config
-
-        self.data_store = data_store or LocalDataStore()
-        self.logger = logger or global_config.get_logger(self.__class__.__name__)
+    def __post_init__(self):
+        super().__post_init__()
         try:
             Configuration.read()
             self._hdx_configured = True
-        except:
+        except Exception:
             self._hdx_configured = False
+        self.configure_hdx()
+        self.dataset = self.fetch_dataset()
 
-    @classmethod
-    def from_dataset_name(cls, dataset_name: str, **kwargs):
-        """Create a downloader for a specific HDX dataset"""
-        config = HDXConfig(dataset_name=dataset_name, **kwargs)
-        return cls(config=config)
+    @property
+    def output_dir_path(self) -> Path:
+        """Path to save the downloaded HDX dataset"""
+        return self.base_path / self.dataset_name
 
-    def _configure_hdx(self):
+    def configure_hdx(self):
         """Configure HDX API if not already configured"""
         if not self._hdx_configured:
             try:
                 Configuration.create(
-                    hdx_site=self.config.hdx_site,
-                    user_agent=self.config.user_agent,
+                    hdx_site=self.hdx_site,
+                    user_agent=self.user_agent,
                     hdx_read_only=True,
                 )
                 self._hdx_configured = True
@@ -90,40 +79,104 @@ class HDXDownloader:
                 self.logger.error(f"Error configuring HDX API: {str(e)}")
                 raise
 
-    def get_dataset(self) -> Dataset:
+    def fetch_dataset(self) -> Dataset:
         """Get the HDX dataset"""
-        self._configure_hdx()
-
         try:
-            self.logger.info(f"Fetching HDX dataset: {self.config.dataset_name}")
-            dataset = Dataset.read_from_hdx(self.config.dataset_name)
+            self.logger.info(f"Fetching HDX dataset: {self.dataset_name}")
+            dataset = Dataset.read_from_hdx(self.dataset_name)
             if not dataset:
-                raise ValueError(
-                    f"Dataset '{self.config.dataset_name}' not found on HDX"
-                )
+                raise ValueError(f"Dataset '{self.dataset_name}' not found on HDX")
             return dataset
         except Exception as e:
             self.logger.error(f"Error fetching HDX dataset: {str(e)}")
             raise
 
+    def _match_pattern(self, value: str, pattern: str) -> bool:
+        """Check if a value matches a pattern"""
+        if isinstance(pattern, str):
+            return pattern.lower() in value.lower()
+        return value == pattern
+
+    def _get_patterns_for_value(self, value: Any) -> List[str]:
+        """Generate patterns for a given value or list of values"""
+        if isinstance(value, list):
+            patterns = []
+            for v in value:
+                patterns.extend(self._get_patterns_for_value(v))
+            return patterns
+
+        if not isinstance(value, str):
+            return [value]
+
+        patterns = []
+        value = value.lower()
+
+        # Add exact match
+        patterns.append(value)
+
+        # Add common variations
+        patterns.extend(
+            [
+                f"/{value}_",  # URL path with prefix
+                f"/{value}.",  # URL path with extension
+                f"_{value}_",  # Filename with value in middle
+                f"_{value}.",  # Filename with value at end
+            ]
+        )
+
+        # If value contains spaces, generate additional patterns
+        if " " in value:
+            # Generate patterns for space-less version
+            no_space = value.replace(" ", "")
+            patterns.extend(self._get_patterns_for_value(no_space))
+
+            # Generate patterns for hyphenated version
+            hyphenated = value.replace(" ", "-")
+            patterns.extend(self._get_patterns_for_value(hyphenated))
+
+        return patterns
+
     def get_dataset_resources(
-        self, dataset: Optional[Dataset] = None
+        self, filter: Optional[Dict[str, Any]] = None, exact_match: bool = False
     ) -> List[Resource]:
-        """Get resources from the HDX dataset"""
-        dataset = dataset or self.get_dataset()
+        """Get resources from the HDX dataset
 
+        Args:
+            filter: Dictionary of key-value pairs to filter resources
+            exact_match: If True, perform exact matching. If False, use pattern matching
+        """
         try:
-            resources = dataset.get_resources()
+            resources = self.dataset.get_resources()
 
             # Apply resource filter if specified
-            if self.config.resource_filter:
+            if filter:
                 filtered_resources = []
                 for res in resources:
                     match = True
-                    for key, value in self.config.resource_filter.items():
-                        if key in res.data and res.data[key] != value:
+                    for key, value in filter.items():
+                        if key not in res.data:
                             match = False
                             break
+
+                        if exact_match:
+                            # For exact matching, check if value matches or is in list of values
+                            if isinstance(value, list):
+                                if res.data[key] not in value:
+                                    match = False
+                                    break
+                            elif res.data[key] != value:
+                                match = False
+                                break
+                        else:
+                            # For pattern matching, generate patterns for value(s)
+                            patterns = self._get_patterns_for_value(value)
+                            if not any(
+                                self._match_pattern(str(res.data[key]), pattern)
+                                for pattern in patterns
+                            ):
+                                match = False
+                                break
+
                     if match:
                         filtered_resources.append(res)
                 resources = filtered_resources
@@ -133,109 +186,325 @@ class HDXDownloader:
             self.logger.error(f"Error getting dataset resources: {str(e)}")
             raise
 
-    def download_dataset(self) -> List[str]:
-        """Download and save all resources from the HDX dataset into the data_store."""
-        try:
-            dataset = self.get_dataset()
-            resources = self.get_dataset_resources(dataset)
-
-            if not resources:
-                self.logger.warning(
-                    f"No resources found for dataset '{self.config.dataset_name}'"
-                )
-                return []
-
-            self.logger.info(
-                f"Found {len(resources)} resource(s) for dataset '{self.config.dataset_name}'"
-            )
+    def get_relevant_data_units(
+        self, source: Union[str, Dict], **kwargs
+    ) -> List[Resource]:
+        """Get relevant data units based on the source type
+
+        Args:
+            source: Either a country name/code (str) or a filter dictionary
+            **kwargs: Additional keyword arguments passed to the specific method
+
+        Returns:
+            List of matching resources
+        """
+        if isinstance(source, str):
+            # If source is a string, assume it's a country and use country-based filtering
+            return self.get_relevant_data_units_by_country(source, **kwargs)
+        elif isinstance(source, dict):
+            # If source is a dict, use it directly as a filter
+            return self.get_dataset_resources(filter=source, **kwargs)
+        else:
+            raise ValueError(f"Unsupported source type: {type(source)}")
 
-            downloaded_paths = []
-            for res in resources:
-                try:
-                    resource_name = res.get("name", "Unknown")
-                    self.logger.info(f"Downloading resource: {resource_name}")
-
-                    # Download to a temporary directory
-                    with tempfile.TemporaryDirectory() as tmpdir:
-                        url, local_path = res.download(folder=tmpdir)
-                        # Read the file and write to the DataStore
-                        with open(local_path, "rb") as f:
-                            data = f.read()
-                        # Compose the target path in the DataStore
-                        target_path = str(
-                            self.config.output_dir_path / Path(local_path).name
-                        )
-                        self.data_store.write_file(target_path, data)
-                        downloaded_paths.append(target_path)
-
-                    self.logger.info(
-                        f"Downloaded resource: {resource_name} to {target_path}"
-                    )
-                except Exception as e:
-                    self.logger.error(
-                        f"Error downloading resource {res.get('name', 'Unknown')}: {str(e)}"
-                    )
-
-            return downloaded_paths
+    def get_relevant_data_units_by_geometry(
+        self, geometry: Union[BaseGeometry, gpd.GeoDataFrame], **kwargs
+    ) -> List[Resource]:
+        raise NotImplementedError(
+            "HDX does not support geometry-based filtering. "
+            "Please use country-based filtering or direct resource filtering instead."
+        )
 
-        except Exception as e:
-            self.logger.error(f"Error downloading dataset: {str(e)}")
-            raise
+    def get_relevant_data_units_by_points(
+        self, points: List[Union[Point, tuple]], **kwargs
+    ) -> List[Resource]:
+        raise NotImplementedError(
+            "HDX does not support point-based filtering. "
+            "Please use country-based filtering or direct resource filtering instead."
+        )
 
+    def get_relevant_data_units_by_country(
+        self,
+        country: str,
+        key: str = "url",
+        **kwargs,
+    ) -> Any:
+        """Get relevant data units for a country
+
+        Args:
+            country: Country name or code
+            key: The key to filter on in the resource data
+            patterns: List of patterns to match against the resource data
+            **kwargs: Additional keyword arguments
+        """
+        country = pycountry.countries.lookup(country)
+        values = [country.alpha_3, country.alpha_2, country.name]
+        return self.get_dataset_resources(
+            filter={key: values},
+        )
 
-class HDXReader:
-    """Reader for HDX datasets"""
+    def get_data_unit_path(self, unit: str, **kwargs) -> str:
+        """Get the path for a data unit"""
+        try:
+            filename = unit.data["name"]
+        except:
+            filename = unit.get("download_url").split("/")[-1]
 
-    def __init__(
-        self,
-        dataset_name: str,
-        data_store: Optional[DataStore] = None,
-        base_path: Optional[Path] = None,
-    ):
-        self.dataset_name = dataset_name
-        self.data_store = data_store or LocalDataStore()
-        self.base_path = base_path or global_config.get_path("hdx", "bronze")
-        self.dataset_path = self.base_path / self.dataset_name
+        return self.output_dir_path / filename
 
     def list_resources(self) -> List[str]:
         """List all resources in the dataset directory using the data_store."""
+        dataset_folder = str(self.output_dir_path)
         # Check if the dataset directory exists in the data_store
         if not (
-            self.data_store.is_dir(str(self.dataset_path))
-            or self.data_store.file_exists(str(self.dataset_path))
+            self.data_store.is_dir(dataset_folder)
+            or self.data_store.file_exists(dataset_folder)
         ):
             raise FileNotFoundError(
-                f"HDX dataset '{self.dataset_name}' not found at {self.dataset_path}. "
+                f"HDX dataset not found at {dataset_folder}. "
                 "Download the data first using HDXDownloader."
             )
-        # List files using the data_store
-        return self.data_store.list_files(str(self.dataset_path))
+        return self.data_store.list_files(dataset_folder)
+
+    def __repr__(self) -> str:
+        return (
+            f"HDXConfig(\n"
+            f"  dataset_name='{self.dataset_name}'\n"
+            f"  base_path='{self.base_path}'\n"
+            f"  hdx_site='{self.hdx_site}'\n"
+            f"  user_agent='{self.user_agent}'\n"
+            f")"
+        )
 
-    def read_resource(
-        self, resource_file: str
-    ) -> Union[pd.DataFrame, gpd.GeoDataFrame]:
-        """Read a specific resource file from the dataset using the data_store."""
-        file_path = str(self.dataset_path / resource_file)
 
-        if not self.data_store.file_exists(file_path):
-            raise FileNotFoundError(
-                f"Resource file {resource_file} not found in dataset {self.dataset_name}"
-            )
+class HDXDownloader(BaseHandlerDownloader):
+    """Downloader for HDX datasets"""
 
+    def __init__(
+        self,
+        config: Union[HDXConfig, dict],
+        data_store: Optional[DataStore] = None,
+        logger: Optional[logging.Logger] = None,
+    ):
+        config = config if isinstance(config, HDXConfig) else HDXConfig(**config)
+        super().__init__(config=config, data_store=data_store, logger=logger)
+
+    def download_data_unit(self, resource: str, **kwargs) -> str:
+        """Download a single resource"""
         try:
-            return read_dataset(self.data_store, file_path)
+            resource_name = resource.get("name", "Unknown")
+            self.logger.info(f"Downloading resource: {resource_name}")
+
+            with tempfile.TemporaryDirectory() as tmpdir:
+                url, local_path = resource.download(folder=tmpdir)
+                with open(local_path, "rb") as f:
+                    data = f.read()
+                # Compose the target path in the DataStore
+                target_path = str(self.config.get_data_unit_path(resource))
+                self.data_store.write_file(target_path, data)
+                self.logger.info(
+                    f"Downloaded resource: {resource_name} to {target_path}"
+                )
+                return target_path
         except Exception as e:
-            raise ValueError(f"Could not read file {file_path}: {str(e)}")
+            self.logger.error(f"Error downloading resource {resource_name}: {str(e)}")
+            return None
+
+    def download_data_units(self, resources: List[Resource], **kwargs) -> List[str]:
+        """Download multiple resources sequentially
+
+        Args:
+            resources: List of HDX Resource objects
+            **kwargs: Additional keyword arguments
+
+        Returns:
+            List of paths to downloaded files
+        """
+        if len(resources) == 0:
+            self.logger.warning("There is no resource to download")
+            return []
 
-    def read_all_resources(self) -> Dict[str, Union[pd.DataFrame, gpd.GeoDataFrame]]:
-        """Read all resources in the dataset directory using the data_store."""
-        resources = self.list_resources()
-        result = {}
+        downloaded_paths = []
+        for resource in tqdm(resources, desc="Downloading resources"):
+            path = self.download_data_unit(resource)
+            if path:
+                downloaded_paths.append(path)
 
-        for resource in resources:
+        return downloaded_paths
+
+    def download(self, source: Union[Dict, str], **kwargs) -> List[str]:
+        """Download data for a source"""
+        resources = self.config.get_relevant_data_units(source, **kwargs)
+        return self.download_data_units(resources)
+
+
+class HDXReader(BaseHandlerReader):
+    """Reader for HDX datasets"""
+
+    def __init__(
+        self,
+        config: Optional[HDXConfig] = None,
+        data_store: Optional[DataStore] = None,
+        logger: Optional[logging.Logger] = None,
+    ):
+        config = config if isinstance(config, HDXConfig) else HDXConfig(**config)
+        super().__init__(config=config, data_store=data_store, logger=logger)
+
+    def resolve_source_paths(
+        self,
+        source: Union[
+            str,  # country code
+            Dict,  # filter
+            Path,  # path
+            str,  # path
+            List[Union[str, Path]],
+        ],
+        **kwargs,
+    ) -> List[Union[str, Path]]:
+        if isinstance(source, (str, Path)):
+            # Could be a country code or a path
+            if self.data_store.file_exists(str(source)) or str(source).endswith(
+                (".csv", ".tif", ".json", ".parquet", ".gz", ".geojson", ".zip")
+            ):
+                source_data_paths = self.resolve_by_paths(source)
+            else:
+                source_data_paths = self.resolve_by_country(source, **kwargs)
+        elif isinstance(source, Dict):
+            resources = self.config.get_relevant_data_units(source=source, **kwargs)
+            source_data_paths = self.config.get_data_unit_paths(resources, **kwargs)
+        elif isinstance(source, Iterable) and all(
+            isinstance(p, (str, Path)) for p in source
+        ):
+            source_data_paths = self.resolve_by_paths(source)
+        else:
+            raise NotImplementedError(f"Unsupported source type: {type(source)}")
+
+        self.logger.info(f"Resolved {len(source_data_paths)} paths!")
+        return source_data_paths
+
+    def load_from_paths(
+        self, source_data_path: List[Union[str, Path]], **kwargs
+    ) -> Any:
+        """Load data from paths"""
+        if len(source_data_path)==1:
+            return read_dataset(self.data_store, source_data_path[0])
+        
+        all_data = {}
+        for file_path in source_data_path:
             try:
-                result[resource] = self.read_resource(resource)
+                all_data[file_path] = read_dataset(self.data_store, file_path)
             except Exception as e:
-                logging.warning(f"Could not read resource {resource}: {str(e)}")
+                raise ValueError(f"Could not read file {file_path}: {str(e)}")
+        return all_data
+
+    def load_all_resources(self):
+        resources = self.config.list_resources()
+        return self.load_from_paths(resources)
+
+    # def read_resource(
+    #     self, resource_file: str
+    # ) -> Union[pd.DataFrame, gpd.GeoDataFrame]:
+    #     """Read a specific resource file from the dataset using the data_store."""
+    #     if not self.dataset_path:
+    #         raise ValueError("No dataset path configured")
+
+    #     file_path = str(self.dataset_path / resource_file)
+
+    #     if not self.data_store.file_exists(file_path):
+    #         raise FileNotFoundError(
+    #             f"Resource file {resource_file} not found in dataset"
+    #         )
+
+    #     try:
+    #         return read_dataset(self.data_store, file_path)
+    #     except Exception as e:
+    #         raise ValueError(f"Could not read file {file_path}: {str(e)}")
+
+    # def read_all_resources(self) -> Dict[str, Union[pd.DataFrame, gpd.GeoDataFrame]]:
+    #     """Read all resources in the dataset directory using the data_store."""
+    #     resources = self.list_resources()
+    #     result = {}
+
+    #     for resource in resources:
+    #         try:
+    #             result[resource] = self.read_resource(resource)
+    #         except Exception as e:
+    #             self.logger.warning(f"Could not read resource {resource}: {str(e)}")
+
+    #     return result
+
+    # def load_from_paths(
+    #     self, source_data_path: List[Union[str, Path]], **kwargs
+    # ) -> Union[
+    #     pd.DataFrame, gpd.GeoDataFrame, Dict[str, Union[pd.DataFrame, gpd.GeoDataFrame]]
+    # ]:
+    #     """Load data from paths"""
+    #     if len(source_data_path) == 1:
+    #         return self.read_resource(str(source_data_path[0]))
+    #     else:
+    #         return self.read_all_resources()
+
+
+class HDXHandler(BaseHandler):
+    """Handler for HDX datasets"""
 
-        return result
+    def __init__(
+        self,
+        dataset_name: str,
+        config: Optional[HDXConfig] = None,
+        downloader: Optional[HDXDownloader] = None,
+        reader: Optional[HDXReader] = None,
+        data_store: Optional[DataStore] = None,
+        logger: Optional[logging.Logger] = None,
+        **kwargs,
+    ):
+        self._dataset_name = dataset_name
+        super().__init__(
+            config=config,
+            downloader=downloader,
+            reader=reader,
+            data_store=data_store,
+            logger=logger,
+            **kwargs,
+        )
+
+    def create_config(
+        self, data_store: DataStore, logger: logging.Logger, **kwargs
+    ) -> HDXConfig:
+        """Create and return a HDXConfig instance"""
+        return HDXConfig(
+            dataset_name=self._dataset_name,
+            data_store=data_store,
+            logger=logger,
+            **kwargs,
+        )
+
+    def create_downloader(
+        self,
+        config: HDXConfig,
+        data_store: DataStore,
+        logger: logging.Logger,
+        **kwargs,
+    ) -> HDXDownloader:
+        """Create and return a HDXDownloader instance"""
+        return HDXDownloader(
+            config=config,
+            data_store=data_store,
+            logger=logger,
+            **kwargs,
+        )
+
+    def create_reader(
+        self,
+        config: HDXConfig,
+        data_store: DataStore,
+        logger: logging.Logger,
+        **kwargs,
+    ) -> HDXReader:
+        """Create and return a HDXReader instance"""
+        return HDXReader(
+            config=config,
+            data_store=data_store,
+            logger=logger,
+            **kwargs,
+        )

gigaspatial/handlers/rwi.py

@@ -1,15 +1,17 @@
 import logging
-from pathlib import Path
-from typing import List, Optional, Union, Dict, Any, Literal
-import pycountry
-import tempfile
+from typing import List, Optional, Union, Literal
+from pydantic.dataclasses import dataclass
+from datetime import datetime
 
-from pydantic import Field, field_validator
+from hdx.data.resource import Resource
+
+from pydantic import Field, ConfigDict
 
 from gigaspatial.core.io.data_store import DataStore
-from gigaspatial.handlers.hdx import HDXConfig, HDXDownloader
+from gigaspatial.handlers.hdx import HDXConfig, HDXDownloader, HDXReader, HDXHandler
 
 
+@dataclass(config=ConfigDict(arbitrary_types_allowed=True))
 class RWIConfig(HDXConfig):
     """Configuration for Relative Wealth Index data access"""
 
@@ -22,16 +24,49 @@ class RWIConfig(HDXConfig):
     country: Optional[str] = Field(
         default=None, description="Country ISO code to filter data for"
     )
+    latest_only: bool = Field(
+        default=True,
+        description="If True, only get the latest resource for each country",
+    )
+
+    def __post_init__(self):
+        super().__post_init__()
+
+    def get_relevant_data_units_by_country(
+        self, country: str, **kwargs
+    ) -> List[Resource]:
+        """Get relevant data units for a country, optionally filtering for latest version"""
+        resources = super().get_relevant_data_units_by_country(
+            country=country, key="url"
+        )
+
+        if self.latest_only and len(resources) > 1:
+            # Find the resource with the latest creation date
+            latest_resource = None
+            latest_date = None
+
+            for resource in resources:
+                created = resource.get("created")
+                if created:
+                    try:
+                        created_dt = datetime.fromisoformat(
+                            created.replace("Z", "+00:00")
+                        )
+                        if latest_date is None or created_dt > latest_date:
+                            latest_date = created_dt
+                            latest_resource = resource
+                    except ValueError:
+                        self.logger.warning(
+                            f"Could not parse creation date for resource: {created}"
+                        )
+
+            if latest_resource:
+                resources = [latest_resource]
 
-    @field_validator("country")
-    def validate_country(cls, value: str) -> str:
-        try:
-            return pycountry.countries.lookup(value).alpha_3
-        except LookupError:
-            raise ValueError(f"Invalid country code provided: {value}")
+        return resources
 
 
-class RelativeWealthIndexDownloader(HDXDownloader):
+class RWIDownloader(HDXDownloader):
     """Specialized downloader for the Relative Wealth Index dataset from HDX"""
 
     def __init__(
@@ -40,118 +75,81 @@ class RelativeWealthIndexDownloader(HDXDownloader):
         data_store: Optional[DataStore] = None,
         logger: Optional[logging.Logger] = None,
     ):
-        if config is None:
-            config = RWIConfig()
-        elif isinstance(config, dict):
-            config = RWIConfig(**config)
-
+        config = config if isinstance(config, RWIConfig) else RWIConfig(**config)
         super().__init__(config=config, data_store=data_store, logger=logger)
 
-    @classmethod
-    def from_config(
-        cls,
-        country: Optional[str] = None,
-        **kwargs,
-    ):
-        """Create a downloader with RWI-specific configurations"""
-        config = RWIConfig(country=country, **kwargs)
-        return cls(config=config)
-
-    def download_dataset(self) -> List[str]:
-        """Download RWI dataset, optionally filtering for a specific country.
-
-        If country is specified, attempts to find and download only the resources
-        relevant to that country. Otherwise, downloads all RWI resources.
-
-        Returns:
-            List of paths to the downloaded files
-        """
-        # If no country specified, download all resources
-        if self.config.country is None:
-            return super().download_dataset()
-
-        # Get all resources from the dataset
-        try:
-            resources = self.get_dataset_resources()
-            if not resources:
-                self.logger.warning(f"No resources found for RWI dataset")
-                return []
-
-            # Prepare country identifiers for matching
-            country_code = self.config.country.lower()
-            country_name = pycountry.countries.lookup(self.config.country).name.lower()
-            country_alpha2 = pycountry.countries.lookup(
-                self.config.country
-            ).alpha_2.lower()
-
-            # Try different matching patterns
-            country_patterns = [
-                f"/{country_code}_",  # URL path with ISO3 prefix
-                f"/{country_code}.",  # URL path with ISO3 followed by extension
-                f"_{country_code}_",  # Filename with ISO3 in middle
-                f"_{country_code}.",  # Filename with ISO3 at end
-                f"/{country_name.replace(' ', '')}_",  # URL with no spaces
-                f"/{country_name.replace(' ', '-')}_",  # URL with hyphens
-                f"/{country_alpha2}_",  # URL with ISO2 code
-                country_name,  # Country name anywhere in URL
-            ]
-
-            # Find matching resources
-            matching_resources = []
-            for resource in resources:
-                # Get the URL safely
-                resource_url = resource.get("url", "")
-                if not resource_url:
-                    continue
-
-                resource_url = resource_url.lower()
-
-                # Check for matches with our patterns
-                if any(pattern in resource_url for pattern in country_patterns):
-                    matching_resources.append(resource)
-
-            if not matching_resources:
-                self.logger.warning(
-                    f"No resources matching country '{self.config.country}' were found. "
-                    f"Consider downloading the full dataset with country=None and filtering afterwards."
-                )
-                return []
-
-            # Download the matching resources
-            downloaded_paths = []
-            for res in matching_resources:
-                try:
-                    resource_name = res.get("name", "Unknown")
-                    self.logger.info(f"Downloading resource: {resource_name}")
-
-                    # Download to a temporary directory
-                    with tempfile.TemporaryDirectory() as tmpdir:
-                        url, local_path = res.download(folder=tmpdir)
-                        # Read the file and write to the DataStore
-                        with open(local_path, "rb") as f:
-                            data = f.read()
-                        # Compose the target path in the DataStore
-                        target_path = str(
-                            self.config.output_dir_path / Path(local_path).name
-                        )
-                        self.data_store.write_file(target_path, data)
-                        downloaded_paths.append(target_path)
 
-                    self.logger.info(
-                        f"Downloaded resource: {resource_name} to {target_path}"
-                    )
+class RWIReader(HDXReader):
+    """Specialized reader for the Relative Wealth Index dataset from HDX"""
 
-                except Exception as e:
-                    resource_name = res.get("name", "Unknown")
-                    self.logger.error(
-                        f"Error downloading resource {resource_name}: {str(e)}"
-                    )
+    def __init__(
+        self,
+        config: Union[RWIConfig, dict] = None,
+        data_store: Optional[DataStore] = None,
+        logger: Optional[logging.Logger] = None,
+    ):
+        config = config if isinstance(config, RWIConfig) else RWIConfig(**config)
+        super().__init__(config=config, data_store=data_store, logger=logger)
 
-            return downloaded_paths
 
-        except Exception as e:
-            self.logger.error(f"Error during country-filtered download: {str(e)}")
+class RWIHandler(HDXHandler):
+    """Handler for Relative Wealth Index dataset"""
 
-            # Fall back to downloading all resources
-            self.logger.info("Falling back to downloading all RWI resources")
-            return super().download_dataset()
+    def __init__(
+        self,
+        config: Optional[RWIConfig] = None,
+        downloader: Optional[RWIDownloader] = None,
+        reader: Optional[RWIReader] = None,
+        data_store: Optional[DataStore] = None,
+        logger: Optional[logging.Logger] = None,
+        **kwargs,
+    ):
+        super().__init__(
+            dataset_name="relative-wealth-index",
+            config=config,
+            downloader=downloader,
+            reader=reader,
+            data_store=data_store,
+            logger=logger,
+            **kwargs,
+        )
+
+    def create_config(
+        self, data_store: DataStore, logger: logging.Logger, **kwargs
+    ) -> RWIConfig:
+        """Create and return a RWIConfig instance"""
+        return RWIConfig(
+            data_store=data_store,
+            logger=logger,
+            **kwargs,
+        )
+
+    def create_downloader(
+        self,
+        config: RWIConfig,
+        data_store: DataStore,
+        logger: logging.Logger,
+        **kwargs,
+    ) -> RWIDownloader:
+        """Create and return a RWIDownloader instance"""
+        return RWIDownloader(
+            config=config,
+            data_store=data_store,
+            logger=logger,
+            **kwargs,
+        )
+
+    def create_reader(
+        self,
+        config: RWIConfig,
+        data_store: DataStore,
+        logger: logging.Logger,
+        **kwargs,
+    ) -> RWIReader:
+        """Create and return a RWIReader instance"""
+        return RWIReader(
+            config=config,
+            data_store=data_store,
+            logger=logger,
+            **kwargs,
+        )

gigaspatial/processing/tif_processor.py

@@ -18,12 +18,12 @@ from gigaspatial.config import config
 @dataclass(config=ConfigDict(arbitrary_types_allowed=True))
 class TifProcessor:
     """
-    A class to handle tif data processing, supporting single-band, RGB, and RGBA data.
+    A class to handle tif data processing, supporting single-band, RGB, RGBA, and multi-band data.
     """
 
     dataset_path: Union[Path, str]
     data_store: Optional[DataStore] = None
-    mode: Literal["single", "rgb", "rgba"] = "single"
+    mode: Literal["single", "rgb", "rgba", "multi"] = "single"
 
     def __post_init__(self):
         """Validate inputs and set up logging."""
@@ -36,10 +36,15 @@ class TifProcessor:
 
         self._load_metadata()
 
+        # Validate mode and band count
         if self.mode == "rgba" and self.count != 4:
             raise ValueError("RGBA mode requires a 4-band TIF file")
         if self.mode == "rgb" and self.count != 3:
             raise ValueError("RGB mode requires a 3-band TIF file")
+        if self.mode == "single" and self.count != 1:
+            raise ValueError("Single mode requires a 1-band TIF file")
+        if self.mode == "multi" and self.count < 2:
+            raise ValueError("Multi mode requires a TIF file with 2 or more bands")
 
     @contextmanager
     def open_dataset(self):
@@ -118,6 +123,16 @@ class TifProcessor:
                     self._tabular = self._to_rgb_dataframe(drop_nodata=True)
                 elif self.mode == "rgba":
                     self._tabular = self._to_rgba_dataframe(drop_transparent=True)
+                elif self.mode == "multi":
+                    self._tabular = self._to_multi_band_dataframe(
+                        drop_nodata=True,
+                        drop_values=[],
+                        band_names=None,  # Use default band naming
+                    )
+                else:
+                    raise ValueError(
+                        f"Invalid mode: {self.mode}. Must be one of: single, rgb, rgba, multi"
+                    )
             except Exception as e:
                 raise ValueError(
                     f"Failed to process TIF file in mode '{self.mode}'. "
@@ -393,6 +408,77 @@ class TifProcessor:
         self.logger.info("Dataset is processed!")
         return data
 
+    def _to_multi_band_dataframe(
+        self,
+        drop_nodata: bool = True,
+        drop_values: list = [],
+        band_names: Optional[List[str]] = None,
+    ) -> pd.DataFrame:
+        """
+        Process multi-band TIF to DataFrame with all bands included.
+
+        Args:
+            drop_nodata (bool): Whether to drop nodata values. Defaults to True.
+            drop_values (list): Additional values to drop from the dataset. Defaults to empty list.
+            band_names (Optional[List[str]]): Custom names for the bands. If None, bands will be named using
+                                            the band descriptions from the GeoTIFF metadata if available,
+                                            otherwise 'band_1', 'band_2', etc.
+
+        Returns:
+            pd.DataFrame: DataFrame containing coordinates and all band values
+        """
+        self.logger.info("Processing multi-band dataset...")
+
+        with self.open_dataset() as src:
+            # Read all bands
+            stack = src.read()
+
+            x_coords, y_coords = self._get_pixel_coordinates()
+
+            # Initialize dictionary with coordinates
+            data_dict = {"lon": x_coords.flatten(), "lat": y_coords.flatten()}
+
+            # Get band descriptions from metadata if available
+            if band_names is None and hasattr(src, "descriptions") and src.descriptions:
+                band_names = [
+                    desc if desc else f"band_{i+1}"
+                    for i, desc in enumerate(src.descriptions)
+                ]
+
+            # Process each band
+            for band_idx in range(self.count):
+                band_data = stack[band_idx]
+
+                # Handle nodata and other values to drop
+                if drop_nodata or drop_values:
+                    values_to_mask = []
+                    if drop_nodata and src.nodata is not None:
+                        values_to_mask.append(src.nodata)
+                    if drop_values:
+                        values_to_mask.extend(drop_values)
+
+                    if values_to_mask:
+                        data_mask = ~np.isin(band_data, values_to_mask)
+                        band_values = np.extract(data_mask, band_data)
+                        if band_idx == 0:  # Only need to mask coordinates once
+                            data_dict["lon"] = np.extract(data_mask, x_coords)
+                            data_dict["lat"] = np.extract(data_mask, y_coords)
+                    else:
+                        band_values = band_data.flatten()
+                else:
+                    band_values = band_data.flatten()
+
+                # Use custom band names if provided, otherwise use descriptions or default naming
+                band_name = (
+                    band_names[band_idx]
+                    if band_names and len(band_names) > band_idx
+                    else f"band_{band_idx + 1}"
+                )
+                data_dict[band_name] = band_values
+
+        self.logger.info("Multi-band dataset is processed!")
+        return pd.DataFrame(data_dict)
+
     def _get_pixel_coordinates(self):
         """Helper method to generate coordinate arrays for all pixels"""
         if "pixel_coords" not in self._cache: