dea-tools 0.3.7.dev28__tar.gz → 0.3.7.dev31__tar.gz

{dea_tools-0.3.7.dev28 → dea_tools-0.3.7.dev31}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: dea-tools
-Version: 0.3.7.dev28
-Summary: Functions and algorithms for analysing Digital Earth Australia data.
+Version: 0.3.7.dev31
+Summary: Open-source tools for geospatial analysis with Digital Earth Australia, Open Data Cube, and Xarray
 Project-URL: Homepage, https://github.com/GeoscienceAustralia/dea-notebooks
 Project-URL: Repository, https://github.com/GeoscienceAustralia/dea-notebooks
 Author-email: Geoscience Australia <earth.observation@ga.gov.au>
@@ -104,38 +104,34 @@ Description-Content-Type: text/markdown
 
 [![DOI](https://img.shields.io/badge/DOI-10.26186/145234-0e7fbf.svg)](https://doi.org/10.26186/145234) [![Apache license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) [![Discord](https://img.shields.io/discord/1212501566326571070?label=Discord&logo=discord&logoColor=white&color=7289DA)](https://discord.com/invite/4hhBQVas5U)
 
-`dea-tools` is an open-source Python package providing functions and algorithms for geospatial analysis using Open Data Cube, Xarray, and Digital Earth Australia.
-It includes utilities for loading data, plotting, spatial and temporal analysis, and applying machine learning to satellite data.
-The package is organised into the following modules:
-
-* `dea_tools.datahandling`: Tools for loading and managing DEA data (e.g. combining products, handling CRSs, pansharpening)
-* `dea_tools.plotting`: Plotting tools for DEA data (e.g. RGB composites, animations, interactive maps)
-* `dea_tools.bandindices`: Functions to calculate remote sensing indices (e.g. NDVI, NDWI)
-* `dea_tools.spatial`: Spatial analysis utilities (e.g. rasterisation, vectorisation, contours, image processing)
-* `dea_tools.temporal`: Tools for temporal analysis (e.g. phenology, time-series statistics, multi-dimensional regression)
-* `dea_tools.classification`: Machine learning utilities (e.g. training and applying models on satellite data)
-* `dea_tools.dask`: Utilities for parallel processing with Dask (e.g. creating scalable Dask clusters)
-* `dea_tools.landcover`: Tools for accessing and visualising DEA Land Cover data
-* `dea_tools.coastal`: Coastal and intertidal analysis tools (e.g. coastal change time series, sunglint mapping)
-* `dea_tools.bom`: Accessing Bureau of Meteorology water data (e.g. gauge and discharge data)
-* `dea_tools.waterbodies`: Accessing and analysing DEA Waterbodies data (e.g. loading waterbody time series)
-* `dea_tools.maps`: Tools for interactive mapping (e.g. folium and ipyleaflet maps)
-* `dea_tools.validation`: Tools for generating validation statistics (e.g. RMSE, R2, correlations)
+`dea-tools` is an open-source Python package for geospatial analysis of satellite data using Digital Earth Australia, Open Data Cube, and Xarray.
+It provides a broad set of utilities for loading, visualising, transforming, and analysing Earth Observation (EO) data across space and time.
+
+The package includes tools for:
+
+* 📦 **Data handling**: Load and combine DEA data products, manage projections and resolutions.
+* 🗺️ **Visualisation**: Create static and interactive maps, RGB plots, and animations.
+* 🛰️ **Remote sensing indices**: Calculate band indices such as NDVI, NDWI, and more.
+* 🌐 **Spatial and temporal analysis**: Apply raster/vector operations, extract contours, compute temporal stats, and model change over time.
+* 🤖 **Machine learning and segmentation**: Train and apply classifiers, or run image segmentation workflows.
+* ⚙️ **Parallel processing**: Set up Dask clusters for scalable processing of large datasets.
+* 🌏 **Domain-specific tools**: Analyse coastal change, intertidal zones, land cover, wetland and waterbody dynamics, and climate datasets.
 
 ## API documentation
 
-A rendered version of the `dea-tools` API is available on the [DEA Knowledge Hub](https://knowledge.dea.ga.gov.au/notebooks/Tools/).
+Full API documentation describing the modules and functions available in `dea-tools` is available on the [DEA Knowledge Hub](https://knowledge.dea.ga.gov.au/notebooks/Tools/).
 
 ## Installation
 
 You can install `dea-tools` from PyPI with `pip` (https://pypi.org/project/dea-tools/).
-By default `dea-tools` will be installed with [minimal dependencies](https://github.com/GeoscienceAustralia/dea-notebooks/blob/develop/pyproject.toml), which excludes `datacube` and other difficult to install packages.
+
+By default `dea-tools` will be installed with [minimal dependencies](https://github.com/GeoscienceAustralia/dea-notebooks/blob/develop/pyproject.toml), which excludes `datacube` and other difficult to install packages:
 
 ```console
 pip install dea-tools
 ```
 
-To install with `datacube` dependencies:
+To install with `datacube` dependencies (note that this requires access to a datacube database):
 ```console
 pip install dea-tools[datacube]
 ```
@@ -175,7 +171,7 @@ To work with this module on the DEA Sandbox or National Computational Infrastruc
 ```python
 import sys
 sys.path.insert(1, "../Tools/")
-import dea_tools.datahandling  # or some other submodule
+from dea_tools.datahandling import load_ard  # or some other function
 ```
 
 Alternatively, you can also do a local installation of `dea-tools`. To do this on the DEA Sandbox, run `pip` from the terminal from within your `dea-notebooks` directory:

{dea_tools-0.3.7.dev28 → dea_tools-0.3.7.dev31}/Tools/README.md

@@ -4,38 +4,34 @@
 
 [![DOI](https://img.shields.io/badge/DOI-10.26186/145234-0e7fbf.svg)](https://doi.org/10.26186/145234) [![Apache license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) [![Discord](https://img.shields.io/discord/1212501566326571070?label=Discord&logo=discord&logoColor=white&color=7289DA)](https://discord.com/invite/4hhBQVas5U)
 
-`dea-tools` is an open-source Python package providing functions and algorithms for geospatial analysis using Open Data Cube, Xarray, and Digital Earth Australia.
-It includes utilities for loading data, plotting, spatial and temporal analysis, and applying machine learning to satellite data.
-The package is organised into the following modules:
-
-* `dea_tools.datahandling`: Tools for loading and managing DEA data (e.g. combining products, handling CRSs, pansharpening)
-* `dea_tools.plotting`: Plotting tools for DEA data (e.g. RGB composites, animations, interactive maps)
-* `dea_tools.bandindices`: Functions to calculate remote sensing indices (e.g. NDVI, NDWI)
-* `dea_tools.spatial`: Spatial analysis utilities (e.g. rasterisation, vectorisation, contours, image processing)
-* `dea_tools.temporal`: Tools for temporal analysis (e.g. phenology, time-series statistics, multi-dimensional regression)
-* `dea_tools.classification`: Machine learning utilities (e.g. training and applying models on satellite data)
-* `dea_tools.dask`: Utilities for parallel processing with Dask (e.g. creating scalable Dask clusters)
-* `dea_tools.landcover`: Tools for accessing and visualising DEA Land Cover data
-* `dea_tools.coastal`: Coastal and intertidal analysis tools (e.g. coastal change time series, sunglint mapping)
-* `dea_tools.bom`: Accessing Bureau of Meteorology water data (e.g. gauge and discharge data)
-* `dea_tools.waterbodies`: Accessing and analysing DEA Waterbodies data (e.g. loading waterbody time series)
-* `dea_tools.maps`: Tools for interactive mapping (e.g. folium and ipyleaflet maps)
-* `dea_tools.validation`: Tools for generating validation statistics (e.g. RMSE, R2, correlations)
+`dea-tools` is an open-source Python package for geospatial analysis of satellite data using Digital Earth Australia, Open Data Cube, and Xarray.
+It provides a broad set of utilities for loading, visualising, transforming, and analysing Earth Observation (EO) data across space and time.
+
+The package includes tools for:
+
+* 📦 **Data handling**: Load and combine DEA data products, manage projections and resolutions.
+* 🗺️ **Visualisation**: Create static and interactive maps, RGB plots, and animations.
+* 🛰️ **Remote sensing indices**: Calculate band indices such as NDVI, NDWI, and more.
+* 🌐 **Spatial and temporal analysis**: Apply raster/vector operations, extract contours, compute temporal stats, and model change over time.
+* 🤖 **Machine learning and segmentation**: Train and apply classifiers, or run image segmentation workflows.
+* ⚙️ **Parallel processing**: Set up Dask clusters for scalable processing of large datasets.
+* 🌏 **Domain-specific tools**: Analyse coastal change, intertidal zones, land cover, wetland and waterbody dynamics, and climate datasets.
 
 ## API documentation
 
-A rendered version of the `dea-tools` API is available on the [DEA Knowledge Hub](https://knowledge.dea.ga.gov.au/notebooks/Tools/).
+Full API documentation describing the modules and functions available in `dea-tools` is available on the [DEA Knowledge Hub](https://knowledge.dea.ga.gov.au/notebooks/Tools/).
 
 ## Installation
 
 You can install `dea-tools` from PyPI with `pip` (https://pypi.org/project/dea-tools/).
-By default `dea-tools` will be installed with [minimal dependencies](https://github.com/GeoscienceAustralia/dea-notebooks/blob/develop/pyproject.toml), which excludes `datacube` and other difficult to install packages.
+
+By default `dea-tools` will be installed with [minimal dependencies](https://github.com/GeoscienceAustralia/dea-notebooks/blob/develop/pyproject.toml), which excludes `datacube` and other difficult to install packages:
 
 ```console
 pip install dea-tools
 ```
 
-To install with `datacube` dependencies:
+To install with `datacube` dependencies (note that this requires access to a datacube database):
 ```console
 pip install dea-tools[datacube]
 ```
@@ -75,7 +71,7 @@ To work with this module on the DEA Sandbox or National Computational Infrastruc
 ```python
 import sys
 sys.path.insert(1, "../Tools/")
-import dea_tools.datahandling  # or some other submodule
+from dea_tools.datahandling import load_ard  # or some other function
 ```
 
 Alternatively, you can also do a local installation of `dea-tools`. To do this on the DEA Sandbox, run `pip` from the terminal from within your `dea-notebooks` directory:

{dea_tools-0.3.7.dev28 → dea_tools-0.3.7.dev31}/Tools/dea_tools/dask.py

@@ -1,4 +1,4 @@
-## dea_dask.py
+# dea_dask.py
 """
 Tools for simplifying the creation of Dask clusters for parallelised computing.
 
@@ -16,7 +16,7 @@ here: https://gis.stackexchange.com/questions/tagged/open-data-cube).
 If you would like to report an issue with this script, you can file one on
 GitHub (https://github.com/GeoscienceAustralia/dea-notebooks/issues/new).
 
-Last modified: June 2022
+Last modified: July 2025
 
 """
 
@@ -24,61 +24,88 @@ import os
 from importlib.util import find_spec
 
 import dask
+import dask.distributed
 from aiohttp import ClientConnectionError
+from odc.io.cgroups import get_cpu_quota
+from odc.stac import configure_rio as cfg_rio
 
 _HAVE_PROXY = bool(find_spec("jupyter_server_proxy"))
-_IS_AWS = "AWS_ACCESS_KEY_ID" in os.environ or "AWS_DEFAULT_REGION" in os.environ
 
 
-def create_local_dask_cluster(spare_mem="3Gb", display_client=True, return_client=False):
+def create_local_dask_cluster(
+    display_client=True,
+    return_client=False,
+    configure_rio=True,
+    n_workers=1,
+    threads_per_worker=None,
+    memory_limit='spare_mem',
+    **kwargs,
+):
     """
-    Using the datacube utils function `start_local_dask`, generate
-    a local dask cluster. Automatically detects if on AWS or NCI.
+    Create a local Dask cluster for parallelised computing using ``dask.distributed.Client``.
 
-    Example use :
+    Example use:
 
-        import sys
-        sys.path.append("../Scripts")
         from dea_dask import create_local_dask_cluster
-
-        create_local_dask_cluster(spare_mem='4Gb')
+        create_local_dask_cluster()
 
     Parameters
     ----------
-    spare_mem : String, optional
-        The amount of memory, in Gb, to leave for the notebook to run.
-        This memory will not be used by the cluster. e.g '3Gb'
-    display_client : Bool, optional
+    display_client : bool, optional
         An optional boolean indicating whether to display a summary of
         the dask client, including a link to monitor progress of the
         analysis. Set to False to hide this display.
-    return_client : Bool, optional
+    return_client : bool, optional
         An optional boolean indicating whether to return the dask client
         object.
+    configure_rio : bool, optional
+       An optional boolean indicating whether to configure ``rasterio``
+       with cloud defaults and unsigned AWS access. Set to False to not
+       apply these defaults.
+    n_workers : int, optional
+        Number of workers to start, default is set to 1 which works well
+        with loading ODC data.
+    threads_per_worker: int, optional
+        Number of threads per each worker, by default this will be set to
+        the number of cpus on the machine.
+    memory_limit: str, float, int, or None, optional
+        Sets the memory limit per worker. Default is 'spare_mem', where 95 % of the available
+        system memory is split among the number of workers, allowing spare memory to be withheld
+        from the cluster.
+        To see other options: https://distributed.dask.org/en/stable/api.html#distributed.Client
+    **kwargs:
+        Additional keyword arguments passed to ``dask.distributed.Client``.
+        For full options, see: https://distributed.dask.org/en/stable/api.html#distributed.Client
 
     """
-    # Attempt to import datacube and raise an error if not available
-    try:
-        from datacube.utils.dask import start_local_dask
-        from datacube.utils.rio import configure_s3_access
-    except ImportError as e:
-        raise ImportError(
-            "`datacube` is required for `create_local_dask_cluster`. "
-            "Please install DEA Tools with the `[datacube]` extra, e.g.: "
-            "`pip install dea-tools[datacube]`"
-        ) from e
-
+    # Ensure that client links correctly launch on DEA Sandbox
     if _HAVE_PROXY:
         # Configure dashboard link to go over proxy
         prefix = os.environ.get("JUPYTERHUB_SERVICE_PREFIX", "/")
         dask.config.set({"distributed.dashboard.link": prefix + "proxy/{port}/status"})
 
-    # Start up a local cluster
-    client = start_local_dask(mem_safety_margin=spare_mem)
-
-    if _IS_AWS:
-        # Configure GDAL for s3 access
-        configure_s3_access(aws_unsigned=True, client=client)
+    # Count cpus if threads_per_worker not provided
+    if threads_per_worker is None:
+        if get_cpu_quota() is not None:
+            threads_per_worker = round(get_cpu_quota())
+        else:
+            threads_per_worker = os.cpu_count()
+
+    # by default split 95% of system memory by the n_workers.
+    if memory_limit =='spare_mem':
+        memory_limit = 0.95 / n_workers
+        
+    # Start client
+    client = dask.distributed.Client(
+        n_workers=n_workers,
+        threads_per_worker=threads_per_worker,
+        memory_limit=memory_limit,
+        **kwargs
+    )
+
+    # configure aws access
+    if configure_rio:
+        cfg_rio(cloud_defaults=True, aws={"aws_unsigned": True}, client=client)
 
     # Show the dask cluster settings
     if display_client:
@@ -86,10 +113,13 @@ def create_local_dask_cluster(spare_mem="3Gb", display_client=True, return_clien
 
         display(client)
 
-    # return the client as an object
+    # Return the client as an object
     if return_client:
         return client
 
+    # Otherwise return none
+    return None
+
 
 def create_dask_gateway_cluster(profile="r5_L", workers=2):
     """
@@ -134,7 +164,9 @@ def create_dask_gateway_cluster(profile="r5_L", workers=2):
 
         # limit username to alphanumeric characters
         # kubernetes pods won't launch if labels contain anything other than [a-Z, -, _]
-        options["jupyterhub_user"] = "".join(c if c.isalnum() else "-" for c in os.getenv("JUPYTERHUB_USER"))
+        options["jupyterhub_user"] = "".join(
+            c if c.isalnum() else "-" for c in os.getenv("JUPYTERHUB_USER")
+        )
 
         cluster = gateway.new_cluster(options)
         cluster.scale(workers)

{dea_tools-0.3.7.dev28 → dea_tools-0.3.7.dev31}/pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "dea-tools"
 dynamic = ["version"]
-description = "Functions and algorithms for analysing Digital Earth Australia data."
+description = "Open-source tools for geospatial analysis with Digital Earth Australia, Open Data Cube, and Xarray"
 authors = [
     { name = "Geoscience Australia", email = "earth.observation@ga.gov.au" }
 ]