sentle 2024.5.2__tar.gz

sentle-2024.5.2/LICENSE.md

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Clemens Mosig
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

sentle-2024.5.2/PKG-INFO

@@ -0,0 +1,19 @@
+Metadata-Version: 2.1
+Name: sentle
+Version: 2024.5.2
+Summary: Sentinel-2 scalable downloader.
+Home-page: UNKNOWN
+Author: Clemens Mosig
+Author-email: clemens.mosig@uni-leipzig.de
+License: UNKNOWN
+Platform: UNKNOWN
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Education
+Classifier: Programming Language :: Python :: 2
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: MacOS :: MacOS X
+Classifier: Operating System :: Microsoft :: Windows
+License-File: LICENSE.md
+
+Download Sentinel-2 data cubes of any scale (larger-than-memory) on any machine with integrated cloud detection, snow masking, harmonization, merging, and temporal composites.
+

sentle-2024.5.2/README.md

@@ -0,0 +1,125 @@
+<p align="center">
+  <a href="https://github.com/cmosig/sentle/"><img src="https://github.com/cmosig/sentle/raw/main/docs/logo.png" alt="sentle"></a>
+</p>
+
+<p align="center">
+<a href="https://opensource.org/licenses/MIT" target="_blank">
+    <img src="https://img.shields.io/badge/License-MIT-blue.svg" alt="License">
+</a>
+<a href="https://peps.python.org/pep-0008/" target="_blank">
+    <img src="https://img.shields.io/badge/code_style-pep8-blue" alt="Black">
+</a>
+</p>
+<p align="center">
+    <em>Download Sentinel-1 & Sentinel-2 data cubes of huge-scale (larger-than-memory) on any machine with integrated cloud
+detection, snow masking, harmonization, merging, and temporal composites.</em>
+</p>
+
+---
+ 
+## Important Note
+
+1) The model for cloud detection will be made available within the next couple of weeks.
+2) **This package is in early alpha stage. There will be bugs!** If you encounter any error, warning, memory issue, etc. please open a GitHub issue with the code to reproduce.
+
+## Installing
+
+```
+pip install sentle
+```
+or 
+```
+git clone git@github.com:cmosig/sentle.git
+cd sentle
+pip install -e .
+```
+
+## Guide
+
+**(1) Setup**
+
+There is only one important function: `process`. Here, you specify all parameters and the function returns a lazy [dask](https://www.dask.org/) array with the shape `(#timesteps, #bands, #pixelsy, #pixelsx)`.
+
+```
+from sentle import sentle
+from rasterio.crs import CRS
+
+da = sentle.process(
+    target_crs=CRS.from_string("EPSG:32633"),
+    bound_left=176000,
+    bound_bottom=5660000,
+    bound_right=216000,
+    bound_top=5700000,
+    datetime="2022-06-17/2023-06-17",
+    target_resolution=10,
+    S2_mask_snow=True,
+    S2_cloud_classification=True,
+    S2_cloud_classification_device="cuda",
+    S1_assets=["vv", "vh"],
+    S2_apply_snow_mask=True,
+    S2_apply_cloud_mask=True,
+    time_composite_freq="7d",
+    num_workers=7,
+)
+```
+This code downloads data for a 40km by 40km area with one year of both Sentinel-1 and Sentinel-2. Clouds and snow are detected and replaced with NaNs. Data is also averaged every 7 days. A lazy dask array is returned:
+
+<p align="center">
+<img src="https://github.com/cmosig/sentle/assets/32590522/f487bba1-3c10-42a2-9b10-356ab2b44825" width="600">
+</p>
+
+Explanation:
+- `target_crs`: Specifies the target CRS that all data will be reprojected to.
+- `target_resolution`:  Determines the resolution that all data is reprojected to in the `target_crs`. 
+- `bound_*`: Bounds in `target_crs` of the area you want to download. Undefined behavior if difference between opposite bounds is not divisable by `target_resolution`.
+- `datetime`: Time range that will be downloaded.
+- `S2_mask_snow`: Whether to compute snow mask for Sentinel-2 data.
+- `S2_cloud_classification`: Whether to a cloud classification layer for Sentinel-2 data.
+- `S2_cloud_classification_device`: Where to run cloud classification. If you have an Nvidia GPU then pass `cuda` otherwise `cpu`(default).
+- `S2_apply_*`: Whether to apply the respective mask, i.e., replace values by NaN.
+- `S1_assets`: Which Sentinel-1 assets to download. Disable Sentinel-1 by setting this to `None`.
+- `time_composite_freq`: Rounding interval across which data is averaged. Uses `pandas.Timestamp.round(time_composite_freq)`. Cloud/snow masks are dropped after masking because they cannot be aggregated.
+- `num_workers`: Number of cores to use. Plan about 4 GiB of memory usage per worker.
+
+**(2) Compute**
+
+You either run `.compute()` on the returned dask array or pass the object to
+`sentle.save_as_zarr(da, path="..."))` which setups zarr storage and saves each chunk as to disk as
+soon as it's ready. The latter enables an area and temporal range to be
+computed that is much larger than the RAM on your machine. 
+
+**(3 Visualize)**
+
+Load the data with xarray and visualize using for example the awesome [lexcube](https://github.com/msoechting/lexcube) package. Here, band B02 is visualized from the above example. One is able to spot the cloud gaps and the spotty coverage during winter.
+
+<p align="center">
+<img src=https://github.com/cmosig/sentle/assets/32590522/33b7f6a0-532e-453b-80db-748d99e753a2/>
+</p>  
+
+## Questions you may have
+
+#### Where can I watch the progress of the download?
+Upon initialization, `sentle` prints a link to a [dask dashboard](https://docs.dask.org/en/latest/dashboard.html). Check the bottom right pane in the Status tab for a progress bar. 
+A variety of other stats are also visible there. If you are working on a remote machine you may need to use [port forwarding](https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding) to access the remote dashboard.
+![image](https://github.com/cmosig/sentle/assets/32590522/c20516b5-7a9e-4e99-953a-9c8325edea7b)
+
+
+#### How do I scale this program?
+Increase the number of workers using the `num_workers` parameter when setting up the `Sentle` class. With default spatial chunk size of 4000, specified by `processing_spatial_chunk_size`, you should plan with 4GiB per worker. At the moment (will change), each worker also initiates its own model on the GPU, meaning more workers will also mean that more GPU VRAM will be used. 
+
+#### My dask graph is too big, what do I do?
+Increase the `processing_spatial_chunk_size` from `4000` to something higher in the `process` function. This will increase spatial chunk sizes, but will also increase worker memory requirements. 
+
+## Contributing
+
+Please submit issues or pull requests if you feel like something is missing or
+needs to be fixed. 
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details.
+
+## Acknowledgments
+
+Thank you to [David Montero](https://github.com/davemlz) for all the
+discussions and his awesome packages which inspired this.

sentle-2024.5.2/sentle/cloud_mask.py

@@ -0,0 +1,54 @@
+import os
+
+import numpy as np
+import pkg_resources
+import torch
+import xarray as xr
+
+
+def load_cloudsen_model(device: str = "cpu"):
+    pkg_path = os.path.dirname(
+        pkg_resources.resource_filename("sentle", "sentle.py"))
+    model_path = os.path.join(pkg_path, "data", "cloudmodel.pt")
+    cloudsen_model = torch.jit.load(model_path)
+    cloudsen_model.eval()
+    cloudsen_model.to(device)
+
+    return cloudsen_model
+
+
+S2_cloud_mask_band = "S2_cloud_classification"
+S2_cloud_prob_bands = [
+    "S2_clear_sky_probability", "S2_thick_cloud_probability",
+    "S2_thin_cloud_probability", "S2_shadow_probability"
+]
+
+
+def compute_cloud_mask(array: np.array, model: torch.jit.ScriptModule,
+                       S2_cloud_classification_device: str):
+
+    assert array.shape == (
+        12, 732,
+        732), "only supporting shape (12, 732, 732) for cloud masking for now"
+
+    # add padding so that shape is divisable by 16 for cloudsen
+    array = np.pad(array, [(0, 0), (2, 2), (2, 2)], "edge")
+
+    # expand one dim because it needs it
+    array = np.expand_dims(array, axis=0)
+
+    # Convert array to torch tensor, divide by 10000
+    # This mantains the array in [0,1]
+    tensor = torch.from_numpy(array) / 10000
+
+    # move to device
+    tensor = tensor.to(S2_cloud_classification_device)
+
+    # Compute the cloud mask
+    with torch.no_grad():
+        cloud_probabilities = model(tensor.type(torch.float32)).cpu().numpy()
+
+    # remove padding again
+    cloud_probabilities = cloud_probabilities[0, :, 2:-2, 2:-2]
+
+    return cloud_probabilities

sentle-2024.5.2/sentle/const.py

@@ -0,0 +1,20 @@
+S2_subtile_size = 732
+
+S2_RAW_BANDS = [
+    'B01', 'B02', 'B03', 'B04', 'B05', 'B06', 'B07', 'B08', 'B8A', 'B09',
+    'B11', 'B12'
+]
+S2_RAW_BAND_RESOLUTION = {
+    'B01': 60,
+    'B02': 10,
+    'B03': 10,
+    'B04': 10,
+    'B05': 20,
+    'B06': 20,
+    'B07': 20,
+    'B08': 10,
+    'B8A': 20,
+    'B09': 60,
+    'B11': 20,
+    'B12': 20
+}