tfv-get-tools 0.2.0__tar.gz → 0.2.2__tar.gz

tfv_get_tools-0.2.2/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 BMT
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

{tfv_get_tools-0.2.0 → tfv_get_tools-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: tfv_get_tools
-Version: 0.2.0
+Version: 0.2.2
 Summary: Tool for downloading and processing data for TUFLOW FV modelling
 Author-email: Alex Waterhouse <alex.waterhouse@apac.bmt.org>, Mitchell Smith <mitchell.smith@apac.bmt.org>, TUFLOW Support <support@tuflow.com>
 License: MIT
@@ -18,6 +18,7 @@ Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
 Requires-Python: <3.14,>=3.9
 Description-Content-Type: text/markdown
+License-File: LICENSE
 Requires-Dist: netCDF4>=1.6.2
 Requires-Dist: tqdm>=4.65.0
 Requires-Dist: xarray>=2023.1.0
@@ -41,6 +42,7 @@ Requires-Dist: sphinx>=7.3.7; extra == "dev"
 Requires-Dist: recommonmark>=0.7.1; extra == "dev"
 Requires-Dist: sphinx-markdown-builder>=0.6.6; extra == "dev"
 Requires-Dist: pyprojroot; extra == "dev"
+Dynamic: license-file
 
 # TFV Get Tools
 
@@ -58,22 +60,22 @@ TFV Get Tools is a Python package that simplifies the process of downloading and
 ### Supported Data Sources
 
 **Atmospheric Data:**
-- [ECMWF ERA5](https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5) (Default - registration required, see [CDS API](https://cds.climate.copernicus.eu/api-how-to))
+- [ECMWF ERA5](https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5) (*Atmos Default* - registration required, see [CDS API](https://cds.climate.copernicus.eu/api-how-to))
 - [NOAA CFSR](https://www.ncei.noaa.gov/data/climate-forecast-system/) (Climate Forecast System Reanalysis)
 - [BARRA2](http://www.bom.gov.au/research/projects/reanalysis/) (Australian Bureau of Meteorology)
 
 **Ocean Data:**
-- [HYCOM](https://www.hycom.org/) (Naval Research Laboratory - Global Ocean Forecast System)
+- [HYCOM](https://www.hycom.org/) (*Ocean Default* Naval Research Laboratory - Global Ocean Forecast System)
 - [Copernicus Marine](https://marine.copernicus.eu/) Global and NWS (registration required, see [Copernicus Marine Service](https://marine.copernicus.eu/))
 
 **Wave Data:**
-- [CSIRO CAWCR](https://data.csiro.au/collection/csiro:39819) (glob_24m, aus_10m, aus_4m, pac_10m, pac_4m)
+- [CSIRO CAWCR](https://data.csiro.au/collection/csiro:39819) (*Wave Default* glob_24m, aus_10m, aus_4m, pac_10m, pac_4m)
 - [Copernicus Marine](https://marine.copernicus.eu/) Global and NWS
 - [ECMWF ERA5](https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5) (registration required, see [CDS API](https://cds.climate.copernicus.eu/api-how-to))
 
 **Tidal Data:**
 - [FES2014](https://www.aviso.altimetry.fr/en/data/products/auxiliary-products/global-tide-fes.html) (AVISO+ Finite Element Solution 2014)
-- [FES2022](https://www.aviso.altimetry.fr/en/data/products/auxiliary-products/global-tide-fes.html) (AVISO+ Finite Element Solution 2022)
+- [FES2022](https://www.aviso.altimetry.fr/en/data/products/auxiliary-products/global-tide-fes.html) (*Tide Default* AVISO+ Finite Element Solution 2022)
 
 ## Installation
 
@@ -255,7 +257,6 @@ This package utilises data from multiple authoritative sources. Please ensure ap
 For questions, bug reports, or feature requests:
 - 📧 **Email**: support@tuflow.com
 - 🐛 **Issues**: Submit via the project repository
-- 📖 **Documentation**: [Link to documentation] (if available)
 
 ## Contributing
 
@@ -275,7 +276,7 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
 
 ## Authors
 
-Developed by [BMT](https://www.bmt.org/), 2025
+Developed by [TUFLOW](https://www.tuflow.com/), 2025
 
 ## Project Status
 

{tfv_get_tools-0.2.0 → tfv_get_tools-0.2.2}/README.md

@@ -14,22 +14,22 @@ TFV Get Tools is a Python package that simplifies the process of downloading and
 ### Supported Data Sources
 
 **Atmospheric Data:**
-- [ECMWF ERA5](https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5) (Default - registration required, see [CDS API](https://cds.climate.copernicus.eu/api-how-to))
+- [ECMWF ERA5](https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5) (*Atmos Default* - registration required, see [CDS API](https://cds.climate.copernicus.eu/api-how-to))
 - [NOAA CFSR](https://www.ncei.noaa.gov/data/climate-forecast-system/) (Climate Forecast System Reanalysis)
 - [BARRA2](http://www.bom.gov.au/research/projects/reanalysis/) (Australian Bureau of Meteorology)
 
 **Ocean Data:**
-- [HYCOM](https://www.hycom.org/) (Naval Research Laboratory - Global Ocean Forecast System)
+- [HYCOM](https://www.hycom.org/) (*Ocean Default* Naval Research Laboratory - Global Ocean Forecast System)
 - [Copernicus Marine](https://marine.copernicus.eu/) Global and NWS (registration required, see [Copernicus Marine Service](https://marine.copernicus.eu/))
 
 **Wave Data:**
-- [CSIRO CAWCR](https://data.csiro.au/collection/csiro:39819) (glob_24m, aus_10m, aus_4m, pac_10m, pac_4m)
+- [CSIRO CAWCR](https://data.csiro.au/collection/csiro:39819) (*Wave Default* glob_24m, aus_10m, aus_4m, pac_10m, pac_4m)
 - [Copernicus Marine](https://marine.copernicus.eu/) Global and NWS
 - [ECMWF ERA5](https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5) (registration required, see [CDS API](https://cds.climate.copernicus.eu/api-how-to))
 
 **Tidal Data:**
 - [FES2014](https://www.aviso.altimetry.fr/en/data/products/auxiliary-products/global-tide-fes.html) (AVISO+ Finite Element Solution 2014)
-- [FES2022](https://www.aviso.altimetry.fr/en/data/products/auxiliary-products/global-tide-fes.html) (AVISO+ Finite Element Solution 2022)
+- [FES2022](https://www.aviso.altimetry.fr/en/data/products/auxiliary-products/global-tide-fes.html) (*Tide Default* AVISO+ Finite Element Solution 2022)
 
 ## Installation
 
@@ -211,7 +211,6 @@ This package utilises data from multiple authoritative sources. Please ensure ap
 For questions, bug reports, or feature requests:
 - 📧 **Email**: support@tuflow.com
 - 🐛 **Issues**: Submit via the project repository
-- 📖 **Documentation**: [Link to documentation] (if available)
 
 ## Contributing
 
@@ -231,7 +230,7 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
 
 ## Authors
 
-Developed by [BMT](https://www.bmt.org/), 2025
+Developed by [TUFLOW](https://www.tuflow.com/), 2025
 
 ## Project Status
 

{tfv_get_tools-0.2.0 → tfv_get_tools-0.2.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "tfv_get_tools"
-version = "0.2.0"
+version = "0.2.2"
 description = "Tool for downloading and processing data for TUFLOW FV modelling"
 authors = [
     { name = "Alex Waterhouse", email = "alex.waterhouse@apac.bmt.org" },

{tfv_get_tools-0.2.0 → tfv_get_tools-0.2.2}/src/tfv_get_tools/providers/_merger.py

@@ -208,30 +208,23 @@ class BaseMerger(ABC):
         download_interval = self.cfg.get("_DOWNLOAD_INTERVAL", "monthly")
 
         if download_interval == "monthly":
-            start_time_strings = [x.stem.split("_")[-2] for x in file_list]
-            end_time_strings = [x.stem.split("_")[-1] for x in file_list]
+            time_stings = [re.search(r'_(\d{8})_(\d{8})', x.stem) for x in file_list]
+            start_time_strings = [x.group(1) for x in time_stings]
+            end_time_strings = [x.group(2) for x in time_stings]
+            
             start_times = pd.DatetimeIndex(
-                [pd.Timestamp(x, unit="h") for x in start_time_strings]
+                [pd.to_datetime(x, format="%Y%m%d") for x in start_time_strings]
             )
             end_times = pd.DatetimeIndex(
-                [pd.Timestamp(x, unit="h") for x in end_time_strings]
+                [pd.to_datetime(x, format="%Y%m%d") for x in end_time_strings]
             )
 
         elif download_interval == "daily":
-            start_time_strings = [x.stem.split("_")[-1] for x in file_list]
-
-            # Handle legacy HYCOM files
-            if start_time_strings and start_time_strings[0] == "0000":
-                if self.verbose:
-                    print("WARNING: Detected legacy HYCOM files with time suffixes")
-                start_time_strings = [
-                    x.name.split(".")[0].split("_")[-2] for x in file_list
-                ]
-
+            start_time_strings = [re.search(r'_(\d{8})_', x.stem).group(1) for x in file_list]
             start_times = pd.DatetimeIndex(
-                [pd.Timestamp(x, unit="h") for x in start_time_strings]
+                [pd.to_datetime(x, format="%Y%m%d") for x in start_time_strings]
             )
-            end_times = start_times + pd.Timedelta("23.9h")
+            end_times = start_times + pd.Timedelta("23.99h")
         else:
             raise ValueError(f"Unknown download interval: {download_interval}")
 

{tfv_get_tools-0.2.0 → tfv_get_tools-0.2.2}/src/tfv_get_tools/providers/wave/era5.py

@@ -99,13 +99,19 @@ class DownloadERA5Wave(BaseDownloader):
                 print("You can find your key at: https://cds.climate.copernicus.eu/user/")
                 print("="*60)
                 
+            elif "cds-beta.climate.copernicus.eu" in error_msg:
+                print("\n" + "="*60)
+                print("CDS API AUTHENTICATION ERROR")
+                print("="*60)
+                print("Your CDS API key appears to be invalid.")
+                print("This is likely due to an update by the Copernicus Climate Data Store.")
+                print("Please check your .cdsapirc file and ensure your API key is correct.")
+                print("You can find your key at: https://cds.climate.copernicus.eu/user/")
+                print("="*60)
+
             else:
-                # For any other errors, show the original message if verbose
-                if self.verbose:
-                    print(f"Failed to download via CDS API: {e}")
-                else:
-                    # Always show some info for unhandled errors
-                    print(f"\nCDS API Error: {e}")
+                print(f"Failed to download via CDS API: {e}")
+
                     
             return False
     
@@ -229,4 +235,4 @@ class MergeERA5Wave(BaseMerger):
         if 'latitude' in merged.coords:
             merged = merged.sortby("latitude")
             
-        return merged, skipped_files
+        return merged, skipped_files

{tfv_get_tools-0.2.0 → tfv_get_tools-0.2.2}/src/tfv_get_tools.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: tfv_get_tools
-Version: 0.2.0
+Version: 0.2.2
 Summary: Tool for downloading and processing data for TUFLOW FV modelling
 Author-email: Alex Waterhouse <alex.waterhouse@apac.bmt.org>, Mitchell Smith <mitchell.smith@apac.bmt.org>, TUFLOW Support <support@tuflow.com>
 License: MIT
@@ -18,6 +18,7 @@ Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
 Requires-Python: <3.14,>=3.9
 Description-Content-Type: text/markdown
+License-File: LICENSE
 Requires-Dist: netCDF4>=1.6.2
 Requires-Dist: tqdm>=4.65.0
 Requires-Dist: xarray>=2023.1.0
@@ -41,6 +42,7 @@ Requires-Dist: sphinx>=7.3.7; extra == "dev"
 Requires-Dist: recommonmark>=0.7.1; extra == "dev"
 Requires-Dist: sphinx-markdown-builder>=0.6.6; extra == "dev"
 Requires-Dist: pyprojroot; extra == "dev"
+Dynamic: license-file
 
 # TFV Get Tools
 
@@ -58,22 +60,22 @@ TFV Get Tools is a Python package that simplifies the process of downloading and
 ### Supported Data Sources
 
 **Atmospheric Data:**
-- [ECMWF ERA5](https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5) (Default - registration required, see [CDS API](https://cds.climate.copernicus.eu/api-how-to))
+- [ECMWF ERA5](https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5) (*Atmos Default* - registration required, see [CDS API](https://cds.climate.copernicus.eu/api-how-to))
 - [NOAA CFSR](https://www.ncei.noaa.gov/data/climate-forecast-system/) (Climate Forecast System Reanalysis)
 - [BARRA2](http://www.bom.gov.au/research/projects/reanalysis/) (Australian Bureau of Meteorology)
 
 **Ocean Data:**
-- [HYCOM](https://www.hycom.org/) (Naval Research Laboratory - Global Ocean Forecast System)
+- [HYCOM](https://www.hycom.org/) (*Ocean Default* Naval Research Laboratory - Global Ocean Forecast System)
 - [Copernicus Marine](https://marine.copernicus.eu/) Global and NWS (registration required, see [Copernicus Marine Service](https://marine.copernicus.eu/))
 
 **Wave Data:**
-- [CSIRO CAWCR](https://data.csiro.au/collection/csiro:39819) (glob_24m, aus_10m, aus_4m, pac_10m, pac_4m)
+- [CSIRO CAWCR](https://data.csiro.au/collection/csiro:39819) (*Wave Default* glob_24m, aus_10m, aus_4m, pac_10m, pac_4m)
 - [Copernicus Marine](https://marine.copernicus.eu/) Global and NWS
 - [ECMWF ERA5](https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5) (registration required, see [CDS API](https://cds.climate.copernicus.eu/api-how-to))
 
 **Tidal Data:**
 - [FES2014](https://www.aviso.altimetry.fr/en/data/products/auxiliary-products/global-tide-fes.html) (AVISO+ Finite Element Solution 2014)
-- [FES2022](https://www.aviso.altimetry.fr/en/data/products/auxiliary-products/global-tide-fes.html) (AVISO+ Finite Element Solution 2022)
+- [FES2022](https://www.aviso.altimetry.fr/en/data/products/auxiliary-products/global-tide-fes.html) (*Tide Default* AVISO+ Finite Element Solution 2022)
 
 ## Installation
 
@@ -255,7 +257,6 @@ This package utilises data from multiple authoritative sources. Please ensure ap
 For questions, bug reports, or feature requests:
 - 📧 **Email**: support@tuflow.com
 - 🐛 **Issues**: Submit via the project repository
-- 📖 **Documentation**: [Link to documentation] (if available)
 
 ## Contributing
 
@@ -275,7 +276,7 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
 
 ## Authors
 
-Developed by [BMT](https://www.bmt.org/), 2025
+Developed by [TUFLOW](https://www.tuflow.com/), 2025
 
 ## Project Status
 

{tfv_get_tools-0.2.0 → tfv_get_tools-0.2.2}/src/tfv_get_tools.egg-info/SOURCES.txt

@@ -1,3 +1,4 @@
+LICENSE
 README.md
 pyproject.toml
 src/tfv_get_tools/__init__.py

{tfv_get_tools-0.2.0 → tfv_get_tools-0.2.2}/tests/test_all_downloaders.py

@@ -9,11 +9,16 @@ from tfv_get_tools import DownloadAtmos, DownloadOcean, DownloadWave
 
 
 # Loopy test :) Add new sources here
+# Turned off Copernicus because of login checks
+# We could mock this properly but I think it 
+# is low value... 
+# There is a test for login failure low which will only trigger
+# if it asks for stdin user/password, which means it is working 
 @pytest.mark.parametrize(
     "source,downloader_class,model",
     [
-        ("COPERNICUS", DownloadOcean, "GLO"),
-        ("COPERNICUS", DownloadWave, "GLO"),
+        # ("COPERNICUS", DownloadOcean, "GLO"),
+        # ("COPERNICUS", DownloadWave, "GLO"),
         ("HYCOM", DownloadOcean, None),
         ("CAWCR", DownloadWave, "glob_24m"),
         ("CAWCR", DownloadWave, "aus_10m"),
@@ -59,36 +64,44 @@ class TestDownloaderIntegration:
     skip_check = True
 
     def test_copernicus_ocean_glo(self):
-        """Test COPERNICUS ocean downloader with GLO model."""
+        """Test COPERNICUS ocean downloader with GLO model.
+        These could be better - I am testing that copernicus kicks off ok
+        and that it triggers a manual user login via terminal"""
         with tempfile.TemporaryDirectory() as temp_dir:
-            result = DownloadOcean(
-                start_date=self.start_date,
-                end_date=self.end_date,
-                xlims=self.xlims,
-                ylims=self.ylims,
-                source="COPERNICUS",
-                model="GLO",
-                out_path=temp_dir,
-                TEST_MODE=self.test_mode,
-                skip_check=self.skip_check,
-            )
-            assert result is not None
+            
+            with pytest.raises(OSError,
+                               match="reading from stdin"):
+                result = DownloadOcean(
+                    start_date=self.start_date,
+                    end_date=self.end_date,
+                    xlims=self.xlims,
+                    ylims=self.ylims,
+                    source="COPERNICUS",
+                    model="GLO",
+                    out_path=temp_dir,
+                    TEST_MODE=self.test_mode,
+                    skip_check=self.skip_check,
+                )
+            
 
     def test_copernicus_wave_glo(self):
-        """Test COPERNICUS wave downloader with GLO model."""
+        """Test COPERNICUS wave downloader with GLO model.
+        These could be better - I am testing that copernicus kicks off ok
+        and that it triggers a manual user login via terminal"""
         with tempfile.TemporaryDirectory() as temp_dir:
-            result = DownloadWave(
-                start_date=self.start_date,
-                end_date=self.end_date,
-                xlims=self.xlims,
-                ylims=self.ylims,
-                source="COPERNICUS",
-                model="GLO",
-                out_path=temp_dir,
-                TEST_MODE=self.test_mode,
-                skip_check=self.skip_check,
-            )
-            assert result is not None
+            with pytest.raises(OSError,
+                    match="reading from stdin"):
+                result = DownloadWave(
+                    start_date=self.start_date,
+                    end_date=self.end_date,
+                    xlims=self.xlims,
+                    ylims=self.ylims,
+                    source="COPERNICUS",
+                    model="GLO",
+                    out_path=temp_dir,
+                    TEST_MODE=self.test_mode,
+                    skip_check=self.skip_check,
+                )
 
     def test_hycom_ocean(self):
         """Test HYCOM ocean downloader."""

{tfv_get_tools-0.2.0 → tfv_get_tools-0.2.2}/tests/test_mergers.py

@@ -378,7 +378,7 @@ class TestFileHandling:
         assert "RANDOM_FILE.nc" not in era5_files
         assert "ERA5_WAVE_20220101_20220131.nc" not in era5_files
 
-    def test_time_filtering(self):
+    def test_time_filtering_simple_ERA5(self):
         """Test filtering files by time range."""
         # Create files with different dates
         test_files = [
@@ -408,6 +408,41 @@ class TestFileHandling:
         assert "ERA5_ATMOS_20220201_20220228.nc" in filtered_files
         # Should exclude March file
         assert "ERA5_ATMOS_20220301_20220331.nc" not in filtered_files
+        
+    def test_time_filtering_painful_HYCOM(self):
+        """Test filtering files by time range."""
+        # Create files with different dates
+        test_files = [
+            "HYCOM_OCEAN_20220101_03h.nc",
+            "HYCOM_OCEAN_20220102_03h_somedbase.nc",
+            "A_LONG_PREFIX_BOUND_TO_GET_YA_HYCOM_OCEAN_20220103_03h_somedbase.nc",
+            "KEEP_ME_HYCOM_OCEAN_20210101_03h.nc",
+            "EXCLUDE_ME_HYCOM_OCEAN_20230101_03h.nc",
+            "WHY_DID_I_EVER_ALLOW_SUCH_FLEXIBILITY_HYCOM_OCEAN_20220301_03h.nc",
+        ]
+        
+        for filename in test_files:
+            (self.in_path / filename).touch()
+        
+        # Test filtering with time constraints
+        merger = MergeHYCOM(
+            in_path=self.in_path,
+            out_path=self.out_path,
+            time_start="2021-01-01", # Keep em all except the EXCLUDE_ME one
+            time_end="2022-09-01",
+            execute=False,
+            verbose=False
+        )
+        
+        files = merger.get_file_list()
+        filtered_files = [f.name for f in files]
+        
+        # Should include Jan and Feb files
+        assert "KEEP_ME_HYCOM_OCEAN_20210101_03h.nc" in filtered_files
+        
+        # Should exclude March file
+        assert "EXCLUDE_ME_HYCOM_OCEAN_20230101_03h.nc" not in filtered_files
+        
 
     def test_no_files_found_error(self):
         """Test error handling when no files match pattern."""