wxdata 1.7__tar.gz → 2.0__tar.gz

{wxdata-1.7/src/wxdata.egg-info → wxdata-2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: wxdata
-Version: 1.7
+Version: 2.0
 Summary: A Python package of end-to-end weather data clients & raw data clients with VPN/PROXY support, data processors that decode variable keys from GRIB format into a plain-language format & various tools for assisting Python automated workflows, querying meteorological datasets and filling gaps in meteorological data.
 Author: Eric J. Drewitz
 Project-URL: Documentation, https://github.com/edrewitz/wxdata/blob/main/Documentation/Landing%20Page.md
@@ -63,13 +63,14 @@ A Python package consisting of the following:
 # Table of Contents
 
 1) [Installation Instructions](https://github.com/edrewitz/WxData?tab=readme-ov-file#installation-instructions)
-2) [Proxy Server Configuration](https://github.com/edrewitz/WxData?tab=readme-ov-file#proxy-server-configuration)
-3) [What makes WxData unique among various meteorological Python packages?](https://github.com/edrewitz/WxData?tab=readme-ov-file#what-makes-wxdata-unique-among-various-meteorological-python-packages)
-4) [WxData Tutorials](https://github.com/edrewitz/WxData?tab=readme-ov-file#wxdata-tutorials)
-5) [WxData Documentation](https://github.com/edrewitz/WxData?tab=readme-ov-file#wxdata-documentation)
-6) [Importing Functions from WxData](https://github.com/edrewitz/WxData?tab=readme-ov-file#importing-functions-from-wxdata)
-7) [Citations](https://github.com/edrewitz/WxData?tab=readme-ov-file#citations)
-8) [Data Sources](https://github.com/edrewitz/WxData?tab=readme-ov-file#data-sources)
+2) [Server List (For End-To-End Clients That Can Rotate Between Servers)](https://github.com/edrewitz/WxData/blob/main/README.md#server-list)
+3) [Proxy Server Configuration](https://github.com/edrewitz/WxData?tab=readme-ov-file#proxy-server-configuration)
+4) [What makes WxData unique among various meteorological Python packages?](https://github.com/edrewitz/WxData?tab=readme-ov-file#what-makes-wxdata-unique-among-various-meteorological-python-packages)
+5) [WxData Tutorials](https://github.com/edrewitz/WxData?tab=readme-ov-file#wxdata-tutorials)
+6) [WxData Documentation](https://github.com/edrewitz/WxData?tab=readme-ov-file#wxdata-documentation)
+7) [Importing Functions from WxData](https://github.com/edrewitz/WxData?tab=readme-ov-file#importing-functions-from-wxdata)
+8) [Citations](https://github.com/edrewitz/WxData?tab=readme-ov-file#citations)
+9) [Data Sources](https://github.com/edrewitz/WxData?tab=readme-ov-file#data-sources)
 
 
 ## Installation Instructions
@@ -114,6 +115,41 @@ i) Install wxdata via Anaconda/Miniconda3 --> `conda install wxdata`
 
 ii) Set up a new environment with an earlier version of Python (must be Python >= 3.10) and then `pip install wxdata`
 
+---------------------------------------------------------
+## Server List
+
+
+
+<img src="https://github.com/edrewitz/WxData/blob/main/icons/noaa_rotation.png?raw=true" width="200" alt="Alt text" /> <img src="https://github.com/edrewitz/WxData/blob/main/icons/ecmwf_rotation.png?raw=true" width="200" alt="Alt text" />
+
+End-To-End clients with multiple servers to pull data from can find the different options for `source` by the table below
+
+`source='noaa' - NOAA/NCEP/NOMADS OR NOAA/NWS/FTP`
+
+`source='ecmwf' - ECMWF Open-Data Server`
+
+`source='aws' - Amazon Web Services (AWS)`
+
+`source='google' - Google Cloud`
+
+
+| Client | NOAA/NCEP/NOMADS | ECMWF Open-Data | Amazon AWS | Google Cloud | NOAA/NWS/FTP |
+| -------- | -------- | -------- | -------- | -------- | -------- |
+| GFS0P25  |Y|N|Y|Y|N|
+| GFS0P25 SECONDARY PARAMETERS  |Y|N|Y|Y|N| 
+| GFS0P50  |Y|N|Y|Y|N|
+| GEFS0P50  |Y|N|Y|Y|N|
+| GEFS0P50 SECONDARY PARAMETERS  |Y|N|Y|Y|N|
+| GEFS0P25  |Y|N|Y|Y|N|
+| ECMWF IFS |N|Y|Y|Y|N|
+| ECMWF IFS Ensemble |N|Y|Y|Y|N|
+| ECMWF AIFS|N|Y|Y|Y|N|
+| ECMWF AIFS Ensemble|N|Y|Y|Y|N|
+| ECMWF IFS Wave |N|Y|Y|Y|N|
+| ECMWF IFS Wave Ensemble |N|Y|Y|Y|N|
+| Get NDFD Grids |N|N|Y|N|Y|
+| RTMA |Y|N|Y|N|N|
+
 ---------------------------------------------------------
 
 ## Proxy Server Configuration
@@ -172,7 +208,13 @@ ii) Set up a new environment with an earlier version of Python (must be Python >
 
 5) Consists of both observational and model data.
 
-6) Has additional tools to assist querying data, resolving gaps in data and automating your Python workflow. 
+6) Has additional tools to assist querying data, resolving gaps in data and automating your Python workflow.
+
+7) Utilizes byte-range requests to subset and speed up download times.
+
+8) Several end-to-end clients have the ability to pull from multiple servers (NOAA/NCEP/NOMADS, ECMWF Open-Data, Amazon Web Services (AWS), Google Cloud and NOAA/NWS/FTP).
+
+9) Clients can automatically rotate to a new server when experiencing connectivity issues to the initial server. 
 
 -----------------------------------------------
     

{wxdata-1.7 → wxdata-2.0}/README.md

@@ -31,13 +31,14 @@ A Python package consisting of the following:
 # Table of Contents
 
 1) [Installation Instructions](https://github.com/edrewitz/WxData?tab=readme-ov-file#installation-instructions)
-2) [Proxy Server Configuration](https://github.com/edrewitz/WxData?tab=readme-ov-file#proxy-server-configuration)
-3) [What makes WxData unique among various meteorological Python packages?](https://github.com/edrewitz/WxData?tab=readme-ov-file#what-makes-wxdata-unique-among-various-meteorological-python-packages)
-4) [WxData Tutorials](https://github.com/edrewitz/WxData?tab=readme-ov-file#wxdata-tutorials)
-5) [WxData Documentation](https://github.com/edrewitz/WxData?tab=readme-ov-file#wxdata-documentation)
-6) [Importing Functions from WxData](https://github.com/edrewitz/WxData?tab=readme-ov-file#importing-functions-from-wxdata)
-7) [Citations](https://github.com/edrewitz/WxData?tab=readme-ov-file#citations)
-8) [Data Sources](https://github.com/edrewitz/WxData?tab=readme-ov-file#data-sources)
+2) [Server List (For End-To-End Clients That Can Rotate Between Servers)](https://github.com/edrewitz/WxData/blob/main/README.md#server-list)
+3) [Proxy Server Configuration](https://github.com/edrewitz/WxData?tab=readme-ov-file#proxy-server-configuration)
+4) [What makes WxData unique among various meteorological Python packages?](https://github.com/edrewitz/WxData?tab=readme-ov-file#what-makes-wxdata-unique-among-various-meteorological-python-packages)
+5) [WxData Tutorials](https://github.com/edrewitz/WxData?tab=readme-ov-file#wxdata-tutorials)
+6) [WxData Documentation](https://github.com/edrewitz/WxData?tab=readme-ov-file#wxdata-documentation)
+7) [Importing Functions from WxData](https://github.com/edrewitz/WxData?tab=readme-ov-file#importing-functions-from-wxdata)
+8) [Citations](https://github.com/edrewitz/WxData?tab=readme-ov-file#citations)
+9) [Data Sources](https://github.com/edrewitz/WxData?tab=readme-ov-file#data-sources)
 
 
 ## Installation Instructions
@@ -82,6 +83,41 @@ i) Install wxdata via Anaconda/Miniconda3 --> `conda install wxdata`
 
 ii) Set up a new environment with an earlier version of Python (must be Python >= 3.10) and then `pip install wxdata`
 
+---------------------------------------------------------
+## Server List
+
+
+
+<img src="https://github.com/edrewitz/WxData/blob/main/icons/noaa_rotation.png?raw=true" width="200" alt="Alt text" /> <img src="https://github.com/edrewitz/WxData/blob/main/icons/ecmwf_rotation.png?raw=true" width="200" alt="Alt text" />
+
+End-To-End clients with multiple servers to pull data from can find the different options for `source` by the table below
+
+`source='noaa' - NOAA/NCEP/NOMADS OR NOAA/NWS/FTP`
+
+`source='ecmwf' - ECMWF Open-Data Server`
+
+`source='aws' - Amazon Web Services (AWS)`
+
+`source='google' - Google Cloud`
+
+
+| Client | NOAA/NCEP/NOMADS | ECMWF Open-Data | Amazon AWS | Google Cloud | NOAA/NWS/FTP |
+| -------- | -------- | -------- | -------- | -------- | -------- |
+| GFS0P25  |Y|N|Y|Y|N|
+| GFS0P25 SECONDARY PARAMETERS  |Y|N|Y|Y|N| 
+| GFS0P50  |Y|N|Y|Y|N|
+| GEFS0P50  |Y|N|Y|Y|N|
+| GEFS0P50 SECONDARY PARAMETERS  |Y|N|Y|Y|N|
+| GEFS0P25  |Y|N|Y|Y|N|
+| ECMWF IFS |N|Y|Y|Y|N|
+| ECMWF IFS Ensemble |N|Y|Y|Y|N|
+| ECMWF AIFS|N|Y|Y|Y|N|
+| ECMWF AIFS Ensemble|N|Y|Y|Y|N|
+| ECMWF IFS Wave |N|Y|Y|Y|N|
+| ECMWF IFS Wave Ensemble |N|Y|Y|Y|N|
+| Get NDFD Grids |N|N|Y|N|Y|
+| RTMA |Y|N|Y|N|N|
+
 ---------------------------------------------------------
 
 ## Proxy Server Configuration
@@ -140,7 +176,13 @@ ii) Set up a new environment with an earlier version of Python (must be Python >
 
 5) Consists of both observational and model data.
 
-6) Has additional tools to assist querying data, resolving gaps in data and automating your Python workflow. 
+6) Has additional tools to assist querying data, resolving gaps in data and automating your Python workflow.
+
+7) Utilizes byte-range requests to subset and speed up download times.
+
+8) Several end-to-end clients have the ability to pull from multiple servers (NOAA/NCEP/NOMADS, ECMWF Open-Data, Amazon Web Services (AWS), Google Cloud and NOAA/NWS/FTP).
+
+9) Clients can automatically rotate to a new server when experiencing connectivity issues to the initial server. 
 
 -----------------------------------------------
     

{wxdata-1.7 → wxdata-2.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "wxdata"
-version = "1.7"
+version = "2.0"
 description = "A Python package of end-to-end weather data clients & raw data clients with VPN/PROXY support, data processors that decode variable keys from GRIB format into a plain-language format & various tools for assisting Python automated workflows, querying meteorological datasets and filling gaps in meteorological data."
 readme = "README.md"
 requires-python = ">=3.10"

{wxdata-1.7 → wxdata-2.0}/src/wxdata/__init__.py

@@ -136,8 +136,8 @@ from wxdata.fems.meta_data import(
 )
 
 # Real-Time Mesoscale Analysis (RTMA)
-# - RTMA Latest
-# - RTMA Comparison Between Two Times
+# - RTMA Latest 
+# - RTMA Comparison Between Two Times 
 from wxdata.rtma.rtma import(
     rtma, 
     rtma_comparison

{wxdata-1.7 → wxdata-2.0}/src/wxdata/aigefs/aigefs.py

@@ -56,7 +56,25 @@ def aigefs_pressure_members(final_forecast_hour=384,
             custom_directory=None,
             chunk_size=8192,
             notifications='off',
-            clear_data=False):
+            clear_data=False,
+            variables=['geopotential height',
+                       'specific humidity',
+                       'temperature',
+                       'u-component of wind',
+                       'v-component of wind',
+                       'vertical velocity (pressure)'],
+            levels=[1000,
+                    925,
+                    850,
+                    700,
+                    600,
+                    500,
+                    400,
+                    300,
+                    250,
+                    150,
+                    100,
+                    50]):
     
     """
     This function downloads, pre-processes and post-processes the latest pressure parameter dataset of the AIGEFS and bins the files to specific folders based on ensemble number.
@@ -116,7 +134,27 @@ def aigefs_pressure_members(final_forecast_hour=384,
     
     17) clear_data (Boolean) - Default=False. When set to False, the scanner safe-guard remains in place (recommended for most users).
         When set to True, the scanner safe-guard is disabled and directory branch is cleared and new data is downloaded. 
-    
+        
+    18) variables (String List) Default=['geopotential height',
+                                        'specific humidity',
+                                        'temperature',
+                                        'u-component of wind',
+                                        'v-component of wind',
+                                        'vertical velocity (pressure)']
+                       
+        
+    19) levels (Integer List) - Default=[1000,
+                                        925,
+                                        850,
+                                        700,
+                                        600,
+                                        500,
+                                        400,
+                                        300,
+                                        250,
+                                        150,
+                                        100,
+                                        50]
     
     Returns
     -------
@@ -174,26 +212,41 @@ def aigefs_pressure_members(final_forecast_hour=384,
         for path, url in zip(paths, urls):
             for i in range(0, stop, 6):
                 if i < 10:
-                    _client.get_gridded_data(f"{url}/aigefs.t{run}z.pres.f00{i}.grib2",
-                                path,
-                                f"aigefs.t{run}z.pres.f00{i}.grib2",
-                                proxies=proxies,
-                                chunk_size=chunk_size,
-                                notifications=notifications)  
+                    _client.byte_range_request(f"{url}/aigefs.t{run}z.pres.f00{i}.grib2",
+                                                f"{url}/aigefs.t{run}z.pres.f00{i}.grib2.idx",
+                                                variables,
+                                                levels,
+                                                'pressure',
+                                                path,
+                                                f"aigefs.t{run}z.pres.f00{i}.grib2",
+                                                proxies=proxies,
+                                                chunk_size=chunk_size,
+                                                notifications=notifications,
+                                                clear_recycle_bin=clear_recycle_bin)
                 elif i >= 10 and i < 100:
-                    _client.get_gridded_data(f"{url}/aigefs.t{run}z.pres.f0{i}.grib2",
-                                path,
-                                f"aigefs.t{run}z.pres.f0{i}.grib2",
-                                proxies=proxies,
-                                chunk_size=chunk_size,
-                                notifications=notifications)  
+                    _client.byte_range_request(f"{url}/aigefs.t{run}z.pres.f0{i}.grib2",
+                                                f"{url}/aigefs.t{run}z.pres.f0{i}.grib2.idx",
+                                                variables,
+                                                levels,
+                                                'pressure',
+                                                path,
+                                                f"aigefs.t{run}z.pres.f0{i}.grib2",
+                                                proxies=proxies,
+                                                chunk_size=chunk_size,
+                                                notifications=notifications,
+                                                clear_recycle_bin=clear_recycle_bin)
                 else:
-                    _client.get_gridded_data(f"{url}/aigefs.t{run}z.pres.f{i}.grib2",
-                                path,
-                                f"aigefs.t{run}z.pres.f{i}.grib2",
-                                proxies=proxies,
-                                chunk_size=chunk_size,
-                                notifications=notifications)     
+                    _client.byte_range_request(f"{url}/aigefs.t{run}z.pres.f{i}.grib2",
+                                                f"{url}/aigefs.t{run}z.pres.f{i}.grib2.idx",
+                                                variables,
+                                                levels,
+                                                'pressure',
+                                                path,
+                                                f"aigefs.t{run}z.pres.f{i}.grib2",
+                                                proxies=proxies,
+                                                chunk_size=chunk_size,
+                                                notifications=notifications,
+                                                clear_recycle_bin=clear_recycle_bin)   
     else:
         print(f"User has latest AIGEFS Pressure Parameter Files\nSkipping Download...")  
         
@@ -411,8 +464,26 @@ def aigefs_single(final_forecast_hour=384,
                     chunk_size=8192,
                     notifications='off',
                     cat='mean',
-                    type_of_level='pressure',
-                    clear_data=False):                   
+                    level_type='pressure',
+                    clear_data=False,
+            variables=['geopotential height',
+                       'specific humidity',
+                       'temperature',
+                       'u-component of wind',
+                       'v-component of wind',
+                       'vertical velocity (pressure)'],
+            levels=[1000,
+                    925,
+                    850,
+                    700,
+                    600,
+                    500,
+                    400,
+                    300,
+                    250,
+                    150,
+                    100,
+                    50]):                   
     
     """
     This function downloads, pre-processes and post-processes the latest AIGEFS Ensemble Mean or Ensemble Spread for either the Pressure or Surface Parameters. 
@@ -476,7 +547,7 @@ def aigefs_single(final_forecast_hour=384,
         1) mean
         2) spread
         
-    17) type_of_level (String) - Default='pressure'. The type of level the data is in.
+    17) level_type (String) - Default='pressure'. The type of level the data is in.
     
         Types of Levels
         ---------------
@@ -488,6 +559,32 @@ def aigefs_single(final_forecast_hour=384,
         When set to True, the scanner safe-guard is disabled and directory branch is cleared and new data is downloaded. 
     
     
+    19) variables (String List) **level_type='pressure'** - Default=['geopotential height',
+                                                                        'specific humidity',
+                                                                        'temperature',
+                                                                        'u-component of wind',
+                                                                        'v-component of wind',
+                                                                        'vertical velocity (pressure)']
+                       
+        When the level_type = 'pressure', the user can filter by variable to the variable they want. (Surface level files are very small 
+        compared to pressure level files).
+        
+    20) levels (Integer List) **level_type='pressure'** - Default=[1000,
+                                                                        925,
+                                                                        850,
+                                                                        700,
+                                                                        600,
+                                                                        500,
+                                                                        400,
+                                                                        300,
+                                                                        250,
+                                                                        150,
+                                                                        100,
+                                                                        50]
+                                                                        
+        When the level_type = 'pressure', the user can filter by level to the level they want. (Surface level files are very small 
+        compared to pressure level files).
+    
     Returns
     -------
     
@@ -512,14 +609,14 @@ def aigefs_single(final_forecast_hour=384,
     '2m_temperature'
     """
     cat = cat.lower()
-    type_of_level = type_of_level.lower()
+    level_type = level_type.lower()
     
     if cat == 'mean':
         cat = 'avg'
     else:
         cat = 'spr'
         
-    if type_of_level == 'pressure':
+    if level_type == 'pressure':
         level = 'pres'
     else:
         level = 'sfc'
@@ -532,7 +629,7 @@ def aigefs_single(final_forecast_hour=384,
         pass    
     
     if custom_directory == None:
-        path = _build_aigefs_single_directory(type_of_level,
+        path = _build_aigefs_single_directory(level_type,
                                   cat)
     else:
         path = _custom_branch(custom_directory)
@@ -545,7 +642,7 @@ def aigefs_single(final_forecast_hour=384,
     url, file, run = _aigefs_single_url_scanner(final_forecast_hour,
                                                         proxies,
                                                         cat,
-                                                        type_of_level)
+                                                        level_type)
     
     download = _local_file_scanner(path, 
                                     file,
@@ -554,7 +651,7 @@ def aigefs_single(final_forecast_hour=384,
                                     model='aigefs')  
     
     if download == True:
-        print(f"Downloading AIGEFS {type_of_level.upper()} {cat.upper()} Files...")
+        print(f"Downloading AIGEFS {level_type.upper()} {cat.upper()} Files...")
         
         _clear_old_data(path)
         
@@ -564,34 +661,73 @@ def aigefs_single(final_forecast_hour=384,
             run = f"{run}"        
         stop = final_forecast_hour + 6
         
+        
         for i in range(0, stop, 6):
-            if i < 10:
-                _client.get_gridded_data(f"{url}aigefs.t{run}z.{level}.{cat}.f00{i}.grib2",
-                            path,
-                            f"aigefs.t{run}z.{level}.{cat}.f00{i}.grib2",
-                            proxies=proxies,
-                            chunk_size=chunk_size,
-                            notifications=notifications)  
-            elif i >= 10 and i < 100:
-                _client.get_gridded_data(f"{url}aigefs.t{run}z.{level}.{cat}.f0{i}.grib2",
-                            path,
-                            f"aigefs.t{run}z.{level}.{cat}.f0{i}.grib2",
-                            proxies=proxies,
-                            chunk_size=chunk_size,
-                            notifications=notifications)  
+            if level_type == 'surface':
+                if i < 10:
+                    _client.get_gridded_data(f"{url}aigefs.t{run}z.{level}.{cat}.f00{i}.grib2",
+                                path,
+                                f"aigefs.t{run}z.{level}.{cat}.f00{i}.grib2",
+                                proxies=proxies,
+                                chunk_size=chunk_size,
+                                notifications=notifications)  
+                elif i >= 10 and i < 100:
+                    _client.get_gridded_data(f"{url}aigefs.t{run}z.{level}.{cat}.f0{i}.grib2",
+                                path,
+                                f"aigefs.t{run}z.{level}.{cat}.f0{i}.grib2",
+                                proxies=proxies,
+                                chunk_size=chunk_size,
+                                notifications=notifications)  
+                else:
+                    _client.get_gridded_data(f"{url}aigefs.t{run}z.{level}.{cat}.f{i}.grib2",
+                                path,
+                                f"aigefs.t{run}z.{level}.{cat}.f{i}.grib2",
+                                proxies=proxies,
+                                chunk_size=chunk_size,
+                                notifications=notifications) 
             else:
-                _client.get_gridded_data(f"{url}aigefs.t{run}z.{level}.{cat}.f{i}.grib2",
-                            path,
-                            f"aigefs.t{run}z.{level}.{cat}.f{i}.grib2",
-                            proxies=proxies,
-                            chunk_size=chunk_size,
-                            notifications=notifications)    
+                if i < 10:
+                    _client.byte_range_request(f"{url}aigefs.t{run}z.{level}.{cat}.f00{i}.grib2",
+                                                    f"{url}aigefs.t{run}z.{level}.{cat}.f00{i}.grib2.idx",
+                                                    variables,
+                                                    levels,
+                                                    'pressure',
+                                                    path,
+                                                    f"aigefs.t{run}z.pres.f00{i}.grib2",
+                                                    proxies=proxies,
+                                                    chunk_size=chunk_size,
+                                                    notifications=notifications,
+                                                    clear_recycle_bin=clear_recycle_bin)
+                elif i >= 10 and i < 100:
+                    _client.byte_range_request(f"{url}aigefs.t{run}z.{level}.{cat}.f0{i}.grib2",
+                                                f"{url}aigefs.t{run}z.{level}.{cat}.f0{i}.grib2.idx",
+                                                variables,
+                                                levels,
+                                                'pressure',
+                                                path,
+                                                f"aigefs.t{run}z.pres.f0{i}.grib2",
+                                                proxies=proxies,
+                                                chunk_size=chunk_size,
+                                                notifications=notifications,
+                                                clear_recycle_bin=clear_recycle_bin)
+                else:
+                    _client.byte_range_request(f"{url}aigefs.t{run}z.{level}.{cat}.f{i}.grib2",
+                                                f"{url}aigefs.t{run}z.{level}.{cat}.f{i}.grib2.idx",
+                                                variables,
+                                                levels,
+                                                'pressure',
+                                                path,
+                                                f"aigefs.t{run}z.pres.f{i}.grib2",
+                                                proxies=proxies,
+                                                chunk_size=chunk_size,
+                                                notifications=notifications,
+                                                clear_recycle_bin=clear_recycle_bin)    
                     
     else:
-        print(f"User has latest AIGEFS {type_of_level.upper()} {cat.upper()} Files\nSkipping Download...")  
+        print(f"User has latest AIGEFS {level_type.upper()} {cat.upper()} Files\nSkipping Download...")  
             
     if process_data == True:
-        print(f"AIGEFS {type_of_level.upper()} {cat.upper()} Data Processing...")    
+        print(f"AIGEFS {level_type.upper()} {cat.upper()} Data Processing...")    
         
         ds = _aigefs_post_processing.aigefs_single_post_processing(path,
                                                                     western_bound,
@@ -605,7 +741,7 @@ def aigefs_single(final_forecast_hour=384,
                                            cat='mean')
                 
         
-        print(f"AIGEFS {type_of_level.upper()} {cat.upper()} Data Processing Complete.")
+        print(f"AIGEFS {level_type.upper()} {cat.upper()} Data Processing Complete.")
         return ds
     else:
         pass