goesgcp 2.0.1__tar.gz → 2.0.3__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.2
2
2
  Name: goesgcp
3
- Version: 2.0.1
3
+ Version: 2.0.3
4
4
  Summary: A package to download and process GOES-16/17 data
5
5
  Home-page: https://github.com/helvecioneto/goesgcp
6
6
  Author: Helvecio B. L. Neto
@@ -41,18 +41,18 @@ Dynamic: summary
41
41
  <!-- badges: end -->
42
42
 
43
43
 
44
- `goesgcp` is a Python utility designed for downloading and reprojecting GOES-R satellite data. This script leverages the `google.cloud` library for accessing data from the Google Cloud Platform (GCP) and `pyproj` for reprojecting data to EPSG:4326, as well as cropping it to a user-defined bounding box.
44
+ `goesgcp` is a Python utility designed for downloading and reprojecting GOES-R satellite data. This script leverages the `google.cloud` library for accessing data from the Google Cloud Platform (GCP) and `rioxarray` for reprojecting data to EPSG:4326 (rectangular grid), as well cropping it to a user-defined bounding box.
45
45
 
46
46
  ## Features
47
47
 
48
- - **Download GOES-R satellite data**: Supports GOES-16 and GOES-17.
48
+ - **Download GOES-R satellite data**: Supports GOES-16 and GOES-18.
49
49
  - **Reprojection and cropping**: Reprojects data to EPSG:4326 and crops to a specified bounding box.
50
50
  - **Flexible command-line interface**: Customize download options, variables, channels, time range, and output format.
51
51
  - **Efficient processing**: Handles large datasets with optimized performance.
52
52
 
53
53
  ## Installation
54
54
 
55
- Install the necessary dependencies via `pip`:
55
+ Install the package via `pip`:
56
56
 
57
57
  ```bash
58
58
  pip install goesgcp
@@ -61,7 +61,7 @@ pip install goesgcp
61
61
 
62
62
  ## Usage
63
63
 
64
- ### Command-Line Arguments
64
+ ### Available Command-Line Arguments
65
65
 
66
66
  The script uses the `argparse` module for handling command-line arguments. Below are the available options:
67
67
 
@@ -89,7 +89,7 @@ goesgcp [OPTIONS]
89
89
  | `--save_format` | Format for saving output files (default: `by_date`). |
90
90
 
91
91
  #### Available GOES Products
92
- A comprehensive list of available GOES products can be found at the following link: [https://github.com/awslabs/open-data-docs/tree/main/docs/noaa/noaa-goes16](https://github.com/awslabs/open-data-docs/tree/main/docs/noaa/noaa-goes16)
92
+ A comprehensive list of available GOES products can be found at the following link: [https://console.cloud.google.com/storage/browser/gcp-public-data-goes-16](https://console.cloud.google.com/storage/browser/gcp-public-data-goes-16)
93
93
 
94
94
  ### Examples
95
95
 
@@ -97,14 +97,14 @@ A comprehensive list of available GOES products can be found at the following li
97
97
  In the example below, the command downloads the 3 most recent files from the GOES-16 satellite for the product ABI-L2-CMIPF. It focuses on the variable CMI (Cloud and Moisture Imagery) from channel 13, which is commonly used for infrared observations. The downloaded files are saved to the specified output directory output/.
98
98
 
99
99
  ```bash
100
- goesgcp --satellite goes-16 --product ABI-L2-CMIPF --var_name CMI --channel 13 --recent 3 --output "output/"
100
+ goesgcp --satellite goes-16 --recent 3 --product ABI-L2-CMIPF --var_name CMI --channel 13 --output "output/"
101
101
  ```
102
102
 
103
103
  #### Download Data for a Specific Time Range
104
- This command retrieves GOES-16 satellite data for the product ABI-L2-CMIPF within the date range 2022-12-15 00:00:00 to 2022-12-20 10:00:00, focusing on hours 5:00 and 6:00 AM. The data is cropped to the geographic bounds of -35° to 5° latitude and -80° to -30° longitude, reprojected with a resolution of 0.045 degrees, and saved in a by_date format for easy organization.
104
+ This command retrieves GOES-16 satellite data for the product ABI-L2-CMIPF within the date range 2022-12-15 00:00:00 to 2022-12-15 12:00:00, focusing on hours 5:00 and 6:00 AM. The data is cropped to the geographic bounds of -35° to 5° latitude and -80° to -30° longitude, reprojected with a resolution of 0.045 degrees, and saved in a by_date format for easy organization.
105
105
 
106
106
  ```bash
107
- goesgcp --satellite goes-16 --product ABI-L2-CMIPF --start '2022-12-15 00:00:00' --end '2022-12-20 10:00:00' --bt_hour 5 6 --save_format by_date --resolution 0.045 --lat_min -35 --lat_max 5 --lon_min -80 --lon_max -30
107
+ goesgcp --satellite goes-16 --product ABI-L2-CMIPF --start '2022-12-15 00:00:00' --end '2022-12-15 12:00:00' --bt_hour 5 6 --save_format by_date --resolution 0.045 --lat_min -35 --lat_max 5 --lon_min -80 --lon_max -30
108
108
  ```
109
109
 
110
110
  ### Contributing
@@ -7,18 +7,18 @@
7
7
  <!-- badges: end -->
8
8
 
9
9
 
10
- `goesgcp` is a Python utility designed for downloading and reprojecting GOES-R satellite data. This script leverages the `google.cloud` library for accessing data from the Google Cloud Platform (GCP) and `pyproj` for reprojecting data to EPSG:4326, as well as cropping it to a user-defined bounding box.
10
+ `goesgcp` is a Python utility designed for downloading and reprojecting GOES-R satellite data. This script leverages the `google.cloud` library for accessing data from the Google Cloud Platform (GCP) and `rioxarray` for reprojecting data to EPSG:4326 (rectangular grid), as well cropping it to a user-defined bounding box.
11
11
 
12
12
  ## Features
13
13
 
14
- - **Download GOES-R satellite data**: Supports GOES-16 and GOES-17.
14
+ - **Download GOES-R satellite data**: Supports GOES-16 and GOES-18.
15
15
  - **Reprojection and cropping**: Reprojects data to EPSG:4326 and crops to a specified bounding box.
16
16
  - **Flexible command-line interface**: Customize download options, variables, channels, time range, and output format.
17
17
  - **Efficient processing**: Handles large datasets with optimized performance.
18
18
 
19
19
  ## Installation
20
20
 
21
- Install the necessary dependencies via `pip`:
21
+ Install the package via `pip`:
22
22
 
23
23
  ```bash
24
24
  pip install goesgcp
@@ -27,7 +27,7 @@ pip install goesgcp
27
27
 
28
28
  ## Usage
29
29
 
30
- ### Command-Line Arguments
30
+ ### Available Command-Line Arguments
31
31
 
32
32
  The script uses the `argparse` module for handling command-line arguments. Below are the available options:
33
33
 
@@ -55,7 +55,7 @@ goesgcp [OPTIONS]
55
55
  | `--save_format` | Format for saving output files (default: `by_date`). |
56
56
 
57
57
  #### Available GOES Products
58
- A comprehensive list of available GOES products can be found at the following link: [https://github.com/awslabs/open-data-docs/tree/main/docs/noaa/noaa-goes16](https://github.com/awslabs/open-data-docs/tree/main/docs/noaa/noaa-goes16)
58
+ A comprehensive list of available GOES products can be found at the following link: [https://console.cloud.google.com/storage/browser/gcp-public-data-goes-16](https://console.cloud.google.com/storage/browser/gcp-public-data-goes-16)
59
59
 
60
60
  ### Examples
61
61
 
@@ -63,14 +63,14 @@ A comprehensive list of available GOES products can be found at the following li
63
63
  In the example below, the command downloads the 3 most recent files from the GOES-16 satellite for the product ABI-L2-CMIPF. It focuses on the variable CMI (Cloud and Moisture Imagery) from channel 13, which is commonly used for infrared observations. The downloaded files are saved to the specified output directory output/.
64
64
 
65
65
  ```bash
66
- goesgcp --satellite goes-16 --product ABI-L2-CMIPF --var_name CMI --channel 13 --recent 3 --output "output/"
66
+ goesgcp --satellite goes-16 --recent 3 --product ABI-L2-CMIPF --var_name CMI --channel 13 --output "output/"
67
67
  ```
68
68
 
69
69
  #### Download Data for a Specific Time Range
70
- This command retrieves GOES-16 satellite data for the product ABI-L2-CMIPF within the date range 2022-12-15 00:00:00 to 2022-12-20 10:00:00, focusing on hours 5:00 and 6:00 AM. The data is cropped to the geographic bounds of -35° to 5° latitude and -80° to -30° longitude, reprojected with a resolution of 0.045 degrees, and saved in a by_date format for easy organization.
70
+ This command retrieves GOES-16 satellite data for the product ABI-L2-CMIPF within the date range 2022-12-15 00:00:00 to 2022-12-15 12:00:00, focusing on hours 5:00 and 6:00 AM. The data is cropped to the geographic bounds of -35° to 5° latitude and -80° to -30° longitude, reprojected with a resolution of 0.045 degrees, and saved in a by_date format for easy organization.
71
71
 
72
72
  ```bash
73
- goesgcp --satellite goes-16 --product ABI-L2-CMIPF --start '2022-12-15 00:00:00' --end '2022-12-20 10:00:00' --bt_hour 5 6 --save_format by_date --resolution 0.045 --lat_min -35 --lat_max 5 --lon_min -80 --lon_max -30
73
+ goesgcp --satellite goes-16 --product ABI-L2-CMIPF --start '2022-12-15 00:00:00' --end '2022-12-15 12:00:00' --bt_hour 5 6 --save_format by_date --resolution 0.045 --lat_min -35 --lat_max 5 --lon_min -80 --lon_max -30
74
74
  ```
75
75
 
76
76
  ### Contributing
@@ -13,6 +13,9 @@ from datetime import datetime, timedelta, timezone
13
13
  from pyproj import CRS, Transformer
14
14
  from google.api_core.exceptions import GoogleAPIError
15
15
 
16
+ import warnings
17
+ warnings.filterwarnings('ignore')
18
+
16
19
 
17
20
  def list_blobs(connection, bucket_name, prefix):
18
21
  """
@@ -31,7 +34,7 @@ def get_directory_prefix(year, julian_day, hour):
31
34
 
32
35
 
33
36
  def get_files_period(connection, bucket_name, base_prefix, pattern,
34
- start, end, bt_hour=[0, 23], bt_min=[0, 60], freq='10 min'):
37
+ start, end, bt_hour=[], bt_min=[], freq=None):
35
38
  """
36
39
  Fetches files from a GCP bucket within a specified time period and returns them as a DataFrame.
37
40
 
@@ -54,11 +57,11 @@ def get_files_period(connection, bucket_name, base_prefix, pattern,
54
57
  files_metadata = []
55
58
 
56
59
  # Generate the list of dates from start to end
57
- current_time = start
58
- while current_time < end:
59
- year = current_time.year
60
- julian_day = str(current_time.timetuple().tm_yday).zfill(3) # Julian day
61
- hour = current_time.hour
60
+ temp = start
61
+ while temp <= end:
62
+ year = temp.year
63
+ julian_day = str(temp.timetuple().tm_yday).zfill(3) # Julian day
64
+ hour = temp.hour
62
65
 
63
66
  # Generate the directory prefix
64
67
  prefix = f"{base_prefix}/{get_directory_prefix(year, julian_day, hour)}"
@@ -71,38 +74,42 @@ def get_files_period(connection, bucket_name, base_prefix, pattern,
71
74
  if pattern in blob.name:
72
75
  files_metadata.append({
73
76
  'file_name': blob.name,
74
- 'last_modified': blob.updated
75
77
  })
76
78
 
77
79
  # Move to the next hour
78
- current_time += timedelta(hours=1)
80
+ temp += timedelta(hours=1)
79
81
 
80
82
  # Create a DataFrame from the list of files
81
83
  df = pd.DataFrame(files_metadata)
82
84
 
83
85
  if df.empty:
84
- print("No files found matching the pattern.")
85
- return pd.DataFrame()
86
+ print("No files found matching the pattern and time range.")
87
+ print(prefix)
88
+ sys.exit(1)
86
89
 
90
+ # Transform file_name to datetime
91
+ df['last_modified'] = pd.to_datetime(df['file_name'].str.extract(r'(\d{4}\d{3}\d{2}\d{2})').squeeze(), format='%Y%j%H%M')
92
+
87
93
  # Ensure 'last_modified' is in the correct datetime format without timezone
88
- df['last_modified'] = pd.to_datetime(df['last_modified']).dt.tz_localize(None)
89
- start = pd.to_datetime(start).tz_localize(None)
90
- end = pd.to_datetime(end).tz_localize(None)
91
-
92
- # Filter the DataFrame based on the date range
93
- df = df[(df['last_modified'] >= start) & (df['last_modified'] < end)]
94
+ df['last_modified'] = pd.to_datetime(df['last_modified']).dt.tz_localize('UTC')
95
+
96
+ # Filter the DataFrame based on the date range (inclusive)
97
+ df = df[(df['last_modified'] >= start) & (df['last_modified'] <= end)]
94
98
 
95
99
  # Filter the DataFrame based on the hour range
96
- df['hour'] = df['last_modified'].dt.hour
97
- df = df[(df['hour'] >= bt_hour[0]) & (df['hour'] <= bt_hour[1])]
100
+ if len(bt_hour) > 1:
101
+ df['hour'] = df['last_modified'].dt.hour
102
+ df = df[(df['hour'] >= bt_hour[0]) & (df['hour'] <= bt_hour[1])]
98
103
 
99
104
  # Filter the DataFrame based on the minute range
100
- df['minute'] = df['last_modified'].dt.minute
101
- df = df[(df['minute'] >= bt_min[0]) & (df['minute'] <= bt_min[1])]
105
+ if len(bt_min) > 1:
106
+ df['minute'] = df['last_modified'].dt.minute
107
+ df = df[(df['minute'] >= bt_min[0]) & (df['minute'] <= bt_min[1])]
102
108
 
103
109
  # Filter the DataFrame based on the frequency
104
- df['freq'] = df['last_modified'].dt.floor(freq)
105
- df = df.groupby('freq').first().reset_index()
110
+ if freq is not None:
111
+ df['freq'] = df['last_modified'].dt.floor(freq)
112
+ df = df.groupby('freq').first().reset_index()
106
113
 
107
114
  return df['file_name'].tolist()
108
115
 
@@ -135,7 +142,7 @@ def get_recent_files(connection, bucket_name, base_prefix, pattern, min_files):
135
142
 
136
143
  # Filter blobs based on the pattern
137
144
  for blob in blobs:
138
- if pattern in blob.name: # You can use "re" here for more complex patterns
145
+ if pattern in blob.name:
139
146
  files.append((blob.name, blob.updated))
140
147
 
141
148
  # Go back one hour
@@ -153,13 +160,22 @@ def crop_reproject(args):
153
160
  Crops and reprojects a GOES-16 file to EPSG:4326.
154
161
  """
155
162
 
156
- file, output = args
163
+ file, output, var_name, lat_min, lat_max, lon_min, lon_max, resolution, save_format = args
157
164
 
158
165
  # Open the file
159
166
  ds = xr.open_dataset(file, engine="netcdf4")
160
167
 
168
+ if var_name is None:
169
+ # Get all variables are 2D
170
+ var_names = [var for var in ds.data_vars if len(ds[var].dims) == 2]
171
+
172
+ # Remove DQF variables
173
+ var_names = [var for var in var_names if 'DQF' not in var]
174
+ else:
175
+ var_names = [var_name]
176
+
161
177
  # Select only var_name and goes_imager_projection
162
- ds = ds[[var_name, "goes_imager_projection"]]
178
+ ds = ds[var_names + ["goes_imager_projection"]]
163
179
 
164
180
  # Get projection
165
181
  sat_height = ds["goes_imager_projection"].attrs["perspective_point_height"]
@@ -208,24 +224,27 @@ def crop_reproject(args):
208
224
  except:
209
225
  pass
210
226
 
211
- # Reproject to EPSG:4326
212
- ds = ds.rio.reproject("EPSG:4326", resolution=resolution)
213
-
214
- # Rename lat/lon coordinates
215
- ds = ds.rename({"x": "lon", "y": "lat"})
227
+ try:
228
+ # Reproject to EPSG:4326
229
+ ds = ds.rio.reproject("EPSG:4326", resolution=resolution)
216
230
 
217
- # Add resolution to attributes
218
- ds[var_name].attrs['resolution'] = "x={:.2f} y={:.2f} degree".format(resolution, resolution)
231
+ # Rename lat/lon coordinates
232
+ ds = ds.rename({"x": "lon", "y": "lat"})
219
233
 
220
- # Crop using lat/lon coordinates, in parallel
221
- ds = ds.rio.clip_box(minx=lon_min, miny=lat_min, maxx=lon_max, maxy=lat_max)
234
+ # Add resolution to attributes
235
+ for var in var_names:
236
+ ds[var].attrs['resolution'] = "x={:.2f} y={:.2f} degree".format(resolution, resolution)
237
+ ds[var].attrs['comments'] = 'Cropped and reprojected to EPSG:4326 by goesgcp'
222
238
 
223
- # Add comments
224
- ds[var_name].attrs['comments'] = 'Cropped and reprojected to EPSG:4326 by goesgcp'
239
+ # Crop using lat/lon coordinates, in parallel
240
+ ds = ds.rio.clip_box(minx=lon_min, miny=lat_min, maxx=lon_max, maxy=lat_max)
225
241
 
226
- # Add global metadata comments
227
- ds.attrs['comments'] = "Data processed by goesgcp, author: Helvecio B. L. Neto (helvecioblneto@gmail.com)"
228
-
242
+ # Add global metadata comments
243
+ ds.attrs['comments'] = "Data processed by goesgcp, author: Helvecio B. L. Neto (helvecioblneto@gmail.com)"
244
+ except Exception as e:
245
+ print(f"Error processing file {file}: {e}")
246
+ pass
247
+
229
248
  if save_format == 'by_date':
230
249
  file_datetime = datetime.strptime(ds.time_coverage_start,
231
250
  "%Y-%m-%dT%H:%M:%S.%fZ")
@@ -241,28 +260,27 @@ def crop_reproject(args):
241
260
  output_directory = f"{output}{year}/{julian_day}/"
242
261
  else:
243
262
  output_directory = output
263
+
244
264
 
245
265
  # Create the output directory
246
266
  pathlib.Path(output_directory).mkdir(parents=True, exist_ok=True)
247
267
 
248
268
  # Save the file
249
269
  output_file = f"{output_directory}{file.split('/')[-1]}"
250
- ds.to_netcdf(output_file, mode='w', format='NETCDF4_CLASSIC')
270
+ ds.to_netcdf(output_file, mode='w')
251
271
 
252
- # Fechar o dataset
253
272
  ds.close()
254
- return
255
273
 
256
274
 
257
275
  def process_file(args):
258
- """ Downloads and processes a file in parallel. """
259
-
260
- bucket_name, blob_name, local_path = args
276
+ """
277
+ Downloads and processes a GOES-16 file.
278
+ """
279
+
280
+ bucket_name, blob_name, local_path, output_path, var_name, lat_min, lat_max, lon_min, lon_max, resolution, \
281
+ save_format, retries = args
261
282
 
262
- # Download options
263
- retries = 5
264
283
  attempt = 0
265
-
266
284
  while attempt < retries:
267
285
  try:
268
286
  # Connect to the bucket
@@ -282,31 +300,28 @@ def process_file(args):
282
300
  log_file.write(f"Failed to download {blob_name} after {retries} attempts. Error: {e}\n")
283
301
 
284
302
  # Crop the file
285
- crop_reproject((local_path, output_path))
303
+ crop_reproject((local_path, output_path, var_name, lat_min, lat_max, lon_min, lon_max, resolution, save_format))
286
304
 
287
305
  # Remove the local file
288
306
  pathlib.Path(local_path).unlink()
289
307
 
308
+ # Create connection
309
+ storage_client = storage.Client.create_anonymous_client()
290
310
 
291
311
  def main():
292
312
  ''' Main function to download and process GOES-16 files. '''
293
313
 
294
-
295
- global output_path, var_name, \
296
- lat_min, lat_max, lon_min, lon_max, \
297
- max_attempts, parallel, recent, resolution, storage_client, \
298
- satellite, product, op_mode, channel, save_format
299
-
300
314
  epilog = """
301
315
  Example usage:
302
316
 
303
- - To download recent 3 files from the GOES-16 satellite for the ABI-L2-CMIPF product:
317
+ - To download recent 3 files from the GOES-16 satellite for the ABI-L2-CMIPF product,
318
+ change resolution to 0.045, and crop the files between latitudes -35 and 5 and longitudes -80 and -30:
304
319
 
305
- goesgcp --satellite goes16 --product ABI-L2-CMIP --recent 3
320
+ goesgcp --satellite goes-16 --product ABI-L2-CMIPF --recent 3 --resolution 0.045 --lat_min -35 --lat_max 5 --lon_min -80 --lon_max -30
306
321
 
307
322
  - To download files from the GOES-16 satellite for the ABI-L2-CMIPF product between 2022-12-15 and 2022-12-20:
308
323
 
309
- goesgcp --start '2022-12-15 00:00:00' --end '2022-12-20 10:00:00' --bt_hour 5 6 --save_format by_date --resolution 0.045 --lat_min -35 --lat_max 5 --lon_min -80 --lon_max -30
324
+ goesgcp --satellite goes-16 --product ABI-L2-CMIPF --start '2022-12-15 09:00:00' --end '2022-12-15 09:50:00' --resolution 0.045 --lat_min -35 --lat_max 5 --lon_min -80 --lon_max -30
310
325
 
311
326
  """
312
327
 
@@ -329,16 +344,16 @@ def main():
329
344
  ]
330
345
 
331
346
  # Set arguments
332
- parser = argparse.ArgumentParser(description='Converts GOES-16 L2 data to netCDF',
347
+ parser = argparse.ArgumentParser(description='Download and process GOES Satellite data files from GCP.',
333
348
  epilog=epilog,
334
349
  formatter_class=argparse.RawDescriptionHelpFormatter)
335
350
 
336
351
  # Satellite and product settings
337
352
  parser.add_argument('--satellite', type=str, default='goes-16', choices=['goes-16', 'goes-18'], help='Name of the satellite (e.g., goes16)')
338
- parser.add_argument('--product', type=str, default='ABI-L2-CMIP', help='Name of the satellite product', choices=product_names)
339
- parser.add_argument('--var_name', type=str, default='CMI', help='Variable name to extract (e.g., CMI)')
353
+ parser.add_argument('--product', type=str, default='ABI-L2-CMIPF', help='Name of the satellite product', choices=product_names)
354
+ parser.add_argument('--var_name', type=str, default=None, help='Variable name to extract (e.g., CMI)')
340
355
  parser.add_argument('--channel', type=int, default=13, help='Channel to use (e.g., 13)')
341
- parser.add_argument('--op_mode', type=str, default='M6C', help='Operational mode to use (e.g., M6C)')
356
+ parser.add_argument('--op_mode', type=str, default='M6', help='Operational mode to use (e.g., M6C)')
342
357
 
343
358
  # Recent files settings
344
359
  parser.add_argument('--recent', type=int, help='Number of recent files to download (e.g., 3)')
@@ -378,7 +393,7 @@ def main():
378
393
  satellite = args.satellite
379
394
  product = args.product
380
395
  op_mode = args.op_mode
381
- channel = str(args.channel).zfill(2)
396
+ channel = args.channel
382
397
  var_name = args.var_name
383
398
  lat_min = args.lat_min
384
399
  lat_max = args.lat_max
@@ -407,9 +422,6 @@ def main():
407
422
  # Create output directory
408
423
  pathlib.Path(output_path).mkdir(parents=True, exist_ok=True)
409
424
 
410
- # Create connection
411
- storage_client = storage.Client.create_anonymous_client()
412
-
413
425
  # Check if the bucket exists
414
426
  try:
415
427
  storage_client.get_bucket(bucket_name)
@@ -417,6 +429,13 @@ def main():
417
429
  print(f"Bucket {bucket_name} not found. Exiting...")
418
430
  sys.exit(1)
419
431
 
432
+ # Check if the channel exists
433
+ if not channel:
434
+ channel = ''
435
+ else:
436
+ channel = str(channel).zfill(2)
437
+ channel = f"C{channel}"
438
+
420
439
  # Set pattern for the files
421
440
  pattern = "OR_"+product+"-"+op_mode+channel+"_G" + satellite[-2:]
422
441
 
@@ -445,7 +464,9 @@ def main():
445
464
 
446
465
  if parallel: # Run in parallel
447
466
  # Create a list of tasks
448
- tasks = [(bucket_name, file, f"tmp/{file.split('/')[-1]}") for file in files_list]
467
+ tasks = [(bucket_name, file, f"tmp/{file.split('/')[-1]}", output_path, var_name,
468
+ lat_min, lat_max, lon_min, lon_max, resolution,
469
+ save_format, max_attempts) for file in files_list]
449
470
 
450
471
  # Download files in parallel
451
472
  with Pool(processes=args.processes) as pool:
@@ -455,11 +476,12 @@ def main():
455
476
  else: # Run in serial
456
477
  for file in files_list:
457
478
  local_path = f"tmp/{file.split('/')[-1]}"
458
- process_file((bucket_name, file, local_path))
479
+ process_file((bucket_name, file, local_path, output_path, var_name,
480
+ lat_min, lat_max, lon_min, lon_max, resolution,
481
+ save_format, max_attempts))
459
482
  loading_bar.update(1)
460
483
  loading_bar.close()
461
484
 
462
- # Remove temporary directory
463
485
  shutil.rmtree('tmp/')
464
486
 
465
487
  if __name__ == '__main__':
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.2
2
2
  Name: goesgcp
3
- Version: 2.0.1
3
+ Version: 2.0.3
4
4
  Summary: A package to download and process GOES-16/17 data
5
5
  Home-page: https://github.com/helvecioneto/goesgcp
6
6
  Author: Helvecio B. L. Neto
@@ -41,18 +41,18 @@ Dynamic: summary
41
41
  <!-- badges: end -->
42
42
 
43
43
 
44
- `goesgcp` is a Python utility designed for downloading and reprojecting GOES-R satellite data. This script leverages the `google.cloud` library for accessing data from the Google Cloud Platform (GCP) and `pyproj` for reprojecting data to EPSG:4326, as well as cropping it to a user-defined bounding box.
44
+ `goesgcp` is a Python utility designed for downloading and reprojecting GOES-R satellite data. This script leverages the `google.cloud` library for accessing data from the Google Cloud Platform (GCP) and `rioxarray` for reprojecting data to EPSG:4326 (rectangular grid), as well cropping it to a user-defined bounding box.
45
45
 
46
46
  ## Features
47
47
 
48
- - **Download GOES-R satellite data**: Supports GOES-16 and GOES-17.
48
+ - **Download GOES-R satellite data**: Supports GOES-16 and GOES-18.
49
49
  - **Reprojection and cropping**: Reprojects data to EPSG:4326 and crops to a specified bounding box.
50
50
  - **Flexible command-line interface**: Customize download options, variables, channels, time range, and output format.
51
51
  - **Efficient processing**: Handles large datasets with optimized performance.
52
52
 
53
53
  ## Installation
54
54
 
55
- Install the necessary dependencies via `pip`:
55
+ Install the package via `pip`:
56
56
 
57
57
  ```bash
58
58
  pip install goesgcp
@@ -61,7 +61,7 @@ pip install goesgcp
61
61
 
62
62
  ## Usage
63
63
 
64
- ### Command-Line Arguments
64
+ ### Available Command-Line Arguments
65
65
 
66
66
  The script uses the `argparse` module for handling command-line arguments. Below are the available options:
67
67
 
@@ -89,7 +89,7 @@ goesgcp [OPTIONS]
89
89
  | `--save_format` | Format for saving output files (default: `by_date`). |
90
90
 
91
91
  #### Available GOES Products
92
- A comprehensive list of available GOES products can be found at the following link: [https://github.com/awslabs/open-data-docs/tree/main/docs/noaa/noaa-goes16](https://github.com/awslabs/open-data-docs/tree/main/docs/noaa/noaa-goes16)
92
+ A comprehensive list of available GOES products can be found at the following link: [https://console.cloud.google.com/storage/browser/gcp-public-data-goes-16](https://console.cloud.google.com/storage/browser/gcp-public-data-goes-16)
93
93
 
94
94
  ### Examples
95
95
 
@@ -97,14 +97,14 @@ A comprehensive list of available GOES products can be found at the following li
97
97
  In the example below, the command downloads the 3 most recent files from the GOES-16 satellite for the product ABI-L2-CMIPF. It focuses on the variable CMI (Cloud and Moisture Imagery) from channel 13, which is commonly used for infrared observations. The downloaded files are saved to the specified output directory output/.
98
98
 
99
99
  ```bash
100
- goesgcp --satellite goes-16 --product ABI-L2-CMIPF --var_name CMI --channel 13 --recent 3 --output "output/"
100
+ goesgcp --satellite goes-16 --recent 3 --product ABI-L2-CMIPF --var_name CMI --channel 13 --output "output/"
101
101
  ```
102
102
 
103
103
  #### Download Data for a Specific Time Range
104
- This command retrieves GOES-16 satellite data for the product ABI-L2-CMIPF within the date range 2022-12-15 00:00:00 to 2022-12-20 10:00:00, focusing on hours 5:00 and 6:00 AM. The data is cropped to the geographic bounds of -35° to 5° latitude and -80° to -30° longitude, reprojected with a resolution of 0.045 degrees, and saved in a by_date format for easy organization.
104
+ This command retrieves GOES-16 satellite data for the product ABI-L2-CMIPF within the date range 2022-12-15 00:00:00 to 2022-12-15 12:00:00, focusing on hours 5:00 and 6:00 AM. The data is cropped to the geographic bounds of -35° to 5° latitude and -80° to -30° longitude, reprojected with a resolution of 0.045 degrees, and saved in a by_date format for easy organization.
105
105
 
106
106
  ```bash
107
- goesgcp --satellite goes-16 --product ABI-L2-CMIPF --start '2022-12-15 00:00:00' --end '2022-12-20 10:00:00' --bt_hour 5 6 --save_format by_date --resolution 0.045 --lat_min -35 --lat_max 5 --lon_min -80 --lon_max -30
107
+ goesgcp --satellite goes-16 --product ABI-L2-CMIPF --start '2022-12-15 00:00:00' --end '2022-12-15 12:00:00' --bt_hour 5 6 --save_format by_date --resolution 0.045 --lat_min -35 --lat_max 5 --lon_min -80 --lon_max -30
108
108
  ```
109
109
 
110
110
  ### Contributing
@@ -13,7 +13,7 @@ with open('requirements.txt') as f:
13
13
 
14
14
  setup(
15
15
  name="goesgcp",
16
- version='2.0.1',
16
+ version='2.0.3',
17
17
  author="Helvecio B. L. Neto",
18
18
  author_email="helvecioblneto@gmail.com",
19
19
  description="A package to download and process GOES-16/17 data",
File without changes
File without changes
File without changes