nc2cog 0.1.3__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- nc2cog-0.1.3/LICENSE +21 -0
- nc2cog-0.1.3/PKG-INFO +360 -0
- nc2cog-0.1.3/README.md +331 -0
- nc2cog-0.1.3/pyproject.toml +45 -0
- nc2cog-0.1.3/setup.cfg +4 -0
- nc2cog-0.1.3/setup.py +22 -0
- nc2cog-0.1.3/src/nc2cog/__init__.py +3 -0
- nc2cog-0.1.3/src/nc2cog/__version__.py +8 -0
- nc2cog-0.1.3/src/nc2cog/analyzer.py +185 -0
- nc2cog-0.1.3/src/nc2cog/cli.py +266 -0
- nc2cog-0.1.3/src/nc2cog/config.py +158 -0
- nc2cog-0.1.3/src/nc2cog/discovery.py +109 -0
- nc2cog-0.1.3/src/nc2cog/errors.py +25 -0
- nc2cog-0.1.3/src/nc2cog/logger.py +48 -0
- nc2cog-0.1.3/src/nc2cog/metadata.py +254 -0
- nc2cog-0.1.3/src/nc2cog/processor.py +534 -0
- nc2cog-0.1.3/src/nc2cog.egg-info/PKG-INFO +360 -0
- nc2cog-0.1.3/src/nc2cog.egg-info/SOURCES.txt +28 -0
- nc2cog-0.1.3/src/nc2cog.egg-info/dependency_links.txt +1 -0
- nc2cog-0.1.3/src/nc2cog.egg-info/entry_points.txt +2 -0
- nc2cog-0.1.3/src/nc2cog.egg-info/requires.txt +5 -0
- nc2cog-0.1.3/src/nc2cog.egg-info/top_level.txt +1 -0
- nc2cog-0.1.3/tests/test_cli_params.py +384 -0
- nc2cog-0.1.3/tests/test_config.py +113 -0
- nc2cog-0.1.3/tests/test_config_projection.py +111 -0
- nc2cog-0.1.3/tests/test_discovery.py +157 -0
- nc2cog-0.1.3/tests/test_extended_cli.py +563 -0
- nc2cog-0.1.3/tests/test_gdal_warnings_fix.py +251 -0
- nc2cog-0.1.3/tests/test_metadata.py +84 -0
- nc2cog-0.1.3/tests/test_projection.py +46 -0
nc2cog-0.1.3/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Sam Chen
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
nc2cog-0.1.3/PKG-INFO
ADDED
|
@@ -0,0 +1,360 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: nc2cog
|
|
3
|
+
Version: 0.1.3
|
|
4
|
+
Summary: Convert netCDF files to Cloud-Optimized GeoTIFF format with advanced compression
|
|
5
|
+
Author: Sam Chen
|
|
6
|
+
License: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/cyberpsyche/nc2cog
|
|
8
|
+
Project-URL: Repository, https://github.com/cyberpsyche/nc2cog
|
|
9
|
+
Keywords: netcdf,cog,geotiff,gdal,geospatial
|
|
10
|
+
Classifier: Development Status :: 4 - Beta
|
|
11
|
+
Classifier: Intended Audience :: Science/Research
|
|
12
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
13
|
+
Classifier: Programming Language :: Python :: 3
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
19
|
+
Classifier: Topic :: Scientific/Engineering :: GIS
|
|
20
|
+
Requires-Python: >=3.9
|
|
21
|
+
Description-Content-Type: text/markdown
|
|
22
|
+
License-File: LICENSE
|
|
23
|
+
Requires-Dist: GDAL>=3.8.0
|
|
24
|
+
Requires-Dist: click>=8.1.0
|
|
25
|
+
Requires-Dist: PyYAML>=6.0
|
|
26
|
+
Requires-Dist: netCDF4>=1.6.0
|
|
27
|
+
Requires-Dist: numpy>=1.21.0
|
|
28
|
+
Dynamic: license-file
|
|
29
|
+
|
|
30
|
+
# nc2cog - netCDF to Cloud-Optimized GeoTIFF Converter
|
|
31
|
+
|
|
32
|
+
[δΈζ](README_zh.md)
|
|
33
|
+
|
|
34
|
+
Convert netCDF files to Cloud-Optimized GeoTIFF format with advanced compression and performance settings.
|
|
35
|
+
|
|
36
|
+
## π Features
|
|
37
|
+
|
|
38
|
+
- **Format Conversion**: Convert netCDF files to Cloud-Optimized GeoTIFF (COG)
|
|
39
|
+
- **Batch Processing**: Process entire directories of netCDF files
|
|
40
|
+
- **Advanced Compression**: Support for deflate, lzw, and jpeg compression with configurable levels
|
|
41
|
+
- **Performance Optimization**: Configurable tile and block sizes for optimal performance
|
|
42
|
+
- **Pyramid Structure**: Customizable overview (pyramid) levels for multi-scale access
|
|
43
|
+
- **GDAL Optimized**: Eliminated GDAL warnings and optimized driver-specific parameters
|
|
44
|
+
- **Parallel Processing**: Multi-threaded conversion for faster processing
|
|
45
|
+
- **Resume Capability**: Resume interrupted conversions
|
|
46
|
+
- **Projection Transformation**: Reproject data from source to target coordinate systems during conversion
|
|
47
|
+
- **Flexible Output**: Output to a directory or directly to a specified `.tif` file
|
|
48
|
+
- **Rich Metadata**: 18 metadata fields automatically written to output COG (source, extent, resolution, min/max, unit, etc.)
|
|
49
|
+
- **Clean COG Layout**: Metadata written during conversion pipeline β no post-creation modifications that break COG layout optimization
|
|
50
|
+
|
|
51
|
+
## π Requirements
|
|
52
|
+
|
|
53
|
+
- Python 3.7+
|
|
54
|
+
- GDAL library with Python bindings
|
|
55
|
+
- click library for CLI handling
|
|
56
|
+
- numpy for numerical operations
|
|
57
|
+
|
|
58
|
+
## π οΈ Installation
|
|
59
|
+
|
|
60
|
+
```bash
|
|
61
|
+
# Clone the repository
|
|
62
|
+
git clone <repository-url>
|
|
63
|
+
cd nc2cog
|
|
64
|
+
|
|
65
|
+
# Install dependencies
|
|
66
|
+
pip install -r requirements.txt
|
|
67
|
+
|
|
68
|
+
# Install GDAL
|
|
69
|
+
# On macOS: brew install gdal
|
|
70
|
+
# On Ubuntu: sudo apt-get install gdal-bin libgdal-dev
|
|
71
|
+
# On Windows: Use OSGeo4W installer
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
## π Quick Start
|
|
75
|
+
|
|
76
|
+
### Single File Conversion
|
|
77
|
+
|
|
78
|
+
Convert a single netCDF file to COG TIFF:
|
|
79
|
+
|
|
80
|
+
```bash
|
|
81
|
+
nc2cog input.nc output/
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
Specify the output filename:
|
|
85
|
+
|
|
86
|
+
```bash
|
|
87
|
+
nc2cog input.nc output/my_custom_name.tif
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
### Batch Conversion
|
|
91
|
+
|
|
92
|
+
Convert all netCDF files in a directory:
|
|
93
|
+
|
|
94
|
+
```bash
|
|
95
|
+
nc2cog input_dir/ output/
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
### Multi-dimensional NetCDF Files
|
|
99
|
+
|
|
100
|
+
For netCDF files containing multiple variables and time dimensions (e.g., `PRE(time, lat, lon)`, `REF(time, lat, lon)`), the tool automatically detects the structure and converts each variable to a separate multi-band COG file, with time steps as bands.
|
|
101
|
+
|
|
102
|
+
```bash
|
|
103
|
+
# Auto-detect and convert all variables
|
|
104
|
+
nc2cog MPF_V4_20251113144500.nc output/
|
|
105
|
+
# Produces: output/PRE.tif (N bands), output/REF.tif (N bands)
|
|
106
|
+
|
|
107
|
+
# Convert only specific variables to a directory
|
|
108
|
+
nc2cog --variables PRE MPF_V4_20251113144500.nc output/
|
|
109
|
+
|
|
110
|
+
# Convert a single variable to a specific output file
|
|
111
|
+
nc2cog --variables PRE MPF_V4_20251113144500.nc output/MPF_v4_PRE.tif
|
|
112
|
+
# Produces: output/MPF_v4_PRE.tif (N bands, each band is a time step)
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
### Advanced Usage
|
|
116
|
+
|
|
117
|
+
With custom compression and performance settings:
|
|
118
|
+
|
|
119
|
+
```bash
|
|
120
|
+
nc2cog input.nc output/ \
|
|
121
|
+
--compression deflate \
|
|
122
|
+
--zlevel 9 \
|
|
123
|
+
--tile-size 1024 \
|
|
124
|
+
--block-size 512 \
|
|
125
|
+
--resampling cubic \
|
|
126
|
+
--overview-levels 2,4,8,16,32 \
|
|
127
|
+
--metadata-source "My Satellite Data"
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
With parallel processing:
|
|
131
|
+
|
|
132
|
+
```bash
|
|
133
|
+
nc2cog input_dir/ output/ --threads 4
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
## βοΈ Command Line Options
|
|
137
|
+
|
|
138
|
+
### Compression Options
|
|
139
|
+
- `--compression` [deflate|lzw|jpeg]: Choose compression algorithm
|
|
140
|
+
- `--zlevel` [1-9]: Set compression level for deflate (default: 6)
|
|
141
|
+
|
|
142
|
+
### Performance Options
|
|
143
|
+
- `--tile-size` INTEGER: Tile size for COG (default: 512)
|
|
144
|
+
- `--block-size` INTEGER: Block size for compression (default: 256)
|
|
145
|
+
|
|
146
|
+
### Pyramid/Overview Options
|
|
147
|
+
- `--resampling` [nearest|bilinear|cubic|...]: Resampling method for overviews (default: nearest)
|
|
148
|
+
- `--overview-levels` TEXT: Overview levels (comma-separated, default: 2,4,8,16)
|
|
149
|
+
|
|
150
|
+
### General Options
|
|
151
|
+
- `--overwrite`: Overwrite existing output files
|
|
152
|
+
- `--dry-run`: Show what would be processed without doing it
|
|
153
|
+
- `--verbose`, `-v`: Enable verbose logging
|
|
154
|
+
- `--resume`: Resume from last processed file
|
|
155
|
+
- `--threads` INTEGER: Number of parallel processing threads (default: 1)
|
|
156
|
+
- `--src-proj` TEXT: Source projection in EPSG format (e.g., EPSG:4326)
|
|
157
|
+
- `--dst-proj` TEXT: Target projection in EPSG format (e.g., EPSG:3857)
|
|
158
|
+
- `--variables` TEXT: Variables to convert in multi-dimensional NC files (comma-separated, e.g., `PRE,REF`). If omitted, all data variables are auto-detected.
|
|
159
|
+
- `--metadata-source` TEXT: Custom source name for COG metadata. If omitted, auto-detected from netCDF global attributes (`source`, `platform`, `institution`).
|
|
160
|
+
- `--version`, `-V`: Show version information and exit.
|
|
161
|
+
|
|
162
|
+
## π Output Path Rules
|
|
163
|
+
|
|
164
|
+
The behavior of the `output_path` argument depends on its format:
|
|
165
|
+
|
|
166
|
+
| Output path ends with | Behavior |
|
|
167
|
+
|-----------------------|----------|
|
|
168
|
+
| `.tif` | Output to the specified file directly |
|
|
169
|
+
| `/` or no extension | Output to a directory (one file per variable for multi-dim) |
|
|
170
|
+
|
|
171
|
+
Examples:
|
|
172
|
+
```bash
|
|
173
|
+
# Directory output: creates output/PRE.tif and output/REF.tif
|
|
174
|
+
nc2cog MPF_V4_20251113144500.nc output/
|
|
175
|
+
|
|
176
|
+
# File output: creates the exact file specified
|
|
177
|
+
nc2cog --variables PRE MPF_V4_20251113144500.nc output/MPF_v4_PRE.tif
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
**Note**: File output (`.tif` ending) only works when converting a single variable. If multiple variables are specified with a `.tif` output path, only the first variable will be converted.
|
|
181
|
+
|
|
182
|
+
## π§ Configuration File
|
|
183
|
+
|
|
184
|
+
Create a `config.yaml` for complex setups:
|
|
185
|
+
|
|
186
|
+
```yaml
|
|
187
|
+
# Processing parameters
|
|
188
|
+
compression: "deflate"
|
|
189
|
+
zlevel: 6
|
|
190
|
+
tile_size: [512, 512]
|
|
191
|
+
block_size: [256, 256]
|
|
192
|
+
|
|
193
|
+
# Output options
|
|
194
|
+
overviews:
|
|
195
|
+
resampling: "nearest"
|
|
196
|
+
levels: [2, 4, 8, 16]
|
|
197
|
+
|
|
198
|
+
# Metadata options
|
|
199
|
+
metadata:
|
|
200
|
+
source: "" # Custom source name (auto-detected from netCDF if empty)
|
|
201
|
+
offset: 0.0 # Data offset
|
|
202
|
+
scale: 1.0 # Data scale factor
|
|
203
|
+
unit: "" # Data unit (auto-detected from netCDF if empty)
|
|
204
|
+
|
|
205
|
+
# Processing control
|
|
206
|
+
overwrite: false
|
|
207
|
+
skip_errors: true
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
Use with: `nc2cog --config config.yaml input.nc output/`
|
|
211
|
+
|
|
212
|
+
## π GDAL Optimization
|
|
213
|
+
|
|
214
|
+
This tool eliminates common GDAL warnings by using driver-appropriate parameters:
|
|
215
|
+
|
|
216
|
+
- **GTiff driver**: Uses `BLOCKXSIZE`/`BLOCKYSIZE` instead of `TILEWIDTH`/`TILEHEIGHT`
|
|
217
|
+
- **COG driver**: Uses `BLOCKSIZE` instead of `BLOCKXSIZE`/`BLOCKYSIZE`
|
|
218
|
+
- **Overview handling**: Optimized to avoid `COPY_SRC_OVERVIEWS` conflicts
|
|
219
|
+
- **Clean COG Layout**: Metadata is written to the intermediate GeoTIFF **before** COG conversion. The GDAL COG driver automatically carries metadata from source to output, eliminating the need for post-creation file modifications that would break COG layout optimization (no "IFD has been rewritten" warnings).
|
|
220
|
+
|
|
221
|
+
## π COG Metadata
|
|
222
|
+
|
|
223
|
+
Each output COG file contains 18 metadata fields written during the conversion pipeline:
|
|
224
|
+
|
|
225
|
+
| Field | Description | Example |
|
|
226
|
+
|-------|-------------|---------|
|
|
227
|
+
| `Coordinate System` | CRS with projection info | `WGS84 (EPSG:4326)` |
|
|
228
|
+
| `Band Count` | Number of bands (time steps) | `10` |
|
|
229
|
+
| `Data Type` | Pixel data type | `Float32` |
|
|
230
|
+
| `Resolution` | Pixel resolution in degrees | `0.01000000` |
|
|
231
|
+
| `Extent` | Bounding box (minLon, minLat, maxLon, maxLat) | `97.095, 37.295, 126.105, 53.405` |
|
|
232
|
+
| `Creation Time` | UTC timestamp of conversion | `2026-05-26 12:08:29` |
|
|
233
|
+
| `Source` | Data source name | `hfioi` |
|
|
234
|
+
| `Compression` | Compression algorithm | `DEFLATE` |
|
|
235
|
+
| `startX` / `startY` | Upper-left corner coordinates | `97.095000` / `53.405000` |
|
|
236
|
+
| `endX` / `endY` | Lower-right corner coordinates | `126.105000` / `37.295000` |
|
|
237
|
+
| `min` / `max` | Global data value range | `0.00` / `7.91` |
|
|
238
|
+
| `offset` / `scale` | Linear transformation parameters | `0.0000` / `1.0000` |
|
|
239
|
+
| `unit` | Data unit from netCDF variable | `mm` |
|
|
240
|
+
| `NODATA` | No-data value | `nan` |
|
|
241
|
+
|
|
242
|
+
### Metadata Source Configuration
|
|
243
|
+
|
|
244
|
+
The `Source` field is determined by the following priority:
|
|
245
|
+
|
|
246
|
+
1. **`--metadata-source` CLI parameter** (highest priority) β explicitly set source name
|
|
247
|
+
2. **netCDF global attributes** β auto-detected from `source`, `platform`, or `institution` attributes
|
|
248
|
+
3. **Empty string** (default fallback)
|
|
249
|
+
|
|
250
|
+
```bash
|
|
251
|
+
# Use custom source name
|
|
252
|
+
nc2cog --metadata-source "FY-4 Satellite" input.nc output/
|
|
253
|
+
|
|
254
|
+
# Auto-detect from netCDF attributes (default behavior)
|
|
255
|
+
nc2cog input.nc output/
|
|
256
|
+
```
|
|
257
|
+
|
|
258
|
+
### Viewing Metadata
|
|
259
|
+
|
|
260
|
+
```bash
|
|
261
|
+
# View all metadata fields
|
|
262
|
+
gdalinfo output.tif
|
|
263
|
+
|
|
264
|
+
# View metadata as JSON
|
|
265
|
+
gdalinfo -json output.tif | python -c "
|
|
266
|
+
import json, sys
|
|
267
|
+
d = json.load(sys.stdin)
|
|
268
|
+
m = d.get('metadata', {}).get('', {})
|
|
269
|
+
for k, v in sorted(m.items()):
|
|
270
|
+
print(f'{k}: {v}')
|
|
271
|
+
"
|
|
272
|
+
```
|
|
273
|
+
|
|
274
|
+
## π― Use Cases
|
|
275
|
+
|
|
276
|
+
### Climate/Meteorological Data
|
|
277
|
+
```bash
|
|
278
|
+
nc2cog climate_data.nc output/ \
|
|
279
|
+
--compression deflate \
|
|
280
|
+
--zlevel 9 \
|
|
281
|
+
--resampling cubic \
|
|
282
|
+
--overview-levels 2,4,8,16,32
|
|
283
|
+
```
|
|
284
|
+
|
|
285
|
+
### Oceanographic Data
|
|
286
|
+
```bash
|
|
287
|
+
nc2cog ocean_data.nc output/ \
|
|
288
|
+
--compression lzw \
|
|
289
|
+
--tile-size 1024 \
|
|
290
|
+
--resampling bilinear \
|
|
291
|
+
--overview-levels 2,4,8
|
|
292
|
+
```
|
|
293
|
+
|
|
294
|
+
### Large Dataset Processing
|
|
295
|
+
```bash
|
|
296
|
+
nc2cog large_dataset/ output/ \
|
|
297
|
+
--threads 4 \
|
|
298
|
+
--compression deflate \
|
|
299
|
+
--zlevel 7 \
|
|
300
|
+
--tile-size 512
|
|
301
|
+
```
|
|
302
|
+
|
|
303
|
+
### Projection Transformations
|
|
304
|
+
With the new projection support, you can transform coordinate systems during conversion:
|
|
305
|
+
|
|
306
|
+
Convert with reprojection from WGS84 to Web Mercator:
|
|
307
|
+
```bash
|
|
308
|
+
nc2cog --src-proj EPSG:4326 --dst-proj EPSG:3857 input.nc output/
|
|
309
|
+
```
|
|
310
|
+
|
|
311
|
+
Or specify only target projection (source will be detected automatically):
|
|
312
|
+
```bash
|
|
313
|
+
nc2cog --dst-proj EPSG:3857 input.nc output/
|
|
314
|
+
```
|
|
315
|
+
|
|
316
|
+
Combine with other conversion options:
|
|
317
|
+
```bash
|
|
318
|
+
nc2cog --src-proj EPSG:4326 --dst-proj EPSG:3857 \
|
|
319
|
+
--compression deflate \
|
|
320
|
+
--zlevel 9 \
|
|
321
|
+
--tile-size 1024 \
|
|
322
|
+
input.nc output/
|
|
323
|
+
```
|
|
324
|
+
|
|
325
|
+
## π€ Contributing
|
|
326
|
+
|
|
327
|
+
1. Fork the repository
|
|
328
|
+
2. Create a feature branch
|
|
329
|
+
3. Make your changes
|
|
330
|
+
4. Add tests if applicable
|
|
331
|
+
5. Submit a pull request
|
|
332
|
+
|
|
333
|
+
## π License
|
|
334
|
+
|
|
335
|
+
MIT License
|
|
336
|
+
|
|
337
|
+
## π Issues
|
|
338
|
+
|
|
339
|
+
Report issues on the GitHub repository.
|
|
340
|
+
|
|
341
|
+
## π Documentation
|
|
342
|
+
|
|
343
|
+
For more detailed documentation:
|
|
344
|
+
- [User Guide](docs/user_guide.md)
|
|
345
|
+
- [GDAL Optimization Details](docs/gdal_optimization.md)
|
|
346
|
+
- [Usage Examples](docs/examples.md)
|
|
347
|
+
- [Installation Guide](INSTALLATION.md)
|
|
348
|
+
- [Project Structure](docs/project_structure.md)
|
|
349
|
+
|
|
350
|
+
## βΉοΈ Additional Information
|
|
351
|
+
|
|
352
|
+
### Overview Level Generation
|
|
353
|
+
|
|
354
|
+
When using `--overview-levels`, note that GDAL intelligently determines which overview levels to actually create based on the source image size. For smaller images, GDAL may generate fewer overview levels than specified to avoid creating overly small and potentially useless overviews.
|
|
355
|
+
|
|
356
|
+
For example, with `--overview-levels 2,4,8,16,32` on a 1781x1572 image, GDAL may only generate levels 2 and 4 (890x786 and 445x393 pixels respectively) as the smaller levels would be too small to be useful.
|
|
357
|
+
|
|
358
|
+
---
|
|
359
|
+
|
|
360
|
+
Made with β€οΈ for the geospatial community.
|
nc2cog-0.1.3/README.md
ADDED
|
@@ -0,0 +1,331 @@
|
|
|
1
|
+
# nc2cog - netCDF to Cloud-Optimized GeoTIFF Converter
|
|
2
|
+
|
|
3
|
+
[δΈζ](README_zh.md)
|
|
4
|
+
|
|
5
|
+
Convert netCDF files to Cloud-Optimized GeoTIFF format with advanced compression and performance settings.
|
|
6
|
+
|
|
7
|
+
## π Features
|
|
8
|
+
|
|
9
|
+
- **Format Conversion**: Convert netCDF files to Cloud-Optimized GeoTIFF (COG)
|
|
10
|
+
- **Batch Processing**: Process entire directories of netCDF files
|
|
11
|
+
- **Advanced Compression**: Support for deflate, lzw, and jpeg compression with configurable levels
|
|
12
|
+
- **Performance Optimization**: Configurable tile and block sizes for optimal performance
|
|
13
|
+
- **Pyramid Structure**: Customizable overview (pyramid) levels for multi-scale access
|
|
14
|
+
- **GDAL Optimized**: Eliminated GDAL warnings and optimized driver-specific parameters
|
|
15
|
+
- **Parallel Processing**: Multi-threaded conversion for faster processing
|
|
16
|
+
- **Resume Capability**: Resume interrupted conversions
|
|
17
|
+
- **Projection Transformation**: Reproject data from source to target coordinate systems during conversion
|
|
18
|
+
- **Flexible Output**: Output to a directory or directly to a specified `.tif` file
|
|
19
|
+
- **Rich Metadata**: 18 metadata fields automatically written to output COG (source, extent, resolution, min/max, unit, etc.)
|
|
20
|
+
- **Clean COG Layout**: Metadata written during conversion pipeline β no post-creation modifications that break COG layout optimization
|
|
21
|
+
|
|
22
|
+
## π Requirements
|
|
23
|
+
|
|
24
|
+
- Python 3.7+
|
|
25
|
+
- GDAL library with Python bindings
|
|
26
|
+
- click library for CLI handling
|
|
27
|
+
- numpy for numerical operations
|
|
28
|
+
|
|
29
|
+
## π οΈ Installation
|
|
30
|
+
|
|
31
|
+
```bash
|
|
32
|
+
# Clone the repository
|
|
33
|
+
git clone <repository-url>
|
|
34
|
+
cd nc2cog
|
|
35
|
+
|
|
36
|
+
# Install dependencies
|
|
37
|
+
pip install -r requirements.txt
|
|
38
|
+
|
|
39
|
+
# Install GDAL
|
|
40
|
+
# On macOS: brew install gdal
|
|
41
|
+
# On Ubuntu: sudo apt-get install gdal-bin libgdal-dev
|
|
42
|
+
# On Windows: Use OSGeo4W installer
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
## π Quick Start
|
|
46
|
+
|
|
47
|
+
### Single File Conversion
|
|
48
|
+
|
|
49
|
+
Convert a single netCDF file to COG TIFF:
|
|
50
|
+
|
|
51
|
+
```bash
|
|
52
|
+
nc2cog input.nc output/
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
Specify the output filename:
|
|
56
|
+
|
|
57
|
+
```bash
|
|
58
|
+
nc2cog input.nc output/my_custom_name.tif
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
### Batch Conversion
|
|
62
|
+
|
|
63
|
+
Convert all netCDF files in a directory:
|
|
64
|
+
|
|
65
|
+
```bash
|
|
66
|
+
nc2cog input_dir/ output/
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
### Multi-dimensional NetCDF Files
|
|
70
|
+
|
|
71
|
+
For netCDF files containing multiple variables and time dimensions (e.g., `PRE(time, lat, lon)`, `REF(time, lat, lon)`), the tool automatically detects the structure and converts each variable to a separate multi-band COG file, with time steps as bands.
|
|
72
|
+
|
|
73
|
+
```bash
|
|
74
|
+
# Auto-detect and convert all variables
|
|
75
|
+
nc2cog MPF_V4_20251113144500.nc output/
|
|
76
|
+
# Produces: output/PRE.tif (N bands), output/REF.tif (N bands)
|
|
77
|
+
|
|
78
|
+
# Convert only specific variables to a directory
|
|
79
|
+
nc2cog --variables PRE MPF_V4_20251113144500.nc output/
|
|
80
|
+
|
|
81
|
+
# Convert a single variable to a specific output file
|
|
82
|
+
nc2cog --variables PRE MPF_V4_20251113144500.nc output/MPF_v4_PRE.tif
|
|
83
|
+
# Produces: output/MPF_v4_PRE.tif (N bands, each band is a time step)
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
### Advanced Usage
|
|
87
|
+
|
|
88
|
+
With custom compression and performance settings:
|
|
89
|
+
|
|
90
|
+
```bash
|
|
91
|
+
nc2cog input.nc output/ \
|
|
92
|
+
--compression deflate \
|
|
93
|
+
--zlevel 9 \
|
|
94
|
+
--tile-size 1024 \
|
|
95
|
+
--block-size 512 \
|
|
96
|
+
--resampling cubic \
|
|
97
|
+
--overview-levels 2,4,8,16,32 \
|
|
98
|
+
--metadata-source "My Satellite Data"
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
With parallel processing:
|
|
102
|
+
|
|
103
|
+
```bash
|
|
104
|
+
nc2cog input_dir/ output/ --threads 4
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
## βοΈ Command Line Options
|
|
108
|
+
|
|
109
|
+
### Compression Options
|
|
110
|
+
- `--compression` [deflate|lzw|jpeg]: Choose compression algorithm
|
|
111
|
+
- `--zlevel` [1-9]: Set compression level for deflate (default: 6)
|
|
112
|
+
|
|
113
|
+
### Performance Options
|
|
114
|
+
- `--tile-size` INTEGER: Tile size for COG (default: 512)
|
|
115
|
+
- `--block-size` INTEGER: Block size for compression (default: 256)
|
|
116
|
+
|
|
117
|
+
### Pyramid/Overview Options
|
|
118
|
+
- `--resampling` [nearest|bilinear|cubic|...]: Resampling method for overviews (default: nearest)
|
|
119
|
+
- `--overview-levels` TEXT: Overview levels (comma-separated, default: 2,4,8,16)
|
|
120
|
+
|
|
121
|
+
### General Options
|
|
122
|
+
- `--overwrite`: Overwrite existing output files
|
|
123
|
+
- `--dry-run`: Show what would be processed without doing it
|
|
124
|
+
- `--verbose`, `-v`: Enable verbose logging
|
|
125
|
+
- `--resume`: Resume from last processed file
|
|
126
|
+
- `--threads` INTEGER: Number of parallel processing threads (default: 1)
|
|
127
|
+
- `--src-proj` TEXT: Source projection in EPSG format (e.g., EPSG:4326)
|
|
128
|
+
- `--dst-proj` TEXT: Target projection in EPSG format (e.g., EPSG:3857)
|
|
129
|
+
- `--variables` TEXT: Variables to convert in multi-dimensional NC files (comma-separated, e.g., `PRE,REF`). If omitted, all data variables are auto-detected.
|
|
130
|
+
- `--metadata-source` TEXT: Custom source name for COG metadata. If omitted, auto-detected from netCDF global attributes (`source`, `platform`, `institution`).
|
|
131
|
+
- `--version`, `-V`: Show version information and exit.
|
|
132
|
+
|
|
133
|
+
## π Output Path Rules
|
|
134
|
+
|
|
135
|
+
The behavior of the `output_path` argument depends on its format:
|
|
136
|
+
|
|
137
|
+
| Output path ends with | Behavior |
|
|
138
|
+
|-----------------------|----------|
|
|
139
|
+
| `.tif` | Output to the specified file directly |
|
|
140
|
+
| `/` or no extension | Output to a directory (one file per variable for multi-dim) |
|
|
141
|
+
|
|
142
|
+
Examples:
|
|
143
|
+
```bash
|
|
144
|
+
# Directory output: creates output/PRE.tif and output/REF.tif
|
|
145
|
+
nc2cog MPF_V4_20251113144500.nc output/
|
|
146
|
+
|
|
147
|
+
# File output: creates the exact file specified
|
|
148
|
+
nc2cog --variables PRE MPF_V4_20251113144500.nc output/MPF_v4_PRE.tif
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
**Note**: File output (`.tif` ending) only works when converting a single variable. If multiple variables are specified with a `.tif` output path, only the first variable will be converted.
|
|
152
|
+
|
|
153
|
+
## π§ Configuration File
|
|
154
|
+
|
|
155
|
+
Create a `config.yaml` for complex setups:
|
|
156
|
+
|
|
157
|
+
```yaml
|
|
158
|
+
# Processing parameters
|
|
159
|
+
compression: "deflate"
|
|
160
|
+
zlevel: 6
|
|
161
|
+
tile_size: [512, 512]
|
|
162
|
+
block_size: [256, 256]
|
|
163
|
+
|
|
164
|
+
# Output options
|
|
165
|
+
overviews:
|
|
166
|
+
resampling: "nearest"
|
|
167
|
+
levels: [2, 4, 8, 16]
|
|
168
|
+
|
|
169
|
+
# Metadata options
|
|
170
|
+
metadata:
|
|
171
|
+
source: "" # Custom source name (auto-detected from netCDF if empty)
|
|
172
|
+
offset: 0.0 # Data offset
|
|
173
|
+
scale: 1.0 # Data scale factor
|
|
174
|
+
unit: "" # Data unit (auto-detected from netCDF if empty)
|
|
175
|
+
|
|
176
|
+
# Processing control
|
|
177
|
+
overwrite: false
|
|
178
|
+
skip_errors: true
|
|
179
|
+
```
|
|
180
|
+
|
|
181
|
+
Use with: `nc2cog --config config.yaml input.nc output/`
|
|
182
|
+
|
|
183
|
+
## π GDAL Optimization
|
|
184
|
+
|
|
185
|
+
This tool eliminates common GDAL warnings by using driver-appropriate parameters:
|
|
186
|
+
|
|
187
|
+
- **GTiff driver**: Uses `BLOCKXSIZE`/`BLOCKYSIZE` instead of `TILEWIDTH`/`TILEHEIGHT`
|
|
188
|
+
- **COG driver**: Uses `BLOCKSIZE` instead of `BLOCKXSIZE`/`BLOCKYSIZE`
|
|
189
|
+
- **Overview handling**: Optimized to avoid `COPY_SRC_OVERVIEWS` conflicts
|
|
190
|
+
- **Clean COG Layout**: Metadata is written to the intermediate GeoTIFF **before** COG conversion. The GDAL COG driver automatically carries metadata from source to output, eliminating the need for post-creation file modifications that would break COG layout optimization (no "IFD has been rewritten" warnings).
|
|
191
|
+
|
|
192
|
+
## π COG Metadata
|
|
193
|
+
|
|
194
|
+
Each output COG file contains 18 metadata fields written during the conversion pipeline:
|
|
195
|
+
|
|
196
|
+
| Field | Description | Example |
|
|
197
|
+
|-------|-------------|---------|
|
|
198
|
+
| `Coordinate System` | CRS with projection info | `WGS84 (EPSG:4326)` |
|
|
199
|
+
| `Band Count` | Number of bands (time steps) | `10` |
|
|
200
|
+
| `Data Type` | Pixel data type | `Float32` |
|
|
201
|
+
| `Resolution` | Pixel resolution in degrees | `0.01000000` |
|
|
202
|
+
| `Extent` | Bounding box (minLon, minLat, maxLon, maxLat) | `97.095, 37.295, 126.105, 53.405` |
|
|
203
|
+
| `Creation Time` | UTC timestamp of conversion | `2026-05-26 12:08:29` |
|
|
204
|
+
| `Source` | Data source name | `hfioi` |
|
|
205
|
+
| `Compression` | Compression algorithm | `DEFLATE` |
|
|
206
|
+
| `startX` / `startY` | Upper-left corner coordinates | `97.095000` / `53.405000` |
|
|
207
|
+
| `endX` / `endY` | Lower-right corner coordinates | `126.105000` / `37.295000` |
|
|
208
|
+
| `min` / `max` | Global data value range | `0.00` / `7.91` |
|
|
209
|
+
| `offset` / `scale` | Linear transformation parameters | `0.0000` / `1.0000` |
|
|
210
|
+
| `unit` | Data unit from netCDF variable | `mm` |
|
|
211
|
+
| `NODATA` | No-data value | `nan` |
|
|
212
|
+
|
|
213
|
+
### Metadata Source Configuration
|
|
214
|
+
|
|
215
|
+
The `Source` field is determined by the following priority:
|
|
216
|
+
|
|
217
|
+
1. **`--metadata-source` CLI parameter** (highest priority) β explicitly set source name
|
|
218
|
+
2. **netCDF global attributes** β auto-detected from `source`, `platform`, or `institution` attributes
|
|
219
|
+
3. **Empty string** (default fallback)
|
|
220
|
+
|
|
221
|
+
```bash
|
|
222
|
+
# Use custom source name
|
|
223
|
+
nc2cog --metadata-source "FY-4 Satellite" input.nc output/
|
|
224
|
+
|
|
225
|
+
# Auto-detect from netCDF attributes (default behavior)
|
|
226
|
+
nc2cog input.nc output/
|
|
227
|
+
```
|
|
228
|
+
|
|
229
|
+
### Viewing Metadata
|
|
230
|
+
|
|
231
|
+
```bash
|
|
232
|
+
# View all metadata fields
|
|
233
|
+
gdalinfo output.tif
|
|
234
|
+
|
|
235
|
+
# View metadata as JSON
|
|
236
|
+
gdalinfo -json output.tif | python -c "
|
|
237
|
+
import json, sys
|
|
238
|
+
d = json.load(sys.stdin)
|
|
239
|
+
m = d.get('metadata', {}).get('', {})
|
|
240
|
+
for k, v in sorted(m.items()):
|
|
241
|
+
print(f'{k}: {v}')
|
|
242
|
+
"
|
|
243
|
+
```
|
|
244
|
+
|
|
245
|
+
## π― Use Cases
|
|
246
|
+
|
|
247
|
+
### Climate/Meteorological Data
|
|
248
|
+
```bash
|
|
249
|
+
nc2cog climate_data.nc output/ \
|
|
250
|
+
--compression deflate \
|
|
251
|
+
--zlevel 9 \
|
|
252
|
+
--resampling cubic \
|
|
253
|
+
--overview-levels 2,4,8,16,32
|
|
254
|
+
```
|
|
255
|
+
|
|
256
|
+
### Oceanographic Data
|
|
257
|
+
```bash
|
|
258
|
+
nc2cog ocean_data.nc output/ \
|
|
259
|
+
--compression lzw \
|
|
260
|
+
--tile-size 1024 \
|
|
261
|
+
--resampling bilinear \
|
|
262
|
+
--overview-levels 2,4,8
|
|
263
|
+
```
|
|
264
|
+
|
|
265
|
+
### Large Dataset Processing
|
|
266
|
+
```bash
|
|
267
|
+
nc2cog large_dataset/ output/ \
|
|
268
|
+
--threads 4 \
|
|
269
|
+
--compression deflate \
|
|
270
|
+
--zlevel 7 \
|
|
271
|
+
--tile-size 512
|
|
272
|
+
```
|
|
273
|
+
|
|
274
|
+
### Projection Transformations
|
|
275
|
+
With the new projection support, you can transform coordinate systems during conversion:
|
|
276
|
+
|
|
277
|
+
Convert with reprojection from WGS84 to Web Mercator:
|
|
278
|
+
```bash
|
|
279
|
+
nc2cog --src-proj EPSG:4326 --dst-proj EPSG:3857 input.nc output/
|
|
280
|
+
```
|
|
281
|
+
|
|
282
|
+
Or specify only target projection (source will be detected automatically):
|
|
283
|
+
```bash
|
|
284
|
+
nc2cog --dst-proj EPSG:3857 input.nc output/
|
|
285
|
+
```
|
|
286
|
+
|
|
287
|
+
Combine with other conversion options:
|
|
288
|
+
```bash
|
|
289
|
+
nc2cog --src-proj EPSG:4326 --dst-proj EPSG:3857 \
|
|
290
|
+
--compression deflate \
|
|
291
|
+
--zlevel 9 \
|
|
292
|
+
--tile-size 1024 \
|
|
293
|
+
input.nc output/
|
|
294
|
+
```
|
|
295
|
+
|
|
296
|
+
## π€ Contributing
|
|
297
|
+
|
|
298
|
+
1. Fork the repository
|
|
299
|
+
2. Create a feature branch
|
|
300
|
+
3. Make your changes
|
|
301
|
+
4. Add tests if applicable
|
|
302
|
+
5. Submit a pull request
|
|
303
|
+
|
|
304
|
+
## π License
|
|
305
|
+
|
|
306
|
+
MIT License
|
|
307
|
+
|
|
308
|
+
## π Issues
|
|
309
|
+
|
|
310
|
+
Report issues on the GitHub repository.
|
|
311
|
+
|
|
312
|
+
## π Documentation
|
|
313
|
+
|
|
314
|
+
For more detailed documentation:
|
|
315
|
+
- [User Guide](docs/user_guide.md)
|
|
316
|
+
- [GDAL Optimization Details](docs/gdal_optimization.md)
|
|
317
|
+
- [Usage Examples](docs/examples.md)
|
|
318
|
+
- [Installation Guide](INSTALLATION.md)
|
|
319
|
+
- [Project Structure](docs/project_structure.md)
|
|
320
|
+
|
|
321
|
+
## βΉοΈ Additional Information
|
|
322
|
+
|
|
323
|
+
### Overview Level Generation
|
|
324
|
+
|
|
325
|
+
When using `--overview-levels`, note that GDAL intelligently determines which overview levels to actually create based on the source image size. For smaller images, GDAL may generate fewer overview levels than specified to avoid creating overly small and potentially useless overviews.
|
|
326
|
+
|
|
327
|
+
For example, with `--overview-levels 2,4,8,16,32` on a 1781x1572 image, GDAL may only generate levels 2 and 4 (890x786 and 445x393 pixels respectively) as the smaller levels would be too small to be useful.
|
|
328
|
+
|
|
329
|
+
---
|
|
330
|
+
|
|
331
|
+
Made with β€οΈ for the geospatial community.
|