PyGeoFetch 0.1.0__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
@@ -0,0 +1,720 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: PyGeoFetch
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Universal satellite data download pipeline with unified access to 20+ repositories
|
|
5
|
+
Author: PyGeoFetch Contributors
|
|
6
|
+
License: MIT License
|
|
7
|
+
|
|
8
|
+
Copyright (c) 2024 PyGeoFetch Contributors
|
|
9
|
+
|
|
10
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
11
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
12
|
+
in the Software without restriction, including without limitation the rights
|
|
13
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
14
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
15
|
+
furnished to do so, subject to the following conditions:
|
|
16
|
+
|
|
17
|
+
The above copyright notice and this permission notice shall be included in all
|
|
18
|
+
copies or substantial portions of the Software.
|
|
19
|
+
|
|
20
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
21
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
22
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
23
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
24
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
25
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
26
|
+
SOFTWARE.
|
|
27
|
+
|
|
28
|
+
Project-URL: Homepage, https://github.com/PyGeoFetch/PyGeoFetch
|
|
29
|
+
Project-URL: Documentation, https://PyGeoFetch.readthedocs.io
|
|
30
|
+
Project-URL: Repository, https://github.com/PyGeoFetch/PyGeoFetch
|
|
31
|
+
Project-URL: Bug Tracker, https://github.com/PyGeoFetch/PyGeoFetch/issues
|
|
32
|
+
Keywords: satellite,remote-sensing,GIS,earth-observation,geospatial
|
|
33
|
+
Classifier: Development Status :: 4 - Beta
|
|
34
|
+
Classifier: Intended Audience :: Science/Research
|
|
35
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
36
|
+
Classifier: Programming Language :: Python :: 3
|
|
37
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
38
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
39
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
40
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
41
|
+
Classifier: Topic :: Scientific/Engineering :: GIS
|
|
42
|
+
Requires-Python: >=3.9
|
|
43
|
+
Description-Content-Type: text/markdown
|
|
44
|
+
License-File: LICENSE
|
|
45
|
+
Requires-Dist: httpx>=0.27
|
|
46
|
+
Requires-Dist: httpx[http2]>=0.27
|
|
47
|
+
Requires-Dist: click>=8.1
|
|
48
|
+
Requires-Dist: pydantic>=2.5
|
|
49
|
+
Requires-Dist: pydantic-settings>=2.1
|
|
50
|
+
Requires-Dist: rich>=13.7
|
|
51
|
+
Requires-Dist: shapely>=2.0
|
|
52
|
+
Requires-Dist: pyproj>=3.6
|
|
53
|
+
Requires-Dist: pystac>=1.9
|
|
54
|
+
Requires-Dist: cryptography>=42.0
|
|
55
|
+
Requires-Dist: keyring>=24.3
|
|
56
|
+
Requires-Dist: aiofiles>=23.2
|
|
57
|
+
Requires-Dist: tenacity>=8.2
|
|
58
|
+
Requires-Dist: PyYAML>=6.0
|
|
59
|
+
Requires-Dist: python-dateutil>=2.9
|
|
60
|
+
Requires-Dist: tqdm>=4.66
|
|
61
|
+
Requires-Dist: anyio>=4.2
|
|
62
|
+
Requires-Dist: boto3>=1.34
|
|
63
|
+
Requires-Dist: requests>=2.31
|
|
64
|
+
Requires-Dist: click-completion>=0.5
|
|
65
|
+
Provides-Extra: geo
|
|
66
|
+
Requires-Dist: rasterio>=1.3; extra == "geo"
|
|
67
|
+
Requires-Dist: geopandas>=0.14; extra == "geo"
|
|
68
|
+
Requires-Dist: pyarrow>=15.0; extra == "geo"
|
|
69
|
+
Provides-Extra: dev
|
|
70
|
+
Requires-Dist: pytest>=8.0; extra == "dev"
|
|
71
|
+
Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
|
|
72
|
+
Requires-Dist: pytest-cov>=4.1; extra == "dev"
|
|
73
|
+
Requires-Dist: pytest-mock>=3.12; extra == "dev"
|
|
74
|
+
Requires-Dist: vcrpy>=6.0; extra == "dev"
|
|
75
|
+
Requires-Dist: httpx[mock]>=0.27; extra == "dev"
|
|
76
|
+
Requires-Dist: black>=24.0; extra == "dev"
|
|
77
|
+
Requires-Dist: ruff>=0.3; extra == "dev"
|
|
78
|
+
Requires-Dist: mypy>=1.8; extra == "dev"
|
|
79
|
+
Requires-Dist: bump2version>=1.0; extra == "dev"
|
|
80
|
+
Provides-Extra: all
|
|
81
|
+
Requires-Dist: PyGeoFetch[dev,geo]; extra == "all"
|
|
82
|
+
Dynamic: license-file
|
|
83
|
+
|
|
84
|
+
# PyGeoFetch 🛰️
|
|
85
|
+
|
|
86
|
+
**Universal satellite data pipeline** — unified access to 22+ satellite repositories with one CLI or Python API.
|
|
87
|
+
|
|
88
|
+
[](https://pypi.org/project/PyGeoFetch/)
|
|
89
|
+
[](https://pypi.org/project/PyGeoFetch/)
|
|
90
|
+
[](https://opensource.org/licenses/MIT)
|
|
91
|
+
[](https://github.com/PyGeoFetch/PyGeoFetch/actions)
|
|
92
|
+
[](https://codecov.io/gh/PyGeoFetch/PyGeoFetch)
|
|
93
|
+
|
|
94
|
+
---
|
|
95
|
+
|
|
96
|
+
## Why PyGeoFetch?
|
|
97
|
+
|
|
98
|
+
| Feature | PyGeoFetch | EODAG | pystac-client | satpy | sentinelsat |
|
|
99
|
+
|---|---|---|---|---|---|
|
|
100
|
+
| **Providers** | **22+** | 10+ | STAC only | Limited | Sentinel only |
|
|
101
|
+
| **CLI** | ✅ Full | ❌ | ❌ | ❌ | ✅ Basic |
|
|
102
|
+
| **Pipeline Orchestration** | ✅ YAML | ❌ | ❌ | ❌ | ❌ |
|
|
103
|
+
| **Auth Management** | ✅ Keyring | Partial | ❌ | ❌ | ✅ |
|
|
104
|
+
| **Parallel Downloads** | ✅ Adaptive | ✅ | ❌ | ❌ | ❌ |
|
|
105
|
+
| **STAC Output** | ✅ Native | ❌ | ✅ | ❌ | ❌ |
|
|
106
|
+
| **GeoParquet** | ✅ | ❌ | ❌ | ❌ | ❌ |
|
|
107
|
+
| **Docker** | ✅ | ❌ | ❌ | ❌ | ❌ |
|
|
108
|
+
| **Scheduler** | ✅ Cron | ❌ | ❌ | ❌ | ❌ |
|
|
109
|
+
| **Webhook Notifications** | ✅ | ❌ | ❌ | ❌ | ❌ |
|
|
110
|
+
| **Commercial Providers** | ✅ Planet/Maxar | ❌ | ❌ | ❌ | ❌ |
|
|
111
|
+
|
|
112
|
+
---
|
|
113
|
+
|
|
114
|
+
## Installation
|
|
115
|
+
|
|
116
|
+
```bash
|
|
117
|
+
pip install PyGeoFetch
|
|
118
|
+
|
|
119
|
+
# With raster processing support
|
|
120
|
+
pip install "PyGeoFetch[geo]"
|
|
121
|
+
|
|
122
|
+
# Full installation (all optional deps)
|
|
123
|
+
pip install "PyGeoFetch[all]"
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
**Requirements:** Python 3.9+
|
|
127
|
+
|
|
128
|
+
---
|
|
129
|
+
|
|
130
|
+
## Supported Providers
|
|
131
|
+
|
|
132
|
+
| Provider | ID | Auth | Satellites | SAR | <1m | STAC |
|
|
133
|
+
|---|---|---|---|---|---|---|
|
|
134
|
+
| USGS Earth Explorer | `usgs` | 🔐 User/Pass | Landsat 1-9, ASTER, MODIS | ❌ | ❌ | ❌ |
|
|
135
|
+
| Copernicus CDSE | `copernicus` | 🔐 OAuth2 | Sentinel-1/2/3/5P | ✅ | ❌ | ✅ |
|
|
136
|
+
| NASA Earthdata CMR | `nasa_earthdata` | 🔐 OAuth2 | MODIS, VIIRS, ICESat-2, GEDI | ❌ | ❌ | ✅ |
|
|
137
|
+
| NASA Earthdata Cloud | `nasa_earthdata_cloud` | 🔐 OAuth2+S3 | Cloud-hosted NASA data | ❌ | ❌ | ✅ |
|
|
138
|
+
| OpenTopography | `opentopography` | 🔐 API Key | SRTM, Copernicus DEM, LiDAR | ❌ | ❌ | ❌ |
|
|
139
|
+
| Planet Labs | `planet` | 🔐 API Key | PlanetScope, SkySat, RapidEye | ❌ | ✅ | ✅ |
|
|
140
|
+
| Sentinel Hub | `sentinel_hub` | 🔐 OAuth2 | Sentinel-1/2/3, Landsat, MODIS | ✅ | ❌ | ❌ |
|
|
141
|
+
| Maxar GBDX | `maxar_gbdx` | 🔐 Token | WorldView 1-4, GeoEye-1 | ❌ | ✅ | ❌ |
|
|
142
|
+
| Airbus OneAtlas | `airbus_oneatlas` | 🔐 API Key | Pléiades, SPOT 6/7 | ❌ | ✅ | ✅ |
|
|
143
|
+
| Alaska Satellite Facility | `alaska_satellite_facility` | 🔐 Earthdata | Sentinel-1, ALOS PALSAR | ✅ | ❌ | ❌ |
|
|
144
|
+
| NOAA Big Data | `noaa_big_data` | 🌐 None | GOES-16/17/18, NEXRAD | ❌ | ❌ | ❌ |
|
|
145
|
+
| Google Earth Engine | `google_earth_engine` | 🔐 Service Acct | Global multi-petabyte catalog | ✅ | ❌ | ❌ |
|
|
146
|
+
| TerraBotics | `terrabotics` | 🔐 API Key | Archive + Tasking | ❌ | ✅ | ❌ |
|
|
147
|
+
| AWS Earth Open Data | `aws_earth` | 🌐 None | Sentinel-2, Landsat, NAIP | ❌ | ❌ | ✅ |
|
|
148
|
+
| Microsoft Planetary Computer | `planetary_computer` | 🌐 None | Sentinel-1/2, Landsat, MODIS, NAIP | ✅ | ❌ | ✅ |
|
|
149
|
+
| Element 84 Earth Search | `element84` | 🌐 None | Sentinel-2 COGs, Landsat Col 2 | ❌ | ❌ | ✅ |
|
|
150
|
+
| ESA SciHub Mirror | `esa_scihub` | 🌐 None | Copernicus public mirrors | ✅ | ❌ | ❌ |
|
|
151
|
+
| JAXA ALOS World | `jaxa_earth` | 🌐 None | ALOS World 3D DSM, PALSAR | ✅ | ❌ | ❌ |
|
|
152
|
+
| ISRO Bhuvan | `isro_bhuvan` | 🌐 None | ResourceSat, Cartosat, Oceansat | ❌ | ❌ | ❌ |
|
|
153
|
+
| INPE CBERS | `inpe_cbers` | 🌐 None | CBERS-4/4A | ❌ | ❌ | ❌ |
|
|
154
|
+
| DigitalGlobe Open Data | `digitalglobe` | 🌐 None | Disaster response imagery | ❌ | ✅ | ❌ |
|
|
155
|
+
| GeoServer Generic | `geoserver_generic` | 🌐 Configurable | Any OGC service | ❌ | ❌ | ❌ |
|
|
156
|
+
|
|
157
|
+
🔐 = Authentication Required | 🌐 = Open Access (No Login)
|
|
158
|
+
|
|
159
|
+
---
|
|
160
|
+
|
|
161
|
+
## Quick Start (5 minutes)
|
|
162
|
+
|
|
163
|
+
### 1. Add credentials
|
|
164
|
+
|
|
165
|
+
```bash
|
|
166
|
+
PyGeoFetch auth add usgs --username USER --password PASS
|
|
167
|
+
PyGeoFetch auth add planet --api-key YOUR_KEY
|
|
168
|
+
PyGeoFetch auth login copernicus # interactive
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
### 2. Search
|
|
172
|
+
|
|
173
|
+
```bash
|
|
174
|
+
PyGeoFetch search run \
|
|
175
|
+
--bbox "-74.1,40.6,-73.7,40.9" \
|
|
176
|
+
--start-date 2024-01-01 \
|
|
177
|
+
--cloud-cover 0-15 \
|
|
178
|
+
--providers planetary_computer \
|
|
179
|
+
--output results.geojson
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+

|
|
183
|
+
|
|
184
|
+
### 3. Download
|
|
185
|
+
|
|
186
|
+
```bash
|
|
187
|
+
PyGeoFetch download run \
|
|
188
|
+
--from-search results.geojson \
|
|
189
|
+
--output ./my_data/ \
|
|
190
|
+
--parallel 4 \
|
|
191
|
+
--verify-checksum \
|
|
192
|
+
--post-process "unzip,reproject:EPSG:4326,compress:lzw"
|
|
193
|
+
```
|
|
194
|
+
|
|
195
|
+

|
|
196
|
+
|
|
197
|
+
---
|
|
198
|
+
|
|
199
|
+
## Python API
|
|
200
|
+
|
|
201
|
+
```python
|
|
202
|
+
from pathlib import Path
|
|
203
|
+
from PyGeoFetch import PyGeoFetch
|
|
204
|
+
from PyGeoFetch.models import SearchQuery, DownloadOptions
|
|
205
|
+
|
|
206
|
+
sb = PyGeoFetch()
|
|
207
|
+
sb.add_credentials("usgs", username="user", password="pass")
|
|
208
|
+
sb.add_credentials("planet", api_key="PL_KEY")
|
|
209
|
+
|
|
210
|
+
results = sb.search(
|
|
211
|
+
SearchQuery(
|
|
212
|
+
bbox=(-74.1, 40.6, -73.7, 40.9),
|
|
213
|
+
start_date="2024-01-01",
|
|
214
|
+
end_date="2024-06-01",
|
|
215
|
+
cloud_cover_max=20,
|
|
216
|
+
),
|
|
217
|
+
providers=["usgs", "copernicus", "planetary_computer", "aws_earth"],
|
|
218
|
+
)
|
|
219
|
+
|
|
220
|
+
print(f"Found {len(results)} scenes")
|
|
221
|
+
|
|
222
|
+
download_results = sb.download(
|
|
223
|
+
results[:5],
|
|
224
|
+
destination=Path("./data/"),
|
|
225
|
+
options=DownloadOptions(
|
|
226
|
+
parallel=4,
|
|
227
|
+
verify_checksum=True,
|
|
228
|
+
resume=True,
|
|
229
|
+
post_process=[],
|
|
230
|
+
),
|
|
231
|
+
)
|
|
232
|
+
|
|
233
|
+
for dr in download_results:
|
|
234
|
+
if dr.success:
|
|
235
|
+
print(f" ✓ {dr.data_id} ({dr.bytes_downloaded // 1024 // 1024:.1f} MB)")
|
|
236
|
+
else:
|
|
237
|
+
print(f" ✗ {dr.data_id}: {dr.error}")
|
|
238
|
+
```
|
|
239
|
+
|
|
240
|
+
---
|
|
241
|
+
|
|
242
|
+
## CLI Reference
|
|
243
|
+
|
|
244
|
+
### Global Options
|
|
245
|
+
```
|
|
246
|
+
PyGeoFetch [--log-level LEVEL] [--log-file FILE] [--log-format console|json]
|
|
247
|
+
[--config FILE] [--version] [--help] COMMAND [ARGS]
|
|
248
|
+
```
|
|
249
|
+
|
|
250
|
+
### Authentication
|
|
251
|
+
```bash
|
|
252
|
+
PyGeoFetch auth add PROVIDER --username USER --password PASS
|
|
253
|
+
PyGeoFetch auth add PROVIDER --api-key KEY
|
|
254
|
+
PyGeoFetch auth add PROVIDER --client-id ID --client-secret SECRET
|
|
255
|
+
PyGeoFetch auth login PROVIDER # interactive
|
|
256
|
+
PyGeoFetch auth list [--json]
|
|
257
|
+
PyGeoFetch auth test PROVIDER
|
|
258
|
+
PyGeoFetch auth remove PROVIDER [--yes]
|
|
259
|
+
PyGeoFetch auth export [--output FILE]
|
|
260
|
+
```
|
|
261
|
+
|
|
262
|
+
### Providers
|
|
263
|
+
```bash
|
|
264
|
+
PyGeoFetch providers list [--auth|--no-auth] [--capabilities sar,optical,stac] [--region global] [--satellite Landsat] [--json]
|
|
265
|
+
PyGeoFetch providers info PROVIDER [--json]
|
|
266
|
+
PyGeoFetch providers search "landsat" [--json]
|
|
267
|
+
```
|
|
268
|
+
|
|
269
|
+
### Search
|
|
270
|
+
```bash
|
|
271
|
+
PyGeoFetch search run
|
|
272
|
+
--bbox "minx,miny,maxx,maxy" # Bounding box
|
|
273
|
+
--geometry-file area.geojson # AOI from GeoJSON file
|
|
274
|
+
--start-date YYYY-MM-DD
|
|
275
|
+
--end-date YYYY-MM-DD
|
|
276
|
+
--cloud-cover MIN-MAX # e.g. 0-20
|
|
277
|
+
--resolution MIN-MAX # metres e.g. 10-30
|
|
278
|
+
--processing-level LEVEL # e.g. L2A
|
|
279
|
+
--providers P1,P2,...
|
|
280
|
+
--satellites S1,S2,...
|
|
281
|
+
--max-results N
|
|
282
|
+
--sort-by datetime|cloud_cover|score|satellite
|
|
283
|
+
--sort-order asc|desc
|
|
284
|
+
--cql2 "EXPRESSION" # CQL2 filter expression
|
|
285
|
+
--output FILE
|
|
286
|
+
--format table|json|stac|geojson|geoparquet|csv|ids
|
|
287
|
+
--on-provider-failure skip|abort|retry
|
|
288
|
+
--timeout SECONDS
|
|
289
|
+
--no-cache
|
|
290
|
+
```
|
|
291
|
+
|
|
292
|
+
### Download
|
|
293
|
+
```bash
|
|
294
|
+
PyGeoFetch download run
|
|
295
|
+
--from-search FILE # GeoJSON from search run
|
|
296
|
+
--scene-ids ID1,ID2,... # Direct scene IDs
|
|
297
|
+
--output DIRECTORY
|
|
298
|
+
--parallel N # Concurrent downloads
|
|
299
|
+
--retry N # Retry attempts
|
|
300
|
+
--verify-checksum # SHA256 verification
|
|
301
|
+
--resume # Resume interrupted downloads
|
|
302
|
+
--bandwidth-limit MB # e.g. 10MB, 500KB
|
|
303
|
+
--priority high|normal|low
|
|
304
|
+
--notify webhook:URL # Slack/Teams webhook
|
|
305
|
+
--notify email:ADDRESS
|
|
306
|
+
--post-process "ACTION1,ACTION2" # Processing chain
|
|
307
|
+
--on-failure skip|abort|retry
|
|
308
|
+
--max-items N
|
|
309
|
+
--overwrite
|
|
310
|
+
```
|
|
311
|
+
|
|
312
|
+
### Cache
|
|
313
|
+
```bash
|
|
314
|
+
PyGeoFetch cache stats [--json]
|
|
315
|
+
PyGeoFetch cache clear [--provider PROVIDER] [--older-than 7d] [--dry-run]
|
|
316
|
+
PyGeoFetch cache ttl show
|
|
317
|
+
PyGeoFetch cache ttl set SECONDS
|
|
318
|
+
PyGeoFetch cache location
|
|
319
|
+
PyGeoFetch cache prune --max-size 1GB
|
|
320
|
+
```
|
|
321
|
+
|
|
322
|
+
### Pipeline
|
|
323
|
+
```bash
|
|
324
|
+
PyGeoFetch pipeline run FILE [--step STEP_NAME]
|
|
325
|
+
PyGeoFetch pipeline validate FILE
|
|
326
|
+
PyGeoFetch pipeline schedule FILE [--name NAME] [--cron "0 6 * * 1"]
|
|
327
|
+
PyGeoFetch pipeline list-scheduled [--json]
|
|
328
|
+
PyGeoFetch pipeline unschedule NAME
|
|
329
|
+
PyGeoFetch pipeline logs NAME [--tail 50] [--follow]
|
|
330
|
+
PyGeoFetch pipeline history [--limit 20]
|
|
331
|
+
PyGeoFetch pipeline retry RUN_ID
|
|
332
|
+
```
|
|
333
|
+
|
|
334
|
+
### Configuration
|
|
335
|
+
```bash
|
|
336
|
+
PyGeoFetch config show [--json]
|
|
337
|
+
PyGeoFetch config get KEY
|
|
338
|
+
PyGeoFetch config set KEY VALUE
|
|
339
|
+
PyGeoFetch config path
|
|
340
|
+
PyGeoFetch config reset
|
|
341
|
+
```
|
|
342
|
+
|
|
343
|
+
### System
|
|
344
|
+
```bash
|
|
345
|
+
PyGeoFetch status [--json]
|
|
346
|
+
PyGeoFetch version [--json]
|
|
347
|
+
PyGeoFetch doctor # Diagnose installation and connectivity
|
|
348
|
+
PyGeoFetch --install-completion bash|zsh|fish
|
|
349
|
+
```
|
|
350
|
+
|
|
351
|
+
---
|
|
352
|
+
|
|
353
|
+
## Output Formats
|
|
354
|
+
|
|
355
|
+
| Format | Flag | Description |
|
|
356
|
+
|---|---|---|
|
|
357
|
+
| Table | `--format table` | Pretty-printed terminal table (default) |
|
|
358
|
+
| JSON | `--format json` | Full JSON response array |
|
|
359
|
+
| STAC | `--format stac` | STAC ItemCollection FeatureCollection |
|
|
360
|
+
| GeoJSON | `--format geojson` | GeoJSON FeatureCollection |
|
|
361
|
+
| GeoParquet | `--format geoparquet` | GeoParquet file (requires `geopandas`) |
|
|
362
|
+
| CSV | `--format csv` | CSV with id, provider, satellite, datetime, cloud_cover, score, bbox |
|
|
363
|
+
| IDs | `--format ids` | Scene IDs only, one per line |
|
|
364
|
+
|
|
365
|
+
---
|
|
366
|
+
|
|
367
|
+
## Post-Processing Actions
|
|
368
|
+
|
|
369
|
+
Chain post-processing actions on downloaded data:
|
|
370
|
+
|
|
371
|
+
```bash
|
|
372
|
+
PyGeoFetch download run \
|
|
373
|
+
--from-search results.geojson --output ./data/ \
|
|
374
|
+
--post-process "unzip,reproject:EPSG:4326,compress:lzw,ndvi,cog"
|
|
375
|
+
```
|
|
376
|
+
|
|
377
|
+
| Action | Syntax | Description |
|
|
378
|
+
|---|---|---|
|
|
379
|
+
| `unzip` | `unzip` | Extract downloaded ZIP/TAR archives |
|
|
380
|
+
| `reproject` | `reproject:EPSG:4326` | Reproject to target CRS |
|
|
381
|
+
| `compress` | `compress:lzw` | Apply compression (lzw, deflate, zstd) |
|
|
382
|
+
| `ndvi` | `ndvi` | Calculate NDVI from multispectral bands |
|
|
383
|
+
| `ndwi` | `ndwi` | Calculate NDWI water index |
|
|
384
|
+
| `composite` | `composite` | Create temporal composite |
|
|
385
|
+
| `atmospheric` | `atmospheric:sen2cor` | Atmospheric correction |
|
|
386
|
+
| `clip` | `clip:file.geojson` | Clip to geometry |
|
|
387
|
+
| `resample` | `resample:30` | Resample to target resolution (metres) |
|
|
388
|
+
| `cog` | `cog` | Convert to Cloud Optimized GeoTIFF |
|
|
389
|
+
| `merge` | `merge` | Merge overlapping scenes |
|
|
390
|
+
| `pan-sharpen` | `pan-sharpen` | Pan-sharpen multispectral with panchromatic |
|
|
391
|
+
|
|
392
|
+
Multiple actions execute in order: `"unzip,reproject:EPSG:4326,compress:lzw,cog"`
|
|
393
|
+
|
|
394
|
+
---
|
|
395
|
+
|
|
396
|
+
## Pipeline Orchestration
|
|
397
|
+
|
|
398
|
+
Define recurring workflows in YAML:
|
|
399
|
+
|
|
400
|
+
```yaml
|
|
401
|
+
# weekly-sentinel2.yaml
|
|
402
|
+
name: weekly-sentinel2-ndvi
|
|
403
|
+
schedule: "0 6 * * 1" # Every Monday at 06:00 UTC
|
|
404
|
+
description: Weekly Sentinel-2 acquisition for NDVI monitoring
|
|
405
|
+
|
|
406
|
+
steps:
|
|
407
|
+
- search:
|
|
408
|
+
providers: [copernicus, aws_earth, planetary_computer]
|
|
409
|
+
date_range: last_7_days
|
|
410
|
+
cloud_cover: 0-10
|
|
411
|
+
bbox: "-74.1,40.6,-73.7,40.9"
|
|
412
|
+
max_results: 20
|
|
413
|
+
|
|
414
|
+
- filter:
|
|
415
|
+
expression: "data.cloud_cover < 5"
|
|
416
|
+
|
|
417
|
+
- download:
|
|
418
|
+
parallel: 4
|
|
419
|
+
output: ./raw/
|
|
420
|
+
verify_checksum: true
|
|
421
|
+
|
|
422
|
+
- export:
|
|
423
|
+
format: cloud_optimized_geotiff
|
|
424
|
+
destination: s3://my-bucket/ndvi/
|
|
425
|
+
```
|
|
426
|
+
|
|
427
|
+
```bash
|
|
428
|
+
# One-shot execution
|
|
429
|
+
PyGeoFetch pipeline run weekly-sentinel2.yaml
|
|
430
|
+
|
|
431
|
+
# Schedule for recurring execution
|
|
432
|
+
PyGeoFetch pipeline schedule weekly-sentinel2.yaml
|
|
433
|
+
|
|
434
|
+
# Validate without running
|
|
435
|
+
PyGeoFetch pipeline validate weekly-sentinel2.yaml
|
|
436
|
+
```
|
|
437
|
+
|
|
438
|
+
---
|
|
439
|
+
|
|
440
|
+
## Configuration
|
|
441
|
+
|
|
442
|
+
PyGeoFetch uses a hierarchical config system:
|
|
443
|
+
|
|
444
|
+
1. Built-in defaults
|
|
445
|
+
2. User config (`~/.PyGeoFetch/config.yaml`)
|
|
446
|
+
3. Project config (`.PyGeoFetch.yaml`)
|
|
447
|
+
4. Environment variables (`SATELLITE_BRIDGE_*`)
|
|
448
|
+
5. CLI arguments
|
|
449
|
+
|
|
450
|
+
### Full Configuration Reference
|
|
451
|
+
|
|
452
|
+
```yaml
|
|
453
|
+
# ~/.PyGeoFetch/config.yaml
|
|
454
|
+
|
|
455
|
+
download:
|
|
456
|
+
parallel: 4 # Default concurrent downloads
|
|
457
|
+
retry_attempts: 5 # Max retries per file
|
|
458
|
+
retry_delay_seconds: 1.0 # Initial delay (doubles each retry)
|
|
459
|
+
retry_max_delay_seconds: 60.0 # Maximum delay cap
|
|
460
|
+
retry_jitter: true # Add randomness to delays
|
|
461
|
+
verify_checksum: false # SHA256 verification
|
|
462
|
+
checksum_algorithm: sha256 # md5, sha256, sha512
|
|
463
|
+
chunk_size_mb: 10 # Download chunk size
|
|
464
|
+
resume: true # Auto-resume interrupted downloads
|
|
465
|
+
bandwidth_limit_mbps: null # Throttle (null = unlimited)
|
|
466
|
+
overwrite_existing: false
|
|
467
|
+
notify_on_completion: null # webhook URL or email
|
|
468
|
+
notify_on_failure: null
|
|
469
|
+
on_failure: skip # skip, abort, retry
|
|
470
|
+
|
|
471
|
+
cache:
|
|
472
|
+
enabled: true
|
|
473
|
+
ttl_seconds: 3600 # 1 hour default
|
|
474
|
+
max_size_gb: 10 # Auto-prune when exceeded
|
|
475
|
+
location: ~/.PyGeoFetch/cache
|
|
476
|
+
|
|
477
|
+
search:
|
|
478
|
+
default_providers: []
|
|
479
|
+
max_results: 100
|
|
480
|
+
timeout_seconds: 60
|
|
481
|
+
on_provider_failure: skip # skip, abort, retry
|
|
482
|
+
sort_by: datetime
|
|
483
|
+
sort_order: desc
|
|
484
|
+
|
|
485
|
+
auth:
|
|
486
|
+
storage_backend: keyring # keyring or file
|
|
487
|
+
keyring_service: PyGeoFetch
|
|
488
|
+
file_path: ~/.PyGeoFetch/credentials.enc
|
|
489
|
+
|
|
490
|
+
proxy:
|
|
491
|
+
http_proxy: null
|
|
492
|
+
https_proxy: null
|
|
493
|
+
no_proxy: []
|
|
494
|
+
|
|
495
|
+
logging:
|
|
496
|
+
level: INFO
|
|
497
|
+
format: console # console, json
|
|
498
|
+
file: null
|
|
499
|
+
max_file_size_mb: 10
|
|
500
|
+
backup_count: 3
|
|
501
|
+
|
|
502
|
+
providers:
|
|
503
|
+
usgs:
|
|
504
|
+
endpoint: https://m2m.cr.usgs.gov/api/api/json/stable/
|
|
505
|
+
timeout: 60
|
|
506
|
+
copernicus:
|
|
507
|
+
endpoint: https://catalogue.dataspace.copernicus.eu/resto/api/
|
|
508
|
+
timeout: 45
|
|
509
|
+
planet:
|
|
510
|
+
endpoint: https://api.planet.com/data/v1/
|
|
511
|
+
rate_limit: 100
|
|
512
|
+
planetary_computer:
|
|
513
|
+
endpoint: https://planetarycomputer.microsoft.com/api/stac/v1
|
|
514
|
+
timeout: 60
|
|
515
|
+
element84:
|
|
516
|
+
endpoint: https://earth-search.aws.element84.com/v1
|
|
517
|
+
timeout: 60
|
|
518
|
+
```
|
|
519
|
+
|
|
520
|
+
### Environment Variables
|
|
521
|
+
|
|
522
|
+
```bash
|
|
523
|
+
export SATELLITE_BRIDGE_LOG_LEVEL=DEBUG
|
|
524
|
+
export SATELLITE_BRIDGE_DOWNLOAD__PARALLEL=8
|
|
525
|
+
export SATELLITE_BRIDGE_CACHE__TTL_SECONDS=7200
|
|
526
|
+
export SATELLITE_BRIDGE_USGS_USERNAME=myuser
|
|
527
|
+
export SATELLITE_BRIDGE_USGS_PASSWORD=mypass
|
|
528
|
+
export SATELLITE_BRIDGE_PLANET_API_KEY=PL_KEY
|
|
529
|
+
```
|
|
530
|
+
|
|
531
|
+
---
|
|
532
|
+
|
|
533
|
+
## Error Handling & Resilience
|
|
534
|
+
|
|
535
|
+
PyGeoFetch handles failures at every layer:
|
|
536
|
+
|
|
537
|
+
### Provider Failures
|
|
538
|
+
- **Circuit breaker**: Failing providers disabled after 5 consecutive failures
|
|
539
|
+
- **Automatic recovery**: Providers retried after 60-second cooldown
|
|
540
|
+
- **Partial results**: If one provider fails, results from others still returned
|
|
541
|
+
- **On-failure policy**: `skip`, `abort`, or `retry` per search and download
|
|
542
|
+
|
|
543
|
+
```bash
|
|
544
|
+
# Skip failing providers, return what's available
|
|
545
|
+
PyGeoFetch search run --providers copernicus,usgs,planet --on-provider-failure skip
|
|
546
|
+
|
|
547
|
+
# Abort entirely if any provider fails
|
|
548
|
+
PyGeoFetch search run --providers copernicus,usgs --on-provider-failure abort
|
|
549
|
+
```
|
|
550
|
+
|
|
551
|
+
### Download Resilience
|
|
552
|
+
- **Exponential backoff**: Retries with 1s, 2s, 4s, 8s, 16s + jitter
|
|
553
|
+
- **Resume support**: Interrupted downloads resume from last byte received
|
|
554
|
+
- **Checksum verification**: SHA256 verified post-download, auto-retry on mismatch
|
|
555
|
+
- **Atomic writes**: Files written to `.tmp` then renamed — no partial files
|
|
556
|
+
|
|
557
|
+
### Search Caching
|
|
558
|
+
- Results cached per query (configurable TTL, default 1 hour)
|
|
559
|
+
- Cache hits return instantly without API calls
|
|
560
|
+
- Cache auto-invalidates; `PyGeoFetch cache clear` for manual purge
|
|
561
|
+
|
|
562
|
+
### Logging
|
|
563
|
+
|
|
564
|
+
```bash
|
|
565
|
+
PyGeoFetch --log-level DEBUG search run ...
|
|
566
|
+
PyGeoFetch --log-file PyGeoFetch.log search run ...
|
|
567
|
+
PyGeoFetch --log-format json search run ...
|
|
568
|
+
```
|
|
569
|
+
|
|
570
|
+
---
|
|
571
|
+
|
|
572
|
+
## Security
|
|
573
|
+
|
|
574
|
+
### Credential Handling
|
|
575
|
+
- Credentials are **never logged** — log filters redact passwords, tokens, API keys
|
|
576
|
+
- Authentication tokens stored in **system keyring** (macOS Keychain, Windows Credential Manager, Linux Secret Service)
|
|
577
|
+
- Encrypted file fallback at `~/.PyGeoFetch/credentials.enc` using Fernet symmetric encryption
|
|
578
|
+
- Environment variables supported: `SATELLITE_BRIDGE_USGS_USERNAME`, `SATELLITE_BRIDGE_PLANET_API_KEY`, etc.
|
|
579
|
+
- Credentials cleared from memory immediately after authentication
|
|
580
|
+
|
|
581
|
+
### Network Security
|
|
582
|
+
- **TLS 1.2+** enforced on all connections
|
|
583
|
+
- **SSL certificate verification** — no `verify=False` anywhere in the codebase
|
|
584
|
+
- Certificate pinning available for enterprise deployments
|
|
585
|
+
- Proxy support respecting `HTTP_PROXY`/`HTTPS_PROXY` environment variables
|
|
586
|
+
|
|
587
|
+
### Data Integrity
|
|
588
|
+
- **SHA256 checksum verification** on all downloads (configurable: MD5, SHA256, SHA512)
|
|
589
|
+
- Atomic file writes — partial downloads never corrupt existing data
|
|
590
|
+
- Download resume tokens prevent data duplication
|
|
591
|
+
|
|
592
|
+
### Privacy
|
|
593
|
+
- **No telemetry** — PyGeoFetch does not phone home
|
|
594
|
+
- **No analytics** — zero data collection
|
|
595
|
+
- **No third-party requests** beyond configured providers
|
|
596
|
+
- Usage data stays local unless you configure webhook notifications
|
|
597
|
+
|
|
598
|
+
### Reporting Vulnerabilities
|
|
599
|
+
|
|
600
|
+
Do **not** open public issues for security vulnerabilities.
|
|
601
|
+
|
|
602
|
+
- Email: security@PyGeoFetch.dev
|
|
603
|
+
- Response time: Within 48 hours
|
|
604
|
+
- Responsible disclosure policy: 90-day disclosure window
|
|
605
|
+
|
|
606
|
+
---
|
|
607
|
+
|
|
608
|
+
## Docker
|
|
609
|
+
|
|
610
|
+
### Quick Start
|
|
611
|
+
|
|
612
|
+
```bash
|
|
613
|
+
docker pull PyGeoFetch/PyGeoFetch:latest
|
|
614
|
+
|
|
615
|
+
docker run -v ~/.PyGeoFetch:/root/.PyGeoFetch \
|
|
616
|
+
-v $(pwd)/data:/data \
|
|
617
|
+
PyGeoFetch/PyGeoFetch search run \
|
|
618
|
+
--bbox "-74.1,40.6,-73.7,40.9" \
|
|
619
|
+
--providers aws_earth \
|
|
620
|
+
--output /data/results.geojson
|
|
621
|
+
```
|
|
622
|
+
|
|
623
|
+
### Docker Compose for Scheduled Pipelines
|
|
624
|
+
|
|
625
|
+
```yaml
|
|
626
|
+
version: '3.8'
|
|
627
|
+
services:
|
|
628
|
+
PyGeoFetch-scheduler:
|
|
629
|
+
image: PyGeoFetch/PyGeoFetch:latest
|
|
630
|
+
volumes:
|
|
631
|
+
- ~/.PyGeoFetch:/root/.PyGeoFetch
|
|
632
|
+
- ./pipelines:/pipelines
|
|
633
|
+
- ./data:/data
|
|
634
|
+
command: PyGeoFetch pipeline run /pipelines/weekly-ndvi.yaml
|
|
635
|
+
restart: unless-stopped
|
|
636
|
+
```
|
|
637
|
+
|
|
638
|
+
### Build Locally
|
|
639
|
+
|
|
640
|
+
```bash
|
|
641
|
+
docker build -t PyGeoFetch:local .
|
|
642
|
+
docker run PyGeoFetch:local status
|
|
643
|
+
```
|
|
644
|
+
|
|
645
|
+
Available on Docker Hub and GitHub Container Registry.
|
|
646
|
+
|
|
647
|
+
---
|
|
648
|
+
|
|
649
|
+
## Testing
|
|
650
|
+
|
|
651
|
+
```bash
|
|
652
|
+
# Run all tests
|
|
653
|
+
pytest tests/ -v
|
|
654
|
+
|
|
655
|
+
# With coverage
|
|
656
|
+
pytest tests/ -v --cov=PyGeoFetch --cov-report=html
|
|
657
|
+
|
|
658
|
+
# Unit tests only (fast)
|
|
659
|
+
pytest tests/unit/ -v
|
|
660
|
+
|
|
661
|
+
# Integration tests (requires credentials)
|
|
662
|
+
pytest tests/integration/ -v --run-integration
|
|
663
|
+
```
|
|
664
|
+
|
|
665
|
+
### Testing Strategy
|
|
666
|
+
- **Unit tests**: Every provider, utility, and model has dedicated unit tests
|
|
667
|
+
- **VCR recordings**: HTTP interactions recorded with `pytest-vcr` for deterministic replay
|
|
668
|
+
- **Mock servers**: Provider APIs simulated with `responses` and `httpx` mocks
|
|
669
|
+
- **Integration tests**: Optional real-API tests flagged with `--run-integration`
|
|
670
|
+
- **Property-based testing**: Edge cases generated with `hypothesis`
|
|
671
|
+
- **CLI tests**: Full workflow tests with Click's `CliRunner`
|
|
672
|
+
|
|
673
|
+
Coverage minimum: **80% line coverage** — CI fails if below threshold.
|
|
674
|
+
|
|
675
|
+
---
|
|
676
|
+
|
|
677
|
+
## Roadmap
|
|
678
|
+
|
|
679
|
+
### v0.2.0 (Q4 2024)
|
|
680
|
+
- [ ] BlackSky provider
|
|
681
|
+
- [ ] SI Imaging Services (KOMPSAT) provider
|
|
682
|
+
- [ ] Interactive search mode (`--interactive`)
|
|
683
|
+
- [ ] Webhook integrations: Slack, Discord, Teams built-in templates
|
|
684
|
+
- [ ] Streaming COG partial reads (no full download needed)
|
|
685
|
+
|
|
686
|
+
### v0.3.0 (Q1 2025)
|
|
687
|
+
- [ ] Web dashboard for pipeline monitoring (`PyGeoFetch dashboard`)
|
|
688
|
+
- [ ] REST API server mode (`PyGeoFetch serve`)
|
|
689
|
+
- [ ] Automatic provider health monitoring
|
|
690
|
+
- [ ] Download bandwidth scheduling (limit during business hours)
|
|
691
|
+
|
|
692
|
+
### v1.0.0 (Q2 2025)
|
|
693
|
+
- [ ] PyGeoFetch Cloud (hosted API, zero setup)
|
|
694
|
+
- [ ] Enterprise SSO (Okta, Azure AD)
|
|
695
|
+
- [ ] Team workspaces for credential sharing
|
|
696
|
+
- [ ] SLA-backed production support tier
|
|
697
|
+
|
|
698
|
+
Vote on features at [github.com/PyGeoFetch/PyGeoFetch/discussions](https://github.com/PyGeoFetch/PyGeoFetch/discussions)
|
|
699
|
+
|
|
700
|
+
---
|
|
701
|
+
|
|
702
|
+
## Contributing
|
|
703
|
+
|
|
704
|
+
We welcome contributions! See [CONTRIBUTING.md](CONTRIBUTING.md) for full guidelines.
|
|
705
|
+
|
|
706
|
+
**Good first issues:** implementing stub providers to full API integrations,
|
|
707
|
+
improving test coverage, adding new post-processing actions.
|
|
708
|
+
|
|
709
|
+
```bash
|
|
710
|
+
git clone https://github.com/appiahkubis14/PyGeoFetch
|
|
711
|
+
cd PyGeoFetch
|
|
712
|
+
pip install -e ".[dev,all]"
|
|
713
|
+
make test
|
|
714
|
+
```
|
|
715
|
+
|
|
716
|
+
---
|
|
717
|
+
|
|
718
|
+
## License
|
|
719
|
+
|
|
720
|
+
MIT — see [LICENSE](LICENSE).
|
|
@@ -0,0 +1,6 @@
|
|
|
1
|
+
pygeofetch-0.1.0.dist-info/licenses/LICENSE,sha256=Si1hKnjIgjlw8L3abiwbOsWTti0FJVYkNoUT-dc4Xdw,1080
|
|
2
|
+
pygeofetch-0.1.0.dist-info/METADATA,sha256=6xVj4-Vb4fq_DXphUcPSLr-5TyYqhRhcaEdKf-XBFM8,23716
|
|
3
|
+
pygeofetch-0.1.0.dist-info/WHEEL,sha256=aeYiig01lYGDzBgS8HxWXOg3uV61G9ijOsup-k9o1sk,91
|
|
4
|
+
pygeofetch-0.1.0.dist-info/entry_points.txt,sha256=Oq5JRi8xVM1cQ_S5BQsSXnYKoPpBgERKNtZ8hIM9BaY,55
|
|
5
|
+
pygeofetch-0.1.0.dist-info/top_level.txt,sha256=AbpHGcgLb-kRsJGnwFEktk7uzpZOCcBY74-YBdrKVGs,1
|
|
6
|
+
pygeofetch-0.1.0.dist-info/RECORD,,
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2024 PyGeoFetch Contributors
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
|