hybig-py 2.0.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- hybig_py-2.0.0/.gitignore +51 -0
- hybig_py-2.0.0/LICENSE +60 -0
- hybig_py-2.0.0/PKG-INFO +507 -0
- hybig_py-2.0.0/README.md +483 -0
- hybig_py-2.0.0/hybig/__init__.py +5 -0
- hybig_py-2.0.0/hybig/browse.py +512 -0
- hybig_py-2.0.0/hybig/browse_utility.py +34 -0
- hybig_py-2.0.0/hybig/color_utility.py +149 -0
- hybig_py-2.0.0/hybig/crs.py +114 -0
- hybig_py-2.0.0/hybig/exceptions.py +17 -0
- hybig_py-2.0.0/hybig/sizes.py +429 -0
- hybig_py-2.0.0/pyproject.toml +52 -0
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
# Temporary output directories
|
|
2
|
+
tmp*
|
|
3
|
+
.DS_Store
|
|
4
|
+
tmp
|
|
5
|
+
config-*.json
|
|
6
|
+
config.json
|
|
7
|
+
|
|
8
|
+
# Byte-compiled / optimized / DLL files
|
|
9
|
+
__pycache__/
|
|
10
|
+
*.py[cod]
|
|
11
|
+
*$py.class
|
|
12
|
+
|
|
13
|
+
# Unit test / coverage reports
|
|
14
|
+
htmlcov/
|
|
15
|
+
.tox/
|
|
16
|
+
.nox/
|
|
17
|
+
.coverage
|
|
18
|
+
.coverage.*
|
|
19
|
+
.cache
|
|
20
|
+
nosetests.xml
|
|
21
|
+
coverage.xml
|
|
22
|
+
*.cover
|
|
23
|
+
.hypothesis/
|
|
24
|
+
.pytest_cache/
|
|
25
|
+
coverage
|
|
26
|
+
test-reports
|
|
27
|
+
tests/reports
|
|
28
|
+
|
|
29
|
+
# Jupyter Notebook
|
|
30
|
+
.ipynb_checkpoints
|
|
31
|
+
|
|
32
|
+
# IPython
|
|
33
|
+
profile_default/
|
|
34
|
+
ipython_config.py
|
|
35
|
+
|
|
36
|
+
# pyenv
|
|
37
|
+
.python-version
|
|
38
|
+
|
|
39
|
+
# Environments
|
|
40
|
+
.env
|
|
41
|
+
.venv
|
|
42
|
+
env/
|
|
43
|
+
venv/
|
|
44
|
+
ENV/
|
|
45
|
+
env.bak/
|
|
46
|
+
venv.bak/
|
|
47
|
+
|
|
48
|
+
# mypy
|
|
49
|
+
.mypy_cache/
|
|
50
|
+
.dmypy.json
|
|
51
|
+
dmypy.json
|
hybig_py-2.0.0/LICENSE
ADDED
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
Copyright © 2022 United States Government as represented by the Administrator of the National Aeronautics and Space Administration. All Rights Reserved.
|
|
2
|
+
|
|
3
|
+
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License.
|
|
4
|
+
You may obtain a copy of the License at
|
|
5
|
+
|
|
6
|
+
http://www.apache.org/licenses/LICENSE-2.0
|
|
7
|
+
|
|
8
|
+
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS,
|
|
9
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
|
|
10
|
+
|
|
11
|
+
---
|
|
12
|
+
|
|
13
|
+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
|
14
|
+
|
|
15
|
+
1. Definitions.
|
|
16
|
+
|
|
17
|
+
"License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document.
|
|
18
|
+
|
|
19
|
+
"Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License.
|
|
20
|
+
|
|
21
|
+
"Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
|
|
22
|
+
|
|
23
|
+
"You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License.
|
|
24
|
+
|
|
25
|
+
"Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files.
|
|
26
|
+
|
|
27
|
+
"Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.
|
|
28
|
+
|
|
29
|
+
"Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below).
|
|
30
|
+
|
|
31
|
+
"Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.
|
|
32
|
+
|
|
33
|
+
"Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution."
|
|
34
|
+
|
|
35
|
+
"Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work.
|
|
36
|
+
|
|
37
|
+
2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.
|
|
38
|
+
|
|
39
|
+
3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed.
|
|
40
|
+
|
|
41
|
+
4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:
|
|
42
|
+
|
|
43
|
+
You must give any other recipients of the Work or Derivative Works a copy of this License; and
|
|
44
|
+
You must cause any modified files to carry prominent notices stating that You changed the files; and
|
|
45
|
+
You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and
|
|
46
|
+
If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License.
|
|
47
|
+
|
|
48
|
+
You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License.
|
|
49
|
+
|
|
50
|
+
5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions.
|
|
51
|
+
|
|
52
|
+
6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file.
|
|
53
|
+
|
|
54
|
+
7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.
|
|
55
|
+
|
|
56
|
+
8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages.
|
|
57
|
+
|
|
58
|
+
9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.
|
|
59
|
+
|
|
60
|
+
END OF TERMS AND CONDITIONS
|
hybig_py-2.0.0/PKG-INFO
ADDED
|
@@ -0,0 +1,507 @@
|
|
|
1
|
+
Metadata-Version: 2.3
|
|
2
|
+
Name: hybig-py
|
|
3
|
+
Version: 2.0.0
|
|
4
|
+
Summary: Python package designed to produce browse imagery compatible with NASA's Global Image Browse Services (GIBS).
|
|
5
|
+
Project-URL: Homepage, https://github.com/nasa/harmony-browse-image-generator
|
|
6
|
+
Project-URL: Issues, https://github.com/nasa/harmony-browse-image-generator/issues
|
|
7
|
+
Author-email: Matt Savoie <savoie@colorado.edu>, Owen Littlejohns <owen.m.littlejohns@nasa.gov>
|
|
8
|
+
Maintainer-email: Matt Savoie <savoie@colorado.edu>, Owen Littlejohns <owen.m.littlejohns@nasa.gov>
|
|
9
|
+
License-File: LICENSE
|
|
10
|
+
Classifier: License :: OSI Approved :: Apache Software License
|
|
11
|
+
Classifier: Operating System :: OS Independent
|
|
12
|
+
Classifier: Programming Language :: Python :: 3
|
|
13
|
+
Requires-Python: >=3.10
|
|
14
|
+
Requires-Dist: gdal==3.6.2
|
|
15
|
+
Requires-Dist: harmony-service-lib~=1.0.27
|
|
16
|
+
Requires-Dist: matplotlib==3.9.0
|
|
17
|
+
Requires-Dist: numpy==1.26.4
|
|
18
|
+
Requires-Dist: pillow==10.3.0
|
|
19
|
+
Requires-Dist: pyproj==3.6.1
|
|
20
|
+
Requires-Dist: pystac~=0.5.6
|
|
21
|
+
Requires-Dist: rasterio==1.3.10
|
|
22
|
+
Requires-Dist: rioxarray==0.15.5
|
|
23
|
+
Description-Content-Type: text/markdown
|
|
24
|
+
|
|
25
|
+
# Harmony Browse Image Generator (HyBIG).
|
|
26
|
+
|
|
27
|
+
This repository contains code designed to produce browse imagery. Its default behaviour
|
|
28
|
+
produces images compatible with the NASA Global Image Browse
|
|
29
|
+
Services ([GIBS](https://www.earthdata.nasa.gov/eosdis/science-system-description/eosdis-components/gibs)).
|
|
30
|
+
|
|
31
|
+
This means that default parameters for images are selected to match the
|
|
32
|
+
visualization generation requirements and recommendations put forth in the GIBS
|
|
33
|
+
Interface Control Document (ICD), which can be found on [Earthdata
|
|
34
|
+
Wiki](https://wiki.earthdata.nasa.gov/display/GITC/Ingest+Delivery+Methods)
|
|
35
|
+
along with [additional GIBS
|
|
36
|
+
documentation](https://nasa-gibs.github.io/gibs-api-docs/).
|
|
37
|
+
|
|
38
|
+
HyBIG creates paletted PNG images and associated metadata from GeoTIFF input
|
|
39
|
+
images. Scientific parameter raster data as well as RGB[A] raster images can
|
|
40
|
+
be converted to browse PNGs. These browse images undergo transformation by
|
|
41
|
+
reprojection, tiling and coloring to seamlessly integrate with GIBS.
|
|
42
|
+
|
|
43
|
+
The repository contains code and infrastructure to support both the HyBIG
|
|
44
|
+
Service as well as `hybig-py`. The HyBIG Service is packaged as a Docker
|
|
45
|
+
container that is deployed to [NASA's
|
|
46
|
+
Harmony](https://harmony.earthdata.nasa.gov/) system. The business logic is
|
|
47
|
+
contained in the [`hybig-py` library](https://pypi.org/project/hybig-py/) which
|
|
48
|
+
exposes functions to generate browse images in python scripts.
|
|
49
|
+
|
|
50
|
+
### hybig-py
|
|
51
|
+
|
|
52
|
+
The browse image generation logic is packaged in the hybig-py
|
|
53
|
+
library. Currently, a single function, `create_browse` is exposed to the user.
|
|
54
|
+
|
|
55
|
+
```python
|
|
56
|
+
def create_browse(
|
|
57
|
+
source_tiff: str,
|
|
58
|
+
params: dict = None,
|
|
59
|
+
palette: str | ColorPalette | None = None,
|
|
60
|
+
logger: Logger = None,
|
|
61
|
+
) -> list[tuple[Path, Path, Path]]:
|
|
62
|
+
"""Create browse imagery from an input geotiff.
|
|
63
|
+
|
|
64
|
+
This is the exposed library function to allow users to create browse images
|
|
65
|
+
from the hybig-py library. It parses the input params and constructs the
|
|
66
|
+
correct Harmony input structure [Message.Format] to call the service's
|
|
67
|
+
entry point create_browse_imagery.
|
|
68
|
+
|
|
69
|
+
Output images are created and deposited into the input GeoTIFF's directory.
|
|
70
|
+
|
|
71
|
+
Args:
|
|
72
|
+
source_tiff: str, location of the input geotiff to process.
|
|
73
|
+
|
|
74
|
+
params: [dict | None], A dictionary with the following keys:
|
|
75
|
+
|
|
76
|
+
mime: [str], MIME type of the output image (default: 'image/png').
|
|
77
|
+
any string that contains 'jpeg' will return a jpeg image,
|
|
78
|
+
otherwise create a png.
|
|
79
|
+
|
|
80
|
+
crs: [dict | None], Target image's Coordinate Reference System.
|
|
81
|
+
A dictionary with 'epsg', 'proj4' or 'wkt' key.
|
|
82
|
+
|
|
83
|
+
scale_extent: [dict | None], Scale Extents for the image. This dictionary
|
|
84
|
+
contains "x" and "y" keys each whose value which is a dictionary
|
|
85
|
+
of "min", "max" values in the same units as the crs.
|
|
86
|
+
e.g.: { "x": { "min": 0.5, "max": 125 },
|
|
87
|
+
"y": { "min": 52, "max": 75.22 } }
|
|
88
|
+
|
|
89
|
+
scale_size: [dict | None], Scale sizes for the image. The dictionary
|
|
90
|
+
contains "x" and "y" keys with the horizontal and veritcal
|
|
91
|
+
resolution in the same units as the crs.
|
|
92
|
+
e.g.: { "x": 10, "y": 10 }
|
|
93
|
+
|
|
94
|
+
height: [int | None], height of the output image in gridcells.
|
|
95
|
+
|
|
96
|
+
width: [int | none], width of the output image in gridcells.
|
|
97
|
+
|
|
98
|
+
palette: [str | ColorPalette | none], either a URL to a remote color palette
|
|
99
|
+
that is fetched and loaded or a ColorPalette object used to color
|
|
100
|
+
the output browse image. If not provided, a grayscale image is
|
|
101
|
+
generated.
|
|
102
|
+
|
|
103
|
+
logger: [Logger | None], a configured Logger object. If None a default
|
|
104
|
+
logger will be used.
|
|
105
|
+
|
|
106
|
+
Note:
|
|
107
|
+
if supplied, scale_size, scale_extent, height and width must be
|
|
108
|
+
internally consistent. To define a valid output grid:
|
|
109
|
+
* Specify scale_extent and 1 of:
|
|
110
|
+
* height and width
|
|
111
|
+
* scale_sizes (in the x and y horizontal spatial dimensions)
|
|
112
|
+
* Specify all three of the above, but ensure values are consistent
|
|
113
|
+
with one another, noting that:
|
|
114
|
+
scale_size.x = (scale_extent.x.max - scale_extent.x.min) / width
|
|
115
|
+
scale_size.y = (scale_extent.y.max - scale_extent.y.min) / height
|
|
116
|
+
|
|
117
|
+
Returns:
|
|
118
|
+
List of 3-element tuples. These are the file paths of:
|
|
119
|
+
- The output browse image
|
|
120
|
+
- Its associated ESRI world file (containing georeferencing information)
|
|
121
|
+
- The auxiliary XML file (containing duplicative georeferencing information)
|
|
122
|
+
|
|
123
|
+
|
|
124
|
+
Example Usage:
|
|
125
|
+
results = create_browse(
|
|
126
|
+
"/path/to/geotiff",
|
|
127
|
+
{
|
|
128
|
+
"mime": "image/png",
|
|
129
|
+
"crs": {"epsg": "EPSG:4326"},
|
|
130
|
+
"scale_extent": {
|
|
131
|
+
"x": {"min": -180, "max": 180},
|
|
132
|
+
"y": {"min": -90, "max": 90},
|
|
133
|
+
},
|
|
134
|
+
"scale_size": {"x": 10, "y": 10},
|
|
135
|
+
},
|
|
136
|
+
"https://remote-colortable",
|
|
137
|
+
logger,
|
|
138
|
+
)
|
|
139
|
+
|
|
140
|
+
"""
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
### library installation
|
|
144
|
+
|
|
145
|
+
The hybig-py library can be installed from PyPI but has a prerequisite
|
|
146
|
+
dependency requirement on the GDAL libraries. Ensure you have an environment
|
|
147
|
+
with the libraries available. You can check on Linux/macOS:
|
|
148
|
+
```bash
|
|
149
|
+
gdal-config --version
|
|
150
|
+
```
|
|
151
|
+
on windows (if GDAL is in your PATH):
|
|
152
|
+
```bash
|
|
153
|
+
gdalinfo --version
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
Once verified, you can simply install the libary:
|
|
157
|
+
|
|
158
|
+
```bash
|
|
159
|
+
pip install hybig-py
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
|
|
163
|
+
### Reprojection
|
|
164
|
+
|
|
165
|
+
GIBS expects to receive images in one of three Coordinate Reference System (CRS) projections.
|
|
166
|
+
|
|
167
|
+
| Region | Code | Name |
|
|
168
|
+
|-------------|-----------|-----------------------------------------------------------|
|
|
169
|
+
| north polar | EPSG:3413 | WGS 84 / NSIDC Sea Ice Polar Stereographic North |
|
|
170
|
+
| south polar | EPSG:3031 | WGS 84 / Antarctic Polar Stereographic |
|
|
171
|
+
| global | EPSG:4326 | WGS 84 -- WGS84 - World Geodetic System 1984, used in GPS |
|
|
172
|
+
|
|
173
|
+
HyBIG processing will attempt to choose a GIBS-suitable target CRS from the
|
|
174
|
+
input image or read it from the inputs. Reprojection is done by resampling via
|
|
175
|
+
nearest neighbor. It is important to note that HyBig outputs are not scientific
|
|
176
|
+
data, but browse imagery and should not be used for scientific analysis.
|
|
177
|
+
|
|
178
|
+
|
|
179
|
+
### Tiling
|
|
180
|
+
|
|
181
|
+
Large output images are divided into smaller, more manageable tiles for
|
|
182
|
+
efficient handling and processing, as per agreement with GIBS. The maximum
|
|
183
|
+
untiled image size generated by HyBIG is 67,108,864 cells (8,192 x 8,192). If
|
|
184
|
+
the output image exceeds this threshold, HyBIG automatically tiles the output
|
|
185
|
+
into multiple 4,096 x 4,096 cell images.
|
|
186
|
+
|
|
187
|
+
Tiled images are labeled with the zero-based column and row numbers inserted
|
|
188
|
+
into the output filename before its
|
|
189
|
+
extension. For example, `VCF5KYR_1991001_001_2018224205008.r01c02.png` represents the
|
|
190
|
+
second row and third column of the output tiles. The tiles at the edges are
|
|
191
|
+
truncated to fit the overall image dimensions. Currently, you cannot override
|
|
192
|
+
this behavior.
|
|
193
|
+
|
|
194
|
+
### Coloring
|
|
195
|
+
|
|
196
|
+
HyBIG images are colored in several ways. A palette can be included in the
|
|
197
|
+
input [STAC
|
|
198
|
+
Item](https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md). If
|
|
199
|
+
an Item's asset contains a value with the role of `palette`, it is assumed to
|
|
200
|
+
be a reference to a remote color table, which is fetched from the asset's
|
|
201
|
+
`href` and parsed as a GDAL color table.
|
|
202
|
+
|
|
203
|
+
If the STAC Item lacks color information, the Harmony message source is
|
|
204
|
+
searched for a related URL with a "content type" of `VisualizationURL` and a
|
|
205
|
+
"type" of `Color Map`. If found, it is presumed to be a remote color table and
|
|
206
|
+
fetched from that location.
|
|
207
|
+
|
|
208
|
+
In the absence of remote color information, the input image itself is searched
|
|
209
|
+
for a color map, which is used if present.
|
|
210
|
+
|
|
211
|
+
If no color information can be found, grayscale is used.
|
|
212
|
+
|
|
213
|
+
### Defaults
|
|
214
|
+
|
|
215
|
+
HyBIG tries to provide GIBS-appropriate default values for the browse image
|
|
216
|
+
outputs. When a user does not provide a target values for the output, HyBIG
|
|
217
|
+
will try to pick an appropriate default.
|
|
218
|
+
|
|
219
|
+
#### Coordinate Reference System (CRS)
|
|
220
|
+
|
|
221
|
+
HyBIG selects a default CRS from the list of GIBS preferred projections. The
|
|
222
|
+
steps followed are simple but effective:
|
|
223
|
+
|
|
224
|
+
1. If the `proj` is `lonlat` use global (`EPSG:4326`)
|
|
225
|
+
1. If the projection latitude of origin is above 80° N use northern (`EPSG:3413`)
|
|
226
|
+
1. If the projection latitude of origin is below -80° N use southern (`EPSG:3031`)
|
|
227
|
+
1. Otherwise use global (`EPGS:4326`)
|
|
228
|
+
|
|
229
|
+
#### Scale Extent (Image Bounds)
|
|
230
|
+
|
|
231
|
+
The default scale extent for an output image is computed by reprojecting the
|
|
232
|
+
input data boundary into the target CRS. It densifies the edges by adding 21
|
|
233
|
+
points ([rasterio's
|
|
234
|
+
default](https://rasterio.readthedocs.io/en/latest/api/rasterio.warp.html#rasterio.warp.transform_bounds))
|
|
235
|
+
to each edge before reprojection to account for non-linear edges produced by
|
|
236
|
+
the transformation ensuring inclusion of all data in the output image.
|
|
237
|
+
|
|
238
|
+
#### Dimensions / Scale Sizes
|
|
239
|
+
|
|
240
|
+
Output image dimensions can be explicitly included as `width` and `height` in
|
|
241
|
+
the harmony message or computed based on the scale extent and scale size
|
|
242
|
+
(resolution).
|
|
243
|
+
|
|
244
|
+
The dimension computations from the scale extent and scale size:
|
|
245
|
+
```
|
|
246
|
+
height = round((scale_extent['ymax'] - scale_extent['ymin']) / scale_size.y)
|
|
247
|
+
width = round((scale_extent['xmax'] - scale_extent['xmin']) / scale_size.x)
|
|
248
|
+
```
|
|
249
|
+
|
|
250
|
+
When a Harmony message contains neither `dimensions` nor `scaleSizes` a default
|
|
251
|
+
set of dimensions is computed.
|
|
252
|
+
|
|
253
|
+
For coarse input data, the resolution (scale size) is used with the scale
|
|
254
|
+
extent to compute the output dimensions. For high resolution data, finer than
|
|
255
|
+
2km per gridcell, the input resolution is used to lookup the closest GIBS
|
|
256
|
+
preferred resolution (Table 4.1.8-1 and -2 from the ICD) and the preferred
|
|
257
|
+
resolution along with the scale extent is used to compute the output image
|
|
258
|
+
dimensions.
|
|
259
|
+
|
|
260
|
+
### Customizations
|
|
261
|
+
|
|
262
|
+
Users can request customizations to the output images such as `crs`,
|
|
263
|
+
`scale_extents`, or `scale_sizes` and dimensions (`height` & `width`) in the
|
|
264
|
+
harmony request. However, the generated outputs may not be compatible with
|
|
265
|
+
GIBS.
|
|
266
|
+
|
|
267
|
+
When a user customizes `scale_extent` or `scale_size`, they must also include a
|
|
268
|
+
`crs` in the request. The units of the cusomized values must match the target
|
|
269
|
+
CRS. For example, specifying a bounding box in degrees requires a target CRS
|
|
270
|
+
also with units of degrees.
|
|
271
|
+
|
|
272
|
+
|
|
273
|
+
## Repository structure:
|
|
274
|
+
|
|
275
|
+
```
|
|
276
|
+
|- 📂 bin
|
|
277
|
+
|- 📂 docker
|
|
278
|
+
|- 📂 docs
|
|
279
|
+
|- 📂 hybig
|
|
280
|
+
|- 📂 harmony_service
|
|
281
|
+
|- 📂 tests
|
|
282
|
+
|- CHANGELOG.md
|
|
283
|
+
|- CONTRIBUTING.md
|
|
284
|
+
|- LICENSE
|
|
285
|
+
|- README.md
|
|
286
|
+
|- dev-requirements.txt
|
|
287
|
+
|- legacy-CHANGELOG.md
|
|
288
|
+
|- pip_requirements.txt
|
|
289
|
+
|- pip_requirements_skip_snyk.txt
|
|
290
|
+
|- pyproject.toml
|
|
291
|
+
|
|
292
|
+
```
|
|
293
|
+
|
|
294
|
+
* `bin` - A directory containing utility scripts to build the service and test
|
|
295
|
+
images. A script to extract the release notes for the most recent version, as
|
|
296
|
+
contained in `CHANGELOG.md` is also in this directory.
|
|
297
|
+
|
|
298
|
+
* `docker` - A directory containing the Dockerfiles for the service and test
|
|
299
|
+
images. It also contains `service_version.txt`, which contains the semantic
|
|
300
|
+
version number of the library and service image. Update this file with a new
|
|
301
|
+
version to trigger a release.
|
|
302
|
+
|
|
303
|
+
* `docs` - A directory with example usage notebooks.
|
|
304
|
+
|
|
305
|
+
* `hybig` - A directory containing Python source code for the HyBIG library.
|
|
306
|
+
This directory contains the business logic for generating GIBS compatible
|
|
307
|
+
browse images.
|
|
308
|
+
|
|
309
|
+
* `harmony_service` - A directory containing the Harmony Service specific
|
|
310
|
+
python code. `adapter.py` contains the `BrowseImageGeneratorAdapter` class
|
|
311
|
+
that is invoked by calls to the Harmony service.
|
|
312
|
+
|
|
313
|
+
* `tests` - A directory containing the service unit test suite.
|
|
314
|
+
|
|
315
|
+
* `CHANGELOG.md` - This file contains a record of changes applied to each new
|
|
316
|
+
release of HyBIG. Any release of a new version should have a record
|
|
317
|
+
of what was changed in this file.
|
|
318
|
+
|
|
319
|
+
* `CONTRIBUTING.md` - This file contains guidance for making contributions to
|
|
320
|
+
HyBIG, including recommended git best practices.
|
|
321
|
+
|
|
322
|
+
* `LICENSE` - Required for distribution under NASA open-source
|
|
323
|
+
approval. Details conditions for use, reproduction and distribution.
|
|
324
|
+
|
|
325
|
+
* `README.md` - This file, containing guidance on developing the library and
|
|
326
|
+
service.
|
|
327
|
+
|
|
328
|
+
* `dev-requirements.txt` - list of packages required for library and service
|
|
329
|
+
development.
|
|
330
|
+
|
|
331
|
+
* `legacy-CHANGELOG.md` - Notes for each version that was previously released
|
|
332
|
+
internally to EOSDIS, prior to open-source publication of the code and Docker
|
|
333
|
+
image.
|
|
334
|
+
|
|
335
|
+
* `pip_requirements.txt` - A list of service Python package dependencies.
|
|
336
|
+
|
|
337
|
+
* `pip_requirements_skip_snyk.txt` - A list of service Python package
|
|
338
|
+
dependencies that are not scanned by snyk for vulnerabilities. This file
|
|
339
|
+
contains only the `GDAL` package. It is separated because snyk's scanning is
|
|
340
|
+
naive and cannot pre-install required libraries so that `pip install GDAL`
|
|
341
|
+
fails and we have no work around.
|
|
342
|
+
|
|
343
|
+
* `pyproject.toml` - Configuration file used by packaging tools, as well as
|
|
344
|
+
other tools such as linters, type checkers, etc.
|
|
345
|
+
|
|
346
|
+
|
|
347
|
+
## Local development:
|
|
348
|
+
|
|
349
|
+
Local testing of service functionality can be achieved via a local instance of
|
|
350
|
+
[Harmony](https://github.com/nasa/harmony). Please see instructions there
|
|
351
|
+
regarding creation of a local Harmony instance.
|
|
352
|
+
|
|
353
|
+
For local development and testing of library modifications or small functions
|
|
354
|
+
independent of the main Harmony application:
|
|
355
|
+
|
|
356
|
+
1. Create a Python virtual environment
|
|
357
|
+
1. Ensure GDAL libraries are accessable in the virtual environment.
|
|
358
|
+
1. Install the dependencies in `pip_requirements.txt`, `pip_requirements_skip_snyk.txt` and `dev-requirements.txt`
|
|
359
|
+
1. Install the pre-commit hooks.
|
|
360
|
+
|
|
361
|
+
|
|
362
|
+
```
|
|
363
|
+
> conda create --name hybig-env python==3.11
|
|
364
|
+
> pip install -r pip_requirements.txt -r pip_requirements_skip_snyk.txt
|
|
365
|
+
> pip install -r dev-requirements.txt
|
|
366
|
+
|
|
367
|
+
> pre-commit install
|
|
368
|
+
```
|
|
369
|
+
|
|
370
|
+
|
|
371
|
+
## Tests:
|
|
372
|
+
|
|
373
|
+
This service utilises the Python `unittest` package to perform unit tests on
|
|
374
|
+
classes and functions in the service. After local development is complete, and
|
|
375
|
+
test have been updated, they can be run in Docker via:
|
|
376
|
+
|
|
377
|
+
```bash
|
|
378
|
+
$ ./bin/build-image
|
|
379
|
+
$ ./bin/build-test
|
|
380
|
+
$ ./bin/run-test
|
|
381
|
+
```
|
|
382
|
+
|
|
383
|
+
The `tests/run_tests.sh` script will also generate a coverage report, rendered
|
|
384
|
+
in HTML, and scan the code with `pylint`.
|
|
385
|
+
|
|
386
|
+
Currently, the `unittest` suite is run automatically within a GitHub workflow
|
|
387
|
+
as part of a CI/CD pipeline. These tests are run for all changes made in a PR
|
|
388
|
+
against the `main` branch. The tests must pass in order to merge the PR.
|
|
389
|
+
|
|
390
|
+
Unit tests are executed automatically by github actions on each Pull Request.
|
|
391
|
+
|
|
392
|
+
|
|
393
|
+
## Versioning:
|
|
394
|
+
|
|
395
|
+
Docker service images and the hybig-py package library adhere to semantic
|
|
396
|
+
version numbers: major.minor.patch.
|
|
397
|
+
|
|
398
|
+
* Major increments: These are non-backwards compatible API changes.
|
|
399
|
+
* Minor increments: These are backwards compatible API changes.
|
|
400
|
+
* Patch increments: These updates do not affect the API to the service.
|
|
401
|
+
|
|
402
|
+
## CI/CD:
|
|
403
|
+
|
|
404
|
+
The CI/CD for HyBIG is run on github actions with the workflows in the
|
|
405
|
+
`.github/workflows` directory:
|
|
406
|
+
|
|
407
|
+
* `run_lib_tests.yml` - A reusable workflow that tests the library functions
|
|
408
|
+
against the supported python versions.
|
|
409
|
+
* `run_service_tests.yml` - A reusable workflow that builds the service and
|
|
410
|
+
test Docker images, then runs the Python unit test suite in an instance of
|
|
411
|
+
the test Docker container.
|
|
412
|
+
* `run_tests_on_pull_requests.yml` - Triggered for all PRs against the `main`
|
|
413
|
+
branch. It runs the workflow in `run_service_tests.yml` and
|
|
414
|
+
`run_lib_tests.yml` to ensure all tests pass for the new code.
|
|
415
|
+
* `publish_docker_image.yml` - Triggered either manually or for commits to the
|
|
416
|
+
`main` branch that contain changes to the `docker/service_version.txt` file.
|
|
417
|
+
* `publish_to_pypi.yml` - Triggered either manually or for commits to the
|
|
418
|
+
`main` branch that contain changes to the `docker/service_version.txt`file.
|
|
419
|
+
* `publish_release.yml`<a name="release-workflow"></a> - workflow runs automatically when there is a change to
|
|
420
|
+
the `docker/service_version.txt` file on the main branch. This workflow will:
|
|
421
|
+
* Run the full unit test suite, to prevent publication of broken code.
|
|
422
|
+
* Extract the semantic version number from `docker/service_version.txt`.
|
|
423
|
+
* Extract the released notes for the most recent version from `CHANGELOG.md`.
|
|
424
|
+
* Build and deploy a this service's docker image to `ghcr.io`.
|
|
425
|
+
* Build the library package to be published to PyPI.
|
|
426
|
+
* Publish the package to PyPI.
|
|
427
|
+
* Publish a GitHub release under the semantic version number, with associated
|
|
428
|
+
git tag.
|
|
429
|
+
|
|
430
|
+
|
|
431
|
+
## Releasing
|
|
432
|
+
|
|
433
|
+
A release consists of a new version hybig-py library published to PyPI and a
|
|
434
|
+
new Docker service image published to github's container repository.
|
|
435
|
+
|
|
436
|
+
A release is made automatically when a commit to the main branch contains a
|
|
437
|
+
changes in the `docker/service_version.txt` file, see the [publish_release](#release-workflow) workflow in the CI/CD section above.
|
|
438
|
+
|
|
439
|
+
Before merging a PR that will trigger a release, ensure these two files are updated:
|
|
440
|
+
|
|
441
|
+
* `CHANGELOG.md` - Notes should be added to capture the changes to the service.
|
|
442
|
+
* `docker/service_version.txt` - The semantic version number should be updated.
|
|
443
|
+
|
|
444
|
+
The `CHANGELOG.md` file requires a specific format for a new release, as it
|
|
445
|
+
looks for the following string to define the newest release of the code
|
|
446
|
+
(starting at the top of the file).
|
|
447
|
+
|
|
448
|
+
```
|
|
449
|
+
## [vX.Y.Z] - YYYY-MM-DD
|
|
450
|
+
```
|
|
451
|
+
|
|
452
|
+
Where the markdown reference needs to be updated at the bottom of the file following the existing pattern.
|
|
453
|
+
```
|
|
454
|
+
[unreleased]:https://github.com/nasa/harmony-browse-image-generator/compare/X.Y.Z..HEAD
|
|
455
|
+
[vX.Y.Z]:https://github.com/nasa/harmony-browse-image-generator/compare/X.Y.Y..X.Y.Z
|
|
456
|
+
```
|
|
457
|
+
|
|
458
|
+
### pre-commit hooks:
|
|
459
|
+
|
|
460
|
+
This repository uses [pre-commit](https://pre-commit.com/) to enable pre-commit
|
|
461
|
+
checking the repository for some coding standard best practices. These include:
|
|
462
|
+
|
|
463
|
+
* Removing trailing whitespaces.
|
|
464
|
+
* Removing blank lines at the end of a file.
|
|
465
|
+
* JSON files have valid formats.
|
|
466
|
+
* [ruff](https://github.com/astral-sh/ruff) Python linting checks.
|
|
467
|
+
* [black](https://black.readthedocs.io/en/stable/index.html) Python code
|
|
468
|
+
formatting checks.
|
|
469
|
+
|
|
470
|
+
To enable these checks locally:
|
|
471
|
+
|
|
472
|
+
```bash
|
|
473
|
+
# Install pre-commit Python package as part of test requirements:
|
|
474
|
+
pip install -r tests/pip_test_requirements.txt
|
|
475
|
+
|
|
476
|
+
# Install the git hook scripts:
|
|
477
|
+
pre-commit install
|
|
478
|
+
|
|
479
|
+
# (Optional) Run against all files:
|
|
480
|
+
pre-commit run --all-files
|
|
481
|
+
```
|
|
482
|
+
|
|
483
|
+
When you try to make a new commit locally, `pre-commit` will automatically run.
|
|
484
|
+
If any of the hooks detect non-compliance (e.g., trailing whitespace), that
|
|
485
|
+
hook will state it failed, and also try to fix the issue. You will need to
|
|
486
|
+
review and `git add` the changes before you can make a commit.
|
|
487
|
+
|
|
488
|
+
It is planned to implement additional hooks, possibly including tools such as
|
|
489
|
+
`mypy`.
|
|
490
|
+
|
|
491
|
+
[pre-commit.ci](pre-commit.ci) is configured such that these same hooks will be
|
|
492
|
+
automatically run for every pull request.
|
|
493
|
+
|
|
494
|
+
## Releasing a new version of the service:
|
|
495
|
+
|
|
496
|
+
Once a new Docker image has been published with a new semantic version tag,
|
|
497
|
+
that service version can be released to a Harmony environment by following the
|
|
498
|
+
directions in the [Harmony Managing Existing Services
|
|
499
|
+
Guide](https://github.com/nasa/harmony/blob/main/docs/guides/managing-existing-services.md).
|
|
500
|
+
|
|
501
|
+
## Get in touch:
|
|
502
|
+
|
|
503
|
+
You can reach out to the maintainers of this repository via email:
|
|
504
|
+
|
|
505
|
+
* david.p.auty@nasa.gov
|
|
506
|
+
* matthew.savoie@colorado.edu
|
|
507
|
+
* owen.m.littlejohns@nasa.gov
|