sfeos-helpers 6.5.1__py3-none-any.whl → 6.6.0__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,43 @@
1
+ Metadata-Version: 2.4
2
+ Name: sfeos_helpers
3
+ Version: 6.6.0
4
+ Summary: Helper library for the Elasticsearch and Opensearch stac-fastapi backends.
5
+ Project-URL: Homepage, https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch
6
+ License: MIT
7
+ Keywords: Elasticsearch,FastAPI,Opensearch,STAC,STAC-API,stac-fastapi
8
+ Classifier: Intended Audience :: Developers
9
+ Classifier: Intended Audience :: Information Technology
10
+ Classifier: Intended Audience :: Science/Research
11
+ Classifier: License :: OSI Approved :: MIT License
12
+ Classifier: Programming Language :: Python :: 3.9
13
+ Classifier: Programming Language :: Python :: 3.10
14
+ Classifier: Programming Language :: Python :: 3.11
15
+ Classifier: Programming Language :: Python :: 3.12
16
+ Classifier: Programming Language :: Python :: 3.13
17
+ Classifier: Programming Language :: Python :: 3.14
18
+ Requires-Python: >=3.9
19
+ Requires-Dist: stac-fastapi-core==6.6.0
20
+ Description-Content-Type: text/markdown
21
+
22
+ # sfeos-helpers
23
+
24
+ Helper utilities for the stac-fastapi project. For full documentation, please see the [main README](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/blob/main/README.md).
25
+
26
+ ## Package Information
27
+
28
+ - **Package name**: sfeos-helpers
29
+ - **Description**: Helper utilities for the stac-fastapi project.
30
+ - **Documentation**: [https://stac-utils.github.io/stac-fastapi-elasticsearch-opensearch/](https://stac-utils.github.io/stac-fastapi-elasticsearch-opensearch/)
31
+ - **Source**: [GitHub Repository](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/)
32
+
33
+ ## Installation
34
+
35
+ This package is a dependency of stac-fastapi-elasticsearch and stac-fastapi-opensearch and is typically installed automatically.
36
+
37
+ ```bash
38
+ pip install stac-fastapi-elasticsearch # or stac-fastapi-opensearch
39
+ ```
40
+
41
+ ## Quick Start
42
+
43
+ For detailed usage and examples, please refer to the [main documentation](https://stac-utils.github.io/stac-fastapi-elasticsearch-opensearch/).
@@ -1,15 +1,18 @@
1
- stac_fastapi/sfeos_helpers/mappings.py,sha256=u7ZHyUu3NPS_sXaflfQPSP2Yleld3jgqCG0yKcQ64w4,8600
2
- stac_fastapi/sfeos_helpers/version.py,sha256=FuGC3fKnAmD4Wk95swJ6qCVBs5mZiShrlRKuSH-voyE,45
1
+ stac_fastapi/sfeos_helpers/mappings.py,sha256=IqYd0tQUG-FnWVg6Fp1MU9audntF_-pMGm-BUTGbZMM,8596
2
+ stac_fastapi/sfeos_helpers/version.py,sha256=zPQp-GtLh4R45hT8V_KAWzwZcP4-jyw7Xjms_eNMZBc,45
3
+ stac_fastapi/sfeos_helpers/aggregation/README.md,sha256=SDlvCOpKyaJrlJvx84T2RzCnGALe_PK51zNeo3RP9ac,2122
3
4
  stac_fastapi/sfeos_helpers/aggregation/__init__.py,sha256=Mym17lFh90by1GnoQgMyIKAqRNJnvCgVSXDYzjBiPQk,1210
4
5
  stac_fastapi/sfeos_helpers/aggregation/client.py,sha256=PPUk0kAZnms46FlLGrR5w8wa52vG-dT6BG37896R5CY,17939
5
6
  stac_fastapi/sfeos_helpers/aggregation/format.py,sha256=qUW1jjh2EEjy-V7riliFR77grpi-AgsTmP76z60K5Lo,2011
6
- stac_fastapi/sfeos_helpers/database/__init__.py,sha256=T0YwePfhG3ukL1oUFCh3FYHA9jZZe36FJRYCQplfb18,2645
7
+ stac_fastapi/sfeos_helpers/database/README.md,sha256=TVYFDD4PqDD57ZsWBv4i4LawaL_DAEIOjM6OQuqwLAU,4049
8
+ stac_fastapi/sfeos_helpers/database/__init__.py,sha256=Kvnz8hpXq_sSz8K5OW3PoPsvh9864Vv1zWhI5hxgd4o,2891
7
9
  stac_fastapi/sfeos_helpers/database/datetime.py,sha256=XMyi9Q09cuP_hj97qbGbHFtelq7WQVPdehUfzqNZFV4,4040
8
10
  stac_fastapi/sfeos_helpers/database/document.py,sha256=LtjX15gvaOuZC_k2t_oQhys_c-zRTLN5rwX0hNJkHnM,1725
9
11
  stac_fastapi/sfeos_helpers/database/index.py,sha256=g7_sKfd5XUwq4IhdKRNiasejk045dKlullsdeDSZTq8,6585
10
12
  stac_fastapi/sfeos_helpers/database/mapping.py,sha256=4-MSd4xH5wg7yoC4aPjzYMDSEvP026bw4k2TfffMT5E,1387
11
- stac_fastapi/sfeos_helpers/database/query.py,sha256=g2iGdfgqpx6o8GoQJBMl3AMmqcbSf792qvKWfWipR5w,4193
12
- stac_fastapi/sfeos_helpers/database/utils.py,sha256=9zU9hEglZb6f-uxOhd95saSw2id9w5PR36dWtyfXTb0,8757
13
+ stac_fastapi/sfeos_helpers/database/query.py,sha256=bbSYe0cLC7oFbhkHR5WTKCF7Ca9iZI3fdanD90KYN98,9476
14
+ stac_fastapi/sfeos_helpers/database/utils.py,sha256=CLtZgoUT37oklc9MsExXsxDviv4bzK-ZP7oxAOXS32Y,11780
15
+ stac_fastapi/sfeos_helpers/filter/README.md,sha256=HT8h7RN7iYgUb-Tf3kOrS2C7_I7dHBZCw1CV1_XVMr4,1947
13
16
  stac_fastapi/sfeos_helpers/filter/__init__.py,sha256=n3zL_MhEGOoxMz1KeijyK_UKiZ0MKPl90zHtYI5RAy8,1557
14
17
  stac_fastapi/sfeos_helpers/filter/client.py,sha256=QQ6gnUEMqzS_qG8G5QHcX9x3LNk5JvCp1rRJltMTmM4,4414
15
18
  stac_fastapi/sfeos_helpers/filter/cql2.py,sha256=Cg9kRYD9CVkVSyRqOyB5oVXmlyteSn2bw88sqklGpUM,955
@@ -26,7 +29,6 @@ stac_fastapi/sfeos_helpers/search_engine/selection/base.py,sha256=106c4FK50cgMmT
26
29
  stac_fastapi/sfeos_helpers/search_engine/selection/cache_manager.py,sha256=5yrgf9JA4mgRNMPDKih6xySF8mD724lEWnXhWud7m2c,4039
27
30
  stac_fastapi/sfeos_helpers/search_engine/selection/factory.py,sha256=vbgNVCUW2lviePqzpgsPLxp6IEqcX3GHiahqN2oVObA,1305
28
31
  stac_fastapi/sfeos_helpers/search_engine/selection/selectors.py,sha256=q83nfCfNfLUqtkHpORwNHNRU9Pa-heeaDIPO0RlHb-8,4779
29
- sfeos_helpers-6.5.1.dist-info/METADATA,sha256=yCy4qbL5pERMI0QAwOwK2PKff7u0Cus4BLjRr8xuGHg,41326
30
- sfeos_helpers-6.5.1.dist-info/WHEEL,sha256=tZoeGjtWxWRfdplE7E3d45VPlLNQnvbKiYnx7gwAy8A,92
31
- sfeos_helpers-6.5.1.dist-info/top_level.txt,sha256=vqn-D9-HsRPTTxy0Vk_KkDmTiMES4owwBQ3ydSZYb2s,13
32
- sfeos_helpers-6.5.1.dist-info/RECORD,,
32
+ sfeos_helpers-6.6.0.dist-info/METADATA,sha256=g9JGbrU5C9k4Tem68ICmSHSJbd1gsHRFCvO--USKuWc,1911
33
+ sfeos_helpers-6.6.0.dist-info/WHEEL,sha256=qtCwoSJWgHk21S1Kb4ihdzI2rlJ1ZKaIurTj_ngOhyQ,87
34
+ sfeos_helpers-6.6.0.dist-info/RECORD,,
@@ -1,5 +1,4 @@
1
1
  Wheel-Version: 1.0
2
- Generator: bdist_wheel (0.45.1)
2
+ Generator: hatchling 1.27.0
3
3
  Root-Is-Purelib: true
4
4
  Tag: py3-none-any
5
-
@@ -0,0 +1,57 @@
1
+ # STAC FastAPI Aggregation Package
2
+
3
+ This package contains shared aggregation functionality used by both the Elasticsearch and OpenSearch implementations of STAC FastAPI. It helps reduce code duplication and ensures consistent behavior between the two implementations.
4
+
5
+ ## Package Structure
6
+
7
+ The aggregation package is organized into three main modules:
8
+
9
+ - **client.py**: Contains the base aggregation client implementation
10
+ - `EsAsyncBaseAggregationClient`: The main class that implements the STAC aggregation extension for Elasticsearch/OpenSearch
11
+ - Methods for handling aggregation requests, validating parameters, and formatting responses
12
+
13
+ - **format.py**: Contains functions for formatting aggregation responses
14
+ - `frequency_agg`: Formats frequency distribution aggregation responses
15
+ - `metric_agg`: Formats metric aggregation responses
16
+
17
+ - **__init__.py**: Package initialization and exports
18
+ - Exports the main classes and functions for use by other modules
19
+
20
+ ## Features
21
+
22
+ The aggregation package provides the following features:
23
+
24
+ - Support for various aggregation types:
25
+ - Datetime frequency
26
+ - Collection frequency
27
+ - Property frequency
28
+ - Geospatial grid aggregations (geohash, geohex, geotile)
29
+ - Metric aggregations (min, max, etc.)
30
+
31
+ - Parameter validation:
32
+ - Precision validation for geospatial aggregations
33
+ - Interval validation for datetime aggregations
34
+
35
+ - Response formatting:
36
+ - Consistent response structure
37
+ - Proper typing and documentation
38
+
39
+ ## Usage
40
+
41
+ The aggregation package is used by the Elasticsearch and OpenSearch implementations to provide aggregation functionality for STAC API. The main entry point is the `EsAsyncBaseAggregationClient` class, which is instantiated in the respective app.py files.
42
+
43
+ Example:
44
+ ```python
45
+ from stac_fastapi.sfeos_helpers.aggregation import EsAsyncBaseAggregationClient
46
+
47
+ # Create an instance of the aggregation client
48
+ aggregation_client = EsAsyncBaseAggregationClient(database)
49
+
50
+ # Register the aggregation extension with the API
51
+ api = StacApi(
52
+ ...,
53
+ extensions=[
54
+ ...,
55
+ AggregationExtension(client=aggregation_client),
56
+ ],
57
+ )
@@ -0,0 +1,61 @@
1
+ # STAC FastAPI Database Package
2
+
3
+ This package contains shared database operations used by both the Elasticsearch and OpenSearch
4
+ implementations of STAC FastAPI. It helps reduce code duplication and ensures consistent behavior
5
+ between the two implementations.
6
+
7
+ ## Package Structure
8
+
9
+ The database package is organized into five main modules:
10
+
11
+ - **index.py**: Contains functions for managing indices
12
+ - [create_index_templates_shared](cci:1://file:///home/computer/Code/stac-fastapi-elasticsearch-opensearch/stac_fastapi/sfeos_helpers/stac_fastapi/sfeos_helpers/database_logic_helpers.py:15:0-48:33): Creates index templates for Collections and Items
13
+ - [delete_item_index_shared](cci:1://file:///home/computer/Code/stac-fastapi-elasticsearch-opensearch/stac_fastapi/sfeos_helpers/stac_fastapi/sfeos_helpers/database_logic_helpers.py:128:0-153:30): Deletes an item index for a collection
14
+ - [index_by_collection_id](cci:1://file:///home/computer/Code/stac-fastapi-elasticsearch-opensearch/stac_fastapi/sfeos_helpers/stac_fastapi/sfeos_helpers/utilities.py:86:0-100:5): Translates a collection ID into an index name
15
+ - [index_alias_by_collection_id](cci:1://file:///home/computer/Code/stac-fastapi-elasticsearch-opensearch/stac_fastapi/sfeos_helpers/stac_fastapi/sfeos_helpers/utilities.py:103:0-115:5): Translates a collection ID into an index alias
16
+ - [indices](cci:1://file:///home/computer/Code/stac-fastapi-elasticsearch-opensearch/stac_fastapi/sfeos_helpers/stac_fastapi/sfeos_helpers/utilities.py:118:0-132:5): Gets a comma-separated string of index names
17
+
18
+ - **query.py**: Contains functions for building and manipulating queries
19
+ - [apply_free_text_filter_shared](cci:1://file:///home/computer/Code/stac-fastapi-elasticsearch-opensearch/stac_fastapi/sfeos_helpers/stac_fastapi/sfeos_helpers/database_logic_helpers.py:51:0-74:16): Applies a free text filter to a search
20
+ - [apply_intersects_filter_shared](cci:1://file:///home/computer/Code/stac-fastapi-elasticsearch-opensearch/stac_fastapi/sfeos_helpers/stac_fastapi/sfeos_helpers/database_logic_helpers.py:77:0-104:5): Creates a geo_shape filter for intersecting geometry
21
+ - [populate_sort_shared](cci:1://file:///home/computer/Code/stac-fastapi-elasticsearch-opensearch/stac_fastapi/sfeos_helpers/stac_fastapi/sfeos_helpers/database_logic_helpers.py:107:0-125:16): Creates a sort configuration for queries
22
+
23
+ - **mapping.py**: Contains functions for working with mappings
24
+ - [get_queryables_mapping_shared](cci:1://file:///home/computer/Code/stac-fastapi-elasticsearch-opensearch/stac_fastapi/sfeos_helpers/stac_fastapi/sfeos_helpers/database_logic_helpers.py:156:0-185:27): Retrieves mapping of Queryables for search
25
+
26
+ - **document.py**: Contains functions for working with documents
27
+ - [mk_item_id](cci:1://file:///home/computer/Code/stac-fastapi-elasticsearch-opensearch/stac_fastapi/sfeos_helpers/stac_fastapi/sfeos_helpers/utilities.py:140:0-150:5): Creates a document ID for an Item
28
+ - [mk_actions](cci:1://file:///home/computer/Code/stac-fastapi-elasticsearch-opensearch/stac_fastapi/sfeos_helpers/stac_fastapi/sfeos_helpers/utilities.py:153:0-175:5): Creates bulk actions for indexing items
29
+
30
+ - **utils.py**: Contains utility functions for database operations
31
+ - [validate_refresh](cci:1://file:///home/computer/Code/stac-fastapi-elasticsearch-opensearch/stac_fastapi/sfeos_helpers/stac_fastapi/sfeos_helpers/utilities.py:41:0-78:5): Validates the refresh parameter value
32
+
33
+ ## Usage
34
+
35
+ Import the necessary components from the database package:
36
+
37
+ ```python
38
+ from stac_fastapi.sfeos_helpers.database import (
39
+ # Index operations
40
+ create_index_templates_shared,
41
+ delete_item_index_shared,
42
+ index_alias_by_collection_id,
43
+ index_by_collection_id,
44
+ indices,
45
+
46
+ # Query operations
47
+ apply_free_text_filter_shared,
48
+ apply_intersects_filter_shared,
49
+ populate_sort_shared,
50
+
51
+ # Mapping operations
52
+ get_queryables_mapping_shared,
53
+
54
+ # Document operations
55
+ mk_item_id,
56
+ mk_actions,
57
+
58
+ # Utility functions
59
+ validate_refresh,
60
+ )
61
+ ```
@@ -42,11 +42,13 @@ from .index import (
42
42
  )
43
43
  from .mapping import get_queryables_mapping_shared
44
44
  from .query import (
45
+ apply_collections_bbox_filter_shared,
46
+ apply_collections_datetime_filter_shared,
45
47
  apply_free_text_filter_shared,
46
48
  apply_intersects_filter_shared,
47
49
  populate_sort_shared,
48
50
  )
49
- from .utils import get_bool_env, validate_refresh
51
+ from .utils import add_bbox_shape_to_collection, get_bool_env, validate_refresh
50
52
 
51
53
  __all__ = [
52
54
  # Index operations
@@ -59,6 +61,8 @@ __all__ = [
59
61
  # Query operations
60
62
  "apply_free_text_filter_shared",
61
63
  "apply_intersects_filter_shared",
64
+ "apply_collections_bbox_filter_shared",
65
+ "apply_collections_datetime_filter_shared",
62
66
  "populate_sort_shared",
63
67
  # Mapping operations
64
68
  "get_queryables_mapping_shared",
@@ -68,6 +72,7 @@ __all__ = [
68
72
  # Utility functions
69
73
  "validate_refresh",
70
74
  "get_bool_env",
75
+ "add_bbox_shape_to_collection",
71
76
  # Datetime utilities
72
77
  "return_date",
73
78
  "extract_date",
@@ -3,8 +3,10 @@
3
3
  This module provides functions for building and manipulating Elasticsearch/OpenSearch queries.
4
4
  """
5
5
 
6
- from typing import Any, Dict, List, Optional
6
+ import logging
7
+ from typing import Any, Dict, List, Optional, Union
7
8
 
9
+ from stac_fastapi.core.utilities import bbox2polygon
8
10
  from stac_fastapi.sfeos_helpers.mappings import Geometry
9
11
 
10
12
  ES_MAX_URL_LENGTH = 4096
@@ -66,6 +68,139 @@ def apply_intersects_filter_shared(
66
68
  }
67
69
 
68
70
 
71
+ def apply_collections_datetime_filter_shared(
72
+ datetime_str: Optional[str],
73
+ ) -> Optional[Dict[str, Any]]:
74
+ """Create a temporal filter for collections based on their extent.
75
+
76
+ Args:
77
+ datetime_str: The datetime parameter. Can be:
78
+ - A single datetime string (e.g., "2020-01-01T00:00:00Z")
79
+ - A datetime range with "/" separator (e.g., "2020-01-01T00:00:00Z/2021-01-01T00:00:00Z")
80
+ - Open-ended ranges using ".." (e.g., "../2021-01-01T00:00:00Z" or "2020-01-01T00:00:00Z/..")
81
+ - None if no datetime filter is provided
82
+
83
+ Returns:
84
+ Optional[Dict[str, Any]]: A dictionary containing the temporal filter configuration
85
+ that can be used with Elasticsearch/OpenSearch queries, or None if datetime_str is None.
86
+ Example return value:
87
+ {
88
+ "bool": {
89
+ "must": [
90
+ {"range": {"extent.temporal.interval": {"lte": "2021-01-01T00:00:00Z"}}},
91
+ {"range": {"extent.temporal.interval": {"gte": "2020-01-01T00:00:00Z"}}}
92
+ ]
93
+ }
94
+ }
95
+
96
+ Notes:
97
+ - This function is specifically for filtering collections by their temporal extent
98
+ - It queries the extent.temporal.interval field
99
+ - Open-ended ranges (..) are replaced with concrete dates (1800-01-01 for start, 2999-12-31 for end)
100
+ """
101
+ if not datetime_str:
102
+ return None
103
+
104
+ # Parse the datetime string into start and end
105
+ if "/" in datetime_str:
106
+ start, end = datetime_str.split("/")
107
+ # Replace open-ended ranges with concrete dates
108
+ if start == "..":
109
+ # For open-ended start, use a very early date
110
+ start = "1800-01-01T00:00:00Z"
111
+ if end == "..":
112
+ # For open-ended end, use a far future date
113
+ end = "2999-12-31T23:59:59Z"
114
+ else:
115
+ # If it's just a single date, use it for both start and end
116
+ start = end = datetime_str
117
+
118
+ return {
119
+ "bool": {
120
+ "must": [
121
+ # Check if any date in the array is less than or equal to the query end date
122
+ # This will match if the collection's start date is before or equal to the query end date
123
+ {"range": {"extent.temporal.interval": {"lte": end}}},
124
+ # Check if any date in the array is greater than or equal to the query start date
125
+ # This will match if the collection's end date is after or equal to the query start date
126
+ {"range": {"extent.temporal.interval": {"gte": start}}},
127
+ ]
128
+ }
129
+ }
130
+
131
+
132
+ def apply_collections_bbox_filter_shared(
133
+ bbox: Union[str, List[float], None]
134
+ ) -> Optional[Dict[str, Dict]]:
135
+ """Create a geo_shape filter for collections bbox search.
136
+
137
+ This function handles bbox parsing from both GET requests (string format) and POST requests
138
+ (list format), and constructs a geo_shape query for filtering collections by their bbox_shape field.
139
+
140
+ Args:
141
+ bbox: The bounding box parameter. Can be:
142
+ - A string of comma-separated coordinates (from GET requests)
143
+ - A list of floats [minx, miny, maxx, maxy] for 2D bbox
144
+ - None if no bbox filter is provided
145
+
146
+ Returns:
147
+ Optional[Dict[str, Dict]]: A dictionary containing the geo_shape filter configuration
148
+ that can be used with Elasticsearch/OpenSearch queries, or None if bbox is invalid.
149
+ Example return value:
150
+ {
151
+ "geo_shape": {
152
+ "bbox_shape": {
153
+ "shape": {
154
+ "type": "Polygon",
155
+ "coordinates": [[[minx, miny], [maxx, miny], [maxx, maxy], [minx, maxy], [minx, miny]]]
156
+ },
157
+ "relation": "intersects"
158
+ }
159
+ }
160
+ }
161
+
162
+ Notes:
163
+ - This function is specifically for filtering collections by their spatial extent
164
+ - It queries the bbox_shape field (not the geometry field used for items)
165
+ - The bbox is expected to be 2D (4 values) after any 3D to 2D conversion in the API layer
166
+ """
167
+ logger = logging.getLogger(__name__)
168
+
169
+ if not bbox:
170
+ return None
171
+
172
+ # Parse bbox if it's a string (from GET requests)
173
+ if isinstance(bbox, str):
174
+ try:
175
+ bbox = [float(x.strip()) for x in bbox.split(",")]
176
+ except (ValueError, AttributeError) as e:
177
+ logger.error(f"Invalid bbox format: {bbox}, error: {e}")
178
+ return None
179
+
180
+ if not bbox or len(bbox) != 4:
181
+ if bbox:
182
+ logger.warning(
183
+ f"bbox has incorrect number of coordinates (length={len(bbox)}), expected 4 (2D bbox)"
184
+ )
185
+ return None
186
+
187
+ # Convert bbox to a polygon for geo_shape query
188
+ bbox_polygon = {
189
+ "type": "Polygon",
190
+ "coordinates": bbox2polygon(bbox[0], bbox[1], bbox[2], bbox[3]),
191
+ }
192
+
193
+ # Return geo_shape query for bbox_shape field
194
+ return {
195
+ "geo_shape": {
196
+ "bbox_shape": {
197
+ "shape": bbox_polygon,
198
+ "relation": "intersects",
199
+ }
200
+ }
201
+ }
202
+
203
+
69
204
  def populate_sort_shared(sortby: List) -> Optional[Dict[str, Dict[str, str]]]:
70
205
  """Create a sort configuration for Elasticsearch/OpenSearch queries.
71
206
 
@@ -5,9 +5,9 @@ in Elasticsearch/OpenSearch, such as parameter validation.
5
5
  """
6
6
 
7
7
  import logging
8
- from typing import Dict, List, Union
8
+ from typing import Any, Dict, List, Union
9
9
 
10
- from stac_fastapi.core.utilities import get_bool_env
10
+ from stac_fastapi.core.utilities import bbox2polygon, get_bool_env
11
11
  from stac_fastapi.extensions.core.transaction.request import (
12
12
  PatchAddReplaceTest,
13
13
  PatchOperation,
@@ -15,6 +15,84 @@ from stac_fastapi.extensions.core.transaction.request import (
15
15
  )
16
16
  from stac_fastapi.sfeos_helpers.models.patch import ElasticPath, ESCommandSet
17
17
 
18
+ logger = logging.getLogger(__name__)
19
+
20
+
21
+ def add_bbox_shape_to_collection(collection: Dict[str, Any]) -> bool:
22
+ """Add bbox_shape field to a collection document for spatial queries.
23
+
24
+ This function extracts the bounding box from a collection's spatial extent
25
+ and converts it to a GeoJSON polygon shape that can be used for geospatial
26
+ queries in Elasticsearch/OpenSearch.
27
+
28
+ Args:
29
+ collection: Collection document dictionary to modify in-place.
30
+
31
+ Returns:
32
+ bool: True if bbox_shape was added, False if it was skipped (already exists,
33
+ no spatial extent, or invalid bbox).
34
+
35
+ Notes:
36
+ - Modifies the collection dictionary in-place by adding a 'bbox_shape' field
37
+ - Handles both 2D [minx, miny, maxx, maxy] and 3D [minx, miny, minz, maxx, maxy, maxz] bboxes
38
+ - Uses the first bbox if multiple are present in the collection
39
+ - Logs warnings for collections with invalid or missing bbox data
40
+ """
41
+ collection_id = collection.get("id", "unknown")
42
+
43
+ # Check if bbox_shape already exists
44
+ if "bbox_shape" in collection:
45
+ logger.debug(
46
+ f"Collection '{collection_id}' already has bbox_shape field, skipping"
47
+ )
48
+ return False
49
+
50
+ # Check if collection has spatial extent
51
+ if "extent" not in collection or "spatial" not in collection["extent"]:
52
+ logger.warning(f"Collection '{collection_id}' has no spatial extent, skipping")
53
+ return False
54
+
55
+ spatial_extent = collection["extent"]["spatial"]
56
+ if "bbox" not in spatial_extent or not spatial_extent["bbox"]:
57
+ logger.warning(
58
+ f"Collection '{collection_id}' has no bbox in spatial extent, skipping"
59
+ )
60
+ return False
61
+
62
+ # Get the first bbox (collections can have multiple bboxes, but we use the first one)
63
+ bbox = (
64
+ spatial_extent["bbox"][0]
65
+ if isinstance(spatial_extent["bbox"][0], list)
66
+ else spatial_extent["bbox"]
67
+ )
68
+
69
+ if len(bbox) < 4:
70
+ logger.warning(
71
+ f"Collection '{collection_id}': bbox has insufficient coordinates (length={len(bbox)}), expected at least 4"
72
+ )
73
+ return False
74
+
75
+ # Extract 2D coordinates (bbox can be 2D [minx, miny, maxx, maxy] or 3D [minx, miny, minz, maxx, maxy, maxz])
76
+ # For 2D polygon, we only need the x,y coordinates and discard altitude (z) values
77
+ minx, miny = bbox[0], bbox[1]
78
+ if len(bbox) == 4:
79
+ # 2D bbox: [minx, miny, maxx, maxy]
80
+ maxx, maxy = bbox[2], bbox[3]
81
+ else:
82
+ # 3D bbox: [minx, miny, minz, maxx, maxy, maxz]
83
+ # Extract indices 3,4 for maxx,maxy - discarding altitude at indices 2 (minz) and 5 (maxz)
84
+ maxx, maxy = bbox[3], bbox[4]
85
+
86
+ # Convert bbox to GeoJSON polygon
87
+ bbox_polygon_coords = bbox2polygon(minx, miny, maxx, maxy)
88
+ collection["bbox_shape"] = {
89
+ "type": "Polygon",
90
+ "coordinates": bbox_polygon_coords,
91
+ }
92
+
93
+ logger.debug(f"Collection '{collection_id}': Added bbox_shape field")
94
+ return True
95
+
18
96
 
19
97
  def validate_refresh(value: Union[str, bool]) -> str:
20
98
  """
@@ -0,0 +1,27 @@
1
+ # STAC FastAPI Filter Package
2
+
3
+ This package contains shared filter extension functionality used by both the Elasticsearch and OpenSearch
4
+ implementations of STAC FastAPI. It helps reduce code duplication and ensures consistent behavior
5
+ between the two implementations.
6
+
7
+ ## Package Structure
8
+
9
+ The filter package is organized into three main modules:
10
+
11
+ - **cql2.py**: Contains functions for converting CQL2 patterns to Elasticsearch/OpenSearch compatible formats
12
+ - [cql2_like_to_es](cci:1://file:///home/computer/Code/stac-fastapi-elasticsearch-opensearch/stac_fastapi/sfeos_helpers/stac_fastapi/sfeos_helpers/filter.py:59:0-75:5): Converts CQL2 "LIKE" characters to Elasticsearch "wildcard" characters
13
+ - [_replace_like_patterns](cci:1://file:///home/computer/Code/stac-fastapi-elasticsearch-opensearch/stac_fastapi/sfeos_helpers/stac_fastapi/sfeos_helpers/filter.py:51:0-56:71): Helper function for pattern replacement
14
+
15
+ - **transform.py**: Contains functions for transforming CQL2 queries to Elasticsearch/OpenSearch query DSL
16
+ - [to_es_field](cci:1://file:///home/computer/Code/stac-fastapi-elasticsearch-opensearch/stac_fastapi/sfeos_helpers/stac_fastapi/sfeos_helpers/filter.py:83:0-93:47): Maps field names using queryables mapping
17
+ - [to_es](cci:1://file:///home/computer/Code/stac-fastapi-elasticsearch-opensearch/stac_fastapi/sfeos_helpers/stac_fastapi/sfeos_helpers/filter.py:96:0-201:13): Transforms CQL2 query structures to Elasticsearch/OpenSearch query DSL
18
+
19
+ - **client.py**: Contains the base filter client implementation
20
+ - [EsAsyncBaseFiltersClient](cci:2://file:///home/computer/Code/stac-fastapi-elasticsearch-opensearch/stac_fastapi/sfeos_helpers/stac_fastapi/sfeos_helpers/filter.py:209:0-293:25): Base class for implementing the STAC filter extension
21
+
22
+ ## Usage
23
+
24
+ Import the necessary components from the filter package:
25
+
26
+ ```python
27
+ from stac_fastapi.sfeos_helpers.filter import cql2_like_to_es, to_es, EsAsyncBaseFiltersClient
@@ -160,7 +160,7 @@ ES_COLLECTIONS_MAPPINGS = {
160
160
  "dynamic_templates": ES_MAPPINGS_DYNAMIC_TEMPLATES,
161
161
  "properties": {
162
162
  "id": {"type": "keyword"},
163
- "extent.spatial.bbox": {"type": "long"},
163
+ "bbox_shape": {"type": "geo_shape"},
164
164
  "extent.temporal.interval": {
165
165
  "type": "date",
166
166
  "format": "strict_date_optional_time||epoch_millis",
@@ -1,2 +1,2 @@
1
1
  """library version."""
2
- __version__ = "6.5.1"
2
+ __version__ = "6.6.0"
@@ -1,714 +0,0 @@
1
- Metadata-Version: 2.1
2
- Name: sfeos-helpers
3
- Version: 6.5.1
4
- Summary: Helper library for the Elasticsearch and Opensearch stac-fastapi backends.
5
- Home-page: https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch
6
- License: MIT
7
- Classifier: Intended Audience :: Developers
8
- Classifier: Intended Audience :: Information Technology
9
- Classifier: Intended Audience :: Science/Research
10
- Classifier: Programming Language :: Python :: 3.9
11
- Classifier: Programming Language :: Python :: 3.10
12
- Classifier: Programming Language :: Python :: 3.11
13
- Classifier: Programming Language :: Python :: 3.12
14
- Classifier: Programming Language :: Python :: 3.13
15
- Classifier: License :: OSI Approved :: MIT License
16
- Requires-Python: >=3.9
17
- Description-Content-Type: text/markdown
18
- Requires-Dist: stac-fastapi.core==6.5.1
19
-
20
- # stac-fastapi-elasticsearch-opensearch
21
-
22
- <!-- markdownlint-disable MD033 MD041 -->
23
-
24
-
25
- <p align="left">
26
- <img src="https://raw.githubusercontent.com/stac-utils/stac-fastapi-elasticsearch-opensearch/refs/heads/main/assets/sfeos.png" width=1000>
27
- </p>
28
-
29
- **Jump to:** [Project Introduction](#project-introduction---what-is-sfeos) | [Quick Start](#quick-start) | [Table of Contents](#table-of-contents)
30
-
31
- [![Downloads](https://static.pepy.tech/badge/stac-fastapi-core?color=blue)](https://pepy.tech/project/stac-fastapi-core)
32
- [![GitHub contributors](https://img.shields.io/github/contributors/stac-utils/stac-fastapi-elasticsearch-opensearch?color=blue)](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/graphs/contributors)
33
- [![GitHub stars](https://img.shields.io/github/stars/stac-utils/stac-fastapi-elasticsearch-opensearch.svg?color=blue)](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/stargazers)
34
- [![GitHub forks](https://img.shields.io/github/forks/stac-utils/stac-fastapi-elasticsearch-opensearch.svg?color=blue)](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/network/members)
35
- [![PyPI version](https://img.shields.io/pypi/v/stac-fastapi-elasticsearch.svg?color=blue)](https://pypi.org/project/stac-fastapi-elasticsearch/)
36
- [![STAC](https://img.shields.io/badge/STAC-1.1.0-blue.svg)](https://github.com/radiantearth/stac-spec/tree/v1.1.0)
37
- [![stac-fastapi](https://img.shields.io/badge/stac--fastapi-6.0.0-blue.svg)](https://github.com/stac-utils/stac-fastapi)
38
-
39
- ## Sponsors & Supporters
40
-
41
- The following organizations have contributed time and/or funding to support the development of this project:
42
-
43
- <p align="left">
44
- <a href="https://healy-hyperspatial.github.io/"><img src="https://raw.githubusercontent.com/stac-utils/stac-fastapi-elasticsearch-opensearch/refs/heads/main/assets/hh-logo-blue.png" alt="Healy Hyperspatial" height="100" hspace="20"></a>
45
- <a href="https://atomicmaps.io/"><img src="https://raw.githubusercontent.com/stac-utils/stac-fastapi-elasticsearch-opensearch/refs/heads/main/assets/am-logo-black.png" alt="Atomic Maps" height="100" hspace="20"></a>
46
- <a href="https://remotesensing.vito.be/"><img src="https://raw.githubusercontent.com/stac-utils/stac-fastapi-elasticsearch-opensearch/refs/heads/main/assets/VITO.png" alt="VITO Remote Sensing" height="100" hspace="20"></a>
47
- </p>
48
-
49
- ## Project Introduction - What is SFEOS?
50
-
51
- SFEOS (stac-fastapi-elasticsearch-opensearch) is a high-performance, scalable API implementation for serving SpatioTemporal Asset Catalog (STAC) data - an enhanced GeoJSON format designed specifically for geospatial assets like satellite imagery, aerial photography, and other Earth observation data. This project enables organizations to:
52
-
53
- - **Efficiently catalog and search geospatial data** such as satellite imagery, aerial photography, DEMs, and other geospatial assets using Elasticsearch or OpenSearch as the database backend
54
- - **Implement standardized STAC APIs** that support complex spatial, temporal, and property-based queries across large collections of geospatial data
55
- - **Scale to millions of geospatial assets** with fast search performance through optimized spatial indexing and query capabilities
56
- - **Support OGC-compliant filtering** including spatial operations (intersects, contains, etc.) and temporal queries
57
- - **Perform geospatial aggregations** to analyze data distribution across space and time
58
- - **Enhanced collection search capabilities** with support for sorting and field selection
59
-
60
- This implementation builds on the STAC-FastAPI framework, providing a production-ready solution specifically optimized for Elasticsearch and OpenSearch databases. It's ideal for organizations managing large geospatial data catalogs who need efficient discovery and access capabilities through standardized APIs.
61
-
62
- ## Common Deployment Patterns
63
-
64
- stac-fastapi-elasticsearch-opensearch can be deployed in several ways depending on your needs:
65
-
66
- - **Containerized Application**: Run as a Docker container with connections to Elasticsearch/OpenSearch databases
67
- - **Serverless Function**: Deploy as AWS Lambda or similar serverless function with API Gateway
68
- - **Traditional Server**: Run on virtual machines or bare metal servers in your infrastructure
69
- - **Kubernetes**: Deploy as part of a larger microservices architecture with container orchestration
70
-
71
- The implementation is flexible and can scale from small local deployments to large production environments serving millions of geospatial assets.
72
-
73
- ## Technologies
74
-
75
- This project is built on the following technologies: STAC, stac-fastapi, FastAPI, Elasticsearch, Python, OpenSearch
76
-
77
- <p align="left">
78
- <a href="https://stacspec.org/"><img src="https://raw.githubusercontent.com/stac-utils/stac-fastapi-elasticsearch-opensearch/refs/heads/main/assets/STAC-01.png" alt="STAC" height="100" hspace="10"></a>
79
- <a href="https://www.python.org/"><img src="https://raw.githubusercontent.com/stac-utils/stac-fastapi-elasticsearch-opensearch/refs/heads/main/assets/python.png" alt="Python" height="80" hspace="10"></a>
80
- <a href="https://fastapi.tiangolo.com/"><img src="https://raw.githubusercontent.com/stac-utils/stac-fastapi-elasticsearch-opensearch/refs/heads/main/assets/fastapi.svg" alt="FastAPI" height="80" hspace="10"></a>
81
- <a href="https://www.elastic.co/"><img src="https://raw.githubusercontent.com/stac-utils/stac-fastapi-elasticsearch-opensearch/refs/heads/main/assets/elasticsearch.png" alt="Elasticsearch" height="80" hspace="10"></a>
82
- <a href="https://opensearch.org/"><img src="https://raw.githubusercontent.com/stac-utils/stac-fastapi-elasticsearch-opensearch/refs/heads/main/assets/opensearch.svg" alt="OpenSearch" height="80" hspace="10"></a>
83
- </p>
84
-
85
- ## Table of Contents
86
-
87
- - [stac-fastapi-elasticsearch-opensearch](#stac-fastapi-elasticsearch-opensearch)
88
- - [Sponsors & Supporters](#sponsors--supporters)
89
- - [Project Introduction - What is SFEOS?](#project-introduction---what-is-sfeos)
90
- - [Common Deployment Patterns](#common-deployment-patterns)
91
- - [Technologies](#technologies)
92
- - [Table of Contents](#table-of-contents)
93
- - [Collection Search Extensions](#collection-search-extensions)
94
- - [Documentation & Resources](#documentation--resources)
95
- - [Package Structure](#package-structure)
96
- - [Examples](#examples)
97
- - [Performance](#performance)
98
- - [Direct Response Mode](#direct-response-mode)
99
- - [Quick Start](#quick-start)
100
- - [Installation](#installation)
101
- - [Running Locally](#running-locally)
102
- - [Using Pre-built Docker Images](#using-pre-built-docker-images)
103
- - [Using Docker Compose](#using-docker-compose)
104
- - [Configuration Reference](#configuration-reference)
105
- - [Datetime-Based Index Management](#datetime-based-index-management)
106
- - [Overview](#overview)
107
- - [When to Use](#when-to-use)
108
- - [Configuration](#configuration)
109
- - [Enabling Datetime-Based Indexing](#enabling-datetime-based-indexing)
110
- - [Related Configuration Variables](#related-configuration-variables)
111
- - [How Datetime-Based Indexing Works](#how-datetime-based-indexing-works)
112
- - [Index and Alias Naming Convention](#index-and-alias-naming-convention)
113
- - [Index Size Management](#index-size-management)
114
- - [Interacting with the API](#interacting-with-the-api)
115
- - [Configure the API](#configure-the-api)
116
- - [Collection Pagination](#collection-pagination)
117
- - [Ingesting Sample Data CLI Tool](#ingesting-sample-data-cli-tool)
118
- - [Elasticsearch Mappings](#elasticsearch-mappings)
119
- - [Managing Elasticsearch Indices](#managing-elasticsearch-indices)
120
- - [Snapshots](#snapshots)
121
- - [Reindexing](#reindexing)
122
- - [Auth](#auth)
123
- - [Aggregation](#aggregation)
124
- - [Rate Limiting](#rate-limiting)
125
-
126
- ## Documentation & Resources
127
-
128
- - **Online Documentation**: [https://stac-utils.github.io/stac-fastapi-elasticsearch-opensearch](https://stac-utils.github.io/stac-fastapi-elasticsearch-opensearch/)
129
- - **Source Code**: [https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch)
130
- - **API Examples**: [Postman Documentation](https://documenter.getpostman.com/view/12888943/2s8ZDSdRHA) - Examples of how to use the API endpoints
131
- - **Community**:
132
- - [Gitter Chat](https://app.gitter.im/#/room/#stac-fastapi-elasticsearch_community:gitter.im) - For real-time discussions
133
- - [GitHub Discussions](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/discussions) - For longer-form questions and answers
134
-
135
- ## Collection Search Extensions
136
-
137
- SFEOS provides enhanced collection search capabilities through two primary routes:
138
- - **GET/POST `/collections`**: The standard STAC endpoint with extended query parameters
139
- - **GET/POST `/collections-search`**: A custom endpoint that supports the same parameters, created to avoid conflicts with the STAC Transactions extension if enabled (which uses POST `/collections` for collection creation)
140
-
141
- These endpoints support advanced collection discovery features including:
142
-
143
- - **Sorting**: Sort collections by sortable fields using the `sortby` parameter
144
- - Example: `/collections?sortby=+id` (ascending sort by ID)
145
- - Example: `/collections?sortby=-id` (descending sort by ID)
146
- - Example: `/collections?sortby=-temporal` (descending sort by temporal extent)
147
-
148
- - **Field Selection**: Request only specific fields to be returned using the `fields` parameter
149
- - Example: `/collections?fields=id,title,description`
150
- - This helps reduce payload size when only certain fields are needed
151
-
152
- - **Free Text Search**: Search across collection text fields using the `q` parameter
153
- - Example: `/collections?q=landsat`
154
- - Searches across multiple text fields including title, description, and keywords
155
- - Supports partial word matching and relevance-based sorting
156
-
157
- - **Structured Filtering**: Filter collections using CQL2 expressions
158
- - JSON format: `/collections?filter={"op":"=","args":[{"property":"id"},"sentinel-2"]}&filter-lang=cql2-json`
159
- - Text format: `/collections?filter=id='sentinel-2'&filter-lang=cql2-text` (note: string values must be quoted)
160
- - Advanced text format: `/collections?filter=id LIKE '%sentinel%'&filter-lang=cql2-text` (supports LIKE, BETWEEN, etc.)
161
- - Supports both CQL2 JSON and CQL2 text formats with various operators
162
- - Enables precise filtering on any collection property
163
-
164
- - **Datetime Filtering**: Filter collections by their temporal extent using the `datetime` parameter
165
- - Example: `/collections?datetime=2020-01-01T00:00:00Z/2020-12-31T23:59:59Z` (finds collections with temporal extents that overlap this range)
166
- - Example: `/collections?datetime=2020-06-15T12:00:00Z` (finds collections whose temporal extent includes this specific time)
167
- - Example: `/collections?datetime=2020-01-01T00:00:00Z/..` (finds collections with temporal extents that extend to or beyond January 1, 2020)
168
- - Example: `/collections?datetime=../2020-12-31T23:59:59Z` (finds collections with temporal extents that begin on or before December 31, 2020)
169
- - Collections are matched if their temporal extent overlaps with the provided datetime parameter
170
- - This allows for efficient discovery of collections based on time periods
171
-
172
- These extensions make it easier to build user interfaces that display and navigate through collections efficiently.
173
-
174
- > **Configuration**: Collection search extensions (sorting, field selection, free text search, structured filtering, and datetime filtering) for the `/collections` endpoint can be disabled by setting the `ENABLE_COLLECTIONS_SEARCH` environment variable to `false`. By default, these extensions are enabled.
175
- >
176
- > **Configuration**: The custom `/collections-search` endpoint can be enabled by setting the `ENABLE_COLLECTIONS_SEARCH_ROUTE` environment variable to `true`. By default, this endpoint is **disabled**.
177
-
178
- > **Note**: Sorting is only available on fields that are indexed for sorting in Elasticsearch/OpenSearch. With the default mappings, you can sort on:
179
- > - `id` (keyword field)
180
- > - `extent.temporal.interval` (date field)
181
- > - `temporal` (alias to extent.temporal.interval)
182
- >
183
- > Text fields like `title` and `description` are not sortable by default as they use text analysis for better search capabilities. Attempting to sort on these fields will result in a user-friendly error message explaining which fields are sortable and how to make additional fields sortable by updating the mappings.
184
- >
185
- > **Important**: Adding keyword fields to make text fields sortable can significantly increase the index size, especially for large text fields. Consider the storage implications when deciding which fields to make sortable.
186
-
187
-
188
- ## Package Structure
189
-
190
- This project is organized into several packages, each with a specific purpose:
191
-
192
- - **stac_fastapi_core**: Core functionality that's database-agnostic, including API models, extensions, and shared utilities. This package provides the foundation for building STAC API implementations with any database backend. See [stac-fastapi-mongo](https://github.com/Healy-Hyperspatial/stac-fastapi-mongo) for a working example.
193
-
194
- - **sfeos_helpers**: Shared helper functions and utilities used by both the Elasticsearch and OpenSearch backends. This package includes:
195
- - `database`: Specialized modules for index, document, and database utility operations
196
- - `aggregation`: Elasticsearch/OpenSearch-specific aggregation functionality
197
- - Shared logic and utilities that improve code reuse between backends
198
-
199
- - **stac_fastapi_elasticsearch**: Complete implementation of the STAC API using Elasticsearch as the backend database. This package depends on both `stac_fastapi_core` and `sfeos_helpers`.
200
-
201
- - **stac_fastapi_opensearch**: Complete implementation of the STAC API using OpenSearch as the backend database. This package depends on both `stac_fastapi_core` and `sfeos_helpers`.
202
-
203
- ## Examples
204
-
205
- The `/examples` directory contains several useful examples and reference implementations:
206
-
207
- - **pip_docker**: Examples of running stac-fastapi-elasticsearch from PyPI in Docker without needing any code from the repository
208
- - **auth**: Authentication examples including:
209
- - Basic authentication
210
- - OAuth2 with Keycloak
211
- - Route dependencies configuration
212
- - **rate_limit**: Example of implementing rate limiting for API requests
213
- - **postman_collections**: Postman collection files you can import for testing API endpoints
214
-
215
- These examples provide practical reference implementations for various deployment scenarios and features.
216
-
217
- ## Performance
218
-
219
- ### Direct Response Mode
220
-
221
- - The `enable_direct_response` option is provided by the stac-fastapi core library (introduced in stac-fastapi 5.2.0) and is available in this project starting from v4.0.0.
222
- - **Control via environment variable**: Set `ENABLE_DIRECT_RESPONSE=true` to enable this feature.
223
- - **How it works**: When enabled, endpoints return Starlette Response objects directly, bypassing FastAPI's default serialization for improved performance.
224
- - **Important limitation**: All FastAPI dependencies (including authentication, custom status codes, and validation) are disabled for all routes when this mode is enabled.
225
- - **Best use case**: This mode is best suited for public or read-only APIs where authentication and custom logic are not required.
226
- - **Default setting**: `false` for safety.
227
- - **More information**: See [issue #347](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/issues/347) for background and implementation details.
228
-
229
- ## Quick Start
230
-
231
- This section helps you get up and running with stac-fastapi-elasticsearch-opensearch quickly.
232
-
233
- ### Installation
234
-
235
- - **For versions 4.0.0a1 and newer** (PEP 625 compliant naming):
236
- ```bash
237
- pip install stac-fastapi-elasticsearch # Elasticsearch backend
238
- pip install stac-fastapi-opensearch # Opensearch backend
239
- pip install stac-fastapi-core # Core library
240
- ```
241
-
242
- - **For versions 4.0.0a0 and older**:
243
- ```bash
244
- pip install stac-fastapi.elasticsearch # Elasticsearch backend
245
- pip install stac-fastapi.opensearch # Opensearch backend
246
- pip install stac-fastapi.core # Core library
247
- ```
248
-
249
- > **Important Note:** Starting with version 4.0.0a1, package names have changed from using periods (e.g., `stac-fastapi.core`) to using hyphens (e.g., `stac-fastapi-core`) to comply with PEP 625. The internal package structure uses underscores, but users should install with hyphens as shown above. Please update your requirements files accordingly.
250
-
251
- ### Running Locally
252
-
253
- There are two main ways to run the API locally:
254
-
255
- #### Using Pre-built Docker Images
256
-
257
- - We provide ready-to-use Docker images through GitHub Container Registry:
258
- - [ElasticSearch backend](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pkgs/container/stac-fastapi-es)
259
- - [OpenSearch backend](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pkgs/container/stac-fastapi-os)
260
-
261
- - **Pull and run the images**:
262
- ```shell
263
- # For Elasticsearch backend
264
- docker pull ghcr.io/stac-utils/stac-fastapi-es:latest
265
-
266
- # For OpenSearch backend
267
- docker pull ghcr.io/stac-utils/stac-fastapi-os:latest
268
- ```
269
-
270
- #### Using Docker Compose
271
-
272
- - **Prerequisites**: Ensure [Docker Compose](https://docs.docker.com/compose/install/) or [Podman Compose](https://podman-desktop.io/docs/compose) is installed on your machine.
273
-
274
- - **Start the API**:
275
- ```shell
276
- docker compose up elasticsearch app-elasticsearch
277
- ```
278
-
279
- - **Configuration**: By default, Docker Compose uses Elasticsearch 8.x and OpenSearch 2.11.1. To use different versions, create a `.env` file:
280
- ```shell
281
- ELASTICSEARCH_VERSION=8.11.0
282
- OPENSEARCH_VERSION=2.11.1
283
- ENABLE_DIRECT_RESPONSE=false
284
- ```
285
-
286
- - **Compatibility**: The most recent Elasticsearch 7.x versions should also work. See the [opensearch-py docs](https://github.com/opensearch-project/opensearch-py/blob/main/COMPATIBILITY.md) for compatibility information.
287
-
288
-
289
-
290
- ## Configuration Reference
291
-
292
- You can customize additional settings in your `.env` file:
293
-
294
- | Variable | Description | Default | Required |
295
- |------------------------------|--------------------------------------------------------------------------------------|--------------------------|---------------------------------------------------------------------------------------------|
296
- | `ES_HOST` | Hostname for external Elasticsearch/OpenSearch. | `localhost` | Optional |
297
- | `ES_PORT` | Port for Elasticsearch/OpenSearch. | `9200` (ES) / `9202` (OS)| Optional |
298
- | `ES_USE_SSL` | Use SSL for connecting to Elasticsearch/OpenSearch. | `true` | Optional |
299
- | `ES_VERIFY_CERTS` | Verify SSL certificates when connecting. | `true` | Optional |
300
- | `ES_API_KEY` | API Key for external Elasticsearch/OpenSearch. | N/A | Optional |
301
- | `ES_TIMEOUT` | Client timeout for Elasticsearch/OpenSearch. | DB client default | Optional |
302
- | `STAC_FASTAPI_TITLE` | Title of the API in the documentation. | `stac-fastapi-<backend>` | Optional |
303
- | `STAC_FASTAPI_DESCRIPTION` | Description of the API in the documentation. | N/A | Optional |
304
- | `STAC_FASTAPI_VERSION` | API version. | `2.1` | Optional |
305
- | `STAC_FASTAPI_LANDING_PAGE_ID` | Landing page ID | `stac-fastapi` | Optional |
306
- | `APP_HOST` | Server bind address. | `0.0.0.0` | Optional |
307
- | `APP_PORT` | Server port. | `8000` | Optional |
308
- | `ENVIRONMENT` | Runtime environment. | `local` | Optional |
309
- | `WEB_CONCURRENCY` | Number of worker processes. | `10` | Optional |
310
- | `RELOAD` | Enable auto-reload for development. | `true` | Optional |
311
- | `STAC_FASTAPI_RATE_LIMIT` | API rate limit per client. | `200/minute` | Optional |
312
- | `BACKEND` | Tests-related variable | `elasticsearch` or `opensearch` based on the backend | Optional |
313
- | `ELASTICSEARCH_VERSION` | Version of Elasticsearch to use. | `8.11.0` | Optional |
314
- | `OPENSEARCH_VERSION` | OpenSearch version | `2.11.1` | Optional |
315
- | `ENABLE_DIRECT_RESPONSE` | Enable direct response for maximum performance (disables all FastAPI dependencies, including authentication, custom status codes, and validation) | `false` | Optional |
316
- | `RAISE_ON_BULK_ERROR` | Controls whether bulk insert operations raise exceptions on errors. If set to `true`, the operation will stop and raise an exception when an error occurs. If set to `false`, errors will be logged, and the operation will continue. **Note:** STAC Item and ItemCollection validation errors will always raise, regardless of this flag. | `false` | Optional |
317
- | `DATABASE_REFRESH` | Controls whether database operations refresh the index immediately after changes. If set to `true`, changes will be immediately searchable. If set to `false`, changes may not be immediately visible but can improve performance for bulk operations. If set to `wait_for`, changes will wait for the next refresh cycle to become visible. | `false` | Optional |
318
- | `ENABLE_COLLECTIONS_SEARCH` | Enable collection search extensions (sort, fields, free text search, structured filtering, and datetime filtering) on the core `/collections` endpoint. | `true` | Optional |
319
- | `ENABLE_COLLECTIONS_SEARCH_ROUTE` | Enable the custom `/collections-search` endpoint (both GET and POST methods). When disabled, the custom endpoint will not be available, but collection search extensions will still be available on the core `/collections` endpoint if `ENABLE_COLLECTIONS_SEARCH` is true. | `false` | Optional |
320
- | `ENABLE_TRANSACTIONS_EXTENSIONS` | Enables or disables the Transactions and Bulk Transactions API extensions. This is useful for deployments where mutating the catalog via the API should be prevented. If set to `true`, the POST `/collections` route for search will be unavailable in the API. | `true` | Optional |
321
- | `STAC_ITEM_LIMIT` | Sets the environment variable for result limiting to SFEOS for the number of returned items and STAC collections. | `10` | Optional |
322
- | `STAC_INDEX_ASSETS` | Controls if Assets are indexed when added to Elasticsearch/Opensearch. This allows asset fields to be included in search queries. | `false` | Optional |
323
- | `ENV_MAX_LIMIT` | Configures the environment variable in SFEOS to override the default `MAX_LIMIT`, which controls the limit parameter for returned items and STAC collections. | `10,000` | Optional |
324
- | `USE_DATETIME` | Configures the datetime search behavior in SFEOS. When enabled, searches both datetime field and falls back to start_datetime/end_datetime range for items with null datetime. When disabled, searches only by start_datetime/end_datetime range. | `true` | Optional |
325
-
326
- > [!NOTE]
327
- > The variables `ES_HOST`, `ES_PORT`, `ES_USE_SSL`, `ES_VERIFY_CERTS` and `ES_TIMEOUT` apply to both Elasticsearch and OpenSearch backends, so there is no need to rename the key names to `OS_` even if you're using OpenSearch.
328
-
329
- ## Datetime-Based Index Management
330
-
331
- ### Overview
332
-
333
- SFEOS supports two indexing strategies for managing STAC items:
334
-
335
- 1. **Simple Indexing** (default) - One index per collection
336
- 2. **Datetime-Based Indexing** - Time-partitioned indexes with automatic management
337
-
338
- The datetime-based indexing strategy is particularly useful for large temporal datasets. When a user provides a datetime parameter in a query, the system knows exactly which index to search, providing **multiple times faster searches** and significantly **reducing database load**.
339
-
340
- ### When to Use
341
-
342
- **Recommended for:**
343
- - Systems with large collections containing millions of items
344
- - Systems requiring high-performance temporal searching
345
-
346
- **Pros:**
347
- - Multiple times faster queries with datetime filter
348
- - Reduced database load - only relevant indexes are searched
349
-
350
- **Cons:**
351
- - Slightly longer item indexing time (automatic index management)
352
- - Greater management complexity
353
-
354
- ### Configuration
355
-
356
- #### Enabling Datetime-Based Indexing
357
-
358
- Enable datetime-based indexing by setting the following environment variable:
359
-
360
- ```bash
361
- ENABLE_DATETIME_INDEX_FILTERING=true
362
- ```
363
-
364
- ### Related Configuration Variables
365
-
366
- | Variable | Description | Default | Example |
367
- |----------|-------------|---------|---------|
368
- | `ENABLE_DATETIME_INDEX_FILTERING` | Enables time-based index partitioning | `false` | `true` |
369
- | `DATETIME_INDEX_MAX_SIZE_GB` | Maximum size limit for datetime indexes (GB) - note: add +20% to target size due to ES/OS compression | `25` | `50` |
370
- | `STAC_ITEMS_INDEX_PREFIX` | Prefix for item indexes | `items_` | `stac_items_` |
371
-
372
- ## How Datetime-Based Indexing Works
373
-
374
- ### Index and Alias Naming Convention
375
-
376
- The system uses a precise naming convention:
377
-
378
- **Physical indexes:**
379
- ```
380
- {ITEMS_INDEX_PREFIX}{collection-id}_{uuid4}
381
- ```
382
-
383
- **Aliases:**
384
- ```
385
- {ITEMS_INDEX_PREFIX}{collection-id} # Main collection alias
386
- {ITEMS_INDEX_PREFIX}{collection-id}_{start-datetime} # Temporal alias
387
- {ITEMS_INDEX_PREFIX}{collection-id}_{start-datetime}_{end-datetime} # Closed index alias
388
- ```
389
-
390
- **Example:**
391
-
392
- *Physical indexes:*
393
- - `items_sentinel-2-l2a_a1b2c3d4-e5f6-7890-abcd-ef1234567890`
394
-
395
- *Aliases:*
396
- - `items_sentinel-2-l2a` - main collection alias
397
- - `items_sentinel-2-l2a_2024-01-01` - active alias from January 1, 2024
398
- - `items_sentinel-2-l2a_2024-01-01_2024-03-15` - closed index alias (reached size limit)
399
-
400
- ### Index Size Management
401
-
402
- **Important - Data Compression:** Elasticsearch and OpenSearch automatically compress data. The configured `DATETIME_INDEX_MAX_SIZE_GB` limit refers to the compressed size on disk. It is recommended to add +20% to the target size to account for compression overhead and metadata.
403
-
404
- ## Interacting with the API
405
-
406
- - **Creating a Collection**:
407
- ```shell
408
- curl -X "POST" "http://localhost:8080/collections" \
409
- -H 'Content-Type: application/json; charset=utf-8' \
410
- -d $'{
411
- "id": "my_collection"
412
- }'
413
- ```
414
-
415
- - **Adding an Item to a Collection**:
416
- ```shell
417
- curl -X "POST" "http://localhost:8080/collections/my_collection/items" \
418
- -H 'Content-Type: application/json; charset=utf-8' \
419
- -d @item.json
420
- ```
421
-
422
- - **Searching for Items**:
423
- ```shell
424
- curl -X "GET" "http://localhost:8080/search" \
425
- -H 'Content-Type: application/json; charset=utf-8' \
426
- -d $'{
427
- "collections": ["my_collection"],
428
- "limit": 10
429
- }'
430
- ```
431
-
432
- - **Filtering by Bbox**:
433
- ```shell
434
- curl -X "GET" "http://localhost:8080/search" \
435
- -H 'Content-Type: application/json; charset=utf-8' \
436
- -d $'{
437
- "collections": ["my_collection"],
438
- "bbox": [-180, -90, 180, 90]
439
- }'
440
- ```
441
-
442
- - **Filtering by Datetime**:
443
- ```shell
444
- curl -X "GET" "http://localhost:8080/search" \
445
- -H 'Content-Type: application/json; charset=utf-8' \
446
- -d $'{
447
- "collections": ["my_collection"],
448
- "datetime": "2020-01-01T00:00:00Z/2020-12-31T23:59:59Z"
449
- }'
450
- ```
451
-
452
- ## Configure the API
453
-
454
- - **API Title and Description**: By default set to `stac-fastapi-<backend>`. Customize these by setting:
455
- - `STAC_FASTAPI_TITLE`: Changes the API title in the documentation
456
- - `STAC_FASTAPI_DESCRIPTION`: Changes the API description in the documentation
457
-
458
- - **Database Indices**: By default, the API reads from and writes to:
459
- - `collections` index for collections
460
- - `items_<collection name>` indices for items
461
- - Customize with `STAC_COLLECTIONS_INDEX` and `STAC_ITEMS_INDEX_PREFIX` environment variables
462
-
463
- - **Root Path Configuration**: The application root path is the base URL by default.
464
- - For AWS Lambda with Gateway API: Set `STAC_FASTAPI_ROOT_PATH` to match the Gateway API stage name (e.g., `/v1`)
465
-
466
- - **Feature Configuration**: Control which features are enabled:
467
- - `ENABLE_COLLECTIONS_SEARCH`: Set to `true` (default) to enable collection search extensions (sort, fields). Set to `false` to disable.
468
- - `ENABLE_TRANSACTIONS_EXTENSIONS`: Set to `true` (default) to enable transaction extensions. Set to `false` to disable.
469
-
470
- ## Collection Pagination
471
-
472
- - **Overview**: The collections route supports pagination through optional query parameters.
473
- - **Parameters**:
474
- - `limit`: Controls the number of collections returned per page
475
- - `token`: Used to retrieve subsequent pages of results
476
- - **Response Structure**: The `links` field in the response contains a `next` link with the token for the next page of results.
477
- - **Example Usage**:
478
- ```shell
479
- curl -X "GET" "http://localhost:8080/collections?limit=1&token=example_token"
480
- ```
481
-
482
- ## Ingesting Sample Data CLI Tool
483
-
484
- - **Overview**: The `data_loader.py` script provides a convenient way to load STAC items into the database.
485
-
486
- - **Usage**:
487
- ```shell
488
- python3 data_loader.py --base-url http://localhost:8080
489
- ```
490
-
491
- - **Options**:
492
- ```
493
- --base-url TEXT Base URL of the STAC API [required]
494
- --collection-id TEXT ID of the collection to which items are added
495
- --use-bulk Use bulk insert method for items
496
- --data-dir PATH Directory containing collection.json and feature
497
- collection file
498
- --help Show this message and exit.
499
- ```
500
-
501
- - **Example Workflows**:
502
- - **Loading Sample Data**:
503
- ```shell
504
- python3 data_loader.py --base-url http://localhost:8080
505
- ```
506
- - **Loading Data to a Specific Collection**:
507
- ```shell
508
- python3 data_loader.py --base-url http://localhost:8080 --collection-id my-collection
509
- ```
510
- - **Using Bulk Insert for Performance**:
511
- ```shell
512
- python3 data_loader.py --base-url http://localhost:8080 --use-bulk
513
- ```
514
-
515
- ## Elasticsearch Mappings
516
-
517
- - **Overview**: Mappings apply to search index, not source data. They define how documents and their fields are stored and indexed.
518
- - **Implementation**:
519
- - Mappings are stored in index templates that are created on application startup
520
- - These templates are automatically applied when creating new Collection and Item indices
521
- - The `sfeos_helpers` package contains shared mapping definitions used by both Elasticsearch and OpenSearch backends
522
- - **Customization**: Custom mappings can be defined by extending the base mapping templates.
523
-
524
- ## Managing Elasticsearch Indices
525
-
526
- ### Snapshots
527
-
528
- - **Overview**: Snapshots provide a way to backup and restore your indices.
529
-
530
- - **Creating a Snapshot Repository**:
531
- ```shell
532
- curl -X "PUT" "http://localhost:9200/_snapshot/my_fs_backup" \
533
- -H 'Content-Type: application/json; charset=utf-8' \
534
- -d $'{
535
- "type": "fs",
536
- "settings": {
537
- "location": "/usr/share/elasticsearch/snapshots/my_fs_backup"
538
- }
539
- }'
540
- ```
541
- - This creates a snapshot repository that stores files in the elasticsearch/snapshots directory in this git repo clone
542
- - The elasticsearch.yml and compose files create a mapping from that directory to /usr/share/elasticsearch/snapshots within the Elasticsearch container and grant permissions for using it
543
-
544
- - **Creating a Snapshot**:
545
- ```shell
546
- curl -X "PUT" "http://localhost:9200/_snapshot/my_fs_backup/my_snapshot_2?wait_for_completion=true" \
547
- -H 'Content-Type: application/json; charset=utf-8' \
548
- -d $'{
549
- "metadata": {
550
- "taken_because": "dump of all items",
551
- "taken_by": "pvarner"
552
- },
553
- "include_global_state": false,
554
- "ignore_unavailable": false,
555
- "indices": "items_my-collection"
556
- }'
557
- ```
558
- - This creates a snapshot named my_snapshot_2 and waits for the action to be completed before returning
559
- - This can also be done asynchronously by omitting the wait_for_completion parameter, and queried for status later
560
- - The indices parameter determines which indices are snapshotted, and can include wildcards
561
-
562
- - **Viewing Snapshots**:
563
- ```shell
564
- # View a specific snapshot
565
- curl http://localhost:9200/_snapshot/my_fs_backup/my_snapshot_2
566
-
567
- # View all snapshots
568
- curl http://localhost:9200/_snapshot/my_fs_backup/_all
569
- ```
570
- - These commands allow you to check the status and details of your snapshots
571
-
572
- - **Restoring a Snapshot**:
573
- ```shell
574
- curl -X "POST" "http://localhost:9200/_snapshot/my_fs_backup/my_snapshot_2/_restore?wait_for_completion=true" \
575
- -H 'Content-Type: application/json; charset=utf-8' \
576
- -d $'{
577
- "include_aliases": false,
578
- "include_global_state": false,
579
- "ignore_unavailable": true,
580
- "rename_replacement": "items_$1-copy",
581
- "indices": "items_*",
582
- "rename_pattern": "items_(.+)"
583
- }'
584
- ```
585
- - This specific command will restore any indices that match items_* and rename them so that the new index name will be suffixed with -copy
586
- - The rename_pattern and rename_replacement parameters allow you to restore indices under new names
587
-
588
- - **Updating Collection References**:
589
- ```shell
590
- curl -X "POST" "http://localhost:9200/items_my-collection-copy/_update_by_query" \
591
- -H 'Content-Type: application/json; charset=utf-8' \
592
- -d $'{
593
- "query": {
594
- "match_all": {}
595
- },
596
- "script": {
597
- "lang": "painless",
598
- "params": {
599
- "collection": "my-collection-copy"
600
- },
601
- "source": "ctx._source.collection = params.collection"
602
- }
603
- }'
604
- ```
605
- - After restoring, the item documents have been restored in the new index (e.g., my-collection-copy), but the value of the collection field in those documents is still the original value of my-collection
606
- - This command updates these values to match the new collection name using Elasticsearch's Update By Query feature
607
-
608
- - **Creating a New Collection**:
609
- ```shell
610
- curl -X "POST" "http://localhost:8080/collections" \
611
- -H 'Content-Type: application/json' \
612
- -d $'{
613
- "id": "my-collection-copy"
614
- }'
615
- ```
616
- - The final step is to create a new collection through the API with the new name for each of the restored indices
617
- - This gives you a copy of the collection that has a resource URI (/collections/my-collection-copy) and can be correctly queried by collection name
618
-
619
- ### Reindexing
620
-
621
- - **Overview**: Reindexing allows you to copy documents from one index to another, optionally transforming them in the process.
622
-
623
- - **Use Cases**:
624
- - Apply changes to documents
625
- - Correct dynamically generated mappings
626
- - Transform data (e.g., lowercase identifiers)
627
- - The index templates will make sure that manually created indices will also have the correct mappings and settings
628
-
629
- - **Example: Reindexing with Transformation**:
630
- ```shell
631
- curl -X "POST" "http://localhost:9200/_reindex" \
632
- -H 'Content-Type: application/json' \
633
- -d $'{
634
- "source": {
635
- "index": "items_my-collection-lower_my-collection-hex-000001"
636
- },
637
- "dest": {
638
- "index": "items_my-collection-lower_my-collection-hex-000002"
639
- },
640
- "script": {
641
- "source": "ctx._source.id = ctx._source.id.toLowerCase()",
642
- "lang": "painless"
643
- }
644
- }'
645
- ```
646
- - In this example, we make a copy of an existing Item index but change the Item identifier to be lowercase
647
- - The script parameter allows you to transform documents during the reindexing process
648
-
649
- - **Updating Aliases**:
650
- ```shell
651
- curl -X "POST" "http://localhost:9200/_aliases" \
652
- -H 'Content-Type: application/json' \
653
- -d $'{
654
- "actions": [
655
- {
656
- "remove": {
657
- "index": "*",
658
- "alias": "items_my-collection"
659
- }
660
- },
661
- {
662
- "add": {
663
- "index": "items_my-collection-lower_my-collection-hex-000002",
664
- "alias": "items_my-collection"
665
- }
666
- }
667
- ]
668
- }'
669
- ```
670
- - If you are happy with the data in the newly created index, you can move the alias items_my-collection to the new index
671
- - This makes the modified Items with lowercase identifiers visible to users accessing my-collection in the STAC API
672
- - Using aliases allows you to switch between different index versions without changing the API endpoint
673
-
674
- ## Auth
675
-
676
- - **Overview**: Authentication is an optional feature that can be enabled through Route Dependencies.
677
- - **Implementation Options**:
678
- - Basic authentication
679
- - OAuth2 with Keycloak
680
- - Custom route dependencies
681
- - **Configuration**: Authentication can be configured using the `STAC_FASTAPI_ROUTE_DEPENDENCIES` environment variable.
682
- - **Examples and Documentation**: Detailed examples and implementation guides can be found in the [examples/auth](examples/auth) directory.
683
-
684
- ## Aggregation
685
-
686
- - **Supported Aggregations**:
687
- - Spatial aggregations of points and geometries
688
- - Frequency distribution aggregation of any property including dates
689
- - Temporal distribution of datetime values
690
-
691
- - **Endpoint Locations**:
692
- - Root Catalog level: `/aggregations`
693
- - Collection level: `/<collection_id>/aggregations`
694
-
695
- - **Implementation Details**: The `sfeos_helpers.aggregation` package provides specialized functionality for both Elasticsearch and OpenSearch backends.
696
-
697
- - **Documentation**: Detailed information about supported aggregations can be found in [the aggregation docs](./docs/src/aggregation.md).
698
-
699
-
700
- ## Rate Limiting
701
-
702
- - **Overview**: Rate limiting is an optional security feature that controls API request frequency on a remote address basis.
703
-
704
- - **Configuration**: Enabled by setting the `STAC_FASTAPI_RATE_LIMIT` environment variable:
705
- ```
706
- STAC_FASTAPI_RATE_LIMIT=500/minute
707
- ```
708
-
709
- - **Functionality**:
710
- - Limits each client to a specified number of requests per time period (e.g., 500 requests per minute)
711
- - Helps prevent API abuse and maintains system stability
712
- - Ensures fair resource allocation among all clients
713
-
714
- - **Examples**: Implementation examples are available in the [examples/rate_limit](examples/rate_limit) directory.
@@ -1 +0,0 @@
1
- stac_fastapi