stac-fastapi-opensearch 6.2.1__tar.gz → 6.4.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {stac_fastapi_opensearch-6.2.1 → stac_fastapi_opensearch-6.4.0}/PKG-INFO +80 -49
- {stac_fastapi_opensearch-6.2.1 → stac_fastapi_opensearch-6.4.0}/README.md +78 -22
- {stac_fastapi_opensearch-6.2.1 → stac_fastapi_opensearch-6.4.0}/setup.py +2 -2
- {stac_fastapi_opensearch-6.2.1 → stac_fastapi_opensearch-6.4.0}/stac_fastapi/opensearch/app.py +60 -13
- {stac_fastapi_opensearch-6.2.1 → stac_fastapi_opensearch-6.4.0}/stac_fastapi/opensearch/database_logic.py +186 -66
- {stac_fastapi_opensearch-6.2.1 → stac_fastapi_opensearch-6.4.0}/stac_fastapi/opensearch/version.py +1 -1
- {stac_fastapi_opensearch-6.2.1 → stac_fastapi_opensearch-6.4.0}/stac_fastapi_opensearch.egg-info/PKG-INFO +81 -50
- {stac_fastapi_opensearch-6.2.1 → stac_fastapi_opensearch-6.4.0}/stac_fastapi_opensearch.egg-info/requires.txt +2 -2
- {stac_fastapi_opensearch-6.2.1 → stac_fastapi_opensearch-6.4.0}/stac_fastapi_opensearch.egg-info/top_level.txt +0 -1
- {stac_fastapi_opensearch-6.2.1 → stac_fastapi_opensearch-6.4.0}/setup.cfg +0 -0
- {stac_fastapi_opensearch-6.2.1 → stac_fastapi_opensearch-6.4.0}/stac_fastapi/opensearch/__init__.py +0 -0
- {stac_fastapi_opensearch-6.2.1 → stac_fastapi_opensearch-6.4.0}/stac_fastapi/opensearch/config.py +0 -0
- {stac_fastapi_opensearch-6.2.1 → stac_fastapi_opensearch-6.4.0}/stac_fastapi_opensearch.egg-info/SOURCES.txt +0 -0
- {stac_fastapi_opensearch-6.2.1 → stac_fastapi_opensearch-6.4.0}/stac_fastapi_opensearch.egg-info/dependency_links.txt +0 -0
- {stac_fastapi_opensearch-6.2.1 → stac_fastapi_opensearch-6.4.0}/stac_fastapi_opensearch.egg-info/entry_points.txt +0 -0
- {stac_fastapi_opensearch-6.2.1 → stac_fastapi_opensearch-6.4.0}/stac_fastapi_opensearch.egg-info/not-zip-safe +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
|
-
Metadata-Version: 2.
|
|
1
|
+
Metadata-Version: 2.1
|
|
2
2
|
Name: stac_fastapi_opensearch
|
|
3
|
-
Version: 6.
|
|
3
|
+
Version: 6.4.0
|
|
4
4
|
Summary: Opensearch stac-fastapi backend.
|
|
5
5
|
Home-page: https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch
|
|
6
6
|
License: MIT
|
|
@@ -15,34 +15,9 @@ Classifier: Programming Language :: Python :: 3.13
|
|
|
15
15
|
Classifier: License :: OSI Approved :: MIT License
|
|
16
16
|
Requires-Python: >=3.9
|
|
17
17
|
Description-Content-Type: text/markdown
|
|
18
|
-
Requires-Dist: stac-fastapi-core==6.2.1
|
|
19
|
-
Requires-Dist: sfeos-helpers==6.2.1
|
|
20
|
-
Requires-Dist: opensearch-py~=2.8.0
|
|
21
|
-
Requires-Dist: opensearch-py[async]~=2.8.0
|
|
22
|
-
Requires-Dist: uvicorn~=0.23.0
|
|
23
|
-
Requires-Dist: starlette<0.36.0,>=0.35.0
|
|
24
18
|
Provides-Extra: dev
|
|
25
|
-
Requires-Dist: pytest~=7.0.0; extra == "dev"
|
|
26
|
-
Requires-Dist: pytest-cov~=4.0.0; extra == "dev"
|
|
27
|
-
Requires-Dist: pytest-asyncio~=0.21.0; extra == "dev"
|
|
28
|
-
Requires-Dist: pre-commit~=3.0.0; extra == "dev"
|
|
29
|
-
Requires-Dist: ciso8601~=2.3.0; extra == "dev"
|
|
30
|
-
Requires-Dist: httpx<0.28.0,>=0.24.0; extra == "dev"
|
|
31
19
|
Provides-Extra: docs
|
|
32
|
-
Requires-Dist: mkdocs~=1.4.0; extra == "docs"
|
|
33
|
-
Requires-Dist: mkdocs-material~=9.0.0; extra == "docs"
|
|
34
|
-
Requires-Dist: pdocs~=1.2.0; extra == "docs"
|
|
35
20
|
Provides-Extra: server
|
|
36
|
-
Requires-Dist: uvicorn[standard]~=0.23.0; extra == "server"
|
|
37
|
-
Dynamic: classifier
|
|
38
|
-
Dynamic: description
|
|
39
|
-
Dynamic: description-content-type
|
|
40
|
-
Dynamic: home-page
|
|
41
|
-
Dynamic: license
|
|
42
|
-
Dynamic: provides-extra
|
|
43
|
-
Dynamic: requires-dist
|
|
44
|
-
Dynamic: requires-python
|
|
45
|
-
Dynamic: summary
|
|
46
21
|
|
|
47
22
|
# stac-fastapi-elasticsearch-opensearch
|
|
48
23
|
|
|
@@ -82,11 +57,10 @@ SFEOS (stac-fastapi-elasticsearch-opensearch) is a high-performance, scalable AP
|
|
|
82
57
|
- **Scale to millions of geospatial assets** with fast search performance through optimized spatial indexing and query capabilities
|
|
83
58
|
- **Support OGC-compliant filtering** including spatial operations (intersects, contains, etc.) and temporal queries
|
|
84
59
|
- **Perform geospatial aggregations** to analyze data distribution across space and time
|
|
60
|
+
- **Enhanced collection search capabilities** with support for sorting and field selection
|
|
85
61
|
|
|
86
62
|
This implementation builds on the STAC-FastAPI framework, providing a production-ready solution specifically optimized for Elasticsearch and OpenSearch databases. It's ideal for organizations managing large geospatial data catalogs who need efficient discovery and access capabilities through standardized APIs.
|
|
87
63
|
|
|
88
|
-
|
|
89
|
-
|
|
90
64
|
## Common Deployment Patterns
|
|
91
65
|
|
|
92
66
|
stac-fastapi-elasticsearch-opensearch can be deployed in several ways depending on your needs:
|
|
@@ -112,26 +86,44 @@ This project is built on the following technologies: STAC, stac-fastapi, FastAPI
|
|
|
112
86
|
|
|
113
87
|
## Table of Contents
|
|
114
88
|
|
|
115
|
-
- [
|
|
116
|
-
- [
|
|
117
|
-
- [
|
|
118
|
-
- [
|
|
119
|
-
- [
|
|
120
|
-
- [
|
|
121
|
-
- [
|
|
122
|
-
- [
|
|
123
|
-
- [
|
|
124
|
-
- [
|
|
125
|
-
- [
|
|
126
|
-
- [
|
|
127
|
-
- [
|
|
128
|
-
- [
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
- [
|
|
132
|
-
- [
|
|
133
|
-
- [
|
|
134
|
-
- [
|
|
89
|
+
- [stac-fastapi-elasticsearch-opensearch](#stac-fastapi-elasticsearch-opensearch)
|
|
90
|
+
- [Sponsors \& Supporters](#sponsors--supporters)
|
|
91
|
+
- [Project Introduction - What is SFEOS?](#project-introduction---what-is-sfeos)
|
|
92
|
+
- [Common Deployment Patterns](#common-deployment-patterns)
|
|
93
|
+
- [Technologies](#technologies)
|
|
94
|
+
- [Table of Contents](#table-of-contents)
|
|
95
|
+
- [Collection Search Extensions](#collection-search-extensions)
|
|
96
|
+
- [Documentation \& Resources](#documentation--resources)
|
|
97
|
+
- [Package Structure](#package-structure)
|
|
98
|
+
- [Examples](#examples)
|
|
99
|
+
- [Performance](#performance)
|
|
100
|
+
- [Direct Response Mode](#direct-response-mode)
|
|
101
|
+
- [Quick Start](#quick-start)
|
|
102
|
+
- [Installation](#installation)
|
|
103
|
+
- [Running Locally](#running-locally)
|
|
104
|
+
- [Using Pre-built Docker Images](#using-pre-built-docker-images)
|
|
105
|
+
- [Using Docker Compose](#using-docker-compose)
|
|
106
|
+
- [Configuration Reference](#configuration-reference)
|
|
107
|
+
- [Datetime-Based Index Management](#datetime-based-index-management)
|
|
108
|
+
- [Overview](#overview)
|
|
109
|
+
- [When to Use](#when-to-use)
|
|
110
|
+
- [Configuration](#configuration)
|
|
111
|
+
- [Enabling Datetime-Based Indexing](#enabling-datetime-based-indexing)
|
|
112
|
+
- [Related Configuration Variables](#related-configuration-variables)
|
|
113
|
+
- [How Datetime-Based Indexing Works](#how-datetime-based-indexing-works)
|
|
114
|
+
- [Index and Alias Naming Convention](#index-and-alias-naming-convention)
|
|
115
|
+
- [Index Size Management](#index-size-management)
|
|
116
|
+
- [Interacting with the API](#interacting-with-the-api)
|
|
117
|
+
- [Configure the API](#configure-the-api)
|
|
118
|
+
- [Collection Pagination](#collection-pagination)
|
|
119
|
+
- [Ingesting Sample Data CLI Tool](#ingesting-sample-data-cli-tool)
|
|
120
|
+
- [Elasticsearch Mappings](#elasticsearch-mappings)
|
|
121
|
+
- [Managing Elasticsearch Indices](#managing-elasticsearch-indices)
|
|
122
|
+
- [Snapshots](#snapshots)
|
|
123
|
+
- [Reindexing](#reindexing)
|
|
124
|
+
- [Auth](#auth)
|
|
125
|
+
- [Aggregation](#aggregation)
|
|
126
|
+
- [Rate Limiting](#rate-limiting)
|
|
135
127
|
|
|
136
128
|
## Documentation & Resources
|
|
137
129
|
|
|
@@ -142,6 +134,37 @@ This project is built on the following technologies: STAC, stac-fastapi, FastAPI
|
|
|
142
134
|
- [Gitter Chat](https://app.gitter.im/#/room/#stac-fastapi-elasticsearch_community:gitter.im) - For real-time discussions
|
|
143
135
|
- [GitHub Discussions](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/discussions) - For longer-form questions and answers
|
|
144
136
|
|
|
137
|
+
## Collection Search Extensions
|
|
138
|
+
|
|
139
|
+
SFEOS implements extended capabilities for the `/collections` endpoint, allowing for more powerful collection discovery:
|
|
140
|
+
|
|
141
|
+
- **Sorting**: Sort collections by sortable fields using the `sortby` parameter
|
|
142
|
+
- Example: `/collections?sortby=+id` (ascending sort by ID)
|
|
143
|
+
- Example: `/collections?sortby=-id` (descending sort by ID)
|
|
144
|
+
- Example: `/collections?sortby=-temporal` (descending sort by temporal extent)
|
|
145
|
+
|
|
146
|
+
- **Field Selection**: Request only specific fields to be returned using the `fields` parameter
|
|
147
|
+
- Example: `/collections?fields=id,title,description`
|
|
148
|
+
- This helps reduce payload size when only certain fields are needed
|
|
149
|
+
|
|
150
|
+
- **Free Text Search**: Search across collection text fields using the `q` parameter
|
|
151
|
+
- Example: `/collections?q=landsat`
|
|
152
|
+
- Searches across multiple text fields including title, description, and keywords
|
|
153
|
+
- Supports partial word matching and relevance-based sorting
|
|
154
|
+
|
|
155
|
+
These extensions make it easier to build user interfaces that display and navigate through collections efficiently.
|
|
156
|
+
|
|
157
|
+
> **Configuration**: Collection search extensions can be disabled by setting the `ENABLE_COLLECTIONS_SEARCH` environment variable to `false`. By default, these extensions are enabled.
|
|
158
|
+
|
|
159
|
+
> **Note**: Sorting is only available on fields that are indexed for sorting in Elasticsearch/OpenSearch. With the default mappings, you can sort on:
|
|
160
|
+
> - `id` (keyword field)
|
|
161
|
+
> - `extent.temporal.interval` (date field)
|
|
162
|
+
> - `temporal` (alias to extent.temporal.interval)
|
|
163
|
+
>
|
|
164
|
+
> Text fields like `title` and `description` are not sortable by default as they use text analysis for better search capabilities. Attempting to sort on these fields will result in a user-friendly error message explaining which fields are sortable and how to make additional fields sortable by updating the mappings.
|
|
165
|
+
>
|
|
166
|
+
> **Important**: Adding keyword fields to make text fields sortable can significantly increase the index size, especially for large text fields. Consider the storage implications when deciding which fields to make sortable.
|
|
167
|
+
|
|
145
168
|
## Package Structure
|
|
146
169
|
|
|
147
170
|
This project is organized into several packages, each with a specific purpose:
|
|
@@ -272,8 +295,12 @@ You can customize additional settings in your `.env` file:
|
|
|
272
295
|
| `ENABLE_DIRECT_RESPONSE` | Enable direct response for maximum performance (disables all FastAPI dependencies, including authentication, custom status codes, and validation) | `false` | Optional |
|
|
273
296
|
| `RAISE_ON_BULK_ERROR` | Controls whether bulk insert operations raise exceptions on errors. If set to `true`, the operation will stop and raise an exception when an error occurs. If set to `false`, errors will be logged, and the operation will continue. **Note:** STAC Item and ItemCollection validation errors will always raise, regardless of this flag. | `false` | Optional |
|
|
274
297
|
| `DATABASE_REFRESH` | Controls whether database operations refresh the index immediately after changes. If set to `true`, changes will be immediately searchable. If set to `false`, changes may not be immediately visible but can improve performance for bulk operations. If set to `wait_for`, changes will wait for the next refresh cycle to become visible. | `false` | Optional |
|
|
298
|
+
| `ENABLE_COLLECTIONS_SEARCH` | Enable collection search extensions (sort, fields). | `true` | Optional |
|
|
275
299
|
| `ENABLE_TRANSACTIONS_EXTENSIONS` | Enables or disables the Transactions and Bulk Transactions API extensions. If set to `false`, the POST `/collections` route and related transaction endpoints (including bulk transaction operations) will be unavailable in the API. This is useful for deployments where mutating the catalog via the API should be prevented. | `true` | Optional |
|
|
276
300
|
| `STAC_ITEM_LIMIT` | Sets the environment variable for result limiting to SFEOS for the number of returned items and STAC collections. | `10` | Optional |
|
|
301
|
+
| `STAC_INDEX_ASSETS` | Controls if Assets are indexed when added to Elasticsearch/Opensearch. This allows asset fields to be included in search queries. | `false` | Optional |
|
|
302
|
+
| `ENV_MAX_LIMIT` | Configures the environment variable in SFEOS to override the default `MAX_LIMIT`, which controls the limit parameter for returned items and STAC collections. | `10,000` | Optional |
|
|
303
|
+
| `USE_DATETIME` | Configures the datetime search behavior in SFEOS. When enabled, searches both datetime field and falls back to start_datetime/end_datetime range for items with null datetime. When disabled, searches only by start_datetime/end_datetime range. | True | Optional |
|
|
277
304
|
|
|
278
305
|
> [!NOTE]
|
|
279
306
|
> The variables `ES_HOST`, `ES_PORT`, `ES_USE_SSL`, `ES_VERIFY_CERTS` and `ES_TIMEOUT` apply to both Elasticsearch and OpenSearch backends, so there is no need to rename the key names to `OS_` even if you're using OpenSearch.
|
|
@@ -415,6 +442,10 @@ The system uses a precise naming convention:
|
|
|
415
442
|
- **Root Path Configuration**: The application root path is the base URL by default.
|
|
416
443
|
- For AWS Lambda with Gateway API: Set `STAC_FASTAPI_ROOT_PATH` to match the Gateway API stage name (e.g., `/v1`)
|
|
417
444
|
|
|
445
|
+
- **Feature Configuration**: Control which features are enabled:
|
|
446
|
+
- `ENABLE_COLLECTIONS_SEARCH`: Set to `true` (default) to enable collection search extensions (sort, fields). Set to `false` to disable.
|
|
447
|
+
- `ENABLE_TRANSACTIONS_EXTENSIONS`: Set to `true` (default) to enable transaction extensions. Set to `false` to disable.
|
|
448
|
+
|
|
418
449
|
|
|
419
450
|
## Collection Pagination
|
|
420
451
|
|
|
@@ -36,11 +36,10 @@ SFEOS (stac-fastapi-elasticsearch-opensearch) is a high-performance, scalable AP
|
|
|
36
36
|
- **Scale to millions of geospatial assets** with fast search performance through optimized spatial indexing and query capabilities
|
|
37
37
|
- **Support OGC-compliant filtering** including spatial operations (intersects, contains, etc.) and temporal queries
|
|
38
38
|
- **Perform geospatial aggregations** to analyze data distribution across space and time
|
|
39
|
+
- **Enhanced collection search capabilities** with support for sorting and field selection
|
|
39
40
|
|
|
40
41
|
This implementation builds on the STAC-FastAPI framework, providing a production-ready solution specifically optimized for Elasticsearch and OpenSearch databases. It's ideal for organizations managing large geospatial data catalogs who need efficient discovery and access capabilities through standardized APIs.
|
|
41
42
|
|
|
42
|
-
|
|
43
|
-
|
|
44
43
|
## Common Deployment Patterns
|
|
45
44
|
|
|
46
45
|
stac-fastapi-elasticsearch-opensearch can be deployed in several ways depending on your needs:
|
|
@@ -66,26 +65,44 @@ This project is built on the following technologies: STAC, stac-fastapi, FastAPI
|
|
|
66
65
|
|
|
67
66
|
## Table of Contents
|
|
68
67
|
|
|
69
|
-
- [
|
|
70
|
-
- [
|
|
71
|
-
- [
|
|
72
|
-
- [
|
|
73
|
-
- [
|
|
74
|
-
- [
|
|
75
|
-
- [
|
|
76
|
-
- [
|
|
77
|
-
- [
|
|
78
|
-
- [
|
|
79
|
-
- [
|
|
80
|
-
- [
|
|
81
|
-
- [
|
|
82
|
-
- [
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
- [
|
|
86
|
-
- [
|
|
87
|
-
- [
|
|
88
|
-
- [
|
|
68
|
+
- [stac-fastapi-elasticsearch-opensearch](#stac-fastapi-elasticsearch-opensearch)
|
|
69
|
+
- [Sponsors \& Supporters](#sponsors--supporters)
|
|
70
|
+
- [Project Introduction - What is SFEOS?](#project-introduction---what-is-sfeos)
|
|
71
|
+
- [Common Deployment Patterns](#common-deployment-patterns)
|
|
72
|
+
- [Technologies](#technologies)
|
|
73
|
+
- [Table of Contents](#table-of-contents)
|
|
74
|
+
- [Collection Search Extensions](#collection-search-extensions)
|
|
75
|
+
- [Documentation \& Resources](#documentation--resources)
|
|
76
|
+
- [Package Structure](#package-structure)
|
|
77
|
+
- [Examples](#examples)
|
|
78
|
+
- [Performance](#performance)
|
|
79
|
+
- [Direct Response Mode](#direct-response-mode)
|
|
80
|
+
- [Quick Start](#quick-start)
|
|
81
|
+
- [Installation](#installation)
|
|
82
|
+
- [Running Locally](#running-locally)
|
|
83
|
+
- [Using Pre-built Docker Images](#using-pre-built-docker-images)
|
|
84
|
+
- [Using Docker Compose](#using-docker-compose)
|
|
85
|
+
- [Configuration Reference](#configuration-reference)
|
|
86
|
+
- [Datetime-Based Index Management](#datetime-based-index-management)
|
|
87
|
+
- [Overview](#overview)
|
|
88
|
+
- [When to Use](#when-to-use)
|
|
89
|
+
- [Configuration](#configuration)
|
|
90
|
+
- [Enabling Datetime-Based Indexing](#enabling-datetime-based-indexing)
|
|
91
|
+
- [Related Configuration Variables](#related-configuration-variables)
|
|
92
|
+
- [How Datetime-Based Indexing Works](#how-datetime-based-indexing-works)
|
|
93
|
+
- [Index and Alias Naming Convention](#index-and-alias-naming-convention)
|
|
94
|
+
- [Index Size Management](#index-size-management)
|
|
95
|
+
- [Interacting with the API](#interacting-with-the-api)
|
|
96
|
+
- [Configure the API](#configure-the-api)
|
|
97
|
+
- [Collection Pagination](#collection-pagination)
|
|
98
|
+
- [Ingesting Sample Data CLI Tool](#ingesting-sample-data-cli-tool)
|
|
99
|
+
- [Elasticsearch Mappings](#elasticsearch-mappings)
|
|
100
|
+
- [Managing Elasticsearch Indices](#managing-elasticsearch-indices)
|
|
101
|
+
- [Snapshots](#snapshots)
|
|
102
|
+
- [Reindexing](#reindexing)
|
|
103
|
+
- [Auth](#auth)
|
|
104
|
+
- [Aggregation](#aggregation)
|
|
105
|
+
- [Rate Limiting](#rate-limiting)
|
|
89
106
|
|
|
90
107
|
## Documentation & Resources
|
|
91
108
|
|
|
@@ -96,6 +113,37 @@ This project is built on the following technologies: STAC, stac-fastapi, FastAPI
|
|
|
96
113
|
- [Gitter Chat](https://app.gitter.im/#/room/#stac-fastapi-elasticsearch_community:gitter.im) - For real-time discussions
|
|
97
114
|
- [GitHub Discussions](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/discussions) - For longer-form questions and answers
|
|
98
115
|
|
|
116
|
+
## Collection Search Extensions
|
|
117
|
+
|
|
118
|
+
SFEOS implements extended capabilities for the `/collections` endpoint, allowing for more powerful collection discovery:
|
|
119
|
+
|
|
120
|
+
- **Sorting**: Sort collections by sortable fields using the `sortby` parameter
|
|
121
|
+
- Example: `/collections?sortby=+id` (ascending sort by ID)
|
|
122
|
+
- Example: `/collections?sortby=-id` (descending sort by ID)
|
|
123
|
+
- Example: `/collections?sortby=-temporal` (descending sort by temporal extent)
|
|
124
|
+
|
|
125
|
+
- **Field Selection**: Request only specific fields to be returned using the `fields` parameter
|
|
126
|
+
- Example: `/collections?fields=id,title,description`
|
|
127
|
+
- This helps reduce payload size when only certain fields are needed
|
|
128
|
+
|
|
129
|
+
- **Free Text Search**: Search across collection text fields using the `q` parameter
|
|
130
|
+
- Example: `/collections?q=landsat`
|
|
131
|
+
- Searches across multiple text fields including title, description, and keywords
|
|
132
|
+
- Supports partial word matching and relevance-based sorting
|
|
133
|
+
|
|
134
|
+
These extensions make it easier to build user interfaces that display and navigate through collections efficiently.
|
|
135
|
+
|
|
136
|
+
> **Configuration**: Collection search extensions can be disabled by setting the `ENABLE_COLLECTIONS_SEARCH` environment variable to `false`. By default, these extensions are enabled.
|
|
137
|
+
|
|
138
|
+
> **Note**: Sorting is only available on fields that are indexed for sorting in Elasticsearch/OpenSearch. With the default mappings, you can sort on:
|
|
139
|
+
> - `id` (keyword field)
|
|
140
|
+
> - `extent.temporal.interval` (date field)
|
|
141
|
+
> - `temporal` (alias to extent.temporal.interval)
|
|
142
|
+
>
|
|
143
|
+
> Text fields like `title` and `description` are not sortable by default as they use text analysis for better search capabilities. Attempting to sort on these fields will result in a user-friendly error message explaining which fields are sortable and how to make additional fields sortable by updating the mappings.
|
|
144
|
+
>
|
|
145
|
+
> **Important**: Adding keyword fields to make text fields sortable can significantly increase the index size, especially for large text fields. Consider the storage implications when deciding which fields to make sortable.
|
|
146
|
+
|
|
99
147
|
## Package Structure
|
|
100
148
|
|
|
101
149
|
This project is organized into several packages, each with a specific purpose:
|
|
@@ -226,8 +274,12 @@ You can customize additional settings in your `.env` file:
|
|
|
226
274
|
| `ENABLE_DIRECT_RESPONSE` | Enable direct response for maximum performance (disables all FastAPI dependencies, including authentication, custom status codes, and validation) | `false` | Optional |
|
|
227
275
|
| `RAISE_ON_BULK_ERROR` | Controls whether bulk insert operations raise exceptions on errors. If set to `true`, the operation will stop and raise an exception when an error occurs. If set to `false`, errors will be logged, and the operation will continue. **Note:** STAC Item and ItemCollection validation errors will always raise, regardless of this flag. | `false` | Optional |
|
|
228
276
|
| `DATABASE_REFRESH` | Controls whether database operations refresh the index immediately after changes. If set to `true`, changes will be immediately searchable. If set to `false`, changes may not be immediately visible but can improve performance for bulk operations. If set to `wait_for`, changes will wait for the next refresh cycle to become visible. | `false` | Optional |
|
|
277
|
+
| `ENABLE_COLLECTIONS_SEARCH` | Enable collection search extensions (sort, fields). | `true` | Optional |
|
|
229
278
|
| `ENABLE_TRANSACTIONS_EXTENSIONS` | Enables or disables the Transactions and Bulk Transactions API extensions. If set to `false`, the POST `/collections` route and related transaction endpoints (including bulk transaction operations) will be unavailable in the API. This is useful for deployments where mutating the catalog via the API should be prevented. | `true` | Optional |
|
|
230
279
|
| `STAC_ITEM_LIMIT` | Sets the environment variable for result limiting to SFEOS for the number of returned items and STAC collections. | `10` | Optional |
|
|
280
|
+
| `STAC_INDEX_ASSETS` | Controls if Assets are indexed when added to Elasticsearch/Opensearch. This allows asset fields to be included in search queries. | `false` | Optional |
|
|
281
|
+
| `ENV_MAX_LIMIT` | Configures the environment variable in SFEOS to override the default `MAX_LIMIT`, which controls the limit parameter for returned items and STAC collections. | `10,000` | Optional |
|
|
282
|
+
| `USE_DATETIME` | Configures the datetime search behavior in SFEOS. When enabled, searches both datetime field and falls back to start_datetime/end_datetime range for items with null datetime. When disabled, searches only by start_datetime/end_datetime range. | True | Optional |
|
|
231
283
|
|
|
232
284
|
> [!NOTE]
|
|
233
285
|
> The variables `ES_HOST`, `ES_PORT`, `ES_USE_SSL`, `ES_VERIFY_CERTS` and `ES_TIMEOUT` apply to both Elasticsearch and OpenSearch backends, so there is no need to rename the key names to `OS_` even if you're using OpenSearch.
|
|
@@ -369,6 +421,10 @@ The system uses a precise naming convention:
|
|
|
369
421
|
- **Root Path Configuration**: The application root path is the base URL by default.
|
|
370
422
|
- For AWS Lambda with Gateway API: Set `STAC_FASTAPI_ROOT_PATH` to match the Gateway API stage name (e.g., `/v1`)
|
|
371
423
|
|
|
424
|
+
- **Feature Configuration**: Control which features are enabled:
|
|
425
|
+
- `ENABLE_COLLECTIONS_SEARCH`: Set to `true` (default) to enable collection search extensions (sort, fields). Set to `false` to disable.
|
|
426
|
+
- `ENABLE_TRANSACTIONS_EXTENSIONS`: Set to `true` (default) to enable transaction extensions. Set to `false` to disable.
|
|
427
|
+
|
|
372
428
|
|
|
373
429
|
## Collection Pagination
|
|
374
430
|
|
|
@@ -6,8 +6,8 @@ with open("README.md") as f:
|
|
|
6
6
|
desc = f.read()
|
|
7
7
|
|
|
8
8
|
install_requires = [
|
|
9
|
-
"stac-fastapi-core==6.
|
|
10
|
-
"sfeos-helpers==6.
|
|
9
|
+
"stac-fastapi-core==6.4.0",
|
|
10
|
+
"sfeos-helpers==6.4.0",
|
|
11
11
|
"opensearch-py~=2.8.0",
|
|
12
12
|
"opensearch-py[async]~=2.8.0",
|
|
13
13
|
"uvicorn~=0.23.0",
|
{stac_fastapi_opensearch-6.2.1 → stac_fastapi_opensearch-6.4.0}/stac_fastapi/opensearch/app.py
RENAMED
|
@@ -7,7 +7,12 @@ from contextlib import asynccontextmanager
|
|
|
7
7
|
from fastapi import FastAPI
|
|
8
8
|
|
|
9
9
|
from stac_fastapi.api.app import StacApi
|
|
10
|
-
from stac_fastapi.api.models import
|
|
10
|
+
from stac_fastapi.api.models import (
|
|
11
|
+
ItemCollectionUri,
|
|
12
|
+
create_get_request_model,
|
|
13
|
+
create_post_request_model,
|
|
14
|
+
create_request_model,
|
|
15
|
+
)
|
|
11
16
|
from stac_fastapi.core.core import (
|
|
12
17
|
BulkTransactionsClient,
|
|
13
18
|
CoreClient,
|
|
@@ -23,7 +28,7 @@ from stac_fastapi.core.rate_limit import setup_rate_limit
|
|
|
23
28
|
from stac_fastapi.core.route_dependencies import get_route_dependencies
|
|
24
29
|
from stac_fastapi.core.session import Session
|
|
25
30
|
from stac_fastapi.core.utilities import get_bool_env
|
|
26
|
-
from stac_fastapi.extensions.core import (
|
|
31
|
+
from stac_fastapi.extensions.core import ( # CollectionSearchFilterExtension,
|
|
27
32
|
AggregationExtension,
|
|
28
33
|
CollectionSearchExtension,
|
|
29
34
|
FilterExtension,
|
|
@@ -32,7 +37,11 @@ from stac_fastapi.extensions.core import (
|
|
|
32
37
|
TokenPaginationExtension,
|
|
33
38
|
TransactionExtension,
|
|
34
39
|
)
|
|
40
|
+
from stac_fastapi.extensions.core.fields import FieldsConformanceClasses
|
|
35
41
|
from stac_fastapi.extensions.core.filter import FilterConformanceClasses
|
|
42
|
+
from stac_fastapi.extensions.core.free_text import FreeTextConformanceClasses
|
|
43
|
+
from stac_fastapi.extensions.core.query import QueryConformanceClasses
|
|
44
|
+
from stac_fastapi.extensions.core.sort import SortConformanceClasses
|
|
36
45
|
from stac_fastapi.extensions.third_party import BulkTransactionExtension
|
|
37
46
|
from stac_fastapi.opensearch.config import OpensearchSettings
|
|
38
47
|
from stac_fastapi.opensearch.database_logic import (
|
|
@@ -47,7 +56,9 @@ logging.basicConfig(level=logging.INFO)
|
|
|
47
56
|
logger = logging.getLogger(__name__)
|
|
48
57
|
|
|
49
58
|
TRANSACTIONS_EXTENSIONS = get_bool_env("ENABLE_TRANSACTIONS_EXTENSIONS", default=True)
|
|
59
|
+
ENABLE_COLLECTIONS_SEARCH = get_bool_env("ENABLE_COLLECTIONS_SEARCH", default=True)
|
|
50
60
|
logger.info("TRANSACTIONS_EXTENSIONS is set to %s", TRANSACTIONS_EXTENSIONS)
|
|
61
|
+
logger.info("ENABLE_COLLECTIONS_SEARCH is set to %s", ENABLE_COLLECTIONS_SEARCH)
|
|
51
62
|
|
|
52
63
|
settings = OpensearchSettings()
|
|
53
64
|
session = Session.create_from_settings(settings)
|
|
@@ -61,14 +72,6 @@ filter_extension.conformance_classes.append(
|
|
|
61
72
|
FilterConformanceClasses.ADVANCED_COMPARISON_OPERATORS
|
|
62
73
|
)
|
|
63
74
|
|
|
64
|
-
# Adding collection search extension for compatibility with stac-auth-proxy
|
|
65
|
-
# (https://github.com/developmentseed/stac-auth-proxy)
|
|
66
|
-
# The extension is not fully implemented yet but is required for collection filtering support
|
|
67
|
-
collection_search_extension = CollectionSearchExtension()
|
|
68
|
-
collection_search_extension.conformance_classes.append(
|
|
69
|
-
"https://api.stacspec.org/v1.0.0-rc.1/collection-search#filter"
|
|
70
|
-
)
|
|
71
|
-
|
|
72
75
|
aggregation_extension = AggregationExtension(
|
|
73
76
|
client=EsAsyncBaseAggregationClient(
|
|
74
77
|
database=database_logic, session=session, settings=settings
|
|
@@ -77,14 +80,16 @@ aggregation_extension = AggregationExtension(
|
|
|
77
80
|
aggregation_extension.POST = EsAggregationExtensionPostRequest
|
|
78
81
|
aggregation_extension.GET = EsAggregationExtensionGetRequest
|
|
79
82
|
|
|
83
|
+
fields_extension = FieldsExtension()
|
|
84
|
+
fields_extension.conformance_classes.append(FieldsConformanceClasses.ITEMS)
|
|
85
|
+
|
|
80
86
|
search_extensions = [
|
|
81
|
-
|
|
87
|
+
fields_extension,
|
|
82
88
|
QueryExtension(),
|
|
83
89
|
SortExtension(),
|
|
84
90
|
TokenPaginationExtension(),
|
|
85
91
|
filter_extension,
|
|
86
92
|
FreeTextExtension(),
|
|
87
|
-
collection_search_extension,
|
|
88
93
|
]
|
|
89
94
|
|
|
90
95
|
|
|
@@ -111,14 +116,51 @@ if TRANSACTIONS_EXTENSIONS:
|
|
|
111
116
|
|
|
112
117
|
extensions = [aggregation_extension] + search_extensions
|
|
113
118
|
|
|
119
|
+
# Create collection search extensions if enabled
|
|
120
|
+
if ENABLE_COLLECTIONS_SEARCH:
|
|
121
|
+
# Create collection search extensions
|
|
122
|
+
collection_search_extensions = [
|
|
123
|
+
# QueryExtension(conformance_classes=[QueryConformanceClasses.COLLECTIONS]),
|
|
124
|
+
SortExtension(conformance_classes=[SortConformanceClasses.COLLECTIONS]),
|
|
125
|
+
FieldsExtension(conformance_classes=[FieldsConformanceClasses.COLLECTIONS]),
|
|
126
|
+
# CollectionSearchFilterExtension(
|
|
127
|
+
# conformance_classes=[FilterConformanceClasses.COLLECTIONS]
|
|
128
|
+
# ),
|
|
129
|
+
FreeTextExtension(conformance_classes=[FreeTextConformanceClasses.COLLECTIONS]),
|
|
130
|
+
]
|
|
131
|
+
|
|
132
|
+
# Initialize collection search with its extensions
|
|
133
|
+
collection_search_ext = CollectionSearchExtension.from_extensions(
|
|
134
|
+
collection_search_extensions
|
|
135
|
+
)
|
|
136
|
+
collections_get_request_model = collection_search_ext.GET
|
|
137
|
+
|
|
138
|
+
extensions.append(collection_search_ext)
|
|
139
|
+
|
|
114
140
|
database_logic.extensions = [type(ext).__name__ for ext in extensions]
|
|
115
141
|
|
|
116
142
|
post_request_model = create_post_request_model(search_extensions)
|
|
117
143
|
|
|
144
|
+
items_get_request_model = create_request_model(
|
|
145
|
+
model_name="ItemCollectionUri",
|
|
146
|
+
base_model=ItemCollectionUri,
|
|
147
|
+
extensions=[
|
|
148
|
+
SortExtension(
|
|
149
|
+
conformance_classes=[SortConformanceClasses.ITEMS],
|
|
150
|
+
),
|
|
151
|
+
QueryExtension(
|
|
152
|
+
conformance_classes=[QueryConformanceClasses.ITEMS],
|
|
153
|
+
),
|
|
154
|
+
filter_extension,
|
|
155
|
+
FieldsExtension(conformance_classes=[FieldsConformanceClasses.ITEMS]),
|
|
156
|
+
],
|
|
157
|
+
request_type="GET",
|
|
158
|
+
)
|
|
159
|
+
|
|
118
160
|
app_config = {
|
|
119
161
|
"title": os.getenv("STAC_FASTAPI_TITLE", "stac-fastapi-opensearch"),
|
|
120
162
|
"description": os.getenv("STAC_FASTAPI_DESCRIPTION", "stac-fastapi-opensearch"),
|
|
121
|
-
"api_version": os.getenv("STAC_FASTAPI_VERSION", "6.
|
|
163
|
+
"api_version": os.getenv("STAC_FASTAPI_VERSION", "6.0.0"),
|
|
122
164
|
"settings": settings,
|
|
123
165
|
"extensions": extensions,
|
|
124
166
|
"client": CoreClient(
|
|
@@ -129,9 +171,14 @@ app_config = {
|
|
|
129
171
|
),
|
|
130
172
|
"search_get_request_model": create_get_request_model(search_extensions),
|
|
131
173
|
"search_post_request_model": post_request_model,
|
|
174
|
+
"items_get_request_model": items_get_request_model,
|
|
132
175
|
"route_dependencies": get_route_dependencies(),
|
|
133
176
|
}
|
|
134
177
|
|
|
178
|
+
# Add collections_get_request_model if collection search is enabled
|
|
179
|
+
if ENABLE_COLLECTIONS_SEARCH:
|
|
180
|
+
app_config["collections_get_request_model"] = collections_get_request_model
|
|
181
|
+
|
|
135
182
|
api = StacApi(**app_config)
|
|
136
183
|
|
|
137
184
|
|
|
@@ -3,8 +3,9 @@
|
|
|
3
3
|
import asyncio
|
|
4
4
|
import logging
|
|
5
5
|
from base64 import urlsafe_b64decode, urlsafe_b64encode
|
|
6
|
+
from collections.abc import Iterable
|
|
6
7
|
from copy import deepcopy
|
|
7
|
-
from typing import Any, Dict,
|
|
8
|
+
from typing import Any, Dict, List, Optional, Tuple, Type
|
|
8
9
|
|
|
9
10
|
import attr
|
|
10
11
|
import orjson
|
|
@@ -16,7 +17,7 @@ from starlette.requests import Request
|
|
|
16
17
|
|
|
17
18
|
from stac_fastapi.core.base_database_logic import BaseDatabaseLogic
|
|
18
19
|
from stac_fastapi.core.serializers import CollectionSerializer, ItemSerializer
|
|
19
|
-
from stac_fastapi.core.utilities import
|
|
20
|
+
from stac_fastapi.core.utilities import bbox2polygon, get_bool_env, get_max_limit
|
|
20
21
|
from stac_fastapi.extensions.core.transaction.request import (
|
|
21
22
|
PartialCollection,
|
|
22
23
|
PartialItem,
|
|
@@ -153,31 +154,95 @@ class DatabaseLogic(BaseDatabaseLogic):
|
|
|
153
154
|
"""CORE LOGIC"""
|
|
154
155
|
|
|
155
156
|
async def get_all_collections(
|
|
156
|
-
self,
|
|
157
|
+
self,
|
|
158
|
+
token: Optional[str],
|
|
159
|
+
limit: int,
|
|
160
|
+
request: Request,
|
|
161
|
+
sort: Optional[List[Dict[str, Any]]] = None,
|
|
162
|
+
q: Optional[List[str]] = None,
|
|
157
163
|
) -> Tuple[List[Dict[str, Any]], Optional[str]]:
|
|
158
|
-
"""
|
|
159
|
-
Retrieve a list of all collections from Opensearch, supporting pagination.
|
|
164
|
+
"""Retrieve a list of collections from Elasticsearch, supporting pagination.
|
|
160
165
|
|
|
161
166
|
Args:
|
|
162
167
|
token (Optional[str]): The pagination token.
|
|
163
168
|
limit (int): The number of results to return.
|
|
169
|
+
request (Request): The FastAPI request object.
|
|
170
|
+
sort (Optional[List[Dict[str, Any]]]): Optional sort parameter from the request.
|
|
171
|
+
q (Optional[List[str]]): Free text search terms.
|
|
164
172
|
|
|
165
173
|
Returns:
|
|
166
174
|
A tuple of (collections, next pagination token if any).
|
|
175
|
+
|
|
176
|
+
Raises:
|
|
177
|
+
HTTPException: If sorting is requested on a field that is not sortable.
|
|
167
178
|
"""
|
|
168
|
-
|
|
169
|
-
|
|
179
|
+
# Define sortable fields based on the ES_COLLECTIONS_MAPPINGS
|
|
180
|
+
sortable_fields = ["id", "extent.temporal.interval", "temporal"]
|
|
181
|
+
|
|
182
|
+
# Format the sort parameter
|
|
183
|
+
formatted_sort = []
|
|
184
|
+
if sort:
|
|
185
|
+
for item in sort:
|
|
186
|
+
field = item.get("field")
|
|
187
|
+
direction = item.get("direction", "asc")
|
|
188
|
+
if field:
|
|
189
|
+
# Validate that the field is sortable
|
|
190
|
+
if field not in sortable_fields:
|
|
191
|
+
raise HTTPException(
|
|
192
|
+
status_code=400,
|
|
193
|
+
detail=f"Field '{field}' is not sortable. Sortable fields are: {', '.join(sortable_fields)}. "
|
|
194
|
+
+ "Text fields are not sortable by default in OpenSearch. "
|
|
195
|
+
+ "To make a field sortable, update the mapping to use 'keyword' type or add a '.keyword' subfield. ",
|
|
196
|
+
)
|
|
197
|
+
formatted_sort.append({field: {"order": direction}})
|
|
198
|
+
# Always include id as a secondary sort to ensure consistent pagination
|
|
199
|
+
if not any("id" in item for item in formatted_sort):
|
|
200
|
+
formatted_sort.append({"id": {"order": "asc"}})
|
|
201
|
+
else:
|
|
202
|
+
formatted_sort = [{"id": {"order": "asc"}}]
|
|
203
|
+
|
|
204
|
+
body = {
|
|
205
|
+
"sort": formatted_sort,
|
|
170
206
|
"size": limit,
|
|
171
207
|
}
|
|
172
208
|
|
|
173
|
-
# Only add search_after to the query if token is not None and not empty
|
|
174
209
|
if token:
|
|
175
|
-
search_after = [token]
|
|
176
|
-
|
|
210
|
+
body["search_after"] = [token]
|
|
211
|
+
|
|
212
|
+
# Apply free text query if provided
|
|
213
|
+
if q:
|
|
214
|
+
# For collections, we want to search across all relevant fields
|
|
215
|
+
should_clauses = []
|
|
216
|
+
|
|
217
|
+
# For each search term
|
|
218
|
+
for term in q:
|
|
219
|
+
# Create a multi_match query for each term
|
|
220
|
+
for field in [
|
|
221
|
+
"id",
|
|
222
|
+
"title",
|
|
223
|
+
"description",
|
|
224
|
+
"keywords",
|
|
225
|
+
"summaries.platform",
|
|
226
|
+
"summaries.constellation",
|
|
227
|
+
"providers.name",
|
|
228
|
+
"providers.url",
|
|
229
|
+
]:
|
|
230
|
+
should_clauses.append(
|
|
231
|
+
{
|
|
232
|
+
"wildcard": {
|
|
233
|
+
field: {"value": f"*{term}*", "case_insensitive": True}
|
|
234
|
+
}
|
|
235
|
+
}
|
|
236
|
+
)
|
|
237
|
+
|
|
238
|
+
# Add the query to the body using bool query with should clauses
|
|
239
|
+
body["query"] = {
|
|
240
|
+
"bool": {"should": should_clauses, "minimum_should_match": 1}
|
|
241
|
+
}
|
|
177
242
|
|
|
178
243
|
response = await self.client.search(
|
|
179
244
|
index=COLLECTIONS_INDEX,
|
|
180
|
-
body=
|
|
245
|
+
body=body,
|
|
181
246
|
)
|
|
182
247
|
|
|
183
248
|
hits = response["hits"]["hits"]
|
|
@@ -301,21 +366,94 @@ class DatabaseLogic(BaseDatabaseLogic):
|
|
|
301
366
|
if not datetime_search:
|
|
302
367
|
return search, datetime_search
|
|
303
368
|
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
|
|
309
|
-
|
|
310
|
-
|
|
311
|
-
|
|
312
|
-
|
|
313
|
-
|
|
314
|
-
|
|
315
|
-
|
|
316
|
-
|
|
369
|
+
# USE_DATETIME env var
|
|
370
|
+
# True: Search by datetime, if null search by start/end datetime
|
|
371
|
+
# False: Always search only by start/end datetime
|
|
372
|
+
USE_DATETIME = get_bool_env("USE_DATETIME", default=True)
|
|
373
|
+
|
|
374
|
+
if USE_DATETIME:
|
|
375
|
+
if "eq" in datetime_search:
|
|
376
|
+
# For exact matches, include:
|
|
377
|
+
# 1. Items with matching exact datetime
|
|
378
|
+
# 2. Items with datetime:null where the time falls within their range
|
|
379
|
+
should = [
|
|
380
|
+
Q(
|
|
381
|
+
"bool",
|
|
382
|
+
filter=[
|
|
383
|
+
Q("exists", field="properties.datetime"),
|
|
384
|
+
Q(
|
|
385
|
+
"term",
|
|
386
|
+
**{"properties__datetime": datetime_search["eq"]},
|
|
387
|
+
),
|
|
388
|
+
],
|
|
389
|
+
),
|
|
390
|
+
Q(
|
|
391
|
+
"bool",
|
|
392
|
+
must_not=[Q("exists", field="properties.datetime")],
|
|
393
|
+
filter=[
|
|
394
|
+
Q("exists", field="properties.start_datetime"),
|
|
395
|
+
Q("exists", field="properties.end_datetime"),
|
|
396
|
+
Q(
|
|
397
|
+
"range",
|
|
398
|
+
properties__start_datetime={
|
|
399
|
+
"lte": datetime_search["eq"]
|
|
400
|
+
},
|
|
401
|
+
),
|
|
402
|
+
Q(
|
|
403
|
+
"range",
|
|
404
|
+
properties__end_datetime={"gte": datetime_search["eq"]},
|
|
405
|
+
),
|
|
406
|
+
],
|
|
407
|
+
),
|
|
408
|
+
]
|
|
409
|
+
else:
|
|
410
|
+
# For date ranges, include:
|
|
411
|
+
# 1. Items with datetime in the range
|
|
412
|
+
# 2. Items with datetime:null that overlap the search range
|
|
413
|
+
should = [
|
|
414
|
+
Q(
|
|
415
|
+
"bool",
|
|
416
|
+
filter=[
|
|
417
|
+
Q("exists", field="properties.datetime"),
|
|
418
|
+
Q(
|
|
419
|
+
"range",
|
|
420
|
+
properties__datetime={
|
|
421
|
+
"gte": datetime_search["gte"],
|
|
422
|
+
"lte": datetime_search["lte"],
|
|
423
|
+
},
|
|
424
|
+
),
|
|
425
|
+
],
|
|
426
|
+
),
|
|
427
|
+
Q(
|
|
428
|
+
"bool",
|
|
429
|
+
must_not=[Q("exists", field="properties.datetime")],
|
|
430
|
+
filter=[
|
|
431
|
+
Q("exists", field="properties.start_datetime"),
|
|
432
|
+
Q("exists", field="properties.end_datetime"),
|
|
433
|
+
Q(
|
|
434
|
+
"range",
|
|
435
|
+
properties__start_datetime={
|
|
436
|
+
"lte": datetime_search["lte"]
|
|
437
|
+
},
|
|
438
|
+
),
|
|
439
|
+
Q(
|
|
440
|
+
"range",
|
|
441
|
+
properties__end_datetime={
|
|
442
|
+
"gte": datetime_search["gte"]
|
|
443
|
+
},
|
|
444
|
+
),
|
|
445
|
+
],
|
|
446
|
+
),
|
|
447
|
+
]
|
|
448
|
+
|
|
449
|
+
return (
|
|
450
|
+
search.query(Q("bool", should=should, minimum_should_match=1)),
|
|
451
|
+
datetime_search,
|
|
452
|
+
)
|
|
453
|
+
else:
|
|
454
|
+
if "eq" in datetime_search:
|
|
455
|
+
filter_query = Q(
|
|
317
456
|
"bool",
|
|
318
|
-
must_not=[Q("exists", field="properties.datetime")],
|
|
319
457
|
filter=[
|
|
320
458
|
Q("exists", field="properties.start_datetime"),
|
|
321
459
|
Q("exists", field="properties.end_datetime"),
|
|
@@ -328,29 +466,10 @@ class DatabaseLogic(BaseDatabaseLogic):
|
|
|
328
466
|
properties__end_datetime={"gte": datetime_search["eq"]},
|
|
329
467
|
),
|
|
330
468
|
],
|
|
331
|
-
)
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
# For date ranges, include:
|
|
335
|
-
# 1. Items with datetime in the range
|
|
336
|
-
# 2. Items with datetime:null that overlap the search range
|
|
337
|
-
should = [
|
|
338
|
-
Q(
|
|
339
|
-
"bool",
|
|
340
|
-
filter=[
|
|
341
|
-
Q("exists", field="properties.datetime"),
|
|
342
|
-
Q(
|
|
343
|
-
"range",
|
|
344
|
-
properties__datetime={
|
|
345
|
-
"gte": datetime_search["gte"],
|
|
346
|
-
"lte": datetime_search["lte"],
|
|
347
|
-
},
|
|
348
|
-
),
|
|
349
|
-
],
|
|
350
|
-
),
|
|
351
|
-
Q(
|
|
469
|
+
)
|
|
470
|
+
else:
|
|
471
|
+
filter_query = Q(
|
|
352
472
|
"bool",
|
|
353
|
-
must_not=[Q("exists", field="properties.datetime")],
|
|
354
473
|
filter=[
|
|
355
474
|
Q("exists", field="properties.start_datetime"),
|
|
356
475
|
Q("exists", field="properties.end_datetime"),
|
|
@@ -363,13 +482,8 @@ class DatabaseLogic(BaseDatabaseLogic):
|
|
|
363
482
|
properties__end_datetime={"gte": datetime_search["gte"]},
|
|
364
483
|
),
|
|
365
484
|
],
|
|
366
|
-
)
|
|
367
|
-
|
|
368
|
-
|
|
369
|
-
return (
|
|
370
|
-
search.query(Q("bool", should=should, minimum_should_match=1)),
|
|
371
|
-
datetime_search,
|
|
372
|
-
)
|
|
485
|
+
)
|
|
486
|
+
return search.query(filter_query), datetime_search
|
|
373
487
|
|
|
374
488
|
@staticmethod
|
|
375
489
|
def apply_bbox_filter(search: Search, bbox: List):
|
|
@@ -540,7 +654,7 @@ class DatabaseLogic(BaseDatabaseLogic):
|
|
|
540
654
|
|
|
541
655
|
search_body["sort"] = sort if sort else DEFAULT_SORT
|
|
542
656
|
|
|
543
|
-
max_result_window =
|
|
657
|
+
max_result_window = get_max_limit()
|
|
544
658
|
|
|
545
659
|
size_limit = min(limit + 1, max_result_window)
|
|
546
660
|
|
|
@@ -679,14 +793,21 @@ class DatabaseLogic(BaseDatabaseLogic):
|
|
|
679
793
|
|
|
680
794
|
"""
|
|
681
795
|
await self.check_collection_exists(collection_id=item["collection"])
|
|
796
|
+
alias = index_alias_by_collection_id(item["collection"])
|
|
797
|
+
doc_id = mk_item_id(item["id"], item["collection"])
|
|
682
798
|
|
|
683
|
-
if not exist_ok
|
|
684
|
-
|
|
685
|
-
|
|
686
|
-
|
|
687
|
-
|
|
688
|
-
|
|
689
|
-
|
|
799
|
+
if not exist_ok:
|
|
800
|
+
alias_exists = await self.client.indices.exists_alias(name=alias)
|
|
801
|
+
|
|
802
|
+
if alias_exists:
|
|
803
|
+
alias_info = await self.client.indices.get_alias(name=alias)
|
|
804
|
+
indices = list(alias_info.keys())
|
|
805
|
+
|
|
806
|
+
for index in indices:
|
|
807
|
+
if await self.client.exists(index=index, id=doc_id):
|
|
808
|
+
raise ConflictError(
|
|
809
|
+
f"Item {item['id']} in collection {item['collection']} already exists"
|
|
810
|
+
)
|
|
690
811
|
|
|
691
812
|
return self.item_serializer.stac_to_db(item, base_url)
|
|
692
813
|
|
|
@@ -903,7 +1024,6 @@ class DatabaseLogic(BaseDatabaseLogic):
|
|
|
903
1024
|
"add",
|
|
904
1025
|
"replace",
|
|
905
1026
|
]:
|
|
906
|
-
|
|
907
1027
|
if operation.path == "collection" and collection_id != operation.value:
|
|
908
1028
|
await self.check_collection_exists(collection_id=operation.value)
|
|
909
1029
|
new_collection_id = operation.value
|
|
@@ -957,8 +1077,8 @@ class DatabaseLogic(BaseDatabaseLogic):
|
|
|
957
1077
|
"script": {
|
|
958
1078
|
"lang": "painless",
|
|
959
1079
|
"source": (
|
|
960
|
-
f"""ctx._id = ctx._id.replace('{collection_id}', '{new_collection_id}');"""
|
|
961
|
-
f"""ctx._source.collection = '{new_collection_id}';"""
|
|
1080
|
+
f"""ctx._id = ctx._id.replace('{collection_id}', '{new_collection_id}');"""
|
|
1081
|
+
f"""ctx._source.collection = '{new_collection_id}';"""
|
|
962
1082
|
),
|
|
963
1083
|
},
|
|
964
1084
|
},
|
|
@@ -1180,7 +1300,7 @@ class DatabaseLogic(BaseDatabaseLogic):
|
|
|
1180
1300
|
"source": {"index": f"{ITEMS_INDEX_PREFIX}{collection_id}"},
|
|
1181
1301
|
"script": {
|
|
1182
1302
|
"lang": "painless",
|
|
1183
|
-
"source": f"""ctx._id = ctx._id.replace('{collection_id}', '{collection["id"]}'); ctx._source.collection = '{collection["id"]}' ;""",
|
|
1303
|
+
"source": f"""ctx._id = ctx._id.replace('{collection_id}', '{collection["id"]}'); ctx._source.collection = '{collection["id"]}' ;""",
|
|
1184
1304
|
},
|
|
1185
1305
|
},
|
|
1186
1306
|
wait_for_completion=True,
|
{stac_fastapi_opensearch-6.2.1 → stac_fastapi_opensearch-6.4.0}/stac_fastapi/opensearch/version.py
RENAMED
|
@@ -1,2 +1,2 @@
|
|
|
1
1
|
"""library version."""
|
|
2
|
-
__version__ = "6.
|
|
2
|
+
__version__ = "6.4.0"
|
|
@@ -1,6 +1,6 @@
|
|
|
1
|
-
Metadata-Version: 2.
|
|
2
|
-
Name:
|
|
3
|
-
Version: 6.
|
|
1
|
+
Metadata-Version: 2.1
|
|
2
|
+
Name: stac-fastapi-opensearch
|
|
3
|
+
Version: 6.4.0
|
|
4
4
|
Summary: Opensearch stac-fastapi backend.
|
|
5
5
|
Home-page: https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch
|
|
6
6
|
License: MIT
|
|
@@ -15,34 +15,9 @@ Classifier: Programming Language :: Python :: 3.13
|
|
|
15
15
|
Classifier: License :: OSI Approved :: MIT License
|
|
16
16
|
Requires-Python: >=3.9
|
|
17
17
|
Description-Content-Type: text/markdown
|
|
18
|
-
Requires-Dist: stac-fastapi-core==6.2.1
|
|
19
|
-
Requires-Dist: sfeos-helpers==6.2.1
|
|
20
|
-
Requires-Dist: opensearch-py~=2.8.0
|
|
21
|
-
Requires-Dist: opensearch-py[async]~=2.8.0
|
|
22
|
-
Requires-Dist: uvicorn~=0.23.0
|
|
23
|
-
Requires-Dist: starlette<0.36.0,>=0.35.0
|
|
24
18
|
Provides-Extra: dev
|
|
25
|
-
Requires-Dist: pytest~=7.0.0; extra == "dev"
|
|
26
|
-
Requires-Dist: pytest-cov~=4.0.0; extra == "dev"
|
|
27
|
-
Requires-Dist: pytest-asyncio~=0.21.0; extra == "dev"
|
|
28
|
-
Requires-Dist: pre-commit~=3.0.0; extra == "dev"
|
|
29
|
-
Requires-Dist: ciso8601~=2.3.0; extra == "dev"
|
|
30
|
-
Requires-Dist: httpx<0.28.0,>=0.24.0; extra == "dev"
|
|
31
19
|
Provides-Extra: docs
|
|
32
|
-
Requires-Dist: mkdocs~=1.4.0; extra == "docs"
|
|
33
|
-
Requires-Dist: mkdocs-material~=9.0.0; extra == "docs"
|
|
34
|
-
Requires-Dist: pdocs~=1.2.0; extra == "docs"
|
|
35
20
|
Provides-Extra: server
|
|
36
|
-
Requires-Dist: uvicorn[standard]~=0.23.0; extra == "server"
|
|
37
|
-
Dynamic: classifier
|
|
38
|
-
Dynamic: description
|
|
39
|
-
Dynamic: description-content-type
|
|
40
|
-
Dynamic: home-page
|
|
41
|
-
Dynamic: license
|
|
42
|
-
Dynamic: provides-extra
|
|
43
|
-
Dynamic: requires-dist
|
|
44
|
-
Dynamic: requires-python
|
|
45
|
-
Dynamic: summary
|
|
46
21
|
|
|
47
22
|
# stac-fastapi-elasticsearch-opensearch
|
|
48
23
|
|
|
@@ -82,11 +57,10 @@ SFEOS (stac-fastapi-elasticsearch-opensearch) is a high-performance, scalable AP
|
|
|
82
57
|
- **Scale to millions of geospatial assets** with fast search performance through optimized spatial indexing and query capabilities
|
|
83
58
|
- **Support OGC-compliant filtering** including spatial operations (intersects, contains, etc.) and temporal queries
|
|
84
59
|
- **Perform geospatial aggregations** to analyze data distribution across space and time
|
|
60
|
+
- **Enhanced collection search capabilities** with support for sorting and field selection
|
|
85
61
|
|
|
86
62
|
This implementation builds on the STAC-FastAPI framework, providing a production-ready solution specifically optimized for Elasticsearch and OpenSearch databases. It's ideal for organizations managing large geospatial data catalogs who need efficient discovery and access capabilities through standardized APIs.
|
|
87
63
|
|
|
88
|
-
|
|
89
|
-
|
|
90
64
|
## Common Deployment Patterns
|
|
91
65
|
|
|
92
66
|
stac-fastapi-elasticsearch-opensearch can be deployed in several ways depending on your needs:
|
|
@@ -112,26 +86,44 @@ This project is built on the following technologies: STAC, stac-fastapi, FastAPI
|
|
|
112
86
|
|
|
113
87
|
## Table of Contents
|
|
114
88
|
|
|
115
|
-
- [
|
|
116
|
-
- [
|
|
117
|
-
- [
|
|
118
|
-
- [
|
|
119
|
-
- [
|
|
120
|
-
- [
|
|
121
|
-
- [
|
|
122
|
-
- [
|
|
123
|
-
- [
|
|
124
|
-
- [
|
|
125
|
-
- [
|
|
126
|
-
- [
|
|
127
|
-
- [
|
|
128
|
-
- [
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
- [
|
|
132
|
-
- [
|
|
133
|
-
- [
|
|
134
|
-
- [
|
|
89
|
+
- [stac-fastapi-elasticsearch-opensearch](#stac-fastapi-elasticsearch-opensearch)
|
|
90
|
+
- [Sponsors \& Supporters](#sponsors--supporters)
|
|
91
|
+
- [Project Introduction - What is SFEOS?](#project-introduction---what-is-sfeos)
|
|
92
|
+
- [Common Deployment Patterns](#common-deployment-patterns)
|
|
93
|
+
- [Technologies](#technologies)
|
|
94
|
+
- [Table of Contents](#table-of-contents)
|
|
95
|
+
- [Collection Search Extensions](#collection-search-extensions)
|
|
96
|
+
- [Documentation \& Resources](#documentation--resources)
|
|
97
|
+
- [Package Structure](#package-structure)
|
|
98
|
+
- [Examples](#examples)
|
|
99
|
+
- [Performance](#performance)
|
|
100
|
+
- [Direct Response Mode](#direct-response-mode)
|
|
101
|
+
- [Quick Start](#quick-start)
|
|
102
|
+
- [Installation](#installation)
|
|
103
|
+
- [Running Locally](#running-locally)
|
|
104
|
+
- [Using Pre-built Docker Images](#using-pre-built-docker-images)
|
|
105
|
+
- [Using Docker Compose](#using-docker-compose)
|
|
106
|
+
- [Configuration Reference](#configuration-reference)
|
|
107
|
+
- [Datetime-Based Index Management](#datetime-based-index-management)
|
|
108
|
+
- [Overview](#overview)
|
|
109
|
+
- [When to Use](#when-to-use)
|
|
110
|
+
- [Configuration](#configuration)
|
|
111
|
+
- [Enabling Datetime-Based Indexing](#enabling-datetime-based-indexing)
|
|
112
|
+
- [Related Configuration Variables](#related-configuration-variables)
|
|
113
|
+
- [How Datetime-Based Indexing Works](#how-datetime-based-indexing-works)
|
|
114
|
+
- [Index and Alias Naming Convention](#index-and-alias-naming-convention)
|
|
115
|
+
- [Index Size Management](#index-size-management)
|
|
116
|
+
- [Interacting with the API](#interacting-with-the-api)
|
|
117
|
+
- [Configure the API](#configure-the-api)
|
|
118
|
+
- [Collection Pagination](#collection-pagination)
|
|
119
|
+
- [Ingesting Sample Data CLI Tool](#ingesting-sample-data-cli-tool)
|
|
120
|
+
- [Elasticsearch Mappings](#elasticsearch-mappings)
|
|
121
|
+
- [Managing Elasticsearch Indices](#managing-elasticsearch-indices)
|
|
122
|
+
- [Snapshots](#snapshots)
|
|
123
|
+
- [Reindexing](#reindexing)
|
|
124
|
+
- [Auth](#auth)
|
|
125
|
+
- [Aggregation](#aggregation)
|
|
126
|
+
- [Rate Limiting](#rate-limiting)
|
|
135
127
|
|
|
136
128
|
## Documentation & Resources
|
|
137
129
|
|
|
@@ -142,6 +134,37 @@ This project is built on the following technologies: STAC, stac-fastapi, FastAPI
|
|
|
142
134
|
- [Gitter Chat](https://app.gitter.im/#/room/#stac-fastapi-elasticsearch_community:gitter.im) - For real-time discussions
|
|
143
135
|
- [GitHub Discussions](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/discussions) - For longer-form questions and answers
|
|
144
136
|
|
|
137
|
+
## Collection Search Extensions
|
|
138
|
+
|
|
139
|
+
SFEOS implements extended capabilities for the `/collections` endpoint, allowing for more powerful collection discovery:
|
|
140
|
+
|
|
141
|
+
- **Sorting**: Sort collections by sortable fields using the `sortby` parameter
|
|
142
|
+
- Example: `/collections?sortby=+id` (ascending sort by ID)
|
|
143
|
+
- Example: `/collections?sortby=-id` (descending sort by ID)
|
|
144
|
+
- Example: `/collections?sortby=-temporal` (descending sort by temporal extent)
|
|
145
|
+
|
|
146
|
+
- **Field Selection**: Request only specific fields to be returned using the `fields` parameter
|
|
147
|
+
- Example: `/collections?fields=id,title,description`
|
|
148
|
+
- This helps reduce payload size when only certain fields are needed
|
|
149
|
+
|
|
150
|
+
- **Free Text Search**: Search across collection text fields using the `q` parameter
|
|
151
|
+
- Example: `/collections?q=landsat`
|
|
152
|
+
- Searches across multiple text fields including title, description, and keywords
|
|
153
|
+
- Supports partial word matching and relevance-based sorting
|
|
154
|
+
|
|
155
|
+
These extensions make it easier to build user interfaces that display and navigate through collections efficiently.
|
|
156
|
+
|
|
157
|
+
> **Configuration**: Collection search extensions can be disabled by setting the `ENABLE_COLLECTIONS_SEARCH` environment variable to `false`. By default, these extensions are enabled.
|
|
158
|
+
|
|
159
|
+
> **Note**: Sorting is only available on fields that are indexed for sorting in Elasticsearch/OpenSearch. With the default mappings, you can sort on:
|
|
160
|
+
> - `id` (keyword field)
|
|
161
|
+
> - `extent.temporal.interval` (date field)
|
|
162
|
+
> - `temporal` (alias to extent.temporal.interval)
|
|
163
|
+
>
|
|
164
|
+
> Text fields like `title` and `description` are not sortable by default as they use text analysis for better search capabilities. Attempting to sort on these fields will result in a user-friendly error message explaining which fields are sortable and how to make additional fields sortable by updating the mappings.
|
|
165
|
+
>
|
|
166
|
+
> **Important**: Adding keyword fields to make text fields sortable can significantly increase the index size, especially for large text fields. Consider the storage implications when deciding which fields to make sortable.
|
|
167
|
+
|
|
145
168
|
## Package Structure
|
|
146
169
|
|
|
147
170
|
This project is organized into several packages, each with a specific purpose:
|
|
@@ -272,8 +295,12 @@ You can customize additional settings in your `.env` file:
|
|
|
272
295
|
| `ENABLE_DIRECT_RESPONSE` | Enable direct response for maximum performance (disables all FastAPI dependencies, including authentication, custom status codes, and validation) | `false` | Optional |
|
|
273
296
|
| `RAISE_ON_BULK_ERROR` | Controls whether bulk insert operations raise exceptions on errors. If set to `true`, the operation will stop and raise an exception when an error occurs. If set to `false`, errors will be logged, and the operation will continue. **Note:** STAC Item and ItemCollection validation errors will always raise, regardless of this flag. | `false` | Optional |
|
|
274
297
|
| `DATABASE_REFRESH` | Controls whether database operations refresh the index immediately after changes. If set to `true`, changes will be immediately searchable. If set to `false`, changes may not be immediately visible but can improve performance for bulk operations. If set to `wait_for`, changes will wait for the next refresh cycle to become visible. | `false` | Optional |
|
|
298
|
+
| `ENABLE_COLLECTIONS_SEARCH` | Enable collection search extensions (sort, fields). | `true` | Optional |
|
|
275
299
|
| `ENABLE_TRANSACTIONS_EXTENSIONS` | Enables or disables the Transactions and Bulk Transactions API extensions. If set to `false`, the POST `/collections` route and related transaction endpoints (including bulk transaction operations) will be unavailable in the API. This is useful for deployments where mutating the catalog via the API should be prevented. | `true` | Optional |
|
|
276
300
|
| `STAC_ITEM_LIMIT` | Sets the environment variable for result limiting to SFEOS for the number of returned items and STAC collections. | `10` | Optional |
|
|
301
|
+
| `STAC_INDEX_ASSETS` | Controls if Assets are indexed when added to Elasticsearch/Opensearch. This allows asset fields to be included in search queries. | `false` | Optional |
|
|
302
|
+
| `ENV_MAX_LIMIT` | Configures the environment variable in SFEOS to override the default `MAX_LIMIT`, which controls the limit parameter for returned items and STAC collections. | `10,000` | Optional |
|
|
303
|
+
| `USE_DATETIME` | Configures the datetime search behavior in SFEOS. When enabled, searches both datetime field and falls back to start_datetime/end_datetime range for items with null datetime. When disabled, searches only by start_datetime/end_datetime range. | True | Optional |
|
|
277
304
|
|
|
278
305
|
> [!NOTE]
|
|
279
306
|
> The variables `ES_HOST`, `ES_PORT`, `ES_USE_SSL`, `ES_VERIFY_CERTS` and `ES_TIMEOUT` apply to both Elasticsearch and OpenSearch backends, so there is no need to rename the key names to `OS_` even if you're using OpenSearch.
|
|
@@ -415,6 +442,10 @@ The system uses a precise naming convention:
|
|
|
415
442
|
- **Root Path Configuration**: The application root path is the base URL by default.
|
|
416
443
|
- For AWS Lambda with Gateway API: Set `STAC_FASTAPI_ROOT_PATH` to match the Gateway API stage name (e.g., `/v1`)
|
|
417
444
|
|
|
445
|
+
- **Feature Configuration**: Control which features are enabled:
|
|
446
|
+
- `ENABLE_COLLECTIONS_SEARCH`: Set to `true` (default) to enable collection search extensions (sort, fields). Set to `false` to disable.
|
|
447
|
+
- `ENABLE_TRANSACTIONS_EXTENSIONS`: Set to `true` (default) to enable transaction extensions. Set to `false` to disable.
|
|
448
|
+
|
|
418
449
|
|
|
419
450
|
## Collection Pagination
|
|
420
451
|
|
|
File without changes
|
{stac_fastapi_opensearch-6.2.1 → stac_fastapi_opensearch-6.4.0}/stac_fastapi/opensearch/__init__.py
RENAMED
|
File without changes
|
{stac_fastapi_opensearch-6.2.1 → stac_fastapi_opensearch-6.4.0}/stac_fastapi/opensearch/config.py
RENAMED
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|