stac-fastapi-elasticsearch 6.2.1__tar.gz → 6.4.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {stac_fastapi_elasticsearch-6.2.1 → stac_fastapi_elasticsearch-6.4.0}/PKG-INFO +80 -48
- {stac_fastapi_elasticsearch-6.2.1 → stac_fastapi_elasticsearch-6.4.0}/README.md +78 -22
- {stac_fastapi_elasticsearch-6.2.1 → stac_fastapi_elasticsearch-6.4.0}/setup.py +2 -2
- {stac_fastapi_elasticsearch-6.2.1 → stac_fastapi_elasticsearch-6.4.0}/stac_fastapi/elasticsearch/app.py +61 -13
- {stac_fastapi_elasticsearch-6.2.1 → stac_fastapi_elasticsearch-6.4.0}/stac_fastapi/elasticsearch/database_logic.py +174 -55
- {stac_fastapi_elasticsearch-6.2.1 → stac_fastapi_elasticsearch-6.4.0}/stac_fastapi/elasticsearch/version.py +1 -1
- {stac_fastapi_elasticsearch-6.2.1 → stac_fastapi_elasticsearch-6.4.0}/stac_fastapi_elasticsearch.egg-info/PKG-INFO +81 -49
- {stac_fastapi_elasticsearch-6.2.1 → stac_fastapi_elasticsearch-6.4.0}/stac_fastapi_elasticsearch.egg-info/requires.txt +2 -2
- {stac_fastapi_elasticsearch-6.2.1 → stac_fastapi_elasticsearch-6.4.0}/stac_fastapi_elasticsearch.egg-info/top_level.txt +0 -1
- {stac_fastapi_elasticsearch-6.2.1 → stac_fastapi_elasticsearch-6.4.0}/setup.cfg +0 -0
- {stac_fastapi_elasticsearch-6.2.1 → stac_fastapi_elasticsearch-6.4.0}/stac_fastapi/elasticsearch/__init__.py +0 -0
- {stac_fastapi_elasticsearch-6.2.1 → stac_fastapi_elasticsearch-6.4.0}/stac_fastapi/elasticsearch/config.py +0 -0
- {stac_fastapi_elasticsearch-6.2.1 → stac_fastapi_elasticsearch-6.4.0}/stac_fastapi_elasticsearch.egg-info/SOURCES.txt +0 -0
- {stac_fastapi_elasticsearch-6.2.1 → stac_fastapi_elasticsearch-6.4.0}/stac_fastapi_elasticsearch.egg-info/dependency_links.txt +0 -0
- {stac_fastapi_elasticsearch-6.2.1 → stac_fastapi_elasticsearch-6.4.0}/stac_fastapi_elasticsearch.egg-info/entry_points.txt +0 -0
- {stac_fastapi_elasticsearch-6.2.1 → stac_fastapi_elasticsearch-6.4.0}/stac_fastapi_elasticsearch.egg-info/not-zip-safe +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
|
-
Metadata-Version: 2.
|
|
1
|
+
Metadata-Version: 2.1
|
|
2
2
|
Name: stac_fastapi_elasticsearch
|
|
3
|
-
Version: 6.
|
|
3
|
+
Version: 6.4.0
|
|
4
4
|
Summary: An implementation of STAC API based on the FastAPI framework with both Elasticsearch and Opensearch.
|
|
5
5
|
Home-page: https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch
|
|
6
6
|
License: MIT
|
|
@@ -15,33 +15,9 @@ Classifier: Programming Language :: Python :: 3.13
|
|
|
15
15
|
Classifier: License :: OSI Approved :: MIT License
|
|
16
16
|
Requires-Python: >=3.9
|
|
17
17
|
Description-Content-Type: text/markdown
|
|
18
|
-
Requires-Dist: stac-fastapi-core==6.2.1
|
|
19
|
-
Requires-Dist: sfeos-helpers==6.2.1
|
|
20
|
-
Requires-Dist: elasticsearch[async]~=8.18.0
|
|
21
|
-
Requires-Dist: uvicorn~=0.23.0
|
|
22
|
-
Requires-Dist: starlette<0.36.0,>=0.35.0
|
|
23
18
|
Provides-Extra: dev
|
|
24
|
-
Requires-Dist: pytest~=7.0.0; extra == "dev"
|
|
25
|
-
Requires-Dist: pytest-cov~=4.0.0; extra == "dev"
|
|
26
|
-
Requires-Dist: pytest-asyncio~=0.21.0; extra == "dev"
|
|
27
|
-
Requires-Dist: pre-commit~=3.0.0; extra == "dev"
|
|
28
|
-
Requires-Dist: ciso8601~=2.3.0; extra == "dev"
|
|
29
|
-
Requires-Dist: httpx<0.28.0,>=0.24.0; extra == "dev"
|
|
30
19
|
Provides-Extra: docs
|
|
31
|
-
Requires-Dist: mkdocs~=1.4.0; extra == "docs"
|
|
32
|
-
Requires-Dist: mkdocs-material~=9.0.0; extra == "docs"
|
|
33
|
-
Requires-Dist: pdocs~=1.2.0; extra == "docs"
|
|
34
20
|
Provides-Extra: server
|
|
35
|
-
Requires-Dist: uvicorn[standard]~=0.23.0; extra == "server"
|
|
36
|
-
Dynamic: classifier
|
|
37
|
-
Dynamic: description
|
|
38
|
-
Dynamic: description-content-type
|
|
39
|
-
Dynamic: home-page
|
|
40
|
-
Dynamic: license
|
|
41
|
-
Dynamic: provides-extra
|
|
42
|
-
Dynamic: requires-dist
|
|
43
|
-
Dynamic: requires-python
|
|
44
|
-
Dynamic: summary
|
|
45
21
|
|
|
46
22
|
# stac-fastapi-elasticsearch-opensearch
|
|
47
23
|
|
|
@@ -81,11 +57,10 @@ SFEOS (stac-fastapi-elasticsearch-opensearch) is a high-performance, scalable AP
|
|
|
81
57
|
- **Scale to millions of geospatial assets** with fast search performance through optimized spatial indexing and query capabilities
|
|
82
58
|
- **Support OGC-compliant filtering** including spatial operations (intersects, contains, etc.) and temporal queries
|
|
83
59
|
- **Perform geospatial aggregations** to analyze data distribution across space and time
|
|
60
|
+
- **Enhanced collection search capabilities** with support for sorting and field selection
|
|
84
61
|
|
|
85
62
|
This implementation builds on the STAC-FastAPI framework, providing a production-ready solution specifically optimized for Elasticsearch and OpenSearch databases. It's ideal for organizations managing large geospatial data catalogs who need efficient discovery and access capabilities through standardized APIs.
|
|
86
63
|
|
|
87
|
-
|
|
88
|
-
|
|
89
64
|
## Common Deployment Patterns
|
|
90
65
|
|
|
91
66
|
stac-fastapi-elasticsearch-opensearch can be deployed in several ways depending on your needs:
|
|
@@ -111,26 +86,44 @@ This project is built on the following technologies: STAC, stac-fastapi, FastAPI
|
|
|
111
86
|
|
|
112
87
|
## Table of Contents
|
|
113
88
|
|
|
114
|
-
- [
|
|
115
|
-
- [
|
|
116
|
-
- [
|
|
117
|
-
- [
|
|
118
|
-
- [
|
|
119
|
-
- [
|
|
120
|
-
- [
|
|
121
|
-
- [
|
|
122
|
-
- [
|
|
123
|
-
- [
|
|
124
|
-
- [
|
|
125
|
-
- [
|
|
126
|
-
- [
|
|
127
|
-
- [
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
- [
|
|
131
|
-
- [
|
|
132
|
-
- [
|
|
133
|
-
- [
|
|
89
|
+
- [stac-fastapi-elasticsearch-opensearch](#stac-fastapi-elasticsearch-opensearch)
|
|
90
|
+
- [Sponsors \& Supporters](#sponsors--supporters)
|
|
91
|
+
- [Project Introduction - What is SFEOS?](#project-introduction---what-is-sfeos)
|
|
92
|
+
- [Common Deployment Patterns](#common-deployment-patterns)
|
|
93
|
+
- [Technologies](#technologies)
|
|
94
|
+
- [Table of Contents](#table-of-contents)
|
|
95
|
+
- [Collection Search Extensions](#collection-search-extensions)
|
|
96
|
+
- [Documentation \& Resources](#documentation--resources)
|
|
97
|
+
- [Package Structure](#package-structure)
|
|
98
|
+
- [Examples](#examples)
|
|
99
|
+
- [Performance](#performance)
|
|
100
|
+
- [Direct Response Mode](#direct-response-mode)
|
|
101
|
+
- [Quick Start](#quick-start)
|
|
102
|
+
- [Installation](#installation)
|
|
103
|
+
- [Running Locally](#running-locally)
|
|
104
|
+
- [Using Pre-built Docker Images](#using-pre-built-docker-images)
|
|
105
|
+
- [Using Docker Compose](#using-docker-compose)
|
|
106
|
+
- [Configuration Reference](#configuration-reference)
|
|
107
|
+
- [Datetime-Based Index Management](#datetime-based-index-management)
|
|
108
|
+
- [Overview](#overview)
|
|
109
|
+
- [When to Use](#when-to-use)
|
|
110
|
+
- [Configuration](#configuration)
|
|
111
|
+
- [Enabling Datetime-Based Indexing](#enabling-datetime-based-indexing)
|
|
112
|
+
- [Related Configuration Variables](#related-configuration-variables)
|
|
113
|
+
- [How Datetime-Based Indexing Works](#how-datetime-based-indexing-works)
|
|
114
|
+
- [Index and Alias Naming Convention](#index-and-alias-naming-convention)
|
|
115
|
+
- [Index Size Management](#index-size-management)
|
|
116
|
+
- [Interacting with the API](#interacting-with-the-api)
|
|
117
|
+
- [Configure the API](#configure-the-api)
|
|
118
|
+
- [Collection Pagination](#collection-pagination)
|
|
119
|
+
- [Ingesting Sample Data CLI Tool](#ingesting-sample-data-cli-tool)
|
|
120
|
+
- [Elasticsearch Mappings](#elasticsearch-mappings)
|
|
121
|
+
- [Managing Elasticsearch Indices](#managing-elasticsearch-indices)
|
|
122
|
+
- [Snapshots](#snapshots)
|
|
123
|
+
- [Reindexing](#reindexing)
|
|
124
|
+
- [Auth](#auth)
|
|
125
|
+
- [Aggregation](#aggregation)
|
|
126
|
+
- [Rate Limiting](#rate-limiting)
|
|
134
127
|
|
|
135
128
|
## Documentation & Resources
|
|
136
129
|
|
|
@@ -141,6 +134,37 @@ This project is built on the following technologies: STAC, stac-fastapi, FastAPI
|
|
|
141
134
|
- [Gitter Chat](https://app.gitter.im/#/room/#stac-fastapi-elasticsearch_community:gitter.im) - For real-time discussions
|
|
142
135
|
- [GitHub Discussions](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/discussions) - For longer-form questions and answers
|
|
143
136
|
|
|
137
|
+
## Collection Search Extensions
|
|
138
|
+
|
|
139
|
+
SFEOS implements extended capabilities for the `/collections` endpoint, allowing for more powerful collection discovery:
|
|
140
|
+
|
|
141
|
+
- **Sorting**: Sort collections by sortable fields using the `sortby` parameter
|
|
142
|
+
- Example: `/collections?sortby=+id` (ascending sort by ID)
|
|
143
|
+
- Example: `/collections?sortby=-id` (descending sort by ID)
|
|
144
|
+
- Example: `/collections?sortby=-temporal` (descending sort by temporal extent)
|
|
145
|
+
|
|
146
|
+
- **Field Selection**: Request only specific fields to be returned using the `fields` parameter
|
|
147
|
+
- Example: `/collections?fields=id,title,description`
|
|
148
|
+
- This helps reduce payload size when only certain fields are needed
|
|
149
|
+
|
|
150
|
+
- **Free Text Search**: Search across collection text fields using the `q` parameter
|
|
151
|
+
- Example: `/collections?q=landsat`
|
|
152
|
+
- Searches across multiple text fields including title, description, and keywords
|
|
153
|
+
- Supports partial word matching and relevance-based sorting
|
|
154
|
+
|
|
155
|
+
These extensions make it easier to build user interfaces that display and navigate through collections efficiently.
|
|
156
|
+
|
|
157
|
+
> **Configuration**: Collection search extensions can be disabled by setting the `ENABLE_COLLECTIONS_SEARCH` environment variable to `false`. By default, these extensions are enabled.
|
|
158
|
+
|
|
159
|
+
> **Note**: Sorting is only available on fields that are indexed for sorting in Elasticsearch/OpenSearch. With the default mappings, you can sort on:
|
|
160
|
+
> - `id` (keyword field)
|
|
161
|
+
> - `extent.temporal.interval` (date field)
|
|
162
|
+
> - `temporal` (alias to extent.temporal.interval)
|
|
163
|
+
>
|
|
164
|
+
> Text fields like `title` and `description` are not sortable by default as they use text analysis for better search capabilities. Attempting to sort on these fields will result in a user-friendly error message explaining which fields are sortable and how to make additional fields sortable by updating the mappings.
|
|
165
|
+
>
|
|
166
|
+
> **Important**: Adding keyword fields to make text fields sortable can significantly increase the index size, especially for large text fields. Consider the storage implications when deciding which fields to make sortable.
|
|
167
|
+
|
|
144
168
|
## Package Structure
|
|
145
169
|
|
|
146
170
|
This project is organized into several packages, each with a specific purpose:
|
|
@@ -271,8 +295,12 @@ You can customize additional settings in your `.env` file:
|
|
|
271
295
|
| `ENABLE_DIRECT_RESPONSE` | Enable direct response for maximum performance (disables all FastAPI dependencies, including authentication, custom status codes, and validation) | `false` | Optional |
|
|
272
296
|
| `RAISE_ON_BULK_ERROR` | Controls whether bulk insert operations raise exceptions on errors. If set to `true`, the operation will stop and raise an exception when an error occurs. If set to `false`, errors will be logged, and the operation will continue. **Note:** STAC Item and ItemCollection validation errors will always raise, regardless of this flag. | `false` | Optional |
|
|
273
297
|
| `DATABASE_REFRESH` | Controls whether database operations refresh the index immediately after changes. If set to `true`, changes will be immediately searchable. If set to `false`, changes may not be immediately visible but can improve performance for bulk operations. If set to `wait_for`, changes will wait for the next refresh cycle to become visible. | `false` | Optional |
|
|
298
|
+
| `ENABLE_COLLECTIONS_SEARCH` | Enable collection search extensions (sort, fields). | `true` | Optional |
|
|
274
299
|
| `ENABLE_TRANSACTIONS_EXTENSIONS` | Enables or disables the Transactions and Bulk Transactions API extensions. If set to `false`, the POST `/collections` route and related transaction endpoints (including bulk transaction operations) will be unavailable in the API. This is useful for deployments where mutating the catalog via the API should be prevented. | `true` | Optional |
|
|
275
300
|
| `STAC_ITEM_LIMIT` | Sets the environment variable for result limiting to SFEOS for the number of returned items and STAC collections. | `10` | Optional |
|
|
301
|
+
| `STAC_INDEX_ASSETS` | Controls if Assets are indexed when added to Elasticsearch/Opensearch. This allows asset fields to be included in search queries. | `false` | Optional |
|
|
302
|
+
| `ENV_MAX_LIMIT` | Configures the environment variable in SFEOS to override the default `MAX_LIMIT`, which controls the limit parameter for returned items and STAC collections. | `10,000` | Optional |
|
|
303
|
+
| `USE_DATETIME` | Configures the datetime search behavior in SFEOS. When enabled, searches both datetime field and falls back to start_datetime/end_datetime range for items with null datetime. When disabled, searches only by start_datetime/end_datetime range. | True | Optional |
|
|
276
304
|
|
|
277
305
|
> [!NOTE]
|
|
278
306
|
> The variables `ES_HOST`, `ES_PORT`, `ES_USE_SSL`, `ES_VERIFY_CERTS` and `ES_TIMEOUT` apply to both Elasticsearch and OpenSearch backends, so there is no need to rename the key names to `OS_` even if you're using OpenSearch.
|
|
@@ -414,6 +442,10 @@ The system uses a precise naming convention:
|
|
|
414
442
|
- **Root Path Configuration**: The application root path is the base URL by default.
|
|
415
443
|
- For AWS Lambda with Gateway API: Set `STAC_FASTAPI_ROOT_PATH` to match the Gateway API stage name (e.g., `/v1`)
|
|
416
444
|
|
|
445
|
+
- **Feature Configuration**: Control which features are enabled:
|
|
446
|
+
- `ENABLE_COLLECTIONS_SEARCH`: Set to `true` (default) to enable collection search extensions (sort, fields). Set to `false` to disable.
|
|
447
|
+
- `ENABLE_TRANSACTIONS_EXTENSIONS`: Set to `true` (default) to enable transaction extensions. Set to `false` to disable.
|
|
448
|
+
|
|
417
449
|
|
|
418
450
|
## Collection Pagination
|
|
419
451
|
|
|
@@ -36,11 +36,10 @@ SFEOS (stac-fastapi-elasticsearch-opensearch) is a high-performance, scalable AP
|
|
|
36
36
|
- **Scale to millions of geospatial assets** with fast search performance through optimized spatial indexing and query capabilities
|
|
37
37
|
- **Support OGC-compliant filtering** including spatial operations (intersects, contains, etc.) and temporal queries
|
|
38
38
|
- **Perform geospatial aggregations** to analyze data distribution across space and time
|
|
39
|
+
- **Enhanced collection search capabilities** with support for sorting and field selection
|
|
39
40
|
|
|
40
41
|
This implementation builds on the STAC-FastAPI framework, providing a production-ready solution specifically optimized for Elasticsearch and OpenSearch databases. It's ideal for organizations managing large geospatial data catalogs who need efficient discovery and access capabilities through standardized APIs.
|
|
41
42
|
|
|
42
|
-
|
|
43
|
-
|
|
44
43
|
## Common Deployment Patterns
|
|
45
44
|
|
|
46
45
|
stac-fastapi-elasticsearch-opensearch can be deployed in several ways depending on your needs:
|
|
@@ -66,26 +65,44 @@ This project is built on the following technologies: STAC, stac-fastapi, FastAPI
|
|
|
66
65
|
|
|
67
66
|
## Table of Contents
|
|
68
67
|
|
|
69
|
-
- [
|
|
70
|
-
- [
|
|
71
|
-
- [
|
|
72
|
-
- [
|
|
73
|
-
- [
|
|
74
|
-
- [
|
|
75
|
-
- [
|
|
76
|
-
- [
|
|
77
|
-
- [
|
|
78
|
-
- [
|
|
79
|
-
- [
|
|
80
|
-
- [
|
|
81
|
-
- [
|
|
82
|
-
- [
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
- [
|
|
86
|
-
- [
|
|
87
|
-
- [
|
|
88
|
-
- [
|
|
68
|
+
- [stac-fastapi-elasticsearch-opensearch](#stac-fastapi-elasticsearch-opensearch)
|
|
69
|
+
- [Sponsors \& Supporters](#sponsors--supporters)
|
|
70
|
+
- [Project Introduction - What is SFEOS?](#project-introduction---what-is-sfeos)
|
|
71
|
+
- [Common Deployment Patterns](#common-deployment-patterns)
|
|
72
|
+
- [Technologies](#technologies)
|
|
73
|
+
- [Table of Contents](#table-of-contents)
|
|
74
|
+
- [Collection Search Extensions](#collection-search-extensions)
|
|
75
|
+
- [Documentation \& Resources](#documentation--resources)
|
|
76
|
+
- [Package Structure](#package-structure)
|
|
77
|
+
- [Examples](#examples)
|
|
78
|
+
- [Performance](#performance)
|
|
79
|
+
- [Direct Response Mode](#direct-response-mode)
|
|
80
|
+
- [Quick Start](#quick-start)
|
|
81
|
+
- [Installation](#installation)
|
|
82
|
+
- [Running Locally](#running-locally)
|
|
83
|
+
- [Using Pre-built Docker Images](#using-pre-built-docker-images)
|
|
84
|
+
- [Using Docker Compose](#using-docker-compose)
|
|
85
|
+
- [Configuration Reference](#configuration-reference)
|
|
86
|
+
- [Datetime-Based Index Management](#datetime-based-index-management)
|
|
87
|
+
- [Overview](#overview)
|
|
88
|
+
- [When to Use](#when-to-use)
|
|
89
|
+
- [Configuration](#configuration)
|
|
90
|
+
- [Enabling Datetime-Based Indexing](#enabling-datetime-based-indexing)
|
|
91
|
+
- [Related Configuration Variables](#related-configuration-variables)
|
|
92
|
+
- [How Datetime-Based Indexing Works](#how-datetime-based-indexing-works)
|
|
93
|
+
- [Index and Alias Naming Convention](#index-and-alias-naming-convention)
|
|
94
|
+
- [Index Size Management](#index-size-management)
|
|
95
|
+
- [Interacting with the API](#interacting-with-the-api)
|
|
96
|
+
- [Configure the API](#configure-the-api)
|
|
97
|
+
- [Collection Pagination](#collection-pagination)
|
|
98
|
+
- [Ingesting Sample Data CLI Tool](#ingesting-sample-data-cli-tool)
|
|
99
|
+
- [Elasticsearch Mappings](#elasticsearch-mappings)
|
|
100
|
+
- [Managing Elasticsearch Indices](#managing-elasticsearch-indices)
|
|
101
|
+
- [Snapshots](#snapshots)
|
|
102
|
+
- [Reindexing](#reindexing)
|
|
103
|
+
- [Auth](#auth)
|
|
104
|
+
- [Aggregation](#aggregation)
|
|
105
|
+
- [Rate Limiting](#rate-limiting)
|
|
89
106
|
|
|
90
107
|
## Documentation & Resources
|
|
91
108
|
|
|
@@ -96,6 +113,37 @@ This project is built on the following technologies: STAC, stac-fastapi, FastAPI
|
|
|
96
113
|
- [Gitter Chat](https://app.gitter.im/#/room/#stac-fastapi-elasticsearch_community:gitter.im) - For real-time discussions
|
|
97
114
|
- [GitHub Discussions](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/discussions) - For longer-form questions and answers
|
|
98
115
|
|
|
116
|
+
## Collection Search Extensions
|
|
117
|
+
|
|
118
|
+
SFEOS implements extended capabilities for the `/collections` endpoint, allowing for more powerful collection discovery:
|
|
119
|
+
|
|
120
|
+
- **Sorting**: Sort collections by sortable fields using the `sortby` parameter
|
|
121
|
+
- Example: `/collections?sortby=+id` (ascending sort by ID)
|
|
122
|
+
- Example: `/collections?sortby=-id` (descending sort by ID)
|
|
123
|
+
- Example: `/collections?sortby=-temporal` (descending sort by temporal extent)
|
|
124
|
+
|
|
125
|
+
- **Field Selection**: Request only specific fields to be returned using the `fields` parameter
|
|
126
|
+
- Example: `/collections?fields=id,title,description`
|
|
127
|
+
- This helps reduce payload size when only certain fields are needed
|
|
128
|
+
|
|
129
|
+
- **Free Text Search**: Search across collection text fields using the `q` parameter
|
|
130
|
+
- Example: `/collections?q=landsat`
|
|
131
|
+
- Searches across multiple text fields including title, description, and keywords
|
|
132
|
+
- Supports partial word matching and relevance-based sorting
|
|
133
|
+
|
|
134
|
+
These extensions make it easier to build user interfaces that display and navigate through collections efficiently.
|
|
135
|
+
|
|
136
|
+
> **Configuration**: Collection search extensions can be disabled by setting the `ENABLE_COLLECTIONS_SEARCH` environment variable to `false`. By default, these extensions are enabled.
|
|
137
|
+
|
|
138
|
+
> **Note**: Sorting is only available on fields that are indexed for sorting in Elasticsearch/OpenSearch. With the default mappings, you can sort on:
|
|
139
|
+
> - `id` (keyword field)
|
|
140
|
+
> - `extent.temporal.interval` (date field)
|
|
141
|
+
> - `temporal` (alias to extent.temporal.interval)
|
|
142
|
+
>
|
|
143
|
+
> Text fields like `title` and `description` are not sortable by default as they use text analysis for better search capabilities. Attempting to sort on these fields will result in a user-friendly error message explaining which fields are sortable and how to make additional fields sortable by updating the mappings.
|
|
144
|
+
>
|
|
145
|
+
> **Important**: Adding keyword fields to make text fields sortable can significantly increase the index size, especially for large text fields. Consider the storage implications when deciding which fields to make sortable.
|
|
146
|
+
|
|
99
147
|
## Package Structure
|
|
100
148
|
|
|
101
149
|
This project is organized into several packages, each with a specific purpose:
|
|
@@ -226,8 +274,12 @@ You can customize additional settings in your `.env` file:
|
|
|
226
274
|
| `ENABLE_DIRECT_RESPONSE` | Enable direct response for maximum performance (disables all FastAPI dependencies, including authentication, custom status codes, and validation) | `false` | Optional |
|
|
227
275
|
| `RAISE_ON_BULK_ERROR` | Controls whether bulk insert operations raise exceptions on errors. If set to `true`, the operation will stop and raise an exception when an error occurs. If set to `false`, errors will be logged, and the operation will continue. **Note:** STAC Item and ItemCollection validation errors will always raise, regardless of this flag. | `false` | Optional |
|
|
228
276
|
| `DATABASE_REFRESH` | Controls whether database operations refresh the index immediately after changes. If set to `true`, changes will be immediately searchable. If set to `false`, changes may not be immediately visible but can improve performance for bulk operations. If set to `wait_for`, changes will wait for the next refresh cycle to become visible. | `false` | Optional |
|
|
277
|
+
| `ENABLE_COLLECTIONS_SEARCH` | Enable collection search extensions (sort, fields). | `true` | Optional |
|
|
229
278
|
| `ENABLE_TRANSACTIONS_EXTENSIONS` | Enables or disables the Transactions and Bulk Transactions API extensions. If set to `false`, the POST `/collections` route and related transaction endpoints (including bulk transaction operations) will be unavailable in the API. This is useful for deployments where mutating the catalog via the API should be prevented. | `true` | Optional |
|
|
230
279
|
| `STAC_ITEM_LIMIT` | Sets the environment variable for result limiting to SFEOS for the number of returned items and STAC collections. | `10` | Optional |
|
|
280
|
+
| `STAC_INDEX_ASSETS` | Controls if Assets are indexed when added to Elasticsearch/Opensearch. This allows asset fields to be included in search queries. | `false` | Optional |
|
|
281
|
+
| `ENV_MAX_LIMIT` | Configures the environment variable in SFEOS to override the default `MAX_LIMIT`, which controls the limit parameter for returned items and STAC collections. | `10,000` | Optional |
|
|
282
|
+
| `USE_DATETIME` | Configures the datetime search behavior in SFEOS. When enabled, searches both datetime field and falls back to start_datetime/end_datetime range for items with null datetime. When disabled, searches only by start_datetime/end_datetime range. | True | Optional |
|
|
231
283
|
|
|
232
284
|
> [!NOTE]
|
|
233
285
|
> The variables `ES_HOST`, `ES_PORT`, `ES_USE_SSL`, `ES_VERIFY_CERTS` and `ES_TIMEOUT` apply to both Elasticsearch and OpenSearch backends, so there is no need to rename the key names to `OS_` even if you're using OpenSearch.
|
|
@@ -369,6 +421,10 @@ The system uses a precise naming convention:
|
|
|
369
421
|
- **Root Path Configuration**: The application root path is the base URL by default.
|
|
370
422
|
- For AWS Lambda with Gateway API: Set `STAC_FASTAPI_ROOT_PATH` to match the Gateway API stage name (e.g., `/v1`)
|
|
371
423
|
|
|
424
|
+
- **Feature Configuration**: Control which features are enabled:
|
|
425
|
+
- `ENABLE_COLLECTIONS_SEARCH`: Set to `true` (default) to enable collection search extensions (sort, fields). Set to `false` to disable.
|
|
426
|
+
- `ENABLE_TRANSACTIONS_EXTENSIONS`: Set to `true` (default) to enable transaction extensions. Set to `false` to disable.
|
|
427
|
+
|
|
372
428
|
|
|
373
429
|
## Collection Pagination
|
|
374
430
|
|
|
@@ -6,8 +6,8 @@ with open("README.md") as f:
|
|
|
6
6
|
desc = f.read()
|
|
7
7
|
|
|
8
8
|
install_requires = [
|
|
9
|
-
"stac-fastapi-core==6.
|
|
10
|
-
"sfeos-helpers==6.
|
|
9
|
+
"stac-fastapi-core==6.4.0",
|
|
10
|
+
"sfeos-helpers==6.4.0",
|
|
11
11
|
"elasticsearch[async]~=8.18.0",
|
|
12
12
|
"uvicorn~=0.23.0",
|
|
13
13
|
"starlette>=0.35.0,<0.36.0",
|
|
@@ -7,7 +7,12 @@ from contextlib import asynccontextmanager
|
|
|
7
7
|
from fastapi import FastAPI
|
|
8
8
|
|
|
9
9
|
from stac_fastapi.api.app import StacApi
|
|
10
|
-
from stac_fastapi.api.models import
|
|
10
|
+
from stac_fastapi.api.models import (
|
|
11
|
+
ItemCollectionUri,
|
|
12
|
+
create_get_request_model,
|
|
13
|
+
create_post_request_model,
|
|
14
|
+
create_request_model,
|
|
15
|
+
)
|
|
11
16
|
from stac_fastapi.core.core import (
|
|
12
17
|
BulkTransactionsClient,
|
|
13
18
|
CoreClient,
|
|
@@ -29,7 +34,7 @@ from stac_fastapi.elasticsearch.database_logic import (
|
|
|
29
34
|
create_collection_index,
|
|
30
35
|
create_index_templates,
|
|
31
36
|
)
|
|
32
|
-
from stac_fastapi.extensions.core import (
|
|
37
|
+
from stac_fastapi.extensions.core import ( # CollectionSearchFilterExtension,
|
|
33
38
|
AggregationExtension,
|
|
34
39
|
CollectionSearchExtension,
|
|
35
40
|
FilterExtension,
|
|
@@ -38,7 +43,11 @@ from stac_fastapi.extensions.core import (
|
|
|
38
43
|
TokenPaginationExtension,
|
|
39
44
|
TransactionExtension,
|
|
40
45
|
)
|
|
46
|
+
from stac_fastapi.extensions.core.fields import FieldsConformanceClasses
|
|
41
47
|
from stac_fastapi.extensions.core.filter import FilterConformanceClasses
|
|
48
|
+
from stac_fastapi.extensions.core.free_text import FreeTextConformanceClasses
|
|
49
|
+
from stac_fastapi.extensions.core.query import QueryConformanceClasses
|
|
50
|
+
from stac_fastapi.extensions.core.sort import SortConformanceClasses
|
|
42
51
|
from stac_fastapi.extensions.third_party import BulkTransactionExtension
|
|
43
52
|
from stac_fastapi.sfeos_helpers.aggregation import EsAsyncBaseAggregationClient
|
|
44
53
|
from stac_fastapi.sfeos_helpers.filter import EsAsyncBaseFiltersClient
|
|
@@ -47,13 +56,16 @@ logging.basicConfig(level=logging.INFO)
|
|
|
47
56
|
logger = logging.getLogger(__name__)
|
|
48
57
|
|
|
49
58
|
TRANSACTIONS_EXTENSIONS = get_bool_env("ENABLE_TRANSACTIONS_EXTENSIONS", default=True)
|
|
59
|
+
ENABLE_COLLECTIONS_SEARCH = get_bool_env("ENABLE_COLLECTIONS_SEARCH", default=True)
|
|
50
60
|
logger.info("TRANSACTIONS_EXTENSIONS is set to %s", TRANSACTIONS_EXTENSIONS)
|
|
61
|
+
logger.info("ENABLE_COLLECTIONS_SEARCH is set to %s", ENABLE_COLLECTIONS_SEARCH)
|
|
51
62
|
|
|
52
63
|
settings = ElasticsearchSettings()
|
|
53
64
|
session = Session.create_from_settings(settings)
|
|
54
65
|
|
|
55
66
|
database_logic = DatabaseLogic()
|
|
56
67
|
|
|
68
|
+
|
|
57
69
|
filter_extension = FilterExtension(
|
|
58
70
|
client=EsAsyncBaseFiltersClient(database=database_logic)
|
|
59
71
|
)
|
|
@@ -61,14 +73,6 @@ filter_extension.conformance_classes.append(
|
|
|
61
73
|
FilterConformanceClasses.ADVANCED_COMPARISON_OPERATORS
|
|
62
74
|
)
|
|
63
75
|
|
|
64
|
-
# Adding collection search extension for compatibility with stac-auth-proxy
|
|
65
|
-
# (https://github.com/developmentseed/stac-auth-proxy)
|
|
66
|
-
# The extension is not fully implemented yet but is required for collection filtering support
|
|
67
|
-
collection_search_extension = CollectionSearchExtension()
|
|
68
|
-
collection_search_extension.conformance_classes.append(
|
|
69
|
-
"https://api.stacspec.org/v1.0.0-rc.1/collection-search#filter"
|
|
70
|
-
)
|
|
71
|
-
|
|
72
76
|
aggregation_extension = AggregationExtension(
|
|
73
77
|
client=EsAsyncBaseAggregationClient(
|
|
74
78
|
database=database_logic, session=session, settings=settings
|
|
@@ -77,14 +81,16 @@ aggregation_extension = AggregationExtension(
|
|
|
77
81
|
aggregation_extension.POST = EsAggregationExtensionPostRequest
|
|
78
82
|
aggregation_extension.GET = EsAggregationExtensionGetRequest
|
|
79
83
|
|
|
84
|
+
fields_extension = FieldsExtension()
|
|
85
|
+
fields_extension.conformance_classes.append(FieldsConformanceClasses.ITEMS)
|
|
86
|
+
|
|
80
87
|
search_extensions = [
|
|
81
|
-
|
|
88
|
+
fields_extension,
|
|
82
89
|
QueryExtension(),
|
|
83
90
|
SortExtension(),
|
|
84
91
|
TokenPaginationExtension(),
|
|
85
92
|
filter_extension,
|
|
86
93
|
FreeTextExtension(),
|
|
87
|
-
collection_search_extension,
|
|
88
94
|
]
|
|
89
95
|
|
|
90
96
|
if TRANSACTIONS_EXTENSIONS:
|
|
@@ -110,14 +116,51 @@ if TRANSACTIONS_EXTENSIONS:
|
|
|
110
116
|
|
|
111
117
|
extensions = [aggregation_extension] + search_extensions
|
|
112
118
|
|
|
119
|
+
# Create collection search extensions if enabled
|
|
120
|
+
if ENABLE_COLLECTIONS_SEARCH:
|
|
121
|
+
# Create collection search extensions
|
|
122
|
+
collection_search_extensions = [
|
|
123
|
+
# QueryExtension(conformance_classes=[QueryConformanceClasses.COLLECTIONS]),
|
|
124
|
+
SortExtension(conformance_classes=[SortConformanceClasses.COLLECTIONS]),
|
|
125
|
+
FieldsExtension(conformance_classes=[FieldsConformanceClasses.COLLECTIONS]),
|
|
126
|
+
# CollectionSearchFilterExtension(
|
|
127
|
+
# conformance_classes=[FilterConformanceClasses.COLLECTIONS]
|
|
128
|
+
# ),
|
|
129
|
+
FreeTextExtension(conformance_classes=[FreeTextConformanceClasses.COLLECTIONS]),
|
|
130
|
+
]
|
|
131
|
+
|
|
132
|
+
# Initialize collection search with its extensions
|
|
133
|
+
collection_search_ext = CollectionSearchExtension.from_extensions(
|
|
134
|
+
collection_search_extensions
|
|
135
|
+
)
|
|
136
|
+
collections_get_request_model = collection_search_ext.GET
|
|
137
|
+
|
|
138
|
+
extensions.append(collection_search_ext)
|
|
139
|
+
|
|
113
140
|
database_logic.extensions = [type(ext).__name__ for ext in extensions]
|
|
114
141
|
|
|
115
142
|
post_request_model = create_post_request_model(search_extensions)
|
|
116
143
|
|
|
144
|
+
items_get_request_model = create_request_model(
|
|
145
|
+
model_name="ItemCollectionUri",
|
|
146
|
+
base_model=ItemCollectionUri,
|
|
147
|
+
extensions=[
|
|
148
|
+
SortExtension(
|
|
149
|
+
conformance_classes=[SortConformanceClasses.ITEMS],
|
|
150
|
+
),
|
|
151
|
+
QueryExtension(
|
|
152
|
+
conformance_classes=[QueryConformanceClasses.ITEMS],
|
|
153
|
+
),
|
|
154
|
+
filter_extension,
|
|
155
|
+
FieldsExtension(conformance_classes=[FieldsConformanceClasses.ITEMS]),
|
|
156
|
+
],
|
|
157
|
+
request_type="GET",
|
|
158
|
+
)
|
|
159
|
+
|
|
117
160
|
app_config = {
|
|
118
161
|
"title": os.getenv("STAC_FASTAPI_TITLE", "stac-fastapi-elasticsearch"),
|
|
119
162
|
"description": os.getenv("STAC_FASTAPI_DESCRIPTION", "stac-fastapi-elasticsearch"),
|
|
120
|
-
"api_version": os.getenv("STAC_FASTAPI_VERSION", "6.
|
|
163
|
+
"api_version": os.getenv("STAC_FASTAPI_VERSION", "6.0.0"),
|
|
121
164
|
"settings": settings,
|
|
122
165
|
"extensions": extensions,
|
|
123
166
|
"client": CoreClient(
|
|
@@ -128,9 +171,14 @@ app_config = {
|
|
|
128
171
|
),
|
|
129
172
|
"search_get_request_model": create_get_request_model(search_extensions),
|
|
130
173
|
"search_post_request_model": post_request_model,
|
|
174
|
+
"items_get_request_model": items_get_request_model,
|
|
131
175
|
"route_dependencies": get_route_dependencies(),
|
|
132
176
|
}
|
|
133
177
|
|
|
178
|
+
# Add collections_get_request_model if collection search is enabled
|
|
179
|
+
if ENABLE_COLLECTIONS_SEARCH:
|
|
180
|
+
app_config["collections_get_request_model"] = collections_get_request_model
|
|
181
|
+
|
|
134
182
|
api = StacApi(**app_config)
|
|
135
183
|
|
|
136
184
|
|
|
@@ -17,7 +17,7 @@ from starlette.requests import Request
|
|
|
17
17
|
|
|
18
18
|
from stac_fastapi.core.base_database_logic import BaseDatabaseLogic
|
|
19
19
|
from stac_fastapi.core.serializers import CollectionSerializer, ItemSerializer
|
|
20
|
-
from stac_fastapi.core.utilities import
|
|
20
|
+
from stac_fastapi.core.utilities import bbox2polygon, get_bool_env, get_max_limit
|
|
21
21
|
from stac_fastapi.elasticsearch.config import AsyncElasticsearchSettings
|
|
22
22
|
from stac_fastapi.elasticsearch.config import (
|
|
23
23
|
ElasticsearchSettings as SyncElasticsearchSettings,
|
|
@@ -170,28 +170,96 @@ class DatabaseLogic(BaseDatabaseLogic):
|
|
|
170
170
|
"""CORE LOGIC"""
|
|
171
171
|
|
|
172
172
|
async def get_all_collections(
|
|
173
|
-
self,
|
|
173
|
+
self,
|
|
174
|
+
token: Optional[str],
|
|
175
|
+
limit: int,
|
|
176
|
+
request: Request,
|
|
177
|
+
sort: Optional[List[Dict[str, Any]]] = None,
|
|
178
|
+
q: Optional[List[str]] = None,
|
|
174
179
|
) -> Tuple[List[Dict[str, Any]], Optional[str]]:
|
|
175
|
-
"""Retrieve a list of
|
|
180
|
+
"""Retrieve a list of collections from Elasticsearch, supporting pagination.
|
|
176
181
|
|
|
177
182
|
Args:
|
|
178
183
|
token (Optional[str]): The pagination token.
|
|
179
184
|
limit (int): The number of results to return.
|
|
185
|
+
request (Request): The FastAPI request object.
|
|
186
|
+
sort (Optional[List[Dict[str, Any]]]): Optional sort parameter from the request.
|
|
187
|
+
q (Optional[List[str]]): Free text search terms.
|
|
180
188
|
|
|
181
189
|
Returns:
|
|
182
190
|
A tuple of (collections, next pagination token if any).
|
|
191
|
+
|
|
192
|
+
Raises:
|
|
193
|
+
HTTPException: If sorting is requested on a field that is not sortable.
|
|
183
194
|
"""
|
|
184
|
-
|
|
195
|
+
# Define sortable fields based on the ES_COLLECTIONS_MAPPINGS
|
|
196
|
+
sortable_fields = ["id", "extent.temporal.interval", "temporal"]
|
|
197
|
+
|
|
198
|
+
# Format the sort parameter
|
|
199
|
+
formatted_sort = []
|
|
200
|
+
if sort:
|
|
201
|
+
for item in sort:
|
|
202
|
+
field = item.get("field")
|
|
203
|
+
direction = item.get("direction", "asc")
|
|
204
|
+
if field:
|
|
205
|
+
# Validate that the field is sortable
|
|
206
|
+
if field not in sortable_fields:
|
|
207
|
+
raise HTTPException(
|
|
208
|
+
status_code=400,
|
|
209
|
+
detail=f"Field '{field}' is not sortable. Sortable fields are: {', '.join(sortable_fields)}. "
|
|
210
|
+
+ "Text fields are not sortable by default in Elasticsearch. "
|
|
211
|
+
+ "To make a field sortable, update the mapping to use 'keyword' type or add a '.keyword' subfield. ",
|
|
212
|
+
)
|
|
213
|
+
formatted_sort.append({field: {"order": direction}})
|
|
214
|
+
# Always include id as a secondary sort to ensure consistent pagination
|
|
215
|
+
if not any("id" in item for item in formatted_sort):
|
|
216
|
+
formatted_sort.append({"id": {"order": "asc"}})
|
|
217
|
+
else:
|
|
218
|
+
formatted_sort = [{"id": {"order": "asc"}}]
|
|
219
|
+
|
|
220
|
+
body = {
|
|
221
|
+
"sort": formatted_sort,
|
|
222
|
+
"size": limit,
|
|
223
|
+
}
|
|
224
|
+
|
|
185
225
|
if token:
|
|
186
|
-
search_after = [token]
|
|
226
|
+
body["search_after"] = [token]
|
|
227
|
+
|
|
228
|
+
# Apply free text query if provided
|
|
229
|
+
if q:
|
|
230
|
+
# For collections, we want to search across all relevant fields
|
|
231
|
+
should_clauses = []
|
|
232
|
+
|
|
233
|
+
# For each search term
|
|
234
|
+
for term in q:
|
|
235
|
+
# Create a multi_match query for each term
|
|
236
|
+
for field in [
|
|
237
|
+
"id",
|
|
238
|
+
"title",
|
|
239
|
+
"description",
|
|
240
|
+
"keywords",
|
|
241
|
+
"summaries.platform",
|
|
242
|
+
"summaries.constellation",
|
|
243
|
+
"providers.name",
|
|
244
|
+
"providers.url",
|
|
245
|
+
]:
|
|
246
|
+
should_clauses.append(
|
|
247
|
+
{
|
|
248
|
+
"wildcard": {
|
|
249
|
+
field: {"value": f"*{term}*", "case_insensitive": True}
|
|
250
|
+
}
|
|
251
|
+
}
|
|
252
|
+
)
|
|
187
253
|
|
|
254
|
+
# Add the query to the body using bool query with should clauses
|
|
255
|
+
body["query"] = {
|
|
256
|
+
"bool": {"should": should_clauses, "minimum_should_match": 1}
|
|
257
|
+
}
|
|
258
|
+
|
|
259
|
+
# Execute the search
|
|
188
260
|
response = await self.client.search(
|
|
189
261
|
index=COLLECTIONS_INDEX,
|
|
190
|
-
body=
|
|
191
|
-
"sort": [{"id": {"order": "asc"}}],
|
|
192
|
-
"size": limit,
|
|
193
|
-
**({"search_after": search_after} if search_after is not None else {}),
|
|
194
|
-
},
|
|
262
|
+
body=body,
|
|
195
263
|
)
|
|
196
264
|
|
|
197
265
|
hits = response["hits"]["hits"]
|
|
@@ -204,7 +272,9 @@ class DatabaseLogic(BaseDatabaseLogic):
|
|
|
204
272
|
|
|
205
273
|
next_token = None
|
|
206
274
|
if len(hits) == limit:
|
|
207
|
-
|
|
275
|
+
next_token_values = hits[-1].get("sort")
|
|
276
|
+
if next_token_values:
|
|
277
|
+
next_token = next_token_values[0]
|
|
208
278
|
|
|
209
279
|
return collections, next_token
|
|
210
280
|
|
|
@@ -289,26 +359,99 @@ class DatabaseLogic(BaseDatabaseLogic):
|
|
|
289
359
|
Returns:
|
|
290
360
|
The filtered search object.
|
|
291
361
|
"""
|
|
362
|
+
# USE_DATETIME env var
|
|
363
|
+
# True: Search by datetime, if null search by start/end datetime
|
|
364
|
+
# False: Always search only by start/end datetime
|
|
365
|
+
USE_DATETIME = get_bool_env("USE_DATETIME", default=True)
|
|
366
|
+
|
|
292
367
|
datetime_search = return_date(datetime)
|
|
293
368
|
|
|
294
369
|
if not datetime_search:
|
|
295
370
|
return search, datetime_search
|
|
296
371
|
|
|
297
|
-
if
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
|
|
302
|
-
|
|
303
|
-
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
|
|
309
|
-
|
|
372
|
+
if USE_DATETIME:
|
|
373
|
+
if "eq" in datetime_search:
|
|
374
|
+
# For exact matches, include:
|
|
375
|
+
# 1. Items with matching exact datetime
|
|
376
|
+
# 2. Items with datetime:null where the time falls within their range
|
|
377
|
+
should = [
|
|
378
|
+
Q(
|
|
379
|
+
"bool",
|
|
380
|
+
filter=[
|
|
381
|
+
Q("exists", field="properties.datetime"),
|
|
382
|
+
Q(
|
|
383
|
+
"term",
|
|
384
|
+
**{"properties__datetime": datetime_search["eq"]},
|
|
385
|
+
),
|
|
386
|
+
],
|
|
387
|
+
),
|
|
388
|
+
Q(
|
|
389
|
+
"bool",
|
|
390
|
+
must_not=[Q("exists", field="properties.datetime")],
|
|
391
|
+
filter=[
|
|
392
|
+
Q("exists", field="properties.start_datetime"),
|
|
393
|
+
Q("exists", field="properties.end_datetime"),
|
|
394
|
+
Q(
|
|
395
|
+
"range",
|
|
396
|
+
properties__start_datetime={
|
|
397
|
+
"lte": datetime_search["eq"]
|
|
398
|
+
},
|
|
399
|
+
),
|
|
400
|
+
Q(
|
|
401
|
+
"range",
|
|
402
|
+
properties__end_datetime={"gte": datetime_search["eq"]},
|
|
403
|
+
),
|
|
404
|
+
],
|
|
405
|
+
),
|
|
406
|
+
]
|
|
407
|
+
else:
|
|
408
|
+
# For date ranges, include:
|
|
409
|
+
# 1. Items with datetime in the range
|
|
410
|
+
# 2. Items with datetime:null that overlap the search range
|
|
411
|
+
should = [
|
|
412
|
+
Q(
|
|
413
|
+
"bool",
|
|
414
|
+
filter=[
|
|
415
|
+
Q("exists", field="properties.datetime"),
|
|
416
|
+
Q(
|
|
417
|
+
"range",
|
|
418
|
+
properties__datetime={
|
|
419
|
+
"gte": datetime_search["gte"],
|
|
420
|
+
"lte": datetime_search["lte"],
|
|
421
|
+
},
|
|
422
|
+
),
|
|
423
|
+
],
|
|
424
|
+
),
|
|
425
|
+
Q(
|
|
426
|
+
"bool",
|
|
427
|
+
must_not=[Q("exists", field="properties.datetime")],
|
|
428
|
+
filter=[
|
|
429
|
+
Q("exists", field="properties.start_datetime"),
|
|
430
|
+
Q("exists", field="properties.end_datetime"),
|
|
431
|
+
Q(
|
|
432
|
+
"range",
|
|
433
|
+
properties__start_datetime={
|
|
434
|
+
"lte": datetime_search["lte"]
|
|
435
|
+
},
|
|
436
|
+
),
|
|
437
|
+
Q(
|
|
438
|
+
"range",
|
|
439
|
+
properties__end_datetime={
|
|
440
|
+
"gte": datetime_search["gte"]
|
|
441
|
+
},
|
|
442
|
+
),
|
|
443
|
+
],
|
|
444
|
+
),
|
|
445
|
+
]
|
|
446
|
+
|
|
447
|
+
return (
|
|
448
|
+
search.query(Q("bool", should=should, minimum_should_match=1)),
|
|
449
|
+
datetime_search,
|
|
450
|
+
)
|
|
451
|
+
else:
|
|
452
|
+
if "eq" in datetime_search:
|
|
453
|
+
filter_query = Q(
|
|
310
454
|
"bool",
|
|
311
|
-
must_not=[Q("exists", field="properties.datetime")],
|
|
312
455
|
filter=[
|
|
313
456
|
Q("exists", field="properties.start_datetime"),
|
|
314
457
|
Q("exists", field="properties.end_datetime"),
|
|
@@ -321,29 +464,10 @@ class DatabaseLogic(BaseDatabaseLogic):
|
|
|
321
464
|
properties__end_datetime={"gte": datetime_search["eq"]},
|
|
322
465
|
),
|
|
323
466
|
],
|
|
324
|
-
)
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
# For date ranges, include:
|
|
328
|
-
# 1. Items with datetime in the range
|
|
329
|
-
# 2. Items with datetime:null that overlap the search range
|
|
330
|
-
should = [
|
|
331
|
-
Q(
|
|
332
|
-
"bool",
|
|
333
|
-
filter=[
|
|
334
|
-
Q("exists", field="properties.datetime"),
|
|
335
|
-
Q(
|
|
336
|
-
"range",
|
|
337
|
-
properties__datetime={
|
|
338
|
-
"gte": datetime_search["gte"],
|
|
339
|
-
"lte": datetime_search["lte"],
|
|
340
|
-
},
|
|
341
|
-
),
|
|
342
|
-
],
|
|
343
|
-
),
|
|
344
|
-
Q(
|
|
467
|
+
)
|
|
468
|
+
else:
|
|
469
|
+
filter_query = Q(
|
|
345
470
|
"bool",
|
|
346
|
-
must_not=[Q("exists", field="properties.datetime")],
|
|
347
471
|
filter=[
|
|
348
472
|
Q("exists", field="properties.start_datetime"),
|
|
349
473
|
Q("exists", field="properties.end_datetime"),
|
|
@@ -356,13 +480,8 @@ class DatabaseLogic(BaseDatabaseLogic):
|
|
|
356
480
|
properties__end_datetime={"gte": datetime_search["gte"]},
|
|
357
481
|
),
|
|
358
482
|
],
|
|
359
|
-
)
|
|
360
|
-
|
|
361
|
-
|
|
362
|
-
return (
|
|
363
|
-
search.query(Q("bool", should=should, minimum_should_match=1)),
|
|
364
|
-
datetime_search,
|
|
365
|
-
)
|
|
483
|
+
)
|
|
484
|
+
return search.query(filter_query), datetime_search
|
|
366
485
|
|
|
367
486
|
@staticmethod
|
|
368
487
|
def apply_bbox_filter(search: Search, bbox: List):
|
|
@@ -543,7 +662,7 @@ class DatabaseLogic(BaseDatabaseLogic):
|
|
|
543
662
|
index_param = ITEM_INDICES
|
|
544
663
|
query = add_collections_to_body(collection_ids, query)
|
|
545
664
|
|
|
546
|
-
max_result_window =
|
|
665
|
+
max_result_window = get_max_limit()
|
|
547
666
|
|
|
548
667
|
size_limit = min(limit + 1, max_result_window)
|
|
549
668
|
|
|
@@ -1,2 +1,2 @@
|
|
|
1
1
|
"""library version."""
|
|
2
|
-
__version__ = "6.
|
|
2
|
+
__version__ = "6.4.0"
|
|
@@ -1,6 +1,6 @@
|
|
|
1
|
-
Metadata-Version: 2.
|
|
2
|
-
Name:
|
|
3
|
-
Version: 6.
|
|
1
|
+
Metadata-Version: 2.1
|
|
2
|
+
Name: stac-fastapi-elasticsearch
|
|
3
|
+
Version: 6.4.0
|
|
4
4
|
Summary: An implementation of STAC API based on the FastAPI framework with both Elasticsearch and Opensearch.
|
|
5
5
|
Home-page: https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch
|
|
6
6
|
License: MIT
|
|
@@ -15,33 +15,9 @@ Classifier: Programming Language :: Python :: 3.13
|
|
|
15
15
|
Classifier: License :: OSI Approved :: MIT License
|
|
16
16
|
Requires-Python: >=3.9
|
|
17
17
|
Description-Content-Type: text/markdown
|
|
18
|
-
Requires-Dist: stac-fastapi-core==6.2.1
|
|
19
|
-
Requires-Dist: sfeos-helpers==6.2.1
|
|
20
|
-
Requires-Dist: elasticsearch[async]~=8.18.0
|
|
21
|
-
Requires-Dist: uvicorn~=0.23.0
|
|
22
|
-
Requires-Dist: starlette<0.36.0,>=0.35.0
|
|
23
18
|
Provides-Extra: dev
|
|
24
|
-
Requires-Dist: pytest~=7.0.0; extra == "dev"
|
|
25
|
-
Requires-Dist: pytest-cov~=4.0.0; extra == "dev"
|
|
26
|
-
Requires-Dist: pytest-asyncio~=0.21.0; extra == "dev"
|
|
27
|
-
Requires-Dist: pre-commit~=3.0.0; extra == "dev"
|
|
28
|
-
Requires-Dist: ciso8601~=2.3.0; extra == "dev"
|
|
29
|
-
Requires-Dist: httpx<0.28.0,>=0.24.0; extra == "dev"
|
|
30
19
|
Provides-Extra: docs
|
|
31
|
-
Requires-Dist: mkdocs~=1.4.0; extra == "docs"
|
|
32
|
-
Requires-Dist: mkdocs-material~=9.0.0; extra == "docs"
|
|
33
|
-
Requires-Dist: pdocs~=1.2.0; extra == "docs"
|
|
34
20
|
Provides-Extra: server
|
|
35
|
-
Requires-Dist: uvicorn[standard]~=0.23.0; extra == "server"
|
|
36
|
-
Dynamic: classifier
|
|
37
|
-
Dynamic: description
|
|
38
|
-
Dynamic: description-content-type
|
|
39
|
-
Dynamic: home-page
|
|
40
|
-
Dynamic: license
|
|
41
|
-
Dynamic: provides-extra
|
|
42
|
-
Dynamic: requires-dist
|
|
43
|
-
Dynamic: requires-python
|
|
44
|
-
Dynamic: summary
|
|
45
21
|
|
|
46
22
|
# stac-fastapi-elasticsearch-opensearch
|
|
47
23
|
|
|
@@ -81,11 +57,10 @@ SFEOS (stac-fastapi-elasticsearch-opensearch) is a high-performance, scalable AP
|
|
|
81
57
|
- **Scale to millions of geospatial assets** with fast search performance through optimized spatial indexing and query capabilities
|
|
82
58
|
- **Support OGC-compliant filtering** including spatial operations (intersects, contains, etc.) and temporal queries
|
|
83
59
|
- **Perform geospatial aggregations** to analyze data distribution across space and time
|
|
60
|
+
- **Enhanced collection search capabilities** with support for sorting and field selection
|
|
84
61
|
|
|
85
62
|
This implementation builds on the STAC-FastAPI framework, providing a production-ready solution specifically optimized for Elasticsearch and OpenSearch databases. It's ideal for organizations managing large geospatial data catalogs who need efficient discovery and access capabilities through standardized APIs.
|
|
86
63
|
|
|
87
|
-
|
|
88
|
-
|
|
89
64
|
## Common Deployment Patterns
|
|
90
65
|
|
|
91
66
|
stac-fastapi-elasticsearch-opensearch can be deployed in several ways depending on your needs:
|
|
@@ -111,26 +86,44 @@ This project is built on the following technologies: STAC, stac-fastapi, FastAPI
|
|
|
111
86
|
|
|
112
87
|
## Table of Contents
|
|
113
88
|
|
|
114
|
-
- [
|
|
115
|
-
- [
|
|
116
|
-
- [
|
|
117
|
-
- [
|
|
118
|
-
- [
|
|
119
|
-
- [
|
|
120
|
-
- [
|
|
121
|
-
- [
|
|
122
|
-
- [
|
|
123
|
-
- [
|
|
124
|
-
- [
|
|
125
|
-
- [
|
|
126
|
-
- [
|
|
127
|
-
- [
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
- [
|
|
131
|
-
- [
|
|
132
|
-
- [
|
|
133
|
-
- [
|
|
89
|
+
- [stac-fastapi-elasticsearch-opensearch](#stac-fastapi-elasticsearch-opensearch)
|
|
90
|
+
- [Sponsors \& Supporters](#sponsors--supporters)
|
|
91
|
+
- [Project Introduction - What is SFEOS?](#project-introduction---what-is-sfeos)
|
|
92
|
+
- [Common Deployment Patterns](#common-deployment-patterns)
|
|
93
|
+
- [Technologies](#technologies)
|
|
94
|
+
- [Table of Contents](#table-of-contents)
|
|
95
|
+
- [Collection Search Extensions](#collection-search-extensions)
|
|
96
|
+
- [Documentation \& Resources](#documentation--resources)
|
|
97
|
+
- [Package Structure](#package-structure)
|
|
98
|
+
- [Examples](#examples)
|
|
99
|
+
- [Performance](#performance)
|
|
100
|
+
- [Direct Response Mode](#direct-response-mode)
|
|
101
|
+
- [Quick Start](#quick-start)
|
|
102
|
+
- [Installation](#installation)
|
|
103
|
+
- [Running Locally](#running-locally)
|
|
104
|
+
- [Using Pre-built Docker Images](#using-pre-built-docker-images)
|
|
105
|
+
- [Using Docker Compose](#using-docker-compose)
|
|
106
|
+
- [Configuration Reference](#configuration-reference)
|
|
107
|
+
- [Datetime-Based Index Management](#datetime-based-index-management)
|
|
108
|
+
- [Overview](#overview)
|
|
109
|
+
- [When to Use](#when-to-use)
|
|
110
|
+
- [Configuration](#configuration)
|
|
111
|
+
- [Enabling Datetime-Based Indexing](#enabling-datetime-based-indexing)
|
|
112
|
+
- [Related Configuration Variables](#related-configuration-variables)
|
|
113
|
+
- [How Datetime-Based Indexing Works](#how-datetime-based-indexing-works)
|
|
114
|
+
- [Index and Alias Naming Convention](#index-and-alias-naming-convention)
|
|
115
|
+
- [Index Size Management](#index-size-management)
|
|
116
|
+
- [Interacting with the API](#interacting-with-the-api)
|
|
117
|
+
- [Configure the API](#configure-the-api)
|
|
118
|
+
- [Collection Pagination](#collection-pagination)
|
|
119
|
+
- [Ingesting Sample Data CLI Tool](#ingesting-sample-data-cli-tool)
|
|
120
|
+
- [Elasticsearch Mappings](#elasticsearch-mappings)
|
|
121
|
+
- [Managing Elasticsearch Indices](#managing-elasticsearch-indices)
|
|
122
|
+
- [Snapshots](#snapshots)
|
|
123
|
+
- [Reindexing](#reindexing)
|
|
124
|
+
- [Auth](#auth)
|
|
125
|
+
- [Aggregation](#aggregation)
|
|
126
|
+
- [Rate Limiting](#rate-limiting)
|
|
134
127
|
|
|
135
128
|
## Documentation & Resources
|
|
136
129
|
|
|
@@ -141,6 +134,37 @@ This project is built on the following technologies: STAC, stac-fastapi, FastAPI
|
|
|
141
134
|
- [Gitter Chat](https://app.gitter.im/#/room/#stac-fastapi-elasticsearch_community:gitter.im) - For real-time discussions
|
|
142
135
|
- [GitHub Discussions](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/discussions) - For longer-form questions and answers
|
|
143
136
|
|
|
137
|
+
## Collection Search Extensions
|
|
138
|
+
|
|
139
|
+
SFEOS implements extended capabilities for the `/collections` endpoint, allowing for more powerful collection discovery:
|
|
140
|
+
|
|
141
|
+
- **Sorting**: Sort collections by sortable fields using the `sortby` parameter
|
|
142
|
+
- Example: `/collections?sortby=+id` (ascending sort by ID)
|
|
143
|
+
- Example: `/collections?sortby=-id` (descending sort by ID)
|
|
144
|
+
- Example: `/collections?sortby=-temporal` (descending sort by temporal extent)
|
|
145
|
+
|
|
146
|
+
- **Field Selection**: Request only specific fields to be returned using the `fields` parameter
|
|
147
|
+
- Example: `/collections?fields=id,title,description`
|
|
148
|
+
- This helps reduce payload size when only certain fields are needed
|
|
149
|
+
|
|
150
|
+
- **Free Text Search**: Search across collection text fields using the `q` parameter
|
|
151
|
+
- Example: `/collections?q=landsat`
|
|
152
|
+
- Searches across multiple text fields including title, description, and keywords
|
|
153
|
+
- Supports partial word matching and relevance-based sorting
|
|
154
|
+
|
|
155
|
+
These extensions make it easier to build user interfaces that display and navigate through collections efficiently.
|
|
156
|
+
|
|
157
|
+
> **Configuration**: Collection search extensions can be disabled by setting the `ENABLE_COLLECTIONS_SEARCH` environment variable to `false`. By default, these extensions are enabled.
|
|
158
|
+
|
|
159
|
+
> **Note**: Sorting is only available on fields that are indexed for sorting in Elasticsearch/OpenSearch. With the default mappings, you can sort on:
|
|
160
|
+
> - `id` (keyword field)
|
|
161
|
+
> - `extent.temporal.interval` (date field)
|
|
162
|
+
> - `temporal` (alias to extent.temporal.interval)
|
|
163
|
+
>
|
|
164
|
+
> Text fields like `title` and `description` are not sortable by default as they use text analysis for better search capabilities. Attempting to sort on these fields will result in a user-friendly error message explaining which fields are sortable and how to make additional fields sortable by updating the mappings.
|
|
165
|
+
>
|
|
166
|
+
> **Important**: Adding keyword fields to make text fields sortable can significantly increase the index size, especially for large text fields. Consider the storage implications when deciding which fields to make sortable.
|
|
167
|
+
|
|
144
168
|
## Package Structure
|
|
145
169
|
|
|
146
170
|
This project is organized into several packages, each with a specific purpose:
|
|
@@ -271,8 +295,12 @@ You can customize additional settings in your `.env` file:
|
|
|
271
295
|
| `ENABLE_DIRECT_RESPONSE` | Enable direct response for maximum performance (disables all FastAPI dependencies, including authentication, custom status codes, and validation) | `false` | Optional |
|
|
272
296
|
| `RAISE_ON_BULK_ERROR` | Controls whether bulk insert operations raise exceptions on errors. If set to `true`, the operation will stop and raise an exception when an error occurs. If set to `false`, errors will be logged, and the operation will continue. **Note:** STAC Item and ItemCollection validation errors will always raise, regardless of this flag. | `false` | Optional |
|
|
273
297
|
| `DATABASE_REFRESH` | Controls whether database operations refresh the index immediately after changes. If set to `true`, changes will be immediately searchable. If set to `false`, changes may not be immediately visible but can improve performance for bulk operations. If set to `wait_for`, changes will wait for the next refresh cycle to become visible. | `false` | Optional |
|
|
298
|
+
| `ENABLE_COLLECTIONS_SEARCH` | Enable collection search extensions (sort, fields). | `true` | Optional |
|
|
274
299
|
| `ENABLE_TRANSACTIONS_EXTENSIONS` | Enables or disables the Transactions and Bulk Transactions API extensions. If set to `false`, the POST `/collections` route and related transaction endpoints (including bulk transaction operations) will be unavailable in the API. This is useful for deployments where mutating the catalog via the API should be prevented. | `true` | Optional |
|
|
275
300
|
| `STAC_ITEM_LIMIT` | Sets the environment variable for result limiting to SFEOS for the number of returned items and STAC collections. | `10` | Optional |
|
|
301
|
+
| `STAC_INDEX_ASSETS` | Controls if Assets are indexed when added to Elasticsearch/Opensearch. This allows asset fields to be included in search queries. | `false` | Optional |
|
|
302
|
+
| `ENV_MAX_LIMIT` | Configures the environment variable in SFEOS to override the default `MAX_LIMIT`, which controls the limit parameter for returned items and STAC collections. | `10,000` | Optional |
|
|
303
|
+
| `USE_DATETIME` | Configures the datetime search behavior in SFEOS. When enabled, searches both datetime field and falls back to start_datetime/end_datetime range for items with null datetime. When disabled, searches only by start_datetime/end_datetime range. | True | Optional |
|
|
276
304
|
|
|
277
305
|
> [!NOTE]
|
|
278
306
|
> The variables `ES_HOST`, `ES_PORT`, `ES_USE_SSL`, `ES_VERIFY_CERTS` and `ES_TIMEOUT` apply to both Elasticsearch and OpenSearch backends, so there is no need to rename the key names to `OS_` even if you're using OpenSearch.
|
|
@@ -414,6 +442,10 @@ The system uses a precise naming convention:
|
|
|
414
442
|
- **Root Path Configuration**: The application root path is the base URL by default.
|
|
415
443
|
- For AWS Lambda with Gateway API: Set `STAC_FASTAPI_ROOT_PATH` to match the Gateway API stage name (e.g., `/v1`)
|
|
416
444
|
|
|
445
|
+
- **Feature Configuration**: Control which features are enabled:
|
|
446
|
+
- `ENABLE_COLLECTIONS_SEARCH`: Set to `true` (default) to enable collection search extensions (sort, fields). Set to `false` to disable.
|
|
447
|
+
- `ENABLE_TRANSACTIONS_EXTENSIONS`: Set to `true` (default) to enable transaction extensions. Set to `false` to disable.
|
|
448
|
+
|
|
417
449
|
|
|
418
450
|
## Collection Pagination
|
|
419
451
|
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|