stac-fastapi-elasticsearch 4.0.0a2__tar.gz → 4.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {stac_fastapi_elasticsearch-4.0.0a2 → stac_fastapi_elasticsearch-4.2.0}/PKG-INFO +8 -4
- {stac_fastapi_elasticsearch-4.0.0a2 → stac_fastapi_elasticsearch-4.2.0}/README.md +7 -3
- {stac_fastapi_elasticsearch-4.0.0a2 → stac_fastapi_elasticsearch-4.2.0}/setup.py +1 -1
- {stac_fastapi_elasticsearch-4.0.0a2 → stac_fastapi_elasticsearch-4.2.0}/stac_fastapi/elasticsearch/app.py +48 -22
- {stac_fastapi_elasticsearch-4.0.0a2 → stac_fastapi_elasticsearch-4.2.0}/stac_fastapi/elasticsearch/config.py +26 -2
- {stac_fastapi_elasticsearch-4.0.0a2 → stac_fastapi_elasticsearch-4.2.0}/stac_fastapi/elasticsearch/database_logic.py +474 -85
- stac_fastapi_elasticsearch-4.2.0/stac_fastapi/elasticsearch/version.py +2 -0
- {stac_fastapi_elasticsearch-4.0.0a2 → stac_fastapi_elasticsearch-4.2.0}/stac_fastapi_elasticsearch.egg-info/PKG-INFO +8 -4
- {stac_fastapi_elasticsearch-4.0.0a2 → stac_fastapi_elasticsearch-4.2.0}/stac_fastapi_elasticsearch.egg-info/requires.txt +1 -1
- stac_fastapi_elasticsearch-4.0.0a2/stac_fastapi/elasticsearch/version.py +0 -2
- {stac_fastapi_elasticsearch-4.0.0a2 → stac_fastapi_elasticsearch-4.2.0}/setup.cfg +0 -0
- {stac_fastapi_elasticsearch-4.0.0a2 → stac_fastapi_elasticsearch-4.2.0}/stac_fastapi/elasticsearch/__init__.py +0 -0
- {stac_fastapi_elasticsearch-4.0.0a2 → stac_fastapi_elasticsearch-4.2.0}/stac_fastapi_elasticsearch.egg-info/SOURCES.txt +0 -0
- {stac_fastapi_elasticsearch-4.0.0a2 → stac_fastapi_elasticsearch-4.2.0}/stac_fastapi_elasticsearch.egg-info/dependency_links.txt +0 -0
- {stac_fastapi_elasticsearch-4.0.0a2 → stac_fastapi_elasticsearch-4.2.0}/stac_fastapi_elasticsearch.egg-info/entry_points.txt +0 -0
- {stac_fastapi_elasticsearch-4.0.0a2 → stac_fastapi_elasticsearch-4.2.0}/stac_fastapi_elasticsearch.egg-info/not-zip-safe +0 -0
- {stac_fastapi_elasticsearch-4.0.0a2 → stac_fastapi_elasticsearch-4.2.0}/stac_fastapi_elasticsearch.egg-info/top_level.txt +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.1
|
|
2
2
|
Name: stac_fastapi_elasticsearch
|
|
3
|
-
Version: 4.0
|
|
3
|
+
Version: 4.2.0
|
|
4
4
|
Summary: An implementation of STAC API based on the FastAPI framework with both Elasticsearch and Opensearch.
|
|
5
5
|
Home-page: https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch
|
|
6
6
|
License: MIT
|
|
@@ -125,6 +125,7 @@ You can customize additional settings in your `.env` file:
|
|
|
125
125
|
| `STAC_FASTAPI_TITLE` | Title of the API in the documentation. | `stac-fastapi-elasticsearch` or `stac-fastapi-opensearch` | Optional |
|
|
126
126
|
| `STAC_FASTAPI_DESCRIPTION` | Description of the API in the documentation. | N/A | Optional |
|
|
127
127
|
| `STAC_FASTAPI_VERSION` | API version. | `2.1` | Optional |
|
|
128
|
+
| `STAC_FASTAPI_LANDING_PAGE_ID` | Landing page ID | `stac-fastapi` | Optional |
|
|
128
129
|
| `APP_HOST` | Server bind address. | `0.0.0.0` | Optional |
|
|
129
130
|
| `APP_PORT` | Server port. | `8080` | Optional |
|
|
130
131
|
| `ENVIRONMENT` | Runtime environment. | `local` | Optional |
|
|
@@ -132,9 +133,12 @@ You can customize additional settings in your `.env` file:
|
|
|
132
133
|
| `RELOAD` | Enable auto-reload for development. | `true` | Optional |
|
|
133
134
|
| `STAC_FASTAPI_RATE_LIMIT` | API rate limit per client. | `200/minute` | Optional |
|
|
134
135
|
| `BACKEND` | Tests-related variable | `elasticsearch` or `opensearch` based on the backend | Optional |
|
|
135
|
-
| `ELASTICSEARCH_VERSION` | Version of Elasticsearch to use. | `8.11.0` | Optional |
|
|
136
|
-
| `
|
|
137
|
-
| `
|
|
136
|
+
| `ELASTICSEARCH_VERSION` | Version of Elasticsearch to use. | `8.11.0` | Optional | |
|
|
137
|
+
| `OPENSEARCH_VERSION` | OpenSearch version | `2.11.1` | Optional
|
|
138
|
+
| `ENABLE_DIRECT_RESPONSE` | Enable direct response for maximum performance (disables all FastAPI dependencies, including authentication, custom status codes, and validation) | `false` | Optional
|
|
139
|
+
| `RAISE_ON_BULK_ERROR` | Controls whether bulk insert operations raise exceptions on errors. If set to `true`, the operation will stop and raise an exception when an error occurs. If set to `false`, errors will be logged, and the operation will continue. **Note:** STAC Item and ItemCollection validation errors will always raise, regardless of this flag. | `false` Optional |
|
|
140
|
+
| `DATABASE_REFRESH` | Controls whether database operations refresh the index immediately after changes. If set to `true`, changes will be immediately searchable. If set to `false`, changes may not be immediately visible but can improve performance for bulk operations. If set to `wait_for`, changes will wait for the next refresh cycle to become visible. | `false` | Optional |
|
|
141
|
+
| `ENABLE_TRANSACTIONS_EXTENSIONS` | Enables or disables the Transactions and Bulk Transactions API extensions. If set to `false`, the POST `/collections` route and related transaction endpoints (including bulk transaction operations) will be unavailable in the API. This is useful for deployments where mutating the catalog via the API should be prevented. | `true` | Optional |
|
|
138
142
|
|
|
139
143
|
> [!NOTE]
|
|
140
144
|
> The variables `ES_HOST`, `ES_PORT`, `ES_USE_SSL`, and `ES_VERIFY_CERTS` apply to both Elasticsearch and OpenSearch backends, so there is no need to rename the key names to `OS_` even if you're using OpenSearch.
|
|
@@ -104,6 +104,7 @@ You can customize additional settings in your `.env` file:
|
|
|
104
104
|
| `STAC_FASTAPI_TITLE` | Title of the API in the documentation. | `stac-fastapi-elasticsearch` or `stac-fastapi-opensearch` | Optional |
|
|
105
105
|
| `STAC_FASTAPI_DESCRIPTION` | Description of the API in the documentation. | N/A | Optional |
|
|
106
106
|
| `STAC_FASTAPI_VERSION` | API version. | `2.1` | Optional |
|
|
107
|
+
| `STAC_FASTAPI_LANDING_PAGE_ID` | Landing page ID | `stac-fastapi` | Optional |
|
|
107
108
|
| `APP_HOST` | Server bind address. | `0.0.0.0` | Optional |
|
|
108
109
|
| `APP_PORT` | Server port. | `8080` | Optional |
|
|
109
110
|
| `ENVIRONMENT` | Runtime environment. | `local` | Optional |
|
|
@@ -111,9 +112,12 @@ You can customize additional settings in your `.env` file:
|
|
|
111
112
|
| `RELOAD` | Enable auto-reload for development. | `true` | Optional |
|
|
112
113
|
| `STAC_FASTAPI_RATE_LIMIT` | API rate limit per client. | `200/minute` | Optional |
|
|
113
114
|
| `BACKEND` | Tests-related variable | `elasticsearch` or `opensearch` based on the backend | Optional |
|
|
114
|
-
| `ELASTICSEARCH_VERSION` | Version of Elasticsearch to use. | `8.11.0` | Optional |
|
|
115
|
-
| `
|
|
116
|
-
| `
|
|
115
|
+
| `ELASTICSEARCH_VERSION` | Version of Elasticsearch to use. | `8.11.0` | Optional | |
|
|
116
|
+
| `OPENSEARCH_VERSION` | OpenSearch version | `2.11.1` | Optional
|
|
117
|
+
| `ENABLE_DIRECT_RESPONSE` | Enable direct response for maximum performance (disables all FastAPI dependencies, including authentication, custom status codes, and validation) | `false` | Optional
|
|
118
|
+
| `RAISE_ON_BULK_ERROR` | Controls whether bulk insert operations raise exceptions on errors. If set to `true`, the operation will stop and raise an exception when an error occurs. If set to `false`, errors will be logged, and the operation will continue. **Note:** STAC Item and ItemCollection validation errors will always raise, regardless of this flag. | `false` Optional |
|
|
119
|
+
| `DATABASE_REFRESH` | Controls whether database operations refresh the index immediately after changes. If set to `true`, changes will be immediately searchable. If set to `false`, changes may not be immediately visible but can improve performance for bulk operations. If set to `wait_for`, changes will wait for the next refresh cycle to become visible. | `false` | Optional |
|
|
120
|
+
| `ENABLE_TRANSACTIONS_EXTENSIONS` | Enables or disables the Transactions and Bulk Transactions API extensions. If set to `false`, the POST `/collections` route and related transaction endpoints (including bulk transaction operations) will be unavailable in the API. This is useful for deployments where mutating the catalog via the API should be prevented. | `true` | Optional |
|
|
117
121
|
|
|
118
122
|
> [!NOTE]
|
|
119
123
|
> The variables `ES_HOST`, `ES_PORT`, `ES_USE_SSL`, and `ES_VERIFY_CERTS` apply to both Elasticsearch and OpenSearch backends, so there is no need to rename the key names to `OS_` even if you're using OpenSearch.
|
|
@@ -1,6 +1,10 @@
|
|
|
1
1
|
"""FastAPI application."""
|
|
2
2
|
|
|
3
|
+
import logging
|
|
3
4
|
import os
|
|
5
|
+
from contextlib import asynccontextmanager
|
|
6
|
+
|
|
7
|
+
from fastapi import FastAPI
|
|
4
8
|
|
|
5
9
|
from stac_fastapi.api.app import StacApi
|
|
6
10
|
from stac_fastapi.api.models import create_get_request_model, create_post_request_model
|
|
@@ -20,6 +24,7 @@ from stac_fastapi.core.extensions.fields import FieldsExtension
|
|
|
20
24
|
from stac_fastapi.core.rate_limit import setup_rate_limit
|
|
21
25
|
from stac_fastapi.core.route_dependencies import get_route_dependencies
|
|
22
26
|
from stac_fastapi.core.session import Session
|
|
27
|
+
from stac_fastapi.core.utilities import get_bool_env
|
|
23
28
|
from stac_fastapi.elasticsearch.config import ElasticsearchSettings
|
|
24
29
|
from stac_fastapi.elasticsearch.database_logic import (
|
|
25
30
|
DatabaseLogic,
|
|
@@ -36,6 +41,12 @@ from stac_fastapi.extensions.core import (
|
|
|
36
41
|
)
|
|
37
42
|
from stac_fastapi.extensions.third_party import BulkTransactionExtension
|
|
38
43
|
|
|
44
|
+
logging.basicConfig(level=logging.INFO)
|
|
45
|
+
logger = logging.getLogger(__name__)
|
|
46
|
+
|
|
47
|
+
TRANSACTIONS_EXTENSIONS = get_bool_env("ENABLE_TRANSACTIONS_EXTENSIONS", default=True)
|
|
48
|
+
logger.info("TRANSACTIONS_EXTENSIONS is set to %s", TRANSACTIONS_EXTENSIONS)
|
|
49
|
+
|
|
39
50
|
settings = ElasticsearchSettings()
|
|
40
51
|
session = Session.create_from_settings(settings)
|
|
41
52
|
|
|
@@ -57,19 +68,6 @@ aggregation_extension.POST = EsAggregationExtensionPostRequest
|
|
|
57
68
|
aggregation_extension.GET = EsAggregationExtensionGetRequest
|
|
58
69
|
|
|
59
70
|
search_extensions = [
|
|
60
|
-
TransactionExtension(
|
|
61
|
-
client=TransactionsClient(
|
|
62
|
-
database=database_logic, session=session, settings=settings
|
|
63
|
-
),
|
|
64
|
-
settings=settings,
|
|
65
|
-
),
|
|
66
|
-
BulkTransactionExtension(
|
|
67
|
-
client=BulkTransactionsClient(
|
|
68
|
-
database=database_logic,
|
|
69
|
-
session=session,
|
|
70
|
-
settings=settings,
|
|
71
|
-
)
|
|
72
|
-
),
|
|
73
71
|
FieldsExtension(),
|
|
74
72
|
QueryExtension(),
|
|
75
73
|
SortExtension(),
|
|
@@ -78,6 +76,27 @@ search_extensions = [
|
|
|
78
76
|
FreeTextExtension(),
|
|
79
77
|
]
|
|
80
78
|
|
|
79
|
+
if TRANSACTIONS_EXTENSIONS:
|
|
80
|
+
search_extensions.insert(
|
|
81
|
+
0,
|
|
82
|
+
TransactionExtension(
|
|
83
|
+
client=TransactionsClient(
|
|
84
|
+
database=database_logic, session=session, settings=settings
|
|
85
|
+
),
|
|
86
|
+
settings=settings,
|
|
87
|
+
),
|
|
88
|
+
)
|
|
89
|
+
search_extensions.insert(
|
|
90
|
+
1,
|
|
91
|
+
BulkTransactionExtension(
|
|
92
|
+
client=BulkTransactionsClient(
|
|
93
|
+
database=database_logic,
|
|
94
|
+
session=session,
|
|
95
|
+
settings=settings,
|
|
96
|
+
)
|
|
97
|
+
),
|
|
98
|
+
)
|
|
99
|
+
|
|
81
100
|
extensions = [aggregation_extension] + search_extensions
|
|
82
101
|
|
|
83
102
|
database_logic.extensions = [type(ext).__name__ for ext in extensions]
|
|
@@ -87,27 +106,34 @@ post_request_model = create_post_request_model(search_extensions)
|
|
|
87
106
|
api = StacApi(
|
|
88
107
|
title=os.getenv("STAC_FASTAPI_TITLE", "stac-fastapi-elasticsearch"),
|
|
89
108
|
description=os.getenv("STAC_FASTAPI_DESCRIPTION", "stac-fastapi-elasticsearch"),
|
|
90
|
-
api_version=os.getenv("STAC_FASTAPI_VERSION", "4.0
|
|
109
|
+
api_version=os.getenv("STAC_FASTAPI_VERSION", "4.2.0"),
|
|
91
110
|
settings=settings,
|
|
92
111
|
extensions=extensions,
|
|
93
112
|
client=CoreClient(
|
|
94
|
-
database=database_logic,
|
|
113
|
+
database=database_logic,
|
|
114
|
+
session=session,
|
|
115
|
+
post_request_model=post_request_model,
|
|
116
|
+
landing_page_id=os.getenv("STAC_FASTAPI_LANDING_PAGE_ID", "stac-fastapi"),
|
|
95
117
|
),
|
|
96
118
|
search_get_request_model=create_get_request_model(search_extensions),
|
|
97
119
|
search_post_request_model=post_request_model,
|
|
98
120
|
route_dependencies=get_route_dependencies(),
|
|
99
121
|
)
|
|
100
|
-
app = api.app
|
|
101
|
-
app.root_path = os.getenv("STAC_FASTAPI_ROOT_PATH", "")
|
|
102
|
-
|
|
103
|
-
# Add rate limit
|
|
104
|
-
setup_rate_limit(app, rate_limit=os.getenv("STAC_FASTAPI_RATE_LIMIT"))
|
|
105
122
|
|
|
106
123
|
|
|
107
|
-
@
|
|
108
|
-
async def
|
|
124
|
+
@asynccontextmanager
|
|
125
|
+
async def lifespan(app: FastAPI):
|
|
126
|
+
"""Lifespan handler for FastAPI app. Initializes index templates and collections at startup."""
|
|
109
127
|
await create_index_templates()
|
|
110
128
|
await create_collection_index()
|
|
129
|
+
yield
|
|
130
|
+
|
|
131
|
+
|
|
132
|
+
app = api.app
|
|
133
|
+
app.router.lifespan_context = lifespan
|
|
134
|
+
app.root_path = os.getenv("STAC_FASTAPI_ROOT_PATH", "")
|
|
135
|
+
# Add rate limit
|
|
136
|
+
setup_rate_limit(app, rate_limit=os.getenv("STAC_FASTAPI_RATE_LIMIT"))
|
|
111
137
|
|
|
112
138
|
|
|
113
139
|
def run() -> None:
|
|
@@ -3,14 +3,14 @@
|
|
|
3
3
|
import logging
|
|
4
4
|
import os
|
|
5
5
|
import ssl
|
|
6
|
-
from typing import Any, Dict, Set
|
|
6
|
+
from typing import Any, Dict, Set, Union
|
|
7
7
|
|
|
8
8
|
import certifi
|
|
9
9
|
from elasticsearch._async.client import AsyncElasticsearch
|
|
10
10
|
|
|
11
11
|
from elasticsearch import Elasticsearch # type: ignore[attr-defined]
|
|
12
12
|
from stac_fastapi.core.base_settings import ApiBaseSettings
|
|
13
|
-
from stac_fastapi.core.utilities import get_bool_env
|
|
13
|
+
from stac_fastapi.core.utilities import get_bool_env, validate_refresh
|
|
14
14
|
from stac_fastapi.types.config import ApiSettings
|
|
15
15
|
|
|
16
16
|
|
|
@@ -86,6 +86,18 @@ class ElasticsearchSettings(ApiSettings, ApiBaseSettings):
|
|
|
86
86
|
indexed_fields: Set[str] = {"datetime"}
|
|
87
87
|
enable_response_models: bool = False
|
|
88
88
|
enable_direct_response: bool = get_bool_env("ENABLE_DIRECT_RESPONSE", default=False)
|
|
89
|
+
raise_on_bulk_error: bool = get_bool_env("RAISE_ON_BULK_ERROR", default=False)
|
|
90
|
+
|
|
91
|
+
@property
|
|
92
|
+
def database_refresh(self) -> Union[bool, str]:
|
|
93
|
+
"""
|
|
94
|
+
Get the value of the DATABASE_REFRESH environment variable.
|
|
95
|
+
|
|
96
|
+
Returns:
|
|
97
|
+
Union[bool, str]: The value of DATABASE_REFRESH, which can be True, False, or "wait_for".
|
|
98
|
+
"""
|
|
99
|
+
value = os.getenv("DATABASE_REFRESH", "false")
|
|
100
|
+
return validate_refresh(value)
|
|
89
101
|
|
|
90
102
|
@property
|
|
91
103
|
def create_client(self):
|
|
@@ -106,6 +118,18 @@ class AsyncElasticsearchSettings(ApiSettings, ApiBaseSettings):
|
|
|
106
118
|
indexed_fields: Set[str] = {"datetime"}
|
|
107
119
|
enable_response_models: bool = False
|
|
108
120
|
enable_direct_response: bool = get_bool_env("ENABLE_DIRECT_RESPONSE", default=False)
|
|
121
|
+
raise_on_bulk_error: bool = get_bool_env("RAISE_ON_BULK_ERROR", default=False)
|
|
122
|
+
|
|
123
|
+
@property
|
|
124
|
+
def database_refresh(self) -> Union[bool, str]:
|
|
125
|
+
"""
|
|
126
|
+
Get the value of the DATABASE_REFRESH environment variable.
|
|
127
|
+
|
|
128
|
+
Returns:
|
|
129
|
+
Union[bool, str]: The value of DATABASE_REFRESH, which can be True, False, or "wait_for".
|
|
130
|
+
"""
|
|
131
|
+
value = os.getenv("DATABASE_REFRESH", "false")
|
|
132
|
+
return validate_refresh(value)
|
|
109
133
|
|
|
110
134
|
@property
|
|
111
135
|
def create_client(self):
|
|
@@ -31,7 +31,7 @@ from stac_fastapi.core.database_logic import (
|
|
|
31
31
|
)
|
|
32
32
|
from stac_fastapi.core.extensions import filter
|
|
33
33
|
from stac_fastapi.core.serializers import CollectionSerializer, ItemSerializer
|
|
34
|
-
from stac_fastapi.core.utilities import MAX_LIMIT, bbox2polygon
|
|
34
|
+
from stac_fastapi.core.utilities import MAX_LIMIT, bbox2polygon, validate_refresh
|
|
35
35
|
from stac_fastapi.elasticsearch.config import AsyncElasticsearchSettings
|
|
36
36
|
from stac_fastapi.elasticsearch.config import (
|
|
37
37
|
ElasticsearchSettings as SyncElasticsearchSettings,
|
|
@@ -128,8 +128,20 @@ async def delete_item_index(collection_id: str):
|
|
|
128
128
|
class DatabaseLogic(BaseDatabaseLogic):
|
|
129
129
|
"""Database logic."""
|
|
130
130
|
|
|
131
|
-
|
|
132
|
-
|
|
131
|
+
async_settings: AsyncElasticsearchSettings = attr.ib(
|
|
132
|
+
factory=AsyncElasticsearchSettings
|
|
133
|
+
)
|
|
134
|
+
sync_settings: SyncElasticsearchSettings = attr.ib(
|
|
135
|
+
factory=SyncElasticsearchSettings
|
|
136
|
+
)
|
|
137
|
+
|
|
138
|
+
client = attr.ib(init=False)
|
|
139
|
+
sync_client = attr.ib(init=False)
|
|
140
|
+
|
|
141
|
+
def __attrs_post_init__(self):
|
|
142
|
+
"""Initialize clients after the class is instantiated."""
|
|
143
|
+
self.client = self.async_settings.create_client
|
|
144
|
+
self.sync_client = self.sync_settings.create_client
|
|
133
145
|
|
|
134
146
|
item_serializer: Type[ItemSerializer] = attr.ib(default=ItemSerializer)
|
|
135
147
|
collection_serializer: Type[CollectionSerializer] = attr.ib(
|
|
@@ -278,6 +290,34 @@ class DatabaseLogic(BaseDatabaseLogic):
|
|
|
278
290
|
)
|
|
279
291
|
return item["_source"]
|
|
280
292
|
|
|
293
|
+
async def get_queryables_mapping(self, collection_id: str = "*") -> dict:
|
|
294
|
+
"""Retrieve mapping of Queryables for search.
|
|
295
|
+
|
|
296
|
+
Args:
|
|
297
|
+
collection_id (str, optional): The id of the Collection the Queryables
|
|
298
|
+
belongs to. Defaults to "*".
|
|
299
|
+
|
|
300
|
+
Returns:
|
|
301
|
+
dict: A dictionary containing the Queryables mappings.
|
|
302
|
+
"""
|
|
303
|
+
queryables_mapping = {}
|
|
304
|
+
|
|
305
|
+
mappings = await self.client.indices.get_mapping(
|
|
306
|
+
index=f"{ITEMS_INDEX_PREFIX}{collection_id}",
|
|
307
|
+
)
|
|
308
|
+
|
|
309
|
+
for mapping in mappings.values():
|
|
310
|
+
fields = mapping["mappings"].get("properties", {})
|
|
311
|
+
properties = fields.pop("properties", {}).get("properties", {}).keys()
|
|
312
|
+
|
|
313
|
+
for field_key in fields:
|
|
314
|
+
queryables_mapping[field_key] = field_key
|
|
315
|
+
|
|
316
|
+
for property_key in properties:
|
|
317
|
+
queryables_mapping[property_key] = f"properties.{property_key}"
|
|
318
|
+
|
|
319
|
+
return queryables_mapping
|
|
320
|
+
|
|
281
321
|
@staticmethod
|
|
282
322
|
def make_search():
|
|
283
323
|
"""Database logic to create a Search instance."""
|
|
@@ -294,8 +334,8 @@ class DatabaseLogic(BaseDatabaseLogic):
|
|
|
294
334
|
return search.filter("terms", collection=collection_ids)
|
|
295
335
|
|
|
296
336
|
@staticmethod
|
|
297
|
-
def apply_datetime_filter(search: Search, datetime_search):
|
|
298
|
-
"""Apply a filter to search
|
|
337
|
+
def apply_datetime_filter(search: Search, datetime_search: dict):
|
|
338
|
+
"""Apply a filter to search on datetime, start_datetime, and end_datetime fields.
|
|
299
339
|
|
|
300
340
|
Args:
|
|
301
341
|
search (Search): The search object to filter.
|
|
@@ -304,17 +344,109 @@ class DatabaseLogic(BaseDatabaseLogic):
|
|
|
304
344
|
Returns:
|
|
305
345
|
Search: The filtered search object.
|
|
306
346
|
"""
|
|
347
|
+
should = []
|
|
348
|
+
|
|
349
|
+
# If the request is a single datetime return
|
|
350
|
+
# items with datetimes equal to the requested datetime OR
|
|
351
|
+
# the requested datetime is between their start and end datetimes
|
|
307
352
|
if "eq" in datetime_search:
|
|
308
|
-
|
|
309
|
-
|
|
353
|
+
should.extend(
|
|
354
|
+
[
|
|
355
|
+
Q(
|
|
356
|
+
"bool",
|
|
357
|
+
filter=[
|
|
358
|
+
Q(
|
|
359
|
+
"term",
|
|
360
|
+
properties__datetime=datetime_search["eq"],
|
|
361
|
+
),
|
|
362
|
+
],
|
|
363
|
+
),
|
|
364
|
+
Q(
|
|
365
|
+
"bool",
|
|
366
|
+
filter=[
|
|
367
|
+
Q(
|
|
368
|
+
"range",
|
|
369
|
+
properties__start_datetime={
|
|
370
|
+
"lte": datetime_search["eq"],
|
|
371
|
+
},
|
|
372
|
+
),
|
|
373
|
+
Q(
|
|
374
|
+
"range",
|
|
375
|
+
properties__end_datetime={
|
|
376
|
+
"gte": datetime_search["eq"],
|
|
377
|
+
},
|
|
378
|
+
),
|
|
379
|
+
],
|
|
380
|
+
),
|
|
381
|
+
]
|
|
310
382
|
)
|
|
383
|
+
|
|
384
|
+
# If the request is a date range return
|
|
385
|
+
# items with datetimes within the requested date range OR
|
|
386
|
+
# their startdatetime ithin the requested date range OR
|
|
387
|
+
# their enddatetime ithin the requested date range OR
|
|
388
|
+
# the requested daterange within their start and end datetimes
|
|
311
389
|
else:
|
|
312
|
-
|
|
313
|
-
|
|
314
|
-
|
|
315
|
-
|
|
316
|
-
|
|
390
|
+
should.extend(
|
|
391
|
+
[
|
|
392
|
+
Q(
|
|
393
|
+
"bool",
|
|
394
|
+
filter=[
|
|
395
|
+
Q(
|
|
396
|
+
"range",
|
|
397
|
+
properties__datetime={
|
|
398
|
+
"gte": datetime_search["gte"],
|
|
399
|
+
"lte": datetime_search["lte"],
|
|
400
|
+
},
|
|
401
|
+
),
|
|
402
|
+
],
|
|
403
|
+
),
|
|
404
|
+
Q(
|
|
405
|
+
"bool",
|
|
406
|
+
filter=[
|
|
407
|
+
Q(
|
|
408
|
+
"range",
|
|
409
|
+
properties__start_datetime={
|
|
410
|
+
"gte": datetime_search["gte"],
|
|
411
|
+
"lte": datetime_search["lte"],
|
|
412
|
+
},
|
|
413
|
+
),
|
|
414
|
+
],
|
|
415
|
+
),
|
|
416
|
+
Q(
|
|
417
|
+
"bool",
|
|
418
|
+
filter=[
|
|
419
|
+
Q(
|
|
420
|
+
"range",
|
|
421
|
+
properties__end_datetime={
|
|
422
|
+
"gte": datetime_search["gte"],
|
|
423
|
+
"lte": datetime_search["lte"],
|
|
424
|
+
},
|
|
425
|
+
),
|
|
426
|
+
],
|
|
427
|
+
),
|
|
428
|
+
Q(
|
|
429
|
+
"bool",
|
|
430
|
+
filter=[
|
|
431
|
+
Q(
|
|
432
|
+
"range",
|
|
433
|
+
properties__start_datetime={
|
|
434
|
+
"lte": datetime_search["gte"]
|
|
435
|
+
},
|
|
436
|
+
),
|
|
437
|
+
Q(
|
|
438
|
+
"range",
|
|
439
|
+
properties__end_datetime={
|
|
440
|
+
"gte": datetime_search["lte"]
|
|
441
|
+
},
|
|
442
|
+
),
|
|
443
|
+
],
|
|
444
|
+
),
|
|
445
|
+
]
|
|
317
446
|
)
|
|
447
|
+
|
|
448
|
+
search = search.query(Q("bool", filter=[Q("bool", should=should)]))
|
|
449
|
+
|
|
318
450
|
return search
|
|
319
451
|
|
|
320
452
|
@staticmethod
|
|
@@ -414,8 +546,9 @@ class DatabaseLogic(BaseDatabaseLogic):
|
|
|
414
546
|
|
|
415
547
|
return search
|
|
416
548
|
|
|
417
|
-
|
|
418
|
-
|
|
549
|
+
async def apply_cql2_filter(
|
|
550
|
+
self, search: Search, _filter: Optional[Dict[str, Any]]
|
|
551
|
+
):
|
|
419
552
|
"""
|
|
420
553
|
Apply a CQL2 filter to an Elasticsearch Search object.
|
|
421
554
|
|
|
@@ -435,7 +568,7 @@ class DatabaseLogic(BaseDatabaseLogic):
|
|
|
435
568
|
otherwise the original Search object.
|
|
436
569
|
"""
|
|
437
570
|
if _filter is not None:
|
|
438
|
-
es_query = filter.to_es(_filter)
|
|
571
|
+
es_query = filter.to_es(await self.get_queryables_mapping(), _filter)
|
|
439
572
|
search = search.query(es_query)
|
|
440
573
|
|
|
441
574
|
return search
|
|
@@ -607,7 +740,7 @@ class DatabaseLogic(BaseDatabaseLogic):
|
|
|
607
740
|
if not await self.client.exists(index=COLLECTIONS_INDEX, id=collection_id):
|
|
608
741
|
raise NotFoundError(f"Collection {collection_id} does not exist")
|
|
609
742
|
|
|
610
|
-
async def
|
|
743
|
+
async def async_prep_create_item(
|
|
611
744
|
self, item: Item, base_url: str, exist_ok: bool = False
|
|
612
745
|
) -> Item:
|
|
613
746
|
"""
|
|
@@ -637,49 +770,123 @@ class DatabaseLogic(BaseDatabaseLogic):
|
|
|
637
770
|
|
|
638
771
|
return self.item_serializer.stac_to_db(item, base_url)
|
|
639
772
|
|
|
640
|
-
def
|
|
773
|
+
async def bulk_async_prep_create_item(
|
|
641
774
|
self, item: Item, base_url: str, exist_ok: bool = False
|
|
642
775
|
) -> Item:
|
|
643
776
|
"""
|
|
644
777
|
Prepare an item for insertion into the database.
|
|
645
778
|
|
|
646
|
-
This method performs pre-insertion preparation on the given `item`,
|
|
647
|
-
|
|
648
|
-
|
|
779
|
+
This method performs pre-insertion preparation on the given `item`, such as:
|
|
780
|
+
- Verifying that the collection the item belongs to exists.
|
|
781
|
+
- Optionally checking if an item with the same ID already exists in the database.
|
|
782
|
+
- Serializing the item into a database-compatible format.
|
|
649
783
|
|
|
650
784
|
Args:
|
|
651
|
-
item (Item): The item to be
|
|
652
|
-
base_url (str): The base URL used
|
|
653
|
-
exist_ok (bool): Indicates whether the item can exist
|
|
785
|
+
item (Item): The item to be prepared for insertion.
|
|
786
|
+
base_url (str): The base URL used to construct the item's self URL.
|
|
787
|
+
exist_ok (bool): Indicates whether the item can already exist in the database.
|
|
788
|
+
If False, a `ConflictError` is raised if the item exists.
|
|
654
789
|
|
|
655
790
|
Returns:
|
|
656
|
-
Item: The item
|
|
791
|
+
Item: The prepared item, serialized into a database-compatible format.
|
|
657
792
|
|
|
658
793
|
Raises:
|
|
659
794
|
NotFoundError: If the collection that the item belongs to does not exist in the database.
|
|
660
|
-
ConflictError: If an item with the same ID already exists in the collection
|
|
795
|
+
ConflictError: If an item with the same ID already exists in the collection and `exist_ok` is False,
|
|
796
|
+
and `RAISE_ON_BULK_ERROR` is set to `true`.
|
|
661
797
|
"""
|
|
662
|
-
|
|
663
|
-
collection_id = item["collection"]
|
|
664
|
-
if not self.sync_client.exists(index=COLLECTIONS_INDEX, id=collection_id):
|
|
665
|
-
raise NotFoundError(f"Collection {collection_id} does not exist")
|
|
798
|
+
logger.debug(f"Preparing item {item['id']} in collection {item['collection']}.")
|
|
666
799
|
|
|
667
|
-
if
|
|
668
|
-
|
|
669
|
-
|
|
800
|
+
# Check if the collection exists
|
|
801
|
+
await self.check_collection_exists(collection_id=item["collection"])
|
|
802
|
+
|
|
803
|
+
# Check if the item already exists in the database
|
|
804
|
+
if not exist_ok and await self.client.exists(
|
|
805
|
+
index=index_alias_by_collection_id(item["collection"]),
|
|
806
|
+
id=mk_item_id(item["id"], item["collection"]),
|
|
670
807
|
):
|
|
671
|
-
|
|
672
|
-
f"Item {
|
|
808
|
+
error_message = (
|
|
809
|
+
f"Item {item['id']} in collection {item['collection']} already exists."
|
|
673
810
|
)
|
|
811
|
+
if self.async_settings.raise_on_bulk_error:
|
|
812
|
+
raise ConflictError(error_message)
|
|
813
|
+
else:
|
|
814
|
+
logger.warning(
|
|
815
|
+
f"{error_message} Continuing as `RAISE_ON_BULK_ERROR` is set to false."
|
|
816
|
+
)
|
|
817
|
+
|
|
818
|
+
# Serialize the item into a database-compatible format
|
|
819
|
+
prepped_item = self.item_serializer.stac_to_db(item, base_url)
|
|
820
|
+
logger.debug(f"Item {item['id']} prepared successfully.")
|
|
821
|
+
return prepped_item
|
|
822
|
+
|
|
823
|
+
def bulk_sync_prep_create_item(
|
|
824
|
+
self, item: Item, base_url: str, exist_ok: bool = False
|
|
825
|
+
) -> Item:
|
|
826
|
+
"""
|
|
827
|
+
Prepare an item for insertion into the database.
|
|
674
828
|
|
|
675
|
-
|
|
829
|
+
This method performs pre-insertion preparation on the given `item`, such as:
|
|
830
|
+
- Verifying that the collection the item belongs to exists.
|
|
831
|
+
- Optionally checking if an item with the same ID already exists in the database.
|
|
832
|
+
- Serializing the item into a database-compatible format.
|
|
833
|
+
|
|
834
|
+
Args:
|
|
835
|
+
item (Item): The item to be prepared for insertion.
|
|
836
|
+
base_url (str): The base URL used to construct the item's self URL.
|
|
837
|
+
exist_ok (bool): Indicates whether the item can already exist in the database.
|
|
838
|
+
If False, a `ConflictError` is raised if the item exists.
|
|
839
|
+
|
|
840
|
+
Returns:
|
|
841
|
+
Item: The prepared item, serialized into a database-compatible format.
|
|
842
|
+
|
|
843
|
+
Raises:
|
|
844
|
+
NotFoundError: If the collection that the item belongs to does not exist in the database.
|
|
845
|
+
ConflictError: If an item with the same ID already exists in the collection and `exist_ok` is False,
|
|
846
|
+
and `RAISE_ON_BULK_ERROR` is set to `true`.
|
|
847
|
+
"""
|
|
848
|
+
logger.debug(f"Preparing item {item['id']} in collection {item['collection']}.")
|
|
849
|
+
|
|
850
|
+
# Check if the collection exists
|
|
851
|
+
if not self.sync_client.exists(index=COLLECTIONS_INDEX, id=item["collection"]):
|
|
852
|
+
raise NotFoundError(f"Collection {item['collection']} does not exist")
|
|
676
853
|
|
|
677
|
-
|
|
854
|
+
# Check if the item already exists in the database
|
|
855
|
+
if not exist_ok and self.sync_client.exists(
|
|
856
|
+
index=index_alias_by_collection_id(item["collection"]),
|
|
857
|
+
id=mk_item_id(item["id"], item["collection"]),
|
|
858
|
+
):
|
|
859
|
+
error_message = (
|
|
860
|
+
f"Item {item['id']} in collection {item['collection']} already exists."
|
|
861
|
+
)
|
|
862
|
+
if self.sync_settings.raise_on_bulk_error:
|
|
863
|
+
raise ConflictError(error_message)
|
|
864
|
+
else:
|
|
865
|
+
logger.warning(
|
|
866
|
+
f"{error_message} Continuing as `RAISE_ON_BULK_ERROR` is set to false."
|
|
867
|
+
)
|
|
868
|
+
|
|
869
|
+
# Serialize the item into a database-compatible format
|
|
870
|
+
prepped_item = self.item_serializer.stac_to_db(item, base_url)
|
|
871
|
+
logger.debug(f"Item {item['id']} prepared successfully.")
|
|
872
|
+
return prepped_item
|
|
873
|
+
|
|
874
|
+
async def create_item(
|
|
875
|
+
self,
|
|
876
|
+
item: Item,
|
|
877
|
+
base_url: str = "",
|
|
878
|
+
exist_ok: bool = False,
|
|
879
|
+
**kwargs: Any,
|
|
880
|
+
):
|
|
678
881
|
"""Database logic for creating one item.
|
|
679
882
|
|
|
680
883
|
Args:
|
|
681
884
|
item (Item): The item to be created.
|
|
682
|
-
|
|
885
|
+
base_url (str, optional): The base URL for the item. Defaults to an empty string.
|
|
886
|
+
exist_ok (bool, optional): Whether to allow the item to exist already. Defaults to False.
|
|
887
|
+
**kwargs: Additional keyword arguments.
|
|
888
|
+
- refresh (str): Whether to refresh the index after the operation. Can be "true", "false", or "wait_for".
|
|
889
|
+
- refresh (bool): Whether to refresh the index after the operation. Defaults to the value in `self.async_settings.database_refresh`.
|
|
683
890
|
|
|
684
891
|
Raises:
|
|
685
892
|
ConflictError: If the item already exists in the database.
|
|
@@ -687,41 +894,72 @@ class DatabaseLogic(BaseDatabaseLogic):
|
|
|
687
894
|
Returns:
|
|
688
895
|
None
|
|
689
896
|
"""
|
|
690
|
-
#
|
|
897
|
+
# Extract item and collection IDs
|
|
691
898
|
item_id = item["id"]
|
|
692
899
|
collection_id = item["collection"]
|
|
693
|
-
|
|
900
|
+
|
|
901
|
+
# Ensure kwargs is a dictionary
|
|
902
|
+
kwargs = kwargs or {}
|
|
903
|
+
|
|
904
|
+
# Resolve the `refresh` parameter
|
|
905
|
+
refresh = kwargs.get("refresh", self.async_settings.database_refresh)
|
|
906
|
+
refresh = validate_refresh(refresh)
|
|
907
|
+
|
|
908
|
+
# Log the creation attempt
|
|
909
|
+
logger.info(
|
|
910
|
+
f"Creating item {item_id} in collection {collection_id} with refresh={refresh}"
|
|
911
|
+
)
|
|
912
|
+
|
|
913
|
+
# Prepare the item for insertion
|
|
914
|
+
item = await self.async_prep_create_item(
|
|
915
|
+
item=item, base_url=base_url, exist_ok=exist_ok
|
|
916
|
+
)
|
|
917
|
+
|
|
918
|
+
# Index the item in the database
|
|
919
|
+
await self.client.index(
|
|
694
920
|
index=index_alias_by_collection_id(collection_id),
|
|
695
921
|
id=mk_item_id(item_id, collection_id),
|
|
696
922
|
document=item,
|
|
697
923
|
refresh=refresh,
|
|
698
924
|
)
|
|
699
925
|
|
|
700
|
-
|
|
701
|
-
raise ConflictError(
|
|
702
|
-
f"Item {item_id} in collection {collection_id} already exists"
|
|
703
|
-
)
|
|
704
|
-
|
|
705
|
-
async def delete_item(
|
|
706
|
-
self, item_id: str, collection_id: str, refresh: bool = False
|
|
707
|
-
):
|
|
926
|
+
async def delete_item(self, item_id: str, collection_id: str, **kwargs: Any):
|
|
708
927
|
"""Delete a single item from the database.
|
|
709
928
|
|
|
710
929
|
Args:
|
|
711
930
|
item_id (str): The id of the Item to be deleted.
|
|
712
931
|
collection_id (str): The id of the Collection that the Item belongs to.
|
|
713
|
-
|
|
932
|
+
**kwargs: Additional keyword arguments.
|
|
933
|
+
- refresh (str): Whether to refresh the index after the operation. Can be "true", "false", or "wait_for".
|
|
934
|
+
- refresh (bool): Whether to refresh the index after the operation. Defaults to the value in `self.async_settings.database_refresh`.
|
|
714
935
|
|
|
715
936
|
Raises:
|
|
716
937
|
NotFoundError: If the Item does not exist in the database.
|
|
938
|
+
|
|
939
|
+
Returns:
|
|
940
|
+
None
|
|
717
941
|
"""
|
|
942
|
+
# Ensure kwargs is a dictionary
|
|
943
|
+
kwargs = kwargs or {}
|
|
944
|
+
|
|
945
|
+
# Resolve the `refresh` parameter
|
|
946
|
+
refresh = kwargs.get("refresh", self.async_settings.database_refresh)
|
|
947
|
+
refresh = validate_refresh(refresh)
|
|
948
|
+
|
|
949
|
+
# Log the deletion attempt
|
|
950
|
+
logger.info(
|
|
951
|
+
f"Deleting item {item_id} from collection {collection_id} with refresh={refresh}"
|
|
952
|
+
)
|
|
953
|
+
|
|
718
954
|
try:
|
|
955
|
+
# Perform the delete operation
|
|
719
956
|
await self.client.delete(
|
|
720
957
|
index=index_alias_by_collection_id(collection_id),
|
|
721
958
|
id=mk_item_id(item_id, collection_id),
|
|
722
959
|
refresh=refresh,
|
|
723
960
|
)
|
|
724
961
|
except ESNotFoundError:
|
|
962
|
+
# Raise a custom NotFoundError if the item does not exist
|
|
725
963
|
raise NotFoundError(
|
|
726
964
|
f"Item {item_id} in collection {collection_id} not found"
|
|
727
965
|
)
|
|
@@ -744,24 +982,41 @@ class DatabaseLogic(BaseDatabaseLogic):
|
|
|
744
982
|
except ESNotFoundError:
|
|
745
983
|
raise NotFoundError(f"Mapping for index {index_name} not found")
|
|
746
984
|
|
|
747
|
-
async def create_collection(self, collection: Collection,
|
|
985
|
+
async def create_collection(self, collection: Collection, **kwargs: Any):
|
|
748
986
|
"""Create a single collection in the database.
|
|
749
987
|
|
|
750
988
|
Args:
|
|
751
989
|
collection (Collection): The Collection object to be created.
|
|
752
|
-
|
|
990
|
+
**kwargs: Additional keyword arguments.
|
|
991
|
+
- refresh (str): Whether to refresh the index after the operation. Can be "true", "false", or "wait_for".
|
|
992
|
+
- refresh (bool): Whether to refresh the index after the operation. Defaults to the value in `self.async_settings.database_refresh`.
|
|
753
993
|
|
|
754
994
|
Raises:
|
|
755
995
|
ConflictError: If a Collection with the same id already exists in the database.
|
|
756
996
|
|
|
997
|
+
Returns:
|
|
998
|
+
None
|
|
999
|
+
|
|
757
1000
|
Notes:
|
|
758
1001
|
A new index is created for the items in the Collection using the `create_item_index` function.
|
|
759
1002
|
"""
|
|
760
1003
|
collection_id = collection["id"]
|
|
761
1004
|
|
|
1005
|
+
# Ensure kwargs is a dictionary
|
|
1006
|
+
kwargs = kwargs or {}
|
|
1007
|
+
|
|
1008
|
+
# Resolve the `refresh` parameter
|
|
1009
|
+
refresh = kwargs.get("refresh", self.async_settings.database_refresh)
|
|
1010
|
+
refresh = validate_refresh(refresh)
|
|
1011
|
+
|
|
1012
|
+
# Log the creation attempt
|
|
1013
|
+
logger.info(f"Creating collection {collection_id} with refresh={refresh}")
|
|
1014
|
+
|
|
1015
|
+
# Check if the collection already exists
|
|
762
1016
|
if await self.client.exists(index=COLLECTIONS_INDEX, id=collection_id):
|
|
763
1017
|
raise ConflictError(f"Collection {collection_id} already exists")
|
|
764
1018
|
|
|
1019
|
+
# Index the collection in the database
|
|
765
1020
|
await self.client.index(
|
|
766
1021
|
index=COLLECTIONS_INDEX,
|
|
767
1022
|
id=collection_id,
|
|
@@ -769,6 +1024,7 @@ class DatabaseLogic(BaseDatabaseLogic):
|
|
|
769
1024
|
refresh=refresh,
|
|
770
1025
|
)
|
|
771
1026
|
|
|
1027
|
+
# Create the item index for the collection
|
|
772
1028
|
await create_item_index(collection_id)
|
|
773
1029
|
|
|
774
1030
|
async def find_collection(self, collection_id: str) -> Collection:
|
|
@@ -798,29 +1054,52 @@ class DatabaseLogic(BaseDatabaseLogic):
|
|
|
798
1054
|
return collection["_source"]
|
|
799
1055
|
|
|
800
1056
|
async def update_collection(
|
|
801
|
-
self, collection_id: str, collection: Collection,
|
|
1057
|
+
self, collection_id: str, collection: Collection, **kwargs: Any
|
|
802
1058
|
):
|
|
803
|
-
"""Update a collection
|
|
1059
|
+
"""Update a collection in the database.
|
|
804
1060
|
|
|
805
1061
|
Args:
|
|
806
|
-
self: The instance of the object calling this function.
|
|
807
1062
|
collection_id (str): The ID of the collection to be updated.
|
|
808
1063
|
collection (Collection): The Collection object to be used for the update.
|
|
1064
|
+
**kwargs: Additional keyword arguments.
|
|
1065
|
+
- refresh (str): Whether to refresh the index after the operation. Can be "true", "false", or "wait_for".
|
|
1066
|
+
- refresh (bool): Whether to refresh the index after the operation. Defaults to the value in `self.async_settings.database_refresh`.
|
|
1067
|
+
Returns:
|
|
1068
|
+
None
|
|
809
1069
|
|
|
810
1070
|
Raises:
|
|
811
|
-
NotFoundError: If the collection with the given `collection_id` is not
|
|
812
|
-
|
|
1071
|
+
NotFoundError: If the collection with the given `collection_id` is not found in the database.
|
|
1072
|
+
ConflictError: If a conflict occurs during the update.
|
|
813
1073
|
|
|
814
1074
|
Notes:
|
|
815
1075
|
This function updates the collection in the database using the specified
|
|
816
|
-
`collection_id` and
|
|
817
|
-
|
|
1076
|
+
`collection_id` and the provided `Collection` object. If the collection ID
|
|
1077
|
+
changes, the function creates a new collection, reindexes the items, and deletes
|
|
1078
|
+
the old collection.
|
|
818
1079
|
"""
|
|
1080
|
+
# Ensure kwargs is a dictionary
|
|
1081
|
+
kwargs = kwargs or {}
|
|
1082
|
+
|
|
1083
|
+
# Resolve the `refresh` parameter
|
|
1084
|
+
refresh = kwargs.get("refresh", self.async_settings.database_refresh)
|
|
1085
|
+
refresh = validate_refresh(refresh)
|
|
1086
|
+
|
|
1087
|
+
# Log the update attempt
|
|
1088
|
+
logger.info(f"Updating collection {collection_id} with refresh={refresh}")
|
|
1089
|
+
|
|
1090
|
+
# Ensure the collection exists
|
|
819
1091
|
await self.find_collection(collection_id=collection_id)
|
|
820
1092
|
|
|
1093
|
+
# Handle collection ID change
|
|
821
1094
|
if collection_id != collection["id"]:
|
|
1095
|
+
logger.info(
|
|
1096
|
+
f"Collection ID change detected: {collection_id} -> {collection['id']}"
|
|
1097
|
+
)
|
|
1098
|
+
|
|
1099
|
+
# Create the new collection
|
|
822
1100
|
await self.create_collection(collection, refresh=refresh)
|
|
823
1101
|
|
|
1102
|
+
# Reindex items from the old collection to the new collection
|
|
824
1103
|
await self.client.reindex(
|
|
825
1104
|
body={
|
|
826
1105
|
"dest": {"index": f"{ITEMS_INDEX_PREFIX}{collection['id']}"},
|
|
@@ -834,9 +1113,11 @@ class DatabaseLogic(BaseDatabaseLogic):
|
|
|
834
1113
|
refresh=refresh,
|
|
835
1114
|
)
|
|
836
1115
|
|
|
1116
|
+
# Delete the old collection
|
|
837
1117
|
await self.delete_collection(collection_id)
|
|
838
1118
|
|
|
839
1119
|
else:
|
|
1120
|
+
# Update the existing collection
|
|
840
1121
|
await self.client.index(
|
|
841
1122
|
index=COLLECTIONS_INDEX,
|
|
842
1123
|
id=collection_id,
|
|
@@ -844,76 +1125,184 @@ class DatabaseLogic(BaseDatabaseLogic):
|
|
|
844
1125
|
refresh=refresh,
|
|
845
1126
|
)
|
|
846
1127
|
|
|
847
|
-
async def delete_collection(self, collection_id: str,
|
|
1128
|
+
async def delete_collection(self, collection_id: str, **kwargs: Any):
|
|
848
1129
|
"""Delete a collection from the database.
|
|
849
1130
|
|
|
850
1131
|
Parameters:
|
|
851
|
-
self: The instance of the object calling this function.
|
|
852
1132
|
collection_id (str): The ID of the collection to be deleted.
|
|
853
|
-
|
|
1133
|
+
kwargs (Any, optional): Additional keyword arguments, including `refresh`.
|
|
1134
|
+
- refresh (str): Whether to refresh the index after the operation. Can be "true", "false", or "wait_for".
|
|
1135
|
+
- refresh (bool): Whether to refresh the index after the operation. Defaults to the value in `self.async_settings.database_refresh`.
|
|
854
1136
|
|
|
855
1137
|
Raises:
|
|
856
1138
|
NotFoundError: If the collection with the given `collection_id` is not found in the database.
|
|
857
1139
|
|
|
1140
|
+
Returns:
|
|
1141
|
+
None
|
|
1142
|
+
|
|
858
1143
|
Notes:
|
|
859
1144
|
This function first verifies that the collection with the specified `collection_id` exists in the database, and then
|
|
860
|
-
deletes the collection. If `refresh` is set to
|
|
861
|
-
function also calls `delete_item_index` to delete the index for the items in the collection.
|
|
1145
|
+
deletes the collection. If `refresh` is set to "true", "false", or "wait_for", the index is refreshed accordingly after
|
|
1146
|
+
the deletion. Additionally, this function also calls `delete_item_index` to delete the index for the items in the collection.
|
|
862
1147
|
"""
|
|
1148
|
+
# Ensure kwargs is a dictionary
|
|
1149
|
+
kwargs = kwargs or {}
|
|
1150
|
+
|
|
1151
|
+
# Verify that the collection exists
|
|
863
1152
|
await self.find_collection(collection_id=collection_id)
|
|
1153
|
+
|
|
1154
|
+
# Resolve the `refresh` parameter
|
|
1155
|
+
refresh = kwargs.get("refresh", self.async_settings.database_refresh)
|
|
1156
|
+
refresh = validate_refresh(refresh)
|
|
1157
|
+
|
|
1158
|
+
# Log the deletion attempt
|
|
1159
|
+
logger.info(f"Deleting collection {collection_id} with refresh={refresh}")
|
|
1160
|
+
|
|
1161
|
+
# Delete the collection from the database
|
|
864
1162
|
await self.client.delete(
|
|
865
1163
|
index=COLLECTIONS_INDEX, id=collection_id, refresh=refresh
|
|
866
1164
|
)
|
|
867
|
-
|
|
1165
|
+
|
|
1166
|
+
# Delete the item index for the collection
|
|
1167
|
+
try:
|
|
1168
|
+
await delete_item_index(collection_id)
|
|
1169
|
+
except Exception as e:
|
|
1170
|
+
logger.error(
|
|
1171
|
+
f"Failed to delete item index for collection {collection_id}: {e}"
|
|
1172
|
+
)
|
|
868
1173
|
|
|
869
1174
|
async def bulk_async(
|
|
870
|
-
self,
|
|
871
|
-
|
|
872
|
-
|
|
1175
|
+
self,
|
|
1176
|
+
collection_id: str,
|
|
1177
|
+
processed_items: List[Item],
|
|
1178
|
+
**kwargs: Any,
|
|
1179
|
+
) -> Tuple[int, List[Dict[str, Any]]]:
|
|
1180
|
+
"""
|
|
1181
|
+
Perform a bulk insert of items into the database asynchronously.
|
|
873
1182
|
|
|
874
1183
|
Args:
|
|
875
|
-
self: The instance of the object calling this function.
|
|
876
1184
|
collection_id (str): The ID of the collection to which the items belong.
|
|
877
1185
|
processed_items (List[Item]): A list of `Item` objects to be inserted into the database.
|
|
878
|
-
|
|
1186
|
+
**kwargs (Any): Additional keyword arguments, including:
|
|
1187
|
+
- refresh (str, optional): Whether to refresh the index after the bulk insert.
|
|
1188
|
+
Can be "true", "false", or "wait_for". Defaults to the value of `self.sync_settings.database_refresh`.
|
|
1189
|
+
- refresh (bool, optional): Whether to refresh the index after the bulk insert.
|
|
1190
|
+
- raise_on_error (bool, optional): Whether to raise an error if any of the bulk operations fail.
|
|
1191
|
+
Defaults to the value of `self.async_settings.raise_on_bulk_error`.
|
|
1192
|
+
|
|
1193
|
+
Returns:
|
|
1194
|
+
Tuple[int, List[Dict[str, Any]]]: A tuple containing:
|
|
1195
|
+
- The number of successfully processed actions (`success`).
|
|
1196
|
+
- A list of errors encountered during the bulk operation (`errors`).
|
|
879
1197
|
|
|
880
1198
|
Notes:
|
|
881
|
-
This function performs a bulk insert of `processed_items` into the database using the specified `collection_id`.
|
|
882
|
-
insert is performed
|
|
883
|
-
`mk_actions` function is called to generate a list of actions for the bulk insert.
|
|
884
|
-
index is refreshed after the bulk insert
|
|
1199
|
+
This function performs a bulk insert of `processed_items` into the database using the specified `collection_id`.
|
|
1200
|
+
The insert is performed synchronously and blocking, meaning that the function does not return until the insert has
|
|
1201
|
+
completed. The `mk_actions` function is called to generate a list of actions for the bulk insert. The `refresh`
|
|
1202
|
+
parameter determines whether the index is refreshed after the bulk insert:
|
|
1203
|
+
- "true": Forces an immediate refresh of the index.
|
|
1204
|
+
- "false": Does not refresh the index immediately (default behavior).
|
|
1205
|
+
- "wait_for": Waits for the next refresh cycle to make the changes visible.
|
|
885
1206
|
"""
|
|
886
|
-
|
|
1207
|
+
# Ensure kwargs is a dictionary
|
|
1208
|
+
kwargs = kwargs or {}
|
|
1209
|
+
|
|
1210
|
+
# Resolve the `refresh` parameter
|
|
1211
|
+
refresh = kwargs.get("refresh", self.async_settings.database_refresh)
|
|
1212
|
+
refresh = validate_refresh(refresh)
|
|
1213
|
+
|
|
1214
|
+
# Log the bulk insert attempt
|
|
1215
|
+
logger.info(
|
|
1216
|
+
f"Performing bulk insert for collection {collection_id} with refresh={refresh}"
|
|
1217
|
+
)
|
|
1218
|
+
|
|
1219
|
+
# Handle empty processed_items
|
|
1220
|
+
if not processed_items:
|
|
1221
|
+
logger.warning(f"No items to insert for collection {collection_id}")
|
|
1222
|
+
return 0, []
|
|
1223
|
+
|
|
1224
|
+
# Perform the bulk insert
|
|
1225
|
+
raise_on_error = self.async_settings.raise_on_bulk_error
|
|
1226
|
+
success, errors = await helpers.async_bulk(
|
|
887
1227
|
self.client,
|
|
888
1228
|
mk_actions(collection_id, processed_items),
|
|
889
1229
|
refresh=refresh,
|
|
890
|
-
raise_on_error=
|
|
1230
|
+
raise_on_error=raise_on_error,
|
|
891
1231
|
)
|
|
892
1232
|
|
|
1233
|
+
# Log the result
|
|
1234
|
+
logger.info(
|
|
1235
|
+
f"Bulk insert completed for collection {collection_id}: {success} successes, {len(errors)} errors"
|
|
1236
|
+
)
|
|
1237
|
+
|
|
1238
|
+
return success, errors
|
|
1239
|
+
|
|
893
1240
|
def bulk_sync(
|
|
894
|
-
self,
|
|
895
|
-
|
|
896
|
-
|
|
1241
|
+
self,
|
|
1242
|
+
collection_id: str,
|
|
1243
|
+
processed_items: List[Item],
|
|
1244
|
+
**kwargs: Any,
|
|
1245
|
+
) -> Tuple[int, List[Dict[str, Any]]]:
|
|
1246
|
+
"""
|
|
1247
|
+
Perform a bulk insert of items into the database synchronously.
|
|
897
1248
|
|
|
898
1249
|
Args:
|
|
899
|
-
self: The instance of the object calling this function.
|
|
900
1250
|
collection_id (str): The ID of the collection to which the items belong.
|
|
901
1251
|
processed_items (List[Item]): A list of `Item` objects to be inserted into the database.
|
|
902
|
-
|
|
1252
|
+
**kwargs (Any): Additional keyword arguments, including:
|
|
1253
|
+
- refresh (str, optional): Whether to refresh the index after the bulk insert.
|
|
1254
|
+
Can be "true", "false", or "wait_for". Defaults to the value of `self.sync_settings.database_refresh`.
|
|
1255
|
+
- refresh (bool, optional): Whether to refresh the index after the bulk insert.
|
|
1256
|
+
- raise_on_error (bool, optional): Whether to raise an error if any of the bulk operations fail.
|
|
1257
|
+
Defaults to the value of `self.async_settings.raise_on_bulk_error`.
|
|
1258
|
+
|
|
1259
|
+
Returns:
|
|
1260
|
+
Tuple[int, List[Dict[str, Any]]]: A tuple containing:
|
|
1261
|
+
- The number of successfully processed actions (`success`).
|
|
1262
|
+
- A list of errors encountered during the bulk operation (`errors`).
|
|
903
1263
|
|
|
904
1264
|
Notes:
|
|
905
|
-
This function performs a bulk insert of `processed_items` into the database using the specified `collection_id`.
|
|
906
|
-
insert is performed synchronously and blocking, meaning that the function does not return until the insert has
|
|
907
|
-
completed. The `mk_actions` function is called to generate a list of actions for the bulk insert.
|
|
908
|
-
|
|
1265
|
+
This function performs a bulk insert of `processed_items` into the database using the specified `collection_id`.
|
|
1266
|
+
The insert is performed synchronously and blocking, meaning that the function does not return until the insert has
|
|
1267
|
+
completed. The `mk_actions` function is called to generate a list of actions for the bulk insert. The `refresh`
|
|
1268
|
+
parameter determines whether the index is refreshed after the bulk insert:
|
|
1269
|
+
- "true": Forces an immediate refresh of the index.
|
|
1270
|
+
- "false": Does not refresh the index immediately (default behavior).
|
|
1271
|
+
- "wait_for": Waits for the next refresh cycle to make the changes visible.
|
|
909
1272
|
"""
|
|
910
|
-
|
|
1273
|
+
# Ensure kwargs is a dictionary
|
|
1274
|
+
kwargs = kwargs or {}
|
|
1275
|
+
|
|
1276
|
+
# Resolve the `refresh` parameter
|
|
1277
|
+
refresh = kwargs.get("refresh", self.async_settings.database_refresh)
|
|
1278
|
+
refresh = validate_refresh(refresh)
|
|
1279
|
+
|
|
1280
|
+
# Log the bulk insert attempt
|
|
1281
|
+
logger.info(
|
|
1282
|
+
f"Performing bulk insert for collection {collection_id} with refresh={refresh}"
|
|
1283
|
+
)
|
|
1284
|
+
|
|
1285
|
+
# Handle empty processed_items
|
|
1286
|
+
if not processed_items:
|
|
1287
|
+
logger.warning(f"No items to insert for collection {collection_id}")
|
|
1288
|
+
return 0, []
|
|
1289
|
+
|
|
1290
|
+
# Perform the bulk insert
|
|
1291
|
+
raise_on_error = self.sync_settings.raise_on_bulk_error
|
|
1292
|
+
success, errors = helpers.bulk(
|
|
911
1293
|
self.sync_client,
|
|
912
1294
|
mk_actions(collection_id, processed_items),
|
|
913
1295
|
refresh=refresh,
|
|
914
|
-
raise_on_error=
|
|
1296
|
+
raise_on_error=raise_on_error,
|
|
1297
|
+
)
|
|
1298
|
+
|
|
1299
|
+
# Log the result
|
|
1300
|
+
logger.info(
|
|
1301
|
+
f"Bulk insert completed for collection {collection_id}: {success} successes, {len(errors)} errors"
|
|
915
1302
|
)
|
|
916
1303
|
|
|
1304
|
+
return success, errors
|
|
1305
|
+
|
|
917
1306
|
# DANGER
|
|
918
1307
|
async def delete_items(self) -> None:
|
|
919
1308
|
"""Danger. this is only for tests."""
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.1
|
|
2
2
|
Name: stac-fastapi-elasticsearch
|
|
3
|
-
Version: 4.0
|
|
3
|
+
Version: 4.2.0
|
|
4
4
|
Summary: An implementation of STAC API based on the FastAPI framework with both Elasticsearch and Opensearch.
|
|
5
5
|
Home-page: https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch
|
|
6
6
|
License: MIT
|
|
@@ -125,6 +125,7 @@ You can customize additional settings in your `.env` file:
|
|
|
125
125
|
| `STAC_FASTAPI_TITLE` | Title of the API in the documentation. | `stac-fastapi-elasticsearch` or `stac-fastapi-opensearch` | Optional |
|
|
126
126
|
| `STAC_FASTAPI_DESCRIPTION` | Description of the API in the documentation. | N/A | Optional |
|
|
127
127
|
| `STAC_FASTAPI_VERSION` | API version. | `2.1` | Optional |
|
|
128
|
+
| `STAC_FASTAPI_LANDING_PAGE_ID` | Landing page ID | `stac-fastapi` | Optional |
|
|
128
129
|
| `APP_HOST` | Server bind address. | `0.0.0.0` | Optional |
|
|
129
130
|
| `APP_PORT` | Server port. | `8080` | Optional |
|
|
130
131
|
| `ENVIRONMENT` | Runtime environment. | `local` | Optional |
|
|
@@ -132,9 +133,12 @@ You can customize additional settings in your `.env` file:
|
|
|
132
133
|
| `RELOAD` | Enable auto-reload for development. | `true` | Optional |
|
|
133
134
|
| `STAC_FASTAPI_RATE_LIMIT` | API rate limit per client. | `200/minute` | Optional |
|
|
134
135
|
| `BACKEND` | Tests-related variable | `elasticsearch` or `opensearch` based on the backend | Optional |
|
|
135
|
-
| `ELASTICSEARCH_VERSION` | Version of Elasticsearch to use. | `8.11.0` | Optional |
|
|
136
|
-
| `
|
|
137
|
-
| `
|
|
136
|
+
| `ELASTICSEARCH_VERSION` | Version of Elasticsearch to use. | `8.11.0` | Optional | |
|
|
137
|
+
| `OPENSEARCH_VERSION` | OpenSearch version | `2.11.1` | Optional
|
|
138
|
+
| `ENABLE_DIRECT_RESPONSE` | Enable direct response for maximum performance (disables all FastAPI dependencies, including authentication, custom status codes, and validation) | `false` | Optional
|
|
139
|
+
| `RAISE_ON_BULK_ERROR` | Controls whether bulk insert operations raise exceptions on errors. If set to `true`, the operation will stop and raise an exception when an error occurs. If set to `false`, errors will be logged, and the operation will continue. **Note:** STAC Item and ItemCollection validation errors will always raise, regardless of this flag. | `false` Optional |
|
|
140
|
+
| `DATABASE_REFRESH` | Controls whether database operations refresh the index immediately after changes. If set to `true`, changes will be immediately searchable. If set to `false`, changes may not be immediately visible but can improve performance for bulk operations. If set to `wait_for`, changes will wait for the next refresh cycle to become visible. | `false` | Optional |
|
|
141
|
+
| `ENABLE_TRANSACTIONS_EXTENSIONS` | Enables or disables the Transactions and Bulk Transactions API extensions. If set to `false`, the POST `/collections` route and related transaction endpoints (including bulk transaction operations) will be unavailable in the API. This is useful for deployments where mutating the catalog via the API should be prevented. | `true` | Optional |
|
|
138
142
|
|
|
139
143
|
> [!NOTE]
|
|
140
144
|
> The variables `ES_HOST`, `ES_PORT`, `ES_USE_SSL`, and `ES_VERIFY_CERTS` apply to both Elasticsearch and OpenSearch backends, so there is no need to rename the key names to `OS_` even if you're using OpenSearch.
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|