sfeos-helpers 6.2.0__tar.gz → 6.2.1__tar.gz

{sfeos_helpers-6.2.0 → sfeos_helpers-6.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.4
 Name: sfeos_helpers
-Version: 6.2.0
+Version: 6.2.1
 Summary: Helper library for the Elasticsearch and Opensearch stac-fastapi backends.
 Home-page: https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch
 License: MIT
@@ -15,6 +15,15 @@ Classifier: Programming Language :: Python :: 3.13
 Classifier: License :: OSI Approved :: MIT License
 Requires-Python: >=3.9
 Description-Content-Type: text/markdown
+Requires-Dist: stac-fastapi.core==6.2.1
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
 
 # stac-fastapi-elasticsearch-opensearch
 
@@ -245,6 +254,7 @@ You can customize additional settings in your `.env` file:
 | `RAISE_ON_BULK_ERROR`        | Controls whether bulk insert operations raise exceptions on errors. If set to `true`, the operation will stop and raise an exception when an error occurs. If set to `false`, errors will be logged, and the operation will continue. **Note:** STAC Item and ItemCollection validation errors will always raise, regardless of this flag. | `false` | Optional |
 | `DATABASE_REFRESH`           | Controls whether database operations refresh the index immediately after changes. If set to `true`, changes will be immediately searchable. If set to `false`, changes may not be immediately visible but can improve performance for bulk operations. If set to `wait_for`, changes will wait for the next refresh cycle to become visible. | `false` | Optional |
 | `ENABLE_TRANSACTIONS_EXTENSIONS` | Enables or disables the Transactions and Bulk Transactions API extensions. If set to `false`, the POST `/collections` route and related transaction endpoints (including bulk transaction operations) will be unavailable in the API. This is useful for deployments where mutating the catalog via the API should be prevented. | `true` | Optional |
+| `STAC_ITEM_LIMIT` | Sets the environment variable for result limiting to SFEOS for the number of returned items and STAC collections. | `10` | Optional |
 
 > [!NOTE]
 > The variables `ES_HOST`, `ES_PORT`, `ES_USE_SSL`, `ES_VERIFY_CERTS` and `ES_TIMEOUT` apply to both Elasticsearch and OpenSearch backends, so there is no need to rename the key names to `OS_` even if you're using OpenSearch.

{sfeos_helpers-6.2.0 → sfeos_helpers-6.2.1}/README.md

@@ -227,6 +227,7 @@ You can customize additional settings in your `.env` file:
 | `RAISE_ON_BULK_ERROR`        | Controls whether bulk insert operations raise exceptions on errors. If set to `true`, the operation will stop and raise an exception when an error occurs. If set to `false`, errors will be logged, and the operation will continue. **Note:** STAC Item and ItemCollection validation errors will always raise, regardless of this flag. | `false` | Optional |
 | `DATABASE_REFRESH`           | Controls whether database operations refresh the index immediately after changes. If set to `true`, changes will be immediately searchable. If set to `false`, changes may not be immediately visible but can improve performance for bulk operations. If set to `wait_for`, changes will wait for the next refresh cycle to become visible. | `false` | Optional |
 | `ENABLE_TRANSACTIONS_EXTENSIONS` | Enables or disables the Transactions and Bulk Transactions API extensions. If set to `false`, the POST `/collections` route and related transaction endpoints (including bulk transaction operations) will be unavailable in the API. This is useful for deployments where mutating the catalog via the API should be prevented. | `true` | Optional |
+| `STAC_ITEM_LIMIT` | Sets the environment variable for result limiting to SFEOS for the number of returned items and STAC collections. | `10` | Optional |
 
 > [!NOTE]
 > The variables `ES_HOST`, `ES_PORT`, `ES_USE_SSL`, `ES_VERIFY_CERTS` and `ES_TIMEOUT` apply to both Elasticsearch and OpenSearch backends, so there is no need to rename the key names to `OS_` even if you're using OpenSearch.

{sfeos_helpers-6.2.0 → sfeos_helpers-6.2.1}/setup.py

@@ -6,7 +6,7 @@ with open("README.md") as f:
     desc = f.read()
 
 install_requires = [
-    "stac-fastapi.core==6.2.0",
+    "stac-fastapi.core==6.2.1",
 ]
 
 setup(

{sfeos_helpers-6.2.0 → sfeos_helpers-6.2.1}/sfeos_helpers.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.1
-Name: sfeos-helpers
-Version: 6.2.0
+Metadata-Version: 2.4
+Name: sfeos_helpers
+Version: 6.2.1
 Summary: Helper library for the Elasticsearch and Opensearch stac-fastapi backends.
 Home-page: https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch
 License: MIT
@@ -15,6 +15,15 @@ Classifier: Programming Language :: Python :: 3.13
 Classifier: License :: OSI Approved :: MIT License
 Requires-Python: >=3.9
 Description-Content-Type: text/markdown
+Requires-Dist: stac-fastapi.core==6.2.1
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
 
 # stac-fastapi-elasticsearch-opensearch
 
@@ -245,6 +254,7 @@ You can customize additional settings in your `.env` file:
 | `RAISE_ON_BULK_ERROR`        | Controls whether bulk insert operations raise exceptions on errors. If set to `true`, the operation will stop and raise an exception when an error occurs. If set to `false`, errors will be logged, and the operation will continue. **Note:** STAC Item and ItemCollection validation errors will always raise, regardless of this flag. | `false` | Optional |
 | `DATABASE_REFRESH`           | Controls whether database operations refresh the index immediately after changes. If set to `true`, changes will be immediately searchable. If set to `false`, changes may not be immediately visible but can improve performance for bulk operations. If set to `wait_for`, changes will wait for the next refresh cycle to become visible. | `false` | Optional |
 | `ENABLE_TRANSACTIONS_EXTENSIONS` | Enables or disables the Transactions and Bulk Transactions API extensions. If set to `false`, the POST `/collections` route and related transaction endpoints (including bulk transaction operations) will be unavailable in the API. This is useful for deployments where mutating the catalog via the API should be prevented. | `true` | Optional |
+| `STAC_ITEM_LIMIT` | Sets the environment variable for result limiting to SFEOS for the number of returned items and STAC collections. | `10` | Optional |
 
 > [!NOTE]
 > The variables `ES_HOST`, `ES_PORT`, `ES_USE_SSL`, `ES_VERIFY_CERTS` and `ES_TIMEOUT` apply to both Elasticsearch and OpenSearch backends, so there is no need to rename the key names to `OS_` even if you're using OpenSearch.

sfeos_helpers-6.2.1/sfeos_helpers.egg-info/requires.txt

@@ -0,0 +1 @@
+stac-fastapi.core==6.2.1

{sfeos_helpers-6.2.0 → sfeos_helpers-6.2.1}/sfeos_helpers.egg-info/top_level.txt

@@ -1 +1,2 @@
+dist
 stac_fastapi

{sfeos_helpers-6.2.0 → sfeos_helpers-6.2.1}/stac_fastapi/sfeos_helpers/database/query.py

@@ -80,11 +80,14 @@ def populate_sort_shared(sortby: List) -> Optional[Dict[str, Dict[str, str]]]:
         This function transforms a list of sort specifications into the format required by
         Elasticsearch/OpenSearch for sorting query results. The returned dictionary can be
         directly used in search requests.
+        Always includes 'id' as secondary sort to ensure unique pagination tokens.
     """
     if sortby:
-        return {s.field: {"order": s.direction} for s in sortby}
+        sort_config = {s.field: {"order": s.direction} for s in sortby}
+        sort_config.setdefault("id", {"order": "asc"})
+        return sort_config
     else:
-        return None
+        return {"id": {"order": "asc"}}
 
 
 def add_collections_to_body(

{sfeos_helpers-6.2.0 → sfeos_helpers-6.2.1}/stac_fastapi/sfeos_helpers/database/utils.py

@@ -76,7 +76,7 @@ def merge_to_operations(data: Dict) -> List:
             nested_operations = merge_to_operations(value)
 
             for nested_operation in nested_operations:
-                nested_operation.path = f"{key}.{nested_operation.path}"
+                nested_operation.path = f"{key}/{nested_operation.path}"
                 operations.append(nested_operation)
 
         else:
@@ -90,6 +90,7 @@ def check_commands(
     op: str,
     path: ElasticPath,
     from_path: bool = False,
+    create_nest: bool = False,
 ) -> None:
     """Add Elasticsearch checks to operation.
 
@@ -101,25 +102,44 @@ def check_commands(
 
     """
     if path.nest:
-        commands.add(
-            f"if (!ctx._source.containsKey('{path.nest}'))"
-            f"{{Debug.explain('{path.nest} does not exist');}}"
-        )
-
-    if path.index or op in ["remove", "replace", "test"] or from_path:
-        commands.add(
-            f"if (!ctx._source{path.es_nest}.containsKey('{path.key}'))"
-            f"{{Debug.explain('{path.key}  does not exist in {path.nest}');}}"
-        )
-
-    if from_path and path.index is not None:
-        commands.add(
-            f"if ((ctx._source{path.es_location} instanceof ArrayList"
-            f" && ctx._source{path.es_location}.size() < {path.index})"
-            f" || (!(ctx._source{path.es_location} instanceof ArrayList)"
-            f" && !ctx._source{path.es_location}.containsKey('{path.index}')))"
-            f"{{Debug.explain('{path.path} does not exist');}}"
-        )
+        part_nest = ""
+        for index, path_part in enumerate(path.parts):
+
+            # Create nested dictionaries if not present for merge operations
+            if create_nest and not from_path:
+                value = "[:]"
+                for sub_part in reversed(path.parts[index + 1 :]):
+                    value = f"['{sub_part}': {value}]"
+
+                commands.add(
+                    f"if (!ctx._source{part_nest}.containsKey('{path_part}'))"
+                    f"{{ctx._source{part_nest}['{path_part}'] = {value};}}"
+                    f"{'' if index == len(path.parts) - 1 else' else '}"
+                )
+
+            else:
+                commands.add(
+                    f"if (!ctx._source{part_nest}.containsKey('{path_part}'))"
+                    f"{{Debug.explain('{path_part} in {path.path} does not exist');}}"
+                )
+
+            part_nest += f"['{path_part}']"
+
+    if from_path or op in ["remove", "replace", "test"]:
+
+        if isinstance(path.key, int):
+            commands.add(
+                f"if ((ctx._source{path.es_nest} instanceof ArrayList"
+                f" && ctx._source{path.es_nest}.size() < {abs(path.key)})"
+                f" || (!(ctx._source{path.es_nest} instanceof ArrayList)"
+                f" && !ctx._source{path.es_nest}.containsKey('{path.key}')))"
+                f"{{Debug.explain('{path.key} does not exist in {path.nest}');}}"
+            )
+        else:
+            commands.add(
+                f"if (!ctx._source{path.es_nest}.containsKey('{path.key}'))"
+                f"{{Debug.explain('{path.key} does not exist in {path.nest}');}}"
+            )
 
 
 def remove_commands(commands: ESCommandSet, path: ElasticPath) -> None:
@@ -130,15 +150,16 @@ def remove_commands(commands: ESCommandSet, path: ElasticPath) -> None:
         path (ElasticPath): Path to value to be removed
 
     """
-    if path.index is not None:
+    commands.add(f"def {path.variable_name};")
+    if isinstance(path.key, int):
         commands.add(
-            f"def {path.variable_name} = ctx._source{path.es_location}.remove({path.index});"
+            f"if (ctx._source{path.es_nest} instanceof ArrayList)"
+            f"{{{path.variable_name} = ctx._source{path.es_nest}.remove({path.es_key});}} else "
         )
 
-    else:
-        commands.add(
-            f"def {path.variable_name} = ctx._source{path.es_nest}.remove('{path.key}');"
-        )
+    commands.add(
+        f"{path.variable_name} = ctx._source{path.es_nest}.remove('{path.key}');"
+    )
 
 
 def add_commands(
@@ -160,21 +181,22 @@ def add_commands(
         value = (
             from_path.variable_name
             if operation.op == "move"
-            else f"ctx._source.{from_path.es_path}"
+            else f"ctx._source{from_path.es_path}"
         )
+
     else:
         value = f"params.{path.param_key}"
         params[path.param_key] = operation.value
 
-    if path.index is not None:
+    if isinstance(path.key, int):
         commands.add(
-            f"if (ctx._source{path.es_location} instanceof ArrayList)"
-            f"{{ctx._source{path.es_location}.{'add' if operation.op in ['add', 'move'] else 'set'}({path.index}, {value})}}"
-            f"else{{ctx._source.{path.es_path} = {value}}}"
+            f"if (ctx._source{path.es_nest} instanceof ArrayList)"
+            f"{{ctx._source{path.es_nest}.{'add' if operation.op in ['add', 'move'] else 'set'}({path.es_key}, {value});}}"
+            f" else ctx._source{path.es_nest}['{path.es_key}'] = {value};"
         )
 
     else:
-        commands.add(f"ctx._source.{path.es_path} = {value};")
+        commands.add(f"ctx._source{path.es_path} = {value};")
 
 
 def test_commands(
@@ -190,14 +212,23 @@ def test_commands(
     value = f"params.{path.param_key}"
     params[path.param_key] = operation.value
 
+    if isinstance(path.key, int):
+        commands.add(
+            f"if (ctx._source{path.es_nest} instanceof ArrayList)"
+            f"{{if (ctx._source{path.es_nest}[{path.es_key}] != {value})"
+            f"{{Debug.explain('Test failed `{path.path}`"
+            f" != ' + ctx._source{path.es_path});}}"
+            f"}} else "
+        )
+
     commands.add(
-        f"if (ctx._source.{path.es_path} != {value})"
-        f"{{Debug.explain('Test failed `{path.path}` | "
-        f"{operation.json_value} != ' + ctx._source.{path.es_path});}}"
+        f"if (ctx._source{path.es_path} != {value})"
+        f"{{Debug.explain('Test failed `{path.path}`"
+        f" != ' + ctx._source{path.es_path});}}"
     )
 
 
-def operations_to_script(operations: List) -> Dict:
+def operations_to_script(operations: List, create_nest: bool = False) -> Dict:
     """Convert list of operation to painless script.
 
     Args:
@@ -215,10 +246,16 @@ def operations_to_script(operations: List) -> Dict:
             ElasticPath(path=operation.from_) if hasattr(operation, "from_") else None
         )
 
-        check_commands(commands=commands, op=operation.op, path=path)
+        check_commands(
+            commands=commands, op=operation.op, path=path, create_nest=create_nest
+        )
         if from_path is not None:
             check_commands(
-                commands=commands, op=operation.op, path=from_path, from_path=True
+                commands=commands,
+                op=operation.op,
+                path=from_path,
+                from_path=True,
+                create_nest=create_nest,
             )
 
         if operation.op in ["remove", "move"]:

sfeos_helpers-6.2.1/stac_fastapi/sfeos_helpers/models/patch.py

@@ -0,0 +1,124 @@
+"""patch helpers."""
+
+import re
+from typing import Any, Dict, Optional, Union
+
+from pydantic import BaseModel, model_validator
+
+regex = re.compile(r"([^.' ]*:[^.'[ ]*)\.?")
+replacements = str.maketrans({"/": "", ".": "", ":": "", "[": "", "]": ""})
+
+
+class ESCommandSet:
+    """Uses dictionary keys to behaviour of ordered set.
+
+    Yields:
+        str: Elasticsearch commands
+    """
+
+    dict_: Dict[str, None] = {}
+
+    def __init__(self):
+        """Initialise ESCommandSet instance."""
+        self.dict_ = {}
+
+    def add(self, value: str):
+        """Add command.
+
+        Args:
+            value (str): value to be added
+        """
+        self.dict_[value] = None
+
+    def remove(self, value: str):
+        """Remove command.
+
+        Args:
+            value (str): value to be removed
+        """
+        del self.dict_[value]
+
+    def __iter__(self):
+        """Iterate Elasticsearch commands.
+
+        Yields:
+            str: Elasticsearch command
+        """
+        yield from self.dict_.keys()
+
+
+def to_es(string: str):
+    """Convert patch operation key to Elasticsearch key.
+
+    Args:
+        string (str): string to be converted
+
+    Returns:
+        _type_: converted string
+    """
+    if matches := regex.findall(string):
+        for match in set(matches):
+            string = re.sub(rf"\.?{match}", f"['{match}']", string)
+
+    return string
+
+
+class ElasticPath(BaseModel):
+    """Converts a JSON path to an Elasticsearch path.
+
+    Args:
+        path (str): JSON path to be converted.
+
+    """
+
+    parts: list[str] = []
+
+    path: Optional[str] = None
+    key: Optional[Union[str, int]] = None
+    nest: Optional[str] = None
+
+    es_path: Optional[str] = None
+    es_key: Optional[str] = None
+    es_nest: Optional[str] = None
+
+    variable_name: Optional[str] = None
+    param_key: Optional[str] = None
+
+    class Config:
+        """Class config."""
+
+        frozen = True
+
+    @model_validator(mode="before")
+    @classmethod
+    def validate_model(cls, data: Any):
+        """Set optional fields from JSON path.
+
+        Args:
+            data (Any): input data
+        """
+        data["parts"] = data["path"].lstrip("/").split("/")
+
+        data["key"] = data["parts"].pop(-1)
+        data["nest"] = "/".join(data["parts"])
+        data["path"] = data["nest"] + "/" + data["key"]
+
+        data["es_key"] = data["key"]
+        data["es_nest"] = "".join([f"['{part}']" for part in data["parts"]])
+        data["es_path"] = data["es_nest"] + f"['{data['es_key']}']"
+
+        if data["key"].lstrip("-").isdigit() or data["key"] == "-":
+            data["key"] = -1 if data["key"] == "-" else int(data["key"])
+            data["es_key"] = (
+                f"ctx._source{data['es_nest']}.size() - {-data['key']}"
+                if data["key"] < 0
+                else str(data["key"])
+            )
+            data["es_path"] = data["es_nest"] + f"[{data['es_key']}]"
+
+        data[
+            "variable_name"
+        ] = f"{data['nest'].replace('/','_').replace(':','_')}_{str(data['key']).replace(':','_')}"
+        data["param_key"] = data["path"].translate(replacements)
+
+        return data

{sfeos_helpers-6.2.0 → sfeos_helpers-6.2.1}/stac_fastapi/sfeos_helpers/version.py

@@ -1,2 +1,2 @@
 """library version."""
-__version__ = "6.2.0"
+__version__ = "6.2.1"

sfeos_helpers-6.2.0/sfeos_helpers.egg-info/requires.txt

@@ -1 +0,0 @@
-stac-fastapi.core==6.2.0

sfeos_helpers-6.2.0/stac_fastapi/sfeos_helpers/models/patch.py

@@ -1,166 +0,0 @@
-"""patch helpers."""
-
-import re
-from typing import Any, Dict, Optional, Union
-
-from pydantic import BaseModel, computed_field, model_validator
-
-regex = re.compile(r"([^.' ]*:[^.'[ ]*)\.?")
-replacements = str.maketrans({"/": "", ".": "", ":": "", "[": "", "]": ""})
-
-
-class ESCommandSet:
-    """Uses dictionary keys to behaviour of ordered set.
-
-    Yields:
-        str: Elasticsearch commands
-    """
-
-    dict_: Dict[str, None] = {}
-
-    def __init__(self):
-        """Initialise ESCommandSet instance."""
-        self.dict_ = {}
-
-    def add(self, value: str):
-        """Add command.
-
-        Args:
-            value (str): value to be added
-        """
-        self.dict_[value] = None
-
-    def remove(self, value: str):
-        """Remove command.
-
-        Args:
-            value (str): value to be removed
-        """
-        del self.dict_[value]
-
-    def __iter__(self):
-        """Iterate Elasticsearch commands.
-
-        Yields:
-            str: Elasticsearch command
-        """
-        yield from self.dict_.keys()
-
-
-def to_es(string: str):
-    """Convert patch operation key to Elasticsearch key.
-
-    Args:
-        string (str): string to be converted
-
-    Returns:
-        _type_: converted string
-    """
-    if matches := regex.findall(string):
-        for match in set(matches):
-            string = re.sub(rf"\.?{match}", f"['{match}']", string)
-
-    return string
-
-
-class ElasticPath(BaseModel):
-    """Converts a JSON path to an Elasticsearch path.
-
-    Args:
-        path (str): JSON path to be converted.
-
-    """
-
-    path: str
-    nest: Optional[str] = None
-    partition: Optional[str] = None
-    key: Optional[str] = None
-
-    es_path: Optional[str] = None
-    es_nest: Optional[str] = None
-    es_key: Optional[str] = None
-
-    index_: Optional[int] = None
-
-    @model_validator(mode="before")
-    @classmethod
-    def validate_model(cls, data: Any):
-        """Set optional fields from JSON path.
-
-        Args:
-            data (Any): input data
-        """
-        data["path"] = data["path"].lstrip("/").replace("/", ".")
-        data["nest"], data["partition"], data["key"] = data["path"].rpartition(".")
-
-        if data["key"].lstrip("-").isdigit() or data["key"] == "-":
-            data["index_"] = -1 if data["key"] == "-" else int(data["key"])
-            data["path"] = f"{data['nest']}[{data['index_']}]"
-            data["nest"], data["partition"], data["key"] = data["nest"].rpartition(".")
-
-        data["es_path"] = to_es(data["path"])
-        data["es_nest"] = f".{to_es(data['nest'])}" if data["nest"] else ""
-        data["es_key"] = to_es(data["key"])
-
-        return data
-
-    @computed_field  # type: ignore[misc]
-    @property
-    def index(self) -> Union[int, str, None]:
-        """Compute location of path.
-
-        Returns:
-            str: path index
-        """
-        if self.index_ and self.index_ < 0:
-
-            return f"ctx._source.{self.location}.size() - {-self.index_}"
-
-        return self.index_
-
-    @computed_field  # type: ignore[misc]
-    @property
-    def location(self) -> str:
-        """Compute location of path.
-
-        Returns:
-            str: path location
-        """
-        return self.nest + self.partition + self.key
-
-    @computed_field  # type: ignore[misc]
-    @property
-    def es_location(self) -> str:
-        """Compute location of path.
-
-        Returns:
-            str: path location
-        """
-        if self.es_key and ":" in self.es_key:
-            return self.es_nest + self.es_key
-        return self.es_nest + self.partition + self.es_key
-
-    @computed_field  # type: ignore[misc]
-    @property
-    def variable_name(self) -> str:
-        """Variable name for scripting.
-
-        Returns:
-            str: variable name
-        """
-        if self.index is not None:
-            return f"{self.location.replace('.','_').replace(':','_')}_{self.index}"
-
-        return (
-            f"{self.nest.replace('.','_').replace(':','_')}_{self.key.replace(':','_')}"
-        )
-
-    @computed_field  # type: ignore[misc]
-    @property
-    def param_key(self) -> str:
-        """Param key for scripting.
-
-        Returns:
-            str: param key
-        """
-        return self.path.translate(replacements)