sfeos-helpers 6.1.0__py3-none-any.whl → 6.2.1__py3-none-any.whl

{sfeos_helpers-6.1.0.dist-info → sfeos_helpers-6.2.1.dist-info}/METADATA

@@ -1,6 +1,6 @@
-Metadata-Version: 2.1
-Name: sfeos-helpers
-Version: 6.1.0
+Metadata-Version: 2.4
+Name: sfeos_helpers
+Version: 6.2.1
 Summary: Helper library for the Elasticsearch and Opensearch stac-fastapi backends.
 Home-page: https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch
 License: MIT
@@ -15,7 +15,15 @@ Classifier: Programming Language :: Python :: 3.13
 Classifier: License :: OSI Approved :: MIT License
 Requires-Python: >=3.9
 Description-Content-Type: text/markdown
-Requires-Dist: stac-fastapi.core==6.1.0
+Requires-Dist: stac-fastapi.core==6.2.1
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
 
 # stac-fastapi-elasticsearch-opensearch
 
@@ -104,6 +112,7 @@ This project is built on the following technologies: STAC, stac-fastapi, FastAPI
 - [Auth](#auth)
 - [Aggregation](#aggregation)
 - [Rate Limiting](#rate-limiting)
+- [Datetime-Based Index Management](#datetime-based-index-management)
 
 ## Documentation & Resources
 
@@ -245,10 +254,86 @@ You can customize additional settings in your `.env` file:
 | `RAISE_ON_BULK_ERROR`        | Controls whether bulk insert operations raise exceptions on errors. If set to `true`, the operation will stop and raise an exception when an error occurs. If set to `false`, errors will be logged, and the operation will continue. **Note:** STAC Item and ItemCollection validation errors will always raise, regardless of this flag. | `false` | Optional |
 | `DATABASE_REFRESH`           | Controls whether database operations refresh the index immediately after changes. If set to `true`, changes will be immediately searchable. If set to `false`, changes may not be immediately visible but can improve performance for bulk operations. If set to `wait_for`, changes will wait for the next refresh cycle to become visible. | `false` | Optional |
 | `ENABLE_TRANSACTIONS_EXTENSIONS` | Enables or disables the Transactions and Bulk Transactions API extensions. If set to `false`, the POST `/collections` route and related transaction endpoints (including bulk transaction operations) will be unavailable in the API. This is useful for deployments where mutating the catalog via the API should be prevented. | `true` | Optional |
+| `STAC_ITEM_LIMIT` | Sets the environment variable for result limiting to SFEOS for the number of returned items and STAC collections. | `10` | Optional |
 
 > [!NOTE]
 > The variables `ES_HOST`, `ES_PORT`, `ES_USE_SSL`, `ES_VERIFY_CERTS` and `ES_TIMEOUT` apply to both Elasticsearch and OpenSearch backends, so there is no need to rename the key names to `OS_` even if you're using OpenSearch.
 
+## Datetime-Based Index Management
+
+### Overview
+
+SFEOS supports two indexing strategies for managing STAC items:
+
+1. **Simple Indexing** (default) - One index per collection
+2. **Datetime-Based Indexing** - Time-partitioned indexes with automatic management
+
+The datetime-based indexing strategy is particularly useful for large temporal datasets. When a user provides a datetime parameter in a query, the system knows exactly which index to search, providing **multiple times faster searches** and significantly **reducing database load**.
+
+### When to Use
+
+**Recommended for:**
+- Systems with large collections containing millions of items
+- Systems requiring high-performance temporal searching
+
+**Pros:**
+- Multiple times faster queries with datetime filter
+- Reduced database load - only relevant indexes are searched
+
+**Cons:**
+- Slightly longer item indexing time (automatic index management)
+- Greater management complexity
+
+### Configuration
+
+#### Enabling Datetime-Based Indexing
+
+Enable datetime-based indexing by setting the following environment variable:
+
+```bash
+ENABLE_DATETIME_INDEX_FILTERING=true
+```
+
+### Related Configuration Variables
+
+| Variable | Description | Default | Example |
+|----------|-------------|---------|---------|
+| `ENABLE_DATETIME_INDEX_FILTERING` | Enables time-based index partitioning | `false` | `true` |
+| `DATETIME_INDEX_MAX_SIZE_GB` | Maximum size limit for datetime indexes (GB) - note: add +20% to target size due to ES/OS compression | `25` | `50` |
+| `STAC_ITEMS_INDEX_PREFIX` | Prefix for item indexes | `items_` | `stac_items_` |
+
+## How Datetime-Based Indexing Works
+
+### Index and Alias Naming Convention
+
+The system uses a precise naming convention:
+
+**Physical indexes:**
+```
+{ITEMS_INDEX_PREFIX}{collection-id}_{uuid4}
+```
+
+**Aliases:**
+```
+{ITEMS_INDEX_PREFIX}{collection-id}                                  # Main collection alias
+{ITEMS_INDEX_PREFIX}{collection-id}_{start-datetime}                 # Temporal alias
+{ITEMS_INDEX_PREFIX}{collection-id}_{start-datetime}_{end-datetime}  # Closed index alias
+```
+
+**Example:**
+
+*Physical indexes:*
+- `items_sentinel-2-l2a_a1b2c3d4-e5f6-7890-abcd-ef1234567890`
+
+*Aliases:*
+- `items_sentinel-2-l2a` - main collection alias
+- `items_sentinel-2-l2a_2024-01-01` - active alias from January 1, 2024
+- `items_sentinel-2-l2a_2024-01-01_2024-03-15` - closed index alias (reached size limit)
+
+### Index Size Management
+
+**Important - Data Compression:** Elasticsearch and OpenSearch automatically compress data. The configured `DATETIME_INDEX_MAX_SIZE_GB` limit refers to the compressed size on disk. It is recommended to add +20% to the target size to account for compression overhead and metadata.
+
 ## Interacting with the API
 
 - **Creating a Collection**:
@@ -557,4 +642,3 @@ You can customize additional settings in your `.env` file:
   - Ensures fair resource allocation among all clients
   
 - **Examples**: Implementation examples are available in the [examples/rate_limit](examples/rate_limit) directory.
-

sfeos_helpers-6.2.1.dist-info/RECORD

@@ -0,0 +1,32 @@
+stac_fastapi/sfeos_helpers/mappings.py,sha256=z6GJFJUE7bRKF9ODc8_ddkb7JCOokMtj4p2LeaQqrQQ,8237
+stac_fastapi/sfeos_helpers/version.py,sha256=TvDysD5xP1CNnI8XDtkkz5sggBM1uuxjhZCv120q3AU,45
+stac_fastapi/sfeos_helpers/aggregation/__init__.py,sha256=Mym17lFh90by1GnoQgMyIKAqRNJnvCgVSXDYzjBiPQk,1210
+stac_fastapi/sfeos_helpers/aggregation/client.py,sha256=PPUk0kAZnms46FlLGrR5w8wa52vG-dT6BG37896R5CY,17939
+stac_fastapi/sfeos_helpers/aggregation/format.py,sha256=qUW1jjh2EEjy-V7riliFR77grpi-AgsTmP76z60K5Lo,2011
+stac_fastapi/sfeos_helpers/database/__init__.py,sha256=T0YwePfhG3ukL1oUFCh3FYHA9jZZe36FJRYCQplfb18,2645
+stac_fastapi/sfeos_helpers/database/datetime.py,sha256=XMyi9Q09cuP_hj97qbGbHFtelq7WQVPdehUfzqNZFV4,4040
+stac_fastapi/sfeos_helpers/database/document.py,sha256=LtjX15gvaOuZC_k2t_oQhys_c-zRTLN5rwX0hNJkHnM,1725
+stac_fastapi/sfeos_helpers/database/index.py,sha256=g7_sKfd5XUwq4IhdKRNiasejk045dKlullsdeDSZTq8,6585
+stac_fastapi/sfeos_helpers/database/mapping.py,sha256=4-MSd4xH5wg7yoC4aPjzYMDSEvP026bw4k2TfffMT5E,1387
+stac_fastapi/sfeos_helpers/database/query.py,sha256=g2iGdfgqpx6o8GoQJBMl3AMmqcbSf792qvKWfWipR5w,4193
+stac_fastapi/sfeos_helpers/database/utils.py,sha256=sgkvjhWtQVGwQLeEXgPR6VoWpdbuVV-WJIGFUsVxqYg,8715
+stac_fastapi/sfeos_helpers/filter/__init__.py,sha256=n3zL_MhEGOoxMz1KeijyK_UKiZ0MKPl90zHtYI5RAy8,1557
+stac_fastapi/sfeos_helpers/filter/client.py,sha256=QwjYWXkevoVS7HPtoXfeSzDy-_GJnFhPJtJM49D14oU,4229
+stac_fastapi/sfeos_helpers/filter/cql2.py,sha256=Cg9kRYD9CVkVSyRqOyB5oVXmlyteSn2bw88sqklGpUM,955
+stac_fastapi/sfeos_helpers/filter/transform.py,sha256=1GEWQSp-rbq7_1nDVv1ApDbWxt8DswJWxwaxzV85gj4,4644
+stac_fastapi/sfeos_helpers/models/patch.py,sha256=s5n85ktnH6M2SMqpqyItR8uLxliXmnSTg1WO0QLVsmI,3127
+stac_fastapi/sfeos_helpers/search_engine/__init__.py,sha256=Bi0cAtul3FuLjFceTPtEcaWNBfmUX5vKaqDvbSUAm0o,754
+stac_fastapi/sfeos_helpers/search_engine/base.py,sha256=9KOLW3NjW9PzWQzqLuhIjQU7FOHdDnB3ZNwDq469JZU,1400
+stac_fastapi/sfeos_helpers/search_engine/factory.py,sha256=nPty3L8esypSVIzl5IKfmqQ1hVUIjMQ183Ksistr1bM,1066
+stac_fastapi/sfeos_helpers/search_engine/index_operations.py,sha256=mYt1C2HEdwtslNwRHiZkvYTWQSZDwBBnKo5YJXfxnDo,5565
+stac_fastapi/sfeos_helpers/search_engine/inserters.py,sha256=o-I_4OowMJetMwRFPdq8Oix_DAkMNGBw4fYyoa5W6s0,10562
+stac_fastapi/sfeos_helpers/search_engine/managers.py,sha256=nldomKmw8iQfOxeGZbBRGG_rWk-vB5Hy_cOjJ2e0ArE,6454
+stac_fastapi/sfeos_helpers/search_engine/selection/__init__.py,sha256=qKd4KzZkERwF_yhIeFcjAUnq5vQarr3CuXxE3SWmt6c,441
+stac_fastapi/sfeos_helpers/search_engine/selection/base.py,sha256=106c4FK50cgMmTpPJkWdgbExPkU2yIH4Wq684Ww-fYE,859
+stac_fastapi/sfeos_helpers/search_engine/selection/cache_manager.py,sha256=5yrgf9JA4mgRNMPDKih6xySF8mD724lEWnXhWud7m2c,4039
+stac_fastapi/sfeos_helpers/search_engine/selection/factory.py,sha256=vbgNVCUW2lviePqzpgsPLxp6IEqcX3GHiahqN2oVObA,1305
+stac_fastapi/sfeos_helpers/search_engine/selection/selectors.py,sha256=q83nfCfNfLUqtkHpORwNHNRU9Pa-heeaDIPO0RlHb-8,4779
+sfeos_helpers-6.2.1.dist-info/METADATA,sha256=Q_L1-BW_owsnpv1KVGneU7O0y-eP31OA-QlcHHiIpZw,34629
+sfeos_helpers-6.2.1.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
+sfeos_helpers-6.2.1.dist-info/top_level.txt,sha256=vqn-D9-HsRPTTxy0Vk_KkDmTiMES4owwBQ3ydSZYb2s,13
+sfeos_helpers-6.2.1.dist-info/RECORD,,

{sfeos_helpers-6.1.0.dist-info → sfeos_helpers-6.2.1.dist-info}/WHEEL

@@ -1,5 +1,5 @@
 Wheel-Version: 1.0
-Generator: bdist_wheel (0.45.1)
+Generator: setuptools (80.9.0)
 Root-Is-Purelib: true
 Tag: py3-none-any
 

stac_fastapi/sfeos_helpers/aggregation/client.py

@@ -313,9 +313,11 @@ class EsAsyncBaseAggregationClient(AsyncBaseAggregationClient):
             )
 
         if aggregate_request.datetime:
-            search = self.database.apply_datetime_filter(
-                search=search, interval=aggregate_request.datetime
+            search, datetime_search = self.database.apply_datetime_filter(
+                search=search, datetime=aggregate_request.datetime
             )
+        else:
+            datetime_search = {"gte": None, "lte": None}
 
         if aggregate_request.bbox:
             bbox = aggregate_request.bbox
@@ -414,6 +416,7 @@ class EsAsyncBaseAggregationClient(AsyncBaseAggregationClient):
                 geometry_geohash_grid_precision,
                 geometry_geotile_grid_precision,
                 datetime_frequency_interval,
+                datetime_search,
             )
         except Exception as error:
             if not isinstance(error, IndexError):

stac_fastapi/sfeos_helpers/database/__init__.py

@@ -30,11 +30,12 @@ Function Naming Conventions:
 """
 
 # Re-export all functions for backward compatibility
-from .datetime import return_date
+from .datetime import extract_date, extract_first_date_from_index, return_date
 from .document import mk_actions, mk_item_id
 from .index import (
     create_index_templates_shared,
     delete_item_index_shared,
+    filter_indexes_by_datetime,
     index_alias_by_collection_id,
     index_by_collection_id,
     indices,
@@ -53,6 +54,7 @@ __all__ = [
     "delete_item_index_shared",
     "index_alias_by_collection_id",
     "index_by_collection_id",
+    "filter_indexes_by_datetime",
     "indices",
     # Query operations
     "apply_free_text_filter_shared",
@@ -68,4 +70,6 @@ __all__ = [
     "get_bool_env",
     # Datetime utilities
     "return_date",
+    "extract_date",
+    "extract_first_date_from_index",
 ]

stac_fastapi/sfeos_helpers/database/datetime.py

@@ -4,14 +4,19 @@ This module provides datetime utility functions specifically designed for
 Elasticsearch and OpenSearch query formatting.
 """
 
+import logging
+import re
+from datetime import date
 from datetime import datetime as datetime_type
 from typing import Dict, Optional, Union
 
 from stac_fastapi.types.rfc3339 import DateTimeType
 
+logger = logging.getLogger(__name__)
+
 
 def return_date(
-    interval: Optional[Union[DateTimeType, str]]
+    interval: Optional[Union[DateTimeType, str]],
 ) -> Dict[str, Optional[str]]:
     """
     Convert a date interval to an Elasticsearch/OpenSearch query format.
@@ -39,8 +44,14 @@ def return_date(
     if isinstance(interval, str):
         if "/" in interval:
             parts = interval.split("/")
-            result["gte"] = parts[0] if parts[0] != ".." else None
-            result["lte"] = parts[1] if len(parts) > 1 and parts[1] != ".." else None
+            result["gte"] = (
+                parts[0] if parts[0] != ".." else datetime_type.min.isoformat() + "Z"
+            )
+            result["lte"] = (
+                parts[1]
+                if len(parts) > 1 and parts[1] != ".."
+                else datetime_type.max.isoformat() + "Z"
+            )
         else:
             converted_time = interval if interval != ".." else None
             result["gte"] = result["lte"] = converted_time
@@ -58,3 +69,53 @@ def return_date(
             result["lte"] = end.strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + "Z"
 
     return result
+
+
+def extract_date(date_str: str) -> date:
+    """Extract date from ISO format string.
+
+    Args:
+        date_str: ISO format date string
+
+    Returns:
+        A date object extracted from the input string.
+    """
+    date_str = date_str.replace("Z", "+00:00")
+    return datetime_type.fromisoformat(date_str).date()
+
+
+def extract_first_date_from_index(index_name: str) -> date:
+    """Extract the first date from an index name containing date patterns.
+
+    Searches for date patterns (YYYY-MM-DD) within the index name string
+    and returns the first found date as a date object.
+
+    Args:
+        index_name: Index name containing date patterns.
+
+    Returns:
+        A date object extracted from the first date pattern found in the index name.
+
+    """
+    date_pattern = r"\d{4}-\d{2}-\d{2}"
+    match = re.search(date_pattern, index_name)
+
+    if not match:
+        logger.error(f"No date pattern found in index name: '{index_name}'")
+        raise ValueError(
+            f"No date pattern (YYYY-MM-DD) found in index name: '{index_name}'"
+        )
+
+    date_string = match.group(0)
+
+    try:
+        extracted_date = datetime_type.strptime(date_string, "%Y-%m-%d").date()
+        return extracted_date
+    except ValueError as e:
+        logger.error(
+            f"Invalid date format found in index name '{index_name}': "
+            f"'{date_string}' - {str(e)}"
+        )
+        raise ValueError(
+            f"Invalid date format in index name '{index_name}': '{date_string}'"
+        ) from e

stac_fastapi/sfeos_helpers/database/index.py

@@ -3,9 +3,13 @@
 This module provides functions for creating and managing indices in Elasticsearch/OpenSearch.
 """
 
+import re
+from datetime import datetime
 from functools import lru_cache
 from typing import Any, List, Optional
 
+from dateutil.parser import parse  # type: ignore[import]
+
 from stac_fastapi.sfeos_helpers.mappings import (
     _ES_INDEX_NAME_UNSUPPORTED_CHARS_TABLE,
     COLLECTIONS_INDEX,
@@ -66,6 +70,59 @@ def indices(collection_ids: Optional[List[str]]) -> str:
     )
 
 
+def filter_indexes_by_datetime(
+    indexes: List[str], gte: Optional[str], lte: Optional[str]
+) -> List[str]:
+    """Filter indexes based on datetime range extracted from index names.
+
+    Args:
+        indexes: List of index names containing dates
+        gte: Greater than or equal date filter (ISO format, optional 'Z' suffix)
+        lte: Less than or equal date filter (ISO format, optional 'Z' suffix)
+
+    Returns:
+        List of filtered index names
+    """
+
+    def parse_datetime(dt_str: str) -> datetime:
+        """Parse datetime string, handling both with and without 'Z' suffix."""
+        return parse(dt_str).replace(tzinfo=None)
+
+    def extract_date_range_from_index(index_name: str) -> tuple:
+        """Extract start and end dates from index name."""
+        date_pattern = r"(\d{4}-\d{2}-\d{2})"
+        dates = re.findall(date_pattern, index_name)
+
+        if len(dates) == 1:
+            start_date = datetime.strptime(dates[0], "%Y-%m-%d")
+            max_date = datetime.max.replace(microsecond=0)
+            return start_date, max_date
+        else:
+            start_date = datetime.strptime(dates[0], "%Y-%m-%d")
+            end_date = datetime.strptime(dates[1], "%Y-%m-%d")
+            return start_date, end_date
+
+    def is_index_in_range(
+        start_date: datetime, end_date: datetime, gte_dt: datetime, lte_dt: datetime
+    ) -> bool:
+        """Check if index date range overlaps with filter range."""
+        return not (
+            end_date.date() < gte_dt.date() or start_date.date() > lte_dt.date()
+        )
+
+    gte_dt = parse_datetime(gte) if gte else datetime.min.replace(microsecond=0)
+    lte_dt = parse_datetime(lte) if lte else datetime.max.replace(microsecond=0)
+
+    filtered_indexes = []
+
+    for index in indexes:
+        start_date, end_date = extract_date_range_from_index(index)
+        if is_index_in_range(start_date, end_date, gte_dt, lte_dt):
+            filtered_indexes.append(index)
+
+    return filtered_indexes
+
+
 async def create_index_templates_shared(settings: Any) -> None:
     """Create index templates for Elasticsearch/OpenSearch Collection and Item indices.
 
@@ -120,11 +177,11 @@ async def delete_item_index_shared(settings: Any, collection_id: str) -> None:
     client = settings.create_client
 
     name = index_alias_by_collection_id(collection_id)
-    resolved = await client.indices.resolve_index(name=name)
+    resolved = await client.indices.resolve_index(name=name, ignore=[404])
     if "aliases" in resolved and resolved["aliases"]:
         [alias] = resolved["aliases"]
         await client.indices.delete_alias(index=alias["indices"], name=alias["name"])
         await client.indices.delete(index=alias["indices"])
     else:
-        await client.indices.delete(index=name)
+        await client.indices.delete(index=name, ignore=[404])
     await client.close()

stac_fastapi/sfeos_helpers/database/query.py

@@ -80,11 +80,14 @@ def populate_sort_shared(sortby: List) -> Optional[Dict[str, Dict[str, str]]]:
         This function transforms a list of sort specifications into the format required by
         Elasticsearch/OpenSearch for sorting query results. The returned dictionary can be
         directly used in search requests.
+        Always includes 'id' as secondary sort to ensure unique pagination tokens.
     """
     if sortby:
-        return {s.field: {"order": s.direction} for s in sortby}
+        sort_config = {s.field: {"order": s.direction} for s in sortby}
+        sort_config.setdefault("id", {"order": "asc"})
+        return sort_config
     else:
-        return None
+        return {"id": {"order": "asc"}}
 
 
 def add_collections_to_body(

stac_fastapi/sfeos_helpers/database/utils.py

@@ -76,7 +76,7 @@ def merge_to_operations(data: Dict) -> List:
             nested_operations = merge_to_operations(value)
 
             for nested_operation in nested_operations:
-                nested_operation.path = f"{key}.{nested_operation.path}"
+                nested_operation.path = f"{key}/{nested_operation.path}"
                 operations.append(nested_operation)
 
         else:
@@ -90,6 +90,7 @@ def check_commands(
     op: str,
     path: ElasticPath,
     from_path: bool = False,
+    create_nest: bool = False,
 ) -> None:
     """Add Elasticsearch checks to operation.
 
@@ -101,25 +102,44 @@ def check_commands(
 
     """
     if path.nest:
-        commands.add(
-            f"if (!ctx._source.containsKey('{path.nest}'))"
-            f"{{Debug.explain('{path.nest} does not exist');}}"
-        )
-
-    if path.index or op in ["remove", "replace", "test"] or from_path:
-        commands.add(
-            f"if (!ctx._source{path.es_nest}.containsKey('{path.key}'))"
-            f"{{Debug.explain('{path.key}  does not exist in {path.nest}');}}"
-        )
-
-    if from_path and path.index is not None:
-        commands.add(
-            f"if ((ctx._source{path.es_location} instanceof ArrayList"
-            f" && ctx._source{path.es_location}.size() < {path.index})"
-            f" || (!(ctx._source{path.es_location} instanceof ArrayList)"
-            f" && !ctx._source{path.es_location}.containsKey('{path.index}')))"
-            f"{{Debug.explain('{path.path} does not exist');}}"
-        )
+        part_nest = ""
+        for index, path_part in enumerate(path.parts):
+
+            # Create nested dictionaries if not present for merge operations
+            if create_nest and not from_path:
+                value = "[:]"
+                for sub_part in reversed(path.parts[index + 1 :]):
+                    value = f"['{sub_part}': {value}]"
+
+                commands.add(
+                    f"if (!ctx._source{part_nest}.containsKey('{path_part}'))"
+                    f"{{ctx._source{part_nest}['{path_part}'] = {value};}}"
+                    f"{'' if index == len(path.parts) - 1 else' else '}"
+                )
+
+            else:
+                commands.add(
+                    f"if (!ctx._source{part_nest}.containsKey('{path_part}'))"
+                    f"{{Debug.explain('{path_part} in {path.path} does not exist');}}"
+                )
+
+            part_nest += f"['{path_part}']"
+
+    if from_path or op in ["remove", "replace", "test"]:
+
+        if isinstance(path.key, int):
+            commands.add(
+                f"if ((ctx._source{path.es_nest} instanceof ArrayList"
+                f" && ctx._source{path.es_nest}.size() < {abs(path.key)})"
+                f" || (!(ctx._source{path.es_nest} instanceof ArrayList)"
+                f" && !ctx._source{path.es_nest}.containsKey('{path.key}')))"
+                f"{{Debug.explain('{path.key} does not exist in {path.nest}');}}"
+            )
+        else:
+            commands.add(
+                f"if (!ctx._source{path.es_nest}.containsKey('{path.key}'))"
+                f"{{Debug.explain('{path.key} does not exist in {path.nest}');}}"
+            )
 
 
 def remove_commands(commands: ESCommandSet, path: ElasticPath) -> None:
@@ -130,15 +150,16 @@ def remove_commands(commands: ESCommandSet, path: ElasticPath) -> None:
         path (ElasticPath): Path to value to be removed
 
     """
-    if path.index is not None:
+    commands.add(f"def {path.variable_name};")
+    if isinstance(path.key, int):
         commands.add(
-            f"def {path.variable_name} = ctx._source{path.es_location}.remove({path.index});"
+            f"if (ctx._source{path.es_nest} instanceof ArrayList)"
+            f"{{{path.variable_name} = ctx._source{path.es_nest}.remove({path.es_key});}} else "
         )
 
-    else:
-        commands.add(
-            f"def {path.variable_name} = ctx._source{path.es_nest}.remove('{path.key}');"
-        )
+    commands.add(
+        f"{path.variable_name} = ctx._source{path.es_nest}.remove('{path.key}');"
+    )
 
 
 def add_commands(
@@ -160,21 +181,22 @@ def add_commands(
         value = (
             from_path.variable_name
             if operation.op == "move"
-            else f"ctx._source.{from_path.es_path}"
+            else f"ctx._source{from_path.es_path}"
         )
+
     else:
         value = f"params.{path.param_key}"
         params[path.param_key] = operation.value
 
-    if path.index is not None:
+    if isinstance(path.key, int):
         commands.add(
-            f"if (ctx._source{path.es_location} instanceof ArrayList)"
-            f"{{ctx._source{path.es_location}.{'add' if operation.op in ['add', 'move'] else 'set'}({path.index}, {value})}}"
-            f"else{{ctx._source.{path.es_path} = {value}}}"
+            f"if (ctx._source{path.es_nest} instanceof ArrayList)"
+            f"{{ctx._source{path.es_nest}.{'add' if operation.op in ['add', 'move'] else 'set'}({path.es_key}, {value});}}"
+            f" else ctx._source{path.es_nest}['{path.es_key}'] = {value};"
         )
 
     else:
-        commands.add(f"ctx._source.{path.es_path} = {value};")
+        commands.add(f"ctx._source{path.es_path} = {value};")
 
 
 def test_commands(
@@ -190,14 +212,23 @@ def test_commands(
     value = f"params.{path.param_key}"
     params[path.param_key] = operation.value
 
+    if isinstance(path.key, int):
+        commands.add(
+            f"if (ctx._source{path.es_nest} instanceof ArrayList)"
+            f"{{if (ctx._source{path.es_nest}[{path.es_key}] != {value})"
+            f"{{Debug.explain('Test failed `{path.path}`"
+            f" != ' + ctx._source{path.es_path});}}"
+            f"}} else "
+        )
+
     commands.add(
-        f"if (ctx._source.{path.es_path} != {value})"
-        f"{{Debug.explain('Test failed `{path.path}` | "
-        f"{operation.json_value} != ' + ctx._source.{path.es_path});}}"
+        f"if (ctx._source{path.es_path} != {value})"
+        f"{{Debug.explain('Test failed `{path.path}`"
+        f" != ' + ctx._source{path.es_path});}}"
     )
 
 
-def operations_to_script(operations: List) -> Dict:
+def operations_to_script(operations: List, create_nest: bool = False) -> Dict:
     """Convert list of operation to painless script.
 
     Args:
@@ -215,10 +246,16 @@ def operations_to_script(operations: List) -> Dict:
             ElasticPath(path=operation.from_) if hasattr(operation, "from_") else None
         )
 
-        check_commands(commands=commands, op=operation.op, path=path)
+        check_commands(
+            commands=commands, op=operation.op, path=path, create_nest=create_nest
+        )
         if from_path is not None:
             check_commands(
-                commands=commands, op=operation.op, path=from_path, from_path=True
+                commands=commands,
+                op=operation.op,
+                path=from_path,
+                from_path=True,
+                create_nest=create_nest,
             )
 
         if operation.op in ["remove", "move"]: