vectordb-bench 0.0.29__tar.gz → 0.0.30__tar.gz

{vectordb_bench-0.0.29 → vectordb_bench-0.0.30}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: vectordb-bench
-Version: 0.0.29
+Version: 0.0.30
 Summary: VectorDBBench is not just an offering of benchmark results for mainstream vector databases and cloud services, it's your go-to tool for the ultimate performance and cost-effectiveness comparison. Designed with ease-of-use in mind, VectorDBBench is devised to help users, even non-professionals, reproduce results or test new systems, making the hunt for the optimal choice amongst a plethora of cloud services and open-source vector databases a breeze.
 Author-email: XuanYang-cn <xuan.yang@zilliz.com>
 Project-URL: repository, https://github.com/zilliztech/VectorDBBench
@@ -295,9 +295,12 @@ Options:
   --force-merge-enabled BOOLEAN   Whether to perform force merge operation
   --flush-threshold-size TEXT     Size threshold for flushing the transaction
                                   log
+  --engine TEXT                   type of engine to use valid values [faiss, lucene]
   # Memory Management
   --cb-threshold TEXT             k-NN Memory circuit breaker threshold
-
+  
+  # Quantization Type
+  --quantization-type TEXT        which type of quantization to use valid values [fp32, fp16]
   --help                          Show this message and exit.
   ```
 
@@ -339,6 +342,49 @@ milvushnsw:
 > - Options passed on the command line will override the configuration file*
 > - Parameter names use an _ not -
 
+#### Using a batch configuration file.
+
+The vectordbbench command can read a batch configuration file to run all the test cases in the yaml formatted configuration file.
+
+By default, configuration files are expected to be in vectordb_bench/config-files/, this can be overridden by setting  
+the environment variable CONFIG_LOCAL_DIR or by passing the full path to the file. 
+
+The required format is:
+```yaml
+commandname:
+  - parameter_name: parameter_value
+    another_parameter_name: parameter_value
+```
+Example:
+```yaml
+pgvectorhnsw:
+  - db_label: pgConfigTest
+    user_name: vectordbbench
+    password: vectordbbench
+    db_name:  vectordbbench
+    host: localhost
+    m: 16
+    ef_construction: 128
+    ef_search: 128
+milvushnsw:
+  - skip_search_serial: True
+    case_type: Performance1536D50K
+    uri: http://localhost:19530
+    m: 16
+    ef_construction: 128
+    ef_search: 128
+    drop_old: False
+    load: False
+```
+> Notes: 
+> - Options can only be passed through configuration files
+> - Parameter names use an _ not -
+
+How to use?
+```shell
+vectordbbench batchcli --batch-config-file <your-yaml-configuration-file>
+```
+
 ## Leaderboard
 ### Introduction
 To facilitate the presentation of test results and provide a comprehensive performance analysis report, we offer a [leaderboard page](https://zilliz.com/benchmark). It allows us to choose from QPS, QP$, and latency metrics, and provides a comprehensive assessment of a system's performance based on the test results of various cases and a set of scoring mechanisms (to be introduced later). On this leaderboard, we can select the systems and models to be compared, and filter out cases we do not want to consider. Comprehensive scores are always ranked from best to worst, and the specific test results of each query will be presented in the list below.

{vectordb_bench-0.0.29 → vectordb_bench-0.0.30}/README.md

@@ -201,9 +201,12 @@ Options:
   --force-merge-enabled BOOLEAN   Whether to perform force merge operation
   --flush-threshold-size TEXT     Size threshold for flushing the transaction
                                   log
+  --engine TEXT                   type of engine to use valid values [faiss, lucene]
   # Memory Management
   --cb-threshold TEXT             k-NN Memory circuit breaker threshold
-
+  
+  # Quantization Type
+  --quantization-type TEXT        which type of quantization to use valid values [fp32, fp16]
   --help                          Show this message and exit.
   ```
 
@@ -245,6 +248,49 @@ milvushnsw:
 > - Options passed on the command line will override the configuration file*
 > - Parameter names use an _ not -
 
+#### Using a batch configuration file.
+
+The vectordbbench command can read a batch configuration file to run all the test cases in the yaml formatted configuration file.
+
+By default, configuration files are expected to be in vectordb_bench/config-files/, this can be overridden by setting  
+the environment variable CONFIG_LOCAL_DIR or by passing the full path to the file. 
+
+The required format is:
+```yaml
+commandname:
+  - parameter_name: parameter_value
+    another_parameter_name: parameter_value
+```
+Example:
+```yaml
+pgvectorhnsw:
+  - db_label: pgConfigTest
+    user_name: vectordbbench
+    password: vectordbbench
+    db_name:  vectordbbench
+    host: localhost
+    m: 16
+    ef_construction: 128
+    ef_search: 128
+milvushnsw:
+  - skip_search_serial: True
+    case_type: Performance1536D50K
+    uri: http://localhost:19530
+    m: 16
+    ef_construction: 128
+    ef_search: 128
+    drop_old: False
+    load: False
+```
+> Notes: 
+> - Options can only be passed through configuration files
+> - Parameter names use an _ not -
+
+How to use?
+```shell
+vectordbbench batchcli --batch-config-file <your-yaml-configuration-file>
+```
+
 ## Leaderboard
 ### Introduction
 To facilitate the presentation of test results and provide a comprehensive performance analysis report, we offer a [leaderboard page](https://zilliz.com/benchmark). It allows us to choose from QPS, QP$, and latency metrics, and provides a comprehensive assessment of a system's performance based on the test results of various cases and a set of scoring mechanisms (to be introduced later). On this leaderboard, we can select the systems and models to be compared, and filter out cases we do not want to consider. Comprehensive scores are always ranked from best to worst, and the specific test results of each query will be presented in the list below.

{vectordb_bench-0.0.29 → vectordb_bench-0.0.30}/vectordb_bench/backend/clients/__init__.py

@@ -27,6 +27,7 @@ class DB(Enum):
     Pinecone = "Pinecone"
     ElasticCloud = "ElasticCloud"
     QdrantCloud = "QdrantCloud"
+    QdrantLocal = "QdrantLocal"
     WeaviateCloud = "WeaviateCloud"
     PgVector = "PgVector"
     PgVectoRS = "PgVectoRS"
@@ -75,6 +76,11 @@ class DB(Enum):
 
             return QdrantCloud
 
+        if self == DB.QdrantLocal:
+            from .qdrant_local.qdrant_local import QdrantLocal
+
+            return QdrantLocal
+
         if self == DB.WeaviateCloud:
             from .weaviate_cloud.weaviate_cloud import WeaviateCloud
 
@@ -201,6 +207,11 @@ class DB(Enum):
 
             return QdrantConfig
 
+        if self == DB.QdrantLocal:
+            from .qdrant_local.config import QdrantLocalConfig
+
+            return QdrantLocalConfig
+
         if self == DB.WeaviateCloud:
             from .weaviate_cloud.config import WeaviateConfig
 
@@ -323,6 +334,11 @@ class DB(Enum):
 
             return QdrantIndexConfig
 
+        if self == DB.QdrantLocal:
+            from .qdrant_local.config import QdrantLocalIndexConfig
+
+            return QdrantLocalIndexConfig
+
         if self == DB.WeaviateCloud:
             from .weaviate_cloud.config import WeaviateIndexConfig
 

{vectordb_bench-0.0.29 → vectordb_bench-0.0.30}/vectordb_bench/backend/clients/aws_opensearch/aws_opensearch.py

@@ -36,6 +36,7 @@ class AWSOpenSearch(VectorDB):
         self.vector_col_name = vector_col_name
 
         log.info(f"AWS_OpenSearch client config: {self.db_config}")
+        log.info(f"AWS_OpenSearch db case config : {self.case_config}")
         client = OpenSearch(**self.db_config)
         if drop_old:
             log.info(f"AWS_OpenSearch client drop old index: {self.index_name}")
@@ -43,6 +44,14 @@ class AWSOpenSearch(VectorDB):
             if is_existed:
                 client.indices.delete(index=self.index_name)
             self._create_index(client)
+        else:
+            is_existed = client.indices.exists(index=self.index_name)
+            if not is_existed:
+                self._create_index(client)
+                log.info(f"AWS_OpenSearch client create index: {self.index_name}")
+
+            self._update_ef_search_before_search(client)
+            self._load_graphs_to_memory(client)
 
     @classmethod
     def config_cls(cls) -> AWSOpenSearchConfig:
@@ -52,7 +61,17 @@ class AWSOpenSearch(VectorDB):
     def case_config_cls(cls, index_type: IndexType | None = None) -> AWSOpenSearchIndexConfig:
         return AWSOpenSearchIndexConfig
 
-    def _create_index(self, client: OpenSearch):
+    def _create_index(self, client: OpenSearch) -> None:
+        ef_search_value = (
+            self.case_config.ef_search if self.case_config.ef_search is not None else self.case_config.efSearch
+        )
+        log.info(f"Creating index with ef_search: {ef_search_value}")
+        log.info(f"Creating index with number_of_replicas: {self.case_config.number_of_replicas}")
+
+        log.info(f"Creating index with engine: {self.case_config.engine}")
+        log.info(f"Creating index with metric type: {self.case_config.metric_type_name}")
+        log.info(f"All case_config parameters: {self.case_config.__dict__}")
+
         cluster_settings_body = {
             "persistent": {
                 "knn.algo_param.index_thread_qty": self.case_config.index_thread_qty,
@@ -64,18 +83,15 @@ class AWSOpenSearch(VectorDB):
             "index": {
                 "knn": True,
                 "number_of_shards": self.case_config.number_of_shards,
-                "number_of_replicas": 0,
+                "number_of_replicas": self.case_config.number_of_replicas,
                 "translog.flush_threshold_size": self.case_config.flush_threshold_size,
-                # Setting trans log threshold to 5GB
-                **(
-                    {"knn.algo_param.ef_search": self.case_config.ef_search}
-                    if self.case_config.engine == AWSOS_Engine.nmslib
-                    else {}
-                ),
+                "knn.advanced.approximate_threshold": "-1",
             },
             "refresh_interval": self.case_config.refresh_interval,
         }
+        settings["index"]["knn.algo_param.ef_search"] = ef_search_value
         mappings = {
+            "_source": {"excludes": [self.vector_col_name], "recovery_source_excludes": [self.vector_col_name]},
             "properties": {
                 **{categoryCol: {"type": "keyword"} for categoryCol in self.category_col_names},
                 self.vector_col_name: {
@@ -86,6 +102,8 @@ class AWSOpenSearch(VectorDB):
             },
         }
         try:
+            log.info(f"Creating index with settings: {settings}")
+            log.info(f"Creating index with mappings: {mappings}")
             client.indices.create(
                 index=self.index_name,
                 body={"settings": settings, "mappings": mappings},
@@ -112,6 +130,18 @@ class AWSOpenSearch(VectorDB):
         """Insert the embeddings to the opensearch."""
         assert self.client is not None, "should self.init() first"
 
+        num_clients = self.case_config.number_of_indexing_clients or 1
+        log.info(f"Number of indexing clients from case_config: {num_clients}")
+
+        if num_clients <= 1:
+            log.info("Using single client for data insertion")
+            return self._insert_with_single_client(embeddings, metadata)
+        log.info(f"Using {num_clients} parallel clients for data insertion")
+        return self._insert_with_multiple_clients(embeddings, metadata, num_clients)
+
+    def _insert_with_single_client(
+        self, embeddings: Iterable[list[float]], metadata: list[int]
+    ) -> tuple[int, Exception]:
         insert_data = []
         for i in range(len(embeddings)):
             insert_data.append(
@@ -129,7 +159,108 @@ class AWSOpenSearch(VectorDB):
         except Exception as e:
             log.warning(f"Failed to insert data: {self.index_name} error: {e!s}")
             time.sleep(10)
-            return self.insert_embeddings(embeddings, metadata)
+            return self._insert_with_single_client(embeddings, metadata)
+
+    def _insert_with_multiple_clients(
+        self, embeddings: Iterable[list[float]], metadata: list[int], num_clients: int
+    ) -> tuple[int, Exception]:
+        import concurrent.futures
+        from concurrent.futures import ThreadPoolExecutor
+
+        embeddings_list = list(embeddings)
+        chunk_size = max(1, len(embeddings_list) // num_clients)
+        chunks = []
+
+        for i in range(0, len(embeddings_list), chunk_size):
+            end = min(i + chunk_size, len(embeddings_list))
+            chunks.append((embeddings_list[i:end], metadata[i:end]))
+
+        clients = []
+        for _ in range(min(num_clients, len(chunks))):
+            client = OpenSearch(**self.db_config)
+            clients.append(client)
+
+        log.info(f"AWS_OpenSearch using {len(clients)} parallel clients for data insertion")
+
+        def insert_chunk(client_idx: int, chunk_idx: int):
+            chunk_embeddings, chunk_metadata = chunks[chunk_idx]
+            client = clients[client_idx]
+
+            insert_data = []
+            for i in range(len(chunk_embeddings)):
+                insert_data.append(
+                    {"index": {"_index": self.index_name, self.id_col_name: chunk_metadata[i]}},
+                )
+                insert_data.append({self.vector_col_name: chunk_embeddings[i]})
+
+            try:
+                resp = client.bulk(insert_data)
+                log.info(f"Client {client_idx} added {len(resp['items'])} documents")
+                return len(chunk_embeddings), None
+            except Exception as e:
+                log.warning(f"Client {client_idx} failed to insert data: {e!s}")
+                return 0, e
+
+        results = []
+        with ThreadPoolExecutor(max_workers=len(clients)) as executor:
+            futures = []
+
+            for chunk_idx in range(len(chunks)):
+                client_idx = chunk_idx % len(clients)
+                futures.append(executor.submit(insert_chunk, client_idx, chunk_idx))
+
+            for future in concurrent.futures.as_completed(futures):
+                count, error = future.result()
+                results.append((count, error))
+
+        from contextlib import suppress
+
+        for client in clients:
+            with suppress(Exception):
+                client.close()
+
+        total_count = sum(count for count, _ in results)
+        errors = [error for _, error in results if error is not None]
+
+        if errors:
+            log.warning("Some clients failed to insert data, retrying with single client")
+            time.sleep(10)
+            return self._insert_with_single_client(embeddings, metadata)
+
+        resp = self.client.indices.stats(self.index_name)
+        log.info(
+            f"""Total document count in index after parallel insertion:
+                {resp['_all']['primaries']['indexing']['index_total']}""",
+        )
+
+        return (total_count, None)
+
+    def _update_ef_search_before_search(self, client: OpenSearch):
+        ef_search_value = (
+            self.case_config.ef_search if self.case_config.ef_search is not None else self.case_config.efSearch
+        )
+
+        try:
+            index_settings = client.indices.get_settings(index=self.index_name)
+            current_ef_search = (
+                index_settings.get(self.index_name, {})
+                .get("settings", {})
+                .get("index", {})
+                .get("knn.algo_param", {})
+                .get("ef_search")
+            )
+
+            if current_ef_search != str(ef_search_value):
+                log.info(f"Updating ef_search before search from {current_ef_search} to {ef_search_value}")
+                settings_body = {"index": {"knn.algo_param.ef_search": ef_search_value}}
+                client.indices.put_settings(index=self.index_name, body=settings_body)
+                log.info(f"Successfully updated ef_search to {ef_search_value} before search")
+
+            log.info(f"Current engine: {self.case_config.engine}")
+            log.info(f"Current metric_type: {self.case_config.metric_type_name}")
+
+        except Exception as e:
+            log.warning(f"Failed to update ef_search parameter before search: {e}")
 
     def search_embedding(
         self,
@@ -151,9 +282,18 @@ class AWSOpenSearch(VectorDB):
 
         body = {
             "size": k,
-            "query": {"knn": {self.vector_col_name: {"vector": query, "k": k}}},
+            "query": {
+                "knn": {
+                    self.vector_col_name: {
+                        "vector": query,
+                        "k": k,
+                        "method_parameters": {"ef_search": self.case_config.efSearch},
+                    }
+                }
+            },
             **({"filter": {"range": {self.id_col_name: {"gt": filters["id"]}}}} if filters else {}),
         }
+
         try:
             resp = self.client.search(
                 index=self.index_name,
@@ -162,6 +302,7 @@ class AWSOpenSearch(VectorDB):
                 _source=False,
                 docvalue_fields=[self.id_col_name],
                 stored_fields="_none_",
+                preference="_only_local" if self.case_config.number_of_shards == 1 else None,
             )
             log.debug(f"Search took: {resp['took']}")
             log.debug(f"Search shards: {resp['_shards']}")
@@ -173,6 +314,7 @@ class AWSOpenSearch(VectorDB):
 
     def optimize(self, data_size: int | None = None):
         """optimize will be called between insertion and search in performance cases."""
+        self._update_ef_search()
         # Call refresh first to ensure that all segments are created
         self._refresh_index()
         if self.case_config.force_merge_enabled:
@@ -182,7 +324,22 @@ class AWSOpenSearch(VectorDB):
         # Call refresh again to ensure that the index is ready after force merge.
         self._refresh_index()
         # ensure that all graphs are loaded in memory and ready for search
-        self._load_graphs_to_memory()
+        self._load_graphs_to_memory(self.client)
+
+    def _update_ef_search(self):
+        ef_search_value = (
+            self.case_config.ef_search if self.case_config.ef_search is not None else self.case_config.efSearch
+        )
+        log.info(f"Updating ef_search parameter to: {ef_search_value}")
+
+        settings_body = {"index": {"knn.algo_param.ef_search": ef_search_value}}
+        try:
+            self.client.indices.put_settings(index=self.index_name, body=settings_body)
+            log.info(f"Successfully updated ef_search to {ef_search_value}")
+            log.info(f"Current engine: {self.case_config.engine}")
+            log.info(f"Current metric_type: {self.case_config.metric_type}")
+        except Exception as e:
+            log.warning(f"Failed to update ef_search parameter: {e}")
 
     def _update_replicas(self):
         index_settings = self.client.indices.get_settings(index=self.index_name)
@@ -200,7 +357,7 @@ class AWSOpenSearch(VectorDB):
         while True:
             res = self.client.cat.indices(index=self.index_name, h="health", format="json")
             health = res[0]["health"]
-            if health != "green":
+            if health == "green":
                 break
             log.info(f"The index {self.index_name} has health : {health} and is not green. Retrying")
             time.sleep(SECONDS_WAITING_FOR_REPLICAS_TO_BE_ENABLED_SEC)
@@ -228,8 +385,16 @@ class AWSOpenSearch(VectorDB):
             "persistent": {"knn.algo_param.index_thread_qty": self.case_config.index_thread_qty_during_force_merge}
         }
         self.client.cluster.put_settings(cluster_settings_body)
+
+        log.info("Updating the graph threshold to ensure that during merge we can do graph creation.")
+        output = self.client.indices.put_settings(
+            index=self.index_name, body={"index.knn.advanced.approximate_threshold": "0"}
+        )
+        log.info(f"response of updating setting is: {output}")
+
         log.debug(f"Starting force merge for index {self.index_name}")
-        force_merge_endpoint = f"/{self.index_name}/_forcemerge?max_num_segments=1&wait_for_completion=false"
+        segments = self.case_config.number_of_segments
+        force_merge_endpoint = f"/{self.index_name}/_forcemerge?max_num_segments={segments}&wait_for_completion=false"
         force_merge_task_id = self.client.transport.perform_request("POST", force_merge_endpoint)["task"]
         while True:
             time.sleep(WAITING_FOR_FORCE_MERGE_SEC)
@@ -238,8 +403,8 @@ class AWSOpenSearch(VectorDB):
                 break
         log.debug(f"Completed force merge for index {self.index_name}")
 
-    def _load_graphs_to_memory(self):
+    def _load_graphs_to_memory(self, client: OpenSearch):
         if self.case_config.engine != AWSOS_Engine.lucene:
             log.info("Calling warmup API to load graphs into memory")
             warmup_endpoint = f"/_plugins/_knn/warmup/{self.index_name}"
-            self.client.transport.perform_request("GET", warmup_endpoint)
+            client.transport.perform_request("GET", warmup_endpoint)

{vectordb_bench-0.0.29 → vectordb_bench-0.0.30}/vectordb_bench/backend/clients/aws_opensearch/cli.py

@@ -1,3 +1,4 @@
+import logging
 from typing import Annotated, TypedDict, Unpack
 
 import click
@@ -5,18 +6,21 @@ from pydantic import SecretStr
 
 from ....cli.cli import (
     CommonTypedDict,
-    HNSWFlavor2,
+    HNSWFlavor1,
     cli,
     click_parameter_decorators_from_typed_dict,
     run,
 )
 from .. import DB
+from .config import AWSOS_Engine, AWSOSQuantization
+
+log = logging.getLogger(__name__)
 
 
 class AWSOpenSearchTypedDict(TypedDict):
     host: Annotated[str, click.option("--host", type=str, help="Db host", required=True)]
-    port: Annotated[int, click.option("--port", type=int, default=443, help="Db Port")]
-    user: Annotated[str, click.option("--user", type=str, default="admin", help="Db User")]
+    port: Annotated[int, click.option("--port", type=int, default=80, help="Db Port")]
+    user: Annotated[str, click.option("--user", type=str, help="Db User")]
     password: Annotated[str, click.option("--password", type=str, help="Db password")]
     number_of_shards: Annotated[
         int,
@@ -38,23 +42,23 @@ class AWSOpenSearchTypedDict(TypedDict):
         ),
     ]
 
-    index_thread_qty_during_force_merge: Annotated[
-        int,
+    engine: Annotated[
+        str,
         click.option(
-            "--index-thread-qty-during-force-merge",
-            type=int,
-            help="Thread count during force merge operations",
-            default=4,
+            "--engine",
+            type=click.Choice(["nmslib", "faiss", "lucene"], case_sensitive=False),
+            help="HNSW algorithm implementation to use",
+            default="faiss",
         ),
     ]
 
-    number_of_indexing_clients: Annotated[
-        int,
+    metric_type: Annotated[
+        str,
         click.option(
-            "--number-of-indexing-clients",
-            type=int,
-            help="Number of concurrent indexing clients",
-            default=1,
+            "--metric-type",
+            type=click.Choice(["l2", "cosine", "ip"], case_sensitive=False),
+            help="Distance metric type for vector similarity",
+            default="l2",
         ),
     ]
 
@@ -64,26 +68,26 @@ class AWSOpenSearchTypedDict(TypedDict):
     ]
 
     refresh_interval: Annotated[
-        int,
+        str,
         click.option(
             "--refresh-interval", type=str, help="How often to make new data available for search", default="60s"
         ),
     ]
 
     force_merge_enabled: Annotated[
-        int,
+        bool,
         click.option("--force-merge-enabled", type=bool, help="Whether to perform force merge operation", default=True),
     ]
 
     flush_threshold_size: Annotated[
-        int,
+        str,
         click.option(
             "--flush-threshold-size", type=str, help="Size threshold for flushing the transaction log", default="5120mb"
         ),
     ]
 
     cb_threshold: Annotated[
-        int,
+        str,
         click.option(
             "--cb-threshold",
             type=str,
@@ -92,8 +96,30 @@ class AWSOpenSearchTypedDict(TypedDict):
         ),
     ]
 
+    quantization_type: Annotated[
+        str | None,
+        click.option(
+            "--quantization-type",
+            type=click.Choice(["fp32", "fp16"]),
+            help="quantization type for vectors (in index)",
+            default="fp32",
+            required=False,
+        ),
+    ]
+
+    engine: Annotated[
+        str | None,
+        click.option(
+            "--engine",
+            type=click.Choice(["faiss", "lucene"]),
+            help="quantization type for vectors (in index)",
+            default="faiss",
+            required=False,
+        ),
+    ]
+
 
-class AWSOpenSearchHNSWTypedDict(CommonTypedDict, AWSOpenSearchTypedDict, HNSWFlavor2): ...
+class AWSOpenSearchHNSWTypedDict(CommonTypedDict, AWSOpenSearchTypedDict, HNSWFlavor1): ...
 
 
 @cli.command()
@@ -117,9 +143,13 @@ def AWSOpenSearch(**parameters: Unpack[AWSOpenSearchHNSWTypedDict]):
             refresh_interval=parameters["refresh_interval"],
             force_merge_enabled=parameters["force_merge_enabled"],
             flush_threshold_size=parameters["flush_threshold_size"],
-            number_of_indexing_clients=parameters["number_of_indexing_clients"],
             index_thread_qty_during_force_merge=parameters["index_thread_qty_during_force_merge"],
             cb_threshold=parameters["cb_threshold"],
+            efConstruction=parameters["ef_construction"],
+            efSearch=parameters["ef_runtime"],
+            M=parameters["m"],
+            engine=AWSOS_Engine(parameters["engine"]),
+            quantization_type=AWSOSQuantization(parameters["quantization_type"]),
         ),
         **parameters,
     )