vectordb-bench 0.0.2__tar.gz → 0.0.3__tar.gz

{vectordb-bench-0.0.2 → vectordb-bench-0.0.3}/.gitignore

@@ -7,3 +7,5 @@ __pycache__
 __MACOSX
 .DS_Store
 build/
+venv/
+.idea/

{vectordb-bench-0.0.2 → vectordb-bench-0.0.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: vectordb-bench
-Version: 0.0.2
+Version: 0.0.3
 Summary: VectorDBBench is not just an offering of benchmark results for mainstream vector databases and cloud services, it's your go-to tool for the ultimate performance and cost-effectiveness comparison. Designed with ease-of-use in mind, VectorDBBench is devised to help users, even non-professionals, reproduce results or test new systems, making the hunt for the optimal choice amongst a plethora of cloud services and open-source vector databases a breeze.
 Author-email: XuanYang-cn <xuan.yang@zilliz.com>
 Project-URL: repository, https://github.com/zilliztech/VectorDBBench
@@ -17,6 +17,7 @@ License-File: LICENSE
 [![version](https://img.shields.io/pypi/v/vectordb-bench.svg?color=blue)](https://pypi.org/project/vectordb-bench/)
 [![Downloads](https://pepy.tech/badge/vectordb-bench)](https://pepy.tech/project/vectordb-bench)
 
+**Leaderboard:** https://zilliz.com/benchmark
 ## Quick Start
 ### Prerequirement
 ``` shell
@@ -41,6 +42,24 @@ Closely mimicking real-world production environments, we've set up diverse testi
 
 Prepare to delve into the world of VectorDBBench, and let it guide you in uncovering your perfect vector database match.  
 
+## Leaderboard
+### Introduction
+To facilitate the presentation of test results and provide a comprehensive performance analysis report, we offer a [leaderboard page](https://zilliz.com/benchmark). It allows us to choose from QPS, QP$, and latency metrics, and provides a comprehensive assessment of a system's performance based on the test results of various cases and a set of scoring mechanisms (to be introduced later). On this leaderboard, we can select the systems and models to be compared, and filter out cases we do not want to consider. Comprehensive scores are always ranked from best to worst, and the specific test results of each query will be presented in the list below.
+
+### Scoring Rules
+
+1. For each case, select a base value and score each system based on relative values. 
+    - For QPS and QP$, we use the highest value as the reference, denoted as `base_QPS` or `base_QP$`, and the score of each system is `(QPS/base_QPS) * 100` or `(QP$/base_QP$) * 100`. 
+    - For Latency, we use the lowest value as the reference, that is, `base_Latency`, and the score of each system is `(Latency + 10ms)/(base_Latency + 10ms)`. 
+
+    We want to give equal weight to different cases, and not let a case with high absolute result values become the sole reason for the overall scoring. Therefore, when scoring different systems in each case, we need to use relative values. 
+
+    Also, for Latency, we add 10ms to the numerator and denominator to ensure that if every system performs particularly well in a case, its advantage will not be infinitely magnified when latency tends to 0.
+
+2. For systems that fail or timeout in a particular case, we will give them a score based on a value worse than the worst result by a factor of two. For example, in QPS or QP$, it would be half the lowest value. For Latency, it would be twice the maximum value.
+
+3. For each system, we will take the geometric mean of its scores in all cases as its comprehensive score for a particular metric.
+
 ## Build on your own
 ### Install requirements
 ``` shell
@@ -69,10 +88,12 @@ $ ruff check vectordb_bench --fix
 
 ## How does it work?
 ### Result Page
-![image](https://github.com/zilliztech/VectorDBBench/assets/105927039/66ab83c4-656e-41a8-a643-d9790faccbeb)
+![image](https://github.com/zilliztech/VectorDBBench/assets/105927039/7f5cdae7-f9f2-4a81-b2e0-e5c6268cd970)
 This is the main page of VectorDBBench, which displays the standard benchmark results we provide. Additionally, results of all tests performed by users themselves will also be shown here. We also offer the ability to select and compare results from multiple tests simultaneously.
 
 The standard benchmark results displayed here include all 9 cases that we currently support for all our clients (Milvus, Zilliz Cloud, Elastic Search, Qdrant Cloud, and Weaviate Cloud). However, as some systems may not be able to complete all the tests successfully due to issues like Out of Memory (OOM) or timeouts, not all clients are included in every case.
+
+All standard benchmark results are generated by a client running on an 8 core, 32 GB host, which is located in the same region as the server being tested. The client host is equipped with an `Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz` processor. Also all the servers for the open-source systems tested in our benchmarks run on hosts with the same type of processor.
 ### Run Test Page
 ![image](https://github.com/zilliztech/VectorDBBench/assets/105927039/a789099a-3707-4214-8052-b73463b8f2c6)
 This is the page to run a test:
@@ -102,17 +123,17 @@ We've developed an array of 9 comprehensive benchmark cases to test vector datab
 - **Medium Dataset, High Filtering Rate:** This case uses a medium dataset (Cohere 1M vectors, 768 dimensions) with a high filtering rate.
 For a quick reference, here is a table summarizing the key aspects of each case:
 
-Case No. | Case Type | Dataset Size | Dataset Type | Filtering Rate | Results |
-|----------|-----------|--------------|--------------|----------------|---------|
-1 | Capacity Case | Large Dim | GIST 100K vectors, 960 dimensions | N/A | Number of inserted vectors |
-2 | Capacity Case | Small Dim | SIFT 100K vectors, 128 dimensions | N/A | Number of inserted vectors |
-3 | Search Performance Case | XLarge Dataset | LAION 100M vectors, 768 dimensions | N/A | Index building time, recall, latency, maximum QPS |
-4 | Search Performance Case | Large Dataset | Cohere 10M vectors, 768 dimensions | N/A | Index building time, recall, latency, maximum QPS |
-5 | Search Performance Case | Medium Dataset | Cohere 1M vectors, 768 dimensions | N/A | Index building time, recall, latency, maximum QPS |
-6 | Filtering Search Performance Case | Large Dataset, Low Filtering Rate | Cohere 10M vectors, 768 dimensions | 1% vectors | Index building time, recall, latency, maximum QPS |
-7 | Filtering Search Performance Case | Medium Dataset, Low Filtering Rate | Cohere 1M vectors, 768 dimensions | 1% vectors | Index building time, recall, latency, maximum QPS |
-8 | Filtering Search Performance Case | Large Dataset, High Filtering Rate | Cohere 10M vectors, 768 dimensions | 99% vectors | Index building time, recall, latency, maximum QPS |
-9 | Filtering Search Performance Case | Medium Dataset, High Filtering Rate | Cohere 1M vectors, 768 dimensions | 99% vectors | Index building time, recall, latency, maximum QPS |
+Case No. | Case Type | Dataset Size  | Filtering Rate | Results |
+|----------|-----------|--------------|----------------|---------|
+1 | Capacity Case | GIST 100K vectors, 960 dimensions | N/A | Number of inserted vectors |
+2 | Capacity Case | SIFT 100K vectors, 128 dimensions | N/A | Number of inserted vectors |
+3 | Search Performance Case | LAION 100M vectors, 768 dimensions | N/A | Index building time, recall, latency, maximum QPS |
+4 | Search Performance Case | Cohere 10M vectors, 768 dimensions | N/A | Index building time, recall, latency, maximum QPS |
+5 | Search Performance Case | Cohere 1M vectors, 768 dimensions | N/A | Index building time, recall, latency, maximum QPS |
+6 | Filtering Search Performance Case | Cohere 10M vectors, 768 dimensions | 1% vectors | Index building time, recall, latency, maximum QPS |
+7 | Filtering Search Performance Case | Cohere 1M vectors, 768 dimensions | 1% vectors | Index building time, recall, latency, maximum QPS |
+8 | Filtering Search Performance Case | Cohere 10M vectors, 768 dimensions | 99% vectors | Index building time, recall, latency, maximum QPS |
+9 | Filtering Search Performance Case | Cohere 1M vectors, 768 dimensions | 99% vectors | Index building time, recall, latency, maximum QPS |
 
 Each case provides an in-depth examination of a vector database's abilities, providing you a comprehensive view of the database's performance.
 
@@ -200,3 +221,64 @@ For the Client, we welcome any parameter tuning to obtain better results.
 Many databases may not be able to complete all test cases due to issues such as Out of Memory (OOM), crashes, or timeouts. In these scenarios, we will clearly state these occurrences in the test results.
 ### Mistake Or Misrepresentation 
 We strive for accuracy in learning and supporting various vector databases, yet there might be oversights or misapplications. For any such occurrences, feel free to [raise an issue](https://github.com/zilliztech/VectorDBBench/issues/new) or make amendments on our GitHub page.
+## Timeout
+In our pursuit to ensure that our benchmark reflects the reality of a production environment while guaranteeing the practicality of the system, we have implemented a timeout plan based on our experiences for various tests.
+
+**1. Capacity Case:**
+- For Capacity Case, we have assigned an overall timeout.
+
+**2. Other Cases:**
+
+For other cases, we have set two timeouts:
+
+- **Data Loading Timeout:** This timeout is designed to filter out systems that are too slow in inserting data, thus ensuring that we are only considering systems that is able to cope with the demands of a real-world production environment within a reasonable time frame.
+
+- **Optimization Preparation Timeout**: This timeout is established to avoid excessive optimization strategies that might work for benchmarks but fail to deliver in real production environments. By doing this, we ensure that the systems we consider are not only suitable for testing environments but also applicable and efficient in production scenarios.
+
+This multi-tiered timeout approach allows our benchmark to be more representative of actual production environments and assists us in identifying systems that can truly perform in real-world scenarios.
+<table>
+    <tr>
+        <th>Case</th>
+        <th>Data Size</th>
+        <th>Timeout Type</th>
+        <th>Value</th>
+    </tr>
+    <tr>
+        <td>Capacity Case</td>
+        <td>N/A</td>
+        <td>Loading timeout</td>
+        <td>24 hours</td>
+    </tr>
+    <tr>
+        <td rowspan="2">Other Cases</td>
+        <td rowspan="2">1M vectors, 768 dimensions</td>
+        <td>Loading timeout</td>
+        <td>2.5 hours</td>
+    </tr>
+    <tr>
+        <td>Optimization timeout</td>
+        <td>15 mins</td>
+    </tr>
+    <tr>
+        <td rowspan="2">Other Cases</td>
+        <td rowspan="2">10M vectors, 768 dimensions</td>
+        <td>Loading timeout</td>
+        <td>25 hours</td>
+    </tr>
+    <tr>
+        <td>Optimization timeout</td>
+        <td>2.5 hours</td>
+    </tr>
+    <tr>
+        <td rowspan="2">Other Cases</td>
+        <td rowspan="2">100M vectors, 768 dimensions</td>
+        <td>Loading timeout</td>
+        <td>250 hours</td>
+    </tr>
+    <tr>
+        <td>Optimization timeout</td>
+        <td>25 hours</td>
+    </tr>
+</table>
+
+**Note:** Some datapoints in the standard benchmark results that voilate this timeout will be kept for now for reference. We will remove them in the future.

{vectordb-bench-0.0.2 → vectordb-bench-0.0.3}/README.md

@@ -3,6 +3,7 @@
 [![version](https://img.shields.io/pypi/v/vectordb-bench.svg?color=blue)](https://pypi.org/project/vectordb-bench/)
 [![Downloads](https://pepy.tech/badge/vectordb-bench)](https://pepy.tech/project/vectordb-bench)
 
+**Leaderboard:** https://zilliz.com/benchmark
 ## Quick Start
 ### Prerequirement
 ``` shell
@@ -27,6 +28,24 @@ Closely mimicking real-world production environments, we've set up diverse testi
 
 Prepare to delve into the world of VectorDBBench, and let it guide you in uncovering your perfect vector database match.  
 
+## Leaderboard
+### Introduction
+To facilitate the presentation of test results and provide a comprehensive performance analysis report, we offer a [leaderboard page](https://zilliz.com/benchmark). It allows us to choose from QPS, QP$, and latency metrics, and provides a comprehensive assessment of a system's performance based on the test results of various cases and a set of scoring mechanisms (to be introduced later). On this leaderboard, we can select the systems and models to be compared, and filter out cases we do not want to consider. Comprehensive scores are always ranked from best to worst, and the specific test results of each query will be presented in the list below.
+
+### Scoring Rules
+
+1. For each case, select a base value and score each system based on relative values. 
+    - For QPS and QP$, we use the highest value as the reference, denoted as `base_QPS` or `base_QP$`, and the score of each system is `(QPS/base_QPS) * 100` or `(QP$/base_QP$) * 100`. 
+    - For Latency, we use the lowest value as the reference, that is, `base_Latency`, and the score of each system is `(Latency + 10ms)/(base_Latency + 10ms)`. 
+
+    We want to give equal weight to different cases, and not let a case with high absolute result values become the sole reason for the overall scoring. Therefore, when scoring different systems in each case, we need to use relative values. 
+
+    Also, for Latency, we add 10ms to the numerator and denominator to ensure that if every system performs particularly well in a case, its advantage will not be infinitely magnified when latency tends to 0.
+
+2. For systems that fail or timeout in a particular case, we will give them a score based on a value worse than the worst result by a factor of two. For example, in QPS or QP$, it would be half the lowest value. For Latency, it would be twice the maximum value.
+
+3. For each system, we will take the geometric mean of its scores in all cases as its comprehensive score for a particular metric.
+
 ## Build on your own
 ### Install requirements
 ``` shell
@@ -55,10 +74,12 @@ $ ruff check vectordb_bench --fix
 
 ## How does it work?
 ### Result Page
-![image](https://github.com/zilliztech/VectorDBBench/assets/105927039/66ab83c4-656e-41a8-a643-d9790faccbeb)
+![image](https://github.com/zilliztech/VectorDBBench/assets/105927039/7f5cdae7-f9f2-4a81-b2e0-e5c6268cd970)
 This is the main page of VectorDBBench, which displays the standard benchmark results we provide. Additionally, results of all tests performed by users themselves will also be shown here. We also offer the ability to select and compare results from multiple tests simultaneously.
 
 The standard benchmark results displayed here include all 9 cases that we currently support for all our clients (Milvus, Zilliz Cloud, Elastic Search, Qdrant Cloud, and Weaviate Cloud). However, as some systems may not be able to complete all the tests successfully due to issues like Out of Memory (OOM) or timeouts, not all clients are included in every case.
+
+All standard benchmark results are generated by a client running on an 8 core, 32 GB host, which is located in the same region as the server being tested. The client host is equipped with an `Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz` processor. Also all the servers for the open-source systems tested in our benchmarks run on hosts with the same type of processor.
 ### Run Test Page
 ![image](https://github.com/zilliztech/VectorDBBench/assets/105927039/a789099a-3707-4214-8052-b73463b8f2c6)
 This is the page to run a test:
@@ -88,17 +109,17 @@ We've developed an array of 9 comprehensive benchmark cases to test vector datab
 - **Medium Dataset, High Filtering Rate:** This case uses a medium dataset (Cohere 1M vectors, 768 dimensions) with a high filtering rate.
 For a quick reference, here is a table summarizing the key aspects of each case:
 
-Case No. | Case Type | Dataset Size | Dataset Type | Filtering Rate | Results |
-|----------|-----------|--------------|--------------|----------------|---------|
-1 | Capacity Case | Large Dim | GIST 100K vectors, 960 dimensions | N/A | Number of inserted vectors |
-2 | Capacity Case | Small Dim | SIFT 100K vectors, 128 dimensions | N/A | Number of inserted vectors |
-3 | Search Performance Case | XLarge Dataset | LAION 100M vectors, 768 dimensions | N/A | Index building time, recall, latency, maximum QPS |
-4 | Search Performance Case | Large Dataset | Cohere 10M vectors, 768 dimensions | N/A | Index building time, recall, latency, maximum QPS |
-5 | Search Performance Case | Medium Dataset | Cohere 1M vectors, 768 dimensions | N/A | Index building time, recall, latency, maximum QPS |
-6 | Filtering Search Performance Case | Large Dataset, Low Filtering Rate | Cohere 10M vectors, 768 dimensions | 1% vectors | Index building time, recall, latency, maximum QPS |
-7 | Filtering Search Performance Case | Medium Dataset, Low Filtering Rate | Cohere 1M vectors, 768 dimensions | 1% vectors | Index building time, recall, latency, maximum QPS |
-8 | Filtering Search Performance Case | Large Dataset, High Filtering Rate | Cohere 10M vectors, 768 dimensions | 99% vectors | Index building time, recall, latency, maximum QPS |
-9 | Filtering Search Performance Case | Medium Dataset, High Filtering Rate | Cohere 1M vectors, 768 dimensions | 99% vectors | Index building time, recall, latency, maximum QPS |
+Case No. | Case Type | Dataset Size  | Filtering Rate | Results |
+|----------|-----------|--------------|----------------|---------|
+1 | Capacity Case | GIST 100K vectors, 960 dimensions | N/A | Number of inserted vectors |
+2 | Capacity Case | SIFT 100K vectors, 128 dimensions | N/A | Number of inserted vectors |
+3 | Search Performance Case | LAION 100M vectors, 768 dimensions | N/A | Index building time, recall, latency, maximum QPS |
+4 | Search Performance Case | Cohere 10M vectors, 768 dimensions | N/A | Index building time, recall, latency, maximum QPS |
+5 | Search Performance Case | Cohere 1M vectors, 768 dimensions | N/A | Index building time, recall, latency, maximum QPS |
+6 | Filtering Search Performance Case | Cohere 10M vectors, 768 dimensions | 1% vectors | Index building time, recall, latency, maximum QPS |
+7 | Filtering Search Performance Case | Cohere 1M vectors, 768 dimensions | 1% vectors | Index building time, recall, latency, maximum QPS |
+8 | Filtering Search Performance Case | Cohere 10M vectors, 768 dimensions | 99% vectors | Index building time, recall, latency, maximum QPS |
+9 | Filtering Search Performance Case | Cohere 1M vectors, 768 dimensions | 99% vectors | Index building time, recall, latency, maximum QPS |
 
 Each case provides an in-depth examination of a vector database's abilities, providing you a comprehensive view of the database's performance.
 
@@ -186,3 +207,64 @@ For the Client, we welcome any parameter tuning to obtain better results.
 Many databases may not be able to complete all test cases due to issues such as Out of Memory (OOM), crashes, or timeouts. In these scenarios, we will clearly state these occurrences in the test results.
 ### Mistake Or Misrepresentation 
 We strive for accuracy in learning and supporting various vector databases, yet there might be oversights or misapplications. For any such occurrences, feel free to [raise an issue](https://github.com/zilliztech/VectorDBBench/issues/new) or make amendments on our GitHub page.
+## Timeout
+In our pursuit to ensure that our benchmark reflects the reality of a production environment while guaranteeing the practicality of the system, we have implemented a timeout plan based on our experiences for various tests.
+
+**1. Capacity Case:**
+- For Capacity Case, we have assigned an overall timeout.
+
+**2. Other Cases:**
+
+For other cases, we have set two timeouts:
+
+- **Data Loading Timeout:** This timeout is designed to filter out systems that are too slow in inserting data, thus ensuring that we are only considering systems that is able to cope with the demands of a real-world production environment within a reasonable time frame.
+
+- **Optimization Preparation Timeout**: This timeout is established to avoid excessive optimization strategies that might work for benchmarks but fail to deliver in real production environments. By doing this, we ensure that the systems we consider are not only suitable for testing environments but also applicable and efficient in production scenarios.
+
+This multi-tiered timeout approach allows our benchmark to be more representative of actual production environments and assists us in identifying systems that can truly perform in real-world scenarios.
+<table>
+    <tr>
+        <th>Case</th>
+        <th>Data Size</th>
+        <th>Timeout Type</th>
+        <th>Value</th>
+    </tr>
+    <tr>
+        <td>Capacity Case</td>
+        <td>N/A</td>
+        <td>Loading timeout</td>
+        <td>24 hours</td>
+    </tr>
+    <tr>
+        <td rowspan="2">Other Cases</td>
+        <td rowspan="2">1M vectors, 768 dimensions</td>
+        <td>Loading timeout</td>
+        <td>2.5 hours</td>
+    </tr>
+    <tr>
+        <td>Optimization timeout</td>
+        <td>15 mins</td>
+    </tr>
+    <tr>
+        <td rowspan="2">Other Cases</td>
+        <td rowspan="2">10M vectors, 768 dimensions</td>
+        <td>Loading timeout</td>
+        <td>25 hours</td>
+    </tr>
+    <tr>
+        <td>Optimization timeout</td>
+        <td>2.5 hours</td>
+    </tr>
+    <tr>
+        <td rowspan="2">Other Cases</td>
+        <td rowspan="2">100M vectors, 768 dimensions</td>
+        <td>Loading timeout</td>
+        <td>250 hours</td>
+    </tr>
+    <tr>
+        <td>Optimization timeout</td>
+        <td>25 hours</td>
+    </tr>
+</table>
+
+**Note:** Some datapoints in the standard benchmark results that voilate this timeout will be kept for now for reference. We will remove them in the future.

{vectordb-bench-0.0.2 → vectordb-bench-0.0.3}/pyproject.toml

@@ -37,6 +37,9 @@ dependencies = [
     "scikit-learn",
     "s3fs",
     "psutil",
+    "polars",
+    "pgvector",
+    "sqlalchemy"
 ]
 dynamic = ["version"]
 

vectordb-bench-0.0.3/tests/test_dataset.py

@@ -0,0 +1,36 @@
+from vectordb_bench.backend.dataset import Dataset
+import logging
+import pytest
+from pydantic import ValidationError
+
+
+log = logging.getLogger("vectordb_bench")
+
+class TestDataSet:
+    def test_iter_dataset(self):
+        for ds in Dataset:
+            log.info(ds)
+
+    def test_cohere(self):
+        cohere = Dataset.COHERE.get(100_000)
+        log.info(cohere)
+        assert cohere.name == "Cohere"
+        assert cohere.size == 100_000
+        assert cohere.label == "SMALL"
+        assert cohere.dim == 768
+
+    def test_cohere_error(self):
+        with pytest.raises(ValidationError):
+            Dataset.COHERE.get(9999)
+
+    def test_init_cohere(self):
+        coheres = [Dataset.COHERE.manager(i) for i in [100_000, 1_000_000, 10_000_000]]
+        for t in coheres:
+            t._validate_local_file()
+
+    def test_iter_cohere(self):
+        cohere_10m = Dataset.COHERE.manager(10_000_000)
+        cohere_10m.prepare(False)
+        for i in cohere_10m:
+            log.debug(i.head(1))
+

{vectordb-bench-0.0.2 → vectordb-bench-0.0.3}/tests/test_models.py

@@ -65,6 +65,6 @@ class TestModels:
 
     def test_test_result_display(self):
         result_dir = config.RESULTS_LOCAL_DIR
-        for json_file in result_dir.glob("*.json"):
+        for json_file in result_dir.glob("result*.json"):
             res = TestResult.read_file(json_file)
             res.display()

{vectordb-bench-0.0.2 → vectordb-bench-0.0.3}/tests/ut_cases.py

@@ -3,32 +3,21 @@ from vectordb_bench.backend.cases import (
     CaseType,
 )
 
-import vectordb_bench.backend.dataset as ds
-from pydantic.dataclasses import dataclass
-
-@dataclass
-class Cohere_S(ds.Cohere):
-    label: str = "SMALL"
-    size: int  = 100_000
-
-@dataclass
-class Glove_S(ds.Glove):
-    label: str = "SMALL"
-    size : int = 100_000
+from vectordb_bench.backend.datase import Dataset, DatasetManager
 
 
 class Performance100K99p(PerformanceCase):
-    case_id: CaseType = CaseType.PerformanceSHigh
+    case_id: CaseType = 100
     filter_rate: float | int | None = 0.99
-    dataset: ds.DataSet = ds.get(ds.Name.Cohere, ds.Label.SMALL)
+    dataset: DatasetManager = Dataset.COHERE.manager(100_000)
     name: str = "Filtering Search Performance Test (100K Dataset, 768 Dim, Filter 99%)"
     description: str = """This case tests the search performance of a vector database with a small dataset (<b>Cohere 100K vectors</b>, 768 dimensions) under a high filtering rate (<b>99% vectors</b>), at varying parallel levels.
 Results will show index building time, recall, and maximum QPS."""
 
 class Performance100K1p(PerformanceCase):
-    case_id: CaseType = CaseType.PerformanceSLow
+    case_id: CaseType = 100
     filter_rate: float | int | None = 0.01
-    dataset: ds.DataSet = ds.get(ds.Name.Cohere, ds.Label.SMALL)
+    dataset: DatasetManager = Dataset.COHERE.manager(100_000)
     name: str = "Filtering Search Performance Test (100K Dataset, 768 Dim, Filter 1%)"
     description: str = (
         """This case tests the search performance of a vector database with a small dataset (<b>Cohere 100K vectors</b>, 768 dimensions) under a low filtering rate (<b>1% vectors</b>), at varying parallel levels.
@@ -37,10 +26,8 @@ Results will show index building time, recall, and maximum QPS.""",
 
 
 class Performance100K(PerformanceCase):
-    case_id: CaseType = CaseType.PerformanceSZero
-    dataset: ds.DataSet = ds.get(ds.Name.Cohere, ds.Label.SMALL)
+    case_id: CaseType = 100
+    dataset: DatasetManager = Dataset.COHERE.manager(100_000)
     name: str = "Search Performance Test (100K Dataset, 768 Dim)"
     description: str = """This case tests the search performance of a vector database with a small dataset (<b>Cohere 100K vectors</b>, 768 dimensions) at varying parallel levels.
 Results will show index building time, recall, and maximum QPS."""
-
-

{vectordb-bench-0.0.2 → vectordb-bench-0.0.3}/vectordb_bench/__init__.py

@@ -18,12 +18,23 @@ class config:
     USE_SHUFFLED_DATA = env.bool("USE_SHUFFLED_DATA", True)
 
     RESULTS_LOCAL_DIR = pathlib.Path(__file__).parent.joinpath("results")
-    CASE_TIMEOUT_IN_SECOND = 24 * 60 * 60
+
+    CAPACITY_TIMEOUT_IN_SECONDS =  24 * 3600 # 24h
+    LOAD_TIMEOUT_1M             = 2.5 * 3600 # 2.5h
+    LOAD_TIMEOUT_10M            =  25 * 3600 # 25h
+    LOAD_TIMEOUT_100M           = 250 * 3600 # 10.41d
+
+    OPTIMIZE_TIMEOUT_1M         =  15 * 60   # 15min
+    OPTIMIZE_TIMEOUT_10M        = 2.5 * 3600 # 2.5h
+    OPTIMIZE_TIMEOUT_100M       =  25 * 3600 # 1.04d
 
 
     def display(self) -> str:
-        tmp = [i for i in inspect.getmembers(self)
-            if not inspect.ismethod(i[1]) and not i[0].startswith('_') \
+        tmp = [
+            i for i in inspect.getmembers(self)
+            if not inspect.ismethod(i[1])
+            and not i[0].startswith('_')
+            and "TIMEOUT" not in i[0]
         ]
         return tmp
 

{vectordb-bench-0.0.2 → vectordb-bench-0.0.3}/vectordb_bench/backend/cases.py

@@ -2,8 +2,10 @@ import typing
 import logging
 from enum import Enum, auto
 
-from . import dataset as ds
-from ..base import BaseModel
+from vectordb_bench import config
+from vectordb_bench.base import BaseModel
+
+from .dataset import Dataset, DatasetManager
 
 
 log = logging.getLogger(__name__)
@@ -44,7 +46,7 @@ class CaseType(Enum):
         if c is not None:
             return c().name
         raise ValueError("Case unsupported")
-    
+
     @property
     def case_description(self) -> str:
         c = self.case_cls
@@ -73,7 +75,10 @@ class Case(BaseModel):
     label: CaseLabel
     name: str
     description: str
-    dataset: ds.DataSet
+    dataset: DatasetManager
+
+    load_timeout: float | int
+    optimize_timeout: float | int | None
 
     filter_rate: float | None
 
@@ -92,6 +97,8 @@ class Case(BaseModel):
 class CapacityCase(Case, BaseModel):
     label: CaseLabel = CaseLabel.Load
     filter_rate: float | None = None
+    load_timeout: float | int = config.CAPACITY_TIMEOUT_IN_SECONDS
+    optimize_timeout: float | int | None = None
 
 
 class PerformanceCase(Case, BaseModel):
@@ -101,7 +108,7 @@ class PerformanceCase(Case, BaseModel):
 
 class CapacityDim960(CapacityCase):
     case_id: CaseType = CaseType.CapacityDim960
-    dataset: ds.DataSet = ds.get(ds.Name.GIST, ds.Label.SMALL)
+    dataset: DatasetManager = Dataset.GIST.manager(100_000)
     name: str = "Capacity Test (960 Dim Repeated)"
     description: str = """This case tests the vector database's loading capacity by repeatedly inserting large-dimension vectors (GIST 100K vectors, <b>960 dimensions</b>) until it is fully loaded.
 Number of inserted vectors will be reported."""
@@ -109,7 +116,7 @@ Number of inserted vectors will be reported."""
 
 class CapacityDim128(CapacityCase):
     case_id: CaseType = CaseType.CapacityDim128
-    dataset: ds.DataSet = ds.get(ds.Name.SIFT, ds.Label.SMALL)
+    dataset: DatasetManager = Dataset.SIFT.manager(500_000)
     name: str = "Capacity Test (128 Dim Repeated)"
     description: str = """This case tests the vector database's loading capacity by repeatedly inserting small-dimension vectors (SIFT 100K vectors, <b>128 dimensions</b>) until it is fully loaded.
 Number of inserted vectors will be reported."""
@@ -117,64 +124,78 @@ Number of inserted vectors will be reported."""
 
 class Performance10M(PerformanceCase):
     case_id: CaseType = CaseType.Performance10M
-    dataset: ds.DataSet = ds.get(ds.Name.Cohere, ds.Label.LARGE)
+    dataset: DatasetManager = Dataset.COHERE.manager(10_000_000)
     name: str = "Search Performance Test (10M Dataset, 768 Dim)"
     description: str = """This case tests the search performance of a vector database with a large dataset (<b>Cohere 10M vectors</b>, 768 dimensions) at varying parallel levels.
 Results will show index building time, recall, and maximum QPS."""
+    load_timeout: float | int = config.LOAD_TIMEOUT_10M
+    optimize_timeout: float | int | None = config.OPTIMIZE_TIMEOUT_10M
 
 
 class Performance1M(PerformanceCase):
     case_id: CaseType = CaseType.Performance1M
-    dataset: ds.DataSet = ds.get(ds.Name.Cohere, ds.Label.MEDIUM)
+    dataset: DatasetManager = Dataset.COHERE.manager(1_000_000)
     name: str = "Search Performance Test (1M Dataset, 768 Dim)"
     description: str = """This case tests the search performance of a vector database with a medium dataset (<b>Cohere 1M vectors</b>, 768 dimensions) at varying parallel levels.
 Results will show index building time, recall, and maximum QPS."""
+    load_timeout: float | int = config.LOAD_TIMEOUT_1M
+    optimize_timeout: float | int | None = config.OPTIMIZE_TIMEOUT_1M
 
 
 class Performance10M1P(PerformanceCase):
     case_id: CaseType = CaseType.Performance10M1P
     filter_rate: float | int | None = 0.01
-    dataset: ds.DataSet = ds.get(ds.Name.Cohere, ds.Label.LARGE)
+    dataset: DatasetManager = Dataset.COHERE.manager(10_000_000)
     name: str = "Filtering Search Performance Test (10M Dataset, 768 Dim, Filter 1%)"
     description: str = """This case tests the search performance of a vector database with a large dataset (<b>Cohere 10M vectors</b>, 768 dimensions) under a low filtering rate (<b>1% vectors</b>), at varying parallel levels.
 Results will show index building time, recall, and maximum QPS."""
+    load_timeout: float | int = config.LOAD_TIMEOUT_10M
+    optimize_timeout: float | int | None = config.OPTIMIZE_TIMEOUT_10M
 
 
 class Performance1M1P(PerformanceCase):
     case_id: CaseType = CaseType.Performance1M1P
     filter_rate: float | int | None = 0.01
-    dataset: ds.DataSet = ds.get(ds.Name.Cohere, ds.Label.MEDIUM)
+    dataset: DatasetManager = Dataset.COHERE.manager(1_000_000)
     name: str = "Filtering Search Performance Test (1M Dataset, 768 Dim, Filter 1%)"
     description: str = """This case tests the search performance of a vector database with a medium dataset (<b>Cohere 1M vectors</b>, 768 dimensions) under a low filtering rate (<b>1% vectors</b>), at varying parallel levels.
 Results will show index building time, recall, and maximum QPS."""
+    load_timeout: float | int = config.LOAD_TIMEOUT_1M
+    optimize_timeout: float | int | None = config.OPTIMIZE_TIMEOUT_1M
 
 
 class Performance10M99P(PerformanceCase):
     case_id: CaseType = CaseType.Performance10M99P
     filter_rate: float | int | None = 0.99
-    dataset: ds.DataSet = ds.get(ds.Name.Cohere, ds.Label.LARGE)
+    dataset: DatasetManager = Dataset.COHERE.manager(10_000_000)
     name: str = "Filtering Search Performance Test (10M Dataset, 768 Dim, Filter 99%)"
     description: str = """This case tests the search performance of a vector database with a large dataset (<b>Cohere 10M vectors</b>, 768 dimensions) under a high filtering rate (<b>99% vectors</b>), at varying parallel levels.
 Results will show index building time, recall, and maximum QPS."""
+    load_timeout: float | int = config.LOAD_TIMEOUT_10M
+    optimize_timeout: float | int | None = config.OPTIMIZE_TIMEOUT_10M
 
 
 class Performance1M99P(PerformanceCase):
     case_id: CaseType = CaseType.Performance1M99P
     filter_rate: float | int | None = 0.99
-    dataset: ds.DataSet = ds.get(ds.Name.Cohere, ds.Label.MEDIUM)
+    dataset: DatasetManager = Dataset.COHERE.manager(1_000_000)
     name: str = "Filtering Search Performance Test (1M Dataset, 768 Dim, Filter 99%)"
     description: str = """This case tests the search performance of a vector database with a medium dataset (<b>Cohere 1M vectors</b>, 768 dimensions) under a high filtering rate (<b>99% vectors</b>), at varying parallel levels.
 Results will show index building time, recall, and maximum QPS."""
+    load_timeout: float | int = config.LOAD_TIMEOUT_1M
+    optimize_timeout: float | int | None = config.OPTIMIZE_TIMEOUT_1M
 
 
 
 class Performance100M(PerformanceCase):
     case_id: CaseType = CaseType.Performance100M
     filter_rate: float | int | None = None
-    dataset: ds.DataSet = ds.get(ds.Name.LAION, ds.Label.LARGE)
+    dataset: DatasetManager = Dataset.LAION.manager(100_000_000)
     name: str = "Search Performance Test (100M Dataset, 768 Dim)"
     description: str = """This case tests the search performance of a vector database with a large 100M dataset (<b>LAION 100M vectors</b>, 768 dimensions), at varying parallel levels.
 Results will show index building time, recall, and maximum QPS."""
+    load_timeout: float | int = config.LOAD_TIMEOUT_100M
+    optimize_timeout: float | int | None = config.OPTIMIZE_TIMEOUT_100M
 
 
 type2case = {

{vectordb-bench-0.0.2 → vectordb-bench-0.0.3}/vectordb_bench/backend/clients/__init__.py

@@ -15,7 +15,7 @@ from .pinecone.pinecone import Pinecone
 from .weaviate_cloud.weaviate_cloud import WeaviateCloud
 from .qdrant_cloud.qdrant_cloud import QdrantCloud
 from .zilliz_cloud.zilliz_cloud import ZillizCloud
-
+from .pgvector.pgvector import PgVector
 
 class DB(Enum):
     """Database types
@@ -35,6 +35,7 @@ class DB(Enum):
     ElasticCloud = "ElasticCloud"
     QdrantCloud = "QdrantCloud"
     WeaviateCloud = "WeaviateCloud"
+    PgVector = "PgVector"
 
 
     @property
@@ -49,8 +50,12 @@ db2client = {
     DB.ElasticCloud: ElasticCloud,
     DB.QdrantCloud: QdrantCloud,
     DB.Pinecone: Pinecone,
+    DB.PgVector: PgVector
 }
 
+for db in DB:
+    assert issubclass(db.init_cls, VectorDB)
+
 
 __all__ = [
     "DB", "VectorDB", "DBConfig", "DBCaseConfig", "IndexType", "MetricType", "EmptyDBCaseConfig",