redisbench-admin 0.11.55__py3-none-any.whl → 0.11.57__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- redisbench_admin/environments/oss_cluster.py +9 -1
- redisbench_admin/run/asm.py +1 -1
- redisbench_admin/run_remote/remote_helpers.py +41 -11
- redisbench_admin/run_remote/standalone.py +2 -3
- redisbench_admin/utils/remote.py +2 -0
- {redisbench_admin-0.11.55.dist-info → redisbench_admin-0.11.57.dist-info}/METADATA +7 -4
- redisbench_admin-0.11.57.dist-info/RECORD +117 -0
- {redisbench_admin-0.11.55.dist-info → redisbench_admin-0.11.57.dist-info}/WHEEL +1 -1
- redisbench_admin/run/ann/pkg/.dockerignore +0 -2
- redisbench_admin/run/ann/pkg/.git +0 -1
- redisbench_admin/run/ann/pkg/.github/workflows/benchmarks.yml +0 -100
- redisbench_admin/run/ann/pkg/.gitignore +0 -21
- redisbench_admin/run/ann/pkg/LICENSE +0 -21
- redisbench_admin/run/ann/pkg/README.md +0 -157
- redisbench_admin/run/ann/pkg/algos.yaml +0 -1294
- redisbench_admin/run/ann/pkg/algosP.yaml +0 -67
- redisbench_admin/run/ann/pkg/ann_benchmarks/__init__.py +0 -2
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/__init__.py +0 -0
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/annoy.py +0 -26
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/balltree.py +0 -22
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/base.py +0 -36
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/bruteforce.py +0 -110
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/ckdtree.py +0 -17
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/datasketch.py +0 -29
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/definitions.py +0 -187
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/diskann.py +0 -190
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/dolphinnpy.py +0 -31
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/dummy_algo.py +0 -25
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/elasticsearch.py +0 -107
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/elastiknn.py +0 -124
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/faiss.py +0 -124
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/faiss_gpu.py +0 -61
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/faiss_hnsw.py +0 -39
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/flann.py +0 -27
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/hnswlib.py +0 -36
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/kdtree.py +0 -22
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/kgraph.py +0 -39
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/lshf.py +0 -25
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/milvus.py +0 -99
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/mrpt.py +0 -41
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/n2.py +0 -28
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/nearpy.py +0 -48
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/nmslib.py +0 -74
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/onng_ngt.py +0 -100
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/opensearchknn.py +0 -107
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/panng_ngt.py +0 -79
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/pinecone.py +0 -39
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/puffinn.py +0 -45
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/pynndescent.py +0 -115
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/qg_ngt.py +0 -102
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/redisearch.py +0 -90
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/rpforest.py +0 -20
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/scann.py +0 -34
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/sptag.py +0 -28
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/subprocess.py +0 -246
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/vald.py +0 -149
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/vecsim-hnsw.py +0 -43
- redisbench_admin/run/ann/pkg/ann_benchmarks/algorithms/vespa.py +0 -47
- redisbench_admin/run/ann/pkg/ann_benchmarks/constants.py +0 -1
- redisbench_admin/run/ann/pkg/ann_benchmarks/data.py +0 -48
- redisbench_admin/run/ann/pkg/ann_benchmarks/datasets.py +0 -620
- redisbench_admin/run/ann/pkg/ann_benchmarks/distance.py +0 -53
- redisbench_admin/run/ann/pkg/ann_benchmarks/main.py +0 -325
- redisbench_admin/run/ann/pkg/ann_benchmarks/plotting/__init__.py +0 -2
- redisbench_admin/run/ann/pkg/ann_benchmarks/plotting/metrics.py +0 -183
- redisbench_admin/run/ann/pkg/ann_benchmarks/plotting/plot_variants.py +0 -17
- redisbench_admin/run/ann/pkg/ann_benchmarks/plotting/utils.py +0 -165
- redisbench_admin/run/ann/pkg/ann_benchmarks/results.py +0 -71
- redisbench_admin/run/ann/pkg/ann_benchmarks/runner.py +0 -333
- redisbench_admin/run/ann/pkg/create_dataset.py +0 -12
- redisbench_admin/run/ann/pkg/create_hybrid_dataset.py +0 -147
- redisbench_admin/run/ann/pkg/create_text_to_image_ds.py +0 -117
- redisbench_admin/run/ann/pkg/create_website.py +0 -272
- redisbench_admin/run/ann/pkg/install/Dockerfile +0 -11
- redisbench_admin/run/ann/pkg/install/Dockerfile.annoy +0 -5
- redisbench_admin/run/ann/pkg/install/Dockerfile.datasketch +0 -4
- redisbench_admin/run/ann/pkg/install/Dockerfile.diskann +0 -29
- redisbench_admin/run/ann/pkg/install/Dockerfile.diskann_pq +0 -31
- redisbench_admin/run/ann/pkg/install/Dockerfile.dolphinn +0 -5
- redisbench_admin/run/ann/pkg/install/Dockerfile.elasticsearch +0 -45
- redisbench_admin/run/ann/pkg/install/Dockerfile.elastiknn +0 -61
- redisbench_admin/run/ann/pkg/install/Dockerfile.faiss +0 -18
- redisbench_admin/run/ann/pkg/install/Dockerfile.flann +0 -10
- redisbench_admin/run/ann/pkg/install/Dockerfile.hnswlib +0 -10
- redisbench_admin/run/ann/pkg/install/Dockerfile.kgraph +0 -6
- redisbench_admin/run/ann/pkg/install/Dockerfile.mih +0 -4
- redisbench_admin/run/ann/pkg/install/Dockerfile.milvus +0 -27
- redisbench_admin/run/ann/pkg/install/Dockerfile.mrpt +0 -4
- redisbench_admin/run/ann/pkg/install/Dockerfile.n2 +0 -5
- redisbench_admin/run/ann/pkg/install/Dockerfile.nearpy +0 -5
- redisbench_admin/run/ann/pkg/install/Dockerfile.ngt +0 -13
- redisbench_admin/run/ann/pkg/install/Dockerfile.nmslib +0 -10
- redisbench_admin/run/ann/pkg/install/Dockerfile.opensearchknn +0 -43
- redisbench_admin/run/ann/pkg/install/Dockerfile.puffinn +0 -6
- redisbench_admin/run/ann/pkg/install/Dockerfile.pynndescent +0 -4
- redisbench_admin/run/ann/pkg/install/Dockerfile.redisearch +0 -18
- redisbench_admin/run/ann/pkg/install/Dockerfile.rpforest +0 -5
- redisbench_admin/run/ann/pkg/install/Dockerfile.scann +0 -5
- redisbench_admin/run/ann/pkg/install/Dockerfile.scipy +0 -4
- redisbench_admin/run/ann/pkg/install/Dockerfile.sklearn +0 -4
- redisbench_admin/run/ann/pkg/install/Dockerfile.sptag +0 -30
- redisbench_admin/run/ann/pkg/install/Dockerfile.vald +0 -8
- redisbench_admin/run/ann/pkg/install/Dockerfile.vespa +0 -17
- redisbench_admin/run/ann/pkg/install.py +0 -70
- redisbench_admin/run/ann/pkg/logging.conf +0 -34
- redisbench_admin/run/ann/pkg/multirun.py +0 -298
- redisbench_admin/run/ann/pkg/plot.py +0 -159
- redisbench_admin/run/ann/pkg/protocol/bf-runner +0 -10
- redisbench_admin/run/ann/pkg/protocol/bf-runner.py +0 -204
- redisbench_admin/run/ann/pkg/protocol/ext-add-query-metric.md +0 -51
- redisbench_admin/run/ann/pkg/protocol/ext-batch-queries.md +0 -77
- redisbench_admin/run/ann/pkg/protocol/ext-prepared-queries.md +0 -77
- redisbench_admin/run/ann/pkg/protocol/ext-query-parameters.md +0 -47
- redisbench_admin/run/ann/pkg/protocol/specification.md +0 -194
- redisbench_admin/run/ann/pkg/requirements.txt +0 -14
- redisbench_admin/run/ann/pkg/requirements_py38.txt +0 -11
- redisbench_admin/run/ann/pkg/results/fashion-mnist-784-euclidean.png +0 -0
- redisbench_admin/run/ann/pkg/results/gist-960-euclidean.png +0 -0
- redisbench_admin/run/ann/pkg/results/glove-100-angular.png +0 -0
- redisbench_admin/run/ann/pkg/results/glove-25-angular.png +0 -0
- redisbench_admin/run/ann/pkg/results/lastfm-64-dot.png +0 -0
- redisbench_admin/run/ann/pkg/results/mnist-784-euclidean.png +0 -0
- redisbench_admin/run/ann/pkg/results/nytimes-256-angular.png +0 -0
- redisbench_admin/run/ann/pkg/results/sift-128-euclidean.png +0 -0
- redisbench_admin/run/ann/pkg/run.py +0 -12
- redisbench_admin/run/ann/pkg/run_algorithm.py +0 -3
- redisbench_admin/run/ann/pkg/templates/chartjs.template +0 -102
- redisbench_admin/run/ann/pkg/templates/detail_page.html +0 -23
- redisbench_admin/run/ann/pkg/templates/general.html +0 -58
- redisbench_admin/run/ann/pkg/templates/latex.template +0 -30
- redisbench_admin/run/ann/pkg/templates/summary.html +0 -60
- redisbench_admin/run/ann/pkg/test/__init__.py +0 -0
- redisbench_admin/run/ann/pkg/test/test-jaccard.py +0 -19
- redisbench_admin/run/ann/pkg/test/test-metrics.py +0 -99
- redisbench_admin-0.11.55.dist-info/RECORD +0 -243
- {redisbench_admin-0.11.55.dist-info → redisbench_admin-0.11.57.dist-info}/entry_points.txt +0 -0
- {redisbench_admin-0.11.55.dist-info → redisbench_admin-0.11.57.dist-info/licenses}/LICENSE +0 -0
|
@@ -201,7 +201,15 @@ def generate_cluster_redis_server_args(
|
|
|
201
201
|
dbfilename = get_cluster_dbfilename(port)
|
|
202
202
|
|
|
203
203
|
command = generate_common_server_args(
|
|
204
|
-
binary,
|
|
204
|
+
binary,
|
|
205
|
+
daemonize,
|
|
206
|
+
dbdir,
|
|
207
|
+
dbfilename,
|
|
208
|
+
enable_debug_command,
|
|
209
|
+
ip,
|
|
210
|
+
logfile,
|
|
211
|
+
port,
|
|
212
|
+
enable_redis_7_config_directives,
|
|
205
213
|
)
|
|
206
214
|
command.extend(
|
|
207
215
|
[
|
redisbench_admin/run/asm.py
CHANGED
|
@@ -145,7 +145,7 @@ class ShardSlotInfo:
|
|
|
145
145
|
"""
|
|
146
146
|
if shard_index < 0 or shard_index >= len(self.shards):
|
|
147
147
|
raise IndexError(
|
|
148
|
-
f"Shard index {shard_index} out of range (0..{len(self.shards)-1})"
|
|
148
|
+
f"Shard index {shard_index} out of range (0..{len(self.shards) - 1})"
|
|
149
149
|
)
|
|
150
150
|
|
|
151
151
|
shard = self.shards[shard_index]
|
|
@@ -205,8 +205,7 @@ def remote_tool_pre_bench_step(
|
|
|
205
205
|
)
|
|
206
206
|
logging.info("Finished up remote tool {} requirements".format(benchmark_tool))
|
|
207
207
|
|
|
208
|
-
|
|
209
|
-
def setup_remote_benchmark_tool_requirements_ftsb(
|
|
208
|
+
def _setup_remote_benchmark_tool_requirements(
|
|
210
209
|
client_public_ip,
|
|
211
210
|
username,
|
|
212
211
|
private_key,
|
|
@@ -218,14 +217,43 @@ def setup_remote_benchmark_tool_requirements_ftsb(
|
|
|
218
217
|
):
|
|
219
218
|
commands = [
|
|
220
219
|
"wget {} -q -O {}".format(tool_link, remote_tool_link),
|
|
221
|
-
"
|
|
222
|
-
"chmod 755 {}".format(remote_tool_link),
|
|
220
|
+
"chmod 755 {}".format(remote_tool_link)
|
|
223
221
|
]
|
|
222
|
+
|
|
223
|
+
# detect if queries_file_link is a s3 URI or http one and act accordingly (use aws cli or wget)
|
|
224
|
+
if queries_file_link.startswith("s3://"):
|
|
225
|
+
commands.append(
|
|
226
|
+
"aws s3 cp {} {} --no-sign-request".format(queries_file_link, remote_input_file)
|
|
227
|
+
)
|
|
228
|
+
else:
|
|
229
|
+
commands.append("wget {} -q -O {}".format(queries_file_link, remote_input_file))
|
|
224
230
|
execute_remote_commands(
|
|
225
231
|
client_public_ip, username, private_key, commands, client_ssh_port
|
|
226
232
|
)
|
|
227
233
|
|
|
228
234
|
|
|
235
|
+
def setup_remote_benchmark_tool_requirements_ftsb(
|
|
236
|
+
client_public_ip,
|
|
237
|
+
username,
|
|
238
|
+
private_key,
|
|
239
|
+
tool_link,
|
|
240
|
+
queries_file_link,
|
|
241
|
+
remote_tool_link,
|
|
242
|
+
remote_input_file,
|
|
243
|
+
client_ssh_port,
|
|
244
|
+
):
|
|
245
|
+
_setup_remote_benchmark_tool_requirements(
|
|
246
|
+
client_public_ip,
|
|
247
|
+
username,
|
|
248
|
+
private_key,
|
|
249
|
+
tool_link,
|
|
250
|
+
queries_file_link,
|
|
251
|
+
remote_tool_link,
|
|
252
|
+
remote_input_file,
|
|
253
|
+
client_ssh_port,
|
|
254
|
+
)
|
|
255
|
+
|
|
256
|
+
|
|
229
257
|
def setup_remote_benchmark_tool_requirements_tsbs(
|
|
230
258
|
client_public_ip,
|
|
231
259
|
username,
|
|
@@ -236,13 +264,15 @@ def setup_remote_benchmark_tool_requirements_tsbs(
|
|
|
236
264
|
remote_input_file,
|
|
237
265
|
client_ssh_port,
|
|
238
266
|
):
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
|
|
267
|
+
_setup_remote_benchmark_tool_requirements(
|
|
268
|
+
client_public_ip,
|
|
269
|
+
username,
|
|
270
|
+
private_key,
|
|
271
|
+
tool_link,
|
|
272
|
+
queries_file_link,
|
|
273
|
+
remote_tool_link,
|
|
274
|
+
remote_input_file,
|
|
275
|
+
client_ssh_port,
|
|
246
276
|
)
|
|
247
277
|
|
|
248
278
|
|
|
@@ -13,7 +13,6 @@ from redisbench_admin.utils.remote import (
|
|
|
13
13
|
)
|
|
14
14
|
from redisbench_admin.utils.ssh import SSHSession
|
|
15
15
|
from redisbench_admin.utils.utils import redis_server_config_module_part
|
|
16
|
-
import tempfile
|
|
17
16
|
|
|
18
17
|
|
|
19
18
|
def ensure_redis_server_available(server_public_ip, username, private_key, port=22):
|
|
@@ -516,7 +515,7 @@ def spin_test_standalone_redis(
|
|
|
516
515
|
return False
|
|
517
516
|
|
|
518
517
|
remote_redis_conf_path = f"{temporary_dir}/redis.conf"
|
|
519
|
-
logging.info(
|
|
518
|
+
logging.info("📁 Copying custom redis.conf to remote host...")
|
|
520
519
|
|
|
521
520
|
copy_result = copy_file_to_remote_setup(
|
|
522
521
|
server_public_ip,
|
|
@@ -541,7 +540,7 @@ def spin_test_standalone_redis(
|
|
|
541
540
|
return False
|
|
542
541
|
|
|
543
542
|
remote_redis_server_path = f"{temporary_dir}/redis-server"
|
|
544
|
-
logging.info(
|
|
543
|
+
logging.info("📁 Copying custom redis-server binary to remote host...")
|
|
545
544
|
|
|
546
545
|
copy_result = copy_file_to_remote_setup(
|
|
547
546
|
server_public_ip,
|
redisbench_admin/utils/remote.py
CHANGED
|
@@ -178,6 +178,8 @@ def connect_remote_ssh(port, private_key, server_public_ip, username):
|
|
|
178
178
|
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
|
|
179
179
|
logging.info("Connecting to remote server {}".format(server_public_ip))
|
|
180
180
|
c.connect(hostname=server_public_ip, port=port, username=username, pkey=k)
|
|
181
|
+
transport = c.get_transport()
|
|
182
|
+
transport.set_keepalive(10) # Send keepalive every 10 seconds
|
|
181
183
|
logging.info("Connected to remote server {}".format(server_public_ip))
|
|
182
184
|
return c
|
|
183
185
|
|
|
@@ -1,7 +1,8 @@
|
|
|
1
|
-
Metadata-Version: 2.
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
2
|
Name: redisbench-admin
|
|
3
|
-
Version: 0.11.
|
|
3
|
+
Version: 0.11.57
|
|
4
4
|
Summary: Redis benchmark run helper. A wrapper around Redis and Redis Modules benchmark tools ( ftsb_redisearch, memtier_benchmark, redis-benchmark, aibench, etc... ).
|
|
5
|
+
License-File: LICENSE
|
|
5
6
|
Author: filipecosta90
|
|
6
7
|
Author-email: filipecosta.90@gmail.com
|
|
7
8
|
Requires-Python: >=3.10.0,<4.0.0
|
|
@@ -9,18 +10,20 @@ Classifier: Programming Language :: Python :: 3
|
|
|
9
10
|
Classifier: Programming Language :: Python :: 3.10
|
|
10
11
|
Classifier: Programming Language :: Python :: 3.11
|
|
11
12
|
Classifier: Programming Language :: Python :: 3.12
|
|
13
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
12
15
|
Requires-Dist: Flask (>=2.0.1,<3.0.0)
|
|
13
16
|
Requires-Dist: Flask-HTTPAuth (>=4.4.0,<5.0.0)
|
|
14
17
|
Requires-Dist: GitPython (>=3.1.12,<4.0.0)
|
|
15
18
|
Requires-Dist: Jinja2 (>=3.0.3,<4.0.0)
|
|
16
|
-
Requires-Dist: PyYAML (>=6.0,<7.0)
|
|
19
|
+
Requires-Dist: PyYAML (>=6.0.1,<7.0.0)
|
|
17
20
|
Requires-Dist: boto3 (>=1.13.24,<2.0.0)
|
|
18
21
|
Requires-Dist: certifi (>=2021.10.8,<2025.0.0)
|
|
19
22
|
Requires-Dist: daemonize (>=2.5.0,<3.0.0)
|
|
20
23
|
Requires-Dist: flask-restx (>=0.5.1,<0.6.0)
|
|
21
24
|
Requires-Dist: humanize (>=2.4.0,<3.0.0)
|
|
22
25
|
Requires-Dist: jsonpath_ng (>=1.5.2,<2.0.0)
|
|
23
|
-
Requires-Dist: matplotlib (>=3.
|
|
26
|
+
Requires-Dist: matplotlib (>=3.9.0,<4.0.0)
|
|
24
27
|
Requires-Dist: numpy (>=2.0.0,<3.0.0)
|
|
25
28
|
Requires-Dist: pandas (>=2.1.2,<3.0.0)
|
|
26
29
|
Requires-Dist: paramiko (>=2.7.2,<4.0.0)
|
|
@@ -0,0 +1,117 @@
|
|
|
1
|
+
redisbench_admin/__init__.py,sha256=kLgzVDw4-frGXtbJH_WKiwWcXaUZYisYBKYYOeSJWvU,473
|
|
2
|
+
redisbench_admin/cli.py,sha256=Kljn7e6zV8kCGRGSfZCf1SdHdvtW8d3kA5LwMba68Zw,8073
|
|
3
|
+
redisbench_admin/commands/__init__.py,sha256=mzVrEtqefFdopyzR-W6xx3How95dyZfToGKm1-_YzeY,95
|
|
4
|
+
redisbench_admin/commands/commands.json.py,sha256=mzVrEtqefFdopyzR-W6xx3How95dyZfToGKm1-_YzeY,95
|
|
5
|
+
redisbench_admin/compare/__init__.py,sha256=DtBXRp0Q01XgCFmY-1OIePMyyYihVNAjZ1Y8zwqSDN0,101
|
|
6
|
+
redisbench_admin/compare/args.py,sha256=rOP8td_GYPYsZiYN4olyYmUsku-oizggEZNRTB_fVro,6336
|
|
7
|
+
redisbench_admin/compare/compare.py,sha256=NvC7RbQUJUKlPdJQj9TjUmk2k8UtiA9wnZWNDV7IM_Q,103315
|
|
8
|
+
redisbench_admin/deploy/__init__.py,sha256=DtBXRp0Q01XgCFmY-1OIePMyyYihVNAjZ1Y8zwqSDN0,101
|
|
9
|
+
redisbench_admin/deploy/args.py,sha256=neLUcQqI__HkJItkQg2C293hl5g3yHG40t171r7-E5Y,1732
|
|
10
|
+
redisbench_admin/deploy/deploy.py,sha256=MtfJbsL97DLrbBYut6zRCzyEMebX4xWoZE-m4-JDRB8,3885
|
|
11
|
+
redisbench_admin/environments/__init__.py,sha256=cD7zfXt0VEmy0b7452HvcAxX_9kVj6Vm213yNdUHP20,95
|
|
12
|
+
redisbench_admin/environments/oss_cluster.py,sha256=qiHjhmkQ6cqvsD-4pLxXX1S1Hb2LarSg5V-XuXCoTZo,8241
|
|
13
|
+
redisbench_admin/environments/oss_standalone.py,sha256=Sl38rUpwJ3wNOl9zn38iK8q2iJi2pRFmaJAZJbuT_SQ,2474
|
|
14
|
+
redisbench_admin/export/__init__.py,sha256=DtBXRp0Q01XgCFmY-1OIePMyyYihVNAjZ1Y8zwqSDN0,101
|
|
15
|
+
redisbench_admin/export/args.py,sha256=v_WjJCNz_LeIFMNwSN6XwRmvSx1K2ys8XS1gK50EM_4,3508
|
|
16
|
+
redisbench_admin/export/common/__init__.py,sha256=DtBXRp0Q01XgCFmY-1OIePMyyYihVNAjZ1Y8zwqSDN0,101
|
|
17
|
+
redisbench_admin/export/common/common.py,sha256=LnvXjMLlJRzMTxiFIjrfRFfDx9JJm88OZHu7lnTOpFA,4331
|
|
18
|
+
redisbench_admin/export/export.py,sha256=u00NjaCbWhCJ319leVlP4ZkqiqZt5FN4Gbag4Poo23M,11274
|
|
19
|
+
redisbench_admin/export/google_benchmark/__init__.py,sha256=DtBXRp0Q01XgCFmY-1OIePMyyYihVNAjZ1Y8zwqSDN0,101
|
|
20
|
+
redisbench_admin/export/google_benchmark/google_benchmark_json_format.py,sha256=OuMaMmmma5VvXA0rcLIQSMxIq81oa5I3xYDFhbWj-IA,1804
|
|
21
|
+
redisbench_admin/export/memtier_benchmark/__init__.py,sha256=DtBXRp0Q01XgCFmY-1OIePMyyYihVNAjZ1Y8zwqSDN0,101
|
|
22
|
+
redisbench_admin/export/memtier_benchmark/memtier_benchmark_json_format.py,sha256=DtBXRp0Q01XgCFmY-1OIePMyyYihVNAjZ1Y8zwqSDN0,101
|
|
23
|
+
redisbench_admin/export/pyperf/__init__.py,sha256=DtBXRp0Q01XgCFmY-1OIePMyyYihVNAjZ1Y8zwqSDN0,101
|
|
24
|
+
redisbench_admin/export/pyperf/pyperf_json_format.py,sha256=g3S68cRAMJiogpUP31-hVltw04xN1LxHszC3fBshO3U,1677
|
|
25
|
+
redisbench_admin/export/redis_benchmark/__init__.py,sha256=DtBXRp0Q01XgCFmY-1OIePMyyYihVNAjZ1Y8zwqSDN0,101
|
|
26
|
+
redisbench_admin/export/redis_benchmark/metrics_definition.py,sha256=d_w37WSLtp0JZ0u-Ge1R3Hv43_bycLRCXBZ9zDR5-uE,2522
|
|
27
|
+
redisbench_admin/export/redis_benchmark/redis_benchmark_csv_format.py,sha256=a0_aqO4NN54FCE-TIDJ2Yqf5aFU4tlC0fHZTcQnDvEE,5780
|
|
28
|
+
redisbench_admin/extract/__init__.py,sha256=DtBXRp0Q01XgCFmY-1OIePMyyYihVNAjZ1Y8zwqSDN0,101
|
|
29
|
+
redisbench_admin/extract/args.py,sha256=OrEwGOd58fu1F4moUrzFq1bUb69mpiMSGeh9tGthq0o,1073
|
|
30
|
+
redisbench_admin/extract/extract.py,sha256=V1rAFwbOkIFfcKLH_82K4zNAcJtaQn3wz60ddzRzxBc,770
|
|
31
|
+
redisbench_admin/grafana_api/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
|
|
32
|
+
redisbench_admin/grafana_api/app.py,sha256=F4bw7ToO9bHB6mj6y9EhiTBxSNzFBHJfWJT7AcZBpl8,3416
|
|
33
|
+
redisbench_admin/grafana_api/args.py,sha256=7chFoUcTASHsSYLgzGWH6VEP_gtdnvK3AgLC_sR4D6o,1212
|
|
34
|
+
redisbench_admin/grafana_api/grafana_api.py,sha256=dG17GCYmWRILmy7h3-OiBeGzuNGnRGUv-jqR-7MRiws,1613
|
|
35
|
+
redisbench_admin/profilers/__init__.py,sha256=DtBXRp0Q01XgCFmY-1OIePMyyYihVNAjZ1Y8zwqSDN0,101
|
|
36
|
+
redisbench_admin/profilers/daemon.py,sha256=xm1P5c0Lv91kllT4DRLDJH4azGuPHtnpgNeGjC6ecpc,12975
|
|
37
|
+
redisbench_admin/profilers/flamegraph.pl,sha256=Za5XE-1gb_U-nzqwoyRwfe1TB182c64gITa-2klWTTA,35898
|
|
38
|
+
redisbench_admin/profilers/perf.py,sha256=HtzzMVsXEJa1H7tOAfKlbFYDn2KnxAG_IU9yKPKZB7w,27772
|
|
39
|
+
redisbench_admin/profilers/perf_daemon_caller.py,sha256=nD97cXmX3JytyafvNMmhUBq40uYrf6vtjdJ1TXZbvVY,4948
|
|
40
|
+
redisbench_admin/profilers/pprof.py,sha256=g7oNC3AtNDTUOBIh_mIi5bFl_b0mL8tqBu6qKvAOrKw,3949
|
|
41
|
+
redisbench_admin/profilers/profilers.py,sha256=4C1xaPyLoPydJ3eBAxW7IlSHG-3qj3A3BAKejiZXEK0,510
|
|
42
|
+
redisbench_admin/profilers/profilers_local.py,sha256=etU6f5EdEq2a6JrakwPVfaoHlBSb0sycTEYg5ugvjVw,12220
|
|
43
|
+
redisbench_admin/profilers/profilers_schema.py,sha256=XvKURihYh9Qu8lB7HXNHqgcb_G6eBznC2J70VxU3c5E,2744
|
|
44
|
+
redisbench_admin/profilers/stackcollapse-perf.pl,sha256=XStXxBheOVjJYdo9Gha0rf5PZb4o1J1IJ-pQY77FV08,10307
|
|
45
|
+
redisbench_admin/profilers/vtune.py,sha256=OMx5ABr5OtWeiqK0dTspdyeAfKlDEzhdXDKocDSa9xM,4618
|
|
46
|
+
redisbench_admin/run/__init__.py,sha256=DtBXRp0Q01XgCFmY-1OIePMyyYihVNAjZ1Y8zwqSDN0,101
|
|
47
|
+
redisbench_admin/run/aibench_run_inference_redisai_vision/__init__.py,sha256=DtBXRp0Q01XgCFmY-1OIePMyyYihVNAjZ1Y8zwqSDN0,101
|
|
48
|
+
redisbench_admin/run/aibench_run_inference_redisai_vision/aibench_run_inference_redisai_vision.py,sha256=03GDbZZSo21f9iBTcNTdBKQAfq_Ghd0zxr-KYe1QIMU,1917
|
|
49
|
+
redisbench_admin/run/ann/__init__.py,sha256=DtBXRp0Q01XgCFmY-1OIePMyyYihVNAjZ1Y8zwqSDN0,101
|
|
50
|
+
redisbench_admin/run/ann/ann.py,sha256=h8VE7fouyNkpMSjNhgjkI_R-aWb-lKIKxvNZPOO8pFY,1288
|
|
51
|
+
redisbench_admin/run/args.py,sha256=4OJIj19fieEjvz7Xnk1rIjaa50R5Kv1IlyMyECHtj0Y,8219
|
|
52
|
+
redisbench_admin/run/asm.py,sha256=Hybna8Xs7SnsP2IBwg6kMsiGPoCK0TlvhyKX94XmOEs,16738
|
|
53
|
+
redisbench_admin/run/cluster.py,sha256=_Y6a8Dbu1cJ7OxhgymKQSZcCmV8cZ3UpGEWL6b6O84Y,6363
|
|
54
|
+
redisbench_admin/run/common.py,sha256=1zJPQtIk3tjvLTjNUgxvDdcviN73psiaDBQ8cRJMIl8,29341
|
|
55
|
+
redisbench_admin/run/ftsb/__init__.py,sha256=DtBXRp0Q01XgCFmY-1OIePMyyYihVNAjZ1Y8zwqSDN0,101
|
|
56
|
+
redisbench_admin/run/ftsb/ftsb.py,sha256=MnviWzIxMPxAw__R1lQT6tTsH7nP7t5ejCXrALm2MUw,2580
|
|
57
|
+
redisbench_admin/run/git.py,sha256=6UYGcTN0MPzf4QDVoJnFkou0yZasLF6jLG7f0zoySq8,3064
|
|
58
|
+
redisbench_admin/run/grafana.py,sha256=iMDgMyJKinpZMTD43rZ1IcRGkadjFjCxaB48mYWkvG4,9421
|
|
59
|
+
redisbench_admin/run/memtier_benchmark/__init__.py,sha256=DtBXRp0Q01XgCFmY-1OIePMyyYihVNAjZ1Y8zwqSDN0,101
|
|
60
|
+
redisbench_admin/run/memtier_benchmark/memtier_benchmark.py,sha256=wTd2olovvFBZ98mOSr6DM5BJsdaiuPteEZzBqeSgbkE,4246
|
|
61
|
+
redisbench_admin/run/metrics.py,sha256=8EQdcZbCiFB_kIR1WtUQNOPV8y74bZ8Dj51Cv0aR4nk,7556
|
|
62
|
+
redisbench_admin/run/modules.py,sha256=9To85oDw2tmUNmTDxOgvKls_46oZRcd2cCt6xNjIWiA,1691
|
|
63
|
+
redisbench_admin/run/redis_benchmark/__init__.py,sha256=DtBXRp0Q01XgCFmY-1OIePMyyYihVNAjZ1Y8zwqSDN0,101
|
|
64
|
+
redisbench_admin/run/redis_benchmark/redis_benchmark.py,sha256=e-Az2uTlt3z2W4uzlUsdxeT8GITpxpGb-Mjb6JxrSWc,6848
|
|
65
|
+
redisbench_admin/run/redisgraph_benchmark_go/__init__.py,sha256=DtBXRp0Q01XgCFmY-1OIePMyyYihVNAjZ1Y8zwqSDN0,101
|
|
66
|
+
redisbench_admin/run/redisgraph_benchmark_go/redisgraph_benchmark_go.py,sha256=DijIoWr73e2t7Zy-UmODzC51IjUh6hH5I0LaPRJW4Vk,2241
|
|
67
|
+
redisbench_admin/run/redistimeseries.py,sha256=x3PA7QoHXu53zs5v0ekK2sVmUnA9_ZF2JxgCDf1Mui4,21331
|
|
68
|
+
redisbench_admin/run/run.py,sha256=AwE4_qkejAXOcDVFmwHjVqYjqi1BFZKdGNrCKBHkAHI,6030
|
|
69
|
+
redisbench_admin/run/s3.py,sha256=pXQXZ1rrwDCWeBegGR4aKzbKqWWwMrmqvIjFxEB3bh4,442
|
|
70
|
+
redisbench_admin/run/ssh.py,sha256=gRW6ROoTKlaxLKhS5tM-Ejjd6zk2iO1KN9dzBzw7GOk,2835
|
|
71
|
+
redisbench_admin/run/tsbs_run_queries_redistimeseries/__init__.py,sha256=DtBXRp0Q01XgCFmY-1OIePMyyYihVNAjZ1Y8zwqSDN0,101
|
|
72
|
+
redisbench_admin/run/tsbs_run_queries_redistimeseries/tsbs_run_queries_redistimeseries.py,sha256=CfXcoLS25pEkhGtka82JwgDZVPV4w2lU_VhI-reH5h4,2514
|
|
73
|
+
redisbench_admin/run/ycsb/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
|
|
74
|
+
redisbench_admin/run/ycsb/ycsb.py,sha256=cs5saVH7C4YpDvzhoa15PwEho59qTVR1E90v_FYjMVw,6873
|
|
75
|
+
redisbench_admin/run_async/__init__.py,sha256=DtBXRp0Q01XgCFmY-1OIePMyyYihVNAjZ1Y8zwqSDN0,101
|
|
76
|
+
redisbench_admin/run_async/async_env.py,sha256=tE1turaaZNHfOaSpGxh62EJWp88zoQFUf3sMbaS7JRA,2408
|
|
77
|
+
redisbench_admin/run_async/async_terraform.py,sha256=ngOQnECUuC20pZwiJItaiBnzlwT2DiKciPTHtqLURe4,11299
|
|
78
|
+
redisbench_admin/run_async/benchmark.py,sha256=S-dsaWGjgsPQxj8sXAACnbtNw5zlJnRFoo53ULbrMEY,1630
|
|
79
|
+
redisbench_admin/run_async/log.py,sha256=cD7zfXt0VEmy0b7452HvcAxX_9kVj6Vm213yNdUHP20,95
|
|
80
|
+
redisbench_admin/run_async/render_files.py,sha256=NMagmx-2hsMET_XN8tkmQz55g-azqW7SjAqaq4GL8F0,2676
|
|
81
|
+
redisbench_admin/run_async/run_async.py,sha256=g2ZOQqj9vXZYaRyNpJZtgfYyY9tMuRmEv3Hh3qWOUs8,14525
|
|
82
|
+
redisbench_admin/run_local/__init__.py,sha256=DtBXRp0Q01XgCFmY-1OIePMyyYihVNAjZ1Y8zwqSDN0,101
|
|
83
|
+
redisbench_admin/run_local/args.py,sha256=LPpqtx1cH1dkkeHjYlaFnAp_TijxnzPZFO2CmYD9ikU,1906
|
|
84
|
+
redisbench_admin/run_local/local_client.py,sha256=gwawMDOBrf7m--uyxu8kMZC5LBiLjbUBSKvzVOdOAas,124
|
|
85
|
+
redisbench_admin/run_local/local_db.py,sha256=9vINqKOs-wDMFEuEHT0I8KO9YnEo_h4NWNk5da3LwSY,7518
|
|
86
|
+
redisbench_admin/run_local/local_helpers.py,sha256=JyqLW2-Sbm35BXjxxfOB1yK7ADdLfcVrq08NLNdIwac,7026
|
|
87
|
+
redisbench_admin/run_local/run_local.py,sha256=TIGhSp4UaVZtoUxKMwS4BMeE7dNQJm9ODBtsDjcGrbI,35333
|
|
88
|
+
redisbench_admin/run_remote/__init__.py,sha256=DtBXRp0Q01XgCFmY-1OIePMyyYihVNAjZ1Y8zwqSDN0,101
|
|
89
|
+
redisbench_admin/run_remote/args.py,sha256=Ef32mg1yNYYHL5g59SzIWZqFB__RNLLriPqiucVyoNg,4826
|
|
90
|
+
redisbench_admin/run_remote/consts.py,sha256=bCMkwyeBD-EmOpoHKni7LjWy5WuaxGJhGhqpi4AL0RQ,386
|
|
91
|
+
redisbench_admin/run_remote/log.py,sha256=cD7zfXt0VEmy0b7452HvcAxX_9kVj6Vm213yNdUHP20,95
|
|
92
|
+
redisbench_admin/run_remote/notifications.py,sha256=-W9fLaftEFNfplBl2clHk37jbYxliDbHftQ62khN31k,2157
|
|
93
|
+
redisbench_admin/run_remote/remote_client.py,sha256=rRmDro1weto01wzqYpId8NMPoizEzSyudXBCjYrBVMs,14128
|
|
94
|
+
redisbench_admin/run_remote/remote_db.py,sha256=EEDeiOZk-godr5EINscEkOJLGWUN3gFfH6RaBzAKbak,14566
|
|
95
|
+
redisbench_admin/run_remote/remote_env.py,sha256=Ux_0QT1unNRlKl3cakzjG5Px1uuxOOfBoF_pnalx_T8,4936
|
|
96
|
+
redisbench_admin/run_remote/remote_failures.py,sha256=IOo6DyxarcwwMPCeN4gWB2JrhuC9iBLwq0nCROqr5ak,1567
|
|
97
|
+
redisbench_admin/run_remote/remote_helpers.py,sha256=OEFjbdqG_FJ5KY6B63jYr21vPomtM3ld_F9MsGRTaCU,11271
|
|
98
|
+
redisbench_admin/run_remote/run_remote.py,sha256=tZqCu1fTfB5gWooVIEsSDoaVfnVRfxeCpn-RLmYI3IM,75476
|
|
99
|
+
redisbench_admin/run_remote/standalone.py,sha256=WRgiY8oMJwDUPht1LAPG9tnOLvhJmhRvVOi3GBnMoX4,27952
|
|
100
|
+
redisbench_admin/run_remote/terraform.py,sha256=vV3eWXNwj7vsnFNqUgCir5ueZS4VYopEyzWiTtoSq0Q,4018
|
|
101
|
+
redisbench_admin/utils/__init__.py,sha256=DtBXRp0Q01XgCFmY-1OIePMyyYihVNAjZ1Y8zwqSDN0,101
|
|
102
|
+
redisbench_admin/utils/benchmark_config.py,sha256=bC2C6rnj89wkkSlOXyyfe0N15unn_M1t1zfskfVkb98,21387
|
|
103
|
+
redisbench_admin/utils/local.py,sha256=zUvyVI9LZMT3qyxs1pO3mXL6Bt_1z9EZUGppaRcWNRA,3890
|
|
104
|
+
redisbench_admin/utils/redisearch.py,sha256=lchUEzpt0zB1rHwlDlw9LLifAnxFWcLP-PePw7TjL-0,1602
|
|
105
|
+
redisbench_admin/utils/redisgraph_benchmark_go.py,sha256=os7EJt6kBxsFJLKkSoANbjMT7-cEq4-Ns-49alk2Tf8,2048
|
|
106
|
+
redisbench_admin/utils/remote.py,sha256=MsRL6x-eekzJOVE6Ho05igZzUX54m-4fRVCQlSYxyw0,42306
|
|
107
|
+
redisbench_admin/utils/results.py,sha256=uKk3uNJ--bSXlUj_HGQ2OaV6MVqmXJVM8xTzFV6EOw4,3267
|
|
108
|
+
redisbench_admin/utils/ssh.py,sha256=QW4AwlocMHJt05QMdN_4f8WeDmxiEwR80ny8VBThq6k,6533
|
|
109
|
+
redisbench_admin/utils/utils.py,sha256=XVSvo1_DdcYwk2jOxL3VPVPbnDnhGYt8ieYfANo6rTo,15085
|
|
110
|
+
redisbench_admin/watchdog/__init__.py,sha256=cD7zfXt0VEmy0b7452HvcAxX_9kVj6Vm213yNdUHP20,95
|
|
111
|
+
redisbench_admin/watchdog/args.py,sha256=nKsG1G6ATOZlAMHMtT9u3kXxduKCbejSZ5x8oB_ynZ8,1312
|
|
112
|
+
redisbench_admin/watchdog/watchdog.py,sha256=0wWYge3x_OMxWrzazNhJif2NK4tKsI963HVZqjczRag,6189
|
|
113
|
+
redisbench_admin-0.11.57.dist-info/METADATA,sha256=2T9Tvwug0CQlh1ueoikzvMk4sBlqjL2JP9argPeR25A,5724
|
|
114
|
+
redisbench_admin-0.11.57.dist-info/WHEEL,sha256=zp0Cn7JsFoX2ATtOhtaFYIiE2rmFAD4OcMhtUki8W3U,88
|
|
115
|
+
redisbench_admin-0.11.57.dist-info/entry_points.txt,sha256=UUawXk_AS-PlieKJ1QxPQXGsRLb6OW_F0MtmA1W0KE8,113
|
|
116
|
+
redisbench_admin-0.11.57.dist-info/licenses/LICENSE,sha256=AAMtfs82zOOvmG68vILivm6lxi2rcOlGObmA8jzxQvw,10768
|
|
117
|
+
redisbench_admin-0.11.57.dist-info/RECORD,,
|
|
@@ -1 +0,0 @@
|
|
|
1
|
-
gitdir: ../../../../.git/modules/redisbench_admin/run/ann/pkg
|
|
@@ -1,100 +0,0 @@
|
|
|
1
|
-
name: ANN benchmarks
|
|
2
|
-
|
|
3
|
-
on: [push, pull_request]
|
|
4
|
-
|
|
5
|
-
jobs:
|
|
6
|
-
build:
|
|
7
|
-
|
|
8
|
-
runs-on: ubuntu-20.04
|
|
9
|
-
strategy:
|
|
10
|
-
matrix:
|
|
11
|
-
include:
|
|
12
|
-
- library: annoy
|
|
13
|
-
dataset: random-xs-20-angular
|
|
14
|
-
- library: dolphinn
|
|
15
|
-
dataset: random-xs-20-angular
|
|
16
|
-
- library: faiss
|
|
17
|
-
dataset: random-xs-20-angular
|
|
18
|
-
- library: flann
|
|
19
|
-
dataset: random-xs-20-angular
|
|
20
|
-
- library: kgraph
|
|
21
|
-
dataset: random-xs-20-angular
|
|
22
|
-
- library: milvus
|
|
23
|
-
dataset: random-xs-20-angular
|
|
24
|
-
- library: mrpt
|
|
25
|
-
dataset: random-xs-20-angular
|
|
26
|
-
- library: n2
|
|
27
|
-
dataset: random-xs-20-angular
|
|
28
|
-
- library: nearpy
|
|
29
|
-
dataset: random-xs-20-angular
|
|
30
|
-
- library: ngt
|
|
31
|
-
dataset: random-xs-20-angular
|
|
32
|
-
- library: nmslib
|
|
33
|
-
dataset: random-xs-20-angular
|
|
34
|
-
- library: hnswlib
|
|
35
|
-
dataset: random-xs-20-angular
|
|
36
|
-
- library: puffinn
|
|
37
|
-
dataset: random-xs-20-angular
|
|
38
|
-
- library: pynndescent
|
|
39
|
-
dataset: random-xs-20-angular
|
|
40
|
-
- library: rpforest
|
|
41
|
-
dataset: random-xs-20-angular
|
|
42
|
-
- library: sklearn
|
|
43
|
-
dataset: random-xs-20-angular
|
|
44
|
-
- library: sptag
|
|
45
|
-
dataset: random-xs-20-angular
|
|
46
|
-
- library: mih
|
|
47
|
-
dataset: random-xs-16-hamming
|
|
48
|
-
- library: datasketch
|
|
49
|
-
dataset: random-s-jaccard
|
|
50
|
-
- library: scann
|
|
51
|
-
dataset: random-xs-20-angular
|
|
52
|
-
- library: elasticsearch
|
|
53
|
-
dataset: random-xs-20-angular
|
|
54
|
-
- library: elastiknn
|
|
55
|
-
dataset: random-xs-20-angular
|
|
56
|
-
- library: opensearchknn
|
|
57
|
-
dataset: random-xs-20-angular
|
|
58
|
-
- library: diskann
|
|
59
|
-
dataset: random-xs-20-angular
|
|
60
|
-
- library: puffinn
|
|
61
|
-
dataset: random-s-jaccard
|
|
62
|
-
- library: pynndescent
|
|
63
|
-
dataset: random-s-jaccard
|
|
64
|
-
- library: vespa
|
|
65
|
-
dataset: random-xs-20-angular
|
|
66
|
-
- library: scipy
|
|
67
|
-
dataset: random-xs-20-angular
|
|
68
|
-
- library: vald
|
|
69
|
-
dataset: random-xs-20-angular
|
|
70
|
-
fail-fast: false
|
|
71
|
-
|
|
72
|
-
steps:
|
|
73
|
-
- uses: actions/checkout@v2 # Pull the repository
|
|
74
|
-
|
|
75
|
-
- name: Install various apt packages
|
|
76
|
-
run: sudo apt-get install -y libhdf5-dev python3-numpy python3-scipy python3-matplotlib python3-sklearn
|
|
77
|
-
|
|
78
|
-
- name: Install dependencies
|
|
79
|
-
run: |
|
|
80
|
-
pip3 install -r requirements.txt
|
|
81
|
-
python3 install.py
|
|
82
|
-
|
|
83
|
-
env:
|
|
84
|
-
LIBRARY: ${{ matrix.library }}
|
|
85
|
-
DATASET: ${{ matrix.dataset }}
|
|
86
|
-
|
|
87
|
-
- name: Run the benchmark
|
|
88
|
-
run: |
|
|
89
|
-
python3 run.py --docker-tag ann-benchmarks-${LIBRARY} --max-n-algorithms 5 --dataset $DATASET --run-disabled --timeout 300
|
|
90
|
-
python3 run.py --docker-tag ann-benchmarks-${LIBRARY} --max-n-algorithms 5 --dataset $DATASET --run-disabled --batch --timeout 300
|
|
91
|
-
sudo chmod -R 777 results/
|
|
92
|
-
python3 plot.py --dataset $DATASET --output plot.png
|
|
93
|
-
python3 plot.py --dataset $DATASET --output plot-batch.png --batch
|
|
94
|
-
python3 -m unittest test/test-metrics.py
|
|
95
|
-
python3 -m unittest test/test-jaccard.py
|
|
96
|
-
python3 create_website.py --outputdir . --scatter --latex
|
|
97
|
-
|
|
98
|
-
env:
|
|
99
|
-
LIBRARY: ${{ matrix.library }}
|
|
100
|
-
DATASET: ${{ matrix.dataset }}
|
|
@@ -1,21 +0,0 @@
|
|
|
1
|
-
MIT License
|
|
2
|
-
|
|
3
|
-
Copyright (c) 2018 Erik Bernhardsson
|
|
4
|
-
|
|
5
|
-
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
-
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
-
in the Software without restriction, including without limitation the rights
|
|
8
|
-
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
-
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
-
furnished to do so, subject to the following conditions:
|
|
11
|
-
|
|
12
|
-
The above copyright notice and this permission notice shall be included in all
|
|
13
|
-
copies or substantial portions of the Software.
|
|
14
|
-
|
|
15
|
-
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
-
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
-
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
-
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
-
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
-
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
-
SOFTWARE.
|
|
@@ -1,157 +0,0 @@
|
|
|
1
|
-
Benchmarking nearest neighbors
|
|
2
|
-
==============================
|
|
3
|
-
|
|
4
|
-
[](https://github.com/erikbern/ann-benchmarks/actions?query=workflow:benchmarks)
|
|
5
|
-
|
|
6
|
-
Doing fast searching of nearest neighbors in high dimensional spaces is an increasingly important problem, but so far there has not been a lot of empirical attempts at comparing approaches in an objective way.
|
|
7
|
-
|
|
8
|
-
This project contains some tools to benchmark various implementations of approximate nearest neighbor (ANN) search for different metrics. We have pregenerated datasets (in HDF5) formats and we also have Docker containers for each algorithm. There's a [test suite](https://travis-ci.org/erikbern/ann-benchmarks) that makes sure every algorithm works.
|
|
9
|
-
|
|
10
|
-
Evaluated
|
|
11
|
-
=========
|
|
12
|
-
|
|
13
|
-
* [Annoy](https://github.com/spotify/annoy)
|
|
14
|
-
* [FLANN](http://www.cs.ubc.ca/research/flann/)
|
|
15
|
-
* [scikit-learn](http://scikit-learn.org/stable/modules/neighbors.html): LSHForest, KDTree, BallTree
|
|
16
|
-
* [PANNS](https://github.com/ryanrhymes/panns)
|
|
17
|
-
* [NearPy](http://pixelogik.github.io/NearPy/)
|
|
18
|
-
* [KGraph](https://github.com/aaalgo/kgraph)
|
|
19
|
-
* [NMSLIB (Non-Metric Space Library)](https://github.com/nmslib/nmslib): SWGraph, HNSW, BallTree, MPLSH
|
|
20
|
-
* [hnswlib (a part of nmslib project)](https://github.com/nmslib/hnsw)
|
|
21
|
-
* [RPForest](https://github.com/lyst/rpforest)
|
|
22
|
-
* [FAISS](https://github.com/facebookresearch/faiss.git)
|
|
23
|
-
* [DolphinnPy](https://github.com/ipsarros/DolphinnPy)
|
|
24
|
-
* [Datasketch](https://github.com/ekzhu/datasketch)
|
|
25
|
-
* [PyNNDescent](https://github.com/lmcinnes/pynndescent)
|
|
26
|
-
* [MRPT](https://github.com/teemupitkanen/mrpt)
|
|
27
|
-
* [NGT](https://github.com/yahoojapan/NGT): ONNG, PANNG, QG
|
|
28
|
-
* [SPTAG](https://github.com/microsoft/SPTAG)
|
|
29
|
-
* [PUFFINN](https://github.com/puffinn/puffinn)
|
|
30
|
-
* [N2](https://github.com/kakao/n2)
|
|
31
|
-
* [ScaNN](https://github.com/google-research/google-research/tree/master/scann)
|
|
32
|
-
* [Elastiknn](https://github.com/alexklibisz/elastiknn)
|
|
33
|
-
* [OpenSearch KNN](https://github.com/opensearch-project/k-NN)
|
|
34
|
-
* [DiskANN](https://github.com/microsoft/diskann): Vamana, Vamana-PQ
|
|
35
|
-
* [Vespa](https://github.com/vespa-engine/vespa)
|
|
36
|
-
* [scipy](https://docs.scipy.org/doc/scipy/reference/spatial.html): cKDTree
|
|
37
|
-
* [vald](https://github.com/vdaas/vald)
|
|
38
|
-
|
|
39
|
-
Data sets
|
|
40
|
-
=========
|
|
41
|
-
|
|
42
|
-
We have a number of precomputed data sets for this. All data sets are pre-split into train/test and come with ground truth data in the form of the top 100 neighbors. We store them in a HDF5 format:
|
|
43
|
-
|
|
44
|
-
| Dataset | Dimensions | Train size | Test size | Neighbors | Distance | Download |
|
|
45
|
-
| ----------------------------------------------------------------- | ---------: | ---------: | --------: | --------: | --------- | -------------------------------------------------------------------------- |
|
|
46
|
-
| [DEEP1B](http://sites.skoltech.ru/compvision/noimi/) | 96 | 9,990,000 | 10,000 | 100 | Angular | [HDF5](http://ann-benchmarks.com/deep-image-96-angular.hdf5) (3.6GB)
|
|
47
|
-
| [Fashion-MNIST](https://github.com/zalandoresearch/fashion-mnist) | 784 | 60,000 | 10,000 | 100 | Euclidean | [HDF5](http://ann-benchmarks.com/fashion-mnist-784-euclidean.hdf5) (217MB) |
|
|
48
|
-
| [GIST](http://corpus-texmex.irisa.fr/) | 960 | 1,000,000 | 1,000 | 100 | Euclidean | [HDF5](http://ann-benchmarks.com/gist-960-euclidean.hdf5) (3.6GB) |
|
|
49
|
-
| [GloVe](http://nlp.stanford.edu/projects/glove/) | 25 | 1,183,514 | 10,000 | 100 | Angular | [HDF5](http://ann-benchmarks.com/glove-25-angular.hdf5) (121MB) |
|
|
50
|
-
| GloVe | 50 | 1,183,514 | 10,000 | 100 | Angular | [HDF5](http://ann-benchmarks.com/glove-50-angular.hdf5) (235MB) |
|
|
51
|
-
| GloVe | 100 | 1,183,514 | 10,000 | 100 | Angular | [HDF5](http://ann-benchmarks.com/glove-100-angular.hdf5) (463MB) |
|
|
52
|
-
| GloVe | 200 | 1,183,514 | 10,000 | 100 | Angular | [HDF5](http://ann-benchmarks.com/glove-200-angular.hdf5) (918MB) |
|
|
53
|
-
| [Kosarak](http://fimi.uantwerpen.be/data/) | 27983 | 74,962 | 500 | 100 | Jaccard | [HDF5](http://ann-benchmarks.com/kosarak-jaccard.hdf5) (2.0GB) |
|
|
54
|
-
| [MNIST](http://yann.lecun.com/exdb/mnist/) | 784 | 60,000 | 10,000 | 100 | Euclidean | [HDF5](http://ann-benchmarks.com/mnist-784-euclidean.hdf5) (217MB) |
|
|
55
|
-
| [NYTimes](https://archive.ics.uci.edu/ml/datasets/bag+of+words) | 256 | 290,000 | 10,000 | 100 | Angular | [HDF5](http://ann-benchmarks.com/nytimes-256-angular.hdf5) (301MB) |
|
|
56
|
-
| [SIFT](http://corpus-texmex.irisa.fr/) | 128 | 1,000,000 | 10,000 | 100 | Euclidean | [HDF5](http://ann-benchmarks.com/sift-128-euclidean.hdf5) (501MB) |
|
|
57
|
-
| [Last.fm](https://github.com/erikbern/ann-benchmarks/pull/91) | 65 | 292,385 | 50,000 | 100 | Angular | [HDF5](http://ann-benchmarks.com/lastfm-64-dot.hdf5) (135MB) |
|
|
58
|
-
|
|
59
|
-
Results
|
|
60
|
-
=======
|
|
61
|
-
|
|
62
|
-
Interactive plots can be found at <http://ann-benchmarks.com>. These are all as of December 2021, running all benchmarks on a r5.4xlarge machine on AWS with `--parallelism 7`:
|
|
63
|
-
|
|
64
|
-
glove-100-angular
|
|
65
|
-
-----------------
|
|
66
|
-
|
|
67
|
-

|
|
68
|
-
|
|
69
|
-
sift-128-euclidean
|
|
70
|
-
------------------
|
|
71
|
-
|
|
72
|
-

|
|
73
|
-
|
|
74
|
-
fashion-mnist-784-euclidean
|
|
75
|
-
---------------------------
|
|
76
|
-
|
|
77
|
-

|
|
78
|
-
|
|
79
|
-
lastfm-64-dot
|
|
80
|
-
------------------
|
|
81
|
-
|
|
82
|
-

|
|
83
|
-
|
|
84
|
-
nytimes-256-angular
|
|
85
|
-
-------------------
|
|
86
|
-
|
|
87
|
-

|
|
88
|
-
|
|
89
|
-
glove-25-angular
|
|
90
|
-
----------------
|
|
91
|
-
|
|
92
|
-

|
|
93
|
-
|
|
94
|
-
Install
|
|
95
|
-
=======
|
|
96
|
-
|
|
97
|
-
The only prerequisite is Python (tested with 3.6) and Docker.
|
|
98
|
-
|
|
99
|
-
1. Clone the repo.
|
|
100
|
-
2. Run `pip install -r requirements.txt`.
|
|
101
|
-
3. Run `python install.py` to build all the libraries inside Docker containers (this can take a while, like 10-30 minutes).
|
|
102
|
-
|
|
103
|
-
Running
|
|
104
|
-
=======
|
|
105
|
-
|
|
106
|
-
1. Run `python run.py` (this can take an extremely long time, potentially days)
|
|
107
|
-
2. Run `python plot.py` or `python create_website.py` to plot results.
|
|
108
|
-
|
|
109
|
-
You can customize the algorithms and datasets if you want to:
|
|
110
|
-
|
|
111
|
-
* Check that `algos.yaml` contains the parameter settings that you want to test
|
|
112
|
-
* To run experiments on SIFT, invoke `python run.py --dataset glove-100-angular`. See `python run.py --help` for more information on possible settings. Note that experiments can take a long time.
|
|
113
|
-
* To process the results, either use `python plot.py --dataset glove-100-angular` or `python create_website.py`. An example call: `python create_website.py --plottype recall/time --latex --scatter --outputdir website/`.
|
|
114
|
-
|
|
115
|
-
Including your algorithm
|
|
116
|
-
========================
|
|
117
|
-
|
|
118
|
-
1. Add your algorithm into `ann_benchmarks/algorithms` by providing a small Python wrapper.
|
|
119
|
-
2. Add a Dockerfile in `install/` for it
|
|
120
|
-
3. Add it to `algos.yaml`
|
|
121
|
-
4. Add it to `.github/workflows/benchmarks.yml`
|
|
122
|
-
|
|
123
|
-
Principles
|
|
124
|
-
==========
|
|
125
|
-
|
|
126
|
-
* Everyone is welcome to submit pull requests with tweaks and changes to how each library is being used.
|
|
127
|
-
* In particular: if you are the author of any of these libraries, and you think the benchmark can be improved, consider making the improvement and submitting a pull request.
|
|
128
|
-
* This is meant to be an ongoing project and represent the current state.
|
|
129
|
-
* Make everything easy to replicate, including installing and preparing the datasets.
|
|
130
|
-
* Try many different values of parameters for each library and ignore the points that are not on the precision-performance frontier.
|
|
131
|
-
* High-dimensional datasets with approximately 100-1000 dimensions. This is challenging but also realistic. Not more than 1000 dimensions because those problems should probably be solved by doing dimensionality reduction separately.
|
|
132
|
-
* Single queries are used by default. ANN-Benchmarks enforces that only one CPU is saturated during experimentation, i.e., no multi-threading. A batch mode is available that provides all queries to the implementations at once. Add the flag `--batch` to `run.py` and `plot.py` to enable batch mode.
|
|
133
|
-
* Avoid extremely costly index building (more than several hours).
|
|
134
|
-
* Focus on datasets that fit in RAM. For billion-scale benchmarks, see the related [big-ann-benchmarks](https://github.com/harsha-simhadri/big-ann-benchmarks) project.
|
|
135
|
-
* We mainly support CPU-based ANN algorithms. GPU support exists for FAISS, but it has to be compiled with GPU support locally and experiments must be run using the flags `--local --batch`.
|
|
136
|
-
* Do proper train/test set of index data and query points.
|
|
137
|
-
* Note that we consider that set similarity datasets are sparse and thus we pass a **sorted** array of integers to algorithms to represent the set of each user.
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
Authors
|
|
141
|
-
=======
|
|
142
|
-
|
|
143
|
-
Built by [Erik Bernhardsson](https://erikbern.com) with significant contributions from [Martin Aumüller](http://itu.dk/people/maau/) and [Alexander Faithfull](https://github.com/ale-f).
|
|
144
|
-
|
|
145
|
-
Related Publication
|
|
146
|
-
==================
|
|
147
|
-
|
|
148
|
-
The following publication details design principles behind the benchmarking framework:
|
|
149
|
-
|
|
150
|
-
- M. Aumüller, E. Bernhardsson, A. Faithfull:
|
|
151
|
-
[ANN-Benchmarks: A Benchmarking Tool for Approximate Nearest Neighbor Algorithms](https://arxiv.org/abs/1807.05614). Information Systems 2019. DOI: [10.1016/j.is.2019.02.006](https://doi.org/10.1016/j.is.2019.02.006)
|
|
152
|
-
|
|
153
|
-
Related Projects
|
|
154
|
-
================
|
|
155
|
-
|
|
156
|
-
- [big-ann-benchmarks](https://github.com/harsha-simhadri/big-ann-benchmarks) is a benchmarking effort for billion-scale approximate nearest neighbor search as part of the [NeurIPS'21 Competition track](https://neurips.cc/Conferences/2021/CompetitionTrack).
|
|
157
|
-
|