endee-llamaindex 0.1.3__tar.gz → 0.1.5a1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: endee-llamaindex
3
- Version: 0.1.3
3
+ Version: 0.1.5a1
4
4
  Summary: Vector Database for Fast ANN Searches
5
5
  Home-page: https://endee.io
6
6
  Author: Endee Labs
@@ -11,13 +11,17 @@ Classifier: Operating System :: OS Independent
11
11
  Requires-Python: >=3.6
12
12
  Description-Content-Type: text/markdown
13
13
  Requires-Dist: llama-index>=0.12.34
14
- Requires-Dist: endee>=0.1.4
14
+ Requires-Dist: endee==0.1.9
15
+ Requires-Dist: fastembed>=0.3.0
16
+ Provides-Extra: gpu
17
+ Requires-Dist: fastembed-gpu>=0.3.0; extra == "gpu"
15
18
  Dynamic: author
16
19
  Dynamic: author-email
17
20
  Dynamic: classifier
18
21
  Dynamic: description
19
22
  Dynamic: description-content-type
20
23
  Dynamic: home-page
24
+ Dynamic: provides-extra
21
25
  Dynamic: requires-dist
22
26
  Dynamic: requires-python
23
27
  Dynamic: summary
@@ -31,18 +35,19 @@ Build powerful RAG applications with Endee vector database and LlamaIndex.
31
35
  ## Table of Contents
32
36
 
33
37
  1. [Installation](#1-installation)
34
- 2. [Setting up Credentials](#2-setting-up-endee-and-openai-credentials)
35
- 3. [Creating Sample Documents](#3-creating-sample-documents)
36
- 4. [Setting up Endee with LlamaIndex](#4-setting-up-endee-with-llamaindex)
37
- 5. [Creating a Vector Index](#5-creating-a-vector-index-from-documents)
38
- 6. [Basic Retrieval](#6-basic-retrieval-with-query-engine)
39
- 7. [Using Metadata Filters](#7-using-metadata-filters)
40
- 8. [Advanced Filtering](#8-advanced-filtering-with-multiple-conditions)
41
- 9. [Custom Retriever Setup](#9-custom-retriever-setup)
42
- 10. [Custom Retriever with Query Engine](#10-using-a-custom-retriever-with-a-query-engine)
43
- 11. [Direct VectorStore Querying](#11-direct-vectorstore-querying)
44
- 12. [Saving and Loading Indexes](#12-saving-and-loading-indexes)
45
- 13. [Cleanup](#13-cleanup)
38
+ 2. [Testing locally](#testing-locally)
39
+ 3. [Setting up Credentials](#2-setting-up-endee-and-openai-credentials)
40
+ 4. [Creating Sample Documents](#3-creating-sample-documents)
41
+ 5. [Setting up Endee with LlamaIndex](#4-setting-up-endee-with-llamaindex)
42
+ 6. [Creating a Vector Index](#5-creating-a-vector-index-from-documents)
43
+ 7. [Basic Retrieval](#6-basic-retrieval-with-query-engine)
44
+ 8. [Using Metadata Filters](#7-using-metadata-filters)
45
+ 9. [Advanced Filtering](#8-advanced-filtering-with-multiple-conditions)
46
+ 10. [Custom Retriever Setup](#9-custom-retriever-setup)
47
+ 11. [Custom Retriever with Query Engine](#10-using-a-custom-retriever-with-a-query-engine)
48
+ 12. [Direct VectorStore Querying](#11-direct-vectorstore-querying)
49
+ 13. [Saving and Loading Indexes](#12-saving-and-loading-indexes)
50
+ 14. [Cleanup](#13-cleanup)
46
51
 
47
52
  ---
48
53
 
@@ -50,12 +55,66 @@ Build powerful RAG applications with Endee vector database and LlamaIndex.
50
55
 
51
56
  Get started by installing the required package.
52
57
 
58
+ ### Basic Installation (Dense-only search)
59
+
53
60
  ```bash
54
61
  pip install endee-llamaindex
55
62
  ```
56
63
 
57
64
  > **Note:** This will automatically install `endee` and `llama-index` as dependencies.
58
65
 
66
+ ### Full Installation (with Hybrid Search support)
67
+
68
+ For hybrid search capabilities (dense + sparse vectors), install with the `hybrid` extra:
69
+
70
+ ```bash
71
+ pip install endee-llamaindex[hybrid]
72
+ ```
73
+
74
+ This includes FastEmbed for sparse vector encoding (SPLADE, BM25, etc.).
75
+
76
+ ### GPU-Accelerated Hybrid Search
77
+
78
+ For GPU-accelerated sparse encoding:
79
+
80
+ ```bash
81
+ pip install endee-llamaindex[hybrid-gpu]
82
+ ```
83
+
84
+ ### All Features
85
+
86
+ To install all optional dependencies:
87
+
88
+ ```bash
89
+ pip install endee-llamaindex[all]
90
+ ```
91
+
92
+ ### Installation Options Summary
93
+
94
+ | Installation | Use Case | Includes |
95
+ |--------------|----------|----------|
96
+ | `pip install endee-llamaindex` | Dense vector search only | Core dependencies |
97
+ | `pip install endee-llamaindex[hybrid]` | Dense + sparse hybrid search | + FastEmbed (CPU) |
98
+ | `pip install endee-llamaindex[hybrid-gpu]` | GPU-accelerated hybrid search | + FastEmbed (GPU) |
99
+ | `pip install endee-llamaindex[all]` | All features | All optional deps |
100
+
101
+ ---
102
+
103
+ ## Testing locally
104
+
105
+ From the project root:
106
+
107
+ ```bash
108
+ python -m venv env && source env/bin/activate # optional
109
+ pip install -e .
110
+ pip install pytest sentence-transformers huggingface-hub
111
+ export ENDEE_API_TOKEN="your-endee-api-token" # or set in endee_llamaindex/test_cases/setup_class.py
112
+
113
+ cd endee_llamaindex/test_cases && PYTHONPATH=.. python -m pytest . -v
114
+ ```
115
+
116
+ See [TESTING.md](TESTING.md) for more options and single-test runs.
117
+
59
118
  ---
60
119
 
61
120
  ## 2. Setting up Endee and OpenAI credentials
@@ -145,7 +204,7 @@ vector_store = EndeeVectorStore.from_params(
145
204
  index_name=index_name,
146
205
  dimension=dimension,
147
206
  space_type="cosine", # Can be "cosine", "l2", or "ip"
148
- precision="medium" # Index precision: "low", "medium", "high", or None
207
+ precision="float16" # Options: "binary", "float16", "float32", "int16d", "int8d" (default: "float16")
149
208
  )
150
209
 
151
210
  # Create storage context with our vector store
@@ -160,9 +219,11 @@ print(f"Initialized Endee vector store with index: {index_name}")
160
219
  |-----------|-------------|---------|
161
220
  | `space_type` | Distance metric for similarity | `cosine`, `l2`, `ip` |
162
221
  | `dimension` | Vector dimension | Must match embedding model |
163
- | `precision` | Index precision setting | `"low"`, `"medium"` (default), `"high"`, or `None` |
164
- | `key` | Encryption key for metadata | 256-bit hex key (64 hex characters) |
222
+ | `precision` | Index precision setting | `"binary"`, `"float16"` (default), `"float32"`, `"int16d"`, `"int8d"` |
165
223
  | `batch_size` | Vectors per API call | Default: `100` |
224
+ | `hybrid` | Enable hybrid search (dense + sparse) | Default: `False` |
225
+ | `M` | Optional HNSW M parameter (bi-directional links) | Optional (backend default if not specified) |
226
+ | `ef_con` | Optional HNSW ef_construction parameter | Optional (backend default if not specified) |
166
227
 
167
228
  ---
168
229
 
@@ -239,16 +300,47 @@ print(response)
239
300
 
240
301
  ### Available Filter Operators
241
302
 
242
- | Operator | Description |
243
- |----------|-------------|
244
- | `FilterOperator.EQ` | Equal to |
245
- | `FilterOperator.NE` | Not equal to |
246
- | `FilterOperator.GT` | Greater than |
247
- | `FilterOperator.GTE` | Greater than or equal |
248
- | `FilterOperator.LT` | Less than |
249
- | `FilterOperator.LTE` | Less than or equal |
250
- | `FilterOperator.IN` | In list |
251
- | `FilterOperator.NIN` | Not in list |
303
+ | Operator | Description | Backend Symbol | Example |
304
+ |----------|-------------|----------------|---------|
305
+ | `FilterOperator.EQ` | Equal to | `$eq` | `rating == 5` |
306
+ | `FilterOperator.IN` | In list | `$in` | `category in ["ai", "ml"]` |
307
+
308
+
309
+ > **Important Notes:**
310
+ > - Currently, the Endee LlamaIndex integration only supports **EQ** and **IN** metadata filters.
311
+ > - Range-style operators (LT, LTE, GT, GTE) are **not** supported in this adapter.
312
+
313
+ ### Filter Examples
314
+
315
+ Here are practical examples showing how to use the supported filter operators:
316
+
317
+ ```python
318
+ from llama_index.core.vector_stores.types import MetadataFilters, MetadataFilter, FilterOperator
319
+
320
+ # Example 1: Equal to (EQ)
321
+ # Find documents with rating equal to 5
322
+ rating_filter = MetadataFilter(key="rating", value=5, operator=FilterOperator.EQ)
323
+ filters = MetadataFilters(filters=[rating_filter])
324
+ # Backend: {"rating": {"$eq": 5}}
325
+
326
+ # Example 2: In list (IN)
327
+ # Find documents in AI or ML categories
328
+ category_filter = MetadataFilter(key="category", value=["ai", "ml"], operator=FilterOperator.IN)
329
+ filters = MetadataFilters(filters=[category_filter])
330
+ # Backend: {"category": {"$in": ["ai", "ml"]}}
331
+
332
+ # Example 3: Combined filters (AND logic)
333
+ # Find AI documents with rating equal to 5
334
+ filters = MetadataFilters(filters=[
335
+ MetadataFilter(key="category", value="ai", operator=FilterOperator.EQ),
336
+ MetadataFilter(key="rating", value=5, operator=FilterOperator.EQ)
337
+ ])
338
+ # Backend: [{"category": {"$eq": "ai"}}, {"rating": {"$eq": 5}}]
339
+
340
+ # Create a query engine with filters
341
+ filtered_engine = index.as_query_engine(filters=filters)
342
+ response = filtered_engine.query("What is machine learning?")
343
+ ```
252
344
 
253
345
  ---
254
346
 
@@ -439,10 +531,14 @@ Delete the index when you're done to free up resources.
439
531
  | `api_token` | `str` | Your Endee API token | Required |
440
532
  | `index_name` | `str` | Name of the index | Required |
441
533
  | `dimension` | `int` | Vector dimension | Required |
442
- | `space_type` | `str` | Distance metric | `"cosine"` |
443
- | `precision` | `str` | Index precision setting | `"medium"` |
444
- | `key` | `str` | Encryption key for metadata (256-bit hex) | `None` |
534
+ | `space_type` | `str` | Distance metric (`"cosine"`, `"l2"`, `"ip"`) | `"cosine"` |
535
+ | `precision` | `str` | Index precision (`"binary"`, `"float16"`, `"float32"`, `"int16d"`, `"int8d"`) | `"float16"` |
445
536
  | `batch_size` | `int` | Vectors per API call | `100` |
537
+ | `hybrid` | `bool` | Enable hybrid search (dense + sparse vectors) | `False` |
538
+ | `sparse_dim` | `int` | Sparse dimension for hybrid index | `None` |
539
+ | `model_name` | `str` | Model name for sparse embeddings (e.g., `'splade_pp'`, `'bert_base'`) | `None` |
540
+ | `M` | `int` | Optional HNSW M parameter (bi-directional links per node) | `None` (backend default) |
541
+ | `ef_con` | `int` | Optional HNSW ef_construction parameter | `None` (backend default) |
446
542
 
447
543
  ### Distance Metrics
448
544
 
@@ -454,38 +550,39 @@ Delete the index when you're done to free up resources.
454
550
 
455
551
  ### Precision Settings
456
552
 
457
- The `precision` parameter controls the trade-off between search accuracy and performance:
553
+ The `precision` parameter controls the vector storage format and affects memory usage and search performance:
458
554
 
459
555
  | Precision | Description | Use Case |
460
556
  |-----------|-------------|----------|
461
- | `"low"` | Faster searches, lower accuracy | Large-scale applications where speed is critical |
462
- | `"medium"` | Balanced performance and accuracy | General purpose applications (default) |
463
- | `"high"` | Slower searches, higher accuracy | Applications requiring maximum precision |
464
- | `None` | Default system precision | Use system defaults |
557
+ | `"float32"` | Full precision floating point | Maximum accuracy, higher memory usage |
558
+ | `"float16"` | Half precision floating point | Balanced accuracy and memory (default) |
559
+ | `"binary"` | Binary vectors | Extremely compact, best for binary embeddings |
560
+ | `"int8d"` | 8-bit integer quantization | High compression, good accuracy |
561
+ | `"int16d"` | 16-bit integer quantization | Better accuracy than int8d, moderate compression |
465
562
 
466
- ### Encryption Support
563
+ ### HNSW Parameters (Optional)
467
564
 
468
- You can encrypt metadata stored in Endee by providing a 256-bit encryption key (64 hex characters). This ensures sensitive information is encrypted at rest.
565
+ HNSW (Hierarchical Navigable Small World) parameters control index construction and search quality. These are **optional** - if not provided, the Endee backend uses optimized defaults.
469
566
 
470
- ```python
471
- # Generate a 256-bit key (example - use a secure method in production)
472
- import secrets
473
- encryption_key = secrets.token_hex(32) # 32 bytes = 64 hex characters
567
+ | Parameter | Description | Impact |
568
+ |-----------|-------------|--------|
569
+ | `M` | Number of bi-directional links per node | Higher M = better recall, more memory |
570
+ | `ef_con` | Size of dynamic candidate list during construction | Higher ef_con = better quality, slower indexing |
571
+
572
+ **Example with custom HNSW parameters:**
474
573
 
475
- # Create vector store with encryption
574
+ ```python
476
575
  vector_store = EndeeVectorStore.from_params(
477
- api_token=endee_api_token,
478
- index_name=index_name,
479
- dimension=dimension,
576
+ api_token="your-token",
577
+ index_name="custom_index",
578
+ dimension=384,
480
579
  space_type="cosine",
481
- precision="medium",
482
- key=encryption_key # Metadata will be encrypted
580
+ M=32, # Optional: custom M value
581
+ ef_con=256 # Optional: custom ef_construction
483
582
  )
484
-
485
- # Important: Store this key securely! You'll need it to access the index later.
486
583
  ```
487
584
 
488
- > **Warning:** If you lose the encryption key, you will not be able to decrypt your metadata. Store it securely (e.g., in a secrets manager).
585
+ **Note:** Only specify M and ef_con if you need to fine-tune performance. The backend defaults work well for most use cases.
489
586
 
490
587
  ---
491
588
 
@@ -7,18 +7,19 @@ Build powerful RAG applications with Endee vector database and LlamaIndex.
7
7
  ## Table of Contents
8
8
 
9
9
  1. [Installation](#1-installation)
10
- 2. [Setting up Credentials](#2-setting-up-endee-and-openai-credentials)
11
- 3. [Creating Sample Documents](#3-creating-sample-documents)
12
- 4. [Setting up Endee with LlamaIndex](#4-setting-up-endee-with-llamaindex)
13
- 5. [Creating a Vector Index](#5-creating-a-vector-index-from-documents)
14
- 6. [Basic Retrieval](#6-basic-retrieval-with-query-engine)
15
- 7. [Using Metadata Filters](#7-using-metadata-filters)
16
- 8. [Advanced Filtering](#8-advanced-filtering-with-multiple-conditions)
17
- 9. [Custom Retriever Setup](#9-custom-retriever-setup)
18
- 10. [Custom Retriever with Query Engine](#10-using-a-custom-retriever-with-a-query-engine)
19
- 11. [Direct VectorStore Querying](#11-direct-vectorstore-querying)
20
- 12. [Saving and Loading Indexes](#12-saving-and-loading-indexes)
21
- 13. [Cleanup](#13-cleanup)
10
+ 2. [Testing locally](#testing-locally)
11
+ 3. [Setting up Credentials](#2-setting-up-endee-and-openai-credentials)
12
+ 4. [Creating Sample Documents](#3-creating-sample-documents)
13
+ 5. [Setting up Endee with LlamaIndex](#4-setting-up-endee-with-llamaindex)
14
+ 6. [Creating a Vector Index](#5-creating-a-vector-index-from-documents)
15
+ 7. [Basic Retrieval](#6-basic-retrieval-with-query-engine)
16
+ 8. [Using Metadata Filters](#7-using-metadata-filters)
17
+ 9. [Advanced Filtering](#8-advanced-filtering-with-multiple-conditions)
18
+ 10. [Custom Retriever Setup](#9-custom-retriever-setup)
19
+ 11. [Custom Retriever with Query Engine](#10-using-a-custom-retriever-with-a-query-engine)
20
+ 12. [Direct VectorStore Querying](#11-direct-vectorstore-querying)
21
+ 13. [Saving and Loading Indexes](#12-saving-and-loading-indexes)
22
+ 14. [Cleanup](#13-cleanup)
22
23
 
23
24
  ---
24
25
 
@@ -26,12 +27,66 @@ Build powerful RAG applications with Endee vector database and LlamaIndex.
26
27
 
27
28
  Get started by installing the required package.
28
29
 
30
+ ### Basic Installation (Dense-only search)
31
+
29
32
  ```bash
30
33
  pip install endee-llamaindex
31
34
  ```
32
35
 
33
36
  > **Note:** This will automatically install `endee` and `llama-index` as dependencies.
34
37
 
38
+ ### Full Installation (with Hybrid Search support)
39
+
40
+ For hybrid search capabilities (dense + sparse vectors), install with the `hybrid` extra:
41
+
42
+ ```bash
43
+ pip install endee-llamaindex[hybrid]
44
+ ```
45
+
46
+ This includes FastEmbed for sparse vector encoding (SPLADE, BM25, etc.).
47
+
48
+ ### GPU-Accelerated Hybrid Search
49
+
50
+ For GPU-accelerated sparse encoding:
51
+
52
+ ```bash
53
+ pip install endee-llamaindex[hybrid-gpu]
54
+ ```
55
+
56
+ ### All Features
57
+
58
+ To install all optional dependencies:
59
+
60
+ ```bash
61
+ pip install endee-llamaindex[all]
62
+ ```
63
+
64
+ ### Installation Options Summary
65
+
66
+ | Installation | Use Case | Includes |
67
+ |--------------|----------|----------|
68
+ | `pip install endee-llamaindex` | Dense vector search only | Core dependencies |
69
+ | `pip install endee-llamaindex[hybrid]` | Dense + sparse hybrid search | + FastEmbed (CPU) |
70
+ | `pip install endee-llamaindex[hybrid-gpu]` | GPU-accelerated hybrid search | + FastEmbed (GPU) |
71
+ | `pip install endee-llamaindex[all]` | All features | All optional deps |
72
+
73
+ ---
74
+
75
+ ## Testing locally
76
+
77
+ From the project root:
78
+
79
+ ```bash
80
+ python -m venv env && source env/bin/activate # optional
81
+ pip install -e .
82
+ pip install pytest sentence-transformers huggingface-hub
83
+ export ENDEE_API_TOKEN="your-endee-api-token" # or set in endee_llamaindex/test_cases/setup_class.py
84
+
85
+ cd endee_llamaindex/test_cases && PYTHONPATH=.. python -m pytest . -v
86
+ ```
87
+
88
+ See [TESTING.md](TESTING.md) for more options and single-test runs.
89
+
35
90
  ---
36
91
 
37
92
  ## 2. Setting up Endee and OpenAI credentials
@@ -121,7 +176,7 @@ vector_store = EndeeVectorStore.from_params(
121
176
  index_name=index_name,
122
177
  dimension=dimension,
123
178
  space_type="cosine", # Can be "cosine", "l2", or "ip"
124
- precision="medium" # Index precision: "low", "medium", "high", or None
179
+ precision="float16" # Options: "binary", "float16", "float32", "int16d", "int8d" (default: "float16")
125
180
  )
126
181
 
127
182
  # Create storage context with our vector store
@@ -136,9 +191,11 @@ print(f"Initialized Endee vector store with index: {index_name}")
136
191
  |-----------|-------------|---------|
137
192
  | `space_type` | Distance metric for similarity | `cosine`, `l2`, `ip` |
138
193
  | `dimension` | Vector dimension | Must match embedding model |
139
- | `precision` | Index precision setting | `"low"`, `"medium"` (default), `"high"`, or `None` |
140
- | `key` | Encryption key for metadata | 256-bit hex key (64 hex characters) |
194
+ | `precision` | Index precision setting | `"binary"`, `"float16"` (default), `"float32"`, `"int16d"`, `"int8d"` |
141
195
  | `batch_size` | Vectors per API call | Default: `100` |
196
+ | `hybrid` | Enable hybrid search (dense + sparse) | Default: `False` |
197
+ | `M` | Optional HNSW M parameter (bi-directional links) | Optional (backend default if not specified) |
198
+ | `ef_con` | Optional HNSW ef_construction parameter | Optional (backend default if not specified) |
142
199
 
143
200
  ---
144
201
 
@@ -215,16 +272,47 @@ print(response)
215
272
 
216
273
  ### Available Filter Operators
217
274
 
218
- | Operator | Description |
219
- |----------|-------------|
220
- | `FilterOperator.EQ` | Equal to |
221
- | `FilterOperator.NE` | Not equal to |
222
- | `FilterOperator.GT` | Greater than |
223
- | `FilterOperator.GTE` | Greater than or equal |
224
- | `FilterOperator.LT` | Less than |
225
- | `FilterOperator.LTE` | Less than or equal |
226
- | `FilterOperator.IN` | In list |
227
- | `FilterOperator.NIN` | Not in list |
275
+ | Operator | Description | Backend Symbol | Example |
276
+ |----------|-------------|----------------|---------|
277
+ | `FilterOperator.EQ` | Equal to | `$eq` | `rating == 5` |
278
+ | `FilterOperator.IN` | In list | `$in` | `category in ["ai", "ml"]` |
279
+
280
+
281
+ > **Important Notes:**
282
+ > - Currently, the Endee LlamaIndex integration only supports **EQ** and **IN** metadata filters.
283
+ > - Range-style operators (LT, LTE, GT, GTE) are **not** supported in this adapter.
284
+
285
+ ### Filter Examples
286
+
287
+ Here are practical examples showing how to use the supported filter operators:
288
+
289
+ ```python
290
+ from llama_index.core.vector_stores.types import MetadataFilters, MetadataFilter, FilterOperator
291
+
292
+ # Example 1: Equal to (EQ)
293
+ # Find documents with rating equal to 5
294
+ rating_filter = MetadataFilter(key="rating", value=5, operator=FilterOperator.EQ)
295
+ filters = MetadataFilters(filters=[rating_filter])
296
+ # Backend: {"rating": {"$eq": 5}}
297
+
298
+ # Example 2: In list (IN)
299
+ # Find documents in AI or ML categories
300
+ category_filter = MetadataFilter(key="category", value=["ai", "ml"], operator=FilterOperator.IN)
301
+ filters = MetadataFilters(filters=[category_filter])
302
+ # Backend: {"category": {"$in": ["ai", "ml"]}}
303
+
304
+ # Example 3: Combined filters (AND logic)
305
+ # Find AI documents with rating equal to 5
306
+ filters = MetadataFilters(filters=[
307
+ MetadataFilter(key="category", value="ai", operator=FilterOperator.EQ),
308
+ MetadataFilter(key="rating", value=5, operator=FilterOperator.EQ)
309
+ ])
310
+ # Backend: [{"category": {"$eq": "ai"}}, {"rating": {"$eq": 5}}]
311
+
312
+ # Create a query engine with filters
313
+ filtered_engine = index.as_query_engine(filters=filters)
314
+ response = filtered_engine.query("What is machine learning?")
315
+ ```
228
316
 
229
317
  ---
230
318
 
@@ -415,10 +503,14 @@ Delete the index when you're done to free up resources.
415
503
  | `api_token` | `str` | Your Endee API token | Required |
416
504
  | `index_name` | `str` | Name of the index | Required |
417
505
  | `dimension` | `int` | Vector dimension | Required |
418
- | `space_type` | `str` | Distance metric | `"cosine"` |
419
- | `precision` | `str` | Index precision setting | `"medium"` |
420
- | `key` | `str` | Encryption key for metadata (256-bit hex) | `None` |
506
+ | `space_type` | `str` | Distance metric (`"cosine"`, `"l2"`, `"ip"`) | `"cosine"` |
507
+ | `precision` | `str` | Index precision (`"binary"`, `"float16"`, `"float32"`, `"int16d"`, `"int8d"`) | `"float16"` |
421
508
  | `batch_size` | `int` | Vectors per API call | `100` |
509
+ | `hybrid` | `bool` | Enable hybrid search (dense + sparse vectors) | `False` |
510
+ | `sparse_dim` | `int` | Sparse dimension for hybrid index | `None` |
511
+ | `model_name` | `str` | Model name for sparse embeddings (e.g., `'splade_pp'`, `'bert_base'`) | `None` |
512
+ | `M` | `int` | Optional HNSW M parameter (bi-directional links per node) | `None` (backend default) |
513
+ | `ef_con` | `int` | Optional HNSW ef_construction parameter | `None` (backend default) |
422
514
 
423
515
  ### Distance Metrics
424
516
 
@@ -430,38 +522,39 @@ Delete the index when you're done to free up resources.
430
522
 
431
523
  ### Precision Settings
432
524
 
433
- The `precision` parameter controls the trade-off between search accuracy and performance:
525
+ The `precision` parameter controls the vector storage format and affects memory usage and search performance:
434
526
 
435
527
  | Precision | Description | Use Case |
436
528
  |-----------|-------------|----------|
437
- | `"low"` | Faster searches, lower accuracy | Large-scale applications where speed is critical |
438
- | `"medium"` | Balanced performance and accuracy | General purpose applications (default) |
439
- | `"high"` | Slower searches, higher accuracy | Applications requiring maximum precision |
440
- | `None` | Default system precision | Use system defaults |
529
+ | `"float32"` | Full precision floating point | Maximum accuracy, higher memory usage |
530
+ | `"float16"` | Half precision floating point | Balanced accuracy and memory (default) |
531
+ | `"binary"` | Binary vectors | Extremely compact, best for binary embeddings |
532
+ | `"int8d"` | 8-bit integer quantization | High compression, good accuracy |
533
+ | `"int16d"` | 16-bit integer quantization | Better accuracy than int8d, moderate compression |
441
534
 
442
- ### Encryption Support
535
+ ### HNSW Parameters (Optional)
443
536
 
444
- You can encrypt metadata stored in Endee by providing a 256-bit encryption key (64 hex characters). This ensures sensitive information is encrypted at rest.
537
+ HNSW (Hierarchical Navigable Small World) parameters control index construction and search quality. These are **optional** - if not provided, the Endee backend uses optimized defaults.
445
538
 
446
- ```python
447
- # Generate a 256-bit key (example - use a secure method in production)
448
- import secrets
449
- encryption_key = secrets.token_hex(32) # 32 bytes = 64 hex characters
539
+ | Parameter | Description | Impact |
540
+ |-----------|-------------|--------|
541
+ | `M` | Number of bi-directional links per node | Higher M = better recall, more memory |
542
+ | `ef_con` | Size of dynamic candidate list during construction | Higher ef_con = better quality, slower indexing |
543
+
544
+ **Example with custom HNSW parameters:**
450
545
 
451
- # Create vector store with encryption
546
+ ```python
452
547
  vector_store = EndeeVectorStore.from_params(
453
- api_token=endee_api_token,
454
- index_name=index_name,
455
- dimension=dimension,
548
+ api_token="your-token",
549
+ index_name="custom_index",
550
+ dimension=384,
456
551
  space_type="cosine",
457
- precision="medium",
458
- key=encryption_key # Metadata will be encrypted
552
+ M=32, # Optional: custom M value
553
+ ef_con=256 # Optional: custom ef_construction
459
554
  )
460
-
461
- # Important: Store this key securely! You'll need it to access the index later.
462
555
  ```
463
556
 
464
- > **Warning:** If you lose the encryption key, you will not be able to decrypt your metadata. Store it securely (e.g., in a secrets manager).
557
+ **Note:** Only specify M and ef_con if you need to fine-tune performance. The backend defaults work well for most use cases.
465
558
 
466
559
  ---
467
560