vectorizer-sdk 3.0.3__tar.gz → 3.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/PKG-INFO +55 -6
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/README.md +54 -5
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/pyproject.toml +1 -1
- vectorizer_sdk-3.2.0/tests/test_retry_after_parse.py +51 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/utils/http_client.py +88 -29
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/vectorizer_sdk.egg-info/PKG-INFO +55 -6
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/vectorizer_sdk.egg-info/SOURCES.txt +1 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/LICENSE +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/rpc/__init__.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/rpc/_codec.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/rpc/async_client.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/rpc/commands.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/rpc/endpoint.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/rpc/pool.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/rpc/sync_client.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/rpc/types.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/setup.cfg +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/tests/test_client_integration.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/tests/test_discovery.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/tests/test_exceptions.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/tests/test_file_operations.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/tests/test_file_upload.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/tests/test_graph.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/tests/test_http_client.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/tests/test_intelligent_search.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/tests/test_mock_transport.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/tests/test_models.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/tests/test_qdrant_advanced.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/tests/test_routing.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/tests/test_sdk_comprehensive.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/tests/test_simple.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/tests/test_umicp.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/tests/test_validation.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/utils/__init__.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/utils/transport.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/utils/umicp_client.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/utils/validation.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/vectorizer/__init__.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/vectorizer/_base.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/vectorizer/admin.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/vectorizer/auth.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/vectorizer/client.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/vectorizer/collections.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/vectorizer/graph.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/vectorizer/search.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/vectorizer/vectors.py +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/vectorizer_sdk.egg-info/dependency_links.txt +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/vectorizer_sdk.egg-info/entry_points.txt +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/vectorizer_sdk.egg-info/requires.txt +0 -0
- {vectorizer_sdk-3.0.3 → vectorizer_sdk-3.2.0}/vectorizer_sdk.egg-info/top_level.txt +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: vectorizer_sdk
|
|
3
|
-
Version: 3.0
|
|
3
|
+
Version: 3.2.0
|
|
4
4
|
Summary: Python SDK for Vectorizer - Semantic search, vector operations, and the VectorizerRPC binary transport (default in v3.x)
|
|
5
5
|
Author-email: HiveLLM Team <team@hivellm.org>
|
|
6
6
|
License: Apache-2.0
|
|
@@ -58,9 +58,38 @@ Dynamic: license-file
|
|
|
58
58
|
A comprehensive Python SDK for the Vectorizer semantic search service.
|
|
59
59
|
|
|
60
60
|
**Package**: `vectorizer_sdk` (PEP 625 compliant)
|
|
61
|
-
**Version**: 3.
|
|
61
|
+
**Version**: 3.2.0
|
|
62
62
|
**PyPI**: https://pypi.org/project/vectorizer-sdk/
|
|
63
63
|
|
|
64
|
+
## v3.2 — backpressure-aware client (HTTP 429 + `Retry-After`)
|
|
65
|
+
|
|
66
|
+
The REST `VectorizerClient` honors server-side bulk-upsert
|
|
67
|
+
backpressure shipped in Vectorizer 3.2.0
|
|
68
|
+
([#263](https://github.com/hivellm/vectorizer/issues/263)). On HTTP
|
|
69
|
+
`429 Too Many Requests` the client parses `Retry-After` (seconds
|
|
70
|
+
form, 1 s default, 30 s cap), sleeps, and retries up to 3 times
|
|
71
|
+
before raising a typed `RateLimitError`. Pre-3.2.0 clients bounced
|
|
72
|
+
429s into a generic 5xx and lost the retry budget. Identical
|
|
73
|
+
semantics ship in every first-party SDK (Rust, Python, TypeScript,
|
|
74
|
+
Go, C#) — see `tests/test_retry_after_parse.py`.
|
|
75
|
+
|
|
76
|
+
## v3.1 — `/insert_vectors` + stable client-id upserts
|
|
77
|
+
|
|
78
|
+
- `insert_vectors(collection, vectors, public_key=None)` — bulk-
|
|
79
|
+
insert pre-computed embeddings with caller-supplied vector ids.
|
|
80
|
+
Skips the embedding pipeline entirely.
|
|
81
|
+
- `insert` / `insert_texts`: the request `id` is now used verbatim
|
|
82
|
+
as the stored `Vector.id` (non-chunked) or as `<id>#<chunk_index>`
|
|
83
|
+
(chunked). Re-running the same payload upserts in place instead
|
|
84
|
+
of duplicating.
|
|
85
|
+
- Chunked vectors expose a flat payload layout (`{content,
|
|
86
|
+
file_path, chunk_index, parent_id, ...user_metadata}`). Legacy
|
|
87
|
+
nested payloads from ≤ 3.0.x stay readable during the deprecation
|
|
88
|
+
window.
|
|
89
|
+
|
|
90
|
+
Client-id contract: non-empty, length ≤ 256, no leading/trailing
|
|
91
|
+
whitespace, must not contain `#`.
|
|
92
|
+
|
|
64
93
|
## v3.0 — VectorizerRPC is the default transport
|
|
65
94
|
|
|
66
95
|
Starting with v3.0, the recommended transport is **VectorizerRPC**: a
|
|
@@ -79,6 +108,8 @@ import vectorizer_sdk
|
|
|
79
108
|
|
|
80
109
|
async def main():
|
|
81
110
|
client = await vectorizer_sdk.connect_async("vectorizer://127.0.0.1:15503")
|
|
111
|
+
# `hello` and `search_basic` are RPC-only (not available on the legacy
|
|
112
|
+
# REST `VectorizerClient`).
|
|
82
113
|
await client.hello(vectorizer_sdk.HelloPayload(client_name="my-app"))
|
|
83
114
|
print(await client.list_collections())
|
|
84
115
|
hits = await client.search_basic("docs", "vector database", limit=5)
|
|
@@ -140,7 +171,7 @@ for a runnable end-to-end example.
|
|
|
140
171
|
pip install vectorizer-sdk
|
|
141
172
|
|
|
142
173
|
# Or specific version
|
|
143
|
-
pip install vectorizer-sdk==3.
|
|
174
|
+
pip install vectorizer-sdk==3.2.0
|
|
144
175
|
```
|
|
145
176
|
|
|
146
177
|
## Package Layout (v3.x)
|
|
@@ -477,6 +508,11 @@ related = await client.get_related_files(
|
|
|
477
508
|
|
|
478
509
|
### Summarization Operations
|
|
479
510
|
|
|
511
|
+
> WARNING: The `/summarize/*` REST endpoints are documented but not yet wired
|
|
512
|
+
> server-side (see `DOC_GAP_ANALYSIS`). The SDK methods below
|
|
513
|
+
> (`summarize_text`, `summarize_context`) will fail until server wiring is
|
|
514
|
+
> complete.
|
|
515
|
+
|
|
480
516
|
#### Summarize Text
|
|
481
517
|
Summarize text using various methods:
|
|
482
518
|
|
|
@@ -509,6 +545,11 @@ summary = await client.summarize_context(
|
|
|
509
545
|
|
|
510
546
|
### Workspace Management
|
|
511
547
|
|
|
548
|
+
> WARNING: `add_workspace`, `list_workspaces`, and `remove_workspace` are
|
|
549
|
+
> exposed via the REST transport through dynamic `__getattr__` delegation,
|
|
550
|
+
> but they are not first-class SDK methods yet. They work at runtime but
|
|
551
|
+
> won't autocomplete in IDEs. A future release will add explicit methods.
|
|
552
|
+
|
|
512
553
|
#### Add Workspace
|
|
513
554
|
Add a new workspace:
|
|
514
555
|
|
|
@@ -537,6 +578,11 @@ await client.remove_workspace(
|
|
|
537
578
|
|
|
538
579
|
### Backup Operations
|
|
539
580
|
|
|
581
|
+
> WARNING: `create_backup`, `list_backups`, and `restore_backup` are
|
|
582
|
+
> exposed via the REST transport through dynamic `__getattr__` delegation,
|
|
583
|
+
> but they are not first-class SDK methods yet. They work at runtime but
|
|
584
|
+
> won't autocomplete in IDEs. A future release will add explicit methods.
|
|
585
|
+
|
|
540
586
|
#### Create Backup
|
|
541
587
|
Create a backup of collections:
|
|
542
588
|
|
|
@@ -664,8 +710,11 @@ await client.create_collection("documents", dimension=768)
|
|
|
664
710
|
await client.insert_texts("documents", [
|
|
665
711
|
{"id": "doc1", "text": "Sample document", "metadata": {"source": "api"}}
|
|
666
712
|
])
|
|
667
|
-
|
|
668
|
-
await client.
|
|
713
|
+
# Update-via-reinsert: re-call `insert_texts` with the same id to replace the record.
|
|
714
|
+
await client.insert_texts("documents", [
|
|
715
|
+
{"id": "doc1", "text": "Sample document (updated)", "metadata": {"updated": True}}
|
|
716
|
+
])
|
|
717
|
+
await client.delete_vectors("documents", ["doc1"])
|
|
669
718
|
|
|
670
719
|
# Reads automatically go to replicas (load balanced)
|
|
671
720
|
results = await client.search_vectors("documents", query="sample", limit=10)
|
|
@@ -702,7 +751,7 @@ The SDK automatically classifies operations:
|
|
|
702
751
|
|
|
703
752
|
| Operation Type | Routed To | Methods |
|
|
704
753
|
|---------------|-----------|---------|
|
|
705
|
-
| **Writes** | Always Master | `insert_texts`, `insert_vectors`, `
|
|
754
|
+
| **Writes** | Always Master | `insert_texts`, `insert_vectors`, `delete_vectors`, `create_collection`, `delete_collection` |
|
|
706
755
|
| **Reads** | Based on `read_preference` | `search_vectors`, `get_vector`, `list_collections`, `intelligent_search`, `semantic_search`, `hybrid_search` |
|
|
707
756
|
|
|
708
757
|
#### Standalone Mode (Single Node)
|
|
@@ -7,9 +7,38 @@
|
|
|
7
7
|
A comprehensive Python SDK for the Vectorizer semantic search service.
|
|
8
8
|
|
|
9
9
|
**Package**: `vectorizer_sdk` (PEP 625 compliant)
|
|
10
|
-
**Version**: 3.
|
|
10
|
+
**Version**: 3.2.0
|
|
11
11
|
**PyPI**: https://pypi.org/project/vectorizer-sdk/
|
|
12
12
|
|
|
13
|
+
## v3.2 — backpressure-aware client (HTTP 429 + `Retry-After`)
|
|
14
|
+
|
|
15
|
+
The REST `VectorizerClient` honors server-side bulk-upsert
|
|
16
|
+
backpressure shipped in Vectorizer 3.2.0
|
|
17
|
+
([#263](https://github.com/hivellm/vectorizer/issues/263)). On HTTP
|
|
18
|
+
`429 Too Many Requests` the client parses `Retry-After` (seconds
|
|
19
|
+
form, 1 s default, 30 s cap), sleeps, and retries up to 3 times
|
|
20
|
+
before raising a typed `RateLimitError`. Pre-3.2.0 clients bounced
|
|
21
|
+
429s into a generic 5xx and lost the retry budget. Identical
|
|
22
|
+
semantics ship in every first-party SDK (Rust, Python, TypeScript,
|
|
23
|
+
Go, C#) — see `tests/test_retry_after_parse.py`.
|
|
24
|
+
|
|
25
|
+
## v3.1 — `/insert_vectors` + stable client-id upserts
|
|
26
|
+
|
|
27
|
+
- `insert_vectors(collection, vectors, public_key=None)` — bulk-
|
|
28
|
+
insert pre-computed embeddings with caller-supplied vector ids.
|
|
29
|
+
Skips the embedding pipeline entirely.
|
|
30
|
+
- `insert` / `insert_texts`: the request `id` is now used verbatim
|
|
31
|
+
as the stored `Vector.id` (non-chunked) or as `<id>#<chunk_index>`
|
|
32
|
+
(chunked). Re-running the same payload upserts in place instead
|
|
33
|
+
of duplicating.
|
|
34
|
+
- Chunked vectors expose a flat payload layout (`{content,
|
|
35
|
+
file_path, chunk_index, parent_id, ...user_metadata}`). Legacy
|
|
36
|
+
nested payloads from ≤ 3.0.x stay readable during the deprecation
|
|
37
|
+
window.
|
|
38
|
+
|
|
39
|
+
Client-id contract: non-empty, length ≤ 256, no leading/trailing
|
|
40
|
+
whitespace, must not contain `#`.
|
|
41
|
+
|
|
13
42
|
## v3.0 — VectorizerRPC is the default transport
|
|
14
43
|
|
|
15
44
|
Starting with v3.0, the recommended transport is **VectorizerRPC**: a
|
|
@@ -28,6 +57,8 @@ import vectorizer_sdk
|
|
|
28
57
|
|
|
29
58
|
async def main():
|
|
30
59
|
client = await vectorizer_sdk.connect_async("vectorizer://127.0.0.1:15503")
|
|
60
|
+
# `hello` and `search_basic` are RPC-only (not available on the legacy
|
|
61
|
+
# REST `VectorizerClient`).
|
|
31
62
|
await client.hello(vectorizer_sdk.HelloPayload(client_name="my-app"))
|
|
32
63
|
print(await client.list_collections())
|
|
33
64
|
hits = await client.search_basic("docs", "vector database", limit=5)
|
|
@@ -89,7 +120,7 @@ for a runnable end-to-end example.
|
|
|
89
120
|
pip install vectorizer-sdk
|
|
90
121
|
|
|
91
122
|
# Or specific version
|
|
92
|
-
pip install vectorizer-sdk==3.
|
|
123
|
+
pip install vectorizer-sdk==3.2.0
|
|
93
124
|
```
|
|
94
125
|
|
|
95
126
|
## Package Layout (v3.x)
|
|
@@ -426,6 +457,11 @@ related = await client.get_related_files(
|
|
|
426
457
|
|
|
427
458
|
### Summarization Operations
|
|
428
459
|
|
|
460
|
+
> WARNING: The `/summarize/*` REST endpoints are documented but not yet wired
|
|
461
|
+
> server-side (see `DOC_GAP_ANALYSIS`). The SDK methods below
|
|
462
|
+
> (`summarize_text`, `summarize_context`) will fail until server wiring is
|
|
463
|
+
> complete.
|
|
464
|
+
|
|
429
465
|
#### Summarize Text
|
|
430
466
|
Summarize text using various methods:
|
|
431
467
|
|
|
@@ -458,6 +494,11 @@ summary = await client.summarize_context(
|
|
|
458
494
|
|
|
459
495
|
### Workspace Management
|
|
460
496
|
|
|
497
|
+
> WARNING: `add_workspace`, `list_workspaces`, and `remove_workspace` are
|
|
498
|
+
> exposed via the REST transport through dynamic `__getattr__` delegation,
|
|
499
|
+
> but they are not first-class SDK methods yet. They work at runtime but
|
|
500
|
+
> won't autocomplete in IDEs. A future release will add explicit methods.
|
|
501
|
+
|
|
461
502
|
#### Add Workspace
|
|
462
503
|
Add a new workspace:
|
|
463
504
|
|
|
@@ -486,6 +527,11 @@ await client.remove_workspace(
|
|
|
486
527
|
|
|
487
528
|
### Backup Operations
|
|
488
529
|
|
|
530
|
+
> WARNING: `create_backup`, `list_backups`, and `restore_backup` are
|
|
531
|
+
> exposed via the REST transport through dynamic `__getattr__` delegation,
|
|
532
|
+
> but they are not first-class SDK methods yet. They work at runtime but
|
|
533
|
+
> won't autocomplete in IDEs. A future release will add explicit methods.
|
|
534
|
+
|
|
489
535
|
#### Create Backup
|
|
490
536
|
Create a backup of collections:
|
|
491
537
|
|
|
@@ -613,8 +659,11 @@ await client.create_collection("documents", dimension=768)
|
|
|
613
659
|
await client.insert_texts("documents", [
|
|
614
660
|
{"id": "doc1", "text": "Sample document", "metadata": {"source": "api"}}
|
|
615
661
|
])
|
|
616
|
-
|
|
617
|
-
await client.
|
|
662
|
+
# Update-via-reinsert: re-call `insert_texts` with the same id to replace the record.
|
|
663
|
+
await client.insert_texts("documents", [
|
|
664
|
+
{"id": "doc1", "text": "Sample document (updated)", "metadata": {"updated": True}}
|
|
665
|
+
])
|
|
666
|
+
await client.delete_vectors("documents", ["doc1"])
|
|
618
667
|
|
|
619
668
|
# Reads automatically go to replicas (load balanced)
|
|
620
669
|
results = await client.search_vectors("documents", query="sample", limit=10)
|
|
@@ -651,7 +700,7 @@ The SDK automatically classifies operations:
|
|
|
651
700
|
|
|
652
701
|
| Operation Type | Routed To | Methods |
|
|
653
702
|
|---------------|-----------|---------|
|
|
654
|
-
| **Writes** | Always Master | `insert_texts`, `insert_vectors`, `
|
|
703
|
+
| **Writes** | Always Master | `insert_texts`, `insert_vectors`, `delete_vectors`, `create_collection`, `delete_collection` |
|
|
655
704
|
| **Reads** | Based on `read_preference` | `search_vectors`, `get_vector`, `list_collections`, `intelligent_search`, `semantic_search`, `hybrid_search` |
|
|
656
705
|
|
|
657
706
|
#### Standalone Mode (Single Node)
|
|
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|
|
4
4
|
|
|
5
5
|
[project]
|
|
6
6
|
name = "vectorizer_sdk"
|
|
7
|
-
version = "3.0
|
|
7
|
+
version = "3.2.0"
|
|
8
8
|
description = "Python SDK for Vectorizer - Semantic search, vector operations, and the VectorizerRPC binary transport (default in v3.x)"
|
|
9
9
|
readme = "README.md"
|
|
10
10
|
requires-python = ">=3.8"
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
"""Retry-After header parser tests for the Python SDK (issue #263, phase9 §7).
|
|
2
|
+
|
|
3
|
+
The full retry loop is exercised end-to-end at the server level by
|
|
4
|
+
``crates/vectorizer-server/tests/backpressure_429.rs``; here we only
|
|
5
|
+
lock in the value-parsing edges that determine how aggressively the
|
|
6
|
+
SDK backs off.
|
|
7
|
+
|
|
8
|
+
These constants are kept in sync with ``utils/http_client.py``:
|
|
9
|
+
|
|
10
|
+
- missing/unparseable header -> 1 s default
|
|
11
|
+
- ``Retry-After: 0`` -> 1 s default (never busy-loop)
|
|
12
|
+
- capped at 30 s so a misconfigured server cannot pin the client
|
|
13
|
+
"""
|
|
14
|
+
|
|
15
|
+
import os
|
|
16
|
+
import sys
|
|
17
|
+
import unittest
|
|
18
|
+
|
|
19
|
+
sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
|
|
20
|
+
|
|
21
|
+
from utils.http_client import _parse_retry_after # type: ignore[import-not-found]
|
|
22
|
+
|
|
23
|
+
|
|
24
|
+
class TestRetryAfterParse(unittest.TestCase):
|
|
25
|
+
def test_missing_header_returns_default(self) -> None:
|
|
26
|
+
self.assertEqual(_parse_retry_after(None), 1)
|
|
27
|
+
|
|
28
|
+
def test_empty_or_whitespace_returns_default(self) -> None:
|
|
29
|
+
self.assertEqual(_parse_retry_after(""), 1)
|
|
30
|
+
self.assertEqual(_parse_retry_after(" "), 1)
|
|
31
|
+
|
|
32
|
+
def test_zero_returns_default_to_avoid_busy_loop(self) -> None:
|
|
33
|
+
self.assertEqual(_parse_retry_after("0"), 1)
|
|
34
|
+
|
|
35
|
+
def test_unparseable_string_returns_default(self) -> None:
|
|
36
|
+
self.assertEqual(_parse_retry_after("not-a-number"), 1)
|
|
37
|
+
|
|
38
|
+
def test_small_values_pass_through_verbatim(self) -> None:
|
|
39
|
+
self.assertEqual(_parse_retry_after("3"), 3)
|
|
40
|
+
self.assertEqual(_parse_retry_after("7"), 7)
|
|
41
|
+
self.assertEqual(_parse_retry_after(" 5 "), 5)
|
|
42
|
+
|
|
43
|
+
def test_large_values_are_capped_at_30s(self) -> None:
|
|
44
|
+
# If this assertion ever flips, audit _RETRY_AFTER_MAX_SECONDS
|
|
45
|
+
# in utils/http_client.py first.
|
|
46
|
+
self.assertEqual(_parse_retry_after("3600"), 30)
|
|
47
|
+
self.assertEqual(_parse_retry_after("31"), 30)
|
|
48
|
+
|
|
49
|
+
|
|
50
|
+
if __name__ == "__main__":
|
|
51
|
+
unittest.main()
|
|
@@ -12,14 +12,23 @@ try:
|
|
|
12
12
|
NetworkError,
|
|
13
13
|
ServerError,
|
|
14
14
|
AuthenticationError,
|
|
15
|
+
RateLimitError,
|
|
15
16
|
)
|
|
16
17
|
except ImportError:
|
|
17
18
|
from exceptions import (
|
|
18
19
|
NetworkError,
|
|
19
20
|
ServerError,
|
|
20
21
|
AuthenticationError,
|
|
22
|
+
RateLimitError,
|
|
21
23
|
)
|
|
22
24
|
|
|
25
|
+
# Issue #263: cap Retry-After respect at this many seconds so a
|
|
26
|
+
# misconfigured server can't pin a client into a half-hour sleep.
|
|
27
|
+
_RETRY_AFTER_MAX_SECONDS = 30
|
|
28
|
+
# Floor for parsed Retry-After so a `0` or missing header still yields
|
|
29
|
+
# a noticeable backoff rather than busy-looping the server.
|
|
30
|
+
_RETRY_AFTER_DEFAULT_SECONDS = 1
|
|
31
|
+
|
|
23
32
|
logger = logging.getLogger(__name__)
|
|
24
33
|
|
|
25
34
|
|
|
@@ -83,42 +92,70 @@ class HTTPClient:
|
|
|
83
92
|
) -> Any:
|
|
84
93
|
"""
|
|
85
94
|
Make an HTTP request.
|
|
86
|
-
|
|
95
|
+
|
|
96
|
+
Honors `Retry-After` on 429 responses (issue #263): the client
|
|
97
|
+
sleeps for the header's value (capped) and retries up to
|
|
98
|
+
``max_retries`` times. After exhaustion a ``RateLimitError`` is
|
|
99
|
+
raised so callers can surface the back-pressure to the user.
|
|
100
|
+
|
|
87
101
|
Args:
|
|
88
102
|
method: HTTP method
|
|
89
103
|
path: API endpoint path
|
|
90
104
|
data: Request data
|
|
91
|
-
|
|
105
|
+
|
|
92
106
|
Returns:
|
|
93
107
|
Response data
|
|
94
108
|
"""
|
|
95
109
|
await self._ensure_session()
|
|
96
|
-
|
|
110
|
+
|
|
97
111
|
url = f"{self.base_url}{path}"
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
112
|
+
attempts_remaining = self.max_retries
|
|
113
|
+
last_429_text: Optional[str] = None
|
|
114
|
+
|
|
115
|
+
while True:
|
|
116
|
+
try:
|
|
117
|
+
async with self._session.request(
|
|
118
|
+
method,
|
|
119
|
+
url,
|
|
120
|
+
json=data if data else None,
|
|
121
|
+
) as response:
|
|
122
|
+
if response.status == 429:
|
|
123
|
+
last_429_text = await response.text()
|
|
124
|
+
if attempts_remaining <= 0:
|
|
125
|
+
raise RateLimitError(
|
|
126
|
+
f"HTTP 429 after {self.max_retries} retries: "
|
|
127
|
+
f"{last_429_text}"
|
|
128
|
+
)
|
|
129
|
+
delay = _parse_retry_after(
|
|
130
|
+
response.headers.get("Retry-After")
|
|
131
|
+
)
|
|
132
|
+
logger.info(
|
|
133
|
+
"Vectorizer 429 — sleeping %.1fs before retry "
|
|
134
|
+
"(remaining attempts=%d)",
|
|
135
|
+
delay,
|
|
136
|
+
attempts_remaining,
|
|
137
|
+
)
|
|
138
|
+
attempts_remaining -= 1
|
|
139
|
+
await asyncio.sleep(delay)
|
|
140
|
+
continue
|
|
141
|
+
|
|
142
|
+
if response.status >= 400:
|
|
143
|
+
error_text = await response.text()
|
|
144
|
+
raise self._handle_error(response.status, error_text)
|
|
145
|
+
|
|
146
|
+
content_type = response.headers.get('Content-Type', '')
|
|
147
|
+
if 'application/json' in content_type:
|
|
148
|
+
return await response.json()
|
|
149
|
+
return await response.text()
|
|
150
|
+
|
|
151
|
+
except (ServerError, AuthenticationError, RateLimitError):
|
|
152
|
+
raise
|
|
153
|
+
except aiohttp.ClientError as e:
|
|
154
|
+
raise NetworkError(f"HTTP request failed: {e}")
|
|
155
|
+
except asyncio.TimeoutError:
|
|
156
|
+
raise NetworkError("Request timeout")
|
|
157
|
+
except Exception as e:
|
|
158
|
+
raise NetworkError(f"Unknown error: {e}")
|
|
122
159
|
|
|
123
160
|
async def get(self, path: str) -> Any:
|
|
124
161
|
"""Make a GET request."""
|
|
@@ -139,15 +176,37 @@ class HTTPClient:
|
|
|
139
176
|
def _handle_error(self, status: int, error_text: str) -> Exception:
|
|
140
177
|
"""Handle HTTP errors and convert to appropriate exceptions."""
|
|
141
178
|
message = f"HTTP {status}: {error_text}"
|
|
142
|
-
|
|
179
|
+
|
|
143
180
|
if status == 401:
|
|
144
181
|
return AuthenticationError(message)
|
|
145
182
|
elif status == 403:
|
|
146
183
|
return AuthenticationError("Access forbidden")
|
|
147
184
|
elif status == 404:
|
|
148
185
|
return ServerError("Resource not found")
|
|
149
|
-
elif status
|
|
186
|
+
elif status == 429:
|
|
187
|
+
# 429 is handled in `request()` via Retry-After; reaching
|
|
188
|
+
# here means the caller bypassed retry handling, so
|
|
189
|
+
# surface a typed RateLimitError instead of a generic 5xx.
|
|
190
|
+
return RateLimitError(message)
|
|
191
|
+
elif status in (500, 502, 503, 504):
|
|
150
192
|
return ServerError(message)
|
|
151
193
|
else:
|
|
152
194
|
return ServerError(message)
|
|
153
195
|
|
|
196
|
+
|
|
197
|
+
def _parse_retry_after(value: Optional[str]) -> float:
|
|
198
|
+
"""Parse a ``Retry-After`` header value (seconds form only).
|
|
199
|
+
|
|
200
|
+
Returns a sane default + caps an unreasonably large server hint
|
|
201
|
+
so a misconfigured server can't pin a client into a long sleep.
|
|
202
|
+
"""
|
|
203
|
+
if not value:
|
|
204
|
+
return _RETRY_AFTER_DEFAULT_SECONDS
|
|
205
|
+
try:
|
|
206
|
+
seconds = float(value.strip())
|
|
207
|
+
except ValueError:
|
|
208
|
+
return _RETRY_AFTER_DEFAULT_SECONDS
|
|
209
|
+
if seconds <= 0:
|
|
210
|
+
return _RETRY_AFTER_DEFAULT_SECONDS
|
|
211
|
+
return min(seconds, _RETRY_AFTER_MAX_SECONDS)
|
|
212
|
+
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: vectorizer_sdk
|
|
3
|
-
Version: 3.0
|
|
3
|
+
Version: 3.2.0
|
|
4
4
|
Summary: Python SDK for Vectorizer - Semantic search, vector operations, and the VectorizerRPC binary transport (default in v3.x)
|
|
5
5
|
Author-email: HiveLLM Team <team@hivellm.org>
|
|
6
6
|
License: Apache-2.0
|
|
@@ -58,9 +58,38 @@ Dynamic: license-file
|
|
|
58
58
|
A comprehensive Python SDK for the Vectorizer semantic search service.
|
|
59
59
|
|
|
60
60
|
**Package**: `vectorizer_sdk` (PEP 625 compliant)
|
|
61
|
-
**Version**: 3.
|
|
61
|
+
**Version**: 3.2.0
|
|
62
62
|
**PyPI**: https://pypi.org/project/vectorizer-sdk/
|
|
63
63
|
|
|
64
|
+
## v3.2 — backpressure-aware client (HTTP 429 + `Retry-After`)
|
|
65
|
+
|
|
66
|
+
The REST `VectorizerClient` honors server-side bulk-upsert
|
|
67
|
+
backpressure shipped in Vectorizer 3.2.0
|
|
68
|
+
([#263](https://github.com/hivellm/vectorizer/issues/263)). On HTTP
|
|
69
|
+
`429 Too Many Requests` the client parses `Retry-After` (seconds
|
|
70
|
+
form, 1 s default, 30 s cap), sleeps, and retries up to 3 times
|
|
71
|
+
before raising a typed `RateLimitError`. Pre-3.2.0 clients bounced
|
|
72
|
+
429s into a generic 5xx and lost the retry budget. Identical
|
|
73
|
+
semantics ship in every first-party SDK (Rust, Python, TypeScript,
|
|
74
|
+
Go, C#) — see `tests/test_retry_after_parse.py`.
|
|
75
|
+
|
|
76
|
+
## v3.1 — `/insert_vectors` + stable client-id upserts
|
|
77
|
+
|
|
78
|
+
- `insert_vectors(collection, vectors, public_key=None)` — bulk-
|
|
79
|
+
insert pre-computed embeddings with caller-supplied vector ids.
|
|
80
|
+
Skips the embedding pipeline entirely.
|
|
81
|
+
- `insert` / `insert_texts`: the request `id` is now used verbatim
|
|
82
|
+
as the stored `Vector.id` (non-chunked) or as `<id>#<chunk_index>`
|
|
83
|
+
(chunked). Re-running the same payload upserts in place instead
|
|
84
|
+
of duplicating.
|
|
85
|
+
- Chunked vectors expose a flat payload layout (`{content,
|
|
86
|
+
file_path, chunk_index, parent_id, ...user_metadata}`). Legacy
|
|
87
|
+
nested payloads from ≤ 3.0.x stay readable during the deprecation
|
|
88
|
+
window.
|
|
89
|
+
|
|
90
|
+
Client-id contract: non-empty, length ≤ 256, no leading/trailing
|
|
91
|
+
whitespace, must not contain `#`.
|
|
92
|
+
|
|
64
93
|
## v3.0 — VectorizerRPC is the default transport
|
|
65
94
|
|
|
66
95
|
Starting with v3.0, the recommended transport is **VectorizerRPC**: a
|
|
@@ -79,6 +108,8 @@ import vectorizer_sdk
|
|
|
79
108
|
|
|
80
109
|
async def main():
|
|
81
110
|
client = await vectorizer_sdk.connect_async("vectorizer://127.0.0.1:15503")
|
|
111
|
+
# `hello` and `search_basic` are RPC-only (not available on the legacy
|
|
112
|
+
# REST `VectorizerClient`).
|
|
82
113
|
await client.hello(vectorizer_sdk.HelloPayload(client_name="my-app"))
|
|
83
114
|
print(await client.list_collections())
|
|
84
115
|
hits = await client.search_basic("docs", "vector database", limit=5)
|
|
@@ -140,7 +171,7 @@ for a runnable end-to-end example.
|
|
|
140
171
|
pip install vectorizer-sdk
|
|
141
172
|
|
|
142
173
|
# Or specific version
|
|
143
|
-
pip install vectorizer-sdk==3.
|
|
174
|
+
pip install vectorizer-sdk==3.2.0
|
|
144
175
|
```
|
|
145
176
|
|
|
146
177
|
## Package Layout (v3.x)
|
|
@@ -477,6 +508,11 @@ related = await client.get_related_files(
|
|
|
477
508
|
|
|
478
509
|
### Summarization Operations
|
|
479
510
|
|
|
511
|
+
> WARNING: The `/summarize/*` REST endpoints are documented but not yet wired
|
|
512
|
+
> server-side (see `DOC_GAP_ANALYSIS`). The SDK methods below
|
|
513
|
+
> (`summarize_text`, `summarize_context`) will fail until server wiring is
|
|
514
|
+
> complete.
|
|
515
|
+
|
|
480
516
|
#### Summarize Text
|
|
481
517
|
Summarize text using various methods:
|
|
482
518
|
|
|
@@ -509,6 +545,11 @@ summary = await client.summarize_context(
|
|
|
509
545
|
|
|
510
546
|
### Workspace Management
|
|
511
547
|
|
|
548
|
+
> WARNING: `add_workspace`, `list_workspaces`, and `remove_workspace` are
|
|
549
|
+
> exposed via the REST transport through dynamic `__getattr__` delegation,
|
|
550
|
+
> but they are not first-class SDK methods yet. They work at runtime but
|
|
551
|
+
> won't autocomplete in IDEs. A future release will add explicit methods.
|
|
552
|
+
|
|
512
553
|
#### Add Workspace
|
|
513
554
|
Add a new workspace:
|
|
514
555
|
|
|
@@ -537,6 +578,11 @@ await client.remove_workspace(
|
|
|
537
578
|
|
|
538
579
|
### Backup Operations
|
|
539
580
|
|
|
581
|
+
> WARNING: `create_backup`, `list_backups`, and `restore_backup` are
|
|
582
|
+
> exposed via the REST transport through dynamic `__getattr__` delegation,
|
|
583
|
+
> but they are not first-class SDK methods yet. They work at runtime but
|
|
584
|
+
> won't autocomplete in IDEs. A future release will add explicit methods.
|
|
585
|
+
|
|
540
586
|
#### Create Backup
|
|
541
587
|
Create a backup of collections:
|
|
542
588
|
|
|
@@ -664,8 +710,11 @@ await client.create_collection("documents", dimension=768)
|
|
|
664
710
|
await client.insert_texts("documents", [
|
|
665
711
|
{"id": "doc1", "text": "Sample document", "metadata": {"source": "api"}}
|
|
666
712
|
])
|
|
667
|
-
|
|
668
|
-
await client.
|
|
713
|
+
# Update-via-reinsert: re-call `insert_texts` with the same id to replace the record.
|
|
714
|
+
await client.insert_texts("documents", [
|
|
715
|
+
{"id": "doc1", "text": "Sample document (updated)", "metadata": {"updated": True}}
|
|
716
|
+
])
|
|
717
|
+
await client.delete_vectors("documents", ["doc1"])
|
|
669
718
|
|
|
670
719
|
# Reads automatically go to replicas (load balanced)
|
|
671
720
|
results = await client.search_vectors("documents", query="sample", limit=10)
|
|
@@ -702,7 +751,7 @@ The SDK automatically classifies operations:
|
|
|
702
751
|
|
|
703
752
|
| Operation Type | Routed To | Methods |
|
|
704
753
|
|---------------|-----------|---------|
|
|
705
|
-
| **Writes** | Always Master | `insert_texts`, `insert_vectors`, `
|
|
754
|
+
| **Writes** | Always Master | `insert_texts`, `insert_vectors`, `delete_vectors`, `create_collection`, `delete_collection` |
|
|
706
755
|
| **Reads** | Based on `read_preference` | `search_vectors`, `get_vector`, `list_collections`, `intelligent_search`, `semantic_search`, `hybrid_search` |
|
|
707
756
|
|
|
708
757
|
#### Standalone Mode (Single Node)
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|