wavemind 2.2.2__tar.gz → 2.2.3__tar.gz

{wavemind-2.2.2 → wavemind-2.2.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: wavemind
-Version: 2.2.2
+Version: 2.2.3
 Summary: Local-first dynamic memory field with vector search and wave-field re-ranking
 License-Expression: MIT
 Project-URL: Homepage, https://github.com/CaspianG/wavemind
@@ -23,6 +23,8 @@ Requires-Dist: faiss-cpu>=1.8; platform_system != "Windows" and extra == "indexe
 Requires-Dist: qdrant-client>=1.9; extra == "indexes"
 Provides-Extra: postgres
 Requires-Dist: psycopg[binary]>=3.1; extra == "postgres"
+Provides-Extra: redis
+Requires-Dist: redis>=5.0; extra == "redis"
 Provides-Extra: otel
 Requires-Dist: opentelemetry-api>=1.25; extra == "otel"
 Requires-Dist: opentelemetry-sdk>=1.25; extra == "otel"
@@ -38,6 +40,7 @@ Requires-Dist: annoy>=1.17; extra == "production"
 Requires-Dist: faiss-cpu>=1.8; platform_system != "Windows" and extra == "production"
 Requires-Dist: qdrant-client>=1.9; extra == "production"
 Requires-Dist: psycopg[binary]>=3.1; extra == "production"
+Requires-Dist: redis>=5.0; extra == "production"
 Requires-Dist: opentelemetry-api>=1.25; extra == "production"
 Requires-Dist: opentelemetry-sdk>=1.25; extra == "production"
 Requires-Dist: opentelemetry-exporter-otlp>=1.25; extra == "production"
@@ -538,7 +541,7 @@ Checked-in result:
 
 | profile | result |
 |---|---:|
-| Cluster planner | 4096 namespaces, 4 nodes, replication factor 2, single-node loss availability `1.000`. |
+| Cluster planner | 4096 namespaces, 4 nodes, replication factor 2, node-loss availability `1.000`, zone-loss availability `1.000`, write quorum `2`. |
 | Hot cache | 2000 lookups, hit rate `0.920`, p99 lookup `0.01 ms`. |
 | Structured payloads | image/audio/table/event retrieval, precision@1 `1.000`, p99 `1.27 ms`. |
 
@@ -567,6 +570,24 @@ SQLite writes.
 
 The same planner is available over HTTP as `POST /cluster-plan`.
 
+Maintenance worker:
+
+```sh
+wavemind maintenance --namespace user:42 --consolidate-steps 10 --consolidate-concepts --json
+```
+
+This runs one deterministic maintenance pass: expired-memory purge, optional
+field/concept consolidation, and index-health repair. Production deployments can
+call the same command from cron, systemd, Kubernetes CronJobs, Celery, RQ, or
+Temporal.
+
+Hot-cache options:
+
+| cache | use case |
+|---|---|
+| `HotMemoryCache` | in-process local API/server cache. |
+| `RedisHotMemoryCache` | shared cache for multiple API workers. Install with `pip install "wavemind[redis]"`. |
+
 ## Structured And Multimodal Memory
 
 WaveMind can store non-text memories as structured text plus metadata. This is
@@ -1017,8 +1038,8 @@ Current read:
 | LongMemEval full retrieval | On the official LongMemEval-S cleaned file, 470 non-abstention session-level questions, WaveMind reaches `evidence_recall@5 0.782` and `precision@1 0.696`; Chroma static reaches `0.518` / `0.355`; Qdrant static reaches `0.520` / `0.355`. | This is now the strongest public memory result in the repo. It is retrieval-only, not final answer quality. |
 | LongMemEval 50-query smoke | On the first 50 non-abstention LongMemEval-S questions, WaveMind reaches `evidence_recall@5 0.920`, `precision@1 0.760`, and `MRR@5 0.827`; Chroma/Qdrant static reach `0.600`, `0.260`, and `0.385`. | This is the fast regression profile for checking current changes before rerunning the full LongMemEval profile. WaveMind wins on quality; latency still needs work. |
 | ANN/index curve | At 50000 generated 128-d vectors, NumPy exact keeps `recall@10 1.000` at `6.49 ms`; quantized int8 keeps `0.934` at `24.92 ms`; Annoy is faster at `4.92 ms` but drops to `0.730` recall; Qdrant local keeps `1.000` recall at `43.49 ms`. | Current local scale boundary is clear: quantized search needs kernel work, Annoy needs tuning/FAISS, and Qdrant should be tested in service mode for a fair production comparison. |
-| Production load | At 100000 generated 128-d vectors, service-mode Qdrant reaches `recall@10 1.000`, avg `10.76 ms`; pgvector HNSW reaches `0.736`, avg `17.76 ms`; at 1M vectors Qdrant reaches `0.506`, avg `45.81 ms`. | Qdrant service is already usable at 100k. The 1M result is not production-grade yet: large-N service settings need tuning before claiming million-memory recall. |
-| Scale readiness | Deterministic 1M-memory simulation validates 4096 namespace placements over 4 nodes with replication factor 2, single-node-loss availability `1.000`, hot-cache hit rate `0.920`, and structured payload precision@1 `1.000`. | This proves routing/cache/payload foundations, not a 10M-vector load-test claim. Real 100k-10M production latency needs service-backed load tests. |
+| Production load | At 100000 generated 128-d vectors, service-mode Qdrant reaches `recall@10 1.000`, avg `10.28 ms`, p99 `21.26 ms`. At 1M, tuned Qdrant reaches `recall@10 0.984`, avg `116.80 ms`, p99 `209.28 ms`; an EF sweep finds `recall@10 0.977`, avg `64.76 ms`, p99 `103.77 ms` at `hnsw_ef=2048` on 30 queries. | 100k is production-grade on the tested machine. 1M recall is now strong, but p99 still needs tuning before claiming a stable sub-100 ms SLO. |
+| Scale readiness | Deterministic 1M-memory simulation validates 4096 namespace placements over 4 nodes with replication factor 2, node-loss availability `1.000`, zone-loss availability `1.000`, hot-cache hit rate `0.920`, and structured payload precision@1 `1.000`. | This proves routing/cache/payload foundations, not a 10M-vector load-test claim or Raft replication. Real 10M latency still needs service-backed load tests on larger hardware. |
 | Memory competitor adapters | WaveMind reaches `precision@1 0.80`, `precision@3 1.00`, stale suppression `1.00` on the small adapter profile. Mem0, Zep, and LangGraph are listed as skipped unless their real packages/services are configured. | This prevents fake competitor claims. The adapter harness is ready; real Mem0/Zep/LangGraph results still need configured installs. |
 | LongMemEval local answer generation | With the same local Ollama `qwen2.5:1.5b`, WaveMind reaches `exact_match 0.240`, `contains_answer 0.380`, `token_f1 0.333`, and `evidence_recall@5 0.920`; Chroma and Qdrant static both reach `0.120`, `0.160`, `0.170`, and `0.600`. | This is the first checked-in end-to-end answer benchmark against Chroma/Qdrant. It is still a 50-question lightweight smoke run, not a full LongMemEval leaderboard score. |
 
@@ -1035,8 +1056,9 @@ Current read:
 | NoMIRACL Russian retrieval | Russian human-annotated multilingual relevance over compact candidate passages. | implemented | WaveMind / Chroma / Qdrant | Keep same-embedding `nDCG@10` at parity, then rerun with sentence-transformers and full MIRACL Russian when disk/service capacity allows it. |
 | ANN/VectorDBBench-style local curve | Recall/latency tradeoff for candidate indexes on generated vectors. | implemented | NumPy exact / quantized int8 / Annoy / Qdrant local | Use this as the local engineering curve; official VectorDBBench remains future work. |
 | Production index profile | Docker-backed 50000-vector profile for persisted FAISS, Qdrant service, and PostgreSQL/pgvector HNSW. | implemented | FAISS / Qdrant service / pgvector | Keep service-mode candidate generation above `0.95` recall@10 and below 10 ms average query latency at 50000 vectors. |
-| Production load profile | 100k and 1M service-backed candidate-index checks. | implemented | Qdrant service / pgvector HNSW / FAISS persisted | 100k Qdrant is strong; 1M Qdrant and pgvector require tuning before production claims. |
-| Scale readiness profile | Cluster placement, single-node-loss simulation, hot-cache behavior, and structured/multimodal payload retrieval. | implemented | Mem0 / Zep / LangGraph persistent memory / GraphRAG target adapters | Use this as production foundation proof before real distributed 100k, 1M, and 10M load tests. |
+| Production load profile | 100k and 1M service-backed candidate-index checks with p95/p99 latency. | implemented | Qdrant service / pgvector HNSW / FAISS persisted | Keep 100k at recall@10 `1.000`; push 1M p99 below 100 ms with recall@10 >= 0.95. |
+| Qdrant 1M HNSW ef sweep | One 1M Qdrant collection queried with multiple `hnsw_ef` values. | implemented | Qdrant service | Repeat with 100+ queries and collection-level HNSW build parameters before claiming a stable 1M SLO. |
+| Scale readiness profile | Cluster placement, node/zone-loss simulation, quorum report, hot-cache behavior, and structured/multimodal payload retrieval. | implemented | Mem0 / Zep / LangGraph persistent memory / GraphRAG target adapters | Use this as production foundation proof before real distributed 10M load tests. |
 | Memory competitor adapter profile | Dynamic-memory scenario wired for external memory frameworks. | implemented | Mem0 / Zep / LangGraph persistent memory | Report real competitor results only when their packages/services are explicitly configured. |
 | [BEIR](https://github.com/beir-cellar/beir) | Standard zero-shot information retrieval quality. | planned | Chroma / Qdrant / FAISS | Stay within 0.02 `nDCG@10` on identical embeddings. |
 | [MTEB Retrieval](https://github.com/embeddings-benchmark/mteb) | Separates encoder quality from retrieval-store quality. | planned | Chroma / Qdrant / FAISS | Prove WaveMind does not reduce same-embedding retrieval quality. |
@@ -1307,15 +1329,16 @@ Checked-in production load points:
 
 ```sh
 python benchmarks/production_load_benchmark.py --sizes 100000 --dim 128 --queries 100 --top-k 10 --engines qdrant-service pgvector faiss-persisted
-python benchmarks/production_load_benchmark.py --sizes 1000000 --dim 128 --queries 50 --top-k 10 --engines qdrant-service --output benchmarks/production_load_qdrant_1m_results.json
+python benchmarks/production_load_benchmark.py --sizes 1000000 --dim 128 --queries 50 --top-k 10 --engines qdrant-service --output benchmarks/production_load_qdrant_1m_tuned_results.json
 ```
 
-| vectors | engine | recall@10 | avg latency | p95 latency | build |
-|---:|---|---:|---:|---:|---:|
-| 100000 | Qdrant service | 1.000 | 10.76 ms | 18.78 ms | 39873.2 ms |
-| 100000 | WaveMind pgvector | 0.736 | 17.76 ms | 23.48 ms | 455703.7 ms |
-| 100000 | WaveMind faiss-persisted | skipped | - | - | - |
-| 1000000 | Qdrant service | 0.506 | 45.81 ms | 65.18 ms | 563945.5 ms |
+| vectors | engine | recall@10 | avg latency | p95 latency | p99 latency | build |
+|---:|---|---:|---:|---:|---:|---:|
+| 100000 | Qdrant service | 1.000 | 10.28 ms | 18.97 ms | 21.26 ms | 27439.3 ms |
+| 100000 | WaveMind pgvector | 0.736 | 17.76 ms | 23.48 ms | - | 455703.7 ms |
+| 100000 | WaveMind faiss-persisted | skipped | - | - | - | - |
+| 1000000 | Qdrant service tuned | 0.984 | 116.80 ms | 153.84 ms | 209.28 ms | 450674.6 ms |
+| 1000000 | Qdrant `hnsw_ef=2048` sweep point | 0.977 | 64.76 ms | 91.18 ms | 103.77 ms | 451912.4 ms |
 
 Read this as an engineering curve, not an official VectorDBBench result. Annoy
 is faster than exact NumPy at 50000 vectors but loses too much recall with the
@@ -1327,9 +1350,9 @@ FAISS persistence and service-mode Qdrant now both preserve exact recall at
 `WAVEMIND_PGVECTOR_EF_SEARCH=400`, which improves recall materially but still
 misses the `0.95` production target and is slower than the other two profiles.
 The 100k load profile shows Qdrant service is already viable for candidate
-generation; the 1M Qdrant profile shows that default service settings are not
-enough for production recall and need HNSW/search tuning before million-memory
-claims.
+generation on the tested machine. The tuned 1M profile crosses the recall target,
+and the EF sweep gets close to the p99 latency target, but 1M should still be
+treated as tuning-in-progress until a 100+ query p99 run stays below 100 ms.
 If a required package, service, or environment variable is missing, the runner
 marks that engine as `skipped` instead of silently falling back to another
 backend.

{wavemind-2.2.2 → wavemind-2.2.3}/README.md

@@ -488,7 +488,7 @@ Checked-in result:
 
 | profile | result |
 |---|---:|
-| Cluster planner | 4096 namespaces, 4 nodes, replication factor 2, single-node loss availability `1.000`. |
+| Cluster planner | 4096 namespaces, 4 nodes, replication factor 2, node-loss availability `1.000`, zone-loss availability `1.000`, write quorum `2`. |
 | Hot cache | 2000 lookups, hit rate `0.920`, p99 lookup `0.01 ms`. |
 | Structured payloads | image/audio/table/event retrieval, precision@1 `1.000`, p99 `1.27 ms`. |
 
@@ -517,6 +517,24 @@ SQLite writes.
 
 The same planner is available over HTTP as `POST /cluster-plan`.
 
+Maintenance worker:
+
+```sh
+wavemind maintenance --namespace user:42 --consolidate-steps 10 --consolidate-concepts --json
+```
+
+This runs one deterministic maintenance pass: expired-memory purge, optional
+field/concept consolidation, and index-health repair. Production deployments can
+call the same command from cron, systemd, Kubernetes CronJobs, Celery, RQ, or
+Temporal.
+
+Hot-cache options:
+
+| cache | use case |
+|---|---|
+| `HotMemoryCache` | in-process local API/server cache. |
+| `RedisHotMemoryCache` | shared cache for multiple API workers. Install with `pip install "wavemind[redis]"`. |
+
 ## Structured And Multimodal Memory
 
 WaveMind can store non-text memories as structured text plus metadata. This is
@@ -967,8 +985,8 @@ Current read:
 | LongMemEval full retrieval | On the official LongMemEval-S cleaned file, 470 non-abstention session-level questions, WaveMind reaches `evidence_recall@5 0.782` and `precision@1 0.696`; Chroma static reaches `0.518` / `0.355`; Qdrant static reaches `0.520` / `0.355`. | This is now the strongest public memory result in the repo. It is retrieval-only, not final answer quality. |
 | LongMemEval 50-query smoke | On the first 50 non-abstention LongMemEval-S questions, WaveMind reaches `evidence_recall@5 0.920`, `precision@1 0.760`, and `MRR@5 0.827`; Chroma/Qdrant static reach `0.600`, `0.260`, and `0.385`. | This is the fast regression profile for checking current changes before rerunning the full LongMemEval profile. WaveMind wins on quality; latency still needs work. |
 | ANN/index curve | At 50000 generated 128-d vectors, NumPy exact keeps `recall@10 1.000` at `6.49 ms`; quantized int8 keeps `0.934` at `24.92 ms`; Annoy is faster at `4.92 ms` but drops to `0.730` recall; Qdrant local keeps `1.000` recall at `43.49 ms`. | Current local scale boundary is clear: quantized search needs kernel work, Annoy needs tuning/FAISS, and Qdrant should be tested in service mode for a fair production comparison. |
-| Production load | At 100000 generated 128-d vectors, service-mode Qdrant reaches `recall@10 1.000`, avg `10.76 ms`; pgvector HNSW reaches `0.736`, avg `17.76 ms`; at 1M vectors Qdrant reaches `0.506`, avg `45.81 ms`. | Qdrant service is already usable at 100k. The 1M result is not production-grade yet: large-N service settings need tuning before claiming million-memory recall. |
-| Scale readiness | Deterministic 1M-memory simulation validates 4096 namespace placements over 4 nodes with replication factor 2, single-node-loss availability `1.000`, hot-cache hit rate `0.920`, and structured payload precision@1 `1.000`. | This proves routing/cache/payload foundations, not a 10M-vector load-test claim. Real 100k-10M production latency needs service-backed load tests. |
+| Production load | At 100000 generated 128-d vectors, service-mode Qdrant reaches `recall@10 1.000`, avg `10.28 ms`, p99 `21.26 ms`. At 1M, tuned Qdrant reaches `recall@10 0.984`, avg `116.80 ms`, p99 `209.28 ms`; an EF sweep finds `recall@10 0.977`, avg `64.76 ms`, p99 `103.77 ms` at `hnsw_ef=2048` on 30 queries. | 100k is production-grade on the tested machine. 1M recall is now strong, but p99 still needs tuning before claiming a stable sub-100 ms SLO. |
+| Scale readiness | Deterministic 1M-memory simulation validates 4096 namespace placements over 4 nodes with replication factor 2, node-loss availability `1.000`, zone-loss availability `1.000`, hot-cache hit rate `0.920`, and structured payload precision@1 `1.000`. | This proves routing/cache/payload foundations, not a 10M-vector load-test claim or Raft replication. Real 10M latency still needs service-backed load tests on larger hardware. |
 | Memory competitor adapters | WaveMind reaches `precision@1 0.80`, `precision@3 1.00`, stale suppression `1.00` on the small adapter profile. Mem0, Zep, and LangGraph are listed as skipped unless their real packages/services are configured. | This prevents fake competitor claims. The adapter harness is ready; real Mem0/Zep/LangGraph results still need configured installs. |
 | LongMemEval local answer generation | With the same local Ollama `qwen2.5:1.5b`, WaveMind reaches `exact_match 0.240`, `contains_answer 0.380`, `token_f1 0.333`, and `evidence_recall@5 0.920`; Chroma and Qdrant static both reach `0.120`, `0.160`, `0.170`, and `0.600`. | This is the first checked-in end-to-end answer benchmark against Chroma/Qdrant. It is still a 50-question lightweight smoke run, not a full LongMemEval leaderboard score. |
 
@@ -985,8 +1003,9 @@ Current read:
 | NoMIRACL Russian retrieval | Russian human-annotated multilingual relevance over compact candidate passages. | implemented | WaveMind / Chroma / Qdrant | Keep same-embedding `nDCG@10` at parity, then rerun with sentence-transformers and full MIRACL Russian when disk/service capacity allows it. |
 | ANN/VectorDBBench-style local curve | Recall/latency tradeoff for candidate indexes on generated vectors. | implemented | NumPy exact / quantized int8 / Annoy / Qdrant local | Use this as the local engineering curve; official VectorDBBench remains future work. |
 | Production index profile | Docker-backed 50000-vector profile for persisted FAISS, Qdrant service, and PostgreSQL/pgvector HNSW. | implemented | FAISS / Qdrant service / pgvector | Keep service-mode candidate generation above `0.95` recall@10 and below 10 ms average query latency at 50000 vectors. |
-| Production load profile | 100k and 1M service-backed candidate-index checks. | implemented | Qdrant service / pgvector HNSW / FAISS persisted | 100k Qdrant is strong; 1M Qdrant and pgvector require tuning before production claims. |
-| Scale readiness profile | Cluster placement, single-node-loss simulation, hot-cache behavior, and structured/multimodal payload retrieval. | implemented | Mem0 / Zep / LangGraph persistent memory / GraphRAG target adapters | Use this as production foundation proof before real distributed 100k, 1M, and 10M load tests. |
+| Production load profile | 100k and 1M service-backed candidate-index checks with p95/p99 latency. | implemented | Qdrant service / pgvector HNSW / FAISS persisted | Keep 100k at recall@10 `1.000`; push 1M p99 below 100 ms with recall@10 >= 0.95. |
+| Qdrant 1M HNSW ef sweep | One 1M Qdrant collection queried with multiple `hnsw_ef` values. | implemented | Qdrant service | Repeat with 100+ queries and collection-level HNSW build parameters before claiming a stable 1M SLO. |
+| Scale readiness profile | Cluster placement, node/zone-loss simulation, quorum report, hot-cache behavior, and structured/multimodal payload retrieval. | implemented | Mem0 / Zep / LangGraph persistent memory / GraphRAG target adapters | Use this as production foundation proof before real distributed 10M load tests. |
 | Memory competitor adapter profile | Dynamic-memory scenario wired for external memory frameworks. | implemented | Mem0 / Zep / LangGraph persistent memory | Report real competitor results only when their packages/services are explicitly configured. |
 | [BEIR](https://github.com/beir-cellar/beir) | Standard zero-shot information retrieval quality. | planned | Chroma / Qdrant / FAISS | Stay within 0.02 `nDCG@10` on identical embeddings. |
 | [MTEB Retrieval](https://github.com/embeddings-benchmark/mteb) | Separates encoder quality from retrieval-store quality. | planned | Chroma / Qdrant / FAISS | Prove WaveMind does not reduce same-embedding retrieval quality. |
@@ -1257,15 +1276,16 @@ Checked-in production load points:
 
 ```sh
 python benchmarks/production_load_benchmark.py --sizes 100000 --dim 128 --queries 100 --top-k 10 --engines qdrant-service pgvector faiss-persisted
-python benchmarks/production_load_benchmark.py --sizes 1000000 --dim 128 --queries 50 --top-k 10 --engines qdrant-service --output benchmarks/production_load_qdrant_1m_results.json
+python benchmarks/production_load_benchmark.py --sizes 1000000 --dim 128 --queries 50 --top-k 10 --engines qdrant-service --output benchmarks/production_load_qdrant_1m_tuned_results.json
 ```
 
-| vectors | engine | recall@10 | avg latency | p95 latency | build |
-|---:|---|---:|---:|---:|---:|
-| 100000 | Qdrant service | 1.000 | 10.76 ms | 18.78 ms | 39873.2 ms |
-| 100000 | WaveMind pgvector | 0.736 | 17.76 ms | 23.48 ms | 455703.7 ms |
-| 100000 | WaveMind faiss-persisted | skipped | - | - | - |
-| 1000000 | Qdrant service | 0.506 | 45.81 ms | 65.18 ms | 563945.5 ms |
+| vectors | engine | recall@10 | avg latency | p95 latency | p99 latency | build |
+|---:|---|---:|---:|---:|---:|---:|
+| 100000 | Qdrant service | 1.000 | 10.28 ms | 18.97 ms | 21.26 ms | 27439.3 ms |
+| 100000 | WaveMind pgvector | 0.736 | 17.76 ms | 23.48 ms | - | 455703.7 ms |
+| 100000 | WaveMind faiss-persisted | skipped | - | - | - | - |
+| 1000000 | Qdrant service tuned | 0.984 | 116.80 ms | 153.84 ms | 209.28 ms | 450674.6 ms |
+| 1000000 | Qdrant `hnsw_ef=2048` sweep point | 0.977 | 64.76 ms | 91.18 ms | 103.77 ms | 451912.4 ms |
 
 Read this as an engineering curve, not an official VectorDBBench result. Annoy
 is faster than exact NumPy at 50000 vectors but loses too much recall with the
@@ -1277,9 +1297,9 @@ FAISS persistence and service-mode Qdrant now both preserve exact recall at
 `WAVEMIND_PGVECTOR_EF_SEARCH=400`, which improves recall materially but still
 misses the `0.95` production target and is slower than the other two profiles.
 The 100k load profile shows Qdrant service is already viable for candidate
-generation; the 1M Qdrant profile shows that default service settings are not
-enough for production recall and need HNSW/search tuning before million-memory
-claims.
+generation on the tested machine. The tuned 1M profile crosses the recall target,
+and the EF sweep gets close to the p99 latency target, but 1M should still be
+treated as tuning-in-progress until a 100+ query p99 run stays below 100 ms.
 If a required package, service, or environment variable is missing, the runner
 marks that engine as `skipped` instead of silently falling back to another
 backend.

{wavemind-2.2.2 → wavemind-2.2.3}/benchmarks/BENCHMARK_LEADERBOARD.md

@@ -18,9 +18,10 @@ This is a compact reader-facing view of checked-in benchmark results. It is not
 | [LongMemEval evidence 50-query smoke](https://github.com/xiaowu0162/LongMemEval) | long-term-agent-memory | evidence recall@k | WaveMind: 0.92 / 15.3 ms | Static vector: 0.6 / 0.337 ms | WaveMind leads on quality |
 | [ANN index latency curve](https://github.com/erikbern/ann-benchmarks) | index-latency | Recall@k | WaveMind numpy: 1 / 6.485 ms | Qdrant local: 1 / 43.5 ms | Quality tie; WaveMind faster |
 | Production index profile | index-latency | Recall@k | WaveMind faiss-persisted: 1 / 3.524 ms | Qdrant service: 1 / 4.414 ms | Quality tie; WaveMind faster |
-| Production load profile 100k | production-scale | Recall@k | WaveMind pgvector: 0.736 / 17.8 ms | Qdrant service: 1 / 10.8 ms | Baseline leads on quality |
-| Production load profile 1M | production-scale | Recall@k | - | Qdrant service: 0.506 / 45.8 ms | No WaveMind result |
-| Scale readiness profile | production-scale | precision@1 | WaveMind structured payloads: 1 / 0.791 ms | - | WaveMind-only check |
+| Production load profile 100k | production-scale | Recall@k | WaveMind pgvector: 0.736 / 17.8 ms | Qdrant service: 1 / 10.3 ms | Baseline leads on quality |
+| Production load profile 1M | production-scale | Recall@k | - | Qdrant service: 0.984 / 116.8 ms | No WaveMind result |
+| Qdrant 1M HNSW ef sweep | production-scale | Recall@k | - | hnsw_ef=2048: 0.977 / 64.8 ms | No WaveMind result |
+| Scale readiness profile | production-scale | precision@1 | WaveMind structured payloads: 1 / 0.837 ms | - | WaveMind-only check |
 | Memory competitor adapter profile | agent-memory | precision@1 | WaveMind: 0.8 / 0.554 ms | - | WaveMind-only check |
 | [LongMemEval answer generation](https://github.com/xiaowu0162/LongMemEval) | long-term-agent-memory | token F1 | WaveMind + qwen2.5:1.5b: 0.333 / - | Chroma static + qwen2.5:1.5b: 0.17 / - | WaveMind leads on quality |
 

{wavemind-2.2.2 → wavemind-2.2.3}/benchmarks/BENCHMARK_REPORT.md

@@ -21,9 +21,10 @@ Planned rows are not claimed wins. They are the public proof path WaveMind must
 | [LongMemEval evidence 50-query smoke](https://github.com/xiaowu0162/LongMemEval) | long-term-agent-memory | implemented | WaveMind: evidence recall@k 0.92, precision@1 0.76, MRR@k 0.83, context saved 0.87, avg latency 15.3, p95 latency 42.9<br>Chroma static: evidence recall@k 0.60, precision@1 0.26, MRR@k 0.39, context saved 0.89, avg latency 13.3, p95 latency 26.0<br>Static vector: evidence recall@k 0.60, precision@1 0.26, MRR@k 0.39, context saved 0.89, avg latency 0.34, p95 latency 2.14<br>Qdrant static: evidence recall@k 0.60, precision@1 0.26, MRR@k 0.39, context saved 0.89, avg latency 180.2, p95 latency 296.7 | Speed up full LongMemEval reruns by reusing per-question candidate indexes or adding a streaming runner mode. |
 | [ANN index latency curve](https://github.com/erikbern/ann-benchmarks) | index-latency | implemented | WaveMind numpy: Recall@k 1.00, avg latency 6.49, p95 latency 6.41, build ms 744.7<br>WaveMind quantized: Recall@k 0.93, avg latency 24.9, p95 latency 37.4, build ms 2088.7<br>WaveMind annoy: Recall@k 0.73, avg latency 4.92, p95 latency 7.37, build ms 4090.1<br>WaveMind faiss: skipped - Install faiss-cpu to use FaissVectorIndex<br>Qdrant local: Recall@k 1.00, avg latency 43.5, p95 latency 59.7, build ms 17525.7 | Tune quantized search kernels, add FAISS on Linux/macOS CI, and test Qdrant service-mode curves beyond 50000 vectors. |
 | Production index profile | index-latency | implemented | WaveMind faiss-persisted: Recall@k 1.00, avg latency 3.52, p95 latency 7.88, build ms 715.9<br>Qdrant service: Recall@k 1.00, avg latency 4.41, p95 latency 5.93, build ms 12269.8<br>WaveMind pgvector: Recall@k 0.81, avg latency 10.9, p95 latency 15.7, build ms 185048.9 | Use the dedicated production load profile for 100000 and 1000000-vector service tests, then tune pgvector and Qdrant for recall/latency. |
-| Production load profile 100k | production-scale | implemented | Qdrant service: Recall@k 1.00, avg latency 10.8, p95 latency 18.8, build ms 39873.2<br>WaveMind pgvector: Recall@k 0.74, avg latency 17.8, p95 latency 23.5, build ms 455703.7<br>WaveMind faiss-persisted: skipped - Set WAVEMIND_FAISS_PATH to use the persisted FAISS backend | Tune pgvector HNSW build/search parameters and add persisted FAISS from the Linux benchmark container. |
-| Production load profile 1M | production-scale | implemented | Qdrant service: Recall@k 0.51, avg latency 45.8, p95 latency 65.2, build ms 563945.5 | Run Qdrant with tuned HNSW/search params, then add FAISS IVF/HNSW and pgvector 1M profiles on a larger disk. |
-| Scale readiness profile | production-scale | implemented | WaveMind cluster planner: simulated memories 1000000, namespaces 4096, nodes 4, replication factor 2, node loss min availability 1.00, placement ms 109.3<br>WaveMind hot cache: queries 2000, capacity 512, hit rate 0.92, evictions 0, p99 lookup ms 0.01<br>WaveMind structured payloads: queries 4, precision@1 1.00, avg latency 0.79, p99 latency ms 1.27 | Move from single-node service profiles to namespace sharding and replicated service runs. |
+| Production load profile 100k | production-scale | implemented | Qdrant service: Recall@k 1.00, avg latency 10.3, p95 latency 19.0, p99 latency ms 21.3, build ms 27439.3<br>WaveMind pgvector: Recall@k 0.74, avg latency 17.8, p95 latency 23.5, build ms 455703.7<br>WaveMind faiss-persisted: skipped - Set WAVEMIND_FAISS_PATH to use the persisted FAISS backend | Tune pgvector HNSW build/search parameters and add persisted FAISS from the Linux benchmark container. |
+| Production load profile 1M | production-scale | implemented | Qdrant service: Recall@k 0.98, avg latency 116.8, p95 latency 153.8, p99 latency ms 209.3, build ms 450674.6 | Tune Qdrant indexing/search params further, then add FAISS IVF/HNSW and pgvector 1M profiles on a larger disk. |
+| Qdrant 1M HNSW ef sweep | production-scale | implemented | hnsw_ef=512: Recall@k 0.75, avg latency 47.2, p95 latency 68.5, p99 latency ms 68.5, max latency ms 68.5<br>hnsw_ef=768: Recall@k 0.85, avg latency 44.0, p95 latency 69.1, p99 latency ms 69.8, max latency ms 69.8<br>hnsw_ef=1024: Recall@k 0.88, avg latency 62.9, p95 latency 81.1, p99 latency ms 85.5, max latency ms 85.5<br>hnsw_ef=1536: Recall@k 0.94, avg latency 65.6, p95 latency 111.2, p99 latency ms 119.7, max latency ms 119.7<br>hnsw_ef=2048: Recall@k 0.98, avg latency 64.8, p95 latency 91.2, p99 latency ms 103.8, max latency ms 103.8 | Repeat with 100+ queries and collection-level HNSW build parameters before claiming a stable production SLO. |
+| Scale readiness profile | production-scale | implemented | WaveMind cluster planner: simulated memories 1000000, namespaces 4096, nodes 4, replication factor 2, node loss min availability 1.00, zone loss min availability 1.00, read quorum 1, write quorum 2, placement ms 115.8<br>WaveMind hot cache: queries 2000, capacity 512, hit rate 0.92, evictions 0, p99 lookup ms 0.00<br>WaveMind structured payloads: queries 4, precision@1 1.00, avg latency 0.84, p99 latency ms 1.07 | Move from single-node service profiles to namespace sharding and replicated service runs. |
 | Memory competitor adapter profile | agent-memory | implemented | WaveMind: precision@1 0.80, precision@3 1.00, stale suppression 1.00, avg latency 0.55, p95 latency 0.83<br>Mem0: skipped - Install Mem0 to run this adapter profile: pip install "mem0ai"<br>Zep: skipped - Install the Zep client package and set ZEP_API_KEY or ZEP_API_URL.<br>LangGraph persistent memory: skipped - Install LangGraph to run this adapter profile: pip install "langgraph" | Add documented setup commands for each competitor adapter and store checked-in results only when those real adapters run. |
 | [LongMemEval answer generation](https://github.com/xiaowu0162/LongMemEval) | long-term-agent-memory | implemented | extractive smoke: queries 20, evidence recall@k 1.00, exact match 0.00, contains answer 0.05, token f1 0.02, avg retrieval ms 3.79, avg generation ms 0.77<br>WaveMind + qwen2.5:0.5b: queries 50, evidence recall@k 0.92, exact match 0.12, contains answer 0.18, token f1 0.18, avg retrieval ms 2.98, avg generation ms 1428.2<br>Chroma static + qwen2.5:0.5b: queries 50, evidence recall@k 0.60, exact match 0.10, contains answer 0.12, token f1 0.13, avg retrieval ms 4.10, avg generation ms 1234.7<br>Qdrant static + qwen2.5:0.5b: queries 50, evidence recall@k 0.60, exact match 0.10, contains answer 0.12, token f1 0.13, avg retrieval ms 63.8, avg generation ms 893.5<br>WaveMind + qwen2.5:1.5b: queries 50, evidence recall@k 0.92, exact match 0.24, contains answer 0.38, token f1 0.33, avg retrieval ms 2.00, avg generation ms 2153.0<br>Chroma static + qwen2.5:1.5b: queries 50, evidence recall@k 0.60, exact match 0.12, contains answer 0.16, token f1 0.17, avg retrieval ms 7.05, avg generation ms 2082.4<br>Qdrant static + qwen2.5:1.5b: queries 50, evidence recall@k 0.60, exact match 0.12, contains answer 0.16, token f1 0.17, avg retrieval ms 100.2, avg generation ms 758.1 | Run all 470 non-abstention questions with a stronger local/API model and add faithfulness/abstention scoring. |
 

{wavemind-2.2.2 → wavemind-2.2.3}/benchmarks/ann_index_curve_benchmark.py

@@ -1,6 +1,7 @@
 from __future__ import annotations
 
 import argparse
+from contextlib import contextmanager
 import json
 import os
 import statistics
@@ -57,11 +58,15 @@ def _recall_at_k(results: list[list[int]], expected: list[set[int]], top_k: int)
     return statistics.mean(recalls) if recalls else 0.0
 
 
-def _p95(values: list[float]) -> float:
+def _percentile(values: list[float], pct: float) -> float:
     if not values:
         return 0.0
     ordered = sorted(values)
-    return ordered[min(len(ordered) - 1, int(len(ordered) * 0.95))]
+    index = min(
+        len(ordered) - 1,
+        max(0, int(round((pct / 100.0) * (len(ordered) - 1)))),
+    )
+    return ordered[index]
 
 
 def run_wavemind_index(
@@ -87,7 +92,10 @@ def run_wavemind_index(
         "engine": f"WaveMind {kind}",
         "recall_at_k": _recall_at_k(ids, expected, top_k),
         "avg_latency_ms": statistics.mean(latencies) if latencies else 0.0,
-        "p95_latency_ms": _p95(latencies),
+        "p50_latency_ms": statistics.median(latencies) if latencies else 0.0,
+        "p95_latency_ms": _percentile(latencies, 95),
+        "p99_latency_ms": _percentile(latencies, 99),
+        "max_latency_ms": max(latencies) if latencies else 0.0,
         "build_ms": build_ms,
         "queries": len(queries),
     }
@@ -108,21 +116,27 @@ def run_qdrant(
         url = ":memory:"
     try:
         from qdrant_client import QdrantClient
-        from qdrant_client.models import Distance, PointStruct, VectorParams
+        from qdrant_client.models import Distance, PointStruct, SearchParams, VectorParams
     except ImportError as exc:
         raise RuntimeError("Install qdrant-client to run the Qdrant ANN curve") from exc
-    if service:
-        client = QdrantClient(url=url, api_key=os.environ.get("WAVEMIND_QDRANT_API_KEY"))
-        engine = "Qdrant service"
-    else:
-        client = QdrantClient(url)
-        engine = "Qdrant local"
+    with _local_no_proxy(url):
+        if service:
+            client = QdrantClient(
+                url=url,
+                api_key=os.environ.get("WAVEMIND_QDRANT_API_KEY"),
+                timeout=float(os.environ.get("WAVEMIND_QDRANT_TIMEOUT", "120")),
+            )
+            engine = "Qdrant service"
+        else:
+            client = QdrantClient(url)
+            engine = "Qdrant local"
     collection_name = f"wavemind_ann_curve_{time.time_ns()}"
     try:
         started = time.perf_counter()
         client.recreate_collection(
             collection_name=collection_name,
             vectors_config=VectorParams(size=int(vectors.shape[1]), distance=Distance.COSINE),
+            timeout=int(os.environ.get("WAVEMIND_QDRANT_COLLECTION_TIMEOUT", "120")),
         )
         batch = []
         for index, vector in enumerate(vectors):
@@ -137,12 +151,26 @@ def run_qdrant(
         latencies: list[float] = []
         for query in queries:
             started = time.perf_counter()
+            search_params = None
+            hnsw_ef = os.environ.get("WAVEMIND_QDRANT_HNSW_EF")
+            exact = os.environ.get("WAVEMIND_QDRANT_EXACT", "").lower() in {
+                "1",
+                "true",
+                "yes",
+                "on",
+            }
+            if hnsw_ef or exact:
+                search_params = SearchParams(
+                    hnsw_ef=int(hnsw_ef) if hnsw_ef else None,
+                    exact=exact or None,
+                )
             hits = list(
                 client.query_points(
                     collection_name=collection_name,
                     query=query.tolist(),
                     limit=top_k,
                     with_payload=False,
+                    search_params=search_params,
                 ).points
             )
             latencies.append((time.perf_counter() - started) * 1000.0)
@@ -151,9 +179,16 @@ def run_qdrant(
             "engine": engine,
             "recall_at_k": _recall_at_k(ids, expected, top_k),
             "avg_latency_ms": statistics.mean(latencies) if latencies else 0.0,
-            "p95_latency_ms": _p95(latencies),
+            "p50_latency_ms": statistics.median(latencies) if latencies else 0.0,
+            "p95_latency_ms": _percentile(latencies, 95),
+            "p99_latency_ms": _percentile(latencies, 99),
+            "max_latency_ms": max(latencies) if latencies else 0.0,
             "build_ms": build_ms,
             "queries": len(queries),
+            "search_params": {
+                "hnsw_ef": int(hnsw_ef) if hnsw_ef else None,
+                "exact": exact,
+            },
         }
     finally:
         if os.environ.get("WAVEMIND_QDRANT_KEEP_COLLECTION", "0").lower() not in {
@@ -204,7 +239,7 @@ def run_size(
         elif key in {"qdrant", "qdrant-local"}:
             try:
                 results.append(run_qdrant(vectors, queries, expected, top_k, service=False))
-            except RuntimeError as exc:
+            except Exception as exc:
                 results.append(
                     {
                         "engine": "Qdrant local",
@@ -215,7 +250,7 @@ def run_size(
         elif key == "qdrant-service":
             try:
                 results.append(run_qdrant(vectors, queries, expected, top_k, service=True))
-            except RuntimeError as exc:
+            except Exception as exc:
                 results.append(
                     {
                         "engine": "Qdrant service",
@@ -235,6 +270,33 @@ def run_size(
     }
 
 
+@contextmanager
+def _local_no_proxy(url: str):
+    if not any(host in url for host in ("127.0.0.1", "localhost", "::1")):
+        yield
+        return
+    original_no_proxy = os.environ.get("NO_PROXY")
+    original_no_proxy_lower = os.environ.get("no_proxy")
+    local_hosts = "127.0.0.1,localhost,::1"
+    os.environ["NO_PROXY"] = (
+        f"{original_no_proxy},{local_hosts}" if original_no_proxy else local_hosts
+    )
+    os.environ["no_proxy"] = (
+        f"{original_no_proxy_lower},{local_hosts}" if original_no_proxy_lower else local_hosts
+    )
+    try:
+        yield
+    finally:
+        if original_no_proxy is None:
+            os.environ.pop("NO_PROXY", None)
+        else:
+            os.environ["NO_PROXY"] = original_no_proxy
+        if original_no_proxy_lower is None:
+            os.environ.pop("no_proxy", None)
+        else:
+            os.environ["no_proxy"] = original_no_proxy_lower
+
+
 def run_benchmark(
     sizes: Iterable[int],
     dim: int,
@@ -276,18 +338,19 @@ def run_benchmark(
 
 def print_table(payload: dict[str, Any]) -> None:
     top_k = payload["scenario"]["top_k"]
-    print(f"| vectors | engine | recall@{top_k} | avg latency | p95 latency | build |")
-    print("|---:|---|---:|---:|---:|---:|")
+    print(f"| vectors | engine | recall@{top_k} | avg latency | p95 latency | p99 latency | build |")
+    print("|---:|---|---:|---:|---:|---:|---:|")
     for size_result in payload["results"]:
         for result in size_result["results"]:
             if result.get("skipped"):
-                print(f"| {size_result['vectors']} | {result['engine']} | skipped | - | - | - |")
+                print(f"| {size_result['vectors']} | {result['engine']} | skipped | - | - | - | - |")
                 continue
             print(
                 f"| {size_result['vectors']} | {result['engine']} | "
                 f"{result['recall_at_k']:.3f} | "
                 f"{result['avg_latency_ms']:.2f} ms | "
                 f"{result['p95_latency_ms']:.2f} ms | "
+                f"{result['p99_latency_ms']:.2f} ms | "
                 f"{result['build_ms']:.1f} ms |"
             )
 

{wavemind-2.2.2 → wavemind-2.2.3}/benchmarks/benchmark_matrix_results.json

@@ -625,14 +625,16 @@
         "recall@10",
         "avg_latency_ms",
         "p95_latency_ms",
+        "p99_latency_ms",
         "build_ms"
       ],
       "current": {
         "Qdrant service": {
           "recall_at_k": 1.0,
-          "avg_latency_ms": 10.75583399971947,
-          "p95_latency_ms": 18.775899952743202,
-          "build_ms": 39873.22780000977
+          "avg_latency_ms": 10.275224001379684,
+          "p95_latency_ms": 18.97340000141412,
+          "p99_latency_ms": 21.25629998045042,
+          "build_ms": 27439.262800035067
         },
         "WaveMind pgvector": {
           "recall_at_k": 0.736,
@@ -645,7 +647,7 @@
           "reason": "Set WAVEMIND_FAISS_PATH to use the persisted FAISS backend"
         }
       },
-      "target": "Reach recall@10 >= 0.95 and avg latency < 20 ms on at least one production service backend at 100000 memories.",
+      "target": "Reach recall@10 >= 0.95 and p99 latency < 100 ms on at least one production service backend at 100000 memories.",
       "next_step": "Tune pgvector HNSW build/search parameters and add persisted FAISS from the Linux benchmark container."
     },
     {
@@ -653,8 +655,8 @@
       "name": "Production load profile 1M",
       "category": "production-scale",
       "status": "implemented",
-      "source": "benchmarks/production_load_qdrant_1m_results.json",
-      "dataset": "1000000 generated normalized 128-d vectors; Qdrant service-only recall@10/latency profile with 50 queries.",
+      "source": "benchmarks/production_load_qdrant_1m_tuned_results.json",
+      "dataset": "1000000 generated normalized 128-d vectors; Qdrant service-only recall@10/latency profile with tuned HNSW search.",
       "competitors": [
         "Qdrant service"
       ],
@@ -662,18 +664,76 @@
         "recall@10",
         "avg_latency_ms",
         "p95_latency_ms",
+        "p99_latency_ms",
         "build_ms"
       ],
       "current": {
         "Qdrant service": {
-          "recall_at_k": 0.506,
-          "avg_latency_ms": 45.80613199970685,
-          "p95_latency_ms": 65.17819996224716,
-          "build_ms": 563945.4993000254
+          "recall_at_k": 0.984,
+          "avg_latency_ms": 116.80030399700627,
+          "p95_latency_ms": 153.84380001341924,
+          "p99_latency_ms": 209.27749999100342,
+          "build_ms": 450674.6019999846
         }
       },
-      "target": "Tune service settings until recall@10 >= 0.95 with p95 latency below 100 ms at 1M vectors.",
-      "next_step": "Run Qdrant with tuned HNSW/search params, then add FAISS IVF/HNSW and pgvector 1M profiles on a larger disk."
+      "target": "Keep recall@10 >= 0.95 and push p99 latency below 100 ms at 1M vectors.",
+      "next_step": "Tune Qdrant indexing/search params further, then add FAISS IVF/HNSW and pgvector 1M profiles on a larger disk."
+    },
+    {
+      "id": "production_load_qdrant_1m_ef_sweep",
+      "name": "Qdrant 1M HNSW ef sweep",
+      "category": "production-scale",
+      "status": "implemented",
+      "source": "benchmarks/production_load_qdrant_1m_ef_sweep_results.json",
+      "dataset": "1000000 generated normalized 128-d vectors; one Qdrant service collection queried with multiple hnsw_ef settings.",
+      "competitors": [
+        "Qdrant service"
+      ],
+      "metrics": [
+        "recall@10",
+        "avg_latency_ms",
+        "p95_latency_ms",
+        "p99_latency_ms"
+      ],
+      "current": {
+        "hnsw_ef=512": {
+          "recall_at_k": 0.7533333333333333,
+          "avg_latency_ms": 47.16066332766786,
+          "p95_latency_ms": 68.49600002169609,
+          "p99_latency_ms": 68.51829995866865,
+          "max_latency_ms": 68.51829995866865
+        },
+        "hnsw_ef=768": {
+          "recall_at_k": 0.85,
+          "avg_latency_ms": 44.041876665626965,
+          "p95_latency_ms": 69.07949998276308,
+          "p99_latency_ms": 69.75289998808876,
+          "max_latency_ms": 69.75289998808876
+        },
+        "hnsw_ef=1024": {
+          "recall_at_k": 0.8833333333333333,
+          "avg_latency_ms": 62.91633000170501,
+          "p95_latency_ms": 81.1338999774307,
+          "p99_latency_ms": 85.5405000038445,
+          "max_latency_ms": 85.5405000038445
+        },
+        "hnsw_ef=1536": {
+          "recall_at_k": 0.9366666666666666,
+          "avg_latency_ms": 65.5569299919686,
+          "p95_latency_ms": 111.21890001231804,
+          "p99_latency_ms": 119.66809997102246,
+          "max_latency_ms": 119.66809997102246
+        },
+        "hnsw_ef=2048": {
+          "recall_at_k": 0.9766666666666667,
+          "avg_latency_ms": 64.75662334123626,
+          "p95_latency_ms": 91.18230000603944,
+          "p99_latency_ms": 103.77270000753924,
+          "max_latency_ms": 103.77270000753924
+        }
+      },
+      "target": "Find a setting that keeps recall@10 >= 0.95 while keeping p99 latency below 100 ms.",
+      "next_step": "Repeat with 100+ queries and collection-level HNSW build parameters before claiming a stable production SLO."
     },
     {
       "id": "scale_readiness",
@@ -701,20 +761,23 @@
           "nodes": 4,
           "replication_factor": 2,
           "node_loss_min_availability": 1.0,
-          "placement_ms": 109.29749999195337
+          "zone_loss_min_availability": 1.0,
+          "read_quorum": 1,
+          "write_quorum": 2,
+          "placement_ms": 115.80999998841435
         },
         "WaveMind hot cache": {
           "queries": 2000,
           "capacity": 512,
           "hit_rate": 0.92,
           "evictions": 0,
-          "p99_lookup_ms": 0.009100011084228754
+          "p99_lookup_ms": 0.003500026650726795
         },
         "WaveMind structured payloads": {
           "queries": 4,
           "precision_at_1": 1.0,
-          "avg_latency_ms": 0.7908999978099018,
-          "p99_latency_ms": 1.2743999832309783
+          "avg_latency_ms": 0.8370749856112525,
+          "p99_latency_ms": 1.0724000167101622
         }
       },
       "target": "Prove the production foundation before heavier 100k, 1M, and 10M vector load tests: deterministic placement, survivable replicas, hot-cache behavior, and structured payload recall.",