hyperstreamdb 0.5.4__tar.gz → 0.6.0__tar.gz

hyperstreamdb-0.6.0/.cargo/audit.toml

@@ -0,0 +1,25 @@
+# Cargo Audit Configuration
+# https://github.com/rustsec/rustsec/tree/main/cargo-audit
+
+[advisories]
+ignore = [
+    # Transitive dependencies & existing ignores
+    "RUSTSEC-2026-0104", # rustls-webpki: CRL panic
+    "RUSTSEC-2026-0099", # rustls-webpki: wildcard name constraints
+    "RUSTSEC-2026-0098", # rustls-webpki: URI name constraints
+    "RUSTSEC-2026-0097", # rand: unsound with custom logger
+    "RUSTSEC-2024-0437", # protobuf: uncontrolled recursion
+    "RUSTSEC-2026-0177", # pyo3: missing Sync bound
+    "RUSTSEC-2026-0176", # pyo3: out-of-bounds read in iterator
+    "RUSTSEC-2023-0071", # rsa: Marvin attack
+    "RUSTSEC-2026-0258", # h2: unbounded empty DATA frames
+    "RUSTSEC-2026-0195", # quick-xml: unbounded namespace allocation
+    "RUSTSEC-2026-0194", # quick-xml: quadratic run time in duplicate attribute check
+    "RUSTSEC-2025-0134", # rustls-pemfile: unmaintained advisory
+    "RUSTSEC-2026-0190", # anyhow: Error::downcast_mut unsoundness
+    "RUSTSEC-2026-0221", # event-listener: !Send tags across thread boundaries
+    "RUSTSEC-2026-0204", # crossbeam-epoch: pointer dereference
+    "RUSTSEC-2026-0234", # rkyv: hash table out of bounds
+    "RUSTSEC-2026-0233", # rkyv: use-after-free
+    "RUSTSEC-2026-0235", # rkyv: Rc/Arc out of bounds
+]

hyperstreamdb-0.6.0/.dockerignore

@@ -0,0 +1,15 @@
+target/
+.git/
+.github/
+.venv/
+venv/
+*.db
+*.parquet
+*.log
+test_*
+rag_db/
+news_db/
+demo/
+benchmarks/
+__pycache__/
+*.pyc

{hyperstreamdb-0.5.4 → hyperstreamdb-0.6.0}/.gitignore

@@ -109,3 +109,4 @@ news_db/
 venv_demo/
 .env
 
+COMMERCIALIZATION_STRATEGY.md

hyperstreamdb-0.6.0/BENCHMARK_FINDINGS.md

@@ -0,0 +1,89 @@
+# Benchmark Findings — `hypersearch` vs Elasticsearch 7.10.2
+
+Step 4.2 of the OpenSearch/ES integration plan. Compares the `hypersearch`
+REST server against a single-node Elasticsearch 7.10.2 (Docker, security
+disabled, 1 shard / 0 replicas, refresh disabled during ingest, default 1 GiB
+JVM heap), feeding both the **same document stream** (one document per HTTP
+POST) on the same host.
+
+- **Tool:** [`benchmarks/competitive/benchmark_es710.py`](benchmarks/competitive/benchmark_es710.py)
+- **Raw results:** `benchmarks/competitive/benchmark_results/es710_hypersearch_*.json` / `.md`
+
+## How to run
+
+```bash
+# From the repo root, after building the binary:
+cargo build -p hyperstreamdb-search --bin hypersearch
+
+# Quick validation run (200 docs, 20 query runs, dim 32) — the M4 gate:
+./venv/bin/python benchmarks/competitive/benchmark_es710.py --quick
+
+# Full target runs (spec Step 4.2):
+./venv/bin/python benchmarks/competitive/benchmark_es710.py --size 100000 --runs 100
+./venv/bin/python benchmarks/competitive/benchmark_es710.py --size 1000000 --runs 100
+
+# hypersearch-only (no ES) or keep the ES container for inspection:
+./venv/bin/python benchmarks/competitive/benchmark_es710.py --skip-es
+./venv/bin/python benchmarks/competitive/benchmark_es710.py --keep-es
+```
+
+The ES 7.10.2 image (`docker.elastic.co/elasticsearch/elasticsearch:7.10.2`,
+~814 MB) is pulled on first run.
+
+## Results — quick run (200 docs, dim 32, 20 runs)
+
+| Operation | HyperStreamDB p50 | HyperStreamDB p95 | ES 7.10.2 p50 | ES 7.10.2 p95 |
+|-----------|-------------------|-------------------|---------------|---------------|
+| Ingest (docs/s) | **879** | — | 221 | — |
+| Refresh (ms) | 19,560 | — | **46** | — |
+| `match` BM25 (ms) | 86.1 | 100.4 | **3.3** | 4.8 |
+| Filtered `match`+`term` (ms) | 106.5 | 120.1 | **2.6** | 2.9 |
+| `knn` HNSW (ms) | 85.6 | 89.0 | n/a¹ | n/a¹ |
+| Hybrid `match`+`knn` (ms) | 101.5 | 109.3 | n/a¹ | n/a¹ |
+
+¹ ES 7.10 has no `dense_vector` type, so `knn` and hybrid are hypersearch-only.
+
+## Findings
+
+1. **Search latency is within the target envelope.** Hypersearch query
+   latencies (86–120 ms p95) fall inside the plan's **50–200 ms** target for
+   website/document search. This is the intended operating point.
+
+2. **ES is far faster on search — by design.** ES 7.10 answers in 3–6 ms
+   because it pins segments in memory (sub-millisecond, JVM heap + hot SSD).
+   Hypersearch trades that latency for an object-storage-native, scale-to-zero
+   model. This is the core positioning trade-off, not a regression.
+
+3. **Ingest throughput favors hypersearch** (879 vs 221 docs/s) for single-doc
+   POSTs, because writes land in the memtable + WAL immediately. (The
+   `_bulk` endpoint, added in M3, improves this further for batch ingestion.)
+
+4. **Refresh (index build) is the one-time cost.** Hypersearch's refresh
+   (19.5 s for 200 docs) includes building the BM25 inverted index + HNSW
+   vector index and committing the manifest/Iceberg metadata; ES's refresh
+   (46 ms) only flushes the translog. As document count grows, hypersearch's
+   per-refresh index build grows too — this is the main area to optimize
+   (incremental index builds, background/async indexing, and the on-demand
+   index-file cache) before the 100k/1M runs.
+
+5. **`knn` and hybrid are differentiators.** ES 7.10 cannot do native vector
+   search; hypersearch serves `knn` and RRF-fused hybrid in the same 50–200 ms
+   envelope with no extra infrastructure.
+
+## Next steps (100k / 1M)
+
+The quick run validates the pipeline end-to-end (the M4 gate). The full
+100k/1M runs are long-running (ingest alone is ~20 min for 1M docs at the
+observed hypersearch rate, ~75 min for ES, plus index builds and query runs)
+and should be run on a dedicated host:
+
+```bash
+./venv/bin/python benchmarks/competitive/benchmark_es710.py --size 100000 --runs 100
+./venv/bin/python benchmarks/competitive/benchmark_es710.py --size 1000000 --runs 100
+```
+
+The S3/MinIO variant (spec: "local FS, then S3 via MinIO") sets
+`HYPERSEARCH_STORAGE_URI=s3://<bucket>/search` with a running MinIO and the
+same commands; the on-demand index-file cache
+([`hyperstreamdb-search/src/index_cache.rs`](hyperstreamdb-search/src/index_cache.rs))
+is what makes the S3 path viable at scale.

{hyperstreamdb-0.5.4 → hyperstreamdb-0.6.0}/CHANGELOG.md

@@ -9,6 +9,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ---
 
+## [0.6.0] - 2026-09-09
+
+### Added
+- **Multi-Protocol Gateway Ecosystem**:
+  - **`hyperstreamdb-search` Service (`hypersearch` binary)**: Dual Elasticsearch 7.10 (Port 9200) and Qdrant (Port 6333) compatible REST APIs over HyperStreamDB tables.
+    - Elasticsearch 7.10 API: Full document CRUD (`POST /{index}/_doc`), hybrid search (`POST /{index}/_search` with BM25 + HNSW kNN + Reciprocal Rank Fusion), index management (`PUT /{index}`, `POST /{index}/_refresh`), and cluster health (`GET /_cluster/health`).
+    - Qdrant REST API: Collection management (`/collections/{name}`), point upsert/retrieval (`/collections/{name}/points`), and vector search (`/collections/{name}/points/search`).
+    - Prometheus metrics exporter on `/metrics` (Port 9090).
+  - **`hyperstreamdb-flight` Gateway Service**: Native Arrow Flight SQL gRPC gateway (Port 50051) enabling zero-copy analytics for DuckDB, Polars, Apache Spark, and JDBC/ODBC BI tools via standard Flight SQL/ADBC.
+- **Multi-Flavor GPU Acceleration & Hardware Auto-Detection**:
+  - Optional GPU acceleration exposed across `hyperstreamdb-search` and `hyperstreamdb-flight` via `cuda`, `wgpu`, `rocm`, `intel`, and `all-gpu` feature flags.
+  - Runtime device selection via `HYPERSEARCH_DEVICE=auto|cuda[:N]|rocm[:N]|intel[:N]|mps|cpu`.
+  - Active compute backend (`compute` block) exposed in `GET /` cluster info and `GET /_cluster/stats`.
+  - `docker/Dockerfile.gpu`: Multi-flavor GPU container image with CUDA 12 runtime, NVRTC JIT compilation, and Vulkan/Mesa drivers for AMD Radeon and Intel Arc.
+  - `docker/docker-compose.gpu.yml`: Compose GPU override with hardware reservations and device pass-through.
+- **Iceberg Compaction Resilience & Index Recovery**:
+  - `Table::recover_indexes_async(&self)` / `recover_indexes(&self)`: Re-indexes data files that are missing overlay index sidecars, recovering fast vector (HNSW) and keyword (BM25) search after external Iceberg tools (Spark `rewriteDataFiles`, Trino `OPTIMIZE`, PyIceberg) compact table data files.
+- **Docker Container Infrastructure**:
+  - `hyperstreamdb/quickstart:latest`: Single all-in-one developer container running ES 7.10 (9200), Qdrant (6333), and Flight SQL (50051).
+  - `hyperstreamdb/search:latest`: Standalone production search microservice.
+  - `hyperstreamdb/flight:latest`: Standalone production Arrow Flight SQL microservice.
+  - `docker/docker-compose.quickstart.yml`: Single-command full stack with MinIO (S3), Project Nessie catalog, and HyperStreamDB.
+  - `docker-compose.production.yml`: Production multi-container configuration with health checks and resource limits.
+- Okapi BM25 keyword scoring (tunable `k1`/`b`) with an English analyzer in the core engine; public `keyword_search_index` API.
+- Smart hybrid trigger fusing keyword (BM25) and vector (HNSW) results via reciprocal rank fusion (RRF, k=60).
+- Background segment index builds with `wait_for_background_tasks_async` for deterministic refresh semantics.
+
 ## [0.5.3] - 2026-06-21
 
 ### Added