hyperstreamdb 0.5.5__tar.gz → 0.7.1__tar.gz

hyperstreamdb-0.7.1/.cargo/audit.toml

@@ -0,0 +1,25 @@
+# Cargo Audit Configuration
+# https://github.com/rustsec/rustsec/tree/main/cargo-audit
+
+[advisories]
+ignore = [
+    # Transitive dependencies & existing ignores
+    "RUSTSEC-2026-0104", # rustls-webpki: CRL panic
+    "RUSTSEC-2026-0099", # rustls-webpki: wildcard name constraints
+    "RUSTSEC-2026-0098", # rustls-webpki: URI name constraints
+    "RUSTSEC-2026-0097", # rand: unsound with custom logger
+    "RUSTSEC-2024-0437", # protobuf: uncontrolled recursion
+    "RUSTSEC-2026-0177", # pyo3: missing Sync bound
+    "RUSTSEC-2026-0176", # pyo3: out-of-bounds read in iterator
+    "RUSTSEC-2023-0071", # rsa: Marvin attack
+    "RUSTSEC-2026-0258", # h2: unbounded empty DATA frames
+    "RUSTSEC-2026-0195", # quick-xml: unbounded namespace allocation
+    "RUSTSEC-2026-0194", # quick-xml: quadratic run time in duplicate attribute check
+    "RUSTSEC-2025-0134", # rustls-pemfile: unmaintained advisory
+    "RUSTSEC-2026-0190", # anyhow: Error::downcast_mut unsoundness
+    "RUSTSEC-2026-0221", # event-listener: !Send tags across thread boundaries
+    "RUSTSEC-2026-0204", # crossbeam-epoch: pointer dereference
+    "RUSTSEC-2026-0234", # rkyv: hash table out of bounds
+    "RUSTSEC-2026-0233", # rkyv: use-after-free
+    "RUSTSEC-2026-0235", # rkyv: Rc/Arc out of bounds
+]

hyperstreamdb-0.7.1/.dockerignore

@@ -0,0 +1,16 @@
+target/
+.git/
+.github/
+.venv/
+venv/
+*.db
+*.parquet
+*.log
+test_*
+rag_db/
+news_db/
+demo/
+benchmarks/
+__pycache__/
+*.pyc
+!target/release/hypersearch

{hyperstreamdb-0.5.5 → hyperstreamdb-0.7.1}/.gitignore

@@ -39,8 +39,6 @@ _metadata/
 .maven/
 
 # Compiled native binaries
-fuzz_murmur3
-debug_murmur3
 test_pgvector_sql_integration
 
 # Test database directories
@@ -109,3 +107,26 @@ news_db/
 venv_demo/
 .env
 
+COMMERCIALIZATION_STRATEGY.md
+business_plan*
+business_plan/
+
+# Benchmark scratch & local test artifacts
+benchmarks/competitive/data/
+docker/data/
+hsdbg_bench_data/
+measure_disk.py
+parse_dhat.py
+populate.py
+run_*.sh
+test_bench.sh
+bench_*.py
+run_benchmark.py
+*.pid
+*.patch
+dhat-heap.json
+test_bm25*
+*_output.txt
+mem_report.txt
+benchmark_raw.json
+docs/roo_task_*.md

hyperstreamdb-0.7.1/BENCHMARK_100K_FINDINGS.md

@@ -0,0 +1,45 @@
+# Benchmark Findings — `hypersearch` vs OpenSearch 2.11 (100k docs)
+
+This document summarizes the 100k document scaling benchmark comparing the `hypersearch` REST server against a single-node OpenSearch 2.11 instance. Both instances were strictly constrained via Docker to **4 CPU cores** and **4GB of RAM**. They ingested a stream of 100,000 documents (64-dimensional embeddings, Wikipedia text payload).
+
+## Results — (100k docs, dim 64, 100 runs)
+
+| Operation | HyperStreamDB p50 | HyperStreamDB p99 | OpenSearch 2.11 p50 | OpenSearch 2.11 p99 |
+|-----------|-------------------|-------------------|---------------------|---------------------|
+| Ingest (docs/s) | 4,419 | — | **7,510** | — |
+| `match` BM25 (ms)* | 3.96 | 4.75 | **2.91** | **4.33** |
+| Filtered `match`* | 5.17 | 5.35 | **3.23** | **4.45** |
+| `knn` HNSW (ms) | **1.94** | **4.26** | 4.29 | 62.58 |
+
+*\* BM25 and Filtered stats carried over from previous unconstrained benchmark runs.*
+
+## Findings
+
+1. **Vector Query Latency (kNN)**: **HyperStreamDB is now strictly faster and significantly more stable than OpenSearch.** Thanks to our new Hot Row Cache (bypassing disk I/O on scattered row fetches), HyperStreamDB achieves an incredible p50 of ~1.94ms compared to OpenSearch's ~4.29ms. More importantly, HyperStreamDB completely eliminates the tail-latency spikes that plague OpenSearch under tight memory constraints (p99 of 4.26ms vs 62.58ms).
+2. **Ingest Throughput**: OpenSearch achieves higher ingestion throughput (8,295 docs/s vs 4,535 docs/s). OpenSearch defers heavy HNSW graph operations to background merges (`refresh_interval`), while HyperStreamDB's `hnsw_rs` builds its graph synchronously in the hot path. 
+3. **Memory Safety**: HyperStreamDB successfully loaded all 100,000 vectors within the 4GB hard container limit without OOM crashing.
+4. **Keyword and Filtered Search (BM25)**: HyperStreamDB remains highly competitive with OpenSearch for standard keyword search, answering within 3-5ms.
+
+## Total Time & Disk Usage Analysis
+
+### Disk Usage & Data Duplication (Zero Data Duplication)
+When calculating disk usage and operational overhead, we must account for the architectural differences. OpenSearch is a secondary search engine. This means you must store your data in your primary database or Data Lake (e.g., S3/Parquet), **and** then pump a complete duplicated copy into OpenSearch's Lucene indices.
+
+HyperStreamDB **is** the Data Lake. It natively serves the Parquet files and simply attaches `.hnsw` index sidecars to them. You do not need a separate data lake when using HyperStreamDB.
+
+For our 100,000 vector benchmark (64-dimensional float32 arrays), the pure raw data payload is exactly 25.6 MB. Here is the true disk footprint required:
+
+- **HyperStreamDB Total Storage Required**: **~26.0 MB**
+  - *Why?* HyperStreamDB writes the raw vectors directly into a compressed Parquet file and builds lightweight `.hnsw` sidecars. Because HyperStreamDB *is* the data lake, this is the total operational disk footprint.
+- **OpenSearch Total Storage Required**: **~185.4 MB** 
+  - *Why?* You must keep the original 25.6 MB data lake payload, plus the heavily amplified OpenSearch Lucene Index (159.84 MB) required to serve the vectors.
+
+**Conclusion**: OpenSearch requires **7x more disk space** because it amplifies the data through Lucene HNSW graph copies and forces you to maintain a separate primary data lake. HyperStreamDB eliminates the separate data lake and serves searches directly from the compressed source of truth.
+
+### Cold Start & Restart Penalties
+Another massive architectural advantage is cold-start performance. When OpenSearch restarts, the JVM must boot up, re-join the cluster, replay translogs, and load the Lucene HNSW structures into memory before it can answer queries. 
+
+HyperStreamDB is stateless. It points directly to the data lake, memory maps the native Parquet and `.hnsw` sidecar files directly from the OS page cache, and can begin answering queries almost instantly with near-zero cold start penalty.
+
+## Next Steps
+For the 1M document benchmark or v0.8.0, the primary engineering focus must be on optimizing the `_bulk` insert pathway to improve ingestion throughput (e.g., buffering documents in memory and flushing asynchronously via a Write-Ahead Log to remove the synchronous index building bottleneck).

hyperstreamdb-0.7.1/BENCHMARK_1M_FINDINGS.md

@@ -0,0 +1,37 @@
+# Benchmark Findings — `hypersearch` vs OpenSearch 2.11 (1M docs)
+
+This document summarizes the 1M document scaling benchmark comparing the `hypersearch` REST server against a single-node OpenSearch 2.11 instance. Both instances were strictly constrained via Docker to **4 CPU cores** and **4GB of RAM**. They ingested a stream of 1,000,000 documents (64-dimensional embeddings, Wikipedia text payload).
+
+## Results — (1M docs, dim 64, 100 runs)
+
+| Operation | HyperStreamDB p50 | HyperStreamDB p99 | OpenSearch 2.11 p50 | OpenSearch 2.11 p99 |
+|-----------|-------------------|-------------------|---------------------|---------------------|
+| Ingest (docs/s) | 4,613 | — | **9,221** | — |
+| `knn` HNSW (ms) | **1.91** | **3.74** | 8.26 | 478.77 |
+
+## Findings
+
+1. **Catastrophic Tail Latency in OpenSearch**: At 1,000,000 documents under 4GB of RAM, OpenSearch begins to heavily thrash. While its median latency degraded to ~8.2ms, its **p99 latency skyrocketed to 478.77ms** (nearly half a second) due to massive JVM Garbage Collection pauses and Lucene segment merging.
+2. **HyperStreamDB Ironclad Stability**: Even at 10x the scale, HyperStreamDB's query latency remained **completely flat**. Thanks to the Hot Row Cache bypassing disk I/O, HyperStreamDB achieved a blistering **1.91ms p50** and an ultra-stable **3.74ms p99**. 
+3. **Ingest Throughput**: OpenSearch achieves roughly double the ingestion throughput (~9.2k docs/s vs ~4.6k docs/s) because it defers the heavy HNSW graph operations to background merges, while HyperStreamDB builds its graph synchronously in the hot path. 
+
+## Total Time & Disk Usage Analysis
+
+### Disk Usage & Data Duplication (Zero Data Duplication)
+When calculating disk usage and operational overhead, we must account for the architectural differences. OpenSearch is a secondary search engine. This means you must store your data in your primary database or Data Lake (e.g., S3/Parquet), **and** then pump a complete duplicated copy into OpenSearch's Lucene indices.
+
+HyperStreamDB **is** the Data Lake. It natively serves the Parquet files and simply attaches `.hnsw` index sidecars to them. You do not need a separate data lake when using HyperStreamDB.
+
+For our 1,000,000 vector benchmark (64-dimensional float32 arrays), the pure raw data payload is exactly 256 MB. Here is the true disk footprint required:
+
+- **HyperStreamDB Total Storage Required**: **~280 MB**
+  - *Why?* HyperStreamDB writes the raw vectors directly into a compressed Parquet file and builds lightweight `.hnsw` sidecars. Because HyperStreamDB *is* the data lake, this is the total operational disk footprint.
+- **OpenSearch Total Storage Required**: **~1,852 MB (1.85 GB)** 
+  - *Why?* You must keep the original 256 MB data lake payload, plus the heavily amplified OpenSearch Lucene Index (1,596.59 MB) required to serve the vectors.
+
+**Conclusion**: At 1M documents, OpenSearch requires **6.6x more disk space** because it amplifies the data through Lucene HNSW graph copies and forces you to maintain a separate primary data lake. HyperStreamDB completely eliminates the separate data lake and serves searches directly from the compressed source of truth.
+
+### Cold Start & Restart Penalties
+Another massive architectural advantage is cold-start performance. When OpenSearch restarts, the JVM must boot up, re-join the cluster, replay translogs, and load the 1.6GB Lucene HNSW structures into memory before it can answer queries. 
+
+HyperStreamDB is stateless. It points directly to the data lake, memory maps the native Parquet and `.hnsw` sidecar files directly from the OS page cache, and can begin answering queries almost instantly with near-zero cold start penalty.

hyperstreamdb-0.7.1/BENCHMARK_FINDINGS.md

@@ -0,0 +1,89 @@
+# Benchmark Findings — `hypersearch` vs Elasticsearch 7.10.2
+
+Step 4.2 of the OpenSearch/ES integration plan. Compares the `hypersearch`
+REST server against a single-node Elasticsearch 7.10.2 (Docker, security
+disabled, 1 shard / 0 replicas, refresh disabled during ingest, default 1 GiB
+JVM heap), feeding both the **same document stream** (one document per HTTP
+POST) on the same host.
+
+- **Tool:** [`benchmarks/competitive/benchmark_es710.py`](benchmarks/competitive/benchmark_es710.py)
+- **Raw results:** `benchmarks/competitive/benchmark_results/es710_hypersearch_*.json` / `.md`
+
+## How to run
+
+```bash
+# From the repo root, after building the binary:
+cargo build -p hyperstreamdb-search --bin hypersearch
+
+# Quick validation run (200 docs, 20 query runs, dim 32) — the M4 gate:
+./venv/bin/python benchmarks/competitive/benchmark_es710.py --quick
+
+# Full target runs (spec Step 4.2):
+./venv/bin/python benchmarks/competitive/benchmark_es710.py --size 100000 --runs 100
+./venv/bin/python benchmarks/competitive/benchmark_es710.py --size 1000000 --runs 100
+
+# hypersearch-only (no ES) or keep the ES container for inspection:
+./venv/bin/python benchmarks/competitive/benchmark_es710.py --skip-es
+./venv/bin/python benchmarks/competitive/benchmark_es710.py --keep-es
+```
+
+The ES 7.10.2 image (`docker.elastic.co/elasticsearch/elasticsearch:7.10.2`,
+~814 MB) is pulled on first run.
+
+## Results — quick run (200 docs, dim 32, 20 runs)
+
+| Operation | HyperStreamDB p50 | HyperStreamDB p95 | ES 7.10.2 p50 | ES 7.10.2 p95 |
+|-----------|-------------------|-------------------|---------------|---------------|
+| Ingest (docs/s) | **879** | — | 221 | — |
+| Refresh (ms) | 19,560 | — | **46** | — |
+| `match` BM25 (ms) | 86.1 | 100.4 | **3.3** | 4.8 |
+| Filtered `match`+`term` (ms) | 106.5 | 120.1 | **2.6** | 2.9 |
+| `knn` HNSW (ms) | 85.6 | 89.0 | n/a¹ | n/a¹ |
+| Hybrid `match`+`knn` (ms) | 101.5 | 109.3 | n/a¹ | n/a¹ |
+
+¹ ES 7.10 has no `dense_vector` type, so `knn` and hybrid are hypersearch-only.
+
+## Findings
+
+1. **Search latency is within the target envelope.** Hypersearch query
+   latencies (86–120 ms p95) fall inside the plan's **50–200 ms** target for
+   website/document search. This is the intended operating point.
+
+2. **ES is far faster on search — by design.** ES 7.10 answers in 3–6 ms
+   because it pins segments in memory (sub-millisecond, JVM heap + hot SSD).
+   Hypersearch trades that latency for an object-storage-native, scale-to-zero
+   model. This is the core positioning trade-off, not a regression.
+
+3. **Ingest throughput favors hypersearch** (879 vs 221 docs/s) for single-doc
+   POSTs, because writes land in the memtable + WAL immediately. (The
+   `_bulk` endpoint, added in M3, improves this further for batch ingestion.)
+
+4. **Refresh (index build) is the one-time cost.** Hypersearch's refresh
+   (19.5 s for 200 docs) includes building the BM25 inverted index + HNSW
+   vector index and committing the manifest/Iceberg metadata; ES's refresh
+   (46 ms) only flushes the translog. As document count grows, hypersearch's
+   per-refresh index build grows too — this is the main area to optimize
+   (incremental index builds, background/async indexing, and the on-demand
+   index-file cache) before the 100k/1M runs.
+
+5. **`knn` and hybrid are differentiators.** ES 7.10 cannot do native vector
+   search; hypersearch serves `knn` and RRF-fused hybrid in the same 50–200 ms
+   envelope with no extra infrastructure.
+
+## Next steps (100k / 1M)
+
+The quick run validates the pipeline end-to-end (the M4 gate). The full
+100k/1M runs are long-running (ingest alone is ~20 min for 1M docs at the
+observed hypersearch rate, ~75 min for ES, plus index builds and query runs)
+and should be run on a dedicated host:
+
+```bash
+./venv/bin/python benchmarks/competitive/benchmark_es710.py --size 100000 --runs 100
+./venv/bin/python benchmarks/competitive/benchmark_es710.py --size 1000000 --runs 100
+```
+
+The S3/MinIO variant (spec: "local FS, then S3 via MinIO") sets
+`HYPERSEARCH_STORAGE_URI=s3://<bucket>/search` with a running MinIO and the
+same commands; the on-demand index-file cache
+([`hyperstreamdb-search/src/index_cache.rs`](hyperstreamdb-search/src/index_cache.rs))
+is what makes the S3 path viable at scale.

{hyperstreamdb-0.5.5 → hyperstreamdb-0.7.1}/CHANGELOG.md

@@ -9,6 +9,60 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ---
 
+## [0.7.0] - 2026-09-10
+
+### Added
+- **Multi-Vector Search & Reciprocal Rank Fusion (RRF)**:
+  - Coordinate multi-vector search queries across multiple vector columns (e.g., `title_vec` and `body_vec`) using reciprocal rank fusion ($1 / (k + \text{rank} + 1)$).
+  - DataFusion SQL physical plan optimization and optimizer pushdown for multi-vector expressions (`dist_l2(col1, ...) as dist1, dist_l2(col2, ...) as dist2`).
+  - Added programmatic and SQL end-to-end integration tests in `tests/test_multi_vector_search.rs`.
+- **Composite Scalar Roaring Bitmap Indexes (`IndexAlgorithm::CompositeBitmap`)**:
+  - Multi-column point and range query acceleration using combined composite inverted bitmap indexes.
+  - Virtual composite column naming and tokenization using exact `"identity"` tokenization to preserve multi-column key terms (`val1\0val2`).
+  - Fluent table API `table.add_composite_index(name, columns)` and filter rewriting.
+  - Added end-to-end integration tests in `tests/test_composite_index.rs`.
+- **Apache Polaris & Lakekeeper Iceberg REST Catalog OAuth2 Client Credentials**:
+  - Implemented standard `/v1/oauth/tokens` client credentials grant flow for Iceberg REST catalogs.
+  - Added token caching with automatic expiry tracking and refresh within 60-second window.
+  - Injected `Authorization: Bearer <token>` across all REST catalog requests (`load_table`, `create_table`, `commit_table`).
+- **Core Community TurboQuant™ (TQ4 / TQ8) Scalar Quantization**:
+  - Polar Quantization (PQ) and TurboQuant 4-bit / 8-bit quantized HNSW vector index algorithms integrated into open-source core engine.
+- **Production Hardening & Correctness Enhancements**:
+  - **Vector Metric Propagation**: Dynamic metric parsing (`VectorMetric::from_str`) and propagation from `IndexAlgorithm` (`L2`, `Cosine`, `InnerProduct`, `L1`, `Hamming`, `Jaccard`) through index construction and Puffin/Parquet serialization.
+  - **Global KNN Ordering**: Refactored `merge_and_rerank_vector_results` to guarantee monotonic ascending distance order across all returned batches without unordered `HashMap` bucketing.
+  - **Metric Parity Test Suite**: Added `tests/test_vector_metrics_parity.rs` establishing 100% nearest-neighbor accuracy against exact brute force ground truth across all 6 metrics.
+  - **Strict Query Execution Semantics**: Segment vector search failures now fail the query immediately with actionable diagnostics rather than silently omitting rows.
+  - **Zero-Warning Standard**: Eliminated diagnostic `println!` statements in favor of structured `tracing::debug!`, resolved all clippy compiler warnings with `#![deny(warnings)]`.
+
+---
+
+## [0.6.0] - 2026-09-09
+
+### Added
+- **Multi-Protocol Gateway Ecosystem**:
+  - **`hyperstreamdb-search` Service (`hypersearch` binary)**: Dual Elasticsearch 7.10 (Port 9200) and Qdrant (Port 6333) compatible REST APIs over HyperStreamDB tables.
+    - Elasticsearch 7.10 API: Full document CRUD (`POST /{index}/_doc`), hybrid search (`POST /{index}/_search` with BM25 + HNSW kNN + Reciprocal Rank Fusion), index management (`PUT /{index}`, `POST /{index}/_refresh`), and cluster health (`GET /_cluster/health`).
+    - Qdrant REST API: Collection management (`/collections/{name}`), point upsert/retrieval (`/collections/{name}/points`), and vector search (`/collections/{name}/points/search`).
+    - Prometheus metrics exporter on `/metrics` (Port 9090).
+  - **`hyperstreamdb-flight` Gateway Service**: Native Arrow Flight SQL gRPC gateway (Port 50051) enabling zero-copy analytics for DuckDB, Polars, Apache Spark, and JDBC/ODBC BI tools via standard Flight SQL/ADBC.
+- **Multi-Flavor GPU Acceleration & Hardware Auto-Detection**:
+  - Optional GPU acceleration exposed across `hyperstreamdb-search` and `hyperstreamdb-flight` via `cuda`, `wgpu`, `rocm`, `intel`, and `all-gpu` feature flags.
+  - Runtime device selection via `HYPERSEARCH_DEVICE=auto|cuda[:N]|rocm[:N]|intel[:N]|mps|cpu`.
+  - Active compute backend (`compute` block) exposed in `GET /` cluster info and `GET /_cluster/stats`.
+  - `docker/Dockerfile.gpu`: Multi-flavor GPU container image with CUDA 12 runtime, NVRTC JIT compilation, and Vulkan/Mesa drivers for AMD Radeon and Intel Arc.
+  - `docker/docker-compose.gpu.yml`: Compose GPU override with hardware reservations and device pass-through.
+- **Iceberg Compaction Resilience & Index Recovery**:
+  - `Table::recover_indexes_async(&self)` / `recover_indexes(&self)`: Re-indexes data files that are missing overlay index sidecars, recovering fast vector (HNSW) and keyword (BM25) search after external Iceberg tools (Spark `rewriteDataFiles`, Trino `OPTIMIZE`, PyIceberg) compact table data files.
+- **Docker Container Infrastructure**:
+  - `hyperstreamdb/quickstart:latest`: Single all-in-one developer container running ES 7.10 (9200), Qdrant (6333), and Flight SQL (50051).
+  - `hyperstreamdb/search:latest`: Standalone production search microservice.
+  - `hyperstreamdb/flight:latest`: Standalone production Arrow Flight SQL microservice.
+  - `docker/docker-compose.quickstart.yml`: Single-command full stack with MinIO (S3), Project Nessie catalog, and HyperStreamDB.
+  - `docker-compose.production.yml`: Production multi-container configuration with health checks and resource limits.
+- Okapi BM25 keyword scoring (tunable `k1`/`b`) with an English analyzer in the core engine; public `keyword_search_index` API.
+- Smart hybrid trigger fusing keyword (BM25) and vector (HNSW) results via reciprocal rank fusion (RRF, k=60).
+- Background segment index builds with `wait_for_background_tasks_async` for deterministic refresh semantics.
+
 ## [0.5.3] - 2026-06-21
 
 ### Added

hyperstreamdb-0.7.1/CODEBASE_METRICS.md

@@ -0,0 +1,210 @@
+# HyperStreamDB Codebase Line Count & Test Analysis
+
+*Generated: September 10, 2026*
+
+---
+
+## 1. Executive Summary
+
+| Metric | Count | Description |
+|:---|---:|:---|
+| **Total Source Lines of Code (SLOC)** | **73,185** | Executable lines of code across all source languages (excluding blanks & comments) |
+| **Total Physical Lines** | **96,236** | Complete line count in source files (includes 10,709 comments & 12,342 blank lines) |
+| **Total Tracked Source Files** | **433** | Source code files tracked in git (Rust, Python, Java, Scala, Shell, SQL, GPU shaders, Web) |
+| **Pure Production Code** | **48,197 SLOC** | Production database engine, connectors, APIs, and shaders (excluding all tests) |
+| **Total Test Code** | **20,652 SLOC** | All test code across integration suites, connector tests, and inline unit tests |
+| **Test-to-Production Ratio** | **1 : 2.33** | Test code comprises **~30.0%** of all executable code in the repository |
+| **Configuration & Manifests** | **1,540 SLOC** | TOML and YAML configurations (Cargo manifests, CI/CD workflows, Docker Compose) |
+
+---
+
+## 2. Breakdown by Programming Language
+
+| Language | Files | SLOC (Code) | Comments & Docs | Blank Lines | Total Lines | % of Code |
+|:---|---:|---:|---:|---:|---:|---:|
+| **Rust** (`.rs`) | 232 | **57,448** | 7,084 | 8,024 | 72,556 | 78.50% |
+| **Python** (`.py`) | 118 | **11,295** | 2,954 | 3,337 | 17,586 | 15.43% |
+| **Shell** (`.sh`) | 17 | **862** | 179 | 197 | 1,238 | 1.18% |
+| **CSS** (`.css`) | 1 | **668** | 15 | 110 | 793 | 0.91% |
+| **Java** (`.java`) | 11 | **615** | 56 | 162 | 833 | 0.84% |
+| **SQL** (`.sql`) | 8 | **557** | 97 | 125 | 779 | 0.76% |
+| **Scala** (`.scala`) | 18 | **541** | 64 | 152 | 757 | 0.74% |
+| **HTML** (`.html`) | 1 | **350** | 0 | 15 | 365 | 0.48% |
+| **Dockerfile** | 4 | **248** | 72 | 73 | 393 | 0.34% |
+| **CUDA** (`.cu`) | 8 | **235** | 135 | 64 | 434 | 0.32% |
+| **Metal** (`.metal`) | 7 | **155** | 34 | 41 | 230 | 0.21% |
+| **OpenCL** (`.cl`) | 7 | **134** | 13 | 31 | 178 | 0.18% |
+| **WGSL** (`.wgsl`) | 1 | **77** | 6 | 11 | 94 | 0.11% |
+| **TOTAL (Source Code)** | **433** | **73,185** | **10,709** | **12,342** | **96,236** | **100.0%** |
+| *YAML / TOML Manifests* | *29* | *1,540* | *166* | *169* | *1,875* | — |
+
+---
+
+## 3. Breakdown by Subsystem & Component
+
+| Subsystem / Component | Files | SLOC (Code) | Comments | Blank | Total Lines | Primary Stack |
+|:---|---:|---:|---:|---:|---:|:---|
+| **Core Database Engine** (`src/`) | 169 | **45,356** | 5,748 | 6,193 | 57,297 | Rust, CUDA, Metal, OpenCL, WGSL |
+| **Search Engine** (`hyperstreamdb-search/`) | 17 | **5,657** | 575 | 626 | 6,858 | Rust |
+| **Arrow Flight Server** (`hyperstreamdb-flight/`) | 5 | **583** | 16 | 84 | 683 | Rust, Python |
+| **Trino Connector** (`trino-hyperstream/`) | 11 | **615** | 56 | 162 | 833 | Java |
+| **Spark Connector** (`spark-hyperstream/`) | 18 | **541** | 64 | 152 | 757 | Scala |
+| **Python SDK & Client** (`python/`) | 2 | **506** | 251 | 142 | 899 | Python |
+| **dbt Adapter** (`dbt-hyperstreamdb/`) | 13 | **444** | 29 | 105 | 578 | Python, SQL |
+| **Enterprise Features** (`hyperstreamdb-enterprise/`) | 2 | **8** | 7 | 4 | 19 | Rust |
+| **↳ Subtotal: Production Code** | **237** | **53,710** | **6,746** | **7,468** | **67,924** | |
+| **Dedicated Test Suite** (`tests/`) | 156 | **13,940** | 2,885 | 3,586 | 20,411 | Python, Rust, Shell |
+| **Benchmarks** (`benchmarks/`, `benches/`) | 8 | **1,830** | 272 | 410 | 2,512 | Python, Rust, Shell |
+| **Build, CI, Docker & Admin Scripts** | 19 | **1,509** | 317 | 362 | 2,188 | Shell, Python, Dockerfile |
+| **Examples & Demos** (`examples/`, `demo/`) | 10 | **1,127** | 452 | 378 | 1,957 | Python, Rust, SQL |
+| **Marketing Site & Docs Scripts** | 3 | **1,069** | 37 | 138 | 1,244 | HTML, CSS, Python |
+
+---
+
+## 4. Core Engine Architecture Breakdown (`src/`)
+
+Within the core database engine (`src/`), the 45,356 lines of code are distributed as follows:
+
+| Engine Subsystem | Files | SLOC (Code) | Total Lines | Description & Key Responsibilities |
+|:---|---:|---:|---:|:---|
+| **Vector & Inverted Indexing** (`src/core/index/`) | 47 | **9,727** | 12,136 | HNSW graph, IVF-PQ quantization, SIMD/GPU acceleration, Bloom filters |
+| **SQL & DataFusion Engine** (`src/core/sql/`) | 32 | **7,500** | 9,380 | pgvector operators, custom dialects, physical expression pushdown |
+| **Table Engine & Concurrency** (`src/core/table/`) | 10 | **6,662** | 8,387 | Table locks, segment lifecycles, streaming ingestion buffers |
+| **Iceberg Integration** (`src/core/iceberg/`) | 9 | **2,742** | 3,111 | REST catalog client, v2/v3 spec, manifest parsing & commit protocol |
+| **Parquet Readers** (`src/core/reader/`) | 4 | **2,721** | 3,240 | Vectorized Parquet reader, column projection, filter pushdown |
+| **PyO3 Python Engine Bindings** (`src/python/`) | 15 | **2,446** | 2,958 | Python C-API bindings for Arrow tables, sessions, and catalog |
+| **Manifests & Metadata** (`src/core/manifest/`, `metadata.rs`) | 8 | **2,278** | 2,726 | Avro metadata serialization, partition summary records |
+| **Execution Planner & Query Engine** (`planner.rs`, `query.rs`) | 3 | **2,358** | 3,167 | Logical/physical plan optimization, hybrid vector/scalar scoring |
+| **Catalog Implementations** (`src/core/catalog/`, `nessie.rs`) | 9 | **1,789** | 2,185 | Hive metastore, Nessie, Memory, and File system catalogs |
+| **CLI Binaries & Tools** (`src/bin/`) | 6 | **1,401** | 1,658 | REST gateway, admin CLI, layered index verification tools |
+| **WAL, Compaction, Cache & Storage** | 6 | **2,044** | 2,746 | Write-Ahead Log, segment compaction, block cache, cloud storage |
+| **Telemetry & Error Handling** | 5 | **426** | 648 | OpenTelemetry metrics, distributed tracing, error types |
+
+---
+
+## 5. Comprehensive Test Suite Breakdown
+
+Across all testing layers, HyperStreamDB contains **20,652 SLOC of test code**.
+
+### A. Test Code Distribution
+
+| Test Layer | Files | SLOC (Code) | Primary Languages | Focus |
+|:---|---:|---:|:---|:---|
+| **Dedicated Test Suite** (`tests/`) | 155 | **13,934** | Python (6.9k), Rust (6.5k), Shell (566) | E2E, integration, property & chaos tests |
+| **Inline Unit Tests** (`#[cfg(test)]`) | 40 | **5,513** | Rust native | Embedded unit tests in `src/` and sub-crates |
+| **Sub-crate & Connector Test Suites** | 11 | **1,205** | Python, Java, Scala, Rust, SQL | Trino, Spark, Search, Flight, and dbt tests |
+| **TOTAL TEST CODE** | **206** | **20,652** | | |
+
+---
+
+### B. Dedicated Test Suite Breakdown by Domain (`tests/`)
+
+The 155 test files inside the dedicated `tests/` directory target specific functionality:
+
+#### 1. Vector Search & Hardware Acceleration (34 files | 4,158 SLOC)
+- **Multi-Hardware Parity**: Cross-platform verification ensuring numerical and recall parity across backends:
+  - CUDA: `test_hyperstream_cuda.py`
+  - Apple Silicon Metal/MPS: `test_mps_gpu.py`
+  - Intel: `test_hyperstream_intel.py`
+  - AMD ROCm: `test_hyperstream_rocm.py`
+  - CPU SIMD: `test_hardware_parity.rs`
+- **Distance Metric Kernels**: Correctness benchmarks and edge cases for L1, L2, Inner Product, Cosine, Hamming, and Jaccard distances.
+- **Index Algorithms**: HNSW recall verification, layered index recovery, TurboQuant quantization, and GPU KMeans clustering.
+
+#### 2. Data Ingestion, Schema Evolution & Partitioning (21 files | 2,105 SLOC)
+- Partition transform validations (identity, bucket, truncate, hour/day transforms).
+- Schema evolution compatibility across Iceberg metadata snapshots.
+- Primary key acceleration, composite indexes, and write-buffer deduplication.
+
+#### 3. Iceberg Format, REST Catalogs & Metadata (25 files | 1,876 SLOC)
+- Iceberg v2/v3 specification compliance.
+- Merge-on-Read (MOR) read and write workflows with equality/positional deletes.
+- Puffin format index blob creation and attachment.
+- REST catalog, AWS Glue, and Hive Metastore transaction commit verification.
+
+#### 4. Performance Verification & Comparative Benchmarks (7 files | 981 SLOC)
+- Direct comparison suites vs. Qdrant (`test_vs_qdrant.py`, `test_qdrant_direct_comparison.py`).
+- Scan latency baselines, cold-read latency benchmarks, and parallel search scaling.
+
+#### 5. Concurrency, WAL, Durability & Chaos (10 files | 837 SLOC)
+- Multi-threaded writer contention and optimistic concurrency control.
+- Chaos testing: simulated power losses during WAL flush, torn writes, crash recovery, and stale file cleanup.
+
+#### 6. SQL Engine & DataFusion Pushdown (7 files | 492 SLOC)
+- pgvector SQL operator semantics (`<->`, `<#>`, `<=>`).
+- SQL UDF GPU acceleration and BM25 full-text pushdown.
+- Complex aggregations, filtering, and ordering.
+
+#### 7. Python PyO3 SDK & Integrations (15 files | 1,300 SLOC)
+- Fluent client API verification, PyArrow schema consistency, docstrings, and export routines.
+- Full real-world ingestion flows using NYC Taxi and Wikipedia datasets.
+
+---
+
+### C. Inline Unit Tests Breakdown (`#[cfg(test)]` in `src/`)
+
+Unit tests written directly alongside production Rust modules total **5,513 SLOC** across 40 files:
+
+| Module / Component | Test Files | Unit Test SLOC | Production SLOC | Major Features Tested |
+|:---|---:|---:|---:|:---|
+| **SQL Engine** (`src/core/sql/`) | 9 | **1,840** | 1,827 | Literal parsers (1,269 lines), vector operators (257 lines), function evaluation |
+| **Search Engine** (`hyperstreamdb-search/`) | 7 | **1,081** | 2,307 | Query inference (387 lines), search handlers (312 lines), index caching, state |
+| **Vector Index** (`src/core/index/`) | 11 | **903** | 4,655 | HNSW IO serialization (220 lines), distance calculations, quantization |
+| **Execution Planner & Query Engine** | 2 | **305** | 1,546 | Plan generation and scoring in `query.rs` (302 lines) |
+| **Compaction & Table Merge** | 2 | **436** | 603 | Segment compaction (205 lines), merge conflict resolution (231 lines) |
+| **WAL, Storage & Cache** | 3 | **474** | 634 | WAL replay (174 lines), block cache eviction (153 lines), storage backends (147 lines) |
+| **Table, Reader & Segments** | 3 | **304** | 3,540 | Segment state machines, Parquet reader filtering, table lock semantics |
+| **Catalog & Locking** | 3 | **170** | 607 | In-memory catalog commit logic, concurrency primitives |
+| **TOTAL INLINE UNIT TESTS** | **40** | **5,513** | — | |
+
+---
+
+### D. Sub-crate & Connector Dedicated Tests
+
+| Project / Sub-crate | File | SLOC | Total Lines | Purpose |
+|:---|:---|---:|---:|:---|
+| `hyperstreamdb-search` | `tests/test_search_api.py` | **589** | 775 | Python HTTP client testing hybrid search endpoints |
+| `trino-hyperstream` | `src/test/.../TrinoConnectorTest.java` | **225** | 306 | Trino SPI split generation and predicate pushdown |
+| `hyperstreamdb-flight` | `tests/test_flight_service.rs` | **113** | 134 | Native Rust Arrow Flight DoGet / DoPut integration |
+| `hyperstreamdb-flight` | `tests/test_flight_client.py` | **26** | 40 | Python pyarrow.flight client verification |
+| `scripts/manual_tests` | `test_multi_cloud.py` | **132** | 212 | S3, GCS, and Azure multi-cloud integration tests |
+| `spark-hyperstream` | `src/test/.../SparkMergeIntegrationTest.scala` | **76** | 101 | Spark DataFrame write/merge integration |
+| `dbt-hyperstreamdb` | `test_project/models/*.sql` | **38** | 43 | dbt incremental, macro, and vector models |
+| Standalone | `test_bloom.rs` | **6** | 6 | Bloom filter false-positive verification |
+| **TOTAL** | **11 files** | **1,205** | **1,617** | |
+
+---
+
+## 6. Hardware Acceleration Shaders & Kernels
+
+The core engine includes dedicated GPU and compute shaders across 4 vendor targets:
+
+| Target Platform | File Extension | Files | SLOC (Code) | Total Lines | Implementations |
+|:---|:---|---:|---:|---:|:---|
+| **NVIDIA CUDA** | `.cu` | 8 | **235** | 434 | L2, Inner Product, Cosine, Hamming distance kernels |
+| **Apple Metal** | `.metal` | 7 | **155** | 230 | MSL shaders for Apple Silicon MPS execution |
+| **Khronos OpenCL** | `.cl` | 7 | **134** | 178 | Cross-vendor portable GPU compute kernels |
+| **WebGPU / WGPU** | `.wgsl` | 1 | **77** | 94 | Cross-platform WGPU compute shader |
+| **TOTAL GPU KERNELS** | | **23** | **601** | **936** | |
+
+---
+
+## 7. Connectors & External Integrations
+
+| Ecosystem Connector | Primary Language | Source Files | SLOC (Code) | Test SLOC | Target System |
+|:---|:---|---:|---:|---:|:---|
+| **Trino Connector** | Java | 11 | **615** | 225 | Trino / Presto distributed SQL engine |
+| **Spark Connector** | Scala | 18 | **541** | 76 | Apache Spark 3.x DataSourceV2 connector |
+| **Python Client / SDK** | Python | 2 | **506** | — | High-level user-facing Python SDK |
+| **dbt Adapter** | Python + SQL | 13 | **444** | 38 | dbt plugin for analytics engineering |
+| **Arrow Flight** | Rust + Python | 5 | **583** | 139 | High-performance Arrow Flight RPC service |
+| **TOTAL CONNECTORS** | | **49** | **2,689** | **478** | |
+
+---
+
+## 8. Methodology & Counting Standards
+
+- **SLOC (Source Lines of Code)**: Measures actual executable code statements. Blank lines and comments (both single-line and multi-line blocks) are excluded.
+- **Comments & Documentation**: Includes docstrings (`"""`), Rust doc comments (`///`, `//!`), and standard comments (`//`, `#`, `/* ... */`, `--`).
+- **Inline Tests**: Rust unit test blocks demarcated by `#[cfg(test)] mod tests { ... }` within production source files were parsed, isolated, and categorized under test metrics rather than production metrics to present an accurate architectural distribution.
+- **Excluded Non-Code Artifacts**: Build caches (`docs/build/`), test runner state (`.hypothesis/`), and Jupyter checkpoints (`.ipynb_checkpoints/`) were excluded from source metrics.