hyperstreamdb 0.5.3__tar.gz → 0.5.4__tar.gz

hyperstreamdb-0.5.4/.cargo/audit.toml

@@ -0,0 +1,11 @@
+[advisories]
+ignore = [
+    "RUSTSEC-2026-0104",
+    "RUSTSEC-2026-0099",
+    "RUSTSEC-2026-0098",
+    "RUSTSEC-2026-0097",
+    "RUSTSEC-2024-0437",
+    "RUSTSEC-2026-0177",
+    "RUSTSEC-2026-0176",
+    "RUSTSEC-2023-0071"
+]

hyperstreamdb-0.5.4/ADMIN_CLI.md

@@ -0,0 +1,59 @@
+# HyperStreamDB Admin CLI
+
+`hyperstreamdb-admin` is the official administrative tooling for HyperStreamDB. It provides cluster operators with safe, easy-to-use commands to manage storage, optimize performance, and clean up historical data.
+
+## Installation / Building
+If you are compiling from source, the admin CLI is built alongside the main engine:
+```bash
+cargo build --release --bin hyperstreamdb-admin
+```
+
+The compiled binary will be located at `target/release/hyperstreamdb-admin`.
+
+## Commands
+
+### 1. `compact`
+Over time, continuous ingestion can lead to fragmentation—creating hundreds or thousands of small data files (often called the "small files problem"). The `compact` command merges these small files into larger, optimized files. This significantly improves query performance by reducing metadata overhead and maximizing the efficiency of vectorized scans.
+
+**Usage:**
+```bash
+hyperstreamdb-admin compact --uri <TABLE_URI> [OPTIONS]
+```
+
+**Options:**
+- `--uri`, `-u`: **(Required)** The storage URI of the table you want to compact. (e.g., `file:///tmp/hdb`, `s3://my-bucket/table`)
+- `--target-file-size`: **(Optional)** The ideal size for the compacted data files. The CLI accepts human-readable string values like `64MB`, `1GB`, `512K`. The default is `64MB`.
+
+**Example:**
+```bash
+hyperstreamdb-admin compact --uri "s3://production-data/events_table" --target-file-size 128MB
+```
+
+### 2. `vacuum`
+HyperStreamDB relies on Multi-Version Concurrency Control (MVCC) and copy-on-write semantics. As data is updated, deleted, or compacted, old files are unlinked from the active manifest but remain on disk to allow time-travel queries and safe rollback.
+
+The `vacuum` command permanently deletes these unreferenced historical files to reclaim physical storage space. 
+
+> [!WARNING]
+> Running vacuum is irreversible. You will not be able to time-travel to versions older than the retention limit once vacuum completes.
+
+**Usage:**
+```bash
+hyperstreamdb-admin vacuum --uri <TABLE_URI> [OPTIONS]
+```
+
+**Options:**
+- `--uri`, `-u`: **(Required)** The storage URI of the table you want to vacuum.
+- `--retain-versions`: **(Optional)** The number of historical manifest versions to keep. Any unreferenced data file not tied to the kept versions will be permanently deleted. Default is `10`.
+
+**Example:**
+```bash
+hyperstreamdb-admin vacuum --uri "s3://production-data/events_table" --retain-versions 5
+```
+
+## E2E Testing
+The CLI is strictly tested for regressions. You can run the administrative automated bash test locally using:
+```bash
+./tests/verify_admin_cli.sh
+```
+This test spins up a mock dataset, runs a compaction pass, and successfully vacuums the resulting orphaned files.

{hyperstreamdb-0.5.3 → hyperstreamdb-0.5.4}/CONTRIBUTING.md

@@ -12,7 +12,7 @@ Before contributing, please read [STEERING.md](STEERING.md) for our development
 
 ### Prerequisites
 
-- **Rust** (stable, 1.75+) — install via [rustup](https://rustup.rs/)
+- **Rust** (stable, 1.80+) — install via [rustup](https://rustup.rs/)
 - **Python** (3.10–3.14) — for binding development and testing
 - **Docker** — for integration tests against MinIO and Nessie
 - **Cargo** and **maturin** — for building Python wheels

{hyperstreamdb-0.5.3 → hyperstreamdb-0.5.4}/Cargo.lock

@@ -1305,9 +1305,9 @@ checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724"
 
 [[package]]
 name = "chacha20"
-version = "0.10.0"
+version = "0.10.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6f8d983286843e49675a4b7a2d174efe136dc93a18d69130dd18198a6c167601"
+checksum = "d524456ba66e72eb8b115ff89e01e497f8e6d11d78b70b1aa13c0fbd97540a81"
 dependencies = [
  "cfg-if",
  "cpufeatures 0.3.0",
@@ -2709,6 +2709,17 @@ dependencies = [
  "pin-project-lite",
 ]
 
+[[package]]
+name = "evmap"
+version = "11.0.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1b8874945f036109c72242964c1174cf99434e30cfa45bf45fedc983f50046f8"
+dependencies = [
+ "hashbag",
+ "left-right",
+ "smallvec",
+]
+
 [[package]]
 name = "fastrand"
 version = "2.4.1"
@@ -2947,6 +2958,21 @@ dependencies = [
  "slab",
 ]
 
+[[package]]
+name = "generator"
+version = "0.8.9"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b3b854b0e584ead1a33f18b2fcad7cf7be18b3875c78816b753639aa501513ae"
+dependencies = [
+ "cc",
+ "cfg-if",
+ "libc",
+ "log",
+ "rustversion",
+ "windows-link",
+ "windows-result",
+]
+
 [[package]]
 name = "generic-array"
 version = "0.14.7"
@@ -3147,6 +3173,12 @@ dependencies = [
  "zerocopy",
 ]
 
+[[package]]
+name = "hashbag"
+version = "0.1.13"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7040a10f52cba493ddb09926e15d10a9d8a28043708a405931fe4c6f19fac064"
+
 [[package]]
 name = "hashbrown"
 version = "0.12.3"
@@ -3484,7 +3516,7 @@ dependencies = [
 
 [[package]]
 name = "hyperstreamdb"
-version = "0.5.3"
+version = "0.5.4"
 dependencies = [
  "ahash",
  "anyhow",
@@ -3524,7 +3556,8 @@ dependencies = [
  "lazy_static",
  "log",
  "metal 0.27.0",
- "metrics",
+ "metrics 0.23.1",
+ "metrics-exporter-prometheus",
  "moka",
  "murmur3",
  "num_cpus",
@@ -3864,9 +3897,9 @@ dependencies = [
 
 [[package]]
 name = "js-sys"
-version = "0.3.102"
+version = "0.3.103"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "03d04c30968dffe80775bd4d7fb676131cd04a1fb46d2686dbffbaec2d9dfd31"
+checksum = "53b44bfcdb3f8d5837a46dae1ca9660a837176eee74a28b229bc626816589102"
 dependencies = [
  "cfg-if",
  "futures-util",
@@ -3899,6 +3932,17 @@ dependencies = [
  "spin",
 ]
 
+[[package]]
+name = "left-right"
+version = "0.11.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0f0c21e4c8ff95f487fb34e6f9182875f42c84cef966d29216bf115d9bba835a"
+dependencies = [
+ "crossbeam-utils",
+ "loom",
+ "slab",
+]
+
 [[package]]
 name = "lexical-core"
 version = "1.0.6"
@@ -4111,6 +4155,19 @@ version = "0.4.33"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad"
 
+[[package]]
+name = "loom"
+version = "0.7.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "419e0dc8046cb947daa77eb95ae174acfbddb7673b4151f56d1eed8e93fbfaca"
+dependencies = [
+ "cfg-if",
+ "generator",
+ "scoped-tls",
+ "tracing",
+ "tracing-subscriber",
+]
+
 [[package]]
 name = "lru-slab"
 version = "0.1.2"
@@ -4238,6 +4295,56 @@ dependencies = [
  "portable-atomic",
 ]
 
+[[package]]
+name = "metrics"
+version = "0.24.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "89550ee9f79e88fef3119de263694973a8adb26c21d75322164fb8c493039fe2"
+dependencies = [
+ "portable-atomic",
+ "rapidhash",
+]
+
+[[package]]
+name = "metrics-exporter-prometheus"
+version = "0.18.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1db0d8f1fc9e62caebd0319e11eaec5822b0186c171568f0480b46a0137f9108"
+dependencies = [
+ "base64 0.22.1",
+ "evmap",
+ "http-body-util",
+ "hyper 1.10.1",
+ "hyper-rustls 0.27.9",
+ "hyper-util",
+ "indexmap 2.14.0",
+ "ipnet",
+ "metrics 0.24.6",
+ "metrics-util",
+ "quanta",
+ "rustls 0.23.41",
+ "thiserror 2.0.18",
+ "tokio",
+ "tracing",
+]
+
+[[package]]
+name = "metrics-util"
+version = "0.20.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "96f8722f8562635f92f8ed992f26df0532266eb03d5202607c20c0d7e9745e13"
+dependencies = [
+ "crossbeam-epoch",
+ "crossbeam-utils",
+ "hashbrown 0.16.1",
+ "metrics 0.24.6",
+ "quanta",
+ "rand 0.9.4",
+ "rand_xoshiro",
+ "rapidhash",
+ "sketches-ddsketch",
+]
+
 [[package]]
 name = "mime"
 version = "0.3.17"
@@ -5230,6 +5337,21 @@ version = "0.2.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "5a651516ddc9168ebd67b24afd085a718be02f8858fe406591b013d101ce2f40"
 
+[[package]]
+name = "quanta"
+version = "0.12.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f3ab5a9d756f0d97bdc89019bd2e4ea098cf9cde50ee7564dde6b81ccc8f06c7"
+dependencies = [
+ "crossbeam-utils",
+ "libc",
+ "once_cell",
+ "raw-cpuid",
+ "wasi 0.11.1+wasi-snapshot-preview1",
+ "web-sys",
+ "winapi",
+]
+
 [[package]]
 name = "quick-error"
 version = "1.2.3"
@@ -5489,12 +5611,39 @@ dependencies = [
  "rand_core 0.9.5",
 ]
 
+[[package]]
+name = "rand_xoshiro"
+version = "0.7.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f703f4665700daf5512dcca5f43afa6af89f09db47fb56be587f80636bda2d41"
+dependencies = [
+ "rand_core 0.9.5",
+]
+
 [[package]]
 name = "range-alloc"
 version = "0.1.5"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "ca45419789ae5a7899559e9512e58ca889e41f04f1f2445e9f4b290ceccd1d08"
 
+[[package]]
+name = "rapidhash"
+version = "4.4.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b5e48930979c155e2f33aa36ab3119b5ee81332beb6482199a8ecd6029b80b59"
+dependencies = [
+ "rustversion",
+]
+
+[[package]]
+name = "raw-cpuid"
+version = "11.6.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "498cd0dc59d73224351ee52a95fee0f1a617a2eae0e7d9d720cc622c73a54186"
+dependencies = [
+ "bitflags 2.13.0",
+]
+
 [[package]]
 name = "raw-window-handle"
 version = "0.6.2"
@@ -5947,6 +6096,12 @@ dependencies = [
  "windows-sys 0.61.2",
 ]
 
+[[package]]
+name = "scoped-tls"
+version = "1.0.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e1cf6437eb19a8f4a6cc0f7dca544973b0b78843adbfeb3683d1a94a0024a294"
+
 [[package]]
 name = "scopeguard"
 version = "1.2.0"
@@ -6169,6 +6324,12 @@ version = "1.0.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "8ee5873ec9cce0195efcb7a4e9507a04cd49aec9c83d0389df45b1ef7ba2e649"
 
+[[package]]
+name = "sketches-ddsketch"
+version = "0.3.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0c6f73aeb92d671e0cc4dca167e59b2deb6387c375391bc99ee743f326994a2b"
+
 [[package]]
 name = "skiplist"
 version = "0.3.0"
@@ -7440,9 +7601,9 @@ checksum = "b8dad83b4f25e74f184f64c43b150b91efe7647395b42289f38e50566d82855b"
 
 [[package]]
 name = "wasm-bindgen"
-version = "0.2.125"
+version = "0.2.126"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8ddb3f79143bced6de84270411622a2699cee572fc0875aeaf1e7867cf9fca1a"
+checksum = "4b067c0c11094aef6b7a801c1e34a26affafdf3d051dba08456b868789aaf9a4"
 dependencies = [
  "cfg-if",
  "once_cell",
@@ -7453,9 +7614,9 @@ dependencies = [
 
 [[package]]
 name = "wasm-bindgen-futures"
-version = "0.4.75"
+version = "0.4.76"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "503b14d284f2c8dac03b819967e155ea753f573586193b2b2c95990cb5d69280"
+checksum = "c62df1340f32221cb9c54d6a27b030e3dba64361d4a95bed55f9aacb44da291d"
 dependencies = [
  "js-sys",
  "wasm-bindgen",
@@ -7463,9 +7624,9 @@ dependencies = [
 
 [[package]]
 name = "wasm-bindgen-macro"
-version = "0.2.125"
+version = "0.2.126"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4e21a184b13fb19e157296e2c46056aec9092264fab83e4ba59e68c61b323c3d"
+checksum = "167ce5e579f6bcf889c4f7175a8a5a585de84e8ff93976ce393efa5f2837aab1"
 dependencies = [
  "quote",
  "wasm-bindgen-macro-support",
@@ -7473,9 +7634,9 @@ dependencies = [
 
 [[package]]
 name = "wasm-bindgen-macro-support"
-version = "0.2.125"
+version = "0.2.126"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fecefd9c35bd935a20fc3fc344b5f29138961e4f47fb03297d88f2587afb5ebd"
+checksum = "f3997c7839262f4ef12cf90b818d6340c18e80f263f1a94bf157d0ec4420380e"
 dependencies = [
  "bumpalo",
  "proc-macro2",
@@ -7486,9 +7647,9 @@ dependencies = [
 
 [[package]]
 name = "wasm-bindgen-shared"
-version = "0.2.125"
+version = "0.2.126"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "23939e44bb9a5d7576fa2b563dc2e136628f1224e88a8deed09e04858b77871f"
+checksum = "dc1b4cb0cc549fcf58d7dfc081778139b3d283a081644e833e84682ad71cea24"
 dependencies = [
  "unicode-ident",
 ]
@@ -7508,9 +7669,9 @@ dependencies = [
 
 [[package]]
 name = "web-sys"
-version = "0.3.102"
+version = "0.3.103"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a6430a72df5eb332242960fe84b3002a241163998241eb596d4f739b9757061d"
+checksum = "8622dcb61c0bcc9fffa6938bed81210af2da9a7e4a1a834b2e37a59b6dfb6141"
 dependencies = [
  "js-sys",
  "wasm-bindgen",

{hyperstreamdb-0.5.3 → hyperstreamdb-0.5.4}/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "hyperstreamdb"
-version = "0.5.3"
+version = "0.5.4"
 edition = "2021"
 license = "MIT AND Apache-2.0"
 description = "HyperStreamDB - Serverless Index-Streaming Database with Overlay Indexing and Vector Search"
@@ -35,11 +35,14 @@ enterprise = []
 python = ["dep:pyo3", "pyo3/extension-module", "dep:numpy", "dep:pythonize", "wgpu", "pollster", "cuda"]
 java = ["dep:jni"]
 proptest = []
+observability = ["dep:metrics-exporter-prometheus"]
 
 # GPU Acceleration Features
 wgpu = ["dep:wgpu"]
 pollster = ["dep:pollster"]
 cuda = ["dep:cudarc"]
+metrics-exporter-prometheus = ["dep:metrics-exporter-prometheus"]
+
 
 [dependencies]
 # core columnar memory format
@@ -134,6 +137,7 @@ regex = "1.10"
 sha2 = "0.10"
 wgpu = { version = "0.20.0", optional = true }
 pollster = { version = "0.4.0", optional = true }
+metrics-exporter-prometheus = { version = "0.18.3", optional = true }
 
 [target.'cfg(not(target_os = "macos"))'.dependencies]
 cudarc = { version = "0.13.0", features = ["driver", "nvrtc", "cuda-12000"], optional = true }
@@ -199,3 +203,5 @@ strip = "debuginfo"         # Keep symbols for crash backtraces, strip DWARF fro
 inherits = "release"
 lto = "fat"                 # Cross-crate LTO for max performance (slower builds)
 codegen-units = 1           # Single unit for LTO effectiveness
+[profile.test]
+debug = 0

hyperstreamdb-0.5.4/ELASTICSEARCH_INTEGRATION_PLAN.md

@@ -0,0 +1,121 @@
+# Implementation Plan: Elasticsearch-Like Search Engine Layer (Add-On Product)
+
+This document outlines the strategy for validating HyperStreamDB's production readiness and building an Elasticsearch-compatible REST API wrapper. 
+
+The REST API is implemented as a **separate, optional add-on package (`hyperstreamdb-search`)** in the Cargo workspace. This ensures the core library remains lightweight, and users who only need library-level access or FFI bindings do not pull in HTTP dependencies.
+
+---
+
+## Architectural Thesis: Cost-Efficiency & Serverless Over Low-Latency
+
+* **Not In-Memory**: Elasticsearch keeps all segments and caches aggressively pinned in RAM to achieve sub-millisecond query execution. This makes it operationally expensive and hard to run in serverless environments.
+* **Target Workload (Website / Doc Search)**: For website search, document indexing, and corporate knowledge bases, query response latencies of **50–200ms** are perfectly acceptable.
+* **Object Storage Native**: By designing for website search, we can load index files (`.hnsw` and `.idx`) and Parquet row groups **on-demand** from object storage (like S3/MinIO) or local disk cache. This enables an extremely cheap, serverless, scale-to-zero operational model.
+
+---
+
+## Step 1: Production Readiness & Quality Validation
+
+Before implementing new features, we must establish a production-readiness baseline. This ensures the engine's core is secure, stable, and conforms to standard best practices.
+
+### 1.1 Code Quality & Safety Checks
+- **Security Audit**: Ensure `cargo audit` runs successfully in CI/CD with zero unignored vulnerabilities.
+- **Static Analysis**: Enforce Clippy linting (`cargo clippy --all-targets --all-features`) with zero warnings or errors.
+- **Formatting Standards**: Verify that all Rust files are styled per `cargo fmt`.
+- **Code Test Coverage**: Ensure all Rust core unit and integration tests compile and run to completion (`cargo test --all-targets`).
+- **Python Compatibility**: Run the full Python test suite (`pytest`) to ensure zero regressions in bindings, catalogs, and table reads/writes.
+
+### 1.2 Documentation Readiness Check
+- **API Reference Compilation**: Verify that `cargo doc --no-deps` builds cleanly without warnings or broken intra-doc links.
+- **Getting Started Guide**: Ensure [GETTING_STARTED.md](GETTING_STARTED.md) is up-to-date with current APIs and dependency instructions.
+- **API Coverage**: Confirm all public catalog methods, index structures, and configuration keys are fully documented.
+
+---
+
+## Step 2: Lexical Processing & Search Enhancements (Crate: `hyperstreamdb`)
+
+To support Elasticsearch-like search capability, we implement base lexical processing helpers within the core library.
+
+### 2.1 Text Analyzers & Tokenizers
+- **New Module**: `src/core/index/analyzer.rs`
+- **Features**:
+  - Lowercase filter.
+  - Standard whitespace/punctuation tokenizer.
+  - Basic english stop-word filter.
+
+### 2.2 BM25 Relevance Scoring
+- **Implementation**: Calculate Okapi BM25 scores during inverted index lookup:
+  $$\text{Score}(D, Q) = \sum_{i=1}^{n} \text{IDF}(q_i) \cdot \frac{f(q_i, D) \cdot (k_1 + 1)}{f(q_i, D) + k_1 \cdot \left(1 - b + b \cdot \frac{|D|}{\text{avgdl}}\right)}$$
+- **Integration**: Integrate the scoring mechanism with the search planner to rank results.
+
+---
+
+## Step 3: The `hyperstreamdb-search` Add-On Crate
+
+Create a new package `hyperstreamdb-search` in the Cargo workspace root.
+
+### 3.1 Cargo Setup
+- **New Crate**: [hyperstreamdb-search/Cargo.toml]
+- **Dependencies**: `hyperstreamdb`, `axum`, `tower`, `tower-http`, `tokio`, `serde`, `serde_json`.
+
+### 3.2 Document Ingestion & Schema Evolution
+- **Endpoint**: `POST /<index_name>/_doc`
+- **Dynamic Schema-Mapping**: Parse arbitrary JSON documents, infer field datatypes, and dynamically evolve the target Iceberg schema (without rewriting existing Parquet data).
+- **Buffered Commits**: Route incoming documents to the active Memtable and Write-Ahead Log (WAL) to ensure low write latency.
+
+### 3.3 Search API
+- **Endpoint**: `POST /<index_name>/_search`
+- **Query Parser**: Parse JSON-based query DSL matching standard search criteria:
+  - `match` (lexical search using inverted index + BM25).
+  - `knn` (HNSW vector search).
+  - `filter` (roaring bitmap scalar pre-filtering).
+- **On-Demand Loading**: Fetch the HNSW and inverted indexes from S3/MinIO on-demand, caching them locally in temp storage for future requests.
+
+---
+
+## Step 4: Verification & Benchmarking
+
+### 4.1 Automated Validation
+- Implement a test suite in `hyperstreamdb-search/tests/test_search_api.py` that verifies:
+  - Document ingestion via HTTP `POST`.
+  - Dynamic table creation and schema evolution.
+  - Hybrid lexical + vector search queries.
+  
+### 4.2 Benchmark Analysis
+- Measure ingestion throughput (docs/sec) and search latency (p95/p99) against a baseline local Elasticsearch instance, targeting the 50–200ms latency envelope.
+
+---
+
+## Step 5: Positioning: Strengths & Weaknesses vs. Elasticsearch
+
+To effectively market this add-on, we must clearly define how it compares to Elasticsearch (ES) so users understand when to choose HyperStreamDB-Search.
+
+### 5.1 Strengths (Competitive Advantages)
+1. **Ultra-Low TCO (Total Cost of Ownership)**:
+   * *Elasticsearch*: Requires expensive instance groups with huge memory allocations (JVM heaps) and fast, hot SSD storage. 
+   * *HyperStreamDB*: Built for serverless object storage (S3/MinIO). When idle, it costs nothing. Index files (`.hnsw` and `.idx`) are fetched on-demand and cached locally.
+2. **Open Data Lakehouse Integration**:
+   * *Elasticsearch*: Uses a proprietary data format. Getting data out for analysis requires heavy ETLs or expensive scroll APIs.
+   * *HyperStreamDB*: Underpinned by Apache Iceberg and Parquet. Other tools (DuckDB, Trino, Spark) can query the exact same files directly in the data lake without moving data.
+3. **Dynamic Schema Evolution**:
+   * *Elasticsearch*: Changing mapping types or structures often requires creating a new index and running a resource-heavy `_reindex` job.
+   * *HyperStreamDB*: Inherits Iceberg's native schema evolution, allowing column additions, drops, and renaming instantly.
+4. **GPU-Accelerated Hybrid Search**:
+   * *Elasticsearch*: Vector search runs on standard JVM threads.
+   * *HyperStreamDB*: Native vector indexing with SIMD/GPU acceleration for low-cost, high-scale HNSW execution.
+
+### 5.2 Weaknesses (Where ES Wins & How to Position It)
+1. **Sub-Millisecond Query Latency**:
+   * *Elasticsearch*: Sub-5ms response times due to aggressive in-memory caching.
+   * *HyperStreamDB*: 50–200ms response times due to S3-native fetch overhead.
+   * *Positioning*: Position HyperStreamDB-Search for **website search, document catalogs, and log archives** where 50–200ms is imperceptible to users, but the 90% hosting cost reduction is highly compelling.
+2. **Ecosystem & Dashboarding**:
+   * *Elasticsearch*: Has mature Kibana integration for visualization and dashboarding.
+   * *HyperStreamDB*: No proprietary visualization frontend.
+   * *Leveraging Grafana, Prometheus & Apache Superset*:
+     - **Prometheus Metrics Endpoint**: The `hyperstreamdb-search` REST server will expose a `/metrics` endpoint (via the `prometheus` crate, already a dependency) providing operational telemetry: query latency histograms, ingestion throughput counters, index cache hit rates, and active connection gauges.
+     - **Grafana + Prometheus Stack**: Grafana natively scrapes Prometheus endpoints, giving operators real-time operational dashboards (query p95/p99, error rates, ingestion backpressure) out of the box with zero custom code.
+     - **Grafana via DuckDB / Trino (Data Dashboards)**: For data-level dashboards, Grafana connects to DuckDB and Trino. Since our tables are standard Iceberg, Grafana can query and visualize HyperStreamDB data with full SQL support.
+     - **Apache Superset**: The premier open-source BI tool for data lakes natively supports Trino, Spark, and DuckDB, offering a robust Kibana-like log viewing and dashboarding experience.
+     - **Elasticsearch API Compatibility**: By ensuring our REST API matches standard Elasticsearch query endpoints, users can configure **Grafana's built-in Elasticsearch datasource** to query HyperStreamDB directly, providing zero-friction dashboard reuse.
+