hyperstreamdb 0.4.0__tar.gz → 0.5.0__tar.gz

hyperstreamdb-0.5.0/.env.example

@@ -0,0 +1,51 @@
+# HyperStreamDB Environment Configuration
+# Copy this file to .env and fill in your credentials.
+# DO NOT commit .env to version control.
+
+# -------------------------------------------------------
+# MinIO (S3-compatible object storage)
+# -------------------------------------------------------
+MINIO_ROOT_USER=minioadmin
+MINIO_ROOT_PASSWORD=minioadmin
+
+# -------------------------------------------------------
+# Nessie PostgreSQL backend
+# -------------------------------------------------------
+NESSIE_POSTGRES_DB=nessie
+NESSIE_POSTGRES_USER=nessie
+NESSIE_POSTGRES_PASSWORD=nessie
+
+# -------------------------------------------------------
+# S3 credentials (used by Nessie catalog and Trino)
+# These should match the MinIO root user/password above
+# unless you are using a separate S3-compatible service.
+# -------------------------------------------------------
+S3_ACCESS_KEY=minioadmin
+S3_SECRET_KEY=minioadmin
+S3_ENDPOINT=http://minio:9000
+S3_REGION=us-east-1
+
+# -------------------------------------------------------
+# Trino S3 credentials (for iceberg/hyperstreamdb catalogs)
+# -------------------------------------------------------
+TRINO_S3_ACCESS_KEY=minioadmin
+TRINO_S3_SECRET_KEY=minioadmin
+
+# -------------------------------------------------------
+# Trino PostgreSQL catalog credentials
+# -------------------------------------------------------
+TRINO_PG_CONNECTION_PASSWORD=indie
+
+# -------------------------------------------------------
+# External database connection (used by Trino entrypoint)
+# -------------------------------------------------------
+PG_CONN_RW=
+
+# -------------------------------------------------------
+# API keys (add as needed for your deployment)
+# -------------------------------------------------------
+# AWS_ACCESS_KEY_ID=
+# AWS_SECRET_ACCESS_KEY=
+# AZURE_STORAGE_ACCOUNT=
+# AZURE_STORAGE_KEY=
+# GCS_SERVICE_ACCOUNT_KEY=

{hyperstreamdb-0.4.0 → hyperstreamdb-0.5.0}/.gitignore

@@ -84,6 +84,8 @@ design_*.md
 docs/MONETIZATION_ARCHITECTURE.md
 docs/IN_MEMORY_COMPONENT_ANALYSIS.md
 docs/SUMMARY.md
+docs/DORIS_OPTIMIZATION_PATTERNS.md
+docs/task_status.md
 benchmarks/competitive/*.md
 benchmarks/competitive/benchmark_results/*.md
 

{hyperstreamdb-0.4.0 → hyperstreamdb-0.5.0}/.instructions.md

@@ -1,6 +1,6 @@
 # HyperStreamDB Development Guidelines
 
-**Foundation:** These guidelines are rooted in [The Pragmatic Programmer](https://pragprog.com/) principles. See `/memories/STEERING.md` for the full foundational principles that guide all projects.
+**Foundation:** These guidelines are rooted in [The Pragmatic Programmer](https://pragprog.com/) principles. See `STEERING.md` for the full foundational principles that guide all projects.
 
 ## Core HyperStreamDB Principles
 

hyperstreamdb-0.5.0/.qwen/settings.json

@@ -0,0 +1,9 @@
+{
+  "context": {
+    "fileName": [
+      "QWEN.md",
+      "STEERING.md"
+    ]
+  },
+  "$version": 4
+}

hyperstreamdb-0.5.0/.qwen/settings.json.orig

@@ -0,0 +1,5 @@
+{
+  "context": {
+    "fileName": ["QWEN.md", "STEERING.md"]
+  }
+}

hyperstreamdb-0.5.0/CHANGELOG.md

@@ -0,0 +1,315 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+---
+
+## [0.5.0] - 2026-06-03
+
+### Added
+- Comprehensive unit test suites for Spark and Trino connectors
+- Thread-local GPU context for concurrent query safety
+- Structured error types and fallback chain for production readiness
+- Query configuration API
+- Metrics instrumentation for observability
+- Release profiles (`release` and `release-lto`) in Cargo.toml
+- `WriteAheadLog::append_fire_and_forget` — non-blocking WAL append that hands off batches to the WAL worker without blocking on `fdatasync`, eliminating ~800 ms write latency per call
+
+### Changed
+- CUDA is now an optional feature (no longer required for CI builds)
+- FFI bounds hardened with panic safety guarantees
+- Public APIs documented with rustdoc
+- **`Table(index_all=False)` is now the default** (previously `True`). Automatic HNSW/BM25 index building on commit is now opt-in. This eliminates a silent 15–18 s background build that previously fired on every `commit()` for any table containing a vector column. To restore the old behaviour: `Table(uri, index_all=True)` or `table.index_all = True`.
+- **`Table(autocommit=False)` is now the default** (previously `True`). Writes accumulate in an in-memory buffer and must be explicitly committed with `table.commit()`. This eliminates unexpected auto-flush overhead during ingestion loops.
+- `write_async` no longer auto-detects columns named `"embedding"` as implicit HNSW index targets. Only columns explicitly registered via `add_index()` or `set_index_columns()` are indexed.
+- `manifest_manager.load_latest_full` replaced with `load_latest` in the `flush_async` hot path, eliminating two unnecessary full manifest scans per commit.
+- Primary key uniqueness check upgraded from O(N²) to O(N) using `HashSet`.
+
+### Fixed
+- Rustdoc warnings across the codebase
+- PyO3 API compatibility issues
+- Compilation errors in core modules
+- Critical broken functionality items
+
+---
+
+## [0.4.0] - 2026-05-10
+
+### Added
+- Tiered manifests with 8MB dynamic chunking for large-scale deployments
+- File-based distributed locking for concurrent write safety
+- OTLP (OpenTelemetry) tracing integration
+- FFI panic safety boundaries
+- Chaos and concurrent writers test suites
+- HNSW SIMD indexing (stabilized, replaced legacy simdeez dependency)
+
+### Changed
+- Modularized query execution engine
+- Refactored monolithic `reader.rs` into modular files
+- Refactored monolithic `manifest.rs` into `manager.rs` and `types.rs`
+- Extracted segment indexing logic into separate builder modules
+- Replaced `std::sync` primitives with `parking_lot` for better concurrency
+- Feature-gated `opencl3` and `wgpu` for binary size reduction
+- Upgraded `hashbrown` dependency and trimmed `sqlx` features
+- Eliminated panicking `unwrap()` calls in core library code
+
+### Fixed
+- Filter fallback to Iceberg manifests when no index exists on the column
+- Hybrid search stability and correctness
+- GPU test fallback when CPU-only environment detected
+- Vector search schema compatibility
+
+---
+
+## [0.3.3] - 2026-04-25
+
+### Fixed
+- Vector search schema alignment
+
+---
+
+## [0.3.2] - 2026-04-23
+
+### Changed
+- Stabilized streaming architecture
+- Switched documentation deployment to GitHub Actions
+
+### Fixed
+- Streaming read and Python bindings
+
+---
+
+## [0.3.1] - 2026-04-15
+
+### Fixed
+- TQ8/blob_type index load dispatch
+
+---
+
+## [0.3.0] - 2026-04-14
+
+### Added
+- **High-Density Storage Milestone**: TQ4 (4-bit) and TQ8 (8-bit) TurboQuant quantization
+- Global runtime support
+- Schema evolution infrastructure
+- Finalized indexing infrastructure
+
+### Fixed
+- Multiple stability fixes across storage and query layers
+
+---
+
+## [0.2.6] - 2026-04-12
+
+### Changed
+- Stabilized parallel execution
+- Modernized logging infrastructure
+- Updated GPU installation guidance
+
+### Fixed
+- Categorical column handling
+- Metal backend type mismatch on macOS
+
+---
+
+## [0.2.3] - 2026-04-11
+
+### Changed
+- Modernized GPU acceleration
+- Aligned with PyTorch standards for device detection
+
+---
+
+## [0.2.1] - 2026-04-10
+
+### Changed
+- Auto-sync pyproject version from Cargo.toml
+- Made cudarc optional for non-CUDA CI builds
+
+### Fixed
+- CUDA and MPS device recognition
+- PK validation memory and schema merge logic
+
+---
+
+## [0.2.0] - 2026-04-09
+
+### Added
+- Core engine refactor
+- WAL deduplication
+- Thread-safe GPU context
+- Hardware-agnostic compute dispatch with thread-local context management
+
+### Changed
+- Extracted read, write, builder, schema, and fluent APIs from monolithic table module
+- Introduced `TableBuilder` to streamline initialization
+- Eliminated implicit tokio runtime creation
+
+---
+
+## [0.1.12] - 2026-04-05
+
+### Added
+- Dynamic GPU backend detection (PyTorch-style `build.rs`)
+- Single-source version management
+
+### Changed
+- Unified version across all metadata files
+- Removed non-standard extra-features mapping
+- Removed native Windows target (WSL2 recommended)
+- Updated PyPI classifiers for Python 3.13 and 3.14 support
+- Optimized CI matrix for universal Python 3.10 abi3 wheels
+
+### Fixed
+- Missing library `ImportError` on Linux wheels
+- PyPI trailing data wheel rejection on macOS
+- Manifest manager Rust compiler error
+
+---
+
+## [0.1.9] - 2026-04-04
+
+### Added
+- Partitioned tables support
+- SQL aggregation fixes
+
+### Changed
+- Standardized Table API across Python integration tests
+- Increased floating-point tolerance for GPU distance kernels
+
+### Fixed
+- Inverted index row ID encoding
+- Integration test stability
+
+---
+
+## [0.1.8] - 2026-04-04
+
+### Changed
+- Transitioned to explicit Device API
+- Broadened Intel GPU detection
+- Standardized Intel GPU backend naming
+
+### Fixed
+- PyArrow schema interoperability
+- Mutability errors in `python_gpu_context.rs`
+
+---
+
+## [0.1.7] - 2026-04-03
+
+### Added
+- **10x ingestion speedup** via Async HNSW and Iceberg v3 ZSTD optimizations
+
+### Changed
+- Release stabilization and documentation improvements
+
+---
+
+## [0.1.6] - 2026-04-02
+
+### Added
+- Strict primary key uniqueness enforcement with inverted index lookups
+
+### Fixed
+- PK uniqueness for upsert operations
+- macOS build compatibility
+
+---
+
+## [0.1.5] - 2026-04-02
+
+### Added
+- HNSW-IVF stabilization
+- Python extras hardware mapping (`all_gpu`, `intel_cpu`)
+
+### Changed
+- Fully internalized `hnsw_rs` to `src/core/index/hnsw_rs` for crates.io compatibility
+- Resolved `unexpected_cfgs` warnings
+
+---
+
+## [0.1.3] - 2026-04-01
+
+### Fixed
+- HNSW crate patch compatibility
+
+---
+
+## [0.1.2] - 2026-03-31
+
+### Added
+- Vector search parameter tuning
+- Enhanced diagnostics and query optimization
+- Python API improvements
+- Explain plan enhancements
+
+### Changed
+- Improved API design and explain output
+- Stabilized hybrid search pipeline and RAG demo
+- CI: gated FFI and connector tests behind `java` feature
+- CI: removed Linux aarch64 from release matrix
+- CI: switched to Zig-based cross-compilation
+- CI: added Rust caching for faster builds
+
+### Fixed
+- Restored `add_index_columns` method for Python/Rust bindings
+- OpenCL linker errors and cross-compilation issues
+- Manylinux compatibility for both architectures
+
+---
+
+## [0.1.1] - 2026-03-30
+
+### Added
+- Vector UDFs and aggregates registered in SQL context
+- Hybrid search stabilization
+- Initial RAG demo support
+
+### Changed
+- Aligned development guidelines with pragmatic programmer principles
+- Registered vector operators with DataFusion for hybrid queries
+
+---
+
+## [0.1.0] - 2026-03-30
+
+### Added
+- Initial release of HyperStreamDB
+- Serverless index-streaming database with overlay indexing
+- Apache Iceberg V2/V3 compliance
+- Persistent scalar (RoaringBitmap) and vector (HNSW) indexes
+- Native SQL support via DataFusion
+- Python bindings via PyO3
+- Multi-backend GPU acceleration (CUDA, ROCm, Metal, Intel XPU)
+- Fluent query API with method chaining
+- pgvector-compatible SQL operators
+
+---
+
+[Unreleased]: https://github.com/rla3rd/hyperstreamdb/compare/v0.5.0...HEAD
+[0.5.0]: https://github.com/rla3rd/hyperstreamdb/compare/v0.4.1...v0.5.0
+[0.4.0]: https://github.com/rla3rd/hyperstreamdb/compare/v0.3.3...v0.4.0
+[0.3.3]: https://github.com/rla3rd/hyperstreamdb/compare/v0.3.2...v0.3.3
+[0.3.2]: https://github.com/rla3rd/hyperstreamdb/compare/v0.3.1...v0.3.2
+[0.3.1]: https://github.com/rla3rd/hyperstreamdb/compare/v0.3.0...v0.3.1
+[0.3.0]: https://github.com/rla3rd/hyperstreamdb/compare/v0.2.6...v0.3.0
+[0.2.6]: https://github.com/rla3rd/hyperstreamdb/compare/v0.2.3...v0.2.6
+[0.2.3]: https://github.com/rla3rd/hyperstreamdb/compare/v0.2.1...v0.2.3
+[0.2.1]: https://github.com/rla3rd/hyperstreamdb/compare/v0.2.0...v0.2.1
+[0.2.0]: https://github.com/rla3rd/hyperstreamdb/compare/v0.1.12...v0.2.0
+[0.1.12]: https://github.com/rla3rd/hyperstreamdb/compare/v0.1.9...v0.1.12
+[0.1.9]: https://github.com/rla3rd/hyperstreamdb/compare/v0.1.8...v0.1.9
+[0.1.8]: https://github.com/rla3rd/hyperstreamdb/compare/v0.1.7...v0.1.8
+[0.1.7]: https://github.com/rla3rd/hyperstreamdb/compare/v0.1.6...v0.1.7
+[0.1.6]: https://github.com/rla3rd/hyperstreamdb/compare/v0.1.5...v0.1.6
+[0.1.5]: https://github.com/rla3rd/hyperstreamdb/compare/v0.1.3...v0.1.5
+[0.1.3]: https://github.com/rla3rd/hyperstreamdb/compare/v0.1.2...v0.1.3
+[0.1.2]: https://github.com/rla3rd/hyperstreamdb/compare/v0.1.1...v0.1.2
+[0.1.1]: https://github.com/rla3rd/hyperstreamdb/compare/v0.1.0...v0.1.1
+[0.1.0]: https://github.com/rla3rd/hyperstreamdb/releases/tag/v0.1.0

hyperstreamdb-0.5.0/CONTRIBUTING.md

@@ -0,0 +1,254 @@
+# Contributing to HyperStreamDB
+
+Thank you for your interest in contributing! HyperStreamDB is a serverless index-streaming database built in Rust with Python and Java bindings. This document outlines how to set up your development environment and the conventions we follow.
+
+## Project Principles
+
+Before contributing, please read [STEERING.md](STEERING.md) for our development philosophy. It covers our approach to code quality, design, and release readiness based on *The Pragmatic Programmer*.
+
+---
+
+## Development Setup
+
+### Prerequisites
+
+- **Rust** (stable, 1.75+) — install via [rustup](https://rustup.rs/)
+- **Python** (3.10–3.14) — for binding development and testing
+- **Docker** — for integration tests against MinIO and Nessie
+- **Cargo** and **maturin** — for building Python wheels
+
+```bash
+# Install Rust
+curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
+
+# Install maturin (Python-Rust build tool)
+pip install maturin
+
+# Clone and build
+git clone https://github.com/rla3rd/hyperstreamdb.git
+cd hyperstreamdb
+cargo build --release
+```
+
+### Python Bindings
+
+```bash
+# Develop Python bindings locally
+maturin develop
+
+# Run Python tests
+pytest tests/
+```
+
+### Docker Services
+
+```bash
+# Start MinIO (S3-compatible storage) and Nessie (Iceberg catalog)
+docker compose -f docker-compose-minio-nessie.yml up -d
+
+# Run integration tests
+pytest tests/integration/
+```
+
+### GPU Acceleration (Optional)
+
+GPU features are optional and feature-gated. To build with GPU support:
+
+```bash
+# With CUDA (NVIDIA)
+cargo build --features cuda
+
+# With WGPU (Vulkan/Metal/ROCm)
+cargo build --features wgpu
+```
+
+See the [README](README.md#gpu-acceleration-optional) for hardware-specific installation guides.
+
+---
+
+## Coding Standards
+
+### Rust
+
+- **Formatting**: Run `cargo fmt` before committing. We use `rustfmt` defaults.
+- **Linting**: Run `cargo clippy --all-features` and resolve all warnings. Treat clippy warnings as errors.
+- **Naming**:
+  - Modules: `snake_case`
+  - Types/Structs: `PascalCase`
+  - Functions/methods: `snake_case`
+  - Constants: `SCREAMING_SNAKE_CASE`
+- **Error handling**: Use `Result<T, HyperstreamError>` throughout. Never use `.unwrap()` in library code — prefer `.expect()` with a descriptive message or proper error propagation.
+- **Documentation**: All `pub` items must have rustdoc comments. Run `cargo doc --no-deps` to verify.
+
+### Python
+
+- Follow [PEP 8](https://peps.python.org/pep-0008/) for style
+- Use type hints on all public functions and methods
+- Docstrings should follow [Google style](https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings)
+
+### Commit Messages
+
+We follow [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/):
+
+```
+<type>(<scope>): <description>
+
+[optional body]
+```
+
+**Types:**
+| Type       | Description                                      |
+|------------|--------------------------------------------------|
+| `feat`     | New feature or functionality                     |
+| `fix`      | Bug fix                                          |
+| `perf`     | Performance improvement                          |
+| `refactor` | Code change that neither fixes a bug nor adds a feature |
+| `docs`     | Documentation only changes                       |
+| `test`     | Adding or correcting tests                       |
+| `chore`    | Maintenance, CI, build, dependency updates       |
+| `ci`       | Continuous Integration configuration changes     |
+
+**Examples:**
+```
+feat(indexing): stabilize HNSW SIMD indexing and remove legacy simdeez dependency
+fix: resolve rustdoc warnings and pyo3 API compatibility
+refactor(gpu): implement hardware-agnostic compute dispatch with thread-local context
+docs: switch to GitHub Actions for Pages deployment
+test: add comprehensive unit test suites for Spark and Trino connectors
+chore: bump version to v0.4.0
+```
+
+---
+
+## Branch Naming Conventions
+
+```
+feat/<feature-name>        # New features
+fix/<issue-or-bug>         # Bug fixes
+refactor/<component>       # Refactoring work
+docs/<topic>               # Documentation changes
+test/<scope>               # Test additions or changes
+chore/<task>               # Maintenance tasks
+```
+
+**Examples:**
+- `feat/tiered-manifests`
+- `fix/hybrid-search-fallback`
+- `refactor/gpu-context`
+- `test/spark-connector`
+
+---
+
+## Pull Request Process
+
+### 1. Create a Draft PR
+
+Start with a **Draft PR** while work is in progress. This allows early feedback without triggering full CI review.
+
+### 2. Self-Review Checklist
+
+Before marking your PR as **Ready for Review**:
+
+- [ ] Code passes `cargo fmt` and `cargo clippy` with zero warnings
+- [ ] All tests pass (`cargo test` and `pytest tests/`)
+- [ ] New features include corresponding unit and/or integration tests
+- [ ] Public API changes have rustdoc documentation
+- [ ] `CHANGELOG.md` is updated (if this is a user-facing change)
+- [ ] No broken windows: compiler warnings, failing tests, or suboptimal patterns
+
+### 3. CI Validation
+
+The CI pipeline runs:
+- **Rust checks**: `cargo check`, `cargo clippy`, `cargo test`
+- **Python tests**: `pytest`
+- **Documentation build**: `cargo doc --no-deps`
+- **Integration tests**: against MinIO and Nessie (when applicable)
+
+All CI checks must pass before merge.
+
+### 4. Review and Merge
+
+- PRs require at least one approval from a maintainer
+- Address all review comments before merging
+- Squash-merge to keep history clean
+
+---
+
+## Testing Requirements
+
+### Unit Tests
+
+Every new feature or bug fix must include unit tests. Use `#[cfg(test)]` modules within the relevant source file:
+
+```rust
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_feature_behavior() {
+        // ...
+    }
+}
+```
+
+### Integration Tests
+
+For features that span multiple modules or involve external systems (storage, catalogs), add integration tests under `tests/integration/`:
+
+```bash
+# Run all integration tests
+pytest tests/integration/
+
+# Run specific test
+pytest tests/integration/test_hybrid_search.py -v
+```
+
+### Benchmarks
+
+Performance-critical changes should include benchmark updates:
+
+```bash
+# Run Criterion benchmarks
+cargo bench
+
+# Quick benchmark suite
+python benchmarks/benchmark_suite.py --quick
+```
+
+---
+
+## Building Connectors
+
+The Spark and Trino connectors require a separate build step:
+
+```bash
+# Build all connector variants
+./build-connectors.sh
+
+# Build with CUDA support
+./build-connectors.sh --cuda
+```
+
+---
+
+## Release Process
+
+Releases are tagged and published by the maintainer. The process involves:
+
+1. Bump version in `Cargo.toml` (single source of truth)
+2. Update `CHANGELOG.md` with release date
+3. Create git tag: `git tag v0.X.Y`
+4. Push tag: `git push origin v0.X.Y`
+5. CI automatically publishes to crates.io and PyPI
+
+---
+
+## Questions or Issues?
+
+- Open a [GitHub Issue](https://github.com/rla3rd/hyperstreamdb/issues) for bugs or feature requests
+- For discussions, use the Discussions tab or reach out directly
+
+---
+
+**Stay Pragmatic. Stay Antigravity.**