benostreamdb 0.10.0__tar.gz

benostreamdb-0.10.0/.cargo/audit.toml

@@ -0,0 +1,25 @@
+# Cargo Audit Configuration
+# https://github.com/rustsec/rustsec/tree/main/cargo-audit
+
+[advisories]
+ignore = [
+    # Transitive dependencies & existing ignores
+    "RUSTSEC-2026-0104", # rustls-webpki: CRL panic
+    "RUSTSEC-2026-0099", # rustls-webpki: wildcard name constraints
+    "RUSTSEC-2026-0098", # rustls-webpki: URI name constraints
+    "RUSTSEC-2026-0097", # rand: unsound with custom logger
+    "RUSTSEC-2024-0437", # protobuf: uncontrolled recursion
+    "RUSTSEC-2026-0177", # pyo3: missing Sync bound
+    "RUSTSEC-2026-0176", # pyo3: out-of-bounds read in iterator
+    "RUSTSEC-2023-0071", # rsa: Marvin attack
+    "RUSTSEC-2026-0258", # h2: unbounded empty DATA frames
+    "RUSTSEC-2026-0195", # quick-xml: unbounded namespace allocation
+    "RUSTSEC-2026-0194", # quick-xml: quadratic run time in duplicate attribute check
+    "RUSTSEC-2025-0134", # rustls-pemfile: unmaintained advisory
+    "RUSTSEC-2026-0190", # anyhow: Error::downcast_mut unsoundness
+    "RUSTSEC-2026-0221", # event-listener: !Send tags across thread boundaries
+    "RUSTSEC-2026-0204", # crossbeam-epoch: pointer dereference
+    "RUSTSEC-2026-0234", # rkyv: hash table out of bounds
+    "RUSTSEC-2026-0233", # rkyv: use-after-free
+    "RUSTSEC-2026-0235", # rkyv: Rc/Arc out of bounds
+]

benostreamdb-0.10.0/.dockerignore

@@ -0,0 +1,16 @@
+target/
+.git/
+.github/
+.venv/
+venv/
+*.db
+*.parquet
+*.log
+test_*
+rag_db/
+news_db/
+demo/
+benchmarks/
+__pycache__/
+*.pyc
+!target/release/bsdb-search

benostreamdb-0.10.0/.env.example

@@ -0,0 +1,51 @@
+# BenoStreamDB Environment Configuration
+# Copy this file to .env and fill in your credentials.
+# DO NOT commit .env to version control.
+
+# -------------------------------------------------------
+# MinIO (S3-compatible object storage)
+# -------------------------------------------------------
+MINIO_ROOT_USER=minioadmin
+MINIO_ROOT_PASSWORD=minioadmin
+
+# -------------------------------------------------------
+# Nessie PostgreSQL backend
+# -------------------------------------------------------
+NESSIE_POSTGRES_DB=nessie
+NESSIE_POSTGRES_USER=nessie
+NESSIE_POSTGRES_PASSWORD=nessie
+
+# -------------------------------------------------------
+# S3 credentials (used by Nessie catalog and Trino)
+# These should match the MinIO root user/password above
+# unless you are using a separate S3-compatible service.
+# -------------------------------------------------------
+S3_ACCESS_KEY=minioadmin
+S3_SECRET_KEY=minioadmin
+S3_ENDPOINT=http://minio:9000
+S3_REGION=us-east-1
+
+# -------------------------------------------------------
+# Trino S3 credentials (for iceberg/benostreamdb catalogs)
+# -------------------------------------------------------
+TRINO_S3_ACCESS_KEY=minioadmin
+TRINO_S3_SECRET_KEY=minioadmin
+
+# -------------------------------------------------------
+# Trino PostgreSQL catalog credentials
+# -------------------------------------------------------
+TRINO_PG_CONNECTION_PASSWORD=indie
+
+# -------------------------------------------------------
+# External database connection (used by Trino entrypoint)
+# -------------------------------------------------------
+PG_CONN_RW=
+
+# -------------------------------------------------------
+# API keys (add as needed for your deployment)
+# -------------------------------------------------------
+# AWS_ACCESS_KEY_ID=
+# AWS_SECRET_ACCESS_KEY=
+# AZURE_STORAGE_ACCOUNT=
+# AZURE_STORAGE_KEY=
+# GCS_SERVICE_ACCOUNT_KEY=

benostreamdb-0.10.0/.gitattributes

@@ -0,0 +1 @@
+*.so binary=true

benostreamdb-0.10.0/.gitignore

@@ -0,0 +1,129 @@
+# ── Rust ────────────────────────────────────────────────────────────────────
+target/
+**/*.rs.bk
+
+# ── Python ──────────────────────────────────────────────────────────────────
+__pycache__/
+*.py[cod]
+*$py.class
+.venv/
+.venv_iceberg/
+.venv-cuda-shims/
+venv/
+venv_verify/
+venv_demo/
+test_venv/
+!venv/requirements.txt
+!venv/.gitkeep
+*.egg-info/
+
+# Compiled Python extension (built by `maturin develop`)
+python/**/*.so
+python/**/*.pyd
+python/**/*.dylib
+
+# ── IDE / editor / tooling ──────────────────────────────────────────────────
+.vscode/
+.idea/
+.metals/
+.continue/
+.qwen/
+.agents/
+
+# ── Compiled native binaries / debug symbols ────────────────────────────────
+*.dylib
+*.dSYM/
+*.so
+*.dll
+*.exe
+test_pgvector_sql_integration
+scratch_mem_test
+
+# ── BenoStream data artifacts ───────────────────────────────────────────────
+*.parquet
+*.idx
+*.hnsw.graph
+*.hnsw.data
+*.del
+_manifest/
+_metadata/
+test_admin_table/
+test_explain_db/
+test_ingestion_db/
+test_ingestion_perf_db/
+test_lancedb_ingestion_perf_db/
+
+# ── Local toolchain downloads (Maven/JDK fetched by build-connectors.sh) ────
+.java/
+.maven/
+
+# ── Local vector DBs, scratch data & generated artifacts ────────────────────
+rag_db/
+news_db/
+data/
+docker/data/
+benchmarks/competitive/data/
+hsdbg_bench_data/
+connector-artifacts/
+dist/
+benchmark_results/
+trino-benostream/src/main/resources/libbenostreamdb.so
+spark-benostream/src/main/resources/libbenostreamdb.so
+
+# ── Documentation build output (Sphinx) ─────────────────────────────────────
+docs/build/
+
+# ── Build / test logs & scratch ─────────────────────────────────────────────
+# Run logs are written here (see docs/BENCHMARKING.md), never committed.
+logs/
+*.log
+*.pid
+*.patch
+dhat-heap.json
+build_errors*.txt
+check_output.txt
+rust_test_output*.log
+python_test_output*.log
+python_feature_test_output*.log
+test_output*.txt
+*_output.txt
+mem_report.txt
+benchmark_raw.json
+benchmarks/competitive/venv/
+
+# ── Test caches ─────────────────────────────────────────────────────────────
+.hypothesis/
+.pytest_cache/
+.ipynb_checkpoints/
+
+# ── Local secrets — never commit ────────────────────────────────────────────
+.streamlit/secrets.toml
+.streamlit/config.toml
+.env
+.env.local
+groq_api_key.txt
+
+# ── Ad-hoc local scripts & scratch (not part of the repo) ───────────────────
+scratch*.rs
+/test
+test.rs
+test_avro.py
+test_investigate.py
+test_mmap.py
+test_vectors.py
+debug_read*.py
+patch_*.py
+patch_*.sh
+fix_*.py
+extract_comments.py
+analyze_comments.py
+all_comments.txt
+interesting_comments.txt
+user_msg.txt
+measure_disk.py
+parse_dhat.py
+populate.py
+run_*.sh
+test_bench.sh
+bench_*.py
+run_benchmark.py

benostreamdb-0.10.0/.instructions.md

@@ -0,0 +1,67 @@
+# BenoStreamDB Development Guidelines
+
+**Foundation:** These guidelines are rooted in [The Pragmatic Programmer](https://pragprog.com/) principles.
+
+## Core BenoStreamDB Principles
+
+### Code Quality
+- **Always attempt to fix warnings if possible** - Don't ignore compiler warnings; resolve them during implementation (Aligns with: Don't Live with Broken Windows #4)
+- Use proper error handling instead of silencing warnings
+- Maintain clean build output (zero clippy warnings)
+- Type safety is paramount
+- Refactor early and often as code is written
+
+### Versioning & Releases
+- Version numbers are injected from git tags during build
+- `Cargo.toml` and `pyproject.toml` use dynamic versioning from `build.rs`
+- Tag format: `v0.X.Y` (e.g., `v0.1.2`)
+- All releases should be tagged and pushed with comprehensive release notes
+
+### Licensing & Attribution
+- Apache 2.0 / MIT dual license
+- Properly attribute optimizations borrowed from iceberg-rust or other Apache-licensed projects
+- Maintain THIRDPARTY_NOTICES.md with all third-party code usage
+- Include inline code comments for borrowed patterns
+
+### API Design
+- Use native Rust names (PascalCase for types: `Cosine`, `L2`, `InnerProduct`)
+- Python API uses lowercase for user-facing functions: `metric="cosine"`
+- Support both primary parameter names and backward-compatible aliases
+- Document all vector distance functions and index parameters
+
+### Testing & Validation
+- Unit tests required for new optimizer rules and query optimizations (Aligns with: Design to Test #20, Test Your Software #21)
+- Explain() output should show correct index usage and query strategy
+- Test suite must pass before any release tagging (Aligns with: Make Quality a Requirements Issue #7)
+- Performance benchmarks are part of testing, not optional extras
+- Integration tests for query planning changes
+- Run full test suite before tagging releases
+- Verify explain() output shows correct index detection
+
+### Packaging
+- Exclude build artifacts, virtual environments from Cargo.toml (Aligns with: Make It Easy to Reuse #11)
+- Ensure PyPI builds succeed without external dependencies
+- Test `maturin develop --release` before release tagging
+- Dynamic versioning from git tags (build.rs injection)
+
+## Current Focus Areas
+
+### Hybrid Query Optimization
+- Support multiple vector columns with ordering
+- Compound primary key detection for automatic hybrid search
+- Vector search always runs after scalar filters (push-down)
+- Proper explain() output showing both scalar and vector execution
+- Use tracer bullets: Implement end-to-end before perfecting details (Aligns with: Use Tracer Bullets #14)
+
+### Iceberg-Rust Optimizations
+- LIMIT pushdown to file scanning layer
+- Row group skipping based on column statistics
+- Manifest metadata caching for performance
+- Predicate pushdown improvements
+- Properly attribute all borrowed code patterns (Aligns with: Communicate! #9, Make It Easy to Reuse #11)
+
+### Observability & Debugging
+- Structured tracing via `tracing` crate (no println!) (Aligns with: Program Close to Problem Domain #16)
+- Proper log levels (debug, info, warn, error)
+- Explain query plans for user visibility (Aligns with: Use Tracer Bullets #14)
+- Document query strategy through explain() output

benostreamdb-0.10.0/.mailmap

@@ -0,0 +1,6 @@
+# Normalize every historical author/committer to the project owner.
+Rick Albright <rla3rd@gmail.com> <rick@hyperstreamdb.com>
+Rick Albright <rla3rd@gmail.com> <ralbright@verityplatform.com>
+Rick Albright <rla3rd@gmail.com> <ralbright@google.com>
+Rick Albright <rla3rd@gmail.com> <rla3rd@users.noreply.github.com>
+Rick Albright <rla3rd@gmail.com> <rla3rd@gmail.com>