isage-vdb 0.2.0.10__tar.gz

isage_vdb-0.2.0.10/.github/agents/sageVDB-agent.agent.md

@@ -0,0 +1,51 @@
+---
+description: 'SageVDB development agent for FAISS-like vector database with pluggable ANNS algorithms'
+tools: ['vscode', 'execute', 'read', 'agent', 'edit', 'search', 'web', 'todo', 'vscode.mermaid-chat-features/renderMermaidDiagram', 'github.vscode-pull-request-github/issue_fetch', 'github.vscode-pull-request-github/suggest-fix', 'github.vscode-pull-request-github/searchSyntax', 'github.vscode-pull-request-github/doSearch', 'github.vscode-pull-request-github/renderIssues', 'github.vscode-pull-request-github/activePullRequest', 'github.vscode-pull-request-github/openPullRequest', 'ms-azuretools.vscode-containers/containerToolsConfig', 'ms-python.python/getPythonEnvironmentInfo', 'ms-python.python/getPythonExecutableCommand', 'ms-python.python/installPythonPackage', 'ms-python.python/configurePythonEnvironment', 'ms-toolsai.jupyter/configureNotebook', 'ms-toolsai.jupyter/listNotebookPackages', 'ms-toolsai.jupyter/installNotebookPackages', 'ms-vscode.cpp-devtools/Build_CMakeTools', 'ms-vscode.cpp-devtools/RunCtest_CMakeTools', 'ms-vscode.cpp-devtools/ListBuildTargets_CMakeTools', 'ms-vscode.cpp-devtools/ListTests_CMakeTools']
+---
+
+# SageVDB Development Agent
+
+## Purpose
+This agent assists with developing, testing, and maintaining the SageVDB C++ vector database library. It understands the FAISS-compatible API design, pluggable ANNS architecture, and multimodal fusion capabilities.
+
+## When to Use
+- Adding or modifying ANNS algorithm plugins
+- Working with vector operations, indexing, or search
+- Implementing multimodal fusion strategies
+- Debugging build/test issues
+- Updating documentation for API changes
+
+## Quick Context
+- **Language**: C++20 core, optional CUDA/FAISS; Python bindings via pybind11
+- **Entry points**: `SageVDB`/`VectorStore`/`QueryEngine` in include/sage_vdb; ANNS plugins under src/anns and registered via `REGISTER_ANNS_ALGORITHM`
+- **Goal**: FAISS-like API surface with better modularity; new ANNS backends should be pluggable without changing public headers
+
+## Ready-Made Commands
+- Configure & build (Release): `cmake -B build -S . -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTS=ON`
+- Build helper script: `./build.sh`
+- Run tests (from build): `ctest --verbose` or `./test_sage_vdb`, `./test_multimodal`
+- Enable FAISS: supply `-DFAISS_ROOT=$CONDA_PREFIX` (matches CI) and build with `ENABLE_FAISS=ON` option if present
+
+## When Editing
+- Keep new ANNS algorithms under src/anns with public headers in include/sage_vdb/anns; register factories in .cpp files only
+- Maintain FAISS-compatible knobs (k/nprobe/index type/metric) and thread params through `DatabaseConfig` and `SearchParams` instead of ad-hoc globals
+- Preserve thread-safety in VectorStore (shared_mutex) and follow existing exception style (`SageVDBException`, runtime_error for unsupported ops)
+- Do not create new local virtual environments (`venv`/`.venv`); use the existing configured Python environment.
+- Update docs: core changes → README.md; multimodal → docs/guides/README_Multimodal.md, docs/USAGE_MODES.md
+
+## Validation Checklist
+- Build succeeds without FAISS; optional FAISS path guarded by ENABLE_FAISS
+- Tests green (`test_sage_vdb`, `test_multimodal`)
+- Public headers remain stable (API-compatible with FAISS-style usage)
+
+## Output Style
+- Provide concrete code examples for ANNS registration patterns
+- Reference existing plugins (brute_force, faiss) as templates
+- Always validate dimension compatibility in multimodal fusion
+- Keep explanations concise with build/test commands ready to copy
+
+## Polyrepo coordination rules
+
+- Treat this repository as the only local source tree; do not assume sibling repositories exist.
+- If a task spans multiple repositories, implement only this repo and explicitly list follow-up repo/version-bump actions.
+- Do not create `venv`/`.venv`; always use the existing configured Python environment.

isage_vdb-0.2.0.10/.github/copilot-instructions.md

@@ -0,0 +1,79 @@
+# Copilot Instructions for SageVDB
+
+These guardrails keep completions consistent with the repo goal: FAISS-style API with a modular, pluggable ANNS core and multimodal fusion.
+
+## Design North Star
+- Mirror FAISS user ergonomics: `IndexType`, `DistanceMetric`, `DatabaseConfig`, `train_index()`, `build_index()`, `search()`, and IVF/HNSW params match FAISS naming where possible.
+- Everything runs through `VectorStore`/`QueryEngine` with algorithm lookup via `anns::ANNSRegistry`. New algorithms must register factories with `REGISTER_ANNS_ALGORITHM` and avoid leaking implementation headers.
+- Multimodal stays additive: `MultimodalSageVDB` composes `ModalityManager` and `FusionEngine` (concat/weighted/attention/etc.) without coupling to specific embedders.
+
+## Preferred APIs
+- For vector DB entry points use `SageVDB` and `create_database()`; validate dimensions with `DatabaseConfig` rather than ad-hoc checks.
+- Use `SearchParams` for k/nprobe/radius/metadata flags; use `anns::QueryConfig` for algorithm-specific knobs; keep `anns_build_params`/`anns_query_params` stringly-typed like FAISS CLI.
+- Metadata flows through `MetadataStore` and `QueryResult.metadata`; keep it optional but stable.
+- Persistence goes through `VectorStore::save/load` and mirrors FAISS index IO semantics; avoid one-off serializers.
+
+## Implementation Rules
+- Respect capabilities: if an algorithm cannot update/delete/range-search, throw `std::runtime_error` with the algorithm name (see `anns::ANNSAlgorithm` defaults).
+- Keep thread safety: `VectorStore` uses shared_mutex for read-heavy paths; do not expose raw internals that bypass locking.
+- Favor C++20 standard library; avoid introducing new deps unless aligned with cmake options (`ENABLE_FAISS`, `ENABLE_MULTIMODAL`, etc.).
+- Do not create new local virtual environments (`venv`/`.venv`); use the existing configured Python environment.
+- Keep code pluggable: new ANNS implementations live under `src/anns/` + public headers under `include/sage_vdb/anns/`; register via factory macro in a `.cpp` file only.
+
+## Testing & Build
+- Default build: `./build.sh`; CI builds with `cmake -B build ...` and optional `-DFAISS_ROOT` when FAISS is present.
+- Tests: from `build/`, run `ctest --verbose` or `./test_sage_vdb`, `./test_multimodal`.
+
+## Python Binding Notes
+- `python/bindings.cpp` exposes the C++ core; keep interface names and docstrings aligned with the C++ API (FAISS-like naming, e.g., `search`, `add`, `train_index`).
+- When adding params, thread them through `DatabaseConfig` -> `VectorStore` -> bindings; keep defaults backward compatible.
+
+## Documentation Expectations
+- Update `README.md` for core vector/ANNS changes; use `docs/guides/README_Multimodal.md` and `docs/USAGE_MODES.md` when changing multimodal flows.
+- Keep option names and examples FAISS-compatible to ease user migration.
+
+## Publishing
+
+**⚠️ IMPORTANT: Publishing requires explicit version update and manual action**
+
+### Publishing Workflow
+1. **Update version** in `pyproject.toml` (current: 0.1.5)
+   - Bug fixes: increment patch (0.1.5 → 0.1.6)
+   - New features: increment minor (0.1.5 → 0.2.0)
+   - Breaking changes: increment major (0.1.5 → 1.0.0)
+
+2. **Build and publish** using `sage-pypi-publisher`:
+   ```bash
+   # One-command workflow (recommended)
+   cd /path/to/sageVDB
+   sage-pypi-publisher publish . -r testpypi --no-dry-run  # Test first
+   sage-pypi-publisher publish . -r pypi --no-dry-run      # Production
+   
+   # publish command auto-detects C++ extension and builds manylinux wheels
+   ```
+
+3. **Git hook behavior**: The pre-push hook will:
+   - Warn if version wasn't updated
+   - Ask [u/y/n]: update now / continue / cancel
+   - Choosing 'y' pushes to GitHub but does NOT publish to PyPI
+
+### Notes
+- Use latest `isage-pypi-publisher` from PyPI
+- PyPI token must be in `~/.pypirc`
+- `publish` command auto-detects package type (no need for --force-manylinux)
+- Push to GitHub and publish to PyPI are separate steps
+- See `docs/ops/RELEASE.md` and `docs/ops/DEPLOYMENT.md` for details
+
+## Polyrepo coordination (mandatory)
+
+- This repository is an independent SAGE sub-repository and is developed/released independently.
+- Do not assume sibling source directories exist locally in `intellistream/SAGE`.
+- For cross-repo rollout, publish this repo/package first, then bump the version pin in `SAGE/packages/sage/pyproject.toml` when applicable.
+- Do not add local editable installs of other SAGE sub-packages in setup scripts or docs.
+
+## 🚫 NEVER_CREATE_DOT_VENV_MANDATORY
+
+- 永远不要创建 `.venv` 或 `venv`（无任何例外）。
+- NEVER create `.venv`/`venv` in this repository under any circumstance.
+- 必须复用当前已配置的非-venv Python 环境（如现有 conda 环境）。
+- If any script/task suggests creating a virtualenv, skip that step and continue with the existing environment.

isage_vdb-0.2.0.10/.github/workflows/ci-tests.yml

@@ -0,0 +1,163 @@
+name: CMake - Build and Test SageVDB
+
+on:
+  pull_request:
+    branches: [ main, main-dev ]
+  push:
+    branches: [ main, main-dev ]
+  workflow_dispatch:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  build-and-test:
+    name: ${{ matrix.os }} | CMake ${{ matrix.build_type }}
+    runs-on: ${{ matrix.os }}
+    env:
+      # Limit CI log level to avoid excessive DEBUG logs
+      sage_vdb_LOG_LEVEL: info
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ ubuntu-latest ]
+        build_type: [ Release, Debug ]
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      # Setup conda environment for FAISS and other dependencies
+      - name: Setup Miniconda
+        uses: conda-incubator/setup-miniconda@v3
+        with:
+          auto-update-conda: true
+          python-version: 3.11
+          channels: conda-forge
+
+      - name: Install dependencies via conda
+        shell: bash -el {0}
+        run: |
+          conda install -y \
+            faiss-cpu \
+            openblas \
+            cmake \
+            make \
+            gcc_linux-64 \
+            gxx_linux-64 \
+            pkg-config
+          python -m pip install --upgrade pip
+          # Editable install ensures the built extension is available under ./sagevdb for pytest.
+          # Avoid CI fragility from unpublished optional integration dependencies.
+          python -m pip install pytest pytest-cov black isort
+          python -m pip install -e . --no-deps
+
+      # Install additional C++ toolchain (Ubuntu)
+      - name: Install C++ toolchain (Ubuntu)
+        if: runner.os == 'Linux'
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y --no-install-recommends \
+            build-essential \
+            cmake \
+            pkg-config \
+            libblas-dev \
+            liblapack-dev
+
+      - name: Show compiler and dependency versions
+        shell: bash -el {0}
+        run: |
+          g++ --version || true
+          gcc --version || true
+          cmake --version || true
+          conda list | grep faiss || true
+          conda list | grep openblas || true
+          echo "Python packages:"
+
+      - name: Configure CMake
+        shell: bash -el {0}
+        run: |
+          cmake -S . -B build \
+            -DCMAKE_BUILD_TYPE=${{ matrix.build_type }} \
+            -DBUILD_TESTS=ON \
+            -DBUILD_PYTHON_BINDINGS=OFF \
+            -DENABLE_MULTIMODAL=ON \
+            -DCMAKE_POSITION_INDEPENDENT_CODE=ON \
+            -DCMAKE_PREFIX_PATH="$CONDA_PREFIX" \
+            -DFAISS_ROOT="$CONDA_PREFIX"
+
+      - name: Build
+        shell: bash -el {0}
+        run: |
+          # Limit parallel jobs to avoid OOM on CI runners
+          cmake --build build --config ${{ matrix.build_type }} --parallel 2
+
+      - name: Verify build directory
+        shell: bash -el {0}
+        run: |
+          if [ ! -d build ]; then
+            echo "Error: build directory not found after build step."
+            exit 1
+          fi
+          echo "Build directory contents:"
+          ls -la build
+
+      - name: Set library path for tests
+        shell: bash -el {0}
+        run: |
+          echo "LD_LIBRARY_PATH=$PWD/build:$CONDA_PREFIX/lib:$LD_LIBRARY_PATH" >> $GITHUB_ENV
+
+      - name: Run all tests
+        shell: bash -el {0}
+        run: |
+          if [ ! -d build ]; then
+            echo "Error: build directory not found before tests."
+            exit 1
+          fi
+          cd build
+          LD_LIBRARY_PATH=$PWD:$CONDA_PREFIX/lib ctest --output-on-failure --verbose
+
+      - name: Run Python tests
+        shell: bash -el {0}
+        run: |
+          python -m pytest -v tests
+
+      - name: Run direct test executables (fallback)
+        if: failure()
+        shell: bash -el {0}
+        run: |
+          echo "Running tests directly..."
+          if [ ! -d build ]; then
+            echo "Build directory not found; skip fallback executable tests."
+            exit 0
+          fi
+          cd build
+          if [ -f ./test_sage_vdb ]; then
+            echo "Running test_sage_vdb..."
+            LD_LIBRARY_PATH=$PWD:$CONDA_PREFIX/lib ./test_sage_vdb || true
+          fi
+          if [ -f ./test_multimodal ]; then
+            echo "Running test_multimodal..."
+            LD_LIBRARY_PATH=$PWD:$CONDA_PREFIX/lib ./test_multimodal || true
+          fi
+
+      - name: Upload test logs on failure
+        if: failure()
+        uses: actions/upload-artifact@v4
+        with:
+          name: sage-db-test-logs-${{ matrix.os }}-${{ matrix.build_type }}
+          path: |
+            build/Testing/**
+            build/*.log
+            build/**/compile_commands.json
+            build/test_sage_vdb
+            build/test_multimodal
+
+      - name: Upload built library
+        if: success()
+        uses: actions/upload-artifact@v4
+        with:
+          name: sage-db-library-${{ matrix.os }}-${{ matrix.build_type }}
+          path: |
+            build/libsage_vdb.so
+            build/_sage_vdb*.so

isage_vdb-0.2.0.10/.github/workflows/quick-test.yml

@@ -0,0 +1,88 @@
+name: Quick Test
+
+on:
+  pull_request:
+    branches: [ main, main-dev ]
+    paths:
+      - 'src/**'
+      - 'include/**'
+      - 'tests/**'
+      - 'CMakeLists.txt'
+      - 'cmake/**'
+  workflow_dispatch:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  quick-test:
+    name: Quick Test (Ubuntu Release)
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      # Use cache for conda dependencies
+      - name: Cache conda dependencies
+        uses: actions/cache@v4
+        with:
+          path: ~/conda_pkgs_dir
+          key: ${{ runner.os }}-conda-${{ hashFiles('**/requirements*.txt') }}
+          restore-keys: |
+            ${{ runner.os }}-conda-
+
+      - name: Setup Miniconda
+        uses: conda-incubator/setup-miniconda@v3
+        with:
+          auto-update-conda: true
+          python-version: 3.11
+          channels: conda-forge
+          use-only-tar-bz2: true
+
+      - name: Install dependencies
+        shell: bash -el {0}
+        run: |
+          conda install -y \
+            faiss-cpu \
+            openblas \
+            cmake \
+            make \
+            gcc_linux-64 \
+            gxx_linux-64
+
+      - name: Configure and Build
+        shell: bash -el {0}
+        run: |
+          cmake -S . -B build \
+            -DCMAKE_BUILD_TYPE=Release \
+            -DBUILD_TESTS=ON \
+            -DBUILD_PYTHON_BINDINGS=OFF \
+            -DENABLE_MULTIMODAL=ON \
+            -DCMAKE_PREFIX_PATH="$CONDA_PREFIX" \
+            -DFAISS_ROOT="$CONDA_PREFIX"
+          
+          cmake --build build --parallel
+
+      - name: Run Tests
+        shell: bash -el {0}
+        run: |
+          if [ ! -d build ]; then
+            echo "Error: build directory not found before tests."
+            exit 1
+          fi
+          cd build
+          LD_LIBRARY_PATH=$PWD:$CONDA_PREFIX/lib ctest --output-on-failure
+
+      - name: Test library loading
+        shell: bash -el {0}
+        run: |
+          echo "Testing library dependencies..."
+          if [ ! -d build ]; then
+            echo "Error: build directory not found before library checks."
+            exit 1
+          fi
+          cd build
+          ldd libsage_vdb.so || true
+          LD_LIBRARY_PATH=$PWD:$CONDA_PREFIX/lib ldd test_sage_vdb || true

isage_vdb-0.2.0.10/.gitignore

@@ -0,0 +1,24 @@
+# Build directories
+build/
+install/
+_codeql_build_dir/
+_codeql_detected_source_root
+
+# Python cache
+__pycache__/
+*/__pycache__/
+**/__pycache__/
+*.pyc
+
+# Compiled extensions
+*.so
+*.o
+*.a
+
+# Specific compiled files
+python/_sage_vdb.cpython-*-linux-gnu.so
+dist/*
+wheelhouse/*
+.env
+.coverage
+pytest.log

isage_vdb-0.2.0.10/.pre-commit-config.yaml

@@ -0,0 +1,89 @@
+# Pre-commit hooks configuration for C++ submodule
+# Installation:
+#   pip install pre-commit
+#   pre-commit install
+#
+# Usage:
+#   pre-commit run --all-files
+#   git commit --no-verify  # Skip hooks temporarily
+
+default_language_version:
+  python: python3.11
+
+repos:
+  # General file checks
+- repo: https://github.com/pre-commit/pre-commit-hooks
+  rev: v6.0.0
+  hooks:
+  - id: trailing-whitespace
+  - id: end-of-file-fixer
+  - id: check-yaml
+    args: [--unsafe]
+  - id: check-json
+  - id: check-toml
+  - id: check-added-large-files
+    args: ['--maxkb=1000']
+  - id: check-merge-conflict
+  - id: check-case-conflict
+  - id: mixed-line-ending
+    args: [--fix=lf]
+  - id: detect-private-key
+
+  # C++: clang-format (code formatting)
+- repo: https://github.com/pre-commit/mirrors-clang-format
+  rev: v19.1.6
+  hooks:
+  - id: clang-format
+    types_or: [c++, c]
+    args: ['-i']
+
+  # CMake: cmake-format
+- repo: https://github.com/cheshirekow/cmake-format-precommit
+  rev: v0.6.13
+  hooks:
+  - id: cmake-format
+    args: [--in-place]
+  - id: cmake-lint
+    args: [--disabled-codes=C0103,C0301]
+
+  # Python (for pybind11 bindings and test scripts)
+- repo: https://github.com/astral-sh/ruff-pre-commit
+  rev: v0.14.2
+  hooks:
+  - id: ruff
+    args: [--fix]
+    types_or: [python, pyi]
+  - id: ruff-format
+    types_or: [python, pyi]
+
+  # Shell scripts
+- repo: https://github.com/shellcheck-py/shellcheck-py
+  rev: v0.10.0.1
+  hooks:
+  - id: shellcheck
+    args: [--severity=warning]
+
+  # YAML formatting
+- repo: https://github.com/macisamuele/language-formatters-pre-commit-hooks
+  rev: v2.15.0
+  hooks:
+  - id: pretty-format-yaml
+    args: [--autofix, --indent, '2']
+
+  # Markdown formatting
+- repo: https://github.com/executablebooks/mdformat
+  rev: 0.7.21
+  hooks:
+  - id: mdformat
+    args: [--wrap, '100']
+    additional_dependencies:
+    - mdformat-gfm
+    - mdformat-black
+
+  # Secret detection
+- repo: https://github.com/Yelp/detect-secrets
+  rev: v1.5.0
+  hooks:
+  - id: detect-secrets
+    args: ['--baseline', '.secrets.baseline']
+    exclude: package.lock.json

isage_vdb-0.2.0.10/.vscode/settings.json

@@ -0,0 +1,3 @@
+{
+    "cmake.configureOnOpen": false
+}

isage_vdb-0.2.0.10/CHANGELOG.md

@@ -0,0 +1,38 @@
+# Changelog
+
+All notable changes to isage-vdb will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.2.0.10] - 2026-05-27
+
+### Fixed
+- Preserved sage-anns adapter metadata and local ID state across `save()` / `load()` using an adapter sidecar file
+- Restored adapter state safely by validating dimension, metric, and algorithm before loading external ANN indexes
+- Normalized sage-anns adapter metadata keys and values to strings before writing through `MetadataStore`
+- Corrected the sage-anns integration example so it reports inserted vector count instead of reusing `dimension`
+
+### Changed
+- Installed Python adapter sources into built wheel/sdist artifacts so `sagevdb.sage_anns` ships with published packages
+- Moved `isage-anns` to an explicit optional `sage-anns` extra and aligned the minimum version with `isage-anns>=0.2.0`
+- Clarified documentation around the native C++ ANNS registry boundary versus the optional Python `backend="sage-anns"` adapter path
+
+## [0.1.10] - 2026-02-14
+
+### Fixed
+- **CRITICAL**: Added `faiss-cpu>=1.7.0` to core dependencies in pyproject.toml
+  - libsage_vdb.so is compiled with FAISS support and requires libfaiss.so at runtime
+  - Previously faiss-cpu was optional, causing OSError when importing sagevdb
+- Improved error message when libfaiss.so is missing during module import
+
+### Changed
+- Removed redundant `faiss` from optional dependencies (now in core)
+
+## [0.1.9] - 2026-02-13
+
+### Added
+- Initial release with FAISS-compatible API
+- Support for multiple index types (FLAT, IVF, HNSW)
+- Metadata filtering and hybrid search
+- Persistent storage (save/load)