openviking 0.3.6__tar.gz → 0.3.6.dev45__tar.gz

openviking-0.3.6.dev45/CONTRIBUTING.md

@@ -0,0 +1,564 @@
+# Contributing Guide
+
+Thank you for your interest in OpenViking! We welcome contributions of all kinds:
+
+- Bug reports
+- Feature requests
+- Documentation improvements
+- Code contributions
+
+---
+
+## Development Setup
+
+### Prerequisites
+
+- **Python**: 3.10+
+- **Go**: 1.22+ (Required for building AGFS components from source)
+- **Rust**: 1.91.1+ (Required for source builds because the bundled `ov` CLI is built during packaging)
+- **C++ Compiler**: GCC 9+ or Clang 11+ (Required for building core extensions, must support C++17)
+- **CMake**: 3.12+
+
+#### Platform-Specific Native Build Tools
+
+- **Linux**: Install `build-essential`; some environments may also require `pkg-config`
+- **macOS**: Install Xcode Command Line Tools (`xcode-select --install`)
+- **Windows**: Install CMake and MinGW for local native builds
+
+#### Supported Platforms (Pre-compiled Wheels)
+
+OpenViking provides pre-compiled **Wheel** packages for the following environments:
+
+- **Windows**: x86_64
+- **macOS**: x86_64, arm64 (Apple Silicon)
+- **Linux**: x86_64, arm64 (manylinux)
+
+For other platforms (e.g., FreeBSD), the package will be automatically compiled from source during installation via `pip`. Ensure you have the [Prerequisites](#prerequisites) installed.
+
+### 1. Fork and Clone
+
+```bash
+git clone https://github.com/YOUR_USERNAME/openviking.git
+cd openviking
+```
+
+### 2. Install Dependencies
+
+We recommend using `uv` for Python environment management:
+
+```bash
+# Install uv (if not installed)
+curl -LsSf https://astral.sh/uv/install.sh | sh
+
+# Sync dependencies and create virtual environment
+uv sync --all-extras
+source .venv/bin/activate  # Linux/macOS
+# or .venv\Scripts\activate  # Windows
+```
+
+#### Local Development & Native Rebuilds
+
+OpenViking defaults to `binding-client` mode for AGFS, which requires pre-built native artifacts. If you modify the **AGFS (Go)** code, the bundled **Rust CLI**, or the **C++ extensions**, or if the pre-built artifacts are not found, you need to re-compile and re-install them. Run the following command in the project root:
+
+```bash
+uv pip install -e . --force-reinstall
+```
+
+This command ensures that `setup.py` is re-executed, triggering rebuilds for AGFS, the bundled `ov` CLI, and the C++ components.
+
+### 3. Configure Environment
+
+Create a configuration file `~/.openviking/ov.conf`:
+
+```json
+{
+  "embedding": {
+    "dense": {
+      "provider": "volcengine",
+      "api_key": "your-api-key",
+      "model": "doubao-embedding-vision-251215",
+      "api_base": "https://ark.cn-beijing.volces.com/api/v3",
+      "dimension": 1024,
+      "input": "multimodal"
+    }
+  },
+  "vlm": {
+    "api_key": "your-api-key",
+    "model": "doubao-seed-2-0-pro-260215",
+    "api_base": "https://ark.cn-beijing.volces.com/api/v3"
+  }
+}
+```
+
+Set the environment variable:
+
+```bash
+export OPENVIKING_CONFIG_FILE=~/.openviking/ov.conf
+```
+
+### 4. Verify Installation
+
+```python
+import asyncio
+import openviking as ov
+
+async def main():
+    client = ov.AsyncOpenViking(path="./test_data")
+    await client.initialize()
+    print("OpenViking initialized successfully!")
+    await client.close()
+
+asyncio.run(main())
+```
+
+### 5. Build Rust CLI (Optional)
+
+The Rust CLI (`ov`) provides a high-performance command-line client for interacting with OpenViking Server.
+
+Even if you do not plan to use `ov` directly, the Rust toolchain is still required when building OpenViking from source because packaging also builds the bundled CLI binary.
+
+```bash
+# Build and install from source
+cargo install --path crates/ov_cli
+
+# Or use the quick install script (downloads pre-built binary)
+curl -fsSL https://raw.githubusercontent.com/volcengine/OpenViking/main/crates/ov_cli/install.sh | bash
+```
+
+After installation, run `ov --help` to see all available commands. CLI connection config goes in `~/.openviking/ovcli.conf`.
+
+---
+
+## Project Structure
+
+```
+openviking/
+├── pyproject.toml        # Project configuration
+├── Cargo.toml            # Rust workspace configuration
+├── third_party/          # Third-party dependencies
+│   └── agfs/             # AGFS filesystem
+│
+├── openviking/           # Python SDK
+│   ├── async_client.py   # AsyncOpenViking client
+│   ├── sync_client.py    # SyncOpenViking client
+│   ├── client/           # Local and HTTP client implementations
+│   ├── console/          # Standalone console UI and proxy service
+│   ├── core/             # Core data models and directory abstractions
+│   ├── message/          # Session message and part models
+│   ├── models/           # Embedding and VLM backends
+│   ├── parse/            # Resource parsers and detectors
+│   ├── resource/         # Resource processing and watch management
+│   ├── retrieve/         # Retrieval system
+│   ├── server/           # HTTP server
+│   ├── service/          # Shared service layer
+│   ├── session/          # Session management and compression
+│   ├── storage/          # Storage layer
+│   ├── telemetry/        # Operation telemetry
+│   ├── trace/            # Trace and runtime tracing helpers
+│   ├── utils/            # Utilities and configuration helpers
+│   └── prompts/          # Prompt templates
+│
+├── crates/               # Rust components
+│   └── ov_cli/           # Rust CLI client
+│       ├── src/          # CLI source code
+│       └── install.sh    # Quick install script
+│
+├── src/                  # C++ extension sources (Python abi3)
+│
+├── tests/                # Test suite
+│   ├── client/           # Client tests
+│   ├── console/          # Console tests
+│   ├── core/             # Core logic tests
+│   ├── parse/            # Parser tests
+│   ├── resource/         # Resource processing tests
+│   ├── retrieve/         # Retrieval tests
+│   ├── server/           # Server tests
+│   ├── service/          # Service layer tests
+│   ├── session/          # Session tests
+│   ├── storage/          # Storage tests
+│   ├── telemetry/        # Telemetry tests
+│   ├── vectordb/         # Vector database tests
+│   └── integration/      # End-to-end tests
+│
+└── docs/                 # Documentation
+    ├── en/               # English docs
+    └── zh/               # Chinese docs
+```
+
+---
+
+## Code Style
+
+We use the following tools to maintain code consistency:
+
+| Tool | Purpose | Config |
+|------|---------|--------|
+| **Ruff** | Linting, Formatting, Import sorting | `pyproject.toml` |
+| **mypy** | Type checking | `pyproject.toml` |
+
+### Automated Checks (Recommended)
+
+We use [pre-commit](https://pre-commit.com/) to automatically run these checks before every commit. This ensures your code always meets the standards without manual effort.
+
+1. **Install pre-commit**:
+   ```bash
+   pip install pre-commit
+   ```
+
+2. **Install the git hooks**:
+   ```bash
+   pre-commit install
+   ```
+
+Now, `ruff` (check & format) will run automatically when you run `git commit`. If any check fails, it may automatically fix the file. You just need to add the changes and commit again.
+
+### Running Checks
+
+```bash
+# Format code
+ruff format openviking/
+
+# Lint
+ruff check openviking/
+
+# Type check
+mypy openviking/
+```
+
+### Style Guidelines
+
+1. **Line width**: 100 characters
+2. **Indentation**: 4 spaces
+3. **Strings**: Prefer double quotes
+4. **Type hints**: Encouraged but not required
+5. **Docstrings**: Required for public APIs (1-2 lines max)
+
+---
+
+## Testing
+
+### Running Tests
+
+```bash
+# Run all tests
+pytest
+
+# Run specific test module
+pytest tests/client/ -v
+pytest tests/server/ -v
+pytest tests/parse/ -v
+
+# Run specific test file
+pytest tests/client/test_lifecycle.py
+
+# Run specific test
+pytest tests/client/test_lifecycle.py::TestClientInitialization::test_initialize_success
+
+# Run by keyword
+pytest -k "search" -v
+
+# Run with coverage
+pytest --cov=openviking --cov-report=term-missing
+```
+
+### Writing Tests
+
+Tests are organized in subdirectories under `tests/`. The project uses `asyncio_mode = "auto"`, so async tests do **not** need the `@pytest.mark.asyncio` decorator:
+
+```python
+# tests/client/test_example.py
+from openviking import AsyncOpenViking
+
+
+class TestAsyncOpenViking:
+    async def test_initialize(self, uninitialized_client: AsyncOpenViking):
+        await uninitialized_client.initialize()
+        assert uninitialized_client._service is not None
+        await uninitialized_client.close()
+
+    async def test_add_resource(self, client: AsyncOpenViking, sample_markdown_file):
+        result = await client.add_resource(
+            path=str(sample_markdown_file),
+            reason="test document"
+        )
+        assert "root_uri" in result
+        assert result["root_uri"].startswith("viking://")
+```
+
+Common fixtures are defined in `tests/conftest.py`, including `client` (initialized `AsyncOpenViking`), `uninitialized_client`, `temp_dir`, `sample_markdown_file`, and more.
+
+---
+
+## Contribution Workflow
+
+### 1. Create a Branch
+
+```bash
+git checkout main
+git pull origin main
+git checkout -b feature/your-feature-name
+```
+
+Branch naming conventions:
+- `feature/xxx` - New features
+- `fix/xxx` - Bug fixes
+- `docs/xxx` - Documentation updates
+- `refactor/xxx` - Code refactoring
+
+### 2. Make Changes
+
+- Follow code style guidelines
+- Add tests for new functionality
+- Update documentation as needed
+
+### 3. Commit Changes
+
+```bash
+git add .
+git commit -m "feat: add new parser for xlsx files"
+```
+
+### 4. Push and Create PR
+
+```bash
+git push origin feature/your-feature-name
+```
+
+Then create a Pull Request on GitHub.
+
+---
+
+## Commit Convention
+
+We follow [Conventional Commits](https://www.conventionalcommits.org/):
+
+```
+<type>(<scope>): <subject>
+
+<body>
+
+<footer>
+```
+
+### Types
+
+| Type | Description |
+|------|-------------|
+| `feat` | New feature |
+| `fix` | Bug fix |
+| `docs` | Documentation |
+| `style` | Code style (no logic change) |
+| `refactor` | Code refactoring |
+| `perf` | Performance improvement |
+| `test` | Tests |
+| `chore` | Build/tooling |
+
+### Examples
+
+```bash
+# New feature
+git commit -m "feat(parser): add support for xlsx files"
+
+# Bug fix
+git commit -m "fix(retrieval): fix score calculation in rerank"
+
+# Documentation
+git commit -m "docs: update quick start guide"
+
+# Refactoring
+git commit -m "refactor(storage): simplify interface methods"
+```
+
+---
+
+## Pull Request Guidelines
+
+### PR Title
+
+Use the same format as commit messages.
+
+### PR Description Template
+
+```markdown
+## Summary
+
+Brief description of the changes and their purpose.
+
+## Type of Change
+
+- [ ] New feature (feat)
+- [ ] Bug fix (fix)
+- [ ] Documentation (docs)
+- [ ] Refactoring (refactor)
+- [ ] Other
+
+## Testing
+
+Describe how to test these changes:
+- [ ] Unit tests pass
+- [ ] Manual testing completed
+
+## Related Issues
+
+- Fixes #123
+- Related to #456
+
+## Checklist
+
+- [ ] Code follows project style guidelines
+- [ ] Tests added for new functionality
+- [ ] Documentation updated (if needed)
+- [ ] All tests pass
+```
+
+---
+
+## CI/CD Workflows
+
+We use **GitHub Actions** for Continuous Integration and Continuous Deployment. Our workflows are designed to be modular and tiered.
+
+### 1. Automatic Workflows
+
+| Event | Workflow | Description |
+|-------|----------|-------------|
+| **Pull Request** | `pr.yml` | Runs **Lint** (Ruff, Mypy) and **Test Lite** (Integration tests on Linux + Python 3.10). Provides fast feedback for contributors. (Displayed as **01. Pull Request Checks**) |
+| **Push to Main** | `ci.yml` | Runs **Test Full** (All OS: Linux/Win/Mac, All Py versions: 3.10-3.14) and **CodeQL** (Security scan). Ensures main branch stability. (Displayed as **02. Main Branch Checks**) |
+| **Release Published** | `release.yml` | Triggered when you create a Release on GitHub. Automatically builds source distribution and wheels, determines version from Git Tag, and publishes to **PyPI**. (Displayed as **03. Release**) |
+| **Weekly Cron** | `schedule.yml` | Runs **CodeQL** security scan every Sunday. (Displayed as **04. Weekly Security Scan**) |
+
+Other repository workflows also exist for PR review automation, Docker image builds, and Rust CLI packaging.
+
+### 2. Manual Trigger Workflows
+
+Maintainers can manually trigger the following workflows from the "Actions" tab to perform specific tasks or debug issues.
+
+#### A. Lint Checks (`11. _Lint Checks`)
+Runs code style checks (Ruff) and type checks (Mypy). No arguments required.
+
+> **Tip**: It is recommended to install [pre-commit](https://pre-commit.com/) locally to run these checks automatically before committing (see [Automated Checks](#automated-checks-recommended) section above).
+
+#### B. Test Suite (Lite) (`12. _Test Suite (Lite)`)
+Runs fast integration tests, supports custom matrix configuration.
+
+*   **Inputs**:
+    *   `os_json`: JSON string array of OS to run on (e.g., `["ubuntu-24.04"]`).
+    *   `python_json`: JSON string array of Python versions (e.g., `["3.10"]`).
+
+#### C. Test Suite (Full) (`13. _Test Suite (Full)`)
+Runs the full test suite on all supported platforms (Linux/Mac/Win) and Python versions (3.10-3.14). Supports custom matrix configuration when triggered manually.
+
+*   **Inputs**:
+    *   `os_json`: List of OS to run on (Default: `["ubuntu-24.04", "macos-14", "windows-latest"]`).
+    *   `python_json`: List of Python versions (Default: `["3.10", "3.11", "3.12", "3.13", "3.14"]`).
+
+#### D. Security Scan (`14. _CodeQL Scan`)
+Runs CodeQL security analysis. No arguments required.
+
+#### E. Build Distribution (`15. _Build Distribution`)
+Builds Python wheel packages only, does not publish.
+
+*   **Inputs**:
+    *   `os_json`: List of OS to build on (Default: `["ubuntu-24.04", "ubuntu-24.04-arm", "macos-14", "macos-15-intel", "windows-latest"]`).
+    *   `python_json`: List of Python versions (Default: `["3.10", "3.11", "3.12", "3.13", "3.14"]`).
+    *   `build_sdist`: Whether to build source distribution (Default: `true`).
+    *   `build_wheels`: Whether to build wheel distribution (Default: `true`).
+
+#### F. Publish Distribution (`16. _Publish Distribution`)
+Publishes built packages (requires build Run ID) to PyPI.
+
+*   **Inputs**:
+    *   `target`: Select publish target (`testpypi`, `pypi`, `both`).
+    *   `build_run_id`: Build Workflow Run ID (Required, get it from the Build run URL).
+
+#### G. Manual Release (`03. Release`)
+One-stop build and publish (includes build and publish steps).
+
+> **Version Numbering & Tag Convention**:
+> This project uses `setuptools_scm` to automatically extract version numbers from Git Tags.
+> *   **Tag Naming Convention**: Must follow the `vX.Y.Z` format (e.g., `v0.1.0`, `v1.2.3`). Tags must be compliant with Semantic Versioning.
+> *   **Release Build**: When a Release event is triggered, the version number directly corresponds to the Git Tag (e.g., `v0.1.0` -> `0.1.0`).
+> *   **Manual/Non-Tag Build**: The version number will include the commit count since the last Tag (e.g., `0.1.1.dev3`).
+> *   **Confirm Version**: After the publish job completes, you can see the published version directly in the **Notifications** area at the top of the Workflow **Summary** page (e.g., `Successfully published to PyPI with version: 0.1.8`). You can also verify it in the logs or the **Artifacts** filenames.
+
+*   **Inputs**:
+    *   `target`: Select publish target.
+        *   `none`: Build artifacts only (no publish). Used for verifying build capability.
+        *   `testpypi`: Publish to TestPyPI. Used for Beta testing.
+        *   `pypi`: Publish to official PyPI.
+        *   `both`: Publish to both.
+    *   `os_json`: Build platforms (Default includes all).
+    *   `python_json`: Python versions (Default includes all).
+    *   `build_sdist`: Whether to build source distribution (Default: `true`).
+    *   `build_wheels`: Whether to build wheel distribution (Default: `true`).
+
+> **Publishing Notes**:
+> *   **Test First**: It is strongly recommended to publish to **TestPyPI** for verification before publishing to official PyPI. Note that PyPI and TestPyPI are completely independent environments, and accounts and package data are not shared.
+> *   **No Overwrites**: Neither PyPI nor TestPyPI allow overwriting existing packages with the same name and version. If you need to republish, you must upgrade the version number (e.g., tag a new version or generate a new dev version). If you try to publish an existing version, the workflow will fail.
+
+---
+
+## Issue Guidelines
+
+### Bug Reports
+
+Please provide:
+
+1. **Environment**
+   - Python version
+   - OpenViking version
+   - Operating system
+
+2. **Steps to Reproduce**
+   - Detailed steps
+   - Code snippets
+
+3. **Expected vs Actual Behavior**
+
+4. **Error Logs** (if any)
+
+### Feature Requests
+
+Please describe:
+
+1. **Problem**: What problem are you trying to solve?
+2. **Solution**: What solution do you propose?
+3. **Alternatives**: Have you considered other approaches?
+
+---
+
+## Documentation
+
+Documentation is in Markdown format under `docs/`:
+
+- `docs/en/` - English documentation
+- `docs/zh/` - Chinese documentation
+
+### Documentation Guidelines
+
+1. Code examples must be runnable
+2. Keep documentation in sync with code
+3. Use clear, concise language
+
+---
+
+## Code of Conduct
+
+By participating in this project, you agree to:
+
+1. **Be respectful**: Maintain a friendly and professional attitude
+2. **Be inclusive**: Welcome contributors from all backgrounds
+3. **Be constructive**: Provide helpful feedback
+4. **Stay focused**: Keep discussions technical
+
+---
+
+## Getting Help
+
+If you have questions:
+
+- [GitHub Issues](https://github.com/volcengine/openviking/issues)
+- [Discussions](https://github.com/volcengine/openviking/discussions)
+
+---
+
+Thank you for contributing!