cyborgdb-migrate 0.1.0__tar.gz

cyborgdb_migrate-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Cyborg Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

cyborgdb_migrate-0.1.0/PKG-INFO

@@ -0,0 +1,50 @@
+Metadata-Version: 2.4
+Name: cyborgdb-migrate
+Version: 0.1.0
+Summary: TUI wizard and CLI for migrating vector data into CyborgDB
+Author: Cyborg Inc.
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/cyborginc/cyborgdb-migrate
+Project-URL: Repository, https://github.com/cyborginc/cyborgdb-migrate
+Project-URL: Issues, https://github.com/cyborginc/cyborgdb-migrate/issues
+Keywords: vector-database,migration,cyborgdb,pinecone,qdrant,weaviate,chromadb,milvus,embeddings,tui
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Database
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+License-File: LICENSE
+Requires-Dist: textual>=0.86.0
+Requires-Dist: cyborgdb
+Requires-Dist: numpy>=1.21.0
+Requires-Dist: tomli>=1.1.0; python_version < "3.11"
+Provides-Extra: pinecone
+Requires-Dist: pinecone; extra == "pinecone"
+Provides-Extra: qdrant
+Requires-Dist: qdrant-client; extra == "qdrant"
+Provides-Extra: weaviate
+Requires-Dist: weaviate-client; extra == "weaviate"
+Provides-Extra: chromadb
+Requires-Dist: chromadb; extra == "chromadb"
+Provides-Extra: milvus
+Requires-Dist: pymilvus; extra == "milvus"
+Provides-Extra: syntax
+Requires-Dist: tree-sitter; extra == "syntax"
+Requires-Dist: tree-sitter-python; extra == "syntax"
+Provides-Extra: all
+Requires-Dist: pinecone; extra == "all"
+Requires-Dist: qdrant-client; extra == "all"
+Requires-Dist: weaviate-client; extra == "all"
+Requires-Dist: chromadb; extra == "all"
+Requires-Dist: pymilvus; extra == "all"
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: pytest-asyncio; extra == "dev"
+Requires-Dist: textual-dev; extra == "dev"
+Dynamic: license-file

cyborgdb_migrate-0.1.0/README.md

@@ -0,0 +1,159 @@
+# cyborgdb-migrate
+
+A TUI wizard and CLI for migrating vector data from popular vector databases into [CyborgDB](https://cyborgdb.com) — the encrypted vector database.
+
+## Features
+
+- **Interactive TUI** — step-by-step wizard powered by [Textual](https://textual.textualize.io/)
+- **Headless CLI** — non-interactive mode for scripts and CI/CD pipelines
+- **5 source connectors** — Pinecone, Qdrant, Weaviate, ChromaDB, Milvus
+- **Encrypted at rest** — every index is AES-encrypted with a key you control
+- **Checkpoint & resume** — automatically saves progress; resume interrupted migrations
+- **Spot-check verification** — post-migration vector and metadata integrity checks
+- **Double-buffered I/O** — overlaps extraction and upsert for maximum throughput
+
+## Installation
+
+```bash
+pip install cyborgdb-migrate
+```
+
+Install with support for your source database:
+
+```bash
+# Individual sources
+pip install "cyborgdb-migrate[pinecone]"
+pip install "cyborgdb-migrate[qdrant]"
+pip install "cyborgdb-migrate[weaviate]"
+pip install "cyborgdb-migrate[chromadb]"
+pip install "cyborgdb-migrate[milvus]"
+
+# All sources at once
+pip install "cyborgdb-migrate[all]"
+```
+
+Optional syntax highlighting for the summary code snippet:
+
+```bash
+pip install "cyborgdb-migrate[syntax]"
+```
+
+## Quick Start
+
+### Interactive (TUI)
+
+```bash
+cyborgdb-migrate
+```
+
+The wizard walks you through:
+
+1. Selecting a source database
+2. Entering credentials and picking an index/collection
+3. Connecting to CyborgDB
+4. Creating or selecting a destination index
+5. Running the migration with live progress
+6. Viewing verification results and a Python quickstart snippet
+
+### Headless (CLI)
+
+Create a TOML config file (see [`example-config.toml`](example-config.toml)):
+
+```toml
+[source]
+type = "pinecone"
+api_key = "${PINECONE_API_KEY}"
+index = "my-index"
+
+[destination]
+host = "http://localhost:8000"
+api_key = "${CYBORGDB_API_KEY}"
+create_index = true
+index_name = "my-cyborgdb-index"
+index_type = "ivfflat"
+
+[options]
+batch_size = 200
+checkpoint_every = 10
+```
+
+Run the migration:
+
+```bash
+cyborgdb-migrate --config migration.toml
+```
+
+Resume an interrupted migration:
+
+```bash
+cyborgdb-migrate --config migration.toml --resume
+```
+
+## Configuration Reference
+
+### `[source]`
+
+| Key | Required | Description |
+|-----|----------|-------------|
+| `type` | Yes | Source database: `pinecone`, `qdrant`, `weaviate`, `chromadb`, `milvus` |
+| `index` | Yes | Index or collection name to migrate from |
+| `namespace` | No | Namespace/partition to migrate (Pinecone, Milvus) |
+| *(other keys)* | Varies | Passed as credentials to the source connector (e.g. `api_key`, `host`) |
+
+### `[destination]`
+
+| Key | Required | Description |
+|-----|----------|-------------|
+| `host` | Yes | CyborgDB server URL |
+| `api_key` | Yes | CyborgDB API key |
+| `index_name` | Yes | Destination index name |
+| `create_index` | No | `true` (default) to create a new index, `false` to use existing |
+| `index_type` | No | `ivfflat` (default) or `ivfpq` |
+| `index_key` | No | Hex-encoded encryption key (for existing indexes) |
+| `key_file` | No | Path to encryption key file (for existing indexes) |
+
+### `[options]`
+
+| Key | Default | Description |
+|-----|---------|-------------|
+| `batch_size` | `100` | Vectors per batch |
+| `checkpoint_every` | `10` | Save checkpoint every N batches |
+| `spot_check_per_batch` | `4` | Vectors sampled per batch for verification |
+
+Environment variables can be referenced as `${VAR_NAME}` anywhere in the config.
+
+## CLI Options
+
+```
+cyborgdb-migrate [OPTIONS]
+
+Options:
+  --config FILE       TOML config file for non-interactive mode
+  --resume            Resume from checkpoint (requires --config)
+  --batch-size INT    Override batch size (default: 100)
+  --log-file FILE     Log file path (default: ./cyborgdb-migrate.log)
+  --quiet             Minimal output (non-interactive only)
+  --version           Show version and exit
+```
+
+## Supported Sources
+
+| Source | Extras | Notes |
+|--------|--------|-------|
+| [Pinecone](https://www.pinecone.io/) | `pinecone` | Supports namespaces |
+| [Qdrant](https://qdrant.tech/) | `qdrant` | Scroll-based pagination |
+| [Weaviate](https://weaviate.io/) | `weaviate` | Supports named vectors |
+| [ChromaDB](https://www.trychroma.com/) | `chromadb` | Local and remote modes |
+| [Milvus](https://milvus.io/) | `milvus` | Supports partitions, content field heuristic |
+
+## Exit Codes
+
+| Code | Meaning |
+|------|---------|
+| `0` | Success |
+| `1` | Configuration or connection error |
+| `2` | Migration completed but spot-check verification failed |
+
+## License
+
+[MIT](LICENSE)

cyborgdb_migrate-0.1.0/pyproject.toml

@@ -0,0 +1,88 @@
+[build-system]
+requires = ["setuptools>=68.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "cyborgdb-migrate"
+version = "0.1.0"
+description = "TUI wizard and CLI for migrating vector data into CyborgDB"
+requires-python = ">=3.10"
+license = "MIT"
+authors = [
+    {name = "Cyborg Inc."},
+]
+keywords = [
+    "vector-database",
+    "migration",
+    "cyborgdb",
+    "pinecone",
+    "qdrant",
+    "weaviate",
+    "chromadb",
+    "milvus",
+    "embeddings",
+    "tui",
+]
+classifiers = [
+    "Environment :: Console",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
+    "Topic :: Database",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = [
+    "textual>=0.86.0",
+    "cyborgdb",
+    "numpy>=1.21.0",
+    "tomli>=1.1.0;python_version<'3.11'",
+]
+
+[project.optional-dependencies]
+pinecone = ["pinecone"]
+qdrant = ["qdrant-client"]
+weaviate = ["weaviate-client"]
+chromadb = ["chromadb"]
+milvus = ["pymilvus"]
+syntax = [
+    "tree-sitter",
+    "tree-sitter-python",
+]
+all = [
+    "pinecone",
+    "qdrant-client",
+    "weaviate-client",
+    "chromadb",
+    "pymilvus",
+]
+dev = [
+    "pytest",
+    "pytest-asyncio",
+    "textual-dev",
+]
+
+[project.scripts]
+cyborgdb-migrate = "cyborgdb_migrate.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/cyborginc/cyborgdb-migrate"
+Repository = "https://github.com/cyborginc/cyborgdb-migrate"
+Issues = "https://github.com/cyborginc/cyborgdb-migrate/issues"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.ruff]
+line-length = 100
+target-version = "py310"
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "W"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+asyncio_mode = "auto"

cyborgdb_migrate-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

cyborgdb_migrate-0.1.0/src/cyborgdb_migrate/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.1.0"

cyborgdb_migrate-0.1.0/src/cyborgdb_migrate/app.py

@@ -0,0 +1,58 @@
+import time
+
+from textual.app import App
+from textual.theme import Theme
+
+from cyborgdb_migrate.models import MigrationState
+from cyborgdb_migrate.screens.welcome import WelcomeScreen
+
+MIN_WIDTH = 80
+MIN_HEIGHT = 24
+
+CYBORGDB_THEME = Theme(
+    name="cyborgdb",
+    primary="#217684",
+    secondary="#56D3DB",
+    accent="#38C3EE",
+    surface="#1a2a2e",
+    background="#111d20",
+    panel="#1e3438",
+    warning="#f5a623",
+    error="#e74c3c",
+    success="#2ecc71",
+    dark=True,
+)
+
+
+class MigrateApp(App):
+    """CyborgDB Migration Wizard TUI application."""
+
+    CSS_PATH = "theme.css"
+    TITLE = "CyborgDB Migration Wizard"
+
+    BINDINGS = [
+        ("ctrl+q", "", ""),  # unbind default quit
+    ]
+
+    _ctrl_c_time: float = 0.0
+
+    def __init__(self, state: MigrationState | None = None) -> None:
+        super().__init__()
+        self.state = state or MigrationState()
+
+    def action_quit(self) -> None:
+        """Disable the default quit action."""
+
+    def _on_key(self, event) -> None:
+        if event.key == "ctrl+c":
+            now = time.monotonic()
+            if now - self._ctrl_c_time < 1.0:
+                self.exit()
+            else:
+                self._ctrl_c_time = now
+                self.notify("Press Ctrl+C again to quit", timeout=1)
+
+    def on_mount(self) -> None:
+        self.register_theme(CYBORGDB_THEME)
+        self.theme = "cyborgdb"
+        self.push_screen(WelcomeScreen(self.state))

cyborgdb_migrate-0.1.0/src/cyborgdb_migrate/checkpoint.py

@@ -0,0 +1,77 @@
+from __future__ import annotations
+
+import json
+import os
+import tempfile
+from dataclasses import asdict, dataclass
+from datetime import datetime, timezone
+from pathlib import Path
+
+CHECKPOINT_DIR = "./cyborgdb-migrate-checkpoints"
+
+
+@dataclass
+class CheckpointData:
+    version: int = 1
+    source_type: str = ""
+    source_index: str = ""
+    dest_index: str = ""
+    namespace: str | None = None
+    cursor: str | None = None
+    vectors_migrated: int = 0
+    vectors_total: int = 0
+    started_at: str = ""
+    updated_at: str = ""
+    batch_size: int = 100
+    batches_completed: int = 0
+
+
+def checkpoint_path(source_type: str, source_index: str, dest_index: str) -> Path:
+    """Return the checkpoint file path for a given migration."""
+    safe_name = f"{source_type}_{source_index}_{dest_index}.json"
+    # Sanitize filename
+    safe_name = safe_name.replace("/", "_").replace("\\", "_")
+    return Path(CHECKPOINT_DIR) / safe_name
+
+
+def save_checkpoint(data: CheckpointData) -> Path:
+    """Atomically save checkpoint data to disk."""
+    path = checkpoint_path(data.source_type, data.source_index, data.dest_index)
+    path.parent.mkdir(parents=True, exist_ok=True)
+
+    data.updated_at = datetime.now(timezone.utc).isoformat()
+    if not data.started_at:
+        data.started_at = data.updated_at
+
+    # Atomic write: write to temp file then rename
+    fd, tmp_path = tempfile.mkstemp(dir=path.parent, suffix=".tmp")
+    try:
+        with os.fdopen(fd, "w") as f:
+            json.dump(asdict(data), f, indent=2)
+        os.replace(tmp_path, path)
+    except Exception:
+        # Clean up temp file on failure
+        try:
+            os.unlink(tmp_path)
+        except OSError:
+            pass
+        raise
+
+    return path
+
+
+def load_checkpoint(source_type: str, source_index: str, dest_index: str) -> CheckpointData | None:
+    """Load checkpoint data if it exists. Returns None if no checkpoint found."""
+    path = checkpoint_path(source_type, source_index, dest_index)
+    if not path.exists():
+        return None
+    with open(path) as f:
+        raw = json.load(f)
+    return CheckpointData(**raw)
+
+
+def delete_checkpoint(source_type: str, source_index: str, dest_index: str) -> None:
+    """Delete a checkpoint file if it exists."""
+    path = checkpoint_path(source_type, source_index, dest_index)
+    if path.exists():
+        path.unlink()

cyborgdb_migrate-0.1.0/src/cyborgdb_migrate/cli.py

@@ -0,0 +1,225 @@
+import argparse
+import sys
+
+from cyborgdb_migrate import __version__
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        prog="cyborgdb-migrate",
+        description="Migrate vector data from other databases into CyborgDB",
+    )
+    parser.add_argument(
+        "--config", metavar="FILE",
+        help="TOML config file for non-interactive mode",
+    )
+    parser.add_argument(
+        "--resume", action="store_true",
+        help="Resume from checkpoint (non-interactive only)",
+    )
+    parser.add_argument(
+        "--batch-size", type=int, default=100,
+        help="Vectors per batch (default: 100)",
+    )
+    parser.add_argument(
+        "--log-file", metavar="FILE",
+        default="./cyborgdb-migrate.log",
+        help="Log file path (default: ./cyborgdb-migrate.log)",
+    )
+    parser.add_argument(
+        "--quiet", action="store_true",
+        help="Minimal output (non-interactive only)",
+    )
+    parser.add_argument("--version", action="version", version=f"%(prog)s {__version__}")
+
+    args = parser.parse_args()
+
+    if args.config:
+        setup_logging(args.log_file)
+        run_headless(args.config, args.batch_size, args.resume, args.log_file, args.quiet)
+    else:
+        if args.resume:
+            print("Error: --resume is only supported with --config", file=sys.stderr)
+            raise SystemExit(1)
+        setup_logging(args.log_file)
+        from cyborgdb_migrate.app import MigrateApp
+        from cyborgdb_migrate.models import MigrationState
+
+        state = MigrationState()
+        state.batch_size = args.batch_size
+        app = MigrateApp(state)
+        app.run()
+
+
+def setup_logging(log_file: str) -> None:
+    import logging
+
+    logging.basicConfig(
+        filename=log_file,
+        level=logging.INFO,
+        format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
+        datefmt="%Y-%m-%d %H:%M:%S",
+    )
+
+
+def run_headless(
+    config_path: str,
+    batch_size: int,
+    resume: bool,
+    log_file: str,
+    quiet: bool,
+) -> None:
+    import logging
+    import threading
+
+    from rich.console import Console
+    from rich.progress import BarColumn, Progress, SpinnerColumn, TextColumn, TimeElapsedColumn
+
+    from cyborgdb_migrate.config import load_config
+    from cyborgdb_migrate.destination import CyborgDestination
+    from cyborgdb_migrate.engine import MigrationEngine
+    from cyborgdb_migrate.sources import SOURCE_REGISTRY
+
+    logger = logging.getLogger("cyborgdb_migrate.headless")
+    console = Console(stderr=True)
+
+    config = load_config(config_path)
+
+    # Override batch_size from CLI if provided and different from default
+    if batch_size != 100:
+        config.batch_size = batch_size
+
+    # Resolve source
+    source_type = config.source_type
+    source_cls = None
+    for name, cls in SOURCE_REGISTRY.items():
+        if name.lower() == source_type.lower() or source_type.lower() in name.lower():
+            source_cls = cls
+            break
+    if source_cls is None:
+        console.print(f"[red]Unknown source type: {source_type}[/red]")
+        raise SystemExit(1)
+
+    source = source_cls()
+    source.configure(config.source_credentials)
+    if not quiet:
+        console.print(f"Connecting to {source.name()}...")
+    source.connect()
+    if not quiet:
+        console.print(f"[green]Connected to {source.name()}[/green]")
+
+    # Inspect source
+    source_info = source.inspect(config.source_index)
+    if not quiet:
+        console.print(
+            f"Source: {source_info.index_or_collection_name} "
+            f"({source_info.dimension}d, {source_info.vector_count:,} vectors)"
+        )
+
+    # Connect to CyborgDB
+    destination = CyborgDestination()
+    destination.connect(config.destination_host, config.destination_api_key)
+    if not quiet:
+        console.print("[green]Connected to CyborgDB[/green]")
+
+    # Set up index
+    if config.create_index:
+        from cyborgdb import Client
+
+        index_key = Client.generate_key(save=False)
+        if not quiet:
+            console.print(f"Generated encryption key (hex): {index_key.hex()}")
+
+        from cyborgdb_migrate.destination import compute_n_lists
+
+        n_lists = compute_n_lists(source_info.vector_count)
+        destination.create_index(
+            name=config.index_name,
+            dimension=source_info.dimension,
+            index_type=config.index_type or "ivfflat",
+            index_key=index_key,
+            n_lists=n_lists,
+            metric=source_info.metric,
+        )
+    else:
+        if config.index_key:
+            index_key = _decode_key(config.index_key)
+        elif config.key_file:
+            with open(config.key_file) as f:
+                index_key = _decode_key(f.read().strip())
+        else:
+            console.print("[red]No index key provided for existing index[/red]")
+            raise SystemExit(1)
+        destination.load_index(config.index_name, index_key)
+
+        # Validate dimension match for existing index
+        dest_dim = destination.get_index_dimension()
+        if dest_dim is not None and dest_dim != source_info.dimension:
+            console.print(
+                f"[red]Dimension mismatch: source has {source_info.dimension}d, "
+                f"destination has {dest_dim}d[/red]"
+            )
+            raise SystemExit(1)
+
+    # Run migration
+    cancel_event = threading.Event()
+
+    def on_progress(update):
+        pass  # Progress handled by rich progress bar below
+
+    engine = MigrationEngine(
+        source=source,
+        destination=destination,
+        source_info=source_info,
+        batch_size=config.batch_size,
+        checkpoint_every=config.checkpoint_every,
+        spot_check_per_batch=config.spot_check_per_batch,
+        on_progress=on_progress,
+        cancel_event=cancel_event,
+    )
+
+    if quiet:
+        result = engine.run(
+            namespace=config.source_namespace,
+            resume=resume,
+        )
+    else:
+        with Progress(
+            SpinnerColumn(),
+            TextColumn("[progress.description]{task.description}"),
+            BarColumn(),
+            TextColumn("[progress.percentage]{task.percentage:>3.0f}%"),
+            TimeElapsedColumn(),
+            console=console,
+        ) as progress:
+            task = progress.add_task("Migrating...", total=source_info.vector_count)
+
+            def progress_callback(update):
+                progress.update(task, completed=update.vectors_migrated)
+
+            engine.on_progress = progress_callback
+            result = engine.run(
+                namespace=config.source_namespace,
+                resume=resume,
+            )
+
+    if not quiet:
+        console.print("\n[green]Migration complete![/green]")
+        console.print(f"  Vectors: {result.vectors_migrated:,} / {result.vectors_expected:,}")
+        console.print(f"  Duration: {result.duration_seconds:.1f}s")
+        console.print(f"  Spot check: {'PASSED' if result.spot_check_passed else 'FAILED'}")
+        console.print(f"  Details: {result.spot_check_details}")
+
+    if not result.spot_check_passed:
+        logger.warning("Spot check failed: %s", result.spot_check_details)
+        raise SystemExit(2)
+
+
+def _decode_key(value: str) -> bytes:
+    """Decode a key from hex, falling back to base64 for backwards compatibility."""
+    try:
+        return bytes.fromhex(value)
+    except ValueError:
+        import base64
+
+        return base64.b64decode(value)