scubiee 0.2.0__tar.gz

scubiee-0.2.0/PKG-INFO

@@ -0,0 +1,185 @@
+Metadata-Version: 2.4
+Name: scubiee
+Version: 0.2.0
+Summary: Scubiee — local Cursor MCP context engine: setup once, ctx init per repo, CUDA/DirectML/CoreML/CPU.
+Project-URL: Homepage, https://github.com/Usmansayed/new-context-engine
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: numpy>=1.26
+Requires-Dist: faiss-cpu>=1.8
+Requires-Dist: requests>=2.31
+Requires-Dist: networkx>=3.2
+Requires-Dist: rapidfuzz>=3.0
+Requires-Dist: psutil>=5.9
+Requires-Dist: tree-sitter<0.26,>=0.23.0
+Requires-Dist: tree-sitter-python<0.26,>=0.23
+Requires-Dist: tree-sitter-javascript<0.26,>=0.23
+Requires-Dist: tree-sitter-typescript<0.25,>=0.23
+Requires-Dist: tree-sitter-json<0.25,>=0.23
+Requires-Dist: tree-sitter-go<0.26,>=0.23
+Requires-Dist: tree-sitter-rust<0.25,>=0.23
+Requires-Dist: tree-sitter-java<0.25,>=0.23
+Requires-Dist: tree-sitter-c<0.25,>=0.23
+Requires-Dist: tree-sitter-cpp<0.25,>=0.23
+Requires-Dist: tree-sitter-ruby<0.25,>=0.23
+Requires-Dist: tree-sitter-c-sharp<0.25,>=0.23
+Requires-Dist: mcp<2,>=1.0
+Requires-Dist: huggingface_hub>=0.20
+Provides-Extra: cpu
+Requires-Dist: fastembed>=0.4; extra == "cpu"
+Requires-Dist: onnxruntime>=1.17; extra == "cpu"
+Provides-Extra: coreml
+Requires-Dist: fastembed>=0.4; extra == "coreml"
+Requires-Dist: onnxruntime>=1.17; extra == "coreml"
+Provides-Extra: dml
+Requires-Dist: fastembed>=0.4; extra == "dml"
+Requires-Dist: onnxruntime-directml>=1.17; extra == "dml"
+Provides-Extra: cuda
+Requires-Dist: fastembed>=0.4; extra == "cuda"
+Requires-Dist: onnxruntime-gpu>=1.17; extra == "cuda"
+Provides-Extra: embed-st
+Requires-Dist: sentence-transformers>=3.0; extra == "embed-st"
+Requires-Dist: torch; extra == "embed-st"
+Requires-Dist: einops; extra == "embed-st"
+Provides-Extra: mcp
+Requires-Dist: mcp<2,>=1.0; extra == "mcp"
+
+# Scubiee
+
+Local Context Engine: Merkle sync → Graphify AST → **mix** compress → CodeRankEmbed (FastEmbed) → TurboQuant/FAISS → Conductor `R_plan`
+
+## Install (no git clone)
+
+Requires **Python 3.10+**. One line installs the package and configures GPU/CPU + Cursor MCP.
+
+**pip**
+
+```bash
+pip install scubiee && ctx setup
+```
+
+**npm** (installs the same Python package, then runs `ctx setup`)
+
+```bash
+npm install -g scubiee
+```
+
+Then `ctx init <repo>` for each codebase, and reload MCP in Cursor (Settings → MCP → refresh).
+
+`ctx setup` picks **CUDA** (NVIDIA), **DirectML** (Windows AMD/Intel), **CoreML** (macOS Metal/ANE), or **CPU**.
+
+From a git checkout (contributors only): `pip install -e .` then `ctx setup`.
+
+## Use
+
+Inside the engine process there are **three managers** only:
+
+| Manager | Role |
+|---------|------|
+| **RuntimeManager** | Workspace lifecycle, publish search generation after sync, serve queries |
+| **IndexManager** | Merkle probe, full index, incremental sync |
+| **ResourceManager** | CPU/RAM admission and embed batching |
+
+A tiny **watchdog** sidecar (not a manager) polls `/health` and restarts the daemon if it crashes. Disable: `CTX_WATCHDOG=0`.
+
+**Install-and-forget check** (isolated port/home):
+
+```powershell
+.\.venv\Scripts\python.exe -u scripts\sim_install_forget.py
+```
+
+## Use
+
+In Cursor: call MCP tools after reload.
+
+CLI (optional):
+
+```powershell
+ctx index .
+ctx search "login validate"
+ctx sync .
+ctx status .
+```
+
+Or: `python -m pipeline index .`
+
+Default pre-embed compression is **`mix`** with a **512-char** cap (locked). Opt out: `CTX_COMPRESS=off`.
+
+## Resource management
+
+Indexing and embedding go through a **Resource Manager** that samples CPU/RAM and adapts throughput:
+
+| Pressure | Behavior |
+|----------|----------|
+| idle | Larger embed batches |
+| normal | Baseline (from accel profile) |
+| busy | Smaller batches + pauses |
+| critical | Background sync deferred; embeds crawl |
+
+```powershell
+ctx resources              # live pressure + hardware
+ctx resources --refresh    # re-detect CPU/RAM/GPU/libs
+ctx init                   # also writes hardware.json + picks fastest ORT backend
+```
+
+Disable: `CTX_RM_DISABLE=1`. Tune: `CTX_RM_MAX_CPU`, `CTX_RM_CRITICAL_CPU`, `CTX_RM_MIN_FREE_RAM_MB`.
+
+## Cursor MCP
+
+After install, MCP tools talk to the local Context Engine service (`CTX_ENGINE_URL`, default `http://127.0.0.1:8765`). The MCP process auto-starts that service if needed.
+
+Tools: `search_code`, `locate_capability`, `grep_code`, `file_outline`, `status`, `sync_index`, `set_repo`, `register_project`.
+
+Dashboard (optional): http://127.0.0.1:8765/dashboard
+
+Advanced (usually unnecessary):
+
+```powershell
+ctx engine status
+ctx engine stop
+ctx engine run .                   # foreground service
+```
+
+`ctx engine start` also starts the watchdog sidecar. Logs: `~/.context-engine/watchdog.log`.
+
+**Registration modes** (dashboard or CLI):
+
+| Mode | Trigger |
+|------|---------|
+| **Automatic** (default) | IDE/MCP open registers the project once, then incremental indexing |
+| **MCP / CLI** | No auto-init; first tool call returns a consent prompt; use `register_project` (optional `always_allow`) or `ctx register` |
+
+```powershell
+ctx settings --mode automatic
+ctx settings --mode mcp_cli
+ctx register . --always-allow --fast
+ctx serve .   # open http://127.0.0.1:8765/dashboard
+```
+
+On open (automatic): resolves a stable **project id**, indexes if needed (`CTX_AUTO_INDEX=1`), then keeps the index fresh every **5 minutes** (changed files only — graph patch + embed together).
+
+## Project data
+
+| Location | Role |
+|----------|------|
+| `<repo>/.context-engine/id.json` | Tiny identity (`project_id`). Prefer gitignore. |
+| `~/.context-engine/prefs.json` | Registration mode + indexing prefs |
+| `~/.context-engine/projects/<project_id>/` | Chunks, graph, merkle, embed cache |
+| `~/.context-engine/registry.json` | Maps `project_id` ↔ paths, `registered`, `always_allow` |
+| `~/.context-engine/vectordb/` | FAISS + TurboQuant collections |
+
+Path moved → id file recovers the store. Id file deleted → registry path recovers. Both gone → new id + reindex.
+
+## Layout
+
+| Path | Role |
+|------|------|
+| `packages/pipeline/` | Index, embed, FAISS+TurboQuant, MCP, sync |
+| `packages/conductor/` | Retrieval fusion (`R_plan`, etc.) |
+| `packages/graphify/` | Bundled AST / graph |
+| `packages/enrich/`, `metadata/`, `repo_ir/` | Chunk enrichment |
+| `scripts/install*` | Setup + MCP install |
+| `tests/` | Unit / integration tests |
+| `docs/` | Design notes + ship notes |
+
+See `VENDOR.md` and `docs/compress-mix-shipped.md`.

scubiee-0.2.0/README.md

@@ -0,0 +1,139 @@
+# Scubiee
+
+Local Context Engine: Merkle sync → Graphify AST → **mix** compress → CodeRankEmbed (FastEmbed) → TurboQuant/FAISS → Conductor `R_plan`
+
+## Install (no git clone)
+
+Requires **Python 3.10+**. One line installs the package and configures GPU/CPU + Cursor MCP.
+
+**pip**
+
+```bash
+pip install scubiee && ctx setup
+```
+
+**npm** (installs the same Python package, then runs `ctx setup`)
+
+```bash
+npm install -g scubiee
+```
+
+Then `ctx init <repo>` for each codebase, and reload MCP in Cursor (Settings → MCP → refresh).
+
+`ctx setup` picks **CUDA** (NVIDIA), **DirectML** (Windows AMD/Intel), **CoreML** (macOS Metal/ANE), or **CPU**.
+
+From a git checkout (contributors only): `pip install -e .` then `ctx setup`.
+
+## Use
+
+Inside the engine process there are **three managers** only:
+
+| Manager | Role |
+|---------|------|
+| **RuntimeManager** | Workspace lifecycle, publish search generation after sync, serve queries |
+| **IndexManager** | Merkle probe, full index, incremental sync |
+| **ResourceManager** | CPU/RAM admission and embed batching |
+
+A tiny **watchdog** sidecar (not a manager) polls `/health` and restarts the daemon if it crashes. Disable: `CTX_WATCHDOG=0`.
+
+**Install-and-forget check** (isolated port/home):
+
+```powershell
+.\.venv\Scripts\python.exe -u scripts\sim_install_forget.py
+```
+
+## Use
+
+In Cursor: call MCP tools after reload.
+
+CLI (optional):
+
+```powershell
+ctx index .
+ctx search "login validate"
+ctx sync .
+ctx status .
+```
+
+Or: `python -m pipeline index .`
+
+Default pre-embed compression is **`mix`** with a **512-char** cap (locked). Opt out: `CTX_COMPRESS=off`.
+
+## Resource management
+
+Indexing and embedding go through a **Resource Manager** that samples CPU/RAM and adapts throughput:
+
+| Pressure | Behavior |
+|----------|----------|
+| idle | Larger embed batches |
+| normal | Baseline (from accel profile) |
+| busy | Smaller batches + pauses |
+| critical | Background sync deferred; embeds crawl |
+
+```powershell
+ctx resources              # live pressure + hardware
+ctx resources --refresh    # re-detect CPU/RAM/GPU/libs
+ctx init                   # also writes hardware.json + picks fastest ORT backend
+```
+
+Disable: `CTX_RM_DISABLE=1`. Tune: `CTX_RM_MAX_CPU`, `CTX_RM_CRITICAL_CPU`, `CTX_RM_MIN_FREE_RAM_MB`.
+
+## Cursor MCP
+
+After install, MCP tools talk to the local Context Engine service (`CTX_ENGINE_URL`, default `http://127.0.0.1:8765`). The MCP process auto-starts that service if needed.
+
+Tools: `search_code`, `locate_capability`, `grep_code`, `file_outline`, `status`, `sync_index`, `set_repo`, `register_project`.
+
+Dashboard (optional): http://127.0.0.1:8765/dashboard
+
+Advanced (usually unnecessary):
+
+```powershell
+ctx engine status
+ctx engine stop
+ctx engine run .                   # foreground service
+```
+
+`ctx engine start` also starts the watchdog sidecar. Logs: `~/.context-engine/watchdog.log`.
+
+**Registration modes** (dashboard or CLI):
+
+| Mode | Trigger |
+|------|---------|
+| **Automatic** (default) | IDE/MCP open registers the project once, then incremental indexing |
+| **MCP / CLI** | No auto-init; first tool call returns a consent prompt; use `register_project` (optional `always_allow`) or `ctx register` |
+
+```powershell
+ctx settings --mode automatic
+ctx settings --mode mcp_cli
+ctx register . --always-allow --fast
+ctx serve .   # open http://127.0.0.1:8765/dashboard
+```
+
+On open (automatic): resolves a stable **project id**, indexes if needed (`CTX_AUTO_INDEX=1`), then keeps the index fresh every **5 minutes** (changed files only — graph patch + embed together).
+
+## Project data
+
+| Location | Role |
+|----------|------|
+| `<repo>/.context-engine/id.json` | Tiny identity (`project_id`). Prefer gitignore. |
+| `~/.context-engine/prefs.json` | Registration mode + indexing prefs |
+| `~/.context-engine/projects/<project_id>/` | Chunks, graph, merkle, embed cache |
+| `~/.context-engine/registry.json` | Maps `project_id` ↔ paths, `registered`, `always_allow` |
+| `~/.context-engine/vectordb/` | FAISS + TurboQuant collections |
+
+Path moved → id file recovers the store. Id file deleted → registry path recovers. Both gone → new id + reindex.
+
+## Layout
+
+| Path | Role |
+|------|------|
+| `packages/pipeline/` | Index, embed, FAISS+TurboQuant, MCP, sync |
+| `packages/conductor/` | Retrieval fusion (`R_plan`, etc.) |
+| `packages/graphify/` | Bundled AST / graph |
+| `packages/enrich/`, `metadata/`, `repo_ir/` | Chunk enrichment |
+| `scripts/install*` | Setup + MCP install |
+| `tests/` | Unit / integration tests |
+| `docs/` | Design notes + ship notes |
+
+See `VENDOR.md` and `docs/compress-mix-shipped.md`.

scubiee-0.2.0/packages/conductor/__init__.py

@@ -0,0 +1,5 @@
+"""Deterministic triple-signal retrieval conductor (Graphify + BM25 + dense)."""
+
+from conductor.conductor import Conductor, ConductorConfig, Hit
+
+__all__ = ["Conductor", "ConductorConfig", "Hit"]